Adds a retry mechanism when deleting containers

Bulk middleware now has a mechanism to retry a delete when the
HTTP response code is 409. This happens when the container still
has objects. It can be useful in a bulk delete where you delete
all the objects in a container and then try to delete the container.
It is very likely that at the end it will fail because the replica
objects have not been deleted by the time the middleware got a
successful response.

Change-Id: I1614fcb5cc511be26a9dda90753dd08ec9546a3c
Closes-Bug: #1253478
Various optimizations to RingBuilder.rebalance()

Overall gain is about 20-22% of time on my laptop. This includes:

* replacing string sort_key with a tuple (because we can and because
  compairing two 3-tuples is faster than comparing two 26-byte strings);
* keeping track of hungriest dev in tier (allows us to use built-in
  dict.__getitem__ as a key instead of lambdas in couple of places);
* remove unnecessary sorted() call in the innermost loop (because we
  don't need to sort all of them or better we don't need to sort
  anything there);
* memoize tiers for each dev (saves just a couple of percents but why
  not).

Performance measurments have been done using this script:
http://paste.openstack.org/show/55609/

Related-Bug: #1262166
Related-Bug: #1261659
Change-Id: If38bd9fe82efc12b01e9aa146e0f4ab565fb6bea
Refactor to share on-disk layout with hash cleanup

Closes-Bug: 1260132
Change-Id: Iaa367c686b8dc49dd53c55a7cca661d9611044f8
Generic means for persisting system metadata.

Middleware or core features may need to store metadata
against accounts or containers. This patch adds a
generic mechanism for system metadata to be persisted
in backend databases, without polluting the user
metadata namespace, by using the reserved header
namespace x-<server_type>-sysmeta-*.

Modifications are firstly that backend servers persist
system metadata headers alongside user metadata and
other system state.

For accounts and containers, system metadata in PUT
and POST requests is treated in a similar way to user
metadata. System metadata is not yet supported for
object requests.

Secondly, changes in the proxy controllers ensure that
headers in the system metadata namespace will pass through
in requests to backend servers.

Thirdly, system metadata returned from backend servers
in GET or HEAD responses is added to the cached info
dict, which middleware can access.

Finally, a gatekeeper middleware module is provided
which filters all system metadata headers from requests
and responses by removing headers with names starting
x-account-sysmeta-, x-container-sysmeta-. The gatekeeper
also removes headers starting x-object-sysmeta- in
anticipation of future support for system metadata being
set for objects. This prevents clients from writing or
reading system metadata.

The required_filters list in swift/proxy/server.py is
modified to include the gatekeeper middleware so that
if the gatekeeper has not been configured in the
pipeline then it will be automatically inserted close
to the start of the pipeline.

blueprint cluster-federation

Change-Id: I80b8b14243cc59505f8c584920f8f527646b5f45
Move the tests from functionalnosetests

Move the tests from functionalnosetests under functional, so we no
longer have two seperate trees for functional tests. This also drops
the 'nose' name from the directory, so that it doesn't end up with
confusion if we move to testr. Further, since there are no longer two
test runs in .functests, it nows looks very close to the other two.

Change-Id: I8de025c29d71f05072e257df24899927b82c1382
Add Swift-account-stats as a related project.

Change-Id: If2620a6b448581697d0d049578ea28faead6fbcf
Merge "Adds a retry mechanism when deleting containers"
Raise diskfile.py module coverage to > 98%

We attempt to get the code coverage (with branch coverage) to 100%,
but fall short because due to interactions between coverage.py and
CPython's peephole optimizer. See:

    https://bitbucket.org/ned/coveragepy/issue/198/continue-marked-as-not-covered

In the main diskfile module, we remove the check for a valid
"self._tmppath" since it is only one of a number of fields that could
be verified and it was not worth trying to get coverage for it. We
also remove the try / except around the close() method call in the
DiskFileReader's app_iter_ranges() method since it will never be
called in a context that will raise a quarantine exception (by
definition ranges can't generate a quarantine event).

We also:

* fix where quarantine messages are checked to ensure the
  generator is actually executed before the check
* in new and modified tests:
  * use assertTrue in place of assert_
  * use assertEqual in place of assertEquals
* fix references to the reserved word "object"

Change-Id: I6379be04adfc5012cb0b91748fb3ba3f11200b48
Merge "Refactor to share on-disk layout with hash cleanup"
Merge "Move the tests from functionalnosetests"
Merge "Add Swift-account-stats as a related project."
Merge "Raise diskfile.py module coverage to > 98%"
Merge "Various optimizations to RingBuilder.rebalance()"
Merge "Generic means for persisting system metadata."