Adds a retry mechanism when deleting containers

Bulk middleware now has a mechanism to retry a delete when the
HTTP response code is 409. This happens when the container still
has objects. It can be useful in a bulk delete where you delete
all the objects in a container and then try to delete the container.
It is very likely that at the end it will fail because the replica
objects have not been deleted by the time the middleware got a
successful response.

Change-Id: I1614fcb5cc511be26a9dda90753dd08ec9546a3c
Closes-Bug: #1253478
Preserve tracebacks from run_in_thread

Now the traceback goes all the way down to where the exception came
from, not just down to run_in_thread. Better for debugging.

Change-Id: Iac6acb843a6ecf51ea2672a563d80fa43d731f23
Various optimizations to RingBuilder.rebalance()

Overall gain is about 20-22% of time on my laptop. This includes:

* replacing string sort_key with a tuple (because we can and because
  compairing two 3-tuples is faster than comparing two 26-byte strings);
* keeping track of hungriest dev in tier (allows us to use built-in
  dict.__getitem__ as a key instead of lambdas in couple of places);
* remove unnecessary sorted() call in the innermost loop (because we
  don't need to sort all of them or better we don't need to sort
  anything there);
* memoize tiers for each dev (saves just a couple of percents but why
  not).

Performance measurments have been done using this script:
http://paste.openstack.org/show/55609/

Related-Bug: #1262166
Related-Bug: #1261659
Change-Id: If38bd9fe82efc12b01e9aa146e0f4ab565fb6bea
Make POST for bulk delete actually work

Change-Id: I568e7e31df3dcbeac20dba6d543a13c0409de00e
Closes-Bug: 1232787
Allow specify arguments to .probetests script

Just so at least I can add -x or -s.

Change-Id: I95543a3086ca5fb292e7899c02646a58296c610a
Remove swiftclient dep on direct_client

Partial Implements: blueprint remove-swiftclient-dependency
Change-Id: I9af7150e5d21d50e5f880e57796314b8f05822d2
Merge "Allow specify arguments to .probetests script"
Merge "Preserve tracebacks from run_in_thread"
Fix checkmount  error parsing in swift-recon

- swift-recon now handles parsing instances where 'mounted' key (in unmounted
  and disk_usage) is an error message instead of a bool.
- Add's checkmount exception handling to the recon umounted endpoint.
- Updates existing unittest to have ismount throw an error.
- Updates unittests to cover the corner cases

Change-Id: Id51d14a8b98de69faaac84b2b34b7404b7df69e9
Up nproc limit on startup.

Separate out setrlimit calls for specific exception handling.

Closes-Bug: #1264561
Change-Id: I5588f19f8d0393409580d17317727977758d5cb3
Merge "Up nproc limit on startup."
Merge "Fix checkmount  error parsing in swift-recon"
Don't report async pendings on exception

If we encounter an exception trying to gather async pendings 'async'
doesn't exist and the cronjob ends up erroring out and leaving behind a
stale lock file.

Change-Id: I70a6d3f00bd2c9ce742e6d16af93804280707040
Whitelist external netifaces requirement

* tox.ini(testenv.install_command): Use the --allow-external and
--allow-insecure options so that pip 1.5 and later will assent to
retrieve the netifaces package even though it's not hosted on PyPI.
The --allow-insecure option is aliased to a clearer
--allow-unverified wording in 1.5, but the old form is being used to
avoid breaking users of 1.4.x and will be valid at least through
1.6.x according to comments in the pip source.

Change-Id: I587b516f3f77c930475ca2eaae5ff73c5d2ab576
Refactor to share on-disk layout with hash cleanup

Closes-Bug: 1260132
Change-Id: Iaa367c686b8dc49dd53c55a7cca661d9611044f8
Generic means for persisting system metadata.

Middleware or core features may need to store metadata
against accounts or containers. This patch adds a
generic mechanism for system metadata to be persisted
in backend databases, without polluting the user
metadata namespace, by using the reserved header
namespace x-<server_type>-sysmeta-*.

Modifications are firstly that backend servers persist
system metadata headers alongside user metadata and
other system state.

For accounts and containers, system metadata in PUT
and POST requests is treated in a similar way to user
metadata. System metadata is not yet supported for
object requests.

Secondly, changes in the proxy controllers ensure that
headers in the system metadata namespace will pass through
in requests to backend servers.

Thirdly, system metadata returned from backend servers
in GET or HEAD responses is added to the cached info
dict, which middleware can access.

Finally, a gatekeeper middleware module is provided
which filters all system metadata headers from requests
and responses by removing headers with names starting
x-account-sysmeta-, x-container-sysmeta-. The gatekeeper
also removes headers starting x-object-sysmeta- in
anticipation of future support for system metadata being
set for objects. This prevents clients from writing or
reading system metadata.

The required_filters list in swift/proxy/server.py is
modified to include the gatekeeper middleware so that
if the gatekeeper has not been configured in the
pipeline then it will be automatically inserted close
to the start of the pipeline.

blueprint cluster-federation

Change-Id: I80b8b14243cc59505f8c584920f8f527646b5f45
Remove trailing slash for consistency

Change-Id: Idd4fd116b6be226e46e33f421883b6fb34947a84
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Move the tests from functionalnosetests

Move the tests from functionalnosetests under functional, so we no
longer have two seperate trees for functional tests. This also drops
the 'nose' name from the directory, so that it doesn't end up with
confusion if we move to testr. Further, since there are no longer two
test runs in .functests, it nows looks very close to the other two.

Change-Id: I8de025c29d71f05072e257df24899927b82c1382
Merge "Don't report async pendings on exception"
Merge "Remove swiftclient dep on direct_client"