Let eventlet.wsgi.server log tracebacks when eventlet_debug is enabled

The "logging" available in eventlet.wsgi.server/BaseHTTPServer doesn't
generally suite our needs, so it should be bypassed using a NullLogger in
production.  But in development it can be useful if tracebacks generated from
inside eventlet.wsgi (say a NameError in DiskFile.__iter__) end up in logs.
Since we already have eventlet_debug parsed inside of run_server we can skip
the NullLogger bypass and let stuff blast out to STDERR when configured for
development/debug logging.

Change-Id: I20a9e82c7fed8948bf649f1f8571b4145fca201d
direct_client not passing args between some functions

The call to _get_direct_account_container in direct_get_account
has several of its args =None instead of set to the value passed
to direct_get_account.

The same applies to _get_direct_account_container in
direct_get_container.

The direct_get_container is only called by the account-reaper
and this bug will have limited impact on it. The marker,
maintained in reap_container, is ignored by direct_get_container.
This is not as bad as it sounds, if the account-reaper successfully
deletes the first 10K objects, assuming the container has > 10K
objects, the next call to direct_get_container will in fact return
the next 10K objects even though it sets marker=None (assuming the
first 10K objects were successfully deleted).

This patch also updates test_direct_get_account and
test_direct_get_container to ensure the appropriate
args are included in the connection query_string.

Closes-Bug: #1369558
Change-Id: If1c8aa1240d38354ebc9b1ebca92dc1c8c36cb5f
Remove unused option from ObjectReplicator.__init__

disk_chunk_size is unused by the ObjectReplicator class. Note that it
*is* used by the object replicator process; it's just that
DiskFileManager is the one that pulls it out of the conf and pays
attention to it. Keeping it as an attribute on ObjectReplicator is
unnecessary.

Change-Id: I1eeef7b33873b4c8bb269ca02dcb067098b6fded
Add new features to swift-drive-audit

This patch adds two new features to swift-drive-audit. The first
is an option in the drive-audit.conf file that allows the operator
to prevent the drives ever being unmounted automatically,
regardless of the amount of errors present. This could be of
benefit in very small systems consisting of only one or two drives
where the operator would like to manually unmount/fix the
particular drive(s) and minimise any potential downtime.

The second is another option in drive-audit.conf that allows the
operator to select a recon directory. This directory will then
have a drive.recon file which will keep an up-to-date record of
the swift drives and any errors associated with them. An example
of the output would be as follows:

{"/srv/node/disk2": "0", "/srv/node/disk3": "25", "/srv/node/disk0": "0",
"/srv/node/disk1": "0", "/srv/node/disk10": "0", "/srv/node/disk7": "0",
"/srv/node/disk4": "137", "/srv/node/disk5": "0", "/srv/node/disk8": "0",
"/srv/node/disk9": "0", "/srv/node/disk6": "0", "/srv/node/disk11": "60"}

This would allow the operator to monitor the errors on the swift
drives without having to spend time searching through logs. Also, if
this is accepted, it should be possible to add an option to
swift-recon that would keep track of this at a system level.

Change-Id: Ib5dacf8622b7363e070c274c7c30c8ead448a055
use get_container_info in ratelimit

Everyone else cooperates to store the container_info cache in the env the
first time it's requested.  This should save a duplicate memcache hit on
requests that ratelimiting looks at.

Change-Id: Ic6411d4619db6b53fafe9fdbf1d0a370d1258c38
updated hacking rules

1) Added comment for H231, which we were already enforcing. H231
is for Python 3.x compatible except statements.

2) Added check for H201, which we were enforcing in reviews
but waiting on hacking checks to be updated. H201 is for bare
except statements, and the update in upstream hacking is to
support the "  # noqa" flag on it.

The H201 check catches some existing bare excepts that are fixed.

Change-Id: I68638aa9ea925ef62f9035a426548c2c804911a8
Provides proper error handling on builder unpickle

This patch provides the necessary error handling while unpickling
a builder file. Earlier if a builder file is empty/invalid/corrupted,
the stacktrace was shown to user with an exit code of 1. This fixes it
to show a user-friendly message and also returns the exit code of 2,
indicating there was a failure.

Change-Id: I51eb24702c422299629f8053d4591dd10f5863f8
Closes-Bug: #1370680
Fixes unit tests to clean up temporary directories

This patch fixes the unit tests to remove the temporary directories
created during run of unit tests. Some of unit tests did not tear down
correctly, whatever it had set it up for running. This would over period
of time bloat up the tmp directory. As on date, there were around 49 tmp
directories left uncleared per round of unit tests. This patch fixes it.

Change-Id: If591375ca9cc87d52c7c9c6dc16c9fb4b49e99fc
Incorrect variable name in the diskfile.py

You should use the suffix variable instead of the hash_ variable when
updates the hashes dictionary in the get_hashes().

Fixes Bug1374777

Change-Id: If9a7ab5b6d271385e4fdc79ff9c216047a2f4d7e
Merge "direct_client not passing args between some functions"
Imported Translations from Transifex

Change-Id: I59b68ecb6f07e2b71e4655699d0165007d764718
Merge "Remove unused option from ObjectReplicator.__init__"
Merge "use get_container_info in ratelimit"
Fix up the return value of launch()

This is "obviously" the right thing to do, except of course it's
pure sugar: the return value is not used anywhere. Except if someone
has a script that imports the whole thing somehow and then does
isinstance(dict). Because that is so much easier than submitting
a patch, I can imagine someone, somewhere doing that.

The fix came in a patch by dfg, see review 121851.

Change-Id: I39ddf70dc2d027b7db6e31097400248dc66eb137
Merge "Incorrect variable name in the diskfile.py"
Limit partition movement when adding a new tier

When adding a new tier (region, zone, node, device) to an existing,
already balanced ring all existing partitions in the existing tiers of
the same level are gathered for reassigning, even when there is
not enough space in the new tier. This will create a lot of unnecessary
replication traffic in the backend network.

For example, when only one region exists in the ring and a new region is
added, all existing parts are selected to reassign, even when the new
region has a total weight of 0. Same for zones, nodes and devices.

This patch limits the number of partitions that are choosen to reassign
by checking for devices on other tiers that are asking for more
partitions.

Failed devices are not considered when applying the limit.

Co-Authored By: Florent Flament <florent.flament-ext@cloudwatt.com>
Change-Id: I6178452e47492da4677a8ffe4fb24917b5968cd9
Closes-Bug: 1367826
Some statements are evaluated twice in the setUp of the
TestObjectReplicator

Remove the duplicated statements.

Closes-Bug: #1374783

Change-Id: If2b55e864fea497d7a7b218adf11eb7749c27765
Merge "Some statements are evaluated twice in the setUp of the TestObjectReplicator"
fix a comment

Change-Id: I6310390f756bc0d326e56fceef32b630f0fe1ca1
Fix ring-builder crash.

If you adjust the replica count after adding devices but before the
first rebalance, the ring builder would crash.

To reproduce:

    $ swift-ring-builder crashy.builder create 8 3 1
    $ swift-ring-builder crashy.builder add r1z1-10.1.1.1:6000/sda1 1
    $ swift-ring-builder crashy.builder set_replicas 5
    $ swift-ring-builder crashy.builder rebalance

Change-Id: Id91cf8680961ccf6e3db153577e99cad545bee9d