Sync netifaces with global-requirements

also take the bit with the setup.py thing, seems harmless?

Change-Id: I387147235727752540a0a173a45b4b057d96899c
Mock time for audit test 'test_sleeper'

The object auditor unit test, test_sleeper, needs to mock out time
to ensure run variations in the test environment do not skew the
results.

Change-Id: I6ef15e88d6f760bc805e4bb6a6270751984624ff
Closes-Bug: 1321019
Fix SLO test with old simplejson

Our requirements.txt contains "simplejson>=2.0.9". With that version,
attempting to decode invalid JSON raises ValueError, not
simplejson.JSONDecodeError. In fact, there is no such thing as
JSONDecodeError; it isn't introduced until a later version. Also,
stdlib's json module doesn't have a JSONDecodeError either. However,
JSONDecodeError inherits from ValueError, so we can just catch that
instead.

Also added a better failure message when invalid JSON is detected;
we'll now fail immediately instead of later in some assertEqual().

Change-Id: I7f0db20b2d72e89c961cb43daa6888ecd878f769
Fix issues with test_wsgi.py and Storage Policies

Discovered some tests that were coupling the code under test with the
storage policies configured in /etc/swift/swift.conf.  There was some
tests that created fake rings in their tempdirs, but didn't reset or
patch the POLICIES global.  So if your local config needed more rings
that the fake's were setting up (just 2) the tests would puke when they
loaded up an app that looked for rings.  I think this probably started
happening when we added eager object ring loading back into the proxy.

 * two TestCases in test_wsgi were missing @patch_policies
 * fixed issue with patch_policies that could cause state to bleed
   between tests
 * patch_policies' legacy and default collections get a FakeRing by
   default

 * drive-by cleanup for test_loadapp_proxy() ring serialized path
   handling
 * drive-by cleanup for test_internal_client that was doing basically
   the same thing as test_wsgi

Change-Id: Ia706000ba961ed24f2c22b81041e53a0c3f302fc
Add test for swift-recon --auditor

Related-Bug: 1329785
Change-Id: I47cecd8a4cd55ca75c2a51153be7bb61c27d0ea0
Parallel object auditor

We are soon going to put servers with a high ratio of disk to CPU
into production as object servers.  One of our concerns with this
configuration is that the object auditor would take too long to
complete its audit cycle.  Therefore we decided to parallelise
the auditor.

The auditor already uses fork(), so we decided to use the parallel
model from the replicator.  Concurrency is set by the concurrency
parameter in the auditor stanza, which sets the number of parallel
checksum auditors.  The actual number of parallel auditing processes
is concurrency + 1 if zero_byte_fps is non-zero.

Only one ZBF process is forked, and a new ZBF process is forked as
soon as the current ZBF process finishes.  Thus the last process
running will always be a ZBF process.

Both forever and once modes are parallelised.

Each checksum auditor process submits a nested dictionary with keys
{'object_auditor_stats_ALL': {'diskn': {..}}} to dump_recon_cache
so that the object_auditor_stats_ALL dict in recon cache consists
of individual sub-dicts for each of the object disks on the server.
The recon cache is no different to before when the checksum auditor
is run in serial mode.  When swift-recon is run, it sums the stats
for the individual disks.

DocImpact

Change-Id: I0ce3db57a43e482d4be351cc522fc9060af6e2d3
Add SAIO fix for Ubuntu 12.04

The version of setup tools in Ubuntu 12.04 is too old and is causing
the SAIO instructions to fail when installing python-swiftclient.

The work around is to install python-swiftclient's dependencies
before running running:

  python setup.py develop

This change adds a note to users of Ubuntu 12.04 to replace step 2
of "Getting the code" with:

  cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; \
                               python setup.py develop; cd -

Change-Id: I63f57bbf1f1158f8740f6137ad55ff49f12a316c
Closes-Bug: #1217288
Fix KeyError on Storage Policy Container Sync

In the proxy, container_info can return a 'storage_policy' of None.  When
you set a header value on a swob.Request to None that effectively just
delete's the key.  One path through the proxy during container sync was
counting on the the 'X-Backend-Storage-Policy-Index' being set which isn't
the case if the cached container_info if for a pre-policies container.

Also clean up some test cruft, tighten up the interface on FakeConn, and add
some object controller tests to exercise more interesting failure and handoff
code paths.

Change-Id: Ic379fa62634c226cc8a5a4c049b154dad70696b3
Container Auditor should log a warning if the devices path contains a non-directory.

If the devices path configured in container-server.conf contains a file
then an uncaught exception is seen in the logs. For example if file foo exists as such
/srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is
not a directory is seen in the logs

This patch was essentially clayg and can be found in the bug

I tested it and wanted to get a feel of the openstack workflow so going through the
commit process

I have added a unit test as well as cleaned up and improved the unit test coverage
for this module.
- unit test for above fix is added
- unit test to verify exceptions that are raised in the module
- unit test to verify the logger's behavior
- unit test to verify mount_check behavior

Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844
Closes-Bug: #1317257
Merge "Add SAIO fix for Ubuntu 12.04"
Small cleanup to ring serialization

Instead of catching TypeError and retrying without mtime, let's just
check and see if mtime is an allowed arg and decide based on
that. This way, if anything else raises TypeError we'll know about it.

Change-Id: Ib394d8176e2bc4bd53ace019ce5816369223c273
Merge "Container Auditor should log a warning if the devices path contains a non-directory."
Merge "Fix SLO test with old simplejson"
Merge "Add test for swift-recon --auditor"
SimpleClient http proxying

Previously, this code was attempting to set up http proxying but it
wasn't working. We noticed after a while when we saw traffic going
through an alternate route instead of our set of http proxies with
container sync.

Additional work and testing by clayg; thanks!

Change-Id: I840b8e55a80c13ae85c65bf68de261d735685b27
Fix pop_queue for move object twice

With the two vector timestamp change some resolution was lost in the queue
entries that could lead to the reconciler being unable to successfully remove
a processed item from the queue in pop_queue.  To ensure the queue entries
with a significant offset can be successfully removed while still handling
the re-enqueued object case issue the DELETE with the timestamp slightly later
than the maximum of the queue entries last modified time (q_record) and
misplaced objects timestamp (q_ts).

Change-Id: I4726243b3f7c4c1e98f0c578e7ffdecf4ec22199
Fix order dependent test in proxy.test_server

TestObjectController.test_POST_backend_headers was being too picky about the
order of backend requests which when pushed through eventlet will not have a
stable order.  This change preserves the expectations and assertions while
removing the dependency on the order of the requests.

Change-Id: I7176ccb9223cd3dfc3c922b8b3d81eb514891d05
Fix exception raising in FakeConn

Timeout isn't an Exception, so Timeouts in tests weren't getting
raised. Instead, you'd sometimes have an HTTPResponse's .status be a
Timeout object, not an integer, which greatly confuses code that
expects an integer.

Also reorder the test that exposed the failure in the gate so it blows
up most times instead of sometimes do demonstrate the failure with out
this fix to FakeConn.

Change-Id: I76367a0575f84cad6b2f03e814f3f16bf96bc7d1
Fix potential missing key error in container_info

If upgrading from a non-storage policy enabled version of
swift to a storage policy enabled version its possible that
memcached will have an info structure that does not contain
the 'storage_policy" key resulting in an unhandled exception
during the lookup.  The fix is to simply make sure we never
return the dict without a storage_policy key defined; if it
doesn't exist its safe to make it '0' as this means you're
in the update scenario and there's xno other possibility.

Change-Id: If8e8f66d32819c5bfb2d1308e14643f3600ea6e9
Merge "Fix pop_queue for move object twice"