Sync netifaces with global-requirements

also take the bit with the setup.py thing, seems harmless?

Change-Id: I387147235727752540a0a173a45b4b057d96899c
Mock time for audit test 'test_sleeper'

The object auditor unit test, test_sleeper, needs to mock out time
to ensure run variations in the test environment do not skew the
results.

Change-Id: I6ef15e88d6f760bc805e4bb6a6270751984624ff
Closes-Bug: 1321019
Fix SLO test with old simplejson

Our requirements.txt contains "simplejson>=2.0.9". With that version,
attempting to decode invalid JSON raises ValueError, not
simplejson.JSONDecodeError. In fact, there is no such thing as
JSONDecodeError; it isn't introduced until a later version. Also,
stdlib's json module doesn't have a JSONDecodeError either. However,
JSONDecodeError inherits from ValueError, so we can just catch that
instead.

Also added a better failure message when invalid JSON is detected;
we'll now fail immediately instead of later in some assertEqual().

Change-Id: I7f0db20b2d72e89c961cb43daa6888ecd878f769
Minor Updates to Storage Policy Docs

Fixed some grammar issues, formatting issues, clarrified some
wording now that its easier to read fully rendered.

Change-Id: Ie803dd1a84d50de7663a7099c32d81807701e188
Replace POLICY and POLICY_INDEX with string literals

Replaced throughout code base &  tox'd. Functional as well
as probe tests pass with and without policies defined.

POLICY --> 'X-Storage-Policy'
POLICY_INDEX --> 'X-Backend-Storage-Policy-Index'

Change-Id: Iea3d06de80210e9e504e296d4572583d7ffabeac
Fix issues with test_wsgi.py and Storage Policies

Discovered some tests that were coupling the code under test with the
storage policies configured in /etc/swift/swift.conf.  There was some
tests that created fake rings in their tempdirs, but didn't reset or
patch the POLICIES global.  So if your local config needed more rings
that the fake's were setting up (just 2) the tests would puke when they
loaded up an app that looked for rings.  I think this probably started
happening when we added eager object ring loading back into the proxy.

 * two TestCases in test_wsgi were missing @patch_policies
 * fixed issue with patch_policies that could cause state to bleed
   between tests
 * patch_policies' legacy and default collections get a FakeRing by
   default

 * drive-by cleanup for test_loadapp_proxy() ring serialized path
   handling
 * drive-by cleanup for test_internal_client that was doing basically
   the same thing as test_wsgi

Change-Id: Ia706000ba961ed24f2c22b81041e53a0c3f302fc
Fix some spelling in comments

Change-Id: Id272dd9f603a42d9b22f80cff93716a5ed15698a
Don't count 412 or 416 as errors in stats

The backend HTTP servers emit StatsD metrics of the form
<server>.<method>.timing and <server>.<method>.errors.timing
(e.g. object-server.GET.timing). Whether something counts as an error
or not is based on the response's HTTP status code.

Prior to this commit, "success" was 2xx, 3xx, or 404, while everything
else was considered "error". This adds 412 and 416 to the "success"
set. Like 404, these status codes indicate that we got the request and
processed it without error, but the response was "no". They shouldn't
be in the same category as statuses like 507 that indicate something
stopped the request from being processed.

Change-Id: I5582a51cf6f64aa22c890da01aaaa077f3a54202
Merge "Replace POLICY and POLICY_INDEX with string literals"
Merge "Minor Updates to Storage Policy Docs"
Fix swift-recon --auditor bug

I introduced a bug in swift-recon --auditor, which Christian
Schwede uncovered.  Christian proposed a fix which replaces the
generator nested_get_value with a recursive function which
returns just one instance of a key.  Christian's fix will break
the parallel object auditor patch which uses the generator to
return all instances of a particular key in a nested dictionary
and then sums those values.

This alternative fix sums over all values returned by
nested_get_value, as is done in the parallel auditor patch.

Change-Id: I6ac403395ab42d11161d6df266e8fbd5a2af9a64
Add test for swift-recon --auditor

Related-Bug: 1329785
Change-Id: I47cecd8a4cd55ca75c2a51153be7bb61c27d0ea0
Parallel object auditor

We are soon going to put servers with a high ratio of disk to CPU
into production as object servers.  One of our concerns with this
configuration is that the object auditor would take too long to
complete its audit cycle.  Therefore we decided to parallelise
the auditor.

The auditor already uses fork(), so we decided to use the parallel
model from the replicator.  Concurrency is set by the concurrency
parameter in the auditor stanza, which sets the number of parallel
checksum auditors.  The actual number of parallel auditing processes
is concurrency + 1 if zero_byte_fps is non-zero.

Only one ZBF process is forked, and a new ZBF process is forked as
soon as the current ZBF process finishes.  Thus the last process
running will always be a ZBF process.

Both forever and once modes are parallelised.

Each checksum auditor process submits a nested dictionary with keys
{'object_auditor_stats_ALL': {'diskn': {..}}} to dump_recon_cache
so that the object_auditor_stats_ALL dict in recon cache consists
of individual sub-dicts for each of the object disks on the server.
The recon cache is no different to before when the checksum auditor
is run in serial mode.  When swift-recon is run, it sums the stats
for the individual disks.

DocImpact

Change-Id: I0ce3db57a43e482d4be351cc522fc9060af6e2d3
Merge "Don't count 412 or 416 as errors in stats"
Merge "Fix swift-recon --auditor bug"
Merge "Fix some spelling in comments"
Fix double request in simple client

The SimpleClient.base_request method need only to call urllib2.urlopen
once to get a response from the server.  Remove the duplicate call.

 * fixup tests that were poorly mocking urllib2
 * mock eventlet sleep because we're all busy

Change-Id: I6da93c689200351e9dbd9416b9c4bf9544571f15
Add SAIO fix for Ubuntu 12.04

The version of setup tools in Ubuntu 12.04 is too old and is causing
the SAIO instructions to fail when installing python-swiftclient.

The work around is to install python-swiftclient's dependencies
before running running:

  python setup.py develop

This change adds a note to users of Ubuntu 12.04 to replace step 2
of "Getting the code" with:

  cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; \
                               python setup.py develop; cd -

Change-Id: I63f57bbf1f1158f8740f6137ad55ff49f12a316c
Closes-Bug: #1217288
Fix KeyError on Storage Policy Container Sync

In the proxy, container_info can return a 'storage_policy' of None.  When
you set a header value on a swob.Request to None that effectively just
delete's the key.  One path through the proxy during container sync was
counting on the the 'X-Backend-Storage-Policy-Index' being set which isn't
the case if the cached container_info if for a pre-policies container.

Also clean up some test cruft, tighten up the interface on FakeConn, and add
some object controller tests to exercise more interesting failure and handoff
code paths.

Change-Id: Ic379fa62634c226cc8a5a4c049b154dad70696b3
Container Auditor should log a warning if the devices path contains a non-directory.

If the devices path configured in container-server.conf contains a file
then an uncaught exception is seen in the logs. For example if file foo exists as such
/srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is
not a directory is seen in the logs

This patch was essentially clayg and can be found in the bug

I tested it and wanted to get a feel of the openstack workflow so going through the
commit process

I have added a unit test as well as cleaned up and improved the unit test coverage
for this module.
- unit test for above fix is added
- unit test to verify exceptions that are raised in the module
- unit test to verify the logger's behavior
- unit test to verify mount_check behavior

Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844
Closes-Bug: #1317257