Mock time for audit test 'test_sleeper'

The object auditor unit test, test_sleeper, needs to mock out time
to ensure run variations in the test environment do not skew the
results.

Change-Id: I6ef15e88d6f760bc805e4bb6a6270751984624ff
Closes-Bug: 1321019
Fix SLO test with old simplejson

Our requirements.txt contains "simplejson>=2.0.9". With that version,
attempting to decode invalid JSON raises ValueError, not
simplejson.JSONDecodeError. In fact, there is no such thing as
JSONDecodeError; it isn't introduced until a later version. Also,
stdlib's json module doesn't have a JSONDecodeError either. However,
JSONDecodeError inherits from ValueError, so we can just catch that
instead.

Also added a better failure message when invalid JSON is detected;
we'll now fail immediately instead of later in some assertEqual().

Change-Id: I7f0db20b2d72e89c961cb43daa6888ecd878f769
Fix issues with test_wsgi.py and Storage Policies

Discovered some tests that were coupling the code under test with the
storage policies configured in /etc/swift/swift.conf.  There was some
tests that created fake rings in their tempdirs, but didn't reset or
patch the POLICIES global.  So if your local config needed more rings
that the fake's were setting up (just 2) the tests would puke when they
loaded up an app that looked for rings.  I think this probably started
happening when we added eager object ring loading back into the proxy.

 * two TestCases in test_wsgi were missing @patch_policies
 * fixed issue with patch_policies that could cause state to bleed
   between tests
 * patch_policies' legacy and default collections get a FakeRing by
   default

 * drive-by cleanup for test_loadapp_proxy() ring serialized path
   handling
 * drive-by cleanup for test_internal_client that was doing basically
   the same thing as test_wsgi

Change-Id: Ia706000ba961ed24f2c22b81041e53a0c3f302fc
Fix some spelling in comments

Change-Id: Id272dd9f603a42d9b22f80cff93716a5ed15698a
Fix swift-recon --auditor bug

I introduced a bug in swift-recon --auditor, which Christian
Schwede uncovered.  Christian proposed a fix which replaces the
generator nested_get_value with a recursive function which
returns just one instance of a key.  Christian's fix will break
the parallel object auditor patch which uses the generator to
return all instances of a particular key in a nested dictionary
and then sums those values.

This alternative fix sums over all values returned by
nested_get_value, as is done in the parallel auditor patch.

Change-Id: I6ac403395ab42d11161d6df266e8fbd5a2af9a64
Add test for swift-recon --auditor

Related-Bug: 1329785
Change-Id: I47cecd8a4cd55ca75c2a51153be7bb61c27d0ea0
Merge "Fix swift-recon --auditor bug"
Merge "Fix some spelling in comments"
Fix double request in simple client

The SimpleClient.base_request method need only to call urllib2.urlopen
once to get a response from the server.  Remove the duplicate call.

 * fixup tests that were poorly mocking urllib2
 * mock eventlet sleep because we're all busy

Change-Id: I6da93c689200351e9dbd9416b9c4bf9544571f15
Add SAIO fix for Ubuntu 12.04

The version of setup tools in Ubuntu 12.04 is too old and is causing
the SAIO instructions to fail when installing python-swiftclient.

The work around is to install python-swiftclient's dependencies
before running running:

  python setup.py develop

This change adds a note to users of Ubuntu 12.04 to replace step 2
of "Getting the code" with:

  cd $HOME/python-swiftclient; sudo pip install -r requirements.txt; \
                               python setup.py develop; cd -

Change-Id: I63f57bbf1f1158f8740f6137ad55ff49f12a316c
Closes-Bug: #1217288
Fix KeyError on Storage Policy Container Sync

In the proxy, container_info can return a 'storage_policy' of None.  When
you set a header value on a swob.Request to None that effectively just
delete's the key.  One path through the proxy during container sync was
counting on the the 'X-Backend-Storage-Policy-Index' being set which isn't
the case if the cached container_info if for a pre-policies container.

Also clean up some test cruft, tighten up the interface on FakeConn, and add
some object controller tests to exercise more interesting failure and handoff
code paths.

Change-Id: Ic379fa62634c226cc8a5a4c049b154dad70696b3
Container Auditor should log a warning if the devices path contains a non-directory.

If the devices path configured in container-server.conf contains a file
then an uncaught exception is seen in the logs. For example if file foo exists as such
/srv/1/node/foo then when the container-auditor runs, the exception that foo/containers is
not a directory is seen in the logs

This patch was essentially clayg and can be found in the bug

I tested it and wanted to get a feel of the openstack workflow so going through the
commit process

I have added a unit test as well as cleaned up and improved the unit test coverage
for this module.
- unit test for above fix is added
- unit test to verify exceptions that are raised in the module
- unit test to verify the logger's behavior
- unit test to verify mount_check behavior

Change-Id: I903b2b1e11646404cfb0551ee582a514d008c844
Closes-Bug: #1317257
Merge "Add SAIO fix for Ubuntu 12.04"
Small cleanup to ring serialization

Instead of catching TypeError and retrying without mtime, let's just
check and see if mtime is an allowed arg and decide based on
that. This way, if anything else raises TypeError we'll know about it.

Change-Id: Ib394d8176e2bc4bd53ace019ce5816369223c273
Merge "Container Auditor should log a warning if the devices path contains a non-directory."
Merge "Fix SLO test with old simplejson"
Merge "Add test for swift-recon --auditor"
SimpleClient http proxying

Previously, this code was attempting to set up http proxying but it
wasn't working. We noticed after a while when we saw traffic going
through an alternate route instead of our set of http proxies with
container sync.

Additional work and testing by clayg; thanks!

Change-Id: I840b8e55a80c13ae85c65bf68de261d735685b27
Fix pop_queue for move object twice

With the two vector timestamp change some resolution was lost in the queue
entries that could lead to the reconciler being unable to successfully remove
a processed item from the queue in pop_queue.  To ensure the queue entries
with a significant offset can be successfully removed while still handling
the re-enqueued object case issue the DELETE with the timestamp slightly later
than the maximum of the queue entries last modified time (q_record) and
misplaced objects timestamp (q_ts).

Change-Id: I4726243b3f7c4c1e98f0c578e7ffdecf4ec22199
Fix order dependent test in proxy.test_server

TestObjectController.test_POST_backend_headers was being too picky about the
order of backend requests which when pushed through eventlet will not have a
stable order.  This change preserves the expectations and assertions while
removing the dependency on the order of the requests.

Change-Id: I7176ccb9223cd3dfc3c922b8b3d81eb514891d05