Some functional tests for static large objects

There's some sort-of-hacky code in there to detect SLO support in
order to skip tests when SLO is off so that the functests won't fail
on older clusters.

Change-Id: I6ad5974a0db7213747b0f4497d08ffc706d3f220
Reorganize SLO unit tests

This brings some sanity to the SLO test app (the thing the middleware
wraps in the unit tests) as well as splits things into multiple test
classes.

This is part of the effort to move all SLO functionality to
middleware, but it's a separate commit to prove that these test
changes don't harm anything when running against the old code.

Change-Id: I52a16f15a80dfaf9b3c595b0e634d52f418caf6c
in case you lose your builder backups

Change-Id: Ica555be2be492c3ec5fdeab738058ff35989a603
workaround probetest race from early response

Change-Id: I594633887c86fc2212850409a37ee2257633a23c
test: improve db_replicator coverage

This patch adds a test for ReplicatorRpc.complete_rsync()
and complete extract_device() coverage.

test_extract_device:
  test the case the parameter is invalid

test_complete_rsync_with_bad_input:
  ensure the use of invalid parameters return a 404 erro

test_complete_rsync:
  validate the returned code in case of success

Change-Id: I59e0d26a1efe59d8beff1e81c2a7edc6de0872e9
Added discoverable capabilities.

Swift can now optionally be configured to allow requests to '/info',
providing information about the swift cluster.  Additionally a HMAC
signed requests to
'/info?swiftinfo_sig=<sign>&swiftinfo_expires=<expires>' can be
configured allowing privileged access to more sensitive information
not meant to be public.

DocImpact
Change-Id: I2379360fbfe3d9e9e8b25f1dc34517d199574495
Implements: blueprint capabilities
Closes-Bug: #1245694
slightly less early quorum

The early quorum change has maybe added a little bit too much
eventual to the consistency of requests in Swift, and users can
sometimes get unexpected
results.

This change gives us a knob to turn in finding the right balance,
by adding a timeout where pending requests can finish after quorum
is achieved.

Change-Id: Ife91aaa8653e75b01313bbcf19072181739e932c
Protect against hash cleanup errors on PUTs

In http://launchpad.net/bugs/1254405 an exception occurs when
finalizing the PUT of an object, but it is obscured by the thread pool
code, so we don't see the actual line where it originated. However,
there are two possible functions where this exception could originate:

    1. renamer()
    2. hash_cleanup_listdir()

If this is an error in renamer(), there is some other waky problem
where the temporary file has been removed. It is likely that this is a
problem where a file name from os.listdir ends up disappearing (but
even this is a rare occurence). Regardless, it is not clear that we
really want an error from hash_cleanup_listdir() from affecting the
return result of the PUT.

To that end, we squelsh OSErrors from hash_cleanup_listdir() for
now. One might argue for all errors, but since os.unlink() and
os.listdir() raise OSError today, that is probably sufficient.

Even that we use "Closes-Bug" below, it is not clear it can even be
determined that this closes that bug report.

Change-Id: I2f55df835c387e4d17cffda74c04c9994aebbe1f
Closes-Bug: 1254405
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Remove unnecessary "is not None" check

From a review comment on https://review.openstack.org/30051 remove
the "is not None" check, as the assignment in the try block will
never assign None as value as the int() built-in will not return it.

There was some concern of long-term maintenance of the DiskFile
class's _quarantine method raising exceptions. If that routine were
ever mistakenly changed to NOT raise an exception, subtle problems
could creep into the code (see https://review.openstack.org/53237).
We address this concern by raising an exception explicitly at the
call sites of DiskFile._quarantine().

Change-Id: I1729a2d77a6b72b4494b24a8838b47ad5272c075
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Push cooperative sleep call down into ThreadPool

The PUT REST API has no idea how writes are performed, so when thread
pools are in use, the sleep is not necessary, though it is still
necessary when thread pools are not in use. Since the ThreadPool
object knows when threads are actually in use, it can take care of
being cooperative with the eventlet hub.

In addition, we can hide the cooperative iterator hook, given that the
only other consumer of this was the auditor, which does not need it
any longer.  The only consumer of the DiskFile class that wants the
cooperative behavior is the REST API layer of the object server, which
is also using thread pools.

Change-Id: Ibc4ac672899f9a35fd68c85d7f56403c19b4f991
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Report path information in failure exceptions

When an error occurs during functional tests that use the
swift_test_client module, the reported error message includes the
method and path:

    ResponseError: 500: 'Internal Error' ('HEAD' \
    '/v1/AUTH_test/d5ce...')

Change-Id: I631cd9e83879fb644778d4ded62625483bf38045
Fix bug in obj updater run_once().

The "not" in front of the ismount() call got accidentally dropped in a
recent change.  This patch adds it back along with a few more tests.

Note that this bug only showed up on an SAIO during probe tests because
I used actually-mounted (virtual) "disks".  So keep that in mind when
building SAIOs for development/testing.

Change-Id: Ia193f3c4b73203605954036863575c22ddab6b03
Merge "slightly less early quorum"
Add HTML reporting for test branch coverage.

When including branch coverage results, also generate HTML reports into
a "cover" subdirectory under the directory in which .unittests resides
(i.e. known location at the top of the swift tree).

Change-Id: I493d74f38755f7bf0d7043052585efb27840b238
Merge "Fix bug in obj updater run_once()."
Merge "Add HTML reporting for test branch coverage."
Merge "Some functional tests for static large objects"
Use TCP_NODELAY for created sockets.

Mark Seger at HP has been looking at small objects, 1 and 2 KB size,
and with Rick Jones' help noticed that TCP protocol traces showed
effects from the Nagel algorithm client-to-server and
server-to-client.

This patch just addresses our WSGI server responses, but does not
address out-bound connections from the various servers.

Change-Id: I11f86df1f56fba1c6ab6084dc1f580c395f072dc
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Account for a platform difference in semaphores

On OS X (and probably other Operating Systems) it isn't possible to
introspect the value of a semaphore. Account for this by skipping a
test about this.

Change-Id: I97824f9fc4e36de4f7a62c8ce53865e6977dfdfe
Merge "Reorganize SLO unit tests"