Some functional tests for object versioning

Also fix some exception-handling stuff in functional tests; at some
point, ResponseError() started needing two more parameters, but the
functional tests (not swift_test_client, just tests.py) still had a
couple spots that were not passing in the new params. Now they're
optional again, and if you omit them, then the stringification of the
ResponseError is just a little less useful than it could be.

Change-Id: I38968c4b590fc04b97b85c5f974c8648291a6689
Mute hash_suffix rmdir failures

Change-Id: I10acfb1a3005ef988ac8a88819bf754b98cea63a
Adjust unit tests to clarify existing device usage

Most of the unit tests run with check_mount=False, so if a device is
chosen by a test that was not created ahead of time, the test will still
work, since the "tmp" directory creation in DiskFile.create() using
mkdirs(), thereby creating the a directory for the device (perhaps that
should be fixed).

This change refactors the test code to use the exiting device
explicitly.

Change-Id: Ib4fbf069596672bb0b98f41573f312eabde38f98
New container sync configuration option

Summary of the new configuration option:

The cluster operators add the container_sync middleware to their
proxy pipeline and create a container-sync-realms.conf for their
cluster and copy this out to all their proxy and container servers.
This file specifies the available container sync "realms".

A container sync realm is a group of clusters with a shared key that
have agreed to provide container syncing to one another.

The end user can then set the X-Container-Sync-To value on a
container to //realm/cluster/account/container instead of the
previously required URL.

The allowed hosts list is not used with this configuration and
instead every container sync request sent is signed using the realm
key and user key.

This offers better security as source hosts can be faked much more
easily than faking per request signatures. Replaying signed requests,
assuming it could easily be done, shouldn't be an issue as the
X-Timestamp is part of the signature and so would just short-circuit
as already current or as superceded.

This also makes configuration easier for the end user, especially
with difficult networking situations where a different host might
need to be used for the container sync daemon since it's connecting
from within a cluster. With this new configuration option, the end
user just specifies the realm and cluster names and that is resolved
to the proper endpoint configured by the operator. If the operator
changes their configuration (key or endpoint), the end user does not
need to change theirs.

DocImpact

Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
use specific hacking rules

Change-Id: I91cb5e30ec081262e2a590d64ba46fddadc7590f
Move all SLO functionality to middleware

This way, with zero additional effort, SLO will support enhancements
to object storage and retrieval, such as:
 * automatic resume of GETs on broken connection (today)
 * storage policies (in the near future)
 * erasure-coded object segments (in the far future)

This also lets SLOs work with other sorts of hypothetical third-party
middleware, for example object compression or encryption.

Getting COPY to work here is sort of a hack; the proxy's object
controller now checks for "swift.copy_response_hook" in the request's
environment and feeds the GET response (the source of the new object's
data) through it. This lets a COPY of a SLO manifest actually combine
the segments instead of merely copying the manifest document.

Updated ObjectController to expect a response's app_iter to be an
iterable, not just an iterator. (PEP 333 says "When called by the
server, the application object must return an iterable yielding zero
or more strings." ObjectController was just being too strict.) This
way, SLO can re-use the same response-generation logic for GET and
COPY requests.

Added a (sort of hokey) mechanism to allow middlewares to close
incompletely-consumed app iterators without triggering a warning. SLO
does this when it realizes it's performed a ranged GET on a manifest;
it closes the iterable, removes the range, and retries the
request. Without this change, the proxy logs would get 'Client
disconnected on read' in them.

DocImpact

blueprint multi-ring-large-objects

Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
Merge "Move all SLO functionality to middleware"
Merge "New container sync configuration option"
Add missing setting to SLO sample conf

Change-Id: I3b3a306e855509d6d03223ea59dfbd109647733c
Consolidating and standardizing x-delete-at format

Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
Expose some SLO constraints via /info

These are the ones that affect what requests a client can make; the
others are just time and speed limits, so they weren't as interesting.

Change-Id: I21c19c950227f02725aafc309a3996fc6749a383
Make test_close_error run much faster

Change-Id: Iddeab82eed2f781837ab41c9e99e8450bb14a974
Mute hash_cleanup_listdir common failures

Change-Id: I0fffa387b5fae793de91870e20cab4e32e94387a
Merge "Add missing setting to SLO sample conf"
ismount should not raise exceptions

The changes from using os.path.ismount to using
swift.common.utils.ismount has caused problems since the new one
raises exceptions in cases where the old one did not. Daemons have
been encountering this and exiting; servers have been 500ing instead
of 507ing in this case, changing handoff behaviors, etc.

Since the new one was specifically written and tested for this new
behavior, I left that original function as ismount_raw and made
ismount do what it did before.

If there really isn't some reason for this new behavior, I'll be glad
to get rid of ismount_raw and just keep ismount. I couldn't see any
reason for the new behavior myself.

Change-Id: I2b5b17f9ed9656cd8804a5ed568170697d0b183d
Use constant time comparison in tempURL

Use constant time comparison when evaluating tempURL to avoid timing
attacks (CVE-2014-0006).

Fixes bug 1265665

Change-Id: I11e4ad83cc4077e52adf54a0bd0f9749294b2a48
Remove an useless dependency to swiftclient

Unit tests in test/unit only have one dependency
to swiftclient in test_direct_client.py. This one
can be easily avoided and this patch removes it.

Change-Id: Ic1c78bc7f7fe426e8f7d8209a783342a0c4f071f
Use constant-time string compare for /info requests

If you GET /info?swiftinfo_sig=<HMAC>&swiftinfo_expires=<TIME>, then
the response contains any admin-only information that's been
registered (via calls to register_swift_info(admin=True)).

The bad news is that the info controller isn't using streq_const_time
to compare the valid HMAC signatures with the passed-in one, leading
to a possible timing attack.

The good news is that this isn't a security hole since there's
absolutely nothing interesting in the admin-only section yet, so even
if an attacker does suss out a valid signature, they don't learn
anything. However, we should still fix this in case anything
interesting ever does get added.

Change-Id: I28b6814def67200ddaa6272e09f6a55fb517fd97
Merge "Expose some SLO constraints via /info"
Merge "use specific hacking rules"