Fix fd leak from get_logger() in python 2.6

Calling get_logger({}) instantiates a logging.handlers.SyslogHandler,
which opens and keeps a socket around (either /dev/log or UDP or
whatever; not important).

Under Python 2.6, all logging handlers instantiated anywhere at all
will live for the entire lifetime of the program; they get stored in
logging._handlerList and logging._handlers. Python 2.7 is very
similar, but uses weakrefs instead of strong references in those
module-level variables, so logging handlers can actually get cleaned
up prior to program exit.

The net effect is that any program that calls get_logger() more than a
fixed number of times will leak file descriptors under Python 2.6.

This commit throws encapsulation out the window and, under 2.6 only,
replaces strong references with weakrefs in logging._handlerList and
logging._handlers, thus avoiding the leak.

Change-Id: I5dc0d1619c5a4500f892b898afd9e3668ec0ee7c
Some functional tests for object versioning

Also fix some exception-handling stuff in functional tests; at some
point, ResponseError() started needing two more parameters, but the
functional tests (not swift_test_client, just tests.py) still had a
couple spots that were not passing in the new params. Now they're
optional again, and if you omit them, then the stringification of the
ResponseError is just a little less useful than it could be.

Change-Id: I38968c4b590fc04b97b85c5f974c8648291a6689
Sanify handoff search depth with non-integer replica counts

On GET, the proxy will go search the primary nodes plus some number of
handoffs for the account/container/object before giving up and
returning a 404. That number is, by default, twice the ring's replica
count. This was fine if your ring had an integral number of replicas,
but could lead to some slightly-odd behavior if you have fractional
replicas.

For example, imagine that you have 3.49 replicas in your object ring;
perhaps you're migrating a cluster from 3 replicas to 4, and you're
being smart and doing it a bit at a time.

On an object GET that all the primary nodes 404ed, the proxy would
then compute 2 * 3.49 = 6.98, round it up to 7, and go look at 7
handoff nodes. This is sort of weird; the intent was to look at 6
handoffs for objects with 3 replicas, and 8 handoffs for objects with
4, but the effect is 7 for everybody.

You also get little latency cliffs as you scale up replica counts. If,
instead of 3.49, you had 3.51 replicas, then the proxy would look at 8
handoff nodes in every case [ceil(2 * 3.51) = 8], so there'd be a
small-but-noticeable jump in the time it takes to produce a 404.

The fix is to compute the number of handoffs based on the number of
primary nodes for the partition, not the ring's replica count. This
gets rid of the little latency cliffs and makes the behavior more like
what you get with integral replica counts.

If your ring has an integral number of replicas, there's no behavior
change here.

Change-Id: I50538941e571135299fd6b86ecd9dc780cf649f5
Documentation of the logs formats used in Swift

Change-Id: I402014627bab7113105a6a63c0b0ce1267735ca0
Mute hash_suffix rmdir failures

Change-Id: I10acfb1a3005ef988ac8a88819bf754b98cea63a
Adjust unit tests to clarify existing device usage

Most of the unit tests run with check_mount=False, so if a device is
chosen by a test that was not created ahead of time, the test will still
work, since the "tmp" directory creation in DiskFile.create() using
mkdirs(), thereby creating the a directory for the device (perhaps that
should be fixed).

This change refactors the test code to use the exiting device
explicitly.

Change-Id: Ib4fbf069596672bb0b98f41573f312eabde38f98
New container sync configuration option

Summary of the new configuration option:

The cluster operators add the container_sync middleware to their
proxy pipeline and create a container-sync-realms.conf for their
cluster and copy this out to all their proxy and container servers.
This file specifies the available container sync "realms".

A container sync realm is a group of clusters with a shared key that
have agreed to provide container syncing to one another.

The end user can then set the X-Container-Sync-To value on a
container to //realm/cluster/account/container instead of the
previously required URL.

The allowed hosts list is not used with this configuration and
instead every container sync request sent is signed using the realm
key and user key.

This offers better security as source hosts can be faked much more
easily than faking per request signatures. Replaying signed requests,
assuming it could easily be done, shouldn't be an issue as the
X-Timestamp is part of the signature and so would just short-circuit
as already current or as superceded.

This also makes configuration easier for the end user, especially
with difficult networking situations where a different host might
need to be used for the container sync daemon since it's connecting
from within a cluster. With this new configuration option, the end
user just specifies the realm and cluster names and that is resolved
to the proper endpoint configured by the operator. If the operator
changes their configuration (key or endpoint), the end user does not
need to change theirs.

DocImpact

Change-Id: Ie1704990b66d0434e4991e26ed1da8b08cb05a37
use specific hacking rules

Change-Id: I91cb5e30ec081262e2a590d64ba46fddadc7590f
Fix container quota MW for handling a bad source path

The copy source must be container/object.
This patch avoids the server to return
an internal server error when user provides
a path without a container.

Fixes: bug #1255049
Change-Id: I1a85c98d9b3a78bad40b8ceba9088cf323042412
Move all SLO functionality to middleware

This way, with zero additional effort, SLO will support enhancements
to object storage and retrieval, such as:
 * automatic resume of GETs on broken connection (today)
 * storage policies (in the near future)
 * erasure-coded object segments (in the far future)

This also lets SLOs work with other sorts of hypothetical third-party
middleware, for example object compression or encryption.

Getting COPY to work here is sort of a hack; the proxy's object
controller now checks for "swift.copy_response_hook" in the request's
environment and feeds the GET response (the source of the new object's
data) through it. This lets a COPY of a SLO manifest actually combine
the segments instead of merely copying the manifest document.

Updated ObjectController to expect a response's app_iter to be an
iterable, not just an iterator. (PEP 333 says "When called by the
server, the application object must return an iterable yielding zero
or more strings." ObjectController was just being too strict.) This
way, SLO can re-use the same response-generation logic for GET and
COPY requests.

Added a (sort of hokey) mechanism to allow middlewares to close
incompletely-consumed app iterators without triggering a warning. SLO
does this when it realizes it's performed a ranged GET on a manifest;
it closes the iterable, removes the range, and retries the
request. Without this change, the proxy logs would get 'Client
disconnected on read' in them.

DocImpact

blueprint multi-ring-large-objects

Change-Id: Ic11662eb5c7176fbf422a6fc87a569928d6f85a1
Merge "Move all SLO functionality to middleware"
Merge "New container sync configuration option"
Add missing setting to SLO sample conf

Change-Id: I3b3a306e855509d6d03223ea59dfbd109647733c
Consolidating and standardizing x-delete-at format

Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
Expose some SLO constraints via /info

These are the ones that affect what requests a client can make; the
others are just time and speed limits, so they weren't as interesting.

Change-Id: I21c19c950227f02725aafc309a3996fc6749a383
Make test_close_error run much faster

Change-Id: Iddeab82eed2f781837ab41c9e99e8450bb14a974
Mute hash_cleanup_listdir common failures

Change-Id: I0fffa387b5fae793de91870e20cab4e32e94387a
Merge "Add missing setting to SLO sample conf"
ismount should not raise exceptions

The changes from using os.path.ismount to using
swift.common.utils.ismount has caused problems since the new one
raises exceptions in cases where the old one did not. Daemons have
been encountering this and exiting; servers have been 500ing instead
of 507ing in this case, changing handoff behaviors, etc.

Since the new one was specifically written and tested for this new
behavior, I left that original function as ismount_raw and made
ismount do what it did before.

If there really isn't some reason for this new behavior, I'll be glad
to get rid of ismount_raw and just keep ismount. I couldn't see any
reason for the new behavior myself.

Change-Id: I2b5b17f9ed9656cd8804a5ed568170697d0b183d
Use constant time comparison in tempURL

Use constant time comparison when evaluating tempURL to avoid timing
attacks (CVE-2014-0006).

Fixes bug 1265665

Change-Id: I11e4ad83cc4077e52adf54a0bd0f9749294b2a48