Fix fd leak from get_logger() in python 2.6

Calling get_logger({}) instantiates a logging.handlers.SyslogHandler,
which opens and keeps a socket around (either /dev/log or UDP or
whatever; not important).

Under Python 2.6, all logging handlers instantiated anywhere at all
will live for the entire lifetime of the program; they get stored in
logging._handlerList and logging._handlers. Python 2.7 is very
similar, but uses weakrefs instead of strong references in those
module-level variables, so logging handlers can actually get cleaned
up prior to program exit.

The net effect is that any program that calls get_logger() more than a
fixed number of times will leak file descriptors under Python 2.6.

This commit throws encapsulation out the window and, under 2.6 only,
replaces strong references with weakrefs in logging._handlerList and
logging._handlers, thus avoiding the leak.

Change-Id: I5dc0d1619c5a4500f892b898afd9e3668ec0ee7c
Some functional tests for object versioning

Also fix some exception-handling stuff in functional tests; at some
point, ResponseError() started needing two more parameters, but the
functional tests (not swift_test_client, just tests.py) still had a
couple spots that were not passing in the new params. Now they're
optional again, and if you omit them, then the stringification of the
ResponseError is just a little less useful than it could be.

Change-Id: I38968c4b590fc04b97b85c5f974c8648291a6689
Sanify handoff search depth with non-integer replica counts

On GET, the proxy will go search the primary nodes plus some number of
handoffs for the account/container/object before giving up and
returning a 404. That number is, by default, twice the ring's replica
count. This was fine if your ring had an integral number of replicas,
but could lead to some slightly-odd behavior if you have fractional
replicas.

For example, imagine that you have 3.49 replicas in your object ring;
perhaps you're migrating a cluster from 3 replicas to 4, and you're
being smart and doing it a bit at a time.

On an object GET that all the primary nodes 404ed, the proxy would
then compute 2 * 3.49 = 6.98, round it up to 7, and go look at 7
handoff nodes. This is sort of weird; the intent was to look at 6
handoffs for objects with 3 replicas, and 8 handoffs for objects with
4, but the effect is 7 for everybody.

You also get little latency cliffs as you scale up replica counts. If,
instead of 3.49, you had 3.51 replicas, then the proxy would look at 8
handoff nodes in every case [ceil(2 * 3.51) = 8], so there'd be a
small-but-noticeable jump in the time it takes to produce a 404.

The fix is to compute the number of handoffs based on the number of
primary nodes for the partition, not the ring's replica count. This
gets rid of the little latency cliffs and makes the behavior more like
what you get with integral replica counts.

If your ring has an integral number of replicas, there's no behavior
change here.

Change-Id: I50538941e571135299fd6b86ecd9dc780cf649f5
Documentation of the logs formats used in Swift

Change-Id: I402014627bab7113105a6a63c0b0ce1267735ca0
Mute hash_suffix rmdir failures

Change-Id: I10acfb1a3005ef988ac8a88819bf754b98cea63a
Adjust unit tests to clarify existing device usage

Most of the unit tests run with check_mount=False, so if a device is
chosen by a test that was not created ahead of time, the test will still
work, since the "tmp" directory creation in DiskFile.create() using
mkdirs(), thereby creating the a directory for the device (perhaps that
should be fixed).

This change refactors the test code to use the exiting device
explicitly.

Change-Id: Ib4fbf069596672bb0b98f41573f312eabde38f98
Fix container quota MW for handling a bad source path

The copy source must be container/object.
This patch avoids the server to return
an internal server error when user provides
a path without a container.

Fixes: bug #1255049
Change-Id: I1a85c98d9b3a78bad40b8ceba9088cf323042412
Consolidating and standardizing x-delete-at format

Change-Id: Idc916da1c7fe1cc43a2c26f7f7ee1d4fcdd52c89
Make test_close_error run much faster

Change-Id: Iddeab82eed2f781837ab41c9e99e8450bb14a974
Mute hash_cleanup_listdir common failures

Change-Id: I0fffa387b5fae793de91870e20cab4e32e94387a
ismount should not raise exceptions

The changes from using os.path.ismount to using
swift.common.utils.ismount has caused problems since the new one
raises exceptions in cases where the old one did not. Daemons have
been encountering this and exiting; servers have been 500ing instead
of 507ing in this case, changing handoff behaviors, etc.

Since the new one was specifically written and tested for this new
behavior, I left that original function as ismount_raw and made
ismount do what it did before.

If there really isn't some reason for this new behavior, I'll be glad
to get rid of ismount_raw and just keep ismount. I couldn't see any
reason for the new behavior myself.

Change-Id: I2b5b17f9ed9656cd8804a5ed568170697d0b183d
Remove an useless dependency to swiftclient

Unit tests in test/unit only have one dependency
to swiftclient in test_direct_client.py. This one
can be easily avoided and this patch removes it.

Change-Id: Ic1c78bc7f7fe426e8f7d8209a783342a0c4f071f
Use constant-time string compare for /info requests

If you GET /info?swiftinfo_sig=<HMAC>&swiftinfo_expires=<TIME>, then
the response contains any admin-only information that's been
registered (via calls to register_swift_info(admin=True)).

The bad news is that the info controller isn't using streq_const_time
to compare the valid HMAC signatures with the passed-in one, leading
to a possible timing attack.

The good news is that this isn't a security hole since there's
absolutely nothing interesting in the admin-only section yet, so even
if an attacker does suss out a valid signature, they don't learn
anything. However, we should still fix this in case anything
interesting ever does get added.

Change-Id: I28b6814def67200ddaa6272e09f6a55fb517fd97
Let swift-object-info skip etag verification

Now swift-object-info has a "-n" option that will cause the etag
verification to be skipped; on large objects, the etag verification
takes the vast majority of the runtime, and sometimes you just want to
know which account owns the 5 GB monstrosity without waiting around while
its checksum is verified.

Change-Id: Id284570633eb7b98046cdb948d7c37a152de1195
Remove dependencies on pep8, pyflakes and flake8

They should be determined by the hacking dependency
implicitely.

Change-Id: I8805f7b0ef9f2b1f59be342e9c6d032330707688
Merge "Consolidating and standardizing x-delete-at format"
Fix Error 400 Header Line Too Long

Fix Error 400 Header Line Too Long when using Identity v3 PKI Tokens

Uses swift.conf max_header_size option to set wsgi.MAX_HEADER_LINE,
allowing the operator to customize this parameter.

The default value has been let to 8192 to avoid unexpected
configuration change on deployed platforms. The max_header_size option
has to be increased (for example to 16384), to accomodate for large
Identity v3 PKI tokens, including more than 7 catalog entries.

The default max header line size of 8192 is exceeded in the following
scenario:
- Auth tokens generated by Keystone v3 API include the catalog.
- Keystone's catalog contains more than 7 services.

Similar fixes have been merged in other projects.

Change-Id: Ia838b18331f57dfd02b9f71d4523d4059f38e600
Closes-Bug: 1190149
Drop origin from get_info calls

The reason for this is that having origin in the get_info calls causes an
infinite loop. The way that code was written it relies on GETorHEAD_base to
populate the data- the only problem is that the HEAD call is wrapped by
cors_validation which calls get_info and round and round we go. imo get_info
should be refactored to not work this way (relying on this other call to do
stuff behind scenes and then magically your stuff is there) because it seems
pretty prone to breaking. But I'll let somebody else do that :).

Fixes bug 1270039

Change-Id: Idad3cedd965e0d5fb068b062fe8fef301c87b75a
Fix swift-dispersion-report when partitions overlap

swift-dispersion-report tries to avoid checking a partition more than
once, so it keeps track of partitions already queried and skips
duplicates.

swift-dispersion-report also keeps track of the number of successful
responses; it counts the number of expected replicas to find, and also
counts the number of replicas actually found, and tells the operator
if the numbers differ.

However, in the case that a partition was duplicated, the
expected-responses counter was incremented, but the actual check was
skipped, so it looked as though some copies were missing. Now we only
increment the expected-responses counter if we're actually going to
perform the check.

Change-Id: I22ac2b8066b62ca7c8ebf099c9f602118bb1a298
release notes and authors update for 1.12

Change-Id: I76c13957a7758287fd40dd2e9fb85277d9288977