Fix fd leak from get_logger() in python 2.6

Calling get_logger({}) instantiates a logging.handlers.SyslogHandler,
which opens and keeps a socket around (either /dev/log or UDP or
whatever; not important).

Under Python 2.6, all logging handlers instantiated anywhere at all
will live for the entire lifetime of the program; they get stored in
logging._handlerList and logging._handlers. Python 2.7 is very
similar, but uses weakrefs instead of strong references in those
module-level variables, so logging handlers can actually get cleaned
up prior to program exit.

The net effect is that any program that calls get_logger() more than a
fixed number of times will leak file descriptors under Python 2.6.

This commit throws encapsulation out the window and, under 2.6 only,
replaces strong references with weakrefs in logging._handlerList and
logging._handlers, thus avoiding the leak.

Change-Id: I5dc0d1619c5a4500f892b898afd9e3668ec0ee7c
Some functional tests for object versioning

Also fix some exception-handling stuff in functional tests; at some
point, ResponseError() started needing two more parameters, but the
functional tests (not swift_test_client, just tests.py) still had a
couple spots that were not passing in the new params. Now they're
optional again, and if you omit them, then the stringification of the
ResponseError is just a little less useful than it could be.

Change-Id: I38968c4b590fc04b97b85c5f974c8648291a6689
Sanify handoff search depth with non-integer replica counts

On GET, the proxy will go search the primary nodes plus some number of
handoffs for the account/container/object before giving up and
returning a 404. That number is, by default, twice the ring's replica
count. This was fine if your ring had an integral number of replicas,
but could lead to some slightly-odd behavior if you have fractional
replicas.

For example, imagine that you have 3.49 replicas in your object ring;
perhaps you're migrating a cluster from 3 replicas to 4, and you're
being smart and doing it a bit at a time.

On an object GET that all the primary nodes 404ed, the proxy would
then compute 2 * 3.49 = 6.98, round it up to 7, and go look at 7
handoff nodes. This is sort of weird; the intent was to look at 6
handoffs for objects with 3 replicas, and 8 handoffs for objects with
4, but the effect is 7 for everybody.

You also get little latency cliffs as you scale up replica counts. If,
instead of 3.49, you had 3.51 replicas, then the proxy would look at 8
handoff nodes in every case [ceil(2 * 3.51) = 8], so there'd be a
small-but-noticeable jump in the time it takes to produce a 404.

The fix is to compute the number of handoffs based on the number of
primary nodes for the partition, not the ring's replica count. This
gets rid of the little latency cliffs and makes the behavior more like
what you get with integral replica counts.

If your ring has an integral number of replicas, there's no behavior
change here.

Change-Id: I50538941e571135299fd6b86ecd9dc780cf649f5
Documentation of the logs formats used in Swift

Change-Id: I402014627bab7113105a6a63c0b0ce1267735ca0
Fix container quota MW for handling a bad source path

The copy source must be container/object.
This patch avoids the server to return
an internal server error when user provides
a path without a container.

Fixes: bug #1255049
Change-Id: I1a85c98d9b3a78bad40b8ceba9088cf323042412
ismount should not raise exceptions

The changes from using os.path.ismount to using
swift.common.utils.ismount has caused problems since the new one
raises exceptions in cases where the old one did not. Daemons have
been encountering this and exiting; servers have been 500ing instead
of 507ing in this case, changing handoff behaviors, etc.

Since the new one was specifically written and tested for this new
behavior, I left that original function as ismount_raw and made
ismount do what it did before.

If there really isn't some reason for this new behavior, I'll be glad
to get rid of ismount_raw and just keep ismount. I couldn't see any
reason for the new behavior myself.

Change-Id: I2b5b17f9ed9656cd8804a5ed568170697d0b183d
Let swift-object-info skip etag verification

Now swift-object-info has a "-n" option that will cause the etag
verification to be skipped; on large objects, the etag verification
takes the vast majority of the runtime, and sometimes you just want to
know which account owns the 5 GB monstrosity without waiting around while
its checksum is verified.

Change-Id: Id284570633eb7b98046cdb948d7c37a152de1195
Remove dependencies on pep8, pyflakes and flake8

They should be determined by the hacking dependency
implicitely.

Change-Id: I8805f7b0ef9f2b1f59be342e9c6d032330707688
Use a tempfile.mkdtemp() based temporary directory

Change-Id: Ie0e8137615348b130b67323d0d2913dc5ebfd5fb
Fix up SAIO to use new gatekeeper

Fix up the SAIO default proxy-server pipeline to use the new
gatekeeper and update to follow the pipeline ordering from proxy
sample.

Change-Id: Iaa4de7abd7fc557cafe6fe40668973e9fc142ca3
Make swift-recon usable on hosts without IPv6

Fixes a bug when swift-recon --sockstat is used on hosts without
IPv6 support.

Tested by disabling IPv6 on Ubuntu 12.04 LTS:

Add "ipv6.disable=1" to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub
sudo update-grub
sudo /sbin/reboot

Closes-Bug: 1270711
Change-Id: Ib31a059e412ac68ca0a3faef4201fec7560d1178
Skip delete_at_update for replication requests

Requests through the object server that are from backend replication
should not send x-delete-at updates to the .expiring_objects account.
Replication is just moving data around or making new replicas, not
creating new data from nothing.

Change-Id: I324864face3ff559822c7a50c50e675e8b889b48
Container Sync: Simple HTTP Proxy load balancing

Change-Id: I021b043b927153bacff48cae648d4d8c5bbad765
Add debug logger on per-test proxy-server

To help debug problems with the proxy-server app setup for each class,
we add a debug logger so that on failure we can see what requests were
actually sent.

Additionally, we move the set_http_connect() call to the same relative
position as the one before it for clarity of intent.

Related bug 1271962

Change-Id: Idc301c06e114b11c358ee4fbc0b2b70ec7743091
Add secondary groups to user during privilege escalation

setgid provides the primary group, setgroups sets the secondary
groups. Prior to this patch, we would do a setgroups with an empty
list, effectively wiping secondary groups. We now verify which
secondary groups the user is member of and escalate the privileges
accordingly.

Change-Id: I33a10edd448b3ac5aa758a8d1d70e582cf421c7d
Closes-Bug: 1269473
Add documentation for GateKeeper

The purpose of GateKeeper mostly relates to the development of new swift code,
so I threw together a guide for development_middleware that covers some basics
with a eye towards metadata handling in-particular.

I also fixed up some missing autodoc's, split out middleware autodoc and added
some ref's here and about so I could link to them from the
development_middleware guide.

DocImpact
Change-Id: I20dd942ea8df9e33c3e794cb49669ffa1332c63e
Merge "Fix up SAIO to use new gatekeeper"
Move test.unit.locale out of gettext's way

When I run unittests on my laptop I get an ImportError from gettext who tries
to "from locale import normalize" and says "ImportError: cannot import name
normalize".  Whit this patch unittests works, not sure why this problem
doesn't show up in my other dev environments.

Also this seems like good form since 'locale' is already a stdlib module.

Change-Id: If0fcd66ce391665b4e4127c610de3246db409a68
Merge "Add debug logger on per-test proxy-server"
Merge "Use a tempfile.mkdtemp() based temporary directory"