Faster swift-dispersion-populate

- Makes swift-dispersion-populate a bit faster when using a larger
  dispersion_coverage with a larger part_power.
- Adds option to only run population for container OR objects
- Adds option to let you resume population at given point (useful if you
  need to resume population after a previous run error'd out or the
  like) by specifying which suffix to start at.

The original populate just randomly used uuid4().hex as a suffix on the
container/object names until all the partition's required where covered.
This isn't a big deal if you're only doing 1% coverage on a ring with a
small part power but takes ages if you're doing 100% on a larger ring.

Change-Id: I52f890a774412c1d6179f12db9081aedc58b6bc2
Make test_translations test our tree

In order to run the correct classes, Python test framework adjusts
sys.path. However, these changes are not propagated to subprocesses.
Therefore, the test actually tries to test installed Swift, not
the one in which it is running.

The usual suggestion is to run "python setup.py develop" before
testing, but it's annoying and error-prone. If you forget it,
you may test the code in /usr very easily, and never know.

Let's just pass the correct path to subprocess. Much safer.

Change-Id: Ic71314e8462cf6e0579d704ffe9fbbfac7e6ba24
Small efficiency improvement in account quotas

If you were performing an object PUT with X-Copy-From and you had
account_quotas in your pipeline but your account did not have a quota
set, then we will now avoid an unnecessary call to get_object_info().

Change-Id: I8592d4ad65e45ecf9e0557bc10ce2c2db19732e8
Proxy: use just one greenthread per client

Instead of using two greenthreads and an eventlet.queue.Queue of size
1 to stream response bodies from backends to clients, just do it with
a single greenthread that reads and writes in a loop. This should
lower the amount of CPU used by the proxy in its response streaming.

Client fairness used to be provided implicitly; since the queue only
held 1 item, the read-from-backend greenthread would block after each
chunk until the write-to-client greenthread got a chance to run. Now
fairness is provided by an explicit eventlet.sleep() call, just as it
is in the object server.

Change-Id: Iae27109f5a3d109ad21ec9a972e39f22150f6dbb
Adding more unit tests for audit_location_generator

Change-Id: I40410fbbb79cea8647074f703e4675364c69d930
Merge "Proxy: use just one greenthread per client"
Merge "Small efficiency improvement in account quotas"
Merge "Make test_translations test our tree"
Add WWW-Authenticate to 401 responses

Per http://www.ietf.org/rfc/rfc2616.txt, when a 401 error is returned, the
Www-Authenticate response header MUST also be returned. The format is
described in http://www.ietf.org/rfc/rfc2617.txt.

Swift supports and/or implements a number of authentication schemes
including tempauth, Keystone, tempurl, formpost and container sync. In
this fix, we use a catch-all, "Swift". The realm is the account (where
known) or "unknown" (bad path or where the 401 is returned from code
that does not have the request). Examples:

     Www-Authenticate: Swift realm="AUTH_1234567889"
     Www-Authenticate: Swift realm="unknown"

Fixes bug #1215491

Change-Id: I03362789318dfa156d3733ef9348795062a9cfc4
Merge "Faster swift-dispersion-populate"
Merge "Adding more unit tests for audit_location_generator"
Merge "Add WWW-Authenticate to 401 responses"
Add an __str__ method to brokers

A few uses of broker.db_file are in printouts where we do need
them, so the administrator may know what's up. Seems like an easy
way to get rid of those is to make brokers identify themselves
with common __str__. Alternative back-end implementations may
supply something other than a filename here, for example a cluster
name and a volume name.

Note that I'm not sure if correct coercion would occur when
brokers are bounced through dictionaries, hence explicit str().

Change-Id: I329788ebd1fbe39ffadcf9f9d5194a74a88dde58
utf8 encode tempurl key

In tempurl middleware, hmac uses the value of account metadata to
generate HMAC-SHA1 signature and hmac must accept a str-type string, not
a unicode string. The meta dict returned from get_info stroges special
chars as unicode however. So just encode it for tempurl using.

Closes-Bug: #1242644
Change-Id: I4be62eea014a573efc4748470de57dccf00e431d
Merge "Add an __str__ method to brokers"
Fix UnboundLocalError on account PUT

Fixes bug-1243973

Change-Id: I67143535c0f7a0c6b53f67329a0bb128a355a4de
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Simplify callers of diskfile.[read|write]_metadata()

As it happens, diskfile.read_metadata() and diskfile.write_metadata()
can take either an open file or a filename as their first arguments
(since xattr.[get|set]xattr() can), so we can clean up a couple places
where we were opening a file just to call read_metadata() or
write_metadata() on it. This results in 2 fewer system calls.

Example strace output:

/* read_metadata(filename) */
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", 0x0, 0) = 10
getxattr("/mnt/sdb1/1/node/sdb1/afile", "user.some.key", "some-value", 10) = 10

/* fp = open(filename); read_metadata(fp) */
open("/mnt/sdb1/1/node/sdb1/afile", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
fgetxattr(4, "user.some.key", 0x0, 0)   = 10
fgetxattr(4, "user.some.key", "some-value", 10) = 10

Change-Id: I321d8663b9e9e47b8f3ee6c21a1b65b408bb80e6
Update my mailmap

Using new email address.

Change-Id: I9d49992e64302ff5d879761fa63dd01cefb63cad
catch decompression errors

Change-Id: Ica380edc2364a5e18cefc26f70710e18ea329cfa
Merge "Update my mailmap"