Use threadpools in the object server for performance.

Without a (per-disk) threadpool, requests to a slow disk would affect
all clients by blocking the entire eventlet reactor on
read/write/etc. The slower the disk, the worse the performance. On an
object server, you frequently have at least one slow disk due to
auditing and replication activity sucking up all the available IO. By
kicking those blocking calls out to a separate OS thread, we let the
eventlet reactor make progress in other greenthreads, and by having a
per-disk pool, we ensure that one slow disk can't suck up all the
resources of an entire object server.

There were a few blocking calls that were done with eventlet.tpool,
but that's a fixed-size global threadpool, so I moved them to the
per-disk threadpools. If the object server is configured not to use
per-disk threadpools, (i.e. threads_per_disk = 0, which is the
default), those call sites will still ultimately end up using
eventlet.tpool.execute. You won't end up blocking a whole object
server while waiting for a huge fsync.

If you decide not to use threadpools, the only extra overhead should
be a few extra Python function calls here and there. This is
accomplished by setting threads_per_disk = 0 in the config.

blueprint concurrent-disk-io
Change-Id: I490f8753d926fdcee3a0c65c5aaf715bc2b7c290
Add parallelism to object expirer daemon.

Two types of parallelism are added:

- concurrency to speed up what a single process does
- a way to run multiple daemons to work on different parts of the work

DocImpact

Change-Id: I48997f68eb2fd8de19a5ee8b9fcdf76dde2ba0ab
get_info - removes duplicate code (Take 3)

Consolidate the different ways in which info of account/container
is gathered, cached, used, updated, etc.

This refactoring increases code reuse and is a basis for later
addition of account ACLs.
Changing the get_info users is left for future.
This staged approach ensures the behaviour is unchanged.

Change-Id: I67b58030d3f9e3bc86bcd7ece0f1dc693c4e08c3
Fixes: Bug #1162199
Fixed Bug 1187200

See Bug 1187200 for a full description of the problem.

Part 1:

X-Delete-At-Container added to X-Delete-At-* info

This fixes the bug by passing the expiring-objects-account's
container name onward to the backend object servers. This is in case
the object servers' expiring_objects_container_divisor happens to be
different than the proxy server's, we want to make sure the host,
partition, and device match up with the container name. Different
container names would be fine, but not with mismatched host,
partition, and device info.

Part 2:

The db_replicator now double checks the disk path's partition against
the partition the ring gives back. If they don't match, it logs the
problem but continues to replicate the database to where it should be
and, on success to all proper nodes, removes the local out of place
database.

Bug 1187200

Change-Id: Id0873a3f2198ce285fe0b0c777738eff38bc2438
Add non-slash delimiter tests

Also expand the logic following the select statement in
list_objects_iter and list_containers_iter to document the state
following the prefix and delimiter tests.

Also remove doc strings which obscure test names when run with
"./.unittests" script.

Performance Note:

The following simple python program illustrates the performance
advantage of not attempting to invoke startswith() method of a string
when we know the argument is an empty string.

    import time

    curs = []
    for i in range(10000):
        curs.append(['nameofthing%d'%(i), 1, 2, 3, 4])

    beg = time.time()
    for i in range(1000):
        x = [r for r in curs]
    end = time.time()
    print "without: %.2f" % (end - beg)

    beg = time.time()
    for i in range(1000):
        y = [r for r in curs if r[0].startswith('')]
    end = time.time()
    print "with: %.2f" % (end - beg)

    # Output:
    # $ python ~/test_startswith.py
    # without: 0.60
    # with: 2.73

Change-Id: I1ac72cd7932953b0efe7db8ff591829b985b8d73
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Local read affinity for GET/HEAD requests.

Now you can configure the proxy server to read from "local" primary
nodes first, where "local" is governed by the newly-introduced
"read_affinity" setting in the proxy config. This is desirable when
the network links between regions/zones are of varying capacities; in
such a case, it's a good idea to prefer fetching data from closer
backends.

The new setting looks like rN[zM]=P, where N is the region number, M
is the optional zone number, and P is the priority. Multiple values
can be specified by separating them with commas. The priority for
nodes that don't match anything is a very large number, so they'll
sort last.

This only affects the ordering of the primary nodes; it doesn't affect
handoffs at all. Further, while the primary nodes are reordered for
all requests, it only matters for GET/HEAD requests since handling the
other verbs ends up making concurrent requests to *all* the primary
nodes, so ordering is irrelevant.

Note that the default proxy config does not have this setting turned
on, so the default configuration's behavior is unaffected.

blueprint multi-region

Change-Id: Iea4cd367ed37fe5ee69b63234541d358d29963a4
Tempauth - remove change made to HTTP_X_AUTH_TOKEN

tempauth changes HTTP_X_AUTH_TOKEN although not required by function
preventing swift extendability.
(prevents other middleware to rely on HTTP_X_AUTH_TOKEN representing
the token sent by the client).

Change-Id: I7ed6f9b5d0a5a203326a3912ef27cf29d2d4e9ce
Fixes: Bug #1189410
Merge "Fixed Bug 1187200"
Merge "Local read affinity for GET/HEAD requests."
Merge "Add non-slash delimiter tests"
Merge "Tempauth - remove change made to HTTP_X_AUTH_TOKEN"
Merge "Use threadpools in the object server for performance."
Merge "Add parallelism to object expirer daemon."
Fix functional tests with keystone.

- Make sure storage_url is string and not unicode or the unicode
  no_safe_quote test will not get this nicely.
- Fixes bug 1190190.

Change-Id: I945f2b94f2063b836c10131fa4bd57938ed48f3c
Merge "get_info - removes duplicate code (Take 3)"
Merge "Fix functional tests with keystone."