Expose Backend container info on deleted containers.

Include basic container info in backend headers on 404 responses from the
container server.  Default empty values are used as placeholders if the
database does not exist.

Specifically the X-Backend-Status-Changed-At, X-Backend-DELETE-Timestamp and
the X-Backend-Storage-Policy-Index value will be needed by the reconciler to
deal with reconciling out of order object writes in the face of recently
deleted containers.

 * Add "status_changed_at" key to the response from ContainerBroker.get_info.
 * Add "Status Timestamp" field to swift.cli.info.print_db_info_metadata.
 * Add "status_changed_at" key to the response from AccountBroker.get_info.

Change-Id: Ie6d388f067f5b096b0f96faef151120ba23c8748
Update FakeLogger with better log level handling

Parameterize logger on some daemons which were previously unparameterized and
try and use the interface in tests.

Change-Id: Id2e3740b1dd569050f4e083617e7dd6a4249027e
Make FakeRing use more real code

The existing FakeRing mock's implementation bit me on some pretty
subtle character encoding issue by-passing the hash_path code that is
normally part of get_part_nodes.  This change tries to exercise more
of the real ring code paths when it makes sense and provide a better
Fake for use in testing.

Change-Id: I87e10b78ed0963344988f73e946ecda7a96cc6a8
Keep status_changed_at up-to-date with status changes.

In particular during container recreation and replication.

When a container-server receives a PUT for a deleted database an extra UPDATE
is issued against the container_stat table to notate the x-timestamp of the
request.

During replication if merge_timestamps causes a container's status to change
(from DELETED to ACTIVE or vice-versa) the status_changed_at field is set to
the current time.

Accurate reporting of status_changed_at is useful for container replication
forensics and allows resolution of "set on create" attributes like the
upcoming storage_policy_index.

Change-Id: I938919eb0dde7f473a18c32d1cb26986c1c3c5b7
Extend direct_client

Rework header handling and add some methods needed by the reconciler.

 * response headers are case insensitive HeaderKeyDicts
 * add direct client container obj put and delete
 * add headers param to direct head object
 * add headers to DirectClientException

DirectClientException is a subclass of ClientException with a convience
constructor.  ClientException now supports an http_headers kwarg.

Exceptions raised from direct_client will include headers.

Change-Id: Ia484d569619df0bf85f973e4e916de2ac6401d5e
Add container-reconciler daemon

This daemon will take objects that are in the wrong storage policy and
move them to the right ones, or delete requests that went to the wrong
storage policy and apply them to the right ones. It operates on a
queue similar to the object-expirer's queue.

Discovering that the object is in the wrong policy will be done in
subsequent commits by the container replicator; this is the daemon
that handles them once they happen.

Like the object expirer, you only need to run one of these per cluster
see etc/container-reconciler.conf.

Change-Id: I5ea62eb77ddcbc7cfebf903429f2ee4c098771c9
Add reconciler probetest outline

You can manually setup a split brain scenario for reconciler testing with the
enqueue script using the machinery from the included probetest.  Evoke the
test as a script with with 'split-brain' command for more help.

Change-Id: I3a7b3167d674eba5f6e4072b176f6c4d29cdcd72
Add Storage Policies to container.backend

Change-Id: I15a96adfa876b6b50de707e2e9dff02191e92f3e
Add Storage Policy to container put/delete object

The object-server will send X-Storage-Policy-Index headers with all
requests to container severs, including X-Delete containers and all
object PUT/DELETE requests.  This header value is persisted in the
pickle file for the update and sent along with async requests from the
object-updater as well.

The container server will extract the X-Storage-Policy-Index header from
incoming requests and apply it to container broker calls as appropriate
defaulting to the legacy storage policy 0 to support seemless migration.

Change-Id: Iad42034a078f30fb053c44d1d8dfc927bd59c908
Merge container storage_policy_index values.

Keep status_changed_at in container databases current with status changes that
occur as a result of container creation, deletion, or re-creation.

Merge container put/delete/created timestamps when handling replicate
responses from remote servers in addition to during the handling of the
REPLICATE request.

When storage policies are configured on a cluster send status_changed_at,
object_count and storage_policy_index as part of container replication sync
args.

Use status_changed_at during replication to determine the oldest active
container and merge storage_policy_index.

Change-Id: Ib9a0dd42c271145e641437dc04d0ebea1e11fc47
Enqueue misplaced objects during container replication

After a container database is replicated, a _post_replicate_hook will enqueue
misplaced objects for the container-reconciler into the .misplaced_objects
containers.  Items to be reconciled are "batch loaded" into the reconciler
queue and the end of a container replication cycle by levering container
replication itself.

Change-Id: I3627efcdea75403586dffee46537a60add08bfda
Add LRUCache to common.utils

This decorator will memonize a function using a fixed size cache that evicts
the oldest entries.  It also supports a maxtime paramter to configure a
"time-to-live" for entries in the cache.

The reconciler code uses this to cache computations of the correct storage
policy index for a container for 30 seconds.

Change-Id: I0f220869e33c461a4100b21c6324ad725da864fa