Add Storage Policy support to Object Updates

The object server will now send its storage policy index to the
container server synchronously and asynchronously (via async_pending).

Each storage policy gets its own async_pending directory under
/srv/node/$disk/objects-$N, so there's no need to change the on-disk
pickle format; the policy index comes from the async_pending's
filename. This avoids any hassle on upgrade. (Recall that policy 0's
objects live in /srv/node/$disk/objects, not objects-0.)  Per-policy
tempdir as well.

Also clean up a couple little things in the object updater. Now it
won't abort processing when it encounters a file (not directory) named
"async_pending-\d+", and it won't process updates in a directory that
does not correspond to a storage policy.

That is, if you have policies 1, 2, and 3, but there's a directory on
your disk named "async_pending-5", the updater will now skip over that
entirely. It won't even bother doing directory listings at all. This
is a good idea, believe it or not, because there's nothing good that
the container server can do with an update from some unknown storage
policy. It can't update the listing, it can't move the object if it's
misplaced... all it can do is ignore the request, so it's better to
just not send it in the first place. Plus, if this is due to a
misconfiguration on one storage node, then the updates will get
processed once the configuration is fixed.

There's also a drive by fix to update some backend http mocks for container
update tests that we're not fully exercising their their request fakes.
Because the object server container update code is resilient to to all manor
of failure from backend requests the general intent of the tests was
unaffected but this change cleans up some confusing logging in the debug
logger output.

The object-server will send X-Storage-Policy-Index headers with all
requests to container severs, including X-Delete containers and all
object PUT/DELETE requests.  This header value is persisted in the
pickle file for the update and sent along with async requests from the
object-updater as well.

The container server will extract the X-Storage-Policy-Index header from
incoming requests and apply it to container broker calls as appropriate
defaulting to the legacy storage policy 0 to support seemless migration.

DocImpact
Implements: blueprint storage-policies
Change-Id: I07c730bebaee068f75024fa9c2fa9e11e295d9bd
Add Storage Policy Support to the Auditor

This patch makes the object auditor policy-aware, so it'll audit
objects in any storage policy.

DocImpact
Implements: blueprint storage-policies
Change-Id: I94e3a7937d9814b9ecef6ca35371e245a43513d3
Add storage policy support for the Replicator

This makes it so that objects stored in all policies get replicated
properly. This is only for rsync replication, not ssync.

DocImpact
Implements: blueprint storage-policies
Change-Id: Ifdb4624841f35953ba80189e669d3ef15d5563fd
Add Storage Policy Support to ssync

This patch makes ssync policy aware so that clusters using storage
policies and ssync replication will replicate objects in all policies.

DocImpact
Implements: blueprint storage-policies
Change-Id: I64879077676d764c6330e03734fc6665bb26f552
Put X-Backend-Timestamp in object 404 responses

This way the container reconciler can tell (sometimes) that an object
was deleted at a certain time.

DocImpact
Implements: blueprint storage-policies
Change-Id: Idaba3255f4109e5150d6c457f913c600fd8923eb
Extend interface on InternalClient

 * add get_object
 * allow extra headers passthrough on HEAD/metadata reqeusts
 * expose (account|container|get_object)_ring properties

Pipeline propety access to the auto_create_account_prefix also allows us to
bypass the early exit on a container HEAD for auto_create_accounts if the
container-updater hasn't cycled yet.

Allow overriding of storage policy index.

This is something the reconciler will need so that it can GET from one
policy, PUT in another, and then DELETE from the first one again.

DocImpact
Implements: blueprint storage-policies
Change-Id: I9b287d15f2426022d669d1186c9e22dd8ca13fb9
Extend direct_client

Rework header handling and add some methods needed by the reconciler.

 * response headers are case insensitive HeaderKeyDicts
 * add direct client container obj put and delete
 * add headers param to direct head object
 * add headers to DirectClientException

DirectClientException is a subclass of ClientException with a convience
constructor.  ClientException now supports an http_headers kwarg.

Exceptions raised from direct_client will include headers.

DocImpact
Implements: blueprint storage-policies
Change-Id: Ia484d569619df0bf85f973e4e916de2ac6401d5e
Add container-reconciler daemon

This daemon will take objects that are in the wrong storage policy and
move them to the right ones, or delete requests that went to the wrong
storage policy and apply them to the right ones. It operates on a
queue similar to the object-expirer's queue.

Discovering that the object is in the wrong policy will be done in
subsequent commits by the container replicator; this is the daemon
that handles them once they happen.

Like the object expirer, you only need to run one of these per cluster
see etc/container-reconciler.conf.

DocImpact
Implements: blueprint storage-policies
Change-Id: I5ea62eb77ddcbc7cfebf903429f2ee4c098771c9
Add reconciler probetest outline

You can manually setup a split brain scenario for reconciler testing with the
enqueue script using the machinery from the included probetest.  Evoke the
test as a script with with 'split-brain' command for more help.

DocImpact
Implements: blueprint storage-policies
Change-Id: I3a7b3167d674eba5f6e4072b176f6c4d29cdcd72
Merge container storage_policy_index

Keep status_changed_at in container databases current with status changes that
occur as a result of container creation, deletion, or re-creation.

Merge container put/delete/created timestamps when handling replicate
responses from remote servers in addition to during the handling of the
REPLICATE request.

When storage policies are configured on a cluster send status_changed_at,
object_count and storage_policy_index as part of container replication sync
args.

Use status_changed_at during replication to determine the oldest active
container and merge storage_policy_index.

DocImpact
Implements: blueprint storage-policies
Change-Id: Ib9a0dd42c271145e641437dc04d0ebea1e11fc47
Add Storage Policy Support to Accounts

This change updates the account HEAD handler to report out per
policy object and byte usage for the account.  Cumulative values
are still reported and policy names are used in the report
out (unless request is sent to an account server directly in
which case policy indexes are used for easier accounting).

Below is an example of the relevant HEAD response for a cluster
with 3 policies and just a few small objects:

   X-Account-Container-Count: 3
   X-Account-Object-Count: 3
   X-Account-Bytes-Used: 21
   X-Storage-Policy-Bronze-Object-Count: 1
   X-Storage-Policy-Bronze-Bytes-Used: 7
   X-Storage-Policy-Silver-Object-Count: 1
   X-Storage-Policy-Silver-Bytes-Used: 7
   X-Storage-Policy-Gold-Object-Count: 1
   X-Storage-Policy-Gold-Bytes-Used: 7

Set a DEFAULT storage_policy_index for existing container rows during
migration.

Copy existing object_count and bytes_used in policy_stat table during
migration.

DocImpact
Implements: blueprint storage-policies
Change-Id: I5ec251f9a8014dd89764340de927d09466c72221
Add functional tests for Storage Policy

 * additional container tests
 * refactor test cross policy copy
 * make functional tests cleanup better

DocImpact
Implements: blueprint storage-policies
Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002
Update swift-container-info to be storage policy aware

Lookup policy name in swift.conf; 'Legacy' container will use
policy-0's name; 'Unknown' is shown if policy not found in swift.conf

DocImpact
Implements: blueprint storage-policies
Change-Id: I450d40dc6e2d8f759187dff36d658e52737ae2a5
Enqueue misplaced objects during container replication

After a container database is replicated, a _post_replicate_hook will enqueue
misplaced objects for the container-reconciler into the .misplaced_objects
containers.  Items to be reconciled are "batch loaded" into the reconciler
queue and the end of a container replication cycle by levering container
replication itself.

DocImpact
Implements: blueprint storage-policies
Change-Id: I3627efcdea75403586dffee46537a60add08bfda
Add LRUCache to common.utils

This decorator will memonize a function using a fixed size cache that evicts
the oldest entries.  It also supports a maxtime paramter to configure a
"time-to-live" for entries in the cache.

The reconciler code uses this to cache computations of the correct storage
policy index for a container for 30 seconds.

DocImpact
Implements: blueprint storage-policies
Change-Id: I0f220869e33c461a4100b21c6324ad725da864fa
Fixes probe tests with non-zero default storage policy

Add headers param to direct_client.direct_get_object, which is used in
probetests to passthrough the X-Storage-Policy-Index header.

Raise SkipTest if default policy type is not replication

DocImpact
Implements: blueprint storage-policies
Change-Id: I19adbbcefbc086c8467bd904a275d55cde596412
Add Storage Policy Support to list_endpoints

This patch makes list_endpoints policy aware so that the object is
looked up in the right ring and its actual locations returned to the
client.

Also added policy support for custom endpoint names which was
missing before and tightened up unit test coverage for both
functional issues addressed here.

DocImpact
Implements: blueprint storage-policies
Change-Id: I56d4b0f4f321a4c72b11ec44d868a194d02ea3a3
Add Storage Policy Support to Container Sync

Have container sync get its object ring from POLICIES now,
update tests to use policy index from container_info and pass
that along for use in ring selection.

This change also introduced the option of specifiying in the cluster info
which of the relam/cluster's is the current realm/cluster.

DocImpact
Implements: blueprint storage-policies
Change-Id: If57d3b0ff8c395f21c81fda76458bc34fcb23257
Add Storage Policy Support to Recon Middleware

Recon middleware returns object ring file MD5 sums; this patch
updates it to include other object files that may be present
because of Storage Policies.  Also adds unit test coverage for
the MD5 reporting function which previously had none.

The recon script will now check all rings the server responds with
match the on-disk md5's regardless of server-type; including any
storage policy object rings.

Note the small change to the ring save method, needed to
stimulate the right code paths in 2.6 and 2.7 versions of
gzip to enable testing of ring MD5 sums.

DocImpact
Implements: blueprint storage-policies
Change-Id: I01efd2999d6d9c57ee8693ac3a6236ace17c5566
Add Storage Policy support to the Account Reaper

Extract X-Storage-Policy-Index header from container listing request
and use it when making direct object DELETE requests.

DocImpact
Implements: blueprint storage-policies
Change-Id: Icd4b2611b4169e46f216ff9a9839af732971a2bf