Fix broker.storage_policy_index after .set_storage_policy_index

Change-Id: I5abd232263ef4443d7f63ae9995a6a996eeca8c7
Merge "Fix broker.storage_policy_index after .set_storage_policy_index" into feature/ec
Store per-object policy index in container

To support the object reconciler (coming Real Soon Now™), we store
each object's storage policy index in the container's 'object'
table. This way, some other daemon (replicator or something) can
locate the misplaced ones and enqueue them for the reconciler.

We keep container_stat.misplaced_object_count so that the reconciling
daemon can short-circuit if there aren't any misplaced
objects. Checking a single integer is much faster than scanning a
multi-million-row 'object' table.

We deliberately do not add the object's policy to any container
listings since that may break poorly-written clients (and aren't they
all). It's just for internal bookkeeping.

Change-Id: Iac81fc002a3636bb3dc73c6a171b72d742b9d2d1
put SPI in output of list_objects_iter

I'm not 100% certain that this is a good idea; if it is, squash it
into the parent commit please. Or maybe leave it separate? Dunno.

Maybe it goes in later with the proxy 404 stuff, because that's about
the only time we'll want this, so... call it premature?

Change-Id: Id495d32f1cf572ab82f44892e56126a165a0bfa2
Add container reconciliation to container updater

Under normal circumstances, an object lives in storage policy N, where
N is the value from its container's container_stat table. However,
under some circumstances involving network failures and such, it can
happen that an object is actually stored in a different policy than
its container.

This commit introduces functionality in the container updater to move
misplaced objects to their proper places. I chose the container
updater because it's already doing a filesystem walk and inspecting
every container DB, so when nothing is misplaced, no extra work is
being done. I used a separate greenpool for the work so that account
stats would still be updated with the same latency as before.

******************** NOT DONE ****************************

stuff that's not done

 * still need statsd tests (marked w/XXX comments)

 * really really needs a probetest or two

 * benchmark ContainerBroker.put_object and make sure it's not too much slower when SPIs match

Change-Id: I92ea9ddc80d89b21c7d98bdfde27eb221b9629ac
Handle storage policies in container replicator

The container replicator will now sync up storage policy between two
different container DBs. The policy from the older container is kept,
or if the containers are the same age, a tiebreaker of lesser-policy
is used (this seems unlikely to occur, but we need some kind of
tiebreaker just in case).

The reason for keeping the older container's policy is that policy
mismatches are most likely to occur when a container is unavailable
for a short span and the application recreates it. When this happens,
the older container DB is likely to contain far more objects than the
newer one, so by picking the older, we minimize the number of
misplaced objects.

Change-Id: I0878fb32f30effd24eb791f6bed62a6045a2e8e5