Move Storage Policy validation from parser to class

Currently we validate Storage Policy conflicts while parsing
swift.conf in parse_storage_policies. This let's us get away
with unnatural things in tests like patch_policies with two
policies that have the same name.

Additionally this change makes the rules a bit easier to
identify and assure coverage.

Change-Id: I6cd48e5a3b3658e106ecd20df0b04fd804c028a4
Implements: blueprint storage-policies
Fix issue with account broker and pre-SPI databases

Set a DEFAULT storage_policy_index for existing container rows during
migration.

Copy existing object_count and bytes_used in policy_stat table during
migration.

Change-Id: I6b3f591bad455dcd072056c28fc5a37c3f618e40
Implements: blueprint storage-policies
Fixes versioning function tests with non-zero default policy

In each PUT, proxy-server will do HEAD request for container sync
and checking the object versions. This patch makes proxy-server
knows the non-zero default policy when HEADing

Change-Id: I29c5715a9dbc11d9f2ca9ccdfa7655beadf354f8
Minor refactor of storage policy helper functions

Pulled from a larger pending review where it became clear that
we didn't need to keep adding get_data_dir(index) type routines
so this patch makes a generic function in the policy module that
constructs strings given a base and a policy index then others
can use it directly or as a partial (as is done with get_data_dir()
for example)

Change-Id: Ia3bed58f4b8d52f44b8ab3e8fad8014231c19c50
Implements: blueprint storage-policies
Quick fix to Storage Policy Support for List Endpoints

The path being sent to get_container_info didn't include the ver
so downstream path parsers were off by one on interpretting
parms.  get_info is mocked in unit tests so not caught there,
found while doing an example for the docs.  The version doesn't
matter in this case, just need a placeholder so the downstream
functions get the parms correct.

Also added policy support for custom endpoint names which was
missing before and tightened up unit test coverage for both
functional issues addressed here.

Change-Id: I56d4b0f4f321a4c72b11ec44d868a194d02ea3a3
Implements: blueprint storage-policies
Add Storage Policy Support to Container Sync

Have container sync get its object ring from POLICIES now,
update tests to use policy index from container_info and pass
that along for use in ring selection.

Change-Id: If57d3b0ff8c395f21c81fda76458bc34fcb23257
Implements: blueprint storage-policies
Add Storage Policy Support to Recon Middleware

Recon middleware returns object ring file MD5 sums; this patch
updates it to include other object files that may be present
because of Storage Policies.  Also adds unit test coverage for
the MD5 reporting function which previously had none.

The recon script will now check all rings the server responds with
match the on-disk md5's regardless of server-type; including any
storage policy object rings.

Note the small change to the ring save method, needed to
stimulate the right code paths in 2.6 and 2.7 versions of
gzip to enable testing of ring MD5 sums.

Change-Id: I01efd2999d6d9c57ee8693ac3a6236ace17c5566
Implements: blueprint storage-policies

Conflicts:
	swift/cli/recon.py
	test/unit/cli/test_recon.py
Adding functional tests for Storage Policy

Change-Id: If17bc7b9737558d3b9a54eeb6ff3e6b51463f002

Conflicts:
	test/functional/__init__.py
Fix X-Backend-Storage-Policy-Index requests

The reconciler needs to make PUT/GET requests to specific Storage Policies
despite the cached value of the container_info.

The existing implementation was pulling the request header after it had
already selected the ring and was placing objects on the wrong devices.

Change-Id: I2f92b9ddc1179e0223bcb33fc4cfead50db0ae78
Fix recursive Policy patching

When using test.unit.patch_policies on a TestCase that subclasses a patched
TestCase the mock policies would be unwound too early.  This change allows you
to subclass a TestCase decorated with patch_policies and get the parent
classes effective POLICIES.

Change-Id: I1b5baefe1ae4281e3be8a5f772994aff4065f981
Cleanup container update headers in obj server tests

The object server tests for container backed headers were doing a match
expectation against the backend HeaderKeyDict using a real dict so in tests
the keys were needlessly case sensitive.  This change updates the tests so
they perform correct assertions with case insensitive header keys and removes
some needless string manipulation in the obj server added to satisfy the
overly picky unittests.

There's also a drive by fix to update some backend http mocks for container
update tests that we're not fully exercising their their request fakes.
Because the object server container update code is resilient to to all manor
of failure from backend requests the general intent of the tests was
unaffected but this change cleans up some confusing logging in the debug
logger output.

Change-Id: Iea5ed128b62fd2dc17f55cd8076b711e6087dd26
cleanup policy parsing a bit more

Change-Id: Ia7053557411e085a58c7904b12a9d07dd2d1ccf6

Conflicts:
	swift/common/storage_policy.py
	test/unit/common/test_storage_policies.py
	test/unit/proxy/controllers/test_obj.py
Add write_fake_ring helper to test.unit

... and some tests to use it.

Change-Id: I570d10af5011edcf8279605f8cd0fd1b3fd16bfa

Conflicts:
	test/unit/cli/test_info.py
Fix cross policy versioning DELETE

In the object_versions section of proxy.controllers.obj's DELETE the
X-Storage-Policy-Index header was not being set on the correct request, which
ment you'd get a 404 when you tried to DELETE an object in a versioned
container.  The 404 was acctually coming from the request to cleanup the old
version after it had already been copied over the current version.

Change-Id: Ibbf57abf16bc38c02e5703113ce5a17fa2a7dae2

Conflicts:
	test/functional/tests.py
Extend interface on InternalClient

 * add get_object
 * allow extra headers passthrough on HEAD/metadata reqeusts
 * expose (account|container|get_object)_ring properties

Pipeline propety access to the auto_create_account_prefix also allows us to
bypass the early exit on a container HEAD for auto_create_accounts if the
container-updater hasn't cycled yet.

Change-Id: I9b287d15f2426022d669d1186c9e22dd8ca13fb9

Conflicts:
	test/unit/account/test_backend.py
Add probe tests for Account Reaper

Change-Id: I1ecb5c25b41a53206b70582dbbe5e9097b86999d
Add probetest for container sync

This change also introduced the option of specifiying in the cluster info
which of the relam/cluster's is the current realm/cluster.

Change-Id: Iec7c936398c5536091e70786c5b36e1fa3abbec0

Conflicts:
	etc/proxy-server.conf-sample
Add Storage Policy support to the Account Reaper

Extract X-Storage-Policy-Index header from container listing request
and use it when making direct object DELETE requests.

Change-Id: Icd4b2611b4169e46f216ff9a9839af732971a2bf
Implements: blueprint storage-policies
Fix cross policy versioning DELETE

In the object_versions section of proxy.controllers.obj's DELETE the
X-Storage-Policy-Index header was not being set on the correct request, which
ment you'd get a 404 when you tried to DELETE an object in a versioned
container.  The 404 was acctually coming from the request to cleanup the old
version after it had already been copied over the current version.

Change-Id: Ibbf57abf16bc38c02e5703113ce5a17fa2a7dae2

Conflicts:
	test/functional/tests.py
Use X-Backend-Storage-Policy-Index

The reconciler adds a number of X-Backend headers to container
responses and also override InternalClient requests which over-ride
container_info when setting X-Backend-Storage-Policy-Index.  Since
only backend requests will ever refer to policy by index, this change
unifies this backend request header with the other headers already
prefixed by Backend.  It also superficially simplifies the proxy's
BaseController translation of policy_index to name since X-Backend
headers are already stripped from outgoing requests.

Change-Id: I7597dac2bde7fc438c96b20e1574508aba5cfa05