feat: integrate shard storage with transport

This patch refines the interface and storage implementation defined in
the last patch and integrates it with the transport layer.

A few updates have been made:
- 'name' -> 'href' for listing shards
- limiting, markers, and detailed are all used
- use of common_utils.fields to clean up shards transport PATCH
- add missing init for schemas
- fix schema issues found: 'location' -> 'uri', __init__.py
- shard resource correctly implements PUT semantics (replaces)

Transport: the admin API concept has been expanded to include
functionality from the public interface *in addition* to admin
functionality. Part of the rationale behind this is to simplify unit
testing. The other part of this is that an admin should be able to do
everything a normal user can do in addition to their special
functions.

Storage: now divided into control and data plane. The bootstrap passes
a control driver down to the transport which *can* be used for
endpoints as needed.

A test suite has been added that exercises the functionality from the
transport side of the shard registry resource.

Finally, the way the FaultyStorage driver tests were handled was
changed. Something about the setattr magic in that suite's setup made
it such that *all* tests would use the Faulty storage driver. This is
possibly related to the use of lazy_property decorators. To address
this issue, this patch promotes the faulty storage driver to setup.cfg
visibility and removes the setattrs.

Change-Id: I5b8cdb3a11d29422762b52f1e15e33167eecb867
Partitally-implements: blueprint storage-sharding
Partially-Closes: 1241686
Closes-Bug: 1243898
Add Tests for non-existing resources

This patch adds tests for action on non-existing queues, messages & claims.

Change-Id: I68517b1b98f668eff86a0bcad19dc9ed8d01267c
Tests-Bug: #1243752
feat: connect sharding manager to control drivers

This patch implements the register/deregister/lookup operations for
the sharding manager.

As a result of this patch, the sharding driver now takes the control
driver as a paramater. This gives it the access needed to make
decisions about queue placement (and lookups) based on the entries in
the control plane.

In order to get the sharding manager working correctly, several
methods had to be overridden to properly handle the QueueNotMapped
error. When a QueueNotMapped error is encountered under normal
conditions, it is equivalent to the target resource not being
found. The overrides mimic the default behavior of those controllers'
methods in the case where the queue isn't found.

An inconsistency was found between the base storage driver and the
concrete implementations. message_controller.list did not provide an
include_claimed parameter. This has been corrected.

To avoid DuplicateOptErrors in the mongodb driver in a sharded
context, a filtering algorithm is applied. To make it clear to data
drivers that they need to apply this filtering algorithm, conf.dynamic
is set by the sharding driver.

Unit tests have been added for the weighted select algorithm.

Up next: sharded queue listing support, caching

Change-Id: Ic36cd657ba13fa4c0b624dc5c71bfe99f9a70139
Partitally-implements: blueprint storage-sharding
feat(shard): queue listing from multiple sources

Sharding need special storage driver wrapper for queue-related
operations. The listing queue algorithm merges the listing result
from multiple storage connections, and return them in sorted order.

This patch also fixes non-configurable default limits in sharding.

Change-Id: If9bd80459bd1a0dc92ea68c42bc28c0f6dde2a65
Partitally-implements: blueprint storage-sharding
chore: Pass cache into drivers

This patch plumbs storage drivers so they all get a cache instance
they can use at their discretion.

I decided to make cache a required param to the initializers, since
any deployment of Marconi of any significance would likely opt to
enable caching anyway, and the cache library has a default in-memory
backend that can be used for simple deployments.

Note also that both data- and control-plane drivers receive a cache
instance, even though the sharding controllers will not use it
(caching will be done in the sharding data driver instead.) I thought
it would be better to pass cache in both cases so we can share test
code and avoid complicating utils.load_storage_driver(). Also, the
control driver may eventually support operations other than sharding;
cache may come in handy then.

Change-Id: I647791af0d7a5914c30cb2489033ec650a455370
Signed-off-by: kgriffs <kurt.griffiths@rackspace.com>
feat(sharding): Cache shard mappings

This patch modifies sharding.Catalog so that it caches the mapping
between (project, queue) ===> shard. This reduces latency for
all data plane operations when sharding is enabled.

Change-Id: I54714c4e6c0ed78ac302ce86aed9bb72b200052b
Partially-Implements: blueprint storage-sharding
fix(mongo): Delete claimed message fails

This patch adds an additional check on the primary node for the correct
claim ID if it fails to match on the secondary. This takes care of the
case when the claimed version of the message document has not yet been
replicated to the secondary before the client attempts to delete it.

Change-Id: Ia22562532351a36bedc980c7c6041517207b2723
Closes-Bug: #1247297