feat: connect sharding manager to control drivers

This patch implements the register/deregister/lookup operations for
the sharding manager.

As a result of this patch, the sharding driver now takes the control
driver as a paramater. This gives it the access needed to make
decisions about queue placement (and lookups) based on the entries in
the control plane.

In order to get the sharding manager working correctly, several
methods had to be overridden to properly handle the QueueNotMapped
error. When a QueueNotMapped error is encountered under normal
conditions, it is equivalent to the target resource not being
found. The overrides mimic the default behavior of those controllers'
methods in the case where the queue isn't found.

An inconsistency was found between the base storage driver and the
concrete implementations. message_controller.list did not provide an
include_claimed parameter. This has been corrected.

To avoid DuplicateOptErrors in the mongodb driver in a sharded
context, a filtering algorithm is applied. To make it clear to data
drivers that they need to apply this filtering algorithm, conf.dynamic
is set by the sharding driver.

Unit tests have been added for the weighted select algorithm.

Up next: sharded queue listing support, caching

Change-Id: Ic36cd657ba13fa4c0b624dc5c71bfe99f9a70139
Partitally-implements: blueprint storage-sharding
feat(shard): queue listing from multiple sources

Sharding need special storage driver wrapper for queue-related
operations. The listing queue algorithm merges the listing result
from multiple storage connections, and return them in sorted order.

This patch also fixes non-configurable default limits in sharding.

Change-Id: If9bd80459bd1a0dc92ea68c42bc28c0f6dde2a65
Partitally-implements: blueprint storage-sharding
chore: Pass cache into drivers

This patch plumbs storage drivers so they all get a cache instance
they can use at their discretion.

I decided to make cache a required param to the initializers, since
any deployment of Marconi of any significance would likely opt to
enable caching anyway, and the cache library has a default in-memory
backend that can be used for simple deployments.

Note also that both data- and control-plane drivers receive a cache
instance, even though the sharding controllers will not use it
(caching will be done in the sharding data driver instead.) I thought
it would be better to pass cache in both cases so we can share test
code and avoid complicating utils.load_storage_driver(). Also, the
control driver may eventually support operations other than sharding;
cache may come in handy then.

Change-Id: I647791af0d7a5914c30cb2489033ec650a455370
Signed-off-by: kgriffs <kurt.griffiths@rackspace.com>
feat(sharding): Cache shard mappings

This patch modifies sharding.Catalog so that it caches the mapping
between (project, queue) ===> shard. This reduces latency for
all data plane operations when sharding is enabled.

Change-Id: I54714c4e6c0ed78ac302ce86aed9bb72b200052b
Partially-Implements: blueprint storage-sharding
fix(mongo): Delete claimed message fails

This patch adds an additional check on the primary node for the correct
claim ID if it fails to match on the secondary. This takes care of the
case when the claimed version of the message document has not yet been
replicated to the secondary before the client attempts to delete it.

Change-Id: Ia22562532351a36bedc980c7c6041517207b2723
Closes-Bug: #1247297
feat(health): make health API storage-aware

Closes-Bug: 1243268
Change-Id: I75c3b46ccb1a5a66e6c12766d3e451fd3478822c
fix(caching): memcached backend disobeys contract

Updated the oslo.cache memcached backend so that set returns
True/False. To do so, it was a matter of checking that memcached
client's return value was nonzero, as per the docs.

Change-Id: I5622fbab1b08eca7dd1ee01e053cb1b9fc860a6e
fix(storage/mongodb): add claimed index hint

This gives a hint to the mongodb query optimizer to use the correct
index when dealing with POST claim requests. It results in 4x fewer
objects being scanned, 7x fewer index entries scanned, and loads only
one index rather than all of them.

Change-Id: Ia48a0953b843663a4d80219c05d4a98f3e3ee6b0
fix(storage/mongodb): use find + hint for delete

This patch replaces find_one in message delete with find().hint(). The
rationale behind this change is that I believe that find_one()
attempts to use all indices, so can incur some slowdown with
unnecessary scanning. find() + hint() avoids these additional scans.

Change-Id: I53193204a9a90958eadcfce8db14c4fc00a040de
WIP(mongo): blindly retry any AutoReconnect error.

Change-Id: Ic8357384779d355ba80472359f2667e01fd2d5b5