chore: Expose commit hash in version module

This patch adds a marconi.version.verify_sha() convenience function,
so that continuous integration scripts can use it to verify marconi has
been installed correctly and/or to detect when the installed version is
out of date (i.e., that "import marconi" pulls in the latest version of
the library.)

Change-Id: I645c3a55a837d588fec55fa9b5c1ec579026b8f5
feat(mongodb): Partition data across multiple databases

This patch causes data to be partitioned across multiple databases in
order to reduce writer lock contention. The "queues" collection is
isolated in its own database, while the messages collection is partitioned
across several other databases. The number of partitions is configurable.

For example, if the number of partitions is set to 4, these databases
will be created in MongoDB:

    marconi_queues
    marconi_messages_p0
    marconi_messages_p1
    marconi_messages_p2
    marconi_messages_p3

Implements: blueprint mongodb-multidb
Change-Id: I399f4a39e5377a381aef489b046bc14155ccb75b
Fix queue's packages namespaces

Some namespaces weren't migrated during the marconi marconi/queues
migration. This patch migrates those.

Change-Id: I70e7140eb30f9b4e50836db076d0618cd1dbad73
fix(proxy): forward marconi headers

Proxy uses the `requests` library to dispatch requests to
multiple marconi servers.  When the result comes back, update
the proxy response headers with the marconi responses.

Change-Id: I151580f174f1a0fc1edca23b43bb7256bcf568fe
Merge "feat(mongodb): Partition data across multiple databases"
Merge "fix(proxy): forward marconi headers"
proxy: mirror structure of marconi queues + bootstrap

This patch adds smarter configuration to the proxy in two steps:

1. mirror the transport implementation used in marconi.queues in
   marconi.proxy
2. add a bootstrap file to take care of start up

Rationale: make configuration work, make deploying easy, make
alternate transport implementations feasible.

Another change: the unit tests are fixed by adding a few changes:
1. add drop functionality to the proxy storage interface
2. use drop/flush in test suite tearDown
3. rm tests.unit.test_config
4. delete queues at the end of the catalogue test (not yet robust)

The rationale for (3) was that test_config did not play nice with
other tests when they were registering their options, and failed as a
result. Furthermore, we should not need to test oslo.config.

Configuration changes: new fields in etc/marconi.conf
- drivers:proxy
- drivers:proxy:storage:{memory.mongodb}
- drivers:proxy:transport:wsgi
- oslo_cache

Also, fix: InternalServerError -> HTTPInternalServerError

Finally, redis was removed from requirements.txt.

Change-Id: If2365a1a738a3975fe6bde7bd07dfdee3460cecd
Implements: blueprint placement-service
feat: separate config for queues and proxy

This patchset separates the configuration of the proxy from that of
the queues server. This was done in order to simplify the
configuration file for each, and because it is not expected that the
proxy and the queues servers would be launched on the same
host. Furthermore, many of the proxy options are not relevant to the
queues server.

Furthermore, to allow this, common.config had to be modified to take a
prog parameter. This enabled the ability to save multiple
configuration files to one directory. See below for details.

The new files are:
- etc/marconi-proxy.conf
- etc/marconi-queues.conf

They are expected to be saved to one of:
- ~/.marconi
- /etc/marconi

Regarding namespaces, queues specific options are associated with the
'queues:*' group and proxy specific options are associated to the
'proxy:*' group.

The appropriate changes are also applied to the test suite and
helpers.

Change-Id: I7cf25e47ecff47934b50c21000b31308e1a4c8a9
Implements: blueprint placement-service
feat: split proxy API into admin and public apps

Rationale: when deploying the proxy, operators should be the only ones
that have access to the administrative functionality - primarily, the
management of partitions. This split makes it possible to deploy the
forwarding portion of the proxy independently of the administrative
portion.

The appropriate changes in tests, setup.cfg, and configuration files
(see: I7cf25e47ecff47934b50c21000b31308e1a4c8a9) were made.

Certain helpers that are reused for wsgi transport implementations
were extracted to a common location to aid reuse.

Unit tests were also simplified for the proxy, with the intent to make
them more thorough in coming patches.

New test requirement: httpretty
- used to perform request mocking during proxy transport unit tests

Change-Id: Ia26981a78c477a896370c48768e71f45c364c769
Implements: blueprint placement-service
feat (proxy/admin): allow partition modifications

This patchset adds one new method to the partitions API
- PATCH /v1/partitions/{partition}

The partitions storage layer is also updated to support a simpler
update interface:
- update(self, name, **kwargs) [with kwargs in ('hosts', 'weight')]

The partitions transport layer is cleared of cruft and reuses JSON
handling utilities from the queues transport layer.

This patch implementation uses jsonschema to achieve self-documenting,
strict format validation of the incoming input.

The partition storage controllers are modified accordingly, as are the
tests. More tests are added for the partitions transport layer to
verify the correctness and to ease future refactorings.

Change-Id: Ifa92b1225421196f95131c2b74e3c07b07c4cfd4
Implements: blueprint placement-service
Closes-Bug: 1230841
fix: force exception content to be text_type

Change-Id: Icc305f57741de88b72d40c677ef81dea918db818
fix: encode keys before caching

This patch addresses an error in the caching layer that prevented it
from playing nice with backends like memcached that expect keys to be
bytestrings rather than unicode strings. The change is made in
oslo.cache to make its use as transparent as possible to users.

Change-Id: Iee48e16393321ff7908807a9f0f7fa20480ec5f4
Closes-Bug: 1230556
Merge "proxy: mirror structure of marconi queues + bootstrap"
Merge "feat: separate config for queues and proxy"
Merge "feat: split proxy API into admin and public apps"
Merge "feat (proxy/admin): allow partition modifications"
Merge "fix: encode keys before caching"
fix(wsgi): Non-ASCII characters in the queue name trigger 500 status

The problem was that queue_name wasn't being validated before being
used to construct log messages. Also, I found that the regex used to
validate queue names was not reliable and needed to be rewritten.

Upon further investigation, I also discovered that the queue name
wasn't being validated for all routes that included it. This was
also fixed. Validating all input is a basic best practice re
application security.

Along the way, I edited the validation error messages for clarity,
and relocated the ValidationFailed exception since it is only used
by the transport driver.

Change-Id: Idf9f6c7b9a88865a6396dfcd72abbce96c774490
Closes-Bug: 1208873
Add Tests for Queue Stats

This patch adds tests for Queue Stats.

Change-Id: I7842eef98e6ed0a9809bea1a9b9d194c41b29d08
Merge "Add Tests for Queue Stats"