doc: Recommend installing marconi via pip

This approach was recommended by mordred on the dev list recently. It
will ensure that the correct deps are installed, distribute is the right
version, etc.

Change-Id: I326053a511db76178d8e63908c3f2287d66c14b9
Follow hacking rules about import

This patch follows the rules about import in HACKING.rst
https://github.com/openstack/marconi/blob/master/HACKING.rst#imports

* reorder imports
* Add missing blank lines before code

Change-Id: I4abcfb3a2640499c5df34cbf75b5eaecb3d19823
chore: Remove GC cruft from storage driver base class

Change-Id: I7314d817be2b183d59d4c2dfbf09123ead6bebda
Merge "Follow hacking rules about import"
Setup storage pipeline in the boostrap instead of driver base

This patch moves pipeline setup into the bootstrap and out of
the storage driver base class, so that the base class can be
inherited by meta-drivers, such as the planned sharding manager,
without introducing a loop in the bootstrapping logic.

Now, a meta-driver is exposed to the transport object that
takes care of wiring up the pipeline for each resource
controller behind the scenes.

As part of this work, the pipeline config was modified to
support configuring different stages depending on the
resource. We create three instances of Pipeline anyway,
so it seemed to make sense to allow the operator to
configure the pipelines independently.

Partially-Implements: blueprint storage-pipeline
Change-Id: Ibdb7d0e9537b1eec38a13f4881df7462039bbf98
'Persist' __getattr__ results

This patch adds a new decorator that helps creating 'cachable'
__getattr__ methods. The decorator sets the returned value as an
attribute of the instance.

The patch also decorates `Pipeline`'s __getattr__ method

Implements blueprint storage-pipeline

Change-Id: I6b052d536d5c6e6edaeec884b3c4a0e8703b1036
fix(queues): Global config used everywhere

This patch modifies the queues code to use manually instantiated configs
rather than the global config. This will allow for more flexibility in
configuring components. For example, the up-and-coming sharding feature
can construct configs based on the shard catalog rather than the local
INI file. Also, this change will make testing easier.

Also in this patch, the SQLite driver schema was consolidated into
one place in preparation for the sharding patch. With this change,
controllers can be instantiated and used independently, since all
the tables are set up when the driver is loaded, not in a piecemeal
fashion as before.

Change-Id: I1afa8ab8c7e6dd9a017f4a9e3a3b1fadbeb32806
Implements: blueprint remove-global-config
Closes-Bug: #1239725
Use stevedore instead of importutils

Current `Pipeline` implementation uses importutils for loading stages.
This patch replaces the usage of importutils with stevedore so that it
is possible to have external stages as well.

Implements blueprint storage-pipeline

Change-Id: I3d1c48ddde0721db38f467f7b723d34ea2033b86
Merge "Use stevedore instead of importutils"
Return a consumer function instead of consuming

Former implementation was called consume_for and used to consume the
whole pipeline. This implementation returns a callable function instead
which makes it possible for wrapped methods to have a `method` keyword
and to control / reuse the consumer.

Implements blueprint storage-pipeline

Change-Id: I8d7ad43028b5615b24f06ca1e116e35e5ab6a145
Validation for messages returned by queue/stats

This patch updates the  queue/stats tests to include validation for message age
(0 <= age <= message_ttl_max), href & created time.

Tests Bug: #1237620

Change-Id: I7c6dfd10b94c8ac64247d687ac31783522b8f3d0
feat: Storage sharding foundation

This patch provides the plumbing for implementing storage
sharding across multiple backends. Sharding is agnostic to
storage driver type and transport type. The new feature is
optional, and disabled by default.

The design eschews placing any kind of sharding reverse proxy
in the network, allowing the storage drivers to continue
communicating directly with their respective backends.

Sharding can be enabled by setting the global "sharding"
option to True. Future patches will add a sharding section to
the config that can be used to tweak the way sharding works when
it is enabled.

Storage drivers are managed by a Catalog class. The Catalog is
responsible for registering and deregistering queues in the
catalog backend, and for looking up an appropriate driver,
according to which shard a particular queue has been assigned.

In the future, this design will make it straightforward to map
individual queues to different storage backends, according to user
preference.

FWIW, I considered enabling sharding by inserting the routing driver
as the last stage in the storage pipeline. However, it felt like
a hack for the following reasons:

  * Doing so orphaned the regular, solitary driver that was
    still always loaded at the end of the pipeline.
  * Since the bootstrap was not aware of the sharding driver,
    it could not be used to provide setup, so the catalog
    object had to be turned into a singleton and options
    had to always be loaded from the global config.
  * The driver would have to be added to each controller
    pipeline, and would have to always be the last stage in
    the pipeline. Introducing a simple "sharded" boolean option
    seemed to be a more straightforward, less error-prone way
    for operators to enable sharding.

Partially-Implements: blueprint storage-sharding
Change-Id: I5190211e81fe4acd311b2cfdd0bae806cc3fec81
fix(bootstrap): Logging CLI options don't work

In the current implementation, Bootstrap creates the config option
inside its initializer. That means that common options such as
those registered by openstack.common.log, are not registered.

This patch modifies the Bootstrap initializer to take a config
instance from the caller, rather than trying to build one
itself. This does two things for us:

1. Avoids "magic" marconi configuration; the caller has full
   control and visibility into how marconi is configured.
2. Allows for passing in the global CONF object from the
   marconi-server command and the reference WSGI app module
   so that common options registered by oslo modules on the
   global CONF instance are picked up.

Hopefully openstack.common.log will be modified at some point
so that it isn't tightly coupled with cfg.CONF.

Implements: blueprint remove-global-config
Change-Id: Ibb6638f99ca2ce4a2a6025f6cd41939bb30fa85a
Merge "Validation for messages returned by queue/stats"
Merge "doc: Recommend installing marconi via pip"
Merge "feat: Storage sharding foundation"
Fix Pep8 indentation errors missed by Pep8 1.4.5

Pep8 1.4.6+ finds additional indentation errors. Fix them in
preparation of being able to switch to a newer Pep8 version.

Change-Id: I50493675c33524e577a0eebf3f9b74037ed9721c
Merge "fix(bootstrap): Logging CLI options don't work"
Move misplaced test under the right package

Change-Id: I3c0c78cb315362f4f2948be91e23685495824b73
fix: one less unnecessary modification

One of the mongo driver methods unnecessary pop()'ed a value
in the old dict while constructing a new dict.

Change-Id: Ief2190807cd0443f9eeb238fe3e86b756c3fbe41
Closes-Bug: 1222935