Add simple capabilities API

Precursor-to: BP selectable-aggregates

The per-deployment variation in API capabilities is effectively
determined by the lack of strict feature parity across storage
drivers.

Currently we require that callers either know a priori which
features are provided and which not, or else to probe the API
and handle the NotImplementedErrors.

With the advent of more complexity in the API (complex queries,
selectable aggregates etc.) it would be useful to allow a more
direct interrogation of capabilities in the current deployment.

We expose the current set of capabilities in the form of a
simple nested dictionary, e.g.:

    {
        'meters': {'pagination': True,
                   'query': {'simple': True,
                             'complex': False}},
        'resources': {'pagination': False,
                      'query': {'simple': True,
                                'metadata': True,
                                'complex': False}},
        'samples': {'pagination': True,
                    'groupby': True,
                    'query': {'simple': True,
                              'complex': True}},
        'statistics': {'pagination': True,
                       'groupby': True,
                       'query': {'simple': True,
                                 'metadata': True,
                                 'complex': False},
                       'aggregation': {'standard': True,
                                       'selectable': {
                                           'max': True,
                                           'min': True,
                                           'sum': True,
                                           'avg': True,
                                           'count': True,
                                           'stddev': True,
                                           'cardinality': True,
                                           'quartile': False}}},
        'alarms': {'query': {'simple': True,
                             'complex': True},
                   'history': {'query': {'simple': True,
                                         'complex': True}}},
        'events': {'query': {'simple': True}},
    }

which is flattened in the usual way (using a similar approach as
used for resource metadata).

Leaf nodes of the capabilities representation are boolean, with
absence being implicitly False.

Change-Id: I8de9e153d7641d5d38ca9b79bab66887257c3010
Selectable aggregation functions for statistics

Addresses: BP selectable-aggregates

Previously, the statistics API always computed and returned a
standard set of aggregate functions.

Now, individual aggregation functions can be selected and even
parameterized via syntax of form:

   /v2/meters/<meter_name>/statistics?aggregate.func=<name>&aggregate.param=<value>

For example in order to calculate the average CPU util only:

  GET /v2/meters/cpu_util/statistics?aggregate.func=avg

  HTTP/1.0 200 OK
  [{"aggregate": {"avg": 0.6858829535841072},
    "avg": 0.6858829535841072,
    "duration_start": "2014-01-30T11:13:23",
    "duration_end": "2014-01-31T16:07:13",
    "duration": 104030.0,
    "period": 0,
    "period_start": "2014-01-30T11:13:23",
    "period_end": "2014-01-31T16:07:13",
    "groupby": null,
    "unit": "%"}]

In the current patch, selectable aggregates are provided by the
sqlalchemy driver only, with support in the mongodb driver to
follow in a subsequent patch.

Change-Id: I6cc095ba5ae16dea3f6b404e72a070ab9ac49c9a
Selectable aggregate support in mongodb

Addresses: BP selectable-aggregates

Add parallel support to mongodb for the selection of
standard aggregate functions.

Test coverage is provided by the scenario test added in
previous patch of this series.

Change-Id: I797b69958733a9401d3825547c95e41672bc4374
Decouple source and sink configuration for pipelines

Addresses: BP decoupled-source-sink-discoverable-resources

Add support for a new decoupled model of pipeline config
which does not conflate the elements specific to sources
and sinks.

Instead of the basic unit of config being a consolidated
pipeline, the sources and sinks may now be specified as
separate lists of dictionaries, before being linked together
once parsed.

This allows source-specific configuration, such as resource
discovery, to be kept focused only on the fine-grained source
while avoiding the necessity for wide duplication of sink-
related config.

The shape of the new config format is best illustrated with
an example:

  ---
  sources:
      - name: meter_source
        interval: 600
        meters:
            - "*"
        sinks:
            - meter_sink
      - name: host_cpu_source
        interval: 120
        meters:
             - "cpu.util.*min"
        resources:
             - "snmp://ip1"
             - "snmp://ip2"
             - "snmp://ip3"
        sinks:
            - meter_sink
            - lossy_sink
      - name: instance_cpu_source
        interval: 60
        meters:
            - "cpu"
        sinks:
            - cpu_sink
  sinks:
      - name: meter_sink
        transformers:
        publishers:
            - rpc://
      - name: lossy_sink
        transformers:
        publishers:
            - udp://addr
      - name: cpu_sink
        transformers:
            - name: "rate_of_change"
              parameters:
                  target:
                      name: "cpu_util"
                      unit: "%"
                      type: "gauge"
                      scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"
        publishers:
            - rpc://

The old pipeline.yaml format continues to be supported as
a deprecated feature to avoid breaking existing deployments.
The semantics of the common individual configuration elements
are identical in the deprecated and decoupled versions.

Change-Id: Ide86c0feba88ae736f2a913b5faa95e640c4ceaf
Merge "Add simple capabilities API"
Merge "Selectable aggregation functions for statistics"
Merge "Selectable aggregate support in mongodb"
Remove code duplication Part 3

Remove code duplication from impl_mongodb and impl_db2 as both of
these drivers use pymongo. This is a multiple step process as there
are several differences between the implementation of these drivers.

The third step is about to move the alarm history related functions
into pymongo_base. Alarm change functions were not implemented in DB2,
the expectation is that the MongoDB implementation will work on DB2
also.

Change-Id: Icff92ac31829669481b512d5504dcc7e2668eb07
Adds time constraints to alarms

This patch allows alarms to have time constraints specified.
If one or more time constraints are specified, the alarm is
evaluated only if the current time is within at least one
time constraint.

A field 'time_constraints' is added to the alarm model
that holds a list of the time constraints. Each time constraint
has the fields
- name		the name of the constraint
- description	description, default is auto-generated from start and duration
- start		starting point(s) of the constraint, in cron format
- duration	duration of the constraint, in seconds
- timezone	optional timezone information

Change-Id: I2d1bcd6728affc31834d7e2f3a0bdd570b2413bb
Blueprint: time-constrained-alarms
Wider selection of aggregates for mongodb

Partially-addresses: BP wider-aggregate-range

Support for two new aggregation functions has been added:

  * stddev: population standard deviation (unparameterized)
  * cardinality: count of distinct values for a sample attribute
    (parameterized by attribute name)

For example in order to calculate the standard deviation of CPU util:

  GET /v2/meters/cpu_util/statistics?aggregate.func=stddev

  HTTP/1.0 200 OK
  [{"aggregate": {"stddev": 0.6858829535841072},
    "duration_start": "2014-01-30T11:13:23",
    "duration_end": "2014-01-31T16:07:13",
    "duration": 104030.0,
    "period": 0,
    "period_start": "2014-01-30T11:13:23",
    "period_end": "2014-01-31T16:07:13",
    "groupby": null,
    "unit": "%"}]

Or to calculate the number of distinct instances that existed for
each tenant contrasted with the total number of instance samples
for that tenant in each 15 min period:

  GET /v2/meters/instance/statistics?aggregate.func=cardinality&aggregate.param=resource_id
                                    &aggregate.func=count
                                    &groupby=project_id&period=900

  HTTP/1.0 200 OK
  [{"count": 19,
    "aggregate": {"count": 19.0, "cardinality/resource_id": 3.0},
    "duration": 328.478029,
    "duration_start": "2014-01-31T10:00:41.823919",
    "duration_end": "2014-01-31T10:06:10.301948",
    "period": 900,
    "period_start": "2014-01-31T10:00:00",
    "period_end": "2014-01-31T10:15:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"},
   {"count": 22,
    "aggregate": {"count": 22.0, "cardinality/resource_id": 4.0},
    "duration": 808.00384,
    "duration_start": "2014-01-31T10:15:15",
    "duration_end": "2014-01-31T10:28:43.003840",
    "period": 900,
    "period_start": "2014-01-31T10:15:00",
    "period_end": "2014-01-31T10:30:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"},
   {"count": 2,
    "aggregate": {"count": 2.0, "cardinality/resource_id": 2.0},
    "duration": 0.0,
    "duration_start": "2014-01-31T10:35:15",
    "duration_end": "2014-01-31T10:35:15",
    "period": 900,
    "period_start": "2014-01-31T10:30:00",
    "period_end": "2014-01-31T10:45:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"}]

Test coverage is provided by the scenario test added in
next patch of this series.

Change-Id: Id3a37622f35d9a7d757c485b4d486b3f01cc6474
Wider selection of aggregates for sqlalchemy

Partially-addresses: BP wider-aggregate-range

Support for two new aggregation functions has been added:

  * stddev: population standard deviation (unparameterized)
  * cardinality: count of distinct values for a sample attribute
    (parameterized by attribute name)

For example in order to calculate the standard deviation of CPU util:

  GET /v2/meters/cpu_util/statistics?aggregate.func=stddev

  HTTP/1.0 200 OK
  [{"aggregate": {"stddev": 0.6858829535841072},
    "duration_start": "2014-01-30T11:13:23",
    "duration_end": "2014-01-31T16:07:13",
    "duration": 104030.0,
    "period": 0,
    "period_start": "2014-01-30T11:13:23",
    "period_end": "2014-01-31T16:07:13",
    "groupby": null,
    "unit": "%"}]

Or to calculate the number of distinct instances that existed for
each tenant contrasted with the total number of instance samples
for that tenant in each 15 min period:

  GET /v2/meters/instance/statistics?aggregate.func=cardinality&aggregate.param=resource_id
                                    &aggregate.func=count
                                    &groupby=project_id&period=900

  HTTP/1.0 200 OK
  [{"count": 19,
    "aggregate": {"count": 19.0, "cardinality/resource_id": 3.0},
    "duration": 328.478029,
    "duration_start": "2014-01-31T10:00:41.823919",
    "duration_end": "2014-01-31T10:06:10.301948",
    "period": 900,
    "period_start": "2014-01-31T10:00:00",
    "period_end": "2014-01-31T10:15:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"},
   {"count": 22,
    "aggregate": {"count": 22.0, "cardinality/resource_id": 4.0},
    "duration": 808.00384,
    "duration_start": "2014-01-31T10:15:15",
    "duration_end": "2014-01-31T10:28:43.003840",
    "period": 900,
    "period_start": "2014-01-31T10:15:00",
    "period_end": "2014-01-31T10:30:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"},
   {"count": 2,
    "aggregate": {"count": 2.0, "cardinality/resource_id": 2.0},
    "duration": 0.0,
    "duration_start": "2014-01-31T10:35:15",
    "duration_end": "2014-01-31T10:35:15",
    "period": 900,
    "period_start": "2014-01-31T10:30:00",
    "period_end": "2014-01-31T10:45:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"}]

Change-Id: I51344a8e5770760342087eb49c748eca69fb38c9
Merge "Adds time constraints to alarms"
Merge "Wider selection of aggregates for mongodb"
Merge "Wider selection of aggregates for sqlalchemy"
Merge "Decouple source and sink configuration for pipelines"
Merge "Remove code duplication Part 3"