Decouple source and sink configuration for pipelines

Addresses: BP decoupled-source-sink-discoverable-resources

Add support for a new decoupled model of pipeline config
which does not conflate the elements specific to sources
and sinks.

Instead of the basic unit of config being a consolidated
pipeline, the sources and sinks may now be specified as
separate lists of dictionaries, before being linked together
once parsed.

This allows source-specific configuration, such as resource
discovery, to be kept focused only on the fine-grained source
while avoiding the necessity for wide duplication of sink-
related config.

The shape of the new config format is best illustrated with
an example:

  ---
  sources:
      - name: meter_source
        interval: 600
        meters:
            - "*"
        sinks:
            - meter_sink
      - name: host_cpu_source
        interval: 120
        meters:
             - "cpu.util.*min"
        resources:
             - "snmp://ip1"
             - "snmp://ip2"
             - "snmp://ip3"
        sinks:
            - meter_sink
            - lossy_sink
      - name: instance_cpu_source
        interval: 60
        meters:
            - "cpu"
        sinks:
            - cpu_sink
  sinks:
      - name: meter_sink
        transformers:
        publishers:
            - rpc://
      - name: lossy_sink
        transformers:
        publishers:
            - udp://addr
      - name: cpu_sink
        transformers:
            - name: "rate_of_change"
              parameters:
                  target:
                      name: "cpu_util"
                      unit: "%"
                      type: "gauge"
                      scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"
        publishers:
            - rpc://

The old pipeline.yaml format continues to be supported as
a deprecated feature to avoid breaking existing deployments.
The semantics of the common individual configuration elements
are identical in the deprecated and decoupled versions.

Change-Id: Ide86c0feba88ae736f2a913b5faa95e640c4ceaf
Wider selection of aggregates for mongodb

Partially-addresses: BP wider-aggregate-range

Support for two new aggregation functions has been added:

  * stddev: population standard deviation (unparameterized)
  * cardinality: count of distinct values for a sample attribute
    (parameterized by attribute name)

For example in order to calculate the standard deviation of CPU util:

  GET /v2/meters/cpu_util/statistics?aggregate.func=stddev

  HTTP/1.0 200 OK
  [{"aggregate": {"stddev": 0.6858829535841072},
    "duration_start": "2014-01-30T11:13:23",
    "duration_end": "2014-01-31T16:07:13",
    "duration": 104030.0,
    "period": 0,
    "period_start": "2014-01-30T11:13:23",
    "period_end": "2014-01-31T16:07:13",
    "groupby": null,
    "unit": "%"}]

Or to calculate the number of distinct instances that existed for
each tenant contrasted with the total number of instance samples
for that tenant in each 15 min period:

  GET /v2/meters/instance/statistics?aggregate.func=cardinality&aggregate.param=resource_id
                                    &aggregate.func=count
                                    &groupby=project_id&period=900

  HTTP/1.0 200 OK
  [{"count": 19,
    "aggregate": {"count": 19.0, "cardinality/resource_id": 3.0},
    "duration": 328.478029,
    "duration_start": "2014-01-31T10:00:41.823919",
    "duration_end": "2014-01-31T10:06:10.301948",
    "period": 900,
    "period_start": "2014-01-31T10:00:00",
    "period_end": "2014-01-31T10:15:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"},
   {"count": 22,
    "aggregate": {"count": 22.0, "cardinality/resource_id": 4.0},
    "duration": 808.00384,
    "duration_start": "2014-01-31T10:15:15",
    "duration_end": "2014-01-31T10:28:43.003840",
    "period": 900,
    "period_start": "2014-01-31T10:15:00",
    "period_end": "2014-01-31T10:30:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"},
   {"count": 2,
    "aggregate": {"count": 2.0, "cardinality/resource_id": 2.0},
    "duration": 0.0,
    "duration_start": "2014-01-31T10:35:15",
    "duration_end": "2014-01-31T10:35:15",
    "period": 900,
    "period_start": "2014-01-31T10:30:00",
    "period_end": "2014-01-31T10:45:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"}]

Test coverage is provided by the scenario test added in
next patch of this series.

Change-Id: Id3a37622f35d9a7d757c485b4d486b3f01cc6474
Wider selection of aggregates for sqlalchemy

Partially-addresses: BP wider-aggregate-range

Support for two new aggregation functions has been added:

  * stddev: population standard deviation (unparameterized)
  * cardinality: count of distinct values for a sample attribute
    (parameterized by attribute name)

For example in order to calculate the standard deviation of CPU util:

  GET /v2/meters/cpu_util/statistics?aggregate.func=stddev

  HTTP/1.0 200 OK
  [{"aggregate": {"stddev": 0.6858829535841072},
    "duration_start": "2014-01-30T11:13:23",
    "duration_end": "2014-01-31T16:07:13",
    "duration": 104030.0,
    "period": 0,
    "period_start": "2014-01-30T11:13:23",
    "period_end": "2014-01-31T16:07:13",
    "groupby": null,
    "unit": "%"}]

Or to calculate the number of distinct instances that existed for
each tenant contrasted with the total number of instance samples
for that tenant in each 15 min period:

  GET /v2/meters/instance/statistics?aggregate.func=cardinality&aggregate.param=resource_id
                                    &aggregate.func=count
                                    &groupby=project_id&period=900

  HTTP/1.0 200 OK
  [{"count": 19,
    "aggregate": {"count": 19.0, "cardinality/resource_id": 3.0},
    "duration": 328.478029,
    "duration_start": "2014-01-31T10:00:41.823919",
    "duration_end": "2014-01-31T10:06:10.301948",
    "period": 900,
    "period_start": "2014-01-31T10:00:00",
    "period_end": "2014-01-31T10:15:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"},
   {"count": 22,
    "aggregate": {"count": 22.0, "cardinality/resource_id": 4.0},
    "duration": 808.00384,
    "duration_start": "2014-01-31T10:15:15",
    "duration_end": "2014-01-31T10:28:43.003840",
    "period": 900,
    "period_start": "2014-01-31T10:15:00",
    "period_end": "2014-01-31T10:30:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"},
   {"count": 2,
    "aggregate": {"count": 2.0, "cardinality/resource_id": 2.0},
    "duration": 0.0,
    "duration_start": "2014-01-31T10:35:15",
    "duration_end": "2014-01-31T10:35:15",
    "period": 900,
    "period_start": "2014-01-31T10:30:00",
    "period_end": "2014-01-31T10:45:00",
    "groupby": {"project_id": "061a5c91811e4044b7dc86c6136c4f99"},
    "unit": "instance"}]

Change-Id: I51344a8e5770760342087eb49c748eca69fb38c9
Merge "Wider selection of aggregates for mongodb"
Merge "Wider selection of aggregates for sqlalchemy"
Merge "Decouple source and sink configuration for pipelines"