alphagov/backdrop-ga-collector — pull request #40 by pwaller (2014-03-26 08:40:08+00:00)
Title: Add plugin architecture

Note: This needs rebasing after #39 is merged. Review should not take place until after that.
----------------------------------------------

This pull request adds a new plugin mechanism.

It currently depends on the `backdrop-collector-plugins` package, which
hasn't yet been reviewed.

The basic idea is that it should be possible to perform arbitrary
transformations to the documents before they are stamped with a `humanId`.

These arbitrary transformations may perform an aggregation on the input
and so need to see all input documents and specify the complete output
document set.

```json
{
"dataType": "content_dashboard_visitors_count",
"query": {
"id": "ga:123456",
"metrics": [
"visitors"
],
"dimensions": [
"customVarValue9"
]
},
"target": {
"url": "http://localhost:3039/data/foo/visitors-count",
"token": "foo-bearer-token"
}
"plugins": [
"ComputeDepartmentKey('customVarValue9')",
"RemoveKey('customVarValue9')",
"AggregateKey(aggregate_count('visitors'))"
]
}
```

The above query configuration shows the new "plugins" configuration
option. If "plugins" is specified, then `backdrop.collector.plugins` is
required to run.

In the example above, plugins are run in turn on all documents, in the order
they are provided. `ComputeDepartmentKey` computes a new `department`
parameter for each document based on the content of `customVarValue9`,
`RemoveKey` deletes the old `customVarValue9`. Finally, because there was
a many-to-one mapping of `customVarValue9` to `department`, the `visitors`
count needs to be summed for documents where all values except `visitors`
are equal. This is achieved through `AggregateKey`, which is capable
of aggregating multiple fields as well as different types of fields.
`aggregate_count` in this case specifies that the aggreagtion should happen
using the `sum` builtin. `aggregate_rate('rate_var', 'count_var')` allows
aggregation of percentages, weighted by counts.

Plugin strings are python expressions evaluated in the
[`backdrop.collector.plugins`](https://github.com/scraperwiki/backdrop-collector-plugins/blob/master/backdrop/collector/plugins/__init__.py) namespace.