rackerlabs/blueflood — pull request #274 by fourk (2014-03-14 23:16:30+00:00)
Title: Kafka ingestion step 1 -- WIP

Changes on a per-module basis:
#`blueflood-kafka-http`.

This is a new module added in this PR. It is really small (2 files and like 50 lines of code), and was created to deal with adding code that has a dependency on both blueflood-kafka and blueflood-http. Maybe there's a better way of solving that problem involving classloaders or something similar.

##blueflood-core

Metric.DataType got moved into its own class from being an inner class.

JSONMetricsContainer now has support for scoping of tenant ID on both the container level and on a per individual JSONMetric basis.

BatchWriter now accepts an implementation of the newly added IMetricsWriter interface. It allows customization of the storage device used during ingestion. Implementation replicating existing functionality added with 'AstyanaxMetricsWriter'.

Several getters in the Metric class renamed for Jackson compatibility.

Metric objects are now comparable for equality with one another.

## blueflood-kafka

MetricDeserializer added that can turn JSON representing Metric objects into Metric objects. TODO: see if I can consolidate MetricDeserializer and JSONMetricsContainer-based serialization and deserialization.

MetricsBatchSerializer added for converting MetricCollection objects to/from JSON. TODO: same as above re: consolidation.

KafkaProducer added that can dump batches of raw metrics (via MetricsCollection) into kafka. TODO: fix the hardcoded configuration, see what code can be conslidated with the existing kafka producer used for rollups.

TODO: Mock Kafka for testing, right now some tests have been disabled because they depend on kafka running locally.

## blueflood-http

Configuration value AUTHORIZED_AGENT_TENANTS added. Any tenants present in that list are allowed to submit metrics on behalf of other tenants.

Bulk http ingestion that is not scoped to an individual tenant is now possible. Make a request to the normal ingestion endpoint, but include an X-MultiTenant header, and make sure that your tenant id is in AUTHORIZED_AGENT_TENANTS. Also, make sure that the metrics you submit contain a 'tenantId' key on their JSON representation.

### other TODOs

I'll be adding an ingestor that feeds off of kafka. I may do that as a separate PR since this one is already at about a thousand lines of code.

Tests. There are a bunch that need to be added.