metamx/druid — pull request #416 by fjy (2014-03-04 22:52:52+00:00)
Title: Refactor and clean-up Druid data ingestion Part 1

Druid data ingestion schemas now consist of 3 things:
1. DataSchema - this JSON object defines the datasource, timestamp, dimensions, metrics and granularities of the ingested data. Each data schema requires a parser, which is used to parse the data. Each parser contains a parseSpec, and the current supported formats are csv, delimited, and json. Both the parser and parseSpec should be easily extendable for different storage formats (csv, tsv, json, etc.) and different delivery formats (string, protoBuf, bytes, etc). Granularity specs include the segment granularity and the query (rollup) granularity. Data schemas can be reused for both batch and real-time ingestion, as well as any new ingestion formats.
2. IOConfig - this JSON object defines where data is coming from and where it is going. For real-time ingestion, it includes a firehose and a plumber. For hadoop indexing, it includes the pathspec, the segment metadata update spec (formerly called the db updater spec), and the segment output path.
3. DriverConfig - this JSON object defines indexing parameters such as target Partition size or max rows in memory. Driver configs for both real-time and batch are optional, and reasonable defaults have been introduced for them.

This PR will introduce the following things:
- See: https://github.com/druid-io/druid-api/pull/9
- deprecated data specs
- introduced parse specs
- rejiggered input row parsers
- changed the firehose interface
- moved the parser from firehose to dataSchema
- moved a lot of configuration out of firehoses and plumbers into driver configs
- removed SegmentGranularity; segmentGranularities in druid now use the java-util Granularity class. Query granularities still use QueryGranularity.
- refactored plumbers

- docs are coming

Oh, and everything is backwards compatible.