vinted/camus — pull request #14 by saulius (2014-10-07 12:07:49+00:00)
Title: Generate less small Avro files while offloading

The idea is to use default partitioning schema (yyyy/mm/dd/HH) for events offloaded from Kafka. Then reprocess this dump for each portal individually with sweepers. We need ETL partition planner for that which will transform files in ETLed directory to hive-compatible directories.

Also: sweepers will now only include records for specific portal (condition in map task). This is necessary because data files in ETLed directory contain data for all portals.

Expected result: a lot less small files and faster processing with fewer, bigger files. Fingers crossed.

@ljank