snowplow/snowplow — pull request #319 by rgabo (2013-08-15 17:07:08+00:00)
Title: Process archived logs in datestamped directories

A new `--process-archive` option of emr-etl-runner allows to process each datestamped directory under the archive bucket, specified by a required `--start` and an optional `--end` argument (if `--end` is omitted, the current day is used).

It does so by creating separate filecrush steps for each datestamped directory without having to move logfiles to the processing bucket. This saves hundreds of thousands of S3 requests and even more time.

If a datestamped directory between the start and end date is missing, the job flow will fail at that particular step so it's by no means foolproof.