spotify/luigi — pull request #324 by bstempi (2014-05-09 18:07:10+00:00)
Title: Feature/s3 emr target

This pull request offers a quick way for testing if EMR output on S3 is present and finished. Given an S3 location, it will check:

1. That the location exists
2. That the _SUCCESS file exists within the location, signaling that the EMR output is complete.

This is accomplished via the `S3EmrTarget`. The `S3EmrTask` external task is also offered as a convenience.

Confusing Terminology Alert
-------------------------------------

The `_SUCCESS` file is a Hadoop construct. This isn't checking that an EMR "step" completed, but rather that a Hadoop job completed. I opted to call it the `S3EmrTarget` instead of the `S3HadoopTarget` to signal that its EMR-friendly. I'm open to a better naming convention,