spotify/luigi — pull request #239 by danielfrg (2013-11-29 01:19:35+00:00)
Title: Basic EMR support using MRJob

I hacked this simple (and maybe a little bit dirty) integration between Luigi and MRJob. Looking for some feedback and see if we can iterate to make this better :)

It needs all MRJob code to be in a separated `.py` file since MRJob takes that `.py` and sends it to the cloud.
On the other hand that makes easy to integrate old MRJob code to luigi.

I am sure there is a solution to make it more friendly like the native Hadoop support luigi has (just define the mapper and reducer), but I am not sure how to do it.

It uses the new S3Target but it actually doesn't do anything special with it, but I think that makes easier to understand the pipeline.

Includes the most basic example from MRJob.