spotify/luigi — pull request #9 by jcrobak (2013-01-06 18:02:32+00:00)
Title: new hadoop_jvm module for java/scala hadoop jobs.

We use scala mapreduce rather than hadoop streaming, so I'm trying to add support for that into luigi. This is a preliminary draft that I was hoping to get some feedback on.

If you don't feel like this belongs in luigi-proper, then we can keep the code in our own place, but it'd be nice to have some of the refactors I did to make it easier to share code. If you think this makes sense in luigi, maybe it'd make sense to have the streaming-specific code subclass the general purpose (poorly named) JvmHadoopJobTask. Thoughts?

commit message:
* new JvmHadoopJobTask and extracted some common functionality from
HadoopJobTask into BaseHadoopJobTask
* moved jobconf logic into the BaseHadoopJobTask
* bare-bones JvmHadoopJobRunner and extracted common functionality
from HadoopJobRunner into static methods (submitting/tracking a
job).