spotify/luigi — pull request #291 by a1k0n (2014-03-11 16:16:32+00:00)
Title: Move n_reduce_tasks hadoop config logic into JobTask

This fixes a (somewhat severe) bug where non-streaming jobs (i.e., jobs
in jars) would have their hadoop config set to 0 reduce tasks.

I apparently broke this when I added the map-only functionality. Jar tasks which always specify a number of reducers are unaffected, but e.g. when you leave the n_reducers default in Scalding, it ends up not running the reducers at all. Assuming this passes all tests, please merge ASAP.

And really, the n_reduce_tasks logic (or even the final_mapper etc for that matter) doesn't really belong in BaseHadoopJobTask at all, as it's specific to the streaming jar used by JobTask's runner.