twitter/scalding — pull request #75 by PlasticLizard (2012-05-05 15:17:04+00:00)
Title: CascadeJob

This is a very simple kind of Job for aggregating other Scalding jobs into a Cascade. The main motiviation for this was the desire to run parallel scalding jobs in a single Amazon EMR Job Flow. Based on discussions on the list, this was the recommended approach.

Testing Note: I could not figure out how to test the "run" method because the TestJob class doesn't seem to call "run"- so I am happy to add a test for this, but I will need some help. However, I have tested this class in practice both in Local Mode and on Amazon EMR and it works fine with real jobs.

Finally, there are two ways to use the new class, inheriting from it or via args

class MyAggregateJob(args : Args) extends CascadeJob(args) { addJobs(new JobA(args), new JobB(args) }

or

scald.rb com.twitter.scalding.CascadeJob --hdfs --jobs JobA JobB

Thanks,

Nathan