duedil-ltd/mrjob — pull request #11 by amedhat (2013-07-05 21:51:50+00:00)
Title: Steps extension

My small hack contribution....

ping @tomhennigan , @tarnfeld
Made a few fairly minor extensions to allow variable steps in an mr job that terminate based on conditions satisifed through the use of job counters....

Such extension can be useful for things like graph oriented jobs.. so I added a graph based problem for testing it in examples, which is breadth first search;

https://github.com/duedil-ltd/mrjob/blob/steps-extension/mrjob/examples/mr_breadth_first_search.py

Basically counters are used to check if at any step, all nodes encountered have been visited and expanded, and if so termination is triggered...to allow the output of the latest step going into the final directory, the output_dir variable is rewired to take its output from the latest step instead of the last one as shown below;

https://github.com/duedil-ltd/mrjob/blob/steps-extension/mrjob/hadoop.py#L334

Possible Extensions:

1- The termination condition function us implemented in runner.py and it basically tests for two very simple criteria, (a) that necessary counts have not occured, (B) or that counts triggering job completion have been detected.

https://github.com/duedil-ltd/mrjob/blob/steps-extension/mrjob/runner.py#L1002

Although such two criteria can cover a good percentage of cases, a better approach is to have a termination_condition function implemented in any job to define the criteria for terminating any step type.. that would require a way to pass that function to the hadoop runner to be able to call it from there...

2- Generalising termination capability by having sequences of terminable and non-terminable step groups, where a termination triggers skipping a terminable chunk of the same nature rather than the whole job