etsy/scalding — pull request #4 by rjhall (2013-01-04 22:35:55+00:00)
Title: Tracing Inputs through the flow

This contains classes to "trace" input rows through the flow. At the end, input rows that were "used" in the construction of output rows are written to files. This allows for further invocations of the job to be run against potentially smaller input files, and still give the same results.

The original intent was to create subsampled data sets which can be used for testing ideas, but where naive subsampling of all the inputs fails (e.g., when the computation involves a join). To do this one can insert the "optionalSubsample" call into their flow at the point where they want subsampling to happen. On subsequent invocations of the job, when running on the output subsets, this operation is ignored.