twitter/scalding — pull request #973 by bholt (2014-07-25 01:39:56+00:00)
Title: Basic reducer estimator support

Add support for reducer estimators. Plugs into Cascading's FlowStepStrategy to run the estimator before the invocation of each job step.

Which reducer estimator is run is controlled by configuration parameters specifying the classname. Another flag controls whether the estimate should override cases where user has explicitly set reducers (`Grouped.withReducers()`).

Currently only one estimator has been implemented: `InputSizeReducerEstimator`, which will look at the amount of input data (at each step) and choose a number of reducers based on the specified "bytes.per.reducer" parameter.

```
hadoop myjob.jar \
-Dscalding.reducer.estimator=com.twitter.scalding.estimator.InputSizeReducerEstimator \
-Dscalding.reducer.estimator.override=true \
-Dscalding.reducer.estimator.bytes.per.reducer=1024 \
com.twitter.scalding.MyJob --hdfs
```

This PR differs from what is currently working its way into internal Twitter by integrating into the ExecutionContext directly, so reducer estimation can be available from old-style Jobs, scalding-as-a-library uses, and in the REPL.