hRaven Reducer Estimator

Adds new "scalding-hraven" module with an estimator that looks up job history in hRaven to determine the actual amount of bytes reaching reducers. The `HRavenHistory` trait can be added to a `ReducerEstimator` to add functionality for looking up job history, and the `RatioBasedReducerEstimator` does this, scaling the estimate provided by `InputSizeReducerEstimator` based on the past ratio between mapper/reducer input sizes.

This PR also includes a couple other minor changes:
- Using a case class as argument to `estimateReducers` to hold the common reducer estimator info (`flow`,`step`,`predecessorSteps`)
- Add optional `fallbackEstimator` which lets estimators be chained

So far I don't have a story for unit testing hRaven. Any ideas are welcome.
None that I know of.


On Mon, Aug 4, 2014 at 10:49 AM, Brandon Holt <notifications@github.com>
wrote:

> In
> scalding-core/src/main/scala/com/twitter/scalding/reducer_estimation/InputSizeReducerEstimator.scala:
>
> >      val fs = f.getPath.getFileSystem(conf)
> >      fs.globStatus(f.getPath)
> >        .map{ s => fs.getContentSummary(s.getPath).getLength }
> >        .sum
> >    }
> >
> > +  protected def totalSize(taps: Iterator[Tap[_, _, _]], conf: JobConf): Option[Long] =
> > +    taps.foldLeft(Option(0L)) {
> > +      // recursive case
> > +      case (Some(total), multi: MultiSourceTap[Tap[_, _, _], _, _]) =>
>
> Yeah, this worried me — first analytics batch job I ran exposed this. Are
> there any other cases you can think of that don't implement Hfs that may
> come up?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/twitter/scalding/pull/996/files#r15780033>.
>



-- 
Oscar Boykin :: @posco :: http://twitter.com/posco