Add more documentation to TypedPipe

Sadly, some of TypedPipe had no scaladoc.
Yes. That is a good fix. "have really heavy skew".

On Thu, Dec 4, 2014 at 3:28 PM, Alex Levenson <notifications@github.com>
wrote:

> In scalding-core/src/main/scala/com/twitter/scalding/typed/TypedPipe.scala:
>
> > @@ -515,7 +627,21 @@ trait TypedPipe[+T] extends Serializable {
> >        .hashLeftJoin(grouped)
> >        .map { case (t, (_, optV)) => (t, optV) }
> >
> > -  /** Build a sketch of this TypedPipe so that you can do a skew-join with another Grouped */
> > +  /**
> > +   * Enables joining when this TypedPipe has some keys with many many values and
> > +   * but many with very few values. For instance, a graph where some nodes have
> > +   * millions of neighbors, but most have only a few.
> > +   *
> > +   * We build a (count-min) sketch of each key's frequency, and we use that
> > +   * to shard the heavy keys across many reducers.
> > +   * This increases communication cost in order to reduce the maximum time needed
> > +   * to complete the join.
> > +   *
> > +   * {@code pipe.sketch(100).join(thatPipe) }
> > +   * will add an extra map/reduce job over a standard join to create the count-min-sketch.
> > +   * This will generally only be beneficial if you have really have skew, where without
>
> really heavy?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/twitter/scalding/pull/1111/files#r21350142>.
>



-- 
Oscar Boykin :: @posco :: http://twitter.com/posco