Add sorted take optimizations

closes #880

Should be good enough, I think.
Got it. I'll update to this approach. I'll have to push it all the way down
to KeyedListLike I think, but that shouldn't be a problem.


On Wed, Jun 11, 2014 at 12:03 PM, Avi Bryant <notifications@github.com>
wrote:

> In scalding-core/src/main/scala/com/twitter/scalding/typed/Grouped.scala:
>
> > @@ -236,6 +239,28 @@ case class IdentityValueSortedReduce[K, V1](
> >    override def mapGroup[V3](fn: (K, Iterator[V1]) => Iterator[V3]) =
> >      ValueSortedReduce[K, V1, V3](keyOrdering, mapped, valueSort, fn, reducers)
> >
> > +  /** We are sorting then taking. Optimized for small take values
> > +   * If we take <= 100 (arbitrarily chosen), we use an in-memory-based method
> > +   * Otherwise, we send all the values to the reducers
> > +   */
> > +  override def take(n: Int) =
> > +    if(n <= 0) {
> > +      // This means don't take anything, which is legal, but strange
> > +      filterKeys(_ => false)
> > +    }
> > +    else if(n <= 100) {
>
> My point is that you could have a method, in Grouped, which is called
> bufferedTake or something, which just has the implementation from this else
> if clause:
>
> def bufferedTake(n: Int) = {
>    implicit val mon = ...
>    val pretake: ...}
>
> and that you change take to just have
>
> else if(n <= 100) {
>   bufferedTake(n)} else {
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/twitter/scalding/pull/894/files#r13667624>.
>



-- 
Oscar Boykin :: @posco :: http://twitter.com/posco