I prefer to push this happen since it will make the usage clear and easy at
least for now.

Best,
Jenna Zhang
Software Engineer | Twitter, Inc
Follow me: @jenna_zz


On Mon, Nov 4, 2013 at 1:34 PM, P. Oscar Boykin <notifications@github.com>wrote:

> In scalding-core/src/main/scala/com/twitter/scalding/typed/TypedPipe.scala:
>
> > @@ -114,10 +114,10 @@ trait TypedPipe[+T] extends Serializable {
> >     * Returns the set of distinct elements in the TypedPipe
> >     */
> >    @annotation.implicitNotFound(msg = "For distinct method to work, the type in TypedPipe must have an Ordering.")
> > -  def distinct(implicit ord: Ordering[_ >: T]): TypedPipe[T] = {
> > +  def distinct(implicit ord: Ordering[_ >: T], reducers: Int = -1): TypedPipe[T] = {
>
> Also, I wonder if we should punt on this until we do a HugeSet[T] api:
>
> trait HugeSet[+T] {
>   def union[U>:T](that: HugeSet[U]): HugeSet[U]
>   def intersection[U>:T](that: HugeSet[U]): HugeSet[U]
>   def filter(fn: T => Boolean): HugeSet[T]
>   def toTypedPipe: TypedPipe[T]
>   def toTypedPipeWithReducers(red: Int): TypedPipe[T]}
>
> Then distinct can return a HugeSet, and then an implicit conversion can go
> back to TypedPipe. Thoughts on taking this further?
>
> In the mean time, you can add this function you want to your own code with
> the RichTypedPipe pattern.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/twitter/scalding/pull/679/files#r7414168>
> .
>