twitter/scalding — pull request #1148 by johnynek (2015-01-01 01:50:46+00:00)
Title: Add a sketch of OrderedBufferable

This is a pull request against an experimental branch (fast-binay-sort) that will try to plumb through binary comparators from scalding all the way down to Hadoop.

This has been shown, perhaps not surprisingly, to have huge wins.

@ianoc How does this look for a starting point?

My thought is that we will check with reflection in all Groups if we get an Ordering that is a subclass of OrderedBufferable. If so, we will wire up the correct stuff so that cascading sees everything. Then we can make a method in TwitterJob like:
```scala
// Handle strings, longs, etc....
implicit def primitiveOrd[T:IsPrimitive]: Ordering[T] = Macros.orderingBufferablePrimitive[T]
implicit def caseClassOrd[T:IsCaseClass]: Ordering[T] = Macros.orderedBufferableCaseClass[T]
implicit def thriftStruftOrd[T:IsThriftStruct] = Macros.orderedBufferableScrooge[T]
```

Does this seem like a sane way to plumb things through? It is minimally invasive in that if you don't opt in, you don't get the performance win.

My thought is that if this looks like a huge win, we consider in scalding 0.14 or 0.15 making this required and you opt of it by creating a Kryo-based OrderedBufferable that will just internally deserialize.