twitter/scalding — pull request #212 by asiegel34 (2012-10-25 21:20:20+00:00)
Title: Proposal: prototype for typesafe fields extension

This is just a proposal. It's a prototype for the typesafe fields extension that just implements map, mapTo, flatMap, and flatMapTo. Currently I've set the script just to generate up to arity 3 to keep things simple and clean, but this is easy to increase to 6 or 8 (or whatever seems right).

Again, this is a proposal only, to get a feel for what this API extension would look like under the covers. Before merging it, I'd want to increase the arity to something more reasonable, implement other RichPipe methods such as filter, implement GroupBuilder methods in a separate trait, and add more tests.

Feedback is welcome. Here's a description of the feature proposal that I previously sent to scalding-dev:

The typesafe API is great, but I sometimes find myself preferring the fields API, particularly in jobs with lots of fields. When I'm using the fields API, I find I crave more type safety. Scalding 0.8.0 took a step in this direction by introducing the semi-typesafe Field class. The traditional way of specifying a map operation in the fields API uses scala symbols as field identifiers, for example:

pipe.map(('x,'y) -> ('z,'w)) { tuple: (Int, Int) => val (x,y) = tuple; (x+y, x-y) }

Three major disadvantages of this API are:

(1) there are no compile-time typographical checks on field identifier names (if i misspell a field name, it won't be detected until flow planning);
(2) there are no compile-time or flow-time type-checks of field data (if 'x and 'y are valid incoming fields, but aren't both Ints, this won't be detected until the job throws a cast exception); and
(3) because the fields are untyped, there's no way to specify custom Orderings, and therefore no way to use group operations on keys whose types don't implement Comparable (even if scala knows about an implicit Ordering for them).

It's also a little annoying (but not quite a "major disadvantage") that in the mapping function, the tuple input has to be explicitly type-declared and then unwrapped.

0.8.0 introduced the concept of a Field object:

val xField = StringField[Int]("x")
val yField = StringField[Int]("y")

etc., so the map operation can now be written

pipe.map((xField, yField) -> (zField, wField)) { tuple: (Int, Int) => val (x,y) = tuple; (x+y, x-y) }

This solves (1), of course; and also (3), because the xField object (say) carries with it an implicit ordering on Int. But the type checking doesn't carry over to the mapping function, so (2) is still an issue.

My proposal for 0.9.0 adds additional type-checking by introducing various new signatures for RichPipe.map, one for each combination of input and output arity. A typical example (the "(2,2)-ary" map function) is

def map[A,B,S,T](fields: ((Field[A],Field[B]),(Field[S],Field[T])))(fn: ((A,B)) => (S,T))
(implicit conv: TupleConverter[(A,B)], implicit setter: TupleSetter[(S,T)], implicit inArity: Arity2, implicit outArity: Arity2)

Now the example map operation can be written

pipe.map((xField, yField) -> (zField, wField)) { case (x,y) => (x+y,x-y) }

and it's fully typesafe: since the Field instances carry type information, the input and output parameters of the associated function implicitly receive their types. This solves all of the preceding issues.

The reason for the extra implicits (inArity and outArity) is that there are two other map implementations with similar signatures (the "(1,3)-ary" and "(3,1)-ary" map functions):

def map[A,S,T,U](fields: (Field[A],(Field[S],Field[T],Field[U])))(fn: (A) => (S,T,U))
(implicit conv: TupleConverter[A], implicit setter: TupleSetter[(S,T,U)], implicit inArity: Arity1, implicit outArity: Arity3)

def map[A,B,C,S](fields: ((Field[A],Field[B],Field[C]),Field[S]))(fn: ((A,B,C)) => S)
(implicit conv: TupleConverter[(A,B,C)], implicit setter: TupleSetter[S], implicit inArity: Arity3, implicit outArity: Arity1)

In all three cases ((2,2)-ary, (1,3)-ary, (3,1)-ary), the type erasure of the "fields" parameter is Tuple2, and the type erasure of the "fn" parameter is Function1. Likewise for "conv" and "setter". But the arity implicits have distinct type erasures, ensuring that each map implementation translates to a distinct Java signature.

This feature solves a lot of problems with the fields API. The downside (and this is the potential controversy) is that it requires adding a lot of new method signatures: n^2 of them, if we want to support up to (n,n)-ary mappings. I'm fairly sure that n=22 is not the right number here; something like n=8 might be appropriate. (I've certainly never mapped more than eight input or output fields.) But this still requires 64 new methods. Then, of course, we need another 64 for the flatMap implementations, and mapTo and flatMapTo...

The same principles can be applied to make a lot of the GroupBuilder logic typesafe. For example, a common pattern is

groupBuilder.plus[Map[Key,Value]]('myField)

but if 'myField is replaced by an object of type Field[Map[Key,Value]], then we get the benefit of some really snazzy type inference:

groupBuilder.plus(myField)

and so on.

I really like this approach, and I think it would make it a lot more comfortable to code up jobs in the fields API. I'd be curious to know what everyone thinks of the unavoidable proliferation of method signatures in RichPipe and GroupBuilder.