twitter/scalding — pull request #1137 by johnynek (2014-12-17 06:20:20+00:00)
Title: Compositional sources (design sketch)

DO NOT MERGE

@isnotinvain this is close to what you and Chandler were talking about right?

You ultimately build a TapProvider (really all a source is). But these (and ReadPathProviders and SchemeProviders) compose well. With this you don't have the combinatorial explosion, and you can still get named sources.

I actually think open source is the wrong place to experiment here. I would probably suggest we do this internally until we get a good design or need to change core Source abstraction (which is a huge compatibility hit).

I thought I had a great idea about using scala's dependent types to avoid casts, but scala seems pretty dumb. I was hoping scala knows this:
```scala
trait Key {
type Inner
}

trait IntKey {
type Inner = Int
}
trait StringKey {
type Inner = String
}
// This should compile
def work(m: Key): m.Inner = m match {
case i: IntKey => 1
case s: StringKey => "1"
}
```
It does not compile. It says:
```
found : Int(1)
required: m.Inner
case i: IntKey => 1
^
test.scala:29: error: type mismatch;
found : String("1")
required: m.Inner
case s: StringKey => "1"
```
Even though you can prove that in the first branch, since IntKey matches, the `m.Inner == Int`. Similarly for StringKey. If we had a smarter compiler, we could clean up a lot (maybe all) of the casts in scalding using this trick on the Mode.