twitter/scalding — pull request #450 by slyphon (2013-06-05 00:44:55+00:00)
Title: Create DistributedCacheFile, a pleasant API for managing files to be distributed via hadoop's DistributedCache

Dealing with hadoop's filecache.DistributedCache is kind of awkward. This gives users an easy way of setting up paths to be added to the distributed cache, and hides the details of symlink naming and registration.

Modifying the example in the wiki:

```scala
import com.maxmind.geoip.LookupService
import com.twitter.scalding.filecache.{DistributedCacheFile, DistributedCacheContext}
import com.twitter.scalding._

class SomeDerpyJob(args: Args) extends Job(args) with DistributedCacheContext {

val geoData = DistributedCacheFile("/geo-data/GeoLiteCity.dat").add(jobConfOption.get)

@transient private lazy val lookupService =
new LookupService(geoData.path, LookupService.GEOIP_MEMORY_CACHE)

def jobConfOption(implicit mode : Mode) = {
mode match {
case Hdfs(_, conf) => Option(conf)
case _ => None
}
}
}
```

I think this makes it more straightforward for users, giving them the benefits:
* Symlnk names are generated based on the filename of the resource plus the hash of fully qualified URI of the resource, which (if the resource at the location doesn't change), means that there won't be accidental collisions in the mappers/reducers.
* The details of keeping track of the different names, and ensuring that the symlink anchor is added correctly is handled
* The configuration vs. mapper phase is handled for the user (no runtime object-cannot-be-serialized errors)

I added mockito as a dependency, as I mock out the DistributedCache trait during tests (which wraps the hadoop DistributedCache static methods). I hope that's acceptable.

I look forward to your bikeshedding, I'm interested to see what color it winds up being :)

Jonathan Simms