twitter/scalding — pull request #762 by themodernlife (2014-01-28 21:31:05+00:00)
Title: Issues using skinny jars, -libjars flag and com.twitter.scalding.Tool as the Main class

Here's the use case:
- Skinny jar, *J.jar*, with a dependency on jar *D.jar* for some functionality in the mappers
- Scalding assembly jar, *S.jar*, with Scalding/Scala/Kryo/Algebird/Chill/etc... (aka fat "framework" jar)

In a perfect world, I would want the following to work, presuming S.jar is installed on all nodes in my Hadoop cluster, or installed into my EMR cluster via a bootstrap action à la https://github.com/Cascading/CascadingSDK#installing-on-aws-elastic-mapreduce

```scala
package example

class MyJob(args : Args) extends Job(args) {
lazy val dep = new SomeThirdPartyDependency("hi there!")

TextLine(args("input"))
.map(() → 'whatever) { _: Unit => dep.doSomething() }
.write(NullSource)
}
```

```bash
hadoop jar J.jar com.twitter.scalding.Tool -libjars D.jar example.MyJob --hdfs --input s3://bucket/file.txt
```

The first problem encountered is that com.twitter.scalding.Tool's classloader couldn't even *find* `example.MyJob`. This was fixed with #752.

Unfortunately Cascading also gets confused in the mappers/reducers because (as far as I can tell) when it tries to deserialize `example.MyJob` it will *not* have J.jar on its classloader's path, because the actual "main" class was com.twitter.scalding.Tool which is one level up in the classloader hierarchy (and classloaders only resolve by going up)!

This is the reason for why setting "cascading.app.appjar.class" fixes some users' issues. This can be done in your job by overriding `config`, this PR makes that the default.

So, there is still one gotcha which wasn't really obvious to me at first but again, because D.jar is added to a child classloader, it will not be visible on the *submitting* node when the class above is serialized, so the SomeThirdPartyDependency val (even if it's *lazy*) will cause an error. If it's hidden behind a `def`, you don't get an error.

The trick there is do

```bash
HADOOP_CLASSPATH=D.jar hadoop jar J.jar com.twitter.scalding.Tool -libjars D.jar example.MyJob --hdfs --input s3://bucket/file.txt
```

For what it's worth, my motivation here is:
- `sbt-assembly` and the "fat jars" approach introduces a lot of needless cycle time into the build
- Getting this right means it should be really easy to launch jobs via e.g. `sbt`
- Installing on EMR via some kind of bootstrap script offers a really nice user experience for developers looking to get started with Scalding