pwendell/spark — pull request #4 by heathermiller (2014-11-05 22:39:54+00:00)
Title: SPARK-3257 Enable :cp to add JARs in spark-shell (Scala 2.11)

**Note:** Targeting this at @pwendell's branch just to have a version of Spark that builds on Scala 2.11 to work off of. I'll retarget this as soon as 2.11 support is in master.

First off, this PR cannot be merged as-is – I had to tweak the build to use a local build of Scala (with the scalac fixes for REPL jar loading, still in the Scala PR queue). So anything that I've committed that's build-related is committed so someone else can see how to build/reproduce. (Build instructions at bottom)

This PR builds upon the companion PR accepted for upstream Scala 2.11.x. It is intended to depend on Scala 2.11.x where x >= 5.
https://github.com/scala/scala/pull/4051

Given that most logic for this fix will be in the mainline Scala compiler, I've added tests in Scala to ensure jar-loading doesn't break in future versions of Scala. Those are here:
https://github.com/heathermiller/scala/commit/d045ddeba0bec4deef8cd5825d4b8c022c726ca8
(we programmatically generate jars, load them, and call methods on stuff we've dynamically loaded in)

## What it looks like:

14/11/05 13:51:14 INFO Main: Created spark context..
Spark context available as sc.

scala> :cp /Users/hmiller/Downloads/test.jar
14/11/05 13:53:03 INFO SparkContext: Added JAR /Users/hmiller/Downloads/test.jar at http://192.168.1.83:61722/jars/test.jar with timestamp 1415224383736
Added '/Users/hmiller/Downloads/test.jar' to classpath.

scala> new testblah.Test
res3: testblah.Test = testblah.Test@5d2b7e11

scala> :cp /Users/hmiller/Downloads/test2.jar
The path '/Users/hmiller/Downloads/test2.jar' cannot be loaded, because existing classpath entries conflict.

scala>

## How to build and test this yourself

First, you need to check out the `repl-cp-fix2` branch of my Scala fork at (until it's merged into upstream Scala 2.11.x):
https://github.com/heathermiller/scala.git

To build using ant do:
$ ./pull-binary-libs.sh (has to be done only once for your working copy)
$ ant build

This produces jars of the compiler and library in build/pack/lib, as well as runner scripts in build/pack/bin.

After that, you need to update the path pointing to your local Scala build in `project/SparkBuild.scala` on line 123, ending in "build/pack/".

To build:
$ sbt/sbt -Pscala-2.11 update (should fetch a current snapshot version of macro paradise, which fixes a binary compatibility issue)
$ sbt/sbt -Pscala-2.11 assembly

To test the spark-shell, of course just do:
$ ./bin/spark-shell

ping @mateiz