twitter/scalding — pull request #708 by miguno (2013-12-07 21:04:12+00:00)
Title: Improve compatibility with Scala 2.10.x

_Disclaimer: I am still getting up to speed with Scala and notably sbt, and the issue addressed by this pull request was simply the blocker to successfully run the "Getting Started" Scalding example with Scala 2.10.3. I do not claim that this is the best way to fix the problem I am describing below, though I can confirm it does nicely work in my 2.10.3 based environment now. If in doubt, treat the pull request and this bug report as a good starting point to address the problem in a different way. :-)_

While trying to build and use Scalding with Scala 2.10.3 (with sbt 0.13.0) I ran into a kind of version conflict between the Scala version wired to to sbt (2.10.2) and my non-sbt Scala 2.10.x version (2.10.3).

**How to reproduce**

1. Create a fresh clone of Scalding, branch `develop`.
1. Change `project/Build.scala` from Scala 2.9.2 to 2.10.3. See my updated [Build.scala](https://github.com/miguno/scalding/commit/8442ba7d4cd768fd90ab15d5e3bb1743882b43ba) in my fork's `scala-2.10.3 branch` for details; I intentionally did not send a pull request with `scalaVersion` changed from 2.9.2 to 2.10.3 so that the default build properties of Scalding remain unchanged.
2. Run `./sbt update && ./sbt clean && ./sbt assembly`.

You will run into the following, misleading error:

```
java.lang.ClassCastException: com.twitter.scalding.Opaque cannot be cast to java.lang.Comparable
at com.twitter.scalding.IntegralComparator.compare(IntegralComparator.scala:54)
```

**How to fix**

As shown in the pull request the relevant line in `project/Build.scala` is:

```scala
// causes problems
scalaVersion := "2.10.3"

// works
scalaVersion in ThisBuild := "2.10.3"
```

Now running `./sbt assembly` will work.

However, this alone is not enough to address Scala 2.10.x support for Scalding. One must also adjust `scripts/scald.rb`.

**Modifications required for scripts/scald.rb**

1. `scald.rb` must be updated if only to update a regex that parses `scalaVersion` from `Build.scala`.
2. `scald.rb` must be updated to find the Scala libraries (scala-compiler, scala-library, scala-reflect) not under `SBT_HOME/boot` but under `$HOME/.ivy2`. This is because sbt 0.13.0 will only ever download and store Scala 2.10.2 under `SBT_HOME/boot`, so the current `scald.rb` will fail to find the three Scala 2.10.3 jars.

**Potential issues of this pull request**

_1) -Dscala.home is not set in the case of Scala 2.10.x_

I could not continue to set `-Dscala.home` for `COMPILE_CMD` in `scald.rb` _when using Scala 2.10.x_ (note: the behavior when using Scala 2.9.2 _is not_ changed, so Twitter's 2.9.2 setup should be fine) because from what I understand there isn't an appropriate value for `scala.home` in the `$HOME/.ivy2` case. In my testing removing `-Dscala.home=...` did not affect running `scald.rb` -- everything still worked.

Note that I couldn't test `scripts/scald-repl.sh` because I have been getting the following error right from the start, i.e. before and after this code change, so I'd say this issue is not related to the pull request.

```
# When trying to run scald-repl.sh (before AND after this code change)
Exception in thread "main" java.lang.NoClassDefFoundError: com/twitter/scalding/ScaldingShell
Caused by: java.lang.ClassNotFoundException: com.twitter.scalding.ScaldingShell
```

_2) Untested with (default) Scala 2.9.2_

I haven't tested the PR with Scala 2.9.2. From what I understand the 1-line change to `Build.scala` should work with 2.9.2, too, but I am not 100% sure. In the case of `scald.rb` it should be even safer: The changes to `scald.rb` only trigger when the Scala version is 2.10.x, so anyone using `scald.rb` with 2.9.x should not be affected at all.

**Pull request details:**

This fix is required to untangle the Scala version "wired" to the
installed version of sbt. For instance, this allows us to build
Scalding with sbt 0.13.0 and scalaVersion set to Scala 2.10.3 even
though sbt 0.13.0 uses Scala 2.10.2.

Without this change we run into the following error when having sbt
0.13.0 and Scala 2.10.3 installed locally, and set scalaVersion to
2.10.2 in projects/Build.scala:

java.lang.ClassCastException: com.twitter.scalding.Opaque cannot be cast to java.lang.Comparable
at com.twitter.scalding.IntegralComparator.compare(IntegralComparator.scala:54)

See the following related discussions:

- Fix compiler library in SBT 0.13.x with Scala != 2.10.2
https://github.com/mpeltonen/sbt-idea/pull/253
- How does sbt choose which Scala version to use?
http://stackoverflow.com/questions/14572148
- How to change Scala version for build definition?
http://stackoverflow.com/questions/11948002