twitter/scalding — Issue #7: Problem in compiling on Ubuntu 11.10
Author: hgavert
Created: 2012-02-09 07:57:02+00:00
State: open

Firstly, I'm not familiar with ruby, sbt or scalding and only starting scala... so I might be doing something obviously wrong. So, here is what I just tried to do. This is Ubuntu 11.10 - and system fully up to date with apt-get. So, check the scala version:

```
me@ulap:~/dev/scala$ scalac -version
Scala compiler version 2.9.0.1 -- Copyright 2002-2011, LAMP/EPFL
```

Then get SBT

```
# https://github.com/harrah/xsbt/wiki/Getting-Started-Setup
me@ulap:~/bin$ wget http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-tools.sbt/sbt-launch/0.11.2/sbt-launch.jar
# echo java -Xmx512M -jar `dirname $0`/sbt-launch.jar "$@" > sbt
```

Then get scalding:

```
me@ulap:~/dev/scala$ git clone http://github.com/twitter/scalding.git
Cloning into scalding...
remote: Counting objects: 134, done.
remote: Compressing objects: 100% (97/97), done.
remote: Total 134 (delta 40), reused 110 (delta 16)
Receiving objects: 100% (134/134), 78.49 KiB, done.
Resolving deltas: 100% (40/40), done.
```

Next change the scala version & compiler

```
# edit built.sbt
# scalaVersion := "2.9.0.1"

# edit scripts/scald.rb
# COMPILE_CMD="scalac"
```

However running sbt update with this version fails:

```
me@ulap:~/dev/scala/scalding$ ~/bin/sbt update
[info] Loading project definition from /home/me/dev/scala/scalding/project
[info] Updating {file:/home/me/dev/scala/scalding/project/}default-b80ea6...
[info] Resolving com.eed3si9n#sbt-assembly;0.7.3 ...
[info] Resolving org.scala-tools.sbt#sbt_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#main_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#actions_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#classfile_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#io_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#control_2.9.1;0.11.2 ...
[info] Resolving org.scala-lang#scala-library;2.9.1 ...
[info] Resolving org.scala-tools.sbt#interface;0.11.2 ...
[info] Resolving org.scala-tools.sbt#logging_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#process_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#classpath_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#launcher-interface_2.9.1;0.11.2 ...
[info] Resolving org.scala-lang#scala-compiler;2.9.1 ...
[info] Resolving org.scala-tools.sbt#incremental-compiler_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#collections_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#api_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#persist_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbinary#sbinary_2.9.0;0.4.0 ...
[info] Resolving org.scala-tools.sbt#compile_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#ivy_2.9.1;0.11.2 ...
[info] Resolving org.apache.ivy#ivy;2.2.0 ...
[info] Resolving com.jcraft#jsch;0.1.31 ...
[info] Resolving commons-httpclient#commons-httpclient;3.1 ...
[info] Resolving commons-logging#commons-logging;1.0.4 ...
[info] Resolving commons-codec#commons-codec;1.2 ...
[info] Resolving org.scala-tools.sbt#completion_2.9.1;0.11.2 ...
[info] Resolving jline#jline;0.9.94 ...
[info] Resolving org.scala-tools.sbt#run_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#task-system_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#tasks_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#tracking_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#cache_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#testing_2.9.1;0.11.2 ...
[info] Resolving org.scala-tools.testing#test-interface;0.5 ...
[info] Resolving org.scala-tools.sbt#compiler-interface;0.11.2 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_8_1;0.11.2 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_8_0;0.11.2 ...
[info] Resolving org.scala-tools.sbt#precompiled-2_9_0;0.11.2 ...
[info] Done updating.
[info] Set current project to scalding (in build file:/home/me/dev/scala/scalding/)
Getting Scala 2.9.0.1 ...

:: problems summary ::
:::: WARNINGS
module not found: org.scala-lang#scala-compiler;2.9.0.1

==== local: tried

/home/me/.ivy2/local/org.scala-lang/scala-compiler/2.9.0.1/ivys/ivy.xml

==== typesafe-ivy-releases: tried

http://repo.typesafe.com/typesafe/ivy-releases/org.scala-lang/scala-compiler/2.9.0.1/ivys/ivy.xml

==== Maven Central: tried

http://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.9.0.1/scala-compiler-2.9.0.1.pom

==== Scala-Tools Maven2 Repository: tried

http://scala-tools.org/repo-releases/org/scala-lang/scala-compiler/2.9.0.1/scala-compiler-2.9.0.1.pom

==== Scala-Tools Maven2 Snapshots Repository: tried

http://scala-tools.org/repo-snapshots/org/scala-lang/scala-compiler/2.9.0.1/scala-compiler-2.9.0.1.pom

module not found: org.scala-lang#scala-library;2.9.0.1

==== local: tried

/home/me/.ivy2/local/org.scala-lang/scala-library/2.9.0.1/ivys/ivy.xml

==== typesafe-ivy-releases: tried

http://repo.typesafe.com/typesafe/ivy-releases/org.scala-lang/scala-library/2.9.0.1/ivys/ivy.xml

==== Maven Central: tried

http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.9.0.1/scala-library-2.9.0.1.pom

==== Scala-Tools Maven2 Repository: tried

http://scala-tools.org/repo-releases/org/scala-lang/scala-library/2.9.0.1/scala-library-2.9.0.1.pom

==== Scala-Tools Maven2 Snapshots Repository: tried

http://scala-tools.org/repo-snapshots/org/scala-lang/scala-library/2.9.0.1/scala-library-2.9.0.1.pom

::::::::::::::::::::::::::::::::::::::::::::::

:: UNRESOLVED DEPENDENCIES ::

::::::::::::::::::::::::::::::::::::::::::::::

:: org.scala-lang#scala-compiler;2.9.0.1: not found

:: org.scala-lang#scala-library;2.9.0.1: not found

::::::::::::::::::::::::::::::::::::::::::::::

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
unresolved dependency: org.scala-lang#scala-compiler;2.9.0.1: not found
unresolved dependency: org.scala-lang#scala-library;2.9.0.1: not found
Error during sbt execution: Error retrieving required libraries
(see /home/me/.sbt/boot/update.log for complete log)
[error] {file:/home/me/dev/scala/scalding/}default-bc2517/*:scala-instance: xsbti.RetrieveException: Could not retrieve Scala 2.9.0.1
[error] Total time: 3 s, completed Feb 9, 2012 9:22:34 AM
me@ulap:~/dev/scala/scalding$

```

So, try with 2.9.0

```
# so, chance the version to 2.9.0
# edit built.sbt
# scalaVersion := "2.9.0"
```

sbt update works:

```
me@ulap:~/dev/scala/scalding$ ~/bin/sbt update
...
[success] Total time: 3 s, completed Feb 9, 2012 9:27:08 AM
```

Then sbt test - which fails like this:

```
me@ulap:~/dev/scala/scalding$ ~/bin/sbt test
[info] Loading project definition from /home/me/dev/scala/scalding/project
[info] Set current project to scalding (in build file:/home/me/dev/scala/scalding/)
[info] Compiling 18 Scala sources to /home/me/dev/scala/scalding/target/scala-2.9.0/classes...
[error] /home/me/dev/scala/scalding/src/main/scala/com/twitter/scalding/Source.scala:145: type mismatch;
[error] found : Seq[cascading.tap.Tap[_ <: cascading.flow.FlowProcess[_], _, _, _]]
[error] required: Seq[cascading.tap.Tap[?>: Nothing <: cascading.flow.FlowProcess, ?>: Nothing <: Any, ?>: Nothing <: Any, ?>: Nothing <: Any]]
[error] case _ => new MultiSourceTap(taps.toSeq : _*)
[error] ^
[error] /home/me/dev/scala/scalding/src/main/scala/com/twitter/scalding/Source.scala:173: type mismatch;
[error] found : Seq[cascading.tap.Tap[_ <: cascading.flow.FlowProcess[_], _, _, _]]
[error] required: Seq[cascading.tap.Tap[?>: Nothing <: cascading.flow.FlowProcess, ?>: Nothing <: Any, ?>: Nothing <: Any, ?>: Nothing <: Any]]
[error] JobConf, RecordReader[_,_], OutputCollector[_,_]](taps.toSeq : _*)
[error] ^
[error] two errors found
[error] {file:/home/me/dev/scala/scalding/}default-bc2517/compile:compile: Compilation failed
[error] Total time: 31 s, completed Feb 9, 2012 9:29:53 AM
```

It would be really helpful to have good documentation on the installation parameters. What to change for different versions of scala or hadoop (I didn't even get that far, but I know it will come) as well as host name settings and whatever you have there.