mattpap/IScala — pull request #21 by hvanhovell (2014-11-11 20:27:54+00:00)
Title: Spark backend for IScala

IScala Spark Backend
=================
Introduction
----------------
The Spark large scale data processing framework provides a REPL which allows for interactive exploration, analysis and manipulation of large amounts of data. This PR adds this Spark backend to IScala. The combination of iScala's/iPythons ease of use and visualization capabilites and Spark's processing power should be incredibly useful. This work is partially inspired and based upon [ISpark](https://github.com/tribbloid/ISpark).

Design & Implementation
--------------------
* One of the major hurdles in creating the Spark backend is the fact that the Scala and Spark IMain classes do not have a shared interface. So in order to get this to work, a (thin) common interface (IMainBackend) was introduced for both IMain classes and the Interpreter now uses this common interface.
* The interpreter class has been modified in order to make it the common gateway for IScala interpreter operations (this intead of accessing an IMain instance directly).
* An interpreter factory option (--interp) has been added to the options. This interpreter factory helps the IScala class construct the Interpreter.

Running
-----------
To start IPython with IScala Spark backend manually, issue:

ipython notebook --KernelManager.kernel_cmd='["$SPARK_HOME/dist/bin/spark-submit", "--master", "$MASTER", "--driver-memory", "2G", "--class", "org.refptr.iscala.IScala", "$ISCALA/IScala.jar", "--connection-file", "{connection_file}", "--parent", "--interp", "org.refptr.iscala.SparkInterpreterFactory"]'

Replace $SPARK_HOME by the location of the spark distribution, $MASTER by the master URL of the Spark master node and $ISCALA by the location of the IScala jar.

NB: Spark is currently Scala 2.10 only!

TODO's
-----------
* Undo separation of Interpreter interface and IScalaInterpreter implementation.
* Tests seem to be broken.
* Created (temporary) SBT logger which doesn't use JLine. The JLine version used in Spark collides with the JLine version used by SBT...
* Spark version used 1.2.0-SNAPSHOT. We should use a production version.
* Reset does not work in Spark. This is because of a bind problem in Spark.