apache/incubator-spark — pull request #60 by ScrapCodes (2013-10-15 10:58:15+00:00)
Title: [SPARK-743] A new resource manager using Akka cluster.

This is only a preview branch for testing and getting an initial feedback.

Improvements over current standalone mode.
1) There is no master, so no need of state replication using zookeeper. But unfortunately spark is still not highly available and that is on my TODO list. (Hint: Work is in progress for making driver state replicated in cas style). Ping me for more details.

2) Very little code and cluster matrices are inbuilt in akka(which is also pluggable) so no extra work for that.

3) We can simplify deployments with this as in I hope nodes will node need to have same path for spark home on all node. Just export the env variable SPARK_HOME and done.

Okay lets see it all in action, so steps to try this out locally !
Steps 1) sbt/sbt assembly
2) start worker(s) but first seed worker is started as follows
`./spark-class org.rm.WorkerRunner 1 1024 ~/work/temp/ 2552`
and then more(if you really want to) like this
`./spark-class org.rm.WorkerRunner 1 1024 ~/work/temp/ 2552`

3) start your job like this `./run-example org.apache.spark.examples.SparkPi rm-mode `
OR
start repl like this.
`$. export MASTER="rm-mode"`
`$. ./spark-shell`

Above is to try out things locally only mentioned for convenience. Testing it distributed would require you to change the seed nodes only.

Also I do realize the package names etc... don't follow spark apache convention but that is easy to fix as this is only to get feedback.

I will be happy to join forces with anyone interested to work on it and welcome criticism and questions alike.