mesos/spark — pull request #160 by dennybritz (2012-08-02 03:44:21+00:00)
Title: Standalone cluster scripts

- Heavily influenced by Hadoop scripts. Basically works the same way, but with argument names :) A list of slaves is expected to be in conf/slaves.
- Added a EC2 script, ec2-standalone, based on the mesos script, that starts a spark cluster in standalone mode. It works the same way as the mesos one (and takes the same arguments). The biggest differences are that there are no zoo keeper nodes, different security groups, and that it executes bin/start-all.sh to setup the cluster.
- Created an AMI for the standalone mode, ami-1ff25976 (https://s3.amazonaws.com/spark-standalone-amis/latest-spark). All it has is the latest spark and scala.
- Tested the scripts locally and on EC2.
- We probably want to write up a tutorial on how to use the scripts, though it should be rather straightforward for people familiar with the hadoop ones.
- One TODO is the configuration. How do we want users to specify things like master port, webUiPort, worker memory, worke cores, and so on? The simplest way would be to put these as env variables into the shark-env.sh, but that can get messy rather quickly.
- Removed explicit netty dependency. This caused trouble because 2 netty jars were included. Akka already has a netty dependency.