apache/incubator-spark — pull request #603 by nicklan (2014-02-15 05:48:26+00:00)
Title: Tachyon scripts

Initial work to get tachyon scripts going. This works in the testing I've done so far. To use:

1. call make-distribution.sh --with-tachyon --tgz
=> This builds the tgz and puts the tachyon fatjar in dist/jars/tachyon-0.4.0-jar-with-dependencies.jar

2. copy the resultant tgz to the hosts in the cluster and unpack

4. edit conf/slaves on the master node and add the slave nodes in the cluster

3. call sbin/start-all.sh --tachyon
=> You should now have a spark and tachyon master running on the master node, and spark and tachyon workers running on all the slave nodes

notes:

- No zookeeper config yet

- All tachyon stuff lives in sbin/tachyon/. This includes config/logs/underfs etc

- The tachyon webapp uses jetty, which requires a jdk and not a jre to run, so on the default amazon ami it won't work. I've still included all the webapp files however, as we probably want to support that on machines where there is a jdk

- the ensure-conf.sh script generates the initial config from the template. it's passed $SPARK_MASTER_IP so it can configure that, and also computes 2/3rds of the total memory of the machine (as reported in /proc/meminfo) and sets that to the size of the ramdisk.