freeman-lab/thunder — pull request #73 by industrial-sloth (2014-12-17 22:57:02+00:00)
Title: updates to launch scripts

This PR provides updated versions of the existing `thunder` and `thunder-submit` scripts which explicitly expose the relevant options available in their Spark equivalents, `pyspark` and `spark-submit`. This is intended to resolve #54. It also updates `thunder-ec2.py` to expose more of the options available in the equivalent `spark-ec2.py` script, and adds a `--thunder-version` option in thunder-ec2.

The new `thunder-shell.py` script is intended as a replacement for the existing `thunder` script (although the current `thunder` script is left in place for now, pending the results of testing). `thunder-submit.py` is intended to replace the existing `thunder-submit`. Both rely on a new module `thunder.utils.launchutils`, which provides a few utility functions for the two scripts and a common set of command-line options. These options are manually built to mirror those in the Spark scala file `SparkSubmitArguments.scala` - this does set up something of a future maintenance nightmare, but considering the difference in languages I'm not seeing a great alternative.

Since these scripts import from `thunder.utils.launchutils`, the script that launches the interactive Thunder shell can no longer be called just `thunder` - with that name, it tries to resolve the `thunder.utils.launchutils` module within a subdirectory of `$THUNDER_HOME/bin`, where the launch scripts live, rather than under `$THUNDER_HOME/python/thunder` where they actually are. (We could address this by moving launchutils.py into bin/, but that then stretches the interpretation of a bin directory rather far.)

A few of the options exposed here have different defaults than in the underlying scala-based CLIs. The `--class` and `--deploy-mode` options don't apply for Thunder, which only supports Python code (only `--deploy-mode client` is currently supported for Python scripts by Spark), and so they aren't included here.

Finally, the `--thunder-version` option in thunder-ec2 lets users specify a Thunder version either by dotted version number (e.g. "0.4.1") or by git commit hash, similarly to how Spark versions are handled by the `--version` flag. The default for `--thunder-version` is "HEAD", which will download the current tip of the Thunder master branch from github. This preserves the current behavior of the script.