amplab/shark — pull request #315 by brianbrunner (2014-03-28 01:16:25+00:00)
Title: Add Python SharkContext

I've added a working implementation of a python SharkContext. The biggest functionality it exposes is a python version of `sql2rdd`, backed by the PythonTableRDD from https://github.com/amplab/shark/pull/303.

The only big issue it has currently is that it uses a hack for setting up the Py4j gateway. It injects a bit of code into the `bin/compute-classpath.sh` file in the `SPARK_HOME` directory to add the `HIVE_CONF_DIR` and the assembled shark jar to the Py4j gateway's classpath. The shark jar must be added after the spark jar, so unfortunately we can't just set the `SPARK_CLASSPATH` env variable. It works and does clean up after itself, but overall this method is less than ideal so if someone has a suggestion I'd be willing to rework it.

You should be able to play around with this by running the `./bin/pyshark` script.