apache/spark — pull request #787 by andrewor14 (2014-05-15 07:25:16+00:00)
Title: [WIP] [SPARK-1808] Route bin/pyspark through Spark submit

*Problem.* For `bin/pyspark`, there is currently no other way to specify Spark configuration properties other than through `SPARK_JAVA_OPTS` in `conf/spark-env.sh`, which is supposedly deprecated. It needs to pick up configurations explicitly specified in `conf/spark-defaults.conf` instead.

*Solution.* Have `bin/pyspark` invoke `bin/spark-submit`, like all of its counterparts in Scala land (i.e. `bin/spark-shell`, `bin/run-example`). This has the additional benefit of making the Spark scripts consistent with each other.

---

*Details.* `bin/pyspark` inherently handles two cases: (1) running python applications and (2) running the python shell. For (1), Spark submit already offers an existing code path to run python applications. For cases in which `bin/pyspark` is given a python file, we can simply call pass the file directly to spark-submit. Here, the JVM launches the python application as a sub-process. This is the simple case:

- `bin/pyspark` passes the python file to Spark submit
- Spark submit passes the python file to `PythonAppRunner`
- `PythonAppRunner` sets up the Py4j GatewayServer on the Java side
- `PythonAppRunner` runs the python file as a sub-process

Case (2) is more involved. We cannot simply run the shell as another application, and use the existing code path in Spark submit as in (1). This is because the keyboard signals will not be propagated to the python interpreter properly, and dealing with each signal individually is cumbersome and likely not comprehensive. Thus, this PR takes the approach of making Python the parent process instead. This allows all keyboard signals to be propagated to the python REPL first, rather than to the JVM first:

- `bin/pyspark` calls `python/pyspark/repl.py`
- `repl.py` calls Spark submit as a sub-process
- Spark submit calls `PythonShellRunner`
- `PythonShellRunner` sets up the Py4j GatewayServer on
- `repl.py` learns the Py4j gateway server port from `PythonShellRunner` through sockets
- `repl.py` creates a SparkContext using this gateway server
- `repl.py` starts a REPL with this SparkContext