apache/spark — pull request #603 by andrewor14 (2014-04-30 23:50:23+00:00)
Title: [SPARK-1688] PySpark throws unhelpful exception when pyspark is not found

Currently, if pyspark cannot be loaded for any reason, Spark throws a random `java.io.EOFException` when trying to read from the python daemon's output. This can be caused by one of the following:

(1) PYTHONPATH is not set
(2) PYTHONPATH does not contain the python directory (or jar, in the case of YARN)
(3) The jar does not contain pyspark files (YARN)
(4) The jar does not contain py4j files (YARN)

This PR distinguishes between these cases and throws a more helpful exception earlier with the appropriate error message.

Tested on both a standalone cluster and a YARN cluster.