apache/spark — pull request #1864 by liancheng (2014-08-09 01:13:27+00:00)
Title: [SPARK-2894][Core] Fixes spark-shell and pyspark CLI options

This PR tries to fix SPARK-2894 in a way different from #1825, namely filtering out all `spark-submit` options from user application options, since we have control over option list of `spark-submit`. By extracting the filtering into a utility shell function, fixing other shell scripts can be much easier, and adding new `spark-submit` options wouldn't need too much duplicated work. In the mid term, we still need a cleaner solution such as what has been discussed in #1715.

All mode `pyspark` supported (Python, IPython and deprecated `pyspark app.py` style) should work. @JoshRosen @davies It would be great if you can help review PySpark related code, thanks!

An open issue is Windows scripts need also to be updated.