apache/spark — pull request #2516 by tigerquoll (2014-09-24 07:28:22+00:00)
Title: Spark Core - [SPARK-3620] - Refactor of SparkSubmit Argument parsing code

Argument processing seems to have gotten a lot of attention lately, so I thought I might throw my contribution into the ring. Attached for consideration and to prompt discussion is a revamp of argument handling in SparkSubmit aimed at making things a lot more consistent. The only things that have been modified are the way that configuration properties are read/ processed and prioritised

Things to note include:
* All configuration parameters can now be consistently set via config file

* Configuration parameters defaults have been removed from the code, and placed into a property file which is read from the class path on startup. There should be no need to trace through 5 files to see what a config parameter defaults to if it is not specified, or have different default values applied in multiple places throughout the code.

* Configuration parameter validation is now done once all configuration parameters have been read in and resolved from various locations, not just when reading the command line.

* All property files (including spark_default_conf) are parsed by Java property handling code. All custom parsing code has been removed. Escaping of characters should now be consistent everywhere.

* All configuration parameters are overridden in the same consistent way - configuration parameters for sparkSubmit are pulled form the following sources in order of priority
1. Entries specified on the command line (except from --conf entries)
2. Entries specified on the command line with --conf
3. Environment variables (including legacy variable mappings)
4. System config variables (eg by using -Dspark.var.name)
5. $(SPARK_DEFAULT_CONF)/spark-defaults.conf or $(SPARK_HOME)/conf/spark-defaults.conf if either exist
6. Hard coded defaults in class path at spark-submit-defaults.prop

* A property file specified by one of the sources listed above gets read in and the properties are considered to be at the priority of the configuration source that specified the file. A property specified in a property file will not override an existing config value already specifiedby that configuration source

The existing argument handling is pretty finicky - chances are high that I’ve missed some behaviour - if this PR is going to be accepted/approved let me know any bugs and I’ll fix them up and document the behaviour for future reference