apache/spark — pull request #3559 by brennonyork (2014-12-02 19:38:31+00:00)
Title: [SPARK-4616][Core] - SPARK_CONF_DIR is not effective in spark-submit

By default the `SPARK_CONF_DIR` is not capable of being set from the `spark-submit` script, but a spark properties file is. After diving through the code it turned out that the `SPARK_CONF_DIR` is actually a cyclic reference as it is referenced within the `load-spark-env.sh` script and can also then be reset within the loaded `spark-env.sh` file. What's worse is that, if the `spark-env.sh` defined a `SPARK_CONF_DIR` it wouldn't be picked up and used to grab the default `spark-defaults.conf` if no `--properties-file` flag was present. As such, it seemed best to provide a `--environment-file` flag that can be used to present an arbitrary bash file to the system which would preload any necessary environment configuration options (including `SPARK_CONF_DIR`). This then solves the original problem that the `SPARK_CONF_DIR` wasn't effective within `spark-submit`, but also removes the cyclic dependency on where and when the `SPARK_CONF_DIR` is loaded.