SPARK-2058: Overriding SPARK_HOME/conf with SPARK_CONF_DIR

Update of PR #997.

With this PR, setting SPARK_CONF_DIR overrides SPARK_HOME/conf (not only spark-defaults.conf and spark-env).
A better way to write this is:
```
deployMode = Option(deployMode).orElse(env.get("DEPLOY_MODE")).orNull
```