SPARK-2058: Overriding config from SPARK_HOME with SPARK_CONF_DIR

Fixes https://issues.apache.org/jira/browse/SPARK-2058

The general idea is to use SPARK_CONF_DIR content instead of SPARK_HOME/conf when defined.

 - When SPARK_CONF_DIR is defined, spark-defaults.conf will be loaded from there. 
 - SPARK_CONF_DIR is added to the classpath, overriding the default conf from SPARK_HOME/conf
Indeed it's not used in the SparkSubmit, but this is done in order to provide the user defined config to the other components such as logging, Scheduler and metrics.