SPARK-2058: Overriding config from SPARK_HOME with SPARK_CONF_DIR

Fixes https://issues.apache.org/jira/browse/SPARK-2058

The general idea is to use SPARK_CONF_DIR content instead of SPARK_HOME/conf when defined.

 - When SPARK_CONF_DIR is defined, spark-defaults.conf will be loaded from there. 
 - SPARK_CONF_DIR is added to the classpath, overriding the default conf from SPARK_HOME/conf
While we're here, this can be one call to `Utils.deleteRecursively(tmpDir)`