SPARK-2058: Overriding config from SPARK_HOME with SPARK_CONF_DIR

Fixes https://issues.apache.org/jira/browse/SPARK-2058

The general idea is to use SPARK_CONF_DIR content instead of SPARK_HOME/conf when defined.

 - When SPARK_CONF_DIR is defined, spark-defaults.conf will be loaded from there. 
 - SPARK_CONF_DIR is added to the classpath, overriding the default conf from SPARK_HOME/conf
Just a clarification: I see you're deleting the directory below, so BeforeAndAfter does not apply here. But still I'd prefer to use Files.createTempDir() instead of a hardcoded name. Just for paranoia.