SPARK-2058: Overriding config from SPARK_HOME with SPARK_CONF_DIR

Fixes https://issues.apache.org/jira/browse/SPARK-2058

The general idea is to use SPARK_CONF_DIR content instead of SPARK_HOME/conf when defined.

 - When SPARK_CONF_DIR is defined, spark-defaults.conf will be loaded from there. 
 - SPARK_CONF_DIR is added to the classpath, overriding the default conf from SPARK_HOME/conf
Use Files.createTempDir(), and delete it when the test is done (using scalatest's BeforeAndAfter); there are several examples of this in other tests.