Assorted clean-up for Spark-on-YARN.

In particular when the HADOOP_CONF_DIRS are not specified.