Assorted clean-up for Spark-on-YARN.

In particular when the HADOOP_CONF_DIR is not not specified.