fixes https://issues.apache.org/jira/browse/SPARK-1825

This is my attempt to fix https://issues.apache.org/jira/browse/SPARK-1825.

Tested on Windows 7 and Hortonworks HDP 2.1 Sandbox, and it works.
Two more problems reported in SPARK-1825(SPARK_HOME, %HADOOP_MAPRED_HOME%) were gone, perhaps by other commits that took after rc5.

WARNING:
But this fix is Hadoop 2.4.0+ only, since it uses new APIs introduced by https://issues.apache.org/jira/browse/YARN-1824.
So, version checking may be needed, but my knowledge for the Spark source code is limited, so I don't know how to do it.

I'm not quite sure what you mean by a config option.  you mean build time?  Otherwise I would expect anything <2.4 to fail the to build.