fixes https://issues.apache.org/jira/browse/SPARK-1825

This is my attempt to fix https://issues.apache.org/jira/browse/SPARK-1825.

Tested on Windows 7 and Hortonworks HDP 2.1 Sandbox, and it works.
Tow more problems reported in SPARK-18259(SPARK_HOME, %HADOOP_MAPRED_HOME%) were gone, perhaps by other commits that took after rc5.

But this fix is Hadoop 2.4.0+ only, since it uses new APIs introduced by https://issues.apache.org/jira/browse/YARN-1824.
So, version checking may be needed, but my knowledge for the Spark source code is limited, so I don't know how to do it.