[SPARK-2460] Optimize SparkContext.hadoopFile api

1 use SparkContext.hadoopRDD() instead of instantiate HadoopRDD directly in SparkContext.hadoopFile.
   SparkContext.hadoopRDD()  Add necessary security credentials to the JobConf before broadcasting it.

2 broadcast jobConf in HadoopRDD, not Configuration. this will resolve the dead lock issue----
now HadoopRDD broadcast Configuration and in each task (compute method) to get jobConf 
then the lock of 
 conf.synchronized {
      val newJobConf = new JobConf(conf)
      initLocalJobConfFuncOpt.map(f => f(newJobConf))
      HadoopRDD.putCachedMetadata(jobConfCacheKey, newJobConf)
      newJobConf
     }
will conflict with hadoop version fix  Hadoop-10456:

hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
synchronized(Configuration.class) {
       REGISTRY.put(this, null);
     }

we can reproduce the bug like this:
hadoop version 2.4.1
spark master branch

hql("SELECT t1.a, t1.b, t1.c FROM table_A t1 JOIN table_A t2 ON (t1.a = t2.a)") 
   
Do you know if there was ever a reason we broadcasted the Configuration and this lambda, just to turn it into a JobConf on the Executor side? Was it really just extra work for no reason?