mesos/spark — pull request #942 by harveyfeng (2013-09-22 21:08:50+00:00)
Title: Cache Configurations and InputFormats across Hadoop file reads. 

Currently motivated by Shark queries on Hive-partitioned tables, where there's a JobConf broadcast for every Hive-partition (i.e., every subdirectory read). This PR only modifies the old Hadoop API RDDs, but similar additions to the new API might reduce computation latencies for FileInputDStream (it only uses the new API right now).
As a bonus, added InputFormats caching, to avoid reflection calls for every RDD#compute().

Few implementation nodes that aren't ideal:
- Added a general soft-reference hashmap in SparkHadoopUtil because I wanted to avoid adding another class to SparkEnv.
- SparkContext default hadoopConfiguration isn't cached. There's no equals() method for Configuration, so there isn't a good way to determine when configuration properties have changed.
- SparkContext does the broadcasting before creating HadoopRDDs. If this is terrible, it might be possible to add a factory for HadoopRDDs.