amplab/shark — pull request #190 by harveyfeng (2013-10-16 06:40:15+00:00)
Title: Minimize Configuration broadcast/serialization overhead for HadoopRDDs/Hive-partition scans.

Note: relies on https://github.com/amplab/shark/pull/189

Previously, there was one broadcasted Configuration for each HadoopRDD created for a Hive partition that's stored on disk - e.g., for a table scan over 1000 partitions, there are 1000 Configuration broadcasts.
This change uses the HadoopFileRDD, so that there is one broadcasted Configuration shared across multiple HadoopRDDs created for the table scan.

Also, moved HadoopRDD row deserialization from processPartition() to preprocessRDD() in TableScanOperator. This makes it more consistent with how partitioned tables are processed, and allows HiveConf to be annotated as transient, which should reduce Spark task sizes.