apache/spark — pull request #1390 by concretevitamin (2014-07-12 23:52:58+00:00)
Title: [SPARK-2443][SQL] Fix slow read from partitioned tables 

This simply incorporates Shark's [#329](https://github.com/amplab/shark/pull/329) into Spark SQL. Implementation credit to @chiragaggarwal.

@marmbrus @rxin @chenghao-intel

## Benchmarks
Generated a local text file with 10M rows of simple key-value pairs. The data is loaded as a table through Hive. Results are obtained on my local machine using hive/console.

Without the fix:

Non-partitioned | Partitioned (1 part)
------------ | -------------
First run: 9.52s end-to-end (1.64s Spark job) | First run: 36.6s (28.3s)
Stablized runs: 1.21s (1.18s) | Stablized runs: 27.6s (27.5s)

With this fix:

Non-partitioned | Partitioned (1 part)
------------ | -------------
First run: 9.57s (1.46s) | First run: 9.30s (1.45s)
Stablized runs: 1.13s (1.10s) | Stablized runs: 1.18s (1.15s)