apache/spark — pull request #1883 by marmbrus (2014-08-11 02:53:29+00:00)
Title: [WIP][SPARK-2961][SQL] Use statistics to skip cached partitions

Adds the ability for `InMemoryColumnarTableScan` to skip partitions that cannot possibly contain any matching rows, based on statistics collected when caching. For example, if we know that in a given partition `max(a) = 10` and there is a predicate `a = 15`, the partition will be skipped.

TODO:
- [ ] More tests
- [ ] Code cleanup