apache/spark — pull request #2188 by liancheng (2014-08-29 00:38:05+00:00)
Title: [SPARK-2961][SQL] Use statistics to prune batches within cached partitions

This PR is based on #1883 authored by @marmbrus. Key differences:

1. Batch pruning instead of partition pruning

When #1883 was authored, batched column buffer building (#1880) hadn't been introduced. This PR combines these two and provide partition batch level pruning, which leads to smaller memory footprints and can generally skip more elements. The cost is that the pruning predicates are evaluated more frequently (partition number multiplies batch number per partition).

1. More filters are supported

Filter predicates consist of `=`, `<`, `<=`, `>`, `>=` and their conjunctions are supported. Disjunctions are not supported yet. In short,

```sql
SELECT * FROM t WHERE i > 0 AND i < 10
```

is executed in a more smart way, while

```sql
SELECT * FROM t WHERE i < 0 OR i > 10
```

still scans the whole table.