apache/spark — pull request #2327 by liancheng (2014-09-09 04:12:54+00:00)
Title: [SPARK-3294][SQL] WIP: eliminates boxing costs from in-memory columnar storage

This is a major refactoring of the in-memory columnar storage implementation, aims to eliminate boxing costs as much as possible. The basic idea is to refactor all major interfaces into a row-based form and use them together with `SpecificMutableRow`. The difficult part is how to adapt all compression schemes, esp. `RunLengthEncoding` and `DictionaryEncoding` to this design. Since in-memory compression is disabled by default for now, and this PR should be strictly better than before no matter in-memory compression is enabled or not, maybe I'll finish that part in another PR.

TODO

- [ ] Benchmark
- [ ] Eliminate boxing costs in `RunLengthEncoding`
- [ ] Eliminate boxing costs in `DictionaryEncoding` (not easy to do without specializing `DictionaryEncoding` for every supported column type)