apache/spark — pull request #1238 by concretevitamin (2014-06-26 23:14:33+00:00)
Title: [SQL] [WIP] Prototype implementation of size estimations for Catalyst logical plans.

The idea is that every Catalyst logical plan gets hold of an Estimates class, the usage of which provides useful estimations on various statistics. See the implementations of `ParquetRelation` and `MetastoreRelation`.

This patch also includes one usage of the estimation interface -- namely, use physical table sizes from the estimate interface to convert an equi-join to a broadcast join (when doing so is beneficial, as determined by a size threshold).

This PR still needs some cleanups, but just want to put it out here first to gather feedback on the interface & high-level approach.

TODOs:
- [ ] Add a separate test suite & improve test coverage.
- [ ] Support `ParquetRelation` in the aforementioned BHJ optimization.