apache/tajo — pull request #192 by jihoonson (2014-10-08 08:46:29+00:00)
Title: TAJO-838: Improve query planner to utilize index

Hi guys.

This is an ongoing work. Even though there still remain critical problems for practical uses, I'd like to share the progress of this issue.

Finally, I succeeded to utilize the index for query processing.
To show the effectiveness of the index, I carried out a simple performance test as follows.

* Environments: an in-house cluster that consists of one master and 32 workers.
* Data: TPC-DS store_sales table at scale factor 100 (41 GB).
* DDL for index creation: create index ss_item_sk_idx on store_sales (ss_item_sk asc null first);
* Test query: select ss_item_sk from store_sales where ss_item_sk = 1; (selectivity = 0.000045139%)
* Result

| | Without disk cache | With disk cache |
|--- | --- | ---|
| Without index | 23.917 | 19.154 |
| With index | 4.207 | 3.995 |

Although the selectivity of the query is very low, I think that this result shows a potential benefit of index.

Here are some remaining issues.
* Selectivity estimation. In the current patch, index utilization is forced when it exists. I'll improve this to use the index when it is beneficial.
* Support index for partitioned tables
* Consider the case when the query predicate includes two or more columns.
* Code refactoring and potential bug fixes
* Add more tests