amplab/shark — pull request #149 by chenghao-intel (2013-09-06 08:53:11+00:00)
Title: CodeGen for Query Engine

Hi, guys,
In order to reduce the dependencies of Hive, and remove the "interpret" running mode for queries, I finish first prototype Query Engine CodeGen.
The pull request is only for prototype review/discussion purpose, Please DO NOT merge to trunk. Any suggestions are welcome.

It generates 3 types of source code:
1) Row Object ( to replace the ObjectInspector + Object )
2) Operator ( to replace the literals within the rdd.mapPartition etc.)
3) ObjectInspector ( backward-compatible with the exists Operator / Evaluator to access the RowObject)

Currently, I merely rewrote the JoinOperator / GroupByPreShuffleOperator / GroupByPostShuffleOperator / ReduceSinkOperator), and also it doesn't support the DISTINCT select yet.

Some open issues (TODOs):
a. ClassLoader (instance per task, not per slave, performance issues)
b. Java Compiler API (called multiple times in generating Operator, should just compile once on master)
c. Performance Tuning (Row Object ser/de need some tuning on performance etc.)
d. Stability (lack of unit tests / integration test)
e. Integrated with Expression Evaluation CodeGen
f. Remove all of the ObjectInspector from the generated operators.

Some tips:
a. JVM flags:
Output the generated java source code in folder
-Dcg_source_code_path=/home/hcheng/git/sotc_cloud-shark/target/cgjava
Increase the permgen memory size
-Xms256m -Xmx1024m -XX:PermSize=512m -XX:MaxPermSize=512m

b. disable / enable the codegen feature:
In shark REPL: set shark.expr.cg=true/false;

We can have detailed discussion on the code review system.

Thanks,
Cheng Hao