apache/tajo — pull request #115 by babokim (2014-08-13 13:10:28+00:00)
Title: TAJO-992: Reduce number of hash shuffle output file.

For this I added the following features.
- HashShuffleAppender which is created a single instance each a ExecutionBlock and Partition in a Worker.
Therefore, all execution block's tasks in a worker share a HashShuffleAppender. Each task's HashShuffleWriteExec calls HashShuffleAppender.appends() every 'tajo.shuffle.hash.appender.buffer.size' tuples(default is 10,000) for coarse-grained lock.
- Splittable IntermediateEntry
If a intermediate file is large, it is difficult to process with multiple tasks. New IntermediateEntry class has page meta data which contains start position and length every 'tajo.shuffle.hash.appender.page.volumn-mb' value(default: 30MB). Repartitioner class use that meta data for making proper number of tasks.
- Failure awareness IntermediateEntry
If specified task is failed, failed task's tuples in the intermediate file should be removed. But this is impossible because that tuples are already written in a file. For this IntermediateEntry has Task's tuple index meta. RawFile's scanner can use this data. But in this patch that meta is not used. I'll create another for this.