DataTorrent/Malhar — pull request #1030 by siyuanh (2014-08-27 07:55:37+00:00)
Title: Add DTFile and performance test for DTFile

The file format of DTFile is same as {@link TFile} with different reader implementation.
It reads data block by block and cache the binary block data into memory to speed up the random read.

The public api of DTFile Reader is as same as it is in TFile.Reader implementation.
Besides, it provides getBlockBuffer(), getKeyOffset(), getKeyLength(), getValueOffset(), getValueLength() method to expose raw block, key, value data to user to avoid unnecessary internal/external data copy

In the performance test, It shows no difference in sequential reads and 20x faster in random reads(If most of them are memory hits)

It also provides a global cache which can be shared by multiple readers and users can constraint size of cache by the num of entries, memory usage(in bytes) or percentage of available heap size