DataTorrent/Malhar — pull request #1029 by siyuanh (2014-08-26 21:59:39+00:00)
Title: Add dtfile to historical data store

The file format of DTFile is same as TFile but different reader implementation. It reads data block by block and cache the binary block data into memory to speed up the random reads.

The reader interface is same as hadoop default TFile implementation. Moreover, it provides getBlockBuffer(), getKeyOffset(), getKeyLength(), getValueOffset(), getValueLength() method to expose raw block, key, value data to user to avoid unnecessary internal/external data copy

In the performance test I see no difference in sequential reads and 20x faster in random reads