First cut at dictionary encoding Ints into Bytes. Only works when there are fewer than 256 unique values - categorical data only
First cut at dictionary encoding Ints into Bytes. Only works when there are fewer than 256 unique values - categorical data only
Fixed minbufsize logic.
Converted some commented printlns to debug log messages.
Fixed minbufsize logic. Converted some commented printlns to debug log messages.

Fixed minbufsize logic. Converted some commented printlns to debug log messages.
Short circuit for _uniques per Bobby's suggestion so that it does not grow if
there are a large number of unique values.
one more test to check if compression is being used
Modifications to current() VS next() semantics in the
EWAHNullableColumnIterator. current() no longer modifies state.

Nullable iterators now assume that columns may be sparse and use the null
bitmap to compress storage of columns.

Nullable Iterators can be wrappers to compressed Int Iterators.

Quick performance result summary with sparse data.

Data - 1,000,000 rows of Ints.

1. (with 3000 unique values, no compression)

space taken in memory without null opt - 4,000,080 bytes
space taken in memory with null opt    -   696,520 bytes

2. (with fewer than 240 unique values, compression turned ON)

space taken in memory without null opt - 1,002,008 bytes
space taken in memory with null opt    -   269,874 bytes
Commit the RLE class alone so that Ram can look at and potentially reuse the RLE encoding/decoding.
Included Reynold's style changes.
Working Run Length Encoding for strings. Multiple implementations including the
streaming encoding one chosen are in the source code.

RLE tested for ints too. But not built into the Builder/Factory yet.

The StringColumnBuilder chooses encoding based on heuristics now. It can also be
forced to pick one "scheme" or the other using ENV variables.

Dictionary - attempts were made to make it a complete decorator like RLE or Null
(but this still does not work)

An attempt to use traits-oriented decorators in scala style also did not work
out due to scala reflection troubles.

An LZF implementation is also present now for comparisons. It does not yet
support a block-based or streaming decompression yet which is neccessary to
avoid decompressing the whole column when needed.

--------

After RLE string compressions here are current comparisons

column	#rows	uncompressed 	compressed
u	250,000	8,501,818	466,466
f	250,000	6,814,436	6,814,436
d	250,000	6,001,288	56
a	250,000	10,502,242	74
id	250,000	5,501,182	54

columns d, a and id are partition columns and are the single-value degenerate
case for RLE.

column had too many transitions in values and heuristics chose to not compress
it.
Added support for LZF compression using blocks with fixed number of rows. Code for both iterators left in for testing/comparison.
Get TBLPROPERTY from SharkConfVars working.
Eliminated RLE List[] function versions in favor of IntArrayList
Added ScalaDoc about different encodings.
move scheme() to the base class

Passing TBLPROPERTIES settings to ColumnarSerde and in turn to the StringColumnBuilder and IntColumnBuilder to allow users to control compression schemes.

Resolved logger issues and removed some scaffolding
Incorporated Reynold's style recommendations into code.
Dictionary using Arrays and fastutil now.
Using an enumeration for compression schemes.
ColumnStats fixes to not allocate as many objects. Also redid computation of previous value and
added tests for counting of transitions.
Added a duplicate method for ByteBufferReader.
Use a duplicated ByteBufferReader reference instead of copying to an IntArrayList during RLE Iteration.
Handle the case where compression parameters are not set and also pass on the cachedTableName without prefix
Moved uniques data structure out of IntColumnStats to the IntColumnBuilder. This is to ensure that this potentially large structure does not get cached in memory at the master.