Merge pull request #9 from tejasapatil/master

While skipping rows, BitFieldReader tries to load the next values w/o ch...
Use project version for release tag
[maven-release-plugin] prepare release 0.7
[maven-release-plugin] prepare for next development iteration
Update to Hive 0.13.1
[maven-release-plugin] prepare release 0.8
[maven-release-plugin] prepare for next development iteration
[DWRF] Bug in low memory mode

Summary:
There's a bug in the string and integer tree writers when using the dictionary encoding, which only occurs when the writers enter low memory mode before the first index stride is written out.  If this happens, when the columns are converted from dictionary encoding to direct encoding, it will fail to create an index entry for the first index stride, which causes an IndexOutOfBoundsException later when the data is being read.

The fix just creates the row index entry if it's needed.

Test Plan:
Rewrote some partitions which ran into this issue and verified I could read them.

Added test cases.

Running ptests
[DWRF] Another bug in low memory mode

Summary: If a string column is written out directly and it's writing the stripe corresponding to the end of the hive.exec.orc.encoding.interval and at the same time it switches to low memory mode, it will not enter the if block to convert the dictionary encoded data to direct encoding.  This is because use dictionary encoding is false, but use carried over direct encoding is also false.  Due to this the data in the dictionary is lost.

Test Plan:
Added a unit test.

Ptests pass.
[DWRF] Yet another bug in low memory mode

Summary:
There is a bug in low memory where if low memory mode is entered in a stride (other than the first) and not enough rows are written to the strip to finish off this stride (e.g. if the number of rows in a stride is 10,000 rows, low memory mode is entered after writing 1000 rows to the stride and any number less than 9000 rows are written before the stripe is flushed) then the index entry will not be populated with the positions for that stride.

To fix this I added a check, if dictionary encoding is being used, and the number of saved indeces is > 0 then write out index entries for the number of saved indeces + 1.  This is because if we are in this case, we must be converting from a dictionary to direct encoding.  So write out an index entry for every stride we've finished so far, plus the one we're currently working on.

Test Plan:
Added a testcases to demonstrate these.  Also added a couple test cases for other edge cases I could think of.

ptests pass
[DWRF] Handle Hive weirdness with null keys in maps

Summary: In Hive, with certain formats the map object inspector will say the size of a map is 1 and return an empty map when getMap() is called (e.g. LazyMap).  This breaks DWRF.  To handle this, don't trust getMapSize, instead treat the map returned by getMap as the source of truth.

Test Plan:
Added a testcase which demonstrates the issue.

Running ptests.
[DWRF] Support getRecordWriter with null FileSystem

Summary: RC file and Sequence file both ignore the FileSystem parameter passed to the getRecordWriter method.  To make ORC more compatible with these old formats, check if the FileSystem parameter is null, and if so, use the RC file method of initializing it.

Test Plan:
Added a test case for it.

Ran ptests.
[DWRF] Support key access/size for null maps

Summary: There are a couple of methods in OrcLazyMapObjectInspector that don't handle nulls gracefully.  Added null checks to them and handle nulls in the same way that RC/Sequence file do.

Test Plan:
Added testcase which demonstrates the NPE.

Also added tests for other complex types, they all fully support nulls.
[DWRF] Accessing out of bounds index of list should return null

Summary: This is to be feature compatible with RC/Sequence file.  If you access an index < 0 or >= the size of the list, it should return null, not throw IndexOutOfBounds.

Test Plan:
Added a test case.

Also added a test case for accessing keys of maps that don't exist.

Running ptests.
[DWRF] Materialize in the OrcLazyObject copy constructor

Summary:
We should call materialize in the OrcLazyObject copy constructor to prevent the copy and the original from both having materialize being called on them and both advancing the tree reader, also this prevents the copy from calling materialize out of order.  Added a comment to this effect in the code.

Also added a null check to all the copy constructors for classes which extend OrcLazyObject.  Previously, if materialize wasn't called previous would be null and this would cause an NPE.  Still, if the value of the first row is null, this could still be null, so we should check for it.

Test Plan:
Added unit tests.

Running ptests.
[DWRF] Handle nulls in non-lazy object inspectors

Summary: Made these fixes for the lazy object inspectors, but forgot about the non-lazy ones.

Test Plan:
Added tests.

Running ptests
[maven-release-plugin] prepare release 0.9
[maven-release-plugin] prepare for next development iteration