PARQUET-57 - Update dev README to clarify two points

Author: Brock Noland <brock@apache.org>

Closes #26 from brockn/PARQUET-57 and squashes the following commits:

ffb2a9a [Brock Noland] Prompt for jira user/pass if not set
0135cdd [Brock Noland] Fix spelling
171d714 [Brock Noland] PARQUET-57 - Update dev README to clarify two points
PARQUET-13: The `-d` option for `parquet-schema` shouldn't have optional argument

Author: Cheng Lian <lian.cs.zju@gmail.com>

Closes #11 from liancheng/fix-cli-arg and squashes the following commits:

85a5453 [Cheng Lian] Reverted the dummy change
47ce817 [Cheng Lian] Dummy change to trigger Travis
1c0a244 [Cheng Lian] The `-d` option for `parquet-schema` shouldn't have optional argument
PARQUET-59: Fix parquet-scrooge test on hadoop-2.

Author: Ryan Blue <rblue@cloudera.com>

Closes #27 from rdblue/PARQUET-59-fix-scrooge-test-on-hadoop-2 and squashes the following commits:

ac34369 [Ryan Blue] PARQUET-59: Fix parquet-scrooge test on hadoop-2.
[maven-release-plugin] prepare release parquet-1.6.0rc1
[maven-release-plugin] prepare for next development iteration
PARQUET-73: Add support for FilterPredicates to cascading schemes

Author: Alex Levenson <alexlevenson@twitter.com>

Closes #34 from isnotinvain/alexlevenson/filter-cascading-scheme and squashes the following commits:

cd69a8e [Alex Levenson] Add support for FilterPredicates to cascading schemes
PARQUET-50: Re-Enable the semver enforcer

Author: Alex Levenson <alexlevenson@twitter.com>

Closes #37 from isnotinvain/alexlevenson/fix-backwards-compat-check and squashes the following commits:

c91335c [Alex Levenson] Bump version to 1.6.1 to make semver enforcer happy
c09c8c1 [Alex Levenson] PARQUET-50: Re-Enable the semver enforcer
Parquet-70: Fixed storing pig schema to udfcontext for non projection case and moved...

... column index access setting to udfcontext so as not to affect other loaders.

I found an problem that affects both the Column name access and column index access due to the way the pig schema is stored by the loader.

##Column Name Access:
The ParquetLoader was only storing the pig schema in the UDFContext when push projection is applied.  In the full load case, the schema was not stored which triggered a full reload of the schema during task execution.  You can see in initSchema references the UDFContext for the schema, but that is only set in push projection.  However, the schema needs to be set in both the job context (so the TupleReadSupport can access the schema) and the UDFContext (so the task side loader can access it), which is why it is set in both locations.  This also meant the requested schema was never set to the task side either, which could cause other problems as well.

##Column Index Access:
For index based access, the problem was that the column index access setting and the requested schema were not stored in the udfcontext and sent to the task side (unless pushProjection was called).  The schema was stored in the job context, but this would be overwritten if another loader was executed first.  Also, the property to use column index access was only being set at the job context level, so subsequent loaders would use column index access even if they didn't request it.

This fix now ensures that both the schema and column index access are set in the udfcontext and loaded in the initSchema method.

JIRA: https://issues.apache.org/jira/browse/PARQUET-70

-Dan

Author: Daniel Weeks <dweeks@netflix.com>

Closes #36 from dcw-netflix/pig-schema-context and squashes the following commits:

f896a25 [Daniel Weeks] Moved property loading into setInput
8f3dc28 [Daniel Weeks] Changed to set job conf settings in both front and backend
d758de0 [Daniel Weeks] Updated to use isFrontend() for setting context properties
b7ef96a [Daniel Weeks] Fixed storing pig schema to udfcontext for non projection case and moved column index access setting to udfcontext so as not to affect other loaders.
PARQUET-69: Committer doc

In order to improve turnaround time on reviewing pull requests here are a few guidelines and informations.

Author: Julien Le Dem <julien@twitter.com>

Closes #23 from julienledem/committer_doc and squashes the following commits:

d4b2e19 [Julien Le Dem] Update REVIEWERS.md
f7d8a5e [Julien Le Dem] Update COMMITTERS.md
b627d50 [Julien Le Dem] Create REVIEWERS.md
0027a98 [Julien Le Dem] Create REVIEWERS.md
6fd49d0 [Julien Le Dem] Create REVIEWERS.md
c6c26e6 [Julien Le Dem] Create REVIEWERS.md
cbbec60 [Julien Le Dem] Create REVIEWERS.md
56b7d3d [Julien Le Dem] Create REVIEWERS.md
a93ddcb [Julien Le Dem] Create REVIEWERS.md
98f1e52 [Julien Le Dem] Create REVIEWERS.md
5d960f4 [Julien Le Dem] Update COMMITTERS.md
7be5f36 [Julien Le Dem] Create REVIEWERS.md
24ef829 [Julien Le Dem] Create REVIEWERS.md
799a067 [Julien Le Dem] Create REVIEWERS.md
8ef1250 [Julien Le Dem] Create REVIEWERS.md
b85f454 [Julien Le Dem] Create REVIEWERS.md
38d1afa [Julien Le Dem] Update COMMITTERS.md
425224d [Julien Le Dem] Update COMMITTERS.md
46249ed [Julien Le Dem] Update COMMITTERS.md
7879de2 [Julien Le Dem] Create COMMITTERS.md
PARQUET-62: Fix binary dictionary write bug.

The binary dictionary writers keep track of written values in memory to
deduplicate and write dictionary pages periodically. If the written
values are changed by the caller, then this corrupts the dictionary
without an error message. This adds a defensive copy to fix the problem.

Author: Ryan Blue <rblue@cloudera.com>

Closes #29 from rdblue/PARQUET-62-fix-dictionary-bug and squashes the following commits:

42b6920 [Ryan Blue] PARQUET-62: Fix binary dictionary write bug.
PARQUET-67: mechanism to add extra metadata in the footer

this expands on the idea proposed by @wesleypeck in https://github.com/Parquet/parquet-mr/pull/185

Author: julien <julien@twitter.com>

Closes #32 from julienledem/extensible_metadata and squashes the following commits:

72e0a50 [julien] mechanism to add extra metadata in the footer
PARQUET-66: Upcast blockSize to long to prevent integer overflow.

Author: Eric Snyder <snyderep@gmail.com>

Closes #33 from snyderep/master and squashes the following commits:

c99802e [Eric Snyder] PARQUET-66: Upcast blockSize to long to prevent integer overflow.
PARQUET-80: upgrade semver plugin version to 0.9.27

To include the fix in:
https://github.com/jeluard/semantic-versioning/pull/39

Author: julien <julien@twitter.com>

Closes #46 from julienledem/upgrade_semver_plugin and squashes the following commits:

30e7247 [julien] upgrade semver plugin version to 0.9.27
PARQUET-75: Fixed string decode performance issue

Switch to using 'UTF8.decode' as opposed to 'new String'

https://issues.apache.org/jira/browse/PARQUET-75

Author: Daniel Weeks <dweeks@netflix.com>

Closes #40 from dcw-netflix/string-decode and squashes the following commits:

2cf53e7 [Daniel Weeks] Fixed string decode performance issue
PARQUET-8: bump scrooge-maven-plugin version

This makes eclipse:eclipse target work.

Author: dvryaboy <dvryaboy@gmail.com>
Author: Dmitriy Ryaboy <dmitriy@twitter.com>

Closes #41 from dvryaboy/PARQUET-8 and squashes the following commits:

f089bac [dvryaboy] trivial change to retrigger travis build.
d06a98e [Dmitriy Ryaboy] bump scrooge-maven-plugin version
do ProtocolEvents fixing only when there is required fields missing in the requested schema

https://issues.apache.org/jira/browse/PARQUET-61
This PR is trying to redo the https://github.com/apache/incubator-parquet-mr/pull/7

In this PR, it fixes the protocol event in a more precise condition:
Only when the requested schema missing some required fields that are present in the full schema

So even if there a projection, as long as the projection is not getting rid of the required field, the protocol events amender will not be called.

Could you take a look at this ? @dvryaboy @yan-qi

Author: Tianshuo Deng <tdeng@twitter.com>

Closes #28 from tsdeng/fix_protocol_when_required_field_missing and squashes the following commits:

ba778b9 [Tianshuo Deng] add continue for readability
d5639df [Tianshuo Deng] fix unused import
090e894 [Tianshuo Deng] format
13a609d [Tianshuo Deng] comment format
ef1fe58 [Tianshuo Deng] little refactor, remove the hasMissingRequiredFieldFromProjection method
7c2c158 [Tianshuo Deng] format
83a5655 [Tianshuo Deng] do ProtocolEvents fixing only when there is required fields missing in the requested schema
PARQUET-63: Enable dictionary encoding for FIXED.

This uses the existing dictionary support introduced for int96. Encoding
and ParquetProperties have been updated to use the dictionary supporting
classes, when requested for write or present during read. This also
fixes a bug in the fixed dictionary values writer, where the length was
hard-coded for int96, 12 bytes.

Author: Ryan Blue <rblue@cloudera.com>

Closes #30 from rdblue/PARQUET-63-add-fixed-dictionary-support and squashes the following commits:

bc34a34 [Ryan Blue] PARQUET-63: Enable dictionary encoding for FIXED.
PARQUET-84: Avoid reading rowgroup metadata in memory on the client side.

This will improve reading big datasets with a large schema (thousands of columns)
Instead rowgroup metadata can be read in the tasks where each tasks reads only the metadata of the file it's reading

Author: julien <julien@twitter.com>

Closes #45 from julienledem/skip_reading_row_groups and squashes the following commits:

ccdd08c [julien] fix parquet-hive
24a2050 [julien] Merge branch 'master' into skip_reading_row_groups
3d7e35a [julien] adress review feedback
5b6bd1b [julien] more tests
323d254 [julien] sdd unit tests
f599259 [julien] review feedback
fb11f02 [julien] fix backward compatibility check
2c20b46 [julien] cleanup readFooters methods
3da37d8 [julien] fix read summary
ab95a45 [julien] cleanup
4d16df3 [julien] implement task side metadata
9bb8059 [julien] first stab at integrating skipping row groups
update scala 2.10

Try to upgrade to scala 2.10

Author: Tianshuo Deng <tdeng@twitter.com>

Closes #35 from tsdeng/update_scala_2_10 and squashes the following commits:

1b7e55f [Tianshuo Deng] fix comment
bed9de3 [Tianshuo Deng] remove twitter artifactory
2bce643 [Tianshuo Deng] publish fix
06b374e [Tianshuo Deng] define scala.binary.version
fcf6965 [Tianshuo Deng] Merge branch 'master' into update_scala_2_10
e91d9f7 [Tianshuo Deng] update version
5d18b88 [Tianshuo Deng] version
83df898 [Tianshuo Deng] update scala 2.10
upgrade scalatest_version to depend on scala 2.10.4

Author: julien <julien@twitter.com>

Closes #52 from julienledem/scalatest_version and squashes the following commits:

945fa75 [julien] upgrade scalatest_version to depend on scala 2.10.4