GitHub PushEvent at 2014-09-13 08:38:59
Repository: incubator-parquet-mr
Actor: dsy88
Payload: {"shas":[["0793e49b85d34135fee9f55030997d95f62af97b","a34d47d11bef4e101aa6199cf903b81b91e09c75@apache.org","PARQUET-57 - Update dev README to clarify two points\n\nAuthor: Brock Noland <brock@apache.org>\n\nCloses #26 from brockn/PARQUET-57 and squashes the following commits:\n\nffb2a9a [Brock Noland] Prompt for jira user/pass if not set\n0135cdd [Brock Noland] Fix spelling\n171d714 [Brock Noland] PARQUET-57 - Update dev README to clarify two points","Brock Noland",false],["0148455170be07f89bd6b9230960a6cd510c7ca6","a50d2185142f97721f64187b05c8deeec99604b1@gmail.com","PARQUET-13: The `-d` option for `parquet-schema` shouldn't have optional argument\n\nAuthor: Cheng Lian <lian.cs.zju@gmail.com>\n\nCloses #11 from liancheng/fix-cli-arg and squashes the following commits:\n\n85a5453 [Cheng Lian] Reverted the dummy change\n47ce817 [Cheng Lian] Dummy change to trigger Travis\n1c0a244 [Cheng Lian] The `-d` option for `parquet-schema` shouldn't have optional argument","Cheng Lian",false],["3a396d3a4000bd2575e5314cdea0ba1e2367804c","359617f561f57a021978c1b1cba8233cc23d2950@cloudera.com","PARQUET-59: Fix parquet-scrooge test on hadoop-2.\n\nAuthor: Ryan Blue <rblue@cloudera.com>\n\nCloses #27 from rdblue/PARQUET-59-fix-scrooge-test-on-hadoop-2 and squashes the following commits:\n\nac34369 [Ryan Blue] PARQUET-59: Fix parquet-scrooge test on hadoop-2.","Ryan Blue",false],["b86b01b1e4f20b41a10dc77d7d5326890fc27867","5c682c2d1ec4073e277f9ba9f4bdf07e5794dabe@twitter.com","[maven-release-plugin] prepare release parquet-1.6.0rc1","julien",false],["08a3c6a7dd41f02bb0c876829c01392e6f2d09a2","5c682c2d1ec4073e277f9ba9f4bdf07e5794dabe@twitter.com","[maven-release-plugin] prepare for next development iteration","julien",false],["0d497c4547934485f2aa9e2e9ead46f26fab7bd2","85ad480e390f2e91d9ccec0570186fd33c4a1638@twitter.com","PARQUET-73: Add support for FilterPredicates to cascading schemes\n\nAuthor: Alex Levenson <alexlevenson@twitter.com>\n\nCloses #34 from isnotinvain/alexlevenson/filter-cascading-scheme and squashes the following commits:\n\ncd69a8e [Alex Levenson] Add support for FilterPredicates to cascading schemes","Alex Levenson",false],["7af955a0ac0182deca139e4a15352ce702691dd4","85ad480e390f2e91d9ccec0570186fd33c4a1638@twitter.com","PARQUET-50: Re-Enable the semver enforcer\n\nAuthor: Alex Levenson <alexlevenson@twitter.com>\n\nCloses #37 from isnotinvain/alexlevenson/fix-backwards-compat-check and squashes the following commits:\n\nc91335c [Alex Levenson] Bump version to 1.6.1 to make semver enforcer happy\nc09c8c1 [Alex Levenson] PARQUET-50: Re-Enable the semver enforcer","Alex Levenson",false],["7b415faaed09eba1103ea30577ef1a32fba7048c","ae4ccca20a6abe2c01e64f7ab4a4ae1600072599@netflix.com","Parquet-70: Fixed storing pig schema to udfcontext for non projection case and moved...\n\n... column index access setting to udfcontext so as not to affect other loaders.\n\nI found an problem that affects both the Column name access and column index access due to the way the pig schema is stored by the loader.\n\n##Column Name Access:\nThe ParquetLoader was only storing the pig schema in the UDFContext when push projection is applied.  In the full load case, the schema was not stored which triggered a full reload of the schema during task execution.  You can see in initSchema references the UDFContext for the schema, but that is only set in push projection.  However, the schema needs to be set in both the job context (so the TupleReadSupport can access the schema) and the UDFContext (so the task side loader can access it), which is why it is set in both locations.  This also meant the requested schema was never set to the task side either, which could cause other problems as well.\n\n##Column Index Access:\nFor index based access, the problem was that the column index access setting and the requested schema were not stored in the udfcontext and sent to the task side (unless pushProjection was called).  The schema was stored in the job context, but this would be overwritten if another loader was executed first.  Also, the property to use column index access was only being set at the job context level, so subsequent loaders would use column index access even if they didn't request it.\n\nThis fix now ensures that both the schema and column index access are set in the udfcontext and loaded in the initSchema method.\n\nJIRA: https://issues.apache.org/jira/browse/PARQUET-70\n\n-Dan\n\nAuthor: Daniel Weeks <dweeks@netflix.com>\n\nCloses #36 from dcw-netflix/pig-schema-context and squashes the following commits:\n\nf896a25 [Daniel Weeks] Moved property loading into setInput\n8f3dc28 [Daniel Weeks] Changed to set job conf settings in both front and backend\nd758de0 [Daniel Weeks] Updated to use isFrontend() for setting context properties\nb7ef96a [Daniel Weeks] Fixed storing pig schema to udfcontext for non projection case and moved column index access setting to udfcontext so as not to affect other loaders.","Daniel Weeks",false],["45e581028ab60949b0303d4a0cfe4eee077bcce8","5c682c2d1ec4073e277f9ba9f4bdf07e5794dabe@twitter.com","PARQUET-69: Committer doc\n\nIn order to improve turnaround time on reviewing pull requests here are a few guidelines and informations.\n\nAuthor: Julien Le Dem <julien@twitter.com>\n\nCloses #23 from julienledem/committer_doc and squashes the following commits:\n\nd4b2e19 [Julien Le Dem] Update REVIEWERS.md\nf7d8a5e [Julien Le Dem] Update COMMITTERS.md\nb627d50 [Julien Le Dem] Create REVIEWERS.md\n0027a98 [Julien Le Dem] Create REVIEWERS.md\n6fd49d0 [Julien Le Dem] Create REVIEWERS.md\nc6c26e6 [Julien Le Dem] Create REVIEWERS.md\ncbbec60 [Julien Le Dem] Create REVIEWERS.md\n56b7d3d [Julien Le Dem] Create REVIEWERS.md\na93ddcb [Julien Le Dem] Create REVIEWERS.md\n98f1e52 [Julien Le Dem] Create REVIEWERS.md\n5d960f4 [Julien Le Dem] Update COMMITTERS.md\n7be5f36 [Julien Le Dem] Create REVIEWERS.md\n24ef829 [Julien Le Dem] Create REVIEWERS.md\n799a067 [Julien Le Dem] Create REVIEWERS.md\n8ef1250 [Julien Le Dem] Create REVIEWERS.md\nb85f454 [Julien Le Dem] Create REVIEWERS.md\n38d1afa [Julien Le Dem] Update COMMITTERS.md\n425224d [Julien Le Dem] Update COMMITTERS.md\n46249ed [Julien Le Dem] Update COMMITTERS.md\n7879de2 [Julien Le Dem] Create COMMITTERS.md","Julien Le Dem",false],["54bb983271adcbcc1519ab9e2288871d69093708","359617f561f57a021978c1b1cba8233cc23d2950@cloudera.com","PARQUET-62: Fix binary dictionary write bug.\n\nThe binary dictionary writers keep track of written values in memory to\ndeduplicate and write dictionary pages periodically. If the written\nvalues are changed by the caller, then this corrupts the dictionary\nwithout an error message. This adds a defensive copy to fix the problem.\n\nAuthor: Ryan Blue <rblue@cloudera.com>\n\nCloses #29 from rdblue/PARQUET-62-fix-dictionary-bug and squashes the following commits:\n\n42b6920 [Ryan Blue] PARQUET-62: Fix binary dictionary write bug.","Ryan Blue",false],["792b1490db122e953c9120279ddc86407ffae3c0","5c682c2d1ec4073e277f9ba9f4bdf07e5794dabe@twitter.com","PARQUET-67: mechanism to add extra metadata in the footer\n\nthis expands on the idea proposed by @wesleypeck in https://github.com/Parquet/parquet-mr/pull/185\n\nAuthor: julien <julien@twitter.com>\n\nCloses #32 from julienledem/extensible_metadata and squashes the following commits:\n\n72e0a50 [julien] mechanism to add extra metadata in the footer","julien",false],["84ebe4c82405895706552ccf1becf6647886663f","c454f511ffc0a24a86f1daf704d4e0ec73539cde@gmail.com","PARQUET-66: Upcast blockSize to long to prevent integer overflow.\n\nAuthor: Eric Snyder <snyderep@gmail.com>\n\nCloses #33 from snyderep/master and squashes the following commits:\n\nc99802e [Eric Snyder] PARQUET-66: Upcast blockSize to long to prevent integer overflow.","Eric Snyder",false],["8474f6d8142c7d509fa1a459956c6284c0cb3ff2","5c682c2d1ec4073e277f9ba9f4bdf07e5794dabe@twitter.com","PARQUET-80: upgrade semver plugin version to 0.9.27\n\nTo include the fix in:\nhttps://github.com/jeluard/semantic-versioning/pull/39\n\nAuthor: julien <julien@twitter.com>\n\nCloses #46 from julienledem/upgrade_semver_plugin and squashes the following commits:\n\n30e7247 [julien] upgrade semver plugin version to 0.9.27","julien",false],["d3cd97a8ad7f1c1df48bf42080d993b861158786","ae4ccca20a6abe2c01e64f7ab4a4ae1600072599@netflix.com","PARQUET-75: Fixed string decode performance issue\n\nSwitch to using 'UTF8.decode' as opposed to 'new String'\n\nhttps://issues.apache.org/jira/browse/PARQUET-75\n\nAuthor: Daniel Weeks <dweeks@netflix.com>\n\nCloses #40 from dcw-netflix/string-decode and squashes the following commits:\n\n2cf53e7 [Daniel Weeks] Fixed string decode performance issue","Daniel Weeks",false],["7a105068e60b7dd6e9f28dd0ccdb9b696a9bc941","64a3afcfbe7d15b14a73c87df01a71195a24e765@gmail.com","PARQUET-8: bump scrooge-maven-plugin version\n\nThis makes eclipse:eclipse target work.\n\nAuthor: dvryaboy <dvryaboy@gmail.com>\nAuthor: Dmitriy Ryaboy <dmitriy@twitter.com>\n\nCloses #41 from dvryaboy/PARQUET-8 and squashes the following commits:\n\nf089bac [dvryaboy] trivial change to retrigger travis build.\nd06a98e [Dmitriy Ryaboy] bump scrooge-maven-plugin version","dvryaboy",false],["f8b06df7a56f92f4bc7dd564ad7ec026e3b4f3da","f8a0b68fd75e68067624c0acb343bfb9a5ed4026@twitter.com","do ProtocolEvents fixing only when there is required fields missing in the requested schema\n\nhttps://issues.apache.org/jira/browse/PARQUET-61\nThis PR is trying to redo the https://github.com/apache/incubator-parquet-mr/pull/7\n\nIn this PR, it fixes the protocol event in a more precise condition:\nOnly when the requested schema missing some required fields that are present in the full schema\n\nSo even if there a projection, as long as the projection is not getting rid of the required field, the protocol events amender will not be called.\n\nCould you take a look at this ? @dvryaboy @yan-qi\n\nAuthor: Tianshuo Deng <tdeng@twitter.com>\n\nCloses #28 from tsdeng/fix_protocol_when_required_field_missing and squashes the following commits:\n\nba778b9 [Tianshuo Deng] add continue for readability\nd5639df [Tianshuo Deng] fix unused import\n090e894 [Tianshuo Deng] format\n13a609d [Tianshuo Deng] comment format\nef1fe58 [Tianshuo Deng] little refactor, remove the hasMissingRequiredFieldFromProjection method\n7c2c158 [Tianshuo Deng] format\n83a5655 [Tianshuo Deng] do ProtocolEvents fixing only when there is required fields missing in the requested schema","Tianshuo Deng",false],["647b8a70f9b7c94cabf9a7ec7bce2e7cbbb4c05b","359617f561f57a021978c1b1cba8233cc23d2950@cloudera.com","PARQUET-63: Enable dictionary encoding for FIXED.\n\nThis uses the existing dictionary support introduced for int96. Encoding\nand ParquetProperties have been updated to use the dictionary supporting\nclasses, when requested for write or present during read. This also\nfixes a bug in the fixed dictionary values writer, where the length was\nhard-coded for int96, 12 bytes.\n\nAuthor: Ryan Blue <rblue@cloudera.com>\n\nCloses #30 from rdblue/PARQUET-63-add-fixed-dictionary-support and squashes the following commits:\n\nbc34a34 [Ryan Blue] PARQUET-63: Enable dictionary encoding for FIXED.","Ryan Blue",false],["5dafd127f3de7c516ce9c1b7329087a01ab2fc57","5c682c2d1ec4073e277f9ba9f4bdf07e5794dabe@twitter.com","PARQUET-84: Avoid reading rowgroup metadata in memory on the client side.\n\nThis will improve reading big datasets with a large schema (thousands of columns)\nInstead rowgroup metadata can be read in the tasks where each tasks reads only the metadata of the file it's reading\n\nAuthor: julien <julien@twitter.com>\n\nCloses #45 from julienledem/skip_reading_row_groups and squashes the following commits:\n\nccdd08c [julien] fix parquet-hive\n24a2050 [julien] Merge branch 'master' into skip_reading_row_groups\n3d7e35a [julien] adress review feedback\n5b6bd1b [julien] more tests\n323d254 [julien] sdd unit tests\nf599259 [julien] review feedback\nfb11f02 [julien] fix backward compatibility check\n2c20b46 [julien] cleanup readFooters methods\n3da37d8 [julien] fix read summary\nab95a45 [julien] cleanup\n4d16df3 [julien] implement task side metadata\n9bb8059 [julien] first stab at integrating skipping row groups","julien",false],["5f39948b2414ea2582892f6447566d7fe4909b4f","f8a0b68fd75e68067624c0acb343bfb9a5ed4026@twitter.com","update scala 2.10\n\nTry to upgrade to scala 2.10\n\nAuthor: Tianshuo Deng <tdeng@twitter.com>\n\nCloses #35 from tsdeng/update_scala_2_10 and squashes the following commits:\n\n1b7e55f [Tianshuo Deng] fix comment\nbed9de3 [Tianshuo Deng] remove twitter artifactory\n2bce643 [Tianshuo Deng] publish fix\n06b374e [Tianshuo Deng] define scala.binary.version\nfcf6965 [Tianshuo Deng] Merge branch 'master' into update_scala_2_10\ne91d9f7 [Tianshuo Deng] update version\n5d18b88 [Tianshuo Deng] version\n83df898 [Tianshuo Deng] update scala 2.10","Tianshuo Deng",false],["24119cc8e41fa07f6ec0a1a33428419f5c99a829","5c682c2d1ec4073e277f9ba9f4bdf07e5794dabe@twitter.com","upgrade scalatest_version to depend on scala 2.10.4\n\nAuthor: julien <julien@twitter.com>\n\nCloses #52 from julienledem/scalatest_version and squashes the following commits:\n\n945fa75 [julien] upgrade scalatest_version to depend on scala 2.10.4","julien",false]],"size":27,"ref":"refs/heads/ByteBufferRead","head":"00b7c7cdc24f035af10286eda019ded1976cd4d3"}