Netflix/aegisthus — pull request #26 by danielbwatson (2014-08-25 19:46:41+00:00)
Title: Support columnar input.

Squashed commit of the following:

commit 9fa588e7dd853f804686b503dd418d8556f15c18
Author: Daniel Watson
Date: Mon Aug 25 12:33:36 2014 -0700

Bump version to 0.2.0.

commit dca36d797970feceeec726b7959b38d5ea5cf6d2
Author: Daniel Watson
Date: Mon Aug 25 12:33:03 2014 -0700

Minor cleanup

commit 9854299a753e5cca8313c91da2cea2aaffed162b
Merge: d288134 1e85e42
Author: Daniel Watson
Date: Mon Aug 25 12:27:55 2014 -0700

Merge remote-tracking branch 'upstream/master' into column_based

Conflicts:
build.gradle
gradle.properties
settings.gradle

commit d288134ce09b949ed64fadb0681dad1aef9e3734
Author: Daniel Watson
Date: Mon Aug 25 11:48:42 2014 -0700

Use CleanOutputFormat.

commit 2c13322453312cf4d0306676e56769ffe72f4524
Author: Daniel Watson
Date: Mon Aug 18 14:59:25 2014 -0700

Add build configuration for distcp

commit 5945fbd838e1efb9c7796f7d0adf6d4f8489ba41
Author: Daniel Watson
Date: Mon Aug 18 14:59:10 2014 -0700

Use the hadoop library selection from master.

commit c276d308a5f7c9dcdbc6de106fbe95f89a71dd57
Author: Daniel Watson
Date: Mon Aug 18 14:53:43 2014 -0700

Explicitly specify the Java version.

commit a40004b37a5f5ad169101b4a16f70ed5474677d5
Author: Daniel Watson
Date: Wed Aug 13 16:51:40 2014 -0700

Copy over distcp from master.

commit b501ebecda58d5d1c219778de34179ce7abf7402
Author: Daniel Watson
Date: Wed Aug 13 13:54:56 2014 -0700

Allow override of AegisthusInputFormat.listStatus

- Allow users of this file to override if desired.

commit 03962a26e3ce7a5658dca38d1920f0a9c6bc9f01
Author: Daniel Watson
Date: Tue Aug 12 14:28:37 2014 -0700

Upgrade to Gradle 1.12.

commit a196f5836d737516e0f298f8388013d13dc17acc
Author: Daniel Watson
Date: Mon Aug 11 12:44:41 2014 -0700

Use the correct input format in the pig loader.

commit 656e511667d39e0f8a77d11278a80cf040381ca4
Author: Daniel Watson
Date: Mon Aug 11 12:44:02 2014 -0700

Fixed output of incorrect data size.

commit 0f4415c5ad59871f1467824fdfc4c6f42a554981
Author: Daniel Watson
Date: Mon Aug 11 12:42:33 2014 -0700

Changed log level to reduce output spam.

- Also switched log statements to use SLF4J style output format instead
of string concatenation.

commit e585bdfdd36a1b77a9736ffb876c26a76f4dc831
Author: Daniel Watson
Date: Fri Aug 8 11:45:05 2014 -0700

Minor code cleanup.

commit e3d030fc801fa751e43415e05227de97fcf7780b
Author: Daniel Watson
Date: Fri Aug 8 11:09:40 2014 -0700

Fixed breakage caused by changes to OffsetScanner.

commit 7ea301e8f454def228e4dbaef533d5e85d94d233
Author: Daniel Watson
Date: Fri Aug 8 11:09:03 2014 -0700

Rename CommitLogScanner and restore the version from master.

commit 3fcf4a03191864b95a576c3d3ee968f4909bfefa
Author: Daniel Watson
Date: Fri Aug 8 10:27:05 2014 -0700

Added the pig column loader

commit 305ce7a45739296a6d23779eeac0d28abefcb529
Author: Daniel Watson
Date: Thu Aug 7 18:06:56 2014 -0700

Restore old input files from master.

commit 364e2097150e1e8acfd01cef50a96c128299ee4e
Author: Daniel Watson
Date: Thu Aug 7 17:58:05 2014 -0700

Move the columnar input classes to a new package

- This will let us run the old and new code in parallel.

commit cd72252ec84b07ed66d2e08a01fad6bb98a7bee9
Author: Daniel Watson
Date: Thu Aug 7 17:36:07 2014 -0700

Fix negative partition numbers.

commit a498d602a2e97360f42c309a9f4f0ac6b803d040
Author: Daniel Watson
Date: Thu Aug 7 16:47:49 2014 -0700

Add back CassReducer from origin/master.

commit f1dbee0f4d5032e22a4a24ba33a592959b993730
Author: Daniel Watson
Date: Thu Aug 7 16:45:07 2014 -0700

Renamed for consistency with other files.

commit 9ee9a7f10e5f2922d5770751722885b46527bcd5
Author: Daniel Watson
Date: Thu Aug 7 16:38:31 2014 -0700

Make reducer class configurable.

commit 64f4c9f5a8af4ab7fdc571490391b34ed5a8afc6
Author: Daniel Watson
Date: Thu Aug 7 16:37:29 2014 -0700

Inline retrieval of inputFormatClass

commit 17351142b6241c37893513c7a15d4da49e2b90a7
Author: Daniel Watson
Date: Thu Aug 7 16:24:49 2014 -0700

Move column type into Aegisthus.

- This is in preparation for making the reducer configurable.

commit 7e3da93e78f9f32c16257c44f68555ba96e6c7bc
Author: Daniel Watson
Date: Thu Aug 7 16:23:08 2014 -0700

Remove unused field.

commit 447b4cd063f263a444477c7cce8b8e3b154841ed
Author: Daniel Watson
Date: Thu Aug 7 16:12:49 2014 -0700

Store a reference to the ExecutorService.

commit 7b4fb0a78e394aa32a7387fd6b3410cc3c89407a
Author: Daniel Watson
Date: Thu Aug 7 16:11:26 2014 -0700

Renamed new sstable reducer.

- This way we can still support the old reducer.

commit 8d4f2d9b8f0e6f8d07506e1a82cb93e314e0ee82
Author: Daniel Watson
Date: Thu Aug 7 16:09:03 2014 -0700

Removed used import.

commit 9c243c73001c5fb8f38812ca10bd0f0ec4abddef
Author: Daniel Watson
Date: Thu Aug 7 16:08:41 2014 -0700

Renamed constant.

commit 2e601e37fcfef37f8c6f2d265ba277bb352bdb1e
Author: Daniel Watson
Date: Thu Aug 7 15:04:38 2014 -0700

Add support for specifying input format from CLI

commit d6016b03c74b20bd00be7830c6b0a32a62b1db20
Author: Daniel Watson
Date: Thu Aug 7 15:03:12 2014 -0700

Add missing import.

commit eeb59b01d25e586f30aacc0b5a593c5f79266865
Author: Daniel Watson
Date: Wed Aug 6 16:55:29 2014 -0700

Include timestamp in the sort.

commit 02ba8a9beea420d52d7674d8976765f64079d61a
Author: Daniel Watson
Date: Wed Aug 6 16:53:25 2014 -0700

Added debug message when skipping ranged tombstone

commit 1c7d9833aeb4d2926476cfc2389e927c88971179
Author: Daniel Watson
Date: Wed Aug 6 16:51:45 2014 -0700

Fix deprecation warnings.

commit c8fcadcee79a9e3bf1a727f3967060c186b450e0
Author: Daniel Watson
Date: Wed Aug 6 16:50:35 2014 -0700

Fix hard coded MR path.

commit 48842247b7367187066e7e251ea923b21ee6311e
Author: Daniel Watson
Date: Wed Aug 6 16:48:27 2014 -0700

Disable a failing test.

- Disabling the test instead of removing it until I find out why agent.json (that the test depends on) was removed.

commit d8a3073084e11d79d241f05e91fd5c8b7ad4b855
Author: Daniel Watson
Date: Wed Aug 6 16:46:05 2014 -0700

Include thrift in the jar.

- Cassandra loads thrift at runtime when we use BytesType.compare

commit 1874fadcfc0f721b66beb41d956991ad93afacf3
Author: Daniel Watson
Date: Wed Aug 6 16:44:49 2014 -0700

Update to the latest stable version of RxJava

commit f8620f7aa8d4e07f72fc9b6d8b981c3ff6a99784
Author: Charles Smith
Date: Sat Jul 26 21:56:05 2014 -0700

missing file

commit 81693c8b162cba5d15f5e185b8456c2d3e067a6d
Author: Charles Smith
Date: Mon Jul 14 02:09:28 2014 -0700

fixed the sorting. Still doesn't seem right

commit 037a32c1041225d185a69cb232f36d095c51933c
Author: Charles Smith
Date: Sun Jul 13 23:54:30 2014 -0700

remove protobuf

commit 90f5df328808ea853e634d91404ba5707f1bedb0
Author: Charles Smith
Date: Sun Jul 13 23:43:30 2014 -0700

A complete rewrite of serialization to Cassandra's OnDiskAtom

Basically what happened is that now we always read from->to sstable.
So in the end the output of the reduce can be a compacted table.

This doesn't take into account the ordering of the sstables as far as
Cassandra is concerned, so the sstables wouldn't dump back into the db
without some issues, but there is no reason we couldn't change this.

Still todo:

- 1. finish ranged tombstone reduce logic
This may be done. Haven't tested it yet, but we are now using Cassandra's code.
- 2. handle json input
We will deprecate this. We could still handle it, but there is no reason to do it
if we output sstables.
3. build a testing framework for this as the requirements are difficult now
Not done, need to build testing for at least the reducer to make sure the new code works.
4. finish the type handling into json
This now works, but I now need to allow the job to output json optionally and then extend
The reader to handle reading sstables, which is what we will want to do internally.
- 5. Sort input into reduce by type.
This is now done. Have to test it, but it appears to work.

To roll this out in NFLX, besides testing, #4 needs to be done. Then the python code will have to discount
the current json snapshot.

commit 6bd409cec2b02d985e65060d7c14dcc774dcc631
Author: Charles Smith
Date: Sat Jul 12 23:30:23 2014 -0700

Further work on column based processing.

Still todo:

1. finish ranged tombstone reduce logic
2. handle json input
3. build a testing framework for this as the requirements are difficult
now
4. finish the type handling into json
5. Sort input into reduce by type.

I think that is it. Finally need to do some performance testing on this
as it seems much slower than the old row code.

commit da9f3b3b69a09d2fbc11c02cc4fb760c94f6931a
Author: Charles Smith
Date: Wed May 28 18:29:02 2014 -0700

working without sorting and column types

commit e3056d51520cf3f749b0b17a0e34df38a06aa1c5
Author: Charles Smith
Date: Wed Apr 30 17:38:55 2014 -0700

major refactor

commit 3eba71ed988e4ea97e84ba8776d64d65480b91cf
Author: Charles Smith
Date: Fri Apr 25 17:12:39 2014 -0700

initial commit for handling bad sstables

I am not at all happy about this change. The basic idea is to work
around bad rows by keeping track of where we are in the index files.
The problem is basically that this mechanism is far reaching and hard to
test. Possibly before this can be fully committed we need to bite the
bullet and create unit tests for the Hadoop features of the code
(inputformats, splits, recordreaders).