dbpedia/distributed-extraction-framework — pull request #27 by nilesh-c (2014-07-02 23:31:17+00:00)
Title: Upgrade to Hadoop 2.2.0 and add custom InputFormat

NOTE: This is still a work in progress. Please do not run/test this as of yet. Wait for my signal. :)

I have set the Hadoop version to 2.2.0 because it has a lot of bug fixes that we need (regarding splittable compression streams) - any version 2.2.0 onwards will be okay.

This PR mainly makes the following changes:

1. Set Hadoop version to 2.2.0 and make certain changes to prevent compile errors (Hadoop API changes?) and changed use of deprecated API to newer API.
2. A custom InputFormat called `DBpediaWikiPageInputFormat` is added (which uses two classes, `SeekableInputStream` and `ByteMatcher`) that supports compressed input. It also supports **splittable** compression methods like bz2 which is a huge benefit for us. Here's how: say we have this input file, *enwiki-latest-pages-articles-multistream.xml.bz2*. It's > 10GB. It's compressed such that Hadoop's splittable input stream implementations can break it into chunks and decompress them on the fly *parallely*. Parallel decompression and subsequently parallel operations on the data.

`WikiPageWritable` is a Hadoop `Writable` implementation that wraps a `WikiPage` and handles its serialization/deserialization.

I may update this PR after I test this out on 5-6 languages in parallel today.

Feature-wise we're left with distributed outputs using a custom OutputFormat implementation which should pretty much wrap up **nildev2**. I shall make another PR for that in a day.