dbpedia/distributed-extraction-framework — pull request #28 by nilesh-c (2014-07-08 02:19:04+00:00)
Title: Optimizations + Upgrade to Hadoop 2.2.0, add custom InputFormat and OutputFormat - V2

This PR mainly makes the following changes:

1. Set Hadoop version to 2.2.0 and make certain changes to prevent compile errors and move from deprecated API to newer API.
2. Some changes like **Use broadcast for Language object; wrap mapper using SparkUtils.kryoWrapFunction** and **Move ontology, disambiguation, commonsSource vals to createExtraction...** were made to solve serialization issues.
3. Various small changes like making writing of intermediate `RDD[WikiPage]` to cache optional/configurable, registering all the classes that Kryo needs to serialize to improve performance and reduce serialization footprint etc. were made.
4. Replace `XmlInputFormat` with a custom InputFormat called `DBpediaWikiPageInputFormat` (which uses two classes, `SeekableInputStream` and `ByteMatcher`) that supports compressed input. It also supports **splittable** compression methods like bz2 which is a huge benefit for us. Here's how: say we have this input file, *enwiki-latest-pages-articles-multistream.xml.bz2*. It's > 10GB. It's compressed such that Hadoop's splittable input stream implementations can break it into chunks and decompress them on the fly *parallely*. Parallel decompression and subsequently parallel operations on the data. Refer to discussion on https://github.com/dbpedia/distributed-extraction-framework/pull/27 for details. This is not anything fancy, but something that most InputFormats have to do. Hadoop's own built-in InputFormat implementations do stuff like this too.
5. A custom OutputFormat called `DBpediaCompositeOutputFormat`) is used to write outputs in parallel instead of streaming it over the master iteratively and writing to Hadoop FileSystem - this obviously eliminates a big performance bottleneck. `WikiPageWritable` is a Hadoop `Writable` implementation that wraps a `WikiPage` and handles its serialization/deserialization. Similarly, `QuadSeqWritable` wraps a `Seq[Quad]` - this one is used for writing outputs.

I think we're ready for reviewing and merging to master, and then asking the community to test this out.