lobid/lodmill — pull request #562 by fsteeg (2014-09-18 10:17:05+00:00)
Title: Replace hand-knit indexing with elasticsearch-hadoop library

This pull request replaces our hand-knit indexing logic with the elasticsearch-hadoop library.

It speeds up the indexing process and simplifies the code and execution.

Previously, the conversion job wrote JSON in the Elasticsearch bulk format into HDFS files, and a second process indexed these files using the Elasticsearch bulk API. Now, instead of first writing the data to files, and then indexing it, the data is now directly written to Elasticsearch in the job's reduce step, using the elasticsearch-hadoop library provided by Elasticsearch itself.