lobid/lodmill — pull request #587 by dr0i (2014-10-10 15:27:14+00:00)
Title: Issue 43 hbz transform new base dump

See hbz/lobid#43 (and https://github.com/hbz/mabxml-elasticsearch , #465 , #538, lobid/lodmill#478)

We store a daily snapshot of the MabXml in elasticsearch, based on
the new full dump of 2014-08. This pull request allows to get the data
from elasticsearch index (`hbz01`) and feed this data (the raw mab xml)
into our transformation and indexing workflow. Documents harvested around 600
docs/s. Thus whole transformation takes around 10h which is twice as fast as
using parallized filesystem snapshot (8 instances).