alphagov/rummager — pull request #161 by heathd (2013-09-02 14:41:36+00:00)
Title: Fast bulk loading

Support bulk loading by reading a stream of indexing commands and data via standard input.

Indexing operations are split into packets of about 1MB in size and sent to elastic search directly.

The indexing process supports a seamless re-indexing by loading new data in a disconnected index before switching over.

Loading a complete government index takes approx 25 seconds on my local machine.