elasticsearch/elasticsearch-py — pull request #25 by rboulton (2013-11-20 16:47:19+00:00)
Title: Add a streaming bulk index helper function.

The existing elasticsearch bulk_index helper is limited in two significant ways:

- it doesn't allow the results of the updates to be seen until all the commands have been processed.
- it doesn't allow actions other than `index` to be performed.

This PR adds a new helper function `streaming_bulk_index` which has neither of these restrictions.

It also adds a BulkCommand helper class, which can be used to pass messages to the `streaming_bulk_index` helper, but easily associate external data with them. I've been using code very similar to this in production to read messages from a RabbitMQ queue, convert them to the appropriate elasticsearch command, and the only ack or nack the message after elasticsearch has indexed them; I store the RabbitMQ message ID in the external data field, and thus don't have to guess it from the command contents.