taskrabbit/elasticsearch-dump — pull request #14 by pderaaij (2014-05-16 06:50:56+00:00)
Title: Added transport method for elasticsearch to use scan and scroll method

When using this tool on our index we've noticed many duplicate documents in our export. For a 114.000 document index we had about 37.000 duplicate documents. We tried to fix this by using a limit of 115.000, but that uses way too much memory of course.

So, we started to look at the code and we found it is using the query method of the search API. The problem with this method is that it will not holds the order of the result set for a period of time. The search API provides a scan and scroll method for this purpose.

This pull requests contains a new transporter which explicitly uses this scan and scroll method of Elasticsearch. To not break current usages I've created a new transporter and it has to be enabled by using some new command flags.

In our case this is solving our issues and creates a correct dump holding all documents without any duplicates.