taskrabbit/elasticsearch-dump — pull request #28 by evantahler (2014-06-13 21:23:05+00:00)
Title: respect `--limit` by dividing by shard count

Now that we are using the scan/scroll API to load data from Elasticsearch, we need to modify how the flag `--limit` is treated in reads.

In most Elasticsearch APIs, limit is literal, in that if you say `{size: 100}`, you will get 100 results. However, the scan/scroll API is special, in that it tries to minimize load on each shard and does not pre-collect results before transmitting. The `size` in this API is actually results per-shard. So if you have 5 shards and say `{size: 100}`, you will actually get ~500 results back (assuming the shard has unsent data to return).

This PR attempts to look up how many shards and index has, and will modify the effective `{size}` to be `limit / shards`.