alphagov/rummager — pull request #69 by fatbusinessman (2013-03-13 12:01:07+00:00)
Title: Evaluate `all_documents` lazily

Instead of loading all the documents into memory at once, we can load them up in batches as they are needed. This should reduce memory use, but more importantly avoids the problem of timeouts as elasticsearch tries to form a response with several thousand documents in it.

**Note**: pulling this into the `index_aliases` branch instead of `master`, because this work builds on the index aliasing and is required to make it work with the `government` index, but I don’t want it to get lost in the rest of the pull request.