openspending/openspending — pull request #557 by stefanw (2013-03-07 21:46:00+00:00)
Title: Streaming download of search results

This enables a streaming download of search results if the given pagesize is bigger than the default page size. This should allow for bigger downloads since they are no longer blocking and don't pile up memory.

## Implementation details
This is done by returning an iterator that does the fetching of the search result pages yielding the entries. I refactored a lot of stuff along the way to make parts of the csv, json and browser library more reusable.

CSV implementation is pretty straight forward. The JSON(P) streaming is a bit more complicated due to the fact that it also contains the `stats` and `facets` keys. I have no idea if I have gotten the `facets` implementation right though (someone should check).

Download for JSON(P) is forced with the iterator with the rationale that you don't want to display huge JSON files in your browser.

This should be backwards compatible to the previous API implementation.