codeforamerica/citygram — pull request #120 by invisiblefunnel (2014-10-02 23:26:45+00:00)
Title: Paginate endpoints

This is a first attempt at optional pagination of endpoints.

**Problem**: some datasets get batch dumps of thousands of records, some API's can't return enough records to cover the entire batch. An example if the NYC 311 dataset, which receives a batch of 3-4k records/day, but the Socrata API sets a (totally reasonable) max `limit` of 1000.

**Precedent**: There is a related precedent set in [rfc5988](http://tools.ietf.org/html/rfc5988) and [hypermedia API's like GitHub](https://developer.github.com/guides/traversing-with-pagination/), but it seems more complex than necessary.

**Solution**:
* Citygram poll's the registered publisher endpoint.
* The response includes a `Next-Page` header with a complete url to retrieve the next page.
* Verify that the `Next-Page` value has the same host, but isn't identical to the requested url.
* Queue up a new job to retrieve the next page.

The other thing to balance is complexity between Citygram and the connector services. Anyway, I could definitely use a sanity check on this. Thanks!

cc @chriswhong