Package: paginate-json
paginate-jsonCLI tool for retrieving JSON from paginated APIs.This tool works against APIs that use the HTTP Link header for pagination. The GitHub API isone example of this.Recipes using this tool:Combined release notes from GitHub with jq and paginate-jsonExport a Mastodon timeline to SQLiteInstallationpipinstallpaginate-jsonOr usepipx:pipxinstallpaginate-jsonUsageRun this tool against a URL that returns a JSON list of items and uses thelink:HTTP header to indicate the URL of the next page of results.It will output a single JSON list containing all of the records, across multiple pages.paginate-json\https://api.github.com/users/simonw/eventsYou can use the--headeroption to send additional request headers. For example, if you have a GitHub OAuth token you can pass it like this:paginate-json\https://api.github.com/users/simonw/events\--headerAuthorization"bearer e94d9e404d86..."Some APIs may return a root level object where the items you wish to gather are stored in a key, like this example from theDatasette JSON API:{"ok":true,"rows":[{"id":1,"name":"San Francisco"},{"id":2,"name":"Los Angeles"},{"id":3,"name":"Detroit"},{"id":4,"name":"Memnonia"}]}In this case, use--key rowsto specify which key to extract the items from:paginate-json\https://latest.datasette.io/fixtures/facet_cities.json\--keyrowsThe output JSON will be streamed as a pretty-printed JSON array by default.To switch to newline-delimited JSON, with a separate object on each line, add--nl:paginate-json\https://latest.datasette.io/fixtures/facet_cities.json\--keyrows\--nlThe output from that command looks like this:{"id": 1, "name": "San Francisco"}
{"id": 2, "name": "Los Angeles"}
{"id": 3, "name": "Detroit"}
{"id": 4, "name": "Memnonia"}Using this with sqlite-utilsThis tool works well in conjunction withsqlite-utils. For example, here's how to load all of the GitHub issues for a project into a local SQLite database.paginate-json\"https://api.github.com/repos/simonw/datasette/issues?state=all&filter=all"\--nl|\sqlite-utilsupsert/tmp/issues.dbissues---nl--pk=idYou can then useother features of sqlite-utilsto enhance the resulting database. For example, to enable full-text search on the issue title and body columns:sqlite-utilsenable-fts/tmp/issues.dbissuestitlebodyUsing jq to transform each pageIf you install the optionaljqorpyjqdependency you can also pass--jq PROGRAMto transform the results of each page using ajq program. Thejqoption you supply should transform each page of fetched results into an array of objects.For example, to extract theidandtitlefrom each issue:paginate-json\"https://api.github.com/repos/simonw/datasette/issues"\--nl\--jq'map({id, title})'paginate-json --helpUsage: paginate-json [OPTIONS] URL

  Fetch paginated JSON from a URL

  Example usage:

      paginate-json https://api.github.com/repos/simonw/datasette/issues

Options:
  --version                Show the version and exit.
  --nl                     Output newline-delimited JSON
  --key TEXT               Top-level key to extract from each page
  --jq TEXT                jq transformation to run on each page
  --accept TEXT            Accept header to send
  --sleep INTEGER          Seconds to delay between requests
  --silent                 Don't show progress on stderr - default
  -v, --verbose            Show progress on stderr
  --show-headers           Dump response headers out to stderr
  --ignore-http-errors     Keep going on non-200 HTTP status codes
  --header <TEXT TEXT>...  Send custom request headers
  --help                   Show this message and exit.