Refactor pagination handling to support more pagination options
Add support for two new pagination options:

  * all_ids=true -- doesn't return records: just a JSON array of their IDs

  * id_set=1,2,3,4,5 -- returns the requests set of records (selected
    by ID)
Rework the periodic indexer to use the new 'id_set' pagination option.

This should make indexing faster for repositories with lots of
records.  The LIMIT/OFFSET approach used by the standard pagination
gets progressively slower as you seek towards the end of a dataset
(which indexing always does).  Instead, we pull back the complete list
of record IDs initially, then request subsets of IDs.
Be a little more selective with regard to which JSON exceptions we catch
Move temp file handling into the batch_imports controller so that retrying on failed transaction works properly.
Don't send a vocab changed notification unless the creation actually succeeds
Make sure the terms sha1 is always set
Make sure ensure_exists always fires (even under the raciest of conditions)
Transaction wrangling
Use a more efficient query for handling id_set queries
Merge pull request #430 from hudmol/loadtest-tweaks

Loadtest tweaks