[569] Fixes to the ONS scraper to make sure we get all results

When a link to a data page is found we mangle it to make sure we get
all of the pages at once, and to make sure we don't miss any. Also added
a test option and a bit more useful logging
[569] Apparently blank field identifiers are new
[569] Make sure we remove duplicate resources that we find
[569] Fix problem with using set with dicts
[569] Fixes
[569] Fetch links from the root page before traversing to "data" page
[569] Refactor to tidy up the code and make the scraper re-usable
[569] Temporary workaround api 502s
[569] Fix for large output (display name instead of dict)
[569] Now deletes any .xml redirector page that returns a 404 (optionally)
[569] Fix typo
[569] Adds original's description to start of newly found description
[642] Cherry-pick from master
[642] Remove unnecessary schema items
[642] Style changes and local tweaks
[642] Missing search restriction