A series of optimisiations and bugfixes aimed at getting the QA reports back live on DGU. Queries now execute fast enough so that Varnish does not time out before they return and templates are better formatted.

* Set filename of the al broken links CSV file to all_broken_links.csv (it was set to the built-in function id() before)
* Changed certain SQLAlchemy queries followed by multiple tsak_status queries into single SQL statements
* All reports only return data for active packages and resources
* Capitalized the openness score descriptions
* Tweaked all the main templates so that long links don't break column widths
* Made the "License not open" check come after the other checks, because we prefer to know if links are genuinely broken, rather than that information being supressed by the license. Only working, non-openly licences links now get the "License not open" message
* Noticed that requests library sometimes fails to get head requests after a 301 redirect, reporting bad port. Can't fix this but have added approprate log messages
Don't treat working links that aren't openly licensed as broken
Option to use internal site_url - needed for Bytemark. Added option to not have every page filtered for insertion of qa.css etc. Much improved logging in tasks.py.
More places where internal site url required.
Correct handling of new resources - expect an "error" when try to get tha failure count in this case.
QA report updated to use CKAN groups, rather than published_by field.
New detailed query for broken dataset results. New field - date of last time link worked. Excluded "Bad content type" from broken links report. Improved "reason" field to include more detail of why the score was given, instead of a description of the score which is useless. Added many data types, based on DGU data formats. Fixed up test_tasks.py
Update requests version to match ckanext-archiver.
Fix funny header in new version of requests, that prevented CKAN API calls(!). See #2942.
Scoring datasets - extension check does not use mimetype library since we have a better list of format that we are interested in. Removing unfinished attempt to correlate mime-types with extension. Promote mime-type as most important thing - holds publishers to account more. Broken datasets report - fixed rows getting jumbled, updated dates being wrong, hours and minutes mixed up.
CSV writer for lists of dicts. Made report dict ordered to help csv writer. Tidied table presentation.
Adding basic format sniffing
sniff_format tests now all pass.
Format sniffer vastly improved, using real files. Notifications from CKAN now alert bulk/priority queue as appropriate. Can now specify group (instead of package) in update command.
Task scoring tidied up. Scoring of extensions fixed. Paster command now defaults to correct queue when you specify a group. Cope with groups with deleted datasets in (!). Broken link reports now cope when no results. File sniffer now copes with unicode in filenames. Sniffer has more logging when it succeeds. Tests fixed up.
Give one star for downloaded but unknown format. Reports still not complete.
Revamp reports of broken links. Add paster commands: view & clean.
Include number of packages and resources in broken results.
Fix bad search-replace.
Added WMS to the sniffed formats. It is a service, but works fine like this.