alphagov/govuk_crawler_worker — pull request #2 by mattbostock (2014-06-26 10:47:44+00:00)
Title: [#73455918] Set user agent for crawler

Set the crawler's HTTP user agent to "GOV.UK Crawler" and add a test for
this.