[#73455918] Set user agent for crawler

Set the crawler's HTTP user agent to "GOV.UK Crawler" and add a test for
this.
Thanks @dcarley, fixed in 513afe3.