[#73455918] Set user agent for crawler

Set the crawler's HTTP user agent to "GOV.UK Crawler" and add a test for
this.
I've changed it to "GOV.UK Crawler Worker" in 41177e. Likewise I wasn't sure about the version number - I'll add a story to the backlog.