[#73455918] Set user agent for crawler

Set the crawler's HTTP user agent to "GOV.UK Crawler" and add a test for
this.
I think there's a small regression here in that every request will instantiate a new client object. Whereas it's beneficial, both in terms of memory usage and connections, to re-use them. From the [http.Client docs](http://golang.org/pkg/net/http/#Client):

> The Client's Transport typically has internal state (cached TCP connections), so Clients should be reused instead of created as needed. Clients are safe for concurrent use by multiple goroutines.

NB: [http.Get](http://golang.org/src/pkg/net/http/client.go?s=8155:8203#L241) is a wrapper for [http.DefaultClient](http://golang.org/src/pkg/net/http/client.go?s=882:2405#L74).