Set user agent for crawler

Set the crawler's HTTP user agent to "GOV.UK Crawler" and add a test for
this.
Reuse HTTP client

Previously, every request would instantiate a new HTTP client.

Use `http.DefaultClient`[1] instead, which is the default client that Go
uses for HTTP requests, e.g. `http.Get` uses it[2].

[1]: http://golang.org/src/pkg/net/http/client.go?#L74
[2]: http://golang.org/src/pkg/net/http/client.go?#L251
Change user agent to 'GOV.UK Crawler Worker'

Which better reflects the name of this project.
Add hostname to user agent

So that we know which server the crawler is running on.

Only check for the string "GOV.UK Crawler Worker" in our tests as it's
not easy for us to mock the system hostname and testing against
`os.Hostname()` would be of minimal benefit.

Also removes the function that trims the body of the HTML response, as
there's no reason for there to be spaces around the text in the response
body.
Rename testServer for user agent test

The function signature for the httptest.Server we initiate here is
different to the `testServer` that precedes it, so use a different
variable name to make it clear that this test server behaves
differently, i.e. it puts the user agent into the response body.
Split HTTP server handler into own variable

To improve readability and to pave the way for passing in a handler to
this function if necessary in future.