Return error if 404 encountered

Cause `http_crawler.Crawl()` to return an error if the URL being
crawled returns a HTTP 404 'Not found' error.

This is then caught by the `CrawlURL()` function in `workflow.go`[1],
meaning the contents of this URL are not written to disk.

[1]:
https://github.com/alphagov/govuk_crawler_worker/blob/2e4e1cc52796d3507e4abd635771346a9acdc519/workflow.go#L62-L65
Return an error if redirect status encountered

Cause `http_crawler.Crawl()` to return an error if a HTTP redirect
status code is encountered.

This will be caught by `CrawlURL()` in `workflow.go`[1] such that the
contents of this URL will not be written to disk.

To prevent the redirect, we specify a custom HTTP client that defines a
`CheckRedirect` function. Go's default `CheckRedirect` function follows
up to 10 redirects[2].

I looked at using `transport.RoundTrip()`[3], which results in a single
transaction (thus redirects aren't followed automatically), but
preferred to keep `http.Client` so that we can use its high-level
features.

[1]:
https://github.com/alphagov/govuk_crawler_worker/blob/2e4e1cc52796d3507e4abd635771346a9acdc519/workflow.go#L62-L65
[2]: http://golang.org/src/pkg/net/http/client.go?s=882:2405#L389
[3]: https://groups.google.com/forum/#!topic/golang-nuts/PmYm3h2J3FE
Use http.Transport instead of http.Client

Prefer `http.Transport` instead of `http.Client`.

Using `transport.RoundTrip` means that we only perform one HTTP
transaction and therefore we do not try to resolve HTTP redirects. This
simplifies the logic around redirect by removing the need to define a
`CheckRedirect` function.

I had previously wanted to keep `http.Client` for its high-level
functions such as handling cookies but after consideration I don't think
we need them.

Also, the one function I can forsee us implementing, HTTP basic auth, is
supported on the `http.Request` interface and so can be used with
`http.Transport`.

This also allows us to return an error (`RedirectError`) in the same way
that we do for HTTP 404 and 5XX status codes.

The list of redirect status codes is cribbed from `http.Client` private
`shouldRedirectGet` function:
http://golang.org/src/pkg/net/http/client.go?s=5094:5155#L217
Move RetryStatusCodes() further down

It's less important than the functions that followed it; move it further
down the file so that the code is easier to read (most important
functions first).

I had wanted to make this function private as it's not used outside of
`http_crawler` in the main code, but it turns out that the Ginkgo tests
fail if it's defined as a private method. Usually we wouldn't want to
test private methods, but in this case I think this test adds value.
Use `http.RedirectHandler` function in test

See: http://golang.org/pkg/net/http/#RedirectHandler
Make it clear that HTML body not related to error
Use http.DefaultTransport

To avoid creating a new transport on each request. This also
retains the default timeouts as used by `http.Client`[1].

See:
http://golang.org/src/pkg/net/http/transport.go#L33

[1]: http://golang.org/src/pkg/net/http/client.go#L162
Merge pull request #23 from alphagov/75178318-skip-non-200-responses

[#75178318] Prevent 404 and 3XX errors from being written to disk