alphagov/govuk_crawler_worker — pull request #23 by mattbostock (2014-07-17 13:50:21+00:00)
Title: [#75178318] Prevent 404 and 3XX errors from being written to disk

These changes prevent HTTP 404 and 3XX responses from being written to disk by returning an error when we encounter a HTTP response with these error codes.

Using `transport.RoundTrip` means that we only perform one HTTP
transaction and therefore we do not try to resolve HTTP redirects. This
simplifies the logic around redirect by removing the need to define a
`CheckRedirect` function.

I had previously wanted to keep `http.Client` for its high-level
functions such as handling cookies but after consideration I don't think
we need them.

Also, the one function I can foresee us implementing, HTTP basic auth, is
supported on the `http.Request` interface and so can be used with
`http.Transport`.

This also allows us to return an error (`RedirectError`) in the same way
that we do for HTTP 404 and 5XX status codes.

The list of redirect status codes is cribbed from `http.Client` private
`shouldRedirectGet` function:
http://golang.org/src/pkg/net/http/client.go?s=5094:5155#L217