alphagov/govuk_crawler_worker — pull request #42 by mattbostock (2014-07-31 15:39:55+00:00)
Title: [#75499462] Prevent crawler repeatedly hitting endpoint that returns 5XX status

Repurpose `ttlHashSet` such that we can increment the value of the keys stored in Redis to represent the number of unsuccessful times we have crawled a URL.

The rationale behind this change is to allow us to stop retrying URLs
that continuously return HTTP error codes, specifically HTTP 5XX errors.

We count crawl retries as follows:

- A value of `-1` means that the URL was successfully crawled within
the key expiry period.
- A value of `0` (zero) means that the URL has not been crawled
previously within the key expiry period (currently 48 hours). Zero
is also the value returned if the key does not yet exist.
- A value of `1` or greater indicates that the URL has been
unsuccesfully crawled as many times.

I've used the constants `AlreadyCrawled` and `NotRecentlyCrawled` to represent `-1` and `0` respectively.

It's worth digging into the commit messages here as there are some unresolved issues I'm going to tackle next, notably:
- The change in 198ea83 may need applying in other places where we execute multiple commands using Redis pipelining.
- We're using `time.Sleep()` in 2c7616d and elsewhere in our existing tests which I think we
should eventually remove; I'll raise a story to improve this later.

* * *

Also, convert some HTTP status codes to use constants from the `net.http` package where possible.