[#75499462] Prevent crawler repeatedly hitting an endpoint that returns 5XX status

Repurpose `ttlHashSet` such that we can increment the value of the keys stored in Redis to represent the number of unsuccessful times we have crawled a URL.

The rationale behind this change is to allow us to stop retrying URLs
that continuously return HTTP error codes, specifically HTTP 5XX errors.

We count crawl retries as follows:

  - A value of `-1` means that the URL was successfully crawled within
    the key expiry period.
  - A value of `0` (zero) means that the URL has not been crawled
    previously within the key expiry period (currently 48 hours). Zero
    is also the value returned if the key does not yet exist.
  - A value of `1` or greater indicates that the URL has been
    unsuccesfully crawled as many times.

I've used the constants `AlreadyCrawled` and `NotRecentlyCrawled` to represent `-1` and `0` respectively.

It's worth digging into the commit messages here as there are some unresolved issues I'm going to tackle next, notably:
  - The change in 198ea83 may need applying in other places where we execute multiple commands using Redis pipelining.
  - We're using `time.Sleep()` in 2c7616d and elsewhere in our existing tests which I think we
should eventually remove; I'll raise a story to improve this later.

* * *

Also, convert some HTTP status codes to use constants from the `net.http` package where possible.
Looking at the failing test in Travis, it might be because this server isn't running on port 80. One way to make sure we're crawling the correct `testServer` is to call `rootURL, _ = url.Parse(server.URL)`.

We'll then have the correct `host:port` for the server we've started. More details here: http://godoc.org/net/http/httptest#Server