Add *CrawlerMessageItem.IsBlacklisted() function

Add a function `IsBlacklisted()` on the `CrawlerMessageItem` interface
that returns true if the item's URL is blacklisted.

We'll be using this in the next commit to check for blacklisted URLs
when reading messages from a RabbitMQ queue.
Check URL is blacklisted when reading from queue

Check if a URL is blacklisted as we read it from the RabbitMQ queue to
prevent us from crawling blacklisted URLs.

Previously we would only check URLs that were extracted from a crawled
HTML document.

Note that in the tests, we inspect the RabbitMQ queue to verify that the
message has been rejected. This is necessary because the channel
returned by `ReadFromQueue()` will always have a zero length (since the
message is rejected), so we can't perform any assertions on that
channel.
Merge pull request #49 from alphagov/75962980-blacklist-urls-read-from-queue

[#75962980] Check URL is blacklisted when reading from queue
Repurpose Add to Incr on TTL hash set

Rename and repurpose the `TTLHashSet.Add()` function so that it now
increments the specified key each time it's called.
Add `*TTLHashSet.Set()` function

Which takes an `int` as a value for the specified key.
Repurpose `*TTLHashSet.Exists()` to `.Get()`

Repurpose and rename `*TTLHashSet.Exists()` so that it now returns the
value stored against the specified key.

Note that a non-existent key will return a value of zero (0).
Use Redis to count crawl retries on a URL

Use the `ttlHashSet` (backed by Redis) to store the number of times
we've tried crawling a URL.

The rationale behind this change is to allow us to stop retrying URLs
that continuously return HTTP error codes, specifically HTTP 5XX errors.

We count crawl retries as follows:

  - A value of `-1` means that the URL was successfully crawled within
    the key expiry period.
  - A value of `0` (zero) means that the URL has not been crawled
    previously within the key expiry period (currently 48 hours). Zero
    is also the value returned if the key does not yet exist.
  - A value of 1 or greater indicates that the URL has been
    unsuccesfully crawled as many times.
Make max crawl retries configurable
Test that a key is incremented sequentially

Extend the existing test for incrementing a key to assert that the key
is incremented sequentially on consecutive calls to `ttlHashSet.Incr()`.

This test exposed the issue explained in the previous commit.
Test that we won't keep retrying URL indefinitely

Add a test to verify that we don't keep trying to crawl an item that
repeatedly returns HTTP 500 errors; in this case we allow 4 retries.

Because nothing should be sent over the `crawled` channel  (since the
`*CrawlMessageItem` being processed will be rejected by `CrawlURL()`),
we use the value stored in the Redis-backed `*TTLHashSet` to determine
if the maximum number of retires has been reached.
Use HTTP status codes from net.http where possible

Use the HTTP status codes from Go's `net.http` package[1] to make the
code more readable.

Once noticeable exception is a HTTP 429 code, which is currently not
exported publicly from that package.

[1]: http://golang.org/src/pkg/net/http/status.go
Convert magic values to constants

Make it clear what `-1` and `0` represent by replacing these values with
constants.

Export these constants publicly so that we can re-use them in our tests
(I'd like to find a better way to do that).
Reinstate `ttlHashSet.Exists()`

It makes the tests more readable and it's useful in the context of this
generic and reusable package.
Rename `val` to `crawlCount`

To make it more obvious what this variable stores.
Defer mutex unlock until function returns

`defer` on `t.mutex.Unlock()` so that the mutex is unlocked whenever
the `*TTLHashSet.Incr()` function returns.

This ensures that the mutex is always unlocked when this function
returns. Previously an error may have caused the function to return but
left the mutex locked, causing other goroutines to block.