Log when the AMQP channel's `notifyClose` channel is closed

We didn't have anything to check whether the `notifyClose` channel was
open. We were only doing something when an `amqp.Error` message was
added to the channel.

By adding this, we'll be able to track when/why a channel might be
closed.
[#76062972] Reduce reliance on `time.Sleep` for tests

Where possible we should use the asynchronous constructs provided by
our testing utilities (ginkgo and gomega).

The majority of tests have been converted to use these new matchers.
The only test that hasn't been converted is one that relies on
timings (TCP reconnection in Redis) where an internal goroutine's
behaviour (read: delay) is time-based.
Merge pull request #46 from alphagov/reduce-reliance-on-time.Sleep-in-tests

[#76062972] Reduce reliance on `time.Sleep` for tests
Merge pull request #48 from alphagov/log-when-the-channel-is-closed

Log when the AMQP channel's `notifyClose` channel is closed
Reorder workflow functions to be in the order that they run

Make it a little easier for people who are looking to follow the
workflow we have in our `main()` function by placing the workflow
functions in the other that we run them.
Explicitly log any errors we have trying to acknowledge messages

`Delivery` objects return an error if they can't be acknowledged. Log
when we receive an error so that we know where these errors are
appearing.
[#76261684] Increase queue read channel buffer based on number of crawlers

Increase the buffer we hold for `Delivery` items to equal the number
of goroutines we're running for crawling.
Merge pull request #50 from alphagov/increase-reader-channel-buffer-to-equal-number-of-crawler-threads

[#76261684] Log acknowledgement errors and increase reader channel buffer
Repurpose Add to Incr on TTL hash set

Rename and repurpose the `TTLHashSet.Add()` function so that it now
increments the specified key each time it's called.
Add `*TTLHashSet.Set()` function

Which takes an `int` as a value for the specified key.
Repurpose `*TTLHashSet.Exists()` to `.Get()`

Repurpose and rename `*TTLHashSet.Exists()` so that it now returns the
value stored against the specified key.

Note that a non-existent key will return a value of zero (0).
Use Redis to count crawl retries on a URL

Use the `ttlHashSet` (backed by Redis) to store the number of times
we've tried crawling a URL.

The rationale behind this change is to allow us to stop retrying URLs
that continuously return HTTP error codes, specifically HTTP 5XX errors.

We count crawl retries as follows:

  - A value of `-1` means that the URL was successfully crawled within
    the key expiry period.
  - A value of `0` (zero) means that the URL has not been crawled
    previously within the key expiry period (currently 48 hours). Zero
    is also the value returned if the key does not yet exist.
  - A value of 1 or greater indicates that the URL has been
    unsuccesfully crawled as many times.
Make max crawl retries configurable
Call `GetReply()` for each Redis command

Call `redis.Client.GetReply()` for each of `INCR` and `EXPIRE` commands
we append to the Redis pipeline.

This resolves an issue whereby the specified key was not being
incremented. Commenting out the line that appends the `EXPIRE` enabled
the incrementing to work as expected, which lead to the discovery that
we should be calling `GetReply()` for each command.

I'm still not 100% _why_ this is the case; indeed this may or may not
need to be applied to the other instances where we pipeline multiple
commands. I also don't understand why it was the `INCR` command that was
ignored, instead of the (later) `EXPIRE` command.

Note that the examples for the Redis client library we are using
calls `GetReply()` for each command appended to the pipeline, which
formed the basis for this fix:

https://github.com/fzzy/radix/blob/89fdcfd1614a851db5f57bfecdf78812d684ea9a/examples/example.go#L86-L91
Test that a key is incremented sequentially

Extend the existing test for incrementing a key to assert that the key
is incremented sequentially on consecutive calls to `ttlHashSet.Incr()`.

This test exposed the issue explained in the previous commit.
Test that we won't keep retrying URL indefinitely

Add a test to verify that we don't keep trying to crawl an item that
repeatedly returns HTTP 500 errors; in this case we allow 4 retries.

Because nothing should be sent over the `crawled` channel  (since the
`*CrawlMessageItem` being processed will be rejected by `CrawlURL()`),
we use the value stored in the Redis-backed `*TTLHashSet` to determine
if the maximum number of retires has been reached.
Use HTTP status codes from net.http where possible

Use the HTTP status codes from Go's `net.http` package[1] to make the
code more readable.

Once noticeable exception is a HTTP 429 code, which is currently not
exported publicly from that package.

[1]: http://golang.org/src/pkg/net/http/status.go
Convert magic values to constants

Make it clear what `-1` and `0` represent by replacing these values with
constants.

Export these constants publicly so that we can re-use them in our tests
(I'd like to find a better way to do that).
Reinstate `ttlHashSet.Exists()`

It makes the tests more readable and it's useful in the context of this
generic and reusable package.
Rename `val` to `crawlCount`

To make it more obvious what this variable stores.