Continue to next item if already crawled

There's nothing further we need to do if a URL has already been crawled,
so continue to the next item.
Flip 'URL exists' logic for readability

Adjust the `if` statement to avoid the negative evaluation,
keep error checking code together and separate out the positive action,
`outbound <- message`.
Continue on error
Simplify if statement
Show Travis badge only for master branch

The Travis build badge was showing as 'failing' because it represented
all branches, some of which were failing.

Change it so that it only represents the master branch, which is what we
care about.
Merge pull request #44 from alphagov/fix-tests-badge

Show Travis badge only for master branch
Merge pull request #43 from alphagov/tidy-continues

Tidy continues and simplify some `if` blocks
Repurpose Add to Incr on TTL hash set

Rename and repurpose the `TTLHashSet.Add()` function so that it now
increments the specified key each time it's called.
Add `*TTLHashSet.Set()` function

Which takes an `int` as a value for the specified key.
Repurpose `*TTLHashSet.Exists()` to `.Get()`

Repurpose and rename `*TTLHashSet.Exists()` so that it now returns the
value stored against the specified key.

Note that a non-existent key will return a value of zero (0).
Use Redis to count crawl retries on a URL

Use the `ttlHashSet` (backed by Redis) to store the number of times
we've tried crawling a URL.

The rationale behind this change is to allow us to stop retrying URLs
that continuously return HTTP error codes, specifically HTTP 5XX errors.

We count crawl retries as follows:

  - A value of `-1` means that the URL was successfully crawled within
    the key expiry period.
  - A value of `0` (zero) means that the URL has not been crawled
    previously within the key expiry period (currently 48 hours). Zero
    is also the value returned if the key does not yet exist.
  - A value of 1 or greater indicates that the URL has been
    unsuccesfully crawled as many times.
Make max crawl retries configurable
Call `GetReply()` for each Redis command

Call `redis.Client.GetReply()` for each of `INCR` and `EXPIRE` commands
we append to the Redis pipeline.

This resolves an issue whereby the specified key was not being
incremented. Commenting out the line that appends the `EXPIRE` enabled
the incrementing to work as expected, which lead to the discovery that
we should be calling `GetReply()` for each command.

I'm still not 100% _why_ this is the case; indeed this may or may not
need to be applied to the other instances where we pipeline multiple
commands. I also don't understand why it was the `INCR` command that was
ignored, instead of the (later) `EXPIRE` command.

Note that the examples for the Redis client library we are using
calls `GetReply()` for each command appended to the pipeline, which
formed the basis for this fix:

https://github.com/fzzy/radix/blob/89fdcfd1614a851db5f57bfecdf78812d684ea9a/examples/example.go#L86-L91
Test that a key is incremented sequentially

Extend the existing test for incrementing a key to assert that the key
is incremented sequentially on consecutive calls to `ttlHashSet.Incr()`.

This test exposed the issue explained in the previous commit.
Test that we won't keep retrying URL indefinitely

Add a test to verify that we don't keep trying to crawl an item that
repeatedly returns HTTP 500 errors; in this case we allow 4 retries.

Note the use of `time.Sleep()` used here to allow time for the
goroutines spawned by `CrawlURL()` to consume the messages queued on the
`deliveries` channel. I'm not proud of this but it's proving difficult
to test this behaviour as the `crawled` channel will always be empty
(because we reject message that contains the errroring URL).

I tried using Gomega's `Eventually`[1] assertion to block until the
`deliveries` channel is empty, but that assertion seems to evaluate to
true before `CrawlURL()` before `ReadFromQueue()` is able to process the
message published to RabbitMQ (which is also published
asynchronously[2]).

We're using `time.Sleep()` elsewhere[3] in our tests which I think we
should eventually remove; I'll raise a story to improve this later.

[1]: http://onsi.github.io/gomega/
[2]: http://godoc.org/github.com/streadway/amqp#Channel.Publish
[3]:
https://github.com/alphagov/govuk_crawler_worker/blob/901afcc42277c8e757bd98121ecb945253378a5d/workflow_test.go#L228
Use HTTP status codes from net.http where possible

Use the HTTP status codes from Go's `net.http` package[1] to make the
code more readable.

Once noticeable exception is a HTTP 429 code, which is currently not
exported publicly from that package.

[1]: http://golang.org/src/pkg/net/http/status.go
Convert magic values to constants

Make it clear what `-1` and `0` represent by replacing these values with
constants.

Export these constants publicly so that we can re-use them in our tests
(I'd like to find a better way to do that).
Reinstate `ttlHashSet.Exists()`

It makes the tests more readable and it's useful in the context of this
generic and reusable package.
Rename `val` to `crawlCount`

To make it more obvious what this variable stores.