Log when we're sleeping a goroutine for a 429 response

When we're told we're making too many requests, and we're sleeping we
should log this action so a user watching the process can better
understand what's going on.
Merge pull request #38 from alphagov/log-when-we-go-to-sleep-for-429-responses

Log when we're sleeping a goroutine for a 429 response
Continue when we reject a message for not extracting URLs

We can't do both rejection and acknowledgement of a message. So if we
reject a message, we should continue to the next one.
Merge pull request #39 from alphagov/continue-when-we-reject-a-message-for-not-extracting-urls

Continue when we reject a message for not extracting URLs
[#75962332] Relax URL path sanitisation

Just use path.Clean() to remove/resolve traversals, which will prevent the
file from being written outside of our root. The kennygrant/sanitize package
was doing too much as described in the story and covered by the additional
test cases. We don't strictly need these now that we've removed that package
but I think it might make sense to prevent regressions if we revisit this in
the future.

NB: About non-latin characters. It seems that the old mirror wrote these
filenames as URL encoded. However some testing shows that Nginx on the
mirror machines won't serve these as desired:

    root@mirror0:/srv/mirror_data/www-origin# echo one > 如何在香港申請英國簽證.html
    root@mirror0:/srv/mirror_data/www-origin# echo two > %E5%A6%82%E4%BD%95%E5%9C%A8%E9%A6%99%E6%B8%AF%E7%94%B3%E8%AB%8B%E8%8B%B1%E5%9C%8B%E7%B0%BD%E8%AD%89.html
    root@mirror0:/srv/mirror_data/www-origin# curl -kH "Host: www-origin.mirror.foo" https://localhost/如何在香港申請英國簽證
    one
    root@mirror0:/srv/mirror_data/www-origin# curl -kH "Host: www-origin.mirror.foo" https://localhost/%E5%A6%82%E4%BD%95%E5%9C%A8%E9%A6%99%E6%B8%AF%E7%94%B3%E8%AB%8B%E8%8B%B1%E5%9C%8B%E7%B0%BD%E8%AD%89one

Our default behaviour of not URL encoding (and now not stripping) these is
correct.
Merge pull request #40 from alphagov/75962332-relax_url_sanitisation

[Fixes #75962332] Relax URL path sanitisation
Fix typo in queue test suite filename
Merge pull request #41 from alphagov/rename-test-suite

Fix typo in queue test suite filename
Repurpose Add to Incr on TTL hash set

Rename and repurpose the `TTLHashSet.Add()` function so that it now
increments the specified key each time it's called.
Add `*TTLHashSet.Set()` function

Which takes an `int` as a value for the specified key.
Repurpose `*TTLHashSet.Exists()` to `.Get()`

Repurpose and rename `*TTLHashSet.Exists()` so that it now returns the
value stored against the specified key.

Note that a non-existent key will return a value of zero (0).
Use Redis to count crawl retries on a URL

Use the `ttlHashSet` (backed by Redis) to store the number of times
we've tried crawling a URL.

The rationale behind this change is to allow us to stop retrying URLs
that continuously return HTTP error codes, specifically HTTP 5XX errors.

We count crawl retries as follows:

  - A value of `-1` means that the URL was successfully crawled within
    the key expiry period.
  - A value of `0` (zero) means that the URL has not been crawled
    previously within the key expiry period (currently 48 hours). Zero
    is also the value returned if the key does not yet exist.
  - A value of 1 or greater indicates that the URL has been
    unsuccesfully crawled as many times.
Make max crawl retries configurable
Call `GetReply()` for each Redis command

Call `redis.Client.GetReply()` for each of `INCR` and `EXPIRE` commands
we append to the Redis pipeline.

This resolves an issue whereby the specified key was not being
incremented. Commenting out the line that appends the `EXPIRE` enabled
the incrementing to work as expected, which lead to the discovery that
we should be calling `GetReply()` for each command.

I'm still not 100% _why_ this is the case; indeed this may or may not
need to be applied to the other instances where we pipeline multiple
commands. I also don't understand why it was the `INCR` command that was
ignored, instead of the (later) `EXPIRE` command.

Note that the examples for the Redis client library we are using
calls `GetReply()` for each command appended to the pipeline, which
formed the basis for this fix:

https://github.com/fzzy/radix/blob/89fdcfd1614a851db5f57bfecdf78812d684ea9a/examples/example.go#L86-L91
Test that a key is incremented sequentially

Extend the existing test for incrementing a key to assert that the key
is incremented sequentially on consecutive calls to `ttlHashSet.Incr()`.

This test exposed the issue explained in the previous commit.
Test that we won't keep retrying URL indefinitely

Add a test to verify that we don't keep trying to crawl an item that
repeatedly returns HTTP 500 errors; in this case we allow 4 retries.

Note the use of `time.Sleep()` used here to allow time for the
goroutines spawned by `CrawlURL()` to consume the messages queued on the
`deliveries` channel. I'm not proud of this but it's proving difficult
to test this behaviour as the `crawled` channel will always be empty
(because we reject message that contains the errroring URL).

I tried using Gomega's `Eventually`[1] assertion to block until the
`deliveries` channel is empty, but that assertion seems to evaluate to
true before `CrawlURL()` before `ReadFromQueue()` is able to process the
message published to RabbitMQ (which is also published
asynchronously[2]).

We're using `time.Sleep()` elsewhere[3] in our tests which I think we
should eventually remove; I'll raise a story to improve this later.

[1]: http://onsi.github.io/gomega/
[2]: http://godoc.org/github.com/streadway/amqp#Channel.Publish
[3]:
https://github.com/alphagov/govuk_crawler_worker/blob/901afcc42277c8e757bd98121ecb945253378a5d/workflow_test.go#L228