alphagov/govuk_crawler_worker — pull request #51 by mattbostock (2014-08-06 10:30:07+00:00)
Title: Reject item only after sleeping or retry increment

Prevent two race conditions by moving the rejection (and requeueing) of
the RabbitMQ message until after we have incremented the number of
retries against that URL or have slept on encountering a HTTP 429.

The race conditions that this solves are:

- The case where a HTTP 5XX error is encountered, the message would
have been requeued and potentially read by `ReadFromQueue()` before
the retry count was incremented on `*TTLHashSet` meaning we might
retry a URL more than the desired maximum number of retries.

- The case where a HTTP 429 error is encountered, the message would
have been requeued and the URL hit again by another goroutine
_before_ the `time.Sleep()`.