alphagov/govuk_crawler_worker — pull request #31 by KushalP (2014-07-23 08:54:09+00:00)
Title: [#73135884] Increase back off time for 429 response codes

Increase the amount of time we back off from crawling when we receive
a 429 HTTP status code. Otherwise we won't have waited long enough for
the server to let us continue.