alphagov/govuk_crawler_worker — pull request #55 by mattbostock (2014-08-07 08:32:01+00:00)
Title: [#76505094] Add non-HTML URLs to the TTLHashSet

To prevent us from retrying non-HTML URLs each time we encounter them,
ensure that they are marked in the TTLHashSet (backed by Redis) as
'already crawled', meaning they won't be retried for as a long as the
TTLHashSet's expiry time (currently 48 hours).