alphagov/govuk_crawler_worker — pull request #49 by mattbostock (2014-08-04 09:02:10+00:00)
Title: [#75962980] Check URL is blacklisted when reading from queue 

Check if a URL is blacklisted as we read it from the RabbitMQ queue to
prevent us from crawling blacklisted URLs.

Previously we would only check URLs that were extracted from a crawled
HTML document.