[#75962980] Check URL is blacklisted when reading from queue 

Check if a URL is blacklisted as we read it from the RabbitMQ queue to
prevent us from crawling blacklisted URLs.

Previously we would only check URLs that were extracted from a crawled
HTML document.
Can I suggest we merge this now if everything else is OK, and we address this separately as part of story #76062972?