alphagov/govuk_crawler_worker — pull request #79 by KushalP (2014-11-07 12:16:40+00:00)
Title: Don't extract URLs that set rel="nofollow" on the element

We should honour whatever nofollow attributes people use instead of blindly extracting all URLs.