alphagov/govuk_crawler_worker — pull request #33 by KushalP (2014-07-28 11:55:25+00:00)
Title: [#75579526] Remove #fragment when extract URLs from the DOM

This change strips the #fragment part of the URL when extracting URLs.
It fixes a bug where we were crawling the same URL multiple times, but
with different fragment identifiers.

Rather than crawling both `gov.uk/` and `gov.uk/#foo`, we'd now only
crawl the former case.