[#75579526] Remove #fragment when extracting URLs from the DOM

This change strips the #fragment part of the URL when extracting URLs.
It fixes a bug where we were crawling the same URL multiple times, but
with different fragment identifiers.

Rather than crawling both `gov.uk/` and `gov.uk/#foo`, we'd now only
crawl the former case.
Merge pull request #33 from alphagov/remove-fragment-when-extracting-URLs

[#75579526] Remove #fragment when extracting URLs from the DOM