alphagov/govuk_crawler_worker — pull request #62 by KushalP (2014-08-08 14:57:34+00:00)
Title: Calling `ExtractURLs()` only returns unique URLs

Many pages on GOV.UK link to the same page multiple times. To prevent
the same URLs being added to the queue, we should make sure that the
set of URLs we extract is unique.

Had to use a `map[string]*url.URL` here and iterate through the keys
because Go doesn't have an in-built Set data structure. Just arrays
and maps.