alphagov/govuk_crawler_worker — pull request #34 by KushalP (2014-07-28 15:17:58+00:00)
Title: Refactor filtering and mapping for url extraction

We make a few changes to the URLs we extract such as filtering out
URLs that don't match a certain predicate and adjusting the values
that we hold.

This change converts these operations to use a common set of
`filterURLs` and `mapURLs` functions that can be passed a function to
call for each item in the array of *url.URL elements.