alphagov/govuk_crawler_worker — pull request #12 by mattbostock (2014-07-03 17:42:31+00:00)
Title: [#74371458]  Refactor ExtractURLs()

Refactor ExtractURLs() to address the following issues:

1. Relative URLs were resolved to use HTTP (not HTTPS), which would
result in unnecessary redirects on our stack:
https://github.com/alphagov/govuk_crawler_worker/blob/378c4189a8c7feb13131a896d90b835e65c862af/crawler_message_item.go#L86

2. Relative URLs that do not start with a leading slash were ignored:
https://github.com/alphagov/govuk_crawler_worker/blob/378c4189a8c7feb13131a896d90b835e65c862af/crawler_message_item.go#L85-L87

3. The `findByElementAttribute` function was doing too much:
a) It extracts URLs from the document DOM
b) It filters blacklisted URLs
c) It resolves relative URLs to absolute URLs
d) It filters by the URL host:
https://github.com/alphagov/govuk_crawler_worker/blob/378c4189a8c7feb13131a896d90b835e65c862af/crawler_message_item.go#L61-L92

To address this;

- The `host` variable on `CrawlerMessageItem` structs has been replaced
with a URL type struct that stores the root URL. I couldn't actually
find anywhere in the code that used the `host` variable on that
struct, and passing in the root URL means we're now able to filter out
the links we extract from documents by comparing them to the root URL,
rather than the URL of the current document being parsed. Previously,
we'd add links for external sites to RabbitMQ but they'd be dropped by
`Crawler.Crawl()`[1].

- Use the same protocol schema as the root URL when resolving relative
URLs.

- Use url.ResolveReference to convert relative URLs to absolute URLs[2].
Note that this method will take both absolute and relative URLs, so we
pass everything through it.

- Separate the functionality of `findByElementAttribute` into 4 new
methods. These methods use the URL struct type[3] for both for their
arguments and return values so that it's easier for us to perform
comparisons on and to manipulate the URLs.

Also, the behaviour of `isBlacklistedPath()` has changed to use
`strings.HasPrefix()` to improve the accuracy of matches.

[1]:
https://github.com/alphagov/govuk_crawler_worker/blob/7f0370b05ba5bbd1c29264d6bafbc2bb82d7f39f/http_crawler/crawler.go#L62-L64
[2]: http://golang.org/pkg/net/url/#URL.ResolveReference
[3]: http://golang.org/pkg/net/url/#URL