alphagov/govuk_crawler_worker — pull request #1 by mattbostock (2014-06-24 14:13:33+00:00)
Title: [#73136218] Write crawled items to disk

Write crawled HTML content to disk by extending the CrawlerMessageItem
interface with a new method, WriteToDisk(), and its supporting method,
FileName().

Use the 'github.com/kennygrant/sanitize' package to sanitise the file
path and filename. Note that the function used, `sanitize.Path()`, is
actually intended for sanitising URLs rather than filenames, but serves
our purpose well because it runs `filepath.Clean()`[1] and also leaves
the base path (i.e. '/government/organisations/') intact. The function
from that same package, `sanitize.Name`, uses `path.Base()` to strip
this out and so is not useful for our case. Should the behaviour of
`sanitize.Path()` change in future, our tests should pick this up.

Adds a new `mirrorRoot` configuration variable which _must_ be set as no
default is defined to prevent the crawler from writing to disk without
this being explicitly set.

Files are written according to their URL path, so the URL
`https://www.gov.uk/government/organisations` will be written as
`/government/organisations.html` inside `mirrorRoot`. Directories are
created as necessary.

Note that by calling `WriteToDisk()` from the `CrawlURL` function in
`workflow.go`, writes to disk are tightly coupled with the act of
crawling. Given the slow speed at which we crawl (1-2 pages per second),
I don't think this is worth addressing now, but we may in future
consider moving this logic to a separate worker which could consume
crawl items from a queue and write them to disk.

[1]: http://golang.org/pkg/path/filepath/#Clean