DRY up CrawlerMessageItem tests

DRY up the CrawlerMessageItem tests by defining the host and URL
fixtures up front.

Also, add sub-page to the example URL we test with so that we can make
use of it in later tests.
`go fmt` third_party.go
Write crawled items to disk

Write crawled HTML content to disk by extending the CrawlerMessageItem
interface with a new method, WriteToDisk(), and its supporting method,
FileName().

Use the 'github.com/kennygrant/sanitize' package to sanitise the file
path and filename. Note that the function used, `sanitize.Path()`, is
actually intended for sanitising URLs rather than filenames, but serves
our purpose well because it runs `filepath.Clean()`[1] and also leaves
the base path (i.e. '/government/organisations/') intact. The function
from that same package, `sanitize.Name`, uses `path.Base()` to strip
this out and so is not useful for our case. Should the behaviour of
`sanitize.Path()` change in future, our tests should pick this up.

Adds a new `mirrorRoot` configuration variable which _must_ be set as no
default is defined to prevent the crawler from writing to disk without
this being explicitly set.

Files are written according to their URL path, so the URL
`https://www.gov.uk/government/organisations` will be written as
`/government/organisations.html` inside `mirrorRoot`. Directories are
created as necessary.

Note that by calling `WriteToDisk()` from the `CrawlURL` function in
`workflow.go`, writes to disk are tightly coupled with the act of
crawling. Given the slow speed at which we crawl (1-2 pages per second),
I don't think this is worth addressing now, but we may in future
consider moving this logic to a separate worker which could consume
crawl items from a queue and write them to disk.

[1]: http://golang.org/pkg/path/filepath/#Clean
Test WriteToDisk() and FilePath() methods

Test the new WriteToDisk() and FilePath() methods on the
CrawlerMessageItem interface.

Move some of the fixture variables inside BeforeEach() so that we can
re-use them across tests.

We also now populate `item.HTMLBody` by default as the new FilePath()
and WriteToDisk() methods will only act on content it detects as HTML.

The test for WriteToDisk() will actually write to disk inside the
directory specified by the MIRROR_ROOT environment variable, which must
be set. I don't think this is an issue given that many of the other
tests are also integration tests, i.e. they rely on networking to test
crawling.

Mocking in Go is also non-trivial. Testify's `mock`
package[1] looks like a nice solution but I don't think it's necessary
in this case.

Also add a test for the URL() method on that interface.

[1]: https://github.com/stretchr/testify#mock-package
Rename fileName, FileName to filePath, FilePath

To be more specific.
Test that caps in URLs are converted to lower case
Update README.md to include MIRROR_ROOT

Which is now required. Content crawled will be written to disk inside
this path.
Delete mirror root after tests run

Delete the mirror root directory, as defined by the `MIRROR_ROOR`
environment variable, after each of the tests has run.

Do this after _each_ test in case so as not to cross-contaminate
fixtures between tests.

Also add a note to the README to warn that this will happen.

Add a new file, `helpers_test.go` to contain the shared function that
deletes the mirror root. This would have been named `test_helpers.go`,
but it seems Ginkgo requires it to be named with the `_test.go` suffix.

I also wanted to raise an error and call `panic()` or `os.Exit()` if
`MIRROR_ROOT` already exists on disk to prevent someone from deleting
files they want to keep but I couldn't get Ginkgo to exit before it runs
AfterEach(), which deletes those files.
Update travis.yml with MIRROR_ROOT env var
Write content to disk in separate goroutines

Move the `WriteToDisk()` method from the `CrawlerMessageItem` interface
to run as part of the main workflow in `workflow.go` because writing is
a separate concern to crawling.

Update the tests accordingly and improve them so that we check for
existence of the file created on disk.

Use two goroutines for symmetry with `CrawlURL()`. Note that we reject
the message without requeueing it as a failure to write to disk is
likely to require some manual intervention and I don't think it's wise
for us to retry.
Explicitly declare channels for readability

Declare the variables we use as
channels before they are used to improve readability.

When I first looked at this code, being new to Go, it took me a moment
to realise that `acknowledge`, `crawlItems` and `extract` are in fact
channels. By using `var` instead of the `:=` shortcut, it should be
clearer what these are.

I also considered (credit to @dancarley) creating a custom type such as:
```go
type itemChan <-chan *CrawlerMessageItem
```

However, I think using this and replacing all instances of `<-chan
*CrawlerMessageItem` in the codebase, while reducing duplication, would
obscure its meaning.
Rename channels to make their intent clearer
Remove second goroutine that calls writeLoop()

I had added the second goroutine for symmetry with `CrawlURL()`[1], but
on second thought there's no need for this. Crawling is time-consuming
whereas writing to disk should be much less so.

[1]:
https://github.com/alphagov/govuk_crawler_worker/blob/a4559142407cbfe50956a777f28f5d96693fa934/workflow.go#L70-L71
Test the content that is written to disk

In the test for WriteItemToDisk(), we were previously only checking that
the file existed on disk.

Expand this test to check the contents of the file written to disk. The
file's existence is confirmed by the fact of it having content and we
also check for errors when reading the file.
CrawlerMessageItem should only know relative paths

Rename the `FilePath()` method on the `CrawlerMessageItem` interface to
`RelativeFilePath()` and change its logic accordingly so that it only
returns a relative file path, to which we append the mirror root
directory in `WriteItemToDisk()` in `workflow.go`.

This better separates the concerns of the workflow versus
`CrawlerMessageItem`, which should not concern itself with the mirror
root.
Pass mirrorRoot as argument to WriteItemToDisk()

So that WriteItemToDisk() doesn't need to care about configuration; this
should be handled in `main.go`.
Rename channel variables to be more descriptive

So that's it clear that these are channels.
Relative file paths shouldn't have a leading slash

Previously we were prepending the relative filepath returned by
RelativeFilePath() with a leading slash.

This isn't correct - a relative path should not have a leading slash.
The reason this was added was so that there would be a slash between the
mirror root path and the relative file path, but this is already handled
by path.Join[1].

Update the tests accordingly.

Also, fix the following tests, which weren't testing for a '.html'
suffix because they were not testing HTML content:

- strips out the domain, protocol, auth and ports
- strips illegal characters

[1]: http://golang.org/pkg/path/#Join
Panic if MIRROR_ROOT environment variable not set

Panic so that the application fails fast.
Use bang to negate expression in case statement

Sorry @dcarley...