ProxyTCP helper for tests

Add a helper for proxying TCP connections. It has a method to kill all
currently connected sessions, which will be used for testing reconnects in
response to network disruption. It's modelled mostly on httptest.Server.
Redis reconnect after error

The radix/redis client closes the connection to the server if it experiences
an error. This is what causes the error caught in 8c907f8. Improve our
reliability by reconnecting the client if we encounter a network or IO
related error (but not a semantic error like an invalid command).

This will not block the operation that first caused the error, meaning that
PublishURL() may submit one already crawled URL and ReadFromQueue() may
reject (requeue) an item.

The reconnect will run in another goroutine but subsequent operations will
be blocking, causing PublishURL() and ReadFromQueue() to then stop
processing until Redis is available again. This is a desirable result
because without the cache we'd expend resources re-crawling already
up-to-date pages.

There is a 2s delay between reconnect attempts so as not to overwhelm the
client or server. But no limit on how many times it will try to reconnect.

I'm unsure about how we're going to monitor this because the health check
endpoint which is going to use ttlHashSet.Ping() will also block and we
don't have any logging inside of ttlHashSet.
DRY up CrawlerMessageItem tests

DRY up the CrawlerMessageItem tests by defining the host and URL
fixtures up front.

Also, add sub-page to the example URL we test with so that we can make
use of it in later tests.
`go fmt` third_party.go
Write crawled items to disk

Write crawled HTML content to disk by extending the CrawlerMessageItem
interface with a new method, WriteToDisk(), and its supporting method,
FileName().

Use the 'github.com/kennygrant/sanitize' package to sanitise the file
path and filename. Note that the function used, `sanitize.Path()`, is
actually intended for sanitising URLs rather than filenames, but serves
our purpose well because it runs `filepath.Clean()`[1] and also leaves
the base path (i.e. '/government/organisations/') intact. The function
from that same package, `sanitize.Name`, uses `path.Base()` to strip
this out and so is not useful for our case. Should the behaviour of
`sanitize.Path()` change in future, our tests should pick this up.

Adds a new `mirrorRoot` configuration variable which _must_ be set as no
default is defined to prevent the crawler from writing to disk without
this being explicitly set.

Files are written according to their URL path, so the URL
`https://www.gov.uk/government/organisations` will be written as
`/government/organisations.html` inside `mirrorRoot`. Directories are
created as necessary.

Note that by calling `WriteToDisk()` from the `CrawlURL` function in
`workflow.go`, writes to disk are tightly coupled with the act of
crawling. Given the slow speed at which we crawl (1-2 pages per second),
I don't think this is worth addressing now, but we may in future
consider moving this logic to a separate worker which could consume
crawl items from a queue and write them to disk.

[1]: http://golang.org/pkg/path/filepath/#Clean
Test WriteToDisk() and FilePath() methods

Test the new WriteToDisk() and FilePath() methods on the
CrawlerMessageItem interface.

Move some of the fixture variables inside BeforeEach() so that we can
re-use them across tests.

We also now populate `item.HTMLBody` by default as the new FilePath()
and WriteToDisk() methods will only act on content it detects as HTML.

The test for WriteToDisk() will actually write to disk inside the
directory specified by the MIRROR_ROOT environment variable, which must
be set. I don't think this is an issue given that many of the other
tests are also integration tests, i.e. they rely on networking to test
crawling.

Mocking in Go is also non-trivial. Testify's `mock`
package[1] looks like a nice solution but I don't think it's necessary
in this case.

Also add a test for the URL() method on that interface.

[1]: https://github.com/stretchr/testify#mock-package
Rename fileName, FileName to filePath, FilePath

To be more specific.
Test that caps in URLs are converted to lower case
Update README.md to include MIRROR_ROOT

Which is now required. Content crawled will be written to disk inside
this path.
Delete mirror root after tests run

Delete the mirror root directory, as defined by the `MIRROR_ROOR`
environment variable, after each of the tests has run.

Do this after _each_ test in case so as not to cross-contaminate
fixtures between tests.

Also add a note to the README to warn that this will happen.

Add a new file, `helpers_test.go` to contain the shared function that
deletes the mirror root. This would have been named `test_helpers.go`,
but it seems Ginkgo requires it to be named with the `_test.go` suffix.

I also wanted to raise an error and call `panic()` or `os.Exit()` if
`MIRROR_ROOT` already exists on disk to prevent someone from deleting
files they want to keep but I couldn't get Ginkgo to exit before it runs
AfterEach(), which deletes those files.
Update travis.yml with MIRROR_ROOT env var
Write content to disk in separate goroutines

Move the `WriteToDisk()` method from the `CrawlerMessageItem` interface
to run as part of the main workflow in `workflow.go` because writing is
a separate concern to crawling.

Update the tests accordingly and improve them so that we check for
existence of the file created on disk.

Use two goroutines for symmetry with `CrawlURL()`. Note that we reject
the message without requeueing it as a failure to write to disk is
likely to require some manual intervention and I don't think it's wise
for us to retry.
Explicitly declare channels for readability

Declare the variables we use as
channels before they are used to improve readability.

When I first looked at this code, being new to Go, it took me a moment
to realise that `acknowledge`, `crawlItems` and `extract` are in fact
channels. By using `var` instead of the `:=` shortcut, it should be
clearer what these are.

I also considered (credit to @dancarley) creating a custom type such as:
```go
type itemChan <-chan *CrawlerMessageItem
```

However, I think using this and replacing all instances of `<-chan
*CrawlerMessageItem` in the codebase, while reducing duplication, would
obscure its meaning.
Rename channels to make their intent clearer
Remove second goroutine that calls writeLoop()

I had added the second goroutine for symmetry with `CrawlURL()`[1], but
on second thought there's no need for this. Crawling is time-consuming
whereas writing to disk should be much less so.

[1]:
https://github.com/alphagov/govuk_crawler_worker/blob/a4559142407cbfe50956a777f28f5d96693fa934/workflow.go#L70-L71
Test the content that is written to disk

In the test for WriteItemToDisk(), we were previously only checking that
the file existed on disk.

Expand this test to check the contents of the file written to disk. The
file's existence is confirmed by the fact of it having content and we
also check for errors when reading the file.
Don't block Redis ops during reconnect

This addresses the concern in the previous commit about breaking the Ping()
health check while another goroutine is attempting to reconnect. It also
makes the behaviour of other operations more predictable by returning errors
while the reconnect is in progress. These errors will be surfaced outside of
ttlHashSet, so we don't necessarily need to log the fact that we're
reconnecting.

I've had to introduce an additional flag protected by an r/w mutex to ensure
that we only create one reconnect goroutine. I've put this in the util
package because I suspect we may use the same pattern for RabbitMQ.

The tests have been adapted to assert that we receive roughly the right
amount of errors for the correct period of time based on the reconnect
interval. Then ensure we can reconnect again afterwards.

This now means that PublishURLs() will write duplicate and already crawled
URLs to the queue. However ReadFromQueue() should reject them until Redis is
available again. So we shouldn't waste too much resource by re-crawling
them.
CrawlerMessageItem should only know relative paths

Rename the `FilePath()` method on the `CrawlerMessageItem` interface to
`RelativeFilePath()` and change its logic accordingly so that it only
returns a relative file path, to which we append the mirror root
directory in `WriteItemToDisk()` in `workflow.go`.

This better separates the concerns of the workflow versus
`CrawlerMessageItem`, which should not concern itself with the mirror
root.
Pass mirrorRoot as argument to WriteItemToDisk()

So that WriteItemToDisk() doesn't need to care about configuration; this
should be handled in `main.go`.
Rename channel variables to be more descriptive

So that's it clear that these are channels.