Set user agent for crawler

Set the crawler's HTTP user agent to "GOV.UK Crawler" and add a test for
this.
Reuse HTTP client

Previously, every request would instantiate a new HTTP client.

Use `http.DefaultClient`[1] instead, which is the default client that Go
uses for HTTP requests, e.g. `http.Get` uses it[2].

[1]: http://golang.org/src/pkg/net/http/client.go?#L74
[2]: http://golang.org/src/pkg/net/http/client.go?#L251
Change user agent to 'GOV.UK Crawler Worker'

Which better reflects the name of this project.
Add hostname to user agent

So that we know which server the crawler is running on.

Only check for the string "GOV.UK Crawler Worker" in our tests as it's
not easy for us to mock the system hostname and testing against
`os.Hostname()` would be of minimal benefit.

Also removes the function that trims the body of the HTML response, as
there's no reason for there to be spaces around the text in the response
body.
Rename testServer for user agent test

The function signature for the httptest.Server we initiate here is
different to the `testServer` that precedes it, so use a different
variable name to make it clear that this test server behaves
differently, i.e. it puts the user agent into the response body.
Split HTTP server handler into own variable

To improve readability and to pave the way for passing in a handler to
this function if necessary in future.
Merge pull request #2 from alphagov/73455918-add-user-agent

[#73455918] Set user agent for crawler
Add REPO_PATH variable to Makefile

To reduce repetition of the repo URL. I'm about to use this for something
else too.
Symlink self into third_party/src

So that we don't clone a new copy of our own code and ignore local changes
when building/testing. This is what most of the CoreOS projects (where
third_party.go originated from) do:

https://github.com/coreos/etcd/blob/v0.4.4/build#L3-6

Should fix a problem that @rjw1 and @mattbostock experienced.

make(1) won't re-run the commands if the symlink already exists because the
target name matches the "file". However you will need to delete your
existing directory if you've already used make in the past.
Merge pull request #3 from alphagov/third_party_symlink_self

Symlink self into third_party/src
Move getEnvDefault to a util package

So that we can re-use it for tests. I did consider moving the settings
variables themselves, but I think it makes sense to confine them to the main
function/package, at the cost of duplicating the defaults in the tests.
Use AMQP_ADDRESS and REDIS_ADDRESS in tests

So that I can run the integration tests against Redis and RabbitMQ running
in virtual machines rather than my host machine e.g.

    REDIS_ADDRESS="10.3.0.50:6379" AMQP_ADDRESS="amqp://test:test@10.3.0.70:5672" make
Fix typo in test name

s/AMQL/AMQP/
Merge pull request #4 from alphagov/tests_amqp_redis_addresses

Use AMQP_ADDRESS and REDIS_ADDRESS in tests
DRY up CrawlerMessageItem tests

DRY up the CrawlerMessageItem tests by defining the host and URL
fixtures up front.

Also, add sub-page to the example URL we test with so that we can make
use of it in later tests.
`go fmt` third_party.go
Write crawled items to disk

Write crawled HTML content to disk by extending the CrawlerMessageItem
interface with a new method, WriteToDisk(), and its supporting method,
FileName().

Use the 'github.com/kennygrant/sanitize' package to sanitise the file
path and filename. Note that the function used, `sanitize.Path()`, is
actually intended for sanitising URLs rather than filenames, but serves
our purpose well because it runs `filepath.Clean()`[1] and also leaves
the base path (i.e. '/government/organisations/') intact. The function
from that same package, `sanitize.Name`, uses `path.Base()` to strip
this out and so is not useful for our case. Should the behaviour of
`sanitize.Path()` change in future, our tests should pick this up.

Adds a new `mirrorRoot` configuration variable which _must_ be set as no
default is defined to prevent the crawler from writing to disk without
this being explicitly set.

Files are written according to their URL path, so the URL
`https://www.gov.uk/government/organisations` will be written as
`/government/organisations.html` inside `mirrorRoot`. Directories are
created as necessary.

Note that by calling `WriteToDisk()` from the `CrawlURL` function in
`workflow.go`, writes to disk are tightly coupled with the act of
crawling. Given the slow speed at which we crawl (1-2 pages per second),
I don't think this is worth addressing now, but we may in future
consider moving this logic to a separate worker which could consume
crawl items from a queue and write them to disk.

[1]: http://golang.org/pkg/path/filepath/#Clean
Test WriteToDisk() and FilePath() methods

Test the new WriteToDisk() and FilePath() methods on the
CrawlerMessageItem interface.

Move some of the fixture variables inside BeforeEach() so that we can
re-use them across tests.

We also now populate `item.HTMLBody` by default as the new FilePath()
and WriteToDisk() methods will only act on content it detects as HTML.

The test for WriteToDisk() will actually write to disk inside the
directory specified by the MIRROR_ROOT environment variable, which must
be set. I don't think this is an issue given that many of the other
tests are also integration tests, i.e. they rely on networking to test
crawling.

Mocking in Go is also non-trivial. Testify's `mock`
package[1] looks like a nice solution but I don't think it's necessary
in this case.

Also add a test for the URL() method on that interface.

[1]: https://github.com/stretchr/testify#mock-package
Rename fileName, FileName to filePath, FilePath

To be more specific.
Test that caps in URLs are converted to lower case