Log to stderr rather than stdout

We don't collect application logs on stdout using logstash, just on
stderr.
Merge pull request #66 from alphagov/log-to-stderr

Log to stderr rather than stdout
Make the `EXPIRE` time for ttl_hash_set configurable through the ENV

Instead of making the expiry time a constant, allow it to be injected
using configuration. This can then be later owned by the environment
that's passed in to be more in keeping with the 12-factor style apps
we try to write.
Inject the `EXPIRE` time for ttl_hash_set using an ENV variable

Read the expire time from the environment variable that we define and
pass it through the call to `NewTTLHashSet`.
Change a raw `Expect` test to poll asynchronously with `Eventually`

This test is flaky as it's only called once to check the value. It
needs to be polled over time as it can take up to a second for Redis
to acknowledge the write.
[#76800748] Allow outputting logs in JSON format

This change allows us to output the logs in JSON rather than plain
text. An application flag has been used rather than changing it over
directly so that we can gracefully move our infrastructure over to use
it.
Blow up with a fatal error if we can't parse `TTL_EXPIRE_TIME`
Be consistent and use `time.Hour` for test `EXPIRE` time
Merge pull request #67 from alphagov/allow-configuring-the-ttl-hash-set-ttl-period

Allow configuring the ttl_hash_set EXPIRE period using the environment
Merge pull request #69 from alphagov/76800748-log-out-as-json

[#76800748] Allow outputting logs in JSON format
Merge pull request #68 from alphagov/change-direct-expects-test-to-be-async

Change a raw `Expect` test to poll asynchronously with `Eventually`
Rename `HTMLBody` field to `ResponseBody`

We're going to support more MIME types than just `text/html` now, so
we should generalise the field.
Wrap the HTTP Response in a `CrawlerResponse` object

This can then hold both the Body and any HTTP headers we want to pass
around and reuse.
`CrawlerResponse` object owns whether or not the body `isHTML`

This is now owned by the respective object as it has ownership over
the HTTP Body response.
Rename `IsHTML` method to `IsBodyHTML`

The `CrawlerResponse` now holds both the header and footer for a given
HTTP response so we can check the actual Content-Type rather than
just matching between `plain`, `html` and `octet` text strings.
`ContentType` returns the content type as a string

This removes any of the extra information that usually litters the
Content-Type for our various different application stacks as well.
Add a list of constants for the content types we're supporting

For now, these will be the content types that we'll accept and write
to disk.
A response can tell whether it holds an `AcceptedContentType`

To make it easier to tell whether or not a given response is supported
add a helpful method on the object that can tell us quickly if that's
the case.
Improve logging for `WriteItemToDisk` workflow function

Rather than using the same log message for, make them specific.
Set Content-Type HTTP header and remove `expectedFilePath` variable

Update these tests for the `item.RelativeFilePath()` so that they
still behave as we expect before we start to manipulate the underlying
code to support multiple file formats.

Also removed the `expectedFilePath` variable as it was confusing to
follow. It didn't make sense to keep on changing this single variable
when the reference could just as easily be kept inlined. This is now
the case so the tests are more self contained.