alphagov/govuk_crawler_worker — pull request #64 by KushalP (2014-08-12 10:51:59+00:00)
Title: [#75962418] Write non-HTML content to disk

This adds support for writing the following file types to disk:
- ATOM
- CSV
- DOCX
- ICS
- JSON
- ODP
- ODS
- ODT
- PDF
- XLS
- XLSX

Created a new `http_crawler.CrawlerResponse` object that will hold the body and headers for the HTTP response. It then provides some helper methods around `Content-Type` values that we support that are used elsewhere in the workflow for the function `WriteItemToDisk`.