18F/foia — pull request #87 by konklone (2014-12-29 19:01:51+00:00)
Title: Mock out HTTP requests in data tests

This uses [`vcrpy`](https://github.com/kevin1024/vcrpy) to mock out HTTP requests as YAML files in test fixtures. To regenerate cached HTML, delete the relevant YAML file, re-run the tests (with a network connection), and then commit the changes.

The tests can now be run without a network connection.

This involved some refactoring to isolate the network calls, so that they could be mocked inside the tests. `vcrpy` does not do universal monkeypatching on `requests` or `urllib`, so the request call has to be made inside the vcr block. I think the general modularization of parsing from downloading is good for the scraper, though I don't think passing in optional arguments like I am is the ideal solution. Nonetheless, I think it's the most effective way of getting this done without extensive refactoring.

Fixes https://github.com/18F/foia-hub/issues/371.