mysociety/za-hansard — pull request #38 by geoffkilpin (2014-02-11 22:15:40+00:00)
Title: PMG scraper updates

There are four major updates contained in this pull request:
* Changing the help test for the fetch-to-limit option so that it refers to reports instead of questions.
* Additional delays have been added (subject to discussion in the email thread). Delays have been moved to the open_url functions.
* The scraper now only logs into the PMG site when it needs to access premium content. As the scraper appears to occasionally be logged out by the site (probably because of the speed at which it is running) it logs in each time a premium page needs to be accessed, instead of maintaining a single session.
* Before the scraper would stop running when a previously seen report was processed - in effect meaning that only the first committee would be scraped. Instead the scraper now only stops processing the current committee.