mysociety/za-hansard — pull request #62 by geoffkilpin (2014-10-18 18:38:08+00:00)
Title: Import all questions

Ensures that NCOP answers are scraped (currently only National Assembly answers are scraped. This fixes #40) and makes a number of improvements to the SayIt import process:

* Imports questions without answers and ensures that new answers are handled correctly. This fixes mysociety/pombola#1120 and #29, and (I realise after already completing this) completes the work being done in the [issues/29-import-unmatched-questions](https://github.com/mysociety/za-hansard/tree/issues/29-import-unmatched-questions) branch.
* Corrects incorrect minister names so that questions and answers are placed in the correct SayIt section. This addresses part of mysociety/pombola#1492.
* Sets the correct answer date (which is not the same as the question date).
* Sets source_url for both questions and answers

A new command is created to apply corrections to existing data without changing IDs or urls. Because fo the way these are applied, the following commands should be run in order immediately after the pull request is applied (i.e. before the scraper is run automatically):

$ ./manage.py za_hansard_q_and_a_scraper --correct-existing-sayit-import --commit
$ ./manage.py za_hansard_q_and_a_scraper --run-all-steps
$ ./manage.py za_hansard_q_and_a_scraper --correct-existing-sayit-import --commit

The corresponding pull request to Pombola (link to be added) must also be applied for this to work.