Queens-Hacks/qcumber-scraper — pull request #23 by pR0Ps (2014-11-29 18:29:43+00:00)
Title: Automated testing

We have different Python versions (2.7, 3.3, 3.4, etc), as well as different parsing libraries (lxml, html5) to support. It's pretty much impossible to manually test all the combinations so we should create a test suite to do it for us.

I'm thinking that we could dump a few different examples of each page type (letter, expanded subject dropdown, course, term, etc), into local files then make some tests that open them and make sure that we can parse everything out of them properly using the scraper.

I'm thinking we should have a file that specifies which subjects, courses, etc that we want to test, as well as an updater script to deal with updating them.

The updater script could use the scraper to grab the pages and store the HTML of them locally, as well as store the actual scraped data. Tests run against the data could test their output and make sure it's correct. This way we can keep track of which pages we're testing against, as well as have an easy way to keep them all updated. We could then add problematic pages as we come across them (Ex: the CISC subject, see #18 ) to prevent regressions.

Some work will have to be done to the scraper to make sure that we can pull out individual courses by name (not id). The rest of the types (subject, section, etc) already pull enough attributes out of the page that we can use to specify them by name, but the `all_courses` function only returns the `_unique` of the course, not anything non-volatile (like the course number). I think this was done just to avoid duplicating information that would be scraped out of the course page (once it was loaded), not for any technical reason so it should be relatively easy to change.

Obviously if the SOLUS html changes and scraper breaks the updater script won't be able to update the stored pages and scraped data, but in those cases we can manually update the files for local testing, then when the problem is fixed, run the updater again.

I don't know if we want to actually store the HTML files in the repo or not as we might run into legal issues (putting private data in a public Git repo).

A huge benefit of this is that we won't need to actually touch SOLUS except when updating the local tests, fixing authentication issues, or actually scraping. This will speed up development, as well as allow us to not hammer SOLUS (and risk getting banned) while testing.

Anyway, this is just a huge ideas dump, let me know your thoughts on it.