unitedstates/inspectors-general — pull request #41 by audiodude (2014-05-27 03:59:13+00:00)
Title: Department of Energy IG scraper

The interesting things here is that we scrape out a "sub_type" for the report based on things like:

"Special Inquiry: DOE/IG-0910" -- 'special_inquiry'
and
"Management Alert: DOE/IG-0906" -- 'management_alert'

These didn't seem like topics, offices, or categories, so that's why I made them sub types.

This is an IG site that includes landing pages and summaries, which this scraper extracts.

We also detect reports that are unreleased by checking if their summary pages for 'report is classified' and 'not available for viewing', because sometimes classified reports from this IG have a stub PDF that simply says "This report is classified. For information on.....".

For some categories of reports ('topics') we use a report 'type' other than report. These are listed in the code under ADDITIONAL_TOPICS.

Also, this is the first scraper that is a class. I was sick of passing around options and year_range and the like, mainly. But I also genuinely needed to set the state of the class in one method that was used by a higher up method, for the report_type.