unitedstates/inspectors-general — pull request #48 by spulec (2014-06-05 13:03:48+00:00)
Title: Add DOT Scraper

The biggest issue with this one is that a large number of the "reports" don't actually have reports.

I ran `./igs --only=dot --since=1996 --dry_run --debug --topics=T,SA` with success so I feel good about the general structure of the scraper.

All of the other topics appear to have some landing pages that don't link to reports.

Examples:
http://www.oig.dot.gov/library-item/6287
http://www.oig.dot.gov/library-item/5061
http://www.oig.dot.gov/library-item/6552
http://www.oig.dot.gov/library-item/2784
http://www.oig.dot.gov/library-item/2786

Example with report: http://www.oig.dot.gov/library-item/6432

I started a blacklist, but it was over 100 items, so I stopped. We can just mark all of these as `missing` if the report url isn't there, but that obviously has downsides. Thoughts?