Completed scraper for Department of Defense IG.

http://www.dodig.mil/pubs/index.cfm

Some interesting things about this scaper:

For each topic area/office, it first downloads the first page of results which contains links to all pages of results, then yields the urls to the main download loop.

It skips reports marked "Internal Use Only" or "Classified" because there is no PDF for those.

It attempts, somewhat successfully, to extract summary text from the summary pages for reports