unitedstates/inspectors-general — pull request #176 by divergentdave (2014-10-17 23:17:33+00:00)
Title: Unique report IDs - Part 2

This branch is a WIP for correcting cases where the same report_id is used for two different reports that fall in the same year, and thus can't be caught by the QA scripts. The first two commits add validation, everything else is tweaks to scrapers. I've taken some of the easy ones already, but there is still plenty to do.

Changes to scrapers tend to be
* Compensating for typos on the websites
* Taking care of edge cases in generating the report_id
* Only scraping each report once, even if they somehow appear on multiple lists/pages

- [ ] agriculture
- [x] arc
- [ ] commerce
- [x] cncs
- [ ] dhs
- [ ] dod
- [ ] dot
- [ ] education
- [ ] energy
- [x] epa
- [ ] exim
- [ ] fdic
- [ ] gpo
- [ ] hhs
- [ ] hud
- [ ] interior
- [x] itc
- [ ] labor
- [ ] loc
- [ ] nasa
- [ ] neh
- [x] nrc
- [x] pbgc
- [ ] ssa
- [ ] treasury
- [x] tigta
- [ ] usaid
- [ ] va

Here's a copy of the list of duplicates I'm working from https://gist.github.com/divergentdave/d520271903ebf8f02776