unitedstates/inspectors-general — pull request #14 by konklone (2014-05-09 22:06:29+00:00)
Title: A runner script, and a file to track which scrapers are 'safe'

This adds a top-level runner script, `./igs`, which can automatically run multiple IG scrapers in turn.

* Running `./igs` by default will run each scraper in `/inspectors` in turn.
* Running `./igs --safe` will limit it to the scrapers whose names appear in `safe.yml`.
* Running `./igs --only=[igs]` where `[igs]` is a comma-separated list of IGs, e.g. `opm,epa`, will limit it to only those scrapers.
* All other command line flags will be passed directly to the individual scrapers.

Unfortunately, this does mean `--safe` and `--only` are no longer safe flags for a scraper to accept. I updated `usps.py` and `epa.py` to rename their `--only` flags to different name.

One ramification: all scrapers now need to change their `utils.run(run)` line to the more verbose:

```python
utils.run(run) if (__name__ == "__main__") else None
```

Annoying, but worth the tradeoff, even if we weren't doing a runner script. This allows scrapers to be imported as modules without auto-triggering a download. This check could have been built into the `utils.run` method itself, but that would have had to use more hacky detection, and it felt cleaner overall this way.