openeventdata/scraper — pull request #47 by ahalterman (2014-05-22 03:52:03+00:00)
Title: Add international/local/wire label to sources

As we kept adding more local sources, we also kept adding to the amount of junk and noise that could appear in the dataset. While local sources give us better micro coverage of events that are interesting to people studying a specific country, they bring three downsides:
- stories written in styles besides the standard wire format that event coders are written for.
- references to specific people, locations, and offices that are not in the actor dictionaries that probably require country-specific dictionaries
- very, very local events that are not of interest to most users of the dataset and can cause confusion and cruft.

To have it both ways, I labeled each source as either wire, international, or local (see the README), allowing us to have separate datasets designed for either comprehensiveness or cleanness. The categories for each news source are debatable and people who have different ideas are welcome to fork and pull request.

I'll hold off merging this until John confirms that the extra column won't destroy everything.