edsu/twarc — pull request #21 by hugovk (2014-10-21 11:18:14+00:00)
Title: Utils to dedupe and sort by ID (date)

Two extra utilities:

* `twarc.py --scrape` resulted in some duplicate tweets. `deduplicate.py` reduced it from 945 to 734. Use like: `utils/deduplicate.py tweets.json > tweets_deduped.json`

* The output of `twarc.py --scrape` was roughly in reverse chronological order, but not exactly. `sort_by_id.py` sorts by ID, which is analogous to sorting by time. Use like: `utils/sort_by_id.py tweets.json > sorted.json`