9-volt/press-kit — pull request #33 by alisnic (2014-06-29 18:42:46+00:00)
Title: Use mongo to store data

Notable changes
- archive fetched pages using gzip (use 4X less disk space)
- validate pages at fetch time, so no invalid data is saved (use 1.5X less disk space)
- removed squasher, it makes no sense if we have a database
- Improve fetchers' networking fallbacks (based on a whole week of continuous testing)
- use more simple progress indicator for parsers (which do not hit the disk). Improves parsing performance
- update dataset
- `make console`
- progress bars everywhere

The main entity for in the DB is a ParsedPage, here's how it looks:
![screen shot 2014-06-28 at 9 23 51 am](https://cloud.githubusercontent.com/assets/1770351/3419991/cff22106-fe8c-11e3-8e0e-80f21432e9fc.png)

`mentions` is a hash of the form `{voronin: true, leanca: false...etc}`. mentions and total mentions are updated on demand for each ParsedPages on analyze-time

Thoughts for the future:
- maybe use map-reduce to generate the dataset? MongoDB supports that.
- create a Day DB entity, which summarises all the mentions for a given day? We can measure how the mentions for a given person evolve in a more granular manner.