jdswinbank/Comet — Issue #1: Pruning the IVORN database
Author: jdswinbank
Created: 2012-02-09 10:32:34+00:00
State: open

When a new event is received, we store its IVORN so that we can use it for de-duplication purposes: we shouldn't process the same event twice.

Currently the IVORNs are stored in a series of [anydbm](http://docs.python.org/library/anydbm.html) databases, which will grow without limit. When a large number of events are received:

1. These databases will get extremely large;
2. Searching the database for previously seen IVORNs will get very expensive.

The simplest way to avoid this is simply to expire IVORNs from the database after some time (a month?). Alternatively, we could investigate a more sophisticated database system which works around the issues.