djcp/linnaeus — pull request #5 by meekmichael (2014-05-14 17:48:41+00:00)
Title: Don't copy all of redis into memory

This set of changes makes us keep the sums for each category
in their own redis key, and then when we need to get the scores for
a number of tokens, we fetch only those tokens plus that sum.

The :total key is created on the fly, so that existing Linnaeus redis stores can be upgraded.

For me, my spam filter application went from 0.35 to 0.008 seconds to check a message. A pretty big difference. Scaling will also improve with large and growing databases.