sunlightlabs/alarms — Issue #66: Caching layer in front of searches
Author: klondike
Created: 2011-11-08 15:41:27+00:00
State: open

This will be more and more important as we add more data sources, especially if we add one that we can't control (like FederalRegister.gov).

* Create a table to cache searches per-subscription_type-per-keyword
- unit is a search, with that joint key (indexed) and then the items contents
* rake command (and cap command) to flush all caches - would need to be done whenever the structure of data changes
* cron task to flush each type of cache, different rate for each type
* Put a cache TTL (or no TTL) on each subscription_type so that even if the cache fails to get flushed, it will be ignored if too old
- fetch all cached searches at once for the given keyword, hash it by subscription_type

Also:
* needs to be easy to turn off TTLs in development.
* proper error handling on the frontend when an API fails or hangs, with no cache to fall back on