mjibson/goread — pull request #51 by russross (2013-06-28 19:14:18+00:00)
Title: vary the update frequency for feeds

Rather than checking each feed on a fixed schedule, this patch varies the frequency of checks based on the history of that feed. There are a few parts:

1. Add an Average field to Feed. This tracks the average interval that we have observed between updates for the feed.

2. Update the Average field whenever new stories are found. The old average is merged with the new interval in such a way that the importance of older intervals gradually decays, so new data points are more important than older ones. This is computed based on the observed delay, not the timestamps in the feeds and stories themselves, since they are so unreliable. Precision doesn't matter here anyway, a rough estimate is good enough.

3. Schedule the next check for a feed by waiting some fraction of the average update interval for that feed. This is bounded on the high and low end to ensure sanity. Also, after a while with no updates, the interval is gradually increased as well. This prevents frequently-updated sources that suddenly stop updating from being checked obsessively forever.