mjibson/goread — pull request #83 by russross (2013-07-01 17:23:47+00:00)
Title: schedule feed updates dynamically based on average update frequency

Rather than checking each feed on a fixed schedule, this patch varies the frequency of checks based on the history of that feed. There are a few parts:

1. Add an Average field to Feed. This tracks the average interval that we have observed between updates for the feed.

2. Update the Average field whenever new stories are found. The old average is merged with the new interval in such a way that the importance of older intervals gradually decays, so new data points are more important than older ones. This is computed based on the observed delay, not the time stamps in the feeds and stories themselves, since they are so unreliable. Precision doesn't matter here anyway, a rough estimate is good enough.

3. Schedule the next check for a feed by waiting some fraction of the average update interval for that feed. This is bounded on the high and low end to ensure sanity. Also, after a while with no updates, the interval is gradually increased.

The process is controlled by a series of const values in utils.go:

* UpdateMin (20 minutes): the minimum time between checks
* UpdateMax (12 hours): the maximum time between checks
* UpdateDefault (3 hours): the default time, mostly for new feeds
* UpdateFraction (0.5): if a feed has a new story on average every N minutes, check for new stories every N * * UpdateFraction minutes. 0.5 means we expect to check each source about twice for each update. Setting it too close to 1 risks just missing updates on sites with scheduled updates (think daily comics) and then waiting a long time before checking again
* UpdateJitter (3 minutes): randomly vary the wait time +/- this much
* UpdateLongFactor (20): if a feed has not been updated in 20 times the normal interval, gradually start slowing down the frequency of checks
* NewIntervalWeight (0.2): a new data point for how frequently a feed is updated is given 20% weight in the overall average, and old values are compressed down to account for the other 80%.