BBC-News/alephant-broker — pull request #37 by stevenjack (2014-12-19 09:50:00+00:00)
Title: [SPIKE] Stale while revalidate

### Problem

Currently the request cycle is governed and reliant on the speed of the request being made to the renderer, this can cause issue when the service is slow or has intermittent issues and can cause the broker to end up hammering the renderer with a number of requests.

### Solution

Follow a `stale-while-revalidate` and `stale-while-error` model, so we change the cache object's ttl to be much higher such as a week/month then store an object inside in the cache which has it's own ttl and update time. This allows us to check this when pulling the item out of the cache, and if it's expired then thread off and do the update. All other subsequent requests still serve out of the cache until the thread has got the new content and updated the cache.

We also have a `validating` property that and `last_validated` so that once the thread is off re-validating the content we don't have lot's of other threads being created to do that same work.

Also the `last_validated` variable allows the operation to timeout, as the thread may have died or not completed for some reason so we don't want the content to remain stale there after.