disqus/django-modeldict — pull request #6 by lyudmildrx (2014-12-18 10:49:54+00:00)
Title: Update CachedDict

Update _local_last_updated only after an actual update takes place.
In the case where updating from remote cache fails, fallback population
from source would trigger and update _local_last_update with correct data.

Motivation behind PR:
In Delivery Hero we are experiencing some weird gargoyle switches behavior. After changing the status of a feature switch about 50% of request would behave as if switch status did not change and the number of 'bad' request would slowly go down to 0. 'Bad cache' seems to be triggered and continuously observed only if there is significant server load. I verified that higher server load leads to faster flush of bad cache and linked it to the reload of uwsgi processes. I also verified that johnny-cache does not cause this, so I turned to gargoyle itself and found a possible bug in local cache implementation in modeldict.base.CachedDict. After some meditation, I saw that under constant load there is a chance that CachedDict would never mark its local cache as invalid.

Testing done on the changes:
We have not done testing in real environments.
For local testing, I would run a few django shells and run spammer (from https://github.com/lyudmildrx/django-modeldict/commit/7b3e0f57390e2acb76da08b4f2a4c2274e9d3c81) in each of them. I would then change the feature switch value and observe the running shells. Some of the shells would continue looping thinking that the feature switch status did not change.
With changes from PR, none of the shells continue looping.