Repository: translate/pootle

Issue:
Mitigate the risk of revision counter also being in 'default' Redis cache
The revision counter goes into the 'default' cache.  This adds the risk of people clearing the default cache and also wiping out the revision counter — even if there's a way to restore it, we might want to prevent removal in the first place. This concern was raised in https://github.com/translate/pootle/issues/3500#issuecomment-73204160



Discussion:
When I first raised the situation in https://github.com/translate/pootle/issues/3500#issuecomment-73204160 I was proposing to whitelist the keys that need to be preserved when running the `clear_cache` command. On a second thought, this doesn't scale well and is error-prone.

I'd instead have a reserved `redis` named entry in `CACHES`, which would use a `DB` number different from any other named caches. This named entry would be used anywhere in the app code for persistent Redis storage. Not sure if we could/should put this in the same `DB` number as `RQ_QUEUES`, which also needs persistent storage — would love your input here @ta2-1.

At the same time, the `clear_cache` command would refuse to run [`cache.clear()`](https://docs.djangoproject.com/en/1.7/topics/cache/#basic-usage) (which uses [`flushdb()`](http://redis.io/commands/flushdb) behind the scenes) for the cache named as `redis`, thus explicitly preventing the data living there from accidental wipe outs.

@julen why do we use redis through the cache framework instead of using it directly for this?

It seems like a lot of our problems here are caused by the inherent design of the cache framework which is meant to behave _as a cache_. It has a timeout, it has automatic culling, it has clearing and flushing and wiping and killing and what not... so yeah.

Currently we have no direct way of reusing the existing Redis connection, while the caching framework provides that for free. Hint: with django-redis we'll be able to gain easy [direct client access](http://niwibe.github.io/django-redis/#_why_use_django_redis).

Let's do it then.

From my side, a more than desirable outcome for this issue would be to reduce the duplication across RQ and Redis cache backend configuration. Good news is [django-rq have us covered](https://github.com/ui/django-rq#support-for-django-redis-and-django-redis-cache) and it'll be cheap to be DRY :)

+1 for the redis DB inited as CACHE, and reusing that settings for RQ.

+1 here too.

Taking over this to ensure it'll be ready for our next release.


Resolving patch:
diff --git a/pootle/core/cache.py b/pootle/core/cache.py
--- a/pootle/core/cache.py
+++ b/pootle/core/cache.py
@@ -19,8 +19,13 @@
 # along with this program; if not, see <http://www.gnu.org/licenses/>.
 
 
+from django.conf import settings
 from django.core.cache import caches, cache as default_cache
 from django.core.cache.backends.base import InvalidCacheBackendError
+from django.core.exceptions import ImproperlyConfigured
+
+
+PERSISTENT_STORES = ('redis', 'stats')
 
 
 def make_method_key(model, method, key):
@@ -56,6 +61,17 @@ def get_cache(cache=None):
     :param cache: The name of the requested cache.
     """
     try:
+        # Check for proper Redis persistent backends
+        # FIXME: this logic needs to be a system sanity check
+        if (cache in PERSISTENT_STORES and
+            (cache not in settings.CACHES or
+            'RedisCache' not in settings.CACHES[cache]['BACKEND'] or
+            settings.CACHES[cache].get('TIMEOUT', '') != None)):
+            raise ImproperlyConfigured(
+                'Pootle requires a Redis-backed caching backend for %r '
+                'with `TIMEOUT: None`. Please review your settings.' % cache
+            )
+
         return caches[cache]
     except InvalidCacheBackendError:
         return default_cache
diff --git a/pootle/core/models.py b/pootle/core/models.py
--- a/pootle/core/models.py
+++ b/pootle/core/models.py
@@ -24,7 +24,7 @@
 from .mixins import TreeItem
 
 
-cache = get_cache()
+cache = get_cache('redis')
 
 
 class Revision(object):


Test patch:
diff --git a/tests/settings.py b/tests/settings.py
--- a/tests/settings.py
+++ b/tests/settings.py
@@ -17,7 +17,19 @@
     'default': {
         'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
         'LOCATION': 'pootle-tests'
-    }
+    },
+    # Must set up entries for persistent stores here because we have a
+    # check in place that will abort everything otherwise
+    'redis': {
+        'BACKEND': 'django_redis.cache.RedisCache',
+        'LOCATION': 'redis://127.0.0.1:6379/1',
+        'TIMEOUT': None,
+    },
+    'stats': {
+        'BACKEND': 'django_redis.cache.RedisCache',
+        'LOCATION': 'redis://127.0.0.1:6379/1',
+        'TIMEOUT': None,
+    },
 }