Repository: learningequality/kolibri

Issue:
Lingering Redis locks from previous dirty shutdowns can cause locks during sign-in that result in errors or timeouts
<!--
Instructions:
 * Fill out the sections below, replace …'s with information about your issue
 * Use the 'preview' function above this text box to verify formatting before submitting
-->

### Observed behavior
<!--
Description of the behavior that was observed, including screenshots or other references when applicable
-->
When using Redis, if Kolibri has an open lock while it's forced to shut down, such that the lock isn't released, then Kolibri will run into issues acquiring locks when it is restarted.

### Expected behavior
<!--
Description of what behavior was expected but did not occur
-->
- Ensure all appropriate defensive code is in place to prevent lingering locks in Redis
- Lingering locks should TBD, options for consideration:
  1. Prevent Kolibri from fully starting without informing the user. This may be appropriate because depending on the lingering lock type, the database could be in an incomplete state, so it would give the user a chance to make a backup. Then perhaps startup could proceed with confirmation from the user
  2. Add a management command to clear locks
  2. Always clear locks on startup
  3. ???

### User-facing consequences
<!--
Implications and real-world consequences for learners, coaches, admins, and other users of the application
-->
Being unable to acquire locks, Kolibri requests may hang, timeout, or raise an error. Depending on cache efficiency, authentication may be impossible for some users.

### Errors and logs
<!--
Relevant logs from:
 * the command line
 * ~/.kolibri/logs/kolibri.txt
 * the browser console

Please wrap errors in triple backticks for clean formatting like this:
```
01:10 info: something happened
01:12 error: something bad happened
```
-->
```
ERROR 2020-08-10 17:18:56,981 django.request Internal Server Error: /api/auth/session/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/rest_framework/viewsets.py", line 116, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/utils/decorators.py", line 67, in _wrapper
    return bound_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/utils/decorators.py", line 63, in bound_func
    return func.__get__(self, type(self))(*args2, **kwargs2)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "./kolibri/core/auth/api.py", line 577, in create
    login(request, user)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/contrib/auth/__init__.py", line 161, in login
    user_logged_in.send(sender=user.__class__, request=request, user=user)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/dispatch/dispatcher.py", line 193, in send
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/contrib/auth/models.py", line 25, in update_last_login
    user.save(update_fields=['last_login'])
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/django/contrib/auth/base_user.py", line 80, in save
    super(AbstractBaseUser, self).save(*args, **kwargs)
  File "./kolibri/core/auth/models.py", line 219, in save
    self.ensure_dataset()
  File "./kolibri/core/auth/models.py", line 231, in ensure_dataset
    inferred_dataset_id = self.infer_dataset(*args, **kwargs)
  File "./kolibri/core/auth/models.py", line 694, in infer_dataset
    return self.cached_related_dataset_lookup("facility")
  File "./kolibri/core/auth/models.py", line 190, in cached_related_dataset_lookup
    dataset_id = dataset_cache.get(key)
  File "./kolibri/core/utils/cache.py", line 124, in get
    return self.cache.get(self.make_key(key), *args, **kwargs)
  File "./kolibri/core/utils/cache.py", line 70, in __exit__
    self.release()
  File "./kolibri/core/utils/cache.py", line 64, in release
    self._lock.release()
  File "/usr/local/lib/python2.7/dist-packages/kolibri/dist/redis/lock.py", line 224, in release
    raise LockError("Cannot release an unlocked lock")
LockError: Cannot release an unlocked lock
```

### Steps to reproduce
<!--
Precise steps that someone else can follow in order to see this behavior
-->
1. Set up Kolibri to use Redis
2. Open Kolibri shell, and run:
```
from kolibri.core.utils.cache import process_cache
lock = process_cache.lock(
    "namespaced_cache_dataset",
    timeout=2000,  # milliseconds
    sleep=0.01,  # seconds
    blocking_timeout=5,  # seconds
    thread_local=True,
)
lock.acquire()
```
3. Close the shell
4. Kolibri will encounter crippling issues

### Context
<!--
Tell us about your environment, including:
 * Kolibri version
 * Operating system
 * Browser
-->
Kolibri 0.14



Discussion:
For the hosted kolibri instance use-case, is it possible to acquire the lock with a certain lease duration? I.e. the lock expires after a certain number of seconds, even if the lock hasn't been released yet.
@aronasorman We can definitely add an expiry. That would be a good change, and it would tie in nicely with acquiring all the locks during start up.
Presumably we'd also need to add handling for edgiest case of the lock being released after the expiry too?
Ha yeah that's definitely a case we may want to handle as well, esp. for the case where a process may still edit the cache even if the lock has expired. The general problem here is when 2 processes think they have the lock.

For that specific problem @rtibbles, I think we can just do a try-catch when the lock has already been released? I think that should be fine. Let me know if there's a better solution for that.

For the general problem though, it'll require a lot more changes to make sure that processes with older locks don't accidentally edit the cache when a new lock lease has been acquired.
Follow-up: we encountered this issue again today in [another incident](https://learneq.pagerduty.com/incidents/P4B4SVI).

One thing I've noticed is that the app waits for 60 seconds or more for the lock to be released before an unspecified timeout hits. So as long as the lock lease is a fraction of this, then the lock lease solution should mitigate these incidents.
This happened again with the main demo site. This was last deployed on August 18th.
@aronasorman I see our configuration check isn't working on the demo server. I see the following in the logs:
```
WARNING  Unable to check Redis settings
WARNING  unknown command `CONFIG`, with args beginning with: `GET`, `maxmemory-policy`
```

Where are the settings for the demo server?
@bjester it seems it's happening without restarting the server or upgrading kolibri too
https://www.notion.so/learningequality/Frequent-Redis-Lock-errors-on-the-demo-server-bacac3f3fd1d491c86ecac75d89a2eee
Apologies for the late reply @bjester -- we don't have any special configuration, so everything in the default values here apply: https://cloud.google.com/memorystore/docs/redis/redis-configs
This has also now been seen in the wild for the diskcache based locks too, not just for Redis.

Resolving patch:
diff --git a/kolibri/core/auth/management/commands/sync.py b/kolibri/core/auth/management/commands/sync.py
--- a/kolibri/core/auth/management/commands/sync.py
+++ b/kolibri/core/auth/management/commands/sync.py
@@ -108,6 +108,7 @@ def handle_async(self, *args, **options):  # noqa C901
             call_command("loaddata", "scopedefinitions")
 
         dataset_cache.clear()
+        dataset_cache.activate()
 
         # try to connect to server
         controller = MorangoProfileController(PROFILE_FACILITY_DATA)
@@ -204,6 +205,7 @@ def handle_async(self, *args, **options):  # noqa C901
             self.job.extra_metadata.update(sync_state=State.COMPLETED)
             self.job.save_meta()
 
+        dataset_cache.deactivate()
         logger.info("Syncing has been completed.")
 
     @contextmanager
diff --git a/kolibri/core/auth/models.py b/kolibri/core/auth/models.py
--- a/kolibri/core/auth/models.py
+++ b/kolibri/core/auth/models.py
@@ -23,13 +23,13 @@
 from __future__ import unicode_literals
 
 import logging
+from threading import local
 
 import six
 from django.contrib.auth.models import AbstractBaseUser
 from django.contrib.auth.models import AnonymousUser
 from django.contrib.auth.models import UserManager
 from django.core import validators
-from django.core.cache import cache
 from django.core.exceptions import ObjectDoesNotExist
 from django.core.exceptions import ValidationError
 from django.db import models
@@ -75,11 +75,37 @@
 from kolibri.core.device.utils import set_device_settings
 from kolibri.core.errors import KolibriValidationError
 from kolibri.core.fields import DateTimeTzField
-from kolibri.core.utils.cache import NamespacedCacheProxy
 from kolibri.utils.time_utils import local_now
 
 logger = logging.getLogger(__name__)
-dataset_cache = NamespacedCacheProxy(cache, "dataset")
+
+
+class DatasetCache(local):
+    def __init__(self):
+        self.deactivate()
+
+    def activate(self):
+        self._active = True
+
+    def deactivate(self):
+        self._active = False
+        self.clear()
+
+    def clear(self):
+        self._cache = {}
+
+    def get(self, key):
+        if self._active:
+            return self._cache.get(key)
+        return None
+
+    def set(self, key, dataset_id):
+        if self._active:
+            self._cache[key] = dataset_id
+        return None
+
+
+dataset_cache = DatasetCache()
 
 
 def _has_permissions_class(obj):
@@ -203,7 +229,7 @@ def cached_related_dataset_lookup(self, related_obj_name):
                 dataset_id = getattr(self, related_obj_name).dataset_id
             except ObjectDoesNotExist as e:
                 raise ValidationError(e)
-            dataset_cache.set(key, dataset_id, 60 * 10)
+            dataset_cache.set(key, dataset_id)
         return dataset_id
 
     def calculate_source_id(self):


Test patch: