Repository: matrix-org/synapse

Issue:
CacheDecoratorTestCase.test_double_get fails on Python 3.8: 4 != 3
### Description

There is a test failure when synapse 1.4.0 is built with Python 3.8.

### Steps to reproduce

- Run the tests using Python 3.8: `PYTHONPATH=. trial-3 tests`

The test suite fails:
```
[...]
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/usr/lib64/python3.8/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/builddir/build/BUILD/synapse-1.4.0/tests/storage/test__base.py", line 308, in test_double_get
    self.assertEquals(a.func2.cache.cache.pop.call_count, 3)
  File "/usr/lib64/python3.8/site-packages/twisted/trial/_synctest.py", line 432, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/lib64/python3.8/unittest/case.py", line 912, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib64/python3.8/unittest/case.py", line 905, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 4 != 3

tests.storage.test__base.CacheDecoratorTestCase.test_double_get
-------------------------------------------------------------------------------
Ran 780 tests in 194.729s

FAILED (skips=1, failures=1, successes=778)
```

### Version information

- **Version**:
1.4.0

- **Install method**:
running tests from the source tree

- **Platform**:
Fedora rawhide (to become Fedora 32) which includes Python 3.8



Discussion:
We don't currently support python 3.8, so I'm going to go ahead and close this, but if you'd like to submit a PR to fix this, that would be great.

Resolving patch:
diff --git a/synapse/util/caches/descriptors.py b/synapse/util/caches/descriptors.py
--- a/synapse/util/caches/descriptors.py
+++ b/synapse/util/caches/descriptors.py
@@ -17,8 +17,8 @@
 import inspect
 import logging
 import threading
-from collections import namedtuple
-from typing import Any, cast
+from typing import Any, Tuple, Union, cast
+from weakref import WeakValueDictionary
 
 from six import itervalues
 
@@ -38,6 +38,8 @@
 
 logger = logging.getLogger(__name__)
 
+CacheKey = Union[Tuple, Any]
+
 
 class _CachedFunction(Protocol):
     invalidate = None  # type: Any
@@ -430,7 +432,7 @@ def _wrapped(*args, **kwargs):
             # Add our own `cache_context` to argument list if the wrapped function
             # has asked for one
             if self.add_cache_context:
-                kwargs["cache_context"] = _CacheContext(cache, cache_key)
+                kwargs["cache_context"] = _CacheContext.get_instance(cache, cache_key)
 
             try:
                 cached_result_d = cache.get(cache_key, callback=invalidate_callback)
@@ -624,14 +626,38 @@ def errback(f):
         return wrapped
 
 
-class _CacheContext(namedtuple("_CacheContext", ("cache", "key"))):
-    # We rely on _CacheContext implementing __eq__ and __hash__ sensibly,
-    # which namedtuple does for us (i.e. two _CacheContext are the same if
-    # their caches and keys match). This is important in particular to
-    # dedupe when we add callbacks to lru cache nodes, otherwise the number
-    # of callbacks would grow.
-    def invalidate(self):
-        self.cache.invalidate(self.key)
+class _CacheContext:
+    """Holds cache information from the cached function higher in the calling order.
+
+    Can be used to invalidate the higher level cache entry if something changes
+    on a lower level.
+    """
+
+    _cache_context_objects = (
+        WeakValueDictionary()
+    )  # type: WeakValueDictionary[Tuple[Cache, CacheKey], _CacheContext]
+
+    def __init__(self, cache, cache_key):  # type: (Cache, CacheKey) -> None
+        self._cache = cache
+        self._cache_key = cache_key
+
+    def invalidate(self):  # type: () -> None
+        """Invalidates the cache entry referred to by the context."""
+        self._cache.invalidate(self._cache_key)
+
+    @classmethod
+    def get_instance(cls, cache, cache_key):  # type: (Cache, CacheKey) -> _CacheContext
+        """Returns an instance constructed with the given arguments.
+
+        A new instance is only created if none already exists.
+        """
+
+        # We make sure there are no identical _CacheContext instances. This is
+        # important in particular to dedupe when we add callbacks to lru cache
+        # nodes, otherwise the number of callbacks would grow.
+        return cls._cache_context_objects.setdefault(
+            (cache, cache_key), cls(cache, cache_key)
+        )
 
 
 def cached(


Test patch: