Repository: matrix-org/synapse

Issue:
`trial` test runs leak memory
#15334 introduced a bug where `HomeServer` instances no longer get garbage collected during `trial` test runs. As a result, memory usage grows to multiple gigabytes during test runs.

See https://github.com/matrix-org/synapse/pull/15334/files#diff-f50ec854a65082453f4086048f20c291ca6020fab973486a9da2285025394cc8R180.
When we garbage collect generation 0, `self.hs` still holds a reference to the `HomeServer`, which gets promoted to generation 1 and then never collected.

Apart from that, there are also a couple of other leaks at play.

---

- [x] Fix `HomeServer`s leaking due to not being GCed.
      https://github.com/matrix-org/synapse/pull/15630
- [x] Fix `FederationRateLimiter` (and referenced `HomeServer`s) leaking due to `_rate_limiter_instances` global.
      https://github.com/matrix-org/synapse/pull/15630
- [x] Fix config cache added in #15284 growing too large (~360 entries). `HomeServerConfig`s are pretty heavy and hold on to a lot of stuff, including Jinja templates. A suitable max size for the cache might be 8-16 entries.
      https://github.com/matrix-org/synapse/pull/15646
- [ ] Fix `HomeServer`s leaking via [synapse.util.caches.caches_by_name](https://github.com/matrix-org/synapse/blob/9bb2eac71962970d02842bca441f4bcdbbf93a11/synapse/util/caches/__init__.py#L39).
- [ ] Fix `HomeServer`s leaking via [synapse.util.caches.CACHE_METRIC_REGISTRY](https://github.com/matrix-org/synapse/blob/9bb2eac71962970d02842bca441f4bcdbbf93a11/synapse/util/caches/__init__.py#L37).
- [ ] Fix `HomeServer`s leaking via [synapse.config.cache._CACHES](https://github.com/matrix-org/synapse/blob/796a4b74823b721c72de07e45718f05e78e1565d/synapse/config/cache.py#L34).

The last 3 are fairly annoying to fix, so I've left them alone.


Discussion:
When I force a full GC every 50 or so gen0 GCs, a trial run no longer OOMs on my laptop. But memory usage still keeps growing and the GCs get slower and slower. So there's a separate memory leak going on.

According to tracemalloc, the most leaked objects are
```
    /home/squah/repos/synapse/synapse/util/caches/lrucache.py:483: size=26.2 MiB, count=217769, average=126 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/jinja2/environment.py:704: size=23.5 MiB, count=183329, average=134 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/jinja2/environment.py:1299: size=16.5 MiB, count=1515, average=11.2 KiB
    /home/squah/repos/synapse/synapse/storage/database.py:469: size=11.2 MiB, count=140743, average=83 B
    /usr/lib/python3.10/functools.py:58: size=10.3 MiB, count=168891, average=64 B
    /usr/lib/python3.10/functools.py:52: size=8257 KiB, count=93189, average=91 B
    /usr/lib/python3.10/linecache.py:137: size=6795 KiB, count=67319, average=103 B
    /home/squah/repos/synapse/synapse/util/caches/lrucache.py:759: size=4862 KiB, count=8643, average=576 B
    /home/squah/repos/synapse/synapse/storage/background_updates.py:547: size=4747 KiB, count=42392, average=115 B
    /home/squah/repos/synapse/synapse/config/_base.py:168: size=4571 KiB, count=55214, average=85 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/jinja2/environment.py:349: size=4557 KiB, count=4106, average=1137 B
    /home/squah/repos/synapse/synapse/util/caches/deferred_cache.py:108: size=4185 KiB, count=105808, average=40 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/twisted/internet/base.py:114: size=3669 KiB, count=12692, average=296 B
    /home/squah/repos/synapse/synapse/util/caches/lrucache.py:502: size=2971 KiB, count=25931, average=117 B
    /home/squah/repos/synapse/synapse/storage/database.py:192: size=2807 KiB, count=35984, average=80 B
    /usr/lib/python3.10/unittest/mock.py:415: size=2564 KiB, count=5466, average=480 B
    /home/squah/repos/synapse/synapse/storage/background_updates.py:509: size=2560 KiB, count=19521, average=134 B
    /usr/lib/python3.10/functools.py:61: size=2388 KiB, count=14553, average=168 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/jinja2/environment.py:350: size=2359 KiB, count=4108, average=588 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/twisted/internet/defer.py:461: size=2306 KiB, count=29910, average=79 B
```
but this isn't too helpful in tracking down the leak.

Resolving patch:
diff --git a/synapse/util/ratelimitutils.py b/synapse/util/ratelimitutils.py
--- a/synapse/util/ratelimitutils.py
+++ b/synapse/util/ratelimitutils.py
@@ -25,10 +25,12 @@
     Iterator,
     List,
     Mapping,
+    MutableSet,
     Optional,
     Set,
     Tuple,
 )
+from weakref import WeakSet
 
 from prometheus_client.core import Counter
 from typing_extensions import ContextManager
@@ -86,7 +88,9 @@
 )
 
 
-_rate_limiter_instances: Set["FederationRateLimiter"] = set()
+# This must be a `WeakSet`, otherwise we indirectly hold on to entire `HomeServer`s
+# during trial test runs and leak a lot of memory.
+_rate_limiter_instances: MutableSet["FederationRateLimiter"] = WeakSet()
 # Protects the _rate_limiter_instances set from concurrent access
 _rate_limiter_instances_lock = threading.Lock()
 


Test patch:
diff --git a/tests/unittest.py b/tests/unittest.py
--- a/tests/unittest.py
+++ b/tests/unittest.py
@@ -229,13 +229,20 @@ def tearDown(orig: Callable[[], R]) -> R:
         #
         # The easiest way to do this would be to do a full GC after each test
         # run, but that is very expensive. Instead, we disable GC (above) for
-        # the duration of the test so that we only need to run a gen-0 GC, which
-        # is a lot quicker.
+        # the duration of the test and only run a gen-0 GC, which is a lot
+        # quicker. This doesn't clean up everything, since the TestCase
+        # instance still holds references to objects created during the test,
+        # such as HomeServers, so we do a full GC every so often.
 
         @around(self)
         def tearDown(orig: Callable[[], R]) -> R:
             ret = orig()
             gc.collect(0)
+            # Run a full GC every 50 gen-0 GCs.
+            gen0_stats = gc.get_stats()[0]
+            gen0_collections = gen0_stats["collections"]
+            if gen0_collections % 50 == 0:
+                gc.collect()
             gc.enable()
             set_current_context(SENTINEL_CONTEXT)