Repository: matrix-org/synapse

Issue:
Synapse 1.44.0 hangs up every few minutes
<!--

**THIS IS NOT A SUPPORT CHANNEL!**
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

Ever since updating to 1.44.0 from 1.43.0, I've experienced frequent hanging from Synapse that lasts 10 seconds to 3 minutes. During this "hanging" period all requests (including client api, federation, static web page, etc) time out, with the rare request getting a response after a few seconds. It does not starve server resources at all, though it causes large Synapse CPU usage spikes up to 50% (average usage is around 5%). While this issue is not taking place, the homeserver performs well even in larger federated rooms.

<!-- Describe here the problem that you are experiencing -->

### Steps to reproduce

- Let a device in a few large federated rooms do an initial sync or wait a few minutes
- Homeserver freezes up for a little bit

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

I expect the server to not freeze up and perform normally like it does when this issue does not happen. The occasional errors I get when this happens are just a byproduct of a connection to a client timing out unexpectedly. Looking at Grafana, it all correlates to large spikes in the federation transmission loop.

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: jepcraft.ddns.net:443
<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: Synapse 1.44.0
- **Install method**: latest `matrixdotorg/synapse` Docker build
- No separate Synapse workers
- Uses both IPv4 and IPv6
<!-- examples: package manager/git clone/pip  -->

- **Platform**:
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->
- Debian GNU/Linux 10 (buster) x86_64
- Physical server that runs applications in Docker



Discussion:
I have noticed similar problems on my server since the upgrade to 1.44.0 and I think I have managed to isolate the cause to dns request storms sent by the matrix server. I could match timeout errors in the nginx proxy log to thousands of dns request in our dns server query log. During these events the dns server can receive around 4000 queries per minute.

I have mitigated the problem somewhat by installing a nameservice cache daemon on the matrix server but it still happens from time to time. Specifically when clients start to connect in the morning.

Let me know if I can provide any more info that can be of use.
Grafana screenshots shared by @Whodiduexpect in Synapse Admins yesterday:

![image](https://user-images.githubusercontent.com/5547783/136925299-164395fb-e6e3-480e-9feb-7fa159e65420.png)

![image](https://user-images.githubusercontent.com/5547783/136925312-cf1d07c8-0528-40f5-bbf5-578033674605.png)

With extra info, also from Synapse Admins yesterday:

> this spike pattern matches up with when it all breaks

> > What are the name of the background jobs that's spiking here? (i.e. what's the label associated with the purple spikes)
>
> federation transmission loop

https://matrix.to/#/!ehXvUhWNASUkSLvAGP:matrix.org/$A_cYK1J2Qlxplek3vbgPDB-JOGDy-cM1Cate7x2FTDw?via=matrix.org&via=matrix.breakpointingbad.com&via=libera.chat
Thanks for the graphs, @Whodiduexpect: it would be useful to see the graphs under "Per-block metrics" and "Federation", which should hopefully give us a better idea of exactly *what* it's doing.

Based on the symptoms reported by @Iamsomeguy my guess would be that it's related to sending out presence updates to all the servers, which is known to be very heavyweight, but I can't see why that would start being an issue in 1.44.0.
Time period is the same as the original screenshots
![image](https://user-images.githubusercontent.com/53096512/137041248-55cfd51b-2c1a-4686-8c24-3c1b930e2d56.png)
![image](https://user-images.githubusercontent.com/53096512/137041341-0fc9a294-a23c-415c-9951-1553e75eb4af.png)
![image](https://user-images.githubusercontent.com/53096512/137041446-cd9d2443-7a06-458b-9d04-1b2ad3430a74.png)
After setting up proper client and federation workers, this issue just vanished (I hope I'm not speaking too soon but it's been quite a while and it hasn't happened). So it seems whatever was happening it's an issue with the main Synapse process only and does not affect any of the workers.
![image](https://user-images.githubusercontent.com/53096512/137060278-2fa25584-16dd-4eb6-a8d3-b67cdb63f945.png)
After that larger spike a bit past 18:40 is when the workers got fully setup to take all supported client and federation requests and from there, there are only much smaller infrequent CPU spikes that don't effect the server, towards the end I was trying to deliberately cause the issue to happen.
A small update regarding my situation. Yesterday I turned off presence:
```yaml
presence:
       enabled: false
```
This has reduced the number of dns queries to about half (~2000/min for two minutes), but I still see upstream timeouts in the nginx log:
```
2021/10/13 08:11:15 [error] 11304#11304: *992382 no live upstreams while connecting to upstream, client: 1.2.3.4, server: matrix.synap.se, request: "OPTIONS /_matrix/client/r0/joined_groups HTTP/1.1", upstream: "http://localhost/_matrix/client/r0/joined_groups", host: "matrix.synap.se"
2021/10/13 08:11:16 [error] 11304#11304: *992379 no live upstreams while connecting to upstream, client: 2.3.4.5, server: matrix.synap.se, request: "OPTIONS /_matrix/client/r0/sync?filter=2&timeout=0&since=s1025296_101588561_2162_2117272_722109_53_39010_1805136_2 HTTP/1.1", upstream: "http://localhost/_matrix/client/r0/sync?filter=2&timeout=0&since=s1025296_101588561_2162_2117272_722109_53_39010_1805136_2", host: "matrix.synap.se"
```
There were 61 such messages during 90 seconds this morning.
In the homeserver log, at the same time, I see around 1500 messages like this:
```
2021-10-13 08:11:38,412 - synapse.logging.context - 70 - WARNING - federation_transaction_transmission_loop-2733 - Re-starting finished log context federation_transaction_transmission_loop-2733
2021-10-13 08:11:38,412 - synapse.logging.context - 70 - WARNING - federation_transaction_transmission_loop-2733 - Re-starting finished log context federation_transaction_transmission_loop-2733
```
Nothing else that looks suspicious.
I haven't set up workers on this host.
@Whodiduexpect Thank you for the graphs. I'm glad to hear workers made things better. Unfortunately I don't think there are any smoking guns there. @Iamsomeguy thank you for the update too.

We're not aware of anything in between 1.43 and 1.44 that would cause this, so this is a bit of a thorny problem to track down. Would either of you be willing to roll back synapse to 1.43 and confirm that you don't see these symptoms any more? (There shouldn't be any problems doing so, e.g. we didn't have any database migrations between those two versions.)
Cross-referencing: @richvdh wondered if someone else was affected, see https://matrix.to/#/!ehXvUhWNASUkSLvAGP:matrix.org/$aTlgICUGpU30pfGVrbrwpBZxBB-0lu3D_x_sp3twifk?via=matrix.org&via=libera.chat&via=underzsea.com
I'm now running 1.43.0 since 07:43 UTC and so far I haven't seen any upstream timeouts in the nginx log. There are lots of DNS requests though and about 5500 "Re-starting finished log context" messages in homeserver.log. This was just after startup. Seems to have calmed down now.
I have also re-enabled presence.
I just got 20 upstream timeouts in the nginx log during 2 minutes after about 20 minutes of running. So, maybe it is not version related. I will keep an eye on it during the day.
I've noticed even with workers a few things depend on the state of the Synapse main process, even if the requests are not going to the master process, notably sending messages. So the lockup will occur and you can sync, scroll up in the room and the history keeps up, etc but event sending does not work. This is blatantly obvious when you upload an image and it uploads instantly to the media repo but it never really sends the event and Element's UI bugs out a little bit but regardless of what it says the image doesn't actually get sent until the lockup has ended. Everything that doesn't go to workers doesn't work during the lockup either, like seeing message edit history.
In terms of whether or not it's a regression specifically in 1.44.0 I now recall this kind of issue happening before but it was nowhere near as bad as it was when I updated and I feel that was more performance related rather than the main process freezing up every 1 in 4 initial syncs on accounts in federated rooms like it currently does. I haven't tested downgrading to 1.43.0, it's possible the issue comes from earlier.
I've narrowed it down even further, I had a generic worker assigned to federation requests but I never had an actual federation sender worker, I added it and now the CPU usage never goes above 5% and the issue is gone for real. I'm not sure how helpful this is but I've confirmed it's in the federation sender part of Synapse.
Just checked and this morning I had only one upstream timeout error in the nginx log and there was one on Saturday. It feels as if it happens when clients first connect in the morning. Our server is private but federated and we are about 10 users. I'm the only user connected to large rooms.

We are still on 1.43.0
Three more timeouts this morning at 08:23-08:24 local time which corresponds to about 5000 requests to our dns cache server.
But overall it looks much better than with 1.44.0
[Someone else in Synapse Admins](https://matrix.to/#/!ehXvUhWNASUkSLvAGP:matrix.org/$Y08ftfyVIzHt8cAz4C8rjkD3XEVHqiM7MkP7D2t9HME?via=matrix.org&via=matrix.breakpointingbad.com&via=libera.chat) seems to be saying they are affected.
@Iamsomeguy What installation method are you using? (Docker? other?)
@callahad Deb package from https://packages.matrix.org/debian/ on Debian 10 buster
I had this or related trouble just now. Using a pip venv to install.
Upgraded straight from 43 to 45.1 which would rarely let me send a message, much less log in anew at all. Tried 44 to no avail, finally I'm back on 43 and could at least log in and seem to be able to send messages again.

I get bursts of multiple `synapse.logging.context - 70 - WARNING - federation_transaction_transmission_loop-2872- Re-starting finished log context federation_transaction_transmission_loop-2872` and `synapse.logging.context - 70 - WARNING - federation_transaction_transmission_loop-2358- Expected logging context sentinel but found federation_transaction_transmission_loop-2872` in the log and my local dnsmasq is overloaded every couple seconds paired with federation DNS error in synapse log:
```
Oct 21 11:51:10 hostname dnsmasq[6205]: Maximum number of concurrent DNS queries reached (max: 150)
Oct 21 11:51:16 hostname dnsmasq[6205]: Maximum number of concurrent DNS queries reached (max: 150)
Oct 21 11:51:23 hostname dnsmasq[6205]: Maximum number of concurrent DNS queries reached (max: 150)
Oct 21 11:51:29 hostname dnsmasq[6205]: Maximum number of concurrent DNS queries reached (max: 150)
Oct 21 11:51:36 hostname dnsmasq[6205]: Maximum number of concurrent DNS queries reached (max: 150)
Oct 21 11:51:43 hostname dnsmasq[6205]: Maximum number of concurrent DNS queries reached (max: 150)
Oct 21 11:52:41 hostname dnsmasq[6205]: Maximum number of concurrent DNS queries reached (max: 150)
Oct 21 11:52:47 hostname dnsmasq[6205]: Maximum number of concurrent DNS queries reached (max: 150)
```

I am also getting this error in element trying to delete old sessions: `request failed: CORS request rejected: https://mydomain:8448/_matrix/client/r0/delete_devices`
For anyone who can readily reproduce this, especially if you're using `pip` to install, would you be willing to try using [git bisect](https://git-scm.com/docs/git-bisect) to identify which commit seems to cause the issue?
> For anyone who can readily reproduce this, especially if you're using `pip` to install, would you be willing to try using [git bisect](https://git-scm.com/docs/git-bisect) to identify which commit seems to cause the issue?

I am happy to help, but need this particular server running productively

Resolving patch:
diff --git a/synapse/app/_base.py b/synapse/app/_base.py
--- a/synapse/app/_base.py
+++ b/synapse/app/_base.py
@@ -31,6 +31,7 @@
 from twisted.internet import defer, error, reactor
 from twisted.logger import LoggingFile, LogLevel
 from twisted.protocols.tls import TLSMemoryBIOFactory
+from twisted.python.threadpool import ThreadPool
 
 import synapse
 from synapse.api.constants import MAX_PDU_SIZE
@@ -48,6 +49,7 @@
 from synapse.metrics.jemalloc import setup_jemalloc_stats
 from synapse.util.caches.lrucache import setup_expire_lru_cache_entries
 from synapse.util.daemonize import daemonize_process
+from synapse.util.gai_resolver import GAIResolver
 from synapse.util.rlimit import change_resource_limit
 from synapse.util.versionstring import get_version_string
 
@@ -338,9 +340,18 @@ async def start(hs: "HomeServer"):
     Args:
         hs: homeserver instance
     """
+    reactor = hs.get_reactor()
+
+    # We want to use a separate thread pool for the resolver so that large
+    # numbers of DNS requests don't starve out other users of the threadpool.
+    resolver_threadpool = ThreadPool(name="gai_resolver")
+    resolver_threadpool.start()
+    reactor.installNameResolver(
+        GAIResolver(reactor, getThreadPool=lambda: resolver_threadpool)
+    )
+
     # Set up the SIGHUP machinery.
     if hasattr(signal, "SIGHUP"):
-        reactor = hs.get_reactor()
 
         @wrap_as_background_process("sighup")
         def handle_sighup(*args, **kwargs):
diff --git a/synapse/util/gai_resolver.py b/synapse/util/gai_resolver.py
new file mode 100644
--- /dev/null
+++ b/synapse/util/gai_resolver.py
@@ -0,0 +1,136 @@
+# This is a direct lift from
+# https://github.com/twisted/twisted/blob/release-21.2.0-10091/src/twisted/internet/_resolver.py.
+# We copy it here as we need to instantiate `GAIResolver` manually, but it is a
+# private class.
+
+
+from socket import (
+    AF_INET,
+    AF_INET6,
+    AF_UNSPEC,
+    SOCK_DGRAM,
+    SOCK_STREAM,
+    gaierror,
+    getaddrinfo,
+)
+
+from zope.interface import implementer
+
+from twisted.internet.address import IPv4Address, IPv6Address
+from twisted.internet.interfaces import IHostnameResolver, IHostResolution
+from twisted.internet.threads import deferToThreadPool
+
+
+@implementer(IHostResolution)
+class HostResolution:
+    """
+    The in-progress resolution of a given hostname.
+    """
+
+    def __init__(self, name):
+        """
+        Create a L{HostResolution} with the given name.
+        """
+        self.name = name
+
+    def cancel(self):
+        # IHostResolution.cancel
+        raise NotImplementedError()
+
+
+_any = frozenset([IPv4Address, IPv6Address])
+
+_typesToAF = {
+    frozenset([IPv4Address]): AF_INET,
+    frozenset([IPv6Address]): AF_INET6,
+    _any: AF_UNSPEC,
+}
+
+_afToType = {
+    AF_INET: IPv4Address,
+    AF_INET6: IPv6Address,
+}
+
+_transportToSocket = {
+    "TCP": SOCK_STREAM,
+    "UDP": SOCK_DGRAM,
+}
+
+_socktypeToType = {
+    SOCK_STREAM: "TCP",
+    SOCK_DGRAM: "UDP",
+}
+
+
+@implementer(IHostnameResolver)
+class GAIResolver:
+    """
+    L{IHostnameResolver} implementation that resolves hostnames by calling
+    L{getaddrinfo} in a thread.
+    """
+
+    def __init__(self, reactor, getThreadPool=None, getaddrinfo=getaddrinfo):
+        """
+        Create a L{GAIResolver}.
+        @param reactor: the reactor to schedule result-delivery on
+        @type reactor: L{IReactorThreads}
+        @param getThreadPool: a function to retrieve the thread pool to use for
+            scheduling name resolutions.  If not supplied, the use the given
+            C{reactor}'s thread pool.
+        @type getThreadPool: 0-argument callable returning a
+            L{twisted.python.threadpool.ThreadPool}
+        @param getaddrinfo: a reference to the L{getaddrinfo} to use - mainly
+            parameterized for testing.
+        @type getaddrinfo: callable with the same signature as L{getaddrinfo}
+        """
+        self._reactor = reactor
+        self._getThreadPool = (
+            reactor.getThreadPool if getThreadPool is None else getThreadPool
+        )
+        self._getaddrinfo = getaddrinfo
+
+    def resolveHostName(
+        self,
+        resolutionReceiver,
+        hostName,
+        portNumber=0,
+        addressTypes=None,
+        transportSemantics="TCP",
+    ):
+        """
+        See L{IHostnameResolver.resolveHostName}
+        @param resolutionReceiver: see interface
+        @param hostName: see interface
+        @param portNumber: see interface
+        @param addressTypes: see interface
+        @param transportSemantics: see interface
+        @return: see interface
+        """
+        pool = self._getThreadPool()
+        addressFamily = _typesToAF[
+            _any if addressTypes is None else frozenset(addressTypes)
+        ]
+        socketType = _transportToSocket[transportSemantics]
+
+        def get():
+            try:
+                return self._getaddrinfo(
+                    hostName, portNumber, addressFamily, socketType
+                )
+            except gaierror:
+                return []
+
+        d = deferToThreadPool(self._reactor, pool, get)
+        resolution = HostResolution(hostName)
+        resolutionReceiver.resolutionBegan(resolution)
+
+        @d.addCallback
+        def deliverResults(result):
+            for family, socktype, _proto, _cannoname, sockaddr in result:
+                addrType = _afToType[family]
+                resolutionReceiver.addressResolved(
+                    addrType(_socktypeToType.get(socktype, "TCP"), *sockaddr)
+                )
+            resolutionReceiver.resolutionComplete()
+
+        return resolution


Test patch: