Repository: matrix-org/synapse

Issue:
Reuse SSL contexts for .well-known lookups
While I was investigating why Synapse 0.99 appeared to be slower than Synapse 0.34 (#4713), I tried disabling the .well-known lookup in the federation agent. Doing so appeared to make my synapse settle at 1.5G of memory usage rather than 2G.

> The .well-known cache sits at around 1500 entries, so I can't believe it's all coming from there - it would imply about 333K per entry, which seems immense. I wonder if the connection pool is hanging on to data it shouldn't. It would probably be fruitful to investigate with a memory profiler.


Discussion:
#4713 was closed in favour of this bug, but this bug only mentions memory. To me it also appears that our synapse 0.99 instance is much slower than it was on 0.34. Should there be a separate issue for the performance regression?
If you have details of a performance regression that is not caused by memory usage, then please do create a new ticket.
I would guess that this shares the root cause with #5395 (assuming synapse 0.99 validates SSL certs during `.well-known` lookups)
> I would guess that this shares the root cause with #5395 (assuming synapse 0.99 validates SSL certs during .well-known lookups)

yes, absolutely. I think the only reason it was bad rather than catastrophic for .well-known is that we tend to make fewer connections for .well-known lookups, and then cache the results for a while.

We can probably apply the same fix to .well-known lookup that we did to #5395.
As of 1.0, my notoriously beset homeserver, which historically settled around 1.4GB on a good day and occasionally exhausted my 4GB of ram entirely and brings down the whole server with swap thrashing, is using under 400MB.
that might just mean that it doesn't have any extremities atm and so isn't having to do much state resolution
I've actually been keeping a close eye on extremities lately; both before and after, no room has had more than 4 and only a handful have had more than 1. The improvement is without doubt due to a change between 0.99.5.2 and 1.0. I think #5417 has fixed a serious long-standing problem.
anecdatally, my HS has gone from idling at 1.3GB to being at 650MB fwiw.
I'm turning this issue into a tracker for applying the #5395 fix to .well-known lookups.

Resolving patch:
diff --git a/synapse/crypto/context_factory.py b/synapse/crypto/context_factory.py
--- a/synapse/crypto/context_factory.py
+++ b/synapse/crypto/context_factory.py
@@ -31,6 +31,7 @@
     platformTrust,
 )
 from twisted.python.failure import Failure
+from twisted.web.iweb import IPolicyForHTTPS
 
 logger = logging.getLogger(__name__)
 
@@ -74,6 +75,7 @@ def getContext(self):
         return self._context
 
 
+@implementer(IPolicyForHTTPS)
 class ClientTLSOptionsFactory(object):
     """Factory for Twisted SSLClientConnectionCreators that are used to make connections
     to remote servers for federation.
@@ -146,6 +148,12 @@ def _context_info_cb(ssl_connection, where, ret):
             f = Failure()
             tls_protocol.failVerification(f)
 
+    def creatorForNetloc(self, hostname, port):
+        """Implements the IPolicyForHTTPS interace so that this can be passed
+        directly to agents.
+        """
+        return self.get_options(hostname)
+
 
 @implementer(IOpenSSLClientConnectionCreator)
 class SSLClientConnectionCreator(object):
diff --git a/synapse/http/federation/matrix_federation_agent.py b/synapse/http/federation/matrix_federation_agent.py
--- a/synapse/http/federation/matrix_federation_agent.py
+++ b/synapse/http/federation/matrix_federation_agent.py
@@ -64,10 +64,6 @@ class MatrixFederationAgent(object):
         tls_client_options_factory (ClientTLSOptionsFactory|None):
             factory to use for fetching client tls options, or none to disable TLS.
 
-        _well_known_tls_policy (IPolicyForHTTPS|None):
-            TLS policy to use for fetching .well-known files. None to use a default
-            (browser-like) implementation.
-
         _srv_resolver (SrvResolver|None):
             SRVResolver impl to use for looking up SRV records. None to use a default
             implementation.
@@ -81,7 +77,6 @@ def __init__(
         self,
         reactor,
         tls_client_options_factory,
-        _well_known_tls_policy=None,
         _srv_resolver=None,
         _well_known_cache=well_known_cache,
     ):
@@ -98,13 +93,12 @@ def __init__(
         self._pool.maxPersistentPerHost = 5
         self._pool.cachedConnectionTimeout = 2 * 60
 
-        agent_args = {}
-        if _well_known_tls_policy is not None:
-            # the param is called 'contextFactory', but actually passing a
-            # contextfactory is deprecated, and it expects an IPolicyForHTTPS.
-            agent_args["contextFactory"] = _well_known_tls_policy
         _well_known_agent = RedirectAgent(
-            Agent(self._reactor, pool=self._pool, **agent_args)
+            Agent(
+                self._reactor,
+                pool=self._pool,
+                contextFactory=tls_client_options_factory,
+            )
         )
         self._well_known_agent = _well_known_agent
 


Test patch:
diff --git a/tests/http/federation/test_matrix_federation_agent.py b/tests/http/federation/test_matrix_federation_agent.py
--- a/tests/http/federation/test_matrix_federation_agent.py
+++ b/tests/http/federation/test_matrix_federation_agent.py
@@ -75,7 +75,6 @@ def setUp(self):
 
         config_dict = default_config("test", parse=False)
         config_dict["federation_custom_ca_list"] = [get_test_ca_cert_file()]
-        # config_dict["trusted_key_servers"] = []
 
         self._config = config = HomeServerConfig()
         config.parse_config_dict(config_dict, "", "")
@@ -83,7 +82,6 @@ def setUp(self):
         self.agent = MatrixFederationAgent(
             reactor=self.reactor,
             tls_client_options_factory=ClientTLSOptionsFactory(config),
-            _well_known_tls_policy=TrustingTLSPolicyForHTTPS(),
             _srv_resolver=self.mock_resolver,
             _well_known_cache=self.well_known_cache,
         )
@@ -691,16 +689,18 @@ def test_get_well_known_unsigned_cert(self):
         not signed by a CA
         """
 
-        # we use the same test server as the other tests, but use an agent
-        # with _well_known_tls_policy left to the default, which will not
-        # trust it (since the presented cert is signed by a test CA)
+        # we use the same test server as the other tests, but use an agent with
+        # the config left to the default, which will not trust it (since the
+        # presented cert is signed by a test CA)
 
         self.mock_resolver.resolve_service.side_effect = lambda _: []
         self.reactor.lookups["testserv"] = "1.2.3.4"
 
+        config = default_config("test", parse=True)
+
         agent = MatrixFederationAgent(
             reactor=self.reactor,
-            tls_client_options_factory=ClientTLSOptionsFactory(self._config),
+            tls_client_options_factory=ClientTLSOptionsFactory(config),
             _srv_resolver=self.mock_resolver,
             _well_known_cache=self.well_known_cache,
         )