Repository: matrix-org/synapse

Issue:
Hostname provided by SNI and in the Host header doesn't match in federation requests
Running synapse since several months behind a proxypass, including the federation port. I'm using apache httpd as reverse proxy. I have to servers (same config). As soon as I update one of those to 0.33.3 (or now 0.33.4), federation don't work anymore. The homeserver 0.33.4 can't send anything to the one still at 0.33.2

I can see a lot of resp 400 from synapse to the rev proxy each time a synapse >= 0.33.3 talks to me (0.33.2). Upgrading this synapse instance to 0.33.4 doesn't change anything. So it's not an incompatibility between >= 0.33.3 and older, but something which broke federation behind a proxypass.

What info would you need to debug this ?


Discussion:
@dani: we need to see the logs for the requests which are failing - probably a large block of logs at INFO log level on both the source & destination server would help.
In fact, it seems that synapse >= 0.33.3 who sends put requests on the federation endpoints are not dispatched correctly to synapse backend, but are being answered by a 400 error message by the frontal httpd directly. It might have something to do with the recent SNI change brought in 0.33.3, which somehow doesn't play well with my proxypass setup (request treated by wrong vhost ?). I'll gather the info and post it here ASAP
Ok, looks like I got it. Requests from other synapse servers (but only those >= 0.33.3) are not valid, because the SNI doesn't match the Host header. Eg, my matrix server is lapiole.org. My ID is @dani:lapiole.org but the DNS name of the box handling it is matrix.lapiole.org (published through the SRV DNS entry), and here are the result:

```
[Tue Sep 18 08:26:15.954501 2018] [ssl:error] [pid 31587] AH02032: Hostname matrix.lapiole.org provided via SNI and hostname lapiole.org provided via HTTP are different
```

And the corresponding access_log line

```
matrix.lapiole.org 52.211.y.x - - [18/Sep/2018:08:26:15 +0200] "PUT /_matrix/federation/v1/send/1537216206283/ HTTP/1.1" 400 - "-" "Synapse/0.33.4" scheme="-"
```

There was no SNI before that, so the request was being treated (through a dedicated vhost listening on a specific port), but now that there's SNI, and that it doesn't match Host, httpd automatically responds with a 400 bad request. It doesn't even reach synapse backend.
Not sure how nginx (or other proxies) are handling this, but those requests are invalid. And it looks like there's no way to ignore this and serve them anyway with Apache httpd. So indeed, 0.33.3 broke reverse-proxyfied federation
The problem has already been rised in issue #2525 which has been closed: "the conclusion was that the current behaviour is actually correct". I do not agree with the conclusion. Requests with a mismatch between SNI and Host headers are invalid (not only Apache httpd rejects them, cloudflare also does for example, see https://support.cloudflare.com/hc/en-us/articles/204872628-Error-1013-HTTP-hostname-and-TLS-SNI-hostname-mismatch)
Indeed there should not be a mismatch between SNI and Host. (The topic of #2525 was what they should be set to.)

I'll have a look at this later.

Resolving patch:
diff --git a/synapse/http/endpoint.py b/synapse/http/endpoint.py
--- a/synapse/http/endpoint.py
+++ b/synapse/http/endpoint.py
@@ -108,7 +108,7 @@ def matrix_federation_endpoint(reactor, destination, tls_client_options_factory=
 
     Args:
         reactor: Twisted reactor.
-        destination (bytes): The name of the server to connect to.
+        destination (unicode): The name of the server to connect to.
         tls_client_options_factory
             (synapse.crypto.context_factory.ClientTLSOptionsFactory):
             Factory which generates TLS options for client connections.
@@ -126,10 +126,17 @@ def matrix_federation_endpoint(reactor, destination, tls_client_options_factory=
         transport_endpoint = HostnameEndpoint
         default_port = 8008
     else:
+        # the SNI string should be the same as the Host header, minus the port.
+        # as per https://github.com/matrix-org/synapse/issues/2525#issuecomment-336896777,
+        # the Host header and SNI should therefore be the server_name of the remote
+        # server.
+        tls_options = tls_client_options_factory.get_options(domain)
+
         def transport_endpoint(reactor, host, port, timeout):
             return wrapClientTLS(
-                tls_client_options_factory.get_options(host),
-                HostnameEndpoint(reactor, host, port, timeout=timeout))
+                tls_options,
+                HostnameEndpoint(reactor, host, port, timeout=timeout),
+            )
         default_port = 8448
 
     if port is None:


Test patch: