Repository: matrix-org/synapse

Issue:
Synapse workers can not connect to master after upgrading to 1.85.0 from 1.84.0  
### Description

After upgrading from 1.84.0 to 1.85.0 worker processes are unable to contact the main process.
Downgrading to 1.84.0 with the same configuration resolves the connection problems.

The configuration is using a shared `instance_map` block with a `main` section.

Could be that there is something wrong with the existing configuration and that it worked by chance.

Looking at the changelog for 1.85.0, pull request #15578 seems relevant.

### Steps to reproduce

Upgrade from 1.84.0 to 1.85.0 using the attached configuraiton.

### Homeserver

private server

### Synapse Version

1.85.0

### Installation Method

Docker (matrixdotorg/synapse)

### Database

PostgreSQL, single server. Not Not restored from backup.

### Workers

Multiple workers

### Platform

Master process and workers are running as containers in a k8s cluster.

### Configuration

Relevant parts of shared homeserver.yaml


    pid_file: /var/run/homeserver.pid
    log_config: "/config/log.config"
    tls_private_key_path: "/etc/ssl/cluster-certificate/tls.key"
    tls_certificate_path: "/etc/ssl/cluster-certificate/tls.crt"

    listeners:
      - port: 8008
        tls: true
        bind_address: '0.0.0.0'
        type: http
        x_forwarded: true
        resources:
          - names: [client]
            compress: false
          - names: [federation]
            compress: false
      - port: 9093
        bind_address: '0.0.0.0'
        tls: true
        type: http
        resources:
         - names: [replication]

    instance_map:
      main:
        host: synapse.default.svc
        port: 9093
        tls: true
      synapse_worker_0:
        host: synapse-worker-0.synapse-worker-headless.default.svc
        port: 9093
        tls: false
      synapse_worker_1:
        host: synapse-worker-1.synapse-worker-headless.default.svc
        port: 9093
        tls: false

    stream_writers:
      events:
        - synapse_worker_0
        - synapse_worker_1

    worker_replication_secret: "secret"
    redis:
      enabled: true
      host: "redis.default.svc"
      port: 6379
      password: "redis-password"

worker.yaml

    worker_name: synapse_worker_0
    worker_app: synapse.app.generic_worker
    worker_log_config: "/config/log.config"

    worker_listeners:
      - port: 8008
        tls: true
        bind_addresses: ['0.0.0.0']
        type: http
        x_forwarded: true
        resources:
          - names: [client]
            compress: false
          - names: [federation]
            compress: false
      - port: 9093
        bind_address: '0.0.0.0'
        tls: false
        type: http
        resources:
         - names: [replication]

### Relevant log output

```shell
Following is observed on the worker nodes

2023-06-07 05:26:49,967 - synapse.http.client - 933 - INFO - GET-53- Error sending request to  POST synapse-replication://master/_synapse/replication/presence_set_state/<redacted>: AttributeError 'str' object has no attribute 'decode'
2023-06-07 05:26:50,395 - synapse.federation.federation_server - 1488 - INFO - PUT-55- Failed to handle edu 'm.presence': SynapseError('502: Failed to talk to master process')
```


### Anything else that would be useful to know?

_No response_


Discussion:
This looks to be a regression when using TLS for replication. 

https://github.com/matrix-org/synapse/blob/d162aecaac52fb467822e319e4c3c5b216c33ca9/synapse/http/replicationagent.py#L77-L81

Those are strings, but `creatorForNetloc` is expecting `bytes`. I don't know why `mypy` didn't pick that up.

Resolving patch:
diff --git a/synapse/http/replicationagent.py b/synapse/http/replicationagent.py
--- a/synapse/http/replicationagent.py
+++ b/synapse/http/replicationagent.py
@@ -76,7 +76,7 @@ def endpointForURI(self, uri: URI) -> IStreamClientEndpoint:
                 endpoint = wrapClientTLS(
                     # The 'port' argument below isn't actually used by the function
                     self.context_factory.creatorForNetloc(
-                        self.instance_map[worker_name].host,
+                        self.instance_map[worker_name].host.encode("utf-8"),
                         self.instance_map[worker_name].port,
                     ),
                     endpoint,


Test patch: