Repository: matrix-org/synapse

Issue:
Unable to configure SMTP to use TLS without STARTTLS
<!--

**THIS IS NOT A SUPPORT CHANNEL!**
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

<!-- Describe here the problem that you are experiencing -->

Hi. I'm the sysadmin operating converser.eu homeserver. It seems that my users can't register email in their accounts. New users can't register when they put an email.

When I try to add an email, there is a POST request to `https://converser.eu/_matrix/client/r0/account/3pid/email/requestToken` but it timeouts. On app.element.io, I've got the error `request failed: ETIMEDOUT` .

In synapse configuration, the identity server is defined. The mailserver is configured too, and the connection to it is ok.

I've got nginx as a proxy. I've got a timeout : 
`2020/08/06 15:32:09 [error] 1067425#1067425: *1948 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 2a0e:xxxxxx, server: converser.eu, request: "POST /_matrix/client/r0/account/3pid/email/requestToken HTTP/2.0", upstream: "http://127.0.0.1:8008/_matrix/client/r0/account/3pid/email/requestToken", host: "converser.eu"`
But no logs related to this in homeserver.log.

Am I missing something ?

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: converser.eu

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: 1.17.0

- **Install method**: debian package
<!-- examples: package manager/git clone/pip  -->

- **Platform**: debian buster + backports
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



Discussion:
I'm one of these users, and in my account, which already has a registered email, I get the `ETIMEDOUT` error when trying to perform a password reset. I don't have access to the server, but the end-user symptom is a `POST` request on https://converser.eu/_matrix/client/r0/account/password/email/requestToken that never gets any answer.
>  But no logs related to this in homeserver.log.

The first thing to try is upping your log level to `INFO` or `DEBUG` and seeing if that gives any additional information.

If the request isn't making it to the homeserver this sounds like an issue with the nginx config.
Here are the logs. On synapse : 
```
2020-08-08 10:00:43,710 - synapse.access.http.8008 - 249 - DEBUG - POST-615- 2a0e:xxxx - 8008 - Received request: POST /_matrix/client/r0/account/3pid/email/requestToken
2020-08-08 10:00:43,711 - synapse.storage.txn - 402 - DEBUG - POST-615- [TXN START] {get_user_id_by_threepid-1643}
2020-08-08 10:00:43,712 - synapse.storage.SQL - 221 - DEBUG - POST-615- [SQL] {get_user_id_by_threepid-1643} SELECT user_id FROM user_threepids WHERE medium = ? AND address = ?
2020-08-08 10:00:43,713 - synapse.storage.SQL - 226 - DEBUG - POST-615- [SQL values] {get_user_id_by_threepid-1643} ['email', 'xxxx']
2020-08-08 10:00:43,715 - synapse.storage.SQL - 240 - DEBUG - POST-615- [SQL time] {get_user_id_by_threepid-1643} 0.001488 sec
2020-08-08 10:00:43,717 - synapse.storage.txn - 489 - DEBUG - POST-615- [TXN END] {get_user_id_by_threepid-1643} 0.005449 sec
2020-08-08 10:00:43,807 - synapse.storage.txn - 402 - DEBUG - POST-615- [TXN START] {get_threepid_validation_session-1647}
2020-08-08 10:00:43,809 - synapse.storage.SQL - 221 - DEBUG - POST-615- [SQL] {get_threepid_validation_session-1647} SELECT address, session_id, medium, client_secret, last_send_attempt, validated_at FROM threepid_validation_session WHERE client_secret = ? AND medium = ? AND address = ? AND validated_at IS NULL LIMIT 1
2020-08-08 10:00:43,811 - synapse.storage.SQL - 226 - DEBUG - POST-615- [SQL values] {get_threepid_validation_session-1647} ['xxxx', 'email', 'xxxx']
2020-08-08 10:00:43,816 - synapse.storage.SQL - 240 - DEBUG - POST-615- [SQL time] {get_threepid_validation_session-1647} 0.002463 sec
2020-08-08 10:00:43,818 - synapse.storage.txn - 489 - DEBUG - POST-615- [TXN END] {get_threepid_validation_session-1647} 0.011665 sec
2020-08-08 10:00:43,900 - synapse.push.mailer - 309 - INFO - POST-615- Sending email to xxxx
2020-08-08 10:06:05,959 - synapse.http.site - 219 - WARNING - POST-615- Error processing request <XForwardedForRequest at 0x7fd4ab7bfba8 method='POST' uri='/_matrix/client/r0/account/3pid/email/requestToken' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.
```

And on my mailserver : 
```
Aug  7 10:04:40 mail1 postfix/submissions/smtpd[26520]: connect from matrix.6clones.net[51.15.191.9]
Aug  7 10:07:49 mail1 postfix/submissions/smtpd[30015]: SSL_accept error from matrix.6clones.net[51.15.191.9]: Connection timed out
Aug  7 10:07:49 mail1 postfix/submissions/smtpd[30015]: lost connection after CONNECT from matrix.6clones.net[51.15.191.9]
Aug  7 10:07:49 mail1 postfix/submissions/smtpd[30015]: disconnect from matrix.6clones.net[51.15.191.9] commands=0/0
```

So synapse is connecting to the mailserver, but the connection fails. I wondered if that was because of the TLS. I've configured Synapse to use 465 port, which is direct TLS, not STARTTLS method. And it seems Synapse doesn't handle it. I've changed the port to 587 and now it works.

So, new question, why Synapse can't connect to a mailserver with direct TLS connection ?
@Toniob What's the value of `email.require_transport_security` in your config?
`require_transport_security: True`

With port 465, it fails, with 587, it works.
The `require_transport_security` configuration option seems to be a bit poorly named (but is inherited from Twisted and [has a decent comment around](https://github.com/matrix-org/synapse/blob/faba873d4b91cf03c5c2d9546a6df781ce58f2cc/docs/sample_config.yaml#L1944-L1949)):

> Uncomment the following to require TLS transport security for SMTP. By default, Synapse will connect over plain text, and will then switch to TLS via STARTTLS *if the SMTP server supports it*. If this option is set, Synapse will refuse to connect unless the server supports STARTTLS.

`require_transport_security` means that STARTTLS is still used, but it requires it to succeed.

I do not believe there is a way to configure the initial connection to the SMTP server use TLS. (Note that Synapse uses [Twisted's `sendmail`](https://twistedmatrix.com/documents/19.2.0/api/twisted.mail.smtp.html#sendmail) method under-the-hood, which also doesn't seem to support this configuration.)

I'll update this issue as a request for just using TLS without STARTTLS.
Sorry to ask here, but I am not able to get this running with synapse v1.41.0. What is now the correct configuration? The only log I see is the info log for "Sending Mail". No warnings or errors.
My current config is:
```
registrations_require_3pid:
    - email
allowed_local_3pids:
    - medium: email
      pattern: '^[^@]+@<my-domain>$'
email:
    smtp_host: smtp.***
    smtp_port: 465
    smtp_user: '****'
    smtp_pass: '****'
    require_transport_security: False
    enable_tls: True 
    notif_from: "Your Friendly %(app)s homeserver <no-reply@<my-domain>>"
    subjects:
        email_validation: "[%(server_name)s] Validate your email"
```
Sadly I am not able to get logs from the email-server. Can someone please verify that this is the correct configuration. Or do you have a hint how I can enable more logs for the twisted library to find out what the problem on my side is? 
Same problem here port 587 works, port 465 does not.

The mail server itself supports 465 without issue, I have other things configured to use it over smtp via port 465 such as nextcloud, pfsense notification, grafana, etc.

Port 465 is always encrypted under SSL, why is it not supported?

Why do these settings even exist, couldn't they be inferred from the configured port?
```yaml
  enable_tls: true
  require_transport_security: true
```

**This is configured in the homeserver.yaml**

because I am on debian mine is here:
`sudo nano /etc/matrix-synapse/conf.d/config.yaml`

but non debian users might be here:
`sudo nano /etc/matrix-synapse/homeserver.yaml`

This is 587 which works:
```yaml
email:
  smtp_host: "smtp.example.com"
  smtp_port: 587
  smtp_user: "noreply@example.com"
  smtp_pass: "ExampleExample"
  notif_from: "%(app)s Homeserver <noreply@example.com>"
  client_base_url: "https://matrix.example.com"
  app_name: "My Matrix"
  enable_tls: true
  require_transport_security: true
```

This is 465, does not work:
```yaml
email:
  smtp_host: "smtp.example.com"
  smtp_port: 465
  smtp_user: "noreply@example.com"
  smtp_pass: "ExampleExample"
  notif_from: "%(app)s Homeserver <noreply@example.com>"
  client_base_url: "https://matrix.example.com"
  app_name: "My Matrix"
#  enable_tls: true
#  require_transport_security: true
```

Also tried this for 465, does not work:
```yaml
email:
  smtp_host: "smtp.example.com"
  smtp_port: 465
  smtp_user: "noreply@example.com"
  smtp_pass: "ExampleExample"
  notif_from: "%(app)s Homeserver <noreply@example.com>"
  client_base_url: "https://matrix.example.com"
  app_name: "My Matrix"
  enable_tls: true
#  require_transport_security: true
```

Resolving patch:
diff --git a/synapse/config/emailconfig.py b/synapse/config/emailconfig.py
--- a/synapse/config/emailconfig.py
+++ b/synapse/config/emailconfig.py
@@ -85,14 +85,19 @@ def read_config(self, config: JsonDict, **kwargs: Any) -> None:
         if email_config is None:
             email_config = {}
 
+        self.force_tls = email_config.get("force_tls", False)
         self.email_smtp_host = email_config.get("smtp_host", "localhost")
-        self.email_smtp_port = email_config.get("smtp_port", 25)
+        self.email_smtp_port = email_config.get(
+            "smtp_port", 465 if self.force_tls else 25
+        )
         self.email_smtp_user = email_config.get("smtp_user", None)
         self.email_smtp_pass = email_config.get("smtp_pass", None)
         self.require_transport_security = email_config.get(
             "require_transport_security", False
         )
         self.enable_smtp_tls = email_config.get("enable_tls", True)
+        if self.force_tls and not self.enable_smtp_tls:
+            raise ConfigError("email.force_tls requires email.enable_tls to be true")
         if self.require_transport_security and not self.enable_smtp_tls:
             raise ConfigError(
                 "email.require_transport_security requires email.enable_tls to be true"
diff --git a/synapse/handlers/send_email.py b/synapse/handlers/send_email.py
--- a/synapse/handlers/send_email.py
+++ b/synapse/handlers/send_email.py
@@ -23,10 +23,12 @@
 
 import twisted
 from twisted.internet.defer import Deferred
-from twisted.internet.interfaces import IOpenSSLContextFactory, IReactorTCP
+from twisted.internet.interfaces import IOpenSSLContextFactory
+from twisted.internet.ssl import optionsForClientTLS
 from twisted.mail.smtp import ESMTPSender, ESMTPSenderFactory
 
 from synapse.logging.context import make_deferred_yieldable
+from synapse.types import ISynapseReactor
 
 if TYPE_CHECKING:
     from synapse.server import HomeServer
@@ -48,7 +50,7 @@ def _getContextFactory(self) -> Optional[IOpenSSLContextFactory]:
 
 
 async def _sendmail(
-    reactor: IReactorTCP,
+    reactor: ISynapseReactor,
     smtphost: str,
     smtpport: int,
     from_addr: str,
@@ -59,6 +61,7 @@ async def _sendmail(
     require_auth: bool = False,
     require_tls: bool = False,
     enable_tls: bool = True,
+    force_tls: bool = False,
 ) -> None:
     """A simple wrapper around ESMTPSenderFactory, to allow substitution in tests
 
@@ -73,8 +76,9 @@ async def _sendmail(
         password: password to give when authenticating
         require_auth: if auth is not offered, fail the request
         require_tls: if TLS is not offered, fail the reqest
-        enable_tls: True to enable TLS. If this is False and require_tls is True,
+        enable_tls: True to enable STARTTLS. If this is False and require_tls is True,
            the request will fail.
+        force_tls: True to enable Implicit TLS.
     """
     msg = BytesIO(msg_bytes)
     d: "Deferred[object]" = Deferred()
@@ -105,13 +109,23 @@ def build_sender_factory(**kwargs: Any) -> ESMTPSenderFactory:
         # set to enable TLS.
         factory = build_sender_factory(hostname=smtphost if enable_tls else None)
 
-    reactor.connectTCP(
-        smtphost,
-        smtpport,
-        factory,
-        timeout=30,
-        bindAddress=None,
-    )
+    if force_tls:
+        reactor.connectSSL(
+            smtphost,
+            smtpport,
+            factory,
+            optionsForClientTLS(smtphost),
+            timeout=30,
+            bindAddress=None,
+        )
+    else:
+        reactor.connectTCP(
+            smtphost,
+            smtpport,
+            factory,
+            timeout=30,
+            bindAddress=None,
+        )
 
     await make_deferred_yieldable(d)
 
@@ -132,6 +146,7 @@ def __init__(self, hs: "HomeServer"):
         self._smtp_pass = passwd.encode("utf-8") if passwd is not None else None
         self._require_transport_security = hs.config.email.require_transport_security
         self._enable_tls = hs.config.email.enable_smtp_tls
+        self._force_tls = hs.config.email.force_tls
 
         self._sendmail = _sendmail
 
@@ -189,4 +204,5 @@ async def send_email(
             require_auth=self._smtp_user is not None,
             require_tls=self._require_transport_security,
             enable_tls=self._enable_tls,
+            force_tls=self._force_tls,
         )


Test patch:
diff --git a/tests/handlers/test_send_email.py b/tests/handlers/test_send_email.py
--- a/tests/handlers/test_send_email.py
+++ b/tests/handlers/test_send_email.py
@@ -23,7 +23,7 @@
 from twisted.mail import interfaces, smtp
 
 from tests.server import FakeTransport
-from tests.unittest import HomeserverTestCase
+from tests.unittest import HomeserverTestCase, override_config
 
 
 @implementer(interfaces.IMessageDelivery)
@@ -110,3 +110,58 @@ def test_send_email(self):
         user, msg = message_delivery.messages.pop()
         self.assertEqual(str(user), "foo@bar.com")
         self.assertIn(b"Subject: test subject", msg)
+
+    @override_config(
+        {
+            "email": {
+                "notif_from": "noreply@test",
+                "force_tls": True,
+            },
+        }
+    )
+    def test_send_email_force_tls(self):
+        """Happy-path test that we can send email to an Implicit TLS server."""
+        h = self.hs.get_send_email_handler()
+        d = ensureDeferred(
+            h.send_email(
+                "foo@bar.com", "test subject", "Tests", "HTML content", "Text content"
+            )
+        )
+        # there should be an attempt to connect to localhost:465
+        self.assertEqual(len(self.reactor.sslClients), 1)
+        (
+            host,
+            port,
+            client_factory,
+            contextFactory,
+            _timeout,
+            _bindAddress,
+        ) = self.reactor.sslClients[0]
+        self.assertEqual(host, "localhost")
+        self.assertEqual(port, 465)
+
+        # wire it up to an SMTP server
+        message_delivery = _DummyMessageDelivery()
+        server_protocol = smtp.ESMTP()
+        server_protocol.delivery = message_delivery
+        # make sure that the server uses the test reactor to set timeouts
+        server_protocol.callLater = self.reactor.callLater  # type: ignore[assignment]
+
+        client_protocol = client_factory.buildProtocol(None)
+        client_protocol.makeConnection(FakeTransport(server_protocol, self.reactor))
+        server_protocol.makeConnection(
+            FakeTransport(
+                client_protocol,
+                self.reactor,
+                peer_address=IPv4Address("TCP", "127.0.0.1", 1234),
+            )
+        )
+
+        # the message should now get delivered
+        self.get_success(d, by=0.1)
+
+        # check it arrived
+        self.assertEqual(len(message_delivery.messages), 1)
+        user, msg = message_delivery.messages.pop()
+        self.assertEqual(str(user), "foo@bar.com")
+        self.assertIn(b"Subject: test subject", msg)