Repository: matrix-org/synapse

Issue:
Unable to configure SMTP to use TLS without STARTTLS
<!--

**THIS IS NOT A SUPPORT CHANNEL!**
**IF YOU HAVE SUPPORT QUESTIONS ABOUT RUNNING OR CONFIGURING YOUR OWN HOME SERVER**,
please ask in **#synapse:matrix.org** (using a matrix.org account if necessary)

If you want to report a security issue, please see https://matrix.org/security-disclosure-policy/

This is a bug report template. By following the instructions below and
filling out the sections with your information, you will help the us to get all
the necessary data to fix your issue.

You can also preview your report before submitting it. You may remove sections
that aren't relevant to your particular case.

Text between <!-- and --​> marks will be invisible in the report.

-->

### Description

<!-- Describe here the problem that you are experiencing -->

Hi. I'm the sysadmin operating converser.eu homeserver. It seems that my users can't register email in their accounts. New users can't register when they put an email.

When I try to add an email, there is a POST request to `https://converser.eu/_matrix/client/r0/account/3pid/email/requestToken` but it timeouts. On app.element.io, I've got the error `request failed: ETIMEDOUT` .

In synapse configuration, the identity server is defined. The mailserver is configured too, and the connection to it is ok.

I've got nginx as a proxy. I've got a timeout : 
`2020/08/06 15:32:09 [error] 1067425#1067425: *1948 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 2a0e:xxxxxx, server: converser.eu, request: "POST /_matrix/client/r0/account/3pid/email/requestToken HTTP/2.0", upstream: "http://127.0.0.1:8008/_matrix/client/r0/account/3pid/email/requestToken", host: "converser.eu"`
But no logs related to this in homeserver.log.

Am I missing something ?

<!--
Describe how what happens differs from what you expected.

If you can identify any relevant log snippets from _homeserver.log_, please include
those (please be careful to remove any personal or private data). Please surround them with
``` (three backticks, on a line on their own), so that they are formatted legibly.
-->

### Version information

<!-- IMPORTANT: please answer the following questions, to help us narrow down the problem -->

<!-- Was this issue identified on matrix.org or another homeserver? -->
- **Homeserver**: converser.eu

<!--
 What version of Synapse is running?

You can find the Synapse version with this command:

$ curl http://localhost:8008/_synapse/admin/v1/server_version

(You may need to replace `localhost:8008` if Synapse is not configured to
listen on that port.)
-->
- **Version**: 1.17.0

- **Install method**: debian package
<!-- examples: package manager/git clone/pip  -->

- **Platform**: debian buster + backports
<!--
Tell us about the environment in which your homeserver is operating
distro, hardware, if it's running in a vm/container, etc.
-->



Discussion:
I'm one of these users, and in my account, which already has a registered email, I get the `ETIMEDOUT` error when trying to perform a password reset. I don't have access to the server, but the end-user symptom is a `POST` request on https://converser.eu/_matrix/client/r0/account/password/email/requestToken that never gets any answer.
>  But no logs related to this in homeserver.log.

The first thing to try is upping your log level to `INFO` or `DEBUG` and seeing if that gives any additional information.

If the request isn't making it to the homeserver this sounds like an issue with the nginx config.
Here are the logs. On synapse : 
```
2020-08-08 10:00:43,710 - synapse.access.http.8008 - 249 - DEBUG - POST-615- 2a0e:xxxx - 8008 - Received request: POST /_matrix/client/r0/account/3pid/email/requestToken
2020-08-08 10:00:43,711 - synapse.storage.txn - 402 - DEBUG - POST-615- [TXN START] {get_user_id_by_threepid-1643}
2020-08-08 10:00:43,712 - synapse.storage.SQL - 221 - DEBUG - POST-615- [SQL] {get_user_id_by_threepid-1643} SELECT user_id FROM user_threepids WHERE medium = ? AND address = ?
2020-08-08 10:00:43,713 - synapse.storage.SQL - 226 - DEBUG - POST-615- [SQL values] {get_user_id_by_threepid-1643} ['email', 'xxxx']
2020-08-08 10:00:43,715 - synapse.storage.SQL - 240 - DEBUG - POST-615- [SQL time] {get_user_id_by_threepid-1643} 0.001488 sec
2020-08-08 10:00:43,717 - synapse.storage.txn - 489 - DEBUG - POST-615- [TXN END] {get_user_id_by_threepid-1643} 0.005449 sec
2020-08-08 10:00:43,807 - synapse.storage.txn - 402 - DEBUG - POST-615- [TXN START] {get_threepid_validation_session-1647}
2020-08-08 10:00:43,809 - synapse.storage.SQL - 221 - DEBUG - POST-615- [SQL] {get_threepid_validation_session-1647} SELECT address, session_id, medium, client_secret, last_send_attempt, validated_at FROM threepid_validation_session WHERE client_secret = ? AND medium = ? AND address = ? AND validated_at IS NULL LIMIT 1
2020-08-08 10:00:43,811 - synapse.storage.SQL - 226 - DEBUG - POST-615- [SQL values] {get_threepid_validation_session-1647} ['xxxx', 'email', 'xxxx']
2020-08-08 10:00:43,816 - synapse.storage.SQL - 240 - DEBUG - POST-615- [SQL time] {get_threepid_validation_session-1647} 0.002463 sec
2020-08-08 10:00:43,818 - synapse.storage.txn - 489 - DEBUG - POST-615- [TXN END] {get_threepid_validation_session-1647} 0.011665 sec
2020-08-08 10:00:43,900 - synapse.push.mailer - 309 - INFO - POST-615- Sending email to xxxx
2020-08-08 10:06:05,959 - synapse.http.site - 219 - WARNING - POST-615- Error processing request <XForwardedForRequest at 0x7fd4ab7bfba8 method='POST' uri='/_matrix/client/r0/account/3pid/email/requestToken' clientproto='HTTP/1.0' site=8008>: <class 'twisted.internet.error.ConnectionLost'> Connection to the other side was lost in a non-clean fashion: Connection lost.
```

And on my mailserver : 
```
Aug  7 10:04:40 mail1 postfix/submissions/smtpd[26520]: connect from matrix.6clones.net[51.15.191.9]
Aug  7 10:07:49 mail1 postfix/submissions/smtpd[30015]: SSL_accept error from matrix.6clones.net[51.15.191.9]: Connection timed out
Aug  7 10:07:49 mail1 postfix/submissions/smtpd[30015]: lost connection after CONNECT from matrix.6clones.net[51.15.191.9]
Aug  7 10:07:49 mail1 postfix/submissions/smtpd[30015]: disconnect from matrix.6clones.net[51.15.191.9] commands=0/0
```

So synapse is connecting to the mailserver, but the connection fails. I wondered if that was because of the TLS. I've configured Synapse to use 465 port, which is direct TLS, not STARTTLS method. And it seems Synapse doesn't handle it. I've changed the port to 587 and now it works.

So, new question, why Synapse can't connect to a mailserver with direct TLS connection ?
@Toniob What's the value of `email.require_transport_security` in your config?
`require_transport_security: True`

With port 465, it fails, with 587, it works.
The `require_transport_security` configuration option seems to be a bit poorly named (but is inherited from Twisted and [has a decent comment around](https://github.com/matrix-org/synapse/blob/faba873d4b91cf03c5c2d9546a6df781ce58f2cc/docs/sample_config.yaml#L1944-L1949)):

> Uncomment the following to require TLS transport security for SMTP. By default, Synapse will connect over plain text, and will then switch to TLS via STARTTLS *if the SMTP server supports it*. If this option is set, Synapse will refuse to connect unless the server supports STARTTLS.

`require_transport_security` means that STARTTLS is still used, but it requires it to succeed.

I do not believe there is a way to configure the initial connection to the SMTP server use TLS. (Note that Synapse uses [Twisted's `sendmail`](https://twistedmatrix.com/documents/19.2.0/api/twisted.mail.smtp.html#sendmail) method under-the-hood, which also doesn't seem to support this configuration.)

I'll update this issue as a request for just using TLS without STARTTLS.

Resolving patch:
diff --git a/synapse/config/emailconfig.py b/synapse/config/emailconfig.py
--- a/synapse/config/emailconfig.py
+++ b/synapse/config/emailconfig.py
@@ -80,6 +80,12 @@ def read_config(self, config, **kwargs):
         self.require_transport_security = email_config.get(
             "require_transport_security", False
         )
+        self.enable_smtp_tls = email_config.get("enable_tls", True)
+        if self.require_transport_security and not self.enable_smtp_tls:
+            raise ConfigError(
+                "email.require_transport_security requires email.enable_tls to be true"
+            )
+
         if "app_name" in email_config:
             self.email_app_name = email_config["app_name"]
         else:
@@ -368,6 +374,14 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs):
           #
           #require_transport_security: true
 
+          # Uncomment the following to disable TLS for SMTP.
+          #
+          # By default, if the server supports TLS, it will be used, and the server
+          # must present a certificate that is valid for 'smtp_host'. If this option
+          # is set to false, TLS will not be used.
+          #
+          #enable_tls: false
+
           # notif_from defines the "From" address to use when sending emails.
           # It must be set if email sending is enabled.
           #
diff --git a/synapse/handlers/send_email.py b/synapse/handlers/send_email.py
--- a/synapse/handlers/send_email.py
+++ b/synapse/handlers/send_email.py
@@ -16,7 +16,12 @@
 import logging
 from email.mime.multipart import MIMEMultipart
 from email.mime.text import MIMEText
-from typing import TYPE_CHECKING
+from io import BytesIO
+from typing import TYPE_CHECKING, Optional
+
+from twisted.internet.defer import Deferred
+from twisted.internet.interfaces import IReactorTCP
+from twisted.mail.smtp import ESMTPSenderFactory
 
 from synapse.logging.context import make_deferred_yieldable
 
@@ -26,19 +31,75 @@
 logger = logging.getLogger(__name__)
 
 
+async def _sendmail(
+    reactor: IReactorTCP,
+    smtphost: str,
+    smtpport: int,
+    from_addr: str,
+    to_addr: str,
+    msg_bytes: bytes,
+    username: Optional[bytes] = None,
+    password: Optional[bytes] = None,
+    require_auth: bool = False,
+    require_tls: bool = False,
+    tls_hostname: Optional[str] = None,
+) -> None:
+    """A simple wrapper around ESMTPSenderFactory, to allow substitution in tests
+
+    Params:
+        reactor: reactor to use to make the outbound connection
+        smtphost: hostname to connect to
+        smtpport: port to connect to
+        from_addr: "From" address for email
+        to_addr: "To" address for email
+        msg_bytes: Message content
+        username: username to authenticate with, if auth is enabled
+        password: password to give when authenticating
+        require_auth: if auth is not offered, fail the request
+        require_tls: if TLS is not offered, fail the reqest
+        tls_hostname: TLS hostname to check for. None to disable TLS.
+    """
+    msg = BytesIO(msg_bytes)
+
+    d: "Deferred[object]" = Deferred()
+
+    factory = ESMTPSenderFactory(
+        username,
+        password,
+        from_addr,
+        to_addr,
+        msg,
+        d,
+        heloFallback=True,
+        requireAuthentication=require_auth,
+        requireTransportSecurity=require_tls,
+        hostname=tls_hostname,
+    )
+
+    # the IReactorTCP interface claims host has to be a bytes, which seems to be wrong
+    reactor.connectTCP(smtphost, smtpport, factory, timeout=30, bindAddress=None)  # type: ignore[arg-type]
+
+    await make_deferred_yieldable(d)
+
+
 class SendEmailHandler:
     def __init__(self, hs: "HomeServer"):
         self.hs = hs
 
-        self._sendmail = hs.get_sendmail()
         self._reactor = hs.get_reactor()
 
         self._from = hs.config.email.email_notif_from
         self._smtp_host = hs.config.email.email_smtp_host
         self._smtp_port = hs.config.email.email_smtp_port
-        self._smtp_user = hs.config.email.email_smtp_user
-        self._smtp_pass = hs.config.email.email_smtp_pass
+
+        user = hs.config.email.email_smtp_user
+        self._smtp_user = user.encode("utf-8") if user is not None else None
+        passwd = hs.config.email.email_smtp_pass
+        self._smtp_pass = passwd.encode("utf-8") if passwd is not None else None
         self._require_transport_security = hs.config.email.require_transport_security
+        self._enable_tls = hs.config.email.enable_smtp_tls
+
+        self._sendmail = _sendmail
 
     async def send_email(
         self,
@@ -82,17 +143,16 @@ async def send_email(
 
         logger.info("Sending email to %s" % email_address)
 
-        await make_deferred_yieldable(
-            self._sendmail(
-                self._smtp_host,
-                raw_from,
-                raw_to,
-                multipart_msg.as_string().encode("utf8"),
-                reactor=self._reactor,
-                port=self._smtp_port,
-                requireAuthentication=self._smtp_user is not None,
-                username=self._smtp_user,
-                password=self._smtp_pass,
-                requireTransportSecurity=self._require_transport_security,
-            )
+        await self._sendmail(
+            self._reactor,
+            self._smtp_host,
+            self._smtp_port,
+            raw_from,
+            raw_to,
+            multipart_msg.as_string().encode("utf8"),
+            username=self._smtp_user,
+            password=self._smtp_pass,
+            require_auth=self._smtp_user is not None,
+            require_tls=self._require_transport_security,
+            tls_hostname=self._smtp_host if self._enable_tls else None,
         )
diff --git a/synapse/server.py b/synapse/server.py
--- a/synapse/server.py
+++ b/synapse/server.py
@@ -34,8 +34,6 @@
 )
 
 import twisted.internet.tcp
-from twisted.internet import defer
-from twisted.mail.smtp import sendmail
 from twisted.web.iweb import IPolicyForHTTPS
 from twisted.web.resource import IResource
 
@@ -442,10 +440,6 @@ def get_room_creation_handler(self) -> RoomCreationHandler:
     def get_room_shutdown_handler(self) -> RoomShutdownHandler:
         return RoomShutdownHandler(self)
 
-    @cache_in_self
-    def get_sendmail(self) -> Callable[..., defer.Deferred]:
-        return sendmail
-
     @cache_in_self
     def get_state_handler(self) -> StateHandler:
         return StateHandler(self)


Test patch:
diff --git a/tests/push/test_email.py b/tests/push/test_email.py
--- a/tests/push/test_email.py
+++ b/tests/push/test_email.py
@@ -45,14 +45,6 @@ class EmailPusherTests(HomeserverTestCase):
 
     def make_homeserver(self, reactor, clock):
 
-        # List[Tuple[Deferred, args, kwargs]]
-        self.email_attempts = []
-
-        def sendmail(*args, **kwargs):
-            d = Deferred()
-            self.email_attempts.append((d, args, kwargs))
-            return d
-
         config = self.default_config()
         config["email"] = {
             "enable_notifs": True,
@@ -75,7 +67,17 @@ def sendmail(*args, **kwargs):
         config["public_baseurl"] = "aaa"
         config["start_pushers"] = True
 
-        hs = self.setup_test_homeserver(config=config, sendmail=sendmail)
+        hs = self.setup_test_homeserver(config=config)
+
+        # List[Tuple[Deferred, args, kwargs]]
+        self.email_attempts = []
+
+        def sendmail(*args, **kwargs):
+            d = Deferred()
+            self.email_attempts.append((d, args, kwargs))
+            return d
+
+        hs.get_send_email_handler()._sendmail = sendmail
 
         return hs
 
diff --git a/tests/rest/client/v2_alpha/test_account.py b/tests/rest/client/v2_alpha/test_account.py
--- a/tests/rest/client/v2_alpha/test_account.py
+++ b/tests/rest/client/v2_alpha/test_account.py
@@ -47,12 +47,6 @@ def make_homeserver(self, reactor, clock):
         config = self.default_config()
 
         # Email config.
-        self.email_attempts = []
-
-        async def sendmail(smtphost, from_addr, to_addrs, msg, **kwargs):
-            self.email_attempts.append(msg)
-            return
-
         config["email"] = {
             "enable_notifs": False,
             "template_dir": os.path.abspath(
@@ -67,7 +61,16 @@ async def sendmail(smtphost, from_addr, to_addrs, msg, **kwargs):
         }
         config["public_baseurl"] = "https://example.com"
 
-        hs = self.setup_test_homeserver(config=config, sendmail=sendmail)
+        hs = self.setup_test_homeserver(config=config)
+
+        async def sendmail(
+            reactor, smtphost, smtpport, from_addr, to_addrs, msg, **kwargs
+        ):
+            self.email_attempts.append(msg)
+
+        self.email_attempts = []
+        hs.get_send_email_handler()._sendmail = sendmail
+
         return hs
 
     def prepare(self, reactor, clock, hs):
@@ -511,11 +514,6 @@ def make_homeserver(self, reactor, clock):
         config = self.default_config()
 
         # Email config.
-        self.email_attempts = []
-
-        async def sendmail(smtphost, from_addr, to_addrs, msg, **kwargs):
-            self.email_attempts.append(msg)
-
         config["email"] = {
             "enable_notifs": False,
             "template_dir": os.path.abspath(
@@ -530,7 +528,16 @@ async def sendmail(smtphost, from_addr, to_addrs, msg, **kwargs):
         }
         config["public_baseurl"] = "https://example.com"
 
-        self.hs = self.setup_test_homeserver(config=config, sendmail=sendmail)
+        self.hs = self.setup_test_homeserver(config=config)
+
+        async def sendmail(
+            reactor, smtphost, smtpport, from_addr, to_addrs, msg, **kwargs
+        ):
+            self.email_attempts.append(msg)
+
+        self.email_attempts = []
+        self.hs.get_send_email_handler()._sendmail = sendmail
+
         return self.hs
 
     def prepare(self, reactor, clock, hs):
diff --git a/tests/rest/client/v2_alpha/test_register.py b/tests/rest/client/v2_alpha/test_register.py
--- a/tests/rest/client/v2_alpha/test_register.py
+++ b/tests/rest/client/v2_alpha/test_register.py
@@ -509,10 +509,6 @@ def make_homeserver(self, reactor, clock):
         }
 
         # Email config.
-        self.email_attempts = []
-
-        async def sendmail(*args, **kwargs):
-            self.email_attempts.append((args, kwargs))
 
         config["email"] = {
             "enable_notifs": True,
@@ -532,7 +528,13 @@ async def sendmail(*args, **kwargs):
         }
         config["public_baseurl"] = "aaa"
 
-        self.hs = self.setup_test_homeserver(config=config, sendmail=sendmail)
+        self.hs = self.setup_test_homeserver(config=config)
+
+        async def sendmail(*args, **kwargs):
+            self.email_attempts.append((args, kwargs))
+
+        self.email_attempts = []
+        self.hs.get_send_email_handler()._sendmail = sendmail
 
         self.store = self.hs.get_datastore()