Repository: scrapy/scrapy

Issue:
Fix SMTP STARTTLS for Twisted >= 21.2.0
## Summary

The [Mail settings](https://docs.scrapy.org/en/latest/topics/email.html#topics-email-settings) don't have an option to choose a TLS version. Only to enforce upgrading connections to use SSL/TLS.
Mail servers like smtp.office365.com dropped support for TLS1.0 and TLS1.1 and now require TLS1.2: https://techcommunity.microsoft.com/t5/exchange-team-blog/new-opt-in-endpoint-available-for-smtp-auth-clients-still/ba-p/2659652 

It seems that scrapy mail doesn't support TLS1.2. The error message (with `MAIL_TLS = True`):

`[scrapy.mail] Unable to send mail: To=['user@gmail.com'] Cc=[] Subject="Test" Attachs=0- 421 b'4.7.66 TLS 1.0 and 1.1 are not supported. Please upgrade/update your client to support TLS 1.2. Visit https://aka.ms/smtp_auth_tls. [AM6P194CA0047.EURP194.PROD.OUTLOOK.COM]'` 

## Motivation

Without TLS1.2 it's not possible anymore to send mails via smtp.office365.com. An option to use TLS1.2 would fix this issue



Discussion:
Scrapy doesn't configure protocol versions explicitly when using `ESMTPSenderFactory` and neither does that class, so I would expect it to use global OpenSSL settings, including TLS 1.2 or 1.3 if supported. See https://twistedmatrix.com/documents/current/core/howto/ssl.html

Can you show the library versions printed by Scrapy?
It depends on the version of scrapy and which library version it uses for OpenSSL?
I use scrapy 2.0.1.
Scrapy prints the following when starting a job:
`[scrapy.utils.log] Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.8.2 (default, Feb 26 2020, 15:09:34) - [GCC 8.3.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Linux-4.15.0-66-generic-x86_64-with-glibc2.2.5`
Thanks for your help!
I suggest trying newer pyOpenSSL/cryptography/Twisted versions first.
I upgraded the versions to the newest:
`cryptography==36.0.1
pyOpenSSL==22.0.0
twisted==21.7.0`

Now I get a new error:
`502 Server does not support secure communication via TLS / SSL`

So it definitely looks like the issue is my mail server now. I will close this feature request.
Thank you for your quick help
Hi @wRAR I might still need your help.

The smtp server supports TLS1.2. I also tested this by creating a script with `smtplib`.
This script successfully sends my email using TLS1.2:

```
import smtplib, ssl

port = 587
smtp_server = 'smtp.office365.com'
sender_email = 'test1@test.com'
receiver_email = 'test2@test.com'
user = 'test3@test.de'
password = input('Type your password and press enter:')
message = 'test'

context = ssl.create_default_context()
context.minimum_version = ssl.TLSVersion.TLSv1_2
with smtplib.SMTP(smtp_server, port) as server:
    server.ehlo()  # Can be omitted
    server.starttls(context=context)
    server.ehlo()  # Can be omitted
    server.login(user, password)
    server.sendmail(sender_email, receiver_email, message)
```

I can't seem to get it to work with `scrapy`. I created a new scrapy project with a newer version(`2.5.1`) as well.
These are all my versions:
`[scrapy.utils.log] INFO: Versions: lxml 4.7.1.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.7.0, Python 3.9.7 (default, Sep  3 2021, 12:37:55) - [Clang 12.0.5 (clang-1205.0.22.9)], pyOpenSSL 22.0.0 (OpenSSL 1.1.1m  14 Dec 2021), cryptography 36.0.1, Platform macOS-11.6.2-x86_64-i386-64bit`

I still get the error:
`[scrapy.mail] ERROR: Unable to send mail: To=['test@test.com'] Cc=[] Subject="tls-test" Attachs=0- 502 Server does not support secure communication via TLS / SSL`

Do you have an idea what I could try next?
Are you using MAIL_TLS and port 587?
Both yes
I'll try to check this later
Hi @wRAR, sorry to bother you. 
Do you have some more ideas of what I could try? 

Actually "Server does not support secure communication via TLS / SSL" is a client error from Twisted and it's misleading because in fact it's the client that doesn't support STARTTLS.

Since Twisted 21.2.0 (https://github.com/twisted/twisted/commit/abef1218a9013223ee237d8179b4705cbfd716c0), `ESMTPSenderFactory` needs to be passed `hostname` to use STARTTLS. Unfortunately this isn't documented, but the whole of `ESMTPSenderFactory` isn't, the public interface seems to be `sendmail()` which we don't use.

The fix to this seems to be just passing `hostname=self.smtphost` to `ESMTPSenderFactory()` in `MailSender._sendmail()`, but only for Twisted 21.2.0 and above.

Resolving patch:
diff --git a/scrapy/mail.py b/scrapy/mail.py
--- a/scrapy/mail.py
+++ b/scrapy/mail.py
@@ -12,7 +12,9 @@
 from email.utils import formatdate
 from io import BytesIO
 
+from twisted.python.versions import Version
 from twisted.internet import defer, ssl
+from twisted import version as twisted_version
 
 from scrapy.utils.misc import arg_to_iter
 from scrapy.utils.python import to_bytes
@@ -126,16 +128,11 @@ def _sent_failed(self, failure, to, cc, subject, nattachs):
                       'mailattachs': nattachs, 'mailerr': errstr})
 
     def _sendmail(self, to_addrs, msg):
-        # Import twisted.mail here because it is not available in python3
         from twisted.internet import reactor
-        from twisted.mail.smtp import ESMTPSenderFactory
         msg = BytesIO(msg)
         d = defer.Deferred()
-        factory = ESMTPSenderFactory(
-            self.smtpuser, self.smtppass, self.mailfrom, to_addrs, msg, d,
-            heloFallback=True, requireAuthentication=False, requireTransportSecurity=self.smtptls,
-        )
-        factory.noisy = False
+
+        factory = self._create_sender_factory(to_addrs, msg, d)
 
         if self.smtpssl:
             reactor.connectSSL(self.smtphost, self.smtpport, factory, ssl.ClientContextFactory())
@@ -143,3 +140,20 @@ def _sendmail(self, to_addrs, msg):
             reactor.connectTCP(self.smtphost, self.smtpport, factory)
 
         return d
+
+    def _create_sender_factory(self, to_addrs, msg, d):
+        from twisted.mail.smtp import ESMTPSenderFactory
+
+        factory_keywords = {
+            'heloFallback': True,
+            'requireAuthentication': False,
+            'requireTransportSecurity': self.smtptls
+        }
+
+        # Newer versions of twisted require the hostname to use STARTTLS
+        if twisted_version >= Version('twisted', 21, 2, 0):
+            factory_keywords['hostname'] = self.smtphost
+
+        factory = ESMTPSenderFactory(self.smtpuser, self.smtppass, self.mailfrom, to_addrs, msg, d, **factory_keywords)
+        factory.noisy = False
+        return factory


Test patch:
diff --git a/tests/test_mail.py b/tests/test_mail.py
--- a/tests/test_mail.py
+++ b/tests/test_mail.py
@@ -4,6 +4,11 @@
 from io import BytesIO
 from email.charset import Charset
 
+from twisted.internet._sslverify import ClientTLSOptions
+from twisted.internet.ssl import ClientContextFactory
+from twisted.python.versions import Version
+from twisted.internet import defer
+from twisted import version as twisted_version
 from scrapy.mail import MailSender
 
 
@@ -121,6 +126,17 @@ def test_send_attach_utf8(self):
         self.assertEqual(text.get_charset(), Charset('utf-8'))
         self.assertEqual(attach.get_payload(decode=True).decode('utf-8'), body)
 
+    def test_create_sender_factory_with_host(self):
+        mailsender = MailSender(debug=False, smtphost='smtp.testhost.com')
+
+        factory = mailsender._create_sender_factory(to_addrs=['test@scrapy.org'], msg='test', d=defer.Deferred())
+
+        context = factory.buildProtocol('test@scrapy.org').context
+        if twisted_version >= Version('twisted', 21, 2, 0):
+            self.assertIsInstance(context, ClientTLSOptions)
+        else:
+            self.assertIsInstance(context, ClientContextFactory)
+
 
 if __name__ == "__main__":
     unittest.main()