Repository: CTFd/CTFd

Issue:
Email cannot be sent if CTF name has an accent
<!--
If this is a bug report please fill out the template below.

If this is a feature request please describe the behavior that you'd like to see.
-->

**Environment**:

  - CTFd Version/Commit: 2.3.2
  - Operating System: Debian 10.3, Python 3.7.3
  - Web Browser and Version: Firefox Nightly, 76.0a1 (2020-04-06)

**What happened?**

After a fresh install of CTFd we set the competition name to "Inté CTF". We configure the email server to use our Postfix relay, and test it with the "forgotten password" feature.
The email never arrives and is bounced by relay servers because the `From` header is invalid.

The actual issue is because the `From` header has the form "CTF name <email address>" (see [here](https://github.com/CTFd/CTFd/blob/master/CTFd/utils/email/smtp.py#L25)). Once encoded in the `msg.as_string()` ([there](https://github.com/CTFd/CTFd/blob/master/CTFd/utils/email/smtp.py#L55)) it is entirely encoded in UTF-8 looking like this `=?utf-8?q?Int=C3=A9_CTF_=3Cctf-noreply=40example=2Ecom=3E?=`, which is invalid against RFC 2407 section 5 ("An 'encoded-word' MUST NOT appear in any portion of an 'addr-spec'"). A correct form would be `=?utf-8?q?Int=C3=A9?= CTF <ctf-noreply@example.com>`.

I have a patch ready but it uses `from email.message import EmailMessage` which is available in Python 3.6+. I'll be happy to open a PR if you confirm me that support for lower Python version can be dropped ^^

<details>
<summary>Patch for `CTFd/utils/email/smtp.py`</summary>

```diff
-from email.mime.text import MIMEText
+from email.message import EmailMessage


# around line 50
-        msg = MIMEText(text)
+        msg = EmailMessage()
+        msg.set_content(text)
         msg["Subject"] = subject
         msg["From"] = mailfrom_addr
         msg["To"] = addr
 
-        smtp.sendmail(msg["From"], [msg["To"]], msg.as_string())
+        smtp.send_message(msg)
```

</details>

**How to reproduce your issue**

- setup a new instance of CTFd
- configure the competition name with non-ascii characters
- configure the email server to use a "real" mail server (not Mailgun)
- try to send an email (with the "forgotten password" for example)
- see the mail server logs

<details>
<summary>
You can see the difference between both methods (`MIMEText` and `EmailMessage`) with the following snippet
</summary>

```python
from email.mime.text import MIMEText
msg = MIMEText("This is a message with accents éèçà")
msg["From"] = "René Côti <rene.coti@example.com>"
msg["To"] = "recipient@example.com"
msg.as_string()

from email.message import EmailMessage
msg = EmailMessage()
msg.set_content("This is a message with accents éèçà")
msg["From"] = "René Côti <rene.coti@example.com>"
msg["To"] = "recipient@example.com"
msg.as_string()
```

</details>

**Any associated stack traces or error logs**
```
# Sent to a gmail recipient
opendkim[839]: 278839FE69: can't parse From: header value ' =?utf-8?q?Int=C3=A9_CTF_=3Cctf-noreply=40example=2Ecom=3E?='
postfix/smtp[8076]: 278839FE69: to=<recipient@gmail.com>, relay=gmail-smtp-in.l.google.com[74.125.206.27]:25, delay=0.75, delays=0.31/0/0.23/0.21, dsn=5.7.1, status=bounced (host gmail-smtp-in.l.google.com[74.125.206.27] said: 550-5.7.1 [185.132.74.134      14] Messages missing a valid address in From: 550 5.7.1 header, or having no From: header, are not accepted. p64si4801636wmp.124 - gsmtp (in reply to end of DATA command))

# Sent to another recipient, with another relay server
postfix/smtpd[7858]: NOQUEUE: reject: RCPT from mailcube2.domain.fr[193.51.52.6]: 550 5.1.1 <ctf-noreply@example.com>: Recipient address rejected: User unknown
 in virtual mailbox table; from=<> to=<ctf-noreply@example.com> proto=ESMTP helo=<mailcube2.domain.fr>
```


Discussion:
Hmmmm I haven't dug into this yet but I just tried to reproduce it using a hosted CTFd instance which uses SMTP and I wasn't able to reproduce. Messages with UTF-8 went through fine. 

Can you replicate this issue with any other SMTP provider? Maybe try gmail? Or if there's some kind of email validator?

I do believe that this is probably an issue but we can't drop Python 2 support yet. Is there a Python 2 compatible fix that would work here? 

Just to give you an update, I did test using my personal Gmail account as the SMTP server, and it did work correctly. Maybe Gmail is more lax on the headers parsing, maybe Postfix is plain buggy, I don't know.

What I did observe though is that the `MIMEText.as_string()` behavior is different on Python 2: it only encodes the "name" part of the From header, giving something like `Int\xc3\xa9 CTF <ctf-noreply@example.com>`. I did not test to see if it was accepted by our Postfix server.

If it is valid, a simple workaround would be to run different code on Python 2 and 3, like this:
```python
import sys
if sys.version_info.major < 3:
from email.mime.text import MIMEText
    from email.mime.text import MIMEText
else:
    from email.message import EmailMessage


    try:
        smtp = get_smtp(**data)
        if sys.version_info.major < 3:
            msg = MIMEText(text)
        else:
            msg = EmailMessage()
            msg.set_content(text)
        msg["Subject"] = subject
        msg["From"] = mailfrom_addr
        msg["To"] = addr

        if sys.version_info.major < 3:
            smtp.sendmail(msg["From"], [msg["To"]], msg.as_string())
        else:
            smtp.send_message(msg)
        smtp.quit()
        return True, "Email sent"
```

Not sure if this is clean enough, or if it's better to wait for the 3.0 (which will drop Python 2 support I guess?)

As a side note, during my investigation I got lost in the stdlib code ^^', and the code for header encoding is a lot more complex on an EmailMessage in Python 3, it has specific parsers for different headers (From is not the same as Subject for example). I tried to reproduce a similar behavior using `email.header` and encoding part of the address manually, but without result yet.
Thanks for doing the investigation on this! 

I think as long as there's a fallback for Python 2 this can go in earlier than v3. It would be nice to understand what the underlying issue is but it's good to know that it's fixed with Python 3. 

As I mentioned before, I think I've seen errors in this SMTP email sending area so I'm willing to push this up and then remove the legacy Python 2 code once v3 hits. The only concern here is that it's a little more difficult to test code that calls two different sending functions. If `EmailMessage` resolves the other issues I've seen I'll definitely patch this before v3. 

As an aside, I would probably argue that this is something to do with Postfix. I imagine gmail sees weird clients and knows to be a little more flexible. 
I *think* the Python 3 version could also work with the `smtp.sendmail(msg["From"], [msg["To"]], msg.as_string())`, so less code change
Perhaps but the docs suggest that send_message has a few other pieces of code in it that wrap sendmail. I imagine we'll just have to do both. 
EDIT: Actually my patch might have been wrong but let me know if this fixes the issue for you. This might be completely wrong. 

@nymous Looking at the test output from switching to EmailMessage, I suspect maybe MIMEText needs to be set to use utf-8 as the email charset. 

Do you think you can try `msg = MIMEText(text, _charset="utf-8")` to see if that would work on your Postfix server? 

Test output:

```
E   AssertionError: Expected call: sendmail('CTFd <noreply@ctfd.io>', ['user@user.com'], 'Content-Type: text/plain; charset="us-ascii"\nMIME-Version: 1.0\nContent-Transfer-Encoding: 7bit\nSubject: Message from CTFd\nFrom: CTFd <noreply@ctfd.io>\nTo: user@user.com\n\nthis is a test')
E   Actual call: sendmail('CTFd <noreply@ctfd.io>', ['user@user.com'], 'Content-Type: text/plain; charset="utf-8"\nContent-Transfer-Encoding: 7bit\nMIME-Version: 1.0\nSubject: Message from CTFd\nFrom: CTFd <noreply@ctfd.io>\nTo: user@user.com\n\nthis is a test\n')
```
Hmm setting the charset actually may work as it appears it's the only difference between sendmail and smtp_message. If you have some way to reproduce this that you could share with me, that would be useful to nail down the issue. 

Python 3 with smtp_message
```
tests/utils/test_email.py Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Subject: Message from CTFd
From: Inté CTF <CTFd>
To: user@user.com

this is a test
```

Python 2 with sendmail
```
tests/utils/test_email.py From nobody Fri Apr 17 15:25:49 2020
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: Message from CTFd
From: Inté CTF <CTFd>
To: user@user.com

this is a test
```


I was able to test this out and EmailMessage resolves a lot of problems. Setting the charset doesn't really help that much and doesn't resolve this problem either. 

With that in mind, this is one of those warts that's going to have to have a version check and then fixed in v3. 

Resolving patch:
diff --git a/CTFd/utils/config/__init__.py b/CTFd/utils/config/__init__.py
--- a/CTFd/utils/config/__init__.py
+++ b/CTFd/utils/config/__init__.py
@@ -53,14 +53,14 @@ def can_send_mail():
 
 
 def get_mail_provider():
-    if app.config.get("MAIL_SERVER") and app.config.get("MAIL_PORT"):
-        return "smtp"
     if get_config("mail_server") and get_config("mail_port"):
         return "smtp"
-    if app.config.get("MAILGUN_API_KEY") and app.config.get("MAILGUN_BASE_URL"):
-        return "mailgun"
     if get_config("mailgun_api_key") and get_config("mailgun_base_url"):
         return "mailgun"
+    if app.config.get("MAIL_SERVER") and app.config.get("MAIL_PORT"):
+        return "smtp"
+    if app.config.get("MAILGUN_API_KEY") and app.config.get("MAILGUN_BASE_URL"):
+        return "mailgun"
 
 
 def mailgun():
diff --git a/CTFd/utils/email/smtp.py b/CTFd/utils/email/smtp.py
--- a/CTFd/utils/email/smtp.py
+++ b/CTFd/utils/email/smtp.py
@@ -1,5 +1,9 @@
+import six
 import smtplib
 from email.mime.text import MIMEText
+
+if six.PY3:
+    from email.message import EmailMessage
 from socket import timeout
 
 from CTFd.utils import get_app_config, get_config
@@ -47,12 +51,22 @@ def sendmail(addr, text, subject):
 
     try:
         smtp = get_smtp(**data)
-        msg = MIMEText(text)
+
+        if six.PY2:
+            msg = MIMEText(text)
+        else:
+            msg = EmailMessage()
+            msg.set_content(text)
+
         msg["Subject"] = subject
         msg["From"] = mailfrom_addr
         msg["To"] = addr
 
-        smtp.sendmail(msg["From"], [msg["To"]], msg.as_string())
+        if six.PY2:
+            smtp.sendmail(msg["From"], [msg["To"]], msg.as_string())
+        else:
+            smtp.send_message(msg)
+
         smtp.quit()
         return True, "Email sent"
     except smtplib.SMTPException as e:


Test patch:
diff --git a/tests/users/test_auth.py b/tests/users/test_auth.py
--- a/tests/users/test_auth.py
+++ b/tests/users/test_auth.py
@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
+import six
 from freezegun import freeze_time
 from mock import patch
 
@@ -304,8 +305,11 @@ def test_user_can_confirm_email(mock_smtp):
         r = client.get("http://localhost/confirm")
         assert "Need to resend the confirmation email?" in r.get_data(as_text=True)
 
-        # smtp.sendmail was called
-        mock_smtp.return_value.sendmail.assert_called()
+        # smtp send message function was called
+        if six.PY2:
+            mock_smtp.return_value.sendmail.assert_called()
+        else:
+            mock_smtp.return_value.send_message.assert_called()
 
         with client.session_transaction() as sess:
             data = {"nonce": sess.get("nonce")}
@@ -334,6 +338,9 @@ def test_user_can_reset_password(mock_smtp):
     """Test that a user is capable of resetting their password"""
     from email.mime.text import MIMEText
 
+    if six.PY3:
+        from email.message import EmailMessage
+
     app = create_ctfd()
     with app.app_context(), freeze_time("2012-01-14 03:21:34"):
         # Set CTFd to send emails
@@ -369,7 +376,13 @@ def test_user_can_reset_password(mock_smtp):
                 "http://localhost/reset_password/InVzZXJAdXNlci5jb20i.TxD0vg.28dY_Gzqb1TH9nrcE_H7W8YFM-U"
             )
             ctf_name = get_config("ctf_name")
-            email_msg = MIMEText(msg)
+
+            if six.PY2:
+                email_msg = MIMEText(msg)
+            else:
+                email_msg = EmailMessage()
+                email_msg.set_content(msg)
+
             email_msg["Subject"] = "Password Reset Request from {ctf_name}".format(
                 ctf_name=ctf_name
             )
@@ -377,9 +390,15 @@ def test_user_can_reset_password(mock_smtp):
             email_msg["To"] = to_addr
 
             # Make sure that the reset password email is sent
-            mock_smtp.return_value.sendmail.assert_called_with(
-                from_addr, [to_addr], email_msg.as_string()
-            )
+            if six.PY2:
+                mock_smtp.return_value.sendmail.assert_called_with(
+                    from_addr, [to_addr], email_msg.as_string()
+                )
+            else:
+                mock_smtp.return_value.send_message.assert_called()
+                assert str(mock_smtp.return_value.send_message.call_args[0][0]) == str(
+                    email_msg
+                )
 
             # Get user's original password
             user = Users.query.filter_by(email="user@user.com").first()
diff --git a/tests/utils/test_email.py b/tests/utils/test_email.py
--- a/tests/utils/test_email.py
+++ b/tests/utils/test_email.py
@@ -1,5 +1,9 @@
+import six
 from email.mime.text import MIMEText
 
+if six.PY3:
+    from email.message import EmailMessage
+
 import requests
 from freezegun import freeze_time
 from mock import Mock, patch
@@ -34,14 +38,25 @@ def test_sendmail_with_smtp_from_config_file(mock_smtp):
         sendmail(to_addr, msg)
 
         ctf_name = get_config("ctf_name")
-        email_msg = MIMEText(msg)
+        if six.PY2:
+            email_msg = MIMEText(msg)
+        else:
+            email_msg = EmailMessage()
+            email_msg.set_content(msg)
+
         email_msg["Subject"] = "Message from {0}".format(ctf_name)
         email_msg["From"] = from_addr
         email_msg["To"] = to_addr
 
-        mock_smtp.return_value.sendmail.assert_called_once_with(
-            from_addr, [to_addr], email_msg.as_string()
-        )
+        if six.PY2:
+            mock_smtp.return_value.sendmail.assert_called_with(
+                from_addr, [to_addr], email_msg.as_string()
+            )
+        else:
+            mock_smtp.return_value.send_message.assert_called()
+            assert str(mock_smtp.return_value.send_message.call_args[0][0]) == str(
+                email_msg
+            )
     destroy_ctfd(app)
 
 
@@ -66,14 +81,24 @@ def test_sendmail_with_smtp_from_db_config(mock_smtp):
         sendmail(to_addr, msg)
 
         ctf_name = get_config("ctf_name")
-        email_msg = MIMEText(msg)
+        if six.PY2:
+            email_msg = MIMEText(msg)
+        else:
+            email_msg = EmailMessage()
+            email_msg.set_content(msg)
         email_msg["Subject"] = "Message from {0}".format(ctf_name)
         email_msg["From"] = from_addr
         email_msg["To"] = to_addr
 
-        mock_smtp.return_value.sendmail.assert_called_once_with(
-            from_addr, [to_addr], email_msg.as_string()
-        )
+        if six.PY2:
+            mock_smtp.return_value.sendmail.assert_called_with(
+                from_addr, [to_addr], email_msg.as_string()
+            )
+        else:
+            mock_smtp.return_value.send_message.assert_called()
+            assert str(mock_smtp.return_value.send_message.call_args[0][0]) == str(
+                email_msg
+            )
     destroy_ctfd(app)
 
 
@@ -85,18 +110,11 @@ def test_sendmail_with_mailgun_from_config_file(fake_post_request):
         app.config["MAILGUN_API_KEY"] = "key-1234567890-file-config"
         app.config["MAILGUN_BASE_URL"] = "https://api.mailgun.net/v3/file.faked.com"
 
-        from_addr = get_config("mailfrom_addr") or app.config.get("MAILFROM_ADDR")
         to_addr = "user@user.com"
         msg = "this is a test"
 
         sendmail(to_addr, msg)
 
-        ctf_name = get_config("ctf_name")
-        email_msg = MIMEText(msg)
-        email_msg["Subject"] = "Message from {0}".format(ctf_name)
-        email_msg["From"] = from_addr
-        email_msg["To"] = to_addr
-
         fake_response = Mock()
         fake_post_request.return_value = fake_response
         fake_response.status_code = 200
@@ -132,18 +150,11 @@ def test_sendmail_with_mailgun_from_db_config(fake_post_request):
         set_config("mailgun_api_key", "key-1234567890-db-config")
         set_config("mailgun_base_url", "https://api.mailgun.net/v3/db.faked.com")
 
-        from_addr = get_config("mailfrom_addr") or app.config.get("MAILFROM_ADDR")
         to_addr = "user@user.com"
         msg = "this is a test"
 
         sendmail(to_addr, msg)
 
-        ctf_name = get_config("ctf_name")
-        email_msg = MIMEText(msg)
-        email_msg["Subject"] = "Message from {0}".format(ctf_name)
-        email_msg["From"] = from_addr
-        email_msg["To"] = to_addr
-
         fake_response = Mock()
         fake_post_request.return_value = fake_response
         fake_response.status_code = 200
@@ -196,18 +207,26 @@ def test_verify_email(mock_smtp):
         )
 
         ctf_name = get_config("ctf_name")
-        email_msg = MIMEText(msg)
+        if six.PY2:
+            email_msg = MIMEText(msg)
+        else:
+            email_msg = EmailMessage()
+            email_msg.set_content(msg)
         email_msg["Subject"] = "Confirm your account for {ctf_name}".format(
             ctf_name=ctf_name
         )
         email_msg["From"] = from_addr
         email_msg["To"] = to_addr
 
-        # Need to freeze time to predict the value of the itsdangerous token.
-        # For now just assert that sendmail was called.
-        mock_smtp.return_value.sendmail.assert_called_with(
-            from_addr, [to_addr], email_msg.as_string()
-        )
+        if six.PY2:
+            mock_smtp.return_value.sendmail.assert_called_with(
+                from_addr, [to_addr], email_msg.as_string()
+            )
+        else:
+            mock_smtp.return_value.send_message.assert_called()
+            assert str(mock_smtp.return_value.send_message.call_args[0][0]) == str(
+                email_msg
+            )
     destroy_ctfd(app)
 
 
@@ -233,16 +252,24 @@ def test_successful_registration_email(mock_smtp):
 
         msg = "You've successfully registered for CTFd!"
 
-        email_msg = MIMEText(msg)
+        if six.PY2:
+            email_msg = MIMEText(msg)
+        else:
+            email_msg = EmailMessage()
+            email_msg.set_content(msg)
         email_msg["Subject"] = "Successfully registered for {ctf_name}".format(
             ctf_name=ctf_name
         )
         email_msg["From"] = from_addr
         email_msg["To"] = to_addr
 
-        # Need to freeze time to predict the value of the itsdangerous token.
-        # For now just assert that sendmail was called.
-        mock_smtp.return_value.sendmail.assert_called_with(
-            from_addr, [to_addr], email_msg.as_string()
-        )
+        if six.PY2:
+            mock_smtp.return_value.sendmail.assert_called_with(
+                from_addr, [to_addr], email_msg.as_string()
+            )
+        else:
+            mock_smtp.return_value.send_message.assert_called()
+            assert str(mock_smtp.return_value.send_message.call_args[0][0]) == str(
+                email_msg
+            )
     destroy_ctfd(app)