Repository: scrapy/scrapy

Issue:
Wrong charset handling in MailSender
Looks like passing `charset='utf-8'` makes a plain text message with `Content-Transfer-Encoding: base64` which then can't be read.

At https://github.com/scrapy/scrapy/blob/5a75b14a5fbbbd37c14aa7317761655ac7706b70/scrapy/mail.py#L81 `set_charset` is called but as the payload is not set yet, the underlying class just sets the headers. When later `set_payload` is called, it doesn't do any encoding, but the `Content-Transfer-Encoding` is already set. Looks like the fix should be passing the encoding to `set_payload` too, like was proposed in #3722 (and `set_charset` may be safe to be removed, not sure about this). Note that we have tests but they don't catch this.

Note also, that all of this seems to be compat code according to the Python docs.
Wrong charset handling in MailSender
Looks like passing `charset='utf-8'` makes a plain text message with `Content-Transfer-Encoding: base64` which then can't be read.

At https://github.com/scrapy/scrapy/blob/5a75b14a5fbbbd37c14aa7317761655ac7706b70/scrapy/mail.py#L81 `set_charset` is called but as the payload is not set yet, the underlying class just sets the headers. When later `set_payload` is called, it doesn't do any encoding, but the `Content-Transfer-Encoding` is already set. Looks like the fix should be passing the encoding to `set_payload` too, like was proposed in #3722 (and `set_charset` may be safe to be removed, not sure about this). Note that we have tests but they don't catch this.

Note also, that all of this seems to be compat code according to the Python docs.
Fix charset handling in MailSender
Changes:
  
Implementation
- Set encoding utf-8 for payload inside send

Fixes #5096, closes #5118


Discussion:
Hello, i am new here and i would like to try and fix this bug. Can you please give me some guidlines and tell me how to recreate this issue?
Sure, you can go ahead.
> Can you please give me some guidlines and tell me how to recreate this issue?

The documentation about using the `MailSender` class is at https://docs.scrapy.org/en/latest/topics/email.html . Then it should be a matter of following the instructions on the issue description (“passing charset='utf-8'”), and verify that indeed the resulting email message cannot be read.
Hello, I have made a small change and now when you send an email with utf-8 encoding the body of the email looks fine and does not have a base64 encoding. Do you want me to do a pull request or maybe a draft pull request to see what I have done and give me feedback?
> Do you want me to do a pull request or maybe a draft pull request to see what I have done and give me feedback?

That would be great! Feel free to make it a regular pull request if it’s ready for review and discussion.
Hello, i am new here and i would like to try and fix this bug. Can you please give me some guidlines and tell me how to recreate this issue?
Sure, you can go ahead.
> Can you please give me some guidlines and tell me how to recreate this issue?

The documentation about using the `MailSender` class is at https://docs.scrapy.org/en/latest/topics/email.html . Then it should be a matter of following the instructions on the issue description (“passing charset='utf-8'”), and verify that indeed the resulting email message cannot be read.
Hello, I have made a small change and now when you send an email with utf-8 encoding the body of the email looks fine and does not have a base64 encoding. Do you want me to do a pull request or maybe a draft pull request to see what I have done and give me feedback?
> Do you want me to do a pull request or maybe a draft pull request to see what I have done and give me feedback?

That would be great! Feel free to make it a regular pull request if it’s ready for review and discussion.


Resolving patch:
diff --git a/scrapy/mail.py b/scrapy/mail.py
--- a/scrapy/mail.py
+++ b/scrapy/mail.py
@@ -96,10 +96,9 @@ def send(
             rcpts.extend(cc)
             msg["Cc"] = COMMASPACE.join(cc)
 
-        if charset:
-            msg.set_charset(charset)
-
         if attachs:
+            if charset:
+                msg.set_charset(charset)
             msg.attach(MIMEText(body, "plain", charset or "us-ascii"))
             for attach_name, mimetype, f in attachs:
                 part = MIMEBase(*mimetype.split("/"))
@@ -110,7 +109,7 @@ def send(
                 )
                 msg.attach(part)
         else:
-            msg.set_payload(body)
+            msg.set_payload(body, charset)
 
         if _callback:
             _callback(to=to, subject=subject, body=body, cc=cc, attach=attachs, msg=msg)


Test patch:
diff --git a/tests/test_mail.py b/tests/test_mail.py
--- a/tests/test_mail.py
+++ b/tests/test_mail.py
@@ -111,7 +111,7 @@ def test_send_utf8(self):
 
         msg = self.catched_msg["msg"]
         self.assertEqual(msg["subject"], subject)
-        self.assertEqual(msg.get_payload(), body)
+        self.assertEqual(msg.get_payload(decode=True).decode("utf-8"), body)
         self.assertEqual(msg.get_charset(), Charset("utf-8"))
         self.assertEqual(msg.get("Content-Type"), 'text/plain; charset="utf-8"')