Repository: pypi/warehouse

Issue:
PyPI sends emails with unicode in header
**Describe the bug**
<!-- A clear and concise description the bug -->
I created a new account on pypi.org, but I am not receiving any email verification link by email - not after accouunt creation and also not after resending the verification link (even if pypi.org claims "Verification email for nicolas@seita.nl resent").

test.pypi.org works for me, using the same account name and email address. I deleted and recreated the account on both pypi.org and test.pypi.org - same behaviour (well, the initial email did sometimes not arrive on test.pypi.org, but the the verification resend always arrived right away).

**Expected behavior**
<!-- A clear and concise description of what you expected to happen -->
I should get an email token after registering, and also when I resend the verification mail from the account page. I know it might take a while, but I am experiencing this since Friday.

**To Reproduce**
<!-- Steps to reproduce the bug, or a link to PyPI where the bug is visible -->
Register on pypi.org, wait for email. Resend the verification email from the account settings page, wait again.

**My Platform**
<!--
    Any details about your specific platform:
    * If the problem is in the browser, what browser, version, and OS?
    * If the problem is with a command-line tool, what version of that tool?
    * If the problem is with connecting to PyPI, include some details about
      your network, including SSL/TLS implementation in use, internet service
      provider, and if there are any firewalls or proxies in use.
-->
Firefox 52



Discussion:
Hello @nhoening! PyPI received bounces for the address you specified which led to it being placed on the suppression list.

I've removed it from the suppression list, can you try again?
Yes, that did it. Thanks for the quick help!
@nhoening in taking a closer look, @di noticed that your domain's mail transfer agent was rejecting the `To` field due to the ö in your last name:

```
emailAddress: Nicolas Höning <nicolas@seita.nl>
diagnosticCode: smtp; 550 To contains invalid characters.
```

Warehouse constructs `To` headers using:

https://github.com/pypa/warehouse/blob/40fbc32fef7c7ffe41cd18f3f8951578555db1aa/warehouse/email/__init__.py#L25-L33

So it appears that either:
  1) Your mail transfer agent no supports unicode
  2) You temporarily changed your profile display name, as it now contains the ö character

Can you confirm if either of these occurred? I may be imagining things, but I briefly recall seeing the non-unicode "variant" of your name in your profile when taking a look initially.

My concern is that future email may not be deliverable.
I can confirm that I tried to rule out the umlaut as a problem. I had used it initially and after a day or two, I went on testing without the umlaut. test.pypi.org also did not work initially, until I removed it, but I thought it must be a glitch. It makes more sense now.

So, either I should talk to my email provider, or you guys decide that [RFC-5322 Article 2.2](https://tools.ietf.org/html/rfc5322#section-2.22) (see below) applies to this problem (I wouldn't be expert enough to tell) and maybe encode header text.

> Header fields are lines beginning with a field name, followed by a
>    colon (":"), followed by a field body, and terminated by CRLF.  A
>    field name MUST be composed of printable US-ASCII characters (i.e.,
>    characters that have values between 33 and 126, inclusive), except
>    colon.  A field body may be composed of printable US-ASCII characters
>    as well as the space (SP, ASCII value 32) and horizontal tab (HTAB,
>    ASCII value 9) characters (together known as the white space
>    characters, WSP).


Reopening this because I think it's something that can be fixed on our end.

Resolving patch:
diff --git a/warehouse/email/services.py b/warehouse/email/services.py
--- a/warehouse/email/services.py
+++ b/warehouse/email/services.py
@@ -11,7 +11,9 @@
 # limitations under the License.
 
 from email.headerregistry import Address
-from email.utils import parseaddr
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+from email.utils import parseaddr, formataddr
 
 from pyramid_mailer import get_mailer
 from pyramid_mailer.message import Message
@@ -68,13 +70,25 @@ def create_service(cls, context, request):
         )
 
     def send(self, subject, body, *, recipient):
-        resp = self._client.send_email(
+        message = MIMEMultipart("mixed")
+        message["Subject"] = subject
+        message["From"] = self._sender
+
+        # The following is necessary to support friendly names with Unicode characters,
+        # otherwise the entire value will get encoded and will not be accepted by SES:
+        #
+        #   >>> parseaddr("Fööbar <foo@bar.com>")
+        #   ('Fööbar', 'foo@bar.com')
+        #   >>> formataddr(_)
+        #   '=?utf-8?b?RsO2w7ZiYXI=?= <foo@bar.com>'
+        message["To"] = formataddr(parseaddr(recipient))
+
+        message.attach(MIMEText(body, "plain", "utf-8"))
+
+        resp = self._client.send_raw_email(
             Source=self._sender,
-            Destination={"ToAddresses": [recipient]},
-            Message={
-                "Subject": {"Data": subject, "Charset": "UTF-8"},
-                "Body": {"Text": {"Data": body, "Charset": "UTF-8"}},
-            },
+            Destinations=[recipient],
+            RawMessage={"Data": message.as_string()},
         )
 
         self._db.add(


Test patch:
diff --git a/tests/unit/email/test_services.py b/tests/unit/email/test_services.py
--- a/tests/unit/email/test_services.py
+++ b/tests/unit/email/test_services.py
@@ -102,9 +102,17 @@ def test_creates_service(self):
         assert sender._db is request.db
 
     def test_send(self, db_session):
+        # Determine what the random boundary token will be
+        import random
+        import sys
+
+        random.seed(42)
+        token = random.randrange(sys.maxsize)
+        random.seed(42)
+
         resp = {"MessageId": str(uuid.uuid4()) + "-ses"}
         aws_client = pretend.stub(
-            send_email=pretend.call_recorder(lambda *a, **kw: resp)
+            send_raw_email=pretend.call_recorder(lambda *a, **kw: resp)
         )
         sender = SESEmailSender(
             aws_client, sender="DevPyPI <noreply@example.com>", db=db_session
@@ -116,13 +124,27 @@ def test_send(self, db_session):
             recipient="FooBar <somebody@example.com>",
         )
 
-        assert aws_client.send_email.calls == [
+        assert aws_client.send_raw_email.calls == [
             pretend.call(
                 Source="DevPyPI <noreply@example.com>",
-                Destination={"ToAddresses": ["FooBar <somebody@example.com>"]},
-                Message={
-                    "Subject": {"Data": "This is a Subject", "Charset": "UTF-8"},
-                    "Body": {"Text": {"Data": "This is a Body", "Charset": "UTF-8"}},
+                Destinations=["FooBar <somebody@example.com>"],
+                RawMessage={
+                    "Data": (
+                        'Content-Type: multipart/mixed; boundary="==============={token}=="\n'  # noqa
+                        "MIME-Version: 1.0\n"
+                        "Subject: This is a Subject\n"
+                        "From: DevPyPI <noreply@example.com>\n"
+                        "To: FooBar <somebody@example.com>\n"
+                        "\n"
+                        "--==============={token}==\n"
+                        'Content-Type: text/plain; charset="utf-8"\n'
+                        "MIME-Version: 1.0\n"
+                        "Content-Transfer-Encoding: base64\n"
+                        "\n"
+                        "VGhpcyBpcyBhIEJvZHk=\n"
+                        "\n"
+                        "--==============={token}==--\n"
+                    ).format(token=token)
                 },
             )
         ]