Repository: python/cpython (Python)

Issue:
email: folding of quoted string in display_name violates RFC
BPO | [36041](https://bugs.python.org/issue36041)
--- | :---
Nosy | @warsaw, @bitdancer, @SwampWalker
PRs | <li>python/cpython#12054</li>
Files | <li>[address_folding_bug.py](https://bugs.python.org/file48155/address_folding_bug.py "Uploaded as text/plain at 2019-02-19.16:45:38 by @SwampWalker"): Illustration of issue, monkey patch fix of issue, illustration of fix</li><li>[address_folding_bug.py](https://bugs.python.org/file48158/address_folding_bug.py "Uploaded as text/plain at 2019-02-20.08:32:07 by @SwampWalker")</li>

<sup>*Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.*</sup>

<details><summary>Show more details</summary><p>

GitHub fields:
```python
assignee = None
closed_at = None
created_at = <Date 2019-02-19.16:45:39.064>
labels = ['type-bug', '3.7', 'expert-email']
title = 'email: folding of quoted string in display_name violates RFC'
updated_at = <Date 2019-11-20.08:30:29.764>
user = 'https://github.com/SwampWalker'
```

bugs.python.org fields:
```python
activity = <Date 2019-11-20.08:30:29.764>
actor = 'ronaldevers'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['email']
creation = <Date 2019-02-19.16:45:39.064>
creator = 'aaryn.startmail'
dependencies = []
files = ['48155', '48158']
hgrepos = []
issue_num = 36041
keywords = ['patch']
message_count = 7.0
messages = ['335975', '336010', '336049', '336093', '336109', '336667', '336668']
nosy_count = 4.0
nosy_names = ['barry', 'r.david.murray', 'aaryn.startmail', 'ronaldevers']
pr_nums = ['12054']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36041'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']
```

</p></details>


<!-- gh-linked-prs -->
### Linked PRs
* gh-122753
* gh-129007
* gh-129008
* gh-129009
* gh-129111
* gh-132371
<!-- /gh-linked-prs -->



Discussion:
When using a policy for an EmailMessage that triggers folding (during serialization) of a fairly long display_name in an address field, the folding process removes the quotes from the display name breaking the semantics of the field.

In particular, for a From address's display name like r'anything@anything.com ' + 'a' * MAX_LINE_LEN the folding puts anything@anything.com unquoted immediately after the From: header. For applications that do sender verification inside and then send it to an internal SMTP server that does not perform its own sender verification this could be considered a security issue since it enables sender spoofing. Receiving mail servers might be able to detect the broken header, but experiments show that the mail gets delivered.

Simple demonstration (reproduced in attachment) of issue:

```py
SMTP_POLICY = email.policy.default.clone(linesep='\r\n', max_line_length=72)
address = Address(display_name=r'anything@anything.com ' + 'a' * 72, addr_spec='dev@local.startmail.org')

message = EmailMessage(policy=SMTP_POLICY)
message['From'] = Address(display_name=display_name, addr_spec=addr_spec)

# Trigger folding (via as_string()), then parse it back in.
msg_string = message.as_string()
msg_bytes = msg_string.encode('utf-8')
msg_deserialized = BytesParser(policy=SMTP_POLICY).parsebytes(msg_bytes)

# Verify badness
from_hdr = msg_deserialized['From']
assert from_hdr != str(address)  # But they should be equal...
```
Since Address itself renders it correctly (str(address)), the problem is going to take a bit of digging to find.  I'm guessing the quoted_string atom is getting transformed incorrectly into something else at some point during the folding.
Hi David, the problem is in email.\_header_value_parser.\_refold_parse_tree.

Specifically, when the parsetree renders too long, it recursively gets newparts = list(part) (the children). When it does this to a BareQuotedString node, the child nodes are unquoted and unescaped and it just happily serializes these.

I thought I had attached a file that monkey patches the \_refold_parse_tree function with a fixed version... let me try again.
I'm afraid I don't have time to parse through the file you uploaded.  Can you produce a pull request or a diff showing your fix?  And ideally some added tests :)  But whatever you can do is great, if you don't have time maybe someone else will pick it up (I unfortunately don't have time, though I should be able to do a review of a PR).
Sure thing, I'll try to produce something tomorrow.
Sorry about the delay. I opened pull request https://github.com/python/cpython/pull/12054 for this. Let me know if you need anything else.
Although I am not personally interested in backporting a fix for this issue, anyone that experiences this issue in python 3.5 can execute the following monkey patch to solve the issue:

```py
def _fix_issue_36041_3_5():
    from email._header_value_parser import QuotedString, ValueTerminal, quote_string
    import email._header_value_parser

    class BareQuotedString(QuotedString):

        token_type = 'bare-quoted-string'

        def __str__(self):
            return quote_string(''.join(str(x) for x in self))

        @property
        def value(self):
            return ''.join(str(x) for x in self)

        @property
        def parts(self):
            parts = list(self)
            escaped_parts = []
            for part in parts:
                if isinstance(part, ValueTerminal):
                    escaped = quote_string(str(part))[1:-1]
                    escaped_parts.append(ValueTerminal(escaped, 'ptext'))
                else:
                    escaped_parts.append(part)
            # Add quotes to the first and last parts.
            escaped_parts[0] = ValueTerminal('"' + str(escaped_parts[0]), 'ptext')
            escaped_parts[-1] = ValueTerminal(str(escaped_parts[-1] + '"'), 'ptext')
            return escaped_parts

    email._header_value_parser.BareQuotedString = BareQuotedString
```
I came to report a similar issue (discovered during writing tests for #118643). I thing they are the parts of the same problem.

The address like `"<one@example.com>," <two@example.com>` can turn into two addresses `<one@example.com>, <two@example.com>` after re-folding.
Still an issue in 3.13. #security-issue.
[Ping!](https://devguide.python.org/getting-started/pull-request-lifecycle/#reviewing) This security issue in modern email has PR #122753 waiting for a reviewer.




Patch:
diff --git a/Lib/email/_header_value_parser.py b/Lib/email/_header_value_parser.py
index ec2215a5e5f33c..3d845c09d415f6 100644
--- a/Lib/email/_header_value_parser.py
+++ b/Lib/email/_header_value_parser.py
@@ -95,8 +95,16 @@
 NLSET = {'\n', '\r'}
 SPECIALSNL = SPECIALS | NLSET
 
+
+def make_quoted_pairs(value):
+    """Escape dquote and backslash for use within a quoted-string."""
+    return str(value).replace('\\', '\\\\').replace('"', '\\"')
+
+
 def quote_string(value):
-    return '"'+str(value).replace('\\', '\\\\').replace('"', r'\"')+'"'
+    escaped = make_quoted_pairs(value)
+    return f'"{escaped}"'
+
 
 # Match a RFC 2047 word, looks like =?utf-8?q?someword?=
 rfc2047_matcher = re.compile(r'''
@@ -2905,6 +2913,15 @@ def _refold_parse_tree(parse_tree, *, policy):
         if not hasattr(part, 'encode'):
             # It's not a terminal, try folding the subparts.
             newparts = list(part)
+            if part.token_type == 'bare-quoted-string':
+                # To fold a quoted string we need to create a list of terminal
+                # tokens that will render the leading and trailing quotes
+                # and use quoted pairs in the value as appropriate.
+                newparts = (
+                    [ValueTerminal('"', 'ptext')] +
+                    [ValueTerminal(make_quoted_pairs(p), 'ptext')
+                     for p in newparts] +
+                    [ValueTerminal('"', 'ptext')])
             if not part.as_ew_allowed:
                 wrap_as_ew_blocked += 1
                 newparts.append(end_ew_not_allowed)
diff --git a/Misc/NEWS.d/next/Security/2024-08-06-11-43-08.gh-issue-80222.wfR4BU.rst b/Misc/NEWS.d/next/Security/2024-08-06-11-43-08.gh-issue-80222.wfR4BU.rst
new file mode 100644
index 00000000000000..0f0661d0b1cf4a
--- /dev/null
+++ b/Misc/NEWS.d/next/Security/2024-08-06-11-43-08.gh-issue-80222.wfR4BU.rst
@@ -0,0 +1,6 @@
+Fix bug in the folding of quoted strings when flattening an email message using
+a modern email policy. Previously when a quoted string was folded so that
+it spanned more than one line, the surrounding quotes and internal escapes
+would be omitted. This could theoretically be used to spoof header lines
+using a carefully constructed quoted string if the resulting rendered email
+was transmitted or re-parsed.


Test patch:
diff --git a/Lib/test/test_email/test__header_value_parser.py b/Lib/test/test_email/test__header_value_parser.py
index 95224e19f67ce5..d60a7039f9d4c6 100644
--- a/Lib/test/test_email/test__header_value_parser.py
+++ b/Lib/test/test_email/test__header_value_parser.py
@@ -3082,13 +3082,40 @@ def test_address_list_with_list_separator_after_fold(self):
         self._test(parser.get_address_list(to)[0],
             f'{a},\n =?utf-8?q?H=C3=BCbsch?= Kaktus <beautiful@example.com>\n')
 
-        a = '.' * 79
+        a = '.' * 79  # ('.' is a special, so must be in quoted-string.)
         to = f'"{a}" <xyz@example.com>, "Hübsch Kaktus" <beautiful@example.com>'
         self._test(parser.get_address_list(to)[0],
-            f'{a}\n'
+            f'"{a}"\n'
             ' <xyz@example.com>, =?utf-8?q?H=C3=BCbsch?= Kaktus '
             '<beautiful@example.com>\n')
 
+    def test_address_list_with_specials_in_long_quoted_string(self):
+        # Regression for gh-80222.
+        policy = self.policy.clone(max_line_length=40)
+        cases = [
+            # (to, folded)
+            ('"Exfiltrator <spy@example.org> (unclosed comment?" <to@example.com>',
+             '"Exfiltrator <spy@example.org> (unclosed\n'
+             ' comment?" <to@example.com>\n'),
+            ('"Escaped \\" chars \\\\ in quoted-string stay escaped" <to@example.com>',
+             '"Escaped \\" chars \\\\ in quoted-string\n'
+             ' stay escaped" <to@example.com>\n'),
+            ('This long display name does not need quotes <to@example.com>',
+             'This long display name does not need\n'
+             ' quotes <to@example.com>\n'),
+            ('"Quotes are not required but are retained here" <to@example.com>',
+             '"Quotes are not required but are\n'
+             ' retained here" <to@example.com>\n'),
+            ('"A quoted-string, it can be a valid local-part"@example.com',
+             '"A quoted-string, it can be a valid\n'
+             ' local-part"@example.com\n'),
+            ('"local-part-with-specials@but-no-fws.cannot-fold"@example.com',
+             '"local-part-with-specials@but-no-fws.cannot-fold"@example.com\n'),
+        ]
+        for (to, folded) in cases:
+            with self.subTest(to=to):
+                self._test(parser.get_address_list(to)[0], folded, policy=policy)
+
     # XXX Need tests with comments on various sides of a unicode token,
     # and with unicode tokens in the comments.  Spaces inside the quotes
     # currently don't do the right thing.