Repository: pyca/cryptography

Issue:
S/MIME signature: "Content-Type: text/plain" added to message (39.0.0)
Hey,

I'm observing a weird behavior when signing with S/MIME since version 39.0.0: it seems like an additional "Content-Type: text/plain" is added to the message after signing it.

My code looks like this:

```
from cryptography import x509
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.serialization import pkcs7

with open("test.cert", "rb") as cc:
    cert = x509.load_pem_x509_certificate(cc.read())

with open("test.key", "rb") as ck:
    key = serialization.load_pem_private_key(ck.read(), None)

msg = "test"

options = [pkcs7.PKCS7Options.DetachedSignature, pkcs7.PKCS7Options.Text]

signed_msg = (
    pkcs7.PKCS7SignatureBuilder()
    .set_data(bytes(msg, "utf-8"))
    .add_signer(cert, key, hashes.SHA256())
    .sign(serialization.Encoding.SMIME, options)
)

with open("/tmp/msg.txt", "wb") as msg_file:
    msg_file.write(signed_msg)
```

With `cryptography 38.0.4`, the relevant part is
```
This is an S/MIME signed message

------A78EE028A05FC73322259A20ED9EAAAE
Content-Type: text/plain

test
------A78EE028A05FC73322259A20ED9EAAAE
```
and `openssl smime -verify -in /tmp/msg.txt -noverify` is successful.

With `cryptography 39.0.0`, I get
```
This is an S/MIME signed message

--===============0873037622973214044==
Content-Type: text/plain

Content-Type: text/plain

test
--===============0873037622973214044==
```
and the verification with `openssl` fails.
After manually removing the additional "Content-Type: text/plain" from the file, it succeeds again.


Discussion:
Thanks for the report. This is our bug. In 39 we rewrote part of the S/MIME code and introduced this issue. We canonicalize the input in text mode (which consists of converting line endings to `\r\n`) and we add the text/plain value in the same function. This is useful since we need to sign over the content-type as well as the payload, but then we call another smime_encode function if you pass `serialization.Encoding.SMIME`, which duplicates it. The tests failed to catch it because they looked for the presence of text/plain and did not assert the number of them and our verification test passed too because we don't extract the signature test value from the output.

We'll get a fix up soon.
I'm not sure I understand why our verification tests still pass here.
They pass because we sign the right thing but embed the wrong thing into the larger smime payload and then the verify call doesn't use what's in the smime payload. The test is https://github.com/pyca/cryptography/blob/d90ed2b2bc6d02720191d6449be098451677b3fd/tests/hazmat/primitives/test_pkcs7.py#L495-L506

but it should be:

```python
        assert sig_pem.count(b"text/plain") == 1
        # Parse the message to get the signed data, which is the
        # first payload in the message
        message = email.parser.BytesParser().parsebytes(sig_pem)
        signed_data = message.get_payload()[0].as_bytes()
        _pkcs7_verify(...)
```

Either of the changes to the test above would have caught this bug.
Gotcha,thanks.

On Wed, Feb 15, 2023 at 7:15 PM Paul Kehrer ***@***.***>
wrote:

> They pass because we sign the right thing but embed the wrong thing into
> the larger smime payload and then the verify call doesn't use what's in the
> smime payload. The test is
> https://github.com/pyca/cryptography/blob/d90ed2b2bc6d02720191d6449be098451677b3fd/tests/hazmat/primitives/test_pkcs7.py#L495-L506
>
> but it should be:
>
>         assert sig_pem.count(b"text/plain") == 1
>         # Parse the message to get the signed data, which is the
>         # first payload in the message
>         message = email.parser.BytesParser().parsebytes(sig_pem)
>         signed_data = message.get_payload()[0].as_bytes()
>         _pkcs7_verify(...)
>
> Either of the changes to the test above would have caught this bug.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pyca/cryptography/issues/8298#issuecomment-1432274003>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAAGBCYZE3NAQRXF4AFSVDWXVWT3ANCNFSM6AAAAAAU47XZFU>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


-- 
All that is necessary for evil to succeed is for good people to do nothing.

@reaperhulk Thanks for the answer and explanation!

Resolving patch:
diff --git a/src/cryptography/hazmat/primitives/serialization/pkcs7.py b/src/cryptography/hazmat/primitives/serialization/pkcs7.py
--- a/src/cryptography/hazmat/primitives/serialization/pkcs7.py
+++ b/src/cryptography/hazmat/primitives/serialization/pkcs7.py
@@ -5,6 +5,7 @@
 import email.base64mime
 import email.generator
 import email.message
+import email.policy
 import io
 import typing
 
@@ -177,7 +178,9 @@ def sign(
         return rust_pkcs7.sign_and_serialize(self, encoding, options)
 
 
-def _smime_encode(data: bytes, signature: bytes, micalg: str) -> bytes:
+def _smime_encode(
+    data: bytes, signature: bytes, micalg: str, text_mode: bool
+) -> bytes:
     # This function works pretty hard to replicate what OpenSSL does
     # precisely. For good and for ill.
 
@@ -192,9 +195,10 @@ def _smime_encode(data: bytes, signature: bytes, micalg: str) -> bytes:
 
     m.preamble = "This is an S/MIME signed message\n"
 
-    msg_part = email.message.MIMEPart()
+    msg_part = OpenSSLMimePart()
     msg_part.set_payload(data)
-    msg_part.add_header("Content-Type", "text/plain")
+    if text_mode:
+        msg_part.add_header("Content-Type", "text/plain")
     m.attach(msg_part)
 
     sig_part = email.message.MIMEPart()
@@ -213,7 +217,18 @@ def _smime_encode(data: bytes, signature: bytes, micalg: str) -> bytes:
 
     fp = io.BytesIO()
     g = email.generator.BytesGenerator(
-        fp, maxheaderlen=0, mangle_from_=False, policy=m.policy
+        fp,
+        maxheaderlen=0,
+        mangle_from_=False,
+        policy=m.policy.clone(linesep="\r\n"),
     )
     g.flatten(m)
     return fp.getvalue()
+
+
+class OpenSSLMimePart(email.message.MIMEPart):
+    # A MIMEPart subclass that replicates OpenSSL's behavior of not including
+    # a newline if there are no headers.
+    def _write_headers(self, generator) -> None:
+        if list(self.raw_items()):
+            generator._write_headers(self)


Test patch:
diff --git a/tests/hazmat/primitives/test_pkcs7.py b/tests/hazmat/primitives/test_pkcs7.py
--- a/tests/hazmat/primitives/test_pkcs7.py
+++ b/tests/hazmat/primitives/test_pkcs7.py
@@ -3,6 +3,7 @@
 # for complete details.
 
 
+import email.parser
 import os
 import typing
 
@@ -289,6 +290,7 @@ def test_smime_sign_detached(self, backend):
 
         sig = builder.sign(serialization.Encoding.SMIME, options)
         sig_binary = builder.sign(serialization.Encoding.DER, options)
+        assert b"text/plain" not in sig
         # We don't have a generic ASN.1 parser available to us so we instead
         # will assert on specific byte sequences being present based on the
         # parameters chosen above.
@@ -298,8 +300,17 @@ def test_smime_sign_detached(self, backend):
         # as a separate section before the PKCS7 data. So we should expect to
         # have data in sig but not in sig_binary
         assert data in sig
+        # Parse the message to get the signed data, which is the
+        # first payload in the message
+        message = email.parser.BytesParser().parsebytes(sig)
+        signed_data = message.get_payload()[0].get_payload().encode()
         _pkcs7_verify(
-            serialization.Encoding.SMIME, sig, data, [cert], options, backend
+            serialization.Encoding.SMIME,
+            sig,
+            signed_data,
+            [cert],
+            options,
+            backend,
         )
         assert data not in sig_binary
         _pkcs7_verify(
@@ -492,10 +503,14 @@ def test_sign_text(self, backend):
         # The text option adds text/plain headers to the S/MIME message
         # These headers are only relevant in SMIME mode, not binary, which is
         # just the PKCS7 structure itself.
-        assert b"text/plain" in sig_pem
-        # When passing the Text option the header is prepended so the actual
-        # signed data is this.
-        signed_data = b"Content-Type: text/plain\r\n\r\nhello world"
+        assert sig_pem.count(b"text/plain") == 1
+        assert b"Content-Type: text/plain\r\n\r\nhello world\r\n" in sig_pem
+        # Parse the message to get the signed data, which is the
+        # first payload in the message
+        message = email.parser.BytesParser().parsebytes(sig_pem)
+        signed_data = message.get_payload()[0].as_bytes(
+            policy=message.policy.clone(linesep="\r\n")
+        )
         _pkcs7_verify(
             serialization.Encoding.SMIME,
             sig_pem,