Repository: certbot/certbot

Issue:
certbot 0.28.0 wrongly split the chain obtained via acmev2 if the chain has CRLF lineending
If you're having trouble using Certbot and aren't sure you've found a bug or
request for a new feature, please first try asking for help at
https://community.letsencrypt.org/. There is a much larger community there of
people familiar with the project who will be able to more quickly answer your
questions.

## My operating system is (include version):
Debian 9.12 (stretch) 

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):
OS package manager 0.28.0-1~deb9u2

## I ran this command and it produced this output:
certbot certonly --webroot -w /var/www/example.com/public_html/ -d example.com -d www.example.com --server 'https://api.buypass.com/acme/directory'

## Certbot's behavior differed from what I expected because:
The ACME v2 api server from buypass.com returns the pem-certificate-chain with CRLF lineending.
The certificate is extracted correctly and get's correctly converted to LF line ending.
The cain file get extracted with a wrong offset, containing a few charactes (in my case up to 35 characters) from the chain file, including the text: "-----END CERTIFICATE-----" and with CRLF linending
Also the fullchain.pem file is reconstructed by appending the cert file and the extracted chain file, resuling in a mixed lineending file, which contains extraneous characters between the certificate and the chain file.
Like this:
xxxxxxxxxxxxxxxxxxxxxxxxxxxAVbdrquLF
-----END CERTIFICATE-----LF
AVbdrquCRLF
-----END CERTIFICATE-----CRLF
-----BEGIN CERTIFICATE-----CRLF
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxCRLF
...


## Here is a Certbot log showing the issue (if available):
###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.
the certbot logs as everyting is OK, without error

## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:
N/A


Discussion:
The behavior is reproducible on master. I think an effort should be made to make this more robust (i.e. don't assume that `dump_` is going to produce exactly the same PEM representation that was fed in using `load_`), because it's caused issues before (#7625). 

Log including the cert response: [letsencrypt.log](https://github.com/certbot/certbot/files/4420754/letsencrypt.log)

The resulting `fullchain.pem` is mangled:

    -----BEGIN CERTIFICATE-----
    MIIGQTCCBCmgAwIBAgIKKYpamf8RmbntxDANBgkqhkiG9w0BAQsFADBLMQswCQYD
    VQQGEwJOTzEdMBsGA1UECgwUQnV5cGFzcyBBUy05ODMxNjMzMjcxHTAbBgNVBAMM
    FEJ1eXBhc3MgQ2xhc3MgMiBDQSA1MB4XDTIwMDQwMjExNDAzN1oXDTIwMDkyOTIx
    NTkwMFowADCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ7zq0phdfxa
    64pdVrpAXfU1p4AXto1dCeolJ9N+4R12CmKd7Dxz/8fIjLabFK7jvtSyyy2Qd6Jy
    4Pdz6sfpPaAsOAQqDWBbqzrMHxa2vOiOmo9tgVRtUf1nUNZkEu+xm3Oiae7W06qN
    lA6PcdjA66Sc+I05LpsPEKNiGgMu5V5yiI444kavEcZITTxq2sSb8BzR+oA9y3QL
    yq0YTyOkZl2Nr1K8GJMVJqXlyK5sTlAH3/MWIgz4GsXovXucghLTaNkjOpdwPeJe
    7fkFOpsA4ShhpbkPqbo3j1g5gnZxi3/r8t3E1pBzXR422smF3SQCFj4GvurVf+Xo
    gn7VZyAB6QECAwEAAaOCAnAwggJsMAkGA1UdEwQCMAAwHwYDVR0jBBgwFoAUJ1Kk
    by0qq0CTkOzWacv+fGE7fEIwHQYDVR0OBBYEFJ3cy8umx1CV7AlF2HtIHPSLwr+/
    MA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIw
    HwYDVR0gBBgwFjAKBghghEIBGgECBzAIBgZngQwBAgEwOgYDVR0fBDMwMTAvoC2g
    K4YpaHR0cDovL2NybC5idXlwYXNzLm5vL2NybC9CUENsYXNzMkNBNS5jcmwwHwYD
    VR0RAQH/BBUwE4IRZjcxMDVkZmYubmdyb2suaW8wagYIKwYBBQUHAQEEXjBcMCMG
    CCsGAQUFBzABhhdodHRwOi8vb2NzcC5idXlwYXNzLmNvbTA1BggrBgEFBQcwAoYp
    aHR0cDovL2NydC5idXlwYXNzLm5vL2NydC9CUENsYXNzMkNBNS5jZXIwggEEBgor
    BgEEAdZ5AgQCBIH1BIHyAPAAdgC72d+8H4pxtZOUI5eqkntHOFeVCqtS6BqQlmQ2
    jh7RhQAAAXE6sXB3AAAEAwBHMEUCIBiAni8JdwL1xRmiuFjJS78J2TlaiWfm42QV
    tF+LPxrkAiEAo2G84qVVjjkYaUWUMK4UroyMzmalWQ8TNFezHwC1ftYAdgBvU3as
    MfAxGdiZAKRRFf93FRwR2QLBACkGjbIImjfZEwAAAXE6sXBNAAAEAwBHMEUCIQCO
    lh2oub5LvnoLSO3nCtPbi9RuuLMS/kql7I4EcID7dwIgDWY9stH/QM6IfanV/U/Q
    Vd6XLFhIA4amdmCLwbLn0lwwDQYJKoZIhvcNAQELBQADggIBAAow2hhq8i5bntcY
    u4w2YQrzni+XWF8zc+Bd4Pp+6YgyZZIcslaTjak2IBFIz3TeGt9S/jKz55nGp9np
    e0rCwqah7dQ4KhW8YSMcWKoPCFbQF/S9xQDdqguLHmB2m7gpUXo8QAY1hq4s3REZ
    k+NWzirRzgvB8HoCKIfSh/72GPRXeb8322hq/WnrOD/5cpYgiJXVgRfKE7l0ZwP8
    kMDeo42p7MbN4Q0dlTW5Djanv6sW8u3mdPFBSCP9zt8efFLUX5Y6pyRKQHCLahgN
    lRb+fUZyuzACnP4YoeOF6wSw1NILRKcGm2BKcooypgz+shs6jSt2mre/WEPpkfVF
    jXbvGEQqHapbfqV1laCHzWgMWdZ8uny3g7LhCtzEdr9FQ634e8nxwcZp2u6vq3dp
    V4PEl+YsrmpFfVCTnZDFk6wrhMyfomJ9tqApbKPogzCODkAIs1l5fya8qFFwoGYH
    KYcmqb83LLiWG9+i25RoopLDf/vrKsRd136Ee8VeS017LgzBV5ywKoLMvRhtUYMr
    ejAMk2+RMDd60hMI7Gi6SL3h+u92Rr2Qiqdn9Uo5Az+HOsb5p/a8OW937Z5d1LqH
    pIHSLCdlbsaydcQBanqZqxkn39lfsa9z6nWMHhB9iUk7IXRLq2GORwKHq06o078K
    IlV+YOkJt4xBc+elm9B5BKCPp+36
    -----END CERTIFICATE-----
    KCPp+36
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIIGFjCCA/6gAwIBAgIJMvoIN2vyPCATMA0GCSqGSIb3DQEBCwUAME4xCzAJBgNV
    BAYTAk5PMR0wGwYDVQQKDBRCdXlwYXNzIEFTLTk4MzE2MzMyNzEgMB4GA1UEAwwX
    QnV5cGFzcyBDbGFzcyAyIFJvb3QgQ0EwHhcNMTcwNTIzMTI1NzM4WhcNMjcwNTIz
    MTI1NzM4WjBLMQswCQYDVQQGEwJOTzEdMBsGA1UECgwUQnV5cGFzcyBBUy05ODMx
    NjMzMjcxHTAbBgNVBAMMFEJ1eXBhc3MgQ2xhc3MgMiBDQSA1MIICIjANBgkqhkiG
    9w0BAQEFAAOCAg8AMIICCgKCAgEAu5hBKP7+r8dtoZT14U1OKtTBSzI+U3nUC4p5
    ht3kD/DR9hWPB6/2Kljii7Ft4AXTSxS30YGwBpqEtIHzxjfgf/fuTb/eBx5wWhxz
    v0RWosEhUwyQqYX0lVEqHruRV6rjlkvtNYkhdEU2kmQs6j4T66p8AbgZLZSq42k+
    lEU3hGQ8NpDiKSvxX5aVorxsRel98AP0VaRVMZfeMYI5DkTcfWkTMrD+dDqGejOB
    S3V1+EstPV1Auf85LiUo0jbwy1ZOppVFftUczMyBwOENOrUwfjDyQnQrn4qhq+TD
    qAMua2/2+gX0jHZ/wwdP4mhSPjmlpNejMmzXbykDu74BJqWfBvz9fpG6C344M0Hv
    WZ4QLOTP5dKwMVTGF6S/QK3jJb8SdB0cDMCzsLBagC/F56+cLMntOXsv4DOvD3HG
    rmrEZKsjJB+0XWIVwdPif4ItQm6MJZq4rGkPymaKsaKAes7OpFflFj7ruQIuc380
    l7c98uVpIuszMTak3L8CaPN6nrpmIUGAaOT1fDdfANKnb5vVosSr6W/ETkF8gFsP
    Ljy09wzrUKH/Gnk7EdoBNwdy5zwP8z1F99j8QUjVVQwckBO/YsgJI6MOXoHqeHS3
    LVT18dZ66FJLYZTIXZQTtLDo3eQ6eL+vSNnakxMjO4nH90BQGy6l0CRPCIi0+a0A
    8HaUbA8CAwEAAaOB+TCB9jAPBgNVHRMBAf8EBTADAQH/MB8GA1UdIwQYMBaAFMmA
    d+BikoL1RpzzuvdMw964o605MB0GA1UdDgQWBBQnUqRvLSqrQJOQ7NZpy/58YTt8
    QjAOBgNVHQ8BAf8EBAMCAQYwHwYDVR0gBBgwFjAKBghghEIBGgECBzAIBgZngQwB
    AgEwPQYDVR0fBDYwNDAyoDCgLoYsaHR0cDovL2NybC5idXlwYXNzLm5vL2NybC9C
    UENsYXNzMlJvb3RDQS5jcmwwMwYIKwYBBQUHAQEEJzAlMCMGCCsGAQUFBzABhhdo
    dHRwOi8vb2NzcC5idXlwYXNzLmNvbTANBgkqhkiG9w0BAQsFAAOCAgEAvrrbRl50
    56kq5vOKuWUcazLEHuJlVyjr1QBpLoq1dZNjpmIFLjOmG7LLMv83Hs2d5Ww0H6Sg
    NlloNjp9JYo5PCruL+djEb5Ct787d0dD+sQ9wkEQNK/fgQ6hqef4Z/lYxmtNqCyR
    xxuK8LcgvMG5D3W9CLZYIqCyGgKZqAE0FtWyinPTzCdvxzCMRZz2dYJLN+2nejMs
    z1+FswjSnOyva0XegBZnjhrHNI3ZyB7nPILephet0AiYyXpXUuEvBXa6qVkeK8mf
    6uGgq+zTXqK52cKu2woOa9TV2/oyC1DzVNoZbfOgJ0Geh2TWeJn8rL+W4VrvxT6K
    w2+lk/YeyXgFd6km3FgEQk4pRnzuV3OtDErdQRW2lssR3gl9u2DF9IJYbzoRiVvy
    7wTHuchTygpG8n/K9ajaWDdEL7+jSvPjU5+HBcHwgcxDSYF3QSLNIsOaBS5O1Hob
    440ryhvw5lGgzqkVsToXJCaaSziOmqbzzNPHgQ0HM+vzvDTzC0Z6ikF0LGjkPSML
    a4qKpyJzIjLOjqPVnI51920yg/zZuN5nz1lQcH+0q7GVeoRaHE2iFyJnHXR+Ljl9
    2SHWMrw1jbhk7D5/u1XmtsS0r1oeHgbF7zi7ESOVLFs2Zcm5kewkPDljznSJTE94
    1YS4d5yABnRwkq6PSaKEvBRshdl84c/Fd7Y=
    -----END CERTIFICATE-----

So there are two problems: a wrong offset, and the `CRLF` line endings.

I will say here that I had some troubles in the past with the PEM encoding because of `CRLF` line endings when porting Certbot on Windows. I fixed that by always emitting cryptographic material with `LF` line endings. On top of this, Boulder explicitly refuses `CRLF` line endings: https://github.com/letsencrypt/boulder/issues/3557

In #7625, issue has been considered as won't fix, because the CA Server was not following the official specifications defined in RFCs. I would say that the wrong offset fall in the same category, and the fix should be from BuyPass.

However. For the `CRLF` line endings, I could not find any clear indication that a PEM file *should* be formated with `LF` line endings. It seems to be a *de facto* standard because historically, all there cryptographic tools have been developped for Unix systems first ...

At the very minimum BuyPass should conform to the usual `LF` line endings, because it will cause very bad time to users in various situation. But we may also handle that on our side, because it seems to me that it is not a violation of specifications to use `CRLF` line endings here.

What do you think @bmw?
I already discussed with buypass CA regarding this (prior to open this issues here).
https://community.buypass.com/t/x2hhvnf/chainfullchain-file-contains-unexpected-characters
It seems that in the referenced RFC (https://tools.ietf.org/html/rfc8555#section-9.1 which further directs to this RFC: https://tools.ietf.org/html/rfc7468#section-3) the eol is defined as:   
`eol        = CRLF / CR / LF`
So it seems the any mentioned combination should be considered as valid.
Thanks for all the info!

Regardless of what Buypass does, I think this is something we should fix on our end since per the links szjozsef provided, the spec allows for CRLF line endings.

The code that needs to be updated is [here](https://github.com/certbot/certbot/blob/4ca86d9482cbb9bca04335bfb7dbd892cefe91c8/certbot/certbot/crypto_util.py#L481-L493) if anyone wants to take a crack at it.

Resolving patch:
diff --git a/certbot/certbot/crypto_util.py b/certbot/certbot/crypto_util.py
--- a/certbot/certbot/crypto_util.py
+++ b/certbot/certbot/crypto_util.py
@@ -8,6 +8,7 @@
 import logging
 import warnings
 
+import re
 # See https://github.com/pyca/cryptography/issues/4275
 from cryptography import x509  # type: ignore
 from cryptography.exceptions import InvalidSignature
@@ -478,6 +479,17 @@ def sha256sum(filename):
         sha256.update(file_d.read().encode('UTF-8'))
     return sha256.hexdigest()
 
+# Finds one CERTIFICATE stricttextualmsg according to rfc7468#section-3.
+# Does not validate the base64text - use crypto.load_certificate.
+CERT_PEM_REGEX = re.compile(
+    b"""-----BEGIN CERTIFICATE-----\r?
+.+?\r?
+-----END CERTIFICATE-----\r?
+""",
+    re.DOTALL # DOTALL (/s) because the base64text may include newlines
+)
+
+
 def cert_and_chain_from_fullchain(fullchain_pem):
     """Split fullchain_pem into cert_pem and chain_pem
 
@@ -486,11 +498,24 @@ def cert_and_chain_from_fullchain(fullchain_pem):
     :returns: tuple of string cert_pem and chain_pem
     :rtype: tuple
 
+    :raises errors.Error: If there are less than 2 certificates in the chain.
+
     """
-    cert = crypto.dump_certificate(crypto.FILETYPE_PEM,
-        crypto.load_certificate(crypto.FILETYPE_PEM, fullchain_pem)).decode()
-    chain = fullchain_pem[len(cert):].lstrip()
-    return (cert, chain)
+    # First pass: find the boundary of each certificate in the chain.
+    # TODO: This will silently skip over any "explanatory text" in between boundaries,
+    # which is prohibited by RFC8555.
+    certs = CERT_PEM_REGEX.findall(fullchain_pem.encode())
+    if len(certs) < 2:
+        raise errors.Error("failed to parse fullchain into cert and chain: " +
+                           "less than 2 certificates in chain")
+
+    # Second pass: for each certificate found, parse it using OpenSSL and re-encode it,
+    # with the effect of normalizing any encoding variations (e.g. CRLF, whitespace).
+    certs_normalized = [crypto.dump_certificate(crypto.FILETYPE_PEM,
+        crypto.load_certificate(crypto.FILETYPE_PEM, cert)).decode() for cert in certs]
+
+    # Since each normalized cert has a newline suffix, no extra newlines are required.
+    return (certs_normalized[0], "".join(certs_normalized[1:]))
 
 def get_serial_from_cert(cert_path):
     """Retrieve the serial number of a certificate from certificate path


Test patch:
diff --git a/certbot/tests/crypto_util_test.py b/certbot/tests/crypto_util_test.py
--- a/certbot/tests/crypto_util_test.py
+++ b/certbot/tests/crypto_util_test.py
@@ -376,17 +376,33 @@ def test_sha256sum(self):
 class CertAndChainFromFullchainTest(unittest.TestCase):
     """Tests for certbot.crypto_util.cert_and_chain_from_fullchain"""
 
+    def _parse_and_reencode_pem(self, cert_pem):
+        from OpenSSL import crypto
+        return crypto.dump_certificate(crypto.FILETYPE_PEM,
+            crypto.load_certificate(crypto.FILETYPE_PEM, cert_pem)).decode()
+
     def test_cert_and_chain_from_fullchain(self):
         cert_pem = CERT.decode()
         chain_pem = cert_pem + SS_CERT.decode()
         fullchain_pem = cert_pem + chain_pem
         spacey_fullchain_pem = cert_pem + u'\n' + chain_pem
+        crlf_fullchain_pem = fullchain_pem.replace(u'\n', u'\r\n')
+
+        # In the ACME v1 code path, the fullchain is constructed by loading cert+chain DERs
+        # and using OpenSSL to dump them, so here we confirm that OpenSSL is producing certs
+        # that will be parseable by cert_and_chain_from_fullchain.
+        acmev1_fullchain_pem = self._parse_and_reencode_pem(cert_pem) + \
+            self._parse_and_reencode_pem(cert_pem) + self._parse_and_reencode_pem(SS_CERT.decode())
+
         from certbot.crypto_util import cert_and_chain_from_fullchain
-        for fullchain in (fullchain_pem, spacey_fullchain_pem):
+        for fullchain in (fullchain_pem, spacey_fullchain_pem, crlf_fullchain_pem,
+                          acmev1_fullchain_pem):
             cert_out, chain_out = cert_and_chain_from_fullchain(fullchain)
             self.assertEqual(cert_out, cert_pem)
             self.assertEqual(chain_out, chain_pem)
 
+        self.assertRaises(errors.Error, cert_and_chain_from_fullchain, cert_pem)
+
 
 if __name__ == '__main__':
     unittest.main()  # pragma: no cover