Repository: certbot/certbot

Issue:
py3.11+ crash with "AttributeError: can't set attribute" upon ACME server error
Python 3.10.8 seems to be fine, the breakage appears to be introduced in Python 3.11 and all subsequent releases (so far).

I think the issue is approximately the same as https://github.com/python/cpython/issues/99856, in that if you raise an immutable exception (which `acme.messages.Error` is, owing to `josepy.util.ImmutableMap`) within a context manager, it will blow up when trying to set its `__traceback__` attribute on exiting the context manager.

It's not totally clear to me whether this is something that Python will fix, or we will be stuck with this behavior. We could put a workaround into`josepy`. @bmw wdyt? 

## My operating system is (include version):
macOS Ventura 13.1

## I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):
`git` @ be3bf316c0e9f29cfe9c90e5dd7b903f88f54504

## I ran this command and it produced this output:
(It's important to use Let's Encrypt production or staging, to trigger a policy error on `newOrder`)

    $ sudo certbot certonly -d example.com --standalone --dry-run
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Simulating a certificate request for example.com
    An unexpected error occurred:
    AttributeError: can't set attribute
    Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

## Certbot's behavior differed from what I expected because:
The ACME error should not crash Certbot, it should show the normal output (snap output below):

    # certbot certonly --webroot -w /tmp -d example.com --dry-run
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Simulating a certificate request for example.com
    An unexpected error occurred:
    Error creating new order :: Cannot issue for "example.com": The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy
    Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

## Here is a Certbot log showing the issue (if available):

    2023-01-22 16:48:20,397:DEBUG:certbot._internal.main:certbot version: 2.3.0.dev0
    2023-01-22 16:48:20,397:DEBUG:certbot._internal.main:Location of certbot entry point: /Users/alex/devel/certbot/venv/bin/certbot
    2023-01-22 16:48:20,397:DEBUG:certbot._internal.main:Arguments: ['-d', 'example.com', '--standalone', '--dry-run']
    2023-01-22 16:48:20,398:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#certbot-route53:auth,PluginEntryPoint#dns-cloudflare,PluginEntryPoint#dns-digitalocean,PluginEntryPoint#dns-dnsimple,PluginEntryPoint#dns-dnsmadeeasy,PluginEntryPoint#dns-gehirn,PluginEntryPoint#dns-google,PluginEntryPoint#dns-linode,PluginEntryPoint#dns-luadns,PluginEntryPoint#dns-nsone,PluginEntryPoint#dns-ovh,PluginEntryPoint#dns-rfc2136,PluginEntryPoint#dns-route53,PluginEntryPoint#dns-sakuracloud,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
    2023-01-22 16:48:20,425:DEBUG:certbot._internal.log:Root logging level set at 30
    2023-01-22 16:48:20,426:DEBUG:certbot._internal.plugins.selection:Requested authenticator standalone and installer None
    2023-01-22 16:48:20,427:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * standalone
    Description: Runs an HTTP server locally which serves the necessary validation files under the /.well-known/acme-challenge/ request path. Suitable if there is no HTTP server already running. HTTP challenge only (wildcards not supported).
    Interfaces: Authenticator, Plugin
    Entry point: standalone = certbot._internal.plugins.standalone:Authenticator
    Initialized: <certbot._internal.plugins.standalone.Authenticator object at 0x1047d9bd0>
    Prep: True
    2023-01-22 16:48:20,427:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.standalone.Authenticator object at 0x1047d9bd0> and installer None
    2023-01-22 16:48:20,427:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator standalone, Installer None
    2023-01-22 16:48:20,495:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/37176138', new_authzr_uri=None, terms_of_service=None), 8dee1ed9e2f5b5ee000bf0a0ccf21bc0, Meta(creation_dt=datetime.datetime(2021, 12, 15, 22, 8, 59, tzinfo=<UTC>), creation_host='1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa', register_to_eff=None))>
    2023-01-22 16:48:20,502:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
    2023-01-22 16:48:20,527:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
    2023-01-22 16:48:21,196:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 830
    2023-01-22 16:48:21,198:DEBUG:acme.client:Received response:
    HTTP 200
    Server: nginx
    Date: Sun, 22 Jan 2023 05:48:21 GMT
    Content-Type: application/json
    Content-Length: 830
    Connection: keep-alive
    Cache-Control: public, max-age=0, no-cache
    X-Frame-Options: DENY
    Strict-Transport-Security: max-age=604800

    {
    "5e5rgib1WlY": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
    "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
    "meta": {
        "caaIdentities": [
        "letsencrypt.org"
        ],
        "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
        "website": "https://letsencrypt.org/docs/staging-environment/"
    },
    "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
    "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
    "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
    "renewalInfo": "https://acme-staging-v02.api.letsencrypt.org/get/draft-ietf-acme-ari-00/renewalInfo/",
    "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
    }
    2023-01-22 16:48:21,200:DEBUG:certbot._internal.display.obj:Notifying user: Simulating a certificate request for example.com
    2023-01-22 16:48:21,213:DEBUG:acme.client:Requesting fresh nonce
    2023-01-22 16:48:21,213:DEBUG:acme.client:Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
    2023-01-22 16:48:21,422:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
    2023-01-22 16:48:21,423:DEBUG:acme.client:Received response:
    HTTP 200
    Server: nginx
    Date: Sun, 22 Jan 2023 05:48:21 GMT
    Connection: keep-alive
    Cache-Control: public, max-age=0, no-cache
    Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
    Replay-Nonce: B37CdOEt6R9D7GpOoUNeF8wtV8GFYCGxye8aslI7QggV7No
    X-Frame-Options: DENY
    Strict-Transport-Security: max-age=604800


    2023-01-22 16:48:21,424:DEBUG:acme.client:Storing nonce: B37CdOEt6R9D7GpOoUNeF8wtV8GFYCGxye8aslI7QggV7No
    2023-01-22 16:48:21,424:DEBUG:acme.client:JWS payload:
    b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "example.com"\n    }\n  ]\n}'
    2023-01-22 16:48:21,436:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
    {
    "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8zNzE3NjEzOCIsICJub25jZSI6ICJCMzdDZE9FdDZSOUQ3R3BPb1VOZUY4d3RWOEdGWUNHeHllOGFzbEk3UWdnVjdObyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctb3JkZXIifQ",
    "signature": "mOaT0Wj7fwaglw3L7GadZwVBS8kmflkLOhUDWjLyJ4-16PFVMQSmY2t-7dN9qNr-KzSi4aAD9a2uHd6el4JYNyQUFXhwoU-vW9VawnwzAX15n8UALcpFNmTAFhU_iXLF1R_JWVT7EQoZzSgQo67lzIBTBcX6nRiFRNXtvuN-aFUJCSJp8_NikVnBLKPkxLq0sbcS7XNcmUvmK53k-R0JidfxRvCJP8ac3ka_0chcSVc5kOKyjMv0gEe5sa6qLkfZiIGUtrtZPyGuZl9odyVDBiH6YMvGyts4uEOktO8FtLZvhbO1a6t12zJDF90q3SXSOwlUfnql4UqGGv1BqHILWA",
    "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImV4YW1wbGUuY29tIgogICAgfQogIF0KfQ"
    }
    2023-01-22 16:48:21,653:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 400 258
    2023-01-22 16:48:21,654:DEBUG:acme.client:Received response:
    HTTP 400
    Server: nginx
    Date: Sun, 22 Jan 2023 05:48:21 GMT
    Content-Type: application/problem+json
    Content-Length: 258
    Connection: keep-alive
    Boulder-Requester: 37176138
    Cache-Control: public, max-age=0, no-cache
    Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
    Replay-Nonce: BEB9MCA0p22E8Dedv8MJ4cDKOCOHPt45Qb1feMqjtDdi1Es

    {
    "type": "urn:ietf:params:acme:error:rejectedIdentifier",
    "detail": "Error creating new order :: Cannot issue for \"example.com\": The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy",
    "status": 400
    }
    2023-01-22 16:48:21,655:DEBUG:certbot._internal.log:Exiting abnormally:
    Traceback (most recent call last):
    File "/Users/alex/.pyenv/versions/3.11.0/lib/python3.11/contextlib.py", line 155, in __exit__
        self.gen.throw(typ, value, traceback)
    File "/Users/alex/devel/certbot/certbot/certbot/_internal/main.py", line 1683, in make_displayer
        yield displayer
    File "/Users/alex/devel/certbot/certbot/certbot/_internal/main.py", line 1736, in main
        return config.func(config, plugins)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/certbot/certbot/_internal/main.py", line 1590, in certonly
        lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/certbot/certbot/_internal/main.py", line 138, in _get_and_save_cert
        lineage = le_client.obtain_and_enroll_certificate(domains, certname)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/certbot/certbot/_internal/client.py", line 516, in obtain_and_enroll_certificate
        cert, chain, key, _ = self.obtain_certificate(domains)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/certbot/certbot/_internal/client.py", line 428, in obtain_certificate
        orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/certbot/certbot/_internal/client.py", line 478, in _get_order_and_authorizations
        orderr = self.acme.new_order(csr_pem)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/acme/acme/client.py", line 138, in new_order
        response = self._post(self.directory['newOrder'], order)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/acme/acme/client.py", line 338, in _post
        return self.net.post(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/acme/acme/client.py", line 711, in post
        return self._post_once(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/acme/acme/client.py", line 724, in _post_once
        response = self._check_response(response, content_type=content_type)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/acme/acme/client.py", line 575, in _check_response
        raise messages.Error.from_json(jobj)
    acme.messages.Error: urn:ietf:params:acme:error:rejectedIdentifier :: The server will not issue certificates for the identifier :: Error creating new order :: Cannot issue for "example.com": The ACME server refuses to issue a certificate for this domain name, because it is forbidden by policy

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "/Users/alex/devel/certbot/venv/bin/certbot", line 33, in <module>
        sys.exit(load_entry_point('certbot', 'console_scripts', 'certbot')())
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/certbot/certbot/main.py", line 19, in main
        return internal_main.main(cli_args)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/certbot/certbot/_internal/main.py", line 1733, in main
        with make_displayer(config) as displayer:
    File "/Users/alex/.pyenv/versions/3.11.0/lib/python3.11/contextlib.py", line 188, in __exit__
        exc.__traceback__ = traceback
        ^^^^^^^^^^^^^^^^^
    File "/Users/alex/devel/certbot/venv/lib/python3.11/site-packages/josepy/util.py", line 191, in __setattr__
        raise AttributeError("can't set attribute")
    AttributeError: can't set attribute
    2023-01-22 16:48:21,661:ERROR:certbot._internal.log:An unexpected error occurred:
    2023-01-22 16:48:21,662:ERROR:certbot._internal.log:AttributeError: can't set attribute



Discussion:
Thanks for looking into and writing this up Alex.

I think we should fix this on our end. In addition to not yet seeing any progress on https://github.com/python/cpython/issues/99856 being resolved, while not at all clear, I think it could be reasonably argued that having an immutable exception class breaks Python's exception API. In Python, [all exceptions must inherit from `BaseException`](https://docs.python.org/3/library/exceptions.html#built-in-exceptions) which [defines `with_traceback` and `add_note` that mutate the exception](https://docs.python.org/3/library/exceptions.html#BaseException). (Interestingly, `with_traceback` seems to work on our error class here, but `add_note` does not.)

I'm curious what you had in mind to fix this though. My first thought is an ideal fix would be to make `acme.messages.Error` mutable with little to no changes to its API. (Does no longer inheriting from `JSONObjectWithFields` constitute an API change? :shrug: ) What do you think?
>Does no longer inheriting from JSONObjectWithFields constitute an API change? 🤷

Don't we get all the deserialization behavior from `JSONObjectWithFields`? 

I don't have a good sense for what the fix should look like. What came to mind for me was overriding `__setattr__` in `acme.messages.Error` to alter its behavior, possibly just to redirect the call to `object.__setattr__`. Maybe creating a mutability mixin? I hope that setting random non-field attributes on josepy objects won't affect its functionality.
> Don't we get all the deserialization behavior from `JSONObjectWithFields`?

Yeah. My thought would be to somehow make `acme.messages.Error` mutable without altering any base classes or to make a mutable version of `JSONObjectWithFields` in `acme` or `josepy` that `acme.messages.Error` can inherit from.
looking into this a bit, it does seem like overriding `__setattr__` is the most straightforward fix since a lot of exception handling code expects to be able to modify `__traceback__` and `__notes__` directly. also, while a mutable JSONObjectWIthFields class would work, i think that'd imply making the traceback serializable, which i suspect we're not interested in doing.

something like this seems to work:

```
--- a/acme/acme/messages.py
+++ b/acme/acme/messages.py
@@ -185,6 +185,12 @@ class Error(jose.JSONObjectWithFields, errors.Error):
             return code
         return None
 
+    def __setattr__(self, name: str, value: Any) -> None:
+        if name in ['__traceback__', '__notes__']:
+            return object.__setattr__(self, name, value)
+        else:
+            return super().__setattr__(name, value)
+
     def __str__(self) -> str:
         result = b' :: '.join(
             part.encode('ascii', 'backslashreplace') for part in
```
oh i also meant to add that since `__notes__` and `__traceback__` don't show up in the exception object's `__slots__`, they're invisible to the JSONy methods in josepy, so i don't think mutating them will affect that code
I have one concern that allowlisting `__traceback__` and `__notes__` may set us up for future breakages. For example, [PEP-3134](https://peps.python.org/pep-3134/) looks like it defines some other possible exception attributes. In there is no harm, I'd be in favor of passing through any attribute mutation.
+1 to allowing modifications of any attributes.

By the way, we successfully used this `object.__setattr__` approach before on `JSONObjectWithFields` based classes in https://github.com/certbot/certbot/pull/6758. I do think it's a bit of a hack and think making a mutable `JSONObjectWithFields` like class in `josepy` that shares code with `JSONObjectWithFields` would be cleaner, but using `object.__setattr__` should work and fix the problem.
sgtm, and yeah agreed that it's mostly a hack -- mostly i'd be worried that some code elsewhere is counting on the immutability of these fields for e.g. hashing or correctness of serialized fields, but if that's not the case, then let's go for it
I'll be around to review this if anybody wants to grab it!

Resolving patch:
diff --git a/acme/acme/messages.py b/acme/acme/messages.py
--- a/acme/acme/messages.py
+++ b/acme/acme/messages.py
@@ -123,6 +123,9 @@ class Error(jose.JSONObjectWithFields, errors.Error):
 
     https://datatracker.ietf.org/doc/html/rfc7807
 
+    Note: Although Error inherits from JSONObjectWithFields, which is immutable,
+    we add mutability for Error to comply with the Python exception API.
+
     :ivar str typ:
     :ivar str title:
     :ivar str detail:
@@ -185,6 +188,10 @@ def code(self) -> Optional[str]:
             return code
         return None
 
+    # Hack to allow mutability on Errors (see GH #9539)
+    def __setattr__(self, name: str, value: Any) -> None:
+        return object.__setattr__(self, name, value)
+
     def __str__(self) -> str:
         result = b' :: '.join(
             part.encode('ascii', 'backslashreplace') for part in


Test patch:
diff --git a/acme/tests/messages_test.py b/acme/tests/messages_test.py
--- a/acme/tests/messages_test.py
+++ b/acme/tests/messages_test.py
@@ -1,6 +1,7 @@
 """Tests for acme.messages."""
 from typing import Dict
 import unittest
+import contextlib
 from unittest import mock
 import warnings
 
@@ -91,6 +92,22 @@ def test_str(self):
             u"Problem for {1.identifier.value}: {1.typ} :: {1.description} :: {1.detail} :: {1.title}").format(
         self.error_with_subproblems, self.subproblem))
 
+    # this test is based on a minimal reproduction of a contextmanager/immutable
+    # exception related error: https://github.com/python/cpython/issues/99856
+    def test_with_context_manager(self):
+        from acme.messages import Error
+
+        @contextlib.contextmanager
+        def context():
+            yield
+
+        try:
+            with context():
+                raise self.error_custom
+        except Error as e:
+            self.assertIsNotNone(self.error_custom.__traceback__)
+
+
 class ConstantTest(unittest.TestCase):
     """Tests for acme.messages._Constant."""