Repository: certbot/certbot

Issue:
[Discussion]: Consider the correct logging level for ARI fetch failures
### Task description

The big red warning that doesn't affect cert renewal is confusing people, so https://github.com/certbot/certbot/pull/10335 changes it down to `info`. But maybe people do want to see this? But maybe not the whole error message or something, maybe we use display util? Let's consider this further after the point release.


Discussion:
there was some discussion about this in a mattermost thread at [here](https://opensource.eff.org/eff-open-source/pl/a93qxq9fppbnpgnd3s4c8p661c). some of the most recent and possibly most relevant messages were:

> **brad:** i actually did want to question the fact ARI failures are now only shown in the log files or if you run with --verbose again, picking up the thread from https://opensource.eff.org/eff-open-source/pl/16x7pxw9bfdkzpfzckr13574uw
>
> my concern is that if we do this, ARI will break, we'll never know, there'll be a (potentially mass) revocation event, and certbot won't be ready to handle it. if users don't see the text and report it, how will we ever know if it breaks?
>
> maybe we can't make this robust and quiet enough to do well, but i personally think the ideal here would be to silently handle things like 500s from the server but print something to the terminal for all other ARI errors
> 
> what do y'all think? am i being paranoid?
>
> (if needed we could maybe keep track of the number of ARI failures and only report after there has been X errors in a row, but i think this is probably overkill on the first version of this)

and

> **erica:** I have a couple thoughts. First is around wondering around types and classes of errors, and which would and would not be acceptable to show to a user, and the relative prevalence of each type. 
>
> Second, and I think more actionable, is around testing. Theoretically sufficiently robust and realistic tests could alert us to some sort of failure in our code. And I also wonder if enough of our users do in fact run in a more verbose manner such that they will be enough to catch the send inform us. Particularly thinking of our more active forum contributors.

to respond to erica's message, for the first point, i think we'd have to make decisions around that on a case by case basis. i feel like if there are messages we'd like to make sure aren't ignored, we could do whatever UX and engineering work necessary to make them palatable

for the second point, we definitely should have testing regardless of anything we do here! i'm just scared that any inevitable differences between our test setups and those used by real users could allow bugs to slip through which could be real bad. without a more evidence and assurances, i would not feel confident that we can rely on people running and reading certbot with --verbose both now and that they'd continue doing so in the future

i'd also like to flag that our code that is most similar to this is https://github.com/certbot/certbot/blob/b70a9f098797c88fe64cf9546f424ae0dbae129c/certbot/src/certbot/ocsp.py. it includes a mixture of logging levels depending on the problem including warning, it runs with the same frequency that ARI does, and we've never had a major problem here

what i'd like to do is try and print an at least somewhat user readable message about things that could indicate a certbot problem with ARI such as 400s from the server in the code changed at https://github.com/certbot/certbot/pull/10335 or server missing from the renewal configuration file in https://github.com/certbot/certbot/pull/10353, while continuing to be silent about other types of errors. the error message shouldn't be very obscure, but tbh, i'm personally less worried about the user knowing how to diagnose the problem rather than it being annoying/scary enough that we're told about it popping up

thoughts? if people, especially multiple people, think this is overkill here, i'm happy to drop this point and we can close this issue. i'm just worried that if we're not more careful about ignoring errors here, we/our users may regret it later if another mass revocation happens
i'm pretty much totally in alignment w/ Brad's argument, especially around the brittleness of relying on users who both run w/ `--verbose` and also frequent the forums. we tend to diagnose the severity of an issue by both the type of bug AND the number of users talking about it in github issues/the forums, so surfacing more information about potential issues makes that signal more useful. and if we end up getting flooded w/ people concerned about non-issues, we can always course correct.

+1 to warning on non-5xx ARI errors and a missing `server` config field
My main consideration here is making sure that the things we print are always helpful to the user. If it's a noisy issue that's only helpful for us debugging, or an error that doesn't really matter and the user can do nothing about, they shouldn't have to see it.

I do like the plan of specifically catching and informing the user about particular errors, if we think there are likely to be actionable errors. In that case, my main concern is that we make it clear that the cert acquisition was still a success (if it was).

I feel like "warning" logging level is useful for things that aren't broken right now but that the user should really actually pay attention to because they might become a problem or are a problem in some cases -- pending deprecation warnings are a good example of that. So we should carefully consider how we're presenting different ARI errors; display util notify might be a gentler solution. But really I think this all depends on the particular errors.
i think we can definitely make it clear cert acquisition was a success. i'm not sure how actionable non-500 ARI errors or missing server fields will be, but i personally still think they're important for the reasons i described above. happy to do what i can tho to make the error text something you like erica

looking into the missing server issue first, i estimate less than 1800 active certbot configs may not have a server field. see https://gist.github.com/bmw/4aa1c8e888c9bd7ab2ca545eaeb121b7

because of this, i think i'm planning to essentially just bump the logging level from https://github.com/certbot/certbot/pull/10353 and play with the wording a bit in the case where there's no server field to use with ARI

alternatively, i could also check the version in the renewal config file and not warn if it's in the version range where we didn't save server when it was the default, but i kind of doubt it's worth that complexity in an initial version of this. once users upgrade and reissue their certs, they'll have a server field again

if people disagree, please lmk!
Which versions were those? Are those distro-released versions? If not, it's not super likely they'll ever upgrade imo. And even if they do, at worst they'll see it once and then we'll save it and be good after that. So I'm not too worried about that case.
Just to list out some common errors:


| error    | error code (if applicable) | show to user | level |
|-------|-----|-----|---|
| server mismatch | 4xx? | probably? | not warning because it's probably our fault? |
| no server in renewal config file | n/a | yes | warning because it's probably their fault? |
| server internal error | 5xx|  no | info |
| directory fetch error | | | |

edited to add directory fetch error, which I am working on adding a check for in https://github.com/certbot/certbot/issues/10342
> Which versions were those? Are those distro-released versions? If not, it's not super likely they'll ever upgrade imo. And even if they do, at worst they'll see it once and then we'll save it and be good after that. So I'm not too worried about that case.

affected versions were [0.5.0, 0.26.0). see [https://gist.github.com/bmw/4aa1c8e888c9bd7ab2ca545eaeb121b7#file-versions-py-L8-L13](https://gist.github.com/bmw/4aa1c8e888c9bd7ab2ca545eaeb121b7#file-versions-py-L8-L13) if you're really curious

i'm not sure if they were distro-released versions. given how few users there are, i wasn't planning to look into it, but i can if you feel it's really important

in that case, i'll go ahead with my plan to bump the logging level from https://github.com/certbot/certbot/pull/10353 and play with the wording a bit in the case where there's no server field to use with ARI unless i hear otherwise

> | error    | error code (if applicable) | show to user | level |
> |-------|-----|-----|---|
> | server mismatch | 4xx? | probably? | not warning because it's probably our fault? |

will and i above expressed wanting to show a message to the user on 400s from the server to avoid any potential problems here going unnoticed. i feel a warning logging level is the easiest and most straightforward way to do that. do you have a preference for something else and if so what?
> will and i above expressed wanting to show a message to the user on 400s from the server to avoid any potential problems here going unnoticed. i feel a warning logging level is the easiest and most straightforward way to do that. do you have a preference for something else and if so what? 

I missed that line in the discussion above. warning logging seems fine.
after looking into the details here a lot more, i'd like to propose a slight change of plans for all of our ARI errors including https://github.com/certbot/certbot/issues/10342 which i'll take on myself to fix up:

my new proposal is that in one place in renewal.py, we catch all Exceptions, log a warning level message like "An error occurred requesting ACME Renewal Information (ARI). If this problem persists and you think it's a bug in Certbot, please open an issue at https://github.com/certbot/certbot/issues/new/choose.", and log the full exception at the debug level. there are a few reasons for this:

* we want to catch all exceptions around ARI to ensure an ARI failure doesn't prevent a renewal attempt from otherwise happening
* since a general Exception catches so much, i think it's important that we don't just silently eat the error. grepping around our code for other instances of `pylint: disable=broad-except`, we always log something at a much higher level which i think we should continue to do here
* warning level messages will not be printed during automated renewal where [we include a --quiet flag](https://github.com/certbot/certbot/blob/33d3162c24ee550bc77255f840bb518f8e9776d7/snap/snapcraft.yaml#L47), [tell others](https://github.com/certbot/certbot/blob/33d3162c24ee550bc77255f840bb518f8e9776d7/certbot/docs/using.rst?plain=1#L876) to [do the same](https://github.com/certbot/certbot/blob/33d3162c24ee550bc77255f840bb518f8e9776d7/certbot/docs/packaging.rst?plain=1#L61), and [set the quiet logging level to error](https://github.com/certbot/certbot/blob/33d3162c24ee550bc77255f840bb518f8e9776d7/certbot/src/certbot/_internal/constants.py#L151-L152). i think i'd personally be fine logging at the error level here ourselves, but my hunch is that warning hits a nice middle ground of making noise for people who run `certbot renew` directly so we can learn about any problems while not spamming all of our users at once if a bug does creep in

if we wanted, i think we could add additional polish here of catching certain exceptions and logging them at a level below warning, BUT i'd like to also propose we don't do that now. i actually don't think we need it now or probably ever

for instance, in the latest released version of certbot, if we fail to fetch the directory during an ARI check, the exception propagates all the way up to [here](https://github.com/certbot/certbot/blob/v4.1.1/certbot/src/certbot/_internal/renewal.py#L651-L656) where we do log the exception at the error level. the same is true if a directory fetch fails when we actually go to renew the certificate with the renew subcommand where i expect it's been that way for many, many years

despite this loud logging, essentially no one is complaining. i think the reality is many of these networking and server 500 errors that could happen very rarely actually do. because of that, i'd prefer we do the simple thing now and only add more complex logic if we see a need really arise

what do y'all think? should i go ahead and implement this or do you deeply dislike it and me doing so would be a waste of my time?



Patch:
diff --git a/acme/src/acme/client.py b/acme/src/acme/client.py
index 238ebd27b71..64c90e06e61 100644
--- a/acme/src/acme/client.py
+++ b/acme/src/acme/client.py
@@ -335,6 +335,12 @@ def renewal_time(self, cert_pem: bytes
         :param bytes cert_pem: cert as pem file
 
         :returns: Tuple of time to attempt renewal, next time to ask for renewal info
+
+        :raises errors.ARIError: If an error occurs fetching ARI from the
+            server. Explicit exception chaining is used so the original error
+            can be accessed through the __cause__ attribute on the ARIError if
+            desired.
+
         """
         now = datetime.datetime.now()
         # https://www.ietf.org/archive/id/draft-ietf-acme-ari-08.html#section-4.3.3
@@ -358,10 +364,9 @@ def renewal_time(self, cert_pem: bytes
         ari_url = renewal_info_base_url + '/' + _renewal_info_path_component(cert)
         try:
             resp = self.net.get(ari_url, content_type='application/json')
-        except (requests.exceptions.RequestException, messages.Error) as error:
-            logger.info("failed to fetch renewal_info URL (%s): %s", ari_url, error)
-            return None, now + default_retry_after
-
+        except Exception as e:  # pylint: disable=broad-except
+            error_msg = f'failed to fetch renewal_info URL {ari_url}'
+            raise errors.ARIError(error_msg, now + default_retry_after) from e
         renewal_info: messages.RenewalInfo = messages.RenewalInfo.from_json(resp.json())
 
         start = renewal_info.suggested_window.start # pylint: disable=no-member
diff --git a/acme/src/acme/errors.py b/acme/src/acme/errors.py
index 5e0a5379d12..9b51b69875f 100644
--- a/acme/src/acme/errors.py
+++ b/acme/src/acme/errors.py
@@ -1,4 +1,5 @@
 """ACME errors."""
+import datetime
 import typing
 from typing import Any
 from typing import List
@@ -149,3 +150,10 @@ def __init__(self, location: str) -> None:
 
 class WildcardUnsupportedError(Error):
     """Error for when a wildcard is requested but is unsupported by ACME CA."""
+
+
+class ARIError(ClientError):
+    """An error occurred during an ARI request and we want to suggest a Retry-After time."""
+    def __init__(self, message: str, retry_after: datetime.datetime) -> None:
+        super().__init__(message, retry_after)
+        self.retry_after = retry_after
diff --git a/certbot/src/certbot/_internal/renewal.py b/certbot/src/certbot/_internal/renewal.py
index 64a7b6af8e5..414b93c4c33 100644
--- a/certbot/src/certbot/_internal/renewal.py
+++ b/certbot/src/certbot/_internal/renewal.py
@@ -325,6 +325,45 @@ def should_renew(config: configuration.NamespaceConfig,
     return False
 
 
+def _ari_renewal_time(config: configuration.NamespaceConfig,
+                      lineage: storage.RenewableCert,
+                      acme_clients: Dict[str, acme_client.ClientV2],
+                      cert_pem: bytes,
+                      )-> Optional[datetime.datetime]:
+    """Return the ARI suggested renewal time if it's available."""
+    # For ARI requests, we want to use the ACME directory URL from which the
+    # cert was originally requested. Since `config.server` can be overridden on
+    # the command line, we're using the server stored in the cert's renewal
+    # conf, i.e. `lineage.server`
+    #
+    # Fixes https://github.com/certbot/certbot/issues/10339
+    if lineage.server:
+        try:
+            # Creating a new ACME client makes a network request, so check if we have
+            # one cached for this cert's server already
+            if lineage.server not in acme_clients:
+                acme_clients[lineage.server] = \
+                    client.create_acme_client(config, server_override=lineage.server)
+            acme = acme_clients.get(lineage.server, None)
+
+            # Attempt to get the ARI-defined renewal time
+            if acme:
+                return acme.renewal_time(cert_pem)[0]
+        except Exception:  # pylint: disable=broad-except
+            # We want to stop errors around ARI preventing renewal so we catch all exceptions here
+            # with a warning asking users to tell us about any problems they are experiencing
+            logger.warning("An error occurred requesting ACME Renewal Information (ARI). If this "
+                           "problem persists and you think it's a bug in Certbot, please open an "
+                           "issue at https://github.com/certbot/certbot/issues/new/choose.")
+            logger.debug("Error while requesting ARI was:", exc_info=True)
+    else:
+        renewal_conf_file = storage.renewal_filename_for_lineagename(config, lineage.lineagename)
+        logger.warning("Skipping ARI check because %s has no 'server' field. This issue will not "
+                       "prevent certificate renewal", renewal_conf_file)
+
+    return None
+
+
 def _default_renewal_time(cert_pem: bytes) -> datetime.datetime:
     """Return an reasonable default time to attempt renewal of the certificate
     based on the certificate lifetime.
@@ -374,32 +413,7 @@ def should_autorenew(config: configuration.NamespaceConfig,
     with open(cert, 'rb') as f:
         cert_pem = f.read()
 
-    renewal_time = None
-    # For ARI requests, we want to use the ACME directory URL from which the
-    # cert was originally requested. Since `config.server` can be overridden on
-    # the command line, we're using the server stored in the cert's renewal
-    # conf, i.e. `lineage.server`
-    #
-    # Fixes https://github.com/certbot/certbot/issues/10339
-    if lineage.server:
-        # Creating a new ACME client makes a network request, so check if we have
-        # one cached for this cert's server already
-        if lineage.server not in acme_clients:
-            try:
-                acme_clients[lineage.server] = \
-                    client.create_acme_client(config, server_override=lineage.server)
-            except Exception as error:  # pylint: disable=broad-except
-                logger.info("Unable to connect to %s to request ACME Renewal Information (ARI). "
-                            "Error was: %s", lineage.server, error)    
-        acme = acme_clients.get(lineage.server, None)
-
-        # Attempt to get the ARI-defined renewal time
-        if acme:
-            renewal_time, _ = acme.renewal_time(cert_pem)
-    else:
-        renewal_conf_file = storage.renewal_filename_for_lineagename(config, lineage.lineagename)
-        logger.warning("Skipping ARI check because %s has no 'server' field. This issue will not "
-                       "prevent certificate renewal", renewal_conf_file)
+    renewal_time = _ari_renewal_time(config, lineage, acme_clients, cert_pem)
 
     now = datetime.datetime.now(datetime.timezone.utc)
 


Test patch:
diff --git a/acme/src/acme/_internal/tests/client_test.py b/acme/src/acme/_internal/tests/client_test.py
index 885c6ac5b9a..7d8f44f9c9c 100644
--- a/acme/src/acme/_internal/tests/client_test.py
+++ b/acme/src/acme/_internal/tests/client_test.py
@@ -548,27 +548,38 @@ def test_renewal_time_with_renewal_info(self, dt_mock):
 
     @mock.patch('acme.client.datetime')
     def test_renewal_time_renewal_info_errors(self, dt_mock):
-        utc_now = datetime.datetime(2025, 3, 15, tzinfo=datetime.timezone.utc)
-        dt_mock.datetime.now.return_value = utc_now
+        def now(tzinfo=None):
+            return datetime.datetime(2025, 3, 15, tzinfo=tzinfo)
+        dt_mock.datetime.now.side_effect = now
+        dt_mock.timedelta = datetime.timedelta
+        dt_mock.timezone = datetime.timezone
+
         self.client.directory = messages.Directory({
             'renewalInfo': 'https://www.letsencrypt-demo.org/acme/renewal-info',
         })
-        # Failure to fetch the 'renewalInfo' URL should return None
-        self.net.get.side_effect = requests.exceptions.RequestException
+        # Failure to fetch the 'renewalInfo' URL should raise an ARIError with the exception raised
+        # by self.net.get as the __cause__ and a Retry-After 6 hours in the future
+        expected_cause = requests.exceptions.RequestException
+        expected_retry_after = now() + datetime.timedelta(seconds=6 * 60 * 60)
+        self.net.get.side_effect = expected_cause
 
         cert_pem = make_cert_for_renewal(
             not_before=datetime.datetime(2025, 3, 12, 00, 00, 00),
             not_after=datetime.datetime(2025, 3, 20, 00, 00, 00),
         )
-        t, _ = self.client.renewal_time(cert_pem)
-        assert t == None
+        with pytest.raises(errors.ARIError) as exception_info:
+            self.client.renewal_time(cert_pem)
+        assert isinstance(exception_info.value.__cause__, expected_cause)
+        assert exception_info.value.retry_after == expected_retry_after
 
         cert_pem = make_cert_for_renewal(
             not_before=datetime.datetime(2025, 3, 12, 00, 00, 00),
             not_after=datetime.datetime(2025, 3, 30, 00, 00, 00),
         )
-        t, _ = self.client.renewal_time(cert_pem)
-        assert t == None
+        with pytest.raises(errors.ARIError) as exception_info:
+            self.client.renewal_time(cert_pem)
+        assert isinstance(exception_info.value.__cause__, expected_cause)
+        assert exception_info.value.retry_after == expected_retry_after
 
     @mock.patch('acme.client.datetime')
     def test_renewal_time_returns_retry_after(self, dt_mock):
diff --git a/certbot/src/certbot/_internal/tests/renewal_test.py b/certbot/src/certbot/_internal/tests/renewal_test.py
index 9ba3a35d4ed..cbe62aeeb36 100644
--- a/certbot/src/certbot/_internal/tests/renewal_test.py
+++ b/certbot/src/certbot/_internal/tests/renewal_test.py
@@ -10,6 +10,7 @@
 import pytz
 
 from acme import challenges
+from acme import errors as acme_errors
 from certbot import configuration
 from certbot import errors
 from certbot._internal import storage
@@ -440,8 +441,35 @@ def test_resilient_ari_directory_fetches(self, mock_renewal_time, mock_ocsp, moc
         mock_ocsp.return_value = True
 
         with mock.patch('certbot._internal.renewal.open', mock.mock_open(read_data=b'')):
-            assert renewal.should_autorenew(self.config, mock_rc, acme_clients)
-            assert mock_renewal_time.call_count == 0
+            with mock.patch('certbot._internal.renewal.logger') as mock_logger:
+                assert renewal.should_autorenew(self.config, mock_rc, acme_clients)
+        assert mock_renewal_time.call_count == 0
+        # Ensure we logged about skipping the ARI check and the underlying exception
+        assert any('ARI' in call.args[0] for call in mock_logger.warning.call_args_list)
+        assert any(call.kwargs.get('exc_info') for call in mock_logger.debug.call_args_list)
+
+
+    @mock.patch('certbot._internal.storage.RenewableCert.ocsp_revoked')
+    def test_resilient_ari_check(self, mock_ocsp):
+        from certbot._internal import renewal
+
+        mock_acme = mock.MagicMock()
+        ari_error = acme_errors.ARIError('some error', datetime.datetime.now())
+        ari_server = 'http://ari'
+        mock_acme.renewal_time.side_effect = ari_error
+        acme_clients = {}
+        acme_clients[ari_server] = mock_acme
+        mock_rc = mock.MagicMock()
+        mock_rc.server = ari_server
+        mock_rc.autorenewal_is_enabled.return_value = True
+        mock_ocsp.return_value = True
+
+        with mock.patch('certbot._internal.renewal.open', mock.mock_open(read_data=b'')):
+            with mock.patch('certbot._internal.renewal.logger') as mock_logger:
+                assert renewal.should_autorenew(self.config, mock_rc, acme_clients)
+        # Ensure we logged about skipping the ARI check and the underlying exception
+        assert any('ARI' in call.args[0] for call in mock_logger.warning.call_args_list)
+        assert any(call.kwargs.get('exc_info') for call in mock_logger.debug.call_args_list)
 
 
 class RestoreRequiredConfigElementsTest(test_util.ConfigTestCase):