Repository: certbot/certbot

Issue:
Certbot 1.32.2 can hang polling Sectigo/ZeroSSL authz
If you're having trouble using Certbot and aren't sure you've found a bug or
request for a new feature, please first try asking for help at
https://community.letsencrypt.org/. There is a much larger community there of
people familiar with the project who will be able to more quickly answer your
questions.

## My operating system is (include version):
Ubuntu 20.04 (amd64)

## I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):
snap (Certbot 1.32.2)

## I ran this command and it produced this output:
2022-12-20:
```
$ sudo certbot certonly --cert-name dynroute53.mattnordhoff.net-ecdsa --key-type ecdsa --manual --manual-auth-hook "/opt/certbot_dns/certbot_dns.py --auth --config /opt/certbot_dns/certbot_dns.ini --route-53" --manual-cleanup-hook "/opt/certbot_dns/certbot_dns.py --cleanup --config /opt/certbot_dns/certbot_dns.ini --route-53" --preferred-challenges dns-01 --server https://acme.zerossl.com/v2/DV90 -d dynroute53.mattnordhoff.net -d www3.mattnordhoff.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for dynroute53.mattnordhoff.net and www3.mattnordhoff.net
Hook '--manual-auth-hook' for dynroute53.mattnordhoff.net ran with output:
 {"value": "\"XB7wfYhYVeZ2IFzi8RcxpLSQItb4qCr54NhZ7GNbJfc\"", "debug_host_name": "_acme-challenge.dynroute53", "zone_id": "Z098584139YZXBJKADLJ1", "version": 2, "debug_change_id": "C06296722GWQXDI0GI4RE", "debug_zone_name": "mattnordhoff.net", "fqdn": "_acme-challenge.dynroute53.mattnordhoff.net."}
Hook '--manual-auth-hook' for www3.mattnordhoff.net ran with output:
 {"debug_host_name": "_acme-challenge.www3", "version": 2, "fqdn": "_acme-challenge.www3.mattnordhoff.net.", "debug_zone_name": "mattnordhoff.net", "value": "\"nlX5TU5kV8EcUY6Ca16Vf2klaBMYOMgfnFo0Kc2dGbM\"", "debug_change_id": "C03741053SSN8JIFWERCD", "zone_id": "Z098584139YZXBJKADLJ1"}
^CExiting due to user request.

2022-12-21 00:00:11,171:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/snap/certbot/2618/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 106, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/snap/certbot/2618/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 164, in _poll_authorizations
    time.sleep(sleep_seconds)
KeyboardInterrupt
```

2022-12-31: It was the snap certbot renew timer; n/a.

(I haven't published `certbot_dns.py`.)

## Certbot's behavior differed from what I expected because:
Recently, on two different systems (both using 1.32.2 from snap), Certbot hung while polling an authz from ZeroSSL (which uses Sectigo's white label ACME API).

The first time, I hit ^C after an hour. The second time, I sent SIGTERM after over 12 hours. Both times, I noticed a CLOSE_WAIT connection to the ACME server, e.g.:

```
$ sudo lsof -ni | rg python
python3   2657947       root    8u  IPv6 232524025      0t0  TCP [2604:a880:400:d0::83:2002]:47062->[2a0e:ac00:c7:d450::5bc7:d450]:https (CLOSE_WAIT)
```

Sectigo/ZeroSSL is often so slow that Certbot gives up and exits, but it's not supposed to *actually freeze*. It seems like there's a bug somewhere. Maybe there's a polling issue. Maybe the connection gets into a weird state and Certbot/requests/Python doesn't handle it well. I dunno.

## Here is a Certbot log showing the issue (if available):
###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.
2022-12-20: https://mn0.us/oT3aso7eoMVo/letsencrypt.log
2022-12-31: https://mn0.us/QruuC4BbvaBJ/letsencrypt.log

It doesn't show much; Certbot doesn't have a log entry for "nothing is happening". :-)

The second one shows an OCSP query timeout (which is somewhat common). The OCSP hostname uses Cloudflare but the ACME hostname does not, AFAICT.

## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:
Not relevant.


Discussion:
I would guess that Certbot is doing its best to respect this response header:

>Retry-After: 86400


Note that Certbots parsing of `Retry-After` headers *might* be buggy, see https://github.com/certbot/certbot/issues/9319
1.) Oh jeez. I didn't even notice the Retry-After header.

That makes sense. The one with a traceback was in the middle of `time.sleep(sleep_seconds)`. (I didn't read the code to check for context, though.)

Well... Now what?

We haven't confirmed it, but everything may be working as designed.

What about considering this a feature request for something like, "Certbot should error out if Retry-After is more than 30 minutes?"

2.) I forgot to mention it, but I'm using UTC.
If I'm reading my logs right, the poll interval makes no sense. It looks like Certbot interprets `Retry-After` like:

* `5`: 5 seconds
* `10`: 0
* `60`: 0 the first time; 60 seconds the second time
* `300`: 0
* `86400`: At least an hour

What in the world?

I wonder if Sectigo is treating the traffic as abusive -- although that's not fair when it starts out at 5 seconds.
From another perspective, it's a bug that Retry-After can exceed the overall issuance timeout (if it's intended to apply to the authz polling, not just the order polling).
> * `10`: 0
> * `60`: 0 the first time; 60 seconds the second time

I haven't checked it *thoroughly* but this is probably a misreading of the logs as I believe Certbot couples a certain retry-after to the type of POST that was send. E.g., if a *challenge* POST returns a retry-value of 10 seconds, Certbot doesn't have an issue with trying an *autz* POST immediately after the *chall* POST.
Ah. I didn't compare the URLs before.

My first log has:

```
2022-12-20 22:55:16,410:DEBUG:acme.client:Sending POST request to https://acme.zerossl.com/v2/DV90/chall/vfH1QY8aC0PddRkyNzc0Zw:
2022-12-20 22:55:17,126:DEBUG:urllib3.connectionpool:https://acme.zerossl.com:443 "POST /v2/DV90/chall/vfH1QY8aC0PddRkyNzc0Zw HTTP/1.1" 200 163
2022-12-20 22:55:17,127:DEBUG:acme.client:Received response:
Retry-After: 10

2022-12-20 22:55:17,130:DEBUG:acme.client:Sending POST request to https://acme.zerossl.com/v2/DV90/chall/s1W6Zsm1LskamUNnQu2eOA:
2022-12-20 22:55:17,712:DEBUG:urllib3.connectionpool:https://acme.zerossl.com:443 "POST /v2/DV90/chall/s1W6Zsm1LskamUNnQu2eOA HTTP/1.1" 200 163
2022-12-20 22:55:17,712:DEBUG:acme.client:Received response:
Retry-After: 10

2022-12-20 22:55:18,715:DEBUG:acme.client:Sending POST request to https://acme.zerossl.com/v2/DV90/authz/POqcNQdCSmPrxb4jQWAx1A:
2022-12-20 22:55:19,406:DEBUG:urllib3.connectionpool:https://acme.zerossl.com:443 "POST /v2/DV90/authz/POqcNQdCSmPrxb4jQWAx1A HTTP/1.1" 200 460
2022-12-20 22:55:19,407:DEBUG:acme.client:Received response:
Retry-After: 5
```

Maybe it's specific to the exact URL, or the exact object, rather than just the type?
> or the exact object

It's  probably that indeed. Certbot seems to be associating the retry after values to the exact object, but the Sectigo ACME server seems not to:

From your first log file:

Timestamp | Type | ID | Retry-After
-- | -- | -- | --
22:54:11 | authz | POqcNQdCSmPrxb4jQWAx1A | 5
22:54:12 | authz | U9Ucx8a8A4G7EOtiUZZpnw | 5
22:55:17 | chall | vfH1QY8aC0PddRkyNzc0Zw | 10
22:55:17 | chall | s1W6Zsm1LskamUNnQu2eOA | 10
22:55:19 | authz | POqcNQdCSmPrxb4jQWAx1A | 5
22:55:20 | authz | U9Ucx8a8A4G7EOtiUZZpnw | 5
22:55:25 | authz | POqcNQdCSmPrxb4jQWAx1A | 5
22:55:26 | authz | U9Ucx8a8A4G7EOtiUZZpnw | 5
(...) |   |   |  
22:56:16 | authz | POqcNQdCSmPrxb4jQWAx1A | 60
22:56:17 | authz | U9Ucx8a8A4G7EOtiUZZpnw | 60
22:57:19 | authz | POqcNQdCSmPrxb4jQWAx1A | 300
22:57:20 | authz | U9Ucx8a8A4G7EOtiUZZpnw | 86400

The last few retry after values go from 300 to 86400 with no regard to the actual object ID. And in your second log, it goes from 5 to 86400 immediately without the 300 in between. Maybe it depends on the current load on how long it delays clients?

Oh and the 10 seconds retry after seems to be just for challenges and for authz the default is 5 it seems.
I am struggling to guess what the intention of Sectigo sending 86400 is vs. a permanent error to make the client go away and come again another day.

I checked lego and acme.sh, they will hang around for 100 seconds and 90 seconds respectively. 

Certbot (without `Retry-After` involved) hangs around for (30 retries * 3 seconds default interval) = 90 seconds.

A simple approach might be to keep [`_poll_authorizations`](https://github.com/certbot/certbot/blob/b81ef33f33d69451f1ab981583240a1bac79f7c0/certbot/certbot/_internal/auth_handler.py#L145-L210)'s current approach of `max_retries`, but to set a hardcoded deadline of 30 minutes on the function, as suggested by @mnordhoff. Any sleep longer than that gets truncated.

That way, we give the server some wiggle room to delay clients, but not excessively so. Notably, it keeps Certbot's default 90 second behavior in the absence of `Retry-After` - we wouldn't be changing that.



Resolving patch:
diff --git a/certbot/certbot/_internal/auth_handler.py b/certbot/certbot/_internal/auth_handler.py
--- a/certbot/certbot/_internal/auth_handler.py
+++ b/certbot/certbot/_internal/auth_handler.py
@@ -55,7 +55,8 @@ def __init__(self, auth: interfaces.Authenticator, acme_client: Optional[client.
 
     def handle_authorizations(self, orderr: messages.OrderResource,
                               config: configuration.NamespaceConfig, best_effort: bool = False,
-                              max_retries: int = 30) -> List[messages.AuthorizationResource]:
+                              max_retries: int = 30,
+                              max_time_mins: float = 30) -> List[messages.AuthorizationResource]:
         """
         Retrieve all authorizations, perform all challenges required to validate
         these authorizations, then poll and wait for the authorization to be checked.
@@ -63,6 +64,7 @@ def handle_authorizations(self, orderr: messages.OrderResource,
         :param certbot.configuration.NamespaceConfig config: current Certbot configuration
         :param bool best_effort: if True, not all authorizations need to be validated (eg. renew)
         :param int max_retries: maximum number of retries to poll authorizations
+        :param float max_time_mins: maximum time (in minutes) to poll authorizations
         :returns: list of all validated authorizations
         :rtype: List
 
@@ -103,7 +105,7 @@ def handle_authorizations(self, orderr: messages.OrderResource,
 
             # Wait for authorizations to be checked.
             logger.info('Waiting for verification...')
-            self._poll_authorizations(authzrs, max_retries, best_effort)
+            self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
 
             # Keep validated authorizations only. If there is none, no certificate can be issued.
             authzrs_validated = [authzr for authzr in authzrs
@@ -143,11 +145,11 @@ def deactivate_valid_authorizations(self, orderr: messages.OrderResource) -> Tup
         return (deactivated, failed)
 
     def _poll_authorizations(self, authzrs: List[messages.AuthorizationResource], max_retries: int,
-                             best_effort: bool) -> None:
+                             deadline_minutes: float, best_effort: bool) -> None:
         """
         Poll the ACME CA server, to wait for confirmation that authorizations have their challenges
         all verified. The poll may occur several times, until all authorizations are checked
-        (valid or invalid), or after a maximum of retries.
+        (valid or invalid), or a maximum of retries, or the polling deadline is reached.
         """
         if not self.acme:
             raise errors.Error("No ACME client defined, cannot poll authorizations.")
@@ -156,6 +158,7 @@ def _poll_authorizations(self, authzrs: List[messages.AuthorizationResource], ma
                                           Optional[Response]]] = {index: (authzr, None)
                             for index, authzr in enumerate(authzrs)}
         authzrs_failed_to_report = []
+        deadline = datetime.datetime.now() + datetime.timedelta(minutes=deadline_minutes)
         # Give an initial second to the ACME CA server to check the authorizations
         sleep_seconds: float = 1
         for _ in range(max_retries):
@@ -184,7 +187,7 @@ def _poll_authorizations(self, authzrs: List[messages.AuthorizationResource], ma
             authzrs_to_check = {index: (authzr, resp) for index, (authzr, resp)
                                 in authzrs_to_check.items()
                                 if authzr.body.status == messages.STATUS_PENDING}
-            if not authzrs_to_check:
+            if not authzrs_to_check or datetime.datetime.now() > deadline:
                 # Polling process is finished, we can leave the loop
                 break
 
@@ -196,6 +199,9 @@ def _poll_authorizations(self, authzrs: List[messages.AuthorizationResource], ma
             retry_after = max(self.acme.retry_after(resp, 3)
                               for _, resp in authzrs_to_check.values()
                               if resp is not None)
+            # Whatever Retry-After the ACME server requests, the polling must not take
+            # longer than the overall deadline (https://github.com/certbot/certbot/issues/9526).
+            retry_after = min(retry_after, deadline)
             sleep_seconds = (retry_after - datetime.datetime.now()).total_seconds()
 
         # In case of failed authzrs, create a report to the user.


Test patch:
diff --git a/certbot/tests/auth_handler_test.py b/certbot/tests/auth_handler_test.py
--- a/certbot/tests/auth_handler_test.py
+++ b/certbot/tests/auth_handler_test.py
@@ -1,5 +1,5 @@
 """Tests for certbot._internal.auth_handler."""
-import functools
+import datetime
 import logging
 import unittest
 
@@ -12,7 +12,6 @@
 from acme import messages
 from certbot import achallenges
 from certbot import errors
-from certbot import util
 from certbot._internal.display import obj as display_obj
 from certbot.plugins import common as plugin_common
 from certbot.tests import acme_util
@@ -227,6 +226,39 @@ def test_max_retries_exceeded(self):
             self.handler.handle_authorizations(mock_order, self.mock_config, False, 1)
         self.assertIn('All authorizations were not finalized by the CA.', str(error.exception))
 
+    @mock.patch('certbot._internal.auth_handler.time.sleep')
+    def test_deadline_exceeded(self, mock_sleep):
+        authzrs = [gen_dom_authzr(domain="0", challs=acme_util.CHALLENGES)]
+        mock_order = mock.MagicMock(authorizations=authzrs)
+
+        orig_now = datetime.datetime.now
+        state = {'time_slept': 0}
+
+        def mock_sleep_effect(secs):
+            state['time_slept'] += secs
+        mock_sleep.side_effect = mock_sleep_effect
+
+        def mock_now_effect():
+            return orig_now() + datetime.timedelta(seconds=state["time_slept"])
+
+        # We will return STATUS_PENDING and ask Certbot to sleep for 20 minutes at a time.
+        interval = datetime.timedelta(minutes=20).seconds
+        self.mock_net.poll.side_effect = _gen_mock_on_poll(status=messages.STATUS_PENDING,
+                                                           wait_value=interval)
+
+        with self.assertRaises(errors.AuthorizationError) as error, \
+             mock.patch('certbot._internal.auth_handler.datetime.datetime') as mock_dt:
+            mock_dt.now.side_effect = mock_now_effect
+            # Polling will only proceed for 30 minutes at most, so the second 20 minute sleep
+            # should be truncated and the polling should be aborted.
+            self.handler.handle_authorizations(mock_order, self.mock_config, False)
+        self.assertIn('All authorizations were not finalized by the CA.', str(error.exception))
+
+        self.assertEqual(mock_sleep.call_count, 3) # 1s, 20m and 10m sleep
+        self.assertEqual(mock_sleep.call_args_list[0][0][0], 1)
+        self.assertAlmostEqual(mock_sleep.call_args_list[1][0][0], interval - 1, delta=1)
+        self.assertAlmostEqual(mock_sleep.call_args_list[2][0][0], interval/2 - 1, delta=1)
+
     def test_no_domains(self):
         mock_order = mock.MagicMock(authorizations=[])
         self.assertRaises(errors.AuthorizationError, self.handler.handle_authorizations,