Repository: certbot/certbot

Issue:
Should auto-recover from bad nonces
I recently got this error trying to issue a cert:

`The client sent an unacceptable anti-replay nonce :: JWS has invalid anti-replay nonce`

This happens from time to time - the server loses track of outstanding nonces. Since the reply always includes a fresh nonce, the client should auto-retry the request.

Should auto-recover from bad nonces
I recently got this error trying to issue a cert:

`The client sent an unacceptable anti-replay nonce :: JWS has invalid anti-replay nonce`

This happens from time to time - the server loses track of outstanding nonces. Since the reply always includes a fresh nonce, the client should auto-retry the request.



Discussion:
I got this error several times today as well, all the errors were from the production server, staging was working fine. Might be something to do with loads, however as noted, it's trivial to implement retry on client side. 

I am getting the same error and retry doesn't solve the problem.
Unfortunately few retries also makes it that the domain hits the certificate limit and now we have to wait 7 days.

We offer this to all our client and now we are worried we are going to have serious issues comes the renewal date.

Ok here is out this finally worked for me, We have to repeat the new certificate step and use the nonce that is sent back with "jws invalid nonce" to get the certificate issued. 
Otherwise if we go through the whole process again it always fails.

There must be a new bug on the server side making resulting in this but overall because of the workaround it is not a show stopper.

BTW, @rezaalavi your comment suggests your developing alternative client. This repo (and this bug) is for the official client implementation.

FWIW I just got this error, updated the letsencrypt code and then stopped getting it.

I was also encountering this error and after update of the Gentoo package and it worked!

I've been hit by that invalid anti-replay nonce problem for my use case (DNS challenge, where I had to wait up to half an hour before the challenge value is published by my DNS provider).

The first time, it was with the staging environment, but it did not reproduced afterwards. I was then hit numerous time with the production environment, so I followed Jacob's suggestion and make the client retries POSTs on badNonce errors.

Attached is the patch (against certbot-0.9.3, the version I use) I've extracted from my deployment of certbot:

[cerbot-acme-clientpy.patch.txt](https://github.com/certbot/certbot/files/659682/cerbot-acme-clientpy.patch.txt)

I'm not a pythonist, so it may not be nice python code: maybe one can clean this attempt and make a pull request from it?

edit: typo in jsha's first name, sorry.
Just out of my curiosity, what stops `boulder` from keeping track of nonces making this a problem that has to be solved by the different ACME clients?
Many nonces go unused, which means we have to expire them after a while. More importantly, we currently keep track of nonces per-frontend and therefore per-DC. When we switch traffic between DCs, any inflight sessions will get badNonce errors.
Because there is no limit how long it can take to perform a challenge when using the `manual` plugin, I'm bumping the priority of this issue. Due to how long it can take for DNS changes to propagate, this problem makes using Certbot with the DNS challenge impossible in some cases (see #4099). 
Attached is an updated version of my previous patch (against 49d46ef, and with a more idiomatic way to propagate the raised exception when max number of attempts is reached):

[~~cerbot-acme-clientpy-v2.patch.txt~~](https://github.com/certbot/certbot/files/726405/cerbot-acme-clientpy-v2.patch.txt)
[cerbot-acme-clientpy-v2-proper-urn.patch.txt](https://github.com/certbot/certbot/files/726528/cerbot-acme-clientpy-v2-proper-urn.patch.txt)

As a workaround, should be appliable against published v0.10.0 and v0.10.1.

Edit: the URN prefix for acme errors has changed ("urn:ietf:params:acme:error:" instead of "urn:acme:error:"); updated patch.
I got this error several times today as well, all the errors were from the production server, staging was working fine. Might be something to do with loads, however as noted, it's trivial to implement retry on client side. 

I am getting the same error and retry doesn't solve the problem.
Unfortunately few retries also makes it that the domain hits the certificate limit and now we have to wait 7 days.

We offer this to all our client and now we are worried we are going to have serious issues comes the renewal date.

Ok here is out this finally worked for me, We have to repeat the new certificate step and use the nonce that is sent back with "jws invalid nonce" to get the certificate issued. 
Otherwise if we go through the whole process again it always fails.

There must be a new bug on the server side making resulting in this but overall because of the workaround it is not a show stopper.

BTW, @rezaalavi your comment suggests your developing alternative client. This repo (and this bug) is for the official client implementation.

FWIW I just got this error, updated the letsencrypt code and then stopped getting it.

I was also encountering this error and after update of the Gentoo package and it worked!

I've been hit by that invalid anti-replay nonce problem for my use case (DNS challenge, where I had to wait up to half an hour before the challenge value is published by my DNS provider).

The first time, it was with the staging environment, but it did not reproduced afterwards. I was then hit numerous time with the production environment, so I followed Jacob's suggestion and make the client retries POSTs on badNonce errors.

Attached is the patch (against certbot-0.9.3, the version I use) I've extracted from my deployment of certbot:

[cerbot-acme-clientpy.patch.txt](https://github.com/certbot/certbot/files/659682/cerbot-acme-clientpy.patch.txt)

I'm not a pythonist, so it may not be nice python code: maybe one can clean this attempt and make a pull request from it?

edit: typo in jsha's first name, sorry.
Just out of my curiosity, what stops `boulder` from keeping track of nonces making this a problem that has to be solved by the different ACME clients?
Many nonces go unused, which means we have to expire them after a while. More importantly, we currently keep track of nonces per-frontend and therefore per-DC. When we switch traffic between DCs, any inflight sessions will get badNonce errors.
Because there is no limit how long it can take to perform a challenge when using the `manual` plugin, I'm bumping the priority of this issue. Due to how long it can take for DNS changes to propagate, this problem makes using Certbot with the DNS challenge impossible in some cases (see #4099). 
Attached is an updated version of my previous patch (against 49d46ef, and with a more idiomatic way to propagate the raised exception when max number of attempts is reached):

[~~cerbot-acme-clientpy-v2.patch.txt~~](https://github.com/certbot/certbot/files/726405/cerbot-acme-clientpy-v2.patch.txt)
[cerbot-acme-clientpy-v2-proper-urn.patch.txt](https://github.com/certbot/certbot/files/726528/cerbot-acme-clientpy-v2-proper-urn.patch.txt)

As a workaround, should be appliable against published v0.10.0 and v0.10.1.

Edit: the URN prefix for acme errors has changed ("urn:ietf:params:acme:error:" instead of "urn:acme:error:"); updated patch.

Resolving patch:
diff --git a/acme/acme/client.py b/acme/acme/client.py
--- a/acme/acme/client.py
+++ b/acme/acme/client.py
@@ -674,8 +674,25 @@ def _get_nonce(self, url):
             self._add_nonce(self.head(url))
         return self._nonces.pop()
 
-    def post(self, url, obj, content_type=JOSE_CONTENT_TYPE, **kwargs):
-        """POST object wrapped in `.JWS` and check response."""
+    def post(self, *args, **kwargs):
+        """POST object wrapped in `.JWS` and check response.
+
+        If the server responded with a badNonce error, the request will
+        be retried once.
+
+        """
+        should_retry = True
+        while True:
+            try:
+                return self._post_once(*args, **kwargs)
+            except messages.Error as error:
+                if should_retry and error.code == 'badNonce':
+                    logger.debug('Retrying request after error:\n%s', error)
+                    should_retry = False
+                else:
+                    raise
+
+    def _post_once(self, url, obj, content_type=JOSE_CONTENT_TYPE, **kwargs):
         data = self._wrap_in_jws(obj, self._get_nonce(url))
         kwargs.setdefault('headers', {'Content-Type': content_type})
         response = self._send_request('POST', url, data=data, **kwargs)


Test patch:
diff --git a/acme/acme/client_test.py b/acme/acme/client_test.py
--- a/acme/acme/client_test.py
+++ b/acme/acme/client_test.py
@@ -642,7 +642,9 @@ def setUp(self):
         self.wrapped_obj = mock.MagicMock()
         self.content_type = mock.sentinel.content_type
 
-        self.all_nonces = [jose.b64encode(b'Nonce'), jose.b64encode(b'Nonce2')]
+        self.all_nonces = [
+            jose.b64encode(b'Nonce'),
+            jose.b64encode(b'Nonce2'), jose.b64encode(b'Nonce3')]
         self.available_nonces = self.all_nonces[:]
 
         def send_request(*args, **kwargs):
@@ -690,7 +692,7 @@ def test_post(self):
         self.net._wrap_in_jws.assert_called_once_with(
             self.obj, jose.b64decode(self.all_nonces.pop()))
 
-        assert not self.available_nonces
+        self.available_nonces = []
         self.assertRaises(errors.MissingNonce, self.net.post,
                           'uri', self.obj, content_type=self.content_type)
         self.net._wrap_in_jws.assert_called_with(
@@ -706,6 +708,25 @@ def test_post_wrong_post_response_nonce(self):
         self.assertRaises(errors.BadNonce, self.net.post, 'uri',
                           self.obj, content_type=self.content_type)
 
+    def test_post_failed_retry(self):
+        check_response = mock.MagicMock()
+        check_response.side_effect = messages.Error.with_code('badNonce')
+
+        # pylint: disable=protected-access
+        self.net._check_response = check_response
+        self.assertRaises(messages.Error, self.net.post, 'uri',
+                          self.obj, content_type=self.content_type)
+
+    def test_post_successful_retry(self):
+        check_response = mock.MagicMock()
+        check_response.side_effect = [messages.Error.with_code('badNonce'),
+                                      self.checked_response]
+
+        # pylint: disable=protected-access
+        self.net._check_response = check_response
+        self.assertEqual(self.checked_response, self.net.post(
+            'uri', self.obj, content_type=self.content_type))
+
     def test_head_get_post_error_passthrough(self):
         self.send_request.side_effect = requests.exceptions.RequestException
         for method in self.net.head, self.net.get: