Repository: certbot/certbot

Issue:
Use the ACMEv2 newNonce endpoint when a new nonce is needed
In our testing, sometimes Akamai fails to reach Let's Encrypt, and returns a 500 with malformed (non-JSON) contents. Certbot reports this as a missing nonce; instead it should report the malformed body and ignore the lack of a nonce.


Discussion:
Here's an example of the exact 503 response and the traceback:
```
<snipped>
2018-06-19 20:00:41,668:INFO:certbot.main:Obtaining a new certificate
2018-06-19 20:00:41,779:DEBUG:certbot.crypto_util:Generating key (2048 bits): /home/xxx/xxx/prod-acmev02-1/conf/keys/6980_key-certbot.pem
2018-06-19 20:00:41,835:DEBUG:certbot.crypto_util:Creating CSR: /home/xxx/xxx/prod-acmev02-1/conf/csr/6980_csr-certbot.pem
2018-06-19 20:00:41,836:DEBUG:acme.client:Requesting fresh nonce
2018-06-19 20:00:41,836:DEBUG:acme.client:Sending HEAD request to https://acme-v02-1.api.letsencrypt.org/acme/new-order.
2018-06-19 20:00:41,991:DEBUG:urllib3.connectionpool:https://acme-v02-1.api.letsencrypt.org:443 "HEAD /acme/new-order HTTP/1.1" 503 0
2018-06-19 20:00:41,992:DEBUG:acme.client:Received response:
HTTP 503
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 271
Expires: Tue, 19 Jun 2018 20:00:41 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 19 Jun 2018 20:00:41 GMT
Connection: keep-alive

2018-06-19 20:00:41,992:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/home/xxx/certbot_venv/bin/certbot", line 11, in <module>
    sys.exit(main())
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/certbot/main.py", line 1315, in main
    return config.func(config, plugins)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/certbot/main.py", line 1206, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/certbot/main.py", line 118, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/certbot/client.py", line 351, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/certbot/client.py", line 294, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/certbot/client.py", line 326, in _get_order_and_authorizations
    orderr = self.acme.new_order(csr_pem)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/acme/client.py", line 779, in new_order
    return self.client.new_order(csr_pem)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/acme/client.py", line 605, in new_order
    response = self._post(self.directory['newOrder'], order)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/acme/client.py", line 90, in _post
    return self.net.post(*args, **kwargs)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/acme/client.py", line 1069, in post
    return self._post_once(*args, **kwargs)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/acme/client.py", line 1079, in _post_once
    data = self._wrap_in_jws(obj, self._get_nonce(url), url, acme_version)
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/acme/client.py", line 1058, in _get_nonce
    self._add_nonce(self.head(url))
  File "/home/xxx/certbot_venv/lib/python2.7/site-packages/acme/client.py", line 1053, in _add_nonce
    raise errors.MissingNonce(response)
MissingNonce: Server HEAD response did not include a replay nonce, headers: {'Content-Length': '271', 'Expires': 'Tue, 19 Jun 2018 20:00:41 GMT', 'Server': 'AkamaiGHost', 'Connection': 'keep-alive', 'Pragma': 'no-cache', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Date': 'Tue, 19 Jun 2018 20:00:41 GMT', 'Content-Type': 'text/html', 'Mime-Version': '1.0'} (This may be a service outage)
2018-06-19 20:00:41,993:ERROR:certbot.log:An unexpected error occurred:

Found the offending code, conveniently commented to tell us that it's not checking for an error:

```
    def head(self, *args, **kwargs):
        """Send HEAD request without checking the response.

        Note, that `_check_response` is not called, as it is expected
        that status code other than successfully 2xx will be returned, or
        messages2.Error will be raised by the server.

        """
        return self._send_request('HEAD', *args, **kwargs)
```

I think it makes sense to check that it's an expected error that we get back, not an unexpected error. Is there somewhere that defines what all the possible expected errors might be? I see that boulder gives a 405:

```
Requesting fresh nonce
Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-acct.
https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-acct HTTP/1.1" 405 0
Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 103
Allow: POST
Replay-Nonce: n_QRYFLZP947pWA4CxKpLOPMfEx1s0y8z_vehqjQu2w
Expires: Mon, 22 Oct 2018 23:10:03 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Mon, 22 Oct 2018 23:10:03 GMT
Connection: keep-alive
```
(I think the `2` in `messages2.Error` is a typo?)
Found related https://github.com/letsencrypt/boulder/issues/417 and turns out this issue is a duplicate of https://github.com/certbot/certbot/issues/521 which claimed to have been fixed in https://github.com/certbot/certbot/pull/523 but seems to not have been
JK, #523 was the proposed solution which didn't get used, and instead #586 got merged.
It seems safe to check the response now.
wait a second

this is what `HEAD newNonce` is for isn't it

are we really not using that
@jsha @cpu are we ok with just fixing this for ACMEv2 servers and leaving it as unresolved for ACMEv1 or is it important to fix it for ACMEv1
readers: this is no longer a bug about error reporting, it is a bug also about implementing support for the newNonce directory endpoint
@ohemorange I don't have strong feelings about whether its important for ACMEv1.

Resolving patch:
diff --git a/acme/acme/client.py b/acme/acme/client.py
--- a/acme/acme/client.py
+++ b/acme/acme/client.py
@@ -89,6 +89,8 @@ def _post(self, *args, **kwargs):
 
         """
         kwargs.setdefault('acme_version', self.acme_version)
+        if hasattr(self.directory, 'newNonce'):
+            kwargs.setdefault('new_nonce_url', getattr(self.directory, 'newNonce'))
         return self.net.post(*args, **kwargs)
 
     def update_registration(self, regr, update=None):
@@ -1106,10 +1108,15 @@ def _add_nonce(self, response):
         else:
             raise errors.MissingNonce(response)
 
-    def _get_nonce(self, url):
+    def _get_nonce(self, url, new_nonce_url):
         if not self._nonces:
             logger.debug('Requesting fresh nonce')
-            self._add_nonce(self.head(url))
+            if new_nonce_url is None:
+                response = self.head(url)
+            else:
+                # request a new nonce from the acme newNonce endpoint
+                response = self._check_response(self.head(new_nonce_url), content_type=None)
+            self._add_nonce(response)
         return self._nonces.pop()
 
     def post(self, *args, **kwargs):
@@ -1130,8 +1137,13 @@ def post(self, *args, **kwargs):
 
     def _post_once(self, url, obj, content_type=JOSE_CONTENT_TYPE,
             acme_version=1, **kwargs):
-        data = self._wrap_in_jws(obj, self._get_nonce(url), url, acme_version)
+        try:
+            new_nonce_url = kwargs.pop('new_nonce_url')
+        except KeyError:
+            new_nonce_url = None
+        data = self._wrap_in_jws(obj, self._get_nonce(url, new_nonce_url), url, acme_version)
         kwargs.setdefault('headers', {'Content-Type': content_type})
         response = self._send_request('POST', url, data=data, **kwargs)
+        response = self._check_response(response, content_type=content_type)
         self._add_nonce(response)
-        return self._check_response(response, content_type=content_type)
+        return response


Test patch:
diff --git a/acme/acme/client_test.py b/acme/acme/client_test.py
--- a/acme/acme/client_test.py
+++ b/acme/acme/client_test.py
@@ -805,7 +805,8 @@ def test_finalize_order_timeout(self):
     def test_revoke(self):
         self.client.revoke(messages_test.CERT, self.rsn)
         self.net.post.assert_called_once_with(
-            self.directory["revokeCert"], mock.ANY, acme_version=2)
+            self.directory["revokeCert"], mock.ANY, acme_version=2,
+            new_nonce_url=DIRECTORY_V2['newNonce'])
 
     def test_update_registration(self):
         # "Instance of 'Field' has no to_json/update member" bug:
@@ -1052,7 +1053,10 @@ def setUp(self):
         self.response = mock.MagicMock(ok=True, status_code=http_client.OK)
         self.response.headers = {}
         self.response.links = {}
-        self.checked_response = mock.MagicMock()
+        self.response.checked = False
+        self.acmev1_nonce_response = mock.MagicMock(ok=False,
+            status_code=http_client.METHOD_NOT_ALLOWED)
+        self.acmev1_nonce_response.headers = {}
         self.obj = mock.MagicMock()
         self.wrapped_obj = mock.MagicMock()
         self.content_type = mock.sentinel.content_type
@@ -1064,13 +1068,21 @@ def setUp(self):
 
         def send_request(*args, **kwargs):
             # pylint: disable=unused-argument,missing-docstring
+            self.assertFalse("new_nonce_url" in kwargs)
+            method = args[0]
+            uri = args[1]
+            if method == 'HEAD' and uri != "new_nonce_uri":
+                response = self.acmev1_nonce_response
+            else:
+                response = self.response
+
             if self.available_nonces:
-                self.response.headers = {
+                response.headers = {
                     self.net.REPLAY_NONCE_HEADER:
                     self.available_nonces.pop().decode()}
             else:
-                self.response.headers = {}
-            return self.response
+                response.headers = {}
+            return response
 
         # pylint: disable=protected-access
         self.net._send_request = self.send_request = mock.MagicMock(
@@ -1082,28 +1094,39 @@ def check_response(self, response, content_type):
         # pylint: disable=missing-docstring
         self.assertEqual(self.response, response)
         self.assertEqual(self.content_type, content_type)
-        return self.checked_response
+        self.assertTrue(self.response.ok)
+        self.response.checked = True
+        return self.response
 
     def test_head(self):
-        self.assertEqual(self.response, self.net.head(
+        self.assertEqual(self.acmev1_nonce_response, self.net.head(
             'http://example.com/', 'foo', bar='baz'))
         self.send_request.assert_called_once_with(
             'HEAD', 'http://example.com/', 'foo', bar='baz')
 
+    def test_head_v2(self):
+        self.assertEqual(self.response, self.net.head(
+            'new_nonce_uri', 'foo', bar='baz'))
+        self.send_request.assert_called_once_with(
+            'HEAD', 'new_nonce_uri', 'foo', bar='baz')
+
     def test_get(self):
-        self.assertEqual(self.checked_response, self.net.get(
+        self.assertEqual(self.response, self.net.get(
             'http://example.com/', content_type=self.content_type, bar='baz'))
+        self.assertTrue(self.response.checked)
         self.send_request.assert_called_once_with(
             'GET', 'http://example.com/', bar='baz')
 
     def test_post_no_content_type(self):
         self.content_type = self.net.JOSE_CONTENT_TYPE
-        self.assertEqual(self.checked_response, self.net.post('uri', self.obj))
+        self.assertEqual(self.response, self.net.post('uri', self.obj))
+        self.assertTrue(self.response.checked)
 
     def test_post(self):
         # pylint: disable=protected-access
-        self.assertEqual(self.checked_response, self.net.post(
+        self.assertEqual(self.response, self.net.post(
             'uri', self.obj, content_type=self.content_type))
+        self.assertTrue(self.response.checked)
         self.net._wrap_in_jws.assert_called_once_with(
             self.obj, jose.b64decode(self.all_nonces.pop()), "uri", 1)
 
@@ -1135,7 +1158,7 @@ def test_post_failed_retry(self):
     def test_post_not_retried(self):
         check_response = mock.MagicMock()
         check_response.side_effect = [messages.Error.with_code('malformed'),
-                                      self.checked_response]
+                                      self.response]
 
         # pylint: disable=protected-access
         self.net._check_response = check_response
@@ -1143,13 +1166,12 @@ def test_post_not_retried(self):
                           self.obj, content_type=self.content_type)
 
     def test_post_successful_retry(self):
-        check_response = mock.MagicMock()
-        check_response.side_effect = [messages.Error.with_code('badNonce'),
-                                      self.checked_response]
+        post_once = mock.MagicMock()
+        post_once.side_effect = [messages.Error.with_code('badNonce'),
+                                      self.response]
 
         # pylint: disable=protected-access
-        self.net._check_response = check_response
-        self.assertEqual(self.checked_response, self.net.post(
+        self.assertEqual(self.response, self.net.post(
             'uri', self.obj, content_type=self.content_type))
 
     def test_head_get_post_error_passthrough(self):
@@ -1160,6 +1182,26 @@ def test_head_get_post_error_passthrough(self):
         self.assertRaises(requests.exceptions.RequestException,
                           self.net.post, 'uri', obj=self.obj)
 
+    def test_post_bad_nonce_head(self):
+        # pylint: disable=protected-access
+        # regression test for https://github.com/certbot/certbot/issues/6092
+        bad_response = mock.MagicMock(ok=False, status_code=http_client.SERVICE_UNAVAILABLE)
+        self.net._send_request = mock.MagicMock()
+        self.net._send_request.return_value = bad_response
+        self.content_type = None
+        check_response = mock.MagicMock()
+        self.net._check_response = check_response
+        self.assertRaises(errors.ClientError, self.net.post, 'uri',
+                          self.obj, content_type=self.content_type, acme_version=2,
+                          new_nonce_url='new_nonce_uri')
+        self.assertEqual(check_response.call_count, 1)
+
+    def test_new_nonce_uri_removed(self):
+        self.content_type = None
+        self.net.post('uri', self.obj, content_type=None,
+            acme_version=2, new_nonce_url='new_nonce_uri')
+
+
 class ClientNetworkSourceAddressBindingTest(unittest.TestCase):
     """Tests that if ClientNetwork has a source IP set manually, the underlying library has
     used the provided source address."""