Repository: certbot/certbot

Issue:
Certbot performs challenge even when reusing valid authz
## My operating system is (include version):

Ubuntu 16.04 (x86-64)

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):

certbot-auto

## I ran this command and it produced this output:

    $ sudo -H certbot certonly --staging --webroot -w /srv/www/certbot -d 2017-12-20.clover.letsencrypt.mattnordhoff.net
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator webroot, Installer None
    Obtaining a new certificate
    Performing the following challenges:
    http-01 challenge for 2017-12-20.clover.letsencrypt.mattnordhoff.net
    Using the webroot path /srv/www/certbot for all unmatched domains.
    Waiting for verification...
    Cleaning up challenges

    IMPORTANT NOTES:
     - Congratulations! Your certificate and chain have been saved at:
       /etc/letsencrypt/live/2017-12-20.clover.letsencrypt.mattnordhoff.net/fullchain.pem
       Your key file has been saved at:
       /etc/letsencrypt/live/2017-12-20.clover.letsencrypt.mattnordhoff.net/privkey.pem
       Your cert will expire on 2018-03-20. To obtain a new or tweaked
       version of this certificate in the future, simply run certbot
       again. To non-interactively renew *all* of your certificates, run
       "certbot renew"

    $ sudo -H certbot delete --cert-name 2017-12-20.clover.letsencrypt.mattnordhoff.net
    Saving debug log to /var/log/letsencrypt/letsencrypt.log

    -------------------------------------------------------------------------------
    Deleted all files relating to certificate
    2017-12-20.clover.letsencrypt.mattnordhoff.net.
    -------------------------------------------------------------------------------

    $ sudo -H certbot certonly --staging --webroot -w /srv/www/certbot -d 2017-12-20.clover.letsencrypt.mattnordhoff.net
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator webroot, Installer None
    Obtaining a new certificate
    Performing the following challenges:
    http-01 challenge for 2017-12-20.clover.letsencrypt.mattnordhoff.net
    Using the webroot path /srv/www/certbot for all unmatched domains.
    Waiting for verification...
    Cleaning up challenges

    IMPORTANT NOTES:
     - Congratulations! Your certificate and chain have been saved at:
       /etc/letsencrypt/live/2017-12-20.clover.letsencrypt.mattnordhoff.net/fullchain.pem
       Your key file has been saved at:
       /etc/letsencrypt/live/2017-12-20.clover.letsencrypt.mattnordhoff.net/privkey.pem
       Your cert will expire on 2018-03-20. To obtain a new or tweaked
       version of this certificate in the future, simply run certbot
       again. To non-interactively renew *all* of your certificates, run
       "certbot renew"

## Certbot's behavior differed from what I expected because:

The second time, even though Let's Encrypt returned a valid authz, Certbot still created the challenge file and `POST`ed to the `/acme/challenge` endpoint despite it being unnecessary.

It's harmless for automated usage, but unfortunate for people using manual mode and suffering through a cumbersome file upload or DNS update process.

## Here is a Certbot log showing the issue (if available):
###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

* First run: https://mn0.us/CgfUq946UzLG9VUFKRMes5t/letsencrypt.log
* Second run: https://mn0.us/Niwqr8QLE_dULTHfVtKccxN/letsencrypt.log

## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:



Discussion:
@bmw thoughts?
I'm actually a little surprised we didn't already have an issue about this. Thanks a lot for opening one.

Much of Certbot's core ACME functionality hasn't changed in a long time (with the exception of ongoing work now for ACMEv2) and instead we've been focusing on building more stuff on top of it. When the relevant code here was written, no one was talking about authz reuse and the new-authz endpoint always returned a new authz with pending challenges.

I agree it's usually harmless, but is an unfortunate burden for people doing things manually. We probably won't get to this in the next couple months and we're focusing on other things (like getting our automated DNS plugins packaged), but we certainly want to fix this at some point. If someone is interested in writing a PR, the relevant code here is in `certbot/auth_handler.py`.

Resolving patch:
diff --git a/certbot/auth_handler.py b/certbot/auth_handler.py
--- a/certbot/auth_handler.py
+++ b/certbot/auth_handler.py
@@ -31,7 +31,7 @@ class AuthHandler(object):
         :class:`~acme.challenges.Challenge` types
     :type auth: :class:`certbot.interfaces.IAuthenticator`
 
-    :ivar acme.client.BackwardsCompatibleClientV2 acme: ACME client API.
+    :ivar acme.client.BackwardsCompatibleClientV2 acme_client: ACME client API.
 
     :ivar account: Client's Account
     :type account: :class:`certbot.account.Account`
@@ -40,9 +40,9 @@ class AuthHandler(object):
         type strings with the most preferred challenge listed first
 
     """
-    def __init__(self, auth, acme, account, pref_challs):
+    def __init__(self, auth, acme_client, account, pref_challs):
         self.auth = auth
-        self.acme = acme
+        self.acme = acme_client
 
         self.account = account
         self.pref_challs = pref_challs
@@ -85,19 +85,26 @@ def handle_authorizations(self, orderr, best_effort=False):
         self.verify_authzr_complete(aauthzrs)
 
         # Only return valid authorizations
-        retVal = [aauthzr.authzr for aauthzr in aauthzrs
-                  if aauthzr.authzr.body.status == messages.STATUS_VALID]
+        ret_val = [aauthzr.authzr for aauthzr in aauthzrs
+                   if aauthzr.authzr.body.status == messages.STATUS_VALID]
 
-        if not retVal:
+        if not ret_val:
             raise errors.AuthorizationError(
                 "Challenges failed for all domains")
 
-        return retVal
+        return ret_val
 
     def _choose_challenges(self, aauthzrs):
-        """Retrieve necessary challenges to satisfy server."""
-        logger.info("Performing the following challenges:")
-        for aauthzr in aauthzrs:
+        """
+        Retrieve necessary and pending challenges to satisfy server.
+        NB: Necessary and already validated challenges are not retrieved,
+        as they can be reused for a certificate issuance.
+        """
+        pending_authzrs = [aauthzr for aauthzr in aauthzrs
+                           if aauthzr.authzr.body.status != messages.STATUS_VALID]
+        if pending_authzrs:
+            logger.info("Performing the following challenges:")
+        for aauthzr in pending_authzrs:
             aauthzr_challenges = aauthzr.authzr.body.challenges
             if self.acme.acme_version == 1:
                 combinations = aauthzr.authzr.body.combinations
@@ -125,7 +132,7 @@ def _has_challenges(self, aauthzrs):
 
     def _solve_challenges(self, aauthzrs):
         """Get Responses for challenges from authenticators."""
-        resp = []  # type: Collection[acme.challenges.ChallengeResponse]
+        resp = []  # type: Collection[challenges.ChallengeResponse]
         all_achalls = self._get_all_achalls(aauthzrs)
         try:
             if all_achalls:
@@ -531,7 +538,7 @@ def _report_failed_challs(failed_achalls):
 
     """
     problems = collections.defaultdict(list)\
-    # type: DefaultDict[str, List[achallenges.KeyAuthorizationAnnotatedChallenge]]
+        # type: DefaultDict[str, List[achallenges.KeyAuthorizationAnnotatedChallenge]]
     for achall in failed_achalls:
         if achall.error:
             problems[achall.error.typ].append(achall)


Test patch:
diff --git a/certbot-nginx/tests/boulder-integration.conf.sh b/certbot-nginx/tests/boulder-integration.conf.sh
--- a/certbot-nginx/tests/boulder-integration.conf.sh
+++ b/certbot-nginx/tests/boulder-integration.conf.sh
@@ -1,3 +1,4 @@
+#!/usr/bin/env bash
 # Based on
 # https://www.exratione.com/2014/03/running-nginx-as-a-non-root-user/
 # https://github.com/exratione/non-root-nginx/blob/9a77f62e5d5cb9c9026fd62eece76b9514011019/nginx.conf
@@ -52,7 +53,7 @@ http {
     listen 5002 $default_server;
     # IPv6.
     listen [::]:5002 $default_server;
-    server_name nginx.wtf nginx2.wtf;
+    server_name nginx.wtf nginx-tls.wtf nginx2.wtf;
 
     root $root/webroot;
 
diff --git a/certbot-nginx/tests/boulder-integration.sh b/certbot-nginx/tests/boulder-integration.sh
--- a/certbot-nginx/tests/boulder-integration.sh
+++ b/certbot-nginx/tests/boulder-integration.sh
@@ -39,8 +39,8 @@ nginx -v
 reload_nginx
 certbot_test_nginx --domains nginx.wtf run
 test_deployment_and_rollback nginx.wtf
-certbot_test_nginx --domains nginx.wtf run --preferred-challenges tls-sni
-test_deployment_and_rollback nginx.wtf
+certbot_test_nginx --domains nginx-tls.wtf run --preferred-challenges tls-sni
+test_deployment_and_rollback nginx-tls.wtf
 certbot_test_nginx --domains nginx2.wtf --preferred-challenges http
 test_deployment_and_rollback nginx2.wtf
 # Overlapping location block and server-block-level return 301
diff --git a/certbot/tests/auth_handler_test.py b/certbot/tests/auth_handler_test.py
--- a/certbot/tests/auth_handler_test.py
+++ b/certbot/tests/auth_handler_test.py
@@ -57,7 +57,7 @@ def test_unrecognized(self):
              errors.Error, self.handler._challenge_factory, authzr, [0])
 
 
-class HandleAuthorizationsTest(unittest.TestCase):
+class HandleAuthorizationsTest(unittest.TestCase):  # pylint: disable=too-many-public-methods
     """handle_authorizations test.
 
     This tests everything except for all functions under _poll_challenges.
@@ -316,6 +316,24 @@ def test_incomplete_authzr_error(self, mock_verify, mock_poll):
         self.assertEqual(
             self.mock_auth.cleanup.call_args[0][0][0].typ, "tls-sni-01")
 
+    def test_validated_challenge_not_rerun(self):
+        # With pending challenge, we expect the challenge to be tried, and fail.
+        authzr = acme_util.gen_authzr(
+                messages.STATUS_PENDING, "0",
+                [acme_util.HTTP01],
+                [messages.STATUS_PENDING], False)
+        mock_order = mock.MagicMock(authorizations=[authzr])
+        self.assertRaises(
+            errors.AuthorizationError, self.handler.handle_authorizations, mock_order)
+
+        # With validated challenge; we expect the challenge not be tried again, and succeed.
+        authzr = acme_util.gen_authzr(
+                messages.STATUS_VALID, "0",
+                [acme_util.HTTP01],
+                [messages.STATUS_VALID], False)
+        mock_order = mock.MagicMock(authorizations=[authzr])
+        self.handler.handle_authorizations(mock_order)
+
     def _validate_all(self, aauthzrs, unused_1, unused_2):
         for i, aauthzr in enumerate(aauthzrs):
             azr = aauthzr.authzr