Repository: certbot/certbot

Issue:
KeyError when using --manual-cleanup-hook with wildcard certificate
## My operating system is (include version):
Ubuntu 16.04

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):
Git - commit `31805c5`

## I ran this command and it produced this output:
```bash
$ certbot --noninteractive --manual --manual-public-ip-logging-ok --preferred-challenges dns --manual-auth-hook "/srv/letsencrypt-lexicon.sh auth" --manual-cleanup-hook "/srv/letsencrypt-lexicon.sh cleanup" --server https://acme-staging-v02.api.letsencrypt.org/directory --cert-name xenial.alberon.co.uk -d xenial.alberon.co.uk,*.xenial.alberon.co.uk -m dave@davejamesmiller.com --agree-tos
...
Traceback (most recent call last):
  File "/srv/certbot/venv/bin/certbot", line 11, in <module>
    load_entry_point('certbot', 'console_scripts', 'certbot')()
  File "/srv/certbot/certbot/main.py", line 1266, in main
    return config.func(config, plugins)
  File "/srv/certbot/certbot/main.py", line 1157, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/srv/certbot/certbot/main.py", line 118, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/srv/certbot/certbot/client.py", line 349, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/srv/certbot/certbot/client.py", line 293, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/srv/certbot/certbot/client.py", line 329, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/srv/certbot/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(resp, best_effort)
  File "/srv/certbot/certbot/auth_handler.py", line 159, in _respond
    self._cleanup_challenges(active_achalls)
  File "/srv/certbot/certbot/auth_handler.py", line 304, in _cleanup_challenges
    self.auth.cleanup(achalls)
  File "/srv/certbot/certbot/plugins/manual.py", line 218, in cleanup
    env = self.env.pop(achall.domain)
KeyError: u'xenial.alberon.co.uk'
```

## Certbot's behavior differed from what I expected because:
It crashed.

This only seems to happen when using `--manual-cleanup-hook` *and* including both `xenial.alberon.co.uk` and `*.xenial.alberon.co.uk`.

I think `self.env.pop(achall.domain)` gets called twice, both times with `achall.domain = 'xenial.alberon.co.uk'`, and the second time it's already been removed. I would guess the key needs to be changed to something else now that one domain may require two separate challenges.

## Here is a Certbot log showing the issue (if available):
https://gist.github.com/davejamesmiller/2a8c2f7a03cac9edd573c767a8fbedec

## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:
N/A


Discussion:
@ohemorange @bmw, this is a bug related to the handling of challenges for wildcard requests in ACMEv2, although it was triggered by running a development version and not a released version, so I wouldn't be surprised if it's already been fixed. I just wanted to point it out to be sure that you were aware of it.
@davejamesmiller , thanks for pointing out the problem you ran into. Development in this area is ongoing, so if you have time, could you periodically try again with more recent commits?
This is probably still an issue in `master`. Thanks for flagging.

EDIT: Although we currently test this case in our integration tests so I'm not sure what's going on. We should definitely look into the issue before the release though.

Resolving patch:
diff --git a/certbot/plugins/manual.py b/certbot/plugins/manual.py
--- a/certbot/plugins/manual.py
+++ b/certbot/plugins/manual.py
@@ -189,7 +189,7 @@ def _perform_achall_with_script(self, achall):
         os.environ.update(env)
         _, out = hooks.execute(self.conf('auth-hook'))
         env['CERTBOT_AUTH_OUTPUT'] = out.strip()
-        self.env[achall.domain] = env
+        self.env[achall] = env
 
     def _perform_achall_manually(self, achall):
         validation = achall.validation(achall.account_key)
@@ -215,7 +215,7 @@ def _perform_achall_manually(self, achall):
     def cleanup(self, achalls):  # pylint: disable=missing-docstring
         if self.conf('cleanup-hook'):
             for achall in achalls:
-                env = self.env.pop(achall.domain)
+                env = self.env.pop(achall)
                 if 'CERTBOT_TOKEN' not in env:
                     os.environ.pop('CERTBOT_TOKEN', None)
                 os.environ.update(env)


Test patch:
diff --git a/certbot/plugins/manual_test.py b/certbot/plugins/manual_test.py
--- a/certbot/plugins/manual_test.py
+++ b/certbot/plugins/manual_test.py
@@ -93,10 +93,10 @@ def test_script_perform(self):
             self.auth.perform(self.achalls),
             [achall.response(achall.account_key) for achall in self.achalls])
         self.assertEqual(
-            self.auth.env[self.dns_achall.domain]['CERTBOT_AUTH_OUTPUT'],
+            self.auth.env[self.dns_achall]['CERTBOT_AUTH_OUTPUT'],
             dns_expected)
         self.assertEqual(
-            self.auth.env[self.http_achall.domain]['CERTBOT_AUTH_OUTPUT'],
+            self.auth.env[self.http_achall]['CERTBOT_AUTH_OUTPUT'],
             http_expected)
         # tls_sni_01 challenge must be perform()ed above before we can
         # get the cert_path and key_path.
@@ -107,7 +107,7 @@ def test_script_perform(self):
             self.auth.tls_sni_01.get_z_domain(self.tls_sni_achall),
             'novalidation')
         self.assertEqual(
-            self.auth.env[self.tls_sni_achall.domain]['CERTBOT_AUTH_OUTPUT'],
+            self.auth.env[self.tls_sni_achall]['CERTBOT_AUTH_OUTPUT'],
             tls_sni_expected)
 
     @test_util.patch_get_utility()
diff --git a/tests/boulder-integration.sh b/tests/boulder-integration.sh
--- a/tests/boulder-integration.sh
+++ b/tests/boulder-integration.sh
@@ -233,6 +233,7 @@ certname="dns.le.wtf"
 common -a manual -d dns.le.wtf --preferred-challenges dns,tls-sni run \
     --cert-name $certname \
     --manual-auth-hook ./tests/manual-dns-auth.sh \
+    --manual-cleanup-hook ./tests/manual-dns-cleanup.sh \
     --pre-hook 'echo wtf2.pre >> "$HOOK_TEST"' \
     --post-hook 'echo wtf2.post >> "$HOOK_TEST"' \
     --renew-hook 'echo deploy >> "$HOOK_TEST"'
@@ -433,7 +434,8 @@ done
 # Test ACMEv2-only features
 if [ "${BOULDER_INTEGRATION:-v1}" = "v2" ]; then
     common -a manual -d '*.le4.wtf,le4.wtf' --preferred-challenges dns \
-        --manual-auth-hook ./tests/manual-dns-auth.sh
+        --manual-auth-hook ./tests/manual-dns-auth.sh \
+        --manual-cleanup-hook ./tests/manual-dns-cleanup.sh
 fi
 
 # Most CI systems set this variable to true.
diff --git a/tests/manual-dns-cleanup.sh b/tests/manual-dns-cleanup.sh
new file mode 100755
--- /dev/null
+++ b/tests/manual-dns-cleanup.sh
@@ -0,0 +1,3 @@
+#!/bin/sh
+curl -X POST 'http://localhost:8055/clear-txt' -d \
+    "{\"host\": \"_acme-challenge.$CERTBOT_DOMAIN.\"}"