Repository: certbot/certbot

Issue:
"certbot renew" error "Please choose an account" if you have multiple accounts
This bug has been independently reported on the forum twice already:

https://community.letsencrypt.org/t/certbot-renew-error-please-choose-an-account/206600
https://community.letsencrypt.org/t/why-is-another-account-added/205179/5

I suspect this broke in 2.7.0 -- which had some changes to configuration handling -- but I'm not certain.

In any case, it appears "`certbot renew`" is now unable to renew certificates if you have multiple accounts with the CA (e.g. because the installation dates to the `acme-v01.api.letsencrypt.org` transitions). It ought to read the account ID out of the renewal configuration, but apparently it is no longer doing so.

---

If you're having trouble using Certbot and aren't sure you've found a bug or
request for a new feature, please first try asking for help at
https://community.letsencrypt.org/. There is a much larger community there of
people familiar with the project who will be able to more quickly answer your
questions.

## My operating system is (include version):
Ubuntu 20.04

## I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):
snap

## I ran this command and it produced this output:
```
$ sudo certbot certonly --manual --manual-auth-hook "/opt/certbot_dns/certbot_dns.py --auth --config /opt/certbot_dns/certbot_dns.ini --route-53" --manual-cleanup-hook "/opt/certbot_dns/certbot_dns.py --cleanup --config /opt/certbot_dns/certbot_dns.ini --route-53" --preferred-challenges dns-01 --staging -d letsencrypt.mattnordhoff.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Please choose an account
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: flash.mattnordhoff.net@2023-10-18T06:12:39Z (4c49)
2: flash.mattnordhoff.net@2021-04-05T21:00:15Z (6ad0)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Requesting a certificate for letsencrypt.mattnordhoff.net
Hook '--manual-auth-hook' for letsencrypt.mattnordhoff.net ran with output:
 [redacted]

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/letsencrypt.mattnordhoff.net/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/letsencrypt.mattnordhoff.net/privkey.pem
This certificate expires on 2024-01-16.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.
$ sudo certbot renew --cert-name letsencrypt.mattnordhoff.net --force-renewal
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/letsencrypt.mattnordhoff.net.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate letsencrypt.mattnordhoff.net with error: Missing command line flag or config entry for this setting:
Please choose an account
Choices: ['flash.mattnordhoff.net@2023-10-18T06:12:39Z (4c49)', 'flash.mattnordhoff.net@2021-04-05T21:00:15Z (6ad0)']

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/letsencrypt.mattnordhoff.net/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
```

(For this example reproduction, I fiddled with `/etc/letsencrypt/accounts/` to have 2 staging accounts.)

## Certbot's behavior differed from what I expected because:
Certbot was unable to determine the account and failed to renew.

## Here is a Certbot log showing the issue (if available):
###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.
```
2023-10-18 06:15:56,270:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2023-10-18 06:15:56,565:DEBUG:certbot._internal.main:certbot version: 2.7.1
2023-10-18 06:15:56,565:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/3390/bin/certbot
2023-10-18 06:15:56,566:DEBUG:certbot._internal.main:Arguments: ['--cert-name', 'letsencrypt.mattnordhoff.net', '--force-renewal', '--preconfigured-renewal']
2023-10-18 06:15:56,566:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-10-18 06:15:56,632:DEBUG:certbot._internal.log:Root logging level set at 30
2023-10-18 06:15:56,634:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/letsencrypt.mattnordhoff.net.conf
2023-10-18 06:15:56,638:DEBUG:certbot.configuration:Var server=https://acme-staging-v02.api.letsencrypt.org/directory (set by user).
2023-10-18 06:15:56,638:DEBUG:certbot.configuration:Var account={'server'} (set by user).
2023-10-18 06:15:56,639:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2023-10-18 06:15:56,660:DEBUG:certbot._internal.renewal:Auto-renewal forced with --force-renewal...
2023-10-18 06:15:56,660:DEBUG:certbot._internal.plugins.selection:Requested authenticator manual and installer None
2023-10-18 06:15:56,661:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * manual
Description: Manual configuration or run your own shell scripts
Interfaces: Authenticator, Plugin
Entry point: EntryPoint(name='manual', value='certbot._internal.plugins.manual:Authenticator', group='certbot.plugins')
Initialized: <certbot._internal.plugins.manual.Authenticator object at 0x7f325d5df700>
Prep: True
2023-10-18 06:15:56,662:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.manual.Authenticator object at 0x7f325d5df700> and installer None
2023-10-18 06:15:56,662:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator manual, Installer None
2023-10-18 06:15:56,796:ERROR:certbot._internal.renewal:Failed to renew certificate letsencrypt.mattnordhoff.net with error: Missing command line flag or config entry for this setting:
Please choose an account
Choices: ['flash.mattnordhoff.net@2023-10-18T06:12:39Z (4c49)', 'flash.mattnordhoff.net@2021-04-05T21:00:15Z (6ad0)']
2023-10-18 06:15:56,807:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 537, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/_internal/main.py", line 1548, in renew_cert
    le_client = _init_le_client(config, auth, installer)
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/_internal/main.py", line 833, in _init_le_client
    acc, acme = _determine_account(config)
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/_internal/main.py", line 731, in _determine_account
    potential_acc = display_ops.choose_account(accounts)
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/display/ops.py", line 86, in choose_account
    code, index = display_util.menu("Please choose an account", labels, force_interactive=True)
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/display/util.py", line 80, in menu
    return obj.get_display().menu(message, choices, default=default, cli_flag=cli_flag,
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/_internal/display/obj.py", line 470, in menu
    raise self._interaction_fail(message, cli_flag, "Choices: " + repr(choices))
certbot.errors.MissingCommandlineFlag: Missing command line flag or config entry for this setting:
Please choose an account
Choices: ['flash.mattnordhoff.net@2023-10-18T06:12:39Z (4c49)', 'flash.mattnordhoff.net@2021-04-05T21:00:15Z (6ad0)']

2023-10-18 06:15:56,809:DEBUG:certbot._internal.display.obj:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-10-18 06:15:56,810:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2023-10-18 06:15:56,811:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/letsencrypt.mattnordhoff.net/fullchain.pem (failure)
2023-10-18 06:15:56,812:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-10-18 06:15:56,813:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/3390/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/_internal/main.py", line 1873, in main
    return config.func(config, plugins)
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/_internal/main.py", line 1642, in renew
    renewed_domains, failed_domains = renewal.handle_renewal_request(config)
  File "/snap/certbot/3390/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 565, in handle_renewal_request
    raise errors.Error(
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2023-10-18 06:15:56,814:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)
```

## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:
N/A


Discussion:
I can reproduce this after manually manipulating directories to create two accounts. I can also reproduce this with Certbot 2.6.0 which came out in May so I don't think this is due to any recent changes on our end here. Not sure what could be causing the new reports...
`certbot renew` succeeded for me in mid-September using 2.6.0.
In your initial post you said you manually fiddled with the accounts directory to get the info to file this issue. I did the same and hit the problem in both new and older versions of Certbot.

For your mid-September renewal, did you have multiple accounts and does it now fail for you with a newer version of Certbot? If so, have any info about that setup and how I could recreate it?
It's just an old server with an `/etc/letsencrypt/` directory dating back to the `acme-v01` era.
> It's just an old server with an `/etc/letsencrypt/` directory dating back to the `acme-v01` era.

Confirmed in my case as well. I deleted old directories (coming from old installations) in /etc/letsencrypt/accounts and now renewal is back working.
Thanks.
Aldo
Ooo thanks for the follow up.

There was something unusual about my previous test setup (setting `server` in `/etc/letsencrypt/cli.ini`) that was causing me to hit this for older versions of Certbot too, but I am now seeing the change in behavior with the newer versions. We'll get this fixed.
@wgreenberg, I think this is a kind of tricky one and I wanted to see if you had any history/insight that might help here.

To reproduce this from a clean setup:

1. Obtain a cert.
2. Temporarily move the `/etc/letsencrypt/accounts/<server URL>/directory/<account hash>` directory somewhere else. (I just moved it up one level where Certbot ignores it.)
3. Obtain a second certificate as part of a different certificate lineage.
4. Move the directory from (2) back to its original location.
5. Run `certbot renew --force-renewal` (making sure you don't have a value of `server` set anywhere like `/etc/letsencrypt/cli.ini` which is what initially prevented me from reproducing this).

Pre-2.7.0, renewal works. Post-2.7.0, renewal fails with the errors shown in the above comments.

What I believe is happening here is both `server` and `account` are saved in the renewal configuration files, however, because of the ordering in [this list](https://github.com/certbot/certbot/blob/436b7fbe28c78608f816f6b18d33dbe41914e103/certbot/certbot/_internal/renewal.py#L45-L48) and the code [here](https://github.com/certbot/certbot/blob/436b7fbe28c78608f816f6b18d33dbe41914e103/certbot/certbot/_internal/renewal.py#L197-L205) that uses it, `server` is restored first.

What then happens is the argument source for `server` is set to `RUNTIME` and since [`server` changes the meaning of `account`](https://github.com/certbot/certbot/blob/436b7fbe28c78608f816f6b18d33dbe41914e103/certbot/certbot/_internal/cli/cli_constants.py#L79-L85), `set_by_user('account')` returns `True` when `set_by_cli('account')` would have returned `False`. This difference causes `account` not to be set on the configuration object before trying to renew the cert and therefore this error.

Any thoughts on how to fix this? My first thoughts having not carefully read through the initial PR that made the changes here would be either:

1. Make a new `ArgumentSource` type for renewal configuration files, ensure `renewal.py` causes any restored arguments to have this argument source, and have `set_by_user` return `False` in this case.
2. Change the behavior of (and possibly rename) `set_by_user` so it also returns `False` in the `RUNTIME` case like `set_by_cli` used to.

(1) seems better but (2) seems easier. Penny for your thoughts?
ack, thanks for getting an easily reproducible test case @bmw! i was able to (i think) capture the failure in this test case, which confirms your theory:

```
diff --git a/certbot/certbot/_internal/tests/renewal_test.py b/certbot/certbot/_internal/tests/renewal_test.py
index edc9eea35..d29457d76 100644
--- a/certbot/certbot/_internal/tests/renewal_test.py
+++ b/certbot/certbot/_internal/tests/renewal_test.py
@@ -253,6 +253,14 @@ class RestoreRequiredConfigElementsTest(test_util.ConfigTestCase):
         self._call(self.config, {'server': constants.V1_URI})
         assert self.config.server == constants.CLI_DEFAULTS['server']
 
+    def test_9805(self):
+        renewalparams = {
+            'server': 'http://foo.com',
+            'account': 'foo@bar.org',
+        }
+        self._call(self.config, renewalparams)
+        self.assertEqual(self.config.account, 'foo@bar.org')   # fails, self.config.account is None
+
```

i think your solutions are reasonable, though for 1 does it semantically make sense to consider a renewal's parameters to be not set by the user? another question around this is, how does `server` actually cause changes to `account`? i don't see any overriding behavior in helpful.py.

another solution could be this:
```
diff --git a/certbot/certbot/_internal/renewal.py b/certbot/certbot/_internal/renewal.py
index a33dd7ee5..bf5901d68 100644
--- a/certbot/certbot/_internal/renewal.py
+++ b/certbot/certbot/_internal/renewal.py
@@ -194,13 +194,14 @@ def restore_required_config_elements(config: configuration.NamespaceConfig,
 
     """
 
+    config_copy = copy.deepcopy(config)
     required_items = itertools.chain(
         (("pref_challs", _restore_pref_challs),),
         zip(BOOL_CONFIG_ITEMS, itertools.repeat(_restore_bool)),
         zip(INT_CONFIG_ITEMS, itertools.repeat(_restore_int)),
         zip(STR_CONFIG_ITEMS, itertools.repeat(_restore_str)))
     for item_name, restore_func in required_items:
-        if item_name in renewalparams and not config.set_by_user(item_name):
+        if item_name in renewalparams and not config_copy.set_by_user(item_name):
             value = restore_func(item_name, renewalparams[item_name])
             setattr(config, item_name, value)
```
since what we're actually trying to avoid here is clobbering params set by the user in the original config, not in the work-in-progress restored config.

Resolving patch:
diff --git a/certbot/certbot/_internal/renewal.py b/certbot/certbot/_internal/renewal.py
--- a/certbot/certbot/_internal/renewal.py
+++ b/certbot/certbot/_internal/renewal.py
@@ -194,6 +194,7 @@ def restore_required_config_elements(config: configuration.NamespaceConfig,
 
     """
 
+    updated_values = {}
     required_items = itertools.chain(
         (("pref_challs", _restore_pref_challs),),
         zip(BOOL_CONFIG_ITEMS, itertools.repeat(_restore_bool)),
@@ -202,7 +203,9 @@ def restore_required_config_elements(config: configuration.NamespaceConfig,
     for item_name, restore_func in required_items:
         if item_name in renewalparams and not config.set_by_user(item_name):
             value = restore_func(item_name, renewalparams[item_name])
-            setattr(config, item_name, value)
+            updated_values[item_name] = value
+    for key, value in updated_values.items():
+        setattr(config, key, value)
 
 
 def _remove_deprecated_config_elements(renewalparams: Mapping[str, Any]) -> Dict[str, Any]:


Test patch:
diff --git a/certbot/certbot/_internal/tests/renewal_test.py b/certbot/certbot/_internal/tests/renewal_test.py
--- a/certbot/certbot/_internal/tests/renewal_test.py
+++ b/certbot/certbot/_internal/tests/renewal_test.py
@@ -253,6 +253,18 @@ def test_ancient_server_renewal_conf(self, mock_set_by_user):
         self._call(self.config, {'server': constants.V1_URI})
         assert self.config.server == constants.CLI_DEFAULTS['server']
 
+    def test_related_values(self):
+        # certbot.configuration.NamespaceConfig.set_by_user considers some values as related to each
+        # other and considers both set by the user if either is. This test ensures all renewal
+        # parameters are restored regardless of their restoration order or relation between values.
+        # See https://github.com/certbot/certbot/issues/9805 for more info.
+        renewalparams = {
+            'server': 'https://example.org',
+            'account': 'somehash',
+        }
+        self._call(self.config, renewalparams)
+        self.assertEqual(self.config.account, renewalparams['account'])
+
 
 class DescribeResultsTest(unittest.TestCase):
     """Tests for certbot._internal.renewal._renew_describe_results."""