Repository: certbot/certbot

Issue:
Reuse ACMEv1 accounts for ACMEv2
Boulder implemented this and making use of it is a better experience for our users and their servers.


Discussion:
The current assumption is accounts are not shared between servers at different URLs which seems like a safe and usually correct assumption, however, boulder implemented the ability to share accounts between ACME versions. (This is the case in production and probably the case in staging too but we should double check the latter.) I think we should continue to make the assumption accounts aren't shared in general and add a special case for Let's Encrypt. Normally I'm against hard coding Let's Encrypt's policies like this and having an ancient version of Certbot come back to bite us later, but this isn't likely to change and allowing the account to be shared is a significant benefit for us, Let's Encrypt, and our users.

The relevant class is `certbot.account.AccountFileStorage`, specifically its usage of `config.accounts_dir`, and especially the `find_all` method. The way I'd do this is to make a data structure containing the server URLs across which accounts can be shared in `accounts.py`, `constants.py`, or `configuration.py`. When `find_all` is called, it can use these to check the account directories for the other endpoints to see what other accounts exist.

I think the big question here is whether we always check these other directories or only if an account doesn't exist in the account directory specifically for that endpoint. The reason for considering the latter is to avoid adding new prompts to Certbot.

If you look at `certbot.main._determine_account.py`, we will prompt if there is more than one matching account. Since people have been changing the server URL to use ACMEv2 and we switched ACMEv2 to the default staging URL, I think this will cause problems. For instance, I think it's fairly common that a user has an ACMEv1 prod account and a staging account for both endpoints. If we make the change to have `find_all` always return both accounts, if you run `certbot renew --dry-run` I'd expect Certbot to crash saying it needs you to choose an account but can't do so because `renew` implies non-interactive :/ Similar problems would happen for people who have already started using ACMEv2 in prod, have a ACMEv1 account, and are using the `--non-interactive` flag.

The other piece of this that I haven't fully thought through is if we use an account from a directory other than the one matching the server URL being used, how do we find that account during renewal? Unfortunately, I think we have to break forwards compatibility here as I don't see a way to make this work with an older version of Certbot. If we're OK with that, the two options I thought of are to add the account directory used to the runtime config and save it in the renewal conf file or to reuse the logic from `find_all` to search other account directories in `certbot.account.AccountFileStorage.load`. The latter seems much easier and less error prone to me.

If you're still with me, let me know if you have any questions about all this. Happy to answer them, talk design, or try to take this issue entirely if prefer.
To keep forward compatibility, we may be able to create symlinks. For example, when reusing an ACMEv1 account for ACMEv2, create the ACMEv2 account directory if it doesn't exist, and create a symlink from the directory for the ACMEv1 account you're using to the ACMEv2 directory.

No idea if this actually works but maybe something to look into. To create this symlink, we'd either have to always make these symlinks for any accounts returned from `find_all` that are from a different directory or we'd have to have other code that takes the account the user picked and creates the symlink.
I have a different idea:

we create an ordered list of related server urls (per special-cased server, such as prod and staging) that aren't interchangeable but instead are preferential

and you look up the correct one in the list and then try ones chronologically earlier than it

so if there's an acmev2 account hanging around, use that if you're an acmev2 url
and if you're an acmev2 url but there's no acmev2 account, just use the acmev1 account

but it's one way -- if you're acmev1, you can't try looking for an acmev2 account

and if acmev3 comes out, try looking for acmev2 then acmev1 

this solves the interactivity problem

for the forward compatibility problem, same symlink fix? if you have to make an acmev2 account and there's no acmev1 account hanging around, also make a symlink from all the earlier-downs to the acmev2 account.

edit: it might actually be easier just to immediately make symlinks up and down

actually this gets cleaner if we don't bother planning for the future, and just assume there won't be acmev3 and we just ever have to do this once and if that changes we can fix it then

I also think it's fine that forward compatibility requires passing `--server`
I think it's rare that someone will actually want to see both their old and new accounts. it's rare that people have multiple accounts in the first place, and the more common case is that someone just made a new account for the new server, and don't actually care which account they use. and if they have to use an old account, they can specify `--server`

Resolving patch:
diff --git a/certbot/account.py b/certbot/account.py
--- a/certbot/account.py
+++ b/certbot/account.py
@@ -16,6 +16,7 @@
 from acme import fields as acme_fields
 from acme import messages
 
+from certbot import constants
 from certbot import errors
 from certbot import interfaces
 from certbot import util
@@ -142,7 +143,11 @@ def __init__(self, config):
                                    self.config.strict_permissions)
 
     def _account_dir_path(self, account_id):
-        return os.path.join(self.config.accounts_dir, account_id)
+        return self._account_dir_path_for_server_path(account_id, self.config.server_path)
+
+    def _account_dir_path_for_server_path(self, account_id, server_path):
+        accounts_dir = self.config.accounts_dir_for_server_path(server_path)
+        return os.path.join(accounts_dir, account_id)
 
     @classmethod
     def _regr_path(cls, account_dir_path):
@@ -156,22 +161,44 @@ def _key_path(cls, account_dir_path):
     def _metadata_path(cls, account_dir_path):
         return os.path.join(account_dir_path, "meta.json")
 
-    def find_all(self):
+    def _find_all_for_server_path(self, server_path):
+        accounts_dir = self.config.accounts_dir_for_server_path(server_path)
         try:
-            candidates = os.listdir(self.config.accounts_dir)
+            candidates = os.listdir(accounts_dir)
         except OSError:
             return []
 
         accounts = []
         for account_id in candidates:
             try:
-                accounts.append(self.load(account_id))
+                accounts.append(self._load_for_server_path(account_id, server_path))
             except errors.AccountStorageError:
                 logger.debug("Account loading problem", exc_info=True)
+
+
+        if not accounts and server_path in constants.LE_REUSE_SERVERS:
+            # find all for the next link down
+            prev_server_path = constants.LE_REUSE_SERVERS[server_path]
+            prev_accounts = self._find_all_for_server_path(prev_server_path)
+            # if we found something, link to that
+            if prev_accounts:
+                if os.path.islink(accounts_dir):
+                    os.unlink(accounts_dir)
+                else:
+                    try:
+                        os.rmdir(accounts_dir)
+                    except OSError:
+                        return []
+                prev_account_dir = self.config.accounts_dir_for_server_path(prev_server_path)
+                os.symlink(prev_account_dir, accounts_dir)
+            accounts = prev_accounts
         return accounts
 
-    def load(self, account_id):
-        account_dir_path = self._account_dir_path(account_id)
+    def find_all(self):
+        return self._find_all_for_server_path(self.config.server_path)
+
+    def _load_for_server_path(self, account_id, server_path):
+        account_dir_path = self._account_dir_path_for_server_path(account_id, server_path)
         if not os.path.isdir(account_dir_path):
             raise errors.AccountNotFound(
                 "Account at %s does not exist" % account_dir_path)
@@ -193,6 +220,9 @@ def load(self, account_id):
                     account_id, acc.id))
         return acc
 
+    def load(self, account_id):
+        return self._load_for_server_path(account_id, self.config.server_path)
+
     def save(self, account, acme):
         self._save(account, acme, regr_only=False)
 
diff --git a/certbot/configuration.py b/certbot/configuration.py
--- a/certbot/configuration.py
+++ b/certbot/configuration.py
@@ -65,8 +65,12 @@ def server_path(self):
 
     @property
     def accounts_dir(self):  # pylint: disable=missing-docstring
+        return self.accounts_dir_for_server_path(self.server_path)
+
+    def accounts_dir_for_server_path(self, server_path):
+        """Path to accounts directory based on server_path"""
         return os.path.join(
-            self.namespace.config_dir, constants.ACCOUNTS_DIR, self.server_path)
+            self.namespace.config_dir, constants.ACCOUNTS_DIR, server_path)
 
     @property
     def backup_dir(self):  # pylint: disable=missing-docstring
diff --git a/certbot/constants.py b/certbot/constants.py
--- a/certbot/constants.py
+++ b/certbot/constants.py
@@ -156,6 +156,12 @@
 ACCOUNTS_DIR = "accounts"
 """Directory where all accounts are saved."""
 
+LE_REUSE_SERVERS = {
+    'acme-staging-v02.api.letsencrypt.org/directory':
+        'acme-staging.api.letsencrypt.org/directory'
+}
+"""Servers that can reuse accounts from other servers."""
+
 BACKUP_DIR = "backups"
 """Directory (relative to `IConfig.work_dir`) where backups are kept."""
 


Test patch:
diff --git a/certbot/tests/account_test.py b/certbot/tests/account_test.py
--- a/certbot/tests/account_test.py
+++ b/certbot/tests/account_test.py
@@ -95,6 +95,7 @@ def test_it(self):
 
 class AccountFileStorageTest(test_util.ConfigTestCase):
     """Tests for certbot.account.AccountFileStorage."""
+    #pylint: disable=too-many-public-methods
 
     def setUp(self):
         super(AccountFileStorageTest, self).setUp()
@@ -159,7 +160,8 @@ def test_find_all_accounts_dir_absent(self):
         self.assertEqual([], self.storage.find_all())
 
     def test_find_all_load_skips(self):
-        self.storage.load = mock.MagicMock(
+        # pylint: disable=protected-access
+        self.storage._load_for_server_path = mock.MagicMock(
             side_effect=["x", errors.AccountStorageError, "z"])
         with mock.patch("certbot.account.os.listdir") as mock_listdir:
             mock_listdir.return_value = ["x", "y", "z"]
@@ -175,6 +177,64 @@ def test_load_id_mismatch_raises_error(self):
         self.assertRaises(errors.AccountStorageError, self.storage.load,
                           "x" + self.acc.id)
 
+    def _set_server(self, server):
+        self.config.server = server
+        from certbot.account import AccountFileStorage
+        self.storage = AccountFileStorage(self.config)
+
+    def test_find_all_neither_exists(self):
+        self._set_server('https://acme-staging-v02.api.letsencrypt.org/directory')
+        self.assertEqual([], self.storage.find_all())
+        self.assertEqual([], self.storage.find_all())
+        self.assertFalse(os.path.islink(self.config.accounts_dir))
+
+    def test_find_all_find_before_save(self):
+        self._set_server('https://acme-staging-v02.api.letsencrypt.org/directory')
+        self.assertEqual([], self.storage.find_all())
+        self.storage.save(self.acc, self.mock_client)
+        self.assertEqual([self.acc], self.storage.find_all())
+        self.assertEqual([self.acc], self.storage.find_all())
+        self.assertFalse(os.path.islink(self.config.accounts_dir))
+        # we shouldn't have created a v1 account
+        prev_server_path = 'https://acme-staging.api.letsencrypt.org/directory'
+        self.assertFalse(os.path.isdir(self.config.accounts_dir_for_server_path(prev_server_path)))
+
+    def test_find_all_save_before_find(self):
+        self._set_server('https://acme-staging-v02.api.letsencrypt.org/directory')
+        self.storage.save(self.acc, self.mock_client)
+        self.assertEqual([self.acc], self.storage.find_all())
+        self.assertEqual([self.acc], self.storage.find_all())
+        self.assertFalse(os.path.islink(self.config.accounts_dir))
+        self.assertTrue(os.path.isdir(self.config.accounts_dir))
+        prev_server_path = 'https://acme-staging.api.letsencrypt.org/directory'
+        self.assertFalse(os.path.isdir(self.config.accounts_dir_for_server_path(prev_server_path)))
+
+    def test_find_all_server_downgrade(self):
+        # don't use v2 accounts with a v1 url
+        self._set_server('https://acme-staging-v02.api.letsencrypt.org/directory')
+        self.assertEqual([], self.storage.find_all())
+        self.storage.save(self.acc, self.mock_client)
+        self.assertEqual([self.acc], self.storage.find_all())
+        self._set_server('https://acme-staging.api.letsencrypt.org/directory')
+        self.assertEqual([], self.storage.find_all())
+
+    def test_upgrade_version(self):
+        self._set_server('https://acme-staging.api.letsencrypt.org/directory')
+        self.storage.save(self.acc, self.mock_client)
+        self._set_server('https://acme-staging-v02.api.letsencrypt.org/directory')
+        self.assertEqual([self.acc], self.storage.find_all())
+
+    @mock.patch('os.rmdir')
+    def test_corrupted_account(self, mock_rmdir):
+        # pylint: disable=protected-access
+        self._set_server('https://acme-staging.api.letsencrypt.org/directory')
+        self.storage.save(self.acc, self.mock_client)
+        mock_rmdir.side_effect = OSError
+        self.storage._load_for_server_path = mock.MagicMock(
+            side_effect=errors.AccountStorageError)
+        self._set_server('https://acme-staging-v02.api.letsencrypt.org/directory')
+        self.assertEqual([], self.storage.find_all())
+
     def test_load_ioerror(self):
         self.storage.save(self.acc, self.mock_client)
         mock_open = mock.mock_open()