Repository: certbot/certbot

Issue:
Allow removing SAN from multidomain certificate when renewing
When one has a multidomain certificate with many SANs (Subject Alternative Names), there can come a situation where one wants to remove certain SANs from the certificate when renewing (e.g. the removed domain is no longer used - "retired", or it has become relevant in another multidomain certificate instead; in both cases, to keep things clean, I want to remove the no longer needed SAN from the multidomain certificate).

The problem is when renewing with the reduced set of SANs, letsencrypt does not modify the existing certificate "branch", but creates a new one, usually with the `-0001` suffix. That is quite understandable, as it is a potentially destructive action; however in many cases (e.g. the one I am describing above), it is not what the user wants/expects.

Example:

Multidomain certificate created and repeatedly renewed with `--domains domain.tld,s1.domain.tld,s2.domain.tld,s3.domain.tld` exists in `/etc/letsencrypt/live/domain.tld`.

Now I want to remove e.g. `s3.domain.tld`, so I run with `--domains domain.tld,s1.domain.tld,s2.domain.tld`. Since there is no way currently to specify the intention to modify the SAN list in the existing `/etc/letsencrypt/live/domain.tld` "branch" (is it called "lineage"?), I end up with `/etc/letsencrypt/live/domain.tld` (and same in `archive` and `renewal`) untouched, and instead a new "branch" ("lineage"?) created in `/etc/letsencrypt/live/domain.tld-0001` with the requested domain list.

This presents a few problems:
- I have to modify server config to point to the new certificate path (i.e. `/etc/letsencrypt/live/domain.tld` --> `/etc/letsencrypt/live/domain.tld-0001`)
- I really don't like the "-0001" stuff optically 😊
- it leaves `/etc/letsencrypt/{live,archive,renewal}/domain.tld` on disk with no further purpose

A workaround could be to (neither was tested):
- delete `/etc/letsencrypt/{live,archive,renewal}/domain.tld` before doing the renewal that removes the SAN – that is a little bit disconcerting, as there will be a period where no cert will be on disk (possible downtime, if the renewal should fail, I have to restore working certificates from archive)
- do as above (i.e. let it create the "branch" with `-0001` suffix), and then somehow "rename" `domain.tld-0001` to `domain.tld` (again, possible downtime, possible to mess up, waste of time...)
- edit the `/etc/letsencrypt/renewal/domain.tld.conf` file - the `domains` key (and perhaps the `[[webroot_map]]` section) - that does not work, as letsencrypt gets the list of domains directly from the certificate itself (found it in code somewhere)

If it is of interest, there is also a relevant [LE Community thread](https://community.letsencrypt.org/t/how-to-prevent-creation-of-etc-letsencrypt-live-domain-tld-0001-when-removing-domains-from-a-domain-tld-multidomain-certificate/8135).

So I propose that letsencrypt has some means to be told that one wants to use an existing "branch" ("mileage"?) for the new certificate, even if it meant to remove some SANs from the domain.

I think from the UI point of view, for lot of cases what is decisive is the main SAN domain name, which is listed first. That often identifies "what" certificate it is, and letsencrypt already uses it for CN and for naming the "branch" ("lineage"?) on disk. In case of renewal with removing domains, the user usually uses the same CN.

So I think that a lot (if not the majority) of these cases would be covered by some option like `--compress` (physics opposite to "--expand" 😉), which would look whether a certificate with same CN but some SANs removed w.r.t. its current list of SANs is requested, and if yes, would update the same certificate "branch" ("lineage"?), instead of creating a new one with `-0001` suffix, with the new certificate with reduced list of SANs.

Users using `compress` in their cron scripts or even `.ini` files would then just have to be advised that for this to work, they have to preserve the CN (i.e. the first domain in the domain list), and should obviously not use the same CN in more than one certificate (that could go into the documentation of the `--compress` option which the user would have to turn on explicitly, thus having been informed/warned).

Thanks.



Discussion:
You are correct to point out our absence of `--compress`. This certainly should be added so I'm adding this to the milestone for our 0.3.0 release.

You may also be interested in the related #318 which is about what the client does if it fails to obtain authorization for all the names in a certificate.

I agree with the above comments but would like to add one thing.

Compress has a different connotation (tar, gzip, etc.) so how about `--shrink` ?

@aptalca you are right about that (I realized that too after submitting the issue, just did not get to add it as a comment). Of course the UI suggested by me is only to give a first shot suggestion, I am not claiming `--compress` is the best name, or that the command-line UI should not be even a bit different in the way it works. :-)

I think this is a needed feature to avoid generating new folders/branch/milage/lineage when adjusting the list of domains.

Let's say you have `example.com,www.example.com` stored in `example.com/`
and `example.com,www.example.com,beta.example.com` stored in `example.com-0001/`

Now when you ask to generate a cert for `example.com` using the `--shrink` flag, which one would be updated?

While this doesn't solve the problem, if `--shrink` applies similar logic to `--expand`, `example.com/` would be updated. This is because removing `www.example.com` from the cert in `example.com/` would be the smallest possible change to the existing certs.

With that said, this may not be the behavior the user wants and what if there is a tie between two certs? We shouldn't just pick one arbitrarily and instead let the user decide which certificate is updated as suggested by @hultqvist in #2128.

The biggest problem around this is that we don't have a way to specify particular lineages from the command line (nor is it clear what the best way to specify them would be). The code changes required would be super-tiny if we had that mechanism available. `-d` flags don't uniquely specify a lineage because they could match a potentially unlimited number of them!

It would make sense to be able to name lineages after something other than the CN. For example, for one of my certs that handles a bunch of "misc" subdomains, any single one of them isn't really an appropriate identifier for the cert. I ended up naming it "misc.&lt;domain&gt;" which is now its CN and a redundant/unused name for it (it resolves since I have a DNS wildcard, but is never used), but it would've been easier if I could just name the lineage that without actually making it a part of the cert.

In my case, I have a handful of mail.\* subdomains for various workrelated webmails in a single cert (as they go to the same actaul server, using the same webmail installation) and one of my clients used to have both their domain.net and domain.se but let their .net expire. Now my renewal fails and what to me seems like a simple solution (removing the domain and webroot information from the .conf file) is not working.

I'm not in a great hurry, I have about a month to get this fixed and assume that worst case scenario is to merely remove the certs and start over by requesting new certs.

It would've been nice to have the option of just dropping some domains from the .conf file and tell it that I have intentionally chosen to no longer have those domains in the given certificate and it would solve it from there.

Just wanted to add the real scenario showing why solving this issue is important for us: on one server, we are currently using one "megacertificate" (from a different CA) for around 70 domain names. So first step in gradual migration to Let's Encrypt would be to issue the same certificate with the same 70 domain names, but this time from Let's Encrypt.

However if I wanted, as a second phase of migration to LE, to later "spin off" a certain group of domain names from this megacertificate to a certificate of its own, I would run into this issue. That is the main reason why I have not migrated yet to Let's Encrypt on that server: if I issued the megacertificate from Let's Encrypt, I could not later do this "spin off" of subsets of domain names; and I don't currently have the time to create right away all the "perfect sets" of domain names that should go into one certificate (I want to do it gradually).

The reason why we did not create those subsets already with our current CA is exactly because renewing certificates was a very demanding manual process for us, so we wanted to minimize the trouble of this process and have only one certificate to renew (and then are happy to be set for another year).

Also the reason to divide the domain name set into subsets with separate certificates is that often, these domain names have very little in common, and it simply does not feel proper to put them all together in one big certificate (sometimes even marketing reasons – to not associate two completely different websites – is relevant).

So, I am postponing migration to Let's Encrypt on that server until this issue is solved (or until I have a very clear idea about the perfect subsets of domain names, which I currently don't, and don't have time to create them at once, and even if I did, I would still like to be free to change my mind about them later, or correct a wrong "spin off").

One server where we already have Let's Encrypt deployed (and are loving it 😄 ) is one with just 3 certificates each with ~2-3 domain names, so there this issue was not a big deal.

Hi there,

I get that this issue is not solved yet - but is there a workaround that can at least fix it for now?

deleting the 0001 files and their non suffixed brethren from `/etc/letsencrypt/{archive,live,renewal}` seems to have done the trick as a workaround.

But AFAIK the paths are still available in config files and broken then. Don't have any 0001 config file around now so I can't provide an example. I wrote a [renewal script](https://github.com/nerdoc/uberspace-tools/blob/master/letsencrypt-renew) for using Let'sEncrypt on Uberspace.de, and dealt with that just by re-adding the new certificate after finding the new path that was just created. It's a really ugly hack.

What @dwt wrote works. Deleting `/etc/letsencrypt/archive/example.com`, `/etc/letsencrypt/live/example.com` and `/etc/letsencrypy/renewal/example.com.conf` will completely remove a lineage. Make sure no webserver is accessing these file paths before doing this!

Since this creates a new lineage, it means that the letsencrypt server thinks you have a certificate that is running out because you are not renewing it. So expect some confusing mails from letsencrypt.

Facing this issue right now.

The whole system of using SANs could be improved - little documentation, and not very intuitive.

No idea why when renewing the certificate, can't just remove SANS?

Why not simply allow the user to specify the name of the renewal config file that is associated with the certificate lineage they want to remove a domain from?  That seems like it would be a simple and unique way to identify the relevant certificate lineage if it can't be automatically determined from the list of domains.

Implemented here:
https://github.com/PaulSD/certbot/commit/84c9f3f8f3968d1185a233b463a05b2a1f2be294

... Or am I missing something here?  (I feel like I must be missing something, because someone else would probably have implemented this same solution long ago if it were really this simple...)

@PaulSD I was reading through this issue and wondering why it wasn't done like this. Your suggestion makes sense to me.

To the workaround of @dwt: While removing all 0001 (etc) occurrencies cleans up the newly produced certificates, this does not help, as these are the new, valid certificates. If you delete them, you have the old one lying around, which are going to expire then. But they are used by the web server.

To be clear: deleting -0001 files doesn't help here, as the next time `letsencrypt certonly` is run, the -0001 files are created again. I want letsencrypt to overwrite the old ones (or give me at least the option to do that in a way).

What I need (for an automatic renewal script) is that either
- the paths of the new certificates stay the same, even when a domain has been deleted, and no -0001 - or whatever - lineage is created, or
- `letsencrypt certonly` spits out the path of the new certificates in a parseable, validated way, so I can get the path using grep/sed/whatever and point the web server to the new certificate.

I am really looking forward to a solution here... ;-)

@PaulSD @cowlicks I also thought of the solution like PaulSD@84c9f3f ... just searched the certbot manual for that option today when I had many SANs to manage in a single web server.  I don't see why this issue has been left unresolved for such a long time.

@yaegashi  I can make a PR with @PaulSD 's commit tomorrow if it is working.

I have a Let's Encrypt certificate with 50 or so subdomains / SANs.  I now want to renew or replace it with a certificate for the top level domain and its www subdomain only.

How can I convert the certificate?  I'm happy for any kind of manual solution because once it's converted I can automate the renewal.

I've read everything above but I'm still unsure how to do it...

What I do in such cases, if everything else fails: backup the cli.ini, remove the whole .config/letsencrypt folder, restart letsencrypt to generate the new config, and overwrite the cli.ini with the backup version, now change the domain settings there.
then run `letsencrypt certonly` again, and it creates a completely new certificate without all the other problems.
Then pass the certificate to the webserver.

It's a bit overkill, but works.

@airblade This should be addressed in 0.10.0 with #3615. Thanks for your use case!


Resolving patch:
diff --git a/certbot/main.py b/certbot/main.py
--- a/certbot/main.py
+++ b/certbot/main.py
@@ -632,7 +632,7 @@ def obtain_cert(config, plugins, lineage=None):
 def renew(config, unused_plugins):
     """Renew previously-obtained certificates."""
     try:
-        renewal.renew_all_lineages(config)
+        renewal.handle_renewal_request(config)
     finally:
         hooks.post_hook(config, final=True)
 
diff --git a/certbot/renewal.py b/certbot/renewal.py
--- a/certbot/renewal.py
+++ b/certbot/renewal.py
@@ -38,6 +38,12 @@ def renewal_conf_files(config):
     """Return /path/to/*.conf in the renewal conf directory"""
     return glob.glob(os.path.join(config.renewal_configs_dir, "*.conf"))
 
+def renewal_file_for_certname(config, certname):
+    """Return /path/to/certname.conf in the renewal conf directory"""
+    path = os.path.join(config.renewal_configs_dir, "{0}.conf".format(certname))
+    if not os.path.exists(path):
+        raise errors.CertStorageError("No certificate found with name {0}.".format(certname))
+    return path
 
 def _reconstitute(config, full_path):
     """Try to instantiate a RenewableCert, updating config with relevant items.
@@ -296,26 +302,33 @@ def _renew_describe_results(config, renew_successes, renew_failures,
     print("\n".join(out))
 
 
-def renew_all_lineages(config):
+def handle_renewal_request(config):
     """Examine each lineage; renew if due and report results"""
 
     # This is trivially False if config.domains is empty
-    if any(domain not in config.webroot_map for domain in config.domains) or config.certname:
+    if any(domain not in config.webroot_map for domain in config.domains):
         # If more plugins start using cli.add_domains,
         # we may want to only log a warning here
-        raise errors.Error("Currently, the renew verb is only capable of "
+        raise errors.Error("Currently, the renew verb is capable of either "
                            "renewing all installed certificates that are due "
-                           "to be renewed; individual domains or lineages cannot be "
-                           "specified with this action. If you would like to "
-                           "renew specific certificates, use the certonly "
+                           "to be renewed or renewing a single certificate specified "
+                           "by its name. If you would like to renew specific "
+                           "certificates by their domains, use the certonly "
                            "command. The renew verb may provide other options "
                            "for selecting certificates to renew in the future.")
+
     renewer_config = configuration.RenewerConfiguration(config)
+
+    if config.certname:
+        conf_files = [renewal_file_for_certname(renewer_config, config.certname)]
+    else:
+        conf_files = renewal_conf_files(renewer_config)
+
     renew_successes = []
     renew_failures = []
     renew_skipped = []
     parse_failures = []
-    for renewal_file in renewal_conf_files(renewer_config):
+    for renewal_file in conf_files:
         disp = zope.component.getUtility(interfaces.IDisplay)
         disp.notification("Processing " + renewal_file, pause=False)
         lineage_config = copy.deepcopy(config)


Test patch:
diff --git a/certbot/tests/main_test.py b/certbot/tests/main_test.py
--- a/certbot/tests/main_test.py
+++ b/certbot/tests/main_test.py
@@ -872,6 +872,16 @@ def test_renew_verb_empty_config(self):
         args = ["renew", "--dry-run", "-tvv"]
         self._test_renewal_common(False, [], args=args, should_renew=False, error_expected=True)
 
+    def test_renew_with_certname(self):
+        test_util.make_lineage(self, 'sample-renewal.conf')
+        self._test_renewal_common(True, [], should_renew=True,
+            args=['renew', '--dry-run', '--cert-name', 'sample-renewal'])
+
+    def test_renew_with_bad_certname(self):
+        self._test_renewal_common(True, [], should_renew=False,
+            args=['renew', '--dry-run', '--cert-name', 'sample-renewal'],
+            error_expected=True)
+
     def _make_dummy_renewal_config(self):
         renewer_configs_dir = os.path.join(self.config_dir, 'renewal')
         os.makedirs(renewer_configs_dir)