Repository: certbot/certbot

Issue:
`certbot renew` doesn't work when the /etc/letsencrypt/cli.ini file contains a list of domains.
**My system setup:**

* Fedora 25 Server (64-bit)
* certbot 0.9.3
* openssl 1.0.2j

**What I did:**

Using the `domains` config line, I successfully generated a certificate for all my domains and subdomains at once, but it apparently causes a problem for renewing them. certbot complains that you can't specify individual domains, even though no domains are specified *on the command line*.

```
↪  sudo certbot renew  --pre-hook "systemctl stop nginx.service"  \
    --post-hook "systemctl start nginx.service"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
No renewals attempted, so not running post-hook
Currently, the renew verb is only capable of renewing all installed
certificates that are due to be renewed; individual domains cannot
be specified with this action. If you would like to renew specific
certificates, use the certonly command. The renew verb may provide
other options for selecting certificates to renew in the future.

<exit code 1>
```

When I comment out the `domains` line in `cli.ini`, `certbot renew` behaves as expected.

```
↪  sudo certbot renew  --pre-hook "systemctl stop nginx.service"  \
    --post-hook "systemctl start nginx.service"
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-----------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/<MY_DOMAIN>.conf
-----------------------------------------------------------------------
Cert not yet due for renewal

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/<MY_DOMAIN>/fullchain.pem (skipped)
No renewals were attempted.
No renewals attempted, so not running post-hook
```


Discussion:
This is expected behavior and is a duplicate of #3824 which is now closed. With that said, we could try and improve the error message here to mention a configuration file as suggested in #3824.
I read #3824, which refers to #2305, which refers to #3615. That last thread was a little confusing to me, but these all seem to drift further and further away from my original point. #3824 asks for a more specific error message, but was closed without getting one. #2305 asks for a way to specify which domains to renew, but was closed in favor of #3615, which does not even implement that, as far as I can tell...

My original point was that the `domains` line in `cli.ini` makes it easier to create certificates the first time but breaks renewing them: If the renew command is only ever going to just renew all active certificates, maybe it could simply ignore the `domains` config line? Renewing all active certs is what I want, and it seems silly to have to edit your certbot config file before creating domain certificates and then again afterward. This sort of global configuration file for an application normally still works for all actions the app is able to do.

The workaround in the meanwhile I guess is to use `certbot certonly` instead. That works --- *if* you can use renewal hooks on the certonly command. But from reading the documentation, I'm not sure that you can.
I agree #3824 was prematurely closed. #2305 was fixed in #3880 with the `--cert-name` flag.

> My original point was that the domains line in cli.ini makes it easier to create certificates the first time

Easier than just specifying the domains on the command line with `-d/--domains`? I suppose it depends on your use case.

> If the renew command is only ever going to just renew all active certificates, maybe it could simply ignore the domains config line?

I'm not sure if we can really do this. `-d/--domains` can be used to select which certificate to use and to change the names found in a certificate. People sometimes attempt this with `certbot renew` and it'd be nice to catch this and print an error message up front instead of doing something they don't expect like ignoring the flag entirely.

> The workaround in the meanwhile I guess is to use certbot certonly instead. That works --- if you can use renewal hooks on the certonly command. But from reading the documentation, I'm not sure that you can.

I think removing your domains from your configuration file is an easier workaround, but yes, `certbot certonly` can be used instead. You can also use renewal hooks with `certonly` and thanks for pointing out that this isn't at all clear from our documentation. I created #4103 to track that issue.

If you decide to go the `certonly` route, make sure you test your set up with `--dry-run` and include the following flag on the command line or in your configuration file:
```
  --keep-until-expiring, --keep, --reinstall
                        If the requested cert matches an existing cert, always
                        keep the existing one until it is due for renewal (for
                        the 'run' subcommand this means reinstall the existing
                        cert). (default: Ask)
```

cc @ohemorange, in case you have any thoughts on this one
>> My original point was that the domains line in cli.ini makes it easier to create certificates the first time

> Easier than just specifying the domains on the command line with -d/--domains?

When you're doing more than just a few domain names, yes, I think so. Or maybe it's not *easier to execute* so much as it is *easier to read*. The `domains = ...` line is a flat list of domain names, so each element has only a single semantic context; the command line to create certificates, on the other hand, has potentially many arguments, each meaning very different things. So putting some things in the config file makes the command line shorter, simpler.

Isn't that what the `domains` line in the config file is for? I guess most of what I'm saying here might not matter if I misunderstood the intended use of the `cli.ini` file.


> -d/--domains can be used to select which certificate to use and to change the names found in a certificate. People sometimes attempt this with certbot renew and it'd be nice to catch this and print an error message up front instead of doing something they don't expect like ignoring the flag entirely.

Well, I'd rather meant to ignore the domains line in `cli.ini`.

The issue here seems pretty obvious to me, so either I'm missing something, or I'm not explaining the problem clearly enough. Maybe considering an example from another app will help?

With git, you can designate which branch you want to push to, either using the command line arguments, or by setting the `push.default` variable in the `.gitconfig` file. But pushing branches is irrelevant for viewing the commit logs. If you were to use some hypothetical command line argument to designate push-to-this-branch, I would expect `git log` to report some sort of error. But I also expect `git log` to ignore configurations like `push.default` or `http.postBuffer` because they have nothing to do with displaying the commit history.

I believe a config file line specifying domains for which to *create* certificates is similarly irrelevant to a command that is meant only to *refresh* all existing certs.
I agree with you that putting domains in a configuration file makes the command easier to read and putting domains there is certainly a valid use of the file.

I think the difference between `certbot renew` and your `git` example is `certbot renew` wasn't written with the idea that it would always refresh all existing certs. From the beginning, we wanted users to be able to control what certificates were updated, it was just never implemented until #3880 due to concerns about the ambiguity of `domains` that I mentioned above. Since `domains` can be used to select certificates, I don't think the value is irrelevant and I'm very hesitant to just throw the value passed to Certbot away.

With all that said, we should do something to support this use case better, even if it is just providing a better error message. I believe @ohemorange has some ideas on things we could do here.

Resolving patch:
diff --git a/certbot/cli.py b/certbot/cli.py
--- a/certbot/cli.py
+++ b/certbot/cli.py
@@ -515,6 +515,13 @@ def _usage_string(self, plugins, help_arg):
 
         return usage
 
+    def remove_config_file_domains_for_renewal(self, parsed_args):
+        """Make "certbot renew" safe if domains are set in cli.ini."""
+        # Works around https://github.com/certbot/certbot/issues/4096
+        if self.verb == "renew":
+            for source, flags in self.parser._source_to_settings.items(): # pylint: disable=protected-access
+                if source.startswith("config_file") and "domains" in flags:
+                    parsed_args.domains = _Default() if self.detect_defaults else []
 
     def parse_args(self):
         """Parses command line arguments and returns the result.
@@ -527,6 +534,8 @@ def parse_args(self):
         parsed_args.func = self.VERBS[self.verb]
         parsed_args.verb = self.verb
 
+        self.remove_config_file_domains_for_renewal(parsed_args)
+
         if self.detect_defaults:
             return parsed_args
 


Test patch:
diff --git a/certbot/tests/cli_test.py b/certbot/tests/cli_test.py
--- a/certbot/tests/cli_test.py
+++ b/certbot/tests/cli_test.py
@@ -39,6 +39,7 @@ def test_read_file(self):
         self.assertEqual(contents, test_contents)
 
 
+
 class ParseTest(unittest.TestCase):
     '''Test the cli args entrypoint'''
 
@@ -61,6 +62,22 @@ def _help_output(self, args):
                 self.assertRaises(SystemExit, self.parse, args, output)
         return output.getvalue()
 
+    @mock.patch("certbot.cli.flag_default")
+    def test_cli_ini_domains(self, mock_flag_default):
+        tmp_config = tempfile.NamedTemporaryFile()
+        # use a shim to get ConfigArgParse to pick up tmp_config
+        shim = lambda v: constants.CLI_DEFAULTS[v] if v != "config_files" else [tmp_config.name]
+        mock_flag_default.side_effect = shim
+
+        namespace = self.parse(["certonly"])
+        self.assertEqual(namespace.domains, [])
+        tmp_config.write(b"domains = example.com")
+        tmp_config.flush()
+        namespace = self.parse(["certonly"])
+        self.assertEqual(namespace.domains, ["example.com"])
+        namespace = self.parse(["renew"])
+        self.assertEqual(namespace.domains, [])
+
     def test_no_args(self):
         namespace = self.parse([])
         for d in ('config_dir', 'logs_dir', 'work_dir'):