Repository: certbot/certbot

Issue:
Interpret replace=True as "replace if possible, otherwise append"
https://community.letsencrypt.org/t/permissions-required-to-etc-letsencrypt-live-for-script/35645/6

In Nginx, say someone's done something like this:

```
server {
  server_name perfectly-good-name.com;
  listen 443 ssl;
  # ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem;
  # ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem;
}
```

They know where the certs will live, so they've put in links to them, but Nginx needs real files there, so they comment them out.

But now the installer's going to fail, because we make the assumption that if someone has a server block with ssl enabled, we shouldn't add any ssl directives because we should try to leave things as much the same as possible.

I propose changing this assumption from an empirical perspective. People run into this all the time, so therefore it's the workflow that people expect should work. Therefore we should support it.

I'd love to hear any reasons why we didn't do this previously, or if we treat this differently in Apache.


Discussion:
@bmw @joona?
My two cents as a user, on varying degrees of improvement that could be made:

1) Least Work: Just improve the error message... "Unable to install" sounds like something went wrong, which triggers the need to find out what happened and evaluate if other things may have broken. It would help if the message said something like:
`Certificate created and stored at <path to certs>. Modification of existing SSL server blocks not supported. Manual configuration will be required. 
NOTE: Most users just need to add the following to their ssl-enabled server block(s):
ssl_certificate /etc/letsencrypt/live/example.org/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.org/privkey.pem;`

2) Most Work: Allow the script to run regardless of validity of existing ssl_certificate configuration. If existing ssl_certificate configuration becomes valid because of cert creation, do nothing. Otherwise, update the config to be valid, using the new certificate. (Possibly with "Do you want to overwrite" type messages for cert files if some already exist.)

3. Middle ground: One could imagine a scope-reduced version of (2) above... Just handle the easiest most common cases... I.e. only those with a single server block using the cert. Any case more complex than that could fall back on a useful error message... Something like "Site configuration too complex for script to safely handle. Certificates are created, but manual configuration will be required."

Thanks.
> I propose changing this assumption from an empirical perspective. People run into this all the time, so therefore it's the workflow that people expect should work. Therefore we should support it.
>
> I'd love to hear any reasons why we didn't do this previously, or if we treat this differently in Apache.

I had to do some digging and testing here as the relevant code even predates my involvement in the project. I agree that we should do this when trying to add the cert and key directives to a virtual host.

To provide the best justification I could come up with for why things are this way, in both Apache and Nginx a virtual host like this is probably technically misconfigured. You're telling your webserver to use HTTPS but not providing a cert or key unless you've added one to your top level config which isn't done by default or a normal practice as far as I know.

This is especially true in Apache where Apache will fail to start if you tell it to use SSL/TLS without a cert and key. Nginx behaves differently and will start, but will refuse connections to the server and write messages to `error.log`. Part of the reason we do this in Nginx may be that the Nginx plugin was written to strongly mimic the way the Apache plugin worked at the time where this makes a little more sense.

With all that said, Nginx still runs in this case and the configuration is valid in either server if there are certs and keys in the top level config. I see no problem fixing this and made #4837 to track the issue in Apache.

In Nginx, this could be done as simply as catching the error thrown if `replace=True` and the directive doesn't exist and calling again with `replace=False`. We could make an additional function in `parser.py` to do this if we wanted. Alternatively, we could just make `replace=True` have this behavior by default, but I'd carefully audit the code to make sure this is what we want to happen in every case.
This case is the one and only time it's called with `replace=True`.

Resolving patch:
diff --git a/certbot-nginx/certbot_nginx/configurator.py b/certbot-nginx/certbot_nginx/configurator.py
--- a/certbot-nginx/certbot_nginx/configurator.py
+++ b/certbot-nginx/certbot_nginx/configurator.py
@@ -198,16 +198,10 @@ def deploy_cert(self, domain, cert_path, key_path,
         cert_directives = [['\n', 'ssl_certificate', ' ', fullchain_path],
                            ['\n', 'ssl_certificate_key', ' ', key_path]]
 
-        try:
-            self.parser.add_server_directives(vhost,
-                                              cert_directives, replace=True)
-            logger.info("Deployed Certificate to VirtualHost %s for %s",
-                        vhost.filep, vhost.names)
-        except errors.MisconfigurationError as error:
-            logger.debug(error)
-            # Presumably break here so that the virtualhost is not modified
-            raise errors.PluginError("Cannot find a cert or key directive in {0} for {1}. "
-                "VirtualHost was not modified.".format(vhost.filep, vhost.names))
+        self.parser.add_server_directives(vhost,
+                                          cert_directives, replace=True)
+        logger.info("Deployed Certificate to VirtualHost %s for %s",
+                    vhost.filep, vhost.names)
 
         self.save_notes += ("Changed vhost at %s with addresses of %s\n" %
                             (vhost.filep,
diff --git a/certbot-nginx/certbot_nginx/parser.py b/certbot-nginx/certbot_nginx/parser.py
--- a/certbot-nginx/certbot_nginx/parser.py
+++ b/certbot-nginx/certbot_nginx/parser.py
@@ -278,8 +278,8 @@ def add_server_directives(self, vhost, directives, replace):
 
         This method modifies vhost to be fully consistent with the new directives.
 
-        ..note :: If replace is True, this raises a misconfiguration error
-        if the directive does not already exist.
+        ..note :: If replace is True and the directive already exists, the first
+        instance will be replaced. Otherwise, the directive is added.
         ..note :: If replace is False nothing gets added if an identical
         block exists already.
 
@@ -464,8 +464,9 @@ def _add_directives(block, directives, replace):
     When replace=False, it's an error to try and add a directive that already
     exists in the config block with a conflicting value.
 
-    When replace=True, a directive with the same name MUST already exist in the
-    config block, and the first instance will be replaced.
+    When replace=True and a directive with the same name already exists in the
+    config block, the first instance will be replaced. Otherwise, the directive
+    will be added to the config block.
 
     ..todo :: Find directives that are in included files.
 
@@ -547,49 +548,46 @@ def find_location(direc):
     location = find_location(directive)
 
     if replace:
-        if location is None:
-            raise errors.MisconfigurationError(
-                'expected directive for {0} in the Nginx '
-                'config but did not find it.'.format(directive[0]))
-        block[location] = directive
-        _comment_directive(block, location)
-    else:
-        # Append directive. Fail if the name is not a repeatable directive name,
-        # and there is already a copy of that directive with a different value
-        # in the config file.
-
-        # handle flat include files
-
-        directive_name = directive[0]
-        def can_append(loc, dir_name):
-            """ Can we append this directive to the block? """
-            return loc is None or (isinstance(dir_name, str) and dir_name in REPEATABLE_DIRECTIVES)
-
-        err_fmt = 'tried to insert directive "{0}" but found conflicting "{1}".'
-
-        # Give a better error message about the specific directive than Nginx's "fail to restart"
-        if directive_name == INCLUDE:
-            # in theory, we might want to do this recursively, but in practice, that's really not
-            # necessary because we know what file we're talking about (and if we don't recurse, we
-            # just give a worse error message)
-            included_directives = _parse_ssl_options(directive[1])
-
-            for included_directive in included_directives:
-                included_dir_loc = find_location(included_directive)
-                included_dir_name = included_directive[0]
-                if not is_whitespace_or_comment(included_directive) \
-                    and not can_append(included_dir_loc, included_dir_name):
-                    if block[included_dir_loc] != included_directive:
-                        raise errors.MisconfigurationError(err_fmt.format(included_directive,
-                            block[included_dir_loc]))
-                    else:
-                        _comment_out_directive(block, included_dir_loc, directive[1])
-
-        if can_append(location, directive_name):
-            block.append(directive)
-            _comment_directive(block, len(block) - 1)
-        elif block[location] != directive:
-            raise errors.MisconfigurationError(err_fmt.format(directive, block[location]))
+        if location is not None:
+            block[location] = directive
+            _comment_directive(block, location)
+            return
+    # Append directive. Fail if the name is not a repeatable directive name,
+    # and there is already a copy of that directive with a different value
+    # in the config file.
+
+    # handle flat include files
+
+    directive_name = directive[0]
+    def can_append(loc, dir_name):
+        """ Can we append this directive to the block? """
+        return loc is None or (isinstance(dir_name, str) and dir_name in REPEATABLE_DIRECTIVES)
+
+    err_fmt = 'tried to insert directive "{0}" but found conflicting "{1}".'
+
+    # Give a better error message about the specific directive than Nginx's "fail to restart"
+    if directive_name == INCLUDE:
+        # in theory, we might want to do this recursively, but in practice, that's really not
+        # necessary because we know what file we're talking about (and if we don't recurse, we
+        # just give a worse error message)
+        included_directives = _parse_ssl_options(directive[1])
+
+        for included_directive in included_directives:
+            included_dir_loc = find_location(included_directive)
+            included_dir_name = included_directive[0]
+            if not is_whitespace_or_comment(included_directive) \
+                and not can_append(included_dir_loc, included_dir_name):
+                if block[included_dir_loc] != included_directive:
+                    raise errors.MisconfigurationError(err_fmt.format(included_directive,
+                        block[included_dir_loc]))
+                else:
+                    _comment_out_directive(block, included_dir_loc, directive[1])
+
+    if can_append(location, directive_name):
+        block.append(directive)
+        _comment_directive(block, len(block) - 1)
+    elif block[location] != directive:
+        raise errors.MisconfigurationError(err_fmt.format(directive, block[location]))
 
 def _apply_global_addr_ssl(addr_to_ssl, parsed_server):
     """Apply global sslishness information to the parsed server block


Test patch:
diff --git a/certbot-nginx/certbot_nginx/tests/parser_test.py b/certbot-nginx/certbot_nginx/tests/parser_test.py
--- a/certbot-nginx/certbot_nginx/tests/parser_test.py
+++ b/certbot-nginx/certbot_nginx/tests/parser_test.py
@@ -273,11 +273,16 @@ def test_replace_server_directives(self):
                            ['server_name', 'example.*'], []
                            ]]])
         mock_vhost.names = set(['foobar.com', 'example.*'])
-        self.assertRaises(errors.MisconfigurationError,
-                          nparser.add_server_directives,
-                          mock_vhost,
-                          [['ssl_certificate', 'cert.pem']],
-                          replace=True)
+        nparser.add_server_directives(
+            mock_vhost, [['ssl_certificate', 'cert.pem']], replace=True)
+        self.assertEqual(
+            nparser.parsed[filep],
+            [[['server'], [['listen', '69.50.225.155:9000'],
+                           ['listen', '127.0.0.1'],
+                           ['server_name', 'foobar.com'], ['#', COMMENT],
+                           ['server_name', 'example.*'], [],
+                           ['ssl_certificate', 'cert.pem'], ['#', COMMENT], [],
+                           ]]])
 
     def test_get_best_match(self):
         target_name = 'www.eff.org'