Repository: certbot/certbot

Issue:
Apache and Nginx plugins should use reload rather than restart
letsencrypt-apache/letsencrypt_apache/configurator.py

> ```
> .. todo:: Try to use reload instead. (This caused timing problems before)                                                                                                    
> ```

Our challenge methods are designed to be used without taking down a web server, so we should make sure we actually don't take down the web server to restart.

Note that Nginx has this issue too, noted in https://github.com/letsencrypt/letsencrypt/pull/950 and worked around.

We need a generic wait_for_challenge method that can be used for both plugins, which will repeatedly attempt to connect to the appropriate port on localhost and ask for the appropriate servername. If the loop times out after, say, five seconds, it should error out and say the configuration to add the challenge failed.



Discussion:
Copying from related PR:

-----jsha------
I think it's a little harder than it sounds, but I'd be interested in being proved wrong. My main concerns were:
(a) Can we be sure connecting to localhost is always the right thing? (probably yes)
(b) what method do I use to connect?

For (b), urllib seems to be the obvious answer, but we'd have to disable cert checking because it would be a snakeoil cert. However, if the new config hasn't loaded, we will get a default cert, so we won't get a nice failure like we wanted. Is there an easy to use connect method that will give me back the DNSNames of the provided cert regardless of cert validity?

-----kuba-----
(a) maybe just connect to the domain in question?
(b) you can use acme.crypto_util.probe_sni for that :)

(a) and (b) together: probe_sni(domain, socket.gethostbyname(domain), config.dvsni_port)

Apache reload _should_ make the new changes effective immediately, as it's using the vanilla apache graceful restart on most (all?) distros that it's available at (all?). Graceful restart _should_ serve the existing connections with the old configuration, and the new ones with the new one. 

@jdkasten can you remember the testing process that led to the todo-comment?

I'll try to tinker with this in the coming days.

I haven't tested the reload since Apache 2.2 back when I initially wrote up
a demo.

Reload used to fail for me about 1/6th of the time.  The configurations
would be edited, reload would be called, but the challenges failed because
the actual server didn't seem to load in the new configuration.

I would manually have to restart the server in order for it to read the new
configuration (challenge configuration)

I also used to test it by running a looped wget bash script while the Let's
Encrypt client was running.  (Just to see any performance implications)  My
guess at the time was that some long lasting connections caused the stall.

When I changed it to restart (a very long time ago)  the failures
disappeared.  I had vowed to return to the problem...

So in other words, if it appears to be working with the simple change to
reload, that is fantastic.  I had problems a few years ago and hadn't
messed with it since.  Apache may have updated their scirpt, or perhaps I
misattributed a different problem with the restart command.

On Sat, Nov 14, 2015 at 1:57 PM, joohoi notifications@github.com wrote:

> Apache reload _should_ make the new changes effective immediately, as
> it's using the vanilla apache graceful restart on most (all?) distros that
> it's available at (all?). Graceful restart _should_ serve the existing
> connections with the old configuration, and the new ones with the new one.
> 
> @jdkasten https://github.com/jdkasten can you remember the testing
> process that led to the todo-comment?
> 
> I'll try to tinker with this in the coming days.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/letsencrypt/letsencrypt/issues/954#issuecomment-156748719
> .

Note: there is almost certainly going to be a race condition on reload for both Nginx and Apache: the servers will take a non-zero amount of time to process the signal, parse the new config, and start serving with it. I did run into this race condition pretty reliably when working on the Nginx code. That's why I recommend adding an explicit loop check to wait until the new config is actually serving before proceeding with processing.

@jsha  Yes, the restart was implemented and //todo added... just to get
around the race condition temporarily.  I think the only viable solution is
to poll the server checking for the changes, it just requires a bit more
code.

IIRC, the reload timings (configuration changes) had a very high variance.

That being said, all webservers can reuse the same code to test for the
challenges.  (It can be added to the plugins directory within letsencrypt)

On Mon, Nov 16, 2015 at 8:56 AM, Jacob Hoffman-Andrews <
notifications@github.com> wrote:

> Note: there is almost certainly going to be a race condition on reload for
> both Nginx and Apache: the servers will take a non-zero amount of time to
> process the signal, parse the new config, and start serving with it. I did
> run into this race condition pretty reliably when working on the Nginx
> code. That's why I recommend adding an explicit loop check to wait until
> the new config is actually serving before proceeding with processing.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/letsencrypt/letsencrypt/issues/954#issuecomment-157096333
> .

I've also been thinking about this lately and wanted to follow up here. At first, I thought we only needed to poll when setting up for challenges. The thinking was that we only really care that new configuration is in effect when boulder is about to perform DV.

Unfortunately, this doesn't work. According to the [Apache documentation](https://httpd.apache.org/docs/2.4/stopping.html#graceful), it's possible for the reload command to return 0 and Apache fails to reload the config. Therefore, we need to poll on every reload to make sure Apache was able to load the new config.

The polling approach is a bit tricky. The Apache plugin calls `restart` a number of times internally after making different changes. This means we'd need to poll for different changes depending on when `restart` is called.

Even worse, the Apache plugin implements `restart` as part of the `IInstaller` interface which is called a number of times by the rest of the code base. This means that `restart` needs to be able to figure out what changes to poll for on its own, based on nothing than its internal state. Perhaps we can do this by examining the state of Augeas, but this seems quite difficult.

Additionally, we can't always poll ourselves. Specifically, proxies/firewalls will be a problem.

Something on the better side: I can't seem to be able to reproduce the issue with loading a new configuration with reload. The new config gets loaded every time.

I guess the problem previously could have had something to do with wget leaving connection open as default (without --no-http-keep-alive option)

However, as @jsha commented, the server takes time reloading the new configuration in any case, so some kind of delay needs to be in place. I haven't tested it with large amount of vhost configurations, so can't say for sure how the amount of different configurations affects the reload times. 

For smaller configuration sets the changes seem instant, at least with debian init scripts. I tested with:

```
sudo /etc/init.d/apache2 reload;curl http://test.local
```

And it produced the desired outcome every single time.

@bmw, what kind of cases are you thinking with the comment about apache failing to reload the configuration? Something that we can't predict with configtest? 

In my tests both debian init script and apache2ctl errored out with return code 1. However all I really tested was configuration syntax errors.

@joohoi, so I actually don't know of a case where the problem occurs, but it is possible according to the Apache documentation I linked above. To quote the documentation of graceful restart (which has the same effect of calling the init script with `reload`):

> When you issue a restart, a syntax check is first run, to ensure that there are no errors in the configuration files. If your configuration file has errors in it, you will get an error message about that syntax error, and the server will refuse to restart. This avoids the situation where the server halts and then cannot restart, leaving you with a non-functioning server.
> 
> This still will not guarantee that the server will restart correctly.

With that said, it appears `restart` has the same behavior and we currently don't do anything to protect against this case.

Perhaps we only have to do polling when setting up challenges. While not failproof, we could configure an additional vhost in the call to `perform`. We can then attempt to poll this vhost a fixed number of times. If we receive the expected response, we continue.

If we never receive the correct response, I think the best behavior is to attempt to continue here as well. The reason is it is possible we're unable to connect to ourselves for whatever reason but Apache has successfully loaded the new config.

For how I see it, as such, we wouldn't gain anything from polling the changes, as the race condition seems to be attributed to the connection keepalive settings from the HTTP client side in the graceful restart (reload).

It'd be nice if we could track that down for sure. There's no hard reason why Apache and Nginx would not be able to start accepting new connections with a new config while old ones are still inflight. And in fact such a problem would effectively kill graceful reload for many sites that have very long keepalives.

The keepalive problem usually effects only a subset of the child processes, and polling separate child processes is close to impossible in a coherent way. I suggest we leave the restart in place and introduce a --reload-only cli flag with appropriate warning to let the user choose whether to take a walk on the thin ice or not.

To recap what's going on here (as a tl;dr and to clear my thoughts): 
- We want to use reload instead of restart in both Apache and Nginx, but:
  - Reloading configuration (and it coming to an effect) takes undefined time, depending on the amount of configuration present etc.
  - There's an underlying keepalive issue with Apache child processes.
  - Polling via HTTP may not be feasible in some corner configuration cases.
  - Apache documentation states that there may be (is?) some corner cases where the new configuration will not be reloaded even though the return codes state so.
- We can use apache2ctl -S to determine some parts of active vhost configuration, but it doesn't take the old child processes running and old configuration in to account.
- Suggestion about letting the user decide if they want to use reload instead of restart, but let them know about the possible shortcomings.

> Reloading configuration (and it coming to an effect) takes undefined time, depending on the amount of configuration present etc.

I believe this is also true of `restart`. My understanding is that both `restart` and `reload` run a config test, send the Apache process a signal, and immediately return. Sure the signal and processing time is different, but both the `restart` and `reload` command return before the process is complete. The actual reload/restart takes an undefined amount of time.

> There's an underlying keepalive issue with Apache child processes.

So I can confirm from testing that Apache 2.4 can handle requests with the new and old configuration at the same time. In my tests, all new connections were handled with the new configuration, but I don't know that this always happens.

Due to the complexity of doing this right, I'm leaning towards either just adding a short call to `time.sleep` after reloading the config for challenges or kicking this out of the 1.0 milestone.

I think it's really important to have this for 1.0. I'm fine with a call to `sleep` as a short-term workaround. I'd suggest at least two seconds as a ballpark figure.


Resolving patch:
diff --git a/letsencrypt-apache/letsencrypt_apache/configurator.py b/letsencrypt-apache/letsencrypt_apache/configurator.py
--- a/letsencrypt-apache/letsencrypt_apache/configurator.py
+++ b/letsencrypt-apache/letsencrypt_apache/configurator.py
@@ -8,6 +8,7 @@
 import shutil
 import socket
 import subprocess
+import time
 
 import zope.interface
 
@@ -96,7 +97,7 @@ def add_parser_arguments(cls, add):
             help="Path to the Apache 'a2enmod' binary.")
         add("init-script", default=constants.CLI_DEFAULTS["init_script"],
             help="Path to the Apache init script (used for server "
-            "reload/restart).")
+            "reload).")
         add("le-vhost-ext", default=constants.CLI_DEFAULTS["le_vhost_ext"],
             help="SSL vhost configuration extension.")
         add("server-root", default=constants.CLI_DEFAULTS["server_root"],
@@ -974,7 +975,7 @@ def is_site_enabled(self, avail_fp):
         return False
 
     def enable_site(self, vhost):
-        """Enables an available site, Apache restart required.
+        """Enables an available site, Apache reload required.
 
         .. note:: Does not make sure that the site correctly works or that all
                   modules are enabled appropriately.
@@ -1009,7 +1010,7 @@ def enable_site(self, vhost):
     def enable_mod(self, mod_name, temp=False):
         """Enables module in Apache.
 
-        Both enables and restarts Apache so module is active.
+        Both enables and reloads Apache so module is active.
 
         :param str mod_name: Name of the module to enable. (e.g. 'ssl')
         :param bool temp: Whether or not this is a temporary action.
@@ -1051,7 +1052,7 @@ def enable_mod(self, mod_name, temp=False):
 
         # Modules can enable additional config files. Variables may be defined
         # within these new configuration sections.
-        # Restart is not necessary as DUMP_RUN_CFG uses latest config.
+        # Reload is not necessary as DUMP_RUN_CFG uses latest config.
         self.parser.update_runtime_variables(self.conf("ctl"))
 
     def _add_parser_mod(self, mod_name):
@@ -1074,16 +1075,16 @@ def _enable_mod_debian(self, mod_name, temp):
         le_util.run_script([self.conf("enmod"), mod_name])
 
     def restart(self):
-        """Restarts apache server.
+        """Reloads apache server.
 
         .. todo:: This function will be converted to using reload
 
-        :raises .errors.MisconfigurationError: If unable to restart due
-            to a configuration problem, or if the restart subprocess
+        :raises .errors.MisconfigurationError: If unable to reload due
+            to a configuration problem, or if the reload subprocess
             cannot be run.
 
         """
-        return apache_restart(self.conf("init-script"))
+        return apache_reload(self.conf("init-script"))
 
     def config_test(self):  # pylint: disable=no-self-use
         """Check the configuration of Apache for errors.
@@ -1158,11 +1159,15 @@ def perform(self, achalls):
 
         sni_response = apache_dvsni.perform()
         if sni_response:
-            # Must restart in order to activate the challenges.
+            # Must reload in order to activate the challenges.
             # Handled here because we may be able to load up other challenge
             # types
             self.restart()
 
+            # TODO: Remove this dirty hack. We need to determine a reliable way
+            # of identifying when the new configuration is being used.
+            time.sleep(3)
+
             # Go through all of the challenges and assign them to the proper
             # place in the responses return value. All responses must be in the
             # same order as the original challenges.
@@ -1201,42 +1206,42 @@ def _get_mod_deps(mod_name):
     return deps.get(mod_name, [])
 
 
-def apache_restart(apache_init_script):
-    """Restarts the Apache Server.
+def apache_reload(apache_init_script):
+    """Reloads the Apache Server.
 
     :param str apache_init_script: Path to the Apache init script.
 
     .. todo:: Try to use reload instead. (This caused timing problems before)
 
     .. todo:: On failure, this should be a recovery_routine call with another
-       restart.  This will confuse and inhibit developers from testing code
+       reload.  This will confuse and inhibit developers from testing code
        though.  This change should happen after
        the ApacheConfigurator has been thoroughly tested.  The function will
        need to be moved into the class again.  Perhaps
        this version can live on... for testing purposes.
 
-    :raises .errors.MisconfigurationError: If unable to restart due to a
-        configuration problem, or if the restart subprocess cannot be run.
+    :raises .errors.MisconfigurationError: If unable to reload due to a
+        configuration problem, or if the reload subprocess cannot be run.
 
     """
     try:
-        proc = subprocess.Popen([apache_init_script, "restart"],
+        proc = subprocess.Popen([apache_init_script, "reload"],
                                 stdout=subprocess.PIPE,
                                 stderr=subprocess.PIPE)
 
     except (OSError, ValueError):
         logger.fatal(
-            "Unable to restart the Apache process with %s", apache_init_script)
+            "Unable to reload the Apache process with %s", apache_init_script)
         raise errors.MisconfigurationError(
-            "Unable to restart Apache process with %s" % apache_init_script)
+            "Unable to reload Apache process with %s" % apache_init_script)
 
     stdout, stderr = proc.communicate()
 
     if proc.returncode != 0:
         # Enter recovery routine...
-        logger.error("Apache Restart Failed!\n%s\n%s", stdout, stderr)
+        logger.error("Apache Reload Failed!\n%s\n%s", stdout, stderr)
         raise errors.MisconfigurationError(
-            "Error while restarting Apache:\n%s\n%s" % (stdout, stderr))
+            "Error while reloading Apache:\n%s\n%s" % (stdout, stderr))
 
 
 def get_file_path(vhost_path):


Test patch: