Repository: certbot/certbot

Issue:
Bad installation not properly reverted
I just ran `letsencrypt install --cert-path $(mktemp -d) --key-path $(mktemp -d) -t` with only Apache installed. Installation failed, but the changes to the vhost were not reverted, leaving directives like this:

```
SSLCertificateFile /tmp/tmp.57eWjciAOH
SSLCertificateKeyFile /tmp/tmp.h7pSVhLI4U
```



Discussion:
Here's a full log of the issue with a fresh install of Let's Encrypt and Apache on Debian Jessie.

```
letsencrypt install --cert-path $(mktemp -d) --key-path $(mktemp -d) --agree-tos --agree-eula -d example.com -t -vv --debug
2015-10-19 11:44:33,409:DEBUG:letsencrypt.cli:Root logging level set at 10
2015-10-19 11:44:33,410:INFO:letsencrypt.cli:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2015-10-19 11:44:33,410:DEBUG:letsencrypt.cli:Arguments: ['install', '--cert-path', '/tmp/tmp.dafUPdfoR9', '--key-path', '/tmp/tmp.7EvbekJSnO', '--agree-tos', '--agree-eula', '-d', 'example.com', '-t', '-vv', '--debug']
2015-10-19 11:44:33,411:DEBUG:letsencrypt.cli:Discovered plugins: PluginsRegistry(PluginEntryPoint#nginx,PluginEntryPoint#standalone,PluginEntryPoint#manual,PluginEntryPoint#webroot,PluginEntryPoint#apache,PluginEntryPoint#null)
2015-10-19 11:44:33,649:DEBUG:letsencrypt.plugins.disco:No installation (PluginEntryPoint#nginx): Could not find configuration root
Traceback (most recent call last):
  File "/root/letsencrypt/letsencrypt/plugins/disco.py", line 103, in prepare
    self._initialized.prepare()
  File "/root/letsencrypt/letsencrypt-nginx/letsencrypt_nginx/configurator.py", line 112, in prepare
    self.conf('server-root'), self.mod_ssl_conf)
  File "/root/letsencrypt/letsencrypt-nginx/letsencrypt_nginx/parser.py", line 29, in __init__
    self.loc = self._set_locations(ssl_options)
  File "/root/letsencrypt/letsencrypt-nginx/letsencrypt_nginx/parser.py", line 179, in _set_locations
    root = self._find_config_root()
  File "/root/letsencrypt/letsencrypt-nginx/letsencrypt_nginx/parser.py", line 202, in _find_config_root
    "Could not find configuration root")
NoInstallationError: Could not find configuration root
2015-10-19 11:44:33,650:DEBUG:letsencrypt.display.ops:Single candidate plugin: * apache
Description: Apache Web Server - Alpha
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = letsencrypt_apache.configurator:ApacheConfigurator
Initialized: <letsencrypt_apache.configurator.ApacheConfigurator object at 0x7fbcc7bebd10>
Prep: True
2015-10-19 11:44:33,655:INFO:letsencrypt_apache.configurator:Created an SSL vhost at /etc/apache2/sites-available/000-default-le-ssl.conf
2015-10-19 11:44:33,662:DEBUG:letsencrypt.reverter:Creating backup of /etc/apache2/sites-available/000-default-le-ssl.conf
2015-10-19 11:44:33,738:INFO:letsencrypt_apache.configurator:Deploying Certificate to VirtualHost /etc/apache2/sites-available/000-default-le-ssl.conf
2015-10-19 11:44:33,739:INFO:letsencrypt_apache.configurator:Enabling available site: /etc/apache2/sites-available/000-default-le-ssl.conf
2015-10-19 11:44:33,819:ERROR:letsencrypt_apache.configurator:Apache Restart Failed!
Restarting apache2 (via systemctl): apache2.service failed!

Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.

2015-10-19 11:44:33,820:DEBUG:letsencrypt.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/root/letsencrypt/letsencrypt/client.py", line 342, in deploy_certificate
    self.installer.restart()
  File "/root/letsencrypt/letsencrypt-apache/letsencrypt_apache/configurator.py", line 1067, in restart
    return apache_restart(self.conf("init-script"))
  File "/root/letsencrypt/letsencrypt-apache/letsencrypt_apache/configurator.py", line 1221, in apache_restart
    "Error while restarting Apache:\n%s\n%s" % (stdout, stderr))
MisconfigurationError: Error while restarting Apache:
Restarting apache2 (via systemctl): apache2.service failed!

Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.


2015-10-19 11:44:33,820:DEBUG:letsencrypt.error_handler:Calling registered functions
2015-10-19 11:44:33,824:DEBUG:letsencrypt.cli:Exiting abnormally:
Traceback (most recent call last):
  File "/root/letsencrypt/venv/bin/letsencrypt", line 9, in <module>
    load_entry_point('letsencrypt==0.1.0.dev0', 'console_scripts', 'letsencrypt')()
  File "/root/letsencrypt/letsencrypt/cli.py", line 1002, in main
    return args.func(args, config, plugins)
  File "/root/letsencrypt/letsencrypt/cli.py", line 402, in install
    args.fullchain_path)
  File "/root/letsencrypt/letsencrypt/client.py", line 342, in deploy_certificate
    self.installer.restart()
  File "/root/letsencrypt/letsencrypt-apache/letsencrypt_apache/configurator.py", line 1067, in restart
    return apache_restart(self.conf("init-script"))
  File "/root/letsencrypt/letsencrypt-apache/letsencrypt_apache/configurator.py", line 1221, in apache_restart
    "Error while restarting Apache:\n%s\n%s" % (stdout, stderr))
MisconfigurationError: Error while restarting Apache:
Restarting apache2 (via systemctl): apache2.service failed!

Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.


Traceback (most recent call last):
  File "/root/letsencrypt/venv/bin/letsencrypt", line 9, in <module>
    load_entry_point('letsencrypt==0.1.0.dev0', 'console_scripts', 'letsencrypt')()
  File "/root/letsencrypt/letsencrypt/cli.py", line 1002, in main
    return args.func(args, config, plugins)
  File "/root/letsencrypt/letsencrypt/cli.py", line 402, in install
    args.fullchain_path)
  File "/root/letsencrypt/letsencrypt/client.py", line 342, in deploy_certificate
    self.installer.restart()
  File "/root/letsencrypt/letsencrypt-apache/letsencrypt_apache/configurator.py", line 1067, in restart
    return apache_restart(self.conf("init-script"))
  File "/root/letsencrypt/letsencrypt-apache/letsencrypt_apache/configurator.py", line 1221, in apache_restart
    "Error while restarting Apache:\n%s\n%s" % (stdout, stderr))
MisconfigurationError: Error while restarting Apache:
Restarting apache2 (via systemctl): apache2.service failed!

Job for apache2.service failed. See 'systemctl status apache2.service' and 'journalctl -xn' for details.
```

This is a result of Apache never actually being prepped for release.  The recovery_routine() should be called in case of failure.  (It hadn't been prepped because it makes testing harder)

Steps to fix.

Add self.recovery_routine() here
https://github.com/letsencrypt/letsencrypt/blob/master/letsencrypt-apache/letsencrypt_apache/configurator.py#L1217

(Note this will require the function to be moved up into the .restart() method.  or an easier approach would be to simply catch the error in ApacheConfigurator.restart() and then do the recovery_routine())

Now this only fixes the symptoms.  The install verb will have to be fixed in #809 to avoid this error all together.

You've probably looked into this issue more than I have, but I'm not sure that will work. The reason I think this is that `recovery_routine` is already being called on the Apache plugin after the error is encountered. The code that makes the call is [here](https://github.com/letsencrypt/letsencrypt/blob/master/letsencrypt/client.py#L322).

It is evident from the logs above that the `ErrorHandler` runs, but there is no additional logging information being added by the call to `self.installer.recovery_routine` which quickly looking through the code seems to mean that the recovery was "successful". I wonder if there is also a problem with registering changes with the `reverter`?

Ahh... sorry @bmw I had forgotten you protected me with your error handling! :+1:    I was quickly trying to go through and comment on things.

I just walked through #809 with Noah and I think we solved that problem.

#809 appears to be a problem with saving configuration changes (Noah is writing a quick PR).  If that doesn't solve this problem (I am not convinced it will immediately), I will take a closer look.   Sorry for the spam and thanks for looking into this.

We need to determine whether #809 did or did not fix this.  I'm going to try to reproduce it now.

@jdkasten confirmed that this is still present.  As a failure to keep config files sane, I think this is an extremely urgent bug.


Resolving patch:
diff --git a/letsencrypt/client.py b/letsencrypt/client.py
--- a/letsencrypt/client.py
+++ b/letsencrypt/client.py
@@ -329,8 +329,26 @@ def deploy_certificate(self, domains, privkey_path,
                 self.installer.save()  # needed by the Apache plugin
 
             self.installer.save("Deployed Let's Encrypt Certificate")
-            # sites may have been enabled / final cleanup
+
+        # sites may have been enabled / final cleanup
+        with error_handler.ErrorHandler(self._rollback_and_restart):
+            self.installer.restart()
+
+    def _rollback_and_restart(self):
+        """Rollback the most recent checkpoint and restart the webserver"""
+        logger.critical("Rolling back to previous server configuration...")
+        try:
+            self.installer.rollback_checkpoints()
             self.installer.restart()
+        except:
+            # TODO: suggest letshelp-letsencypt here
+            logger.critical("Failure to rollback config "
+                            "changes and restart your server")
+            logger.critical("Please submit a bug report to "
+                            "https://github.com/letsencrypt/letsencrypt")
+            raise
+        logger.critical("Rollback successful; your server has "
+                        "been restarted with your old configuration")
 
     def enhance_config(self, domains, redirect=None):
         """Enhance the configuration.


Test patch:
diff --git a/letsencrypt/tests/client_test.py b/letsencrypt/tests/client_test.py
--- a/letsencrypt/tests/client_test.py
+++ b/letsencrypt/tests/client_test.py
@@ -166,6 +166,27 @@ def test_deploy_certificate(self):
         self.assertEqual(installer.save.call_count, 2)
         installer.restart.assert_called_once_with()
 
+    def test_deploy_certificate_restart_failure_with_recovery(self):
+        installer = mock.MagicMock()
+        installer.restart.side_effect = [errors.PluginError, None]
+        self.client.installer = installer
+
+        self.assertRaises(errors.PluginError, self.client.deploy_certificate,
+                          ["foo.bar"], "key", "cert", "chain", "fullchain")
+        installer.rollback_checkpoints.assert_called_once_with()
+        self.assertEqual(installer.restart.call_count, 2)
+
+    def test_deploy_certificate_restart_failure_without_recovery(self):
+        installer = mock.MagicMock()
+        installer.restart.side_effect = errors.PluginError
+        installer.rollback_checkpoints.side_effect = errors.ReverterError
+        self.client.installer = installer
+
+        self.assertRaises(errors.PluginError, self.client.deploy_certificate,
+                          ["foo.bar"], "key", "cert", "chain", "fullchain")
+        installer.rollback_checkpoints.assert_called_once_with()
+        self.assertEqual(installer.restart.call_count, 1)
+
     @mock.patch("letsencrypt.client.enhancements")
     def test_enhance_config(self, mock_enhancements):
         self.assertRaises(errors.Error,