Repository: certbot/certbot

Issue:
Nginx installer not properly reloading configuration 
Detailed conversation could be found here:
https://community.letsencrypt.org/t/cannot-renew-create-a-new-cert-when-i-had-no-issue-previously/103040/

## My operating system is (include version):
Ubuntu 18.04.3 LTS
Nginx 1.17.4 (note, this isn't the version of nginx that's part of Ubuntu's repo's)

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):


## I ran this command and it produced this output:
`sudo certbot renew --cert-name ackis.duckdns.org --debug-challenges --dry-run`

```
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ackis.duckdns.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ackis.duckdns.org
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges
Attempting to renew cert (ackis.duckdns.org) from /etc/letsencrypt/renewal/ackis.duckdns.org.conf produced an unexpected error: Failed authorization procedure. ackis.duckdns.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ackis.duckdns.org/.well-known/acme-challenge/7bO_DNxtDDyO_hPdRJcpGWEJHaLTRwtTsMWpWtQREDE [174.3.126.96]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - -  - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: ackis.duckdns.org
   Type:   unauthorized
   Detail: Invalid response from
   https://ackis.duckdns.org/.well-known/acme-challenge/7bO_DNxtDDyO_hPdRJcpGWEJHaLTRwtTsMWpWtQREDE
   [174.3.126.96]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body>\r\n<center><h1>404 Not
   Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
```

## Certbot's behavior differed from what I expected because:

Certbot should have renewed the certificate.

## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:

```
# HTTP server - redirect to HTTPS
server {
        listen                                  80;
        listen                                  [::]:80;
        server_name                             www.ackis.duckdns.org ackis.duckdns.org;

        location / {
                return                          301 https://ackis.duckdns.org$request_uri;
        }

        # Workaround LE and certbot not working with nginx
        location /.well-known/acme-challenge/ {
                                                root /var/www/letsencrypt;
        }

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;

        add_header                              Strict-Transport-Security "max-age=31536000;";
        add_header                              X-Frame-Options SAMEORIGIN;
        add_header                              X-Content-Type-Options nosniff;
        add_header                              X-XSS-Protection "1; mode=block";
}

# www domain - redirect to domain without www
server {
        listen                                  443 ssl;
        listen                                  [::]:443 ssl;
        server_name                             www.ackis.duckdns.org;

        location / {
                return                          301 https://ackis.duckdns.org$request_uri;
        }

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;

        ssl_certificate                         /etc/letsencrypt/live/www.ackis.duckdns.org/fullchain.pem;
        ssl_certificate_key                     /etc/letsencrypt/live/www.ackis.duckdns.org/privkey.pem;

        server_tokens                           off;
        etag                                    off;

        add_header                              Strict-Transport-Security "max-age=31536000;";
        add_header                              X-Frame-Options SAMEORIGIN;
        add_header                              X-Content-Type-Options nosniff;
        add_header                              X-XSS-Protection "1; mode=block";
}

server {
        listen                                  443 ssl default_server;
        listen                                  [::]:443 ssl;
        server_name                             ackis.duckdns.org;

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;

        ssl_certificate                         /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem; # managed by Certbot
        ssl_certificate_key                     /etc/letsencrypt/live/ackis.duckdns.org/privkey.pem; # managed by Certbot

        server_tokens                           off;
        etag                                    off;

        add_header                              Strict-Transport-Security "max-age=31536000;";
        add_header                              X-Frame-Options SAMEORIGIN;
        add_header                              X-Content-Type-Options nosniff;
        add_header                              X-XSS-Protection "1; mode=block";

        location ~* /\.\./ {
                deny                            all;
                return                          404;
        }

        location ~* "^(?:.+\.(?:htaccess|make|txt|test|markdown|md|engine|inc|info|install|module|profile|po|sh|.*sql|theme|tpl(?:\.php)?|xtmpl)|code-style\.pl|/Entries.*|/Re$
                return                          404;
        }

        location = /favicon.ico {
                try_files                       /favicon.ico =204;
        }

        location / {
                root                            /var/www/internet;
                index                           index.html;
        }
        location /nginx_status {
                access_log                      syslog:server=localhost,tag=nginx_access_admin,severity=info;
                error_log                       syslog:server=localhost,tag=nginx_error_admin;
                allow                           192.168.0.0/24;
                deny                            all;

                auth_basic                      "Restricted access";
                auth_basic_user_file            /etc/nginx/auth/admin.htpasswd;

                stub_status                     on;
        }

}

```


Discussion:
> Nginx 1.17.4 (note, this isn't the version of nginx that's part of Ubuntu's repo's)

Was it installed from https://nginx.org/en/linux_packages.html#Ubuntu or elsewhere?
> > Nginx 1.17.4 (note, this isn't the version of nginx that's part of Ubuntu's repo's)
> 
> Was it installed from https://nginx.org/en/linux_packages.html#Ubuntu or elsewhere?

Yes, it's from their official repo.
Thanks for opening the issue. I wonder if there's something different in the systemd service that gets installed with the nginx version from Nginx repositories. I'm thinking of anything that would cause the nginx restarts to silently fail.
I tried to reproduce this with Ubuntu 18.04.3, nginx 1.17.4 from nginx mainline, Certbot 0.31.0 from PPA, using the posted configuration (with the domain name substituted and chopped-off regex fixed). Everything worked OK.

Would you be able to zip up your entire `/etc/nginx` directory and upload it? From your thread, I believe there are a large number of other virtualhosts on the server also. They could be having an influence.

e.g. Certbot reloads nginx (indirectly) via SIGHUP, which is asynchronous. Your nginx takes a long time to reload its full configuration. So long that the challenge requests arrive before the new config is actually applied. The challenges fail.

I'd also like to see whether there are nginx packages from multiple locations installed:

    dpkg-query -l "*nginx*"
`dpkg-query -l "*nginx*"`

```
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                Version        Architecture   Description
+++-===================-==============-==============-===========================================
ii  nginx               1.17.4-1~bioni amd64          high performance web server
rc  nginx-common        1.10.3-0ubuntu all            small, powerful, scalable web/proxy server
un  nginx-core          <none>         <none>         (no description available)
ii  nginx-doc           1.14.0-0ubuntu all            small, powerful, scalable web/proxy server
un  nginx-extras        <none>         <none>         (no description available)
un  nginx-full          <none>         <none>         (no description available)
un  nginx-light         <none>         <none>         (no description available)
ii  python-certbot-ngin 0.31.0-1+ubunt all            transitional dummy package
ii  python-certbot-ngin 0.23.0-1       all            Nginx plugin documentation for Certbot
ii  python3-certbot-ngi 0.31.0-1+ubunt all            Nginx plugin for Certbot

```

And my nginx config files are here: [nginx.tar.gz](https://github.com/certbot/certbot/files/3713170/nginx.tar.gz)

I've faced similar issues on my deployment (16.04 LTS, NGINX PPA, Certbot PPA)
Details here: https://community.letsencrypt.org/t/unable-to-renew-cert-using-nginx-plugin-fresh-cert-creation-succeeds/102843
My pastebin links, converted to attachments: [certbot issue 7422 - nginx -t.txt](https://github.com/certbot/certbot/files/3704995/certbot_help_topic_102843_-_nginx_-t.txt)
[certbot debug output with split root domain blocks - http tls.txt](https://github.com/certbot/certbot/files/3704997/certbot_with_split_root_domain_blocks___http_tls.txt)

Since reporting that issue, I reviewed my certbot per-site configuration files. I removed some pre and post hooks for legacy certificates, and unified them to all use the nginx validation and installer plugins. However the impacted certificates, including my test-case in that thread, were already inline with the modern plugins and no special hooks.

I've been able to complete those outstanding renewals, but I had to rerun `certbot renew` repeatedly as it churned through the renewals, failing to complete some renewals with similar failures consistent with improperly reloaded server configs.

In my test cases, I've been able to make renewals succeed just by running `systemctl reload nginx` repeatedly in a second window.
My nginx reloads instantly, but seems to take over a minute to restart. 
@Ackis Yup, that's similar to my experience. Essentially instant returns of `systemctl reload nginx`.  (Using `time` it is 0.026s) Restarting it again takes longer, though still much less than a minute. (roughly 0.120s clock `time`) I'm not sure if that's async. I recall reloads being slower in the past, but the systemd service files may have changed how that's done.

After leaving the server unattended for a few weeks I'll often find that nginx is running as root and working, but no longer linked to the systemd service. Leading me to suspect some disordered nginx launching during `certbot renew` scheduled runs, similar to #5486.

Explicit reloads have been sufficient for me in applying the .well-known validation interception hook to the config. @alexzorin may be describing the issue correctly, given that I also have a bunch of individual nginx server blocks. (Around 30 server{} blocks counting www and root entries separately.) And a sightly smaller number of individual certs, though I have been consolidating some.

This issue continues to be present with my certs as they come up for renewal. 

```
# dpkg-query -l "*nginx*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                  Version                         Architecture                    Description
+++-=====================================================-===============================-===============================-===============================================================================================================
ii  nginx                                                 1.16.1-1~xenial                 amd64                           high performance web server
un  nginx-common                                          <none>                          <none>                          (no description available)
un  nginx-core                                            <none>                          <none>                          (no description available)
un  python-certbot-nginx                                  <none>                          <none>                          (no description available)
un  python-certbot-nginx-doc                              <none>                          <none>                          (no description available)
ii  python3-certbot-nginx                                 0.31.0-1+ubuntu16.04.1+certbot+ all                             Nginx plugin for Certbot
```

```
# cat /lib/systemd/system/nginx.service
[Unit]
Description=nginx - high performance web server
Documentation=http://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target
```
Also having this issue.

To test I made the vhost modifications manually, did an Nginx reload, and made the request.

It took around 16 seconds for Nginx reload to complete, then a further 16 seconds before the response changed from 301/HTTPS to 200/token, for a total of 32 seconds for the correct response to be sent.

So could it be that the reload delay, or presumably more likely the second delay whilst waiting for the response to change, be the issue?

I've tried disabling any fastcgi and proxy caches in Nginx, but still having this 32 second gap.

EDIT:

So if I run interactively and sit around for 32 seconds it works, obviously not very helpful for auto-renewals.

**Fail**:
```
certbot renew --cert-name example.com -vvv
```

**Success**:
```
certbot certonly --cert-name example.com -vvv --force-interactive --debug-challenges
```

Ubuntu 18.04.3 LTS
nginx/1.16.1
certbot 0.31.0
I'm also having this problem... It seems to be very much related to the time it takes to reload nginx, which causes the authentication to fail (gets a 404) when it requests immediately after reload. On this particular server we have 400+ .conf files that are being loaded in, with proxy_pass statements. I'm not entirely sure why the long reload, but I suspect it could be due to bad domains in some of the config files which nginx is failing to resolve which takes some time, or just the parse time of that many config files?

I realized I could force a bit of a slowdown in the nginx authenticator by adding a delay after reload, which looks to solve the problem completely in my case. I simply modified the ```certbot/certbot-nginx/certbot_nginx/_internal/configurator.py``` file and added a ```time.sleep(5)``` after the nginx_restart call (line 913):
```
        nginx_restart(self.conf('ctl'), self.nginx_conf)
        time.sleep(5)
```

This allows the nginx server to finish reloading before the api calls are made, and so it then correctly responds with the valid token instead of a 404.

This is not a proper solution, and it likely has to do with a specifically (badly) configured nginx, but it does fix the issue is this particular case. Maybe there could be a certbot nginx option to add a delay for slow web servers?
Yes, I think this bug can be summarised with the existing todo in `certbot/certbot-nginx/certbot_nginx/_internal/configurator.py`.

```
TODO: Check for expected servername and loop until it
appears or return an error if looping too long.
```

We basically need to wait until the restart has completed before carrying on.
Okay, so here's a pull request that waits for the restart to complete. #7740

Resolving patch:
diff --git a/certbot-nginx/certbot_nginx/_internal/configurator.py b/certbot-nginx/certbot_nginx/_internal/configurator.py
--- a/certbot-nginx/certbot_nginx/_internal/configurator.py
+++ b/certbot-nginx/certbot_nginx/_internal/configurator.py
@@ -77,6 +77,9 @@ def add_parser_arguments(cls, add):
         add("ctl", default=constants.CLI_DEFAULTS["ctl"], help="Path to the "
             "'nginx' binary, used for 'configtest' and retrieving nginx "
             "version number.")
+        add("sleep-seconds", default=constants.CLI_DEFAULTS["sleep_seconds"], type=int,
+            help="Number of seconds to wait for nginx configuration changes "
+            "to apply when reloading.")
 
     @property
     def nginx_conf(self):
@@ -912,7 +915,7 @@ def restart(self):
         :raises .errors.MisconfigurationError: If either the reload fails.
 
         """
-        nginx_restart(self.conf('ctl'), self.nginx_conf)
+        nginx_restart(self.conf('ctl'), self.nginx_conf, self.conf('sleep-seconds'))
 
     def config_test(self):
         """Check the configuration of Nginx for errors.
@@ -1159,7 +1162,7 @@ def _redirect_block_for_domain(domain):
     return redirect_block
 
 
-def nginx_restart(nginx_ctl, nginx_conf):
+def nginx_restart(nginx_ctl, nginx_conf, sleep_duration):
     """Restarts the Nginx Server.
 
     .. todo:: Nginx restart is fatal if the configuration references
@@ -1167,6 +1170,8 @@ def nginx_restart(nginx_ctl, nginx_conf):
         before restart.
 
     :param str nginx_ctl: Path to the Nginx binary.
+    :param str nginx_conf: Path to the Nginx configuration file.
+    :param int sleep_duration: How long to sleep after sending the reload signal.
 
     """
     try:
@@ -1190,10 +1195,12 @@ def nginx_restart(nginx_ctl, nginx_conf):
 
     except (OSError, ValueError):
         raise errors.MisconfigurationError("nginx restart failed")
-    # Nginx can take a moment to recognize a newly added TLS SNI servername, so sleep
-    # for a second. TODO: Check for expected servername and loop until it
-    # appears or return an error if looping too long.
-    time.sleep(1)
+    # Nginx can take a significant duration of time to fully apply a new config, depending
+    # on size and contents (https://github.com/certbot/certbot/issues/7422). Lacking a way
+    # to reliably identify when this process is complete, we provide the user with control
+    # over how long Certbot will sleep after reloading the configuration.
+    if sleep_duration > 0:
+        time.sleep(sleep_duration)
 
 
 def _determine_default_server_root():
diff --git a/certbot-nginx/certbot_nginx/_internal/constants.py b/certbot-nginx/certbot_nginx/_internal/constants.py
--- a/certbot-nginx/certbot_nginx/_internal/constants.py
+++ b/certbot-nginx/certbot_nginx/_internal/constants.py
@@ -1,6 +1,9 @@
 """nginx plugin constants."""
 import platform
 
+from acme.magic_typing import Any
+from acme.magic_typing import Dict
+
 FREEBSD_DARWIN_SERVER_ROOT = "/usr/local/etc/nginx"
 LINUX_SERVER_ROOT = "/etc/nginx"
 PKGSRC_SERVER_ROOT = "/usr/pkg/etc/nginx"
@@ -15,7 +18,8 @@
 CLI_DEFAULTS = dict(
     server_root=server_root_tmp,
     ctl="nginx",
-)
+    sleep_seconds=1
+) # type: Dict[str, Any]
 """CLI defaults."""
 
 


Test patch:
diff --git a/certbot-nginx/tests/configurator_test.py b/certbot-nginx/tests/configurator_test.py
--- a/certbot-nginx/tests/configurator_test.py
+++ b/certbot-nginx/tests/configurator_test.py
@@ -460,11 +460,13 @@ def test_get_openssl_version(self, mock_popen):
         self.assertEqual(self.config._get_openssl_version(), "")
 
     @mock.patch("certbot_nginx._internal.configurator.subprocess.Popen")
-    def test_nginx_restart(self, mock_popen):
+    @mock.patch("certbot_nginx._internal.configurator.time")
+    def test_nginx_restart(self, mock_time, mock_popen):
         mocked = mock_popen()
         mocked.communicate.return_value = ('', '')
         mocked.returncode = 0
         self.config.restart()
+        mock_time.sleep.assert_called_once_with(0.1234)
 
     @mock.patch("certbot_nginx._internal.configurator.subprocess.Popen")
     def test_nginx_restart_fail(self, mock_popen):
diff --git a/certbot-nginx/tests/test_util.py b/certbot-nginx/tests/test_util.py
--- a/certbot-nginx/tests/test_util.py
+++ b/certbot-nginx/tests/test_util.py
@@ -56,6 +56,7 @@ def get_nginx_configurator(self, config_path, config_dir, work_dir, logs_dir,
         backups = os.path.join(work_dir, "backups")
 
         self.configuration.nginx_server_root = config_path
+        self.configuration.nginx_sleep_seconds = 0.1234
         self.configuration.le_vhost_ext = "-le-ssl.conf"
         self.configuration.config_dir = config_dir
         self.configuration.work_dir = work_dir