Repository: certbot/certbot

Issue:
High CPU utilization / long delay during "Cleaning up challenges"
## My operating system is (include version):
`Linux nginx-primary 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux`

## I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):
```
snap --version
snap    2.58.2
snapd   2.58.2
series  16
ubuntu  18.04
kernel  4.15.0-112-generic
```

## Certbot's behavior differed from what I expected because:
Previous to the release version `2.3.0` (`Feb 14, 2023`), the average time between `"Cleaning up challenges"` and the next log line output on our machine was approximately `30 seconds`. Starting `Feb 15, 2023`, after `certbot` was auto-updated to `2.3.0`, we started getting high CPU utilization reports (using ~95% of the CPU) when attempting to secure a new certificate. Also, the time between `"Cleaning up challenges"` and the next line spiked to `1 hour to 1.5 hours`, and then eventually seemingly settled at an average of `~3 minutes`.


## Here is a Certbot log showing the issue (if available):
### Example log history prior to 2.3.0 / Feb 14, 2023
```
2023-02-12 10:29:30,109:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-02-12 10:30:00,318:DEBUG:certbot._internal.client:CSR: CSR(file='/etc/letsencrypt/csr/264088_csr-certbot.pem', data=b'-----BEGIN CERTIFICATE REQUEST-----\n[redcated]\n', form='pem')

2023-02-13 00:44:13,040:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-02-13 00:44:45,110:DEBUG:certbot._internal.client:CSR: CSR(file='/etc/letsencrypt/csr/264203_csr-certbot.pem', data=b'-----BEGIN CERTIFICATE REQUEST-----\n[redacted]\n-----END CERTIFICATE REQUEST-----\n', form='pem')

2023-02-13 00:59:12,926:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-02-13 00:59:44,519:DEBUG:certbot._internal.client:CSR: CSR(file='/etc/letsencrypt/csr/264213_csr-certbot.pem', data=b'-----BEGIN CERTIFICATE REQUEST-----\n[redacted]\n-----END CERTIFICATE REQUEST-----\n', form='pem')
```
Note the time difference between the two lines.

### Example log history after 2.3.0 upgrade / Feb 15, 2023 and onward
```
2023-02-16 01:25:30,996:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-02-16 02:21:00,166:DEBUG:certbot._internal.client:CSR: CSR(file=None, data=b'-----BEGIN CERTIFICATE REQUEST-----\n[redacted]\n', form='pem')

2023-02-17 10:16:18,451:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-02-17 11:50:31,763:DEBUG:certbot._internal.client:CSR: CSR(file=None, data=b'-----BEGIN CERTIFICATE REQUEST-----\n[redacted]\n-----END CERTIFICATE REQUEST-----\n', form='pem')

2023-02-22 14:43:55,446:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-02-22 14:47:04,871:DEBUG:certbot._internal.client:CSR: CSR(file=None, data=b'-----BEGIN CERTIFICATE REQUEST-----\n[redacted]\n-----END CERTIFICATE REQUEST-----\n', form='pem')

2023-02-22 20:10:23,585:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-02-22 20:13:38,653:DEBUG:certbot._internal.client:CSR: CSR(file=None, data=b'-----BEGIN CERTIFICATE REQUEST-----\n[redacted]\n-----END CERTIFICATE REQUEST-----\n', form='pem')

2023-02-22 20:04:09,426:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-02-22 20:07:21,719:DEBUG:certbot._internal.client:CSR: CSR(file=None, data=b'-----BEGIN CERTIFICATE REQUEST-----\n[redacted]\n', form='pem')
```
Note the vastly increased time difference between the two lines.

## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:
Using NGINX challenges.

I was attempting to dig into this further, but from what I could tell, the NGINX plugin doesn't seem to write log entries very often, so I'm not sure what it's doing during "Cleaning up challenges" that is taking so much longer compared to the previous release.

Our server has about ~2000 domains / certificates active.


Discussion:
Sorry to hear that! Do you happen to know the exact certbot version that corresponds to your earlier logs? Prior to release 2.3.0 generally, the version was 1.32.0 -- were you using that?
Does doing a dry run also exhibit the high CPU usage?

    certbot certonly --nginx -d example.com --dry-run

If it does, something that might help bisect this issue might be to try install different versions of Certbot and the nginx plugin [in a pip virtual environment](https://certbot.eff.org/instructions?ws=nginx&os=pip) and try run that above command. You'd have to amend the instructions to be something more like:

    sudo /opt/certbot/bin/pip install certbot==2.1.0 certbot-nginx==2.1.0

It would be helpful to know, which of the following releases the issue first appeared in:

- 2.0.0
- 2.1.0
- 2.3.0
> Sorry to hear that! Do you happen to know the exact certbot version that corresponds to your earlier logs? Prior to release 2.3.0 generally, the version was 1.32.0 -- were you using that?

Great question, I was able to get that info from the log files.

Previous version:
`2023-02-14 20:30:59,521:DEBUG:certbot._internal.main:certbot version: 1.32.2`

Current version:
`2023-02-15 05:56:04,506:DEBUG:certbot._internal.main:certbot version: 2.3.0`
> Does doing a dry run also exhibit the high CPU usage?
> 
> ```
> certbot certonly --nginx -d example.com --dry-run
> ```
> 

It does, here's the output of `top` when running a dry run:

```
top - 15:31:25 up 924 days, 13:05,  2 users,  load average: 1.27, 1.13, 1.20
Tasks: 135 total,   3 running,  87 sleeping,   0 stopped,   0 zombie
%Cpu(s): 98.3 us,  1.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.2 si,  0.5 st
KiB Mem :  4039260 total,   479264 free,  1623292 used,  1936704 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2061952 avail Mem 

  PID USER            PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
15093 [redacted]      20   0  519452 472364   9976 R  98.7 11.7  13197:28 python3
24295 [redacted]      20   0   96036  54628  15456 R  97.4  1.4   0:12.33 python3
```

I'll also mention that the command line output (not the log file):
```
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for [redacted]

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: [redacted]
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up A for [redacted] - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for [redacted] - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.
```

Between the `"Saving debug log"` and the next line `"Simulating a certificate request"` took what felt like ~30 seconds.

The entire dryrun process ran for over 4 minutes 45 seconds. After the `"Hint: The Certificate Authority..."` nothing was outputted for several minutes.

I will try to attempt the pip virtual environment approach you recommend and will report back, but wanted to pass along this datapoint in the meantime.

The other thing I'll mention is, the CPU spike alarms we're receiving seemingly only happen when adding new domains to our system. I haven't verified yet, but assuming the automatic renewals are still working, we're not getting CPU alerts when that process is running. I haven't dug through logs yet to determine how long each renewal is taking and if that is also askew. I haven't verified any of that yet so take it with a grain of salt and I may be incorrect.
I have a mild suspicion that my change in #9475 may have caused this performance regression. Checking that the issue is absent in Certbot v2.0.0 but present in v2.1.0 should confirm whether that's the case.
I'm currently stuck here and still trying to figure out (I'm not a python expert)

```
/opt/certbot/bin/pip install certbot==2.1.0 certbot-nginx==2.1.0
ERROR: Could not find a version that satisfies the requirement certbot==2.1.0 (from versions: 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.9.2, 0.9.3, 0.10.0, 0.10.1, 0.10.2, 0.11.0, 0.11.1, 0.12.0, 0.13.0, 0.14.0, 0.14.1, 0.14.2, 0.15.0, 0.16.0, 0.17.0, 0.18.0, 0.18.1, 0.18.2, 0.19.0, 0.20.0, 0.21.0, 0.21.1, 0.22.0, 0.22.1, 0.22.2, 0.23.0, 0.24.0, 0.25.0, 0.25.1, 0.26.0, 0.26.1, 0.27.0, 0.27.1, 0.28.0, 0.29.0, 0.29.1, 0.30.0, 0.30.1, 0.30.2, 0.31.0, 0.32.0, 0.33.0, 0.33.1, 0.34.0, 0.34.1, 0.34.2, 0.35.0, 0.35.1, 0.36.0, 0.37.0, 0.37.1, 0.37.2, 0.38.0, 0.39.0, 0.40.0, 0.40.1, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.8.0, 1.9.0, 1.10.0, 1.10.1, 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.21.0, 1.22.0, 1.23.0)
ERROR: No matching distribution found for certbot==2.1.0
```

```
/opt/certbot/bin/pip --version
pip 21.3.1 from /opt/certbot/lib/python3.6/site-packages/pip (python 3.6)
```

Seemingly cannot downgrade via `snap` either:
```
snap info certbot
name:      certbot
summary:   Automatically configure HTTPS using Let's Encrypt
publisher: Certbot Project (certbot-eff✓)
store-url: https://snapcraft.io/certbot
contact:   https://github.com/certbot/certbot/issues
license:   unset
description: |
  The objective of Certbot, Let's Encrypt, and the ACME (Automated
  Certificate Management Environment) protocol is to make it possible
  to set up an HTTPS server and have it automatically obtain a
  browser-trusted certificate, without any human intervention. This is
  accomplished by running a certificate management agent on the web
  server.
  
  This agent is used to:
    - Automatically prove to the Let's Encrypt CA that you control the website
    - Obtain a browser-trusted certificate and set it up on your web server
    - Keep track of when your certificate is going to expire, and renew it
    - Help you revoke the certificate if that ever becomes necessary.
commands:
  - certbot
services:
  certbot.renew: oneshot, enabled, inactive
snap-id:      wy7i66qPx4neXr6m9rTh7Y40h8EhtZFh
tracking:     latest/stable
refresh-date: 9 days ago, at 04:35 UTC
channels:
  latest/stable:    2.3.0      2023-02-14 (2772) 45MB classic
  latest/candidate: ↑                                 
  latest/beta:      2.3.0      2023-02-14 (2772) 45MB classic
  latest/edge:      2.4.0.dev0 2023-02-23 (2800) 45MB classic
installed:          2.3.0                 (2772) 45MB classic
```

My Python version may not be compatible
```
python3 --version
Python 3.6.9
```

I see running `pip install` in verbose mode, seeing errors like:
```
Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/8c/d6/ee08b79927971b029861e5b37f19b49e0343c5013f31698cc42a212debf5/certbot-2.0.0-py3-none-any.whl#sha256=9e6f16be2a8e007b041099b9b6bc4d92566bc40f1066053ab00afbf4e4ca80de (from https://pypi.org/simple/certbot/) (requires-python:>=3.7)
  Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/09/df/955c86b860b5da2a63a34e27326958a8864e68be141bac9136ae2822b93e/certbot-2.0.0.tar.gz#sha256=05a63c25d4a622dfb63b8d346cb97cba86d38ca6e7d71d2bbbc9223995731c13 (from https://pypi.org/simple/certbot/) (requires-python:>=3.7)
  Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/f7/32/a12e996d9b235e09ad2b72bde3557dfebbfbb99049d4ad248bdbf7c87511/certbot-2.1.0-py3-none-any.whl#sha256=0f7b7bf81bca57be9441dc90d42a589f844f7abc758ee4220ea1e3c6ccb3a4bf (from https://pypi.org/simple/certbot/) (requires-python:>=3.7)
  Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/25/92/9c48e2e9ad7f31d29c71b5b087cd97bfbaf1891f1ae2a88f8e99a930a40c/certbot-2.1.0.tar.gz#sha256=1f4409739f8454956963b4c78fc1c959db2bef85ea6fc6f08951d7275ec2595d (from https://pypi.org/simple/certbot/) (requires-python:>=3.7)
  Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/a1/b4/6ce3242684917998539fa738b96a61a81231a0cd5107a9d9c97e2f5f409e/certbot-2.2.0-py3-none-any.whl#sha256=6bbefdbf937f3e58d0d127d98b3ede5c616bfa34af664d432a82c0d5b55004ab (from https://pypi.org/simple/certbot/) (requires-python:>=3.7)
  Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/ea/2a/c84869ed98f10adace2f9ff026425104e26078c7339e8c8e751f5ae7790f/certbot-2.2.0.tar.gz#sha256=6d11bda7167d219ecba94e33907c65bfc0b2a3a3d783c525341a7b16d93ea6e9 (from https://pypi.org/simple/certbot/) (requires-python:>=3.7)
  Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/a9/e0/857515440a208b28957e0c02d45a9fe296d7c5aeac286b166f50cd5328e1/certbot-2.3.0-py3-none-any.whl#sha256=adcaa6ee6346fb890af034971d5da165acf421b9e79bd7bb284c61317209c1e3 (from https://pypi.org/simple/certbot/) (requires-python:>=3.7)
  Link requires a different Python (3.6.9 not in: '>=3.7'): https://files.pythonhosted.org/packages/d3/79/2ed6c08fc864e735221c14a62c5fc269988cb61955fcb24df8bffd0fffd1/certbot-2.3.0.tar.gz#sha256=93d39f0a9547b1251bf8101bd73f26dbaf3e0eb13448b8690f6e0dd9aa4dcd8a (from https://pypi.org/simple/certbot/) (requires-python:>=3.7)
```

Cannot seem to upgrade `python3` on our Ubuntu instance:
```
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3 is already the newest version (3.6.7-1~18.04).
```

Resolving patch:
diff --git a/certbot-nginx/certbot_nginx/_internal/configurator.py b/certbot-nginx/certbot_nginx/_internal/configurator.py
--- a/certbot-nginx/certbot_nginx/_internal/configurator.py
+++ b/certbot-nginx/certbot_nginx/_internal/configurator.py
@@ -1061,8 +1061,8 @@ def get_version(self) -> Tuple[int, ...]:
 
         product_name, product_version = version_matches[0]
         if product_name != 'nginx':
-            logger.warning("nginx derivative %s is not officially supported by "
-                           "Certbot.", product_name)
+            logger.warning("NGINX derivative %s is not officially supported by"
+                           " certbot", product_name)
 
         nginx_version = tuple(int(i) for i in product_version.split("."))
 
diff --git a/certbot-nginx/certbot_nginx/_internal/nginxparser.py b/certbot-nginx/certbot_nginx/_internal/nginxparser.py
--- a/certbot-nginx/certbot_nginx/_internal/nginxparser.py
+++ b/certbot-nginx/certbot_nginx/_internal/nginxparser.py
@@ -33,18 +33,6 @@
 logger = logging.getLogger(__name__)
 
 
-class UnsupportedDirectiveException(RuntimeError):
-    """Exception when encountering an nginx directive which is not supported
-    by this parser."""
-
-    directive_name: str
-    line_no: int
-
-    def __init__(self, directive_name: str, line_no: int) -> None:
-        self.directive_name = directive_name
-        self.line_no = line_no
-
-
 class RawNginxParser:
     # pylint: disable=pointless-statement
     """A class that parses nginx configuration with pyparsing."""
@@ -86,7 +74,6 @@ class RawNginxParser:
 
     def __init__(self, source: str) -> None:
         self.source = source
-        self.whitespace_token_group.addParseAction(self._check_disallowed_directive)
 
     def parse(self) -> ParseResults:
         """Returns the parsed tree."""
@@ -96,12 +83,6 @@ def as_list(self) -> List[Any]:
         """Returns the parsed tree as a list."""
         return self.parse().asList()
 
-    def _check_disallowed_directive(self, _source: str, line: int, results: ParseResults) -> None:
-        # *_by_lua_block might be first or second result, due to optional leading whitespace
-        toks = [t for t in results[0:2] if isinstance(t, str) and t.endswith("_by_lua_block")]
-        if toks:
-            raise UnsupportedDirectiveException(toks[0], line)
-
 
 class RawNginxDumper:
     """A class that dumps nginx configuration from the provided tree."""
diff --git a/certbot-nginx/certbot_nginx/_internal/parser.py b/certbot-nginx/certbot_nginx/_internal/parser.py
--- a/certbot-nginx/certbot_nginx/_internal/parser.py
+++ b/certbot-nginx/certbot_nginx/_internal/parser.py
@@ -223,12 +223,6 @@ def _parse_files(self, filepath: str, override: bool = False) -> List[UnspacedLi
                                "supported.", item)
             except pyparsing.ParseException as err:
                 logger.warning("Could not parse file: %s due to %s", item, err)
-            except nginxparser.UnsupportedDirectiveException as e:
-                logger.warning(
-                    "%s:%d contained the '%s' directive, which is not supported by Certbot. The "
-                    "file has been ignored, which may prevent Certbot from functioning properly. "
-                    "Consider using the --webroot plugin and manually installing the certificate.",
-                    item, e.line_no, e.directive_name)
         return trees
 
     def _find_config_root(self) -> str:


Test patch:
diff --git a/certbot-nginx/tests/nginxparser_test.py b/certbot-nginx/tests/nginxparser_test.py
--- a/certbot-nginx/tests/nginxparser_test.py
+++ b/certbot-nginx/tests/nginxparser_test.py
@@ -14,7 +14,6 @@
 from certbot_nginx._internal.nginxparser import loads
 from certbot_nginx._internal.nginxparser import RawNginxParser
 from certbot_nginx._internal.nginxparser import UnspacedList
-from certbot_nginx._internal.nginxparser import UnsupportedDirectiveException
 import test_util as util
 
 FIRST = operator.itemgetter(0)
@@ -357,68 +356,6 @@ def test_edge_cases(self):
         parsed = loads("")
         assert parsed == []
 
-    def test_lua(self):
-        # https://github.com/certbot/certbot/issues/9066
-        with pytest.raises(UnsupportedDirectiveException):
-            loads("""
-        location /foo {
-            content_by_lua_block {
-                ngx.say('Hello World')
-            }
-        }
-        """)
-
-        # Without leading whitespace
-        with pytest.raises(UnsupportedDirectiveException):
-            loads("""
-        location /foo {content_by_lua_block {
-                ngx.say('Hello World')
-            }
-        }
-        """)
-
-        # Doesn't trigger if it's commented or not in the right position.
-        parsed = loads("""
-        location /foo {server_name content_by_lua_block;
-            #content_by_lua_block {
-            # ngx.say('Hello World')
-            # }
-        }
-        """)
-        assert parsed == \
-            [
-                [['location', '/foo'],
-                [['server_name', 'content_by_lua_block'],
-                 ['#', 'content_by_lua_block {'],
-                 ['#', " ngx.say('Hello World')"],
-                 ['#', ' }']
-                ]]
-            ]
-
-        # *_by_lua should parse successfully.
-        parsed = loads("""
-            location / {
-                set $a 32;
-                set $b 56;
-                set_by_lua $sum
-                    'return tonumber(ngx.arg[1]) + tonumber(ngx.arg[2])'
-                    $a $b;
-                content_by_lua '
-                    ngx.say("foo");
-                ';
-            }
-        """)
-        assert parsed == \
-            [
-                [['location', '/'],
-                 [['set', '$a', '32'],
-                  ['set', '$b', '56'],
-                  ['set_by_lua', '$sum',
-                   "'return tonumber(ngx.arg[1]) + tonumber(ngx.arg[2])'", '$a', '$b'
-                  ],
-                  ['content_by_lua', '\'\n                    ngx.say("foo");\n                \'']
-                 ]]
-            ]
 
 class TestUnspacedList(unittest.TestCase):
     """Test the UnspacedList data structure"""
diff --git a/certbot-nginx/tests/parser_test.py b/certbot-nginx/tests/parser_test.py
--- a/certbot-nginx/tests/parser_test.py
+++ b/certbot-nginx/tests/parser_test.py
@@ -4,8 +4,6 @@
 import shutil
 import sys
 from typing import List
-import unittest
-from unittest import mock
 
 import pytest
 
@@ -533,14 +531,6 @@ def test_invalid_unicode_characters_in_ssl_options(self):
             for output in log.output
         )
 
-    @mock.patch('certbot_nginx._internal.parser.logger.warning')
-    def test_load_unsupported_directive_logged(self, mock_warn):
-        nparser = parser.NginxParser(self.config_path)
-        nparser.config_root = nparser.abs_path('unsupported_directives.conf')
-        nparser.load()
-        assert mock_warn.call_count == 1
-        assert "which is not supported by Certbot" in mock_warn.call_args[0][0]
-
 
 if __name__ == "__main__":
     sys.exit(pytest.main(sys.argv[1:] + [__file__]))  # pragma: no cover
diff --git a/certbot-nginx/tests/testdata/etc_nginx/unsupported_directives.conf b/certbot-nginx/tests/testdata/etc_nginx/unsupported_directives.conf
deleted file mode 100644
--- a/certbot-nginx/tests/testdata/etc_nginx/unsupported_directives.conf
+++ /dev/null
@@ -1,11 +0,0 @@
-# This configuration file contains unsupported direcives.
-
-server {
-    listen 80;
-
-    location / {
-        foobar_by_lua_block {
-            ngx.say("Hello World")
-        }
-    }
-}