Repository: certbot/certbot

Issue:
Followups from disabling build isolation from pip
In #8255 which fixed https://github.com/certbot/certbot/issues/8252, we disabled pip's build isolation used when building wheels from sources with a `pyproject.toml` file. My understanding is that the main downside of this change is that we are now responsible for ensuring that the dependencies a project specifies in their `pyproject.toml` file following [PEP 518](https://www.python.org/dev/peps/pep-0518/#specification) are already installed in the build environment before trying to install that package. (There are some other minor downsides such as being unable to install packages with conflicting build requirements discussed at https://www.python.org/dev/peps/pep-0517/#recommendations-for-build-frontends-non-normative, but I'm not particularly worried about any of them in the context of Certbot right now). My first question here is this understanding correct? Are there any other problems I am not aware of?

As a test of that understanding, #8255 says:

> It may however be problematic if some libraries require another build system than `setuptools` and do not provide a fallback to a `setuptools` build. For the record, `dns-lexicon`, that I maintain, uses `poetry` and so a PEP 517 compliant definition of a build system, but provides also this fallback (https://github.com/AnalogJ/lexicon/blob/master/setup.py).

If `dns-lexicon` or another project removed their `setup.py` file and we couldn't use their wheel, this approach of turning off pip's build isolation should continue to work as long as we have any dependencies, such as `poetry`, installed in the build environment correct?

If I'm understanding things correctly, my next question is if there is future work to be done here. Should we be disabling build isolation in other locations such as the DNS plugin snaps, Dockerfiles, CI config, or our scripts in `tools` such as the `venv*.py` scripts or the release script? How will we ensure that build dependencies continue to be properly satisfied as we update the pinnings of our dependencies? Do we want to make a request or start a conversation with any of our dependencies about improvements here?

Two ideas for the last question so far are:

1. Asking all of our dependencies to pin exact versions of their PEP 518 style build dependencies.
2. Ask `pip` for a way to have it continue to install the PEP 518 style build dependencies for us with the ability to further constrain those dependencies with a constraints file. These dependencies could be installed either in an isolated environment or the normal installation environment, although I think the former is preferred because it'd allow us to pin tools like `poetry` without unnecessarily adding it to our installation (or manually removing it after installing any packages that need it to build).


Discussion:
Another thing we may want to think about is if we want to do anything extra to control the order our dependencies are installed. For instance, how can we make sure `cffi` is installed before `cryptography` is built in the Certbot snap?
> 
> 
> In #8255 which fixed #8252, we disabled pip's build isolation used when building wheels from sources with a `pyproject.toml` file. My understanding is that the main downside of this change is that we are now responsible for ensuring that the dependencies a project specifies in their `pyproject.toml` file following [PEP 518](https://www.python.org/dev/peps/pep-0518/#specification) are already installed in the build environment before trying to install that package. (There are some other minor downsides such as being unable to install packages with conflicting build requirements discussed at https://www.python.org/dev/peps/pep-0517/#recommendations-for-build-frontends-non-normative, but I'm not particularly worried about any of them in the context of Certbot right now). My first question here is this understanding correct? Are there any other problems I am not aware of?

Yes, we are responsible for installing whatever build dependency a given python library is about to be installed. I will note the three following significant behavior with pip:
* for projects without `pyproject.toml`, pip will assume a setuptools approach and use this
* for projects with `pyproject.toml` but without `[build-system]` section, pip will also assume a setuptools approach and use this
* for editable installations, pip require a setuptools approach, so a `setup.py` shim as set in `dns-lexicon` is required for now

> 
> As a test of that understanding, #8255 says:
> 
> > It may however be problematic if some libraries require another build system than `setuptools` and do not provide a fallback to a `setuptools` build. For the record, `dns-lexicon`, that I maintain, uses `poetry` and so a PEP 517 compliant definition of a build system, but provides also this fallback (https://github.com/AnalogJ/lexicon/blob/master/setup.py).
> 
> If `dns-lexicon` or another project removed their `setup.py` file and we couldn't use their wheel, this approach of turning off pip's build isolation should continue to work as long as we have any dependencies, such as `poetry`, installed in the build environment correct?

Indeed. In the situation of a project using `poetry` as its build system, and without providing the `setup.py` shim, a call to `pip install --no-isolation-build` package will output the following kind of error if `poetry` is not installed beforehand:
```
Processing /home/aferrand/Dev/lexicon
    Preparing wheel metadata ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 357, in run
    resolver.resolve(requirement_set)
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 564, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 549, in _generate_metadata
    return generate_metadata(
  File "/home/aferrand/Dev/lexicon/venv/lib/python3.8/site-packages/pip/_internal/operations/build/metadata.py", line 36, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/home/aferrand/Dev/lexicon/venv/share/python-wheels/pep517-0.8.2-py2.py3-none-any.whl/pep517/wrappers.py", line 176, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/home/aferrand/Dev/lexicon/venv/share/python-wheels/pep517-0.8.2-py2.py3-none-any.whl/pep517/wrappers.py", line 265, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
  File "/tmp/tmp67ui7r33", line 86, in _build_backend
    obj = import_module(mod_path)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'poetry'
```

> 
> If I'm understanding things correctly, my next question is if there is future work to be done here. Should we be disabling build isolation in other locations such as the DNS plugin snaps, Dockerfiles, CI config, or our scripts in `tools` such as the `venv*.py` scripts or the release script? How will we ensure that build dependencies continue to be properly satisfied as we update the pinnings of our dependencies? Do we want to make a request or start a conversation with any of our dependencies about improvements here?

I think that in the long term, avoiding the build isolation is fighting against a really decent design decision. But it requires involvement from the Python libraries maintainers (as it goes for wheels, that PyPA strongly recommends to build to help distributing Python projects).

I believe that the sane pinning dependency logic that is applied on Certbot is also applicable on build system for Python libraries. Having a global failure of building the sdist you provide is something that you want to avoid at all cost as a library maintainer. Given that PEP 517/518 actually provide a (almost) complete isolation of the build system runtime, it makes really sense in my opinion to always pin all build dependencies, and validate the versions bumping on each release of your library with proper integration tests.

On that matter I think I will propose a PR to `cryptography` to fix its `setuptools` version.

> 
> Two ideas for the last question so far are:
> 
>     1. Asking all of our dependencies to pin exact versions of their PEP 518 style build dependencies.
> 
>     2. Ask `pip` for a way to have it continue to install the PEP 518 style build dependencies for us with the ability to further constrain those dependencies with a constraints file. These dependencies could be installed either in an isolated environment or the normal installation environment, although I think the former is preferred because it'd allow us to pin tools like `poetry` without unnecessarily adding it to our installation (or manually removing it after installing any packages that need it to build).

Given what I said previously, we should start with 1, since this effort goes in the direction of a global improvement of the building and distribution problematic of libraries in Python. As of, 2. could work by applying a constraint on a range of dependencies, but following 1., these kind of constraints would mostly be incompatible since the versions of the build system would be pinned by the libraries maintainers.

> 
> 
> Another thing we may want to think about is if we want to do anything extra to control the order our dependencies are installed. For instance, how can we make sure `cffi` is installed before `cryptography` is built in the Certbot snap?

Given the accumulated modifications done on our `snapcraft.yaml` to mitigate downsides of the opiniated build approach provided by the python snap plugin, I am more and more tempted to take a radical approach, and override the whole build phase with our own build logic, that would be consistent through specific integration tests on our side, and reusable on various situations (snaps, windows installer, bare installation in venv).

In particular, the whole problem here came from the need to have `wheel` installed in the venv that will build the snap, and that we solved by using `--system-site-packages` because there were no other way to tweak the venv before pip install kicks in. We have a strong experience in building highly stable and reusable virtual environments with `certbot-auto` and development environments, and maybe we should take advantage of it in the snap building process.
> On that matter I think I will propose a PR to `cryptography` to fix its `setuptools` version.

If you do that, please drop a link here as I would be interested to see how that conversation goes. Will `dns-lexicon` also be pinning an exact version of `poetry` at https://github.com/AnalogJ/lexicon/blob/v3.4.1/pyproject.toml#L3 in future releases?

While I think I understand your thinking here, I personally think we'll be hard pressed to get all of our dependencies to lock things down like this. I think pinning things down exactly is annoying as a package maintainer as you now have to deal with things like if `poetry`/`setuptools` fix a build problem on someone's machine in a new release, you'll be getting people bugging you to do a new release of your project just to update your pinnings. I think it also means more work for us in the long term because we have to continue to monitor all of our dependencies for problems here whenever we update our pinnings if we want to be sure to avoid problems like we had in https://github.com/certbot/certbot/issues/8252. Finally, we'll be forced to fall back to `--no-build-isolation` if we cannot convince all of them to make this change.

I personally like the 2nd idea I listed in my prior post better. I think it makes things easier for package maintainers while allowing people to pin things down in a way that works in their specific build environment rather than the package maintainer trying to come up with a pinning that works everywhere. It also gives us more control because it allows us to fix the problem on our end once and for all without changes to our dependencies (outside of `pip`). I think `pip` in theory should be able to make use of its new dependency resolution to ensure that both the package's build requirements and the user's further constraints on them are satisfied.

> I am more and more tempted to take a radical approach, and override the whole build phase with our own build logic

I'd still like to avoid this if we can. It's more code for us to maintain and I think in theory the more customization we do, the more potential problems we'll hit that others have not encountered before. I'd personally prefer we try to improve the Python plugin upstream to fit our need here.
In general I am not sure yet to pin `poetry` in the setup of `dns-lexicon`. As of now, before opening a PR or something, I now think I should gather some opinion around how build dependencies should be handled in the scope of PEP 517-518, and maybe the best good practices have not appeared yet, and this feature needs more time to become mature.

At that point, I would like to note that pinning build dependencies should not create a burden on user side: since the build is done in isolation, the specific version of `poetry` or `setuptools` is used only for the package that requires it, and is independant from the actual runtime of the modules that will be executed. It is true that if a new version of build packages fixes some problems for people running specific environments, then the maintainer would need ultimately to update these pinned dependencies to solve that, but I am not sure that it is significant compared to runtime problem you can encounter with regular dependencies (the ones for the runtime itself).

At last for the complete override of build command during the snap build, I take in fact the specific design of Python plugin v2, which is to follow closely what a Python developer would do with a virtual environment. I think that if we keep that, we stays on the expected design. Also note that on Python plugin v2, the actual logic hold by the plugin is very limited (create venv + pip install), but the few operations done are also really constraint, since you basically cannot hook into the process. I agree that improving the Python plugin itself would be the best, but I keep in mind the trade-off done here.
I think starting a broader conversation makes sense. If you want to start that, the way I would recommend doing it is posting to the [distutils-sig](https://mail.python.org/mailman3/lists/distutils-sig.python.org/) mailing list. I would frame the conversation as how the Python community should pin down build dependencies for consistency and reliability. Again, if you start that thread, please post a link here.

It's true that your approach pushes the burden of pinning things down properly onto the package maintainer, but what I'm saying is that I think many package maintainers won't want to accept that burden. I also think that a pip flag like `--build-constraints` gives users another tool to deal with these problems rather than setting `--no-build-isolation` which causes the package's build dependencies to not be checked at all and therefore should be a last resort in my opinion.

Until changes are made upstream in some way though, I personally think we should fix things in places other than the Certbot snap to ensure we don't hit problems like https://github.com/certbot/certbot/issues/8252 in the future. If you have ideas on how to do that, please share!
Indeed maybe a flag like `--build-constraints` could to it. Anyway I will start the discussion and put a link here.

About fixing other places, well, it is not technically a fix, but it cannot happen for now in places where certbot support officially an installation method, because the virtual environment is each time pinned with pipstrap to an old version of pip (9.x) that does not support isolation builds.

It is to keep in mind however if we need and decide to upgrade the pinned version of pip in the future. 
That's good to know it's not a problem where we use pipstrap. In that case, I think we should look for the places we don't use pipstrap and either start using it or set `--no-build-isolation`. This is not a complete list, but I think we don't use pipstrap in places like our DNS plugin snaps, CI, and tox.

I also think we should add a comment to `pipstrap` flagging that we need to be careful about upgrading `pip` and reference these issues.
I think I just hit a similar problem here in our oldest tests when trying to update one of our dependencies. See https://github.com/certbot/certbot/pull/8289.
The remaining TODO list here in my opinion is:

- [ ] fix this problem in our DNS plugin snaps (attempted in https://github.com/certbot/certbot/pull/8319, but that had to be reverted)
- [ ] fix this problem in our release script
- [ ] double check that there aren't more locations that I missed
- [ ] start a broader conversation with pip about how this problem should be handled in general or create an issue for us to track doing so
Since we reverted https://github.com/certbot/certbot/pull/8319 for now, I added it back to the TODO list. I wrote about the approach I personally think is probably the best to try and fix it in https://github.com/certbot/certbot/pull/8323#issue-494352977.

~~If you work on this again, please don't test in Certbot's CI until you expect it to work because it generates a ton of failure emails. I'm unfortunately not aware of a good way to prevent this.~~ I've filtered these emails a bit so people should free to test in with our CI.

Again, I'm happy to get you an ARM machine to play with temporarily if you want.

Resolving patch:
diff --git a/tools/pip_install.py b/tools/pip_install.py
--- a/tools/pip_install.py
+++ b/tools/pip_install.py
@@ -59,9 +59,13 @@ def certbot_normal_processing(tools_path, test_constraints):
     certbot_requirements = os.path.normpath(os.path.join(
         repo_path, 'letsencrypt-auto-source/pieces/dependency-requirements.txt'))
     with open(certbot_requirements, 'r') as fd:
-        data = fd.readlines()
+        certbot_reqs = fd.readlines()
+    with open(os.path.join(tools_path, 'pipstrap_constraints.txt'), 'r') as fd:
+        pipstrap_reqs = fd.readlines()
     with open(test_constraints, 'w') as fd:
-        data = "\n".join(strip_hashes.process_entries(data))
+        data_certbot = "\n".join(strip_hashes.process_entries(certbot_reqs))
+        data_pipstrap = "\n".join(strip_hashes.process_entries(pipstrap_reqs))
+        data = "\n".join([data_certbot, data_pipstrap])
         fd.write(data)
 
 
@@ -72,7 +76,8 @@ def merge_requirements(tools_path, requirements, test_constraints, all_constrain
     # Here is the order by increasing priority:
     # 1) The general development constraints (tools/dev_constraints.txt)
     # 2) The general tests constraints (oldest_requirements.txt or
-    #    certbot-auto's dependency-requirements.txt for the normal processing)
+    #    certbot-auto's dependency-requirements.txt + pipstrap's constraints
+    #    for the normal processing)
     # 3) The local requirement file, typically local-oldest-requirement in oldest tests
     files = [os.path.join(tools_path, 'dev_constraints.txt'), test_constraints]
     if requirements:
@@ -82,17 +87,18 @@ def merge_requirements(tools_path, requirements, test_constraints, all_constrain
         fd.write(merged_requirements)
 
 
-def call_with_print(command):
+def call_with_print(command, env=None):
+    if not env:
+        env = os.environ
     print(command)
-    subprocess.check_call(command, shell=True)
+    subprocess.check_call(command, shell=True, env=env)
 
 
-def pip_install_with_print(args_str, disable_build_isolation=True):
-    command = ['"', sys.executable, '" -m pip install --disable-pip-version-check ']
-    if disable_build_isolation:
-        command.append('--no-build-isolation ')
-    command.append(args_str)
-    call_with_print(''.join(command))
+def pip_install_with_print(args_str, env=None):
+    if not env:
+        env = os.environ
+    command = ['"', sys.executable, '" -m pip install --disable-pip-version-check ', args_str]
+    call_with_print(''.join(command), env=env)
 
 
 def main(args):
@@ -113,20 +119,22 @@ def main(args):
             else:
                 certbot_normal_processing(tools_path, test_constraints)
 
+            env = os.environ.copy()
+            env["PIP_CONSTRAINT"] = all_constraints
+
             merge_requirements(tools_path, requirements, test_constraints, all_constraints)
             if requirements:  # This branch is executed during the oldest tests
                 # First step, install the transitive dependencies of oldest requirements
                 # in respect with oldest constraints.
-                pip_install_with_print('--constraint "{0}" --requirement "{1}"'
-                                       .format(all_constraints, requirements))
+                pip_install_with_print('--requirement "{0}"'.format(requirements),
+                                       env=env)
                 # Second step, ensure that oldest requirements themselves are effectively
                 # installed using --force-reinstall, and avoid corner cases like the one described
                 # in https://github.com/certbot/certbot/issues/7014.
                 pip_install_with_print('--force-reinstall --no-deps --requirement "{0}"'
                                        .format(requirements))
 
-            pip_install_with_print('--constraint "{0}" {1}'.format(
-                all_constraints, ' '.join(args)))
+            pip_install_with_print(' '.join(args), env=env)
 
 
 if __name__ == '__main__':
diff --git a/tools/pipstrap.py b/tools/pipstrap.py
--- a/tools/pipstrap.py
+++ b/tools/pipstrap.py
@@ -1,46 +1,17 @@
 #!/usr/bin/env python
 """Uses pip to upgrade Python packaging tools to pinned versions."""
 from __future__ import absolute_import
-
 import os
-import shutil
-import tempfile
 
 import pip_install
 
 
-# We include the hashes of the packages here for extra verification of
-# the packages downloaded from PyPI. This is especially valuable in our
-# builds of Certbot that we ship to our users such as our Docker images.
-#
-# An older version of setuptools is currently used here in order to keep
-# compatibility with Python 2 since newer versions of setuptools have dropped
-# support for it.
-REQUIREMENTS = r"""
-pip==20.2.4 \
-    --hash=sha256:51f1c7514530bd5c145d8f13ed936ad6b8bfcb8cf74e10403d0890bc986f0033 \
-    --hash=sha256:85c99a857ea0fb0aedf23833d9be5c40cf253fe24443f0829c7b472e23c364a1
-setuptools==44.1.1 \
-    --hash=sha256:27a714c09253134e60a6fa68130f78c7037e5562c4f21f8f318f2ae900d152d5 \
-    --hash=sha256:c67aa55db532a0dadc4d2e20ba9961cbd3ccc84d544e9029699822542b5a476b
-wheel==0.35.1 \
-    --hash=sha256:497add53525d16c173c2c1c733b8f655510e909ea78cc0e29d374243544b77a2 \
-    --hash=sha256:99a22d87add3f634ff917310a3d87e499f19e663413a52eb9232c447aa646c9f
-"""
+_REQUIREMENTS_PATH = os.path.join(os.path.dirname(__file__), "pipstrap_constraints.txt")
 
 
 def main():
-    with pip_install.temporary_directory() as tempdir:
-        requirements_filepath = os.path.join(tempdir, 'reqs.txt')
-        with open(requirements_filepath, 'w') as f:
-            f.write(REQUIREMENTS)
-        pip_install_args = '--requirement ' + requirements_filepath
-        # We don't disable build isolation because we may have an older
-        # version of pip that doesn't support the flag disabling it. We
-        # expect these packages to already have usable wheels available
-        # anyway so no building should be required.
-        pip_install.pip_install_with_print(pip_install_args,
-                                           disable_build_isolation=False)
+    pip_install_args = '--requirement "{0}"'.format(_REQUIREMENTS_PATH)
+    pip_install.pip_install_with_print(pip_install_args)
 
 
 if __name__ == '__main__':
diff --git a/windows-installer/construct.py b/windows-installer/construct.py
--- a/windows-installer/construct.py
+++ b/windows-installer/construct.py
@@ -46,9 +46,11 @@ def _compile_wheels(repo_path, build_path, venv_python):
     wheels_project = [os.path.join(repo_path, package) for package in certbot_packages]
 
     with _prepare_constraints(repo_path) as constraints_file_path:
-        command = [venv_python, '-m', 'pip', 'wheel', '-w', wheels_path, '--constraint', constraints_file_path]
+        env = os.environ.copy()
+        env['PIP_CONSTRAINT'] = constraints_file_path
+        command = [venv_python, '-m', 'pip', 'wheel', '-w', wheels_path]
         command.extend(wheels_project)
-        subprocess.check_call(command)
+        subprocess.check_call(command, env=env)
 
 
 def _prepare_build_tools(venv_path, venv_python, repo_path):
@@ -61,15 +63,20 @@ def _prepare_build_tools(venv_path, venv_python, repo_path):
 
 @contextlib.contextmanager
 def _prepare_constraints(repo_path):
-    requirements = os.path.join(repo_path, 'letsencrypt-auto-source', 'pieces', 'dependency-requirements.txt')
-    constraints = subprocess.check_output(
-        [sys.executable, os.path.join(repo_path, 'tools', 'strip_hashes.py'), requirements],
+    reqs_certbot = os.path.join(repo_path, 'letsencrypt-auto-source', 'pieces', 'dependency-requirements.txt')
+    reqs_pipstrap = os.path.join(repo_path, 'tools', 'pipstrap_constraints.txt')
+    constraints_certbot = subprocess.check_output(
+        [sys.executable, os.path.join(repo_path, 'tools', 'strip_hashes.py'), reqs_certbot],
+        universal_newlines=True)
+    constraints_pipstrap = subprocess.check_output(
+        [sys.executable, os.path.join(repo_path, 'tools', 'strip_hashes.py'), reqs_pipstrap],
         universal_newlines=True)
     workdir = tempfile.mkdtemp()
     try:
         constraints_file_path = os.path.join(workdir, 'constraints.txt')
         with open(constraints_file_path, 'a') as file_h:
-            file_h.write(constraints)
+            file_h.write(constraints_pipstrap)
+            file_h.write(constraints_certbot)
             file_h.write('pywin32=={0}'.format(PYWIN32_VERSION))
         yield constraints_file_path
     finally:


Test patch: