Repository: certbot/certbot

Issue:
Should we pin dependency versions?
Currently we pin specific versions of all our of Python dependencies in `certbot-auto` but don't do this in our PyPI packages. Should we change this? I'd be very interested to see what other popular client side Python projects with OS dependencies do. (I'm not personally aware of any and over the past 2 years I've learned why.)

There was discussion of this in IRC between @erikrose, @ohemorange, and @zjs, but I didn't get a chance to respond while everyone was still in the channel. Thinking about this for a bit, I think I'm against pinning while we have our current distribution methods, but let's see where this helps us.

#### certbot-auto

We already pin specific dependency versions and check its hash.

#### OS packages

While we're currently relying and encouraging systems to package Certbot and for people to use those packages, pinning specific versions of our dependencies isn't feasible. While it'd be nice to only support only one version of our dependencies at a time, the packages available in different distros vary wildly and we need to support multiple versions. Pinning specific versions makes testing different package versions harder for us (see [tox.ini](https://github.com/certbot/certbot/blob/master/tox.ini#L37) and #4049) and makes OS packaging harder. I see no benefit here.

Pinning `dependency<next.major.version` is interesting. This makes it clear that we don't expect Certbot to work with new major versions of our dependencies. If Certbot or its tests are run with a new dependency version though, it'll definitely fail if we pin, but it may not if we don't. Which behavior do we want?

#### pip install

We never tell anyone to install this way and I am perfectly fine not supporting this installation method as was suggested in IRC. In general, I think Python does a poor job supporting `pip` installing packages with OS dependencies which we have many of. Unless you have an intimate understanding of the project and understand you need to install `libffi-dev`, `gcc`, `libaugeas`, and more (or luckily happen to have those packages installed), `pip install` isn't going to work.

#### developer setup

`tools/venv.sh` failing slows down developers, but:

1. If this is our primary motivation for this, we can fix it without affecting packaging.
2. It's beneficial for us to know when a dependency changes that breaks Certbot (although we could test this during CI or nightly testing).


Discussion:
**OS packages**

This can't be an unsolved problem. If it's standard or standard-ish to pin `dependency<next.major.version`, but major versions get updated all the time, no way do things just always stop working when the next version of the dependency comes out. @erikrose?

**pip install**

Yeah, I'm fine with not bothering for this one.

**developer setup**

We should definitely not have broken dependencies break the developer workflow, especially with so many people actively developing Certbot.

CI, nightly testing, or even regular manual removing of conservative pins could all work here.
#### OS packages

> If it's standard or standard-ish to pin dependency<next.major.version, but major versions get updated all the time, no way do things just always stop working when the next version of the dependency comes out.

Major versions come out seldom for most Python packages. `click` (which we don't use) is about the only one I can think of that revs all the time, but that's because Armin is strict about semver and click is still young. I think we release certbot often enough that we can broaden our pinnings as new major versions are found to work.

#### pip install
I agree that we should say "you're on your own" to people using `pip install`. In fact, I think I was the one who suggested it on IRC. Ordinarily, `pip install` shoud be fine for dev, but our tightly coupled multi-package architecture rules that out.

#### developer setup
> We should definitely not have broken dependencies break the developer workflow,

As far as development goes, I think `venv.sh` is fine. We just need to mention it conspicuously. Deps pinned at `whatever.we.need ≤ us < next.major.version` should provide the latitude to pull in upstream bug fixes while not foisting the debugging of new major versions on every random contributor.
Note for posterity: in my mind, the primary motivation for pinning Python dependencies < some_major_version is so venv.sh provides a more reliably working env for (new) devs.
@erikrose, if the primary motivation is developer setup, why not do this in a way that doesn't affect packaging? We could do things like have `tools/venv.sh` use the `certbot-auto` requirements file and then `pip install` certbot in editable mode for example. I'd be open to other ideas too, but I'm not convinced this is beneficial for any packaging efforts and we can accomplish this for devs without affecting packaging.
The secondary motivation for this is to address the devs who will inevitably use `pip install -e .` to set up their dev env, ignoring the documentation. Tertiarily: people using certbot as a lib.

But it's fine with me if you want to close this.

If I were architecting everything over from the ground up, I'd merge all our tightly coupled packages (certbot-apache, certbot-nginx, etc.) and save a bunch of release paperwork. acme, which is actually usable on its own, would remain separate, and it would follow semver, getting a new major version when its network API changed, at minimum. Then we really could rely on pip rather than needing to have venv.sh and explain it to people, as package install order would no longer matter. I have a moderate allergy to custom tooling. :-)
> The secondary motivation for this is to address the devs who will inevitably use pip install -e . to set up their dev env, ignoring the documentation.

Those people also need to know about our OS dependencies though and because of this I unfortunately don't think Certbot is set up very well to support this use case.

> Tertiarily: people using certbot as a lib.

I believe this only affects them if they're using `pip install` which we agreed we're not worried about. Perhaps we should think differently about this case or offer a better way to install Certbot for people using it as a lib where it's not packaged though.

I think we should do something here. To build on what I said about OS packages before for @ohemorange's benefit, if we say we require `python-augeas<=0.5.0` and only a newer version is available, Certbot will immediately crash saying the right dependencies aren't available. If we don't pin, Certbot will at least try to work with the new dependency. I'm not aware of any client side Python projects that pin their dependencies like this. I asked Harlan how this would work in Debian and he said that a newer version of our dependency would be uploaded without being tested with a version of Certbot.

With this in mind, I don't think pinning helps OS packages and actually creates additional problems for them. Pinning also doesn't help any of our other distribution methods (that we said we cared about) so I don't think our solution here should affect packaging.

Proposal: Use the `letsencrypt-auto` requirements file to install deps in `venv.sh`, and the relevant `tox` tests. We can either modify the requirements file and/or the release process so the latest released version of Certbot isn't installed here or we can just install latest released version and overwrite it with the local packages. I'll also probably create a nightly test that tests with the latest packages so we can be aware of breakage as it occurs without slowing down development.

Thoughts?

cc @ohemorange 
> I asked Harlan how this would work in Debian and he said that a newer version of our dependency would be uploaded without being tested with a version of Certbot.

Would be uploaded where? As in, if that gets pushed, it'll just go forward regardless of any of our pinning info?
To further explain as well as give a concrete example, if Certbot depends on `python-cryptography` and someone wants to upload a new version of `python-cryptography`, it is uploaded and released in the Debian repositories without checking that the new version of `python-cryptography` works with Certbot. If a user upgrades their system (or installs Certbot for the first time) they will get this new version of `python-crypytography` and the old version is removed if it was installed.

If we pinned an older version of cryptography, Certbot is broken because it will immediately crash saying the required version of `python-cryptography` isn't available. If we don't pin, Certbot will try to work with the new version and error out as normal if a problem occurs.
So, to confirm because this surprises me, it doesn't keep around multiple versions of OS packages as needed to resolve dependencies in the system? There's no mechanism for that at all? It just blows away old versions and lets things break?
Correct. Linux distros in general only have one version of a package installed a time. There are exceptions to this such as `python2` and `python3` or Gentoo with its [slots](https://devmanual.gentoo.org/general-concepts/slotting/) system, but this isn't generally done and updating a package removes previously installed versions of it.

EDIT: To add more to explain the distros behavior, the way they work around the problem is not constantly taking updates from upstream. They take a set of packages and test/remove them until they believe the set is stable. Then they release this set and promise to maintain security/bug fixes for them for a number of years. This is how most distros work.

Obviously rolling release distros like Debian sid, Arch Linux, or Gentoo work differently, but at least in the case of Debian sid, Harlan said a new dependency would be uploaded (removing the old package) without testing its compatibility with Certbot.
@bmw Good point about devs needing OS-packaged dependencies. We could probably get rid of Augeus, but we'll never get rid of openssl.

> Linux distros in general only have one version of a package installed a time.

In Debian, the convention is to add a number to the end of a library name when backward compatibility breaks: libfoo1, libfoo2. You can have 1 specific version of each major version installed at once.

So Debian's Python packaging subsystem, if it indeed not equipped to handle different major versions of a Python package at once, is a bit of a departure from how shared libs have long been handled. I expect this is due to Python's own deficient namespacing, where requests 1.0 and requests 2.0 are both imported as `requests`.

Having venv.sh use the le-auto pinnings seems like a reasonable alternative approach to me.
Interesting. I didn't know that Debian offered multiple versions of a library at the same time in the general case. I knew there were exceptions, but didn't realize this was a normal policy. Good job Debian.

I have yet see a distro do this for Python dependencies though. Sure many distros offer a Python 2 and a Python 3 version of a package, but I've never seen a distro offer multiple versions for the same Python version.

I'm happy to write the PR to use requirements in `venv.sh`, but I want to make sure @ohemorange is OK with this approach due to her interest here.
Sounds like there's unlikely to be a perfect answer that never breaks anything here, so that works for me.

Resolving patch:
diff --git a/certbot-apache/setup.py b/certbot-apache/setup.py
--- a/certbot-apache/setup.py
+++ b/certbot-apache/setup.py
@@ -11,7 +11,7 @@
     'acme=={0}'.format(version),
     'certbot=={0}'.format(version),
     'mock',
-    'python-augeas<=0.5.0',
+    'python-augeas',
     # For pkg_resources. >=1.0 so pip resolves it to a version cryptography
     # will tolerate; see #2599:
     'setuptools>=1.0',
diff --git a/letsencrypt-auto-source/build.py b/letsencrypt-auto-source/build.py
--- a/letsencrypt-auto-source/build.py
+++ b/letsencrypt-auto-source/build.py
@@ -21,14 +21,17 @@ def build(version=None, requirements=None):
     :arg version: The version to attach to the script. Default: the version of
         the certbot package
     :arg requirements: The contents of the requirements file to embed. Default:
-        contents of letsencrypt-auto-requirements.txt
+        contents of dependency-requirements.txt, letsencrypt-requirements.txt,
+        and certbot-requirements.txt
 
     """
     special_replacements = {
         'LE_AUTO_VERSION': version or certbot_version(DIR)
     }
     if requirements:
-        special_replacements['letsencrypt-auto-requirements.txt'] = requirements
+        special_replacements['dependency-requirements.txt'] = ''
+        special_replacements['letsencrypt-requirements.txt'] = ''
+        special_replacements['certbot-requirements.txt'] = requirements
 
     def replacer(match):
         token = match.group(1)


Test patch: