Repository: certbot/certbot

Issue:
Allow DNS plugin snaps to build if they rely on unreleased acme/certbot
One problem I think we're going to have with DNS plugin snaps is if the DNS plugin depends on an unreleased version of `acme` or `certbot`, we will be unable to build the snap until our next release. We had this problem in the past in the Certbot snap as described at https://github.com/basak/certbot-snap-build/issues/12.

I think fixing this problem is important because not being able to test changes until its time for a release is how we end up wasting a lot of time during the release and creating point releases.

I think fixing this will be tricky though. We may be able to tackle this problem at the same time as https://github.com/certbot/certbot/issues/7946. If acme and certbot aren't installed, this isn't an issue. Going this route may require dramatic changes to the snapcraft Python plugin or us not being able to use it all in the DNS plugin snaps. I think we could maybe accomplish this through some combination of passing `--no-deps` to `pip` and stripping out the `acme` and `certbot` dependencies in the plugins.

Another idea would be to make the unreleased version of acme and certbot available to the DNS plugin snap during the build process. I'm open to other ideas too if people have them.
Externally snapped plugins contain other copy of Certbot
The following discussion was (hackily) copied from https://github.com/basak/certbot-snap-build/issues/20:

I said:

> I may be mistaken, but I believe each separately packaged snap plugin includes its own copy of Certbot and all of its dependencies. Is there anyway we can avoid this by having the plugin essentially depend on the Certbot snap, pull it into its build environment, and then only install dependencies that aren't already satisfied?

Robie said:

>Good idea!
>
>I think this would work in the case of most plugins. They could ship only the CERTBOT_PLUGIN_PATH directory and any dependency directories and that would make the plugin snaps much smaller.
>
>If the list of directories is available, then I believe this is trivial to configure in snapcraft.yaml. I'm not sure how to get that list automatically though.
>
>So I think our architecture allows it, but I'm not sure how in the build step to ensure that it is done correctly in the general case. In specific cases it should be relatively easy to arrange by considering each plugin individually.

I said:

>What cases do you have in mind where you think this would not work?
>
>To set this up, are you imagining the plugin including a subset of the directories under `$SNAP/lib/python3.5/site-packages`?
>
>Is there any (easy) way to load the Certbot snap or its files into the build environment of the plugin snap? The Certbot snap itself is the real source of truth about what packages are already available and can be excluded from the plugin snap. Alternatively, I think we'd need something like a separate file to list the installed packages which plugin snaps could use to know which packages they can exclude.

Robie said:

> >What cases do you have in mind where you think this would not work?
>
>I think I had in mind cases where the required plugin dependencies collide/conflict with certbot's required dependencies. However then of course the plugin wouldn't work just from pip without snap, so perhaps that's not an issue at all.
>
>Another example: if a third party plugin were to use augeas (and the main certbot snap did not), it'd have a dependency on libaugeas as well as the augeas lenses in /usr/lib/ somewhere. A plugin snap build wouldn't automatically know that those directories must not be stripped. This would have to be handed manually by the plugin snap developer, but it wouldn't make anything impossible I don't think.
>
> > To set this up, are you imagining the plugin including a subset of the directories under $SNAP/lib/python3.5/site-packages?
>
>Right. At the same path, but with most of the other directories removed.
>
> > Is there any (easy) way to load the Certbot snap or its files into the build environment of the plugin snap?
>
>There's build-snaps which might work. I think this might give you the installed snap available in the build environment only. At that point you could examine /snap/certbot/current. This might need some thought wrt. the version you're getting at build time versus the intended production version the snap will be intended to work with.

I said:

> > if a third party plugin were to use augeas (and the main certbot snap did not), it'd have a dependency on libaugeas as well as the augeas lenses in /usr/lib/ somewhere.
>
>If the main Certbot snap did not contain augeas but the plugin snap did, do you know if the plugin would be able to dynamically load augeas? Would paths be set up properly for this to work while the code is being run by the Certbot snap? I haven't looked into the details of how snaps (and especially classic snaps) manage to keep their code isolated from the rest of the system.
>
> > There's build-snaps which might work. I think this might give you the installed snap available in the build environment only. At that point you could examine /snap/certbot/current. This might need some thought wrt. the version you're getting at build time versus the intended production version the snap will be intended to work with.
>
>Thanks for the link! With some tinkering this seems promising to me. It seems like problem of different versions (and the fact the Certbot snap masks dependencies installed in the external plugin snap) could be largely solved by making it clear what dependencies will be installed in the Certbot snap and announcing any changes that we expect to break things such as ABI changes well in advance. Since most of Certbot's dependencies are entirely written in Python and their APIs have had few significant changes over the past 4+ years, I wouldn't expect there to be too many problems here.


Discussion:
> Another idea would be to make the unreleased version of acme and certbot available to the DNS plugin snap during the build process. I'm open to other ideas too if people have them.

This is typically to make these kind of solutions possible during the release that I am for a migration from Travis to a pipeline-oriented CI. Indeed we could imagine a job where the certbot snap is built, then stored as an artifact without pushing it to snapstore. At the right time in the pipeline, the job(s) responsible to build and test the dns plugins snaps would retrieve the artifact, install the certbot snap, and test the dns snap with it.

This way you can test the whole stack of snaps without any publication, and so validate everything before making the actual publication, without a dramatic change in how the snaps are built. 
I agree!
To be clear, my intention of this issue here is to track the problem that during the build process for DNS plugin snaps, acme and certbot are currently installed in the DNS plugin snap from PyPI which won't work if the plugin requires an unreleased version of acme or certbot.

The problem/solution Adrien described also exists where  when we go to test the DNS plugin snap, we'll need to make sure the Certbot snap it is used with is built from the same source because otherwise the Certbot snap may not have the changes the plugin snap requires. I added a comment about this at https://github.com/certbot/certbot/issues/8041#issuecomment-642849813.
One idea I had to accomplish both this and #7946 is to create an environment variable like `NO_CERTBOT_DEPS` which when it is set, causes the plugins to not add dependencies on `acme` and `certbot`. In addition to this, to avoid the possibility of building wheels during a release outside of the snaps with this value set, I think we should probably raise an exception if this environment variable is set and `'bdist_wheel' in sys.argv[1:]` is true in `setup.py`.

I briefly tested all of this outside of the snaps and it seemed to work.

Once these changes are made, I think we could then just set this environment variable in the plugin's snapcraft.yml file. I think you can use `build-environment` to do this.
Wait, how is that working? Plugins call into `certbot` code; see [here](https://github.com/certbot/certbot/blob/master/certbot-dns-dnsimple/certbot_dns_dnsimple/_internal/dns_dnsimple.py#L7) for example. When you said "it seemed to work", did you mean that the snap built successfully? Also, did you consider telling the snap not to stage the `acme` and `certbot` subdirectories in `lib` in the `snapcraft.yaml`? Doing so would mean there's no chance of accidentally building non-snap wheels without the deps.
> Wait, how is that working?

Because `pip`/`setuptools` just installs our code. It doesn't run it.

> When you said "it seemed to work", did you mean that the snap built successfully?

Like I said, I tested this outside of snaps, but on Ubuntu 20.04, pip happily installed the plugin into a virtual environment without the `acme` and `certbot` dependencies. This is somewhat similar to running `pip install --no-deps`, however, the plugin's other dependencies are installed.

> did you consider telling the snap not to stage the `acme` and `certbot` subdirectories in `lib` in the `snapcraft.yaml`?

I did not and I think something like that would work, but there are other problems to resolve. That approach solves https://github.com/certbot/certbot/issues/7946, but I don't think it would solve this issue. The problem here is the `acme` and `certbot` packages in the root of the repo are currently not available to the plugin snap when it is building. I think at least by default, snapcraft wants a layout like:
```
source files/
  - snap/
    - snapcraft.yml
```
This is the layout we currently have in the proof of concept as you can see by the use of `.` to refer to the DNS plugin source [here](https://github.com/certbot/certbot/blob/master/certbot-dns-dnsimple/snap/snapcraft.yaml#L12).

Because of this layout, I don't think `acme` and `certbot` in the parent directories are available to snapcraft while it is building the DNS plugin. Since they are not being used, the acme and certbot packages are currently being installed from PyPI. The problem I described in this issue is that pulling these packages from PyPI will not work when the plugin's acme or certbot dependencies is something like `acme>=1.6.0.dev0`. No such package will exist on PyPI so the snap build will fail.

We could try to get these packages exposed to snapcraft in some way. I'm not sure how hard this would be to do and potential gotchas along the way.

Two benefits I think my approach has over something like this though, at least how I have imagined we'd make it work are:

1. Customizing the snapcraft build may prevent us from making use of build.snapcraft.io to build our snaps for other architectures which Adrien and I talked about doing at https://opensource.eff.org/eff-open-source/pl/zj6aurg8ijd9zntwh8t5krdoga. At least when doing the builds through the web UI, you cannot customize the build at all.
2. All of `acme` and `certbot`'s dependencies are not included in the plugin snap without having to explicitly unstage them and keep the list up-to-date.

This is not at all to say we couldn't or shouldn't look into an approach like this, it's just the potential problems and downsides I wanted to flag.

> Doing so would mean there's no chance of accidentally building non-snap wheels without the deps.

I personally feel pretty confident there's no chance with the `bdist_wheel` check I described which is something [we already do](https://github.com/certbot/certbot/blob/bcb3554836471b0c18fa00862ba73bf7c3396c42/certbot/setup.py#L67-L69), but I agree with you it completely removes the concern and need for any mitigations at all though. It's a trade-off to make.
> If the main Certbot snap did not contain augeas but the plugin snap did, do you know if the plugin would be able to dynamically load augeas? Would paths be set up properly for this to work while the code is being run by the Certbot snap? I haven't looked into the details of how snaps (and especially classic snaps) manage to keep their code isolated from the rest of the system.

Classic snaps just have their code available but at a different mount point, and it's the snap developer's responsibility to set up everything so the right things get loaded (eg. with `PATH`, `LD_LIBRARY_PATH` and so on). I think snapcraft helps here and there, such as using `patchelf` to adjust binaries to find their (non-`dlopen()`) dynamically linked libraries in the right place automatically.

So I think it should be possible, but perhaps not trivial, and might have to be figured out on a case-by-case basis. In the case of augeas, it's possible to tweak the Python end of it to `dlopen()` from a different place, for example.

Resolving patch:
diff --git a/certbot-dns-dnsimple/setup.py b/certbot-dns-dnsimple/setup.py
--- a/certbot-dns-dnsimple/setup.py
+++ b/certbot-dns-dnsimple/setup.py
@@ -12,12 +12,19 @@
 # Remember to update local-oldest-requirements.txt when changing the minimum
 # acme/certbot version.
 install_requires = [
-    'acme>=0.31.0',
-    'certbot>=1.1.0',
     'setuptools',
     'zope.interface',
 ]
 
+if not os.environ.get('EXCLUDE_CERTBOT_DEPS'):
+    install_requires.extend([
+        'acme>=0.31.0',
+        'certbot>=1.1.0',
+    ])
+elif 'bdist_wheel' in sys.argv[1:]:
+    raise RuntimeError('Unset EXCLUDE_CERTBOT_DEPS when building wheels '
+                       'to include certbot dependencies.')
+
 setuptools_known_environment_markers = (StrictVersion(setuptools_version) >= StrictVersion('36.2'))
 if setuptools_known_environment_markers:
     install_requires.append('mock ; python_version < "3.3"')


Test patch: