Repository: easybuilders/easybuild-easyblocks

Issue:
Python packages are being installed to local/lib/python*/dist-packages due to tweaks to sysconfig.py in OS (Ubuntu 22.04)
**Problem description**

Hi. Recently, one of my machines started experiencing problems with installing EasyBuild via the bootstrap method.

```
== 2022-11-26 22:18:10,235 easyblock.py:3774 INFO Running method sanity_check_step part of step sanitycheck
== 2022-11-26 22:18:10,281 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/eb_tmp/local/lib/python3.10/dist-packages/easybuild/base/exceptions.py:124 in __init__): Failed to isolate installed egg dir for easybuild-framework (at easybuild/eb_tmp/local/lib/python3.10/dist-packages/easybuild/easyblocks/e/easybuildmeta.py:235 in sanity_check_step)
```

I use a custom build script I wrote based on the official instructions in the Lua, Lmod and EasyBuild documentations:
https://github.com/anselmicz/easybuild-installer/blob/main/easybuild.sh

My current builds fail during the sanity check phase, I'm enclosing the build log, environment at the time of installation, and trace output as a gist:
https://gist.github.com/anselmicz/72d1a0239cf67b685ed878aef2f61359

The error itself comes from https://github.com/boegel/easybuild-easyblocks/blob/develop/easybuild/easyblocks/e/easybuildmeta.py#L235

In a Slack conversation, @ocaisa mentioned that the error may be somehow connected to
https://github.com/easybuilders/easybuild-easyblocks/pull/2805

**System**

***OS***
Debian bookworm/sid

***Custom dependencies***
```
Lua/5.1.4.9
Lmod/8.5
```

***System packages***
```
python3/3.10.6
python3-pip/22.3
wget/1.21.3
gcc/12.2.0
make/4.3
rsync/3.2.6
tcl-dev/8.6.11
libreadline-dev/8.2
libibverbs-dev/43.0
xdot/1.2
```

***Python packages***
```
setuptools/65.6.3
python-graph-core/1.8.2
python-graph-dot/1.8.2
archspec/0.2.0
autopep8/2.0.0
GitPython/3.1.29
pep8/1.7.1
pycodestyle/2.10.0
Rich/12.6.0
```

Do you have any thoughts on this? I tried lowering both the temporary and to-be-installed versions of EasyBuild, but the issue persists both on `4.6.1` and  `4.6.0`, and I can't try `4.5.4` because then the `setuptools` and `setup.py` problems arise.


Discussion:
@anselmicz Can you share the contents of `/apps/all/EasyBuild/4.6.2`, in particular the contents of the `lib*/python*/site-packages` subdirectory that should be there?

I have also tried lowering the `setuptools` and `pip` versions to no avail.

@boegel I have updated the gist with a directory tree of the `/apps/all/EasyBuild/4.6.2`

`lib/python3.10/site-packages` is empty, however, everything seems to install under `/apps/all/EasyBuild/4.6.2/local`, including `lib`... this is probably the distribution difference, but here the path is `.../lib/python3.10/dist-packages`.

Same goes for the temporary EasyBuild installation, instead of installing under the prefix, `/tmp/$USER/eb_tmp`, the actual location of `bin`, `lib`, etc. is under `/tmp/$USER/eb_tmp/local`

I remember dealing with this problem before, but the issue has always been exclusively with the temporary installation of EasyBuild that could have always been fixed by accommodating the `PYTHONPATH`, i. e. setting `export PYTHONPATH=$(/bin/ls -rtd -1 $EB_TMPDIR/local/lib*/python*/dist-packages | tail -1):$PYTHONPATH`.
Here is a Dockerfile to reproduce the error using Ubuntu 22.04 LTS:

```
FROM ubuntu:jammy-20221130
#FROM ubuntu:focal-20221130

ARG DEBIAN_FRONTEND="noninteractive"

RUN set -eux \
  && apt-get update -qq \
  && apt-get -y upgrade \
  && apt-get install -y -q --no-install-recommends \
      lmod \
      python3-pip \
  && apt-get clean \
  && python3 -m pip install --ignore-installed --prefix /tmp/eb_tmp easybuild \
  && rm -rf /tmp/hsperfdata* /var/*/apt/*/partial /var/lib/apt/lists/* /var/log/apt/term*

RUN set -x \
  && export LOGNAME=root \
  && export PATH=$(find /tmp/eb_tmp -type d -name bin):$PATH \
  && export PYTHONPATH=$(find /tmp/eb_tmp/ -type d -name *-packages | tail -1) \
  && export EB_PYTHON=python3 \
  && . /etc/profile.d/lmod.sh \
  && eb --allow-use-as-root-and-accept-consequences --install-latest-eb-release -d --prefix /tmp/easybuild \
  ;  cat /tmp/eb-*/easybuild-EasyBuild-4.6.2-*.log

ENTRYPOINT ["/bin/bash"]
```

When you `docker build` the same thing using `FROM ubuntu:focal-20221130` it works. So there has to be something different with Ubuntu 22.04 LTS, which I haven't figured out, yet. One of the obvious things, as mentioned: `jammy` installs everything in `/tmp/eb_tmp/local/`, while `focal` installs directly into `/tmp/eb_tmp`.

@anselmicz, is it the latest Ubuntu LTS for you, too?

Any ideas? Is there a fix or config change needed in Ubuntu, or with EasyBuild?
@crubb my report deals with Debian Unstable

I confirm I can reproduce the error in `jammy`. However, this is not at all that surprising, because, based on my understanding of Ubuntu versioning, `jammy` is based on Debian `bookworm` (current `sid` / unstable). It also makes sense that it might work correctly in `focal`, given it's based on `bullseye` where it works too (at least for me).
root cause of the problems seems to be the following: https://discuss.python.org/t/linux-distro-patches-to-sysconfig-are-changing-pip-install-prefix-outside-virtual-environments/18240/14

Essentially, some distros patched Python's sysconfig to change the default installation scheme from `posix_prefix` to `posix_local`.

I found a bug report for Ubuntu, which essentially sits dormant after more or less stating that it's a design choice: https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1967920
@crubb I can see how that affect the installation of EasyBuild with EasyBuild (since that's done with the system Python), but how does that affect installing Python packages with a Python that was installed with EasyBuild?!
@crubb Good catch! This (temporary) workaround seems to work for me:
```
--- /lib/python3.10/sysconfig.py.orig	2022-12-13 17:38:06.221735514 +0100
+++ /lib/python3.10/sysconfig.py	2022-12-13 17:40:28.549111500 +0100
@@ -271,7 +271,7 @@
         if deb_build in ('deb', 'deb_system'):
             prefix_scheme = 'deb_system'
         else:
-            prefix_scheme = 'posix_local'
+            prefix_scheme = 'posix_prefix'

     return {
         'prefix': prefix_scheme,
```

@boegel, looking at the links @crubb provided, it seems like in Debian, `sysconfig.py` got some extreme mods between the versions:
https://lists.debian.org/debian-python/2022/03/msg00039.html.
So far this seems to be distro specific, so the EasyBuild Python should be unaffected.
I actually borrowed from Debian and stole their `DEB_PYTHON_INSTALL_LAYOUT` environment variable. Set it to `eb`, and it uses `posix_prefix` as the `prefix_scheme`:

```
--- sysconfig.py-orig   2022-12-13 21:37:25.489016711 +0000
+++ sysconfig.py        2022-12-13 21:37:52.088975055 +0000
@@ -270,6 +270,8 @@
         deb_build = os.environ.get('DEB_PYTHON_INSTALL_LAYOUT', 'posix_local')
         if deb_build in ('deb', 'deb_system'):
             prefix_scheme = 'deb_system'
+        elif deb_build in ('eb'):
+            prefix_scheme = 'posix_prefix'
         else:
             prefix_scheme = 'posix_local'
```
I just run into the same problem while generating a test-report on a `ubuntu-22.04` container ([recipe](https://github.com/easybuilders/easybuild-containers/blob/main/ubuntu-22.04/Dockerfile), [image @ ghcr.io](https://github.com/easybuilders/easybuild-containers/pkgs/container/ubuntu-22.04) ) for `EasyBuild-4.7.0.eb`: https://github.com/easybuilders/easybuild-easyconfigs/pull/17065#issuecomment-1375629863

With the described workaround I was able to build it successfully.
Did some more digging, found some more info:

* ["sysconfig default scheme change in Python 3.10" (on `debian-python` mailing list)](https://lists.debian.org/debian-python/2022/03/msg00039.html);
* In https://github.com/pypa/pip/issues/10978 it's highlighted that this problem isn't specific to Debian/Ubuntu at all, it also affects Fedora 37 (so soon also RHEL & friends?), or even Fedora 36 (based on [this](https://bugzilla.redhat.com/show_bug.cgi?id=2026979) and [this](https://bugzilla.redhat.com/show_bug.cgi?id=2097183)), but there they didn't even bother to introduce a custom installation scheme like Debian/Ubuntu did with `posix_local` (which is not standard in Python, see https://docs.python.org/3/library/sysconfig.html#installation-paths), they just fiddled with the default `posix_prefix` instead, making it harder to detect;
  * [this](https://src.fedoraproject.org/rpms/python3.10/pull-request/124) seems to suggest the change was rolled back in Fedora though...
* Spack has been dealing with this too, see:
  * https://github.com/spack/spack/issues/37632
Updates to the `PythonPackage` easyblock are implemented in #2977 to deal with this, using that EasyBuild installs without problems on Ubuntu 22.04.
@boegel redirected me here about the problem on Ubuntu 22.04. I could not follow easybuild installation guide on [Installing EasyBuild tutorial](https://docs.easybuild.io/installation/#eb_as_module). The procedure fails at step 2 with the error:

```
build failed (first 300 chars): Failed to isolate installed egg dir for easybuild-framework (took 1 min 10 secs)
```
 and no other obvious information. The solution discussed here seem to be used with the flag `--include-easyblocks-from-pr 2977`, as in:

```
eb --include-easyblocks-from-pr 2977 --install-latest-eb-release --prefix /tmp/easybuild
```

and it worked immediately as expected on my system.
@akesandgren mentioned that he was hitting some problems when testing #2977, so I'm re-opening this issue to try and figure out what was overlooked (I haven't been able to reproduce this problem):

```
== 2023-08-24 16:23:42,987 pythonpackage.py:285 INFO Subdirectories of /scratch/ebtest4/software/EasyBuild/4.8.0/local: ['lib']
== 2023-08-24 16:23:42,988 build_log.py:171 ERROR EasyBuild crashed with an error (at easybuild/base/exceptions.py:126 in __init__): Failed to install EasyBuild packages: [Errno 22] Invalid argument: '/scratch/ebtest4/software/EasyBuild/4.8.0/lib' (at easybuild/easyblocks/e/easybuildmeta.py:155 in install_step)
```
I did a quick debugging session on this with @akesandgren, and it's now clear why he was hitting the `Invalid argument` error, which is triggered by EasyBuild trying to symlink `<prefix>/local/lib` to `<prefix>/lib` while `<prefix>/lib` already exists.

In @akesandgren's Ubuntu 22.04 container, Python 3.10 is installed, but `pip` is *not*.
The `EasyBuildMeta` easyblock automatically enables installing EasyBuild with `pip install --prefix ...` for sufficiently recent versions of EasyBuild, if `pip` is installed (and recent enough). Since that condition is not met, the `EasyBuildMeta` easyblock falls back to using `python setup.py install --prefix ...`.

Apparently whether the active Python naming scheme of the system Python is `posix_prefix` (the default) or `posix_local` (the custom one in Ubuntu 22.04) only matters when installing Python packages with `pip install --prefix ...` (or `python3 -m pip install --prefix ...`), *not* when using `python3 setup.py install --prefix ...`.

The changes in #2977 make `PythonPackage` prepare an empty `<prefix>/local/python*/site-packages` directory, based on the active Python naming scheme being `posix_local`, but the installation is actually done in `<prefix>/local/python*/site-packages` (as it should be!).
`PythonPackage.install_step` then tries to symlink `local/lib` to `lib` after installing the `easybuild-framework` package, which fails because `<prefix>/lib` exists already.

Resolving patch:
diff --git a/easybuild/easyblocks/generic/pythonpackage.py b/easybuild/easyblocks/generic/pythonpackage.py
--- a/easybuild/easyblocks/generic/pythonpackage.py
+++ b/easybuild/easyblocks/generic/pythonpackage.py
@@ -562,11 +562,32 @@ def get_installed_python_packages(self, names_only=True, python_cmd=None):
         else:
             return pkgs
 
+    def using_pip_install(self):
+        """
+        Check whether 'pip install --prefix' is being used to install Python packages.
+        """
+        if self.install_cmd.startswith(PIP_INSTALL_CMD):
+            self.log.debug("Using 'pip install' for installing Python packages: %s" % self.install_cmd)
+            return True
+        else:
+            self.log.debug("Not using 'pip install' for installing Python packages (install command template: %s)",
+                           self.install_cmd)
+            return False
+
+    def using_local_py_install_scheme(self):
+        """
+        Determine whether the custom 'posix_local' Python installation scheme is actually used.
+        This requires that 'pip install --prefix' is used, since the active Python installation scheme
+        doesn't matter when using 'python setup.py install --prefix'.
+        """
+        # see also  https://github.com/easybuilders/easybuild-easyblocks/issues/2976
+        py_install_scheme = det_py_install_scheme(python_cmd=self.python_cmd)
+        return py_install_scheme == PY_INSTALL_SCHEME_POSIX_LOCAL and self.using_pip_install()
+
     def compose_install_command(self, prefix, extrapath=None, installopts=None):
         """Compose full install command."""
 
-        using_pip = self.install_cmd.startswith(PIP_INSTALL_CMD)
-        if using_pip:
+        if self.using_pip_install():
 
             pip_version = det_pip_version(python_cmd=self.python_cmd)
             if pip_version:
@@ -603,7 +624,7 @@ def compose_install_command(self, prefix, extrapath=None, installopts=None):
                 # otherwise, self.src is a list of dicts, one element per source file
                 loc = self.src[0]['path']
 
-        if using_pip:
+        if self.using_pip_install():
             extras = self.cfg.get('use_pip_extras')
             if extras:
                 loc += '[%s]' % extras
@@ -638,11 +659,13 @@ def py_post_install_shenanigans(self, install_dir):
         * dealing with 'local' subdirectory in install directory in case 'posix_local' installation scheme was used;
         * symlinking site-packages to dist-packages if only the former is available;
         """
-        py_install_scheme = det_py_install_scheme(python_cmd=self.python_cmd)
-
-        if py_install_scheme == PY_INSTALL_SCHEME_POSIX_LOCAL:
+        if self.using_local_py_install_scheme():
+            self.log.debug("Looks like the active Python installation scheme injected a 'local' subdirectory...")
             handle_local_py_install_scheme(install_dir)
+        else:
+            self.log.debug("Looks like active Python installation scheme did not inject a 'local' subdirectory, good!")
 
+        py_install_scheme = det_py_install_scheme(python_cmd=self.python_cmd)
         if py_install_scheme != PY_INSTALL_SCHEME_POSIX_PREFIX:
             symlink_dist_site_packages(install_dir, self.all_pylibdirs)
 
@@ -777,13 +800,12 @@ def test_step(self, return_output_ec=False):
 
                     # if posix_local is the active installation scheme there will be
                     # a 'local' subdirectory in the specified prefix;
-                    # see also https://github.com/easybuilders/easybuild-easyblocks/issues/2976
-                    py_install_scheme = det_py_install_scheme(python_cmd=self.python_cmd)
-                    if py_install_scheme == PY_INSTALL_SCHEME_POSIX_LOCAL:
+                    if self.using_local_py_install_scheme():
                         actual_installdir = os.path.join(test_installdir, 'local')
                     else:
                         actual_installdir = test_installdir
 
+                    self.log.debug("Pre-creating subdirectories in %s: %s", actual_installdir, self.all_pylibdirs)
                     for pylibdir in self.all_pylibdirs:
                         mkdir(os.path.join(actual_installdir, pylibdir), parents=True)
                 except OSError as err:
@@ -828,14 +850,14 @@ def install_step(self):
         # if posix_local is the active installation scheme there will be
         # a 'local' subdirectory in the specified prefix;
         # see also https://github.com/easybuilders/easybuild-easyblocks/issues/2976
-        py_install_scheme = det_py_install_scheme(python_cmd=self.python_cmd)
-        if py_install_scheme == PY_INSTALL_SCHEME_POSIX_LOCAL:
+        if self.using_local_py_install_scheme():
             actual_installdir = os.path.join(self.installdir, 'local')
         else:
             actual_installdir = self.installdir
 
         # create expected directories
         abs_pylibdirs = [os.path.join(actual_installdir, pylibdir) for pylibdir in self.all_pylibdirs]
+        self.log.debug("Pre-creating subdirectories %s in %s...", abs_pylibdirs, actual_installdir)
         for pylibdir in abs_pylibdirs:
             mkdir(pylibdir, parents=True)
 


Test patch: