Repository: spack/spack

Issue:
external find python should not generate python3.x unless python3 is the same version
### Steps to reproduce

On some systems (eg RHEL) with `/etc/alternatives`, there are multiple versions
of python in /usr under names like `/usr/bin/python3.6`, `/usr/bin/python3.8`, etc.
`spack external find python` seems to find all of them, but there can
only be one `/usr/bin/python3`.

I claim this is a mistake and that `external find` should not produce
`python3.x` unless `python3` exists in the same directory and runs the
same version.  Thus, external find should never produce multiple versions
in the same directory such as `/usr/bin`.

**How to reproduce**

On one system that uses `/etc/alternatives` where `/usr/bin/python3.8`
exists but `/usr/bin/python3` still points to 3.6, run:

```
$ spack external find python
==> The following specs have been detected on this system
    and added to /home/krentel/.spack/packages.yaml
python@2.7.18  python@3.6.8  python@3.8.16

$ cat ~/.spack/packages.yaml 
packages:
  python:
    externals:
    - spec: python@2.7.18+bz2+crypt+ctypes+dbm~lzma+nis+pyexpat~pythoncmd+readline+sqlite3+ssl~tkinter+uuid+zlib
      prefix: /usr
    - spec: python@3.6.8+bz2+crypt+ctypes+dbm+lzma+nis+pyexpat~pythoncmd+readline+sqlite3+ssl~tkinter+uuid+zlib
      prefix: /usr
    - spec: python@3.8.16+bz2+crypt+ctypes+dbm+lzma+nis+pyexpat~pythoncmd+readline+sqlite3+ssl~tkinter+uuid+zlib
      prefix: /usr
```

The problem here is that spack claims that both `python 3.6.8` and
`3.8.16` exist and both exist in `/usr`.  Of course, there is only **one**
`/usr/bin/python3` (which still points to 3.6).

The `intel-xed` package depends on `3.7:`.  So, `spack spec intel-xed`
succeeds (thinking it's getting 3.8), but then `install` falls over
when it actually gets 3.6.

This actually affects many packages.  Xed stands out just because it
asks for 3.7:.

```
$ spack spec intel-xed
...
 -   intel-xed@2023.07.09%gcc@8.5.0~debug~pic build_system=generic arch=linux-rhel8-zen
[e]      ^python@3.8.16%gcc@8.5.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat~pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,4c24573,ebdca64,f2fd060 arch=linux-rhel8-zen

 $ spack install intel-xed
[+] /usr (external python-3.8.16-zevk7meiwbvjbuerlqkmpcis6nmiv5i6)
==> Installing intel-xed-2023.07.09-rbpy5eljvirht67et7rbocvo4pgefeoi
...
==> intel-xed: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 1:
    './mfile.py' '--clean'
See build log for details:
  /scratch/krentel/python/spack-repo/build-stage/spack-stage-intel-xed-2023.07.09-rbpy5eljvirht67et7rbocvo4pgefeoi/spack-build-out.txt
```

Of course, it fails because xed `mfile.py` tries to run `/usr/bin/env python3`
(nothing wrong with that) and it gets 3.6.

So, IMO the current `spack external find python` is broken because it
produces python3.8 but can't deliver 3.8.

IMO, the simplest fix is to refuse to generate `python3.x` unless
`/python3` exists in the same directory and produces the same version.
In particular, there should only be one python3 in /usr/bin.

A more complicated fix would be to create a wrapper for the other
python3.x, but you'd have to create a new directory and wrapper for
each "other" python that exists in the same directory but is different
from python3.

If you want to do that, fine, but IMO it's not worth it.  Besides, if I really
want that, I can do it myself.

ping @adamjstewart 


### Error message

See above.

### Information on your system

$ spack debug report
* **Spack:** 0.21.0.dev0 (bd165ebc4dbd0001d4e125e1caeb9fe09064c632)
* **Python:** 3.8.16
* **Platform:** linux-rhel8-zen2
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I disagree that it's a "bug" to detect both. Both Python 3.6 and 3.8 are indeed installed, and work fine if called through the `pythonX.Y` executable.

BUT! I agree that it isn't very useful to detect both if many packages are hard-coded to expect `python3` (or `python`) and won't build as a result. We can try to change them, but there are far too many that expect this.

So I'm not necessarily opposed to dropping support for external Python if multiple versions are installed in the same prefix. In fact, anything that reduces my maintenance burden is appreciated, and limiting the scope of external Pythons we support sounds good to me.

This reminds me of similar issues during the Python 2 -> 3 transition period where Ubuntu got rid of the `python` executable and only had `python2` and `python3` executables. Many scripts were still hard-coded to use `/usr/bin/env python`, and would fail as a result. This was "solved" by only supporting Python 3, but clearly the same issue still persists. We also added a `+pythoncmd` variant that defines whether or not `python` exists in addition to `pythonX` and `pythonX.Y`. We could do something similar for `+pythoncmdcmd` (name TBD) if we really wanted to support this, but then every package has to define whether or not it expects `pythonX` to work.

In summary, everything is Python's fault for allowing multiple versions to be installed in a single directory but not making it clear how users should invoke scripts or define shebangs if multiple versions are supported.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/python/package.py b/var/spack/repos/builtin/packages/python/package.py
--- a/var/spack/repos/builtin/packages/python/package.py
+++ b/var/spack/repos/builtin/packages/python/package.py
@@ -326,7 +326,7 @@ class Python(Package):
     # An in-source build with --enable-optimizations fails for python@3.X
     build_directory = "spack-build"
 
-    executables = [r"^python[\d.]*[mw]?$"]
+    executables = [r"^python\d?$"]
 
     @classmethod
     def determine_version(cls, exe):


Test patch: