Repository: spack/spack

Issue:
clingo fails with python 3.6 on ppc64le RHEL 7.x
### Steps to reproduce

Clingo does not work with the /usr/bin/python3 3.6 that comes with
RedHat 7.x on ppc64le.  The initial bootstrapping fails with:

```
$ spack solve zlib  
==> Error: module 'clingo' has no attribute 'Control'
```

I tried this on a power9 with RHEL 7.7 with /usr/bin/python3 3.6.8,
rpm package python3-3.6.8-10.el7.ppc64le.

How to reproduce:

1. move .spack to another name, eg: dot-spack, so you have a clean, empty setup

2. make a fresh clone of spack develop

3. source setup-env.sh, and edit config.yaml to set concretizer to clingo

4. run: spack compiler find

```
$ spack compiler find
==> Added 1 new compiler to /home/krentel/.spack/linux/compilers.yaml
    gcc@4.8.5
==> Compilers are defined in the following files:
    /home/krentel/.spack/linux/compilers.yaml
```

5. run: spack solve zlib

```
$ spack solve zlib
==> Error: module 'clingo' has no attribute 'Control'
```

----------

You can also reproduce the problem on lassen at LLNL.  The thing about
LLNL machines is that they come with the StdEnv module that puts
another python from /usr/tcetmp/bin in front of your path.  If you
move that out of the way and use /usr/bin/python3, then it has the
same problem.

You could unload the StdEnv module, or add /usr/bin to the front of
your path.  Anything that puts /usr/bin/python3 before /usr/tcetmp
will get back to the bare RH 7.x python and reproduce the problem.

You can even mkdir some directory, copy /usr/bin/python3 to the new
directory and put that in front of your path.  That pretty much
confirms the problem is python because it's the only new thing in your
path.

Notes:

1. As I mentioned, the problem is in python3 shipped with RHEL 7.x on
ppc64le.  If you build a new python, even the same 3.6 rev, then it
works.

2. The problem is specific to ppc64le.  It doesn't happen on x86_64,
even with the same RH rev.  I haven't tried ARM.

3. I haven't tried RH 8.x.  If someone can suggest a RH 8.x powerpc
machine, maybe something at LLNL or Sandia, I could try it there.

I don't know if the problem is that the RH python lacks some feature,
or if spack doesn't detect or use is correctly.  But if the shipped
python won't work with clingo, then that adds an extra step to the
bootstrap process to build a new python.

I've seen this sort of problem before where the same rev of RH on
different architectures don't provide exactly the same set of
features.  (But more often with compiler features.)

ping @alalazo 


### Error message

``` 
$ spack -d solve zlib

==> [2021-09-01-12:27:21.791669] Imported solve from built-in commands
==> [2021-09-01-12:27:21.793301] Imported solve from built-in commands
==> [2021-09-01-12:27:21.794287] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/bootstrap.yaml
==> [2021-09-01-12:27:21.818877] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/config.yaml
==> [2021-09-01-12:27:21.870298] Reading config file /home/krentel/clingo/spack-repo/etc/spack/config.yaml
==> [2021-09-01-12:27:21.917261] DATABASE LOCK TIMEOUT: 3s
==> [2021-09-01-12:27:21.917325] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-09-01-12:27:21.917923] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/bootstrap.yaml
==> [2021-09-01-12:27:21.938049] DATABASE LOCK TIMEOUT: 3s
==> [2021-09-01-12:27:21.938110] PACKAGE LOCK TIMEOUT: No timeout
==> [2021-09-01-12:27:21.938208] [BOOTSTRAP CONFIG SCOPE] name=_builtin
==> [2021-09-01-12:27:21.938851] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/home/krentel/clingo/spack-repo/etc/spack/defaults
==> [2021-09-01-12:27:21.938911] [BOOTSTRAP CONFIG SCOPE] name=defaults/linux, path=/home/krentel/clingo/spack-repo/etc/spack/defaults/linux
==> [2021-09-01-12:27:21.938977] [BOOTSTRAP CONFIG SCOPE] name=system, path=/etc/spack
==> [2021-09-01-12:27:21.939030] [BOOTSTRAP CONFIG SCOPE] name=system/linux, path=/etc/spack/linux
==> [2021-09-01-12:27:21.939094] [BOOTSTRAP CONFIG SCOPE] name=site, path=/home/krentel/clingo/spack-repo/etc/spack
==> [2021-09-01-12:27:21.939146] [BOOTSTRAP CONFIG SCOPE] name=site/linux, path=/home/krentel/clingo/spack-repo/etc/spack/linux
==> [2021-09-01-12:27:21.939208] [BOOTSTRAP CONFIG SCOPE] name=user, path=/home/krentel/.spack
==> [2021-09-01-12:27:21.939259] [BOOTSTRAP CONFIG SCOPE] name=user/linux, path=/home/krentel/.spack/linux
==> [2021-09-01-12:27:21.962406] [BOOTSTRAP ROOT SPEC] clingo: clingo-bootstrap@spack+python %gcc target=ppc64le
==> [2021-09-01-12:27:21.962470] [BOOTSTRAP MODULE clingo] Try importing from Python
==> [2021-09-01-12:27:21.962851] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/bootstrap.yaml
==> [2021-09-01-12:27:22.144795] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/config.yaml
==> [2021-09-01-12:27:22.202184] Reading config file /home/krentel/clingo/spack-repo/etc/spack/config.yaml
==> [2021-09-01-12:27:22.322394] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python %gcc target=ppc64le ^python@3.6 /rfu4la457mdbpoffk2g5hikj2hhoek4s" provides the "clingo" Python module
==> [2021-09-01-12:27:22.322660] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/repos.yaml
==> [2021-09-01-12:27:22.330052] Reading config file /home/krentel/clingo/spack-repo/etc/spack/defaults/config.yaml
==> [2021-09-01-12:27:22.381029] Reading config file /home/krentel/clingo/spack-repo/etc/spack/config.yaml
Traceback (most recent call last):
  File "/home/krentel/clingo/spack-repo/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/main.py", line 797, in main
    return _invoke_command(command, parser, args, unknown)
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/main.py", line 525, in _invoke_command
    return_val = command(parser, args)
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/cmd/solve.py", line 106, in solve
    specs, dump=dump, models=models, timers=args.timers, stats=args.stats
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/solver/asp.py", line 1691, in solve
    return driver.solve(setup, specs, dump, models, timers, stats, tests)
  File "/home/krentel/clingo/spack-repo/lib/spack/spack/solver/asp.py", line 332, in solve
    self.control = clingo.Control()
AttributeError: module 'clingo' has no attribute 'Control'
```

### Information on your system

* **Spack:** 0.16.2-4111-7adacf9
* **Python:** 3.6.8
* **Platform:** linux-rhel7-power9le
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I can reproduce that on Lassen. It seems the extension is not ABI compatible with that Python, even though I am trying to get more information. The weird thing is that the extension has been compiled against a Python 3.6 coming from `manylinux2014` so it _should_ be compatible with _any_ Python 3.6. I'll post findings here as I understand more.
I checked all the dynamic symbols and the only ones visible in `clingo` come from GLIBC_2.17 which is allowed, see [here](https://www.python.org/dev/peps/pep-0599/#the-manylinux2014-policy). 

There's one thing we do though that _might_ explain the ABI incompatibility. In the same document there are requirements for sticking with:
```
CXXABI_1.3.7, CXXABI_TM_1 is also allowed
GLIBCXX_3.4.19
GCC_4.8.0
```
We don't "export" those symbols, but clingo requires C++14 (which is fully supported from GCC 5 iirc). To avoid having to ship a standard C++ library, since we don't need to catch C++ exceptions across libraries we statically link `libstdc++` and `libgcc_s` coming from the GCC 9.3.0 in the `manylinux2014` image. Bottomline, this Python 3.6 is compatible with the extension:
```
$ python3
Python 3.6.4 (default, Feb  5 2018, 14:42:43) 
[GCC 4.9.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
```
this one is not:
```
$ /usr/bin/python3.6
Python 3.6.8 (default, Sep 26 2019, 15:48:23) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
```
one difference that strikes me is GCC 4.8.5 vs. GCC 4.9.3 Maybe it's a red herring...
Must be something else. I built Python 3.6.10 with Spack + GCC4.8.5 and:
```python
Python 3.6.10 (default, Sep  1 2021, 13:36:46) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import clingo
>>> dir(clingo)
['Application', 'ApplicationOptions', 'Assignment', 'Backend', 'Configuration', 'Control', 'Flag', 'Function', 'HeuristicType', 'Infimum', 'MessageCode', 'Model', 'ModelType', 'Number', 'Observer', 'ProgramBuilder', 'PropagateControl', 'PropagateInit', 'Propagator', 'PropagatorCheckMode', 'SolveControl', 'SolveHandle', 'SolveResult', 'StatisticsArray', 'StatisticsMap', 'String', 'Supremum', 'Symbol', 'SymbolType', 'SymbolicAtom', 'SymbolicAtomIter', 'SymbolicAtoms', 'TheoryAtom', 'TheoryAtomIter', 'TheoryElement', 'TheoryTerm', 'TheoryTermType', 'Trail', 'TruthValue', 'Tuple_', '_Slice', '_Symbol', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__version__', '_error_code', '_error_message', 'ast', 'clingo_main', 'parse_files', 'parse_program', 'parse_term']
```
Still have to understand what happens with the system Python.
Got what is the issue by using the Python interpreter in super verbose mode. Using the system Python:
```console
$ /usr/bin/python3 -vvv
[ lots of output ]
>>> import clingo
[ ... ]
# trying /home/bob/.spack/bootstrap/store/linux-centos7-ppc64le/gcc-9.3.1/clingo-bootstrap-spack-rfu4la457mdbpoffk2g5hikj2hhoek4s/lib/python3.6/site-packages/clingo.cpython-36m-ppc64le-linux-gnu.so
[ ... ]
```
The Python interpreter at some point is trying to load:
```
clingo.cpython-36m-ppc64le-linux-gnu.so
```
in the correct directory, but fails. If one looks carefully at it **the name of the library is wrong**, since apparently the system Python resolves the host architecture to ppc64le instead of powerpc64le. In fact, copying the binary we have to the name this Python expects we can import clingo:
```console
$ cp ~/.spack/bootstrap/store/linux-centos7-ppc64le/gcc-9.3.1/clingo-bootstrap-spack-rfu4la457mdbpoffk2g5hikj2hhoek4s/lib/python3.6/site-packages/clingo.cpython-36m-powerpc64le-linux-gnu.so  ~/.spack/bootstrap/store/linux-centos7-ppc64le/gcc-9.3.1/clingo-bootstrap-spack-rfu4la457mdbpoffk2g5hikj2hhoek4s/lib/python3.6/site-packages/clingo.cpython-36m-ppc64le-linux-gnu.so 
$ PYTHONPATH=~/.spack/bootstrap/store/linux-centos7-ppc64le/gcc-9.3.1/clingo-bootstrap-spack-rfu4la457mdbpoffk2g5hikj2hhoek4s/lib/python3.6/site-packages /usr/bin/python3.6
Python 3.6.8 (default, Sep 26 2019, 15:48:23) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import clingo 
>>> dir(clingo)
['Application', 'ApplicationOptions', 'Assignment', 'Backend', 'Configuration', 'Control', 'Flag', 'Function', 'HeuristicType', 'Infimum', 'MessageCode', 'Model', 'ModelType', 'Number', 'Observer', 'ProgramBuilder', 'PropagateControl', 'PropagateInit', 'Propagator', 'PropagatorCheckMode', 'SolveControl', 'SolveHandle', 'SolveResult', 'StatisticsArray', 'StatisticsMap', 'String', 'Supremum', 'Symbol', 'SymbolType', 'SymbolicAtom', 'SymbolicAtomIter', 'SymbolicAtoms', 'TheoryAtom', 'TheoryAtomIter', 'TheoryElement', 'TheoryTerm', 'TheoryTermType', 'Trail', 'TruthValue', 'Tuple_', '_Slice', '_Symbol', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__version__', '_error_code', '_error_message', 'ast', 'clingo_main', 'parse_files', 'parse_program', 'parse_term']
```
So the bottomline is that **the issue is not stemming from an ABI incompatibility but from the Python interpreter constructing the wrong name for the extension library to load** :sweat_smile: 

------

Confirmed that all the other Python interpreters look for the correct name:
```
# extension module 'clingo' loaded from '~/.spack/bootstrap/store/linux-centos7-ppc64le/gcc-9.3.1/clingo-bootstrap-spack-rfu4la457mdbpoffk2g5hikj2hhoek4s/lib/python3.6/site-packages/clingo.cpython-36m-powerpc64le-linux-gnu.so'
```
And corresponding issue: https://github.com/pypa/manylinux/issues/687
So it seems RedHat has a patch to "fix" the architecture names, which break manylinux on Power PC see [here](https://src.fedoraproject.org/fork/churchyard/rpms/python3/blame/00274-fix-arch-names.patch?identifier=test_email-mktime). The binary works with other compiled Python interpreters, since the name of the architecture is the expected one.
@tgamblin @becker33 One fix we can adopt to workaround this is to copy the extension library to the alternative name if we are on a Power architecture as soon as we have installed the corresponding buildcache.
Would it be a simple workaround to add a symlink in the clingo
bootstrap to cover the other name?

I tried that manually and it seems to work.
Yeah, I think that or copying should be the workaround. We should do it from Spack code, so we don't have to touch the buildcache which seems to be built and packaged correctly.
@alalazo I think making the symlink as @mwkrentel suggests is the way to go.  I'm tempted to say we should just fix the buildcache here, so that it always has the two names.  Adding symlinks like this seems like it should be part of creating packages for `ppc64le`.
Any reason not to make a new buildcache?
Creating a symlink or copying the library to have an alternative name is a quick fix. Recreating and uploading the buildcache takes longer and needs more validation (we have CI to check for `x86_64` but need to validate manually `ppc64le` and `aarch64` binaries). I feel like it's not less hacky since the binary is technically portable and correct, it is just running on a patched interpreter. 

Finally, a few lines fix in the `bootstrap.py` module with a comment pointing to the issue has better visibility among developers than a corresponding fix in the scripts generating the pipeline.
> Adding symlinks like this seems like it should be part of creating packages for ppc64le.

~I don't think so, unless:~
1. ~We package software using the RedHat system Python as an external~
2. ~We use RedHat patch to change the architecture name in an interpreter built by Spack~

Thinking about that some more, the interpreter we use goes in general in the hash of the buildcache, so every extension is made for a specific interpreter - and each interpreter is consistent with itself. This issue really arise because we want to build a "universal" Python buildcache that works on any Python 3.X interpreter + ppc64le. So I don't think we need a general fix for ppc64le.
The thing is that basically the only Power machines we care about run RHEL. This isn’t a special case; it’s kind of the only case, at least until IBM decides it cares about HPC again.

Also I’m kind of worried that other distros may do things like this, too. IIRC, Ubuntu calls the architecture `ppc64el` instead of `ppc64le`. Are there issues with Ubuntu on power?
So I’d say do the quick fix for now, but we need to get more machines validating the build caches w/different containers. We can add runners to them; we should have some power9 runners at LLNL that we can use; also GCP has some (though I dunno if we can use them effectively — they are AIX).
Do you know if the expected arch name here is something we can query from the host python? Seems like there should be a programmatic way to know the expected libname.
> Are there issues with Ubuntu on power?

I didn't check that, will do today. 

> Do you know if the expected arch name here is something we can query from the host python? Seems like there should be a programmatic way to know the expected libname.

I think the answer is:
```
sysconfig.get_config_var("EXT_SUFFIX")
```
which should get the extension suffix for the Python extensions (from the link reporting the issue on manylinux).
> sysconfig.get_config_var("EXT_SUFFIX")

Ok then maybe the thing to do is to symlink that if it’s different from the manylinux name — that seems more robust than including everything we know of in the buildcache; it might save us on platforms we don’t anticipate.
> Are there issues with Ubuntu on power?

Running on an emulated Ubuntu under Docker:
```console
root@bd0b62bf1402:/# uname -m
ppc64le
root@bd0b62bf1402:/# python3
Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_var("EXT_SUFFIX")
'.cpython-38-powerpc64le-linux-gnu.so'
```
So it seems they use a different name for `uname -m` but they don't patch the Python interpreter.
Just tried on ubuntu 20.04 ppc64le under Docker + Qemu:
```console
root@23bf6e6f7e4c:/spack# python3 bin/spack -d solve  zlib
[ ... ]
==> [2021-09-02-14:10:34.221393] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python %gcc target=ppc64le ^python@3.8 /uxkpc2euofdkgveonftklstnxyha5wsf" provides the "clingo" Python module
==> [2021-09-02-14:10:34.224783] Reading config file /spack/etc/spack/defaults/repos.yaml
==> [2021-09-02-14:10:34.480049] Reading config file /root/.spack/linux/compilers.yaml
==> [2021-09-02-14:10:34.516172] Reading config file /spack/etc/spack/defaults/packages.yaml
==> [2021-09-02-14:10:35.046197] opt_criterion(16, deprecated versions used)
==> [2021-09-02-14:10:35.046799] opt_criterion(15, version weight)
==> [2021-09-02-14:10:35.047451] opt_criterion(14, number of non-default variants (roots))
==> [2021-09-02-14:10:35.048011] opt_criterion(13, multi-valued variants)
==> [2021-09-02-14:10:35.048767] opt_criterion(12, preferred providers for roots)
==> [2021-09-02-14:10:35.049367] opt_criterion(11, number of non-default variants (non-roots))
==> [2021-09-02-14:10:35.050105] opt_criterion(9, preferred providers (non-roots))
==> [2021-09-02-14:10:35.050822] opt_criterion(8, compiler mismatches)
==> [2021-09-02-14:10:35.051532] opt_criterion(7, version badness)
==> [2021-09-02-14:10:35.052148] opt_criterion(6, count of non-root multi-valued variants)
==> [2021-09-02-14:10:35.052715] opt_criterion(5, non-preferred compilers)
==> [2021-09-02-14:10:35.053235] opt_criterion(4, target mismatches)
==> [2021-09-02-14:10:35.053888] opt_criterion(3, non-preferred targets)
==> [2021-09-02-14:10:35.084335] Best of 1 considered solutions.
==> [2021-09-02-14:10:35.088205] Optimization Criteria:
  Priority  Criterion                                 Value
  1         deprecated versions used                      0
  2         version weight                                0
  3         number of non-default variants (roots)        0
  4         multi-valued variants                         0
  5         preferred providers for roots                 0
  6         number of non-default variants (non-roots)    0
  7         preferred providers (non-roots)               0
  8         compiler mismatches                           0
  9         version badness                               0
  10        count of non-root multi-valued variants       0
  11        non-preferred compilers                       0
  12        target mismatches                             0
  13        non-preferred targets                         0

zlib@1.2.11%gcc@9.3.0+optimize+pic+shared arch=linux-ubuntu20.04-ppc64le
```
so it seems it's just RedHat system Python at the moment.

Resolving patch:
diff --git a/lib/spack/spack/bootstrap.py b/lib/spack/spack/bootstrap.py
--- a/lib/spack/spack/bootstrap.py
+++ b/lib/spack/spack/bootstrap.py
@@ -5,8 +5,11 @@
 from __future__ import print_function
 
 import contextlib
+import fnmatch
 import json
 import os
+import os.path
+import re
 import sys
 
 try:
@@ -80,6 +83,7 @@ def _try_import_from_store(module, abstract_spec_str):
         sys.path.extend(module_paths)
 
         try:
+            _fix_ext_suffix(candidate_spec)
             if _python_import(module):
                 msg = ('[BOOTSTRAP MODULE {0}] The installed spec "{1}/{2}" '
                        'provides the "{0}" Python module').format(
@@ -100,6 +104,71 @@ def _try_import_from_store(module, abstract_spec_str):
     return False
 
 
+def _fix_ext_suffix(candidate_spec):
+    """Fix the external suffixes of Python extensions on the fly for
+    platforms that may need it
+
+    Args:
+        candidate_spec (Spec): installed spec with a Python module
+            to be checked.
+    """
+    # Here we map target families to the patterns expected
+    # by pristine CPython. Only architectures with known issues
+    # are included. Known issues:
+    #
+    # [RHEL + ppc64le]: https://github.com/spack/spack/issues/25734
+    #
+    _suffix_to_be_checked = {
+        'ppc64le': {
+            'glob': '*.cpython-*-powerpc64le-linux-gnu.so',
+            're': r'.cpython-[\w]*-powerpc64le-linux-gnu.so',
+            'fmt': r'{module}.cpython-{major}{minor}m-powerpc64le-linux-gnu.so'
+        }
+    }
+
+    # If the current architecture is not problematic return
+    generic_target = archspec.cpu.host().family
+    if str(generic_target) not in _suffix_to_be_checked:
+        return
+
+    # If there's no EXT_SUFFIX (Python < 3.5) or the suffix matches
+    # the expectations, return since the package is surely good
+    ext_suffix = sysconfig.get_config_var('EXT_SUFFIX')
+    if ext_suffix is None:
+        return
+
+    expected = _suffix_to_be_checked[str(generic_target)]
+    if fnmatch.fnmatch(ext_suffix, expected['glob']):
+        return
+
+    # If we are here it means the current interpreter expects different names
+    # than pristine CPython. So:
+    # 1. Find what we have installed
+    # 2. Create symbolic links for the other names, it they're not there already
+
+    # Check if standard names are installed and if we have to create
+    # link for this interpreter
+    standard_extensions = fs.find(candidate_spec.prefix, expected['glob'])
+    link_names = [re.sub(expected['re'], ext_suffix,  s) for s in standard_extensions]
+    for file_name, link_name in zip(standard_extensions, link_names):
+        if os.path.exists(link_name):
+            continue
+        os.symlink(file_name, link_name)
+
+    # Check if this interpreter installed something and we have to create
+    # links for a standard CPython interpreter
+    non_standard_extensions = fs.find(candidate_spec.prefix, '*' + ext_suffix)
+    for abs_path in non_standard_extensions:
+        directory, filename = os.path.split(abs_path)
+        module = filename.split('.')[0]
+        link_name = os.path.join(directory, expected['fmt'].format(
+            module=module, major=sys.version_info[0], minor=sys.version_info[1])
+        )
+        if os.path.exists(link_name):
+            continue
+        os.symlink(abs_path, link_name)
+
+
 @_bootstrapper(type='buildcache')
 class _BuildcacheBootstrapper(object):
     """Install the software needed during bootstrapping from a buildcache."""


Test patch: