Repository: spack/spack

Issue:
Installation issue: `Cython compiler 'cython' cannot compile programs` when building `py-scipy`
### Steps to reproduce the issue

I think I'm running into two different issues with compiling `py-scipy`.  First of all, due to https://github.com/serge-sans-paille/pythran/issues/2101 I'm running into the error
```
AttributeError: type object 'PatternTransform' has no attribute 'typePatterns'. Did you mean: 'CallPatterns'?
```
reported at https://github.com/scipy/scipy/issues/18390.  I believe this one should be worked around by adding
```py
    # https://github.com/serge-sans-paille/pythran/issues/2101
    conflicts("python@3.11:", when="@:0.12.1")
```
to the `py-pythran` recipe in Spack.  However, after that I still get another compilation error

```console
$ spack -e . spec py-scipy                                                                                                                                     
Input spec                                                                                                                                                                                                        
--------------------------------                                                                                                                                                                                  
 -   py-scipy                                                                                                                                                                                                     
                                                                                                                                                                                                                  
Concretized                                                                                                                                                                                                       
--------------------------------                                                                                                                                                                                  
 -   py-scipy@1.11.4%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512                                                                                                                           
[+]      ^openblas@0.3.24%gcc@12.3.0~bignuma~consistent_fpcsr+fortran~ilp64+locking+pic+shared build_system=makefile symbol_suffix=none threads=none arch=linux-rhel7-skylake_avx512                              
[+]          ^gmake@4.4.1%gcc@12.3.0~guile build_system=generic arch=linux-rhel7-skylake_avx512                                                                                                                   
[+]          ^perl@5.38.0%gcc@12.3.0+cpanm+opcode+open+shared+threads build_system=generic patches=714e4d1 arch=linux-rhel7-skylake_avx512                                                                        
[+]              ^berkeley-db@18.1.40%gcc@12.3.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-rhel7-skylake_avx512                                                                      
[+]      ^pkgconf@1.9.5%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512                                                                                                                         
[+]      ^py-cython@0.29.36%gcc@12.3.0 build_system=python_pip patches=c4369ad arch=linux-rhel7-skylake_avx512
[+]          ^py-setuptools@63.4.3%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]      ^py-meson-python@0.13.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]          ^meson@1.2.1%gcc@12.3.0 build_system=python_pip patches=0f0b1bd,ae59765 arch=linux-rhel7-skylake_avx512
[+]          ^py-pyproject-metadata@0.7.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]              ^py-packaging@23.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]          ^py-tomli@2.0.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]              ^py-flit-core@3.9.0%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]      ^py-numpy@1.24.3%gcc@12.3.0 build_system=python_pip patches=873745d arch=linux-rhel7-skylake_avx512
[+]      ^py-pip@23.1.2%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]      ^py-pybind11@2.11.0%gcc@12.3.0~ipo build_system=cmake build_type=Release generator=ninja arch=linux-rhel7-skylake_avx512
[+]          ^cmake@3.27.7%gcc@12.3.0~doc+ncurses+ownlibs build_system=generic build_type=Release arch=linux-rhel7-skylake_avx512
[+]              ^curl@8.4.0%gcc@12.3.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-rhel7-skylake_avx512
[+]                  ^nghttp2@1.57.0%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^ninja@1.11.1%gcc@12.3.0+re2c build_system=generic arch=linux-rhel7-skylake_avx512
[+]              ^python@3.11.6%gcc@12.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=13fa8bf,b0615b2,ebdca64,f2fd060 arch=linux-rhel7-skylake_avx512
[+]              ^re2c@2.2%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]      ^py-pythran@0.12.0%gcc@12.3.0 build_system=python_pip patches=5ccf989 arch=linux-rhel7-skylake_avx512
[+]          ^py-beniget@0.4.1%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]          ^py-gast@0.5.4%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]          ^py-ply@3.11%gcc@12.3.0 build_system=python_pip arch=linux-rhel7-skylake_avx512
[+]      ^py-wheel@0.41.2%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]      ^python@3.10.13%gcc@12.3.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,ebdca64,f2fd060 arch=linux-rhel7-skylake_avx512
[+]          ^bzip2@1.0.8%gcc@12.3.0~debug~pic+shared build_system=generic arch=linux-rhel7-skylake_avx512
[+]              ^diffutils@3.9%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^expat@2.5.0%gcc@12.3.0+libbsd build_system=autotools arch=linux-rhel7-skylake_avx512
[+]              ^libbsd@0.11.7%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]                  ^libmd@1.0.4%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^gdbm@1.23%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^gettext@0.22.3%gcc@12.3.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-rhel7-skylake_avx512
[+]              ^libiconv@1.17%gcc@12.3.0 build_system=autotools libs=shared,static arch=linux-rhel7-skylake_avx512
[+]              ^libxml2@2.10.3%gcc@12.3.0+pic~python+shared build_system=autotools arch=linux-rhel7-skylake_avx512
[+]              ^tar@1.34%gcc@12.3.0 build_system=autotools zip=pigz arch=linux-rhel7-skylake_avx512
[+]                  ^pigz@2.7%gcc@12.3.0 build_system=makefile arch=linux-rhel7-skylake_avx512
[+]                  ^zstd@1.5.5%gcc@12.3.0+programs build_system=makefile compression=none libs=shared,static arch=linux-rhel7-skylake_avx512
[+]          ^libffi@3.4.4%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^libxcrypt@4.4.35%gcc@12.3.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-rhel7-skylake_avx512
[+]          ^ncurses@6.4%gcc@12.3.0~symlinks+termlib abi=none build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^openssl@3.1.3%gcc@12.3.0~docs+shared build_system=generic certs=mozilla arch=linux-rhel7-skylake_avx512
[+]              ^ca-certificates-mozilla@2023-05-30%gcc@12.3.0 build_system=generic arch=linux-rhel7-skylake_avx512
[+]          ^readline@8.2%gcc@12.3.0 build_system=autotools patches=bbf97f1 arch=linux-rhel7-skylake_avx512
[+]          ^sqlite@3.43.2%gcc@12.3.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^util-linux-uuid@2.38.1%gcc@12.3.0 build_system=autotools arch=linux-rhel7-skylake_avx512
[+]          ^xz@5.4.1%gcc@12.3.0~pic build_system=autotools libs=shared,static arch=linux-rhel7-skylake_avx512
[+]          ^zlib-ng@2.1.4%gcc@12.3.0+compat+opt build_system=autotools arch=linux-rhel7-skylake_avx512
```


### Error message

<details><summary>Error message</summary>

<pre>

  ../meson.build:1:0: ERROR: Cython compiler 'cython' cannot compile programs

  A full log can be found at /lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/build_stage/spack-stage-py-scipy-1.11.4-eys3x6zg35sw4g2wijiyofnknd2nyjx6/spack-src/build/meson-logs/meson-log.tx
t
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Exception information:
Traceback (most recent call last):
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_internal/operations/build/metadata.p
y", line 35, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 713, i
n prepare_metadata_for_build_wheel
    return super().prepare_metadata_for_build_wheel(
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", li
ne 186, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", li
ne 311, in _call_hook
    self._subprocess_runner(
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_internal/utils/subprocess.py", line 
252, in runner
    call_subprocess(
  File "/lustre/scratch/scratch/cceamgi/repo/hpc-spack-0.21/mg-initial-stack/spack/opt/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
ath_placeholder__/__spack_path_placeholder__/__spack_p/linux-rhel7-skylake_avx512/gcc-12.3.0/py-pip-23.1.2-uaklrl2rqg3i2kg5xgliq6qjq4emp73f/lib/python3.10/site-packages/pip/_internal/utils/subprocess.py", line 
224, in call_subprocess
    raise error
pip._internal.exceptions.InstallationSubprocessError: Preparing metadata (pyproject.toml) exited with 1
</pre></details>


### Information on your system

* **Spack:** 0.21.1 (e30fedab102f9281a220fb4fae82e3f8c43a82ac)
* **Python:** 3.6.8
* **Platform:** linux-rhel7-skylake_avx512
* **Concretizer:** clingo

### Additional information

[spack-build-env.txt](https://github.com/spack/spack/files/14462311/spack-build-env.txt)
[spack-build-out.txt](https://github.com/spack/spack/files/14462312/spack-build-out.txt)

CC: @adamjstewart  @rgommers

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
Adding the conflict seems correct to me.

> to the py-pythran recipe in Spack.  However, after that I still get another compilation error

The concretization above and the attached `spack-build-env.txt` still show Pythran 0.12.0 being used for Python 3.10. It's not clear from the logs what the actual problem is - could be the same one, or a different one.

Pythran 0.12.0 won't be tested well (if at all) with SciPy 1.11.4, since the metadata in https://github.com/scipy/scipy/blob/maintenance/1.11.x/pyproject.toml shows:
```toml
    # Reason for `<`: future-proofing (0.14.0 released and known to work)
    "pythran>=0.12.0,<0.15.0",
```

`pip` is always going to be choosing the highest version in that range, so I would not be surprised if 0.12.0 turns out to have an issue. Could you try bumping the lower bound for `py-pythran` in the `py-scipy` issue until your problem goes away?

> The concretization above and the attached `spack-build-env.txt` still show Pythran 0.12.0 being used for Python 3.10.

Yeah, this is part of a larger environment (the goal is to try and build `py-alphafold@2.2.4 +cuda`, but it seems to be an almost impossible task), so there are other constraints involved.  If I try to do `py-alphafold@2.2.4 +cuda ^py-pythran@0.12.2` the environment refuses to concretize before of other conflicts somewhere else.

> Could you try bumping the lower bound for `py-pythran` in the `py-scipy` issue until your problem goes away?

Sorry, I'm not sure what you're suggesting exactly.  Are you suggesting to change https://github.com/spack/spack/blob/763f444d632516cc14db800f3306bba441faec31/var/spack/repos/builtin/packages/py-scipy/package.py#L81 to require, for example, `py-pythran@0.12.2:`?  I'm not sure that'd work because of the problem above with other `py-alphafold` deps.
You can also try #42213 which adds newer versions of many of these packages.
> to require, for example, `py-pythran@0.12.2:`? I'm not sure that'd work because of the problem above with other `py-alphafold` deps.

Yes, that is what I was suggesting indeed. `py-pythran` is only a build dependency of `py-scipy`, so I was hoping that the separate per-package concretization of build dependencies would avoid any conflicts here. Not sure if that feature is opt-in or if it works differently than I imagined if there are still conflicts.
> Not sure if that feature is opt-in or if it works differently than I imagined if there are still conflicts.

See https://spack.readthedocs.io/en/latest/build_settings.html#duplicate-nodes

We would need to add `tags = ["build-tools"]` to the package to make it the default behavior, and only in Spack 0.21+
Ok, the incompatibility of `py-pythran@0.12.2` with `py-alphafold@2.2.4` may have been somehow (newer versions of other packages?) fixed in the `develop` branch of Spack, because in spack v0.21.1 I can't concretize `spack spec --fresh py-alphafold@2.2.4 ^py-scipy@1.11.4 ^py-pythran@0.12.2`, while this spec is successfully concretized in `develop`.  However I can't really test building `py-alphafold@2.2.4` is successful on `develop`, my current focus is trying to get a stack working with a stable version of spack and backporting dozens of recipes of python packages isn't feasible (I'm backporting only the few ones I can handle).

I'll open a PR to add the incompatibility of `py-pythran@:0.12.1` with `python@3.11:` and will consider this issue fixed for the time being.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-pythran/package.py b/var/spack/repos/builtin/packages/py-pythran/package.py
--- a/var/spack/repos/builtin/packages/py-pythran/package.py
+++ b/var/spack/repos/builtin/packages/py-pythran/package.py
@@ -67,6 +67,8 @@ class PyPythran(PythonPackage):
 
     # https://github.com/serge-sans-paille/pythran/issues/1937
     conflicts("%apple-clang@13:", when="@:0.10")
+    # https://github.com/serge-sans-paille/pythran/issues/2101
+    conflicts("^python@3.11:", when="@:0.12.1")
 
     @property
     def headers(self):


Test patch: