Repository: spack/spack

Issue:
Again py-scipy with Intel issues - undefined symbols
### Steps to reproduce

I am building py-scipy with Intel and I am still having problems with undefined symbols as mentioned in earlier issues that were closed as completed. The problem is that during the py-scipy install it uses `ifort` as Fortran compiler because of https://github.com/spack/spack/blob/d8e566d5542e2c8f0508a83922ba1eada5d473cd/var/spack/repos/builtin/packages/py-scipy/package.py#L124 - side note: but with the `gfortran` compiler flags, so that doesn't seem right either -, but when it links the Fortran extension it uses `gfortran`:
```
...
INFO: compiling Fortran sources
  INFO: Fortran f77 compiler: /gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/spack/lib/spack/env/intel/ifort -Wall -g -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
  Fortran f90 compiler: /gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/spack/lib/spack/env/intel/ifort -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
  Fortran fix compiler: /gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/spack/lib/spack/env/intel/ifort -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -fPIC -O3 -funroll-loops
  INFO: compile options: '-DHAVE_CBLAS -Ibuild/src.linux-x86_64-3.9/build/src.linux-x86_64-3.9/scipy/integrate -I/gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/envs/py-scipy-intel-test/install/intel/2022.0.1/py-numpy-1.22.3-vkrg7fi/
lib/python3.9/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.9/numpy/distutils/include -I/discover/swdev/jcsda/spack-stack/miniconda-3.9.7/include/python3.9 -c'
  INFO: ifort:f77: build/src.linux-x86_64-3.9/scipy/integrate/_test_odeint_banded-f2pywrappers.f
  INFO: ifort:f77: scipy/integrate/tests/banded5x5.f
  ifort: command line warning #10157: ignoring option '-W'; argument is of wrong type
  ifort: command line warning #10006: ignoring unknown option '-ffixed-form'
  ifort: command line warning #10006: ignoring unknown option '-fno-second-underscore'
  ifort: command line warning #10157: ignoring option '-W'; argument is of wrong type
  ifort: command line warning #10006: ignoring unknown option '-ffixed-form'
  ifort: command line warning #10006: ignoring unknown option '-fno-second-underscore'
  INFO: /gpfsm/dulocal/sles12/other/gcc/10.1.0/bin/gfortran -Wall -g -Wall -g -shared build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/scipy/integrate/_test_odeint_bandedmodule.o build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/build/src.linux-x86_64-3.9/scipy/integrate/fortranobject.o build/temp.linux-x86_64-3.9/scipy/integrate/tests/banded5x5.o build/temp.linux-x86_64-3.9/build/src.linux-x86_64-3.9/scipy/integrate/_test_odeint_banded-f2pywrappers.o -L/gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/envs/py-scipy-intel-test/install/intel/2022.0.1/openblas-0.3.19-2tgjxxs/lib -Lbuild/temp.linux-x86_64-3.9 -Wl,-rpath=/gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/envs/py-scipy-intel-test/install/intel/2022.0.1/openblas-0.3.19-2tgjxxs/lib -llsoda -lmach -lopenblas -lopenblas -lgfortran -o build/lib.linux-x86_64-3.9/scipy/integrate/_test_odeint_banded.cpython-39-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.9/link-version-scipy.integrate._test_odeint_banded.map
  INFO: building 'scipy.interpolate.interpnd' extension
  INFO: compiling C sources
  INFO: C compiler: /gpfsm/dnb32/dheinzel/spack-stack/spack-stack-scipy-intel/spack/lib/spack/env/intel/icc -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /discover/swdev/jcsda/spack-stack/miniconda-3.9.7/include -I/discover/swdev/jcsda/spack-stack/miniconda-3.9.7/include -fPIC -O2 -isystem /discover/swdev/jcsda/spack-stack/miniconda-3.9.7/include -fPIC
...
```

### Error message

```
python3
from scipy.stats import skew
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/stats/__init__.py", line 453, in <module>
    from ._stats_py import *
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/stats/_stats_py.py", line 38, in <module>
    from scipy.spatial.distance import cdist
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/spatial/__init__.py", line 108, in <module>
    from ._geometric_slerp import geometric_slerp
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/spatial/_geometric_slerp.py", line 9, in <module>
    from scipy.spatial.distance import euclidean
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/spatial/distance.py", line 123, in <module>
    from ..special import rel_entr
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/special/__init__.py", line 652, in <module>
    from . import _basic
  File "/gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/special/_basic.py", line 15, in <module>
    from . import _specfun
ImportError: /gpfsm/dswdev/jcsda/spack-stack/spack-stack-v1/envs/skylab-1.0.0-intel-2022.0.1/install/intel/2022.0.1/py-scipy-1.8.0-eiuk7wc/lib/python3.9/site-packages/scipy/special/_specfun.cpython-39-x86_64-linux-gnu.so: undefined symbol: __svml_pow4_l9
```
Similar errors on other systems when using Intel.

### Information on your system

Both on HPC (Nasa Discover) and on AWS EC2 with different versions of the Intel compiler (2021.4.0 and 2022.0.1), `py-scipy@1.8.0`.

```
[py-scipy-intel-test] dheinzel@discover32:/discover/nobackup/dheinzel/spack-stack/spack-stack-scipy-intel> spack debug report
* **Spack:** 0.19.0.dev0 (aea27b394b6b2dc184d8ca3718fd57308d7312e8)
* **Python:** 3.9.7
* **Platform:** linux-sles12-haswell
* **Concretizer:** clingo
```

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
@rgommers can you take a look at this?
@climbfuji could you point out these previous issues? I 

> `_specfun.cpython-39-x86_64-linux-gnu.so: undefined symbol: __svml_pow4_l9`

I haven't seen that one before I think. I just searched for SVML and can't find anything in the issue tracker. 

`_specfun` should be linking `npymath`, but that should not include SVML. My first guess is that something went wrong in the `numpy`  build.
Here is a list of issues that I think are related, and I agree it has to do with numpy:
- https://github.com/spack/spack/issues/8616
- https://github.com/numpy/numpy/issues/14146
- https://github.com/numpy/numpy/pull/13132
- https://github.com/spack/spack/pull/12170
- https://github.com/spack/spack/issues/10361
- https://github.com/scipy/scipy/issues/13853
> The problem is that during the py-scipy install it uses `ifort` as Fortran compiler because of
> 
> https://github.com/spack/spack/blob/d8e566d5542e2c8f0508a83922ba1eada5d473cd/var/spack/repos/builtin/packages/py-scipy/package.py#L124

This is probably a bug in the Spack package build definition. Just setting `F90` is not enough I think, and that `gfortran` flags are being used seems to confirm that. It should be something like:
```
python setup.py config --compiler=intel --fcompiler=ifort install
```
or even
```
# Could be `intel` or `intelem` compiler, there's quite a few flavors here in numpy.distutils
python setup.py config --compiler=intel --fcompiler=ifort build_clib --compiler=intel --fcompiler=intelem install
```
I'll note that that is being taken care of for the Fujitsu compiler, but not for Intel Fortran: https://github.com/spack/spack/blob/d8e566d5542e2c8f0508a83922ba1eada5d473cd/var/spack/repos/builtin/packages/py-scipy/package.py#L116-L120

@climbfuji would you be able to provide the exact spack build/install command to reproduce this? 
@rgommers I was under the impression that `python setup.py ...` was deprecated and everything should go through a build frontend like pip instead. Happy to expand `setup.cfg` to set the correct options for other compilers. What is the full list of compilers that are accepted inputs, and which keys can take arbitrary values? Spack supports a lot of uncommon compilers...
```python
$ spack python
Spack version 0.19.0.dev0
Python 3.8.9, Darwin x86_64
>>> import spack
>>> spack.compilers.supported_compilers()
['aocc', 'apple-clang', 'arm', 'cce', 'clang', 'dpcpp', 'fj', 'gcc', 'intel', 'msvc', 'nag', 'nvhpc', 'oneapi', 'pgi', 'rocmcc', 'xl', 'xl_r']
```
> @rgommers I was under the impression that `python setup.py ...` was deprecated and everything should go through a build frontend like pip instead. Happy to expand `setup.cfg` to set the correct options for other compilers

Ah yes, `setup.cfg` works for this too - and is indeed technically nicer. `setup.py` deprecated, yeah technically setuptools maintainers have been saying that, but it's not going anywhere any time soon. And for this kind of argument passing I find it easier than figuring out how to get `pip` to pass on the options.

> What is the full list of compilers that are accepted inputs, and which keys can take arbitrary values? Spack supports a lot of uncommon compilers

The way these names are defined is via the `compiler_type` attribute of compiler classes in `numpy.distutils`, e.g. https://github.com/numpy/numpy/blob/main/numpy/distutils/intelccompiler.py#L12. As you can see in that file, there's a bunch of different names for various flavors of using the Intel compilers (I know, it's really bad) with different ways of having Python built. Not sure what of that is relevant here - I think Spack will build Python with the same compilers, and most tweaks are for a Python built with GCC/MSVC. I think `intel` and `ifort` will do. There's also Fortran compiler flavors at https://github.com/numpy/numpy/blob/main/numpy/distutils/fcompiler/intel.py#L32.

The `arm`, `fujitsu` and `nag` compilers at https://github.com/numpy/numpy/tree/main/numpy/distutils/fcompiler are probably still relevant; the rest I have no idea. 

I think we probably want to make the minimal change necessary to make Intel compilers work with SciPy 1.8.x-1.9.1 here. This stuff is all going to go away when the Meson build lands. And I'm actually in the process of adding CI for `ifort` and MKL: https://github.com/scipy/scipy/pull/16957.
Thank you both for looking at this. I was going in a similar direction, albeit not in the spack framework itself, but in the packages. I'll better wait for you to do this the right way!
> > The problem is that during the py-scipy install it uses `ifort` as Fortran compiler because of
> > https://github.com/spack/spack/blob/d8e566d5542e2c8f0508a83922ba1eada5d473cd/var/spack/repos/builtin/packages/py-scipy/package.py#L124
> 
> This is probably a bug in the Spack package build definition. Just setting `F90` is not enough I think, and that `gfortran` flags are being used seems to confirm that. It should be something like:
> 
> ```
> python setup.py config --compiler=intel --fcompiler=ifort install
> ```
> 
> or even
> 
> ```
> # Could be `intel` or `intelem` compiler, there's quite a few flavors here in numpy.distutils
> python setup.py config --compiler=intel --fcompiler=ifort build_clib --compiler=intel --fcompiler=intelem install
> ```
> 
> I'll note that that is being taken care of for the Fujitsu compiler, but not for Intel Fortran:
> 
> https://github.com/spack/spack/blob/d8e566d5542e2c8f0508a83922ba1eada5d473cd/var/spack/repos/builtin/packages/py-scipy/package.py#L116-L120
> 
> @climbfuji would you be able to provide the exact spack build/install command to reproduce this?

Just saw that. I added the intelem option in the same place before I posted this issue, but that didn't help. I am now doing essentially the same for py-numpy, too. And set LDSHARED="icc -shared" as well. Trying to throw everything at it and see what happens.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-scipy/package.py b/var/spack/repos/builtin/packages/py-scipy/package.py
--- a/var/spack/repos/builtin/packages/py-scipy/package.py
+++ b/var/spack/repos/builtin/packages/py-scipy/package.py
@@ -118,6 +118,15 @@ def set_fortran_compiler(self):
             with open("setup.cfg", "w") as f:
                 f.write("[config_fc]\n")
                 f.write("fcompiler = fujitsu\n")
+        elif self.spec.satisfies("%intel") or self.spec.satisfies("%oneapi"):
+            if self.spec.satisfies("target=x86:"):
+                with open("setup.cfg", "w") as f:
+                    f.write("[config_fc]\n")
+                    f.write("fcompiler = intel\n")
+            elif self.spec.satisfies("target=x86_64:"):
+                with open("setup.cfg", "w") as f:
+                    f.write("[config_fc]\n")
+                    f.write("fcompiler = intelem\n")
 
     def setup_build_environment(self, env):
         # https://github.com/scipy/scipy/issues/9080
@@ -129,6 +138,11 @@ def setup_build_environment(self, env):
             if self.spec.satisfies("^py-numpy@1.16:1.17"):
                 env.set("NPY_DISTUTILS_APPEND_FLAGS", "1")
 
+        # https://github.com/scipy/scipy/issues/14935
+        if self.spec.satisfies("%intel ^py-pythran") or self.spec.satisfies("%oneapi ^py-pythran"):
+            if self.spec["py-pythran"].version < Version("0.12"):
+                env.set("SCIPY_USE_PYTHRAN", "0")
+
         # Pick up Blas/Lapack from numpy
         self.spec["py-numpy"].package.setup_build_environment(env)
 


Test patch: