Repository: scipy/scipy
Pull request: BUG/CI: Compile and run tests with `ifx` + `MKL` on Linux
<!-- 
Thanks for contributing a pull request! Please ensure that
your PR satisfies the checklist before submitting:
http://scipy.github.io/devdocs/dev/contributor/development_workflow.html#checklist-before-submitting-a-pr

Also, please name and describe your PR as you would write a
commit message:
http://scipy.github.io/devdocs/dev/contributor/development_workflow.html#writing-the-commit-message.
However, please only include an issue number in the description, not the title,
and please ensure that any code names containing underscores are enclosed in backticks.

Depending on your changes, you can skip CI operations and save time and energy: 
http://scipy.github.io/devdocs/dev/contributor/continuous_integration.html#skipping

Note that we are a team of volunteers; we appreciate your
patience during the review process.

Again, thanks for contributing!
-->

#### Reference issue
<!--Example: Closes gh-WXYZ.-->

Fixes https://github.com/scipy/scipy/issues/20728

#### What does this implement/fix?
<!--Please explain your changes.-->

#### Additional information
<!--Any additional information you think is important.-->

@rgommers I am copy-pasting your reviews here which I will address next week. I think they are for ``PROPACK``. Will send a separate PR there.

1. in the first commit, splitting whitespace changes from a ``cdotc`` change would be good
2. in the second commit, keeping ``ccdotc`` a function rather than really inlining it (just like arpack-ng did) would be better for readability
3. https://github.com/scipy/scipy/issues/20728#issuecomment-2226342356


Problem statement:
BUG: sparse.linalg: Segfault in `arpack` with `ifx`
### Describe your issue.

I am installing scipy from source on an HPC on top of python 3.11.6. The code builds without errors, but `scipy.test()` segfaults at `scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py`. We have intel MKL on the HPC, so I built scipy with 
```
CC=icx CXX=icpx FC=ifx python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp
```

### Reproducing Code Example

```python
python dev.py test -s sparse
```


### Error message

```shell
ninja: Entering directory `/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build'
[4/4] Generating scipy/generate-version with a custom command
Build OK
💻  meson install -C build --only-changed
Installing, see meson-install.log...
Installation OK
SciPy from development installed path at: /work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages
Running tests for scipy version:1.13.0, installed at:/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.11.6, pytest-7.4.3, pluggy-1.3.0
rootdir: /work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0
configfile: pytest.ini
plugins: hypothesis-6.102.4
collected 11638 items / 61 deselected / 11577 selected

scipy/sparse/csgraph/tests/test_connected_components.py ........                                                                                                             [  0%]
scipy/sparse/csgraph/tests/test_conversions.py ...                                                                                                                           [  0%]
scipy/sparse/csgraph/tests/test_flow.py ...........................................                                                                                          [  0%]
scipy/sparse/csgraph/tests/test_graph_laplacian.py ......................................................................................................................... [  1%]
............................................................................................................................................................................ [  2%]
............................................................................................................................................................................ [  4%]
............................................................................................................................................................................ [  5%]
............................................................................................................................................................................ [  7%]
............................................................................................................................................................................ [  8%]
.................................................................................................................................................                            [ 10%]
scipy/sparse/csgraph/tests/test_matching.py ................................                                                                                                 [ 10%]
scipy/sparse/csgraph/tests/test_pydata_sparse.py ssssssssssssssssssssssssssssssssss                                                                                          [ 10%]
scipy/sparse/csgraph/tests/test_reordering.py ...                                                                                                                            [ 10%]
scipy/sparse/csgraph/tests/test_shortest_path.py ...................................                                                                                         [ 11%]
scipy/sparse/csgraph/tests/test_spanning_tree.py .                                                                                                                           [ 11%]
scipy/sparse/csgraph/tests/test_traversal.py ........                                                                                                                        [ 11%]
scipy/sparse/linalg/_dsolve/tests/test_linsolve.py .s.s.s.s.ss..........s.....................                                                                               [ 11%]
scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py .Fatal Python error: Segmentation fault

Current thread 0x0000146b7118b740 (most recent call first):
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 883 in extract
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1357 in eigs
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1577 in eigsh
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 248 in eval_evec
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py", line 425 in test_hermitian_modes
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/_lib/_testutils.py", line 106 in __call__
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 745 in scipy_tests
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 761 in run
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/action.py", line 461 in execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/task.py", line 473 in execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 176 in execute_task
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 220 in run_tasks
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/runner.py", line 254 in run_all
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_run.py", line 265 in _execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_base.py", line 570 in execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/cmd_base.py", line 150 in parse_execute
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/doit/api.py", line 50 in run_tasks
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 251 in run_doit_task
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/pydevtool/cli.py", line 267 in callback
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 783 in invoke
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1434 in invoke
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1688 in invoke
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/rich_click/rich_command.py", line 126 in main
  File "/work2/noaa/co2/sbasu/packages/python/3.11.6/lib/python3.11/site-packages/click/core.py", line 1157 in __call__
  File "/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/dev.py", line 1486 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy._lib._fpumode, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, scipy.special._ufuncs_cxx, scipy.special._cdflib, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2 (total: 50)
Segmentation fault (core dumped)
```


### SciPy/NumPy/Python version and system information

```shell
Python 3.11.6 (main, May 15 2024, 11:36:54) [GCC Intel(R) C++ gcc 11.3.1 mode] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, scipy, numpy; print(scipy.__version__, numpy.__version__, sys.version_info); scipy.show_config()
1.13.0 1.26.4 sys.version_info(major=3, minor=11, micro=6, releaselevel='final', serial=0)
/work2/noaa/co2/sbasu/packages/sources/scipy-1.13.0/build-install/lib/python3.11/site-packages/scipy/__config__.py:154: UserWarning: Install `pyyaml` for better output
  warnings.warn("Install `pyyaml` for better output", stacklevel=1)
{
  "Compilers": {
    "c": {
      "name": "intel-llvm",
      "linker": "ld.bfd",
      "version": "11.3.1",
      "commands": "icx"
    },
    "cython": {
      "name": "cython",
      "linker": "cython",
      "version": "3.0.10",
      "commands": "cython"
    },
    "c++": {
      "name": "intel-llvm",
      "linker": "ld.bfd",
      "version": "11.3.1",
      "commands": "icpx"
    },
    "fortran": {
      "name": "intel-llvm",
      "linker": "ld.bfd",
      "version": "2023.1.0",
      "commands": "ifx"
    },
    "pythran": {
      "version": "0.14.0",
      "include directory": "../../../python/3.11.6/lib/python3.11/site-packages/pythran"
    }
  },
  "Machine Information": {
    "host": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "build": {
      "cpu": "x86_64",
      "family": "x86_64",
      "endian": "little",
      "system": "linux"
    },
    "cross-compiled": true
  },
  "Build Dependencies": {
    "blas": {
      "name": "mkl-dynamic-lp64-iomp",
      "found": true,
      "version": "2023.1",
      "detection method": "pkgconfig",
      "include directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../include",
      "lib directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../lib/intel64",
      "openblas configuration": "unknown",
      "pc file directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/2023.1.0/lib/pkgconfig"
    },
    "lapack": {
      "name": "mkl-dynamic-lp64-iomp",
      "found": true,
      "version": "2023.1",
      "detection method": "pkgconfig",
      "include directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../include",
      "lib directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/latest/lib/pkgconfig/../../lib/intel64",
      "openblas configuration": "unknown",
      "pc file directory": "/apps/spack-managed/gcc-11.3.1/intel-oneapi-mkl-2023.1.0-4cujjco7etbwl34hwrtw3ree7dwhxnci/mkl/2023.1.0/lib/pkgconfig"
    },
    "pybind11": {
      "name": "pybind11",
      "version": "2.12.0",
      "detection method": "config-tool",
      "include directory": "unknown"
    }
  },
  "Python Information": {
    "path": "/work2/noaa/co2/sbasu/packages/python/3.11.6/bin/python3.11",
    "version": "3.11"
  }
}
```

Hints from issue discussion:
Thanks for the report @AgilentGCMS. That ARPACK code has had recurring issues with some BLAS functions (`cdotc` & co.) that are sensitive to Fortran compiler ABI conventions. MKL is built with the opposite convention as the default one (gfortran).

Maybe this particular crash was always an issue with MKL from OneAPI, or maybe it changed with https://github.com/scipy/scipy/pull/20232 (that was included in 1.13.0, which you're testing here). Do you know if this is a regression from 1.12.x or not for your setup?
I can reproduce this on Windows with scipy-1.13.1 and OneAPI MKL 2024.1. 
Thanks @cgohlke for confirming. So it's a problem that's not platform-dependent.

I'll add a bit of context, since it may not be worth starting to debug here - the maintenance state of our ARPACK copy is very bad:
- We're at ARPACK 3.3.0, from 2015 (last upgrade done in gh-5901). So it's about 9 years and 6 minor releases behind https://github.com/opencollab/arpack-ng. Open PRs to upgrade were never finished: gh-10375, gh-14786. I think a lot of issues may be hiding in ARPACK that have been solved upstream. The upgrade to 3.7.0 in the first PR did also uncover some regressions which made it harder to finish that PR.
- The end goal is to get rid of ARPACK (and PROPACK) completely, as discussed in gh-18566.

It's also still possible that this is an MKL bug. We do test with Accelerate, which uses the same ABI conventions as MKL, and things work as expected there.
Seems like I am able to reproduce the segmentation fault with `FC=ifx` and C/C++ compilers from conda. `ifx` + `MKL` and `ifx` + `OpenBLAS` combination fails with the same segmentation fault. However, `gfortran` + `MKL` combination works just fine. Will try with `ifort` tomorrow.

My conclusion is that `ifx` usage gives problems, `MKL` doesn't.

**Commands**

`python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp` (`sparse` test succeed)

`FC=ifx python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp` (`sparse` test fail with segmentation fault)

`FC=ifx python dev.py build` (`sparse` test fail with segmentation fault)

**Logs**

```bash
SciPy from development installed path at: /home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages
Running tests for scipy version:1.15.0.dev0+1043.bbe993d, installed at:/home/czgdp18079/Quansight/scipy/build-install/lib/python3.12/site-packages/scipy
============================= test session starts ==============================
platform linux -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/czgdp18079/Quansight/scipy
configfile: pytest.ini
plugins: anyio-4.4.0, xdist-3.6.1, cov-5.0.0, hypothesis-6.104.2, timeout-2.3.1
collected 12224 items / 348 deselected / 11876 selected                        

scipy/sparse/csgraph/tests/test_connected_components.py ........         [  0%]
scipy/sparse/csgraph/tests/test_conversions.py ...                       [  0%]
scipy/sparse/csgraph/tests/test_flow.py ................................ [  0%]
...........                                                              [  0%]
scipy/sparse/csgraph/tests/test_graph_laplacian.py ..................... [  0%]
........................................................................ [  1%]
........................................................................ [  1%]
........................................................................ [  2%]
........................................................................ [  3%]
........................................................................ [  3%]
........................................................................ [  4%]
........................................................................ [  4%]
........................................................................ [  5%]
........................................................................ [  6%]
........................................................................ [  6%]
........................................................................ [  7%]
........................................................................ [  7%]
........................................................................ [  8%]
........................................................................ [  9%]
........................................................................ [  9%]
.........................                                                [  9%]
scipy/sparse/csgraph/tests/test_matching.py ............................ [ 10%]
....                                                                     [ 10%]
scipy/sparse/csgraph/tests/test_pydata_sparse.py sssssssssssssssssssssss [ 10%]
sssssssssssssssssssssssssssssssssss                                      [ 10%]
scipy/sparse/csgraph/tests/test_reordering.py ...                        [ 10%]
scipy/sparse/csgraph/tests/test_shortest_path.py ....................... [ 10%]
.................................................                        [ 11%]
scipy/sparse/csgraph/tests/test_spanning_tree.py .                       [ 11%]
scipy/sparse/csgraph/tests/test_traversal.py ........                    [ 11%]
scipy/sparse/linalg/_dsolve/tests/test_linsolve.py .s.s.s.s.ss.......... [ 11%]
s............................                                            [ 11%]
scipy/sparse/linalg/_eigen/arpack/tests/test_arpack.py .Fatal Python error: Segmentation fault

Current thread 0x00007efc3801b740 (most recent call first):
  ...
```
OK I bumped the ARPACK version in #21110. Maybe it can solve this issue too. No idea what the changes are as it still has issues unresolved upstream too. 

Simultaneously, I started rewriting ARPACK work. The code is exactly what anyone would expect from a F77 monolith.
With your PR, the segmentation fault disappears from `_arpack` but comes in `test_svds`,

**Commands**

`python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp` (succeeds with `sparse` module)

`FC=ifx python dev.py build` (`sparse` test fail with segmentation fault)

`FC=ifx python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp` (`sparse` test fail with segmentation fault)

`FC="ifort -diag-disable=10448" python dev.py build` (`sparse` test fail with segmentation fault)

`FC="ifort -diag-disable=10448" python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp` (`sparse` test fail with segmentation fault)

**Logs**

```bash
scipy/sparse/linalg/_eigen/tests/test_svds.py .......................... [ 14%]
................................................s.ss..s.ssss....s.ss..s. [ 15%]
ssss....s.ss..s.ssss....s.ss..s.ssss....s.ss..s.ssss....s.ss..s.ssss.... [ 16%]
............ssssssss.................................................... [ 16%]
............................................s.ss....s.ss....s.ss....s.ss [ 17%]
....s.ss....s.ss....s.ss....s.ss....s.ss....s.ss....s.ss....s.ss........ [ 17%]
........................................................................ [ 18%]
..............................s.Fatal Python error: Fatal Python error: Segmentation fault


```
Sigh... Language of scientific computing. 
I think the problem is with `ifx` and not `MKL`/`OpenBLAS`. With `gfortran`, both `MKL` and `OpenBLAS` pass all the tests in `sparse` module. With `ifx`, both `MKL` and `OpenBLAS` give segmentation fault at the same point.
Could you run the test suite with `-v` option so that we know which test failed? 
Yes. Now I will debug the tests. I wanted to see what all combinations work and what all fail. I have started with `ifx` and `MKL` combination.

One weird observation with the following code (basically an MRE of the failing test),

```python
import numpy as np
from scipy.sparse.linalg import svds

def test_svd_simple(A, k, real, transpose, lo_type):
    A = np.asarray(A)
    A = np.real(A) if real else A
    A = A.T if transpose else A
    A2 = lo_type(A)

    u, s, vh = svds(A2, k, solver='propack', random_state=0)
    print(u, s, vh)

A1 = [[1, 2, 3], [3, 4, 3], [1 + 1j, 0, 2], [0, 0, 1]]
test_svd_simple(A1, 1, False, False, np.asarray)
```

**Randomised behaviour**

Results produced are correct but seems like segmentation fault is random, sometimes it happens sometimes it doesn't. Seems like coming from LLVM generated code by `ifx`.

```bash
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
[[0.39378471+0.32618658j]
 [0.63317198+0.51091137j]
 [0.15324317+0.21149959j]
 [0.06920502+0.05994875j]] [7.06729714] [[0.37610466-0.27127444j 0.46980582-0.38147804j 0.48909242-0.42367566j]]
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
Segmentation fault (core dumped)
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
Segmentation fault (core dumped)
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
Segmentation fault (core dumped)
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
Segmentation fault (core dumped)
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
Segmentation fault (core dumped)
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
Segmentation fault (core dumped)
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
Segmentation fault (core dumped)
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
Segmentation fault (core dumped)
(scipy-dev) czgdp18079@scipy-linux-9:~/Quansight/scipy$ python ../debug.py
[[0.39378471+0.32618658j]
 [0.63317198+0.51091137j]
 [0.15324317+0.21149959j]
 [0.06920502+0.05994875j]] [7.06729714] [[0.37610466-0.27127444j 0.46980582-0.38147804j 0.48909242-0.42367566j]]
```

**Notes**

With `pdb`, no segmentation fault comes up.

Commit message: DEV: Update PROPACK

Patch to review:
diff --git a/.github/workflows/linux_intel_oneAPI.yml b/.github/workflows/linux_intel_oneAPI.yml
new file mode 100644
index 000000000000..0ca443ed7940
--- /dev/null
+++ b/.github/workflows/linux_intel_oneAPI.yml
@@ -0,0 +1,95 @@
+name: Intel Fortran tests
+
+on:
+  push:
+    branches:
+      - maintenance/**
+  pull_request:
+    branches:
+      - main
+      - maintenance/**
+
+permissions:
+   contents: read  # to fetch code (actions/checkout)
+
+# The environment variable LINUX_BASEKIT_URL and LINUX_HPCKIT_URL
+# store the URL for downloading Intel oneAPI.
+# Reference - https://github.com/oneapi-src/oneapi-ci/blob/b4a96bd1888c130fcb73524d2b77b90f43681cbc/.github/workflows/build_all.yml#L11-L12
+env:
+  LINUX_BASEKIT_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/fdc7a2bc-b7a8-47eb-8876-de6201297144/l_BaseKit_p_2024.1.0.596_offline.sh
+  LINUX_HPCKIT_URL: https://registrationcenter-download.intel.com/akdlm/IRC_NAS/7f096850-dc7b-4c35-90b5-36c12abd9eaa/l_HPCKit_p_2024.1.0.560_offline.sh
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
+  cancel-in-progress: true
+
+jobs:
+  get_commit_message:
+    name: Get commit message
+    uses: ./.github/workflows/commit_message.yml
+
+  msvc_ifx_blas:
+    name: py3.12, dev.py
+    needs: get_commit_message
+    # Ensure (a) this doesn't run on forks by default, and
+    #        (b) it does run with Act locally (`github` doesn't exist there)
+    if: >
+      needs.get_commit_message.outputs.message == 1
+      && (github.repository == 'scipy/scipy' || github.repository == '')
+    runs-on: ubuntu-24.04
+    strategy:
+      matrix:
+        python-version: ["3.12"]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4.1.1
+        with:
+          submodules: recursive
+
+      - name: Setup Conda
+        uses: conda-incubator/setup-miniconda@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+          channels: conda-forge
+          channel-priority: true
+          activate-environment: scipy-dev
+          use-only-tar-bz2: false
+          miniforge-variant: Mambaforge
+          miniforge-version: latest
+          use-mamba: true
+
+      - name: Update Conda Environment
+        run: mamba env update -n scipy-dev -f environment.yml
+
+      - name: cache install
+        id: cache-install
+        uses: actions/cache@v3
+        with:
+          path: |
+              /opt/intel/oneapi/
+          key: install-${{ env.LINUX_BASEKIT_URL }}-${{ env.LINUX_HPCKIT_URL }}-compiler
+      - name: Install oneAPI Base kit
+        if: steps.cache-install.outputs.cache-hit != 'true'
+        run: |
+          printenv LINUX_BASEKIT_URL
+          bash tools/install_intel_oneAPI_linux.sh $LINUX_BASEKIT_URL
+      - name: Install oneAPI HPC kit
+        if: steps.cache-install.outputs.cache-hit != 'true'
+        run: |
+          printenv LINUX_HPCKIT_URL
+          bash tools/install_intel_oneAPI_linux.sh $LINUX_HPCKIT_URL
+
+      - name: Initialise Intel oneAPI and Build SciPy
+        shell: bash -l {0}
+        run: |
+          . /opt/intel/oneapi/setvars.sh --force
+          conda activate scipy-dev
+          FC=ifx python dev.py build -C-Dblas=mkl-dynamic-lp64-iomp -C-Dlapack=mkl-dynamic-lp64-iomp
+
+      - name: Test scipy
+        shell: bash -l {0}
+        run: |
+          . /opt/intel/oneapi/setvars.sh --force
+          conda activate scipy-dev
+          python dev.py test
diff --git a/scipy/interpolate/tests/test_fitpack2.py b/scipy/interpolate/tests/test_fitpack2.py
index 407eb0d736c6..02c716d43935 100644
--- a/scipy/interpolate/tests/test_fitpack2.py
+++ b/scipy/interpolate/tests/test_fitpack2.py
@@ -137,8 +137,8 @@ def test_out_of_range_regression(self):
             for ext in [2, 'raise']:
                 assert_raises(ValueError, spl, xp, **dict(ext=ext))
             for ext in [3, 'const']:
-                assert_allclose(spl(xp, ext=ext), xp_clip**3, atol=1e-16)
-                assert_allclose(cls(x, y, ext=ext)(xp), xp_clip**3, atol=1e-16)
+                assert_allclose(spl(xp, ext=ext), xp_clip**3, atol=2e-16)
+                assert_allclose(cls(x, y, ext=ext)(xp), xp_clip**3, atol=2e-16)
 
         # also test LSQUnivariateSpline [which needs explicit knots]
         t = spl.get_knots()[3:4]  # interior knots w/ default k=3
diff --git a/scipy/optimize/meson.build b/scipy/optimize/meson.build
index 626d34ed30c3..1becf75989a7 100644
--- a/scipy/optimize/meson.build
+++ b/scipy/optimize/meson.build
@@ -106,7 +106,7 @@ py3.extension_module('_slsqp',
   [f2py_gen.process('slsqp/slsqp.pyf'), 'slsqp/slsqp_optmz.f'],
   fortran_args: fortran_ignore_warnings,
   link_args: version_link_args,
-  dependencies: [fortranobject_dep],
+  dependencies: [lapack_dep, blas_dep, fortranobject_dep],
   install: true,
   link_language: 'fortran',
   subdir: 'scipy/optimize'
diff --git a/scipy/optimize/slsqp/slsqp_optmz.f b/scipy/optimize/slsqp/slsqp_optmz.f
index ef0f1644d122..b2e00a693e3b 100644
--- a/scipy/optimize/slsqp/slsqp_optmz.f
+++ b/scipy/optimize/slsqp/slsqp_optmz.f
@@ -37,7 +37,7 @@
       SUBROUTINE slsqp (m, meq, la, n, x, xl, xu, f, c, g, a,
      *                  acc, iter, mode, w, l_w, jw, l_jw,
      *                  alpha, f0, gs, h1, h2, h3, h4, t, t0, tol,
-     *                  iexact, incons, ireset, itermx, line, 
+     *                  iexact, incons, ireset, itermx, line,
      *                  n1, n2, n3)
 
 C   SLSQP       S EQUENTIAL  L EAST  SQ UARES  P ROGRAMMING
@@ -263,15 +263,15 @@ SUBROUTINE slsqp (m, meq, la, n, x, xl, xu, f, c, g, a,
       CALL slsqpb  (m, meq, la, n, x, xl, xu, f, c, g, a, acc, iter,
      * mode, w(ir), w(il), w(ix), w(im), w(is), w(iu), w(iv), w(iw), jw,
      * alpha, f0, gs, h1, h2, h3, h4, t, t0, tol,
-     * iexact, incons, ireset, itermx, line, 
+     * iexact, incons, ireset, itermx, line,
      * n1, n2, n3)
- 
+
       END
 
       SUBROUTINE slsqpb (m, meq, la, n, x, xl, xu, f, c, g, a, acc,
      *                   iter, mode, r, l, x0, mu, s, u, v, w, iw,
      *                   alpha, f0, gs, h1, h2, h3, h4, t, t0, tol,
-     *                   iexact, incons, ireset, itermx, line, 
+     *                   iexact, incons, ireset, itermx, line,
      *                   n1, n2, n3)
 
 C   NONLINEAR PROGRAMMING BY SOLVING SEQUENTIALLY QUADRATIC PROGRAMS
@@ -284,6 +284,7 @@ SUBROUTINE slsqpb (m, meq, la, n, x, xl, xu, f, c, g, a, acc,
      *                 k, j, la, line, m, meq, mode, n, n1, n2, n3
       LOGICAL          badlin
 
+
       DOUBLE PRECISION a(la,n+1), c(la), g(n+1), l((n+1)*(n+2)/2),
      *                 mu(la), r(m+n+n+2), s(n+1), u(n+1), v(n+1), w(*),
      *                 x(n), xl(n), xu(n), x0(n),
@@ -631,6 +632,7 @@ SUBROUTINE lsq(m,meq,n,nl,la,l,g,a,b,xl,xu,x,y,w,jw,mode)
 c     coded            Dieter Kraft, april 1987
 c     revised                        march 1989
 
+
       DOUBLE PRECISION l,g,a,b,w,xl,xu,x,y,
      .                 diag,ZERO,one,ddot_sl,xnorm
 
@@ -674,7 +676,7 @@ SUBROUTINE lsq(m,meq,n,nl,la,l,g,a,b,xl,xu,x,y,w,jw,mode)
          CALL dcopy_ (i1-n2, l(i2), 1, w(i3), n)
          CALL dscal_sl (i1-n2,     diag, w(i3), n)
          w(i3) = diag
-         w(IF-1+i) = (g(i) - ddot_sl (i-1, w(i4), 1, w(IF), 1))/diag
+         w(IF-1+i) = (g(i) - ddot_sl(i-1, w(i4), 1, w(IF), 1))/diag
          i2 = i2 + i1 - n2
          i3 = i3 + n1
          i4 = i4 + n
@@ -842,6 +844,7 @@ SUBROUTINE lsei(c,d,e,f,g,h,lc,mc,LE,me,lg,mg,n,x,xnrm,w,jw,mode)
 
       INTEGER          jw(*),i,ie,IF,ig,iw,j,k,krank,l,lc,LE,lg,
      .                 mc,mc1,me,mg,mode,n
+
       DOUBLE PRECISION c(lc,n),e(LE,n),g(lg,n),d(lc),f(LE),h(lg),x(n),
      .                 w(*),t,ddot_sl,xnrm,rnorm(1),dnrm2_,epmach,ZERO
       DATA             epmach/2.22d-16/,ZERO/0.0d+00/
@@ -971,6 +974,7 @@ SUBROUTINE lsi(e,f,g,h,LE,me,lg,mg,n,x,xnorm,w,jw,mode)
 C     20.03.1987, DIETER KRAFT: REVISED TO FORTRAN 77
 
       INTEGER          i,j,LE,lg,me,mg,mode,n,jw(lg)
+
       DOUBLE PRECISION e(LE,n),f(LE),g(lg,n),h(lg),x(n),w(*),
      .                 ddot_sl,xnorm,dnrm2_,epmach,t,one
       DATA             epmach/2.22d-16/,one/1.0d+00/
@@ -1046,6 +1050,7 @@ SUBROUTINE ldp(g,mg,m,n,h,x,xnorm,w,INDEX,mode)
 C               3: ITERATION COUNT EXCEEDED BY NNLS
 C               4: INEQUALITY CONSTRAINTS INCOMPATIBLE
 
+
       DOUBLE PRECISION g,h,x,xnorm,w,u,v,
      .                 ZERO,one,fac,rnorm,dnrm2_,ddot_sl,diff
       INTEGER          INDEX,i,IF,iw,iwdual,iy,iz,j,m,mg,mode,n,n1
@@ -1337,6 +1342,7 @@ SUBROUTINE hfti(a,mda,m,n,b,mdb,nb,tau,krank,rnorm,h,g,ip)
 
       INTEGER          i,j,jb,k,kp1,krank,l,ldiag,lmax,m,
      .                 mda,mdb,n,nb,ip(n)
+
       DOUBLE PRECISION a(mda,n),b(mdb,nb),h(n),g(n),rnorm(nb),factor,
      .                 tau,ZERO,hmax,diff,tmp,ddot_sl,dnrm2_,u,v
       diff(u,v)=       u-v
diff --git a/scipy/optimize/tests/test_optimize.py b/scipy/optimize/tests/test_optimize.py
index 8a591156a80c..0ee21185ac23 100644
--- a/scipy/optimize/tests/test_optimize.py
+++ b/scipy/optimize/tests/test_optimize.py
@@ -19,6 +19,7 @@
 import pytest
 from pytest import raises as assert_raises
 
+import scipy
 from scipy import optimize
 from scipy.optimize._minimize import Bounds, NonlinearConstraint
 from scipy.optimize._minimize import (MINIMIZE_METHODS,
@@ -2915,6 +2916,9 @@ def jacobian2(x):
 
 # This test is about handling fixed variables, not the accuracy of the solvers
 @pytest.mark.xfail_on_32bit("Failures due to floating point issues, not logic")
+@pytest.mark.xfail(scipy.show_config(mode='dicts')['Compilers']['fortran']['name'] ==
+                   "intel-llvm",
+                   reason="Failures due to floating point issues, not logic")
 @pytest.mark.parametrize('method', eb_data["methods"])
 @pytest.mark.parametrize('kwds', eb_data["kwds"])
 @pytest.mark.parametrize('bound_type', eb_data["bound_types"])
diff --git a/scipy/optimize/tests/test_slsqp.py b/scipy/optimize/tests/test_slsqp.py
index cab46291b91e..7d7648eeb8bd 100644
--- a/scipy/optimize/tests/test_slsqp.py
+++ b/scipy/optimize/tests/test_slsqp.py
@@ -6,6 +6,7 @@
 from pytest import raises as assert_raises
 import pytest
 import numpy as np
+import scipy
 
 from scipy.optimize import fmin_slsqp, minimize, Bounds, NonlinearConstraint
 
@@ -489,6 +490,9 @@ def f(x):
         assert_(sol.success)
         assert_allclose(sol.x, 0, atol=1e-10)
 
+    @pytest.mark.xfail(scipy.show_config(mode='dicts')['Compilers']['fortran']['name']
+                       == "intel-llvm",
+                       reason="Runtime warning due to floating point issues, not logic")
     def test_inconsistent_inequalities(self):
         # gh-7618
 
diff --git a/scipy/sparse/linalg/_propack/PROPACK b/scipy/sparse/linalg/_propack/PROPACK
index a9ed9736ad52..300f803c5ac3 160000
--- a/scipy/sparse/linalg/_propack/PROPACK
+++ b/scipy/sparse/linalg/_propack/PROPACK
@@ -1 +1 @@
-Subproject commit a9ed9736ad52ff76ae1777922b700b13ca2bf0ae
+Subproject commit 300f803c5ac3372ceb65ba446c71c90f128814b1
diff --git a/scipy/sparse/linalg/_propack/meson.build b/scipy/sparse/linalg/_propack/meson.build
index 671472495859..d5433fe6a30a 100644
--- a/scipy/sparse/linalg/_propack/meson.build
+++ b/scipy/sparse/linalg/_propack/meson.build
@@ -32,6 +32,7 @@ d_src = [
 ]
 
 c_src = [
+  'PROPACK/complex8/ccdotc.F',
   'PROPACK/complex8/cblasext.F',
   'PROPACK/complex8/cgemm_ovwr.F',
   'PROPACK/complex8/cgetu0.F',
@@ -52,6 +53,7 @@ z_src = [
   'PROPACK/complex16/dbsvd.F',
   'PROPACK/complex16/dgemm_ovwr.F',
   'PROPACK/complex16/printstat.F',
+  '../_eigen/arpack/ARPACK/SRC/zzdotc.f',
   'PROPACK/complex16/zblasext.F',
   'PROPACK/complex16/zgemm_ovwr.F',
   'PROPACK/complex16/zgetu0.F',
@@ -89,7 +91,7 @@ foreach ele: elements
     ele[1],
     fortran_args: [
       fargs_propack,
-      fortran_ignore_warnings, 
+      fortran_ignore_warnings,
       _fflag_Wno_intrinsic_shadow,
       _fflag_Wno_uninitialized,
       _fflag_fpp,
diff --git a/tools/install_intel_oneAPI_linux.sh b/tools/install_intel_oneAPI_linux.sh
new file mode 100644
index 000000000000..c39b0a5131de
--- /dev/null
+++ b/tools/install_intel_oneAPI_linux.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+# SPDX-FileCopyrightText: 2020 Intel Corporation
+#
+# SPDX-License-Identifier: MIT
+
+URL=$1
+COMPONENTS=$2
+
+curl --output webimage.sh --url "$URL" --retry 5 --retry-delay 5
+chmod +x webimage.sh
+./webimage.sh -x -f webimage_extracted --log extract.log
+rm -rf webimage.sh
+WEBIMAGE_NAME=$(ls -1 webimage_extracted/)
+if [ -z "$COMPONENTS" ]; then
+  sudo webimage_extracted/"$WEBIMAGE_NAME"/bootstrapper -s --action install --eula=accept --log-dir=.
+  installer_exit_code=$?
+else
+  sudo webimage_extracted/"$WEBIMAGE_NAME"/bootstrapper -s --action install --components="$COMPONENTS" --eula=accept --log-dir=.
+  installer_exit_code=$?
+fi
+rm -rf webimage_extracted
+exit $installer_exit_code


Code review comments:
diff_hunk: @@ -106,7 +106,7 @@ py3.extension_module('_slsqp',
   [f2py_gen.process('slsqp/slsqp.pyf'), 'slsqp/slsqp_optmz.f'],
   fortran_args: fortran_ignore_warnings,
   link_args: version_link_args,
-  dependencies: [fortranobject_dep],
+  dependencies: [lapack_dep, blas_dep, fortranobject_dep],
path: scipy/optimize/meson.build
text: @user1:
Is this actually needed, or is that a left-over from debugging?

@author:
Ah! yes. Debugging left over.

@author:
```suggestion
  dependencies: [fortranobject_dep],
```