Repository: pyodide/pyodide

Issue:
New signature mismatch in `scipy.sparse.linalg.svds`
## 🐛 Bug

I am seeing new fatal errors in the scipy test suite in https://github.com/lesteve/scipy-tests-pyodide since a few days ago. They are likely related to the numpy version upgrade in #3593 although I have no idea why.

### To Reproduce

You can reproduce it with the [Pyodide alpha console](https://pyodide.org/en/latest/console.html) and the following snippet:

```py
import numpy as np

from scipy.sparse.linalg import svds

rng = np.random.default_rng(0)
A = rng.random((10, 10))

res = svds(A, k=3, which='LM', random_state=0)
```

```
Stack (most recent call first):
  File "/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 578 in extract
  File "/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", line 1691 in eigsh
  File "/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/_svds.py", line 339 in svds
  File "<console>", line 1 in <module>
  File "/lib/python311.zip/_pyodide/_base.py", line 363 in run_async
  File "/lib/python311.zip/pyodide/console.py", line 378 in runcode
  File "/lib/python311.zip/pyodide/console.py", line 492 in runcode
  File "/lib/python311.zip/asyncio/events.py", line 80 in _run
  File "/lib/python311.zip/pyodide/webloop.py", line 319 in run_handle

Uncaught RuntimeError: null function or function signature mismatch
    at f2py_rout__arpack_dseupd (000ffae6:0x5e92)
    at fortran_call (0003ecf2:0x65ac)
    at _PyCFunctionWithKeywords_TrampolineCall (pyodide.asm.js:1375:97)
    at _PyObject_MakeTpCall (pyodide.asm.wasm:0x199bba)
    at PyObject_Vectorcall (pyodide.asm.wasm:0x19a1a4)
    at _PyEval_EvalFrameDefault (pyodide.asm.wasm:0x25f947)
    at PyEval_EvalCode (pyodide.asm.wasm:0x255ed8)
    at builtin_eval (pyodide.asm.wasm:0x2531d6)
    at _PyEval_EvalFrameDefault (pyodide.asm.wasm:0x25e78b)
    at gen_send_ex2 (pyodide.asm.wasm:0x1adc62)
```

Debugging a bit more, the function is called with 23 arguments:
```
call_indirect (param i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32 i32) (result i32)
```

but has 25 arguments:
```
(func $dseupd_ (;219;) (export "dseupd_") (param $var0 i32) (param $var1 i32) (param $var2 i32) (param $var3 i32) (param $var4 i32) (param $var5 i32) (param $var6 i32) (param $var7 i32) (param $var8 i32) (param $var9 i32) (param $var10 i32) (param $var11 i32) (param $var12 i32) (param $var13 i32) (param $var14 i32) (param $var15 i32) (param $var16 i32) (param $var17 i32) (param $var18 i32) (param $var19 i32) (param $var20 i32) (param $var21 i32) (param $var22 i32) (param $var23 i32) (param $var24 i32) (result i32)
```

### Expected behavior

No fatal error

### Environment

- Pyodide Version<!-- (e.g. 1.8.1) -->: alpha




Discussion:
Thanks for the report @lesteve! Too bad there isn't an "I hate this" react...
Here is the summary of my investigation:
- very likely due to https://github.com/numpy/numpy/pull/19388 there has been changes in f2py in numpy 1.24
- Here are is an excerpt of the diff on the arpackmodule:
```
diff -u /tmp/{working,problematic}/src.emscripten_3_1_32_wasm32-3.11/scipy/sparse/linalg/_eigen/arpack
/_arpackmodule.c
```
```diff
@@ -1110,10 +1165,10 @@
 d,z,info = sseupd(rvec,howmny,select,sigma,bmat,which,nev,tol,resid,v,iparam,ipntr,workd,workl,info,[ldz,n,ncv,ldv,lworkl])\n\nWrapper for ``sseupd``.\
 \n\nParameters\n----------\n"
 "rvec : input int\n"
-"howmny : input string(len=1)\n"
+"howmny : input bytes\n"
 "select : input rank-1 array('i') with bounds (ncv)\n"
 "sigma : input float\n"
-"bmat : input string(len=1)\n"
+"bmat : input bytes\n"
 "which : input string(len=2)\n"
 "nev : input int\n"
 "tol : input float\n"
@@ -1134,42 +1189,40 @@
```

```diff
@@ -1405,7 +1472,7 @@
 #endif
 /*callfortranroutine*/
             Py_BEGIN_ALLOW_THREADS
-                (*f2py_func)(&rvec,howmny,select,d,z,&ldz,&sigma,bmat,&n,which,&nev,&tol,resid,&ncv,v,&ldv,iparam,ipntr,workd,workl,&lworkl,&info,slen(howmny),slen(bmat),slen(which));
+                (*f2py_func)(&rvec,&howmny,select,d,z,&ldz,&sigma,&bmat,&n,which,&nev,&tol,resid,&ncv,v,&ldv,iparam,ipntr,workd,workl,&lworkl,&info,slen(which));
             Py_END_ALLOW_THREADS
 if (PyErr_Occurred())
   f2py_success = 0;
#+end_src
```

So `howmny` and `bmat` have changed types from `string(len=1)` to `bytes`. The function is called is taking 23 arguments with the f2py changes instead of 25. The previous (numpy 1.23.5) behaviour has additional arguments with `slen(howmny)` and `slen(bmat)`.

The thing is that the f2c'ed dseupd.f still has the old signature with 25 arguments
```
packages/scipy/build/scipy-1.9.3/scipy/sparse/linalg/_eigen/arpack/ARPACK/SRC/dseupd.c
```

```c
/* Subroutine */ int dseupd_(logical *rvec, char *howmny, logical *select,
        doublereal *d__, doublereal *z__, integer *ldz, doublereal *sigma,
        char *bmat, integer *n, char *which, integer *nev, doublereal *tol,
        doublereal *resid, integer *ncv, doublereal *v, integer *ldv, integer
        ,*iparam, integer *ipntr, doublereal *workd, doublereal *workl,
        integer *lworkl, integer *info, ftnlen howmny_len, ftnlen bmat_len,
        ftnlen which_len)
```

About the possible way forward, there may be a few different possibilities:
- patch `scipy/sparse/linalg/_eigen/arpack/arpack.pyf.src` and use `character*1` rather than `character` everywhere. This seems to work locally
- in `pyodide-build/pyodide_build/_f2c_fixes.py` change the signature somehow, it does not seem like the `_len` variables are used in the function so maybe that could work
- patch somehow the call site adding 1 for the length of `howmny` and `bmat`, not sure how easy this is
- something else?

I am not sure whether this is worth reporting to numpy since `f2py` and `f2c` don't generate the same signature, which may be an issue for us in other places ... I am not sure also how to put together a smaller reproducer to explain our issue with not too much code  ...
I think this problem is created in `_f2c_fixes.py`, quite possibly in the function `char1_args_to_int`.

The issue is that if you apply `character*1`, it produces an `ftnlen` argument at the end. So if you had:
```
f(a, b, c)
character *1 a, character*2 b, real c
```
then f2c makes a C function like:
```C
f(char* a, char* b, real* c, ftnlen a_len, ftnlen b_len)
```
but clapack and probably openblas have hand modified these functions to remove the ftnlen arguments, since they are kind of wrong for fixed length strings -- we don't need to pass a length, it's always 1 (respectively 2). But then we're f2cing other code that tries to call f, and it generates a signature like `f(char* a, char* b, real* c, ftnlen a_len, ftnlen b_len)` which is 5 i32 args and the actual function has 3 i32 args so there's a problem. So in `_f2c_fixes.py` there's automated tooling to remove these `ftnlen` args. But if the function being called was not defined in clapack, then it will have these `ftnlen` args so it needs to be left alone. 
Thanks for the details, now I understand a bit more the reason for the char to int replacement!

In this case though the Fortran code is in scipy (in `scipy/sparse/linalg/_eigen/arpack/ARPACK/SRC/dseupd.f` so not related to BLAS or LAPACK) and is `f2c`ed by Pyodide right? So the change in `f2py` in numpy 1.24 breaks the consistency between the `f2py`ed code and the `f2c`ed code (maybe for good reasons ...). 

I tried to follow your advice and replace `character` by `int` in `dseupd.f` code in `fix_f2c_input`, but the thing is that you have some code like `if (bmat .ne. 'I' .and. bmat .ne. 'G')` which fails to compile, so I guess that you will need to replace this by their `ord` equivalent as well. I tried to do this, this compiles but I get a runtime error that I need to investigate (no signature mismatch so this goes a step further). The Python traceback is:
```
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/_svds.py", line 339
, in svds
    _, eigvec = eigsh(XH_X, k=k, tol=tol ** 2, maxiter=maxiter,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", 
line 1691, in eigsh
    return params.extract(return_eigenvectors)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python3.11/site-packages/scipy/sparse/linalg/_eigen/arpack/arpack.py", 
line 585, in extract
    raise ArpackError(ierr, infodict=self.extract_infodict)
scipy.sparse.linalg._eigen.arpack.arpack.ArpackError: ARPACK error -15: HOWMNY must 
be one of 'A' or 'S' if RVEC = .true.
```

Somehow it feels slightly easier to patch the `pyf` to use `character*1` instead of `character` and avoid the `f2py` changes in numpy 1.24 seems but maybe I am missing something :wink: 
 
> it feels slightly easier to

Not sure. I encourage you to try anything that you think might work. My patches are all "this is the first thing I tried that worked" I am not sure that they are the best possible approach for the cases they cover nor for any new cases.
> it feels slightly easier to

Not sure. I encourage you to try anything that you think might work. My patches are all "this is the first thing I tried that worked" I am not sure that they are the best possible approach for the cases they cover nor for any new cases.

Resolving patch:
diff --git a/pyodide-build/pyodide_build/_f2c_fixes.py b/pyodide-build/pyodide_build/_f2c_fixes.py
--- a/pyodide-build/pyodide_build/_f2c_fixes.py
+++ b/pyodide-build/pyodide_build/_f2c_fixes.py
@@ -268,6 +268,21 @@ def fix_line(line: str) -> str:
                 """
             )
 
+    # In numpy 1.24 f2py changed its treatment of character argument. In
+    # particular it does not generate a ftnlen parameter for each
+    # character parameter but f2c still generates it. The following code
+    # removes unneeded ftnlen parameters from the f2ced signature. The
+    # problematic subroutines and parameters are the ones with a type character
+    # in scipy/sparse/linalg/_eigen/arpack/arpack.pyf.src
+    if "eupd.c" in str(f2c_output):
+        # put signature on a single line to make replacement more
+        # straightforward
+        regrouped_lines = regroup_lines(lines)
+        lines = [
+            re.sub(r",?\s*ftnlen\s*(howmny_len|bmat_len)", "", line)
+            for line in regrouped_lines
+        ]
+
     with open(f2c_output, "w") as f:
         f.writelines(lines)
 


Test patch:
diff --git a/packages/scipy/test_scipy.py b/packages/scipy/test_scipy.py
--- a/packages/scipy/test_scipy.py
+++ b/packages/scipy/test_scipy.py
@@ -63,6 +63,7 @@ def runtest(module, filter):
     runtest("odr", "explicit")
     runtest("signal.tests.test_ltisys", "TestImpulse2")
     runtest("stats.tests.test_multivariate", "haar")
+    runtest("sparse.linalg._eigen", "test_svds_parameter_k_which")
 
 
 @pytest.mark.driver_timeout(40)