Repository: scipy/scipy

Issue:
BUG: Build failure due to problems with shebang line in cythoner.py
I ran into a problem running `dev.py` that appears to be caused by the shebang line `#!python3`  in the file `scipy/_build_utils/cythoner.py`. If I change it to `#!/usr/bin/env python` then the build works fine.

Most files in scipy with a shebang line use `#!/usr/bin/env python`. Only files in the `_build_utils` use `#!python3`.

Error message when running `python dev.py build`:

```shell
Meson build setup OK
💻  ninja -C /mnt/c/Users/Jozsef/OSS/scipy-test/build
ninja: Entering directory `/mnt/c/Users/Jozsef/OSS/scipy-test/build'
[3/1562] Generating 'scipy/_lib/_ccallback_c.cpython-310-x86_64-linux-gnu.so.p/_ccallback_c.c'.
FAILED: scipy/_lib/_ccallback_c.cpython-310-x86_64-linux-gnu.so.p/_ccallback_c.c
/mnt/c/Users/Jozsef/OSS/scipy-test/scipy/_build_utils/cythoner.py ../scipy/_lib/_ccallback_c.pyx scipy/_lib/_ccallback_c.cpython-310-x86_64-linux-gnu.so.p/_ccallback_c.c
/bin/sh: 1: /mnt/c/Users/Jozsef/OSS/scipy-test/scipy/_build_utils/cythoner.py: not found
[12/1562] Compiling C++ object scipy/_lib/_uarray/_uarray.cpython-310-x86_64-linux-gnu.so.p/_uarray_dispatch.cxx.o
ninja: build stopped: subcommand failed.
Build failed!
```

If I try running `cythoner.py` directly:

```shell
-bash: /mnt/c/Users/Jozsef/OSS/scipy-conda/scipy/_build_utils/cythoner.py: python3: bad interpreter: No such file or directory
```

I'm using conda with WSL (Ubuntu).


Discussion:
Thanks for the report @JozsefKutas. We shouldn't be relying on shebang lines at all, let's fix this to make sure it uses the correct Python interpreter that was already detected by Meson.
This turns out to be slightly annoying. In the below, the `find_program` version works but the `files` one does not:
```meson
cython_tree = declare_dependency(sources: _cython_tree)

cython_cli = find_program('_build_utils/cythoner.py')
invoke_cython = files('_build_utils/cythoner.py')

cython_gen = generator(py3,
  arguments : [invoke_cython, '@INPUT@', '@OUTPUT@'],
  output : '@BASENAME@.c',
  depends : _cython_tree)

cython_gen_cpp = generator(cython_cli,
  arguments : ['@INPUT@', '@OUTPUT@', '--cplus'],
  output : '@BASENAME@.cpp',
  depends : [_cython_tree])
```

The error is:
```
scipy/meson.build:216:0: ERROR: generator keyword argument 'arguments' was of type array[File | str] but should have been array[str] that cannot be empty
```

`generator` takes a single executable, and it seems hard to give it `py3 <source_root>/_build_utils/cythoner.py` where `py3` is the detected Python install from:
```meson
py_mod = import('python')
py3 = py_mod.find_installation()
```

`find_program` is the wrong thing to use, but it's not clear to me if there's a tweak to make this work - in lieu of changing `generator` to `custom_target` everywhere (which would be a lot of churn).


I had a look at it some more. We're not using it wrong, the chapter on generators in The Meson Manual uses as its one example what we are doing:
```meson
idl_compiler = find_program('dsc_compiler.py')

g = generator(idl_compiler,
  output: ...
```
This seems like a design issue in Meson, relying on a shebang is a problem. Here we have a system that has `python` but not `python3`, and in gh-16937 we have an example of a system in the opposite situation (has `python3` but not `python`.

@JozsefKutas can you please confirm that you really don't have a `python3` executable at all? Or does changing the shebang to `#!/usr/bin/env python3` work too?
The relevant Meson issue, with rules for shebang to use per platform, is https://github.com/mesonbuild/meson/issues/3911. 

> @JozsefKutas can you please confirm that you really don't have a `python3` executable at all? Or does changing the shebang to `#!/usr/bin/env python3` work too?

@JozsefKutas a ping on this, can you please check?
Hi, thanks for looking into this and I'm sorry for not responding earlier.

I've just tried with `#!/usr/bin/env python3` and the build runs without any issues.

I can run `python3` in console without problems (both in and out of the conda scipy environment, and with conda completely deactivated), but for some reason it doesn't work with the shebang line (again, regardless of whether conda is activated or not). I've only encountered the problem in WSL.
The problem is that `#!foo` is an invalid shebang altogether, shebang lines are supposed to contain the full path to an executable (that executable can be the env program) plus optionally a single argument (e.g. a program that the env executable will search for on $PATH).

Attempting to run this program directly in any way results in failure -- no matter what system you use.

...

There's actually a more interesting issue. Meson's find_program implementation has a method called `_shebang_to_cmd()`, which is used if the command doesn't have the executable bit set (or on Windows, if the extension isn't in windows_exts). This checks the shebang using very rough heuristics to find a script interpreter, but not actually following the same rules that an OS kernel does. Namely, it special cases any shebang that *ends* with `python3`, even if it begins with nonsense such as the empty string, and runs via its own sys.executable which it knows satisfies "a python3".

This is why, usually, the cythoner.py script works fine. It is not marked executable in git, and thus isn't either executable in a git clone or an sdist. So Meson special cases it in find_program, and logs this:

```
Program _build_utils/cythoner.py found: YES (/usr/bin/python3 /tmp/scipy/scipy/_build_utils/cythoner.py)
```

Apparently, this is broken on WSL as it is somehow executable nonetheless, likely some quirk of WSL permissions. So Meson simply tries to run it directly.
> Thanks for the report @JozsefKutas. We shouldn't be relying on shebang lines at all, let's fix this to make sure it uses the correct Python interpreter that was already detected by Meson.

I am not entirely sure what this is trying to say, but I think it's probably wrong. This script uses pure barebones stdlib python, and does not care what python interpreter it is run with, since all it does is execute the shell command `cython`.

Resolving patch:
diff --git a/scipy/_build_utils/cythoner.py b/scipy/_build_utils/cythoner.py
--- a/scipy/_build_utils/cythoner.py
+++ b/scipy/_build_utils/cythoner.py
@@ -1,4 +1,4 @@
-#!python3
+#!/usr/bin/env python3
 """ Scipy variant of Cython command
 
 Cython, as applied to single pyx file.


Test patch: