Repository: mesonbuild/meson

Issue:
Broken `python.dependency()` with PyPy, maybe
**Describe the bug**

Meson is erroring out when trying to detect `python.dependency()` when the interpreter is PyPy:

```
Compiler for C supports arguments -fvisibility=hidden: YES 
Configuring pycairo-1.20.0.egg-info using configuration
Dependency python found: NO

cairo/meson.build:35:0: ERROR: Python dependency not found
```

That line in `cairo/meson.build` is calling `python.dependency()`.

**To Reproduce**

Encountered trying to build [pycairo 1.20.0](https://github.com/pygobject/pycairo) in [conda-forge](https://github.com/conda-forge/pycairo-feedstock); specifically the pull request conda-forge/pycairo-feedstock#23. Can be reproduced locally by checking out the PR branch on a Linux machine and running `./build-locally.py linux_64_python3.6.____73_pypy`.

**Expected behavior**

No surprising Python-related failures.

**system parameters**
* Is this a [cross build](https://mesonbuild.com/Cross-compilation.html) or just a plain native build (for the same computer)? Native
* what operating system? CentOS 6.10
* what Python version are you using? 3.6 with PyPy 7.3.3
* what `meson --version`: 0.57.1
* what `ninja --version` if it's a Ninja build: 1.10.2

**More Information**

The root cause of this issue seems to be the same as initially reported in #4841. Meson expects the PyPy library to be found in `${base}/bin`, which is not the case for conda-forge.

I see that the logic in this key line:

https://github.com/mesonbuild/meson/blob/1be13710adba2126f5731fccdf698a9d405a9d67/mesonbuild/modules/python.py#L149

was inverted in a later pull request #5873 by @jpakkane. The original logic seems correct to me, and it's not clear to me what that change had to do with the rest of the pull request — maybe something unintended snuck in?

**edit**: I did a quick check and `link_libpython` is false for conda-forge PyPy, so I think that the original logic would avoid the problem in this particular case.


Discussion:
OK, furthermore, I believe that `python.dependency()` will never work on OSX with PyPy. In the Python dependency constructor, the dep can be found in one of two ways: either using pkg-config, or the SYSCONFIG method. The pkg-config method is disabled in PyPy, and the SYSCONFIG method is not activated on OSX. So if you're using PyPy on OSX, the dep can never be found.
@pkgw what would be the best way to detect alternatively? I don't have a Mac, but I don't mind submitting a PR on your behalf if I find the time.

I see you reference a change in boolean logic. Maybe some light will be shed!

Note: I am not at all familiar with pypy
I *think* that if the logic is switched back to `self.is_found = largs is not None or not self.link_libpython`, everything would work, because `link_libpython` is false for PyPy. And that still seems to me to be the correct logic: treat the dependency as found/OK if either you found the libpython library to link against, *or* you don't need to link against it.
PyPy dev here.
Now that projects are starting to use meson in earnest, it would be nice to fix this. What would a solution look like? However meson decides to use python3.8.so or python3.8.dll, could we get it to use the PyPy equivalents?
> treat the dependency as found/OK if either you found the libpython library to link against, _or_ you don't need to link against it

Not an python expert, but that makes sense, yes. @jpakkane was that change intentional in https://github.com/mesonbuild/meson/pull/5873? 
```diff
-        self.is_found = largs is not None or not self.link_libpython
+        self.is_found = largs is not None or self.link_libpython
```
> However meson decides to use python3.8.so or python3.8.dll, could we get it to use the PyPy equivalents?

This is platform dependent, but without pkg-config to serve as an authority on the subject (i.e. for the sysconfig resolver) we essentially poke at distutils to see if a distutils built extension would contain link arguments for libpython. If so, we need to link against it too.

Aside: this is sad because distutils is deprecated but as far as I can tell there is no replacement for this data and you basically need to memorize the per platform conventions.

This issue claims that that is being correctly detected, but a logic switch is inverted causing it to be effectively ignored?

I'll take a look at this when I get home.
I thought `sysconfig` was the canonical way to find this information. For instance, `sysconfig.get_config_var('LDLIBRARY')` on linux and macOS should return the shared object. This is much simpler for PyPy to deal with since it is a one-liner. Are there guides around how to add pypy to pkg-config ?
@mattip here is how the current `/usr/lib64/python3.pc` looks on my system:

```
$ cat /usr/lib64/pkgconfig/python3.pc
# See: man pkg-config
prefix=/usr
exec_prefix=/usr
libdir=/usr/lib64
includedir=/usr/include

Name: Python
Description: Build a C extension for Python
Requires:
Version: 3.10
Libs.private: -lcrypt -ldl  -lm
Libs:
Cflags: -I${includedir}/python3.10
```
That sysconfig entry says what the shared library is. It doesn't say whether extension modules should link to it.

Maybe only programs which embed the interpreter should link to libpython. That's how it works on Linux these days, for example. It's the reason pkg-config now has two profiles for python, one called `python.pc` and one called `python-embed.pc`.
Thanks for the explanations. The pkg-config file seems like something a distro packager would supply, so not really something for a tarball-download install of pypy. I wonder if homebrew/debian/fedora packagers are supplying it for pypy.

> Maybe only programs which embed the interpreter should link to libpython

PyPy currently does not support embedding very well. There is a way to convince CFFI to support it, but I am not aware of projects doing this. So maybe meson could just say "embedding PyPy not supported" on linux and macOS for now, until a real need arises. For windows, see #10080 since it  needs to resolve all symbols when linking.
The pkg-config files come from the CPython project in collaboration with distro vendors, and reflect the consensus on what CPython's ecosystem should look like. That is also what guides distutils behavior.

If PyPy doesn't provide pkg-config files then of course we will use sysconfig instead. :) At the end of the day it's all just a question of which use cases exist and what compile and link arguments to expect in each of these use cases.

If PyPy doesn't consider embedding usable, it need not be supported. Meson uses the `embed: true` kwarg in meson.build when using the python module (this correctly handles the difference between python < 3.8 versus versions that began the split between embedding and building extensions).

So for `embed: true` should we just error out in PyPy with "PyPy doesn't support embedding"?
> So for `embed: true` should we just error out in PyPy with "PyPy doesn't support embedding"?

I am not sure. Do you have examples of projects that use embedding? I was more thinking that it would be easier to avoid even looking for a link library if `embed: false`. But maybe that doesn't make sense and using `sysconfig` is a better strategy.
The common one is programs like weechat (an IRC client) that support user scripts for a variety of languages while not actually being written in those languages.

(Weechat can do user scripts written in perl, python, ruby, lua, javascript, php (???) and I think maybe a couple I'm forgetting.)
Since weechat calls `Py_Initialize()`, which is not currently supported on PyPy, even if meson could set everything up properly the link would fail. But if PyPy did get support for `Py_Initialize()` we would be back at the starting point.
One thing about pkg-config, is that since it doesn't require running python (whatever implementation), which is super helpful when cross compiling.

Resolving patch:
diff --git a/mesonbuild/modules/python.py b/mesonbuild/modules/python.py
--- a/mesonbuild/modules/python.py
+++ b/mesonbuild/modules/python.py
@@ -158,7 +158,7 @@ def _find_libpy(self, python_holder: 'PythonInstallation', environment: 'Environ
         if largs is not None:
             self.link_args = largs
 
-        self.is_found = largs is not None or self.link_libpython
+        self.is_found = largs is not None or not self.link_libpython
 
         inc_paths = mesonlib.OrderedSet([
             self.variables.get('INCLUDEPY'),


Test patch: