Repository: mesonbuild/meson

Issue:
Python 3.8 and `python3-config --libs`
Quoting https://bugzilla.redhat.com/show_bug.cgi?id=1718290:

> The build no longer links to libpython, resulting in undefined references to Python API.
Code that embeds Python (rather than building an extension module) needs to pass `--embed` to any `python3-config --libs` invocation to build with Python 3.8.
More information: 
https://docs.python.org/3.8/whatsnew/3.8.html#debug-build-uses-the-same-abi-as-release-build

In libratbag, we are using swig to generate a Python library that we link with a dependencies of `dependency('python3')`. I wasn't able to find out how the pkg-config was invoced, so I can not really submit a patch.


Discussion:
The python module handles the first part this since #4187

edit: oh, it's the other way around in that pkg-config no longer makes it link against libpython.. never mind then :/ I guess we need to an an option to the python module somehow for this new case.
There is a new "python-embed.pc" in 3.8, have you tried that?
Yep, changing to `dependency('python3-embed')` fixes the issue, thanks!
Thanks everybody. I guess we can close this issue.
I'm not sure this is the correct fix; the fix should work on all python versions. If it is, we should document it somewhere?
The bigger issue is how should Python dependencies looked up via the Python module work? That one probably needs a new kwarg because you can not specify which dependency name to use.
I've also got one bug https://bugzilla.redhat.com/show_bug.cgi?id=1718283.

I think proper way would be to use some kwarg which would choose between -embed and normal python.
Seems reasonable.
Checking for `python3-embed.pc` is not sufficient as it might not even exist (it does not on Debian until the **default** Python version becomes 3.8). The correct way would be to call `python3.8-config --ldflags --embed`.

Also related bug: https://gitlab.freedesktop.org/gstreamer/gst-python/issues/28
> The bigger issue is how should Python dependencies looked up via the Python module work? That one probably needs a new kwarg because you can not specify which dependency name to use.

yes, precisely, I reckon an `embed` argument to the dependency method would be the desirable API
> Checking for `python3-embed.pc` is not sufficient as it might not even exist (it does not on Debian until the **default** Python version becomes 3.8).

If `python3-embed.pc` does not exist, and `python3.pc` is a symlink to `python-3.8.pc`, then your distribution is flawed and MUST be fixed.

If `python3-embed.pc` does not exist, and `python3.pc` is a symlink to `python-3.7.pc`, then your meson.build project will build using the default version of python on your distribution, which is python 3.7, so this issue is not topical to your use case.

> The correct way would be to call `python3.8-config --ldflags --embed`.

Incorrect, projects which expect to build against some version of python 3.x must not hardcode `python3.8-config` but rather use python3-config, which shall be a symlink to the default version of python3, which on your distro is python 3.7.

If it were appropriate to look up `python3.8-config`, then it would likewise be appropriate to declare a `dependency('python3.8-embed')`, since `python3.8-embed.pc` MUST exist (even if `python3-embed.pc` does not), for any installation of python 3.8 that includes development headers, regardless of whether it is the default, unversioned 3.x namespace.
@eli-schwartz You're missing the point.

There could be multiple Python versions installed on a system, which is the case on Debian for example (right now: 2.7, 3.7 and 3.8). One Python 3 version is going to be the default (and have the corresponding `python3.pc` etc), but you can select different Python versions during the build and that also always worked with both autotools and meson until Python 3.8 appeared.

See https://gitlab.freedesktop.org/gstreamer/gst-python/ for an example meson build system that allows to select the Python version. Using `python3.pc` unconditionally is not going to work there, you'd have to use either `python-3.X.pc` (and the `-embed` version if X > 7) or the corresponding `python3.X-config` (with `--embed` if needed for > 3.7).
If you're not using pkg-config in the first place, then why are you suggesting that "checking python3-embed.pc is insufficient"?

Moreover, why are you suggesting that the correct way is to call `python3.8-config`, which is a false equivalence, since if you're going to hardcode or detect the version in the python*-config tool, you can hardcode or detect the version in the python*-embed.pc file too.

Using meson's python module via find_installation().dependency() already checks the major.minor version of the detected python binary, then looks for `'python-{}.pc'.format(pkg_version)` explicitly, never python3.pc, so that's fine.

The recommendation in previous comments "have you tried python3-embed.pc" was strictly a recommendation for users who are currently using `dependency('python3')`, rather than the python module, so I assumed that was happening. In such cases, it makes perfect sense to first try `dependency('python3-embed')`.

The same logic would always apply to autotools, which doesn't automatically resolve `PKG_CHECK_MODULES(PYTHON, [python3], , )` to the major.minor version, and you get to do all the magic yourself.
A bit hacky, but you could do arbitrary version selection yourself (no need for the python module) like this:

```
python = find_program('python3')
pythonver = run_command(python, '-c', 'import sys; print("%s.%s" % sys.version_info[:2], end="")').stdout()

# or via language_version()
python = import('python')
pythonver = python.find_installation('python3').language_version()

python_dep = dependency('python-@0@-embed'.format(pythonver), version: '>=3', required: false)
if not python_dep.found()
    python_dep = dependency('python-@0@'.format(pythonver))
endif
```

Alternatively (this prints up to 3 missing unrequired dependencies first):

```
pythonver = get_option('pythonver')
python = dependency('', required: false)
foreach base : ['python@0@-embed', 'python@0@', 'python-@0@-embed', 'python-@0@']
    if not python.found()
        python = dependency(base.format(pythonver), version: '>=3', required: false)
    endif
endforeach

if not python.found()
    error('no suitable version of python@0@ found'.format(pythonver))
endif
```
> If you're not using pkg-config in the first place, then why are you suggesting that "checking python3-embed.pc is insufficient"?

Checking for a generic `python3*.pc` is insufficient. My comment was about having this as the "official" workaround for this problem. It won't allow selecting a specific Python version.

`python-3.8-embed.pc` if Python 3.8 is selected would work too for example, or using `python-3.8-config`.

Your workaround to parse the Python version and then look for the corresponding `.pc` file would work though, thanks for that!

Resolving patch:
diff --git a/mesonbuild/interpreter.py b/mesonbuild/interpreter.py
--- a/mesonbuild/interpreter.py
+++ b/mesonbuild/interpreter.py
@@ -2013,6 +2013,7 @@ def get_cross_property_method(self, args, kwargs):
                                       'build_always_stale',
                                       'console'},
                     'dependency': {'default_options',
+                                   'embed',
                                    'fallback',
                                    'language',
                                    'main',
diff --git a/mesonbuild/modules/python.py b/mesonbuild/modules/python.py
--- a/mesonbuild/modules/python.py
+++ b/mesonbuild/modules/python.py
@@ -27,7 +27,7 @@
     InvalidArguments,
     FeatureNew, FeatureNewKwargs, disablerIfNotFound
 )
-from ..interpreter import ExternalProgramHolder, extract_required_kwarg
+from ..interpreter import ExternalProgramHolder, extract_required_kwarg, permitted_kwargs
 from ..build import known_shmod_kwargs
 from .. import mlog
 from ..environment import detect_cpu_family
@@ -47,6 +47,7 @@ def __init__(self, python_holder, environment, kwargs):
         super().__init__('python', environment, None, kwargs)
         self.name = 'python'
         self.static = kwargs.get('static', False)
+        self.embed = kwargs.get('embed', False)
         self.version = python_holder.version
         self.platform = python_holder.platform
         self.pkgdep = None
@@ -62,9 +63,11 @@ def __init__(self, python_holder, environment, kwargs):
         if DependencyMethods.PKGCONFIG in self.methods and not python_holder.is_pypy:
             pkg_version = self.variables.get('LDVERSION') or self.version
             pkg_libdir = self.variables.get('LIBPC')
+            pkg_embed = '-embed' if self.embed and mesonlib.version_compare(self.version, '>=3.8') else ''
+            pkg_name = 'python-{}{}'.format(pkg_version, pkg_embed)
 
             # If python-X.Y.pc exists in LIBPC, we will try to use it
-            if pkg_libdir is not None and Path(os.path.join(pkg_libdir, 'python-{}.pc'.format(pkg_version))).is_file():
+            if pkg_libdir is not None and Path(os.path.join(pkg_libdir, '{}.pc'.format(pkg_name))).is_file():
                 old_pkg_libdir = os.environ.get('PKG_CONFIG_LIBDIR')
                 old_pkg_path = os.environ.get('PKG_CONFIG_PATH')
 
@@ -74,11 +77,11 @@ def __init__(self, python_holder, environment, kwargs):
                     os.environ['PKG_CONFIG_LIBDIR'] = pkg_libdir
 
                 try:
-                    self.pkgdep = PkgConfigDependency('python-{}'.format(pkg_version), environment, kwargs)
-                    mlog.debug('Found "python-{}" via pkgconfig lookup in LIBPC ({})'.format(pkg_version, pkg_libdir))
+                    self.pkgdep = PkgConfigDependency(pkg_name, environment, kwargs)
+                    mlog.debug('Found "{}" via pkgconfig lookup in LIBPC ({})'.format(pkg_name, pkg_libdir))
                     py_lookup_method = 'pkgconfig'
                 except MesonException as e:
-                    mlog.debug('"python-{}" could not be found in LIBPC ({})'.format(pkg_version, pkg_libdir))
+                    mlog.debug('"{}" could not be found in LIBPC ({})'.format(pkg_name, pkg_libdir))
                     mlog.debug(e)
 
                 if old_pkg_path is not None:
@@ -89,16 +92,16 @@ def __init__(self, python_holder, environment, kwargs):
                 else:
                     os.environ.pop('PKG_CONFIG_LIBDIR', None)
             else:
-                mlog.debug('"python-{}" could not be found in LIBPC ({}), this is likely due to a relocated python installation'.format(pkg_version, pkg_libdir))
+                mlog.debug('"{}" could not be found in LIBPC ({}), this is likely due to a relocated python installation'.format(pkg_name, pkg_libdir))
 
             # If lookup via LIBPC failed, try to use fallback PKG_CONFIG_LIBDIR/PKG_CONFIG_PATH mechanisms
             if self.pkgdep is None or not self.pkgdep.found():
                 try:
-                    self.pkgdep = PkgConfigDependency('python-{}'.format(pkg_version), environment, kwargs)
-                    mlog.debug('Found "python-{}" via fallback pkgconfig lookup in PKG_CONFIG_LIBDIR/PKG_CONFIG_PATH'.format(pkg_version))
+                    self.pkgdep = PkgConfigDependency(pkg_name, environment, kwargs)
+                    mlog.debug('Found "{}" via fallback pkgconfig lookup in PKG_CONFIG_LIBDIR/PKG_CONFIG_PATH'.format(pkg_name))
                     py_lookup_method = 'pkgconfig-fallback'
                 except MesonException as e:
-                    mlog.debug('"python-{}" could not be found via fallback pkgconfig lookup in PKG_CONFIG_LIBDIR/PKG_CONFIG_PATH'.format(pkg_version))
+                    mlog.debug('"{}" could not be found via fallback pkgconfig lookup in PKG_CONFIG_LIBDIR/PKG_CONFIG_PATH'.format(pkg_name))
                     mlog.debug(e)
 
         if self.pkgdep and self.pkgdep.found():
@@ -345,6 +348,9 @@ def extension_module_method(self, args, kwargs):
 
         return self.interpreter.func_shared_module(None, args, kwargs)
 
+    @noPosargs
+    @permittedKwargs(permitted_kwargs['dependency'])
+    @FeatureNewKwargs('python_installation.dependency', '0.53.0', ['embed'])
     def dependency_method(self, args, kwargs):
         dep = PythonDependency(self, self.interpreter.environment, kwargs)
         return self.interpreter.holderify(dep)


Test patch:
diff --git a/test cases/frameworks/1 boost/meson.build b/test cases/frameworks/1 boost/meson.build
--- a/test cases/frameworks/1 boost/meson.build	
+++ b/test cases/frameworks/1 boost/meson.build	
@@ -30,8 +30,8 @@ extralibdep = dependency('boost', modules : ['thread', 'system', 'log_setup', 'l
 pymod = import('python')
 python2 = pymod.find_installation('python2', required: host_machine.system() == 'linux', disabler: true)
 python3 = pymod.find_installation('python3', required: host_machine.system() == 'linux', disabler: true)
-python2dep = python2.dependency(required: host_machine.system() == 'linux', disabler: true)
-python3dep = python3.dependency(required: host_machine.system() == 'linux', disabler: true)
+python2dep = python2.dependency(required: host_machine.system() == 'linux', embed: true, disabler: true)
+python3dep = python3.dependency(required: host_machine.system() == 'linux', embed: true, disabler: true)
 
 # compile python 2/3 modules only if we found a corresponding python version
 if(python2dep.found() and host_machine.system() == 'linux')