Repository: mesonbuild/meson

Issue:
`python.extension_module()` linking with libpython when it should not
Since version 0.63.0 Meson automatically adds a dependency on libpython when needed compiling extension modules. To determine whether to link to libpython Meson consults `distutils`. On Python 3.7 on Linux, `distutils` says that libpython should not be linked:
```python
>>> import distutils.core
>>> cmd = distutils.core.Distribution().get_command_obj('build_ext')
>>> cmd.ensure_finalized()
>>> cmd.get_libraries(distutils.core.Extension('dummy', []))
[]
```
However, the `pkg-config` file for python contains these linker arguments
```
$ pkg-config python3 --libs
-L/opt/_internal/cpython-3.7.15/lib -lpython3.7m
```
This code https://github.com/mesonbuild/meson/blob/fd43842041e8ca83b1baee94f053c814e144f98b/mesonbuild/modules/python.py#L524-L543
deals with adding the libpython dependency as need. In the case the libpython dependency is explicitly requested, only the compile arguments are used, however, in the case when it is not explicitly requested, all the arguments are added, including the link arguments found via `pkg-config`. I think that only the compile arguments should be included in this case too when the extension module should not be linked with libpython.

I'll prepare a PR, but I would like to check with @eli-schwartz if he agrees on the diagnosis.


Discussion:
This diagnosis confuses me since we specifically check for link_libpython and remove the link arguments (keeping only the compile ones).

However, on 3.7 you are supposed to link to libpython, except on Debian...
Oh, I think I see it now.

Edit: or not? This should already work?
I think the patch makes it clearer. I actually encountered the issue on the manylinux docker images, which are based on AlmaLinux. I suspect Debian may be patching the pkg-condig too.
The thing is that the patch is exactly what I thought you meant, and I don't see how that's possible to be a problem.

The internal dependency is `embed: false`. If you provide one we don't know what it is so we partial it.
I stepped through the thing with `pdb` and the linker flags are added. I'm afraid I don't know enough of the surrounding code to understand exactly why. However, I just verified that this solves the issue on Linux and a segfault on macOS.

The pkg-config definition oh the system looks like this:
```
# cat /opt/_internal/cpython-3.7.15/lib/pkgconfig/python3.pc
# See: man pkg-config
prefix=/opt/_internal/cpython-3.7.15
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: Python
Description: Python library
Requires:
Version: 3.7
Libs.private: -lcrypt -lpthread -ldl  -lutil
Libs: -L${libdir} -lpython3.7m
Cflags: -I${includedir}/python3.7m
```
Let me know if there is anything else that may be useful to better understand the root cause.

Edit: yes, I verified that this is the pkg-config picked up by Meson.
I think the bug is most likely in the discrepancy between `distutils` and `pkg-config` for Python 3.7. But fixing it in the Python seems unlikely to happen at this point. I think that dealing with it in Meson is the only option.
Also note that the distinction from `python3.pc` and `python3-embed.c` comes only starting with Python 3.8, so as long as Meson supports Python 3.7, we need to filter out the link flags.
> The internal dependency is `embed: false`.

I had a closer look at the Meson python module code and I understood better what you mean with the above sentence. When the dependency is looked up with `embed=False` the `python3.pc` is looked up and when `embed=True` the `python3-embed.pc` is used instead. As I was writing above, up to Python 3.7 there is no `python3-embed.pc` thus this does not have the effect of removing the linking to libpython as per the `python3.pc` reported above.

Resolving patch:
diff --git a/mesonbuild/cmake/interpreter.py b/mesonbuild/cmake/interpreter.py
--- a/mesonbuild/cmake/interpreter.py
+++ b/mesonbuild/cmake/interpreter.py
@@ -131,7 +131,7 @@ class OutputTargetMap:
     rm_so_version = re.compile(r'(\.[0-9]+)+$')
 
     def __init__(self, build_dir: Path):
-        self.tgt_map = {}          # type: T.Dict[str, T.Union['ConverterTarget', 'ConverterCustomTarget']]
+        self.tgt_map: T.Dict[str, T.Union['ConverterTarget', 'ConverterCustomTarget']] = {}
         self.build_dir = build_dir
 
     def add(self, tgt: T.Union['ConverterTarget', 'ConverterCustomTarget']) -> None:
@@ -220,38 +220,38 @@ def __init__(self, target: CMakeTarget, env: 'Environment', for_machine: Machine
         self.full_name = target.full_name
         self.type = target.type
         self.install = target.install
-        self.install_dir = None  # type: T.Optional[Path]
+        self.install_dir: T.Optional[Path] = None
         self.link_libraries = target.link_libraries
         self.link_flags = target.link_flags + target.link_lang_flags
-        self.depends_raw = []   # type: T.List[str]
-        self.depends = []       # type: T.List[T.Union[ConverterTarget, ConverterCustomTarget]]
+        self.depends_raw: T.List[str] = []
+        self.depends: T.List[T.Union[ConverterTarget, ConverterCustomTarget]] = []
 
         if target.install_paths:
             self.install_dir = target.install_paths[0]
 
-        self.languages = set()          # type: T.Set[str]
-        self.sources = []               # type: T.List[Path]
-        self.generated = []             # type: T.List[Path]
-        self.generated_ctgt = []        # type: T.List[CustomTargetReference]
-        self.includes = []              # type: T.List[Path]
-        self.sys_includes = []          # type: T.List[Path]
-        self.link_with = []             # type: T.List[T.Union[ConverterTarget, ConverterCustomTarget]]
-        self.object_libs = []           # type: T.List[ConverterTarget]
-        self.compile_opts = {}          # type: T.Dict[str, T.List[str]]
-        self.public_compile_opts = []   # type: T.List[str]
+        self.languages: T.Set[str] = set()
+        self.sources: T.List[Path] = []
+        self.generated: T.List[Path] = []
+        self.generated_ctgt: T.List[CustomTargetReference] = []
+        self.includes: T.List[Path] = []
+        self.sys_includes: T.List[Path] = []
+        self.link_with: T.List[T.Union[ConverterTarget, ConverterCustomTarget]] = []
+        self.object_libs: T.List[ConverterTarget] = []
+        self.compile_opts: T.Dict[str, T.List[str]] = {}
+        self.public_compile_opts: T.List[str] = []
         self.pie = False
 
         # Project default override options (c_std, cpp_std, etc.)
-        self.override_options = []  # type: T.List[str]
+        self.override_options: T.List[str] = []
 
         # Convert the target name to a valid meson target name
         self.name = _sanitize_cmake_name(self.name)
 
-        self.generated_raw = []     # type: T.List[Path]
+        self.generated_raw: T.List[Path] = []
 
         for i in target.files:
-            languages = set()       # type: T.Set[str]
-            src_suffixes = set()    # type: T.Set[str]
+            languages: T.Set[str] = set()
+            src_suffixes: T.Set[str] = set()
 
             # Insert suffixes
             for j in i.sources:
@@ -483,7 +483,7 @@ def process_object_libs(self, obj_target_list: T.List['ConverterTarget'], linker
                 # suffix and just produces object files like `foo.obj`. Thus we have to do our best to
                 # undo this step and guess the correct language suffix of the object file. This is done
                 # by trying all language suffixes meson knows and checking if one of them fits.
-                candidates = [j]  # type: T.List[str]
+                candidates = [j]
                 if not any(j.endswith('.' + x) for x in exts):
                     mlog.warning('Object files do not contain source file extensions, thus falling back to guessing them.', once=True)
                     candidates += [f'{j}.{x}' for x in exts]
@@ -517,7 +517,7 @@ def _append_objlib_sources(self, tgt: 'ConverterTarget') -> None:
 
     @lru_cache(maxsize=None)
     def _all_source_suffixes(self) -> 'ImmutableListProtocol[str]':
-        suffixes = []  # type: T.List[str]
+        suffixes: T.List[str] = []
         for exts in lang_suffixes.values():
             suffixes.extend(exts)
         return suffixes
@@ -583,8 +583,8 @@ def log(self) -> None:
 
 class CustomTargetReference:
     def __init__(self, ctgt: 'ConverterCustomTarget', index: int) -> None:
-        self.ctgt = ctgt    # type: ConverterCustomTarget
-        self.index = index  # type: int
+        self.ctgt = ctgt
+        self.index = index
 
     def __repr__(self) -> str:
         if self.valid():
@@ -599,8 +599,8 @@ def filename(self) -> str:
         return self.ctgt.outputs[self.index]
 
 class ConverterCustomTarget:
-    tgt_counter = 0  # type: int
-    out_counter = 0  # type: int
+    tgt_counter = 0
+    out_counter = 0
 
     def __init__(self, target: CMakeGeneratorTarget, env: 'Environment', for_machine: MachineChoice) -> None:
         assert target.current_bin_dir is not None
@@ -612,14 +612,14 @@ def __init__(self, target: CMakeGeneratorTarget, env: 'Environment', for_machine
         self.cmake_name = str(self.name)
         self.original_outputs = list(target.outputs)
         self.outputs = [x.name for x in self.original_outputs]
-        self.conflict_map = {}                          # type: T.Dict[str, str]
-        self.command = []                               # type: T.List[T.List[T.Union[str, ConverterTarget]]]
+        self.conflict_map: T.Dict[str, str] = {}
+        self.command: T.List[T.List[T.Union[str, ConverterTarget]]] = []
         self.working_dir = target.working_dir
         self.depends_raw = target.depends
-        self.inputs = []                                # type: T.List[T.Union[str, CustomTargetReference]]
-        self.depends = []                               # type: T.List[T.Union[ConverterTarget, ConverterCustomTarget]]
-        self.current_bin_dir = target.current_bin_dir   # type: Path
-        self.current_src_dir = target.current_src_dir   # type: Path
+        self.inputs: T.List[T.Union[str, CustomTargetReference]] = []
+        self.depends: T.List[T.Union[ConverterTarget, ConverterCustomTarget]] = []
+        self.current_bin_dir = target.current_bin_dir
+        self.current_src_dir = target.current_src_dir
         self.env = env
         self.for_machine = for_machine
         self._raw_target = target
@@ -652,7 +652,7 @@ def ensure_absolute(x: Path) -> Path:
         # Ensure that there is no duplicate output in the project so
         # that meson can handle cases where the same filename is
         # generated in multiple directories
-        temp_outputs = []  # type: T.List[str]
+        temp_outputs: T.List[str] = []
         for i in self.outputs:
             if i in all_outputs:
                 old = str(i)
@@ -664,11 +664,11 @@ def ensure_absolute(x: Path) -> Path:
         self.outputs = temp_outputs
 
         # Check if the command is a build target
-        commands = []  # type: T.List[T.List[T.Union[str, ConverterTarget]]]
+        commands: T.List[T.List[T.Union[str, ConverterTarget]]] = []
         for curr_cmd in self._raw_target.command:
             assert isinstance(curr_cmd, list)
             assert curr_cmd[0] != '', "An empty string is not a valid executable"
-            cmd = []  # type: T.List[T.Union[str, ConverterTarget]]
+            cmd: T.List[T.Union[str, ConverterTarget]] = []
 
             for j in curr_cmd:
                 if not j:
@@ -775,25 +775,25 @@ def __init__(self, build: 'Build', subdir: Path, src_dir: Path, install_prefix:
         self.env = env
         self.for_machine = MachineChoice.HOST # TODO make parameter
         self.backend_name = backend.name
-        self.linkers = set()  # type: T.Set[str]
+        self.linkers: T.Set[str] = set()
         self.fileapi = CMakeFileAPI(self.build_dir)
 
         # Raw CMake results
-        self.bs_files = []    # type: T.List[Path]
-        self.codemodel_configs = None  # type: T.Optional[T.List[CMakeConfiguration]]
-        self.cmake_stderr = None  # type: T.Optional[str]
+        self.bs_files: T.List[Path] = []
+        self.codemodel_configs: T.Optional[T.List[CMakeConfiguration]] = None
+        self.cmake_stderr: T.Optional[str] = None
 
         # Analysed data
         self.project_name = ''
-        self.languages = []  # type: T.List[str]
-        self.targets = []  # type: T.List[ConverterTarget]
-        self.custom_targets = []  # type: T.List[ConverterCustomTarget]
+        self.languages: T.List[str] = []
+        self.targets: T.List[ConverterTarget] = []
+        self.custom_targets: T.List[ConverterCustomTarget] = []
         self.trace: CMakeTraceParser
         self.output_target_map = OutputTargetMap(self.build_dir)
 
         # Generated meson data
-        self.generated_targets = {}  # type: T.Dict[str, T.Dict[str, T.Optional[str]]]
-        self.internal_name_map = {}  # type: T.Dict[str, str]
+        self.generated_targets: T.Dict[str, T.Dict[str, T.Optional[str]]] = {}
+        self.internal_name_map: T.Dict[str, str] = {}
 
         # Do some special handling for object libraries for certain configurations
         self._object_lib_workaround = False
@@ -890,7 +890,7 @@ def analyse(self) -> None:
         self.trace.parse(self.cmake_stderr)
 
         # Find all targets
-        added_target_names = []  # type: T.List[str]
+        added_target_names: T.List[str] = []
         for i_0 in self.codemodel_configs:
             for j_0 in i_0.projects:
                 if not self.project_name:
@@ -927,7 +927,7 @@ def analyse(self) -> None:
 
         # First pass: Basic target cleanup
         object_libs = []
-        custom_target_outputs = []  # type: T.List[str]
+        custom_target_outputs: T.List[str] = []
         for ctgt in self.custom_targets:
             ctgt.postprocess(self.output_target_map, self.src_dir, custom_target_outputs, self.trace)
         for tgt in self.targets:
@@ -1026,9 +1026,9 @@ def assign(var_name: str, value: BaseNode) -> AssignmentNode:
         # Add the run script for custom commands
 
         # Add the targets
-        processing = []   # type: T.List[str]
-        processed = {}   # type: T.Dict[str, T.Dict[str, T.Optional[str]]]
-        name_map = {}   # type: T.Dict[str, str]
+        processing: T.List[str] = []
+        processed: T.Dict[str, T.Dict[str, T.Optional[str]]] = {}
+        name_map: T.Dict[str, str] = {}
 
         def extract_tgt(tgt: T.Union[ConverterTarget, ConverterCustomTarget, CustomTargetReference]) -> IdNode:
             tgt_name = None
@@ -1056,13 +1056,13 @@ def process_target(tgt: ConverterTarget) -> None:
             detect_cycle(tgt)
 
             # First handle inter target dependencies
-            link_with = []              # type: T.List[IdNode]
-            objec_libs = []             # type: T.List[IdNode]
-            sources = []                # type: T.List[Path]
-            generated = []              # type: T.List[T.Union[IdNode, IndexNode]]
-            generated_filenames = []    # type: T.List[str]
-            custom_targets = []         # type: T.List[ConverterCustomTarget]
-            dependencies = []           # type: T.List[IdNode]
+            link_with: T.List[IdNode] = []
+            objec_libs: T.List[IdNode] = []
+            sources: T.List[Path] = []
+            generated: T.List[T.Union[IdNode, IndexNode]] = []
+            generated_filenames: T.List[str] = []
+            custom_targets: T.List[ConverterCustomTarget] = []
+            dependencies: T.List[IdNode] = []
             for i in tgt.link_with:
                 assert isinstance(i, ConverterTarget)
                 if i.name not in processed:
@@ -1123,7 +1123,7 @@ def process_target(tgt: ConverterTarget) -> None:
             install_tgt = options.get_install(tgt.cmake_name, tgt.install)
 
             # Generate target kwargs
-            tgt_kwargs = {
+            tgt_kwargs: TYPE_mixed_kwargs = {
                 'build_by_default': install_tgt,
                 'link_args': options.get_link_args(tgt.cmake_name, tgt.link_flags + tgt.link_libraries),
                 'link_with': link_with,
@@ -1131,7 +1131,7 @@ def process_target(tgt: ConverterTarget) -> None:
                 'install': install_tgt,
                 'override_options': options.get_override_options(tgt.cmake_name, tgt.override_options),
                 'objects': [method(x, 'extract_all_objects') for x in objec_libs],
-            }  # type: TYPE_mixed_kwargs
+            }
 
             # Only set if installed and only override if it is set
             if install_tgt and tgt.install_dir:
@@ -1148,12 +1148,12 @@ def process_target(tgt: ConverterTarget) -> None:
                 tgt_kwargs['pic'] = tgt.pie
 
             # declare_dependency kwargs
-            dep_kwargs = {
+            dep_kwargs: TYPE_mixed_kwargs = {
                 'link_args': tgt.link_flags + tgt.link_libraries,
                 'link_with': id_node(tgt_var),
                 'compile_args': tgt.public_compile_opts,
                 'include_directories': id_node(inc_var),
-            }  # type: TYPE_mixed_kwargs
+            }
 
             if dependencies:
                 generated += dependencies
@@ -1195,7 +1195,7 @@ def process_custom_target(tgt: ConverterCustomTarget) -> None:
             # directory.
 
             detect_cycle(tgt)
-            tgt_var = tgt.name  # type: str
+            tgt_var = tgt.name
 
             def resolve_source(x: T.Union[str, ConverterTarget, ConverterCustomTarget, CustomTargetReference]) -> T.Union[str, IdNode, IndexNode]:
                 if isinstance(x, ConverterTarget):
@@ -1214,7 +1214,7 @@ def resolve_source(x: T.Union[str, ConverterTarget, ConverterCustomTarget, Custo
                     return x
 
             # Generate the command list
-            command = []  # type: T.List[T.Union[str, IdNode, IndexNode]]
+            command: T.List[T.Union[str, IdNode, IndexNode]] = []
             command += mesonlib.get_meson_command()
             command += ['--internal', 'cmake_run_ctgt']
             command += ['-o', '@OUTPUT@']
@@ -1226,12 +1226,12 @@ def resolve_source(x: T.Union[str, ConverterTarget, ConverterCustomTarget, Custo
             for cmd in tgt.command:
                 command += [resolve_source(x) for x in cmd] + [';;;']
 
-            tgt_kwargs = {
+            tgt_kwargs: TYPE_mixed_kwargs = {
                 'input': [resolve_source(x) for x in tgt.inputs],
                 'output': tgt.outputs,
                 'command': command,
                 'depends': [resolve_source(x) for x in tgt.depends],
-            }  # type: TYPE_mixed_kwargs
+            }
 
             root_cb.lines += [assign(tgt_var, function('custom_target', [tgt.name], tgt_kwargs))]
             processed[tgt.name] = {'inc': None, 'src': None, 'dep': None, 'tgt': tgt_var, 'func': 'custom_target'}
diff --git a/mesonbuild/modules/python.py b/mesonbuild/modules/python.py
--- a/mesonbuild/modules/python.py
+++ b/mesonbuild/modules/python.py
@@ -121,6 +121,17 @@ def __init__(self, name: str, environment: 'Environment',
         if libpc and not self.is_found:
             mlog.debug(f'"python-{self.version}" could not be found in LIBPC, this is likely due to a relocated python installation')
 
+        # The "-embed" version of python.pc was introduced in 3.8, and distutils
+        # extension linking was changed to be considered a non embed usage. Before
+        # then, this dependency always uses the embed=True file because that is the
+        # only one that exists,
+        #
+        # On macOS and some Linux distros (Debian) distutils doesn't link extensions
+        # against libpython, even on 3.7 and below. We call into distutils and
+        # mirror its behavior. See https://github.com/mesonbuild/meson/issues/4117
+        if not self.embed and not self.link_libpython and mesonlib.version_compare(self.version, '< 3.8'):
+            self.link_args = []
+
 
 class PythonFrameworkDependency(ExtraFrameworkDependency, _PythonDependencyBase):
 
@@ -529,17 +540,8 @@ def extension_module_method(self, args: T.List['TYPE_var'], kwargs: 'TYPE_kwargs
 
             kwargs['install_dir'] = self._get_install_dir_impl(False, subdir)
 
-        new_deps = []
-        has_pydep = False
-        for dep in mesonlib.extract_as_list(kwargs, 'dependencies'):
-            if isinstance(dep, _PythonDependencyBase):
-                has_pydep = True
-                # On macOS and some Linux distros (Debian) distutils doesn't link
-                # extensions against libpython. We call into distutils and mirror its
-                # behavior. See https://github.com/mesonbuild/meson/issues/4117
-                if not self.link_libpython:
-                    dep = dep.get_partial_dependency(compile_args=True)
-            new_deps.append(dep)
+        new_deps = mesonlib.extract_as_list(kwargs, 'dependencies')
+        has_pydep = any(isinstance(dep, _PythonDependencyBase) for dep in new_deps)
         if not has_pydep:
             pydep = self._dependency_method_impl({})
             if not pydep.found():


Test patch: