Repository: mesonbuild/meson

Issue:
Add `pure` setting to `python_installation` object
tl;dr of this proposal is: make it possible for Python projects to set `pure` in a single place, rather than have to repeat `pure: false` over and over.

This follows up on https://github.com/mesonbuild/meson/pull/10514#issuecomment-1161405026 (Cc @eli-schwartz)

### What is the problem?

For SciPy we currently have 99 lines of `pure: false` in `meson.build` files. For each subdir, we typically have one instance of:
```meson

python_sources = [
  '__init__.py',
  'xxx.py',  # list of file names
]

py3.install_sources(
  python_sources,
  pure: false,
  subdir: 'scipy/some_subdir'
)
```

That `pure: false` boilerplate not only gets repetitive, there are two other issues with it:
1. If you forget it, things will still work in CI, and then on some random Linux distro that happens to still define `purelib` and `platlib` to different directories, the install will be broken at runtime (because Python packages should be in a single source tree, not split - I have seen this issue once already).
2. It makes the average maintainer/contributor, who will not know what `purelib`/`platlib` is about, wonder whether this is relevant - waste of time.

### What does `pure` mean again?

The best description is probably https://peps.python.org/pep-0491/#what-s-the-deal-with-purelib-vs-platlib.

 The last sentence says: _"In practice a wheel should have only one of “purelib” or “platlib” depending on whether it is pure Python or not and those files should be at the root with the appropriate setting given for “Root-is-purelib”."_ This makes sense; and explains why things break if you forget `pure: false` once in a Python package that contains compiled code - the structure of the Python package gets lost if files get scattered over both `purelib` _and_ `platlib`.

Let me also emphasize that the whole purelib/platlib distinction is Linux-only, for a minority of Linux distros, and a leftover from a time where people jumped through hoops to save a little bit of disk space. It makes very little sense as a design in today's context, and should be hidden as much as possible (unfortunately it cannot be removed from Python packaging completely).

### How can we improve things here in Meson?

Because all files for a complete Python package need to have the same `pure` setting, it makes sense for that setting to live on the `python_installation` object, rather than having to specify it every time you use `python_installation.install_sources`. I'm not sure what the best way of adding it there; if the typical code in the top-level `meson.build` looks like:
```meson
py_mod = import('python')
py3 = py_mod.find_installation()
py3_dep = py3.dependency()
```
then it could be a keyword to `find_installation()`, or a separate method like `py3 = py3.set_purelib('false')`.

It should be possible to add something like this in a backwards-compatible way; if it's not used then `install_sources` continues to default to `true`, and if it is given then that overwrites the default. 


Discussion:
A somewhat similar situation happens with the `install: true` argument to `extension_module()`. That is not nearly as much of a problem, because if you forget it things break immediately rather than at runtime on a few Linux distros only - but it's also a lot of boilerplate. For `py3.extension_module`, in any Python package you are likely to want to install every single extension, but you now have to type `install: true` on every line. Adding it to the `python_installation` object once as a default could be considered, if my proposal for `pure` above is acceptable.
I'm not 100% sure what I think about this, but if we do implement this I think we'd probably want to do it as a kwarg on find_installation(), rather than a method that mutates an existing found installation.

Aside: another possible solution is implementing some `functools.partial`-like method of preparing a function with set arguments, a topic that has come up a few times as well in relation to generator() being an inferior solution to custom_target in a for loop in many cases.

Resolving patch:
diff --git a/mesonbuild/modules/python.py b/mesonbuild/modules/python.py
--- a/mesonbuild/modules/python.py
+++ b/mesonbuild/modules/python.py
@@ -14,6 +14,7 @@
 from __future__ import annotations
 
 from pathlib import Path
+import copy
 import functools
 import json
 import os
@@ -67,7 +68,7 @@ class PythonIntrospectionDict(TypedDict):
 
     class PyInstallKw(TypedDict):
 
-        pure: bool
+        pure: T.Optional[bool]
         subdir: str
         install_tag: T.Optional[str]
 
@@ -75,6 +76,7 @@ class FindInstallationKw(ExtractRequired):
 
         disabler: bool
         modules: T.List[str]
+        pure: T.Optional[bool]
 
     _Base = ExternalDependency
 else:
@@ -409,6 +411,7 @@ def __init__(self, name: str, command: T.Optional[T.List[str]] = None,
             'variables': {},
             'version': '0.0',
         }
+        self.pure: bool = True
 
     def _check_version(self, version: str) -> bool:
         if self.name == 'python2':
@@ -472,7 +475,7 @@ def _get_path(self, state: T.Optional['ModuleState'], key: str) -> None:
         return rel_path
 
 
-_PURE_KW = KwargInfo('pure', bool, default=True)
+_PURE_KW = KwargInfo('pure', (bool, NoneType))
 _SUBDIR_KW = KwargInfo('subdir', str, default='')
 
 
@@ -485,6 +488,7 @@ def __init__(self, python: 'PythonExternalProgram', interpreter: 'Interpreter'):
         self.variables = info['variables']
         self.suffix = info['suffix']
         self.paths = info['paths']
+        self.pure = python.pure
         self.platlib_install_path = os.path.join(prefix, python.platlib)
         self.purelib_install_path = os.path.join(prefix, python.purelib)
         self.version = info['version']
@@ -599,7 +603,8 @@ def dependency_method(self, args: T.List['TYPE_var'], kwargs: 'TYPE_kwargs') ->
     def install_sources_method(self, args: T.Tuple[T.List[T.Union[str, mesonlib.File]]],
                                kwargs: 'PyInstallKw') -> 'Data':
         tag = kwargs['install_tag'] or 'runtime'
-        install_dir = self._get_install_dir_impl(kwargs['pure'], kwargs['subdir'])
+        pure = kwargs['pure'] if kwargs['pure'] is not None else self.pure
+        install_dir = self._get_install_dir_impl(pure, kwargs['subdir'])
         return self.interpreter.install_data_impl(
             self.interpreter.source_strings_to_files(args[0]),
             install_dir,
@@ -610,7 +615,8 @@ def install_sources_method(self, args: T.Tuple[T.List[T.Union[str, mesonlib.File
     @noPosargs
     @typed_kwargs('python_installation.install_dir', _PURE_KW, _SUBDIR_KW)
     def get_install_dir_method(self, args: T.List['TYPE_var'], kwargs: 'PyInstallKw') -> str:
-        return self._get_install_dir_impl(kwargs['pure'], kwargs['subdir'])
+        pure = kwargs['pure'] if kwargs['pure'] is not None else self.pure
+        return self._get_install_dir_impl(pure, kwargs['subdir'])
 
     def _get_install_dir_impl(self, pure: bool, subdir: str) -> P_OBJ.OptionString:
         if pure:
@@ -733,6 +739,7 @@ def _find_installation_impl(self, state: 'ModuleState', display_name: str, name_
         KwargInfo('required', (bool, UserFeatureOption), default=True),
         KwargInfo('disabler', bool, default=False, since='0.49.0'),
         KwargInfo('modules', ContainerTypeInfo(list, str), listify=True, default=[], since='0.51.0'),
+        _PURE_KW.evolve(default=True, since='0.64.0'),
     )
     def find_installation(self, state: 'ModuleState', args: T.Tuple[T.Optional[str]],
                           kwargs: 'FindInstallationKw') -> ExternalProgram:
@@ -797,6 +804,8 @@ def find_installation(self, state: 'ModuleState', args: T.Tuple[T.Optional[str]]
                 raise mesonlib.MesonException('{} is missing modules: {}'.format(name_or_path or 'python', ', '.join(missing_modules)))
             return NonExistingExternalProgram()
         else:
+            python = copy.copy(python)
+            python.pure = kwargs['pure']
             return python
 
         raise mesonlib.MesonBugException('Unreachable code was reached (PythonModule.find_installation).')


Test patch:
diff --git a/test cases/python/7 install path/meson.build b/test cases/python/7 install path/meson.build
--- a/test cases/python/7 install path/meson.build	
+++ b/test cases/python/7 install path/meson.build	
@@ -8,5 +8,13 @@ project('install path',
 py = import('python').find_installation()
 py.install_sources('test.py')
 py.install_sources('test.py', pure: false)
+install_data('test.py', install_dir: py.get_install_dir() / 'data')
+install_data('test.py', install_dir: py.get_install_dir(pure: false) / 'data')
+
+py_plat = import('python').find_installation(pure: false)
+py_plat.install_sources('test.py', subdir: 'kw')
+py_plat.install_sources('test.py', pure: true, subdir: 'kwrevert')
+install_data('test.py', install_dir: py_plat.get_install_dir() / 'kw/data')
+install_data('test.py', install_dir: py_plat.get_install_dir(pure: true) / 'kwrevert/data')
 
 subdir('structured')
diff --git a/test cases/python/7 install path/test.json b/test cases/python/7 install path/test.json
--- a/test cases/python/7 install path/test.json	
+++ b/test cases/python/7 install path/test.json	
@@ -6,7 +6,13 @@
     {"type": "file", "file": "pure/alpha/two.py"},
     {"type": "file", "file": "pure/alpha/three.py"},
     {"type": "file", "file": "pure/beta/one.py"},
+    {"type": "file", "file": "plat/kw/test.py"},
+    {"type": "file", "file": "pure/kwrevert/test.py"},
     {"type": "file", "file": "plat/test.py"},
-    {"type": "file", "file": "pure/test.py"}
+    {"type": "file", "file": "pure/test.py"},
+    {"type": "file", "file": "plat/data/test.py"},
+    {"type": "file", "file": "pure/data/test.py"},
+    {"type": "file", "file": "plat/kw/data/test.py"},
+    {"type": "file", "file": "pure/kwrevert/data/test.py"}
   ]
 }