Repository: scipy/scipy

Issue:
Bug: Meson build with dev.py fails to detect SciPy with debian python
While trying to build SciPy with meson on a 32bit machine for [#15814](https://github.com/scipy/scipy/issues/15814)  (using docker image `ubuntu:i386:bionic`; though this is not an architecture specific bug, I think)

After installing python3.8  and other required deps (through `apt` in ubuntu), and then python deps with pip3 (installed through [get-pip.py](https://bootstrap.pypa.io/get-pip.py)).

On running `python3.8 dev.py -j 2 --build-only` , build goes through fine; even the installation goes through fine.
```
Meson build setup OK
Build OK
Installing, see meson-install.log...
Installation OK
```

But later the script fails when [`get_project_info`](https://github.com/scipy/scipy/blob/653fd090fe18e28254986cd4c446c47b7051ef57/dev.py#L385) is called in dev.py script to import scipy, with the error message attached below.

```bash
Trying to find scipy from development installed path at: /__w/scipy/scipy/build-install/lib/python3.8/site-packages
Traceback (most recent call last):
  File "dev.py", line 393, in get_project_info
    test, version, mod_path = runtests.import_module()
  File "/__w/scipy/scipy/runtests.py", line 356, in import_module
    __import__(PROJECT_MODULE)
ModuleNotFoundError: No module named 'scipy'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "dev.py", line 635, in <module>
    main(argv=sys.argv[1:])
  File "dev.py", line 282, in main
    test, version, mod_path = get_project_info()
  File "dev.py", line 403, in get_project_info
    test, version, mod_path = runtests.import_module()
  File "/__w/scipy/scipy/runtests.py", line 356, in import_module
    __import__(PROJECT_MODULE)
ModuleNotFoundError: No module named 'scipy'
Error: Process completed with exit code 1.
```

The problem is that in dev.py, [`get_site_packages`](https://github.com/scipy/scipy/blob/653fd090fe18e28254986cd4c446c47b7051ef57/dev.py#L556) method returns `'/scipy/build-install/lib/python3.8/site-packages'` but actually meson installs scipy to `'/scipy/build-install/lib/python3/dist-packages'`

Feel free to check logs about the error and for reproducibility [here](https://github.com/AnirudhDagar/scipy/runs/6197379647?check_suite_focus=true).

### Additional Information
When I run ` python3.8 -c "import site; print(site.getsitepackages())"` I get the following:
```
['/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages']
```

I guess this issue is also discussed in mesonbuild/meson#8739, explaining problems with debian python and dist-packages vs site-packages or python3.x vs python3, depending on what python (system, venv, compiled) is being used.

Edit 1: Adding more info for someone who reads this later. Great blog on [Python, Debian, and the install locations](https://ffy00.github.io/blog/02-python-debian-and-the-install-locations/) and meson patching similar problem in https://github.com/mesonbuild/meson/pull/9288 

#### SciPy/NumPy/Python version information
`1.9.0.dev0+1925.653fd09 1.19.0 sys.version_info(major=3, minor=8, micro=0, releaselevel='final', serial=0)`

### Pitch
I can suggest a hacky fix to tweak `dev.py` specifically in [`get_project_info`](https://github.com/scipy/scipy/blob/1a945cae6ddc9186318cf34cfe1756cdd86eea34/dev.py#L385) or more specifically `PATH_INSTALLED` variable to also fall back to check for other possible installation paths that meson used in debian like here, for example utilizing path from `['/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages', '/usr/lib/python3.8/dist-packages']` and stripping them to
`['/lib/python3.8/dist-packages', '/lib/python3/dist-packages', '/lib/python3.8/dist-packages']` and then joining it with the prefix.

There is probably a better way to do this. I can send a PR after a discussion if this is indeed a valid bug.

cc @rgommers @Smit-create 


Discussion:
Thanks for the detailed bug report @AnirudhDagar. You diagnosed it correctly, it's indeed due to that mess Debian made. It looks like it boils down to:

```python
>>> import sys
>>> from sysconfig import get_path
>>> from pathlib import Path
>>> plat_path = Path(get_path("platlib"))
```
where `plat_path` gives `python3.8/site-packages` while things get installed to `python3/dist-packages`. 

I don't think there's a correct solution, so a hacky one to extend `get_site_packages` and see if the constructed path exists and trying `python3` and `dist-packages` fallbacks if not seems like the right thing to do here. A PR would be great.

Resolving patch:
diff --git a/dev.py b/dev.py
--- a/dev.py
+++ b/dev.py
@@ -50,6 +50,7 @@
 
 import sys
 import os
+import warnings  # noqa: E402
 from pathlib import Path
 import platform
 # the following multiprocessing import is necessary to prevent tests that use
@@ -57,6 +58,13 @@
 # import is enough...
 import multiprocessing
 
+# distutils is required to infer meson install path
+# if this needs to be replaced for Python 3.12 support and there's no
+# stdlib alternative, use the hack discussed in gh-16058
+with warnings.catch_warnings():
+    warnings.filterwarnings("ignore", category=DeprecationWarning)
+    from distutils import dist
+    from distutils.command.install import INSTALL_SCHEMES
 
 # In case we are run from the source directory, we don't want to import the
 # project from there:
@@ -550,7 +558,18 @@ def copy_openblas():
 
 
 def get_site_packages():
-    plat_path = Path(get_path('platlib'))
+    """
+    Depending on whether we have debian python or not,
+    return dist_packages path or site_packages path.
+    """
+    if 'deb_system' in INSTALL_SCHEMES:
+        # Debian patched python in use
+        install_cmd = dist.Distribution().get_command_obj('install')
+        install_cmd.select_scheme('deb_system')
+        install_cmd.finalize_options()
+        plat_path = Path(install_cmd.install_platlib)
+    else:
+        plat_path = Path(get_path('platlib'))
     return str(Path(PATH_INSTALLED) / plat_path.relative_to(sys.exec_prefix))
 
 
@@ -576,8 +595,6 @@ def build_project(args):
 
     setup_build(args, env)
 
-    site_dir = get_site_packages()
-
     cmd = ["ninja", "-C", args.build_dir]
     if args.parallel > 1:
         cmd += ["-j", str(args.parallel)]
@@ -593,6 +610,8 @@ def build_project(args):
 
     install_project(args)
 
+    site_dir = get_site_packages()
+
     if args.win_cp_openblas and platform.system() == 'Windows':
         if copy_openblas() == 0:
             print('OpenBLAS copied')
diff --git a/do.py b/do.py
--- a/do.py
+++ b/do.py
@@ -97,6 +97,7 @@ def task_meta(cls, **kwargs):
 import os
 import subprocess
 import sys
+import warnings
 import shutil
 import json
 import datetime
@@ -106,6 +107,15 @@ def task_meta(cls, **kwargs):
 import errno
 import contextlib
 from sysconfig import get_path
+
+# distutils is required to infer meson install path
+# if this needs to be replaced for Python 3.12 support and there's no
+# stdlib alternative, use CmdAction and the hack discussed in gh-16058
+with warnings.catch_warnings():
+    warnings.filterwarnings("ignore", category=DeprecationWarning)
+    from distutils import dist
+    from distutils.command.install import INSTALL_SCHEMES
+
 from pathlib import Path
 from collections import namedtuple
 from types import ModuleType as new_module
@@ -300,8 +310,7 @@ def __init__(self, args=None):
             self.installed = self.build.parent / (self.build.stem + "-install")
         # relative path for site-package with py version
         # i.e. 'lib/python3.10/site-packages'
-        py_lib_path = Path(get_path('platlib')).relative_to(sys.exec_prefix)
-        self.site = self.installed / py_lib_path
+        self.site = self.get_site_packages()
 
     def add_sys_path(self):
         """Add site dir to sys.path / PYTHONPATH"""
@@ -310,6 +319,21 @@ def add_sys_path(self):
         os.environ['PYTHONPATH'] = \
             os.pathsep.join((site_dir, os.environ.get('PYTHONPATH', '')))
 
+    def get_site_packages(self):
+        """
+        Depending on whether we have debian python or not,
+        return dist_packages path or site_packages path.
+        """
+        if 'deb_system' in INSTALL_SCHEMES:
+            # debian patched python in use
+            install_cmd = dist.Distribution().get_command_obj('install')
+            install_cmd.select_scheme('deb_system')
+            install_cmd.finalize_options()
+            plat_path = Path(install_cmd.install_platlib)
+        else:
+            plat_path = Path(get_path('platlib'))
+        return self.installed / plat_path.relative_to(sys.exec_prefix)
+
 
 @contextlib.contextmanager
 def working_dir(new_dir):


Test patch: