Repository: mesonbuild/meson

Issue:
built-in python find_installation failure using msi package on windows
The following meson.build file:

```
project('test',` 'c')

pymod = import('python')
python3 = pymod.find_installation()
```
gives the following error output when run on windows using the MSI package (tested with both 0.49.2 and 0.50)

```
> meson build
The Meson build system
Version: 0.50.0
Source dir: c:\cygwin64\home\bricha3\meson-test
Build dir: c:\cygwin64\home\bricha3\meson-test\build
Build type: native build
Project name: test
Project version: undefined
Native C compiler: clang (clang 7.0.1 "clang version 7.0.1 (tags/RELEASE_701/final)")
Build machine cpu family: x86_64
Build machine cpu: x86_64
Using meson's python ['C:\\Program Files\\Meson\\meson.exe', 'runpython']

meson.build:4:0: ERROR: <ExternalProgram 'python3' -> ['C:\\Program Files\\Meson\\meson.exe', 'runpython']> is not a valid python

A full log can be found at c:\cygwin64\home\bricha3\meson-test\build\meson-logs\meson-log.txt

```

The log file indicates no further information:

```
Build started at 2019-04-05T11:04:13.179697
Main binary: C:\Program Files\Meson\meson.exe
Python system: Windows
The Meson build system
Version: 0.50.0
Source dir: c:\cygwin64\home\bricha3\meson-test
Build dir: c:\cygwin64\home\bricha3\meson-test\build
Build type: native build
Project name: test
Project version: undefined
Sanity testing C compiler: clang
Is cross compiler: False.
Sanity check compiler command line: clang c:\cygwin64\home\bricha3\meson-test\build\meson-private\sanitycheckc.c -o c:\cygwin64\home\bricha3\meson-test\build\meson-private\sanitycheckc.exe
Sanity check compile stdout:

-----
Sanity check compile stderr:

-----
Running test binary command: c:\cygwin64\home\bricha3\meson-test\build\meson-private\sanitycheckc.exe
Native C compiler: clang (clang 7.0.1 "clang version 7.0.1 (tags/RELEASE_701/final)")
Build machine cpu family: x86_64
Build machine cpu: x86_64
Using meson's python ['C:\\Program Files\\Meson\\meson.exe', 'runpython']

meson.build:4:0: ERROR: <ExternalProgram 'python3' -> ['C:\\Program Files\\Meson\\meson.exe', 'runpython']> is not a valid python
```


Discussion:
When I run the latest meson code manually with python, I do not see this issue - only with the MSI package.

CCing @MathieuDuponchelle who wrote that code.
Further follow-up - the python in the package seems to work fine when called through meson. When I create a simple one-liner python test file, it runs perfectly through meson:

```
c:\....>meson runpython test.py
Hello, world!
```

Therefore it looks like some internal check in meson itself is failing to recognise the valid python.
The issue appears to be with the call to "links_against_libpython()" call in the "INTROSPECT_COMMAND" used to check the python. If I copy that introspect command to a test python file, and run it through meson runpython, I get the following error output:

```
Traceback (most recent call last):
  File "mesonbuild\mesonmain.py", line 122, in run
  File "mesonbuild\mesonmain.py", line 91, in run_runpython_command
  File "C:\python\lib\runpy.py", line 263, in run_path
  File "C:\python\lib\runpy.py", line 96, in _run_module_code
  File "C:\python\lib\runpy.py", line 85, in _run_code
  File "test.py", line 26, in <module>
    'link_libpython': links_against_libpython(),
  File "test.py", line 14, in links_against_libpython
    from distutils.core import Distribution, Extension
ModuleNotFoundError: No module named 'distutils.core'
```

Removing the final line of the print statement gives:

```
{"variables": {"prefix": ".", "exec_prefix": ".", "py_version": "3.6.8", "py_version_short": "3.6", "py_version_nodot": "36", "installed_base": ".", "base": ".", "installed_platbase": ".", "platbase": ".", "projectbase": "C:\\Program Files\\Meson", "abiflags": "", "LIBDEST": "Lib", "BINLIBDEST": "Lib", "INCLUDEPY": "Include", "EXT_SUFFIX": ".pyd", "EXE": ".exe", "VERSION": "36", "BINDIR": "C:\\Program Files\\Meson", "SO": ".pyd", "userbase": "C:\\Users\\bricha3\\AppData\\Roaming\\Python", "srcdir": "C:\\Program Files\\Meson"}, "paths": {"stdlib": "Lib", "platstdlib": "Lib", "purelib": "Lib\\site-packages", "platlib": "Lib\\site-packages", "include": "Include", "platinclude": "Include", "scripts": "Scripts", "data": "."}, "install_paths": {"stdlib": "\\lib\\python3.6", "platstdlib": "\\lib\\python3.6", "purelib": "\\lib\\python3.6\\site-packages", "platlib": "\\lib\\python3.6\\site-packages", "include": "\\include\\python3.6", "platinclude": "include\\python3.6", "scripts": "\\bin", "data": "."}, "version": "3.6", "platform": "win-amd64", "is_pypy": false}
```
This is just missing the "link_libpython: true" bit at the end, which I get if I run the full introspection command through python normally.
this function was added in https://github.com/mesonbuild/meson/pull/4187/ , CC @lazka :)
I can make this particular error go away by copying over the following files to the distutils folder in the meson installation folder (c:\Program Files\Meson\lib\distutils). They probably should be included in the MSI installation package.

archive_util.py
cmd.py
config.py
core.py
debug.py
dep_util.py
dir_util.py
dist.py
errors.py
extension.py
fancy_getopt.py
file_util.py
spawn.py
util.py

Is there a reason not all the distutils files are packaged? Are other parts of the standard python library omitted similarly, because if so, it reduces the usefulness of the python module, since we can't rely on having a complete python installation available?
Error message running that introspection code is now:

```
Traceback (most recent call last):
  File "mesonbuild\mesonmain.py", line 122, in run
  File "mesonbuild\mesonmain.py", line 91, in run_runpython_command
  File "C:\python\lib\runpy.py", line 263, in run_path
  File "C:\python\lib\runpy.py", line 96, in _run_module_code
  File "C:\python\lib\runpy.py", line 85, in _run_code
  File "test.py", line 28, in <module>
    'link_libpython': links_against_libpython(),
  File "test.py", line 17, in links_against_libpython
    cmd = Distribution().get_command_obj('build_ext')
  File "C:\Program Files\Meson\lib\distutils\dist.py", line 857, in get_command_obj
    klass = self.get_command_class(command)
  File "C:\Program Files\Meson\lib\distutils\dist.py", line 843, in get_command_class
    raise DistutilsModuleError("invalid command '%s'" % command)
distutils.errors.DistutilsModuleError: invalid command 'build_ext'
```
copying over the build_ext file, or the rest of the .py files in the command subfolder, does not appear to fix this issue for me, so not sure what else is going on here.
> this function was added in #4187 , CC @lazka :)

I think it's a fair assumption that we require a complete Python install, so the error looks correct to me. We could extend the msi installer to include distutils.
+1 for including a more complete python.

Documenting here my current workaround for anyone else who hits this issue...

What I'm doing for now is using "required: false" when searching for python 3, and then for the custom_target commands to call python scripts using it, I put in ["meson" "runpython"] directly in them if python3.found() is false. Works well enough for me now at the cost of extra "if" statements.
The problem here is that both cx_freeze and pyinstaller try to be clever and only bundle the minimal set of things you actually include (even for your own code which seems a bit stupid). This makes sense for most programs but not for us since we want _everything_. Both of them require extra command line arguments to add modules they don't detect automatically.

And even if you do that for all the missing things, you overflow Window's maximum command line length.

Resolving patch:
diff --git a/mesonbuild/mesonmain.py b/mesonbuild/mesonmain.py
--- a/mesonbuild/mesonmain.py
+++ b/mesonbuild/mesonmain.py
@@ -81,14 +81,18 @@ def add_command(self, name, add_arguments_func, run_func, help, aliases=[]):
             self.commands[i] = p
 
     def add_runpython_arguments(self, parser):
+        parser.add_argument('-c', action='store_true', dest='eval_arg', default=False)
         parser.add_argument('script_file')
         parser.add_argument('script_args', nargs=argparse.REMAINDER)
 
     def run_runpython_command(self, options):
         import runpy
-        sys.argv[1:] = options.script_args
-        sys.path.insert(0, os.path.dirname(options.script_file))
-        runpy.run_path(options.script_file, run_name='__main__')
+        if options.eval_arg:
+            exec(options.script_file)
+        else:
+            sys.argv[1:] = options.script_args
+            sys.path.insert(0, os.path.dirname(options.script_file))
+            runpy.run_path(options.script_file, run_name='__main__')
         return 0
 
     def add_help_arguments(self, parser):
diff --git a/mesonbuild/modules/python.py b/mesonbuild/modules/python.py
--- a/mesonbuild/modules/python.py
+++ b/mesonbuild/modules/python.py
@@ -42,9 +42,8 @@
 
 
 def run_command(python, command):
-    _, stdout, _ = mesonlib.Popen_safe(python.get_command() + [
-        '-c',
-        command])
+    cmd = python.get_command() + ['-c', command]
+    _, stdout, _ = mesonlib.Popen_safe(cmd)
 
     return stdout.strip()
 
@@ -265,8 +264,7 @@ def get_pkgconfig_variable(self, variable_name, kwargs):
             return super().get_pkgconfig_variable(variable_name, kwargs)
 
 
-INTROSPECT_COMMAND = '''
-import sysconfig
+INTROSPECT_COMMAND = '''import sysconfig
 import json
 import sys
 
diff --git a/msi/createmsi.py b/msi/createmsi.py
--- a/msi/createmsi.py
+++ b/msi/createmsi.py
@@ -84,6 +84,28 @@ def get_all_modules_from_dir(self, dirname):
         modules = ['mesonbuild.' + modname + '.' + x for x in modules if not x.startswith('_')]
         return modules
 
+    def get_more_modules(self):
+        # Python packagers want to minimal and only copy the things that
+        # they can see that are used. They are blind to many things.
+        return ['distutils.archive_util',
+                'distutils.cmd',
+                'distutils.config',
+                'distutils.core',
+                'distutils.debug',
+                'distutils.dep_util',
+                'distutils.dir_util',
+                'distutils.dist',
+                'distutils.errors',
+                'distutils.extension',
+                'distutils.fancy_getopt',
+                'distutils.file_util',
+                'distutils.spawn',
+                'distutils.util',
+                'distutils.version',
+                'distutils.command.build_ext',
+                'distutils.command.build',
+                ]
+
     def build_dist(self):
         for sdir in self.staging_dirs:
             if os.path.exists(sdir):
@@ -91,7 +113,7 @@ def build_dist(self):
         main_stage, ninja_stage = self.staging_dirs
         modules = self.get_all_modules_from_dir('mesonbuild/modules')
         modules += self.get_all_modules_from_dir('mesonbuild/scripts')
-        modules += ['distutils.version']
+        modules += self.get_more_modules()
         modulestr = ','.join(modules)
         python = shutil.which('python')
         cxfreeze = os.path.join(os.path.dirname(python), "Scripts", "cxfreeze")


Test patch: