Repository: mesonbuild/meson

Issue:
Python OSError when downloading patch in wrap-file on PyPy-3.8
Meson fails to download the patch file for the official pybind11 wrap from WrapDB.

```
  Run-time dependency pybind11 found: NO (tried pkgconfig and cmake)
  Looking for a fallback subproject for the dependency pybind11
  Downloading pybind11 source from https://github.com/pybind/pybind11/archive/refs/tags/v2.10.0.tar.gz
  Download size: 716362
  Downloading: ..........
  Downloading pybind11 patch from https://wrapdb.mesonbuild.com/v2/pybind11_2.10.0-1/get_patch
  Download size: 1158
  Downloading: ..........

  ERROR: Unhandled python OSError. This is probably not a Meson bug, but an issue with your build environment.
  Traceback (most recent call last):
    File "/tmp/pip-build-env-lldhk6f8/overlay/lib/pypy3.8/site-packages/mesonbuild/wrap/wrap.py", line 780, in apply_patch
      shutil.unpack_archive(path, self.subdir_root)
    File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/shutil.py", line 1237, in unpack_archive
      func(filename, extract_dir, **kwargs)
    File "/opt/hostedtoolcache/PyPy/3.8.15/x64/lib/pypy3.8/shutil.py", line 1136, in _unpack_zipfile
      raise ReadError("%s is not a zip file" % filename)
  shutil.ReadError: /home/runner/work/pyhepmc/pyhepmc/subprojects/packagecache/pybind11_2.10.0-1_patch.zip is not a zip file
```

This happens on ubuntu-latest image from Github Actions, pypy-3.8.

I encounter this problem when trying to install the Python package that I am migrating from cmake build to meson-python at the moment.

https://github.com/scikit-hep/pyhepmc/actions/runs/3845655154/jobs/6550052598

The installation works on macos-latest with Python-3.8, so I assume it is a problem with running meson on pypy-3.8.

This message seems to be wrong in this case.
```
  ERROR: Unhandled python OSError. This is probably not a Meson bug, but an issue with your build environment.
```


Discussion:
Interesting. What is the contents of /home/runner/work/pyhepmc/pyhepmc/subprojects/packagecache/pybind11_2.10.0-1_patch.zip and why does python's zipfile module claim that it's not a zipfile? It should be checksummed before attempting to unpack it, and raise `WrapException(f'Incorrect hash for {what}:\n {expected} expected\n {dhash} actual.')` instead. (This is caught and formatted as a meson "ERROR: {string}", and does not print tracebacks or claims about unhandled python tracebacks.)

Since it works with python-3.8 but not pypy-3.8, that would seem to imply that CPython and PyPy have different ideas about what constitutes a zipfile??? I would not expect that at all.
I haven't looked into the code, but could this be related to the difference in life-times of objects in CPython and PyPy?

In CPython, you can do
```
open("foo", "w").write("stuff")
```
This will close the file "foo" immediately, since CPython uses refcounting and only uses a garbage collector for cycles.

PyPy does not use refcounting, so the file may still be open (and incompletely written) after that line.
Hmm, could help, let's find out. https://github.com/eli-schwartz/meson/tree/pypy-wrap (commit adecd32a4d868453d92a3a528d8a8e3a72a2ae8d)

Our implementation actually returns a tuple of `hashlib.sha256().hexdigest(), NamedTemporaryFile().name` which we... do not actually close, indeed. The actual comparison to see if the hash is correct uses that hashlib buffer, which we update in blocks at the same time as we write blocks to the temporary file, which means that it's going to be correct even in cases where the file write itself is catastrophically broken or else not fully written, of which the latter case actually counts as a Meson bug.

Once we are satisfied that that all worked out and downloaded correctly, we os.rename() the still-open file which may or may not have gotten closed at random, but which we only have a string attribute for by this point in time. This is indeed quite semantically incorrect.

Can you try installing Meson from my forked branch and see if that works?
@eli-schwartz Thanks for looking into this, this analysis makes sense to me. I cannot easily check the patched version on my end, because I already changed my code to work around that issue (I don't use the pybind11 wrap anymore, but install pybind11 with Python, like scipy), and I cannot run pypy locally on my M1 Mac. In any case, I think it would be good to add a unit test for this on your end, which should reveal the bug.
In theory, we are already testing this as part of the unittests. It's just that the unittests all run in CI with CPython (because that's what the distros we test on use -- we don't use actions/setup-python), and CPython doesn't tickle this bad practice.


I see that you run most tests with Github actions. You can generate a job that uses `actions/setup-python@v4` to set pypy-3.8 as the default python interpreter which will be used by meson.
```
    - uses: actions/setup-python@v4
      with:
        python-version: "pypy-3.8"
```
That's what I do when I test whether my extension also works under PyPy in addition to CPython.
Well, the thing is that all our Linux tests use custom docker containers with a large number of things preinstalled, many running on distros that Actions doesn't natively provide (and therefore setup-python can't work on).

And our macOS tests hard-wipe the setup-python versions of python to stop them from interfering with homebrew, because `brew install XXX` is currently broken across the entirety of github.com if XXX has some version of python anywhere in its dependency tree, and tries to install or upgrade it. Also since we test homebrew python, not the Actions one. :D

Our Windows tests are split between cygwin, msys2, and native Windows, and the native Windows ones only run in Azure pipelines.
If you cannot run the full test suite on PyPy, then you could test a subset on PyPy. I guess you need the docker containers to get all kinds of compilers, but this is a python/meson issue that is independent of compilers. You can use whatever compilers are available in a vanilla ubuntu-X image on Github for this specific case. I think there is no way around that, frankly, because people will run into these issues. I just happened to be the first one.
I verified this fix works: https://github.com/eli-schwartz/pyhepmc/actions/runs/3881149796/jobs/6619846035

Double checked locally after installing pypy3, and e.g. `LinuxlikeTests::test_wrap_with_file_url` fails before my patch and succeeds after it.

Trying to figure out what the best place to test this in CI is...

Resolving patch:
diff --git a/mesonbuild/wrap/wrap.py b/mesonbuild/wrap/wrap.py
--- a/mesonbuild/wrap/wrap.py
+++ b/mesonbuild/wrap/wrap.py
@@ -676,7 +676,7 @@ def get_data(self, urlstring: str) -> T.Tuple[str, str]:
             except urllib.error.URLError as e:
                 mlog.log(str(e))
                 raise WrapException(f'could not get {urlstring} is the internet available?')
-        with contextlib.closing(resp) as resp:
+        with contextlib.closing(resp) as resp, tmpfile as tmpfile:
             try:
                 dlsize = int(resp.info()['Content-Length'])
             except TypeError:


Test patch:
diff --git a/run_project_tests.py b/run_project_tests.py
--- a/run_project_tests.py
+++ b/run_project_tests.py
@@ -1104,7 +1104,7 @@ def __init__(self, category: str, subdir: str, skip: bool = False, stdout_mandat
         TestCategory('swift', 'swift', backend not in (Backend.ninja, Backend.xcode) or not shutil.which('swiftc')),
         # CUDA tests on Windows: use Ninja backend:  python run_project_tests.py --only cuda --backend ninja
         TestCategory('cuda', 'cuda', backend not in (Backend.ninja, Backend.xcode) or not shutil.which('nvcc')),
-        TestCategory('python3', 'python3', backend is not Backend.ninja),
+        TestCategory('python3', 'python3', backend is not Backend.ninja or 'python3' not in sys.executable),
         TestCategory('python', 'python'),
         TestCategory('fpga', 'fpga', shutil.which('yosys') is None),
         TestCategory('frameworks', 'frameworks'),
@@ -1591,7 +1591,7 @@ def clear_transitive_files() -> None:
     clear_transitive_files()
 
     print('Meson build system', meson_version, 'Project Tests')
-    print('Using python', sys.version.split('\n')[0])
+    print('Using python', sys.version.split('\n')[0], f'({sys.executable!r})')
     if 'VSCMD_VER' in os.environ:
         print('VSCMD version', os.environ['VSCMD_VER'])
     setup_commands(options.backend)
diff --git a/run_tests.py b/run_tests.py
--- a/run_tests.py
+++ b/run_tests.py
@@ -343,6 +343,10 @@ def print_system_info():
     print('')
     print(flush=True)
 
+def subprocess_call(cmd, **kwargs):
+    print(f'$ {mesonlib.join_args(cmd)}')
+    return subprocess.call(cmd, **kwargs)
+
 def main():
     print_system_info()
     parser = argparse.ArgumentParser()
@@ -380,7 +384,7 @@ def main():
         cmd = mesonlib.python_command + ['run_meson_command_tests.py', '-v']
         if options.failfast:
             cmd += ['--failfast']
-        returncode += subprocess.call(cmd, env=env)
+        returncode += subprocess_call(cmd, env=env)
         if options.failfast and returncode != 0:
             return returncode
         if no_unittests:
@@ -393,11 +397,11 @@ def main():
             cmd = mesonlib.python_command + ['run_unittests.py', '--backend=' + backend.name, '-v']
             if options.failfast:
                 cmd += ['--failfast']
-            returncode += subprocess.call(cmd, env=env)
+            returncode += subprocess_call(cmd, env=env)
             if options.failfast and returncode != 0:
                 return returncode
         cmd = mesonlib.python_command + ['run_project_tests.py'] + sys.argv[1:]
-        returncode += subprocess.call(cmd, env=env)
+        returncode += subprocess_call(cmd, env=env)
     else:
         cross_test_args = mesonlib.python_command + ['run_cross_test.py']
         for cf in options.cross:
@@ -408,7 +412,7 @@ def main():
                 cmd += ['--failfast']
             if options.cross_only:
                 cmd += ['--cross-only']
-            returncode += subprocess.call(cmd, env=env)
+            returncode += subprocess_call(cmd, env=env)
             if options.failfast and returncode != 0:
                 return returncode
     return returncode
diff --git a/test cases/frameworks/11 gir subproject/test.json b/test cases/frameworks/11 gir subproject/test.json
--- a/test cases/frameworks/11 gir subproject/test.json	
+++ b/test cases/frameworks/11 gir subproject/test.json	
@@ -9,5 +9,5 @@
     {"type": "expr", "file": "usr/lib/?libgirlib.so"},
     {"type": "file", "platform": "cygwin", "file": "usr/lib/libgirsubproject.dll.a"}
   ],
-  "skip_on_jobname": ["azure", "cygwin", "macos", "msys2"]
+  "skip_on_jobname": ["azure", "cygwin", "macos", "msys2", "pypy"]
 }
diff --git a/test cases/frameworks/7 gnome/test.json b/test cases/frameworks/7 gnome/test.json
--- a/test cases/frameworks/7 gnome/test.json	
+++ b/test cases/frameworks/7 gnome/test.json	
@@ -36,5 +36,5 @@
     {"type": "file", "file": "usr/include/simple-resources.h"},
     {"type": "file", "file": "usr/include/generated-gdbus.h"}
   ],
-  "skip_on_jobname": ["azure", "cygwin", "macos", "msys2"]
+  "skip_on_jobname": ["azure", "cygwin", "macos", "msys2", "pypy"]
 }