Repository: mesonbuild/meson

Issue:
Meson crashes while running GLib gdbus-proxy-threads test
This is currently a blocker for adding FreeBSD CI to GLib. It is easily reproducible and it makes it almost impossible to complete the GLib test on FreeBSD.
```
$ ~/gnome/source/meson/meson.py test gdbus-proxy-threads
ninja: Entering directory `/tmp/ram/glib-build'
ninja: no work to do.
Traceback (most recent call last):
  File "/home/lantw44/gnome/source/meson/meson.py", line 29, in <module>
    sys.exit(mesonmain.main())
  File "/home/lantw44/gnome/source/meson/mesonbuild/mesonmain.py", line 379, in main
    return run(sys.argv[1:], launcher)
  File "/home/lantw44/gnome/source/meson/mesonbuild/mesonmain.py", line 278, in run
    return mtest.run(remaining_args)
  File "/home/lantw44/gnome/source/meson/mesonbuild/mtest.py", line 759, in run
    return th.run_special()
  File "/home/lantw44/gnome/source/meson/mesonbuild/mtest.py", line 692, in run_special
    self.run_tests(tests)
  File "/home/lantw44/gnome/source/meson/mesonbuild/mtest.py", line 666, in run_tests
    self.drain_futures(futures)
  File "/home/lantw44/gnome/source/meson/mesonbuild/mtest.py", line 682, in drain_futures
    self.process_test_result(result.result())
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/lantw44/gnome/source/meson/mesonbuild/mtest.py", line 262, in run
    return self._run_cmd(wrap + cmd + self.test.cmd_args + self.options.test_args)
  File "/home/lantw44/gnome/source/meson/mesonbuild/mtest.py", line 353, in _run_cmd
    (stdo, stde) = p.communicate()
  File "/usr/local/lib/python3.6/subprocess.py", line 843, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/usr/local/lib/python3.6/subprocess.py", line 1505, in _communicate
    selector.register(self.stdout, selectors.EVENT_READ)
  File "/usr/local/lib/python3.6/selectors.py", line 351, in register
    key = super().register(fileobj, events, data)
  File "/usr/local/lib/python3.6/selectors.py", line 237, in register
    key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data)
  File "/usr/local/lib/python3.6/selectors.py", line 224, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/usr/local/lib/python3.6/selectors.py", line 39, in _fileobj_to_fd
    "{!r}".format(fileobj)) from None
ValueError: Invalid file object: <_io.BufferedReader name=6>
```

It took 30 seconds for meson to crash, so it looked like a timeout issue. However, `gdbus-proxy-threads` test itself completed in less than 5 seconds. It exited and became a zombie process before meson crashed.
```
 └─┬= 93390 lantw44 python3 /home/lantw44/gnome/source/meson/meson.py test gdbus-proxy-threads (python3.6)
   └──= 93399 lantw44 <defunct>
```

Interestingly, meson doesn't crash when running the same test with `-v` flag.
```
$ ~/gnome/source/meson/meson.py test -v gdbus-proxy-threads
ninja: Entering directory `/tmp/ram/glib-build'
ninja: no work to do.
/gdbus/proxy/vs-threads: OK
1/1 glib:gio / gdbus-proxy-threads          OK       2.58 s 

OK:         1
FAIL:       0
SKIP:       0
TIMEOUT:    0
```

Running `meson test` with `-v` avoids the crash, but it still doesn't work with GitLab runner. It seems there is a leftover `dbus-daemon` process when running `meson test` with `-v` and GitLab runner waits forever for it. The test https://gitlab.gnome.org/GNOME/glib/-/jobs/70974 already runs for a week and GitLab runner still waits for it.

Related comments on GNOME GitLab:
- https://gitlab.gnome.org/Infrastructure/GitLab/issues/286#note_273597
- https://gitlab.gnome.org/Infrastructure/GitLab/issues/286#note_273789


Discussion:
Looks like we're not killing process groups correctly after each test. Not sure what's going on because it seems to be BSD-specific?
After spending time reading `mtest.py` and `subprocess.py`, I think I understand the problem better now. Although it cannot be reproduced on GNU/Linux, it is not a BSD-specific issue but a bug of Python subprocess `communicate` function: https://bugs.python.org/issue30154.

When `communicate` is called, Python blocks itself on a call to `poll` to wait for data written to stdout and stderr. It exits the `poll` loop when all monitored file descriptors are closed. After it exits the loop, it waits for the child process to exit and return. It works most of the time because file descriptors are closed on process exit, so Python can exit the loop when the child process exits. However, if the child process forks a grandchild process and the grandchild process never exits, both stdout and stderr will still be open after the child process exits. This causes `communicate` to block forever until the timeout is expired.

`gdbus-proxy-threads` is a test which forks at least two child processes. One of them is `dbus-daemon` and another one is a watcher which is responsible for killing `dbus-daemon` when the test is done. On GNU/Linux, the test works correctly, so `dbus-daemon` is terminated when the test exits. On FreeBSD, the watcher process crashes and doesn't kill the `dbus-daemon` process, causing stdout and stderr to be kept open after the main process exits. This triggers the problem described above and seems to cause the second call to `communicate` to fail.

The ideal fix of the problem will be fixing the `gdbus-proxy-threads` crash on FreeBSD. However, it is still nice if meson itself can handle the case and explain the problem to users, so users will be able to easily understand what the problem is.
The leftover process problem is now fixed in GLib: https://gitlab.gnome.org/GNOME/glib/merge_requests/251. I still hope the crash problem can be fixed in meson. I think having a misbehaving test crashes the build system isn't something users expect to happen.

Meson is still not reliable enough to run GLib tests on FreeBSD. Meson puts wrong RPATH on the generated executable, so the test still fails on the CI machine because it uses the system-installed GLib instead of the one built under the build directory: https://gitlab.gnome.org/GNOME/glib/-/jobs/86836.
> est still fails on the CI machine because it uses the system-installed GLib instead of the one built under the build directory

Are you sure this is not caused by them using the "other" style of rpath or runpath or whatever it was where system paths override entries in RPATH instead of the other way around?
> Are you sure this is not caused by them using the "other" style of rpath or runpath or whatever it was where system paths override entries in RPATH instead of the other way around?

There is no LD_LIBRARY_PATH set in the environment and LDFLAGS already includes `-Wl,--disable-new-dtags`. I think it is really a RPATH problem because the first item in RPATH is `$ORIGIN/../../../../../../../../../../../usr/local/lib`. You can see the output of `ldd` and `readelf` in this comment: https://gitlab.gnome.org/GNOME/glib/merge_requests/251#note_296739. The test was done with meson 0.47.1.

Resolving patch:
diff --git a/mesonbuild/mtest.py b/mesonbuild/mtest.py
--- a/mesonbuild/mtest.py
+++ b/mesonbuild/mtest.py
@@ -354,7 +354,21 @@ def preexec_fn():
                     # There's nothing we can do (maybe the process
                     # already died) so carry on.
                     pass
-            (stdo, stde) = p.communicate()
+            try:
+                (stdo, stde) = p.communicate(timeout=1)
+            except subprocess.TimeoutExpired:
+                # An earlier kill attempt has not worked for whatever reason.
+                # Try to kill it one last time with a direct call.
+                # If the process has spawned children, they will remain around.
+                p.kill()
+                try:
+                    (stdo, stde) = p.communicate(timeout=1)
+                except subprocess.TimeoutExpired:
+                    stdo = b'Test process could not be killed.'
+                    stde = b''
+            except ValueError:
+                stdo = b'Could not read output. Maybe the process has redirected its stdout/stderr?'
+                stde = b''
         endtime = time.time()
         duration = endtime - starttime
         stdo = decode(stdo)


Test patch: