Repository: python-trio/trio

Issue:
use pidfd on Linux when available
Recent versions of the Linux kernel finally added a much more reasonable way to interact with subprocesses called "pidfd". The main advantages are:

- Using an fd as a handle for interacting with a process avoids a bunch of weird edge cases that pids have, like pids being recycled out from under you so you accidentally kill the wrong process (this isn't an issue for the specific way trio interacts with subprocesses, because we only work with child processes and the relationship between a parent and child is special, but in more general cases it's an issue)
- You can wait for child processes using `epoll`!!!!
- There may be more fancy features coming that are built on top of pidfd, like the ability to automatically kill a child when the last pidfd reference to it is closed.

Probably the epoll integration is the first thing we care about, since it provides an alternative to the rube goldberg thread-based construction we use now. Of course we'll still have to maintain that indefinitely because of RHEL and friends, but it'll be nice to avoid it when running on a modern system. That also means this isn't like, a huge urgent priority, because the user-visible impact is basically nil. But we'll want it eventually, so might as well have an issue to collect information until we're ready.

Initial pidfd support landed in kernel 5.1; the epoll integration landed in 5.3. And Ubuntu 19.10 (out next week) will have a 5.3 kernel, so pidfd is starting to actually filter out to users. I believe that in 19.10 there still aren't any glibc wrappers for the new syscalls, so if you want to access them you have to [call `syscall` directly](https://manpages.courier-mta.org/htmlman2/syscall.2.html). We could do that with ctypes or something though – it's janky but not difficult.

The biggest blocker here is that we will definitely need to test the new code, and AFAIK none of the CI services we have access to will ship a pidfd-capable kernel for... a while. They all seem to stick to LTS releases, and sometimes they're slow to upgrade even then... Azure is still on Ubuntu 16.04, though Travis and Github Actions both have 18.04. So probably we won't be able to actually *test* pidfd support until sometime after Ubuntu 20.04 LTS is released in April 2020. (Unfortunately since this is a kernel feature, using docker to pull down a newer release doesn't help, and I'm not aware of any hosted CI services that let you bring-your-own-kernel.)


Discussion:
Python 3.9 will have a `os.pidfd_open` wrapper built-in: https://github.com/python/cpython/pull/17063

It's pretty simple so we might end up ignoring this and using cffi instead, in order to support older Python versions, but wanted to mention for reference.
Note you can obtain a pidfd by doing `open('/proc/PID', O_DIRECTORY)`. `pidfd_open` is mainly useful if you need to be robust against environments where `/proc` isn't mounted. Of course, to do anything useful with the pidfd you do need to make new syscalls, so that factoid may not save you writing your own ffi.
Oh huh, good point :-). And we need fallback code anyway for old systems, so we could use that for environments where there's no `/proc`.

> to do anything useful with the pidfd you do need to make new syscalls, so that factoid may not save you writing your own ffi.

The number one motivation for using this would be to replace our current hack of spawning a thread for every child process just to sit and wait for notification that it's finished. To do that with pidfd, we don't need any other syscalls, just `await trio.hazmat.wait_readable(pidfd)`.

What I'm not sure of is how we can tell whether the fd we pulled out of `/proc` is actually a new-style pidfd that we can pass to epoll, or a regular old-style directory fd. Do you (or anyone) know of any behavioral differences we could use to sniff that out? I guess we could try to guess by parsing `os.uname().release` but I'd *really* prefer to avoid that if possible...
On November 7, 2019 9:32:20 AM GMT+01:00, "Nathaniel J. Smith" <notifications@github.com> wrote:
>Oh huh, good point :-). And we need fallback code anyway for old
>systems, so we could use that for environments where there's no
>`/proc`.
>
>> to do anything useful with the pidfd you do need to make new
>syscalls, so that factoid may not save you writing your own ffi.
>
>The number one motivation for using this would be to replace our
>current hack of spawning a thread for every child process just to sit
>and wait for notification that it's finished. To do that with pidfd, we
>don't need any other syscalls, just `await
>trio.hazmat.wait_readable(pidfd)`.
>
>What I'm not sure of is how we can tell whether the fd we pulled out of
>`/proc` is actually a new-style pidfd that we can pass to epoll, or a
>regular old-style directory fd. Do you (or anyone) know of any
>behavioral differences we could use to sniff that out? I guess we could
>try to guess by parsing `os.uname().release` but I'd *really* prefer to
>avoid that if possible...

You can only get pollable pidfds from pidfd_open(). To tell them apart a statfs() looking at the filesystem type should be enough. Proper pidfd_open() pidfds are anonymous inodes.


Huh. You never know who's reading these github threads :-). @brauner Thanks for the info, and for implementing pidfd! It's a massive, massive improvement.
I believe this bit of gibberish works to get a compatible version of `pidfd_open` across past and future Python versions:

```python
import sys
if sys.platform == "linux":
    try:
        # Python 3.9+
        from os import pidfd_open
    except ImportError:
        import os
        import ctypes
        _cdll_for_pidfd_open = ctypes.CDLL(None, use_errno=True)
        _cdll_for_pidfd_open.syscall.restype = ctypes.c_long
        _cdll_for_pidfd_open.syscall.argtypes = [
            ctypes.c_long,  # syscall number
            ctypes.c_int,   # fd
            ctypes.c_uint,  # flags
        ]
        __NR_pidfd_open = 434
        def pidfd_open(fd, flags):
            result = _cdll_for_pidfd_open.syscall(
                __NR_pidfd_open, fd, flags
            )
            if result < 0:
                err = ctypes.get_errno()
                raise OSError(err, os.strerror(err))
            return result
```

(Well, it doesn't try to support Python 2; the `sys.platform` check is wrong there, and maybe other things too. In case anyone cares about that. Fortunately Trio doesn't :-).)
Except I'm not sure if those `argtypes` are correct. Maybe arguments to the libc `syscall` wrapper are always `long`, even when the underlying syscall takes `int`? The interaction of varargs and ABIs is super obscure so I'm not really sure, but the musl `syscall()` implementation does pull out the arguments using `va_arg(..., long)`.
> Huh. You never know who's reading these github threads :-). @brauner Thanks for the info, and for implementing pidfd! It's a massive, massive improvement.

Thank you for the kind words. :)
There is more to come. :)

Resolving patch:
diff --git a/trio/_subprocess.py b/trio/_subprocess.py
--- a/trio/_subprocess.py
+++ b/trio/_subprocess.py
@@ -1,5 +1,6 @@
 import os
 import subprocess
+import sys
 from typing import Optional
 
 from ._abc import AsyncResource, SendStream, ReceiveStream
@@ -11,6 +12,39 @@
 )
 import trio
 
+# Linux-specific, but has complex lifetime management stuff so we hard-code it
+# here instead of hiding it behind the _subprocess_platform abstraction
+can_try_pidfd_open = True
+try:
+    from os import pidfd_open
+except ImportError:
+    if sys.platform == "linux":
+        import ctypes
+        _cdll_for_pidfd_open = ctypes.CDLL(None, use_errno=True)
+        _cdll_for_pidfd_open.syscall.restype = ctypes.c_long
+        # pid and flags are actually int-sized, but the syscall() function
+        # always takes longs. (Except on x32 where long is 32-bits and syscall
+        # takes 64-bit arguments. But in the unlikely case that anyone is
+        # using x32, this will still work, b/c we only need to pass in 32 bits
+        # of data, and the C ABI doesn't distinguish between passing 32-bit vs
+        # 64-bit integers; our 32-bit values will get loaded into 64-bit
+        # registers where syscall() will find them.)
+        _cdll_for_pidfd_open.syscall.argtypes = [
+            ctypes.c_long,  # syscall number
+            ctypes.c_long,  # pid
+            ctypes.c_long,  # flags
+        ]
+        __NR_pidfd_open = 434
+
+        def pidfd_open(fd, flags):
+            result = _cdll_for_pidfd_open.syscall(__NR_pidfd_open, fd, flags)
+            if result < 0:
+                err = ctypes.get_errno()
+                raise OSError(err, os.strerror(err))
+            return result
+    else:
+        can_try_pidfd_open = False
+
 
 class Process(AsyncResource):
     r"""A child process. Like :class:`subprocess.Popen`, but async.
@@ -151,6 +185,21 @@ def _init(
             if self.stderr is not None:
                 os.close(stderr)
 
+        self._pidfd = None
+        if can_try_pidfd_open:
+            try:
+                fd = pidfd_open(self._proc.pid, 0)
+            except OSError:
+                # Well, we tried, but it didn't work (probably because we're
+                # running on an older kernel, or in an older sandbox, that
+                # hasn't been updated to support pidfd_open). We'll fall back
+                # on waitid instead.
+                pass
+            else:
+                # It worked! Wrap the raw fd up in a Python file object to
+                # make sure it'll get closed.
+                self._pidfd = open(fd)
+
         if self.stdin is not None and self.stdout is not None:
             self.stdio = StapledStream(self.stdin, self.stdout)
 
@@ -205,19 +254,32 @@ async def aclose(self):
                 with trio.CancelScope(shield=True):
                     await self.wait()
 
+    def _close_pidfd(self):
+        if self._pidfd is not None:
+            self._pidfd.close()
+            self._pidfd = None
+
     async def wait(self):
         """Block until the process exits.
 
         Returns:
           The exit status of the process; see :attr:`returncode`.
         """
-        if self.poll() is None:
-            async with self._wait_lock:
-                if self.poll() is None:
+        async with self._wait_lock:
+            if self.poll() is None:
+                if self._pidfd is not None:
+                    await trio.hazmat.wait_readable(self._pidfd)
+                else:
                     await wait_child_exiting(self)
-                    self._proc.wait()
-        else:
-            await trio.hazmat.checkpoint()
+                # We have to use .wait() here, not .poll(), because on macOS
+                # (and maybe other systems, who knows), there's a race
+                # condition inside the kernel that creates a tiny window where
+                # kqueue reports that the process has exited, but
+                # waitpid(WNOHANG) can't yet reap it. So this .wait() may
+                # actually block for a tiny fraction of a second.
+                self._proc.wait()
+                self._close_pidfd()
+        assert self.returncode is not None
         return self.returncode
 
     def poll(self):
@@ -227,7 +289,10 @@ def poll(self):
           The exit status of the process, or ``None`` if it is still
           running; see :attr:`returncode`.
         """
-        return self._proc.poll()
+        result = self._proc.poll()
+        if result is not None:
+            self._close_pidfd()
+        return result
 
     def send_signal(self, sig):
         """Send signal ``sig`` to the process.


Test patch: