Repository: pantsbuild/pants

Issue:
Interrupting an `(Interactive)Process` should attempt to kill the process gracefully
Currently, both `Process` and `InteractiveProcess` will send `KILL` when they are dropped (via `tokio`s facilities for `kill_on_drop`). But in both cases, kill doesn't give the underlying process time to do its own teardown, and this can lead to resource leaks. `pants` itself has [concerns](https://github.com/pantsbuild/pants/issues/12996) with being killed without getting a TERM first, and end user processes like `uvicorn` would like to be able to run their user shutdown hooks before dying.

To do this, we will likely need to implement a workaround for the lack of `AsyncDrop`: by essentially adding an optional `TERM` and synchronous wait in a `Drop` guard wrapped around relevant processes, or optionally gracefully interrupting graph nodes.


Discussion:
This lightly relates to #12603 as well.
#13678 added the `ManagedChild` wrapper around interactive processes (i.e., those spawned by `run`, `test --debug`, etc):
https://github.com/pantsbuild/pants/blob/6b583e2efc4415c8dab2798f0223423c9fce1954/src/rust/engine/process_execution/src/children.rs#L9-L61

On `Drop`, that struct currently SIGKILLs a process group that the managed process is placed in on startup. I think that to fix this ticket, the `ManagedChild` should optionally use a more graceful signal to kill the process.

There are a few factors here: the codepath which consumes `ManagedChild` does attempt to explicitly kill the child if the `Session` (i.e. the run of pants) is cancelled (with Ctrl+C): https://github.com/pantsbuild/pants/blob/6b583e2efc4415c8dab2798f0223423c9fce1954/src/rust/engine/src/intrinsics.rs#L704-L715 ...but it's possible that that will race with the client going away (because the main loop, well above that code, will also exit if the `Session` is cancelled), which would cause that particular bit of code to not run, and for `Drop` to run instead. So, two potential approaches would be:
1. Have the caller of `ManagedChild` `spawn` waiting for the process and `Session.is_cancelled`, such that even if the client has gone away, the foreground code can go through its teardown process. This would basically be equivalent to saying that `ManagedChild` shouldn't be treated as "async drop/cancellation safe".
2. Synchronously wait in `impl Drop for ManagedChild` (with a tiny `thread::sleep` loop and a deadline), and acknowledge that sometimes we will block the foreground code while it attempts to disappear after cancellation. This is safe (for this caller at least) because we only run one interactive process at a time, and since the sleep is bounded, we can't deadlock the executor.

Either should work!

Resolving patch:
diff --git a/testprojects/src/python/coordinated_runs/waiter.py b/testprojects/src/python/coordinated_runs/waiter.py
--- a/testprojects/src/python/coordinated_runs/waiter.py
+++ b/testprojects/src/python/coordinated_runs/waiter.py
@@ -7,11 +7,6 @@
 import time
 from multiprocessing import Process
 
-waiting_for_file = sys.argv[1]
-pid_file = sys.argv[2]
-child_pid_file = sys.argv[3]
-attempts = 60
-
 
 def run_child():
     while True:
@@ -19,21 +14,46 @@ def run_child():
         time.sleep(1)
 
 
-child = Process(target=run_child, daemon=True)
-child.start()
+def main():
+    waiting_for_file = sys.argv[1]
+    pid_file = sys.argv[2]
+    child_pid_file = sys.argv[3]
+    cleanup_wait_time = int(sys.argv[4])
+    attempts = 60
 
-with open(child_pid_file, "w") as pf:
-    pf.write(str(child.pid))
+    child = Process(target=run_child, daemon=True)
+    child.start()
 
-with open(pid_file, "w") as pf:
-    pf.write(str(os.getpid()))
+    with open(child_pid_file, "w") as pf:
+        pf.write(str(child.pid))
 
-try:
-    while not os.path.isfile(waiting_for_file):
-        if attempts <= 0:
-            raise Exception("File was never written.")
-        attempts -= 1
-        sys.stderr.write("Waiting for file {}\n".format(waiting_for_file))
-        time.sleep(1)
-finally:
-    child.terminate()
+    with open(pid_file, "w") as pf:
+        pf.write(str(os.getpid()))
+
+    try:
+        while not os.path.isfile(waiting_for_file):
+            if attempts <= 0:
+                raise Exception("File was never written.")
+            attempts -= 1
+            sys.stderr.write("Waiting for file {}\n".format(waiting_for_file))
+            sys.stderr.flush()
+            time.sleep(1)
+
+    except KeyboardInterrupt:
+        sys.stderr.write("keyboard int received\n")
+        sys.stderr.flush()
+
+    finally:
+        sys.stderr.write("waiter cleaning up\n")
+        sys.stderr.flush()
+
+        child.terminate()
+        if cleanup_wait_time > 0:
+            time.sleep(cleanup_wait_time)
+
+        sys.stderr.write("waiter cleanup complete\n")
+        sys.stderr.flush()
+
+
+if __name__ == "__main__":
+    main()


Test patch:
diff --git a/tests/python/pants_test/pantsd/pantsd_integration_test.py b/tests/python/pants_test/pantsd/pantsd_integration_test.py
--- a/tests/python/pants_test/pantsd/pantsd_integration_test.py
+++ b/tests/python/pants_test/pantsd/pantsd_integration_test.py
@@ -499,15 +499,23 @@ def test_pantsd_invalidation_stale_sources(self):
             rm_rf(test_path)
 
     def _assert_pantsd_keyboardinterrupt_signal(
-        self, signum: int, regexps: list[str] | None = None
+        self,
+        signum: int,
+        regexps: list[str] | None = None,
+        not_regexps: list[str] | None = None,
+        cleanup_wait_time: int = 0,
     ):
         """Send a signal to the thin pailgun client and observe the error messaging.
 
         :param signum: The signal to send.
         :param regexps: Assert that all of these regexps match somewhere in stderr.
+        :param not_regexps: Assert that all of these regexps do not match somewhere in stderr.
+        :param cleanup_wait_time: passed throught to waiter, dictated how long simulated cleanup will take
         """
         with self.pantsd_test_context() as (workdir, config, checker):
-            client_handle, waiter_pid, child_pid, _ = launch_waiter(workdir=workdir, config=config)
+            client_handle, waiter_pid, child_pid, _ = launch_waiter(
+                workdir=workdir, config=config, cleanup_wait_time=cleanup_wait_time
+            )
             client_pid = client_handle.process.pid
             waiter_process = psutil.Process(waiter_pid)
             child_process = psutil.Process(waiter_pid)
@@ -516,6 +524,9 @@ def _assert_pantsd_keyboardinterrupt_signal(
             assert child_process.is_running()
             checker.assert_started()
 
+            # give time to enter the try/finally block in the child process
+            time.sleep(5)
+
             # This should kill the client, which will cancel the run on the server, which will
             # kill the waiting process and its child.
             os.kill(client_pid, signum)
@@ -525,6 +536,9 @@ def _assert_pantsd_keyboardinterrupt_signal(
             for regexp in regexps or []:
                 self.assertRegex(client_run.stderr, regexp)
 
+            for regexp in not_regexps or []:
+                self.assertNotRegex(client_run.stderr, regexp)
+
             # pantsd should still be running, but the waiter process and child should have been
             # killed.
             time.sleep(5)
@@ -532,10 +546,33 @@ def _assert_pantsd_keyboardinterrupt_signal(
             assert not child_process.is_running()
             checker.assert_running()
 
-    def test_pantsd_sigint(self):
+    def test_pantsd_graceful_shutdown(self):
+        """Test that SIGINT is propgated to child processes and they are given time to shutdown."""
+        self._assert_pantsd_keyboardinterrupt_signal(
+            signal.SIGINT,
+            regexps=[
+                "Interrupted by user.",
+                "keyboard int received",
+                "waiter cleaning up",
+                "waiter cleanup complete",
+            ],
+            cleanup_wait_time=0,
+        )
+
+    def test_pantsd_graceful_shutdown_deadline(self):
+        """Test that a child process that does not respond to SIGINT within 5 seconds, is forcibly
+        cleaned up with a SIGKILL."""
         self._assert_pantsd_keyboardinterrupt_signal(
             signal.SIGINT,
-            regexps=["Interrupted by user."],
+            regexps=[
+                "Interrupted by user.",
+                "keyboard int received",
+                "waiter cleaning up",
+            ],
+            not_regexps=[
+                "waiter cleanup complete",
+            ],
+            cleanup_wait_time=6,
         )
 
     def test_sigint_kills_request_waiting_for_lock(self):
diff --git a/tests/python/pants_test/pantsd/pantsd_integration_test_base.py b/tests/python/pants_test/pantsd/pantsd_integration_test_base.py
--- a/tests/python/pants_test/pantsd/pantsd_integration_test_base.py
+++ b/tests/python/pants_test/pantsd/pantsd_integration_test_base.py
@@ -56,7 +56,7 @@ def attempts(
 
 
 def launch_waiter(
-    *, workdir: str, config: Mapping | None = None
+    *, workdir: str, config: Mapping | None = None, cleanup_wait_time: int = 0
 ) -> tuple[PantsJoinHandle, int, int, str]:
     """Launch a process that will wait forever for a file to be created.
 
@@ -74,6 +74,7 @@ def launch_waiter(
         file_to_make,
         waiter_pid_file,
         child_pid_file,
+        str(cleanup_wait_time),
     ]
     client_handle = run_pants_with_workdir_without_waiting(argv, workdir=workdir, config=config)
     waiter_pid = -1