Repository: DataBiosphere/toil

Issue:
Fixes to prevent hanging on zombie children have made test for children to all be dead fail intermittently
As @mr-c noted, we have a flaky test here now:

https://ucsc-ci.com/databiosphere/toil/-/jobs/8639#L5368

```
________________ SingleMachineBatchSystemTest.testProcessEscape ________________
Traceback (most recent call last):
  File "/builds/databiosphere/toil/src/toil/test/batchSystems/batchSystemTest.py", line 665, in testProcessEscape
    fcntl.lockf(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB)
BlockingIOError: [Errno 11] Resource temporarily unavailable
```

_Originally posted by @mr-c in https://github.com/DataBiosphere/toil/issues/4022#issuecomment-1024890016_

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1137)
┆friendlyId: TOIL-1137



Discussion:
@w-gao This is almost certainly related to #3974 and the fix for #3965. We now don't hang around until these child processes are dead enough to unlock their files before returning from the batch system shutdown, so the files appear locked still.

Unless we start looping through process group members somehow and making sure they are all either gone or waiting to be reaped, I'm not sure how we fix this 100%. Maybe we want to bring back that kill loop but with a timeout this time? Or maybe we do really want to enumerate processes in the group?
Ah, [testProcessEscape()](https://github.com/DataBiosphere/toil/blob/master/src/toil/test/batchSystems/batchSystemTest.py#L570) expects children to all be dead after batch system shutdown, but #3974 removes that guarantee.

I wonder if we can just add a slight delay after we send each SIGKILL [here](https://github.com/DataBiosphere/toil/blob/master/src/toil/batchSystems/singleMachine.py#L324), since normal processes should exit fairly quickly with SIGKILL? To be safe, we can also bring back the kill loop but with a timeout.

> Or maybe we do really want to enumerate processes in the group?

Would a sigkill signal to the process group reach all child processes? If not, we could enumerate the processes in the group and send a kill signal individually, but I think we still need to wait a while for the signal to propagate.

Resolving patch:
diff --git a/src/toil/batchSystems/singleMachine.py b/src/toil/batchSystems/singleMachine.py
--- a/src/toil/batchSystems/singleMachine.py
+++ b/src/toil/batchSystems/singleMachine.py
@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import datetime
 import logging
 import math
 import os
@@ -302,10 +303,14 @@ def _stop_now(self, popens: Sequence[subprocess.Popen]) -> List[int]:
 
         return pgids
 
-    def _stop_and_wait(self, popens: Sequence[subprocess.Popen]) -> None:
+    def _stop_and_wait(self, popens: Sequence[subprocess.Popen], timeout: int = 5) -> None:
         """
         Stop the given child processes and all their children. Blocks until the
-        processes are gone.
+        processes are gone or timeout is passed.
+
+        :param popens: The processes to stop and wait on.
+        :param timeout: The number of seconds to wait for all process groups to
+                        be gone.
         """
 
         pgids = self._stop_now(popens)
@@ -313,27 +318,54 @@ def _stop_and_wait(self, popens: Sequence[subprocess.Popen]) -> None:
         for popen in popens:
             # Wait on all the children
             popen.wait()
+            logger.debug('Process %s known to batch system %s is stopped; it returned %s',
+                         popen.pid, id(self), popen.returncode)
 
-            logger.debug('Process %s known to batch system %s is stopped; it returned %s', popen.pid, id(self), popen.returncode)
+        # Make sure all child processes have received their kill signal
+        self._wait_for_death(pgids, timeout)
 
-        for pgid in pgids:
-            try:
-                # Send a kill to the group again, to see if anything in it
-                # is still alive. Our first kill might not have been
-                # delivered yet.
-                os.killpg(pgid, signal.SIGKILL)
-                # If that worked it is still alive, let user know that we may leave
-                # behind dead but unreaped processes.
-                logger.warning('Sent redundant shutdown kill to surviving process group %s known to batch system %s', pgid, id(self))
-                logger.warning('Make sure your jobs are cleaning up child processes appropriately to avoid zombie '
-                               'processes possibly being left behind.')
+    def _wait_for_death(self, pgids: List[int], timeout: int = 5):
+        """
+        Wait for the process groups to be killed. Blocks until the processes
+        are gone or timeout is passed.
 
-            except ProcessLookupError:
-                # The group is actually gone now.
-                pass
-            except PermissionError:
-                # The group is not only gone but reused
-                pass
+        :param pgids: The list of process group ids.
+        :param timeout: The number of seconds to wait for all process groups to
+                        be gone.
+        """
+        # TODO: this opens a PGID reuse risk; someone else might've reaped the
+        #  process and its PGID may have been re-used.
+
+        start = datetime.datetime.now()
+        while len(pgids) > 0 and (datetime.datetime.now() - start).total_seconds() < timeout:
+            new_pgids: List[int] = []
+            for pgid in pgids:
+                try:
+                    # Send a kill to the group again, to see if anything in it
+                    # is still alive. Our first kill might not have been
+                    # delivered yet.
+                    os.killpg(pgid, signal.SIGKILL)
+
+                    # If we reach here, something in the process group still
+                    # exists.
+                    new_pgids.append(pgid)
+                except ProcessLookupError:
+                    # The group is actually gone now.
+                    pass
+                except PermissionError:
+                    # The group is not only gone but reused
+                    pass
+
+            pgids = new_pgids
+            if len(pgids) > 0:
+                time.sleep(0.1)
+
+        if len(pgids) > 0:
+            # If any processes are still alive, let user know that we may leave
+            # behind dead but unreaped processes.
+            logger.warning('Processes were not reaped in groups: %s.', str(pgids))
+            logger.warning('Make sure your jobs are cleaning up child processes appropriately to avoid zombie '
+                           'processes possibly being left behind.')
 
     def _pollForDoneChildrenIn(self, pid_to_popen):
         """
@@ -550,7 +582,7 @@ def _handleChild(self, pid: int) -> None:
             # Before we reap it (if possible), kill its PID as a PGID to make sure
             # it isn't leaving children behind.
             # TODO: This is a PGID re-use risk on Mac because the process is
-            # reaped already and the PGID may have been reused.
+            #  reaped already and the PGID may have been reused.
             try:
                 os.killpg(pid, signal.SIGKILL)
             except ProcessLookupError:
@@ -572,25 +604,7 @@ def _handleChild(self, pid: int) -> None:
 
         # Last attempt to make sure all processes in the group have received
         # their kill signals.
-        # TODO: this opens a PGID reuse risk; we reaped the process and its
-        #  PGID may have been re-used.
-        try:
-            # Send a kill to the group again, to see if anything in it
-            # is still alive. Our first kill might not have been
-            # delivered yet.
-            os.killpg(pid, signal.SIGKILL)
-            # If that worked it is still alive, let user know that we may leave
-            # behind dead but unreaped processes.
-            logger.warning('Sent redundant job completion kill to surviving process group %s known to batch system %s', pid, id(self))
-            logger.warning('Make sure your jobs are cleaning up child processes appropriately to avoid zombie '
-                           'processes possibly being left behind.')
-
-        except ProcessLookupError:
-            # It is dead already
-            pass
-        except PermissionError:
-            # It isn't ours actually. Ours is dead.
-            pass
+        self._wait_for_death([pid])
 
         # Free up the job's resources.
         self.coreFractions.release(coreFractions)


Test patch: