Repository: Parsl/parsl

Issue:
workqueue atexit cleanup raises an error with Python 3.12 when actually cleaning up
**Describe the bug**

If Work Queue has no been cleaned up by the time this test process exits, for example due to some exception during development:
```
$ pytest parsl/tests/ --config parsl/tests/configs/workqueue_ex.py 
```

then this stack trace appears:

```
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl/executors/workqueue/executor.py", line 317, in atexit_cleanup
    self.shutdown()
  File "/home/benc/parsl/src/parsl/parsl/executors/workqueue/executor.py", line 733, in shutdown
    self.provider.cancel(kill_ids)
  File "/home/benc/parsl/src/parsl/parsl/providers/local/local.py", line 271, in cancel
    retcode, stdout, stderr = self.channel.execute_wait(cmd, self.cmd_timeout)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/benc/parsl/src/parsl/parsl/channels/local/local.py", line 58, in execute_wait
    proc = subprocess.Popen(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/lib/python3.12/subprocess.py", line 1883, in _execute_child
    self.pid = _fork_exec(
               ^^^^^^^^^^^
RuntimeError: preexec_fn not supported at interpreter shutdown
```

Parsl removed atexit handling because of this sort of thing in PR #3165; but PR #3269 added this back in for Work Queue and Task Vine with the claim that "These handlers never spawn new threads or processes" - which looks like it is incorrect.

I think there are more circumstances where this would appear beyond running Pytest against broken local changes, but I have not dug into creating a reproduction case.


Discussion:
cc @tphung3 
It looks like the providers fork some cleanup processes when cancelling blocks. Working on it..
More than half of the providers use `self.execute_wait` to remove jobs, which spawns a new process running a shell cmd. Since it seems unavoidable, `atexit` handlers should be removed.
On a related note, does parsl dfk work with the `with` context now? I see `__enter__` and `__exit__` methods for the dfk @benclifford.
@tphung3 yeah, `with` context management was introduced in #3260 basically as an attempt to salvage this shutdown behaviour, but requiring a little more user engagement. Here's an example from the test suite:

```
    with parsl.load(config=config) as dfk:
        assert isinstance(dfk, DataFlowKernel)

        bash_future = foo(1, stdout=tmpd_cwd / 'foo.stdout')
        assert bash_future.result() == 0

        with open(tmpd_cwd / 'foo.stdout', 'r') as f:
            assert f.read() == "2\n"
```

On the positive side, `__exit__` gets more exception information than `atexit` handlers do, and I hope that can lead to some more interesting optional "wait for all tasks unless there's an exception, in which case don't wait for all tasks" shutdown behaviour, to address a common usability problem.

Resolving patch:
diff --git a/parsl/executors/workqueue/executor.py b/parsl/executors/workqueue/executor.py
--- a/parsl/executors/workqueue/executor.py
+++ b/parsl/executors/workqueue/executor.py
@@ -3,7 +3,6 @@
 high-throughput system for delegating Parsl tasks to thousands of remote machines
 """
 
-import atexit
 import threading
 import multiprocessing
 import logging
@@ -298,24 +297,6 @@ def __init__(self,
         if self.init_command != "":
             self.launch_cmd = self.init_command + "; " + self.launch_cmd
 
-        # register atexit handler to cleanup when Python shuts down
-        atexit.register(self.atexit_cleanup)
-
-        # Attribute indicating whether this executor was started to shut it down properly.
-        # This safeguards cases where an object of this executor is created but
-        # the executor never starts, so it shouldn't be shutdowned.
-        self.is_started = False
-
-        # Attribute indicating whether this executor was shutdown before.
-        # This safeguards cases where this object is automatically shut down (e.g.,
-        # via atexit) and the user also explicitly calls shut down. While this is
-        # permitted, the effect of an executor shutdown should happen only once.
-        self.is_shutdown = False
-
-    def atexit_cleanup(self):
-        # Calls this executor's shutdown method upon Python exiting the process.
-        self.shutdown()
-
     def _get_launch_command(self, block_id):
         # this executor uses different terminology for worker/launch
         # commands than in htex
@@ -325,8 +306,6 @@ def start(self):
         """Create submit process and collector thread to create, send, and
         retrieve Parsl tasks within the Work Queue system.
         """
-        # Mark this executor object as started
-        self.is_started = True
         self.tasks_lock = threading.Lock()
 
         # Create directories for data and results
@@ -713,14 +692,6 @@ def shutdown(self, *args, **kwargs):
         """Shutdown the executor. Sets flag to cancel the submit process and
         collector thread, which shuts down the Work Queue system submission.
         """
-        if not self.is_started:
-            # Don't shutdown if the executor never starts.
-            return
-
-        if self.is_shutdown:
-            # Don't shutdown this executor again.
-            return
-
         logger.debug("Work Queue shutdown started")
         self.should_stop.value = True
 
@@ -741,7 +712,6 @@ def shutdown(self, *args, **kwargs):
         self.collector_queue.close()
         self.collector_queue.join_thread()
 
-        self.is_shutdown = True
         logger.debug("Work Queue shutdown completed")
 
     @wrap_with_logs


Test patch: