Repository: Parsl/parsl

Issue:
HTEX worker watchdog does not start workers
**Describe the bug**
The HTEX `Manager.worker_watchdog` method creates a worker process object when a worker fails, but [does not start](https://github.com/Parsl/parsl/blob/741293a76c2ad114b48bcc927701258b7ba7f355/parsl/executors/high_throughput/process_worker_pool.py#L401-L411) the process.

**To Reproduce**
1. Run a function that kills the worker
```python
@python_app
def kill_worker():
    import sys
    sys.exit(2)
```
2. Observe that no new worker process starts.

**Expected behavior**
The worker watchdog should recognize that the worker is dead and start a new process.

**Environment**
 - Parsl version: 2023.11.20


Discussion:

Resolving patch:
diff --git a/parsl/executors/high_throughput/process_worker_pool.py b/parsl/executors/high_throughput/process_worker_pool.py
--- a/parsl/executors/high_throughput/process_worker_pool.py
+++ b/parsl/executors/high_throughput/process_worker_pool.py
@@ -398,15 +398,7 @@ def worker_watchdog(self, kill_event: threading.Event):
                     except KeyError:
                         logger.info("Worker {} was not busy when it died".format(worker_id))
 
-                    p = self.mpProcess(target=worker, args=(worker_id,
-                                                            self.uid,
-                                                            self.worker_count,
-                                                            self.pending_task_queue,
-                                                            self.pending_result_queue,
-                                                            self.ready_worker_queue,
-                                                            self._tasks_in_progress,
-                                                            self.cpu_affinity),
-                                       name="HTEX-Worker-{}".format(worker_id))
+                    p = self._start_worker(worker_id)
                     self.procs[worker_id] = p
                     logger.info("Worker {} has been restarted".format(worker_id))
 
@@ -423,18 +415,7 @@ def start(self):
 
         self.procs = {}
         for worker_id in range(self.worker_count):
-            p = self.mpProcess(target=worker,
-                               args=(worker_id,
-                                     self.uid,
-                                     self.worker_count,
-                                     self.pending_task_queue,
-                                     self.pending_result_queue,
-                                     self.ready_worker_queue,
-                                     self._tasks_in_progress,
-                                     self.cpu_affinity,
-                                     self.available_accelerators[worker_id] if self.accelerators_available else None),
-                               name="HTEX-Worker-{}".format(worker_id))
-            p.start()
+            p = self._start_worker(worker_id)
             self.procs[worker_id] = p
 
         logger.debug("Workers started")
@@ -476,6 +457,25 @@ def start(self):
         logger.info("process_worker_pool ran for {} seconds".format(delta))
         return
 
+    def _start_worker(self, worker_id: int):
+        p = self.mpProcess(
+            target=worker,
+            args=(
+                worker_id,
+                self.uid,
+                self.worker_count,
+                self.pending_task_queue,
+                self.pending_result_queue,
+                self.ready_worker_queue,
+                self._tasks_in_progress,
+                self.cpu_affinity,
+                self.available_accelerators[worker_id] if self.accelerators_available else None,
+            ),
+            name="HTEX-Worker-{}".format(worker_id),
+        )
+        p.start()
+        return p
+
 
 def execute_task(bufs):
     """Deserialize the buffer and execute the task.


Test patch:
diff --git a/parsl/tests/test_error_handling/test_htex_worker_failure.py b/parsl/tests/test_error_handling/test_htex_worker_failure.py
--- a/parsl/tests/test_error_handling/test_htex_worker_failure.py
+++ b/parsl/tests/test_error_handling/test_htex_worker_failure.py
@@ -19,7 +19,8 @@ def kill_worker():
 
 
 @pytest.mark.local
-def test_htex_worker_failure():
-    with pytest.raises(WorkerLost):
-        f = kill_worker()
-        f.result()
+def test_htex_worker_failures():
+    for _ in range(3):
+        with pytest.raises(WorkerLost):
+            f = kill_worker()
+            f.result()