Repository: Parsl/parsl

Issue:
Parsl hangs when using MPI Mode HTEx if resource specs are not found
**Describe the bug**
I found, accidentally, that Parsl will hang if a task is sent to HTEx running in "MPI Mode" if you fail to provide a `parsl_resource_dict` as an explicitly-named keyword argument.

It appears that the the resource requirements are not sent along with the task if you provide the resource specification as a keyword argument. 
The lack of task records means that the task is not registered in the `tasks_to_nodes` map in `MPIScheduler` and therefore not found when the last completes.

```
2024-05-13 12:07:17.349 parsl:409 8518 Result-Pusher [ERROR]  Got an exception: 1
Traceback (most recent call last):
  File "/home/lward/Work/ExaLearn/parsl/parsl/executors/high_throughput/process_worker_pool.py", line 403, in push_results
    r = self.task_scheduler.get_result(block=True, timeout=push_poll_period)
  File "/home/lward/Work/ExaLearn/parsl/parsl/executors/high_throughput/mpi_resource_management.py", line 213, in get_result
    nodes_to_reallocate = self._map_tasks_to_nodes[task_id]
KeyError: 1
```


**To Reproduce**

Remove the `parsl_resource_requirements` from the input arguments to an MPI app in any of the tests.

**Expected behavior**

Parsl at least fails with an error message, at least. It might be good for Parsl to substitute some default values (1 rank per node, 1 node) with a warning so that tasks still run.

**Environment**
 - OS: Ubuntu
 - Python version: 
 - Parsl version: `master` as of today.

**Distributed Environment**
- Where are you running the Parsl script from ? Desktop
- Where do you need the workers to run ? Same as parsl



Discussion:
If it's not an urgent fix, I can work on this over the weekend!
@R1j1t I think there are two things going on here. I think you should only try to fix the first one at first, in one PR. You might like to try the second one too, but it is much more subtle (and perhaps controversial), so if you attempt that, you should do that in a separate PR.

One part (perhaps easiest) is to make sure that at submit time (in `HighThroughputExecutor.submit`, perhaps, where it is very easy to fail) that if enable_mpi_mode is set, then validate the resource specs that are expected, and raise an exception right away if they are not. This is a good piece of the code to raise an exception, because it will result in that exception being handed back to the user pretty cleanly - so the user can quite easily see the explanation you are giving in the exception text.

One part, which is more architectural and more subtle, is that exceptions happening in some places will cause Parsl to hang because they do not correctly turn into a top level Parsl failure in the submitting process: for example, if the process worker pool `ResultPusher` thread gets an exception of any kind during the get result section, the code logs an exception and then does nothing else to converge towards a result.

I suspect that might be because sometimes, in not-encrypted mode, the worker pool gets random connections from elsewhere and tries to process those as results. That failure mode is more safely dealt with by turning on `encryption=True` (introduced in PR #3030) and maybe we should not be trying to ignore bad stuff happening at this layer any more. In which case, we can adopt an Erlang/OTP inspired approach and exit the process worker pool entirely. This will propagate failure of that task (and all others) upwards towards the user (at the expense of killing other tasks being performed by that worker pool).

(for comparison, issue #3404 is another "some piece goes wrong and we ignore that and hang")
> If it's not an urgent fix, I can work on this over the weekend!

This is not urgent -- this MPI work is an ongoing prototype that needs thoughtful fixes rather than hurried fixes.

Thanks @benclifford for the information. I will look into these suggestions in context of the codebase once I start, but your comments have already made my learning curve much smoother!

Resolving patch:
diff --git a/parsl/executors/high_throughput/executor.py b/parsl/executors/high_throughput/executor.py
--- a/parsl/executors/high_throughput/executor.py
+++ b/parsl/executors/high_throughput/executor.py
@@ -645,7 +645,8 @@ def submit(self, func, resource_specification, *args, **kwargs):
         Returns:
               Future
         """
-        validate_resource_spec(resource_specification)
+
+        validate_resource_spec(resource_specification, self.enable_mpi_mode)
 
         if self.bad_state_is_set:
             raise self.executor_exception
diff --git a/parsl/executors/high_throughput/mpi_prefix_composer.py b/parsl/executors/high_throughput/mpi_prefix_composer.py
--- a/parsl/executors/high_throughput/mpi_prefix_composer.py
+++ b/parsl/executors/high_throughput/mpi_prefix_composer.py
@@ -8,8 +8,18 @@
                    'mpiexec')
 
 
+class MissingResourceSpecification(Exception):
+    """Exception raised when input is  not supplied a resource specification"""
+
+    def __init__(self, reason: str):
+        self.reason = reason
+
+    def __str__(self):
+        return f"Missing resource specification: {self.reason}"
+
+
 class InvalidResourceSpecification(Exception):
-    """Exception raised when Invalid keys are supplied via resource specification"""
+    """Exception raised when Invalid input is supplied via resource specification"""
 
     def __init__(self, invalid_keys: Set[str]):
         self.invalid_keys = invalid_keys
@@ -18,13 +28,19 @@ def __str__(self):
         return f"Invalid resource specification options supplied: {self.invalid_keys}"
 
 
-def validate_resource_spec(resource_spec: Dict[str, str]):
+def validate_resource_spec(resource_spec: Dict[str, str], is_mpi_enabled: bool):
     """Basic validation of keys in the resource_spec
 
     Raises: InvalidResourceSpecification if the resource_spec
         is invalid (e.g, contains invalid keys)
     """
     user_keys = set(resource_spec.keys())
+
+    # empty resource_spec when mpi_mode is set causes parsl to hang
+    # ref issue #3427
+    if is_mpi_enabled and len(user_keys) == 0:
+        raise MissingResourceSpecification('MPI mode requires optional parsl_resource_specification keyword argument to be configured')
+
     legal_keys = set(("ranks_per_node",
                       "num_nodes",
                       "num_ranks",


Test patch:
diff --git a/parsl/tests/test_mpi_apps/test_mpi_mode_enabled.py b/parsl/tests/test_mpi_apps/test_mpi_mode_enabled.py
--- a/parsl/tests/test_mpi_apps/test_mpi_mode_enabled.py
+++ b/parsl/tests/test_mpi_apps/test_mpi_mode_enabled.py
@@ -6,6 +6,8 @@
 from parsl import python_app, bash_app
 from parsl.tests.configs.htex_local import fresh_config
 
+from parsl.executors.high_throughput.mpi_prefix_composer import MissingResourceSpecification
+
 import os
 
 EXECUTOR_LABEL = "MPI_TEST"
@@ -168,3 +170,11 @@ def test_simulated_load(rounds: int = 100):
         total_ranks, nodes = future.result(timeout=10)
         assert len(nodes) == futures[future]["num_nodes"]
         assert total_ranks == futures[future]["num_nodes"] * futures[future]["ranks_per_node"]
+
+
+@pytest.mark.local
+def test_missing_resource_spec():
+
+    with pytest.raises(MissingResourceSpecification):
+        future = mock_app(sleep_dur=0.4)
+        future.result(timeout=10)
diff --git a/parsl/tests/test_mpi_apps/test_resource_spec.py b/parsl/tests/test_mpi_apps/test_resource_spec.py
--- a/parsl/tests/test_mpi_apps/test_resource_spec.py
+++ b/parsl/tests/test_mpi_apps/test_resource_spec.py
@@ -19,7 +19,8 @@
 )
 from parsl.executors.high_throughput.mpi_prefix_composer import (
     validate_resource_spec,
-    InvalidResourceSpecification
+    InvalidResourceSpecification,
+    MissingResourceSpecification
 )
 
 EXECUTOR_LABEL = "MPI_TEST"
@@ -122,18 +123,22 @@ def test_top_level():
 
 @pytest.mark.local
 @pytest.mark.parametrize(
-    "resource_spec, exception",
+    "resource_spec, is_mpi_enabled, exception",
     (
-        ({"num_nodes": 2, "ranks_per_node": 1}, None),
-        ({"launcher_options": "--debug_foo"}, None),
-        ({"num_nodes": 2, "BAD_OPT": 1}, InvalidResourceSpecification),
-        ({}, None),
+        ({"num_nodes": 2, "ranks_per_node": 1}, False, None),
+        ({"launcher_options": "--debug_foo"}, False, None),
+        ({"num_nodes": 2, "BAD_OPT": 1}, False, InvalidResourceSpecification),
+        ({}, False, None),
+        ({"num_nodes": 2, "ranks_per_node": 1}, True, None),
+        ({"launcher_options": "--debug_foo"}, True, None),
+        ({"num_nodes": 2, "BAD_OPT": 1}, True, InvalidResourceSpecification),
+        ({}, True, MissingResourceSpecification),
     )
 )
-def test_resource_spec(resource_spec: Dict, exception):
+def test_resource_spec(resource_spec: Dict, is_mpi_enabled: bool, exception):
     if exception:
         with pytest.raises(exception):
-            validate_resource_spec(resource_spec)
+            validate_resource_spec(resource_spec, is_mpi_enabled)
     else:
-        result = validate_resource_spec(resource_spec)
+        result = validate_resource_spec(resource_spec, is_mpi_enabled)
         assert result is None