Repository: Parsl/parsl

Issue:
SimpleLauncher should help the user understand why they are not getting multiple nodes
**Is your feature request related to a problem? Please describe.**

When Parsl runs workers inside a batch system, each block of workers can contain multiple compute nodes.

In order to run across all of the compute nodes inside that block, users can specify a `Launcher`, which is usually specific to the particular kind of HPC system that they are using: https://parsl.readthedocs.io/en/stable/reference.html#launchers

There is also the `SimpleLauncher` which doesn't try to run workers across multiple nodes, but only runs on the single node it is already running on.

This can be confusing to users when asking for multiple worker nodes, which are then unused if they do not specify the correct launcher but instead choose the SimpleLauncher.

**Describe the solution you'd like**

SimpleLauncher should inspect its `nodes_per_block` parameter and raise an exception if users are trying to use it with multiple nodes in a block, rather than 1 node per block.

Sometimes advanced users genuinely do want to run this way (for example in @yadudoc 's HTEX MPI prototype) and so the `__init__` method should get a new parameter "permit_multiple_nodes" which should allow users to specify multiple nodes per block without getting this exception. This new parameter should have a default value of `False` so that users who do not specify this option will get the new exception, and the old behaviour is only seen by advanced users.

In addition to implementing this, there should be three test cases:
* test that the launcher works with one node per block specified
* test that the launcher raises an exception with some other number of nodes per block, for example 64 nodes per block
* test that advanced users can disable the exception when using 64 nodes per block but setting permit_multiple_nodes=True


**Additional context**
this comes from discussion with @Andrew-S-Rosen 


Discussion:
CCing @WardLT since the exception (and proposed kwarg to circumvent it) is relevant to his workflows as well. 
Food for thought: would it not make more sense to log a warning? Then there is no need for a new kwarg. 
Other similar code, the wrapped launcher, logs a warning in similar circumstances and it would be more internally consistent to do follow that style. Mostly this comes from my preference of "you aren't going to read the warnings, and this is almost definitely the wrong thing to be doing". But yeah, I'd be fine with logging a warning like the wrapped launcher does a few lines below.
That is, my view is:
The behaviour right now is you are asking the launcher to do X and it cannot do X, but instead it does Y, which is not X. It might be more desirable to raise an exception that X cannot be done, rather than doing something else that is not X.


Until there is robust, first-class support for Apps based on Python functions that launch MPI subprocesses, I would lean towards not raising an exception since it will break people's codes. Anyone who watches Logan's YouTube videos for instance. 😉

Edit: https://youtu.be/0V4Hs4kTyJs?si=zbuDT7xxRnWjKfUn (8:20)
I’d like to give this a shot. 
@Barry-dE ok great. See @Andrew-S-Rosen 's comments that I agree with - this should probably be a warning log message, rather than an exception that I initially proposed at the top of the issue.

If you look in launchers.py, the `WrappedLauncher` class has some warning code for nodes_per_block > 1 that looks quite like what I am imagining now.
Working on it. Will submit a PR once all the tests pass. 
I will like to work on this @benclifford 
I made a PR for this issue @benclifford . I will like to have your review on it. 
https://github.com/Parsl/parsl/pull/3214

Resolving patch:
diff --git a/parsl/launchers/launchers.py b/parsl/launchers/launchers.py
--- a/parsl/launchers/launchers.py
+++ b/parsl/launchers/launchers.py
@@ -11,11 +11,13 @@ class SimpleLauncher(Launcher):
     def __init__(self, debug: bool = True) -> None:
         super().__init__(debug=debug)
 
-    def __call__(self, command: str, tasks_per_node: int, nodes_per_block: int, permit_multiple_nodes: bool = False) -> str:
+    def __call__(self, command: str, tasks_per_node: int, nodes_per_block: int) -> str:
+
+        if nodes_per_block > 1:
+            logger.warning('Simple Launcher only supports single node per block. '
+                           f'Requested nodes: {nodes_per_block}. '
+                           'You may be getting fewer workers than expected')
 
-        if nodes_per_block > 1 and not permit_multiple_nodes:
-            logger.warning("SimpleLauncher only supports 1 node per block. "
-                           "Set permit_multiple_nodes=True to allow multiple nodes per block.")
         return command
 
 


Test patch: