Repository: Parsl/parsl

Issue:
Extreme Scale mpi workers do not register capacity fields
mpi_worker_pool.py does not register with the appropriate fields to get tasks issued, so registrations default to having 0 capacity and no tasks are issued to extreme scale workers.

Compare `create_reg_message` in `mpi_worker_pool.py` and in htex's `process_worker_pool.py`.

Config representation is broken
The config representation appears to be broken:
```
In [1]: from parsl.configs.htex_local import config

In [1]: config
Out[2]: Config()
```

Git bisect tells me the last commit where this error didn't happen was c6deae01a2840b8617c0b3975c9f8c99882a0c60. The commits immediately after that I haven't tested because I'm getting some error from the `enforce` package. Then by 695d2763449026f9779a61a2293960ea531d89bb I see the error. So this appears to have something to do with the type checking.

cc'ing @benclifford 


Discussion:

Removing the `typechecked` decorator [here](https://github.com/Parsl/parsl/blob/master/parsl/config.py#L55) fixes the representation. We'll need to poke around to figure out how to make them play nicely.
`RepresentationMixin` gets the list of parameters to display by inspecting the argument list of `__init__` and expecting that argument list to explicitly list all the relevant arguments:

```
>>> inspect.getfullargspec(Config.__init__)
FullArgSpec(args=['self', 'executors', 'app_cache', 'checkpoint_files', 'checkpoint_mode', 'checkpoint_period', 'data_management_max_threads', 'lazy_errors', 'retries', 'run_dir', 'strategy', 'monitoring', 'usage_tracking'], varargs=None, varkw=None, defaults=(None, True, None, None, None, 10, True, 0, 'runinfo', 'simple', None, False), kwonlyargs=[], kwonlydefaults=None, annotations={})
```

The `@typechecked` decorator replaces that `__init__` with a different init that performs its own type checking and then passes on every keyword arg to the original `__init__`  (so doing similar to RepresentationMixin). That replacement `__init__` does not list the arguments explicitly:

```
>>> inspect.getfullargspec(Config.__init__)
FullArgSpec(args=[], varargs='args', varkw='kwargs', defaults=None, kwonlyargs=[], kwonlydefaults=None, annotations={})
```
This specific problem is fixed by #927 which modifies RepresentationMixin to look for a single layer of `functools.update_wrapper` wrapping - which is used by the @typechecked decorator. #927 does not make RepresentationMixin cope with arbitrary decorators, though.

Resolving patch:
diff --git a/parsl/executors/extreme_scale/mpi_worker_pool.py b/parsl/executors/extreme_scale/mpi_worker_pool.py
--- a/parsl/executors/extreme_scale/mpi_worker_pool.py
+++ b/parsl/executors/extreme_scale/mpi_worker_pool.py
@@ -99,6 +99,9 @@ def create_reg_message(self):
                'os': platform.system(),
                'hname': platform.node(),
                'dir': os.getcwd(),
+               'prefetch_capacity': 0,
+               'worker_count': (self.comm.size - 1),
+               'max_capacity': (self.comm.size - 1) + 0,  # (+prefetch)
         }
         b_msg = json.dumps(msg).encode('utf-8')
         return b_msg


Test patch:
diff --git a/parsl/tests/sites/test_local_exex.py b/parsl/tests/sites/test_local_exex.py
new file mode 100644
--- /dev/null
+++ b/parsl/tests/sites/test_local_exex.py
@@ -0,0 +1,87 @@
+import argparse
+
+import pytest
+
+import parsl
+from parsl.dataflow.dflow import DataFlowKernel
+from parsl.app.app import App
+from parsl.tests.conftest import load_dfk
+from parsl.tests.configs.exex_local import config
+
+parsl.clear()
+dfk = DataFlowKernel(config=config)
+parsl.set_stream_logger()
+
+import logging
+logger = logging.getLogger(__name__)
+
+
+@App("python", dfk, executors=['Extreme_Local'])
+def python_app_2():
+    import os
+    import threading
+    import time
+    time.sleep(1)
+    return "Hello from PID[{}] TID[{}]".format(os.getpid(), threading.current_thread())
+
+
+@App("python", dfk, executors=['Extreme_Local'])
+def python_app_1():
+    import os
+    import threading
+    import time
+    time.sleep(1)
+    return "Hello from PID[{}] TID[{}]".format(os.getpid(), threading.current_thread())
+
+
+@App("bash", dfk)
+def bash_app(stdout=None, stderr=None):
+    return 'echo "Hello from $(uname -a)" ; sleep 2'
+
+
+@pytest.mark.local
+def test_python(N=2):
+    """Testing basic python functionality."""
+
+    r1 = {}
+    r2 = {}
+    for i in range(0, N):
+        r1[i] = python_app_1()
+        r2[i] = python_app_2()
+    print("Waiting ....")
+
+    for x in r1:
+        print("python_app_1 : ", r1[x].result())
+    for x in r2:
+        print("python_app_2 : ", r2[x].result())
+
+    return
+
+
+@pytest.mark.local
+def test_bash():
+    """Testing basic bash functionality."""
+
+    import os
+    fname = os.path.basename(__file__)
+
+    x = bash_app(stdout="{0}.out".format(fname))
+    print("Waiting ....")
+    print(x.result())
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-n", "--num", default=10,
+                        help="Count of apps to launch")
+    parser.add_argument("-d", "--debug", action='store_true',
+                        help="Count of apps to launch")
+    parser.add_argument("-c", "--config", default='local',
+                        help="Path to configuration file to run")
+    args = parser.parse_args()
+    load_dfk(args.config)
+    if args.debug:
+        parsl.set_stream_logger()
+
+    test_python()
+    test_bash()