Repository: Parsl/parsl

Issue:
parsl vs typeguard 4 doesn't work
**Describe the bug**
multiprocessing.Queue is used as a generic type in "monitoring/monitoring.py"; example case; see [1]
multiprocessing.Queue is not a "type" (at-least in Python 3.11.7)
Quoting pydoc:
"multiprocessing.Queue = Queue(maxsize=0) method of multiprocessing.context.DefaultContext instance
    Returns a queue object"
This is causing code that uses MonitoringHub to fail.

[1]: See: https://github.com/Parsl/parsl/blob/479cf9db614b8f1e954f177ecbbc0ddccfb4e581/parsl/monitoring/monitoring.py#L171

**To Reproduce**
1. Setup Parsl 2024.1.8 with Python 3.11.7
2. Run the following test script:

```
> cat test.py                                                                                                                                                                                                                                      (episim37)
import parsl
from parsl.monitoring.monitoring import MonitoringHub
from parsl.config import Config
from parsl.executors import HighThroughputExecutor
from parsl.addresses import address_by_hostname

config = Config(
   executors=[
       HighThroughputExecutor(
           label="local_htex",
           cores_per_worker=1,
           max_workers=4,
           address=address_by_hostname(),
       )
   ],
   monitoring=MonitoringHub(
       hub_address=address_by_hostname(),
       hub_port=55055,
       monitoring_debug=False,
       resource_monitoring_interval=10,
   ),
   strategy='none'
)

parsl.load(config)
```
3. Run script
```
python test.py
```
4. See error
```
Traceback (most recent call last):
  File "/home/parantapa/test.py", line 25, in <module>
    parsl.load(config)
  File "/home/parantapa/miniconda3/envs/episim37/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 1428, in load
    cls._dfk = DataFlowKernel(config)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/parantapa/miniconda3/envs/episim37/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 116, in __init__
    self.hub_interchange_port = self.monitoring.start(self.run_id, self.run_dir)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/parantapa/miniconda3/envs/episim37/lib/python3.11/site-packages/parsl/monitoring/monitoring.py", line 171, in start
    comm_q: Queue[Union[Tuple[int, int], str]]
            ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'method' object is not subscriptable
```

**Expected behavior**
Code to not crash

**Environment**
 - OS: ArchLinux
 - Python version: 3.11.7
 - Parsl version: 2024.1.8

**Distributed Environment**
- Where are you running the Parsl script from ? [e.g. Laptop/Workstation, Login node, Compute node]
Laptop

- Where do you need the workers to run ? [e.g. Same as Parsl script, Compute nodes, Cloud nodes]
Laptop


Discussion:

Resolving patch:
diff --git a/parsl/monitoring/monitoring.py b/parsl/monitoring/monitoring.py
--- a/parsl/monitoring/monitoring.py
+++ b/parsl/monitoring/monitoring.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import os
 import socket
 import time
@@ -11,7 +13,8 @@
 import parsl.monitoring.remote
 
 from parsl.multiprocessing import ForkProcess, SizedQueue
-from multiprocessing import Process, Queue
+from multiprocessing import Process
+from multiprocessing.queues import Queue
 from parsl.utils import RepresentationMixin
 from parsl.process_loggers import wrap_with_logs
 from parsl.utils import setproctitle
@@ -20,7 +23,7 @@
 
 from parsl.monitoring.message_type import MessageType
 from parsl.monitoring.types import AddressedMonitoringMessage, TaggedMonitoringMessage
-from typing import cast, Any, Callable, Dict, Optional, Sequence, Tuple, Union
+from typing import cast, Any, Callable, Dict, Optional, Sequence, Tuple, Union, TYPE_CHECKING
 
 _db_manager_excepts: Optional[Exception]
 
@@ -171,7 +174,19 @@ def start(self, run_id: str, run_dir: str) -> int:
         self.logger.debug("Initializing ZMQ Pipes to client")
         self.monitoring_hub_active = True
 
-        comm_q: Queue[Union[Tuple[int, int], str]]
+        # This annotation is incompatible with typeguard 4.x instrumentation
+        # of local variables: Queue is not subscriptable at runtime, as far
+        # as typeguard is concerned. The more general Queue annotation works,
+        # but does not restrict the contents of the Queue. Using TYPE_CHECKING
+        # here allows the stricter definition to be seen by mypy, and the
+        # simpler definition to be seen by typeguard. Hopefully at some point
+        # in the future, Queue will allow runtime subscripts.
+
+        if TYPE_CHECKING:
+            comm_q: Queue[Union[Tuple[int, int], str]]
+        else:
+            comm_q: Queue
+
         comm_q = SizedQueue(maxsize=10)
 
         self.exception_q: Queue[Tuple[str, str]]


Test patch:
diff --git a/parsl/tests/test_bash_apps/test_stdout.py b/parsl/tests/test_bash_apps/test_stdout.py
--- a/parsl/tests/test_bash_apps/test_stdout.py
+++ b/parsl/tests/test_bash_apps/test_stdout.py
@@ -1,6 +1,7 @@
 import os
 
 import pytest
+import typeguard
 
 import parsl.app.errors as perror
 from parsl.app.app import bash_app
@@ -44,7 +45,11 @@ def test_bad_stdout_specs(spec):
     try:
         fn.result()
     except Exception as e:
-        assert isinstance(e, TypeError) or isinstance(e, perror.BadStdStreamFile), "Exception is wrong type"
+        # This tests for TypeCheckError by string matching on the type name
+        # because that class does not exist in typeguard 2.x - it is new in
+        # typeguard 4.x. When typeguard 2.x support is dropped, this test can
+        # become an isinstance check.
+        assert "TypeCheckError" in str(type(e)) or isinstance(e, TypeError) or isinstance(e, perror.BadStdStreamFile), "Exception is wrong type"
     else:
         assert False, "Did not raise expected exception"