Repository: Parsl/parsl

Issue:
Monitoring Bug on ALCF Theta
**Describe the bug**

I get some errors when using monitoring on Theta, which I think are causing problems with executors. When using monitoring, my jobs sent to a `ThreadPoolExecutor` execute but I do not see a "Task Completed" notification and the jobs eventually stall:

```
2020-08-08 17:10:06,436 - root - ERROR - Exception reading IO counters for child memory_percent. Recorded IO usage may be incomplete
Traceback (most recent call last):
  File "/lus/theta-fs0/projects/CSC249ADCD08/colmena/env/lib/python3.7/site-packages/parsl/monitoring/monitoring.py", line 551, in monitor
    d['psutil_process_disk_write'] += child.io_counters().write_bytes
AttributeError: 'Process' object has no attribute 'io_counters'
```


**To Reproduce**

The following script computes on my desktop and stalls on the Theta login node:

```python
import logging

from concurrent.futures import as_completed

import parsl
from parsl import ThreadPoolExecutor, python_app
from parsl.addresses import address_by_hostname
from parsl.config import Config
from parsl.monitoring import MonitoringHub

#logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.INFO)

config = Config(
    executors=[
        ThreadPoolExecutor(label="local_threads", max_threads=1)
    ],
    monitoring=MonitoringHub(
        hub_address=address_by_hostname(),
        hub_port=55055,
        monitoring_debug=False,
        resource_monitoring_interval=10,
    ),
    strategy=None,
)

parsl.load(config)


@python_app
def func(x):
    return x + 1


futures = [func(i) for i in range(32)]
for f in as_completed(futures):
    print(f.result())
```

**Expected behavior**

No scary error messages and my jobs all executing completely.


**Environment**
 - OS: Centos
 - Python version: 3.7
 - Parsl version: 1.0

**Distributed Environment**
- Where are you running the Parsl script from ? Launch node
- Where do you need the workers to run ? Some on compute nodes, some on launch node



Discussion:
monitoring isn't supported on ThreadPoolExecutor - so don't be surprised that it doesn't work with anything other than HighThroughputExecutor.

Do you know which version of the `psutil` library you are using? It would be interesting to see what you get if you log
`child.as_dict()` on that exception to show what information is available - for example, I see this on a test process:

```
>>> p.as_dict()
{'threads': [pthread(id=1, user_time=0.05, system_time=0.02)], 'num_threads': 1, 'cpu_percent': 0.0, 'memory_full_info': None, 'cpu_times': pcputimes(user=0.05, system=0.02, children_user=903.06, children_system=621.44), 'cpu_affinity': [0, 1, 2, 3, 4, 5, 6, 7], 'memory_info': pmem(rss=0, vms=7020544, shared=0, text=897024, lib=0, data=573440, dirty=0), 'ionice': pionice(ioclass=<IOPriority.IOPRIO_CLASS_NONE: 0>, value=0), 'nice': 0, 'open_files': None, 'memory_percent': 0.0, 'io_counters': None, 'memory_maps': None, 'status': 'sleeping', 'environ': None, 'num_ctx_switches': pctxsw(voluntary=636, involuntary=6), 'cmdline': ['/bin/bash', '/home/benc/tmp/cue/rootfile-benc-bsgjdeodu3poa1neabtg'], 'connections': None, 'gids': pgids(real=0, effective=0, saved=0), 'ppid': 0, 'cwd': None, 'pid': 1, 'uids': puids(real=0, effective=0, saved=0), 'name': 'rootfile-benc-b', 'num_fds': None, 'username': 'root', 'cpu_num': 2, 'exe': '/bin/bash', 'terminal': '/dev/pts/0', 'create_time': 1596012245.43}
```

The jobs stalling thing is also interesting - the above exception happening should not stop job progress because that exception is caught and handled entirely by that code. However if psutil is behaving unexpectedly, it's possible that there is another exception happening which isn't being caught and logged. I've encountered that in a few places when parsl launches threads, that exceptions fall off the top of the call stack; and in my LSST branch I have code which wraps most threads with another wrapper that properly logs exceptions falling off the top to help diagnose those problems.

You could do something similar here: put a try block around the entire body of `monitoring.py:monitor` making no assumptions about any of the code being well behaved, and catch and log anything that goes wrong.

I'm using `psutil` version 5.7.0. 

Thanks for the suggestion on the additional monitoring for Thread workers. I would not be surprised if the root of this error are exceptions not being caught and returned properly. 
@WardLT Thanks for reporting the issue. As Ben pointed out, monitoring does not work for `ThreadPoolExecutor`. 

I just tried your minimal example with Parsl master, it did hang. But when I switched it to `HighThroughputExecutor`, it worked fine for me. I will dig into why it hangs.

BTW, I would like to note the issue #1069 . If later on you use monitoring on Theta compute nodes, you will probably see message missing with more than 40 workers/node. As far as I know, this is likely still an issue on Theta.
@ZhuozhaoLi  When I run @WardLT 's script on Theta, I see all the tasks return, but the DFK appears to be hung at the end with these lines at the end of the parsl.log :
```
2020-08-10 15:55:07.158 parsl.monitoring.monitoring:267 [INFO]  Terminating Monitoring Hub
2020-08-10 15:55:07.159 parsl.monitoring.monitoring:283 [INFO]  Waiting for Hub to receive all messages and terminate
```
> @ZhuozhaoLi When I run @WardLT 's script on Theta, I see all the tasks return, but the DFK appears to be hung at the end with these lines at the end of the parsl.log :
> 
> ```
> 2020-08-10 15:55:07.158 parsl.monitoring.monitoring:267 [INFO]  Terminating Monitoring Hub
> 2020-08-10 15:55:07.159 parsl.monitoring.monitoring:283 [INFO]  Waiting for Hub to receive all messages and terminate
> ```

I tried to run Logan's script for 10 times, I think 2 out of 10 succeeded, but the rest hung. For the hung ones, I saw these lines:

```
2020-08-10 16:07:30.552 parsl.dataflow.dflow:907 [INFO]  DFK cleanup initiated
2020-08-10 16:07:30.553 parsl.dataflow.dflow:810 [INFO]  Summary of tasks in DFK:
2020-08-10 16:07:30.553 parsl.dataflow.dflow:823 [INFO]  Tasks in state States.done: 1
2020-08-10 16:07:30.553 parsl.dataflow.dflow:823 [INFO]  Tasks in state States.launched: 31
2020-08-10 16:07:30.553 parsl.dataflow.dflow:829 [INFO]  End of summary
```
That summary looks weird if you're expecting all of the tasks to be in `done` state at exit.

Resolving patch:
diff --git a/parsl/monitoring/db_manager.py b/parsl/monitoring/db_manager.py
--- a/parsl/monitoring/db_manager.py
+++ b/parsl/monitoring/db_manager.py
@@ -428,11 +428,12 @@ def start(self, priority_queue, node_queue, resource_queue) -> None:
             if resource_messages:
                 logger.debug(
                     "Got {} messages from resource queue, {} reprocessable".format(len(resource_messages), len(reprocessable_first_resource_messages)))
-                self._insert(table=RESOURCE, messages=resource_messages)
+
+                insert_resource_messages = []
                 for msg in resource_messages:
                     task_try_id = str(msg['task_id']) + "." + str(msg['try_id'])
                     if msg['first_msg']:
-
+                        # Update the running time to try table if first message
                         msg['task_status_name'] = States.running.name
                         msg['task_try_time_running'] = msg['timestamp']
 
@@ -442,6 +443,12 @@ def start(self, priority_queue, node_queue, resource_queue) -> None:
                             if task_try_id in deferred_resource_messages:
                                 logger.error("Task {} already has a deferred resource message. Discarding previous message.".format(msg['task_id']))
                             deferred_resource_messages[task_try_id] = msg
+                    else:
+                        # Insert to resource table if not first message
+                        insert_resource_messages.append(msg)
+
+                if insert_resource_messages:
+                    self._insert(table=RESOURCE, messages=insert_resource_messages)
 
             if reprocessable_first_resource_messages:
                 self._insert(table=STATUS, messages=reprocessable_first_resource_messages)
diff --git a/parsl/monitoring/monitoring.py b/parsl/monitoring/monitoring.py
--- a/parsl/monitoring/monitoring.py
+++ b/parsl/monitoring/monitoring.py
@@ -300,24 +300,32 @@ def monitor_wrapper(f,
         Wrap the Parsl app with a function that will call the monitor function and point it at the correct pid when the task begins.
         """
         def wrapped(*args, **kwargs):
-            command_q = Queue(maxsize=10)
+            # Send first message to monitoring router
+            try:
+                monitor(os.getpid(),
+                        task_id,
+                        monitoring_hub_url,
+                        run_id,
+                        logging_level,
+                        sleep_dur,
+                        first_message=True)
+            except Exception:
+                pass
+
+            # create the monitor process and start
             p = Process(target=monitor,
                         args=(os.getpid(),
                               try_id,
                               task_id,
                               monitoring_hub_url,
                               run_id,
-                              command_q,
                               logging_level,
                               sleep_dur),
                         name="Monitor-Wrapper-{}".format(task_id))
             p.start()
+
             try:
-                try:
-                    return f(*args, **kwargs)
-                finally:
-                    command_q.put("Finished")
-                    p.join()
+                return f(*args, **kwargs)
             finally:
                 # There's a chance of zombification if the workers are killed by some signals
                 p.terminate()
@@ -472,26 +480,36 @@ def monitor(pid,
             task_id,
             monitoring_hub_url,
             run_id,
-            command_q,
             logging_level=logging.INFO,
-            sleep_dur=10):
+            sleep_dur=10,
+            first_message=False):
     """Internal
     Monitors the Parsl task's resources by pointing psutil to the task's pid and watching it and its children.
     """
-    import psutil
     import platform
+    import time
+
+    radio = UDPRadio(monitoring_hub_url,
+                     source_id=task_id)
+
+    if first_message:
+        msg = {'run_id': run_id,
+               'task_id': task_id,
+               'hostname': platform.node(),
+               'first_msg': first_message,
+               'timestamp': datetime.datetime.now()
+        }
+        radio.send(msg)
+        return
 
+    import psutil
     import logging
-    import time
-    import queue
 
     format_string = "%(asctime)s.%(msecs)03d %(name)s:%(lineno)d [%(levelname)s]  %(message)s"
     logging.basicConfig(filename='{logbase}/monitor.{task_id}.{pid}.log'.format(
         logbase="/tmp", task_id=task_id, pid=pid), level=logging_level, format=format_string)
-    logging.debug("start of monitor")
 
-    radio = UDPRadio(monitoring_hub_url,
-                     source_id=task_id)
+    logging.debug("start of monitor")
 
     # these values are simple to log. Other information is available in special formats such as memory below.
     simple = ["cpu_num", 'cpu_percent', 'create_time', 'cwd', 'exe', 'memory_percent', 'nice', 'name', 'num_threads', 'pid', 'ppid', 'status', 'username']
@@ -501,7 +519,6 @@ def monitor(pid,
     pm = psutil.Process(pid)
     pm.cpu_percent()
 
-    first_msg = True
     children_user_time = {}
     children_system_time = {}
     total_children_user_time = 0.0
@@ -515,7 +532,7 @@ def monitor(pid,
             d["try_id"] = try_id
             d['resource_monitoring_interval'] = sleep_dur
             d['hostname'] = platform.node()
-            d['first_msg'] = first_msg
+            d['first_msg'] = first_message
             d['timestamp'] = datetime.datetime.now()
 
             logging.debug("getting children")
@@ -559,17 +576,10 @@ def monitor(pid,
             d['psutil_process_time_system'] += total_children_system_time
             logging.debug("sending message")
             radio.send(d)
-            first_msg = False
         except Exception:
             logging.exception("Exception getting the resource usage. Not sending usage to Hub", exc_info=True)
 
-        try:
-            msg = command_q.get(block=False)
-            if msg == "Finished":
-                logging.info("Received task finished message. Ending the monitoring loop now.")
-                break
-        except queue.Empty:
-            logging.debug("Have not received any message.")
-
         logging.debug("sleeping")
         time.sleep(sleep_dur)
+
+    logger.info("Monitor exiting")


Test patch: