Repository: Parsl/parsl

Issue:
Monitoring is consuming excessive amounts of CPU time
**Describe the bug**
Running a top-level workflow script on a NERSC login node, I am seeing that with monitoring enabled, processes associated with the workflow consume ~2 full cores 100% of the time, even when the workflow is idling,  awaiting Slurm job(s) to start running.  This has the consequence that after a couple of hours, said processes trigger some sort of watchdog which then *kills* the interactive top-level workflow script.   This, of course, is not good for the batch jobs once they begin to run.

Running the same workflow *without* monitoring reveals ~5% CPU consumption (vs 200%).  Are there some additional configuration knobs one may use to safely reduce this activity?  I have successfully used the following in my executor config (thanks @benclifford !), but they seem to have no impact on monitoring:

poll_period=5000
heartbeat_period=300
heartbeat_threshold=1200

And for then for monitoring I am using this default(?) stanza:
resource_monitoring_interval=60

(Is there documentation for these parameters, along with the units?)

**To Reproduce**
1. Setup Parsl 0.8.0 with Python 3.7.2 on cori login node
2. Configure a test workflow (requiring one or more batch jobs) to enable monitoring
3. Run top-level workflow script, then run 'top'

**Expected behavior**
Looking at the CPU consumed by the workflow scripts (using 'top') before the batch job is dispatched will reveal the problem.

**Environment**
 - OS: 
Linux cori02 4.12.14-150.27-default #1 SMP Thu Jul 11 11:24:28 UTC 2019 (bf2abc2) x86_64 x86_64 x86_64 GNU/Linux

 - Python 3.7.2
 - Parsl 0.8.0 (+fixes/enhancements in 'master')

**Distributed Environment**
- Where are you running the Parsl script from ? Cori.nersc.gov login node
- Where do you need the workers to run ? Cori batch (either KNL or Haswell) via htex 



Discussion:
Will look it to this high CPU consumption issue soon.

I think the MonitoringHub parameter documentation can be found here https://parsl.readthedocs.io/en/latest/stubs/parsl.monitoring.MonitoringHub.html.

Resolving patch:
diff --git a/parsl/monitoring/db_manager.py b/parsl/monitoring/db_manager.py
--- a/parsl/monitoring/db_manager.py
+++ b/parsl/monitoring/db_manager.py
@@ -365,7 +365,7 @@ def _migrate_logs_to_internal(self, logs_queue, queue_tag, kill_event):
             self.logger.debug("""Checking STOP conditions for {} threads: {}, {}"""
                               .format(queue_tag, kill_event.is_set(), logs_queue.qsize() != 0))
             try:
-                x, addr = logs_queue.get(block=False)
+                x, addr = logs_queue.get(timeout=0.1)
             except queue.Empty:
                 continue
             else:
@@ -392,9 +392,10 @@ def _get_messages_in_batch(self, msg_queue, interval=1, threshold=99999):
             if time.time() - start >= interval or len(messages) >= threshold:
                 break
             try:
-                x = msg_queue.get(block=False)
+                x = msg_queue.get(timeout=0.1)
                 # self.logger.debug("Database manager receives a message {}".format(x))
             except queue.Empty:
+                self.logger.debug("Database manager has not received any message.")
                 break
             else:
                 messages.append(x)
diff --git a/parsl/monitoring/monitoring.py b/parsl/monitoring/monitoring.py
--- a/parsl/monitoring/monitoring.py
+++ b/parsl/monitoring/monitoring.py
@@ -245,6 +245,7 @@ def start(self, run_id):
                                 name="Monitoring-DBM-Process"
         )
         self.dbm_proc.start()
+        self.logger.info("Started the Hub process {} and DBM process {}".format(self.queue_proc.pid, self.dbm_proc.pid))
 
         try:
             udp_dish_port, ic_port = comm_q.get(block=True, timeout=120)


Test patch: