Repository: Parsl/parsl

Issue:
0.7.2 monitoring hangs in pytest
With the config added in benc-pytest-monitoring, 5762f434f518488881f3dae9cd18d29a25423c4c

pytest hangs after test completion:

```
$ pytest parsl/tests/test_bash_apps/test_apptimeout.py  --config parsl/tests/configs/local_threads_monitoring.py
============================= test session starts ==============================
platform linux -- Python 3.7.2+, pytest-4.0.2, py-1.8.0, pluggy-0.9.0
rootdir: /home/benc/parsl/src/parsl/parsl/tests, inifile: pytest.ini
plugins: xdist-1.26.1, ordering-0.6, forked-1.0.2, cov-2.6.1
collected 1 item                                                               

parsl/tests/test_bash_apps/test_apptimeout.py .                          [100%]

=========================== 1 passed in 3.65 seconds ===========================
```

The corresponding monitoring _hub.log shows:

```
2019-03-18 17:32:07.166 monitoring_hub:180 [INFO]  Monitoring Hub initialized
2019-03-18 17:32:07.774 monitoring_hub:232 [INFO]  Terminating Monitoring Hub
2019-03-18 17:32:07.774 monitoring_hub:235 [INFO]  Waiting Hub to receive all messages and terminate
2019-03-18 17:32:10.791 monitoring_hub:238 [INFO]  Received STOP from Hub
2019-03-18 17:32:10.791 monitoring_hub:241 [INFO]  Terminating Hub
2019-03-18 17:32:10.835 monitoring_hub:232 [INFO]  Terminating Monitoring Hub
2019-03-18 17:32:10.835 monitoring_hub:235 [INFO]  Waiting Hub to receive all messages and terminate

```

which looks as if the monitoring hub is trying to shut down twice.

Vague hypothesis: perhaps once when DFK is closed, and once when `__del__` is called by the garbage collector?

It's possible that this doesn't happen in manual tests because the garbage collector probably never tries to collect a MonitoringHub (or anything else) because the process is ready to exit; but if my hypothesis is right, this might cause similar hanging behaviour in interactive use.



Discussion:
I think your hypothesis is right. Currently, `MonitoringHub` waits for a `STOP` message from `Hub` before exiting since we want all the messages to be received. For the first time, `MonitoringHub` can receive that `STOP` message from `Hub`, But for the second time, the `Hub` is already closed, and the `MonitoringHub` would never receive that message -- thus it hangs.
It's probably more in consistent with how other resources are managed in parsl to perform the shutdown when the DFK shuts down, and avoid using `__del__` entirely

Resolving patch:
diff --git a/parsl/monitoring/monitoring.py b/parsl/monitoring/monitoring.py
--- a/parsl/monitoring/monitoring.py
+++ b/parsl/monitoring/monitoring.py
@@ -182,6 +182,7 @@ def start(self):
         self.logger.info("Monitoring Hub initialized")
 
         self.logger.debug("Initializing ZMQ Pipes to client")
+        self.monitoring_hub_active = True
         self._context = zmq.Context()
         self._dfk_channel = self._context.socket(zmq.DEALER)
         self._dfk_channel.set_hwm(0)
@@ -229,10 +230,11 @@ def send(self, mtype, message):
         self.logger.debug("Sending message {}, {}".format(mtype, message))
         return self._dfk_channel.send_pyobj((mtype, message))
 
-    def __del__(self):
+    def close(self):
         if self.logger:
             self.logger.info("Terminating Monitoring Hub")
-        if self._dfk_channel:
+        if self._dfk_channel and self.monitoring_hub_active:
+            self.monitoring_hub_active = False
             self._dfk_channel.close()
             self.logger.info("Waiting Hub to receive all messages and terminate")
             try:
@@ -244,8 +246,8 @@ def __del__(self):
             self.queue_proc.terminate()
             self.priority_msgs.put(("STOP", 0))
 
-    def close(self):
-        return self.__del__()
+    def __del__(self):
+        self.close()
 
     @staticmethod
     def monitor_wrapper(f, task_id, monitoring_hub_url, run_id, sleep_dur):


Test patch: