Repository: Parsl/parsl

Issue:
1 second workflow takes 31 seconds with monitoring enabled
**Describe the bug**
The below workflow takes 31 seconds to run with monitoring enabled. With monitoring disabled, it takes 1 second.

A clear and concise description of what the bug is.

**To Reproduce**
Run this code with master = ea8333c3249fdf3b21691b7a66fb4e9c8ed37ec0

```
import parsl

from parsl.monitoring.monitoring import MonitoringHub
from parsl.configs.local_threads import config
from parsl.addresses import address_by_hostname

config.monitoring=MonitoringHub(
       hub_address=address_by_hostname(),
       hub_port=55055,
       resource_monitoring_interval=10,
   )

parsl.load(config)

@parsl.python_app()
def foo():
    return 7

print(foo().result())
```

```
time python md.py 
7

real    0m31.128s
user    0m1.000s
sys     0m0.260s
```

**Expected behavior**
The workflow should run much closer to the runtime with monitoring disabled:

```
time python md.py 
7

real    0m0.829s
user    0m0.692s
sys     0m0.124s
```

**Environment**
my laptop, python 3.7, parsl master ea8333c3249fdf3b21691b7a66fb4e9c8ed37ec0



Discussion:
I met the same issue yesterday. It is caused by waiting for `Hub` process to join (the code below)

https://github.com/Parsl/parsl/blob/a3dfd360ad01b8962bd43eea077c8e58c79d345f/parsl/monitoring/monitoring.py#L267-L269

The log showed `Hub` finished 30 seconds before the `join` completed.
```
2019-11-14 04:49:11.283 hub:430 [INFO]  Hub finished
```
```
2019-11-14 04:49:08.143 monitoring_hub:267 [INFO]  Waiting for Hub to receive all messages and terminate
2019-11-14 04:49:38.352 monitoring_hub:269 [DEBUG]  Finished waiting for Hub termination
```
It is probably related to the problem in multiprocessing `Queue`.

https://stackoverflow.com/questions/42350933/joining-multiprocessing-queue-takes-a-long-time
Was trying to further investigate this issue. I added a few lines of code below to `Hub`.

```
        self.sock.close()
        self.logger.info("UDP socker closed.")
        self.dfk_channel.close()
        self.logger.info("DFK channel closed.")
        self.ic_channel.close()
        self.logger.info("IC channel closed.")
        self._context.destroy()
        self.logger.info("ZMQ context destroyed.")
        self.logger.info("Hub finished")
```

The logs showed that it is the zmq context that took such a long time to destroy and blocked the process from joining.

```
2019-11-15 19:32:40.583 hub:436 [INFO]  UDP socker closed.
2019-11-15 19:32:40.583 hub:438 [INFO]  DFK channel closed.
2019-11-15 19:32:40.583 hub:440 [INFO]  IC channel closed.
2019-11-15 19:33:07.572 hub:442 [INFO]  ZMQ context destroyed.
2019-11-15 19:33:07.572 hub:443 [INFO]  Hub finished
```

Will further check what is happening here.

Resolving patch:
diff --git a/parsl/monitoring/monitoring.py b/parsl/monitoring/monitoring.py
--- a/parsl/monitoring/monitoring.py
+++ b/parsl/monitoring/monitoring.py
@@ -370,11 +370,13 @@ def __init__(self,
 
         self._context = zmq.Context()
         self.dfk_channel = self._context.socket(zmq.DEALER)
+        self.dfk_channel.setsockopt(zmq.LINGER, 0)
         self.dfk_channel.set_hwm(0)
         self.dfk_channel.RCVTIMEO = int(self.loop_freq)  # in milliseconds
         self.dfk_channel.connect("tcp://{}:{}".format(client_address, client_port))
 
         self.ic_channel = self._context.socket(zmq.DEALER)
+        self.ic_channel.setsockopt(zmq.LINGER, 0)
         self.ic_channel.set_hwm(0)
         self.ic_channel.RCVTIMEO = int(self.loop_freq)  # in milliseconds
         self.logger.debug("hub_address: {}. hub_port_range {}".format(hub_address, hub_port_range))


Test patch: