Repository: Parsl/parsl

Issue:
"ZMQError: Operation cannot be accomplished in current state" triggers shutdown
## Brief description and request
I've been seeing an error crop up in longer running Parsl DFK's which trigger a graceful, but unwanted shutdown. I can't find a reliable/minimal example to cause this trigger other than "run a while." So, I'm going to post as much debug info I can about our setup and am seeking advice on things to test or other information which would be helpful in debugging this error.

I'm happy to test whatever is needed, I can also make direct code modifications 

## Backtrace of error thrown
 The final error is in the title but here is the backtrace, it reliably fails on this line:

```
... (there are higher functions in this stack from our program)
  File "/home/lnaden/github/qcfractal/qcfractal/queue/parsl_adapter.py", line 94, in count_running_workers
    if hasattr(executor, 'connected_workers'):
  File "/home/lnaden/miniconda3/envs/qca/lib/python3.7/site-packages/parsl/executors/high_throughput/executor.py", line 463, in connected_workers
    workers = self.command_client.run("WORKERS")
  File "/home/lnaden/miniconda3/envs/qca/lib/python3.7/site-packages/parsl/executors/high_throughput/zmq_pipes.py", line 40, in run
    self.zmq_socket.send_pyobj(message, copy=True)
  File "/home/lnaden/miniconda3/envs/qca/lib/python3.7/site-packages/zmq/sugar/socket.py", line 606, in send_pyobj
    return self.send(msg, flags=flags, **kwargs)
  File "/home/lnaden/miniconda3/envs/qca/lib/python3.7/site-packages/zmq/sugar/socket.py", line 395, in send
    return super(Socket, self).send(data, flags=flags, copy=copy, track=track)
  File "zmq/backend/cython/socket.pyx", line 725, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 772, in zmq.backend.cython.socket.Socket.send
  File "zmq/backend/cython/socket.pyx", line 247, in zmq.backend.cython.socket._send_copy
  File "zmq/backend/cython/socket.pyx", line 242, in zmq.backend.cython.socket._send_copy
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Operation cannot be accomplished in current state
```

Below this it behaves as though a standard shutdown signal and the next log item is `[I 190716 15:38:08 dflow:821] DFK cleanup initiated`

## Additional info

`if hasattr(executor, 'connected_workers'):` is inside a loop over the values: `parsl.dataflow.dflow.DataFlowKernel.executors.values()` 

Near as I can tell, this error is only thrown and logged at the higher level programs which are running Parsl and this does not appear in any of the logs or files from the `runinfo` folder. 

Leading up to the crash, there are multiple "loss of Manager" errors. The first set of lost managers happened ~6 hours after the processes were started, which is expected given the wall time of the cluster would have stopped them. However, after that initial set, we started loosing managers every 5-30 minutes seemingly randomly, all due to "too many missed heartbeats." This behavior is the same on 2 separate clusters (similar hardware, same SLURM scheduler). Not sure if this is related, but it's an observation.

Config file:
```python
Config(
    app_cache=True,
    checkpoint_files=None,
    checkpoint_mode=None,
    checkpoint_period=None,
    data_management_max_threads=10,
    executors=[HighThroughputExecutor(
        address='calogin1',
        cores_per_worker=16,
        heartbeat_period=30,
        heartbeat_threshold=120,
        interchange_port_range=(55000, 56000),
        label='QCFractal_Parsl_Slurm_Executor',
        launch_cmd='process_worker_pool.py {debug} {max_workers} -p {prefetch_capacity} -c {cores_per_worker} -m {mem_per_worker} --poll {poll_period} --task_url={task_url} --result_url={result_url} --logdir={logdir} --block_id={{block_id}} --hb_period={heartbeat_period} --hb_threshold={heartbeat_threshold} ',
        managed=True,
        max_workers=48,
        mem_per_worker=None,
        poll_period=10,
        prefetch_capacity=0,
        provider=SlurmProvider(
            'normal_q',
            channel=LocalChannel(envs={}, script_dir=None, userhome='/home/lnaden/qcarchive/run_cc'),
            cmd_timeout=10,
            exclusive=True,
            init_blocks=0,
            launcher=SingleNodeLauncher(),
            max_blocks=24,
            min_blocks=0,
            move_files=True,
            nodes_per_block=1,
            parallelism=1,
            scheduler_options='#SBATCH --exclusive\n#SBATCH -A themolssi\n',
            walltime='06:00:00',
            worker_init='source /home/lnaden/qcarchive/boot_qcarchive_env.sh'
        ),
        storage_access=[],
        suppress_failure=False,
        worker_debug=False,
        worker_logdir_root=None,
        worker_port_range=(54000, 55000),
        worker_ports=None,
        working_dir=None
    )],
    lazy_errors=True,
    monitoring=None,
    retries=0,
    run_dir='runinfo',
    strategy='simple',
    usage_tracking=False
)

```

Setup info: SLURM Cluster, executing locally on the head node in a background process. Directly calling and manipulating the `DataFlowKernel` after providing it a `Config` 


Discussion:
We have some more info for this bug. There are several conditions to make this happen we have found:

1. There had to be tasks to complete, which run as normal
2. The tasks in the Parsl Executor needed to deplete
3. The blocks were issued a `HOLD_BLOCK` command due to enough time passing without jobs in the queue.
4. The workers timed out, such that there were 0 blocks running/submitted/pending, and 0 connected workers. 
5. New tasks were added to the queue
6. New blocks were allocated: (`X/0/0 running/submitted/pending blocks,`), but no workers ever reported in (`and 0 connected workers`). 
7. The blocks would eventually be stopped and new blocks created (resulting in new scheduler jobs)
8. At some point the ZMQ error is thrown.

We can verify the worker process does start on the individual nodes, but they never seem able to report back to the central executor. We also can do this 

Attached is a tarball which has not only the `runinfo` directory for 2 runs, one with and without `worker_debug=True` respectively, but also our logfile which can actually shows the ZMQ bug. If need be, I can also get another runinfo folder from a production, but its about the same results.
 
cc @mattwelborn who was able to reliably reproduce the bug and got the runinfo, and cc @yadudoc who wanted to be pinged on this when we could reliably recreate this bug. Sorry it took so long for more info.

[blocks_no_workers_zmq.tar.gz](https://github.com/Parsl/parsl/files/3547145/blocks_no_workers_zmq.tar.gz)

Here's another failure involving zmq. Interestingly, this one does not show the same symptoms as the above case. Namely, no `HOLD_BLOCK` commands are issued and no workers time out.

[logs.tar.gz](https://github.com/Parsl/parsl/files/3547886/logs.tar.gz)

Any update here? This is an item that we are seeing often on half a dozen clusters.

Resolving patch:
diff --git a/parsl/executors/high_throughput/zmq_pipes.py b/parsl/executors/high_throughput/zmq_pipes.py
--- a/parsl/executors/high_throughput/zmq_pipes.py
+++ b/parsl/executors/high_throughput/zmq_pipes.py
@@ -4,6 +4,7 @@
 import time
 import pickle
 import logging
+import threading
 
 logger = logging.getLogger(__name__)
 
@@ -23,12 +24,27 @@ def __init__(self, ip_address, port_range):
 
         """
         self.context = zmq.Context()
-        self.zmq_socket = self.context.socket(zmq.REQ)
-        self.port = self.zmq_socket.bind_to_random_port("tcp://{}".format(ip_address),
-                                                        min_port=port_range[0],
-                                                        max_port=port_range[1])
+        self.ip_address = ip_address
+        self.port_range = port_range
+        self.port = None
+        self.create_socket_and_bind()
+        self._lock = threading.Lock()
 
-    def run(self, message):
+    def create_socket_and_bind(self):
+        """ Creates socket and binds to a port.
+
+        Upon recreating the socket, we bind to the same port.
+        """
+        self.zmq_socket = self.context.socket(zmq.REQ)
+        self.zmq_socket.setsockopt(zmq.LINGER, 0)
+        if self.port is None:
+            self.port = self.zmq_socket.bind_to_random_port("tcp://{}".format(self.ip_address),
+                                                            min_port=self.port_range[0],
+                                                            max_port=self.port_range[1])
+        else:
+            self.zmq_socket.bind("tcp://{}:{}".format(self.ip_address, self.port))
+
+    def run(self, message, max_retries=3):
         """ This function needs to be fast at the same time aware of the possibility of
         ZMQ pipes overflowing.
 
@@ -37,8 +53,24 @@ def run(self, message):
         in ZMQ sockets reaching a broken state once there are ~10k tasks in flight.
         This issue can be magnified if each the serialized buffer itself is larger.
         """
-        self.zmq_socket.send_pyobj(message, copy=True)
-        reply = self.zmq_socket.recv_pyobj()
+        reply = '__PARSL_ZMQ_PIPES_MAGIC__'
+        with self._lock:
+            for i in range(max_retries):
+                try:
+                    self.zmq_socket.send_pyobj(message, copy=True)
+                    reply = self.zmq_socket.recv_pyobj()
+                except zmq.ZMQError:
+                    logger.exception("Potential ZMQ REQ-REP deadlock caught")
+                    logger.info("Trying to reestablish context")
+                    self.zmq_socket.close()
+                    self.context.destroy()
+                    self.context = zmq.Context()
+                    self.create_socket_and_bind()
+
+        if reply == '__PARSL_ZMQ_PIPES_MAGIC__':
+            logger.error("Command channel run retries exhausted. Unable to run command")
+            raise Exception("Command Channel retries exhausted")
+
         return reply
 
     def close(self):


Test patch: