Repository: Parsl/parsl

Issue:
Parsl stuck after 30 minutes on HTCondor
**Describe the bug**
Using Parsl with HighThroughputExecutor and CondorProvider, Parsl becomes stuck after 30 minutes of no results from a manager.

**To Reproduce**
1. Run the following script with HTCondor available
  <details>
    <summary>Python script</summary>
    <!-- have to be followed by an empty line! -->

```python
#!/usr/bin/env python3

import os
import parsl
from tqdm import tqdm


def get_parsl_config(num_jobs):
    condor_init = "source {}/environment.sh\n".format(os.environ["HOME"])
    provider = parsl.providers.CondorProvider(
        init_blocks=num_jobs,
        max_blocks=num_jobs,
        worker_init=condor_init
    )
    parsl_executor = parsl.executors.HighThroughputExecutor(
        label="HTCondor",
        address=parsl.addresses.address_by_hostname(),
        max_workers=1,
        provider=provider,
    )
    parsl_config = parsl.config.Config(
        executors=[parsl_executor],
    )
    return parsl_config


@parsl.python_app
def work(sleep):
    import time
    time.sleep(sleep)
    return sleep

parsl_config = get_parsl_config(4)
parsl.load(parsl_config)
futures = [work(i + 28 * 60) for i in range(0, 4 * 60, 60)]
results = []
for future in tqdm(futures):
    results.append(future.result())

print(results)
```
  </details>

The `~/environment.sh` script that is executed here just sets `PATH`, `PYTHONPATH` and related variables to allow executing `process_worker_pool.py` with the needed libraries available on my particular setup.

 2. Wait for the HTCondor jobs to start and wait 30 minutes.
 3. The script will be stuck at 50%. Only the first two `future.result()` calls will return (the ones with 28 and 29 min of sleep), the others won't.
 
I also tested it with with various other sleep times. If the sleep time is below 30 min, it works. If it is above or equal it gets stuck. The HTCondor jobs continue running fine and the workers report having finished all the tasks.

**Expected behavior**
All four calls should return after a waiting time of 31 min and the script should print `[1680, 1740, 1800, 1860]`.

**Environment**
 - OS: CentOS Linux release 7.9.2009
 - Python version 3.6 and 3.8
 - Parsl version 1.0.0 and 1.1.0

**Distributed Environment**
- Where are you running the Parsl script from? Login node
- Where do you need the workers to run? HTCondor cluster nodes that run the same OS and software as the login node

**Additional Information**
From some debugging I did it looks to me like the connection `result_outgoing` of the Manager stops working, but I'm not sure if this is the entire problem.



Discussion:
can you attach a copy of your runinfo/ directory for a run with parsl 1.1.0?
an off-the-cuff thought without further evidence is that your network environment doesn't let network connections last longer than 30 minutes (that's the sort of thing I wouldn't expect in a clean network, but is the sort of thing I'd expect in some routed environments involving network address translation).
Here is the runinfo directory.
[runinfo.zip](https://github.com/Parsl/parsl/files/6384471/runinfo.zip)

@jrueb Thanks for getting us the logs. It looks like 4 tasks were dispatched to 4 workers and all of them completed execution:
```
> cat HTCondor/*/*/worker_0.log |  grep Completed
2021-04-27 14:15:56.166 worker_log:551 Rank:0 [INFO]  Completed task 2
2021-04-27 14:16:56.348 worker_log:551 Rank:0 [INFO]  Completed task 3
2021-04-27 14:14:52.326 worker_log:551 Rank:0 [INFO]  Completed task 1
2021-04-27 14:17:57.496 worker_log:551 Rank:0 [INFO]  Completed task 4
```

The obvious item that jumps out from the logs is that the `CondorProvider.walltime` is set to the default of 10 minutes. Could you retry running with a more reasonable walltime of say `00:45:00`? I'm a bit surprised that the managers getting dropped are not being detected. There's no indication in the `interchange.log` that the managers disconnected, or from the manager logs that there was some failure. Similarly, the `parsl.log` has no logged errors/exceptions. 

Do you think you could also pop in these additional config options for better debug logging?
```
Config(executors=[
      HighThroughputExecutors(
            worker_debug=True,       # This will add debug logs
            heartbeat_period=10,     # More frequent heartbeats to detect network drops
            heartbeat_threshold=30,  # Report failure if heartbeats were not received in a 30s period
      )

```

Here is the runinfo with walltime, worker_debug, heartbeat_period, and heartbeat_threshold set as mentioned.
[runinfo.zip](https://github.com/Parsl/parsl/files/6386317/runinfo.zip)

After further testing I think the `result_outgoing` connection of `process_worker_pool` `Manager` gets broken. Is there a reason why there are no heartbeats being sent over it?
If the Managers are connecting and then losing connection that might be an issue to discuss with your network admins. In the last log you shared, I do not see the HighThroughputExecutor detecting any Manager disconnects, but the logs show that all dispatched tasks finish execution. Can you share your latest logs please?
There are no changes to the logs. The connections are in the weird state of being connected but still nothing is received.
I would like to discuss this with my admins. If you could provide just a few lines of code that sets up the same type of connections that is used here, it would help a lot.
Try this script -> https://gist.github.com/yadudoc/a8bba2b31167c08633cfa5563e5fbce7
You'll have to give the worker-side the server address over command line args.

Resolving patch:
diff --git a/parsl/executors/high_throughput/executor.py b/parsl/executors/high_throughput/executor.py
--- a/parsl/executors/high_throughput/executor.py
+++ b/parsl/executors/high_throughput/executor.py
@@ -395,6 +395,8 @@ def _queue_management_worker(self):
                             exception = deserialize(msg['exception'])
                             self.set_bad_state_and_fail_all(exception)
                             break
+                        elif tid == -1 and 'heartbeat' in msg:
+                            continue
 
                         task_fut = self.tasks.pop(tid)
 
diff --git a/parsl/executors/high_throughput/interchange.py b/parsl/executors/high_throughput/interchange.py
--- a/parsl/executors/high_throughput/interchange.py
+++ b/parsl/executors/high_throughput/interchange.py
@@ -463,7 +463,7 @@ def start(self, poll_period=None):
                     tasks_requested = int.from_bytes(message[1], "little")
                     self._ready_manager_queue[manager]['last_heartbeat'] = time.time()
                     if tasks_requested == HEARTBEAT_CODE:
-                        logger.debug("[MAIN] Manager {} sent heartbeat".format(manager))
+                        logger.debug("[MAIN] Manager {} sent heartbeat via tasks connection".format(manager))
                         self.task_outgoing.send_multipart([manager, b'', PKL_HEARTBEAT_CODE])
                     else:
                         logger.debug("[MAIN] Manager {} requested {} tasks".format(manager, tasks_requested))
@@ -521,7 +521,10 @@ def start(self, poll_period=None):
                     for b_message in b_messages:
                         r = pickle.loads(b_message)
                         try:
-                            self._ready_manager_queue[manager]['tasks'].remove(r['task_id'])
+                            if int(r['task_id']) != -1:
+                                self._ready_manager_queue[manager]['tasks'].remove(r['task_id'])
+                            elif 'heartbeat' in r:
+                                logger.debug("[MAIN] Manager {} sent heartbeat via results connection".format(manager))
                         except Exception:
                             # If we reach here, there's something very wrong.
                             logger.exception("Ignoring exception removing task_id {} for manager {} with task list {}".format(
diff --git a/parsl/executors/high_throughput/process_worker_pool.py b/parsl/executors/high_throughput/process_worker_pool.py
--- a/parsl/executors/high_throughput/process_worker_pool.py
+++ b/parsl/executors/high_throughput/process_worker_pool.py
@@ -209,7 +209,7 @@ def create_reg_message(self):
         b_msg = json.dumps(msg).encode('utf-8')
         return b_msg
 
-    def heartbeat(self):
+    def heartbeat_to_incoming(self):
         """ Send heartbeat to the incoming task queue
         """
         heartbeat = (HEARTBEAT_CODE).to_bytes(4, "little")
@@ -248,7 +248,7 @@ def pull_tasks(self, kill_event):
                                                                                         pending_task_count))
 
             if time.time() > last_beat + self.heartbeat_period:
-                self.heartbeat()
+                self.heartbeat_to_incoming()
                 last_beat = time.time()
 
             if pending_task_count < self.max_queue_size and ready_worker_count > 0:
@@ -313,6 +313,7 @@ def push_results(self, kill_event):
         logger.debug("[RESULT_PUSH_THREAD] push poll period: {}".format(push_poll_period))
 
         last_beat = time.time()
+        last_result_beat = time.time()
         items = []
 
         while not kill_event.is_set():
@@ -325,6 +326,10 @@ def push_results(self, kill_event):
             except Exception as e:
                 logger.exception("[RESULT_PUSH_THREAD] Got an exception: {}".format(e))
 
+            if time.time() > last_result_beat + self.heartbeat_period:
+                last_result_beat = time.time()
+                items.append(pickle.dumps({'task_id': -1, 'heartbeat': True}))
+
             # If we have reached poll_period duration or timer has expired, we send results
             if len(items) >= self.max_queue_size or time.time() > last_beat + push_poll_period:
                 last_beat = time.time()


Test patch: