Repository: Parsl/parsl

Issue:
Interchange KeyError due to too many heartbeat missed
Running this code snipset on Theta always fails.
```
import parsl
from parsl.providers import LocalProvider, CobaltProvider
from parsl.channels import LocalChannel
from parsl.launchers import SingleNodeLauncher, AprunLauncher
from parsl.config import Config
from parsl.executors import HighThroughputExecutor
from parsl.app.app import python_app  # , bash_app
from parsl.addresses import address_by_hostname
import time

theta_config = Config(
    executors=[
        HighThroughputExecutor(
            label="theta_htex",
            max_workers=1,
            address=address_by_hostname(),
            heartbeat_threshold=6,
            heartbeat_period=2,
            provider=CobaltProvider(
                queue='debug-flat-quad',
                account='CSC249ADCD01',
                launcher=AprunLauncher(overrides="-d 64"),
                worker_init='source activate parsl-issues',
                init_blocks=1,
                max_blocks=1,
                min_blocks=0,
                nodes_per_block=1,
                cmd_timeout=60,
                walltime='01:00:00',
           ),
        )
    ],
    strategy='simple',
)

dfk = parsl.load(theta_config)

@python_app
def double(x):
    return x * 2

if __name__ == '__main__':

    fut1 = [double(i) for i in range(10)]
    res1 = [f.result() for f in fut1]
    print("Task 1 result is {}".format(res1))
    print("Sleeping for 4 minutes")
    time.sleep(240)
    print("Finished sleeping")
    fut2 = [double(1) for i in range(10, 20)]
    res2 = [f.result() for f in fut2]
    print("Task 2 result is {}".format(res2))
    print("Done")
```

It would raise the exception below

```
Process Process-1:
Traceback (most recent call last):
  File "/home/zzli/anaconda3/envs/funcx/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/zzli/anaconda3/envs/funcx/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/zzli/anaconda3/envs/funcx/lib/python3.6/site-packages/parsl-0.8.0-py3.6.egg/parsl/executors/high_throughput/interchange.py", line 563, in starter
    ic.start()
  File "/home/zzli/anaconda3/envs/funcx/lib/python3.6/site-packages/parsl-0.8.0-py3.6.egg/parsl/executors/high_throughput/interchange.py", line 458, in start
    tasks_inflight = len(self._ready_manager_queue[manager]['tasks'])
KeyError: b'7c51d275c32f'
```

Took an initial scan of the code. It seems like this is because the manager is removed from the `ready_manager_queue` due to too many heartbeats missed in https://github.com/Parsl/parsl/blob/4cea89bdea96f44b74d5c6eb4030e6c6679a31f1/parsl/executors/high_throughput/interchange.py#L514
but is not removed from `interesting_managers`.



Discussion:
Is this a situation where managers are failing and the failing manager is being handled badly? Or do you think this should work if the pasted exception was fixed?

> Is this a situation where managers are failing and the failing manager is being handled badly? Or do you think this should work if the pasted exception was fixed?

I think it should work if we properly fix the exception.
I can confirm that changing the line https://github.com/Parsl/parsl/blob/4cea89bdea96f44b74d5c6eb4030e6c6679a31f1/parsl/executors/high_throughput/interchange.py#L514
to `self._ready_manager_queue[manager]['active'] = False` works.

I am testing now with a newly added line `interesting_managers.remove(manager)` after https://github.com/Parsl/parsl/blob/4cea89bdea96f44b74d5c6eb4030e6c6679a31f1/parsl/executors/high_throughput/interchange.py#L514
> Is this a situation where managers are failing and the failing manager is being handled badly? Or do you think this should work if the pasted exception was fixed?

The manager is killed by the strategy because there is no task for 2 minutes.

Resolving patch:
diff --git a/parsl/executors/high_throughput/interchange.py b/parsl/executors/high_throughput/interchange.py
--- a/parsl/executors/high_throughput/interchange.py
+++ b/parsl/executors/high_throughput/interchange.py
@@ -512,6 +512,8 @@ def start(self, poll_period=None):
                         self.results_outgoing.send(pkl_package)
                         logger.warning("[MAIN] Sent failure reports, unregistering manager")
                 self._ready_manager_queue.pop(manager, 'None')
+                if manager in interesting_managers:
+                    interesting_managers.remove(manager)
 
         delta = time.time() - start
         logger.info("Processed {} tasks in {} seconds".format(count, delta))


Test patch: