Repository: Parsl/parsl

Issue:
KeyError in HighThroughputExecutor `submit` (under high concurrent submission load?)
**Describe the bug**

I've seen this in CI on a PR that changes only documentation (#2024). I'm assuming this PR is a heisenbug in HTEX, not related to the documentation change. 

This happened in the fibonacci recursive test, which probably pushes on htex concurrency quite hard: it's likely that tasks are being launched concurrently from multiple threads (due to the use of join apps) and the tasks are all extremely short (they wrap `+`). 

```
DEBUG    parsl.executors.high_throughput.executor:executor.py:562 Pushing function <function add at 0x7f4393a7e6a8> to queue with args ()
ERROR    parsl.dataflow.dflow:dflow.py:317 Task 374 failed after 0 retry attempts
Traceback (most recent call last):
  File "/home/travis/build/Parsl/parsl/parsl/dataflow/dflow.py", line 286, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
  File "/home/travis/build/Parsl/parsl/parsl/dataflow/dflow.py", line 439, in _unwrap_remote_exception_wrapper
    result = future.result()
  File "/opt/python/3.6.7/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/opt/python/3.6.7/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/travis/build/Parsl/parsl/parsl/dataflow/dflow.py", line 493, in launch_if_ready
    task_id, task_record['func'], *new_args, **kwargs)
  File "/home/travis/build/Parsl/parsl/parsl/dataflow/dflow.py", line 582, in launch_task
    exec_fu = executor.submit(executable, self.tasks[task_id]['resource_specification'], *args, **kwargs)
  File "/home/travis/build/Parsl/parsl/parsl/executors/high_throughput/executor.py", line 579, in submit
    return self.tasks[task_id]
KeyError: 116
```

A clear and concise description of what the bug is.

**To Reproduce**
heisenbug.

**Expected behavior**
This failure shouldn't happen. (it doesn't happen most of the time)

**Environment**
CI
PR #2024 on top of `master` 3a03ec4b4b2ad013ac79ab8e185be5ed54b94068




Discussion:
My gut feeling to start with is/was that duplicate htex task IDs are being allocated for separate tasks.

This code in htex submit is not thread-safe, because i) `+=` is not atomic and ii) the value of the counter taken for the task_id can be changed after the increment - see https://julien.danjou.info/atomic-lock-free-counters-in-python/

But, executor.submit calls are supposed to be serialised by the DFK, by the use of a lock to protect against this, by this code:
```
        with self.submitter_lock:
            exec_fu = executor.submit(executable, self.tasks[task_id]['resource_specification'], *args, **kwargs)
```



Resolving patch:
diff --git a/parsl/executors/high_throughput/executor.py b/parsl/executors/high_throughput/executor.py
--- a/parsl/executors/high_throughput/executor.py
+++ b/parsl/executors/high_throughput/executor.py
@@ -561,7 +561,8 @@ def submit(self, func, resource_specification, *args, **kwargs):
             args_to_print = tuple([arg if len(repr(arg)) < 100 else (repr(arg)[:100] + '...') for arg in args])
         logger.debug("Pushing function {} to queue with args {}".format(func, args_to_print))
 
-        self.tasks[task_id] = Future()
+        fut = Future()
+        self.tasks[task_id] = fut
 
         try:
             fn_buf = pack_apply_message(func, args, kwargs,
@@ -576,7 +577,7 @@ def submit(self, func, resource_specification, *args, **kwargs):
         self.outgoing_q.put(msg)
 
         # Return the future
-        return self.tasks[task_id]
+        return fut
 
     @property
     def scaling_enabled(self):


Test patch: