Repository: Parsl/parsl

Issue:
Race condition in handling of close-in-time dependency failures
**Describe the bug**
If two dependencies for an app fail close in time, then the `launch_if_ready` dependency handling code can fire multiple times roughly simultanously.

This results in at least an invalid state exception in parsl.log - I haven't investigated further to see if more bad things happen.

**To Reproduce**
A test program follows, along with a strategically placed sleep statement in the DFK to increase the chances of the race condition happening.

```
import parsl
import time

@parsl.python_app
def failapp():
  import time
  time.sleep(1)
  raise ValueError("Deliberate failure")

@parsl.python_app
def merge(a,b):
  return True

parsl.load()

f1 = failapp()
f2 = failapp()
f3 = merge(f1,f2)
print(f3.exception())

time.sleep(20)
```

and 

```
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
index 05d49cca5e..e24e4ed625 100644
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -496,6 +496,9 @@ class DataFlowKernel(object):
         or callback.
         """
         task_id = task_record['id']
+        if task_id == 2:
+          logger.debug("BENC: sleeping because task id 2, fiddling for races")
+          time.sleep(1)
         if self._count_deps(task_record['depends']) == 0:
 
             # We can now launch *task*
```

This results in dependency failure handling happening twice, once correctly, and the second time, two errors - an invalid state exception (which I would expect from this race) and a `TypeError` which I have not investigated.

```
2022-02-15 11:30:12.495 parsl.app.errors:126 [DEBUG]  Reraising exception of type <class 'TypeError'>
2022-02-15 11:30:12.498 parsl.dataflow.dflow:288 [DEBUG]  Task 2 try 0 failed
2022-02-15 11:30:12.498 parsl.dataflow.dflow:332 [ERROR]  Task 2 failed after 0 retry attempts
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 285, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 470, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/home/benc/parsl/src/parsl/parsl/app/errors.py", line 131, in reraise
    reraise(t, v, tb)
  File "/home/benc/parsl/virtualenv-3.9/lib/python3.9/site-packages/six.py", line 718, in reraise
    raise value.with_traceback(tb)
  File "/home/benc/parsl/src/parsl/parsl/app/errors.py", line 159, in wrapper
    return func(*args, **kwargs)  # type: ignore
TypeError: merge() missing 2 required positional arguments: 'a' and 'b'
2022-02-15 11:30:12.499 concurrent.futures:331 [ERROR]  exception calling callback for <Future at 0x7f1a61d5e580 state=finished returned RemoteExceptionWrapper>
Traceback (most recent call last):
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 285, in handle_exec_update
    res = self._unwrap_remote_exception_wrapper(future)
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 470, in _unwrap_remote_exception_wrapper
    result.reraise()
  File "/home/benc/parsl/src/parsl/parsl/app/errors.py", line 131, in reraise
    reraise(t, v, tb)
  File "/home/benc/parsl/virtualenv-3.9/lib/python3.9/site-packages/six.py", line 718, in reraise
    raise value.with_traceback(tb)
  File "/home/benc/parsl/src/parsl/parsl/app/errors.py", line 159, in wrapper
    return func(*args, **kwargs)  # type: ignore
TypeError: merge() missing 2 required positional arguments: 'a' and 'b'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 329, in _invoke_callbacks
    callback(self)
  File "/home/benc/parsl/src/parsl/parsl/dataflow/dflow.py", line 339, in handle_exec_update
    task_record['app_fu'].set_exception(e)
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 540, in set_exception
    raise InvalidStateError('{}: {!r}'.format(self._state, self))
concurrent.futures._base.InvalidStateError: FINISHED: <AppFuture at 0x7f1a61d45eb0 state=finished raised DependencyError>
```

**Expected behavior**
Two dependencies failing close in time should behave the same as separated-in-time dependency failures. The above strange messages (and potentially other misbehaviour) should not happen.

**Environment**

master c61023d87e73cbd4190bd1ce03fefab22606767a

python 3.9



Discussion:
I've dug into this a bit more. There is a race condition in `launch_if_ready` in the DFK.

`launch_if_ready` is called when each dependency `Future` completes.

So it should work like this:

say task T has three dependency futures, `A, B, C`, then three callbacks are registered to invoke `launch_if_ready`, one on each future.

When `launch_if_ready` runs, it checks the list of dependency futures to see if they are all completed. If not, then it does nothing, and if they are all completed, then it launches or processes failures for T.

So an example of how this is meant to work (and how it usually works):

B completes
`launch_if_ready` runs, and sees `B.done()` but not `A.done()` and not `C.done()`. So no-op

later, A completes
`launch_if_ready` runs, and sees `B.done()` and `A.done` but not `C.done()`. So no-op

later, C completes
`launch_if_ready` runs, and sees `B.done()` and `A.done` and `C.done()`. So process launch or failure for `T`.

However, in the circumstances leading to this issue report, this is not how things happen. This issue is triggered when two dependencies complete close in time - exacerbated by the high concurrency of the local thread executor, but I think this can happen without that:

B completes
`launch_if_ready` runs, and sees `B.done()` but not `A.done()` and not `C.done()`. So no-op

later, A completes, and soon after C completes. [this is the concurrency that is not handled]
`launch_if_ready` runs (due to A callback), and sees `B.done()` and `A.done` and `C.done()`. So either failure handling or launch of `T` happens. This makes some changes to the task record: specifically `sanitize_and_wrap` maps values out of dependency Futures, and removes any failed Futures from `args` and `kwargs`. [this is referred to as STEP X below]

`launch_if_ready` runs again, possibly concurrently with the above execution, or possibly afterwards. It sees no dependency Futures at all (because they were mapped away by the previous `launch_if_ready` execution of `sanitize_and_wrap`). So even if there previously were dependency errors, `T` will be launched.

In the test case, this leads to:
```
TypeError: merge() missing 2 required positional arguments: 'a' and 'b'
```
because the two arguments were Futures containing exceptions and were therefore removed from the args list by the first execution of `sanitize_and_wrap`.
This failure then leads to an attempt to set `T`'s future to a failure. However `T`'s Future is already in a failure state, from the first round of dependency processing in STEP X, and so this generates `concurrent.futures._base.InvalidStateError`

I don't think any of this second attempt to run is made visible to end users observing T's Future, because they would only see the (correct) dependency exception set in STEP X. It only appears in log messages.

I haven't investigated what happens when two launches like this happen when dependencies are valid - eg do two copies of T get launched?

I think the solution is to make `launch_if_ready` handle that case that it may already have launched the task, and is being called *after* that. At present, it assumes that is only being called before launch time, or is the final launching invocation.

Resolving patch:
diff --git a/parsl/dataflow/dflow.py b/parsl/dataflow/dflow.py
--- a/parsl/dataflow/dflow.py
+++ b/parsl/dataflow/dflow.py
@@ -243,9 +243,7 @@ def _create_task_log_info(self, task_record):
         return task_log_info
 
     def _count_deps(self, depends):
-        """Internal.
-
-        Count the number of unresolved futures in the list depends.
+        """Count the number of unresolved futures in the list depends.
         """
         count = 0
         for dep in depends:
@@ -497,38 +495,50 @@ def launch_if_ready(self, task_record):
         ready to run - launch_if_ready will not incorrectly launch that
         task.
 
+        It is also not an error to call launch_if_ready on a task that has
+        already been launched - launch_if_ready will not re-launch that
+        task.
+
         launch_if_ready is thread safe, so may be called from any thread
         or callback.
         """
+        exec_fu = None
+
         task_id = task_record['id']
-        if self._count_deps(task_record['depends']) == 0:
+        with task_record['task_launch_lock']:
+
+            if task_record['status'] != States.pending:
+                logger.debug(f"Task {task_id} is not pending, so launch_if_ready skipping")
+                return
+
+            if self._count_deps(task_record['depends']) != 0:
+                logger.debug(f"Task {task_id} has outstanding dependencies, so launch_if_ready skipping")
+                return
+
+            # We can now launch the task or handle any dependency failures
 
-            # We can now launch *task*
-            new_args, kwargs, exceptions_tids = self.sanitize_and_wrap(task_record['args'],
-                                                                       task_record['kwargs'])
+            new_args, kwargs, exceptions_tids = self._unwrap_futures(task_record['args'],
+                                                                     task_record['kwargs'])
             task_record['args'] = new_args
             task_record['kwargs'] = kwargs
+
             if not exceptions_tids:
                 # There are no dependency errors
-                exec_fu = None
-                # Acquire a lock, retest the state, launch
-                with task_record['task_launch_lock']:
-                    if task_record['status'] == States.pending:
-                        try:
-                            exec_fu = self.launch_task(
-                                task_record, task_record['func'], *new_args, **kwargs)
-                            assert isinstance(exec_fu, Future)
-                        except Exception as e:
-                            # task launched failed somehow. the execution might
-                            # have been launched and an exception raised after
-                            # that, though. that's hard to detect from here.
-                            # we don't attempt retries here. This is an error with submission
-                            # even though it might come from user code such as a plugged-in
-                            # executor or memoization hash function.
-
-                            logger.debug("Got an exception launching task", exc_info=True)
-                            exec_fu = Future()
-                            exec_fu.set_exception(e)
+                try:
+                    exec_fu = self.launch_task(
+                        task_record, task_record['func'], *new_args, **kwargs)
+                    assert isinstance(exec_fu, Future)
+                except Exception as e:
+                    # task launched failed somehow. the execution might
+                    # have been launched and an exception raised after
+                    # that, though. that's hard to detect from here.
+                    # we don't attempt retries here. This is an error with submission
+                    # even though it might come from user code such as a plugged-in
+                    # executor or memoization hash function.
+
+                    logger.debug("Got an exception launching task", exc_info=True)
+                    exec_fu = Future()
+                    exec_fu.set_exception(e)
             else:
                 logger.info(
                     "Task {} failed due to dependency failure".format(task_id))
@@ -541,19 +551,19 @@ def launch_if_ready(self, task_record):
                 exec_fu.set_exception(DependencyError(exceptions_tids,
                                                       task_id))
 
-            if exec_fu:
-                assert isinstance(exec_fu, Future)
-                try:
-                    exec_fu.add_done_callback(partial(self.handle_exec_update, task_record))
-                except Exception:
-                    # this exception is ignored here because it is assumed that exception
-                    # comes from directly executing handle_exec_update (because exec_fu is
-                    # done already). If the callback executes later, then any exception
-                    # coming out of the callback will be ignored and not propate anywhere,
-                    # so this block attempts to keep the same behaviour here.
-                    logger.error("add_done_callback got an exception which will be ignored", exc_info=True)
+        if exec_fu:
+            assert isinstance(exec_fu, Future)
+            try:
+                exec_fu.add_done_callback(partial(self.handle_exec_update, task_record))
+            except Exception:
+                # this exception is ignored here because it is assumed that exception
+                # comes from directly executing handle_exec_update (because exec_fu is
+                # done already). If the callback executes later, then any exception
+                # coming out of the callback will be ignored and not propate anywhere,
+                # so this block attempts to keep the same behaviour here.
+                logger.error("add_done_callback got an exception which will be ignored", exc_info=True)
 
-                task_record['exec_fu'] = exec_fu
+            task_record['exec_fu'] = exec_fu
 
     def launch_task(self, task_record, executable, *args, **kwargs):
         """Handle the actual submission of the task to the executor layer.
@@ -719,7 +729,7 @@ def check_dep(d):
 
         return depends
 
-    def sanitize_and_wrap(self, args, kwargs):
+    def _unwrap_futures(self, args, kwargs):
         """This function should be called when all dependencies have completed.
 
         It will rewrite the arguments for that task, replacing each Future


Test patch: