Repository: PrefectHQ/prefect

Issue:
Dask CancelledError when running flow in parallel
I managed to get the DaskExecutor to work with the simple example with inc/dec/add/list_sum. However, I'm running (pretty reliably) into what seems to be concurrency errors. When I run Dask with just one thread in a process, it works fine. However, when I give it multiple threads in one process, I'm consistently getting errors.
I reduced the example flow to the following:

```
with Flow("dask-example") as flow:
	incs = inc.map(x=range(100))
	decs = dec.map(x=range(100))
```

The error I'm getting when running with Dask, one worker process (+ one scheduler process) and multiple threads in that worker process:

```
[2019-04-18 15:06:18,588] INFO - prefect.TaskRunner | Task 'Constant[range]': Starting task run...
[2019-04-18 15:06:18,588] INFO - prefect.TaskRunner | Task 'Constant[range]': Starting task run...
... (more init for Task Contant) ...
[2019-04-18 15:06:18,598] INFO - prefect.TaskRunner | Task 'dec': Starting task run...
[2019-04-18 15:06:18,602] INFO - prefect.TaskRunner | Task 'inc': Starting task run...
[2019-04-18 15:06:18,826] DEBUG - prefect.TaskRunner | Task 'dec': Handling state change from Pending to Mapped
**[2019-04-18 15:06:18,856] INFO - prefect.TaskRunner | Unexpected error: CancelledError('run_fn-bc3361b2-937f-4d22-b02e-7bdeafe9f539-86',)
[2019-04-18 15:06:18,862] ERROR - prefect.TaskRunner | Task 'dec': unexpected error while running task: CancelledError('run_fn-bc3361b2-937f-4d22-b02e-7bdeafe9f539-86',)**
[2019-04-18 15:06:18,869] INFO - prefect.TaskRunner | Task 'dec': finished task run for task with final state: 'Failed'
[2019-04-18 15:06:18,897] DEBUG - prefect.TaskRunner | Task 'inc': Handling state change from Pending to Mapped
[2019-04-18 15:06:18,999] INFO - prefect.TaskRunner | Task 'dec[0]': Starting task run...
[2019-04-18 15:06:18,999] INFO - prefect.TaskRunner | Task 'dec[1]': Starting task run...
[2019-04-18 15:06:18,999] INFO - prefect.TaskRunner | Task 'dec[2]': Starting task run...
[2019-04-18 15:06:18,999] DEBUG - prefect.TaskRunner | Task 'dec[0]': Handling state change from Pending to Running
[2019-04-18 15:06:19,000] DEBUG - prefect.TaskRunner | Task 'dec[1]': Handling state change from Pending to Running
[2019-04-18 15:06:19,001] DEBUG - prefect.TaskRunner | Task 'dec[2]': Handling state change from Pending to Running
[2019-04-18 15:06:19,001] DEBUG - prefect.TaskRunner | Task 'dec[0]': Calling task.run() method...
[2019-04-18 15:06:19,002] DEBUG - prefect.TaskRunner | Task 'dec[0]': Handling state change from Running to Success
[2019-04-18 15:06:19,002] DEBUG - prefect.TaskRunner | Task 'dec[1]': Calling task.run() method...
[2019-04-18 15:06:19,002] DEBUG - prefect.TaskRunner | Task 'dec[2]': Calling task.run() method...
[2019-04-18 15:06:19,002] INFO - prefect.TaskRunner | Task 'dec[0]': finished task run for task with final state: 'Success'
... (further runs of tasks dec[x]/inc[x] which all seem to run fine, even though dec has already been cancelled) ...
```

If I include the add.map(x.incs, y.incs) in the flow after the inc/dec, then there's an additional error. However, this just seems to be a side-effect of the inc/dec failing before. Perhaps the error in this case should be a bit more meaningful, or execution just stopped. At least the error shouldn't be unexpected I think, as the input task failed already.

```
[2019-04-18 14:47:14,581] INFO - prefect.TaskRunner | Task 'add': Starting task run...
[2019-04-18 14:47:14,581] INFO - prefect.TaskRunner | Unexpected error: TypeError("'CancelledError' object does not support indexing",)
[2019-04-18 14:47:14,581] DEBUG - prefect.TaskRunner | Task 'add': Handling state change from Pending to Failed
[2019-04-18 14:47:14,581] DEBUG - prefect.TaskRunner | Task 'add': task has been mapped; ending run.
```

However my question is more related to how I can debug this properly. I've been trying to run it with various of the suggested debug settings and trying to dig into the logs to see if I could find anything useful, but so far unfortunately fruitless. I'd be grateful for any suggestions.


Discussion:
I also looked in the Dask code for various locations where a CancelledError could be thrown, as well as the locations in Prefect where these errors could be caught and logged. The logging occurs in the task_runner.py, while the exception is most likely thrown in client.py of Dask. However, there are various reasons why this error may be thrown and I'm unsure which one of them is actually throwing it. I tried running with raise_on_exception to true in order to get more information about the exception than just the repr, but in my case this setting didn't seem to do anything.
I'm not 100% sure but I have a suspicion - mapping with the `DaskExecutor` is implemented via a `worker_client`.  I wonder if the `worker_client` behaves poorly with a `YarnCluster`?  Might be worth raising an issue on `dask-yarn` to ask.
@fvannee I've replicated your flow and what I believe is a similar setup (one scheduler, one worker with 10 threads) but so far I'm unable to replicate the error. I feel like I saw errors similar to this during testing when the client would lose connection to the scheduler early, or garbage collect the dask futures aggressively, but we've added protections against that beyond Dask's usual ones.

@cicdw If you look at the first time the error is thrown, it's on a task with no "[1]" after it, so I believe it's prior to mapping.
Also, I think you are correct, your second error is indeed due to receiving the `CanceledError` object in the downstream task.

This (can) be a feature of Prefect -- failed tasks can still return results to tasks that have triggers allowing them to run when upstream tasks failed. By default, failed tasks return the error that caused them to fail, but you could replace that with anything you wanted. We often use that to introspect errors -- if a task fails, the following task can have a `all_failed` trigger and it'll receive the traceback from the first task!

Debugging this is tricky -- I have a feeling this is the Dask scheduler / client communication breaking down. Prefect tries to protect against this by keeping references to futures and using `fire_and_forget`, but I'll keep thinking if there's anywhere else it might be slipping through.
Thanks to both of you for the help so far! I did some more investigation on the dask-yarn, but couldn't really find a reason why this would be causing it. Therefore, I tried running it without it on just Dask and the issue occurs there as well. Therefore, I think it has indeed something to do with Dask parallelization itself and/or Prefect's handling of this.
I managed to break it down into an example that reproduces 100% on my machine. I hope this replicates on your setup as well. I also have some more detailed logging available.

Start a scheduler and a worker with 3 threads:
```
dask-scheduler > scheduler.log 2>&1
dask-worker --nanny --nthreads 3 127.0.0.1:8786 > worker.log 2>&1

```
Run the Prefect example:
`python3 dask_test.py`

The *first* run of this *always* fails on my machine. However, if I don't restart the Dask scheduler/worker, second run and so on will run fine. As soon as I restart my Dask scheduler/worker, the first run after that fails again.

[dask_test.py.txt](https://github.com/PrefectHQ/prefect/files/3097513/dask_test.py.txt)
[scheduler.log](https://github.com/PrefectHQ/prefect/files/3097514/scheduler.log)
[worker.log](https://github.com/PrefectHQ/prefect/files/3097515/worker.log)

Also, interestingly enough the following sequences can also be reproduced 100% on my setup. I'm a bit in doubt why, because some of these sequences are hard to relate to the issue we're seeing:
- start scheduler, start worker, run flow (fail), run flow (success), run flow (success) (etc. success from here)
- start scheduler, start worker, restart worker, run flow (fail)
- start scheduler, start worker, run flow (fail), restart worker, run flow (fail)
- start scheduler, start worker, run flow (fail), run flow (success), restart worker, run flow (success)
- start scheduler, start worker, run flow (fail), run flow (success), restart worker/scheduler, run flow (fail)

With or without nanny doesn't seem to make much of a difference, although I have seen a few cases where 'start scheduler, start worker with --no-nanny, run flow' succeeds the first time. However, it also fails very often.
@fvannee thanks so much for the reproducible example, I'm able to reproduce! I'll try to dig into this and will update as I proceed.
Progress!  I can reproduce this issue without using Prefect by mocking up our pipeline; if you start a `dask-scheduler` and a `dask-worker` in the same way as above, and run: 
```python
from distributed import Client, fire_and_forget, worker_client

def inc(x):
    return x + 1

def dec(x):
    return x - 1

def mapper(fn, arr):
    with worker_client(separate_thread=True) as client:
        futures = client.map(fn, arr, pure=False)
    fire_and_forget(futures)
    return futures

def waiter(futs):
    with worker_client(separate_thread=True) as client:
        return client.gather(futs)

def pipeline(fn, arr):
    futs = mapper(fn, arr)
    return waiter(futs)

with Client(address='localhost:8786') as client:
    f = client.submit(pipeline, inc, list(range(100)))
    fire_and_forget(f) # this appears to be irrelevant
    g = client.submit(pipeline, dec, list(range(100)))
    fire_and_forget(g) # this appears to be irrelevant
    val = client.gather([f,g])
```
you should get a prominent `CancelledError`.  We probably need to revisit our use of `fire_and_forget`
Just to note, I also ran this snippet and got the same `CancelledError` 👍 
Nice work! Could this be an issue with Dask itself then? Because apparently it only occurs the first time after we start the scheduler. Does Dask do something different the second run?
Oh, I tried to run your snippet, but I'm actually unable to reproduce it with just that snippet. This looks very strange to me. Should I see the CancelledError in the output of that script itself, or in the Dask worker log? It looks like it runs to completion for me:

```
2019-04-19 21:04:31,150 - distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2
2019-04-19 21:04:31,150 - distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms
2019-04-19 21:04:31,153 - distributed.comm.tcp - DEBUG - Setting TCP keepalive: nprobes=10, idle=10, interval=2
2019-04-19 21:04:31,153 - distributed.comm.tcp - DEBUG - Setting TCP user timeout: 30000 ms
2019-04-19 21:04:31,161 - distributed.client - DEBUG - Started scheduling coroutines. Synchronized
2019-04-19 21:04:31,165 - distributed.client - DEBUG - Submit pipeline(...), pipeline-7ce51c2724f8710a63fbc8d5c0574b40
2019-04-19 21:04:31,167 - distributed.client - DEBUG - Submit pipeline(...), pipeline-569445b5f43d75455da7b05ce6c7c1c8
2019-04-19 21:04:31,168 - distributed.client - DEBUG - Waiting on futures to clear before gather
2019-04-19 21:04:31,456 - distributed.client - DEBUG - Client receives message {'op': 'key-in-memory', 'key': 'pipeline-7ce51c2724f8710a63fbc8d5c0574b40', 'type': b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08builtins\x94\x8c\x04list\x94\x93\x94.'}
2019-04-19 21:04:31,488 - distributed.client - DEBUG - Client receives message {'op': 'key-in-memory', 'key': 'pipeline-569445b5f43d75455da7b05ce6c7c1c8', 'type': b'\x80\x04\x95\x15\x00\x00\x00\x00\x00\x00\x00\x8c\x08builtins\x94\x8c\x04list\x94\x93\x94.'}
2019-04-19 21:04:31,504 - distributed.client - DEBUG - Release key pipeline-7ce51c2724f8710a63fbc8d5c0574b40
2019-04-19 21:04:31,505 - distributed.client - DEBUG - Release key pipeline-569445b5f43d75455da7b05ce6c7c1c8

```
@fvannee the `CancelledError` should occur in your main process.  Here's something super interesting: try recreating _without_ writing the logs to a file.  I.e., spin up your dask cluster via:
```
dask-scheduler
dask-worker --nanny --nthreads 3 127.0.0.1:8786
```
and then try running the snippet.

It seems that when I write logs to a file, the error _doesn't_ occur. 🤔 
My experience with @cicdw's script:

- if all three processes (scheduler, worker, python) are running it for the first time, the python script results in a `CancelledError`. If any of them have been run before, it works.

- if I comment out the `fire_and_forget` in the `mapper` function, I can't get the error (it works)

Resolving patch:
diff --git a/src/prefect/engine/executors/dask.py b/src/prefect/engine/executors/dask.py
--- a/src/prefect/engine/executors/dask.py
+++ b/src/prefect/engine/executors/dask.py
@@ -142,6 +142,7 @@ def map(self, fn: Callable, *args: Any) -> List[Future]:
         elif self.is_started:
             with worker_client(separate_thread=True) as client:
                 futures = client.map(fn, *args, pure=False)
+                return client.gather(futures)
         else:
             raise ValueError("This executor has not been started.")
 


Test patch: