Repository: PrefectHQ/prefect

Issue:
Prefect mapping on Dask can sometimes hang indefinitely
## Archived from the [Prefect Public Slack Community](https://join.slack.com/t/prefect-public/shared_invite/enQtNzE5OTU3OTQwNzc1LTQ5M2FkZmQzZjI0ODg1ZTBmOTc0ZjVjYWFjMWExZDAyYzBmYjVmMTE1NTQ1Y2IxZTllOTc4MmI3NzYxMDlhYWU)

**thisishaoyu**: Dask question here:
I am new to distributed systems, and want to run a large number of tasks (N = 10k+) in parallel. Afaik, Prefect+Dask is scalable to 10k+ parallel tasks (although I know it's ideal to batch many small tasks to few longer-running tasks if possible).

However, trying different small toy tasks (and various combinations of `--nprocs` `--nthreads` on workers), it seems that consistently
- Running Client() locally starts immediately, as expected.
- Running with one worker server, starting takes much longer as N increases
For large N:
```
[2019-09-30 22:59:15,781] INFO - prefect.TaskRunner | Task 'stage_0': Starting task run...
distributed.utils_perf - INFO - full garbage collection released 561.93 MB from 0 reference cycles (threshold: 10.00 MB)
distributed.core - INFO - Event loop was unresponsive in Worker for 34.31s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
```
- Running on multiple servers, the Dask scheduler also takes much way longer than the expected ~1ms overhead/task, before any activity happens (on the client servers or the Dask scheduler dashboard).

Why is scheduling taking so long? How could I adjust scheduling policy (<https://distributed.dask.org/en/latest/scheduling-policies.html>) to speed up running large number of small tasks?

**chris**: Hi Jeff - are you using mapping to create the 10k tasks?

**thisishaoyu**: Yes

**thisishaoyu**: Actually, I am using the `create_product` earlier with e.g. len(params1) = len(params2) = 100 --&gt; len(prod) = 10k

**chris**: and what is `stage_0` which is hanging?

**thisishaoyu**: 
```
[2019-09-30 22:59:15,781] INFO - prefect.TaskRunner | Task 'stage_0': Starting task run...
distributed.utils_perf - INFO - full garbage collection released 561.93 MB from 0 reference cycles (threshold: 10.00 MB)
distributed.core - INFO - Event loop was unresponsive in Worker for 34.31s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
```

**thisishaoyu**: is the current log output

**chris**: right, but what is `stage_0` doing?  I’m a bit surprised to see Dask complaining before any tasks are even run - I think I might be missing some context

**thisishaoyu**: `create_product` ran immediately before, and it ran quickly and successfully

**thisishaoyu**: `stage_0` currently generates a small random Pandas dataframe and saves it to a tmp directory in file titled `f"{param1}_{param2}.csv"`. Space isn't an issue.

**chris**: ok ok I think I’m following now -&gt; it appears that `stage_0` begins its run but freezes, is that correct?

**thisishaoyu**: It appears so from the log. There is no activity on the Dask dashboard.
If I recall correctly, with smaller N, it also freezes (for a much shorter while) before spitting out output.

**chris**: ok yea, this is related to an issue I’ve been digging into in Prefect Cloud as well; I haven’t been able to pinpoint the root cause yet,  but it seems that mapping from a worker (which is what Prefect does) causes a bottleneck somewhere in Dask

**thisishaoyu**: Any workarounds for the parallel execution of a large number of tasks?

**chris**: ultimately expanding your workers’ memory allocation is the only thing i’ve found that consistently works; I’ve been running workflows in Prefect Cloud that map over ~12-24k tasks and while they do eventually  run I also run into these “zombie” tasks regularly.  I’ll try to prioritize fixing this because I agree it is very frustrating.  I suspect I’ll have to fundamentally change how we move data around through the Dask Executor

**thisishaoyu**: Which parameters expand worker memory allocation?

**thisishaoyu**: `--memory-limit` in Dask?

**chris**: oh my apologies, I’m thinking in kubernetes --&gt; I usually have to increase the size of my nodes

**chris**: <@ULVA73B9P> archive “Prefect mapping on Dask can sometimes hang indefinitely”




Discussion:
I am actively investigating this 
I think we can solve this bottleneck when mapping first begins with this pattern: https://github.com/dask/distributed/issues/3108#issuecomment-537693221

Resolving patch:
diff --git a/src/prefect/engine/executors/dask.py b/src/prefect/engine/executors/dask.py
--- a/src/prefect/engine/executors/dask.py
+++ b/src/prefect/engine/executors/dask.py
@@ -156,14 +156,19 @@ def map(self, fn: Callable, *args: Any, **kwargs: Any) -> List[Future]:
         if not args:
             return []
 
+        apply = lambda f, *args, **kwargs: f(*args, **kwargs)
+        N = len(args[0])
+
         dask_kwargs = self._prep_dask_kwargs()
         kwargs.update(dask_kwargs)
 
         if self.is_started and hasattr(self, "client"):
-            futures = self.client.map(fn, *args, **kwargs)
+            func = self.client.scatter(fn)
+            futures = self.client.map(apply, [func] * N, *args, **kwargs)
         elif self.is_started:
             with worker_client(separate_thread=True) as client:
-                futures = client.map(fn, *args, **kwargs)
+                func = client.scatter(fn)
+                futures = client.map(apply, [func] * N, *args, **kwargs)
                 return client.gather(futures)
         else:
             raise ValueError("This executor has not been started.")


Test patch: