Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: usage_guide.md

Use `prefect-dask` to speed up code with parallelization. The `DaskTaskRunner` is a parallel task runner that submits tasks to the scheduler. By default, a temporary Dask cluster is created for the duration of the flow run. For example, this flow counts up to 10 in parallel (note that the output is not sequential). ```python import time from prefect import flow, task from prefect_dask import DaskTaskRunner @task def shout(number): time.sleep(0.5) print(f"#{number}") @flow(task_runner=DaskTaskRunner) def countto(highestnumber): for number in range(highest_number): shout.submit(number) if name == "main": count_to(10) ``` If you already have a Dask cluster running, either local or cloud hosted, you can provide the connection URL via an `address` argument. To configure your flow to use the `DaskTaskRunner`: Make sure the `prefect-dask` collection is installed as described earlier: `pip install prefect-dask`. In your flow code, import `DaskTaskRunner` from `prefectdask.taskrunners`. Assign it as the task runner when the flow is defined using the `task_runner=DaskTaskRunner` argument. For example, this flow uses the `DaskTaskRunner` configured to access an existing Dask cluster at `http://my-dask-cluster`. ```python from prefect import flow from prefectdask.taskrunners import DaskTaskRunner @flow(task_runner=DaskTaskRunner(address="http://my-dask-cluster")) def my_flow(): ... ``` `DaskTaskRunner` accepts the following optional parameters: | Parameter | Description | | | | | address | Address of a currently running Dask scheduler. | | cluster_class | The cluster class to use when creating a temporary Dask cluster. It can be either the full class name (for example, `"distributed.LocalCluster"`), or the class itself. | | clusterkwargs | Additional kwargs to pass to the `clusterclass` when creating a temporary Dask cluster. | | adaptkwargs | Additional kwargs to pass to `cluster.adapt` when creating a temporary Dask cluster. Note that adaptive scaling is only enabled if `adaptkwargs` are provided. | | client_kwargs | Additional kwargs to use when creating a . | !!! warning "Multiprocessing safety" Note that, because the `DaskTaskRunner` uses multiprocessing, calls to flows in scripts must be guarded with `if name == "main":` or you will encounter warnings and errors. If you don't provide the `address` of a Dask scheduler, Prefect creates a temporary local cluster automatically. The number of workers used is based on the number of cores available to your execution environment. The default provides a mix of processes and threads that should work well for most workloads. If you want to specify this explicitly, you can pass values for `nworkers` or `threadsperworker` to `clusterkwargs`. ```python DaskTaskRunner( clusterkwargs={"nworkers": 4, "threadsperworker": 2} ) ``` If you use a Dask collection, such as a `dask.DataFrame` or `dask.Bag`, to distribute the work across workers and achieve parallel computations, use one of the context managers `getdaskclient` or `getasyncdask_client`: ```python import dask from prefect import flow, task from prefectdask import DaskTaskRunner, getdask_client @task def compute_task(): with getdaskclient() as client: df = dask.datasets.timeseries("2000", "2001", partition_freq="4w") summary_df = df.describe().compute() return summary_df @flow(task_runner=DaskTaskRunner()) def dask_flow(): prefectfuture = computetask.submit() return prefect_future.result() dask_flow() ``` The context managers can be used the same way in both `flow` run contexts and `task` run contexts. !!! warning "Resolving futures in sync client" Note, by default, `daskcollection.compute()` returns concrete values while `client.compute(daskcollection)` returns Dask Futures. Therefore, if you call `client.compute`, you must resolve all futures before exiting out of the context manager by either: setting `sync=True` ```python with getdaskclient() as client: df = dask.datasets.timeseries("2000", "2001", partition_freq="4w") summary_df = client.compute(df.describe(), sync=True) ``` calling `result()` ```python with getdaskclient() as client: df = dask.datasets.timeseries("2000", "2001", partition_freq="4w") summary_df = client.compute(df.describe()).result() ``` For more information, visit the docs on . There is also an equivalent context manager for asynchronous tasks and flows:

```python import asyncio import dask from prefect import flow, task from prefectdask import DaskTaskRunner, getasyncdaskclient @task async def compute_task(): async with getasyncdask_client() as client: df = dask.datasets.timeseries("2000", "2001", partition_freq="4w") summary_df = await client.compute(df.describe()) return summary_df @flow(task_runner=DaskTaskRunner()) async def dask_flow(): prefectfuture = await computetask.submit() return await prefect_future.result() asyncio.run(dask_flow()) ``` !!! warning "Resolving futures in async client" With the async client, you do not need to set `sync=True` or call `result()`. However, you must `await client.compute(dask_collection)` before exiting the context manager. To invoke `compute` from the Dask collection, set `sync=False` and call `result()` before exiting out of the context manager: `await dask_collection.compute(sync=False)`. The `DaskTaskRunner` is capable of creating a temporary cluster using any of . This can be useful when you want each flow run to have its own Dask cluster, allowing for per-flow adaptive scaling. To configure, you need to provide a `cluster_class`. This can be: A string specifying the import path to the cluster class (for example, `"dask_cloudprovider.aws.FargateCluster"`) The cluster class itself A function for creating a custom cluster You can also configure `clusterkwargs`, which takes a dictionary of keyword arguments to pass to `clusterclass` when starting the flow run. For example, to configure a flow to use a temporary `dask_cloudprovider.aws.FargateCluster` with 4 workers running with an image named `my-prefect-image`: ```python DaskTaskRunner( clusterclass="daskcloudprovider.aws.FargateCluster", clusterkwargs={"nworkers": 4, "image": "my-prefect-image"}, ) ``` Multiple Prefect flow runs can all use the same existing Dask cluster. You might manage a single long-running Dask cluster (maybe using the Dask ) and configure flows to connect to it during execution. This has a few downsides when compared to using a temporary cluster (as described above): All workers in the cluster must have dependencies installed for all flows you intend to run. Multiple flow runs may compete for resources. Dask tries to do a good job sharing resources between tasks, but you may still run into issues. That said, you may prefer managing a single long-running cluster. To configure a `DaskTaskRunner` to connect to an existing cluster, pass in the address of the scheduler to the `address` argument: ```python DaskTaskRunner(address="tcp://...") ``` One nice feature of using a `DaskTaskRunner` is the ability to scale adaptively to the workload. Instead of specifying `n_workers` as a fixed number, this lets you specify a minimum and maximum number of workers to use, and the dask cluster will scale up and down as needed. To do this, you can pass `adapt_kwargs` to `DaskTaskRunner`. This takes the following fields: `maximum` (`int` or `None`, optional): the maximum number of workers to scale to. Set to `None` for no maximum. `minimum` (`int` or `None`, optional): the minimum number of workers to scale to. Set to `None` for no minimum. For example, here we configure a flow to run on a `FargateCluster` scaling up to at most 10 workers. ```python DaskTaskRunner( clusterclass="daskcloudprovider.aws.FargateCluster", adapt_kwargs={"maximum": 10} ) ``` Dask annotations can be used to further control the behavior of tasks. For example, we can set the of tasks in the Dask scheduler: ```python import dask from prefect import flow, task from prefectdask.taskrunners import DaskTaskRunner @task def show(x): print(x) @flow(task_runner=DaskTaskRunner()) def my_flow(): with dask.annotate(priority=-10): future = show(1) # low priority task with dask.annotate(priority=10): future = show(2) # high priority task ``` Another common use case is annotations: ```python import dask from prefect import flow, task from prefectdask.taskrunners import DaskTaskRunner @task def show(x): print(x) @flow( task_runner=DaskTaskRunner( clusterkwargs={"nworkers": 1, "resources": {"GPU": 1, "process": 1}} ) ) def my_flow(): with dask.annotate(resources={'GPU': 1}): future = show(0) # this task requires 1 GPU resource on a worker with dask.annotate(resources={'process': 1}): future = show(1) future = show(2) future = show(3) ```