Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: dask-ray-task-runners.md

description: Learn how to use the Prefect Dask and Ray task runners for parallel or distributed task execution. tags: tasks task runners flow configuration parallel execution distributed execution Dask Ray search: boost: 2 Task runners provide an execution environment for tasks. In a flow decorator, you can specify a task runner to run the tasks called in that flow. The default task runner is the . !!! note "Use `.submit` to run your tasks asynchronously" To run tasks asynchronously use the `.submit` method when you call them. If you call a task as you would normally in Python code it will run synchronously, even if you are calling the task within a flow that uses the `ConcurrentTaskRunner`, `DaskTaskRunner`, or `RayTaskRunner`. Many real-world data workflows benefit from truly parallel, distributed task execution. For these use cases, the following Prefect-developed task runners for parallel task execution may be installed as . runs tasks requiring parallel execution using . runs tasks requiring parallel execution using . These task runners can spin up a local Dask cluster or Ray instance on the fly, or let you connect with a Dask or Ray environment you've set up separately. This enables you to take advantage of massively parallel computing environments. Use Dask or Ray in your flows to choose the execution environment that fits your particular needs. !!! note "Remote storage" To ensure that tasks executing in Dask or Ray have access to task result storage, particularly when accessing a Dask or Ray instance outside of your execution environment, configure . Configure a task runner for a flow. Start with the . This task runner runs all tasks synchronously and is useful as a debugging tool in conjunction with async code. Import the `SequentialTaskRunner`, specify a `task_runner` on the flow, and call the tasks with `.submit()`. ```python hl_lines="2 12 15 16" from prefect import flow, task from prefect.task_runners import SequentialTaskRunner @task def say_hello(name): print(f"hello {name}") @task def say_goodbye(name): print(f"goodbye {name}") @flow(task_runner=SequentialTaskRunner()) def greetings(names): for name in names: say_hello.submit(name) say_goodbye.submit(name) if name == "main": greetings(["arthur", "trillian", "ford", "marvin"]) ``` Save this code as `sequential_flow.py` and run it. <div class="terminal"> ```bash python sequential_flow.py ``` </div> If you take out the log messages and just look at the printed output of the tasks, you see they're executed in sequential order: <div class="terminal"> ``` bash hello arthur goodbye arthur hello trillian goodbye trillian hello ford goodbye ford hello marvin goodbye marvin ``` </div> This simple flow doesn't gain much from parallel execution, but it illustrates how simple it is to take advantage of the . Configure your flow to use the `DaskTaskRunner`: Ensure the `prefect-dask` collection is installed by running `pip install prefect-dask`. In your flow code, import `DaskTaskRunner` from `prefectdask.taskrunners`. Assign it as the task runner when the flow is defined using the `task_runner=DaskTaskRunner` argument. Use the `.submit` method when calling task-decorated functions. This is the same flow as above, with a few minor changes to use `DaskTaskRunner` where we previously configured `SequentialTaskRunner`. Install `prefect-dask`, make these changes, then save the updated code as `dask_flow.py`. ```python hl_lines="2 12 18" from prefect import flow, task from prefectdask.taskrunners import DaskTaskRunner @task def say_hello(name): print(f"hello {name}") @task def say_goodbye(name): print(f"goodbye {name}") @flow(task_runner=DaskTaskRunner()) def greetings(names): for name in names: say_hello.submit(name) say_goodbye.submit(name) if name == "main": greetings(["arthur", "trillian", "ford", "marvin"]) ``` Note that, because you're using `DaskTaskRunner` in a script, you must use `if name == "main":` or you'll see warnings and

Run `dask_flow.py`. If you get a warning about accepting incoming network connections, that's okay - everything is local in this example. <div class="terminal"> ```bash python dask_flow.py ``` </div> `DaskTaskRunner` automatically creates a local Dask cluster, then starts executing all of the task runs in parallel. The results do not return in the same order as the sequential code above. Abbreviated output: <div class="terminal"> ```bash goodbye marvin hello arthur goodbye ford hello trillian ``` </div> Notice what happens if you do not use the `submit` method when calling tasks: ```python from prefect import flow, task from prefectdask.taskrunners import DaskTaskRunner @task def say_hello(name): print(f"hello {name}") @task def say_goodbye(name): print(f"goodbye {name}") @flow(task_runner=DaskTaskRunner()) def greetings(names): for name in names: say_hello(name) say_goodbye(name) if name == "main": greetings(["arthur", "trillian", "ford", "marvin"]) ``` Run the script: <div class="terminal"> ```bash python dask_flow.py ``` </div> Once again, the tasks run sequentially. Here's the output with logs removed. <div class="terminal"> ```bash hello arthur goodbye arthur hello trillian goodbye trillian hello ford goodbye ford hello marvin goodbye marvin ``` </div> The task runs are not submitted to the `DaskTaskRunner`; instead, they run sequentially. To flexibly apply the appropriate task runner for your workflow, use the same flow as above, with a few minor changes to use the where you previously configured `DaskTaskRunner`. Configure your flow to use the `RayTaskRunner`: Install `prefect-ray` into your environment with `pip install -U prefect-ray`. In your flow code, import `RayTaskRunner` from `prefectray.taskrunners`. Specify the task runner when the flow is defined using the `task_runner=RayTaskRunner` argument. !!! warning "Ray environment limitations" While we're excited about parallel task execution through Ray to Prefect, there are some inherent limitations with Ray: Support for Python 3.11 is . Ray's Windows support is currently in beta. See the for further compatibility information. Save this code in `ray_flow.py`. ```python hl_lines="2 12" from prefect import flow, task from prefectray.taskrunners import RayTaskRunner @task def say_hello(name): print(f"hello {name}") @task def say_goodbye(name): print(f"goodbye {name}") @flow(task_runner=RayTaskRunner()) def greetings(names): for name in names: say_hello.submit(name) say_goodbye.submit(name) if name == "main": greetings(["arthur", "trillian", "ford", "marvin"]) ``` Now run `ray_flow.py` `RayTaskRunner`, which automatically creates a local Ray instance, then immediately starts executing all of the tasks in parallel. If you have an existing Ray instance, you can provide the address as an argument to run tasks in the instance. See for details. Many workflows include a variety of tasks, and not all of them benefit from parallel execution. You'll most likely want to use the Dask or Ray task runners and spin up their respective resources only for those tasks that need them. Because task runners are specified on flows, you can assign different task runners to tasks by using to organize those tasks. This example uses the same tasks as the previous examples, but on the parent flow, `greetings()`, you use the default `ConcurrentTaskRunner`. Then you call a `ray_greetings()` subflow that uses the `RayTaskRunner` to execute the same tasks in a Ray instance. ```python from prefect import flow, task from prefectray.taskrunners import RayTaskRunner @task def say_hello(name): print(f"hello {name}") @task def say_goodbye(name): print(f"goodbye {name}") @flow(task_runner=RayTaskRunner()) def ray_greetings(names): for name in names: say_hello.submit(name) say_goodbye.submit(name) @flow() def greetings(names): for name in names: say_hello.submit(name) say_goodbye.submit(name) ray_greetings(names) if name == "main": greetings(["arthur", "trillian", "ford", "marvin"]) ``` If you save this as `ray_subflow.py` and run it, you'll see that the flow `greetings` runs as you'd expect for a concurrent flow, then flow `ray-greetings` spins up a Ray instance to run the tasks again.