Package: prefect-dask
Coordinate and parallelize your dataflow withprefect-daskVisit the full docshereto see additional examples and the API reference.Theprefect-daskcollection makes it easy to include distributed processing for your flows. Check out the examples below to get started!Getting StartedIntegrate with Prefect flowsPerhaps you're already working with Prefect flows. Say your flow downloads many images to train your machine learning model. Unfortunately, it takes a long time to download your flows because your code is running sequentially.After installingprefect-daskyou can parallelize your flow in three simple steps:Add the import:from prefect_dask import DaskTaskRunnerSpecify the task runner in the flow decorator:@flow(task_runner=DaskTaskRunner)Submit tasks to the flow's task runner:a_task.submit(*args, **kwargs)The parallelized code  runs in about 1/3 of the time in our test!  And that's without distributing the workload over multiple machines.
Here's the before and after!=== "Before"
```python hl_lines="1"
# Completed in 15.2 secondsfrom typing import List
from pathlib import Path

import httpx
from prefect import flow, task

URL_FORMAT = (
    "https://www.cpc.ncep.noaa.gov/products/NMME/archive/"
    "{year:04d}{month:02d}0800/current/images/nino34.rescaling.ENSMEAN.png"
)

@task
def download_image(year: int, month: int, directory: Path) -> Path:
    # download image from URL
    url = URL_FORMAT.format(year=year, month=month)
    resp = httpx.get(url)

    # save content to directory/YYYYMM.png
    file_path = (directory / url.split("/")[-1]).with_stem(f"{year:04d}{month:02d}")
    file_path.write_bytes(resp.content)
    return file_path

@flow
def download_nino_34_plumes_from_year(year: int) -> List[Path]:
    # create a directory to hold images
    directory = Path("data")
    directory.mkdir(exist_ok=True)

    # download all images
    file_paths = []
    for month in range(1, 12 + 1):
        file_path = download_image(year, month, directory)
        file_paths.append(file_path)
    return file_paths

if __name__ == "__main__":
    download_nino_34_plumes_from_year(2022)
```=== "After"```python hl_lines="1 8 26 35"
# Completed in 5.7 seconds

from typing import List
from pathlib import Path

import httpx
from prefect import flow, task
from prefect_dask import DaskTaskRunner

URL_FORMAT = (
    "https://www.cpc.ncep.noaa.gov/products/NMME/archive/"
    "{year:04d}{month:02d}0800/current/images/nino34.rescaling.ENSMEAN.png"
)

@task
def download_image(year: int, month: int, directory: Path) -> Path:
    # download image from URL
    url = URL_FORMAT.format(year=year, month=month)
    resp = httpx.get(url)

    # save content to directory/YYYYMM.png
    file_path = (directory / url.split("/")[-1]).with_stem(f"{year:04d}{month:02d}")
    file_path.write_bytes(resp.content)
    return file_path

@flow(task_runner=DaskTaskRunner(cluster_kwargs={"processes": False}))
def download_nino_34_plumes_from_year(year: int) -> List[Path]:
    # create a directory to hold images
    directory = Path("data")
    directory.mkdir(exist_ok=True)

    # download all images
    file_paths = []
    for month in range(1, 12 + 1):
        file_path = download_image.submit(year, month, directory)
        file_paths.append(file_path)
    return file_paths

if __name__ == "__main__":
    download_nino_34_plumes_from_year(2022)
```The original flow completes in 15.2 seconds.However, with just a few minor tweaks, we were able to reduce the runtime by nearlythreefolds, down to just5.7seconds!Integrate with Dask client/cluster and collectionsSuppose you have an existing Dask client/cluster and collection, like adask.dataframe.DataFrame, and you want to add observability.Withprefect-dask, there's no major overhaul necessary because Prefect was designed with incremental adoption in mind! It's as easy as:Adding the importsSprinkling a fewtaskandflowdecoratorsUsingget_dask_clientcontext manager on collections to distribute work across workersSpecifying the task runner and client's address in the flow decoratorSubmitting the tasks to the flow's task runner=== "Before"```python
import dask.dataframe
import dask.distributed



client = dask.distributed.Client()


def read_data(start: str, end: str) -> dask.dataframe.DataFrame:
    df = dask.datasets.timeseries(start, end, partition_freq="4w")
    return df


def process_data(df: dask.dataframe.DataFrame) -> dask.dataframe.DataFrame:

    df_yearly_avg = df.groupby(df.index.year).mean()
    return df_yearly_avg.compute()


def dask_pipeline():
    df = read_data("1988", "2022")
    df_yearly_average = process_data(df)
    return df_yearly_average

dask_pipeline()
```=== "After"```python hl_lines="3 4 8 13 15 19 21 22"
import dask.dataframe
import dask.distributed
from prefect import flow, task
from prefect_dask import DaskTaskRunner, get_dask_client

client = dask.distributed.Client()

@task
def read_data(start: str, end: str) -> dask.dataframe.DataFrame:
    df = dask.datasets.timeseries(start, end, partition_freq="4w")
    return df

@task
def process_data(df: dask.dataframe.DataFrame) -> dask.dataframe.DataFrame:
    with get_dask_client():
        df_yearly_avg = df.groupby(df.index.year).mean()
        return df_yearly_avg.compute()

@flow(task_runner=DaskTaskRunner(address=client.scheduler.address))
def dask_pipeline():
    df = read_data.submit("1988", "2022")
    df_yearly_average = process_data.submit(df)
    return df_yearly_average

dask_pipeline()
```Now, you can conveniently see when each task completed, both in the terminal and the UI!14:10:09.845|INFO|prefect.engine-Createdflowrun'chocolate-pony'forflow'dask-flow'14:10:09.847|INFO|prefect.task_runner.dask-ConnectingtoanexistingDaskclusterattcp://127.0.0.1:5925514:10:09.857|INFO|distributed.scheduler-Receiveclientconnection:Client-8c1e0f24-9133-11ed-800e-86f2469c4e7a14:10:09.859|INFO|distributed.core-Startingestablishedconnectiontotcp://127.0.0.1:5951614:10:09.862|INFO|prefect.task_runner.dask-TheDaskdashboardisavailableathttp://127.0.0.1:8787/status14:10:11.344|INFO|Flowrun'chocolate-pony'-Createdtaskrun'read_data-5bc97744-0'fortask'read_data'14:10:11.626|INFO|Flowrun'chocolate-pony'-Submittedtaskrun'read_data-5bc97744-0'forexecution.14:10:11.795|INFO|Flowrun'chocolate-pony'-Createdtaskrun'process_data-090555ba-0'fortask'process_data'14:10:11.798|INFO|Flowrun'chocolate-pony'-Submittedtaskrun'process_data-090555ba-0'forexecution.14:10:13.279|INFO|Taskrun'read_data-5bc97744-0'-FinishedinstateCompleted()14:11:43.539|INFO|Taskrun'process_data-090555ba-0'-FinishedinstateCompleted()14:11:43.883|INFO|Flowrun'chocolate-pony'-FinishedinstateCompleted('All states completed.')ResourcesFor additional examples, check out theUsage Guide!InstallationGet started by installingprefect-dask!=== "pip"```bash
pip install -U prefect-dask
```=== "conda"```bash
conda install -c conda-forge prefect-dask
```Requires an installation of Python 3.7+.We recommend using a Python virtual environment manager such as pipenv, conda, or virtualenv.These tasks are designed to work with Prefect 2. For more information about how to use Prefect, please refer to thePrefect documentation.FeedbackIf you encounter any bugs while usingprefect-dask, feel free to open an issue in theprefect-daskrepository.If you have any questions or issues while usingprefect-dask, you can find help in either thePrefect Discourse forumor thePrefect Slack community.Feel free to star or watchprefect-daskfor updates too!ContributingIf you'd like to help contribute to fix an issue or add a feature toprefect-dask, pleasepropose changes through a pull request from a fork of the repository.Here are the steps:Fork the repositoryClone the forked repositoryInstall the repository and its dependencies:pip install -e ".[dev]"Make desired changesAdd testsInsert an entry toCHANGELOG.mdInstallpre-committo perform quality checks prior to commit:pre-commit installgit commit,git push, and create a pull request