Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: workers.md

description: Learn how Prefect flow deployments enable configuring flows for scheduled and remote execution with workers. tags: workers orchestration flow runs deployments triggers tutorial search: boost: 2 Learn how workers enable flows for scheduled and remote execution. installed and running on your machine. In the previous section of the tutorial, you learned how: work pools are a bridge between the Prefect orchestration layer and infrastructure for flow runs, and can be dynamically provisioned. to transition from persistent infrastructure to dynamic infrastructure with `flow.deploy` instead of `flow.serve`. Work pools that rely on client-side workers enable you to run work flows in your own Docker containers, Kubernetes clusters, and serverless environments such as AWS ECS, Azure Container Instances, and GCP Cloud Run. The following diagram summarizes the architecture of a worker-based work pool deployment: ```mermaid graph TD subgraph your_infra["Your Execution Environment"] worker["Worker"] subgraph flowruninfra[Flow Run Infra] flowruna(("Flow Run A")) end subgraph flowruninfra_2[Flow Run Infra] flowrunb(("Flow Run B")) end end subgraph api["Prefect API"] Deployment --> |assigned to| work_pool work_pool(["Work Pool"]) end worker --> |polls| work_pool worker --> |creates| flowruninfra worker --> |creates| flowruninfra_2 ``` <sup>Notice above that the worker is in charge of provisioning the flow run infrastructure. In the context of this tutorial, that flow run infrastructure is an ephemeral Docker container to host each flow run. Different create different types of flow run infrastructure.</sup> Next, create a work pool and worker to deploy your tutorial flow. You will execute it later with the Prefect API. For this tutorial you will create a Docker type work pool through the CLI. Using the Docker work pool type means that all work sent to this work pool runs within a dedicated Docker container using a Docker client available to the worker. !!! tip "Other work pool types" There are for serverless computing environments such as AWS ECS, Azure Container Instances, Google Cloud Run, and Vertex AI. Kubernetes is also a popular work pool type. To set up a Docker type work pool, run: <div class="terminal"> ```bash prefect work-pool create --type docker my-docker-pool ``` </div> To confirm the work pool creation was successful, run: <div class="terminal"> ```bash prefect work-pool ls ``` </div> You should see your new `my-docker-pool` listed in the output. Next, check that you can see this work pool in your Prefect UI. Navigate to the Work Pools tab and verify that you see `my-docker-pool`

When you click into `my-docker-pool`, you should see a red status icon signifying that this work pool is not ready. To make the work pool ready, start a worker. Workers are a lightweight polling process that kick off scheduled flow runs on a specific type of infrastructure (such as Docker). To start a worker on your local machine, open a new terminal and confirm that your virtual environment has `prefect` installed. Run the following command in this new terminal to start the worker: <div class="terminal"> ```bash prefect worker start --pool my-docker-pool ``` </div> You should see the worker start. It's now polling the Prefect API to check for any scheduled flow runs it should pick up and then submit for execution. Youll see your new worker listed in the UI under the Workers tab of the Work Pools page with a recent last polled date. You should now see a `Ready` status indicator on your work pool. Keep this terminal session active for the worker to continue to pick up jobs. Since you are running this worker locally, the worker will terminate if you close the terminal. In a production setting this worker should run as a . Next, deploy your tutorial flow to `my-docker-pool`. From the previous steps, you now have: A work pool A worker Update your `repo_info.py` file to build a Docker image, and update your deployment so your worker can execute it. Make the following updates in `repo_info.py`: Change `flow.serve` to `flow.deploy`. Tell `flow.deploy` which work pool to deploy to. Tell `flow.deploy` the name for the Docker image to build. The updated `repo_info.py` looks like this: ```python hllines="17-22" title="repoinfo.py" import httpx from prefect import flow @flow(log_prints=True) def getrepoinfo(repo_name: str = "PrefectHQ/prefect"): url = f"https://api.github.com/repos/{repo_name}" response = httpx.get(url) response.raiseforstatus() repo = response.json() print(f"{repo_name} repository statistics :") print(f"Stars : {repo['stargazers_count']}") print(f"Forks : {repo['forks_count']}") if name == "main": getrepoinfo.deploy( name="my-first-deployment", workpoolname="my-docker-pool", image="my-first-deployment-image:tutorial", push=False ) ``` !!! note "Why the `push=False`?" For this tutorial, your Docker worker is running on your machine, so you don't need to push the image built by `flow.deploy` to a registry. When your worker is running on a remote machine, you must push the image to a registry that the worker can access. Remove the `push=False` argument, include your registry name, and ensure you've to push the image to a registry. Now that you've updated your script, run it to deploy your flow to the work pool: <div class="terminal"> ```bash python

``` </div> Prefect builds a custom Docker image containing your workflow code that the worker can use to dynamically spawn Docker containers whenever this workflow needs to run. !!! note "Dockerfile" In this example, Prefect generates a Dockerfile for you that builds an image based on one of Prefect's published images. The generated Dockerfile copies the current directory into the Docker image and installs any dependencies listed in a `requirements.txt` file. To use a custom Dockerfile, specify the path to the Dockerfile using the `DeploymentImage` class: ```python hllines="21-25" title="repoinfo.py" import httpx from prefect import flow from prefect.deployments import DeploymentImage @flow(log_prints=True) def getrepoinfo(repo_name: str = "PrefectHQ/prefect"): url = f"https://api.github.com/repos/{repo_name}" response = httpx.get(url) response.raiseforstatus() repo = response.json() print(f"{repo_name} repository statistics :") print(f"Stars : {repo['stargazers_count']}") print(f"Forks : {repo['forks_count']}") if name == "main": getrepoinfo.deploy( name="my-first-deployment", workpoolname="my-docker-pool", image=DeploymentImage( name="my-first-deployment-image", tag="tutorial", dockerfile="Dockerfile" ), push=False ) ``` To update your deployment, modify your script and rerun it. You'll need to make one update to specify a value for `job_variables` to ensure your Docker worker can successfully execute scheduled runs for this flow. See the example below. The `job_variables` section allows you to fine-tune the infrastructure settings for a specific deployment. These values override default values in the specified work pool's . When testing images locally without pushing them to a registry (to avoid potential errors like docker.errors.NotFound), we recommend including an `imagepullpolicy` job_variable set to `Never`. However, for production workflows, push images to a remote registry for more reliability and accessibility. Set the `imagepullpolicy` as `Never` for this tutorial deployment without affecting the default value set on your work pool: ```python hllines="21" title="repoinfo.py" import httpx from prefect import flow @flow(log_prints=True) def getrepoinfo(repo_name: str = "PrefectHQ/prefect"): url = f"https://api.github.com/repos/{repo_name}" response = httpx.get(url) response.raiseforstatus() repo = response.json() print(f"{repo_name} repository statistics :") print(f"Stars : {repo['stargazers_count']}") print(f"Forks : {repo['forks_count']}") if name == "main": getrepoinfo.deploy( name="my-first-deployment", workpoolname="my-docker-pool", jobvariables={"imagepull_policy": "Never"}, image="my-first-deployment-image:tutorial", push=False ) ``` To register this update to your deployment's parameters with Prefect's API, run: <div class="terminal"> ```bash python repo_info.py ``` </div> Submit a flow-run to the work pool: <div class="terminal"> ```bash prefect deployment run 'getrepoinfo/my-deployment' ``` </div> !!! danger "Common Pitfall" Store and run your deploy scripts at the root of your repo, otherwise the built Docker file may be missing files that it needs to execute. !!! tip "Multiple deployments" A Prefect flow can have more than one deployment. This pattern is useful for a flow to run in different execution environments. Learn how to configure deployments in YAML with . contain deep dives into Prefect components provide step-by-step recipes for common Prefect operations including: - -