Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: work-pools.md

description: Learn how Prefect deployments can be configured for scheduled and remote execution with work pools. tags: work pools orchestration flow runs deployments tutorial search: boost: 2 Learn how to configure Prefect deployments for scheduled and remote execution with work pools. Work pools are a bridge between the Prefect orchestration layer and infrastructure for flow runs that can be dynamically provisioned. To transition from persistent infrastructure to dynamic infrastructure, use `flow.deploy` instead of `flow.serve`. !!! tip "" Earlier in the tutorial you used `serve` to deploy your flows. For many use cases, `serve` is sufficient to meet scheduling and orchestration needs. Work pools are optional. If infrastructure needs escalate, work pools become a handy tool. But you're not locked into one method and can combine approaches as needed. The primary reason to use work pools is for dynamic infrastructure provisioning and configuration. For example, you might have a workflow that has expensive infrastructure requirements and is run infrequently. In this case, you don't want an idle process running within that infrastructure. !!! note "Deployment definition methods differ slightly for work pools" When you use work-pool-based execution, you define deployments differently. Deployments for workers are configured with `deploy`, which requires additional configuration. You cannot use a deployment created with `serve` with a work pool. Other advantages to using work pools include: Configuring default infrastructure configurations on your work pools that all jobs inherit and can override. Allowing platform teams to use work pools to expose opinionated (and enforced) interfaces to the infrastructure that they oversee. Allowing work pools to prioritize (or limit) flow runs through the use of . Prefect provides several . Prefect Cloud provides a Prefect Managed work pool option that is the simplest way to run workflows remotely. A cloud-provider account, such as AWS, is not required with a Prefect Managed work pool. !!! note "Prefect Cloud" This tutorial uses Prefect Cloud to deploy flows to work pools. Managed execution and push work pools are available in only. If you are not using Prefect Cloud, learn more about work pools below and then proceed to the that uses worker-based work pools. In your terminal, run the following command to set up a work pool named `my-managed-pool` of type `prefect:managed`. <div class="terminal"> ```bash prefect work-pool create my-managed-pool --type prefect:managed ``` </div> Lets confirm that the work pool was successfully created by running the following command. <div class="terminal"> ```bash prefect work-pool ls ``` </div> You should see your new `my-managed-pool` in the output list. Navigate to the Work Pools tab in the UI and verify that `my-managed-pool` is listed. You can select Edit from the three-dot menu on the work pool card to view the details of your work pool. Work pools contain configuration details to provision infrastructure for flow runs. For example, you can specify additional Python packages or environment variables for all deployments that use this work pool. Note that individual deployments can override the work pool configuration. Now that youve set up your work pool, you can deploy a flow to it. Deploy your tutorial flow to

From the previous steps, you now have: A work pool Update your `repo_info.py` file to create a deployment in Prefect Cloud. The updates that you need to make to `repo_info.py` are: Change `flow.serve` to `flow.deploy`. Tell `flow.deploy` which work pool to deploy to. The updated `repo_info.py` looks like this: ```python hllines="17-23" title="repoinfo.py" import httpx from prefect import flow @flow(log_prints=True) def getrepoinfo(repo_name: str = "PrefectHQ/prefect"): url = f"https://api.github.com/repos/{repo_name}" response = httpx.get(url) response.raiseforstatus() repo = response.json() print(f"{repo_name} repository statistics :") print(f"Stars : {repo['stargazers_count']}") print(f"Forks : {repo['forks_count']}") if name == "main": getrepoinfo.from_source( source="https://github.com/prefecthq/demos.git", entrypoint="repoinfo.py:getrepo_info" ).deploy( name="my-first-deployment", workpoolname="my-managed-pool", ) ``` In the `from_source` method, specify the source of your flow code. In the `deploy` method, specify the name of your deployment and the name of the work pool that you created earlier. You can store your flow code in any of several types of remote storage. This example uses a GitHub repository, but you can also use a Docker image (more information on that later in this tutorial). Alternatively, you can store your flow code in cloud provider storage such as AWS S3, or within a different git-based cloud provider such as GitLab or Bitbucket. !!! note In the example above, you stored your code in a GitHub repository. If you make changes to the flow code, you must push those changes to your own GitHub account and update the `source` argument of `from_source` to point to your repository. Now that you've updated your script, you can run it to register your deployment on Prefect Cloud: <div class="terminal"> ```bash python repo_info.py ``` </div> You should see a message in the CLI that your deployment was created and instructions for how to run it. <div class="terminal"> ```bash Successfully created/updated all deployments! Deployments Name Status Details get-repo-info/my-first-deployment | applied To schedule a run for this deployment, use the following command: $ prefect deployment run 'get-repo-info/my-first-deployment' You can also run your flow through the Prefect UI: https://app.prefect.cloud/account/ abc/workspace/123/deployments/deployment/xyz ``` </div> Navigate to your Prefect Cloud UI and view your new deployment. Click the Run button to trigger a run of your deployment. Because you configured this deployment with a Prefect Managed work pool, Prefect Cloud will run your flow on your behalf. You can view the logs in the UI. You're ready to submit a flow-run to the work pool. Run the deployment from the CLI (or the UI): <div class="terminal"> ```bash prefect deployment run 'getrepoinfo/my-deployment' ``` </div> Prefect Managed work pools are a great way to get started with Prefect. See the for more details. For more control over the infrastructure that your flows run on, we recommend Prefect Cloud's push work pools. Serverless push work pools scale infinitely and provide more configuration options than Prefect Managed work pools. Prefect provides push work pools for AWS ECS on Fargate, Azure Container Instances, Google Cloud Run, and Modal. Using a push work pool requires sufficient permissions on your cloud provider account. This example uses GCP. Prerequisites: Install the and . If you already have the gcloud CLI installed, update to the latest version with `gcloud components update`. The following permissions in your GCP project: resourcemanager.projects.list serviceusage.services.enable iam.serviceAccounts.create iam.serviceAccountKeys.create

artifactregistry.repositories.create Docker installed to build and push images to your registry. Install . To set up a work pool named `my-cloud-run-pool` of type `cloud-run:push`, run: <div class="terminal"> ```bash prefect work-pool create --type cloud-run:push --provision-infra my-cloud-run-pool ``` </div> Using the `--provision-infra` flag allows you to select a GCP project for your work pool and automatically configure it to execute flows through Cloud Run. In your GCP project, this command activates the Cloud Run API, and creates a service account and a key (if they don't already exist). In your Prefect workspace, this command creates a for storing the service account key. Here's an abbreviated example output from running the command: <div class="terminal"> ```bash Provisioning infrastructure for your work pool my-cloud-run-pool will require: Updates in GCP project central-kit-405415 in region us-central1 Activate the Cloud Run API for your project Activate the Artifact Registry API for your project Create an Artifact Registry repository named prefect-images Create a service account for managing Cloud Run jobs: prefect-cloud-run Service account will be granted the following roles: Service Account User Cloud Run Developer Create a key for service account prefect-cloud-run Updates in Prefect workspace Create GCP credentials block my--pool-push-pool-credentials to store the service account key Proceed with infrastructure provisioning? [y/n]: y Activating Cloud Run API Activating Artifact Registry API Creating Artifact Registry repository Configuring authentication to Artifact Registry Setting default Docker build namespace Creating service account Assigning roles to service account Creating service account key Creating GCP credentials block Provisioning Infrastructure 100% 0:00:00 Infrastructure successfully provisioned! Created work pool 'my-cloud-run-pool'! ``` </div> After infrastructure provisioning completes, you will be logged into your new Artifact Registry repository. The default Docker build namespace is set to the URL of the repository. While the default namespace is set, any images you build without specifying a registry or username/organization are pushed to the repository. To use this functionality, write your deploy script like this: ```python hllines="14" title="exampledeploy_script.py" from prefect import flow from prefect.deployments import DeploymentImage @flow(log_prints=True) def my_flow(name: str = "world"): print(f"Hello {name}! I'm a flow running on Cloud Run!") if name == "main": my_flow.deploy( name="my-deployment", workpoolname="above-ground", cron="0 1 *", image=DeploymentImage( name="my-image:latest", platform="linux/amd64", ) ) ``` Run the script to create the deployment on the Prefect Cloud server. Running this script builds a Docker image with the tag `<region>-docker.pkg.dev/<project>/<repository-name>/my-image:latest` and pushes it to your repository. !!! tip Make sure you have Docker running locally before running this script. Note that you only need to include an object of the `DeploymentImage` class with the argument `platform="linux/amd64` if you're building your image on a machine with an ARM-based processor. Otherwise, you can pass `image="my-image:latest"` to `deploy`. Also note that the `cron` argument schedules the deployment to run at 1:00 AM every day. See the docs for more information on scheduling options. See the for more details and example commands for each cloud provider. If you need more control over your infrastructure, want to run your workflows in Kubernetes, or are running a self-hosted Prefect server instance, continue to the . You will learn how to use work pools that rely on a worker and customize Docker images for container-based infrastructure.