Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: ci-cd.md

description: CI/CD resources for working with Prefect. tags: CI/CD continuous integration continuous delivery search: boost: 2 Many organizations deploy Prefect workflows through their CI/CD process. Each organization has their own unique CI/CD setup, but a common pattern is to use CI/CD to manage Prefect . Combining Prefect's deployment features with CI/CD tools enables efficient management of flow code updates, scheduling changes, and container builds. This guide uses to implement a CI/CD process, but these concepts are generally applicable across many CI/CD tools. Note that Prefect's primary ways for creating deployments, a `.deploy` flow method or a `prefect.yaml` configuration file, are both designed for building and pushing images to a Docker registry. In this example, you'll write a GitHub Actions workflow that runs each time you push to your repository's `main` branch. This workflow builds and pushes a Docker image containing your flow code to Docker Hub, then deploys the flow to Prefect Cloud. Your CI/CD process must be able to authenticate with Prefect in order to deploy flows. Deploy flows securely and non-interactively in your CI/CD process by saving your `PREFECTAPIURL` and `PREFECTAPIKEY` . This allows them to be accessed in your CI/CD runner's environment without exposing them in any scripts or configuration files. In this scenario, deploying flows involves building and pushing Docker images, so add `DOCKERUSERNAME` and `DOCKERPASSWORD` as secrets to your repository as well. Create secrets for GitHub Actions in your repository under Settings -> Secrets and variables -> Actions -> New repository secret: To deploy your flow through GitHub Actions, you need a workflow YAML file. GitHub looks for workflow YAML files in the `.github/workflows/` directory in the root of your repository. In their simplest form, GitHub workflow files are made up of triggers and jobs. The `on:` trigger is set to run the workflow each time a push occurs on the `main` branch of the repository. The `deploy` job is comprised of four `steps`: `Checkout` clones your repository into the GitHub Actions runner so you can reference files or run scripts from your repository in later steps. `Log in to Docker Hub` authenticates to DockerHub so your image can be pushed to the Docker registry in your DockerHub account. is an existing GitHub action maintained by Docker. `with:` passes values into the Action, similar to passing parameters to a function. `Setup Python` installs your selected version of Python. `Prefect Deploy` installs the dependencies used in your flow, then deploys your flow. `env:` makes the `PREFECTAPIKEY` and `PREFECTAPIURL` secrets from your repository available as environment variables during this step's execution. For reference, the examples below live in their respective branches of . === ".deploy" ``` . .github/ workflows/ deploy-prefect-flow.yaml flow.py requirements.txt ``` `flow.py` ```python from prefect import flow @flow(log_prints=True) def hello(): print("Hello!") if name == "main":

name="my-deployment", workpoolname="my-work-pool", image="myregistry/myimage:myimagetag", ) ``` `.github/workflows/deploy-prefect-flow.yaml` ```yaml name: Deploy Prefect flow on: push: branches: main jobs: deploy: name: Deploy runs-on: ubuntu-latest steps: name: Checkout uses: actions/checkout@v4 name: Log in to Docker Hub uses: docker/login-action@v3 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} name: Setup Python uses: actions/setup-python@v5 with: python-version: '3.11' name: Prefect Deploy env: PREFECTAPIKEY: ${{ secrets.PREFECTAPIKEY }} PREFECTAPIURL: ${{ secrets.PREFECTAPIURL }} run: | pip install -r requirements.txt python flow.py ``` === "prefect.yaml" ``` . .github/ workflows/ deploy-prefect-flow.yaml flow.py prefect.yaml requirements.txt ``` `flow.py` ```python from prefect import flow @flow(log_prints=True) def hello(): print("Hello!") ``` `prefect.yaml` ```yaml name: cicd-example prefect-version: 2.14.11 build: prefectdocker.deployments.steps.builddocker_image: id: build_image requires: prefect-docker>=0.3.1 imagename: myregistry/my_image tag: myimagetag dockerfile: auto push: prefectdocker.deployments.steps.pushdocker_image: requires: prefect-docker>=0.3.1 imagename: "{{ buildimage.image_name }}" tag: "{{ build_image.tag }}" pull: null deployments: name: my-deployment entrypoint: flow.py:hello work_pool: name: my-work-pool workqueuename: default job_variables: image: "{{ build-image.image }}" ``` `.github/workflows/deploy-prefect-flow.yaml` ```yaml name: Deploy Prefect flow on: push: branches: main jobs: deploy: name: Deploy runs-on: ubuntu-latest steps: name: Checkout uses: actions/checkout@v4 name: Log in to Docker Hub uses: docker/login-action@v3 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} name: Setup Python uses: actions/setup-python@v5 with: python-version: '3.11' name: Prefect Deploy env: PREFECTAPIKEY: ${{ secrets.PREFECTAPIKEY }} PREFECTAPIURL: ${{ secrets.PREFECTAPIURL }} run: | pip install -r requirements.txt prefect deploy -n my-deployment ``` After pushing commits to your repository, GitHub automatically triggers a run of your workflow. Monitor the status of running and completed workflows from the Actions tab of your repository. View the logs from each workflow step as they run. The `Prefect Deploy` step includes output about your image build and push, and the creation/update of your deployment. <div class="terminal"> ```bash Successfully built image '*/cicd-example:latest' Successfully pushed image '*/cicd-example:latest' Successfully created/updated all deployments! Deployments Name Status Details hello/my-deployment applied ``` </div> In more complex scenarios, CI/CD processes often need to accommodate several additional considerations to enable a smooth development workflow: Making code available in different environments as it advances through stages of development Handling independent deployment of distinct groupings of work, as in a monorepo Efficiently using build time to avoid repeated work This addresses each of these considerations with a combination of Prefect's and GitHub's capabilities. Which deployment processes should run are automatically selected when changes are pushed depending on two conditions: ```yaml on: push: branches: stg main paths: "project_1/" ``` `branches:` - which branch has changed. This ultimately selects which Prefect workspace a deployment is created or updated in. In this example, changes on the `stg` branch will deploy flows to a staging workspace, and changes on the `main` branch will deploy flows to a production workspace. `paths:` - which project folders' files have changed. Since each project folder contains its own flows, dependencies, and `prefect.yaml`, it represents a complete set of logic and configuration that can be deployed

Each project in this repository gets its own GitHub Actions workflow YAML file. The `prefect.yaml` file in each project folder depends on environment variables that are dictated by the selected job in each CI/CD workflow, enabling external code storage for Prefect deployments that is clearly separated across projects and environments. ``` . cicd-example-workspaces-prod # production bucket project_1 project_2 cicd-example-workspaces-stg # staging bucket project_1 project_2 ``` Since the deployments in this example use S3 for code storage, it's important that push steps place flow files in separate locations depending upon their respective environment and project so no deployment overwrites another deployment's files. Since building Docker images and installing Python dependencies are essential parts of the deployment process, it's useful to rely on caching to skip repeated build steps. The `setup-python` action offers so Python packages do not have to be downloaded on repeat workflow runs. ```yaml name: Setup Python uses: actions/setup-python@v5 with: python-version: "3.11" cache: "pip" ``` ``` Using cached prefect-2.16.1-py3-none-any.whl (2.9 MB) Using cached prefect_aws-0.4.10-py3-none-any.whl (61 kB) ``` The `build-push-action` for building Docker images also offers . If you are not using GitHub, other remote are available as well. ```yaml name: Build and push id: build-docker-image env: GITHUBSHA: ${{ steps.get-commit-hash.outputs.COMMITHASH }} uses: docker/build-push-action@v5 with: context: ${{ env.PROJECT_NAME }}/ push: true tags: ${{ secrets.DOCKERUSERNAME }}/${{ env.PROJECTNAME }}:${{ env.GITHUB_SHA }}-stg cache-from: type=gha cache-to: type=gha,mode=max ``` ``` importing cache manifest from gha:* DONE 0.1s [internal] load build context transferring context: 70B done DONE 0.0s [2/3] COPY requirements.txt requirements.txt CACHED [3/3] RUN pip install -r requirements.txt CACHED ``` Prefect provides its own GitHub Actions for and . These actions can simplify deploying with CI/CD when using `prefect.yaml`, especially in cases where a repository contains flows that are used in multiple deployments across multiple Prefect Cloud workspaces. Here's an example of integrating these actions into the workflow above: ```yaml name: Deploy Prefect flow on: push: branches: main jobs: deploy: name: Deploy runs-on: ubuntu-latest steps: name: Checkout uses: actions/checkout@v4 name: Log in to Docker Hub uses: docker/login-action@v3 with: username: ${{ secrets.DOCKER_USERNAME }} password: ${{ secrets.DOCKER_PASSWORD }} name: Setup Python uses: actions/setup-python@v5 with: python-version: "3.11" name: Prefect Auth uses: PrefectHQ/actions-prefect-auth@v1 with: prefect-api-key: ${{ secrets.PREFECTAPIKEY }} prefect-workspace: ${{ secrets.PREFECT_WORKSPACE }} name: Run Prefect Deploy uses: PrefectHQ/actions-prefect-deploy@v3 with: deployment-names: my-deployment requirements-file-paths: requirements.txt ``` The `docker/login-action` GitHub Action supports pushing images to a wide variety of image registries. For example, if you are storing Docker images in AWS Elastic Container Registry, you can add your ECR registry URL to the `registry` key in the `with:` part of the action and use an `AWSACCESSKEYID` and `AWSSECRETACCESSKEY` as your `username` and `password`. ```yaml name: Login to ECR uses: docker/login-action@v3 with: registry: <aws-account-number>.dkr.ecr.<region>.amazonaws.com username: ${{ secrets.AWSACCESSKEY_ID }} password: ${{ secrets.AWSSECRETACCESS_KEY }} ``` Check out the if you're using Terraform to manage your infrastructure.