Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: automations.md

description: Prefect walkthrough on how to use automations and common best practices tags: automations event-driven trigger title: Automations and Common Use Cases search: boost: 2 The shows you how to configure one within the UI. This guide describes the following use cases: Create a simple notification automation in just a few UI clicks Build upon an event-based automation Combine into a multi-layered responsive deployment pattern !!! cloud-ad "Available only on Prefect Cloud" Automations are a Prefect Cloud feature. Python installed Prefect Authenticated to a A set up to handle the deployments Automations allow you to take actions in response to triggering events recorded by Prefect. This example grabs data from an API and sends a notification based on the end state. Start by pulling hypothetical user data from an endpoint and then performing data cleaning and transformations. First create a simple extract method that pulls the data from a random user data generator endpoint. ```python from prefect import flow, task, getrunlogger import requests import json @task def fetch(url: str): logger = getrunlogger() response = requests.get(url) raw_data = response.json() logger.info(f"Raw response: {raw_data}") return raw_data @task def clean(raw_data: dict): print(raw_data.get('results')[0]) results = raw_data.get('results')[0] logger = getrunlogger() logger.info(f"Cleaned results: {results}") return results['name'] @flow def build_names(num: int = 10): df = [] url = "https://randomuser.me/api/" logger = getrunlogger() copy = num while num != 0: raw_data = fetch(url) df.append(clean(raw_data)) num -= 1 logger.info(f"Built {copy} names: {df}") return df if name == "main": listofnames = build_names() ``` The data cleaning workflow has visibility into each step, and sends a list of names to the next step of the pipeline. Next, send a notification based off a completed state outcome. Configure a notification that shows when to look into our workflow logic. Prior to creating the automation, confirm the notification location. Create a notification block to help define where the notification is sent. Navigate to the blocks page on the UI, and click into creating an email notification block. Go to the automations page to create your first automation. Next, find the trigger type. In this case, use a flow completion. Finally, create the actions for when the trigger is hit. In this case, create a notification to showcase the completion. Now the automation is ready to be triggered from a flow run completion. Run the file locally and see that the notification is sent to your inbox after the completion. It may take a few minutes for the notification to arrive. !!! Tip "No deployment created" Keep in mind, you do not need to create a deployment to trigger your automation, where a state outcome of a local flow run helped trigger this notification block. You are not required to create a deployment to trigger a notification. Now that you've seen how to create an email notification from a flow run completion, see how to kick off a deployment run in response to an event. Create an automation to kick off a deployment instead of a notification. Explore how to programmatically create this automation by taking advantage of Prefect's REST API. See the as a reference for interacting with the Prefect Cloud automation endpoints. Create a deployment to kick off some work based on how long a flow is

For example, if the `buildnames` flow takes too long to execute, you can kick off a deployment with the same `buildnames` flow, but replace the `count` value with a lower number to speed up completion. Create a deployment with a `prefect.yaml` file or a Python file that uses `flow.deploy`. === "prefect.yaml" Create a `prefect.yaml` file like this one for our flow `build_names`: ```yaml name: automations-guide prefect-version: 2.13.1 build: null push: null pull: prefect.deployments.steps.setworkingdirectory: directory: /Users/src/prefect/Playground/automations-guide deployments: name: deploy-build-names version: null tags: [] description: null entrypoint: test-automations.py:build_names parameters: {} work_pool: name: tutorial-process-pool workqueuename: null job_variables: {} schedule: null ``` === ".deploy" To follow a more Python-based approach to create a deployment, use `flow.deploy` as in the example below. ```python if name == "main": build_names.deploy( name="deploy-build-names", workpoolname="tutorial-process-pool" image="myregistry/myimage:myimagetag", ) ``` Grab your `deployment_id` from this deployment using the CLI and embed it in your automation. !!! Tip "Find deployment_id from the CLI" Run `prefect deployment ls` in an authenticated command prompt. ```bash prefect deployment ls Deployments Name ID Extract islands/island-schedule d9d7289c-7a41-436d-8313-80a044e61532 build-names/deploy-build-names 8b10a65e-89ef-4c19-9065-eec5c50242f4 ride-duration-prediction-backfill/backfill-deployment 76dc6581-1773-45c5-a291-7f864d064c57 ``` Create an automation with a POST call to programmatically create the automation. Ensure you have your `apikey`, `accountid`, and `workspace_id`. ```python def createeventdriven_automation(): apiurl = f"https://api.prefect.cloud/api/accounts/{accountid}/workspaces/{workspace_id}/automations/" data = { "name": "Event Driven Redeploy", "description": "Programmatically created an automation to redeploy a flow based on an event", "enabled": "true", "trigger": { "after": [ "string" ], "expect": [ "prefect.flow-run.Running" ], "for_each": [ "prefect.resource.id" ], "posture": "Proactive", "threshold": 30, "within": 0 }, "actions": [ { "type": "run-deployment", "source": "selected", "deployment_id": "YOUR-DEPLOYMENT-ID", "parameters": "10" } ], "owner_resource": "string" } headers = {"Authorization": f"Bearer {PREFECTAPIKEY}"} response = requests.post(api_url, headers=headers, json=data) print(response.json()) return response.json() ``` After running this function, you will see the changes that came from the post request within the UI. Keep in mind the context will be "custom" on UI. Run the underlying flow and see the deployment kick off after 30 seconds. This results in a new flow run of `build_names`. You can see this new deployment get initiated with the custom parameters outlined above. In a few quick changes, you can programmatically create an automation that deploys workflows with custom parameters. You can take this a step further by using your own .yaml version of the automation, and registering that file with the UI. This simplifies the requirements of the automation by declaring it in its own .yaml file, and then registering that .yaml with the API. First start with creating the .yaml file to house the automation requirements: ```yaml title="automation.yaml" name: Cancel long running flows description: Cancel any flow run after an hour of execution trigger: match: "prefect.resource.id": "prefect.flow-run.*" match_related: {} after: "prefect.flow-run.Failed" expect: "prefect.flow-run.*" for_each: "prefect.resource.id" posture: "Proactive" threshold: 1 within: 30 actions: type: "cancel-flow-run" ``` Make a helper function that applies this YAML file with the REST API function. ```python import yaml from utils import post, put def createorupdate_automation(path: str = "automation.yaml"): """Create or update an automation from a local YAML file""" with open(path, "r") as fh: payload = yaml.safe_load(fh) automations = post("/automations/filter") existing_automation = [a["id"] for a in automations if a["name"] == payload["name"]] automationexists = len(existingautomation) > 0 if automation_exists: print(f"Automation '{payload['name']}' already exists and will be updated") put(f"/automations/{existing_automation[0]}", payload=payload) else: print(f"Creating automation '{payload['name']}'") post("/automations/", payload=payload) if name == "main": createorupdate_automation() ``` Find a complete repo with these APIs examples in this . In this example, you created the automation by registering the .yaml file with a helper function. Use webhooks to expose the events

This allows you to extend the functionality of deployments and respond to changes in your workflow. By exposing a webhook endpoint, you can kick off workflows that trigger deployments, all from an event created from an HTTP request. Create this webhook in the UI to create these dynamic events. ```JSON { "event": "model-update", "resource": { "prefect.resource.id": "product.models.{{ body.model_id}}", "prefect.resource.name": "{{ body.friendly_name }}", "runcount": "{{body.runcount}}" } } ``` From this input, you can create an exposed webhook endpoint. Each webhook corresponds to a custom event created, where you can react to it downstream with a separate deployment or automation. For example, you can create a curl request that sends the endpoint information such as a run count for your deployment. ```console curl -X POST https://api.prefect.cloud/hooks/34iV2SFke3mVa6y5Y-YUoA -d "modelid=adhoc" -d "runcount=10" -d "friendly_name=test-user-input" ``` From here, you can make a webhook that is connected to pulling in parameters on the curl command, and then it kicks off a deployment that uses these pulled parameters. Go into the event feed to automate straight from this event. This allows you to create automations that respond to these webhook events. From a few clicks in the UI, you can associate an external process with the Prefect events API that can trigger downstream deployments. In the next section, you will explore event triggers that automate the kickoff of a deployment run. You can take this a step further by creating a deployment that triggers when a flow run takes longer than expected. You can take advantage of Prefect's library that uses an LLM to classify the data. Marvin is great at embedding data science and data analysis applications within your pre-existing data engineering workflows. In this case, you can use to help make the dataset more information rich. Install Marvin with `pip install marvin` and . You can add a trigger to run a deployment in response to a specific event. Here's an example with Marvin's AI functions. It will take in a Pandas DataFrame and use the AI function to analyze it. Here is an example of pulling in that data and classifying using Marvin AI. The dummy data is based on classifications you have already created. ```python from marvin import ai_classifier from enum import Enum import pandas as pd @ai_fn def generatesyntheticuserdata(buildof_names: list[dict]) -> list: """ Generate additional data for userID (numerical values with 6 digits), location, and timestamp as separate columns and append the data onto 'buildofnames'. Make userID the first column """ @flow def createfakeuser_dataset(df): artifactdf = generatesyntheticuserdata(df) print(artifact_df) createtableartifact( key="fake-user-data", table=artifact_df, description= "Dataset that is comprised of a mix of autogenerated data based on user data" ) if name == "main": createfakeartifact() ``` Kick off a deployment with a trigger defined in a `prefect.yaml` file. Specify what to trigger when the event stays in a running state for longer than 30 seconds. ```yaml name: automations-guide prefect-version: 2.13.1 build: null push: null pull: prefect.deployments.steps.setworkingdirectory: directory: /Users/src/prefect/Playground/marvin-extension deployments: name: create-fake-user-dataset triggers: enabled: true match: prefect.resource.id: "prefect.flow-run.*" after: "prefect.flow-run.Running", expect: [], for_each: ["prefect.resource.id"], parameters: param_1: 10 posture: "Proactive" version: null tags: [] description: null entrypoint: marvin-extension.py:createfakeuser_dataset parameters: {} work_pool: name: tutorial-process-pool workqueuename: null job_variables: {} schedule: null ``` You've seen how to create automations with the UI, REST API, and a trigger defined in a `prefect.yaml` deployment definition. To learn more about Prefect events, which can trigger automations, see the . Prefect Cloud can receive external events webhooks. See the to learn how to create them.