Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: overriding-job-variables.md

description: Learn how to override job variables on a work pool for a given deployment. tags: deployments flow runs work pools job variables environment variables search: boost: 2 As described in the guide, there are two ways to deploy flows to work pools: using a `prefect.yaml` file or using the `.deploy()` method. In both cases, you can override job variables on a work pool for a given deployment. While exactly which job variables are available to be overridden depends on the type of work pool you're using at a given time, this guide will explore some common patterns for overriding job variables in both deployment methods. First of all, what are "job variables"? Job variables are infrastructure related values that are configurable on a work pool, which may be relevant to how your flow run executes on your infrastructure. Job variables can be overridden on a per-deployment or per-flow run basis, allowing you to dynamically change infrastructure from the work pools defaults depending on your needs. <hr> Let's use `env` - the only job variable that is configurable for all work pool types - as an example. When you create or edit a work pool, you can specify a set of environment variables that will be set in the runtime environment of the flow run. For example, you might want a certain deployment to have the following environment variables available: ```json { "EXECUTION_ENV": "staging", "MYNOTSOSECRETCONFIG": "plumbus", } ``` Rather than hardcoding these values into your work pool in the UI and making them available to all deployments associated with that work pool, you can override these values on a per-deployment basis. Let's look at how to do that. Say we have the following repo structure: ``` tree . README.md requirements.txt demo_project daily_flow.py ``` ... and we have some `demo_flow.py` file like this: ```python import os from prefect import flow, task @task def dosomethingimportant(notsosecret_value: str) -> None: print(f"Doing something important with {notsosecret_value}!") @flow(log_prints=True) def some_work(): environment = os.environ.get("EXECUTION_ENVIRONMENT", "local") print(f"Coming to you live from {environment}!") notsosecretvalue = os.environ.get("MYNOTSOSECRET_CONFIG") if notsosecret_value is None: raise ValueError("You forgot to set MYNOTSOSECRETCONFIG!") dosomethingimportant(notsosecret_value) ``` In this case, let's also say we have the following deployment definition in a `prefect.yaml` file at the root of our repository: ```yaml deployments: name: demo-deployment entrypoint: demoproject/demoflow.py:some_work work_pool: name: local schedule: null ``` !!! note While not the focus of this guide, note that this deployment definition uses a default "global" `pull` step, because one is not explicitly defined on the deployment. For reference, here's what that would look like at the top of the `prefect.yaml` file: ```yaml pull: prefect.deployments.steps.gitclone: &clonerepo repository:

branch: main ``` To provide the `EXECUTIONENVIRONMENT` and `MYNOTSOSECRETCONFIG` environment variables to this deployment, we can add a `jobvariables` section to our deployment definition in the `prefect.yaml` file: ```yaml deployments: name: demo-deployment entrypoint: demoproject/demoflow.py:some_work work_pool: name: local job_variables: env: EXECUTION_ENVIRONMENT: staging MYNOTSOSECRETCONFIG: plumbus schedule: null ``` ... and then run `prefect deploy -n demo-deployment` to deploy the flow with these job variables. We should then be able to see the job variables in the `Configuration` tab of the deployment in the UI: If you want to use environment variables that are already set in your local environment, you can template these in the `prefect.yaml` file using the `{{ $ENVVARNAME }}` syntax: ```yaml deployments: name: demo-deployment entrypoint: demoproject/demoflow.py:some_work work_pool: name: local job_variables: env: EXECUTIONENVIRONMENT: "{{ $EXECUTIONENVIRONMENT }}" MYNOTSOSECRETCONFIG: "{{ $MYNOTSOSECRETCONFIG }}" schedule: null ``` !!! note This assumes that the machine where `prefect deploy` is run would have these environment variables set. <div class="terminal"> ```bash export EXECUTION_ENVIRONMENT=staging export MYNOTSOSECRETCONFIG=plumbus ``` </div> As before, run `prefect deploy -n demo-deployment` to deploy the flow with these job variables, and you should see them in the UI under the `Configuration` tab. If you're using the `.deploy()` method to deploy your flow, the process is similar, but instead of having your `prefect.yaml` file define the job variables, you can pass them as a dictionary to the `job_variables` argument of the `.deploy()` method. We could add the following block to our `demoproject/dailyflow.py` file from the setup section: ```python if name == "main": flow.from_source( source="https://github.com/zzstoatzz/prefect-monorepo.git", entrypoint="src/demoproject/demoflow.py:some_work" ).deploy( name="demo-deployment", workpoolname="local", # can only .deploy() to a local work pool in prefect>=2.15.1 job_variables={ "env": { "EXECUTIONENVIRONMENT": os.environ.get("EXECUTIONENVIRONMENT", "local"), "MYNOTSOSECRETCONFIG": os.environ.get("MYNOTSOSECRETCONFIG") } } ) ``` !!! note The above example works assuming a couple things: the machine where this script is run would have these environment variables set. <div class="terminal"> ```bash export EXECUTION_ENVIRONMENT=staging export MYNOTSOSECRETCONFIG=plumbus ``` </div> `demoproject/dailyflow.py` already exists in the repository at the specified path Running this script with something like: <div class="terminal"> ```bash python demoproject/dailyflow.py ``` </div> ... will deploy the flow with the specified job variables, which should then be visible in the UI under the `Configuration` tab. When running flows, you can pass in job variables that override any values set on the work pool or deployment. Any interface that runs deployments can accept job variables. Custom runs allow you to pass in a dictionary of variables into your flow run infrastructure. Using the same `env` example from above, we could do the following: Similarly, runs kicked off via CLI accept job variables with the `-jv` or `--job-variable` flag. ```bash prefect deployment run \ --id "fb8e3073-c449-474b-b993-851fe5e80e53" \ --job-variable MYNEWENV_VAR=42 \ --job-variable HELLO=THERE ``` Additionally, runs kicked off via automation actions can use job variables, including ones rendered from Jinja templates.