Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: specifying-upstream-dependencies.md

description: Run tasks in a desired order tags: tasks dependencies wait_for search: boost: 2 Learn how to run tasks in a desired order. You can provide results from a task to other tasks (or subflows) as upstream dependencies. Prefect uses upstream dependencies in two ways: To populate dependency arrows in the flow run graph To determine execution order for units of work that depend on each other !!! note "Tasks vs. other functions" Only results from tasks inform Prefect's ability to determine dependencies. Return values from functions without task decorators, including subflows, do not carry the same information about their origin as task results. When using non-sequential task runners such as the or , the order of execution for submitted tasks is not guaranteed unless you specify their dependencies. For example, compare how tasks submitted to the `ConcurrentTaskRunner` behave with and without upstream dependencies by clicking on the tabs below. === "Without dependencies" ```python @flow(log_prints=True) # Default task runner is ConcurrentTaskRunner def flowoftasks(): first.submit() second.submit() third.submit() @task def first(): print("I'm first!") @task def second(): print("I'm second!") @task def third(): print("I'm third!") ``` ```text hl_lines="7-9" Flow run 'pumpkin-puffin' - Created task run 'first-0' for task 'first' Flow run 'pumpkin-puffin' - Submitted task run 'first-0' for execution. Flow run 'pumpkin-puffin' - Created task run 'second-0' for task 'second' Flow run 'pumpkin-puffin' - Submitted task run 'second-0' for execution. Flow run 'pumpkin-puffin' - Created task run 'third-0' for task 'third' Flow run 'pumpkin-puffin' - Submitted task run 'third-0' for execution. Task run 'third-0' - I'm third! Task run 'first-0' - I'm first! Task run 'second-0' - I'm second! Task run 'second-0' - Finished in state Completed() Task run 'third-0' - Finished in state Completed() Task run 'first-0' - Finished in state Completed() Flow run 'pumpkin-puffin' - Finished in state Completed('All states completed.') ``` === "With dependencies" ```python @flow(log_prints=True) # Default task runner is ConcurrentTaskRunner def flowoftasks(): first_result = first.submit() secondresult = second.submit(firstresult) third.submit(second_result) @task def first(): print("I'm first!") @task def second(input): print("I'm second!") @task def third(input): print("I'm third!") ``` ```text hl_lines="7 9 11" Flow run 'statuesque-waxbill' - Created task run 'first-0' for task 'first' Flow run 'statuesque-waxbill' - Submitted task run 'first-0' for execution. Flow run 'statuesque-waxbill' - Created task run 'second-0' for task 'second' Flow run 'statuesque-waxbill' - Submitted task run 'second-0' for execution. Flow run 'statuesque-waxbill' - Created task run 'third-0' for task 'third' Flow run 'statuesque-waxbill' - Submitted task run 'third-0' for execution. Task run 'first-0' - I'm first! Task run 'first-0' - Finished in state Completed() Task run 'second-0' - I'm second! Task run 'second-0' - Finished in state Completed() Task run 'third-0' - I'm third! Task run 'third-0' - Finished in state Completed() Flow run 'statuesque-waxbill' - Finished in state Completed('All states

``` A task or subflow's upstream dependencies can be inferred automatically through its inputs, or stated explicitly with the `wait_for` parameter. When a result from a task is used as input for another task, Prefect automatically recognizes the task that result originated from as an upstream dependency. This applies to every way you can run tasks with Prefect, whether you're calling the task function directly, calling , or calling . Subflows similarly recognize tasks results as upstream dependencies. ```python from prefect import flow, task @flow(log_prints=True) def flowoftasks(): upstream_result = upstream.submit() downstream1result = downstream1.submit(upstreamresult) downstream2result = downstream2.submit(upstreamresult) mappedtaskresults = mappedtask.map([downstream1result, downstream2_result]) finaltask(mappedtask_results) @task def upstream(): return "Hello from upstream!" @task def downstream_1(input): return input @task def downstream_2(input): return input @task def mapped_task(input): return input @task def final_task(input): print(input) ``` <figure markdown="span"> <figcaption>Flow run graph displaying inferred dependencies with the "Dependency grid" layout selected</figcaption> </figure> Tasks that do not share data can be informed of their upstream dependencies through the `wait_for` parameter. Just as with automatic dependencies, this applies to direct task function calls, , , and subflows. !!! warning "Differences with `.map()`" Manually defined upstream dependencies apply to all tasks submitted by `.map()`, so each mapped task must wait for all upstream dependencies passed into `wait_for` to finish. This is distinct from automatic dependencies for mapped tasks, where each mapped task must only wait for the upstream tasks whose results it depends on. ```python from prefect import flow, task @flow(log_prints=True) def flowoftasks(): upstream_result = upstream.submit() downstream1result = downstream1.submit(waitfor=[upstream_result]) downstream2result = downstream2.submit(waitfor=[upstream_result]) mappedtaskresults = mappedtask.map([1, 2], waitfor=[downstream1result, downstream2result]) finaltask(waitfor=mappedtaskresults) @task def upstream(): pass @task def downstream_1(): pass @task def downstream_2(): pass @task def mapped_task(input): pass @task def final_task(): pass ``` <figure markdown="span"> <figcaption>Flow run graph displaying manual dependencies with the "Dependency grid" layout selected</figcaption> </figure> For more complex workflows, parts of your logic may require additional resources, different infrastructure, or independent parallel execution. A typical approach for addressing these needs is to execute that logic as separate runs from within a flow. Compose deployment runs into a flow and treat them as upstream dependencies by calling from within a task. Given a deployment `process-user` of flow `parallel-work`, a flow of deployments might look like this: ```python from prefect import flow, task from prefect.deployments import run_deployment @flow def flowofdeployments(): deploymentrun1 = rundeploymenttask.submit( flow_name="parallel-work", deployment_name="process-user", parameters={"user_id": 1}, ) deploymentrun2 = rundeploymenttask.submit( flow_name="parallel-work", deployment_name="process-user", parameters={"user_id": 2}, ) downstreamtask(waitfor=[deploymentrun1, deploymentrun2]) @task(taskrunname="Run deployment {flowname}/{deploymentname}") def rundeploymenttask( flow_name: str, deployment_name: str, parameters: dict ): run_deployment( name=f"{flowname}/{deploymentname}", parameters=parameters ) @task def downstream_task(): print("I'm downstream!") ``` By default, deployments started from `rundeployment` will also appear as subflows for tracking purposes. Disable this behavior by setting the `assubflow` parameter for `run_deployment` to `False`. <figure markdown="span"> <figcaption>Flow run graph displaying deployments as dependencies with the "Dependency grid" layout selected</figcaption> </figure>