Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: big-data.md

description: Tips for using Prefect when dealing with big data. tags: big data flow configuration parallel execution distributed execution caching search: boost: 2 This guide explores methods to reduce the processing time or memory utilization of Prefect workflows that process large amounts of data, without editing your Python code. Prefect provides multiple options to optimize your Python code for speed, memory, compute, or disk space. This guide uses examples for how to: Remove task introspection with `quote` to save time running your code. Write task results to cloud storage such as S3 using a block to save memory. Save data to disk within a flow rather than using results. Cache task results to save time and compute. Compress results written to disk to save space. Use a for parallelizable operations to save time. When a task is called from a flow, each argument is introspected by Prefect, by default. To speed up your flow runs, disable this behavior for a task by wrapping the argument using . To demonstrate, here's a basic example that extracts and transforms some New York taxi data: ```python hllines="2 26" title="etquote.py" from prefect import task, flow from prefect.utilities.annotations import quote import pandas as pd @task def extract(url: str): """Extract data""" dfraw = pd.readparquet(url) print(df_raw.info()) return df_raw @task def transform(df: pd.DataFrame): """Basic transformation""" df["tipfraction"] = df["tipamount"] / df["total_amount"] print(df.info()) return df @flow(log_prints=True) def et(url: str): """ET pipeline""" df_raw = extract(url) df = transform(quote(df_raw)) if name == "main": url = "https://d37ci6vzurychx.cloudfront.net/trip-data/yellowtripdata2023-09.parquet" et(url) ``` Introspection can take significant time when the object being passed is a large collection, such as dictionary or DataFrame, where each element needs to be visited. Note that using `quote` reduces execution time at the expense of disabling task dependency tracking for the wrapped

By default, the results of task runs are stored in memory in your execution environment. This behavior makes flow runs fast for small data, but can be problematic for large data. You can save memory by writing results to disk. In production, you'll generally want to write results to a cloud provider storage such as AWS S3. Prefect lets you use a storage block from a Prefect Cloud integration library such as to save your configuration information. Learn more about . Install the relevant library, register the block type with the server, and create your block. Then reference the block in your flow: ```python ... from prefect_aws.s3 import S3Bucket mys3block = S3Bucket.load("MYBLOCKNAME") ... @task(resultstorage=mys3_block) ``` The result of the task will write to S3, rather than be stored in memory. To save memory and time with big data, you don't need to pass results between tasks. Instead, write and read data to disk directly in your flow code. Prefect has integration libraries for each of the major cloud providers. Each library contains blocks with methods that make it convenient to read and write data to and from cloud object storage. The has step-by-step examples for each cloud provider. Caching allows you to avoid unnecessarily re-running tasks. Caching can save you time and compute. Note that caching requires task result persistence. Caching is discussed in detail in the . If you're using Prefect's task result persistence, you can save disk space by compressing the results. Specify the result type with `compressed/` prefixed: ```python @task(result_serializer="compressed/json") ``` Read about for more details. The tradeoff of using compression is that it takes time to compress and decompress the data. Prefect's task runners allow you to use the Dask and Ray Python libraries to run tasks in parallel and distributed across multiple machines. This can save you time and compute when operating on large data structures. See the for details.