Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: results.md

description: Prefect results capture the data returned from your flows and tasks. tags: flows subflows tasks states results search: boost: 2 Results represent the data returned by a flow or a task. When calling flows or tasks, the result is returned directly: ```python from prefect import flow, task @task def my_task(): return 1 @flow def my_flow(): taskresult = mytask() return task_result + 1 result = my_flow() assert result == 2 ``` When working with flow and task states, the result can be retrieved with the `State.result()` method: ```python from prefect import flow, task @task def my_task(): return 1 @flow def my_flow(): state = mytask(returnstate=True) return state.result() + 1 state = myflow(returnstate=True) assert state.result() == 2 ``` When submitting tasks to a runner, the result can be retrieved with the `Future.result()` method: ```python from prefect import flow, task @task def my_task(): return 1 @flow def my_flow(): future = my_task.submit() return future.result() + 1 result = my_flow() assert result == 2 ``` Sometimes your flows or tasks will encounter an exception. Prefect captures all exceptions in order to report states to the orchestrator, but we do not hide them from you (unless you ask us to) as your program needs to know if an unexpected error has occurred. When calling flows or tasks, the exceptions are raised as in normal Python: ```python from prefect import flow, task @task def my_task(): raise ValueError() @flow def my_flow(): try: my_task() except ValueError: print("Oh no! The task failed.") return True my_flow() ``` If you would prefer to check for a failed task without using `try/except`, you may ask Prefect to return the state: ```python from prefect import flow, task @task def my_task(): raise ValueError() @flow def my_flow(): state = mytask(returnstate=True) if state.is_failed(): print("Oh no! The task failed. Falling back to '1'.") result = 1 else: result = state.result() return result + 1 result = my_flow() assert result == 2 ``` If you retrieve the result from a failed state, the exception will be raised. For this reason, it's often best to check if the state is failed first. ```python from prefect import flow, task @task def my_task(): raise ValueError() @flow def my_flow(): state = mytask(returnstate=True) try: result = state.result() except ValueError: print("Oh no! The state raised the error!") return True my_flow() ``` When retrieving the result from a state, you can ask Prefect not to raise exceptions: ```python from prefect import flow, task @task def my_task(): raise ValueError() @flow def my_flow(): state = mytask(returnstate=True) mayberesult = state.result(raiseon_failure=False) if isinstance(maybe_result, ValueError): print("Oh no! The task failed. Falling back to '1'.") result = 1 else: result = maybe_result return result + 1 result = my_flow() assert result == 2 ``` When submitting tasks to a runner, `Future.result()` works the same as `State.result()`: ```python from prefect import flow, task @task def my_task(): raise ValueError() @flow def my_flow(): future = my_task.submit() try: future.result() except ValueError: print("Ah! Futures will raise the failure as well.") mayberesult = future.result(raiseon_failure=False) print(f"Got {type(maybe_result)}") return True my_flow() ``` When calling flows or tasks, the result is returned directly: ```python import asyncio from prefect import flow, task @task async def my_task(): return 1 @flow async def my_flow(): taskresult = await mytask() return task_result + 1 result =

assert result == 2 ``` When working with flow and task states, the result can be retrieved with the `State.result()` method: ```python import asyncio from prefect import flow, task @task async def my_task(): return 1 @flow async def my_flow(): state = await mytask(returnstate=True) result = await state.result(fetch=True) return result + 1 async def main(): state = await myflow(returnstate=True) assert await state.result(fetch=True) == 2 asyncio.run(main()) ``` !!! important "Resolving results" Prefect 2.6.0 added automatic retrieval of persisted results. Prior to this version, `State.result()` did not require an `await`. For backwards compatibility, when used from an asynchronous context, `State.result()` returns a raw result type. You may opt-in to the new behavior by passing `fetch=True` as shown in the example above. If you would like this behavior to be used automatically, you may enable the `PREFECTASYNCFETCHSTATERESULT` setting. If you do not opt-in to this behavior, you will see a warning. You may also opt-out by setting `fetch=False`. This will silence the warning, but you will need to retrieve your result manually from the result type. When submitting tasks to a runner, the result can be retrieved with the `Future.result()` method: ```python import asyncio from prefect import flow, task @task async def my_task(): return 1 @flow async def my_flow(): future = await my_task.submit() result = await future.result() return result + 1 result = asyncio.run(my_flow()) assert result == 2 ``` The Prefect API does not store your results . Instead, the result is persisted to a storage location in your infrastructure and Prefect stores a reference to the result. The following Prefect features require results to be persisted: Task cache keys Flow run retries If results are not persisted, these features may not be usable. Persistence of results requires a and a . Prefect sets defaults for these, and you should not need to adjust them until you want to customize behavior. You can configure results on the `flow` and `task` decorators with the following options: `persist_result`: Whether the result should be persisted to storage. `result_storage`: Where to store the result when persisted. `result_serializer`: How to convert the result to a storable form. Persistence of the result of a task or flow can be configured with the `persistresult` option. The `persistresult` option defaults to a null value, which will automatically enable persistence if it is needed for a Prefect feature used by the flow or task. Otherwise, persistence is disabled by default. For example, the following flow has retries enabled. Flow retries require that all task results are persisted, so the task's result will be persisted: ```python from prefect import flow, task @task def my_task(): return "hello world!" @flow(retries=2) def my_flow(): my_task() ``` Flow retries do not require the flow's result to be persisted, so it will not be. In this next example, one task has caching enabled. Task caching requires that the given task's result is persisted: ```python from prefect import flow, task from datetime import timedelta @task(cachekeyfn=lambda: "always", cache_expiration=timedelta(seconds=20)) def my_task(): return "hello world!" @task def myothertask(): ... @flow def my_flow(): my_task() myothertask() ``` Persistence of results can be manually toggled on or off: ```python from prefect import flow, task @flow(persist_result=True) def my_flow(): ... @task(persist_result=False) def my_task(): ... ``` Toggling persistence manually will always override any behavior that Prefect would

You may also change Prefect's default persistence behavior with the `PREFECTRESULTSPERSISTBYDEFAULT` setting. To persist results by default, even if they are not needed for a feature change the value to a truthy value: ``` prefect config set PREFECTRESULTSPERSISTBYDEFAULT=true ``` Task and flows with `persistresult=False` will not persist their results even if `PREFECTRESULTSPERSISTBY_DEFAULT` is `true`. can be configured with the `resultstorage` option. The `resultstorage` option defaults to a null value, which infers storage from the context. Generally, this means that tasks will use the result storage configured on the flow unless otherwise specified. If there is no context to load the storage from and results must be persisted, results will be stored in the path specified by the `PREFECTLOCALSTORAGE_PATH` setting (defaults to `~/.prefect/storage`). ```python from prefect import flow, task from prefect.filesystems import LocalFileSystem, S3 @flow(persist_result=True) def my_flow(): my_task() # This task will use the flow's result storage @task(persist_result=True) def my_task(): ... my_flow() # The flow has no result storage configured and no parent, the local file system will be used. newflow = myflow.withoptions(resultstorage=S3(bucket_path="my-bucket")) new_flow() # The flow and task within it will use S3 for result storage. ``` You can configure this to use a specific storage using one of the following: A storage instance, e.g. `LocalFileSystem(basepath=".my-results")` A storage slug, e.g. `'s3/dev-s3-block'` The path of the result file in the result storage can be configured with the `resultstoragekey`. The `resultstoragekey` option defaults to a null value, which generates a unique identifier for each result. ```python from prefect import flow, task from prefect.filesystems import LocalFileSystem, S3 @flow(resultstorage=S3(bucketpath="my-bucket")) def my_flow(): my_task() @task(persistresult=True, resultstoragekey="mytask.json") def my_task(): ... myflow() # The task's result will be persisted to 's3://my-bucket/mytask.json' ``` Result storage keys are formatted with access to all of the modules in `prefect.runtime` and the run's `parameters`. In the following example, we will run a flow with three runs of the same task. Each task run will write its result to a unique file based on the `name` parameter. ```python from prefect import flow, task @flow() def my_flow(): hello_world() hello_world(name="foo") hello_world(name="bar") @task(persistresult=True, resultstorage_key="hello-{parameters[name]}.json") def hello_world(name: str = "world"): return f"hello {name}" my_flow() ``` After running the flow, we can see three persisted result files in our storage directory: ```shell $ ls ~/.prefect/storage | grep "hello-" hello-bar.json hello-foo.json hello-world.json ``` In the next example, we include metadata about the flow run from the `prefect.runtime.flow_run` module: ```python from prefect import flow, task @flow def my_flow(): hello_world() @task(persistresult=True, resultstoragekey="{flowrun.flowname}{flowrun.name}hello.json") def hello_world(name: str = "world"): return f"hello {name}" my_flow() ``` After running this flow, we can see a result file templated with the name of the flow and the flow run: ``` ls ~/.prefect/storage | grep "my-flow" my-flowindustrious-trouthello.json ``` If a result exists at a given storage key in the storage location, it will be overwritten. Result storage keys can only be configured on tasks at this time. can be configured with the `resultserializer` option. The `resultserializer` option defaults to a null value, which infers the serializer from the context. Generally, this means that tasks will use the result serializer configured on the flow unless otherwise specified. If there is no context to load the serializer from, the serializer defined by `PREFECTRESULTSDEFAULT_SERIALIZER` will be used. This setting defaults to Prefect's pickle

You may configure the result serializer using: A type name, e.g. `"json"` or `"pickle"` &mdash; this corresponds to an instance with default values An instance, e.g. `JSONSerializer(jsonlib="orjson")` Prefect provides a `CompressedSerializer` which can be used to wrap other serializers to provide compression over the bytes they generate. The compressed serializer uses `lzma` compression by default. We test other compression schemes provided in the Python standard library such as `bz2` and `zlib`, but you should be able to use any compression library that provides `compress` and `decompress` methods. You may configure compression of results using: A type name, prefixed with `compressed/` e.g. `"compressed/json"` or `"compressed/pickle"` An instance e.g. `CompressedSerializer(serializer="pickle", compressionlib="lzma")` Note that the `"compressed/<serializer-type>"` shortcut will only work for serializers provided by Prefect. If you are using custom serializers, you must pass a full instance. The Prefect API does not store your results in most cases for the following reasons: Results can be large and slow to send to and from the API. Results often contain private information or data. Results would need to be stored in the database or complex logic implemented to hydrate from another source. There are a few cases where Prefect will store your results directly in the database. This is an optimization to reduce the overhead of reading and writing to result storage. The following data types will be stored by the API without persistence to storage: booleans (`True`, `False`) nulls (`None`) If `persist_result` is set to `False`, these values will never be stored. The Prefect API tracks metadata about your results. The value of your result is only stored in . Result metadata can be seen in the UI on the "Results" page for flows. Prefect tracks the following result metadata: Data type Storage location (if persisted) When running your workflows, Prefect will keep the results of all tasks and flows in memory so they can be passed downstream. In some cases, it is desirable to override this behavior. For example, if you are returning a large amount of data from a task it can be costly to keep it memory for the entire duration of the flow run. Flows and tasks both include an option to drop the result from memory with `cacheresultin_memory`: ```python @flow(cacheresultin_memory=False) def foo(): return "pretend this is large data" @task(cacheresultin_memory=False) def bar(): return "pretend this is biiiig data" ``` When `cacheresultin_memory` is disabled, the result of your flow or task will be persisted by default. The result will then be pulled from storage when needed. ```python @flow def foo(): result = bar() state = bar(return_state=True) state.result() future = bar.submit() future.result() @task(cacheresultin_memory=False) def bar(): return "pretend this is biiiig data" ``` If both `cacheresultin_memory` and persistence are disabled, your results will not be available downstream. ```python @task(persistresult=False, cacheresultinmemory=False) def bar(): return "pretend this is biiiig data" @flow def foo(): result = bar() state = bar(return_state=True) state.result() future = bar.submit() future.result() ``` Result storage is responsible for reading and writing serialized data to an external location. At this time, any file system block can be used for result storage. A result serializer is responsible for converting your Python object to and from bytes. This is necessary to store the object outside of Python and retrieve it

Pickle is a standard Python protocol for encoding arbitrary Python objects. We supply a custom pickle serializer at `prefect.serializers.PickleSerializer`. Prefect's pickle serializer uses the project by default to support more object types. Alternative pickle libraries can be specified: ```python from prefect.serializers import PickleSerializer PickleSerializer(picklelib="custompickle") ``` Benefits of the pickle serializer: Many object types are supported. Objects can define custom pickle support. Drawbacks of the pickle serializer: When nested attributes of an object cannot be pickled, it is hard to determine the cause. When deserializing objects, your Python and pickle library versions must match the one used at serialization time. Serialized objects cannot be easily shared across different programming languages. Serialized objects are not human readable. We supply a custom JSON serializer at `prefect.serializers.JSONSerializer`. Prefect's JSON serializer uses custom hooks by default to support more object types. Specifically, we add support for all types supported by . By default, we use the standard Python `json` library. Alternative JSON libraries can be specified: ```python from prefect.serializers import JSONSerializer JSONSerializer(jsonlib="orjson") ``` Benefits of the JSON serializer: Serialized objects are human readable. Serialized objects can often be shared across different programming languages. Deserialization of serialized objects is generally version agnostic. Drawbacks of the JSON serializer: Supported types are limited. Implementing support for additional types must be done at the serializer level. Prefect uses internal result types to capture information about the result attached to a state. The following types are used: `UnpersistedResult`: Stores result metadata but the value is only available when created. `LiteralResult`: Stores simple values inline. `PersistedResult`: Stores a reference to a result persisted to storage. All result types include a `get()` method that can be called to return the value of the result. This is done behind the scenes when the `result()` method is used on states or futures. Unpersisted results are used to represent results that have not been and will not be persisted beyond the current flow run. The value associated with the result is stored in memory, but will not be available later. Result metadata is attached to this object for storage in the API and representation in the UI. Literal results are used to represent . The values contained by these results must always be JSON serializable. Example: ``` result = LiteralResult(value=None) result.json() ``` Literal results reduce the overhead required to persist simple results. The persisted result type contains all of the information needed to retrieve the result from storage. This includes: Storage: A reference to the that can be used to read the serialized result. Key: Indicates where this specific result is in storage. Persisted result types also contain metadata for inspection without retrieving the result: Serializer type: The name of the type. The `get()` method on result references retrieves the data from storage, deserializes it, and returns the original object. The `get()` operation will cache the resolved object to reduce the overhead of subsequent calls. When results are persisted to storage, they are always written as a JSON document. The schema for this is described by the `PersistedResultBlob` type. The document contains: The serialized data of the result. A full description of that can be used to deserialize the result data. The Prefect version used to create the result.