Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: blocks.md

description: Prefect blocks package configuration storage, infrastructure, and secrets for use with deployments or flow scripts. tags: blocks storage secrets configuration search: boost: 2 Prefect blocks store configuration and provide an interface for interacting with external systems. Blocks expose methods that provide functionality specific to the systems they interface with. For example, blocks can be used to download data from or upload data to an S3 bucket, query data from or write data to a database, or send a message to a Slack channel. Block types are Python classes with a handy UI webform for configuration. Blocks are instantiation of these classes with specific values. Configure blocks through Python code or via a form in the UI. Access blocks for use in Python code. Block values are stored in Prefect Cloud or your self-hosted Prefect server instance. Blocks can be shared with other users in your Prefect Cloud workspace. To see block types available for configuration, use `prefect block type ls` from the CLI or navigate to the Blocks page in the UI and click +. !!! tip "Blocks and parameters" Blocks are useful for configuration that needs to be shared across flow runs and between flows. For configuration that will change between flow runs, we recommend using . Commonly used block types come built-in with Prefect. These block types can be created via the UI and used without installing any additional packages. | Block | Slug | Description | | -- | -- | | | | `azure` | Stores data as a file on Azure Data Lake and Azure Blob Storage. | | | `custom-webhook` | Calls custom webhooks. | | | `discord-webhook` | Calls Discord webhooks. | | | `date-time` | Stores a datetime value. | | | `docker-container` | Runs a command in a container. | | | `docker-registry` | Connects to a Docker registry. Requires a Docker Engine to be connectable. | | | `gcs` | Store data as a file on Google Cloud Storage. | | | `github` | Interacts with files stored on public GitHub repositories. | | | `json` | Stores JSON data. | | | <span class="no-wrap">`kubernetes-cluster-config`</span> | Stores configuration for interaction with Kubernetes clusters. | | | `kubernetes-job` | Runs a command as a Kubernetes Job. | | | `local-file-system` | Stores data as a file on a local file system. | | `mattermost-webhook` | Sends notifications via a provided Mattermost webhook. | | | `ms-teams-webhook` | Sends notifications via a provided Microsoft Teams webhook. | | | `opsgenie-webhook` | Sends notifications via a provided Opsgenie webhook. | | | `pager-duty-webhook` | Sends notifications via a provided PagerDuty webhook. | | | `process` | Run a command in a new process. | | | `remote-file-system` | Stores data as a file on any remote file system that supports `fsspec`. | | | `s3` | Stores data as a file on AWS S3. | | | `secret` | Stores a secret value. The value will be obfuscated when this block is logged or shown in the UI. | | | `sendgrid-email` | Sends notifications via Sendgrid email. | | | `slack-webhook` | Sends notifications via a provided Slack webhook. | | | `smb` | Stores data as a file on a SMB

| | | `string` | Stores a string value. | | | `twilio-sms` | Sends notifications via Twilio SMS. | !!! Warning The `S3`, `Azure`, `GCS`, and `GitHub` blocks are deprecated in favor of the the corresponding `S3Bucket`, `AzureBlobStorageCredentials`, `GCSBucket`, and `GitHubRepository` blocks found in the . Some block types that appear in the UI can be created immediately, and then the corresponding integration library must be installed for use. For example, an AWS Secret block can be created, but not used until the is installed. Block types can be created by anyone and optionally shared with the community. You'll find block types available for consumption in many of the published . If a block type is not available in the UI, you can via the CLI. | Integration | Block | Slug | | -- | -- | -- | | | | `ecs-task` | | | | `minio-credentials` | | | | `s3-bucket` | | | | `azure-blob-storage-credentials` | | | | `azure-container-instance-credentials` | | | | `azure-container-instance-job` | | | | `azure-cosmos-db-credentials` | | | | `azureml-credentials` | | | | `bitbucket-credentials` | | | | `bitbucket-repository` | | | | `databricks-credentials` | | | |`dbt-cli-bigquery-target-configs` | | | | `dbt-cli-profile` | | | | `dbt-cloud-credentials` | | | | `dbt-cli-global-configs` | | | | `dbt-cli-postgres-target-configs` | | | | `dbt-cli-snowflake-target-configs` | | | | `dbt-cli-target-configs` | | | | `docker-host` | | | | `docker-registry-credentials` | | | | `email-server-credentials` | | | | `bigquery-warehouse` | | | | `cloud-run-job` | | | | `gcp-credentials` | | | | `gcpsecret` | | | | `gcs-bucket` | | | | `vertex-ai-custom-training-job` | | | | `github-credentials` | | | | `github-repository` | | | | `gitlab-credentials` | | | | `gitlab-repository` | | | | `kubernetes-credentials` | | | | `shell-operation`| | | | `slack-credentials` | | | | `slack-incoming-webhook` | | | | `snowflake-connector` | | | | `snowflake-credentials` | | | | `database-credentials` | | | | `sqlalchemy-connector` | Blocks are classes that subclass the `Block` base class. They can be instantiated and used like normal classes. To instantiate a block that stores a JSON value, use the `JSON` block: ```python from prefect.blocks.system import JSON jsonblock = JSON(value={"theanswer": 42}) ``` To retrieve this saved value use the `.save()` method: ```python json_block.save(name="life-the-universe-everything") ``` To update saved block value stored for a given block, overwrite the existing block by passing `overwrite=True`: ```python json_block.save(overwrite=True) ``` Create a new JSON block by setting the `name` parameter to a new value: ```python json_block.save(name="actually-life-the-universe-everything") ``` Note that blocks can also be created and updated via the . The block name can be used to load the block: ```python hl_lines="6" from prefect import flow from prefect.blocks.system import JSON @flow def whatisthe_answer(): json_block = JSON.load("life-the-universe-everything") print(jsonblock.value["theanswer"]) if name == "main": whatisthe_answer() # 42 ``` Alternatively, load a block with the unique slug that is a combination of the block type slug and the block name. To load our JSON block from above, run the following: ```python hl_lines="3" from prefect.blocks.core import Block json_block = Block.load("json/life-the-universe-everything") print(json_block.value["the-answer"]) #42 ``` Delete a block with the `.delete()` method: ```python from prefect.blocks.core import Block

``` Alternatively, use the CLI to delete specific blocks with a given slug or id: ```bash prefect block delete json/life-the-universe-everything ``` ```bash prefect block delete --id <my-id> ``` To create a custom block type, define a class that subclasses `Block`. The `Block` base class builds on Pydantic's `BaseModel`, so custom blocks can be . Here's a block that represents a cube and holds information about the length of each edge in inches: ```python from prefect.blocks.core import Block class Cube(Block): edgelengthinches: float ``` You can include methods on a block to provide functionality. Here's the same cube block with methods to calculate the volume and surface area of the cube: ```python hl_lines="6-10" from prefect.blocks.core import Block class Cube(Block): edgelengthinches: float def get_volume(self): return self.edgelengthinches3 def getsurfacearea(self): return 6 self.edge_length_inches*2 ``` Use the new `Cube` block type in a flow: ```python from prefect import flow rubikscube = Cube(edgelength_inches=2.25) rubiks_cube.save("rubiks-cube") @flow def calculatecubesurfacearea(cubename): cube = Cube.load(cube_name) print(cube.getsurfacearea()) if name == "main": calculatecubesurface_area("rubiks-cube") # 30.375 ``` All block values are encrypted before being stored. If you have values that you would not like visible in the UI or in logs, use the `SecretStr` field type provided by Pydantic to automatically obfuscate those values. This functionality can be useful for fields that are used to store credentials such as passwords and API tokens. Here's an example of an `AWSCredentials` block that uses `SecretStr`: ```python hl_lines="8" from typing import Optional from prefect.blocks.core import Block from pydantic import SecretStr # if pydantic version >= 2.0, use: from pydantic.v1 import SecretStr class AWSCredentials(Block): awsaccesskey_id: Optional[str] = None awssecretaccess_key: Optional[SecretStr] = None awssessiontoken: Optional[str] = None profile_name: Optional[str] = None region_name: Optional[str] = None ``` Because `awssecretaccess_key` has the `SecretStr` type hint assigned to it, the value of that field will not be exposed if the object is logged: ```python awscredentialsblock = AWSCredentials( awsaccesskey_id="AKIAJKLJKLJKLJKLJKLJK", awssecretaccesskey="secretaccess_key" ) print(awscredentialsblock) ``` Prefect's `SecretDict` field type allows you to add a dictionary field to your block that will have values at all levels automatically obfuscated in the UI or in logs. This functionality is useful for blocks where typing or structure of secret fields is not known until configuration time. Here's an example of a block that uses `SecretDict`: ```python from typing import Dict from prefect.blocks.core import Block from prefect.blocks.fields import SecretDict class SystemConfiguration(Block): system_secrets: SecretDict system_variables: Dict systemconfigurationblock = SystemConfiguration( system_secrets={ "password": "p@ssw0rd", "apitoken": "token123456789", "private_key": "<private key here>", }, system_variables={ "selfdestructcountdown_seconds": 60, "selfdestructcountdownstoptime": 7, }, ) ``` `systemsecrets` will be obfuscated when `systemconfigurationblock` is displayed, but `systemvariables` will be shown in plain-text: ```python print(systemconfigurationblock) ``` The way that a block is displayed can be controlled by metadata fields that can be set on a block type's subclass. Available metadata fields include: | Property | Description | | -- | -- | | \blocktype_name | Display name of the block in the UI. Defaults to the class name. | | \blocktype_slug | Unique slug used to reference the block type in the API. Defaults to a lowercase, dash-delimited version of the block type name. | | \logourl | URL pointing to an image that should be displayed for the block type in the UI. Default to `None`. | | \_description | Short description of block type. Defaults to docstring, if provided. | | \codeexample | Short code snippet shown in UI for how to load/use block

Default to first example provided in the docstring of the class, if provided. | Blocks are composable - a block can be used within other blocks. You can create a block type that uses functionality from another block type by declaring it as an attribute. Nestable blocks are loosely coupled, as configuration can be changed for each block independently. This allows configuration to be shared across multiple use cases. To illustrate, here's a an expanded `AWSCredentials` block that includes the ability to get an authenticated session via the `boto3` library: ```python from typing import Optional import boto3 from prefect.blocks.core import Block from pydantic import SecretStr class AWSCredentials(Block): awsaccesskey_id: Optional[str] = None awssecretaccess_key: Optional[SecretStr] = None awssessiontoken: Optional[str] = None profile_name: Optional[str] = None region_name: Optional[str] = None def getboto3session(self): return boto3.Session( awsaccesskeyid = self.awsaccesskeyid awssecretaccesskey = self.awssecretaccesskey awssessiontoken = self.awssessiontoken profilename = self.profilename region_name = self.region ) ``` The `AWSCredentials` block can be used within an S3Bucket block to provide authentication when interacting with an S3 bucket: ```python hl_lines="5" import io class S3Bucket(Block): bucket_name: str credentials: AWSCredentials def read(self, key: str) -> bytes: s3client = self.credentials.getboto3_session().client("s3") stream = io.BytesIO() s3client.downloadfileobj(Bucket=self.bucket_name, key=key, Fileobj=stream) stream.seek(0) output = stream.read() return output def write(self, key: str, data: bytes) -> None: s3client = self.credentials.getboto3_session().client("s3") stream = io.BytesIO(data) s3client.uploadfileobj(stream, Bucket=self.bucket_name, Key=key) ``` You can use this `S3Bucket` block with previously saved `AWSCredentials` block values in order to interact with the configured S3 bucket: ```python mys3bucket = S3Bucket( bucketname="mys3_bucket", credentials=AWSCredentials.load("myawscredentials") ) mys3bucket.save("mys3bucket") ``` Saving block values like this links the values of the two blocks so that any changes to the values stored for the `AWSCredentials` block with the name `myawscredentials` will be seen the next time that block values for the `S3Bucket` block named `mys3bucket` is loaded. Values for nested blocks can also be hard coded by not first saving child blocks: ```python mys3bucket = S3Bucket( bucketname="mys3_bucket", credentials=AWSCredentials( awsaccesskey_id="AKIAJKLJKLJKLJKLJKLJK", awssecretaccesskey="secretaccess_key" ) ) mys3bucket.save("mys3bucket") ``` In the above example, the values for `AWSCredentials` are saved with `mys3bucket` and will not be usable with any other blocks. Let's add a `bucket_folder` field to your custom `S3Bucket` block that represents the default path to read and write objects from (this field exists on ). Add the new field to the class definition: ```python hl_lines="4" class S3Bucket(Block): bucket_name: str credentials: AWSCredentials bucket_folder: str = None ... ``` Then with either Prefect Cloud or your self-hosted Prefect server instance. If you have any existing blocks of this type that were created before the update and you'd prefer to not re-create them, migrate them to the new version of your block type by adding the missing values: ```python mys3bucket_block = S3Bucket.load("my-s3-bucket", validate=False) mys3bucketblock.bucketpath = "my-default-bucket-path" mys3bucket_block.save("my-s3-bucket", overwrite=True) ``` Prefect comes with many blocks pre-registered and ready to use. If you do not have a block available for use, you can register it. Blocks can be registered from a Python module available in the current environment with a CLI command like this: <div class="terminal"> ```bash prefect block register --module prefect_aws.credentials ``` </div> This command is useful for registering all blocks found within a module in a . Alternatively, if a custom block has been created in a `.py` file, the block can also be registered with the CLI command: <div class="terminal"> ```bash prefect block register --file my_block.py ``` </div> The registered block will then be available for configuration.