Project: Prefect (Orchestration & Management / Scheduling & Orchestration)
File: moving-data.md

description: Moving data to and from cloud providers tags: data storage read data write data cloud providers AWS S3 Azure Storage Azure Blob Storage Azure GCP Google Cloud Storage GCS moving data search: boost: 2 Learn how to use Prefect to move data to and from AWS, Azure, and GCP blob storage. Prefect Authenticated with (or self-hosted instance) A cloud provider account (for example, ) In the CLI, install the Prefect integration library for your cloud provider: === "AWS" provides blocks for interacting with AWS services. <div class="terminal"> ```bash pip install -U prefect-aws ``` </div> === "Azure" provides blocks for interacting with Azure services. <div class="terminal"> ```bash pip install -U prefect-azure ``` </div> === "GCP" provides blocks for interacting with GCP services. <div class="terminal"> ```bash pip install -U prefect-gcp ``` </div> Register the new block types with Prefect Cloud (or with your self-hosted Prefect server instance): === "AWS" <div class="terminal"> ```bash prefect block register -m prefect_aws ``` </div> === "Azure" <div class="terminal"> ```bash prefect block register -m prefect_azure ``` </div> === "GCP" <div class="terminal"> ```bash prefect block register -m prefect_gcp ``` </div> A confirmation message in the CLI shows that several block types were registered. The UI shows the new block types listed. Create a storage bucket in the cloud provider account. Ensure the bucket is publicly accessible, or create a user or service account with the appropriate permissions to fetch and write data to the bucket. If the bucket is private, there are several options to authenticate: At deployment runtime, ensure the runtime environment is authenticated. Create a block with configuration details and reference it when creating the storage block. If you saved credential details in a block, you can use a credentials block specific to the cloud provider or use a more generic secret block. You can create through the UI or Python code. Below you use Python code to create a credentials block for your cloud provider. !!! Warning "Credentials safety" Reminder, don't store credential values in public locations such as public git platform repositories. The examples below use environment variables to store credential values. === "AWS" ```python import os from prefect_aws import AwsCredentials myawscreds = AwsCredentials( awsaccesskey_id="123abc", awssecretaccesskey=os.environ.get("MYAWSSECRETACCESS_KEY"), ) myawscreds.save(name="my-aws-creds-block", overwrite=True) ``` === "Azure" ```python import os from prefect_azure import AzureBlobStorageCredentials myazurecreds = AzureBlobStorageCredentials( connectionstring=os.environ.get("MYAZURECONNECTIONSTRING"), ) myazurecreds.save(name="my-azure-creds-block", overwrite=True) ``` === "GCP" We recommend specifying the service account key file contents as a string, rather than the path to the file, because that file might not be available in your production environments. ```python import os from prefect_gcp import GCPCredentials mygcpcreds = GCPCredentials( serviceaccountinfo=os.environ.get("GCPSERVICEACCOUNTKEYFILE_CONTENTS"), ) mygcpcreds.save(name="my-gcp-creds-block", overwrite=True) ``` Run the code to create the block. You should see a message that the block was created. You can create a block for the chosen cloud provider using Python code or the

This example uses Python code. === "AWS" Note that the `S3Bucket` block is not the same as the `S3` block that ships with Prefect. The `S3Bucket` block used in this example is part of the `prefect-aws` library and provides additional functionality. Next, reference the credentials block created above. ```python from prefect_aws import S3Bucket s3bucket = S3Bucket.create( bucket="my-bucket-name", credentials="my-aws-creds-block" ) s3bucket.save(name="my-s3-bucket-block", overwrite=True) ``` === "Azure" Note that the `AzureBlobStorageCredentials` block is not the same as the Azure block that ships with Prefect. The `AzureBlobStorageCredentials` block used in this example is part of the `prefect-azure` library and provides additional functionality. Azure blob storage doesn't require a separate block. The connection string used in the `AzureBlobStorageCredentials` block can encode the information needed. === "GCP" Note that the `GcsBucket` block is not the same as the `GCS` block that ships with Prefect. The `GcsBucket` block is part of the `prefect-gcp` library and provides additional functionality. Next, reference the credentials block created above. ```python from prefectgcp.cloudstorage import GcsBucket gcsbucket = GcsBucket( bucket="my-bucket-name", credentials="my-gcp-creds-block" ) gcsbucket.save(name="my-gcs-bucket-block", overwrite=True) ``` Run the code to create the block. You should see a message that the block was created. Use your new block inside a flow to write data to your cloud provider. === "AWS" ```python from pathlib import Path from prefect import flow from prefect_aws.s3 import S3Bucket @flow() def uploadtos3(): """Flow function to upload data""" path = Path("mypathto/my_file.parquet") aws_block = S3Bucket.load("my-s3-bucket-block") awsblock.uploadfrompath(frompath=path, to_path=path) if name == "main": uploadtos3() ``` === "Azure" ```python from prefect import flow from prefect_azure import AzureBlobStorageCredentials from prefectazure.blobstorage import blobstorageupload @flow def uploadtoazure(): """Flow function to upload data""" blobstoragecredentials = AzureBlobStorageCredentials.load( name="my-azure-creds-block" ) with open("mypathto/my_file.parquet", "rb") as f: blobstorageupload( data=f.read(), container="my_container", blob="mypathto/my_file.parquet", blobstoragecredentials=blobstoragecredentials, ) if name == "main": uploadtoazure() ``` === "GCP" ```python from pathlib import Path from prefect import flow from prefectgcp.cloudstorage import GcsBucket @flow() def uploadtogcs(): """Flow function to upload data""" path = Path("mypathto/my_file.parquet") gcs_block = GcsBucket.load("my-gcs-bucket-block") gcsblock.uploadfrompath(frompath=path, to_path=path) if name == "main": uploadtogcs() ``` Use your block to read data from your cloud provider inside a flow. === "AWS" ```python from prefect import flow from prefect_aws import S3Bucket @flow def downloadfroms3(): """Flow function to download data""" s3_block = S3Bucket.load("my-s3-bucket-block") s3block.getdirectory( frompath="mypathto/myfile.parquet", localpath="mypathto/myfile.parquet" ) if name == "main": downloadfroms3() ``` === "Azure" ```python from prefect import flow from prefect_azure import AzureBlobStorageCredentials from prefectazure.blobstorage import blobstoragedownload @flow def downloadfromazure(): """Flow function to download data""" blobstoragecredentials = AzureBlobStorageCredentials.load( name="my-azure-creds-block" ) blobstoragedownload( blob="mypathto/my_file.parquet", container="my_container", blobstoragecredentials=blobstoragecredentials, ) if name == "main": downloadfromazure() ``` === "GCP" ```python from prefect import flow from prefectgcp.cloudstorage import GcsBucket @flow def downloadfromgcs(): gcs_block = GcsBucket.load("my-gcs-bucket-block") gcsblock.getdirectory( frompath="mypathto/myfile.parquet", localpath="mypathto/myfile.parquet" ) if name == "main": downloadfromgcs() ``` Check out the , , and docs to see additional methods for interacting with cloud storage providers. Each library also contains blocks for interacting with other cloud-provider services.