Project: Flink (App Definition and Development / Streaming & Messaging)
File: azure.md

title: Azure Blob Storage weight: 4 type: docs aliases: /deployment/filesystems/azure.html /ops/filesystems/azure <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> is a Microsoft-managed service providing cloud storage for a variety of use cases. You can use Azure Blob Storage with Flink for reading and writing data as well in conjunction with the Flink supports accessing Azure Blob Storage using both or . {{< hint info >}} Azure recommends using abfs:// for accessing ADLS Gen2 storage accounts even though wasb:// works through backward compatibility. {{< /hint >}} {{< hint warning >}} abfs:// can be used for accessing the ADLS Gen2 storage accounts only. Please visit Azure documentation on how to identify ADLS Gen2 storage account. {{< /hint >}} You can use Azure Blob Storage objects like regular files by specifying paths in the following format: ```plain // WASB unencrypted access wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path> // WASB SSL encrypted access wasbs://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path> // ABFS unecrypted access abfs://<your-container>@$<your-azure-account>.dfs.core.windows.net/<object-path> // ABFS SSL encrypted access abfss://<your-container>@$<your-azure-account>.dfs.core.windows.net/<object-path> ``` See below for how to use Azure Blob Storage in a Flink job: ```java // Read from Azure Blob storage env.readTextFile("wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>"); // Write to Azure Blob storage stream.writeAsText("wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>"); // Use Azure Blob Storage as checkpoint storage Configuration config = new Configuration(); config.set(CheckpointingOptions.CHECKPOINT_STORAGE, "filesystem"); config.set(CheckpointingOptions.CHECKPOINTS_DIRECTORY, "wasb://<your-container>@$<your-azure-account>.blob.core.windows.net/<object-path>"); env.configure(config); ``` To use `flink-azure-fs-hadoop`, copy the respective JAR file from the `opt` directory to the `plugins` directory of your Flink distribution before starting Flink, e.g. ```bash mkdir ./plugins/azure-fs-hadoop cp ./opt/flink-azure-fs-hadoop-{{< version >}}.jar ./plugins/azure-fs-hadoop/ ``` `flink-azure-fs-hadoop` registers default FileSystem wrappers for URIs with the wasb:// and wasbs:// (SSL encrypted access) scheme. Hadoop's WASB Azure Filesystem supports configuration of credentials via the Hadoop configuration as outlined in the . For convenience Flink forwards all Flink configurations with a key prefix of `fs.azure` to the Hadoop configuration of the filesystem. Consequently, the azure blob storage key can be configured in via: ```yaml fs.azure.account.key.<accountname>.blob.core.windows.net: <azurestorage_key> ``` Alternatively, the filesystem can be configured to read the Azure Blob Storage key from an environment variable `AZURESTORAGEKEY` by setting the following configuration keys in . ```yaml fs.azure.account.keyprovider.<account_name>.blob.core.windows.net: org.apache.flink.fs.azurefs.EnvironmentVariableKeyProvider ``` Hadoop's ABFS Azure Filesystem supports several ways of configuring authentication. Please visit the documentation on how to configure. {{< hint info >}} Azure recommends using Azure managed identity to access the ADLS Gen2 storage accounts using abfs. Please refer to for more details. Please visit the for the list of services that support Managed Identities. Flink clusters deployed in those Azure services can take advantage of Managed Identities. {{< /hint >}} Azure blob storage key can be configured in via: ```yaml fs.azure.account.key.<accountname>.dfs.core.windows.net: <azurestorage_key> ``` {{< top >}}