Project: StarRocks (App Definition and Development / Database)
File: authenticate_to_azure_storage.md

displayed_sidebar: "English" sidebar_label: Microsoft Azure Storage description: "This topic describes how to authenticate with Azure" From v3.0 onwards, StarRocks can integrate with Microsoft Azure Storage (Azure Blob Storage or Azure Data Lake Storage) in the following scenarios: Batch load data from Azure Storage. Back up data from and restore data to Azure Storage. Query Parquet and ORC files in Azure Storage. Query , , , and tables in Azure Storage. StarRocks supports the following types of Azure Storage accounts: Azure Blob Storage Azure Data Lake Storage Gen1 Azure Data Lake Storage Gen2 In this topic, Hive catalog, file external table, and Broker Load are used as examples to show how StarRocks integrates with Azure Storage by using these types of Azure Storage accounts. For information about the parameters in the examples, see , , and . StarRocks supports using one of the following authentication methods to access Blob Storage: Shared Key SAS Token NOTE When you load data or directly query files from Blob Storage, you must use the wasb or wasbs protocol to access your data: If your storage account allows access over HTTP, use the wasb protocol and write the file path as `wasb://<container>@<storageaccount>.blob.core.windows.net/<path>/<filename>`. If your storage account allows access over HTTPS, use the wasbs protocol and write the file path as `wasbs://<container>@<storageaccount>.blob.core.windows.net/<path>/<filename>`. Configure `azure.blob.storageaccount` and `azure.blob.sharedkey` as follows in the statement: ```SQL CREATE EXTERNAL CATALOG hivecatalogazure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storageaccount" = "<blobstorageaccountname>", "azure.blob.sharedkey" = "<blobstorageaccountshared_key>" ); ``` Configure `azure.blob.storageaccount`, `azure.blob.sharedkey`, and the file path (`path`) as follows in the statement: ```SQL CREATE EXTERNAL TABLE externaltableazure ( id varchar(65500), attributes map<varchar(100), varchar(2000)> ) ENGINE=FILE PROPERTIES ( "path" = "wasb[s]://<container>@<storageaccount>.blob.core.windows.net/<path>/<filename>", "format" = "ORC", "azure.blob.storageaccount" = "<blobstorageaccountname>", "azure.blob.sharedkey" = "<blobstorageaccountshared_key>" ); ``` Configure `azure.blob.storageaccount`, `azure.blob.sharedkey`, and the file path (`DATA INFILE`) as follows in the statement: ```SQL LOAD LABEL test_db.label000 ( DATA INFILE("wasb[s]://<container>@<storageaccount>.blob.core.windows.net/<path>/<filename>") INTO TABLE testingestion2 FORMAT AS "parquet" ) WITH BROKER ( "azure.blob.storageaccount" = "<blobstorageaccountname>", "azure.blob.sharedkey" = "<blobstorageaccountshared_key>" ); ``` Configure `azure.blob.storageaccount`, `azure.blob.container`, and `azure.blob.sastoken` as follows in the statement: ```SQL CREATE EXTERNAL CATALOG hivecatalogazure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storageaccount" = "<blobstorageaccountname>", "azure.blob.container" = "<blobcontainername>", "azure.blob.sastoken" = "<blobstorageaccountSAS_token>" ); ``` Configure `azure.blob.storageaccount`, `azure.blob.container`, `azure.blob.sastoken`, and the file path (`path`) as follows in the statement: ```SQL CREATE EXTERNAL TABLE externaltableazure ( id varchar(65500), attributes map<varchar(100), varchar(2000)> ) ENGINE=FILE PROPERTIES ( "path" = "wasb[s]://<container>@<storageaccount>.blob.core.windows.net/<path>/<filename>", "format" = "ORC", "azure.blob.storageaccount" = "<blobstorageaccountname>", "azure.blob.container" = "<blobcontainername>", "azure.blob.sastoken" = "<blobstorageaccountSAS_token>" ); ``` Configure `azure.blob.storageaccount`, `azure.blob.container`,

and the file path (`DATA INFILE`) as follows in the statement: ```SQL LOAD LABEL test_db.label000 ( DATA INFILE("wasb[s]://<container>@<storageaccount>.blob.core.windows.net/<path>/<filename>") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.blob.storageaccount" = "<blobstorageaccountname>", "azure.blob.container" = "<blobcontainername>", "azure.blob.sastoken" = "<blobstorageaccountSAS_token>" ); ``` StarRocks supports using one of the following authentication methods to access Data Lake Storage Gen1: Managed Service Identity Service Principal NOTE When you load data or query files from Data Lake Storage Gen1, you must use the adl protocol to access your data and write the file path as `adl://<datalakestoragegen1name>.azuredatalakestore.net/<path>/<file_name>`. Configure `azure.adls1.usemanagedservice_identity` as follows in the statement: ```SQL CREATE EXTERNAL CATALOG hivecatalogazure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.usemanagedservice_identity" = "true" ); ``` Configure `azure.adls1.usemanagedservice_identity` and the file path (`path`) as follows in the statement: ```SQL CREATE EXTERNAL TABLE externaltableazure ( id varchar(65500), attributes map<varchar(100), varchar(2000)> ) ENGINE=FILE PROPERTIES ( "path" = "adl://<datalakestoragegen1name>.azuredatalakestore.net/<path>/<file_name>", "format" = "ORC", "azure.adls1.usemanagedservice_identity" = "true" ); ``` Configure `azure.adls1.usemanagedservice_identity` and the file path (`DATA INFILE`) as follows in the statement: ```SQL LOAD LABEL test_db.label000 ( DATA INFILE("adl://<datalakestoragegen1name>.azuredatalakestore.net/<path>/<file_name>") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls1.usemanagedservice_identity" = "true" ); ``` Configure `azure.adls1.oauth2clientid`, `azure.adls1.oauth2credential`, and `azure.adls1.oauth2endpoint` as follows in the statement: ```SQL CREATE EXTERNAL CATALOG hivecatalogazure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2clientid" = "<applicationclientid>", "azure.adls1.oauth2credential" = "<applicationclient_credential>", "azure.adls1.oauth2endpoint" = "<OAuth2.0authorizationendpoint_v2>" ); ``` Configure `azure.adls1.oauth2clientid`, `azure.adls1.oauth2credential`, `azure.adls1.oauth2endpoint`, and the file path (`path`) as follows in the statement: ```SQL CREATE EXTERNAL TABLE externaltableazure ( id varchar(65500), attributes map<varchar(100), varchar(2000)> ) ENGINE=FILE PROPERTIES ( "path" = "adl://<datalakestoragegen1name>.azuredatalakestore.net/<path>/<file_name>", "format" = "ORC", "azure.adls1.oauth2clientid" = "<applicationclientid>", "azure.adls1.oauth2credential" = "<applicationclient_credential>", "azure.adls1.oauth2endpoint" = "<OAuth2.0authorizationendpoint_v2>" ); ``` Configure `azure.adls1.oauth2clientid`, `azure.adls1.oauth2credential`, `azure.adls1.oauth2endpoint`, and the file path (`DATA INFILE`) as follows in the statement: ```SQL LOAD LABEL test_db.label000 ( DATA INFILE("adl://<datalakestoragegen1name>.azuredatalakestore.net/<path>/<file_name>") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls1.oauth2clientid" = "<applicationclientid>", "azure.adls1.oauth2credential" = "<applicationclient_credential>", "azure.adls1.oauth2endpoint" = "<OAuth2.0authorizationendpoint_v2>" ); ``` StarRocks supports using one of the following authentication methods to access Data Lake Storage Gen2: Managed Identity Shared Key Service Principal NOTE When you load data or query files from Data Lake Storage Gen2, you must use the abfs or abfss protocol to access your data: If your storage account allows access over HTTP, use the abfs protocol and write the file path as `abfs://<container>@<storageaccount>.dfs.core.windows.net/<path>/<filename>`. If your storage account allows access over HTTPS, use the abfss protocol and write the file path as `abfss://<container>@<storageaccount>.dfs.core.windows.net/<path>/<filename>`. Before you start, you need to make the following preparations: Edit the virtual machines (VMs) on which your StarRocks cluster is deployed. Add the managed identities to these

Make sure that the managed identities are associated with the role (Storage Blob Data Reader) authorized to read data in your storage account. Configure `azure.adls2.oauth2usemanagedidentity`, `azure.adls2.oauth2tenantid`, and `azure.adls2.oauth2client_id` as follows in the statement: ```SQL CREATE EXTERNAL CATALOG hivecatalogazure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2usemanaged_identity" = "true", "azure.adls2.oauth2tenantid" = "<serviceprincipaltenant_id>", "azure.adls2.oauth2clientid" = "<serviceclientid>" ); ``` Configure `azure.adls2.oauth2usemanagedidentity`, `azure.adls2.oauth2tenantid`, `azure.adls2.oauth2client_id`, and the file path (`path`) as follows in the statement: ```SQL CREATE EXTERNAL TABLE externaltableazure ( id varchar(65500), attributes map<varchar(100), varchar(2000)> ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://<container>@<storageaccount>.dfs.core.windows.net/<path>/<filename>", "format" = "ORC", "azure.adls2.oauth2usemanaged_identity" = "true", "azure.adls2.oauth2tenantid" = "<serviceprincipaltenant_id>", "azure.adls2.oauth2clientid" = "<serviceclientid>" ); ``` Configure `azure.adls2.oauth2usemanagedidentity`, `azure.adls2.oauth2tenantid`, `azure.adls2.oauth2client_id`, and the file path (`DATA INFILE`) as follows in the statement: ```SQL LOAD LABEL test_db.label000 ( DATA INFILE("adfs[s]://<container>@<storageaccount>.dfs.core.windows.net/<path>/<filename>") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.oauth2usemanaged_identity" = "true", "azure.adls2.oauth2tenantid" = "<serviceprincipaltenant_id>", "azure.adls2.oauth2clientid" = "<serviceclientid>" ); ``` Configure `azure.adls2.storageaccount` and `azure.adls2.sharedkey` as follows in the statement: ```SQL CREATE EXTERNAL CATALOG hivecatalogazure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storageaccount" = "<storageaccount_name>", "azure.adls2.sharedkey" = "<sharedkey>" ); ``` Configure `azure.adls2.storageaccount`, `azure.adls2.sharedkey`, and the file path (`path`) as follows in the statement: ```SQL CREATE EXTERNAL TABLE externaltableazure ( id varchar(65500), attributes map<varchar(100), varchar(2000)> ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://<container>@<storageaccount>.dfs.core.windows.net/<path>/<filename>", "format" = "ORC", "azure.adls2.storageaccount" = "<storageaccount_name>", "azure.adls2.sharedkey" = "<sharedkey>" ); ``` Configure `azure.adls2.storageaccount`, `azure.adls2.sharedkey`, and the file path (`DATA INFILE`) as follows in the statement: ```SQL LOAD LABEL test_db.label000 ( DATA INFILE("adfs[s]://<container>@<storageaccount>.dfs.core.windows.net/<path>/<filename>") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.storageaccount" = "<storageaccount_name>", "azure.adls2.sharedkey" = "<sharedkey>" ); ``` Before you start, you need to create a service principal, create a role assignment to assign a role to the service principal, and then add the role assignment to your storage account. As such, you can make sure that this service principal can successfully access the data in your storage account. Configure `azure.adls2.oauth2clientid`, `azure.adls2.oauth2clientsecret`, and `azure.adls2.oauth2clientendpoint` as follows in the statement: ```SQL CREATE EXTERNAL CATALOG hivecatalogazure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2clientid" = "<serviceclientid>", "azure.adls2.oauth2clientsecret" = "<serviceprincipalclient_secret>", "azure.adls2.oauth2clientendpoint" = "<serviceprincipalclient_endpoint>" ); ``` Configure `azure.adls2.oauth2clientid`, `azure.adls2.oauth2clientsecret`, `azure.adls2.oauth2clientendpoint`, and the file path (`path`) as follows in the statement: ```SQL CREATE EXTERNAL TABLE externaltableazure ( id varchar(65500), attributes map<varchar(100), varchar(2000)> ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://<container>@<storageaccount>.dfs.core.windows.net/<path>/<filename>", "format" = "ORC", "azure.adls2.oauth2clientid" = "<serviceclientid>", "azure.adls2.oauth2clientsecret" = "<serviceprincipalclient_secret>", "azure.adls2.oauth2clientendpoint" = "<serviceprincipalclient_endpoint>" ); ``` Configure `azure.adls2.oauth2clientid`, `azure.adls2.oauth2clientsecret`, `azure.adls2.oauth2clientendpoint`, and the file path (`DATA INFILE`) as follows in the statement: ```SQL LOAD LABEL test_db.label000 ( DATA INFILE("adfs[s]://<container>@<storageaccount>.dfs.core.windows.net/<path>/<filename>") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.oauth2clientid" = "<serviceclientid>", "azure.adls2.oauth2clientsecret" = "<serviceprincipalclient_secret>", "azure.adls2.oauth2clientendpoint" = "<serviceprincipalclient_endpoint>" ); ```