Project: Alluxio (Runtime / Cloud Native Storage)
File: Azure-Data-Lake.md

layout: global title: Azure Data Lake Storage Gen1 This guide describes how to configure Alluxio with {:target="_blank"} as the under storage system. Azure Data Lake Storage is an enterprise-wide hyper-scale repository for big data analytic workloads. Azure Data Lake enables you to capture data of any size, type, and ingestion speed in one single place for operational and exploratory analytics. It is designed to store and analyze large amounts of structured, semi-structured, and unstructured data. For more information about Azure Data Lake Storage Gen1, please read its {:target="_blank"}. Note: Azure Data Lake Storage Gen1 will be retired on Feb 29, 2024. Be sure to migrate to Azure Data Lake Storage Gen2 prior to that date. See how {:target="_blank"}. If you haven't already, please see before you get started. In preparation for using Azure Data Lake Storage Gen1 with Alluxio, [create a new Data Lake storage in your Azure account](https://docs.microsoft.com/en-us/azure/data-lake-store/data-lake-store-get-started-portal){:target="_blank"} or use an existing Data Lake storage. <table class="table table-striped"> <tr> <td markdown="span" style="width:30%">`<AZURE_DIRECTORY>`</td> <td markdown="span">The directory you want to use, either by creating a new directory or using an existing one</td> </tr> <tr> <td markdown="span" style="width:30%">`<AZURE_ACCOUNT>`</td> <td markdown="span">Your Azure storage account</td> </tr> <tr> <td markdown="span" style="width:30%">`<CLIENT_ID>`</td> <td markdown="span">See {:target="_blank"} for instructions on how to retrieve the application (client) ID and authentication key (also called the client secret) for your application</td> </tr> <tr> <td markdown="span" style="width:30%">`<AUTHENTICATION_KEY>`</td> <td markdown="span">See {:target="_blank"} for instructions on how to retrieve the application (client) ID and authentication key (also called the client secret) for your application</td> </tr> <tr> <td markdown="span" style="width:30%">`<TENANT_ID>`</td> <td markdown="span">See {:target="_blank"} for instructions on how to retrieve the tenant ID</td> </tr> </table> You also need to set up {:target="_blank"} for your storage account. To use Azure Data Lake Storage Gen1 as the UFS of Alluxio root mount point, you need to configure Alluxio to use under storage systems by modifying `conf/alluxio-site.properties`. If it does not exist, create the configuration file from the template. ```shell $ cp conf/alluxio-site.properties.template conf/alluxio-site.properties ``` Specify the underfs address by modifying `conf/alluxio-site.properties` to include: ```properties alluxio.dora.client.ufs.root=adl://<AZUREACCOUNT>.azuredatalakestore.net/<AZUREDIRECTORY>/ ``` Specify the application ID, authentication key and tenant ID for the Azure AD application used for the Azure account of the root mount point by adding the following properties in `conf/alluxio-site.properties`: ```properties fs.adl.account.<AZUREACCOUNT>.oauth2.client.id=<CLIENTID> fs.adl.account.<AZUREACCOUNT>.oauth2.credential=<AUTHENTICATIONKEY> fs.adl.account.<AZUREACCOUNT>.oauth2.refresh.url=https://login.microsoftonline.com/<TENANTID>/oauth2/token ``` After these changes, Alluxio should be configured to work with Azure Data Lake storage as its under storage system, and you can run Alluxio locally with it. Once you have configured Alluxio to Azure Data Lake Storage Gen1, try to see that everything works.