Package: adlfs
Filesystem interface to Azure-Datalake Gen1 and Gen2 StorageQuickstartThis package can be installed using:pip install adlfsorconda install -c conda-forge adlfsTheadl://andabfs://protocols are included in fsspec's known_implementations registry
in fsspec > 0.6.1, otherwise users must explicitly inform fsspec about the supported adlfs protocols.To use the Gen1 filesystem:importdask.dataframeasddstorage_options={'tenant_id':TENANT_ID,'client_id':CLIENT_ID,'client_secret':CLIENT_SECRET}dd.read_csv('adl://{STORE_NAME}/{FOLDER}/*.csv',storage_options=storage_options)To use the Gen2 filesystem you can use the protocolabfsoraz:importdask.dataframeasddstorage_options={'account_name':ACCOUNT_NAME,'account_key':ACCOUNT_KEY}ddf=dd.read_csv('abfs://{CONTAINER}/{FOLDER}/*.csv',storage_options=storage_options)ddf=dd.read_parquet('az://{CONTAINER}/folder.parquet',storage_options=storage_options)Acceptedprotocol/uriformatsinclude:'PROTOCOL://container/path-part/file''PROTOCOL://container@account.dfs.core.windows.net/path-part/file'oroptionally,ifAZURE_STORAGE_ACCOUNT_NAMEandanAZURE_STORAGE_<CREDENTIAL>issetasanenvironmentalvariable,thenstorage_optionswillbereadfromtheenvironmentalvariablesTo read from a public storage blob you are required to specify the'account_name'.
For example, you can accessNYC Taxi & Limousine Commissionas:storage_options={'account_name':'azureopendatastorage'}ddf=dd.read_parquet('az://nyctlc/green/puYear=2019/puMonth=*/*.parquet',storage_options=storage_options)DetailsThe package includes pythonic filesystem implementations for both
Azure Datalake Gen1 and Azure Datalake Gen2, that facilitate
interactions between both Azure Datalake implementations and Dask.  This is done leveraging theintake/filesystem_specbase class and Azure Python SDKs.Operations against both Gen1 Datalake currently only work with an Azure ServicePrincipal
with suitable credentials to perform operations on the resources of choice.Operations against the Gen2 Datalake are implemented by leveragingAzure Blob Storage Python SDK.Setting credentialsThestorage_optionscan be instantiated with a variety of keyword arguments depending on the filesystem. The most commonly used arguments are:connection_stringaccount_nameaccount_keysas_tokentenant_id,client_id, andclient_secretare combined for an Azure ServicePrincipal e.g.storage_options={'account_name': ACCOUNT_NAME, 'tenant_id': TENANT_ID, 'client_id': CLIENT_ID, 'client_secret': CLIENT_SECRET}anon:TrueorFalse. The default value for anon (i.e. anonymous) is Truelocation_mode: valid values are "primary" or "secondary" and apply to RA-GRS accountsFor more argument details see all arguments forAzureBlobFileSystemhereandAzureDatalakeFileSystemhere.The following environmental variables can also be set and picked up for authentication:"AZURE_STORAGE_CONNECTION_STRING""AZURE_STORAGE_ACCOUNT_NAME""AZURE_STORAGE_ACCOUNT_KEY""AZURE_STORAGE_SAS_TOKEN""AZURE_STORAGE_TENANT_ID""AZURE_STORAGE_CLIENT_ID""AZURE_STORAGE_CLIENT_SECRET"The filesystem can be instantiated for different use cases based on a variety ofstorage_optionscombinations. The following list describes some common use cases utilizingAzureBlobFileSystem, i.e. protocolsabfsoraz. Note that all cases require theaccount_nameargument to be provided:Anonymous connection to public container:storage_options={'account_name': ACCOUNT_NAME, 'anon': True}will assume theACCOUNT_NAMEpoints to a public container, and attempt to use an anonymous login. Note, the default value foranonis True.Auto credential solving using Azure's DefaultAzureCredential() library:storage_options={'account_name': ACCOUNT_NAME, 'anon': False}will useDefaultAzureCredentialto get valid credentials to the containerACCOUNT_NAME.DefaultAzureCredentialattempts to authenticate via themechanisms and order visualized here.Azure ServicePrincipal:tenant_id,client_id, andclient_secretare all used as credentials for an Azure ServicePrincipal: e.g.storage_options={'account_name': ACCOUNT_NAME, 'tenant_id': TENANT_ID, 'client_id': CLIENT_ID, 'client_secret': CLIENT_SECRET}.Append BlobTheAzureBlobFileSystemacceptsall of the Async BlobServiceClient arguments.By default, write operations create BlockBlobs in Azure, which, once written can not be appended. It is possible to create an AppendBlob usingmode="ab"when creating and operating on blobs. Currently, AppendBlobs are not available if hierarchical namespaces are enabled.