Package: pyarrowfs-adlgen2
pyarrowfs-adlgen2pyarrowfs-adlgen2 is an implementation of a pyarrow filesystem for Azure Data Lake Gen2.It allows you to use pyarrow and pandas to read parquet datasets directly from Azure without the need to copy files to local storage first.Installationpip install pyarrowfs-adlgen2Reading datasetsExample usage with pandas dataframe:importazure.identityimportpandasaspdimportpyarrow.fsimportpyarrowfs_adlgen2handler=pyarrowfs_adlgen2.AccountHandler.from_account_name('YOUR_ACCOUNT_NAME',azure.identity.DefaultAzureCredential())fs=pyarrow.fs.PyFileSystem(handler)df=pd.read_parquet('container/dataset.parq',filesystem=fs)Example usage with arrow tables:importazure.identityimportpyarrow.datasetimportpyarrow.fsimportpyarrowfs_adlgen2handler=pyarrowfs_adlgen2.AccountHandler.from_account_name('YOUR_ACCOUNT_NAME',azure.identity.DefaultAzureCredential())fs=pyarrow.fs.PyFileSystem(handler)ds=pyarrow.dataset.dataset('container/dataset.parq',filesystem=fs)table=ds.to_table()Configuring timeoutsTimeouts are passed to azure-storage-file-datalake SDK methods. The timeout unit is in seconds.importazure.identityimportpyarrowfs_adlgen2handler=pyarrowfs_adlgen2.AccountHandler.from_account_name('YOUR_ACCOUNT_NAME',azure.identity.DefaultAzureCredential(),timeouts=pyarrowfs_adlgen2.Timeouts(file_system_timeout=10))# or mutate it:handler.timeouts.file_client_timeout=20Writing datasetsWith pyarrow version 3 or greater, you can write datasets from arrow tables:importpyarrowaspaimportpyarrow.datasetpyarrow.dataset.write_dataset(table,'name.pq',format='parquet',partitioning=pyarrow.dataset.partitioning(schema=pyarrow.schema([('year',pa.int32())]),flavor='hive'),filesystem=pyarrow.fs.PyFileSystem(handler))With earlier versions, files must be opened/written one at a time:As of pyarrow version 1.0.1,pyarrow.parquet.ParquetWriterdoes not supportpyarrow.fs.PyFileSystem, but data can be written to open files:withfs.open_output_stream('container/out.parq')asout:df.to_parquet(out)Or with arrow tables:importpyarrow.parquetwithfs.open_output_stream('container/out.parq')asout:pyarrow.parquet.write_table(table,out)Accessing only a single container/file-systemIf you do not want, or can't access the whole storage account as a single filesystem, you can usepyarrowfs_adlgen2.FilesystemHandlerto view a single file system within an account:importazure.identityimportpyarrowfs_adlgen2handler=pyarrowfs_adlgen2.FilesystemHandler.from_account_name("STORAGE_ACCOUNT","FS_NAME",azure.identity.DefaultAzureCredential())All access is done through the file system within the storage account.Set http headers for files for pyarrow >= 5You can set headers for any output files by using themetadataargument tohandler.open_output_stream:importpyarrowfs_adlgen2fs=pyarrowfs_adlgen2.AccountHandler.from_account_name("theaccount").to_fs()metadata={"content_type":"application/json"}withfs.open_output_stream("container/data.json",metadata)asout:out.write("{}")Note that the spelling is different than you might expect! For a list of valid keys, seeContentSettings.You can do this for pyarrow >= 5 when usingpyarrow.fs.PyFileSystem, and for any pyarrow if using the handlers
from pyarrowfs_adlgen2 directly.Running testsTo run the integration tests, you need:Azure Storage Account V2 with hierarchial namespace enabled (Data Lake gen2 account)To configure azure login (f. ex. use$ az loginor set up environment variables, seeazure.identity.DefaultAzureCredential)Install pytest, f. ex.pip install pytestNB! All data in the storage account is deleted during testing, USE AN EMPTY ACCOUNTAZUREARROWFS_TEST_ACT=thestorageaccount pytest