Package: cloud-arrow
Cloud Arrow
Python Library to read and write Parquet and Deltalake files from ADLS and GCFS 

                                Dic 2023
		        Alfredo Lorie Bernardo, Ignacio Rodriguez Sanchez					

                             version 0.6.0IntroductionCloud Arrowis a python library to provide read and write capabilities forParquetandDeltalakefiles
(without relying on spark) from/to the main cloud providers object storage service (Azure ADLSGen2, Google GCSFS, AWS S3 -> coming soon)
and the Local FileSystem.
The main goal is to provide a single and unified API for reading and writing files from python programs.
This library is available in PyPI and distributed under the GNU license.4Up to date, Cloud Arrow heavily relies on the Apache Arrow under the hood and some fsspec-compatible filesystems implementations to connect
and interact with the cloud providers object storage.DownloadGitHub:https://github.com/a24lorie/Cloud-ArrowHighlightsFast file access with transparent multi-threading.Support for reading and writing partitioned datasetsFiltering and partition pruning to avoid unnecessary data movementRead, write, and manage Delta Lake tables with Python or Rust without Spark or JavaUsing Cloud-ArrowReading from ADLSGen2To read a parquet o deltalake files from Azure ADLSGen2 we first must need to create an instance of ADLSObjectStorage
using the client-secret authentication mechanism by providing the following arguments:tentant_idRequired - The Azure Tenant Id used in oauth flowsclient_id: Required - Service principal client id for authorizing requestsclient_secret: Required - Service principal client secret for authorizing requestsaccount_name: Required - The name of the azure storage accountcontainer: Required - Container namefromcloud_arrow.adlsimportADLSObjectStorageadls_object_storage=ADLSObjectStorage(tenant_id="AzureTentantId",client_id="AzureClientId",client_secret="AzureClientSecret",account_name="storage-account",container="container")Use the adls_object_storage object created and call the read_to_pandas method to retrieve files in a pandas DataFrame by specifying:file_format: Required - can be on of "parquet" or "deltalake"path: Required - The object storage location to the dataset (can be a single file name or directory name)filters: Optional - pyarrow.compute.Expression or List[Tuple] or List[List[Tuple]]. Seepyarrow filterfor more detailsresultParquet=adls_object_storage.read_to_pandas(file_format="parquet",path="path_to_parquet")resultDelta=adls_object_storage.read_to_pandas(file_format="deltalake",path="path_to_deltalake")print(resultParquet.info())print(resultDelta.info())Writing to ADLS Gen2To write a parquet o deltalake files from Azure ADLSGen2 we first must need to create an instance of ADLSObjectStorage
using the client-secret authentication mechanism by providing the following arguments:tentant_idRequired - The Azure Tenant Id used in oauth flowsclient_id: Required - Service principal client id for authorizing requestsclient_secret: Required - Service principal client secret for authorizing requestsaccount_name: Required - The name of the azure storage accountcontainer: Required - Container namefromcloud_arrow.adlsimportADLSObjectStoragefromcloud_arrow.coreimportDeltaLakeWriteOptionsfromcloud_arrow.coreimportParquetWriteOptionsadls_object_storage=ADLSObjectStorage(tenant_id="AzureTentantId",client_id="AzureClientId",client_secret="AzureClientSecret",account_name="storage-account",container="container")Use the adls_object_storage object created and call the write method to save the dataset in parquet or deltalake format to ADLSGen2 by specifying::table: Required - pandas DataFrame or Arrow Tablefile_format: Required -  Required - can be on of "parquet" or "deltalake"path: Required - The object storage location to store the datasetwrite_options: Required - Options to write the files including:partitions: Allows to specify which columns to use to split the dataset.compression_codec: Allow to specify the compression codec (None, snappy, sz4, brotli, gzip, zstd)existing_data_behavior:  Allows to specify how to handle data that already exists in the destination.# Example writing parquetadls_object_storage.write(table=table,file_format="parquet",path=path,write_options=ParquetWriteOptions(partitions=["col1","col2",...],compression_codec="snappy",existing_data_behavior="overwrite_or_ignore")# 'error', 'overwrite_or_ignore', 'delete_matching')# Example writing deltalakeadls_object_storage.write(table=table,file_format="deltalake",path=path,write_options=DeltaLakeWriteOptions(partitions=["col1","col2",...],compression_codec="snappy",existing_data_behavior="overwrite")# 'error', 'append', 'overwrite', 'ignore')Reading from GCSFSTo write a parquet o deltalake files from Google GCSFS we first must need to create an instance of GCSFSObjectStorage
by providing the following arguments:projectRequired - Project id of the Google Cloud projectaccess: Required - Access method to the file system (read_only, read_write or full_control)token: Required - Authentication method for Google Cloud Storage (None, google_default, cache, anon, browser, cloud or credentials filename path). SeeGCSFileSystemfor more details.bucket: Required - Name of the bucket or container that hold the data.default_location: Optional - Default location where buckets are created.fromcloud_arrow.gcsfsimportGCSFSObjectStoragegcsfs_object_storage=GCSFSObjectStorage(project="GCSFSProjectId",access="read_only",bucket="GCSFSBucket",token="/path/google-secret.json",default_location="")google-secret.jsonIn the previous example thecredentials filename pathwas used to authenticate. You can create thegoogle-secret.jsonfilehere.
You don't need to manually fill in JSON by hand, the below example is provided to show you what the end result should look like.
You should be able to read, write, and delete objects from at least one bucket.{"type":"service_account","project_id":"$YOUR_GOOGLE_PROJECT_ID","private_key_id":"...","private_key":"...","client_email":"...","client_id":"...","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://accounts.google.com/o/oauth2/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_x509_cert_url":""}Use the gcsfs_object_storage object created and call the read_to_pandas method to retrieve files in a pandas DataFrame by specifying:file_format: Required - can be on of "parquet" or "deltalake"path: Required - The object storage location to the dataset (can be a single file name or directory name)filters: Optional - pyarrow.compute.Expression or List[Tuple] or List[List[Tuple]]. Seepyarrow filterfor more detailsresultParquet=gcsfs_object_storage.read_to_pandas(file_format="parquet",path="path_to_parquet")resultDelta=gcsfs_object_storage.read_to_pandas(file_format="deltalake",path="path_to_deltalake")print(resultParquet.info())print(resultDelta.info())Writing to GCSFSWhen creating a cloud-arrow writer to Google Cloud Storage create an instance of GCSFSObjectStorage with the arguments:projectRequired - Project id of the Google Cloud projectaccess: Required - Access method to the file system (read_only, read_write or full_control)token: Required - Authentication method for Google Cloud Storage (None, google_default, cache, anon, browser, cloud or credentials filename path).
SeeGCSFileSystemfor more details.bucket: Required - Name of the bucket or container that hold the data.default_location: Optional - Default location where buckets are created.fromcloud_arrow.gcsfsimportGCSFSObjectStoragefromcloud_arrow.coreimportDeltaLakeWriteOptionsfromcloud_arrow.coreimportParquetWriteOptionsgcsfs_object_storage=GCSFSObjectStorage(project="GCSFSProjectId",access="read_only",bucket="GCSFSBucket",token="/path/google-secret.json",default_location="")Use the gcsfs_object_storage object created and call the write method to save the dataset in parquet or deltalake format to GCSFS by specifying::table: Required - pandas DataFrame or Arrow Tablefile_format: Required -  Required - can be on of "parquet" or "deltalake"path: Required - The object storage location to store the datasetwrite_options: Required - Options to write the files including:partitions: Allows to specify which columns to use to split the dataset.compression_codec: Allow to specify the compression codec (None, snappy, sz4, brotli, gzip, zstd)existing_data_behavior:  Allows to specify how to handle data that already exists in the destination.# Example writing parquetgcsfs_object_storage.write(table=table,file_format="parquet",path=path,write_options=ParquetWriteOptions(partitions=["col1","col2",...],compression_codec="snappy",existing_data_behavior="overwrite_or_ignore")# 'error', 'overwrite_or_ignore', 'delete_matching')# Example writing deltalakegcsfs_object_storage.write(table=table,file_format="deltalake",path=path,write_options=DeltaLakeWriteOptions(partitions=["col1","col2",...],compression_codec="snappy",existing_data_behavior="overwrite")# 'error', 'append', 'overwrite', 'ignore')ContributingThe Cloud Arrow library welcomes contributors from all developers, regardless of your experience or programming background.
If you find a bug, send apull requestand we'll discuss things. If you are not familiar with "pull request" term I recommend reading the followingarticlefor better understanding
We value kind communication and building a productive, friendly environment for maximum collaboration and fun.