Package: pdcloud
pdcloudpdcloudis a Python package designed to simplify and accelerate the onboarding of data stored in cloud environments. It's built to be cloud-agnostic, allowing seamless access to dataframes stored across various cloud platforms.Simplifying Cloud Data Access withpdcloudpdcloudoffers a unified interface to interact with multiple cloud storage providers, abstracting away the complexities of dealing with different cloud-specific APIs. Key advantages include:Cloud-Agnostic Interface: One interface to access data across Azure, AWS, GCP, and more, removing the need to understand each cloud provider's specifics.Streamlined Data Operations: Whether reading or writing data,pdcloudprovides a consistent, intuitive API, simplifying cloud data operations.Optimized Data Handling: Leveraging PyArrow and Parquet,pdcloudensures efficient, fast, and cost-effective data processing.Throughpdcloud, users gain a straightforward, efficient path to access and manipulate cloud-stored data, irrespective of the underlying cloud platform.Benefits of Using PyArrow and Parquetpdcloudleverages the power of PyArrow and Parquet for data storage and processing, offering several key advantages:Efficient Data Storage: Parquet stores data in a columnar format, which is more space-efficient compared to row-based storage, especially for analytical queries.Optimized for Performance: PyArrow's columnar memory format enables fast data access and efficient in-memory computing, which is crucial for analytics.Cross-platform Support: Parquet is supported across multiple programming languages and platforms, ensuring compatibility and flexibility.Scalability: Ideal for handling large datasets, Parquet efficiently scales to accommodate massive volumes of data.Data Compression: Parquet supports various compression techniques, significantly reducing storage costs and improving I/O performance.Schema Evolution: Parquet supports schema evolution, allowing modification of the schema over time without the need to rewrite the dataset.By using PyArrow and Parquet,pdcloudensures that data is stored and accessed in the most efficient, performant, and cost-effective manner.Design Choices inpdcloudpdcloudis crafted with the vision of simplifying data access across various cloud platforms. Key design choices include:Unified API: A single, intuitive interface for all cloud storage operations, regardless of the cloud provider.Abstraction Layer: Abstracts the complexities of each cloud provider's API, providing a seamless experience.Cloud-Agnostic Approach: Designed to be adaptable to different cloud environments, ensuring flexibility and broad applicability.Optimized Data Processing: Integration with PyArrow and Parquet for efficient data handling, suitable for both small and large-scale datasets.Focus on Performance and Scalability: Ensures efficient data operations, catering to the needs of both individual users and large enterprises.These design choices reflect our commitment to providing a versatile, efficient, and user-friendly tool for cloud-based data management.Key FeaturesCloud Agnostic: Compatible with major cloud providers, enabling access to data regardless of its cloud location.Efficient Data Onboarding: Reduces the steps involved in data transfer and processing, moving away from traditional methods like SFTP/FTP.Direct Data Access: Facilitates direct access to data through simple cloud configurations and connection strings.Standardized Data Format: Utilizes Parquet format for data storage and retrieval, ensuring efficiency and uniformity.MotivationThe goal ofpdcloudis to revolutionize how data providers share and users access data. By eliminating the cumbersome process of data transfer and storage,pdcloudenables users to onboard data swiftly and efficiently. Upon signing necessary data agreements, users can instantly access data provided by vendors through unique cloud configurations, significantly cutting down the time and resources typically spent on data integration.FeaturesCloud agnostic: Works with Azure Blob Storage, with planned support for AWS S3 and Google Cloud Storage.Asynchronous and synchronous read/write operations.Utilizes Apache Arrow for efficient data handling.InstallationpipinstallpdcloudUsageAzure Storage AdapterimportpandasaspdfrompdcloudimportAzureStoragefrompdcloudimportLib# Initialize the Azure Storage Adapterconnection_string=""azure_storage=AzureStorage(connection_string)# Define the container namecontainer_name="library"# Create an instance of the Lib classlib=Lib(container=container_name,storage=azure_storage)# Read and process all data objects from the containerall_data:pd.DataFrame=lib.read_all()print("All Data:",all_data)# Read and process a specific data object from the containerdata_object_name="mydata"specific_data:pd.DataFrame=lib.read(data_object_name)print("Specific Data Object:",specific_data)# Write a DataFrame to the same Containerlib.write("mydata",data=df,overwrite=True)# Write a DataFrame to a different containerlib.write("mydata",container="library",data=df,overwrite=True)ContributingContributions to pdcloud are welcome! Please read our contributing guidelines for details on how to contribute to the project.LicenseThis project is licensed under the MIT License.