Package: polystores
polystoresPolystores is an abstraction and a collection of clients to interact with cloud storages.Install$pipinstall-UpolystoresN.B. this module does not include by default the cloud storage's client requirements
to keep the library lightweight, the user needs to install the appropriate module to use withpolystores.Install S3pipinstall-Upolystores[s3]Install GCSpipinstall-Upolystores[gcs]Install Azure Storagepipinstall-Upolystores[azure]StoresThis module includes clients and stores abstraction that can be used to interact with AWS S3, Azure Storage, and Google Cloud Storage.S3Normal instantiationfrompolystores.stores.s3_storeimportS3Stores3_store=S3Store(endpoint_url=...,access_key_id=...,secret_access_key=...,session_token=...,region=...)Using env varsexportAWS_ENDPOINT_URL=...exportAWS_ACCESS_KEY_ID=...exportAWS_SECRET_ACCESS_KEY=...exportAWS_SECURITY_TOKEN=...
exprotAWS_REGION=...And then you can instantiate the storefrompolystores.stores.s3_storeimportS3Stores3_store=S3Store()Using a clientfrompolystores.stores.s3_storeimportS3Stores3_store=S3Store(client=client)Important methodss3_store.list(bucket_name,prefix='',delimiter='',page_size=None,max_items=None,keys=True,prefixes=True)s3_store.list_prefixes(bucket_name,prefix='',delimiter='',page_size=None,max_items=None)s3_store.list_keys(bucket_name,prefix='',delimiter='',page_size=None,max_items=None)s3_store.check_key(key,bucket_name=None)s3_store.get_key(key,bucket_name=None)s3_store.read_key(key,bucket_name=None)s3_store.upload_bytes(bytes_data,key,bucket_name=None,overwrite=False,encrypt=False,acl=None)s3_store.upload_string(string_data,key,bucket_name=None,overwrite=False,encrypt=False,acl=None,encoding='utf-8')s3_store.upload_file(filename,key,bucket_name=None,overwrite=False,encrypt=False,acl=None,use_basename=True)s3_store.download_file(key,local_path,bucket_name=None,use_basename=True)s3_store.upload_dir(dirname,key,bucket_name=None,overwrite=False,encrypt=False,acl=None,use_basename=True)s3_store.download_dir(key,local_path,bucket_name=None,use_basename=True)GCSNormal instantiationfrompolystores.stores.gcs_storeimportGCSStoregcs_store=GCSStore(project_id=...,credentials=...,key_path=...,key_path=...,scopes=...)Using a clientfrompolystores.stores.gcs_storeimportGCSStoregcs_store=GCSStore(client=client)Important methodsgcs_store.list(key,bucket_name=None,path=None,delimiter='/',blobs=True,prefixes=True)gcs_store.upload_file(filename,blob,bucket_name=None,use_basename=True)gcs_store.download_file(blob,local_path,bucket_name=None,use_basename=True)gcs_store.upload_dir(dirname,blob,bucket_name=None,use_basename=True)gcs_store.download_dir(blob,local_path,bucket_name=None,use_basename=True)Azure StorageNormal instantiationfrompolystores.stores.azure_storeimportAzureStoreaz_store=AzureStore(account_name=...,account_key=...,connection_string=...)Using env varsexportAZURE_ACCOUNT_NAME=...exportAZURE_ACCOUNT_KEY=...exportAZURE_CONNECTION_STRING=...And then you can instantiate the storefrompolystores.stores.azure_storeimportAzureStoreaz_store=AzureStore()Using a clientfrompolystores.stores.azure_storeimportAzureStoreaz_store=AzureStore(client=client)Important methodsaz_store.list(key,container_name=None,path=None,delimiter='/',blobs=True,prefixes=True)az_store.upload_file(filename,blob,container_name=None,use_basename=True)az_store.download_file(blob,local_path,container_name=None,use_basename=True)az_store.upload_dir(dirname,blob,container_name=None,use_basename=True)az_store.download_dir(blob,local_path,container_name=None,use_basename=True)Running testspytestLicense