Package: dblue-stores
Dblue StoresDblue Store is an abstraction and a collection of clients to interact with cloud storages.Install$pipinstall-Udblue-storesN.B. this module does not include by default the cloud storage's client requirements
to keep the library lightweight, the user needs to install the appropriate module to use withdblue-stores.Install S3pipinstall-Udblue-stores[s3]Install GCSpipinstall-Udblue-stores[gcs]Install Azure Storagepipinstall-Udblue-stores[azure]Install SFTPpipinstall-Udblue-stores[sftp]StoresThis module includes clients and stores abstraction that can be used to interact with AWS S3, Azure Storage, Google Cloud Storage and SFTP.S3Normal instantiationfromdblue_stores.stores.s3importS3Stores3_store=S3Store(endpoint_url=...,access_key=...,secret_key=...,session_token=...,region=...)Using env varsexportAWS_ENDPOINT_URL=...exportAWS_ACCESS_KEY=...exportAWS_SECRET_KEY=...exportAWS_SECURITY_TOKEN=...
exprotAWS_REGION=...And then you can instantiate the storefromdblue_stores.stores.s3importS3Stores3_store=S3Store()Using a clientfromdblue_stores.stores.s3importS3Stores3_store=S3Store(client=client)Important methodss3_store.list(bucket_name,prefix='',delimiter='',page_size=None,max_items=None,keys=True,prefixes=True)s3_store.list_prefixes(bucket_name,prefix='',delimiter='',page_size=None,max_items=None)s3_store.list_keys(bucket_name,prefix='',delimiter='',page_size=None,max_items=None)s3_store.check_key(key,bucket_name=None)s3_store.get_key(key,bucket_name=None)s3_store.read_key(key,bucket_name=None)s3_store.upload_bytes(bytes_data,key,bucket_name=None,overwrite=False,encrypt=False,acl=None)s3_store.upload_string(string_data,key,bucket_name=None,overwrite=False,encrypt=False,acl=None,encoding='utf-8')s3_store.upload_file(filename,key,bucket_name=None,overwrite=False,encrypt=False,acl=None,use_basename=True)s3_store.download_file(key,local_path,bucket_name=None,use_basename=True)s3_store.upload_dir(dirname,key,bucket_name=None,overwrite=False,encrypt=False,acl=None,use_basename=True)s3_store.download_dir(key,local_path,bucket_name=None,use_basename=True)GCSNormal instantiationfromdblue_stores.stores.gcsimportGCSStoregcs_store=GCSStore(project_id=...,credentials=...,key_path=...,keyfile_dict=...,scopes=...)Using a clientfromdblue_stores.stores.gcsimportGCSStoregcs_store=GCSStore(client=client)Important methodsgcs_store.list(key,bucket_name=None,path=None,delimiter='/',blobs=True,prefixes=True)gcs_store.upload_file(filename,blob,bucket_name=None,use_basename=True)gcs_store.download_file(blob,local_path,bucket_name=None,use_basename=True)gcs_store.upload_dir(dirname,blob,bucket_name=None,use_basename=True)gcs_store.download_dir(blob,local_path,bucket_name=None,use_basename=True)Azure StorageNormal instantiationfromdblue_stores.stores.azureimportAzureStoreaz_store=AzureStore(account_name=...,account_key=...,connection_string=...)Using env varsexportAZURE_ACCOUNT_NAME=...exportAZURE_ACCOUNT_KEY=...exportAZURE_CONNECTION_STRING=...And then you can instantiate the storefromdblue_stores.stores.azureimportAzureStoreaz_store=AzureStore()Using a clientfromdblue_stores.stores.azureimportAzureStoreaz_store=AzureStore(client=client)Important methodsaz_store.list(key,container_name=None,path=None,delimiter='/',blobs=True,prefixes=True)az_store.upload_file(filename,blob,container_name=None,use_basename=True)az_store.download_file(blob,local_path,container_name=None,use_basename=True)az_store.upload_dir(dirname,blob,container_name=None,use_basename=True)az_store.download_dir(blob,local_path,container_name=None,use_basename=True)Running testspytestPublishInstall twinepipinstalltwineBuild distributionpythonsetup.pysdistPublish to pypitwineuploaddist/*CreditsMost of the code are borrowed fromhttps://github.com/polyaxon/polystores