Package: cloudstorageio
cloudstorageioStorage agnostic IO interface for humansDeveloped and tested on python 3.6+GitHubPyPiGetting StartedThese instructions will get you a copy of the project up and running on your local machine.S3 configspipinstallawscli--upgrade--userset your aws configssudoaptinstallawscli$awsconfigure
AWSAccessKeyID[None]:AKIAIOSFODNN7EXAMPLE
AWSSecretAccessKey[None]:wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Defaultregionname[None]:us-west-2
Defaultoutputformat[None]:jsonclick herefor more info about configuration and installationGoogle cloud storage configsProvide authentication credentials to your application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS.
Replace [FILE_NAME] with the file of the JSON file that contains your service account keyGOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json"click herefor more info about configuration and installationDropbox workspace configsProvide app access token by setting the environment variable DROPBOX_TOKEN.
Replace [TOKEN] with the generated token of your dropbox appDROPBOX_TOKEN='TOKEN'click herefor more info about configuration and installationInstallationThe default installation includes only AWS connection.pip install cloudstorageioTo add Google Drive install withdriveextra:pipinstall"cloudstorageio[drive]"To add Google Storage install withgoogleextra:pipinstall"cloudstorageio[google]"To add Dropbox install withdropboxextra:pipinstall"cloudstorageio[dropbox]"To install all extras run:pipinstall"cloudstorageio[all]"UsageWrite and read filefromcloudstorageioimportCloudInterfacegoogle_file_path='gs://bucket-name/path-to-file/sample-file.txt's3_file_path='s3://bucket-name/path-to-file/sample-pic.jpg'ci=CloudInterface()# Write text to Google cloud storage filewithci.open(google_file_path,'w')asf:f.write("Lorem Ipsum is simply dummy text")# ORci.save(google_file_path,"Lorem Ipsum is simply dummy text")# Read picture from S3 storagewithci.open(s3_file_path,'rb')asf:s3_output=f.read()# binary content of picture# ORci.fetch(s3_file_path)Remove, list, and check folder/filefromcloudstorageioimportCloudInterfacedropbox_folder_path='dbx://bucket-name/folder's3_file_path='s3://bucket-name/path-to-file/sample-pic.jpg'ci=CloudInterface()ci.isfile(s3_file_path)# returns Trueci.isdir(dropbox_folder_path)# returns Trueci.remove(s3_file_path)# removes fileci.listdir(dropbox_folder_path)# lists folder contentCopy filefromcloudstorageioimportCloudInterfacedropbox_file_path='dbx://bucket-name/path-to-file/sample-pic.jpg'gs_file_path='gs://bucket-name/path-to-file/sample-file.txt'ci=CloudInterface()ci.copy(from_path=dropbox_file_path,to_path=gs_file_path)# copies dropbox file to gsCopy dirfromcloudstorageioimportCloudInterfaces3_dir='s3://bucket-name/sample_folder'gs_dir='gs://bucket-name/sample_folder'ci=CloudInterface()ci.copy_dir(source_dir=s3_dir,dest_dir=gs_dir)# copies s3 folder to gsPowered byCognaize`# Changelog
All major cloudstorageio changes and pypi releases are listed hereChanges[0.0.1] - 2019-01-29Created project skeleton[0.0.2] - 2019-02-14Implemented S3 & Google Cloud Storage Interfaces[0.0.3] - 2019-02-28Implemented Cloud Storage InterfaceImplemented Local interface[0.0.4] - 2019-05-31Add delete, listdir & isfile/isdir methods for all interfaces[0.0.5] - 2019-07-09Implemented DropBox interface[0.0.6] - 2019-07-24Add copy_batch for folder copying[0.0.7] - 2019-08-08Add Google Drive interface[0.0.8] - 2019-08-30Add Async Cloud Interface[0.0.9] - 2019-10-18Structure changes (file renames/moves)Implemented scalable unittests[0.0.10] - 2020-07-16Commented Overwriting loggers,add cognaize logo in Readme[0.1.0] - 2020-08-05Add copy_dir for path to path copyChange copy_batch for list to list copyUsages in README[0.1.1] - 2020-08-14README changes[0.1.2] - 2020-11-10Using multipart upload when copying from local to S3Pypi releasesSee CHANGES references[0.12.1] - 2019-03-12[0.0.3][1.0.5] - 2019-06-18[0.0.4][1.0.8] - 2019-08-01[0.0.6][1.0.9] - 2019-10-24[0.0.9][1.1.0] - 2020-08-05[0.1.0][1.1.1] - 2020-08-14[0.1.1][1.1.2] - 2020-11-10[0.1.2][1.1.3] - 2020-11-10Update listdir for s3[1.2.1] - 2020-11-10Add AWS session token support[1.2.3] - 2020-11-10Remove dropbox package dependency (dropbox will not be supported with this version)[1.2.4] - 2020-11-10Add option include_files for listdir[1.2.7] - 2022-10-05Updated build to setuptools[1.2.8] - 2022-05-04Fix error on listdir for s3 if the folder did not include subfoldersAdd option include_files for listdir for non-recursive case[1.2.9] - 2022-05-05Add github workflows for mergingUpdate tests to use AWS mockerMove moto dependency to dev-requirements[1.2.10] - 2023-06-30Fix listdir implementation to include full paths in S3Fix path concatenation in S3[1.2.11] - 2023-07-03Addmototo test requirements[1.2.12] - 2023-07-06Add exclude options tocopy_dirfunction in S3[1.2.15] - 2023-09-29Fix issue with document deepcopyAdd Dropbox, Google Storage and Drive as extra dependencies