Package: gcs-uri
gcs-uriSimple API to copy files to and from Google Cloud StorageInstallationpipinstallgcs-uriUsagegcs-uriexposes the following functions as its main public APIcopy_filecopy_dircopy_filesThese functions do exactly what they sound like they do.copy_filewill copy a source file (either a local file or a remote blob in GCS) to destination file (either a local file or remote blob in GCS).copy_dirwill recursively copy the contents of a directory (either a local directory or a remote "directory" in GCS) to a destination directory (either a local directory or a remote "directory" in GCS)copy_fileswill copy a list of source files (either local files or remote blobs in GCS or a mix of local files/remote blobs) to a corresponding set of destination files (either local files or remote blobs in GCS of a mix of local files/remote blobs)If the second argument tocopy_filesis of typestr | Path | Blob(as opposed to a Sequence), then this argument is treated like a directory and each of the source files are "flattened" (i.e. folder delimiters are removed) and copied under the destintation directory.The idea being that you can pass just about any object to these functions and the functions will figures how to do the copying.ExamplesLocal file -> local fileIn this casecopy_filebehaves just likeshutil.copy2orcp, copying the source file to the destination file locally.src='/my/src/file.txt'dst='/my/dst/file.txt'copy_file(src,dst)srcanddstcan also bepathlib.Pathobjects:frompathlibimportPathsrc=Path('/my/src/file.txt')dst=Path('/my/dst/file.txt')copy_file(src,dst)Local dir -> local dirIn this casecopy_dirbehaves just likeshutil.copytree(or somewhat like rsync, butcopy_dirwill "re-copy" all files to the destination whether they exist in the the destination or not).src='/my/src'dst='/my/dst'copy_dir(src,dst)# if there was a file `/my/src/a/b.txt` after `copy_dir`# there would then be a file `/my/dst/a/b.txt`The source and destination can include or omit a trailing slash and the results are the same as above.Local file -> remote file (upload)To copy a file to a google cloud bucket, barely anything has to change, the destination should simply be a google storage URI:src='/my/src/file.txt'dst='gs://my-bkt/dst/file.txt'copy_file(src,dst)If you would likegcs-urito use a particular Google Storage Client, this can be provided as a keyword(-only) argument (the same applies tocopy_dir):fromgoogle.cloudimportstorageclient=storage.Client()src='/my/src/file.txt'dst='gs://my-bkt/dst/file.txt'copy_file(src,dst,client=client)If no client is provided and either of the source or destinations (or both) are determined to represent a remote location thengcs-uriwill try to instantiate a client by callingstorage.Client().Note, we can providedgcs-uriwith "richer" objects (instead of just strings):frompathlibimportPathfromgoogle.cloudimportstorageclient=storage.Client()src=Path('/my/src/file.txt')dst=storage.Blob.from_string('gs://my-bkt/dst/file.txt',client=client)copy_file(src,dst)Local dir -> remote dir (upload)The concepts from the previous sections apply here:src='/my/src'dst='gs://my-bkt/dst'copy_dir(src,dst)# if there was a file `/my/src/a/b.txt` after `copy_dir`# there would then be a blob `gs://my-bkt/dst/a/b.txt`Remote file -> local file (download)src='gs://my-bkt/src/file.txt'dst='/my/dst/file.txt'copy_file(src,dst)Remote dir -> local dir (download)src='gs://my-bkt/src'dst='/my/dst'copy_dir(src,dst)Remote file -> remote file (transfer)src='gs://my-bkt/src/file.txt'dst='gs://my-other-bkt/dst/file.txt'copy_file(src,dst)Remote dir -> remote dir (transfer)src='gs://my-bkt/src'dst='gs://my-other-bkt/dst'copy_dir(src,dst)List of local files -> list of remote filessrcs=['/my/src/file1.txt','/my/src/file2.txt']dsts=['gs://my-bkt/dst/file1.txt','gs://my-bkt/dst/file2.txt']copy_files(srcs,dsts)# copies: /my/src/file1.txt -> gs://my-bkt/dst/file1.txt# copies: /my/src/file2.txt -> gs://my-bkt/dst/file2.txtList of local files -> remote dirsrcs=['/my/src/file1.txt','/my/src/file2.txt']dst='gs://my-bkt/dst'copy_files(srcs,dst)# copies: /my/src/file1.txt -> gs://my-bkt/dst/my-src-file1.txt# copies: /my/src/file2.txt -> gs://my-bkt/dst/my-src-file2.txtAPI# src/gcs_uri.pydefcopy_file(src:str|Path|Blob,dst:str|Path|Blob,*,client:Client|None=None,quiet:bool=False,)->None:"""Copy a single file.If `src` and `dst` are both determined to be local files then `client` is ignored."""defcopy_dir(src:str|Path|Blob,dst:str|Path|Blob,*,client:Client|None=None,quiet:bool=False,)->None:"""Copy a directory (recursively).If `src` and `dst` are both determined to be local directoriesthen `client` is ignored."""defcopy_files(srcs:Sequence[str|Path|Blob],dsts:str|Path|Blob|Sequence[str|Path|Blob],*,client:Client|None=None,quiet:bool=False,)->None:"""Copy a list of files.If `dsts` is a `str | Path | Blob` it is treated as a directoryand each of the files in `srcs` will have its name "flattened" and will becopied under `dsts`.if `dsts` is a `Sequence[str | Path | Blob]` it is zipped with `srcs`, i.e.each file in `srcs` is copied to its corresponding entry in `dsts`."""TestsThis package comes with some basic end-to-end (e2e) tests. They require an active google cloud project with the google storage API enabled.To help with running them there is a utility script in the root of this repo:run_e2e_tests.py.usage: run_e2e_tests.py [-h] [-v] [-c GOOGLE_APPLICATION_CREDENTIALS]
                        [-u TEST_STORAGE_URI]

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  -c GOOGLE_APPLICATION_CREDENTIALS, --google-application-credentials GOOGLE_APPLICATION_CREDENTIALS
                        Google cloud service account to use.
  -u TEST_STORAGE_URI, --test-storage-uri TEST_STORAGE_URI
                        Google storage uri to use when running e2e tests.This script requires you to provided a service account json file as we'll as a URI to a location in google cloud which the tests will use to copy blobs to/from. (IMPORTANT:allblobs at and beneath the location you specifify will be removed - the bucket itself willnotbe removed).So, run the e2e tests with something like:python-mrun_e2e_tests-c"path/to/service-account.json"-u"gs://my-bkt/gcs-uri-tests"ContributingHave or install a recent version ofpoetry(version >= 1.1)Fork the repoSetup a virtual environment (however you prefer)Runpoetry installRunpre-commit installAdd your changes (adding/updating tests is always nice too)Commit your changes + push to your forkOpen a PR