Package: gs-wrap
gs-wrapgs-wrapwrapsGoogle Cloud Storage APIfor multi-threaded data manipulation including copying, reading, writing and
hashing.Originally, we used ourgsutilwrap,
a thin wrapper aroundgsutilcommand-line interface, to simplify
the deployment and backup tasks related to Google Cloud Storage.
However,gsutilwrapwas prohibitively slow at copying many objects into
different destinations.Therefore we developedgs-wrapto accelerate these operations while keeping
it equally fast or faster thangsutilwrapat other operations.While thegoogle-cloud-storagelibrary provided by Google offers sophisticated features and good performance,
its use cases and behavior differ fromgsutil.
Since we wanted the simplicity and usage patterns ofgsutil, we createdgs-wrap, which wrapsgoogle-cloud-storagein its core and with its
interface set to behave likegsutil.gs-wrapis not the first Python library wrapping Google Cloud Storage API.cloud-storage-clienttakes a similar approach and aims to manage both Amazon’s S3 and Google Cloud
Storage. Parts of it are also based ongoogle-cloud-storage, however the
library’s behaviour differs fromgsutilwhich made it hard to use as an
in-place replacement forgsutilwrap. Additionally, the library did not
offer all needed operations, for example copying to many destinations, reading,
writing and hashing.The main strength ofgs-wrapis the ability to copy many objects from many
different paths to multiple destinations, while still mimickinggsutilinterface. A direct comparison of performance betweengs-wrapandgsutilwrapcan be found in thesection Benchmarks.UsageYou need to create a Google Cloud Storage bucket to use this client library.
Follow along with theofficial Google Cloud Storage documentationto
learn how to create a bucket.Connect to your Google Cloud Storage bucketFirst a client for interacting with the Google Cloud Storage API needs to be
created. This one uses internally theStorage Clientfromgoogle-cloud-storage.One parameter can be passed to the client:The Google Cloud Storageprojectwhich the client acts on behalf of. It will
be passed when creating the internal client. If not passed, falls back to the
default inferred from the locally authenticatedGoogle Cloud SDKenvironment. Each project needs a separate
client. Operations between two different projects are not supported.importgswrapclient=gswrap.Client()# project is optionalList objects in your bucketWarningWildcards (*, **, ?, [chars], [char range]) are not supported by
Google Cloud Storage API and neither bygs-wrapat the moment
[2019-01-16]. Reasons are that thegsutilwith wildcards can hardly be
equivalently reconstructed and that the toplevel search is extremely
inefficient. More information aboutgsutilwildcards can be found here:https://cloud.google.com/storage/docs/gsutil/addlhelp/WildcardNamesclient.ls(gcs_url="gs://your-bucket/your-dir",recursive=False)# gs://your-bucket/your-dir/your-subdir1/# gs://your-bucket/your-dir/your-subdir2/# gs://your-bucket/your-dir/file1client.ls(gcs_url="gs://your-bucket/your-dir",recursive=True)# gs://your-bucket/your-dir/your-subdir1/file1# gs://your-bucket/your-dir/your-subdir1/file2# gs://your-bucket/your-dir/your-subdir2/file1# gs://your-bucket/your-dir/file1Copy objects within Google Cloud StorageIf both the source and destination URL are cloud URLs from the same provider,gsutilcopies data “in the cloud” (i.e. without downloading to and
uploading from the machine where you rungs-wrap).Noteclient.cp() runs single-threaded by default. When multi-threading is
activated, the maximum number of workers is the number of processors on the
machine, multiplied by 5. This is the multi-threading default of theThreadPoolExecuter from the concurrent.futures library.Copy file within Google Cloud Storage# your-bucket before:# gs://your-bucket/file1client.cp(src="gs://your-bucket/file1",dst="gs://your-bucket/your-dir/",recursive=True)# your-bucket after:# gs://your-bucket/file1# gs://your-bucket/your-dir/file1# your-backup-bucket before:# "empty"client.cp(src="gs://your-bucket/file1",dst="gs://your-backup-bucket/backup-file1",recursive=False)# your-backup-bucket after:# gs://your-backup-bucket/backup-file1Copy directory within Google Cloud Storage# your-bucket before:# "empty"client.cp(src="gs://your-bucket/some-dir/",dst="gs://your-bucket/another-dir/",recursive=False)# google.api_core.exceptions.GoogleAPIError: No URLs matched# your-bucket before:# gs://your-bucket/some-dir/file1# gs://your-bucket/some-dir/dir1/file11# Destination URL without slashclient.cp(src="gs://your-bucket/some-dir/",dst="gs://your-bucket/another-dir",recursive=True)# your-bucket after:# gs://your-bucket/another-dir/file1# gs://your-bucket/another-dir/dir1/file11# Destination URL with slashclient.cp(src="gs://your-bucket/some-dir/",dst="gs://your-bucket/another-dir/",recursive=True)# your-bucket after:# gs://your-bucket/another-dir/some-dir/file1# gs://your-bucket/another-dir/some-dir/dir1/file11# Choose to copy multi-threaded. (default=False)client.cp(src="gs://your-bucket/some-dir/",dst="gs://your-bucket/another-dir",recursive=True,multithreaded=True)# your-bucket after:# gs://your-bucket/another-dir/file1# gs://your-bucket/another-dir/dir1/file11Upload objects to Google Cloud StorageNoterecursivecauses directories, buckets, and bucket subdirectories to be
copied recursively. If you upload from local disk to Google Cloud Storage
and set recursive toFalse,gs-wrapwill raise an exception and inform you that no URL matched.
This mimicks the behaviour ofgsutilwhen no wildcards are used.# Your local directory:# /home/user/storage/file1# /home/user/storage/file2# your-bucket before:# "empty"client.cp(src="/home/user/storage/",dst="gs://your-bucket/local/",recursive=True)# your-bucket after:# gs://your-bucket/local/storage/file1# gs://your-bucket/local/storage/file2Download objects from Google Cloud StorageNoterecursivecauses directories, buckets, and bucket subdirectories to be
copied recursively. If you upload from local disk to Google Cloud Storage
and set recursive toFalse,gs-wrapwill raise an exception and inform you that no URL matched.
This mimicks the behaviour ofgsutilwhen no wildcards are used.importos# Current your-bucket:# gs://your-bucket/file1client.cp(src="gs://your-bucket/file1",dst="/home/user/storage/file1")# Your local directory:# /home/user/storage/file1Copy, download and upload with parametersNoteAll parameters can be used for any kind ofcpoperation.# Parameter: no_clobber example:importos# File content before: "hello"os.stat("/home/user/storage/file1").st_mtime# 1537947563client.cp(src="gs://your-bucket/file1",dst="/home/user/storage/file1",no_clobber=True)# no_clobber option stops from overwriting.# File content after: "hello"os.stat("/home/user/storage/file1").st_mtime# 1537947563client.cp(src="gs://your-bucket/file1",dst="/home/user/storage/file1",no_clobber=False)# File content after: "hello world"os.stat("/home/user/storage/file1").st_mtime# 1540889799# Parameter: recursive and multi-threaded example:# Your local directory:# /home/user/storage/file1# ...# /home/user/storage/file1000# your-bucket before:# "empty"# Execute normal recursive copy in multiple threads.client.cp(src="/home/user/storage/",dst="gs://your-bucket/local/",recursive=True,multithreaded=True)# your-bucket after:# gs://your-bucket/local/storage/file1# ...# gs://your-bucket/local/storage/file1000# Parameter: preserve_posix example:# Your file before:# /home/user/storage/file1# e.g. file_mtime: 1547653413 equivalent to 2019-01-16 16:43:33client.cp(src="/home/user/storage/file1",dst="gs://your-backup-bucket/file1",preserve_posix=False)# your-backup-bucket after:# gs://your-backup-bucket/file1 e.g. "no metadata file_mtime"# Preserve the POSIX attributes. POSIX attributes are the metadata of a file.client.cp(src="/home/user/storage/file1",dst="gs://your-backup-bucket/file1",preserve_posix=True)# your-backup-bucket after:# gs://your-backup-bucket/file1 e.g. file_mtime: 2019-01-16 16:43:33Perform multiple copy operations in one callsources_destinations=[# Copy on Google Cloud Storage('gs://your-bucket/your-dir/file','gs://your-bucket/backup-dir/file'),# Copy from gcs to local('gs://your-bucket/your-dir/file',pathlib.Path('/home/user/storage/backup-file')),# Copy from local to gcs(pathlib.Path('/home/user/storage/new-file'),'gs://your-bucket/your-dir/new-file'),# Copy locally(pathlib.Path('/home/user/storage/file'),pathlib.Path('/home/user/storage/new-file'))]client.cp_many_to_many(srcs_dsts=sources_destinations)Remove files from Google Cloud Storage# your-bucket before:# gs://your-bucket/fileclient.rm(url="gs://your-bucket/file")# your-bucket after:# "empty"# your-bucket before:# gs://your-bucket/file1# gs://your-bucket/your-dir/file2# gs://your-bucket/your-dir/sub-dir/file3client.rm(url="gs://your-bucket/your-dir",recursive=True)# your-bucket after:# gs://your-bucket/file1Read and write files in Google Cloud Storageclient.write_text(url="gs://your-bucket/file",text="Hello, I'm text",encoding='utf-8')client.read_text(url="gs://your-bucket/file",encoding='utf-8')# Hello I'm textclient.write_bytes(url="gs://your-bucket/data",data="I'm important data".encode('utf-8'))data=client.read_bytes(url="gs://your-bucket/data")data.decode('utf-8')# I'm important dataCopy os.stat() of a file or metadata of a blobNotePOSIX attributes include meta information about a file. When copying a file
locally or copying a file within Google Cloud Storage, the POSIX attributes
are always preserved. On the other hand, when downloading or uploading file
to Google Cloud Storage, the POSIX attributes is only preserved whenpreserve_posixis set to True.file=pathlib.Path('/home/user/storage/file')file.touch()print(file.stat())# os.stat_result(st_mode=33204, st_ino=19022665, st_dev=64769, st_nlink=1,# st_uid=1000, st_gid=1000, st_size=0, st_atime=1544015997,# st_mtime=1544015997, st_ctime=1544015997)# Upload does not preserve POSIX attributes.client.cp(src=pathlib.Path('/home/user/storage/file'),dst="gs://your-bucket/file")stats=client.stat(url="gs://your-bucket/file")stats.creation_time# 2018-11-21 13:27:46.255000+00:00stats.update_time# 2018-11-21 13:27:46.255000+00:00stats.content_length# 1024 [bytes]stats.storage_class# REGIONALstats.file_atime# Nonestats.file_mtime# Nonestats.posix_uid# Nonestats.posix_gid# Nonestats.posix_mode# Nonestats.md5# b'1B2M2Y8AsgTpgAmY7PhCfg=='stats.crc32c# b'AAAAAA=='# Upload with preserve_posix also copy POSIX attributes to blob.# POSIX attributes are the metadata of a file.# It also works for downloading.client.cp(src=pathlib.Path('/home/user/storage/file'),dst="gs://your-bucket/file",preserve_posix=True)stats=client.stat(url="gs://your-bucket/file")stats.creation_time# 2018-11-21 13:27:46.255000+00:00stats.update_time# 2018-11-21 13:27:46.255000+00:00stats.content_length# 1024 [bytes]stats.storage_class# REGIONALstats.file_atime# 2018-11-21 13:27:46stats.file_mtime# 2018-11-21 13:27:46stats.posix_uid# 1000stats.posix_gid# 1000stats.posix_mode# 777stats.md5# b'1B2M2Y8AsgTpgAmY7PhCfg=='stats.crc32c# b'AAAAAA=='Check correctness of copied file# Check modification time when copied with preserve_posix.client.same_modtime(path='/home/user/storage/file',url='gs://your-bucket/file')# Check md5 hash to ensure content equality.client.same_md5(path='/home/user/storage/file',url='gs://your-bucket/file')# Retrieve hex digests of MD5 checksums for multiple URLs.urls=['gs://your-bucket/file1','gs://your-bucket/file2']client.md5_hexdigests(urls=urls,multithreaded=False)DocumentationThe documentation is available onreadthedocs.SetupIn order to use this library, you need to go through the following steps:Select or create a Cloud Platform project.Enable billing for your project.Enable the Google Cloud Storage API.Setup Authentication using the Google Cloud SDK.InstallationInstall gs-wrap with pip:pip3installgs-wrapDevelopmentCheck out the repository.In the repository root, create the virtual environment:python3-mvenvvenv3Activate the virtual environment:sourcevenv3/bin/activateInstall the development dependencies:pip3install-e.[dev]We use tox for testing and packaging the distribution. Assuming that the virtual
environment has been activated and the development dependencies have been
installed, run:toxPre-commit ChecksWe provide a set of pre-commit checks that lint and check code for formatting.Namely, we use:yapfto check the formatting.The style of the docstrings is checked withpydocstyle.Static type analysis is performed withmypy.isortto sort your imports for you.Various linter checks are done withpylint.Doctests are executed using the Pythondoctest module.pyicontract-lintlints contracts
in Python code defined withicontract library.twineto check the README for invalid markup
which prevents it from rendering correctly on PyPI.Run the pre-commit checks locally from an activated virtual environment with
development dependencies:./precommit.pyThe pre-commit script can also automatically format the code:./precommit.py--overwriteBenchmarksAssuming that the virtual environment has been activated, the development
dependencies have been installed and thePYTHONPATHhas been set to the
project directory, run the benchmarks with:./benchmark/main.py*NAMEOFYOURGCSBUCKET*Here are some of our benchmark results:Benchmark list 10000 files:TESTEDTIMESPEEDUPgswrap3.22 s-gsutilwrap3.98 s1.24 xBenchmark upload 10000 files:TESTEDTIMESPEEDUPgswrap45.12 s-gsutilwrap34.85 s0.77 xBenchmark upload-many-to-many 500 files:TESTEDTIMESPEEDUPgswrap2.14 s-gsutilwrap65.2 s30.49 xBenchmark download 10000 files:TESTEDTIMESPEEDUPgswrap43.92 s-gsutilwrap43.01 s0.98 xBenchmark download-many-to-many 500 files:TESTEDTIMESPEEDUPgswrap5.85 s-gsutilwrap62.93 s10.76 xBenchmark copy on remote 1000 files:TESTEDTIMESPEEDUPgswrap5.09 s-gsutilwrap4.47 s0.88 xBenchmark copy-many-to-many-on-remote 500 files:TESTEDTIMESPEEDUPgswrap6.55 s-gsutilwrap62.76 s9.57 xBenchmark remove 1000 files:TESTEDTIMESPEEDUPgswrap3.16 s-gsutilwrap3.66 s1.16 xBenchmark read 100 files:TESTEDTIMESPEEDUPgswrap16.56 s-gsutilwrap64.73 s3.91 xBenchmark write 30 files:TESTEDTIMESPEEDUPgswrap2.67 s-gsutilwrap32.55 s12.17 xBenchmark stat 100 files:TESTEDTIMESPEEDUPgswrap6.39 s-gsutilwrap48.15 s7.53 xAll results of our benchmarks can be foundhere.VersioningWe followSemantic Versioning.
The version X.Y.Z indicates:X is the major version (backward-incompatible),Y is the minor version (backward-compatible), andZ is the patch version (backward-compatible bug fix).