Package: cos-utils
Cloud Object Storage CLITable of content:Getting StartedListing Cloud Object Storage bucketsListing the content of a Cloud Object Storage bucketUploading files to a Cloud Object Storage bucketDownloading files from a Cloud Object Storage bucketRemoving files from a Cloud Object Storage bucketGetting startedThe utility requires Python 3.6 or above.InstallationYou can install the utility fromPyPIor from thesource.Install from pypi.org$ pip install cos-utils --upgradeInstall from source code$ git clone https://github.com/CODAIT/cos-utils.git
$ cd cos-utils
$ pip install .ConfigurationSet theAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYenvironment variables based on yourCloud Object Storage HMAC credentials.$ export AWS_ACCESS_KEY_ID=...
$ export AWS_SECRET_ACCESS_KEY=...Listing Cloud Object Storage bucketsYou can run the list utility in a terminal window using the generated console script$ list_buckets --helpor explicitly$ python -m cos_utils.list_buckets --helpThe help lists required and optional parameters.usage: list_buckets [-h] pattern

List buckets in Cloud Object Storage instance.

positional arguments:
  pattern     Bucket name spec (supported wildcards: * and ?)

optional arguments:
  -h, --help  show this help message and exit

Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be
defined to run the utility.List all buckets$ list_buckets *On Linux, Unix and MacOS wildcards need to be escaped to prevent shell expansion:list_files <bucket-name> \*.Apply a filterUse the*(any character) and?(one character) wildcards to define a filter condition.For example, to limit output to buckets starting withdata-:$ list_buckets data-*Listing the content of a Cloud Object Storage bucketYou can run the list utility in a terminal window using the generated console script$ list_files --helpor explicitly$ python -m cos_utils.list_files --helpThe help lists required and optional parameters.usage: list_files [-h] bucket pattern

List the content of a Cloud Object Storage bucket.

positional arguments:
  bucket      Bucket name
  pattern     Object key spec (supported wildcards: * and ?)

optional arguments:
  -h, --help  show this help message and exit

Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be
defined to run the utility.List the content of<bucket-name>$ list_files <bucket-name> *On Linux, Unix and MacOS wildcards need to be escaped to prevent shell expansion:list_files <bucket-name> \*.Apply a filterUse the*(any character) and?(one character) wildcards to define a filter condition.For example, to limit output to files ending in.png:$ list_files <bucket-name> *.pngUploading files to a Cloud Object Storage bucketYou can run the upload utility in a terminal window using the generated console script$ upload_files --helpor explicitly$ python -m cos_utils.upload_files --helpThe help lists required and optional parameters. The examples listed below explain them in detail.usage: upload_files [-h] [-p PREFIX] [-r] [-s] [-w] bucket pattern

Upload files to a Cloud Object Storage bucket.

positional arguments:
  bucket                Bucket name
  pattern               File or directory spec (supported wildcards: * and ?)

optional arguments:
  -h, --help            show this help message and exit
  -p PREFIX, --prefix PREFIX
                        Key name prefix
  -r, --recursive       Include files in subdirectories
  -s, --squash          Exclude subdirectory name from key name
  -w, --wipe            Clear bucket prior to upload

Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be
defined to run the utility.Example scenarioThe</path/to/local/directory>contains the following directories and files:file1.png
file2.png
file3.jpg
file4.txt
dir1/file5.gif
dir1/file6.png
dir1/dir2/file7.png
dir1/dir3/file8.jpg
dir1/dir3/file1.pngIn the examples given below<bucket-name>refers to an existing bucket in Cloud Object Storage.Upload directoriesYou can upload the content of any directory.Upload the content of</path/to/local/directory>to bucket<bucket-name>$ upload_files <bucket-name> </path/to/local/directory>Bucket<bucket-name>contains the following objects:file1.png
file2.png
file3.jpg
file4.txtSame as before but clear the bucket first before uploadingSpecify the optional--wipeparameter to clear the bucket before upload.$ upload_files <bucket-name> </path/to/local/directory> --wipeBucket<bucket-name>contains the following objects:file1.png
file2.png
file3.jpg
file4.txtSame as before but include subdirectoriesSpecify the optional--recursiveparameter include files in subdirectories.$ upload_files <bucket-name> </path/to/local/directory> --wipe --recursiveBucket<bucket-name>contains the following objects:file1.png
file2.png
file3.jpg
file4.txt
dir1/file5.gif
dir1/file6.png
dir1/dir2/file7.png
dir1/dir3/file8.jpg
dir1/dir3/file1.pngSame as before but don't use subdirectory names during object key generationSpecify the optional--squashparameter to ignore subdirectory names during object key generation.$ upload_files <bucket-name> </path/to/local/directory> --wipe --recursive --squashBucket<bucket-name>contains the following objects. Note that</path/to/local/directory>contains two files namedfile1.png. Firstfile1.pngis uploaded and later overwritten with the content ofdir1/dir3/file1.png.file2.png
file3.jpg
file4.txt
file5.gif
file6.png
file7.png
file8.jpg
file1.pngSame as before but include a static key name prefixSpecify the optional--prefix <prefix>parameter to add<prefix>to the object key for every file.$ upload_files <bucket-name> </path/to/local/directory> --wipe --recursive --squash --prefix dataBucket<bucket-name>contains the following objects:data/file2.png
data/file3.jpg
data/file4.txt
data/file5.gif
data/file6.png
data/file7.png
data/file8.jpg
data/file1.pngUpload filesYou can upload a single file by specifying</path/to/local/directory/filename>.$ upload_files <bucket-name> /path/to/local/directory/file1.png --wipeBucket<bucket-name>contains the following object:file1.pngYou can upload multiple files by specifying a pattern</path/to/local/directory/filename-pattern>$ upload_files <bucket-name> /path/to/local/directory/*.png --wipeOn Linux, Unix and MacOS wildcards need to be escaped to prevent shell expansion:/path/to/local/directory/\*.png.Bucket<bucket-name>contains the following objects:file1.png
file2.pngUse the--recursiveparameter to extend the search to subdirectories of/path/to/local/directory/.$ upload_files <bucket-name> /path/to/local/directory/*.png --wipe --recursivefile1.png
file2.png
dir1/file6.png
dir1/dir2/file7.png
dir1/dir3/file1.pngDownloading files from a Cloud Object Storage bucketYou can run the download utility in a terminal window using the generated console script$ download_files --helpor explicitly$ python -m cos_utils.dowload_files --helpThe help lists required and optional parameters. The examples listed below explain them in detail.usage: download_files [-h] [-d TARGET_DIR] bucket pattern

Download objects from a Cloud Object Storage bucket.

positional arguments:
  bucket                Bucket name
  pattern               Object key spec (supported wildcards: * and ?)

optional arguments:
  -h, --help            show this help message and exit
  -d TARGET_DIR, --target_dir TARGET_DIR
                        Local target directory. Defaults to the current
                        directory.

Environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY must be
defined to run the utility.Download complete bucket contentYou can download the complete content of a bucket to the current directory:$ download_files <bucket-name> *On Linux, Unix and MacOS wildcards need to be escaped to prevent shell expansion:download_files <bucket-name> \*.Same as before but specify a target directoryUse the--target_dir </path/to/local/dir>parameter to specify an existing directory where the downloaded files will be stored:$ download_files <bucket-name> * --target_dir /tmp/downloadsUse wildcards to selectively download filesUse the*(any character) and?(one character) wildcards to define a filter condition.Download only png files$ download_files <bucket-name> *.pngDownload files that contain a certain string in their name$ download_files <bucket-name> *fil*Removing files from a Cloud Object Storage bucketYou can run the remove utility in a terminal window using the generated console script$ remove_files --helpor explicitly$ python -m cos_utils.remove_files --helpRemove all files from a bucket$ remove_files <bucket-name>LicenseApache-2.0