Saravanan666/pypi-org huggingface dataset:
Package: s3-streaming
s3-streaming: handling (big) S3 files like regular filesStoring, retrieving and using files in S3 is a regular activity so it should be easy. It should also ...stream the datahave an api that is python file-io likehandle some of the desearization and compression stuff because why notInstallpipinstalls3-streamingStreaming S3 objects like regular filesThe basicsOpening and reading S3 objects is similar to regular python io. The only difference is that you need to provide aboto3.session.Sessioninstance to handle the bucket access.importboto3froms3streamingimports3_openwiths3_open('s3://bucket/key',boto_session=boto3.session.Session())asf:fornext_lineinf:print(next_line)Injecting deserialization and compression handling in streamConsider a file that isgzipcompressed and contains lines ofjson. There's some boilerplate in dealing with that,
but why bother? Just handle that in stream.froms3streamingimports3_open,deserialize,compressionreader_settings=dict(boto_session=boto3.session.Session(),deserializer=deserialize.json_lines,compression=compression.gzip)withs3_open('s3://bucket/key.gzip',**reader_settings)asf:fornext_lineinf:print(next_line.keys())# because the file was decompressed ...print(next_line.values())#   ... and the json is now a loaded dict!Otherdeserializeoptions includecsvcsv_as_dicttsvtsv_as_dictstring
Package: s3sync
s3-syncSync AWS S3 storage:Sync file from s3 -> s3Sync file from s3 -> local storageSync file from local storage -> s3Installationpipinstalls3syncS3 sync s3 to s3Read from parameterss3-sync--sync-s3-buckettrue\# using 5 concurrent thread default is 2--thread5\--source-bucket-namesource-bucket\--source-region-nameap-southeast-1\--source-access-key-idsource-access-key\--source-secret-accesssource-secret\--target-bucket-nametarget-bucket\--target-region-nameap-southeast-3\--target-access-key-idtarget-access-key\--target-secret-accesstarget-secretRead from json fileCreate example json file as bellow example, you can name this file whatever{"source":{"bucket_name":"source-bucket","region_name":"ap-southeast-1","access_key_id":"source-access-key","secret_access_key":"source-secret-key"},"target":{"bucket_name":"target-bucket","region_name":"ap-southeast-3","access_key_id":"target-access-key","secret_access_key":"target-secret-key"}}Then execute commands3-sync --sync-s3-bucket true -json /home/ubuntu/config.json --thread 5S3 sync s3 to local (Give absolute path for this)Read from parameterss3-sync--sync-s3-localtrue\# using 5 concurrent thread default is 2--thread5\--source-bucket-namesource-bucket\--source-region-nameap-southeast-1\--source-access-key-idsource-access-key\--source-secret-accesssource-secret-key\--target-local-path/home/ubuntu/sync-s3/my-folderRead from json fileCreate example json file as bellow example, you can name this file whatever{"source":{"bucket_name":"source-bucket","region_name":"ap-southeast-1","access_key_id":"source-access-key","secret_access_key":"source-secret-key"},"target":{"path":"/home/ubuntu/sync-s3/my-folder"}}Then execute commands3-sync --sync-s3-local true -json /home/ubuntu/config.json --thread 5S3 sync local to s3Read from parameterss3-sync--sync-local-s3true\# using 5 concurrent thread default is 2--thread5\--source-local-path/home/ubuntu/sync-s3/my-folder\--target-bucket-nametarget-bucket\--target-region-nameap-southeast-3\--target-access-key-idtarget-access-key\--target-secret-accesstarget-secret-keyRead from json fileCreate example json file as bellow example, you can name this file whatever{"source":{"path":"/home/ubuntu/sync-s3/my-folder"},"target":{"bucket_name":"target-bucket-name","region_name":"ap-southeast-3","access_key_id":"target-access-key","secret_access_key":"target-secret-key"}}Then execute commands3-sync --sync-local-s3 true -json /home/ubuntu/config.json --thread 5Help & BugsIf you are still confused or found a bug,
pleaseopen the issue. All bug reports are appreciated, some
features have not been tested yet due to lack of free time.Licenses3-syncreleased under MIT. SeeLICENSEfor more details.
Package: s3sync-util
S3SyncUtilityS3SyncUtilityis a Python tool using Boto3 to simplify syncing local directories with Amazon S3, for easy uploads and downloads.InstallationYou can install S3Sync usingpipby providing the Git repository URL:pipinstallgit+https://github.com/GeonsV3Collab/S3SyncUtility.git@development

or

pipinstalls3sync_utilConfigurationInitialize the S3Sync configuration:s3sync config initThis will interactively create a.config.inifile with your S3 bucket and prefix settings.Modify`.config.ini`:You can manually edit the`.config.ini`file to change the configuration options.UsageBefore using the utility, make sure you have the required dependencies installed.To uploadfiles/directoriesto S3:s3sync upload --directory <local_directory> --s3-bucket <bucket_name> --s3-prefix <prefix>To downloadfiles/directoriesfrom S3:s3sync download --s3-bucket <bucket_name> --s3-prefix <prefix> --directory <local_directory>Versioning ExampleMajor version bump:v1.0.0(Breaking backward compatibility)Minor version bump:v1.1.0(Backward-compatible new features)Patch version bump:v1.1.1(Backward-compatible bug fixes)Alpha release:v2.0.0-alpha(Initial development)Beta release:v2.0.0-beta(Feature-complete, testing phase)Dev version:v2.0.0-dev(Development version)Stable release:v2.0.0(Production-ready)LicenseThis project is licensed under theMIT License.MIT LicenseCopyright (c) 2023 Geons Logix and V3 Analytics - CollaborationPermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Package: s3-talenttech-oss
S3 CRUD clientPre-installexportS3_ENDPOINT_URL=[s3amocrm.apicompatibleserver]exportS3_BUCKET=[bucketname]exportS3_ACCESS_KEY_ID=[s3keyid]exportS3_SECRET_ACCESS_KEY=[s3secretkey]Usagepip3installs3-talenttech-ossfroms3.clientimportClientclient=Client()client.create_dir('dir')client.create_file('dir/file.txt','qwerty')content=client.read_file('dir/file.txt')client.create_file('dir/file_another.txt','qwerty')lst=client.bulk_read('dir')print(lst)>>>['qwerty','qwerty']client.delete_dir('dir')client.upload_file(src,dst,print_progress=True)
Package: s3-to-disk
S3ToDiskThis is a simple tool to synchronizes S3 to a local folder. It usesaws s3 synccommand behind the scene.This package works on Python versions:2.7.x and greater3.6.x and greaterUsageSet your aws credentials as explained inaws-cli/getting-startedRuns3-to-disk --helpto get more information about parameters.Runs3-to-disk -D s3 -L log --dry-runto clone all your S3
Package: s3tos3backup
s3tos3backup is a tool for managing backups for Amazon S3 storage with IAM role support.UsageSimples3tos3backupWith all options at runtimes3tos3backup --access_key=FOO --secret_key=BAR -b BAZ -o 14Supported options-c FILE, --config FILE Config file name. Defaults to ~/.s3tos3backup
--access_key AWS_ACCESS_KEY AWS Access Key
--secret_key AWS_SECRET_KEY AWS Secret Key
-b BUCKET_NAME, --bucket BUCKET_NAME Bucket name
-o REMOVE_OLDER_DAYS, --remove-older-days REMOVE_OLDER_DAYS Remove backups older than days
-v, --verbose         Enable verbose output
-d, --debug           Enable debug output
--remove-only         Remove only old backups. Do not do a backup
--backup-only         Do a backup. Do not remove only old backups
--configure           Save configuration and exitConfiguration[main]
access_key = FOO
secret_key = BAR
bucket_name = BAZ
remove_older_days = 14DevelopmentRunning testspython setup.py testRunning PEP8 checkerpep8 --config=pep8.rc src/s3tos3backup
Package: s3-to-sftp
Simple application that listens on a SQS Queue and transfers files from S3
to a SFTP Server.The authentication can be done via username/password or username/private key.
Private key should be stored “as-is” in AWS Secrets Manager.The container must have an environment variableSFTP_TARGET, that follows the format below.
You can set an enviroment variable for each of the below, using the property name as environment variable name.The value in environment variables overrides the values otherwise set.BuildYou can build the docker image with the following commandsdocker build . -t s3-to-sftpAlternatively, using docker-composedocker-compose buildDeploy with ECS Compose-XWe highly recommend that you store the value forSFTP_TARGETin AWS Secrets Manager.Install compose-xpython3 -m venv compose-x
source compose-x/bin/activate
pip install pip -U; pip install ecs-composexDeployYou might need to update the settings in the aws.yaml file to meet your environment settings.
On start, the connection to the SFTP server is not established, so the deployment won’t fail
because of that. However, it requires to have theSFTP_TARGETdetails setup in before it
starts listening on the SQS queue jobs.ecs-compose-x up -d templates -f docker-compose.yaml -f aws.yaml -p s3-to-sftp-testingMonitoringIn AWS CloudWatch, you will see new metrics in a Namespace calledS3ToSFTPwhich is going to have
some metrics, here for statistics.These metrics are published usingEMF.SFTP_TARGET format{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "id": "sftp-secret-format",
    "type": "object",
    "additionalProperties": false,
    "properties": {
        "host": {
            "type": "string",
            "format": "idn-hostname"
        },
        "port": {
            "type": "number",
            "minimum": 1,
            "maximum": 65535
        },
        "username": {
            "type": "string"
        },
        "password": {
            "type": "string"
        },
        "default_path": {
            "type": "string"
        },
        "private_key": {
            "type": "string"
        },
        "private_key_pass": {
            "type": "string"
        }
    },
    "required": [
        "host",
        "port",
        "username"
    ]
}
Package: s3tree
S3Tree 🌲S3Tree allows you to access files stored on Amazon AWS S3 as a simple directory tree. You can traverse the tree and read files as easily as
youos.walk.>>>imports3tree>>>s3tree.config.aws_access_key_id='<AWS-ACCESS-KEY>'>>>s3tree.config.aws_secret_access_key='<AWS-SECRET-KEY>'>>>tree=s3tree.S3Tree(bucket_name='my-awesome-bucket')>>>len(tree)77>>>forobjintree:print(obj.name)...adminassets...index.jsInstallationTo install S3Tree, use pipenv (or pip):$ pipenv install s3treeDocumentationUsing AWS credentialsThes3treemodule exposes aconfigobject that can be used to set the AWS credentials you want to use
globally. You should ideally do this before you make any instances ofS3Tree.>>>imports3tree>>>s3tree.config.aws_access_key_id='<AWS-ACCESS-KEY>'>>>s3tree.config.aws_secret_access_key='<AWS-SECRET-KEY>'Credentials can also be passed while creating anS3Treeinstance:>>>tree=s3tree.S3Tree('dummy-bucket',aws_access_key_id='<AWS-ACCESS-KEY>',aws_secret_access_key='<AWS-SECRET-KEY>')Passing the credentials during instance creation overrides the global config.Fetching a treeTheS3Treeobject represents a tree at any given path. The path can be specified while creating a new tree.
If no path, or/is passed, the root if the bucket is fetched.>>>tree=s3tree.S3Tree(bucket_name='dummy')# tree at the root of the bucket `dummy`>>>tree=s3tree.S3Tree(bucket_name='dummy',path='/admin/css')# tree under the path `/admin/css`>>>tree=s3tree.S3Tree(bucket_name='dummy',path='admin/css')# this works too.Thetreeobject above is an iterable that contains all files and directories in this tree.len(tree)gives you the total size of this tree.
If you want to access files and directories separately:>>>tree.directories# iterable for all the directories in the tree>>>tree.files# iterable for all the files in the treeThe S3Tree object can be easily represented as JSON:>>>tree.as_jsonThe Directory objectEach element intree.directoriesis aDirectoryobject. This has attributes that help you
display the directory in a human-friendly manner, and methods to fetch the tree under itself.>>>mydir=tree.directories[0]>>>mydir.name# the name of this directorycss>>>mydir.path# the full path of this directory/admin/css>>>child_tree=mydir.get_tree()# retrieve and store the tree under `mydir` to `child_tree`>>>json_data=mydir.as_json# JSON representation of this directoryThe File objectEach element intree.filesis aFileobject, which has attributes and methods to display properties
and read the file.>>>myfile=tree.files[0]>>>myfile.name# name of this fileindex.js>>>myfile.size# human-readable size of this file4KB>>>contents=myfile.read()# reads the file and stores its contents in `contents`>>>json_data=myfile.as_json# JSON representation of this file obj
Package: s3-tree
s3-treeuses the tree command to display the objects in an s3 bucket in a tree-like diagram.dependenciestreepython3aws cliinstallation$ pip install s3-treeusage$ s3-tree bucketnameexample$ s3-tree bucketname
bucketname
├── asset-manifest.json
├── favicon.ico
├── index.html
├── manifest.json
├── precache-manifest.e8c8442b93de34204de5f9b23fa0174b.js
├── service-worker.js
└── static
    ├── css
    │   ├── main.43b5e879.chunk.css
    │   └── main.43b5e879.chunk.css.map
    ├── js
    │   ├── 1.f6579156.chunk.js
    │   ├── 1.f6579156.chunk.js.map
    │   ├── main.36bbb0f4.chunk.js
    │   ├── main.36bbb0f4.chunk.js.map
    │   ├── runtime~main.229c360f.js
    │   └── runtime~main.229c360f.js.map
    └── media
        ├── her.37588412.png
        ├── me.e69004b8.png
        └── us.f114bc8d.jpg

4 directories, 17 files
Package: s3-uncompress
s3-uncompressThis library is used to uncompress files that are in s3. It creates a CompressedFile object wich have two methods with extraction capabilities using memory or disk.Object creation exampleIf the file name contains the extension, the compression type will be get from there, if not, the compressed_type parameter needs to be setted (Ex: "zip", "rar", "x-rar", "x-rar-compressed", "vnd.rar").CompressedFile(s3_bucket_name='example-source-bucket', s3_key='example.zip', compressed_type=None)uncompress_using_memoryReads the compressed object directly from an S3 bucket and loads the content in bytes over memory. Then iterates to extract each file to the defined S3 destination.uncompress_using_memory(s3_target_bucket='example-target-bucket', s3_target_key='example/target/key')uncompress_using_diskDownloads the compressed object from an S3 bucket to a local path. Then iterates the content to extract each file to the same local path, and finally upload all files to the defined S3 destination and delete local_path.uncompress_using_disk(local_path='example_local_path', s3_target_bucket='example-target-bucket', s3_target_key='example/target/key')Supported Formatsziprar
Package: s3-unique-bucket
s3-unique-bucketUtility to create a unique bucket for each S3 account, useful for deployment scenarios.Did you ever need a bucket to upload your deployment templates into? Are you always annoyed by having to create them manually, not really making your deployment pipeline truly automatic? Well, worry no more!This script is aneasyandidempotentway to just create an S3 bucket that will have a very high chance to avoid name conflicts.
If run a second time, the script recognizes the bucket already exists and will only emit the bucket's name for reuse.Examplebucket=$(s3-unique-bucket)awss3cpfoo.txts3://$bucket/foo.txtHow it worksThe bucket name is generated from a user-settable prefix, the sha1 hash of the account ID with base32 encoding, and the region the bucket is created in.
Then the script checks if a bucket with this name exists on the provided account. If not, it is created. Then finally the name of the bucket is emitted viastdout.DocumentationSet the AWS credentials and region with environment variables.AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_DEFAULT_REGIONThe rest is configured with command line arguments.-p/--prefix: User-settable prefix (OPTIONAL)Default:account-bucketThe bucket name starts with this settable prefix. It can also be empty. Note that a dash ('-') is automatically added between the prefix and the hash if prefix is not empty.-p/--prefix: AWS Account Region (CONDITIONAL)Condition: Must be provided if environment variableAWS_DEFAULT_REGIONisn't set.Which region the bucket shall be created in.
Package: s3upload
A python module for uploading data to s3 using the multipart API.InstallationInstall viapip:$pipinstalls3uploadCLIUse the s3upload CLI to upload from stdin or from data provided on the command line.Usage:s3upload[options]Options:-h,--helpshowthishelpmessageandexit-bBUCKET,--bucket=BUCKETthes3buckettouploadto-kKEY,--key=KEYthenameofthekeytocreateinthebucket-KAWS_KEY,--aws_key=AWS_KEYawsaccesskey-sAWS_SECRET,--aws_secret=AWS_SECRETawssecretkey-dDATA,--data=DATAthedatatouploadtos3--ifleftblankwillbereadfromSTDIN-tTHREADS,--threads=THREADSnumberofthreadstousewhileuploadinginparallelModuleYou can also interface with the module from python:>>>froms3uploadimportuploaddefupload(bucket,aws_access_key,aws_secret_key,iterable,key,progress_cb=None,threads=5,replace=False,secure=True):''' Upload data to s3 using the s3 multipart upload API.

    Args:
        bucket: name of s3 bucket
        aws_access_key: aws access key
        aws_secret_key: aws secret key
        iterable: The data to upload. Each 'part' in the list
        will be uploaded in parallel. Each part must be at
        least 5242880 bytes (5mb).
        key: the name of the key to create in the s3 bucket
        progress_cb: will be called with (part_no, uploaded, total)
        each time a progress update is available.
        threads: the number of threads to use while uploading.
        replace: will replace the key in s3 if set to true. (Default is false)
        secure: use ssl when talking to s3. (Default is true)
'''LicenceMIT
Package: s3-upload-download
s3_upload_download with dirictory positionUtilities for s3 download and upload file.Function for s3 download to the local path including s3 sub folder pathfrom  s3_upload_download import s3_transfers_utlitss3_transfers_utlits.download_s3_file(bucket_name, s3_folder,file_extension, destination_path)Parameters:bucket_name: The name of the s3 bucket 
    s3_folder: The folder path in the s3 bucket including subfolders 
    file_extension: extension of file such as .parquet or .csv 
    destination_path: a relative or absolute directory path in the local file systemExample Usage
.. code-block:: pythonfrom  s3_upload_download import s3_transfers_file

    bucket_name='test_bucket'
    s3_folder='test_folder'
    file_extension='.parquet'
    destination_path=f'home/momataj/test_folder/'
    s3_transfers_file.download_s3_file(bucket_name, s3_folder, file_extension, destination_path)Function for s3 upload from the local path including s3 sub folder path and file extensionfrom  s3_upload_download import s3_transfers_utlitss3_transfers_utlits.upload_s3_file(bucket_name,source_path,base_dir,file_extension)Parameters:bucket_name: the name of the s3 bucket 
    source_path: local dirctory where file is located such as /home/{username}/{foldername} 
    base_dir: a relative or absolute directory path in the local file system such as /home/{username}/ 
    file_extension: extension of file such as .parquet or .csvExample Usage
.. code-block:: python
from  s3_upload_download import s3_transfers_filebucket_name='test_bucket'
    s3_folder='test_folder'
    file_extension='.parquet'
    base_dir='home/momataj/'
    sourceDir=f'home/momataj/test_folder/'
    s3_transfers_file.upload_s3_file(bucket_name,sourceDir,base_dir,file_extension)
Package: s3uploader
A simple application to upload files to an Amazon S3 bucket. Behind the scenes this script uses the awesome Boto3
library to access the Amazon Web Services.QuickstartThe application uses the AWS credentials file located in ~/.aws/credentialsusage: s3uploader.py [-h] [–verbose] [–store-paths] bucket file [file …]Upload files to an Amazon S3 bucketpositional arguments:bucket         Name of the S3 bucket to use
file           File to upload to the S3 bucketoptional arguments:-h,--helpshow this help message and exit--verboseGive verbose output while uploading files--store-pathsStore the file path in the S3 object nameOptions--store-pathsIf this flag is set the S3 object will stored with the path you specify, otherwise the filenamewill be used as the
object name. Using this approach it is possible to simulate a directory structure on S3.--verboseGive verbose output while uploading files to S3.
Package: s3-uploader
Command line tool for uploading resources to AWS S3.This is part of Onfido’s team blobs store project.The purpose of this tool is to upload and manage the versions of your project’s
dependencies that later could be resolved by the complimentary tool
thedependencies-resolver.Installing$pipinstalls3-uploaderHow to useExample for using the tool to upload a resources3-uploader-bmy-s3-bucket-fmy_big_blob-lpath/to/blob/in/other/projectOr you can upload to a specific location in S3s3-uploader-bmy-s3-bucket-fmy_big_blob-ppath/to/file/in/s3-lpath/to/blob/in/other/projectGet the projectClone the git repository$gitclonehttps://github.com/onfido/s3-uploaderInstall a virtualenv$sudoapt-getinstallpython-virtualenvCreate a new virtualenv$virtualenvs3_uploader_veInstall project’s requirements$s3_uploader_ve/bin/pipinstall-rrequirements.txtReporting IssuesIf you have suggestions, bugs or other issues specific to this library, open
an issue, or just be awesome and make a pull request out of it.
Package: s3-upload-split
Python S3 Upload SplitS3 Upload Split is used to stream the content of an iterator to multiple S3 objects based on a provided regular
expression. The iterator must be a list of dictionary, typically the resulset of a SQL query. Files will be calleddata-{pattern}.jsonwhere{pattern}is the match found using your regex.Installpip install s3-upload-splitUsageImportimportrefromsqlalchemyimportcreate_enginefroms3_upload_splitimportSplitUploadS3bucket='YOUR_BUCKET_NAME'# ex: my-bucketprefix='OUTPUT_PATH'# ex: db1/output/dev/regex=re.compile(r'YOUR_REGEX')# ex: \\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\D+(\\d{4}-\\d{2})-\\d{2}engine=create_engine('sqlite:///bookstore.db')# https://github.com/pranaymethuku/bookstore-database/blob/master/database/bookstore.dbwithengine.connect()ascon:iterator=con.execute('SELECT * FROM book')SplitUploadS3(bucket,prefix,regex,iterator).handle_content()LimitationsIt creates one thread per matched pattern using your regex, so take it into account when you use that module. This is
typically useful if your regex matches months in the input iterator.
Package: s3-uri-to-url
S3 URI to URLCommand line tool that converts S3 URIs to a valid aws console URL.Assumes you are already logged into the appropriate account - this tool does not handle credentials or sessions at all.Usage$ s3-uri-to-url s3://sample_bucket/any/long/prefix-string/
https://s3.console.aws.amazon.com/s3/buckets/sample_bucket?prefix=any/long/prefix-string/Installationpip install s3_uri_to_url