Saravanan666/pypi-org huggingface dataset:
Package: s3query
# S3QueryS3Query opens file or files in s3 file system as text file, will perform unzip
if necessary. It also handles hadoop liked part-* files and treat all files
within a directory as one single file.# Example>>> from s3query import s3open
>>> import sys
>>>
>>> with s3open("s3://s3query/data/testgz/") as fgz:
...     for line in fgz:
...         sys.stdout.write(line)
...
one
two
three
four
five
six
seven
eight
nine
>>> with s3open("s3://s3query/data/test/") as fplain:
...     for line in fplain:
...         sys.stdout.write(line)
...
one
two
three
four
five
six
seven
eight
nine
Package: s3-reader
s3-readerPython library to read S3 file as local file.RequirementPython 3.9, 3.10, 3.11Installation$pipinstalls3-readerUsagefrom s3_reader import File


def check_s3_file(path):
    file = File(path)

    with open(file.path) as f:
        ...

check_s3_file('s3://<bucket>/path/to/file')In this example, the S3 file is downloaded whenfile = File(path)is
executed, and it is stored as a temporary file.file.pathrefers to the path of the locally stored temporary file.At the end of thecheck_s3_filefunction, the file object is deleted, and
consequently, the temporary file is also deleted.If path refers to a local file instead of an S3 file, File simply copies the
path, and you can use the File object in the same manner.
Package: s3repo
This is a tool to manage simple Debian repositories in Amazon S3.Example:$ cat ~/.boto
[Credentials]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY

$ s3repo --bucket=mybucket --codename=precise --prefix=private/mystuff init

$ s3repo --bucket=mybucket --codename=precise --prefix=private/mystuff add ~/myapp.debBy default add removes old versions unless you specify –keep-oldIn /etc/apt/sources.list or whatever:deb s3://XXXXXXXXXXXXXXXXXXXX:YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY@s3.amazonaws.com/mybucket/private/mystuff precise mainThe s3:// transport for APT is from apt-s3 (https://github.com/kyleshank/apt-s3)
Package: s3resumable
S3ResumableThis modules provides a helper class to download files from S3 using boto3.
To be able to recover incomplete downloads, it downloads files in parts of
the configured size.InstallationUsingpip:pipinstalls3resumableUsingdocker-compose:docker-composebuildUsageThe basic usage of s3resumable module can be summarized as declare aboto3client and pass it onS3Resumableclass:importboto3froms3resumableimportS3Resumables3client=boto3.client('s3')s3resumable=S3Resumable(s3client)s3resumable.download_file('my_bucket','my_key','my_download_dir')This will download the file in parts (15mb by default) and once downloaded
all the parts will join them in one file.A CLI can also be used. Check the help:s3resumable--helpQAIn order to check QA, you can use docker-compose:docker-composebuild
docker-composerunpy27qa
docker-composerunpy37qa
Package: s3same
s3same(“sesame”, like“open
sesame”) creates
unique-per-repo AWS credentials for Travis CI artifact uploading to S3
and encrypts those credentials with the repo’s public key.InstallationRunning the commandsudopipinstalls3samewill install the latest stable release ofs3sameas a command in/usr/local/bin.Usage$ s3same --help
Usage: s3same [OPTIONS] REPO

Options:
  --pro               Use Travis CI Pro
  --github TEXT       GitHub token
  --owner TEXT        GitHub owner
  --s3-bucket TEXT    S3 bucket for artifacts
  --aws-region TEXT   AWS region
  --aws-key TEXT      AWS key
  --aws-secret TEXT   AWS secret
  --aws-profile TEXT  AWS profile
  --nuke              Nuke the entire s3same setup on IAM
  --help              Show this message and exit.Let’s assume you’ve got your AWS credentials in`~/.aws/credentials<http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files>`__.
Go to your GitHub settings andcreate a personal access
tokenfors3same(the
token must have therepopermission). Create~/.s3samewith:GITHUB_TOKEN=put your token here
S3_BUCKET=the bucket to which artifacts will be uploaded
AWS_PROFILE=some profile
AWS_REGION=your AWS regionIf your AWS credentials are in the default profile, you can omit theAWS_PROFILEline.With all the credentials in place, running$ s3same some-repo-name --owner some-user --prowill create an AWS IAM user unique to the repo
(s3same_travis__some-user__some-repo-name), add that user to thes3same_travisAWS IAM group (creating the group if it doesn’t exist)
to give it the necessary permissions (defined by thes3same_travispolicy, which will be created if it doesn’t exist), generate an AWS key
and secret for that user, use the public key for the given repo on
Travis CI to encrypt the key and secret, and print out the YAML snippet
to use for artifact uploading credentials.ConfigurationSeveral configuration parameters can be specified by the environment, in
the~/.s3samefile, or on the command line. The name/syntax for
environment variables and within the~/.s3samefile are the same.
Anything in the~/.s3samefile overrides the corresponding
environment variable and anything passed as a command-line parameter
overrides the environment variables and the~/.s3samefile. Setting
an AWS key/secret pair overrides specifying an AWS configuration
profile.ParameterVariable NameCommand-LineGitHub TokenGITHUB_TOKEN--githubGitHub OwnerGITHUB_OWNER--ownerS3 BucketS3_BUCKET--s3-bucketAWS RegionAWS_REGION--aws-regionAWS KeyAWS_ACCESS_KEY_ID--aws-keyAWS SecretAWS_SECRET_ACCESS_KEY--aws-secretAWS ProfileAWS_PROFILE--aws-profile
Package: s3seek
s3seekFile-like classes for interacting with AWS S3 buckets. In particular, for
seek and partial download when reading.Based onhttps://alexwlchan.net/2019/02/working-with-large-s3-objects/developmentpipinstall-e.[dev]# Install using pip including development extraspre-commitinstall# Enable pre-commit hookspre-commitrun--all-files# Run pre-commit hooks without committing# Note pre-commit is configured to use:# - seed-isort-config to better categorise third party imports# - isort to sort imports# - black to format codepip-compile# Freeze dependenciespytest# Run testscoveragerun--source=s3seek-mpytest&&coveragereport-m# Run tests, print coveragemypy.# Type checkingpipdeptree# Print dependenciesGlobal git ignores perhttps://help.github.com/en/github/using-git/ignoring-files#configuring-ignored-files-for-all-repositories-on-your-computerFor release to PyPI seehttps://packaging.python.org/tutorials/packaging-projects/
Package: s3selectparser
s3selectparserExperimental S3 Select parserInstallationpipinstalls3selectparserUsageThe underlying pyparsing parser is exposed ass3_select_parser.froms3selectparserimports3_select_parserparsed=s3_select_parser.parseString('''SELECTa as my_alias_a,b my_alias_b,c,"AS","AS" as "FrOM",my_func(a, another_func(47, c)),'hello ''billy'' 'FROMS3Object[*]''')print(parsed.asDict())
Package: s3shutil
s3shutil is the easiest to use and fastest way of moving around directories and files in s3.NoteDecember 1st, 2023. Just released sync operationSync operation allows you to incrementally copy to destination files that
were added to source since the last copy
Supports all directions: s3 to s3, s3 to local drive, local drive to s3.InstallationWe recommend installing from the official PyPI repository.$pipinstalls3shutilDesign PrinciplesA simple and intuitive string based API.Symmetric API: download and uploads work equallyExposes powerful and performant one-liners.Emulate the well knownshutilstandard module API.Use performance boosts behind the scenes (multithreading, batching, server to server operations)No dependencies except boto3Using s3shutils3shutil usesboto3internally and we assume you have your credentials set up properly.Using s3shutil is super easy:Import is mandatory, no suprises here:imports3shutilThen you can do powerful things with simple one liners::# download a tree from s3s3shutil.copytree('s3://bucket/my/path','/home/myuser/files/')# upload a tree to s3s3shutil.copytree('/home/users/pics/','s3://bucket/path/archive/')# copy between two s3 locations# same or different buckets3shutil.copytree('s3://bucket2/files/someth/','s3://bucket1/backup/old/')# delete (recursively) entire prefixs3shutil.rmtree('s3://bucket/my-files/documents/')Just released! (December 2023), tree_sync operation:Only copies files that are missing in the destination.
Also deletes extra files.# sync downloads3shutil.tree_sync('s3://bucket/files/docs/','/home/myuser/docs')# sync uploads3shutil.tree_sync('/home/myuser/files/','s3://bucket/files/docs-v2/')# sync two bucket locationss3shutil.tree_sync('s3://bucket/files/docs/','s3://bucket2/a/b/c')Conclusionss3shutil will notice alone if the location is s3 (starts with s3://) or not
All operations have a similar string based API of powerfull one linersTest Matrixs3shutil is thoroughly unit tested in all the combinations of:Python Versions:3.123.113.103.93.83.7And boto3 Versions:1.331.301.281.271.261.251.241.23ContactJust use it! You can send an email as wellandyworms@gmail.com.
All emails are (eventually) answered.
Also read the code, fork, open a PR, start a discussion.
Package: s3sign
A library to generate AWS S3 Signed RequestsUsagePUT a filefrom s3sign import S3PUTSigner

file_name = 'tos3.png'
mime_type = 'image/png'
valid = 60  # seconds
# S3_NAME, AWS_ACCESS_KEY, AWS_SECRET_KEY can be set as environment
# variables or passed as keyword arguments
signer = S3PUTSigner()

# Get the signature
put_signature = signer.get_signed_url(file_name, valid, mime_type)

# Use the signature to upload a file
url = put_signature['signed_url']
headers = put_signature['headers']
content = open(file_name, 'rb').read()

# PUT the object
requests.put(url, data=content, headers=headers)GET a filesigner = S3GETSigner()

object_name = <the object name>
signed_url = signer.get_signed_url(object_name, valid)
url = signed_url['signed_url']
requests.get(url)Run testexport S3_BUCKET=<your-bucket>
export AWS_ACCESS_KEY=<your-access-key>
export AWS_SECRET_KEY=<your-secret-key>
python -m unittest discover
Package: s3simplebackup
CLI for backing up to S3UsageInstall AWS CLI and add your credentialsRun “pip install -e .” in the programs parent directoryRun script from a directory to backup OR point to directory.First variable is the target object and second variable is the destination bucket.If the json config default bucket is set then you can skip the second variable to use the default.To use a wildcard you must place it in quotations ‘*’.To use recursive wildcards you must use ‘**’ see glob module.ExamplesExample to upload current working directory:s3simplebackup . testbucketExample to upload a home Directory:s3simplebackup ~ testbucketExample to upload a folder:s3simplebackup folder testbucketExample to upload all python scripts in Directory with default bucket:s3simplebackup '*.py'Example to upload all python scripts recursively in directory with default bucket:s3simplebackup '**.py'
Package: s3-simple-compress
S3 Simple Compress ZIPDocumentation for AWS S3 In-Memory File Compression (zip) ProgramOverviewThe AWS S3 file compression (zip) program is a tool that allows users to compress one or more files stored in S3 into a single zip file all in memory without the need to download to your Hard Disk. This can be useful for reducing file sizes and saving storage costs.RequirementsBefore using the AWS S3 file compression (zip) program, you'll need to have the following:- An AWS account
- Access to the AWS S3 service
- Basic knowledge of command line and AWSInstallationHow to install the projectOverviewThe AWS S3 file compression (zip) program is a tool that allows users to compress one or more files stored in S3 into a single zip file all in memory without the need to download to your Hard Disk. This can be useful for reducing file sizes and saving storage costs.
RequirementsBefore using the AWS S3 file compression (zip) program, you'll need to have the following:BashAn AWS account
Access to the AWS S3 service
Basic knowledge of command line and AWSInstallation
How to install the projectFor installation of the project's CLI, we recommend using poetry to install:poetry add s3-simple-compressAlthough this is only a recommendation! You can also install the project with your preferred package manager, such as pip:pip install s3-simple-compressHow to use the programTo use the AWS S3 file compression (zip) program, follow the steps below:- First, we need to import our package
- Then, instantiate the class
- We may or may not need to call the `credentials` method, depending on whether `~/.aws/credentials` already exists or not
- Finally, we just need to call the `zipping_in_s3` methodThe program will compress the specified files and save the compressed zip file in the specified S3 bucket.Code Examplefroms3_compress.zipping_s3importZippingS3zips3=ZippingS3()zips3.credentials(ACCESS_KEY='test',SECRET_KEY='test',SESSION_TOKEN='us-east-1')zips3.zipping_in_s3('test','','zip_name')Here's a Python code example that implements the AWS S3 file compression (zip) program's functionality 100% in memory.
Package: s3sourcedependencies
UsageExtends setuptools by adding a new setup keywords3dependencieswhich will
download and install the specified dependencies from S3.An example:setup(
    name='demo',
    version='0.5.1',
    author='Ion Scerbatiuc',
    packages=find_packages(),
    setup_requires=['s3sourcedependencies'],
    s3dependencies=[
        's3://my.sdist.bucket/private.dependency-0.x.tar.gz',
        's3://my.other.sdist.bucket/other.private.dependency-1.x.tar.gz',
    ],
)In order for this to work you will need to have theAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYenvironment variables available to the process
installing the package.Can be combined withs3sourceuploaderto package, distribute and use private packages as dependencies for other
private packages.Current limitationsonly .tar.gz source distributions are supportedonly buckets available to a single AWS account can be usedthe private packages are downloaded each time regardless if they are already installed
Package: s3splitmerge
Welcome tos3splitmergeDocumentationFeaturesSplit:split big data file on (>=500MB) in common data format CSV, TSV, JSON intoInstallpip installawswrangler==2.10.0--no-depss3splitmergeis released on PyPI, so all you need is:$pipinstalls3splitmergeTo upgrade to latest version:$pipinstall--upgrades3splitmergeMerge Multiple AWS S3 Json File into One Big1. Input DataFiles:s3://my-bucket/input/date=2000-01-01/a.json # 6MB
s3://my-bucket/input/date=2000-01-01/b.json # 600MB
s3://my-bucket/input/date=2000-01-01/c.json # 120MB
s3://my-bucket/input/date=2000-01-02/...
...Content:{"id": 1, "value": "a", ...}
{"id": 2, "value": "b", ...}
{"id": 3, "value": "c", ...}Normalize file size to approximately 6MB. If smaller than 6MB, keep it as it is:s3://my-bucket/input-normalized/date=2000-01-01/a-1.json # 6MB

s3://my-bucket/input-normalized/date=2000-01-01/b-1.json # 6MB
s3://my-bucket/input-normalized/date=2000-01-01/b-2.json # 6MB
...
s3://my-bucket/input-normalized/date=2000-01-01/b-100.json # 6MB

s3://my-bucket/input-normalized/date=2000-01-01/c-1.json # 6MB
s3://my-bucket/input-normalized/date=2000-01-01/c-2.json # 6MB
...
s3://my-bucket/input-normalized/date=2000-01-01/c-20.json # 6MBPerformance per file ETL using AWS Lambda:s3://my-bucket/output-normalized/date=2000-01-01/a-1.parquet # 6MB

s3://my-bucket/output-normalized/date=2000-01-01/b-1.parquet # 6MB
s3://my-bucket/output-normalized/date=2000-01-01/b-2.parquet # 6MB
...
s3://my-bucket/output-normalized/date=2000-01-01/b-100.parquet # 6MB

s3://my-bucket/output-normalized/date=2000-01-01/c-1.parquet # 6MB
s3://my-bucket/output-normalized/date=2000-01-01/c-2.parquet # 6MB
...
s3://my-bucket/output-normalized/date=2000-01-01/c-20.parquet # 6MBMerge file into Bigger one for better Athena Query performance:s3://my-bucket/output-normalized/date=2000-01-01/part-1.parquet # 500MB
s3://my-bucket/output-normalized/date=2000-01-01/part-2.parquet # 500MB
...
Package: s3sqlite-db
Welcome to s3sqlite-dbThis is an extension toencode/databasesthat allows using sqlite database with AWS S3.
Main purpose for this is use with AWS Lambda, to download sqlite db to Lambda on db connect and upload back on disconnect.Installation$pipinstalls3sqlite-dbUsageYou can use S3Database as async context manager:S3_BUCKET='my-bucket'S3_KEY='database.sqlite'DATABASE_URL=f's3sqlite://{S3_BUCKET}/{S3_KEY}'asyncwithS3Database(DATABASE_URL)asdb:query=table.select()db.fetch_all(query)or with async framework like FastAPI:fromfastapiimportFastAPIapp=FastAPI()database=S3Database(DATABASE_URL)@app.on_event("startup")asyncdefstartup():awaitdatabase.connect()@app.on_event("shutdown")asyncdefshutdown():awaitdatabase.disconnect()ConfigurationBy default if remote database was modified, exception is raised, butignote_conflicts=Trueargument can be specified, to force overwrite.asyncwithS3Database(DATABASE_URL,ignote_conflicts=True)asdb:...Also you can specify a path to download local copy of db.
This can be useful when working locally (not on AWS Lambda), or when whorking with several sqlite databaases at the same time.asyncwithS3Database(DATABASE_URL,local_path='/path/to/db.sqlite')asdb:...
Package: s3stash
Very simple module that uses boto3 to stash a file in S3.Example:from orgtup import AwsCredentials
from s3stash import Stash

bucket = 'mycoolbucket'
credentials = AwsCredentials('myaccesskey', 'mysecretkey')
stash = Stash(credentials, bucket)
stash.stash_string('my cool data that I want saved to S3')LicenseApache 2.0
Package: s3stat
This python module uses the really nicegoaccessutility
to provide you with an amazing Amazon log file analyser tool that is relatively easy to install, and is extremely
easy to extend.
GOACCESS version needed: 0.8.5 (it doesn’t work with 0.9+)Installationpip install s3statThis installss3stat.pyin your PYTHONPATH in case you would like to run it from the command line.QuickstartInstall goaccessYou should installgoaccessNoteCloudfront log file processing requires goaccess 0.7.1+Generating an AWS userFirst you should create a user that has approriate rights to read your log files, and you should have its AWS access keys ready.Log in to theaws consoleCreate a new user and select the option to generate an access key for the userSave the access key and secure keys, as these will be needed soonOpen thePermissionstab for the user, and attach a new user policy. Select custom policy, and copy the following:{
  "Statement": [
    {
      "Sid": "Stmt1334764540928",
      "Action": [
        "s3:GetBucketAcl",
        "s3:GetBucketLogging",
        "s3:GetObject",
        "s3:ListAllMyBuckets",
        "s3:ListBucket",
        "s3:PutBucketAcl",
        "s3:PutBucketLogging",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::*"
      ]
    },
    {
      "Sid": "Stmt1334764631669",
      "Action": [
        "cloudfront:GetDistribution",
        "cloudfront:GetDistributionConfig",
        "cloudfront:GetStreamingDistribution",
        "cloudfront:GetStreamingDistributionConfig",
        "cloudfront:ListDistributions",
        "cloudfront:ListStreamingDistributions",
        "cloudfront:UpdateDistribution",
        "cloudfront:UpdateStreamingDistribution"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}Set up logging in your bucketsFirst you should ask Amazon to generate logs for your buckets and cloudfront distributions.Run this scripts3stat.py <aws key> <aws secret> <bucket> <log_path>This will download all the log files for today, and start a goaccess instance in your console.For further options you might run:s3stat.py -hExtendingActually s3stat was designed to be easy to add to your pythonic workflow, as a result it defines
a single class that you can subclass to process the results in json format.:import s3stat

class MyS3Stat(s3stat.S3Stat):

    def process_results(self, json):
        print json

    def process_error(self, exception, data=None):
        print data
        raise exception

mytask = MyS3Stat(bucket, log_path, for_date, (aws_key, aws_secret))
mytask.run()Where theaws_*parameters are optional, if missing then they are taken from the environment variables as provided by boto.
The process_error method currently is called only when the JSON decoding fails, thusdatais the non-decodeable string, while
exception is the ValueError raised by Python.ToDoprovide a command that adds logging to specified buckets and cloudfront distributions
Package: s3streamcat
S3 Stream Cat powers you to instantly check the content of S3 file.
Especially it supports streaming and printing data from compressed files.Compression Format SupportedGzipXZBZIPYou might find it most useful when you don’t actually want to download the file
however want to check the sample data, or to grep for few records matching
certain search.
Typical usage often looks like this:s3streamcat s3://bucketname/dir/file_path
s3streamcat s3://bucketname/dir/file_path | more
s3streamcat s3://bucketname/dir/file_path | grep somethingTo try different encoding in case utf-8 is unable to decode your stream:s3streamcat -e latin-1 s3://bucketname/dir/file_path.gzTo tune number of bytes to be fetched at a time:s3streamcat -b 1024 s3://bucketname/dir/file_path.gzConfigurationsIf you have aws client installed on your systems3streamcatwill workout of the boxIf not you will have to have file at locationHOME/.aws/credentialswith aws cred:[default]
aws_access_key_id=<put your aws access key>
aws_secret_access_key=<put your aws secret key>DependanciesIts written for Python3 which is the present and future of the languagelibssl-devlibffi-devpython3-devUbuntu users can install dependancy packages with:sudoapt-getinstall-ylibssl-devlibffi-devsudoapt-getinstallpython3-dev
Package: s3streamer
OverviewA frontend module to upload files to AWS S3 storage. The module supports large files as it chunks them into smaller sizes and recombines them into the original file in the specified S3 bucket. It employs multiprocessing, and there is the option of specifying the size of each chunk as well as how many chunks to send in a single run. The defaults are listed inOptional Argumentsbelow.PrerequisitesAn AWS S3 bucket to receive uploads.An AWS Lambda function to perform backend tasks.The AWS CloudFormation template to create these resources are available in the project’s GitHub repository.Required (Positional) ArgumentsPosition 1: Filename (local full / relative path to the file)Optional (Keyword) Argumentspath: Destination path in the S3 bucket (default: /)parts: Number of multiprocessing parts to send simultaneously (default: 5)partsize: Size of each part in MB (default: 100)tmp: Location of local temporary directory to store temporary files created by the module (default: ‘/tmp’)purge: Whether to purge the specified file instead of uploading it (default: False)requrl: The endpoint URL for backend Lambda function (default: ‘https://<api_endpoint_url>’)reqapikey: The API key for backend Lambda function (default: ‘<api_key>’)UsageInstallation:pip3installs3streamer# orpython3-mpipinstalls3streamerIn Python3:# To upload a new file.froms3streamer.s3streamerimportmultipartresponse=multipart('myfile.iso','installer/images')# To remove a file from S3.froms3streamer.s3streamerimportmultipartresponse=multipart('myfile.iso','installer/images',purge=True)In BASH:# To upload a new file.python3-c\"from s3streamer.s3streamer import multipart; \
response = multipart(\
'myfile.iso', \
'installer/images')"# To remove a file from S3.python3-c\"from s3streamer.s3streamer import multipart; \
response = multipart(\
'myfile.iso', \
'installer/images', \
purge = True)"If the upload is successful, the file will be available atinstaller/images/myfile.iso.