Saravanan666/pypi-org huggingface dataset:
Package: s3lify
S3lifyS3lify, a script to deploy secure (SSL) single page application (SPA) or HTML static site on AWS S3, using S3, Route53, Cloudfront and ACM.How does it work?InstallInstall S3lify. It's a Python script, run the pip command belowpip install s3lifyThen navigate to the root of the directory that contains your site, then initialize S3lifys3lify initThen edit the config files3lify.ymlto match your site's informationSetup ProcessSetup S3lifys3lify setupIt creates the website on S3 by creating a new website bucketIt creates an DNS entry on Route53 pointing to S3 website bucketIt provisions an SSL certificate using ACM, and validates the domain by adding the and ACM CNAME on Route53(If the domain is purchased through Route53Domains, it will update the DNS server name)It creates a new Cloudfront distribution and attaches the SSL certificate provided by ACMIt updates the Route53 with the Cloudfront's domain nameNow your SSL site is ready to deploy. You will be able to access your site via 'https://yoursite.com'Ready to deploy!Deploy Processs3lify deployIt purges all files in S3 bucketIt invalidates all objects in cloudfrontUpload the directory to S3Sites updated successfullyThat's it!AWS Service UsedS3Route53Route53 DomainsCloudfrontACMFAQCan I use only S3 website?Yes. In thes3lify.ymlsetdistribution: s3I deploy my site, but I don't see the changes.Make sure you build your site first, then runs3lify deployCommandspip install s3lify: Install S3lifys3lify init: Init S3lify in the directorys3lify setup: Setup S3lify and all the AWS services neededs3lify deploy: Deploy the sites3lify status: see the status of the siteConfigAt the root of the directory, outside of the build folder, create the config files3lify.yml# s3lify.yml
# -----------------------------------------------------------------------------
# S3lify config
# -----------------------------------------------------------------------------

#:: AWS credentials and region
aws_region: us-east-1
# aws_access_key_id: 
# aws_secret_access_key: 

#:: Site Info
# The domain without the 'www.'. It will create the S3 bucket with it
domain:  # 'mysite.com'

# The directory containing the site to upload, from the CWD running s3now
site_directory: ./mysite

# For working with SPA, point the error_file to 'index.html' 
index_file: index.html # index.html
error_file: error.html # error.html

#:: Purge files.
purge_files: True         # To delete all the files on S3
purge_exclude_files:      # Files not to delete on purge
  - index.html
  - error.html

#:: distribution
# The type of distribution
# s3 | route53 | cloudfront
# default: s3
# s3: to deploy to s3 only
# route53: will deploy on s3 and set the domain on Route53
# cloudfront: deploy on s3, set route53, set ACM for SSL and activate cloudfront 
distribution: cloudfront

#:: update_route53domains_dns
# default: True
# when true it will attempt to update the domain DNS with the route53 Name servers
update_route53domains_dns: True

#:: invalidate_cloudfront_objects
# To invalidate cloudfront objects, so it can retrieve new contents
invalidate_cloudfront_objects: TrueLicense: MIT
Package: s3like
s3likeA small package that is used by COMP to read and write model results to S3 like object storage systems. This means that this package is compatible with DigitalOcean Spaces and AWS S3, since DO Spaces uses the same API as AWS S3.Setup:pipinstalls3likeexportOBJ_STORAGE_ACCESS=...exportOBJ_STORAGE_SECRET=...exportOBJ_STORAGE_ENDPOINT=...exportOBJ_STORAGE_EDGE=...exportOBJ_STORAGE_BUCKET=...Use:imports3like# run_model returns data that is compliant with the COMP outputs api.local_result,task_id=run_model(**kwargs)remote_result=s3like.write_to_s3like(task_id,local_result)round_trip=s3like.read_from_s3like(remote_result)assertlocal_result==round_tripTest:py.test-v
Package: s3lite
S3LiteMinimal async s3 client implementation in python.InstallationRequirements:Python 3.9+pipinstalls3liteExamplesList bucketsfroms3liteimportClientclient=Client("key-id","secret-key","https://s3-endpoint")forbucketinawaitclient.ls_buckets():print(bucket.name)Create new bucketfroms3liteimportClientclient=Client("key-id","secret-key","https://s3-endpoint")bucket=awaitclient.create_bucket("new-bucket")Upload objectfromioimportBytesIOfroms3liteimportClientclient=Client("key-id","secret-key","https://s3-endpoint")object_from_path=awaitclient.upload_object("new-bucket","test-image.jpg","./local-image.png")withopen("document.txt","rb")asf:# Make sure to open file in read-binary modeobject_from_file=awaitclient.upload_object("new-bucket","document.txt",f)bytes_io=BytesIO(b"test file content")object_from_bytesio=awaitclient.upload_object("new-bucket","test.txt",bytes_io)Download objectfroms3liteimportClientclient=Client("key-id","secret-key","https://s3-endpoint")saved_path=awaitclient.download_object("new-bucket","test-image.jpg","./local-image.png")bytes_io=awaitclient.download_object("new-bucket","document.txt",in_memory=True)partial_bytes=awaitclient.download_object("new-bucket","test.txt",in_memory=True,offset=4,limit=6)Generate presigned urlfroms3liteimportClientclient=Client("key-id","secret-key","https://s3-endpoint")obj=awaitclient.upload_object("new-bucket","test-image.jpg","./local-image.png")presigned_url=obj.share()# Url will be valid for 1 daypresigned_url_1h=client.share("new-bucket","test.txt",3600)# Url will be valid for 1 hourDelete objectfroms3liteimportClientclient=Client("key-id","secret-key","https://s3-endpoint")saved_path=awaitclient.delete_object("new-bucket","test-image.jpg")
Package: s3lncoll
s3lncoll========Read files from S3 as defined by a key prefix and map them by lines toa set of optionally gzip compressed output files in S3, with theoutput files limited by (pre-compressed) file size.  The string "{}"in the output key will be substituted with the (zero-based) index ofthe output files.::s3lncoll: Line stream s3 files into ~uniform lumps in S3Usage: s3lncoll {{arguments}} {{options}}Arguments:from [text]  S3 URL prefix to clumpto [text]    S3 URL for target clump ('{}' will be the count)Options:-h, --help             Show this help message and exit-H, --HELP             Help for all sub-commands-D, --debug            Enable debug logging-d, --delete           Delete source files/keys-j, --json             Validate each line as JSONM-q, --quiet            Be quiet, be vewy vewy quiet-V, --version          Report installed version-z, --compress         Ccompress (gzip) the target(s)-b, --blocksize [int]  Maximum size of pre-compressed output files in bytes. (default: 1048576)Architecture============s3lncoll has a pipe and filter architecture which streams a set of keys as defined by a prefixthrough a `LineStream`. `LineStream` reads the files under the keys and spits out a single linevia an iterator. `RotatingFileCtx` receives that stream of lines and aggregates them into chunkedfiles (of a maximum size or a single line, whichever is the larger), followed by flushing thelines out to a provided S3 path.+----------------------------------------------------------------------------------------------------+|                                                                                                    ||                   +------------+                +-------------------------+                        ||                   |            |                |                         |                        ||    bucket Keys    | LineStream |     Lines      |     RotatingFileCtx     |  S3 Files              ||    ------------>  |            |  ------------> |                         | ------------>          ||                   |            |                |                         |                        ||                   +------------+                +-------------------------+                        ||                                                                                                    ||                   +---------------------------+                                                    ||                   |                           |                                                    ||                   |     cmd.py: Scheduler     |                                                    ||                   |                           |                                                    ||                   +---------------------------+                                                    ||                                                                                                    ||                                                                                          s3lncoll  |+----------------------------------------------------------------------------------------------------+
Package: s3-loader
s3-loaderInstallpip install s3-loaderUsage:$s3-loader[OPTIONS]COMMAND[ARGS]...Options:--install-completion: Install completion for the current shell.--show-completion: Show completion for the current shell, to copy it or customize the installation.--help: Show this message and exit.Commands:auth: to set aws credetialsempty-bucket: delete everything in an s3 bucketupload-dir: upload an entire directory to s3upload-file: upload a single file to s3s3-loader authto set aws credetialsUsage:$s3-loaderauth[OPTIONS]Options:--aws-access-key-id TEXT: Your aws access key  [required]--aws-secret-access-key TEXT: Your aws secret acess key  [required]--help: Show this message and exit.s3-loader empty-bucketdelete everything in an s3 bucketUsage:$s3-loaderempty-bucket[OPTIONS]Options:--bucket-name TEXT: Your bucket name  [required]--help: Show this message and exit.s3-loader upload-dirupload an entire directory to s3Usage:$s3-loaderupload-dir[OPTIONS]Options:--path TEXT: Path to your directory  [required]--bucket-name TEXT: Your bucket name  [required]--help: Show this message and exit.s3-loader upload-fileupload a single file to s3Usage:$s3-loaderupload-file[OPTIONS]Options:--local-file TEXT: Local file name  [required]--bucket TEXT: Your bucket name  [required]--s3-file TEXT: Name the file will use on s3  [required]--help: Show this message and exit.
Package: s3local
s3localCache the object in s3 to localhost.Create a cache corresponding to s3 and automatically create a path for localhost and return it.Once downloaded files remain in localhost as cache, the second migration download will be skippedworks on python3.6 or higherSettingsaws auth support following.environment variablesprofile(use --aws-profile option.)instance profileExamplesdownload object and list object$s3localdownload-us3://mybucket/artifacts/--debug2021-05-1411:27:13,367DEBUG-Copying:s3://mybucket/artifacts/main.log>/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log2021-05-1411:27:13,367DEBUG-Copying:s3://mybucket/artifacts/main2.log>/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log2021-05-1411:27:13,367DEBUG-Copying:s3://mybucket/artifacts/main3.log>/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log# next download is skip$s3localdownload-us3://mybucket/artifacts/--debug2021-05-1414:08:02,970DEBUG-skipalreadyexistsinlocal:s3://mybucket/artifacts/main.log2021-05-1414:08:02,970DEBUG-skipalreadyexistsinlocal:s3://mybucket/artifacts/main2.log2021-05-1414:08:02,970DEBUG-skipalreadyexistsinlocal:s3://mybucket/artifacts/main3.log# overwrite download. (not skip)$s3localdownload-us3://mybucket/artifacts/--debug--no-skip-exist2021-05-1411:27:13,367DEBUG-Copying:s3://mybucket/artifacts/main.log>/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log2021-05-1411:27:13,367DEBUG-Copying:s3://mybucket/artifacts/main2.log>/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log2021-05-1411:27:13,367DEBUG-Copying:s3://mybucket/artifacts/main3.log>/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.logBy default$HOME/.s3localis the root directory.The format of path in local is as follows:$HOME/.s3local/s3/${bucket}/${key}You can change root by setting an environment variable S3LOCAL_ROOT.$s3locallist-local-us3://mybucket/artifacts/
/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log
/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log
/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.logupload object$s3localupload-stox.ini-us3://mybucket/test/2023-05-3110:44:08,474INFO-Copyingtos3:tox.ini=>s3://mybucket/test/tox.iniPython APIdownloadfroms3localimportDownloaders3local=Downloader("s3://mybucket/artifacts/")list=s3local.list_local_path(download=True)print(list)#=> [#     "/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main.log",#     "/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main2.log",#     "/Users/hiroshi.toyama/.s3local/s3/mybucket/artifacts/main3.log",# ]uploadfroms3localimportUploaders3local=Uploader("s3://mybucket/artifacts/")uploader.upload("output/hoge.txt")#=> s3://mybucket/artifacts/hoge.txtuploader.upload("output")#=> s3://mybucket/artifacts/output/hoge.txtInstallationpipinstalls3localCIinstall test package$ ./scripts/ci.sh installtest$ ./scripts/ci.sh run-testflake8 and black and pytest.release pypi$ ./scripts/ci.sh releasegit tag and pypi release.
Package: s3-log-parse
A simple tool for parsing AWS S3 access logs. Can either be used from Python or via a command line wrapper.InstallationUse pippip install s3-log-parseCommand Line usageThe s3logparse command line script will reformat logs according to the following conventionsAll fields are tab separatedNo fields are quotedNull text fields (represented by -) are emptyTimestamp is ISO8601 formattedNull numeric fields are converted to 0The script can accept a single argument, the name of a log files3logparse mys3logfile.txtpiping will also workcat allmys3logs/* | s3logparsePython APIThe parse_log_lines function iterates over raw log lines and converts them to a namedtuple with the following attributes: bucket_owner, bucket, timestamp, remote_ip, requester, request_id, operation, s3_key, request_uri, status_code, error_code, bytes_sent, object_size, total_time, turn_around_time, referrer, user_agent, version_idHere is a simple example which extracts IP addresses from logsfroms3logparseimports3logparsewithopen('mys3logs.txt')asfh:forlog_entryinparse_log_lines(fh.readlines()):print(log_entry.ip_address)
Package: s3-log-query
S3 Log QueryS3 Log Query is a Python library that takes one or more log files stored in S3, fetches and aggregates log messages from the files based on certain parameters, and returns the messages in order of occurrence. This library utilizes S3 Select and multithreading for performance.InstallationUse the package managerpipto install S3 Log Query.pipinstalls3-log-queryUsagefroms3_log_queryimportLogQuerylogs=LogQuery(log_paths={'server1':'server1/server1.log','db server':'db_server/db_server.log','server2':'server2/server2.log','server3':'server3/server3/log'},s3_bucket='mikethoun-logs')logs.query(start='02/28/2020 5:20:55',entries=100,keys=['server1','db server'],min_severity=logging.WARN)ContributingPull requests are welcome.LicenseApache License 2.0
Package: s3logs
Get files from directory by mask and push them to S3Installpipinstalls3logsUsages3logsconfig.confWhere config.conf can use that structure:[S3]access_key=<S3_KEY>secret_key=<S3_SECRET_KEY>host=<s3.example.com>bucket=<bucket_name>chunk_size=<bytes, default=52428800>[logs]suffix=.gzkey_suffix=.gzdirectory=/var/log/nginx/depth=30[map]example.com-access.log=example/accessexample.com-error.log=example/errormysite.me.access.log=mysite/accessWhen it used with that config, script takes all files in directory/var/log/nginx/, filter only those, which ends with.gzand send it to S3, according to map.For example, /var/log/nginx now consists of:example.com-access.log
example.com-access.log.0.gz
example.com-access.log.1.gz
example.com-error.log
example.com-error.log.0.gz
example.com-error.log.1.gz
mysite.me.access.log
mysite.me.access.log.0.gz
mysite.me.error.log
mysite.me.error.log.0.gzSo, if today is 9 December 2015, and your hostname is node1, on your S3<bucket_name>would be those keys:node1/example/access/2015-12-09.gz
node1/example/access/2015-12-08.gz
node1/example/error/2015-12-09.gz
node1/example/error/2015-12-08.gz
node1/mysite/access/2015-12-09.gzBecause we have not explain how maps mysite.me.error.log.0.gz - it would be skipped.Script also checks whether file exists in S3 and push only those, which are not.Parameterdepthstops pushing, if filename is older, than.<depth>.gz
Package: s3m
S3M - is a sqlite3 wrapper for multithreaded python applications.Table of contentsMultithreadingDocumentationInstallationUsageMultithreading supportS3M prevents multiple database operations (or transactions, can be disabled) from running at once.
That’s basically the whole point of this library.DocumentationDocumentation is available atRead the DocsInstallationpipinstalls3morpythonsetup.pyinstallThere are no requirements.UsageThe usage is pretty much the same as with built-insqlite3.
Package: s3manage
Utilities to help manage S3 bucketsFree software: BSD licenseInstallationpip install s3manageDocumentationhttps://s3manage.readthedocs.io/DevelopmentTo run the all tests run:toxNote, to combine the coverage data from all the tox environments run:Windowsset PYTEST_ADDOPTS=--cov-append
toxOtherPYTEST_ADDOPTS=--cov-append toxChangelog0.3.0 (2017-06-08)Added “size” and “last_modified” fields to outputChanged output to a tableStylized output for easier reading and comprehension0.2.0 (2017-06-07)Added Unicode-based iconsImproved Python 3 support0.1.0 (2017-05-26)First release on PyPI.
Package: s3manager
Python Script to upload files to AWSThis script is a wrapper over boto3 which makes developer easy to upload files to AWSGetting StartedSteps to get started:-pip install s3managerSteps to Use:-from s3manager import S3manager
s3_object_manager = S3manager(bucket_name="bucketname", access_key=<access_key>, access_secret=<access_secret>)To upload file to s3s3_object_manager.upload_file(file_name=<file_name>, filepath=<filepath>, acl='public-read')In response result will have url to the file uploaded and the key name of file in bucketto delete the objects3_object_manager.delete_file(key)to download the zip of multiple filess3_object_manager.download_files_in_zip(['fileobjkey1','fileobjkey2'],'nameofnewzip')this will create a zip file in current directory and will return zip file namePrerequisitesboto3 should be installedBuilt With[python3][boto3]AuthorsAnshul Gupta-Initial work- [AwsS3Manager]LicenseThis project is licensed under the MIT License - see theLICENSE.mdfile for details
Package: s3manifestcollectstatic
s3manifestcollectstaticOptimized collectstatic for S3ManifestStaticStorageInstallationInstall the packagepip install s3manifestcollectstaticAdds3manifestcollectstatictoINSTALLED_APPSDescriptioncollectstaticcan take a long time. When used withstorages.backends.s3boto3.S3ManifestStaticStorage,collectstaticuploads
the files twice, once without the hash at the end of the file name, and once
with the hash.  Also, it doesn't use multiple threads to upload to s3.s3manifestcollectstaticuploads the files only once, uses threads to speed
things up, and doesn't upload the files that are already on S3.collectstatic: (Around 20 minutes)$ time ./manage.py collectstatic --noinput

604 static files copied, 646 post-processed.
./manage.py collectstatic --noinput  29,94s user 2,27s system 2% cpu 20:25,06 totals3manifestcollectstatic: (Around 30 seconds)$ time ./manage.py s3manifestcollectstatic

604 static files copied to '/tmp/tmpbw0q_5lq', 646 post-processed.
Start the upload of 604 files
Uploading the manifest
./manage.py s3manifestcollectstatic  10,95s user 1,92s system 49% cpu 26,269 totalIf you want to reupload the files use-f:./manage.py s3manifestcollectstatic -fYou can limit the maximum number of workers using-w:./manage.py s3manifestcollectstatic -w 4Tested with:Python 3.9, Django 3.2, django-storages 1.11Python 3.11, Django 4.2, django-storages 1.13
Package: s3-memoize
Memoization function decorators using Amazon S3 for PythonInstallpipinstalls3-memoizeUsageSetup AWS credentials (e.g.myprofileordefault).Make a S3 bucket (e.g.mybucketname).Create test.py.fromfunctoolsimportlru_cachefroms3_memoizeimports3_fifo_cache,s3_lru_cacheBUCKET_NAME='mybucketname'# @lru_cache(maxsize=2, typed=False)@s3_fifo_cache(maxsize=2,typed=False,bucket_name=BUCKET_NAME)# @s3_lru_cache(maxsize=2, typed=False, bucket_name=BUCKET_NAME)deftest(num):print(f'test:{num}')returnnum*2print(test.cache_clear())# print(test.cache_set_expiration(days=1))print(test.cache_info())print(test(10))print(test.cache_info())print(test(10))print(test.cache_info())print(test(20))print(test.cache_info())print(test(20))print(test.cache_info())print(test(10))print(test.cache_info())print(test(30))print(test.cache_info())print(test(30))print(test.cache_info())Run. You can omitAWS_PROFILEif it’sdefault.AWS_PROFILE=myprofilepythontest.pyAuthorSusumu OTA
Package: s3-metadata-tagger
AWS S3 Metadata TaggerThe S3 Metadata tagger adds information in the form of metadata to files saved in S3.To do this, the central handler takes a file location and a metadata extracting function.
It first checks whether the file already contains the requested information via aHEADrequest.
If it does not, it downloads the file, invokes extracting function, and adds the metadata to
the s3 object with a inplaceCOPY, MetadataDirective="REPLACE"operation.This package comes with two optional variants for metadata extraction:pdf: for determining the number of pages in a pdfpicture: for determining the dimension of an imageUsageThe entrypoint into the tagger is theobject_tagger.tag_filefunction.It expects anobject_tagger.S3ObjectPath(key, bucket)and aobject_tagger.MetadataHandler(already_tagged, extraction_function, versioning_tag)object as its parameters.
The parameters of theMetadataHandlerare as follows:already_tagged: a function which receives the metadata tags already present on the object, and returns a boolean indicating whether the object should be tagged.extraction_function: a function receiving the path to the downloaded object, and returning astring -> stringdictionary embodying the metadata to add to the objectversioning_tags: astring -> stringdictionary which contains further tags to add to the s3 object, which can for example be used for tag versioningThe function tries to extract the metadata and add it to the object for up to three times.
On success, the added metadata is returned, upon failure an exception is thrown.For an example, see the service utilizing this library for automatically tagging pdfs uploaded to s3 via aws lambdain the examples directory.Structureobject_taggercontains the higher-level orchestration:object_tagger.pycontains all the logic for checking whether the file has already been tagged, downloading it, invoking the metadata script, creating the tag object, and adding it to the s3 resource.The metadata scripts are stored in their respective folderspdf_taggerThe pdf tagger usesPyPDF2to determine the amount of pages in a pdf.
Install with the[pdf]extra option.picture_taggerUsingPillow, the script gets thewidthandheightof the passed image.
Install with the[picture]extra option.TestingBothpdf_taggerandpicture_taggercome with unittests.
There is also an integration test intests/test_object_tagger.py, which expects
alocalstackinstance to run in the background.
Furthermore, the following environment variables need to be set:LOCALSTACK_S3_ENDPOINT_URL=http://localhost:4566AWS_ACCESS_KEY_ID=testAWS_SECRET_ACCESS_KEY=test