Saravanan666/pypi-org huggingface dataset:
Package: s3url
s3url is a simple command line utility for generating time-sensitive, sharable
urls for private s3 files.### ExamplesGenerate a signed URL valid for 24 hours to downloadfile/to/share.txtfrom
the bucket namedmy-bucket:$ s3url my-bucket/file/to/share.txtAlso acceptss3://style URIs:$ s3url s3://my-bucket/file/to/share.txtOrhttp(s)://URIs (with the bucket as the subdomain or in the path):$ s3urlhttp://s3.amazonaws.com/my-bucket/file/to/share.txt$ s3urlhttps://my-bucket.s3.amazonaws.com/file/to/share.txtSpecify the time until expiration (1 hour and 30 minutes from now):$ s3url -e 1h30m my-bucket/file/to/share.txtAWS credentials are loaded from environment variables by default. Export the
variablesAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY. To specify
them at the command line, use:$ s3url -k YourAccessKeyId:YourSecretAccessKey my-bucket/file/to/share.txt### Installation$ pip install s3url### LicenseCopyright © 2014 Ford Hurley <ford.hurley@gmail.com>This work is free. You can redistribute it and/or modify it under the
terms of the Do What The Fuck You Want To Public License, Version 2,
as published by Sam Hocevar. See the COPYING file for more details.[![PyPI version](http://img.shields.io/pypi/v/s3url.svg)](https://pypi.python.org/pypi/s3url)![Build status](https://www.codeship.io/projects/eafe1020-a532-0131-51d0-3e9695abd440/status)
Package: s3-url-helper
OverviewS3Url is a helper class that can help with simple operations on AWS S3 objects.InstallationGet fromhttps://pypi.org/project/s3-url-helper/e.gpip install s3-url-helperUsagefrom s3_url import S3Url

file_url = S3Url(f's3://test-bucket/prefix/file.json')

# url component properties
assert file_url.bucket == 'test-bucket'
assert file_url.key == 'prefix/file.json'

# factory methods
file_url = S3Url.from_url('s3://test-bucket/prefix/file.json')
file_url = S3Url.from_bucket_key(bucket='test-bucket', key='prefix/file.json')

# access underlying boto3 s3 resource object
boto3_obj = file_url.object

# return url string
url_str: str = file_url.url
url_str: str = str(file_url)
assert url_str == 's3://test-bucket/prefix/file.json'

# check if file exists
exists: bool = file_url.exists()
assert exists

# check if any files exist in prefix (url should end with /)
prefix_exists = S3Url('s3://test-bucket/prefix/').prefix_exists()
assert prefix_exists

# read text/json
file_content: str = file_url.read_text()
file_content_json: json = file_url.read_json()

# delete file
file_url.delete()
assert not file_url.exists()

# write text/json
file_url.write_text("test data")
file_url.write_json({"testEntry": "test data"})
assert file_url.exists()

# copy to another object
file_url.copy_to('s3://test-bucket/prefix/file-copy.json')
S3Url('s3://test-bucket/prefix/another-file-copy.json').copy_from(file_url)

# delete all filed in prefix
prefix_url = S3Url('s3://test-bucket/prefix/')
prefix_url.delete_dir()

assert not prefix_url.prefix_exists()
assert not file_url.exists()

# see tests for more examplesDevelopment notesInstall dev dependencies:
pip install '.[dev]'
pip install '.[build]'build/upload:py -m build
py -m twine upload --repository pypi dist/*todo - write docstrings
Package: s3urls
S3urlsParse and buildAmazon S3URLs.UsageParse S3 URLsfroms3urlsimportparse_url>>>parse_url('https://my-bucket.s3.amazonaws.com/my-key/'){'bucket':'my-bucket','key':'my-key/'}>>>parse_url('https://s3-eu-west-1.amazonaws.com/my-bucket/my-key/'){'bucket':'my-bucket','key':'my-key/'}>>>parse_url('s3://my-bucket/my-key'){'bucket':'my-bucket','key':'my-key/'}>>>parse_url('s3://user@my-bucket/my-key'){'bucket':'my-bucket','key':'my-key/','credential_name':'user'}Build S3 URLsfroms3urlsimportbuild_url>>>build_url('s3','my-bucket','my-key/')'s3://my-bucket/my-key/'>>>build_url('s3','my-bucket','my-key/',credential_name='user')'s3://user@my-bucket/my-key/'>>>build_url('bucket-in-path','my-bucket','my-key/')'https://s3.amazonaws.com/my-bucket/my-key/'>>>build_url('bucket-in-path','my-bucket','my-key/',region='eu-west-1')'https://s3-eu-west-1.amazonaws.com/my-bucket/my-key/'>>>build_url('bucket-in-netloc','my-bucket','my-key/')'https://my-bucket.s3.amazonaws.com/my-key/'
Package: s3util
s3utilA command line tool for s3 with autopath completions and features like vim, cat, recursive restores and batch operationsPath auto completionTab completion of s3 paths and bucketss3util vimedit any file on s3 or create new file on s3 using vims3util batch deletedelete multiple directories or file in batch of 1000 objectsoperations availble$ s3util vim <path_to_file>to edit file on vim$ s3util cat <space_seperated_paths_to_file>to print content of file$ s3util rm <space_seperated_paths_to_file>to delete objects on paths$ s3util restore <space_seperated_paths_to_file>to restore objects from glacier on paths$ s3util ls <space_seperated_paths_to_file>to list objects on paths$ s3util cp <path1> <path2>Copy path1 to path2$ s3util mv <path1> <path2>Move path1 to path2--recursivecommand is available forcat,restore,rm,ls,cp,mvto performe operation on all the objects on the path--profilecommand can be used to use a particular profile
exs3util --profile <profile_name> <operation> <path> <options>Install Instruction$ pip install s3utilfollow below setup for autocompletion according to the terminal you are using (bash, zsh etc)For Local development$ git clone https://github.com/karannaoh/s3util.git$ cd s3util/$ pip install -r requirement.txt$ python3 setup.py install$ python3 setup.py sdist bdist_wheel$ pip install s3utilFor autocompletion to work,For bash,$ activate-global-python-argcompleteand addeval "$(register-python-argcomplete s3util)"in .bashrc fileFor zshenablebashcompinit$ autoload -U bashcompinit$ bashcompinitand addeval "$(register-python-argcomplete s3util)"in .zshrc filefollowhttps://github.com/kislyuk/argcompletefor other programs
Package: s3vault
s3vaultA library for storing documents in AWS s3Before you startFor best results, create a bucket with the following settings:Encryption onVersioning onInstallationpip install s3vaultEnvironment variablesThe following environment variables can optionally be provided:AWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_S3_REGION_NAMEUsagefroms3vault.vaultimportVaultvault=Vault(bucket,owner_id,collection)You can also provideaws_access_key_id,aws_secret_access_keyandaws_s3_region_name, otherwise if will default to the values in the environment variables above.Upload a file from a urlvault.upload_from_url(url,filename,extension='pdf')url:the url to fetch the file to store fromfilenamethe Key at which to store the object. Convention::collection/:owner_id/:object_idTODO:this method should use the same convention as the below methods to enforce consistencyGet a filevault.get_object(object_id)vault.get_versions(object_id)
Package: s3vaultlib
S3Vaultlib is a Python Library and CLI tool that enable you to
implements a secure vault / configuration datastore for your AWS
platform by using AWS resources: CloudFormation, S3, IAM, KMS S3Vaultlib
it’syet another vaultwith the goal to give easy maintainability,
use only AWS resource and with strong security patterns in mind.Why a vault?It’s a common pattern in SRE and DevSecOps to create resources
environment unaware and configure the resource automatically when is
deployed in a specific environmentS3Vaultlib FeaturesUse Server Side Encryption to store the objects on S3 with per-role
KMS keyUse per role encryption with least privilege patterns to access the
vault. Each role in the vaultcan only consumeits own
keymaterialsSpecial elevated privileged mode with a specific role able to produce
and configure keymaterials, with only temporary accessSave, retrieve, update objects in the vaultIntegrates flawlessly with Ansible by exposing an action plugin that
allows you to expand templates by using variables / keymaterials from
the vaultPowerful CLI to create, manage and update the objects in the vaultEasy maintainable via simple yaml fileExpose a flexyble python library to extend functionalities or
implement the retrieval of keymaterials from your code.S3vaultlib ArchitectureS3Vaultlib requires no installation or security patches / updates.The architecture leverages entirely on AWS existing resource to create a
secure vault with Role Base Access Control, versioning and region
awareness.It integrates with theIAMto generate the necessary roles and
policies,KMSto generate per-role keys,S3to configure the
bucket policies to enforce high level of security andCloudFormationto create the Infrastructure as Code that combine all the above in a
powerful vault.Check In depth Architecture for more informationHOW-TOsExample scenariosProvisioning a vault: A simple example to see how to provision a vault via the command line
interfaceConfigure NGINX with S3Vaultlib: A simple
example where we deploy an environment unaware NGINX instance and
it’s configured via S3Vaultlib ansible pluginCLI UsageThe complete documentation can be found here:
CLI UsageAlternativesCurrently there are several alternative patterns used.Configuration / Keymaterials encrypted in gitPlease don’t do this, really!Vaultby HashicorpFull featured vault system, widely used in the DevOPS community.
But it’s also yet another system to deploy and maintain in high
availability and also, it requires keymaterials for the
installation (since is not a native AWS component)AWS Secret ManagerVery valid alternative offered by AWS. Still lack a bit of
flexibility to be used transparently in your bootstrap pipelines
for EC2 / Dockers / Lambdas / Applications
Package: s3web
s3webSimple upload files to amazon s3 as a website.Usages3web –config path/to/s3web.cfg path/to/directory-to-uploadAn example of a s3web.cfg:[s3web]
api_key = Some-api-key
api_secret = The-api-secret
domain_name = www.example.comThen just set the CNAME and all that other stuff.
Package: s3wm
S3WM🛠️⚙️This is a WIP project. Don't beleive in README ⚙️🛠️This project is a yet anotherWindow manager.Main Idea behind this project ismodularityand
window managerconfiguration in Python.You can evenbind a python functionto some key combination.ConfigurationMain configuration file must be located in$HOME/.s3wm_conf.py.
S3WM configuration examples can be found in examples folder.How to Run/test# start nested X11 session with XephyrXephyr:1+xinerama-screen1280x720-reset-terminate&# Run it with proper DISPLAY ENVDISPLAY=:1.0s3wm# To grab or release host keys press `Ctrl` + `Shift`
Package: s3workers
s3workersHelper to simplify concurrent access to object scanning in AWS S3 buckets.Free software: MIT licenseDocumentation:https://s3workers.readthedocs.io.FeaturesS3workers provides faster list and delete operations on S3 buckets by opening up simultaneous
connections to issue distinct sets of shared prefix queries. Effectively, this splits up the query
space into 36 independent queries (26 alpha and 10 numeric prefixes). For example, a request to list
all objects in themyfancy/bucket would result in concurrent list queries to S3 for everything
frommyfancy/a...throughmyfancy/b...and everything frommyfancy/0...throughmyfancy/9..., all at the same time, reporting and collating the results locally.SelectionThe default output of s3workers is to simply list (or delete) all objects found at the prefix
requested. However, often it is advantageous to restrict the output to only those matching certain
criteria. The--selectoption provides the ability for evaluating matches using any normal
Python operators or builtins against one or more of the following variables provided to the selector
for each object found:name: The full S3 key name, everythingexceptthe bucket name (string)size: The number of bytes as used by the S3 object (integer).md5: The MD5 hash of the S3 object (string).last_modified: The timestamp indicating the last time the S3 object was changed (string).ReductionIn cases where aggregation of some kind is desired, s3workers provides the ability to execute
reduction logic against an accumulator value. For example, to produce a sum of the size of all
selected S3 objects or to even group the size according to MD5 values. See the usage output for
examples. In all cases, the same variables provided by selection are also provided when reducing.CreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject
template.History0.3.0 (2017-01-01)Refactor/reorg code for usability/readability; add docs; add tests.0.2.0 (2016-12-30)Minor fixes, adding docs, using common logging options.0.1.0 (2016-12-28)First release on PyPI.
Package: s3-zip-uploader-Ivan-Yukish
Zip to S3 file uploader with concurrency supportHow to start the applicationThe commands:First you have to install package:$ pip install s3-zip-uploader-Ivan-YukishThen run the uploader:$ python3 -m s3_zip_uploader_Ivan_Yukish https://example.com/path-to-zip-archive bucket_example_name s3_key_prefix --concurrency 8 --verboseApplication descriptionBefore running app you should set up your AWS CLI account credentials:docs.Running tests:pytestFile structureOur file structure is:├── s3_zip_uploader_Ivan_Yukish
│      ├── __init__.py
│      ├── main.py
│      ├── uploader.py
├── tests
│      ├── fixtures.py
│      ├── tests.py
├── .gitignore
├── LICENCE
├── pytest.ini
├── README.md
├── requirements.txt
└── setup.py
Package: s4api
A Python3 client library for the S4 REST APIThis is the Python3 client library for S4 Services APIs:1.Text Analytics Service2.Fully-Managed GraphDB Database in the Cloud3.Knowledge Graphs AccessInstallationEasiest method$(sudo)pip3installs4apiAlternate methodsDownload the project’s source code fromGitHuband run:$python3setup.pyorDownload the package directly fromPyPI.DocumentationDocumentation is generated by Sphinx-autodoc and is availablehere.Please check theS4 Wikifor more information, code examples and tutorials.
Package: s4-clarity
Used in numerous labs around the world, the S4-Clarity library provides an easy-to-use integration with the BaseSpace Clarity LIMS API. The package includes:Classes representing familiar Clarity API entities that provide read-write access to most properties.Helper methods that simplify common operations.Base classes for scripts that integrate with Clarity: EPPs, DSAs, and shell scripts.Utilities that help with Clarity-related tasks, such as managing config slices, or automating the completion of a Step.The S4-Clarity library lets developers interact with the Clarity API in fewer lines of code. With HTTP and XML boilerplate out of the way, you’ll have your integration built in no time at all.froms4.clarity.scriptsimportTriggeredStepEPPLibraryVolume=2.0MolWeightBasePair=660*1e6# micrograms / molAssumedBasePairs=400.0TargetMolarity=4.0Overage=4classNormalization(TriggeredStepEPP):defon_record_details_enter(self):self.prefetch(self.PREFETCH_INPUTS,self.PREFETCH_OUTPUTS)foriomapinself.step.details.iomaps:library_concentration=iomap.input["Concentration"]library_molarity=library_concentration/(AssumedBasePairs*MolWeightBasePair)iomap.output["Concentration"]=library_concentrationiomap.output["Molarity (nM)"]=library_molarityiomap.output["Library Vol (uL)"]=LibraryVolumeiomap.output["Tris HCl (uL)"]=LibraryVolume*(library_molarity/TargetMolarity-1)self.lims.artifacts.batch_update(self.step.details.outputs)if__name__=="__main__":Normalization.main()InstallationTo install s4-clarity, simply use pip:$ pip install s4-clarity
Package: s4d
S4Dis an open source package for speaker diarization.The aim ofS4Dis to provide an educational and efficient toolkitfor speaker diarization including the whole chain of treatmentthat goes from the audio data to the analysis of the system performance.PREREQUISITESSidekit for Diarizationrequires the following software installed for your platform:[Python](http://www.python.org)[NumPy](http://www.numpy.org/)[Scipy](http://http://www.scipy.org/)[Pandas](http://http://www.pandas.org/)[GLPK](https://www.gnu.org/software/glpk/)[Sphinx 1.1.0 or newer](http://http://sphinx-doc.org/) to build the documentation[SIDEKIT](https://lium.univ-lemans.fr/sidekit/)INSTALLATIONWe recommend the use of a virtual environment (e.g. [Miniconda](https://conda.io/miniconda.html) or [Virtualenv](https://virtualenv.readthedocs.io/en/latest/)).TUTORIALSOnce your installation is complete, you can take a look at the [tutorials](https://git-lium.univ-lemans.fr/Meignier/s4d/tree/master/tutorials).Authors:Sylvain Meignier &
Anthony Larcher &
Pierre-Alexandre Broux &
Florent DesnousVersion:0.1.4.1 of 2020/01/23
Package: s4-platform-api
S4 Platform APIRequirementss4-platform-apirequires Python 3.9+.ContributingThis may change in the future, but for now, please followconventional commitsstyle. The short summary of this is, when you open a PR or make a commit:If you change code, prefix your PR title withfix:if it's a fix orfeat:if it adds a feature.If you change the API, include the textBREAKING CHANGEin your PR title.If you don't change code at all, prefix the PR title withci:,docs:,test:,build:, or something else.The version of the library will be incremented according to the text in merged PRs, so that it followssemver.Clarifications"Adding a feature" includes, at least, any time you make something available to users of the library which wasn't available before. If you remove or change current behaviour, you're changing the API.Developer SetupInstall poetry:brew install poetryInstall dependencies:poetry installExport AUTH0_DOMAIN=dev-d3p316sq.us.auth0.com and AUTH0_AUDIENCE=https://dev-d3p316sq.us.auth0.com/api/v2/as environment variablesExport AUTH0_TEST_CLIENT_ID=D9wQOscnTJnXEmimsviFiDimppJ7A1bK and AUTH0_TEST_CLIENT_SECRET as environment variables. Secret's value is in LastPass as "Auth0 test client secret" in "Shared-Platypus" folderTestingUnit testsFrom the root directory:poetry run pytest test/Integration testsFrom the root directory:poetry run pytest integration_test/All testsFrom the root directory:poetry run pytestBuildingTo build a source and binary wheel distribution:poetry buildExamplesScripts from theexamplesdirectory use the library. These can be run from the command line.`poetry run examples/get_prospective_task 614e1f114640ba4b197af7ea`The fileexamples/.envcontains configurable parameters used by the examples scripts. Edit this as necessary or
override by setting environment variables on the system running the scripts.HOST_URL=http://example.org/orange get_prospective_task 614e1f114640ba4b197af7eaURI OverrideIf this library is run within an AWS deployment application queries need to be altered to use local networking. If the
environment variableURI_OVERRIDEis present queries to the application will be overwritten to use this value.
e.g.
If connection is initialized with a value ofhttps://app.laboratory.plat.s4hq.comand the environment variableURI_OVERRIDEexists with a valuehttp://localhost:8080, a query intended to go tohttps://app.laboratory.plat.s4hq.com/blueHealthwill actually go tohttp://localhost:8080/blueHealth
Package: s4s
# Service for Script
An one-line tool to wrap a simple script or command into an HTTP server.# Install
pip3 install s4s# Usage
s4s –help# Examples
s4s -p 7777 ‘echo $text’
curl -F text=hellohttp://127.0.0.1:7777s4s -p 7777 nl
curl -Ffile=@/path/to/filehttp://127.0.0.1:7777
Package: s4u2p
This is an small extension to the python kerberos package.
It adds support for “services for user to self/proxy” (s4u2s/s4u2p) kerberos functionality.
It adds authGSSImpersonation* functions that let you (as a server) act on behalf of a user.
No actual delegated ticket from a user needs to be forwarded to the server component.
Instead the server account needs to be set up as described here:http://k5wiki.kerberos.org/wiki/Manual_Testing#Services4User_testingFor some overview of s4u see this:http://msdn.microsoft.com/en-us/magazine/cc188757.aspx#S2
Package: s4u.upgrade
This package implements a very minimal upgrade framework for use in Python
applications. Its design consists of three parts:context providers, which are responsible for setting up the context
required for an upgrade step. For example establishing a connection to
a SQL server or configuring SQLALchemy.upgrade steps, which are functions that perform the actual upgrade
steps.acommandline toolto run theupgrade stepsand requiredcontext
providers.In order to keep the framework minimal there are a few things which are
deliberately not supported:No versioning of the environment is done; the tool will always run all
all upgrade steps. Each step must support being run multiple times
without unexpected results.Downgrades are not supported.Depdendencies between upgrade steps are not supported.Here is a basic context provider which configures SQLAlchemy:@upgrade_context('sql')
def setup_sqlalchemy(options):
    engine = create_engine('postgresql:///projectA')
    return {'sql-engine': engine}Here is an example upgrade step to add missing tables and indices in a project
using SQLALchemy:@upgrade_step(require=['sql'])
def add_missing_tables(environment):
    meta.metadata.create_all(environment['sql-engline'])And this is how you run the upgrade:$ bin/upgrade --scan my.package
Package: s5
SciNet Super Simple Secrets ServerS⁵ is a minimalist key manager. The server will ask for a secret (e.g. a password) and store it in memory until the client queries and receives it. The specific use case is a queued job in a high performance computing (HPC) environment that needs to use encryption, but the encryption key should not be passed in plaintext when the job is submitted.When the server starts, it generates a random 96-byte token. The high 48 bytes are used to identify the client, and the low 48 bytes are used as a one-time pad. The client and server communicate via HTTP, despite not using transport layer security, the encrypted response provides resilience to a man-in-the-middle attack.By default, the token is stored along with the host and port information in the user's home directory. This makes it automatically accessible to a client process running on a compute node that shares a filesystem with the node (e.g. a login node) where the server is running. If the client is not on a shared filesystem, the connection information (including the token) can be specified as command line arguments or the client file can be manually copied.The server shuts down by default once a successful query is made, but this behaviour can be adjusted with the--success-maxcommand line argument.UsageStart the server withs5serverand type in the secret. The connection information is saved to~/.s5client.jsonby default. If the client will be running on a node that shares a filesystem and can reach the server's node by its hostname, then nothing needs to be done. Run the client withs5clientand receive the secret into standard output.
Package: s5cmd
s5cmd Python DistributionsOverviews5cmdis a very fast S3 and local filesystem execution tool. It comes with
support for a multitude of operations including tab completion and wildcard
support for files, which can be very handy for your object storage workflow
while working with large number of files.This project provides the infrastructure for building thes5cmdPython wheels.
For more information abouts5cmd, please refer tohttps://github.com/peak/s5cmd.The Python wheels provided here contain the officials5cmdexecutable, which
is sourced from theGitHub releases.
Once the wheel is installed, a convenient launcher executable is automatically
placed in the PATH. This launcher is created during installation by pip,
leveraging the[project.scripts]configuration defined in thepyproject.tomlfile.PlatformsThe following platforms are supported by the binary wheels:OSArchWindows64-bit32-bitARM64Linux Intelmanylinux 64-bitmusllinux 64-bitmanylinux 32-bitmusllinux 32-bitLinux ARMmanylinux AArch64musllinux AArch64Linux PowerPCmanylinux ppc64lemusllinux ppc64lemacOS 10.10+IntelmacOS 11+Apple SiliconLicenseThis project is maintained by Jean-Christophe Fillion-Robin from Kitware Inc. It
is covered by the OSI-approved MIT License.s5cmdis distributed under the OSI-approved MIT License. For further details
regardings5cmd, please visithttps://github.com/peak/s5cmd.