Saravanan666/pypi-org huggingface dataset:
Package: aws-excom
aws-excomAn interactive wrapper aroundaws ecs execute-command.InstallationRequiresAWS CLIandAWS CLI Session Managerto be installed on the system and configured with appropriate credentials.Then, to install the script:pipinstallaws-excomUsageaws-excomThen follow the prompts to start running commands.After running once, you can run:aws-excom--lastto skip the interactive part of the script and immediately replay the last command you
constructed. This may be useful if you accidentally exit a running session.By default, all AWS commands will use your default profile and region. To override these, pass
the following arguments:aws-excom--profilefoo--regionus-east-1By default you'll be given an interactive prompt to enter the command to execute. To skip this,
you may optionally provide a '--command' argument eg.:aws-excom--command"python3"This is handy for writing aliases for common commands:# File: aws-excom-django-python-shell.sh#!/bin/shaws-excom--command"python3 manage.py shell""$@"Local DevelopmentBefore installing the package in development mode, you may find it convenient to create a
virtual environment in which to install dependencies. The recommended way of doing this is
using Python's built-invenvmodule:# Create a directory for your environments if you don't have one alreadymkdir~/.venv# Your environment can have any namepython-mvenv~/.venv/my-aws-excom-envNext you need to activate the environment in your shell:source~/.venv/my-aws-excom-env/bin/activateNow you're ready to install the package and its dependencies:cdaws-excom# -e: Editable modepipinstall-e.You can verify this has worked by running:whichaws-excomThis prints the location of the main script installed by the package, which should be
inside your virtual environment:~/.venv/my-aws-excom-env/aws-excom/bin/aws-excom.Now, runningaws-excomin your shell will use the code you have locally. Any changes
you make to source files will be automatically used when you next run the command.Other Python VersionsTo test the package against different Python versions, first ensure that any required
versions are installed. They should be accessible usingpython3.8/python3.7or
similar.Now you can simply create a suitable virtual environment using this Python version:# Your environment can have any namepython3.8-mvenv~/.venv/my-py38-aws-excom-envAfter activating the environment, you can verify the environment contains the Python
version you've chosen using:python--versionInside this environmentaws-excomwill now run using this Python version.TestingInstall the optional development tools for testing and linting:pipinstall-e.[dev]Run the test suite using:pytest
Package: aws-exporter
AWS Prometheus ExporterPrometheus exporter for generic AWS metrics.InstallationUsageConfigurationMetricsAWS BackupAWS SNSAWS EC2 AMIsconfigurationAWS EC2 InstancesInstallation$gitclonegit@github.com:ClarkSource/aws-exporter.git
$cdaws-exporter
$pipinstall--user--upgrade.UsageJust start the exporter with read only credentials on AWS. This is using boto, so the usual rules for passing credentials apply.$aws-exporterThe exporter should be exposed on port8000ConfigurationAWS_EXPORTER_LOG_LEVEL- [debug,info,warning,error,critical,fatal]MetricsAWS Backupaws_backup_job_collector_successaws_backup_job_size_bytesaws_backup_job_percent_doneaws_backup_vault_collector_successaws_backup_vault_recovery_pointsAWS SNSsns_platform_application_collector_successsns_platform_application_enabledsns_platform_application_cert_expiryAWS EC2 AMIsec2_ami_collector_successec2_ami_creation_dateconfigurationAWS_EXPORTER_EC2_AMI_OWNERS- additional AMI owners, comma separated.AWS EC2 Instancesec2_instance_collector_successec2_instance_creation_date
Package: aws-ext
AWS EXTThe aws_ext python package contains some useful functions (built on top of boto3)
for managing some aws services. At the moment only some utilities for the Aws Glue Data catalogInstallationpipinstallaws_extUsage (command line)export LOGLEVEL=INFOAWS_PROFILE=prd aws_ext get_tables_with_many_versions [] 10aws_ext get_tables_with_many_versions "[db1,db2]" 20aws_ext delete_old_tables_versions [] 10 FalseUsage (python library)importboto3importaws_extsession=boto3.session.Session()GLUEfromaws_extimportglue_databasesglue_client=session.client("glue")Extracting tables with (too) many versionsglue_databases.get_tables_with_many_versions(glue_client,database_name="mydb",threshold=1)Deleting old tables versionsglue_databases.delete_old_tables_versions(glue_client,database_name="mydb",keep=1,dryrun=True)
Package: aws-feature-store
aws_feature_storeIt is a simplified implementation of SageMaker Feature Store approach.InstallationUse the package managerpipto install foobar.pipinstallaws_feature_storeInitialize feature groupfromaws_feature_storeimportFeatureGroup,FeatureDefinition,FeatureTypeEnumbucket_name='{bucket_for_feature_store}'s3_folder='{folder_for_feature_store}'my_feature_name='{your_feature_name}'feature_group_name=f'{my_feature_name}/commit_id={my_feature_name}_{commit_id}'feature_group=FeatureGroup(name=feature_group_name,boto3_session=boto3_session,s3_uri=f"s3://{bucket_name}/{s3_folder}")Create feature groupdefcreate_feature_group(feature_group):description="What is my feature group about"feature_script_repo="{repo_link_to_script}"data_source="{what data are used}"record_identifier_feature_name="column name to store id"event_time_feature_name="{column name to store timestamp}"partition_columns=['biz_id','customer_id']feature_definitions=[FeatureDefinition(feature_name="column_name1",feature_type=FeatureTypeEnum.INTEGRAL),FeatureDefinition(feature_name="column_name2",feature_type=FeatureTypeEnum.STRING),]feature_group.create(record_identifier_name=record_identifier_feature_name,event_time_feature_name=event_time_feature_name,feature_script_repo=feature_script_repo,partition_columns=partition_columns,data_source=data_source,description=description,file_format='parquet/json',feature_definitions=feature_definitions)returnfeature_groupiffeature_group.exists()isNone:feature_group=create_feature_group(feature_group)Ingest dataimportpandasaspddata=pd.read_json('data.json')feature_group.ingest_data_frame(data,f"mlfow_parent_run_id={parent_run_id}/{filename_without_extention}")
Package: awsfile-helper
AWS File Helper (awsfile-helper)Content Management and Retrieval for Cloud and Local StorageCode Usage (Authorization)You must have an AWS Access Key and Secret Access key to connect to AWS.Upon obtaining these keys, usefrombaseblockimportCryptoBaseos.environ['AWS_ACCESS_KEY']=CryptoBase().encrypt_str('<my-access-key>')os.environ['AWS_SECRET_KEY']=CryptoBase().encrypt_str('<my-secret-key>')This will place your encrypted credentials into the environment.TheAwsAPIclass will retrieve and decrypt these credentials to login.Code Usage (General)The following code will retrieve any file:fromawsfile_helperimportFindOwlFiled_cloud_file=FindS3File(file_name='config/graphviz/stylesheets/nlu',file_ext='yaml',file_version='0.1.0').process()The bucket is assumed to bedata-core-bast.We can modify this assumption down the road if we create new buckets for different clients, etc.A file may also be retrieved without a version, like this:d_cloud_file=FindS3File(file_name='config/graphviz/stylesheets/nlu',file_ext='yaml').process()In that case, the system will first look in the environment, using a key configuration that relies on thefile_namelike this:CONFIG_GRAPHVIZ_STYLESHEETS_NLU_VERSION.  If no value is found in the environment, the system will consider this awildcardmatch and set the version to*.  This forces the system to list the contents of the qualified path and choose the latest version.  This operation takes an additional 2-3 milliseconds, assuming the network is running smoothly.Code Usage (Ontologies)Ontologies require a special finder class, because we typically have two files we want to retrieve - an OWL model and a TXT file with synonyms.Therefore, assume that an S3 bucket exists with two files we want to retrieve:syllabus.owlandsyllabus.txt.The following code will retrieve these files:fromawsfile_helperimportFindOwlFiled_cloud_file=FindOwlFile(file_name='syllabus',version='0.1.0').process()Note that we did not specify the file extension within thefile_namevariable.It is permissible to specify explicit file names such assyllabus.txtor partial file names such.The result dictionary is keyed by file name and (with redacted contents) looks like this:{"owl":{"path":"<local path to OWL file">,"version":"<version of OWL file>","contents":"<contents of OWL file>"},"txt":{"path":"<local path to txt file">,"version":"<version of txt file>","contents":"<contents of txt file>"}}
Package: awsfind
awsfindsearches across all accounts and regions known to chaim for the instance(s)You should be able to obtainCrossAccountReadOnlychaim credentials for
every account.The script will obtain chaimrropermissions for each account in turn,
deleting them when it has finished with that account, and then, using a
thread per region query the AWS API for each specific instance mentioned
on the command line.Once all instances have been found the script stops and displays the
results.Should the script need to visit every account it will take approx. 13
minutes to do so.installClone this repository, and enter the dirgit clone https://github.com/ConnectedHomes/awsfind.git
cd awsfindIf you don't currently usepoetryget it withpip[3] install poetry --userOptional: If you intend to develop this script install the dependenciespoetry installOptional: the script can be run from the development environment withpoetry run ifindInstall the script to the users local python installationpoetry build
vers=$(poetry version|sed 's/ /-/')
pip[3] install dist/${vers}*whl --userYou should now have a scriptifindin your python user directories$ which ifind
/home/chris/.local/bin/ifind

$ ifind -h
ifind 0.3.0
 ifind - AWS Instance Finder

Searches across all accounts and regions for instances using chaim credentials

search accounts in alphabetical order
    ifind <instance-id> <instance-id> ... <instance-id>

to search accounts in random order (maybe quicker)
    ifind -r <instance-id> <instance-id> ... <instance-id>You can now add your.local/bindirectory to your PATH if you haven't
already.UsageYou can run directly from this repository withpoetry run ifindor
install it as above$ ifind <instance-id> <instance-id> ... <instance-id>The above command will search through all accounts in alphabetical order,
searching all regions for the instance ids.If you want to search accounts in a random order (which maybe quicker)
then add-rto the command$ ifind -r i-0b7ff13d0219b8b58 i-014c4b3c01153aef8It displays it's current progress$ ifind i-0b7ff13d0219b8b58 i-014c4b3c01153aef8 i-09d8cfbb5fc425d26 i-0b42d2ae0db8cf231
ifind 0.3.0
Searching 119 accounts in 16 regions for 4 instances


  7/119 1/4    biqlite-qa-uki.e. looking in the 7th of 119 accounts for the last remaining instance
id, having found 3 others.Once it has found all the instances it stops, displaying the resultsAccount                       Region    Name                          Instance ID
------------------------------------------------------------------------------------------
biqlite-dev-uk                eu-west-1 UNNAMED                       i-0b7ff13d0219b8b58
biqlite-dev-uk                eu-west-1 UNNAMED                       i-014c4b3c01153aef8
biqlite-firmware              eu-west-1 simplicity                    i-09d8cfbb5fc425d26
biqlite-qa-uk                 eu-west-1 UNNAMED                       i-0b42d2ae0db8cf231

search took 59s
Package: aws-flask-lambda
aws_flask_lambdaInstallationInstall the package using pip:pipinstallaws-flask-lambdaUsagefromaws_flask_lambdaimportFlaskLambdafromflaskimportrequest,jsonifyapp=FlaskLambda(__name__)@app.route('/greet',methods=['GET','POST'])defgreet():name=request.form.get('name','World')message=f'Hello,{name}!'return(jsonify({'message':message}),200,{'Content-Type':'application/json'})if__name__=='__main__':app.run(debug=True)
Package: aws-flask-lambda-swagger-ui
aws_flask_lambda_swagger_uiInstallationInstall the package using pip:pipinstallaws-flask-lambda-swagger-uiUsagefromflaskimportFlaskfromaws_flask_lambda_swagger_uiimportget_swaggerui_blueprintapp=Flask(__name__)SWAGGER_URL='/aws/test/docs'API_URL='https://petstore.swagger.io/v2/swagger.json'swaggerui_blueprint=get_swaggerui_blueprint(SWAGGER_URL,API_URL,config={'app_name':"AWS Lambda application"})app.register_blueprint(swaggerui_blueprint,url_prefix=SWAGGER_URL)if__name__=='__main__':app.run(debug=True)
Package: aws-flask-swagger-ui
aws-flask-swagger-uiSimple Flask blueprint for addingSwagger UIto your flask application.
Including a WSGI adapter forAWS API Gateway/Lambda Proxy Integrationto allows you to use WSGI-compatible middleware and frameworks like Flask and Django with the AWS API Gateway/Lambda proxy integration for your Swagger documentation.Included Swagger UI version:see hereInstallationpip install aws-flask-swagger-uiUsageSimple usage example is shown below for more options check the fileextend examples:fromflaskimportFlaskfromaws_flask_swagger_uiimportget_swaggerui_blueprint,flask_ui_responseapp=Flask(__name__)swaggerui_blueprint=get_swaggerui_blueprint("/api-doc",aws_gw_config={"exportType":"oas30","parameters":{"extensions":"integrations","extensions":"apigateway","extensions":"authorizers",},},)app.register_blueprint(swaggerui_blueprint)deflambda_handler(event,context):returnflask_ui_response(app,event,context,base64_content_types={"image/png"})AWS Gateway Configurationhttp://mysite.com=https://restApiId.execute-api.region.amazonaws.com/stage/In order that the above example works correctly the Lambda function must be connected as Proxy to the endpointhttp://mysite.com/api-doc/Configure your API Gateway with a{proxy+}resource with anANYmethod. Your "Method Response" should likely include anapplication/json"Response Body for 200" that uses theEmptymodel.Because API Gateway doesn't match the root folder with {proxy+} definition, your default URL should contain index.html. It is suggested to create a mock integration on your path/api-docto return a 301. (ex:/api-doc => 301=>/api-doc/index.html) Source code based on Terraform to achieve this can be found in thisarticle.Lambda Test EventThe Lambda function must have the permissions to export the API definition!If you wish to use the "Test" functionality in Lambda for your function, you will need a "API Gateway AWS Proxy" event. Check the event JSON objects in theeventsfolder.To update your test event, click "Actions" -> "Configure test event".Within the Events you need to update the"apiId"and"stage"with values for your AWS account.Protect documentation with passwordIf you create an environment variable like, SWAGGER_PASSWORD=abcThen you will need to pass a query parameter in the URL like,http://mysite.com/api-doc/?pass=abcIf you don't have the environment variable then endpoint is not password protected and you can access it as per normalhttp://mysite.com/api-doc/ConfigurationThe blueprint supports overloading all Swagger UI configuration options that can be JSON serialized.
Seeswagger-ui configurationfor options.Plugins and function parameters are not supported at this time.OAuth2 parameters can be found atswagger-ui oauth2.LicenseThis library is licensed under theApache 2.0 License.TestClone the repo and run pytestgitclonehttps://github.com/tb102122/aws_flask_swagger_ui.git
python-mvenvvirtualenv
virtualenv/bin/activate
pipinstall--upgradepip,setuptools,wheel
pipinstallflake8pytestboto3pytest-cov
pipinstall.
flake8.
pytest
Package: awsflock
AWS flockflock-like functionality for applications in AWS, using dynamodb as a
backend for synchronization.A CLI tool as simple as the classicflockcommand.Use CasesSynchronizing multiple automated jobs across workers in AWSWriting scripts that only one human operator can be running at a time (e.g.
production deployment tools with terraform, CFN, etc)Synchronizing jobs in non-AWS systems (e.g. GitHub Actions) using AWS
credentialsUsageFull usage info can be found withawsflock--helpCreate the table in DynamoDB in order to start usingawsflock:awsflocktable-createAcquire a lock, with a 15 minute expiration, and get the lockLEASE_ID:LEASE_ID="$(awsflockacquireLockFoo--lease-duration'15 minutes')"Renew the lock, getting back the newLEASE_IDand reducing the expiration
window to 5 minutes, specified in seconds:LEASE_ID="$(awsflockrenewLockFoo"$LEASE_ID"--lease-duration'300s')"Release the lock, so that others may use it:awsflockreleaseLockFoo"$LEASE_ID"Attempt to acquire another lock, but don't block and wait for it to be
acquired:LEASE_ID="$(awsflockacquireLockBar--no-wait)"if[$?-eq0];then# lock acquired ...else# lock not acquired ...fiBehavior / ModelLocks are held for a limited period of time.
After that time, if not renewed, the lock expires and another worker may reclaim
the lock.
While the lock is held, the worker holding it has a "lease" on the lock, proven
by a givenLEASE_ID.
ALEASE_IDcan be used to "renew" or "release" a given lock.awsflockrequires a table in DynamoDB to store active locks.
By default, the table name isawsflock, but custom names can be used.
You must create the table before locks can be used (it will never be created
automatically).
Locks are identified by name, and those names are unique keys into theawsflocktable.Locks have a limited lifetime (default: 2 hours) if not explicitly released,
after which they may be "reclaimed" by anyone trying to acquire that lock.NOTE:Local clock time is compared against lock expirations to determine whether
or not reclamation may be tried. The default reclamation window (5 seconds)
is more than sufficient for most use-cases, but assumes that your clocks
are synchronized by NTP or a similar protocol. Usage where clocks cannot be
trusted may result in incorrect lock reclamations.When you acquire a lock, you get back aLEASE_ID. TheLEASE_IDcan then be
used to renew the lock or release it. In this way, locks are held by a single
owner for a limited period of time, and theLEASE_IDconstitutes proof of
ownership.CHANGELOG0.2.0More advanced parsing of durations0.1.1Minor fixup0.1.0Initial release
Package: awsfunctions
# AWS Functions[![CircleCI](https://circleci.com/gh/productml/awsfunctions/tree/master.svg?style=svg)](https://circleci.com/gh/productml/awsfunctions/tree/master)A collection of AWS functions based on [boto3](http://boto3.readthedocs.io/en/latest/).The goal of this library is to provide a higher level abstraction ofboto3for convenience.
Package: aws-get-secret
AWS Get SecretA library letting you easily retrieve a secret value from Aws Secrets ManagerRemarksThe project is written byDeividas Tamkus, supervised byLaimonas Sutkusand is owned byiDenfy. This is an open source
library intended to be used by anyone.iDenfyaims
to share its knowledge and educate market for better and more secure IT infrastructure.Related technologyThis project utilizes the following technology:AWS(Amazon Web Services).AWS SecretsManager.InstallThe project is built and uploaded to PyPi. Install it by using pip.pipinstallaws-get-secretOr directly install it through source../build.sh-icDescriptionThis project allows you to get secret values from Aws Secrets Manager by just knowing secret's ID.ExamplesTo retrieve a secret calledMyCoolSecretuse the following code snippet below.fromaws_get_secret.get_secretimportGetSecretsecret_getter=GetSecret()secret_value=secret_getter.MyCoolSecretprint(f'My super secret is:{secret_value}!')Release history1.0.1Fixed breaking bug.1.0.0Initial commit.
Package: aws-glue-artifact
Welcome toaws_glue_artifactDocumentationA lot of serverless AWS Service supports versioning and alias for deployment. It made the blue / green deployment, canary deployment and rolling back super easy.AWS Lambda Versioning and AliasAWS StepFunction Versioning and AliasAWS SageMaker Model Registry VersioningHowever, AWS Glue does not support this feature. This library provides a way to manage AWS Glue versioning and alias so you can deploy AWS Glue Jobs with confident.Pleaseread this tutorialto learn how to use this library.Installaws_glue_artifactis released on PyPI, so all you need is to:$pipinstallaws-glue-artifactTo upgrade to latest version:$pipinstall--upgradeaws-glue-artifact
Package: aws-glue-container-launcher
Welcome toaws_glue_container_launcherDocumentationAWS Big Data Blog - Develop and test AWS Glue version 3.0 and 4.0 jobs locally using a Docker containerintroduced a method using AWS maintained glue container to perform local development, unit testing, and interactive jupyter notebook on local laptop or in CI environment. I personally use this to accelerate AWS Glue ETL development, improve development experience and ETL logic visibility, and bring the quality of ETL code to the next level by adding lots of unit test and integration test.aws_glue_container_launcheris a zero-dependency, pure python library that can easily create shell script to perform tasks introduced in the AWS blog. This project is the core building block in my personal AWS Glue ETL project CI/CD best practice that is used to delivery high-quality ETL pipeline for my clients.DependenciesYou need to have docker installed on your local laptop or CI environment. You can test it by runningdocker--versionin your terminal.Python3.6 + to install this library and run the shell script.MacOS / Linux / Windows OS to run the shell script, but I only tested on MacOS and Linux.Usagerun glue job locally in container, see example atrun pytest locally in containerrun jupyter lab locally in containerREPL SHELL (TO DO)Visual Studio Code (TODO)Installaws_glue_container_launcheris released on PyPI, so all you need is to:$pipinstallaws-glue-container-launcherTo upgrade to latest version:$pipinstall--upgradeaws-glue-container-launcher
Package: awsglue-dev
aws-glue-libsThis repository contains libraries used in theAWS Glueservice. These libraries extendApache Sparkwith additional data types and operations for ETL workflows. They are used in code generated by the AWS Glue service and can be used in scripts submitted with Glue jobs.Contentawsglue-- This Python package includes the Python interfaces to the AWS Glue ETL library.LicensingThe libraries in this repository licensed under theAmazon Software License(the "License"). They may not be used except in compliance with the License, a copy of which is included here in the LICENSE file.