Saravanan666/pypi-org huggingface dataset:
Package: awsdf
Submodulesawsdf.aws moduleThis module enables connecting to AWS and extracting metadata in
pandas dataframes.Installing from PyPI:pip install -U awsdfUSAGE:import awsdfaws_account = awsdf.Account(profile_name=”<PROFILE_NAME>”)glue_databases_df = aws_account.glue_get_databases()class awsdf.aws.Account(aws_access_key_id=None,
aws_secret_access_key=None, aws_session_token=None, region_name=None,
profile_name=None)Instantiate class object for connecting to AWS and retriving
metadata from AWS__init__(aws_access_key_id=None, aws_secret_access_key=None,
aws_session_token=None, region_name=None, profile_name=None)Provide access keys OR Profile name to connect to AWS account.
Keys take preceedenceParameters:aws_access_key_id (string) – AWS access key IDaws_secret_access_key (string) – AWS secret access keyaws_session_token (string) – AWS temporary session tokenregion_name (string) – AWS regionprofile_name (string) – AWS profile nameglue_get_jobs() -> DataFrameGet AWS Glue jobsReturns:dataframeglue_get_job_history(job_name, no_of_runs=1) -> DataFrameRetrieve glue job historyArguments:job_name – Name of job to retrive historyKeyword Arguments:no_of_runs – No of runs to retrive in descending order
(default: {1})Returns:dataframeglue_get_databases() -> DataFrameGet AWS Glue jobsReturns:dataframeglue_get_tables(dbname=None) -> DataFrameGet AWS Glue tablesKeyword Arguments:dbname – Database Name for which to retrieve tables (default:
{None})Returns:dataframeglue_get_fields(dbname, tablename) -> DataFrameGet AWS Glue table columnsKeyword Arguments:dbname – Database Name for table  tablename – Database Name
for which to retrieve columnsReturns:dataframeathena_data_dictionary(include_dbs: list = [], exclude_dbs: list
= []) -> DataFrameGet AWS Athean data dictionary. A data frame with all databases,
tables & fields with datatypesKeyword Arguments:include_dbs (optional) – list of databases to be included
exclude_dbs (optional) – list of databases to be excluded if
include_dbs list is empty.Returns:dataframequicksight_get_datasources() -> DataFrameGet QuickSight datasourcesKeyword Arguments:N/AReturns:dataframequicksight_get_datasets(includeDetails: bool = False) ->
DataFrameGet QuickSight datasetsKeyword Arguments:includeDetails (optional) – Include addition details i.e.
ConsumedSpiceCapacityInBytes & Owner. Default=FalseReturns:dataframequicksight_get_dataset_permissions(AwsAccountId: str, DataSetId:
str)Get QuickSight dataset permissionsKeyword Arguments:AwsAccountId – AWS account id DataSetId – Dataset idReturns:dataframequicksight_get_dataset_details(datasetId: str) -> dictGet QuickSight dataset detailsKeyword Arguments:DataSetId – Dataset idReturns:dataframequicksight_get_dashboards(includeDetails: bool = False) ->
DataFrameGet QuickSight dashboardsKeyword Arguments:includeDetails (optional) –NOT IMPLEMENTEDInclude
addition details. Default=FalseReturns:dataframequicksight_get_dashboard_details(dashboardId: str) -> dictGet QuickSight dashboard detailsKeyword Arguments:dashboardId – Dashboard idReturns:dictionaryModule contents
Package: awsdocs
awsdocsTable of contentsGeneral infoTechnologiesSetupFeaturesStatusGeneral infoWhenever I find myself struggling in the world of AWS services, I always go to the samepageand select the service in question. Since most of the time I am either writing terraform or python in the terminal when I need this page, I thought why not create a pip package that will open the docs in my default browser from the command-line. Thus, a simple project called awsdocs was born.Technologiespypi to bundle the project into a pip packagePython 3.7argparseSetuppip install awsdocsThen pass through the service you wish to know more about - eg:awsdocs s3FeaturesOpens aws documentation in your default browserSimple to useToDo:Since this code works on the URL path of the documentation, the format of the different services is not always intuitive.awsdocs ssmdoes not work, butawsdocs systems-managerdoes. Of course, I can use the choices option in argparse to make this explicit to the user, but I don't want to have to manually enter all the services. Need to think of an idea to get round that one.StatusIn Progress
Package: awsdsc
Awsdsc (abbreviation of AWS describe)Universal describe command for AWS resources.Installationpip install awsdscBasic UsageFirstly setup AWS credentials (usually by runningaws configure) before using.Then runawsdsc [--profile ${YOUR_PROFILE_NAME}].
You will be prompted to input type and query for AWS resources to describe.For more details, please runawsdsc --helpand check the help text.
Package: awsdumplogs
# DumplogsDumplogs is a command line tool (python) for dumping the aws cloudwatch log streams into local files.## MotivationOn AWS cloud, Users can get historical AWS cloudwatch logs by exporting the log group into S3 bucket. See [AWS logs S3 Export](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3Export.html)
The exported file are all in gz format and are placed in separate directory. one way to download them into loacl machine is to use “aws s3 cp s3://WholeBucket LocalFolder –recursive” , and then unzip all the gz files. This operation on AWS is inevitable fussiness.A more direct approach to dump the logs into local files is to use “aws logs” cli command. But aws-cli logs only allows max. 50 log streams to be retrieved at a time.
In order to dump the whole logs streams from a log group, user need record the ‘next_token’ every time by himself.Dumplogs is inspired by “aws logs” , it aims to dump (and filter) the whole log streams from a AWS log group to local text files.## Installation`bash pip3 installhttps://github.com/xinhuagu/dumplogs.git`or`bash python3-mpip installhttps://github.com/xinhuagu/dumplogs.git`## UsageMake sure the AWS_PROFILE is set properly`bash export AWS_PROFILE=my_profile ``bash usage: dumplogs[-h][--profileAWS_PROFILE][-oOUTPUT] group_name `-h –helphelp information-o –outputoutput directory, default: current directory–profileaws profile, default: AWS_PROFILE env vargroup_nameaws log group name## Example`bash dumplogs/aws/lambda/dpa-kafka-logs-otest--profileDEV `or`bash dumplogs/aws/lambda/dpa-kafka-logs`## ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.## License[Revised BSD License](https://github.com/xinhuagu/dumplogs/blob/master/LICENSE.txt)
Package: aws-dynamodb-parser
AWS DynamoDB ParserAWS DynamoDB utility for parsing DynamoDB responsesInstallationpip install aws-dynamodb-parserPackage Contentsparseparseis the main method exposed for parsing DynamoDB responses when usingboto3.Theparsefunction can handleall of the data types AWS DynamoDB supportsas of October 2020.ExamplesAssuming we have a tableTABLE_NAMEcontaining the following entries, withidset as the partition key andtimestampas the sort key.All three fields included are strings.idtimestampnamea77b5fc0-75cb-408c-bebf-863873506cce2020-10-01 13:13:47.388492First test entrya77b5fc0-75cb-408c-bebf-863873506cce2020-10-01 13:15:25.376589Second test entryfa853ff0-706e-45db-b6ae-aa6a8a1f78562020-10-01 13:16:47.720778Third test entryParsing the result from aget_itemrequest:importboto3fromaws_dynamodb_parserimportparsedynamodb_client=boto3.client("dynamodb")response=dynamodb_client.get_item(TableName="TABLE_NAME",Key={"id":{"S":"a77b5fc0-75cb-408c-bebf-863873506cce"},"timestamp":{"S":"2020-10-01 13:13:47.388492"}})item=response.get("Item",{})print(item)# {'id': {'S': 'a77b5fc0-75cb-408c-bebf-863873506cce'}, 'timestamp': {'S': '2020-10-01 13:13:47.388492'}, 'name': {'S': 'First test entry'}}entry=parse(item)print(entry)# {'id': 'a77b5fc0-75cb-408c-bebf-863873506cce', 'timestamp': '2020-10-01 13:13:47.388492', 'name': 'First test entry'}Parsing the result from aqueryrequest:importboto3fromaws_dynamodb_parserimportparsedynamodb_client=boto3.client("dynamodb")response=dynamodb_client.query(TableName="TABLE_NAME",KeyConditionExpression="#id = :id",ExpressionAttributeNames={"#id":"id"},ExpressionAttributeValues={":id":{"S":"a77b5fc0-75cb-408c-bebf-863873506cce"}})items=response.get("Items",[])print(items)# [{'id': {'S': 'a77b5fc0-75cb-408c-bebf-863873506cce'}, 'timestamp': {'S': '2020-10-01 13:13:47.388492'}, 'name': {'S': 'First test entry'}}, {'id': {'S': 'a77b5fc0-75cb-408c-bebf-863873506cce'}, 'timestamp': {'S': '2020-10-01 13:15:25.376589'}, 'name': {'S': 'Second test entry'}}]entries=parse(items)print(entries)# [{'id': 'a77b5fc0-75cb-408c-bebf-863873506cce', 'timestamp': '2020-10-01 13:13:47.388492, 'name': 'First test entry'}, {'id': 'a77b5fc0-75cb-408c-bebf-863873506cce', 'timestamp': '2020-10-01 13:15:25.376589', 'name': 'Second test entry'}]DocumentationUsers can get the docstring help by running:fromaws_dynamodb_parserimportparsehelp(parse)LinksGithubPyPITest PyPI
Package: aws-easy-mfa
aws_easy_mfaaws_easy_mfais a simple wrapper for AWS's boto3 that makes it much simpler to use with MFA.Simply wrap any function invoking a boto3 session with theaws_easy_mfadecorator and you're off!aws_easy_mfawill take care of all the tedius MFA login steps.InstallationYou can installaws_easy_mfavia pippipinstallaws-easy-mfaA simple exampleUsing boto3 to list bucket names in s3 with theaws_easy_mfadecorator.fromaws_easy_mfaimportaws_easy_mfa@aws_easy_mfa(mfa_device_arn='arn-of-the-mfa-device',mfa_token_code='code-from-token')deflist_s3_buckets(session=None):# Create an S3 client using the sessions3_client=session.client('s3')# List all S3 bucketsresponse=s3_client.list_buckets()# Extract the bucket names from the responsebuckets=response['Buckets']bucket_names=[bucket['Name']forbucketinbuckets]returnbucket_namesOptional argumentsYou can control your MFA timeout by includingduration_secondsas shown below@aws_easy_mfa(mfa_device_arn='arn-of-the-mfa-device',mfa_token_code='code-from-token',duration_seconds=3600)deflist_s3_buckets(session=None):...In addition, if you would like your MFA-authorized sesssion for further usage, you can add thereturn_session=Trueflag to your decorator, like this@aws_easy_mfa(mfa_device_arn='arn-of-the-mfa-device',mfa_token_code='code-from-token',duration_seconds=3600,return_session=True)deflist_s3_buckets(session=None):...The session is returned as a second argument.  So for example instead ofbucket_names=list_s3_buckets()enabling this flag returns your session asbucket_names,session=list_s3_buckets()Direct session retrievalYou can also create an MFA session without usage of the decorator as shown below:fromaws_easy_mfa.utilitiesimportget_mfa_credentials,create_boto3_session# get credentialscredentials=get_mfa_credentials(mfa_device_arn='arn-of-the-mfa-device',mfa_token_code='code-from-token',duration_seconds)# generate mfa-authorized sessionsession=create_boto3_session(credentials)
Package: awsec
AwSecWelcome to AwSec. You can find more extensive documentation over atreadthedocs.This is a small cli tool to make life a little easier for the overworked responder.Contributions are welcome. Just get in touch.QuickstartSimplypip install awsecand get going. The cli is available asawsecand
you can runawsec --helpto get up to speed on what you can do.DevelopmentThis project usespoetryfor dependency management andpre-commitfor local checks.
Package: aws-ec2-auto-scaling-instance-running-scheduler
AWS EC2 AutoScaling Instance Running SchedulerThis is an AWS CDK Construct to make EC2 AutoScaling instance running schedule (only running while working hours(start/stop)).
But only capacity min value is 0 for the AutoScalingGroup.ResourcesThis construct creating resource list.EventBridge Scheduler execution roleEventBridge SchedulerInstallTypeScriptnpminstallaws-ec2-auto-scaling-instance-running-scheduleroryarnaddaws-ec2-auto-scaling-instance-running-schedulerPythonpipinstallaws-ec2-auto-scaling-instance-running-schedulerExamplenpminstallaws-ec2-auto-scaling-instance-running-schedulerimport{Ec2AutoScalingInstanceRunningScheduler}from'aws-ec2-auto-scaling-instance-running-scheduler';newEc2AutoScalingInstanceRunningScheduler(stack,'Ec2AutoScalingInstanceRunningScheduler',{targets:[{groupName:'example-scaling-group',runningDesiredCapacity:2,startSchedule:{timezone:'UTC',},stopSchedule:{timezone:'UTC',},},],});LicenseThis project is licensed under the Apache-2.0 License.
Package: aws-ec2-instance-connect-custom-resource
AWS EC2 Instance Connect Custom ResourceAWS EC2 instance connect custom resourceResourcesThis construct creating resource list.AWS Custom ResourceLambda Function Iam RoleLambda FunctionCustom Resource Iam RoleCustom PolicyInstallTypeScriptnpminstallaws-ec2-instance-connect-custom-resourceoryarnaddaws-ec2-instance-connect-custom-resourcePythonpipinstallaws-ec2-instance-connect-custom-resourceExamplenpminstallaws-ec2-instance-connect-custom-resourceimport{EC2InstanceConnectCustomResource}from'aws-ec2-instance-connect-custom-resource';newEC2InstanceConnectCustomResource(stack,'EC2InstanceConnectCustomResource',{subnetId:'example-subnet-id',securityGroupIds:['example-security-group-id',],});LicenseThis project is licensed under the Apache-2.0 License.
Package: aws-ec2-provisioner
CLI tool to provision infrastructure and configure a simple web app in AWS.The infrastructure components will be name elb-project_name, asg-project_nameand so on. The tool will reuse any
pre-existing components based on the name. However, if there are several  components with the same name - one will be
chosen at randomPrerequisitesAWS AccountVPC with at least one public subnetAt least 9 available private IP addresses in the subnet (allows for 1 EC2 instance to be provisioned)AWS CLI needs to be configured for a user with necessary IAM permissions (TBD)LimitationsTo simplify the design all the infrastructure are being created in the single
subnet, and this subnet is required to be apublicone. Because the architecture doesn't include a NAT gateway
all the provisioned instanceswill be assigned a public IPaddress, so they can download the necessary packages.There isno input validationInstallationpip install aws-ec2-provisionerorgit clone git@github.com:valeramaniuk/aws-ec2-provisioner.git
cd aws-ec2-provisioner
pip install .Usagerunaws_ec2_provisionerwith the following options:--vpc-id <VPC where you want to deploy the app>
--subnet-id <Subnet where you want to deploy the app. Should in the selected VPC>
--project-name <Name of the project. You won't be able to create 2 projects with the same name>
--aws-profile <The name of access key/secret key pair (~/.aws/credentials)>
--min-asg-size <Minimum size of the autoscaling group>
--max-asg-size <Maximum size of the autoscaling group>
--instance-type <INT. Choose from a predefined list>
--scaling-target-value <INT. Scaling policy target value % >If you omit any of the options you will be prompted interactively. In this case, the tool will look up the configuredaws profiles, the VPCs available for the chosen profile/region and the subnets available for the chosen VPC.aws_ec2_provisioner --helpfor all available optionsAvailable instace types:1. t2.micro
2. t2.meduimInfrastructure to be created:ELBListener80Instance port80Health checkHTTP:80/Autoscaling groupScales in/out to maintain Average CPU utilization across all instances in the Autoscaling Group.Launch configuration associates public IP by defaultUser data in Launch configuration installs and startsngnixSecurity GroupsELB security groupingress tcp:80 from anywhereAPP security groupingress only fromELB security groupno SSH accessbecause the APP servers are in the public subnet (architecture limitation)TODO:Load based autoscalingAdditional regionChoose aws_profile interactively based on ~/.aws/credentialsChoose vpc-id interactively after specifying the aws-profile and regionChoose subnet-id interactively after specifying the subnetProper logging instead of print statementsTestsIdempotency (with limitations)Ability to rollbackMove instance user data to separate file, so theuser datafile may be specified at runtimeAdd bumpversionSupport for multiple subnets. Private/publicProvision a NAT gateway as necessaryInformative UIDestroy all provisioned infrastructureCatch all botocore exceptions
Package: aws-ec2-scheduler
# Uniplaces AWS SchedulerThe AWS Scheduler is a script that given a set of profiles schedule the uptime of ec2 instances.## Getting StartedInstall through pip.pip install aws-ec2-schedulerThe initial configuration is done in the `aws.conf` file. Copy the `aws.conf.dist`to `aws.conf` and configure with the right credentials.By default `aws.conf` reads schedule profiles from `scheduler.json` but you can changethe directory from where you want to load the profiles.To run the scheduler just runec2scheduler -c path/to/config_file#### ProfilesThe scheduler expects you to create profiles to start and stop the machines.You can just copy the `scheduler.json.dist` and start creating your new profiles from there.```{"profiles": [ // an array of profiles{"name": "staging", // profile name"region": "us-west-1", // aws region"schedule": { // weekday schedule"monday": {"start": 9, // start hour"stop": 19 // stop hour},"tuesday": {"start": 9,"stop": 19},"wednesday": {"start": 9,"stop": 19},"thursday": {"start": 9,"stop": 19},"friday": {"start": 9,"stop": 19},"saturday": {"start": 9,"stop": 19},"sunday": {"start": 9,"stop": 19}},"instance_tags": [ // tags from the instances you want to attach this profile schedule"jobs-staging", // this tag should be the name of the instance tag:Name"admin-staging","reports-staging"],"elb_names": [ // add this configuration if you want to reregister this elb instances"example-tag-elb-name"]}]}```
Package: aws-ecr-cleanup
Image Cleanup for Amazon ECRThe script removes stale ECR images. Although ECR offers lifecycle policies that serve a similar task, the logic this script implements cannot be achieved by using them.The intention is to protect the set of images, that meet the criteria :comes from a specified period, counting from the oldestsignificant tag,counts no less than N-images, counting from the oldest significant tag (if less than N comes from the specified period).Significant tagsare those that are somehow meaningful for the release process, ie. 'dev', 'rc', 'prod', 'stable' and so on.As the production/stable images are usually older than their development counterparts, using the oldest tag date as an anchor to calculate the protected period, means protecting some set of images that directly precede the current production release.ExampleThe diagram below explains how script works, assuming:protected period = one year,protected count = 13,significant tags = dev, rc, prod,ecr repositories limited to: hello_world, other_ecr_repo.By default, it runs in dry mode. To print what images will be removed, run:python3 main.py --config <config_file.yaml>with config file content:- name: Flush untagged images older than 7 days, leave no less than 7 images
  protected_period: 7
  protected_count: 7
  significant_tags: '*'
  protected_repositories:
    - some_ignored_repository
- name: Protect images are that created for a year since a last significant tag, leave no less than 13 images
  protected_period: 365
  protected_count: 13
  significant_tags:
    - dev
    - rc
    - prod
    - latest
  repositories:
    - hello_worldWhen no repository list is provided, the script will iterate through all of the ECR repositories that are accessible by the AWS account used.
To skip some repositories useprotected_repositoriesoption.To actually remove the images, run:python3 main.py --config <config_file.yaml> --apply
Package: aws-ecr-migration
AWS ECR MigrationShort descriptionProject used to migrate docker images between cloud and a local machine.PrerequisitesLocal prerequisitesDocker installed.This project installed with:pipinstallaws-ecr-migrationor:./install.shCloud prerequisitesECR repository createdUsagePulling from ECR repositoryfromaws_ecr_migration.manageimportManagerfromaws_ecr_migration.aws_credentialsimportAwsCredentialsmanager=Manager(credentials=AwsCredentials(),remote_repository='remote/repository')manager.pull()Pushing an image to ECR repositoryfromaws_ecr_migration.manageimportManagerfromaws_ecr_migration.aws_credentialsimportAwsCredentialsmanager=Manager(credentials=AwsCredentials(),remote_repository='remote/repository')manager.push_image('myimage')Pushing a running container to ECR repositoryfromaws_ecr_migration.manageimportManagerfromaws_ecr_migration.aws_credentialsimportAwsCredentialsmanager=Manager(credentials=AwsCredentials(),remote_repository='remote/repository')manager.push_container('mycontainer')Release history1.2.1Documentation updates.1.2.0Add ability to pull specific image.1.1.0Add repository_exists and repository_empty methods.1.0.0Initial.
Package: awsecs
AWSCONConnect to SSM console of any deployed instance.PrereqsSetup your profiles using the aws cliawsconfigure--profilePROFILE_NAMEYou can also leave out the--profile PROFILE_NAMEparam to set yourdefaultcredentialsRefer to this doc for more informationhttps://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.htmlThis code now has an external dependency on fzf please install it using your preferred package manager.Setuppippip install awsconsourcegit clone git@github.com:sergiopena/awscon.git
cd awscon
python setup.py installTestsNo test at the moment... SHAME!How to use?An exported env var named AWS_PROFILE is required, as this is the profile that will be used to retrieve running ec2 instancesProfile must include the region, currently we only support one region.Runawsconit will retrieve your EC2 running instances and display a menu to connect to them.ContributeContributions are more than welcomed!Distributepython setup.py sdist bdist_wheel
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
twine upload dist/*CreditsInspired by the AWS profile switcherawspLicenseBSD
Package: aws-ecs-cluster
AWS ECS ClusterA custom ECS Cluster resource with an ability to completely
delete itself (even if it contains active tasks within).RemarksThe project is written byLaimonas Sutkusand is owned byiDenfy. This is an open source
library intended to be used by anyone.iDenfyaims
to share its knowledge and educate market for better and more secure IT infrastructure.Release history1.3.0Add md files.1.2.0Dependency update 1.60.0 - 2.0.0.1.1.0Force CDK dependency version update to 1.44.0.1.0.1Add permissions to put logs to log group.1.0.0Initial project. Not tested. No extensive readme.
Package: aws-ecs-deploy
Cli written in Python to make Deployment easierIn order to use this. You need the latest version of python installed.
After that follow the steps below.move to theclidirectory (the parent directory of this file)Create python virtual environment withpython3 -m venv envActive virtual env withsource env/bin/activateInstall python dependencies withpython3 -m pip install -r requirements.txtInstall the cli in virtual environment withpip install -e .BasicsView available commands withdeploy --helpView options of individual command withdeploy COMMAND --helpCreating Stackdeploy create-stack -n STACK_NAMEEx:deploy create-stack -n resourcesCreating multiple stack at onceDeploy core withdeploy create-stacks -cDeploy all services according to config yml files
withdeploy create-stack -s
Package: aws-ecs-exec
aws-ecs-execA cli tool to conveniently execute commands in AWS ECS Tasks.RequirementsPython 3.10+AWS CLIThe tool assumes that theawscommand is installed on your path.InstallationInstall usingpippipinstallaws-ecs-execUsageQuickstart:$ecs-execFor help:$ecs-exec--helpKnown issuesThis has been developed and only tested on MacOS.No pagination is used with boto3 calls, so there will likely be errors
if page limits are exceeded.Does not support tasks with multiple containers.