Saravanan666/pypi-org huggingface dataset:
Package: awsglue-local
awsglueThe awsglue Python package contains the Python portion of theAWS Gluelibrary. This library extendsPySparkto support serverless ETL on AWS.Note that this package must be used in conjunction with the AWS Glue service and is not executable independently. Many of the classes and methods use the Py4J library to interface with code that is available on the Glue platform. This repository can be used as a reference and aid for writing Glue scripts.While scripts using this library can only be run on the AWS Glue service, it is possible to import this library locally. This may be helpful to provide auto-completion in an IDE, for instance. To import the library successfully you will need to install PySpark, which can be done using pip:pip install pysparkContentThis package contains Python interfaces to the key data structures and methods used in AWS Glue. The following are some important modules. More information can be found in the public documentation.GlueContextThe filecontext.pycontains the GlueContext class. GlueContext extends PySpark'sSQLContextclass to provide Glue-specific operations. Most Glue programs will start by instantiating a GlueContext and using it to construct a DynamicFrame.DynamicFrameThe DynamicFrame, defined indynamicframe.py, is the core data structure used in Glue scripts. DynamicFrames are similar to Spark SQL'sDataFramesin that they represent distributed collections of data records, but DynamicFrames provide more flexible handling of data sets with inconsistent schemas. By representing records in a self-describing way, they can be used without specifying a schema up front or requiring a costly schema inference step.DynamicFrames support many operations, but it is also possible to convert them to DataFrames using thetoDFmethod to make use of existing Spark SQL operations.TransformsThetransformsdirectory contains a variety of operations that can be performed on DynamicFrames. These include simple operations, such asDropFields, as well as more complex transformations likeRelationalize, which flattens a nested data set into a collection of tables that can be loaded into a Relational Database. Once imported, transforms can be invoked using the following syntax:TransformClass.apply(args...)Additional ResourcesTheaws-glue-samplesrepository contains sample scripts that make use of awsglue library and can be submitted directly to the AWS Glue service.The publicGlue Documentationcontains information about the AWS Glue service as well as addditional information about the Python library.
Package: awsglue-local-dev
awsglueThe awsglue Python package contains the Python portion of theAWS Gluelibrary. This library extendsPySparkto support serverless ETL on AWS.Note that this package must be used in conjunction with the AWS Glue service and is not executable independently. Many of the classes and methods use the Py4J library to interface with code that is available on the Glue platform. This repository can be used as a reference and aid for writing Glue scripts.While scripts using this library can only be run on the AWS Glue service, it is possible to import this library locally. This may be helpful to provide auto-completion in an IDE, for instance. To import the library successfully you will need to install PySpark, which can be done using pip:pip install pysparkContentThis package contains Python interfaces to the key data structures and methods used in AWS Glue. The following are some important modules. More information can be found in the public documentation.GlueContextThe filecontext.pycontains the GlueContext class. GlueContext extends PySpark'sSQLContextclass to provide Glue-specific operations. Most Glue programs will start by instantiating a GlueContext and using it to construct a DynamicFrame.DynamicFrameThe DynamicFrame, defined indynamicframe.py, is the core data structure used in Glue scripts. DynamicFrames are similar to Spark SQL'sDataFramesin that they represent distributed collections of data records, but DynamicFrames provide more flexible handling of data sets with inconsistent schemas. By representing records in a self-describing way, they can be used without specifying a schema up front or requiring a costly schema inference step.DynamicFrames support many operations, but it is also possible to convert them to DataFrames using thetoDFmethod to make use of existing Spark SQL operations.TransformsThetransformsdirectory contains a variety of operations that can be performed on DynamicFrames. These include simple operations, such asDropFields, as well as more complex transformations likeRelationalize, which flattens a nested data set into a collection of tables that can be loaded into a Relational Database. Once imported, transforms can be invoked using the following syntax:TransformClass.apply(args...)Additional ResourcesTheaws-glue-samplesrepository contains sample scripts that make use of awsglue library and can be submitted directly to the AWS Glue service.The publicGlue Documentationcontains information about the AWS Glue service as well as addditional information about the Python library.
Package: awsglue-questdb-writer
AWS Glue / PySpark QuestDB writerA simple writing utility for writing to QuestDB from PySpark / AWS Glue#Introduction
A very simple convenience library created due to difficulty in getting custom builds of the AWS Glue Libs for AWS for local development.
The standard release is Python 3.6 and requires some changes to be made in order to add in extra libraries.The InfluxDB writer is a potential alternative to this, but I didn't have much of a chance to get it working due
to dependencies and it not easily supporting PySpark.#InstallationInstall this via pippip3 install awsglue-questdb-writer#UsageIn your AWS Glue / PySpark Job include the file viafrom awsglue_questdb_writer import *Usage is by passing a DF to the function, this should ideally be a DF from a SparkSQL output like Glue creates (e.g. from the from_catalog) as that is what has been tested.Important to note:All Timestamps must be datetime objectsNanosecond precision (required by QuestDB) is currently only your timestamp precision with added zerosIf you need real nanosecond permission you must be on Python 3.7 and update the library to use it (See comments)QuestDB is whitespace sensitive, all datetimes are quoted but any other fields with whitespace will cause this to fail (silently)There is no socket response from this library (it is designed to be unmonitored and high throughput) - if errors are in your input it will fail silently (PR's welcome)There is a convenience line to drop unwanted fields prior to passing this into the function to write to QuestDBargs=getResolvedOptions(sys.argv,['TempDir','JOB_NAME','db_name','temp_workflow_bucket','questdb_host','questdb_port'])sc=SparkContext()glueContext=GlueContext(sc)spark=glueContext.spark_sessionjob=Job(glueContext)job.init(args['JOB_NAME'],args)allDaily=glueContext.create_dynamic_frame.from_catalog(database=args['db_name'],table_name="daily",transformation_ctx="allDaily",)df=allDaily.toDF()tdf=df.withColumn('reading_date_time',F.to_timestamp(df['reading_date_time'],'%Y-%m-%dT%H:%M:%S.%f'))tdf=tdf.drop(*["ingestion_date","period_start","period_end","quality_method","event","import_reactive_total","export_reactive_total"])write_to_quest(df=tdf,measurement="meter_id",table="daily",timestamp_field="reading_date_time",args=args)job.commit()#License
See LICENSE for full details
Package: awsglue-rclone
awsglue-rcloneawsglue-rclone is a wrapper for RClone designed for use with AWS Glue Python Shell, rclone everything within Glue!InstallationInstall awsglue-rclone from source:$gitclonehttps://github.com/zhiweio/awsglue-rclone.git&&cdawsglue-rclone/
$python3setup.pyinstallUsageInstallrcloneon local machine (Glue shell)fromawsglue_rcloneimportRClonerclone=RClone()Build AWS S3 remote configfromawsglue_rclone.configimportS3Configcn_s3_conf=S3Config(alias="cn",properties={"region":"cn-north-1","access_key_id":"xxx","secret_access_key":"xxx",},)Createrclone.configfilerclone.create_remote(cn_s3_conf)Build from Secrets Managersecrets:rclone/aws_s3/us_west{"region":"us-west-1","access_key_id":"xxx","secret_access_key":"xxx",}us_s3_conf=S3Config(alias="us",secret="rclone/aws_s3/us_west",)rclone.create_remote(us_s3_conf,append=True)List remote filesret=rclone.ls("cn:bucket/yourfolder"){"code":0,"out":"     2079 README.md\n","error":""}Print verboseret=rclone.ls("cn:bucket/yourfolder",debug=True){"code":0,"out":"     2079 README.md\n","error":"2024/02/08 17:00:09 DEBUG : rclone: Version\"v1.64.2\"starting with parameters [\"rclone\"\"--config=/tmp/opt/.config/rclone.conf\"\"-vv\"\"ls\"\"cn:bucket/yourfolder/\"]\n2024/02/08 17:00:09 DEBUG : Creating backend with remote\"cn:bucket/yourfolder/\"\n2024/02/08 17:00:09 DEBUG : Using config file from\"/tmp/opt/.config/rclone.conf\"\n2024/02/08 17:00:09 DEBUG : fs cache: renaming cache item\"cn:bucket/yourfolder/\"to be canonical\"cn:bucket/yourfolder\"\n2024/02/08 17:00:09 DEBUG : 7 go routines active\n"}Copy files between two remotes, setnio=Trueto save command output (stdout/stderr) into temporary filesrclone.copy("cn:bucket/yourfolder","us:bucket/yourfolder",nio=True){"code":0,"out":"/var/folders/bj/h5g9fnl93dg6tllvsp4hr73r0000gn/T/tmpuhom0swf.stdout","error":"/var/folders/bj/h5g9fnl93dg6tllvsp4hr73r0000gn/T/tmp4jf_9nok.stderr"}Build other remote Config providersfromawsglue_rclone.configimportConfig,RemoteTypes# provider for onedrive storageclassOneDriveConfig(Config):defcreate(self):token={"access_token":self.properties["access_token"],"token_type":self.properties.get("token_type","Bear"),"refresh_token":self.properties["refresh_token"],"expiry":"2018-08-26T22:39:52.486512262+08:00",}token=json.dumps(token)drive_id=self.properties["drive_id"]drive_type="business"returntextwrap.dedent(f"""[{self.alias}]type ={RemoteTypes.onedrive}token ={token}drive_id ={drive_id}drive_type ={drive_type}""")Authors@zhiweioLicenseGPL-3.0
Package: aws-glue-sessions
Glue Interactive Sessions Jupyter kernelGlue Interactive Sessions Jupyter kernel enables you to useAWS Glue Interactive Sessionsfrom your favorite IDEs. This kernel integrates almost anywhere Jupyter does including your favorite IDEs.
Documentation on how to install and use the kernel can also be foundhere.InstallationUse the following to install the kernel on your machine:The command,install-glue-kernels, installs the jupyter kernelspec for both pyspark and spark kernels.pip3 install --upgrade jupyter boto3 aws-glue-sessionsinstall-glue-kernelsMake sure your IAM role have permissions to run Gluehttps://docs.aws.amazon.com/glue/latest/dg/set-up-iam.htmlGetting StartedUsing the kernel via a local Jupyter NotebookTo start a local Jupyter Notebook, complete the following steps.Run the following command to launch Jupyter Notebook.jupyter notebookChoose New, and then choose one of the AWS Glue kernels to begin coding against AWS Glue.Using the kernel via other IDEsYou may also use this kernel via VSCode or other IDEs that support Jupyter kernels.
To set up the Glue Interactive Sessions kernel for Visual Studio Code followthese instructions.ChangeLogv1.0.4 (2023-12-01)Added support for inferring region from AWS_REGION environment variable.v1.0.3 (2023-11-15)Added support for handling special characters in owner tag.v1.0.2 (2023-10-16)There is no change in customer behavior with this kernel version.v1.0.1 (2023-10-13)There is no change in customer behavior with this kernel version.v1.0.0 (2023-09-05)Kernel defaults to Glue version4.0if a default Glue version wasn’t specified.New magics %session_type ,  %%assume_role, %%tags%list_sessions  now also displays session status and creation time.Support for additional visualizations through %matplotlib magicConfigurations can now be provided through magics, environment variables and AWS config files.Fix for VSCode startup issue
Package: aws-google-saml
Google SAML AuthThis is a utility to obtain temporary Amazon Web Services (AWS) Security Token Service (STS) credentials for use on the local Command Line Interface (CLI).This is an enhancement on the popularAWS Google Authapplication, which uses a requests library to authenticate to Google before authenticating to AWS via SAML.This application works similarly, however bypasses the need to authenticate into Google by using the user's existing Google web browser session to post the SAML assertion used for AWS authentication back to this application via local HTTP callback.Getting StartedThis project relies on Python (specifically, we've only tested onPython 3). Please first install Python3 using BrewbrewinstallpythonYou'll then need to configure profiles to use in your~/.aws/configfile. An example below:[profile profile-name]
region = ap-southeast-2
account = 123456789012
google_config.google_idp_id = ABCDE1234
google_config.role_name = production-engineer
google_config.google_sp_id = 000000000000Running the applicationReady? Start the app with the following commandpython3google-saml-auth.py--profileprofile-nameAdministrator Instructions// TODO: How to setup application in Google SAML Console
Package: awsh
awshAnother SSH utility for AWSInstallation$ pip install awshIf you never configured awscli before.$ aws configureAlternatively it can be downloaded onpypi.RequirementsPython 2 or 3What does it do?It gives you a list of hosts you have access to and preconfiguresshwith the proper key and user (assuming they have not been overridden by
a configuration management tool). You can filter by instance name and if
it matches only one instance it will log to it.Use CasesYou are running Elasticbeanstalk, EMR or something that creates it’s
own instance automatically.You are using different key-set for different instances and you can’t
remember which one does what.Instances are coming and going and you lost track of what is
available.Usageusage: awsh [-h] [--users USERS [USERS ...]] [--region REGION]
                   [-i KEY_PATH] [-c COMMAND] [-r REMOTE_HOST]
                   [-p REMOTE_PORT] [-l LOCAL_PORT] [--keys KEYS]
                   [--timeout TIMEOUT] [--console-output] [--version]
                   [filter]

SSH into AWS instances. The default user list assumes that your instances runs
on Ubuntu and or Amazon's AMIs. ex: "awsh --users user1 user2 --region us-
west-2 --keys '~/.keys' instance-name". Note that "awsh --users user1 user2
instance-name" will not be parsed properly due to the nature of nargs. In that
case you may want to do: "awsh instance-name --users user1 user2".

positional arguments:
  filter                Optional name filter. If only one instance is found,
                        it will connect to it directly.

optional arguments:
  -h, --help            show this help message and exit
  --users USERS [USERS ...]
                        Specify the users to try.
  --region REGION       Specify the aws region.
  -i KEY_PATH, --key-path KEY_PATH
                        Specific key path, overrides, --keys
  -c COMMAND, --command COMMAND
                        Translates to ssh -C
  -r REMOTE_HOST, --remote-host REMOTE_HOST
                        Open a tunnels. Translates to ssh -L <local-port
                        >:<remote-host>:<remote-port> <selected-aws-host>
  -p REMOTE_PORT, --remote-port REMOTE_PORT
                        Port to use on the remote host.
  -l LOCAL_PORT, --local-port LOCAL_PORT
                        Port to use on the local host. Get overwritten by
                        remote port if not defined.
  --keys KEYS           Directory of the private keys.
  --timeout TIMEOUT     SSH connection timeout.
  --console-output      Display the instance console out before logging in.
  --version             Returns awsh's version.
Package: aws-hashicorp-packer-reaper
AWS Hashicorp Packer ReaperHashicorp Packer is a great tool for building Amazon Machine Images. However, sometimes the virtual machine running packer
is not stopped. This utility stops or terminated all virtual machines with the name tag 'Packer Builder'You can use it as a command line utility or install it as an AWS Lambda function and stop the spend , NoOps style!Since version 1.0.9, the default name tag was removed. To make it work with the reaper add a run_tags block to the builder:source"amazon-ebs" "ubuntu"{name="my_ubuntu"run_tags={Name="Packer Builder"}install the packer reaperto install the packer reaper, type:pipinstallaws-hashicorp-packer-reapershow running packer instancesTo show running packer instances:$aws-hashicorp-packer-reaperlist

i-06ac951992dbc11a1(PackerBuilder)launched13minutesago-terminated
i-035ebe427a538c829(PackerBuilder)launched4minutesago-running
INFO:2packerbuilderinstancesfoundusage with multiple profilesIf you have multiple AWS Profiles (e.g. when working with AWS Organisations and SSO) you can specify the profile to use on the command line before executing theaws-hashicorp-packer-reaper.$AWS_PROFILE=not-my-default-profileaws-hashicorp-packer-reaperlist

i-0d72937e04c1123a9(PackerBuilder)launched31minutesago-running
i-0712d9bd593adb2fb(PackerBuilder)launched11minutesago-running
INFO:2packerbuilderinstancesfoundstop running packer instancesTo stop running packer instances older than 2 hours:$aws-hashicorp-packer-reaperstop--older-than2h

INFO:stoppingi-035ebe427a538c829(PackerBuilder)created2hoursago
INFO:totalof1runninginstancesstoppedterminate running packer instancesTo terminate stopped and running packer instances older than 24 hours:aws-hashicorp-packer-reaper--verboseterminate--older-than24h

INFO:terminatingi-035ebe427a538c829(PackerBuilder)created25hoursago
INFO:totalof1instancesterminateddeploy the packer reaperTo deploy the packer reaper as an AWS Lambda, type:gitclonehttps://github.com/binxio/aws-hashicorp-packer-reaper.gitcdaws-hashicorp-packer-reaper
awscloudformationdeploy\--capabilitiesCAPABILITY_IAM\--stack-nameaws-hashicorp-packer-reaper\--template-file./cloudformation/aws-hashicorp-packer-reaper.yamlThis will install the packer reaper in your AWS account and run every hour, stopping packer instances
launched more than 2 hours ago and terminate instances older than 24 hours. You can change the schedule
and the action to meet your requirements.read the blog too!https://binx.io/blog/2020/03/27/how-to-terminate-packer-instances-on-aws/
Package: aws-hcs-cli
aws-hcs-cli
Python script for CLI and SDK access to AWS via ADFS while requiring MFA access usinghttps://duo.com/History and Purpose
Harman used to use "the Legendary" aws-adfs CLI tool to login to our AWS accounts. It worked great, especially the DUO 2FA support. Eventually, I decided to write my own similar tool but make it Harman-specific so that we could tailor it to our needs. Since this tool will be used by Harman employees only I had that option. I then morphed it a little more for our use cases.DUO 2FA Requirements
In order for Duo 2FA to work properly Automatic Push needs to be enabled.Installation
Python 3.6+ is recommended as python2 is EOL January 2020.
It is highly recommended to use an application like Pipx to install and use python cli applications.
Follow the pipx installation documentation then simply run pipx install aws-hcs-cli
Experimental Binaries are available on the releases page. These are new and in testing Releases
See the installation options For additional options page for step by step instructions for installing in various environments
Package: awshelp
awshelpawshelp forwards arguments toawsunless-hor--helpare presentinstall + setuppipinstallawshelpaliasaws=awshelpmotivationI am sick and tired of:awscli 1.x$awss3--helpUnknown options: --helpawscli 2.x$awss3--helpusage: aws [-h] [--profile PROFILE] [--debug]optional arguments:-h, --help         show this help message and exit--profile PROFILE--debugin action$awss3cp--help|head-10awshelp: aws does not like --help, but I got you!CP()                                                                      CP()NAMEcp -DESCRIPTIONCopies a local file or S3 object to another location locally or in S3.it even works if you have arguments!$awss3cpfoobar--help|head-10awshelp: aws does not like --help, but I got you!CP()                                                                      CP()NAMEcp -DESCRIPTIONCopies a local file or S3 object to another location locally or in S3.
Package: awshelpers
This package represents a collection of “helper” modules to increase efficiency when working
with AWS’s Python SDK.Searching though readthedocs.io is fine however having to constantly refer back to function signatures while
composing Lambda’s is time consuming and repetitive.Theawshelperspackage attempts to address this issue in a variety of ways.Type HintingWhere ever possible and warranted proper docstrings have been added to expose data types within function
signatures. When coupled with an IDE like Pycharm importing these commented functions into your code will allow
for faster and more accurate composition.Predictable Naming ConventionsAWS has created an SDK that follows sensible naming conventions likeget_*,list_*,describe_*etc. Theawshelperspackage follows a similar style. Since keywords likeget,list,sendcan be shared across
modules it is suggested that the various modules you mayimportuseaskeyword when importing these modules
avoiding naming collisions.Below are two examples::>>> from awshelpers.ssm import get as ssm_get
>>> from awshelpers.s3 import get as s3_getEventA module for displaying Lambda event/payload data. Capable of…formatted dump of event dataparsing specific AWS Services (SNS) Message payloadsTo use simply do::>>> from awshelpers.event import dump as evt_dump
>>> d = evt_dump.payload(event)Simple Systems ManagerA module that interacts with EC2 instances that have the SSM agent installed.http://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.htmlTo use simply do::>>> from awshelpers.ssm import get as ssm_get, list as ssm_list, send as ssm_send
>>> command_result = ssm_send.command(['xx-xxxxxx'],'AWS-RunShellScript', 240, 'My SSM command',['ls -alh','ping -c 3 8.8.8.8'], 'us-west-2', 'ssm-command-bucket', 'service-name')S3A module that interacts with S3.To use simply do::>>> from awshelpers.s3 import get as s3_get
>>> s = s3_get.item('bucket-name','path/to/my/file.txt','/tmp/output.txt')EC2A module that interacts with EC2.To use simply do::>>> from awshelpers.ec2 import describe as ec2_describeElastic IPsList the elastic IPs associated to one or more instances::>>> ec2_describe.eip_addresses(['eni-xxxxxx','eni-yyyyyy'], 'PublicIp')TagsList the tags associated with an instance::>>> ec2_describe.tags([i-xxxxxxxxxxxxxxxxx])