Package: idem-aws
AWS Cloud Provider for Idem.Aboutidem-awshelps manage AWS withidem.idem-aws source codeidem-aws documentationWhat is POP?This project is built withpop, a Python-based
implementation ofPlugin Oriented Programming (POP). POP seeks to bring
together concepts and wisdom from the history of computing in new ways to solve
modern computing problems.For more information:Intro to Plugin Oriented Programming (POP)pop-awesomepop-createWhat is Idem?This project is built withidem, an idempotent,
imperatively executed, declarative programming language written in Python. This project extends
idem!For more information:Idem Project WebsiteIdem Project docs portalGetting StartedPrerequisitesPython 3.8+git(if installing from source, or contributing to the project)IdemNoteIt is recommended that you install Idem using Poetry. Poetry is a tool for virtual environment and dependency management. See theIdem Getting Started guidefor more information.InstallationYou can installidem-awsfrom PyPI, a source repository, or a local directory.Before you installidem-aws, ensure that you are in the same directory as yourpyproject.tomlfile. Optionally, you can specify the directory containing yourpyproject.tomlfile by using the--directory=DIRECTORY(-C)option.Install from PyPITo installidem-awsfrom PyPI, run the following command:poetryaddidem-awsInstall from sourceYou can also installidem-awsdirectly from the source repository:poetryaddgit+https://gitlab.com/vmware/idem/idem-aws.gitIf you don’t specify a branch, Poetry uses the latest commit on themasterbranch.Install from a local directoryClone theidem-awsrepository. Then run the following command to install from the cloned directory:poetryadd~/path/to/idem-awsSetupAfter installation the AWS Idem Provider execution and state modules will be accessible to the pophub.
In order to use them we need to set up our credentials.Create a new file calledcredentials.yamland populate it with credentials.
If you are using localstack, then theidandkeycan be bogus values.
Thedefaultprofile will be picked up automatically byidem.There are many ways aws providers/profiles can be stored. Seeacct backendsfor more information.There are multiple authentication backends foridem-awswhich each have their own unique set of parameters.
The following examples show some of the parameters that can be used in these backends to define profiles.
All backends end up creating a boto3 session under the hood and storing it in thectxvariable that gets passed
to all idemexecandstatefunctions.All authentication backends support two optional parameters,endpoint_urlandprovider_tag_key.  Theendpoint urlis used to specify an alternate destination for boto3 calls, such as a localstack server or custom dynamodb server.
Theprovider_tag_keyis used when creating new resources.idem-awswill only interact with resources that are tagged
with the the customizableprovider_tag_keykey.credentials.yaml:aws:default:endpoint_url:http://localhost:4566use_ssl:Falseaws_access_key_id:localstackaws_secret_access_key:_region_name:us-west-1You can also useaws_session_tokenwith Idem for temporary security credentialsaws:default:endpoint_url:http://localhost:4566use_ssl:Falseaws_access_key_id:localstackaws_secret_access_key:_region_name:us-west-1aws_session_token:my_tokenAdditionally, you can use AWS AssumeRole with Idemaws:default:endpoint_url:http://localhost:4566use_ssl:Falseaws_access_key_id:localstackaws_secret_access_key:_region_name:us-west-1assume_role:role_arn:arn:aws:iam::999999999999999:role/xacct/developerrole_session_name:IdemSessionNameIfregion_nameis unspecified in the acct profile, it can come fromacct.extrasin the idem config file:# idem.cfgacct:extras:aws:region_name:us-west-1You can also authenticate withaws-google-authif it is installed.aws.gsuite:my-staging-env:username:user@gmail.compassword:this_is_available_but_avoid_itrole_arn:arn:aws:iam::999999999999999:role/xacct/developeridp_id:9999999sp_id:999999999999region:us-east-1duration:36000account:developerThe google profile example is not nameddefault. To use it, it will need to be specified explicitly in an idem state.ensure_resource_exists:aws.ec2.vpc.present:-acct_profile:my-staging-env-name:idem_aws_vpc-cidr_block:10.0.0.0/24It can also be specified from the command line when executing states.idemstate--acct-profilemy-staging-envmy_state.slsIt can also be specified from the command line when calling an exec module directly.idemexec--acct-profilemy-staging-envboto3.client.ec2.describe_vpcsThe last step to get up and running is to encrypt the credentials file and add the encryption key and encrypted file
path to the ENVIRONMENT.Theacctcommand should be available asacctis a requisite ofidemandidem-aws.
Encrypt the the credential file.acctencryptcredentials.yamloutput:-A9ZkiCSOjWYG_lbGmmkVh4jKLFDyOFH4e4S1HNtNwI=Add these to your environment:exportACCT_KEY="-A9ZkiCSOjWYG_lbGmmkVh4jKLFDyOFH4e4S1HNtNwI="exportACCT_FILE=$PWD/credentials.yaml.fernetIf no acct_file is supplied, then the default awscli credentials that are picked-up by botocore will be used.
Missing cli options will be filled in by botocore from awscli config.
The order of config priority is:acct profileidem config fileawscli configYou are ready to use idem-aws!Execution ModulesOnce everything has been set up properly, execution modules can be called directly byidem.
Execution modules mirror the namespacing of the boto3.client and boto3.resource modules and have the same parameters.For example, this is how you could list Vpcs from the command line with idem:idemexecboto3.client.ec2.describe_vpcsYou can specify parameters as well.
In the case of boto3 resources, args will be passed to the resource constructor and kwargs will be passed to the operation like so:idemexecboto3.resource.ec2.Vpc.create_subnetvpc-71d00419CidrBlock="10.0.0.0/24"StatesStates are also accessed by their relative location inidem-aws/idem_aws/states.
For example,idem-aws/idem_aws/states/aws/ec2/vpc.pycontains a functionabsent().
In my state file I can create a state that uses theabsentfunction like so.my_state.sls:idem_aws_test_vpc:aws.ec2.vpc.absent:-name:"idem_aws_test_vpc"This state can be executed with:idemstatemy_state.slsidem statealso has some flags that can significantly boost the scalability and performance of the run.
Let’s use this new state which verifies that 100 vpcs are absent:{%foriinrange(100)%}idem_aws_test_vpc_{{i}}:aws.ec2.vpc.absent:-name:"idem_aws_test_vpc_{{i}}"{%endfor-%}State can be executed with--runtimeparallelto make full use of idem’s async execution calls:idemstate--runtimeparallelmy_state.slsRemote storage for enforced state managementIdem-aws supports remote storage for Idem’s enforced state management feature. That is, Idem can
store esm data on AWS S3 bucket. DynamoDB will be used as a file lock to prevent multiple users/processes
access the same storage file concurrently. To use remote storage, the esm profile need to be added to
the credential profile like the following:aws:default:use_ssl:Trueaws_access_key_id:AAAAAAAAA5CDFSDER3UQaws_secret_access_key:eHjPASFWERSFwVXKlsdfS4afDregion_name:eu-west-2esm:bucket:"idem-state-storage-bucket"dynamodb_table:"idem-state-storage-table"key:"/idem-state/demo-storage.json"This esm file means that Idem will use AWS S3 bucket “idem-state-storage-bucket” and DynamoDB table
“idem-state-storage-table” in region eu-west-2. The “key” is the file path to which the esm data
will be read and stored. Both S3 bucket and DynamoDB table need to be created before using the feature.
The DynamoDB table should have the primary key as string “LockID” and nothing else.