Package: terraformpy
TerrafompyTerraformpy is a library and command line tool to supercharge your Terraform configs using a full fledged Python environment!Terraformis an amazing tool.  Like, really amazing.  When working with code that is managing third-party service definitions, and actually applying changes to those definitions by invoking APIs, a high-degree of confidence in the change process is a must-have, and that’s where Terraform excels.  The work flow it empowers allow teams to quickly make changes across a large (and ever growing) footprint in multiple providers/regions/technologies/etc.But as your definitions grow theHCLsyntax very quickly leaves a lot to be desired, and is it ever verbose… So many definitions of variables and outputs need to be repeated, over and over, as you compose more modules that use each other.SinceHCLis “fully JSON compatible” and Python is great at generating JSON data, we built Terraformpy to provide a more productive environment to build and maintain complex Terraform configs.  It has been used daily in production atNerdWalletsince 2016 and has proven very valuable in accelerating our adoption of Terraform across our engineering organization.Installing TerraformpyThe recommended way to install and use Terraformpy is viaPipenvAn example would look like:$mkdirmy-terraform-project$cdmy-terraform-project$pipenvinstallterraformpyYou can then run Terraformpy usingpipenv run:$pipenvrunterraformpy...Or you can usepipenv shellto activate the virtualenv so you don’t need to usepipenv run.  The rest of this document assumes that you’ve runpipenv shelland can just runterraformpydirectly.Using the CLI toolTheterraformpycommand line tool operates as a shim for the underlyingterraformtool.  When invoked it will first find all*.tf.pyfiles in the current directory, loading them using theimpmodule, generate a file namedmain.tf.json, and then invoke underlying tool.# just replace terraform in your regular workflowterraformpyplan-out=tf.plan# review changes...# apply them!
# since we're going to operate on the generated plan here, we don't event need to use terraformpy anymoreterraformapplytf.planEach of the*.tf.pyfiles uses a declarative syntax, using objects imported from this library.  You don’t need to define a main function, you just create instances of classes (anonymous or otherwise) in the root of the module (you’re building regular Python code here).  Since you’re in a full blown Python environment there is no limit on what you can do – import things, connect to databases, etc.Writing.tf.pyfilesTheterraformpyname space provides a number of classes that map directly to things you declare in normal.tf.files.  To write your definitions simply import these classes and begin creating instances of them.  Below is the first example from theTerraform getting start guide.fromterraformpyimportProvider,ResourceProvider('aws',profile='default',region='us-east-1')Resource('aws_instance','example',ami='ami-2757f631'instance_type='t2.micro')Things you can import fromterraformpy:Resource-https://www.terraform.io/docs/configuration/resources.htmlProvider-https://www.terraform.io/docs/configuration/providers.htmlVariable-https://www.terraform.io/docs/configuration/variables.htmlOutput-https://www.terraform.io/docs/configuration/outputs.htmlModule-https://www.terraform.io/docs/configuration/modules.htmlData-https://www.terraform.io/docs/configuration/data-sources.htmlTerraform-https://www.terraform.io/docs/configuration/terraform.htmlSee theexamples/dir for fully functional examples.InterpolationSo far, we’ve only used terraformpy anonymously, but the returned instances of theDataandResourceclasses offer handy interpolation attributes.  For example, a common task is using theDataclass to fetch remote data:ami=Data('aws_ami','ecs_ami',most_recent=True,filter=[dict(name='name',values=['\*amazon-ecs-optimized']),dict(name='owner-alias',values=['amazon'])])Resource('aws_instance','example',ami=ami.id,instance_type='m4.xlarge')Here we simply refer to the id attribute on the ami object when creating theaws_instance.  During the compile phase it would be converted to the correct syntax:"${data.aws_ami.ecs_ami.id}".This works by having a custom__getattr__function on ourDataandResourceobjects that will turn any attribute access for an attribute name that doesn’t exist into the Terraform interpolation syntax.BackendConfiguring a backend happens in theTerraformobject. SeeConfiguring a Terraform Backendfor more details.Bellow we are using an S3 Backend:Terraform(backend=dict(s3=dict(region="us-east-1",bucket="terraform-tfstate-bucket",key="terraform.tfstate",workspace_key_prefix="my_prefix",dynamodb_table="terraform_locks",)))ModulesSince Terraformpy gives you the full power of Python we encourage you to use “Resource Collections” (see the next section) when you’re building your own modular functionality and you don’t plan on sharing these modules outside of your current organization.You can however leverage existing HCL modules using theModuleobject if you want to use pre-built, existing modules:Module("consul",source="hashicorp/consul/aws",version="0.0.5",servers=3)Resource CollectionsA common pattern when building configs using Python is to want to abstract a number of different resources under the guise of a single object – which is the same pattern native Terraform modules aim to solve.  In terraformpy we provide aResourceCollectionbase class for building objects that represent multiple resources.You can useSchematicsto define the fields and perform validation.As an example, when provisioning an RDS cluster you may want to have a standard set of options that you ship with all your clusters.  You can express that with a resource collection:fromschematicsimporttypesfromschematics.typesimportcompoundfromterraformpyimportResource,ResourceCollectionclassRDSCluster(ResourceCollection):# Defining attributes of your resource collection is like defining a Schematics Model, in fact the# ResourceCollection class is just a specialized subclass of the Schematics Model class.## Each attribute becomes a field on the collection, and can be provided as a keyword when constructing# an instance of your collection.## Validation works the same as in Schematics.  You can attach validators to the fields themselves and# also define "validate_field" functions.name=types.StringType(required=True)azs=compound.ListType(types.StringType,required=True)instance_class=types.StringType(required=True,choices=('db.r3.large',...))# The create_resources function is invoked once the instance has been created and the kwargs provided have been# processed against the inputs.  All of the instance attributes have been converted to the values provided, so# if you access self.name in create_resources you're accessing whatever value was provided to the instancedefcreate_resources(self):self.param_group=Resource('aws_rds_cluster_parameter_group','{0}_pg'.format(self.name),family='aurora5.6',parameter=[{'name':'character_set_server','value':'utf8'},{'name':'character_set_client','value':'utf8'}])self.cluster=Resource('aws_rds_cluster',self.name,cluster_identifier=self.name,availability_zones=self.azs,database_name=self.name,master_username='root',master_password='password',db_cluster_parameter_group_name=self.param_group.id)self.instances=Resource('aws_rds_cluster_instance','{0}_instances'.format(self.name),count=2,identifier='{0}-${{count.index}}'.format(self.name),cluster_identifier=self.cluster.id,instance_class=self.instance_class)That definition can then be imported and used in your terraformpy configs.frommodules.rdsimportRDSClustercluster1=RDSCluster(name='cluster1',azs=['us-west-2a','us-west-2b','us-west-2c'],instance_class='db.r3.large')# you can then refer to the resources themselves, for interpolation, through the attrs# i.e. cluster1.cluster.idVariantsResource definitions that exist across many different environments often only vary slightly between each environment. To facilitate the ease of definition for these differences you can use variant grouping.First create the folders:configs/stage/,configs/prod/,configs/shared/.  Inside each of them place a__init__.pyto make them packages.Next create the fileconfigs/shared/instances.py:fromterraformpyimportResourceResource('aws_instance','example',ami=ami.id,prod_variant=dict(instance_type='m4.xlarge'),stage_variant=dict(instance_type='t2.medium'))Then createconfigs/stage/main.tf.py:fromterraformpyimportVariantwithVariant('stage'):importconfigs.shared.instancesSince the import of the instances file happens inside of the Variant context then the Resource will be created as if it had been defined like:fromterraformpyimportResourceResource('aws_instance','example',ami=ami.id,instance_type='t2.medium')Multiple providersDepending on your usage of Terraform you will likely end up needing to use multiple providers at some point in time. To usemultiple providers in Terraformyou define them using aliases and then reference those aliases in your resource definitions.To make this pattern easier you can use the TerraformpyProviderobject as a context manager, and then any resources created within the context will automatically have that provider aliases referenced:fromterraformpyimportResource,ProviderwithProvider("aws",region="us-west-2",alias="west2"):sg=Resource('aws_security_group','sg',ingress=['foo'])assertsg.provider=='aws.west2'Using file contentsOften times you will want to include the contents of a file that is located alongside your Python code, but when runningterraformalong with the${file('myfile.json')}interpolation function pathing will be relative to where the compiledmain.tf.jsonfile is and not where the Python code lives.To help with this situation a function namedrelative_fileinside of theterraformpy.helpersnamespace is provided.fromterraformpyimportResourcefromterraformpy.helpersimportrelative_fileResource('aws_iam_role','role_name',name='role-name',assume_role_policy=relative_file('role_policy.json'))This would produce a definition that leverages the${file(...)}interpolation function with a path that reads therole_policy.jsonfile from the same directory as the Python code that defined the role.HooksTerraformy offers a “hooks” system that allows you to modify objects on the fly at compile
time.  This can be useful to apply transformations so that users of objects do not need
to worry about some of the idiosyncratic schema details of Terraform’s JSON syntax,
most notably“Attributes as Blocks”.The best example of this is theaws_security_groupobject type, which requires that itsingressandegressblocks have all of their attributes present, even if they arenull.  To have users have to type out all of the different attributes and set them toNoneis cumbersome, so instead you can use the hook that we ship with this distribution:fromterraformpy.hooks.awsimportinstall_aws_security_group_attributes_as_blocks_hookinstall_aws_security_group_attributes_as_blocks_hook()Now, users only need to specify the attributes they care about in rules and the hook will
take care of filling in all of the optional attributes that are mandatory to appear in the
final compiled JSON.For more information on how the hooks work see thetest_hooks_aws.pyfile and the
inline comments for theadd_hookfunction on the different object types.Notes and GotchasSecurity Group Rules andselfWhen creatingaws_security_group_ruleResourceobjects you cannot passself=Trueto the object since Python already passes aselfargument into the constructor.  In this case you’ll need to specify it directly in the_values:sg=Resource('aws_security_group_rule','my_rule',_values=dict(self=True),vpc_id=vpc.id,...)Developer notesRunning testsWe use tox to run tests.  While developing locally you can run:toxFormatting with blackWe use black to format code.  To apply formatting run:tox -e black -- .Release StepsMake a branchMake your changesBump the version in the VERSION file and add an entry to the CHANGELOG.md fileOpen a PR, tag @NerdWalletOSS/dynamorm in your PR descriptionOnce approved and merged to master the new version will be pushed to pypi