Package: claranet-tfwrapper
claranet-tfwrappertfwrapperis a python wrapper forOpenTofuand legacyTerraformwhich aims to simplify their usage and enforce best practices.Note: the termTerraformis used in this documentation when talking about generic concepts like providers, modules, stacks and the HCL based domain specific language.Table Of Contentsclaranet-tfwrapperTable Of ContentsFeaturesSetup DependenciesRuntime DependenciesRecommended setupInstallationSetup command-line completionUpgrade from tfwrapper v7 or olderRequired filesConfigurationtfwrapper configurationStacks configurationsStates centralization configurationHow to migrate from one backend to another for state centralizationStacks file structureUsageStack bootstrapWorking on stacksPassing optionsEnvironmentS3 state backend credentialsAzure Service Principal credentialsAzure authentication isolationGCP configurationGKE configurationsStack configurations and credentialsStack pathDevelopmentTestsDebug command-line completionPython code formattingChecksREADME TOCUsing OpenTofu development buildsgit pre-commit hooksReview and merge open Dependabot PRsTagging and publishing new releases to PyPIFeaturesOpenTofu and Terraform behaviour overridingState centralization enforcementStandardized file structureStack initialization from templatesAWS credentials cachingAzure credentials loading (both Service Principal or User)GCP and GKE user ADC supportPlugins cachingTab completionSetup Dependenciespython3>= 3.8.1 <4.0python3-pippython3-venvpipx(recommended)Runtime Dependenciesterraform>= 0.10(>= 0.15for fully working Azure backend with isolation due tohttps://github.com/hashicorp/terraform/issues/25416)azure-cliwhen using context based Azure authenticationRecommended setupOpenTofu 1.6+ (recommended) or Terraform 1.0+ (warning: versions above 1.6 are not open-source, and may cause legal issues depending on the context you are using it).An AWS S3 bucket and DynamoDB table for state centralization in AWS.An Azure Blob Storage container for state centralization in Azure.Installationtfwrapper should installed using pipx (recommended) or pip:pipxinstallclaranet-tfwrapperSetup command-line completionAdd the following to your shell's interactive configuration file, e.g..bashrcfor bash:eval"$(register-python-argcompletetfwrapper-etfwrapper)"You can then press the completion key (usuallyTab ↹) twice to get your partially typedtfwrappercommands completed.Note: the-e tfwrapperparameter adds an suffix to the defined_python_argcompletefunction to avoid clashes with other packages (seehttps://github.com/kislyuk/argcomplete/issues/310#issuecomment-697168326for context).Upgrade from tfwrapper v7 or olderIf you used versions of the wrapper older than v8, there is not much to do when upgrading to v8
except a little cleanup.
Indeed, the wrapper is no longer installed as a git submodule of your project like it used to be instructed and there is no longer anyMakefileto activate it.Just clean up each project by destroying the.wrappersubmodule:gitrm-fMakefile
gitsubmoduledeinit.wrapper
rm-rf.git/modules/.wrapper
gitrm-f.wrapperThen check the staged changes and commit them.Required filestfwrapper expects multiple files and directories at the root of a project.confStacks configurations are stored in theconfdirectory.templatesThetemplatesdirectory is used to store the state backend configuration template and the Terraform stack templates used to initialize new stacks. Using a git submodule is recommended.The following files are required:templates/{provider}/common/state.tf.jinja2: AWS S3 or Azure Blob Storage state backend configuration template.templates/{provider}/basic/main.tf: the default Terraform configuration for new stacks. The wholetemplate/{provider}/basicdirectory is copied on stack initialization.For example with AWS:mkdir-ptemplates/aws/commontemplates/aws/basic# create state configuration template with AWS backendcat<< 'EOF' > templates/aws/common/state.tf.jinja2{% if region is not none %}{% set region = '/' + region + '/' %}{% else %}{% set region = '/' %}{% endif %}terraform {backend "s3" {bucket = "my-centralized-terraform-states-bucket"key    = "{{ client_name }}/{{ account }}/{{ environment }}{{ region }}{{ stack }}/terraform.state"region = "eu-west-1"dynamodb_table = "my-terraform-states-lock-table"}}resource "null_resource" "state-test" {}EOF# create a default stack templates with support for AWS assume rolecat<< 'EOF' > templates/aws/basic/main.tfprovider "aws" {region     = var.aws_regionaccess_key = var.aws_access_keysecret_key = var.aws_secret_keytoken      = var.aws_token}EOFFor example with Azure:mkdir-ptemplates/azure/commontemplates/azure/basic# create state configuration template with Azure backendcat<< 'EOF' > templates/azure/common/state.tf.jinja2{% if region is not none %}{% set region = '/' + region + '/' %}{% else %}{% set region = '/' %}{% endif %}terraform {backend "azurerm" {subscription_id      = "00000000-0000-0000-0000-000000000000"resource_group_name  = "my-resource-group"storage_account_name = "my-centralized-terraform-states-account"container_name       = "terraform-states"key = "{{ client_name }}/{{ account }}/{{ environment }}{{ region }}{{ stack }}/terraform.state"}}EOF# create a default stack templates with support for Azure credentialscat<< 'EOF' > templates/azure/basic/main.tfprovider "azurerm" {subscription_id = var.azure_subscription_idtenant_id       = var.azure_tenant_id}EOF.runThe.rundirectory is used for credentials caching and plan storage.mkdir.run
cat<< 'EOF' > .run/.gitignore*!.gitignoreEOF.gitignoreAdding the following.gitignoreat the root of your project is recommended:cat<< 'EOF' > .gitignore.terraformterraform.tfstateterraform.tfstate.backupterraform.tfvarsEOFConfigurationtfwrapper uses yaml files stored in theconfdirectory of the project.tfwrapper configurationtfwrapper uses some default behaviors that can be overridden or modified via aconfig.ymlfile in theconfdirectory.---always_trigger_init:False# Always trigger `terraform init` first when launching `plan` or `apply` commandspipe_plan_command:"cat"# Default command used when you're invoking tfwrapper with `--pipe-plan`use_local_azure_session_directory:False# Use the current user's Azure configuration in `~/.azure`. By default, the wrapper uses a local `azure-cli` session and configuration in the local `.run` directory.Stacks configurationsStacks configuration files use the following naming convention:conf/${account}_${environment}_${region}_${stack}.ymlHere is an example for an AWS stack configuration:---state_configuration_name:"aws"# use "aws" backend state configurationaws:general:account:&aws_account"xxxxxxxxxxx"# aws account for this stackregion:&aws_regioneu-west-1# aws region for this stackcredentials:profile:my-aws-profile# should be configured in .aws/configterraform:legacy:false# Use legacy version of the tool instead of OpenTofu, defaults to false. Only used if version >= 1.6.0.version:"1.0"# OpenTofu version that tfwrapper will use for this stack. Automatically downloaded if it's not available locally. Defaults to 1.0vars:# variables passed to terraformaws_account:*aws_accountaws_region:*aws_regionclient_name:my-client-name# arbitrary client nameHere is an example for a stack on Azure configuration using user mode and AWS S3 backend for state storage:---state_configuration_name:"aws-demo"# use "aws" backend state configurationazure:general:mode:user# Uses personal credentials with MFAdirectory_id:&directory_id"00000000-0000-0000-0000-000000000000"# Azure Tenant/Directory UIDsubscription_id:&subscription_id"11111111-1111-1111-1111-111111111111"# Azure Subscription UIDterraform:legacy:false# Use legacy version of the tool instead of OpenTofu, defaults to false. Only used if version >= 1.6.0.version:"1.0"# OpenTofu version that tfwrapper will use for this stack. Automatically downloaded if it's not available locally. Defaults to 1.0vars:subscription_id:*subscription_iddirectory_id:*directory_idclient_name:client-name#Replace it with the name of your clientIt is using your account linked to a Microsoft Account. You must have access to the Azure Subscription if you want to use Terraform.Here is an example for a stack on Azure configuration using Service Principal mode:---azure:general:mode:service_principal# Uses an Azure tenant Service Principal accountdirectory_id:&directory_id"00000000-0000-0000-0000-000000000000"# Azure Tenant/Directory UIDsubscription_id:&subscription_id"11111111-1111-1111-1111-111111111111"# Azure Subscription UIDcredentials:profile:customer-profile# To stay coherent, create an AzureRM profile with the same name as the account-alias. Please checkout `azurerm_config.yml.sample` file for configuration structure.terraform:legacy:false# Use legacy version of the tool instead of OpenTofu, defaults to false. Only used if version >= 1.6.0.version:"1.0"# OpenTofu version that tfwrapper will use for this stack. Automatically downloaded if it's not available locally. Defaults to 1.0vars:subscription_id:*subscription_iddirectory_id:*directory_idclient_name:client-name# Replace it with the name of your clientThe wrapper uses the Service Principal's credentials to connect the Azure subscription. The given Service Principal must have access to the subscription.
The wrapper loadsclient_id,client_secretandtenant_idproperties from yourconfig.ymlfile located in~/.azurerm/config.yml.~/.azurerm/config.ymlfile structure example:---claranet-sandbox:client_id:aaaaaaaa-bbbb-cccc-dddd-zzzzzzzzzzzzclient_secret:AAbbbCCCzzz==tenant_id:00000000-0000-0000-0000-000000000000customer-profile:client_id:aaaaaaaa-bbbb-cccc-dddd-zzzzzzzzzzzzclient_secret:AAbbbCCCzzz==tenant_id:00000000-0000-0000-0000-000000000000Here is an example for a GCP/GKE stack with user ADC and multiple GKE instances:---gcp:general:mode:adc-userproject:&gcp_projectproject-namegke:-name:kubernetes-1zone:europe-west1-c-name:kubernetes-2region:europe-west1terraform:legacy:false# Use legacy version of the tool instead of OpenTofu, defaults to false. Only used if version >= 1.6.0.version:"1.0"# OpenTofu version that tfwrapper will use for this stack. Automatically downloaded if it's not available locally. Defaults to 1.0vars:gcp_region:europe-west1gcp_zone:europe-west1-cgcp_project:*gcp_projectclient_name:client-nameYou can declare multiple providers configurations, context is set up accordingly.⚠ This feature is only supported for Azure stacks for now and only works withAzure authentication isolation---azure:general:mode:service_principal# Uses an Azure tenant Service Principal accountdirectory_id:&directory_id"00000000-0000-0000-0000-000000000000"# Azure Tenant/Directory UIDsubscription_id:&subscription_id"11111111-1111-1111-1111-111111111111"# Azure Subscription UIDcredentials:profile:customer-profile# To stay coherent, create an AzureRM profile with the same name as the account-alias. Please checkout `azurerm_config.yml.sample` file for configuration structure.alternative:mode:service_principal# Uses an Azure tenant Service Principal accountdirectory_id:"00000000-0000-0000-0000-000000000000"# Azure Tenant/Directory UIDsubscription_id:"22222222-2222-2222-2222-222222222222"# Azure Subscription UIDcredentials:profile:claranet-sandbox# To stay coherent, create an AzureRM profile with the same name as the account-alias. Please checkout `azurerm_config.yml.sample` file for configuration structure.terraform:version:"1.0"# OpenTofu version that tfwrapper will use for this stack. Automatically downloaded if it's not available locally. Defaults to 1.0legacy:false# Use legacy version of the tool instead of OpenTofu, defaults to false. Only used if version >= 1.6.0.vars:subscription_id:*subscription_iddirectory_id:*directory_idclient_name:client-name# Replace it with the name of your clientThis configuration is useful when having various service principals with a dedicated rights scope for each.The wrapper will generate the following Terraform variables that can be used in your stack:<config_name>_azure_subscription_idwith Azure subscription ID. From the example, variable is:alternative_subscription_id = "22222222-2222-2222-2222-222222222222"<config_name>_azure_tenant_idwith Azure tenant ID. From the example, variable is:alternative_tenant_id = "00000000-0000-0000-0000-000000000000"<config_name>_azure_client_idwith Service Principal client id. From the example, variable is:alternative_client_id = "aaaaaaaa-bbbb-cccc-dddd-zzzzzzzzzzzz"<config_name>_azure_client_secretwith Service Principal client secret. From the example, variable is:alternative_client_secret = "AAbbbCCCzzz=="Also, an isolation context is set to the local.run/aure_<config_name>directory for each configuration.States centralization configurationTheconf/state.ymlconfiguration file defines the configuration used to connect to state backends.These backends can be ofAWS S3and/orAzureRMtypes.The resources for these backends are not created by tfwrapper, and thus must exist beforehand:AWS: an S3 bucket (and optionally but highly recommended a DynamoDB table for locking). It is also recommended to enable versioning on the S3 bucket.Azure: a Blob storage accountYou can use other backends (e.g. Google GCS or Hashicorp Consul) not specifically supported by the wrapper, if you manage authentication yourself and omit theconf/state.ymlfile or make it empty:---Example configuration with both AWS and Azure backends defined:---aws:-name:"aws-demo"general:account:"xxxxxxxxxxx"region:eu-west-1credentials:profile:my-state-aws-profile# should be configured in .aws/configazure:# This backend use storage keys for authentication-name:"azure-backend"general:subscription_id:"xxxxxxx"# the Azure account to use for state storageresource_group_name:"tfstates-xxxxx-rg"# The Azure resource group with state storagestorage_account_name:"tfstatesxxxxx"-name:"azure-alternative"general:subscription_id:"xxxxxxx"# the Azure account to use for state storageresource_group_name:"tfstates-xxxxx-rg"# The Azure resource group with state storagestorage_account_name:"tfstatesxxxxx"# This backend use Azure AD authentication-name:"azure-ad-auth"general:subscription_id:"xxxxxxx"# the Azure account to use for state storageresource_group_name:"tfstates-xxxxx-rg"# The Azure resource group with state storagestorage_account_name:"tfstatesxxxxx"azuread_auth:truebackend_parameters:# Parameters or options which can be used by `state.j2.tf` template filestate_snaphot:"false"# Example of Azure storage backend optionNote: the first backend will be the default one for stacks not definingstate_backend_type.How to migrate from one backend to another for state centralizationIf for example you have both an AWS and Azure state backend configured in yourconf/state.ymlfile,
you can migrate your stack state from one backend to another.Here is a quick howto:Make sure your stack is clean:$cdaccount/path/env/your_stack
$tfwrapperinit
$tfwrapperplan# should return no changesChange your backend in the stack configuration yaml file:---#state_configuration_name: 'aws-demo' # previous backendstate_configuration_name:"azure-alternative"# new backend to useBack in your stack directory, you can perform the change:$cdaccount/path/env/your_stack
$rm-vstate.tf# removing old state backend configuration$tfwrapperbootstrap# regen a new state backend configuration based on the stack yaml config file$tfwrapperinit# Terraform will detect the new backend and propose to migrate it$tfwrapperplan# should return the same changes diff as beforeStacks file structureTerraform stacks are organized based on their:account: an account alias which may refer to provider accounts or subscriptions, e.g.project-a-prod,customer-b-dev.environment:production,preproduction,dev, etc. Withglobalas a special case eliminating theregionpart.region:eu-west-1,westeurope, etc.stack: defaults todefault.web,admin,tools, etc.The following file structure is then enforced:# project root
└── account
│   └── environment
│       └── region
│           └── stack
└── account
    └── _global
        └── stackA real-life example:# project root
├── aws-account-1
│   ├── _global
│   │   └── default
│   │       └── main.tf
│   └── production
│       ├── eu-central-1
│       │   └── web
│       │       └── main.tf
│       └── eu-west-1
│           ├── default
│           │   └── main.t
│           └── tools
│               └── main.tf
└── aws-account-2
    └── backup
        └── eu-west-1
            └── backup
                └── main.tfUsageStack bootstrapAfter creating aconf/${account}_${environment}_${region}_${stack}.ymlstack configuration file you can bootstrap it.# you can bootstrap using the templates/{provider}/basic stacktfwrapper-a${account}-e${environment}-r${region}-s${stack}bootstrap# or another stack template, for example: templates/aws/foobartfwrapper-a${account}-e${environment}-r${region}-s${stack}bootstrapaws/foobar# or from an existent stack, for example: customer/env/region/stacktfwrapper-a${account}-e${environment}-r${region}-s${stack}bootstrapmycustomer/dev/eu-west/runIn the special case of a global stack, the configuration file should instead be named asconf/${account}_global_${stack}.yml.Working on stacksYou can work on stacks from their directory or from the root of the project.# working from the root of the projecttfwrapper-a${account}-e${environment}-r${region}-s${stack}plan# working from the root of a stackcd${account}/${environment}/${region}/${stack}tfwrapperplanYou can also work on several stacks sequentially with theforeachsubcommand from any directory under the root of the project.
By default,foreachselects all stacks under the current directory,
so if called from the root of the project without any filter,
it will select all stacks and execute the specified command in them, one after another:# working from the root of the projecttfwrapperforeach--tfwrapperinitAny combination of the-a,-e,-rand-sarguments can be used to select specific stacks,
e.g. all stacks for an account across all environments but in a specific region:# working from the root of the projecttfwrapper-a${account}-r${region}foreach--tfwrapperplanThe same can be achieved with:# working from an account directorycd${account}tfwrapper-r${region}foreach--tfwrapperplanComplex commands can be executed in a sub-shell with the-S/--shellargument, e.g.:# working from an environment directorycd${account}/${environment}tfwrapperforeach-S'pwd && tfwrapper init >/dev/null 2>&1 && tfwrapper plan 2>/dev/null -- -no-color | grep "^Plan: "'Passing optionsYou can pass anything you want toterraformusing--.tfwrapperplan---targetresource1-targetresource2Environmenttfwrapper sets the following environment variables.S3 state backend credentialsThe default AWS credentials of the environment are set to point to the S3 state backend. Those credentials are acquired from the profile defined inconf/state.ymlAWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_SESSION_TOKENAzure Service Principal credentialsThose AzureRM credentials are loaded only if you are using the Service Principal mode. They are acquired from the profile defined in~/.azurerm/config.ymlARM_CLIENT_IDARM_CLIENT_SECRETARM_TENANT_IDAzure authentication isolationAZURE_CONFIG_DIRenvironment variable is set to the local.run/azuredirectory if global configuration valueuse_local_azure_session_directoryis set totrue, which is the default, which is the default.If you have multiple configurations in your stacks, you also have<CONFIG_NAME>_AZURE_CONFIG_DIRwhich is set to the local.run/azure_<config_name>directory.GCP configurationThose GCP related variables are available from the environment when using the example configuration:TF_VAR_gcp_regionTF_VAR_gcp_zoneTF_VAR_gcp_projectGKE configurationsEach GKE instance has its own kubeconfig, the path to each configuration is available from the environment:TF_VAR_gke_kubeconfig_${gke_cluster_name}kubeconfig is automatically fetched by the wrapper (using gcloud) and stored inside the.rundirectory of your project.
It is refreshed automatically at every run to ensure you point to correct Kubernetes endpoint.
You can disable this behaviour by settingrefresh_kubeconfig: neverin your cluster settings.---gcp:general:mode:adc-userproject:&gcp_projectproject-namegke:-name:kubernetes-1zone:europe-west1-crefresh_kubeconfig:neverStack configurations and credentialsTheterraform['vars']dictionary from the stack configuration is accessible as Terraform variables.The profile defined in the stack configuration is used to acquire credentials accessible from Terraform.
There is two supported providers, the variables which will be loaded depends on the used provider.TF_VAR_client_name(if set in .yml stack configuration file)TF_VAR_aws_accountTF_VAR_aws_regionTF_VAR_aws_access_keyTF_VAR_aws_secret_keyTF_VAR_aws_tokenTF_VAR_azurerm_regionTF_VAR_azure_regionTF_VAR_azure_subscription_idTF_VAR_azure_tenant_idTF_VAR_azure_state_access_key(removed inv11.0.0)Stack pathThe stack path is passed to Terraform. This is especially useful for resource naming and tagging.TF_VAR_accountTF_VAR_environmentTF_VAR_regionTF_VAR_stackDevelopmentTestsAll new code contributions should come with unit and/or integrations tests.To run those tests locally, usetox:poetryruntox-epyLinters are also used to ensure code respects our standards.To run those linters locally:poetryruntox-elintDebug command-line completionYou can get verbose debugging information forargcompleteby defining the following environment variable:export_ARC_DEBUG=1Python code formattingOur code is formatted withblack.Make sure to format all your code contributions withblack ${filename}.Hint: enable auto-format on save withblackin yourfavorite IDE.ChecksTo run code and documentation style checks, runtox -e lint.In addition toblack --check, code is also checked with:flake8, a wrapper forpycodestyleandpyflakes.flake8-docstrings, a wrapper forpydocstyle.README TOCThisREADME's table of contentis formatted withmd_toc.Keep in mind to update it withmd_toc --in-place github README.md.Using OpenTofu development buildsTo build and use development versions of OpenTofu, put them in a~/.terraform.d/versions/X.Y/X.Y.Z-dev/folder:# git clone https://github.com/opentofu/opentofu.git ~/go/src/github.com/opentofu/opentofu# cd ~/go/src/github.com/opentofu/opentofu# go build ./cmd/tofu/# ./tofu versionOpenTofuv1.6.0-dev
onlinux_amd64# mkdir -p ~/.terraform.d/versions/1.6/1.6.0-dev# mv ./opentofu ~/.terraform.d/versions/1.6/1.6.0-dev/git pre-commit hooksSome git pre-commit hooks are configured in.pre-commit-config.yamlfor use with thepre-commit tool.Using them helps avoiding to push changes that will fail the CI.They can be installed locally with:# pre-commit installIf updating hooks configuration, run checks against all files to make sure everything is fine:# pre-commit run --all-files --show-diff-on-failureNote: thepre-committool itself can be installed withpiporpipx.Review and merge open Dependabot PRsUse thescripts/merge-dependabot-mrs.shscript frommasterbranch to:list open Dependabot PRs that are mergeable,review, approve and merge them,pull changes from github and pushing them to origin.Just invoke the script without any argument:# ./scripts/merge-dependabot-mrs.shCheck the help:# ./scripts/merge-dependabot-mrs.sh --helpTagging and publishing new releases to PyPIUse thescripts/release.shscript frommasterbranch to:bump the version with poetry,updateCHANGELOG.md,commit these changes,tag with last CHANGELOG.md item content as annotation,bump the version with poetry again to mark it for development,commit this change,push all commits and tags to all remote repositories.This will trigger a Github Actions job to publish packages to PyPI.To invoke the script, pass it the desired bump rule, e.g.:# ./scripts/release.sh minorFor more options, check the help:# ./scripts/release.sh --help