Terragrunt — Becoming one with its internal behaviors
By: Patrick Picard
Tags: Terragrunt, Terraform

Terragrunt — The multi-tool

Terragrunt, a small wrapper around Terraform, adds a ton of capabilities. In my previous posts, I focused on capabilities and using it in novel ways.

In this article, I focus on how Terragrunt behaves to accomplish the capabilities. I share some lessons learned and how to troubleshoot. To start with, I go over a sample hierarchical configuration structure, Terragrunt’s order of operation, caching, source path gotchas, and content generation.

Terragrunt Configuration Structure

Let’s start with a two-level hierarchical Terragrunt configuration. A terragrunt.hcl in a sub-folder and one in the parent (this one would be shared with all its sub-folders if they have an include directive.

The folder structure below shows the hierarchy of Terragrunt configurations

├── core-iam

│ └── terragrunt.hcl

├── core-organization

│ └── terragrunt.hcl

├── credentials

│ └── terragrunt.hcl

├── log-analytics

│ └── terragrunt.hcl

└── terragrunt.hcl

Let’s start with the core-organization/terragrunt.hcl:

dependency "credentials" {

config_path = "../credentials"

}

source = "git::ssh:// terraform {source = "git::ssh:// git@github.com /xxxxxx/terraform-azurerm-core-organization.git" extra_arguments "force_subscription" {

commands = [

"init",

"apply",

"destroy",

"refresh",

"import",

"plan",

"taint",

"untaint"

] env_vars = {

ARM_TENANT_ID = dependency.credentials.outputs.tenant_id

ARM_CLIENT_ID = dependency.credentials.outputs.client_id

ARM_CLIENT_SECRET = dependency.credentials.outputs.client_secret # Need to force a subscription id when logging in as SPN.

ARM_SUBSCRIPTION_ID = local.config.management.subscription_id

}

}

} locals {

config = yamldecode(file(format("%s/config.yaml", get_env("AZ_PIPELINE_CONFIG_PATH"))))

} inputs = {

root_management_group_name = local.config.global.root_management_group_name

connectivity_subscription_ids = [

for sub in local.config.subscriptions.connectivity : sub.id

] # Configuration abbreviated

} include {

path = find_in_parent_folders()

}

And the root folder terragrunt.hcl

locals {

rscfg = yamldecode(file(find_in_parent_folders("remote_state.yaml")))

} remote_state {

backend = "azurerm"

generate = {

path = "backend.tf"

if_exists = "overwrite"

}

config = {

tenant_id = local.rscfg.remote_state.tenant_id

subscription_id = local.rscfg.remote_state.subscription_id

resource_group_name = local.rscfg.remote_state.common.resource_group_name

storage_account_name = local.rscfg.remote_state.common.storage_account_name

container_name = local.rscfg.remote_state.common.container_name key = "${path_relative_to_include()}/terraform.tfstate" snapshot = true }

} generate "provider" {

path = "provider.tf"

if_exists = "overwrite"

contents = <<EOF

provider "azurerm" {

version = "~>2.0"

features {}

}

EOF

}

Now that we have configurations to work with, let’s go over how Terragrunt behaves.

Order of Operations

When you run any of the Terragrunt commands (init, plan, apply, destroy), it performs a series of tasks; some of which depend on the configuration. To make things simple, let’s use terragrunt plan as the action executed from the core-organization/ folder.

Terragrunt looks at the current folder terragrunt.hcl and follows all the instructions in it:

Look at every dependency and make sure their state is available

Initialize the module defined in the terraform.source. Terragrunt automatically initializes modules. No need to run terragrunt init first. From this step, you will see the creation of a hidden folder named .terragrunt-cache (more details shortly).

Materialize include statements. In this case, we search for a terragrunt.hcl in a parent folder.

The parent configuration has a remote_state and generate directives. The remote_state is evaluated and generates backend.tf on the fly. This file is saved in the terragrunt cache along with the other .tf files. provider.tf is generated similarly.

The locals/inputs are calculated and the extra_arguments generate environment variables. This information is passed down to Terraform to deploy the module. Up to this point, it was all Terragrunt magic and control was handed over to terraform.

.terragrunt-cache

Terragrunt uses a cache as its working directory. This folder is critical for troubleshooting and can be a source of headaches (see next section).

In the cache, guid like paths are created. When you navigate to the lowest level, you notice a few interesting things.

ls -la .terragrunt-cache/OfnG7YSki89VGJUc7uL4eruVFG4/BoxvdZRzDTvrVMxcmMGN5EdtYOU

total 76

drwxr-xr-x 6 patrick patrick 4096 Dec 21 09:43 .

drwxr-xr-x 3 patrick patrick 4096 Dec 18 22:40 ..

drwxr-xr-x 6 patrick patrick 4096 Dec 18 22:40 .git

-rw-r--r-- 1 patrick patrick 64 Dec 18 22:40 .gitignore

-rw-r--r-- 1 patrick patrick 225 Dec 18 22:40 .pre-commit-config.yaml

drwxr-xr-x 4 patrick patrick 4096 Dec 21 09:43 .terraform

-rw-r--r-- 1 patrick patrick 239 Dec 21 09:43 .terragrunt-module-manifest

-rw-r----- 1 patrick patrick 27 Dec 18 22:40 .terragrunt-source-version

-rw-r--r-- 1 patrick patrick 2656 Dec 18 22:40 README.md

-rw-r--r-- 1 patrick patrick 456 Dec 21 09:43 backend.tf

drwxr-xr-x 4 patrick patrick 4096 Dec 18 22:40 examples

-rw-r--r-- 1 patrick patrick 1216 Dec 18 22:40 locals.tf

-rw-r--r-- 1 patrick patrick 1304 Dec 18 22:40 main.tf

-rw-r--r-- 1 patrick patrick 1472 Dec 18 22:40 output.tf

-rw-r--r-- 1 patrick patrick 106 Dec 21 09:43 provider.tf

-rwxr-xr-x 1 patrick patrick 1775 Dec 21 09:43 terragrunt.hcl

drwxr-xr-x 2 patrick patrick 4096 Dec 18 22:40 tests

-rw-r--r-- 1 patrick patrick 1397 Dec 18 22:40 variables.tf

-rw-r--r-- 1 patrick patrick 629 Dec 18 22:40 variables_naming.tf

This folder contains the files in the module referenced by the terraform.module directive. In the .terraform folder, dependencies that were initialized are stored. You can also see the backend.tf and provider.tf that we generated earlier.

So if you are unsure of your remote_state and generate directives, have a look at the generated files!

If you wanted, you could navigate to this folder and do a terraform apply. However, the inputs are not available since Terragrunt generates those on the fly and does not materialize them into a tfvars for example.

The Terragrunt cache is a great tool, but sometimes you will run into behavior that does not align with the configuration you have. It is perfectly safe to destroy the cache (rm -rf .terragrunt-cache) and re-run terragrunt to regenerate it.

Source Paths

In the first configuration, we defined a module path using a SSH URL which is reproduced again:

Terragrunt supports all the path types that terraform supports. However, there are some behavioral differences that are worth noting. Remote URL’s (SSH, Github, etc) are considered immutable. Once Terragrunt has cached it, it will NOT attempt to retrieve it again. With that said, it is best to use versioning to make the URL variable and force re-downloads as you upgrade your modules.

Modules that use relative paths are ALWAYS refreshed on every run. This is useful when doing local development with a related module that you are modifying.

source = "../../..//core-organization"

This behavior is documented, but often glossed over. This has caught me and my peers many times. You have been warned.

Relative vs. Remote URLs

Generated Content

Earlier, we looked a remote_state and generate directive. These inform Terragrunt to create files on the fly which are persisted in the Terragrunt cache.

generate "provider" {

path = "provider.tf"

if_exists = "overwrite"

#......

}

The content generation behavior is controlled using the if_exists attribute. This attribute supports 3 options:

overwrite — generate the file and overwrite if the file already exists

— generate the file and overwrite if the file already exists skip — generate the file, but leave existing file if one exists

— generate the file, but leave existing file if one exists error — exit with an error if a file with the path defined already exists.

I personally prefer the overwrite option so that I am in control. When Terragrunt generates a file, it includes a hash in the content so it knows if the content changed between runs (see the Sig:)

# Generated by Terragrunt. Sig: nIlQXj57tbuaRZEa

provider "azurerm" {

version = "~>2.0"

features {}

}

I recommend that you use a consistent name for provider and backend configuration. For example, if someone had a providers.tf in the remote module and you generate provider.tf, terraform will complain about duplicate configuration.

Conclusion

There is a lot to unpack out of this article. I shared a lot of important knowledge tidbits that you gain after using Terragrunt for a while. Many of which I have learned through battle scars! It is important to be one with the tools you use.

I shared a sample hierarchical configuration to showcase how Terragrunt performs its “wrapper” operations before handing over to Terraform.. Understanding terragrunt caching is very important for troubleshooting and helps understanding the behavior of terraform module sources and configuration generation.

Enjoy!