Verifying Deployment Using Terraform State Files
By: Venkat Sekar
Tags: Terraform, Hashicorp, DevOps, Iac, Open Source

Verifying Deployment Using Terraform State Files

Your First Step Towards Using Terraform in Production

Follow Along

GitLab: https://gitlab.com/hashmapinc/oss/tfstate_verifier

Overview

As I’ve continued working with Hashicorp’s Terraform open-source solution, I remain impressed. Terraform allows me to use Infrastructure as Code to provision and manage almost any cloud, infrastructure, or service that’s required. It comes packed with a great set of automation, reuse, and sharing capabilities that are supported by an active community of developers.

As a best practice for Terraform development, it’s highly recommended to test your deployments. This article, Terraform Code Quality, offers a good overview of testing the resource using well-known libraries and utilities. There are other widely available articles too that attest to the suggested approach.

One common behavior in most of these solutions involves connecting and querying Azure resources and ensuring that they are in conformance with the desired state.

Also, part of the unit testing strategy involves the following steps:

Creating resources from scratch.

Running various tests and ensuring the deployment is successful.

Teardown of the infrastructure.

This is typically done in a secluded environment to not affect development, like adversely impacting QA environments accidentally.

It would not be easy to perform such an introspection of the resources in production environments. In large enterprises, production environments are maintained by a separate operations team. The resources would be in a separate subscription and only accessible from a specific set of networks.

In production environments, we typically deploy premium solutions and enable more security features too. How do we validate if these were enabled or not?

In this blog post, I will approach the verification of resources that have been deployed by Terraform based on its state file.

An Opinionated Approach to Verification

We start with the base understanding:

Terraform, after deploying the resources, maintains the state. If there were any errors in deploying a particular resource, it would be errored out and not maintained in the state file.

Resources deployed using Terraform are not altered manually under normal conditions.

By reading the state file, we can confirm that a specific resource got deployed. However, we want to answer questions beyond that. For example:

We want to ensure “blob public access” is disabled.

Is the type of storage account RAGRS?

Is the object replication on the storage account configured?

Does a specific ADF instance have the System Managed Identity defined?

Some of these features might be turned off in the DEV, QA environment, but not necessarily in the production environment.

NOTE: This article is centered around Azure resources deployed using Terraform, but the pattern demonstrated here would apply to other implementations performed using Terraform.

Adoption Methodology

The Terraform state file is a simple JSON file. While Go and Ruby are very popular languages in system development, for me to take on the added burden of learning one of those languages to simply parse a JSON file isn’t compelling enough to make the time investment.

I am much more experienced in application and data analytics/engineering development using languages such as Java, Python, SQL, and Scala. Adopting existing libraries like TerraGrunt (Go-based), InSpec (Ruby-based), etc., was a non-starter. So to keep it simple, I used Python.

Implementation Logic

Before we proceed, I recommend reading the article Terraform — State demystified. It provides an in-depth view of what is being maintained in the Terraform state file.

I prefer to keep things simple. Instead of suggesting to explore and learn a tool or library, I prefer to see what can be done using the existing language libraries. Therefore during this walkthrough, I will use the following:

Python 3+

Pytest

It is easy to achieve the outcome we want with the Terraform State file being a JSON file.

We start by copying the state file to local storage. If you follow best practices, it is recommended that the state files be stored in a remote location like S3 or Azure Blob, etc. The Terraform State Pull command can be used to download the state file.

Once downloaded, use the python JSON package to parse the file into memory. Execute different test functions (ex based on pytest library) and assert if the deployed resources have met the criteria.

An Implementation Example

For the rest of the article, I will be walking through this verification process using code from :

https://gitlab.com/hashmapinc/oss/tfstate_verifier

Azure Resource

Below is the layout of resources that are deployed in Azure using Terraform. The scripts are found in the ‘HMAP-TFSTATE-RG.’

Keeping it simple, we create a Storage Account, Key Vault, and an Azure Data Factory (ADF). The ADF is linked to the Storage Account and Key Vault.

Verifications

As part of a sample, we verify the following:

Is the storage account deployed?

Is the resource group deployed?

Is the ADF deployed?

Does ADF have a System Managed Identity?

Is the Storage Account publicly accessible?

As reflected earlier, we parse the Terraform State file (TFStateParser.py). Once parsed, we pass the instance to a specific test function that asserts for desired conditions.

The code for verifying if the storage account is publicly accessible is displayed below. Nothing fancy, just simple and plain. This code is present in ‘test/test_HMAP_TFSTATE_RG.py.’

The pytest framework executes the tests and creates a report, which we demonstrate in the section below.

Example Run

I have the sample Terraform State file ‘examples/example_terraform.tfstate’; the private key pieces of information have been replaced with dummy values for obvious reasons.

The tests are executed using the following commands:

pip install -r requirements.txt pytest -vr s — state_file=../examples/example_terraform.tfstate \ — html=../examples/test_report.html

The output of execution:

The report is also captured in an HTML file (examples/test_report.html):

As you can see, this clearly reflects which of the configurations are conformant and which are not. Also, this script does not query Azure. For this reason, you could take the state any number of times and run these verifications to indicate any failures on future deployments.

One of the tests is “Failed” purposefully, as a demonstration in the illustration above.

Limitations

This verification is only possible for resources that have a Terraform provider. This verification would not work for resources like the budget, for which a provider is not available.

As new features are added in Azure, more time is required for those features to be added to the Terraform provider. During this time, these verifications cannot be done using this approach.

Final Thoughts

This form of verification, though simple, could be thought of as an aid in your team's initial journey to IAC adoption.

Need Assistance with Your Terraform Deployment?

Hashmap offers a range of enablement workshops and assessment services, cloud migration services, and consulting service packages as part of our service offerings — we would be glad to work through your specific requirements — please reach out.