Automate Terraform Testing with GitHub Actions and Terratest
By: Petri Autero
Tags: Test Automation, Infrastructure As Code, Continuous Integration, Github Actions, Terraform

In this blog post I go through my experience of integrating the following tools to write tests and automate IaC testing:

Terraform — Tool for building, changing, and versioning infrastructure safely and efficiently.

— Tool for building, changing, and versioning infrastructure safely and efficiently. Terratest — A swiss army knife for testing infrastructure code.

— A swiss army knife for testing infrastructure code. GitHub Actions — Create custom software development life cycle (SDLC) workflows directly in your GitHub repository.

Here is what this post will cover:

Getting Started

The following steps guide you through the steps of creating a simple module, writing an example and test cases for it, and configuring GitHub Workflows to run the tests. You can optionally fork the example repository in GitHub and experiment with the workflows and examples there.

Prerequisites

To be able to go through the steps in the blog post, you need to meet the following prerequisites:

Access to GitHub and GitHub Actions — note that GitHub Actions are free of charge for public repositories.

— note that GitHub Actions are free of charge for public repositories. Go 1.13 +

Terraform 0.12.20 + : I suggest using a tool like tfenv — check out this blog post for reference.

: I suggest using a tool like — check out this blog post for reference. AWS Account and Credentials available for Terraform

Directory structure

The following code block shows the directory structure we’ll be using with the example:

gopath/src/static-website

├── .github

│ └── workflows # GitHub workflow definitions

├── examples

│ └── static-site # Test used for integration test

└── test # Terratest test files

└── fixtures

└── static-site # Test fixture for unit test

NOTE: You will have to create all files under your GOPATH location.

Create a Sample Terraform Module

As the blog post is focusing more on the testing aspect, rather than the module itself, we‘ll keep the module really simple. All the inputs, resources and outputs are embedded within a single file:

# static-website/main.tf # ~~~~~~~~~~~~~~~~~~~~~~~

# DEPLOY A STATIC WEBSITE

# ~~~~~~~~~~~~~~~~~~~~~~~



# REQUIRE A SPECIFIC TERRAFORM VERSION OR HIGHER

terraform {

required_version = ">= 0.12.20"

}



# ---------

# VARIABLES

# ---------

variable "website_name" {

description = "The name of your static website."

type = string

}



variable "region" {

description = "AWS region this bucket should reside in"

type = string

}



variable "force_destroy" {

description = "Delete all objects from the bucket so that the bucket can be destroyed even when not empty"

type = bool

default = false

}



variable "tags" {

description = "Tags to add to resources"

type = map(string)

default = {}

}



# ------------------------------------------------

# CREATE THE S3 WEBSITE BUCKET AND ATTACH A POLICY

# ------------------------------------------------

resource "aws_s3_bucket" "website" {

bucket = var.website_name

acl = "public-read"

tags = var.tags

region = var.region

force_destroy = var.force_destroy



website {

index_document = "index.html"

error_document = "error.html"

}

}



resource "aws_s3_bucket_policy" "website" {

bucket = aws_s3_bucket.website.id

policy = data.aws_iam_policy_document.website.json

}



data "aws_iam_policy_document" "website" {

statement {

actions = [

"s3:GetObject"]



resources = [

"${aws_s3_bucket.website.arn}/*"]



principals {

type = "AWS"

identifiers = ["*"]

}

}

}



# -------

# OUTPUTS

# -------

output "s3_bucket_website_endpoint" {

value = aws_s3_bucket.website.website_endpoint

description = "Website endpoint for the S3 bucket"

}



output "s3_bucket_name" {

description = "Name of of website bucket"

value = aws_s3_bucket.website.id

}

This module deploys a simple S3 website we’ll be using for tests.

Write an example for the module

Following the best practices for Terraform modules, we will write an example for the module, containing an end-to-end sample:

# static-website/examples/static-site/main.tf # --------------------------

# CONFIGURE OUR AWS PROVIDER

# --------------------------



provider "aws" {

region = var.region

}



# ---------

# VARIABLES

# ---------

variable "website_name" {

description = "The name of your static website. Note that this needs to be globally unique"

type = string

default = "terratest-with-gh-actions"

}



variable "region" {

description = "AWS region this bucket should reside in"

type = string

default = "us-east-1"

}



variable "tags" {

description = "Tags to add to resources"

type = map(string)

default = {

CI = "true"

}

}



# ----------------------------------------------

# CREATE THE S3 WEBSITE AND STORE SAMPLE CONTENT

# ----------------------------------------------

module "website" {

source = "../../"



region = var.region

website_name = var.website_name



// We want to be able to destroy after the test

force_destroy = true

tags = var.tags

}



resource "aws_s3_bucket_object" "index" {

key = "index.html"

bucket = module.website.s3_bucket_name

source = "index.html"

acl = "public-read"

content_type = "text/html"

}



resource "aws_s3_bucket_object" "error" {

key = "error.html"

bucket = module.website.s3_bucket_name

source = "error.html"

acl = "public-read"

content_type = "text/html"

}



# -------

# OUTPUTS

# -------

output "s3_bucket_name" {

description = "Name of of website bucket"

value = module.website.s3_bucket_name

}



output "s3_bucket_website_endpoint" {

value = module.website.s3_bucket_website_endpoint

description = "Website endpoint for the S3 bucket"

}

Also, you’ll have to create two files that will be uploaded to the bucket as part of the example:

# static-website/examples/static-site/index.html Hello, index! -------------------------------------------------------------------- # static-website/examples/static-site/error.html Hello, error!

Write a Unit Test for the Module

Write a test fixture for unit tests

Before we get started on the unit test, create a test fixture:

# static-website/test/fixtures/static-site/main.tf provider "aws" {

region = "us-east-1"

}



module "website" {

source = "../../../"



region = "us-east-1"

website_name = "test-fixture"

}

Prepare Go Modules

Before you can begin writing tests, you will first have to initialize Go Modules for the test project. In your test directory, type the following command:

# static-website/test ❯ go mod init

Write a unit test

Now that the module has been initalized, we need to import a couple of libraries:

# static-website/test ❯ go get github.com/gruntwork-io/terratest

❯ go get github.com/stretchr/testify

Next we’ll create the unit test:

# static-website/test/static_site_unit_test.go // +build unit



package test



import (

"github.com/gruntwork-io/terratest/modules/terraform"

test_structure "github.com/gruntwork-io/terratest/modules/test-structure"

"github.com/stretchr/testify/assert"

"path/filepath"

"testing"

)



func TestStaticSiteValidity(t *testing.T) {

t.Parallel()

_fixturesDir := test_structure.CopyTerraformFolderToTemp(t, "../", "test/fixtures")

exampleDir := filepath.Join(_fixturesDir, "static-site")

terratestOptions := &terraform.Options{

TerraformDir: exampleDir,

Vars: map[string]interface{}{},

}

t.Logf("Running in %s", exampleDir)

output := terraform.InitAndPlan(t, terratestOptions)

assert.Contains(t, output, "2 to add, 0 to change, 0 to destroy", "Plan OK and should attempt to create 2 resources")

}

As you can see, the unit test isn’t creating any resources. We’re only validating that we can run terraform plan successfully and get the expected output.

At this point, we’re ready to run the unit test. Note that the Terraform AWS provider requires you to have valid AWS Credentials on your PATH.

# static-website/test ❯ go test -v -tags=unit

...

--- PASS: TestStaticSiteValidity (14.11s)

Test output in Go can sometimes be a bit verbose and hard to read. If you want prettier output, you can try for example gotestsum:

# static-website/test ❯ gotestsum --format short-verbose -- -tags=unit

PASS TestStaticSiteValidity (12.63s)

PASS . DONE 1 tests in 15.416s

Write an Integration Test for the Module

Now that our unit test is passing, we can add an integration test:

# static-website/test/static_site_integration_test.go // +build integration



package test



import (

"fmt"

http_helper "github.com/gruntwork-io/terratest/modules/http-helper"

"github.com/gruntwork-io/terratest/modules/logger"

"github.com/gruntwork-io/terratest/modules/random"

"github.com/gruntwork-io/terratest/modules/retry"

"github.com/gruntwork-io/terratest/modules/terraform"

test_structure "github.com/gruntwork-io/terratest/modules/test-structure"

"github.com/stretchr/testify/assert"

"path/filepath"

"strings"

"testing"

"time"

)



const WebsiteNameKey = "website_name"

const WebsiteOutput = "s3_bucket_website_endpoint"



func TestStaticSiteDeployment(t *testing.T) {

t.Parallel()



// Uncomment these when doing local testing if you need to skip any stages.

//os.Setenv("SKIP_bootstrap", "true")

//os.Setenv("SKIP_apply", "true")

//os.Setenv("SKIP_perpetual_diff", "true")

//os.Setenv("SKIP_website_tests", "true")

//os.Setenv("SKIP_destroy", "true")



_examplesDir := test_structure.CopyTerraformFolderToTemp(t, "../", "examples")

exampleDir := filepath.Join(_examplesDir, "static-site")



test_structure.RunTestStage(t, "bootstrap", func() {

random := strings.ToLower(random.UniqueId())

randomName := fmt.Sprintf("terratest-gh-actions-%s", random)

test_structure.SaveString(t, exampleDir, WebsiteNameKey, randomName)

})



// At the end of the test, run `terraform destroy` to clean up any resources that were created

defer test_structure.RunTestStage(t, "teardown", func() {

terraformOptions := test_structure.LoadTerraformOptions(t, exampleDir)

terraform.Destroy(t, terraformOptions)

})



// Apply the infrastructure

test_structure.RunTestStage(t, "apply", func() {

websiteName := test_structure.LoadString(t, exampleDir, WebsiteNameKey)

terratestOptions := &terraform.Options{

// The path to where your Terraform code is located

TerraformDir: exampleDir,

Vars: map[string]interface{}{

"website_name": websiteName,

},

}

// Save the terraform oprions for future reference

test_structure.SaveTerraformOptions(t, exampleDir, terratestOptions)

terraform.InitAndApply(t, terratestOptions)

})



// Run perpetual diff

test_structure.RunTestStage(t, "perpetual_diff", func() {

terraformOptions := test_structure.LoadTerraformOptions(t, exampleDir)

planResult := terraform.Plan(t, terraformOptions)



// Make sure the plan shows zero changes

assert.Contains(t, planResult, "No changes.")

})



// Run HTTP tests

test_structure.RunTestStage(t, "website_tests", func() {

terraformOptions := test_structure.LoadTerraformOptions(t, exampleDir)

websiteEndpoint := terraform.OutputRequired(t, terraformOptions, WebsiteOutput)



testURL(t, websiteEndpoint, "", 200, "Hello, index!")

testURL(t, websiteEndpoint, "notfound", 404, "Hello, error!")

})

}



func testURL(t *testing.T, endpoint string, path string, expectedStatus int, expectedBody string) {

url := fmt.Sprintf("%s://%s/%s", "http", endpoint, path)

actionDescription := fmt.Sprintf("Calling %s", url)

output := retry.DoWithRetry(t, actionDescription, 10, 10 * time.Second, func() (string, error) {

statusCode, body := http_helper.HttpGet(t, url, nil)

if statusCode == expectedStatus {

logger.Logf(t, "Got expected status code %d from URL %s", expectedStatus, url)

return body, nil

}

return "", fmt.Errorf("got status %d instead of the expected %d from %s", statusCode, expectedStatus, url)

})

assert.Contains(t, output, expectedBody, "Body should contain expected text")

}

This time, the test creates actual resources in AWS and runs a series of tests to validate the results. Again, before running the test, make sure you have AWS Credentials set up.

# static-website/test ❯ go test -v -tags=integration

...

--- PASS: TestStaticSiteDeployment (62.44s)

As you can see, we’re using Go build tags to distinguish unit and integration tests. This will be very useful when set up the GitHub workflows for our tests.

Setup GitHub Actions to Automate the Tests

Set up AWS Secrets in the GitHub repository

As we use the AWS provider in both the unit and integration tests, you will have to provide secrets in the repository. Navigate to “Settings/Secrets” and add your AWS credentials:

AWS Credentials

At the time of writing, GitHub Actions doesn’t unfortunately support user / organisation -level secrets, you will have to configure those at the repository level.

Set up GitHub Actions for Unit Tests

Now that everything is prepared, it’s time to automate everything with GitHub Actions.

Let’s start by creating a workflow for the unit tests:

# static-website/test/.github/workflows/unit.yml name: Unit Tests

on: push

jobs:

go-tests:

name: Run Go Unit Tests

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v1

- uses: actions/setup-go@v1

with:

go-version: 1.13

- uses: autero1/action-terraform@v0.1.0

with:

terraform_version: 0.12.21

- name: Download Go Modules

working-directory: test

run: go mod download

- name: Run Go Tests

working-directory: test

run: go test -v -tags=unit

env:

AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}

AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

There are multiple optimisations we could add, such as caching Go Modules, but we’ll skip those for simplicity.

Now, when you push your changes to you repository, after a while you should see something like this under the Actions tab:

Unit Test results

GitHub actions might complain about a failed annotation, most likely due to stderr output by Terratest: Running in /tmp/… The test won’t fail for that, so you can disregard the warning.

Set up GitHub Actions for Integration Tests

In the final step of this tutorial, we’ll create a workflow for the integration tests:

# static-website/test/.github/workflows/integration.yml name: Integration Tests

on: push

jobs:

go-tests:

name: Run Go Integration Tests

runs-on: ubuntu-latest

steps:

- uses: actions/checkout@v1

- uses: actions/setup-go@v1

with:

go-version: 1.13

- uses: autero1/action-terraform@v0.1.0

with:

terraform_version: 0.12.21

- name: Download Go Modules

working-directory: test

run: go mod download

- name: Run Go Tests

working-directory: test

run: go test -v -tags=integration

env:

AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}

AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

To make it easier to test, we’ll run the integration tests on push events, as well. In real life, you might not want to run all the integration tests on every push. You can read more about the workflow triggers in the official documentation.

Optional: Make the unit test fail

The main module specifies the minimum Terraform version:

# static-website/main.tf # REQUIRE A SPECIFIC TERRAFORM VERSION OR HIGHER

terraform {

required_version = ">= 0.12.20"

}

To make the Unit test run fail, change the Terraform version in the Unit test workflow to 0.12.19 and observe the results:

- uses: autero1/action-terraform@v0.1.0

with:

terraform_version: 0.12.19 <- Use an earlier version

You should now see your Unit Test workflow fail:

Failed Unit Test

To further enhance your tests, you could use a build matrix to continuously test against multiple Terraform versions.

Tips and Tricks

Here are some general tips and tricks to make you more successful when you write tests for your infrastructure code.

Use randomised names for resources in integration tests

Unlike unit tests, integration tests deploy actual resources. To avoid conflicts with resource names, use a random prefix or suffix for your resources. When you’re using randomised names and run your tests locally step by step, create a bootstrap test step to prepare and store the variables to avoid creating a new random name for each iteration.

test_structure.RunTestStage(t, "bootstrap", func() {

random := strings.ToLower(random.UniqueId())

randomName := fmt.Sprintf("terratest-gh-actions-%s", random)

test_structure.SaveString(t, exampleDir, WebsiteNameKey, randomName)

})

Use a separate test account and clean up resource regularly

Sometimes the cleanup phase might fail and you might start building up unwanted resources in your test account. To avoid running into quota issues or paying for unwanted resources, use resource tags for whenever you can, to identify resources that were created as part of your testing cycles. You can also set up a tool like cloud-nuke to regularly clean up your test account.

Use retry loops in your tests

When you create resources, it might sometimes take a while until your resources are accessible. You can avoid intermittent errors caused by eventual consistency by using retry loops in your integration tests:

output := retry.DoWithRetry(t, actionDescription, 10, 10 * time.Second, func() (string, error) {

statusCode, body := http_helper.HttpGet(t, url, nil)

if statusCode == expectedStatus {

logger.Logf(t, "Got expected status code %d from URL %s", expectedStatus, url)

return body, nil

}

return "", fmt.Errorf("got status %d instead of the expected %d from %s", statusCode, expectedStatus, url)

})

This example runs a particular action max 10 times, with 10 second wait time between retries.

Develop your tests with stepwise execution

One particularly useful way of writing the tests is developing them with stepwise execution. Rather than running the test from start to end, you can use os.Setenv("SKIP_xxx") to only run specific stages in your test like this:

...

os.Setenv("SKIP_bootstrap", "true")

os.Setenv("SKIP_apply", "true")

//os.Setenv("SKIP_validation_tests", "true")

... test_structure.RunTestStage(t, "validation_tests", func() {

...

This way you can run your validation steps, etc. without having to redeploy and destroy the resources over and over again — dramatically speeding up your feedback cycle.

Next Steps

This post is just a brief introduction to the capabilities of Terratest and GitHub Actions. To learn more:

Happy testing!