CI/CD for Terragrunt IaC through Atlantis
By: Angelos Kaltsikis
Tags: Atlantis, Terragrunt, Gitops, Terraform, Infrastructure As Code

The old days

At Beat, we run all our workloads on AWS. We use infrastructure as code (IaC) to create and manage our AWS resources. As mentioned in one of our previous medium articles, Scalable Terraform Architecture, our initial IaC was based on AWS Cloudformation. We created a JSON based configuration, which was being committed in Github in order to leverage the VCS benefits. However, as the files were getting bigger and more complex we found out that even meticulous code reviews could fail to find possible problems. Another drawback we found was the workflow of Cloudformation didn’t require any peer review before applying the changes submitted. This means that any user that had access to update the Cloudformation stack could do so from a local file in their laptop whenever they wanted to.

Two years ago we decided to switch to Terraform 0.11 with a single repository to store all of the Terraform code. Terraform code was more readable and better organized than the huge JSON blobs we used in Cloudformation, so it seemed that the first problem improved. In order to solve the second problem while searching, we came across a very promising project, called Atlantis. This project could help us practice usual GitOps on our Terraform code which seemed to be exactly what we were looking for. Last but not least this tool conformed to our vision of giving our developers a secure autonomous way to create and manage infrastructure after being reviewed by peers without the need of providing access to each developer to run things from their laptop.

Nowadays

Lately we decided to use Terragrunt in our environments IaC, moving from an architecture of a few remote states to one architecture of a few thousand of remote states along with the change of the binary that was now required to run them (Terragrunt instead of Terraform). Reasons behind can be found on our medium blog Scalable Terraform Architecture. Subsequently this created a series of challenges to make our Atlantis installation perform as expected. Lets focus now on the steps and optimizations that were required to make Atlantis work with our new architecture seamlessly. As we had introduced Terragrunt, we could leverage its benefits to push one step further our vision of giving our developers access to all of our repositories and allow them to actively participate in parts that were available till then only to DevOps Engineers.

Wrapper for Terragrunt

Terragrunt is notorious for its very detailed logs which at the time were not very configurable. There are multiple ways to handle that, which can be found with a simple search on the internet (some based on wrapper scripts and some based on regex that strip logs). We decided to fork one in order to modify it as needed and also share it with the community. This wrapper covers both the plan & apply methods of Terragrunt and can be also used outside Atlantis in a CI system for example.

Wrapper for less verbose Terragrunt output (plan/apply)

Terragrunt Custom Workflow

Atlantis, fortunately, supports some customization. As seen in the documentation, you can create a custom Workflow with Terragrunt. As we have decided to use a wrapper around Terragrunt, we have used the following custom workflow.

Atlantis Custom Workflow for Terragrunt

Atlantis Custom Dockerfile

We are running our Atlantis inside Kubernetes which means that in order to run Terragrunt in an Atlantis Pod, we had to modify the default Docker image that it runs on. We also identified that we are going to need to support more Terraform Versions than the ones already supported by Atlantis (the latest version for each minor release). As we migrated from a few Terraform remote state files to a few thousand, we are going to certainly need to use local copies for the Terraform Providers to speed things up & save tons of bandwidth, in case we want to run multiple plan/apply commands. Another reason to have those providers locally was that before Terraform 0.13, having manually downloaded community providers was a prerequisite otherwise Terraform was failing to run.

Atlantis Repo level configuration

In the past, we used a hardcoded Atlantis configuration for each repository like in this example. While Atlantis doesn’t require setting this configuration, we prefer using it as it gives us more fine-grained settings for each repository. During our search for how we could create the Atlantis repository configuration and how to make the creation more dynamic, we came across a project which dynamically generates it. This was a project in its early steps but we decided to use it and of course contribute with bug fixes, new feature requests, and development.

Running Terragrunt Atlantis Config Generation at the user’s end

Initially, we started by installing this tool on our laptops, added it as a pre-commit hook for our repository which created a number of issues. First of all the users had to download this tool. Second, it was generating a huge YAML configuration which the user had to push to VCS in each PR. Third, the execution of the scripts created problems for the users’ local workflow as it required several seconds to run for all the subdirectories (>2000). However, we started immediately investigating how we could tackle the above problems. We assessed different solutions like running the script on different stages in the git workflow, running the script on a Github action, or in our CI but none of those solved all of the problems at the same time. We took a step back and thought, where this functionality should have been in the first place? Eureka, Atlantis it is.

Moving Terragrunt Atlantis Config Generation inside Atlantis

Unfortunately, Atlantis didn’t give us the level of customization we needed to run this config generation at the step we believed is the most efficient (after the clone of each repository), through custom workflows or by any other means. However, after diving into the code of Atlantis we found out that most likely we could achieve what we wanted with a small hack. It seemed that we could instruct Atlantis to run whatever we want as a post-checkout git hook. This gave us the ability to create a logic where Atlantis runs a script that exists on a certain folder for each repository, which means that we can easily instruct Atlantis to execute the config generation script with the flags needed by each repository. Of course, as we moved into the generation of the Atlantis Config inside Atlantis we needed to download the Terragrunt Atlantis Config binary inside our Docker Image.

Optimizations for running Parallel Plan/Apply in Atlantis

Atlantis started supporting Parallel Plan/Apply since v0.13 but this feature has been only available in case you were using Terraform Workspaces. We don’t use Terraform workspaces, however, this was only needed to bypass the Atlantis locking mechanism. This means that in theory, this could work perfectly if each Terragrunt directory is executed in a unique workspace. As we were running some major updates on almost all of our IaC during this time, we were very interested in the possibility of speeding things up. By this time an update has been pushed to the Atlantis config generation which made this functionality a reality. We started immediately testing it. While the core functionality was working as expected we faced some other issues which needed some optimizations. The first step Atlantis executes, in its parallel mode, is to copy the whole directory for each repository in a different workspace directory, which triggers a few thousand calls to the Atlantis config generation script. We pushed an update to our post-checkout hooks which instructed Atlantis to run the generation only once, which solved this issue. As we were using workspaces in Atlantis now it was a bit tedious for the user to add both directory and workspace in case they needed to run a specific plan/apply. As a result, we created a PR for the generator to improve the user experience by requiring only the generated project name by the user. Last but not least to truly speed up our execution we had to increase both the Pod Resources Requests/Limits & the disk size in case we needed to run commands on all of our environments.

The Dockerfile that includes all the aforementioned optimisations

The bash script Atlantis runs to call a certain script (if existing) for each IaC repository

The bash script which resides in each IaC repository

Where Next?

In Beat, we love to constantly evolve our offerings, improve the user experience while improving the underlying base for them. Following those principals, we have leveraged the power of separate AWS Accounts to achieve isolation between resources that are being deployed by different teams, giving them more autonomy and fewer things to worry about. Of course, as Atlantis is the tool that our developers use to deploy/manage their Terragrunt configuration files we have deployed multiple Atlantis deployments to intensify the separation and isolation between different parts of the infrastructure. Even though the current solution is stable and fast, as parallel execution reduces the times dramatically, we are still looking for ways to improve our current setup and the tooling around it. For example, we are evaluating the caching of Modules to reduce the network costs and trying to solve other known Atlantis limitations (like supporting multiple stage plans) which are expected to improve our users’ experience a lot.

If you found this article interesting and you are looking for your new challenge, check out our open roles.