Automate Your Infrastructure with GitOps, Terraform and CircleCi
By: John Clarke
Tags: Terraform, Gitops, Gcp, Github, Circleci

CircleCI definition file

Now we’ve got all of the bits we need to get the workflow up and working in CircleCI.

The first thing we’ll need to do is build the workflow definition that we want CircleCI to execute for us. To do this we add a .circleci folder to the root of the project containing the Terraform definition, and in that add a config.yml file, based on the one below

This uses version 2 of CircleCI’s infrastructure and definition syntax. First we set up some defaults that will be used in each of the jobs defined in the file. A job is a single unit of the process we want to run, these jobs are then built into a workflow at the end of the file using rules to control when and how each step is run. The main default we set up is the Docker image that we want to use to run each job, in this instance it’s a custom image in the public Docker registry that I’ve built from the Docker file we looked at earlier.

It’s worth noting at this stage that CircleCI spins up a fresh container for each job, so in theory we could use a different image for each job. We don’t want to do that in this case, be we do need to be able to persist state between jobs, we’ll see how to do that when we look at the jobs in detail.

The first job we define, init , first checks out the target Git repo into the container, and then creates a new file in the same directory containing the GCP authentication token held an environment variable we’ll create when we configure CircleCI itself. This file is used to allow Terraform to authenticate with our project in GCP and read state, as well as create our infrastructure. Finally we run terraform init which will pull the latest state for the infrastructure, and ensure all of the correct Terraform plugins are installed and ready to go.

Obviously all of this will have changed the working directory from what was checked out from source control, and we want to ensure all of those updates are available in any subsequent jobs. We can do that by using CircleCI’s persist_to_workspace operation, to effectively push the current directory, to enable us to pop it in later jobs in the pipeline.

The next job we’ll define is plan , which runs a terraform plan . First we can pop the workspace we persisted above, which get’s all of the updates from the previous job into the fresh container that’s been spun up for this job. Using that state, we can run the terraform plan command, saving the output so that again, we can use it in later jobs as and when we need to. We also take this plan, and save it as a build artifact so we have it available to be manually reviewed. Finally, we once again save the modified state for later jobs to use.

The final job we define is apply , which we use to apply the Terraform plan we previously created, show the results, and then store them as a build artifact so again, we can review them later.

That’s all of the building blocks for the pipeline in place, now we use CircleCI’s workflows facility to define how they’re used to construct our pipeline.

First off, we want to run the init job; we want to do this in all situations the pipeline is triggered. Next we want run plan , but not until init has completed, so set init as a required job for plan to run thus ensuring they run sequentially (by default, CircleCI will try and run jobs in parallel where it can). Finally, we want to run apply once plan has completed, so that the changes get applied to our infrastructure. The caveat here, though, is that we only want to run apply if the change triggering CircleCI was made to the master branch of the project — you’ll recall from our GitOps workflow that only changes to master get applied, anything else is only planned for manual review prior to a merge. We can ensure this happens by filtering the branches that cause apply to run down to master only.

So, that’s our CircleCI workflow defined. We’ll get it checked into GitHub, and then we can head off and complete the processes by attaching the GitHub project to CircleCI, so that the triggers start firing, the jobs start running, and we close the circle (so to speak!).