3 tips to manage drift from manual changes in Terraform
Tags: Terraform, Azure, DevOps, Infrastructure As Code, AWS

3 tips to manage drift from manual changes in Terraform

How do you manage manual infrastructure change with Terraform? If someone changes something manually on the infrastructure, how do you handle it on Terraform, and how do you run Terraform plan next time?

Terraform import subcommand

There are different ways to manage manual changes on your infrastructure in Terraform, depending on the case. Here are 3 options :

Change the Terraform code manually to update your Terraform State

Let’s say you have you have a security group that was changed manually by one of your team members, like opening an HTTP port for a specific subnet, and you discover this at the next terraform apply . This is an easy case.

You will have the diff in the terraform output (be it on CI, or on your laptop). That means that you can add this difference as a snippet directly on your terraform code and apply it. So you will manually add the code after the manual deployment, which means that at the next terraform apply , Terraform will notice that the state doesn’t have this new item (in our example, allowing HTTP for a sub network) and will require your cloud provider (AWS for example) to do it, and AWS will return : “I already have it”. So, in this easy case, the difference is just going to be written to the TFstate file, and you solved your case manually. It’s a bit like fixing a conflict during a git merge.

So:

Write the missing code

Push it

Apply it

The delta disappears

And the final state is updated on the state file.

Reverse Terraform the new existing environment with a tool like Terraformer or Terraforming.

If you have much more difficult case, like a full VM or even worse maybe a full Kubernetes with 10,000 nodes run manually by your new team member, I would not try to write the code manually like I just did for a very simple case. I would use a tool like Terraformer or Terraforming.

This kind of tools scan your cloud provider accounts for resources and parameters, and once they have all the parameters (let’s say for example on a VM : hard disk, IP address, operating system type, etc…) they will turn all those elements into valid Terraform code and create a TFState from the code.

Terraformer is developed by a Sergey in Tel-aviv from the Waze SRE team at Google. It will do something like 80% of the job, and you just need to adapt some things here and there. Terraforming is a community project.

So, you will end up with:

the code

an existing manual deployment,

and a state file

Problem now is that this is not really merged to your existing infrastructure repo and that’s the third subject : chose what you import.

Import specific parts of your new code with the Terraform import subcommand

The third option is to take that code and import it this time differently by using the terraform import subcommand to be very precise / picky about what you import.

Let’s say, as an example that there is a specific IP address that was launched manually for an old load balancer, but production needs to keep this particular IP address and you want to be sure that it’s under control on terraform. You can write the corresponding terraform code and import the corresponding ID on AWS.

It’s quite different from just pushing / applying some code and checking if “it works” on the other side. You have this code, and you know that if you execute it right now, it is going to be duplicate (in this precise example because you can have hundreds of IP address with a name; it was not the case in the previous example with the security group where you can only have one port 80 open).

So in this example, I would define this very specific IP address through an importer like Terraformer (so you have the code if you don’t know how to write it), and copy/paste it on my existing code and then do not apply directly but import it with the terraform import subcommand.

Here’s an example of how the terraform import command looks like.

Just zooming in on the terraform --help command

So, as a wrap-up, here the key tips to manage drift from manual changes in Terraform

The 3 ways I would recommend to manage manual changes in your infrastructure with Terraform, depending on the situation would be: