Source-controlled Infrastructure for Software Developers
By: Norbert Varga
Tags: Terraform, Infrastructure As Code, Emarsys Dev, Google Cloud Platform, DevOps

Source-controlled Infrastructure for Software Developers

Photo on pxfuel

System engineering is not my field, I’m a developer guy, in-and-out. Of course, I know some basic Unix tools, not afraid to use vim, remember something about networks and hardware stuff, but none of these topics ever touched my heart to deepen my knowledge in system engineering.

For most of us, software developers, infrastructure is a pre-built tool either provided us by a system engineering team or a third party. You learn how to deploy your application in it and live-on. And this was fine until DevOps and NoOps became a thing. In the last decade, system engineers got newer and newer toys (like Chef or Puppet) to manage application infrastructure much more easily. On the other hand, using Infrastructure as a Service provider and microservice architecture became the standard architecture for scalable software development. I still haven’t felt the urge to check these tools out (although they sound really fancy), Infrastructure as a Service providers gave enough control over the infrastructure that made me feel comfortable using it, deployment became much easier along with scaling an application. And now the cloud providers are the big players in the game, changing the way we develop software.

Nowadays you’ll certainly be dead if you’re not thinking in cloud-native applications. Building in the cloud is the future of scalable and maintainable applications.

With this new era came a new mindset and new tools have arrived, too. This mindset is the Infrastructure as Code and its best hammer is called Terraform, which blew my mind when I first encountered it.

What is Infrastructure as Code?

Let’s see how it is described in Wikipedia:

Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools.

So you write some kind of code that is readable for us humans and also can be parsed via a tool than can modify our infrastructure in the described way. How awesome is that?!

IaC tools can be divided into several categories, from the simplest scripts, through configuration management until powerful server provisioning tools. Although custom scripts (good ol’ rsync ) and configuration management tools existed way before Infrastructure as Code was a thing, server provisioning tools (Terraform, Pulumi) are rather new.

What is Terraform?

Terraform is an open-source Infrastructure as Code command-line tool that can create, manage and update infrastructure resources such as physical machines, VMs, network switches, containers and more. Almost any infrastructure type can be represented as a resource in Terraform.

It’s fully written in Go and created by Mitchell Hashimoto and HashiCorp, the very same company that gave us Vagrant or Vault, just to mention a few of their very well-renowned products.

Infrastructure management is achieved through different providers. A provider is responsible for understanding API interactions and exposing resources. Providers can be any IaaS (e.g. Google Cloud Platform, AWS, Microsoft Azure, etc.), PaaS (e.g. Heroku), or SaaS services (Gitlab, PagerDuty, Cloudflare) or just an ordinary MySql database. It supports more than 100 different providers.

How does it work?

You can create your configuration using a high-level configuration language known as Hashicorp Configuration Language (HCL), or optionally in JSON (which comes handy when you want to write your own tool around Terraform).

With these configuration files, you can plan and apply modifications or even visualize infrastructure simply by invoking the related Terraform commands. Planning means you can see what will be changed in the infrastructure in detail, comparing the current state of the infrastructure to what state it would be if we apply the changes in the plan.

HCL provides us with options to define and use variables, simple data structures, loops to create the desired resources under a provider. We can even reference resources and their properties in another resource or wait for resources to be created if another one depends on that resource.

Creating our first infrastructure code

Well, the first thing you need is the Terraform itself. You can do this in several ways in different operating systems, but for sake of example, let’s install it on macOS through Homebrew with brew install terraform command.

Now that we have Terraform installed, we can create our config file. First, we need a provider where we want to provision our resources. I chose the Google Cloud Platform for this example. I created a project for it in GCP’s cloud console, created a service account, downloaded the credentials and exported the content of the whole file in a specific env var that is used by Google Terraform Provider so we can perform authenticated change requests:

$ export GOOGLE_CREDENTIALS=/path/to/credentials.json

This is used by Google Terraform Provider to make changes in the managed resources.

You can find in depth guide to create a Google Project in the documentation and more on what service accounts are and how to use them.

So with the project id and a specific (in this case, europe-west3) region, we can set up the Google provider:

Next, we can use the terraform init command to initialize our infrastructure state from the config file.

This will create a .terraform folder with our provisioned resources’ current state using a local backend (which is not a good practice as you don’t want to lose these state files if your machine broke down, but we will come back to change this later). A backend for Terraform is a place where Terraform will look for the state file, in this case, it is the current folder where the command is run. The output of it looks something like this:

Let’s create our first resource. Remember what I said about storing state locally? Let’s use a remote backend and create a GCP Storage bucket for it

and plan out our infrastructure change with terraform plan command:

Terraform will create a plan and shows us what it is about to change, in this case, a new bucket named project-bucket-terraform in europe-west3 region with other default properties, like storage class, will be created. If everything looks fine, it’s time to make these changes with the terraform apply command. This (in most cases) outputs the same thing as when we ran the previous plan and asks us if we wanted to really execute these changes.

Now let’s switch our local backend to a remote one. To do this, we need to add a new backend to our config file:

Then run terraform init again. Terraform will recognize that we are going to switch our backend to a remote one and will ask us if we want to copy our current state file into the bucket we created earlier.

That’s it! Our infrastructure’s state backed up remotely and updated automatically in the future. Adding additional resources from here on only requires a few new lines in the config file.

Bookkeeping infrastructure changes

Photo by Susan Yin on Unsplash

Terraform files can be considered as source code, thus it’s better to be stored in a version control system where we can keep track of all our infrastructure changes, versioning our config and not making changes manually, which is hard to track or even unfold why it’s ever happened. We can even automate reviewing and applying changes made with Terraform with tools like Atlantis or GitHub Actions.

Terraform is a powerful tool in the cloud era, especially for non-system engineers. It provides a new way to build up infrastructure, quicker than any time before. It can help startups to grow without requiring a whole team of system engineers, but a few who could do a lot more and faster with Terraform than a team of system engineers without it.

More to read:

- https://en.wikipedia.org/wiki/Infrastructure_as_code

- https://www.terraform.io/docs