PART 1.1: INFRASTRUCTURE PIPELINE — Building a Scalable App Environment with Infrastructure and Deployment
By: Markus Herkommer
Tags: Azure Devops, Terraform, Azure, Infrastructure, Continuous Integration

PART 1.1: INFRASTRUCTURE PIPELINE — Building a Scalable App Environment with Infrastructure and Deployment

image by author

This is one part of a series. So if you have not read the PART 0: OVERVIEW you can go there and read it to get an overview of what we will actually doing here …

Introduction

In the last PART 1 INFRASTRUCTURE we have created the resources we need for our scalable web application. In this part we want to automate the infrastructure by creating a pipeline for it. The goal of it is like with every other pipeline. We can deliver our infrastructure via continuous integration and approval process for different stages.

We need to download terraform inside our build agent. For this we have different possibilities. We could do this “manually” (bash script) or use the terraform build/release tasks. I wanted to use the build tasks, because they take a lot of work away from you, but they can not deal with workspaces and that sucks. The first “problem” was to select a workspace. This is not supported. So you can do it with bash. And yes this works. But the problem is, that the task does not take the selected workspace. It takes me hours to identify what is going wrong there, but there is a bug in that task.

So I decided to use the build task to download terraform to the agents and do the rest (authentication and terraform actions) via bash commands.

Furthermore I have to say that I had some issues to determine/fix while building this pipelines. This may differs totally from environment to environment. But the good thing is when you will try to fix your personal issues you will learn a lot of how terraform and azure works.

Prerequisites

You need something to do before we can really start with the pipelines… First you need to work through the first part (PART 1), where we have created our terraform configurations. These can also be downloaded/cloned from https://dev.azure.com/sternschleuder/Notifier/_git/Infrastructure?version=GBfeature%2Fpart1.

Terraform Build/Release Task

Go to the terraform build/release tasks and install the task into you azure devops environment.

Azure

First delete all resources that we have created in the PART 1 of the series (best way is to delete the resource group we used there), because we want to create everything new with our pipelines. And to avoid different states and other problems.

Then we create a resource group which is explicitly used for creating the resources. Or in other words to persist our terraform state remotely, so we can work with pipelines, local and with other developers sharing the same state.

So for this please create a resource group named “notifier-resource-group”. We need a storage account which used by terraform to save the states. So create one and name it “notifiertfstore” and choose the even created resource group as resource group for the storage. Then go inside the storage and choose “Storage Explorer” and right click on “BLOB CONTAINERS” and “Create blob container” and name it “notifiertfstate”. This is the location where terraform will create the states.

And a good thing would be to add a group in azure and add your user account to it (the account where you have logged in to azure cli “az login“). This is a good thing when we wanted to use terraform plan/apply from local or if other devs wanted to do this. Else there could be some auth errors when accessing the key vaults resources. So go to “Azure Active Directory” click “Groups” and create one named “notifier-devs”.

Code Changes

First open the “backend.tf” file and replace that code with the following:

backend.tf

terraform {

backend "azurerm" {

tenant_id = YOUR_TENANT_ID

subscription_id = YOUR_SUBSCRIPTION_ID

resource_group_name = "notifier-resource-group"

storage_account_name = "notifiertfstore"

container_name = "notifiertfstate"

key = "terraform-notifier.tfstate"

}

}

Here we are going to use our new created remote backend for persisting the terraform state. Please set here your tenant and subscription id (You can get this information by “az account list” — id is the subscription id!).

Now we need to change our “common.yaml” to the following. We add here our service connection to have also access to our keyvaults. Do not forget to replace the placeholder in capitals with your information! The “YOUR_SERVICE_CONNECTION_OBJECT_ID” you will get from the azure resource manager service connection created later in the build pipeline section. After the service connection is created you will find the service connection name and object id in azure in azure ad under app registrations.

common.tf

tenant_id: YOUR_TENANT_ID kv_allow:

YOUR_SERVCE_CONNECTION_NAME: # Service Connection (principle which used in azuredevops pipeline)

object_id: YOUR_SERVICE_CONNECTION_OBJECT_ID

secret_permissions: ["get", "list", "delete", "set"]

notifier-devs: # Allow group

object_id: YOUR_NOTIFIER_DEVS_OBJECT_ID

secret_permissions: ["get", "list", "delete", "set", "recover", "backup", "restore"]

Build Pipeline

First we create the build pipeline and initialize terraform for our workspaces and validate them. Furthermore we create the artifact with the terraform files to later create the resources in the release pipeline.

Let’s got to azure devops now and navigate to your notifier project pipelines. The press “create pipeline” and choose Azure Repos Git for your source. Select the “Infrastructure” repository. Choose then “Starter pipeline” and click “Save and run” und yes commit it directly into the master branch. After the job has been successfully finished. Check out the repository changes from the origin. After that you should see “azure-pipelines.yml” file und your “Infrastructure” folder. From here we start to add our needed build stuff. So open the file in your editor and let’s go…

First thing to do is delete all stuff in there and then copy the following.

azure-pipelines.yaml

trigger:

- master resources:

repositories:

- repository: self pool:

vmImage: 'ubuntu-latest'

The trigger sets a branch where the pipeline will be automatically triggered. The repository source is to include the repositories which needed in the build. This is in our case only “self” (the repository which triggered the build). And the pool where we define our vmImage. We set it to latest ubuntu.

If we have this done, we can go to the steps. First delete all the code inside the steps section. As the first action we install terraform with the “Terraform Installer” task.

azure-pipelines.yaml

azure-pipelines.yml steps:

- task: TerraformInstaller@0

displayName: Install Terraform Latest

inputs:

terraformVersion: 'latest'

Then we need to authorize to azure. We do this with a “Azure CLI” task. But before we can do this we have to create a service connection in azure devops to add azure into azure devops. For this do the following:

Click on “Project settings” Click on “Service connections” in the “Project Settings” sidebar Click on “Create service connection” Choose “Azure Resource Manager” and click “Next” Click “Service principal (automatic)” and “Next” Select Scope Level “Subscription” and choose your azure subscription Leave the resource group empty, so you have access to all resource groups from your subscription. (maybe there will pop up an auth window where you have to login with your azure credentials) And enter a name for the Azure Resource Manager connection (In my case “ARM Notifier”)

After we have created the service connection, we can add the task. Leave a blank line after the last step and put the following code in to it… And care about indentation! (yaml is very sensitive with this)

azure-pipelines.yaml

- task: AzureCLI@1

displayName: Authorize Azure

inputs:

azureSubscription: 'ARM Notifier'

scriptLocation: inlineScript

inlineScript: |

echo "##vso[task.setvariable variable=AZURE_CLIENT_ID;issecret=false]${servicePrincipalId}"

echo "##vso[task.setvariable variable=AZURE_CLIENT_SECRET;issecret=true]${servicePrincipalKey}"

echo "##vso[task.setvariable variable=AZURE_SUBSCRIPTION_ID;issecret=false]$(az account show --query 'id' -o tsv)"

echo "##vso[task.setvariable variable=AZURE_TENANT_ID;issecret=false]${tenantId}"

addSpnToEnvironment: true

This task authorize us with our created “ARM Notifier” service connection. Now we can initialize an validate our configurations. We do this with following bash task where we are using the “ARM” environment variables. Which we putting under the azure cli task with line break in-between.

azure-pipelines.yaml

- bash: |

terraform init

for ENV in "acc" "prd"

do

terraform workspace select $ENV || terraform workspace new $ENV

terraform validate

done

workingDirectory: '$(System.DefaultWorkingDirectory)'

displayName: 'Terraform Init/Validate configuration'

env:

ARM_CLIENT_ID: $(AZURE_CLIENT_ID)

ARM_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)

ARM_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)

ARM_TENANT_ID: $(AZURE_TENANT_ID)

We directly go further by adding another task which copies our terraform configs the artifact staging directory. So put…

azure-pipelines.yaml

- task: CopyFiles@2

displayName: Copy Terraform Configs

inputs:

SourceFolder: '.'

Contents: '**'

TargetFolder: '$(build.ArtifactStagingDirectory)'

CleanTargetFolder: true

OverWrite: true

And the last task is to publish the artifact…

azure-pipelines.yaml

- task: PublishBuildArtifacts@1

displayName: Publish Terraform Artifacts

inputs:

PathtoPublish: '$(build.ArtifactStagingDirectory)'

ArtifactName: 'tf'

publishLocation: 'Container'

So — if we have all done this, our build pipeline should work! Please trigger your build pipeline to see if it is working. Best to push the code changes and the should the pipeline automatically be triggered.

Release Pipeline

The release pipeline is responsible for applying our terraform changes in azure. First we are going to create a new release pipeline. For this go again to azure devops -> pipelines -> releases and create a pipeline. Select the “empty job” template and press apply. Then name the stage “acc plan” and close the sidebar window. The release pipelines (at the moment) can not be edited via yaml. So we have to use the user interface. But after we have closed the sidebar our pipeline looks like this:

image by author

I have already renamed the pipeline name (on the top) to “Infrastructure Release”. We need to choose the artifact now (in the graphic above it is already done). Press the add button in the artifacts section and choose build as source type. Choose the notifier project and the Infrastructure build pipeline as source like in the graphic below and press add.

image by author

Now we add some tasks here. For this please click on the “acc plan” stage on the job link. Then we are in the task view. Click the plus button to add the “terraform install” task. The task here is almost the same like before in the build pipeline, but we have to use the ui here. But there is nothing more to fill out here.

Next we add a further task to authorize to azure. Please select an Azure CLI task and choose our “ARM Notifier” for the “Azure Resource Manager connection”. The script type is “Shell” and choose inline script andput the following into it:

echo "##vso[task.setvariable variable=AZURE_CLIENT_ID;issecret=false]${servicePrincipalId}"

echo "##vso[task.setvariable variable=AZURE_CLIENT_SECRET;issecret=true]${servicePrincipalKey}"

SUBSCRIPTION_ID=`az account show --query 'id' -o tsv`

echo "selected subscription ${SUBSCRIPTION_ID}"

echo "##vso[task.setvariable variable=AZURE_SUBSCRIPTION_ID;issecret=false]${SUBSCRIPTION_ID}"

echo "##vso[task.setvariable variable=AZURE_TENANT_ID;issecret=false]${tenantId}"

Please open the “Advanced” tab and enter the “$(working_dir)” as “Working Directory”. This is an variable which will be set later when have create a task group from our tasks…

Now we came to the last task to initialize, sleect workspace and call the terraform plan or apply action. Choose a new bash task choose “Inline” and put in the following:

terraform init

terraform workspace select $(env) || terraform workspace new $(env) if [ "$(tf_action)" = "apply" ]; then

terraform $(tf_action) -auto-approve

else

terraform $(tf_action)

fi

This script uses two variables (“$(env)” and “$(tf_action)”). Also these vars will be set later… Check here the auto approve flag when applying the changes. And we need to add here the environment variables which came from the azure cli task. So please set these in the “Environment Variables” tab. Add the following:

Name Value

ARM_CLIENT_ID $(AZURE_CLIENT_ID)

ARM_CLIENT_SECRET $(AZURE_CLIENT_SECRET)

ARM_SUBSCRIPTION_ID $(AZURE_SUBSCRIPTION_ID) ARM_TENANT_ID$(AZURE_TENANT_ID)