Create Pipeline with Terraform & Setup Container Image Scans with Snyk in AWS CodeBuild
By: Lukonde Mwila
Tags: Software Development, Software Engineering, Programming, Docker, DevOps

Containerising applications offers numerous benefits. However, you want to make sure that your containers are based on secure images and that you can be aware of any vulnerabilities in your applications and their dependencies. A security platform like Snyk offers this kind of in-depth container image scanning and security.

In this post, I’ll be demonstrating how you can setup a two stage pipeline in AWS with GitHub as the source along with the CI/CD tools CodeBuild and CodePipeline using Terraform. In addition, we will create an application that we will create a Docker image from to scan, test and configure continuous monitoring in Snyk. This is a good technique to ensure that you secure container images and test them before pushing them into your image repository.

I would recommend that you following along with the source code for this post which is available here.

If you enjoy the post, feel free to buy me a coffee here ☕️ 😃 .

Create Application & Docker Files

Let’s get started by creating the application that we’ll be containerising. If you’ve read some of my previous posts then you can probably guess what kind of application we’ll be working with 😃. That’s right, a React application. As usual, we’re going to keep things very simple because the focus of this post isn’t application development. All we’ll be doing is ensuring that our production grade Docker image is a React build that serves its static content through an Nginx web server. We will also have a second Docker image that will be used to run application tests, as well as check for any existing vulnerabilities in our image using Snyk.

Inside my project folder, I create a directory called docker-application. Inside this docker application folder, I run the following command:

npx create-react-app . --use-npm

This will create a basic React application with npm as the package manager. I want to make two updates to the React application. The first is in the App.test.js file. I update the default test to the following that checks if the main App component successfully renders when mounted onto the DOM.

import React from 'react';

import ReactDOM from "react-dom";

import App from './App'; test('renders without crashing', () => {

const div = document.createElement("div");

ReactDOM.render(<App />, div);

ReactDOM.unmountComponentAtNode(div);

});

The second update is to create an nginx directory in the root of the docker-application folder with a default.conf file. The default.conf file will container the configuration to setup a web server that serves static content from the React build and listens for traffic on port 3000. When our Docker image is being created, this configuration file will be copied over from our local project into the image snapshot.

server {

listen 3000; location / {

root /usr/share/nginx/html;

index index.html index.htm;

}

}

Alright, let’s move on to the Dockerfiles. Whilst in the same directory, we can create both Dockerfiles using the terminal:

touch Dockerfile Dockerfile.dev

The Dockerfile.dev file will be used to run a container that executes our tests.

Dockerfile.dev

FROM node:13.14-buster-slim AS alpine

WORKDIR /app

COPY package.json .

RUN npm install

COPY . . CMD ["npm", "run", "test"]

The second Dockerfile will be used to create our production grade image.

Dockerfile

FROM node:13.14-buster-slim as build

WORKDIR /app

COPY package*.json ./

RUN npm install

COPY . .

RUN npm run build FROM nginx

EXPOSE 3000

COPY ./nginx/default.conf /etc/nginx/conf.d/default.conf

COPY --from=build /app/build /usr/share/nginx/html

You can then proceed to create a Docker image and create a container to ensure that your React application’s test is passing successfully.

docker build -t yourname/react-debug -f Dockerfile.dev .

docker run -e CI=true yourname/react-debug

Once you’ve done that, you should see a result similar to the following:

> docker-application@0.1.0 test /app

> react-scripts test PASS src/App.test.js

✓ renders without crashing (26ms) Test Suites: 1 passed, 1 total

Tests: 1 passed, 1 total

Snapshots: 0 total

Time: 1.912s

Ran all test suites.

Create Snyk Account & Store API Token in AWS Secrets Manager

Signing up for a Snyk account is pretty straightforward. Once you’re signed up and logged into your account. Hover over your username in the top right corner and click on the General Settings option from the dropdown menu that appears. Once you’re on the Account Settings page on the General tab, fetch your API Token. We’re going to be storing this token in AWS Secrets Manager. You can do this using the AWS Console. If you haven’t done this before, not to worry. The process of filling in the relevant fields in Secrets Manager is very intuitive, but in case of any issues you can refer to the documentation here.

Testing Snyk Locally

As much as we’ll be using the Snyk CLI in the CI build stage of our pipeline, it would still be a good idea to test our application locally for before pushing our changes to a repository. To do this, you can install the Snyk CLI using one of the methods outlined here. Once you have that installed, you can check the version with the following command:

snyk --version

Next, we’ll want to authenticate with the CLI tool so that we can track any vulnerabilities that are detected in our Docker image and keep track of them continuously in our Snyk account. To do this, you run the following command in the CLI:

snyk config set api=XXXXXXXX

Once that is done, you can run a test on the Docker image you created earlier, as well as setup continuous monitoring with the following commands:

snyk test --docker yourname/react-debug:latest --file=Dockerfile.dev

snyk monitor --docker yourname/react-debug:latest --file=Dockerfile.dev

The first command execution should produce a list of vulnerabilities detected in the image. To monitor these issues, you can head over to your Snyk Dashboard and you’ll be able to get details of security issues with your image along with the categorised severity of each.

Configure BuildSpec File for CodeBuild

If you’re new to AWS CodeBuild, it’s essentially a CI (Continuous Integration) service or tool that compiles source code, runs tests, and produces software packages or artifcats that are ready to be deployed. We are going to be configuring the build stage of the pipeline using a configuration file saved as buildspec.yml. This file will specify the phases which represent the commands CodeBuild runs during each phase of the build. You can read more details on the buildspec file here.

install: install dependencies you may need for your build

pre_build: final commands to execute before build

build: actual build commands

post_build: finishing touches

You can create this file in the root directory of your project (outside of the docker-application folder). Here’s what we want to achieve in the CI stage of our pipeline:

Install Snyk CLI

Authenticate with Snyk authentication token

Build Docker image for testing

Create container and run application tests

Scan container image and setup monitoring

Build production Docker image

Login to Docker CLI

Push production grade Docker image to Docker Hub

buildspec.yml



phases:

install:

runtime-versions:

docker: 18

pre_build:

commands:

# Install Snyk

- echo Install Snyk

- curl -Lo ./snyk "

- chmod -R +x ./snyk

# Snyk auth

- ./snyk config set api="$SNYK_AUTH_TOKEN"

# Build Docker image for testing

- echo Building the Docker image for testing...

- docker build -t yourname/dkr-scanned-react-container-image-test -f ./docker-application/Dockerfile.dev ./docker-application

build:

commands:

- echo Build started on `date`

# Run tests with built Docker image

- echo Run react tests...

- docker run -e CI=true yourname/dkr-scanned-react-container-image-test

# Scan Docker image with Snyk

- ./snyk test --severity-threshold=medium --docker yourname/dkr-scanned-react-container-image-test:latest --file=./docker-application/Dockerfile.dev

- ./snyk monitor --docker yourname/dkr-scanned-react-container-image-test:latest --file=./docker-application/Dockerfile.dev

# Build the production Docker image

- echo Building the production Docker image...

- docker build -t yourname/dkr-scanned-react-container-image ./docker-application/

# Log in to the Docker CLI

- echo "$DOCKER_PW" | docker login -u "$DOCKER_ID" --password-stdin

post_build:

commands:

# Take these images and push them to Docker hub

- echo Pushing the Docker images...

- docker push yourname/dkr-scanned-react-container-image version: 0.2phases:runtime-versions:docker: 18commands:# Install Snyk- echo Install Snyk- curl -Lo ./snyk " https://github.com/snyk/snyk/releases/download/v1.210.0/snyk-linux - chmod -R +x ./snyk# Snyk auth- ./snyk config set api="$SNYK_AUTH_TOKEN"# Build Docker image for testing- echo Building the Docker image for testing...- docker build -t yourname/dkr-scanned-react-container-image-test -f ./docker-application/Dockerfile.dev ./docker-applicationcommands:- echo Build started on `date`# Run tests with built Docker image- echo Run react tests...- docker run -e CI=true yourname/dkr-scanned-react-container-image-test# Scan Docker image with Snyk- ./snyk test --severity-threshold=medium --docker yourname/dkr-scanned-react-container-image-test:latest --file=./docker-application/Dockerfile.dev- ./snyk monitor --docker yourname/dkr-scanned-react-container-image-test:latest --file=./docker-application/Dockerfile.dev# Build the production Docker image- echo Building the production Docker image...- docker build -t yourname/dkr-scanned-react-container-image ./docker-application/# Log in to the Docker CLI- echo "$DOCKER_PW" | docker login -u "$DOCKER_ID" --password-stdincommands:# Take these images and push them to Docker hub- echo Pushing the Docker images...- docker push yourname/dkr-scanned-react-container-image

You may notice the following environment variables in the above configuration file:

SNYK_AUTH_TOKEN, DOCKER_ID, DOCKER_PW

The values for each of these are obtained from Secrets Manager and are configured as environment variables for use by CodeBuild. This is provisioned in our Terraform code which we will be moving on to next.

Lastly, make sure to push your existing code changes to a GitHub repository that will be used in the source stage of your pipeline.

Using Terraform for AWS Infrastructure

We’ll start by creating a bucket for the backend remote state of our infrastructure and a DynamoDB table for state locking in the case that you’re working within a team. Terraform must store state about your managed infrastructure and configuration, and the backends are responsible for storing the state and providing an API for state locking to prevent multiple runs for multiple Terraform components. Also, I’ll be making use of Terragrunt which is a wrapper for Terraform to keep our IaC (Infrastructure as Code) DRY.

Prerequisites:

AWS CLI tool

Configured AWS profile with CLI

Terraform and Terragrunt

Setting Up The Backend Remote State

Create S3 bucket for backend remote state using the AWS CLI tool:

aws s3api create-bucket --bucket <your-bucket-name> --region eu-west-1 --create-bucket-configuration LocationConstraint=eu-west-1

Create DynamoDB table for state locking with a single attribute LockID which will also be the Hash key:

aws dynamodb create-table --table-name <your-table-name> --attribute-definitions AttributeName=LockID,AttributeType=S --key-schema AttributeName=LockID,KeyType=HASH --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

Folder Structure & Configuration

As you can imagine, there are different ways of structuring your IaC. I’ll be keeping my Terraform modules and the code for the environments to be deployed in the same repository but in separate folders. At the root of the project directory, I’ll have the parent Terragrunt configuration file and a file consisting of sensitive variables. Here’s what it will look like:

├── docker-application/

├── infra-live/prod/terragrunt.hcl

├── infra-modules/cicd

├── buildspec.yml

├── sensitive.tfvars

└── terragrunt.hcl

In our parent terragrunt.hcl file, we set up the remote backend configuration and the table to be used for state locking:

terragrunt.hcl

remote_state {

backend = "s3"

generate = {

path = "backend.tf"

if_exists = "overwrite_terragrunt"

}

config = {

bucket = "<s3-bucket-name>" key = "${path_relative_to_include()}/terraform.tfstate"

region = "eu-west-1"

encrypt = true

dynamodb_table = "<dynamo-db-table-name>"

}

}

We’re going to have the following some files and variable values that we don’t want committed as part of our source code, so we’ll include them in the .gitignore file. In the senstive.tfvars file you can set the following variables that will be added to the CLI by Terragrunt:

sensitive.tfvars

profile = xxx

region = xxx

github_secret_name = xxx

docker_secret_name = xxx

snyk_secret_name = xxx

profile — add a profile name for an AWS account configured in the CLI on your machine

region — the selected region you want to deploy your infrastructure in

github_secret_name — the name of the secret store containing your GitHub personal access token

docker_secret_name — the name of the secret store container your Docker ID and Docker password

snyk_secret_name — the name of the secret store with you Snyk authentication token

Lastly, in this subsection, here’s the .gitignore to make sure we omit the relevant files and folders from our git history:

.gitignore

.DS_Store

.idea

*.iml

**/.terragrunt-cache/* # Local .terraform directories

**/.terraform/* # .tfstate files

*.tfstate

*.tfstate.*

.idea/* # .tfvars files

sensitive.tfvars # Crash log files

crash.log # Ignore override files as they are usually used to override resources locally and so

# are not checked in

override.tf

override.tf.json

*_override.tf

*_override.tf.json # Include override files you do wish to add to version control using negated pattern

#

# !example_override.tf # Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan

# example: *tfplan*

AWS Infrastructure Modules

Inside our infra-modules folder, we’ll add a sub folder called cicd for the infrastructure related to the pipeline we’ll be setting up.

├── cloudwatch/

├── codebuild/

├── codepipeline/

├── lambda/

├── main.tf

├── provider.tf

└── variables.tf

CloudWatch — setup event tracking for changes in the status of the pipeline and send state changes to Lambda

CodeBuild — setup CI for building and testing of our Docker images, as well as configure scans and continuous monitoring with Snyk

CodePipeline — setup two stage pipeline with GitHub source and a CodeBuild project for the build stage

Lambda — deploy a function that will receive CloudWatch events and push notifications to Slack

Main Resource Files

In this section, I’ll only demonstrate the main files, but all the source code is available in a public repository here.

cloudwatch.tf

resource "aws_cloudwatch_event_rule" "main" {

name = "${var.name}-${var.environment}"

description = var.description event_pattern = <<PATTERN

{

"source": [

"aws.codepipeline"

],

"detail-type": [

"CodePipeline Pipeline Execution State Change"

],

"resources": [

"${var.codepipeline_arn}"

],

"detail": {

"pipeline": [

"${var.codepipeline_name}"

],

"state": [

"RESUMED",

"FAILED",

"CANCELED",

"SUCCEEDED",

"SUPERSEDED",

"STARTED"

]

}

}

PATTERN

} resource "aws_cloudwatch_event_target" "main" {

rule = aws_cloudwatch_event_rule.main.name

target_id = var.targetId

arn = var.resource_arn

input_transformer {

input_template = <<DOC

{

"pipeline": <pipeline>,

"state": <state>

}

DOC

input_paths = {

pipeline: "$.detail.pipeline",

state: "$.detail.state"

}

}

}

codebuild.tf

resource "aws_codebuild_project" "main" {

name = "${var.name}-${var.environment}"

service_role = aws_iam_role.main.arn

build_timeout = "10" artifacts {

type = "CODEPIPELINE"

} environment {

compute_type = "BUILD_GENERAL1_SMALL"

image = var.image

type = "LINUX_CONTAINER"

privileged_mode = true environment_variable {

name = "STAGE_NAME"

value = var.environment

} environment_variable {

name = "DOCKER_ID"

value = var.docker_id

} environment_variable {

name = "DOCKER_PW"

value = var.docker_pw

} environment_variable {

name = "SNYK_AUTH_TOKEN"

value = var.snyk_auth_token

}

} source {

type = "CODEPIPELINE"

buildspec = "buildspec.yml"

}

}

codepipeline.tf

resource "aws_codepipeline" "main" {

name = "${var.name}-${var.environment}"

role_arn = aws_iam_role.main.arn artifact_store {

location = "${aws_s3_bucket.main.bucket}"

type = "S3"

} stage {

name = "Source"

action {

name = "Source"

category = "Source"

owner = "ThirdParty"

provider = "GitHub"

version = "1"

output_artifacts = ["SourceArtifact"] configuration = {

Owner = var.github_org

Repo = var.repository_name

PollForSourceChanges = "true"

Branch = var.branch_name

OAuthToken = var.github_token

}

}

} stage {

name = "Build" action {

name = "Build"

category = "Build"

owner = "AWS"

provider = "CodeBuild"

input_artifacts = ["SourceArtifact"]

output_artifacts = ["BuildArtifact"]

version = "1" configuration = {

ProjectName = var.project_name

PrimarySource = "SourceArtifact"

}

run_order = 2

}

}

}

lambda.tf

data "archive_file" "lambda_zip" {

type = "zip"

source_file = "lambda/function_src/index.js"

output_path = "lambda/function_src/index.zip"

} resource "aws_lambda_function" "main" {

filename = data.archive_file.lambda_zip.output_path

function_name = "${var.function_name}-${var.environment}"

role = aws_iam_role.iam_for_lambda.arn

handler = var.handler

source_code_hash = data.archive_file.lambda_zip.output_base64sha256

runtime = var.runtime

} resource "aws_lambda_permission" "cloudwatch_trigger_lambda" {

statement_id = "cloudwatch-codepipeline-trigger-lambda"

action = "lambda:InvokeFunction"

function_name = aws_lambda_function.main.function_name

principal = "events.amazonaws.com"

source_arn = var.source_arn

}

function_src/index.js

main.tf

AWS Infrastructure Environment

Inside our infra-live folder, we’ll add a sub folder called prod for the environment we’ll be setting up. Inside prod, we add a child terragrunt.hcl file which specifies from where to download the Terraform code (the modules folder), as well as the environment-specific values for the input variables in that Terraform code.

terragrunt.hcl

inputs = {

environment = "prod"

branch_name = "master"

} include {

# The find_in_parent_folders() helper will

# automatically search up the directory tree to find the root terragrunt.hcl and inherit

# the remote_state configuration from it.

path = find_in_parent_folders()

} terraform {

source = "../../infra-modules/cicd" extra_arguments "conditional_vars" {

# built-in function to automatically get the list of

# all commands that accept -var-file and -var arguments

commands = get_terraform_commands_that_need_vars() arguments = [

"-lock-timeout=10m",

"-var", "module=${path_relative_to_include()}"

] required_var_files = [

"${get_parent_terragrunt_dir()}/sensitive.tfvars"

]

}

}

Deploy Infrastructure to AWS

Inside the infra-live/prod folder, run the following commands on the terragrunt.hcl file to initialise, configure the backend, and deploy the infrastructure: