Go & GitHub Workflows: Automating Database-Dependent Application Tests
By: Blaine Moser
Tags: Golang, Github, MySQL, Go, Github Actions

Recently I developed a Go application¹ with a test suite that depends on MySQL to run.

This article demonstrates how the test suite can be run during a GitHub Actions Workflow.

Automating the Docker build-and-push process, while ensuring that the image is only pushed to Docker Hub if the tests pass adds efficiency to my deployment pipeline.

The contents of the Dockerfile, some shell scripts and the Workflow’s configuration (defined in a file called main.yml ) will be the focus here.

I am assuming that the reader is familiar with the basics of both Docker and GitHub Actions.

That said, I am somewhat of a beginner to GitHub Actions and would appreciate feedback from more experienced individuals if these are not best practices.

The Dockerfile

Step one of the Dockerfile instructs the build to pull the latest Golang image and defines some build arguments that set environment variables.

FROM golang:latest as go LABEL author=”Joe Soap <joesoap@somedomain.com>” ARG DB_HOST

ENV DB_HOST=$DB_HOST

ARG DB_PORT

ENV DB_PORT=$DB_PORT

ARG DB_USERNAME

ENV DB_USERNAME=$DB_USERNAME

ARG DB_PASSWORD

ENV DB_PASSWORD=$DB_PASSWORD

ARG DB_DATABASE

ENV DB_DATABASE=$DB_DATABASE

ARG DB_CONNECTION

ENV DB_CONNECTION=$DB_CONNECTION

Next, the build is instructed to create a project directory, install the Revel framework² there, then point the working directory to the project directory and copy the contents of the project thereto.

RUN mkdir $GOPATH/github.com

RUN mkdir $GOPATH/github.com/…

RUN mkdir $GOPATH/github.com/…/…

RUN go get github.com/revel/revel

RUN go get github.com/revel/cmd/revel

RUN export PATH=”$PATH:$GOPATH/bin”

WORKDIR $GOPATH/github.com/…/…

COPY . .

This is followed by an instruction to install the project’s Go dependencies:

RUN go list -m -u all

…and an instruction to grant execution permission to two shell scripts in the project’s root:

RUN chmod +x ./start.sh

RUN chmod +x ./test.sh

The final instruction runs the test.sh script. The image build will complete if the tests pass (more on this later):

RUN ./test.sh

EXPOSE 9000

CMD [“./start.sh”]

The Shell Scripts

The two shell scripts here are start.sh and test.sh .

At their outset, both scripts execute commands that create the app’s schema (if it does not exist already) and run any outstanding database migrations to bring the schema up to date.

Thereafter, start.sh will start the API service, whereas test.sh will run the application tests. test.sh is only run during the build process (per the instruction in the Dockerfile RUN ./test.sh ). start.sh is run when the container itself is run ( CMD ["./start.sh"] ).

During runtime the app’s container accesses the host OS’ MySQL service over the host’s network (more on this later).³ The tests therefore need to mirror this behaviour.

The contents of test.sh are:

#!/bin/bash echo “executing test shell”

go run command/main.go make-database

go run command/main.go migrate up echo “running tests”

if revel test -a . -v | grep -q ‘All Tests Passed.’; then

echo “All Tests Passed.”

else raise error “API tests failed”

fi

The first two commands ( ...make-database and ...migrate up ) create and/or update the database. When executed during the build, a fresh schema will always be created and brought up to date.

The third and final command runs the application tests ( revel test -a . -v ) and evaluates the outcome by raising an error if the test command’s output is not “All Tests Passed.” In this case, the build process will fail and the latest image on Docker Hub will not be replaced.

One can see the tests’ output as they are executed during the Workflow:

Excerpt from a Workflow’s output

The main.yml

For brevity, only sections of the main.yml file that are necessary to expose the host’s network to the build will be discussed in detail.

First, main.yml defines the Workflow’s triggers:

name: CI to Docker Hub

on:

push:

branches: [ master ]

pull_request:

branches: [ master ]

workflow_dispatch:

Next, it defines the job that the Workflow will be performing (in this case there is only one — build ). It also specifies the host OS in which the build will occur (the latest Ubuntu), using the property runs-on :

jobs:

build:

runs-on: ubuntu-latest

ubuntu-latest has MySQL installed by default, and build ’s first and second steps are (respectively) to start the service:

steps:

- name: Set up MySQL

run: |

sudo /etc/init.d/mysql start

…and then check that it is active:

- name: Check MySQL Connection

run: |

mysql --version

sudo apt-get install -y mysql-client

mysql --host 127.0.0.1 --port 3306 -uroot -proot -e "SHOW DATABASES"

The password being used here ( -proot ) is “root” because this is the root user’s default MySQL password on ubuntu-latest .

After the MySQL service is started and checked, the job then checks out the repository and logs in to Docker Hub using GitHub secrets⁴.

- name: Check Out Repo

uses: actions/checkout@v2 - name: Login to Docker Hub

uses: docker/login-action@v1

with:

username: ${{ secrets.- }}

password: ${{ secrets.- }}

The following step uses Docker Buildx, which will build the Docker image.⁵

- name: Set up Docker Buildx

id: buildx

uses: docker/setup-buildx-action@v1

with:

driver-opts: network=host



It is important to specify the driver option network=host here as this will expose the host OS’ network (and in turn, its MySQL service) to the build.

The following step is responsible for building the Docker image and pushing it to Docker Hub under the latest tag. It does this using the Docker build and push action:

- name: Build and push

id: docker_build

uses: docker/build-push-action@v2

with:

context: ./

file: ./Dockerfile

network: host

allow: network.host

build-args: |

DB_HOST=localhost

DB_PORT=3306

DB_PASSWORD=root

DB_USERNAME=root

DB_CONNECTION=mysql

push: true

tags: ${{ secrets.- }}/someproject:latest

cache-from: type=local,src=/tmp/.buildx-cache

cache-to: type=local,dest=/tmp/.buildx-cache

It is important to specify network:host and allow: network.host under the with: property here.

One should also take note of the build arguments ( build-args: ) here. These are defined in the Dockerfile (see above), and avail the host OS’ MySQL credentials to the image as environment variables ( localhost is the host OS’ network. MySQL will be running on the default port 3306 in ubuntu-latest ).

When the test.sh script runs during this step (see above), the build will not complete if any of the tests fail.

The final two steps echo the image digest to the terminal and run layer-caching⁶ (respectively):

- name: Image digest

run: echo ${{ steps.docker_build.outputs.digest }}



- name: Cache Docker layers

uses: actions/cache@v2

with:

path: /tmp/.buildx-cache

key: ${{ runner.os }}-buildx-${{ github.sha }}

restore-keys: |

${{ runner.os }}-buildx-

Here is the main.yml file in its entirety:

name: CI to Docker Hub

on:

push:

branches: [ master ]

pull_request:

branches: [ master ]

workflow_dispatch:

jobs:

build:

runs-on: ubuntu-latest

steps:

- name: Set up MySQL

run: |

sudo /etc/init.d/mysql start - name: Check MySql Connection

run: |

mysql --version

sudo apt-get install -y mysql-client

mysql --host 127.0.0.1 --port 3306 -uroot -proot -e "SHOW DATABASES" - name: Check Out Repo

uses: actions/checkout@v2 - name: Login to Docker Hub

uses: docker/login-action@v1

with:

username: ${{ secrets.- }}

password: ${{ secrets.- }}



- name: Set up Docker Buildx

id: buildx

uses: docker/setup-buildx-action@v1

with:

driver-opts: network=host - name: Build and push

id: docker_build

uses: docker/build-push-action@v2

with:

context: ./

file: ./Dockerfile

network: host

allow: network.host

build-args: |

DB_USERNAME=root

DB_PASSWORD=root

DB_PORT=3306

DB_HOST=localhost

DB_CONNECTION=mysql

push: true

tags: ${{ secrets.- }}/someproject:latest

cache-from: type=local,src=/tmp/.buildx-cache

cache-to: type=local,dest=/tmp/.buildx-cache



- name: Image digest

run: echo ${{ steps.docker_build.outputs.digest }} - name: Cache Docker layers

uses: actions/cache@v2

with:

path: /tmp/.buildx-cache

key: ${{ runner.os }}-buildx-${{ github.sha }}

restore-keys: |

${{ runner.os }}-buildx-

Recap

If you need to make a host OS’ network available to the Docker build process during a GitHub Workflow, this is one way to do it.

One can also run a shell script during a Docker build. The script can create an up-to-date schema (by running whatever database versioning commands your application uses), as well as run any application tests that depend on that schema.

To do this one must ensure that the host’s network is exposed to the build process. This is achieved through:

use of appropriate build arguments in the Dockerfile; configuring the GitHub Workflow so that the host network is exposed to the build; and, declaring build arguments containing the host OS’ MySQL credentials and setting them at the build step of the Workflow via the build-args property.

Thank you for reading this!