Using GitHub Actions to speed up CI/CD in data science projects
By: Luis Henrique Zanandréa Paese
Tags: Docker, Data Science, Github, Github Actions

Inserting Data Science in the CI/CD process

The Data Science field is full of different frameworks, dependencies, and different languages that can be used according to the need and abilities of the data scientist, but a common truth amongst them is that they all have the possibility of being encapsulated by a containerization process, helping to ensure the reproducibility of the project.

With that in mind, the example used by me to deploy the automation tool of GitHub Actions involves the development of a web application using R’s Shiny library. Nevertheless, the same workflow implementation could be used to deploy APIs developed using Python’s FastAPI, i.e., or any other framework that can be encapsulated in a Docker container.

The project can be found here: paeselhz/ghActionsDockerShiny. I won’t be entering in details of the development of the application, because the example used by me is relatively simple and have no elaborate development. The focus of this article is the containerization of the project, and the workflow automation to build the image and store it in Docker Hub, making it available for further downloads.

Creating the Dockerfile

For those familiarized with the Dockerfile and its syntax, the execution is the same as expected in a project that will be developed, built, and ran locally with Docker. In it, we declare the base image that will be used for further installation of libraries and dependencies, as well as the configuration of the project, file copy, and other steps that usually can be added to a Dockerfile.

FROM rocker/shiny:4.0.0 RUN apt-get update \

&& apt-get install -y \

libxml2-dev \

libglpk-dev \

&& install2.r \

--error \

dplyr \

shiny \

purrr \

highcharter \

shinyWidgets \

shinycssloaders \

devtools \

xml2 \

igraph \

readr



RUN R -e "devtools::install_github('wilsonfreitas/rbcb')" COPY . /srv/shiny-server RUN chmod -R 777 /srv/shiny-server

This script, which is located in the project root directory, is responsible to gather an image that already has Shiny and its dependencies installed, and the installation of libraries that will be used by the app developed within R.

Creating the Workflow file

To GitHub Actions know which steps need to be taken for the workflow automation, it becomes necessary to create a file within the project that will be located at .github/workflows/main.yml, the file syntax is the same as any YAML file, being easy to code. In case the user does not want to do this process locally and commit the changes, GitHub itself has an online code editor for the creation of the workflow.

In this file are declared a few steps such as the name of the workflow, the triggers that will be used to deploy the workflow execution, and the jobs that it will be responsible for executing. The name and trigger parts of the file are highly customizable, and the user can change it in many ways, moreover, in the part of the job, there are a few steps that are needed for the job to login in Docker Hub, configure BuildX (a tool that will be used to build the image), configure QEMU (a tool that will be allowing multi-platform builds), deploy the built image to Docker Hub, logout and clean the machine to ensure that no processes are still running.

# Setting up a Workflow to work with Github Actions name: ci # Controls to when trigger the GH Action

# Below are configurations to the following triggers:

# - commits at master branch

# - tag commits at the project

# - scheduled to run at 01:00GMT

# The user can also configure triggers at pull requests

# as well as remove branches from triggering GH Actions

on:

push:

branches: [ master ]

tags: [ '*.*.*' ]

schedule:

- cron: '0 1 * * *' # Below there is the job configuration to build the image

# and push it to a DockerHub repository

jobs:

docker:

runs-on: ubuntu-latest

steps:

-

name: Checkout

uses: actions/checkout@v2

-

name: Prepare

id: prep

run: |

DOCKER_IMAGE=<USER_NAME>/<REPOSITORY_NAME>

VERSION=noop

if [ "${{ github.event_name }}" = "schedule" ]; then

VERSION=nightly

elif [[ $GITHUB_REF == refs/tags/* ]]; then

VERSION=${GITHUB_REF#refs/tags/}

elif [[ $GITHUB_REF == refs/heads/* ]]; then

VERSION=$(echo ${GITHUB_REF#refs/heads/} | sed -r 's#/+#-#g')

if [ "${{ github.event.repository.default_branch }}" = "$VERSION" ]; then

VERSION=edge

fi

fi

TAGS="${DOCKER_IMAGE}:${VERSION}"

if [[ $VERSION =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then

MINOR=${VERSION%.*}

MAJOR=${MINOR%.*}

TAGS="$TAGS,${DOCKER_IMAGE}:${MINOR},${DOCKER_IMAGE}:${MAJOR},${DOCKER_IMAGE}:latest"

elif [ "${{ github.event_name }}" = "push" ]; then

TAGS="$TAGS,${DOCKER_IMAGE}:sha-${GITHUB_SHA::8}"

fi

echo ::set-output name=version::${VERSION}

echo ::set-output name=tags::${TAGS}

echo ::set-output name=created::$(date -u +'%Y-%m-%dT%H:%M:%SZ')

-

name: Set up QEMU

uses: docker/setup-qemu-action@v1

-

name: Set up Docker Buildx

uses: docker/setup-buildx-action@v1

-

name: Login to DockerHub

if: github.event_name != 'pull_request'

uses: docker/login-action@v1

with:

username: ${{ secrets.DOCKERHUB_USERNAME }}

password: ${{ secrets.DOCKERHUB_TOKEN }}

-

name: Build and push

id: docker_build

uses: docker/build-push-action@v2

with:

context: .

file: ./Dockerfile

platforms: linux/amd64

push: ${{ github.event_name != 'pull_request' }}

tags: ${{ steps.prep.outputs.tags }}

labels: |

org.opencontainers.image.title=${{ github.event.repository.name }}

org.opencontainers.image.description=${{ github.event.repository.description }}

org.opencontainers.image.url=${{ github.event.repository.html_url }}

org.opencontainers.image.source=${{ github.event.repository.clone_url }}

org.opencontainers.image.version=${{ steps.prep.outputs.version }}

org.opencontainers.image.created=${{ steps.prep.outputs.created }}

org.opencontainers.image.revision=${{ github.sha }}

org.opencontainers.image.licenses=${{ github.event.repository.license.spdx_id }}

The workflow code has almost none external dependencies, given that the creation of the Docker image name and its tags are within this code, however, it needs a pair of Secrets to log in at Docker Hub, in this case, the username used by Docker, and a Token to log in at Docker Hub (which can be generated here). With the username and Token, the user just needs to go at their repository, in the Settings tab and add the token in the Secrets subpage, as seen in the image below:

With these steps, the project should be able to be executed using GitHub Actions to allow the automation of the build, test, and deploy processes.

In the example used by this article, the final image can be found at Docker Hub here, and tested locally by running the command:

`docker run -p 3838:3838 lhzpaese/ghactions_docker_shiny:latest`