DevOps: A Hands-On Beginner's Guide To Build Your Own CI-CD Pipeline At Home From Scratch
By: Saptadip Sarkar
Tags: Heroku, Docker, Pipeline, Cicd Pipeline, Slack

The feeling of shipping a code is great. But if the code is never executed for users, then it is no more than a (digital) waste product. To prevent this, modern software developers ship functionality to their users in short iterations and small increments.

This is where CI-CD comes into the picture. Continuous Integration and Continuous Deployment or CI-CD Pipeline is a way to ship code in small increments and iterations. In this learning tutorial we will describe how to quickly set up a simple & modern CI-CD Pipeline from scratch just by using few freely available hosted services.

Requirements

Basic knowledge of Python and Docker

and A GitHub account(free). GitHub is a web-based hosting service for version control using Git.

account(free). GitHub is a web-based hosting service for version control using Git. A Travis CI account(free). Travis is a hosted service for Continuous Integration work. It’s free for public GitHub repositories

account(free). Travis is a hosted service for Continuous Integration work. It’s free for public GitHub repositories A Docker Hub account(free). It is a service provided by Docker for finding and sharing container images with other users.

account(free). It is a service provided by Docker for finding and sharing container images with other users. A Heroku account(free). Heroku is a cloud platform like Amazon Web Services(AWS,)to host your application.

account(free). Heroku is a cloud platform like Amazon Web Services(AWS,)to host your application. A Slack account(free). It is a cloud-based set of proprietary team collaboration tools and services.

Setup Overview

The overall workflow for setting up a complete CI-CD pipeline can be divided into the following steps:

Write a Python Web Application. Push the code to GitHub. Setup Travis CI to continuously run automated test. Create a Docker image for the web application. Push Docker image to Docker Hub. Deploy Docker image to Heroku. Notify deployment status on Slack.

Now we will explain in detail how each of the above steps are configured and linked with each other so that we have a fully functional modern CI-CD pipeline.

The complete code is available in GitHub under this link.

Step-1: Write A Web Application In Python

Let’s build a small “Order Placement” web application using Python Flask Framework. Flask is a micro web framework written in Python which is classified as a micro-framework because it does not require particular tools or libraries.

Our application is a very simple one as our main objective is to learn the basics of a CI-CD pipeline, NOT building a feature rich python web application. On the landing page of our web application, when you click on the “Get Started” button, the application displays different types of order count under “Summary” tab, by querying a back-end MYSQL database which is running on Heroku.

Below are the screen shots of different pages of our web application.

As mentioned earlier, the application has been developed using Python Flask Framework standards. Explaining the Python Flask Programming model is beyond the scope of this tutorial; however for the convenience of the beginners, we will briefly explain the important parts of the program.

On your local machine, clone the below GitHub repository:

using the below git clone command:

In case git is not already installed on your system, then use the below command(for Ubuntu) to install the package:

sudo apt-get update

sudo apt-get install git

After the repository is successfully cloned on your system, verify that the below directory structure is present:

d3f3@:~/OrderPlacementApp$ ls -latr

total 24

drwxrwxr-x 7 user root 281 Dec 25 19:55 ..

-rw-r--r-- 1 user root 43 Dec 25 19:55 requirements.txt

-rw-r--r-- 1 user root 1154 Dec 25 19:55 main.py

-rw-r--r-- 1 user root 383 Dec 25 19:55 dbconnect.py

-rw-r--r-- 1 user root 22 Dec 25 19:55 Procfile

-rw-r--r-- 1 user root 472 Dec 25 19:55 Dockerfile

-rw-r--r-- 1 user root 302 Dec 25 19:55 .travis.yml

drwxr-xr-x 2 user root 57 Dec 25 19:55 .travis

drwxr-xr-x 3 user root 111 Dec 25 19:55 .idea

drwxr-xr-x 6 user root 73 Dec 25 19:55 static

drwxr-xr-x 2 user root 64 Dec 25 19:55 templates

drwxr-xr-x 8 user root 163 Dec 25 19:55 .git

drwxr-xr-x 7 user root 189 Dec 25 19:55 .

d3f3@:~/OrderPlacementApp$

Now let’s explore some of the important files. Don’t worry if we skip some of the files from the explanation. We will explain them in the later part of this tutorial.

The main program file is main.py that contains the python flask application code.

that contains the python flask application code. The database connection string is defined inside dbconnect.py file.

file. The HTML templates are defined under:

templates/dashboard.html

templates/header.html

templates/main.html

All the static files like css, images, javascripts etc are stored under:

static/css

static/js

static/images

static/fonts

So our application code is now ready to be committed to ‘your’ GitHub repository.

Step-2: Push Your Code To GitHub

Create an account in GitHub first, if you don’t have one already. Then login to your account on GitHub and create a public repository called “OrderPlacementApp”.

Now it’s time to initialize Git locally and push your code to GitHub:

[OrderPlacementApp] $ git init

[OrderPlacementApp] $ git add *

[OrderPlacementApp] $ git commit -m "Initial commit"

[OrderPlacementApp] $ git remote add origin git@github.com:<YOUR_GITHUB_USERNAME>/OrderPlacementApp.git

[OrderPlacementApp] $ git push -u origin master

If the last command above complains about access rights, make sure you’ve added your SSH key to your GitHub account.

You can enable Slack notification on you GitHub repository to automatically send a notification to your slack channel when a push or commit a new code. The detailed process of how to setup slack notification will be described in Step-7. A sample slack notification message is shown below:

Step-3: Connect Travis CI To Run Automatic Build On Every Commit

Enabling Travis CI to start a build at each Push and Pull request for your GitHub repository consist of TWO parts:

The FIRST step is quite simple. It is just about flipping a switch in your Travis account which is in front of your GitHub ‘OrderPlacementApp’ repository, as shown below:

After login to your Travis account, in case you do not see your GitHub repository listed, then click on the ‘Sync account’ button as shown below:

The LAST step of activating Travis CI is to add a file called “.travis.yml” in the root of your project directory. For our Order Placement Application, this file contains:

sudo: required services:

- docker language: python

python:

- "3.5" script:

- sh .travis/deploy_dockerhub.sh after_success:

- sh .travis/deploy_heroku.sh notifications:

slack: SLACK-USERNAME:SLACK-TOKEN

The above file is written in YML format. So basically whenever you commit your code in GitHub repository, Travic CI automatically detects the change and starts executing the tests that are pre-defined in ‘.travis.yml’ file. In this way you can often discover very quickly if your commit broke something, and fix it before it becomes a problem.

Note that in ‘notification’ section, you have to add your own Slack Username and your Slack channel specific token. We will later explain in Step-7 how to generate this token by using “Travis CI” application, available in Slack App Directory.

Now add the file to Git, then commit and push your changes:

[OrderPlacementApp] $ git add .travis.yml

[OrderPlacementApp] $ git commit -m "Add Travis CI configuration"

[OrderPlacementApp] $ git push

Go to the Travis CI dashboard. After a short amount of time Travis should notice your code changes and start the build/test process.