Infrastructure as Code: How we Revamped our Development/Release process to embrace our team scale
By: Kareem Emad
Tags: Infrastructure As Code, DevOps, Automation, Infrastructure, Continuous Delivery

In this article, we show how we handled our Continuous Deployment process in the past, what drove us to change what we have, and what efforts we made to make the process as simple/effective as possible.

The article may give an initial devops taste, but in its essence, the release process affects all teams and how they function on their own or collaborate with other teams, we will show how roles is being shifted just by changing the process.

About us

We at Halan help offering ride-hailing and on demand logistics solutions (installments, Delivery, payments online, E-commerce) for the under-served population by the existing technologies in transportation and other services, you can check out our app here (android / iOS)

Giving the old process its due, the good and the bad

We will start by how we handled our process in the past and how that affected us either positively or negatively.

For my team, we have nearly everything deployed over AWS with GitHub as our version control system, so we need to figure out a way to ship features from our GitHub repos to AWS services.

Version Controlling Deployments

We relied on AWS code pipeline, we set triggers to start a deployment pipeline whenever a new commit is merged into one of the main branches representing the known stages: dev, prod and staging environments.

overview of how deployments are triggered

This flow means that any feature on the master branch is on production, that surely makes merging to master a taboo topic to discuss and not everyone had that permission to merge your PR on master.

Same thing was for other branches but that wasn’t really a problem, since dev and staging envs aren’t really that scary.

Our QC started testing and verifying features when they are merged on the staging, which means they test first on staging env then if verified, the feature will be merged into the master branch. that seems to work fine but let’s look at the downsides of such process.

This allowed us to test and deploy multiple features at the same time on each stage (dev, staging), which is good and bad at the same time because:

That means developers don’t have to wait for each other to gain control over the dev/staging environment.

But it means that features on dev stage could mistakenly cause each other to either not work or work (false positive), and Both of those cases are really bad and might cause code to go on production while it’s not really working.

Diverging Branches, Unnecessary rebasing

As you may have already noticed, we said that in order to deploy on dev env, you need to merge your code on the dev branch. Well that part sadly quickly messed up the timeline of our branch as:

Not all features that has been tested on dev do go into production so we had like ghost commits laying around in our unified dev branch.

An experimentation lab without a doubt, people may discover they have been doing something wrong after a while and decide to drop their code/commits and start a new one, but that still leaves us with their old commits merged on the dev branch.

This kind of behavior produced what we call divergence between branches, to merge a feature branch that has master as its base into dev, you need to resolve conflicts! which is awkwardly weird you literally conflicting with ghosts and resolving things that are never going to be on production anymore, that surely gives you two poor options:

Build a new branch for your feature forked from your original feature branch and resolve conflicts on it then when done testing, merge the original branch into master.

Resolve conflicts on your branch then cherry pick your commits into new branch when merging into master.

With more developers joining our team, things will start to get more ugly and divergence will surely take a hike up.

Revamping the build/deploy process

First thing we had in mind is how to get rid of those divergent branches, we need a new method to safely deliver features to production seamlessly and control how many features are deployed on production without involving source control with deployment business.

A little review on AWS

First we need to identify the main elements that will play a role with us in the process:

Github : source control having repos for every micro-service out there.

: source control having repos for every micro-service out there. Amazon ECR : a Docker image registry that is divided into `repos`, each repo represents a micro-service also and contains the images that has been pushed for service X and can be used or deployed to this service directly to make it work.

: a Docker image registry that is divided into `repos`, each repo represents a micro-service also and contains the images that has been pushed for service X and can be used or deployed to this service directly to make it work. Amazon ECS/EC2/EKS: this is where our micro-services live, along with their environments, ECS is divided into clusters, each cluster holds a number of services each one has task definition that holds info like environment variables, compute specs, network, etc.

So we need to find a way to bind those together to make it easier for us as dev team to deliver features and for QC to control and verify those features.

We figured we could define a middleman service, a service that is responsible for deploying other services. We had two main commands we wanted this service to handle:

Build Command: we want to be able to build an image and push it to ECR from any branch/commit we want from a certain GitHub repo to a certain ECR repo.

we want to be able to build an image and push it to ECR from any branch/commit we want from a certain GitHub repo to a certain ECR repo. Deploy Command: we want to be able to deploy a certain image from a repo on ECR to all clusters holding service X.

Let’s take each of those commands under the microscope and see what each of those involve, we begin with the Build Command:

First of all, we need to clone the repo at the specified commit/branch.

Build a docker image using the Dockerfile in that repo.

Push the built image to ECR repo with a new tag if needed or latest.

While all those steps are certainly doable in our new deploy service, but we found that it’s cumbersome since building docker images consumes a lot of resources and that will limit our new service to a few concurrent deployments or make it so gigantic in resources.

We figured we could instead of doing all this work on the service itself, we could start a new workflow on Github on that repo and pass to it all task details such as which branch to checkout and what ECR repo to push to, when it’s finally done, it will signal the user back using a web-hook. Of-course there are other solutions for that such as Jenkins for better security to make sure your code builds on machines you manage.

That changed the workflow we mentioned above, it now goes as follows:

Receive build details from user.

Trigger a Github action on the specified repo.

action on the specified repo. Github actions will spin a new machine that will notify us back with build status when done, it will also tag the commit with the Jira Card Number you working on.

Second Command is the Depoly command, which is way simpler than the build Command, we here only have to:

Receive Deployment details from user.

Start a new deployment task on each cluster that contians the service and provide the image url in ECR.

deployment workflow

That’s really it for deployment! just a facilitation job between two AWS products, let’s compare this workflow to what we had before and see how our release strategy changed.

Using the new process to deploy your code on any of the stages (prod, dev, staging) does not require any kind of merges , the code is totally independent of the deployment, we could have 20 features ready on the master branch and only choose to deploy 10 of them.

But this also imply that we can only have one feature being tested on the dev branch unless two devs agreed to rebase on each other.

Deployment History and Tagging

This also provided us a better rollback strategy against production issues, Now we have a tagged master branch for each feature/task we had, whenever we face a production issue and want to rollback a feature we can easily navigate back to where this feature started and rollback to the commit before it if possible.

Also that allowed us to merge all working features on master branch, even if they are not going to be deployed , we can still have them on master ready for the next batch of features to be on production when needed, that will surely help our team focus on delivering tasks without worrying about when that task will be merged into master to be deployed, here we merge first then worry about deployments.

Next Steps

We currently working to automate our process better, including environment variables and machine specs, they all will be available as code, whenever we change those specs, that will automatically reflect on our machines on AWS in the respective Environment, no need to hand-code those values each time on AWS.

With this, we will have better control and visibility on all our services configurations, everything in our infrastructure will be in code and one click away to update! also that makes changing our infrastructure independent of the service provider, we could switch from AWS to Google cloud for example and only need to change the commands used by the deploy service to deploy our infrastructure on the new service provider.