The Release Manager model
By: Italo Santos
Tags: Logistics, Technology, Sre, Continuous Delivery, Site Reliability Engineer

However, someone else “accidentally” merged another pull request before and didn’t ship it to production (no issues until now), and the previously merged code contained a bug that caused a serious incident.

The first action is to revert the last known pull request, which doesn’t solve the problem and the next actions will be: start calling people and digging into the last merged pull requests as a try to identify the buggy code to do the right revert.

The above scenario highlights a few problems:

No explicit recovery point in case of failure

No release or changelog mechanism

Lack of protection for human error

Error-prone & Miscommunication issues

The schedule

Deploy calendar notification on Slack channel

As an attempt to better organize deploys, a shared calendar was created with scheduled deployment windows where anyone from the engineering team could simply pick a free time to deploy their code to production

The scheduled windows were divided into 4 windows:

Morning : 7:00 AM to 8:00 AM BRT

: 7:00 AM to 8:00 AM BRT Lunch : 1:00 PM to 2:00 PM BRT

: 1:00 PM to 2:00 PM BRT Late Afternoon : 6:00 PM to 7:00 PM BRT

: 6:00 PM to 7:00 PM BRT Early Morning: 2:00 AM to 5:00 AM BRT

This addresses the visibility issue but still has a communication overhead and starts to slow down the engineering team since the team starts growing and becomes hard to find a spot on the deployment windows.

What about Continuous Deployment?

First of all, there’s the somewhat pedantic fact that deployment does not imply release.

But what makes continuous deployment special is deploying every change that passes the automated tests (and optionally a short QA gate) to production.

Continuous deployment might seem extreme and dangerous

Teams usually need some breathing room between development and deploy, especially with database migrations changes when it’s very important to have a clear safe point to rollback in case of failure and ensure that the code and the database are backward compatible and so on. And because of that only a few companies properly run this practice nowadays…

As an attempt to solve the productivity issue we started some tests with what we called “continuous deployment for real”… In the first design, we tried to automate the current deployment strategy since the development cycle already uses branching to split environments.

Continuous deployment draft back 2018

The initial implementation was made with the staging environment, which already had its own branch and was a safe place to fail, and definitely improve the engineering productivity because everyone was able to just push the code to the branch and get it ready into a real-world environment…

But (this world is so cruel) we face some problems, which in fact could be very harmful in the production environment. One of the challenges comes with database migrations, in the first place it wasn’t so hard to automate the execution of the migration if it exists, but to scale the process we’ll also need to create a lot of tooling to safeguard failure scenarios and create a stable recovery point in case of a rollback.

To be Continued…

Using the branching approach in mind, we also tried to use the Gitflow workflow and couple each branch to an infrastructure environment (i.e.: development, staging, and production)

However, the GitFlow is a bit complex which should take too much effort to automate the entire process. Furthermore, we face an organizational issue where we have multiple teams working in several features that can have different release dates and would be very hard to align everyone to make a single release.

Automating GitFlow

Evolving for stability

All previous approaches were driven by speed and productivity, but this philosophy becomes a bottleneck when the engineering team starts to grow which makes that the “deployment queue” becomes bigger and bigger, which doesn’t leave too much time to observe the system in production.

During the 2018’s Black Friday we identified that we need to change the way we ship changes to production because we’ve identified that we were moving so fast that was hurting us, mainly because we were shipping bad code to production and increasing the number of outages.

Loggi’s 2018 Outage Main Reasons

As the company grows the business becomes more critical and less susceptible to failures. That is a very common path for growing companies, the Facebook is an example of that, where Zuckerberg said in 2014 company’s F8 conference, referring to the “Move Fast and Break Things” philosophy

“What we realized over time is that it wasn’t helping us to move faster because we had to slow down to fix these bugs and it wasn’t improving our speed.”

But slowness isn’t good to a Startup, and we put ourselves to think about how can we keep moving fast and stable…

Changing (a bit) the development cycle

Since we’ve dropped the idea to go with the GitFlow + Continuous Deployment approach, mainly for the production environment, we’ve decided to get the best from Feature Branch Workflow and Google’s SRE Release Engineering Branching strategy

The main branch is the single source of truth and code into this branch is considered stable and ready for deploy

And, with that said we keep the development cycle very similar to the Feature Branch where all feature development should take place in a dedicated branch instead and merged back in the main branch

Image by Kasun Siyambalapitiya

Google’s SRE releasing strategy said that projects don’t release directly from the mainline, instead, a new branch is created from the mainline and named with a specific revision version and become read-only.

We’ve dropped the GitFlow strategy because of the huge number of branches coming and going, we decided to use git tag as a version marker which is like a branch but doesn’t change allowing us to keep the atomicity.

Even after all the previous work we still have the organizational issue mentioned before and still an open question:

What is a release for us? And when we should do it?

At the end of 2018’s holidays, we introduce the concept of the daily deploy which was our definition of a release and also when it will happen.

The Red / Black deployment

Since our main goal was to increase the stability we started the year using a more reliable deployment approach that we called “Push do Macunha” in honor of its creator.

The design uses a similar approach of the Canary Deployment but we rename it to “Red / Black” (believe it or not, we already had the blue/green terminology in our platform)

On a daily basis scheduled time, a new tag is automatically generated, and the pipeline is started leaving the code ready to be deployed, waiting to be approved.

Loggi’s “canary-like” deployment overview

The Release Manager

Ben Treynor advocates on your talk about Keys for SRE that the SRE team needs to keep the developer teams into the production loop and share around 5% of the operation work and, during the 2018’s holidays (including Black Friday) the deployment process was being handled by the SRE team which increase the operational load and moreover, transforming the deploy process in a “mystic creature”.

Image from me.me

SRE team doesn’t run the production alone, opposite of that, we believe in a shared responsibility environment that everyone on the engineering team can run Loggi’s main application deployment process and, inspired by Google’s Release Engineer discipline we’ve created the Release Manager.

The main objective of the Release Manager role is to spread the knowledge, the operational load across the engineering team scaling the deployment process in multiple teams.

The Release Manager is responsible for the following actions:

Coordinate the Loggi’s main application deployment process.

Monitor and take the decision to move forward or rollback the code.

In case of a rollback, it will be in charge of identifying and reverting the offending PR.

We encourage everyone to be part of the rotation, but there is no obligation to be part of the process and, with that said we’ve also created a Release Manager Rotation, which is a list with the names of everyone which applied to be a release manager.

The first version of the Release Manager rotation sheet

The first list born with 15 names and today we have more than 40 release managers

The deployment process isn’t linear

Even with the process running for a long time we realized that the new release manager’s confidence was low, mostly because of the number of decisions which need to be taken during the deployment mainly in failure cases

Deployment workflow overview

To reach the 40+ release managers we have today, we needed to address this issue in some way, which is also an issue for new SRE members.

So one of the new SRE members that were recently joined the team had attended Loggi’s Xboarding, and also the SRE’s, realized that it could use the same methodology to give more context to new release managers in a more practical way, and the Release Manager Onboarding was born.

The process was born to be self-regulated creating a continuous learning environment and for that, we’ve created a classification list based o the previous experience as a release manager (RM for short)

Full RM — The experienced release manager that has already done several deploys and is able to “spread the word” to the other members.

Assistant RM — A less experienced release manager which already attended some deploys as shadow and has absorbed sufficient context to run an entire deploy by themselves paring with a Full RM.

Shadow RM — Since the process born to “spread the word” and transfer the knowledge across the engineering team, anyone can be part of the deploy process as a Shadow (another inspiration from our hiring process), which is an observer.

The face of the Continous Learning approach

Everything is done through a conference call allowing everyone to participate and the Full RM should follow the normal flow of the deployment, but the difference is that it will share its screen with everyone to share their knowledge and experience

References