State of CI/CD and the omnipresent git flow
By: Mykhailo Gorianskyi
Tags: Continuous Delivery, Gitflow, Continuous Integration

Let’s address the elephant in the room — the commonly seen way of delivering software today.

In most of the teams I worked with, there was a bias to use git flow or git flow-like branching strategies. While this is indeed a very popular choice, it’s not always the best one.

Since its inception in 2010 git flow still is quite popular and actively being used in many companies and teams. I encountered it in outsourcing companies, open-source, e-commerce, web agencies, tech departments of otherwise non-tech companies. Even tech consultancy firms would rather suggest that branching strategy. Even for new greenfield projects.

While git flow is thought out and solid, it has its purpose. It’s all about control, separation of concerns, release management. It fits nicely in a project where the trust level is low (security or seniority concerns) or in a project that has an established release cadence.

Speaking about infrequent releases, however, it was established over the years that velocity is one of the driving factors of successful software development teams. Having a high momentum, being able to evolve fast, get fast feedback and pivot early correlates highly with organizational and personal performance. The faster the team is able to accelerate its technical operation, the more performant it is.

However, git flow was not made with performance in mind. Its guardrails and bureaucracy do not put delivery acceleration first.

Git flow and CI/CD

Any development team will inevitably use a delivery pipeline to integrate, build and deploy their code. More often than not, this tool will be referred to as a “CI/CD pipeline”. And there’s a high chance that in an almost idiomatic manner the actual practices of CI (and CD) will be completely or partially neglected or misunderstood.

Let’s dive a bit deeper to understand what issues are associated with applying CI and CD to practice and what does git flow has to do with it.

The lack of CI in CI/CD

Continuous Integration definition states:

Everyone Commits To the Mainline Every Day

What actually tends to happen with git flow is developers start their day by branching off of develop branch for a particular feature request. It’s fairly common for the feature to take more than 1 day to be complete. To make it worse, even if the commits start landing immediately as the work progresses, they still end up in a developer’s personal feature branch. The code is not integrated into the mainline until the feature is complete. It’s not integrated continuously.

Software development is supposed to be an activity heavily shaped by communication. What developers tend to do instead is get sucked in their development bubbles only to reemerge a couple of days later with a solution.

Long living personal feature branches are the product of this. In case a feature branch contains big enough change, they yield larger PRs. Those take longer to review, test, and fix if necessary. As a result code reviews become much less thorough. This queues up the changes to develop, delays releases, and causes conflicts. It’s common to have to rebase a PR because a new commit lands in develop, which resets the approval counter and restarts the review process.

All of this consumes a decent chunk of attention for the sake of balancing the process. Instead, the process should be set up around minimizing context switches and allow for focusing on the code and keeping your eyes on the goal.

Alternative: Work with your PO to ensure that the user stories are as small and atomic as feasible. The workload should be small enough to be completed within a day or less. Try to make your CI pipeline lightweight/performant enough to allow commits to flow as often as needed. By landing commits to trunk often we facilitate the CI principles and keep the code integrated.

Not exactly CD in CI/CD

If you approach an average development team that practices git flow and ask if they can deploy their code right now, most probably they’ll reply with something like:

Yes, of course, we can. We only have to:

Create a release branch Merge release branch back to master and develop Tag master

Which isn’t exactly “right now”.

Another approach that developers take with git flow is creating releases after every merge to develop, so that master is always deployable. This certainly helps, but requires context switching and involves a good deal of manual actions performed on the regular basis.

It should be possible to perform a release with a “click of a button”. And it’s important that developers aren’t the only privileged class to be able to do it.

The goal of continuous delivery is to release software quickly, safely, and in a sustainable way.

Alternative: Consider choosing a more streamlined branching strategy, such as TBD, github flow, gitlab flow, etc. Automate everything. Minimize manual interaction.

Unnecessary complexity

It’s fairly easy to complicate the delivery or even development of any system, by adding a whole new level of complexity: unknowingly or traditionally. Here are some examples you might stumble upon.

Ephemeral infrastructure

It’s not that uncommon to encounter developer teams that prefer to prepare the environment by hand via a UI. Not using the Infrastructure as Code or issuing one-off API requests without any state to rely on. This doesn’t make it easy to quickly reproduce the infrastructure, be it recovery or a new environment. Also, it is a decision that is not always straightforward to change and might involve migration to get the resources under some sort of state without downtime.

You could hear something like — “But we will only need to do it once”. In practice, however, you should multiply this by the number of environments that you would ever have. Also, the infrastructure is rarely amorphous — small updates and fixes are to be expected (again, multiplied by the number of environments). If for whatever reason the infrastructure needs to be re-created, it might take more time and not be done in exactly the same way.

This also contributes to the low Bus factor.

Alternative: Everything as Code. Adopt infra tools (Terraform, CloudFormation, CDK, Pulumi). Prefer an actual shell-script file to a snippet in README.md. A new hire should be able to start working within a couple of hours.

Detached IaC

Some teams prefer to keep their infrastructure configs separate from business logic source code. Be it to avoid an unwanted change, because of strict/siloed responsibilities, the lack of peer trust, or any other.

This requires much more logic in the delivery pipeline to ensure atomicity and sequence in case a feature branch in an App repository depends on a particular feature branch in an Infrastructure repository. The same thing applies in the case of a serverless application — code and infrastructure go hand-in-hand. Handling rollbacks in all dependent repositories is also an issue that needs solving.

Even though the intent was to decouple code from infrastructure, the coupling will still be there. Only this time it’s logical and this logic has to be moved to the delivery pipeline. More moving parts and complexity.

Combine this with git flow and you could imagine the balance that needs to be stricken to deliver a feature, requiring both business logic and infrastructure changes in two separate repositories. Also, consider failure mediation in this scenario.

Alternative: Use git to handle the atomicity. Consider using a mono-repo and keep your XaC and business logic code together.

Env-awareness

Looking at the code you might at times find references and conditions for the type of environment the application might be running in. The code might load different configs files, based on whether the app detects that it runs in production or not. Or there might be a switch statement listing all possible environments and it would perform different actions based on what environment it’s running in. Alternatively, configs might be “grouped” into a batch, and the app gets all the possible configurations and chooses one during run-time. At times you might even find dedicated environment variables specifically for running the application locally.

These are all violations of rule 3 of the 12 factor app. The application should be environment-agnostic.

This makes the application unnecessarily more complex and brittle. The application would need more care to be in sync with the stage names.

Alternative: Use the same set of environment variables for all stages, provide different values. Avoid baking-in environment knowledge.

Low trust

A low level of trust is not just a fintech thing. Missing trust is all around us, in many aspects of the project and team life. Pull requests tend to turn into battlegrounds of code style and nitpicking. Technical solutions tend to be forced by the middle management. Released software tends to be re-tested by hand after successful delivery. Deploying on Fridays tends to be prohibited company-wide. The list goes on. Such work environment is not very enjoyable nor very productive.

Alternative: Numbers and tools carry no intent and are easier to trust. Leave code style reviews to a linter. Leave smoke/acceptance/non-functional testing to an automated test suite. Build quick POCs to demonstrate possible KPI improvements. Prefer weekly “technical debt cleanup” sessions to extra-thorough code reviews.

Considerations

Git flow is just one of the branching strategies. The one that favors scheduled releases.

Choosing a branching strategy should follow an intent. Extrapolate and choose git flow if it matches yours, not because everybody is doing it.

Throwing git flow’s feature branches against a build server does not constitute CI.

It’s not a straightforward task to set up CD process with git flow.

Following CI and CD practices and principles might require certain cultural and toolchain changes, compared to what is considered the norm in your company today.

If it hurts — do it more often. If you are afraid of getting out of comfort and safety of feature branches and/or breaking the mainline — be sure to have a good enough test suite and practice committing to mainline as often as required.

Being nervous/uncertain while releasing means the CI/CD pipeline has room for improvement. The release process should best be as uneventful as possible.