The Transition to Trunk-Based Development
By: Bekzod Babamuradov
Tags: Git, Trunk Based Development, Software Development, Gitflow, DevOps

In the early days of Super Dispatch, the Backend team practiced Git Flow to manage the development process.

We were about three people managing the backend, and we didn’t have a Continuous Integration process. We usually merged and deployed large chunks of code every 3–4 weeks when we fully built a new functionality.

It was okay for us at the beginning. Later, we figured out that for a startup that had to move fast to survive and grow, it was not a good enough speed. We weren’t able to learn quickly because of the long feedback loops.

Challenges When Practicing Git Flow

When working with long-lived feature branches that Git Flow allows, we had several challenges that ultimately slowed us down and reduced the quality of the software we built.

Code reviews were painful because we would usually merge a large chunk of code to the trunk (or the “master” branch in Git terminology).

Since each Git Pull Request consisted of many lines of changes, it became hard to review the code. It also resulted in long back-and-forth discussions. Over time, the team stopped reviewing pull requests altogether. Everyone only cared about their work and did not pay much attention to what other people were working on. The codebase became inconsistent, and knowledge-silos started emerging because everyone owned only some part of the codebase.

Acceptance and regression testing would be conducted on the feature branch that a developer was working on.

Once a developer finished the implementation of a feature in their long-lived feature branch, it would get deployed to a server, and a QA engineer would conduct the manual testing of the functionality. This approach sometimes introduced unexpected bugs into the system after the tested feature branch was merged into the “master” branch. Other times, it resulted in overhead because we had to resolve conflicts when merging the feature branch to the “master” branch. We couldn’t be sure that the merge process did not introduce any bugs in the system.

Refactoring the code was hard because of long-lived branches.

Several times engineers initiated the refactoring of some of the core components, but either they failed to finish it, or it took them a long time. Looking back, we think it is partially because one engineer would spend several days refactoring a component to which another engineer was also making changes in order to add new functionality. Since changes to the code weren’t being merged to the “master” branch daily, it caused a lot of merge conflicts, which delayed the releases and demotivated the engineers to refactor at all.

The Proposal

At some point, we knew that something had to change, but we couldn’t figure out what until we discovered Trunk-based development.

I want to say that from then on, we quickly switched to trunk-based development, and it solved all of these challenges, but that would be a lie.

The team was scared when the Trunk-based development approach was proposed. There was a strong pushback. Most of the team firmly believed that:

It was impossible to merge changes to the “master” branch and release the “master” branch to the production environment on a daily basis.

The code that hadn’t been manually tested should not be merged into the “master” branch.

The code that hadn’t been manually tested should not be deployed to the Production environment.

Without Git Flow, we would lose the ability to do hotfixes.

With similar beliefs, the switch to Trunk-based development would require a significant mindset shift that we couldn’t make right away. Although we understood that trunk-based development was not a silver bullet and it alone would not make our team more productive, what we learned from the book The DevOps Handbook seemed promising:

trunk-based development predicts higher throughput and better stability, and even higher job satisfaction and lower rates of burnout.

The Transition to the Trunk-Based Development

Despite a strong pushback at the beginning, we decided to take baby steps. We agreed that over the next couple of months, we would:

Imagine that the “develop” branch that we used to deploy to our Staging environment was a trunk that we should strive to always keep in a releasable state.

Write more automated tests to increase the test coverage of the codebase.

Learn to split tasks into small and releasable batches.

Make sure that the feature branches that we created were always short-lived.

Learn and be comfortable with Branch by Abstraction and Feature flags.

As we experimented more with trunk-based development, we identified our knowledge gaps and the lack of continuous integration in our development process. In the beginning, it was hard to follow the new conventions. Over time, we increased our test coverage. The commitment to practice trunk-based development forced us to learn to split tasks in the backlog into smaller tasks that were releasable. We learned how to use feature flags and how to branch by abstraction when needed.

Once we felt comfortable and discovered that our fears weren’t real, we removed the “develop” branch and updated our continuous integration pipeline that enabled continuous deployment.

The Outcome

Now, we use only the “master” branch to build a Docker image that we deploy to the Staging environment and then deploy the same Docker image to the Production environment. Deployments to the Production environment are done regularly through self-service (click of a button), either by a developer or a QA engineer.

Since we learned to split tasks in the backlog into smaller and releasable tasks, our pull requests became smaller. Code reviews are now done quickly and with more accuracy, which results in more frequent deployments, fewer merge conflicts, and fewer bugs. When our builds break because of issues, we get fast feedback from our pipeline, and we fix the issues immediately. We are also able to refactor our codebase more aggressively now as we learn more about the business and as we adapt our architecture to our scale.

The transition to the Trunk-based development required a significant mindset shift, and it took us some time to be comfortable with it. We thought it would never work. However, we persisted, and now we can’t imagine ever going back.