Straight to Prod
By: Simon Gerber
Tags: Software Development, Programming, Continuous Delivery, Continuous Integration, Software Engineering

Straight to Prod

All aboard the production express! (Source: Fikri Rasyid on Unsplash)

Branching considered harmful?

When I was a kid, we didn’t have branches. Well — we did, sort of, but you sat them in the corner and tried to pretend they didn’t exist. They were heavy, unpleasant things. Expensive to create, difficult to maintain, impossible to merge.

When Git came along with its lightweight branching and distributed-ness, it solved many problems — and created many more. We shape our tools, and then our tools shape us.

We do what we make easy to do. Once branching and — more importantly — merging was quick and painless, we started doing it more often. Feature-branches and pull-requests entered the common vernacular. Gitflow was created. I look at those Gitflow diagrams and think, I could build a treehouse in those branches! But for many people this was an improvement. And if I’m honest with myself, I was one of them.

If you’ll indulge me for a moment, let’s gather ‘round the story tree before we proceed with the article proper. When I was an eager young software engineer still finishing my degree, I joined a long-running project just in time to be part of the Great CVS to SVN migration. This was too long ago for me to remember anything much about using CVS, but I had some involvement with the “release process” as it was rewritten for SVN.

It was a strange beast, this release process. It tagged every build, which was fine. But hotfixing involved going back to the tag and committing more stuff against that tag. This took shameless advantage of the fact that in SVN the main difference between a tag and a branch is wishful thinking.

If we had to cherry-pick a commit to the hotfix branch (which was actually a tag), we… I don’t know if I can admit this. We… just wrote the code again. By hand. From memory. My great contribution to improving this process was to discover the “compare with trunk” option in Eclipse and teach the team how to use it.

We also did not have unit tests or even nightly builds, let alone a CI server. We did not have feature flags. We developed on trunk. If code wasn’t ready to commit… we didn’t commit it. It sat, on our machines, day after day, week after week. Sometimes we copied our working directories to a network drive as insurance against hard-drive failure. Other times we threw caution to the winds and just let it sit there.

Then, when we were close to committing, the race was on! Someone else might also be close to committing. The challenge was to get your changes in first and leave the others with the excruciating merge. Did the rush to push first result in oversights and stupid bugs? Yes. Yes it did.

Branches are better than this, right? Well, maybe.

Everything is contextual. I don’t know your context. I don’t know you, your company, your colleagues or how you work. But I’m going to make an assertion anyway. A big, bold, potentially scary statement.

Branches have led you astray. Gitflow is not your friend.

You should be committing straight to master.

Does the thought frighten you? Why? This is not a rhetorical or facetious question. What are you afraid of? What have those eyes of yours seen? Please think about this seriously.

When I suggest committing to master I am often meet with resistance. To be fair, it’s usually justified based on past experience. Including my own. Almost all of us have seen, at some point in our careers, trunk based development gone terribly, horribly wrong. Many of the objections I collect fall into one of the buckets below:

It might break something It’s not ready We like keeping ‘master’ clean and stable How can we allow ‘juniors’ to push code straight in without review? More than a handful of developers working on the same codebase at the same time is going to make a mess. “It won’t work here” — there’s something special about us which means CI/CD is not a good fit.

When I discuss removing feature branches with my fellow engineers, I am talking to intelligent, well-reasoned people. They have seen branches work, and trunk based development fail. If going back to the life without branches is dangerous and hard, why would we bother?

In my experience, and despite my initial experience, I have three main reasons it is worth pursuing:

Continuous Integration [CI] and Continuous Delivery/Deployment [CD] have well documented, commonly reported benefits. Many of those benefits are predicated on making the design -> code -> test -> release -> measure loop as short as possible. The longer you delay pushing to master, the longer you delay some part of the CI/CD feedback loop. A common reaction at this point is to assert that you can merge master into your branch as often as you like! This is true, but your code is still hidden from others. Don’t hide your code. It’s rude. Push to master. It turns out that the mental muscles you need to exercise when thinking about how to push to master without breaking everything are the same mental muscles you need to evolve a distributed system. The more practiced you become at trunk-based-development [TBD], the better you become at evolving APIs and data schemas. TBD helps you win at micro-services. And vice-versa. It’s a nifty little feedback loop that enhances the code/release/measure loop we’re also building upon. Feature-toggles, which will be used to safely manage TBD, can decouple “deploy code” from “release feature”. This has the fantastic benefit of letting a feature be turned on ‘when the business is ready’. No longer does everyone have to be aligned around release cycles. This in turn can go a long way to eliminating the ‘release crunch’ which has an all-round positive effect on morale, efficiency and quality.

Deny yourself trunk and you deny yourself this ecosystem of benefits. Long-lived feature branches aren’t quite as bad as bonsai kittens, but they’re close.

With respect to the objections listed above, I have a personal belief that they all boil down to, “Not sure how this works, so it probably doesn’t.” It’s easy to be led down this path. Much advice about implementing CD and TBD can be summarised as “just use feature flags!” Easy to say — hard to master. You can find some good, even great, information online about working with feature-flags. But while feature flags are cool and all — ‘death by feature flags’ is a thing. Why create unnecessary complexity?

In this article I will take a deeper dive into some specific techniques for making every commit production ready.

Additions are safe, removals require thought

Am I showing my age when I talk about battles with RMI and Java serialisation? Maybe… But no doubt there are those of you out there who’ll one-up me with your CORBA car-crashes and DCOM disasters. Then we had pickling and WSDL and DTD, now we have REST and thrift and Avro and gRPC and protobuf and GraphQL, but it’s all the same thing. If you’ve ever wrestled with backwards compatibility across APIs, binary formats or schema versions, you have an advantage. You need to wear the same “thinking hat” for trunk-based-development.

Adding new fields to a class/record/‘data transfer object’ [DTO] is backwards compatible. Removing them is not, unless we can be 100% certain they’re unused. Modifying a field is a ‘remove then add’ in one operation — which means the ‘remove’ rules apply.

One key insight towards making small commits each individually production safe is to realise there’s no difference between worrying about compatibility with something you persisted to disk earlier, or an older API, and compatibility with code from an ‘incomplete’ feature.

Additions are safe. In place modifications are unsafe. Removal is only safe once you can guarantee the subject is unused. Following on from that, the following basic building blocks are safe:

Adding an unused column to a table

Adding an unused field to a DTO

Adding an unused parameter to a method

Adding an unused method to a class

Adding an unused class to a module

Adding an unused module to a repository

Note that ‘unused’ means unused by production code. Unit and integration tests should give us confidence that the code being added is going to work.

We can now use these building blocks to lay out some common playbooks for adding new features. The following scenarios will assume a generic stack with a statically typed language, RPC based incoming end-points, a domain logic layer, an ORM and a relational storage layer. The scenarios chosen all have a front-end/back-end split. This is because I expect this boundary to be familiar to the majority of readers. The ‘plays’ themselves are equally applicable across or within any architectural boundary, for example between two back-end services in a micro-services architecture.