The Three Ways to DevOps paradise
By: Frederik Hamann
Tags: Transformation, Software Development, Organizational Culture, DevOps, Agile Methodology

1) The fundamental idea of DevOps

Historically software developers (from here on Devs) and IT operations (from here on Ops) have always had contrary goals. While Devs tries to respond to changes in the market by deploying features in production as quickly as possible, Ops is responsible for providing stable, secure, reliable software and is not inclined to introduce anything that could jeopardize this burden.

As a result, Dev and Ops often act as adversaries, and testing and info-security activities are performed only at the end of projects. This conflict leads to many unpleasant outcomes: You will have extremely long lead times, impaired quality of work, deploying the functional changes to your production environment are chaotic, your employees are frustrated, and your customers are unhappy.

The idea of DevOps is to resolve this conflict. Devs, Quality Assurance (QA), Ops, and Infosec, should closely collaborate for the organization’s success. Therefore, organizations should break down barriers between the separate departments and integrate QA, Ops, and Infosec expertise into their software delivery teams. In the long run, the spotlight should shift from merely focusing on implementing user features to improving workflow and making IT more productive overall. The goal is to create a safe work system with usually small teams that quickly and self-sufficiently develop, test, and deploy code.

Thus you can ensure high Devs and Ops productivity, short lead times, increased employee and customer satisfaction, and win (back) the marketplace.

History digression:

The basic idea of DevOps is not groundbreaking. DevOps is based on Lean Manufacturing, Toyota Kata, and the logical successor of the Agile movement.

Lean Manufacturing says that the best predictor of quality, customer satisfaction, and employee happiness is lead time.

Toyota Kata explains that it’s all about the improvement of daily work.

And (as you all know) Agile focuses on delivering software frequently within a couple of weeks, emphasizing shorter batch sizes and incremental work.

So based on the fact that pretty much every single tech-related company holds an infinite amount of expertise on agile development (or do you know any firm that is not super agile these days?), every other organization should be able to understand DevOps principles and put them in place everywhere.

And yet, this is not the case. This is probably due to the effortful agile transformations that everybody has gone through over the last years, leaving little to no energy to adapt or further develop. Or perhaps because some companies have not yet fully grasped the concepts of DevOps.

In the following paragraphs, we’re going to clear with the misconception that DevOps only means the introduction of new cloud-based deployment pipeline tools and talk about the most frequent ambiguities you can see in some organizations’ execution of DevOps.

2) The Three Ways of DevOps — and where organizations are inconsistent in their execution

The Three Ways (Source: Gene Kim “The Three Ways: The Principles Underpinning DevOps”)

To resolve the conflict between Devs and Ops and improve your organization’s productivity, follow the Three Ways: Fast flow of information, immediate feedback, and continuous improvement.

#1 Fast flow of information

The idea of the fast flow is relatively self-explanatory. It is about achieving a fast-flow ‘downstream’ which means from Devs to Ops to Customer. There are specific measures how you can ensure this:

a) Make work visible. Okay, admittedly, this step is one everybody manages to cross from the list. Kanban and sprint planning boards are in place in almost every organization these days. However, how you use them is more important, which brings us to the next point on the list.

b) Minimize work in progress. If there is a visible pile in one of the Kanban board lines — don’t just ignore it, add more cards or start something new but rather examine as a team what the reason for the bottleneck problem is.

This happens relatively often when organizational barriers between Devs and Ops are only broken on paper and project leads reward acting busy more than solving issues.

c) Eliminate waste. This can either be partially done work and goes together with task switching, different processes for documentation that nobody uses, and working on extra features.

If external influence drives your project direction too often (e.g., “the super important next steerco”) and you always have the “there is something more important now”-mindset, this is bound to happen.

d) Minimize batch size. After you have just piled up waste in your process, you might feel inclined to increase the batch size “because the project sponsor really needs this feature next week.” If you follow your please-everyone attitude, you will have issues identifying and clearing defects at a later stage.

e) Reduce the number of handoffs. Whenever some information passes, you need communication: Requesting, specifying, prioritizing, scheduling, solving conflicts, verifying — all done with meetings, emails, calls, cooperation platforms.

You may want to think that you’re covered since you just implemented Jira and Confluence, and thus, coordination between your Devs and Ops is easy. But you may want to ask yourself two things.

First: How many tickets and meetings do you need to perform a release? And secondly: How often do either Devs or Ops receive tickets for which they either didn’t know the context nor the urgency to work on them appropriately. This is a classic pitfall if you have too many handoffs.

f) Continually identify and elevate your constraints. These can be very diverse. Either very technical such as long waiting time for environment creation takes or code deployments due to a lot of manual steps, or governance oriented, such as decision committees that prevent one from making changes without maximally complicated approval processes and meetings. The former does usually not apply to those companies that have already made steps in the right direction. However, the latter is an issue that you can find in almost every organization led by some kind of misguided control freak of some sort.

#2 Constant flow of feedback

While you just learned how to achieve fast flow ‘downstream’, the Second Way pursues the constant flow of information ‘upstream’ from Ops to Devs to ensure learning in the company-internal delivery process with several activities:

a) Swarm to solve problems. This one might sound a bit dramatic, but you might want to stop both Devs and Ops from working when something unexpected happens during your delivery (e.g., a change breaks the continuous build or test processes). Swarming enables learning, and if you don’t do it right away, reconstructing the problem becomes an issue. You will insert new work in your delivery process, which will eventually pile up.

You probably know the situation in which external pressure forces you to do some workarounds, and you promise yourself and others to make up for it when you have more time — it’s not going to happen. Technical debt will accumulate.

b) Keep pushing quality to the source. The effectiveness of approval processes declines as you further push the approval away from where you perform work. Sounds familiar? Yes, it is very similar to what you just learned in 1f).

Ask yourself, how should your project sponsor assess a Devs or Ops situation without adequate knowledge of the work or potential implications? It’s only a stamp given out for the sake of feeling important.

Also, please refrain from having Devs create vast amounts of documentation due to one or two negative incidents in the process. Devs should develop and not write novels, and if you are honest to yourself — nobody will read more than 20% of all Confluence pages you created.

Instead, you need to enable everybody to find and fix problems in their area of control as part of their daily work, pushing responsibility back to those who perform the job.

c) Optimize for downstream work centers. A part of learning is to design your software not exclusively for features but also to deliver on NFR (e.g., performance, stability, testability) in your early development steps. This will help you to avoid building up technical debt, and Ops will thank you for it.

#3 Continuous experimentation and improvement

The First Way and the Second Way ensure a fast flow of information from left to right and back from right to left. The Third Way emphasizes the necessity for continuous improvement of your internal processes and externally directed experimentation. You can do this the following way:

a) Promote a culture of fear-free evaluation. Developers will eventually make mistakes, but you’re on the wrong path if your first reaction is to eliminate these individuals from your project or organization. You will ruin team spirit, promote fear rather than mindfulness in people who do safety-critical work, impede safety investigations and make organizations more bureaucratic — with approval meetings and obsolete documentation.

b) Institutionalize the improvement of daily work. Reserve a fixed amount of time to work on technical debt, fix defects, and refactor problematic code and environments. Sounds obvious, but when has that ever happened in your projects?

c) Globalize local improvements. You need to regularly conduct peer reviews in the form of actual blameless post-mortem meetings. I’m sure most people would say that they already do this, but think about which meetings are usually the first to be canceled when time is scarce?

d) Experiment with the customer in short iterations. Feedback should not only come from within the company. The goal is to shorten development cycles, so you can more quickly release features (e.g. with A/B testing) for your customers and react upon their feedback with software improvements.

3) What you should bear in mind when you introduce DevOps in your organization

Once you deeply understood the Three Ways, you want to avoid all these pitfalls mentioned above and misconceptions and generate some quick wins for your business. So, where to start your DevOps transformation?

a) Identify your first DevOps value stream. Choose a value stream that currently doesn’t meet your business expectations — I’m sure there will be at least one.

Before you think about acquiring new tools and focusing on the technical aspects of a DevOps transformation (e.g., building a CI/CD pipeline — which I will discuss in the next blog post), you want to understand how to deliver value to the customer. Who is responsible for what work and which steps to take to improve flow. You focus on where work is stuck in the process (due to change approval processes and so on) and where significant rework is generated.

You will be surprised. The issues of processing time with one important application that your business department is complaining about for months may perhaps be originated in responsibility issues and not technical performance.

b) Create an accountable transformation team. This is the interesting part of your DevOps journey. Now you can start breaking up the silo thinking between Devs and Ops. You will select a group of employees who you will hold accountable for achieving a clearly defined, measurable system-level result (e.g. reduce the deployment lead time from ‘code committed into version control’ to ‘running in production’ by 50%) or business outcome (e.g., provide faster on-demand releases to respond faster to customer feedback).

You should organize this group with flat hierarchies, composed of multiple, cross-functional disciplines (e.g., Product Owner, Devs, Ops, QA, Infosec). They should be independent and able to design and run user experiments, build and deliver new features, deploy and run their code in production, and fix any defects without manual dependencies on other teams (“You build it, you run it”). If each team member has the potential to become a generalist, you break down ‘siloization’ and reduce the number of handoffs in your organization.

c) Fund services and products instead of projects. Instead of project-based funding, which only measures the achievement of time, scope, and budget, product-based funding values business outcomes such as revenue and customer lifetime value compared to the amount of time required or lines of code needed.

This will give you the baseline to measure the success of your DevOps transformation and ideally use this as an argument to further implement in your organization.

d) Give space for cultural development. This piece of advice may be the most important one. Everything you have read before about self-sufficient DevOps teams with responsibilites and accountabilities for their own business results only works out if you have a leadership in place which supports learning and absolutely stops micromanagement. Only if you can experiment often, fail often, blamelessly reflect on your mistakes, you will be able to constantly improve and eventually delivery the business value in the fast-paced manner you expect.