Defining DevOps Transformations — Part 1
By: Abn Amro
Tags: Continuous Delivery, Transformation, Tech, Abn Amro Developer, DevOps

Defining DevOps Transformations — Part 1

In this first part of a two-part blog post, Titus Nachbauer, technology enthusiast and DevOps/CICD expert at ABN AMRO, is going to explore what a DevOps transformation is and why you need to work on transforming your software organization. He will show why he thinks Continuous Delivery is at the core of DevOps transformations and why cultural aspects are most important for a successful transformation. In the second part (coming soon), he will look at crucial metrics and how they could support a DevOps transformation.

Picture by Josh Calabrese

Continuous Delivery of Valuable Software

Have you ever wondered if the organization you work for will still exist in ten or twenty years? Is it older or younger than twenty years? Younger than ten? If you are an engineer, you will probably realize that ten years is a very long time in IT. It is almost impossible to predict, what you will be doing in ten years or what technology you will be using. I do predict one constant: we will still have Continuous Delivery as the main principle of what we do as IT engineers.

I am saying the above with confidence because twenty years ago the Agile principles already mentioned the “[…] Continuous Delivery of valuable software”. What’s more, the Lean movement has something very similar as a guiding principle, with the focus on the elimination of waste and continuous improvement. And Lean has been around for more than 30 years.

But more importantly, you should focus on the benefits and reasons to work on Continuous Delivery. Let’s define what Continuous Delivery is:

“Continuous Delivery is the ability to promote small software changes to production quickly and with confidence in a sustainable and repeatable way.”

-(inspired by continuousdelivery.com and many others)

So, what am I saying here? I am emphasizing that small changes are good, and I am saying that you want to have a high degree of confidence when you deliver them. Confidence in this context comes down to this: I know that it will work and as a team of engineers we feel that we can guarantee that if the software were to fail, it would do so gracefully.

Ask yourself in which situation you would feel better as an engineer. Is it when at the end of the day you feel stressed because your code is not working? Or is it when you are pleased that you made a useful change that adds value to the product? And if you are pleased and less stressed, doesn’t that add to the ability to sustain the pace at which you are working?

Remarks

To work at a sustainable pace, you need automation. Get rid of any manual steps in your testing process and your delivery process becomes controlled and repeatable. This is so essential to Continuous Delivery that many people confuse this part with the whole.

To get a high degree of confidence you must test your changes as soon as possible, locally on your development machines and in an automated way. Unit tests and local test environments for integration testing are therefore essential. Then you can commit changes that (almost) never break anything on production. And if they do break something, you can undo it swiftly with another small change. By the way: small in this context means 1 line of code or one new function at maximum.

Delivering swiftly means it should take about five minutes to commit the code and a maximum of 45 minutes to have the automated pipeline check and deliver it to production. Delivery does not imply release, there might be a manual step before you release your change to the public. The whole process of creating, checking, and delivering the change should be done in about an hour.

In the above, I went over some techniques and practices that are part of improving your Continuous Delivery. Some of these tend to lead to lots of discussions, such as pair programming. It is beyond the scope of this article to discuss what practices are essential and have been proven to contribute. If you want to know more about this, refer to the State of DevOps Report.

The Need to Adapt

Back to DevOps transformations. Why do you need to rethink and improve the way in which you deliver software? Consider the following: In the current economy, change is fast and often unexpected. New companies with innovative power manage to disrupt traditional industries. Do you think your industry will be an exception?

Just ask friends who work in retail what Amazon did to them. Or ask video stores (if you can find any) how the arrival of Netflix changed their business. No industry is safe from this disruption, because it is based on a simple premise: whoever can automate faster and more reliably will outcompete the other businesses in their field because they can innovate faster or at least respond fast to innovations by competitors.

Financial institutions are certainly feeling some pressure to adapt. Fintechs are nibbling at their customer base, payment service providers are eroding some of their traditional services and the Big Tech companies are offering convenient new payment options like Google Pay and Apple Pay. Banks and insurance companies know they must become more agile and faster at delivering software to compete.

What is DevOps Really?

To attack the adaptation problem I described above, many organizations are going through a DevOps or IT transformation. They are trying to apply the values and principles of DevOps to their organization. This can be very challenging. To find out why, let’s address some common misconceptions about DevOps. DevOps is not about simply putting operations people into development teams. Neither is it about automating everything or putting your software in the cloud.

Instead, DevOps is all about rigorous experimentation, fast feedback, and an unimpeded flow of small changes. This is a slightly paraphrased version of the Three Ways of DevOps coined by Gene Kim, one of the founding fathers of the DevOps movement. Doesn’t this sound very similar to what I defined above as Continuous Delivery? Indeed, the ideas about fast feedback and flow are taken from Continuous Delivery and the Lean Manufacturing. Add to that an environment in which people feel free to collaborate, experiment, and learn and you have the basic ingredients of DevOps.

DevOps requires a scientific mindset: Make observations that describe problems, be curious about problems and investigate them; create hypotheses about the problems; test your hypotheses by recording data; also consider that you might be wrong by testing the null hypothesis; refine your hypotheses based on the results of your tests. Don’t assume anything and keep asking questions.

Start with the Goal

In The DevOps Handbook Gene Kim describes how you should start a DevOps transformation by selecting willing and able teams and moving them through a transformation. He describes how a transformation may grow through collaboration with other teams and dissemination of ideas and practices. This, as opposed to a big bang transformation, where you try to move a lot of teams to a new way of working all at once.

Gene Kim’s recommendation is essential advice. If you want to instill a mindset of experimentation and learning, you must organize your DevOps transformation in the same manner: start small, experiment, and measure.

To succeed, DevOps transformations need a clear vision and a common goal or objective to achieve. Management needs to translate the pressure from the marketplace to clear and achievable objectives that lead to measurable results. The OKR system of Objectives and Key results is one way of doing this. I will not go into detail about this.

Implementing the Three Ways

Assuming we have defined a clear and measurable goal, let’s look at how we can implement the Three Ways of DevOps:

To get to an unimpeded flow of small changes, get rid of anything that halts or slows down your flow. To do this you use Systems Thinking. By analyzing the work processes and interactions between parts of your organization, you gain insight into possible improvements.

What is the current bottleneck? Is it a dependency on another team? Go and fix it first! Is it a company policy that is slowing you down? Escalate it and let them fix it! Is it that you are too distracted by keeping legacy systems running? Consider decommissioning them or if someone really needs that old system, let them maintain it. To implement fast feedback loops you need to automate your testing, your provisioning, and deployments and you need to shift as much testing as possible to the beginning of the process. Fast feedback means that at every step of the way you try to find mistakes as fast as possible. This is done by writing good unit tests and by creating local development environments that support fast iterations in every possible way. To enable experimentation use the power of the cloud and containers to make it easy to spin up and destroy (parts of) systems that you need to try something out. Enable canary deployments, A/B testing, and gather customer feedback to make sure you can identify and undo changes that do not help your business. Motivate teams to try out things, enable them to buy their own licenses where possible, and let them share the learnings from their experiments and production issues.

Survival of the Fittest

It cannot be overstated how important Continuous Delivery and good DevOps practices are to an organization’s survival. The State of DevOps Report may give you an indication of whether you are with the elite performers or are falling behind. Simply said: if you are not doing this right now, you will be at risk to be outcompeted within a few years, no matter how stable and traditional your industry may seem right now.

The role of engineers in this is to be curious about new practices and ways to improve Continuous Delivery. Share knowledge about practices and engage in communication inside and outside your team. The culture of the organization is shaped by the behavior of individual engineers. And as you can see from the points above, just trying to improve flow, feedback and experimentation will push your organization to adjust policies, structures, and processes.

Culture is Key

What can management do to support engineers? To enable experimentation and innovation, you need a few pre-requisites. If people in your organization are too scared to speak up about failures, then they will not be able to learn from past mistakes or prevent them in the future. So, first, you need to improve psychological safety by team-building exercises, and by sharing failures and learnings widely and consistently. Managers must lead by example and show their own failures and learnings. The transparency that comes with Objectives and Key Results (OKRs) is a great way of doing this.

Second, your organization will need to give engineers time and resources to foster a culture of continuous learning. As an engineer, you need to be able to decide what is most important to improve in your code. You need to feel empowered to take the trainings you are interested in and to try out the tools you want to use. With the right support you will aspire to be the best at what you do, and you will have fun at work. If you feel like that you are more likely to stay and keep your valuable experience in the organization.

Finally, management should keep repeating that your organization is driven by IT and by great engineers just as much as by business value and great business managers. The organization should listen to the engineers’ concerns and foster an engineering culture where software is of high quality and repetitive or compliance-related work is automated away. In other words: together you should steer towards a low cognitive load for engineers. This you concentrate on your core business of understanding and automating the domain that you work in (see Team Topologies).

Steps of a DevOps Transformation

Entire libraries have been written about how to approach DevOps transformations and organizational change. As a former DevOps consultant, I will try to summarize what I have learned from literature and practice. To me, a DevOps transformation consists of the following steps (some of these may be executed concurrently):

Start with a clear vision and objective Create an environment in which people feel safe and are rewarded for sharing failure Create time and space for continuous learning and experimentation Emphasize and strengthen an engineering culture Emphasize and incentivize collaboration Analyze/map your value stream per customer-facing service Adjust team structures to minimize communication overhead and to enable teams to take end-to-end responsibility Iterate in small steps towards your goal by improving:

Flow of Work

Fast Feedback Loops

Ability to Experiment

Figure 1: Steps in a DevOps Transformation. The colors indicate similar steps.

In figure 1, I show a typical distribution of responsibilities to take these steps. The first four steps are mainly a responsibility for the entire organization (driven by higher management). The last steps are mainly the responsibility of a department or group of teams. Collaboration, the value stream map, and optimizing processes are shared responsibilities. This distribution does not imply that management should not care about the work the team is doing and vice versa. They should work together on every aspect.

Conclusion

The crucial point here is that the cultural aspect of the transformation is leading and must be structured and supported heavily by management and carried by the engineers. Without the right culture, all your efforts to improve will fizzle and fail. Culture follows practice. As Nicole Forsgren and her collaborators found, promoting Continuous Delivery practices helps improve culture [Accelerate]. This implies that an important part of the change is bottom-up with engineers driving the adoption of improvement practices and capabilities.

Practical Advice

If you are planning to conduct a DevOps transformation, start with the objective and formulate clear goals on each of the cultural aspects. Concentrate on fostering a generative and open engineering culture, by implementing Continuous Delivery practices. Lead by example by sharing your own OKRs and by sharing your failures and learnings as well.

As engineers, we need to demand time to work on improving our automation and all other practices that can save time and make us more effective. Support engineering communities within your organization and improve communication and collaboration in your own team.

Doing this will ensure a better experience for engineers, better software delivery performance, and better innovative capability. It will not be easy and may take years. Whenever you are unsure what to do next, keep going back to the fundamentals. Can we improve flow, fast feedback, or the ability to experiment? Are we reaping all the benefits of Continuous Delivery? Rest assured that the results will be worth the effort.

Continue Reading

If you are currently part of a DevOps transformation and you are wondering how you are faring, you will need metrics to get a good picture of your current state. You need to be able to measure whether a certain improvement to your process, or to your application achieves the desired effect.

To learn more, read part 2 of this blog post (coming soon), in which I will explain how to achieve situational awareness in a DevOps transformation by using metrics.