The Power of a “Reference” CI/CD Pipeline
By: Drew Manning
Tags: Ci Cd Pipeline, Automation, Cicd, DevOps

On the CI/CD journey within large companies, stakeholders fall into different categories, each presenting challenges when driving top-down change. The first category are individuals that want to adopt CI/CD but have no idea what it really means or how hard it is. Another category are individuals with a “tools only” perspective, who want to automate — which is good — but haven’t considered the people and process side of things. The final category are the we-can’t-do-it-here nay sayers that haven’t yet embraced a transformational mindset.

The common thread for all these archetypes is a lack of a practical understanding of CI/CD. We need a way to demystify the terminology of CI/CD and get past the hype. We need something demonstrable that resonates with the business, and avoids the futile “other companies have done it this way” line of discussion.

Enter the “Reference CI/CD Pipeline.”

What is a Reference CI/CD Pipeline?

A reference CI/CD pipeline is an automated process from Dev to Production for a specific component of a company’s (or line of business’s) software product. It’s the first iteration of an end-to-end pipeline that you want to use as the “standard” to drive CI/CD adoption across teams.

Just as important as the automated pipeline, the scrum team that is responsible for the software component will adopt CI/CD best practices, such as regular code merges, test automation, robust data mocking, etc. This team will be the reference for the people/process changes that other scrum teams need to adopt.

Overall, the reference pipeline defines clearly what the target state technology, organization and processes should be to achieve CI/CD.

What do Reference CI/CD Pipelines Address?

Reference pipelines solve a number of challenges for the enterprise:

Achievable in Near Term (i.e., a Few Months)

Provide a Concrete Vision of the Future

Can be Implemented Separately from Mainline SDLC Activities

Achievable in Near Term (i.e., a Few Months)

Top-down CI/CD initiatives at large companies may fall victim to the “boil the ocean” problem: scope is simply too large to realistically implement successfully. These large initiatives often attempt to implement change across many teams and products at once, and can be expensive from a technology and people perspective, and very difficult to implement.

If done right, the reference CI/CD pipeline is scoped small enough to deliver in a few months.

Provides a Concrete Vision of the Future

Ask 10 people what their definition of DevOps or CI/CD is, and you might get 10 different answers. Many people equate DevOps and CI/CD only with automation or simply with tools, and while automation and tools are a key part of CI/CD, it is only one part. In my opinion, automation and tools are NOT the most important part. The part that can often be missed is the people and process side of CI/CD. A reference CI/CD pipeline makes it “real”, by defining the automation and the people/process changes needed.

Most teams have some level of automation on their pipelines already, which may include build, deployment and possibly test automation. The reference CI/CD pipeline should fill any critical gaps that allow those pipelines to go end-to-end from Dev to Production. For example, new automation capabilities required in reference CI/CD pipelines that I’ve worked on include:

Auto-promotion : establishing artifact promotion rules requires new capabilities that assess if quality gates are met on the pipeline, and then automatically “push” artifacts and relevant configurations to the next environment. The goal is to convert human decisioning — and the subsequent push-button promotion — to system decisioning and auto-promotion, and for teams to trust that decision.

: establishing artifact promotion rules requires new capabilities that assess if quality gates are met on the pipeline, and then automatically “push” artifacts and relevant configurations to the next environment. The goal is to convert human decisioning — and the subsequent push-button promotion — to system decisioning and auto-promotion, and for teams to trust that decision. Additional Automated Testing: in order to certify that one component can be deployed independently of the others, additional testing is likely needed. For example, consumer-driven contract testing will allow teams to feel confident that their API changes won’t impact consumers of that service. Similarly, component-level performance testing may be needed.

People/process changes I’ve observed include:

New Skills : Teams will need to get better at data mocking, performance testing and consumer driven contract testing, and new skills are likely required; new skills also may involve learning new tools, such as Pact and JMeter. In addition, microservices development — which is a lot harder than many people think — requires new skills. In my experience, teams need to internalize the 12 factor app. https://12factor.net/

: Teams will need to get better at data mocking, performance testing and consumer driven contract testing, and new skills are likely required; new skills also may involve learning new tools, such as Pact and JMeter. In addition, microservices development — which is a lot harder than many people think — requires new skills. In my experience, teams need to internalize the 12 factor app. https://12factor.net/ New Processes: teams need to define quality gates for the environments that are on the end-to-end path, including requirements from QA and Ops teams. Other processes include the use of metrics to drive desired behaviors: reducing time to fix bugs, fix broken builds, merge code, etc. The visibility that metrics give can be uncomfortable for teams moving to CI/CD, so a lot of practice is needed with this new process!

Can be Implemented Separately from Mainline SDLC Activities

Implementing technology and process changes can be very disruptive, so limiting the “blast radius” of the reference pipeline initiative is important. Because the scope of the reference CI/CD pipeline is (usually) one team and (usually) one software component, work can be done with little impact on other day to day BAU activities.

Other teams will appreciate that the kinks are worked out before they are asked to adopt the pipeline!

Challenges with this Approach

As one might suspect, driving change through reference CI/CD pipelines has its own challenges, but these can be overcome through smart planning and execution. Challenges include:

Strong Leadership:

Like all change initiatives, top-down sponsorship is needed, as well as a strong leader to guide the team(s) through the change. The absence of sponsorship and leadership will definitely put your reference CI/CD pipeline initiative at risk! Pick a sponsor that you believe is willing to try something new, and a day-to-day leader who is respected by teams, and who understands CI/CD pipelines, perhaps one of your leaders from the DevOps function (if you have one).

Like all change initiatives, top-down sponsorship is needed, as well as a strong leader to guide the team(s) through the change. The absence of sponsorship and leadership will definitely put your reference CI/CD pipeline initiative at risk! Pick a sponsor that you believe is willing to try something new, and a day-to-day leader who is respected by teams, and who understands CI/CD pipelines, perhaps one of your leaders from the DevOps function (if you have one). Tight Scope Management :

When asked “How should we improve our end to end pipeline?” many team members have good ideas, but unfortunately, they can take a long time to implement. In addition, external dependencies can slow down progress. Overall, as the reference initiative moves along, the potential for scope creep increases. I recommend taking a “minimum viable product” (MVP) approach, which Agile scrum teams should be familiar with. Remember that the goal is to deliver MVP in a few months, and iterate from there.

: When asked “How should we improve our end to end pipeline?” many team members have good ideas, but unfortunately, they can take a long time to implement. In addition, external dependencies can slow down progress. Overall, as the reference initiative moves along, the potential for scope creep increases. I recommend taking a “minimum viable product” (MVP) approach, which Agile scrum teams should be familiar with. Remember that the goal is to deliver MVP in a few months, and iterate from there. Keepin’ It Real:

Arguably, the quickest way to get a piece of software end-to-end is to use a fake/dummy software component. This might be a temptation when creating a reference pipeline. On the positive side, there should be very little friction in moving the software through the SDLC, but, in my opinion, this approach should be avoided because it lacks the people/process changes that teams must adopt. Fundamentally, how do we get end-to-end scrum teams to think and behave differently if we don’t use something they are responsible to build, test and operate?

How to Move Forward in your Company

So where to go from here? If you are an executive trying to get your organization to move to CI/CD, and are being stymied by slow progress, I recommend you consider implementing a reference CI/CD pipeline.

Some steps you can take:

Engage your DevOps/Automation team : This team will be the subject matter experts for a reference CI/CD pipeline, and should offer resources in moving it forward. If you don’t have a DevOps/Automation team, ask around and see if you can find any champions for CI/CD and automation.

: This team will be the subject matter experts for a reference CI/CD pipeline, and should offer resources in moving it forward. If you don’t have a DevOps/Automation team, ask around and see if you can find any champions for CI/CD and automation. Pick a Product and a Team : The key is to start small. Select a microservice or API built by a team that is open to doing things in new ways. The team should be following Agile methodology. The product should be a strategic one that would benefit from faster time to market.

: The key is to start small. Select a microservice or API built by a team that is open to doing things in new ways. The team should be following Agile methodology. The product should be a strategic one that would benefit from faster time to market. Draw an End-to-end Picture : In “selling” this initiative to the stakeholders, a diagram can really help to show what you want to do; include enough technical detail that your technology stakeholders understand the vision, and also information that is interesting to the business, such as speed to market and metrics.

: In “selling” this initiative to the stakeholders, a diagram can really help to show what you want to do; include enough technical detail that your technology stakeholders understand the vision, and also information that is interesting to the business, such as speed to market and metrics. Engage the Exec Sponsor : Once you have done the items above, it’s time to engage the executive responsible for the selected product. Are they on-board? Ideally, the sponsor announces this initiative, so all teams are aware and will align with the plan.

: Once you have done the items above, it’s time to engage the executive responsible for the selected product. Are they on-board? Ideally, the sponsor announces this initiative, so all teams are aware and will align with the plan. Ensure the “Big 3” are Represented: Engineering, QA, and Ops are all required for this initiative, so you will need to make sure you have representation from those teams.

Conclusion

Driving CI/CD adoption in large companies is hard. It requires that you start small and iterate. Reference CI/CD pipelines are one important way to define both the end-to-end pipeline automation and the new ways of working. They let you work out the kinks before broader team adoption.

Fundamentally, when you want to drive change with your stakeholders, “Let me show you” is always more effective than “Lets talk about it”!