How We Got to Zero Bugs and Implemented a Zero Bug Policy
By: Andrew Fulton
Tags: Technology, Software Development, Bug Tracking, Product Management, SaaS

So, how many bugs do you have in your backlog? Hundreds? More? Less? You’re not entirely sure? Well, I can tell you how many bugs are in our backlog right now. The number of bugs in our backlog is … wait for it… ZERO. Zero, zilch, nada. You’re not reading this wrong. We have zero bugs in our backlog. — Not only do we have zero bugs in our backlog, but we plan to keep it that way with our Zero Bug Policy. How did we achieve this incredible milestone and how do we plan to adhere to Zero Bugs moving forward? The answer is not as complicated as you might think, but not without some hurdles.

First, a little bit of history

Just over 2 years ago when I joined ConceptShare to lead the Quality Team there were over 350 documented bugs in the backlog (not crazy, but, not small either). My first question after peering deep into the abyss was “Can I delete all the bugs that are over one year old? What about all the bugs that are over six months old.?” The answer, of course, was No — there was knowledge contained within that extensive list of bugs, knowledge we could use to enhance the quality of our product and in turn enhance the experience for our customers. At the same time, we still needed a way to tackle the backlog of old bugs along with any new bugs that were impacting customer experience, AND continue development of new features and product improvements.

Why is getting to and having zero bugs important?

Having bugs in the backlog collecting dust really sucks.

It sucks for our customers because bugs impact how our product performs for them — we don’t like that. We want our customers using and experiencing our product the way it was meant to be experienced — no workarounds, no “oopsies, we missed that”, no “we’ll get that out……once these other bugs are fixed …”. Our progressive policy puts our customers front and center, allowing us to focus on their experience with our product and shifting the conversation from “It would be nice if ConceptShare did X properly” to “That worked great! Let’s explore more ways ConceptShare can help us do our work”. It sucks for our Creative Operations Advisors, who are our front line support for our customers. Instead of spending their time hearing the same complaints from customers about known bugs and not being able to provide any real timeline on when fixes will be in place, they can spend their time showing our customers the new and exciting ways ConceptShare helps them get their work done. That means they can shift their focus from customer service/support to true customer success. Finally, it sucks for our Product Development Team. Navigating through existing deficiencies when building new features adds unnecessary time and complexity to their work, meaning fewer features and improvements, which ultimately impacts our customers. Being able to say “what big thing are we shipping next?” is incredibly refreshing from “Should we fix these bugs before starting this feature?” Existing bugs aren’t ‘gotcha’s’ on new work, and keeps our development team focused on ideas and innovation, not bug fixes.

Getting buy-in

Having the idea to get to Zero Bugs, and implementing a strategy and policy around it was one thing. Before we could even plan how we were going to get it done, we needed support and buy-in from the Senior Management Team. Otherwise, shipping new features and responding to customer requests would always put bug fixes on the back burner. Here’s the case we made:

- Why was this an important goal to strive for? 1. for our Customers(Externally) and 2. For members of our Team (Internally)

- What is the current cost of our backlog of bugs (productivity, inability to ship more features faster), lost revenue, impacts on growth in accounts, and money wasted on unnecessary infrastructure to maintain performance levels)

- How would we get to zero?

- How do we maintain our Zero Bug Policy once we hit zero?

It took time and a few cycles of discussion to get our CEO fully on-board. The idea of dedicating development time solely to reducing our bug count didn’t seem possible. However, explaining the above made it clear this wasn’t just about having zero bugs. This was about so much more for our customers, for our team, and for our ability to continue to innovate and build a product our customers love. (and our bottom line!)

We were able to get him from “WTF! There’s no way we’re dedicating that much effort to that” to “This makes absolute sense. What do you need to get this done?” Not only did we win him over, but he wrote his own post on the topic and is now a zealous proponent of zero bugs!

Getting Started

We started tackling the backlog by setting a timeline with monthly goals we felt were achievable. The Product Team committed to reducing the backlog by ten bugs per month in conjunction with tackling new bugs that came in AND keep feature work moving along at a respectable velocity. As you can imagine, this worked well for the first month, ok the second month, and by the third month, we were struggling to keep up with both the backlog and new bugs because risking feature timelines to meet bug reduction targets proved to be a tough call forced on developers. To catch up, we did a blast of bug fixing. Backlog and ‘new’ bugs were on the decline. This got us back on track, but, within four months we found ourselves falling back into old ways — bugs were being pushed to the side for feature work and so on. Something had to give — leave the bugs as is and continue on the current path, or, figure out a new approach.

Adjusting the plan

Since our first approach to getting to zero bugs didn’t quite work out as expected, we started to think through different ways to tackle this. It turns out, the successful approach to addressing bugs was right in front of us, we just never considered to apply the approach to bug fixes. At ConceptShare we work in what we call “Feature Teams” — letting developers team up to work on the features that interest them the most. It’s a great approach because we get to challenge ourselves and choose what we work on. Happy worker mindset and all. So, we thought what if we treated bugs like any other feature? Would anyone on the dev team be interested in crushing bugs until they were all gone?It turns out; there were members of the team who were happy to take on this challenge and get us to Zero Bugs. A team formed, made a plan, and got to work.

We’ve done it! Zero Bugs. Now What?

After 18 months of dedicated work, some debates on how to focus our efforts, some small tweaks to our process and two months ahead of our target date — we achieved our goal — we have zero bugs in our backlog. I repeat. We have ZERO bugs in our backlog. It’s a fantastic milestone to hit. Our team is ecstatic. So, now what?

We implemented a new Zero Bugs Policy:

“Our policy mandates that for any given release, the highest priority is to eliminate bugs over building new features. This is to reduce the cost of fixing bugs, being responsive to our users, and keeping the quality of our product as high as possible.”

2. Our Bug Team has transitioned back to product feature work, allowing them to choose their next feature(s) while keeping in mind they, along with other members of our Product Team, remain committed to our Zero Bug Policy.

3. Our Product Team continues to strive to grow our product without the worry of being delayed because of old bugs. We can really put our foot on the gas now, and ship many more features and improvements faster than we could before

4. Our Creative Operations Advisors are now able to focus completely on helping our customers realize everything they can achieve with our product — without having to wait for Bug X and Bug Y to be fixed.

5. And most importantly, our customers can better utilize ConceptShare to work smarter, faster, and better, without being bogged down by annoying bugs in the software.

How do we stay at Zero Bugs?

To accomplish this, we have augmented our Bug Process (which probably resembles any other bug process out there at a high level) to align with our official Zero Bug Policy. These are the changes we’ve made:

Bug triage

Our triage meetings would typically occur on a weekly basis by a group of stakeholders (PMs/COAs/Dev Team Member(s) and QA). Bugs will now be triaged upon arrival by our QA team who will get any additional information from stakeholders if severity/impact is not clear. Bugs will then be sent to the Product Team for someone to grab.

Triaged bugs are immediately claimed

Triaged bugs no longer sit around and wait for someone to start working on them.We have slack built into feature timelines so developers can also dedicate time to pick up any new bugs without risking their feature work.

QA Team taking a more aggressive approach

If a triaged bug is not claimed (i.e. work has started) within a specific time frame our QA team will reach out to the Product Team to get the bug assigned so work may commence

Increasing our release iterations

We will strive to be even more aggressive in our release cycles, so our bug fixes make their way into our customers’ hands faster rather than being bound to a specific feature release.

Wrapping Up

Zero bugs, has quite the ring to it, and it’s been quite the journey so far with some ups/downs and crazy rabbit holes — but it doesn’t stop here. Over the past year and a half, we have accomplished so much more than just adopting a new policy on how we handle our bugs here at ConceptShare. This is now part of our culture — built into the way we work as we develop more features and capabilities of the highest quality, quality which will permeate out to our most important stakeholder, our customers.

I’d like to also say an enormous “THANK YOU!” to the team that made this happen. (check out how we celebrated with a bug shaped pinata.)

I’m curious — has anyone else tried to get to zero bugs, and if so — how did you do it? What did you learn?