What Happened to Tests?
By: Borja Burgos
Tags: QA, Testing, Observability, Software Development, Microservices

In recent years, we’ve made it trivially easy to build and operate increasingly complex systems, yet we still [try to] test these systems as if they were the simple applications of generations ago. Talk to developers about testing at their organizations and you’ll hear no shortage of alarming stories. Stories about unit tests purposely built to increase coverage, mocked interfaces behaving nothing like the real interfaces in production, ever-increasing rates of test and build flakiness, ever-decreasing rates of developer productivity, complex end-to-end tests that take weeks to debug, and worse. All things considered, rare is the occasion you find an application developer or an engineer in test satisfied about testing at their organizations.

It’s unfortunate, given the speed at which engineering teams are able to ship code, tests have the potential to become the single best indicator of performance, correctness, and quality of a system before reaching production. But when teams lack the tools and processes to collect, organize, analyze, debug, and ultimately understand tests and test data, then testing becomes a chore often relegated to a tertiary role.

Popular sentiment in tech right now is: why bother with pre-production testing? After all, the great tooling we have today allows us to run experiments, test in production easily, and do canary deployments, partial deployments, blue/green deployments, automatic roll-backs, etc. Without dwelling into the merits of software testing, as a practice, we shouldn’t think of these modern deployment/release strategies as alternatives to testing — they’re complimentary after all. No matter the mechanisms an organization has to mitigate the impact of a bug in production, its impact will be orders of magnitude more costly than catching the same bug in testing.

We’ve seen it coming

In 2013 we co-founded Tutum, a startup in the container ecosystem. In August 2015, Docker acquired Tutum. Containers, cloud, and automation changed the ball game, and over the past six years, we’ve witnessed countless innovators, early adopters, and more traditional enterprises go through, and struggle with, the most significant technological transformation in recent years. How we develop changed, how we build changed, how we deploy changed, how we operate changed, how we monitor changed, but how we test? That hasn’t changed.

We first experienced the limitations of our testing tools while at Docker, where our team was building modern applications, moving [extremely] fast, and most importantly, helping customers do the same. We ran a complex SaaS and built enterprise software, composed of dozens of services, deployed across multiple data centers, developed using a handful of languages and frameworks. But, in due time, we realized that our customers, across industries, with teams from 10 to 2000 engineers, with 3-tier applications or thousands of microservices, were experiencing our same problems. We were not alone.

These problems manifested themselves as an increasing number of seemingly small issues around testing that started as minor inconveniences and steadily grew in complexity and difficulty. Before we knew it, our CI was more often red than not, plagued with flaky tests that couldn’t be reproduced, broken tests that couldn’t be debugged, and regressions that were not being caught until they reached production. And worst of all, developers across organizations were spending precious hours unsuccessfully triaging failures.

The tipping point

Like every software team out there, during our tenure at Docker, we experienced occasional outages in our production systems. We marveled at the team’s ability and speed, resolving these complicated bugs in our distributed application within minutes of a pager alert. That got us thinking, how come we can fix a production outage so quickly, yet our CI remains broken with a test we can’t debug for almost a week?

And that’s when it clicked. Whereas in production we had APM, infrastructure monitoring, structured logs, error tracking, exception handling, distributed tracing, metrics, and pretty dynamic visualizations to make sense of it all, in CI all we were left with was a log dump of our failed build.

After working with engineering teams at every maturity level, and across every possible industry, we realized something: as the complexity of an application increases, the visibility and value teams derive from their tests rapidly approaches zero. While it may be easy to debug a unit test in a simple application, what happens when an integration test, or even worse an end-to-end test, fails in a complex system? Your build is broken for a week. And a broken CI pipeline is the biggest bottleneck of many software engineering teams.

On top of that, most teams today lack the ability to collect, organize, and analyze the rich information coming from tests. In this day and age, you’re lucky if your team is using something as battle-tested [and archaic] as JUnit XML to point out which tests are failing in a specific build. But even then, how do you know which commit is to blame? How can you easily assign and track a test regression to one particular service, dependency, or developer? And how do you track this over time?

This lack of visibility results in a negative cascading effect: if a developer cannot easily debug a broken test, they are disincentivized from writing additional tests or caring about existing tests. On the flip side, the easier it is to debug a test, the higher the incentive and ROI (return on investment) to write new and maintain old tests. The problems exacerbate when the team responsible for writing, maintaining, and running tests isn’t the same team responsible for fixing them. The DevOps movement helped bridge the gap and eliminate the animosity that existed between Developer and Operations. But we’ve yet to address the partnership and camaraderie needed to run an effective DevTest organization.

We took a step back and asked ourselves, what do we want from testing? What would testing need to look like to be a competitive advantage? How would this redefine the relationship to testing for both the individual and the organization? We started covering whiteboards with potential ideas. Now that we have the early manifestations of some of these ideas, the solution seems obvious in retrospect.

Towards a solution

With the adoption of containers, microservices, cloud, automation, and others, we’ve seen this problem quickly spread across teams. Having experienced it firsthand, and having helped other engineering organizations go through the challenges of modern-day testing, the problem soon became an obsession of ours. In 2018, Fernando Mayo and I co-founded Undefined Labs, a startup focused on solving modern software engineering problems. We knew testing would be our first undertaking.

The solution we’re working on is unique in many ways, yet inspired by custom tooling built at the world’s best engineering organizations — we call it Scope.

As mentioned before, one of the biggest problems that became clear from the beginning was the lack of visibility in testing. In a complex system, when a test runs, failure can come in thousands of different ways. Debugging failure without an in-depth understanding of what the system is doing is nearly impossible. To solve this, we looked at modern monitoring and observability technologies and practices, mostly being used in production, and applied them to the development and testing lifecycles of an application. Scope automatically instruments your tests and applications and provides distributed traces, structured logs, environmental data, exceptions, and much more to make debugging a piece of cake.

Unlike production monitoring tools, focused on live environments, Scope’s value comes in understanding the nuances of tests, the development process, and how application developers and testers work. From commits, PRs, authors, and reviews, to CI, build IDs, test suites and test executions, everything is purposely built for testing. Simultaneously, Scope serves as the DevTest bridge between QA, engineers in test, service owners, and application developers.

How Scope collects, organizes, presents, and communicates issues with teams are modeled after internal tooling used by organizations such as Google, Twitter, Lyft, and Netflix. Scope’s visualizations are purposely built for testing, from grid-views to quickly assess which commit is to blame, to historical graphs to understand trends, and catch performance regressions.

But this is just the beginning. And given the insurmountable amount of invaluable data in tests, we’re actively researching how to best apply ML (Machine Learning) to make testing an even more significant competitive advantage — as demonstrated by recent academic and Facebook research.

Scope is the product that will save developers and engineering organizations countless hours of their most frustrating and demoralizing work, debugging tests in failed CI. But our ambitions don’t stop with debugging. With Scope, our goal is to provide complete developer visibility. Visibility to understand how an application behaves, what a test is doing, and how, where it is failing, when it is slow, and most importantly: why? Tests are a treasure trove of information. Information that your organization is already generating, and all of which is going to waste. Scope makes it easy to tap into all of it.

Tests can help teams understand trends and identify all sorts of regressions — from a silent breaking change on an API, to a recent commit introducing a suboptimal SQL query, to an entire codebase slowing down over time. Scope provides the right signals with which to improve engineering productivity, and unveil critical areas for improvement — understand which critical paths in your application aren’t currently being tested, remove unused legacy code, eliminate redundant tests, reduce flakiness, increase meaningful coverage, and catch incompatibilities between services before you ship them to production.

We’re incredibly excited to share Scope with you. Sign up now for early access and turn your organization’s testing into a competitive advantage.