Define the right Quality Metrics and collect them automatically in one place
By: Mohsen Nasiri
Tags: App Script, Metrics Collection, Quality Metrics, Datadog, Collecting Metrics

How to define meaningful quality metrics that matter to you and your product, and using App Script to collect them all in Google Sheets

Introduction

In modern software development setup, logs and metrics are all over the place. With Kibana, Datadog, and Grafana easily integratable with any programming language and infrastructure providers, collecting logs is not a problem. As a Quality Assurance expert, what I am interested in are those specific logs, or metrics, that reflects the quality of our product, and finding ways to monitor those metrics on a regular basis and seeing their trends over time.

In order to talk about what I exactly do to achieve this goal, we need to break down that last sentence and separate those goals one by one:

“… specific logs that reflect the quality…”

“… finding ways to monitor them…”

And “…see how they change over time”

This is what this article is about. I am going to talk about how I approached these objectives, the lessons I learned from my mistakes, and the final steps that I took to finally have a clear understanding of How is the current state of quality in our product and how it changes over time in a fully automated setup, so my engineering teams can constantly have a clear overview of the product’s quality and see how the decision they make influences it.

What is a “Quality Metric”

A quality metric is any metric of any given type (size, time, count, etc) that either directly, or indirectly, represents the quality of a product or service. Let’s take a web app as an example. What types of metrics can we think of that directly represent its quality? From the most obvious to least, we can name a few:

Page Response Time

This refers to the time calculated from the moment a user interacts with the application until the moment that the interaction results in an effect.

For example, the time it takes when you click on someone’s profile picture in a social media app until the profile page and its content is fully loaded is the Page Response Time.

This refers to the time calculated from the moment a user interacts with the application until the moment that the interaction results in an effect. For example, the time it takes when you click on someone’s profile picture in a social media app until the profile page and its content is fully loaded is the Page Response Time. Uptime Status

It shows how often a web page been up and running by showing a percentage of its uptime.

If a web app is down too often, then it means there is a chance that at a point in time and when the service was needed, it was unavailable for some users, therefore there is no quality to define for it to begin with. Over a period of time, the average uptime percentage can define the quality of the service, since Quality of Service is directly affected by its availability.

It shows how often a web page been up and running by showing a percentage of its uptime. If a web app is down too often, then it means there is a chance that at a point in time and when the service was needed, it was unavailable for some users, therefore there is no quality to define for it to begin with. Over a period of time, the average uptime percentage can define the quality of the service, since Quality of Service is directly affected by its availability. Failure Rate

It refers to the relative number of failed interactions that users experience while using a service. While using a web app, or any sort of software for that matter, the more issue, errors, empty screens, and such that users experience, it would be fair to say that the lower product quality is.

As you might have noticed I stated in the definition of quality metrics “either directly or indirecty”. The three examples above are metrics that directly reflect the quality of a product or software. For instance, in the case of the Page Response Time, it is fair to say that:

The lower the response time is, the higher the quality

… or in case of the Failure Rate:

The higher failure rate than user experience, the lower the quality.

But not all metrics can directly be linked to the quality of a product. Sometimes such metrics only play a role, minor or major, and might end up affecting the quality. These metrics usually can be found not so visibly around the product itself, but rather around the resources that are allocated to the product; resources being infrastructural, developing teams, or related to the users’ experience. Two products might have a similar response time and failure rate, but the majority of users might simply claim “It just feels better when I use one and not the other” which might be translated in terms of quality.

As a couple of examples for metrics that indirectly influence that quality, we can mention a couple:

Bugs count over a period of time

Although detecting failure rate can be easily achieved by looking at your log monitoring service, the number of bugs per week, month, or sprint tells a different story.

Simply because of the bugs count in a development team is increasing, that doesn’t necessarily mean that quality is getting lower since all the other metrics that we have already been through might still be at a good rate, but one conclusion that we can draw is that “Something’s wrong and that something might, later on, result in lower quality”.

As an example, the reason behind this increase could be pressure from the product owner to release new features, which down the road, might end up affecting the quality.

Although detecting failure rate can be easily achieved by looking at your log monitoring service, the number of bugs per week, month, or sprint tells a different story. Simply because of the bugs count in a development team is increasing, that doesn’t necessarily mean that quality is getting lower since all the other metrics that we have already been through might still be at a good rate, but one conclusion that we can draw is that “Something’s wrong and that something might, later on, result in lower quality”. As an example, the reason behind this increase could be pressure from the product owner to release new features, which down the road, might end up affecting the quality. UX analysis metrics

There are tools out there that capture user’s interactions with a web app, and run analysis on those measurements to extract meaningful metrics. FullStory and HorJar are two of these tools that enable the developing team to see how happy, unpleased, or frustrated the users were using their product, by showing them metrics such as the number of rage clicks (user spam-clicking a UI element) or dead clicks (Clicks on elements that don’t trigger an action), which can reflect the possibility that the current UX can benefit from improvements and end up increasing the overall quality.

Which quality metrics should we aim for?

Always choose the metrics that are relevant to your product, not the ones that are often mentioned on the web that you can find on a “Top 10 quality metrics” articles that a simple Google search leads you to.

First of all, many of the quality metrics that were relevant in the olden days of software development are not fitting in today’s rapid architecture. I would argue that metrics such as Change Failure Percentage, MTBF (Mean Time Between Failures), and MTTR (Mean Time To Recover/Repair) are either not so easy to measure, or they simply are not applicable anymore in today’s small scrum/vertical developer teams in the tech scene.

And secondly, a quality metric that is very valuable to one product, does not necessarily bring the same value when measured for another.

Let’s go over this by comparing two different products: A flight and hotel booking web app, like Kayak, and a social media app, like TikTok, with millions of young and impatient users.

For the booking app, the failure rate is the most important quality metric which should be as low as it can be at all times since the booking process is usually a long process and collects users’ info, travel details, and their payment info. Response time though, on the other hand, is not as crucial, since booking processes are usually long ones that people are used to already, and collecting thousands of flight info from other search engines consumes a lot of time anyway.

On the other hand and for the social media app, it is the opposite: The response time has to be as low as possible since most users won’t wait for too long for content to load, and if nothing loads in a short timespan, they simply swipe to the next one, while failure rate, although still important, is not as critical as the booking app.

The best way, in my opinion, is to sit with your product owner/manager and iterate over a list of possible candidates for quality metrics and find out which ones are more valuable to spend time on their monitoring and collection.

How to measure and monitor the metrics

Depending on the metric you choose, there is usually a tool that is made to collect them. Here is a list of well-known tools for some of the quality metrics that I used in the past or currently use:

For the response time of the backend services and failure rate, there are log collecting tools that can be integrated with your services, like Datadog, Sentry, and Kibana

For the Page Response Time and Uptime Status of the frontend pages, there is StatusCake

And for UX quality, there is Sentry and HotJar