Data Observability: Four Key Statistics for Monitoring Numeric Values
By: Kyle Kirwan
Tags: Snowflake, Data Monitoring, Data Quality, Bigquery, Data Observability

Data Observability: Four Key Statistics for Monitoring Numeric Values

Detecting shifts in numeric distributions is a key part of any successful data observability strategy. In this post, we cover four key statistics for monitoring numeric data and how to track them in Snowflake, Redshift, and BigQuery.

Picture this: you work for an awesome SaaS business (“snacks-as-a-service”, obviously) as a data scientist. You just got a new dataset that tracks, among other things, the amount each customer is spending on snacks. Armed with this new data, you build a dashboard to give your growth team better insight into customer spending behavior. The dashboard even becomes part of their weekly team review meeting.

Sounds great, right? Customers are snacking. The growth team is happy. You’re happy.

Cut to a few months later when you get a frantic message from the growth team. The average spend per customer drastically decreased last week. They’re scrambling to figure out what happened, and since they discovered this on their Friday morning review, now somebody is probably losing their Saturday and Sunday to an investigation.

The team is turning to you for help, but you haven’t thought about this dashboard in months. You had no idea that these trends were even showing up. Is it a data problem? Is it a business change? The team is suddenly fighting a big fire that might be threatening the business. And you’re wishing you knew about this sooner so there was more time to react.

In this scenario, once you shipped the dashboard, you moved on to the next priority project and stopped paying attention to the data. Did a product change make it easier to have smaller transaction sizes? Was there a mistake in a marketing campaign configuration? Did an upstream data source silently break?

In an ideal world, as the owner of the dataset, you would always be the first to know about anything as serious as this sudden distribution shift. The shift could indicate that there was a data outage that requires your attention or that there was a business problem that requires someone else’s. Either way, finding out on Friday days after it silently occurred creates chaos for everyone.

If we were tracking some basic statistics about our user’s transaction sizes and had some alerts set up to catch sudden movements in those statistics, we could have kept our weekend debug-free. Let’s dig into four key statistics that — if we were tracking them — might have told us something was going wrong sooner.

First: what are statistical moments?

We often think of numeric data in terms of its distribution: the shape of the data. One common way to visualize a distribution is with a histogram (or maybe a kernel density plot if you’re fancy).

But while visual summaries are helpful to human eyes, there are times when we want to describe the shape of the data numerically. One of those times is within the context of data observability when we are asking a machine to alert us to sudden changes in the distribution’s shape.

The field of statistics uses several standard metrics to summarize these distributions, and they’re referred to as the “moments” of the distribution:

The 1st moment is the expected value, or the mean.

The 2nd moment is the variance.

The 3rd moment is the skewness.

The 4th moment is the kurtosis.

1st Moment: Mean

The mean — or more specifically the expected value — is the most commonly used moment. The mean is a convenient way to measure the overall location of your numeric values on the number line (the one with negative values to the left and positive values to the right).

Why it matters

When working with a large set of numbers, the mean is a great starting point for understanding the dataset. The mean gives you a sense of the center of the dataset and gives you an easy-to-understand view of the general quantity of your numeric value. If the average transaction amount is $0.57, you have a very different business situation than if it’s $927.

If you are tracking the mean of a numeric column in your dataset, a sudden increase or decrease likely indicates that the shape of your dataset has changed. The mean won’t tell you why a change occurred. The change could be due to a few distant outliers, an overall shift in the dataset, or anything in between.

Snack example

In our initial example, a drastic decrease in the average payment per customer sparked concern from the growth team. They were checking a dashboard on a weekly basis and noticed the average payment dropped from $50 to $35. This could mean the business is bringing in significantly less revenue overall, say 30% less, which is a big deal to any company.

Tracking and alerting on movements in the mean could have provided an earlier warning sign, giving you time to ensure it’s not a data problem or giving the growth team more time to look for a behavioral explanation.

Example SQL

Too easy, right?