Unsupervised Data Monitoring
By: Jeremy Stanley
Tags: Machine Learning, Data Quality, Anomaly Detection, Unsupervised Learning, Data Science

Finding and characterizing data chaos with unsupervised learning

To illustrate how this works in practice, let’s introduce an anomaly into this dataset. We will use our command-line client to trigger one of our chaos operations: TimeColumnZero . This introduces zero values into a column at a specific point in time.

Causing chaos in a table by inserting zero values into a column on a given date for a subset of rows.

The column numtickets in fact_listing will now contain 30% zero values on 2021–01–17 whenever the venuestate is equal to “NY.” This illustrates a common data quality issue — an invalid value suddenly appears for a fraction of rows in a key data segment.

Next, we can re-run the table anomaly checks, and we find that the no anomalous records variant fails (as it is looking for any meaningful change):

A failing table anomaly check after having introduced chaos into the table.

We can click into view details to see the explanation:

Summarizing the anomaly in natural language and evaluating its severity.

The table anomaly check has correctly identified that the column numtickets has a sudden increase in 0 values. Note that we never told it to look at this column or to look for zero values.

It also scores the anomaly's severity (this one is strong) and compares that to a learned threshold for this table, which accounts for how much background noise there is in each column. Because the severity exceeds the threshold, this check fails and is highlighted in the user interface (and could notify the user in Slack or Teams).

Scrolling down, we see a custom visualization selected based on the anomaly type and data distribution:

The algorithm chooses a dynamic visualization to help contextualize the anomaly, in this case showing the distribution of the top values in the column on the prior and current dates as a tornado.

In this case, a top values chart shows the most common numtickets values and compares the distribution on 2021–01–16 (the left bars) to the distribution on 2021–01-17 (the right bars). You can see that the value 0 was not there before and is now suddenly 10.4% of records.

Scrolling further, we can see the Root Cause Explanation, which analyzes the raw data underlying our unsupervised model to identify if there are segments of the data where the issue is most prominent:

A root cause analysis performed to identify where the anomalous records occurred.

As you can see, the algorithm correctly identifies that NY is where the anomaly occurred. 100% of the anomalous rows are in that state, but only 29.8% of the population rows are in NY.

This automatic identification of where an issue occurs is powerful. Without it, a user would need to examine records, trace lineages, or repeatedly query and visualize the data to isolate the issue.

The algorithm can find even more complex issues, such as when the relationship changes between columns. For example, consider this chaos operation:

Introducing a more complex form of chaos, where a single column is randomly shuffled on a date.

Here we are shuffling the priceperticket column so that the values no longer correspond to the correct rows. The actual values remain the same (and have the same distribution and mean), but the relationship between that column and other columns in the table has been broken.

In this case, the anomaly cannot be easily summarized in natural language, but we can still score its severity.

In this case, the algorithm identifies that priceperticket is the most problematic column, and that the issue is related to listid and totalprice as well.

Examining the distribution of priceperticket values on 1/16 and 1/17 shows that the anomaly is the strongest in the lowest and highest priceperticket values:

This visualization bins the continuous column into deciles. While the distribution is unchanged between the two days, we can see that the anomaly severity (color) is more intense in extreme deciles.

We can examine a sample of individual rows, where we can see that the algorithm is scoring every value in the table for how anomalous it is:

The most anomalous row in a sample of the data — with color indicating how much credit each value contributed towards the anomaly.

In this example, the $1,960 price per ticket doesn’t make any sense in the context of 28 tickets for $1,568. Plus, the price per ticket is unusually high for the Vampire Weekend show.

Zooming out, we can see this effect across many rows: