Building resilient data products quickly (Part 1 of 2)
By: Matt Wright
Tags: Microsoft, Azure, Data Science, Analytics, Software Development

Building resilient data products quickly (Part 1 of 2)

Anomaly detection through the lens of online dating

Author Matt Wright is joined for this article by co-author Deepsha Menghani

Data scientists on the Customer Growth Analytics team at Microsoft are spoiled. For years we’ve enjoyed the privilege of working alongside a world-class data engineering team, whose charter includes maintaining our high-quality data platform. They bring innovative solutions to tough problems, but with limited capacity, they aren’t always able to provide immediate support, and sometimes we have to get creative.

My team recently had to ingest and integrate new data from over 90 independent sources. We were designing a model that required this data, and we had to ship a functional prototype in front of an aggressive deadline. What is the fastest way to identify data quality issues, maneuver around them, and deliver on one’s commitments? That’s what we had to find out. Drawing inspiration from a couple of unexpected places (including online dating and shipbuilding), we found a feasible solution in just a few days.

In this article, the first of two in a series (see part 2 here), I describe how we chose between streaming and batch data processing to ingest our new data sources. And I provide a step-by-step guide on the method we used to identify anomalies using just a few built-in functions in Azure Data Explorer.

Photo by Will Myers on Unsplash

To stream or to batch?

At the beginning of the data product development lifecycle, one must decide how to ingest data from one’s upstream sources. There are two high-level data ingestion paradigms: batch processing and streaming. With batch processing, your system accumulates data over time and, as the name implies, processes it in batches — often on a daily or hourly cadence. Streaming data, on the other hand, is ingested and processed in real time. Azure Data Explorer (ADX) works great for both approaches.

While choosing between streaming and batch processing, you must consider the many tradeoffs. Sometimes this decision is easy: If you need an answer from your data at the exact same time that you generate it, use streaming data processing. Such examples include autonomous vehicle operations, factory floor automation, or point-of-sale fraud detection.

Most scenarios are not so obvious, and like us, you’ll need to invest a lot of time up front evaluating the requirements driven by your user scenarios. If your users will engage with your solution only every few days, or every week, then building streaming data ingestion model will require more time and money than necessary, which is especially hard to justify if your solution has yet to prove its end-to-end value.

In the end, we all use data for the same reason: To answer questions about real-life events, enabling us to uncover and seize new opportunities.

It’s also important to remember that streaming versus batch processing is sometimes a false dichotomy. Your solution might require both methods at different layers in the stack, and the line between streaming and batch processing starts to fade when your batches are sufficiently small. For example, you may need to stream real-time events for anomaly detection just so you can meet your data quality requirements for batch processing downstream. In the end, we all use data for the same reason: To answer questions about real-life events, enabling us to uncover and seize new opportunities. If you think these opportunities could evaporate before you identify them in your data, then you need to reduce your batches to a finer grain or consider streaming ingestion.

The most important point is this: If you spend time up front analyzing your user scenarios to clarify your requirements, you will save time in the end and prevent rework. As they say, slow is smooth and smooth is fast. You might also consider a staged approach as you move from a prototype to a production solution. You can start with daily batch processing in your initial release, with plans to reduce batches or integrate streaming ingestion in a future release. This phased approach is the path we chose to get our prototype out the door as soon as possible.

After we identified the 90+ data sources we would need for our model, and after deciding to process the data in daily batches, we needed to get familiar with the quirks of each data source before we took a hard dependency. I like to imagine the next stage in the process as though we had looked at the profile picture of each data source, swiped right, and found a match. Now it was time to meet them all in real life.

Pro tip: Get to know your data before you decide to put a ring on it.

Go on a date with your data using Azure Data Explorer

Have you ever been stood up on a first date? It sucks. No one likes to wait uncomfortably alone at a table for two. Similarly, when I query data that is supposed to be refreshed daily, I like to see timestamps that correspond with my expectations. Call me selfish, but I also like to have a rough idea about how many rows I should get in return. If I expect to get one million rows every time I execute a query and then one day I get five million instead, it would be like showing up to a date with my crush only to discover that she had invited four of her friends. It’s fine, I guess. But it would have been nice to get a warning!

To identify the bad data — or at least the data that fell outside our expectations — we first summarized the daily ingestion counts from a sample of each data source and appended those counts to a table. The data sources here are aggregated daily, but as we explored above, you can design a batch data processing solution at any time grain, or even settle on a real-time event-driven streaming solution. Azure Data Explorer handles both scenarios. If you go with a streaming solution, you can bin your data it at any time grain and the methodology described below will be the same.

Here is an example of the results from our daily aggregation.

Figure 1

This aggregation makes it easy to produce trendlines for each individual data source. In Figure 2 below, you can see the weekly seasonality and the positive trend of one of our sample data sources, which we will later decompose. You can also see the outliers, where the red trendline dropped from a daily count of approximately 900 to less than 100 on a single day at the end of May, and then it dropped again from 6,000 to 5,000 — followed by four days with counts of less than 100 — near the end of June.

Figure 2

After visually inspecting a few of these data sources, we knew we would probably find outliers in all of them. We had to expose everything — all 94 data sources — to avoid getting catfished on our first real date. We would need to decompose the daily ingestion counts from each trendline to account for seasonality and trend. Then we could identify the outliers in each data source.

We used the built-in series_decompose function in ADX, as shown in Figure 3 below, to decompose the trendlines. This function extracts the baseline, seasonality, trend, and residuals of each time series. We did this in a single query to decompose all 94 data sources we explored. ADX makes this process easy by returning all the components of each time series as a dynamic data type in the same tabular format that we started with. Here is an example in Figure 3 below. Notice this requires only four additional lines of code.

Figure 3

Finally, we calculated z-scores based on the mean and standard deviation, which we extracted from the residual values above using the series_stats_dynamic function. Then we compared those z-scores against a pre-defined threshold to flag anomalies. The results in Figure 4 below show each anomaly we described in the examples above had been flagged as such.

Figure 4

After commenting the where clause on line 3 of query above, we had results for all 94 data sources over a 180-day period. ADX returned these results in less than one second.

Now we knew, and we could visualize, how often each data source had produced anomalies. Each row in the matrix below represents one of the 94 data sources we profiled over a 180-day period, and each column represents a single day of aggregation. The blue squares represent the good data, and the white squares represent the anomalies. There are 16,920 elements in the matrix, and 412 were bad. This enabled us to go to the individual owners of each data source and inquire about the data completeness issues we would likely face if we took a dependency on this data. It also enabled us to identify outliers moving forward in time.