When Concept Drift Is Semantic Drift
By: Jenny Kwan
Tags: Concept Drift, Product Management, Data Engineering, Big Data, Data Science

Sometimes your models fail because the world breaks. Sometimes your models die because your observations of the world break. The distinction is harder to discern than you think.

Understanding the difference and how to conquer semantic drift leads to much more powerful models.

What Is Concept Drift?

I see many articles online about concept drift assuming that the reader has a background in statistics. Here’s an alternate approach:

A complex world exists outside of your organization. Change in this external world often breaks your models — COVID-19 and our collective response, changes in USA Federal Reserve policy, or the rise of TikTok. We call these changes concept drift.

Concept is the relationship between inputs and outputs of a model, with the outputs often being the model predictions. For example:

Show a prospective customer two sweaters, sweater A and sweater B. Assuming that this prospect is a female between the ages of 30 and 35 in the Northeast USA in a summer month, the prospect chooses sweater A 82% of the time. All of a sudden, the balance between sweater A and sweater B shifts to 27% sweater A and 73% sweater B. What happened?

In the context of a predictive model, the product choices, demographics, geography, and seasonality are inputs. The output is the predicted likelihood of purchasing each product. This relationship is the concept.

If this shift in purchasing patterns occurs out of the blue, we assume something unseen about the world has changed. Maybe the color is no longer in vogue. Perhaps animal rights campaigns have affected the consumption of wool. This drift is concept drift.

Adapting to a Changing World

When the concept of a model drifts, models have to be rebuilt. There are two options:

Use the same model type and parameters (and code), and train it from scratch on new data that represents the latest state of the world. This data should be unpolluted by the old world. An analogy is taking an infant and training it on a world that has always had TikTok. Failing the first option, throw away the code and go through the painstaking process of data science research to discover the new model type and parameters that fit the latest state of the world. This is like designing a cyborg optimized for TikTok.

Note that, in both cases, the new model wouldn’t work on the old state of the world. Old world? Old model. New world? New model.

The only time to use the same model on both the new world and the old world is when you seek a model that is stable across both. In other words, you desire a model based on inputs unaffected by the shifting winds of TikTok, colors from Milan, or animal rights campaigns.

Has the World Changed?

Whenever models break, data scientists often start with the assumption that the underlying concept must have drifted. After all, it’s easier than trying to gather somehow some magical inputs that may not exist and may be impossible (or too expensive) to acquire.

There is a crucial caveat to this assumption: Most organizations do the bulk of their data science research on data that has to do with a specific product, service, or platform. Often, the product captures that data itself. And when this is the case, changes in the platform and changes in how it captures data about itself can introduce drift that has nothing to do with the outside world.

This different kind of drift is semantic drift. And it takes a lot of work and discipline to be able to separate this from concept drift.

What is Semantic Drift?

The term semantic drift has applications outside of statistics and machine learning. Here, we refer to the meaning of data:

What does a variable or feature mean?

What does the value of a variable or feature mean?

What is an observation?

Let’s take gender identity as the variable in question. Upon registration, an online shopping platform asks prospective customers to identify their gender as either male or female. If the online shopping platform adds a third gender identity option, unspecified. The meaning, or semantics, of the gender identity variable changes. Let’s set the shifting politics of gender identity aside. Perhaps certain people wish to leave their gender marker unspecified to reduce their personal feeling of being surveilled. The point is that gender identity used to have two choices, and it now has three.

A typical data science use case is separating the population of prospects into partitions to treat differently. I wish to bucket males separately from females for marketing.

When the system changes how it defines gender identity, models behave strangely. Individual prospects who answered female in the old data may have chosen unspecified if given a chance. But they responded to that question on registration, and an organization wouldn’t ask them again.

Now, data scientists have to deal with a population of prospects where their value for the gender identity variable means something different depending on when the person registered.

Often data scientists don’t know that their datasets contain multiple semantic versions. And if they do, they have no easy way to tell which prospects are which.

Conquering Semantic Drift

A host of specific best practices for conquering semantic drift exist. This product management and software engineering discipline is rich and is beyond the scope of this post.

The basics boil down to this:

Products, services, and platforms that generate data must product-manage the data they generate as a first-class citizen.

Help data scientists by tagging data with the semantic version, which requires code-as-deployed versioning discipline.

Ensure data scientists can thoroughly understand how data of a specific semantic version was collected by preserving code-as-deployed such that it is identifiable by version.

Product managers and product software engineers are not the only ones with work to do. Data scientists and data engineers do as well. The primary task is:

Data infrastructure should be able to rationalize data sets, such as user journeys, that include observations of differing semantic versions.

Thoughts and feedback, please!