Are Sophisticated Machine Learning Designs Your Go-To Solution? Here’s Why They Shouldn’t Be
By: Melih Bahar
Tags: Ensemble Learning, Feature Engineering, Data Science, Machine Learning, Domain Knowledge

In data science, especially with machine learning, asking the right questions guides you to valuable solutions. Sometimes, however, there is more than one question to answer or different points of view caused by contradictory or uncertain answers to those questions. I call these “multiple-component” problems. The challenge is designing a solution that takes all these components into account, like a domain expert would.

As a data scientist at Riskified, my daily work revolves around our ATO (Account Takeover) product and improvement research. Briefly, an ATO is when a bad actor gains access to another party’s legitimate account. The product decides whether a login is an ATO by using internal data supplemented by additional external data during login.

Why is ATO a multiple-component problem? For instance, one component could be the customer’s history, to answer the question “is the customer behaving similar to their history in this specific login.” Another could be the login’s raw data, to answer “does this login look suspicious by itself.” Mostly, the answers aren’t straightforward, and considering only some of them may lead to wrong decisions.

In this post, I’ll share with you how we approached this problem for ATO and why the newest, state-of-the-art, or most sophisticated solution isn’t always the best option. You can provide simpler, faster, but not inferior results for your multiple-component problems without rushing into sophisticated designs. Feature engineering and domain knowledge can go a long way.

First, let’s understand the problem.

The Experts’ Thought Process

Initially, we focused on how a “perfect model” approaches this problem — human analysts. The more we engaged with analysts, the more obvious it became that we had several questions at play. This diagram shows an (over)simplification of analysts’ thought processes.

Some of the different components an analyst needs to consider before deciding on ATO

Say we get a login with a high possibility of a proxy (some or all of the login data is hidden or changed). You might think that it’s not a good sign because it looks like the person’s hiding something. But what if the known behavior of this specific customer has always included a proxy, for some reason? Are we familiar with the account’s online patterns, such as IP or cookie behaviors? Does it act like a bot?

Analysts evaluate and answer all of these questions (and more) before making a decision. So, how do you model this problem?

Modelling the Problem

There are several options:

Neural Network

While it’s potentially a good solution, it requires a lot more data than traditional methods, and more importantly, it’s not easy to explain its decisions.

Explainability of our decisions is crucial, and as the data we have is insufficient, we ruled out this option. One Model

You can design features for different components and feed them to only one model — it’s a good baseline. Ensembling

You can model each component separately and ensemble them to achieve a final decision.

We decided on the last two options, and I’ll discuss them more extensively later. But first, let’s talk features.

Feature Engineering

Before even starting to design, we primarily engineered the features. Once the features were ready, they could be used for any method.

Feature engineering using both raw data and domain knowledge in order to create informative features

We leveraged as much domain knowledge and prior experience as possible, collaborating with analysts to carefully engineer features for each component. This way, we could accurately tell the story from each component while making sure they didn’t contradict or correlate.

For example, while some features look for suspicious (or safe) elements in the login, others are comparative features that look at the customer’s history to check for behavior anomalies. Others still may try to catch potential bots. The choices are many.

Armed with features, let’s take a closer look at our possible solutions.

One Model

We built the baseline by using the information from each component as features for only one model. It’s tricky though, as throwing many features into a model without considering them carefully causes more harm than good. Let me explain:

Baseline model in a nutshell

Too many features in a model can cause overfitting, since the model might learn patterns using only some features. If the sets aren’t representative, some features may become irrelevant and harm the model. Finally, while boosted tree algorithms are robust to correlated features, other algorithms are not and this might decrease performance.

While choosing the features, we tried to avoid information that is too-component-specific to reduce potential noise. We also engineered cross-component features to reveal possible interactions. For example, if both customer history and online activity are available, the model can draw information from both.

This method has several advantages. Technically, there’s only one model to optimize. Most production environments support a predefined set of algorithms, and adding a new algorithm or a different design isn’t always straightforward. Here, no change in production is needed and it’s simple to implement as we can use supported algorithms. It learns potential interactions between components. And lastly, only one training set and one test set are needed.

There are also disadvantages, of course. Potentially, it may have noise caused by different components and segments working together. This might cause a performance drop in sub-segments and/or overall. It’s harder to optimize performance for all sub-segments.

Ensembling

Ensembling is an intuitive way of modelling multiple components, as it imitates how analysts think. The main idea is clear — each component has its own learner and answers only one question using the relevant population segment with the specific components’ features.

Ensembling in a nutshell

Simply put, for each login you’d have multiple scores and then use a meta-model with those scores as the features.

For ATO, we selected the relevant engineered features for each component. We made sure there weren’t too many features per component, as the sets were smaller and overfitting was more likely.

Additionally, we engineered helper features for the meta model to help the right component tell the right story. For instance, we hint at which score is more trustworthy by revealing if a login belongs to a segment.

What about pros and cons?

First, it’s very modular. It’s easy to optimize each learner for its specific component. It’s easier to maintain smaller models and retrain only a specific learner or the meta-model. Also, it’s more robust to noise caused by irrelevant data, since each learner uses only the relevant segment.

However, not only is it technically more complicated, it also means changes in production and thus longer deployment times. Potentially, it can’t catch cross-component patterns or interactions. Also, set creation is challenging since each component requires its own representative sets.

Results

The best version of one model showed a great performance boost with more than a 0.2 increase in ROC AUC, compared to the model in production. We were excited as we expected ensembles to perform even better!

However, things got interesting when we saw the results of the ensemble models. While “better,” the difference in ROC AUC was hardly significant, being only 0.005!

ROC curve showing the best version of one_model vs. 3 different versions of ensembling

After spending so much time optimizing each learner and meta-model and training several versions, a clear win by ensembling was inevitable(!). But what actually happened?

The major boost came from smart feature engineering itself and not the model design! When other pros and cons are also taken into account, one model becomes the clear choice for the final solution!

Bottom Line

Just like we discovered for our ATO problem, the importance of domain knowledge and smart feature engineering is inevitable in designing solutions for real-world problems. Domain knowledge is the most efficient way of telling the right story, and feature engineering communicates that to models. These help your models “think” like domain experts. Often, well-engineered features are more important than model designs.

Multiple-component models are common in data science. Most often than not, performance isn’t the only criterion you need to optimize for a solution. Simplicity of the model design, explainability, ease of implementation in production, and many other factors also need to be taken under consideration, including the tradeoffs between them.

Summary table

As shown in our example, for valuable solutions you can invest more in engineering informative features with your domain knowledge for the problem at hand, before rushing into complicated designs.