“MRMR” Explained Exactly How You Wished Someone Explained to You
By: Samuele Mazzanti
Tags: Uber, Feature Selection, Data Science, Machine Learning, Python

MRMR (acronym for Maximum Relevance — Minimum Redundancy) is a feature selection algorithm that has gained new popularity after the pubblication — in 2019 — of this paper by Uber engineers:

Screenshot from: source.

The authors show how — thanks to MRMR — they achieved great results in automating feature selection in various marketing applications, among which:

user acquisition,

cross/up-selling,

user churn and reactivation.

However, MRMR is not a recent discovery. It was first proposed in this paper published in 2005 by two Berkeley researchers:

Screenshot from: source

The authors claim that, out of tens of thousands of genes, selecting just few of them (for instance 50) is sufficient for achieving the maximum level of accuracy for the task (predicting a disease). So the point is: how do we find the 50 lucky ones? This is exactly what MRMR was originally designed for.

From these first two examples, it’s evident how flexible is MRMR, which can prove useful in completely different domains, from marketing to genomics.

In this article we will see how MRMR works, and how it is different from other popular feature selection algorithms like Boruta (if you want to know more about Boruta, I’ve written a dedicated post: Boruta Explained Exactly How You Wished Someone Explained to You)

Minimal-optimal vs. All-relevant

Feature selection algorithms can be broadly classified in two categories:

minimal-optimal (such as MRMR);

(such as MRMR); all-relevant (such as Boruta).

Feature selection is like playing darts… [Figure by Author]

Minimal-optimal methods seek to identify a small set of features that — put together — have the maximum possible predictive power. On the other hand, all-relevant algorithms are designed to select all the features that — individually — have any predictive power at all.

Thus, if feature A and feature B are both relevant, but they bring more or less the same information, an all-relevant method will select them both, whereas a minimal-optimal method will select only one of them and discard the other one.

This may seem a suble distinction, but it makes a huge difference in practice. Imagine a scenario in which we have 10 thousand features. You use Boruta and you find out that 5 thousand of them have some kind of relationship with the target variable. But 5 thousand is still an unmanageable number of features.

In the example brought by Ding and Peng, you could likely achieve the same (or often higher) accuracy with a far smaller model, for instance 50 features. But, how to choose them? Taking the top 50 features from Boruta doesn’t work. In fact, many of them may be highly correlated with each other: they wouldn’t add much information.

MRMR was developed to overcome this issue.

“The best K features are not the K best features”

Let’s take a toy example. Say that our task is to predict the income of some people, knowing some of their (or their relatives’) characteristics. We dispose of these features:

A toy dataset. [Figure by Author]

Let’s also imagine that we know the causal relationships between the variables (this is a privilege that we don’t have often in real life, however it’s useful for understanding the logic behind MRMR).

Causal links between variables [Figure by Author]

Knowing the causal network would make feature selection ridiculously easy: we would choose {Age, IQ, Height}, since these are the only ones that have a direct causal relationship with the target variable. Adding any of the others would throw in unnecessary noise.

However, these are not necessarily the features that — taken individually — have the strongest predictive value with respect to the target variable. This is summarised by saying that:

“The best K features are not the K best features.” (Adapted from T. M. Cover).

What does this mean?

For instance, take Mother’s IQ: it’s not directly linked to Income, but it’s linked with IQ that is, in turn, heavily linked to Income. So, if you test it, you will find a quite strong relationship between Mother’s IQ and Income. But since we have already IQ in our model, there’s no point in keeping also Mother’s IQ.

Best K features vs. K best features, with K = 3. Each feature is associated with a measure of “strength” (F-statistic) of its relationship with Income. [Figure by Author]

In general terms, this implies that it’s not enough to look at individual measures of association between each feature and the target variable.

We wish to find the best K features, not the K best features!

This is exactly the problem with Boruta: it provides an individual evaluation for each feature. Indeed, in our example, Boruta would select all the 7 features, since all of them have some statistical dependence with Income.

And this could make sense in cases where we don’t have too many features. But, in real applications, it often happens that:

we don’t know the causal relationships linking the features and the target variable;

we have too many features;

there is high redundancy within the features.

In these cases, it may be that an all-relevant algorithm such as Boruta is too “indulgent”, whereas MRMR is capable of removing unnecessary features.

Causal links between variables [Figure by Author]

So, now the question is: how does MRMR work?

Under the hood of MRMR

When using MRMR, you are basically required to make only one choice: deciding the number of features that you want to keep. We will call this number K. In our example, we will take K=3.

In real applications, one can choose K based on domain knowledge or other constraints, such as model capacity, machine memory or time available.

MRMR works iteratively. At each iteration, it identifies the best feature (according to a rule) and adds it to the basket of selected features. Once a feature goes into the bucket, it cannot ever come out.

In our example, these are the outcomes that we would get at the end of each iteration: