“MRMR” Explained Exactly How You Wished Someone Explained to You | by Samuele Mazzanti | Towards Data Science

MRMR (acronym for Maximum Relevance — Minimum Redundancy) is a feature selection algorithm that has gained new popularity after the pubblication — in 2019 — of this paper by Uber engineers:
The authors show how — thanks to MRMR — they achieved great results in automating feature selection in various marketing applications, among which:
user acquisition,
cross/up-selling,
user churn and reactivation.
However, MRMR is not a recent discovery. It was first proposed in this paper published in 2005 by two Berkeley researchers:
The authors claim that, out of tens of thousands of genes, selecting just few of them (for instance 50) is sufficient for achieving the maximum level of accuracy for the task (predicting a disease). So the point is: how do we find the 50 lucky ones? This is exactly what MRMR was originally designed for.
From these first two examples, it’s evident how flexible is MRMR, which can prove useful in completely different domains, from marketing to genomics.
In this article we will see how MRMR works, and how it is different from other popular feature selection algorithms like Boruta (if you want to know more about Boruta, I’ve written a dedicated post: Boruta Explained Exactly How You Wished Someone Explained to You)
Feature selection algorithms can be broadly classified in two categories:
minimal-optimal (such as MRMR);
all-relevant (such as Boruta).
Minimal-optimal methods seek to identify a small set of features that — put together — have the maximum possible predictive power. On the other hand, all-relevant algorithms are designed to select all the features that — individually — have any predictive power at all.
Thus, if feature A and feature B are both relevant, but they bring more or less the same information, an all-relevant method will select them both, whereas a minimal-optimal method will select only one of them and discard the other one.
This may seem a suble distinction, but it makes a huge difference in practice. Imagine a scenario in which we have 10 thousand features. You use Boruta and you find out that 5 thousand of them have some kind of relationship with the target variable. But 5 thousand is still an unmanageable number of features.
In the example brought by Ding and Peng, you could likely achieve the same (or often higher) accuracy with a far smaller model, for instance 50 features. But, how to choose them? Taking the top 50 features from Boruta doesn’t work. In fact, many of them may be highly correlated with each other: they wouldn’t add much information.
MRMR was developed to overcome this issue.
Let’s take a toy example. Say that our task is to predict the income of some people, knowing some of their (or their relatives’) characteristics. We dispose of these features:
Let’s also imagine that we know the causal relationships between the variables (this is a privilege that we don’t have often in real life, however it’s useful for understanding the logic behind MRMR).
Knowing the causal network would make feature selection ridiculously easy: we would choose {Age, IQ, Height}, since these are the only ones that have a direct causal relationship with the target variable. Adding any of the others would throw in unnecessary noise.
However, these are not necessarily the features that — taken individually — have the strongest predictive value with respect to the target variable. This is summarised by saying that:
“The best K features are not the K best features.” (Adapted from T. M. Cover).
What does this mean?
For instance, take Mother’s IQ: it’s not directly linked to Income, but it’s linked with IQ that is, in turn, heavily linked to Income. So, if you test it, you will find a quite strong relationship between Mother’s IQ and Income. But since we have already IQ in our model, there’s no point in keeping also Mother’s IQ.
In general terms, this implies that it’s not enough to look at individual measures of association between each feature and the target variable.
We wish to find the best K features, not the K best features!
This is exactly the problem with Boruta: it provides an individual evaluation for each feature. Indeed, in our example, Boruta would select all the 7 features, since all of them have some statistical dependence with Income.
And this could make sense in cases where we don’t have too many features. But, in real applications, it often happens that:
we don’t know the causal relationships linking the features and the target variable;
we have too many features;
there is high redundancy within the features.
In these cases, it may be that an all-relevant algorithm such as Boruta is too “indulgent”, whereas MRMR is capable of removing unnecessary features.
So, now the question is: how does MRMR work?
When using MRMR, you are basically required to make only one choice: deciding the number of features that you want to keep. We will call this number K. In our example, we will take K=3.
In real applications, one can choose K based on domain knowledge or other constraints, such as model capacity, machine memory or time available.
MRMR works iteratively. At each iteration, it identifies the best feature (according to a rule) and adds it to the basket of selected features. Once a feature goes into the bucket, it cannot ever come out.
In our example, these are the outcomes that we would get at the end of each iteration:
What is the rule that determines the choice of the “Best” feature at each iteration — namely IQ at step 1, Age at step 2 and Height at step 3? This is the core of MRMR.
“Maximum Relevance - Minimum Redundancy” is so called because — at each iteration — we want to select the feature that has maximum relevance with respect to the target variable and minimum redundancy with respect to the features that have been selected at previous iterations.
In practice, at each iteration i, a score is computed for each feature to be evaluated (f):
The best feature at iteration i is the one having the highest score. As simple as that.
The only question now is how to calculate the numerator and the denominator. Based on this decision, the authors of Uber paper identify many variants of MRMR, such as MID, MIQ, FCD, FCQ, FRQ, RFCQ and RFRQ. However:
Since FCQ is embarassingly simple and fast and works generally better than the other algorithms, I will focus on this variant (by the way, extending to other variants is straightforward).
The relevance of a feature f at the i-th iteration (numerator) is computed as the F-statistic between the feature and the target variable (here to know more about the F-test). The redundancy (denominator) is computed as the average (Pearson) correlation between the feature and all the features that have been selected at previous iterations. So, the formula becomes:
where i is the i-th iteration, f is the feature that is being evaluated, F is F-statistic and corr is Pearson correlation. Note that correlation is taken in absolute value. In fact, if two features have correlation .9 or -.9 it makes no difference: in both cases they are highly redundant.
To make things even more explicit, imagine that we are at the 3rd iteration. This is what happened previously:
IQ has been selected at the 1st iteration;
Age has been selected at the 2nd iteration.
Which feature should we select next?
All we need to compute the score of each feature are the F-statistics and the correlations:
As we said above, the next best feature is the one with the highest score, in this case Height.
The iterative process that we have described above is easily implemented in Python:
This implementation is very easy to understand. However, it is suboptimal. Can you figure out why?
In the code above, we are calculating a lot of correlations that we will never use. In fact, at line 6, we are processing all the possible pairs of features. This means F * (F - 1) / 2 pairs. For F = 10,000, this would mean 50 million correlations!
We could save much more time by processing only the pairs of features that we need at each iteration, i.e. computing correlations “as we go”. This can save us a lot of time.
After all, at each iteration, we only need to compute all the correlations between the feature selected at the previous iteration and all the features that have never been selected, and store them in a correlation matrix. This means:
1st iteration: no correlations needed. In fact, no feature has been selected yet: it’s enough to select the feature having the highest score (i.e. the highest F-statistic).
2nd iteration: F - 1 correlations needed.
3rd iteration: F - 2 correlations needed.
...
Kth iteration: F - K + 1 correlations needed.
Now, doing the math is easy: this procedure requires to compute less than F * (K - 1) correlations. If F=10,000 and K=50, this leads to a much more reasonable number: less than 500 thousand correlations.
This improved version would look like this:
This is almost equivalent to the previous version, except for lines 11 and 21 – 23.
You can find a ready-to-use version of MRMR for classification problems in my Github. It can be installed through:
pip install mrmr_selection
This is a snippet of how you can use it on a Pandas dataframe:
from mrmr import mrmr_classiffrom sklearn.datasets import make_classification# create some dataX, y = make_classification(n_samples = 1000, n_features = 50, n_informative = 10, n_redundant = 40)X = pd.DataFrame(X)y = pd.Series(y)# use mrmr classificationselected_features = mrmr_classif(X, y, K = 10)
We have seen why MRMR is a useful feature selection step in many practical problems: because it tries to find a small set of features that are relevant with respect to the target variable and are scarcely redundant with each other.
MRMR is valuable not only because it is effective (as proved in Uber paper), but also because its simplicity makes it fast and easily implementable in any pipeline.
Thank you for reading! I hope you found this post useful.
I appreciate feedback and constructive criticism. If you want to talk about this article or other related topics, you can text me at my Linkedin contact.