How to Use Variance Thresholding For Robust Feature Selection | by Bex T. | Towards Data Science

Today, it is common for datasets to have hundreds if not thousands of features. On the surface, this might seem like a good thing — more features give more information about each sample. But more often than not, these additional features don’t provide that much value and introduce unnecessary complexity.
The biggest challenge of Machine Learning is to create models that have robust predictive power by using as few features as possible. But given the massive sizes of today’s datasets, it is easy to lose the oversight of which features are important and which ones aren’t.
That’s why there is an entire skill to be learned in the ML field — feature selection. Feature selection is the process of choosing a subset of the most important features while trying to retain as much information as possible.
As an example, let’s say we have a dataset of body measurements such as weight, height, BMI, etc. Basic feature selection techniques should be able to drop BMI by finding out that BMI can be represented by weight and height.
In this article, we will explore one such feature selection technique called Variance Thresholding. This technique is a quick and lightweight way of eliminating features with very low variance, i. e. features with not much useful information.
For those who are not familiar, variance, as the name suggests, shows the variability in a distribution in a single metric. It shows how spread out the distribution is and shows the average squared distance from the mean:
Obviously, distributions with bigger values yield a bigger variance because each difference is squared. But the main thing we care about in ML is that the distribution actually contains useful information. For example, consider this distribution:
Computing the variance with Numpy shows us that the distribution has 0 variance or in other words completely useless. Using a feature with zero-variance only adds to model complexity, not to its predictive power. Consider another one:
Similarly, this one is almost made up of a single constant. Distributions that go around a single constant with a few exceptions are also useless. In other words, any feature or distribution with close to 0 variance should be dropped.
Manually computing variances and thresholding them can be a lot of work. Fortunately, Scikit-learn provides VarianceThreshold estimator which can do all the work for us. Just pass a threshold cut-off and all features below that threshold will be dropped.
To demonstrate VarianceThreshold, we will be working with the Ansur dataset. This dataset records measurements of the human body in every imaginable way. Both male and female datasets contain 108 features or measurements of almost 6000 (4000 male, 2000 female) US Army Personnel. We will be focusing on the male dataset:
First, let’s get rid of the features with zero-variance. We will import VarianceThreshold from sklearn.feature_selection:
We initialize it just like any other Scikit-learn estimator. The default value for the threshold is always 0. Also, the estimator only works with numeric data obviously and it will raise an error if there are categorical features present in the dataframe. That’s why, for now, we will subset the numeric features into another dataframe:
So, we have got 98 numeric features. Let’s now fit the estimator to the data and get its results:
Directly calling fit_transform will return the dataframe as a numpy array with features dropped. But sometimes, we don't want the result in that format because the column names will be removed. Consider the alternative:
First, we fit the estimator to data and call its get_support() method. It returns a boolean mask with True values for columns which are not dropped. We can then use this mask to subset our DataFrame like so:
Let’s check the shape of the DataFrame to see if there were any constant columns:
>>> ansur_male_num.shape(4082, 98)
Nope, we still have the same number of features. Now, let’s drop features with variances close to 0:
With a threshold of 1, only 1 feature got dropped.
Often, it is not fair to compare the variance of a feature to another. The reason is that as the values in the distribution get bigger, the variance grows exponentially. In other words, the variances will not be on the same scale. Consider this example:
The above features all have different medians, quartiles, and ranges — completely different distributions. We cannot compare these features to each other.
One method we can use is normalizing all features by dividing them by their mean:
This method ensures that all variances are on the same scale:
Now, we can use the estimator with a lower threshold like 0.005 or 0.003:
As you can see, we were able to drop 50 features from the dataset. Now, let’s test if we did the right thing by dropping so many features.
We will check this by training two RandomForestRegressor to predict a person’s weight in pounds: the first one on the final, feature selected dataset and the second one on the full, numeric-feature only dataset.
Both training and test score suggest a really high performance without overfitting. Now, let’s train the same model on the full numeric-only dataset:
As you can see, even by dropping 50 features we were able to build a pretty powerful model.
Even though Variance Thresholding is a simple method, it can go a long way when performing a feature selection. However, keep in mind that this technique does not take into account the relationship between features or the connection between features and target. Therefore, always double-check that using VT brings performance increase or at least lowers model complexity by doing something like we did with RandomForestRegressor.
Also, check out the official user guide on feature selection by Scikit-learn — there, you can learn how to insert VT estimators in pipeline instances. The guide contains information on other feature selection techniques as well.
If you don’t know what to read next, here, I have picked some for you: