How to Mitigate Overfitting with Feature Selection | by Rukshan Pramoditha | Towards Data Science

This is the final part of the “Addressing the problem of overfitting” article series. Today, we’re continuing from Part 4. Last time, in the 3rd step of Part 4, we’ve built a random forest model on the “heart disease” dataset. There, we used all 13 features of the dataset. Today, we’ll modify that model by considering only the most important features.
First, we visualize the feature importances of that random forest model by using the feature_importances_ attribute.
It is very clear that not all features contribute to the model. Some features are not important to the model and we can remove them.
When we remove the least important features from the model, we reduce the model’s complexity and some noise in the data. Doing so will help to further mitigate overfitting — By author
The x-axis of the above graph contains the feature importance scores for each feature. We can specify a threshold value (here, we choose 0.025) for the importance score such that all features that have scores under that threshold value to be eliminated. For that, we can use the Scikit-learn SelectFromModel class.
from sklearn.feature_selection import SelectFromModelselector = SelectFromModel(rfclf, threshold=0.025)features_important = selector.fit_transform(X, y)
This returns a 2D-Numpy array that has values of only the important features decided by the threshold value. By converting it to a Pandas dataframe,
pd.DataFrame(features_important).head()
We can obtain the following table:
If we also get the original dataframe,
X.head()
We’ve removed 4 features: fbs, restecg, trestbps and chol. These features do not give enough contribution to the model. Keeping them in the model will increase the model’s complexity and that will result in the model overfitting the data.
The order of the importance scores of removed features is:
fbs < restecg < trestbps < chol
If we consider the feature “sex”, its importance value is more than twice the value of “chol”. If we also remove the feature “sex”, the model will lose some important data. That will result in the model underfitting. That’s why we’ve chosen 0.025 for the threshold value that selects the features that we want to keep in the model. You can easily decide the value for the threshold by looking at the feature importance graph.
Finally, we can use the modified dataframe with removed features (Dataframe 1) to build the random forest model again. This time, we use only 9 features. In other words, we can say that we’ve reduced the dimensionality of data. But, keep in mind that the original values in the dataset remain unchanged. You can compare this with Part 3 of the series where we’ve discussed a dimensionality reduction method that finds a new set of features that contains different values than in the original dataset.
The feature selection method that we discussed today can be applied to tree-based models such as decision trees, random forests, etc. For linear regression and logistic regression models, backward elimination and forward selection are mostly preferred. If you want to learn them in detail, read this article written by me.
This is the end of today’s part and also the “Addressing the problem of overfitting” article series. We’ve discussed 5 different techniques to address the problem of overfitting under 5 articles. As additional content, I’ll add some guidelines to select the best technique.
So far, we’ve discussed the following techniques that can be used to mitigate overfitting:
Cross-validation
Regularization
Dimensionality Reduction
Creating Ensembles
Feature Selection
You do not need to apply all these techniques to mitigate overfitting. Here are some guidelines that will help you to select the best technique.
First, you need to realize that your model is overfitting before applying any technique. If you get a higher train score and relatively a much lower test score, your model overfits the data.
It is always better to evaluate the model by using Cross-validation that helps you to detect plenty of options to mitigate overfitting.
It is better to combine Cross-validation with a hyperparameter tuning technique like Grid Search or Random Search that finds a set of optimal values for defined hyperparameters. That will help you to mitigate overfitting. Hyperparameter tuning can be done with any model. For example, we can use it to limit the growth of a decision tree. That process is a type of Regularization discussed in Part 2.
The other type of regularization is adding a regularization term to the loss function. This works well for linear models such as logistic regression, linear regression as well for tree-based models such as XGBoost. For logistic regression, the L2 variant of regularization is applied by default in Scikit-learn. L1 variant is also available. For linear regression, there is no option to specify the regularization in the model itself. However, Scikit-learn provides 3 different options for that. When we apply L1 regularization to linear regression, it is called Lasso regression. When we apply L2 regularization to linear regression, it is called Ridge regression. When we apply both L1 and L2 variants at the same time to linear regression, it is called ElasticNet regression.
Dimensionality reduction techniques can be applied to almost any dataset. The most popular DR technique is the Principal Component Analysis (PCA). It has many advantages other than addressing the problem of overfitting.
If you want to find non-linear patterns in the data, you can use Ensemble techniques such as random forests and XGBoost models rather than using a single decision tree.
You can also combine two techniques together for better results. For example, you can create Ensemble models while applying regularization to limit the growth of individual trees. You can create Ensemble models while selecting the most important features in the dataset.
After creating a model, you can identify the most important features and remove the unnecessary features from the dataset and rebuild the model by using only the most important features. For Ensemble models like the random forest, XGBoost, this can be easily done using the feature_importances_ attribute. For linear regression and logistic regression models, backward elimination and forward selection can be done using separate classes.
Now, you’ll have hands-on experience for applying 5 different techniques to address the problem of overfitting. I suggest you apply them to different datasets. You can apply more than one technique to a dataset and see the outputs and then choose the best one by looking at the outputs.
For your convenience, I’ve decided to make a list that consists of all the parts of this article series plus another useful post. Now, you can access all of them in one place by clicking on the following image.
This is the end of today’s post. My readers can sign up for a membership through the following link to get full access to every story I write and I will receive a portion of your membership fee.
Sign-up link: https://rukshanpramoditha.medium.com/membership
Thank you so much for your continuous support! See you in the next story. Happy learning to everyone!
Special credit goes to Mockup Photos on Unsplash, who provides me with a nice cover image for this post (I did some modifications to the image: added some text on it and removed some parts).