2 Types of Duplicate Features in Machine Learning | by Samarth Agrawal | Towards Data Science

Two things distinguish top data scientists from others in most cases: Feature Creation and Feature Selection. i.e., creating features that capture deeper/hidden insights about the business or customer and then making the right choices about which features to choose for your model.
Feature Selection is the process of reducing the number of input variables when developing a predictive model.
After an extensive Feature Engineering step, you would end up with a large number of features. You may not use all the features in your model. You would be interested in feeding your model only those significant features or remove the ones that do not have any predictive power. It reduces the computational cost of model training and also improves the performance of the model.
In the previous post, we have seen how to detect constant and quasi-constant features.
This post is for identifying all the duplicate features. These can be of two types:
Duplicate Values: When two features have the same set of valuesDuplicate Index: When the value of two features are different, but they occur at the same index
Duplicate Values: When two features have the same set of values
Duplicate Index: When the value of two features are different, but they occur at the same index
As shown in the example image, the year of the sale for a car is the same as the manufacture year; then, these two features essentially say the same thing. Your machine learning model won’t learn anything insightful by keeping both these features in training. You are better off dropping one of the features. Likewise, there can be many more such features, and you need a programmatic way of identifying this.
1) Use the get_duplicate_features functions to get all the constant features.
2) Store all the duplicate features as a list for removing from the dataset.
3) Drop all such features from the dataset. We can see that the number of features has dropped to 251 from 301.
As shown in the example image — all the ‘Camry’ cars are from 2018 and all the ‘Corolla’ cars from 2019. There is nothing insightful for your machine learning model from these features in training. I can also do integer encoding for the Car Model to replace Camry with 2018 and Corolla with 2019. Then it is the same as case 1 above of Duplicate Values. You are better off dropping one of these two features. Likewise, there can be many more such features, and you need a programmatic way of identifying this.
1) Use the get_duplicate_features functions to get all the constant features.
2) Store all the duplicate index features as a list for removing from the dataset.
3) Drop all such features from the dataset. We can see that the number of features has dropped to 226 from 251.
Keeping duplicate features in your dataset introduces the problem of multicollinearity. — In the case of linear models, weights distribution between the two features will be problematic. — If you are using tree-based modes, it won’t matter unless you are looking at feature importance.  — In the case of distance-based models, it will make that feature count more in the distance.
Code Snippet for detecting features having Duplicate Values or Duplicate Index:
# Import required functionsfrom fast_ml.utilities import display_allfrom fast_ml.feature_selection import get_duplicate_features# Use the function to get the results in dataframeduplicate_features = get_duplicate_features(df)display_all(duplicate_features)# All the constant features stored in a listduplicate_features_list = duplicate_features['feature2'].to_list()# Drop all the constant features from the datasetdf.drop(columns = duplicate_features_list, inplace=True)
Package by the Data-Scientists for the Data Scientists ; with Scikit-learn type fit() transform() functionality
There are numerous inbuilt functionalities to make the life of a data scientist much easier
— Exploratory Data Analysis
— Missing Value Treatment
— Outlier Treatment
— Feature Engineering
— Feature Selection
— Model Development
— Model Evaluation
If you enjoyed this, follow me on medium for more.
Interested in collaborating? Let’s connect on Linkedin.
Please feel free to write your thoughts/suggestions/feedback.
Kaggle link
Notebook is available at the following location with fully functional code: