Stop Using Pandas get_dummies() for Feature Encoding | by Satyam Kumar | Towards Data Science

Machine learning algorithms require the input data source in a specific format of numerical vectors. Feature engineering is an important component of a data science model development life cycle, which refers to converting the raw data to numerical format fit for training a robust model.
A data scientist spends about 80% of the time on data preparation and feature engineering. The performance of the model depends on the feature engineering strategies. The raw dataset contains various data types of features including numerical, categorical, date time, etc. There are various feature-engineering techniques that convert different data types of data features to numerical vectors.
Dummy Encoding refers to an encoding strategy to convert a categorical feature to a numerical vector format. There are various other techniques to encode a categorical feature including Count Encoder, One Hot Encoder, Tf-Idf Encoder, etc.
pd.get_dummies() is a function from Pandas that performs dummy encoding in a single line of code. Data scientists mostly use this for feature encoding, but it’s not recommended to use it in production or Kaggle competitions. In this article, we will discuss the reason behind it and what the best choice is for the get_dummies() function.
The get_dummies() function from the Pandas library can be used to convert a categorical variable into dummy/indicator variables. It is in a way a static technique for encoding in its behavior.
We will take a random dataset with 2 numerical and 1 categorical feature (‘color’) for further demostration. The ‘color’ categorical variable has 3 unique categories: green, red, blue.
You can observe the encoded results for pd.get_dummies() . The usage of this function is not recommended in production or on Kaggle as it is in a way static in nature in its behavior. It cannot learn the characteristics from the training data and hence is unable to propagate its findings onto the test dataset.
The categorical feature color has 3 feature values: green, red, blue. In the training sample that causes encoding the color feature into 3 feature categories. But the test data may or may not have all the feature values, which may cause data mismatch issues while modeling.
The blue feature value is missing in the color feature, which causes the absence of the blue feature column in the encoded data. This is caused because pd.get_dummies does not learn the characteristics of training data and will further cause feature mismatch while prediction.
#to print the encoded features for train datapd.get_dummies(X_train)#to print the encoded features for test datapd.get_dummies(X_test)
The only advantage of pd.get_dummies() is its easy interpretability, and the fact that it returns a pandas data frame with clean column names.
One-hot Encoder is a popular feature encoding strategy that performs similar to pd.get_dummies() with added advantages. It encodes a nominal or categorical feature by assigning one binary column per category per categorical feature. Scikit-learn comes with the implementation of the one-hot encoder.
From the above results of One hot encoder for training and test data, one can observe that the characteristics of encoding are saved while encoding the categorical variable for training data and its finding or feature values are propagated while encoding the test data.
Although the blue feature values were not present in the test data, one hot encoder created a feature column for the blue category, as it was present in the training pipeline.
from sklearn.preprocessing import OneHotEncoder# one hot encodingenc = OneHotEncoder(sparse=False)color_onehot = enc.fit_transform(X_train[['color']])#to print the encoded features for train datapd.DataFrame(color_onehot, columns=list(enc.categories_[0]))# tranform encoding for test datatest_onehot = enc.transform(X_test[['color']])#to print the encoded features for train datapd.DataFrame(test_onehot, columns=list(enc.categories_[0]))
pd.get_dummies() returns a pandas data frame with clean column names after feature encoding, still, it's not recommended to use it for production or in Kaggle competitions. One-hot encoded or Count Vectorizer strategies should be preferred as it carries the characteristics of feature values for each categorical features.
New feature categories can also be handled using handle_unknown=’ignore’ parameter for One hot encoder, which can further cause data mismatch issues in pd.get_dummies()
[1] Scikit-learn documentation: https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html
Thank You for Reading