5 Categorical Encoding Tricks you need to know today as a data scientist | by Philippe Bouaziz,Data Scientist at Africa4Data,PhD | Towards Data Science

Categorical encoding is a fundamental step in exploratory data analysis (EDA) for most advanced data science projects whether your developing machine learning models or business analytics. Most libraries including scikit-learn use a variety of data encoding methods. Due to the high quantity of data, finding tricks for getting the best categorical encoding in python is a massive advantage for becoming a unicorn data scientist. In this article, we will review the 5 most successful open source short python code lines which can be combined for making a first-class categorical encoding report.
For this article, we will be analyzing the sample chocolate bar rating dataset you can find here: chocolate-bar.
Introduction
Many scenarios are dealing with categorical data, the classical method consists of detecting cells with categorical values, count their numbers, and understanding their types (binary, multi-values, ...). Five mains methods can be found in most Kaggle competitions, let us review them:
1. One-hot encoding
One Hot Encoding is a process known as encoding categorical variables into dummy variables. This data processing method convert binary categorical columns (yes/ no, male/female,...) into 0/1 binary vector in which 0 indicates the absence of the row belonging to that category. This method can be tricky if used for non-binary multidimensional variables that will result in adding non-useful columns. For instance, if we were to have a column representing x colors one hot encoding would result in x additional columns (colors_green, colors_blue,...).
The optimal categorical variable for hot encoding consists of independent features of 2–4 values, for more dimensions alternatives methods can be easily applied.
The full documentation can be found on GitHub. For now, let’s start with the case at hand, loading it as so:
# import librariesimport pandas as pd# Hot encoding the variables salt, sugar, sweetener_without_sugara = pd.get_dummies(choc.salt)b = pd.get_dummies(choc.sugar)c = pd.get_dummies(choc['sweetener_without_sugar'])# Concatenate new variables with the dataframechoc = pd.concat([choc,a,b,c], axis = 1)# Drop non encoded binary variableschoc = choc.drop(['salt', 'sugar','sweetener_without_sugar'], axis = 1)choc
2. Count encoding
Count encoding consists of filling a categorical value in a column with his count in the categorical columns. For instance, if the value "green" appears 5 times in the colors column, then each "green" would be replaced with the number 5.
The optimal categorical variable for count encoding consists of independent features of more than 4 values. The full documentation can be found on GitHub. Let’s start :
# Create the encoderimport category_encoders as cecat_features = ['lecithin']count_enc = ce.CountEncoder()# Transform the features, rename the columns with the _count suffix, and join to dataframecount_encoded = count_enc.fit_transform([cat_features])data = choc.join(count_encoded.add_suffix("_count"))data
3.Leave-one-out(Loo) encoding
Leave-one-out Encoding is a label-based encoder. This method consists of replacing a label value in a categorical column of the test dataset with the label mean of value A for value B if value B is removed from the dataset. Encoded values range create a model that tends to generalize better. The full documentation can be found on GitHub. Let’s code :
a = choc.drop('review_date', axis = 1)X = a.iloc[:,0:2800]y = choc.iloc[:,-1]# Split the datafrom sklearn.model_selection import train_test_splitX_train, X_test,y_train,y_test = train_test_split(X, y, test_size=0.3)# Create the encoderfrom category_encoders import TargetEncoderenc = TargetEncoder(cols=['country_of_bean_origin'])# Fit the encodertraining_set = enc.fit_transform(X_train, y_train)training_set
4. Label Encoding
This approach involves converting each value in a column to a number. Consider a dataset of bridges having a column names bridge-types having below values. Though there will be many more columns in the dataset, to understand label-encoding, we will focus on one categorical column only. The full documentation can be found on GitHub. Let’s code :
5. Weight of Evidence
This method measures the separation between customers who defaulted or paid back on a loan. This method of supervised encoding considers the target (label) variable for improving the quality of prediction. The full documentation can be found on GitHub. Let’s code :
# Create the encoderfrom category_encoders import WOEEncoderenc = WOEEncoder(cols=['specific_bean_origin_or_bar_name','company_location','country_of_bean_origin','beans','company'])#Fit the encodertraining_set = enc.fit_transform(X_train, y_train)training_set
If you have some spare time I’d recommend, you’ll read this:
towardsdatascience.com
Sum Up
Refer to this link categorical encoding for a complete categorical encoding of chocolate bar rating using these tools.
This brief overview is a reminder of the importance of using the right categorical encoding methods using python in data science. This post has the scope to covered 5 essential Python categorical encoding tools for making a complete exploration workflow, as well as useful documentation.
I hope you enjoy it, keep exploring!!!