4 Categorical Encoding Concepts to Know for Data Scientists | by Cornellius Yudha Wijaya | Towards Data Science

The categorical data type is data divided by category or group — for example, Gender, Education, and Birth Place. Each element in the categorical data is used to group information on specific labels, different from numerical data where the numerical data information is in numbers. The problem with the unprocessed categorical data is the machine learning issue where many models cannot accept categorical data. This issue is why we need Categorical Encoding.
Categorical Encoding is a process where we transform categorical data into numerical data. There are many categorical encodings techniques, but I want to explain the basic process that every data scientist needs to know. In this article, I will outline four categorical encoding concepts to know for data scientists. Let's get into it.
One-Hot Encoding or OHE is a technique that transforms all the elements on a categorical column into new columns represented by 0 or 1 (binary values) to signify the presence of the category value. Let's see an example in the image below.
As we can see in the image above, a single categorical column (Origin) was transformed into three different numerical columns where each column represents elements from the categorical data. In the example, the values 'USA' would become the 'Origin_USA' column with 0 (not present) and 1 (present) after One-Hot Encoding.
One-Hot Encoding is helpful for categorical columns without any order (Ordinal) and less cardinality. Why no ordinal and less cardinal? because One-Hot Encoding treats all the values within the categorical column as equal, removing the ordinal information. Also, a high cardinal categorical column would produce many columns, leading to the curse of dimensionality.
One more note is the One-Hot Encoding might not suitable for Tree-based Machine Learning because OHE causes inefficiency during splitting. You might want to read it more in the article below.
towardsdatascience.com
Let's use a python code example to understand the concept better.
import pandas as pdimport seaborn as snsmpg = sns.load_dataset('mpg')mpg.head()
In this example, I would use the mpg dataset from the seaborn package. The data contains both categorical and numerical columns we could experiment with.
For the next step, I would use the Pandas package using a function called get_dummies to do the OHE.
pd.get_dummies(mpg, drop_first = True, columns = ['origin'] ).head()
I pass the 'drop_first' parameter to be True to drop the transformed first column in the function above. We do this because the information was redundant (dummy-variable trap), and the other two columns already contained the information of the dropped column.
In the previous section, we talked about One-Hot Encoding, where we transformed the categorical column into many different numerical columns. Label Encoding was different — the process transforms categorical column elements into numerical values without any additional column. Let's see the process in the image below.
As we can see from the image above, the categorical column transformed into a numerical value (Primary School = 1, High School = 2, and so on).
The process is easy and informative. However, Label Encoding is mostly only applicable for the Ordinal or categorical data with meaningful order. In my example above, I assume that the 'Education' data is Ordinal data because there is a significant order between the values. Additionally, Label Encoding would produce an equal space between the numerical value — if you feel your categorical data order is not similar, then Label Encoding might not capture that information.
Why is Label Encoding not applicable to the non-ordinal data? Because it causes a prioritization issue. The machine learning model would often generalized high value with a higher priority than the lower value — and if our categorical value doesn't have any meaningful order but treat it as it has one, it would distort the information.
I am often creating my function but let's try to use the Sklearn Python package for the Label Encoding process.
from sklearn import preprocessingle = preprocessing.LabelEncoder()mpg['origin_le'] = le.fit_transform(mpg['origin'])mpg.sample(5)
The Label Encoding transforms the 'origin' column into a numerical one. The order of the alphabet decided the values; that is why 'europe' values were assigned as 0 and 'USA' as 2.
However, as you can see that the Label Encoder from Sklearn substitutes the data only by using the alphabetical information, which might be wrong in the ordinal data case. That is why I often create my transformer rather than relying on the package's function.
Although, if you want to rely on Sklearn; it provides OrdinalEncoder class that could transform the category into the label we assigned.
One-Hot Encoding's major weakness is the features it produced are equivalent to the categorical cardinal, which causes dimensionality issues when the cardinality is too high. One way to alleviate this problem is to represent the categorical data into a lesser number of columns, and that is what Hash Encoding did.
Hash Encoding represents the categorical data into numerical value by the hashing function. Hashing is often used in data encryption or data comparison, but the main part is still similar — transform one feature to another using hashing function.
I would not explain in deep regarding the hashing process, but you could read this following paper to understand one of the most used hashing functions, md5.
The main advantage of using Hash Encoding is that you can control the number of numerical columns produced by the process. You could represent categorical data with 25 or 50 values with five columns (or any number you want). Let's try to do Hash Encoding with a coding example.
First, let's install a particular package for categorical Encoding called category_encoders .
pip install category_encoders
This Python package contains many functions for the Categorical Encoding process and works well with the Featuretools package (The category_encoders package developed to work with the Featuretools). You could check my following article to know what is Featuretools did.
towardsdatascience.com
Using the category_encoders, let's try to Hash Encode the category data in the sample mpg dataset. Our dataset has 'model_year' data with 13 cardinal, and I want to transform it into five numerical features. To do that, we could try the following code.
import category_encoders as ceencoder=ce.HashingEncoder(cols='model_year',n_components=5)hash_res = encoder.fit_transform(mpg['model_year'])hash_res.sample(5)
Hash Encoding would return numerical features that represent the categorical data. Let's concatenate the Hash Encoding result with the original data to have a comparison.
pd.concat([encoder.fit_transform(mpg['model_year']), mpg], axis =1).sample(5)
As we can see from the image above, the 'model_year' data have been transformed into numerical values 0 or 1, and the Machine Learning model could use the data for training.
However, Hash Encoding has two significant weaknesses. First, because we transform the data into fewer features, there would be an information loss. Second, since a high number of categorical values are represented into a smaller number of features, different categorical values could be represented by the same Hash values — this is called a collision.
But, many Kaggle competitors use Hash Encoding to win the competition, so it is worth a try.
Leave One Out Encoding is similar to Target Encoding, but it adds one more step to handle overfitting. To understand the LOOE process, let me try to explain the target encoding concept first. The target encoding technique is a categorical encoding that transforms the categorical data into numerical value using the target (y) information. Let's use a sample image to understand the concept better.
We have 'Pclass' as our independent feature and 'Survived' as the target in the image above. When applying the Target Encoding technique, it transforms the Class 2 values into 0.5, Class 3 into 0, and Class 1 into 1. This transformation base on the information of the 'Survived' data or the target.
The process is simple — the total number of survived classes (1) for each element (Class 1, Class 2, Class 3) are divided by the total number for each class. Class 2 has 1 survived and 1 not-survived, which is reflected as 1/2 or 0.5 when using the Target Encoding technique.
Leave One Out Encoding has an exact approach with the Target Encoding except that LOOE excludes the current row's target in the calculation to ease the outlier effect. This means the calculation result between the True and False classes of the target could be different for each class. LOOE function could also introduce the Gaussian noise distribution to decrease overfitting.
Let's use a code example to understand the concept even better. I would use the titanic dataset from seaborn for this sample.
titanic = sns.load_dataset('titanic')#Passing value 0.1 at sigma parameter to introduce noiseloo_encoder=ce.LeaveOneOutEncoder(cols='pclass', sigma = 0.1)loo_res = loo_encoder.fit_transform(titanic['pclass'], titanic['survived']).rename(columns = {'pclass': 'loo_pclass'})pd.concat([loo_res,titanic], axis =1).sample(5)
The image above shows that the 'loo_pclass' feature is the Categorical Encoding based on the independent and the target. However, each 'loo_pclass' has a different value even with the same values in the target and independent. This is because we introduce the noise during the categorical encoding process.
Do not forget that we use LOOE based on the target data, specifically after splitting the data into training and test data. The training data is where we fit the LOOE and transform both the training and the test data.
Categorical data is inherently different from numerical data and needs a preprocessing step before using it for machine learning modeling problems.
In this article, I outline 4 Categorical Encoding techniques that Data scientists need to know; they are:
One-Hot EncodingLabel EncodingHash EncodingLeave One Out Encoding
One-Hot Encoding
Label Encoding
Hash Encoding
Leave One Out Encoding
I hope it helps!
If you enjoy my content and want to get more in-depth knowledge regarding data or just daily life as a Data Scientist, please consider subscribing to my newsletter here.
If you are not subscribed as a Medium Member, please consider subscribing through my referral.
Visit me on my LinkedIn or Twitter.