All About Target Encoding For Classification Tasks | by Nishant Mohan | Towards Data Science

Recently I did a project wherein the target was multi-class. It was a simple prediction task and the dataset involved both categorical as well as numerical features.
For those of you who are wondering what multi-class classification is: If you want to answer in ‘0 vs 1’, ‘clicked vs not-clicked’ or ‘cat vs dog’, your classification problem is binary; if you want to answer in ‘red vs green vs blue vs yellow’ or ‘sedan vs hatch vs SUV’, then the problem is multi-class.
Therefore, I was researching suitable ways to encode the categorical features. No points for guessing, I was taken to medium articles enumerating benefits of mean target encoding and how it outperforms other methods and how you can use category_encoders library to do the task in just 2 lines of code. However, to my surprise, I found that no article demonstrated this on multi-class target. I went to the documentation of category_encoders and found that it does not say anything about supporting multi-class targets. I dug deeper, scouring through the source code and realized that the library only works for binary or continuous targets.
So I thought: “Inside of every problem lies an opportunity.” — Robert Kiposaki
Going deep, I went straight for the original paper by Daniele Micci-Barreca that introduced mean target encoding. Not only for regression problem, the paper gives the solution for both binary classification as well as multi-class classification. This is the same paper that category_encoders cites for target encoding as well.
While there are several articles explaining target encoding for regression and binary classification problems, my aim is to implement target encoding for multi-class variables. However, before that, we need to understand how it’s done for binary targets. In this article, I cover an overview of the paper that introduced target encoding, and show by example how target encoding works for binary problems.
Here is what the paper says for categorical target, in a nutshell:
(Skip this section if you wish to understand through an example)
One can map every occurrence of a category to the probability estimate of the target attribute. In a classification scenario, the numerical representation corresponds to the posterior probability of the target, conditioned by the value of the categorical attribute. This is effectively a numerical representation of the expected value of the target, given the value of the categorical attribute. In order to avoid overfitting due to a small number of observations in a category, smoothing of the means is also applied.
Let’s look at this practically. In binary problem the target is either 0 or 1. Then, the probability estimate for a category within a categorical variable can be given by Empirical Bayesian probability, P(Y=1|X=Xi), i.e.
where n(TR) is the total row count, n(Y) is the total row count with 1’s in target, n(i) is the count of rows with i-th category and n(iY) is the count of rows with target 1 in i-th category . Therefore, the fraction in the first term represents probability of 1 in the i-th category, while the fraction in second term represents probability of 1 in overall data. λ is a function which gives monotonic weight which helps when we have a small number of several categories, increasing with n(i) count from 0 to 1.
If you have used TargetEncoder from category_encoders library, k is the ‘min_sample_leaf’ parameter, and f is the ‘smoothing’ parameter.
Introducing the weighting factor makes sense because when the sample size is large, we should assign more credit to the posterior probability estimate provided by the first term above. However, if the sample size is small, then we replace the probability estimate with the null hypothesis given by the prior probability of the dependent attribute (i.e. mean of all Ys). With this transformation, missing values are handled by treating them as just another variable.
The paper extends the same concept to multi-class target as well. We have one new feature for each class of target. Each feature is then effectively a numerical representation of the expected value of exactly one class of target, given the value of the categorical attribute. Of course, the number of inputs can increase significantly if the target has a very high number of classes. However, practically the number of classes is usually small. Read more on this in my next article.
In the next section, we look at binary classification situation, with an example.
Let’s look at a binary target example.
What is the probability of Target being 1 if the category of feature Gender is ‘Male’?
It’s 1/2=0.5.
Similarly, what is the probability of Target being 1 if the category of feature Gender is ‘Female’?
It’s 1/4=0.25.
Simple enough?
Wait, now if you replace all occurrences of ‘Female’ with 0.25, you risk what is called overfitting. This is because you did not consider that overall probability of 1’s is not 0.5. It’s 4/9=0.4.
To account for this fact, we add a weight to this ‘prior’ information, using the formula shown in the previous section.
Setting min_sample_leaf, k=1 and smoothing, f=1,
for ‘Male’, we have two rows, so n=2;
λ(‘Male’)=1/(1+exp(-(2–1)/1))=0.73    # Weight Factor for 'Male'  Target Statistic=(Weight Factor * Probability of 1 for Males)                  + ((1-Weight Factor) * Probability of 1 Overall)S(‘Male’)=(0.73 * 0.5) + ((1–0.73) * 0.4) = 0.485
Similarly, for ‘Female’, we have four rows, so n=4;
λ(‘Female’)=1/(1+exp(-(4–1)/1))=0.95    # Weight Factor for 'Female'Target Statistic=(Weight Factor * Probability of 1 for Females)                  + ((1-Weight Factor) * Probability of 1 Overall)S(‘Female’)=(0.95 * 0.25) + ((1–0.95) * 0.4) = 0.259
If you look closer, you may notice that λ helps giving importance to those categories for which number of rows are more. In our example, there were 4 rows of Gender ‘Female’, as opposed to only 2 rows of Gender ‘Male’. The corresponding Weight Factor were 0.95 and 0.73.
Thus, replace all occurrences of ‘Male’ with 0.485 and ‘Female’ with 0.259. Value for ‘Other’ can be calculated in a similar fashion.
Congratulations! You just implemented target encoding.
Don’t trust me?
Check for yourself by running this code, which does the same using your favorite category_encoders library:
!pip install category_encodersimport category_encoders as cex=['Male','Male','Female','Female','Female','Female','Other','Other','Other']y=[1,0,0,0,0,1,1,1,0]print(ce.TargetEncoder().fit_transform(x,y))
In this article, I introduced a shortcoming of the TargetEncoder class of category_encoders library. I explained and summarized the paper that introduced target encoding. I explained through an example how the categories can be replaced with a numerical value.
My next article will focus completely on how we can do the same for multi-class targets. Stay tuned!
Edit: Published the next one, demonstrating and implementing multi-class target encoding here.
Connect with me on LinkedIn
Check out some of my interesting projects on GitHub