Why you shouldn't be using LabelEncoder for categorical features | by Àlex Escolà Nixon | Towards Data Science

When dealing with categorical features choosing an appropriate way to encode them might not be so straight forward. This is a very important aspect to take into account, which will determine the value that a feature will be adding to a model.
The same applies to ordinal features, which are a very common type of categorical data in which its different categories present a natural order. An example could be a temperature feature taking the categories cool, mild and hot.
A fairly common mistake is to just assign a unique numerical value to each category of a feature, disregarding any order it may have. And similarly with nominal features; with the difference that they don’t naturally present any order.
The idea in this post is to explore and understand why this is not a good idea, using an example with ordinal features to better illustrate the point, and which other approaches we should be using instead.
Find all the code along with explanations also as a jupyter notebook.
Mapping the inherent ordinality of a feature to a wrong scale will have a very negative impact on the performance of the model (that is, proportional to the relevance of the feature).
As mentioned, ordinal features have a natural ordering, or to put it another way, they have an ordinal scale with a ranking. So if we want to preserve the features’ value after encoding it, we want to make sure that it resembles their ranking.
An intuitive way to think about it, is by the way a decision tree sets its thresholds. During training, a decision tree will learn the optimal features to set at each node, as well as an optimal threshold which will determine the path unseen samples will follow through each node.
If we encode an ordinal feature using a simple LabelEncoder, that could lead to a feature having say 1 represent warm, 2 which maybe would translate to hot, and a 0 representing boiling. In such case, the result will end up being a tree with an unnecessarily high amount of splits, and hence a much higher complexity for what should be simpler to model.
Though actually seeing why this is a bad idea will be more intuitive than just words (decision trees to the rescue :). Let’s use a simple example to illustrate the above, consisting on two ordinal features containing a range with the amount of hours spend by a student preparing for an exam and the average grade of all previous assignments, and a target variable indicating whether the exam was past or not. I’ve defined the dataframe’s columns as pd.Categorical:
The advantage of defining a categorical column as a pandas’ categorical, is that we get to establish an order among its categories. This allows for much faster sorting based on the established order rather than lexical sorting. And it can also be used as a simple way to get codes for the different categories according to their order.
That being said, for a more general approach with fit / transform methods we should be looking at other tools, which I will go through in the next section.
So the dataframe we’ll be using looks as follows:
   Hours_of_dedication   Assignments_avg_grade  Result0                20-25                       B    Pass1                20-25                       C    Pass2                 5-10                       F    Fail3                 5-10                       C    Fail4                40-45                       B    Pass5                  0-5                       D    Fail6                15-20                       C    Fail7                20-25                       A    Pass8                30-35                       B    Pass9                 5-10                       B    Fail10               10-15                       D    Fail...
And as mentioned, we can obtain the assigned codes to each category according to their ordering in the categories parameter using the pd.Series.cat.codes method:
X = df.apply(lambda x: x.cat.codes)print(X)    Hours_of_dedication  Assignments_avg_grade  Result0                     4                      3       11                     4                      2       12                     1                      0       03                     1                      2       04                     7                      3       15                     0                      1       06                     3                      2       07                     4                      4       18                     6                      3       19                     1                      3       010                    2                      1       0...
Now let’s fit a DecisionTreeClassifier, and see how the tree has defined the splits:
from sklearn import treedt = tree.DecisionTreeClassifier()y = X.pop('Result')dt.fit(X, y)
The sklearn.tree module provides a function to plot a fitted decision tree. Let’s use it to see what the decision tree looks like:
g = tree.plot_tree(dt,                    feature_names = X.columns,                   class_names=['Failed', 'Pass'],                   filled = True,                   label='all',                   rounded=True)
Is that all?? Well... yes! I’ve actually set the features in such a way that there is this simple and obvious relation between the Hours of dedication feature and whether the exam is passed or not, making it clear that the problem should be very easy to model.
Now let’s try to do the same by directly encoding all features with an encoding scheme we could have obtained for instance through a LabelEncoder, so disregarding the actual ordinality of the features, and just assigning a value at random:
from sklearn.preprocessing import LabelEncoderX_wrong = df.apply(LabelEncoder().fit_transform)dt_wrong = tree.DecisionTreeClassifier()dt_wrong.fit(X_wrong, y)g = tree.plot_tree(dt_wrong,                    feature_names = X_wrong.columns,                   class_names=['Fail', 'Pass'],                   filled = True,                   label='all',                   rounded=True)
As expected the tree structure is way more complex than necessary for the simple problem we’re trying to model. In order for the tree to correctly predict all training samples it has expanded until a depth of 4, when a single node should suffice.
This will imply that the classifier is likely to overfit, since we’re drastically increasing the complexity. And by pruning the tree and tuning the necessary parameters to prevent overfitting we are not solving the problem either, since we’ve added too much noise by wrongly encoding the features.
In the example above, we’ve seen a simple way to encode ordinal features using pd.Categorical. But what we’ll be interested in most of the times is not only to transform a given set of features, but also to replicate the applied transformations on unseen data.
For that we have the OrdinalEncoder from Category Encoders. This class enables us to:
Define a dictionary mapping categories to codes
Fit the encoder with the mapping dictionary and transform on unseen data
Let’s see how to use it with an example, containing several features:
The OrdinalEncoder receives a mapping parameter expecting a list of dictionaries, each of which containing the keys col and mapping:
Then we can create a new instance of the encoder with:
import category_encoders as ceencoder = ce.OrdinalEncoder(mapping = ordinal_cols_mapping,                              return_df = True)
So by transforming on the same dataframe as above, we get:
df_train = encoder.fit_transform(df)print(df_train)    Outlook  Temperature  Humidity  Windy  PlayTennis0         2            2         1      0           01         2            2         1      1           02         1            2         1      0           13         0            1         1      0           14         0            0         0      0           15         0            0         0      1           06         1            0         0      1           17         2            1         1      0           08         2            0         0      0           19         0            1         0      0           110        2            1         0      1           111        1            1         1      1           112        1            2         0      0           113        0            1         1      1           0
So to summarise, preserving the ordinality of the features once encoding them is crucial, otherwise as made clear with this example we’ll lose all their predictable power and just add noise to our model.
Find all the code along with explanations also as a jupyter notebook. Hope you enjoyed and found this helpful!