Categorical Feature Encoding in Python | Towards Data Science

Categorical data is a common type of non-numerical data that contains label values and not numbers. Some examples include:
Colors: Red, Green, Blue
Cities: New York, Austin, Denver
Gender: Male, Female
Place: First, Second, Third
According to Wikipedia, “a categorical variable is a variable that can take on one of a limited, and usually fixed number of possible values.”
It is common to refer to a possible value of a categorical variable as a level.
There are several different types of categorical data including:
Binary: A variable that has only 2 values. For example, True/False or Yes/No.
Ordinal: A variable that has some order associated with it like our place example above.
Nominal: A variable that has no numerical importance, for example color or city.
Many machine learning algorithms cannot work with categorical data directly. They require data to be numeric. Therefore, it is essential to know how to encode categorical variables.
We will use data from Kaggle’s Categorical Feature Encoding Challenge II. This challenge features a dataset with only categorical variables, making it perfect for exploring different methods for categorical encoding.
The dataset contains binary, nominal, and ordinal features. We will explore methods for encoding each type of feature.
The full notebook can be found here.
Binary features are those with only two possible values.
We have five binary features in our dataset. As we can see below, the first three are already numeric and don’t require any manipulation, bin_3 and bin_4 are strings that will need to be encoded.
Because these are binary features, we can use Panda’s replace() to encode them:
df['bin_3'] = df['bin_3'].replace({'T':1}, 'F':0})df['bin_4'] = df['bin_4'].replace({'Y':1, 'N':0})
Here we pass a dictionary to replace() with the current value as the key and the desired value as the value. Below we can see our binary features encoded. (For now, we will ignore null values.)
Using replace() is very helpful, for binary features, but what if we have categorical features with more categories?
Ordinal features are those with some order associated with them. We can tell from the sample of ordinal features below these features have an order that may be important.
From Kaggle’s ranking system we know that a position of grandmaster is higher than those of novice or contributor.
The machine learning model may be able to use the order information to make better predictions and we want to preserve it.
For ordinal features, we use integer encoding. To integer encode our data we simply convert labels to integer values.
While there are many methods for integer encoding, we will discuss two here:
Sklearn’s LabelEncoder()
Panda’s map()
We can label encode data with Sklearn’s LabelEncoder():
from sklearn.preprocessing import LabelEncoder#label encoder can't handle missing valuesordinal_features['ord_1'] = ordinal_features['ord_1'].fillna('None')# Label encode ord_1 featurelabel_encoder = LabelEncoder()ordinal_features['ord_1'] = label_encoder.fit_transform(ordinal_features['ord_1'])# Print sample of datasetordinal_features.head()
Above we see the encoded feature ord_1. We can see the value grandmaster has been encoded with the integer 2, novice with the inter 5, and none with the integer 4.
While using LabelEncoder() is very quick and easy, it may not be the best choice here: the order of our encoding is not exactly right. Also, we had to handle our null values before being able to use it.
Another downside to LabelEncoder() is the fact that the documentation states it should be used for encoding target values (y) and not for the inputs (x). Let’s explore a different method for encoding our ordinal features.
Another option here is to use map().
Panda’s map() substitutes each value with another specified value, similar to replace() that we used above. Here we create a dictionary with our desired mapping and apply the mapping to our series:
# create dictionary of ordinal to integer mappingord_2 = {'Freezing':1,             'Cold':2,             'Warm':3,             'Hot':4,             'Boiling Hot':5,             'Lava Hot':6}# apply using mapdf['ord_2'] = df.ord_2.map(ord_2)
Using map() allowed us to specify the order of the values in our categorical feature to ensure they are in a meaningful arrangement.
These methods should only be used for ordinal features, where the order matters. For features where order is not important we must explore other techniques.
Nominal features are categorical features that have no numerical importance. Order does not matter.
One-hot encoding is a better technique when order doesn’t matter. For example, in the feature nom_1 we have shapes. There is no obvious order here. One shape is not better than another.
In a situation like this, where order doesn’t matter, integer encoding could lead to poor model performance and should not be used.
In one hot encoding, a new binary (dummy) variable is created for each unique value in the categorical variable. In the chart above, we had three unique colors and so we create three new features, one for each color. If the value is true, the integer 1 is placed in the field, if false then a 0.
Here we can use Panda’s get_dummies() to one hot encode our nominal features.
This method converts a categorical variable to dummy variables and returns a dataframe. The drop_first parameter is helpful to get k-1 dummies by removing the first level.
nominal_features = pd.get_dummies(nominal_features, drop_first=True)
We can see from the sample of our one hot encoded nominal features above that this type of encoding can greatly increase our number of columns. We input 10 columns and after encoding now have over 5000!
In the situation of high cardinality features, those with many possible values, we may need to do some manipulation prior to encoding. For example, for values occurring only a small percent of the time, we could group them into an “other” category.
An alternative approach could be to remove categorical variables from the dataset. While removing features is always an option, it may not be the best way to deal with categorical variables as we will lose potentially important information.
As the dataset we’ve been working with contains only categorical features, this is clearly not an option in this situation!
We explored basic techniques for encoding categorical variables according to their type:
Binary
Ordinal
Nominal
While the techniques reviewed are by no means the only options for encoding categorical variables, they are a great starting place.
In future articles, we’ll explore other categorical encoding techniques.