Encoding Categorical Features. Introduction | by Yang Liu | Towards Data Science

In machine learning projects, one important part is feature engineering. It is very common to see categorical features in a dataset. However, our machine learning algorithm can only read numerical values. It is essential to encoding categorical features into numerical values.
Here we will cover three different ways of encoding categorical features:
1. LabelEncoder and OneHotEncoder
2. DictVectorizer
3. Pandas get_dummies
For your convenience, the complete code can be found in my github.
The data set we use here is from UCI Machine Learning Repository. It is used to predict whether a patient has kidney disease using various blood indicators as features. We use pandas to read the data in.
# load datadf = pd.read_csv(‘datasets/chronic_kidney_disease.csv’, header=None,  names=[‘age’, ‘bp’, ‘sg’, ‘al’, ‘su’, ‘rbc’, ‘pc’, ‘pcc’, ‘ba’, ‘bgr’, ‘bu’, ‘sc’, ‘sod’, ‘pot’,  ‘hemo’, ‘pcv’, ‘wc’, ‘rc’, ‘htn’, ‘dm’, ‘cad’, ‘appet’, ‘pe’, ‘ane’, ‘class’])# head of dfdf.head(10)
We can see from the table, we have several categorical features, such as ‘rbc’(red blood cells), ‘pc’(pus cell), ‘pcc’(pus cell clumps) and so on. It is obvious the dataset contains missing values, but since this is out scope of the topic in this blog, we’ll not cover how to do it here. But filling missing values is essential and should be done before encoding categorical features. You can refer to my github to see how to perform filling missing values. The completed dataset that ready for categorical encoding is shown as follows.
The labelEncoder and OneHotEncoder only works on categorical features. We need first to extract the categorial featuers using boolean mask.
# Categorical boolean maskcategorical_feature_mask = X.dtypes==object# filter categorical columns using mask and turn it into a listcategorical_cols = X.columns[categorical_feature_mask].tolist()
LabelEncoder converts each class under specified feature to a numerical value. Let’s go through the steps to see how to do it.
Instantiate a LabelEncoder object:
# import labelencoderfrom sklearn.preprocessing import LabelEncoder# instantiate labelencoder objectle = LabelEncoder()
Apply LabelEncoder on each of the categorical columns:
# apply le on categorical feature columnsX[categorical_cols] = X[categorical_cols].apply(lambda col: le.fit_transform(col))X[categorical_cols].head(10)
Note that the output of LabelEncoder is still a dataframe. The results is shown as following:
As we can see, all the categorical feature columns are binary class. But if the categorical feature is multi class, LabelEncoder will return different values for different classes. See the following example, the ‘Neighborhood’ feature has as many as 24 classes.
In this case, using LabelEncoder only is not a good choice, since it brings in a natural ordering for different classes. For example, under ‘Neighborhood’ feature, class_a has value 5 but class_b has value 24, is class_b ‘greater’ than class_a? The answer is obviously no. Thus allowing model learning this result will lead to poor performance. Therefore, for dataframe containing multi class features, a further step of OneHotEncoder is needed. Let’s see the steps to do it.
Instantiate OneHotEncoder object:
# import OneHotEncoderfrom sklearn.preprocessing import OneHotEncoder# instantiate OneHotEncoderohe = OneHotEncoder(categorical_features = categorical_feature_mask, sparse=False ) # categorical_features = boolean mask for categorical columns# sparse = False output an array not sparse matrix
We need to specify categorical feature using its mask inside OneHotEncoder. The sparse=False argument outputs a non-sparse matrix.
Apply OneHotEncoder on DataFrame:
# apply OneHotEncoder on categorical feature columnsX_ohe = ohe.fit_transform(X) # It returns an numpy array
Note that the output is a numpy array, not a dataframe. For each class under a categorical feature, a new column is created for it. For example, there are 20 columns created for the ten binary class categorical features.
As we can see, the LabelEncoder and OneHotEncoder usually need to be used together as two steps procedure. An more convenient way is using DictVectorizer which can achieve these two steps all at once.
First, we need to convert the dataframe into a dictionary. This can be achieved by Pandas to_dict method.
# turn X into dictX_dict = X.to_dict(orient='records') # turn each row as key-value pairs# show X_dictX_dict
The orient='records' is required to turn the data frame into a {column:value} format. The result is a list of dictionaries, among which each dictionary represent one sample. Note that, in this case we don’t need to extract the categorical features, we can convert the whole dataframe into a dict. This is one advantage compared to LabelEncoder and OneHotEncoder.
Now we instantiate a DictVectorizer:
# DictVectorizerfrom sklearn.feature_extraction import DictVectorizer# instantiate a Dictvectorizer object for Xdv_X = DictVectorizer(sparse=False) # sparse = False makes the output is not a sparse matrix
The sparse=False makes the output to be a non-sparse matrix.
DictVectorizer fit and transform on the converted dict:
# apply dv_X on X_dictX_encoded = dv_X.fit_transform(X_dict)# show X_encodedX_encoded
The result is a numpy array:
Each row represents a sample and each column represents a feature. If we want to know what feature for each column, we can check the vocabulary of this DictVectorizer:
# vocabularyvocab = dv_X.vocabulary_# show vocabvocab
Pandas get_dummies method is a very straight forward one step procedure to get the dummy variables for categorical features. The advantage is you can directly apply it on the dataframe and the algorithm inside will recognize the categorical features and perform get dummies operation on it. Here is how to do it:
# Get dummiesX = pd.get_dummies(X, prefix_sep='_', drop_first=True)# X headX.head()
The prefix_sep='_' makes each class has a unique name separated by the delimiter. The drop_first=True drops one column from the resulted dummy features. The purpose is to avoid multicollinearity. Here is the results:
LabelEncoder and OneHotEncoder is usually need to be used together as a two steps method to encode categorical features. LabelEncoder outputs a dataframe type while OneHotEncoder outputs a numpy array. OneHotEncoder has the option to output a sparse matrix. DictVectorizer is a one step method to encode and support sparse matrix output. Pandas get dummies method is so far the most straight forward and easiest way to encode categorical features. The output will remain dataframe type.
As my point of view, the first choice method will be pandas get dummies. But if the number of categorical features are huge, DictVectorizer will be a good choice as it supports sparse matrix output.