Guide to Encoding Categorical Features Using Scikit-Learn For Machine Learning
By: Jason Chong
Tags: Feature Encoding, Data Science, Machine Learning, Scikit Learn, Data Preprocessing

Guide to Encoding Categorical Features Using Scikit-Learn For Machine Learning

One of the most crucial preprocessing steps in any machine learning project is feature encoding. It is the process of turning categorical data in a dataset into numerical data. It is essential that we perform feature encoding because most machine learning models can only interpret numerical data and not data in text form.

In this article, we will learn:

The difference between a nominal variable and an ordinal variable

How OneHotEncoder and OrdinalEncoder can be used to encode these variables respectively

Why the Scikit-learn library is preferred over the Pandas library when it comes to encoding categorical features

As usual, I will demonstrate these concepts through a practical case study using the students’ performance in exams dataset on Kaggle.

You can find the complete notebook on my GitHub here.

Data description

Photo by Annie Spratt on Unsplash

The students’ performance in exams dataset consists of marks secured by 1,000 students in the math, reading and writing subjects. Along with these test scores, the description of each student such as their gender, race/ethnicity, parental level of education, lunch and test preparation course are also included in the dataset.

The first 5 rows of the dataset

Problem statement

The goal of this project is to build a machine learning pipeline which includes feature encoding as well as a regression model to predict a random student’s test score given his/her description. This is an example of a regression problem in machine learning as our target variable, test score has a continuous distribution.

Check for missing values and data types

As we can see from the diagram above, we have 5 categorical (text) variables and 3 numerical variables in the dataset.

As it turns out, all the predictor variables are categorical variables and all the target variables are numerical variables.

Create a single target variable (mean score)

For simplicity purposes, let’s take the average of the 3 test scores i.e. math, reading and writing so that we are left with a single target variable to predict.

Correlation between the 3 test scores

Since the 3 test scores are highly positively correlated with each other, it is appropriate for us to just take the average.

As a result, our new data frame is as follows.

Our new data frame with only a single target variable (mean score)

Exploratory data analysis (EDA)

Exploratory data analysis is the process of analysing and visualising the variables in a dataset. Although this step is not necessary for feature encoding, I personally like to get a feel of my data before performing machine learning. So, feel free to skip past this section if you are only interested to learn about feature encoding.

In this section, we will explore how the different features in the dataset influence the outcome of a student’s test score.

Gender

Barplot of mean score by gender

Female students perform better than male students on average.

Race/ethnicity

Barplot of mean score by race/ethnicity

There appears to be an increasing trend in mean scores as we progress from group A to group E i.e. group A students perform the worst while group E students perform the best.

However, in the absence of any further information, it is difficult for us to draw any meaningful conclusion.

Parental level of education

Barplot of mean score by parental level of education

For the most part, the results are aligned with what we would expect. As the parental level of education increases, the mean score also increases.

Lunch

Barplot of mean score by lunch

Students with standard lunch perform better than those with free/reduced lunch.

Lunch can be seen as a proxy for the financial background of the students. Therefore, it makes sense that, on average, students that are financially better off do better in school compared to those that are financially worse off.

Test preparation course

Barplot of mean score by test preparation course

This should come at no surprise at all. The more prepared students are for their test, the better they will perform.

Mean score

Distplot of the target variable (mean score)

Difference between nominal variable and ordinal variable

Now that we have fully explored the variables in the dataset, we can move on to the main purpose of this article and that is encoding categorical features.

But before we dive into feature encoding, it is important that we first contrast the difference between a nominal variable and an ordinal variable. This is so that we can match the right encoder in Scikit-learn with the right type of variable.

A nominal variable is a categorical variable where its data does not follow a logical ordering. Some examples of nominal data include:

Gender (Male or female)

Blood type (A, B, AB or O)

Political party (Democratic or Republican)

An ordinal variable, on the other hand, is also a categorical variable except its data follows a logical ordering. Some examples of ordinal data include:

Socioeconomic status (low income, middle income or high income)

Education level (high school, bachelor’s degree, master’s degree or PhD)

Satisfaction rating (extremely dislike, dislike, neutral, like or extremely like)

As we will see later on, nominal variables are encoded using Scikit-learn OneHotEncoder while ordinal variables are encoded using Scikit-learn OrdinalEncoder.

Encoding nominal variables

The nominal variables in our dataset include:

Gender

Race/ethnicity

Lunch

Test preparation course

The categories in these features do not have a natural order or ranking.

In this section, we will explore two different ways to encode nominal variables, one using Scikit-learn OneHotEnder and the other using Pandas get_dummies. Once I have demonstrated the two techniques, I will also outline the several benefits OneHotEncoder has over get_dummies when performing feature encoding.

We will refer to the gender column in our data frame for this exercise.

Scikit-learn OneHotEncoder

Scikit-learn OneHotEncoder

As we can see, OneHotEncoder has created two columns to represent the two categories in the gender column, one for male and one for female.

Female students will receive a value of 1 in the female column and a value of 0 in the male column. Conversely, male students will receive a value of 0 in the female column and a value of 1 in the male column.

Nevertheless, the most important takeaway from the illustration above is that OneHotEncoder has successfully transformed what was originally a categorical (text) variable into a numerical variable that our machine learning model can interpret.

Pandas get_dummies

Pandas get_dummies

Similarly, get_dummies has encoded the gender feature by creating two columns to represent the two categories in the gender feature, male and female.

Why OneHotEncoder is better than get_dummies

Now, although both approaches give the same result, OneHotEncoder is generally preferred over get_dummies due to the following reasons:

Under OneHotEncoder, our original data frame remains the same size and therefore it is much easier for us to manage and explore.

OneHotEncoder can be incorporated as part of a machine learning pipeline using Scikit-learn whereas get_dummies require a more manual approach to feature encoding. This makes OneHotEncoder more efficient at transforming both the training set as well as the test set when we are performing machine learning.

Under OneHotEncoder, we can use the GridSearch function in Scikit-learn to evaluate and choose the best preprocessing parameters just like how we would use GridSearch to find the best hyperparameters for a machine learning model.

Cross-validation scores are more reliable under OneHotEncoder than get_dummies.

We will further explore the idea of building a machine learning pipeline towards the end of this article.

Encoding ordinal variable

The only ordinal variable in our data frame is the parental level of education feature. As education level can be seen as a progression, this feature is classified as an ordinal variable.

In this section, we will again consider two approaches to encoding ordinal variables, one using Scikit-learn OrdinalEncoder and the other using Pandas map method. Similar to the previous section, OrdinalEncoder has advantages over the map method when performing feature encoding.

Scikit-learn OrdinalEncoder

OrdinalEncoder differs from OneHotEncoder such that it assigns incremental values to the categories of an ordinal variable. This helps machine learning algorithms to pick up on an ordinal variable and subsequently use the information that it has learned to make more accurate predictions.

In order to use OrdinalEncoder, we have to first specify the order in which we would like to encode our ordinal variable, parental level of education. Here, we want the order to go as follows: some high school, high school, some college, associate’s degree, bachelor’s degree and finally master’s degree.

Scikit-learn OrdinalEncoder

Note that OrdinalEncoder has assigned the value of 0 to some high school, 1 to high school, 2 to some college, 3 to associate’s degree, 4 to bachelor’s degree and 5 to master’s degree.

Pandas map method

Pandas map method

The Pandas map method is a more manual approach to encoding ordinal variables where we individually assign numerical values to the categories in an ordinal variable.

Although it replicates the result of the OrdinalEncoder, it is not ideal for encoding ordinal variables with a high number of unique categories.

Make column transformer

For the remainder of this article, we will only consider the use of OneHotEncoder and OrdinalEncoder as means of encoding the categorical variables in our dataset.

In this section, we will combine OneHotEncoder and OrdinalEncoder into a single-step column transformer.

Single-step column transformer which consists of OneHotEncoder and OrdinalEncoder

Build machine learning pipeline

First and foremost, what is a pipeline and why do we use it?

Essentially, a pipeline chains together multiple steps in the machine learning process where the output of each step is used as the input to the next step. It is typically used to chain data preprocessing procedures (e.g. imputation for missing values, scaling and feature encoding) together with modelling into one cohesive workflow.

Here, we will build two pipelines that share the same column transformer that we have created above but with a different machine learning model, one using linear regression and the other using gradient boosting.

We will then assess the prediction results from the two pipelines using mean absolute error (MAE) as well as root mean squared error (RMSE). The model with a lower prediction error is deemed as more accurate than the other.

We will start off by splitting our data into a training set and a test set. Here, I have assigned 30% of the original data as our test set as well as a seed for reproducibility.

Train test split

We can now move on to building our pipeline.

Two pipelines, one using linear regression and the other using gradient boosting

With predictions ready from the two pipelines, we can proceed to evaluate the accuracy of these predictions using mean absolute error (MAE) and mean squared error (RMSE).

MAE and RMSE of pipelines

We can conclude that linear regression is slightly more accurate than gradient boosting.

Although these are not the most accurate predictions from a machine learning standpoint, my goal for this article is to demonstrate that you can build a pipeline which chains data preprocessing steps such as feature encoding together with modelling into one single cohesive workflow using Scikit-learn.

Conclusion

To summarise, in this article, we have learned the difference between a nominal variable and an ordinal variable as well as how to properly encode them using Scikit-learn OneHotEncoder and LabelEncoder.

We also discussed the benefits of performing feature encoding using the Scikit-learn library over the Pandas library i.e. the flexibility in building a machine learning pipeline.

Finally, we compared the accuracy of two separate pipelines at predicting students’ test score. We conclude that linear regression is slightly more accurate than gradient boosting due to the lower mean absolute error (MAE) as well as root mean squared error (RMSE).

I hope you have picked up a thing or two from this article. Thank you so much for reading. I will be publishing more tutorial articles relating to the Scikit-learn library in the coming weeks so do follow me for future updates.

I look forward to seeing you in my next article. Happy learning!

Follow me on my platforms