What is One-Hot Encoding and how to use Pandas get_dummies function | by B. Chen | Towards Data Science

I get questions all the time about some concepts in machine learning and what they really mean, for example, in one of my previous articles Google’s 7 steps of Machine Learning in practice, I get asked what is one-hot encoding, and when should we use it? That’s is a great question and every machine learning practitioner needs to know the answer, so let’s talk about it and dive into some code with Pandas get_dummies().
In the step of data processing in machine learning, we often need to prepare our data in specific ways before feeding into a machine learning model. One of the examples is to perform a One-Hot encoding on categorical data.
One Hot Encoding is a process in the data processing that is applied to categorical data, to convert it into a binary vector representation for use in machine learning algorithms
One of the major problems with machine learning is that a lot of algorithms cannot work directly with categorical data. Categorical data [1] are variables that can take on one of a limited number of possible values. Some examples are:
The sex of a person: female or male.
The airline travel class: First Class, Business Class, and Economy Class.
The computer vendor: Lenovo, HP, Dell, Apple, Acer, Asus, and Others.
Therefore, we need a way to convert categorical data into a numerical form and our machine learning algorithm can take in that as input. The most widely used encoding techniques are:
Integer Encoding: encodes the values as integer.One-Hot Encoding: encodes the values as a binary vector array.Dummy Variable Encoding: same as One-Hot Encoding, but one less column.
Integer Encoding: encodes the values as integer.
One-Hot Encoding: encodes the values as a binary vector array.
Dummy Variable Encoding: same as One-Hot Encoding, but one less column.
In the above examples, each value in a categorical variable represents a different category. Some categories may have a natural relationship (known as a natural ordering) to each other. The airline travel class variable does have a natural ordering of values. This type of categorical variable is known as an ordinal variable. Ordinal variables should be treated differently in machine learning because the ordinality generally comes with some significance. In the case of airline travel class, the passengers in the higher classes tend to be wealthier and older. Integer Encoding is used to convert ordinal variable to numerical data, for example
First Class is 1, Business Class is 2, and Economy Class is 3
The integer values have a natural ordered relationship between each other and machine learning algorithms may be able to learn this relationship [2].
For categories variables where no such ordinal relationship exists, Integer Encoding could result in poor performance or unexpected results [2]. Fortunately, One-Hot Encoding is a way to combat this. One-Hot Encoding simply creates one column for every possible value and put a 1 in the appropriate column. Below is the equivalent One-Hot Encoding representation.
With One-Hot Encoding, the binary vector arrays representation allows a machine learning algorithm to leverage the information contained in a category value without the confusion caused by ordinality.
However, there is some redundancy in One-Hot encoding. For instance, in the above example, if we know that a passenger’s flight ticket is not First Class and not Economy Class, then it must be Business Class. So we only need to use two of these three dummy-coded variables as a predictor. More generally, the number of dummy-coded variables needed is one less than the number of possible values, which is K-1. In statistics, this is called a dummy encoding variable, or dummy variable. Below is the equivalent dummy encoding variable representation:
Dummy encoding variable is a standard advice in statistics to avoid the dummy variable trap, However, in the world of machine learning, One-Hot encoding is more recommended because dummy variable trap is not really a problem when applying regularization [3].
Before we diving into tutorial, let’s first pull up an example dataset we can work with. For this tutorial, we will be working on the Titanic Dataset from Kaggle. This is a very famous dataset and very often is a student’s first step in machine learning. Let’s import pandas and load data to get started.
import pandas as pddef load_data():    return pd.read_csv('data/train.csv')df = load_data()df.head()
We created a function load_data() to load train.csv file into pandas DataFrame.
Let’s pretend we want to create a One-Hot encoding variable for the Sex column. Here is how we create it.
pd.get_dummies(df.Sex)
What it does is create one column for every possible value and they are two possible values for Sex. It tells you whether it is female or male by putting a 1 in the appropriate column.
Generally speaking, if we have K possible values for a categorical variable, we will get K columns to represent it.
There is some redundancy in One-Hot encoding. For instance, in the above Sex One-Hot encoding, a person is either male or female. So we only need to use one of these two dummy-coded variables as a predictor. More generally, the number of dummy-coded variables needed is one less than the number of possible values, which is K-1. In statistics, this is called a dummy encoding variable, or dummy variable.
By default, the get_dummies() does not do dummy encoding, but One-Hot encoding. To produce an actual dummy encoding from a DataFrame, we need to pass drop_first=True
pd.get_dummies(df.Sex, drop_first=True)
By default, get_dummies() use variable values as column names. For instance, male and female in the above Sex column example. To identify where the dummy variables came from, we can use the prefix argument. For example:
pd.get_dummies(df.Sex, prefix='Sex')
Now, let’s try to get One-Hot encoding for a variable that has more than two possible values. For instance, there is an Embarked column and we can do value_counts() to check its different values.
df.Embarked.value_counts()S    644C    168Q     77Name: Embarked, dtype: int64
Let’s use get_dummies() for that
pd.get_dummies(df.Embarked, prefix='Embarked')
You might be thinking we have still got to attach the One-Hot encoding variables to the DataFrame. To do that, we first assign the output to a variable called embarked_dummies
embarked_dummies = pd.get_dummies(df.Embarked, prefix='Embarked')
Then, we use pd.concat() to attach embarked_dummies to the DataFrame df.
df = pd.concat([df, embarked_dummies], axis=1)
The way pd.concat() works is we can concatenate columns side by side or rows on top of one another. So in this case, we concatenate df and embarked_dummies along columns (axis =1). In the end, we overwrite the original df.
By running df.head(), we can see that we’ve now got the Embarked_C, Embarked_Q, and Embarked_S.
We have been passing a Series to get_dummies(), but we’re gonna pass a DataFrame.
df = load_data() # reset dataframedf = pd.get_dummies(df, columns=['Sex', 'Embarked'])
df is a DataFrame. We pass columns a list and we tell it Sex and Embarked are the columns we want to dummy.
By running df.head(), we can see that the original Sex column and Embarked column are gone, they have been replaced with the dummy columns.
Thanks for reading.
Please checkout the notebook on my Github for the source code.
Stay tuned if you are interested in the practical aspect of machine learning.
[1] Categorical variable in Wikipedia
[2] Why one-hot encoding in machine learning
[3] Are you getting burned by one-hot encoding