Encoding Categorical Variables: One-hot vs Dummy Encoding | by Rukshan Pramoditha | Towards Data Science

You’ll often find categorical variables in your datasets. A categorical variable has a finite number of categories or labels for its values. For example, Gender is a categorical variable that can take “Male” and “Female” for its values.
Categorical variables are mainly divided into two types:
Ordinal categorical variables: These are categorical variables whose values follow a natural order. For example, in the variable Educational level, its values of “Elementary school”, “Some college” and “Graduate degree” follow an order in which “Graduate degree” is the highest educational level and “Elementary school” is the lowest.
Nominal categorical variables: These are categorical variables whose values do not follow a natural order. For example, Gender is a nominal categorical variable whose values of “Male” and “Female” do not follow an order.
Categorical variables usually have strings for their values. Many machine learning algorithms do not support string values for the input variables. Therefore, we need to replace these string values with numbers. This process is called categorical variable encoding.
Here, we’ll discuss two different types of encoding:
One-hot encoding
Dummy encoding
We’ll begin with one-hot encoding.
In one-hot encoding, we create a new set of dummy (binary) variables that is equal to the number of categories (k) in the variable. For example, let’s say we have a categorical variable Color with three categories called “Red”, “Green” and “Blue”, we need to use three dummy variables to encode this variable using one-hot encoding. A dummy (binary) variable just takes the value 0 or 1 to indicate the exclusion or inclusion of a category.
In one-hot encoding,
“Red” color is encoded as [1 0 0] vector of size 3.
“Green” color is encoded as [0 1 0] vector of size 3.
“Blue” color is encoded as [0 0 1] vector of size 3.
Dummy encoding also uses dummy (binary) variables. Instead of creating a number of dummy variables that is equal to the number of categories (k) in the variable, dummy encoding uses k-1 dummy variables. To encode the same Color variable with three categories using the dummy encoding, we need to use only two dummy variables.
In dummy encoding,
“Red” color is encoded as [1 0] vector of size 2.
“Green” color is encoded as [0 1] vector of size 2.
“Blue” color is encoded as [0 0] vector of size 2.
Dummy encoding removes a duplicate category present in the one-hot encoding.
Both one-hot and dummy encoding can be implemented in Pandas by using its get_dummies function.
import pandas as pdpd.get_dummies(data, prefix, dummy_na, columns, drop_first)
data — Here we specify the data that we need to encode. It can be a NumPy array, Pandas Series or a DataFrame.
prefix — If we specify a prefix, it will add to the column names so that we can easily identify the columns. The prefix can be specified as a string for a single column name. For multiple column names, it is defined as a dictionary mapping column names to prefixes. For more details, see the examples below.
dummy_na — If False (default), missing values (NaNs) are ignored when encoding the variables. If True, this will return missing data in a separate category.
columns — This specifies the column names to be encoded. If None (default), all categorical columns in the data parameter will be encoded. If you specify column names as a list, only the specified columns will be encoded.
drop_first — This is the most important parameter. This takes a boolean value, True or False. If False (default), this will perform one-hot encoding. If True, this will drop the first category of each categorical variable, create k-1 dummy variables for each categorical variable and perform dummy encoding.
Now, we use the diamonds dataset (see the source and license information at the bottom) to see both types of encoding in action.
import pandas as pddf = pd.read_csv("diamonds.csv")df.head()
df.shape
The dataset contains 53,940 instances and 10 variables.
Let’s see if there are any missing values in the dataset.
df.isnull().sum().sum()
Since this returns 0, there are no missing values in the dataset.
Let’s see how many categorical variables are in the dataset.
df.info()
The categorical variables have object or category datatype. So, there are 3 categorical variables in the dataset. They are cut, color and clarity.
Let’s see the unique categories or labels of the cut variable.
df["cut"].unique()
There are 5 unique categories in the cut variable. To encode this variable, we need to create 5 dummy variables in one-hot encoding and 4 dummy variables in dummy encoding.
Likewise, the color and clarity variables have unique categories.
df["color"].unique()
df["clarity"].unique()
Now, we apply one-hot encoding to the color variable alone and see the results.
one_hot = pd.get_dummies(df["color"],                         prefix="color",                         drop_first=False)one_hot
This returns a Pandas DataFrame of encoded data. Let’s see the first few rows of it.
See how the text specified in the prefix parameter is combined with the category names of color.
Now, we add one-hot encoding of the color variable to the dataset.
one_hot_df = pd.get_dummies(df, prefix="color",                             columns=["color"],                             drop_first=False)one_hot_df.head()
Now, we apply one-hot encoding to all categorical variables in the dataset.
one_hot_df = pd.get_dummies(df, prefix={'color':'color',                                        'cut':'cut',                                        'clarity':'clarity'},                            drop_first=False)
Now, we see the shape of the encoded dataset.
one_hot_df.shape
The encoded dataset has 27 variables. This is because one-hot encoding has added 20 extra dummy variables when encoding the categorical variables. So, one-hot encoding expands the feature space (dimensionality) in your dataset.
To implement dummy encoding to the data, you can follow the same steps performed in one-hot encoding. The only difference is that you should set the drop_first parameter to True instead of False.
dummy_df = pd.get_dummies(df, prefix={'color':'color',                                       'cut':'cut',                                      'clarity':'clarity'},                          drop_first=True)
Now, we see the shape of the encoded dataset.
dummy_df.shape
The encoded dataset has 24 variables. This is because dummy encoding has added 17 extra dummy variables when encoding the categorical variables. So, dummy encoding also expands the feature space (dimensionality) in your dataset.
Both one-hot and dummy encoding can be implemented in Scikit-learn by using its OneHotEncoder function.
from sklearn.preprocessing import OneHotEncoderohe = OneHotEncoder(categories, drop, sparse)encoded_data = ohe.fit_transform(original_data)#Returns a NumPy array of encoded data
categories — The default is ‘auto’ that automatically determines the categories in each variable.
drop — The default is None that performs one-hot encoding. To perform dummy encoding, set this parameter to ‘first’ that drops the first category of each variable.
sparse — Set this to False to return the output as a NumPy array. The default is True which returns a sparse matrix.
Here also, we use the same diamonds dataset. We apply one-hot encoding to all categorical variables in the dataset.
from sklearn.preprocessing import OneHotEncoderohe = OneHotEncoder(categories='auto',                     drop=None,sparse=False)ohe_df = pd.DataFrame(ohe.fit_transform(df)
Now, we see the shape of the encoded dataset.
Ohh. What? The encoded dataset has 13,687 variables. Can this happen? Yes, this is because Scikit-learn’s OneHotEncoder() function has created dummy variables for numerical variables also. To avoid this, we should only pass categorical data when calling its fit_transform method.
ohe_df = pd.DataFrame(ohe.fit_transform(df[['cut',                                             'color', 'clarity']]))
However, this returns only the encoded categorical data, not the entire dataset with numerical variables.
You can simply do this by specifying the drop parameter to ‘first’.
ohe = OneHotEncoder(categories='auto',                     drop='first',sparse=False)
Both types of encoding can be used to encode ordinal and nominal categorical variables. However, if you strictly want to keep the natural order of an ordinal categorical variable, you can use label encoding instead of the two types of encoding that we’ve discussed above.
Let’s say we have a categorical variable Quality with three categories called “Fair”, “Good” and “Premium”. The natural order of these categories is:
Fair < Good < Premium => 0 < 1 < 2
We can encode this as follows.
One advantage of label encoding is that it does not expand the feature space at all as we just replace category names with numbers. Here, we do not use dummy variables.
The major disadvantage of label encoding is that machine learning algorithms may consider there may be relationships between the encoded categories. For example, an algorithm may interpret Premium (2) as two times better than Good (1). Actually, there is no such relationship between the categories.
To avoid this, label encoding should only be applied to target (y) values, not to input (X) values.
Label encoding can be applied by using Scikit-learn’s LabelEncoder function. Now, we apply it to the cut variable in our diamonds dataset. This is for illustration purposes only as we do not use label encoding to encode input (X) values.
from sklearn.preprocessing import LabelEncoderdf['cut_enc'] = LabelEncoder().fit_transform(df['cut'])df.head(10)
The new encoded data column (cut_enc) has been added at right. We can now remove the cut variable.
The get_dummies() function returns encoded data with variable names. We can also add prefixes to dummy variables in each categorical variable name.
The get_dummies() function returns the entire dataset with numerical variables also.
Both expand the feature space (dimensionality) in your dataset by adding dummy variables. However, dummy encoding adds fewer dummy variables than one-hot encoding does.
Dummy encoding removes a duplicate category in each categorical variable. This avoids the dummy variable trap.
This is the end of today’s post.
Please let me know if you’ve any feedback.
Meanwhile, you can sign up for a membership to get full access to every story I write and I will receive a portion of your membership fee.
Thank you so much for your continuous support! See you in the next story. Happy learning to everyone!
Diamonds dataset source:
The diamonds dataset can be downloaded at https://www.kaggle.com/shivam2503/diamonds
Diamonds dataset license:
The diamonds dataset is licensed by a Public Domain license as defined here. It means that the dataset has been donated to the public.