Data Science Tip #005: Two Correct Ways to Perform One-Hot Encoding! | by David Hundley | Towards Data Science

Hello, hello everybody! I hope you all are enjoying a nice Labor Day weekend. I personally took Friday off as well to extend my weekend into four days, and I had a lot of great quality time with my daughters these last few days. But you know me, I’m always itching to keep producing in some capacity!
Before we get into the post, please be sure to reference my personal GitHub for all the code we’ll get into below. It’s pretty simple, but if you’d like to follow along in a concise Jupyter notebook, then I’d encourage you to check that out. For brevity’s sake, I’m not going to go into what one-hot encoding is in this post and instead encourage you to check out a former post of mine where I go into that more thoroughly.
Alrighty, so when I first learned about one-hot encoding, I was curiously taught one way that I would argue is the “wrong” thing to do for the data science field. It’s not that it doesn’t work to produce encoded features, but it doesn’t go far enough for applied machine learning. And it feels like I keep seeing this particular method over and over in all the tutorials I’ve come across.
That incorrect way? Pandas’ “get_dummies” function.
To demonstrate why NOT to use Pandas’ “get_dummies” and why to use the other options I’ll recommend, let’s first quickly setup our project by importing the necessary Python libraries as well as creating a tiny DataFrame with fake animal data. (I’ll explain what we’re importing down below and how to install the second option if you haven’t installed it before.)
# Importing the required librariesimport pandas as pdimport joblibfrom sklearn import preprocessingfrom category_encoders import one_hot# Creating a small DataFrame with fake animal dataanimals_df = pd.DataFrame({'animal': ['cat', 'dog', 'bird', 'monkey', 'elephant', 'cat', 'bird']})
As you can see, I intentionally duplicated a few values to demonstrate that our encoding will work. With our DataFrame in place, let’s go ahead and see what happens when we use the basic Pandas “get_dummies” function.
# Performing one-hot encoding with Pandas' "get_dummies" functionpandas_dummies = pd.get_dummies(animals_df['animal'])
You might be wondering, what’s the problem? Didn’t this do the trick for us? While it did work in this very specific instance, this method does NOT allow for future transformation in an inference pipeline. Just like we export a model or data scaler for inference transformation, there are means to export something to perform consistent one-hot encoding. Unfortunately, Pandas “get_dummies” does not allow for this. (Why this is not taught better in more data science courses is baffling to me!)
Fortunately, there are ways to get around this, and I’m about to show you two of them!
The first correct way is built right into Scikit-Learn. Just like any of the ML algorithms, there’s a piece of the Scikit-Learn library called OneHotEncoder that allows you to fit your data to an instantiated encoder object and then later transform similar data to appropriately fill all the columns as required.
But here’s the thing... I’m not wild about Scikit-Learn’s implementation of this. It works and is technically correct, but it’s a tiny bit... kludgy? See for yourself in the code below.
# Instantiating the Scikit-Learn OHE objectsklearn_ohe = preprocessing.OneHotEncoder()# Fitting the animals DataFrame to the Scikit-Learn one-hot encodersklearn_dummies = sklearn_ohe.fit_transform(animals_df)# Using the output dummies and transformer categories to produce a cleaner looking dataframesklearn_dummies_df = pd.DataFrame(data = sklearn_dummies.toarray(),                                   columns = sklearn_ohe.categories_)# Dumping the transformer to an external pickle filejoblib.dump(sklearn_ohe, 'sklearn_ohe.pkl')
As you can see in the Jupyter screenshot, the direct output of a transformed DataFrame is NOT a DataFrame. It’s actually a sparse matrix of float values. It is indeed possible to produce a DataFrame with that output, but it requires a little extra work. Not impossible, but also not intuitive. The first time I learned this, it took me several hours to wrap my brain around the output. Additionally, it produces float values in the result as opposed to integer values from the original dataset, which isn’t exactly expected behavior. But again, that can be altered, so we can still call this one correct method.
The second correct way to perform one-hot encoding is by using a special Python library called Category Encoders. If you haven’t used it before, all you have to do is a quick pip or conda install. For your convenience, I’ve pasted the respective shell commands to do so down below.
pip install category_encodersconda install -c conda-forge category_encoders
I was actually introduced to this library by some of my fellow data scientists in my day job, and I personally like it a lot better than Scikit-Learn’s implementation. I think it’s pretty evident why in the code below. Here’s the specific documentation about Category Encoder’s version of the OneHotEncoder, too.
# Instantiating the Category Encoders OHE objectce_ohe = one_hot.OneHotEncoder(use_cat_names = True)# Fitting the animals DataFrame to the Category Encoders one-hot encoderce_dummies = ce_ohe.fit_transform(animals_df)# Dumping the transformer to an external pickle filejoblib.dump(ce_ohe, 'ce_ohe.pkl')
As you can see, the output of fitting the data to the transformer produces a nice, clean DataFrame. Additionally, it appends the original “animal” column name with the value from the column to produce clean column names. Now, I actually think this is a better implementation because when you start one-hot encoding multiple features, it’s nice to know in the output from which feature they were originally sourced!
(If there’s *any* downside at all, the dumped pickle file for Scikit-Learn’s implementation is a lot small, but the size is almost negligible. We’re talking bytes versus kilobytes here, which is almost nothing these days, but I feel like I still gotta share that with y’all.)
Just to prove these transformers are doing what we want them to do, I re-imported them from their exported pickle files and performed encoding on a new verison of the animals DataFrame. It’s pretty similar to the original one, but I re-ordered the values.
# Creating a newly ordered animals dataframenew_animals_df = pd.DataFrame({'animal': ['cat', 'cat', 'dog', 'monkey', 'elephant', 'cat', 'bird', 'dog']})# Importing the exported pickle filesimported_sklearn_ohe = joblib.load('sklearn_ohe.pkl')imported_ce_ohe = joblib.load('ce_ohe.pkl')# Running the new animals DataFrame through the Scikit-Learn imported transformerloaded_sklearn_dummies = imported_sklearn_ohe.transform(new_animals_df)loaded_sklearn_dummies_df = pd.DataFrame(data = loaded_sklearn_dummies.toarray(),                                          columns = imported_sklearn_ohe.categories_)# Running the new animals DataFrame through the Category Encoders imported transformerloaded_ce_dummies = imported_ce_ohe.transform(new_animals_df)
Voila, it works! Each of the outputted transformations share the same characteristics as we first saw when we performed transformation on the original dataset. I didn’t demo it in this post, but there are even baked in features that dictate the proper behavior of what happens when an unknown or null value is passed. Neat!
And that wraps up another post! As nice as Pandas’ “get_dummies” functionality works, you really shouldn’t be using it for applied ML purposes. It causes a very fault tolerant, kludgy implementation in something like an API, and using either Scikit-Learn’s or Category Encoder’s one-hot encoder is much easier to use, anyway. Hope y’all appreciated this post! Be sure to check out my past tips, and please recommend anything you want to see in future posts! Until then, I’m signing off and wish you a happy Labor Day weekend.