3 Easy Ways to Handle Categorical Data -Python
By: Athisha R K
Tags: Pandas Dataframe, Data Science, Scikit Learn, Python3, Data Preprocessing

Photo by Edgar Soto on Unsplash

Introduction

The NOIR System is to classify data as Nominal, Ordinal, Interval, and Ratio. Right now, only a basic understanding of Nominal and Ordinal is required.

Nominal: Categories without any implied order. For example, different blood groups like A+ve, O-ve, etc. do not have an implied ordering among them.

Categories without any implied order. For example, different blood groups like A+ve, O-ve, etc. do not have an implied ordering among them. Ordinal: Categories with implied ordering. For example, the disease can have several stages: onset, acute, and chronic. Each is more severe than the previous one.

Examples of Nominal and Ordinal Data Types

A categorical Feature can be either Nominal or Ordinal datatype. Here are few examples:

iris.csv: species [‘setosa’ ‘versicolor’ ‘virginica']

Seaborn Tips Data-set: sex, smoker, day, time are categorical features

Categorical features cannot be processed by the computer directly. They need to be encoded to enable further processing and analysis. Encoding is the transformation of categorical variables to binary or numerical counterparts.

Encoding Examples

In this article, I’ll be explaining 3 easy ways of encoding in Python.

Prerequisites

Familiarity with Google Colab Notebooks, Jupyter Notebooks (or any other equivalent tool)

Basic understanding of Pandas DataFrame and sci-kit learn library in Python.

Let’s begin…

We shall begin by importing the necessary libraries in python.

Import Necessary Libraries

Seaborn library is used to plot graphs in python (similar to matplotlib). It also provides quick access to some example datasets for documenting seaborn or generating reproducible examples for bug reports. The list of available datasets can be obtained using get_dataset_names() as shown below.

List of Available Datasets in Seaborn

I will be using the “Diamonds” dataset. Please feel free to experiment with other datasets like iris, titanic, planets, and tips.

Load the dataset: Diamonds

This classic dataset contains the prices and other attributes of almost 54,000 diamonds. We can also observe that there are no missing values in the dataset.

Shape of the Dataset and Perecentage of Missing Values

Many beautiful graphs can be drawn with this data using seaborn and matplotlib as shown below (GitHub).

Scatterplot of Carat vs Price w.r.t. Cut

Scatterplot of Depth vs Price w.r.t Cut

The categorical features are Cut, Color and Clarity. These three features needs to be encoded before further processing is done.

Different classes in each feature

Method 1: Using apply() for Cut feature

apply() function allows the user to apply a function on each value of a feature in a dataframe. When we use apply(), it gives us the liberty to map it to any value. The mapping is done for ‘Cut’ as shown below.

Mapping done by CutEncoding( ) for the value ‘cut’

Encoding Cut using apply()

Method 2: Using Label Encoder for Color feature

sklearn.preprocessing.LabelEncoder() encodes the value between 0 to n-1. In our data, Color has 7 classes. So the encoded values ranges from 0 to 6. The below code snippet achieves that goal.

Label Encoding of Color (sklearn.preprocessing)

Method 3: Using DummyEncoding for Clarity Feature

pandas.get_dummies() converts the categorical feature to indicator or dummy variables.

Example: Consider a small dataset having Tennis? as a feature. Possible values for Tennis? are Yes, No, Maybe. The result of get_dummies() is shown below.

Illustration of get_dummies()

get_dummies() can be useful when we are concerned about only one class of a particular feature. When we are trying to shortlist tennis players, the data is regrouped to tennis players and non-tennis players. In such a case, only Tennis_Yes feature is enough.

The same concept can be applied for ‘Clarity’ in our dataset.

Dummy Encoding of Clarity

Another Quick Example: Iris Dataset

Load the dataset using seaborn.load_dataset()

Load iris dataset

2. Apply LabelEncoder() for species feature

Label Encoding of ‘species’

3. Final results

Useful Resources

There are a lot many ways to encode data. These three are the easiest among them. Let me know your thoughts/questions in the comments section below :)