Unleash the Power of Pandas ‘category’ Dtype: Make Categorical Feature Serves Well for Machine Learning
By: Kefei Mo
Tags: Machine Learning, Pandas, Category, Data Science, Python

Explain how ‘category’ dtype works

Quote pandas documentation about categorical data (1.0.5):

All values of categorical data are either in categories or np.nan. Order is defined by the order of categories, not lexical order of the values. Internally, the data structure consists of a categories array and an integer array of codes which point to the real value in the categories array.

Even more confusing now? Don’t worry, let me try to explain it.

Hash table

The mechanism behind is the idea of ‘hash table’. Here is an illustration of how a hash table works. (We use the ‘gender’ feature as an example.)

illustration of the mechanism behind ‘category’ dtype (image by author)

First, we create a HASH TABLE to map the unique values ‘M’ and ‘F’ (a.k.a. keys) into ‘0’ and ‘1’ (a.k.a hashes). Then we ENCODE the original dataset: use ‘0’ and ‘1’ to replace ‘M’ and ‘F’, respectively. The encoded dataset will be stored in memory. This process usually can (but not always) save memory usage (I will explain later). Note the value ‘0’ and ‘1’ are not very interpretative. Instead of showing the encoded dataset, when display, Pandas maps the encoded dataset back to the original ‘M’ and ‘F’ values (INV_ENCODE). In other words, the new dataset with the ‘category’ dtype looks identical to the original one (but they are not the same.)

Data types and their memory usage

One thing to keep in mind is that: different data types might consume different memory usage. Here is a list of common data types and their memory usage.

Calculate memory usage

We have gone through how the category data type works hash table. We have reviewed the common data types and their memory usage. Now it is time for us to talk about memory usage.

You can use the following formulas to approximately calculate the memory usage.

For inherent Python data types (e.g. int64, float64, object),

memory_usage = bytes * n_rows

e.g. ‘gender’ feature (in ‘object’ dtype)

bytes = 8 (8 bytes for object data type);

n_rows = 130,821;

memory_usage = 8 * 130,821 = 1,046,568.

We can check this result with ‘nbytes’ function:

print(df_original.gender.nbytes)

For category dtype, the formula is given by:

bytes_hashed * num_of_row + (0 + bytes_object) * n_unique, where, bytes_hashed = hash_func (n_unique)

This formula consists of two parts: one from the encoded dataset (i.e., df_encode), one from the hash table.

illustration of memory usage calculation (image by author)

e.g. ‘gender’ feature (in ‘category’ dtype)

n_unique = 2,

bytes_hashed = 1. (we can use int8 to encode all the unique values; an int8 dtype takes up 1 byte.)

bytes_object =8. (memory space taken by keys in the hash table; pandas always use ‘object’ type to save the hash table keys.)

So, memory_usage = 1* 130,821+ (0 + 8) * 2 = 130,837

Again, we can check with ‘nbytes’ function

Why ‘category’ dtype not always save space?

The formula explains why sometimes, by ‘categorizing’ data, it saves memory but sometimes not. It has something to do with how many different classes are there (i.e., n_unique). When n_unique is small, pandas can use a small-sized data type to encode the original dataset. e.g., if the n_unique is less than 255, pandas can use int8 to encode, which only takes 1 byte.

However, when n_unique is large, e.g., for event_id, n_unique=107,900, it requires an int32 to encode the original dataset, and each int32 item takes 4 bytes. This is not too bad comparing using int64 in the original dataset, which takes 8 bytes. But don’t forget the hash table also takes memory storage, and Pandas always uses the ‘object’ data type to store the keys within the hash table. Since n_unique is so large as close to the n_rows (82%), the event_id category dataset would take more memory storage.

number of sub-classes within each feature and portion comparing to the total rows

More examples.

Note: The above formula and ‘nbytes’ function are just rough approximations to calcualte memory usage. e.g. memory usage by data frame indeces is not included. The fomula is a good start to understand the category dtype mechanicsm, however, if you want to analyse memory usage more precisely, you might want to check the memory_usage() function.

Some comments

You see, using category dtype not always helps to save memory usage. Also, it is not the only way or the best option if saving memory is your priority. e.g., we can change the ‘gender’ feature to ‘bool’ type to save more space.

However, I am still on the side with using category dtype. One reason is using category dtype is safer and fool-proofed. e.g., the following command is not a wise choice but it would not raise any warning when doing so. You have to check carefully before ‘astype’ your data.

Using ‘category’ dtype would not cause such a problem. Pandas always made the right choice to encode the data for you.

Plus, using category dtype would not lose any interoperability. We are still reading a dataset with the original sub-classes names instead of some encoded numbers.

Enough about saving memory. Let’s move on to the next session to see other benefits and why it matters for machine learning. (I would not sell it if saving memory is the only thing category dtype capable of. 😉)