Don’t Make These 3 Pandas Mistakes
By: Roman Orac
Tags: Pandas, Analytics, Data Science, Programming, Python

Don’t Make These 3 Pandas Mistakes

Only a fool learns from his own mistakes. The wise man learns from the mistakes of others.

Photo by shiyang xu on Unsplash

Have you ever planned you’d need an hour to finish a short task, but then you spend a whole day working on it? If yes, welcome to my world!

In this article, I present 3 pandas mistakes that took me much longer to solve than they should. I also share the link to the Notebook with examples at the end of this article.

Only a fool learns from his own mistakes. The wise man learns from the mistakes of others.

See my pandas articles to learn more about Data Analysis with pandas:

1. How NOT to visualize a Weighted Average

Photo by Victor Freitas on Unsplash

The weighted average is similar to an ordinary average, except that some data points contribute more than others.

Weighted arithmetic mean from Wikipedia

This mistake with pandas occurred to me when I was working on an Exploratory Data Analysis — a typical day for a Data Scientist.

I extracted the data from the database, loaded it with pandas and followed a piece of advice from an ancient philosopher:

A picture is worth a thousand words.

In other words, I’ve visualized the data.

Something caught my eye on the plot and I wasn’t sure what at first.

Mistake

Let’s repeat the steps and see if you spot the mistake. First, let’s define a sample DataFrame:

df = pd.DataFrame({"bin": [0, 1, 2, 4, 5], "value": [1, 2, 3, 2, 5]})

Entries in df DataFrame

Calculate the weighted average, which is 3.15.

weighted_average = (df.bin * df.value).sum() / df.value.sum()

… and visualize the plot:

# visualize the plot

ax = df.plot.bar(x="bin", y="value", figsize=(14, 7)) # plot the weighted average

ax.axvline(weighted_average, color="black", linestyle="--", label="Weighted average")

Bar plot with a weighted average line

It took me a while to figure out what was the mistake.

The weighted average is 3.15, but on the plot, it’s shown at ~4.20. What’s happening here? A bug in matplotlib?

Solution

Not really! The problem is that bin 3 is not present in the DataFrame. Let’s add it and set the value to 0 as it’s not present.

df = df.append({"bin": 3, "value": 0}, ignore_index=True)

df = df.sort_values(["bin"]).reset_index(drop=True)

Entries in df DataFrame with bin 3 set to 0

We don’t need to recalculate the weighted average as it stays the same.

Let’s look at the plot now. Better, right?

# visualize the plot

ax = df.plot.bar(x="bin", y="value", figsize=(14, 7)) # plot the weighted average

ax.axvline(weighted_average, color="black", linestyle="--", label="Weighted average")

Bar plot with a fixed weighted average line

2. How NOT to categorize a numeric column

Photo by v2osk on Unsplash

This mistake is a beauty. I see it time and time again when doing code reviews.

It mostly happens when Data Scientists split the dataset into train and test sets.

Mistake

After splitting the dataset into train and test set, we have two disjoint DataFrames. Now, we would like to bin a numerical column into 10 categories.

Let’s generate random values for the train and test set:

df_train = pd.DataFrame({"feat": np.random.rand(1000)})

df_test = pd.DataFrame({"feat": np.random.rand(1000)})

Values in df_train DataFrame (df_test has similar values).

Pandas has qcut function for quantile-based discretization:

Pandas qcut function: Discretize variable into equal-sized buckets based on rank or based on sample quantiles. For example, 1000 values for 10 quantiles would produce a categorical object indicating quantile membership for each data point.

Let’s apply the pandas function on the train and test set:

df_train.loc[:, "feat_bin"] = pd.qcut(df_train.feat, 10, labels=False) df_test.loc[:, "feat_bin"] = pd.qcut(df_test.feat, 10, labels=False)

Some would think: Great, job completed!

But they’ve actually made a bad mistake.

Solution

Photo by Erik Mclean on Unsplash

The test set is an out of sample set. So the numerical column in the test set should be transformed with boundaries “learned” from the train set.

Pandas qcut function has retbins arguments, which returns the boundaries.

See below, retbins=True returns boundaries with which the train set column was transformed:

# feat_bins has the boundaries df_train.loc[:, "feat_bin"], feat_bins = pd.qcut(df_train["feat"], 10, labels=False, retbins=True)

These are the boundaries:

Values in feat_bins

These boundaries will be used to transform the values in the test set.

We need to add negative and positive infinity to those boundaries as the test set can have unseen values in the training set.

Let’s add negative infinity on the one end and positive infinity on the other:

feat_bins = np.concatenate(([-np.inf], feat_bins[1:-1], [np.inf]))

Values in feat_bins with positive and negative infinity

Now, let’s transform the test set using these boundaries:

df_test.loc[:, "feat_bin"] = pd.cut(df_test.feat, feat_bins, labels=False)

Mistake number two fixed. Great job!

3. How NOT to display duplicates

This mistake happened to me!

I was working on an analytical task and I had to connect entries from the database with entries from an online tool that we use.

This task is a piece of cake for an experienced Data Scientist like me — famous last words.

Mistake

I saved query results from a database to a CSV file and loaded them into pandas:

# random entries that simulate the database query df_db = pd.DataFrame({"key": [1, 2, 3, None], "value": [2, 2, 3, 1]})

Entries in the df_db DataFrame.

Then I extracted entries with the online tool:

# random entries that simulate the the online tool df_ot = pd.DataFrame({"key": ["3", "4", "5", "Null"], "value": [1, 2, 1, 0]})

Entries in the df_ot DataFrame

The last step was to JOIN the Dataframes and calculate some statistics on the duplicates:

df = pd.concat((df_db, df_ot))

Entries in the df DataFrame

Let’s check for duplicates:

df[df.key.duplicated()]

Empty result?!? Why? Two entries with key=3 are present in the Dataframe. Is this a pandas bug?

Solution

Photo by Julie Molliver on Unsplash

No, it’s not a pandas bug. The online tool uses a special string value for NULL, so pandas reads the whole column as a string.

When looking for duplicates, it compares 3 with “3”.

Let’s fix it. First, we need to filter out Null values and Null string.

df = df[df.key != "Null"]

df = df[df.key.notnull()]

Then we convert the key column to int as it should be in the first place.

df.key = df.key.astype(int)

Let’s check the duplicates, now:

df[df.key.duplicated()]

Voila! The third mistake fixed.

Conclusion

Photo by Petr Vyšohlíd on Unsplash

Hopefully, you’ve learned something new and you won’t need to spend time Googling for solutions to these mistakes in case you encounter them in your analysis.

Here is the link to the Jupyter notebook with the example.

Before you go

Some of the links above are affiliate links and if you go through them to make a purchase I’ll earn a commission. Keep in mind that I link courses because of their quality and not because of the commission I receive from your purchases.

Follow me on Twitter, where I regularly tweet about Data Science and Machine Learning.