Finding and removing duplicate rows in Pandas DataFrame | by B. Chen | Towards Data Science

In data preprocessing and analysis, you will often need to figure out whether you have duplicate data and how to deal with them.
In this article, you’ll learn the two methods, duplicated() and drop_duplicates(), for finding and removing duplicate rows, as well as how to modify their behavior to suit your specific needs. This article is structured as follows:
Finding duplicate rowsCounting duplicate and non-duplicate rowsExtracting duplicate rows with locDetermining which duplicates to mark with keepDropping duplicate rows
Finding duplicate rows
Counting duplicate and non-duplicate rows
Extracting duplicate rows with loc
Determining which duplicates to mark with keep
Dropping duplicate rows
For demonstration, we will use a subset from the Titanic dataset available on Kaggle.
import pandas as pddef load_data():     df_all = pd.read_csv('train.csv')    # Take a subset    return df_all.loc[:300, ['Survived', 'Pclass', 'Sex', 'Cabin', 'Embarked']].dropna()df = load_data()
Please check out Notebook for the source code.
To find duplicates on a specific column, we can simply call duplicated() method on the column.
>>> df.Cabin.duplicated()0      False1      False9      False10     False14     False       ...  271    False278    False286    False299    False300    FalseName: Cabin, Length: 80, dtype: bool
The result is a boolean Series with the value True denoting duplicate. In other words, the value True means the entry is identical to a previous one.
To take a look at the duplication in the DataFrame as a whole, just call the duplicated() method on the DataFrame. It outputs True if an entire row is identical to a previous row.
>>> df.duplicated()0      False1      False9      False10     False14     False       ...  271    False278    False286    False299    False300    FalseLength: 80, dtype: bool
To consider certain columns for identifying duplicates, we can pass a list of columns to the argument subset:
>>> df.duplicated(subset=['Survived', 'Pclass', 'Sex'])0      False1      False9      False10      True14      True       ...  271     True278     True286     True299     True300     TrueLength: 80, dtype: bool
The result of the duplicated() is a boolean Series, and we can add them up to count the number of duplicates. Behind the theme, True get converted to 1 and False get converted to 0, then it adds them up.
# Count duplicate on a column>>> df.Cabin.duplicated().sum()11
Just like before, we can count the duplicate in a DataFrame and on certain columns.
# Count duplicate in a DataFrame>>> df.duplicated().sum()3# Count duplicate on certain columns>>> df.duplicated(subset=['Survived', 'Pclass', 'Sex']).sum()70
If you want to count the number of non-duplicates (The number of False), you can invert it with negation (~)and then call sum():
# Count the number of non-duplicates>>> (~df.duplicated()).sum()77
Pandas duplicated() returns a boolean Series. However, it is not practical to see a list of True and False when we need to perform some data analysis.
We can Pandas loc data selector to extract those duplicate rows:
# Extract duplicate rowsdf.loc[df.duplicated(), :]
loc can take a boolean Series and filter data based on True and False. The first argument df.duplicated() will find the rows that were identified by duplicated(). The second argument : will display all columns.
There is an argument keep in Pandas duplicated() to determine which duplicates to mark. keep defaults to 'first', which means the first occurrence gets kept, and all others get identified as duplicates.
We can change it to 'last' keep the last occurrence and mark all others as duplicates.
There is a third option we can use keep=False. It marks all duplicates as True and allows us to see all duplicate rows.
We can use Pandas built-in method drop_duplicates() to drop duplicate rows.
df.drop_duplicates()
Note that we started out as 80 rows, now it’s 77. By default, this method returns a new DataFrame with duplicate rows removed. We can set the argumentinplace=True to remove duplicates from the original DataFrame.
df.drop_duplicates(inplace=True)
The argument keep can be set for drop_duplicates() as well to determine which duplicates to keep. It defaults to 'first' to keep the first occurrence and drop all other duplicates.
Similarly, we can set keep to 'last' to keep the last occurrence and drop other duplicates.
# Use keep='last' to keep the last occurrence df.drop_duplicates(keep='last')
And we can set keep to False to drop all duplicates.
# To drop all duplicatesdf.drop_duplicates(keep=False)
Similarly, to consider certain columns for dropping duplicates, we can pass a list of columns to the argument subset:
# Considering certain columns for dropping duplicatesdf.drop_duplicates(subset=['Survived', 'Pclass', 'Sex'])
Pandas duplicated() and drop_duplicates() are two quick and convenient methods to find and remove duplicates. It is important to know them as we often need to use them during the data preprocessing and analysis.
I hope this article will help you to save time in learning Pandas. I recommend you to check out the documentation for the duplicated() and drop_duplicates() API and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
All Pandas cut() you should know for transforming numerical data into categorical data
Using Pandas method chaining to improve code readability
How to do a Custom Sort on Pandas DataFrame
All the Pandas shift() you should know for data analysis
When to use Pandas transform() function
Pandas concat() tricks you should know
Difference between apply() and transform() in Pandas
All the Pandas merge() you should know
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More tutorials can be found on my Github