Find duplicate rows in a Dataframe based on all or selected columns

16 Feb, 2022
In this article, we will be discussing how to find duplicate rows in a Dataframe based on all or a list of columns. For this, we will use Dataframe.duplicated() method of Pandas. 
Syntax : DataFrame.duplicated(subset = None, keep = ‘first’)Parameters: subset: This Takes a column or list of column label. It’s default value is None. After passing columns, it will consider them only for duplicates.keep: This Controls how to consider duplicate value. It has only three distinct value and default is ‘first’.  
If ‘first’, This considers first value as unique and rest of the same values as duplicate.
If ‘last’, This considers last value as unique and rest of the same values as duplicate.
If ‘False’, This considers all of the same values as duplicates.
Returns: Boolean Series denoting duplicate rows.  
Let’s create a simple dataframe with a dictionary of lists, say column names are: ‘Name’, ‘Age’ and ‘City’.  
Python3
# Import pandas libraryimport pandas as pd # List of Tuplesemployees = [('Stuti', 28, 'Varanasi'),            ('Saumya', 32, 'Delhi'),            ('Aaditya', 25, 'Mumbai'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Mumbai'),            ('Aaditya', 40, 'Dehradun'),            ('Seema', 32, 'Delhi')            ] # Creating a DataFrame objectdf = pd.DataFrame(employees,                  columns = ['Name', 'Age', 'City']) # Print the Dataframedf
Output :  
Example 1: Select duplicate rows based on all columns. Here, We do not pass any argument, therefore, it takes default values for both the arguments i.e. subset = None and keep = ‘first’. 
Python3
# Import pandas libraryimport pandas as pd # List of Tuplesemployees = [('Stuti', 28, 'Varanasi'),            ('Saumya', 32, 'Delhi'),            ('Aaditya', 25, 'Mumbai'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Mumbai'),            ('Aaditya', 40, 'Dehradun'),            ('Seema', 32, 'Delhi')            ] # Creating a DataFrame objectdf = pd.DataFrame(employees,                  columns = ['Name', 'Age', 'City']) # Selecting duplicate rows except first# occurrence based on all columnsduplicate = df[df.duplicated()] print("Duplicate Rows :") # Print the resultant Dataframeduplicate
Output :  
Example 2: Select duplicate rows based on all columns. If you want to consider all duplicates except the last one then pass keep = ‘last’ as an argument. 
Python3
# Import pandas libraryimport pandas as pd # List of Tuplesemployees = [('Stuti', 28, 'Varanasi'),            ('Saumya', 32, 'Delhi'),            ('Aaditya', 25, 'Mumbai'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Mumbai'),            ('Aaditya', 40, 'Dehradun'),            ('Seema', 32, 'Delhi')            ] # Creating a DataFrame objectdf = pd.DataFrame(employees,                  columns = ['Name', 'Age', 'City']) # Selecting duplicate rows except last# occurrence based on all columns.duplicate = df[df.duplicated(keep = 'last')] print("Duplicate Rows :") # Print the resultant Dataframeduplicate
Output :  
Example 3: If you want to select duplicate rows based only on some selected columns then pass the list of column names in subset as an argument.  
Python3
# import pandas libraryimport pandas as pd # List of Tuplesemployees = [('Stuti', 28, 'Varanasi'),            ('Saumya', 32, 'Delhi'),            ('Aaditya', 25, 'Mumbai'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Mumbai'),            ('Aaditya', 40, 'Dehradun'),            ('Seema', 32, 'Delhi')            ] # Creating a DataFrame objectdf = pd.DataFrame(employees,                  columns = ['Name', 'Age', 'City']) # Selecting duplicate rows based# on 'City' columnduplicate = df[df.duplicated('City')] print("Duplicate Rows based on City :") # Print the resultant Dataframeduplicate
Output :  
Example 4: Select duplicate rows based on more than one column name. 
Python3
# import pandas libraryimport pandas as pd # List of Tuplesemployees = [('Stuti', 28, 'Varanasi'),            ('Saumya', 32, 'Delhi'),            ('Aaditya', 25, 'Mumbai'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Delhi'),            ('Saumya', 32, 'Mumbai'),            ('Aaditya', 40, 'Dehradun'),            ('Seema', 32, 'Delhi')            ] # Creating a DataFrame object df = pd.DataFrame(employees,                   columns = ['Name', 'Age', 'City']) # Selecting duplicate rows based# on list of column namesduplicate = df[df.duplicated(['Name', 'Age'])] print("Duplicate Rows based on Name and Age :") # Print the resultant Dataframeduplicate
Output :  
 
reenadevi98412200
Python pandas-dataFrame
Python-pandas
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.