Python | Pandas dataframe.drop_duplicates()

30 Jun, 2021
Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric python packages. Pandas is one of those packages and makes importing and analyzing data much easier.An important part of Data analysis is analyzing Duplicate Values and removing them. Pandas drop_duplicates() method helps in removing duplicates from the data frame. 
Syntax: DataFrame.drop_duplicates(subset=None, keep=’first’, inplace=False)Parameters: subset: Subset takes a column or list of column label. It’s default value is none. After passing columns, it will consider them only for duplicates. keep: keep is to control how to consider duplicate value. It has only three distinct value and default is ‘first’.  
If ‘first’, it considers first value as unique and rest of the same values as duplicate.
If ‘last’, it considers last value as unique and rest of the same values as duplicate.
If False, it consider all of the same values as duplicates
inplace: Boolean values, removes rows with duplicates if True.Return type: DataFrame with removed duplicate rows depending on Arguments passed.  
To download the CSV file used, Click Here. 
 
How to Remove Duplicate Rows in Pandas Dataframe? | GeeksforGeeks - YouTubeGeeksforGeeks529K subscribersHow to Remove Duplicate Rows in Pandas Dataframe? | GeeksforGeeksWatch laterShareCopy linkInfoShoppingTap to unmuteIf playback doesn't begin shortly, try restarting your device.More videosMore videosYou're signed outVideos you watch may be added to the TV's watch history and influence TV recommendations. To avoid this, cancel and sign in to YouTube on your computer.CancelConfirmSwitch cameraShareInclude playlistAn error occurred while retrieving sharing information. Please try again later.Watch on0:000:000:00 / 5:37•Live•<div class="player-unavailable"><h1 class="message">An error occurred.</h1><div class="submessage"><a href="https://www.youtube.com/watch?v=e5oOvU0VWBQ" target="_blank">Try watching this video on www.youtube.com</a>, or enable JavaScript if it is disabled in your browser.</div></div>
Example #1: Removing rows with same First Name In the following example, rows having same First Name are removed and a new data frame is returned. 
Python
# importing pandas packageimport pandas as pd # making data frame from csv filedata = pd.read_csv("employees.csv") # sorting by first namedata.sort_values("First Name", inplace = True) # dropping ALL duplicate valuesdata.drop_duplicates(subset ="First Name",                     keep = False, inplace = True) # displaying datadata
Output: As shown in the image, the rows with same names were removed from data frame.  
Example #2: Removing rows with all duplicate values In this example, rows having all values will be removed. Since the csv file isn’t having such a row, a random row is duplicated and inserted in data frame first. 
Python
#importing pandas packageimport pandas as pd # making data frame from csv filedata = pd.read_csv("employees.csv") #length before adding rowlength1 = len(data) # manually inserting duplicate of a row of row 440data.loc[1001] = [data["First Name"][440],                  data["Gender"][440],                  data["Start Date"][440],                  data["Last Login Time"][440],                  data["Salary"][440],                  data["Bonus %"][440],                  data["Senior Management"][440],                  data["Team"][440]]                    # length after adding rowlength2=  len(data) # sorting by first namedata.sort_values("First Name", inplace=True) # dropping duplicate valuesdata.drop_duplicates(keep=False,inplace=True) # length after removing duplicateslength3=len(data) # printing all data frame lengthsprint(length1, length2, length3)
Output: As shown in the output image, the length after removing duplicates is 999. Since the keep parameter was set to False, all of the duplicate rows were removed. 
 
simmytarika5
Python pandas-dataFrame
Python pandas-dataFrame-methods
Python-pandas
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Different ways to create Pandas Dataframe
Enumerate() in Python
How to Install PIP on Windows ?
*args and **kwargs in Python
Python Classes and Objects
Python OOPs Concepts
Introduction To PYTHON
Python | os.path.join() method
Convert integer to string in Python
Create a Pandas DataFrame from Lists