Concatenate Pandas DataFrames Without Duplicates

16 Feb, 2022
In this article, we are going to concatenate two dataframes using pandas module.
In order to perform concatenation of two dataframes, we are going to use the pandas.concat().drop_duplicates() method in pandas module.
Step-by-step Approach: 
Import module.
Load two sample dataframes as variables.
Concatenate the dataframes using pandas.concat().drop_duplicates() method.
Display the new dataframe generated.
Below are some examples which depict how to perform concatenation between two dataframes using pandas module without duplicates:
Example 1:
Python3
# Importing pandas libraryimport pandas as pd # loading dataframesdataframe1 = pd.DataFrame({'columnA': [20, 30, 40],                           'columnB': [200, 300, 400]}) dataframe2 = pd.DataFrame({'columnA': [50, 20, 60],                           'columnB': [500, 200, 600]}) # Concatenating dataframes without duplicatesnew_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates() # Display concatenated dataframenew_dataframe
Output:
Here, we have concatenated two dataframes using pandas.concat() method.
Example 2:
Python3
# Importing pandas libraryimport pandas as pd # loading dataframesdataframe1 = pd.DataFrame({'name': ['rahul', 'anjali', 'kajal'],                           'age': [23, 28, 30]}) dataframe2 = pd.DataFrame({'name': ['devesh', 'rashi', 'anjali'],                           'age': [20, 15, 28]}) # Concatenating two dataframes without duplicatesnew_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates() # Resetting indexnew_dataframe = new_dataframe.reset_index(drop=True) # Display dataframe generatednew_dataframe
Output:
As shown in the output image, we get the concatenation of dataframes without removing duplicates.
Example 3:
Python3
# Importing pandas libraryimport pandas as pd # Loading dataframesdataframe1 = pd.DataFrame({'empname': ['rohan', 'hina', 'alisa', ],                           'department': ['IT', 'admin', 'finance', ],                           'designation': ['Sr.developer', 'administrator', 'executive', ]}) dataframe2 = pd.DataFrame({'empname': ['rishi', 'huma', 'alisa', ],                           'department': ['cyber security', 'HR', 'finance', ],                           'designation': ['penetration tester', 'HR executive', 'executive', ]}) # Concatenating two dataframes without duplicatesnew_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates() # Resetting indexnew_dataframe = new_dataframe.reset_index(drop=True) # Display dataframe generatednew_dataframe
Output:
Here is another example, which depicts how to concatenate two dataframes.
Output dataset for the Example 3
idevesh
germanshephered48
Picked
Python pandas-dataFrame
Python-pandas
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Python Classes and Objects
Python OOPs Concepts
Introduction To PYTHON
How to drop one or multiple columns in Pandas Dataframe
Python | os.path.join() method
Check if element exists in list in Python
How To Convert Python Dictionary To JSON?
Python | Get unique values from a list
Create a directory in Python