3 Key Differences Between Merge and Concat Functions of Pandas | by Soner Yıldırım | Towards Data Science

Pandas is a highly-efficient and widely used data analysis tool. The core data structure of Pandas is dataframe which represents data in tabular form with labeled rows and columns. There are many powerful and flexible functions and methods of DataFrame that ease and expedite the data cleaning and analysis process.
Data science projects usually require us to gather data from different sources. Hence, as part of data preparation, we may need to combine dataframes. Both concat and merge functions are used to combine dataframes. In this post, I will explain 3 key differences between them.
Concat function concatenates dataframes along rows or columns. We can think of it as stacking up multiple dataframes.
Merge combines dataframes based on values in shared columns. Merge function offers more flexibility compared to concat function because it allows combinations based on a condition.
Consider you have a dataset than contains credit card transactions. You have a bunch of columns about the details of a transaction and one column indicating customer ID. Another dataframe includes more detailed information about a customer and the customer ID as well. In order to combine these two dataframes, we can merge them on “customer ID” column so that entries match. A case that we would use concat is to combines dataframes that contain credit card transactions in 2019 and 2020. We can just stack them up using concat function.
Only concat function has axis parameter. Merge is used to combine dataframes side-by-side based on values in shared columns so there is no need for axis parameter.
We can concatenate dataframes along rows (on top of each other) or columns (side-by-side) depending on the value of axis parameter. The default value of the axis parameter is 0, which indicates combining along rows.
Let’s do some examples.
import numpy as npimport pandas as pddf1 = pd.DataFrame({'A':[1,2,3,4],'B':[True,False,True,True],'C':['C1','C2','C3','C4']})df2 = pd.DataFrame({'A':[5,7,8,5],'B':[False,False,True,False],'C':['C1','C3','C5','C8']})
Axis=0:
Axis=1:
Join is a parameter of concat function and how is a parameter of merge function. Their goal is the same but the way they work is a little different.
Join specifies how to handle indices in case the dataframes have different indices. Let’s create the previous dataframes with different indices.
df2 = pd.DataFrame({'A':[5,7,8,5],'B':[False,False,True,False],'C':['C1','C3','C5','C8']},index=[2,3,4,5])
We just changed the index of the second dataframe by passing index values to index parameter.
Join parameter takes two values, outer and inner.
Outer: Take all the indices (default value of join parameter)
Inner: Take only shared indices
We take all the indices but second dataframe do not have rows with 0 and 1 index. Thus, the values in the second datarame are filled with NaN (default missing value marker). Similarly, row 4 and 5 in the first dataframe are filled NaN values.
If we set join as inner, only the indices that exist in both dataframes will show up. So, we will not have any missing values due to concatenation.
The how parameter of the merge function works in a similar way. The possible values for how are inner, outer, left, right.
inner: only rows with same values in the column specified by on parameter (default value of how parameter)
outer: all the rows
left: all rows from left DataFrame
right: all rows from right DataFrame
The figures below represent the concept of how parameters more clearly.
Before doing some examples, let us recall our dataframes first.
We will merge df1 and df1 on column ‘C’.
The values that both df1 and df1 have in column C are C1 and C3 so only these values show up when how is set as ‘inner’.
How about outer?
When outer is chosen for how parameter, merged dataframe includes all values of column ‘C’. Common values are not duplicated and values that do not exist in df1 or df2 are filled with NaN.
When left is chosen for how parameter, merged dataframe includes all rows from left DataFrame. The columns that come from right dataframe are filled with NaN values if the value in column ‘C’ (the column that is passed to on parameter) is not present in right dataframe. Right option is rarely used because we can just change the order of dataframe in merge function (instead of (df1,df2) use (df2,df1)).
Thank you for reading. Please let me know if you have any feedback.