All the Pandas merge() you should know for combining datasets | by B. Chen | Towards Data Science

Pandas provides various built-in functions for easily combining datasets. Among them, merge() is a high-performance in-memory operation very similar to relational databases like SQL. You can use merge() any time when you want to do database-like join operations.
In this article, we’ll be going through some examples of combining datasets using Pandas merge() function. We will cover the following common usages and should help you get started with data combinations.
The simplest call without any key columnSpecifying key columns using onMerging using left_on and right_onVarious forms of joins: inner, left, right and outerUsing validate to avoid invalid records
The simplest call without any key column
Specifying key columns using on
Merging using left_on and right_on
Various forms of joins: inner, left, right and outer
Using validate to avoid invalid records
Please check out Notebook for the source code.
When you use merge(), the simplest call must have two arguments: the left DataFrame and the right DataFrame. For example, to combine df_customer and df_info:
df_customer = pd.DataFrame({    'id': [1, 2, 3, 4],    'name': ['Tom', 'Jenny', 'James', 'Dan'],})df_info = pd.DataFrame({    'id': [2, 3, 4, 5],    'age': [31, 20, 40, 70],    'sex': ['F', 'M', 'M', 'F']})pd.merge(df_customer, df_info)
By default, the function will combine data on common columns (It is the column id in our example) and produces only the result that matches in both left and right DataFrames.
The following is an equivalent statement if you prefer to call merge from the left DataFrame.
df_customer.merge(df_info) 
You can specify the common columns for merging. To do so, pass an additional argument on as the name of the common column, here 'id' in our example, to merge() function:
pd.merge(df_customer, df_info, on='id')
If you use on , you have to make sure the column you specify must be present in both left and right DataFrames.
To combine data on multiple common columns, you can pass a list toon:
pd.merge(df_customer, df_order, on=['id', 'name'])
It might happen that the column on which you want to merge the DataFrames have different names. For such merges, you will have to specify the left_on as the left DataFrame name and right_on as the right DataFrame name, for example:
pd.merge(  df_customer,   df_info_2,   left_on='id',   right_on='customer_id')
The result will contain both id and customer_id columns.
They are 4 types of joins available to Pandas merge() function. The logic behind these joins is very much the same that you have in SQL when you join tables. You can perform a type of join by specifying the how argument with the following values:
inner: the default join type in Pandas merge() function and it produces records that have matching values in both DataFrames
left: produces all records from the left DataFrame and the matched records from the right DataFrame
right: produces all records from the right DataFrame and the matched records from the left DataFrame
outer: produces all records when there is a match in either left or right DataFrame
And below is how the Venn Diagram looks like for our test dataset
df_customer = pd.DataFrame({    'id': [1,2,3,4],    'name': ['Tom', 'Jenny', 'James', 'Dan'],})df_info = pd.DataFrame({    'id': [2,3,4,5],    'age': [31,20,40,70],    'sex': ['F', 'M', 'M', 'F']})pd.merge(df_customer, df_info, on='id', how=?)
By default, Pandas merge() is performing the inner join and it produces only the set of records that match in both DataFrame.
pd.merge(df_customer, df_info, on='id')
And below is the equivalent SQL query:
SELECT * from customerINNER JOIN infoON customer.id = info.id
To explicitly specify the inner join, you can set the argument how='inner'
pd.merge(df_customer, df_info, how='inner', on='id')
The left join produces all records from the left DataFrame, and the matched records from the right DataFrame. If there is no match, the left side will contain NaN. You can set the argument how='left' to do left join:
pd.merge(df_customer, df_info, how='left', on='id')
And below is the equivalent SQL query:
SELECT * from customerLEFT OUTER JOIN infoON customer.id = info.id
The right join produces all records from the right DataFrame, and the matched records from the left DataFrame. If there is no match, the right side will contain NaN. You can set the argument how='right' to do right join:
pd.merge(df_customer, df_info, how='right', on='id')
And below is the equivalent SQL query:
SELECT * from customerRIGHT OUTER JOIN infoON customer.id = info.id
The outer join produces all records when there is a match in either left or right DataFrame. NaN will be filled for no match on either sides. You can set the argument how='outer' to do outer join:
pd.merge(df_customer, df_info, how='outer', on='id')
And below is the equivalent SQL query:
SELECT * from customerFULL OUTER JOIN infoON customer.id = info.id
The result of merge() might have an increased number of rows if the merge keys are not unique. For example
df_customer = pd.DataFrame({    'id': [1,2,3,4],    'name': ['Tom', 'Jenny', 'James', 'Dan'],})df_order_2 = pd.DataFrame({    'id': [2, 2, 4, 4],    'product': ['A', 'B' ,'A', 'C'],    'quantity': [31, 21, 20,40],    'date': pd.date_range('2019-02-24', periods=4, freq='D')})
Both df_customer and df_order_2 have 4 records. But, you will get a result with 6 records when running the following merge statement:
pd.merge(df_customer, df_order_2, how='left', on='id')
Here are the reasons:
the how='left' will produce all records from df_customer, and the matched records from df_order_2.
In addition, the id in df_order_2 is not unique and all the matching records will be combined and returned.
This is an example of one-to-many merge. It is a valid scenario in our example, in which a customer can have multiple orders. However, one-to-many might be invalid in some other cases, for example, there are two records with the id value 2 in df_info
df_customer = pd.DataFrame({    'id': [1, 2, 3, 4],    'name': ['Tom', 'Jenny', 'James', 'Dan'],})df_info = pd.DataFrame({    'id': [2, 2, 3, 4, 5],    'age': [31, 21, 20, 40, 70],    'sex': ['F', 'F', 'M', 'M', 'F']})
And the merge result will be ended up with 2 different records for the same customer Jenny:
pd.merge(df_customer, df_info, how='left', on='id')
This is certainly wrong because the same customer cannot have different information. To avoid this problem, we can set the argument validate to '1:1' , so it checks if merges keys are unique in both left and right DataFrames. It will raise a MergeError if the validation fails, for example:
pd.merge(df_customer, df_info, how='left', on='id', validate='1:1')
The argument validate takes one of the following values, so you can use it to validate different merge outputs.
one_to_one or 1:1 : check if merge keys are unique in both left and right datasets.
one_to_many or 1:m: check if merge keys are unique in left dataset.
many_to_one or m:1: check if merge keys are unique in right dataset.
many_to_many or m:m: allowed, but does not result in checks.
Pandas merge() function is a simple, powerful, and high-performance in-memory operation very similar to relational databases like SQL.
I hope this article will help you to save time in combining datasets. I recommend you to check out the documentation for the merge() API and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
Pandas resample() tricks you should know for manipulating time-series data
How to do a Custom Sort on Pandas DataFrame
When to use Pandas transform() function
Pandas concat() tricks you should know
Difference between apply() and transform() in Pandas
Using Pandas method chaining to improve code readability
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More tutorials can be found on my Github