Merging DataFrames with pandas | pd.merge()
By: Ravjot Singh
Tags: Data Manipulation, Machine Learning, Artificial Intelligence, Python, Data Science

Types of Merging DataFrame in Python

In this article, you’ll learn how multiple DataFrames could be merged in python using Pandas library.

Introduction

Merging DataFrames is the core process to start with data analysis and machine learning tasks. It is one of the tool-kits which every Data Analyst or Data Scientist should master because in almost all the cases data comes from multiple source and files. You may need to bring all the data in one place by some sort of join logic and then start your analysis.

People who work with SQL like query languages might know the importance of this task. Pandas has full-featured, high performance in-memory join operations idiomatically very similar to relational databases like SQL.

Hands on Learning

1. Importing the required libraries

2. Loading the datasets

We are loading the datasets using the pandas library- pd.read_csv() function.

Importing the datasets on which we will be working

3. Data Analysis Part

Adding a new column “NOC” to Bronze Medals dataset which consists of short name codes of each country in the dataset.

Creating a new column in Bronze Medals dataset

Changing the order of the columns of Bronze Medals dataset.

Changing the order of columns of the dataset with the inbuilt function by just passing the order in a list

Similarly, assigning a new column “NOC” to Silver Medals dataset, which consists of short name codes.

Creating a new column in Silver Medals dataset

Changing the order of the columns of Silver Medals dataset.

4. Merging Datasets

Merging ALL columns

This function takes all the columns of both the datasets to merge i.e. the rows of merge dataframe consists of all rows where the “NOC”, “Country” and “Total” columns are identical in both the dataframes Bronze and Silver Medals.

Empty DataFrame

Thus this merge gives us an empty dataframe because the columns values doesn’t matches.

Merging ON

So now, we our taking a particular column to merge on. “NOC” in our case.

Passing “on” parameter to pd.merge() function

All the remaining columns are appended to our right. The column labels that are identical will change with their suffixes _x and _y according to their origin.

_x — for the first argument to merge _y — for the second argument

Merging ON Multiple Columns

As the country_x and country_y columns are same.

To eliminate the identical columns use “Country” column in the “on” parameter along with “NOC” to merge on both the columns.

Passing multiple columns to “on” parameter

In this way, merge function differs from concatinating (pd.concat() funtion) as merging is allowing to merge on multiple columns.

Using SUFFIXES

We can tailor the column labels with “suffixes” parameter. With this function we can take whatever custom names to choose for the column labels that are identical.

Passing “suffixes” parameter inside the pd.merge() function

Merging with INNER join

Function merge does an inner join by default i.e. it extracts the rows that matches in the joining column (‘on’ parameter) from both dataframes and enclose them together in a joined dataframe.

Inner join is by default

We can specify how=’inner’ but it is a default behavior for the function.So, it is optional.

Merging with LEFT join

By specifying how=’left’ keeps all the rows of the left dataframe in the merged dataframe.

2. For rows in the left dataframe with matches in the right dataframe

Non-joining columns of right dataframe are appended to the left dataframe.

3. For rows in the left dataframe with no matches in the right dataframe

Non joining columns are filled with nulls.

Merging with left join

For example- The row from the Germany is retained with a null value. Because Silver Medals dataframe have no Germany row.

Merging with RIGHT join

Conversely, we can do right join.

Merging with right join

In this example we have retained all 5 rows from the Silver Medals dataframe. This time Italy has nan entry in the total bronze medals column beacuse Bronze Medals dataframe has no Italy row.

Merging with OUTER join

Finally the union of all the rows from both the dataframes can be preserved with an outer join.

Merging with outer join

With the argument how=’outer’, we can clearly see that Germany is not present in the Silver medals winning countries and Italy in the Bronze medals winning countries.

5. Updating DataFrame

Filling the NaN values in the dataframe with zeros using .fillna() function.

Filling all the NaN values with 0 (zero)

Updated DataFrame

All the NaN values are filled with zeros.

Adding a new column “Total Medals” that adds the Bronze and Silver medals of each country.

Created new column “Total Medals”

6. Visualizations Part

Some visualizations are made to get some better insights from the data.

6A) Showing the barplot of Total Bronze and Silver Medals of Each country

The 1st barplot is plotted between Total bronze medals and country column of the dataset.

The 2nd barplot is plotted between Total silver medals and country column of the dataset.

From both the plots, it can be concluded that United States have won more Bronze and Silver medals as compared to other countries.

6B) Showing Total medals vs Country column

This plot shows the count of Total Medals won by each country.

6C) Gridspec

Code:

Output:

This brings us to the end of this article. I hope you have understood the basics of Merging DataFrames using pandas in Python clearly. Make sure you practice as much as possible.

If you wish to check out more resources related to Data Science and Machine Learning you can refer to my Github account.

You can also check my Data Science Portfolio on Github account.