Join the Tables. Understanding the merge() and the... | by Suraj Gurav | Towards Data Science

A data science project begins with the data. The required data is gathered from different sources, assessed, cleaned, and stored for further analysis and modeling.
Data is the key ingredient in data science project
However, having multiple datasets in a single project is quite confusing. While data cleaning, the same process needs to be applied to all the datasets, which is a tedious task. This might also result in some errors. In such cases, it is a good idea to combine all the datasets together to get a single view of the entire data. This process is often necessary when the raw data is collected from different files, worksheets, data tables, and even from web scraping.
towardsdatascience.com
In Python, there are two ways to combine the data from different DataFrames. Hence, this is the story of these 2 methods —
pandas.DataFrame.merge Vs pandas.concat
DataFrame being the most commonly used pandas object is used to store tabular data of different data types in the form of rows and columns.
merge() and Concat() are two very useful methods in pandas to combine the different DataFrames
Let us create three DataFrames, df_one, df_two, df_three
countries = {'ID':["EU101","AS101","EU103","EU102","US111"],\             'Country': ["Germany", "India", "Belgium", "Austria",   "Mexico"]}states = {'Countryname': ["France", "India", "Belgium", "Japan", "Mexico","Spain"],\         'statecount': [15, 7, 10, 9, 32, 10]}europe ={'ID':["EU101","AS101","EU103","EU102","US112"],\        'EU_member': ["Yes", "No", "No", "No", "No"],\        'Airports': [36,126,5,6,77]}df_one = pd.DataFrame(countries, index = [0,1,2,3,4])df_two = pd.DataFrame(states, index = [1,2,3,4,5,6])df_three = pd.DataFrame(europe, index = [0,1,2,3,4])
It couples the pandas DataFrames along a particular axis. It attaches the datasets together either along rows or columns.
Suppose, you want to combine the three DataFrames df_one, df_two, and df_three row-wise so that they will form a single DataFrame.
Using Concat(), you just need to pass the names of the DataFrames in a list.
df_four = pd.concat([df_one, df_two])
By default, both the DataFrames are simply stitched together, one below other. All the columns and rows from both the DataFrames are retained.
To join the tables side-by-side, an additional argument axis = 1 must be added to concat().
df_four = pd.concat([df_one, df_two], axis = 1)
Simple !!
All the DataFrames are connected together as a single DataFrame. But wait, how would I know, which data comes from which DataFrame ??
Pandas have an answer for it. Pandas provide an option to label the DataFrames with a key, so that, after concatenation, you will know the DataFrame from where the data comes. Such labeling of data actually makes it easy to extract the data corresponding to a particular DataFrame.
df_four = pd.concat([df_one, df_two], keys=[“df_one”,”df_two”])
While combining the data from multiple DataFrames, we really have control over what to get in the resulting DataFrame. It can be realized using thejoin argument in Concat(). Pandas give us 2 options in this argument.
outer : Union of DataFrames
outer : Union of DataFrames
df_five = pd.concat([df_one, df_two], axis = 1, join = 'outer',\                   keys=['df_one', 'df_two'])
The rows corresponding to all the indices are retained in the resulting DataFrame.
inner : Intersection of DataFrames
inner : Intersection of DataFrames
df_five = pd.concat([df_one, df_two], axis = 1, join = 'inner',\                   keys=['df_one', 'df_two'])
In an inner join, all the indices common to both the DataFrames df_one and df_two are retained in the resulting DataFrame. Pandas perform outer join along rows by default. Documented information about it can be found here.
It combines DataFrames in database-style, i.e. DataFrames are joined on common columns or indices. Often, we deal with the different datasets which have different types of data about the same entity. Such datasets are usually linked to each other through one or two common columns.
merge() combines the DataFrames on the basis of values from common columns.
df_four = df_one.merge(df_three)
As you can see, both the tables, df_oneand df_three are joined on the common column “ID” which is available in both the tables and the rows which have the same value for column “ID” in both the tables will be retained in the resulting table. merge() automatically identifies the common columns in both the tables.
With its long list of customization options, merge() is considered as the most flexible method to join the tables or DataFrames.
Most often statement I heard about merge() is,
merge() joins the tables in SQL-style
You don’t know about SQL or Structured Query Language ???
No problem at all. Here I show the simplest way to understand merge(). Everyone is aware of Venn diagrams in set theory and the basics of Venn diagrams such as the union, the intersection of sets. Below four pictures will give your simplest and the clearest view of merge(). The pictures are self-explanatory by just looking at the colors.
merge() provides 4 options to get control over the resulting dataset and these can be passed in “how” argument within merge(), such as,
df_one.merge(df_three, how="inner")
By default, merge() provides inner join, hence no need to mention it specifically. An inner join is similar to the intersection of sets whereas the outer join is similar to the union of sets.
So, here are the merge() options,
In merge(), there is a huge list of optional arguments are available which can be explored here.
pandas.DataFrame.join() can be considered as a simple form of merge() with less number of optional arguments. It does merge()-like join operations. Unlike merge(), join() couples the two DataFrames on common indices. Hence, to connect both the tables or DataFrames together, we should set the same column as an index in both the DataFrames. This table joining option can also be explored in the documentation.
Thanks for your time and reading !!!
I hope you enjoyed this simple explanation about pandas two important features to join the datasets. Whether to choose Concat() or merge() is completely our choice but I mostly use merge() as it offers a wide range of options. If you liked the article, then feel free to add feedback and connect with me on LinkedIn.