Pandas merge, concat, append: Master Advanced Joins | Towards Data Science

With each data science project or dataset, you want to perform several analyses and create plots to find insights. Often, the raw data never comes in one massive table but in many separate ones. To answer your questions, you should have the skills to join multiple tables into one and then perform operations on them.
You can acquire these skills by learning different kinds of merge operations such as inner join, left and right joins, self and anti joins, merging on indexes, etc.
∘ Introduction ∘ Setup ∘ Pandas merge() ∘ Basic Exploration ∘ Question 1: ∘ Question 2 ∘ Pandas merge(), suffixes ∘ Pandas merge(), left_on and right_on ∘ Pandas merging on indexes ∘ Pandas joining a table to itself ∘ Semi-join Pandas ∘ Anti-join Pandas ∘ Pandas concat() ∘ Pandas vertical concatenation of dataframes ∘ Pandas append() ∘ Types of Data Mappings ∘ Verifying concatenations ∘ Conclusion
The sample data and the notebook that was used in this article can be found in this GitHub repo.
Pandas provide several methods for performing merges on dataframes. Among all the others merge() method is the most flexible. It is a dataframe method and the general syntax is as follows:
df1.merge(df2, on='common_column')
When combining tables, there are two terminologies you should be familiar with: The name of the table you use first is called the left table while the other is called the right table. In the code snippet above, the left table is df1 and the right table is df2. Also, the verbs join, combine, and merge are all used interchangeably.
Now let’s see how we perform an inner join:
An inner join will only return rows that have matching values in both tables. During the joining process, you will have to know the common table name which exists in both tables.
Let’s say we have these two tables:
>>> user_usage.head()
>>> user_devices.head()
This data was downloaded from the KillBillerapplication. KillBiller was a free service that compared every mobile tariff in the UK and Ireland. The first, user_usage table contains monthly statistics of the mobile usage of users. user_devices table provides details about each user's phone such as operating system and phone model.
How many users use Android OS and how many use iOS?
To answer this question, we will need the information from both tables. There is one linking attribute between both tables: use_id. We will use this column in our merge:
Number of users for each OS: android    157                             ios          2Name: platform, dtype: int64
It looks like there is a huge difference between the two operating systems in our dataset.
In the merge above, we used an example of an inner join. In merge() function, how argument is set to inner by default so we did not have to write it out. When merging two tables using the merge() function, we use on argument to specify the common column. If there are multiple, it is also possible to pass a list of columns to the argument and pandas will take care of the rest.
Note that as a right table, I subset the user_devices table to exclude irrelevant columns to the question.
Now, as we explore further, we will notice that number of given users in two datasets is different:
Clearly, the number of matching user IDs in both tables were 159. This means there are user IDs which are in user_devices table and not in user_usage table and vice versa. So, the next question we want to ask is:
How many users use Android OS and iOS, including all the users that aren’t in user_usage table?
We can answer this question by using either a left or a right join. First, let’s look at the general case of one-sided joins:
A left join will return all of the rows of the left table including matching rows from the right table. A right join will return all of the rows of the right table including matching rows from the left table. Note that in both types of joins, pandas will put an NaN value to the cells with no match.
Number of users for each OS: android    184                             ios         88Name: platform, dtype: int64
The same result can be achieved using a left join. We can just swap the tables and pass left keyword for the how parameter:
Number of users for each OS: android    184                             ios         88Name: platform, dtype: int64
We use an outer join to return the intersection and all the other rows from both tables:
Let’s say we have columns with identical names in both of the tables. Since in our case, we don’t have columns with the same name I will just copy out one column into the other:
Now we have monthly_mb column in both tables. Let's see what happens when we join them:
If we look at the column names, the two columns with the same names have been renamed by adding _x and _y suffixes. Here, _x is for the left table and _y is for the right. This is pandas default behavior when we don't specify the suffixes. We can control the suffixes easily:
It is much better now. Use such suffixes to indicate the columns table of origin when they have identical names. Note that suffixes parameter takes a tuple of values, not a list and order does matter!
There is also a case where you want to join on a common column but the columns have different names. pandas provides a handy argument for doing just that without changing the names of the columns. Let's rename the use_id columns so that I can show you an example.
Now, let’s combine the tables with an inner join:
We use left_on and right_on parameters of the merge function to specify the names of common columns. You should pass the name of the column from the left table to left_on and vice versa. Note that when we use differently named common column, pandas will keep both of the columns as duplicates, so don't forget to drop one of them:
It is also possible to merge on a common index. The syntax for any type of join using indexes is the same. pandas is smart enough to know that you are referring to the index:
Things will get a little tricky when we have indexes with different names. Similar to columns, we will have to use left_on and right_on parameters. Also, there are two more parameters to indicate that we are merging on an index:
We set left_index and right_index to True to indicate we are merging on indexes. This also gives duplicate values: one as an index and one as a column. So, don't forget to drop the column when you are merging!
If the tables you are merging have MultiLevel indexes, you can pass a list of index names to the on or left/right_on parameters and you will be set!
There might be cases where you may want to join a table to itself. Joining a table to itself can be useful when you have: 1. Hierarchical relationships 2. Sequential relationships 3. Graph data
General case of joining a table to itself:
As you can see, a self merge can be pretty useful at times. When you use a self join, you often join on different columns of the same table.
All the joins we had done so far have returned data from both tables. But sometimes, you would want to filter your table based on values on some other table. pandas does not provide this functionality directly. But we can engineer the steps pretty easily.
Semi-joins are useful when you want to subset your data based on observations in other tables. Semi-joins:
1. Returns the intersection of two tables, similar to an inner join.
2. Returns only the columns from the left table, not the right.
3. No duplicates.
Back to our little tables:
As you can see, semi-join is exactly like an inner join but it only includes the results from the left table. Now back to our example data, we want to know the stats for all the users which exist in both tables. But we don’t want the final table to contain columns of the user_devices table. We can approach this problem in three steps:
After we have the inner join, we want to subset our original left table with the values in the final table:
The variable in_both now is a Boolean Series which we can use to subset the left table.
>>> users_in_both.shape(159, 4)
The results are as we expected. If you compare this join to the very first inner join we did, it is the exact same table without the columns of the right table:
>>> inner.shape(159, 8)
This is a tricky case. Please, reread and practice this section more to really understand it)))
Anti-join is the complete opposite of a semi-join. Anti-join:
1. Returns the left table, excluding the intersection
2. Returns only columns from the left table.
As an example, let’s do the complete opposite of the previous step in semi-join. Let’s find out all the users which are only in the user_usage table. Step 1:
Now, I introduce you to another helpful parameter: indicator. When set to True, the resulting table of the merge will have one extra column, _merge. It indicates whether a row is a result of both tables or one side. As you see, there is left_only string in the rows that did not have a match. Now, we want to subset for those rows:
I used .loc subsetting to immediately get the user IDs. Now we can use this pandas series to subset the original left table:
>>> left_only_users.shape(81, 4)
Again, this is a bit more complicated, so get practicing...
Finally, we covered most of the merge function. Now, pd.concat() is used for concatenating/adding two or more tables along a particular axis. It also provides some functionalities to manipulate axes. A common way the function is used is when adding tables vertically.
To add two or more tables vertically, we use concat() method, with the axis argument set to index.
Let’s load one of the sample tables in three chunks:
Pay attention to how I used the chunksize parameter! It is useful to know as it will be immensely helpful when loading large datasets!
When using concat() function, we always pass the tables to be concatenated in a list. To specify the concatenation axis, we use axis parameter. I highly recommend you pass string values to axis parameter such as index and columns. Passing 0 or 1 can be confusing for a lot of people. So, it is a good practice to improve code readability.
concat() function does not pay attention to axis labels. If you have numbered indexes in all of your tables, they will be mixed when you concatenate. So, a helpful paramter for resolving the issue is ignore_index:
This will reset the index and it will count from 0 to n-1.
Another use case for concat() is when you have similar tables but with different column names. When you concatenate tables with differing column names, use join parameter with a suitable string value.
Using concat with inner value resulted in a one-column dataframe since it is the only common column between tables. Using left, right, or outer would have expected results but values stacked vertically.
.append() method:
A simplified version of concat() method
A simplified version of concat() method
2. Supports: ignore_index
3. Does NOT support: join and other differences.
Another difference is that it is a DataFrame method. Meaning that it is called on a DataFrame rather than explicitly like pd.concat. Simple example:
We saw several types of joins above. In each one, we specifically told the methods the type of join we want and just took a leap of faith that it will return the correct result. However, there are methods to further ensure that we get the correct results. To do the task in code, you have to understand 4 types of data mappings in merges:
One to One
One to One
2. One to Many
3. Many to One
4. Many to Many
Let’s elaborate. When we use a one_to_one mapping, we expect that our join should return one and only one match for each row in the left table. If we are joining a table with 200 rows with a one_to_one mapping, the result should also contain 200 rows.
Similarly, if it is one_to_many, we allow multiple matches for a single row in our left table. Let’s how we implement the idea in code:
merge() method provides a helpful parameter for validating these kinds of merges. It accepts only four values:
one_to_one
one_to_one
2. one_to_many
3. many_to_one
4. many_to_many
What’s more, pandas will raise a MergeError if we validate on an incorrect mapping:
https://gist.github.com/64bb2102575ffc9edf587cebbcd59605
As you can see, the merge raised an error. Let’s set it to the correct mapping:
It is also possible to validate on axes when using .concat() method. It has verify_integrity parameter which is set to False by default. When set to True, it will raise a ValueError if the axis or columns have duplicate names:
Finally, the things I wanted to cover in this article are over. We talked about so many different methods for joining data. Believe it or not, there is still more to this topic such as merge_ordered() and merge_asof() methods but they might be a topic for another article.
This just shows how deep and large pandas library is and what you can do with it. I intended this article as a one-stop place to learn about combining data. Hope you will use it in practice.
Read more articles related to the topic: