Joining Datasets with Python’s Pandas | by Thiago Carvalho | Towards Data Science

It’s definitely not uncommon to work with more than one dataset when performing your analysis. Therefore, there’s an abundant amount of methods to bring this data together.
SQL call those operations ‘Joins’ or ‘Unions’; in other languages and tools, you may find functions like Merge or LookUp to do the job.
In this article, I’ll go through some of the functions we can use to join datasets with Pandas.
I’ll run my code in a Jupyter notebook, and the only thing we need for the examples is Pandas.
import pandas as pd
We’ll start by defining some dummy data for the examples, I’ll use lists for simplification, but you’re definitely encouraged to load a dataset.
# create 5 lists with the same sizenames = ['bob', 'bella', 'blue', 'suzy', 'gunter', 'joe', 'holly', 'coco', 'elliot']species = ['dog', 'cat', 'velociraptor', 'dog', 'penguin', 'squid', 'cat', 'cat', 'horse']age = [1, 6, 70, 8, 3, 1, 2, 13, 3]weight = [10, 5, 15, 7, 4, 1, 3, 2, 380]color = ['brown', 'black', 'blue', 'black', 'black', 'gray', 'white', 'orange', 'white']
After defining the lists, we can create our data frames.
# create 3 data frames with the values from the listsdf1 = pd.DataFrame( {'name': names[:3],                     'species': species[:3],                     'age': age[:3]})df2 = pd.DataFrame( {'name': names[3:6],                     'species': species[3:6],                     'age': age[3:6]})df3 = pd.DataFrame( {'name': names[6:],                     'species': species[6:],                     'age': age[6:]})print(df1, '\n')print(df2, '\n')print(df3)
Ok, we have three datasets with the same columns and size. Let’s say we want to group those in a single data frame.
For that, we can use .concat, which is a function that accepts a list of data frames and concatenates them into one.
# .concat to join the dataframes, like a 'union all'df_list = [df1, df2, df3]df = pd.concat(df_list)df
Cool, Pandas matched the columns and returned an almost perfect data frame without much effort.
But what if the column names don’t match? or what if we’re missing a column?
# test with mismatching and missing columnsdf1 = pd.DataFrame( {'name': names[:3],                     'specie': species[:3],                     'age': age[:3]})df2 = pd.DataFrame( {'name': names[3:6],                     'species': species[3:6],                     'age': age[3:6]})df3 = pd.DataFrame( {'name': names[6:],                     'age - years': age[6:]})print(df1, '\n')print(df2, '\n')print(df3)
For the ‘species’ column, I changed its name in the first df and removed it from the last; I’ve also renamed ‘age’ to ‘age — years’.
# concat with mismatching and missing columnsdf_list = [df1, df2, df3]df = pd.concat(df_list)df
Ok, so Pandas .concat requires the names of the columns to be exact matches. If a column is unique to a dataset, just like a full join, it’ll fill the gaps with null values.
There’s one more thing we need to pay attention to, the indexes.
Since we didn’t define unique indexes when creating our data frame, Pandas set us with some default values starting from zero, but it doesn't reset them when we use .concat.
A unique index is always a good idea. In this case, we can use .reset_index to create a new column with proper values or use .set_index to define one of the columns as the index.
But let’s try an even simpler solution that fits our case.
df1 = pd.DataFrame( {'name': names[:3],                     'species': species[:3],                     'age': age[:3]})df2 = pd.DataFrame( {'name': names[3:6],                     'species': species[3:6],                     'age': age[3:6]})df3 = pd.DataFrame( {'name': names[6:],                     'species': species[6:],                     'age': age[6:]})# since we didn't define the indexes when creating the dataframes we can ignore them when concatenatingdf_list = [df1, df2, df3]df = pd.concat(df_list, ignore_index=True)df
With a single parameter, we ignored the indexes and got new ones in the concatenated result.
Another handy parameter is ‘keys’, which allows us to identify the data source with a new index level.
df1 = pd.DataFrame( {'name': names[:3],                     'species': species[:3],                     'age': age[:3]},                     index = [1,2,3])df2 = pd.DataFrame( {'name': names[3:6],                     'species': species[3:6],                     'age': age[3:6]},                     index = [10,11,12])df3 = pd.DataFrame( {'name': names[6:],                     'species': species[6:],                     'age': age[6:]},                     index = [100,200,300])# we can pass 'keys' which creates another index level to identify the concatenated data framesdf_list = [df1, df2, df3, df1]df = pd.concat(df_list, keys=['df1', 'df2', 'df3', 'df4'])df
By default, .concat uses the columns as keys and append the values as rows. But what if we want to concatenate columns to our data frame?
First, let’s define some new columns to concatenate.
df4 = pd.DataFrame( {'weight': weight,                     'color': color                     })df4
Similarly, we could add the lists directly to our data frame by assigning them to a column, like so:
df['color'] = color
But a data frame could have lots of fields, and passing them one by one wouldn’t be the best option.
The concatenate function accepts a parameter for ‘axis’, which allows us to do just that — concatenate columns.
df = pd.concat([df, df4], axis=1)df
We can also change the behaviour of the join.
Let’s try making the ‘name’ column our index and creating another dataset with an extra column to experiment with.
df_list = [df1, df2]df = pd.concat(df_list)df.set_index('name', inplace=True)df5 = pd.DataFrame( {'species': species,                     'age': age,                     'score': [9,10,10,8,6,9,3,4,10]},                     index = names)print(df, '\n')print(df5)
Note that our first data frame has fewer values than the second. When we perform an inner join, it should only bring the rows where the indexes match.
# by default concat behaves like an outer join, or a union all# we can change that with the 'join' parameterdf_list = [df, df5]df = pd.concat(df_list, axis=1, join='inner')df
Now let’s have a look at another function called .append.
This function has similar behaviour to .concat. The previous take multiple data frames and concatenates them into a new one. The append method will use an existing data frame to add the data.
Both will return a data frame, but the way you call them is different. You’ll use Pandas.concat() and DataFrame.append().
Let’s check some examples.
df1.append(df2)
df = df1.append(df2)df = df.append(df3)print(df, '\n')df = df1.append([df2, df3])print(df)
# append a rowdf.append(pd.Series(['oliver', 'monkey', 13], index=['name', 'species', 'age']), ignore_index=True) 
Awesome, with .concat and .append, we can perform most of the joins we might need. Now let’s check a more robust solution named .merge.
Starting with something simple, let’s see how .merge performs a join.
I’ll define another data frame similar to the one we’re already using, but with one more column and one less record.
print(df, '\n')df6 = pd.DataFrame( {'name': names[1:],                     'age': age[1:],                     'score': [10,10,8,6,9,3,4,10]})print(df6)
Merge allows us to select which column will be the key; in this case, let’s use ‘name’.
merged_df = pd.merge(df, df6, on='name')merged_df
Unlike what we saw earlier, a merge is by default an inner join — That means unless told otherwise, it’ll only return the matching rows from both datasets.
We can also notice that the columns present in both datasets are separated, even though they contain the same values.
Lastly, when we perform an inner join like the above, both data frames must have the key column with the same name.
You can select multiple columns as key, like a composite key, and you can also select which kind of join you’ll use.
merged_df = pd.merge(df, df6, how='left', on=['name', 'age'])merged_df
merged_df = pd.merge(df6, df, how='right', on=['name', 'age'])merged_df
Great! Besides all that, the merge function also helps us to validate and understand the data were merging.
# 'one_to_one' or ‘1:1’merged_df = pd.merge(df6, df.append(df1), how='right', on=['name', 'age'], validate='one_to_one')merged_df
For example, the validate parameter will raise an error if the data frames don’t respect the criteria you chose.
Other useful options are:
‘one_to_many’ or ‘1:m’ — Checks if the left keys are unique;
‘many_to_one’ or ‘m:1’— Checks if the right keys are unique;
The ‘indicator’ parameter adds a column to the data frame explaining the keys' relationship.
merged_df = pd.merge(df6, df.append(df1), how='outer', on=['name', 'age'], indicator=True)merged_df
*If you pass a string to the indicator parameter, it’ll be used as the created column's name.
We don’t need to specify which column contains the key since, by default, Pandas will assume the index is the key.
Let’s try setting the name as our index and merging those data frames again.
df.set_index('name', inplace=True)df6.set_index('name', inplace=True)print(df)print(df6)
pd.merge(df6, df)
If we don’t want Pandas to reset the index, we have to use the right_index and left_index parameters.
pd.merge(df6, df, how='outer', left_index=True, right_index=True)
If we want to merge with an index on one side and with a key on the other, we can specify the right_on and left_on parameters.
They accept the column's name holding the key, just like we saw before, but will be applied only to that specific side, using the index column on the other side.
pd.merge(df, df6, how='right', right_on='name', left_index=True)
Great! We got a look at .concat and .append, two convenient functions for joining two data frames. Then we explored .merge, an even better option with lots of flexibility.
Pandas have even more methods to help you work with multiple datasets; it’s not unusual to spend time building the logic to solve your problem and then finding out an already implemented solution for that in a library.
So I encourage you to get a look at some of those other functions such as .compare, .combine_first, and .merge_asof.
Thanks for reading my article. I hope you enjoyed it!
References:Python for Data Analysis — Wes McKinney;Pandas — Concat;Pandas — Merge;Pandas — Append;Pandas — Merge, join, concatenate and compare;