How to Combine Data in Pandas — 5 Functions You Should Know | by Yong Cui | Towards Data Science

When we use Pandas to process data, one common task is to combine data from different sources. In this article, I’ll review 5 Pandas functions that you can use for data merging, as listed below. Each of these functions has its link to the official documentation if you want to take a look.
* concat* join* merge* combine* append
For the current tutorial, let’s use two simple DataFrame objects, as shown below. Please note that I’ll make slight modifications where applicable to better illustrate the corresponding functions.
The concat function is named after concatenation, which allows you to combine data side by side horizontally or vertically.
When you combine data that have the same columns (or most of them are the same, practically), you can call concat by specifying axis to 0, which is actually the default value too.
>>> pd.concat([df0, df1.rename(columns={"c": "a", "d": "b"})], axis=0)   a  b0  1  41  2  52  3  60  2  51  3  62  4  7
When you combine data with rows indicating the same entities (i.e., research data for the same ordered subjects) and columns of different data, you can concatenate them side by side.
>>> pd.concat([df0, df1], axis=1)   a  b  c  d0  1  4  2  51  2  5  3  62  3  6  4  7
By default, when you combine data horizontally (i.e., along the columns), Pandas tries to use the index. When they’re not identical, you’ll see NaNs to fill the non-overlapping ones, as shown below.
>>> df2 = df1.copy()>>> df2.index = [1, 2, 3]>>> pd.concat([df0, df2], axis=1)     a    b    c    d0  1.0  4.0  NaN  NaN1  2.0  5.0  2.0  5.02  3.0  6.0  3.0  6.03  NaN  NaN  4.0  7.0
If this isn’t the desired behavior and you want to them to align perfectly by ignoring the index, you should reset the index before concatenation:
>>> pd.concat([df0.reset_index(drop=True), df2.reset_index(drop=True)], axis=1)   a  b  c  d0  1  4  2  51  2  5  3  62  3  6  4  7
Please note that resetting index for df0 here is optional, because its index happens to be 0-based, which will match the reset index of df2.
Compared to concat, join is specialized in joining the columns between DataFrame objects using index.
>>> df0.join(df1)   a  b  c  d0  1  4  2  51  2  5  3  62  3  6  4  7
When the index are different, the joining keeps the rows from the left DataFrame by default. The rows from the right DataFrame with no matching index in the left DataFrame are removed, as shown below:
>>> df0.join(df2)   a  b    c    d0  1  4  NaN  NaN1  2  5  2.0  5.02  3  6  3.0  6.0
However, this behavior can be changed by setting the how parameter. The available options are: left, right, outer, and inner, and their behaviors are shown below.
# "right" uses df2’s index>>> df0.join(df2, how="right")     a    b  c  d1  2.0  5.0  2  52  3.0  6.0  3  63  NaN  NaN  4  7# "outer" uses the union>>> df0.join(df2, how="outer")     a    b    c    d0  1.0  4.0  NaN  NaN1  2.0  5.0  2.0  5.02  3.0  6.0  3.0  6.03  NaN  NaN  4.0  7.0# "inner" uses the intersection>>> df0.join(df2, how="inner")   a  b  c  d1  2  5  2  52  3  6  3  6
The merge function is mostly like joining actions in a database if you have such experience. Compared to join, merge is more general, which can execute merging operations on columns and indexes. Because we’ve covered join using index-based merging, we’ll be more focused on column-based merging. Let’s see a quick example.
>>> df0.merge(df1.rename(columns={"c": "a"}), on="a", how="inner")   a  b  d0  2  5  51  3  6  6
The on parameter defines which columns the two DataFrame objects will merge on. Please note that you can specify a single column as a string or multiple columns in a list. These columns must be present in both DataFrame objects. More generally, you can specify the merging columns from the left DataFrame and the right DataFrame, respectively, as shown below.
>>> df0.merge(df1, left_on="a", right_on="c")   a  b  c  d0  2  5  2  51  3  6  3  6
It’s pretty much the same result as the previous merging, except with separate columns of a and c.
There are many other optional parameters you can play with the merging. I want to highlight two here.
how: it defines what type of merging that is to be performed. Supported types include left, right, inner (the default one), outer, and cross. Everything should be straightforward except the last one cross, which creates the cartesian product from both frames, as shown below.
>>> df0.merge(df1, how="cross")   a  b  c  d0  1  4  2  51  1  4  3  62  1  4  4  73  2  5  2  54  2  5  3  65  2  5  4  76  3  6  2  57  3  6  3  68  3  6  4  7
suffixes: when the two DataFrame objects have the same columns other than to be merged, this parameter sets how these columns should be renamed to using the suffixes. By default, the suffixes for the left and right data frame is “_x” and “_y”, and you can supply your custom ones. Here’s an example.
>>> df0.merge(df1.rename(columns={"c": "a", "d": "b"}), on="a", how="outer", suffixes=("_l", "_r"))   a  b_l  b_r0  1  4.0  NaN1  2  5.0  5.02  3  6.0  6.03  4  NaN  7.0
The combine function perform column-wise combination between two DataFrame object, and it is very different from the previous ones. What makes combine special is that it takes a function parameter. This function takes two Series with each corresponding to the merging column from each DataFrame and returns a Series to be the final values for element-wise operations for the same columns. Sounds confusing? Let’s see an example in the following code snippet.
The taking_larger_square function operates on column a in df0 and df1 and column b in df0 and df1. Between the two a and two b columns, taking_larger_square takes the squares of the values from the larger column. In this case, df1’s columns a and b will be taken for squares, which produces the final values as shown in the above code snippet. For your reference, the renamed df1 is shown below.
>>> df1.rename(columns={"c": "a", "d": "b"})   a  b0  2  51  3  62  4  7
Although it’s not covered here, there is another closely related function combine_first, which simply combines two frames using the first frame’s non-null values.
So far, most of the operations we have discussed are towards combining the data column-wise. How about row-wise operations? The append function is specialized in appending rows to an existing DataFrame object, creating a new one. Let’s see an example first.
>>> df0.append(df1.rename(columns={"c": "a", "d": "b"}))   a  b0  1  41  2  52  3  60  2  51  3  62  4  7
Does the above operation look familiar to you? I hope it does, because it’s like what you can achieve with concat when you set axis=0. However, what makes append unique is that you can actually append a dict object, which provides us with the flexibility of appending different kinds of data.
>>> df0.append({"a": 1, "b": 2}, ignore_index=True)   a  b0  1  41  2  52  3  63  1  2
A trivial example is shown above. Please note that you must set ignore_index to True, because a dictionary object doesn’t have index information that the DataFrame can use.
In this article, we reviewed 5 mostly used functions for combining data in Pandas. Here’s a quick recap.
concat: combine data row-wise and column-wise
join: combine data row-wise using index
merge: combine data column-wise, like the database joining operations
combine: combine data column-wise having between-column (the same columns) element-wise operations
append: append data in the form of DataFrame or dict object row-wise
Thanks for reading this article. Stay connected by signing up my newsletter. Not a Medium member yet? Support my writing by using my membership link.