Joining DataFrames in Julia. Learn all the joins — inner, outer... | by Bence Komarniczky | Towards Data Science

What is a join? Why would we do it? And how would we do it using DataFrames.jl? In this post, I’ll show some practical but simple examples on how to join DataFrames.
For full access to all Medium articles — including mine — consider subscribing here.
Last time, we figured out how to index, sort, and aggregate our data using DataFrames.jl. Joins is another very common and important operation that arises in the world of tabulated data. A join across two DataFrames is the action of combining the two datasets based on shared column values that exist across the two tables. We call this column (or columns) the key. So, each record from the first table is matched to a record in the second table — as long as the records’ values are the same. Let’s demonstrate this via a small example. First, we set up table A:
5×2 DataFrame Row │ id     name     │ Int64  String─────┼───────────────   1 │     1  Alice   2 │     2  Bob   3 │     3  Claire   4 │     4  Daniel   5 │     5  Edward
This table has all the ids of individuals and their names. Let’s say we have another table, which has the earnings of these individuals:
6×2 DataFrame Row │ id     salary     │ Int64  Int64─────┼───────────────   1 │     3    4078   2 │     4    2395   3 │     5    3662   4 │     6    2202   5 │     7    2564   6 │     8    4545
We now have 2 tables:
A_names holds the ids and names of people
B_earnings holds the ids and earnings.
We would like to combine the 2 tables so that we can see the names and earnings together. We do a join!
3×3 DataFrame Row │ id     name    salary     │ Int64  String  Int64─────┼───────────────────────   1 │     3  Claire    4078   2 │     4  Daniel    2395   3 │     5  Edward    3662
Let’s go through this in detail. Arguments 1 and 2 are the two tables we’re joining. Argument 3 (on) tells us what the key column is. We’ll use this column to match the observations across the tables.
As you can see, we ended up with 3 rows and 3 columns. Now go back to the beginning and check out how the 2 original datasets looked like. Make sure you understand why we ended up with these rows only.
Since we’re using innerjoin, we’re only keeping ids that appear in both datasets — everything else we throw away.
There are other kinds of joins too:
inner join: take the ids that appear in both tables
left join: take all the ids from the left (first) table and the value of the right table
right join: same as left join but keep all the ids from the second table
outer join: take all ids from both tables, regardless of whether or not they appear in the other table
Fear not, we will now go through all of these joins one-by-one! First, let’s do a left join and see what happens to ids that are only in the first dataset:
5×3 DataFrame Row │ id     name    salary     │ Int64  String  Int64?─────┼────────────────────────   1 │     1  Alice   missing   2 │     2  Bob     missing   3 │     3  Claire     4078   4 │     4  Daniel     2395   5 │     5  Edward     3662
Here we kept all of the observations from table A no matter what is going on in table B. For records without a match in table A, the earnings column has missingas value. This makes sense as we never actually saw those earning figures.Of course, there is also a right join. This keeps all rows from the second table.
If you want to have all ids from both tables, use an outer join:
8×3 DataFrame Row │ id     name     salary     │ Int64  String?  Int64?─────┼─────────────────────────   1 │     1  Alice    missing   2 │     2  Bob      missing   3 │     3  Claire      4078   4 │     4  Daniel      2395   5 │     5  Edward      3662   6 │     6  missing     2202   7 │     7  missing     2564   8 │     8  missing     4545
Loads more missing values, but we can see all the ids now. Check out the new missing names for ids 6–8! This is the outer join in action.
These 4 joins make up the basics of table merging. If nothing else, remember these four:
inner: keeps only rows whose keys appear in both tables
left/right: keeps only keys that appear in the left (1st) or right (2nd) table
outer: keeps all the keys from both tables
The above joins will get you very far with your data analytics needs, but I want to introduce you to some lesser-known, but still useful joins.
Let’s say you want to look at people’s names for whom we have earnings data, but you don’t actually want to have all the columns from the second table. That’s what semi-join does. It gives you the same rows as an inner join, but doesn’t add any columns from the 2nd table:
3×2 DataFrame Row │ id     name     │ Int64  String─────┼───────────────   1 │     3  Claire   2 │     4  Daniel   3 │     5  Edward
I hear you, this is useful but not that crazy. Check out a cross-join:
30×4 DataFrame Row │ id     name    id_1   salary     │ Int64  String  Int64  Int64─────┼──────────────────────────────   1 │     1  Alice       3    4078   2 │     1  Alice       4    2395   3 │     1  Alice       5    3662   4 │     1  Alice       6    2202   5 │     1  Alice       7    2564   6 │     1  Alice       8    4545   7 │     2  Bob         3    4078   8 │     2  Bob         4    2395   9 │     2  Bob         5    3662  10 │     2  Bob         6    2202  11 │     2  Bob         7    2564  ⋮  │   ⋮      ⋮       ⋮      ⋮  21 │     4  Daniel      5    3662  22 │     4  Daniel      6    2202  23 │     4  Daniel      7    2564  24 │     4  Daniel      8    4545  25 │     5  Edward      3    4078  26 │     5  Edward      4    2395  27 │     5  Edward      5    3662  28 │     5  Edward      6    2202  29 │     5  Edward      7    2564  30 │     5  Edward      8    4545                      9 rows omitted
Wow, that blew up quickly 💥! A cross-join takes all the rows from table A and for each row, it matches it to every single row of table B. At first impression, this might not make any sense, but it’s a great method for finding all combinations of 2 tables. Our new table has 30 rows, which is 5 (rows of table A) x 6 (rows of table B).
To convince you that this is indeed useful. Let’s say you want to design a new Twix by changing the ingredients. To understand profitability, you also need to figure out the total costs of the chocolate:
Now you believe me? cross joins make chocolate innovation easier, so they are useful!
Now that we have better chocolate, let’s learn how to join on multiple columns. Extending the above joins to work with 2 keys is very easy. In fact, all you have to do is pass a vector of symbols or strings to the on argument of the join functions. To demonstrate this, let’s copy and add another column to both of our datasets. This will contain city names where the users live.
julia> display(C_names)5×3 DataFrame Row │ id     name    city     │ Int64  String  String─────┼─────────────────────────   1 │     1  Alice   New York   2 │     2  Bob     London   3 │     3  Claire  London   4 │     4  Daniel  New York   5 │     5  Edward  Londonjulia> display(D_earnings)6×3 DataFrame Row │ id     salary  city     │ Int64  Int64   String─────┼─────────────────────────   1 │     3    4078  New York   2 │     4    2395  New York   3 │     5    3662  New York   4 │     6    2202  London   5 │     7    2564  New York   6 │     8    4545  New York
One way you can think of this is that we have 2 separate databases. One in New York and one in London. With the systems not knowing about each other, they keep track of the users’ ids separately. So the name of user 1 in London is not the same as user 1 in New York. Indeed they are different users! So when we merge the 2 tables we want to make sure that the names and earnings are not only matched in user ids but also on the database name. Let’s do some joining on both columns then:
1×4 DataFrame Row │ id     name    city      salary     │ Int64  String  String    Int64─────┼─────────────────────────────────   1 │     4  Daniel  New York    2395
As we can see, only Daniel appears in both datasets and lives in the same place.
One problem that you might face is that your key columns don’t have the same names across your DataFrames. If that’s the case then you have 2 options:
You can rename the columns using rename!
or you can pass a mapping of names as the on parameter
Now what we want to do is to join the new earnings table with the names but we want to use another_id in the new table as our joining key.
Here we tell Julia to map => the id column to the another_id column when doing the joins. This same exact format is used when you need to rename columns.
After reading this article, you should be armed with the following join functions:
inner: for looking at overlaps
outer: for keeping everything
left/right: for keeping all rows from one table
semi: for keeping only columns from the first table, while doing an inner join
cross: for creating the product of the 2 tables — i.e. having every row matched up with every row.
You also know how to do join on multiple keys or different key names. Thanks for reading!