Joins with Apache Spark
By: Emml Asimadi
Tags: Sql Joins, Spark, Apache Spark

RE: Data, Joins are everywhere it is what enables us join data sources ( or tables) to form a richer and more complete view of our data, based on keys they have in common.

In this blog I review JOINs as used by the defacto* Big Data framework Apache Spark. More than any other system, Apache Spark’s unified approach to data processing allows us to work with a large variety of data sources/file types (text, csv, json, parquet, avro… s3, relational DB, NoSQL DBs etc) all in the same environment, making joins inevitable albeit COSTLY.

Using Apache Spark’s DataFrame API (same is possible in SparkSQL, Dataset APIs) we illustrate some interesting characteristics of joins in Spark. In this first blog we keep things at a high-level without worrying about Spark’s behavior under the hood or join performance.

Below is the general syntax for joins in Spark.

Spark JOIN Syntax

In the above syntax, just like what we may be familiar with in SQL world, we specify the left table e.g dfLeft , the right table e.g dfright , the joinCondition which specifies which keys should match for a successful join and joinType (inner, outer etc…) which specifies which rows/data should be returned in the results.

The joins we are familiar with from the relational world (as captured in the image above) are known as equi-joins (wiki). This type of join specifies the join condition only with equality comparator e.g dfLeft.key == dfRight.key . Spark however enables joins beyond this 😲…. by allowing the joinCondition to be an arbitrary Boolean expression not just equality as in the case of equi-joins. So we can do more fancy stuff like join two tables if keyA is in a certain range of keyB keyA == keyB +/- 5 or if keyA is contained in the array KeyB 😲 keyB.contains(keyA) or an even more complex join could be, join two tables base on text fields, if they are within a certain edit distance of each other… so this join will succeed even when there are minor typos in the key field edit_distance(keyA,keyB) <= 5 . [NB: the syntax for conditions used in this paragraph are not formal, they just to illustrate in a readable way]

Apache Spark lists the following type of joins inner , outer , left_outer , right_outer , left_semi , left_anti , natural_join , cross or cartesian joins. Most other joins are possible by adding where/filter conditions on the core joins.

Understanding by Doing

Now true to my minimalist yet effective approach 😊….let’s try this seemingly trivial example for illustrating joins in Spark. Take note of the two tables below and try to guess the output of each joinType (inner, outer, left_outer, right_outer …etc). The two tables have only ID =1 in common.

Tables dfLeft and dfRight to illustrate joins.

Inner Join

keep only records whose key exists in both tables. This the default joinType.

Output (you guessed right 😊)

Output of inner Join

Outer Join

retain records from both tables, inserting null in either table when there is no matching record.

Output

Output of outer Join

Left Outer Join

retain all records on the left side, keep null in right side where there are no match keys.

Output

Right Outer Join

You can guess :)

Left Semi Join

Keep only the left table (don’t show the right table), where the joinCondition is met. This is a useful filtering mechanism.

Output

Output of left_semi Join

Left Anti Join

This is the complement of the left_semi join, in this case you return only the left table where the joinCondition is NOT met.

Output

Output of left_anti Join

Cross (Cartesian) Join

This join can easily result in large output so use cautiously. Join each row on the left with every row on the right😲. So for our three rows tables, the resulting joined table ha 3x3=9 rows.

Output

Output of crossJoin

Using Arbitrary Join Conditions

This is what makes joins in Spark even more powerful — the ability to specify sophisticated joinConditions not just equality.

In below example we want to join the two tables if and only if their labels have one or more characters in common😲 (put in another way - if they have at least one character in common). To do this we define a joinCondition that does just that — takes in the two columns and checks whether their intersect is more than or equal to one character in length. 😊

Output (can you believe it 😲)

Can you find the atleast one character that is causing leftLabel to join to rightLabel ?? well done to you!!!

Output of a inner Join on a custom condition.

Conclusion

Hope this lived up to the expectation of a simple yet effective demonstration of Spark’s Join capability. As you may already suspect, joining in Spark or any other distributed system for that matter may not always be trivial because of the distributed nature of data in such systems. Thus there are a few things to be aware of in terms of performance. We discuss that later.

Feel free to comment, share, suggest or question.! etc….