About Joins in Spark 3.0. Tips for efficient joins in Spark SQL. | by David Vrba | Towards Data Science

One of the very frequent transformations in Spark SQL is joining two DataFrames. The syntax for that is very simple, however, it may not be so clear what is happening under the hood and whether the execution is as efficient as it could be.
Spark provides a couple of algorithms for join execution and will choose one of them according to some internal logic. This choice may not be the best in all cases and having a proper understanding of the internal behavior may allow us to lead Spark towards better performance.
Spark 3.0 provides a flexible way to choose a specific algorithm using strategy hints:
dfA.join(dfB.hint(algorithm), join_condition)
and the value of the algorithm argument can be one of the following: broadcast, shuffle_hash, shuffle_merge. Before Spark 3.0 the only allowed hint was broadcast, which is equivalent to using the broadcast function:
dfA.join(broadcast(dfB), join_condition)
In this note, we will explain the major difference between these three algorithms to understand better for which situation they are suitable and we will share some related performance tips. Finally, we will show some benchmarks to compare the execution times for each of these algorithms.
Spark decides what algorithm will be used for joining the data in the phase of physical planning, where each node in the logical plan has to be converted to one or more operators in the physical plan using so-called strategies. The strategy responsible for planning the join is called JoinSelection. Among the most important variables that are used to make the choice belong:
the hint
the joining condition (whether or not it is equi-join)
the join type (inner, left, full outer, ...)
the estimated size of the data at the moment of the join
BroadcastHashJoin (we will refer to it as BHJ in the next text) is the preferred algorithm if one side of the join is small enough (in terms of bytes). In that case, the dataset can be broadcasted (send over) to each executor. This has the advantage that the other side of the join doesn’t require any shuffle and it will be beneficial especially if this other side is very large, so not doing the shuffle will bring notable speed-up as compared to other algorithms that would have to do the shuffle. Here you can see a physical plan for BHJ, it has to branches, where one of them (here it is the branch on the right) represents the broadcasted data:
Spark will choose this algorithm if one side of the join is smaller than the autoBroadcastJoinThreshold, which is 10MB as default. There are various ways how Spark will estimate the size of both sides of the join, depending on how we read the data, whether statistics are computed in the metastore and whether the cost-based optimization feature is turned on or off. We will cover the logic behind the size estimation and the cost-based optimizer in some future post. The default size of the threshold is rather conservative and can be increased by changing the internal configuration. For example, to increase it to 100MB, you can just call
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 100 * 1024 * 1024)
The optimal value will depend on the resources on your cluster. Broadcasting a big size can lead to OoM error or to a broadcast timeout. The timeout is related to another configuration that defines a time limit by which the data must be broadcasted and if it takes longer, it will fail with an error. The default value of this setting is 5 minutes and it can be changed as follows
spark.conf.set("spark.sql.broadcastTimeout", time_in_sec)
Besides the reason that the data might be large, there is also another reason why the broadcast may take too long. Imagine a situation like this
dfA = spark.table(...)dfB = (  data  .withColumn("x", udf_call())  .groupBy("id").sum("x"))dfA.join(dfB.hint("broadcast"), "id")
In this query we join two DataFrames, where the second dfB is a result of some expensive transformations, there is called a user-defined function (UDF) and then the data is aggregated. Suppose that we know that the output of the aggregation is very small because the cardinality of the id column is low. That means that after aggregation, it will be reduced a lot so we want to broadcast it in the join to avoid shuffling the data. The problem however is that the UDF (or any other transformation before the actual aggregation) takes to long to compute so the query will fail due to the broadcast timeout. Besides increasing the timeout, another possible solution for going around this problem and still leveraging the efficient join algorithm is to use caching. If we change the query as follows
dfA = spark.table(...)dfB = (  data  .withColumn("x", udf_call())  .groupBy("id").sum("x")).cache()dfB.count()dfA.join(dfB.hint("broadcast"), "id")
the query will be executed in three jobs. The first job will be triggered by the count action and it will compute the aggregation and store the result in memory (in the caching layer). The second job will be responsible for broadcasting this result to each executor and this time it will not fail on the timeout because the data will be already computed and taken from the memory so it will run fast. Finally, the last job will do the actual join.
SortMergeJoin (we will refer to it as SMJ in the next) is the most frequently used algorithm in Spark SQL. If neither of the DataFrames can be broadcasted, Spark will plan the join with SMJ if there is an equi-condition and the joining keys are sortable (which is the case in most standard situations).
SMJ requires both sides of the join to have correct partitioning and order and in the general case this will be ensured by shuffle and sort in both branches of the join, so the typical physical plan looks like this
As you can see there is an Exchange and Sort operator in each branch of the plan and they make sure that the data is partitioned and sorted correctly to do the final merge. To understand the logic behind this Exchange and Sort, see my previous article where I explain why and how are these operators added to the plan. The shuffle and sort are very expensive operations and in principle, they can be avoided by creating the DataFrames from correctly bucketed tables, which would make the join execution more efficient.
Another joining algorithm provided by Spark is ShuffledHashJoin (SHJ in the next text). If you don’t call it by a hint, you will not see it very often in the query plan. The reason behind that is an internal configuration setting spark.sql.join.preferSortMergeJoin which is set to True as default. In other words, whenever Spark can choose between SMJ and SHJ it will prefer SMJ. The reason why is SMJ preferred by default is that it is more robust with respect to OoM errors. In the case of SHJ, if one partition doesn’t fit in memory, the job will fail, however, in the case of SMJ, Spark will just spill data on disk, which will slow down the execution but it will keep running.
If you switch the preferSortMergeJoin setting to False, it will choose the SHJ only if one side of the join is at least three times smaller then the other side and if the average size of each partition is smaller than the autoBroadcastJoinThreshold (used also for BHJ). This is to avoid the OoM error, which can however still occur because it checks only the average size, so if the data is highly skewed and one partition is very large, so it doesn’t fit in memory, it can still fail.
Similarly to SMJ, SHJ also requires the data to be partitioned correctly so in general it will introduce a shuffle in both branches of the join. However, as opposed to SMJ, it doesn’t require the data to be sorted, which is actually also a quite expensive operation and because of that, it has the potential to be faster than SMJ.
The situation in which SHJ can be really faster than SMJ is when one side of the join is much smaller than the other (it doesn’t have to be tiny as in case of BHJ) because in this case, the difference between sorting both sides (SMJ) and building a hash map (SHJ) will manifest. Here you can see the physical plan for SHJ:
All the previous three algorithms require an equi-condition in the join. If there is no equi-condition, Spark has to use BroadcastNestedLoopJoin (BNLJ) or cartesian product (CPJ). BNLJ will be chosen if one side can be broadcasted similarly as in the case of BHJ. Both BNLJ and CPJ are rather slow algorithms and are encouraged to be avoided by providing an equi-condition if it is possible.
Let’s compare the execution time for the three algorithms that can be used for the equi-joins. In this benchmark we will simply join two DataFrames with the following data size and cluster configuration:
Data:  size(dfA): 194 GB  size(dfB): 816 MBCluster:  driver: m4.xlarge  executor: 4 x c5.4xlarge (together 64 cores)Platform: Databricks (runtime 7.0 with Spark 3.0.0)
To run the query for each of the algorithms we use the noop datasource, which is a new feature in Spark 3.0, that allows running the job without doing the actual write, so the execution time accounts for reading the data (which is in parquet format) and execution of the join. The used PySpark code is bellow and the execution times are in the chart (the vertical axis shows execution time, so the smaller bar the faster execution):
dfA = spark.read.parquet(...)dfB = spark.read.parquet(...)(  dfA.join(dfB.hint(algorithm), "user_id")  .write  .mode("overwrite")  .format("noop")  .save())
It is also good to know that SMJ and BNLJ support all join types, on the other hand, BHJ and SHJ are more limited in this regard because they do not support the full outer join.
Using the hints in Spark SQL gives us the power to affect the physical plan. As I already noted in one of my previous articles, with power comes also responsibility. When we decide to use the hints we are making Spark to do something it wouldn’t do otherwise so we need to be extra careful. What can go wrong here is that the query can fail due to the lack of memory in case of broadcasting large data or building a hash map for a big partition. On the other hand, if we don’t use the hint, we may miss an opportunity for efficient execution because Spark may not have so precise statistical information about the data as we have. This is also related to the cost-based optimizer — how it handles the statistics and whether it is even turned on in the first place (by default it is still off in Spark 3.0 and we will describe the logic related to it in some future post). Also if we don’t use the hint, we will barely see the ShuffledHashJoin because the SortMergeJoin will be almost always preferred even though it will provide slower execution in many cases.
We have seen that in the case when one side of the join is very small we can speed it up with the broadcast hint significantly and there are some configuration settings that can be used along the way to tweak it. If one side of the join is not very small but is still much smaller than the other side and the size of the partitions is reasonable (we do not face data skew) the shuffle_hash hint can provide nice speed-up as compared to SMJ that would take place otherwise.
I also need to mention that using the hints may not be that convenient in production pipelines where the data size grows in time. Using the hint is based on having some statistical information about the data that Spark doesn’t have (or is not able to use efficiently), but if the properties of the data are changing in time, it may not be that useful anymore.