Spark Optimization Techniques — Broad Cast Join
By: Vamsi Krishna
Tags: Hadoop, Spark, Spark Optimization, Big Data, Broadcast Join

Why do we need Broad Cast Join ?

While joins are very common and powerful, they warrant special performance consideration as they may require large network transfers or even create datasets beyond our capability to handle.

As the saying goes, the cross product of big data and big data is an out-of-memory exception.

Another factor causing slow running spark jobs could be the join type. By default, Spark uses the SortMergejoin type which must first sort the left and right sides of data before merging them.

What is Broad Cast Join ?

broadcast join is also called as broadcast hash join (or) Map-side join.

When 1 of the data frame/RDD is small enough to fit in the memory, it is broadcasted over to all the executors where the larger data frame / RDD resides and a hash join is performed.

This has 2 phases:

broadcast-> the smaller data frame / RDD is broadcasted across the executors in the cluster where the larger data frame / RDD is located.

hash join-> A standard hash join is performed on each executor.

There is no shuffling involved in this and can be much quicker.

What are the important properties in Broadcast Join ?

spark.sql.broadcastTimeout:

This property controls how long executors will wait for broadcasted tables.

Default value: 300 seconds (5 minutes or 300000ms)

spark.sql.autoBroadcastJoinThreshold:

The high-level APIs can automatically convert join operations into broadcast joins. The property name where you can set the threshold value for Broadcast join is Spark.sql.autoBroadcastJoinThreshold. If the size of the table is smaller than the this property value then it will do broadcast always.

If table size is < 10 MB size → Broad cast join will be used

If table size is > 10 MB size → sort merge join will be used which is the default join in spark 2.3

Default value: 10485760 (10 MB)

if you set spark.sql.autoBroadcastJoinThreshold to -1 then it will disable the broadcast join

Broad Cast Join Hands-on:

let’s pretend that the captainsDF is huge and the citiesDF is tiny.

Let’s broadcast the citiesDF and join it with the captainsDF.

We can Force BroadcastHashJoin with broadcast hint (as function) as follows:

scala> val joinedDF = captainsDF.join(

| broadcast(citiesDF),

| captainsDF(“city”) === citiesDF(“city”)

| )

joinedDF: org.apache.spark.sql.DataFrame = [first_name: string, city: string … 2 more fields]

scala> joinedDF.explain()

== Physical Plan ==

*(2) BroadcastHashJoin [city#6], [city#22], Inner, BuildRight

:- *(2) Project [_1#2 AS first_name#5, _2#3 AS city#6]

: +- *(2) Filter isnotnull(_2#3)

: +- LocalTableScan [_1#2, _2#3]

+- BroadcastExchange HashedRelationBroadcastMode(List(input[0, string, true]))

+- *(1) Project [_1#19 AS city#22, _2#20 AS country#23]

+- *(1) Filter isnotnull(_1#19)

+- LocalTableScan [_1#19, _2#20]

Spark is smart enough to return the same physical plan, even when the broadcast() method isn’t used. (Look at the screen shot)

Important points about broadcast join:

* Spark broadcast joins are perfect for joining a large DataFrame with a small DataFrame.Broadcast joins cannot be used when joining two large DataFrames.

* If both of the tables are huge then using the broadcast hash join on them is not a good strategy.Because both huge tables cannot sit in a single machine & they have to be distributed into the partitions & put into multiple machines.

* If both the tables are < 10 MB, then the Smaller table should be last of the join and that will be broadcasted

it broad casts which ever table defined latter in the query and that table size should be in the limit of the configuration mentioned.

* In broad cast hash join, full copy of the small DF will be sent to each executor. If that small DF does not fit into the executor’s RAM, then it will lead to OOM errors.

If Broadcast Hash join throws OOM issues, we can Increase the storage memory of executors. You cannot increase specific ones, it’s Increasing the memory of all executors.

Connect with me on linkedin at https://www.linkedin.com/in/vamsi-krishna-0636441ba/