Handling Data Skew in Apache Spark
By: Dima Statz
Tags: Broadcast Join, Aws Emr, Skewness, Apache Spark, Key Salting

Even Distribution vs Distribution With Skew

Introduction

One of the well-known problems in parallel computational systems is data skewness. Usually, in Apache Spark, data skewness is caused by transformations that change data partitioning like join, groupBy, and orderBy. For example, joining on a key that is not evenly distributed across the cluster, causing some partitions to be very large and not allowing Spark to process data in parallel. Since this is a well-known problem, there is a bunch of available solutions for it. In this article, I will share my experience of handling data skewness in Apache Spark.

Input

In this specific example, we will talk about an hourly batch processing of CDN access logs. Different CDNs produce log files with different formats and sizes. In order to simplify the processing, we are running a preprocessor task that creates parquet formatted files with equal sizes around 30MB. All files are saved in AWS S3. The batch sizes vary from 50GB to 1.5TB depending on traffic volumes on CDNs. For all examples in this article, I used events batch that consists of 3570 files (356945717 rows) with a total size of 107.4GB size.

Goal

One of the steps in our data pipeline is data enrichment. The idea is to enrich each log line with different types of metadata like GeoData, UserAgent, OwnerMetadata, etc. For example, if log lines have owner_id field and OwnerMetadata has owner_id, owner_name, we would like to add an owner_name column to log lines. I guess data enrichment is very common to many data pipelines. And probably the code looks like that

val df = spark.read.parquet(“s3://…”)

val geoDataDf = spark.read.parquet(“s3://…”)

val userAgentDf = spark.read.parquet(“s3://…”)

val ownerMetadataDf = spark.read.parquet(“s3://…”) df

.join(geoDataDf, exprGeo, “left”)

.join(userAgentDf, exprUserAgent, “left”)

.join(ownerMetadataDf, exprOwnerMetadata, “left”)

.write

.parquet(“s3://…”)

Join Types

Apache Spark has 3 different join types: Broadcast joins, Sort Merge joins and Shuffle Joins. Starting from Apache Spark 2.3 Sort Merge and Broadcast joins are most commonly used, and thus I will focus on those two. You can find more information about Shuffle joins here and here.

The key idea of the Sort Merge algorithm is to first sort the relations by the join keys so that interleaved linear scans will encounter these sets at the same time and then merge so that each executor will hold a particular set of joining keys. As you can imagine this kind of strategy can be expensive: nodes need to use the network to share data.

In Broadcast joins, Spark sends an entire copy of a lookup table to each executor. Clearly, in this method, each executor is self-sufficient in performing join operations. The network is used only for broadcast.

Sort Merge joins

So we have an events data frame created by reading 3570 AWS S3 object with the total size of 107GB. The created data frame has 356945717 rows. We are performing join operations with the metadata table that has ±1100000 rows in order to enrich events with additional information. After running the job and looking at Spark UI we can see the following picture:

After 12 min after the job starts:

After 21 min after the job starts:

And finally, it takes 1.2h to complete

Clearly, we can see one long-running task with a massive shuffle read. After taking a closer look at this long-running task, we can see that it processed almost 50% of the input(see Shuffle Read Records column).

The question is why the processing of the given job wasn’t properly parallelized. In Apache Spark, Sort Merge Joins send all records with the same join key to the same partition, and it looks like almost 50% of our input rows have the same value in the join column. By running the following query on my input files, I can see that indeed this is the case and we have a classic data skewness problem.

spark.read.parquet(“s3://…”)

.select(“o_id”)

.groupBy(“o_id”)

.agg(count(“o_id”).as(“count_o_id”))

.sort(desc(“count_o_id”))

Broadcast joins

Obviously, broadcast joins look like a good approach to solve the data skewness problem. We have an events data frame that almost perfectly distributed across the Apache Spark cluster. We have metadata data frames that are much smaller than the events data frame. Intuitively we would like to avoid moving the events data frame which size is up to 1.5TB and instead send the entire copy of metadata tables to each Spark Executor. So, I am changing the code to

val df = spark.read.parquet(“s3://...”)

val geoDataDf = spark.read.parquet(“s3://...”)

val userAgentDf = spark.read.parquet(“s3://...”)

val ownerMetadataDf = spark.read.parquet(“s3://...”) df

.join(broadcast(geoDataDf), exprGeo, “left”)

.join(broadcast(userAgentDf), exprUserAgent, “left”)

.join(broadcast(ownerMetadataDf), exprOwnerMetadata, “left”)

.write

.parquet(“s3://...”)

Here I am using the broadcast keyword as a hint to Apache Spark to broadcast the right side of join operations. Note that Apache Spark automatically translates joins to broadcast joins when one of the data frames smaller than the value of spark.sql.autoBroadcastJoinThreshold. See Apache Spark documentation for more info. Now let’s run the code and analyze the job run by using Spark UI.

The job was completed just in 5.2 minutes, a tremendous improvement comparing to the previous run with Sort Merge join. Let’s dig deeper into the data distribution statistics

As you can see in the attached Tasks Metrics table, broadcast joins work just great. We started with evenly distributed data, and doing broadcast doesn’t change the partitioning. You can see in 25th, Median, and 75th percentiles that the data remains evenly distributed and there is 0 shuffling. And the performance is just great, data enrichment of 107GB parquet files is completed in 5.2 min including IO for write. Improving tun time from 1.2h to 5.2min is very impressive.

The only question is whether such an approach works great also for 500GB, 1TB, and 2TB of input data. What is the restriction on the size of broadcasted data frames? Unfortunately, broadcast joins suffer from the following problems and cannot be used always as a solution for Data Skewness:

Scalability. Broadcast joins has a hardcoded limit on the size of broadcasted tables — 8GB table size or 512000000 rows making this solution not scalable. See the source code here. High memory footprint. Broadcasted data should fully fit into the memory of both driver and executor. For example, when a job runs 100 executors and the broadcasted data frame is 1GB the price for using broadcast joins is an additional 100GB of RAM. Also, if your broadcast table tends to increase, you will see the following exception very often and you will need to adjust the Spark Executor’s and Driver’s memory size frequently.

java.lang.OutOfMemoryError: Not enough memory to build and broadcast the table to all worker nodes. As a workaround, you can either disable broadcast by setting spark.sql.autoBroadcastJoinThreshold to -1 or increase the spark driver memory by setting spark.driver.memory to a higher value at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec$$anonfun$relationFuture$1$$anonfun$apply$1.apply(BroadcastExchangeExec.scala:122)

3. Performance. Obviously, broadcasting operations are expensive because all executors need to receive a copy of the table. So, when the input size is big and a job needs a lot of executors, the broadcast time is increasing and may become higher than the join cost itself.

Key Salting

There is another well-known technique to overcome this problem — key salting. The idea is to invent a new key, which guarantees an even distribution of data. Bellow, you can see the possible ‘key salting’ implementation. Here we add one column with uniformly distributed values to the big data frame. And then we add one column to the small data frame, and finally we ‘explode’ the small data frame, meaning we create new rows for each ‘old_id’ and each n in [0, salt] range.

def saltedJoin(df: DataFrame, buildDf: DataFrame, joinExpression: Column, joinType: String, salt: Int): DataFrame = { import org.apache.spark.sql.functions._

val tmpDf = buildDf.withColumn(“slt_range”, array(Range(0, salt).toList.map(lit): _*))



val tableDf = tmpDf.withColumn(“slt_ratio_s”, explode(tmpDf(“slt_range”))).drop(“slt_range”) val streamDf = df.withColumn(“slt_ratio”, monotonically_increasing_id % salt)



val saltedExpr = streamDf(“slt_ratio”) === tableDf(“slt_ratio_s”)

&& joinExpression streamDf.join(tableDf, saltedExpr, joinType).drop(“slt_ratio_s”).drop(“slt_ratio”) }

Now we can join by using both new and old join columns. By making use of scala implicit classes we can rewrite our enrichment flow as follows

val df = spark.read.parquet(“s3://...”)

val geoDataDf = spark.read.parquet(“s3://...”)

val userAgentDf = spark.read.parquet(“s3://...”)

val ownerMetadataDf = spark.read.parquet(“s3://...”) df

.saltedJoin(geoDataDf, exprGeo, “left”, 200)

.saltedJoin(userAgentDf, exprUserAgent, “left”, 200)

.saltedJoin(ownerMetadataDf, exprOwnerMetadata, “left”, 200)

.write

.parquet(“s3://...”)

Now we can run and see the results

We can see a solid improvement from 1.2h to 26 min compared to the previous run using Sort Merge join. It is not so impressive though after we saw the Broadcast Joins in action, 5.2 min for doing exactly the same job. Let’s see how well the data is distributed

The distribution is pretty symmetric around the median value. Probably it can be improved by increasing the salt value and spark.sql.shuffle.partitions.

Summary

The handling of data skew depends on many parameters like data volume, data variety, cluster configuration, and processing goals. So there is no simple recipe for doing it. You have to examine your data, code, and decide what works best for you. But here are some facts that will help you to make the decision.