Apache Spark Internals: Tips and Optimizations
By: Javier Ramos
Tags: Hadoop, Spark, Data Engineering, Big Data, Data Science

Introduction

The goal of this post is to dig a bit deeper into the internals of Apache Spark to get a better understanding of how Spark works under the hood, so we can write optimal code that maximizes parallelism and minimized data shuffles.

This is an extract from my previous article which I recommend reading after this one. I assume you already have basic knowledge of Spark.

Spark Optimizations

In Spark you write code that transform the data, this code is lazy evaluated and, under the hood, is converted to a query plan which gets materialized when you call an action such as collect() or write(). Spark divides the data into partitions which are handle by executors, each one will handle a set of partitions. Operations that are executed within a single partition are called narrow operations and include functions such map() or filter(). In the other hand, aggregations are wide operations that require moving the data across nodes which is very expensive. The query plan itself can be of two major types: a logical plan and a physical plan which we will discuss later.

Remember, the main rule regarding Spark performance is: Minimize Data Shuffles. This is caused by wide operations in Spark such joins or aggregations which are very expensive because of the data shuffle.

Wide vs Narrow Dependencies. Source: https://medium.com/@dvcanton/wide-and-narrow-dependencies-in-apache-spark-21acf2faf031

Always try to reduce the number of data shuffles, actually, the goal of most of the optimizations that we are going to talk, is to try to do this: reduce the amount of data sent over the network.

Let’s now review some of the optimizations…

Filter Early

This is really simple but very important. Filter the data as early as possible so you don’t process data that will be discarded later on. The more you can push filters down to the source, and particularly where you can limit queries by partition key, the better. When possible, specify all the components of the partition key in the filter statement. Also, if you are going to use a set of data from disk more than once, make sure to use cache() to keep it in Spark memory rather than reading from disk each time.

A good rule of thumb is to use the coalesce() method after filtering data to reduce the number of partitions.

Spark Partitions and Spark Joins

This is critical in Spark, I really recommend this article where it explains the different optimizations in detail.

Set the right number of partitions

You need to set the right number of partitions to maximize parallelism in your cluster. If you have too few partitions then you will not be utilizing all cores available in the cluster and wasting resources. Having too many partitions will cause an excessive overhead in managing many small tasks as well as data movement reducing the performance.

The general recommendation for Spark is to have 4x of partitions to the number of cores in cluster available for application, but of course, this will depend on your use case. When in doubt, it’s a good idea to be wrong on the side of a larger number of tasks. This needs to align with the number of executors and the memory per executor which we will review later.

Avoid Data Skew

In general, you want your data to be evenly distributed across all the nodes, this is specially important when performing joins. Your join keys should have a even distribution to avoid data skews. Skewed data leads to performance degradation of parallel processing or even OOM(out of memory) crashes and should be avoided.

You can diagnose data skewness of your data by looking at the Spark UI and checking the time spent on each task. If one or more tasks are taking longer than the others, you have unbalanced partitions.

Some solutions may are:

Repartition data on a more evenly distributed key. This is the preferred option, use a key that naturally distributes the data across partitions before performing any wide operation.

on a more evenly distributed key. This is the preferred option, use a across partitions before performing any wide operation. Use salting which synthesizing an additional random key for better distribution of the data.

which synthesizing an additional random key for better distribution of the data. Split data into skewed and non-skewed data and work with them in parallel by redistributing skewed data.

and work with them in parallel by redistributing skewed data. If one of the data sets to join is small, like a fact table, use broadcast variables which we will discuss later on. This is useful to do lookups on fact tables.

which we will discuss later on. This is useful to do lookups on fact tables. Use broadcast joins when joining two data sets and one is quite small, this has the same benefits as broadcast variables. A more advanced feature is iterative broadcast joins where we split the data and do many small broadcast joins instead of a bigger one.

Let’s have a look to broadcast joins a it more in details…

Spark Joins

By default, Spark user Sort Merge Join which works great for large data sets.

Sort Merge Join. Source: https://towardsdatascience.com/the-art-of-joining-in-spark-dcbd33d693c

The idea is to sort the partitions before the join to reduce the amount of data shuffle, however sorting is itself an expensive operation, the performance of this join will change greatly depending on the source data on both sides of the join, if the data is already shuffled it is very fast, if not, Spark will need to perform an exchange and sort, which causes a data shuffle.

Sort merge join when exchange is required. source: https://towardsdatascience.com/should-i-repartition-836f7842298c

This Join technique works great when both data sets are large but when you join a table with a small fact table, then the advantages are lost. In this case, and specially if you have a cluster which has enough RAM available, you can use broadcast joins.

Broadcast Joins

In the case of broadcast joins, Spark will send a copy of the data to each executor and will be kept in memory, this can increase performance by 70% and in some cases even more. The concept of broadcast joins is similar to broadcast variables which we will discuss later, however broadcast joins are handled automatically by Spark, all you need to do is tell Spark which table you want to broadcast:

df = fact_table.join(broadcast(dimension_table), fact_table.col("dimension_id") === dimension_table.col("id"))

Note that when using broadcast joins the data is shared between the cores, but each executor will have its own copy, hence balancing cores and executors is important and we will discuss this later.

For more information about Spark joins check this article.

Repartition before expensive or multiple joins

The repartition() method allows us to change the distribution of the data on the cluster. This will cause data shuffle which is expensive but it will improve performance if done before certain types of joins or before multiple joins, as long as you specify the column name. As we seen before, Spark needs to be aware of the data distribution to make use of it before the sorted merge join. This can be done using the bucketing which is storing the data in a pre-shuffled and possibly pre-sorted state where the information about bucketing is stored in the metastore. But even if your data is sorted and already shuffled on disk for one of the tables, Spark will not know about it and still do a re sorting and full shuffle of both tables. This is why repartition your data to match the partitions of one side of the join is needed and can reduce data shuffle so only one side(table) of the join is distributed over the network. The idea is that by specifying the column, Spark under the hood adds metadata to the logical plan so it knows that it does not need to move the data. For example, if one table is partition by ID into 100 partitions. By re partitioning the data you avoid data shuffle:

df.repartition(100, "id").join(..., "id")

The goal is to have the same number of partitions on both sides of the join to avoid data exchanges. Data shuffle will also occur if the number of partitions differ from this property:

spark.sql.shuffle.partitions // default 200

which controls the number of partitions during the shuffle, and used by the sort merge join to repartition and sort the data before the join. I really recommend reading this article which goes more into details on how joins and data partition work.

A good rule of thumb is to try to re partition before multiple joins or very expensive joins, to avoid sort merge join re shuffling the data over and over again.

If you need to decrease the number of partitions, use coalesce instead of repartition() method, because it minimizes data shuffles and doesn’t trigger a data exchange. After a filter remember to use coalesce(), because you will have less data and this is much more efficient than repartition() since it minimize data shuffle. Also, always persist or cache your data after re partition to minimize data shuffle. But be remember that repartition is itself an expensive operation, it moves the data all over the cluster, so try to use it just once and only when completely necessary; and always remember to do narrow operations first.

It is important to remember that some wide operations like group by, change the number of partitions.

Repartition before writing to storage

It is always a good idea to write your data in a way that is optimized to read. In HDFS you want to use a columnar format such Parquet to increase performance of read operations when performing column based operations. I talked about this in this article. The idea is that if you create partitions using a the right folder structure, then the read operations do not need to scan all the data in disk but just the folders/partitions specified.

You can partition your data on write using:

df.write.partitionBy('key').json('/path...')

By storing data in different folders, query engines can skip large amounts of data scan from disk, increasing the read performance. For example, in Spark SQL, if you partition by date, if you specify a date in the where clause, then only one folder is read, instead of the whole data set, so the query will take less than a second instead of minutes.

If the folder structure is like this:

/mydata/2020-02-02/...

/mydata/2020-02-03/...

/mydata/2020-02-04/...

/mydata/2020-02-05/...

Then, by running this:

SELECT * from mydata WHERE date = '2020-02-03'

Only the data for 2020–02–03 is read instead of the whole data set.

Avoid disk spills

When joining large data set, Spark needs to store intermediate data during the data shuffle, if the executor does not have enough memory, it will move it to the disk and then join will become extremely slow, make sure you set the right amount of memory(spark.executor.memory) per executor and reduce your data size. You can also change the size of the buffer by setting: spark.shuffle.file.buffer

Set the right number of executors, cores and memory

It is important to set the number of executors according to the number of partitions. Your aim is too maximize parallelism and make sure your Spark executors are busy throughout the duration of the Job and all cores are used in the nodes.

Remember that each executor handles a sub set of the data, that is, a set of partitions. Also, each executor used 1 or more cores as set with the property:

spark.executor.cores

When running with YARN is set to 1.

In Spark, we achieve parallelism by splitting the data into partitions which are the way Spark divides the data. The partitions are spread over the different nodes and each node have a set of executors. Then each executor manages one or more partitions. Note, that at the computational level, each executor performs a set of tasks so each the executor will apply a set of functions to the same partition. So, this is why you may want to have more than one core per executor, so you can run independent task in parallel. It is important to understand that each executor will have its own local and independent data in memory which includes broadcast variables(which will discuss later) and accumulators, both of these use quite a bit of memory; however, these are shared between cores. So, having just one core per executor will mean that all the data needs to replicated for each executor. Also, I/O operations, specially on splitable file formats, can take advantage of multiple cores when reading or writing a partition, maximizing throughput.

In summary, when you use broadcast variables, accumulators or you read/write a lot of data, you want to have more than one core per executor, but not too many, otherwise some of the cores will not be used. For more information check this article.

As a rule of thumb 3–5 cores per executor is a good choice. Of course, this dependents on the number of partitions and the size of the partitions. If you have very small partitions and you don’t use much memory like broadcast variables, then less cores is recommended.

Your goal is to find the balance between the number of executors and cores, plus the right amount of memory for each executor. You can set executor memory, and number of executors and cores when you run spark-submit.

# Run on a Spark standalone cluster in cluster deploy mode with supervise

./bin/spark-submit \

--class org.apache.spark.examples.SparkPi \

--master spark://207.184.161.138:7077 \

--deploy-mode cluster \

--supervise \

--executor-memory 20G \

--total-executor-cores 100 \

/path/to/examples.jar \

1000

You can also pass them as Spark properties.

Setting the right amount of memory per executor is also important, this needs to be based on your use of accumulator, broadcast variables and the size of your data when doing joins and the data is shuffle as we seen before. Make sure you set alerts so you are aware of data spills which are very inefficient.

Broadcast Variables

We already talked about broadcast joins which are extremely useful and quite popular, because it is common to join small data sets with big data sets, for example; when you join your table with a small fact table uploaded from a CSV file.

In this case, the idea is to copy the data to each executor so no data needs to be moved, and the join is done locally because one side of the join is stored on each node completely in memory. This is the idea of broadcasting in Spark, both for joins and for variables.

Joins are managed by Spark for you under the hood, so they are easy to use. Broadcast Variables need to be created in your code. You can store any JVM object as long as it is serializable. We will talk about serialization in the next section.

To create a broadcast variable:

val df = spark.sparkContext.broadcast(data)

Note that the data cannot be a data frame or data set, it needs to be a regular object, so you need to call the collect() method to get all the data before sending it to the executors.

Spark Serialization

There are several articles and books that teach you how to optimize your Spark code, however, the single most efficient thing you can do to increase Spark performance across all the code is to get rid of the the Java Serialization.

Java Serialization is very inefficient and also insecure; and can really slow down your jobs.

You need to understand how Spark runs the applications. In a nutshell, the driver needs to serialize your code and send it to all the nodes, so broadcast variables and the job itself needs to be transferred over the network, besides that, intermediate data and metadata needs to be serialized as well. This happens very often under the hood and it can be a bottle neck for your application.

If you have run into the famous “Task not serializable” error in Spark, then you know what I’m talking about. Understanding which parts of your code needs to be serialize and which ones don’t is critical. Your goal is to identify these objects and optimize them by using another serializable format. To solve serialization issues, I really suggest having a look to this article. In general, ensure all the objects passed to closure are serializable.

The most famous Spark alternative to Java serialization is Kryo Serialization which can increase the Serialization performance by several order of magnitude.

Kryo Performance vs Java, Source: https://docs.mulesoft.com/mule-runtime/3.9/improving-performance-with-the-kryo-serializer

Many Spark developers choose not to use it because you need to register every class you use using:

Kryo kryo = new Kryo();

kryo.register(SomeClass.class);

But I really recommend you enable Kryo from the beginning of your project to take advantage of the performance boost. Note that if you rely on the DataSet API, then you may not need Kryo since you classes will use Tungsten encoders which are even more efficient than Kryo and we will discuss them later.

Spark Catalyst

I left the best optimization to the end. Spark has a secret weapon that increases your job efficiently tremendously and the best part is that you “almost” don’t have to do anything to use it, it runs under the hood. We already touch upon this feature before: Spark Catalyst Engine. It basically rewrites your code in an optimal way.

Catalyst is available on the Data Frame API and partially in the Data Sets API. It is not available in the RDD API. It is part of the Spark SQL layer and the idea behind it is to use all the optimizations done over many years in the RDBMS world and bring them to Spark SQL. Since SQL provides a know mathematical model, Spark Catalyst can understand the data, make assumptions and optimize the code. Under the hood, Spark runs a complicated workflow which completely rewrites your code into a harder to understand but much more efficient one.

Think of the Spark Data Frame API as a declarative language, not real code. You are kind of writing on top this engine that will interpret your code and optimize it. This is the process that happens when you trigger an action in Spark:

Catalyst workflow

In a nutshell, it creates an optimized logical plan which them is divided in multiple physical plans. Then, it generates the code for the physical plans and it distributes the code to the executors (Remember why serializing was important?). Catalyst generates an optimized physical query plan from the logical query plan by applying a series of transformations like predicate push-down, column pruning, and constant folding on the logical plan.

Catalyst Optimizer has two types of optimizations:

Cost-Based Optimizer : Since Data Frames are based in SQL, Catalyst can calculate the cost of each path and analyzes which path is cheaper, and then executes that path to improve the query execution.

: Since Data Frames are based in SQL, Catalyst can calculate the cost of each path and analyzes which path is cheaper, and then executes that path to improve the query execution. Rule-Based optimizer: These include constant folding, predicate push-down, projection pruning, null propagation, Boolean simplification, and other rules.

Catalyst will also auto perform broadcast joins when one side of the join is small, the threshold can be set using this property:

spark.sql.autoBroadcastJoinThreshold

Project Tungsten

Data Frames and Data Set APIs also benefit for Project Tungsten which aims to fix the serialization issues that we mentioned before, memory management and performance. It uses an optimized in memory format and off-heap memory which does not need to be garbage collected.

It optimizes Spark jobs for CPU and memory efficiency by doing the following:

Off-Heap Memory Management using binary in-memory data representation, this is the Tungsten row format. Cache Locality which is about cache-aware computations with cache-aware layout for high cache hit rates. Whole-Stage Code Generation (CodeGen).

This is why you need to use encoders when using Data Set API, these are in charge of the off heap optimizations. You get this for free in the Data Frames and Data Set APIs but Datasets are better optimize. For more information check this great article.

What about RDDs? Catalyst cannot do anything about it, this is the do it your self approach. Actually Catalyst generates RDD code, this is the end result. Also, you need to manage serialization.

Catalyst cannot do anything about it, this is the do it your self approach. Actually Catalyst generates RDD code, this is the end result. Also, you need to manage serialization. Does Catalyst Optimizes DataSet API? Yes and No. Initially, the Data Set API was slower since Catalyst “couldn’t see” what you were doing in the code since data sets syntax is broader than the fixed SQL syntax. But if you use the latest version of Spark (3.0), then most of the optimizations are available. Still, Data Frames API will be a bit more efficient in optimizing your queries; with DataSets you need to be more careful, but if you write good Spark code, the differences are minimal. Plus, Datasets have better space and speed performance, data sets Tungsten optimization is better than data frames. My recommendation is that Spark SQL with Data Frames should be used by data scientists which no deep knowledge of Spark, Catalyst will optimize the code for them. Data Set API is ideal for Scala engineers with good knowledge of Spark internals.

Spark APIs comparison. Source: https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html

Do I need Kryo if I use the Data Set API? Yes and No. Tungsten will manage your case classes using encoders, but there are still other parts of the code that needs to be Serialized and enabling Kryo would help. If you use the data set API, I recommend to skip Kryo unless you need to maximize performance.

Other Spark Optimizations

Use reduceBy/aggregateBy methods instead of groupBy when possible, since they reduce data shuffle. It works faster because Spark knows about the combined output with a common key on each partition before shuffling the data.

methods instead of when possible, since they reduce data shuffle. It works faster because Spark knows about the combined output with a common key on each partition before shuffling the data. Avoid reduceByKey when the input and output value types are different, use use aggregateByKey instead; if the types are the same use reduceByKey if not aggregateByKey .

when the input and output value types are different, use use instead; . Use localCheckpoint () method before repartition and write. This is to create stage barrier between shuffle stage and the write operation.

() method before repartition and write. This is to create stage barrier between shuffle stage and the write operation. Don’t use collect () method. It will try to move all data in to the driver and where it may run out of memory and crash. Only use it for debug purposes or to create broadcast variables which are small in size by definition.

() method. It will try to move all data in to the driver and where it may run out of memory and crash. Only use it for debug purposes or to create broadcast variables which are small in size by definition. Use the right file format . Avro for row level operations and Parquet or ORC for column based operations such Spark SQL. Compress the data and make sure that the format is splitable, so it can be read in parallel by Spark readers. Splittable formats are lso, bzip2, snappy , etc. Non-splittable gzip, zip, lz4…

. such Spark SQL. Compress the data and make sure that the format is splitable, so it can be read in parallel by Spark readers. , etc. Non-splittable gzip, zip, lz4… Monitoring and observability is critical is Spark. Use explain() method before writing to disk as last step to verify that all the optimizations that we talked about take place. Check that the expected Catalyst optimizations are done and filters are pushed to the source, queries are re arranged, data shuffles are minimized, etc. Enable the Spark history server to analyze and observe the evolution of your jobs and detect any change that may cause performance issues.

As you can see, the goal of all these optimizations is to reduce data movement. To do this, always filter as much data as you can. Try to minimize wide operations. Cache the data sets if they are going to be used multiple times. Make sure your data is partitioned evenly before performing joins or writing to disk. Be aware of data locality and remember to use Spark UI to and the explain() method to understand Spark logical and physical plan.

Conclusion

In this article, we unfolded the internals of Spark to be able to understand how it works and how to optimize it. Regarding Spark, we can summarize what we learned with two key takeaways:

Minimize data shuffles and maximize data locality.

Use Data Frames or Data Sets high level APIs to take advantages of the Spark optimizations.

If you are a Scala developer, remember that Spark high order functions are not the same ones as in Scala and you really need to understand how Spark and the Catalyst engine work.

Good luck on your Big Data Journey!

Remember to clap if you enjoyed this article and follow me for more updates!