10 Ways to Improve Spark Performance
By: Vishal Mishra
Tags: Hadoop, Performance Tuning, Spark, Big Data Engineering, Big Data

As Apache Spark works in a distributed manner, it needs a lot of resources like memory and cores (CPU) to complete a job, actually, that depends on the input data and some other factors as well. So, tuning the spark job is not that easy. Sometimes you don't know that your spark job is utilizing the correct amount of resources or not. You try to check this via spark UI and wonder why your application is running so long. So, there are certain ways that can help in improving the performance with the Spark jobs. In this article, we are going to understand 10 ways that can help in improving the spark performance. Let’s start with this.

1. Use Right File Format

Choosing an appropriate file format is very important as a good file format can provide multiple benefits such as Faster Read Time, Faster Write Time, etc. In Hadoop, there are multiple file formats and they have their own benefits. The most widely used file formats with SPark and Hadoop are Parquet, Avro, and ORC.

Avro is a row-based file format so it is good when you want to read the whole row from a table. Avro is also useful when the schema for a dataset keeps changing. It supports schema evaluation in a better way. Avro also works better with serialization/deserialization use cases. It also supports block-level compression.

Parquet is columnar file storage. It groups together column values and stores them adjacent to each other. This is particularly useful if the spark or any data processing framework needs access to a subset of columns from a dataset and it can access those very fast without reading the entire row. Parquet supports both File-Level Compression and Block-Level Compression.

ORC reduces the size of original data up to 70% hence as a result, it is faster to retrieve the data. ORC is another columnar storage format that works better than text, sequence, and RCFile format. ORC stores the data in stripes with metadata along with the data like — min/max/count/sum etc. ORC uses Bloom filters. Bloom filter is a data structure that can tell you, rapidly and memory-efficiently, whether an element is present in a dataset or not. Bloom filter uses a Hash function for this purpose.

2. Use Right File Compression

File compression reduces the size of the original data and makes it suitable for data transfer/shuffle operation. Apache Spark uses the codecs provided by Hadoop to read the compressed files. We can get the list of supported codecs in the following class — org.apache.spark.sql.catalyst.util.CompressionCodecs.

private val shortCompressionCodecNames = Map(

"none" -> null,

"uncompressed" -> null,

"bzip2" -> classOf[BZip2Codec].getName,

"deflate" -> classOf[DeflateCodec].getName,

"gzip" -> classOf[GzipCodec].getName,

"lz4" -> classOf[Lz4Codec].getName,

"snappy" -> classOf[SnappyCodec].getName)

However, there is a problem of parallelism to read a compressed file so we need to choose a conversion file format that is splittable. Splittable compressed files are processed as any parallelizable file — by multiple executors. Bzip2 is natively splittable. Other ones like LZO became splittable thanks to the work of the Open Source community.

3. Use Right Configuration — Memory and Executors

This is by far the most important factor in tuning spark jobs. Efficient memory use is essential to good performance. Spark uses memory mainly for storage and execution. During execution, we also need to define the number of executors to perform a specific task. Sometimes it becomes difficult to obtain the proper metric for memory and executors as every spark job is different and requires a different amount of resources to perform a specific task.

While assigning memory and executors to a spark application, one needs to take care of resources for other operations also like OS processes, garbage collection, off-heap memory which takes a few portions of memory and executors. So from the available resources, you need to keep aside resources required by these processes.

If you are using a shared cluster and you are enabling Dynamic Resource Allocation, then there is one more property that you need to enable which is called External Shuffle Service. Let’s understand what this property does. Suppose one executor has just finished writing the data and becomes idle and another executor is reading the data written by the first executor, and let’s say dynamic resource allocation is enabled, in this case, the first executor will be killed as it’s sitting idle and so the metadata regarding that data will be gone and thus, executor-2 will not know how to read the data anymore. So instead of giving the responsibility to the executor for shuffling, External Shuffle Service takes this responsibility and performs the job better without losing any data. Below are the properties, you need to set to enable external shuffle service.

spark.dynamicAllocation.enabled

spark.shuffle.service.enabled

spark.shuffle.service.port

spark.dynamicAllocation.schedulerBacklogTimeout

spark.dynamicAllocation.executorIdleTimeout

4. Data Serialization

Data Serialization helps in reducing memory usage and also determines a good network performance.

Spark supports two serialization libraries, as follows:

Java Serialization

Kryo Serialization

Java Serialization is the default for most of the types. This works with any class and that’s why it is more flexible. However, the objects after serialization are bigger in size so it is slow as compared to the Kryo Serialization.

Kryo Serialization is the default for shuffling RDDs and simple types. This is 4x faster and 10x more compact than the Java Serializer. However, this serializer requires you to register custom classes. Below is the simple code demo to register a class with Kryo Serialization.

val conf = new SparkConf.setMaster(<master url>)setAppName(<App Name>) conf.registerKryoClasses(Array(classOf[class1],classOf[class2]))

val sc = new SparkConext(conf)

If your objects are large, you may also need to increase the buffer max size. Increase this if you get a “buffer limit exceeded” exception inside Kryo.

spark.kryoserializeer.buffer.max=64Mb

5. Handle Skewed Data and Use of Salting

Data skew is the common data problem which can affect the spark application jobs badly. The cause of the data skew problem is the uneven distribution of the underlying data. Uneven partitioning is sometimes unavoidable in the overall data layout or the nature of the query.

So, how will you find out whether data is skewed or not? It’s not that difficult to find. In the Spark UI, you will observe that sometimes few of the tasks are taking longer on a particular executor as compared to others. This tells us that the partition on this executor is huge and tasks are running very slow because of this. This is a data skew problem.

There are multiple techniques used to handle skewed data. You can repartition the data as well but this is an expensive operation. So you need to be careful using this option.

If you are doing a join operation on a skewed dataset one of the tricks is to increase the spark.sql.autoBroadcastJoinThreshold value so that smaller tables get broadcasted.

Another popular way of handling skewed data is Salting. In a SQL join operation, the join key can be changed to redistribute data in an even manner so that processing for a partition does not take more time. If 70–80% of data belongs to the same key, then generate multiple keys by adding random values to the original key. You can use monotonically_increasing_id() for generating random values. This will help in partitioning the data well during the join operation.

6. Broadcasting

Join operations in Spark is sometimes the biggest source of performance problems. There are multiple ways of joining tables in Spark -

ShuffleHashJoin — A ShuffleHashJoin is the most basic way to join tables in Spark. This is the same Hadoop Map Reduce process which involves 3 steps — Map, Shuffle and Reduce. It maps through both the tables, shuffles the data based on the output (join) key, and then reduces the data and provides the join output.

Shuffle Hash join works best

When data distribute evenly with the key you are joining on. Have an adequate number of keys for parallelism.

For example — If you have 2 Million rows in one of the tables but only 200 keys then, this will not work as this will get parallelism more than 200.

BroadcastHashJoin — BroadcastHashJoin is also a common way of joining tables but in this case, one of the tables must be small enough to fit in a single machine. In Broadcast Hash join, a small table gets copied over to every datanode in the cluster and executors use the local copy of this table to join with the bigger table. Hence we don't need the shuffle phase in this join so this helps in improving performance. This is automatic for many data formats like parquet (if one of the tables is small) but may need to provide a hint to enforce the broadcast hash join by using keyword — broadcast() for other types.

import org.apache.spark.sql.functions.broadcast

broadcast(spark.table("src")).join(spark.table("records"), "key").show()

Cartesian Join — A Cartesian join can easily explode the number of rows so you need to avoid this at all cost and need to find alternate methods to perform this kind of a join.

7. Avoid Data Shuffling

During spark join operations, lots of data shuffle across data nodes which is a very expensive operation. Your goal would be to reduce the amount of data shuffle wherever possible. There are certain transformations that you can avoid to reduce the data shuffle. Here are some recommendations which you can follow —

Use ReduceByKey() or aggregateByKey() instead of GroupByKey() as groupByKey() shuffles all the keys whereas, in reduceByKey, data is combined at each partition, and only one output for one key at each partition is sent over the network. If you want to decrease the no of partitions then use coalesce() instead of repartition() as repartition does a full shuffle of data whereas coalesce tries to avoid the shuffle. Output partitions can be uneven in size in the case of Coalesce. However, use repartition() when you wanted to increase the number of partitions.

8. Proper Use of Cache/Persist

Spark provides an optimization mechanism to store the intermediate computations of a spark dataframe so that it can be reused in subsequent actions. We can store the intermediate RDD/Dataframe results using cache() and persist(). So why we need this in the first place. The reason is in spark, we apply transformations and create different rdds as at each step. These rdds gets evaluated only when an action is called. Suppose you have called an action but later you realized that you need to use a different action on the last transformed rdd, in this case, the second action will have to recompute the rdd since start and apply all the transformation again. This will increase the cost and time. So it's better to save an intermediate state of an rdd (which was computed through various transformations) and then use the persisted rdd for the second action.

There are different storage levels provided for the persist() which can be used depending on the use case whereas cache() only uses the default storage level i.e. MEMORY_ONLY.

persist() storage levels -

MEMORY_ONLY

MEMORY_AND_DISK

MEMORY_ONLY_SER

MEMORY_AND_DISK_SER

DISK_ONLY

When you are going to cache using an in-memory columnar format, by tuning the batchSize property, you can further improve Spark performance.

spark.conf.set("spark.sql.inMemoryColumnarStorage.compressed", true)

spark.conf.set("spark.sql.inMemoryColumnarStorage.batchSize",10000)

9. Degree of Parallelism

Spark is all about parallel computations. Hence we need to define how many levels of parallelism we want in processing the data. Too low parallelism means the job will be running longer and may end up throwing Out of Memory exception whereas too high parallelism would require a large number of resources (cores) which can be an expensive operation. So defining the degree of parallelism depends on the number of cores available in the cluster. Ideally, we should make RDD with the number of partition is equal to the number of cores in the cluster by this all the partitioned will be processed in parallel and equal resources will be used.

There are 2 properties which can be used to increase the level of parallelism -

spark.default.parallelism

spark.sql.shuffle.partitions

spark.sql.shuffle.partitions is used when you are dealing with spark SQL or dataframe API.

10. Tuning Garbage Collection

Spark allows users to persistently cache data so that it can be reused later to avoid the overhead caused by repeated computations. Spark’s executor divides JVM heap space into 2 fractions -

Storage of RDD as a result of Persist/Cache action. Memory consumption during RDD transformation.

Therefore, the less memory space RDD takes up, the more heap space is left for program execution, which increases GC efficiency otherwise the performance will be significantly impacted due to the large no of buffered objects in the older generations.

So when GC is observed as too frequent or long-lasting, it may indicate that memory space is not used efficiently by Spark process or application. You can improve performance by explicitly cleaning up cached RDD’s after they are no longer needed.

Garbage-First GC (G1 GC) is the recommended garbage collector. The G1 collector is planned by Oracle as the long term replacement for the CMS GC. Most importantly, with respect to the CMS, the G1 collector aims to achieve both high throughput and low latency.

Other ways

Avoid Un-necessary Calculations

When you have heavy initializations like initializing classes, database connections then go for mapPartitions() rather than map() as mapPartitions() does this once for each partition instead of doing it on every DataFrame row.

Use Vectorization

Similar to the Hive, we can use vectorization in Spark as well. Vectorization works with Columnar data format only. It processes a batch of rows at a time instead of a single row.

Use Bucketing

Bucketing is a useful concept that is not only used in Hive but also can be used in Spark. With bucketing data is already shuffled and sorted hence bucket joins avoid data shuffle operations significantly.

Conclusion

We have come to the end of the article. I have tried to cover most of the performance improvement techniques which I have also utilized in some of the projects. However, Spark performance tuning is a continuous process that never stops and also it depends on the use-case, amount of data and cluster resources, and many other things. I hope, this article will help you in some way. Keep learning and keep sharing !!!