Apache Spark: Caching. Apache Spark provides an important... | by Vipul Bhardwaj | Towards Data Science

Apache Spark provides an important feature to cache intermediate data and provide significant performance improvement while running multiple queries on the same data. In this article, we will compare different caching techniques, benefits of caching, and when to cache our data.
How to cache
Refer DataSet.scala
df.cache
The cache method calls persist method with default storage level MEMORY_AND_DISK. Other storage levels are discussed later.
df.persist(StorageLevel.MEMORY_AND_DISK)
When to cache
The rule of thumb for caching is to identify the Dataframe that you will be reusing in your Spark Application and cache it. Even if you don’t have enough memory to cache all of your data you should go-ahead and cache it. Spark will cache whatever it can in memory and spill the rest to disk.
Benefits of caching DataFrame
Reading data from source(hdfs:// or s3://) is time consuming. So after you read data from the source and apply all the common operations, cache it if you are going to reuse the data.
By caching you create a checkpoint in your spark application and if further down the execution of application any of the tasks fail your application will be able to recompute the lost RDD partition from the cache.
If you don’t have enough memory data will be cached at the local disk of executor which will also be faster than reading from the source.
If you can only cache a fraction of data it will also improve the performance, the rest of the data can be recomputed by spark and that’s what resilient in RDD means.
Caching methods in Spark
We can use different storage levels for caching the data. Refer: StorageLevel.scala
DISK_ONLY: Persist data on disk only in serialized format.
MEMORY_ONLY: Persist data in memory only in deserialized format.
MEMORY_AND_DISK: Persist data in memory and if enough memory is not available evicted blocks will be stored on disk.
OFF_HEAP: Data is persisted in off-heap memory. Refer spark.memory.offHeap.enabled in Spark Doc.
We can explicitly specify whether to use replication while caching data by using methods such as DISK_ONLY_2, MEMORY_AND_DISK_2, etc.
We can also specify whether to serialize data while storing. Methods like MEMORY_ONLY_SER etc. Using serialized format will increase processing time but decrease the memory footprint.
Serialization with replication is also available e.g. MEMORY_ONLY_SER_2
Comparison between different storage levels
Below tests are run on the local machine with 12GB driver memory and input data of size 14GB and time was recorded using `spark.time` for each iteration. It outputs run time of a query which was executed 10 times in a loop.
import org.apache.spark.storage.StorageLevelval df = spark.read.option("header",true).csv("/test/test.csv")df.persist(StorageLevel.DISK_ONLY)for(_ <- 0 until 10){    spark.time(df.filter($"category_code" like  "%electronics%").count)}
Note: Remember that cache() in spark is lazily evaluated. So when first action was called, at that time data will be cached. Caching will take additional time which can be seen in the graph above. For re-runs we observe significant performance benefit with cache.
Conclusion
In this post, we learned that caching is an important technique to optimize spark jobs. We should cache our data if we are going to use it multiple times in our code. It can give us a significant performance benefit.