# Optimization Techniques for a Distributed In-Memory Computing Platform by Leveraging SSD
## 1 Introduction  
As the big data industry develops rapidly, there have been a number of research efforts to build distributed processing frameworks, such as MapReduce [1] of Hadoop [2], that can effectively store and process \"Big Data\". However, the performance of Hadoop based on normal spindle disks (HDD) can be degraded due to the Hadoop distributed file system's (HDFS) [3] read-write operations, especially for machine learning workloads where there can be a lot of iterative jobs and intermediate data. To address this problem, the Spark [4] framework was introduced, which can effectively cache the intermediate data to memory so that the cluster computing platform can dramatically improve the overall performance. However, according to a comprehensive study analyzing Spark's performance behaviors [5], Spark still can have performance degradation problems due to some straggler tasks. As a result of analyzing the tasks that affect the whole job completion time of Spark, _garbage collection_, _shuffle write_, and _shuffle read_ are identified as main factors that can negatively affect the performance of the Spark system. Unfortunately, a detailed analysis of the main reasons why these aforementioned tasks affect the task processing in Spark and potential solutions have not yet been thoroughly investigated. In this paper, we first analyze the specific factors that can explain why garbage collection, shuffle write, and shuffle read affect the task processing in Spark and present associated situations and possible solutions through extensive experiments on a Spark cluster. Specifically, to analyze the factors causing the degradation of \"whole job completion time\" of the Spark, we performed various experiments with PageRank [6], transitive closure [7], TeraSort [8], and k-means clustering [9] workloads. Based on our extensiveexperimental results, we found potential performance degradation factors in the Spark system that can be summarized as follows:  
1. **The performance degradation on Java garbage collection**: Since Spark is running on Java Virtual Machines (JVM), Java garbage collections can especially occur when there is a lack of Spark executor memory, i.e., JVM heap size. 2. **The performance degradation on shuffle spill**: When the shuffle write is being processed, if the shuffle memory of the Spark executor memory (JVM heap) is insufficient, Spark will spill the shuffle data out to the disk (HDD). In this case, Spark has to serialize the data to write and de-serialize to read the data from the disk. Since a CPU resource is required for the serialization and de-serialization processes, this can slow down the overall processing of tasks. 3. **The performance degradation on shuffle read blocked time**: As we will see from the experimental results, if the number of tasks keeps increasing in iterative stages as in the transitive closure workload, there can be scheduling overhead and Java garbage collections due to lack of Spark executor memory. This can cause shuffle read tasks to be blocked which can result in poor performance. The main contribution of this paper is that we propose effective Spark cluster configuration strategies that can actually improve the overall system performance by utilizing SSDs in order to overcome the physical memory limits of the cluster. In a typical cluster computing environment consisting of commodity servers, it would be difficult to set up a large amount of main memory. Therefore, we address the performance degradation problems in a distributed in-memory computing system that can occur due to insufficient main memory amounts by effectively leveraging SSDs. Our optimization strategy is two-fold as follows. First, we change the capacity fraction ratios of the shuffle and storage spaces in the \"Spark JVM Heap Configuration\". According to the experimental results of different workloads, we observe the performance differences depending on the workload's memory usage patterns.