# Optimization Techniques for a Distributed In-Memory Computing Platform by Leveraging SSD
## 6 Conclusions  
In this paper, we have investigated the main factors for the performance degradation of the Spark system running on top of a commodity-server-based computing cluster with insufficient available main memories. After experimentation and analysis, we presented alternatives that can improve the overall performance.  
Java garbage collection occurs when the storage space of the JVM heap is insufficient due to a lack of physical memory. Java GC makes tasks wait for garbage collection so that the overall job completion time increases. The shuffle spill occurs when the shuffle space of the JVM heap is insufficient during the shuffle phase. Shuffle spill increases the CPU overhead to perform serialization for spilling intermediate shuffle data to the disk due to the lack of shuffle space. In the TC workload experiment, shuffle read blocked time makes the task wait for reading shuffle data through the network because of the lack of shuffle space. All of these factors can potentially increase the overall job completion time which can seriously affect the performance of the Spark system.  
To address these problems, we construct a cluster with an SSD and cache the RDD both on the memory _and_ SSD separately by utilizing the SSD to supplement the storage space of the memory. In addition, we adjust the JVM heap configuration for expanding the shuffle space. As a result, we could achieve a 30% performance improvement for the PageRank workload and a 42% performance improvement for the TC workload. We have identified that the shuffle spill can be a key factor of performance degradation and showed through experimentation that in workloads consisting of a number of iterations and shuffling, expanding the shuffle space can provide significant performance gains. In addition, we found that different memory usage patterns of jobs can affect the total execution time depending on the storage/shuffle memory percentage allocation in the JVM. According to the performance analysis of PageRank and k-means clustering, memory allocation in the JVM that is well-tuned to the workload characteristics can significantly improve job completion time.  
Integrating these findings into the Spark platform would be one of our future works. For example, if workloads can be characterized in terms of the amounts of shuffle data, an optimized configuration can be automatically applied to accelerate the processing of target workloads. Therefore, in heterogeneous server configurations, developing a workload memory usage-aware scheduling system can improve the overall performance of a Spark-based cluster.  
Conceptualization, [PERSON] ([PERSON]); methodology, [PERSON] ([PERSON]) and [PERSON]; software, [PERSON] and J.L. ([PERSON]); validation, J.C., [PERSON] ([PERSON]) and [PERSON] ([PERSON]); investigation, [PERSON] ([PERSON]) and J.-S.K.; resources, [PERSON] ([PERSON]) and J.-S.K.; data curation, [PERSON] and J.L. ([PERSON]); writing--original draft preparation, [PERSON] and J.L. ([PERSON]); writing--review and editing, [PERSON] ([PERSON]) and J.-S.K.; visualization, [PERSON] ([PERSON]); supervision, [PERSON] ([PERSON]) and J.-S.K.; project administration, [PERSON] ([PERSON]) and [PERSON]-[PERSON]; funding acquisition, [PERSON] ([PERSON]). All authors have read and agreed to the published version of the manuscript.  
This research was supported by the Basic Science Research Program (NRF-2020R1F1A1072696) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT, GRRC program of Gyeonggi Province (No. GRRC-KAU-2017-B01, \"Study on the Video and Space Convergence Platform for 360 VR Services\"), and ITRC (Information Technology Research Center) support program (IITP-2021-2018-0-01423).  
Not applicable.  
Available upon request.  
The authors declare no conflict of interest.