# Optimization Techniques for a Distributed In-Memory Computing Platform by Leveraging SSD
## 2 Background and Related Research Work
### Related Work  
There have been many related studies in the literature regarding performance improvements of the Spark platform as follows. Table 1 summarize related work according to the subjects. * **Improving performance of Spark shuffle:** The optimization of shuffle performance in Spark [11] analyzes the bottleneck on running a Spark job and presents two alternatives, columnar compression and shuffle file consolidation. Because spilling all data of the in-memory buffer is a burden for the OS, the solution is to write fewer, larger files in the first place. [PERSON] et al. presented a new adaptive I/O method for collective data shuffling [12]. They adapt the accumulation of shuffle blocks to the individual rate of processing for each reducer task, while coordinating the reducers to collaborate in the optimal selection of the sources (i.e., where to fetch shuffle blocks from). In this way, they balance loads well and avoid stragglers with reducing the memory usage for buffer. \begin{table}
\begin{tabular}{c c} \hline \hline
**Categories** & **Improvement Methods** \\ \hline Spark Shuffle Improvement & Network and Block Optimization [11; 12; 13] \\  & Cost-Effectiveness [14] \\ \hline Performance Analysis, Modeling & I/O-aware Analytical Model [15] \\  & Empirical Performance Model [16] \\ \hline Parameter Tuning & Empirical Tuning [17; 18] \\  & Auto-Tuning [19] \\ \hline Memory Optimization & Memory Optimization [20] \\ \hline JVM and GC Overhead & JVM Overhead [21] \\  & GC Overhead [22] \\ \hline Cache Management Policy & RDD Policy [23] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of related work. Riffle [13] is one of the most efficient shuffle services for large-scale data analytics. Riffle merges fragmented intermediate shuffle files into larger block files and thus converts small, random disk I/O requests into large, sequential ones. Riffle also mixes both merged and unmerged block files to minimize merge operation overhead. [PERSON] et al. suggest a cost-effective shuffle service by combining cheap but slow storage with fast but expensive storage to achieve good performance [14]. They run TPC-DS, CloudSort, and Big Data Benchmark on their system and show a reduction of resource usage by up to 59%. They all emphasize the improvement of shuffle performance and cost-effectiveness by coordinating network transfer or reducing disk I/O. However, in our study, we configure the JVM heap to reduce shuffle spill which is a bottleneck in the shuffle phase and job completion time. * **Performance analysis, modeling, and optimization for Spark:** The authors of [15] show that storage I/O plays a heavy role in the in-memory cluster computing frameworks and propose an I/O-aware analytical model to reason through the performance of Spark programs. The proposed model can analytically explain and predict the runtime behavior of iterative algorithms that are computation/shuffle-heavy algorithms. They also apply the proposed model on cost optimization in Google Cloud. [PERSON] et al. show the performance analysis of Spark and Flink based on their experimental results using representative workloads in a comparative manner [16]. They identify a set of the four most important parameters having a major influence on performance. The task parallelism, the network behaviour during the shuffle phase, the memory, and the data serialization are the most important parameters they pick. On the other hand, in our study, we focus on the performance improvement methodology by appropriately choosing the type of _storage_ and allocating the amount of storage and shuffle areas in the _Spark's JVM heap_ according to the memory usage patterns of target workloads. * **Parameter Tuning for Spark:** There have been some research trials to improve the performance of Spark by tuning all configuration parameters. [PERSON] et al. show their experience of parameter tuning of Spark by trial and error [17]. They pick 12 key application instance-specific parameters and assess their impact using real executions on a petaflop supercomputer.