# Optimization Techniques for a Distributed In-Memory Computing Platform by Leveraging SSD
## 5 Discussion and Summary  
### Discussion  
We analyzed the main factors of potential performance degradation problems based on the feature of workload and processing stages. Our extensive experimental results are summarized with respect to applying the performance optimization techniques of the Spark platform for various workloads as follows:  
* **The performance degradation by Java garbage collection**: In the PageRank workload with the 500 MB dataset and 1 GB dataset, the GC occurs when there is insufficient storage space of the JVM heap to store the RDD. In the Distinct0 stage that reads the input file from the HDFS and caches it into the RDD, the GC occurs. We expand the storage space of the JVM heap through the configuration to solve this GC problem. We can improve performance to reduce GC because the storage space of the JVM heap can be expanded. In Figures 3 and 5, with the same RDD caching option, the_3 configuration shows the best performance in the Distinct0 stage. In addition, in PageRank with the 1 GB dataset, some options fail in the _flatMap_ stage due to the lack of memory. The GC overhead increases so much that the stage fails or goes into an infinite loop. Thus, we construct the cluster with SSDs to solve this problem. It shows a performance improvement and succeeds in the job that failed using only memory, as seen in Figure 6, M&S_1 and S_1.
* **The performance degradation by shuffle spill**: In the PageRank workload with the 500 MB dataset and 1 GB dataset, in the _flatMap_ stage, we can see that option M&S_1 shows the best performance because it has the least amount of shuffle spill (Figure 4: M&S_1 is 30% faster than N_1; Figure 6: M&S_1 is 40% faster than N_3). PageRank has many shuffle tasks. Thus, when the shuffle space of the JVM heap is insufficient for shuffling the data through the network, the shuffle spill occurs. Therefore, in order to reduce the shuffle spill, expanding the shuffle space of the JVM heap becomes the key factor of performance improvement. Furthermore, we can improve the performance by storing the RDD both on the memory and SSD. This can make the executor expand the shuffle memory of the JVM heap to reduce shuffle spill. If there are more iterations, the performance from the _flatMap_ stage would be the key point of the performance improvement. In the 1 GB dataset experiment, the job execution time of S_3 is the best option, because RDDs are cached on the SSD only, and there is sufficient heap memory on the executors. Thus, in the S_3 option, the _Distinct_ stages are faster than any other option. However, if the iteration number increases, the _flatMap_ stage affects the job execution time. Thus, the M&S_1 option can achieve great performance in this case. Through these analyses, we can identify that shuffle has a key effect on job completion time. Thus, we have to expand the shuffle memory of the JVM heap and cache the RDD both in the memory and SSD to obtain enough shuffle memory space for preventing shuffle spill.
* **The performance degradation by shuffle read blocked time**: There is shuffle read blocked time on the TC workload. It occurs when there are a lot of tasks in the stage and each task needs to read the previous RDD through the network. As the result of the TC experiment in Figure 7), the M&S option is faster than option M. In the same RDD caching option, expanding the shuffle space of the JVM heap is faster than expanding the storage space. The reason for improved performance is that by expanding shuffle space of the JVM heap, the shuffle read blocked time decreases in each task.