# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Spark-Bench**} & \multicolumn{3}{c}{**DRA Enabled**} & \multicolumn{3}{c}{**BToP Method**} & \multicolumn{3}{c}{**Energy Saving**} \\  & \multicolumn{3}{c}{**DRA Disabled**} & & \multicolumn{3}{c}{**Energy Savings**} \\ \hline \multirow{2}{*}{**Terasort**} & **Max** & **Equiv.** & **Energy** & **Static** & **Equiv.** & **Energy** & **Per Job** & **Per Year** & **Per Year** \\  & **Exec.** & **Nodes** & **(Wh/job)** & **Exec.** & **Nodes** & **(Wh/job)** & **(\%)** & **(\%)** & **(\%)** \\ \hline
10 GB & 80 & 10 & 398.36 & 18 & 3 & 188.37 & 52.71 & 2207.41 & 329.57 \\
100 GB & 120 & 15 & 770.60 & 40 & 5 & 330.53 & 57.11 & 4626.02 & 690.66 \\
1 TB & 120 & 15 & 3049.95 & 99 & 13 & 2380.5 & 21.95 & 7037.26 & 1050.66 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Energy savings in using Spark with BToP method in lieu of Dynamic Resource Allocation (DRA) enabled for Spark-Bench Terasort of 10 GB, 100 GB, and 1 TB. ### Performance Prediction Models

For a better resource allocation framework, [PERSON] and [PERSON] [40] leveraged the multi-stage execution structure of Apache Spark jobs to develop several hierarchical models for predicting job execution time, memory footprint for RDD creation, and I/O overhead. These performance metrics were extracted from a simulated run of an actual job in a limited scale on real homogeneous cluster by analyzing Spark event log files. To alleviate the effect of data skew in the simulation, input data was divided into multiple sections where sampled data was extracted with equal probability. The authors predicted the job performance based on the limited scale execution job performance data and evaluated their framework with four real-world applications. In each case, the prediction accuracy was high for execution time and memory, but varied with different applications and simulation setup for I/O cost. Although this performance prediction approach is also based on modeling system behavior, its experimental evaluation process is very time consuming and complex, with required calculations of the Stagetime, Tasktime, Startup time, and Cleanup time of job and stage, the I/O cost details for each task through TaskIOWrite and TaskIORead, and the average RDD memory footprint for each stage. In contrast to these complex performance prediction models, which might not always provide accurate and consistent results at different stages, the BToP method provides a much simpler and more reliable database of executor resource consumption signatures optimized by its algorithm from the historical job performance data of each specific application and system for unwavering precision in behavioral replication. ### Optimal Resource Ratio by Distribution Probability

In exploring the effectiveness of executor resource quota connecting to stage performance with various input sizes, [PERSON] et al. [41] used the time distribution probability and quota ratio of CPU cores and memory to determine the optimal ratio for each executor for a typical Spark job. By combining the resource ratio, variance probability and variant total time with the specific resource quota configuration, the authors found the correlated inflection point for best performance to allocate resources accordingly. However, such tedious executor quota calculation is quite complex, especially when it is very difficult to manually fine tune Spark performance for numerous different job characteristics to overcome varying bottlenecks at every processing stage. That is what we try to avoid with the BToP method which automates the resource provisioning process after a database of optimal executor resource consumption signatures has been generated by its algorithm from historical performance data of different workloads. ### Resource Scheduling Algorithm

In the resource scheduling area, [PERSON] et al.