# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
In the same way, for the 1 TBworkload, we only used the first 20 sampled data points within the range from 4 to 95 executors on the x-axis to get the best fit of its actual elbow curve for computation in the BToP method. Our goal is to have the best fitted elbow curve from the limited preview data points within the range of interest where the best trade-off point could be located to provision only the optimal number of executors needed for a workload to reserve the remaining executor resources for other jobs in a multi-tenant Spark YARN cluster to improve the overall system throughput. As such, we could ignore the outlier data points far beyond the turn of the elbow on the runtime curve when curve fitting the elbow graph for implementing the BToP algorithm. In fact, we do not recommend provisioning more executor resources beyond the best trade-off points, which offers rapidly diminishing returns and, possibly, adverse effects, when it comes to runtime performance and energy efficiency. Running Spark with DRA enabled, we notice that Spark dynamically fluctuates the utilized resources up to a maximum of 80 executors for the 10 GB workload and 119-120 executors for the 100 GB and 1 TB workloads. The Spark-Bench Terasort tests with DRA enabled result in 321.897 s for 10 GB, 588.674 s for 100 GB, and 2923.138 s for 1 TB (Figure 5 and Table 1). Running Spark-Bench Terasort with BToP method (DRA disabled), the recommended optimal numbers of executors are 11-13-18 for 10 GB, 33-36-40 for 100 GB, and 85-89-94-99 for 1 TB. By extracting the corresponding runtimes from the fitted elbow curves of Spark-Bench Terasort (Figure 5), we obtain the durations of Spark with BToP method (DRA disabled) for 10 GB, 100 GB, and 1 TB workloads as shown in Figure 3:5

These corresponding durations derived from the optimal numbers of executors recommended by the BToP method consistently outperform the durations of Spark with DRA enabled for all three workloads. The performance gains are up to 10.76% for 1 TB, up to 18.85% for 100 GB, and up to 64.37% for 10 GB (Table 1). The question remains whether the difference in runtimes for performance gain is significant enough to justify the additional ground work required for applying the BToP method to Spark, particularly for applications outside of a production environment. The answer to that question depends on the performance criteria and Service Level Objectives (SLO) in each individual case. Users will have to decide whether to apply the BToP method by weighing the additional benefits in performance gain and energy saving against the extra work involved in creating a database of resource consumption signature for dynamic job profiling. Our Spark-Bench Terasort experiment proves that the BToP method is more advantageous for production runs since the BToP algorithm recommends the optimal number of executors by matching a workload to its equivalent predetermined resource consumption signature for behavioral replication. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Spark-Bench Terasort**} & \multirow{2}{*}{**DRA Enabled (s)**} & \multicolumn{2}{c}{**BToP Method (DRA Disabled)**} & \multirow{2}{*}{**Improved Performance (\%)**} \\ \cline{3-4} \cline{5-5}  & & **Executors** & & **Duration (s)** \\ \hline \multirow{3}{*}{10 GB} & \multirow{3}{*}{321.897 s} & 11 & 129.43 & 59.79 \\ \cline{3-4} \cline{5-5}  & & 13 & 122.06 & 62.08 \\ \cline{3-4} \cline{5-5}  & & 18 & 114.70 & 64.37 \\ \hline \multirow{3}{*}{100 GB} & \multirow{3}{*}{588.674 s} & 33 & 513.50 & 12.77 \\ \cline{3-4} \cline{5-5}  & & 36 & 498.77 & 15.27 \\ \cline{3-4} \cline{5-5}  & & 40 & 477.73 & 18.85 \\ \hline \multirow{3}{*}{1 TB} & \multirow{3}{*}{2923.138 s} & 85 & 2739.04 & 6.30 \\ \cline{3-4} \cline{5-5}  & & 89 & 2694.84 & 7.81 \\ \cline{3-4} \cline{5-5}  & & 94 & 2644.34 & 9.54 \\ \cline{3-4} \cline{5-5}  & & 99 & 2608.56 & 10.76 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance of Spark-Bench Terasort with Dynamic Resource Allocation (DRA) enabled vs.