# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
# Table of number of executors over range of slopes from -0.25 to -39.25 for x=sqrt(-a/slope)
Slope Terasort 106B Terasort 100 GB Terasort 100 GB Terasort 1 TB -0.25 40.24 164.62 569.18 -1.25 18.00 73.62 254.54 -2.25 13.41 54.87 189.73 -3.25 11.16 45.66 157.86 -4.25 9.76 39.93 138.05 -5.25 8.78 35.92 124.21 -6.25 8.05 32.92 113.84 -7.25 7.47 30.57 105.69 -8.25 7.01 28.66 99.08 -9.25 6.62 27.06 93.57 -10.25 6.28 25.71 88.89 -11.25 6.00 24.54 84.85...
#Table of acceleration over range of slopes from -0.25 to -39.25 for |accele|=|2a(1/pow(slope,3))|
Slope Terasort 10 GB Terasort 100 GB Terasort 17B -0.25 51818.24 867164.16 10366899.20 -1.25 414.55 6937.31 82935.19 -2.25 71.08 1189.53 14220.71 -3.25 23.59 394.70 4718.66 -4.25 10.55 176.50 2110.10 -5.25 5.60 93.64 1119.41 -6.25 3.32 55.50 663.48 -7.25 2.12 35.56 425.06 -8.25 1.44 24.13 288.47 -9.25 1.02 17.12 204.67 -10.25 0.75 12.58 150.42 -11.25 0.57 9.52 113.77...
Table of acceleration, slope, and number of executors over incremental change in acceleration #Accele Terasort 106B Terasort 100 GB Terasort 17B #Change Accele Slope Executor Accele Slope Executor Accele Slope Executor Accele Slope Executor 1 3.32 -6.25 8.05 5.82 -13.25 22.61 10.06 -25.25 56.64 2 5.60 -5.25 8.78 9.52 -11.25 24.54 16.88 -21.25 61.74 3 10.55 -4.25 9.76 12.58 -10.25 25.71 22.71 -19.25 64.86 4 10.55 -4.25 9.76 17.12 -9.25 27.06 31.56 -17.25 68.52 5 23.59 -3.25 11.16 24.13 -8.25 28.66 37.75 -16.25 70.60 6 23.59 -3.25 11.16 24.13 -8.25 28.66 37.75 -16.25 70.60 7 23.59 -3.25 11.16 24.13 -8.25 28.66 45.67 -15.25 72.88 8 8 23.59 -3.25 11.16 35.56 -7.25 30.57 55.98 -14.25 75.39 9 23.59 -3.25 11.16 35.56 -7.25 30.57 55.98 -14.25 75.39 10 23.59 -3.25 11.16 35.56 -7.25 30.57 69.63 -13.25 78.18 12 23.59 -3.25 11.16 35.56 -6.25 32.92 69.63 -13.25 78.18...

For Terasort 106B, the recommended number of executors is 11.16  For Terasort 106B, the recommended number of executors is 13.41  For Terasort 106B, the recommended number of executors is 18.00  For Terasort 106B, the recommended number of executors is 32.92  For Terasort 106B, the recommended number of executors is 35.92  For Terasort 106B, the recommended number of executors is 39.93  For Terasort 17B, the recommended number of executors is 84.85  For Terasort 17B, the recommended number of executors is 88.89  For Terasort 17B, the recommended number of executors is 93.57  For Terasort 17B, the recommended number of executors is 99.08 ********************************

First recommended number of executors for same workload provides highest efficiency in performance/energy ratio. Subsequent number(s) slightly improves job runtime. **Figure 6.** Applying BToP algorithm to Spark-Bench Terasort 10 GB, 100 GB, and 1 TB workloads to tabulate the number of executors over range of slopes, acceleration over range of slopes, and recommended acceleration, slope, and optimal number of executors over range of incremental changes in acceleration per slope increment, to output the final recommended optimal numbers of executors for each workload. _a._ The algorithm computes the number of executors over a range of slopes from the first derivative of \(f(x)=\left(\frac{a}{x}\right)+b\) and the acceleration over a range of slopes from the second derivative (Figure 6). Taking the second derivative of the function \(f(x)\), which is the derivative of the slope, we have the acceleration of the rate of change in number of executor resources:

\[f^{\prime\prime}\left(x\right)=f^{\prime}\left(f^{\prime}(x)\right) \tag{5}\]

\[=\lim_{\Delta x\to 0}\frac{\left[\left(\frac{\left[f(x+\Delta x)-f(x)\right]}{ \Delta x}\right)-\left(\frac{\left[f(x)-f(x-\Delta x)\right]}{\Delta x}\right) \right]}{\Delta x} \tag{6}\]

Figure 3:0

as the second symmetric derivative.