# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
From (2),

Figure 3:1

The BToP algorithm finds the optimal number of executors recommended for a workload by locating the best trade-off point at the bottom of the elbow curve where assigning more resources no longer significantly reduces the job duration, and therefore, reduces the overall system efficiency in resource utilization and energy consumption. Taking the parameter \(a\) in \(f(x)=(a/x)+b\) as input, our program computes and tabulates the number of executors over a range of slopes from \(-0.25\) to \(-39.25\) for \(x=\sqrt{-a/slope}\), and the acceleration over a range of slopes from \(-0.25\) to \(-39.25\) for \(\left|acceleration\right|=\left|2a\times slope^{-3}\right|\). \(b\)The algorithm applies the Chain Rule to search for break points and major plateaus on the graphs of acceleration, slope, and executor resources over a range of incremental changes in acceleration per slope increment (Figures 6 and 7). Applying the Chain Rule, the rate of change in acceleration with respect to executors is:

Figure 3:2

The BToP algorithm looks for break points on the graphs to compute a table of recommended acceleration, corresponding slope, and optimal number of executors, when the change in acceleration in the current slope increment is greater than or equal to the target value of change in acceleration per slope increment, and the change in acceleration in the next slope increment is less than the target value of change in acceleration per slope increment (Figures 6 and 7). Finally, the BToP algorithm searches for all major plateaus lasting at least 7 increments of change in acceleration on the graph of executors versus change in acceleration per slope increment, which corresponds to the graph of slope versus change in acceleration per slope increment and the graph of acceleration versus change in acceleration per slope increment. The minimum plateau length of 7 increments of change in acceleration per slope increment specified for a major plateau could be adjusted by user to further fine tune the search for the first recommended optimal executor resource value. For example, if we decrease the minimum plateau length to 5 increments from 7, the first recommended number of executors for Terasort 1 TB rounded to the nearest whole number will be 81 executors instead of 85 executors according to the plot of executors versus change in acceleration per slope increment in Figure 3:3 Similarly, if we increase the minimum plateau length to 8 increments from 7, the first recommend number of executors for Terasort 1 TB rounded to the nearest whole number will be 89 executors instead of 85 executors. However, if we change the minimum plateau length from 7 increments to 5 or 8 increments for Terasort 10 GB and 100 GB, the first number of recommended executors will not change. Apparently, the fine-tuning option has a more sensitive impact to the first number of recommended executors when the workload is heavy such as in the range of 1 TB or larger. As such, users have the option to set a smaller value for the minimum plateau length in the BToP algorithm to further get a slightly more efficient resource provisioning when the workload is heavy. Figure 3:4 The optimal numbers of executors for SparkBench Terasort 10 GB, 100 GB, and 1 TB workloads are identified by the major plateaus lasting at least seven increments on the graphs. The algorithm searches for break points in the changes in acceleration and outputs: (**a**) recommended acceleration, (**b**) corresponding slope and (**c**) executors versus change in acceleration per slope increment. The BToP algorithm extracts the exact number of executors at the best trade-off point on the elbow curve and outputs it as the recommended optimal number of executors for a workload (Figures 6 and 7).