# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
The BToP algorithm finds the optimal number of executors recommended for a workload by locating the best trade-off point on the elbow curve where assigning more executors no longer significantly reduces the job execution time, and therefore, reduces the overall system efficiency in resource utilization and energy consumption. The first recommended number of executors for the same workload provides the highest efficiency in system performance and energy consumption ratio. The subsequent recommended number(s) of executors lowers the job runtime a little bit more, but at a much less efficient performance/energy ratio. However, increasing the number of executors beyond the recommended range does not necessarily translate into any further performance gain and, on the contrary, might adversely increase the job duration. The experimental results prove that Spark using the BToP method could still consistently outperform Spark with DRA enabled in all three tested workloads of 10 GB, 100 GB, and 1 TB, despite the sophistication and robustness of Spark's DRA. _Step 5._ Repeat steps 2-4 to gather enough resource provisioning data points for different workloads to build a database of resource consumption signatures for subsequent job profiling. _Step 6._ Repeat steps 1-5 to recalibrate the database of resource consumption signatures if there are any major changes to step 1. The BToP method is based on behavioral replication, which is accomplished by the creation and maintenance of a database of resource consumption signatures for job profiling to match dynamically-submitted jobs to their recommended optimal static number of resources computed by the BToP algorithm. However, we do not suggest the simple scaling of the insights gained from a small-scale run to a later large-scale execution due to the expected difference in the behavior of several parts of each cluster size including the communications, synchronizations, and convergences of its nodes. Even for the same cluster-computing system, step 6 of the BToP method requires the user to \"[r]repeat steps 1-5 to recalibrate the database of resource consumption signatures if there are any major changes to step 1\", which could be a change in the configuration or fine tuning of the architecture, software, and hardware of the cluster targeted for calibration. _Step 7._ Use the database of resource consumption signatures to match dynamically-submitted production jobs to their recommended optimal number of executors for efficient resource provisioning. For compute-intensive or I/O intensive applications, which are practically different kinds of workloads, their differences are inherently factored into their unique historical performance data used by the BToP algorithm to create their own specific executor resource consumption signatures for later job profiling and behavior replication of corresponding workloads. ## 5 Analysis of Spark with BToP method vs. Spark with DRA Enabled

### Performance Gain

From the graphs of job duration vs executors of Spark-Bench Terasort, we notice that the durations of 10 GB, 100 GB, and 1 TB workloads eventually get longer instead of shorter, even though the number of executors continue to increase after 20, 60, and 95, respectively. This clearly indicates that adding far more executor resources long after the best trade-off point adversely increases Spark's runtime instead of reducing it. This phenomenon is more noticeable with small workloads of 10 GB and 100 GB than with the large 1 TB workload, due to growing framework overhead and resource contention. As such, for the 10 GB workload, we had to limit the number of preview data points used for curve fitting to only the first 7 sampled data points within the range from 1 to 20 executors on the x-axis to get the best fit of its actual elbow curve and to avoid any skew caused by the outliers beyond that range. Similarly, for the 100 GB workload, we only used the first 17 sampled data points within the range from 1 to 60 executors on the x-axis to get the best fit of its actual elbow curve.