# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
[PERSON] et al. suggested setting 1.5 tasks per core as a rule of thumb for better performance across workloads. Moreover, Spark users should configure one big executor per node instead of many small executors with fewer allocated resources resulting in higher resource utilization, which might lead to worse performance due to resource contention. In our experiment on a small 24-node Spark YARN cluster, we are more interested in the performance improvement impacted by the BToP method on Spark than the further fine-tuning of Spark at the expense of the total available number of executors. Therefore, we run the experiment with a single core executor to have the largest possible available number of executors for testing. In other words, due to the resource constraint in our small 24-node cluster, we run the experiment with the minimum value of vCore set for an executor to have the full performance spectrum of up to 192 executors, which is the maximum numbers of executors configurable for that cluster. Such configuration eliminates the benefits of running multiple tasks in a single JVM. However, our purpose here is to find the ratio of improvement in resource utilization, performance, and energy saving in using the BToP method on the same cluster under test. If the cluster could be properly fine-tuned with more resources as in a production environment, the performance curve is expected to shift, while the ratio of improvement should still be consistent for using Spark with the BToP method instead of the built-in DRA mechanism. ### BToP Method Implementation in Spark

We now implement the seven steps outlined in Section 2 for efficient resource provisioning in Spark to apply the BToP method to Spark-Bench Terasort (Figure 1). _Step 1._ Complete the configuration and fine tuning of the architecture, software, and hardware of the production cluster-computing system targeted for calibration. _Step 2._ Collect necessary preview job performance data from historical runtime performances or sampled executions on the same target production system, configured exactly as in step 1, as reference points for each workload (Figure 3). In our experiment, we gathered approximately 20 sample points for each workload to plot the elbow curves. However, a slightly smaller number of sample runs would also be enough for us to obtain the fitted duration elbow curves for computation of the recommended number of executors

Figure 3: Plots of preview data points of job duration vs. executors for Spark-Bench Terasort 10 GB, 100 GB and 1 TB workloads at different vertical scales. for efficient resource allocation. In a production environment, if the necessary preview data to fit the elbow curve could be extracted from the system's existing historical performance data, then no further sample runs would be required. Additionally, the BToP method imposes no overhead on the framework since it neither makes any invasive change to the cluster computing system nor burdens any of its resources after the database of resource consumption signatures has been established. The preview data of Spark-Bench Terasort indicates that when the allocated number of executors is too small for a workload, the job duration is very high due to insufficient processing resources. However, provisioning executor resources more than what is needed for a workload might result in an actual decrease in performance at some point in time after the turn of the elbow on the runtime curve. Although increasing parallelization would normally improve job performance, the growing framework overhead and potential resource contention due to an excessive number of allocated executors could eventually degrade the overall performance. In our experiment, this observed phenomenon of performance degradation of Spark on YARN despite the steady increase in executor resources is more obvious with 10 GB and 100 GB workloads than with a 1 TB workload (Figure 3).