# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
Spark using BToP method (with DRA disabled). The dynamic job profiling is expected to be more precise for identical repetitive jobs in production environment. Moreover, its database of resource consumption signatures is built from historical data and sampled executions of the same target cluster system which has been configured exactly as it was initially set up at time of calibration for later production runs. Thus, Spark with BToP method appears to be always slightly faster than Spark with DRA enabled due to the inherent optimization of the BToP method. Nevertheless, when DRA is enabled, Spark relies on its built-in DRA mechanism, its executor request and executor remove policies, its job scheduler, and users' configurations of the dynamic allocation properties to efficiently allocate and deallocate resources as needed. Therefore, Spark with the BToP method might be more suitable for further optimizing the throughput of a target cluster in production environment, while Spark with DRA enabled is already efficient enough for general purpose applications. We choose Spark-Bench Terasort tests for evaluating the effectiveness of Spark using the BToP method, since the datasets of different sizes generated through a random sequence by Teragen to be sorted by Terasort will always be consistent and their results can be replicated. For many other kinds of workloads, having several different numbers of incremental data sizes needed in the experiment would require us to generate several custom datasets, which will not be public standard datasets. In that case, readers might not be able to verify the test results. On the other hand, even if different kinds of workloads are used, they will not change the effectiveness of the BToP method based on its ingenious design and intelligent nature in behavioral replication. Noticeably, the difference in workloads will be inherently factored into the unique historical performance data of each specific kind of workload used by the BToP method and its algorithm to generate the recommended number of executor resources for efficient allocation as the optimal resource consumption signatures for subsequent behavioral replication. In fact, applying the BToP method to Spark-Bench Teragen gave us the same consistent improvement in performance results as Terasort which we consider as redundant and unnecessary, as it would repeat a point which has already been proven for the BToP method. ### Energy Saving

The performance gain in using Spark with the BToP method also leads to some energy savings, which can be estimated and quantified by using the following energy consumption equation:

\[\text{Energy(N)}=[\text{Time}_{\text{run}}(\text{N})\times\text{Power}_{ \text{active}}(\text{N})]+[\text{Time}_{\text{idle}}\times\text{Power}_{ \text{idle}}] \tag{10}\]

The energy consumption per job can be computed from the linear sum of job duration multiplied by active power and idle duration multiplied by idle power [5; 35; 36]. These power models based on a linear interpolation of CPU utilization have been verified to be accurate with I/O workloads for this class of server, since network and disk activity contribute negligibly to dynamic power consumption [37; 38]. Although the power consumption at the node level varies with the computation phases and workload, it could be computed at the executor level based on the number of allocated executors at each different phase. Spark's DRA mechanism provides an elastic scaling ability which monitors the load of the current application and allows executors to be allocated as needed through the acquisition and release of executors accordingly during run time. As such, we simply estimate the worst-case values of energy consumption of each workload by using the maximum number of executors assigned for DRA.