# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
[42] proposed the Prophet scheduling algorithm to dynamically dispatch executors in terms of expected resource fragmentation, over-allocation, resource contention, and subsequent dramatic performance degradation due to unexpected over-allocation with its task back-off mechanism. Prophet chooses the optimal number of executors based on a greedy approach and the predictability of future executor resource demand. Although Prophet can learn and leverage the patterns in the executors' scheduling to best match time-varying executor demands to resource availability to reduce the makespan of YARN's default capacity and fair schedulers, it operates only at the scheduling level, and is specifically designed for data-parallel computation frameworks. Our BToP method, which is not really a resource scheduler, offers a novel and universal solution for optimal resource provisioning in any system characterizable by a trade-off elbow curve, and does not require any modification to the underlying system. As such, the BToP method could be used in conjunction with any resource scheduling methods, including Prophet, to get the most out of system performance improvement in Spark. ### Energy Conservation Algorithm

To decrease energy consumption in Spark, [PERSON] et al. [36] proposed a scale-down algorithm to remove under-utilized nodes to reduce the clusters' size and a Virtual Machine (VM) migration algorithm to properly adjust the load on over-utilized nodes of Spark clusters on the Cloud. Noticeably, the energy consumption of Spark clusters in the experiment was only simulated on CloudSim and not tested in real working conditions. These scaling down of operational clusters and VM migration resembles the energy conservation approach proposed for Hadoop MapReduce in [PERSON] et al. [37]. However, the authors in [36] did not consider the effect of Spark's built-in DRA, a sophisticated dynamic allocation mechanism which is not available in Hadoop MapReduce. Spark's DRA already handles the dynamic adjustment of resource allocation at the executor level by scaling down or ramping up the number of dispatched executors according to demand during run time. Thus, the real effectiveness of applying the additional scaling down algorithm on top of Spark's built-in DRA has yet to be verified in real working conditions since there might be cases where the algorithm could be in discord and interfere with the built-in DRA or vice versa. Since our invented BToP method matches static resources at the executor level to workloads according to a database of optimal resource consumption signatures, it can be safely combined with [PERSON] et al. 's scale-down algorithm for better energy saving in Spark. ## 7 Conclusions

Spark-Bench Terasort test confirms that our proposed Best Trade-off Point method for efficient resource provisioning in Apache Spark consistently outperforms Spark with its built-in Dynamic Resource Allocation enabled, in both runtime performance and energy efficiency. Although Spark's robust and sophisticated DRA mechanism is effectual for general purpose applications including any workload with erratic and unpredictable behavior, the BToP method might be preferable in production environments, where workload behaviors are predictable and repetitive, and can be replicated for optimal performance and energy saving. By optimizing resource provisioning, the BToP method improves overall system throughput and prevents cluster underutilization as well as starvation among applications running in Spark's multi-tenancy environment. This study shows that significant aggregate annual energy saving can be achieved when our proposed BToP method is adopted and consistently applied to all Big Data processing jobs running on all Spark and Hadoop clusters in today's large datacenters.