# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
Apache Spark, which can access HDFS datasets without being tied to the two-stage MapReduce paradigm [17], also supports running application JARs in HDFS. In this experiment, we evaluate the effectiveness of resource allocation of Apache Spark in Hadoop YARN cluster mode. The BToP method is a breakthrough method for optimal resource provisioning in the scientific, computing, and economic communities, since there has been no solution for finding the best trade point between performance and resources in the past. It is a unique and innovative method for efficient resource provisioning in Spark as well as MapReduce, among many other applications and systems. Although there is an abundance of on-going development and research work to improve many features of Spark, we focus on the dynamic resource allocation mechanism in this paper, since it is the main feature directly related to the objective of the BToP method. This feature is not available in HadoopMapReduce, which relies on YARN for resource allocation at the container level. However, containers allocation is inefficient and does not support a high-level of cloud elasticity. Here, we closely investigated Spark's built-in DRA because we wanted to show that despite its robustness and sophistication, our innovative BToP method can still outperform it in terms of optimal resource provisioning for efficient performance and energy saving, as shown in our experiment comparing Spark with DRA enabled and Spark using the BToP method with DRA disabled. If other features of Spark, which are not directly related to executor resource allocation, also improve its performance, such as the use of RDDs and in-memory processing vs Hadoop MapReduce's use of persistent storage and batch processing, they have already been indirectly taken into account by the intelligent BToP method and algorithm, which generate the optimal resource provisioning values from the actual historical system performance data. In YARN cluster mode, each instance of SparkContext runs an independent set of executor processes, while YARN provides facilities for scheduling across applications. Multiple Spark jobs initiated by different threads may run concurrently within each Spark application which gets its own executor processes. Spark runs _long-running_ processes and threads, which stay up through the entire duration of the application and execute tasks in multiple threads, to avoid the overhead of repeatedly invoking tasks [9; 10]. Allocation of executor resources on the cluster can be controlled by Spark YARN client using the _-num-executors_ option, which overrides Spark's built-in DRA mechanism [18]. We use this option to apply our BToP method and algorithm to determine the optimal static number of executors required for a workload throughout its entire execution. Apache Spark could relinquish executors when they are no longer used, and acquire executors when they are needed, according to its DRA mechanism to gracefully decommission an executor by preserving its state before its removal using timeout [19; 20]. Spark's DRA offering an elastic resource scaling ability, which is missing in MapReduce, helps prevent under-utilization of cluster resources allocated for an application and starvation of others in a multi-tenant system environment. To determine whether Spark with DRA enabled or Spark using the BToP method with DRA disabled would be most suitable for certain circumstances, we need to examine Spark's architecture with its RDD, its distributed execution and DRA mechanism. ### Spark Architecture and Resilient Distributed Dataset (RDD)

Spark's driver programs access Spark through a SparkContext object, which represents a connection to a computing cluster. A driver program typically manages a few Worker Node processes, also known as Executors, which are launched at the beginning of a Spark application, and typically run for the entire lifetime of an application.