# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
### Spark's Dynamic Resource Allocation (DRA)

Although pre-allocating a large enough number of resources for a workload could improve performance, it might lead to underutilization of cluster resources and starvation of other applications. In addition, unlike the MapReduce task, which resides in a process and is killed when the task is finished, the Spark task is a thread residing in a process known as executor, which is launched at the start of Spark application and is not killed until the application is finished. To solve these problems,an elastic resource scaling ability, also known as Dynamic Resource Allocation feature, has been added to Spark since version 1.2 as a built-in mechanism for acquiring and releasing executors during runtime according to the load of current application. This robust feature is particularly useful when multiple applications share resources in a Spark cluster. To enable DRA, we set _spark.dynamicAllocation.enabled_ to true, and enable _spark.shuffle.service.enabled_ for external shuffle data transmission to support dynamically-added and -removed executors. The minimum and maximum numbers of executors that should be allocated to an application can be specified through _spark.dynamicAllocation.minExecutors_ and _spark.dynamicAllocation.maxExecutors_. The initial number of executors can be set in the _spark.dynamicAllocation.initialExecutors_ parameter. The behavior of DRA can be further fine-tuned with several properties listed in Spark's official document for DRA [19]. To disable DRA for a Spark job without hard-coding the properties in the SparkConfig file, we specified the _-num-executors_ in Spark-Bench env.sh settings at time of job submission. Spark's _ExecutorAllocationManager_ calculates the maximum number of executors it requires through pending and running tasks. If the current target executor number exceeds the actual number which is needed, Spark stops adding new executors and notifies cluster manager to cancel any extra pending requests. If the new target executor number remains the same, Spark stops sending requests to the cluster manager. Unlike static allocation as in the BToP method, where all resources are provisioned at the start of the job execution, the DRA mechanism could request and remove resources dynamically during run-time. If the current executor number cannot satisfy the desired executor number, Spark updates the target number and boosts it with the number of executors to add for the first round which is doubled in each subsequent round. To acquire executors when they are needed and to relinquish executors when they are no longer used, Spark uses several different timeouts for its request and remove policies, respectively. The request for executors is triggered when there have been pending tasks for _spark.dynamicAllocation.schedulerBacklogTimeout_ seconds, and then triggered again every _spark.dynamicAllocation.sustainedSchedulerBacklogTimeout_ seconds thereafter if the queue of pending tasks persists. To efficiently deal with the common slow start and fast ramp-up in actual needs, the number of executors requested in each round increase exponentially, in the order of 1, 2, 4, 8, 16, and so on, from the previous round. Any unused executor is removed when it has been idle for more than _spark.dynamicAllocation.executorIdleTimeout_ seconds (the default value is 60 s) [19]. To handle multiple parallel jobs from separate threads which can run simultaneously inside a SparkContext instance, Spark's scheduler runs jobs in a FIFO fashion with an option to use a round robin fair scheduler to prevent large jobs at the head of the queue from significantly delayed, later jobs in the queue [18; 19]. ## 4 Spark-Bench Terasort Experiment with BToP Method

### Test System Specifications

To illustrate our BToP method for obtaining the optimal number of executors for different workloads, we use the Teragen and Terasort components of the Spark-Bench Terasort test to experiment with 10 GB, 100 GB, and 1 TB datasets.