# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
Spark-Bench, which contains a comprehensive set of benchmark workloads for applications in machine learning, graph computation, SQL queries, and streaming application, is a Spark specific benchmarking suite proposed by [PERSON] et al. from IBM TJ Watson Research Center as a standard benchmark suite for Apache Spark [25; 26; 27]. The Terasort component among many others not mentioned in [26; 27] was later added to the updated Spark-Bench on GitHub [28]. We conducted the Spark-Bench Terasort tests on a real 24-node homogeneous Hadoop cluster running Cloudera CDH-5.6 YARN with Spark 1.5.0. The cluster has 2 racks of 12 nodes each. HDFS storage is 261 TB (Raw), and 80 TB (usable after replication overhead). The NameNodes are VM of 4 cores and 24 GB of RAM, each running on Intel Xeon E5-2690 physical hosts of 8 cores and 16 threads,2.9 GHz base frequency and 3.8 GHz max turbo frequency, and TDP of 135 W. The DataNodes are physical systems running Intel Xeon E3-1240 v3, 3.4 GHz base frequency and 3.8 GHz max turbo frequency, and TDP of 80 W. Each node has 4 cores, 8 threads, 32 GB of RAM, two 6 TB hard disks and 1 Gbit network bandwidth. All nodes are connected to a switch with a backplane speed of 48 Gbps. ### Spark Configuration in the Experiment

For Spark running in YARN cluster mode, the _-nun-executors_ option to the Spark YARN client controls how many executors it will allocate on the cluster (as _spark.executor.instances_ configuration property), while _-executor-memory_ (as _spark.executor.memory_ configuration property) and _-executor-cores_ (as _spark.executor.cores_ configuration property) control the resources per executor. When the _-nun-executors_ option is specifically assigned to a certain number of executors at runtime, it supersedes the _spark.dynamicAllocation.enabled_ configuration which is then disabled for that job submission. For the three 10 GB, 100 GB, and 1 TB workloads, we first ran Spark-Bench Terasort with DRA enabled, and then ran Spark Bench Terasort again over a dozen times to have enough sampled data points for plotting a smooth curve, each time with a different _-nun-executors_ number specifically assigned through the Spark option SPARK_EXECTOR_INSTANES = (_-nun-executors_) in Spark-Bench env.sh settings, which automatically turned off dynamic allocation. Spark is designed for Big Data processing, but its version 1.5.0 used in our experiment is unstable when we run Spark-Bench Terasort of 1 TB of data on our 24 nodes homogeneous Hadoop cluster. To process heavy workloads in Spark, we need to occasionally maximize disk and network utilization which can cause timeout exceptions in many operations in Spark such as RPC timeout, heartbeat timeout, connection timeout, acknowledgement wait, storage blockManagerSlave timeout, and storage blockManagerMaster timeout. We find that the default value of RPC communication timeout set to 120 s is clearly insufficient for a heavy 1 TB workload processed by Spark 1.5.0 with DRA enabled. To get Spark-Bench Terasort to complete the job, we increase the timeouts by a large margin in setting the SPARK_RPC_ASKTIMEOUT = 800 s and adding the umbrella SPARK_NETWORK_TIMEOUT = 800 s to cover all timeout exceptions in Spark for heavy workloads. On the other hand, there is no problem running Spark-Bench Terasort on a 1 TB dataset by manually assigning a static Spark's _-nun-executors_ without changing any default settings of timeouts in Spark. It appears that the DRA mechanism generates more disk and network utilization overhead in constantly monitoring, ramping up, and releasing executor resources, and therefore, causes timeout problems in processing heavy workloads. Spark's inherent timeout errors in processing heavy workloads have been reported by many users, especially for Spark versions older than version 1.6.0. Even on Spark version 1.6, [PERSON] et al.