# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
For the BToP method, which is based on behavioral replication to effectively match an optimal static number of executors from the database of resource consumption signatures with its corresponding workload, we can directly compute the actual energy consumption of each specific workload. Since the actual executor resources allocated and utilized by a workload constantly change in Spark with DRA enabled, it would be difficult to ascertain an apparent fluctuating power consumption. However, for a heavy workload of 1 TB, we observe from the logged output files that the 119 to 120 executors, as the maximum number of executors for DRA, are fully utilized most of the time in Spark-Bench Terasort execution. Moreover, even if the maximum number of single task executor resources as single thread processes were not fully used, they would not be released until the long running application has finished. So, it is reasonable to use the maximum number of executors assigned in DRA for the heavy workload of 1 TB to estimate its energy consumption. For Spark-Bench Terasort of 1 TB, we compare the estimated energy saving between the maximum number of executors utilized in DRA (120 executors equivalent to 15 nodes) and the highest number of executors recommended by the BToP algorithm (99 executors equivalent 13 nodes) based on a 24-node Spark YARN cluster with a maximum of 8 executors per node. For an active power consumption per node of 250 W, idle power of 235 W, and an average job arrival time of 3000 s, we have the following energy consumptions results:

\[E(15)=[2923.14\text{ s}\times(250\text{ W}\times 15)]+[(3000\text{ s}-2923.14\text{ s})\times 235\text{ W}]\] \[=10,979,37.1\text{ j}=3049.95\text{ Wh per job}\]

\[E(13)=[2608.56\text{ s}\times(250\text{ W}\times 13)]+[(3000\text{ s}-2608.56\text{ s})\times 235\text{ W}]\] \[=8,569,808.4\text{ j}=2380.5\text{ Wh per job}\]

Hence, by provisioning executor resources with the BToP method, we reduce the energy consumption by about 21.95%. This translates to (0.66945 kWh saved per job) \(\times\) [((365 \(\times\) 24) h/year/((3000/3600) h/job)] =7037.26 kWh saved per year. According to the US Department of Energy, the May 2017 average retail price of electricity for commercial customers in California was $0.1493 per kWh. Thus, the annual energy saving amounts to $1050.66 for the given 1 TB computing job with arrival time of 3000 s [39] (Table 2). From the output files of Spark-Bench Terasort of 10 GB and 100 GB, we observe that Spark with DRA enabled frequently ramps up and down within the range of 0 to 80 and 120 executors, respectively, according to the need of each stage of the jobs. However, the fluctuating resource utilization rarely reached the maximum available number of executors allocated for DRA in both 10 GB and 100 GB workloads. As previously rationalized for the 1 TB workload, since their estimated energy consumptions would be inconsistent if they are modeled after a moving target, we use their maximum numbers of executors assigned for DRA, 80 executors for 10 GB workload and 120 executors for 100 GB workload, as ceiling values to calculate their energy consumption for the worst case. As such, the energy consumption for Spark with DRA enabled could typically be slightly less than the estimated worst-case values. Be that as it may, using Spark with the BToP method instead of Spark with DRA enabled for Spark-Bench Terasort of 10 GB and 100 GB workloads with an average job arrival time of 3000 s will result in an energy efficiency of 52.71% per job and 57.11% per job, respectively. That is equivalent to an annual energy saving of $329.57 for the given 10 GB computing job and $690.66 for the given 100 GB computing job (Table 2). ## 6 Related Work

There have been many studies on resource allocation in Spark to improve its performance. But our BToP method is the first to prove that there is still room for further optimization of resource provisioning at the executor level in Apache Spark, despite its sophisticated built-in DRA mechanism.