# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 1 Introduction  
The Gartner Inc. research firm has forecasted that the rapidly-growing cloud ecosystem will have up to 25 billion IoT sensor devices connected by 2020 [1]. This large number of devices will generate hundreds of zettabytes of information in the cloud to be analyzed by Big Data processing engines, such as Hadoop MapReduce and Apache Spark, to deliver practical value in business, technology, and manufacturing processes for better innovation and more intelligent decisions. In such an era of exponential growth in Big Data, energy efficiency has become an important issue for the ubiquitous Hadoop and Spark ecosystems. It is now established that the energy consumption cost of a server over its useful life has far exceeded its original capital expenditure [2]. Gartner estimated in a July 2016 report that Google at the time had 2.5 million servers and counting [3]. As such, the energy consumption associated with datacenters has become a major concern, as more companies have more datacenters with over a million servers. In fact, datacenter electricity consumption is projected to increase to roughly 140 billion kilowatt-hours annually by 2020, the equivalent annual output of 50 power plants, costing American businesses $13 billion annually in electricity bills, and emitting nearly 100 million metric tons of carbon pollution per year [4]. This energy expense, largely incurred for Big Data processing, could be reduced through more efficient resource provisioning in MapReduce and Spark among other data processing frameworks. We previously proposed the innovative Best Trade-off Point (BToP) method and algorithm for obtaining the best trade-off between performance and computing resources for energy efficiency in anyworkload running on Hadoop MapReduce [5]. Since then, we have further explored more software modules and data processing engines running on the Hadoop ecosystem including Apache Spark for similar or different approaches to optimize resource provisioning for performance efficiency. In this paper, we address the same question of how to allocate the optimal number of executor resources for a workload in Apache Spark. Since Spark is already quite efficient in resource utilization with its built-in Dynamic Resource Allocation (DRA) mechanism, we initially did not expect to gain any further performance improvement by applying the proposed BToP method to Spark. However, our Spark-Bench Terasort experimental results prove that Spark using the BToP method could still outperform Spark with DRA enabled, particularly in production environments, where job profiling for behavioral replication will lead to the most optimal resource provisioning. In the MapReduce programming model, multiple MapReduce jobs must be strung together to create a data pipeline with data read from disk in between every stage of the pipeline and written back to disk when completed. Apache Spark effectively minimizes these excessive disk I/O bottlenecks by keeping all activities in memory whenever possible. The performance difference between Hadoop and Spark has been evaluated by many researchers. [PERSON] et al. [6] tested the performances of Hadoop and Spark using Wordcount dataset of various sizes ranging from 287 KB to 5.5 GB and observed that Spark's performance was better than Hadoop. However, as the dataset size grew, the running times grew very quickly, even faster than a power function, and speedup decreased significantly. [PERSON] et al. [7] confirmed that the Wordcount speedup of Spark over Hadoop was influenced by the input data size. It increased when moving from 1 GB to 5 GB and decreased when moving from 5 GB to 10 GB. The authors in [7] conducted High-Bench benchmarks and found that Spark was more efficient than Hadoop in dealing with a large amount of data in major cases. Unlike Hadoop, which stores the data on disk after each map or reduce action, Spark processes data in-memory based on RDDs.