# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 1 Introduction
If the system has insufficient memory to support huge input size, the performance will become limited when Spark runs out of the memory it needs for the creation of new RDDs. As such, Spark performance can be seriously affected by not having enough memory and can even become slower than Hadoop. In their evaluation of several major architectural components in MapReduce and Spark frameworks, [PERSON] et al. [8] observed that for batch jobs, Spark was a better choice for smaller datasets, since MapReduce could be several times faster than Spark when experimenting with larger datasets. For Terasort of 1 GB input, Spark was faster overall than MapReduce, because Spark had lower control overhead than MapReduce. However, for Terasort of 100 GB and 500 GB inputs, MapReduce was 1.5 times and 1.8 times faster than Spark, respectively, since MapReduce had a more efficient execution model for shuffling data than Spark. For iterative operations in machine learning such as K-Means, Spark was 1.5 times faster than MapReduce in its first iteration, and more than 5 times faster in subsequent operations. RDD caching effectively reduced the CPU overhead for parsing text to objects which was often the bottleneck for each iteration. The experimental results showed that Spark was faster than MapReduce by approximately 2.5 times for Wordcount, and 5 times for both K-means and PageRank. [PERSON] and [PERSON] [9; 10] examined the iterative algorithms including PageRank, K-means, and logistic regression in five resource management frameworks including Hadoop and Spark, and found that Spark could reduce significant performance overheads, including network I/O, disk I/O, scheduling, and synchronization, which slowed down MapReduce, by using RDD caching and reusing it in MapReduce-like parallel operator to optimize iterative and interactive operations. In comparing the memory-enhanced and iterative-aware version of Flame-MR-It, a Java-based MapReduce framework that improves Hadoop performance without modifying its programming APIs, with Hadoop and Spark on Amazon EC2 cloud platform, [PERSON] et al. [11] proved that Flame-MR could reduce Hadoop execution times by up to 65% through memory management optimization while providing competitive results compared to Spark without modifying the source code of the applications, and also using less memory. Spark consumes 62% and 25% more memory than Flame-MR-It for PageRankand Connected Components, respectively. The performance of iterative applications is improved by the utilization of long-lived Workers, the avoidance of writing intermediate results to HDFS, and the implementation of data cache to maximize in-memory processing and minimize the use of disk. Although the emerging Spark with its high-speed in-memory computations does not replace Hadoop MapReduce for low latency data process, many interactive data mining, iterative algorithms of machine learning, and data streaming applications previously available on MapReduce have now become deprecated and substituted by equivalent applications on Spark. Apache Spark also has the functionality to process structured data in Hive and SQL, streaming data from Flume, Twitter, and HDFS, and graphs. Therefore, a large portion of datacenter workloads will eventually be processed by Apache Spark. Hence, we expect that there will be many more research groups and Big Data companies including Databricks [12], a company founded by the creators of Apache Spark, working on the improvement of Spark's performance efficiency and energy saving upon its maturity. Meanwhile, at the high level, many of the prior research works on Hadoop MapReduce resource provisioning to accomplish certain application performance goals through dynamic job profiling, as in [PERSON] [13], cluster size elasticizing, as in [PERSON] et al. [14], scaling past execution results to meet given Service Level Objectives (SLO), as in [PERSON] et al. [15], and matching an application's resource consumption with a database of similar signatures of other applications, as in [PERSON] et al.