# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
Each Executor represents a unique resource unit in Spark, which is a combination of a certain number of CPU cores and memory, as specified through _spark.executor.cores_ and _spark.executor.memory_, for requesting resources from the cluster manager (Figure 2). Figure 2: Spark Architecture in Cluster Mode. The number of CPU cores occupied by each task is defined through _spark.task.cpus_ as the smallest running unit in Spark execution layer. The number of required executors is derived from the number of task execution units. SparkContext is used to build an RDD, which is Spark's fundamental abstraction for distributing data and computation, and for running parallel operations. RDDs are immutable collections of objects supporting in-memory data storage distributed across clusters. Spark's efficiency is achieved through parallelization of processing across multiple cluster nodes and minimization of data replication between those nodes. Its fault-tolerance is achieved in part by logging the lineage of transformations applied to coarse-grained sets of data for recovery when needed. In case of data loss or node failure, an RDD will have sufficient information of how it was derived from other RDDs to recover the lost partition without any costly replication [21; 22; 23]. The Spark programming model is based on two types of high order functions which are transformations to create new RDD and actions to compute results without changing the original data. Transformations are lazily evaluated, since they are not executed until a subsequent action has the need for the result. This lazy operator approach further improves performance by avoiding unnecessary data processing when result is not subsequently needed. However, it can also introduce processing bottlenecks and cause application stall. For better cluster performance, RDDs persist in memory whenever possible. Spark jobs often comprise multiple sequential stages. When an action is called on an RDD, a Directed Acyclic Graph (DAG) of stages is built from the RDD lineage graph at the low level, and submitted to the DAG scheduler. Operators are divided into stages which contain pipelined transformations with narrow dependencies. Each stage contains some tasks based on the partition of the input data with one task for every partition. A task is a combination of data and computation which is assigned to an executor pool thread. The stages are passed on to the Task Scheduler which launches tasks through cluster manager [24]. ### Distributed Execution in Spark

Although Spark starts up in YARN, it does all task scheduling, task forming, and process execution independent of YARN. The Spark driver runs inside of YARN's ApplicationMaster, which is responsible for driving the application and requesting resources from YARN. Spark job runs as a Java process on a JVM. Typically, Spark program execution starts by calling sc.textFile() to create input RDDs from external data. New RDDs are then defined by using narrow transformations such as filter(), map(), and union(), which does not require data to be shuffled across the partitions, or wide transformations such as groupByKey() and reduceByKey(), which does require data to be shuffled. Any intermediate RDDs which will need to be reused are kept in memory through persist(). The sequence of commands creates an RDD lineage, a DAG of RDDs, which could later be used in a called action. Output results are obtained by launch actions, such as reduce(), collect(), count(), first(), and take(), on RDDs to begin parallel computation, which is then optimized and executed by Spark. A lot of Spark's API revolves around passing functions to its operators to run them on the cluster. Spark automatically takes the user's function and distributes it to executors. A user's code in a single driver program is automatically distributed for execution on multiple cluster nodes. All work is expressed in either creating new RDDs, transforming existing RDDs, or calling operations on RDDs to compute a result [18; 21; 22; 23].