# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
[29], in their evaluation of major big data resource management systems in the context of cloud computing environment, reported that their experimentations crashed on several occasions when the dataset was larger than 500 GB. Spark is memory intensive and all operations take place in memory. As such, it has limitations in handling very large dataset applications, and may crash when memory is no longer available for further operations. The authors in [29] confirm that before the release of version 1.5, Spark was not capable of handling datasets larger than the size of RAM and the problem of handling larger dataset persists in the newer releases with different performance overheads. Generally, Spark's performance could easily be improved by just adding more cores and more memory. By increasing the number of cores per executor, we can force an increase in percentage of utilization and a reduction in execution time. To max out the cluster, we can start with evenly-divided memory and cores [30]. For a node with 32 GB of RAM, using YARN's _DefaultResourceCalculator_ setup, which only takes the available memory into account when doing its calculation, the default resource allocation can be potentially up to 32 containers per node. However, each Worker Node in our 24-node Hadoop cluster has only 8 vCores as available logical cores from the 4 physical cores with hyperthreading, which gives 8 threads. Although each node has 32 GB of RAM and YARN's default container allocation size is 1 GB, we are restricted to running only 8 containers of 1 vCore each per node under YARN's _DominantResourceCalculator_ setup to prevent overutilization of CPU resources [31]. Following the same approach for CPU base resource allocation, Spark does its own task scheduling and process execution independent of YARN. Its configured Spark driver and executor cannot be larger than the size of a YARN container. And the total resources requested in terms of CPU and memory cannot exceed the available resources on the cluster for a user's job. In YARN cluster mode, the ApplicationMaster as a non-executor container runs the Spark driver which takes up its own resources assigned through the _-driver-memory_ and _-driver-cores_ properties. Thus, the available resources are determined by the following equations [32]:

Total vCores available >= _spark.executor.cores_\(\times\)_spark.executor.instances_\(+\)1 core for driver

Total Memory available >= (_spark.executor.memory_\(+\)executor memory overhead)

\(\times\)_spark.executor.instances_\(+\)_spark.driver.memory_\(+\)_drivermemoryoverhead_

In our experiment, we set the _spark.executor.memory_\(=\)1 GB and the _spark.executor.cores_\(=\)1 to have a total of 192 executors with 8 executors per node. The total memory used by 24 nodes is equal to only 192 GB out of 768 GB of available RAM. As such, we can optionally increase the heap size up to 3 GB without affecting the number of executors per node. In general, a single executor should not have more than 64 GB of RAM to avoid excessive garbage collection delays. We notice that with a single core executor, the overall Spark-Bench Terasort performance result in our experiment is quite slow, since the benefits of running multiple tasks in a single JVM have been eliminated by the single core executor configuration. To fine tune Spark's performance, we can increase the number of cores per executor to raise the percentage of CPU utilization at the expense of the total available number of executor instances. Ideally, to achieve full write throughput, up to 5 vCores should be assigned to each executor to support up to 5 concurrent tasks since beyond that, HDFS client could have trouble with too many concurrent threads [33]. [PERSON] et al. [26] observed in their Spark-Bench experiment that the workload execution time decreased as the number of vCores increases up to 6 cores, since Spark could launch more tasks concurrently. Nevertheless, overcommitting CPU resources, as in assigning 8 vCores, could lead to a performance degradation due to resource contention.