# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method  
The innovative Best Trade-off Point (BToP) method helps us find the exact optimal number of executor resources for a workload as the best trade-off point between performance and computing resources on a runtime elbow curve fitted from sampled executions of the target cluster. In addition to the cluster-computing frameworks Hadoop and Spark, the proposed BToP techniques could be used in a large variety of systems and applications which utilize a trade-off curve as a powerful tool for making informed decisions [5]. This paper will evaluate only the current most prominent cluster-computing framework, Apache Spark, running on YARN for efficiency in resource provisioning. The seven sequential steps for implementing the proposed BToP method on a cluster-computing system are as follows (Figure 1):  
1. Complete the configuration and fine tuning of the architecture, software, and hardware of the production cluster-computing system targeted for calibration. 2. Collect necessary preview job performance data from historical runtime performances or sampled executions on the same target production system, configured exactly as in step 1, as reference points for each workload. 3. Curve-fit the preview data to obtain the fit parameters \(a\) and \(b\) in the runtime elbow curve function \(f(x)=(a/x)+b\), where \(x\) is the number of executor resources. 4. Input the fit parameter \(a\) to the BToP algorithm to obtain the recommended optimal number of executors for a workload (Algorithms 1 and 2). 1. The algorithm computes the number of executors over a range of slopes from the first derivative of \(f(x)=(a/x)+b\) and the acceleration over a range of slopes from the second derivative (Algorithms 1 and 2). 2. The algorithm applies the Chain Rule to search for break points and major plateaus on the graphs of acceleration, slope, and executor resources over a range of incremental changes in acceleration per slope increment (Algorithms 1 and 2). 3. The algorithm extracts the exact number of executors at the best trade-off point on the elbow curve and outputs it as the recommended optimal number of executors for a workload (Algorithms 1 and 2). * Repeat steps 2-4 to gather enough resource provisioning data points for different workloads to build a database of resource consumption signatures for subsequent job profiling. * Repeat steps 1-5 to recalibrate the database of resource consumption signatures if there are any major changes to step 1. * Use the database of resource consumption signatures to match dynamically submitted production jobs to their recommended optimal number of executors for efficient resource provisioning. The crux of the proposed BToP method is not tied to any specific system. These seven steps for implementing the BToP method and algorithm for efficient resource provisioning could also be used with other types of software components or data processing engines in the Hadoop ecosystem, any computing system, network data routing system, cluster microarchitecture system, payload engine system including but not limited to vehicle, aircraft/plane/jet, boat/ship, and rocket. This method could also be used with the yield curve of various types of securities, the convex iso yield curve, also known as convex isoquant curve, and the indifference curve in economics and manufacturing such as the semiconductor/IC yield curve, the manufacturing quality control curve, and any other types of trade-off curves with an elbow shape, non-inverted or inverted, for making intelligent decisions. In general, the proposed Best Trade-off Point method can be used in any other types of applications which rely on an elbow curve \(f(x)=(a/x)+b\) or an inverted elbow curve \(f(x)=-(a/x)+b\) of performance vs. resources for good decision making including efficient resource provisioning. An elbow curve is a form of an exponential function with a negative growth rate.