# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
_Step 3._ Curve-fit the preview data to obtain the fit parameters \(a\) and \(b\) in the runtime elbow curve function \(f(x)=(a/x)+b\), where \(x\) is the number of executor resources (Figure 4). The best trade-off point on the runtime elbow curve should be the location where no further significant decrease in job duration could be obtained by continuing to increase the number of executors. Since the rate of descending of the job duration is the downhill slope of the graph, the target point could be found in the area where the slope is gentle and no longer steep, and the vertical movement is nearly flat. To find the slope, we take the derivative of the polynomial function

\[f(x)=\left(\frac{a}{x}\right)+b \tag{1}\]

where \(x\) is the number of executors. The derivative of \(f(x)\) is a slope of a tangent line at a point \(x\) on a graph \(f(x)\). It is equivalent to the slope of a secant line between two points \(x\) and \(x+\Delta x\) on the graph, where \(\Delta x\) approaches 0. \[f^{\prime}(x)=\lim_{\Delta x\to 0}\frac{[f(x+\Delta x)-f(x)]}{ \Delta x} \tag{2}\]

Figure 4: Fitting Spark-Bench Terasort preview data points for 1 TB to the function \(f(x)=(a/x)+b\). From (1),

\[f^{\prime}(x)=-ax^{-2} \tag{3}\]

and therefore,

\[x=\sqrt{-a/f^{\prime}(x)} \tag{4}\]

where \(f^{\prime}(x)<0\) for a downhill slope with a negative value. Using GNUplot [34] to curve-fit the preview data points, we obtain the fit parameters \(a\) and \(b\) of the graph function \(f(x)=(a/x)+b\). GNUplot fit command uses Levenberg-Marquardt Algorithm (LMA), also known as the damped least-squares (DLS) method, which is used to solve non-linear least squares problems. LMA interpolates between the Gauss-Newton algorithm (GNA) and the method of gradient descent. However, LMA is more robust than the Gauss-Neuton algorithm since, in many cases, it could find a solution even if it starts very far off the final minimum. The GNUplot fit command is used to find a set of parameters that best fits the input data to the user-defined function, which is \(f(x)=(a/x)+b\) here. The fit is judged based on the Sum of Squared Residuals (SSR) between the input data and the function values, evaluated at the same places on the curve. LMA will try to minimize the weighted SSR or chsquare. A reduced chsquare much larger than 1.0 may be caused by incorrect data error estimates, data errors not normally distributed, systematic measurement errors, 'outliers', or an incorrect model function. The parameter error estimates, which are readily obtained from the variance-covariance matrix after the final iteration, is reported as \"asymptotic standard errors\" (Figure 4). With the obtained fit parameter \(a\), we then plot the three fitted elbow curves of job duration vs. executors for Spark-Bench Terasort 10 GB, 100 GB, and 1 TB workloads (Figure 5). The resulting performances of Spark with DRA enabled for all three workloads indicate that Spark's DRA mechanism is already quite effective in achieving generally good performance. As such, it appears initially from the graphs that there might be no more room for further improvement in running Spark with the BToP method which disables the built-in DRA.