# Best Trade-Off Point Method for Efficient Resource Provisioning in Spark
## 2 Best Trade-Off Point (BToP) Method
It is an exponential decay function \(f(x)\) = \((a/x)+b\) where \(f(x)\) approaches \(b\) instead of zero when \(x\) approaches infinity (Algorithm 1). On the other hand, an inverted elbow curve is a function with inverted exponential growth \(f(x)=-(a/x)+b\) where \(f(x)\) approaches \(b\) instead of infinity when \(x\) approaches infinity (Algorithm 2). ```
Input: Parameter \(a\) for a workload with runtime curve \(f(x)=(a/x)+b\) Output: Optimal number of executors foreachincremental slope valuedo output number of executors x=sqrt(-a/slope); end foreachincremental slope valuedo output absolute value of acceleration=2a/pow(slope, 3); end foreachincremental target value of change in accelerationdo foreachincremental slope valuedo ifchange in acceleration in the current slope increment is >= to the target value AND change in acceleration in the next slope increment is < the target valuethen output acceleration, slope, and number of executors; store number of tasks in an array register; break; end end foreachincremental target of change in accelerationdo ifnumber of executors do output number of executors as recommended optimal value; break; end end foreach
```
Input: Parameter \(a\) for a workload with runtime curve \(f(x)=-(a/x)+b\) output: Optimal number of executors foreachincremental slope valuedo  output number of executors x=sqrt(a/slope);  end foreachincremental slope valuedo  output absolute value of acceleration=2a/pow(slope, 3);  end foreachincremental target value of change in accelerationdo foreachincremental slope valuedo ifchange in acceleration in the current slope increment is >= to the target value AND change in acceleration in the next slope increment  is < the target valuethen  output acceleration, slope, and number of executors;  store number of tasks in an array register;  break;  end end foreachincremental target of change in accelerationdo ifnumber of executors do not change in 7 incrementsthen  output number of executors as recommended optimal value;  break;  end end
```
**Algorithm 2** Best Trade-off Point for an inverted elbow curve \(f(x)=-(a/x)+b\)

Typically, in Data Science applications, we could have an inverted elbow curve, also known as a knee curve (negative elbow curve), of performance vs. data, where increasing the size of data no longer improves performance beyond a certain point. Thus, the BToP algorithm for an inverted elbow curve could simply be derived from the BToP algorithm for a non-inverted elbow curve as shown in Algorithm 2. And the sequential steps of the BToP method will still be the same, with the exception that the elbow curve function is now inverted as \(f(x)=-(a/x)+b\). ```
Input: Parameter \(a\) for a workload with runtime curve \(f(x)=-(a/x)+b\) output: Optimal number of executors foreachincremental slope valuedo  output number of executors x=sqrt(a/slope);  end foreachincremental slope valuedo  output absolute value of acceleration=2a/pow(slope, 3);  end foreachincremental target value of change in accelerationdo foreachincremental slope valuedo ifchange in acceleration in the current slope increment is >= to the target value AND change in acceleration in the next slope increment  is < the target valuethen  output acceleration, slope, and number of executors;  store number of tasks in an array register;  break;  end end foreachincremental target of change in accelerationdo ifnumber of executors do not change in 7 incrementsthen  output number of executors as recommended optimal value;  break;  end end end
```

**Algorithm 3** Best Trade-off Point for an inverted elbow curve \(f(x)=-(a/x)+b\)

## 3 Resource Provisioning in Apache Spark

Our BToP method for optimal resource provisioning is expected to work for many other types of parallel processing frameworks running on Hadoop YARN beyond MapReduce. One such example is Apache Spark, which has recently gained momentum in terms of popularity for in-memory processing of Big Data analytic applications with better sorting performance for large clusters.