# Hyperparameter Optimization Using Successive Halving with Greedy Cross Validation
## 2 Materials and Methods
### The Greedy Successive Halving Algorithm
For the current study, the halving factor was set equal to 3, which matches the default value used in scikit-learn's implementation of the successive halving algorithm [15]. \[N_{iter}=\left\lfloor\log_{h}(N_{max}/N_{min})\right\rfloor+1 \tag{1}\]  
Once the number of successive halving iterations is known, the number of ML models to use and the number of training cases to use for any iteration can be easily calculated using exponential functions of the form \(y=a\cdot e^{x\cdot b}\). Specifically, for any iteration \(i\) (where \(i=0\) for the first iteration), the number of ML models that are needed for the next iteration (\(N_{models}\)) can be determined by Equation (2), while the number of training cases to use for the current iteration (\(N_{cases}\)) can be determined by Equation (3). \[ \begin{array}{l}N_{models}=n(M)*e^{-(i+1)\cdot b_{models}}\\ \text{where :}\\ M=\text{the set of candidate models}\\ n(M)=\text{the cardinality of }M\\ b_{models}=\frac{\ln(2\ /\ n(M))}{-N_{intr}+1}\\ N_{cases}=N_{min}*e^{i\cdot b_{cases}}\\ \text{where :}\\ b_{cases}=\frac{\ln(N_{max}\ /\ N_{min})}{N_{intr}-1}\\ \end{array}  \tag{2}\]  
As soon as the number of training cases to use for the current iteration (\(N_{cases}\)) is known (per Equation (3)), a random sample of \(N_{cases}\) training cases can be drawn from the complete set of training data, which can then be randomly subdivided into \(k\) folds. Next, Equation (2) can be used to calculate the number of candidate ML models (\(N_{models}\)) that will be needed as input for the next iteration. If the current iteration also happens to be the algorithm's final iteration, then \(N_{models}\) will naturally be equal to 1, indicating that the single, best-performing model is to be returned. Once the value of \(N_{models}\) is known, the evaluation of the candidate models for the current iteration can begin. It is at this point that the greedy successive halving algorithm diverges from the standard successive halving algorithm. With standard successive halving, the next step would be to completely evaluate the performance of every remaining candidate ML model by using standard cross validation. After the overall performance of every remaining candidate ML model has been identified, the standard successive halving algorithm would select the \(N_{models}\) best-performing models, which would subsequently be advanced to the next iteration [11]. Any models that did not perform sufficiently well to survive to the next iteration would be discarded. Note that with standard successive halving, every remaining candidate model must be fully evaluated before the algorithm can proceed to the next iteration. In contrast to standard successive halving, the greedy successive halving algorithm does not require every remaining candidate ML model to be fully evaluated before it is able to proceed to the next iteration. Instead, greedy successive halving is able to advance to the next iteration as soon as it has fully evaluated just \(N_{models}\) of the remaining candidate models. This key advantage is attributable to greedy successive halving's use of greedy cross validation, which automatically prioritizes the evaluation of the most promising ML models. A complete description of the proposed greedy successive halving algorithm is provided in Algorithm 1 below.