# Hyperparameter Optimization Using Successive Halving with Greedy Cross Validation
## 2 Materials and Methods
### Evaluative Experiments
If the greedy successive halving algorithm could be shown to select ML models of statistically comparable quality to those selected by the standard successive halving algorithm while requiring less wall-clock time to do so, then it could be reasonably concluded that the greedy successive halving algorithm is superior to the standard successive halving algorithm. Among the classifiers, the performance of the final ML model chosen by each of the competing successive halving algorithms was measured in terms of classification accuracy, while among the regressors, the performance of the final model chosen by the competing successive halving algorithms was measured in terms of the mean absolute error (MAE) of the model's predictions. For each experiment, a naive, exhaustive search of all of the experiment's candidate models was first performed using standard \(k\)-fold cross validation without successive halving. Since an exhaustive search evaluates every possible candidate model using the complete set of training data for a given experiment, the results of the exhaustive search established a baseline wall-clock processing time for each experiment, as \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Dataset** & **Instances** & **Features** & **Target** \\ \hline California Housing & 20,640 & 8 & Real \\ \hline Diabetes & 442 & 10 & Real \\ \hline Wine Recognition & 178 & 13 & Three Classes \\ \hline Wisconsin Diagnostic Breast Cancer & 569 & 29 & Two Classes \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Characteristics of datasets used in the evaluative experiments. well as a baseline performance value for the ground truth optimal model among the set of candidate models considered during the experiment. The mean performance values for the ground truth optimal models identified during the experiments are provided in Table 2 below. Note that the values reported for the California Housing and Diabetes datasets indicate the average MAE for the corresponding experiments' ground truth optimal models, while the values reported for the Wine Recognition and Wisconsin Diagnostic Breast Cancer datasets indicate the average classification accuracy for the corresponding experiments' ground truth optimal models. To better contextualize the MAE values reported in the table, it is also worthwhile to note that the raw values for the target variable in the California Housing dataset ranged from 0.15 to 5.0, while the raw values for the target variable in the Diabetes dataset ranged from 25 to 346. After all of the experiments were complete, both the performance of the final models chosen by each variant of the successive halving algorithm and the wall-clock time required by each variant were standardized as percentages of their corresponding baseline values. For example, a standardized performance value of 1.02 for a regressor model would indicate that the mean absolute error for the chosen model was 2% greater than the ground truth optimal model, while a standardized performance value of 0.98 for a classifier model would indicate that the classification accuracy of the chosen model was 2% less than the ground truth optimal model. Similarly, a standardized time of 0.30 would indicate that a successive halving algorithm required only 30% as much wall-clock time as a corresponding exhaustive search of the same set of candidate models. Using this approach allowed the performance of the competing algorithms to be compared across experimental conditions in a statistically valid way. Finally, the standardized wall-clock times and model performance values for the standard and greedy successive halving algorithms were compared against each other using [PERSON]'s \(t\)-tests [27]. Unlike most other \(t\)-tests, [PERSON]'s \(t\)-tests allow the independent samples being compared to have unequal variances.