# Hyperparameter Optimization Using Successive Halving with Greedy Cross Validation
## 3 Results
### Experiment Results: Wall-Clock Time  
A necessary first step in comparing the performance of greedy successive halving versus that of standard successive halving was to assess how quickly the competing algorithms were able to complete the task of selecting a final ML model from among a set of candidate models. Tables 3-5 below summarize the average wall-clock time required by each algorithm to complete the ML model selection task for sets of 250, 500, and 1000 candidate models, respectively. As noted in Section 2, each value reported in these tables has been standardized as a proportion of the corresponding wall-clock time required to complete a naive, exhaustive search, and reflects the mean of 30 different trials for the specified experimental condition. The probability values reported in the tables originate from [PERSON]'s \(t\)-tests that directly compare the standardized wall-clock times required by the competing algorithms under identical experimental conditions. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**ML Algorithm**} & \multicolumn{2}{c}{**Greedy Successive**} & \multicolumn{2}{c}{**Standard Successive**} \\  & & \multicolumn{2}{c}{**Halving**} & \multicolumn{2}{c}{**Halving**} \\  & & \multicolumn{2}{c}{**\(k\) = 5**} & \multicolumn{2}{c}{\(k\) = 10**} & \multicolumn{2}{c}{\(k\) = 5} & \multicolumn{2}{c}{\(k\) = 10**} \\ \hline California & Passive Aggressive & 0.041 *** & 0.041 *** & 0.069 & 0.077 \\ Housing & Tweedie & 0.112 *** & 0.093 *** & 0.182 & 0.167 \\ \hline \multirow{2}{*}{Diabetes} & Passive Aggressive & 0.242 *** & 0.128 *** & 0.678 & 0.640 \\  & Tweedie & 0.341 *** & 0.156 *** & 1.056 & 0.893 \\ \hline \multirow{3}{*}{Wine Recognition} & Bernoulli Naive & \multirow{3}{*}{0.260 ***} & \multirow{3}{*}{0.187 ***} & \multirow{3}{*}{0.957} & \multirow{3}{*}{0.972} \\  & Bayes & & & & \\ \cline{1-1} Recognition & Decision Tree & 0.251 *** & 0.166 *** & 0.839 & 0.857 \\ \cline{1-1}  & Deep Neural & \multirow{3}{*}{0.120 ***} & \multirow{3}{*}{0.103 ***} & \multirow{3}{*}{0.470} & \multirow{3}{*}{0.551} \\  & Network & & & & \\ \hline \multirow{3}{*}{Wisconsin Diagnostic} & Bernoulli Naive & \multirow{3}{*}{0.359 ***} & \multirow{3}{*}{0.284 ***} & \multirow{3}{*}{0.891} & \multirow{3}{*}{0.925} \\  & Bayes & & & & \\ \cline{1-1} \cline{4-4} \cline{6-6} \cline{5-6}  
\(p<0.001\) for a [PERSON]'s \(t\)-test comparing wall-clock times for the greedy and standard successive halving algorithms. As shown above in Tables 3-5, the greedy successive halving algorithm was consistently able to complete the ML model selection task much more quickly than the standard successive algorithm, regardless of the dataset that was used in the experiment, the type of machine learning algorithm, the number of candidate models being evaluated, or the number of folds used for cross validation. Put differently, the results reported in Tables 3-5 indicate that greedy successive halving required less wall-clock time to complete the ML model selection task than standard success halving in all 60 of the experimental conditions considered in this study, with the superior performance of greedy successive halving being statistically significant at the \(p<0.001\) level in every case. The evidence obtained from the experiments thus strongly suggests that greedy successive halving will be significantly faster than standard successive halving when performing hyperparameter optimization