# Hyperparameter Optimization Using Successive Halving with Greedy Cross Validation
## 3 Results
### Experiment Results: Quality of Chosen Models
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & **ML Algorithm** & \multicolumn{2}{c}{**Greedy Successive Halving**} & \multicolumn{2}{c}{**Standard Successive Halving**} \\  & **Algorithm** & \(k\) = 5 & \(k\) = 10 & \(k\) = 5 & \(k\) = 10 \\ \hline \multirow{3}{*}{California Housing} & Passive Aggressive Tweedie & 1.023 & 1.016 & 1.010 & 1.008 \\  & Tweedie & 1.020 & 1.018 & 1.004 & 1.002 \\ \hline \multirow{3}{*}{Diabetes} & Passive Aggressive Tweedie & 1.049 & 1.068 & 1.030 & 1.039 \\  & Tweedie & 1.054 & 1.035 & 1.031 & 1.025 \\ \hline \multirow{3}{*}{Wine Recognition} & Bernoulli & 0.961 & 0.971 & 0.974 & 0.977 \\  & Naive Bayes & 0.912 & 0.936 & 0.926 & 0.936 \\ \cline{1-1}  & Deep Neural Network & 0.968 & 0.983 & 0.973 & 0.983 \\ \hline \multirow{3}{*}{Wisconsin Diagnostic Breast Cancer} & Bernoulli & 0.992 & 0.992 & 0.994 & 0.993 \\ \cline{1-1}  & Naive Bayes & 0.974 & 0.975 & 0.974 & 0.977 \\ \cline{1-1}  & Deep Neural Network & 0.989 & 0.991 & 0.988 & 0.993 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Mean standardized quality of final ML models chosen from a set of 500 candidate models, relative to ground truth optimal model. experimental condition in which a decision tree was used in conjunction with 1000 different candidate models and 10 cross-validation folds to classify cases for the Wisconsin Diagnostic Breast Cancer dataset. In this rare exception, the standard successive halving algorithm chose final ML models whose classification accuracy was, on average, 0.5% closer to the classification accuracy of the ground truth optimal model than the final models chosen by the greedy successive halving algorithm, with this difference being statistically significant at the \(p<0.01\) level. In all other cases, greedy successive halving was observed to perform identically with standard successive halving in terms of its ability to select high-performing ML models. Finally, it is worth noting that neither the standard successive halving algorithm nor the greedy successive halving algorithm will choose the ground truth optimal model on average. Among the 24 experimental conditions involving regression problems, both the greedy and standard successive halving algorithms were observed to select final ML models whose mean absolute errors were slightly higher on average than the true optimal ML model within the set of available candidate models. Similarly, among the 36 experimental conditions involving classification problems, both the greedy and standard successive halving algorithms were observed to select final ML models whose classification accuracies were slightly lower on average than the true optimal ML model within the set of available candidate models. For both greedy and standard successive halving, then, the choice to use a successive halving algorithm to perform hyperparameter optimization rather than relying on an exhaustive search represents a tradeoff in which substantial gains in search speed are realized in exchange for an outcome that will, on average, be slightly sub-optimal.