# Hyperparameter Optimization Using Successive Halving with Greedy Cross Validation
###### Abstract  
Training and evaluating the performance of many competing Artificial Intelligence (AI)/ Machine Learning (ML) models can be very time-consuming and expensive. Furthermore, the costs associated with this hyperparameter optimization task grow exponentially when cross validation is used during the model selection process. Finding ways of quickly identifying high-performing models when conducting hyperparameter optimization with cross validation is hence an important problem in AI/ML research. Among the proposed methods of accelerating hyperparameter optimization, successive halving has emerged as a popular, state-of-the-art early stopping algorithm. Concurrently, recent work on cross validation has yielded a greedy cross validation algorithm that prioritizes the most promising candidate AI/ML models during the early stages of the model selection process. The current paper proposes a greedy successive halving algorithm in which greedy cross validation is integrated into successive halving. An extensive series of experiments is then conducted to evaluate the comparative performance of the proposed greedy successive halving algorithm. The results show that the quality of the AI/ML models selected by the greedy successive halving algorithm is statistically identical to those selected by standard successive halving, but that greedy successive halving is typically more than 3.5 times faster than standard successive halving.  
hyperparameter optimization; successive halving; greedy cross validation; machine learning; artificial intelligence +
Footnote †: journal:  
definition  
The art and science of searching for the combination of hyperparameter settings that yields the best-performing ML model is known as _hyperparameter optimization_[1], and it is this activity that resides at the core of the current paper. Using hyperparameter optimization is standard practice when developing ML-based solutions since no single ML algorithm or set of hyperparameter values will perform best in all use cases or with all datasets [4; 5].  
When performed competently, hyperparameter optimization identifies an ML model that yields optimal or near-optimal performance. Such an outcome has many obvious benefits, particularly in competitive contexts such as business or national defense. Nevertheless, hyperparameter optimization is plagued by a set of ubiquitous problems. First, many hyperparameters are real-valued. This suggests that there are often an infinite number of possible models in a given ML scenario, which makes it mathematically impossible to conduct an exhaustive search for an optimal model. Although training and evaluating an infinite number of candidate models is not possible, it is feasible to consider a large, finite set of models. Using hyperparameter optimization to train and evaluate the comparative performance of hundreds or thousands of candidate ML models can, however, be very time-consuming and expensive, particularly in the era of big data wherein training just one model may take hours or days [6]. Finally, training and evaluating ML models commonly relies on cross validation (discussed later), which can exponentially increase the time and costs associated with hyperparameter optimization [7]. In light of these considerations, AI/ML practitioners often find themselves in an awkward situation: on the one hand, hyperparameter optimization must be used to find the best-performing model possible given the available budget, but on the other hand, hyperparameter optimization can be both very expensive and very time consuming, and no researcher or organization is blessed with an infinite budget. Finding methods of reducing the time and other costs associated with hyperparameter optimization is therefore of great interest to the AI/ML community [8].