Automatic Termination for Hyperparameter Optimization

1 INTRODUCTION . While the performance of machine learning algorithms crucially depends on their hyperparameters , setting them correctly is typically a tedious and expensive task . Hyperparameter optimization ( HPO ) emerged as a new sub-field in machine learning that tries to automatically determine how to configure a machine learning algorithm . One of the most successful strategies for HPO is Bayesian optimization ( BO ; Močkus , 1975 ; Chen et al. , 2018 ; Snoek et al. , 2012 ; Melis et al. , 2018 ) which iteratively trains a probabilistic model on the evaluations of the tuned algorithm . to select the most promising next candidate point that trades-off exploration and exploitation . In practice , the quality of the final solution found by BO heavily depends on a user defined budget , such as wall-clock time or the number of iterations , which needs to be defined in advanced . If this budget is too small , BO might return hyperparameters that result in a poor predictive performance . If the budget is too large , compute resources will be wasted and , in some cases , it may result in overfitting as we will show in our experiments . Automatically stopping BO is a rather under-explored topic in the literature . A simple baseline is to stop if BO has not found a better solution than the current best incumbent for some successive iterations , which is in the same vein as early stopping for neural network training . Another approach is to track probability of improvement ( Lorenz et al. , 2016 ) or expected improvement ( Nguyen et al. , 2017 ) , and stop the optimization process once it falls below a given threshold . However , determining this threshold may in practice be less intuitive than setting the number of iterations or the wall-clock time . Instead of stopping BO completely , McLeod et al . ( 2018 ) propose to switch to local optimization when the global regret is smaller than a pre-defined target . This condition can also be used to terminate BO early , but it comes with additional complexity such as identifying a ( convex ) region for local optimization and again a predefined budget . In this work , we propose a simple and interpretable automatic termination criterion for BO . In our criterion , we construct high-probability confidence bound on the regret ( i.e. , the difference of our current solution to the global optimum ) exploiting the probabilistic model of the objective . Thus , users are now asked to specify a desired tolerance that defines how accurate should the final solution be compared to global optimum . In addition , we propose to determine the threshold via a cross-validation estimate of the generalization error . This choice takes into account the irreducible discrepancy between the actual objective and the target function optimized via BO , namely the difference between the performance on new data ( i.e. , the population risk ) and the validation error . Our extensive empirical evaluation on a variety of HPO and NAS benchmarks suggests that our method is more robust and effective in maintaining the final solution quality than common baselines ( Lorenz et al. , 2016 ; Nguyen et al. , 2017 ) . We also surface overfitting effects in HPO on both small and large datasets , arguably an overlooked problem in the literature , and demonstrate that our termination criterion helps to mitigate it . 2 BACKGROUND . Bayesian optimization ( BO ) refers to methods of optimizing a black-box objective f : Γ → R in an iterative manner . At every step t , a learner selects an input γt ∈ Γ and observes a noisy output yt , f ( γt ) + εt , where εt is typically assumed to be i.i.d . ( sub ) -Gaussian noise with some variance ( proxy ) σ2ε . The decision of the next input to evaluate depends on a probabilistic model , used to approximate the objective f , and an acquisition function , which determines the decision rule . A popular choice for the probabilistic model is a Gaussian process ( GP ) : f ∼ GP ( µ , κ ) ( Rasmussen & Williams , 2006 ) , specified by some mean function µ : Γ → R and some kernel κ : Γ × Γ → R. As observations y1 : t = [ y1 , . . . , yt ] > for the selected inputs Gt = { γ1 , . . . , γt } are being collected , they are used to update the posterior belief of the model defined by the posterior mean µt ( γ ) and variance σ2t ( γ ) as : µt ( γ ) = κt ( γ ) T ( Kt + σ 2 εI ) −1y1 : t ( 1 ) σ2t ( γ ) = κ ( γ , γ ) − κt ( γ ) > ( Kt + σ2εI ) −1κt ( γ ) , ( 2 ) where ( Kt ) i , j = κ ( γi , γj ) and κt ( γ ) T = [ κ ( γ1 , γ ) , . . . , κ ( γt , γ ) ] T . The next input to query is determined by an acquisition function that aims to trade off exploration and exploitation . Common choices include probability of improvement ( Kushner , 1963 ) , entropy search ( Hennig & Schuler , 2012 ) , GP upper-confidence bound ( Srinivas et al. , 2010 ) , to name a few . The convergence of BO can be quantified via ( simple ) regret , i.e. , the sub-optimality in function value : rt : = f ( γ ∗ t ) − f ( γ∗ ) , ( 3 ) where γ∗ is the global optimizer of f and γ∗t = arg minγ∈Gt f ( γ ) . Specifying adequate tolerance that defines how small the regret should be to terminate BO is of high importance as it determine both the quality and the cost of the solution . However , this criterion can not be directly evaluated in practice , as the input γ∗ and the optimum f ( γ∗ ) are not known . Hyperparameter optimization ( HPO ) is a widely considered application for BO . Consider a supervised learning setting training a machine learning model ( e.g. , a neural network ) M on some feature-response data points D = { ( xi , yi ) } ni=1 sampled i.i.d . from some unknown data distribution P . The model is obtained by running a training algorithm ( e.g. , optimizing the weights of the neural network via SGD ) on D , and the model returned also depends on hyperparameters γ ( e.g. , learning rates used , batch size , etc. ) . We use the notationMγ ( x ; D ) to refer to the prediction that the model produced byM makes for an input x , when trained with hyperparameters γ on data D. Given some loss function ` ( · , · ) , the population risk of the model on unseen data points is given by the expected loss EP [ ` ( y , Mγ ( x , D ) ) ] . The main objective of HPO is to identify hyperparameters γ , such that the resulting model minimizes the population risk : f ( γ ) = EP [ ` ( y , Mγ ( x , D ) ) ] , γ∗ = arg min γ∈Γ f ( γ ) . ( 4 ) In practice , however , the population risk can not be evaluated since P is unknown . Thus , typically , it is estimated on a separate finite validation set DV drawn from the same distribution P . As the result , practical HPO focuses on minimizing the empirical estimate f̂ ( γ ) of the expected loss f ( γ ) leading to ( probably different ) optimizer γ∗D : f̂ ( γ ) = 1 |DV | ∑ xi , yi∈DV ` ( yi , Mγ ( xi , D ) ) , γ∗D = arg min γ∈Γ f̂ ( γ ) . ( 5 ) At its core , BO-based HPO evaluates noisy empirical estimate f̂ ( γt ) for promising hyperparameters γt for some finite number of iterations and the final performance after termination heavily depends on that number . Alternatively , one can also terminate BO when sufficiently close to the global optima , i.e. , using an analogue to the simple regret rt for the validation loss f̂ ( γ ) and f̂ ( γ∗t ) = min γ∈Gt f̂ ( γ ) : r̂t : = f̂ ( γ ∗ t ) − f̂ ( γ∗D ) . ( 6 ) Inconsistency in the optimization objective . Importantly , the true HPO objective f ( γ ) in Eq . ( 4 ) and the empirical surrogate f̂ ( γ ) in Eq . ( 5 ) used for tuning by BO generally do not coincide . Therefore , existing BO approaches may yield sub-optimal solutions to the population risk minimization , even if they succeed in globally optimizing f̂ ( γ ) . This issue , however , is typically neglected in practical HPO . In contrast , we propose a termination condition for BO motivated by the discrepancy in the objectives . 3 TERMINATION CRITERION FOR HYPERPARAMETER OPTIMIZATION . This section firstly motivates why early termination of HPO can be beneficial and then addresses the following two questions : ( 1 ) How to estimate the unknown simple regret and ( 2 ) What threshold of the simple regret can be used to stop HPO . 3.1 MOTIVATION FOR THE TERMINATION CRITERION . We start by analysing the true discrepancy of interest : between the population risk f ( γ∗t ) at some input γ∗t and true optimum f ( γ ∗ ) . We then observe that this discrepancy sums from the statistical error of the empirical BO objective f̂ ( γ ) as well the sub-optimality of the BO candidates ( encoded in the simple regret r̂t ) . The key insight of the following proposition is that iterative reducing of r̂t to 0 may not bring any benefits if the statistical error dominates . Proposition 1 . Consider the expected loss f and its estimator f̂ defined in Eqs . ( 4 ) and ( 5 ) , respectively , and assume the statistical error of the estimator is bounded as ||f̂−f ||∞ ≤ st for some st ≥ 0 . Let γ∗ and γ∗D be their optimizers : γ ∗ = arg minγ∈Γ f ( γ ) and γ ∗ D = arg minγ∈Γ f̂ ( γ ) . Let γ ∗ t be some candidate solution to minγ∈Γ f̂ ( γ ) with sub-optimality in function value r̂t : = f̂ ( γ∗t ) − f̂ ( γ∗D ) . Then the gap in generalization performance f ( γ∗t ) − f ( γ∗ ) can be bounded as follows : f ( γ∗t ) − f ( γ∗ ) ≤ f ( γ∗t ) − f̂ ( γ∗t ) ︸ ︷︷ ︸ ≤ st + f̂ ( γ∗t ) − f̂ ( γ∗D ) ︸ ︷︷ ︸ =r̂t + f̂ ( γ∗D ) − f̂ ( γ∗ ) ︸ ︷︷ ︸ ≤0 + f̂ ( γ∗ ) − f ( γ∗ ) ︸ ︷︷ ︸ ≤ st ≤ 2 st + r̂t . Moreover , without further restrictions on f , f̂ , γ∗t and γ ∗ , the upper bound is tight . Proof : While the second inequality is due to the definition of γ∗t , the others can proved as follows : f ( γ∗t ) − f̂ ( γ∗t ) ≤ |f ( γ∗t ) − f̂ ( γ∗t ) | ≤ max γ∈Γ |f ( γ ) − f̂ ( γ ) | = ||f̂ − f ||∞ ≤ st , γ∗D = arg min γ∈Γ f̂ ( γ ) −→ ∀γ ∈ Γ : f̂ ( γ∗D ) − f̂ ( γ ) ≤ 0 −→ f̂ ( γ∗D ) − f̂ ( γ∗ ) ≤ 0 . The proposition provides the discrepancy bound in terms of the statistical error st and simple regret r̂t . This naturally incites terminating HPO at a candidate γ∗t for which the simple regret r̂t is of the same magnitude as the statistical error st ( as further reduction in r̂t may not improve notably the true objective ) . However , neither of the quantities st and r̂t is known . Below , we propose a termination criterion that relies on estimates of both quantities . Firstly , we show how to use confidence bounds on f̂ ( γ ) to obtain high probability upper bounds on the simple regret r̂t Srinivas et al . ( 2010 ) ; Ha et al . ( 2019 ) . Secondly , we estimate the statistical error st in the case of cross-validation ( Stone , 1974 ; Geisser , 1975 ) where the model performance is defined as an average over several training-validation runs . To this end , we rely on the statistical characteristics ( i.e. , variance or bias ) of such cross-validation-based estimator that are theoretically studied by Nadeau & Bengio ( 2003 ) ; Bayle et al . ( 2020 ) . When cross validation is not used , one could define an intuitive threshold in advance due to the usage of interpretable upper bound of simple regret .

Summary: This paper studies the problem of pre-specifying the optimal termination criterion for Bayesian optimization. Different from prior work that tracks the value of acquisition function, this paper proposes an automatic termination criterion for BO. In particular, they construct a high-probability confidence bound on the regret, and then the users can specify a desired tolerance that shows how accurate the final solution should be compared to the global optimal. They estimate the threshold via a cross-validation estimate of the generalization error. Empirically, they design two evaluation metrics, relative test error change RYC and relative time change RTC, and compare to the comprehensive prior work. The results demonstrate the effectiveness of their proposed approach.