Automatic Termination for Hyperparameter Optimization

1 INTRODUCTION . While the performance of machine learning algorithms crucially depends on their hyperparameters , setting them correctly is typically a tedious and expensive task . Hyperparameter optimization ( HPO ) emerged as a new sub-field in machine learning that tries to automatically determine how to configure a machine learning algorithm . One of the most successful strategies for HPO is Bayesian optimization ( BO ; Močkus , 1975 ; Chen et al. , 2018 ; Snoek et al. , 2012 ; Melis et al. , 2018 ) which iteratively trains a probabilistic model on the evaluations of the tuned algorithm . to select the most promising next candidate point that trades-off exploration and exploitation . In practice , the quality of the final solution found by BO heavily depends on a user defined budget , such as wall-clock time or the number of iterations , which needs to be defined in advanced . If this budget is too small , BO might return hyperparameters that result in a poor predictive performance . If the budget is too large , compute resources will be wasted and , in some cases , it may result in overfitting as we will show in our experiments . Automatically stopping BO is a rather under-explored topic in the literature . A simple baseline is to stop if BO has not found a better solution than the current best incumbent for some successive iterations , which is in the same vein as early stopping for neural network training . Another approach is to track probability of improvement ( Lorenz et al. , 2016 ) or expected improvement ( Nguyen et al. , 2017 ) , and stop the optimization process once it falls below a given threshold . However , determining this threshold may in practice be less intuitive than setting the number of iterations or the wall-clock time . Instead of stopping BO completely , McLeod et al . ( 2018 ) propose to switch to local optimization when the global regret is smaller than a pre-defined target . This condition can also be used to terminate BO early , but it comes with additional complexity such as identifying a ( convex ) region for local optimization and again a predefined budget . In this work , we propose a simple and interpretable automatic termination criterion for BO . In our criterion , we construct high-probability confidence bound on the regret ( i.e. , the difference of our current solution to the global optimum ) exploiting the probabilistic model of the objective . Thus , users are now asked to specify a desired tolerance that defines how accurate should the final solution be compared to global optimum . In addition , we propose to determine the threshold via a cross-validation estimate of the generalization error . This choice takes into account the irreducible discrepancy between the actual objective and the target function optimized via BO , namely the difference between the performance on new data ( i.e. , the population risk ) and the validation error . Our extensive empirical evaluation on a variety of HPO and NAS benchmarks suggests that our method is more robust and effective in maintaining the final solution quality than common baselines ( Lorenz et al. , 2016 ; Nguyen et al. , 2017 ) . We also surface overfitting effects in HPO on both small and large datasets , arguably an overlooked problem in the literature , and demonstrate that our termination criterion helps to mitigate it . 2 BACKGROUND . Bayesian optimization ( BO ) refers to methods of optimizing a black-box objective f : Γ → R in an iterative manner . At every step t , a learner selects an input γt ∈ Γ and observes a noisy output yt , f ( γt ) + εt , where εt is typically assumed to be i.i.d . ( sub ) -Gaussian noise with some variance ( proxy ) σ2ε . The decision of the next input to evaluate depends on a probabilistic model , used to approximate the objective f , and an acquisition function , which determines the decision rule . A popular choice for the probabilistic model is a Gaussian process ( GP ) : f ∼ GP ( µ , κ ) ( Rasmussen & Williams , 2006 ) , specified by some mean function µ : Γ → R and some kernel κ : Γ × Γ → R. As observations y1 : t = [ y1 , . . . , yt ] > for the selected inputs Gt = { γ1 , . . . , γt } are being collected , they are used to update the posterior belief of the model defined by the posterior mean µt ( γ ) and variance σ2t ( γ ) as : µt ( γ ) = κt ( γ ) T ( Kt + σ 2 εI ) −1y1 : t ( 1 ) σ2t ( γ ) = κ ( γ , γ ) − κt ( γ ) > ( Kt + σ2εI ) −1κt ( γ ) , ( 2 ) where ( Kt ) i , j = κ ( γi , γj ) and κt ( γ ) T = [ κ ( γ1 , γ ) , . . . , κ ( γt , γ ) ] T . The next input to query is determined by an acquisition function that aims to trade off exploration and exploitation . Common choices include probability of improvement ( Kushner , 1963 ) , entropy search ( Hennig & Schuler , 2012 ) , GP upper-confidence bound ( Srinivas et al. , 2010 ) , to name a few . The convergence of BO can be quantified via ( simple ) regret , i.e. , the sub-optimality in function value : rt : = f ( γ ∗ t ) − f ( γ∗ ) , ( 3 ) where γ∗ is the global optimizer of f and γ∗t = arg minγ∈Gt f ( γ ) . Specifying adequate tolerance that defines how small the regret should be to terminate BO is of high importance as it determine both the quality and the cost of the solution . However , this criterion can not be directly evaluated in practice , as the input γ∗ and the optimum f ( γ∗ ) are not known . Hyperparameter optimization ( HPO ) is a widely considered application for BO . Consider a supervised learning setting training a machine learning model ( e.g. , a neural network ) M on some feature-response data points D = { ( xi , yi ) } ni=1 sampled i.i.d . from some unknown data distribution P . The model is obtained by running a training algorithm ( e.g. , optimizing the weights of the neural network via SGD ) on D , and the model returned also depends on hyperparameters γ ( e.g. , learning rates used , batch size , etc. ) . We use the notationMγ ( x ; D ) to refer to the prediction that the model produced byM makes for an input x , when trained with hyperparameters γ on data D. Given some loss function ` ( · , · ) , the population risk of the model on unseen data points is given by the expected loss EP [ ` ( y , Mγ ( x , D ) ) ] . The main objective of HPO is to identify hyperparameters γ , such that the resulting model minimizes the population risk : f ( γ ) = EP [ ` ( y , Mγ ( x , D ) ) ] , γ∗ = arg min γ∈Γ f ( γ ) . ( 4 ) In practice , however , the population risk can not be evaluated since P is unknown . Thus , typically , it is estimated on a separate finite validation set DV drawn from the same distribution P . As the result , practical HPO focuses on minimizing the empirical estimate f̂ ( γ ) of the expected loss f ( γ ) leading to ( probably different ) optimizer γ∗D : f̂ ( γ ) = 1 |DV | ∑ xi , yi∈DV ` ( yi , Mγ ( xi , D ) ) , γ∗D = arg min γ∈Γ f̂ ( γ ) . ( 5 ) At its core , BO-based HPO evaluates noisy empirical estimate f̂ ( γt ) for promising hyperparameters γt for some finite number of iterations and the final performance after termination heavily depends on that number . Alternatively , one can also terminate BO when sufficiently close to the global optima , i.e. , using an analogue to the simple regret rt for the validation loss f̂ ( γ ) and f̂ ( γ∗t ) = min γ∈Gt f̂ ( γ ) : r̂t : = f̂ ( γ ∗ t ) − f̂ ( γ∗D ) . ( 6 ) Inconsistency in the optimization objective . Importantly , the true HPO objective f ( γ ) in Eq . ( 4 ) and the empirical surrogate f̂ ( γ ) in Eq . ( 5 ) used for tuning by BO generally do not coincide . Therefore , existing BO approaches may yield sub-optimal solutions to the population risk minimization , even if they succeed in globally optimizing f̂ ( γ ) . This issue , however , is typically neglected in practical HPO . In contrast , we propose a termination condition for BO motivated by the discrepancy in the objectives . 3 TERMINATION CRITERION FOR HYPERPARAMETER OPTIMIZATION . This section firstly motivates why early termination of HPO can be beneficial and then addresses the following two questions : ( 1 ) How to estimate the unknown simple regret and ( 2 ) What threshold of the simple regret can be used to stop HPO . 3.1 MOTIVATION FOR THE TERMINATION CRITERION . We start by analysing the true discrepancy of interest : between the population risk f ( γ∗t ) at some input γ∗t and true optimum f ( γ ∗ ) . We then observe that this discrepancy sums from the statistical error of the empirical BO objective f̂ ( γ ) as well the sub-optimality of the BO candidates ( encoded in the simple regret r̂t ) . The key insight of the following proposition is that iterative reducing of r̂t to 0 may not bring any benefits if the statistical error dominates . Proposition 1 . Consider the expected loss f and its estimator f̂ defined in Eqs . ( 4 ) and ( 5 ) , respectively , and assume the statistical error of the estimator is bounded as ||f̂−f ||∞ ≤ st for some st ≥ 0 . Let γ∗ and γ∗D be their optimizers : γ ∗ = arg minγ∈Γ f ( γ ) and γ ∗ D = arg minγ∈Γ f̂ ( γ ) . Let γ ∗ t be some candidate solution to minγ∈Γ f̂ ( γ ) with sub-optimality in function value r̂t : = f̂ ( γ∗t ) − f̂ ( γ∗D ) . Then the gap in generalization performance f ( γ∗t ) − f ( γ∗ ) can be bounded as follows : f ( γ∗t ) − f ( γ∗ ) ≤ f ( γ∗t ) − f̂ ( γ∗t ) ︸ ︷︷ ︸ ≤ st + f̂ ( γ∗t ) − f̂ ( γ∗D ) ︸ ︷︷ ︸ =r̂t + f̂ ( γ∗D ) − f̂ ( γ∗ ) ︸ ︷︷ ︸ ≤0 + f̂ ( γ∗ ) − f ( γ∗ ) ︸ ︷︷ ︸ ≤ st ≤ 2 st + r̂t . Moreover , without further restrictions on f , f̂ , γ∗t and γ ∗ , the upper bound is tight . Proof : While the second inequality is due to the definition of γ∗t , the others can proved as follows : f ( γ∗t ) − f̂ ( γ∗t ) ≤ |f ( γ∗t ) − f̂ ( γ∗t ) | ≤ max γ∈Γ |f ( γ ) − f̂ ( γ ) | = ||f̂ − f ||∞ ≤ st , γ∗D = arg min γ∈Γ f̂ ( γ ) −→ ∀γ ∈ Γ : f̂ ( γ∗D ) − f̂ ( γ ) ≤ 0 −→ f̂ ( γ∗D ) − f̂ ( γ∗ ) ≤ 0 . The proposition provides the discrepancy bound in terms of the statistical error st and simple regret r̂t . This naturally incites terminating HPO at a candidate γ∗t for which the simple regret r̂t is of the same magnitude as the statistical error st ( as further reduction in r̂t may not improve notably the true objective ) . However , neither of the quantities st and r̂t is known . Below , we propose a termination criterion that relies on estimates of both quantities . Firstly , we show how to use confidence bounds on f̂ ( γ ) to obtain high probability upper bounds on the simple regret r̂t Srinivas et al . ( 2010 ) ; Ha et al . ( 2019 ) . Secondly , we estimate the statistical error st in the case of cross-validation ( Stone , 1974 ; Geisser , 1975 ) where the model performance is defined as an average over several training-validation runs . To this end , we rely on the statistical characteristics ( i.e. , variance or bias ) of such cross-validation-based estimator that are theoretically studied by Nadeau & Bengio ( 2003 ) ; Bayle et al . ( 2020 ) . When cross validation is not used , one could define an intuitive threshold in advance due to the usage of interpretable upper bound of simple regret .

Summary: The paper proposes a new approach for automatic termination of hyperparameter optimization based on Bayesian Optimization (BO).  The idea is to construct high-probability confidence bound on the regret and then determine when to terminate the BO process. Empirical experiments are conducted on various real-world HPO and NAS benchmarks to show the efficacy of the proposed approach.