Hyperparameter Optimization Is Deceiving Us, and How to Stop It

1 Introduction . Machine learning can be informally thought of as a double-loop optimization problem . The inner loop is what is typically called training : It learns the parameters of some model by running a training algorithm on a training set . This is usually done to minimize some training loss function via an algorithm such as stochastic gradient descent ( SGD ) . Both the inner-loop training algorithm and the model are parameterized by a vector of hyperparameters ( HPs ) . Unlike the learned output parameters of a ML model , HPs are inputs provided to the learning algorithm that guide the learning process , such as learning rate and network size . The outer-loop optimization problem is to find HPs ( from a set of allowable HPs ) that result in a trained model that performs the best in expectation on “ fresh ” examples drawn from the same source as the training set , as measured by some loss or loss approximation . An algorithm that attempts this task is called a hyperparameter optimization ( HPO ) procedure [ 12 , 20 ] . From this setup comes the natural question : How do we pick the subspace for the HPO procedure to search over ? The HPO search space is enormous , suffering from the curse of dimensionality ; training , which is also expensive , has to be run for each HP configuration tested . Thus , we have to make hard choices . With limited compute resources , we typically pick a small subspace of possible HPs and perform grid search or random search over that subspace . This involves comparing the empirical performance of the resulting trained models , and then reporting on the model that performs best in ∗Corresponding author : https : //cacioepe.pe 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . terms of a chosen validation metric [ 20 , 37 , 41 ] . For grid search , the grid points are often manually set to values put forth in now-classic papers as good rules-of-thumb concerning , for example , how to set the learning rate [ 36 , 45 , 46 , 56 ] . In other words , how we choose which HPs to test can seem rather ad-hoc . We may have a good rationale in mind , but we often elide the details of that rationale on paper ; we choose an HPO configuration without explicitly justifying our choice . Much recent empirical work has critiqued this practice [ 7 , 11 , 16 , 48 , 50 , 53 , 62 , 65 ] . The authors examine HPO configuration choices in prior work , and find that those choices can have an outsize impact on convergence , correctness , and generalization . They therefore argue that more attention should be paid to the origins of empirical gains in ML , as it is often difficult to tell whether measured improvements are attributable to training or to well-chosen ( or lucky ) HPs . Yet , this empirical work does not suggest a path forward for formalizing this problem or addressing it theoretically . To this end , we argue that the process of drawing conclusions using HPO should itself be an object of study . Our contribution is to put forward , to the best of our knowledge , the first theoreticallybacked characterization for making trustworthy conclusions about algorithm performance using HPO . We model theoretically the following empirically-observed problem : When comparing two algorithms , J and K , searching one subspace can pick HPs that yield the conclusion that J outperforms K , whereas searching another can select HPs that entail the opposite result . In short , the way we choose hyperparameters can deceive us—a problem that we call hyperparameter deception . We formalize this problem , and prove and empirically validate a defense against it . Importantly , our proven defense does not make any promises about ground-truth algorithm performance ; rather , it is guaranteed to avoid the possibility of drawing inconsistent conclusions about algorithm performance within some bounded HPO time budget t. In summary , we : • Formalize the process of drawing conclusions from HPO ( epistemic HPO , Section 3 ) . • Leverage the flexible semantics of modal logic to construct a framework for reasoning rigorously about 1 ) uncertainty in epistemic HPO , and 2 ) how this uncertainty can mislead the conclusions drawn by even the most well-intentioned researchers ( Section 4 ) . • Exercise our logical framework to demonstrate that it naturally suggests defenses with guarantees against being deceived by EHPO , and offer a specific , defended-random-search EHPO ( Section 5 ) . 2 Preliminaries : Problem Intuition and Prevalence in ML Research . Principled HPO methods include grid search [ 41 ] and random search [ 3 ] . For the former , we perform HPO on a grid of HP-values , constructed by picking a set for each HP and taking the Cartesian product . For the latter , the HP-values are randomly sampled from chosen distributions . Both of these HPO algorithms are parameterized themselves : Grid search requires inputting the spacing between different configuration points in the grid , and random search requires distributions from which to sample . We call these HPO-procedure-input values hyper-hyperparameters ( hyper-HPs ) .2 To make HPO outputs comparable , we also introduce the notion of a log : Definition 1 . A log � records all the choices and measurements made during an HPO run , including the total time T it took to run . It has all necessary information to make the HPO run reproducible . A log can be thought of as everything needed to produce a table in a research paper : code , random seed , choice of hyper-HPs , information about the learning task , properties of the learning algorithm , all of the observable results . We formalize all of the randomness in HPO in terms of a random seed r and a pseudo-random number generator ( PRNG ) G. Given a seed , G deterministically produces a sequence of pseudo-random numbers : all numbers lie in some set I ( typically 64-bit integers ) , i.e . r ∈ I and PRNG G : I → I∞ . With this , we can now define HPO formally : Definition 2 . An HPO procedure H is a tuple ( H∗ , C , Λ , A , M , G , X ) where H∗ is a randomized algorithm , C is a set of allowable hyper-HPs ( i.e. , allowable configurations for H∗ ) , Λ is a set of allowable HPs ( i.e. , of HP sets λ ) , A is a training algorithm ( e.g . SGD ) , M is a model ( e.g . VGG16 ) , G is a PRNG , and X is some dataset ( usually split into train and validation sets ) . When run , H∗ takes as input a hyper-HP configuration c ∈ C and a random seed r ∈ I , then proceeds to run Aλ ( on Mλ 2We provide a glossary of all definitions and symbols for reference at the beginning of the Appendix . using G ( r ) and data3 from X ) some number of times for different HPs λ ∈ Λ . Finally , H∗ outputs a tuple ( λ∗ , � ) , where λ∗ is the HP configuration chosen by HPO and � is the log documenting the run . Running H is a crucial part of model development . As part of an empirical , scientific procedure , we specify different training algorithms and a learning task , run potentially many HPO passes , and try to make general conclusions about overall algorithm performance . That is , we aim to develop knowledge regarding whether one of the algorithms outperforms the others . However , recent empirical findings indicate that it is actually really challenging to pick hyper-HPs that yield reliable knowledge about general algorithm performance . In fact , it is a surprisingly common occurrence to be able to draw inconsistent conclusions based on our choice of hyper-HPs [ 11 , 16 , 48 , 65 ] . An example illustrating the possibility of drawing inconsistent conclusions from HPO . As a. first step to studying HPO as a procedure for developing reliable knowledge , we provide an example of how being inadvertently deceived by HPO is a real problem , even in excellent research ( we give an additional example in the Appendix ) .4 We first reproduce Wilson et al . [ 72 ] , in which the authors trained VGG16 with different optimizers on CIFAR-10 ( Figure 1a ) . This experiment uses grid search , with a powers-of-2 grid for the learning rate α crossed with the default HPs for Adam . Based on the best-performing HPO per algorithm ( α = 1 ) , it is reasonable to conclude that non-adaptive methods ( e.g. , SGD ) perform better than adaptive ones ( e.g. , Adam [ 42 ] ) , as the non-adaptive optimizers demonstrate higher test accuracy . However , this setting of grid search ’ s hyper-HPs directly informs this particular conclusion ; using different hyper-HPs makes it possible to conclude the opposite . Inspired by Choi et al . [ 11 ] , we perform grid search over a different subspace , tuning both learning rate and Adam ’ s � parameter . Our results entail the logically opposite conclusion : Non-adaptive methods do not outperform adaptive ones . Rather , when choosing the HPs that maximize test accuracy , all of the optimizers essentially have equivalent performance ( Figure 1b , Appendix ) . Notably , as we can see from the confidence intervals in Figure 1 , satisfying statistical significance is not sufficient to avoid being deceived about comparative algorithm performance [ 73 ] . Thus , we will require additional tools aside from statistical tests to reason about this , which we discuss in Sections 4 & 5 . This example is not exceptional , or even particularly remarkable , in terms of illustrating the hyperparameter deception problem . We simply chose it for convenience : The experiment does not require highly-specialized ML sub-domain expertise to understand it , and it is arguably broadly familiar , as it very well-cited [ 72 ] . However , we emphasize that hyperparameter deception is rather common . Additional examples can be found in numerous empirical studies across ML subfields [ 7 , 11 , 16 , 49 , 50 , 53 , 60 , 65 ] ( Appendix ) . This work shows that reported results tend to 3Definition 2 does not preclude cross-validation , as this can be part of H∗ . The input dataset X can be split in various ways , as a function of the random seed r. 4All code can be found at https : //github.com/pasta41/deception . be impressive for the tested hyper-HP configurations , but that modifying HPO can lead to vastly different performance outcomes that entail contradictory conclusions . More generally , it is possible to develop results that are wrong about performance , or else correct about performance but for the wrong reasons ( e.g. , by picking “ lucky ” hyperparameters ) . Neither of these outcomes constitutes reliable knowledge [ 27 , 47 ] . As scientists , this is disheartening . We want to have confidence in the conclusions we draw from our experiments . We want to trust that we are deriving reliable knowledge about algorithm performance . In the sections that follow , our aim is to study HPO in this reliable-knowledge sense : We want to develop ways to reason rigorously and confidently about how we derive knowledge from empirical investigations involving HPO .

Summary: This paper argues that hyperparameter optimization can lead researchers to draw opposite conclusions when comparing different algorithms. In particular, the authors show that this happens not only as a result of the choice of hyperparameters, but also as a result of the hyperparameter search procedure (e.g., including the search space choice).   To alleviate this concern, the authors propose a novel framework based on modal logic to determine whether a hyperparameter search procedure is "defended" against such deceptions. The argument and proposal is demonstrated through a running example based on work from Wilson et al., showing that according to a user-defined skepticism threshold one could either not draw any conclusion or draw the opposite conclusion from what is claimed in the paper.