Hyperparameter Optimization Is Deceiving Us, and How to Stop It

1 Introduction . Machine learning can be informally thought of as a double-loop optimization problem . The inner loop is what is typically called training : It learns the parameters of some model by running a training algorithm on a training set . This is usually done to minimize some training loss function via an algorithm such as stochastic gradient descent ( SGD ) . Both the inner-loop training algorithm and the model are parameterized by a vector of hyperparameters ( HPs ) . Unlike the learned output parameters of a ML model , HPs are inputs provided to the learning algorithm that guide the learning process , such as learning rate and network size . The outer-loop optimization problem is to find HPs ( from a set of allowable HPs ) that result in a trained model that performs the best in expectation on “ fresh ” examples drawn from the same source as the training set , as measured by some loss or loss approximation . An algorithm that attempts this task is called a hyperparameter optimization ( HPO ) procedure [ 12 , 20 ] . From this setup comes the natural question : How do we pick the subspace for the HPO procedure to search over ? The HPO search space is enormous , suffering from the curse of dimensionality ; training , which is also expensive , has to be run for each HP configuration tested . Thus , we have to make hard choices . With limited compute resources , we typically pick a small subspace of possible HPs and perform grid search or random search over that subspace . This involves comparing the empirical performance of the resulting trained models , and then reporting on the model that performs best in ∗Corresponding author : https : //cacioepe.pe 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . terms of a chosen validation metric [ 20 , 37 , 41 ] . For grid search , the grid points are often manually set to values put forth in now-classic papers as good rules-of-thumb concerning , for example , how to set the learning rate [ 36 , 45 , 46 , 56 ] . In other words , how we choose which HPs to test can seem rather ad-hoc . We may have a good rationale in mind , but we often elide the details of that rationale on paper ; we choose an HPO configuration without explicitly justifying our choice . Much recent empirical work has critiqued this practice [ 7 , 11 , 16 , 48 , 50 , 53 , 62 , 65 ] . The authors examine HPO configuration choices in prior work , and find that those choices can have an outsize impact on convergence , correctness , and generalization . They therefore argue that more attention should be paid to the origins of empirical gains in ML , as it is often difficult to tell whether measured improvements are attributable to training or to well-chosen ( or lucky ) HPs . Yet , this empirical work does not suggest a path forward for formalizing this problem or addressing it theoretically . To this end , we argue that the process of drawing conclusions using HPO should itself be an object of study . Our contribution is to put forward , to the best of our knowledge , the first theoreticallybacked characterization for making trustworthy conclusions about algorithm performance using HPO . We model theoretically the following empirically-observed problem : When comparing two algorithms , J and K , searching one subspace can pick HPs that yield the conclusion that J outperforms K , whereas searching another can select HPs that entail the opposite result . In short , the way we choose hyperparameters can deceive us—a problem that we call hyperparameter deception . We formalize this problem , and prove and empirically validate a defense against it . Importantly , our proven defense does not make any promises about ground-truth algorithm performance ; rather , it is guaranteed to avoid the possibility of drawing inconsistent conclusions about algorithm performance within some bounded HPO time budget t. In summary , we : • Formalize the process of drawing conclusions from HPO ( epistemic HPO , Section 3 ) . • Leverage the flexible semantics of modal logic to construct a framework for reasoning rigorously about 1 ) uncertainty in epistemic HPO , and 2 ) how this uncertainty can mislead the conclusions drawn by even the most well-intentioned researchers ( Section 4 ) . • Exercise our logical framework to demonstrate that it naturally suggests defenses with guarantees against being deceived by EHPO , and offer a specific , defended-random-search EHPO ( Section 5 ) . 2 Preliminaries : Problem Intuition and Prevalence in ML Research . Principled HPO methods include grid search [ 41 ] and random search [ 3 ] . For the former , we perform HPO on a grid of HP-values , constructed by picking a set for each HP and taking the Cartesian product . For the latter , the HP-values are randomly sampled from chosen distributions . Both of these HPO algorithms are parameterized themselves : Grid search requires inputting the spacing between different configuration points in the grid , and random search requires distributions from which to sample . We call these HPO-procedure-input values hyper-hyperparameters ( hyper-HPs ) .2 To make HPO outputs comparable , we also introduce the notion of a log : Definition 1 . A log � records all the choices and measurements made during an HPO run , including the total time T it took to run . It has all necessary information to make the HPO run reproducible . A log can be thought of as everything needed to produce a table in a research paper : code , random seed , choice of hyper-HPs , information about the learning task , properties of the learning algorithm , all of the observable results . We formalize all of the randomness in HPO in terms of a random seed r and a pseudo-random number generator ( PRNG ) G. Given a seed , G deterministically produces a sequence of pseudo-random numbers : all numbers lie in some set I ( typically 64-bit integers ) , i.e . r ∈ I and PRNG G : I → I∞ . With this , we can now define HPO formally : Definition 2 . An HPO procedure H is a tuple ( H∗ , C , Λ , A , M , G , X ) where H∗ is a randomized algorithm , C is a set of allowable hyper-HPs ( i.e. , allowable configurations for H∗ ) , Λ is a set of allowable HPs ( i.e. , of HP sets λ ) , A is a training algorithm ( e.g . SGD ) , M is a model ( e.g . VGG16 ) , G is a PRNG , and X is some dataset ( usually split into train and validation sets ) . When run , H∗ takes as input a hyper-HP configuration c ∈ C and a random seed r ∈ I , then proceeds to run Aλ ( on Mλ 2We provide a glossary of all definitions and symbols for reference at the beginning of the Appendix . using G ( r ) and data3 from X ) some number of times for different HPs λ ∈ Λ . Finally , H∗ outputs a tuple ( λ∗ , � ) , where λ∗ is the HP configuration chosen by HPO and � is the log documenting the run . Running H is a crucial part of model development . As part of an empirical , scientific procedure , we specify different training algorithms and a learning task , run potentially many HPO passes , and try to make general conclusions about overall algorithm performance . That is , we aim to develop knowledge regarding whether one of the algorithms outperforms the others . However , recent empirical findings indicate that it is actually really challenging to pick hyper-HPs that yield reliable knowledge about general algorithm performance . In fact , it is a surprisingly common occurrence to be able to draw inconsistent conclusions based on our choice of hyper-HPs [ 11 , 16 , 48 , 65 ] . An example illustrating the possibility of drawing inconsistent conclusions from HPO . As a. first step to studying HPO as a procedure for developing reliable knowledge , we provide an example of how being inadvertently deceived by HPO is a real problem , even in excellent research ( we give an additional example in the Appendix ) .4 We first reproduce Wilson et al . [ 72 ] , in which the authors trained VGG16 with different optimizers on CIFAR-10 ( Figure 1a ) . This experiment uses grid search , with a powers-of-2 grid for the learning rate α crossed with the default HPs for Adam . Based on the best-performing HPO per algorithm ( α = 1 ) , it is reasonable to conclude that non-adaptive methods ( e.g. , SGD ) perform better than adaptive ones ( e.g. , Adam [ 42 ] ) , as the non-adaptive optimizers demonstrate higher test accuracy . However , this setting of grid search ’ s hyper-HPs directly informs this particular conclusion ; using different hyper-HPs makes it possible to conclude the opposite . Inspired by Choi et al . [ 11 ] , we perform grid search over a different subspace , tuning both learning rate and Adam ’ s � parameter . Our results entail the logically opposite conclusion : Non-adaptive methods do not outperform adaptive ones . Rather , when choosing the HPs that maximize test accuracy , all of the optimizers essentially have equivalent performance ( Figure 1b , Appendix ) . Notably , as we can see from the confidence intervals in Figure 1 , satisfying statistical significance is not sufficient to avoid being deceived about comparative algorithm performance [ 73 ] . Thus , we will require additional tools aside from statistical tests to reason about this , which we discuss in Sections 4 & 5 . This example is not exceptional , or even particularly remarkable , in terms of illustrating the hyperparameter deception problem . We simply chose it for convenience : The experiment does not require highly-specialized ML sub-domain expertise to understand it , and it is arguably broadly familiar , as it very well-cited [ 72 ] . However , we emphasize that hyperparameter deception is rather common . Additional examples can be found in numerous empirical studies across ML subfields [ 7 , 11 , 16 , 49 , 50 , 53 , 60 , 65 ] ( Appendix ) . This work shows that reported results tend to 3Definition 2 does not preclude cross-validation , as this can be part of H∗ . The input dataset X can be split in various ways , as a function of the random seed r. 4All code can be found at https : //github.com/pasta41/deception . be impressive for the tested hyper-HP configurations , but that modifying HPO can lead to vastly different performance outcomes that entail contradictory conclusions . More generally , it is possible to develop results that are wrong about performance , or else correct about performance but for the wrong reasons ( e.g. , by picking “ lucky ” hyperparameters ) . Neither of these outcomes constitutes reliable knowledge [ 27 , 47 ] . As scientists , this is disheartening . We want to have confidence in the conclusions we draw from our experiments . We want to trust that we are deriving reliable knowledge about algorithm performance . In the sections that follow , our aim is to study HPO in this reliable-knowledge sense : We want to develop ways to reason rigorously and confidently about how we derive knowledge from empirical investigations involving HPO .

Summary: This paper concerns an important and broad issue in machine learning regarding hyperparameter optimisation (HPO) and that it might lead to inconsistent outcomes and conclusions about approaches and their performance, in particular the performance that is relative to other approaches. The authors argue that the analysis of an HPO should be an object of study by itself. The paper takes into account some existing related work in regard to HPO. It provides definitions for an (HPO) log, an HPO procedure, an epistemic HPO procedure, to formalise the reasoning process regarding HPOs, among other definitions. It provides an illustration (an example) of how HPOs can be deceptive (that example is also referenced in further sections to help the narrative). As a possible solution, the authors suggest a process (they call it “epistemic hyperparameter optimization” or “EHPO”) and a logical framework (with syntax and semantics provided). That provides a way to analyse uncertainty in EHPO and reason about inconsistent HPO conclusions, to which such uncertainty might lead. The authors define a t-non-deceptive EHPO via a definition that depends on some time (i.e. “t-”) notion (relevant to execution time of experiments). They also show how, theoretically, a defended reasoner can be made, the use of which could result in a guaranteed t-non-deceptive EHPO. Then, they present a defended version of a random search for HPO and present a theorem statement regarding such a search’s t-non-deceptiveness under certain assumptions. Then, the authors explain and show results for an experiment using a modified version of the random search on the illustrative example, and finally they conclude the paper and provide a summary of some practical takeaways.