Response Modeling of Hyper-Parameters for Deep Convolutional Neural Networks

1 INTRODUCTION . The choice of Hyper-Parameters ( HP ) – such as initial learning rate , batch size , and weight decay – has shown to greatly impact the generalization performance of Deep Neural Network ( DNN ) training ( Keskar et al. , 2017 ; Wilson et al. , 2017 ; Li et al. , 2019 ; Yu & Zhu , 2020 ) . By increasing the complexity of network architectures ( from high to low parameterized models ) and training datasets ( class number and samples ) , the manual intervention to tune these parameters for optimization becomes a practically expensive and highly challenging task . Therefore , the problem of Hyper-Parameter Optimization ( HPO ) becomes central to developing highly efficient training workflows . Recent studies shift the gear toward development of a meaningful metric measure to explain effective HP tuning for DNN training . This is done in several behavioural studies , including changes in loss surfaces ( Keskar et al. , 2017 ) , input perturbation analysis ( Novak et al. , 2018 ) , and the energy norm of the covariance of gradients ( Jastrzebski et al. , 2020 ) , just to name a few . In fact , the abstract formulation of the HPO problem , as highlighted by Bergstra & Bengio ( 2012 ) , can be modelled by λ∗ ← arg min λ∈Λ { Ex∼M [ L ( x ; Aλ ( X ( train ) ) ] } , ( 1 ) where , X ( train ) and x are random variables , modelled by some natural distributionM , that represent the train and validation data , respectively , L ( · ) is some expected loss , and Aλ ( X ( train ) ) is a learning algorithm that maps X ( train ) to some learned function , conditioned on the hyper-parameter set λ . Note that this learned function , denoted as f ( θ ; λ ; X ( train ) ) , involves its own inner optimization problem . The HPO in ( 1 ) highlights two optimization problems of which optimization over λ can not occur until optimization over f ( θ ; λ ; X ( train ) ) is complete . This fact applies heavy computational burden for HPO . Bergstra & Bengio ( 2012 ) reduce this burden by attempting to solve the following λ∗ ← arg min λ∈Λ τ ( λ ) , ( 2 ) where τ is called the hyper-parameter response function or response surface , and Λ is some set of choices for λ ( i.e . the search space ) . The goal of the response surface is to introduce an auxiliary 1 Under review as a conference paper at ICLR 2021 function parameterized by λ of which its minimization is directly correlated to minimization of the objective function f ( θ ) . Little advancements in an analytical model of the response surface has led to estimating it by ( a ) running multiple trials of different HP configurations ( e.g . grid searching ) , using evaluation against validation sets as an estimate to τ ; or ( b ) characterizing the distribution model of a configuration ’ s performance metric ( e.g . cross-validation performances ) to numerically define a relationship between τ and λ . An important shift occurred when Bergstra & Bengio ( 2012 ) showed that random searching is more efficient to grid searching , particularly when optimizing high-dimensional HP sets . To mitigate the time complexity and increase overall performance , subsequent methods attempted to characterize the distribution model for such random configurations ( Snoek et al. , 2012 ; Eggensperger et al. , 2013 ; Feurer et al. , 2015a ; b ; Klein et al. , 2017 ; Falkner et al. , 2018 ) or employed population control ( Young et al. , 2015 ; Jaderberg et al. , 2017 ) or early-stopping ( Karnin et al. , 2013 ; Li et al. , 2017 ; 2018 ) . However , these methods suffer from ( a ) additional internal HPs that require manual tuning facilitated by extensive domain knowledge ; ( b ) heavy computational overhead whereby the optimization process takes days to weeks in most cases ( Li et al. , 2017 ; Falkner et al. , 2018 ; Yu & Zhu , 2020 ) ; ( c ) poor generalization across model selection , datasets , and general experimental configurations ( e.g . optimizers ) ; and ( d ) strong dependence on a manually defined search ranges that heavily influences results ( Choi et al. , 2020 ; Sivaprasad et al. , 2020 ) . Importantly , these ranges are generally chosen based on intuition , expert domain knowledge , or some form of a priori knowledge . In this paper , we employ the notion of knowledge gain ( Hosseini & Plataniotis , 2020 ) to model a response surface – solvable with low computational overhead – and use it to perform automatic HPO that does not require any a priori knowledge while still achieving competitive performance against baselines and existing state of the art ( SOTA ) methods . Our goal is therefore to develop an algorithm that is fully autonomous and domain independent that can achieve competitive performance ( not necessarily superior performance ) . We restrict our response surface to consider a single HP , namely the initial learning rate η , and support this choice by noting that the initial learning rate is the most sensitive and important HP towards final model performance ( Goodfellow et al. , 2016 ; Bergstra & Bengio , 2012 ; Yu & Zhu , 2020 ) ( see also Figure 10 in Appendix C ) . We demonstrate how our method ’ s optimization directly correlates to optimizing model performance . Finally , we provide empirical measures of the computational requirements of our algorithm and present thorough experiments on a diverse set of Convolutional Neural Network ( CNN ) and Computer Vision dataset that demonstrate the generalization of our response surface . The main contributions of this work are as follows : 1 . Inspired by knowledge gain , we introduce a well-defined , analytical response surface using the low-rank-factorization of convolution weights ( Equation 5 ) . 2 . We propose a dynamic tracking algorithm of low computational overhead on the order of minutes and hours , dubbed autoHyper , to optimize our response surface and conduct HPO . 3 . This algorithm requires no domain knowledge , human intuition , or manual intervention , and is not bound by a manually set searching space , allowing for completely automatic setting of the initial learning rate ; a novelty for deep learning practitioners . 1.1 RELATED WORKS . We leave extensive analysis of the related works to established surveys ( Luo , 2016 ; He et al. , 2019 ; Yu & Zhu , 2020 ) but present a general overview here . Grid searching and manual tuning techniques that require extensive domain knowledge trial various configurations and retain the best . Random search ( Bergstra & Bengio , 2012 ) was proven to be more efficient , particularly in high-dimensional cases , but these methods suffer from redundancy and high computational overhead . Bayesian optimization ( Snoek et al. , 2012 ; Eggensperger et al. , 2013 ; Feurer et al. , 2015a ; b ; Klein et al. , 2017 ) techniques attempt to characterize the distribution model of the random HP configurations . They fail to properly define the response surface τ and resolve to estimating it by rationalizing a Gaussian process over sampling points . The use of neural networks over Gaussian to model the generalization performance was shown to have better computational performance ( Snoek et al. , 2015 ; Springenberg et al. , 2016 ) . Furthermore , the early stopping methods ( Karnin et al. , 2013 ; Li et al. , 2017 ; 2018 ) spawn various configurations with equal resource distributions , successively stopping poorperforming configurations and reassigning resources dynamically . Population-based training ( PBT ) 2 Under review as a conference paper at ICLR 2021 methods ( Young et al. , 2015 ; Jaderberg et al. , 2017 ) follow an evolutionary approach by spawning various experimental configurations and adapting poor-performing trials to warm restart with inherited learnable parameters and HPs . In addition , other methods such as orthogonal array tuning ( Zhang et al. , 2019 ) , box-constrained derivative-free optimization ( Diaz et al. , 2017 ) , reverse dynamics algorithm for SGD optimization ( Maclaurin et al. , 2015 ) , and hybrid methods ( Swersky et al. , 2013 ; 2014 ; Domhan et al. , 2015 ; Falkner et al. , 2018 ; Kandasamy et al. , 2016 ) exist but demonstrate no significant benefits over the previous techniques . Generally , each of these methods suffer from high computational overheads – on the order of days to weeks to converge – as well as additional internal HPs that heavily influence performance and generalization . In recent years , many Python libraries have also been developed that include these optimization methods ( Bergstra et al. , 2013 ; Kotthoff et al. , 2017 ; Akiba et al. , 2019 ) . 2 A NEW RESPONSE SURFACE MODEL . In this section , we motivate and develop a new response surface model τ ( λ ) based on the low-rank factorization of convolutional weights in a CNN . Unlike the common approach of cross-validation performance measures , we define a new measure on the well-posedness of the intermediate layers of a CNN and relate this measure to the general performance of the network . We first start by adopting the low-rank measure of convolution weights . 2.1 KNOWLEDGE GAIN VIA LOW-RANK FACTORIZATION . Consider a four-way array ( 4-D tensor ) W ∈ RN1×N2×N3×N4 as the convolution weights of an intermediate layer of a CNN ( N1 and N2 being the height and width of kernel size , and N3 and N4 to the input and output channel size , respectively ) . Under the convolution operation , the input feature maps F I ∈ RW×H×N3 are mapped to an arbitrary output feature map FO ∈ RW×H×N4 by FO : , : ,i4 = N3∑ i3=1 F I : , : ,i3 ∗W : , : ,i3 , i4 . W ( 4-D Tensor ) unfold−−−→Wd ( 2-D Matrix ) factorize then decompose−−−−−−−−−−−−−→ ÛdΣ̂dV̂ Td︸ ︷︷ ︸ Ŵd +Ed We note the importance of factorizing the unfolded matrix Wd using a low-rank factorization ( we use the Variational Bayesian Matrix Factorization ( VBMF ) ( Nakajima et al. , 2013 ) ) . Without this factorization , the presence of noise will inhibit proper analysis . This noise Ed will “ capture ” the randomness of initialization and ignoring it will allow us to better analyze our unfolded matrices and make our response surface robust to initialization method . Following the definition of Knowledge Gain ( KG ) from Hosseini & Plataniotis ( 2020 ) , one can now define a metric for each network layer using the norm energy of the low-rank factorization as Gd ( Ŵd ) = 1 Nd · σ1 ( Ŵd ) N ′ d∑ i=1 σi ( Ŵd ) . ( 3 ) where , σ1 ≥ σ2 ≥ . . . ≥ σNd are the associated low-rank singular values in descending order . Here Nd = rank { Ŵd } and the unfolding can be done in either input or output channels i.e . d ∈ { 3 , 4 } . For more information on KG as well as its efficient algorithmic computation , we refer the reader to Hosseini & Plataniotis ( 2020 ) . The metric defined in ( 3 ) is normalized such that Gd ∈ [ 0 , 1 ] and can be used to probe CNN layers to monitor their efficiency in the carriage of information from input to output feature maps . We can further parameterize the KG by the HP set λ , epoch t , and network layer ` as Ḡd , t , ` ( λ ) . A perfect network and set of HPs would yield Ḡd , T , ` ( λ ) = 1 ∀ ` ∈ [ L ] where L is the number of layers in the network and T is the last epoch . In this case , network layer functions as a better autoencoder through iterative training and the carriage of information throughout the network is maximized . Conversely , Ḡd , T , ` ( λ ) = 0 indicates that the information flow is very weak such that the mapping is effectively random ( ‖Ed ‖ is maximized. ) . 3 Under review as a conference paper at ICLR 2021

Summary: The paper proposes an efficient framework to search for the optimal initial learning rate to train neural networks. The key idea is to introduce Knowledge Gain, a metric derived from the singular values of each layer, to indicate the convergency quality of training. Taking advantage of the metric, a logarithmic grid search algorithm (AutoHyper) is proposed to search for the optimal learning rate according to Eq 5 via short-time training (e.g. for 5 epochs), which is demonstrated to be very efficient and take effect to some extent.