Differentiable Hyper-parameter Optimization

Hyper-parameters are widely present in machine learning . Concretely , large amount of hyper-parameters exist in network layers , such as kernel size , channel size and the hidden layer size , which directly affect performance of the model . Thus , hyper-parameter optimization is crucial for machine learning . Current hyper-parameter optimization always requires multiple training sessions , resulting in a large time consuming . To solve this problem , we propose a method to fine-tune neural network ’ s hyper-parameters efficiently in this paper , where optimization completes in only one training session . We apply our method for the optimization of various neural network layers ’ hyper-parameters and compare it with multiple benchmark hyper-parameter optimization models . Experimental results show that our method is commonly 10 times faster than traditional and mainstream methods such as random search , Bayesian optimization and many other state-of-art models . It also achieves higher quality hyper-parameters with better accuracy and stronger stability . 1 INTRODUCTION . HPO ( hyper-parameter optimization ) is one of the most critical parts in auto-ML ( Thornton et al. , 2012 ; Domhan et al. , 2015 ; Kotthoff et al. , 2017 ) . Simple and low-dimensional hyper-parameters can be adjusted manually . It is also practical to use grid search or combine grid search with manual adjustment to deal with this simple problem ( Montavon et al. , 2012 ; gri , 2007 ; Hinton , 2010 ) . With the number of hyper-parameters continues increasing , manual tuning and grid search get ineffective . For a slightly large neural network model , we can use BO ( bayesian optimization ) ( Snoek et al. , 2012 ) or ZOOpt ( zeroth-order optimization ) ( Liu et al. , 2018 ) to optimize hyper-parameters . BO uses Gaussian process regression to fit mean and variance of objective function . However , a large number of matrix operations are needed in the fitting process , and multiple training sessions are needed to evaluate the predicted hyper-parameters . Time consuming increases along with each training session , and the total time consuming is positively correlated with the number of training sessions . Therefore , BO is still not suitable for large amount of hyper-parameters and so is ZOOpt . In addition to BO and ZOOpt , many evolutionary algorithms ( Young et al. , 2015 ) have also been applied . For example , genetic algorithm ( Goldberg ) is often used for HPO . Evolutionary algorithms can often avoid the problem of local optimal . However , a fatal flaw is the low time efficiency . When dealing with large-scale systems , random search ( ran , 2012 ) is a practical and more efficient method to solve HPO . Based on experiences , random search may perform better than BO in some cases with better time efficiency and accuracy . In addition , currently very effective HPO methods are probably HB ( Hyperband ( Li et al. , 2016 ) and DEHB ( Awad et al. , 2021 ) . They accelerate the convergence and make it twice faster than random search . Totally , all current models have a common defect which is a requirement for multiple training sessions of the fine-tuned model . While each training session is always expensive in time consuming . To get rid of this limit , we propose DHPO ( differentiable hyper-parameter optimization ) , which accomplish HPO within one training session . Concretely , we change the form of hyper-parameters and include them in the calculation of forward propagation to achieve differentiable hyper-parameters . In this way , hyper-parameters are differentiable with the goal of minimize the loss function . When the session is over , hyper-parameters complete optimization together with network ’ s parameters . Therefore , only one training session is needed in DHPO . The contributions of this paper are summarized as follows . • In this paper , we propose DHPO , which is the first model to solve hyper-parameter optimization in only one training session . • The proposed approach is universal . It is an idea for HPO which can be applied to various kinds of hyper-parameters in neural network . • We conduct extensive experiments on various kinds of neural network layers compared with multiple benchmark HPO models to evaluate the extreme efficiency and high performance of DHPO . In the remaining of this paper , Section 3 describes the proposed method . Experiments are conducted and analyzed in Section 4 . We overview related work in Section 5 . Section 6 draws the conclusions and future work . 2 BACKGROUND . In this section , we introduce related techniques briefly . Darts ( Liu et al. , 2019 ) is used to solve automatic network architecture search , which is described to determine the best operation from multiple candidate operators locally and globally . Take Figure 1 as example . It is selecting an activation for the fully connected layer from { Tanh , Sigmoid , Relu , Softmax } and supposing Relu is the best choice . Grid search costs four training sessions to evaluate each activation . While Darts can select the hyper-parameters in a single training session . Firstly , Darts calculates features X transformed by the fully connected layer and then uses each activation to process X once to get four non-linear features , denoted as o = [ o1 , o2 , o3 , o4 ] . Further , softmax ( α ) assigns a weight to each oi and aggregates weighted oi by summing them , where αi is a trainable parameter . An ideal train always ends up with max softmax ( α ) → 1 as shown in Figure 1 . Darts in Figure 1 finally determines Relu as the optimal activation with softmax ( alpha ) 3 very close to 1 . Accordingly , softmax ( α ) i ≈ 0 , i 6= 3 meaning a shield to the information contained in o1 , o2 , o4 , which is equivalent to using Relu only . 3 METHOD . As introduced in the background , Darts still has fatal defects . On the one hand , max softmax ( α ) doesn ’ t always converge to a level very close to 1 . The cases where multiple candidates occupy similar weights always exist especially for a large amount of candidates , which means Darts fails to distinguish different candidates and hit the optimal . On the other hand , Darts is only suitable for hyper-parameter with independent candidates . For example , the channel size is beyond Darts ’ s capability . If we ’ d like to apply Darts to solve channel size , we should set a candidate for each possible value of channel size . Then there will beO ( n2 ) channels in total , which is space expensive . Motivated by this , we propose DHPO , aiming at solving all these problems existing in Darts and current HPO models . In the following , we first take the optimization of convolution layer ’ s channel size as example to draw the core of DHPO in Section 3.1 . Then , we apply DHPO to more hyperparameters in Section 3.2 . 3.1 DIFFERENTIABLE HYPER-PARAMETER OPTIMIZATION . The target of DHPO is to make hyper-parameter θ differentiable , and we achieve it through constructing trainable parameter α to substitute θ in the training . Obviously , the core of the idea is to point out the limitations for α and give out a universal method to construct and apply α . In this section , we first declare the sufficient conditions for α to substitute θ based on Theorem 1 in Section 3.1.1 . Then we explain how to construct α and use it to control the structure of neural network , namely that is the forward propagation under α in Section 3.1.2 . In the end , we use Theorem 1 to prove that α is able to substitute θ as Theorem 2 in Section 3.1.3 . 3.1.1 LIMITATIONS FOR α Our target is to construct trainable parameter α to substitute θ in the training . We will point out the sufficient conditions for α in the following . Firstly , we define found sufficient conditions as a new relation between α and θ , which is defined as expressible as Definition 3.1 . Then , we prove that if α is expressible for θ , then we can replace θ with α in neural network as Theorem 1 . The definition and theorem are as follows . Definition 3.1 . Expressible : For a neural network F : Rd 7→ R with W and Θ as parameters and hyper-parameters respectively , hyper-parameter θ2 ∈ Ω2 is expressible for hyper-parameter θ1 ∈ Ω1 if and only if there is a surjective h : Ω2 7→ Ω1 ( θ1 = h ( θ2 ) ) which makes network structure under F ( W |θ1 ) and that under F ( W |θ2 ) are the same . Theorem 1 . If θ2 is expressible for θ1 , then we can replace θ1 with θ2 in neural network . Proof . If θ2 is expressible for θ1 , then there is a surjective from θ2 to θ1 . Therefore , ∀θ1 ∈ Ω1 , ∃θ2 ∈ Ω2 , h ( θ2 ) = θ1 . Meanwhile , F ( W |θ2 ) and F ( W |θ1 ) share the same structures according to Definition 3.1 . So we can replace θ1 with θ2 in a neural network . According to Theorem 1 and Definition 3.1 , we can replace θ with α in neural network as long as the constructed α satisfies the following two conditions . One is that there is a surjective θ = h ( α ) . The other is that the network structure controlled by α is the same to that of θ under θ = h ( α ) . 3.1.2 FORWARD PROPAGATION UNDER α In this section , we first propose the method to construct expressible α as described above , and then explain how α controls the structure of neural network , i.e. , applying α in the forward propagation . φ ( X|α ) = sup ( θ ) ∑ i=1 i ∗ ( X ? κi ) = σ ( ( softmax ( α ) ) ·A− σ ( β ) ) ∗ a ) ( 1 ) We take convolution layer ’ s channel size as example to construct trainable α , which should be expressible for θchannel size ( abbreviated as θ ) . We construct α as α = [ α1 , · · · , αsup ( θ ) ] and α ∈ Rsup ( θ ) , sup ( θ ) = max θ . Formula 1 describes the forward propagation in this convolution layer under α. X ∈ RW×H is the input of convolution layer which is a single-channel image . κi represents the convolution kernel of i-th channel , and we prepare sup ( θ ) candidate kernels in the initialization of neural network . ( ? ) represents convolution operation and ( · ) represents matrix multiplication . A = [ 1 · · · 1 . . . . . . 0 1 ] T is a lower triangular matrix . γ ∈ R is a trainable parameter and a ∈ R is a large constant . Obviously , the core of formula 1 is the calculation of . We visualize the calculation in Figure 2 . Two cases are given in this figure . We take the left as example . α is first transformed by softmax activation and 0 < softmax ( α ) i < 1 . Then we get A by multiplying matrix softmax ( α ) and matrixA , A1 = 1 and Ai > Aj if i > j . Next , we subtract σ ( γ ) from Ai and multiply it by a large constant a , a > 1 . In this time , −a < ( Ai − σ ( γ ) ) ∗ a < a , and it maintains the same monotonicity as A . In the last step , we use σ again to map ( Ai − σ ( γ ) ) ∗ a to a value close to 1 or 0 . Finally , i = σ ( ( Ai−σ ( γ ) ) ∗a ) . In this example , 1 , 2 , 3 are very close to 1 , and the others are very close to 0 . In this occasion , we think that the channel size is 3 . Based on the above construction , α has the same effect as θchannel size . Thus , it is reasonable to substitute θchannel size with α . We will prove this in the following Section 3.1.3 . 3.1.3 EXPRESSIBLE FOR θ The goal is to prove that we can substitute θchannel size with α . According to Theorem 1 , we just need to prove that α is expressible for θchannel size . Before proof , we first list three key properties of in the following which are necessary to the proof . is constructed from α and has three key characteristics . ¬ 1 ≈ 1 . 1 > · · · > sup ( θ ) . ® There is an index t dividing into two sets εbig = { i|1 ≤ i ≤ t } and εsmall = { i|t+ 1 ≤ i ≤ sup ( θ ) } . The items in εbig are all close to 1 and items in εsmall are all close to 0 . The first property ensures that at least one channel is selected . The second property ensures the selected channels are always the first few rather than random several in candidate channels . The third property makes unselected channels blocked away , especially when a 7→ ∞ . Based on these features , we have Theorem 2 . Theorem 2 . When a 7→ ∞ , α is expressible for θ under h ( α ) = rounded ∑ . Proof . If we would like to prove α is expressible for θ , we just need to prove h ( α ) is surjective and network structure controlled by α is the same to that of θ under θ = h ( α ) according to the discussion in Section 3.1.1 . Obviously , θ = h ( α ) is a surjective . We then prove convolution layers under θ and α have same structure . Suppose that the network structure controlled by θ is a convolution layer composed of the first θ channels from candidates . As for the structure controlled by α , we can also see it as a convolution layer composed of the first θ channels according to the third characteristic of . Because noise from unselected channels will fade out with a→∞ . At this time , F ( X|α ) ⇔ F ( X|θ ) . Based on above discussions , we finally achieve the differentiable channel size . Since α is trainable and expressible for θchannel size . Then we can replace θchannel size withα in this convolution layer . When the training session is over , we take rounded ∑ as the optimized θchannel size .

Summary:  This paper introduces a method for hyper-parameter optimization (HPO) for deep neural networks. Its main idea is to replace hyper-parameters by trainable parameters, which can be included in the training process of the network itself. The proposed method is applied to hyper-parameter tunning for two types of networks, namely CNN and FNN on two datasets MNIST and SVHN. It is also compared with several baselines.