Differentiable Hyper-parameter Optimization

Hyper-parameters are widely present in machine learning . Concretely , large amount of hyper-parameters exist in network layers , such as kernel size , channel size and the hidden layer size , which directly affect performance of the model . Thus , hyper-parameter optimization is crucial for machine learning . Current hyper-parameter optimization always requires multiple training sessions , resulting in a large time consuming . To solve this problem , we propose a method to fine-tune neural network ’ s hyper-parameters efficiently in this paper , where optimization completes in only one training session . We apply our method for the optimization of various neural network layers ’ hyper-parameters and compare it with multiple benchmark hyper-parameter optimization models . Experimental results show that our method is commonly 10 times faster than traditional and mainstream methods such as random search , Bayesian optimization and many other state-of-art models . It also achieves higher quality hyper-parameters with better accuracy and stronger stability . 1 INTRODUCTION . HPO ( hyper-parameter optimization ) is one of the most critical parts in auto-ML ( Thornton et al. , 2012 ; Domhan et al. , 2015 ; Kotthoff et al. , 2017 ) . Simple and low-dimensional hyper-parameters can be adjusted manually . It is also practical to use grid search or combine grid search with manual adjustment to deal with this simple problem ( Montavon et al. , 2012 ; gri , 2007 ; Hinton , 2010 ) . With the number of hyper-parameters continues increasing , manual tuning and grid search get ineffective . For a slightly large neural network model , we can use BO ( bayesian optimization ) ( Snoek et al. , 2012 ) or ZOOpt ( zeroth-order optimization ) ( Liu et al. , 2018 ) to optimize hyper-parameters . BO uses Gaussian process regression to fit mean and variance of objective function . However , a large number of matrix operations are needed in the fitting process , and multiple training sessions are needed to evaluate the predicted hyper-parameters . Time consuming increases along with each training session , and the total time consuming is positively correlated with the number of training sessions . Therefore , BO is still not suitable for large amount of hyper-parameters and so is ZOOpt . In addition to BO and ZOOpt , many evolutionary algorithms ( Young et al. , 2015 ) have also been applied . For example , genetic algorithm ( Goldberg ) is often used for HPO . Evolutionary algorithms can often avoid the problem of local optimal . However , a fatal flaw is the low time efficiency . When dealing with large-scale systems , random search ( ran , 2012 ) is a practical and more efficient method to solve HPO . Based on experiences , random search may perform better than BO in some cases with better time efficiency and accuracy . In addition , currently very effective HPO methods are probably HB ( Hyperband ( Li et al. , 2016 ) and DEHB ( Awad et al. , 2021 ) . They accelerate the convergence and make it twice faster than random search . Totally , all current models have a common defect which is a requirement for multiple training sessions of the fine-tuned model . While each training session is always expensive in time consuming . To get rid of this limit , we propose DHPO ( differentiable hyper-parameter optimization ) , which accomplish HPO within one training session . Concretely , we change the form of hyper-parameters and include them in the calculation of forward propagation to achieve differentiable hyper-parameters . In this way , hyper-parameters are differentiable with the goal of minimize the loss function . When the session is over , hyper-parameters complete optimization together with network ’ s parameters . Therefore , only one training session is needed in DHPO . The contributions of this paper are summarized as follows . • In this paper , we propose DHPO , which is the first model to solve hyper-parameter optimization in only one training session . • The proposed approach is universal . It is an idea for HPO which can be applied to various kinds of hyper-parameters in neural network . • We conduct extensive experiments on various kinds of neural network layers compared with multiple benchmark HPO models to evaluate the extreme efficiency and high performance of DHPO . In the remaining of this paper , Section 3 describes the proposed method . Experiments are conducted and analyzed in Section 4 . We overview related work in Section 5 . Section 6 draws the conclusions and future work . 2 BACKGROUND . In this section , we introduce related techniques briefly . Darts ( Liu et al. , 2019 ) is used to solve automatic network architecture search , which is described to determine the best operation from multiple candidate operators locally and globally . Take Figure 1 as example . It is selecting an activation for the fully connected layer from { Tanh , Sigmoid , Relu , Softmax } and supposing Relu is the best choice . Grid search costs four training sessions to evaluate each activation . While Darts can select the hyper-parameters in a single training session . Firstly , Darts calculates features X transformed by the fully connected layer and then uses each activation to process X once to get four non-linear features , denoted as o = [ o1 , o2 , o3 , o4 ] . Further , softmax ( α ) assigns a weight to each oi and aggregates weighted oi by summing them , where αi is a trainable parameter . An ideal train always ends up with max softmax ( α ) → 1 as shown in Figure 1 . Darts in Figure 1 finally determines Relu as the optimal activation with softmax ( alpha ) 3 very close to 1 . Accordingly , softmax ( α ) i ≈ 0 , i 6= 3 meaning a shield to the information contained in o1 , o2 , o4 , which is equivalent to using Relu only . 3 METHOD . As introduced in the background , Darts still has fatal defects . On the one hand , max softmax ( α ) doesn ’ t always converge to a level very close to 1 . The cases where multiple candidates occupy similar weights always exist especially for a large amount of candidates , which means Darts fails to distinguish different candidates and hit the optimal . On the other hand , Darts is only suitable for hyper-parameter with independent candidates . For example , the channel size is beyond Darts ’ s capability . If we ’ d like to apply Darts to solve channel size , we should set a candidate for each possible value of channel size . Then there will beO ( n2 ) channels in total , which is space expensive . Motivated by this , we propose DHPO , aiming at solving all these problems existing in Darts and current HPO models . In the following , we first take the optimization of convolution layer ’ s channel size as example to draw the core of DHPO in Section 3.1 . Then , we apply DHPO to more hyperparameters in Section 3.2 . 3.1 DIFFERENTIABLE HYPER-PARAMETER OPTIMIZATION . The target of DHPO is to make hyper-parameter θ differentiable , and we achieve it through constructing trainable parameter α to substitute θ in the training . Obviously , the core of the idea is to point out the limitations for α and give out a universal method to construct and apply α . In this section , we first declare the sufficient conditions for α to substitute θ based on Theorem 1 in Section 3.1.1 . Then we explain how to construct α and use it to control the structure of neural network , namely that is the forward propagation under α in Section 3.1.2 . In the end , we use Theorem 1 to prove that α is able to substitute θ as Theorem 2 in Section 3.1.3 . 3.1.1 LIMITATIONS FOR α Our target is to construct trainable parameter α to substitute θ in the training . We will point out the sufficient conditions for α in the following . Firstly , we define found sufficient conditions as a new relation between α and θ , which is defined as expressible as Definition 3.1 . Then , we prove that if α is expressible for θ , then we can replace θ with α in neural network as Theorem 1 . The definition and theorem are as follows . Definition 3.1 . Expressible : For a neural network F : Rd 7→ R with W and Θ as parameters and hyper-parameters respectively , hyper-parameter θ2 ∈ Ω2 is expressible for hyper-parameter θ1 ∈ Ω1 if and only if there is a surjective h : Ω2 7→ Ω1 ( θ1 = h ( θ2 ) ) which makes network structure under F ( W |θ1 ) and that under F ( W |θ2 ) are the same . Theorem 1 . If θ2 is expressible for θ1 , then we can replace θ1 with θ2 in neural network . Proof . If θ2 is expressible for θ1 , then there is a surjective from θ2 to θ1 . Therefore , ∀θ1 ∈ Ω1 , ∃θ2 ∈ Ω2 , h ( θ2 ) = θ1 . Meanwhile , F ( W |θ2 ) and F ( W |θ1 ) share the same structures according to Definition 3.1 . So we can replace θ1 with θ2 in a neural network . According to Theorem 1 and Definition 3.1 , we can replace θ with α in neural network as long as the constructed α satisfies the following two conditions . One is that there is a surjective θ = h ( α ) . The other is that the network structure controlled by α is the same to that of θ under θ = h ( α ) . 3.1.2 FORWARD PROPAGATION UNDER α In this section , we first propose the method to construct expressible α as described above , and then explain how α controls the structure of neural network , i.e. , applying α in the forward propagation . φ ( X|α ) = sup ( θ ) ∑ i=1 i ∗ ( X ? κi ) = σ ( ( softmax ( α ) ) ·A− σ ( β ) ) ∗ a ) ( 1 ) We take convolution layer ’ s channel size as example to construct trainable α , which should be expressible for θchannel size ( abbreviated as θ ) . We construct α as α = [ α1 , · · · , αsup ( θ ) ] and α ∈ Rsup ( θ ) , sup ( θ ) = max θ . Formula 1 describes the forward propagation in this convolution layer under α. X ∈ RW×H is the input of convolution layer which is a single-channel image . κi represents the convolution kernel of i-th channel , and we prepare sup ( θ ) candidate kernels in the initialization of neural network . ( ? ) represents convolution operation and ( · ) represents matrix multiplication . A = [ 1 · · · 1 . . . . . . 0 1 ] T is a lower triangular matrix . γ ∈ R is a trainable parameter and a ∈ R is a large constant . Obviously , the core of formula 1 is the calculation of . We visualize the calculation in Figure 2 . Two cases are given in this figure . We take the left as example . α is first transformed by softmax activation and 0 < softmax ( α ) i < 1 . Then we get A by multiplying matrix softmax ( α ) and matrixA , A1 = 1 and Ai > Aj if i > j . Next , we subtract σ ( γ ) from Ai and multiply it by a large constant a , a > 1 . In this time , −a < ( Ai − σ ( γ ) ) ∗ a < a , and it maintains the same monotonicity as A . In the last step , we use σ again to map ( Ai − σ ( γ ) ) ∗ a to a value close to 1 or 0 . Finally , i = σ ( ( Ai−σ ( γ ) ) ∗a ) . In this example , 1 , 2 , 3 are very close to 1 , and the others are very close to 0 . In this occasion , we think that the channel size is 3 . Based on the above construction , α has the same effect as θchannel size . Thus , it is reasonable to substitute θchannel size with α . We will prove this in the following Section 3.1.3 . 3.1.3 EXPRESSIBLE FOR θ The goal is to prove that we can substitute θchannel size with α . According to Theorem 1 , we just need to prove that α is expressible for θchannel size . Before proof , we first list three key properties of in the following which are necessary to the proof . is constructed from α and has three key characteristics . ¬ 1 ≈ 1 . 1 > · · · > sup ( θ ) . ® There is an index t dividing into two sets εbig = { i|1 ≤ i ≤ t } and εsmall = { i|t+ 1 ≤ i ≤ sup ( θ ) } . The items in εbig are all close to 1 and items in εsmall are all close to 0 . The first property ensures that at least one channel is selected . The second property ensures the selected channels are always the first few rather than random several in candidate channels . The third property makes unselected channels blocked away , especially when a 7→ ∞ . Based on these features , we have Theorem 2 . Theorem 2 . When a 7→ ∞ , α is expressible for θ under h ( α ) = rounded ∑ . Proof . If we would like to prove α is expressible for θ , we just need to prove h ( α ) is surjective and network structure controlled by α is the same to that of θ under θ = h ( α ) according to the discussion in Section 3.1.1 . Obviously , θ = h ( α ) is a surjective . We then prove convolution layers under θ and α have same structure . Suppose that the network structure controlled by θ is a convolution layer composed of the first θ channels from candidates . As for the structure controlled by α , we can also see it as a convolution layer composed of the first θ channels according to the third characteristic of . Because noise from unselected channels will fade out with a→∞ . At this time , F ( X|α ) ⇔ F ( X|θ ) . Based on above discussions , we finally achieve the differentiable channel size . Since α is trainable and expressible for θchannel size . Then we can replace θchannel size withα in this convolution layer . When the training session is over , we take rounded ∑ as the optimized θchannel size .

Summary: This paper proposes to tune hyperparameters in a differentiable way by using a modified version of softmax function. It covers the tuning for three kinds of hyper-parameters as examples: channel size, kernel size and hidden layer size. Experiments on MNIST and SVHN (which are small scale datasets by modern DNN standards) show improvements on previous methods. The proposed method itself is reasonable, however, this paper misses many previous works on this topic, which makes it hard to appreciate its novelty.