Differentiable Hyper-parameter Optimization
Authors: Bozhou Chen
Hongzhi Wang
Chenmin Ba
Venue: ICLR 2022 Submitted (2022)
Keywords: Hyper-parameter Optimization

Abstract: Hyper-parameters are widely present in machine learning.
Concretely, large amount of hyper-parameters exist in network layers, such as kernel size, channel size and the hidden layer size, which directly affect performance of the model.
Thus, hyper-parameter optimization is crucial for machine learning. Current hyper-parameter optimization always requires multiple training sessions, resulting in a large time consuming.
To solve this problem, we propose a method to fine-tune neural network's hyper-parameters efficiently in this paper, where optimization completes in only one training session.
We apply our method for the optimization of various neural network layers' hyper-parameters and compare it with multiple benchmark hyper-parameter optimization models.
Experimental results show that our method is commonly 10 times faster than traditional and mainstream methods such as random search, Bayesian optimization and many other state-of-art models. It also achieves higher quality hyper-parameters with better accuracy and stronger stability.

Under review as a conference paper at ICLR 2022
DIFFERENTIABLE HYPER-PARAMETER OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Hyper-parameters widely present in machine learning. Concretely, large amount of hyper--arameterr exist in network layers, such as kernel size, channel size and the hidden layer size, which directly affect performance of the model. Thus, paper-par optimization is crucial for machine learning. Current hyper-parameter optimization always requires multiple training sessions, result- ing in large time consuming. To solve this problem, we propose a method to fine-tune neural network's hyper-parameters efficiently in this paper, where op- timization completes in only one training session. We apply our method for the optimization of various neural network layers' sxper-parameters and compare it with multiple benchmark sults show that our method is commonly 10 times faster than traditional and main- stream methods such as random search, Bayesian optimization and many other state-of-art models. It also achieves higher quality parra-pprreeters with better accuracy and stronger stability.
optimization models. Experimental re-
INTRODUCTION
HPO(hyper-paramets optimization) is one of the most critical parts in auto-MI. (hornton et al., 2012 Domhan et al., 2015; Kotthoff al. 2017). Simple and low-dimensional hy hyper-parameters can be adjusted manually. It is also practical to use grid search or combine grid search with manual
adjustment to deal with this simple problem (Montavon et al. 2012 gri 2007: Hinton 2010).
With the number of hyper-parameters continues increasing, manual tuning and grid search get inef- fective. For slightly large neural network model, we can use BObayssiaan optimization) (Snoek) et al. 2012) Z00pt(zeroth-ordl optimization) (Liu et al. 2018) to optimize hyper-paune apper-parameters.. BO uses Gaussian process regression to fit mean and variance of objective function. However, a large number of matrix operations are needed in the fitting process, and multiple training sessions are needed to evaluate the predicted training session, and the total time consuming is positively correlated with the number of training sessions. Therefore, BO is still not suitable for large amount of hyper-parameters and so is ZOOpt.
Time consuming increases along with each
In addition to BO and Z0Opt, many evolutionary algorithms (Young al. 2015) have also been applied. For example, genetic algorithm (Goldberg) is often used for HPO. Evolutionary algorithms can often avoid the problem of local optimal. However, a fatal flaw the low time efficiency.
When dealing with large-scale systems, random search (ran 2012) is practical and more efficient method solve HPO. Based on experiences, random search may perform better than BO in some cases with better time efficiency and accuracy. In addition, currently very effective HPO methods are probably HB(Hyperband (Li al. 2016) and DEHB (Awad et al. 202T). They accelerate the convergence and make it twice faster than random search.
Totally, all current models have a common defect which a requirement for multiple training ses- sions of the fine-tuned model. While each training session is always expensive in time consuming. To get rid of this limit, we propose DiFodiifferentiable hyper-parameter optimization), which ac- complish HPO within one training session. Concretely, we change the form of include them the calculation of forward propagation to achieve differentiable hyper-paramen parameters. differentiable with the goal of minimize the loss function. When complete optimization together with network's parameters.
and
In this way, hyper-parameed the session is over, Therefore, only training session is needed in DHPO.

---

Under review as a conference paper at ICLR 2022
The contributions of this paper summarized follows.
In this paper, we propose DHPO, which the first model to solve hyper-parameter optimization in only one training session.
The proposed approach is universal. is an idea for HPO which can be applied to various kinds of
neural network.
We conduct extensive experiments on various kinds of neural network layers compared with mul- tiple benchmark HPO models to evaluate the extreme efficiency and high performance of DHPO.
In the remaining of this paper, Section the proposed method. Experiments are conducted and analyzed Section We overview related work in Section Section 6 draws the conclusions and future work.
2 BACKGROUND
candidate operators
Tanh
O so tmtx /ttxama(aaaa a2 ul (2, E as features
Sigmoid
ideal train with selectrede
231
Relu
000
Softmax
imput layer E Rm hidden layer E Rd
Figure 1: Select an activation with Darts
In this section, we introduce related techniques briefly.
Darts (Liu et al. 2019) is used to solve automatic network architecture search, which is de- scribed determine the best operation from multiple candidate operators locally and globally. Take Figure [ as example. It is selecting an activation for the fully connected layer from (Tanh, Sigmoid, Relu, Softmax)
supposing Relu is best choice. Grid search costs four training sessions evaluate each activation. While Darts select the hyper-parameters in a single training session. Firstly, Darts calculates features X transformed by the fully con- nected layer and then uses each activation process X once to get four non-linear features, de- noted as o = [o1, 02, 03, 0z). Further, so softmar(a)) assigns weight to each O, and aggregates weighted
summing them, where a; is trainable parameter. An ideal train always ends up with max sof softmaz(() + 1 as shown in Figure | Darts in Figure finallyy determines Relu as the optimal activation with so/ so lmmaar alpha)s very close 1. Accordingly, sofflooa (maa((a) (o); i meaning a shield to the information contained in O2, which is equivalent to using Relu only.
3 METHOD
As introduced in the background, Darts still has fatal defects. On the one hand, max so fooftaaaaa doesn't always converge to level very close to The cases where multiple candidates occupy
2

---

Under review as a conference paper at ICLR 2022
similar weights always exist especially for large amount of candidates, which means Darts fails to distinguish different candidates and hit the optimal. On the other hand, Darts is only suitable for aaper-prraetter with independent candidates. For example, the channel size is beyond Darts's 's capability. If we'd like to apply Darts to solve channel size, we should set a candidate for each possible value of channel size. Then there will be On channels in total, which is space expensive.
Motivated by this, we propose DHPO, aiming at solving all these problems existing in Darts and current HPO models. In the following, we first take the optimization of convolution layer's channel size as example to draw the core of DHPO in Section 3.1 Then, we apply DHPO to more hyper- parameters Section 3.2
3.1 DIFFERENTIABLE HYPER-PARAMETE PARAMETER OPTIMIZATION
The target of DHPO is to make hyper-parameter 0 differentiable, and we achieve it through con- structing trainable parameter a to substitute in the training. Obviously, the core of the idea is to point out the limitations for and give out universal method to construct and apply a.
In this section, we first declare the sufficient conditions for a to substitute based on Theorem in Section 3.1.1] Then we explain how to construct a and use it to control the structure of neural network, namely that is the forward propagation under a in Section 3.2 In the end, we use Theorem 1 to prove that is able substitute Theorem in Section 3.1.3
3.1.1 LIMITATIONS FOR a
Our target is to construct trainable parameter a to substitute 0 in the training. We will point out the sufficient conditions for a in the following.
Firstly, we define found sufficient conditions as new relation between and 0 which is defined as expressible as Definition 3.T Then, we prove that if a is expressible for then we can replace with a in neural network Theorem [ The definition and theorem are follows.
Definition 3.1. Expressible: For a neural network F Rdd +> TR with W and θ as parameters and hyper-parameters respectively, hyper-parameter 02 E S2g is expressible for hyper-parameter 01 E Si if and only if there is surjective h G2 + O (1 = h(02)) which makes network structure under F(W102) and that under F(W)02) are the same.
Theorem 1. 0f02 is expressible for 0, then we can replace 0₁ with 02 in neural network.
Proof. 02 is expressible for 0,, then there is a surjective from Gg to 01. Therefore, V01 E Q2, e Q2 h(O)) 01. Meanwhile, F(W10, and F(W (01) share the same structures according Defi- nition 3.1 So we can replace 0, with 0g in a neural network.
we can replace 9 with a in neural network as long as the constructed a satisfies the following two conditions. One is that there is surjective 0 = h(a). The other is that the network structure controlled by a is the same to that of under 9 = h(a))
According to Theorem I and Definition 3.1
3.1.2 FORWARD PROPAGATION UNDER &
In this section, we first propose the method to construct expressible as described above, and then explain how a controls the structure of neural network, i.e., applying a in the forward propagation.
sup(O) Σ
o ((X(a):
E (X * ki))
c= ==((sfftmaz()) A- o(B)) + a)
We take convolution layer's channel size as example to construct trainable a, which should be expressible for Dehanneleeise (abbreviated (attoneeeiseeabbbbiiieeee
[aj,,.. [e))(())] and a E R=up()), sup(f) = max f. Formula 1 describes the forward propagation in this convolu- tion layer under X R W is the imput of convolution layer which is single-channel image. N; represents the convolution kernel i-th channel, and we prepare sup(f) candidate kernels in the initialization of neural network. (x) represents convolution operation and (.) represents matrix
0). We construct
3

---

Under review as a conference paper at ICLR 2022
multiplication. A
is lower triangular matrix. Y trainable parameter and
E R is a large constant.
Obviously, the core formula the calculation of visualize the calculation in Figure Two cases given in this figure. We take the left as example. is first transformed by softmax activation and < softmax [);; < Then we get A by multiplying matrix so softma((a) and matrix A. Ar = 1 and A, > Aj i > Next, we subtract (r7) from A, and multiply it by large constant a. a > 1. In this time, -a < (A, o(7)) aa < a, and it maintains the same monotonicity as A. In the last step, = ([(A) * a). In this example, é 62, ea are very close to 1, and the others are very close to O. In this occasion, think that the channel size is 3.
again to map (A, a(7)
to value close
0. Finally,
M
K1
W
M
softmax(u)
softmax (a)
A a so s0ftmaa (m) A
A = so softmax(a) A
(Al o())
==3
(A-
0 6
((A o()) a)
a(A a(r)) a)
Figure 2: Forward propagation under a in DHPO.
Based on the above construction, has the same effect Achannel size- Thus, it is reasonable to substitute Ochannet with a. We will prove this in the following Section 3.1.3
3.1.3 EXPRESSIBLE FOR 0
The goal is to prove that we can substitute Ochannel_size with a. According to Theorem I we just need to prove that is expressible for DehanneLesse: Before proof, first list three key properties of e in the following which are necessary to the proof.
is constructed from and has three key characteristies. De 2 > Esup(0). 3 There is an index / dividing into two sets Ebig {<i/1 S i () and Emmall {eil + sup(O)): ). The items Stig are all close to and items Ssmalt are all close to The first property ensures that at least one channel is selected. The second property ensures the selected channels are always the first few rather than random several in candidate channels. The third property makes unselected channels blocked away, especially when + oc. Based on these features, have Theorem 2 Theorem 2. When + oc, is expressible for 0 under f((a)= rounded Ee.
Proof. If we would like to prove a is expressible for 6, we just need to prove h(aa) is surjective and network structure controlled by is the same to that of 0 under 0 h(a:) according to the discussion in Section[...II
Obviously, 0 h((a) is surjective. We then prove convolution layers under 0 and have same
4

---

Under review as a conference paper at ICLR 2022
structure. Suppose that the network structure controlled by 8 is convolution layer composed of the first 8 channels from candidates. As for the structure controlled by a. we can also see as a convolution layer composed of the first 6 channels according to the third characteristic of Because noise from unselected channels will fade out with a - co. At this time, F(X(o)) e F(. IT
Based on above discussions, we finally achieve the differentiable channel size. Since a is trainable and expressible for channel size. Then
replace 0 channel size with a in this convolution layer.
When the training session is over, we take rounded Σ as the optimized @ehannel.size-
3.2 MULTIPLE HYPER- HYPER-PARAMETER PARAMETERS OPTIMIZATION
Based on the idea discussed in Section 3.1 we can apply DHPO to large amount of hyper- parameters rather than just channel size.
In this section, we select two widely used and typical hyper-parametee explain how deal with them in DHPO, and other hyper-parann parameters can be processed in a similar way. These typical hyper- parameters are convolution kernel size and hidden layer size.
3.2.1 KERNEL SIZE
In this section, we introduce method of making kernel size differentiable. The challenge here that we should deal with multi-dimensional hyper-parameter. Therefore, we should make DHPO suitable for multi-dimensional
peper-prrmmtter rather than just one dimensional hyper-parametcr like channel size. We achieve this by extending to multiple dimensions and the detail is as follows.
Firstly, we construct e from a as described Section 3.1 and a candidate kernel K. For the one- dimensional kernel, x E (Reup(0aarnd..is)) (Okernet size abbreviated as 0). And we use x O 6 as the kernel in forward propagation. For the 2-dimensional kernel K E (esup(e)xeupti we use ④ M as
61
62
(up(())
E2
E2
Esup(0)
selected kernel, M
In this way, the features from unselected
Esup(O) items E; are blocked away since E; O.
(Esup(0)
(O)
Then a is expressible for 0 under h(a) = rounded E. Obviously, h(()) is a surjective. When the large constant - 0o. In this time, kernel size under and that under the same under 0 = h(a). So we can substitute 0 @kernel.size with in neural network. When the training session is over, we take rounded Σ as the optimized Okernel sise.
Based on convolution kernel size, we can design differentiable hyper-paramett for pooling kernel size, convolution stride and etc. in a similar way.
3.2.2 HIDDEN LAYER SIZE
In this section, we apply DHPO to one of the most common and important hyper parpmeerrramtters... hidden layer size. Commonly, the hidden layer size has thousands of candidates. Therefore, it is difficult for sample-based models such as BO and random search. In contrast, DHPO can solve it easily both theoretically and practically since hidden layer size is differentiable in DSA.
Again, we first construct from a as described above and use to control the hidden layer size. We use (X- W + b) @ as the output of the fully connected layer. (X (X-W+ W +b) is the original output and In this way, unselected last few features will blocked away, which is equivalent to use smaller hidden layer size. When the training session is over, we take rounded Σ as optimized @pidden.aize-
its dimension is sup(0psideen.iia
4 EXPERIMENT
To verify the effectiveness of the proposed approach, we conduct HPO on two different neural net- works with multiple datasets, and compare it with various baselines. Section 4.1 briefly introduces
5

---

Under review as a conference paper at ICLR 2022
the neural network to be tuned, Section 4.2 gives the basic settings, Section 4.3 displays the experi-
mental results, and Section 4.4 conducts a case study to show the DHPO tuning process.
4.1
MODEL TO BE TUNED
9028x288 8828:28 8828:28
8828/28
6026x28 8828:28 8828x28 8028,28
1x64
1825/28
1x12
imput
conv block
pool
conv block 2
pool
faltten
linear block
Figure 3: Deep convolution neural network to be tuned
In the experiment, apply DHPO to two classical models, i.e. deep convolutional neural networks and MI.P(multi-layer perceptron), which include various kinds of hyper-parameters, DNN(deep convolution neural network) is lightweight convolution network designed with reference to Kuni- hiko & Fukushima (1980); Lecun & Bottou (198);; Behnke & Sven (2003). Even though DNN is not the best model to process images, it is a good network which can distinguish the performence of various HPO approaches since has large number of syper-parameters DNN's network structure as shown in Figure 3 which a lightweight implementation. And MLP is composed fully connected layers. DNN used to evaluate HPO for convolution layer and pooling layer. MLP is used to evaluate HPO for fully connected layer.
4.2 EXPERIMENTAL SETTINGS
In this section, we introduce the experimental settings.
4.2.1 DATASET
MNIST (LeCun & Cortes 2010 and SVHN (sv 2011) are well known lightweight validation im- age datasets, and they are suitable for DNN. Therefore, we use MNIST and SVHN the benchmark datasets on DNN. On MIP, we choose iris, wine, car and agaricus-lepiot! Basic message of the six datasets is shown in Appendix
4.2.2 BASELINE
The mainstream algorithms used for HPO contains random search, BO, evolutionary algorithm and so on. We finally choose the following HPO algorithm as baselines:
Random search tran 2012) is the most universal HPO algorithm, which can randomly select points the entire search space. So is able to break the limit of local optimal solution.
Zoopt (Liu et al., 2018) does not rely the gradient of the objective function, but learns from samples of the search space instead. It is suitable for HPO tasks that are not differentiable, with many local minimal even unknown but only testable.
Bayes optimization (Snoek et al. (012) is based on Gaussian process regression to fit the distri- bution between samples and objective. It has a practical effect and wide range of applications.
Genetic algorithm (Goldberg) belongs to evolutionary algorithm. relying on the mutation and genetic of maintained population. It often used in HPO and NAS(network architecture search). Particle swarm algorithm (Kennedy & Eberhart 1995) also belongs to evolutionary algorithm, which finds the optimal solution through collaboration and information sharing between individ- uals in the group. Simple implementation, few hyperparameters, and wide applications.
HyperBand (Li et al. 2016) regards the process of finding the optimal hyper-parameee parameters as non-random exploration
the infinite arm bandit under the condition of limited resources. DEHB (Awad et al.) (202]) is developed based on differential evolution and HB, which furnished with better time performance and convergence effect.
https://archise iss uci eduu dataasets

---

Under review as a conference paper at ICLR 2022
4.2.3 METRICS
Time consuming reflects the time interval from when data and model are loaded on the GPU to when the algorithm stops. If data set is loaded multiple times, the loading time is also included.
Accuracy reflects the performance of neural network generated by HPO models.
Mean of top-K accuracy. HPO model always gives multiple sets of hyper-parameters and corre- sponding accuracy. We calculate the mean accuracy of top-K models.
Loss sequence reflects the state of objective and convergence effect in a training session.
4.2.4 PARAMETER SETTINGS
Tuned hyper-parameters We fine-tune convolution layer's channel size, convolution layer's ker- nel size, pooling layer's pooling type and pooling layer's kernel size in DNN model, totally 16 hyper-parameters. Among them, channel size [1, 8], kernel size [2,5], pooling type [MaxPool, Pool Additionally, 3 hidden layer size are optimized in MLP.
DNN We uses the Adam optimizer with learning rate 0.001 and uses cross-entropy loss function. batch size = 500 and each training session contains 30 epochs. MLP All are the same to DNN except for 150 epochs in training session.
BO BO model is initialized with 5 samples, and continues iterating for 25 rounds. Z00pt Iteration budgets is 30, and other syper-parameteed algorithm iterates 30 times. Evolutionary algorithm The population size and iterations are both 10, and mutation probability is 0.001. HB Iterate 81 times keep fair with other models, because of its different iteration principles. DEHB We set budget and max budget as 2 and 50, respectively.
the built-in configurations [ Random search The
Considering the influence of parameter's initialization, three-times repeated independent experi- ment is conducted and our experiments are conducted on one GTX 3060Ti GPU.
4.3 EXPERIMENTAL RESULTS
Table 1: Time consuming and accuracy on MNIST and SVHN. Column 3-9 corresponds to various statistical accuracy(%). denotes that repeat DHPO 10 times, and represents once DHPO with three repeated independent experiments. BLANK corresponds to the blank group without any hyper-parameter optimization by setting all the channel size and set all the kernel size 3.
ALL MEAN
ALL
TOPS
TOPS
TOPI0
TOP10
MODEL
TIME(S)
TOPI
STD
MEAN
STD
MEAN
STD
MNIST
BO
6,298
98.66
97.32
1.14
98.51
0.11
98.33
0.2
RAND
6,308
98.45
95.81
2.58
98.1
0.19
97.78
0.39
13,918
96.17
82.82
13.04
0.05
98.37
DEHB
6,073
98.37
96.7
1.58
98.19
0.11
97.95
0.28
DHPO+
1,708
99.01
98.57
0.19
98.7
0.05
98.57
0.19
DHPO1
98.71
BLANK
98.27
BO
11,043
87.29
83.3
3.95
86.81
0.27
$6.49
0.38
700pt
10,215
85.53
82.73
2.91
85.43
0.08
85.09
0.4
RAND
10.193
86.18
79.81
85.37
0.87
84.12
DEHB
10,194
85.61
81.18
4.51
83.15
0.28
84.45
0.73
DHPO
2,811
87.44
85.22
86.78
0.53
85.22
DHPO1
1,070
85.05
BLANK
84.95
As shown in Table [ DHPO always complete tasks faster than any other HPO model several times with ensuring good performance. Taking MNIST as example, DHPO* wins the game by taking only 28% of DEHB's time consuming. Additionally, DHPOj can also in accuracy with fewer repeats, which means DHPO is stable. The stability can also be reflected from top-K accuracy's standard deviation, which is only 0.19 for all samples under DHPO.
the advantage
IIn DHPO, we invoke Darts to solve pooling type and we have proved it's effective here in Appendix B httpp::/ thu.. cww/ ixi r/x00pt

---

Under review as a conference paper at ICLR 2022
Table 2: Time consuming and accuracy small feature datasets.
IRIS
WINE TIME(s)
CAR TIME(s)
AGARICUS
MODEL
TIME(s)
TOP1
TOP1
TOP1
TIME(s)
TOP1
BO
45
100
46
99.07
93.26
100
ZO0pt
44
100
45
100
47
92.87
71
100
RAND
GA
148
98.89
151
96.3
171
93.55
274
100
PSO
142
100
145
99.07
171
93.35
167
100
HB
24
100
24
87.04
26
69.36
99.94
DEHB
48
100
45
100
49
93.26
100
100
92.49
DHPO
100
96.3
91.43
5
100
On feature datasets shown in Table DHPO also has absolute advantage in efficiency. While in some cases its accuracy is not the best compared to baselines. This is because DHPO trains the network while optimizing In this experiment, we set all epochs to be consistent in order to ensure the fairness of the experiment.
resulting in a need for more epochs than other models.
MNIST
SVHN
AGARICUS
30
101
200pt
10
10:1
DI
DIIPO
BO
20-1
10"2
DEHB
RAND
10
GA
ANN
DEHB
DHPO
PS0
0 10
15
20
25
30
10
25
75
100
125
150
Figure 4: Loss sequence in the training session
As shown in Figure 4 DHPO shows a better loss sequence on MNIST, SVHN and AGARICUS. can be seen that the loss of DHPO is not dominant the beginning, and then slowly drops to the lowest of all models. This corresponds to the two stages DHPO in the actual training process. One is the hyper-parameter optimization stage, and the other is the training stage. DHPO always looking for the best hyper-parameters the heginning, which caused the loss dropping very slowly. Later, the optimal byper-parameters are too small for WINE and CAR. As result, the second stage has not been conducted enough yet. As for IRIS, its distribution simple. So small number of epochs is satisfied.
fixed, and the loss drops quickly then. However, the epochs
100
BO
Z00pt
RAND
10-1
DEHB
PSO
DHPO
10-2
10-3
100
200
300
400
500
Figure 5: Loss sequence comparison in 500 epochs training.
Further, EPOCH to 500 and conduct HPO WINE again. The results and the loss sequence are shown Appendix [ and Figure 55) respectively. easy to see from the figure that the DHPO takes absolute advantage in the later stage of training, and finally achieves good performance.
8

---

Under review as a conference paper at ICLR 2022
4.4 CASE STUDY
In this case study, we track DHPO for whole optimization with running DNN on MNIST by recording syper-parameters generated by DHPO after each epoch. And observe whether hyper- parameters are really optimized in DHPO.
parure
0.970
291 poions
15
321 parss
0.965
2th params
0.40
2th parems
3th params
params
0.30
0.25
0.950
0.15
10
Loss sequence comparison
Accuracy trend along with hyper-parameters
Figure 6: case study on MNIST
In some singleton of DHPO, five groups of hyper- paramerrsseetess given out by DHPO along with the training session. We evaluate these five groups of sypee-pprameters on DNN chronologically and visualize the performances in F The later prper-parameters have stronger convergence capa- bilities according to the loss. Cerformanceacccurayy mean) and stability std) under these hyper-parameters also shows an increasing trend. As the training session progresses, the quality of hyper-parameters gradually increases, which demonstrates that DHPO is an effective HPO method.
5 RELATED WORK
From a large scale, our work belongs to automatic machine learning. With the emergence of large- scale machine learning models, manual tuning (Montavon al. 2012 gri 2007 is no longer ap- plicable. While random algorithm (ran) 2012) and grid search (Hinton 2010) are ineffective with the target of performance and space, respectively. Golldbrr uses evolutionary algorithm to opti- mize paremeparaeters... These algorithms have good effect in exploring the distribution of the objective function value with the sample points and jumping out of the local optimal solution ignore the time consuming. A more stable optimization now belongs to zeroth-order optimiza- tion (Liu et al. 2018) and Bayesian optimization algorithm (Snoek et al. 2012). whose algorithms complementary in principle. BO fits the distribution between samples and objective function with Gaussian process regression. This is suitable for the case where objective is differentiable. Zeroth-order optimization is more suitable for con-differentiable situations, and can solve the situa- tion with multiple local optimal. However, these algorithms still need to repeatedly run the model to be adjusted, which brings large time consuming. order solve this problem, the HyperBand (Li et al. 2016) changed the verification process. Epochs in a training session grow from I along with iterations. And part of samples eliminated after each iteration. Only one sample survives in the end. DEHB (Awad et al. 2021) combines the differential evolution and HyperBand for further optimization. On the whole, all current HPO algorithms rely on multiple training sessions to obtain feedback to adjust
Only DHPO can optimize the speer-prrmeeters in the single training session, which is undoubtedly meaningful work for automatic machine learning.
CONCLUSION & FUTURE WORK
In this paper, DHPO solves the time efficiency problem of HPO, and we also mentioned that DHPO less affected by the initialization parameters. As long certain number of rounds, it always overcome some more difficult cuper-parameters. Furthermore, we plan to study the why to apply DHPO hyper-parameters and more models rather than just neural network.

---

Under review as a conference paper at ICLR 2022
REFERENCES
An empirical evaluation of deep architectures on problems with many factors of variation. In Inter- national Conference on Machine Learning. 2007.
Reading digits in natural images with unsupervised feature learning. nips workshop on deep learning & unsupervised feature learning, 2011.
Random search for hyper- prper-parameter optimization. Journal of Machine Learning Research, 13(1): 281-305, 2012.
N. Awad, N. Mallik, and F. Hutter. Dehb: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. 2021.
Behnke and Sven. Hierarchical neural networks for image interpretation. Springer, 2003.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparam- eter optimization of deep neural networks by extrapolation of learning curves. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
D. E. Goldberg. Genetic algorithm in search.
G. Hinton. A practical guide to training restricted boltzmann machines(i]. Momentum, 9(1):926 947, 2010.
Kennedy and R. Eberhart. Particle swarm optimization. lemm95-internationall Conference on Neural Networks, 1995.
Lars Kotthoff, Chris Thornton, Holger H Hoos, Frank Hutter, and Kevin Leyton- weka 2.0: Automatic model selection and hyperparameter optimization in weka. The Journal of Machine Learning Research, 18(1):826-830) 2017.
Leyton-Brown. Brown. Auto-
Kunihiko and Fukushima. Neocognitron: self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 1980.
Y. Lecun and L. Bottou. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 2):278-2324, 1998.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
L. Li, K. Jamicson, G. Desalvo, A. Rostamizadeh, and A. Talwalkar. Hyperband: A novel bandit- based approach hyperparameter optimization. Journal of Machine Learning Research, 18: 1-52, 2016.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In ICER. OpenReview.net, 2019.
Yu-Ren Liu, Yi-Qi Hu, Hong Qian, Yang Yu, and Chao Qian. Zoopt: Toolbox for derivative-free optimization, 2018.
G. Montavon, G. B. Orr, and KR Müller. Efficient backprop. 10. 1011077998-33302-5599 3):9-48, 2012.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, pp. 2951-2959, 2012.
Chris Thorton,, Frank Hutter, Holger Hoos, and Kevin Leyton- Brown. Auto-weka: Automated selection and tlyper-parameter optimization of classification algorithms. CoRR, abs/1208.3719, 2012.
Steven R Young, Derek C Rose, Thomas P Karnowski, Boung-Hwan Lim, and Robert M Patton. Optimizing deep learning bypee-parameters through an evolutionary algorithm. In Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, pp. 1-5, 2015.
10

---

Under review as a conference paper at ICLR 2022
A
META INFORMATION OF DATASETS
Table 3: Dataset meta information
Dataset
#Train/ #Test
#Attributes
#Class
IRIS
120/ 30
WINE
142/ 36
13
CAR
1 1,382/ 346
6
AGARICUS
6,499 / 1,625
116
MNIST
60,000 /10,000
1#28*28
10
SVHN
73,257 /26,032
3*32#32
10
B PROOF ABOUT DARTS'S EFFECTIVENESS WHEN DEALING WITH LOW DIMENSIONAL CANDIDATES
MaxPool
0.5
0.9
0.1
0.6
011
0.57
O so (max(());
0.4
0.5
AvgPool
0.5
0.9
0.6
Figure 7: The forward propagation of pooling layer
We take Maximum pooling and average pooling as candidates. We will prove Darts is effective in this occasion. A tooee type E {MarPool, Pool). RR2 According to Darts, the output of this pooling layer is calculated as formula visualized in Figure 7
exp(a1)
exp(a2)
((X(a=) exp((n) + exp(on)
* MMaxPoo(XX)+ + exp(()) + eap((2)
AugPPol(XX
(2)
Theorem 3. When 0; az + j. is expressible for 0 under h((x).
h(a)) POOL.CANDIDATES (arg max(a:)
POOL CASDIDATES(O) = Mox MaxPoo Pool
(3)
POOL CANDIDATES(]) AugPool
Proof. When >> a22 according to formula
0 Oupe MarPool. F(X represents eap(oy)
a maximum pooling operation, namely f(X MaxPool(X.. Because explan tezp(az)
1, exp(ag)
R O, then f(X ax) v MaxPool(X)) (X). Assuming the proportion of error informa- exp(ag)
tion is 7 (xplan)+ezplee With 0i >> 02,7 + 0. Then f(X(a) + f(X(0) under 7.
Proof the same for 02 > On-
According to Theorem I and Theorem 3 as long as a; > aj. Darts is effective then. If < 0.001 tolerant, then |aj - agl 6.9 can meet it. It is easy when a E R², So Darts effective this occasion.
11

---

Under review as a conference paper at ICLR 2022
C EXPERIMENT RESULT ON SMALL DATASETS
Table Time consuming and accuracy on small feature datasets.
ALL MEAN
ALL
TOP5 MEAN
TOP5
TOP10 MEAN
TOP10
MODEL
(IME(S)
TOP1
STD
STD
STD
IRIS
BO
45
100
99.67
0.58
100
Z0Opt
44
100
99.56
0.74
100
0
100
RAND
38
100
98.67
5.96
100
100
GA
148
98.89
99.81
0.44
100
0
100
PSO
142
100
94.76
15.88
100
100
HB
24
100
46.54
24.72
100
0
100
DEHB
48
100
99.33
2.21
100
0
100
DHPO*
100
100
100
0
100
DHPO
100
WINE
BO
46
99.07
94.38
5.05
98.33
0.37
98.06
0.5
Z00pt
45
100
95.77
3.68
98.89
0.69
98.43
0.72
RAND
45
100
95.56
3.88
99.07
0.59
98.33
0.91
GA
151
96.3
96.45
1.62
99.26
0.37
98.89
0,56
PSO
145
99.07
77.18
21.38
97.78
0.74
97.5
0.59
НB
24
87.04
51.96
16.63
90.37
1.99
88.52
2.63
DEHB
45
100
94.1
7.03
98.52
0.74
97.87
0.83
DHPO*
9
100
95.56
4.16
98.89
1.36
95.56
4.16
DHPO
96.3
CAR
BO
52
93.26
91.02
4.98
92.99
0.15
92.86
0.17
ZOOpt
47
92.87
92.18
0.44
92.72
0.12
92.58
0.17
RAND
48
93.16
91.57
1.64
92.85
0.22
92.63
0.27
GA
171
93.55
92.45
2.37
93.47
0.19
93.38
0.16
PSO
171
93.35
90.06
8.77
93.43
0.11
93.33
0.13
HB
26
69.36
67.44
9.01
87.38
1.61
80.41
8.31
DEHB
49
93.26
90.89
5.82
93.12
0.12
93.02
0.14
DHPO*
10
92.49
91.65
0.82
92.31
0.23
91.65
0.82
DHPO
91.43 3
AGARICUS
BO
69
100
98.23
6.81
100
0
100
200pt
71
100
100
100
0
100
RAND
64
100
99.99
0.03
100
100
GA
274
100
100
100
0
100
PSO
167
100
90.74
16.42
100
0
100
HB
40
99.94
58.13
15.74
99.94
0
99.93
0.01
DEHB
59
100
100
100
100
DHPO*
16
100
100
100
0
100
DHPO
100
12

---

Under review as a conference paper at ICLR 2022
D
LOSS SEQUENCE ON IRIS, WINE AND CAR
IRIS
WINE
CAR
Diut
14
13
06
03
Figure 8: Loss sequence three small datasets
RESULT OF 500 EPOCH TRAINING ON WINE
TIME
TOP1
TOP5
ALL
BO
153
99.07
98.7
96.98
700pt
154
99.07
98.52
96.73
RAND
151
99.07
98.52
96.98
GA
556
99.07
99.63
97.27
DEHB
160
100
98.89
97.31
PSO
529
100
99.26
88.91
DHPO*
36
100
100
98.06
HB
24
87.04
90.37
51.96
DHPO
10
97.22
13


Reviews:
[{"review_id": "k6MQg-k9Txi", "reviewer": "Reviewer_Qr8L", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Using a modified version of softmax to select channel size and kernel size seems novel which is not achieved by DARTS. However, there exist previous works on neural architecture search which can do this (e.g., Wan, Alvin, et al. \"Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.). Without comparison with these previous works, it is hard to appreciate the contribution and novelty of this proposed method. \n\nBesides, the experiments are conducted only on MNIST and SVHN which are small scale by the standard in this topic. Considering that most papers in neural architecture search report performance on  ImageNet, it is important that the authors also report the performance there.", "summary_of_the_paper": "This paper proposes to tune hyperparameters in a differentiable way by using a modified version of softmax function. It covers the tuning for three kinds of hyper-parameters as examples: channel size, kernel size and hidden layer size. Experiments on MNIST and SVHN (which are small scale datasets by modern DNN standards) show improvements on previous methods. The proposed method itself is reasonable, however, this paper misses many previous works on this topic, which makes it hard to appreciate its novelty.", "summary_of_the_review": "In summary, the proposed method is ok but this paper ignores important previous works, which makes it hard to appreciate the contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hPTgQy8n-XV", "reviewer": "Reviewer_EFJq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper is a contribution in the right direction and has potential to cut down the hyperparameter optimization time greatly. The paper provides an elegant way to determine the optimum hyperparameters via a differentiable scheme.\n\nHaving said that, I have several concerns about the experiments.\n1.\tAnother method which is closely related is Darts (Liu et al (2019)). Why it is not selected as a baseline in the comparisons?\n2.\tTwo datasets MNIST and SVHN are used for the experiments. Both of them are similar and contain images of digits. It would be useful if the authors could show the performance of their method on other diverse datasets like CIFAR-10 or CIFAR-100.\n3.\tSection 4.2.3: the authors report the accuracy of optimized networks, but it is not clear if that is on the validation set or on the test set.\n4.\tTable 1: I don’t see much the benefit of the proposed method DHPO. Take the version DHPO+ as example. Compared to BLANK (i.e. train the network without hyper-parameter tunning), DHPO+ is just slightly better than BLANK, there is no significant difference. For example, 98.71% vs. 98.27% for MNIST and 85.05% vs 84.95% for SVHN.\n5.\tI am not sure why the authors use the version DHPO* (running 10 times) of their method when all the other methods are run only 3 times? Is this a fair comparison? If all of the methods including DHPO+ run the same 3 times, DHPO+ is not better than the baselines on datasets SVHN, Wine, and Car.\n6.\tFinally, it would be nice if the authors could apply their methods to neural architecture search using standard benchmarks e.g. NAS-101 etc. This will truly show the effectiveness of their method.\n\nMinor Comments:\nThe paper seems to be written in a rush. The text and presentation could be significantly improved. \n- the citation of Goldberg in the introduction section is not correctly done (missing publication year).\n- space missing after BO and the following parenthesis, HB and the parenthesis, DHPO and the parenthesis.\n-random search is cited incorrectly as (ran, 2012)!\n- One page 2: incomplete sentence “While Darts can select the hyperparameters …”\n", "summary_of_the_paper": "\nThis paper introduces a method for hyper-parameter optimization (HPO) for deep neural networks. Its main idea is to replace hyper-parameters by trainable parameters, which can be included in the training process of the network itself.\nThe proposed method is applied to hyper-parameter tunning for two types of networks, namely CNN and FNN on two datasets MNIST and SVHN. It is also compared with several baselines.\n", "summary_of_the_review": "To summarize, this paper addresses the HPO problem by transforming the hyper-parameters to trainable parameters and including them in the training process of the network. This approach can speed up the optimization time and allow the hyper-parameter optimization to happen in one single training session. Overall, the idea is interesting but the experiments are not convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BjKUyY7YDbn", "reviewer": "Reviewer_iUkA", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper strengths:\n+ The reparameterization of hyperparameters to make them derivable seems good. The idea is a good idea, even if already existing in the literature.\n\nPaper weaknesses:\n- This paper is not novel, a lot of existing work already tested this idea and showed that it already works. See the following references:\n\nFranceschi, L., Donini, M., Frasconi, P. & Pontil, M.. (2017). Forward and Reverse Gradient-Based Hyperparameter Optimization. Proceedings of the 34th International Conference on Machine Learning, in PMLR 70:1165-1173\n\nStamoulis, D., Ding, R., Wang, D., Lymberopoulos, D., Priyantha, B., Liu, J., & Marculescu, D. (2019). Single-path nas: Designing hardware-efficient convnets in less than 4 hours. arXiv preprint arXiv:1904.02877.\n\nZhang, C., Ren, M., & Urtasun, R. (2018). Graph hypernetworks for neural architecture search. arXiv preprint arXiv:1810.05749.\n\nTwo of those references involve hypernetworks, with the same idea as proposed in this paper, which is define a larger network containing all subnetworks embedded within the hyperparameter search space.\n\n- The selected benchmarks are outdated.\n- Going back to the first point, the methods selected for comparison with the proposed approach are also outdated.\n- The paper is poorly written, hard to read and the ideas are not clearly stated.\n", "summary_of_the_paper": "The paper presents a gradient-based hyperparameter optimization method, wherein a derivable parameterization is proposed for various popular CNN hyperparameters including kernel size, number of channels and hidden layer size. Experiments show promise in terms of convergence speed when evaluated on simple benchmarks such as MNIST and SVNH, however this is irrelevant because a variant of this method already exists in the literature and nothing new is proposed.", "summary_of_the_review": "This paper proposes ideas that are already existing in the literature, therefore it should be rejected. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "BGIKel-ZFZZ", "reviewer": "Reviewer_CR1f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. On MNIST and SVHN, the proposed method has the best top1-error and the lowest STD. It also has the fastest convergence rate.\n\nWeaknesses:\n1. The authors claimed that they proposed the first model to solve hyper-parameter optimization in only one training session. However, they also said Darts could select the hyper-parameters in a single training session. Even if their method handled the argmax problem of DARTS, SNAS [1] has also considered this problem by the Gumbel-softmax trick. Hence, using differentiable methods is not their contribution.\n\n2. The proposed theorems are trivial and meaningless. The authors try to handle the cases where multiple candidates occupy similar weights in DARTS. So, the theorem should prove that. At least, it is necessary to prove the three key characteristics in Section 3.1.3. \n\n3. The paper didn't explain their method well. They just introduced their reparameterization. What is its differential? It is not a trivial problem [2]. Why it can solve the DARTS problem? It cannot be explained just in examples. Moreover, there are many writing issues and typos. For example, \n(1) Formula 1 should be written in a sentence.\n(2) The sentence \"It is selecting an activation for the fully connected layer and supposing Relu is the best choice.\" is weird.\n(3) What does the sentence \"Firstly, we define found sufficient conditions as a new relation between \\alpha and \\theta.\" mean?\n\n4. The experiments only used small data. The performance of the proposed method is not clear on the big data, like ImageNet. At least, CIFAR10 and CIFAR100 are needed. Further, this method should also work for other tasks not only for classification tasks. \n\n5. The baselines are old and not enough. The baselines of Bayes optimization should include the first prize of Black Box Optimization competition in NeurIPS 2020, HEBO [3]. The SOTA of multi-fidelity methods should include BOHB [4] or BOSS [5].\n\n[1] Xie, S., Zheng, H., Liu, C., & Lin, L. (2018). SNAS: stochastic neural architecture search. arXiv preprint arXiv:1812.09926.\n[2] Franceschi, L., Donini, M., Frasconi, P., & Pontil, M. (2017, July). Forward and reverse gradient-based hyperparameter optimization. In International Conference on Machine Learning (pp. 1165-1173). PMLR.\n[3] Cowen-Rivers, A. I., Lyu, W., Wang, Z., Tutunov, R., Jianye, H., Wang, J., & Ammar, H. B. (2020). Hebo: Heteroscedastic evolutionary bayesian optimisation. arXiv e-prints, arXiv-2012.\n[4] Falkner, S., Klein, A., & Hutter, F. (2018, July). BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning (pp. 1437-1446). PMLR.\n[5] Huang, Y., Li, Y., Ye, H., Li, Z., & Zhang, Z. (2020). An Asymptotically Optimal Multi-Armed Bandit Algorithm and Hyperparameter Optimization. arXiv preprint arXiv:2007.05670.\n\n", "summary_of_the_paper": "The authors proposed a differentiable method to optimize neural networks and hyperparameters simultaneously like DARTS. And, they used a new parameterization to represent the original discrete hyperparameters. They claimed that it could eliminate the impact of rounding.", "summary_of_the_review": "As mentioned in Main Review, the contribution of the paper is not enough. The novelty is not strong, since it is not the first model. The theorems are trivial and meaningless to describe the reparameterization, since the specific mapping is given. The experiments are not comprehensive enough considering the dataset and baselines. Moreover, the method itself is not explained well. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a gradient-based hyperparameter optimization method, wherein a differentiable reparameterization is proposed for various popular CNN hyperparameters including kernel size, number of channels and hidden layer size.

All reviewers have pointed out the lack of novelty (such reparameterizations are standard) and lack of convincing experiments.

The authors didn't write any rebuttal.

Overall, there is a large consensus among the reviewers that this paper is not ready for publication at ICLR.
Decision: Reject