DIFFERENTIABLE HYPER-PARAMETER OPTIMIZATION
Year: 2021

Abstract:
Hyper-parameters are widely present in machine learning. Concretely, large amount of hyper-parameters exist in network layers, such as kernel size, channel size and the hidden layer size, which directly affect performance of the model. Thus, hyper-parameter optimization is crucial for machine learning. Current hyper-parameter optimization always requires multiple training sessions, resulting in a large time consuming. To solve this problem, we propose a method to fine-tune neural network’s hyper-parameters efficiently in this paper, where optimization completes in only one training session. We apply our method for the optimization of various neural network layers’ hyper-parameters and compare it with multiple benchmark hyper-parameter optimization models. Experimental results show that our method is commonly 10 times faster than traditional and mainstream methods such as random search, Bayesian optimization and many other state-of-art models. It also achieves higher quality hyper-parameters with better accuracy and stronger stability.

text: Hyper-parameters are widely present in machine learning. Concretely, large amount of hyper-parameters exist in network layers, such as kernel size, channel size and the hidden layer size, which directly affect performance of the model. Thus, hyper-parameter optimization is crucial for machine learning. Current hyper-parameter optimization always requires multiple training sessions, resulting in a large time consuming. To solve this problem, we propose a method to fine-tune neural network’s hyper-parameters efficiently in this paper, where optimization completes in only one training session. We apply our method for the optimization of various neural network layers’ hyper-parameters and compare it with multiple benchmark hyper-parameter optimization models. Experimental results show that our method is commonly 10 times faster than traditional and mainstream methods such as random search, Bayesian optimization and many other state-of-art models. It also achieves higher quality hyper-parameters with better accuracy and stronger stability.

heading: 1 INTRODUCTION
text: HPO(hyper-parameter optimization) is one of the most critical parts in auto-ML (Thornton et al., 2012; Domhan et al., 2015; Kotthoff et al., 2017). Simple and low-dimensional hyper-parameters can be adjusted manually. It is also practical to use grid search or combine grid search with manual adjustment to deal with this simple problem (Montavon et al., 2012; gri, 2007; Hinton, 2010).
With the number of hyper-parameters continues increasing, manual tuning and grid search get ineffective. For a slightly large neural network model, we can use BO(bayesian optimization) (Snoek et al., 2012) or ZOOpt(zeroth-order optimization) (Liu et al., 2018) to optimize hyper-parameters. BO uses Gaussian process regression to fit mean and variance of objective function. However, a large number of matrix operations are needed in the fitting process, and multiple training sessions are needed to evaluate the predicted hyper-parameters. Time consuming increases along with each training session, and the total time consuming is positively correlated with the number of training sessions. Therefore, BO is still not suitable for large amount of hyper-parameters and so is ZOOpt.
In addition to BO and ZOOpt, many evolutionary algorithms (Young et al., 2015) have also been applied. For example, genetic algorithm (Goldberg) is often used for HPO. Evolutionary algorithms can often avoid the problem of local optimal. However, a fatal flaw is the low time efficiency.
When dealing with large-scale systems, random search (ran, 2012) is a practical and more efficient method to solve HPO. Based on experiences, random search may perform better than BO in some cases with better time efficiency and accuracy. In addition, currently very effective HPO methods are probably HB(Hyperband (Li et al., 2016) and DEHB (Awad et al., 2021). They accelerate the convergence and make it twice faster than random search.
Totally, all current models have a common defect which is a requirement for multiple training sessions of the fine-tuned model. While each training session is always expensive in time consuming. To get rid of this limit, we propose DHPO(differentiable hyper-parameter optimization), which accomplish HPO within one training session. Concretely, we change the form of hyper-parameters and include them in the calculation of forward propagation to achieve differentiable hyper-parameters. In this way, hyper-parameters are differentiable with the goal of minimize the loss function. When the session is over, hyper-parameters complete optimization together with network’s parameters. Therefore, only one training session is needed in DHPO.
The contributions of this paper are summarized as follows.
• In this paper, we propose DHPO, which is the first model to solve hyper-parameter optimization in only one training session. • The proposed approach is universal. It is an idea for HPO which can be applied to various kinds of hyper-parameters in neural network. • We conduct extensive experiments on various kinds of neural network layers compared with multiple benchmark HPO models to evaluate the extreme efficiency and high performance of DHPO.
In the remaining of this paper, Section 3 describes the proposed method. Experiments are conducted and analyzed in Section 4. We overview related work in Section 5. Section 6 draws the conclusions and future work.

heading: 2 BACKGROUND
text: In this section, we introduce related techniques briefly.
Darts (Liu et al., 2019) is used to solve automatic network architecture search, which is described to determine the best operation from multiple candidate operators locally and globally. Take Figure 1 as example. It is selecting an activation for the fully connected layer from {Tanh, Sigmoid,Relu, Softmax} and supposing Relu is the best choice. Grid search costs four training sessions to evaluate each activation. While Darts can select the hyper-parameters in a single training session. Firstly, Darts calculates features X transformed by the fully connected layer and then uses each activation to process X once to get four non-linear features, denoted as o = [o1, o2, o3, o4]. Further, softmax(α) assigns a weight to each oi and aggregates weighted oi by summing them, where αi is a trainable parameter. An ideal train always ends up with max softmax(α)→ 1 as shown in Figure 1. Darts in Figure 1 finally determines Relu as the optimal activation with softmax( alpha)3 very close to 1. Accordingly, softmax(α)i ≈ 0, i 6= 3 meaning a shield to the information contained in o1, o2, o4, which is equivalent to using Relu only.

heading: 3 METHOD
text: As introduced in the background, Darts still has fatal defects. On the one hand, max softmax(α) doesn’t always converge to a level very close to 1. The cases where multiple candidates occupy
similar weights always exist especially for a large amount of candidates, which means Darts fails to distinguish different candidates and hit the optimal. On the other hand, Darts is only suitable for hyper-parameter with independent candidates. For example, the channel size is beyond Darts’s capability. If we’d like to apply Darts to solve channel size, we should set a candidate for each possible value of channel size. Then there will beO(n2) channels in total, which is space expensive.
Motivated by this, we propose DHPO, aiming at solving all these problems existing in Darts and current HPO models. In the following, we first take the optimization of convolution layer’s channel size as example to draw the core of DHPO in Section 3.1. Then, we apply DHPO to more hyperparameters in Section 3.2.

heading: 3.1 DIFFERENTIABLE HYPER-PARAMETER OPTIMIZATION
text: The target of DHPO is to make hyper-parameter θ differentiable, and we achieve it through constructing trainable parameter α to substitute θ in the training. Obviously, the core of the idea is to point out the limitations for α and give out a universal method to construct and apply α.
In this section, we first declare the sufficient conditions for α to substitute θ based on Theorem 1 in Section 3.1.1. Then we explain how to construct α and use it to control the structure of neural network, namely that is the forward propagation under α in Section 3.1.2. In the end, we use Theorem 1 to prove that α is able to substitute θ as Theorem 2 in Section 3.1.3.
3.1.1 LIMITATIONS FOR α
Our target is to construct trainable parameter α to substitute θ in the training. We will point out the sufficient conditions for α in the following.
Firstly, we define found sufficient conditions as a new relation between α and θ, which is defined as expressible as Definition 3.1. Then, we prove that if α is expressible for θ, then we can replace θ with α in neural network as Theorem 1. The definition and theorem are as follows. Definition 3.1. Expressible: For a neural network F : Rd 7→ R with W and Θ as parameters and hyper-parameters respectively, hyper-parameter θ2 ∈ Ω2 is expressible for hyper-parameter θ1 ∈ Ω1 if and only if there is a surjective h : Ω2 7→ Ω1(θ1 = h(θ2)) which makes network structure under F(W |θ1) and that under F(W |θ2) are the same. Theorem 1. If θ2 is expressible for θ1, then we can replace θ1 with θ2 in neural network.
Proof. If θ2 is expressible for θ1, then there is a surjective from θ2 to θ1. Therefore, ∀θ1 ∈ Ω1,∃θ2 ∈ Ω2, h(θ2) = θ1. Meanwhile, F(W |θ2) and F(W |θ1) share the same structures according to Definition 3.1. So we can replace θ1 with θ2 in a neural network.
According to Theorem 1 and Definition 3.1, we can replace θ with α in neural network as long as the constructed α satisfies the following two conditions. One is that there is a surjective θ = h(α). The other is that the network structure controlled by α is the same to that of θ under θ = h(α).
3.1.2 FORWARD PROPAGATION UNDER α
In this section, we first propose the method to construct expressible α as described above, and then explain how α controls the structure of neural network, i.e., applying α in the forward propagation.
φ(X|α) = sup(θ)∑ i=1 i ∗ (X ? κi)
= σ((softmax(α)) ·A− σ(β)) ∗ a)
(1)
We take convolution layer’s channel size as example to construct trainable α, which should be expressible for θchannel size(abbreviated as θ). We construct α as α = [α1, · · · , αsup(θ)] and α ∈ Rsup(θ), sup(θ) = max θ. Formula 1 describes the forward propagation in this convolution layer under α. X ∈ RW×H is the input of convolution layer which is a single-channel image. κi represents the convolution kernel of i-th channel, and we prepare sup(θ) candidate kernels in the initialization of neural network. (?) represents convolution operation and (·) represents matrix
multiplication. A =
[ 1 · · · 1
. . . . . . 0 1
]T is a lower triangular matrix. γ ∈ R is a trainable parameter and
a ∈ R is a large constant. Obviously, the core of formula 1 is the calculation of . We visualize the calculation in Figure 2. Two cases are given in this figure. We take the left as example. α is first transformed by softmax activation and 0 < softmax(α)i < 1. Then we get A by multiplying matrix softmax(α) and matrixA, A1 = 1 and Ai > Aj if i > j. Next, we subtract σ(γ) from Ai and multiply it by a large constant a, a > 1. In this time, −a < (Ai − σ(γ)) ∗ a < a, and it maintains the same monotonicity as A. In the last step, we use σ again to map (Ai − σ(γ)) ∗ a to a value close to 1 or 0. Finally, i = σ((Ai−σ(γ)) ∗a). In this example, 1, 2, 3 are very close to 1, and the others are very close to 0. In this occasion, we think that the channel size is 3.
Based on the above construction, α has the same effect as θchannel size. Thus, it is reasonable to substitute θchannel size with α. We will prove this in the following Section 3.1.3.
3.1.3 EXPRESSIBLE FOR θ
The goal is to prove that we can substitute θchannel size with α. According to Theorem 1, we just need to prove that α is expressible for θchannel size. Before proof, we first list three key properties of in the following which are necessary to the proof.
is constructed from α and has three key characteristics. ¬ 1 ≈ 1.  1 > · · · > sup(θ). ® There is an index t dividing into two sets εbig = { i|1 ≤ i ≤ t} and εsmall = { i|t+ 1 ≤ i ≤ sup(θ)}. The items in εbig are all close to 1 and items in εsmall are all close to 0. The first property ensures that at least one channel is selected. The second property ensures the selected channels are always the first few rather than random several in candidate channels. The third property makes unselected channels blocked away, especially when a 7→ ∞. Based on these features, we have Theorem 2. Theorem 2. When a 7→ ∞, α is expressible for θ under h(α) = rounded ∑ .
Proof. If we would like to prove α is expressible for θ, we just need to prove h(α) is surjective and network structure controlled by α is the same to that of θ under θ = h(α) according to the discussion in Section 3.1.1. Obviously, θ = h(α) is a surjective. We then prove convolution layers under θ and α have same
structure. Suppose that the network structure controlled by θ is a convolution layer composed of the first θ channels from candidates. As for the structure controlled by α, we can also see it as a convolution layer composed of the first θ channels according to the third characteristic of . Because noise from unselected channels will fade out with a→∞. At this time, F(X|α)⇔ F(X|θ).
Based on above discussions, we finally achieve the differentiable channel size. Since α is trainable and expressible for θchannel size. Then we can replace θchannel size withα in this convolution layer.
When the training session is over, we take rounded ∑ as the optimized θchannel size.

heading: 3.2 MULTIPLE HYPER-PARAMETERS OPTIMIZATION
text: Based on the idea discussed in Section 3.1, we can apply DHPO to a large amount of hyperparameters rather than just channel size.
In this section, we select two widely used and typical hyper-parameters to explain how to deal with them in DHPO, and other hyper-parameters can be processed in a similar way. These typical hyperparameters are convolution kernel size and hidden layer size.

heading: 3.2.1 KERNEL SIZE
text: In this section, we introduce method of making kernel size differentiable. The challenge here is that we should deal with multi-dimensional hyper-parameter. Therefore, we should make DHPO suitable for multi-dimensional hyper-parameter rather than just one dimensional hyper-parameter like channel size. We achieve this by extending to multiple dimensions and the detail is as follows.
Firstly, we construct from α as described in Section 3.1 and a candidate kernel κ. For the onedimensional kernel, κ ∈ Rsup(θkernel size)(θkernel size abbreviated as θ). And we use κ as the kernel in forward propagation. For the 2-dimensional kernel κ ∈ Rsup(θ)×sup(θ), we use κ M as
selected kernel, M =  1 2 · · · sup(θ) 2 2 · · · sup(θ) ... ... . . .
... sup(θ) sup(θ) · · · sup(θ) . In this way, the features from unselected items i are blocked away since i ≈ 0. Then α is expressible for θ under h(α) = rounded ∑ . Obviously, h(α) is a surjective. When the large constant a → ∞. In this time, kernel size under θ and that under α are the same under θ = h(α). So we can substitute θkernel size with α in neural network. When the training session is over, we take rounded ∑ as the optimized θkernel size.
Based on convolution kernel size, we can design differentiable hyper-parameter for pooling kernel size, convolution stride and etc. in a similar way.

heading: 3.2.2 HIDDEN LAYER SIZE
text: In this section, we apply DHPO to one of the most common and important hyper-parameters, hidden layer size. Commonly, the hidden layer size has thousands of candidates. Therefore, it is difficult for sample-based models such as BO and random search. In contrast, DHPO can solve it easily both theoretically and practically since hidden layer size is differentiable in DSA.
Again, we first construct from α as described above and use to control the hidden layer size. We use (X ·W +b) as the output of the fully connected layer. (X ·W +b) is the original output and its dimension is sup(θhidden size). In this way, unselected last few features will be blocked away, which is equivalent to use a smaller hidden layer size. When the training session is over, we take rounded ∑ as optimized θhidden size.

heading: 4 EXPERIMENT
text: To verify the effectiveness of the proposed approach, we conduct HPO on two different neural networks with multiple datasets, and compare it with various baselines. Section 4.1 briefly introduces
the neural network to be tuned, Section 4.2 gives the basic settings, Section 4.3 displays the experimental results, and Section 4.4 conducts a case study to show the DHPO tuning process.

heading: 4.1 MODEL TO BE TUNED
text: In the experiment, we apply DHPO to two classical models, i.e. deep convolutional neural networks and MLP(multi-layer perceptron), which include various kinds of hyper-parameters. DNN(deep convolution neural network) is a lightweight convolution network designed with reference to Kunihiko & Fukushima (1980); Lecun & Bottou (1998); Behnke & Sven (2003). Even though DNN is not the best model to process images, it is a good network which can distinguish the performence of various HPO approaches since it has a large number of hyper-parameters. DNN’s network structure is as shown in Figure 3, which is a lightweight implementation. And MLP is composed of 4 fully connected layers. DNN is used to evaluate HPO for convolution layer and pooling layer. MLP is used to evaluate HPO for fully connected layer.

heading: 4.2 EXPERIMENTAL SETTINGS
text: In this section, we introduce the experimental settings.

heading: 4.2.1 DATASET
text: MNIST (LeCun & Cortes, 2010) and SVHN (svh, 2011) are well known lightweight validation image datasets, and they are suitable for DNN. Therefore, we use MNIST and SVHN as the benchmark datasets on DNN. On MLP, we choose iris, wine, car and agaricus-lepiota1. Basic message of the six datasets is shown in Appendix A.

heading: 4.2.2 BASELINE
text: The mainstream algorithms used for HPO contains random search, BO, evolutionary algorithm and so on. We finally choose the following HPO algorithm as baselines:
• Random search (ran, 2012) is the most universal HPO algorithm, which can randomly select points in the entire search space. So it is able to break the limit of local optimal solution. • Zoopt (Liu et al., 2018) does not rely on the gradient of the objective function, but learns from samples of the search space instead. It is suitable for HPO tasks that are not differentiable, with many local minimal or even unknown but only testable. • Bayes optimization (Snoek et al., 2012) is based on Gaussian process regression to fit the distribution between samples and objective. It has a practical effect and wide range of applications. • Genetic algorithm (Goldberg) belongs to evolutionary algorithm, relying on the mutation and genetic of maintained population. It is often used in HPO and NAS(network architecture search). • Particle swarm algorithm (Kennedy & Eberhart, 1995) also belongs to evolutionary algorithm, which finds the optimal solution through collaboration and information sharing between individuals in the group. Simple implementation, few hyperparameters, and wide applications. • HyperBand (Li et al., 2016) regards the process of finding the optimal hyper-parameters as a non-random exploration on the infinite arm bandit under the condition of limited resources. • DEHB (Awad et al., 2021) is developed based on differential evolution and HB, which is furnished with better time performance and convergence effect. 1https://archive.ics.uci.edu/ml/datasets

heading: 4.2.3 METRICS
text: • Time consuming reflects the time interval from when data set and model are loaded on the GPU to when the algorithm stops. If data set is loaded multiple times, the loading time is also included. • Accuracy reflects the performance of neural network generated by HPO models. • Mean of top-K accuracy. HPO model always gives multiple sets of hyper-parameters and corre-
sponding accuracy. We calculate the mean accuracy of top-K models. • Loss sequence reflects the state of objective and convergence effect in a training session.

heading: 4.2.4 PARAMETER SETTINGS
text: Tuned hyper-parameters We fine-tune convolution layer’s channel size, convolution layer’s kernel size, pooling layer’s pooling type and pooling layer’s kernel size in DNN model, totally 16 hyper-parameters. Among them, channel size ∈ [1, 8], kernel size ∈ [2, 5], pooling type ∈ [MaxPool, AvgPool]2. Additionally, 3 hidden layer size are optimized in MLP.
DNN We uses the Adam optimizer with learning rate = 0.001 and uses cross-entropy loss function. batch size = 500 and each training session contains 30 epochs. MLP All are the same to DNN except for 150 epochs in training session.
BO BO model is initialized with 5 samples, and continues iterating for 25 rounds. ZOOpt Iteration budgets is 30, and other hyper-parameters use the built-in configurations.3 Random search The algorithm iterates 30 times. Evolutionary algorithm The population size and iterations are both 10, and mutation probability is 0.001. HB Iterate 81 times to keep fair with other models, because of its different iteration principles. DEHB We set min budget and max budget as 2 and 50, respectively.
Considering the influence of parameter’s initialization, a three-times repeated independent experiment is conducted and our experiments are conducted on one GTX 3060Ti GPU.

heading: 4.3 EXPERIMENTAL RESULTS
text: As shown in Table 1, DHPO can always complete tasks faster than any other HPO model several times with ensuring good performance. Taking MNIST as example, DHPO? wins the game by taking only 28% of DEHB’s time consuming. Additionally, DHPO† can also ensure the advantage in accuracy with fewer repeats, which means DHPO is stable. The stability can also be reflected from top-K accuracy’s standard deviation, which is only 0.19 for all samples under DHPO.
2In DHPO, we invoke Darts to solve pooling type and we have proved it’s effective here in Appendix B 3https://github.com/polixir/ZOOpt
On feature datasets shown in Table 2, DHPO also has absolute advantage in efficiency. While in some cases its accuracy is not the best compared to baselines. This is because DHPO trains the network while optimizing hyper-parameters resulting in a need for more epochs than other models. In this experiment, we set all epochs to be consistent in order to ensure the fairness of the experiment.
As shown in Figure 4, DHPO shows a better loss sequence on MNIST, SVHN and AGARICUS. It can be seen that the loss of DHPO is not dominant at the beginning, and then slowly drops to the lowest of all models. This corresponds to the two stages of DHPO in the actual training process. One is the hyper-parameter optimization stage, and the other is the training stage. DHPO is always looking for the best hyper-parameters in the beginning, which caused the loss dropping very slowly. Later, the optimal hyper-parameters are fixed, and the loss drops quickly then. However, the epochs are too small for WINE and CAR. As a result, the second stage has not been conducted enough yet. As for IRIS, its distribution is simple. So a small number of epochs is satisfied.
Further, we set EPOCH to 500 and conduct HPO on WINE again. The results and the loss sequence are shown in Appendix E and Figure 5, respectively. It is easy to see from the figure that the DHPO takes an absolute advantage in the later stage of training, and finally achieves a good performance.

heading: 4.4 CASE STUDY
text: In this case study, we track DHPO for a whole optimization with running DNN on MNIST by recording hyper-parameters generated by DHPO after each epoch. And we observe whether hyperparameters are really optimized in DHPO.
In some singleton of DHPO, five groups of hyper-parameters are given out by DHPO along with the training session. We evaluate these five groups of hyper-parameters on DNN chronologically and visualize the performances in Figure 6. The later hyper-parameters have stronger convergence capabilities according to the loss. Performance(accuracy mean) and stability(accuracy std) under these hyper-parameters also shows an increasing trend. As the training session progresses, the quality of hyper-parameters gradually increases, which demonstrates that DHPO is an effective HPO method.

heading: 5 RELATED WORK
text: From a large scale, our work belongs to automatic machine learning. With the emergence of largescale machine learning models, manual tuning (Montavon et al., 2012; gri, 2007) is no longer applicable. While random algorithm (ran, 2012) and grid search (Hinton, 2010) are ineffective with the target of performance and space, respectively. Goldberg uses evolutionary algorithm to optimize hyper-parameters. These algorithms have a good effect in exploring the distribution of the objective function value with the sample points and jumping out of the local optimal solution if we ignore the time consuming. A more stable optimization now belongs to zeroth-order optimization (Liu et al., 2018) and Bayesian optimization algorithm (Snoek et al., 2012), whose algorithms are complementary in principle. BO fits the distribution between samples and objective function with Gaussian process regression. This is suitable for the case where objective is differentiable. Zeroth-order optimization is more suitable for non-differentiable situations, and can solve the situation with multiple local optimal. However, these algorithms still need to repeatedly run the model to be adjusted, which brings a large time consuming. In order to solve this problem, the HyperBand (Li et al., 2016) changed the verification process. Epochs in a training session grow from 1 along with iterations. And a part of samples is eliminated after each iteration. Only one sample survives in the end. DEHB (Awad et al., 2021) combines the differential evolution and HyperBand for further optimization. On the whole, all current HPO algorithms rely on multiple training sessions to obtain feedback to adjust hyper-parameters. Only DHPO can optimize the hyper-parameters in the single training session, which is undoubtedly a meaningful work for automatic machine learning.

heading: 6 CONCLUSION & FUTURE WORK
text: In this paper, DHPO solves the time efficiency problem of HPO, and we also mentioned that DHPO is less affected by the initialization parameters. As long as a certain number of rounds, it can always overcome some more difficult hyper-parameters. Furthermore, we plan to study the why to apply DHPO on more hyper-parameters and more models rather than just neural network.

heading: A META INFORMATION OF DATASETS
text: Table 3: Dataset meta information
Dataset #Train/ #Test #Attributes #Class IRIS 120/ 30 4 3 WINE 142/ 36 13 3 CAR 1,382/ 346 6 4 AGARICUS 6,499 / 1,625 116 2 MNIST 60,000 /10,000 1*28*28 10 SVHN 73,257 /26,032 3*32*32 10

heading: B PROOF ABOUT DARTS’S EFFECTIVENESS WHEN DEALING WITH LOW DIMENSIONAL CANDIDATES
text: Figure 7: The forward propagation of pooling layer
We take Maximum pooling and average pooling as candidates. We will prove Darts is effective in this occasion. θpool type ∈ {MaxPool, AvgPool}. α ∈ R2. According to Darts, the output of this pooling layer is calculated as formula 2 visualized in Figure 7.
φ(X|α) = exp(α1) exp(α1) + exp(α2) ∗MaxPool(X) + exp(α2) exp(α1) + exp(α2) ∗AvgPool(X) (2)
Theorem 3. When αi αj , i 6= j, α is expressible for θ under h(α). h(α) = POOL CANDIDATES(arg max{α}) POOL CANDIDATES(0) = MaxPool
POOL CANDIDATES(1) = AvgPool
(3)
Proof. When α1 α2, according to formula 3, θpool type = MaxPool. F(X|θ) represents a maximum pooling operation, namely f(X|θ) = MaxPool(X). Because exp(α1)exp(α1)+exp(α2) ≈ 1, exp(α2)exp(α1)+exp(α2) ≈ 0, then f(X|α) ≈MaxPool(X). Assuming the proportion of error information is γ, γ = exp(α2)exp(α1)+exp(α2) . With α1 α2, γ → 0. Then f(X|α)⇔ f(X|θ) under γ. Proof is the same for α2 α1.
According to Theorem 1 and Theorem 3, as long as αi αj , Darts is effective then. If γ ≤ 0.001 is tolerant, then |α1 − α2| > 6.9 can meet it. It is easy when α ∈ R2. So Darts is effective in this occasion.

heading: C EXPERIMENT RESULT ON SMALL DATASETS

heading: D LOSS SEQUENCE ON IRIS, WINE AND CAR

heading: E RESULT OF 500 EPOCH TRAINING ON WINE
text: TIME TOP1 TOP5 ALL BO 153 99.07 98.7 96.98 ZOOpt 154 99.07 98.52 96.73 RAND 151 99.07 98.52 96.98
GA 556 99.07 99.63 97.27 DEHB 160 100 98.89 97.31 PSO 529 100 99.26 88.91
DHPO? 36 100 100 98.06 HB 24 87.04 90.37 51.96 DHPO† 10 97.22

Review summaries:
 This paper introduces a method for hyper-parameter optimization (HPO) for deep neural networks. Its main idea is to replace hyper-parameters by trainable parameters, which can be included in the training process of the network itself. The proposed method is applied to hyper-parameter tunning for two types of networks, namely CNN and FNN on two datasets MNIST and SVHN. It is also compared with several baselines. 

The authors proposed a differentiable method to optimize neural networks and hyperparameters simultaneously like DARTS. And, they used a new parameterization to represent the original discrete hyperparameters. They claimed that it could eliminate the impact of rounding.

This paper proposes to tune hyperparameters in a differentiable way by using a modified version of softmax function. It covers the tuning for three kinds of hyper-parameters as examples: channel size, kernel size and hidden layer size. Experiments on MNIST and SVHN (which are small scale datasets by modern DNN standards) show improvements on previous methods. The proposed method itself is reasonable, however, this paper misses many previous works on this topic, which makes it hard to appreciate its novelty.