Adaptive Hierarchical Hyper-gradient Descent
Authors: RENLONG JIE
Junbin Gao
Andrey Vasnev
Minh-Ngoc Tran
Venue:  (2021)
Keywords: learning rate adaptation
hyper-gradient descent
meta learning
optimisation
hierarchical system

Abstract: Adaptive learning rates can lead to faster convergence and better final performance
for deep learning models. There are several widely known human-designed adap-
tive optimizers such as Adam and RMSProp,  gradient based adaptive methods
such as hyper-descent and L4, and meta learning approaches including learning
to learn. However, the issue of balancing adaptiveness and over-parameterization
is still a topic to be addressed.   In this study,  we investigate different levels of
learning rate adaptation based on the framework of hyper-gradient descent, and
further propose a method that adaptively learns the model parameters for combin-
ing different levels of adaptations. Meanwhile, we show the relationship between
adding regularization on over-parameterized learning rates and building combi-
nations of different levels of adaptive learning rates.  The experiments on several
network architectures including feed-forward networks, LeNet-5 and ResNet-18/34
show  that  the  proposed  multi-level  adaptive  approach  can  outperform  baseline
adaptive methods in a variety circumstances with statistical significance.

Under review as a conference paper at ICLR 2021
HIERARCHICAL HYPER-GRADIENT DE- SCENT
ADAPTIVE
Anonymous authors Paper under double-blind review
ABSTRACT
In this study, we investigate learning rate adaption at different levels based the hyper-gradient descent framework and propose a method that adaptively learns the optimizer parameters by combining different levels of adaptations. Meanwhile, we show the relationship between regularizing over parameterrzzee learning rates and building combinations of adaptive learning rates at different levels. The experi- ments on several network architectures, including feed-forward networks, LeNet-5 and ResNet- 18/34, show that the proposed multi-level adaptive approach significantly outperforms baseline adaptive methods in a variety of circumstances.
INTRODUCTION
The basic optimization algorithm for training deep neural networks is the gradient descent method (GD), which includes stochastic gradient descent (SGD), mini-batch gradient descent, and batch gradient descent. The model parameters are updated according to the first-order gradients of the empirical risks with respect to the parameters being optimized, while back-propagation is imple- mented for calculating the gradients of parameters (Ruder 2016). Naïve gradient descent methods apply fixed learning rates without any adaptation mechanisms. However, considering the change of available information during the learning process, SGD with fixed learning rates can result in inefficiency and requires large amount of computing resources in eyper-parameter searching. One solution is to introduce a learning rate adaptation. This idea can be traced back the work on gain adaptation for connectionist learning methods (Sutton] 1992) and related extensions for non-linear cases (Schraudolph 1999, Yu et al. 2006). In recent years, optimizers with adaptive updating rules were developed in the context of deep learning. while the learning rates are still fixed in training. The proposed methods include AdaGrad (Duchi et al. 2011), RMSProp (Tieleman and Hinton, 2012). and Adam (Kingma and Ba 2015). In addition, there optimizers aiming to address the convergence issue in Adam (Reddi et al. al. 2 Luo et al. 2018) and to rectify the variance of the adaptive learning rate (Liu et al. 2019). Other techniques, such Lookahead, can also achieve variance reduction and stability improvement with negligible extra computational cost (Zhang et al. 2019).
Even though the adaptive optimizers with fixed learning rates can converge faster than SGD in wide range of tasks, the updating rules are designed manually while more introduced. Another idea to use objective function information and update the learning rates as trainable parameters. These methods were introduced as automatic differentiation, where the hyper-parameters can be optimized with backpropagation (Maclaurin et al. 2015: Baydin al. 2018). As gradient-basee based hyper-parameter online approach (Franceschi al., 2017). With the idea of citoo-dfffeeentationn. learning rates can be updated in real-time with the corresponding derivatives of the empirical risk (Almeida et al. 1998), which can be generated to all types of optimizers for deep neural networks (Baydin et al. 2017). Another step size adaptation approach called "L4", is based on the linearized expansion of the loss functions, which rescales the gradient to make fixed predicted progress on the loss (Rolinck and Martius 2018). Furthermore, by addressing the issue of poor generalization performance of adaptive methods, dynamically bound for gradient methods was introduced to build a gradual transition between adaptive approach and SGD (Luo et al. 2018).
are
parameter optimization methods, they can be implemented as an
Another set of approaches train an RNN (recurrent neural network) agent to generate the optimal learning rates in the next step given the historical training information, known as "learning to learn"

---

Under review as a conference paper at ICLR 2021
(Andrychowicz al. 2 This approach empirically outperforms hand-designed optimizers in variety of learning tasks, but another study has shown that it may not be effective for long horizons (Lv et al. 2017). The generalization abilitt of this approach can be improved by using meta training samples and hierarchical LSTMs (long short-term memory) (Wichrowska et al. 2017).
Beyond the adaptive learning rate, learning rate schedules also improve the convergence of optimizers, including time-based decay, step decay, exponential decay (Li and Arora 2019). The most fundamental and widely applied one is piece-wise step-decay learning rate schedule, which could vastly improve the convergence of SGD and even adaptive optimizers Luo et al. 2018 Liu al. ).19). It can be further improved by introducing a statistical test to determine when to apply step-decar (Lang et al. [2019] / et al. (2020). Also, there are works on warm-restart (O d'doogguue and Candes 2015; Loshchilov and Hutter 2017). which could improve the performance of SGD anytime when training deep neural networks.
We find that the existing gradient or model-based learning rate adaptation methods including hyper- gradient descent, L4 and learning to learn only focus on global adaptation, which could be further extended to multi-level cases. That focus aims to introduce locally shared adaptive learning rates such as the layer-wise learning rate and parameter-wise learning rate and considers all levels" information in determining the updating step-size for each parameter. The main contribution of our study can be summarized as follows:
We introduce hierarchical learning rate structures for neural networks and apply hyper- gradient descent to obtain adaptive learning rates at different levels.
We introduce a set of regularization techniques for learning rates to address the balance of global and local adaptations and show the relationship with weighted combinations.
We propose an algorithm implementing the combination of adaptive learning rates at multiple levels for model parameter updating.
MULTI-LEVEL ADAPTATION METHODS
2.1 LAYER-WISE, UNIT-WISE AND PARAMETER-WE WISE ADAPTATION
In the paper on hyper-descent (Baydin et al. 2017), the learning rate is set to be scalar. However, to make the most of learning rate adaptation, in this study, we introduce layer-wist wise or even parameter- wise updating rules, where the learning rate a in each iteration time step is considered to be vector (layer-wise) or even a list of matrices (parameter-wise). For the sake of simplicity, we collect all the learning rates in vector: Or, = (ar1,t, ..., aN..:² Correspondingly, the objective f()) is function of 0 (01, .2..... 0N) collecting all the model parameters. In this case, the derivative of the objective function f with respect to each learning rate can be written as
0f(0.-1) (f ON,C-1)
N 0f(01,-1,..., 1x 00
(I)
00.,1-1
da;,1-1
00,
80,,1-1
where N is the total number of all the model parameters. Eq. wise updating, where we associate a learning rate with a special group of parameters, and each parameter group is updated according to its only learning rate. Notice that although there is dependency between 0--1 and 0,-2 with: ar-1 at-2 (0,-2) where 3 is the updating rate of hyper-gradient descent, we consider that ar-1 is calculated after 0,-2 and thus change of Q1-1 will not result in a change of f.-2. Assume 0, = u( 1,, a:) is the updating rule, where O, {0.)=-0 and a is the learning rate, then the basic gradient descent method for each group gives 0 = u( (1-1,, (1-11,001-1) 014-1) = 0 0 S (0, Hence for gradient descent
be generalized to group-
@f(0.-1) Vo, () a.... " (0.1,,
(0.
(2)
das,-11
Here ai,-1 is a scalar with index i at time step / - 1, corresponding to the learning rate of the äth group, while the shape of f(0) is the same as the shape of 0,. We particularly consider two special cases: (1) In layer-wise adaptation, 0, is the weight matrix of ith layer, and a, is the particular learning rate for this layer. (2) In parameter-wise adaptation, 0 corresponds to a certain parameter involved in the model, which can be an element of the weight matrix in certain layer.
2

---

Under review as a conference paper at ICLR 2021
2.2 REGULARIZATION ON ADAPTIVE LEARNING RATES
The selection of adaptation level should depend on a case-by-case basis. Global or parameter- wise adaptation is usually not the optimal choice across all circumstances. Recall that for deep neural networks, we typically use a relatively large architecture with regularization. This idea can also be applied learning rate space with parameter structure. To address over-parameterization in implementing lower-level learning rate adaptation, we introduce regularization on learning rates to control the flexibility. First, for layer-wise adaptation, can add the following regularization term to the loss function
Lirree Xlayer I(a
where is the indices for each layer, Alaye is the layer-wise regularization coefficient, ay and ag the layer-wise and global-wise adaptive learning rates. A larger Atayer can push each layer's learning rate towards the global learning rate across all the layers. Given particular Cg the gradient of the loss function with respect to the learning rate on in layer can be written as
ÓL a)
mL (0,
aL reg (0,a)
dag
dan,
dan,
(4)
( Vor.- ( 11- 2Atayer (gg,t).
Notice that the time step index of layer-wise regularization term is t rather than - 1, which ensures that we push the layer-wise learning rates towards the corresponding global learning rates of the current step f. Denoting by hı,r-1 (o, V
(1,-1),, then the updating rule for
learning rates can written as
L (0, (0,a) = 01,111 1 3(-h,,-1 + 2Amyer - ag,t)).
04,1 = 01,
(5)
dag
Eq. (5) has close form solution but only applicable in the two-levels case. However, there is an extra hyper-parameter Auaee to be tuned. addition, when there are more levels, components of learning rates at different levels can be interdependent. To construct workable updating scheme for Eq. ( we replace ai and ag. with their relevant approximations. We take the strategy of using their updated version without considering regularization, i.c., dg,b 01 + Sh1, and åg.t= 0g + Bhg,t-1, where f - ( (VV u()),2, Qg.-11
20, is the global h for all parameters. Here we regard i and &g as the "virtual" layer-wise and global-wise learning rates for time step and taking them into the right-hand side of Eq. (5) gives the new updating rule as follows
ai, = 01,11 + Bhh 1 2.y (01, ågge)) (1 2,3 Xtyer Jái, 219
(6)
Notice that in Eq. (6). the two terms are actually weighted average of the layer-wise learning rate âr, and global learning rate åg.t at the current time step. Since we hope to push the layer-wise learning rates towards the global one, the parameters should meet the constraint: 0 < 2,3 < 1, and thus they can be optimized using hyper-parameter searching within a bounded interval as well gradient-based
paraameerr optimizations. We can also consider the case where three levels of learning rate adaptations are involved, including global-wise, layer-wise, and parameter-wiss adaptation. If introduce two more regularization terms control the variation of parameter-wise learning rate with respect to layer-wise learning rate and global learning rates, the regularization loss can be written as
Li para
Alayer E(ar -xg)² Appra_laye layer Σ E(aw a1)? + Apara (aw P
where Cppl the learning rate for the p-th parameter inside layer The second and third terms push each parameter-wis wise learning rate towards the layer-wise learning rate and the global learning rates, respectively. Like the two-level case, the updating rule with this three-level regularization can be approximated by the weighted combination of three components under "virtual approximation". The detail of the updating rule for the three-levels case is given by Algorithm[1] in Section 2.3] We also provided discussion the bias of implementing "virtual approximation" in Appendix A.1.
In general, we can organize all the learning rates in a tree structure. For example, in the three-level case above, og will be the root node, while (az) are the children node at level of the tree and {at))
3

---

Under review as a conference paper at ICLR 2021
the children node of ag leaf nodes at level three of the tree. In general case, assume there are L levels in the tree. Denote the set of all the paths from the root node to each of leave nodes as P and a path is denoted by p = {ai d2...., ar,} where an is the root node, and a is the left node on the path. On this path, denote (1 all the ancestor nodes of og along the path, i.e., ancestors(i) {ai,..., 03-1); ). We will construct regularizer to push a; towards each of its parents. Then the regularization can be written as
Lirreg ΣΣ Σ
aay)
pEP a. og Eancestor(s)
Under this pair-wise L2 regularization, the updating rule for any leave node learning rate or, can be given by the following theorem whose proof is provided in Appendix A.2
Theorem 1. Under virtual approximation, the effect of applying pair-wise L2 regularization Eq. 7 resulty in performing weighted linear combination of virtual learning rates different levels of = Σド=1 75ā0 with Σj=1 = 1, where each component 0z calculated by assuming no regularization.
Remarks:
Theorem actually suggests that a similar updating rule can be obtained for the learning rate at any level on the path. All these have been demonstrated in Algorithm | for the three-level case.
2.3 PROSPECTIVE OF ADAPTIVE LEARNING RATE COMBINATION
Motivated by the analytical derivation in Section 2.2 we can consider combining adaptive learning rates at different levels as substitute and approximation of regularization on the differences in learning rates. This makes the effect of learning rate regularization trainable with gradient-based methods. In general form, assume that we have L levels, which could include global-level, layer-level, unit-level and parameter-level etc, Theorem suggests the following updating rule:
ar = Σj= 7j e. In more general form, we can implement non-linear models such as neural networks model the final adaptive learning rates with respect the learning rates at different levels:a; = g(a1,;; 02.,....r1.111 0), where 0 is the vector of parameters of the non-linear model. We can treat the combination weights (71:--.,774 as trainable parameters, which can also be globally shared or parameter/laerr-seeiffe weights. We only need these different levels of learning rate to have hicrarchical relationship to apply this method. For example, we can further introduce "filter level" to replace layer-level for the convolutional neural network if there no clear layer structure, where the parameters each filter will share the same learning rate.
In this study, we consider the globally shared combination
As the real learning rates implemented in model parameter updating are weighted combina- tions, the corresponding gradient matrices cannot be directly used for learning rate updating. In this case, we first break down the gradient by the combined learning rate to three levels, each of them to update the learning rate at each level, and then calculate the combination by the updated learning rates. Especially, hy a hy and hy. are calculated by the gradients of model losses without regularization, is shown in Eq. 8
hp.t = ())
= VoS(O.-1,,)) Vou(O,-2,)))
Daptt
(f(0.a)
hi,
dan.
(8)
af(0,a)
hggt
dai
where to = - hu. E, hp and f1,1 Epeth layer hp and f(0, a) corresponds to the model loss Temadee (0, (x) in Section 2.2 Algorithm is the full updating rules for the newly proposed optimizer with three levels, which can be denoted as combined adaptive multi-level hyper-gradient descent (CAM-HD). In Algorithm| [ we introduce the general form gradient descent based optimizers (Reddi al. 2018 Luo et al. 2018); for SGD, Φe (g 9t and (g ...))) 1, while
'eere we form represent the sum of the element-wise product but compute in the simplest way.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Updating the rule of three-level CAM-HD
input: 0oo, 3, 6, T
initialization: 0o, 71,0, 72 Y3,0, Op.0s or,0, 00, 0i,0 = 7.00pp0 + 72,004,0 + 73,000 for t E 1,2, ..... 2... do
= (f)
Update hp.ts h1, and hgg by Eq. (8)
Op ap,t-1 3p p p.t-1 Jay.e1
01(4-11) dap--1
ap.t-1 11
01(0,-1) 0o
01, 01, - Ep 8
dar
01,111
1172,-11 1 Ep Mp.t 01,1-1 - -
0t 04-1
0t-1 8g E1 Ep 80 1-1 001-1
01
Da
Bg
p. 71,--1 1 + 14,,1 + 73,1-10
71, 71,-1
71,-1 0 Σp da 0m 8 Um
OL dap--1
80xp,111 Ep da 71,111
0L
22 = -2,-1 8 8-22111
12,111 Ep 8a, p.1- 0
0L
0a,
1/1-111 -ō0,1-1 1 Ep 8a p,1-1
33,t = 7311- 8 B1..1
-3.111 Ep 80 p.t-1 B1..1
0L da
-3,1111 - -c0t-1 Ep Br p,--1-1
al
71 7 nn(m + 72 73), 72 = 7 + 7/ + 73), 73 = 71/(11 + 2 + 73)
my = O1 (g1 !t)
V = Vi (91, I..
0, 8 p m m//VV.
end
return 0ry 71,7, Y2,r> 73,7', Op, 04,77 OC
9; Meanwhile, the corresponding u(O,-2, a) should be changed accordingly. Notice that in each updating time step of Algorithm [ we re-normalize the combination weights 71 7/2 and 73 to ensure that their summation is always even after updating with stochastic gradient-based methods. An alternative way of doing this is to implement Softmax function. In addition, the training of ys can also be extended to multi-level cases, which means different layers. For the updating rates 8p. 8g and 3, of the learning rates at different levels, we set: 3, 9 B, B1 n23, Bg n.3, where is shared parameter. This setting will make the updating steps of learning rates at different levels be on the same scale considering their difference in the number of parameters. An alternative way is take the average based on the number of parameters in Eq. (8) at first.
for Adam, (i 9()
1 9: and (e(g).....)))
18
Bz diag(E) 1 1
have different combination weights in
2.4 CONVERGENCE ANALYSIS AND ALGORITHM COMPLEXITY
The proposed CAM-HD is not an independent optimization method, which can be applied in any gradient-based updating rules. Its convergence properties highly depend on the base optimizer that applied. By referring to the discussion on convergence (Baydin al. 2017). we introduce Kppt = (f)o + (1 ((t))a.., where the function r(t) is selected to satisfy (t( - 0 as t -> oc, and a:s0 is a chosen constant value. Then we can demonstrate the convergence analysis for the three-level case in the following theorem.
Theorem 2 (Convergence under mild assumptions about f). Suppose that f is convex and L-Lipschitz smooth with || | (0)] < Mp. [[Vif(0)]| < M, [[ < Mg for some fixed Mg. M,. M, and all 0. Then Or 0 if axx L/L where L is the Lipschitz constant for all the gradients and t T()) + 0 as f. + oo, where the Og are generated according to (non-stochastic) gradient descent.
In the above theorem, p is the gradient of target function w.r.t. model parameter with index Vi the average gradient of target function w.r.t. parameters in layer with index I, and V, is the global average gradient of target function w.r.t. all model parameters. The proof of this theorem is given in Appendix A.3. Notice that when we introduce Kp.t instead of a;, in Algorithm I the corresponding
gradients Sa 7-1-1
a will also be replaced by OL(O) Di 3Ng,e-1 Do 009,1-1 0Ng,e-1 T()).
aL(O)
5

---

Under review as a conference paper at ICLR 2021
We also made an analysis the number of parameters and algorithm complexity (See Appendix A.4). Our method will not increase the number of model parameters but requires extra space complexity AT during training. Also it requires extra time complexity but at least one-order smaller than training with baseline setting, while the absolute ratio is smaller than the inverse of batch size AT
T << 1 1/mb-
EXPERIMENTS
We use the feed-forward neural network models and different types of convolutions neural networks on multiple benchmark datasets to compare with existing baseline optimizers. For each learning task, the following optimizers will be applied: (a) standard baseline optimizers such as Adam and SGD; (b) hyper-gradient descent (Baydin et al., 2017); (c) L4 stepsize adaptation for standard optimizers (Rolinek and Martius 2018): (d) Adabound optimizer (Luo et al. 2018); (e) RAdam optimizer (Liu cct al. 2019); and (f) the proposed adaptive combination of different levels of hyper-descent. The implementation of (b) is based the code provided with the original paper. For each experiment, provide the average curve and standard error bar in each time step with ten runs.
3.1
TUNING
To compare the effect of CAM-HD with baseline optimizers, we first do hyperparameter tuning for the model training process with baseline optimizers by referring related papers (Kingma and Ba 2015; Baydin et al. 2017 Rolinek and Martius 2018 Luo et al. 2018)) as well as implementing an independent grid search. We mainly consider the hyper-pane papper-praaeters of batch size, learning rate, and momentum for models with different architecture. The search space for batch size is the set of {2") m==3...., while the search space for learning rate, hyper-gradient updating rate and combination weight updating rate (CAM-HD-Ir) are {10-1 ..... 10 10-4). {10-1 ..... 10 10} and {10-1,... ..... 10 +} respectively. The selection criterion is the 5-fold cross-validation loss by early-stopping at the patience of 3. The optimized eyper-parameter for the tasks in this paper are given in Table [ For the learning tasks with recommended learning rate schedules, we will apply these schedules as well.
Table I: Hyperparameter Settings for Experiments
Architecture
Dataset
(Adam)
Hyper-grad (SGDISGDN)
Hyper-grad (Adam)
Ir
CAM-HD-Ir
(SGDISGDN)
MI.P
32
0.0003
1.00-007
0.01
2LP MI.P
MNIST
64
0.001
1.005-07 07
0.01
128
1.00F-07
0.01
MNIST
256
0.001
1.005-03
1.00E-08
0.03
LeNet-5
CIFARIO
256
0.001
1.00F-03
1.00F.08
0.03
SVHN
128
0.001
1.00E-03
1.00E-08
0.03
ResNet-18 18 ResNet-34
256
0.1
0.001
1.00F-06
1.00E-08
0.001
CIFARIO
256
0.1
0.001
1.005-06
1.00E-08
0.001
3.2 COMBINATION RATIO AND MODEL PERFORMANCES
First, we perform study on the combination of different level learning rates. The simulations based on image classification tasks on MNIST and CIFARIO 10 (LeCun et al.] [1988 Krizhevsky and Hinton 2012). One feed-forward neural network with three hidden layers of size [100, 100, 100] and two convolutional network models, including LeNet-5 (LeCun et al. 2013) and ResNet-18 (He et al. 2016), are implemented. We use full training sets of MNIST and CIFARIO for training and full test sets for validation. In each case, two levels of learning rates are considered, which are the global and layer-wise adaptation for FFNN, and global and filter-wise adaptation for CNNs. Adam-CAM-HI optimizer is implemented in all three simulations. We change the combination weights of two levels in each case to see the change of model performance in terms of test classification accuracy at epoch 10, with the corresponding updating rate 8 O. Another hyper-parameter setting follows Table [ We conduct ten runs at each combination weights with different parameter initializations for all three simulations and draw the error bars for standard errors. The result is given in Figure I We can that in all three cases, the optimal performance is neither at full global level nor full layer/filter level, but combination of two levels of adaptive learning rates. Still, the differences between the endpoints

---

Under review as a conference paper at ICLR 2021
77.4
71.2
63.5
27.0
63.0:
76.0
07.7
67.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0-4
0.6 0.8
1.0
0.0
0.2
0.4
0.6 0-8 10
(37 Gamma (Global, FFNN, MNSST
(b)
Gomma 18lebos, LoNet- CIAALO)
[ct Gammaa (Gicc, Reshiet CIRAR10)
Figure The diagram of model performances (at epoch 10) trained by Adam-CAM-HD with
different fixed combination ratios in the case of two-level learning rates adaptation.
96
98.0
90.0
Valdatton Accurno
07.5
97.5
97.0
06-
06.5
96.5
Adam CAM levell
96.0
Adam CAM (3-lxel
Adbm CAM HD levee
94
Adam-HD ttp-gred 1e-91
Adam CAM levell
93.5
Adam-HD trp-grad Ie-91
Adam CAM levell
2
Adam-HD (tp-grad 16-91
Adom CAM levell
Adam-DD the grad 16-71
AdamHD trp prp-graddde 16-71
Adam-HD (tp-grad 16-71
Adam
94.3
Adam
94.5
Adam
10
20 75
10
20
75
10
20
25
(6) Time Step Oiateen now (100,1000 1000)
(5) Time Step Iiltction sine (1000,100)
[21 Time Step (liiden sine (1000, 10001)
Figure 2: The comparison of learning curves of FFNN on MNIST with different adaptive optimizers.
and the optimal combination in terms of model performance have some statistical significance level. This supports analysis in Section [.2] Also, in real training processes, it is possible that the learning in favor of different combination weights in various stages and this requires the online updating of the combination weights.
3.3 FEED FORWARD NEURAL NETWORK FOR IMAGE CLASSIFICATION
This experiment is conducted with fced-forwarts neural networks for image classification on MNIST, including 60,000 training examples and 10,000 test examples. We use the full training set for training and the full test set for validation. Three FFNN with three different hidden layer configurations implemented, including [100, 100], [1000, 100], and [1000, 1000]. Adaptive optimizers including syper-gradient updating rates, and proposed Adam-CAM-HD applied.
Adam, Adam-HD with For Adam-CAM-HD, we apply three-level parameter-layer-global adaptation with initialization of 71 7/2 0.3 and 7/- 0.4, and two-level layer-global adaptation with 71 72 0.5. Figure 2 shows the validation accuracy for different optimizers during the training process of 30 epochs. We can learn that both the two-level and three-level CdMm-CAM-HD outperform the baseline Adam optimizer with optimized hyper-parameters significantly. For Adam-HD, we find that the default hyper-gradient updating rate (10-²) ") for Adam applied (Baydin et al. 2017 is not optimal in experiments, while optimized one of 10 can outperform Adam but still worse than Adam-CAM- with default hyper-gradient updating rate (10-7).
3.4 LENET-5 FOR IMAGE CLASSIFICATION
The second experiment is done with LeNet-5, an early-year convolutional neural network without involving many building and training tricks. We compare set of adaptive Adam optimizers including Adam, Adam-HD, Adam-CAM-HD. and L4 for the image classification learning task of MNIST, CIFARIO and SVHN (Netzer al. 2011). For Adam-CAM-HD,
apply two-level setting with
filter-wist and global learning rates adaptation and initialize ?1 = 0.2, 7/2 = 0.8. We also implement an exponential decay function 7tt) (xp(-rt)
discussed in Section 2.4 with rate 0.002, while is the number of iterations. For L4. we implement the recommended L4 learning rate of 0.15. For Adabound and R AAdam, also apply the recommended typee--aramettes in the original
7

---

Under review as a conference paper at ICLR 2021
papers. The other ppper-prrmette settings optimized in Table I
As can see in Figure
99.2
95.0
60-
86-
06.8
Adam
Adam-CAN
Adam-CAM
08.4
AdamelID [Pievell
AdamolD (Pievel)
Adam-ilD (Pinvel)
Adami4
Adamil
Adam-l4
68.2
Adtbound
45
Adtbourd
Adsbourd
96.0
RAdam
40
RAdam
80-
RAdam
Adam
(4) Fpach (MAUT)))
061 Fooch (CIFART))
(cl Fpoch countk
Figure 3: The comparison of learning curves of training LeNet-5 with different adaptive optimizers.
Adam- CAam-CAM-HD again shows the advantage over other methods in all the three sub- sub-experiments. except MNIST L4 that could perform better in a later stage. The experiment on CIFARIO and SVHN indicates that the recommended hyper-parameters for Adabound, R Adam and could fail in some cases with unstable accuracy curves. On the other hand, Adam-HD can not significantly outperform Adam with the recommended and optimized hyper-gradient updating rate shared with Adam-CAM-HD. The corresponding summary of test performance is given in Table 2 in which the test accuracy of Adam-CAM-HD outperform other optimizers on both CIFARIO 10 and SVHN. Especially, it gives significantly better results than Adam and Adam-HD for all the three datasets.
Table 2: Summary of test performances with LeNet-5
MNIST
CIFARIO
SVHN
Test acc
Test S.E
Test acc
Test S.E
Test acc
Test S.E
Adam-CAM-HD
98.93
0.07
65.55
0.18
87.58
0.37
Adam-HD
98.83
0.05
63.3
0.66
86.94
0.13
Adam-L4
99.19
0.05
63.76
0.26
85.44
0.42
Adabound
99.11
0.05
59.79
0.70
87.22
0.14
RAdam
98.94
0.06
61.17
0.79
87.31
0.41
Adam
98.89
0.05
63.88
0.45
86.82
0.16
3.5 RESNET FOR IMAGE CLASSIFICATION
In the third experiment, we apply ResNets for image classification task CIFAR10. We compare Adam and its adaptive optimizers, as well as SGD with Nestorov momentum (SGDN) and corre- sponding adaptive optimizers for training both R and ResNet-34. For SGDN methods, apply a learning rate schedule, in which the learning rate is initialized to a default value of 0. and reduced 0.01 10% (for (GDN-CAM-HID after epoch 150. The momentum is set to be 0.9 for all SGDN methods. For CAam-CCM-HD SGDN-CAM-HDD we apply two-level CAM-HD with the same setting as the second experiment. In addition, we apply an exponential decay function with decay rate 0.001. The validation accuracy results, training loss, and validation loss are shown in Figure 4 We can see that the validation accuracy of Adam-CAM-HD reaches about 90% in 40 epochs and consistently outperforms Adam, L4 and Adam-HD optimizers in a later stage. The L4 optimizer with recommended
and an optimized weight-decay rate of 0.0005 (instead of 1e-4 applied in other Adam-based optimizers) and ResNet-34, while its training loss outperforms all other methods but with potential over-fitting. Adam-HID achieves better training loss than Adam after epoch 100. However, we find that the validation performance is not good with a default hyper-gradient coefficient of 10-8 (shared with Adam-CAM-HDD Instead, an optimized coefficient of 10 can make safe but small improvement from Adam. RAdam performs slightly better than Adam-CAM-HD in terms of validation accuracy, but the validation eross- cross-entropy of both RAdam and Adabound are worse than our method. Also, we find that in training ResNet- 18/34, the validation accuracy and validation loss of SGDN-CAM-HI slightly outperform SGDN in most epochs even after the resetting of the learning rate at epoch 150.
outperform baseline Adam for both ResNet-
8

---

for Under review as a conference paper at ICLR 2021
00
Adwm-Il
Adwm-CAMA4
Adwm-Il
AdwI-CAMA41
Pereetoge Acuuo
80
Acam
AdamiL4
Adalbound
Adam-L
Adabound
70
Acart-L4
2.0
Adam
1.0
Adam
60
Acaibound
Cooss SGDN-EAMM
0.8
Acam
80
SGON
50
100
156
200
50
100
150
200
50
100
150
200
(0) Time Step Nolidotioon Accurzcy1
061 Tiiee Step (Training Loss)
10 Time Stop Ivalidation Loss!
00-
1.6
80-
1.5
1.4
Acom CAM HD
70-
Acam-HD
Acart
Adamn
Acam
Acaltound
500%
500%
peccontegg
RAdeem
Acom
SCON CAMADD
ScoN
6.0
100
300
(4) Thmee Step Dialidatton Arcurscyl
(4) Time Step STraming 1oo
(1) Time Step CVn townoon Tree))
Figure 4: The learning curves of training ResNet on CIFARIO with adaptive optimizers.
The test performances of dif- ferent optimizers for ResNet-18 and ResNet-34 after 200 epoch of training are shown Ta- ble 3 Notice that the results of SGDN and SGDN-CAM-HD achieved with piece-wise constant learning rates schedule, while the results of Adam-based optimizers are achieved without learning rate schedule. We can learn that the proposed CAM- HD method can improve the corresponding baseline method (Adam, Adam-HD and SGDN) with statistical significance in almost every case. Adam-CAM-HD performs bit worse than RAdam but comparable to Adabound in terms of the average test accuracy for both ResNet-18 and ResNet-34, As higher-level adaptation method, CAM-HD can be applied on top of further improvement.
Table 3: Summary of test performances with ResNet-18/34
ResNet-18 18
ResNet-34
Test ace
Test S.E
Test acc
Test S.E
Adam
86. 94
0.13
87.87
0.2
Adam-HD
87.26
0.35
88.48
0.48
Adam-L4
87.81
0.22
88.02
0.15
Adabound
90,29
0.15
90.15
0.30
RAdam
91.54
0.17
91.76
0.28
Adam-CAM-HD
90.10
0.23
90.18
0.06
SGDN
93.04
0.23
92.93
0.19
SGDN-CAM-II
93.2
0.24
93.47
0.23
CONCLUSION
In this study, we propose a saudient-based learning rate adaptation strategy by introducing hicrarchical multiple-level learning rates deep neural networks. By considering the relationship between regu- larization and the combination of adaptive learning rate at different levels, we further propose a joint algorithm for adaptively learning each level's combination weight. Experiments on FFNN, LeNet-5, and ResNet-18/34 indicate that the proposed methods outperform the standard ADAM/SGDN and other baseline methods with statistical significance. Although the advantage is not fully guaranteed, method achieves higher adaptation level and can be continuously reduced to baseline methods under a specific set of ceper-parameters. This could bring more thoughts and further study on implementing hierarchical learning rate system for deep neural networks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
L. B. Almeida, Т. Langlois. J. D. Amaral, and A. Plakhov. Parameter adaptation in stochastic optimization. On-Line Learning in Neural Nenworks, Publications of the Newton Institute, pages 111-134, 1998.
M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffmann, D. Pfau, Schaul, B. Shillingford. and N. De Freitas. Learning learn by gradient descent by gradient descent. In NeurIPS, pages 3981-3989, 2016.
A. G. Baydin, R. Comissh D. M. Rubio, M. Schmidt, and F. Wood. Online learning rate adaptation with hypergradient descent. ICLR, 2017.
A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: survey. JMLR, 18(153), 2018.
J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12:2121-2159, 2011.
L. Franceschi, M. Donini, P. Frasconi, and M. Pontil. Forward and reverse gradient-based hyperpa- rameter optimization. In ICML, pages 1165-1173. JMLR. org, 2017.
K. He and Sun. Convolutional neural networks constrained time cost. In CVPR, pages 5353-5360, 2015.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
H. Karimi, J. Nutini, and M. Schmidt. Linear convergence of gradient and proximal-gradient methods under the polyak-lojasiewicz condition. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 795-811. Springer, 2016.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015. 2015.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. University of Toronto, 2012.
H. Lang, L. Xiao, and P. Zhang. Using statistics automate stochastic optimization. In Advances in Neural Information Processing Systems, pages 9540-9550, 2019.
Y. LeCun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-234 1998.
Y. LeCun al. Lenet-5, convolutional neural networks. URL.: http://yaan.. lecun. com/exdb/lenet, 20: 5, 2015.
Z. Li and S. Arora. An exponential learning rate schedule for deep learning. In International Conference on Learning Representations, 2019.
I. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond. ICLR, 2019.
I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. 2017.
L. Luo, Y. Xiong, Y. Liu, and X. Sun. Adaptive gradient methods with dynamic bound learning rate. In ICIR, 2018.
K. Lv, S. Jiang, and J. Li. Learning gradient descent: Better generalization and longer horizons, In ICMI., pages 2247-2255. JMLR. org, 2017.
D. Maclaurin, D. Duvenaud, and Adams. Gradient-based hyperparameter optimization through reversible learning. In ICML, pages 2113-2122, 2015.
10

---

Under review as a conference paper at ICLR 2021
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http::/ www.:odd..aaaa.o...o.o.oo.... htt.:/a..........ooo.atooaaaaaaa
ni nips2011 hww.suumbesssssss..ooom pddf
B. O' d'donoguue and E. Candes. Adaptive restart for accelerated gradient schemes. Foundations of computational mathematics, 15(3):115-732, 2015.
Reddi, S. Kale, and S. Kumar, On the convergence of adam and beyond. In ICLR, 2018.
M. Rolinek and G. Martius. L4: Practical loss-based stepsize adaptation for deep learning. In NeurlPS, pages 6433 6443, 2018.
S. Ruder. An overview of gradient descent optimization algorithms. arXiv: 2016.
N. N. Schraudolph. Local gain adaptation stochastic gradient descent. 1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), volume 2, pages 569-574 vol.2, 1999.
R. Sun. Optimization for deep learning: theory and algorithms. arXiv:1912.08957. 2019.
R. S. Sutton. Gain adaptation beats least squares. In Proceedings of the 7th Yale workshop on adaptive and learming systems, volume 161168, 1992.
Tieleman and G. Hinton. Rmsprop: Divide the gradient by rumning average of its recent magnitude. coursera: Neural networks for machine learning. Tech. Rep.. Technical report, page 31, 2012.
O. Wichrowska, Maheswaranathan, M. W. Hoffman, S. Colmenarejo, M. Denil, N. de Freitas, and J. Sohl-Dickstein Learned optimizers that scale and generalize. In ICML, pages 176600. JMLR. org, 2017.
J. Yu, D. Aberdeen, and N. N. Schraudolph. Fast online policy gradient learning with smd gain vector adaptation. In NeurIPS, pages 1185-1192, 2006.
M. Zhang, Lucas, Ba, and G. Hinton. Lookahead optimizer: steps forward, step back. In NeurlPS, pages
2019.
P. Zhang, H. Lang, Q Liu, and L. Xiao. Statistical adaptive stochastic gradient methods. arXiv:2002. 10597, 2020.

---

Under review as a conference paper at ICLR 2021
A APPENDIX FOR PAPER: ADAPTIVE MULTI-LEVEL HYPER-GRADIENT DESCENT
A.1 ANALYSIS ABOUT VIRTUAL. APPROXIMATION
Consider the difference between Eq. (5) and Eq. 6 in the paper:
ai,- - ai, = Atayer âg,t) (ai,t- ag,)))
(9)
Based on the setting of multi-level adaptation, on the right-hand side of Eq. (9. global learning rate updated without regularization åg,t Qg.t. For the layer-wise learning rates, the difference is given by Ōg Or, 2,3 2ayrr (Cu,t (.g.)) which corresponds to the gradient with respect to the regularization term. Thus, Eq. (9) can be rewritten as:
oi, OKt 2,3 Xtayer (2 (2g)a (al. ag.t)) -4
Dgg.t 01
JOLA
(10)
which is the error of the virtual approximation introduced in Eq. (6) in the paper. If 43 << 1 or Oud + 1, this approximation becomes accurate. Another way for handling Eq. (4 in the paper is to 01.1 implement the previous previous-step learning rates in the regularization term.
(11) Since we have 04, = dr,t 2,3 (a og,,z) and مr,+ 04,--1 + ,31, using the learning rates in the last step for regularization will introduce a higher variation from term 3/1,t-1, with respect to the true learning rates in the current step. Thus, we consider the proposed virtual approximation works better than last-step approximation.
O1, 01, 3(- + 2Ammer(O0I,-
Qg.t-1)).
A.2 PROOF OF THEOREM 1:
Proof. Consider the learning regularizer
Lir (a)
ΣΣ Σ a
-
(12)
pEP P. a, a Eparents(s)
To apply the derivative of Lteree with respect az, The terms in Eq. 12 involving ar are only (as; a2)² where oj is an ancestor on the path from the root to the leave node ar, Hence
descent method to update the learning rate a at level L, we need to work out
On ax)
L
da,,
daL,
(13)
o., f (0.1)) VV u(O, 1)-1)++
Σ 0, Encenstoors(L)
Dai, -
As there are exactly L --1 ancestors on the path, we can simply use the index j = 1,2, L The corresponding updating function for Un,t is:
a = On,t-1 1: +
22L - az,)))
L-1 )LSan,)) +
L-1
áz, - -23
Σ
(2 (2372,
(14)
j=1
/=1
Σ Trājt.
j=1
where
L-1
7L 28
X
(15)
j=1
91 = 28
for j = 2,., L..,..---. 1-1.
This form satisfies of = -ン with Σ 7j 1. This completes the proof.
12

---

Under review as a conference paper at ICLR 2021
A.3 PROOF OF THEOREM 2:
Proof. We take three-level's case discussed Section22oo example, which includes global level, layer-level and parameter-level. Suppose that the target function convex, L-Lipschitz smooth all levels, which means for all 0, and 02:
V vff((1)) - Vpf(f22)1 - 1p - 82
[V,) v.f(02))l - 4i|0 021
(16)
I (V))())) - vof(022)l S 40l10 - 02ll
L = max
and its gradient with respect to parameter-wise, layer-wise, global-wise parameter groups satisfy [V)]())] < Mn [V,](00]] < Mr / M, for some fixed Mp. M,, M, and all 0. Then the effective combined learning rate for each parameter satisfies:
la = 11pp--10pp + a + - 10
$ (1 + - + +
Σ
7p.t- max (
1=
+1 max ( ([((),)))) + V/( /11
(17)
$ 00 +
Sp.t-111 max
1=
+ + 1 max
5.0 + ( M2 + M²)
where p,. refers to the value of parameter indexed by p at time step 0r.s refers the set/vector of parameters in layer with index 1 at time step i, and 0g. refers to the whole set of model parameters at time step i. In addition, Mp and nı are the total number of parameters and number of the layers, and we have applied 0 7p. R, 7o 1. This gives an upper bound for the learning rate in each particular time step, which is (tt) as t - 20. By introducing Kp,t = r(t)a + (1 - a()) where the function r()) is selected to satisfy (t) - 0 as t + co, so we have Kp.t - ase as + Do. 0o < 1. for larger enough we have 1/(L+11) + < kp.t < 1/ I., and the algorithm converges when the corresponding -rasient-based optimizer converges for such learning rate under assumptions about f. This follows the discussion in (Karimi et al. 2016 Sun 2019).
A.4 ALGORITHM COMPLEXITY
A.4.1 NUMBER OF PARAMETERS AND SPACE COMPLEXITY
The proposed adaptive optimizer is for efficiently updating the model parameters, while the final model parameters will not be increase by introducing CAM-HD optimizer. However, during the training process, several extra intermediate variables introduced. For example, in the discussed three-level's 's case for feed-forward neural network with Player layers, we need to restore hp.s, h4e and
hg.t, which have the sizes of Shhp.)) Σ/ Rtaaa- (ny + 1)m4+1. S(hy,) Fyayer and S(hg.t)
respectively, where n; is the number of units in ith layer. Also, learning rates Ctp,t, a1 ag. and
take the sizes of S(ap.)) Emer -ipe- (ny 1)m(11. S(ar,) Players S(ag,t) 1, S(ag,) and S(a Mayger (ny + 1) respectively. Also we need small set of scalar parameters to (=1 restore 71, 72 and 73 and other coefficients.
Consider the fact that in training the baseline models, we need to restore model parameters, corresponding gradients, as well as the intermediate gradients during the implementation of chain rule, CAM-HD will take twice of the space for storing intermediate variables in the worst case. For two-level learning rate adaptation considering global and layer-wise learning rates, the extra space complexity by CAM-HD will be one to two orders' smaller than that of baseline model during training.
13

---

Under review as a conference paper at ICLR 2021
A.4.2 TIME COMPLEXITY
In CAM-HD, we need to calculate gradient of loss with respect to the learning rates at each level, which are hp,t, fu, and hg.r in three-level's s case. However, the gradient of each parameter is already known during normal model training, the extra computational cost comes from taking summations and updating the lowest-level learning rates. In general, this cost is in linear relation with the number of differentiable parameters in the original models. Here we discuss the case of feed-forward networks and convolutional networks.
Recall that for feed-forward neural network the whole computational complexity is:
Tlkyer
T()) Omm Thiter
Σ
74-1 -1122)
(18)
1=2
where m is the number of training examples, Piter is the iterations of training, and is the number of units in the 1-th layer. On the other hand, when using three-level CAM-HD with, where the lowest level is parameter-wise matrix element summations to calculate h4, for all layers, as well as a list summation to calculate hg. In addition, two clement-wiss summations will also be implemented for calculating O'p,t and ar Therefore, the extra computational cost of using CAM-HD is (T O(, nutr D1=2 (n, 71-1 + ni)), where n, is the number of batch size, which usually larger than 100. This extra cost is more than one-order smaller than the computational complexity of training model without learning rate adaptation. For the cases when the lowest level is layer-wise, only one element matrix product is needed in each layer to calculate hu,z For convolutional neural networks, we have learned that the total time complexity of all convolutional layers is (He and Sun
need Raaye element products to calculate hp.t for all layers,
Rayyr
for training. Notice that m, mm/nb is the
2015):
Reano
(ng
Viter
Σ
(ML-1 of ny mt))
(19)
where is the index of a convolutional layer, and Boonv layer is the depth (number of convolutional layers). ny is the number of filters in the 1-th layer, while n4-1 is known the number of input channels of the /-th layer. 8t is the spatial size of the filter. my is the spatial size of the output feature map. If we consider convolutional filters as layers, the extra computational cost for CAM-HD in this case is AT(n) = Onn, Titer Ee Cageentttterrrroo smaller than the cost of model without learning rate adaptation.
tayer ((m-11 sf + 1) n,)), which is still more than one order
Therefore, for large networks, applying CAM-HD will not significantly increase the computational cost from the theoretical prospective.
14

---

Under review as a conference paper at ICLR 2021
A.5
SUPPLEMENTARY EXPERIMENTAL RESULTS
A.5.1 LEARNING OF COMBINATION WEIGHTS
The following figures including Figure 5 Figure 6 Figure 7 and Figure give the learning curves of combination weights with respect to the number of training iterations in each experiments, in which each curve averaged by 5 trials with error bars. Through these figures, we can compare the updating curves with different models, different datasets and different CAM-HD optimizers.
0.45
0.38
0.400
0.36
0.300
0.275
0.25
0.30
0.20
1000012500 Iteration 100), Adam)
2500 5000
2500 5000 10000:.500
2500 5000 10000
Iteration IPNNN 11000. 1001, Adami
(c) Iteration (PINN (1000, 1000). Adam)
Figure 5: Learning curves of ys for FFNN on MNIST with Adam.
0.8
0.7
0.6
Value
0.5
0.5-
0.5
0.4
0.4
0.3
02
(LaNet-5, SGD, MNIST)
(LeNet-5,, SCDN, KNIST)
Iteration (LNNer55, Adam, MNIST]
Figure Learning curves of ys for LeNet-5 on MNIST with SGD, SGDN and Adam.
0.8
0.8
0.7
value
0.3
0.2
500
10000
1500
2000
1000
2000
(a) Iteration (LoNoc-5, Adam, CIFAR10)
(b) Itaration (LeNett-5, Adam, SVHN]
Figure 7: Learning curves of ys for LeNet-5 on CIFARIO and SVHN.
Figure 5 corresponds to the experiment of FFNN on MNIST in Section 3.3 which is a three-level We that for different FFNN architecture, the learning behaviors of ys also show different patterns, although trained on a same dataset. Meanwhile, the standard errors for multiple trials are much smaller relative the changes of the average combination weight values.
correspond to the learning curves of ys in the experiments of LeNet-5 for MNIST image classification with SGD, SGDN and Adam, which trained on 10% of original training dataset.
15

---

Under review as a conference paper at ICLR 2021
0.8
Y1
0.8
0.7
0.7-
0.6
value
0.4
0.4
0.3
0.3
2000
4000
6000
0000
2000
4000
6000
8000
(a) iteration (CIFARIO, Reshet-18 18, SGDN)
161 Iteration (CIFAR10,, ResNet-18 Adam)
Figure 8: Learning curves of ys for ResNet-18.
In addition, Figure 7 corresponds to the learning curves of ys in the experiments of LeNet-5 for CIFARIO and SVHN image classification with Adam-CAM-HI.
As is shown in Figure for SGD-CAM-HD, SGDN-CAM-HD and Adam-CAM-HD, the equilibrium values of combination weights are different from each other. Although the initialization 7/1 0.2, 72 0.8 and the updating rate 8 0.03 are of '11 and "22 only change in small proportion when training with Adam-CAM-BR CAM-HD, while the change much more significant towards larger filter/layerwwise adaptation when SGD-CAM-HD or SGDN-CAM-BR average value of weight for layer-wise adaptation "/1 jumps from 0.2 to 0.336 in the first epoch, then drop back to 0.324 before keeping increasing till about 0.388. For Adam-CAM-HD, the average Y1 moves from 0.20 to 0.211 with about 5% change. In Figure 7 both the two subplots are models trained with Adam-CAM-HD For the updating curves in Figure 7 a). which is trained CIFARIO with CAm--C,M-DD, the combination weight for filter-wise adaptation moves from 0.20 to 0.188. Meanwhile, for the updating curves in Figure ([),, which is trained SVHN, the combination weight for filter-wit wise adaptation moves from 0.20 to 195.
be the same for the three optimizers, the values
is implemented. The numerical results show that for SGDN-CAM-
the
The similar effect also be observed from the learning eurves of 7s for ResNet- 18, which is given in Figure 8 and we only take the first 8,000 iterations. Again, we find that in training ResNet- 18 CIFARTO, the combination weights of SGD/SGIN-CAMM-
change much faster than that of Adam-CAM-HD There are several reasons for this effect: First, in the cases when rs do not move significantly, apply Adam-CAM-HD, where the main learning rate (Ic-3) only about 1%-6% of the learning rate of SGD or SGDN (1e-1). In Algorithm 1, we can see that the updating rate of ys proportion of alpha given other terms unchanged. Thus, for the same tasks, if the same value of updating rate is applied, the updating scale of ys for Adam-CAM-HDD can be much smaller than that for EGDN-CAM-HD. Second, this does not mean that if we apply much larger ō for Adam-CAM-HE CAM-HD, the combination weights will still not change significantly or the performance will not be improved. It simply means that using a small can also achieve good performance due to the goodness of initialisation points. Third, it is possible that Adam requires lower level of combination ratio adaptation for the same network architecture compared with SGD/SGDN due to the fact that Adam itself involves stronger adaptiveness.
A.5.2 OTHER EXPERIMENTAL RESULTS
In Figure Figure 3 and Figure 4 of the paper, we have shown the curves of validation accuracies to compare different adaptive optimizers in a variety learning tasks. Here we further provide the training and validation cross-entropy loss curves for corresponding methods in these tasks. Figure the full results of FFNNs, and Figure is the results LeNet-5.
16

---

Under review as a conference paper at ICLR 2021
162
Adam CAN-HD (3-evdl
153
Adam 13 levvvll
HWdden the 1001000
Adem-CAN-IID [Poeve]
Adem-CAM-1ID (Paeell
Adammt1 trp-gred
Adtmmtt (hp-gred
Hd Irp-grad Ie-1i
Adam-HD tr--gadd 1e
Adam
Adem
154
thp-grad
1.48
10
20
10
20 [Traiinng
10
20
Accurscy
Adam CAN (3-lovd))
1,510
Adam CAM (-ivvil
1564
Adam CAM 12 (2
Adam CAM HD 12 level
Adam HD good Ic
Adam HD thp Ic
H4d0e- gre 0100000000000
Adam-HD trp-grad
1,500
Adamott (tp-grad 16-7
1.52
Adam
Adam
1.495
CAM--CMHHID Diveel
Adam CAN 12 leveli
AdamotD ttp-grad Ie-71 Adam
Adam lhe good grdggolde99 Le
1.485
25 (d) Time Stap (Valiation Accurscy
10
20
20
20
20
20
(6) Time Stop (Training Loss)
(1) Time Stop (Vla ination Loos)
1,510
Hodden stre 11:000 1050]
154
1,505
godd
12
Shadgrad
Adam
Adam
1.495
Adem-CAM-IID (Panel))
AdMm-CAM-ID (3avvl)
1.50
AdmmHDD
1.405
1,480
20
Figure 9: The comparison of learning curves of FFNN on MNIST with different adaptive optimizers.
17

---

Under review as a conference paper at ICLR 2021
00.2
4. Adum-CAK CAN HD
4. Adum-CAK Sam--CALIN
+ AdammllD Datevlll
Adam-IID (Atvell
65.0
Adamil
Adamil
Adebourd
Adebourd
RAdam
RAdam
03.6
Adam-CAM-HD
0.
Adamm
Adam
Adam-H A4
Adabound
BAdam
08.0
0.0
10
10
10
(N) Ezoch (Valddaion Accuncc))
Epoch (Taining lesel
6c) Epoch (allaat Lons)
Adam-CAN--DD
1.8
Adam-CAN-HD
1.6
55
Adtm-CAk CAM-HD
1.4
1.4
-50
Adtm La
AdamalD Dievell
Adtbound
RAdim
Adam
1.0
(d) Epoch IValidaion Accuracy)
Epoch (Training Losil
Epech (Validation Losal
AdMm-CAM-ID
Adam-CAM-HD
AdamvlD (Pievel)
AdamviD (Pievel)
86
1.0
Adobourd d
Adamel4
Adthourd
Adamild
RAdam
RAdam
04
Adum
0.55
Adum
022
0.6
RAdim
0.4
80
(5)
Accuracy)
(Trsining
Epoch
Figure 10: The comparison learning curves training LeNet-5 on MNIST, CIFARIO and SVHN with different adaptive optimizers.
18


Reviews:
[{"review_id": "gRC1Bc8271V", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I really appreciate the response from the authors. Some of my original concerns have been addressed, and additional experiments help to show the benefits of CAM-HD, so I have increased my score to 5. But, after reading other reviews and responses, I still believe that this work needs to be compared to advanced learning rate adaptation methods. Most reviewers have pointed out the presentation and insufficient experiments, so it's better to submit the improved version to one of upcoming conferences.\n\n**Summary**\nThis work proposes an optimizer that adaptively determines a learning rates from different levels (global, layer-wise, parameter-wise) based on the hypergradient framework.  The proposed optimizer introduces many additional hyperparameters, and empirical evidence is not strong compared to baselines.\n\n**Detailed comments**\n\nThe proposed method adaptively adjusts learning rates at different levels (parameter-wise, layer-wise, and global). This needs to be compared to previous learning rate schedulers, but I found that only a basic scheduler has been compared. How about the performance of CAM-HD compared to previous approaches?\n* Using Statistics to Automate Stochastic Optimization, NeurIPS’19. \n* Statistical Adaptive Stochastic Gradient Methods, Arxiv’20. (This work is an extension of the neurips paper above).\n* Large Batch Training of Convolutional Networks (Arxiv’17). This work also proposes a ayer-wise Adaptive Rate Scaling.\n* SGDR: Stochastic Gradient Descent with Warm Restarts, ICLR’17.\n\nThis seems to be very sensitive to newly introduced hyperparameters (beta and gammas). This method requires extensive grid search to find a suitable beta, because the optimal beta is different across tasks (as in Table 1). \n\nThe proposed optimizer has been validated only on small-scale datasets. It’s difficult to predict how to behave in training on a large-scale dataset (such as ImageNet). In addition, I’m not quite sure that this approach works well with strong data augmentation techniques (for instance, auto augmentation). Also, I guess that the proposed scheduler will not be working well on large-batch settings (for instance, batch size >= 1K). \n\n“For the learning tasks with recommended learning rate schedules, we will apply these schedules as well.” -> In experiments, just a simple step decay lr rule has been applied. What about using cosine annealing lr schedulers? It could improve the baseline consistently.", "extra_scores": {}, "extra_text": {}}, {"review_id": "X78J1aL2CgW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality\n\n+ The authors have published code for replicability.\n+ The baseline evaluations are of reasonable quality.\n+ The proofs and theorems around convergence and complexity are high quality.\n- The results on their metrics are poor and ambiguous. They only evaluate on validation accuracy / cross entropy, which do not improve substantially.\n- Combination ratio results look inconsistent across datasets & models. \n- The analysis is limited and there are few generalizable insights to be gleaned from the paper.\n- There’s no transfer analysis or generality analysis, implying that each task will have to have its hyper-parameters tuned independently.\n- It’s not demonstrated that overparameterization is a problem. Networks are already overparameterized without issue.\n\nClarity\nTheir combination ratios plot is unclear. What should be taken away from these changes in Gamma? The analysis is unclear. Convergence analysis is unclear. There are plenty of typos. \n\n\nOriginality\n\nThe regularization methodology is somewhat novel, to my knowledge. Hypergradients are known and this is an extension to more parameters which will interact with one another. Multiple levels of tree based interacting hyperparameters is novel, to my knowledge.\n\nSignificance\n\nThis result isn’t a substantial improvement over existing methods. There aren’t clear insights to be gleaned that will generalize to other work on hypergradients (other than that it’s possible to try to regularize with a hierarchical parameter scheme).\nThis may be evidence that trying to tune every parameter simultaneously is overkill / too challenging.\n\n\n\nPros\n\nThe authors present a novel method for learning hyperparameters of a model at multiple levels which typically are not manually tuned. The potential upside of a working method here is high, as the learning rate dramatically impacts model performance. They publish code for reproducibility. The authors propose and implement a novel regularization method for their new hyperparameters as well. There are proofs backing convergence claims made by the authors. The baselines have been tuned and the presentation is honest.\n\nCons\n\nThe paper’s results are ambiguous. The paper isn’t carefully written - it’s unclear what ‘levels of adaptations’ refer to until after the introduction. It’s not clear what conclusions should be drawn from the combination ratios graph as gamma changes (Figure 1). The paper’s analysis value is limited, it’s unclear what solid & general insights about hypergradients can be taken away from it.\n\n\n\nOther feedback:\n\nTypo in the abstract: combination -> combinations\nTypo in the first line of introduction: is gradient -> is the gradient\nTypo in the second sentence of the second paragraph of the introduction: function -> functions\nTypo in the third paragraph of the introduction: horizon -> horizons\nTypo in first paragraph of Experiments: lowercase p in ‘The Proposed -> The proposed’\nLikely better not to shorten ‘Feed Forward Neural Network’ to FFNN in the section heading (though this is fine in the body of the text).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QzF89c7o_sD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Setting appropriate learning rate for network optimization is an important task in deep learning applications. This paper investigates the setting of learning rates for network parameters in different levels, e.g., individual parameter, each layer and global levels. By setting the constraints on the learning rates at multiple scales, the paper derived a hierarchical learning rate setting approach, which is the combination of adaptive learning rates at different levels. \n\nOverall, this proposed learning rate setting method seems to be interesting, however, I have some concerns on the setting of the hyper-parameters of the proposed method. \n\n1. The paper proposed to set the constraints at different levels by Eqn. (7). I have concern on how to automatically set the hyper-parameters of combination weights.  The details on setting / learning these hyper-parameters should be clarified in a more clear way. \n\n2. What are the possible reasons that the proposed hierarchical learning rate can improve the baseline optimizer Adam and SGD?\n\n3. The proposed learning rates setting method can be combined with any gradient-based network optimizer. More combinations and corresponding results and comparisons should be given to show how much the proposed technique can improve the different baseline gradient-based optimization methods.  \n\n4. The comparisons with more network optimizers should be given in the experiments. \n\n\n----\nPost rebuttal comments：\nThanks for the responses from the authors. These responses partially solved my questions. I think that the initialization of hyper-parameters of combination weights seem to be heuristic, and it is unclear on the effects/robustness of its initialization on the optimization performance. My questions on more comparisons and more combinations with other optimizers are not well answered. \n\nI also read the other reviews and responses, I agree with other reviewers on the concerns of experiments, justifications, robustness, etc. Considering that it needs some additional works to solve all these concerns, I suggest the authors to improve the paper and submit it to one following conference. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ico3elPoDqc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary of the paper\n\nThe paper investigates the setting of hyper-gradient descent in the context of adapting learning rates at different levels in a neural network (e.g. per layer). The paper derives an equivalence between regularization of such learning rates and a weighted combination of non-regularized adapted learning rates. Experiments on small datasets and small network architectures are performed and show an improvement over baselines including Adam and SGD.\n\n### Main review\n\nAfter the introduction (which is well-written) I found the paper relatively hard to follow. Sections 2.2 and 2.3 are long and introduce a large variety of options that *could* be used without giving the reader a clear perspective of where this discussion is going. In the experiments, a lot of different combinations of small networks and datasets are used, and while the learning curves look promising, it is hard to assess how robustly the improvements may transfer to other settings, because e.g. the parameters $\\gamma$ are initialized differently in the experiments. The experiment of Fig.1 shows that the variation in the measurements can be very high with respect to $\\gamma$, such that the effect of the algorithm as such is a bit hard to assess. (And e.g. Fig.7 seems to indicate that the $\\gamma$s sometimes do not move from their initial values.) This makes me wonder how much of an improvement it is to go from optimizing hyper-parameters of e.g. SGDN or Adam to optimizing \"hyper-hyper-parameters\". \n\nWhen reading the paper I also wondered if any relation between the behavior of the adapted learning rates and other quantities in training (e.g. momentum per parameter) can be established. Does the adaptation counteract or increase momentum effects? Or does this depend on other quantities? Any such connection could provide insight and/or help with intuitions about the adaptation behavior. I did not see any attempt into such a direction in the paper.\n\nThe relation between regularization and weighting of unregularized values (Theorem 1) seems interesting.\n\nAs the experiments go as far as ResNet-34 on CIFAR10, it would be interesting if a comparison to existing results from the literature could be made. E.g. the accuracies in Fig.4 seem to go into the area of 92-93%, which seems appropriate for this combination, but it is hard to tell from the graph alone. (It would be even more impressive if a clear statement could be made that shows that the final accuracy is better than for the optimizers used in a work that is (close to) state of the art (under given constraint like e.g. architecture=ResNet-18).\n\nA (brief) discussion of the additional computational cost (in terms of memory and operation count) in the main body text would be useful. \n\n* Pros: The paper takes hyper-gradient to many new multi-levels.\n* Cons: The paper is somewhat hard to follow, and it is not immediately clear how transferable the experimental results are to other settings.\n\n### Minor details and comments\n* The font in the figures is very small in some cases, e.g. Figure 5 is very hard to read even when zooming in considerably on the screen.\n* Typos/nits:\n  * p.3 \"leave nodes\" -> leaf\n  * p.4 \"use each of them to updated the learning rate\" -> update\n  * p.6 \"tunning\"\n  * p.8 \"in later stage\"; \"our methods achieves\" -> method\n  * in several places: CMA -> CAM\n  * Lowercasing in reference titles, e.g. \"Rmsprop\", \"smd gain\"", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an optimization framework that automatically adapts the learning rates at different levels of a neural network  based on hypergradient descent.  The AC and reviewers all found the approach interesting and promising and appreciate the author feedback.

We strongly encourage the authors to incorporate the points and additional results provided in their response to the reviewers.

Additionally, some concerns remain to be addressed regarding initialization of hyper-parameters of combination weights. Specifically it would be important to further investigate the impact of such initialization on the optimization performance. Furthermore, additional experiments with other network optimizers would be valuable as pointed out in the reviews.
Decision: Reject