Differentiable Self-Adaptive Learning Rate
Authors: Bozhou Chen
Hongzhi Wang
Chenmin Ba
Venue: ICLR 2022 Submitted (2022)
Keywords: self-adaptive learning rate

Abstract: Adaptive learning rate has been studied for a long time. In the training session of neural networks, learning rate controls update stride and direction in a multi-dimensional space. A large learning rate may cause failure to converge, while a small learning rate will make the convergence too slow.
Even though some optimizers make learning rate adaptive to the training, e.g., using first-order and second-order momentum to adapt learning rate, their network's parameters are still unstable during training and converges too slowly in many occasions.
To solve this problem, we propose a novel optimizer which makes learning rate differentiable with the goal of minimizing loss function and thereby realize an optimizer with truly self-adaptive learning rate. We conducted extensive experiments on multiple network models compared with various benchmark optimizers. It is shown that our optimizer achieves fast and high qualified convergence in extremely short epochs, which is far more faster than those state-of-art optimizers.

Under review as a conference paper at ICLR 2022
DIFFERENTIABLE SELF-ADAPTIVE LEARNING RATE
Anonymous authors
Paper under double-blind review
ABSTRACT
Adaptive learning rate has been studied for long time. In the training session neural networks, learning rate controls update stride and direction in a multi- dimensional space. A large learning rate may cause failure to converge, while a small learning rate will make the convergence too slow. Even though some optimizers make learning rate adaptive to the training, e.g., using first-order and second-order momentum to adapt learning rate, their network's parameters are still unstable during training and converges too slowly in many occasions. To solve this problem, we propose a novel optimizer which makes learning rate differentiable with the goal of minimizing loss function and thereby realize an optimizer with truly self self-adaptive learning rate. We conducted extensive experiments on multi- ple network models compared with various benchmark optimizers. It is shown that our optimizer achieves fast and high qualified convergence in extremely short epochs, which far more faster than those state-of-art optimizers.
INTRODUCTION
Learning rate is one of the most important hyper-parameters in artificial neural networks, and is the core of an optimized] In training session of neural network, optimizer takes the duty to update network parameters, and directly affects the training speed and final effect.
The early (GD(ttochastic gradient descent) method (Lemaréchal 2012 1944) uses a fixed learning rate in the training session. Usually, we choose a large Tearning rale to accelerate training. However, this may cause network's parameter to fluctuate around the extreme point and even cannot converge under SGD (Ruder 2016), which a very unstable optimization. Momentum (Sutskever et al., (013b) takes the exponential moving average of historical gradient as a stride. When parameters get into an oscillation, positive and negative gradients will neutralize with each other, thereby reducing the update stride and getting rid of the fluctuation.
Courant et al. 1994 Curry
However, learning rate under Momentum cannot adapt to the training session. Actually, learning rate needs be larger the beginning of training session and smaller at the later stages. AdaGrad (Duchi et al. 2011) multiply learning with the reciprocal of two-norm of historical gradients to have learning rate reduce along with training session. While AdaGrad is interfered lot by historical gradient, i.e., learning rate in AdaGrad keeps decreasing along with training resulting that learning rate is too small in the late stage of training. RMSProp (Graves) 2013) replaces two-norm AdaGrad with exponential moving average of historical gradients to reduce the interference. Adam (Kingma & Ba, 2014) combines Momentum and RMSProp, so that can solve fluctuations and make learning rate adaptive in the same time. However, Adam still has many limitations. Firstly, Adam usually cannot converge well in the late stages of the training. Secondly, after an oscillation, learning rate is usually very small resulting that it converges very slowly in next several epochs. Thirdly, the update stride limited by the learning rate. However, learning rate cannot grow quickly when a large stride is needed. In word, Adam is not sensitive enough.
Therefore, none of the current optimizers are stable and sensitive enough to ensure a fast and high qualified convergence. To achieve these two goals, propose novel optimizer differnnnee self-adaptive learning rate), which makes learning rate adaptive quickly and accurately. The learning rate DSA is differentiable with the goal of minimizing loss function. DSA is able to get rid of oscillation in only one or two epochs, since the learning rate in DSA can increases or decreases very sensitively and significantly with the help of learning rate's gradient.
https::/eet wik weriiedia..roo http:///iwiweeeeannggaaaaaaa

---

Under review as a conference paper at ICLR 2022
Based above discussions and our researches, contributions of DSA are summarized follows.
We are the first to make learning rate in neural network differentiable with the goal minimizing loss function such that learning rate adapt with clear instruction.
DSA can optimize a wide range of network models and can solve two classical and common problems in machine learning, which are grad loss and hard convergence in the later of training- We conduct extensive experiments on multiple neural network models compared with various state-of-art optimizers. Experimental results have demonstrated the absolute advantage of DSA in speed, stability and performance.
In the remaining of this paper, Dection@ the proposed method. Experiments are conducted in Section 3 We overview related work in Section[] Section draws the conclusions.
2
METHOD
In this section, we propose our approach, DSAdiifferentiable self-adaptive learning rate). In Sec- tion 2.1] we declare the motivation of DSA. The algorithm of DSA will be given out in Section 2.2
in detail. Then we will explain how to apply DSA to training process in Section 2.3
2.1 MOTIVATION
In this section, we discuss the motivation of the proposed approach in detail.
The early optimizer SGD updates parameters as formula I and has three major defects. Firstly, the fixed learning rate in SGD cannot adapt along with the training session. Therefore, it sensitive enough and usually stops convergence in the late stage of training. Secondly, it is slow. ge is always a small number, so the stride always small too. result, the network converges slowly. Thirdly, it is unstable, which we call it oscillation as shown in Figure 6. A slightly larger learning rate may cause parameter crosses extreme point repeatedly.
0 = - 17
(1)
Many optimizers have been proposed to solve these problems. Among them, Adam is one of the most stable and efficient as formula 2 the formula, network's parameter, and gt is its gra- dient in (-th step. represents the first-order momentum of gradient, and v, is the second-order momentum. 17 is the learning rate, and e an infinitesimal to avoid being divided by zero
m, = B1 * M + (1 - -31) gt
= B2 * Ut + (1 -2) gt
m
0 = 0 -
Vez +
Adam substitutes m, for Sr to make the parameter of neural network more stable along with the training session. i22 amplifies the update stride reasonably and makes learning rate adaptive in each epoch. However, unstable occasion still exist commonly especially when 0 > 1), 0 will get into long-term oscillation as F Additionally, slow convergence always exists after a big oscillation of parameter, because m, always steps around learning rate 17 in the late stage of training and results in failure for further optimization. In a word, current optimizers are still not sensitive and stable enough.
that time while o is still large. Apart from that, updating stride
To make the learning rate significantly sensitive enough and supply more stable optimization, we propose DSA(differentiable self-adaptive learning rate). In DSA, we design a loss func- tion L(n) with learning rate 1) as its independent variable. 17 differentiable with the goal of min T(n).Therefore we can use the gradient of learning rate to update the learning rate to make it more sensitive. We will give the details of DSA in section 2.2
2.2 DSA ALGORITHM
In this section, we firstly introduce the basic idea of DSA and then perform three progressive opti- mizations to basic DSA.
2

---

Under review as a conference paper at ICLR 2022
2.2.1 BASIC ALGORITHM
Our target is to make learning rate differentiable to get more sensitive and stable optimizer. To achieve this, we design two-step training and we will talk about it in the following.
SGD
min L(e - 17 * g()))
0.
0
min L()) 8
0 7oo
go
0.
171
go
Figure 1: A decomposition diagram of the optimization process within Epoch
In DSA, the training of each epoch consists of steps shown in Figure The first step is performed with the goal of ming L(0). The second step is performed with the goal of min, [(n), L(n) & L(O 0 g(0). Actually, Ø is function with learning rate 7) as its only variable. Therefore, have the second step rather than assigning 0g - vi * to 0j immediately to take further optimization.
g(0)). After the first step, we get temporary variable 0, 0 &
The second step the core of DSA. Through this step, we achieve rate is differentiable, because 7 is variable of L(n). It means that learning rate be adaptive to minimize the loss function directly. Therefore, the learning rate will be extremely sensitive. Secondly, the loss function be optimized further. After the second step, we have (000 *g1) L(o, - %/ * 9o). Actually, L(f0- Y/o 9o) is the result of SGD, so further optimization achieved.
goals. Firstly, the learning
When using gradient descent solve the second step, we will get learning rate's gradient (ea(--g() g(O)) through derivation in Appendix A Then iterative equations of DSA can be summarized as formula 3 g(0) represents gradient of with the goal of ming I.(0). / controls the update stride of learning rate 7). In the formula, Ū = 0 - 1] g(0) corresponds to the goal of ming ((0) and
8 * Eo.((-g(0) g(0)) corresponds to the goal of min, I(e - * g())- And finally, parameter 0 is updated according to t = 0 * gf)).
0 = 0 g())
1 = 1 - B Σ
g(0) ((()))
0,8
0 = 0 * (0)
In basic DSA, uniform learning rate for each network's parameter. Actually, these param- eters usually have different requirements for learning rate, because they often need to be updated at different speeds. Therefore, make learning parameter specific in the following Section 2.2.2
2.2.2 PARAMETER SPECIFIC LEARNING RATE
In this section, we solve parameter's individual needs for learning rate. To achieve it, we assign specific learning rate for each parameter. In the following, we first declare the change to iterative equations of DSA, then visualize and explain the process of adapting learning rate in DSA in detail.
If learning rate is specific for each network's parameter, then the gradient of parameter 0 should be changed from Σ,,g( (000(-((0) (0) g())) to - g()). Accordingly, iterative equations of DSA should be changed to formula 4 According to formula if g(θ) x g(0,-1) > 0, the learning rate 7) should increase. Otherwise, should decrease.

---

Under review as a conference paper at ICLR 2022
S
S
DSA
s + 9 <0
SGD
7 171
( 0(0)
1/
R
DSA
SGD
T
Learning rate increases
Learning rate decreases
Figure 2: Visualization of learning rate adaptation. Parameter arrives at point R from point S after last iteration and will arrive at point P if we use SGD. S-R-P is track of SGD and S-R-T track of DSA. g(0) and g(0) corresponds to the gradient of parameter at point P and R, respectively.
0 = 0, 7/- 9(0,-1)
T/: 7/-1 1 3 (- (-)(0) g(0r
(4)
= 7/ + 3 (g(0) (((-1))
0, 0,-1 7te g(
Now use Figure 2 to explain how the learning rate is adaptive in DSA based on formula 4 In the left example, P still haven't 't reached the extreme point, so learning rate should increase at R. In this case, formula also indicates that learning rate should increase because g(0) * g(0) > 0. In the right example, P has crossed the extreme point, learning rate should decrease R to avoid that.
In this case, formula 4 also indicates that learning rate should decrease because g(0) g()) 0. From both of the two examples, we can conclude that learning rate in DSA is adaptive reasonably.
Even though the learning rate is now parameter specific, DSA still not deal with the grad loss because Ye g(f:-1) may be very small sometimes. And we will solve in Section 2.2.3
2.2.3 EXPONENTIAL LEARNING RATE
In this section, we try to solve grad loss in DSA. We first state the change for DSA to solve the grad loss and then explain why is reasonable.
Totally, we make two changes to DSA. Firstly, we use exponential the learning rate 2 instead of r). (()) Secondly, we use [o(0]}=> to replace g0). is infinitesimal and 9(0)
{1, 0, -1]. Then the [r00]
iterative equation changed to formula 5
g(0,1 1)
0 = 0z-1 2
1g(0,-1)
-g(0) *g(0_1)
1) = 11-1 - 3+
(5)
g(0) g +
g(0c-1)
0, = 8--1 - t
19(0,-1) +e
To solve grad loss, the key is the second change. After this change, the update stride is absolutely controlled by the learning rate and has nothing to do with the value of gradient. Then the grad loss
4

---

Under review as a conference paper at ICLR 2022
can be solved. Additionally, we use exponential learning rate to avoid negative value of learning rate. Otherwise parameter will move to reverse.
While in practice, the occasions of crossing the extreme point still exist under DSA. The attribution of these occasions can be divided into two types. One is that learning rate increases so fast that parameter crosses the extreme point. The other is that the learning rate decreases slowly that DSA is not able to avoid crossing the extreme point. It affects the stability of DSA, and we will solve this problem in Section 2.2.4
2.2.4
2.2.4 INCREASE SLOWLY AND DECREASE FAST
To reduce occasions where the parameter crosses extreme point, we use a small stride 32 when the learning rate should increase, and large stride $1 in the opposite. The reason is follows. A small stride can solve the occasion that the learning rate increases too fast and parameter crosses the extreme point. A large stride solve the occasion that the learning rate decreases too slowly so that DSA is not able to avoid crossing the extreme point.
Thus the update equation of learning rate changed formula 6 According formula 6 when learning rate should increase, ( g(,=1) is positive, then A, equals the small stride 82. When learning rate should decrease, (0) g( (0-- is negative, then A, equals the large stride 31.
81 + B2 g(0) g(O, +E
g(0) * g(00-1)
B2 31 2
The = 1h-1
B1 > 82
(6)
2
Through the above three improvement, the learning rate in DSA gets very sensitive, and DSA can finally take extremely quick and stable optimization. Then we will illustrate the method of applying DSA to training process in Section 2.3
2.3 TRAINING METHOD
In this section, we try to apply DSA to the training sessions to both small and large datasets. is easy apply DSA to solve small datasets with updating network after each epoch, which is called batch training. While for large datasets, it is necessary to divide th dataset into mini-batches and updating parameters after each mini-batch, which is called mini-batch training.
In the following, we first explain how to apply DSA to solve small datasets with batch training and then draw the method of dealing with large datasets.
We summarize the process of applying DSA to batch training in Algorithm 1 of Appendix simple application of formula 5 and formula
B which
Now we explain how to solve large datasets with DSA. As described above, the dataset should be di- vided into multiple mini-batches. We can still update network after reading all the mini-batches just like a batch training. However, it will be very slow to converge and fail to capture local features of dataset.
update parameters after each mini-batch, i.e. mini-batch training, the neural network can capture local features of dataset well and take faster optimization because of more backward propagation than batch training. However, the mini-batch training always fails capture global features of dataset due to the excessive attention to the latest mini-batch in each optimization.
more effective training method is a mix of them. We first train model for several epochs with mini-batch training to capture the local features. Then we use batch training to train the model continually. In this way, both local features of datasets and global features will be captured. For the second stage, the optimizer should be able to have neural network converge fast in as few epochs as possible. Otherwise, the local features will faded out along with batch training. There is no doubt DSA is the best option for the second stage. As for the first stage, choose Adamax optimizer. Adamax shows stable performance some cases as discussed Kingma & Ba (2014), which can be seen in experimental result, too.
5

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENT
To verify the performance of the proposed approaches, we conduct extensive experiments. In this section, we first introduce the basic settings necessary for experiments in Section 3.1] Then we will show the results and take analyses in Section 3.2 Nextly, two case studies will be conducted in Section 3.3 Finally, we will take sensitivity analysis for DSA and put the result and analyses in Appendix H
3.1 EXPERIMENT SETTING
1980 Lecun & Bottou 1998) and MLP (Gardner & Dorling, 1998). On FMP, we choose the well known and widely used validation datasets MNIST (LeCun & Cortes 2010). SVHN (svh 2011) 1), CIFARIO and CIFARI00 (Krizhevsky et al. 2009). On MLP, we choose iris, wine, car and agaricus-lepiota as validation data set, since they are distinguishable to different optimizers. The basic information of these datasets is shown in Appendix
Dataset We applied DSA to FMP GGaham 2014), DNN Kunihiko & Fukushima
Baseline According to the different optimizer's features, we choose the following optimizers as the competitors, i.e., SGD (Leaaréchal] 2012). Momentum (Sutskever et al. 2013b), AdaGrad (Duchi
jet al. 2011 AdaDelta (Zeiler 2012), RMSPrrp (Graves 2013), Adam (Kinmma & Ba 2014), Adamax (Kingma & Ba 2014) and AdamW (Loshchilov & Hutter, 2017). These optimizers are
designed to solve different problems in the early optimizer. Among them, SGD is the early optimizer.
Momentum is a speed-up optimizer. Others are learning rate adaptive optimizers.
Metrics We recorded the following metries of the classifier to measure the effect of different opti- mizers from different perspectives, i.c., accuracy, Fl-score, recall and precision of a trained neural network (Pedregosa et al. 2011). Additionally, the loss sequence along with training session is also recorded to demonstrate the convergence effect.
Implementations FMP is designed with reference to Graham et al.
2018): Graham & van der
Maaten (2017) as the structure of FMP is visualized in Appendix D DNN is designed with refer- ence to Kunihiko & Fukushima (1980): Lecun & Bottou (1998); Behnke & Sven (003) and we set channel size as a small value to simplify the model to make more sensitive to be used to dis- tinguish different optimizers' convergence speed. MLP sequences 5 fully connected layers. The output feature dimensions each layer are: 32, 64, 256 and 128. The output of the first and third layers are processed using sigmoid activation. The output of the second and the fourth layers are processed by prelu activation (He et al., 2015). The features output by the neural network are pro- cessed by log_softtn softmax activation and cross-entropy loss function. For FMP, the batchsize of MNIST, SVHN, CIFARIO, CIFARIO0 is 64, 64, 128, 256 respectively, and the training epochs(Ti TT2) is set as (50, (75,15), (75,15), (50,15) respectively. For DNN, the batchsize of MNIST, SVHN, CI- FAR10, CIFAR 100 is 64, 64, 64, 32 respectively, and the training epochs(Ti, T2) is set as (15,15). For MLP. epochs set as 100. For DSA, 770 is set as -12, B1 is set as 0.6, and B2 is set as 0.3. For SGD, the learning rate is set as 0.01. For Momentum, B is set as 0.9, and the learning rate is set as 0.01. The remaining apper-parameters of optimizers use the built-in values (Paszke al. 2019) Our experiments are conducted on GTX 3090 GPU for FMP and GTX 3060Ti GPU for the others.
3.2 EXPERIMENTAL RESULTS
In this section, we show the results of three groups of experiments. Firstly, we apply DSA to the large convolution network FMP to show that DSA is able to optimize complex model in short epochs. Secondly, we apply DSA to a lightweight and sensitive convolution network DNN and compare with other benchmark optimizers to show the superiority of DSA in speed and performance. Thirdly, we apply DSA to MLP to show the performance of DSA in batch training compared with baselines and the ability to process feature datasets.
To show that DSA is suitable for large datasets and complex neural network, we firstly apply DSA on FMP which is to process four image datasets and record the key metrics. We have not taken any measures that can additionally enhance the effect of the model, such as data expansion and dropout, to clearly compare the effect of optimizers. We set the training epoch 90 at most and show
https://archive: is. uci sedu/m) /atasets

---

Under review as a conference paper at ICLR 2022
the results in Table 1 after removing optimizers with poor performance. From the results, DSA achieves an accuracy of 99.73% when FMP running on MNIST and an improvement about percentage CIFARI00 compared to Adamax. Obviously, DSA can deal with large neural network and datasets in extremely short epochs and still gain extraordinary performance.
Table 1: The results FMP. Eight different optimizers tried. Only Adamax and Adam competent for this task in short epochs. We have removed useless control group from this table. ACCU, FI, RC, PCS denotes aceuracy, fl-score, recall and precision, respectively.
MNIST
SVHN
ACCU
F1
RC
PCS
ACCU
Fl
RC
PCS
Adamax DSA Adam
99.52
99.52
99.52
99.52
95.42
95.11
95.29
94.98
99.73
99.73
99.73
99.73
96.21
95.95
95.89
96.02
70.21
69.23
72.56
69.69
CIFARIO
CIFAR100
ACCU
F1
RC
PCS
ACCU
F1
RC
PCS
Adamax DSA
86.91
86.99
87.17
86.91
51.13
51.09
52.89
51.13
88.84
88.84
88.86
88.84
59,50
59.63
60.02
59.50
To distinguish these optimizers clearly, we conduct experiments on lightweight convolution neural network DNN referencing Kunihiko & Fukushimal (1980); Lecun & Bottou (198); [Behnke & Sven (2003), where more optimizers can converge to a reasonable result in short epochs. Key part of the results is shown in Table 2 and the details are placed in Appendix E Further,
visualize the loss trend along with epoch as Figure X of Appendix E It is easy to see that DSA converges with an extremely fast speed and touches the limit of DNN far more quickly than any other optimizers.
Table 2: Result on DNN. The four metrics are accuracy. fl-score, recall and precision. In the right tabular, we stack the result of SVHN and CIFAR100 together. The first three lines are for SVHN and the others is for CIFARI00. MNIST
SVHN & CIFAR10
SGD
98.44
98.43
98.43
98.43
AdaDelta
60.42
56.39
57.54
56.69
Monmentum
95.69
95.65
95.75
95.63
Adamax
86.52
85.12
85.35
84.97
AdaDelta
97.35
97.33
97.37
97.33
DSA
87.35
86.18
86.25
86.16
AdaGrad
98.48
98.46
98.47
98.46
SGD
41.70
40.65
42.50
41.70
Adam
97.68
97.65
97.68
97.66
Monmentum
43.89
42.21
43.82
43.89
AdamW
98.42
98.41
98.40
98.42
AdaGrad
53.48
53.50
53.69
53.48
Adamax
98.80
98.79
98.79
98.79
Adamax
60.23
59.52
60.23
98.93
98.92
98.92
98.92
DSA
62.15
62.08
62.08
62.15
IRIS
WINE
CAR
AGARICUS
109:
12
12-1
337
322
Figure 3: Loss trend on MLP.
We then conduct experiments on four small feature datasets with result in Table Appendix to show the performance of DSA in batch training and processing feature datasets. Among the four datasets, WINE and CAR are the most discriminative datasets for DSA and other optimizers, where DSA obtains an increase of about 10 percentage in accuracy and even double performance in fll score and recall. Loss trend is visualized in Figure and DSA converges 50 epochs, 60 epochs, 30 epochs WINE, CAR, AGAICUS respectively, which is impossible for other optimizers. Further, we set epoch as 1,000 to see the performance of other optimizers as shown Figure [ We can that other optimizers stop convergence near 0.1 IRIS, where only DSA converge to 0. On

---

Under review as a conference paper at ICLR 2022
1 of other optimizer's epochs to converge the same level, which AGARICUS, DSA only 30
means DSA is 30 times faster than other optimizers.
IRIS
WINE
CAR
AGARICUS
187
09
27
12
Figure 4: Loss trend on MLP with 1000 epochs.
3.3
CASE STUDY
In this section, conduct studies. The first is simple regression problem calculate the sum of four numbers. The second is minimization problem.
Loss Trend
Track w1
10-2.
ADUP ADADOON
10-1
10-2
10
PONDITUM
10
400
Track W/2
Track
Track W4
109
105
10
10
10-2
10
200
400
600
800
1600
200
400
600
8Bc
1000
700
420
600
800
1000
Figure 5: Result on sum of four numbers.
As the first case, we solve a simple regression problem named the sum of four numbers with target of min ||w T y|1, where
[wh W2, W3, w4] R.A. Obviously, the best value of each w; 1. We set this case to show the concrete optimization process of each optimizer and compare the convergence speed and stability among them from more intuitive perspective. As shown in Figure [5] Momentum and DSA are the most suitable optimizers for this task. While Momentum converges quickly but unstable according the loss trend. Therefore. DSA completes optimization more quickly. The reason can be seen in the track of w1, where Momentum has a large oscillation when close to 1 and take many epochs calm down as shown in Figure [5]ckkck w2). While DSA solves this occasion with a quick adaptive of learning rate and quickly converge to 1.
The second case is minimization of min adaptive ability of DSA. On the one hand, the track of (W1, w2) will always have oscillation. The smaller oscillation is, the stronger adaptive ability is. On the other hand, the change of update stride in each iteration also reflects the optimizer's adaptive ability. Our study consists of two groups, one for SGD and Momentum and the other for RMSProp. AdaGrad, Adam, Adamax and Adam W. the first line of Figure 66, SGD has violent oscillation along wy before calming down and moving to target slowly. Momentum also has violent oscillation along both w, and W2, but it moves faster than SGD. Far more better than SGD and Momentum, DSA is able move the target along straight
wi + b w². We set this case to show the strong
8

---

Under review as a conference paper at ICLR 2022
line without any visible oscillation. Additionally, the stride continues increasing in the beginning and decreases quickly in the end. In the second line of Figure deetall in Appendix G). we select hard initial position and observe the tracks. Other adaptive optimizers all experienced big wave before calm down, especially for RMSProp and AdaGrad. Adam moves to the target more smoothly, but the speed movement to the target within short iterations.
slow. DSA solves all these questions, with efficiently calming down and quick
DSA
SGD
Momentum
200
180
67-
-011
-025
-0.55
0.55
-1.00
1149
-1.
-05
-06
-43
60
-00
-27
64
DSA
Adam
Adamax
6964
3600
0434
0C20
3001
0524
36.44
361
610
-0487
SEM
D0X
-046
-045
-2.4
-6.1
Figure 6: Track visualization. Track starts from (-0.06,0.001,, a 1,b 1000.
RELATED WORK
Stochastic gradient descent (Lemaréchal 2012 is the most commonly used optimizer. Based a fixed learning rate, the gradient itself determines the update stride, which has serious efficiency problems and is greatly affected by grad loss. Momentum (Sutskever al. 2013b) speeds up the training process by adding momentum information. AdaGrad (Duchi et al. (011)) addss adds two-norm of historical gradient on the basis of SGD that the learning rate can be adaptive. That is, it will keep decreasing in the whole training session. RMSProp (Graves) 2013) introduces second-order momen- tum solve the problem that AdaGrad is greatly affected by historical gradients. AdaDelta is also an adaptive optimizer. Although it does not have the concept of learning rate, it is analogous to New- ton's 's method find a more accurate stride for each update. Adam (Kingma Ba/ 2014) combines Momentum and RMSProp, owns both of their advantages. Based on Adam, Adam (Loshchilov) Hutter 2017) adds a regular term to achieve better convergence effect. Adamax extends the two norm the infinite norm to obtain more stable and concise results. SparseAdam (Kingma & Ba 2014) is designed to deal with sparse tensors. L-BFGS algorithm (Schmidi) (2005) is more suitable for large-scale numerical calculations, and has the characteristics of fast convergence of Newton's method. ASGD (Sutskever et al.) (013a) is asynchronous stochastic gradient descent, which is mostly used in large data parallel systems.
CONCLUSION & FUTURE WORK
In this paper, we propose the optimizer with truly self-adaptive learning rate for fast and stable con- vergence. Compareed with existing optimizers, DSA has stronger adaptive capabilities and com- petent for variety of machine learning tasks. While this requires reasonable initial value of the learning rate and stride size 31 and B2. In addition, in the later stage of training, the learning rate is still in an active state, which means pointless adaptation. Therefore, how determine reasonable initial value or eliminate the negative influence of the initial value and make the learning rate of DSA converge stably will be the main issue to be studied next.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Reading digits in natural images with unsupervised feature learning. nips workshop on deep learning & unsupervised feature learning, 2011. 1.
Behnke and Sven. Hierarchical neural networks for image interpretation. Springer, 2003.
Richard Courant et al. Variational methods for the solution of problems of equilibrium and vibra- tions. Lecture notes in pure and applied mathematics, pp. 1-1, 1994.
Haskell Curry. The method of steepest descent for non-linear minimization problems. Quarterly of Applied Mathemutics, 2(3):258-28) 261, 1944.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Matt W Gardner and SR Dorling, Artificial neural networks (the multilayer perceptron) review of applications in the atmospheric sciences. Atmospheric environment, 32(14-15):2227-6336
1998.
Benjamin Graham. Fractional max-pooling, arXiv preprint arXiv:1412.607/ 2014.
Benjamin Graham and Laurens der Maaten. Submanifold sparse convolutional networks. arXiv preprint 0tt1://00601377. 2017.
Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. CVPR, 2018
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint wrXiv:1308.0850 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026,1034, 2015.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kunihiko and Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cyberneties, 1980.
Y. Lecun and L. Bottou. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2778-2324, 1998.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Claude Lemaréchal. Cauchy and the gradient method. Doc Math Extra, 251(254):10, 2012.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1r100:171..01101, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Gareett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Coumapeaa,, M. Brucher, M. Perrot. and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2833 2011.
10

---

Under review as a conference paper at ICLR 2022
Sebastian Ruder. An overview of gradient descent optimization algorithms. CoRR, abs/1609.04747, 2016. htl.://ppe//aaa..ooooo////////3/
Mark Schmidt. minfunc: unconstrained differentiable multivariate optiiizatinn in matlab. Software available http://www. cs. ubc. caf" schmdmm/Sfffwarrmmiiffnne him, 2005.
llya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning, pp- 1149-1147. PMLR, 2013a.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initial- ization and momentum in deep learning. In International conference on machine learming, pp- 1139-1144 1147. PMLR, 2013b.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.

---

Under review as a conference paper at ICLR 2022
A CALCULATE THE GRADIENT OF LEARNING RATE
According to the description in Section 2.2.1 we have Σ(n) - I(0- - " g()) and Ø x ? is learning rate and g()) is gradient of / with the goal of ming L(0). Obviously, rj's gradient
aL(()) and we can calculate it with formula 7
dry
aL(n)
dL(6
17 * (())
dn
dn
aT.(0)
dn
OL(0)
00
00
On
8
OL(6)
06
* g
... 0 17 g())
00
dn
8,8
0(0 11 g(0))
(())
On
0,0
Σ
(
g()))
0,0
B PSEUDO CODE OF DSA WHEN APPLIED TO BATCH TRAINING
Algorithm In Line 2-3, we take once forward and backward propagation to calculate the gradient for 8 which is used for calculating the learning rate's gradient. In Line 4-5, compute the learning rate's gradient and update the learning rate according to its gradient. In Line 6, we finally update network's parameter based updated the learning rate and gradient calculated in Line 2.
Input: neural network model f()n), data set X, y. initial learning rate yoo, learning rate update step size 31 and B2, the number of iterations T, infinitesimal Output: trained neural network model f(Or)
for
T do ÿ - f(X (00.)1))((--1)) +- Vo.-
2:
Calculate gradients of 0, b Calculate according to formula
3:
+ 0:-1 t
9(9,
10(
4:
9 + f(X 0), (()
V
b Calculate gradients
5: Th < 7/-11 8
0101-g(0.-11 (((0)_g(0--
8-88
Update learning rate according to
formula
6:
0g x- 0,-1 2 * g
b Update network's parameters according to formula
7: end for
12

---

Under review as a conference paper at ICLR 2022
C
META INFORMATION OF DATASETS
Table 3: Dataset information
Dataset
#Train/ #Test
#Attributes
#Class
MNIST
60,000 /10,000
1*28*28
10
SVHN
73,257 126,032
3*32#32
10
CIFARIO
50,000 /10,000
3*32#32
10
CIFARI00
50,000 /10,000
3*32*32
100
IRIS
120/ 30
WINE
142/ 36
13
CAR
1,382/
AGARICUS
6,499 1,625
116
D STRUCTURE OF FMP
We visualize the structure of FMP in Figure 7 supposing the input is image of 28 28. FMP composed of convolution block and linear block. Each convolution block ends up with fractional maxpool. Each output of convolution layer is processed by prelu activation.
Prantimn
come block, (64@28x28) conalatiie
cono block, (128@ 28x20)
linear block, (1024x512 512ccals))s
block,
block,
(256028x28)
(256@28x28)
input
conv blocks (512@28x28)
conv block, (512@28x28)
Figure 7: Structure of FMP.
13

---

Under review as a conference paper at ICLR 2022
E
RESULTS OF LARGE DATASETS IN DETAIL
Table 4: Dnn MNIST MNIST FI-SCORE
result detail
ACCU
RECALL
PRECISION
SGD
98.44
98.43
98.43
98.43
Monmentum
95.69
95.65
95.75
95.63
RMSProp
11.35
2.04
1.14
10
Adadelta
97.35
97.33
97.37
97.33
Adagrad
98.48
98.46
98.47
98.46
ADAM
97.68
97.65
97.68
97.66
ADAMW
98.42
98.41
98.4
98.42
ADAMax
99.02
99.02
99.02
99.02
DSA
98.93
98.92
98.92
98.92
Table 5: Dnn SVHN result detail
SVHN
ACCU
FI-SCORE
RECALL
PRECISION
SGD
19.59
3.28
1.96
10
Monmentum
19.59
3.28
1.96
10
RMSProp
19.59
3.28
1.96
10
Adadelta
60.42
56.39
57.54
56.69
Adagrad
19.59
3.28
1.96
10
ADAM
19.59
3.28
1.96
10
ADAMW
19.59
3.28
1.96
10
ADAMax
86.52
85.12
85.35
84.97
DSA
87.35
86.18
86.25
86.16
Table 6: Dnn cifar 10 result detail
CIFAR10
ACCU
FI-SCORE
RECALL
PRECISION
SGD
41.7
40.65
42.5
41.7
Monmentum
43.89
42.21
43.82
43.89
RMSProp
1.82
1
10
Adadelta
40.52
40.55
43.11
40,52
Adagrad
53.48
53.5
53.69
53.48
ADAM
10
1.82
10
ADAMW
1.82
10
ADAMax
60.23
59.52
60
60,23
DSA
62.15
62.08
62.08
62.15
14

---

Under review as a conference paper at ICLR 2022
MNIST
SVHN
CIFARIO
10
-0
10-4
20-4
AUM
20
ADMPAC
10
AGMMAX
20"4
25
Figure 8: Loss trend on DNN with different optimizers.
F RESULTS OF SMALL DATASETS IN DETAIL.
Table 7: Result on MI.P with four feature datasets.
IRIS
WINE
ACCU
RC
ACCU
RC
PCS
SGD
20.00
11.11
06.67
33.33
77.78
56.91
52.70
62.20
Monmentum
20.00
11.11
06.67
33.33
66.67
48.53
44.44
53.57
RMSProp
43.33
20.16
14.44
33.33
44.44
20.51
14.81
33.33
Adadelta
100.0
100.0
100.0
80.56
55.19
64.29
Adagrad
100.0
100.0
100.0
100.0
80.56
59.02
55.19
64.29
ADAM
100.0
100.0
100.0
100.0
80.56
59.45
56.52
64.29
ADAMW
100.0
100.0
100.0
100.0
80.56
59.02
55.19
64.29
ADAMax
100.0
100.0
100.0
100.0
83.33
79.10
78.89
80.26
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
CAR
AGARICUS
ACCU
F1
RC
PCS
ACCU
FI
RC
PCS
SGD
69.36
20.48
17.34
25.00
52.43
34.40
26.22
50.00
Monmentum
69.36
20.48
17.34
25.00
52.43
34.40
26.22
50.00
RMSProp
69.36
20.48
17.34
25.00
52.43
34.40
26.22
50.00
Adadelta
69.36
20.48
17.34
25.00
90.34
90.23
90.98
90.05
Adagrad
88.73
59.47
54.83
66.86
99.94
99.94
99.94
99.94
ADAM
77.46
33.05
33.35
33.86
99.94
99.94
99.94
99.94
ADAMW
77.46
33.05
33.35
33.86
99.94
99.94
99.94
99.94
ADAMax
74.28
29.12
30.23
30.38
99.75
99.75
99.76
99.75
DSA
98.55
96.31
97.68
95.08
100.0
100.0
100.0
100.0
15

---

Under review as a conference paper at ICLR 2022
G
RESULTS OF TRACK ADAPTIVE OPTIMIZERS IN DETAIL
3000
3000
+0.03
2075
EX0
-0.00
-008
-021
-8.00
-2.r AdaGrad
DSA
RMSPROP
0334
5228
0.044
0228
3851
0838
B008
0214
1M0
1.84
8084
0304
.800
000
-055
-044
+0.0
-30
-45
380
Adam
Adamax
AdamW W
Figure 9: Start from (-0.06,0.001), a= 1,6 1000
H SENSITIVITY ANALYSIS
In this section, we analyse the effect of different hyper-parametee of DSA. There are 3 hyper- parameters in DSA, i..., the initial value of learning rate Tlo updating stride Bn and 82. The larger Yo and 82 is, the faster DSA moves in the beginning. The larger 31 is, the faster DSA converges in the end. A group of DSA's tracks with different hyper-parameters No or 32 increases, the convergence will speed up because of larger stride. When 31 increases, the convergence will speed up because of the stronger adaptive capacity.
contained in Figure TO When
77
122, 81 0.6, B2 0.3
770
39,31 0.6, 3/ 0.3
110
6, B1 -06, ;3 = 0.3
100
678
0.0
000
800
-425
-22
-523
-433
2.50
-1.00
-16
288
-11
-34
-22
60
31
24
04
82
02
-12
-17
770
12,
91
0.6,
32
0.6 770
12,
31
0.9,
83
0.3
770
12,
00.,, 93
0.3
300
030
-22
-05
-2.
-056
-150
23
44
-22
-2.4
-00
-04
-63
20
-10
-60
-46
-0A
-03
Figure 10: Start from (-1,1), = 1,b 95
16


Reviews:
[{"review_id": "mXEgE-c2-l2", "reviewer": "Reviewer_57Xy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has serious shortcomings.\n\n- The authors claim to introduce a novel method in gradient-based hyperparameter optimization (adjusting the learning rate by using the derivative of an objective with respect to the learning rate) when this has been done previously many times including by some highly cited papers such as \n \nMaclaurin, D., Duvenaud, D. and Adams, R. Gradient-based hyperparameter optimization through reversible learning. ICML 2015. https://arxiv.org/abs/1502.03492\n \nBaydin, A.G., Cornish, R., Rubio, D.M., Schmidt, M. and Wood, F. Online learning rate adaptation with hypergradient descent. ICLR 2018. https://arxiv.org/abs/1703.04782\n \nand a series of more recent papers covering similar approaches. The first paper deriving this method is actually from the 1990s (Almeida et al., 1998) and this is explained in the ICLR 2018 paper listed above.\n \n- The method they claim to introduce and its derivation has the same analytical form as the method in the paper from ICLR 2018 listed above. So they are repeating the derivation of a method previously published at least twice without having any review or mention of related work which is quite easy to find.\n\n- The submitted paper does not have any coverage whatsoever of related work on gradient-based hyperparameter optimization, and the related works section and introduction cover only very basic level adaptive learning rate algorithms such as AdaGrad or RMSProp. A good pointer for the authors to discover and survey related work would be to start with this paper by Bengio (2000):\n\nY. Bengio. Gradient-based optimization of hyperparameters. Neural Computation, 12(8):1889–1900, 2000. doi: 10.1162/089976600300015187.\n\nand follow the trail of papers citing this paper up to our current decade.\n\nIn addition to these serious shortcomings, the paper is written in a way that makes vague and unsubstantiated statements such as:\n\n- “we … realize an optimizer with truly self-adaptive learning rate”\n- “our optimizer achieves fast and high qualified convergence in extremely short epochs”\n- “Adam usually cannot converge well in the late stages of the training”\n- “In a word, Adam is not sensitive enough”\n- “None of the current optimizers are stable and sensitive enough to ensure a fast and high qualified convergence”\n\nand many others throughout the paper.\n", "summary_of_the_paper": "The paper concerns an adaptive learning rate algorithm based on the derivative of the objective with respect to the learning rate. He authors introduce the technique that they call “differentiable self-adaptive learning rate” (DSA) and present experimental results comparing their technique with conventional optimizers including Adam, AdaGrad, SGD with momentum in tasks including classification with MNIST, CIFAR10, CIFAR100, SVHN datasets.", "summary_of_the_review": "The paper makes the incorrect claim that the presented technique is novel and it crucially fails at finding and citing any instances of related work which are very easy to find with simple search terms like “gradient-based optimization of hyperparameters”.\n\nI believe that this can be an honest case of an independent rediscovery of the technique by the authors combined with a lack of experience in finding related work and writing the paper in a more grounded way (e.g., avoiding vague statements). I believe that there is value in the experiments presented by the authors and I would like to encourage the authors to keep working in this problem space and improving their manuscript to get it eventually published after addressing the crucial shortcomings and finding a new angle to present their contributions.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "cpisJBGTT77", "reviewer": "Reviewer_bLyu", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The presented method is already known, the authors do not cite the preceding works in that line of research, and pretend that \"[they] are the first to make learning rate in neural network differentiable with the goal of minimizing loss function\".\n\nSome non-cited preceding papers:\n * Seminal paper: *Learning to learn by gradient descent by gradient descent*, Andrychowicz 2016.\n * Paper with explicit optimization of $\\eta$ by gradient descent in the context of feeforward neural networks: *Meta-SGD: Learning to Learn Quickly for Few-Shot Learning*, Li 2018.\n * Early preprint in the same in the same line of work: *Speed learning on the fly*, Massé 2016.\n\nBesides, the notation lacks rigour: \n * $\\tilde{L}(\\eta)$ is not truly a function of the learning rate $\\eta$, it also depends on the parameters $\\theta$.\n * The use of $\\tilde{\\theta}$ for the SGD update of $\\theta$ is a bit clumsy: using $\\theta_t$ and $\\theta_{t + 1}$ would have been clearer. Besides, the use of $\\tilde{\\theta}$ without index $t$ to denote the training step shadows some freedom that we have when using the algorithm: we may want to train $\\eta$ each $T$ steps (with arbitrary $T > 0$) instead of each $1$ step.", "summary_of_the_paper": "The authors propose to tune the learning rate of a SGD, by training it by gradient descent.", "summary_of_the_review": "This work is practically contained in existing papers, which are ignored by the authors. Strong reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Dy8eoSa81m8", "reviewer": "Reviewer_thhp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well organized with an intuitive explanation and extensive experiments, and it can be considered as a good empirical study on investigating adaptive learning rates for solving machine learning problems. However, there are a few weaknesses of the paper, and I am listing them (with questions that could improve reviewers' understanding) here. \n\n1: First of all, the DSA method is in fact minimizing the loss function given current iterate $\\theta$ and gradient $g$. Thus, if the modified loss function can be solved exactly, it would produce the optimal $\\eta$ on the direction with current iterate and gradient. However, how could the author justify that it is better than a constant (or indeed any random) learning rate? Can you justify that in theory? \n\n2. Secondly, the novelty is weak as more literature review in optimization field would give rich content related to the DSA method. The classical way of deriving adaptive learning rate (mostly for the deterministic problem, but also stochastic problem in more recent literature) is line search. Line search finds appropriate learning rate along the same direction that DSA does. In addition, implicit SGD[1] was developed with implicit schema to derive a learning rate. Also, more recent work of deriving adaptive learning rate was proposed for variance reduced method [2].\n\n3. Indeed, the modified loss function is a one-dimension optimization sub-problem. The way that DSA deals with the sub-problem is to either propose a gradient descent or use hyper-parameter to perform one-step update. Could you solve it approximately by using the newton method? In [2], a predefined sub-problem was solved approximately by newton method.\n\n4. As authors define the terms in the paper, such as \"stable\", \"sensitive\", \"grad loss\" etc, this paper tries to explain the issues that was caused by non-adaptive method. However, the proposed DSA method is not solving the challenging optimization problem in the field. Could the weakness of convergence (as authors mention) for other optimizers be analyzed more? For example, optimizer often struggles to escape the saddle point, Adam with constant step-size only converges to neighborhood of local minimum, etc. I strongly feel that more in-depth understanding of issues with classical optimizers is needed for the proposed method to address the concerns in the field.\n\n5. The empirical study is extensive. However, a few confusion that I am having are:\n\n5.1 How DSA and other algorithms were configured? It seems like DSA also adopts batch training.\n\n5.2 Does DSA incur additional cost? Could it be more fair to compare algorithms with respect to computational cost instead of epoch?\n\n5.3 The plots for oscillation are specific for some problems, and would it cover general case?\n\n5.4 Other algorithm requires fine-tuned learning rate, would it be fair to compare with fine-tuned version?\n\n\nReferences:\n[1] see https://sites.google.com/view/panos-toulis/implicit-sgd\n[2] see https://arxiv.org/abs/2102.09700\n", "summary_of_the_paper": "This paper presents a method of adaptive learning rate in training deep neural networks and solving machine learning problems. The proposed method derives the adaptive learning rate by minimizing the loss function on the direction of $\\\\theta - \\eta g$, with $\\eta$ being treated as a parameter. ", "summary_of_the_review": "In summary, I'd love to see improvement in this paper. Based on my understanding of the field, I strongly recommend authors analyze the pro and cons of other algorithms and investigate in-depth the issues of not having a truly adaptive learning rate. with adaptive learning rate in developing optimization methods, it is often necessary to include theoretical analysis. And, theoretical analysis and empirical study can work together to understand the proposed method better. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BMi_UpafGS7", "reviewer": "Reviewer_piNT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors propose an optimizer called DSA, which can increase or decrease very sensitively and significantly with the help of the learning rate’s gradient, which overcomes the disadvantages of the existing algorithms e.g., AdaGrad, RMSProp, and Adam. The proposed DSA is the first optimizer that has a clear instruction of directing the loss function, which applies to a wide range of DNN models as well as two classical machine learning problems. The paper has presented extensive experiments and proofed the values of DSAN as what I believe so far. ", "summary_of_the_paper": "The authors propose a novel optimizer that makes the learning rate of a deep neural network to be differentiable in a self-adaptive manner, which overcomes the problems of existing problems DNN optimizers i.e., insensitive and unstable.", "summary_of_the_review": "The paper is ready for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper deals with the problem of adjusting the learning rate during gradient descent optimisation. Unfortunately the proposed approach is very similar to methods already presented in the literature and no significant contribution can be recognised. During the rebuttal, the author(s) have acknowledged their ignorance about the relevant literature and provided some further clarifications that did not turn into a revision of the reviewers’ initial assessment of the work.
Decision: Reject