ADA+: A GENERIC FRAMEWORK WITH MORE ADAPTIVE EXPLICIT ADJUSTMENT FOR LEARNING RATE
Authors: Yue Zhao
Xiangsheng Huang
Ludan Kou
Venue:  (2020)
Keywords: Optimization
Adaptive Methods
Convergence
Convolutional Neural Network

Abstract: Although adaptive algorithms have achieved significant success in training deep neural networks with faster training speed, they tend to have poor generalization performance compared to SGD with Momentum(SGDM). One of the state-of-the-art algorithms, PADAM, is proposed to close the generalization gap of adaptive methods while lacking an internal explanation. This work pro- poses a general framework, in which we use an explicit function Φ(·) as an adjustment to the actual step size, and present a more adaptive specific form AdaPlus(Ada+). Based on this framework, we analyze various behaviors brought by different types of Φ(·), such as a constant function in SGDM, a linear function in Adam, a concave function in Padam and a concave function with offset term in AdaPlus. Empirically, we conduct experiments on classic benchmarks both in CNN and RNN architectures and achieve better performance(even than SGDM).


Under review as a conference paper at ICLR 2020
ADA+: A GENERIC FRAMEWORK WITH MORE ADAP- TIVE EXPLICIT ADJUSTMENT FOR LEARNING RATE
Anonymous authors Paper under double-blind review
ABSTRACT
Although adaptive algorithms have achieved significant success in training deep neural networks with faster training speed, they tend have poor generaliza- tion performance compared to SGD WITH MOMENTUM(SGDMM, One of the state-of-the-art algorithms, PADAM, is proposed to close the generalization gap of adaptive methods while lacking poses a general framework, in which we use an explicit function 2(-) as an adjustment to the actual step size, and present Based on this framework, we analyze various behaviors brought by different types of (f-), such as constant function in SGDM, lin- ear function in ADAM, a concave function in PADAM and a concave function with offset term in ADAPLUS. Empirically, sic benchmarks both in CNN and RNN architectures and achieve better per- formance(even than SGDM). The code is anonymously provided in https: cetponffiles..cmmaaa/aaeeee
internal explanation. This work pro-
adaptive specific form
ADAPLUS(ADA+)
conduct experiments on clas-
/wd.a//daalla..a lus__ zip
INTRODUCTION
First-order optimization algorithms play a significant role in training deep neural networks. One of the dominant methods is SGD(Robbins & Monro 195I). which updates parameters in a very improvement of SGD that combines raw gradients reduce the interference of noise gradients and has proved itself to be an efficient first-order optimization algorithm. Besides, adaptive variants of SGD, such ADA- Hinton 2012), A & Ba 2015). have recently emerged and achieved success due to their convenient and fast automatic learning rate adjustment mechanisms. However, since ADAM is indicated not to converge in certain instances, AMSGRAD et al. 2018) suggests maintaining a non-increasing step size by maximum his- torical term, so as to deal with the non-convergence problem.
concise form. SGD WITH MOMENTUM(Polyal 1964) is momentum-bart
GRAD (Duchi et al., 201 AMSPRoP(Tielemaa
On the hand, adaptive methods are widely applied due their quicker convergence at early stage. However, there remain potential perils adaptivity (Wilson et al.) 2017). which suggest worse performance than the fine-tuned SGD/SGDM. Therefore, many architectures in the field of computer vision, such as CGGNet(Simonyan & Zisserman] 2015). Res et al.) (016)), DenseNet(Huang et al. 2017). and natural language processing lasks/Klein et al. 2017). would be inclined to choose SGD-like algorithms to act as an optimizer. On the other hand, the 'small learning rate dilemma' (Chen & Gu 2018 is also a challenge for adaptive methods, that is, it is difficult to apply the well-worked learning rate decaying strategy in SGD to adaptive gradient meth- ods. With better performance, the PADAM algorithm
& Gu 2018) proposed solve this problem. However, the motivation of this partially adaptive momentum fails to explain the intrinsic implication of doing grid search for the power p of the second-order moment estimation, along with the relationship between the desired learning rate and the partially adaptive term.
In this work, we first propose an adaptive adjustment function, which is a map based on histori- cal gradient information to adjust the actual step size, and a framework that can explain the above phenomenon explicitly. Through our analysis of this function, which combines the SGD-like algo- rithm with the ADAM-like algorithm, rithms. Moreover, we propose a brief but efficient specific form, called ADAPLUS(ADA+) with better adaptability to cope with the aforementioned challenges effectively. Empirically, conduct
further explain the different behaviors of different algo-

---

Under review as a conference paper at ICLR 2020
experiments popular models and tasks both in CNN and RNN architectures with ADAPLUS. Comparing favorably to SGD-like approaches, gains faster convergence, reduced oscillation and better performance. Compared with ADAM-like methods, it can adapt to variable learning rate schedule, thus achieving significant improvement in performance complex deep neural network architectures.
2
PRELIMINARIES AND MOTIVATIONS
2.1 NOTATIONS
As is generally agreed, we use lowercase letters for scalars, lowercase bold letters a for vectors, and uppercase bold letters A for matrices. Then, we set 0, IR parameter vector of a sequence in d-dimensional space, and use the scalar 0z,; to represent the i-th elements in vector 0,. The set of all positive definite x d matrices is denoted by S Besides, operators are defined as following. For vectors a and b in the same dimension, we use va to donate element-wise squares, a/ to donate element-wise divisions. These notations are the same for matrices. For given matrix A E S and vector b E Rd, we use IIx, Ab) as projection operator which means arg minagx A//2 b) where X is convex which has bounded diameter Doo, i.e., l| bllsc S Doc, v a. b. X. Additionally, with mild abuse of the notation of norms, we represent the TP norm formation of an exponential moving average (EMA) of g: like this:
1 lp - o.
13 U--1 + (1 - (1- 19t 17
(2.1)
(1 35
3 (p(e-t) |g.1"
1=1
2.2 MOTIVATIONS
As is described in Figure [ for optimizers, we treat their adjustment term of a, as a mapping from Ige lp to the actual adjustment value, i.e.,
0
4 ( llp)
m
(2.2)
To avoid confusion, we call a the learning rate, which is parameter that needs tuning and applying a decaying schedule. Apart from that, 47 [.]. is called the actual step size. In this section, we summarize popular first-order stochastic optimization algorithms into the following four categories:
+ ((1gt (p))
2.0
Adam
1.5
OurMethod
Padam
1.0
SGD
xx
0.5
+
ll
0.5
2.0
Figure Different choices for @C1lg. lpp,; changing from Adam to SGD. We propose specific function T (11ge lIp) Vllgel + A, which shown above.
2

---

Under review as a conference paper at ICLR 2020
Type I
SGDM (Constant function):
0++1 0, mr, i.e, ((1geelp)= =1
(2.3)
the actual step size of each update equals to a,, which is applied without adaptive adjustments. The high learning rate in early stage makes the loss function fluctuate significantly the surface, and its learning speed is not as fast as adaptive optimization algorithms and their variants.
It can be seen from the formula in SGD WITH MOMENTUM(SADNN
Type ADAM (Linear function):
0 0, 1 m., i.e., (11ge () 119:112-
(2.4)
Extremes of the learning rate Wilson et al. 2017) in ADAM-like algorithms have brought many issues. As shown in Figure I when the gradient is very large, the function value grows greatly in linear rate, which leads to a rapid decrease in the step size; the other hand, once the gradient is extremely small (ie, when the norm of EMA of gr approaches 0), ((lgrl))) |19:|1 is very close O, which makes more than essential to select a much smaller learning rate(a few orders of magnitude smaller than SGDM) to get a rea- sonable actual step size. However, the much smaller initial learning rate makes ADAM- algorithms not flexible enough to adapt to the various learning rate schedule. Although their early convergence faster, the final performance tends to poor due to the small ac- tual step size in the final stage. A proposed solution, ADAMW (Loshchilov & Hutter Hutter]2019), trying to change the way the weight attenuation is updated, achieves a better generaliza- tion performance, but there still remains gap between adaptive methods and SGD-like methods.
Type IIII PADAM (Concave function):
02-1 0:
Ot
m.s i.e., ((1gg 11
(2.5)
1
Note that the here means partial rather than L norm, its default setting is PADAM has achieved superior performance(Chen & Gu 2018) in computer vision experi- ments, but this algorithm only introduces a new hyper-parameter p and does grid searches for it without realizing the essential reason for its improvement relative to ADAM. The internal cause that concave function is applied rather than the linear function in ADAM. Once is extremely small and ||9el12 E E), the mapping value of Φ(.) would be much larger in PADAM than in ADAM; therefore, PADAM can adapt to larger learning rate as thus flexibly adapting to the variable learning rate scheme.
Type IV ADAPLUS (Concave function with offset):
0.41 0 Vigl+++ my i.e., ((1la)))) v 11 +A.
(2.6)
In ADAPLUS, we extend the concave function by introducing an offset A and present default setting as above. This form of (b-) not only directly inherits advantages of PADAM, as is depicted in Figure but also makes better guarantee for larger learning rates. The offset A makes sure that ((-) can altogether avoid the extreme situation. Even when [|g:11 - O, a more extensive learning rate a, allowed. Besides, when l|g. || rela- tively large, we can also adaptively constrain the updates and achieve 'naturally annealing' like ADAM, but more moderately. Thus, we reckon that the proposed algorithm has better adaptive performance than both ADAM-like and SGD-like algorithms, which is why we call it ADAPLUS(AAA+)
So far, we may surprisingly notice that this offset A is the that we used to apply in optimizers indeed. Nevertheless, the previous is only used avoid dividing by zero to keep the numerical stability. In fact, through our analysis above, the design of offset A can effectively avoid the occurrence of extreme step sizes, thus saliently improving adaptivity and performance.
It is bit different from the formation in Chen & Gu 2018 but actually the same.
3

---

Under review as a conference paper at ICLR 2020
We present default setting above, due to its excellent performance and concise form; however, there would be more elegant forms remaining for us to explore the future. For instance, replace the square function with log(-) or tanh(-) and on, tune into p-th power function to apply PADAM-like grid search.
THE PROPOSED FRAMEWORK AND METHOD
3.1 GENERIC FRAMEWORK
As is shown in Section 2.2 we propose a generic framework that includes SGD-like and ADAM-like algorithms. With the adaptive adjustment function ((lggellp) to combine algorithms in an organized way, behavioral characteristics of different algorithms can be explicitly observed.
Algorithm 1 GENERIC FRAMEWORK
Input: 0 E X: learning rate (ar) momentum parameters (B1))=1 $2; LP norm parameter p; function for step estimation (--).
1: Set mg = 0, Uh = 0, 00 =
2: for
to do
3: gi = f(++))
4: m :10mh:11 (1 B1e)gr
5: D 8201-1 + 82)1g.1 9.1
6: i. max (01-1, v4)
7: Depending on whether AMSGrad is desired, the previous line can be optionally annotated.
8:
0++1 x,dias (6//r)
θ
m
9: end for
10: // We omit the bias-correction terms and other mise for clarity.
According to our analysis, different settings of the adaptive adjustment function (( (pp) will lead different behaviors of the algorithm, such ((11):ll)) for SGDM and ((1g g lp for ADAM. In practice, we can selectively decide whether or not to open AMSGRAD according to the actual situation. We will present a specific algorithm that has concise form and demonstrates superior performance in Section 5
3.2 SPECIFIC FORMATION FOR ADAPLUS
Algorithm ADAPLUS
Input: 0, X: learning rate {ar)==; momentum parameters Bu 82g offset term A.
Set my 0, Un
O, 00
2: for t = 1 to T do
3:
gt fxx(
4:
31e)g. 5: e, B20111 (1- 32) 9:
m. B1
6: T, = max (7)
7: Depending on whether AMSGrad is desired, the previous line can be optionally annotated.
8: 0 xI Dr..din(Veetad
(. vu++A mc
9: end for
// We omit the bias-correction terms and other misc for clarity.
Here, we express the specific algorithm corresponding the Type IV function mentioned in Section following. It is special case of Algorithm 1
2.2
0,
0z
Mr i.e., (11gg Vllg.l1 + +
(3.1)
Vlg.+++
Two main contributions are made, including the concave function v2 and an offset term A, so the algorithm avoid extreme actual step sizes and achieve superior empirical results.
4

---

Under review as a conference paper at ICLR 2020
4 CONVERGENCE ANALYSIS OF ADAPLUS
In this section, we provide a convergence analysis based on standard online convex optimiza- tion framework(Zinkevid) 2003). For each time step [7], there sequences of parameters {0:) and convex loss functions {J+)²=1- Let be a bounded convex feasible set, which cludes {0)):-1 and the optimal solution 0. The optimal solution 0 is defined to achieve empirical risk minimization(ERMM) i.e., 0 argmin Σ f() We use regret Rr to donate the entire
DEx
difference between Σt= f((O,) and its minimum value, i.c.,
R Σ fi (0)) min E f((0) Σ f(
(4.1)
With further assumption of bounded gradients, we establish convergence analysis of ADAPLUS to ensure the bound of regret follows.
Theorem 1. Let o a/Vi 31, 32 (0,1)
(0, and Bn S B1, 4 E [T]
Assume that the convex feasible set X' has a bounded diameter Doo, i.e., V a. X, || yll
D Doo. Also, assume that loss functions fo(-) are convex and have bounded gradients, i.e. G
O, 0 E X and t E [T). f:())l x I Go For 0₂ generated using the ADAPLUS (Algorithm
we have the following regret bound:
D2
a (1 B1) /1 log T
PT
(ÔT, A)T
20 (1
31)
(1 - B1) (1 7) 32
(4.2)
D2
T
d
31t
+ 2 21 - B1)
01 +A
This theorem leads directly to the following two corollaries. We will provide the proof of Theorem
and Corollary 2 in Appendix(A)
Corollary 1. Under the conditions in Theorem 7 and supposing B1e -31/t, B2 2 0, we have the
following regret bound:
D
a a (-) VI+ log
Rr S 2a (1 B1)
(pTR + A)T
19.r,ell2
(1 (1-31)² 7) 32
(4.3)
31dD2 (1+
2a (1 31) (1-
Corollary 2. Under the conditions in Corollary 7 we further have the following regret bound:
dD2 p)
a a B1) dG ( log
3,dD2 v1++
RT 5
2a 20(1- -B))
(1- (1-
(1
(4.4)
which means Ry = Ö(VT), where Ö(-) donates the omission of fogarithmic
EXPERIMENTS
5.1 EXPERIMANTAL SETTINGS
In this section, conduct full experiments classical tasks both in CNN and RNN architectures, and compare ADAPLUS with popular algorithms, such as SGD/SGDM. ADAM and AMSGRAD to evaluate performance. Unless otherwise specified, ADAPLUS that utilized in practice is based on the implementation of AMSGRAD, i.e., i, = max (ū,-1, v.) is applied, as is described in Algorithm 2 Overall experiments summarized in Table [
In the field computer vision, we consider three different architectures on the standard CIFAR- 10/100 Krizhes(ki et al. 2009). We VGGNNtt-16(Simonn & Zisserman 2015
5

---

Under review as a conference paper at ICLR 2020
Table An Overview of Experiments.
Tasks
Architectures
Datasets
Framework
VGGNet-16(Simonyan & Zisserman
2015
CV
ResNet-50 He al., 2016
CIFAR-10/100
Torch
DenseNet-121
Huang et 2017
NLP
OpenNMT Klein et al. 2017)
IWSLT15
Tensorflow
Res et al., 2016) and DenseNet- 121 221@Han et al. 2017 where there was clear dis- tinction in the number of layers to better identify different behaviors of algorithms. We employ the fixed budget of 200 epochs and multiply the learning rates by 0.1 at 100th and 150th epochs. In the Neural Machine Translation experiment(?), we chose the learning rate attenuation scheme lu- ong234, which means after 2/3 num train steps, we start halving the learning rate for times before finishing. Using the same parameter settings the benchmark (Luong al. 2017) of 12000 steps on Titan XP.
total
For reproducibility, provide the codes anonymously in https://anonff: BaV7Ed6enb/Adare lus_ znpland put full details of experiments in the Appendix C
com/
5.2 CONVOLUTIONAL, NEURAL NETWORK ON CIFAR-1 10/100
In this section, we train three different models with various numbers of layers accurately compare the performance of optimizers. We also conducted experiments both on CIFAR- 10 and CIFAR-100 examine the impacts of complex tasks and different datasets on the behavior of optimizers. We report training loss and test accuracy in the following figures.
In general, we notice that in deep neural network architecture the final performance of SGDM is significantly better than ADAM and AMSGRAD, which shows issues in generalization of ADAM- like algorithms once again. Additionally, our surprise, our approach ADAPLUS outperforms SGDM in the various models and tasks almost at all the time step (with significant differences or bit similar).
AdaPlus
AdaPlus
AdaPlus
SGOM
Padam
Padam
Podaim
AMSGrad
Training Loss
AMSGrad
Treiming Loss
AM5Grad
Adam
Adam
Adam
Training
150
1G
53
120
055
136
Epochs
Epachs
Epochs
(a) VG Ci
(b) ResNet-50 Cifar- 10
(c) DenseNet-121 Cifar-10
Tost Accunccys %
Test Accuracy'
Accumny
AdaPlus
Adalfuus SCOM
AdaPlus
SCIDM
SCIM
AMSGrad
AMSGrad
AMSGma
Adam
Adam
12
300
120
148
Epochs
(d) VGGNet-16 on Cifar-10
(c) ResNet-50 on Cifar-10
(f) DenseNet-121 on Cifar-10
Figure 2: Training loss and test accuracy of different architectures on Cifar-10.

---

Under review as a conference paper at ICLR 2020
14
AdaPlus SGOM
14
Adalfus
AdaPlus
Padem
Padam
Padam
10
AMSGrad
Training Loss
AM5Grad
Treining Loss
AM5Grad
Adam
Adam
Adam
64
60
100 Epochs
1G
25
055
134
Epachs
Epochs
(a) VGGNet-16 Cifar-100
(b) ResiNet-S0 Cifar-100
(c) DenseNet-121 Cifar-100
Test Acturad %
Test Accuracy
AdaPlus
Ada
AdaPlus
SCDM
SCOM
SCDM
Padam
AMSGrad
AMSGrad
AMSGmad
Adam
50
200
Epochs
Epochs
(d) VGGNet-16 on Cifar- 100
(c) ResNet-50 on Cif
(f) DenseNet-121 on Cifar- 100
Figure Training loss and test accuracy of different architectures on Ci0ar-100.
Summary of behaviors. Empirically, the behaviors of various algorithms match with what analyze in Section 2.2 Primarily, ADAPLUS has similar behaviors to PADAM, showing faster con- vergence and less volatility than SGDM in the early stage. The results of PADAM and SGDM mostly identical, sometimes SGDM wins while other times PADAM wins. However, ADAPLUS has achieved better performance than both of them due the introduction of offset term A. In addition, both ADAM and AMSGRAD perform poorly in the end, although their early training losses decline rapidly and their convergence faster.
Summary of performances. For VGGNet-166, 16, ResNet-50 and DenseNet-1211 121, ADAPLUS has achieved test accuracies of 0.26%, 0.46%, and 0.15% higher than SGDM in Cifar-10 tasks; while in Cifar- 100 tasks, the gaps are about 1.18%, 1.62% and 0.54%. In terms of architectures, the most significant improvement of ADAPLUS incurs in ResNet-50, followed by VGGNet- 16 and DenseNet- 121. ADAPLUS has achieved an accuracy of about 80% in ResNet-50 on CIFAR-100, which is significant improvement over SGDM. terms of complexity of tasks, when the tasks are more complex, ADAPLUS can gain better promotions, since architectures have been improved more by ADAPLUS on CIFAR-100 than on CIFAR-I0.
5.3 NEURAL MACHINE TRANSLATION
In this section, we evaluate the performance of ADAPLUS classic NLP task Neural Machine Translation (NMT))Klein et al., 2017). Due to the particularity of NLP problems, adaptive opti- mization algorithms and their variants do not perform as well SGD; therefore, many researchers prefer to use fine-tuned SGD (illson et al. 2017).
OpenNMT has standard benchmarks with open source codes[Lunng et al. 2017), from which we select to conduct the IWSLT15 Since there is no TensorFlow implementation for PADAM, we will not consider but focus on the comparison of ADAPLUS with the state-of-the-an benchmark. As is generally agreed, standard SGD with Iy = 1.0 can achieve the best average performance of 26.1 in the model with beam = 10. We apply ADAPLUS to the same task in this section. Besides, experiment in the Vietnam-to-Ennee Enngisl task of 15k steps in Appendix D using the same settings as ADASHIFT (Zhuu et al. 2019). which also yields better results than SGD.
Vietnam task with the same settings as its benchmark.

---

Under review as a conference paper at ICLR 2020
Figure shows that AMSGRAD'S performance is worse than ADAM and that despite the fluctuations of ADAM are smaller, SGD performs indeed better. Therefore, we do not apply AMSGRAD in ADAPLUS, ie., the sixth line of Algorithm i annotated. Our NMT experiment further proves that ADAPLUS is more suitable for NL.P problems with recurrent neural network model and sparse data set. As is shown in Figure + ADAPLUS is almost always superior to all other methods. Finally, has achieved the best BLEU slightly higher than SGD, as shown in Table
102
AdaPlus
SGD
82
Training Perplexity
AMSGrad
Adam
Test BLEU
20.5
37.1
25.2
AdaPlus
SGD
AMSGrad
7.2
2002
4000
9200
8:00
13000
12000
2002
4000
6000
8000
10000
12100
Steps
Steps
(a) Training perplexity
(b) Test BLEU
Figure 4: Training perplexity and test BLEU NMT.
Table Best BLEU for 12k steps on IWSLT15 English-to- Vietnam.
Optimizer
SGD
ADAM
AMSGRAD
ADAPLUS
Best BLEU
26.12
24.74
23.17
26.31
6 FUTURE WORK
Although we present a generic framework and an explicit algorithm with excellent performance, there would be more elegant forms remaining for us explore in the future.
1. The choice of function (--). For instance, we can replace the square function with log(-) tanh(-) and so on. Also, the square function can be tuned into p-th power function to apply PADAM-like grid search.
2. The choice of LP Norm parameter p. Different LP Norm parameter can be tried to get better numerical performance.
CONCLUSION
This work proposes novel generic framework, in which explicitly analyze different behaviors brought by various types of ( such as the constant function in SGDM, the linear function in ADAM, the concave function in PADAM and the concave function with offset term in ADAPLUS. With better adaptivity as is demonstrated, ADAPL.US has achieved remarkable superior results in both CNN and RNN experiments. Our main contributions can be summarized as follows:
A generic framework. Combining ADAM-like algorithms with SGD-like algorithms, the adaptive adjustment function ((.) suggests a generic framework we desired.
Explicit analysis different algorithm behaviors by ((-). Based on the explicit analysis of (c.), we explain the different behaviors of different algorithms, which is fundamental reason never mentioned before.
The proposal of concave function with offset term. In ADAPLUS, we propose offset term A, which can further avoid extreme actual step sizes based on the concave function and achieve superior performance.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jinghui Chen and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. CoRR, hts/1806.06663, 2018. URL http::/aaxiiiii/ org/
abs/ /106/10....0 06763
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 120ul):2111-2159 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. OpenNMT: Open- source toolkit for neural machine translation. Proceedings of ACL 2017, System Demonstra- tions, pp. 67-72, Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https: / www.waclwee. www.:w/wwwww..or.........o...... httpp//rrr...tooogggg//oeoo
4012
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer-
ence on Learning Representations, 2019. URL
BkqERiCqY?
Minh-Thaie Hinh-Thang Luong, Eugene Brevdo, and Rui Zhao. Neural machine translation (seq2seq) tutorial.
http:://ghhhhh..oom/eesstttt//nn 2017.
H. McMahan and Matthew Streeter. Adaptive bound optimization for online convex optimization. COLT 2010 - The 23rd Conference on Learning Theory, 02 2010.
B.T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Com- putational Mathematics and Mathematical Physics, 4(5):1 17, 1964. ISSN 0041-5553. doi:
http://0000000000111015500500000 com/ cttp://ccinnno//rrrreeee///////5
URL http://t///// /ttp:///wwwww. www. reciencedirett..
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statisties, pp. 400 400-407,11 1951.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, May 2015.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):66 31, 2012.
Ashia C Wilson, Rebecca Roclofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 48-4158, 2017.
Zhiming Zhou, Qingru Zhang, Guansong Lu, Hongwei Wang, Weinan Zhang, and Yong Yu. Adashift: Decorrelation and convergence of adaptive learning rate methods. In International Conference on Learning Representations, 2019. http:httss///pperret
net forum?
ldd=HkgkkhRck]
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pp. 928-936, 2003.

---

Under review as a conference paper at ICLR 2020
A PROOF OF CONVERGENCE
A.1 PROOF OF THEOREM I
Proof. Since fe(-) are convex, which means V x, E &
fi(y) 2 V((())
(A.1)
Then, we have
RT
(J: Se (0))
(A.2)
S Σ Bg O. --)
Consider the explicit update formula in Algorithm 2 and the notation V. diag (V.). Here, get 0.+1 II x.(+++)+ (0 - an + - m.
We use Q V.+ S donate the matrix for simplification. Besides, find that Q (V,) is the general formation in Algorithm long as it satisfies the requirement of being positive definite matrices, the following derivation suits as well.
For the definition of 0*, it holds that 1Lx.Q (0') 0", va E X. Using Lemma
we have
| (0.-1 00) S $ (0. a. 0²)
(A.3)
Expanding the squared norm on the right side of the inequality and further expanding the momentum term, there
(0+1
0*)
(0:
0*)
+
Q2 2
20z (m, 0: - 0-))
(A.4)
Q (0: 0*)
+
Q,
(1))1d) 2a, (B + (1- 31d) ge. 0, 0*)
2
Rearranging the terms in the above inequality, the following first inequality holds,
(g. 0, - )*) )
Q/ (0, - 00
(00)00) Q (0 - 03/1/9]
Q, mm
2
20, (1 311)
2 - 31e)
2
B1e
, 0r)
1 B1
(A.5)
Q (0, 0r)
Q (0.+1 -0)1-1))1
2 a2
Q,
2
m
20, (1
311)
310)
2
+ 2 B1,)
31rCs
Q
+
Bi
| 0*)
20z (1 310)
and the last inequality holds due to Cauchy-Schwarz and Young's inequality.
Combining A.2 with A.5
have
10

---

Under review as a conference paper at ICLR 2020
T
(0r) fo (0;)
1
Σ
T
i (0, - -0)) ||= -
l (0l+1 - -0) 2
+
at
0 m
2
20 (1 - Bie)
2
2(1 (10)
2
1=1
2 +
31,
o (0, - 03)13 (00-0))
+
Q 1
m
2 (1 818)
20 (1 - B1t)
(A.6)
As is separately proven in Lemma bound.
Lemma 3 and Lemma 4 we can further get the desired regret
T
RT
Σ
(0.) - Se (0:))
t=1
D
a a + B1) vI + log? T
d
0C
(ir, + A)T +
Σ
(A.7)
20: (1 31)
(1- 31)² (1 V/ 32
1=1
D2
B1t
/ + A
2(1 31)
04
A.2
PROOF OF COROLLARY II
Proof.
Since assume that 81, BiA4, A E (0, 1), have
D2
T
Bi
Ves +
2 2(1 - 31)
(A.8)
D2
T
B
V
(Te, + A)
2 - 31)
Besides, due to the definition of 0r,i, we have
A
04,5
32)
5 + B2)
Σ
3
19
+ 1
(A.9)
B2
Since A > 0 given constant, which is chosen to be relatively small enough, it always holds that 3 p > 0, s.t., A S p. Goo. Then, we can further improve the result (A.9 V [T].
11, + 41 B2)
3/-1 Σ
Goo G. S (1++p) p) Goos
(A.10)
which leads to
D2 2(1 31)
P1
Ut,i + A
8 D2 (11 G
T
(A.11)
2a (1 - 31)
m=1
3idD3 (11 G1+
2a (2(1-)) (1) 2)²
11

---

Under review as a conference paper at ICLR 2020
Submitting
A.11
into A.7
we extend the conclusion in Theorem
| to Corollary
T
Rr
[ee
(0.) - fe (6,))
1=1
D
d
(iT, + +AT + a + 31) v1 log 7
Σ
11g.r,ll22
(A.12)
20
(1
B1)
(1 31) (1 7/ 32
i=1
111
D v(1 + Goc
2a 81) (1 21
A.3
PROOF OF COROLLARY
Proof.
Primarily, proved in (A.10), holds that
2 Σ
,r, + A < (1 B2)
32
Gx
50 G)
(A.13)
j=1
Thus, we have
D2 D3 VT
d
Σ
v UT. +A
20 (1 -3 31)
D&.V VT
Σ
(A.14)
20 (1 - B1) 1=1
dD2 (1+ Gx T
2a (1-3) -11)
Second, due to the property of convex functions and bounded gradients, we further have
Σ 19.T
st. S dG VT,
(A.15)
Pluging A.14 and (5.15)
back into A.12 we finally get the result of Corollary
T
Rr
Ife (0.) - (0,)
1=1
dD /(1 + P) GoO
a a dG /(1 log T
(A.16)
2a (1 - B1)
(1 - 31) v)11 82
BidD2 (1 Goo
2a (1 31) (1
which means Ry = Ö(VT). where (()) donates the omission of logarithmic factors.
B AUXILIARY LEMMAS
B. PROOF OF LEMMA I
Lemma 1 McMahan & Streeter 2010); Reddi et al. (2018); Chen & Gu (2018)). For any Q E
S4 and convex feasible X C Rd suppose ui arg m Q=7²² @ 21 | and u2
arg aigging |/ x 22) || then we have 1/2 (ui u2)1l2 I || IQ" 1/2 /2 - x2) ||2.
We will not provide the proof here, since it will be exactly the same as the reference and has been proved many times.
12

---

Under review as a conference paper at ICLR 2020
B.2 PROOF OF LEMMA
Lemma 2. Under the conditions in Theorem 7 we have
Σ
02
Q
1 m
310
Q
m
[]]
(1 81) (1 7)v 82 Σ 191.1
a11 B1) vI + log T
2 31d)
Proof. Similar to (Reddi al.
2018
Chen & Gu
2018), first describe the upper bound of
a.mi in the following derivation, since
Q: - VV++
a
+
$ donate the update rule.
T
at
Qt
m
T
at
m
Ut, + A
04 m
aT . m
Ve. +
UT. + A
04 m
OT m
(B.1)
01, + A
UT, + A
d
Σ/= (1- B15) 35 95.
T
a4 m
a
Σ
Ut A
(1 (2) Σラ= -/--111 -1 + A
Σラ of !g 1
(Ej= B 8 19
2 m
D1, A
VT
32)
A VE 11 + 1-B2
These several equalities holds due to the update rule and definitions of ag, Vt, and my Also, the first inequality holds due the definition of Ür while the second inequality holds due to B1z S
B E [T] and Cauchy-Schward inequality. Since Σ HT- S 1-9, and fe(-) has bounded gradients, we have
1=l
a
Q
m
O4
m
av Go
Σラ=1 11
T-j
119.11
Ut + A
31) V T T(1 -12) 1=1
Djm 0
T
T-1
19
A
1-
Oi m
avG
Σ 3 gi
(B.2)
+
(1- B1) T( 322
E j=1 12
T
04 m
av
3 1
Ui.i +
(1- T -32)
04 m
ay Gx
T
T goaal
Dr.i +A (1- -2)
j=1 1=1
where the second inequality is indeed strictly less for A
> 0, the third inequality holds due to the
property of concave function. At last, as is assumed, 7 v3, (0. 1) is introduced.
8.
13

---

Under review as a conference paper at ICLR 2020
T
T-1
d
grmic
We further expand
4:
m
in the same way, which leads to
00.+4 A
T
Σ
Q1
mi/
1=1
VVG.
T
$
99.d
(1 31) v1
12 1=1
aVG2
d
T
T
70
(1 B1) vT
B2
iml
1=1
j
avG
d
T
19t.sl
< (1 B1) VI 132
Σ
(B.3)
7)
Vt
2m1 1=1
avGx
(1 (1-7)(1 (1 31) VI B2
Vi
i= t=1
av
Σ
11ghT:ll2
Σ
(1-7) 31) V1 - 32
ay Go (1 + log 7')
(1 7) - 31) V1 32 111.T
Similarly, with Σ J5- r the second incquality holds. The third inequality also holds by j=
Cauchy-Schwaad inequality, while the last inequality holds due to the bound on harmonic sum:
E1 l.
Thus, we finally get
at ( 31.)
Q, m: |
+ B1c Q me |']}
(B.4)
a a(1+ 31) Voo (1 + log T
d
191.T,1122
2 (1 - 82 VT 32
B.3 PROOF OF LEMMA 3
Lemma 3. Under the conditions in Theorem 7 we have
T | (0. -8)1 o (0.+1 - -00
2)
D2
d
Σ 1=1
<
Σ
v (Dr, + A)T.
2a, (1- - 31c)
2a (1 (1)
i=1
14

---

Under review as a conference paper at ICLR 2020
Proof. Consider
Q (0, - - (0c+1 - -00) 03)13
204 (1 - B1e)
T
01,3 +
[0c+1. -0,)² (0z,. - -0:)²
20, (1 B1)
[=1 1=1
Vo, + A (01.5 -0,)²
Vo, +A
V
(0e.s - 07
2(1 B1)
at
0r-1
1=2
D2
01.1 A
T
d
U4,5 A
0 A
2(1- (1)
01
0t-1
1=1
D2
UT. + +
2(1 - B1)
O/T
1=1
D2
(ØT,a + A)T,
20
31)
(B.5)
where the first inequality based on 3ic S 31, and the second inequality is due the definition of a bounded feasible set.
B.4 PROOF OF LEMMA
Lemma 4, Under the conditions in Theorem 7 we have
T
31,
0 (0, -00 07)
D2
T
d
B1
(0,
<
A. 2 B1)
+
20r (1 Bie)
[=1
Proof. Consider
811
Q (0. -00)
20h (1 B1t)
1=1
31c v 0e,i + A
(Be.s 8;)² 20; (1 - -B1)
(B.6)
D
T d
Bir
Uf. +A,
2
21
31
1=1
where the first inequality is based on 31 31 and the non-decreasing property of or. and the second inequality is due to the definition of bounded feasible set.
C EXPERIMENTAL SETTINGS
C.1 CIFAR CLASSIFICATION
31 and 32. We conduct experiments in (81,82) = {00.9, 0.99), 0.999). For ADAM and AMSGRAD, we use (0.9, 0.99); while for SGDM, PADAM and ADAPLUS, we use (0.9, 0.999).
Learning Rate. For ADAM and AMSGRAD, do grid search in {0.001, 0.0001). For SGDM, PADAM and ADAPLUS, we do grid search in (0.05, 0.1, 0.2, 0.3), and most opti- mal results using Ir = 0.1.
15

---

Under review as a conference paper at ICLR 2020
Learning Rate Schedule. We conduct a learning rate schedule of decaying the learning rate by 0.1 at the 100th and 150th epoches all 200 epoches.
Off'st.. For ADAPLUS, do grid search in A
1/2] to get best results.
Weight Decaying. For ADAM and AMSGRAD, we set wwight-deeaaaeeeeeeeeeeeeeee decaying = le - 4; while for SGDM, PADAM and ADAPLUS, we set weight weight-decaying
5e
Best Setting, For ADAPLUS, we fine-tune the hyper-parameters and achieve best results when Ir 0.1, 1/8 in VGGNet--iffrll0 ResNet-Cifarloe and DenseNet-Cifal
Ir 0.2, = 1/2 in CUGNet-Cifarlue = 0.1, 1/4 in BesNet-Cifar and
Ir 0.3, À 1/2 in DenseNet-Cifar10)
C.2 NEURAL MACHINE TRANSLATION
31 and 32. We conduct experiments in (81,88) {(0.9. 0.99), (0.9, 0.999)}. For ADAM and AMSGRAD, we use (0.9, 0.99); while for SGDM, PADAM and ADAPLUS, we use (0.9, 0.999).
Learning Rate. For ADAM and AMSGRAD, we do grid search in {0.001, 0.0001 we searched learning rates in {0.1, 0.5, 2.0) for SGD, and get the optimal result when Ir 1.0.
Learning Rate Schedule. We conduct a learning rate scheme luong234, which means after 2/3 num train steps, we start halving the learning rate for 4 times in all 12000 steps.
Offset. For ADAPLUS, we do grid search A (1/2, 1,2] to get best results.
Other Details. We keep all other details the same with the benchmark(LLunng et al. 2017 there also a json file in our codes to demonstrate parameter settings.
Best Setting, For ADAPLUS, we achieve the best result when Ir 2.0, A 2.0 or 1.0, A 1.0.
C.3 ADDITIONAL EXPERIMENTS
31 and 82. We conduct experiments in (8),32) = {0.99 0.99), (0.9, 0.999) For ADAM and AMSGRAD, we use (0.9, 0.99); while for SGDM, PADAM and ADAPLUS, we use (0.9, 0.999).
Learning Rate. For ADAM and AMSGRAD, do grid search in {0.001, 0.0001}. searched learning rates in {0.1, 0.5, 1.0, 2.0, for SGD, and get the optimal result when Ir 1.0.
Learning Rate Schedule. We conduct a learning rate scheme self. which means after 8000 train steps, we start halving the learning rate every 1000 steps within 15000 steps of training.
Offset. For ADAPLUS, we do grid search A (1/8,1/4.1/2) to get best results.
Other Details. We keep all other details the same with ADASHIFT et al. 2019)
Best Setting. For ADAPLUS, achieve the best result when 0.5, A 0.125.
D ADDITIONAL EXPERIMENTS IN NMT
We also conduct another NMT experiment on IWSLT15. Unlike Section 5.3 we apply different learning rate schedule, extend the total number of training steps to 15000 steps, and change the task into Deetnam-to-English We conduct learning rate scheme self, which means after 8000 train steps, we start halving the learning rate every 1000 steps within the total 15000 steps of training. We have fine-tuned different optimizers and compare their best performance, where the results of ADAM and AMSGRAD are similar that in (Zhou et al. 2019). The experimental results still show that ADAPLUS has significant advantages over other algorithms, both in terms of convergence speed and final performance.
16

---

Under review as a conference paper at ICLR 2020
AdaPlus
25.0
SGD
Training Perplexity
Adam AMSGrad
20.0
Test BLEU
60
17.5
12.5
AdaPlus
SGD
18.0
Adam
AMSGrad
2000
4200
5000
5000
10000
12203
14205
2000
4000
6000
5000
10000
12203
14003
Steps
Steps
(a) Training perplexity
(b) Test BLEU
Figure 5: Training perplexity and test BLEU on NMT.
Table 3: Best BLEU for 15k steps IWSLT15 Vietnam-to-Englian
Optimizer
SGD
ADAM
AMSGRAD
ADAPLUS
Best BLEU
25.02
21.68
18.92
25.35
17


Reviews:
[{"review_id": "SygSmKDatB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a general framework for adaptive algorithms, and presents a specific form: ADAPLUS. In the theory part, this work gives convergence analysis of ADAPLUS. For experiments, this work analyzes several algorithm's empirical performances including SGDM, ADAM, AMSGRAD, PADAM, ADAPLUS on CV and NLP tasks. \n\n1. This paper's analysis is not solid enough to support its claim. Since this paper gives a general framework and claims that offset term can achieve superior performance, it is better to give the convergence analysis of general algorithm in the framework and discuss the benefit of the offset term theoretically. Actually, the author gives almost the same theoretical result as ADAM type algorithms, from which I did not see the advantage of using ADAPLUS. \n\n2. And the experiment shows that ADAPLUS performs on par with PADAM on CV task. \nI wonder why the authors did not give the experimental result of PADAM on NLP task? \n\n3.The notation in this paper is quite confusing. In page 2 definition (2.1), v is a vector so the L^p norm of g is actually a vector. But in Figure on page 2, the paper treat it as a scalar and I cannot understand it without further explanation from the authors.\n\nTherefore, I think this work doesn't make enough contribution and the novelty is not enough for ICLR standard.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gUbdrTFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a modification to the ADAM optimizer by introducing an adjustment function, which consists of a square root function and an extra parameter delta. Although the modification is very simple and easy to implement, the theoretical analysis is weak and the empirical performances of the proposed method are similar to the previous adaptive methods.   \n\nHere are my main concerns of the current paper:\n1. The presentation is a little bit confusing in the motivation section. According to equation (2.1), the $L^p$ norm of $\\mathbf{g}_t$ is defined as a vector. However, it seems that the authors treat $\\|\\mathbf{g}_t\\|_p$ as a scalar when presenting the intuition of the proposed method in Figure 1.\n\n2. I do not understand why Figure 1 says the proposed method is better than Padam. It seems to me that if Padam choose a specific p, it can recover the proposed $\\Phi$ function, or even better than the proposed method. Therefore, I do not think the intuition of the proposed method is correct.\n\n3. For the convergence analysis, the authors only consider the convex setting, which I think is meaningless. Because the proposed method is designed for training neural networks, such convergence guarantee in convex setting is not enough. There exist some work such as [1] have proved the convergence guarantee of the adaptive algorithms including Padam in the nonconvex setting. \n\n4. There is one missing baseline Yogi [2] in the current paper.\n\n5. For experimental results, the performance of the proposed method is very similar to the Padam. Due to the close formulation of the proposed method and Padam, it seems to me that the proposed method is  just a more careful hyperparameter tuning process.\n\n6. In NMT experiments, why there is no Padam baseline? In addition, the authors should also report the test perplexity to validate the generalization performance of the proposed optimizer.\n\n7. To fully evaluate the performance of the proposed method, the authors should at least conduct an experiment on the task of language model.\n\nMinor comments:\nThere is an unknown citation in section 5.1.\n\nReference:\n[1]. Zhou, Dongruo, et al. \"On the convergence of adaptive gradient methods for nonconvex optimization.\" arXiv preprint arXiv:1808.05671 (2018).\n[2]. Zaheer, Manzil, et al. \"Adaptive methods for nonconvex optimization.\" Advances in Neural Information Processing Systems. 2018.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g4Pa9zcS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new variation on adaptive learning rate algorithm that builds on a prior work Padam, which is also a concurrent submission to this conference. The method is empirically validated through through various domains and neural networks architecture. Though the empirical results are extensive, I am leaning towards reject because (1) The method is a very small variation; no insight is provided for why. (2) The theory is not very useful in justifying the method. (3) The empirical results are weak.\n\n(1) This work builds on the prior work Padam, which is not a well justified algorithm. Same criticisms hence can be made to this paper. I don't see any strong arguments from the paper that well justifies the methodology either.\nThe problems are listed below:\n\nHere's what I can find of the authors' interpretation of the prior work Padam: \n\n\"The internal cause is that a concave function is applied rather than the linear function in ADAM.\nOnce ε is extremely small and |g| ∈ (0, ε), the mapping value of Φ(·) would be much\nlarger in PADAM than in ADAM; therefore, PADAM can adapt to larger learning rate α,\nthus flexibly adapting to the variable learning rate scheme.\"\n\n(i) I don't think phrasing the cause as a result of concavity gives arise to any new insights. Simply, what this is trying to say is that Φ(·) is large when |g| is small, that's why PADAM can adapt to large learning rate. \n\nBuilding on Padam, the paper further justifies the proposed method by:\n\n\" This form of Φ(·) not only directly inherits advantages of PADAM,\nas is depicted in Figure 1, but also makes a better guarantee for larger learning rates. The\noffset ∆ makes sure that Φ(·) can altogether avoid the extreme situation. Even when\n|g| → 0, a more extensive learning rate αt is allowed. \"\n\n(ii) The paper inherits a serious problem from Padam, that is to assume large learning rate is important for learning rate decay. Padam didn't explain this, nor does this paper. So the justification is very weak. \n(iii) The introduction of the offset is not novel. Just as the author noted, this is almost the same term epsilon in original updates. It's not rare that one also tunes epsilon for some optimization problems.\n\n(2) Continuing the last point, the theoretical analysis focus on the convergence proof of the algorithm. The proof is not new. Theory also is not useful in justifying the method.\n\n(3) The method seems not to be able to beat the previous baseline Padam, which makes it questionable as a practical algorithm. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors proposed a general framework, which uses an explicit function as an adjustment to the actual learning rate, and presented a more adaptive specific form Ada+. Based on this framework, they analyzed various behaviors brought by different types of the function. Empirical experiments on benchmarks demonstrate better performance than some baseline algorithms. The main concern of this paper is: (1) lack of justification or interpretation for the proposed framework; (2) the performance of the proposed algorithm is on a par with Padam; (3) missing comparison with some other baselines on more benchmark datasets. Plus, the authors did not submit response.  I agree with the reviewers’ evaluation.
Decision: Reject