Training Deep Neural Networks with Partially Adaptive Momentum
Authors: Jinghui Chen
Dongruo Zhou
Yiqi Tang
Ziyan Yang
Yuan Cao
Quanquan Gu
Venue:  (2020)

Abstract: Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes ``over adapted''. We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter $p$, to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to a stationary point in the stochastic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.

Under review as a conference paper at ICLR 2020
TRAINING DEEP NEURAL NETWORKS WITH PAR- TIALLY ADAPTIVE MOMENTUM
Anonymous authors Paper under double-blind review
ABSTRACT
Adaptive gradient methods, which adopt historical gradient information to auto- matically adjust the learning rate, despite the nice property of fast convergence, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method, which unifies the Adam/Amsgrad with SGD by introducing a partial adaptive parameter p. to achieve the best from both worlds. We also prove the convergence rate of our proposed algorithm to stationary point in the stochas- tic nonconvex optimization setting. Experiments on standard benchmarks show that our proposed algorithm can maintain fast convergence rate as Adam/ Amsgrad while generalizing as well as SGD in training deep neural networks. These re- sults would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.
INTRODUCTION
Stochastic gradient descent (SGD) is now one of the most dominant approaches for training deep neural networks (Goodfellow et al., 2016). In each iteration, SGD only performs one parameter update on a mini-batch of training examples. SGD is simple and has been proved to be efficient, especially for tasks on large datasets. In recent years, adaptive variants of SGD have emerged and shown their successes for their convenient automatic learning rate adjustment mechanism. Adagrad (Duchi al. 201T is probably the first along this line of research, and significantly outperforms vanilla SGD in the sparse gradient scenario. Despite the first success, Adagrad was later found to demonstrate degraded performance especially in cases where the loss function is nonconvex or the gradient is dense. Many variants of Adagrad, such as RMSprop (Hinton et al. 2012). Adam (Kingma & Ba 2015), Adadelta (Zeiler 2012), Nadam (Dozat 2016), were then proposed to address these challenges by adopting exponential moving average rather than the arithmetic average used in Adagrad. This change largely mitigates the rapid decay of learning rate in Adagrad and hence makes this family of algorithms, especially Adam, particularly popular on various tasks. Recently, has also been observed (Reddi et al.] (018)) that Adam does not converge in some settings where rarely encountered large gradient information quickly dies out due the "short momery" problem of exponential moving average. To address this issue, Amsgrad (Reddi et al. 2018)) has been proposed keep an extra "long term memory" variable to preserve the past gradient information and correct the potential convergence issue in Adam. There are also some other variants of adaptive gradient method such as SC-Adagrad SC-RMSprop (Mukkamala & Hein 2017). which derives logarithmic regret bounds for strongly convex functions.
On the other hand, people recently found that for largely over- pee-parameterized neural networks, e.g., more complex modern convolutional neural network (CNN) architectures such as VGGNet (He) ct al. 2016),, ResNet (He et al.) 2016), Wide ResNet (Zagoruyko & Komodakis) 2016). DenseNet (Huang et al. 2017), training with Adam or its variants typically generalizes worse than SGD with momentum, even when the training performance is better. In particular, people found that carefully- tuned SGD, combined with proper momentum, weight decay and appropriate learning rate decay schedules, can significantly outperform adaptive gradient algorithms eventually (Wilson et al.] 2017). As result, even though adaptive gradient methods are relatively easy to tune and converge

---

Under review as a conference paper at ICLR 2020
faster at the early stage, recent advances in designing neural network structures are all reporting their performances by training their models with SGD-momentum (He et al., 2016: Zagoruyko & Komodakis 2016 Huang et al. 2017 [Simonyan & Zisserman 2014 Ren et al. 2015 Xie et al. 2017| Howard et al. 2017). Different from SGD,, which adopts a universal learning rate for all coordinates, the effective learning rate of adaptive gradient methods, i.e., the universal base learning rate divided by the second order moment term, is different for different coordinates. Due to the normalization of the second order moment, some coordinates will have very large effective learning rates. To alleviate this problem, one usually chooses a smaller base learning rate for adaptive gradi- ent methods than SGD with momentum. This makes the learning rate decay schedule less effective when applied to adaptive gradient methods, since a much smaller base learning rate will soon di- minish after several rounds of decay. We refer to the above phenomenon as the "small learning rate dilemma" (see more details in Section B
With all these observations, natural question is:
Can we take the best from both Adam and SGD-momentum, i.e., design an algorithm that not only enjoys the fast convergence rate as Adam, but also generalizes as well as SGD-momentum?
In this paper, we answer this question affirmatively. We close the generalization gap of adaptive gradient methods by proposing a new algorithm, called partially adaptive momentum estimation (Padam) method, which unifies Adam/Amsgrad with SGD-momentum to achieve the best of both worlds, by a partially adaptive parameter. The intuition behind our algorithm is: by controlling the degree of adaptiveness, the base learning rate in Padam does not need to be small other adaptive gradient methods. Therefore, it can maintain a larger learning rate while preventing the gradient explosion. We note that there exist several studies (Zaheer et al.] 2018 Loshchilov & Hutter 2019 Luo et al. 2019) that also attempted to address the same research question. In detail, Yogi (Zaheer et al.]| (2018) studied the effect of adaptive denominator constant e and minibatch size in the convergence of adaptive gradient methods. Adam W (Loshchilov & Hutter 2019) proposed to fix the weight decay regularization in Adam by decoupling the weight decay from the gradient update and this improves the generalization performance of Adam. AdaBound (Luo et al. (219)) applies dynamic bound of learning rate on Adam and make them smoothly converge to a constant final step size as in SGD. Our algorithm is very different from Yogi, AdamW and AdaBound. Padam is built upon a simple modification of Adam without extra complicated algorithmic design and it comes with rigorous convergence guarantee in the nonconvex stochastic optimization setting.
We highlight the main contributions of our work as follows:
We propose novel and simple algorithm Padam with partially adaptive parameter, which re- solves the "small learning rate dilemma" for adaptive gradient methods and allows for faster con- vergence, hence closing the gap of generalization.
We provide a convergence guarantee for Padam nonconvex optimization. Specifically, we prove that the convergence rate of Padam stationary point for stochastic nonconvex optimization is
E d
1/2
d T
(1.1)
T3/4
where ......, gr are the stochastic gradients and gi:ra = 191,s s ......... gr tie gradients are &oo bounded, [I.I] matches the convergence rate of SGD in terms of the rate of T.
When the stochas-
................................
We also provide thorough experiments about our proposed Padam method training modern deep neural architectures. We empirically show that Padam achieves the fastest convergence speed while generalizing well SGD with momentum. These results suggest that practitioners should pick up adaptive gradient methods once again for faster training of deep neural networks.
Last but not least, compared with the recent work adaptive gradient methods, such Yogi Zaheer et al.] 2018), AdamW (Loshchilov & Hutter 2019). AdaBound (Luo et al., 2019). our proposed Padam achieves better generalization performance than these methods in our experiments.
1.1 ADDITIONAL RELATED WORK
Here review additional related work that is not covered before. Zhang al. (011)) proposed a normalized direction direction-preserving Adam (ND-Adam). which changes the adaptive terms from in- dividual dimensions to the whole gradient vector. Keskar & Socher (2017) proposed improve
2

---

Under review as a conference paper at ICLR 2020
the generalization performance by switching from Adam to SGD. On the other hand, despite the great successes of adaptive gradient methods for training deep neural networks, the convergence guarantees for these algorithms are still understudied. Most convergence analyses of adaptive gra- dient methods are restricted to online convex optimization (Duchi et al. 2011 Kingma & Ba 2015) Mukkamala & Hein 2017, Reddi al., 2018). A few recent attempts have been made lyze adaptive gradient methods for stochastic nonconvex optimization. More specifically, [asu et al. (2018) proved the convergence rate RMSProp and Adam when using deterministic gradient rather than stochastic gradient. Li & Orabona (0118) analyzed convergence rate of AdaGrad under both convex and nonconvex settings but did not consider more complicated Adam-type algorithms. |Ward al. (2018) also proved the convergence rate of AdaGrad under both convex and nonconvex settings without considering the effect of stochastic momentum. Chen et al. (2018) provided a convergence analysis for a class of Adam-type algorithms for nonconvex optimization. Zou & Shen (2018) ana- lyzed the convergence rate of AdaHB and AdaNAG, two modified version of AdaGrad with the use of momentum. However, none of these results are directly applicable to Padam. Our convergence analysis in Section is is quite general and implies the convergence rate of Adam/AMSGrad for non- convex optimization. In terms of learning rate decay schedule, Wu et al. (2018) studied the learning rate schedule via short-horizon bias. Xu et al. (2016); Davis et al. (2019) analyzed the convergence of stochastic algorithms with geometric learning rate decay. Ge al. (2019) studied the learning rate schedule for quadratic functions.
Notation: Scalars are denoted by lower case letters, vectors by lower case bold face letters, and matrices by upper case bold face letters. For vector E Rdd, we denote the fg norm of x by
||x|| = - V 22,, the foo norm of x by ||x||0 = max |zs//. For sequence of vectors {x } we denote by Jyys the i-th element in Xj. We also denote XL.t,i [ri.::..., xt,1'" With slightly abuse of notation, for two vectors and b, we denote a2 as the element-wise square, a" as the element-wist power operation, a/b as the element-wiss division and max(a, b) as the element- wise maximum. We denote by diag(a) diagonal matrix with diagonal entries di,... ad. Given two sequences } and [bn}. J. we write an O(b,) if there exists positive constant C such that + co. Notation Ö(-) hides logarithmic factors.
an S Cb2 and on o(b,)) if on +
2 REVIEW OF ADAPTIVE GRADIENT METHODS
Various adaptive gradient methods have been proposed in order to achieve better performance on various stochastic optimization tasks. Adagrad (Duchi et al., (01I) among the first methods with adaptive learning rate for each individual dimension, which motivates the research on adaptive gradient methods the machine learning community. In detail, Adagrad adopts the following update form:
X X₁ Ole
g: /ve
where V, = 7 1=1
Σ gj
(Adagrad)
where stands for the stochastic gradient V (xx:, and ag = a/ a/viisi is the step size. In this paper, we call 0g base learning rate, which the same for all coordinates of xt, and we call as/ / ve, effective learning rate for the i-th coordinate of x,, which varies across the coordinates. Adagrad is proved to enjoy a huge gain in terms of convergence especially in sparse gradient situations. Empir- ical studies also show a performance gain even for non-sparse gradient settings. RMSprop (Hinton) eet al., 2012) follows the idea of adaptive learning rate and it changes the arithmetic averages used for Adagrad to exponential moving averages. Even though RMSprop is an empirical method with no thcoretical guarantee, the outstanding empirical performance of RMSprop raised people's interests in exponential moving average variants of Adagrad. Adam (Kingma & Ba 2015 is the most popular exponential moving average variant of Adagrad. It combines the idea of RMSprop and momentum acceleration, and takes the following update form:
X(+1 = X; Ci
where m, -,m,-1 1 (1 31)g V B2V + (1 32) (Adam)
1 'The formula equivalent to the one from the original paper Duuhi et al. 2011) after simple manipulations. Heee for simplicity and consistency, we ignore the bias correction step the original paper of Adam. Yet adding the bias correction step will not affect the argument the paper.
3

---

Under review as a conference paper at ICLR 2020
Adam also requires 0t alvt f for the sake of convergence analysis. In practice, any decaying step size or even constant step size works well for Adam. Note that if we choose 81 = 0, Adam basically reduces to RMSprop. Reddi et al. (2018) identified a non-convergence issue in Adam. Specifically, Adam does not collect long-term memory of past gradients and therefore the effective learning rate could be increasing in some cases. They proposed modified algorithm namely Amsgrad. More specifically, Amsgrad adopts an additional step to ensure the decay of the effective learning rate at/ Vv, and its key update formula is as follows:
X++1 = X
m
where V, = max (Vr-1,
(Amsgrad)
m. and V; are the same as Adam. By introducing the V. term, Reddi et al. 2018) corrected some mistakes in the original proof of Adam and proved an O(1/vII /T) convergence rate of Amsgrad for convex optimization. Note that all the theoretical guarantees on adaptive gradient methods (Adagrad, Adam, Amsgrad) are only proved for convex functions.
3 THE PROPOSED ALGORITHM
In this section, we propose new algorithm for bridging the generalization gap for adaptive gradient methods. Specifically, we introduce a partial adaptive parameter p to control the level of adaptive- ness of the optimization procedure. The proposed algorithm displayed in Algorithm |
Algorithm Partially adaptive momentum estimation method (Padam)
input: initial point E X:; step sizes (av)): adaptive parameters B1, B2, P E 1/2]
set mg = 0, Vo = 0, Vo = 0
for t .....,TT do
g: = V)((x,E))
m, -1 + (1 31 31
V. = /2V,-1 + +(1 82)g
V. mnxx(V.-11 Vt)
Xt+1 X₁ 0g m
end for
Output: Choose Xout from {x with probability au-1/(E as)
In Algorithm gi denotes the stochastic gradient and V, can be seen as a moving average over the second order moment of the stochastic gradients. As we can see from Algorithm D the key difference between Padam and Amsgrad (Reddi et al.] (018)) is that: while m, is still the momentum as in Adam/Amsgrad, is now "partially adapted" by the second order moment, i.c.,
X = Xx at m,
where v, = max(Vo-1. Vt).
(Padam)
We call E [0. 1/2) the partially adaptive parameter. Note that 1/2 is the largest possible value for p and larger p will result in non-convergence in the proof (see the proof details the sup- plementary materials). When + O, Algorithm [ reduces to SGD with momentun and when p = 1/2, Algorithm is exactly Amsgrad. Therefore, Padam indeed unifies Amsgrad and SGD with momentum.
With the notations defined above, we are able to formally explain the "small learning rate dilemma". In order to make things clear, we first emphasize the relationship between adaptiveness and learning rate decay. We refer the actual learning rate applied to m, as the effective learning rate, i.e., 0g v Now suppose that a learning rate decay schedule is applied to 0g.. If p is large, then early stages, the effective learning rate az/Of, could be fairly large for certain coordinates with small Dr.i value To prevent those coordinates from overshooting we need to enforce smaller Or, and therefore the base learning rate must be set small (Keskar & Socher) 2017] Wilson et al.) 2017). As result, after several rounds of decaying, the learning rates of the adaptive gradient methods are too small
Thhe only difference hetween Padam with p 0 and SGD-Momentum is extra constant factor (1- 1-11, which can moved into the learning rate such that the update rules for these two algorithms are identical. "The coordinate Tr 's are much less than 1 for most commonly used network architectures. See Figure in the supplementary materials.
4

---

Under review as a conference paper at ICLR 2020
to make any significant progress in the training proces: We call this phenomenon "small learn- ing rate dilemma". It is also easy to see that the larger is, the more severe "small learning rate dilemmaa" is. This suggests that intuitively, parameter p. and choosing < 1/2 can potentially make Padam suffer less from the "small learning rate dilemma" than Amsgrad, which justifies the range of p in Algorithm [ We will show in our experiments (Section 5) that Padam with < 1/2 can adopt an equally large base learning rate as SGD with momentum.
should consider using Padam with a proper adaptive
Note that even though in Algorithm [ the choice of ar, covers different choices of learning rate decay schedule, the main focus of this paper is not about finding the best learning rate decay schedule, but designing new algorithm to control the adaptiveness for better empirical generalization result. In other words, our focus is not on ar, but on V.. For this reason, we simply fix the learning rate decay schedule for all methods in the experiments to provide a fair comparison for different methods.
Figure [ shows the comparison of test error performances under the different partial adaptive parame- ter p for ResNet both CIFAR- 10 and CIFAR-100 datasets. We can observe that larger p will lead to fast convergence at early stages and worse generalization per- formance later, while a smaller be- haves more like SGD with momen- tum: slow in early stages but finally catch up. With proper choice of p (e.g., 1/8 in this case), Padam can obtain the best of both worlds. Note that besides Algorithm T. our partially adaptive idea can also be applied to other adaptive gradient methods such as Adagrad, Adadelta, RMSprop, AdaMax (Kingma) & Ba 2013). For the sake of conciseness, we do not list the partially adaptive versions for other adaptive gradient methods here. We also would like to comment that Padam is totally different from the p-norm generalized version of Adam in Kingma & Ba (015)). which induces AdaMax method when p 0o. In their case, p-norm is used generalize 2- 2-norm of their current and past gradients while keeping the scale of adaptation unchanged. In sharp contrast, we intentionally change (reduce) the scale of the adaptive term in Padam to get better generalization performance. Finally, note that in Algorithm[]] we remove the bias correction step used in the original Adam paper following Reddi al. (2018). Nevertheless, our arguments and theory are applicable the bias correction version as well.
p
p 1/4
645-
Test
180
Ldo
WEpochs
#Epochs
(a) CIFAR-10
(b) CIFAR-100
Figure 1:
Performance comparison of Padam with different choices of p for training ResNet on CIFAR-10 CIFAR-100)
4 CONVERGENCE ANALYSIS OF THE PROPOSED ALGORITHM
In this section, we establish the convergence theory of Algorithm | in the stochastic nonconvex op- timization setting, i.e., we aim at solving the following stochastic nonconvex optimization problem
min [(x) = Egg [x):
X
where& is random variable satisfying certain distribution, f(x: Rdd + R is L-smooth nonconvex function. In the stochastic setting, one cannot directly access the full gradient of fxx). Instead, one can only get unbiased estimators of the gradient of fxx,, which is (); This setting has been studied in Ghadimi & Lan (2013) 2016). We first introduce the following assumptions.
Assumption 4.1 (Bounded Gradient). fxx = Exf(x; has -bundedd stochastic gradient. That is, for any &. we assume that ||V f(: E) G.o.
It is worth mentioning that Assumption 4.1 is slightly weaker than the Ey-boundedness assump- tion /V 2)1l2 < G2 used in Reddi et al. (2016); Chen et al. (2018). Since [ E S || [x: 122 < Vdl f olo, the fg-boundedness assumption implies Assumption 4.1 with
*This does not mean the learning rate decay schedule weakens adaptive gradient method, On the contrary. applying the learning rate decay schedule still gives performance boost to the adaptive gradient methods in general but this performance boost is not significant SGD momentum.
5

---

Under review as a conference paper at ICLR 2020
Go G2. Meanwhile, Go will be tighter than G2 by factor of vd when each coordinate of ff(x; {) almost equals to each other.
Assumption 4.2 (L-smooth). f(x) = fx; {) is L-smooth: for any x.y Rd. it satisfied that [f(x - f(y) (Vf(y), y) < 2/2 ||x y
Assumption 4.2] is frequently used analysis of gradient-based algorithms. is equivalent to the L L-gradient Lipschitz condition, which is often written as [ - Vfly)l2 s L||x yll2. Next we provide the main convergence rate result for our proposed algorithm.
Theorem 4.3 (Padam). Algorithm suppose that E [o, 1/2), 31 < 3 and at = a for t = 1...., T, under Assumptions 4.1 and 4.2 let Aʃ- f(x1) infx fxx), for any g (ma 4p
1), 1], the output X of Algorithm satisfies that
E (
[2]
M1
M2d
Maad
Σ (Brrr 1-a
(4.1)
Ta
T
T(1-01/2
where
2 + 10 M3 (1 B2)ap
4LG
SLG B1)
B1
Mi = Af, M2
d(1 31)
(1- 22) -1- 3/
31
4.3 we can see that M1 and M3 are independent of the number of iterations 7' and dimension d. In addition, if | 11o0 O1), it is easy to see that M2 also has an upper bound that is independent of T and d.
Remark 4.4. From Theorem
The following corollary is special case of Theorem 4.3
when E [0, 1/4] and q 0.
Corollary 4.5. Under the same conditions in Theorem 4.3 if p E [0,1/4], Padam's output satisfies
B f(
M1 Mad
M5a
E
(4.2)
Ta
T
where M1 and M2 and A are the same as in Theorem 4.3 and Mý is defined as follows:
ALG
SLG 21 B1)
(1 8
1- )
M/ -2) (1 - (1 /,""")
(
Remark 4.6. Corollary
4.5 simplifies the result of Theorem 4.3 by choosing g = 0 under the
condition p E [0, 1/4] We remark that this choice of q is optimal in important special case
studied in Duchi et al. (201]: Reddi al. (2018): when the gradient vectors are sparse, we assume
that Σ gg.r, < VdT. Then for > 0, it follows that
d (2 11 |BTT,
(4.3)
T1-9/2
4.3) implies that the upper bound provided by 442 strictly better than 4.1 with g >> There- fore when the gradient vectors are sparse, Padam achieves faster convergence when E [0, 1/4]. Remark 4.7. We show the convergence rate under different choices of step size If
= (T
1 1/2
then by 4.2
have
112] o (2 II (2) 1/2
I3
(4.4)
73/4
Note that the convergence rate given by (4.4) is related to the sum of gradient norms Di= 1 As mentioned in Remark 4.6 when the stochastic gradients gI.T,, 1,..., d are sparse,
have Din |g1.T.1112 & VdT Ducci al. (201 1)). More specifically, suppose Σd= |lg = (dd/ /2222//r222////// + d/T). When 1/2, have E|JlV ( (xmm) || O(at/1 /VT + d/T), which matches the rate 0(1/VT achieved by nonconvex SGD (Ghadimi & Lan 2016), considering the dependence of T. It is worth noting that nonconvex SGD achieves Omaxx. V :) VT) O(vd/T) convergence rate under Assumption 4.1 thus the convergence rate of our algorithm is strictly better than that of SGD by factor of d1/4 when T is large.
(dd (T) for some < s
< 1/2. We have [E[| / (x)ut)

---

Under review as a conference paper at ICLR 2020
Remark 4.8. If we set 1/vT which is not related to E
||g::r;s
then 4.2)
suggests that
E V [1:] o ( +
d
T T
When E |g. < vdT Duchi al. 2011
Reddi et al.
2018 we further have
( = o
(J+
d d T 7
which matches the convergence result in nonconvex SGD dependence of T.
(Ghadimi & Lan 2016) considering the
5 EXPERIMENTS
In this section, we empirically evaluate our proposed algorithm for training various modern deep learning models and test them on several standard benchmarks o We show that for nonconvex loss functions in deep learning, our proposed algorithm still enjoys fast convergence rate, while its generalization performance is as good as SGD with momentum and much better than existing adaptive gradient methods such as Adam and Amsgrad. All experiments AWS p3.8xlarge servers which come with Intel Xeon E5 CPU and 4 NVIDIA Tesla V100 GPUs (16G RAM per GPU). All experiments are implemented in Pytorch version 0.4.1 within Python 3.6.4.
conducted on Amazon
Adam
Anngged
Amograd Adammi
Arragnad
Train
Adsei
Train Loos
Adsilunnd
Pudare
Train
Pacarr
Adailoundd
Adalbound
Fadam
e
Epochs
Eports
Epachs
(a) Train Loss for VGGNet
(b) Train Loss for ResNet
(c) Train Loss for WideResNet
635
Amognad
Aragrad
Aduert
Adbuni
Pacces
0.66
Epochs
Epocts
Epschs
(d) Test Error for VGGNet
(e) Test Error for ResNet
(f) Test Error for WideResNet
Figure 2: Train loss and test error (top-1) plots of three CNN architectures on CIFAR-10 dataset.
We compare Padam against several state-of- algorithms, including: (1) SGD-momentum, (2)
Adam (Kingma & Ba, 2015), (3) Amsgrad (Reddi et al. 2018), (4) AdamW (Loshchilov & Hutter
2019) (5) Yogi (Zaheer et al. 2018) and (6) AdaBound (Luo et al. 2019 We use several popular
datasets for image classifications and language modeling: CIFAR- 10 (Krzhevsky & Hinton 2009).
C10 (Krizhevsky & Hinton 2009), ImageNet dataset (ILSVRC2012) (Deng et al. 20099 and
Penn Treebank dataset (Marcus et al. (993). We adopt three popular CNN architectures to evaluate the performance of our proposed algorithms on image classification task: VGGNet-16 (Simonyan & Zisserman 2014). Residual Neural Network (ResNet-18) (He al. 2016). Wide Residual Network (WRN-16-4) (Zagoruyko & Komodakis 2016). We test the language modeling task using 2-layer and 3-layer Long Short-TTerm Memory (LS network (Hochreiter & Schmidhuber 1997). We test CIFAR-10, CIFAR-100 and Penn Treebank tasks for 200 epochs. For all experiments, we use fixed multi-stage learning rate decaying scheme: we decay the learning rate by 0.1 at the 100th and 150th epochs. We test ImageNet tasks for 100 epochs with similar multi-stage learning rate decaying
'The code is available in the anonymous link

---

Under review as a conference paper at ICLR 2020
scheme at the 30th, 60th and S0th epochs. We perform grid search validation set to choose the best hyper-yppraee paperr-prameters for each algorithm. Details about the datasets, CNN architectures and all the specific model parameters used in the following experiments can be found in the supplementary materials.
5.1 EXPERIMENTAL RESULTS
We compare our proposed algorithm with other baselines on training the aforementioned three modern CNN architectures for image classifica- tion on the CIFAR-I0, CIFAR-100 and also ImageNet datasets. Due to the paper length limit, we leave all experimental results regarding CIFAR- dataset and all test re- sult tables in the supplementary ma- terials. Figure 2 plots the train loss and test error (top-1 error) against training epochs on the CIFAR-10 dataset. As can see from the figure, at the early stage of the training process, (partially) adaptive gradient methods including Padam, make rapid progress lowing both the train loss and the test error, while SGD with momentum con- verges relatively slowly. After the first learning rate decaying at the 100-th epoch, different algorithms start behave differently. SGD with momentum makes huge drop while fully adaptive gradient methods (Adam and Amsgrad) start to generalize badly. Padam, on the other hand, maintains relatively good generalization performance and also holds the lead over other algorithms. After the second decaying at the 150-th epoch, Adam and Amsgrad basically lose all the power of traversing through the parameter space due the "small learning dilemma", while the performance of SGD with momentum finally catches up with Padam. AdamW, Yogi and Ad- aBound indeed improve the performance compared with original Adam but the performance still worse than Padam.
SGD
Nameeau
Adars
Adam
Ansgred
Amagidd
TostEree
Acaloand
Fadar
Text
Adallond
Epochs
Epochs
Top-1 Error, VGGNet
(b) Top-5 Error, VGGNet
Aerggiad
Ayrigrod
Eree
Text Erree
Pademn
0251
Epocts
Eportss
(c) Top-1 Error, ResNet
(d) Top-5 Error, ResNet
Figure 3: Top-1 and Top-5 test error for VGGNet and ResNet on ImageNet dataset.
Overall we can see that Padam achieves the best of both worlds (i.e., Adam and SGD with momen- tum): it maintains faster conver- gence rate while also generalizing well as SGD with momentum in the end. Figure 3 plots the Top- I and Top-5 error against training epochs on the ImageNet dataset for both VGGNet and ResNet. We can see that on ImageNet, all methods behave similarly in our CIFAR-10 experiments. Padam method again obtains the best from both worlds by achieving the fastest convergence while generalizing as well as SGD with momentum. Even though methods such AdamW, Yogi and AdaBound have better per- formance than standard Adam, they still suffer from a big generalization gap on ImageNet dataset. Note that did not conduct WideResNet experiment Imagenet dataset due to GPU memory limits.
Sepp Mamenrtur
Ain Amegral
Abbn
Text Parphotyy
Text Seeploa
Ansignud
Addioonn
Abblcund
Padaa
129
Epochs
Epochs
(a) 2-layer LSTM
(b) 3-yayer LSTM
Figure 4: Test perplexity for 2-layer and 3-layer LSTM model on Penn Treebank dataset.
We also perform experiments the language modeling tasks to test proposed algorithm on Long Short-Tern Memory (LSTM) network (Hochreiter Schmidhuber 1997). where adaptive gradient methods such as Adam are currently the mainstream optimizers for these tasks. Figure 4
8

---

Under review as a conference paper at ICLR 2020
plots the test perplexity against training epochs the Penn Treebank dataset (Marcus et al. 1993) for both 2-layer LSTM and 3-layer LSTM models. We can observe that the differences on simpler 2-layer LSTM model is not very obvious but more complicated 3-layer LSTM model, different algorithms have quite different optimizing behaviors. Even though Adam, Amsgrad and Adam W have faster convergence in the early stages, Padam achieves the best final test perplexity this language modeling task for both of our experiments.
CONCLUSIONS AND FUTURE WORK
In this paper, proposed Padam, which unifies Adam/Amsgrad with SGD-momentum. With an appropriate choice of the partially adaptive parameter, we show that Padam can achieve the best from both worlds, i.e., maintaining fast convergence rate while closing the generalization gap. We also provide a theoretical analysis towards the convergence rate of Padam to stationary point for stochastic nonconvex optimization.
While the empirical generalization performances achieved by Padam backup our hypothesis of "small learning rate dilemma", it is still unclear how learning rate decay affects the performance of adaptive gradient methods in theory. We leave it as important future direction.
REFERENCES
Amitabh Basu, Soham De, Anirbit Mukherjee, and Enayat Ullah. Convergence guarantees for rm- sprop and adam in non-convex optimization and their comparison to nesterov acceleration on autoencoders. arXiv preprint arXiv:/807.06760 2018.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for nonconvex optimization. arXiv preprint 1rXiv:1808.. 02941, 2018.
Damek Davis, Dmitriy Drusvyatskiy, and Vasileios Charisopoulos. Stochastic algorithms with geo- metric step decay converge linearly sharp functions. arXiv preprint arXiv: 10Xv:1197700957, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 248-24882 255.. leee, 2009.
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 120ul):2121-2155 2011 1.
Rong Ge, Sham M Kakade, Rahul Kidambi, and Praneeth Netrapalli. The step decay schedule: A near optimal, geometrically decaying learning rate procedure. arXiv preprint arXiv:/900 12838, 2019.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochas- tic programming. SIAM Journal Optimization, (3(4))2341-2366 2013.
Saced Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic programming. Mathematical Programming, 156(1-2):59-99, 2016.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1.
MIT' press Cambridge, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.

---

Under review as a conference paper at ICLR 2020
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04880 2017.
Gao Huang, Zhuang Liu, Kilian Q Weinberger, and Laurens van der Maaten. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. arXiv preprint arXiv: 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. arXiv preprint arXiv:/805.088 14, 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Liangehen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. In Proceedings of the 7th International Conference Learning Repre- sentations, New Orleans, Louisiana, May 2019.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. 1993.
Mahesh Chandra Mukkamala and Matthias Hein. Variants of rmsprop and adagrad with logarithmic regret bounds. International Conference on Machine Learning, pp. 2545-2553, 2017.
Sashank Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In International conference on machine learning, pp. 314 323, 2016.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference Learning Representations, 2018
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pp. 91-99, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 10Xiv:140911556 2014.
Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes, from any initialization. arXiv preprint 188. 2018
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 4151-4161 2017.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger Grosse. Understanding short-horizon bias in stochastic meta-optimization. In International Conference on Learning Representations, 2018.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987-5995. IEEE, 2017.
Yi Xu, Qihang Lin, and Tianbao Yang. Accelerate stochastic subgradient method by leveraging local growth condition. arXiv preprint arXiv: 1607.00007 2016.
Sergey Zagoruyko and Nikos Komodakis. 160.01665.0 2016.
Wide residual networks.
arXiv preprint
10

---

Under review as a conference paper at ICLR 2020
Manzil Zaheer, Sashank Reddi, Devendra Singh Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. In Advances in Neural Information Processing Systems, 2018.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint 121.5512,,. 2012.
Zijun Zhang. Lin Ma, Zongpeng Li, and Chuan Wu. Normalized direction-preserving adam. arXiv preprint arXiv: 04546, 2017.
Fangyu Zou and Li Shen. On the convergence of adagrad with momentum for training deep neural networks. arXiv preprint arXiv: /808 808.03408, 2018.

---

Under review as a conference paper at ICLR 2020
A PROOF OF THE MAIN THEORY
In this section, we provide a detailed version of proof of Theorem
4.3
A.1 PROOF OF THFOREM 4.3
Let X X1 and
B1
B1
2: = X
+
(x, 10,-1111
x1-1)
x
Xt-1:
(A.1)
1-
1 31
1 31
we have the following lemmas:
Lemma A.I. Let Ze be defined in A. For 2 2, we have
Z++1 Z, 1 31
81
(a.
V, P) (a, v,.) -
- -o.
(A.2)
and
31
Z(+1 21
(ai-1 V as V. m a4 V."."
(A.3)
1 81
For /= = 1, we have
22 - 21 = -( V gi.
(A.4)
Lemma A.2. Let 2₁ be defined in [A.1 Fort 2 we have
B1
|z++1 2112 - 0 P g: 2 +
|x,- - Xx X2
1 31
Lemma A.3. Let z, be defined in A.. Fort 2 2, have
81
[[V - V (xx))22 S T.
||x, - x1-1
B1
Lemma A.4. Let v. and m, be defined in Algorithm
| Then under Assumption 4.1 have
[V x I G / I G and ||m, |/cc I G
Lemma A.5. Suppose that f has -buunde stochastic gradient. Let 31, 32 be the weight param-
eters, a,, t = I....,T
T be the step sizes in Algorithm [ and E [max 4p 1, 0), 1]. We denote
7 31/ 3. Suppose that a and γ I then under Assumption 4.1 we have the following
two results:
T(L+a)/2(00211 - B1)
- (L+o-14
Σ
E
p m:
2] <
E
(1- (1-82)201-9)) (1-)
1=1
and
T
22 2((
|B1::slla 1-9
Σ
|
pg:
2 <
E
Σ
(1 2p
1=1
Now we are ready to prove Theorem
Proof of Theorem
4.3 Since f is L-smooth, we have:
f(z(+1) & fzz) + ff(z,)) (zt+1- - 2t) + 2 |,,+1 -z113
21) + (V)(z)) v((x))) (Ze+1 - -2)+ 1
= f(z)) + vx(x:) (21+1
2c+1 - 2el12
12
(A.5)
In the following, bound /, Ja and Ta separately.
12

---

Under review as a conference paper at ICLR 2020
Bounding term 4: When t = 1, we have
V (x(x)) (z2 -21= -VV(xi)) &.
(A.6)
For 2 2, we have
vx(x,) (2441 2x)
B1 (ac-1 1-1 - ai V,
-V((x)
m ar V. gr
"g
31
31 (x(xx) (a--1 Vi- V
m:-1 Vs(())) ViVig gr,
(A.7)
1-B1
where the first equality holds due A.3 Lemma A.1 in (A.7) we have
For Vf(x))) (1(xx:)((a00))) (ac-1
V (ac-i
aa V m
f fxx) (av-1 -a- Vi m) - (x) |a--1V
- Vi V 11,,1 11100
s 6
(aviiiilll a V. 1 1.1
- -lo,,, -
(A.8)
The first inequality holds because for a positive diagonal matrix A, we have x Ay I |x|| |A |byl The second inequality holds due to a V as V." ł O. Next we bound vx(x) T az V."g gt. We have
ff(xx) at V. 'g:
[x(xx) a Vi.g V -a- V
S -VV((x)) 01 V. + [V)(x))) V,v - 04 V |1::|>>
S -xx( 04 V.. + 6he Vi- |1, - | V. Il.1
fx)( a--1 a V. + Ghe (1a-- - 1l:
(A.9)
The first inequality holds because for a positive diagonal matrix A, we have x T Ay I
||||
||A 1,1 loo. The second inequality holds due V > at V. z Substituting (A.8)
and A.9 into (A.7). we have
fx,) (Z++1 z+) S (xx: ai V.igi + I- B
G2 110- - ||a00,
(A.10)
Bounding term 12: For t 2 1, we have
(VS(z)) - [x())) (ze+1 - 2c)
$ (f(z)) ff(x)) ||z411 - -2o112
31
31
(Ilar V.'g 0g +
||x4-11 - -xoll2)
L|[x - -x4-112
1- B1
- 1- B1
L 31
B1
|au V. p "gglls
1|
-x-xl11112 + L
||x:
x1
31
1 81
B1
2
< <|1a, V,
2L
||x: 1xx--x-111 x
(A.11)
T-
1-BBi-
where the second inequality holds because of Lemma A.I and Lemma A.2 the last inequality holds due to Young's 's inequality.
Bounding term Is: For 1, we have
B1
2 2, < マ || V
V."gill
+ 1- Bi
||x,-1 - xella
B1
V. 8g
+
11 31
||x--1 1x+
(A.12)
13

---

Under review as a conference paper at ICLR 2020
The first inequality is obtained by introducing Lemma A.1
For = 1, substituting A.6) A.I1 and (A.12) into A.5
taking expectation and rearranging
terms, we have
(2f( - f(z1)]
{1] )) ||x| x
$ vx(x)) a1 V + 2 V 1
4L (1 3,
B| vf(x1)
g1 + 21 a1 V ?
$ B|da; 1 + 21| 01 V gi ||
(A.13)
where the last inequality holds because
(x) V
g1 S d. |V((()) V gilloo S d. Gx Gh--- = =dG222
For 2 2, substituting (A.10). terms, we have
AIII and (A.12) into (A.5)
taking expectation and rearranging
E (z 1) G aa
[(zi) + G |
a
8 2111)11))
)1
11
31
B1
2
E
Vf(x)) an- Ving: + 2L 01 V g:
4L
||x
x 11
1- B1
E
VS(x)) 04 - 21
V.
1 "8
g+
4L
31
2
|00-1 V, m 1
1-
B1
<
E
V.
ge
+ 4L
31
2
||a:-1
V
m 112
1-
B1
(A.14)
where the equality holds because E[g inequality holds because of Lemma we have
(xx) conditioned on fxxx)) and V the second Telescoping (A.14) for t T and adding with (A.13).
O2 E
[x)))11
G 0 "|
(f(eTH))
PallarV7"
E <
f(Z1) +
+ da G2-20
31
1 1- B1
31
2
2L
Σ
E
V,
V
4L
at V.,mm
1-
G |an
E
A
+ day 122 2-
+ 2L
Σ
E
as
V 7 g: l2
1- 31
1=1
4L
31
T
a 1
p m 11
(A.15)
B1
(=1
By Lemma A.5
we have
7(2+0)/20000(1 - [31)G (1+g-4p)
1-9
Σ
B V
m
E
Σ
(A.16)
(1-B2) -1
where 7 3. We also have
7(1+0)/2 2 (1 4p)
d
1-9
Σ
2 8: 113
E
1gr
l12
(A.17)
)2
t=1
14

---

Under review as a conference paper at ICLR 2020
Substituting A.16 and (A.17) into (A.15), and rearranging (A.15). we have
T
a
V
Σ 1 a411
1=
G2p
G | V1
A5+
+ dan G2-2p 2p
21 0tt1
1-3 31
2LG
((1+g)/2
2 GG (1+4-4p)
d
1-9
1BITT11122
Ei= 0t-1
(1-
32)
4LG
31
2
21(++))/2d3
-1- 31) 0 410-44) 4p)
1-9
E
Σ
|BIT,IIle)
11-B) Bi
(1-32) 1-12)2p(1 -7)
1= a1-1
1=1
4
22
2G 2p
Af +
+ dG
Ta
T
1
B1
da
1-9
4LG
8LG Bi)
B1
)
+
E
BI.,,1l2))
7//-
(1 32 2p
(1 - 32)
1- B1
(A.18)
where the second inequality holds because 0g = a. Rearranging (A118). we obtain
Mi Ta
M2d
ad9 Ma
1-9 1B1.T.112))
V
(xou))
+
Σ
T
7/2-0)/2
where (M)}² are defined in Theorem
4.3 This completes the proof.
A.2 PROOF OF COROLLARY
Proof of Corollary 4.5 From Theorem
4.3 let E [0,1/4], we have E [0, 1]. Setting g = 0, we
have
(xoul) 11
M Ta
M2 d
Mía VT
E
Σ Bur.lll))
T
where Mj and M2 are defined in Theorem 4.3 and M2 is defined in Corollary the proof.
4.5 This completes
B PROOF OF TECHNICAL LEMMAS
B.1 PROOF OF LEMMA A.I
Proof. By definition, have
B1
B1
Z++1 = X +
(x+41 - x1) = 11
X++1
X4.
1 B1
1 1 31
Then we have
Bi
Zt+1 - Z; =
(xt+1 - -xi)
(xi x(-1)
1- B]
1- 31
31
1 1 31 ( (-ar 0/ V. mm)) ) +
0t-1 V m--1-
1- 31
15

---

Under review as a conference paper at ICLR 2020
The equities above based on definition. Then we have
21+1 - Z, = -a,
V.
31
3 m + (1- (1-81188 +
at-1 Vi-mm
1- B1
1 31
31
m1 a0-1 -
a V.
as V.
1- B1
31
04-1 V |-| m
I - (.. (ar v--i)
Ot Vi'g.
1-
f1
91 [ (ar
(o. V "0l Vii "
(xi-1 - x1) 01 V.V," V 'g:
1- 31
The equalities above follow by combining the like terms.
B.2 PROOF OF LEMMA A.2
Proof. By Lemma A.1
have
B1
1141
2r/12-
I I- (ax,
(x1-1- xx) 0r -i gr
1 31
B1
I-
V..
||x,-11 xz|12 +
31
00,00
where the inequality holds because the term 31/(1 31) is positive, and triangle inequality. Con-
sidering that V, S 0t VT-1. when > 0, we have
- (ax, (a. (a. S 1.
00,00
With that fact, the term above be bound as:
B1
|z - fella - 1l 112
+ 1- 31
Xx:-1 Xx Xxe12-
This completes the proof.
B.3 PROOF OF LEMMA A.3
Proof. For term (2+ [xx) we have:
[z. (x(x))l2 s L| x,l12
B
(Xt x
$
B1
B1
) -||x: -||x; xc-1
< - Br
where the last inequality holds because the term
31/(1 -8)
B1) is positive.
B.4 PROOF OF LEMMA A.4
Proof of Lemma
A.4 Since f has -oundddd stochastic gradient, for any x and &,
| [x:
fl
Goo. Thus, have
[x)
[)x; & (x; Goo-
Next bound || |la. We have
|mml 0 < G. Suppose that ||molooo I Goo. then for mi+1,
we have
= =| + 1-18))
< Billmullo + (1 (11-3i)leen 11)114110
< Goc + +(1 (1-31p) (1-31)Ga
Goo.
16

---

Under review as a conference paper at ICLR 2020
Thus, for any > 0, we have ||m || - G Finally we bound Velloe. First we have l/vollso
Volloo = 0 I 2x Suppose that Villl I G and |/vell0 S G Note that we have
1e411lo = || + (1 - -2)) 1lcc
< 821 llll Vz 11.0 + (1- 82) 12)
- 02 + (1 - -3
Vi S G2 Thus, for any t 2 0, we = C.o
and by definition, we have l/ =
have Vello G
B.5 PROOF OF LEMMA A.5
Proof. Recall that Lt.j. mejj: It.5 denote the j-th coordinate of V., m, and gr. We have
d
a IV m 12]
= af E
d
m
S a
t,i
j= (1 (1- (1- )),)²
= of E
(B.1)
E/= - 82) 92,
where the first inequality holds because 0, 2 U2 Next we have
d
1 (1 - B1) 99
a
E
(1 -30 "37,)
a1
31)²
EEi=1 81 3 19g (1+0 1p) (E)=1 1110119110000000
E
(1 (1-32)22 )2
(2) 8/ 3/-3 "5-192
i=1
a2 B,)2
2j=1 81 G(l to 4p) )E j= 87 '19,/11 (1
E
(1 )2
Ej 82 ) 07,3)20
i=1
(1 - B1 B1)G
(1+9-4p
1 (1 -t++p)
(B.2)
(1-B2) j2p
E/= 8/2 02,0
1=1
where the first inequality holds due to Cauchy inequality, the second inequality holds because
lو I Goo, the last inequality holds because E)=1 8/- S (1 B1)-1, Note that
d Ei= 1 ( 4t4
8 10gal (1 41-
Σ
(B.3)
i=1 E 93
(3) 93,320
==
i=1
j=1
where the equality holds due to the definition of y. Substituting
B.2
and
B.3
into
B.I. we have
mz a2 31 G()
(1+g 4p)
a
E
195
(B.4)
(1 B2)
j=1
Telescoping B.4 for 1 to T, we have
21
B1 G (1-
d
E
mm/ [3]]
E
1-9
(1 Pp
i=1
a1 - 81 G ((L--44p))
d
T
T
E
1-9
Σ
7-1
(1 32
)=1
a2 81 (1++--11 (1+
d Σ :=1==1
T
E
19
1-0
(B.5)
(1 Bz) 2m (1- 7)
17

---

Under review as a conference paper at ICLR 2020
Finally, we have
(1-91/2
9j
1
-2(2+4)/2
i=1
/=1
-
$
T1(10)/22
Σ
|g.T,
(B.6)
where the first and second inequalities hold due to Hölder's inequality. Substituting B.6 into B have
7(+4a)/2d002( G00 (2 7)
(2+4-4p
1-9
Σ
2
m
E
Specifically, taking B1 O, have m, -g, then
r
(2 (1+0 (1+o-44p
d
Igi lll 1-a
Σ
a B
IV
gt
Σ
(1 (1-32 2P
C EXPERIMENT DETAILS
C.I DATASETS
We use several popular datasets for image classifications.
CIFAR-I0 (Krizhevsky & Hinton 2009): it consists of a training set of 50, 000 32 x 32 color images from 10 classes, and also 10, 000 test images.
CIFAR-100 (Krizhevsky Hinton 2009 is similar to CIFAR- 10 but contains 100 image classes with 600 images for each.
2009): ILSVRC2012 contains 1.28 million training images, and 50k validation images over 1000 classes
ImageNet dataset (ILSVRC2012) (Deng et al.
In addition, we adopt Penn Treebank (PTB) dataset (Mareus et al. 1993). which is widely used in Natural Language Processing (NLP) research. Note that word-level PTB dataset does not contain capital letters, numbers, and punctuation. Models are evaluated based on the perplexity metric (lower is better).
C.2 ARCHITECTURES
We use modified VGG-16 architecture for this exper- iment. The VGG-16 network uses only 3 x 3 convolutional layers stacked on top of each other for increasing depth and adopts
VGGNet (Simonyan & Zisserman 2014):
pooling reduce volume size. Finally, two fully-connected layers] are followed by softmax classifier.
ResNet He al. 2016): Residual Neural Network (ResNet) introduces novel architecture with "skip connections' and features heavy use of batch normalization. Such skip connections are also known as gated units or gated recurrent units and have a strong similarity to recent successful elements applied in RNNs. We use ResNet-18 for this experiment, which contains 2 blocks for each type of basic convolutional building blocks in He al. (2016).
For CIFAR experiments, change the ending two fully-connected layers from 2048 nodes to 512 nodes. For ImageNet experiments, batch normalized version (vgg16 hn) provided Pytorch official package
18

---

Under review as a conference paper at ICLR 2020
Legio Scale
-3-
v min
10000 20380 36600 40030 seece 00000 70000 50000 #lterations
Figure 5:
Plot of max and min values across all coordinates of V, against the iteration number for ResNet model on CIFARIO dataset.
Wide ResNet (Zagorykoo Komodakis 2016):
Wide Residual Network further exploits the "skip connections" used in ResNet and in the meanwhile increases the width of residual networks. In detail, we use the 16 layer Wide ResNet with 4 multipliers (WRN-16-4) in our experiments.
LSTM (Hochreiter & Schmidhuber y: Long Short-Term Memory (LSTM) network is a special kind of Recurrent Neural Network (RNN), capable of learning long-term dependencies. It was introduced by Hy/Hchhriiter & Schmidhuber 1997). and was refined and popularized in many followup work.
C.3 PARAMETER SETTINGS
We perform grid searches to choose the best age classification and language modeling tasks. For the base learning rate, do grid search over {0.0001, 0.001, 0.01, 1, 10, 100} for all algorithms, and choose the partial adaptive pa- rameter p from (2/5, 1/5, 1/8, (0.9, 0.99, 0.999). Specifically, in terms of image classification experiments, for SGD with mo- mentum, the base learning rate is set to be 0.1 with a momentum parameter of 0.9. For all adaptive gradient methods except Padam, we set the base learning rate as 0.001. For AdaBound. the final learning rate is set to be 0.1. For Padam, the base learning rate is set be 0.1 and the partially adaptive parameter p is set to be 1/8 due to its best empirical performance. For Adam, Amsgrad, the momentum parameters are set to be $1 0.9, 83 0.99. And for other adaptive gradient methods including Padam, we set 31 0.9, _3 0.999. The weight decay factor is set to 5 x 10-4 for all methods except Adam which adopts a different weight decay scheme. For Adam W, the normalized weight decay factor is set to 2.5 x 10 for CIFAR and 5 x 10-2 2 for ImageNet. For Yogi, e is set as 10 as suggested in the original paper. For AdaBound, the final learning rate is set 0.1 suggested in Luo et al. (2019). The minibatch sizes for CIFAR-10 10 and CIFAR-100 are set to be 128 and for ImageNet dataset we set it to be 256. Regarding the L.STM experiments, for SGD with momentum, the base learning rate is set to be 1 for 2-layer LSTTM model and 10 for 3-layer LSTM. The momentum parameter is set to be 0.9 for both models. For all adaptive gradient methods except Padam and Yogi, set the base learning rate as 0.001. For Yogi, we set the base learning rate as 0.01 for 2-layer LSTM model and 0.1 for 3-layer LSTM model. For Padam, we the base learning rate as 0.01 for 2-layer LSTM model and for 3-layer LSTM model. For all adaptive gradient methods, we set 3 = 0.9, B2 0.999. In terms of algorithm specific parameters, for Padam, we set the partially adaptive parameter p as 0.4 for 2-layer LSTM model and 0.2 for 3-layer LSTM model. For AdaBound, we set the final learning rate as 10 for 2-layer LSTM model and 100 for 3-layer LSTM model. For Yogi, e is set as 10-3 3 as suggested in the original paper. The weight decay factor is set to 1.2 x 10-6 for all methods except AdamW W which adopts different weight decay scheme. For A, the normalized weight decay factor is set to 4 x 10-4, The minibatch size is set be 20 for all LSTM experiments.
for all algorithms in both im-
and the second order moment parameter |8 from
D ADDITIONAL EXPERIMENTS
As a sanity check, we first plot the max and min of v, across all the coordinates in Figure detail, the maximum value of V, around 0.04 in the end and the minimum value is around 3 10-8
19

---

Under review as a conference paper at ICLR 2020
max (
min (v.)) (in(())); coordinates are quite different.
0.55. We see that the effective learning rates for different
SGD Homartum
Namantum
Train Loss
Yog!
AdMeWW
Vay
Acudbound
piptoord
Padam
Padam
Padami
Epochs
Epochs
Epochs
(a) Train Loss
for VGGNet
(b) Train Loss for ResNet
(c) Train Loss for WideResNet
0.8
Nomantum
SCD Semmriam
Vementam
07
Adart
Adam
0.59
Amigrad
Amiggdd
Test brror
0:4
Adamii
Test Error
Adslourd
Padam
Podam
Padaar
3.20
0.2-
0.29
Epochs
Epochs
Epochs
(d) Test Error for VGGNet
(c) Test Error for ResNet
(f) Test Error for WideResNet
Figure 6: Train loss and error (top-1 error) of three CNN architectures on CIFAR-100. In all cases, Padam achieves the fastest training procedure among all methods and generalizes as well as SGD with momentum.
Figure 6 plots the train loss and test error (top-1 error) against training epochs on the CIFAR-100 dataset. We can that Padam achieves the best from both worlds by maintaining faster conver- gence rate while also generalizing as well as SGD with momentum in the end.
Tables aandeessnnnnaahaaa the test accuracy of all algorithms on CIFAR-IC dataset and CIFAR- dataset respectively. On CIFAR-10 10 dataset, methods such as Adam and Amsgrad have the lowest test ac- curacies. Even though followup works such as AdamW, Yogi, AdaBound improve upon original Adam, they still fall behind or barely match the performance of SGD with momentum. In contrast, Padam achieves the highest test accuracy
VGGNet and WideResNet for CIFAR-I0, VGGNet and ResNet for CIFAR-100, On the other two tasks (ResNet for CIFAR-10 and WideResNet for CIFAR- (00), Padam is also on a par with SGD with momentum the final epoch (differences less than 0.2%). This suggests that practitioners should once again, use fast to converge partially adap- tive gradient methods for training deep neural networks, without worrying about the generalization performances.
Table I: Final test accuracy of all algorithms on CIFAR-10 dataset. Bold number indicates the best result.
Test Accuracy (%)
Models
SGDM
Adam
Amsgrad
AdamW
Yogi
AdaBound
Padam
VGGNet 93.71
92.21
92.54
93.54
92.94
93.28
93.78
ResNet
95.00
92.89
93.53
94.56
93.92
94.16
94.94
WideResNet 95.26
92.27
92.91
95.08
94.23
93.85
95.34
Table 3 shows the final test accuracy of all algorithms on ImageNet dataset. Again, we can observe that Padam achieves the best test accuracy VGGNet (both Top-1 and Top-5) and Top-1 accuracy on ResNet. It stays very close to the best baseline of Top-1 accuracy on ResNet model.
Table 4 shows the final test perplexity of all algorithms the Penn Treebank dataset. As we observe from the table, Padam achieves the best (lowest) test perplexity on both 2-layer LSTM and 3-layer L.STM models.
20

---

Under review as a conference paper at ICLR 2020
Table Final test accuracy of algorithms on CIFAR-100 dataset. Bold number indicates the best result.
Test Accuracy (%)
Models
SGDM
Adam
Amsgrad
AdamW
Yogi
AdaBound
Padam
VGGNet
73.32
66.60
69.40
73.03
72.35
72.00
74.39
ResNet
77.77
71.72
72.62
76.69
74.55
75.29
77.85
WideResNet
76.66
71.83
73.02
76.04
74.47
73.49
76.42
Table 3: Final test accuracy on ImageNet dataset
Test Accuracy (%)
Models
SGDM
Adam
Amsgrad
AdamW
Yogi
AdaBound
Padam
ResNet Top-1
70.23
63.79
67.69
67.93
68.23
68.13
70.07
ResNet Top-5
89.40
85.61
88.15
88.47
88.59
88.55
89.47
VGGNet 1
73.93
69.52
69.61
69.89
71.56
70.00
74.04
VGGNet Top-5
91.82
89.12
89.19
89.35
90.25
89.27
91.93
Table 4: Final test perplexity (lower is better) on Penn Treebank dataset
Test Perplexity
Models
SGDM
Adam
Amsgrad
AdamW W
Yogi
AdaBound
Padam
2-layer LSTM
63.37
61.58
62.56
63.93 63.24
64.13
63.14 60.89
61.53
3-layer LSTM
61.22
60.44
61.92
60.01
58.48
21


Reviews:
[{"review_id": "rkxzgXBx9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "It has been empirically observed that adaptive optimizers such as Adam/Amsgrad\nlead to worse generalization than SGD + momentum when used to train neural\nnetworks.\nMotivated by this observation the authors suggest Padam, a modification of Adam/Amsgrad.\nPadam contains a parameter p that, when set to 0 reduces their method to SGD + momentum\nand when set to 1/2 reduces their method to Adam/Amsgrad.\n\nThe Padam iteration is roughly $x^{k+1} = x^k - \\alpha * g / v^p$\nwhere $g$ is an exponential moving average of stochastic gradients and\n$v$ is an exponential moving average of the squared gradient.\n\nIn that way Padam is capable of interpolating between the two methods in order\nto find a good trade-off between the improved convergence of the one and the\nimproved generalization capability of the other.\n\nThe authors also suggest an explanation for why the generalization gap\nof Adam happens:\nThey claim that it is due to the \"small-learning rate dilemma\" that happens\nas follows.\nA small second moment of the stochastic gradients as approximated by $v^{1/2}$\n(think variance) in can lead to large effective steps in some components.\nTo balance this effect out, Adam needs to choose smaller step sizes than\nSGD + momentum.\nIf the same learning rate schedule is used based on a smaller base step size,\nAdam under-trains at the end of training.\n\nThe authors suggest that Padam with p < 1/2 can use larger learning rates\nbecause it does not have as large effective steps if second moments are small.\n\nThe authors also prove convergence rates for making gradients small with Padam.\nThey use the setting of nonconvex optimization and not online convex regret\nanalysis like the Adam/Amsgrad papers.\n\nFinally, the presented experiments using image data sets and standard neural\nnetwork architectures suggest that Padam indeed shares the advantages of both\nSGD + momentum and Adam and thus obtains a best of both worlds.\n\nI suggest to accept the paper.\nIt introduces an elegant generalization of Amsgrad that appears to be\nempirically useful in experiments.\nThe experiments seem to be fairly performed (in terms of hyperparam search).\nThe rigorous convergence analysis is laudable although perhaps not as relevant\nas the practical usefulness of the suggested approach.\n\nI do however suggest that some changes be made.\n1. The \"small learning rate dilemma\" phenomenon needs to be more clearly defined\n\tand explained.\n\tThe whole relationship of SGD step size schedules to generalization\n\t(e.g. simulated annealing analogy) is certainly nontrivial.\n\tI would rather the paper not make some nonrigorous claims if there is no\n\tproof.\n\tOr state clearly whether something is conjecture or proposition (with proof).\n\n2. Explain why the same learning rate schedule should or is (in practice) used\n\tfor Adam as for SGD + momentum, as this does not seem to make sense given\n\tthe aforementioned dilemma.\n\nSpecific notes / suggestions:\n- Page 2: \"We proposed a novel\" -> \"We propose a novel\"\n\n- Page 4: \"bridging this generalization gap\"\n\t\"this\" does not make sense to me in that context, rather use \"the\"\n\n- Page 5: Figure 1, p = 1/16 seems to still be the most attractive in terms of\n\tgeneralization in the long run. Why would I not want to wait till the model\n\tis fully trained?\n\n- Page 4: \"It is very likely that Adam/Amsgrad is \"over-adaptive\"\n\tThis seems to me a strong claim and the explanation that follows it to me\n\tis not rigorous enough.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJle8vkKKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "% post author response %\nThanks for your detailed response. \n\nR1. Note that in almost all classical optimization routines, the learning rate has a (very intuitive) scaling on the problem parameters - for e.g. in gradient descent, the learning rate looks like 1/smoothness. This is mirrored in the definition of Newton methods - where, in the direction of hessian^{-1} grad, one uses a scale free line search to estimate an appropriate stepsize between 0 and 1 (to emphasize, this value has *no* dependence on problem scaling). While this begins to fall apart with the case of adaptive gradient methods, how can we even hope to justify the potentially arbitrary power of, say, 1/4 or 1/8 used by Padam? This is the reason behind my comment that the algorithm is unnatural. By using such a power of the smoothness of the problem, the other component of the learning rate (alpha_t) is no longer a scale free quantity. It has to depend on other problem dependent parameters for the overall learning rate to be scaled appropriately based on the problem characteristics. \n\nR2. While the paper performs grid search (sec C.3 in the paper) for the partially adaptive parameter, the lambda value for YOGI is set as one suggested in their paper. I can accept the claim of authors if I see more experiments tuning the lambda parameter for YOGI as well.  Otherwise, I dont quite see why one specific parameter value for lambda works for every problem.\n\nR3/R5. My point is that the original paper for adaptive methods (adam/adagrad) never mentions learning rate decay. This seems to have been added in subsequently just to boost the performance. What I do not understand is what is the specific advantage of adaptive methods over SGD if every component used by SGD (including learning rate decay) is used even by adaptive methods. Even from a theoretical bound perspective, to the best of my knowledge, there is no clear indication that the partially adaptive momentum methods actually improve over vanilla sgd (+momentum) in settings that do not involve dense parameters/gradients.\n\nR4. Again, the learning rate decay has a specific use in the papers I referenced in my review. Somehow, the bounds presented in the paper do not reflect the use of a step decay schedule on the learning rates, so, I see that the theorem (aside from the assumptions mentioned) is detached from the practical results even in this respect. \n%%\n\nThis paper considers generalization issues experienced by adaptive gradient methods compared to well-tuned SGD + momentum, a topic that is of interest in the development of optimization methods for deep learning. The paper is well-written and elaborates on (i) issues faced by adaptive gradient methods in contrast to standard SGD + momentum, (ii) presents experimental results on training standard conv-net based architectures on image classification benchmarks and on training LSTMs on PTB and (iii) presenting theoretical analysis relating convergence of the method to a first-order critical point for smooth stochastic non-convex optimization.\n\nI have questions about certain aspects of the paper, which I will elaborate below:\n\n— If one takes a step back to understand the origins of diagonal adaptation methods (introduced by Adagrad), this was motivated by the infeasibility of using the inverse square root of a full pre-conditioning adaptation matrix. If we think of such full matrix adaptation methods, is this paper implying the use of other matrix powers (other than a square root) as used by adagrad (or other adaptation approaches)? This appears very unnatural to me.\n\n-- If the main issue is that it is not possible to typically use a large learning rate at the start with ADAM or other preconditioning methods and that leads to using other powers of the diagonal adaptation matrix, a more natural fix would be to use a conservative (trust region inspired) approach to reduce aggressive steps at the start. What I mean is as follows: Suppose H_t is a preconditioning matrix, g_t is the gradient (or the discounted sum of gradients with/without bias correction). The current approach is to make H_t diagonal and use H_t ^{-1/2}g_t as the update matrix. One option to prevent aggressive initial steps is to use (H_t + \\lambda)^{-1/2} g_t, either with having lambda fixed through the optimization or making it a function of iterations. \n\n— I am not aware that papers which employ adaptive methods also tend to use some form of learning rate decay (on the alpha_t ’s) - at least, by looking at the original papers of Adam/adagrad, I do not see the combination of learning rate decay and adaptive methods. In a sense, if one has an `\"adaptive\" optimization method, it’d be unnatural to have to use some form of step decay of the learning rates (alpha_t's) in conjunction with these methods. One would typically just use SGD+momentum with some form of such a step decay of the learning rates. This to me is a serious shortcoming - it appears to make the use of adaptive methods almost irrelevant because the only hyper-parameter that it gets rid of is the dependence on the initial learning rate (because, for SGD, we anyway have the other hyper-parameters like momentum, stepdecay factor, when to decay learning rate etc.)\n\n— With regards to theory (and connections to experiments): Despite the fact that the paper employs a step decay schedule on the learning rates (alpha_t), their theorem statement (or any corollary) doesn’t actually employ this specific step size decay scheme (on the alpha_t ’s) and attempts to understand what are the advantages of the step decay schedule on the convergence statements provided.  The step decay schedule has featured in several recent efforts in the stochastic optimization community (both with convex (https://arxiv.org/pdf/1607.01027.pdf, https://arxiv.org/pdf/1904.12838.pdf)/non-convex (e.g. https://arxiv.org/pdf/1907.09547.pdf) objectives), where, the results indicate non-trivial advantages of using these step-decay schemes on alpha_t’s (though, these are with non-adaptive optimization methods).\n\n— Along these lines (of the previous point), it is important to note what the performance of the optimization method is when alpha_t’s are fixed to a specific value (without being decayed over the course of optimization) - since this relates to the most standard definition (and advantage) associated with using adaptive gradient methods. My understanding is that this result continues to be fairly sub-optimal compared to using SGD+momentum with a step decay schedule. \n\nOther minor comments:\n— I do not understand the use of the term “second order” momentum for calling variables that have a running average of squared gradients. This term is misleading in what it represents.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1eUQxIZcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new variation on adaptive learning rate algorithm that unifies SGD with momentum and Adam/Amsgrad. They provided convergence proof for this algorithm. The effectiveness of the proposed method was demonstrated through various domains and neural networks architecture. Though the empirical results are extensive, I am leaning towards reject because (1) The reason why the method works isn't clear. (2) The theory doesn't justify the practice. (3) The practical usefulness of the algorithm isn't clear. Here are my detailed comments:\n\n(1) The paper provides an observation which they call \"small learning rate dilema\": One often uses a smaller base learning rate for adaptive gradient methods than SGD with momentum. This makes the boost one can gain from applying learning rate decay to adaptive gradient methods not as significant as applying to SGD with momentum. Based on this observation, they propose to penalize the adaptiveness by adjusting the value p in their algorithm. However, the proposed adjustment seems like a trivial one, without giving too much insights into why learning rate decay is not compatible to adaptiveness. An insightful analysis should try to first answer the following questions:     \n      (a) Why shall one start with a large learning rate in the beginning?\n      (b) Why does learning rate decay gives a boost to performance?\n      (c) If one uses an adaptive method, how does it affect one's choice for the initial learning rate? \n      (d) and how does the adaptiveness changes the effect of learning rate decay?\nTo answer those questions, I suggest the author to read [1] where they gave partial answers to (a) (b). If one tries to do similar analysis performed in [1] for adaptive methods, one might be able to answer (c) and (d). \n\n(2) I have two criticisms to the theoretical analysis carried out in the paper. The most important issue is that the analysis is not useful to show the effectiveness of the proposed method:\n     (a) The rate of convergence matches with SGD + momentum. So it is not better than the baseline. \n     (b) It does not show the relationship of adaptiveness and decaying learning rate schedule. \nThe second criticism is related to the novelty of the theorems.  Please correct me on this because I did not go over the theorems carefully. But based on my crude assessment,  theorems are mostly mechanical applications of prior work to the current extended version. Hence it does not provide any further insights into the convergence of nonconvex optimization methods.\n\n(3) Though the empirical results are good, where the proposed method matched or outperformed all previous methods in The method introduces one extra hyperparameter, p, for tuning. It is then questionable whether the algorithm is efficient in terms of hyperparameter searches, i.e., how many hyparameter sweeps are needed for finding a good run, versus baseline methods like SGD+momentum. Since the performance of the methods are mostly the same, if the proposed method requires as many hyperparameter tuning as SGD+momentum, then it makes the proposed method less useful in practice.\n\n[1] Understanding short-horizon bias in stochastic meta-optimization. Wu et. al. ICLR 2018.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends Adam by adding another hyperparameter that allows the second moments to be raised to a power p other than 1/2. This certainly seems worth trying. The paper is well written, and the experiments seem reasonably complete. But some of the reviewers and I feel like the contribution is a bit obvious and incremental. The "small learning rate dilemma" needs a bit more justification: since the denominator has a different scale, the learning rates for different values of p are not directly comparable. It could very well be that Adam's learning rate has to be set too small due to some outlier dimensions, but showing this would require some evidence. From the experiments, it does seem like there's some practical benefit, though it's not terribly surprising that adding an additional hyperparameter will result in improved performance. The reviewers think the theoretical analysis is a straightforward extension of prior work (though I haven't checked myself). Overall, it doesn't seem to me like the contribution is quite enough for publication at ICLR.

Decision: Reject