Decaying momentum helps neural network training
Authors: John Chen
Anastasios Kyrillidis
Venue:  (2020)
Keywords: sgd
momentum
adam
optimization
deep learning

Abstract: Momentum is a simple and popular technique in deep learning for gradient-based optimizers. We propose a decaying momentum (Demon) rule, motivated by decaying the total contribution of a gradient to all future updates. Applying Demon to Adam leads to significantly improved training, notably competitive to momentum SGD with learning rate decay, even in settings in which adaptive methods are typically non-competitive. Similarly, applying Demon to momentum SGD rivals momentum SGD with learning rate decay, and in many cases leads to improved performance. Demon is trivial to implement and incurs limited extra computational overhead, compared to the vanilla counterparts. 

Under review as a conference paper at ICLR 2020
DECAYING MOMENTUM HELPS NEURAL NETWORK TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
Momentum is simple and popular technique in deep learning for gradient- based optimizers. We propose a decaying momentum (DEMON) rule, motivated by de- caying the total contribution of a gradient to all future updates. Applying DEMON to Adam leads to significantly improved training, notably competitive to momen- tum SGD with learning rate decay, even in settings in which adaptive methods are typically non-competitive. Similarly, applying DEMON to momentum SGD rivals momentum SGD with learning rate decay, and some cases leads to improved performance. DEMON is trivial to implement and incurs limited extra computa- tional overhead, compared to the vanilla counterparts.
INTRODUCTION
Deep Neural Networks (DNNs) have drastically advanced the state-of oftte-of-thh-art performance in many computer science applications, including computer vision (Krizhevsky al. 2012), (He al. 2016 Ren et al.] (2055), natural language processing (Mikolov et [ll.]2013) Bahdanau et al. 2014 Gehring ct al. 2017) and speech recognition (Sak et al.] [0144] Sercuu et al. 2016). Yet, the face of such significant developments, the age-old stochastic gradient descent (SGD), and the accelerated variant SGD with momentum (SGDM), algorithm remains one the most, if not the most, popular method for training DNNs (Sutskever al.
2013
Goodfellow al. 2016 Wilson al. 2017).
Adaptive methods (Duchi et al. 2011 Zeiler 2012 Hinton et al. 2012 Kingma Ba 2014 Ma &
Yarats 2018) sought to simplify the training process, while providing similar performance. How- ever, while they are often used by practitioners, there are cases where their use leads to a performance gap (Wilson al., 2017, Shah et al., 2018). At the same time, much of the state-of-the-art perfor- mance on highly contested benchmarks Such as the image classification dataset ImageNet-hare been produced with SGDM (Krizhevsky al. [012 He et al. 2016 Xie al. 2017 Zagoruyko &
Komodakis 2016 Huang et al.) 2017: Ren et al. 2015: Howard et al. 2017).
Nevertheless, key factor in any algorithmic success still lies in hyperparameter tuning. For exam- ple, in the literature above, they obtain such performance with a well-tuned SGD with momentum and a learning rate decay schedule, or with proper hyperparameter tuning adaptive methods. Slight changes in learning rate, learning rate decay. momentum, and weight decay (amongst others) can drastically alter performance. Hyperparameter tuning is arguably one of the most time consum- ing parts of training DNNs, and researchers often resort to costly grid search. Thus, finding new and simple papaa--arreeter tuning routines that boost the performance of state of the art algorithms is of ultimate importance and one of the most pressing problems in machine learning.
The focus of this work on the momentum parameter and how we can boost the performance of training methods with simple technique. Momentum helps speed up learning in directions of low curvature, without becoming unstable in directions of high curvature. Minimizing the objective function L(-), the simplest and most common momentum method, SGDM, is given by the following recursion for variable vector 0, E TRP:
0 0, + my,,
8, B0; 1 9r.
The coefficient B t-trriiiionaall. selected constant in [o. 1] controls how quickly the momentum decays, 9: represents stochastic gradient, usually E[g.] V ((,,), and n > 0 is the step size.
But how do we select 3? The most prominent choice among practitioners is
0.9. This is
supported by recent works that prescribe it (Chen al. 2016 Kingma & Ba 2014 Hinton al.

---

Under review as a conference paper at ICLR 2020
Reddi et al., ) and by the fact that most common softwares, such as PyTorch (Paszke et al. 2017). declare 3 0.9 as the default value in their optimizer implementations. However, there no indication that this choice is universally well-behaved.
2012
There are papers that attempt to tune the momentum parameter. Under an asynchronous distributed setting, (Mitliagkas et al., 2016) observe that running SGD asynchronously is similar to adding momentum-like term to SGD; they also provide experimental evidence that naively setting / = 0.9 would result in momentum "overdose", leading to suboptimal performance. As another example, YellowFin (Zhang & Mitliagkas, (017) is a learning rate and momentum adaptive method for both the synchronous and asynchronous setting, motivated by a quadratic model analysis and some ro- bustness insights. The main message of that work is that, like momentum acceleration needs to be carefully selected based on properties of the objective, the data, and the underlying computational resources. Finally, moving from classical DNN settings towards generative adversarial networks (GANs), the proposed momentum values tend to decrease from 3 0.9 (Mirza Osindero 2014 Radford et al. 2015 Arjovsky et al. 2017), taking even negative values (Gidel et al. 2018)..
In this paper, we introduce a novel momentum decay rule which significantly surpasses the perfor- mance of both Adam and SGDM (as they are used currently), in addition to other state-of-the-art adaptive learning rate and adaptive momentum methods, across a variety of datasets and networks. In particular, our findings can be summarized as follows:
We propose a new momentum decay rule, motivated by decaying the total contribution of a gradient all future updates, with limited overhead and additional computation. i) Using the momentum decay rule with Adam, we observe large performance gains-relative to vanilla Adam-wherr the network continues to learn for far longer after Adam begins to plateau, and suggest that the momentum decay rule should be used as default for this method. iii We observe comparative performance for SGDM between momentum decay and learning rate decay; an interesting result given the unparalleled effectiveness learning rate decay schedule.
Experiments are provided on various datasets, including MNIST, CIFAR-10, CIFAR- 100, STL-100, Penn Treebank (PTB), and networks, including Convolutional Neural Networks (CNN) with Resid- ual architecture (ResNet) (He et al. (0016). Wide Residual architecture (Wide ResNet) (Zagoruyko)
& Komodakis 2016), Non-Residual architecture (VGG-16) (Simonyan & Zisserman, 2014). Recur-
rent Neural Networks (RNN) with Long Short-Term Memory architecture (LSTM) (Hochreiter & Schmidhuber 1997). Variational AutoEncoders (VAE) (Kingma Welling 2015). and the recent
Noise Conditional Score Network (NCSN) (Song & Ermon,2011
2 PRELIMINARIES
Plain stochastic gradient descent motions. Let O, E IR be the parameters of the network at time step t, where 7) E IR is the learning rate/step size, and gr is the stochastic gradient w.r.t. 6g for empirical loss L(-), such that E[g.] V ((6.). Then, plain stochastic gradient descent (SGD) uses It. Vt. Here, the step size r could also be time dependent, n)
the recursion: 6,+1 = 0, but practice shows that decreasing the value of 1) regular or predefined intervals works favorably compared to decreasing the value of at every iteration.
SGDM is parameterized by RR the momentum coefficient, and follows the recursion:
Oc+1 0, + 1/tt, Ur = B gt.
where in E Ip accumulates momentum. Observe that for B O, the above recursion is equivalent to SGD, Common values for /3 are closer to one, with 3 = 0.9 the most used value (Ruder 2016).
Adaptive gradient descent motions. These algorithms utilize current and past gradient information to design preconditioning matrices that better approximate the local curvature of L(.). Beginning with AdaGrad (Duchi al., 2011), the SGD recursion, per coordinate of f, becomes:
0411 00,3 "
Varate" It.sr V1,
where G2 E RPxP is usually a diagonal preconditioning matrix as a summation of squares of past gradients, and E > 0 small constant.
RMSprop (Hinton et al. 2012) substitutes the ever accumulating matrix G, with a root mean squared operation. Denoting the average of squared gradients as EgoS per iteration compute: &
2

---

Under review as a conference paper at ICLR 2020
- + og), where 32 was first proposed 0.9. Here, denotes the per-coordinate multiplication. Then, RMSprop updates as where a momentum term can also be optionally added:
02,1
Vt.
Finally, Adam (Kingma Ba 2014), in addition, keeps an exponentially decaying average of past gradients: E1+ B1 & + Bi) It, leading to the recursion!
0.+1. = 02
A1,
141,1
where usually 31 0.9 and 32 0.999. Observe that Adam is equivalent to RMSprop when 31 = 0, and when no bias correction is applied results in the same recursion.
3 DEMON: DECAYING MOMENTUM ALGORITHM
Algorithm DEMON in SGDM
Algorithm 2 DEMON in Adam
1: Parameters: # of iterations T, step size 17,
1: Parameters: # of iterations T, step size n,
momentum initial value Bini-
momentum initial value Biiit. 82, e = 10-8
2: e4 02 0, otherwise randomly initialized.
2: 02 E0 0, otherwise randomly
3: for = o...., do
initialized.
(1-4)
for O,... T do
4:
Be = Bui
(1 (--) - (1- $)
4:
Be Bimi
5:
Oc+1 = 0, 119t + BrVr
(1- 1an8--Bin Bunl (1-4)
6:
Ut+1 3,0% 179t
5:
Ett = :B 227 + 32) (ge gg)
7: end for
6:
me, It,1 Bem--1.5
7:
0c+1,8 = O2,i Verog,
11
8: end for
Motivation and interpretation. DEMON is motivated by linear learning rate decay models which reduce the impact of gradient current and future updates. By decaying the momentum parameter, we decay the total contribution of a gradient to all future updates. Our goal here is to present concrete, effective, and easy-to-use momentum decay procedure which we show in the experimental section. By Occam's Razor, a linear decay achieves such a goal since it is simple and requires tuning in most cases. The key component is the momentum decay schedule:
(1-$)
B
Piii (1-9m))+ +uml) (1-
(1)
Above, the fraction (1-t/T) refers to the proportion of iterations remaining. The interpretation of this rule comes from the following argument: Assume fixed momentum parameter 3r = 8; e.g., 0.9, as literature dictates. For our discussion, we will use the SGDM recursion. We know that
0o = 0, and 2t _ gt- Then, the main recursion can be unrolled into:
0.+1 = 0, 199t - n39-11 n 00 119t
+
Σ (8'- gc-i))
Interpreting the above recursion. particular gradient term 9b contributes total of E; 3 of its "energy" all future gradient updates. Moreover, for an asymptotically large number of iterations, we know that B contributes on up t-1 1 terms. Then, Σ 8 = 3 Σi= B = 3/(1 3). Thus, in our quest for decaying schedule and for simple linear momentum decay, it natural consider scheme where the cumulative momentum is decayed to 0. Let Biii be the initial B; then at current step i with total 7 steps, we design the decay routine such that: 3/(1-B) = (--ITT)
Bimi)). This leads to equation I Although 3 changes in subsequent iterations, this is typically very close approximation since 35 3 for a particular g' diminishes much faster than 3 changes.
Connection to previous algorithms. DEMON introduces implicit discount factor. The main recursions of the algorithm are the same with standard algorithms in machine learning. E.g., for 3, B = 0.9 obtain SGD with momentum, and for 3 0 obtain plain SGD in Algorithm
For clarity, will skip the bias correction step this description of Adam; Kingma & Ba 2014
3

---

Under review as a conference paper at ICLR 2020
in Algorithm 2 for 31 0.9 with a slightly adjustment of learning rate obtain Adam, while for 31 = O we obtain a non- cannaucumulatiee AdaGrad algorithm. We choose to apply DEMON to a slightly adjusted Adam adam-instead of vanilla Adam to isolate the effect of the momentum parameter, since the momentum parameter adjusts the magnitude of the current gradient as well in vanilla Adam.
Efficiency. DEMON requires only limited extra overhead and computation in comparison to the vanilla counterparts, for the computation of 3,-
Practical suggestions. For settings in which Bii is typically large, such image classification, we advocate for decaying momentum from Siiit at L= = O, to 0 at I = 7 as a general rule. We also observe and report improved performance by delaying momentum decay till later epochs. In many cases, performance can be further improved by decaying to a small negative value, such as 0.3.
4 RELATED WORK
There are numerous techniques for automatic hyperparameter tuning. The most widely used learning rate adaptive methods, starting with AdaGrad (Duchi et al.) (201T). AdaDelta (Zeiler 2012), RMSprop (Hinton et al. 2012), and Adam (Kingma & Bal 4). Adam (Kingma & Ba 201 4). the most popular, introduced a momentum term, which is combined with the current gradient before multiplying with an adaptive learning rate. Interest in closing the generalization difference between adaptive methods and SGDM led to AMSGrad (Reddi et al., 2019), replacing the exponential mov- ing average of squared gradients with the maximum, QHAdam (Ma & Yarats 2018), a variant of QHM which is capable of recovering variety of optimization algorithms, A (Loshchilov & Hutter 2017), ). by fixing the weight decay of Adam, and Padam (Chen & Gu 2018). by lowering the
exponent of the second moment.
Asynchronous methods are commonly used in deep learning, and (Mitliagkas et al. 2016) show that running SGD asynchronously is similar to adding momentum-lik term to SGD without assump- tions of convexity of the objective function. They demonstrate this natural connection empirically on CNNs. This implies that the momentum parameter needs to be tuned according to the level of asynchrony. YellowFin (Zhang & Mitliagkas| (017) is a learning rate and momentum adaptive method for both the synchronous and asynchronous setting motivated by a quadratic model analysis and robustness insights. In the non-convex setting, STORM (Cutkosky & Orabona 2019) uses a variant of momentum for variance reduction.
There is substantial research, both empirical and theoretical, into the convergence of momentum methods (Wibisono & Wilson| 2013: Wibisono et al. 2016; Wilson al., 2016 Kidambi et al.) 2018). In addition, (Sutskever et al. 2013) explored momentum schedules, with even increasing mo- mentum schedules during training. inspired by Nesterov's 's routines for convex optimization. (Smith) et al. 2017) explores increasing batch size instead of decreasing the learning rate, as well as scal- ing batch size to increasing learning rate or increasing momentum. (Smith) 2018) introduces cycles of simultancously increasing learning rate and decreasing momentum followed by simultancously decreasing learning rate and increasing momentum. There is some work into reducing oscillations during training, by adapting the momentum (Odonoghue & Candes 2015). There is also work into adapting momentum in well-conditioned convex problems as opposed to setting to zero (Srinivasan eet al.,,220001 2018). Another approach in this area is to keep several momentum vectors according dif- ferent 3 and combining them (Lucas et al. 2018). We are aware of the theoretical work of (Yuan) 2016) which prove under certain conditions that SGDM is equivalent SGD with a rescaled learn- ing rate. however our experiments in the deep learning setting show slightly different behavior and understanding why is an exciting direction of research.
Smaller values of 3 have gradually been employed for Generative Adversarial Networks (GAN), and recent developments in game dynamics (Gidel et al. 2018) show negative momentum is helpful.
EXPERIMENTS
We separate experiments into those with adaptive learning rate and those with adaptive momentum. All settings, with exact parameeassessrss. are briefly summarized in Table [ and comprehensively detailed in Appendix A We report improved performance by delaying the application of DEMON where applicable, and report performance across different number of total epochs to demonstrate effectiveness regardless of the training budget. Note that the predefined number of epochs we run all experiments affects the proposed decaying momentum routine, by definition of 3,,
4

---

Under review as a conference paper at ICLR 2020
Experiment short name
Model
Dataset
Optimizer
R -EW-IIAAARD---EOONSEXXXX
ResNet18
CIFARIO
DEMON SGDM
RN18 0
ResNet18
CIFARIO
DEMON Adam
VGG-16
CIFARIO0
DEMON SGDM
VSG16-C1FAK100-DENNG
VGG-16
CIFARIO0
DEMON Adam
WRN-STLL1C-DEMON33E.
Wide ResNet 16-8
STL10
DEMON SGDM
WRU-STTL1G-DEMONAAaaa
Wide ResNet 16-8
STL10
DEMON Adam
LSSTX-P7BB-DEEONNGGDM
LSTM RNN
Penn TreeBank
DEMON SGDM
L.S7X-P78-D5M0NAAdmm
LSTM RNN
Penn TreeBank
DEMON Adam
VAB-WWNIST-DEMONSSDDY
VAE
MNIST
DEMON SGDM
TAB-W0IST-DDEMNNdaaa
VAE
MNIST
DEMON Adam
DESN-CCFFARO-DDEMONAAE
NCSN
CIFARIO
DEMON Adam
Table 1: Summary of experimental settings.
30 epochs
75 epochs
150 epachs
300 epochs
Adam
16.58 +: .18
13.63
+: .22
11,90 1: .06
11.94 +:: .06
AMSGrad
16.98 1: 36
13.43
1 .14
11.83
1: .12
10.48 1:
QHAdam
16.41 + 38
15.55
25
13.78
08
13.36
DEMON Adam
11.75 +
9.69
= .10
8.83 + .08
8.44 = .05
Table 2: RN1 8-C1 FAR1 C C-DRMONAdam generalization eror. The number of epochs was predefined before
the execution of the algorithms.
5.1 ADAPTIVE METHODS
At first, we apply DEMON Adam (Algorithm 2 to variety of models and tasks. We select vanilla Adam as the baseline algorithm and include more recent state-of-the-art adaptive learning rate meth- ods Quasi-Hyperbolic Adam (QHAdam) (Ma & Yarats 2018) and AMSGrad (Reddi et al., 2019) in our comparison. See Appendix A.2.I for details. We tune all learning rates in roughly multiples of 3 and try to keep all other parameters close to those recommended in the original literature. For DEMON Adam, we leave Pimi = 0.9, 32 = 0.999 and decay from Bui to O in all experiments.
Residual Neural Network (RN1 BEL---CIFARII (0A.AAARRR0DEEMAAMm... We train ResNet18 (He et al.) 2016) model on the CIFAR- 10 dataset. With DEMON Adam, we achieve the generalization error reported in the literature (He 2016) for this model, attained using SGDM and curated learning rate decay schedule, whilst all other methods are non-competitive. Refer to Table for exact results.
In Figure 2 (Top row, two left-most plots), DEMON Adam is able learn in terms of both loss and accuracy after other methods have plateaued. Running seeds, DEMON Adam outperforms all other methods by a large 2%-5% generalization error margin with small and large number of epochs.
Non-Residual Neural Network (VGG16- -CIFAAARO--DEMONAAa. -EMONAddam) For the CIFAR- 100 dataset, train an adjusted VGG-16 model (Simonyan & Zisserman, 2014). Similarly to the previous setting, we observe similar learning behavior of DEMON Adam, where it continues to improve after other methods appear begin to plateau. We note that this behavior results in 1-3% decrease in generalization error than typically reported results with the same model and task (Sankaranarayanan al. 2018), which are attained using SGDM and a curated learning rate decay schedule.
Running 5 seeds, DEMON Adam achieves an improvement of 3%-6% generalization error margin over other methods, both for a small and large number of epochs. Refer to Figure (Top row, right-most plot) and Table for more details.
Wide Residual Neural Network (NRN-STL1 O-DEMONAAam)) The STL-10 dataset presents different challenge with significantly smaller number of images than the CIFAR datasets, but in higher resolution. We train a Wide Residual 16-8 model (Zagoruyko & Komodakis 2016) for this task. In this setting, note again the behavior of DEMON Adam significantly outperforming other methods in the latter stages of training.
Running 5 seeds, DEMON Adam outperforms all other methods by 0.5%-2% generalization error margin with a small and large number epochs. Refer to Figure (Bottom row, left-most plot) and 'abble for more details.
5

---

Under review as a conference paper at ICLR 2020
VGG-16
Wide Residual 16-8
epechs
150 epochs
300 epochs
50 epochs
100 epochs
200 epochs
Adam
37,98 1 .00
33.62 1
.26
18.65
AMSGrad
40.67 = 65
34,46 + 21
31,62 + .12
21.73 + 225
19.35 + 20
18.21 = .18
QUIAdam
36.53 1 20
32.96 1:
30.97 1 110
21.25
1: 22
19.81
4 11
18.52 .25
DEMON Adam
32.40 .19
28.84 1: 188
27.11 1.1
19,42
1:42::00 .10
18.36
17.62 d: .12
Table VGG1 (0G16-CIFARRI ---CIAAALL--DEOORAaa and WRN-STLLL STLLO-DEMONAdann generalization enorr The num- her of epochs was predefined before the exccution the algorithms.
LSTM
VAE
NCSN
25 spochs
39 epochs
50 spochs
100 epochs
200 opochs
512 opochs
Adam
115.54 644
115.02 : 52
136.28 + .18
134.64 + .14
134.66 +
8.15 .20
AMSGrad
108.07 = .19
107.87 = 25
137,89
+ .12
135.69 = 03
134.75
+ .18
QHAdam
112.45
136.69
134.4
.12
DEMON Adam
101.57 .32
101.44 1 .47
134.46 1 .17
134,12 1 .08
133.87 1. 211
8.07 1 .08
Table 4: PIB-LLTX aww.--DDEEOOocaaanggnnaaaaaiiiii perplexity, VAE-WNIST-DEMOONEE generalization loss
M.NEES--CFFARRO--EENNNA.. inception score.
LSTM (PTB-LSTT-DDENONAC)
Language modeling can have gradient distributions which are sharp; for example, in the case of rare words. We use an LSTM (Hochreiter & Schmidhuberj 1997) model to this task. We observe overfitting for all adaptive methods.
Similar to above, running 5 seeds, DEMON Adam outperforms all other methods by a 6-14 general- ization perplexity margin. Refer to Figure 2 (Bottom row, middle plot) and Table for details.
Variational AutoEncoder (VAE CAEE-NIST MINIST-DEMME DEMONAddmm) Generative models are a branch of un- supervised learning that try learn the data distribution. VAEs (Kingma & Welling 2015 pair generator network with second Neural Network, recognition model that performs approximate inference, and can be trained with backpropagation. We train VAEs on the MNIST dataset.
Running 5 seeds, DEMON Adam outperforms all other methods, particularly for smaller number of epochs. Refer to Figure (Bottom row, right-most plot) and Table for more details.
Noise Conditional Score Network (NCSN-CIFAR] O-DEMONAdam) NCSN (Song & Ermon 2019 is recent generative network achieving state-of the-art inception score on CIFARIO. NCSN estimates the gradients of the data distribution with score matching. Samples are then produced via Langevin dynamics using those gradients. We train a NCSN on the CIFAR dataset and, using the official implementation, were unable reproduce the reported score in the literature. NSCN trained with Adam achieves a superior inception score in Table[4] however the produced images in Figure] exhibit noticeably unnatural green compared to those produced by DEMON Adam.
Figure I: Randomly selected CIFARIO images generated with NCSN. Left: Real CIFARIO images. Middle:
Adam. Right: DEMON Adam.
5.2 ADAPTIVE MOMENTUM METHODS
We apply DEMON SGDM (Algorithm) to a variety of models and tasks. Since SGDM with learn- ing rate decay is most often used to achieve the tlateoof-the-ar results with the architectures and tasks in question, we include SGDM with learning rate decay as the target to beat. SGDM with learning rate decay implemented with decay validation error plateau, where hand-tune the number of epochs to define plateau. Recent adaptive momentum methods included this sec- tion Aggregated Momentum (og (Lucas al. 2018), and Quasi-Hyperiolic Momentum

---

Under review as a conference paper at ICLR 2020
Thening Bedoe
Trang wittt
Th tht S737
21.
3140
03
6.50
3010
3.10
371
Figure 2: Top row, two left-most plots: RN18 CIII--CFFRICO U-DEMONAda for 300 epochs. Top row, right-most plot: VGG1 6- WRN-STLI O-DEMONAdam for 200 epochs. Bottom row, middle plot: PTB-LISTN-DeRONAdn for 25 epochs. Bottom row, right-most plot: MAE-MNIST-DEMON represent training and generalization metrics respectively. Shaded bands represent one standard deviation.
for 300 epochs. Bottom row, left-most plot:
for 200 epochs. Dotted and solid lines
30 epocks
75 epochs
150 epochs
300 epochs
SGDM learning rate docay
11.29 + .35
9.05 = .07
8.26 + 00
1.97 + .14
AggMo
18.85 + .27
13.02 223
11,95
QUM
14.65
1: .24
12.66
1. 19
11,27
+ 1 .13
10.42 1: .05
DEMON SGDM
10.89 4: .12
8.97 2
8.39 4:
Table 5: RN18 8 8- CIFFRRI C- DEMMONSGDM generalization error. The number of epochs was predefined before the execution of the algorithms.
(QHM) (Ma & Yarats 2018). We exclude accelerated SGD (Jain et al. 2017) due to difficulties in tuning. See Appendix [A.2.2 for details. Similar to the last section, we tune all learning rates in roughly multiples of 3 and try to keep all other parameters close those recommended in the original literature. For DEMON SGDM, we leave Biic = 0.9 for most experiments and generally decay from Bui to O.
Residual Neural Network (RN1 B-CBIA-CI------- SAWEPAARRO-DDMONSODO We train a ResNet18 model on the CIFAR-10 dataset. With DEMON SGDM, we achieve better generalization error than SGDM with learning rate decay, the optimizer for producing state-of-the-art results with ResNet architecture. The better performance of decaying momentum relative learning rate decay is surprising.
Running 5 seeds, DEMON SGDM outperforms all other adaptive momentum methods by a large 3%-8% validation error margin with small and large number of epochs and is competitive or better than SGDM with learning rate decay. In Figure 3 (Top row, two left-most plots), DEMON SGDM is observed to continue learning after other adaptive momentum methods appear to begin to plateau.
Non-Residual Neural Network (VGG1 6-CTFAR1 00-DEMONSGDM) For the CIFAR- 100 dataset, we train an adjusted VGG-16 model. In Figure 3 (Top row, right-most plot), we observe DEMON SGDM to learn slowly initially in loss and error, but similar to the previous setting it continues to learn after other methods begin to plateau, resulting in superior final generalization error.
Running 5 seeds, DEMON SGDM achieves improvement of 1%-8% generalization enror margin over all other methods. Refer to for more details.
Wide Residual Neural Network (WRN-STLIO STL----EMOOEEDDM,, We train a Wide Residual 16-8 model for the STL-10 10 dataset. In Figure (Bottom row; left-most plot), training in both loss and error slows down quickly for other adaptive momentum methods with large gap with SGDM learning rate decay. DEMON SGDM continues to improve and eventually catches up to SGDM learning rate decay.
Running 5 seeds, DEMON SGDM outperforms all other methods by a 1.5%-2% generalization error margin with small and large number of epochs. Refer Table 6 for more details.
7

---

Under review as a conference paper at ICLR 2020
VGG-16
Wide Residnal 16-8
75 epochs
150 epachs
300 epochs
75 epochs
150 epochs
300 epechs
SGDM learning decay
30.65
29.74
21.05
17.83 1:
15.16 1: .36
AggMo
42.85
89
34.25 + 24
32.32 = .18
22.70 + .11
20.06 + 31
17.90 :.13
QHI
42.14
1
33.87
32.45
.13
22.86 +: .15
19:40 1: 23
17.79
1: .08
DEMON SGDM
34.35
44
30.59
28.99
1:16
19.45 d: .20
15.98
13.67 .13
Table VGG1 6 --IARIIARROOO DDMMONSGDM and PRX-SSTL1O-DEMONSOD generalization enoor. The num- her of epochs was predefined before the exccution.
LSTM
VAE
25 epochs
39 epochs
50 epochs
100 epochs
200 epachs
SGDM learning tale decay
89.59 1. .07
87,57 = .11
140.51
1 .73
139.54 1 .34
137.33
= .49
AggMo
+ .16
139.69
139.07
137.64
QHM
94.47 + .19
94.44
.13
145,84
+ .39
140.92
.19
137.64
.20
DEMON SGDM
88.33 1 .16
88.32
= .12
139.32 1: .23
137.51
1 29
135.95
.21
Table 7: PTB-18 1 DMMDROONGMM (perplexity) and WTW.MNNISS-OOOOIIIIIII
(gggno/oiiiiiMmggglallaaiiiiiin (generalization loss) exper-
iments.
LSTM LAST-DDEMONGDDA We train an RNN with LSTM architecture for the PTB lan- guage modeling task. Running 5 seeds, DEMON SGDM slightly outperforms other adaptive mo- mentum methods in generalization perplexity, and is competitive with SGDM with learning rate decay. Refer to Figure 3 (Bottom row, middle plot) and Table for more details.
Variational AutoEncoder -VAE-MMTST-DE -DDEONSGGDM We train the generative model VAE on the MNIST dataset. Running seeds, DEMON SGDM outperforms all other methods by 2%. 6% generalization error for a small and large number of epochs. Refer to Figure 3 (Bottom row, right-most plot) and Table 7 for more details.
Thaiig
To sing e11r
02
Thame mithatineess
Traneggeed woleeeeeeoon
1033
Trantri and Ardon
325
132
80 10
Figure 3: Top two left-most plot: MAMB--CIAARO-DEMONNEE for 300 epochs. Bottom row. left-most plots: epochs. Bottom row, middle plot: DDB-ISST-DEMONSCDD
for 300 epochs. Top row,
mosh plot: VGG1 S-CIFAA100--EMOOOO
CAN-SSTL1--DEMNNSSE
25
epochs. Bottom row, right-most plot: DEE-MNIST-DEEMMNSSO for 200 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent standard deviation.
CONCLUSION
We show the effectiveness of the proposed momentum decay rule, DEMON, across a number of datasets and architectures. The adaptive optimizer Adam combined with DEMON is empirically substantially superior to the popular Adam, in addition to other state-of-the-art adaptive learning rate algorithms, suggesting drop-in replacement. Surprisingly, it is also demonstrated that DEMON SGDM comparable to SGDM with learning rate decay. In cases where budget is limited, DEMON SGDM may be preferable. DEMON is computationally cheap, easy to understand and use, and hope is useful in practice and as a subject of future research.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv preprint arXiv:1701.0723 07777, 2017.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv: 109.04333 2014.
Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Revisiting dis- tributed synchronous SGD. arXiv preprint arXiv: 00981, 2016.
Jinghui Chen and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. arXiv preprint arXi:: 2018
Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. arXiv preprint arXiv:1905. 10018, 2019.
Timothy Dozat. 2016.
Incorporating nesterov momentum into adam. ICLR Workshop, (1):20132016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(uu):2121-2159, 2011.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence sequence learning. In Proceedings of the 34th International Conference Machine Learning-Volume 70, pp. 1243-1252. JMLR. org, 2017.
Euhanna Ghadimi, Hamid Reza Feyzmahdavian, and Mikael Johansson. Global convergence of the heavyball method for convex optimization. arXiv preprint arXiv:1412.7457, 2014.
Gauthier Gidel, Reyhane Askari Hemmat, Mohammad Pezeshki, Remi Lepriol, Gabriel Huang, Si- mon Lacoste-Julics and Ioannis Mitliagkas. Negative momentum for improved game dynamics. arXiv preprint arXiv: /007.0800000000 0444004 2018.
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14:8, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.0488 000041.4411 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700 4708, 2017.
Prateek Jain, Sham M. Kakade, Rahul Kidambi, Praneeth Netrapalli, and Aaron Sidford. Acceler- ating stochastic gradient descent for least squares regression. arXiv preprint arXiv:1704.08227, 2017.
Rahul Kidambi, Pranceth Netrapalli, Prateck Jain, and Sham Kakade, On the insufficiency of ex- isting momentum schemes for stochastic optimization. In 2018 Information Theory and Applica- tions Workshop (ITA), pp. 1-9. IEEE, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6 14/2.6980, 2014.

---

Under review as a conference paper at ICLR 2020
Diederik P Kingma and Max Welling. arXiv:1312.611,, 2015.
Auto-encoding variational bayes.
arXiv preprint
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, pp. 1097-1105, 2012.
Laurent Lessard, Benjamin Recht, and Andrew Packard. Analysis and design of optimization algo- rithms via integral quadratic constraints. SIAM Journal on Optimization, 26(1):57-95, 2016.
Iyya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.0501, 2017.
James Lucas, Shengyang Sun, Richard Zemel, and Roger Grosse. Aggregated momentum: Stability through passive damping. arXiv preprint arXiv:/804.00335, 2018.
Jerry Ma and Denis Yarats. Quasi-hyperbolic momentum and Adam for deep learning. arXiv preprint (Xiv:1810006800) 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp. 5111-3119, 2013.
Mchdi Mirza and Simon Osindero. arXiv:1411.1784 2014.
Conditional generative adversarial nets. arXiv preprint
Ioannis Mitliagkas, Ce Zhang, Stefan Hadjis, and Christopher Ré. Asynchrony begets momentum, with an application to deep learning. In 2016 54th Anmual Allerton Conference on Communica- tion, Control, and Computing (Allerton), pp. 997-1004. IEEE, 2016.
Yurii Nesterov. A method for solving the convex programming problem with convergence rate of (1/k2). Soviet Mathematics Doklady, 27())3372376, 1983.
Brendan Odonoghue and Emmanuel Candes. Adaptive restart for accelerated gradient schemes.
Foundations of computational mathematics, 15(3):715-732, 2015.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06444 2015.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence Adam and beyond. arXiv preprint arXiv: /904. 09237,, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv: 1609.0660.00444 04747, 2016.
Haşim Sak, Andrew Senior, and Françoise Beaufays.
Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Fifteenth annual conference of the international speech communication association, 2014.
Swami Sankaranarayanan, Arpit Jain, Rama Chellappa, and Ser Nam Lim. Regularizing deep net- works using efficient layerwise adversarial training. arXiv preprint arXiv:1705.07819 2018.
Tom Sercu, Christian Puhrsch. Brian Kingsbury, and Yann LeCun. Very deep multilingual convolu- tional neural networks for LVCSR. In 2016 IEEE International Conference on Acoustics. Speech IEEE, 2016.
and Signal Processing (ICASSP), pp.
10

---

Under review as a conference paper at ICLR 2020
Vatsal Shah, Anastasios Kyrillidis,, and Sujay Sanghavi. Minimum norm solutions do not always generalize well for over- wwrrpparamteerized problems. arXiv preprint arXiv:1811.07055, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Leslie Smith. A disciplined approach to neural network hyper-parameteet Part learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv: /803:098000000 2018.
Samuel Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe Lc. Don't 't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.0044 2017.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprini arXiv: (000005000,0.000000000000 2019.
Vishwak Srinivasan, Adepu Ravi Sankar, and Vineeth N Balasubramanian. Adine: an adaptive momentum method for stochastic gradient descent. In Proceedings of the ACM India Joint Inter- national Conference on Data Science and Management of Data, pp. 249-256. ACM, 2018.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. the importance initial- ization and momentum in deep learning. In International conference on machine learming, pp- 1139-1147, 2013.
Andre Wibisono and Ashia C Wilson. On accelerated methods in optimization. arXiv preprint 150900050..3666, 2015.
Andre Wibisono, Ashia C Wilson, and Michael Jordan. variational perspective on accelerated methods in optimization. proceedings of the National Academy of Sciences, 113(4))BB7511- E7358, 2016.
Ashia C Wilson, Benjamin Recht, and Michael Jordan. A Lyapumov analysis of momentum meth- ods in optimization. arXiv preprint arXiv: I. 1.02635, 2016.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 4148 4158, 2017.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Yuan. On the influence of momentum acceleration on online learning. Journal of Machine Learning Research, 17(192):1-66 16,11666222 2016.
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXin:1605.07146, 2016.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint 122i::1212.5701, 2012.
Jian Zhang and Ioannis Mitliagkas. Yellowfin and the art of momentum tuning. arXiv preprint arXiv: 0ae:l706.03771 2017.

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTS
We evaluated the momentum decay rule with Adam and SGDM on Residual CNNs, Non Residual CNNS, RNNs and generative models. For CNNs, we used the image classification datasets CI- FARIO, CIFAR100 and STL10 datasets. For RNNs, used the language modeling dataset PTB. For generative modeling, we used the MNIST and CIFARIO datasets. For each network dataset pair other than NSCN, evaluated Adam, QHAdam, AMSGrad, DEMON Adam, AggMo, QHM, DE- MON SGDM, and SGDM with learning rate decay. For adaptive learning rate methods and adaptive momentum methods, we generally perform grid search over the learning rate. For SGDM, we generally perform a grid search over learning rate and initial momentum. For SGDM learning rate decay, the learning rate is decayed by a factor of 0.1 after there is no improvement in validation loss for the best of [1, 2, 3, 5, 10, 20, 30, 40} epochs.
A.1 SETUP
We describe the six test problems in this paper.
CIFAR1O ResNet18 CIFARIO contains 60,000 32x32x3 images with a 50,000 training set, 10,000 test set split. There are 10 classes. ResNet18 (He al.) (016) is 18 layers deep CNN with skip connections for image classification. Trained with a batch size of 128.
CIFAR100 VGG16 CIFAR100 is fine-grained version of CIFAR-10 and contains 60,000 32x32x3 images with a 50,000 training set, 10,000 test set split. There are 100 classes. VGG16 (Simonyan & Zisserman 2014) is 16 layers deep CNN with extensive use of 3x3 convolutional filters. Trained with a batch size of 128
STL10 Wide ResNet 16-8 STL10 contains 1300 96x96x3 images with a 500 training set, 800 test set split. There are 10 classes. Wide ResNet 16-8 (Zagoruyko Komodakis 2016) is 16 layers deep ResNet which is 8 times wider. Trained with a batch size of 64.
PTB-LSTM PTB is an English text corpus containing 929,000 training words, 73,000 validation words, and 82,000 test words. There are 10,000 words in the vocabulary. The model is stacked LSTMs (Hochreiter & Schmidhuber, 1997) with layers, 650 units per layer, and dropout 0.5.
Trained with batch size of 20.
MNIST VAE MNIST contains 60,000 32x32x1 grayscale images with a 50,000 training set, 10,000 test set split. There are 10 classes of 10 digits. VAE (Kingma & Welling 2015) with three dense encoding layers and three dense decoding layers with a latent space of size 2. Trained with batch size of 100.
CIFAR10 NCSN CIFAR1O contains 60,000 32x32x3 images with 50,000 training set, 10,000 test set split. There are 10 classes. NCSN (Song & Ermon 2019) is a recent state-of-the-art generative model which achieves the best reported inception score. We compute inception scores based on total of 50000 samples. We follow the exact implementation in and defer details to the original paper.
A.2 METHODS
A.2.1
ADAPTIVE LEARNING RATE
Adam (Kingma & Ba, 2014), as previously introduced in section 2 keeps exponentially decaying average of squares of past gradients to adapt the learning rate. also introduces an exponentially decaying average of gradients.
The Adam algorithm is parameterized by learning rate > 0, discount factors 31 < 1 and B2 < 1, small constant e, and uses the update rule:
E441 = 31 Eg + (1- St.
E09 = 32 E + (1 - 32) (ge o gt),
6go9
0,41
O n
c9
1-11 Vt.
Ve
12

---

Under review as a conference paper at ICLR 2020
AMSGrad (Reddi et al. 2019) resolves issue in the proof of Adam related to the exponential moving average E7" where Adam does not converge for simple optimization problem. Instead of exponential moving average, AMSGrad keeps running maximum of go5g.
The AMSGrad algorithm is parameterized by learning rate 7) > O, discount factors 31 < and
82 < 1, a small constant c, and uses the update rule:
Et+1 B1 E + (1 -11) gt,
Ege $2 gg (1 82) (ge ot),
Egog = max Egog (+1,
1,i Et,0)
O 01,5 1fi.tt
& 11, 11,
where El and E441 are defined identically to Adam.
QHAdam (Quasi-Hyperbolic Adam) (Ma Yarats 2018) extends QHM (Quasi-Hyperbolic Mo- mentum), introduced further below, to replace both momentum estimators in Adam with quasi- hyperbolic terms. This quasi-hyperbolic formulation is capable recovering Adam and NAdam (Dozat 2016), amongst others.
The QHAdam algorithm is parameterized by learning rate 7 discount factors 31 and 82 < 1, 11, 1/2 E R, small constant c, and uses the update rule:
E 81 3 + (1 -1) gt.
E44 B2 _0 (1 32) (gt o gt),
E/ (1 + $ ) 3-1.20 Ehtn
1+1 (1+ 3 ggggg E(+1
(1 v1) gr + c++
01+1, 0.. -
Vt,
v u2)9t V2 Eg0g
+
where E4 and Eggg are defined identically to Adam.
A.2.2 ADAPTIVE MOMENTUM
AggMo (Aggregated Momentum) (Lucas et al. 2018) takes linear combination of multiple mo- mentum buffers. maintains K momentum buffers, each with a different discount factor, and averages them for the update.
The Ag algorithm parameterized by learning rate > 0, discount factors 8 E R K and uses the update rule:
( = (3)))(7)(66 + 9t.
Vi [],
K
0 = 01,5 -1
(
VL.
K
QHM HQuesiiHye Momentum) (Ma & Yarats 2018 is weighted average of the momentum and plain SGD. QHM capable recovering Nesterov Momentum (Nesterov) 1983). Synthesized Nesterov Variants (Lessard et al. 2016), accSGD (Jain et al. 2017) and others.
The QHM algorithm is parameterized by learning rate vi > 0, discount factor 3 < immediate discount factor E R, and uses the update rule:
Ei = 3 0. + (1 - (3-3)) St.
0 = 02, (1 v) gt + v. 63
E1+1
VL.
13

---

Under review as a conference paper at ICLR 2020
A.3
OPTIMIZER HYPERPARAMETERS
Table 8: Best parameters for CIFAR-10 with ResNet-18.
Optimization method
epochs
other parameters
Adam
30
0.001
Adam
75
0.001
81 = 0.9, 82 = 0.999
Adam
150
0.001
Adam
300
0.0003
AMSGrad
30
0.001
AMSGrad
0.001
81 0.9, 82 = 0.999
AMSGrad
150
0.001
AMSGrad
300
0.001
QHAdam
30
0.001
Adam
75
0.0003
P1 0.7, = 1.0, 31 = 0.9, 82 = 0.99
QHAAdmm
150
0.0003
QHAdam
300
0.0003
DEMON Adam
30
0.0001
DEMON Adam
0.0001
BAN 0.9, |32 0.999
DEMON Adam
150
0.0001
DEMON Adam
300
0.0001
AggMo
30
0.03
AggMo
75
0.01
8 [0, 0.99
AggMo
150
0.01
AggMo
300
0.01
QHM
30
1.0
QHM
0.7, 8 0.999
QHM
150
0.3
QHM
300
0.3
DEMON SGDM
30
0.1
DEMON SGDM
75
0.1
6mi = 0.9
DEMON SGDM
150
0.03
DEMON SGDM
300
SGDM learning decay
30
0.1
31 0.9, patience 5
SGDM learning rate decay
B1 = 0.9, patience
SGDM learning rate decay
150
0.1
B1 = 0.9, patience 20
SGDM learning rate decay
300
0.1
B1 = 0.9, patience 40
14

---

Under review as a conference paper at ICLR 2020
Table Best parameters for CIFAR-100 with VGG-16.
Optimization method
epochs
other parameters
Adam
75
0.0003
Adam
150
0.0003
81 0.9, 82 = 0.999
Adam
300
0.0003
AMSGrad
75
0.0003
AMSGrad
150
0.0003
81 0.9, 82 = 0.999
AMSGrad
300
0.0003
QHAdam
75
0.0003
QHAdam
1.50
0.0003
11 0.7, 12 1.0, 0.9, 82 0.99
QHAdam
300
0.0003
DEMON Adam
75
0.00003
DEMON Adam
150
0.00003
Ban = 0.9, f3 0.999
DEMON Adam
300
0.00003
AggMo
75
0.001
AggMo
150
0.001
8 = (0,00.9.00
AggMo
300
0.001
QHM
75
0.1
QHM
150
0.03
0.7, 9 0.999
QHM
300
0.03
DEMON SGDM
75
0.1
DEMON SGDM
150
0.03
Bui
DEMON SGDM
300
0.03
SGDM learning rate decay
75
patience
SGDM learning rate decay
150
0.03
31 0.9, patience = 20
SGDM learning rate decay
300
0.03
81 0.9, patience = 30
15

---

Under review as a conference paper at ICLR 2020
Table 10: Best parameters for STL10 with Wide ResNet 16-8.
Optimization method
epochs
Adam
50
0.001
Adam
100
0.0003
B1 0.9, |32 0.999
Adam
200
0.0003
AMSGrad
50
0.0003
AMSGrad
100
0.0003
81 0.9, 32 0.999
AMSGrad
200
0.0003
QHAdam
50
0.0003
QHAdam
100
0.0003
11 = 0.7, 12 1.0, 0.9, 0.99
QHAdam
200
0.0003
DEMON Adam
50
0.00003
DEMON Adam
100
0.00003
= = 0.999 0.999
DEMON Adam
200
0.00003
AggMo
50
0.03
AggMo
100
0.03
== [0,00...00999
AggMo
200
0.01
QHM
50
0.3
QHM
100
0.7, B 0.999
QHM
200
0.3
DEMON SGDM
50
DEMON SGDM
100
0.1
Smi = 0.9
DEMON SGDM
200
0.1
SGDM learning rate decay
50
0.1
31 0.9, patience = 10
SGDM learning rate decay
100
0.1
31 0.9, patience = 10
SGDM learning rate decay
200
0.1
31 0.9, patience 20
Table 11: Best parameters for PTB with LSTM architecture.
Optimization method
epochs
other parameters
Adam
25
0.0003
0.9, 82 0.999
Adam
39
0.0003
AMSGrad
25
0.001
0.9, 92 0.999
AMSGrad
39
0.001
QHAdam
25
0.0003
QHAdam
39
0.0003
1/1
0.7. 12 1.0, 0.9, 82 0.999
DEMON Adam
25
0.0001
Bus 0.9, 82 0.999
DEMON Adam
39
0.0001
AggMo
25
0.03
== 00. 0.99
AggMo
39
0.03
QII
25
1.0
V 0.7, 3 = 0.999
QHM
39
1.0
DEMON SGDM
25
1.0
Am 0.5. 8yinal - 0.5
DEMON SGDM
39
1.0
9mnx 0.3, 3 0.5
SGDM learning rate decay
25
0.1
31 0.9. smooth learning rate decay
SGDM learning decay
39
1.0
0.0. smooth learning rate decay
16

---

Under review as a conference paper at ICLR 2020
Table 12: Best parameters for MNIST with VAE.
Optimization method
epochs
other parameters
Adam
0.001
Adam
100
0.001
81 = 0.9, 52 0.999
Adam
200
0.001
AMSGrad
50
0.001
AMSGrad
100
0.001
81 0.9, 0| 0.999
AMSGrad
200
0.001
QHAdam
50
0.001
QHAdam
100
0.001
11 0.7, 1.0, 81 0.9, 93 0.99
QHAdam
200
0.001
DEMON Adam
0.0001
DEMON Adam
0.0001
Bun = 0.9, 03 = 0.999
DEMON Adam
200
0.0001
AggMo
50
0.000003
AggMo
100
0.000003
8= 10, 0.99
AggMo
200
0.000003
QHM
50
0,0001
QHM
100
0.00003
= 0.7, = 0.999
QHM
200
0.00003
DEMON SGDM
50
0.00001
DEMON SGDM
0.00001
Amn
DEMON SGDM
200
0.000003
SGDM learning rate decay
0.00001
31 0.9, patience
SGDM learning rate decay
100
0.000003
3, = 0.9, patience
SGDM learning rate decay
200
0.000003
81 0.9. patience 20
17

---

Under review as a conference paper at ICLR 2020
B CONVERGENCE ANALYSIS
We analyze the global convergence of DEMON CM in the convex setting, following (Ghadimi et al. 2014). For an objective function f which is convex, continuously differentiable, gradient V f-() is Lipschitz continuous with constant L, our goal is to show that f(Or) converges to the optimum f 1,., T. Our following theorem with decreasing momentum, where 0 is the average of 0g for t
holds for constant learning rate and Br decaying with L.
Theorem 1. Assume that f is convex, continuously differentiable, its gradient vf()) is Lipschitz continuous with constant L, with decreasing momentum, but constant step size, as in:
3 =
t+l 142
a E (0, 2t)
We consider the CM iteration in non-stochastic settings, where:
0++1 = 0g - (V.) + 3e (0, 0t-1).
Then, the sequence {0t]T= generated by the CM iteration, with decreasing momentum, satisfies:
f(8r) - 11 (1L+ 2
where Or is the Cesaro average of the iterates: Or + Et Oz.
Proof. Let B. = r+2 and
Pt ; (0. - 02-1) We consider the CM iteration non-stochastic settings, where:
oc+1 = 0g - f(,) + 3c (0, - 0(-1).
Using the definition of above, one can easily prove that:
8141 + Pi+1 (1 + 00+1 t+
0, = 02 Pt
a(++2)
f(.)).
t+1
Using this expression, will amalyze the term ||0,+1 Pt+1 10
|10-11 + pt+1 - 1¹112- = ||0: + Pc - 11 20(t+2) 1+1
(0, + pt 0" vf((.)) + (a((t2)) [VV5(()))1
|00 Pt 11
2all+2) [((-1)
(0, 0--1, v.(())))
2a(2+2) (0. 8*, f.(())) (
(
Since f is convex, continuously differentiable, its gradient is Lipschitz continuous with constant L, then
(I < (0, - 8* f.((.)). f(0.) - fr + 25 (V < (0: 8", Vf(())). S(0.) (0, < (0: O VS().))
(2)
(3)
(4)
Substituting the above inequalities leads to
11 + Pi+1 1- < ||0: Pt- 1- 2a(l+2) (f(0.) f(0-11)
((++1)
2a (1- x)++2)
22 - -f" [V)))
T(++1)
a(1 (a(l+)) 2) 2
XL+2) ((
|V +
(Vf(0))
+++1
where x E (0, 1] is a parameter weighting 2 and 3 Grouping together terms yields
(0((+2) + 201(1422) (f(Ot) f*) + ||0 + Ptt1 1-11 2a(l+2) < a(+1)
(((+))
(f(0)-1) - ( + 110 + Pt: - 0²2
a(t-2)
[((++) ((1)) 2) [Vf(()1
++1 ++1
18

---

Under review as a conference paper at ICLR 2020
The last term is non-positive when E [0,
141 (2z))] so it can be dropped. Summing t
1+2
,,... T yields
20
Σ (1()
2all+2)
(f(O.) ff) + |10++1 +P++1 8²11²) ((f+1)
2a(l+2)
u(++) (f(6)-1) f*) + 110 + p. - -0-12)
implying that:
2ad
Σ
f3
110 1-11
Since:
T 2nd ((r(0)) - s 2nd Σ
T
(f(0,) 30 Σ (f(O)) - f³),
1=1
we further have:
3a
Σ
- f3) -5)
(3a(f(0)) -f) + 110 -8113)
++)++1++
1=1
Due to the convexity of f.
f(0.) s 1 Σ f(0.).
1=1
observe that
T
[(Or)- - 1" S +
Σ (S(O.) - - TयxT (2a(S(0)) - 1*) + +110 - 0-113).
t=1
Since [(0i)- f < 40
11 by Lipschitz continuous gradients, setting À and observing (1+ 1)/(1+)/ + 2 2/3 gives the result.
19

---

Under review as a conference paper at ICLR 2020
C
ADDITIONAL PLOTS
Theining validation loss
Triihing and valditiion loss
Vordaaen houa
0.0
Additional empirical results on adaptive learning rate methods. Left plot: VGG1 6-CT6FCF11100 DO-FFRRR0-DDMONAAC for 300 epochs. Right plot: WRN-STL1 O-DEMONAdam for 200 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent standard deviation.
Figure
Thaining and velicitiion lows
Traiting and velticiion eETOTS
0.43
1.4
ACOPT
0.35
335
0.30
USMON Adum Ideaa slatiig
BEMCN Adam idecay startin at epochs)
030
033
430
3.0
003
3.5
10
15
Training and valicition loss
Training and val ducion CrTOTS
0.35
0.33
430
0.71
USMUN baam Heaay startig spoct
DEMON Adam Isecay starting at 75% spocts)
-35
0.-
032
001
425
3.0
000
3.0
20
20
Training and valicition loso
Training and validacion errors
0.57
Thenerg (3ttwe))
DIMON oonn (secay stating epect in
8.00
DIMON adam Iecday Stariin at spoctsl -300
bhaneeg MIe
002
330
3.0
080
200
1.20
1.00
102
3.0
1.0
1p20he
Top row: 30 epochs. Middle row: 75 epochs. Bottom row: 150 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent standard deviation.
Figure 5: Additional empirical results RN1 D0-CIFAR10-DEMOOOON
20

---

Under review as a conference paper at ICLR 2020
Training and valication l005
Traiting and valiarion erior
10
3.
Thaato (6steet
DTACH Jdwn (secay eartig at spert to
32-
10
04
DAMON adam locaa stattng at 75% cpoctsl
30
92
10
20
30
03
50
sp003s
8020ha
Training and valicatton loss
Traihing and valideiion eriors
4.-
10
AMSSrad
Thanro Dettees
EEMCH Adan (decay sardin at opect 0
32-
10
Adam inecay statt epochsl
03
02-
01
60
300
170
1.00
20
60
90
80
100
170
140
s000hs
1850ha
Figure 6: Additional empirical results Bottom row: 150 epochs. Dotted and solid lines represent training and generalization metrics re- spectively. Shaded bands represent 1 standard deviation.
Top row: 75 epochs.
Training and valicatton loss
Traihing and validecion errors
2.0
250
0.43
1.0
AMSCred
1.3
DEMCH Adan (decay sarding at opect 0
DSMON Adam idecay strting epocts)
1.25
027
1.0
0.5
052
1.00
033
0.2
032
002
30
42
10
30
10507>
Thaining and velicittion losk
Training and oeliderion erroes
1.0
0.00
330
226
66125
6.000
0.4
6.50
AMSCme
2.6
6025
DEHOR Jdam tieccs ting epect on
CEHOR Asam idecce ting at 75% exsches)
230
42
00
00
100
20
42
60
80
100
18003>
19407>
Figure 7: Additional empirical results on WRN DEMONAAa.......m......m........ Top row: 50 epochs. Bottom row: 100 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent standard deviation.
21

---

Under review as a conference paper at ICLR 2020
Training and validat perpieiity
Acart
Trainin pariaaty LOUMNR
10
30
320
opocha
Figure 8: Additional empirical results on DTE-LSST--EEMONAA for epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent standard deviation.
3500
2900
147.5
Quiders
Oilddn
AMSCred
EEACH idecay eartin at spoct
(decay martin spect
142.5
1460
-400
135.
1350
1825
1325
152
Figure 9: Additional empirical results on VAE epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent standard deviation.
Left: 50 epochs. Right: 100
Thaining and velication loos
Thaining and selletionn loas
1.4
Traniro cetted
3.0
2.5
Od cutt 10C
2.0
00
3.0
125
222
300
200
125
200
69cc85
084077
Figure 10:
Additional empirical results on adaptive momentum methods. Left plot: VGG1 RBG16-CIFAR100000 DFFFRRROODDEMONGGDM for 300 epochs. Right plot: WRN-STL10 DDSSLLIDDDMMOSCDA for 200 epochs. Dotted and solid lines represent training and generalization metries respectively. Shaded bands represent standard deviation.
22

---

Under review as a conference paper at ICLR 2020
Training valication
Traihing
Off
Appio
neminn decy
0.30
OONOR
0.0
023
425
Tranre
0.20
2.4
001
Training velicatton
Traihing validation
Off
1.2
0.71
Poptt:
wemiing decay
425
DENOH
0.2
0.5
Tranteg
033
410.5
2.4
002
Training and veliditioo look
Training and validerion
CH1
2.8
0.27
seurning dacay
025
Tranteo testeed
DENOR
0.33
0.0
230.
002
Figure II: Additional empirical results on RN18 CIFAR1O DEMONSSGM.
-DMNNIDDDNNNNNNN Top row: 30 epochs. Middle row: 75 epochs. Bottom row: 150 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent standard deviation.
23

---

Under review as a conference paper at ICLR 2020
Training and valication lo0s
Traiting and validaiion errors
3.0
35
233
04
30
Treerr
GPN
2.0
CENCO SOGM Mocaa stating spean
CePiCh SCCM (accay stattng at 7..5% spocts)
30
92
1.0
50
SpC032
8p00ha
Training and valicatton loss
Traihing and validacion errors
5.-
1.0
36
SCEM
DENON SCOM (decay martin at opect 0)
GENON itecay sortt epechsl
2.-
03-
Trenre
2.0
32
00
60
300
170
1.0
20
60
90
80
100
170
140
s000hs
Figure 12: Additional empirical results 0100110--2AR11--EEONSSSSSS Top 75 epochs. Bottom row: 150 epochs. Dotted and solid lines represent training and generalization metrics re- spectively. Shaded bands represent 1 standard deviation.
Training and valicatton loss
Training and validation errors
280
250
0.4
3.00
1.3
1.25
0.2
1.0
0.25
OUN
-12
0.2
2.00
0.33
SCDM
0.5
CEHOR SoDM dooaa stating zpoch d
CEnION SUDM (ocaa sorting 15% spocts)
032
30
10
30
50
sp00hs
spsott
Thaining and velicittion loss
Training and diniinn errnT
2.02
250
0.42
340
SCEM
DEMOR SOOM (docay starting spect 0)
DENON (dacay staring spectal
3.25
023
1.02
0.75
0.2
033
0.2
0.2
42
00
00
100
20
42
00
100
08003>
Figure 13: Additional empirical results WRN-STLL STL10 0 DEMMONSGDM Top row: 50 epochs. Bot- tom row: 100 epochs. Dotted and solid lines represent training and generalization metrics respec- tively. Shaded bands represent 1 standard deviation.
24

---

Under review as a conference paper at ICLR 2020
Training validation
AgyM:
975
SCEP laarring rale schocule
GCEM (deowy chaiting apach
500
82.
Figure 14: Additional empirical results on PTB-LSTM-DEMONEE for 39 epochs. Dotted and solid lines represent training and generalization metries respectively. Shaded bands represent standard deviation.
Training valicatton
Tratning
Of12
City
152.5
Applic
:225
152.5
Appii
1525
wariing dicky
nouming decyy
1100
2100
1500
(EHOR
DONOR
145.0
143.0
$45.0
143.0
142.5
5025
1973
1375
1975
1375
125.8
1154
135.0
Figure 15: Additional empirical results on VAE VAR-WISST-DEMNSOOW Left: 50 epochs. Right: 100 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent standard deviation.
25


Reviews:
[{"review_id": "HJguTJHpKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new decaying momentum rule to further improve neural network training. The idea is motivated by decaying the total contribution of a gradient to all future updates. The authors also extend this idea on to Adam and show that it improved upon the vanilla Adam.\n\n- The intuition of using momentum decaying scheme is not quite clear to me. Why having a linear momentum decay schedule is better? This question is not answered in the paper. I hope the authors could provide more illustrative explanations regarding this.\n\n- This paper focused essentially on empirical evaluations. I do appreciate that the authors conduction various experiments using different dataset and architecture but I do not understand why the authors want to separate the experiments into parts: adaptive methods and adaptive momentum methods. Not to mention the confusing names, it makes no sense to say the same words again for different optimizers. I would suggest the authors to combine the results together for better comparison.\n\n- CM formulation mentioned in the paper is actually different from the commonly refer SGD with momentum implementation, which is the optimizer that is widely used in the community. Therefore, it is important to at least add SGD with momentum as one baseline in the experiments. \n\n- According to https://github.com/kuangliu/pytorch-cifar, Resnet18 on CIFAR10 can achieve at least 93% accuracy using simple SGD with momentum. It seems that the current reported results are not fully optimized. I would suggest the authors to check the parameter settings and make sure all the hyperparameters for baseline methods are fully tuned.\n\n- Aside from SGD with momentum, the experiments part also lacks several important baselines. The author should also consider comparing the AdamW, Padam mentioned in the paper.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bye55GlTKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose Demon schedule that automatically decays momentum. The paper is very well written, and the experiments are performed on an, extensive, compared to typical papers in the domain, suite of tasks. Unfortunately, while the paper is technically well executed, I have fundamental issues with novelty. Based on the current state-of-the-art understanding of optimization in deep learning, it is quite expected that decaying momentum has an analogous effect to decaying learning rate or increasing batch size (see [1,2]). Given this \"Similarly, applying DEMON to momentum SGD rivals momentum SGD with learning rate decay\" is not surprising, and \"and in many cases leads to improved performance\" warrarnts a bit of scepticism.\n\nMore precisely, a very similar analysis as in \"Motivation and interpretation.\" can be already found in [1]. Similarly, [2] already suggests decreasing momentum. Based on this, experiments in Table. 2 (and the other similar Tables) should compare to decaying learning rate or batch size in Adam and other adaptive methods using an analogous schedule. It might be surprisng, but adaptive methods do benefit from learning rate schedules. Analogously, could you please compare performance of Demon to the recommendation in [2] to decay momentum together with increasing learning rate initially? \n\nUnless a more detailed experiments demonstrate that tuning momentum brings additional benefit on top of its effect on the effective learning rate, there isn't unfortunately in my onion enough practical value in the proposed method.\n\nReferences:\n\n[1] Smith et al, Don't Decay the Learning Rate, Increase the Batch Size, https://arxiv.org/abs/1711.00489\n[2] Fast.ai documentation on one cycle method, https://docs.fast.ai/callbacks.one_cycle.html", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xkhtvaKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the author propose a decaying momentum rule to improve algorithm. Furthermore, he apply this rule in momentum SGD and Adam, then use experiment to prove the algorithm.\n\nIn the experiment, it training on many different dataset and compare with many baseline. The algorithm with decaying momentum rule get much better result than all other algorithm. Furthermore, the paper is well written and east to follow.\n\nHowever, I still have a question about this paper. \n\nEven the algorithm get a good performance, the intuition of the algorithm is not clear. In fact, it is reasonable to decay the total contribution of a gradient to all future updates; it is still unclear why you choose such $\\beta$. As you say, the contribution of previous item is $ \\sum \\beta^i=\\frac \\beta {(1-\\beta)} $, however, this is only correct why $\\beta$ is constant, and when the $\\beta_t$ is change like your definition, what will the contribution become? Furthermore, even the contribution of previous item is $ \\frac \\beta {(1-\\beta)} $, why you want the contribution is equal to $ (1-\\frac t T)\\beta_{init}/(1-\\beta_{init})$, it may need more comment about why you choose the value.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new decaying momentum rule to improve existing optimization algorithms for training deep neural networks, including momentum SGD and Adam. The main objections from the reviewers include: (1) its novelty is limited compared with prior work; (2) the experimental comparison needs to be improved (e.g., the baselines might not be carefully tuned, and learning rate decay is not applied, while it usually boosts the performance of all the algorithms a lot). After reviewer discussion, I agree with the reviewers’ evaluation and recommend reject.
Decision: Reject