Title: DECAYING MOMENTUM HELPS NEURAL NETWORK TRAINING

Abstract: Momentum is a simple and popular technique in deep learning for gradient-based optimizers. We propose a decaying momentum (DEMON) rule, motivated by decaying the total contribution of a gradient to all future updates. Applying DEMON to Adam leads to significantly improved training, notably competitive to momentum SGD with learning rate decay, even in settings in which adaptive methods are typically non-competitive. Similarly, applying DEMON to momentum SGD rivals momentum SGD with learning rate decay, and in some cases leads to improved performance. DEMON is trivial to implement and incurs limited extra computational overhead, compared to the vanilla counterparts.

1. INTRODUCTION:

Deep Neural Networks (DNNs) have drastically advanced the state-of-the-art performance in many computer science applications, including computer vision (Krizhevsky et al., 2012) , (He et al., 2016; Ren et al., 2015) , natural language processing (Mikolov et al., 2013; Bahdanau et al., 2014; Gehring et al., 2017) and speech recognition (Sak et al., 2014; Sercu et al., 2016) . Yet, in the face of such significant developments, the age-old stochastic gradient descent (SGD), and the accelerated variant SGD with momentum (SGDM), algorithm remains one of the most, if not the most, popular method for training DNNs (Sutskever et al., 2013; Goodfellow et al., 2016; Wilson et al., 2017) .

Adaptive methods (Duchi et al., 2011; Zeiler, 2012; Hinton et al., 2012; Kingma & Ba, 2014; Ma & Yarats, 2018) sought to simplify the training process, while providing similar performance. However, while they are often used by practitioners, there are cases where their use leads to a performance gap (Wilson et al., 2017; Shah et al., 2018) . At the same time, much of the state-of-the-art performance on highly contested benchmarks-such as the image classification dataset ImageNet-have been produced with SGDM (Krizhevsky et al., 2012; He et al., 2016; Xie et al., 2017; Zagoruyko & Komodakis, 2016; Huang et al., 2017; Ren et al., 2015; Howard et al., 2017) .

Nevertheless, a key factor in any algorithmic success still lies in hyperparameter tuning. For example, in the literature above, they obtain such performance with a well-tuned SGD with momentum and a learning rate decay schedule, or with a proper hyperparameter tuning in adaptive methods. Slight changes in learning rate, learning rate decay, momentum, and weight decay (amongst others) can drastically alter performance. Hyperparameter tuning is arguably one of the most time consuming parts of training DNNs, and researchers often resort to a costly grid search. Thus, finding new and simple hyper-parameter tuning routines that boost the performance of state of the art algorithms is of ultimate importance and one of the most pressing problems in machine learning.

The focus of this work is on the momentum parameter and how we can boost the performance of training methods with a simple technique. Momentum helps speed up learning in directions of low curvature, without becoming unstable in directions of high curvature. Minimizing the objective function L(•), the simplest and most common momentum method, SGDM, is given by the following recursion for variable vector θ t ∈ R p : θ t+1 = θ t + ηv t , v t = βv t-1 -g t . The coefficient β-traditionally, selected constant in [0, 1]-controls how quickly the momentum decays, g t represents a stochastic gradient, usually E[g t ] = ∇L(θ t ), and η > 0 is the step size.

But how do we select β? The most prominent choice among practitioners is β = 0.9. This is supported by recent works that prescribe it (Chen et al., 2016; Kingma & Ba, 2014; Hinton et al., 2012; Reddi et al., 2019) , and by the fact that most common softwares, such as PyTorch (Paszke et al., 2017) , declare β = 0.9 as the default value in their optimizer implementations. However, there is no indication that this choice is universally well-behaved.

There are papers that attempt to tune the momentum parameter. Under an asynchronous distributed setting, (Mitliagkas et al., 2016) observe that running SGD asynchronously is similar to adding a momentum-like term to SGD; they also provide experimental evidence that naively setting β = 0.9 would result in a momentum "overdose", leading to suboptimal performance. As another example, YellowFin (Zhang & Mitliagkas, 2017 ) is a learning rate and momentum adaptive method for both the synchronous and asynchronous setting, motivated by a quadratic model analysis and some robustness insights. The main message of that work is that, like η, momentum acceleration needs to be carefully selected based on properties of the objective, the data, and the underlying computational resources. Finally, moving from classical DNN settings towards generative adversarial networks (GANs), the proposed momentum values tend to decrease from β = 0.9 (Mirza & Osindero, 2014; Radford et al., 2015; Arjovsky et al., 2017) , taking even negative values (Gidel et al., 2018) .

In this paper, we introduce a novel momentum decay rule which significantly surpasses the performance of both Adam and SGDM (as they are used currently), in addition to other state-of-the-art adaptive learning rate and adaptive momentum methods, across a variety of datasets and networks. In particular, our findings can be summarized as follows:

i) We propose a new momentum decay rule, motivated by decaying the total contribution of a gradient to all future updates, with limited overhead and additional computation. ii) Using the momentum decay rule with Adam, we observe large performance gains-relative to vanilla Adam-where the network continues to learn for far longer after Adam begins to plateau, and suggest that the momentum decay rule should be used as default for this method. iii) We observe comparative performance for SGDM between momentum decay and learning rate decay; an interesting result given the unparalleled effectiveness of learning rate decay schedule. (Hochreiter & Schmidhuber, 1997) , Variational AutoEncoders (VAE) (Kingma & Welling, 2015) , and the recent Noise Conditional Score Network (NCSN) (Song & Ermon, 2019) .

2. PRELIMINARIES:

Plain stochastic gradient descent motions. Let θ t ∈ R p be the parameters of the network at time step t, where η ∈ R is the learning rate/step size, and g t is the stochastic gradient w.r.t. θ t for empirical loss L(•), such that E[g t ] = ∇L(θ t ). Then, plain stochastic gradient descent (SGD) uses the recursion:

θ t+1 = θ t -η • g t , ∀t.

Here, the step size η could also be time dependent, η t , but practice shows that decreasing the value of η at regular or predefined intervals works favorably compared to decreasing the value of η at every iteration.

SGDM is parameterized by β ∈ R, the momentum coefficient, and follows the recursion:

θ t+1 = θ t + ηv t , v t = βv t-1 -g t ,

where v t ∈ R p accumulates momentum. Observe that for β = 0, the above recursion is equivalent to SGD. Common values for β are closer to one, with β = 0.9 the most used value (Ruder, 2016) .

Adaptive gradient descent motions. These algorithms utilize current and past gradient information to design preconditioning matrices that better approximate the local curvature of L(•). Beginning with AdaGrad (Duchi et al., 2011) , the SGD recursion, per coordinate i of θ, becomes:

θ t+1,i = θ t,i - η √ Gt,ii+ε • g t,i , ∀t,

where G t ∈ R p×p is usually a diagonal preconditioning matrix as a summation of squares of past gradients, and ε > 0 a small constant.

RMSprop (Hinton et al., 2012) substitutes the ever accumulating matrix G t with a root mean squared operation. Denoting the average of squared gradients as E g•g t

, per iteration we compute:

E g•g t+1 = β 2 • E g•g t + (1 -β 2 ) • (g t • g t ),

where β 2 was first proposed as 0.9. Here, • denotes the per-coordinate multiplication. Then, RMSprop updates as-where a momentum term can also be optionally added:

θ t+1,i = θ t,i - η √ E g•g t+1,i +ε • g t,i , ∀t.

Finally, Adam (Kingma & Ba, 2014) , in addition, keeps an exponentially decaying average of past gradients:

E g t+1 = β 1 • E g t + (1 -β 1 ) • g t , leading to the recursion: 1 θ t+1,i = θ t,i - η √ E g•g t+1,i +ε • E g t+1,i , ∀t,

where usually β 1 = 0.9 and β 2 = 0.999. Observe that Adam is equivalent to RMSprop when β 1 = 0, and when no bias correction is applied results in the same recursion.

3. DEMON: DECAYING MOMENTUM ALGORITHM:

Algorithm 1 DEMON in SGDM 1: Parameters: # of iterations T , step size η, momentum initial value β init . 2: v t = θ t = 0, otherwise randomly initialized. 3: for t = 0, . . . , T do 4: 

β t = β init • 1- t T (1-βinit)+βinit(1-t T ) 5: θ t+1 = θ t -ηg t + β t v t 6: v t+1 = β t v t -

β t = β init • 1- t T (1-βinit)+βinit(1-t T ) 5: E g•g t+1 = β 2 • E g•g t + (1 -β 2 ) • (g t • g t ) 6: m t,i = g t,1 + β t m t-1,i 7: θ t+1,i = θ t,i - η √ E g•g t+1,i +ε • m t,i

8: end for Motivation and interpretation. DEMON is motivated by linear learning rate decay models which reduce the impact of a gradient to current and future updates. By decaying the momentum parameter, we decay the total contribution of a gradient to all future updates. Our goal here is to present a concrete, effective, and easy-to-use momentum decay procedure which we show in the experimental section. By Occam's Razor, a linear decay achieves such a goal since it is simple and requires no tuning in most cases. The key component is the momentum decay schedule:

β t = β init • (1-t T ) (1-βinit)+βinit(1-t T )

.

(1)

Above, the fraction (1 -t/T ) refers to the proportion of iterations remaining. The interpretation of this rule comes from the following argument: Assume fixed momentum parameter β t ≡ β; e.g., β = 0.9, as literature dictates. For our discussion, we will use the SGDM recursion. We know that v 0 = 0, and v t = βv t-1 -g t . Then, the main recursion can be unrolled into:

θ t+1 = θ t -ηg t -ηβg t-1 -ηβ 2 g t-2 + ηβ 3 v t-2 = • • • = θ t -ηg t -η • t i=1 β i • g t-i

Interpreting the above recursion, a particular gradient term g t contributes a total of η i β i of its "energy" to all future gradient updates. Moreover, for an asymptotically large number of iterations, we know that β contributes on up to t-1 terms. Then,

∞ i=1 β i = β ∞ i=0 β i = β/(1-β).

Thus, in our quest for a decaying schedule and for a simple linear momentum decay, it is natural to consider a scheme where the cumulative momentum is decayed to 0. Let β init be the initial β; then at current step t with total T steps, we design the decay routine such that: β/(1-β) = (1-t/T )β init /(1-β init ). This leads to equation 1. Although β changes in subsequent iterations, this is typically a very close approximation since β i β i+1 . . . β t for a particular g i diminishes much faster than β changes.

Connection to previous algorithms. DEMON introduces an implicit discount factor. The main recursions of the algorithm are the same with standard algorithms in machine learning. E.g., for β t = β = 0.9 we obtain SGD with momentum, and for β = 0 we obtain plain SGD in Algorithm 1; in Algorithm 2, for β 1 = 0.9 with a slightly adjustment of learning rate we obtain Adam, while for β 1 = 0 we obtain a non-accumulative AdaGrad algorithm. We choose to apply DEMON to a slightly adjusted Adam-instead of vanilla Adam-to isolate the effect of the momentum parameter, since the momentum parameter adjusts the magnitude of the current gradient as well in vanilla Adam.

Efficiency. DEMON requires only limited extra overhead and computation in comparison to the vanilla counterparts, for the computation of β t .

Practical suggestions. For settings in which β init is typically large, such as image classification, we advocate for decaying momentum from β init at t = 0, to 0 at t = T as a general rule. We also observe and report improved performance by delaying momentum decay till later epochs. In many cases, performance can be further improved by decaying to a small negative value, such as 0.3.

4. RELATED WORK:

There are numerous techniques for automatic hyperparameter tuning. The most widely used are learning rate adaptive methods, starting with AdaGrad (Duchi et al., 2011) , AdaDelta (Zeiler, 2012), RMSprop (Hinton et al., 2012), and Adam (Kingma & Ba, 2014) . Adam (Kingma & Ba, 2014) , the most popular, introduced a momentum term, which is combined with the current gradient before multiplying with an adaptive learning rate. Interest in closing the generalization difference between adaptive methods and SGDM led to AMSGrad (Reddi et al., 2019) , replacing the exponential moving average of squared gradients with the maximum, QHAdam (Ma & Yarats, 2018) , a variant of QHM which is capable of recovering a variety of optimization algorithms, AdamW (Loshchilov & Hutter, 2017) , by fixing the weight decay of Adam, and Padam (Chen & Gu, 2018) , by lowering the exponent of the second moment.

Asynchronous methods are commonly used in deep learning, and (Mitliagkas et al., 2016) show that running SGD asynchronously is similar to adding a momentum-like term to SGD without assumptions of convexity of the objective function. They demonstrate this natural connection empirically on CNNs. This implies that the momentum parameter needs to be tuned according to the level of asynchrony. YellowFin (Zhang & Mitliagkas, 2017 ) is a learning rate and momentum adaptive method for both the synchronous and asynchronous setting motivated by a quadratic model analysis and robustness insights. In the non-convex setting, STORM (Cutkosky & Orabona, 2019 ) uses a variant of momentum for variance reduction.

There is substantial research, both empirical and theoretical, into the convergence of momentum methods (Wibisono & Wilson, 2015; Wibisono et al., 2016; Wilson et al., 2016; Kidambi et al., 2018) . In addition, (Sutskever et al., 2013) explored momentum schedules, with even increasing momentum schedules during training, inspired by Nesterov's routines for convex optimization. (Smith et al., 2017) explores increasing batch size instead of decreasing the learning rate, as well as scaling batch size to increasing learning rate or increasing momentum. (Smith, 2018) introduces cycles of simultaneously increasing learning rate and decreasing momentum followed by simultaneously decreasing learning rate and increasing momentum. There is some work into reducing oscillations during training, by adapting the momentum (Odonoghue & Candes, 2015) . There is also work into adapting momentum in well-conditioned convex problems as opposed to setting to zero (Srinivasan et al., 2018) . Another approach in this area is to keep several momentum vectors according to different β and combining them (Lucas et al., 2018) . We are aware of the theoretical work of (Yuan, 2016) which prove under certain conditions that SGDM is equivalent to SGD with a rescaled learning rate, however our experiments in the deep learning setting show slightly different behavior and understanding why is an exciting direction of research.

Smaller values of β have gradually been employed for Generative Adversarial Networks (GAN), and recent developments in game dynamics (Gidel et al., 2018) show a negative momentum is helpful.

5. EXPERIMENTS:

We separate experiments into those with adaptive learning rate and those with adaptive momentum. All settings, with exact hyper-parameters, are briefly summarized in 

5.1. ADAPTIVE METHODS:

At first, we apply DEMON Adam (Algorithm 2) to a variety of models and tasks. We select vanilla Adam as the baseline algorithm and include more recent state-of-the-art adaptive learning rate methods Quasi-Hyperbolic Adam (QHAdam) (Ma & Yarats, 2018) and AMSGrad (Reddi et al., 2019) in our comparison. See Appendix A.2.1 for details. We tune all learning rates in roughly multiples of 3 and try to keep all other parameters close to those recommended in the original literature. For DEMON Adam, we leave β init = 0.9, β 2 = 0.999 and decay from β init to 0 in all experiments.

Residual Neural Network (RN18-CIFAR10-DEMONAdam). We train a ResNet18 (He et al., 2016) model on the CIFAR-10 dataset. With DEMON Adam, we achieve the generalization error reported in the literature (He et al., 2016) for this model, attained using SGDM and a curated learning rate decay schedule, whilst all other methods are non-competitive. Refer to Table 2 for exact results.

In Figure 2 (Top row, two left-most plots), DEMON Adam is able to learn in terms of both loss and accuracy after other methods have plateaued. Running 5 seeds, DEMON Adam outperforms all other methods by a large 2%-5% generalization error margin with a small and large number of epochs.

Non-Residual Neural Network (VGG16-CIFAR100-DEMONAdam). For the CIFAR-100 dataset, we train an adjusted VGG-16 model (Simonyan & Zisserman, 2014) . Similarly to the previous setting, we observe similar learning behavior of DEMON Adam, where it continues to improve after other methods appear to begin to plateau. We note that this behavior results in a 1-3% decrease in generalization error than typically reported results with the same model and task (Sankaranarayanan et al., 2018) , which are attained using SGDM and a curated learning rate decay schedule.

Running 5 seeds, DEMON Adam achieves an improvement of 3%-6% generalization error margin over all other methods, both for a small and large number of epochs. Refer to Figure 2 (Top row, right-most plot) and Table 3 for more details.

Wide Residual Neural Network (WRN-STL10-DEMONAdam). The STL-10 dataset presents a different challenge with a significantly smaller number of images than the CIFAR datasets, but in higher resolution. We train a Wide Residual 16-8 model (Zagoruyko & Komodakis, 2016) for this task. In this setting, we note again the behavior of DEMON Adam significantly outperforming other methods in the latter stages of training.

Running 5 seeds, DEMON Adam outperforms all other methods by a 0.5%-2% generalization error margin with a small and large number of epochs. Refer to Figure 2 (Bottom row, left-most plot) and Table 3 for more details. LSTM (PTB-LSTM-DEMONAdam). Language modeling can have gradient distributions which are sharp; for example, in the case of rare words. We use an LSTM (Hochreiter & Schmidhuber, 1997) model to this task. We observe overfitting for all adaptive methods.

Similar to above, running 5 seeds, DEMON Adam outperforms all other methods by a 6-14 generalization perplexity margin. Refer to Figure 2 (Bottom row, middle plot) and Table 4 for details.

Variational AutoEncoder (VAE-MNIST-DEMONAdam). Generative models are a branch of unsupervised learning that try to learn the data distribution. VAEs (Kingma & Welling, 2015) pair a generator network with a second Neural Network, a recognition model that performs approximate inference, and can be trained with backpropagation. We train VAEs on the MNIST dataset.

Running 5 seeds, DEMON Adam outperforms all other methods, particularly for smaller number of epochs. Refer to Figure 2 (Bottom row, right-most plot) and Table 4 for more details.

Noise Conditional Score Network (NCSN-CIFAR10-DEMONAdam). NCSN (Song & Ermon, 2019 ) is a recent generative network achieving state-of-the-art inception score on CIFAR10. NCSN estimates the gradients of the data distribution with score matching. Samples are then produced via Langevin dynamics using those gradients. We train a NCSN on the CIFAR10 dataset and, using the official implementation, were unable to reproduce the reported score in the literature. NSCN trained with Adam achieves a superior inception score in Table 4 , however the produced images in Figure 1 exhibit a noticeably unnatural green compared to those produced by DEMON Adam. 

5.2. ADAPTIVE MOMENTUM METHODS:

We apply DEMON SGDM (Algorithm 1) to a variety of models and tasks. Since SGDM with learning rate decay is most often used to achieve the state-of-the-art results with the architectures and tasks in question, we include SGDM with learning rate decay as the target to beat. SGDM with learning rate decay is implemented with a decay on validation error plateau, where we hand-tune the number of epochs to define plateau. Recent adaptive momentum methods included in this section are Aggregated Momentum (AggMo) (Lucas et al., 2018) , and Quasi-Hyperbolic Momentum (QHM) (Ma & Yarats, 2018) . We exclude accelerated SGD (Jain et al., 2017) due to difficulties in tuning. See Appendix A.2.2 for details. Similar to the last section, we tune all learning rates in roughly multiples of 3 and try to keep all other parameters close to those recommended in the original literature. For DEMON SGDM, we leave β init = 0.9 for most experiments and generally decay from β init to 0.

Residual Neural Network (RN18-CIFAR10-DEMONSGDM). We train a ResNet18 model on the CIFAR-10 dataset. With DEMON SGDM, we achieve better generalization error than SGDM with learning rate decay, the optimizer for producing state-of-the-art results with ResNet architecture. The better performance of decaying momentum relative to learning rate decay is surprising.

Running 5 seeds, DEMON SGDM outperforms all other adaptive momentum methods by a large 3%-8% validation error margin with a small and large number of epochs and is competitive or better than SGDM with learning rate decay. In Figure 3 (Top row, two left-most plots), DEMON SGDM is observed to continue learning after other adaptive momentum methods appear to begin to plateau. Non-Residual Neural Network (VGG16-CIFAR100-DEMONSGDM). For the CIFAR-100 dataset, we train an adjusted VGG-16 model. In Figure 3 (Top row, right-most plot), we observe DEMON SGDM to learn slowly initially in loss and error, but similar to the previous setting it continues to learn after other methods begin to plateau, resulting in superior final generalization error.

Running 5 seeds, DEMON SGDM achieves an improvement of 1%-8% generalization error margin over all other methods. Refer to Table 6 for more details.

Wide Residual Neural Network (WRN-STL10-DEMONSGDM). We train a Wide Residual 16-8 model for the STL-10 dataset. In Figure 3 LSTM (PTB-LSTM-DEMONSGDM). We train an RNN with LSTM architecture for the PTB language modeling task. Running 5 seeds, DEMON SGDM slightly outperforms other adaptive momentum methods in generalization perplexity, and is competitive with SGDM with learning rate decay. Refer to Figure 3 (Bottom row, middle plot) and Table 7 for more details.

Variational AutoEncoder (VAE-MNIST-DEMONSGDM). We train the generative model VAE on the MNIST dataset. Running 5 seeds, DEMON SGDM outperforms all other methods by a 2%-6% generalization error for a small and large number of epochs. Refer to Figure 3 (Bottom row, right-most plot) and Table 7 for more details. 

6. CONCLUSION:

We show the effectiveness of the proposed momentum decay rule, DEMON, across a number of datasets and architectures. The adaptive optimizer Adam combined with DEMON is empirically substantially superior to the popular Adam, in addition to other state-of-the-art adaptive learning rate algorithms, suggesting a drop-in replacement. Surprisingly, it is also demonstrated that DEMON SGDM is comparable to SGDM with learning rate decay. In cases where budget is limited, DEMON SGDM may be preferable. DEMON is computationally cheap, easy to understand and use, and we hope it is useful in practice and as a subject of future research.

A EXPERIMENTS:

We evaluated the momentum decay rule with Adam and SGDM on Residual CNNs, Non Residual CNNS, RNNs and generative models. For CNNs, we used the image classification datasets CI-FAR10, CIFAR100 and STL10 datasets. For RNNs, we used the language modeling dataset PTB. For generative modeling, we used the MNIST and CIFAR10 datasets. For each network dataset pair other than NSCN, we evaluated Adam, QHAdam, AMSGrad, DEMON Adam, AggMo, QHM, DE-MON SGDM, and SGDM with learning rate decay. For adaptive learning rate methods and adaptive momentum methods, we generally perform a grid search over the learning rate. For SGDM, we generally perform a grid search over learning rate and initial momentum. For SGDM learning rate decay, the learning rate is decayed by a factor of 0.1 after there is no improvement in validation loss for the best of {1, 2, 3, 5, 10, 20, 30, 40} epochs.

A.1 SETUP:

We describe the six test problems in this paper.

• CIFAR10 -ResNet18 CIFAR10 contains 60,000 32x32x3 images with a 50,000 training set, 10,000 test set split. There are 10 classes. ResNet18 (He et al., 2016) • PTB -LSTM PTB is an English text corpus containing 929,000 training words, 73,000 validation words, and 82,000 test words. There are 10,000 words in the vocabulary. The model is stacked LSTMs (Hochreiter & Schmidhuber, 1997) with 2 layers, 650 units per layer, and dropout of 0.5. Trained with a batch size of 20.

• MNIST -VAE MNIST contains 60,000 32x32x1 grayscale images with a 50,000 training set, 10,000 test set split. There are 10 classes of 10 digits. VAE (Kingma & Welling, 2015) with three dense encoding layers and three dense decoding layers with a latent space of size 2. Trained with a batch size of 100.

• CIFAR10 -NCSN CIFAR10 contains 60,000 32x32x3 images with a 50,000 training set, 10,000 test set split. There are 10 classes. NCSN (Song & Ermon, 2019 ) is a recent state-of-the-art generative model which achieves the best reported inception score. We compute inception scores based on a total of 50000 samples. We follow the exact implementation in and defer details to the original paper.

A.2 METHODS:



A.2.1 ADAPTIVE LEARNING RATE:

Adam (Kingma & Ba, 2014), as previously introduced in section 2, keeps an exponentially decaying average of squares of past gradients to adapt the learning rate. It also introduces an exponentially decaying average of gradients.

The Adam algorithm is parameterized by learning rate η > 0, discount factors β 1 < 1 and β 2 < 1, a small constant , and uses the update rule:

E g t+1 = β 1 • E g t + (1 -β 1 ) • g t , E g•g t+1 = β 2 • E g•g t + (1 -β 2 ) • (g t • g t ), θ t+1,i = θ t,i - η √ E g•g t+1,i +ε • E g t+1,i , ∀t.

AMSGrad (Reddi et al., 2019) resolves an issue in the proof of Adam related to the exponential moving average E g•g t

, where Adam does not converge for a simple optimization problem. Instead of an exponential moving average, AMSGrad keeps a running maximum of E g•g .

The AMSGrad algorithm is parameterized by learning rate η > 0, discount factors β 1 < 1 and β 2 < 1, a small constant , and uses the update rule:

E g t+1 = β 1 • E g t + (1 -β 1 ) • g t , E g•g t+1 = β 2 • E g•g t + (1 -β 2 ) • (g t • g t ), Êg•g t+1,i = max( Êg•g t,i , E g•g t,i ), θ t+1,i = θ t,i - η Êg•g t+1,i +ε • E g t+1,i , ∀t,

where E g t+1 and E g•g t+1 are defined identically to Adam. QHAdam (Quasi-Hyperbolic Adam) (Ma & Yarats, 2018) extends QHM (Quasi-Hyperbolic Momentum), introduced further below, to replace both momentum estimators in Adam with quasihyperbolic terms. This quasi-hyperbolic formulation is capable of recovering Adam and NAdam (Dozat, 2016) , amongst others.

The QHAdam algorithm is parameterized by learning rate η > 0, discount factors β 1 < 1 and β 2 < 1, ν 1 , ν 2 ∈ R, a small constant , and uses the update rule:

E g t+1 = β 1 • E g t + (1 -β 1 ) • g t , E g•g t+1 = β 2 • E g•g t + (1 -β 2 ) • (g t • g t ), Êg t+1 = (1 + β t+1 1 ) -1 • E g t+1 , Êg•g t+1 = (1 + β t+1 2 ) -1 • E g•g t+1 , θ t+1,i = θ t,i -η (1 -ν 1 ) • g t + ν 1 • Êg t+1 (1 -ν 2 )g 2 t + ν 2 • Êg•g t+1 + , ∀t,

where E g t+1 and E g•g t+1 are defined identically to Adam.

A.2.2 ADAPTIVE MOMENTUM:

AggMo (Aggregated Momentum) (Lucas et al., 2018) takes a linear combination of multiple momentum buffers. It maintains K momentum buffers, each with a different discount factor, and averages them for the update.

The AggMo algorithm is parameterized by learning rate η > 0, discount factors β ∈ R K , and uses the update rule:

(E g t+1 ) (i) = β (i) • (E g t ) (i) + g t , ∀i ∈ [1, K], θ t+1,i = θ t,i -η 1 K • K i=1 (E g t+1 ) (i) , ∀t.

QHM (Quasi-Hyperbolic Momentum) (Ma & Yarats, 2018 ) is a weighted average of the momentum and plain SGD. QHM is capable of recovering Nesterov Momentum (Nesterov, 1983) , Synthesized Nesterov Variants (Lessard et al., 2016) , accSGD (Jain et al., 2017) and others.

The QHM algorithm is parameterized by learning rate η > 0, discount factor β < 1, immediate discount factor ν ∈ R, and uses the update rule:

E g t+1 = β • E g t + (1 -β) • g t , θ t+1,i = θ t,i -η (1 -ν) • g t + ν • E g t+1 , ∀t.

A.3 OPTIMIZER HYPERPARAMETERS 

B CONVERGENCE ANALYSIS:

We analyze the global convergence of DEMON CM in the convex setting, following (Ghadimi et al., 2014) . For an objective function f which is convex, continuously differentiable, its gradient ∇f (•) is Lipschitz continuous with constant L, our goal is to show that f ( θT ) converges to the optimum f * with decreasing momentum, where θT is the average of θ t for t = 1, ..., T . Our following theorem holds for a constant learning rate and β t decaying with t. Theorem 1. Assume that f is convex, continuously differentiable, its gradient ∇f (•) is Lipschitz continuous with constant L, with a decreasing momentum, but constant step size, as in:

β t = 1 t • t+1 t+2 , α ∈ 0, 2 3L .

We consider the CM iteration in non-stochastic settings, where:

θ t+1 = θ t -α∇f (θ t ) + β t (θ t -θ t-1 ) .

Then, the sequence {θ t } T t=1 generated by the CM iteration, with decreasing momentum, satisfies:

f ( θT ) -f * ≤ θ1-θ 2 T 3 4 L + 1 2α

, where θT is the Cesaro average of the iterates: θT

= 1 T T t=1 θ t . Proof. Let β t = 1 t • t+1 t+2 and p t = 1 t (θ t -θ t-1

). We consider the CM iteration in non-stochastic settings, where:

θ t+1 = θ t -α∇f (θ t ) + β t (θ t -θ t-1 ) .

Using the definition of p t above, one can easily prove that:

θ t+1 + p t+1 = (1 + 1 t + 1 )θ t+1 - 1 t + 1 θ t = θ t + p t -α(t+2) t+1 ∇f (θ t ).

Using this expression, we will analyze the term θ t+1 + p t+1 -θ 2 :

θ t+1 + p t+1 -θ 2 = θ t + p t -θ 2 -2α(t+2) t+1 θ t + p t -θ , ∇f (θ t ) + α(t+2) t+1 2 • ∇f (θ t ) 2 = θ t + p t -θ 2 -2α(t+2) t(t+1) θ t -θ t-1 , ∇f (θ t ) -2α(t+2) t+1 θ t -θ , ∇f (θ t ) + α(t+2) t+1 2 • ∇f (θ t ) 2

Since f is convex, continuously differentiable, its gradient is Lipschitz continuous with constant L, then

EQUATION

EQUATION

Substituting the above inequalities leads to

θ t+1 + p t+1 -θ 2 ≤ θ t + p t -θ 2 -2α(t+2) t(t+1) (f (θ t ) -f (θ t-1 )) -2α (1-λ)(t+2) L(t+1) • ∇f (θ t ) 2 -2αλ t+2 t+1 (f (θ t ) -f * ) -α λ(t+2) L(t+1) • ∇f (θ t ) 2 + α(t + 2) t + 1 2 • ∇f (θ t ) 2

where λ ∈ (0, 1] is a parameter weighting (2) and (3). Grouping together terms yields 2α(t+2) t(t+1) + 2αλ(t+2) t+1 (f (θ t ) -f * ) + θ t+1 + p t+1 -θ 2 ≤ 2α(t+2) t(t+1) (f (θ t-1 ) -f * ) + θ t + p t -θ 2

+ α(t+2) t+1 α(t+2) t+1 -2(1-λ) L -λ L ∇f (θ t ) 2 .

The last term is non-positive when α ∈ [0, t+1 t+2 ( 2-λ L )] so it can be dropped. Summing over t = 1, ..., T yields 2αλ Since f (θ 1 ) -f * ≤ L 2 θ 1 -θ 2 by Lipschitz continuous gradients, setting λ = 1 and observing (t + 1)/(t + 2) ≥ 2/3 gives the result. 

:

For clarity, we will skip the bias correction step in this description of Adam; seeKingma & Ba (2014).

Figures and Tables:

Figure 1: Randomly selected CIFAR10 images generated with NCSN. Left: Real CIFAR10 images. Middle: Adam. Right: DEMON Adam.

Figure 2: Top row, two left-most plots: RN18-CIFAR10-DEMONAdam for 300 epochs. Top row, right-most plot: VGG16-CIFAR100-DEMONAdam for 300 epochs. Bottom row, left-most plot: WRN-STL10-DEMONAdam for 200 epochs. Bottom row, middle plot: PTB-LSTM-DEMONAdam for 25 epochs. Bottom row, right-most plot: VAE-MNIST-DEMONAdam for 200 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent one standard deviation.

Figure 3: Top row, two left-most plot: RN18-CIFAR10-DEMONSGDM for 300 epochs. Top row, right-most plot: VGG16-CIFAR100-DEMONSGDM for 300 epochs. Bottom row, left-most plots: WRN-STL10-DEMONSGDM for 200 epochs. Bottom row, middle plot: PTB-LSTM-DEMONSGDM for 25 epochs. Bottom row, right-most plot: VAE-MNIST-DEMONSGDM for 200 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

) (f (θ t ) -f * ) + θ t+1 + p t+1 -θ 2 ) (f (θ t-1 ) -f * ) + θ t + p t -θ 2 , θ t ) -f * ) ≤ 3α(f (θ 1 ) -f * ) + θ 1 -θ 2 . θ t ) -f * ) ≤ 2αλ T t=1 t+2 t+1 (f (θ t ) -f * ) ≤ 3αλ T t=1 (f (θ t ) -f * ),we further have:3αλ T t=1 (f (θ t ) -f * ) ≤ 3 2 3α(f (θ 1 ) -f * ) + θ 1 -θ 2 .Due to the convexity of f , f (θ 1 ) -f * ) + 3 2 θ 1 -θ 2 .

Figure 6: Additional empirical results on VGG16-CIFAR100-DEMONAdam. Top row: 75 epochs. Bottom row: 150 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Figure 7: Additional empirical results on WRN-STL10-DEMONAdam. Top row: 50 epochs. Bottom row: 100 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Figure 8: Additional empirical results on PTB-LSTM-DEMONAdam for 39 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Figure 9: Additional empirical results on VAE-MNIST-DEMONAdam. Left: 50 epochs. Right: 100 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Figure 10: Additional empirical results on adaptive momentum methods. Left plot: VGG16-CIFAR100-DEMONSGDM for 300 epochs. Right plot: WRN-STL10-DEMONSGDM for 200 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Figure 11: Additional empirical results on RN18-CIFAR10-DEMONSGDM. Top row: 30 epochs. Middle row: 75 epochs. Bottom row: 150 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Figure 12: Additional empirical results on VGG16-CIFAR100-DEMONSGDM. Top row: 75 epochs. Bottom row: 150 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Figure 13: Additional empirical results on WRN-STL10-DEMONSGDM. Top row: 50 epochs. Bottom row: 100 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Figure 14: Additional empirical results on PTB-LSTM-DEMONSGDM for 39 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Figure 15: Additional empirical results on VAE-MNIST-DEMONSGDM. Left: 50 epochs. Right: 100 epochs. Dotted and solid lines represent training and generalization metrics respectively. Shaded bands represent 1 standard deviation.

Table 1 and comprehensively detailed in Appendix A. We report improved performance by delaying the application of DEMON where applicable, and report performance across different number of total epochs to demonstrate effectiveness regardless of the training budget. Note that the predefined number of epochs we run all experiments affects the proposed decaying momentum routine, by definition of β t . Summary of experimental settings.

Table content: <table><tr><td>Experiment short name</td><td/><td colspan="2">Model</td><td>Dataset</td><td>Optimizer</td></tr><tr><td colspan="2">RN18-CIFAR10-DEMONSGDM</td><td colspan="2">ResNet18</td><td colspan="2">CIFAR10</td><td>DEMON SGDM</td></tr><tr><td colspan="2">RN18-CIFAR10-DEMONAdam</td><td colspan="2">ResNet18</td><td colspan="2">CIFAR10</td><td>DEMON Adam</td></tr><tr><td colspan="4">VGG16-CIFAR100-DEMONSGDM VGG-16</td><td colspan="2">CIFAR100</td><td>DEMON SGDM</td></tr><tr><td colspan="4">VGG16-CIFAR100-DEMONAdam VGG-16</td><td colspan="2">CIFAR100</td><td>DEMON Adam</td></tr><tr><td>WRN-STL10-DEMONSGDM</td><td/><td colspan="3">Wide ResNet 16-8 STL10</td><td>DEMON SGDM</td></tr><tr><td>WRN-STL10-DEMONAdam</td><td/><td colspan="3">Wide ResNet 16-8 STL10</td><td>DEMON Adam</td></tr><tr><td>LSTM-PTB-DEMONSGDM</td><td/><td colspan="2">LSTM RNN</td><td colspan="2">Penn TreeBank DEMON SGDM</td></tr><tr><td>LSTM-PTB-DEMONAdam</td><td/><td colspan="2">LSTM RNN</td><td colspan="2">Penn TreeBank DEMON Adam</td></tr><tr><td>VAE-MNIST-DEMONSGDM</td><td/><td>VAE</td><td/><td>MNIST</td><td>DEMON SGDM</td></tr><tr><td>VAE-MNIST-DEMONAdam</td><td/><td>VAE</td><td/><td>MNIST</td><td>DEMON Adam</td></tr><tr><td colspan="2">NCSN-CIFAR10-DEMONAdam</td><td colspan="2">NCSN</td><td colspan="2">CIFAR10</td><td>DEMON Adam</td></tr><tr><td/><td colspan="2">30 epochs</td><td>75 epochs</td><td>150 epochs</td><td>300 epochs</td></tr><tr><td>Adam</td><td colspan="2">16.58 ± .18</td><td>13.63 ± .22</td><td>11.90 ± .06</td><td>11.94 ± .06</td></tr><tr><td>AMSGrad</td><td colspan="2">16.98 ± .36</td><td>13.43 ± .14</td><td>11.83 ± .12</td><td>10.48 ± .12</td></tr><tr><td>QHAdam</td><td colspan="2">16.41 ± .38</td><td>15.55 ± .25</td><td>13.78 ± .08</td><td>13.36 ± .11</td></tr><tr><td>DEMON Adam</td><td colspan="2">11.75 ± .15</td><td>9.69 ± .10</td><td>8.83 ± .08</td><td>8.44 ± .05</td></tr></table>

RN18-CIFAR10-DEMONAdam generalization error. The number of epochs was predefined before the execution of the algorithms.

Table content: <table/>

VGG16-CIFAR100-DEMONAdam and WRN-STL10-DEMONAdam generalization error. The number of epochs was predefined before the execution of the algorithms.

Table content: <table><tr><td/><td/><td>VGG-16</td><td/><td colspan="2">Wide Residual 16-8</td><td/></tr><tr><td/><td>75 epochs</td><td>150 epochs</td><td>300 epochs</td><td>50 epochs</td><td>100 epochs</td><td>200 epochs</td></tr><tr><td>Adam</td><td>37.98 ± .20</td><td>33.62 ± .11</td><td>31.09 ± .09</td><td>23.35 ± .20</td><td>19.63 ± .26</td><td>18.65 ± .07</td></tr><tr><td>AMSGrad</td><td>40.67 ± .65</td><td>34.46 ± .21</td><td>31.62 ± .12</td><td>21.73 ± .25</td><td>19.35 ± .20</td><td>18.21 ± .18</td></tr><tr><td>QHAdam</td><td>36.53 ± .20</td><td>32.96 ± .11</td><td>30.97 ± .10</td><td>21.25 ± .22</td><td>19.81 ± .18</td><td>18.52 ± .25</td></tr><tr><td>DEMON Adam</td><td>32.40 ± .19</td><td>28.84 ± .18</td><td>27.11 ± .19</td><td>19.42 ± .10</td><td>18.36 ± .11</td><td>17.62 ± .12</td></tr><tr><td/><td colspan="2">LSTM</td><td/><td>VAE</td><td/><td>NCSN</td></tr><tr><td/><td>25 epochs</td><td>39 epochs</td><td>50 epochs</td><td>100 epochs</td><td>200 epochs</td><td>512 epochs</td></tr><tr><td>Adam</td><td>115.54 ± .64</td><td>115.02 ± .52</td><td>136.28 ± .18</td><td>134.64 ± .14</td><td>134.66 ± .17</td><td>8.15 ± .20</td></tr><tr><td>AMSGrad</td><td>108.07 ± .19</td><td>107.87 ± .25</td><td>137.89 ± .12</td><td>135.69 ± .03</td><td>134.75 ± .18</td><td>-</td></tr><tr><td>QHAdam</td><td>112.52 ± .23</td><td>112.45 ± .39</td><td>136.69 ± .17</td><td>134.84 ± .08</td><td>134.12 ± .12</td><td>-</td></tr><tr><td>DEMON Adam</td><td>101.57 ± .32</td><td>101.44 ± .47</td><td>134.46 ± .17</td><td>134.12 ± .08</td><td>133.87 ± .21</td><td>8.07 ± .08</td></tr></table>

PTB-LSTM-DEMONAdam generalization perplexity, VAE-MNIST-DEMONAdam generalization loss and NCSN-CIFAR10-DEMONAdam inception score.

Table content: <table/>

VGG16-CIFAR100-DEMONSGDM and WRN-STL10-DEMONSGDM generalization error. The number of epochs was predefined before the execution.

Table content: <table><tr><td/><td/><td/><td colspan="2">VGG-16</td><td/><td/><td>Wide Residual 16-8</td></tr><tr><td/><td colspan="2">75 epochs</td><td colspan="2">150 epochs</td><td colspan="2">300 epochs</td><td>75 epochs</td><td>150 epochs</td><td>300 epochs</td></tr><tr><td>SGDM learning rate decay</td><td colspan="2">35.29 ± .59</td><td colspan="2">30.65 ± .31</td><td colspan="2">29.74 ± .43</td><td>21.05 ± .27</td><td>17.83 ± 0.39</td><td>15.16 ± .36</td></tr><tr><td>AggMo</td><td colspan="2">42.85 ± .89</td><td colspan="2">34.25 ± .24</td><td colspan="2">32.32 ± .18</td><td>22.70 ± .11</td><td>20.06 ± .31</td><td>17.90 ± .13</td></tr><tr><td>QHM</td><td colspan="2">42.14 ± .79</td><td colspan="2">33.87 ± .26</td><td colspan="2">32.45 ± .13</td><td>22.86 ± .15</td><td>19.40 ± .23</td><td>17.79 ± .08</td></tr><tr><td>DEMON SGDM</td><td colspan="2">34.35 ± .44</td><td colspan="2">30.59 ± .26</td><td colspan="2">28.99 ± .16</td><td>19.45 ± .20</td><td>15.98 ± .40</td><td>13.67 ± .13</td></tr><tr><td/><td/><td/><td colspan="2">LSTM</td><td/><td/><td>VAE</td></tr><tr><td/><td/><td colspan="2">25 epochs</td><td colspan="2">39 epochs</td><td colspan="2">50 epochs</td><td>100 epochs</td><td>200 epochs</td></tr><tr><td colspan="2">SGDM learning rate decay</td><td colspan="2">89.59 ± .07</td><td colspan="2">87.57 ± .11</td><td colspan="2">140.51 ± .73</td><td>139.54 ± .34</td><td>137.33 ± .49</td></tr><tr><td>AggMo</td><td/><td colspan="2">89.09 ± .16</td><td colspan="2">89.07 ± .15</td><td colspan="2">139.69 ± .17</td><td>139.07 ± .26</td><td>137.64 ± .20</td></tr><tr><td>QHM</td><td/><td colspan="2">94.47 ± .19</td><td colspan="2">94.44 ± .13</td><td colspan="2">145.84 ± .39</td><td>140.92 ± .19</td><td>137.64 ± .20</td></tr><tr><td>DEMON SGDM</td><td/><td colspan="2">88.33 ± .16</td><td colspan="2">88.32 ± .12</td><td colspan="2">139.32 ± .23</td><td>137.51 ± .29</td><td>135.95 ± .21</td></tr></table>

PTB-LSTM-DEMONSGDM (perplexity) and VAE-MNIST-DEMONSGDM (generalization loss) experiments.

Table content: <table/>

is an 18 layers deep CNN with skip connections for image classification. Trained with a batch size of 128. • CIFAR100 -VGG16 CIFAR100 is a fine-grained version of CIFAR-10 and contains 60,000 32x32x3 images with a 50,000 training set, 10,000 test set split. There are 100 classes. VGG16 (Simonyan & Zisserman, 2014) is a 16 layers deep CNN with extensive use of 3x3 convolutional filters. Trained with a batch size of 128 • STL10 -Wide ResNet 16-8 STL10 contains 1300 96x96x3 images with a 500 training set, 800 test set split. There are 10 classes. Wide ResNet 16-8 (Zagoruyko & Komodakis, 2016) is a 16 layers deep ResNet which is 8 times wider. Trained with a batch size of 64.

Table content: <table/>

Best parameters for CIFAR-10 with ResNet-18.

Table content: <table><tr><td>Optimization method</td><td>epochs</td><td>η</td><td>other parameters</td></tr><tr><td>Adam</td><td>30</td><td>0.001</td><td/></tr><tr><td>Adam Adam</td><td>75 150</td><td>0.001 0.001</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>Adam</td><td>300</td><td>0.0003</td><td/></tr><tr><td>AMSGrad</td><td>30</td><td>0.001</td><td/></tr><tr><td>AMSGrad AMSGrad</td><td>75 150</td><td>0.001 0.001</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>AMSGrad</td><td>300</td><td>0.001</td><td/></tr><tr><td>QHAdam</td><td>30</td><td>0.001</td><td/></tr><tr><td>QHAdam QHAdam</td><td>75 150</td><td>0.0003 0.0003</td><td>ν1 = 0.7, ν2 = 1.0, β1 = 0.9, β2 = 0.99</td></tr><tr><td>QHAdam</td><td>300</td><td>0.0003</td><td/></tr><tr><td>DEMON Adam</td><td>30</td><td>0.0001</td><td/></tr><tr><td>DEMON Adam DEMON Adam</td><td>75 150</td><td>0.0001 0.0001</td><td>βinit = 0.9, β2 = 0.999</td></tr><tr><td>DEMON Adam</td><td>300</td><td>0.0001</td><td/></tr><tr><td>AggMo</td><td>30</td><td>0.03</td><td/></tr><tr><td>AggMo AggMo</td><td>75 150</td><td>0.01 0.01</td><td>β = [0, 0.9, 0.99]</td></tr><tr><td>AggMo</td><td>300</td><td>0.01</td><td/></tr><tr><td>QHM</td><td>30</td><td>1.0</td><td/></tr><tr><td>QHM QHM</td><td>75 150</td><td>0.3 0.3</td><td>ν = 0.7, β = 0.999</td></tr><tr><td>QHM</td><td>300</td><td>0.3</td><td/></tr><tr><td>DEMON SGDM</td><td>30</td><td>0.1</td><td/></tr><tr><td>DEMON SGDM DEMON SGDM</td><td>75 150</td><td>0.1 0.03</td><td>βinit = 0.9</td></tr><tr><td>DEMON SGDM</td><td>300</td><td>0.03</td><td/></tr><tr><td>SGDM learning rate decay</td><td>30</td><td>0.1</td><td>β1 = 0.9, patience = 5</td></tr><tr><td>SGDM learning rate decay</td><td>75</td><td>0.1</td><td>β1 = 0.9, patience = 20</td></tr><tr><td>SGDM learning rate decay</td><td>150</td><td>0.1</td><td>β1 = 0.9, patience = 20</td></tr><tr><td>SGDM learning rate decay</td><td>300</td><td>0.1</td><td>β1 = 0.9, patience = 40</td></tr></table>

Best parameters for CIFAR-100 with VGG-16.

Table content: <table><tr><td>Optimization method</td><td>epochs</td><td>η</td><td>other parameters</td></tr><tr><td>Adam</td><td>75</td><td>0.0003</td><td/></tr><tr><td>Adam</td><td>150</td><td>0.0003</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>Adam</td><td>300</td><td>0.0003</td><td/></tr><tr><td>AMSGrad</td><td>75</td><td>0.0003</td><td/></tr><tr><td>AMSGrad</td><td>150</td><td>0.0003</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>AMSGrad</td><td>300</td><td>0.0003</td><td/></tr><tr><td>QHAdam</td><td>75</td><td>0.0003</td><td/></tr><tr><td>QHAdam</td><td>150</td><td>0.0003</td><td>ν1 = 0.7, ν2 = 1.0, β1 = 0.9, β2 = 0.99</td></tr><tr><td>QHAdam</td><td>300</td><td>0.0003</td><td/></tr><tr><td>DEMON Adam</td><td>75</td><td>0.00003</td><td/></tr><tr><td>DEMON Adam</td><td>150</td><td>0.00003</td><td>βinit = 0.9, β2 = 0.999</td></tr><tr><td>DEMON Adam</td><td>300</td><td>0.00003</td><td/></tr><tr><td>AggMo</td><td>75</td><td>0.001</td><td/></tr><tr><td>AggMo</td><td>150</td><td>0.001</td><td>β = [0, 0.9, 0.99]</td></tr><tr><td>AggMo</td><td>300</td><td>0.001</td><td/></tr><tr><td>QHM</td><td>75</td><td>0.1</td><td/></tr><tr><td>QHM</td><td>150</td><td>0.03</td><td>ν = 0.7, β = 0.999</td></tr><tr><td>QHM</td><td>300</td><td>0.03</td><td/></tr><tr><td>DEMON SGDM</td><td>75</td><td>0.1</td><td/></tr><tr><td>DEMON SGDM</td><td>150</td><td>0.03</td><td>βinit = 0.9</td></tr><tr><td>DEMON SGDM</td><td>300</td><td>0.03</td><td/></tr><tr><td>SGDM learning rate decay</td><td>75</td><td>0.1</td><td>β1 = 0.9, patience = 5</td></tr><tr><td>SGDM learning rate decay</td><td>150</td><td>0.03</td><td>β1 = 0.9, patience = 20</td></tr><tr><td>SGDM learning rate decay</td><td>300</td><td>0.03</td><td>β1 = 0.9, patience = 30</td></tr></table>

Best parameters for STL10 with Wide ResNet 16-8.

Table content: <table><tr><td>Optimization method</td><td>epochs</td><td>η</td><td/></tr><tr><td>Adam</td><td>50</td><td>0.001</td><td/></tr><tr><td>Adam</td><td>100</td><td>0.0003</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>Adam</td><td>200</td><td>0.0003</td><td/></tr><tr><td>AMSGrad</td><td>50</td><td>0.0003</td><td/></tr><tr><td>AMSGrad</td><td>100</td><td>0.0003</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>AMSGrad</td><td>200</td><td>0.0003</td><td/></tr><tr><td>QHAdam</td><td>50</td><td>0.0003</td><td/></tr><tr><td>QHAdam</td><td>100</td><td>0.0003</td><td>ν1 = 0.7, ν2 = 1.0, β1 = 0.9, β2 = 0.99</td></tr><tr><td>QHAdam</td><td>200</td><td>0.0003</td><td/></tr><tr><td>DEMON Adam</td><td>50</td><td>0.00003</td><td/></tr><tr><td>DEMON Adam</td><td>100</td><td>0.00003</td><td>βinit = 0.9, β2 = 0.999</td></tr><tr><td>DEMON Adam</td><td>200</td><td>0.00003</td><td/></tr><tr><td>AggMo</td><td>50</td><td>0.03</td><td/></tr><tr><td>AggMo</td><td>100</td><td>0.03</td><td>β = [0, 0.9, 0.99]</td></tr><tr><td>AggMo</td><td>200</td><td>0.01</td><td/></tr><tr><td>QHM</td><td>50</td><td>0.3</td><td/></tr><tr><td>QHM</td><td>100</td><td>0.3</td><td>ν = 0.7, β = 0.999</td></tr><tr><td>QHM</td><td>200</td><td>0.3</td><td/></tr><tr><td>DEMON SGDM</td><td>50</td><td>0.1</td><td/></tr><tr><td>DEMON SGDM</td><td>100</td><td>0.1</td><td>βinit = 0.9</td></tr><tr><td>DEMON SGDM</td><td>200</td><td>0.1</td><td/></tr><tr><td>SGDM learning rate decay</td><td>50</td><td>0.1</td><td>β1 = 0.9, patience = 10</td></tr><tr><td>SGDM learning rate decay</td><td>100</td><td>0.1</td><td>β1 = 0.9, patience = 10</td></tr><tr><td>SGDM learning rate decay</td><td>200</td><td>0.1</td><td>β1 = 0.9, patience = 20</td></tr></table>

Best parameters for PTB with LSTM architecture.

Table content: <table><tr><td>Optimization method</td><td>epochs</td><td>η</td><td>other parameters</td></tr><tr><td>Adam Adam</td><td>25 39</td><td>0.0003 0.0003</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>AMSGrad AMSGrad</td><td>25 39</td><td>0.001 0.001</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>QHAdam QHAdam</td><td>25 39</td><td colspan="2">0.0003 ν1 = 0.7, ν2 = 1.0, β1 = 0.9, β2 = 0.999 0.0003</td></tr><tr><td>DEMON Adam DEMON Adam</td><td>25 39</td><td>0.0001 0.0001</td><td>βinit = 0.9, β2 = 0.999</td></tr><tr><td>AggMo AggMo</td><td>25 39</td><td>0.03 0.03</td><td>β = [0, 0.9, 0.99]</td></tr><tr><td>QHM QHM</td><td>25 39</td><td>1.0 1.0</td><td>ν = 0.7, β = 0.999</td></tr><tr><td>DEMON SGDM</td><td>25</td><td>1.0</td><td>βinit = 0.5, β f inal = -0.5</td></tr><tr><td>DEMON SGDM</td><td>39</td><td>1.0</td><td>βinit = 0.3, β f inal = -0.5</td></tr><tr><td>SGDM learning rate decay</td><td>25</td><td>0.1</td><td>β1 = 0.9, smooth learning rate decay</td></tr><tr><td>SGDM learning rate decay</td><td>39</td><td>1.0</td><td>β1 = 0.0, smooth learning rate decay</td></tr></table>

Best parameters for MNIST with VAE.

Table content: <table><tr><td>Optimization method</td><td>epochs</td><td>η</td><td>other parameters</td></tr><tr><td>Adam</td><td>50</td><td>0.001</td><td/></tr><tr><td>Adam</td><td>100</td><td>0.001</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>Adam</td><td>200</td><td>0.001</td><td/></tr><tr><td>AMSGrad</td><td>50</td><td>0.001</td><td/></tr><tr><td>AMSGrad</td><td>100</td><td>0.001</td><td>β1 = 0.9, β2 = 0.999</td></tr><tr><td>AMSGrad</td><td>200</td><td>0.001</td><td/></tr><tr><td>QHAdam</td><td>50</td><td>0.001</td><td/></tr><tr><td>QHAdam</td><td>100</td><td>0.001</td><td>ν1 = 0.7, ν2 = 1.0, β1 = 0.9, β2 = 0.99</td></tr><tr><td>QHAdam</td><td>200</td><td>0.001</td><td/></tr><tr><td>DEMON Adam</td><td>50</td><td>0.0001</td><td/></tr><tr><td>DEMON Adam</td><td>100</td><td>0.0001</td><td>βinit = 0.9, β2 = 0.999</td></tr><tr><td>DEMON Adam</td><td>200</td><td>0.0001</td><td/></tr><tr><td>AggMo</td><td>50</td><td>0.000003</td><td/></tr><tr><td>AggMo</td><td>100</td><td>0.000003</td><td>β = [0, 0.9, 0.99]</td></tr><tr><td>AggMo</td><td>200</td><td>0.000003</td><td/></tr><tr><td>QHM</td><td>50</td><td>0.0001</td><td/></tr><tr><td>QHM</td><td>100</td><td>0.00003</td><td>ν = 0.7, β = 0.999</td></tr><tr><td>QHM</td><td>200</td><td>0.00003</td><td/></tr><tr><td>DEMON SGDM</td><td>50</td><td>0.00001</td><td/></tr><tr><td>DEMON SGDM</td><td>100</td><td>0.00001</td><td>βinit = 0.9</td></tr><tr><td>DEMON SGDM</td><td>200</td><td>0.000003</td><td/></tr><tr><td>SGDM learning rate decay</td><td>50</td><td>0.00001</td><td>β1 = 0.9, patience = 5</td></tr><tr><td>SGDM learning rate decay</td><td>100</td><td>0.000003</td><td>β1 = 0.9, patience = 5</td></tr><tr><td>SGDM learning rate decay</td><td>200</td><td>0.000003</td><td>β1 = 0.9, patience = 20</td></tr></table>

Peer review (iclr2020):
This paper proposed a new decaying momentum rule to further improve neural network training. The idea is motivated by decaying the total contribution of a gradient to all future updates. The authors also extend this idea on to Adam and show that it improved upon the vanilla Adam. - The intuition of using momentum decaying scheme is not quite clear to me. Why having a linear momentum decay schedule is better? This question is not answered in the paper. I hope the authors could provide more illustrative explanations regarding this. - This paper focused essentially on empirical evaluations. I do appreciate that the authors conduction various experiments using different dataset and architecture but I do not understand why the authors want to separate the experiments into parts: adaptive methods and adaptive momentum methods. Not to mention the confusing names, it makes no sense to say the same words again for different optimizers. I would suggest the authors to combine the results together for better comparison. - CM formulation mentioned in the paper is actually different from the commonly refer SGD with momentum implementation, which is the optimizer that is widely used in the community. Therefore, it is important to at least add SGD with momentum as one baseline in the experiments. - According to https://github.com/kuangliu/pytorch-cifar, Resnet18 on CIFAR10 can achieve at least 93% accuracy using simple SGD with momentum. It seems that the current reported results are not fully optimized. I would suggest the authors to check the parameter settings and make sure all the hyperparameters for baseline methods are fully tuned. - Aside from SGD with momentum, the experiments part also lacks several important baselines. The author should also consider comparing the AdamW, Padam mentioned in the paper.