SoftAdam: Unifying SGD and Adam for better stochastic gradient descent
Authors: Abraham J. Fetterman
Christina H. Kim
Joshua Albrecht
Venue:  (2020)
Keywords: Optimization
SGD
Adam
Generalization
Deep Learning

Abstract: Abstract Stochastic gradient descent (SGD) and Adam are commonly used to optimize deep neural networks, but choosing one usually means making tradeoffs between speed, accuracy and stability. Here we present an intuition for why the tradeoffs exist as well as a method for unifying the two in a continuous way. This makes it possible to control the way models are trained in much greater detail. We show that for default parameters, the new algorithm equals or outperforms SGD and Adam across a range of models for image classification tasks and outperforms SGD for language modeling tasks.

Under review as a conference paper at ICLR 2020
SOFTADAM: UNIFYING SGD AND ADAM FOR BETTER STOCHASTIC GRADIENT DESCENT
Anonymous authors
Paper under double-blind review
ABSTRACT
Stochastic gradient descent (SGD) and Adam are commonly used to optimize deep neural networks, but choosing one usually means making tradeoffs between speed, accuracy and stability. Here present an intuition for why the tradeoff's exist as well as a method for unifying the two in a continuous way. This makes it possible to control the way models are trained in much greater detail. We show that for default parameters, the new algorithm equals or outperforms SGD and Adam across range of models for image classification tasks and outperforms SGD for language modeling tasks.
INTRODUCTION
One of the most common methods of training neural networks is stochastic gradient descent (SGD) (Bottou et al. (2016). SGD has strong theoretical guarantees, including convergence in locally non-convex optimization problems (Lee et al. (2016)). It also shows improved generalization and stability when compared to other optimization algorithms (Smith & Le (2018)).
There have been various efforts in improving the speed and generalization of SGD, One popular modification to use an adaptive gradient (Duchi et al. (20TT)) which scales the gradient step size to be larger in directions with consistently small gradients. Adam, implementation that combines SGD with momentum and an adaptive step size inversely proportional to the RMS gradient, has been particularly successful speeding up training and solving particular problems (Kingma & Ba (2014)). However, at other problems pays a penalty in worse generalization (Wilson et al. (2017 Keskar & Socher (2017), and it requires additional modifications achieve convergence
guarantee (Reddi et al. 2018): Li & Orabonal (2018)).
Here develop intuition for adaptive gradient methods that allows to unify Adam with SGD in a natural way. The new optimizer, SoftAdam, descends in a direction that mixes the SGD with Adam update steps. As such, should be able to achieve equal or better optimization results across a variety of problems.
1.1 RELATED WORK
Several authors have recently tried to combine Adam and SGD to get the best of both worlds. How- ever, these have not always enabled better generalization or performance across different problems.
In one study, the optimization algorithm was switched from Adam to SGD during training based on a scale-free criterion, preventing the addition of a new hyper-parameter (Keskar & Socher (2017). The result is that the longer the convolutional networks were trained on Adam, the worse their generalization performance compared to SGD. The best performance came from switching to SGD in this case soon possible.
Another recent algorithm takes the approach of clipping the large Adam updates to make them more similar to SGD the training nears the end (Luo et al. (2019)) However, this approach requires two new both SGD and Adam.
the rate at which the training is switched over, and learning rate for
Similar to this work, partially adaptive methods (Chen & Gu (018)) can allow arbitrary mixing between SGD and Adam. However, in that work the step size is not strictly smaller than the SGD

---

Under review as a conference paper at ICLR 2020
step and the same guarantees cannot be made about convergence. It is of interest to see whether there is any advantage over these methods.
Other algorithms have shown improvements SGD by averaging weights many steps (Polyak & Juditsky 1992); Zhang et al. [Izmaiooy et al. (018)). These algorithms are complementary to the algorithm developed here, they require underlying algorithm to choose the step direction at any point.
2 SGD AND ADAM
The fundamental idea of gradient descent is to follow the path of steepest descent to an optimum. Stochastic gradient descent enables us to optimize much larger problems by using randomly sub- sampled training data. The stochastic gradient descent algorithm will minimize the loss function 20; which is parameterized by 0 and takes input training data x.
9t - 0: - 0, aggt.
where a is a learning rate that may vary with t and x, is the training data selected for the batch at step t. The convergence rate can be improved further by using running average of the gradient, initializing with mg <- 0. This method, known as momentum (Goh (2017), may be written as,
9t VoJ (0: 21)00101111111
my 31 mm111 (1 31) gt.
0: - 0, am,i
further development that has improved convergence, especially for LSTM and language modeling tasks, involves the second gradient as well. This specific version is known as the Adam algorithm (Kingma & Ba (2014))
+- [o] Tt (0: 1==0
m 81m-- (1 (1) 9t.
Pt 320 + (1 (22 g
am,
0: 8 Vor""
where m. = m2/VI B and 0. = wt/ VI 1 By are unbiased estimators the first and second
moment respectively.
2.1 CONVERGENCE
In order to analyze the convergence of these algorithms, we can consider a second-order approxi- mation of J on its combined argument (0; x)) in the region of x0,;
(z) = =z T H b=2,
2
where H; is the Hessian of (z) around z,. This gives us the gradient,
VJ(z)) H224 - b.
which becomes the SGD update step,
2411 2t (H₁z b)-
2

---

Under review as a conference paper at ICLR 2020
Unrolling this update step be shown to lead to expression for the distance from the optimal value z", in the basis Hessian eigenvectors Çy:
Σ (zo (1) adi) Ec-
2t
We can see that the learning stable if the learning rate a satisfies,
add 1.
In addition, find that the value for the learning rate that leads the fastest overall convergence
is,
2
\1 An
where a and Xn are the max and min eigenvalues of H. respectively.
2.2 ADAPTIVE MOMENT ESTIMATION
rather than a single learning rate a, we were to use a diagonal matrix D such that the update is,
1++ Dgt,
may be able to modify the diagonal entries [do, d1, dnl such that faster overall convergence achieved. For example, in the special case that the Hessian diagonal, the convergence rate for the i-th element becomes,
(1 d;Ai),
which has the solution d; = A In this situation, if the eigenvalues X; are known, the algorithm converge to the minimum exactly step. This corresponds with some intuition behind adaptive moment methods: that taking a step with a "constant" size in every direction toward the target will reach convergence faster than taking step proportional the gradient size.
Because the eigenvalues and cigenvectors not known priori, for practical algorithm we must rely on an approximation to find d; One technique named AdaGrad (Duchi et al. (201T) prescribes the diagonal elements:
di E V
(2)
For building intuition, consider the special case where the Hessian is diagonal,
oh. (A, bz)²
we compare the AdaGrad coefficient the optimal value for Combining this with Equation d, X finding,
(3)
+ V. (Niza - 6,)
A;
As long ad2 this will be true when,
by
a
Σ
Zts
constant w.r.t.
(4)
which is the sum squared distance of the parameter value 24, to its optimum 2; = b,/ This reveals some the tradeoffs being made by AdaGrad. It will perform ideally when all parameters start the same distance from their targets.
3

---

Under review as a conference paper at ICLR 2020
This may be true average if the initializations 2os and optima 2 can be made over the same subspaces. That is, if 27 is uncorrelated to Aç we can expect this to have good performance on average.
However, there can be sigmificant errors in both overestimating and underestimating the eigenvalue. One would expect that for typical problem by and X; might be drawn from uncorrelated distribu- tions. In this case, large values of 2 will be likely to correspond to small values A Since 20 can only be drawn from the average distribution (no information is known at this point), the estimated A is more likely to be large, as the initialization will be far from the optimum. Intuitively, the gradi- is large because the optimum is far from the initialization, but the algorithm mistakes this large gradient for a large eigenvalue.
On the other hand, when the parameter is initialized close to its optimum, the algorithm will mistak- enly believe the eigenvalue is small, and so take relatively large steps. Although they do not affect the overall convergence much on their own (since the parameter near its optimum), these steps can add significant noise to the process, making it difficult accurately measure gradients and therefore find the optimum other parameters.
This problem will be significantly worse for Adam, which forgets its initialization with some decay factor B3. In that case, as each parameter reaches its optimum, estimated eigenvalue X drops and the step size gets correspondingly increased. In fact, the overall algorithm can be divergent as each parameter reaches its optimum, as the step size will grow unbounded unless a is scheduled to decline properly bounding method like AMSGrad is used (Reddi al. (2018)).
In addition, reviewing our earlier assumption of small e, these algorithms will perform worse for small eigenvalues X;- c/a. This might be especially bad in Adam where late in training when the learning rate a ~ Σ. Z+z
2
is small.
We finally note that the Hessians in deep learning problems not diagonal (Sagun al. (2016) Li et al. (2019)). As such, each element might be better optimized by a learning rate that better serves both min and max cigenvalues.
SMOOTHED ADAPTIVE GRADIENT
Overall, this understanding has led us to believe that adaptive moments might effectively estimate X; when it is large, but might be less effective when it small. In order to incorporate this infor- mation about large eigenvalues, well optimize the learning rate to account for variation in the eigenvalues contributing to convergence of a particular component, we consider the an update to Eq.
d, = at (1+1)) A + n/
where Ā is an average eigenvalue and is a new hyper-parameter that controls the weighting of the eigenvalue estimation. Here we have added À to the numerator so that does not need to absorb the changes to the RMS error as it does in Eq. + This also recovers the SGD convergence guarantees, since the step always within factor of 17 to an SGD step. In addition, this will allow us to recover SGD with momentum exactly in the limit + 0. We use the adaptive gradient estimation,
(g + (g
Uti 22 + 0t
RJ 2
X
)agg
where in is the mean value of ut, write,
+ 7))
d; 1 // va/ (e2 vz)
One issue with the above estimation is that its variance is very large at the beginning of training (Li et al. (2019)) It was suggested that this is the reason that warmup is needed for Adam and shown
4

---

Under review as a conference paper at ICLR 2020
Algorithm The SoftAdam Algorithm
Input: 0o F:: initial parameters, for 0)(=1 learning rate, Owed B1, 82, yy 6: other hyperpa- rameters, (0): loss function
Output: 0r E
no, to + 0
> Initialize moments to
for t + 1... T
+ VoJ(z)
Find gradient P. Update first moment
m - 3im--1 + (1- 31)
32 t min (12, 1 1 1/1)
" - 1 + (1 (1- B22) 56
b. Update second moment
1/2 meanelem [0g]
Average over all elements
sqrt (1 82) (1 322)
d. (1 + 11)
1 1 114 + mr / (6r + +2))
Calculate the denominator D Perform the update
0: = 0.-1 mide
end for return AT
that rectifying it can make warmup unnecessary. Where i, is the average of n, elements and toc the average noa, we define r; = v n no and:
a (1+1)
d 1+7 Th + W meeorl 0/s/ (e2 + D.)
This finally forms the basis for our algorithm.
3.1
ALGORRITHM
Our algorithm differs from Adam in a few other ways. First, the biased gradient estimate used rather than the unbiased one. This matches the SGD implementation of momentum, and also avoids magnifying the large variance of early gradient estimates:
m --m4-1 + (1 31) gt-
In addition, the second moment v is calculated in an unbiased way using effective (2),, or by abuse of notation 32e:
Bec min B2, 1 - 1) U)
12 + (1-12e) B2x) gc.
This has a negligable impact on the performance of the algorithm, but makes tracking the moment over time easier since it does not need to be un-biased later. We then calculate the ratio of the number of samples n, used to calculate moment v, to steady state number of samples the average Moc:
Thg
B2
Tloo
32r
We finally note that the weight decay should be calculated separately from this update as AdamW (Loshchilov & Hutter (2017)).
5

---

Under review as a conference paper at ICLR 2020
Top-1 validation accuracy (%) CIFAR-10 for different AdamW, SGD and SoftAdam. SoftAdam is restricted to
Table 1:
ARCHITECTURE
ADAMW
SGD
SOFTADAM
AlexNet
74.55 0.37
77.10 0.50
79.30 + 0.25
VGGI9 (BN)
92.50 0.25
9 0.18
93.89 0.12
PreResNet-56
91.93 0.18
94.02 + 0.03
ResNet-110
92.11 0.04
94.54 0.29
94.75 + 0.15
DenseNet-100
93.25 + 0.19
95.04 + 0.08
95.00 0.11
These steps combined and summarized in Algorithm 1 A reference implementation for the PyTorch package (Paszke et al. (2017) is provided in Appendix A
4 EMPIRICAL RESULTS
In order to test the ability of this algorithm to reach better optima, performed testing variety of different deep learning problems. In these problems we keep rj at the default value of Because the algorithm is the same as SGDM if 0 and is comparable to Adam when rj c-l, getting results at least as good as those algorithms is just a matter parameter tuning. These results are intended show that SoftAdam performs remarkably well with common parameter choice. For the best performance, the hyper-parameter and learning rate schedule a should be optimized for particular problem.
4.1 CIFAR-10
We trained a variety of networks! AlexNet (rizhevsky et al. (012)), VGG19 with batch nor- malization (Simonyan & Zisserman 2014)), Nes with bottleneck blocks (He et al. (0155). PreResNet-56 with bottleneck blocks etal DenseNet-BC with I==100 and k=12 (Huang al. (2016) the CIFAR-I0 dataset (Krizhevsky (2012) using SGD, AdamW and SoftAdam. For each model and optimization method, the weight decay was varied over [le-4,2e-4,5e-4, For each
3,5e-3, For A the learning rate varied over optimizer and architecture and the best result was chosen, and three runs with separate initializa- tions were used go generate the final data. The learning schedule reduced the learning rate by factor of 10 at 50% and 75% through the total number of epochs. The results are summarized in Ta- [ We find that SoftAdam equals or outperforms SGD training classifiers on this dataset. Due to the larger optimal weight decay constant, SoftAdam achieves lower validation loss at higher train loss than SGD.
4.2 PENN TREEBANK
We trained a 3-yer LSTM with 1150 hidden units per layer on the Penn Treebank dataset (Mikolov al. (2010)) in the same manner Merity et al. (2017). For SoftAdam the weight drop was increased from 0.5 to 0.6. Results for the average of three random initializations are shown in Figure 2 (a) and are summarized in Table 2 For these parameters, SoftAdam outperforms SGD significantly but does not quite achieve the same results as Adam. Note that for this experiment we chose Adam instead of AdamW for comparison due its superior performance.
4.3 IWSLT' GERMAN TO ENGLISH
We also trained transformer using the fairseq package by Ott al. (201)) the IWSLT" 14 Ger- man to English dataset. Results for each method with optimized hyperparameters are summarized Table 3 Note that no warmup used for training SoftAdam, but warmup is used for AdamW and
Implementation based the public repository at https: / //gi i thub. cato..com/bbaa bbom/baatt rpaw/ pytorch classi

---

Under review as a conference paper at ICLR 2020
1000 mg)
35
05
(al) AlexNet Train Loss
(a2) AlexNet Val. Loss
(a3) AlexNet Val. Acc.
Blaret Mort)
24-
34
0231
52
(bl) VGG19 Train Loss
(b2) VGG19 Val. Loss
(b3) VGG19 Val. Acc.
02
85
344
081
84
4301
62
(c1) PreResNet Train Loss
(c2) PreResNet Val. Loss
(c3) PreResNet Val. Acc.
idemm plaan
1g0 (lar)
(lear
idemm Stant 19C Maar))
sertccam (Marr
ademn Maare)
85
84-
047
85
82
(dl) ResNet Train Loss
(d2) ResNet Val. Loss
(d3) ResNet Val. Acc.
plaat og Pea))
saaes plane
02
641
8.2
82
(el) DenseNet Train Loss
(e2) DenseNet Val. Loss
(e3) DenseNet Val. Acc.
Figure I: Training results for CI Architectures are (a) AlexNet, (b) VGG19 (BN), (c) PreResNet-56, (d) ResNet-110, (e) DenseNet-100 100 (BC) using SoftAdam, Adam W and SGD. Each line indicates the average of 3 training runs with random initializations.
7

---

Under review as a conference paper at ICLR 2020
70
(a) 3-layer LSTM on PTB
(b) fairseq on IWSLT" 14 De-En
Figure 2: Language model validation perplexity during training for SoftAdam, Adam/ W and SGD.
Table LSTM validation perplexity, test perplexity, and test loss on the Penn Treebank dataset for SoftAdam, Adam and SGD.
OPTIMIZER
VAL. PPL.
SoftAdam
63.63 + 0.19 66.64 0.85
SGD
Adam
62.03 = 0.34
SGD. Following Liu et al. (2019), we use a linear learning rate schedule. For these parameters, we again find that Aom outperforms SGD significantly.
CONCLUSIONS
In this paper, we have motivated and demonstrated a new optimization algorithm that naturally unifies SGD and Adam.
We have focused our empirical results on the default hyper-paramen parameter setting, = 1, and predeter- mined learning schedules. With these parameters, the algorithm shown to produce optimization that is better than or equal to SGD and Adam on image classification tasks. It also performed sig- nificantly better than SGD language modeling tasks.
Together with finding the optimal values for we expect a better understanding of the learning schedule to bring light the way in which the adaptive gradient methods improve convergence. SoftAdam now also makes it possible to create a learning schedule on which may be another fruitful avenue of research, expanding on the work of Ward al. (2018).
Better understanding of how adaptive gradients improve the convergence of practical machine learn- ing models during training will enable larger models be trained more accurately in less time. This paper provides useful intuition for how that occurs and provides new algorithm that can be used improve performance across diverse set of problems.
Table 3: Best results for training IWSLT' 14 De-En with different optimization algorithms.
OPTIMIZER
VAL. PPL.
BLEU SCORE
SGD
4.14
31.43
SoftAdam
4.02
32.76
AdamW
3.87
34.44
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Lo Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning, arXiv preprint arXiv:1606.00 04838, 2016.
Jinghui Chen and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(uul::121-21159 2011.
2.2
Gabriel Goh. Why momentum really works. Distill, 2017.
doi:
URL
2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint arXiv: 2015.
4.1
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. arXiv preprint arXi:: 0.0000000022.. 2016. 4.1
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, 2016. 4.1
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
1.1
Nitish Shirish Keskar and Richard Socher Improving generalization performance by switching from adam to sgd. arXiv preprint 172.:217,,. 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.698.. 2014.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University Toronto, 05 2012. 4.1
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097- 2012.
Jason D. Lee, Max Simchowitz, Michael I. Jordan, and Benjamin Recht. Gradient descent converges to minimizers. arXiv preprint 1rXiv:1602.04991. 2016. [
Xiaoyu Li and Francesco Orabona. On the convergence of stochastic gradient descent with adaptive stepsizes. arXiv preprint / 14, 2018.
Xinyan Li, Qilong Gu, Yingxue Zhou, Tiancong Chen, and Arindam Banerjee. Hessian based analysis of sgd for deep nets: Dynamics and generalization. arXiv preprint 19ti::197..11000
2019. 2.2
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv: 03000006000000355,
2019. 4.3
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 2017. 3.1
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv:1902.0900 2019. [.]
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing 1stm lan- guage models. 2017.
Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Eleventh annual conference of the international speech communication association, 2010. 4.2

---

Under review as a conference paper at ICLR 2020
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019. 4.3
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017. 3.1
Boris Polyak and Anatoli Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838 855, 1992. 1.1
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018. URL /pp[/ttttes:/eeeeeeeeee
net/ Fort//orrmiiidee ryQu R2
Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singu- larity and beyond. arXiv preprint ar1iv: 1.07476, 2016. 2.2
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:1409.1555 2014. 4.1
Samuel L. Smith and Quoe Le. bayesian perspective on generalization and stochastic gradient
descent. In International Conference Learning Representations, 2018. URL https:// openreview net./oormmmdd-Bii11110
Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes, from any initialization. arXiv preprint 1rXh::186.0018//1 2018.
Ashia C. Wilson, Rebecca Roelofs, Mitchell Stem, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. arXiv preprint
08292, 2017. [
Michael R. Zhang, James Lucas, Geoffrey E. Hinton, and Jimmy Ba. Lookahead optimizer: steps forward, step back. CoRR, abs/1907.08610, 2019. URL.ttpp: htt.::p://aa.....orrabbbb/////// 08610
A REFERENCE PYTORCH IMPLEMENTATION
import math import torch
from torch. optim. optimizer import Optimizer import numpy np
class SoftAdam Optimizer):
def .nint..
self.
params.
Ir=le-1, betas=(0.9, 0.999). eps=le-8.
eta=1.0,
weight we
nesterov=False
): defaults = diet
Ir=lr, betas=betas.
eps=eps,
eta=eta,
10

---

Under review as a conference paper at ICLR 2020
wrighh.decassweihhh.deayy nesterov=nesterov,
super (SoftAdam, self). init (params, defaults)
def step (self, closurc=None):
loss None
if closure is not None: loss closure
for group in self param groups: for in group params"]: if p.grad is None: continue grad = p. grad....... data
if grad. is is_sparse: raise RuntimeBrror
"SoftAdam does support sparse gradients"
nesterov group [" nesterov
state = self state [p]
State initialization if len statee == 0:
state step"]
# Exponential moving average of gradient values state exp avg"l = torch. zeros like data
# Exponential moving average of # squared gradient values
state exp-avg-sq") torch, zeros-like data
exp_avg. exp_avg-sq =
state [" exp_avg "] state (" exp avg sq "]
betal, beta2 = group betas"]
state (ratel"sste) step "J += 1
beta2_hat min(
beta2, 1.0 1.0 / (state [" step "])
beta (1 beta2) / (1 beta2 hat) eta_hat2 group [" eta"] group I" ela"] * r-beta
Decay the first and second moment with the # running average coefficient
exp avg mul (betal), add (1 betal, grad) exp_avg-sq wwl.
addcmul_
1 - beta2-hat, grad, grad
Create temporary tensor for the denominator
11

---

Under review as a conference paper at ICLR 2020
denom exp-avg-v g-g.............. mul
/ eta_hat2
torch mean ( exp avg sq)
group ""pppp"""s") group [" eps") "]
denom. sqrt () add
1 group eta "I up. (eta.haa))
wd = group l" weigght group["Ir"]
p. data add (-wd, p. data)
Ir group (1 group["etat "I)
if nesterov: p. a addediv.
-Ir.eff betal, exp-avg, denom
p data addediv
-Ir_eff (1 - betal). grad, denom
else:
p. data addediv addediv-(-Ir.eed) I-Ir-eff, exp-avg, denom)
return loss
12


Reviews:
[{"review_id": "rkx2eAItFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tends to explain how the tradeoffs between convergence speed and convergence performance are made by different optimization methods. Moreover, the paper modifies Adam-like updating rules and proposes a novel optimization methods, SoftAdam. Finally, the paper performs numerical experiments on traditional image classification tasks as well as language modeling tasks.\n\nPros\nThe paper involves the language modeling tasks in empirical results besides traditional image classification tasks, which helps to explain the convergence performance of optimization methods in a wider range of applications.\n\nCons\n1. The writing of this paper is not well organised. Section 1 lacks detailed description of the main idea and the proposed optimization methods, which actually confuses the reader. Section 2 describes too much details of SGD and Adam, and lacks a clear \"intuition\" which readers exactly expect.\n2. The notation in the paper is little confusing. In the update rule of z_{t+1} in Section 2, what is meaning of z? In Section 3, n_t and n_\\infty are used before a proper defination, and what is relationship between \\alpha and \\alpha_t in the implementation of the SoftAdam?\n3. The motivation of the proposed method is weak. Such a weak motivation is mainly because of the insufficient \"intuition\" in Section 2. The author mentions \"the Hessians in deep learning problems are not diagonal\", but does not provide further explanation on why more importance should be lay on serving both max and min eigenvalues.\n4. There are also several minor problems on the numerical results. Firstly, why the colors of \"softAdam\" and \"sgd\" are switched several times in Figure 1? Secondly, the figural result in Figure 1 and he numerical results on language processing models both lack a display of the confidence range.\n     ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1ghLD70FH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new algorithm which brings closer SGD and Adam while also incorporating new corrections to improve behavior of the optimizer in contexts where there is very small or very large eigen values.\n\nDecision\n\nI vote for weak rejection because the core modification proposed to Adam is minor and is mostly supported by intuition and preliminary experiments.\n\nJustification\n\nThere is many modifications proposed, but most are secondary corrections for stability, such as the warm-up schedule with the redefinition of beta_2. These modifications could as well be incorporated in Adam without the core modification that is the smoothing presented in section 3. These additional modifications also make it difficult to measure the importance of the core contribution. Without getting rid of them, an ablation study on toy problems (even synthetic data) would be necessary for a better understanding.\n\nIn section 3.1, the temporal definition of beta_2t is integrating a warm-up. While the reason for doing so is supported in introduction of section 3, the effect of this modification should be weighted against no warm-up, and also compared with its effect on Adam.\n\nThere is an error in algorithm 1. The last element of the last line (Perform the update) should be \\alpha (1 - \\eta) m_t/d_t. The code in Appendix corroborates this correction. Minor related note, the use of d_t to define the denominator of what d_i represents in section 3 is very confusing. I would suggest to use the ratio notation of d_i from the equations in the algorithm for coherence.\n\nIf we get pass the warm-up scheduling, by massaging the equation we get that the algorithm is different from Adam on 2 points, 1) the bias are not corrected and 2) the denominator sqrt(v) + epsilon is replaced by sqrt(v) + sqrt(mean(v) + epsilon^2). I have difficulty convincing myself that smoothing by the average is solving the issues raised in the paper and there is no experiments to study its effect directly.\n\nThe experiments are on 3 datasets, but only the computer vision ones are run on multiple architectures. Caption of figure 1 explains that each model is trained 3 times, but the source of variation between each run is not described. Are the models initialized differently? In any case, there is an overlap for 3 of the 5 models between SGD and SoftAdam which makes the comparison rather unconvincing. There is no standard deviation for Adam, and none on Penn Treebank dataset and IWSLT. For a better comparison, all hyper-parameters of the algorithms should be optimized for each run. I understand that SoftAdam is meant to be close to both SGD and Adam, but using the same hyper-parameters may induces misleading results by favoring some (model, optimizer) combination nevertheless.\n\nMinor comments\n\nIn section 2, second paragraph, the term 'mini-batch' should be used instead of 'batch'.\nIn section 2, last sentence, the betas should have no t.\nIn section 2.1, fourth equation (unnumbered), the eigen vector xi_i is presented as a vector and then used as a scalar. Notation should be uniformed.\nIn Section 2.1 around equation (2), the use of i and j is incoherent.\nIn Section 3:\n- Overall, this understanding *of* has\n- we consider the *an* update", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeabQxpFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work proposed a new algorithm called softAdam to unify the advantages of both Adam and SGD. The authors also showed experiments to backup their theoretical results. \n\nPros:\n\nThe authors provided analysis of different algorithms including Adam and SGD on simple quadratic function, then proposed a new algorithm called softAdam which outperforms both Adam and SGD. Experiment results backup their theory. \n\nCons:\n\n- The novelty of this work is limited. The main contribution of this work is to provide a new adaptive gradient method called softAdam, which changes the update rules for some parameters including \\beta. However, neither intuition or theoretical guarantees are provided in this paper. I recommend the authors to add some explanation about why softAdam outperforms other existing algorithms. Besides, the difference between softAdam and original Adam method is little. \n- The theoretical analysis about existing adaptive methods provides nothing new. The authors showed some analysis on quadratic model, which is a very simple model and hence can not reflect the true model we face in the practice. I suggest the authors provide some analysis on more general model, including convex functions and non-convex functions. \n- The settings of experiments are limited. The authors should at least compare softAdam with other baseline algorithms on some modern deep learning tasks including Imagenet.  \n\nMinor comments:\n\n- Page 4, section 3, 'this understanding of has'... lacks object.\n- This paper lacks some references in this area. \n\nJ. Chen and Q. Gu. Closing the generalization gap of adaptive gradient methods in training\ndeep neural networks. arXiv preprint arXiv:1806.06763, 2018.\nWard, R., Wu, X. and Bottou, L. (2018). Adagrad stepsizes: Sharp convergence over nonconvex\nlandscapes, from any initialization. arXiv preprint arXiv:1806.01811 .\nLi, X. and Orabona, F. (2018). On the convergence of stochastic gradient descent with adaptive\nstepsizes. arXiv preprint arXiv:1805.08114 .\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agreed that the proposed modification was minor. I encourage the authors to pursue in this direction, as they mentioned in their rebuttal, before resubmitting to another conference.
Decision: Reject