Admeta: A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers with Bidirectional Looking
Authors: Yineng Chen
Zuchao Li
Lefei Zhang
Bo Du
hai zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: optimizer
double exponential moving average
bidirectional looking
Adam
SGD

Abstract: Optimizer is an essential component for the success of deep learning, which guides the neural network to update the parameters according to the loss on the training set. SGD and Adam are two classical and effective optimizers on which researchers have proposed many variants, such as SGDM and RAdam. In this paper, we innovatively combine the backward-looking and forward-looking aspects of the optimizer algorithm and propose a novel \textsc{Admeta} (\textbf{A} \textbf{D}ouble exponential \textbf{M}oving averag\textbf{E} \textbf{T}o \textbf{A}daptive and non-adaptive momentum) optimizer framework. For backward-looking part, we propose a DEMA variant scheme, which is motivated by a metric in the stock market, to replace the common exponential moving average scheme. While in the forward-looking part, we present a dynamic lookahead strategy which asymptotically approaching a set value, maintaining its speed at early stage and high convergence performance at final stage. Based on this idea, we provide two optimizer implementations, \textsc{AdmetaR} and \textsc{AdmetaS}, the former based on RAdam and the latter based on SGDM. Through extensive experiments on diverse tasks, we find that the proposed \textsc{Admeta} optimizer outperforms our base optimizers and shows advantages over recently proposed competitive optimizers. We also provide theoretical proof of these two algorithms, which verifies the convergence of our proposed \textsc{Admeta}.

Under review as a conference paper at ICLR 2023
ADMETA: A NOVEL DOUBLE EXPONENTIAL MOVING AVERAGE TO ADAPTIVE AND NON-ADAPTIVE MOMEN- TUM OPTIMIZERS WITH BIDIRECTIONAL LOOKING
Anonymous authors
Paper under double- blind review
ABSTRACT
Optimizer is an essential component for the success of deep learning, which guides the neural network to update the parameters according to the loss on the training set. SGD and Adam are two classical and effective optimizers which researchers have proposed many variants, such as SGDM and RAdam. In this paper, we innovatively combine the backward-lookinn looking and forward- optimizer algorithm and propose a novel ADMETA (A Double exponential Moving averagE. To Adaptive and non-adaptive momentum) optimizer framework. For backward-looking part, we propose DEMA variant scheme, which is motivated by metric in the stock market, replace the common exponential moving average part, we present a dynamic lookahead
aspects of the
scheme. While in the strategy which asymptotically approaching set value, maintaining its speed early stage and high convergence performance at final stage. Based on this idea, we provide two optimizer inplementations, ADMETAR and ADMETAS, the former based on R Adam and the latter based on SGDM. Through extensive experiments on diverse tasks, we find that the proposed ADMETA optimizer outperforms our base optimizers and shows advantages over recently proposed competitive optimizers. We also provide theoretical proof of these two algorithms, which verifies the convergence of proposed ADMETA.
INTRODUCTION
The field of training neural network dominated by gradient decent optimizers for long time, which use first order method. Typical ones include SGD (Robbins & Monro 195T) and SGD with momentum (SGDM) (Sutskever et al.] 2013), which are simple yet efficient algorithms and enjoy even better resulting convergence than many recently proposed optimizers. However, it suffers the disadvantage of low speed initial stage and poor performance sparse training datasets. This shortcoming can not be ignored since with the development of deep learning, the amount of data becomes much larger, and the model becomes much more complex. Time to train a network is also considered important metric when evaluating optimizer. To address this issue, optimizers with adaptive learning rate have been proposed which use nonuniform stepsizes to scale the gradient while training, and the usual implementation is scaling the gradient by square roots of some kind of combination of the squared values of historical gradients. By far the most used are Adam (Kingma & Ba 2014) and AdamW (Loshchilov & Hutter 2017) due their simplicity and high training speed in early stage. Despite their popularity, Adam and many variants like of it (such as RAdam (Liu et al., 2019h) is likely to achieve worse generalization ability than non-adaptive optimizers, observing that their performance quickly plateaus on validation sets.
To achieve a better tradeoff, researchers have made many improvements based on SGD and Adam family optimizers. One attempt is switching from adaptive learning rate methods to SGD, based on the idea of complementing each other's advantages. However, a sudden change from one optimizer to another in set epoch step is not applicable because different algorithms make characteristic choices at saddle points and tend to converge to final points whose loss functions nearby have different geometry (Im et al., 2016). Therefore, many optimizers based on this idea seek for a smooth switch. The representative ones are AdaBound (Luo et al. (019)) and SWATS (Keskar & Socher 2017). The second attempt is proposing new method further accelerate SGDM, including introducing power

---

Under review as a conference paper at ICLR 2023
exponent (pbSGD Zhou al., 2020 aggregated momentum (AggMo (Lucas et al. 2018)) and warm restarts (SGDR (Loshchilov & Hutter 2016)). The third attempt is modifying the process of optimizers with adaptive learning rate achive better local optimum, which is the most popular field in recent researches (Zhuang et al. 2020 Li et al. 2020). Due to space constraints, please see more related work in Appendix A
We focus in this paper on the use of historical and future information about the optimization process of the model, both of which we argue are important for models to reach their optimal points. To this end, we introduce a bidirectional view, backward-looking and forward-looking In the backward-looking view, EMA is an exponentially decreasing weighted moving average, which is used as trend-type indicator in terms of the optimization process. And since the training uses a mini-bar batch strategy, each batch is likely have deviations from the whole, so may mislead the model to the local optimal point. Inspired by stock market indicators, DEMA (Mulloy 1994) is an exponential average calculated on the traditional EMA and current input, which can effectively maintain the trend while reducing the impact caused by short-term bias. We thus replace the traditional exponential moving average (EMA) with double exponential moving average (DEMA). is worth noting that our usage is not equivalent to the original DEMA, but rather variant of it. In the forward-looking part, since we observe that a constant weight adopted by the original Lookahead optimizer (Zhang et al.) (2019) control the scale of fast weights and slow weights in each synchronization period makes the early stage training slow and lossy, we propose a new dynamic strategy which adopts an asymptotic weight for improvement. By applying these two ideas, we propose ADMETA optimizer with ADMETAR and ADMETAS implementations based on RAdam and SGDM respectively.
Extensive experiments have been conducted on computer vision (CV), natural language processing (NLP) and audio processing tasks, which demonstrate that our method achieves better convergence results compared other recently proposed optimizers. Further analysis show that ADMETAS achieves higher generalization ability than SGDM and ADMETAR achieves better convergence results and maintain high speed in initial stage compared to other adaptive learning rate methods. We further find that DEMA and dynamic looking strategy can improve performance compared to EMA and constant strategy, respectively. In addition, we provide convergence proof of our proposed ADMETA in convex and non-convex optimizations.
2
ADMETA
2.1
BACKGROUND
The role of the optimizer in model training is to minimize the loss on the training set and thus drive the learning of model parameters. Formally, consider a loss function f: Ed + R that is bounded below greater than zero, where represents the field of real numbers, denotes the dimension of the parameter and thus Rd denotes d-dimensional Euclidean space. The optimization problem can be formulated as: minge (f), where # indicates parameter whose domain is F and F C Rd. If we define the optimum parameter of the above loss function 0", then the optimization objective can be written as:
0- arg min f(0). AEF
(1)
Optimizers iteratively update parameters make them close to the optimum as training step t increases, that is to make: lim 110,
0- = O.
Stochastic gradient algorithm SGD (Robbins & Monro 1951) optimizes f by iteratively updating
parameter O, at step in the opposite direction of the stochastic gradient g(0;-1; &,) where & is the input variables of the t-th mini-batch in training datasets. For the sake of clarity, we abbreviate (e) as 9r for the rest of the paper unless specified. SGD optimization aims to calculate the updated model parameters based on the previous model parameters, the current gradient and the learning rate. Define learning rate as og, the update process is summarized as follows:
0, 0 Gr
Original SGD tend to vibrate along the process due to the mini-batee strategy and not using of past gradients. What's 's more, this disadvantage also results in its long- time plateaus in valleys and saddle points, thus slowing the speed. To smooth the oscillation and speed up convergence rate, momentum, also known as Polyak's Heavy Ball (Polyak 1964), is introduced to modify SGD. Momentum step
2

---

Under review as a conference paper at ICLR 2023
t is often denoted as m, and obtained by iterative calculation with dampening coefficient B. Thus, the update process of SGD with momentum (SGDM) (Sutskever et al. 2013) becomes as follows:
my = - + (1 - 3)ge,
(3)
02 0:-1 a.
(4)
Although momentum works well, the uniform stepsize on every parameter is also another factor to limit the speed, especially in large datasets and sparse datasets. To further accelerate the update, adaptive learning rate optimizer is introduced which adopts an individual stepsize for each parameter based on their unique update process.
Since a smoothing mechanism is employed in the calculation of stepsize, two dampening coefficients, 31 and $g. are intro-
my = - 1 + (1 -1)g
(5)
duced for balancing the current and historical information.
Rs = 82 + (1 32)
(6)
Adam (Kingma & Ba 2014, a typical adaptive learning rate
O: = 0:-1 - ax//t// V
(7)
optimizer, is implemented as follows:
where my indicates the first momentum, corresponding to the momentum in SGDM; 02 indicates the second momentum.
To emphasize the functionality of Dt, we call it adaptive item for the rest of the paper. Adam may sometimes converge to bad local optimum, partly due its large variance in the early stage. To fix this issue, RAdam (Liu et al., 2019) in- troduces a further rectified item r' and split the update process into two sub-processes sequen- tially connected:
Pco 2/(1 -82) 1, Pt = Pox. - 21,32/ B3),
(8)
(9)
(pr - -p) 2)
Tt x
V (px -)) 2)pp
(10)
0,-1 0p7M4
Pt S 4
(11)
0-1 catemee /
Pt > 4
2.2 BACKWARD-LOOKING
In fact, the calculation of momentum my in Eq. (3) and Eq. 5 is an exponential moving average (EMA) on gradient 9t- EMA, also known exponential weighted moving average, can be used to estimate the local mean value of variables, so that the update variables is related to historical values over period of time. Formally, EMA is expressed as:
S, 3S + (1 (11-3)p 3)p::
(12)
where the variable S' is denoted as S; at time and is the new assigned values. Particularly, S, Pt without using EMA. In Eq. B3.. SGDM employs EMA to take a moving average of the past gradients. While in Eq. 5p. Adam and RAdam further apply EMA on the square of past gradients to construct the adaptive item. In the EMA, the moving average of the variable S at time t is roughly equal to the average of the values p over the past 1/(1 - B) steps. This makes the moving average vary more at the beginning, so a bias correction is proposed and used in Adam (Eq. 7 and in RAdam (Eq. (1I) when p > 4.
EMA can be regarded as obtaining the average values of the variables over time. Compared with the direct assignment of values variables, the change curve of the values obtained by moving average is smoother and less jittery, and the moving average does not fluctuate greatly when inputting outliers, which is very important for the optimization using sampled mini-batch. Although efficiint, EMA is not necessarily the best strategy for using historical information when it comes to the backward- looking part. Although it can effectively suppress the vibration caused by mini-batch training by performing the moving average on ft. it also brings lag time that affects the convergence speed and increases with the length of the moving average. What's more, it can result in overshoot problem (An) al.) 2018), one possible reason is that EMA might make the wrong use of historical gradients in the final stage and thus have a "burden" to converge to optimum.
Double Exponential Moving Average (DEMA), first proposed by Mulloy ( (1994), is a faster moving average strategy and was invented to reduce the lag time of EMA. Thus, motivated by the advantage of DEMA, we developed DEMA variant for the model optimization. It is worth noting DEMA is not simply taking a moving average of historical gradients twice, instead, it takes the moving average of the linear combination of the current gradient the moving average of past gradients. The form of
3

---

Under review as a conference paper at ICLR 2023
our DEMA variant can be written as:
DEMA EMA out
(NEMA in + KKge).
(13)
where jr and i: are coefficients that control the scale of current gradient and only depends on B. From the formula EMA- = 1 gi. past gradients follow a fixed proportionality, that is, the ratio of gradient weight at one time to gradient weight at the previous time is B.
Due to the use of minibatch training strategy, the input is randomly sampled. The effect of each minibatch towards optimization is varied. Therefore, applying a fixed proportional past gradients is not a reasonable approach since it does not take into account the changeable situation. The disadvantage of overshoot that EMA usually has may also be caused by the above reasons 2018).
(An et al.)
Thus, we deal with the relationship between the historical gradients and the current gradient more flexibly by further
T. = A + gt,
(14)
controlling the proportion of past gradients. Our design of coefficients in DEMA is also for this purpose. Based on Eq. algorithm is:
he rggt + plt + V
(15)
my = 3mm-1 + (1 3)
(16)
(13), our actual implementation
where 7 is the output of EMA" with O initial value and m, is the output of EMA out also initiated with O. and 3 are dampening coefficients of inner EMA and outer EMA respectively, v is bias item, which is set to a small amount that decreases exponentially to 0 and chosen X'g1. The bias item does not affect the convergence proof, so for the sake of brevity, is omitted for the rest of this paper and the details can be seen in the code. Please refer to Appendix for more comparison and discussion between EMA and DEMA.
2.3 FORWARD-LOOKING
Focusing on gradient history, that is, backward-looking, optimizer is conducive alleviating the vibration problem in the optimization process and preventing it from being misled by local noise information. However, since the optimization problem of the deep neural network very complex, the optimizer can make the optimization process more robust by pre-exploration, so as to obtain better optimization results, which is called forward-looking.
Based on Reptile algorithm and advances understand- ing the loss surface, Zhang al. (2019) proposed Looka- head optimizer, which introduces two update processes and averages fast and slow weights periodically. The algorithm can be expressed as the cycle of the following process:
Pre-exploration 8, = =PPTIMM(())-)))
Synchronization (every ls steps)
Φ = Φt-k + 11(0011 Φt-k) 02 Φ2.
where OPTIM(-) denotes a chosen optimizer, & denotes the synchronization period, or in other words, the period of forward-lookee looking. denotes the slow weights, 6, denotes the fast weight updated with a chosen optimizer, and v is a constant coefficient controlling the proportion of slow weights and fast weights in each synchronization. Generally, the chosen optimizer be arbitrary.
We can get an intuitive explanation of Lookahead optimizer from the pseudo code above: Guided by fast weight 0,, the slow weight de updates by taking linear interpolation between itself and the fast weight. Every time the fast weight updates & steps, the slow weight updates step. The update direction of slow weight can be regarded as 0, - Φe from the equation. Therefore, 7) can also be interpreted as the stepsize of slow weight in each synchronization. In order not to be confused with the stepsize of fast weight. we rename the stepsize of slow weight as stepsize,. The recommended value of in (Zhang et al. 2019) is 0.5 and 0.8.
In the original Lookahead optimizer implementation, the fast and slow optimization processes were synchronized according to given period, and parameters are fused at fixed ratio during synchronization. However, optimization is a continuous process. In different optimization stages, fast optimization steps have different guiding effects on parameters. We argue that using fixed stepsize, in each synchronization is not an optimal strategy, and may even lead to negative effects. For this consideration, we turns the constant into a π that changes over step monotonously and asymptotically. Generally, n. is a function that starts from I and converges to a set value and depends only on the step f. In this setting, the proportion of slow weights increases and this part gradually
4

---

Under review as a conference paper at ICLR 2023
turns into the original Lookahead method. In other words, the slow weights in method adopts faster stepsize, at the beginning, and it asymptotically slows down as processing. Specifically, we define two asymptotic functions for T/:
= 0.5 * (1 (1+ 0.01 1/ = 0.8 (1 0.1v 3.8
(17)
thus we call this as dynamic asymptotic lookahead. The two functions are designed to turn t/t from 0.5 and 0.8 respectively. Notably, these asymptotic functions may not be the best. We just find that it works well and maybe future work can done to investigate a more suitable one. For the sake of clarity, we will use the latter one the rest of the paper and the results of experiments trained from scratch are based on this function unless specified.
To illustrate the advantages of our dynamic lookhead strategy over no lookhead and the original constant lookahead, we give an optimization example in Figure [ In region D which is around early stage, the direction of the update is relatively stable and a large stepsize, is needed. 0, -> 04 denotes the update of fast weights. A constant lookahead method will slow the update process in each synchronization period, as can be seen in 0 - 02. In our method, fast weights share more proportion in each synchronization period in early stage, thus updating more fast, as can be seen in 01 03- In region (2), which around final stage, the direction of the update is relatively ocillated, and small stepsize, is needed. fast weights tend to overshoot the optimum, as can be seen in 0g - 0g. Lookahead optimizer can achieve better convergence result than gen- eral algorithm as it averages the weights to make them more close the optimum point, as can be seen in 0s - 0g. In our method, the proportion of fast weights have already reduced asymptotically to a set value, thus achieve similar efficacy Lookahead op- timizer as can be seen in 03- + 07.
f(x
5GD
SG0 with Lookahcad
dynamic (Ours)
0) is large, large maintain the convergence speed
d. d) small, a relative small y can achieve better convergence
From these analysis, we demonstrate that our dynamic lookhead strategy method improves the robustness of training.
Figure 1: Comparison between no lookahead, constant lookahead and dynamic lookahead.
2.4 IMPLEMENTATION OF ADMETAR AND ADMETAS
Since optimizers of the Adam family and SGD family have their own advantages and disadvantages, and the bidirectional looking optimizer framework and improvement we propose do not have too many restrictions on the basic optimizer, we have implemented improved versions ADMETAR and ADMETAS based on RAdam and SGDM optimizer. The final algorithm form is shown in Algorithm Detailed proof of convergence and convergence rate for our ADMETAR and ADMETAS is putted Appendix and D
I and 2
EXPERIMENTS
In this section, we demonstrate the effectiveness of our optimizer by turning to an empirically explo- ration of different datasets and different models to compare some popular optimizers. Specifically, we conduct experiments typical CV, NLP, and audio processing tasks. Influenced by the Transformer structure, models are becoming deeper and larger, and therefore training is becoming more difficult. is mainly used for large models. Therefore,
The current paradigm of pre-trate nne-trrining-fine-tmee compare optimizers not only in the maining-from-sceee
setup, but also in the fine-tuning setup.
In this section, we compare proposed optimizer with several typical optimizers, including classic SGD [Robbins & Monro [951 and Adam (Kingma & Ba 2014). our base, SGDM (Sutskever et all 2013 and RAdam (Liu al. 2019). the current state-of-the-art AdaBelief (Zhuang al. 2020).
Notably, employed nesterno momentum baseline
Nesterov 1983 in the SGDM for stronger comparison
5

---

Under review as a conference paper at ICLR 2023
and the optimizer combined of many modules, Ranger (Wright 2019). Please refer to Appendix for more experimental details.
Algorithm 1: ADMETAR Optimizer. All
Notations:
operations element-wise.
ax: learning rate at step
Initialize 8, E F. do + O, mo t- 0,
A, G, 3r, .2: the momentum coefficients
up 0, To + 0, ( <
6: a small value used to avoid zero de- nominator
for 12, do
t t+1
k: synchronization period
9: + V.f((0.)
IL (v) = argmin N (
I - A4-1 t
h4 - kgt + =ul
p 25 - 10 (A+ x) = 10 X 9
my < mm
+ (1 hi
Algorithm 2: ADMETAS Optimizer. All oper-
88
Pt < Poo-
2t 1-
ations are dlement-wise.
if the variance is tractable, i.e,
Initialize 0 E F, 20 < 0, mo + 0, To + 0,
Pt 4, then
0 for 1= do
De + 820 + -/2
[p) -1ppm
t 3 t t+1
Tt 6-
(px 1)(p -Bpp:
9r t (f((())
m4 f I 3
me Ut t- bt
1-31
I. A1-11 + 9
0+t1 f IT 1T, (6t- 04t voi+e
re
he + filt + plx
my + Bm (1 B3
else
0.+1 <- 02 0
0,41 + /,,, (8, m)))
if % = 0:
t+ % k 0:
Φ: n + (1 m)) d1
Φt - 1, + (1 - (7:) 0r
0, + Φt
0 + de
end for return x
end for return x
3.1 IMAGE CLASSIFICATION
Consistent with general optimizer researches (Zhuang et al. 2020). we conduct experiments on two image classification tasks, CIFAR-10 and CIFAR- 100 (Krizhevsky et al., 2009) in CV field, and the results are presented in Table For model baselines, we choose the popular and leading performance ResNet- 110 (He et al., 2016) and PyramidNet (Han et al. (017), respectively. From the experimental results, whether in CIFAR-IO 10 or CIFAR- 100 dataset, and based on the ResNet- 110 or PyramidNet model, SGDM achieves better results than SGD, indicating that backward-looking improves the optimization effect. EMA with rectified item in RAAam performs better than EMA in Adam, suggesting that better backward-looking process can lead to performance gains. Comparing SGDM and R RAdam, we find that SGDM has a performance advantage, showing that though Adam uses adaptive learning rate improve the speed of convergence, it is lossy for performance.
Among optimizers with adaptive learning rate, AdaBelief achieves better results than Adam and RAdam in CIFAR-10 with PyramidNet and CIFAR-100 with ResNet-110 and PyramidNet. Ranger, which combines forward and backward looking, achieves better performance than the backward- looking-only RAdam in CIFAR-10 and CIFAR- with PyramidNet. Our ADMETAR achieves consistent improvement over the optimizer baseline RAdam, which also confirms the gain of bidirec- tional looking for optimization. And ADMETAR has better results than Ranger, indicating that our bidirectional looking is better than Ranger's s simple combination of multiple optimization features. Our ADMETAS also performs better than SGDM, further demonstrating the adaptability of our approach, which not only performs well in Adam family, but also works in SGD family.
Following the previous practice (Luu et al. 2019), we visualize the optimization process of the ResNet-110 model with Adam, RAdam, SGDM, and our ADMETAS, ADMETAR optimizers on the CIFAR-10 10 and CIFAR-106 100 datasets in Figure 2. As can be seen from the training loss figure, the above optimizers can successfully train the model to converge a stable state, but ADMETAS obtains the lowest training loss on 10, while AdaBelief obtains the training loss on CIFAR- 100. In terms of performance on the test set, ADMETAS has obtained the best generalization ability,

---

Under review as a conference paper at ICLR 2023
Table Results on CIFAR-IO and CIFAR-100 test sets,
CIFAR-10
CIFAR-100
Model
ResNet-110
PyramidNet
ResNet-110
PyramidNet
SGD
90,27:4:0115
91.52200003
65.70:1:0.25
76.51+0.006
SGDM
93.68:+0.20
95.08-20.13
72.07=00.28
79.49=00111
Adam
91.89:10.23
94,551:0.24
68.45.10.43
76.72:10.32
RAdam
93.09:+0.05
94,58++0.14
70.39++0.08
76.02:00.53
Ranger
92.85:1:0.34
94.76:1:0.03
68.96-1:0.68
76.15:10.08
AdaßBelief
92.81.100.6
94.70.10.00
70.88.10.07
76.57.10.04
ADMETAR
93.63-1-0.22
94.81:1:0.19
71.00:1:0.05
76.82-1:0.07
ADMETAS
94.12.10.17
95.30 0.08
0..24
1..34
CHAM 1D
CHAMR-
CIRAR-102
CIFAR-100
SCD
000
Rideo
1.0
0:15
0.55
159
Epods
Epeci
Epads
Spoch
Figure 2: Training loss and test accuracy comparison on CIFAR-10 and CIFAR-100 datasets.
which shows that the lower the loss of the training set may not necessarily lead to the better the generalization ability of the model. In addition, from the accuracy of the test set, the convergence speed of the SGD family including SGDM and ADMETAS is generally slower than that of the Adam family (Adam, RAdam, Ranger, AdaBelief and ADMETAR), but the final convergence result of the SGD family better than the Adam family. However, our ADMETAR achieve more comparable performance the SGD family, while maintaining the advantage of the fast convergence of the Adam family. ADMETAR has the highest results on the test set in the early stage of optimization (< 80 epoch), which demonstrates that bidirectional looking improves both accuracy and speed, making ADMETAR efficient and effective optimizer implementation.
Compared ResNet-110, PyramidNet has more complicated structure and can achieve better results in these tasks. In cases where the model is strong enough, the selection of optimizer will not be the main factor for the final performance. As shown Table compared to Adam, RAdam and AdaBelief achieve just bit of improvement on CIFAR-10 task and even achieve worse results on CIFAR-100 100 task, which also verifies our above claims.
3.2 NATURAL LANGUAGE UNDERSTANDING
As general Al component, the general capability requirement for various tasks and various models a basic requirements for optimizers. We evaluate the adaptability of our ADMETA optimizer on the finetune training scenario with current popular pre-trained language models. Since the SGD family converges slowly on the finetune stage of the Transformer architecture, we only compare the various optimizers of the Adam family here. Specifically, we conduct experiments based on the pre-trained language model BERT (Devlin et 2018) on three natural language understanding tasks, GLUE benchmark WWang et al. 2018), machine reading comprehension (SQuAD v1.1 and v2.0 (Rajpurkar) et al. 2016)) and named entity recognition (NER-CoNILL (Sang & De Meulder] (003)). We report results for two model sizes, BERT hase and BERT lange to explore whether model size has an effect on the optimizer.
In Table 2, we report the results of on the development set of datasets of the GLUE benchmark, where Acc, MCC, SCC are abbreviations of accuracy, Matthews Correlation and Spearman Correlation Coefficient, respectively. First, under the BERT-base model, compared with the basic optimizer RAdam, ADMETAR achieves consistent improvement. The most significant improvement is obtained on RTE and CoLA, which indicates that our ADMETA optimizer exhibits greater stability for low- resource optimization. On the other seven datasets, some of them slightly improved. This is
7

---

Under review as a conference paper at ICLR 2023
Table 2: Development results on GLUE benchmark.
MNLI m/mm (Acc)
QQP (Fs)
QNLI
SST-2
CoLA (MCC)
STS-B (SCC)
MRPC (F1)
RTE
Model
Optim
Average
(Acc)
(Acc)
(Acc)
AdamW
83.85/84.08 08
87.72
90.74
93.23
60.32
89.11
90.85
67.51
82.92
Ranger
83,80/84.24
87.83
90.76
92.32
58.87
89.19
90.05
68.59
82.68
BERT
AdaBeliel
83,91/84.42
86.76
90.92
92.55
58.05
88,
90.38
67.87
82.42
RAdam
83,91/84.24
87.66
90.88
92.20
59.31
89.07
90.91
70.04
83.00
ADMETAR
83,90/84.53
87.91
91.14
93.35
62.07
89.62
91.47
71.48
83.87
AdamW
86.05/86.55
88.58
92.40
93.00
59.58
89.21
91.67
71.12
83.95
Ranger
86.53/86.58
88.58
92.39
93.46
63.81
89.73
92.04
72.56
84.89
BERT Lagg
AdaBellif
85.5 65.59/86.25
86.99
92.42
93.00
61.11
90.17
91.28
72.92
84. 19
RAdam
86.40/86.72
36
92.35
93.69
62.61
89.64
91.29
71.48
84.48
ADMETAR
86.21/86.54
88.54
92.63
93.69
64.12
89.92
92.10
73.65
85.11
Table 3: Results on SQuAD vl.1 and v2.0 development sets and NER-CONLLO3 test sets.
SQuAD v1.1
SQuAD v2.0
NER-CONLLL8
Model
Optim
EM
Fi
EM
F1
P
R
Fi
AdamW
80.87
88.39
72.63
75.99
94.65
95,24
94.94
Ranger
81.30
88.58
73.32
76.73
94.47
95.17
94.82
BERT
AdaRelief
80.63
88,10
72.97
76.25
93.79
94.60
94.19
RAdam
80.68
88.
73.21
76.49
94.61
95.42
95.01
ADMETAR
81.55
88.69
73.81
77.19
AdamW
83.31
90.39
76.67
80.02
94.77
95.73
95.24
Ranger
84.21
90.97
77.22
80.35
95.24
95.89
95.56
BERT Iege
AdaBelief
83.53
90.42
77.48
80.57
94.28
95.17
94,72
RAdam
84.17
90.90
77.39
80.72
94.80
95.64
95.22
ADMETAR
84.25
90.92
77.08
80.36
95.38
95.93
95.65
because most of the parameters of the model in the pre-training seaiiiing-finettniin fine-tuning paradigm have converged to certain extent in the pre-training stage, so the further advantage of the optimizer in finetune is not apparent. And when the model switched to larger BERT-large, most tasks receive performance gains, except for CoLA and RTE using Adam W optimizer. Due to the further increase in model parameters, the low-resource dataset is not enough to fine-tune the large model, will even reduce the model performance. But RAdam with rectified item, Ranger with bidirectional looking, and our ADMETAR handle the low-resource challenge well, continue to improve performance, and take advantage of large models. Our ADMETAR achieves the best results on these two low-resource datasets, demonstrating the effectiveness of our bidirectional looking approach.
In Table 3 we further report the results of machine reading comprehension and named entity recognition. ADMETAR achieved improvements at both model sizes in SQuAD v1. dataset, while similar improvements were achieved in SQuAD v2.0 with more complex models, illustrating that our optimizer is model-independent. Named entity recognition has reached a very accurate level with the help of pre-trained language models, and ADMETAR optimizer also brings performance improvements over such a strong baseline, showing that optimization is also a bottleneck that restricts further performance improvement in addition to model structure and data.
3.3 AUDIO CLASSIFICATION
Like images and natural language, speech one of the mainstream fields deep learning re- search. In speech processing, there are also large number of pre-trained large models, such as
Wav2vec (Schneider et al. 2019). To highlight the inppe-indeeendentt nature of the optimizer. we also conduct experiments on two typical tasks of audio classification, keyword spotting (SUPERB) (Yang et al. 2021) and language identification (Common Language) (Sinisetty et al., 2021). ). We employ Wav2vec 2.0 as the baseline model and report the results of each optimizer in Table In addition,
we also list the training time of each optimizer to evaluate the impact of the bidirectional looking mechanism the optimizer time overhead
ADMETAR shows better classification accuracy than AdamW, RAdam, Ranger and AdaBelief, which consistent with the experimental conclusions image and natural language tasks. Consistent
Notably, the reported training time only for rough comparison due to the influence of environments.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Results on speech keyword spotting and language identification tasks.
SUPERB
Common Language
Optim
Training
Training
Adam
98.26
10m444
79.45
8h27m333
AdaBelief
98.41
11m20s
80.29
8h28m25s
Ranger
98.35
11m50s
81.18
8h29m55s
RAdam
11m30s
80.35
8h28m388
ADMETAR
98.50
11m54s
81.57
8h30m155
Table 5: Ablation study on ADMETA optimizer.
Optim
CIFAR-10
A
Optim
CIFAR-10
RAdam
93.09
SGDM
93.68
ADMETAR
93.63
ADMETAS
94,12
-DEMA
93.24
-0.39
-DEMA
89.13
-4.99
-LB
92.29
-0.95
89.88
-4.24
-1.F
93. 14
-0.10
-IF
93.51
-0.61
-LB-LF
92.36
-0.88
-LB-LF
89.80
-4.32
ADMETAR w/ constant I.F
93.03
-0.60
ADMETAS w/ constant I.P
93.75
-0.37
results across image, natural language, and speech modalities verify the task- sisk-independence of our optimizer. Comparing the training time of ADMETAR with AdamW, RAdam, Ranger, and AdaBelief, our ADMETAR have different degrees of increase due to the additional computation and storage the optimization process. Ranger and our ADMETAR increased the time most, but it can still be regarded as slight compared to the overall training time. Therefore, it can be concluded that the bidirectional looking mechanism adopted by ADMETA optimizer will bring additional computational overhead and increase the training time, but compared with the overall training cost, it very small. ADMETA achieves better performance without increasing model parameters and training data, and does not have any impact on the inference time of the model, which achieves a better tradeoff.
4 ABLATION STUDY
We perform ablation study on various designs of ADMETA bidirectional looking in this section. -DEMA means removing the DEMA mechanism in bockwarddlooking and using the original EMA. -LB means complete removal backward-lookes
-LF means complete removal of forward-looking. -L.B-LF means remove bidirectional looking the same time. w/ constant LF means the original Lookahead mechanism the forward-looking The results are evaluated using the ResNet-1 10 model on the test set of CIFAR-10. According to the results shown in Table 5 it be found that the improvement of SGDM compared with SGD initially shows the advantage of backward-looking, And compared with Adam, RAdam reveals that the EMA with the rectified item backward-looking is more suitable for the training of the model than the original EMA. Our ADMETA (including ADMETAR and ADMETAS) achieved the best results. After removing DEMA and replacing dynamic lookahead with constant lookahead, respectively. the performance drops, indicating that both DEMA and dynamic asymptotic lookahead play an important role in stable optimization. After further removing the backward-lookinn the forward-lookien and the bidirectional looking, the results drop further, validating our argument that bidirectional looking is beneficial for optimization.
CONCLUSION
In this paper, we introduce a bidirectional looking optimizer framework, exploring the use of historical and future information for optimization. For backward-looking we introduce a DEMA scheme to replace the traditional EMA strategy, while for forward-looking, we propose a dynamic asymptotic lookahead strategy to replace the constant lookahead scheme. In this way, we propose ADMETA optimizer, and provide two implement versions, ADMETAR and ADMETAS, which are based on adaptive and non-adanaa adaptive momentum optimizers, RAdam and SGDM respectively. We verify the benefits of ADMETA with intuitive examinations and various experiments, showing the effectiveness of our proposed optimizer. Please refer to Appendix F for future work discussion.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ahmet Alacaoglu, Yura Malitsky, Panayotis Mertikopoulos, and Volkan Cevher. A new regret analysis for adam-type algorithms. In International conference on machine learning, pp. 202-210. PMLR, 2020.
Wangpeng An, Haoqian Wang, Qingyun Sun, Jun Xu, Qionghai Dai, and Lei Zhang. A pid controller the approach for stochastic optimization of deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. $522-8531 2018.
Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint arXiv:2002.09018, 2020.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33: 1449--1460, 2020.
Edwin F Beckenbach and Richard Bellman. Inequalities, volume 30. Springer Science & Business Media, 2012.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of a class of adam-type algorithms for non-convex optimization. arXiv preprint arXiv:1808.02944 2018.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810:v881004466 2018.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Hannes Fassold. Adafamily: A family of adam-like adaptive gradient methods. arXiv preprint arXiv:2203.01603, 2022.
Gabriel Goh. Why momentum really works. Distill, 2017. doi: 10.23915/disiill.oo URL
http://distill pub/2017 pttt::// 20000111mmmmmttur
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5927-5935, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get for free. arXiv preprint
2017.
Daniel Jiwoong Im, Michael Tao, and Kristin Branson. An empirical analysis of the optimization of deep network loss surfaces. arXiv preprint 1rXiv:1612.04001 2016.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to sgd. arXiv preprint arXiv: 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:14/2.6900 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Wenjie Li, Zhaoyang Zhang, Xinjiang Wang, and Ping Luo. Adax: Adaptive gradient descent with exponential long term memory. arXiv preprint arXiv:2004.09740. 2020.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint 1rXiv:1908.03266 2019.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:/608 009883, 2016.
10

---

Under review as a conference paper at ICLR 2023
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:171..5500,,22 2017.
James Lucas, Shengyang Sun, Richard Zemel, and Roger Grosse. Aggregated momentum: Stability through passive damping. arXiv preprint Frxiiv/804.000 2018.
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv:1902.09844, 2019.
Xuezhe Ma. Apollo: An adaptive parameter-wise diagonal quasi-newton method for nonconvex stochastic optimization. arXiv preprini arXiv:2009. 13586, 2020.
Brendan McMahan and Matthew Streeter. Adaptive bound optimization for online convex opti- mization. arXiv preprint arXiv: urXiv: 1002.4908, 2010.
Patrick G Mulloy. Smoothing data with faster moving averages. Stocks & Commodities, 12(1):11-19, 1994.
Yurii E Nesterov. A method for solving the convex programming problem with convergence rate o (1/k 2). In Dokl. akad. nauk Sssr, volume 269, pp- 543-547, 1983.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3884 2013.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr computa- fional mathematics and mathematical physics, 4(5):1-17, 1964.
Martin Popel and Ondřej Bojar. Training tips for the transformer model. arXiv preprint arXiv:1804.002447 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 106.:00000055550. 2016.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.092377 092377, 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Erik Sang and Fien De Meulder. Introduction the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050, 2003.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05886 2019.
Ganesh Sinisetty, Pavlo Ruban, Oleksandr Dymoy, and Mirco Ravanelli, Commonlanguage, June 2021. bttpstts:: /ttp::::///...og//o.ooooo 520..222e2ooo..033333
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. International conference on machine learning, pp. 1139-1147, PMLR, 2013.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop, coursera: Neural networks for machine learning. University of Toronto, Technical Report, 6, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint arXiv: 1804 07461, 2018.
11

---

Under review as a conference paper at ICLR 2023
Jianyu Wang, Vinayak Tantia, Nicolas Ballas, and Michael Rabbat. Lookahead converges to stationary points of smooth non-convex functions. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8604-8600 IEEE, 2020.
Yizhou Wang, Yue Kang, Can Qin, Huan Wang, Yi Xu, Yulun Zhang, and Yun Fu. Adapting stepsizes by momentumized gradients improves optimization and generalization. arXiv preprint vrXiv:2106.11514. 2021.
Boxi Weng, Jian Sun, Alireza Sadeghi, and Gang Wang. Adapid: An adaptive pid optimizer for training deep neural networks. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3943-3947. IEEE, 2022.
Less Wright. Ranger a synergistic optimizer. Rergger-Dee--eaannnn--ooo-..mmm
httpss//gittubb.omm//essssw
2019.
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2/05500051, 2021.
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10665-10673, 2021.
Matthew D) Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 12Xxv:1212.5701, 2012.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey Hinton. Lookahead optimizer: k steps forward, step back. Advances in neural information processing systems, 32, 2019.
Beitong Zhou, Jun Liu, Weigao Sun, Ruijuan Chen, Claire J Tomlin, and Ye Yuan. pbsgd: Powered stochastic gradient descent methods for accelerated non-convex optimization. IJCAI, pp. 3258- 3266, 2020.
Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C TTatikonda, Nicha Dvoreek, Xenophon Pa- pademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances in neural information processing systems, 33:18795-18806, 2020.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03), pp. 928-929--88 9389366 2003.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A RELATED WORK
As an important part of machine learning and deep learning, optimizer has received much attention in recent years. The optimizer plays a prominent role in the convergence speed and the convergence effect of the model. To seek good properties like fast convergence, good generalization and robustness, many algorithms have been put forward recently, and they can be divided into four families according to their characteristics and motivation.
SGD Family In this family, the optimizers adopt the method of update like
01 0 a;
where 8, denotes the parameter to be optimized at iteration step and my refers to some combination of past gradients (such as EMA), which can represented as (gt 92..... Original SGD (Robbins) & Monro 1951) directly minus the product of global learning rate and the gradient at each step. Despite of its simplicity, it is still widely used in many datasets. However, SGD blamed for its low convergence rate and high fluctuation, thus many methods have been proposed to accelerate the speed and smooth the update process. One efficient optimizer to tackle this issue is SGDM (Sutskever et al., 2013), which uses exponential moving average (EMA, also known as momentum) to replace the gradient with an exponential weight decay of past gradients. SGDM-Nesterov (Nesterov 1983) is a variant of SGDM which modifies the momentum by computing gradient based on the approximation of the next position and thus changing the descent direction. Experiments have shown that Nesterov momentum tends to achieve a higher speed and performance.
Adam Family The Adam family optimizers usually update parameters by
0: 0: amm//// V
where ie is the adaptive item and can be represented as f2(gî .2.....9)). Compared to SGD family, instead of using uniform learning rate, this kind of optimizer computes an individual learning rate for each parameter due to the effect of the denominator voe in the equation. is usually an cumension-reductionn approximation to the matrix which contains the information second order curvature, such as Fisher matrix (Pascanu & Bengio 2013).
Adadelta Zeiler (2012), Adagrad (Duchi al., 2011) and RMSprop (Telleman & Hinton 2012) are early optimizers in this family. A stand out generation is Adam (Kingma & Baj 2014) which combines the RMSprop with Adagrad. It has been widely used in a wide range datasets and works well even with sparse gradients. However, there are problems with Adam with respect to convergence and generalization, thus many methods have been proposed to make improvements
Based on the large variance in the early stage that may leads to a bad optimum, heuristic warmup (Vaswani al. [017]] Popel & Bojar 2018) and RAdam (Liu al. 2019) are proposed, of which the former starts with a small initial learning rate and the latter introduces rectified item. To fix the convergence error, Reddi et al./ (2019) proposed AMSGrad which requires the non-dccreasing property of the second momentum. In fact, this method can be interpolated into other Adam family algorithms to guarantee the convergence function, AdaBelief (Zhuang et al.) 2020) and AdaMomentum (Wang al., 2021) are proposed. More recently, there are still numerous studies devoted to improving Adam, such as AdaX (Li al., 2020) and AdaFamily (Fassold 2022). However, we notice that most researches put a solid emphasis on modifying the second momentum term, i.e., the adaptive item and ignores the possibility to make a relative overall change to the algorithms.
convex situations. Considering curvature of the loss
Stochastic Second-Order Family In the stochastic second-order optimizers, parameters are up- dated using second-order information related to Hessian matrix. The update process is typically written
0, = 0,- - 0r H me:
where is the Hessian matrix or approximation matrix to it. Ideally, they can achieve better results than the first order optimizers (like Adam family and SGD family), but their practicality is limited
13

---

Under review as a conference paper at ICLR 2023
Optimum
Sauuion
momentum)
with DEMA
Momentun 0.00
Momentum B- 0.80
Leaming rote o 3.0015
Figure 3: EMA vs. DEMA in SGD optimizer. Please refer to our online demo https:
goog le cww.ccoiiiii.ioo. htttm/view//ggg zer admeta for more comparison.
due to the large computational cost of the second order information, like the fisher hessian matrix. Some methods have been proposed using low-rank decomposition and approximating hessian diagonal to reduce the cost, like Apollo (Ma 2020), AdHessian (Yao et al. (021] and Shampoo (Anil
al. 2020
Other Optimizers There are some algorithms that are not convenient to be categorized into the above families and list some examples here. Motivated by PID controller, SGD-PID (An et al.. 2018) takes an analogy between gradient and the input error in a automatic control system. Analysis show that it reduce the overshoot problem in SGD and SGD variants. Furthermore, Weng et al. (002) applied PID into Adam and proposed AdaPID optimizer.
Lookahead (Zhang al. 2019) optimizer updates two sets of weight wherein "fast weights" function as a guider to search for the direction and "slow weights" follows the guide to achieve better optimization. Ranger (Wright 2019) optimizer further combines RAdam and Lookahead to get compound algorithm and shows a better convergence performance.
Discussion To show the advantage of bidirectional looking, we propose ADMETA optimizer. Specifically, it is based on the idea of considering backward-looking and forward-looking. wherein DEMA plays a important role in the former aspect and dynamic asymptotic forward-looking strategy serves for the latter aspect.
In practical use, we provide two versions, ADMETAS and ADMETAR, using the framework of ADMETA and based SGDM and RAdam respectively. Specifically, ADMETAS replace the traditionally used EMA in Backward-looking part SGDM with DEMA and add the forward-looking part which is derived from Lookahead optimizer. ADMETAR is based RAdam in the same way. The second order family is also introduced above because the framework of ADMETA A can also be applied in this family, and it is remained the future work.
B EMA vs. DEMA
To corroborate our analysis of EMA and DEMA, we compared the optimization process of EMA and DEMA on the SGD optimizer according to the practice of (Goh) 2017). Using the same learning rate and starting from the same starting point, the convergence process is shown in Figure 3 The decent surface in the figure is the convex quadratic, which is a useful model despite its simplicity, for comprises important structure, the "valleys", which is often studied an example in momentum- based optimizers. As demonstrated in Figure on the one hand, DEMA achieves faster speed than EMA, which can be easily seen by comparing the distance the optimal point at the same time; the other hand, DEMA achieves better convergence results than EMA as can be seen in the distance between the point of convergence and optimum.
14

---

Under review as a conference paper at ICLR 2023
C PROOF OF CONVERGENCE
In this section, following (Chen et al., 2018), (Alacaoglu et al., 2020) and (Reddi et al., 2019). we provide detailed proof' of convergence for ADMETAR and ADMETAS optimizers in convex and non-convex situations.
C.I CONVERGENCE ANALYSIS IN CONVEX AND NON-CONVEX OPTIMIZATION
Optimization problem For deterministic problems, the problem to be optimized is minger f(0), where denotes the loss function. For online optimization, the problem minger Σt= fe(0), where Jr. is the loss function of the model with the given parameters at the 1-th step.
The criteria for judging convergence in convex and non-convex cases are different. For con- vex optimization, following (Reddi et al.) 2019), the goal is to ensure R(T) o(T), i.e., 4om R(T)/T = 0. For non-convex optimization, following (Chen et al, 2018). the goal
is to ensure mineepyy E [(O.)] =o(T)
Theorem C.1. (Convergence of ADMETAR for convex optimization)
<ル 8 < 1,, i Let {0t} be the sequence obtained from ADMETAR, 0 S A,
and Us I 1141, 4E E [T]. Suppose E F, where J. C R and has bounded diameter D Doe. iee. 110, - 01 S Doo, VI E [T]. Assume f(0) is convex function and l[gelloe is bounded. Denote the optimal point 9. For Os generated, ADMETAR achieves the regret:
T R(T) E1s.(.)
Theorem C.2. (Convergence of ADMETAR for non-convex optimization) Under the ussumptions:
ff exits and is Lipschit:.comtiuuussi.. [=)( - Vf(y)]l < -||x -al--yll. vll. Vx, y; f is also lower
bounded.
step the algorithm can access bounded noisy gradient It. and the true gradient f is also bounded.
The noisy gradient is unbiased, and has independent noise, i.c. 9r V.((.)) + 8c [[ = 0 and
ô,1ō,, vi jj.
Assume minje(a(V1)) 2 <> and az /t, then for any T we have:
V S + (Q1 + Q2logT)
min
where Qi and Q2 are constants independent Tf
Theorem C.3. (Convergence of ADMETAS for convex optimization)
Let {0t} be the sequence obtained by ADMETAS,
S X.3
at V
VI [T]. Suppose
F. where F Rdd and has bounded diameter D D i.e. 10, 01100 < <D>o. Vt E [T]] Assume f(0) is a convex function and l19ulloo is bounded. Denote the optimal point as O. For 0, generated, ADMETAS achieves the regret:
T R(T)- -(00
Theorem C.4. (Convergence of ADMETAS for non-convex optimization) Under the assumptions:
f f exits and is Lipschitt-comtiiuuusi.. [z) - f((y)l L||x vll. Vx, y: f is also lower bounded.
At step t. the algorithum can access a bounded noisy gradient 9t. and the true gradient V is also bounded.
15

---

Under review as a conference paper at ICLR 2023
The noisy gradient is unbiased, and has independent noise, i.e. 9r vf((0)) 6τ. [[6] 0 and ررقيي, vi + j.
Assume 0₁ = a/Vi, then for any T we have:
min E | v (4' + QglogT)
where Qi and Q2 are constants independent of
Before formally proving the theorems, here list some remarks and preparations. Remark 1. For brevity, omit the rectified item of ADMETAR in the proof. However, it does not influence the proof since it can be integrated into the learning rate.
Remark 2. Following [Lo et al. 2019), the bias correction 1/1 Of of the first momentum my is omitted in the convergence of ADMETAR. Since 1/1 - 3f is bounded above and below 10, the order of the terms used is not affected, thus hardly affecting the proof.
Remark 3. The forward-looking part is not considered the proof. On the one hand, explanations and proof's of constant Lookahead have been given in (Zhang al. 2019) and (Wang al. 2020), which can imitated by our dynamic method. On the other hand, forward-looking part is exactly the interpolation of fast weights and slow weights each synchronization period, and the fast weights are updated by the given optimizer. Therefore, the convergence proof is equivalent to only proving convergence of fast weights.
Lemma C.5. if l19 be is bounded,i.e. T, then I,, h4 and m, are also bounded.
l19e|loo S Goo, Vt E [T), where Go is a constant independent
Proof. First of all, we prove |/// s (1 +1+ oo by induction:
when f =
| = 11/1/loo S Go
Suppose / h satisfies, then for = k 1
| | 1 S Tol + |19
$ (A + (l 1loot s (1+ A)G
Next, for |
|loello ||gg + plelloo I <||looll0 2l I [K (1 A)
Since ma is the moving average of h4 where i=1,....t we can get that it is also bounded following the proof of I.
In this way, we can redefine Go by enlarging and the bounded stochastic gradient assumption in the theorem is equivalent to assuming [|9: lo: /e [/ |/mel/> < G
Remark for non-convex optimization, the same way, the bounded noisy gradient assumption equivalent to II. ||4 ll. [elll, |mm]]l < Hl where H is a constant independent of T. This remark will be used in several places in the following proof.
Lemma C.6 (Generalized Hölder inequality, (Beckenbach & Bellman 2012)). For U, RT and positive P, q, r such that +
have
Σ 0,0 < ||>||
This is common mathematical inequality, so the proof is omitted here.
Lemma C.7 (nonexpansiveness property of arg min ||-I|- (cMahan & Streeter, 2010 For any #EF
Q E St,iie. Q is a Positive definite matrice and convex feasible set C Rd, suppose U1 = arg min 1/2 _ z1)ll and = arg min ZEF
1/2 (x 22) then we have / - u2) <
zEF
1111/2(21 =2)
16

---

Under review as a conference paper at ICLR 2023
Proof. First, claim that (u1 21, Quu u1)) 2 0 and (uz -22 Quu u2)) 2 (We only prove the former as the proofs are exactly the same). Otherwise, consider a small 8, we have U1 8(u2- u1 F
(un 5(u2 u1) 21, Q(1 ō(u2 u1) 21))
2 (u) 21. Q(ui - -21)) + 8 ( - u1. Qu2 - u1)) + 5(u) 21, Q(u2 - u1))
2
If there exists (ny 21. (u2 u1)) < 0, can be chosen so small that it satisfies 202
U1 Q(u2 u1)) + 8(u1 21. (u2 un)) which contradicts the definition of u1.
Using the above claim, we further have
(ui 21, Quu2 u1)) (u2 22, Q(u u1)) 2 0
(2 21, Q(u2 - u1)) 2 (u2 u1, Quu2 u1))
(18)
Also, observing the following
((u2 u1 - (z2 21), Q(( - u1) - (22 21))} 2 0
= (U2 u1, (222-21)) =1)) - (u2 U1 Q(u2 un )} + (22 - 21, Q(22 - 21))
(19)
Combining (18 and 19 we have the required result.
C.2 CONVERGENCE ANALYSIS OF ADMETAR FOR CONVEX OPTIMIZATION
Lemma C.8. consider
my = -1m4-1 + -1) Vt 2
it follows that
(14, O2 0) =(m+ 0-1 - O)
31
{m--1. 00 02-11
31
1
+
((me, Or 0) -mm--1,0011- 07).
B1
Proof. By definition of me h4
1-8, m
1
T-B.
81
m4-1. Thus, have
1
91
(hr, O: -0) 0)
31 {mc Oz 0)
(m
1
B1
9
B1
{me, 0g - 0)
(m--1, 0,-111 -0) -1 (me-1, - 00n--1,0-00000 -
1
B1
1
B1
1
((me, 02 0) (m--1, 0--1 -0) (mm--1,0-1 -0) 0)
31
B1
(m4-1, 02 - 01-1)-
1 B1
Lemma C.9 (Bound for
ΣT
aelle; 1/4
aell0,1/4mll m/1)). Under Assumption in Theorem 1, we have
(1- (1-31)av/14 + log7 T
Σ 0
1/4 mel1
Σ |
(11 -7)
B1)
Proof. First, bound ||0; 1/4 m. m2 1|_ From the definition of m, and ut, it follows that
m = (1 - -B) Σ
3 hy = (1-32) 3 nm
j=1
==1
17

---

Under review as a conference paper at ICLR 2023
Then we have
Σ j= (1 - -1888 h3.s) 2
)
2
|0, 1074/1m/
mill
S 1101 1/1 "//m/mlll mpll
1/2
Σ j=1 (1 - 32)
(1-32) "n3,
(1- 31)2 (1)
d
Ei 11 1193)
V1 B2
v V j=1 83 8//011 "13
(1 - (1)
VI B2
7 Σj= (32 M3s:1t)a)* $
(8/2 (/A)2331/A)a((-))) (/)())))))))
(E)- [).,1)) +10+) $12
V VEi=188 02 $2-1
(1 31) d
t
31 - 112.51
VI B2 1=1
(1 (1- 81)
d
3
(20)
V(1 - 32) 1=1 f=1
where the first inequality follows from the fact that 6///2 2 U/,4 generalized Hölder inequality for
1/2 the second one follows from the
3 1= le and the third one follows from the sum of geometric series and the assumption y way, we can bound D ake 1/,
y (3
by (8 and p q 4, r= 2,
8 < 1. In this
T
(1 )2)
d
T
Σ
1/4 mell x
V(1 (1 7)
i=1 t=1
j=1
(1 B1)
T
01
V(I - 82) 7)
=1j=11
(1 - 31) (1-
d Phgal
T
ag
V(I 82)(1 7)
i=1==
1 31
d
T
a
h3
V(1 31) 7)
(1 - 3:)av 1 + log T
h2
V(1 (12 7)
(1 31)ay 1 + log T
32) - 7)
where the first inequality follows from 20 The first equality is by changing order of summation.
The second inequality follows from the fact that ETS a, < - The third inequality is by
Schwar-Sc The last inequality is by using E/ S 1+ log
Theorem C.10. (Convergence of ADMETAR for convex optimization) Let {0c} be the se- quence obtained from ADMETAR, O < X, L.
// < 1. 1,as
a and
Vi
18

---

Under review as a conference paper at ICLR 2023
2 S vt-1, vt E [T.. Suppose I E F, where F C. R and has bounded diameter
Dæo i.e. 110, - 01| I Deo: Vt E [T]. Assume f(0) is a convex function and llgellee is bounded. Denote the optimal point 9. For O2 generated. ADMETAR achieves the regret:
T R(T) -1000)
Proof. Bound for Σ (m -))-
As r E F, we get
Oct1 10, (0, a, 1/2 m;)
min 10 (z - (0: 0, 1/2 m)))ll.
.E
Furthermore, II
F.vE (x) = x for all x E F. Using Lemma C.7 with = 0.+1 and u2 = 8, we have
the following:
| (0.+1 - 0)]1 = | (0t - 0000 1/2 m4 0)1l
| / (0: 0)12 + ažlls, 1/111/411 moll2 20y ame, 02 - 8)
(21)
we rearrange and divide both sides of 2T by 2a, to get
(m,, O2 - 8 -
1
1
1/1
(0: 0)1?
100
1/
10]+1-0112- (0c+1 - 0)] 2 (0.+1 11//1(0/+1 0)11
0t 11, 1/1 mmel?
mmll
20
20y
(0,
(0,
0)12
1
110e
1/1
201-1
20g
1/2
+
1
1
(0i, - 0,)²
"106 1/4 m
2
Org
a1-1
1=1
- 110e
(0, (0,
0)11
(0c+1 0)1
204-1
200
+
D2
d
1/2
6//2
+
at
100
1/1 mel2
(22)
2
at-1
2
i=1
where the last inequality is due to the fact that 0,,i 2 02 1,-1, ae
1 2 a
and the definition of D
Summing (22 over t = L.... T and using that 0g 0 yields
T
D2
d
T
m,, 02 - 0) 0) <
20T
T.
2
a
0
1/4
mel
1=1
Bound for Σt= (m--1, 0,1 -()-
T-1
(m-1, 00-1 -6))
1m1-1.81-1 01) Σ
(m,, G2 - 0141)
1=1
110- 1/4
melllle// /1 (Be41 8)1 -
m
1107
1/4
mmll
l|
66. / [11 ,, [1
(0,-au) (0.-ar0) 00,,0 112m1 - 11, -I1 5, (0.)
1/4
|111,, 1/4 mell
m
||,,, 1/4 m2 2 Ore
19

---

Under review as a conference paper at ICLR 2023
where the first inequality follows from Hölder inequality and the second inequality is due lemma
Bound for (mr, Or 0).
(mr. Or 0) 116; 1/4
my |1110)/ -
0)1
s arlli; 1/1 1/4 mr +
1
11 (0 0)1
4ar
D
d
s sarlee, 1/4 mrl? +
11/2
4CT
where the first inequality follows from Hölder inequality and the second inequality follows from Young's inequality. The last inequality is due the definition of Dxo-
After all these preparations. we obtain:
T
T
T
(1r, 02 0) = 1 B1
B1
(mT, Or 0--07
+
m 1, 0z)
-o)
+ Σ
m,, O2
0)
t=1
P
D2
1/2 T
+
T
au 06
1/4 //4mell?
+
D
T,
1/2
Bi
4or i=1
20T i=1
1=1
Or 00
1/4 m/mlll?
- D2 d
T
1/2
2 31
Σ
aell0; mell?
4ar B1) i=1
T,
2(1 - B1)
1=1
(2 - 1-311)D vT
+2+ B....aaaI I log T
1/2 TT
Σ |
+
4a -11)
21 32)
This proves that ΣT bias of EMA is small (Zuang et al. 2020), thus (L4) approaches E(g.) as step increases. Since h4 kgt + pTe, h2 is the same order as Ji when the time is long enough, thus have
==1 (h, 0 8) O( Suppose the optimizer runs for long time, the
T
01 0, - - -)
(23)
In addition, due to the convexity of f(.), we have
T
I
R(T) = 1(0)) fc(x)) s Σ
(g O2 0)
t=1
t=1
Combined with 23 we complete the proof.
C.3 CONVERGENCE ANALYSIS OF ADMETAR FOR NON-CONVEX OPTIMIZATION
Lemma C.11. Set 0o xı in Algorithm, and define z₁ as
B 3
2, = 02 +
(0, 0 0-1)), V1 11
(24)
Then the following holds true
3
aN
04-1
-adh/Vo
Zt41
m4-1
1
B
Vir
/
20

---

Under review as a conference paper at ICLR 2023
Proof. By the update of ADMETAR, we have
0u+1 - 0₁ -aym// 81 a0 + (1 B1)he)/ Vo. 0g
at U
= 3
(0: - -1--1) - -cs(1 - 11)144/1 /v
ap-1 v0
[0, - 01-11 + $1
at
V -1
- (0.-0:-1) - -00--33111110001 3//he/ Vo,
-
(t-1
Vor
(0, - 02-1) -1 31
at
at-1
m a1 - -1) Vor
(25)
Vūt
V
Since we also have
(1-1(11 0.+1 - 0, (1- 31)0 + 11(+0++1-111 1004+11800 -0,) -,,)-11 11-111111
Combined with 25 we have
(1 - 31) + 31 - -b)
=(1 - -1
+ 3i +1)00++ 02-1) - B1
at a--1
m4-1 - a0l1 - B/)be/ V
The
vi-1
Divide both sides by 1 - B1, we have
$1
oc+1 +
(0t+1 - 00)
1 B1
-0, +
31
(0. 0,-1)
81
a
01-1
Mi-1
/clle/Vo.. br.
1 B1
1
B1
Vt
v
02-1
Lemma C.12. Suppose that the conditions in Theorem
C.2
hold, then
E [ff(2+11) - f(21)] < T₁
(26)
1=1
where
(
Ti -E
V fZz) 1- 31
81
ai
a:-1
mi-1)
Vo:
/ 0;
T2 E
ai (V(((()))) /4/ /x/VVe
27
B1
04-1
T3 E
1 81
Vo:
vo--1
mi-1
T4 E
Σ L
/ V 01
Proof. By the Lipschitz smoothness of Vf,
f(z+1) s f(xe) + (z)( ( #411 - -z) -x)+
|| - =x122
2
Based (C.18 we have
21

---

Under review as a conference paper at ICLR 2023
E[f( - f(21) =E
f(++1) f((z)
1
< E
V f2()) 2641
22) +
21+1
2i|1
E
(z)),, 1-31
31
aj
a;-1
m²-1)
V0s
0;
E
ai (Vx)), /v//
L
|| - 20l1
1
| - 2il1
+ E
Tı + T2
+
2
2
i=1
Then, using inequality ||a + b112 v 2||a|1 + 2|16|2 and combined with lemma C.I1
E
|111 211
< Ta T4
Lemma C.13. In this part, we bound Tı, T2, T3
Proof.
Bound for T1
Ti - E
(V
B1
a₁
0i
mi-1)
1
31
V0;
v 0
F
[=()))]]
|m=
11
81
Vos
Vo-11
0:-1
<H²
3 E
a;-1
31
Vi
03
i=1
Bound for T3
d
Ta <LE
31
Σ
a1
0:-1
(m--1)3
31
V0g
v
/0--1
j=1
j
3
LHdE
d
02
0:-1
27
1- 31
V
1 j=1
Bound for T2
T2 - E
=-E
E -V())
(27)
22

---

Under review as a conference paper at ICLR 2023
The second term of (2 can be bounded
E
0 (V)(2))
f22)2 Vf((0)) + 2 2l Voll
L
31
2
Oi 1/ 0:-1
+
|lyhs 0ell
2
B1
1
31
2
E
04 04-1
0
|lat/// o/1l
2
1
B1
where the second inequality is due to [V] - v((0)) < L| - 0z1l. Then consider the first term of 27
0 V f(,)), hv/ (v/
=
(V (0.) 9/ vv
+ +nF Σ (V.(O).
1=1
Consider the term with x
E
03 (V (V.(()), 9///
E
0. (V)))))) (V110)
E
0; (V(V)(()), (/000)0)/////00
+ E
a (V (V.)),) vv)
(28)
For the second term in RHS of
28 we have
E
E
VF())) (,, Vo. a-11V( + E
Σ a: az (V
+
E
on (V
>
([(()),
(
an aa-
2H² F
Σ on (nn/V
(29)
23

---

Under review as a conference paper at ICLR 2023
where the last equation is because given 0g 0--1, E ( 8:-1] 0 and ||,||| 2H Further, we have
E
E
(0(( -a-
j=l
2 E
(V [[(00))]11(00))) - a
1=2 j=1
2 2
Σ
(as((V a.-1/ -1-1(V
(30)
Substitute
and
into 28
we then get
0 Vf(())) 9:/ vo)
d
<2HºE
(a./(V Di); -0.- -- + 2
(an/Voi))
0s (VS(O))) v(OO\/
(31)
Then, consider the term with μ. Suppose the optimizer runs for a long time, the bias of EMA is small (Zhuang et al. 2020), thus E(T,) approaches Egg.) as step increases. In other words, we can bound it the same way as the term with k. After these bounds, we finally get
L T3 <
B1
E 2
a4
ml
01
E
lach./
/ o/1
B1
2
+ 2(K + H)H
(as/(Ver))
am-1/
1=2 j=1
d
+ 2(x + W
(an (a1//oi)s
- (n +)E
Σ ai (F(())
1=1
Lemma C.14. Suppose the conditions in theorem C.2 holds, then we have
a
(VS())))
SE
C
+C2 Σ
0 1 m4-1
C3
at
1:-1
V
0
Voz
i=1
[-1
a-11
C4
+CB
Vo+
2=2
where C1, C2 Cg, C4 and Css independent of the step.
24

---

Under review as a conference paper at ICLR 2023
Proof. Combine lemma
C.I2
and lemma C.I3.
we get
E
(2(+1)
f21)
2
11
E
a
31
0
0
1=1
j=1
+
B1
LH²E E
Qn
ai-1
1
B1
v0;
v 01-1
5
+ E
Σ
T.
81
+ E
Qs-1
0:-1
E
la,h /oll?
2
+ + + 1HH
0i-1
Vog
2- j=1
+ 2(K E
Σ
(ai/ (an/Voi)s
(I
u)E
j=1
By merging similar terms in above inequality and noticing that K + p> 0, we get
E
0 (V)(),) VS(0.)/V
0z)
H2
2H
E
a2
a1-1
(1 (I) + p))
V0z
V
27
B1
LH E
a2
0:
1 B1
K μ
v
0;-1
2L
E
+
2
B1
E
Σ
mi
2(K + (d)
Ui
2(K 11)
1-
0
2H2 E
Σ
(a1/Vor)) 01)s
E(f(21) - f(2+1))
1=1
2
C
0
+ + Σ
a; 1-
+ C3
at
1t
1
01-1
v /
C4
Og 1
+Cs
(32)
Vog
Theorem C.15. (Convergence of ADMETAR for non-convex optimization)
Under the assumptions:
f f exits and is Lipschit:-comtiuuussi,, ||Vf [z) V f((y)l < 4||x - vll. Vx. y: f is also lower bounded.
At step t. the algorithum can access a bounded noisy gradient 9t. and the true gradient V is also bounded.
25

---

Under review as a conference paper at ICLR 2023
The noisy gradient is unbiased, and has independent noise, i.e. 9r v.((0)) + 6τ. [[6] 0 and ررردد, vi + j.
Assume minjeld (vi); 2 e> 0 and a// n/VI, then for any T we huve:
m E [ + (Q) Q2logT)
where Q1 and Q2 constants independent of
Proof. We bound non-constant terms in RHS of 32 which is given by
T
E
C
ag24
+22
+
0 1 -
C3
al
04-1
Vūz
0;=1
/
a6-1
+O4
+ +CB
/ i2-1
t=
Bound the term with C₁.
Note that minzeja] (v 01 2 minjeld (ht)s 2 <> O, thus we have
a
E
Vuz
<E
Σ
T
ache
ahu
a evi
2
|14,11
c
H²a
2
T
a²a²
(1 + log 7)
where the first inequality is due to (tt) 2 (ft-1);, and the last inequality due to E S
1 + log T
Bound the term with C2.
Apply the same proof above, we get
2
H²a² +
0 m
<
log
8x-1
2
Bound the term with C3-
T
d
T
Ct-1
Or
E
=
(Vir-1)
(Vot)s
01
QT
d
a1
da
=
E
<
(33)
(Vo1);
(VeT),
(v11)
j=1
where the first equality is due to(0,); 2 (0,-1); 1)3 and a, I a,-1, and the second equality is due to telescope sum.
26

---

Under review as a conference paper at ICLR 2023
Bound the term with Ca-
at-1
E
V 8
T-1
1-11
=F
10
V
Σ
a1
a:-1
Vūz
00-1
1=2
2
c²
where the first inequality is due to ((0s/ (o///Vi- - 00-1/V00-1// S 1//c.
Then have for ADMETAR,
T
T
1
agly
+
C2
Σ
Q3- 1
+ C3
a2
(34)
Vū
0=-1
t=1
1=2
1=2
T-1
27
Q4
at-1
+0+
Cs
(35)
V0
t=2
(1 + log T) + C2 H²a²
lo T) + Cs da
da²
<C1
+
+
+Cs
(36)
c²
c²
Furthermore, due to || S H, we have (0t), S H22, then we get
1
a/ e((v))) 2
H
Thus we have
T 1
VT
E
a (V(())
E
v
((0)))
>
min E (())
H
H
LE|T
(37)
Combine 366 and
37
have
min EE(T]
F. [[)))())
< VT
H
(C] C2) C2)
H²²
(1
(1 log T) + Cs da + +C da²
c x +03
(Q) + log
This completes the proof.
C.4 CONVERGENCE ANALYSIS ADMETAS FOR CONVEX OPTIMIZATION
Lemma C.16 (Bound for Σin 0n mm/l). Under Assumption in Theorem 3, we huve
T
Σ
| < 22dG22 VT
27

---

Under review as a conference paper at ICLR 2023
Proof. First, bound |mm]l
|mel s d| m.ll0 - dG
(38)
Now we can bound Di arllmell?
T at od
cellmell S dG
S
2od0 VT
Vi
Theorem C.17. (Convergence of ADMETAS for convex optimization)
Let {0t} be the sequence obtained by ADMETAS, Vt [T.. Suppose
S X. 1 at = vi'
F. where F C Bd and has bounded diameter Deo, i.e. 11, Olloo I Dxo, Vt E [T].. Assume f(0) is a convex function and 119a|l>o is bounded. Denote the optimal point as O. For Oe generated, ADMETAS achieves the regret:
T R(T) I(n(O)) - -((0)))(((0]] -(VT)
Proof. Bound for Et- (m,, 02 0). From the update process, get
|10+1 - 011 -100 - amm1l1 100 [ 2a, O2 0) a
thus have
T
T Σ
T at
m, 0e - 0)
(10. 011 10 - -112) + -
mm 1
20,
2
[=1
consider the left-hand side
(110; 0|1 - |10:41 0112)
204
100
0|
Σ
1
1
110, 011
| 01
201
20
20
20T
2=
dDD22
dD
dD²
+
201
204
201-1
200T
Finally,we get
Σ
{mg 8) <
dD2
+
Σ
T
0
||
201
2
i=1
Bound for Σtm (m 1, 0,11 -g))-
T
T-1
(m-1,00-
- 0,-10 -0)) =
Σ
(m, 0g 0
(m,, amm))
adl ael!mell
28

---

Under review as a conference paper at ICLR 2023
Bound for (mr, Or 0).
(mr, 8) S ar [rll 1 - 01/
4ar
dD
S Sarl mrll2 +
4or
where the first inequality follows from Young's inequality. Combine all these preparations, we obtain
(he O2 - 03
(IMT, OT - 0) -(mo, Oo 03) + (mo, 0o
B
T
B
+
(m,, O1 - 0) +
1 11.1-11 -ba)
1-
3
1=1
t=1
8
B
(mr,, Or
+
(m--1,
001 O,)
(m,, Os -
B
B
8
dD 00
+ Σ as
amllmell?
dD²
0't || 1
B
Aoer
200r
B
dDoc
20.3 B
+ 2
+
a
dG2
VT
1
3
AoeT
This proves that Σ (1e, Og 0) O(VT). Suppose the optimizer runs for a long time, the
bias of EMA is small (Zhuang al. 2020), thus E(4) approaches E(g.) as step increases. Since
h4 = kgt + plg, he is the same order as 9r when the time is long enough, thus we have
Σ
(g, 0, - O(VT)
(39)
In addition, due to the convexity of f(), we have
T
T
R(T)- = Ee(00 - Se(x) s Σ (g 02 0)
-
1=1
Combined with 39 we complete the proof.
C.5 CONVERGENCE ANALYSIS OF ADMETAS FOR NON-CONVEX OPTIMIZATION
Lemma C.18. Set fg & 02 Algorithm, and define 2, as
1 (0. 0,-1), V1 2, 0,
(40)
Then the following holds
B
Zt+1
(ar C4 04. (0.1))m-1 or.be
1 B
Proof. By the update rule of ADMETAS, we have
0++1 0, = -a = -as aa133mm1111 + (1- 3)hb]
3 at
(0, =
0--1) - -a(1- hh
04-1
(a- 1) -0c-00 0--1) (01-0:-1) - ar(1 B)he
-( 0--1)
ar-1
=((0, 9-11) -B(a 0 - B8)he
(41)
29

---

Under review as a conference paper at ICLR 2023
Since also have
- 01) (1 -3) Oc+1 0, = (3)+4 + +((00+1
Combined with we have
(1 3)0++1 3(0++1 06) =(1 -01-1)08- 3(0, 0(-1)
- -((a as (1-1)mm-1 - ar(1 - B)
Divide both sides by 3
B
Oct1 + 1- B (0c+1 - 01) =0, +
(0. 0--1)
1 3
3
1 (ax: at- mm
achh
Lemma C.19. Suppose that the conditions in Theorem C.4 hold, then
E -
-
Σ Ti
where
B1
T E
(2)); 1- 31
(a, a
T2 E
0i V [(+)), h4)
B (a: a4- 1
T3 E
T4 =
||as
Proof. By the Lipschitz smoothness of Vf.
=)+++ + L 11 - 24
5 & f(x)) + (V)(2) 441 2d)
a41
Based
C.18 we have
E[((a+1) - f(21) =E
Σ f(z+t1) -
<
V f22). 2+11 - 21) || 20l1?
8
=- E
(Z:(;;) 1- T- -asa--ii-- Ci- -i-i))m--)
E
a. (i(()(z)))))) hi)
+ E
| 2011
Then, using inequality ||a + b112 < 2 + 21bb and combined with lemma
C.18
|| |
1
< Ta + T4
30

---

Under review as a conference paper at ICLR 2023
Lemma C.20. In this part, bound T2, T3, T4
Proof. Bound for Ti
B
T E
14-1 (2)
ai a4-1
1 1-
B E
H
0; a4-1l 1|
5
B
1 a
1 3
where the second and last inequality is due the monotone decreasing property of oş Bound for T3
3
T3
LH
(a; 0;-1)²
1 1- 8
B
2a
L E
a0 -4-11
1
3
B
2a
LH
1-
where the monotone decreasing property of a; is also used Bound for T4
S 2
La + log
t
where the second inequality is due to Ei= Bound for T2
1
$ + logT
T3 = - F
V (V,))) Ad)
1==1
E
Σ -
(42)
The second term of (42 can be bounded
E
[V(2z) - VS(O)),)))
S
[Vf(2)
V=((,)112 + hl0
1
3
1
E
04
111
E
Maaht 12
2
2
12
B
1
a² H
2
1-
8
i=1
22
9
I
+1
(1 + logT)
2
3
31

---

Under review as a conference paper at ICLR 2023
where the second inequality is due to [2)) - v)((e)) L| - - Then, consider the first term of (22)
E
(V. h,)
E
V (0.)) Ag + miT
f())). vf(())) + +99
tE Σ a: (V.(0). +6.)
=(x+ M)
a: (V)()),)) v)((.))
The second and third equality holds for the follow reasons: on the one hand, 9: vf(0,)+ + in
which F( O, so according to (Chen et al. 2018), given 0,, F = 0: On the other hand, suppose the optimizer runs for a long time, the bias of EMA small (Zhuang et al. 2020), thus E.(,) approaches E(ge) step increases. Finally, can finally bound T2
2² T2 -
L
B
2
+1 1
(1+ logT) + (
[a(VF(())FTO))
t
2
1
1-3
Theorem C.21. (Convergence of ADMETAS in non-convex stochastic optimizution) Under the ussumptions:
f exits and is Lipschit--comiinoussie, [[V V s T||x- 1ilr-yll, -lls Vx, y J is also lower bounded.
step the algorithum can access bounded noisy gradient gt. and the true gradient f f is also bounded.
The noisy gradient is unbiased, and has independent noise, i.e. gr v.((0)) 8G, E[ 0 and
8; 1,,, vi # j.
And ar a/ Vt, then for any T have:
(V(())) s + (Q' QglogT)
min E
where Q2 and Q2 are constants independent of
Proof. We combine lemma C.18 of the objective. First, have
lemma C.19 and lemma C.20 to bound the overall expected descent
E Eif(241) f(a)) ST +
T3
T4
(43)
H22
B
2
L
1
(1 + log
(44)
3
2
1 3
(K + P)E
Σ
ai (V,, V(((.))
(45)
B
+20
I + H (1 + log
(46)
1-
32

---

Under review as a conference paper at ICLR 2023
Notice that
T v(())))
T
E
ac(V
2 E
|VS(0))1
2 V min E [[V)()))) EE|T]
(47)
Rearrange
43).
combined with 47 and notice that x + μr >> O, we have
min F. [1V]
E
Σ a
(VS))))
v(().))
(E(T]
VT
H²2212
a²
+ HH'La (1 (1+
VT
K K+
2
1 1-
1
B
B
- S(67)))
H²
+ 2a
LH + Elf(z1) -
K + p
1-
3
1-
VT (Q' + Q2logT)
where 2" is the optimal of f, i.e. 2" arg min f(2) This completes the proof.
C.6 CONVERGENCE ANALYSIS OF FORWARD-LOOKING
In this section, based Wang et al. (2020), further analysis forward-looking part to complete the convergence proof of ADMETA optimizer.
According (Zhang al. 2019) Lookahead is an algorithm that can be combined with any standard optimization method. The same is true for dynamic lookahead method in forward-lookinn part. What's more, optimizers with forward-looking is essentially processing with two loops as discussed in the main text. The fast weight is updated by optimizers, while the slow weight is updated by interpolating with fast weight every given period. other words, the slow weight updated passively. Therefore, though the slow weight is relevant optimizers, it is almost irrelevant to the selection of optimizers. For this reason, we only prove the convergence of forward-looking of ADMETAS, which can be casily extended the ADMETAR.
Remarks:(some preliminaries) Based on the design of the asymptotic dynamic weight 1)t of the forward-looking part, it can be concluded that when it runs for a long time, d is highly close to the set point, at which we can safely assume that nh is a constant and thus denote 17. In this way, the analysis of dynamic lookahead is the same as the case of static lookahead.
According to algorithm of ADMETA, the slow weight oi updates every k steps. We can assume that the slow weight is trained in sync with fast weight. For this purpose, all we should do is to stipulate 0rk+1 Φrk, where denotes the synchronization period, N and l < k.
Define 1/+ = n00 + (1 - 0,, then according to the update of 0, and dr. we have
Vt+1 = Vt m,am,
and on each period of synchronization, we have
Urk 0Tk = (1 - ( 0TK) =
Vrk ΦT (f0k *Tk)
Theorem C.22. (convergence of forward-lookiigg part)
Suppose [(() is L-smooth, i.e, [V)(x) v ( < L||a - ull, Va, y. The bias of noisy gradient is
bounded, iee, |ōcl S a, where 8; V (0.)) St. Then have that:
T
Σ E
< 0
T
1=0
33

---

Under review as a conference paper at ICLR 2023
Proof. Following the L-smooth property, we have
f(y++1) f(y)) -nax(( m²) + r |
(48)
Taking the expectation of both sides,
(WV pl)) (V(())))) + (V)e), l.)
(49)
Consider the term with K,
EKVF().). (V/() V.(O..
2 /1/ V s(0)))
- - - - -
2
(| |/ - - 2-- 10 - 0,1121
(50)
Suppose the optimizer runs for a long time, the bias of EMA is small enough, thus E(I,) approaches E(ge). For this reason, we can estimate the term with μ in (49) the same way as (50).
Based on the bounded bias gradient assumption and inequality that (a + b)2 - 2a² + 212, we have:
[ 24² 2x 2x2
||]l S 4(p² +)))) +422 + x3)0² (51)
Combined with
48
49
50
and
rearrange the inequality and take the expectation
B]f() SBI (w:)] par(u + x)
ma R)
18 V /
EEII V 1(0))
2
+ ( (1 - TT2 - + k)
El - 0,112] + 2(μ²² + x² x3 LE
2
+ 2(x² + xn)) a Lo2
Since the learning rate is decreasing zero, we can safely assume that after several iterations. 1 - -n,,L > 0. Then, summing over one outer loop
Elf - -[f((rr)
N0 + k)
B(l V [[(+++++)11))) 2(2² x² x>)km// Lo²
2
n + 4(12 + x² ))00 k-1
Elll (((++++)11
n1 - 22
(1- k)
Blll@re+t 3
(52)
2
1=0
34

---

Under review as a conference paper at ICLR 2023
Consider the last term of [22 we have
1-1
Billortt1 - 0 = 1l| - 0-k+i|91
a
11
m m ?
j=0
1-1
=2 E
grk+sll
+ +2
Σ
Tx+tsl* 1|
/=0
j=
<4x2a
(r+k+ V[((++++)))
4 +4
((++++))
12
+ 442 E
Σ (I+k+s ((((++))))+)
+4p2
f ++++))
<A(k² + 11000 + 14112 + &
V [
<A(k²² + 12) + +4(x² + x²) Ma²
V f(0, f(0+++))1
where the first equality using the property that 0.* 0rk Φr4+1. Summing from = to 1, we get,
0+x++++1 0-x+el11
<2(n2 22 + 1(0² )2)
B ((+++)))1
=2(n² + k )
B(| 11 Σ l
1=1+1
=2(n² + oz - 2 + x² )2 k
- jj - 1)
1=
( + (k 1) achieves its maximal value when j 0. Therefore, we have
Σ Bll 0-1+1l19
<2(k² + 2 - 1) + +222 + +33)a2uk(k 1)
Bill f0, (11))11
1=0
Here, we can finally bound the the last term of (2)
E B[((v((2))) [y+xx)
(0 + R)
k-1
Ell
[[ M
B
B ((
2
na(++1) + k)
BIIV//(++++)]
(53)
2
1=0
35

---

Under review as a conference paper at ICLR 2023
where
G= 2= +
+ (1²² (1) p)m(1 1)) - 1)
and
M (1( 4(1² x²2 mor
2
+ (k² )( n) - 1 oi - 1)a
When a is small enough, M is below zero, for which the second inequality of 53 holds. Summing from 0 t T - 1, we get
E[f(Ur))] - Ev)((
s - 7/1 + k)
T-1
T ((V+(+++))) 2(μ² + kn)
1)++
V
Lo
a
T=U
1=0
T=0
T-1
+ (n² + +)
L
Σ
Orkk
Following Wang et al. 2020), we first assume the learning rate a as a fixed constant, then rearrange the inequality above, we get
1 T-1k-1
2[5(2o) sent] s 4(p² + x² mo)oo
Tk ΣΣ
E V
maTk(u +
#
+ 2(x + =3)
Define T as Yk and set the learning rate to 1/VT
())( Sinsl
4(1² + xx² LD
M B[I
n (2 + k)
(u+ K)vT
+ 2(x² 12)
ok-
0
D ANALYSIS OF CONVERGENCE RATE
For convex situation, adopt the regret function estimate the convergence rate. And for convex situation, we adopt the minimum of the expectation of the squared gradient to estimate the convergence, which are corresponding
the proof of convergence since the process of the convergence proof is actually the process of finding the convergence rate.
From T notice that the convergence rates for all optimizers for convex case are of magnitude of O(L/VT) and for non-convex are of O(logT'/VT) which means essence, algorithms based on gradient decent follows a similar rate constraint. However, the convergence speed of different optimizers may attribute many other factors, such on the implementation. Therefore additional statistical experiments are needed for analysis, as we did in Table
E EXPERIMENTAL DETAILS
E.1 HYPERPARAMETER TUNING
For ADMETA optimizer, we first determined rough value range for learning rate and lambda with the toy model according to the visualization as Figure B While for other baseline optimizers, we refer to the recommendec/default hyperparameter settings in the original paper. In this way, get the rouge range of the hyperparameter in optimizers. Then, we search the hyperparameters in the adjacent interval, which is listed in the following three subsections.
36

---

Under review as a conference paper at ICLR 2023
Table 6: The comparison of convergence rate of several optimizers.
Case
Optim
Source
Convergence rough estimation)
SGD
Zinkevich 2003
Eis dr
Z0y
Convex
AMSGrad
Reddi ct al. 2019
D2 vT
"1
Dist 0T,/24 + 21] Ei= Σi=1
20
(1-3, 1- 11-22 E I1
ADMETAS
40x
Te ( + do vT
ADMETAR
(- 2 V7
21 TT + 2/02 12111 5)
(2ttt Eis lla
SGD
AMSGrad
Chen al
2018
(c. 1 log - C2 Co + c.)
Non-corvex
تيييع (1=²) شرب H La + log2")
ADMETAS
Hf 1 - za² (1=²) LH + {b} - s(=111)
ADMETAR
(K: + K2) شوكة (1 + log T) K3 _ेe + K₁ dيي + Ks)
Table 7: Optimizer hyperparameter settings on the CIFAR task.
SGD
SGDM
Adam
RAdam
Ranger LR
AdaBelict
ADMETAR
ADMETAS
Model
task
LR
LR
LR
LR
LR
LR
LR
B
ResNet-110
CIFAR-10 10
0,1
0.1
0.001
0.01
0.01
0.001
0.05
0.1
0,05
0.2
CIFAR-100 100
0.1
0.1
0.001
0.01
0.01
0.01
0.05
0.05
0.05
0.1
PyramidNet
CIFAR-10
0.1
0.1
0.001
0.01
0.01
0.001
0.01
0.1
0.05
0.4
CIFAR
0.001
0.01
0.05
0.1
E.2 IMAGE CLASSIFICATION
We conduct image classification experiments on CIFAR-I0 and CIFAR- datasets, which trained on a single NVIDIA RTX-3090 GPU. Typical architectures like ResNet-110 and PyramidNet employed as the baseline models. In the ResNet-1 10 architecture, there are 54 stacked identical 3 x 3 convolutional layers with 54 two-layer Residual Units (He et al.) 2016).
While in the PyramidNet architecture, there are 110 layers with widening factor of 48 (Han et al. 2017). We set the training batch size to 128 and the validation batch size to 256. Both model is trained with 160 epochs. Milestone schedule is adopted as the learning rate decay strategy, with learning rate decaying at the end of 80-th and 120-th epochs by O.
We report the hyperparameters tuning for our proposed ADMETA and other optimizers for reproduction of our experiments. For all optimizers, the weight decay is fixed as le - 4. The searching scheme of hyperparameter settings for each optimizer concluded as follows:
For SGD and SGDM, the momentum is fixed as 0.9, and the best-performing learning rate is searched from (0.01, 0.05, 0.1) and recommended values in original paper. For our AD- METAS, the À is set to fixed 0.9 and we search the best ress-perfornnn 3 from {0.1, 0.2, 0.440 and learning rate from {0.01, 0.05, 0.1}.
For all adaptive learning rate optimizers, hyperparameters 31, 32 and are set to B1 = 0.9, 32 0.999 and le-9 respectively. For Adam, RAdam and AdaBelief optimizer, the learning rate is searched from 1. 0.01, 0.001). For Ranger, and k are set to 1) = 0.5 and k = 6 according to Wright (2019). The learning rate is searched from {0.1, 0.01, 0.001}. And for our ADMETAR, the setting of k is the same as Ranger, and we search À from {0.05, 0.1, 0.2, 0.3, 0.4) and learning rate from 0.05, 0.01).
The resulting hyperparameters reported in the paper are shown in Table where LR is the abbrevia- tion of learning rate.
37

---

Under review as a conference paper at ICLR 2023
Table Optimizer hyperparameter settings on the GLUE benchmark.
MNLI
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
Model
Optim
IR
X
IR
A
IR
IR
A
I.
A
IR
A
IR
A
ER
X
Adam
2e:5
3e.5
3e
2e.5
Se-5
Se.5
4e-5
6e-5
RAdam
2e-5
2e.5
6e-5
4e.5
le4
4e-4
1.5e-
5e-4
BERT Saue
Ranger
5o-5
5c-5
le4
8c-5
20-4
5o-4
42-4
1c-3
AdaBelief
56-4
5c-4
5e-4
Be-4
4e-4
6c-4
5e-4
6c-4
ADMETAR
1.5044
0.08
le-4
0.36
0.03
7e-4
0.02
1.2c-3
AdamW
2e.5
2e.5
2e.5
2e.5
6e:5
5e-5
4e-5
2e-5
RAdam
26-5
2e-5
56-5
4e-5
lc-4
20-4
80-5
5e-4
BERT
Ranger
56-5
5o-3
5o-5
6c-5
6o-5
5c4
So4
5c-4
AdaBelief
4e-4
ADMETAR
1.5e
0.08
8e-5
0.2
8
0.03
9e.5
0.3
7e.4
0.02
le.3
0.03
6e
0.08
Se
0.1
Table 9: Hyperparameter settings of SQuAD v1.1 and v2.0 development sets.
SQuAD v1.1
SQuAD v2.0
NER-CONLLO3
Model
Optim
LR
LR
AdamW
5e-5
5c-5
6c-5
RAdam
le-4
5e-5
5c-5
BERT base
Ranger
le-4
8e-5
le-4
AdaBelief
le-3
8e-4
5e-4
ADMETAR
4e-4
0.05
3e-4
0.2
2e-4
0.3
AdamW
2e-5
5e-5
2e-5
RAdam
6e-5
Se-5
3e-5
BERT large
Ranger
le-4
8e-5
5e-5
AdaBelief'
8e-4
8e-4
4e-4
ADMETAR
4e-4
0.05
3e-4
0.2
1.5e-4
0.2
E.3 NATURAL LANGUAGE UNDERSTANDING
In the NLU experiments, we employ pre-trained language model BERT (Devlin et 2018 as our backbone. There are two model sizes for BERT: BERT and BERT large. where the base model size has 12 Transformer layers with 768 hidden size, 12 self-attention heads and 110 model parameters and the large model size has 24 Transformer layers with 1024 hidden size, 16 self-attention heads and 340M parameters.
In natural language understanding, we perform experiments on three modeling types of tasks: text classification, machine reading comprehension and token classification. The text classification uses the GLUE benchmark the evaluation data set, the machine reading comprehension uses SQuAD v1.1 and v2.0, and the token classification uses the
named entity recognition data set.
We train the eight tasks in GLUE benchmark for 3 epochs on single NVIDIA RTX-3090 GPU, except for MRPC, which trained for epochs due to its relatively small data size. The maximum sequence length is set 128 and the training batch size is set to 32. SQuAD v1.1 and SQuAD v2.0 are trained for 2 epochs with two GPUs. The maximum sequence length set to 384 and the training batch size per device set to 12. And COR-CONLLO03 trained for for 3 epochs single GPU. The training batch size per device is set to 8.
Because of the pre-training www.ttaiinnnniin-ttmme fine-tuning paradigm, we only employ the adaptive learning rate optimizer. We set 31, 32. and weight decay of these optimizers to 0.9, 0.999, le-8 and 0.0 respectively. n and are 0.5 and in the Ranger optimizer and ADMETA uses the same value of & as Ranger. We perform hyperparameter tuning on the learning rate and X, and the resulting hyperparameters reported the paper are shown Table and
E.4 AUDIO CLASSIFICATION
Based on Wav2vec (Schneider et al. 2019), the Wav2vec 2.0 (Baevski et al., (020) is a framework for seff-supervised learning of speech representations which is composed of 3 modules: feature
38

---

Under review as a conference paper at ICLR 2023
Table 10: Hyperparameter settings of SUPERB and Common Language.
SUPERB
Common Language
Optim
LR
LR
AdamW
3c-5
3e-4
AdaBelicf
8c-4
2c-3
Ranger
3e-4
Se-4
RAdam
8e-5
5e-4
ADMETAR
Se-4
0.05
2e-3
0.2
encoder, contextualized representations and quantization module. In the feature encoder, there are 7 blocks with temporal convolutions that have 512 channels for each block and a relative positional embeddings of the convolutional layer modeling has kernel size of 128 and 16 groups.
Among the configurations of Wav2vec 2.0, we choose Wav2vec 2.0 model, which has 12 Trans- former blocks, 95M parameters and 8 attention heads, with model dimension of 768 and inner dimension (FFN) of 3072. We finetune Wav2vec 2.0base for keyword spotting and language identifica- tion on SUPERB dataset (Yang et al., (202T) and Common Language (Sinisetty et al. 202I) dataset respectively. The dataset size of keyword spotting is smaller than that of language identification, so we use a single NVIDIA RTX-3390 GPU for training on the SUPERB dataset, and use four GPUs for parallel training on the Common Language dataset. The keyword spotting model is trained for 5 epochs with training batch size 32 and language identification model for 10 epochs with training batch size 8 per device.
Due to the same reason as in NLU experiments, i.e. the pre-training- -inniininnnn---iingggg paradigm, we only employ adaptive learning rate optimizers here. For all optimizers chosen, fix 81 0.9, 82 0.999, E = le 8 and set weight decay to 0.0. The learning rate is searched from (5e-5, 8c-5, lc-4, 3e-4, 5e-4, 8c-4), and for ADMETAR, À is searched from {0.05, 0.1 0.2}. The resulting hyperparameters reported in the paper are shown in Table 10
F FUTURE WORK
In the future work, for backward-looking part, though DEMA provides a more flexible way to deal with past gradients, it is still unable intelligently judge the value of certain historical gradient information, such as discarding some obviously unreasonable gradients caused by noise. A better optimizer may have the ability to forget these wrong information and take advantage of what works, just working like human brains. For forward-looking part, our method takes the constant coefficient into dynamic one. It is kind of like milestone scheme of learning rate decay strategies some extent. However, several experiments (Huang et al.) [017]] Ma 2020) have shown that cosine strategy (Loshchilov & Hutter 2016) works better. Therefore, we will follow the cosine scheme and propose a new forward- llokknd strategy that may work even better.
G PERFORMANCE OF SGDM AND ADMETAS ON FINETUNE SETTING
In this section, we test the performance of SGDM and ADMETAS on fintune setting and the results shown in Table II For keyword spotting (SUPERB) (Yang et al. 202T) task, train the models for epochs and use Wav2vec (Schneider et al. 2019) as the baseline model. And for CIFAR-10 (Krizhevsky al., 2009) task, we train the model for 40 epochs from the checkpoint already trained with Adam using learning rate of 0.001 for 160 epochs. The baseline model of CI ResNet- 10 (He et al.) 2016) with deep CNN architecture. We report the results of best hyperparameter settings for SGD and ADMETAS via grid searching.
we notice that SUPERB task, compared to adaptive learning rate methods, SGDM achieves worse results in SUPERB task, but not by much, which shows that SGDM can also be used in finetune setting. While ADMETAS can achieve better result than any other learning rate methods used in experiment, demonstrating the advantage of our approach. This phenomenon contradicts the mainstream view that SGD family not suitable for finetune task. While for CIFAR-10 task, SGDM and ADMETAS both improve the performance compared the start point. However, they
From Table I1
39

---

Under review as a conference paper at ICLR 2023
Table 11: Performance of SGDM and ADMETAS on finetune setting.
Optimizer SUPERB
CIFAR-10
SGDM
98.25 98.54
91.71
ADMETAS
91.87
Table 12: Performance of SGD family optimizers in CIFAR task.
Optim
CIFAR-10
CIFAR-100
ADMETAS
94.12
73.74
SGDM SGDM (I==0.5)
93.68
72.07
93.65
73.48
both obviously worse than the performance of training the task from scratch using SGDM and ADMETAS respectively, which shows that pre-training is a very strong approach that makes the model achieve good state.
The reason why ADMETAS performs better than SGDM in finetune setting may lie in two aspects. On the one hand, DEMA scheme the backward- looking part reduces the overshoot problem that may do harm especially near convergence. On the other hand, the forward-looking part improves the stability of the training process.
H INFLUENCE OF DIFFERENT LEARNING RATES IN SGD FAMILY OPTIMIZERS
Since the learning rate of 0.5 for SGDM a recommended value Han al. (017) but not in (He et al. 2016). to alleviate the influence of different learning rates, we also try the performance of SGDM with learning rate of 0.5 in the ResNet- network and the results listed in Table [2
The results show that choosing a large learning rate for SGDM may increase the performance, as shown that when setting learning rate to 0.5 instead of 0.1, the recommended value in ResNet-1 10. However, this is not always true since the performance on CIFAR-10 when using the learning rate of 0.5 does not get prompted.
40


Reviews:
[{"review_id": "vRRrvZN_Pzn", "reviewer": "Reviewer_RS9p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors found that using a variant of DEMA to replace the EMA used in past optimizers and using a dynamic lookahead optimizer can achieve a better result. Motivated by that, the author proposed a novel optimizer framework Admeta, and implement it on SGD and Radam. Empirical results demonstrate the superior performance of Admeta compared to other optimizers.", "summary_of_the_review": "This work has introduced a new optimizer with modifications on two parts. Despite some deficiencies in theory, the paper is well written.", "strength_and_weaknesses": "The strengths of the paper include an interesting framework to analyze the optimizer with modifications to each part, and a nice visalization figure show the comparison between DEMA and EMA. In my views, the limitations of the paper are that some results are not so good, especially in some relatively large models or datasets, such as MNLI with large BERT model. (But it should not be a big problem for the optimizer.) \nMy detailed comments are listed as below.\n\n1) Theorems 1-4 provide convergence proof on AdmetaR and AdmetaS, the two implementation versions of Admeta based on RAdam and SGD respectively. Before the main proof in the appendix, the authors make many assumptions and omit many things and argue that this argument can be attended to these cases. For example, the authors omit the forward-looking part in the convergence proof.  Are these assumptions and neglections reasonable? More discussion should be put in this part.\n\n2) As in some related works, the author should do some convergence rate analysis of AdmetaR in the convex and nonconvex settings to demonstrate its high convergence speed?\n\n3) The authors should introduce the hyperparameters tuning with more details since it is very important for the application of this optimizer in other tasks.\n\n4) In the ablation study part, I notice that AdmetaS – DEMA achieves worse result than SGDM, which may indicate that DEMA is not always a good method compared to EMA unless combined with a dynamic method. What may be the reason for this? The author needs to explain more.\n\n5) Separating optimizers into forward-looking and backward-looking parts is not a novel view. In fact, it is already used in Ranger ([1]). What is the difference of Admeta compared to Ranger optimizer. \n\n\n6) I notice that authors use SGD with Nesterov momentum in the experiment. It would be more interesting to see Nadam, Adam with Nesterov momentum in the comparisons. How about incorporating Nesterov’s approach in DEMA?\n\n[1] Less Wright. Ranger - a synergistic optimizer. https://github.com/lessw2020/ Ranger-Deep-Learning-Optimizer, 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RLuZj7QYpI", "reviewer": "Reviewer_kSFP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article shows that exponential moving averages have several weaknesses, namely momentum and Lookahead optimizers. The authors then propose modifications corresponding to these weaknesses and propose a framework, Admeta, to analyze optimizers. Theoretical results are provided to show the convergence and the reasons for such modifications. Empirical results on various tasks are provided to demonstrate that the proposed method outperforms the baseline optimizer and the currently proposed optimizer.\n\nIn the implementation, the authors propose new variants of Adam and SGD using the DEMA method and a dynamic look-ahead strategy to avoid the potential negative effects of traditionally used EMA and look-ahead optimizers. The paper also provides evidence that both implementations converge in the convex and non-convex cases.", "summary_of_the_review": "The paper is well written and easy to read. To clearly explain these methods, the authors combine text, formulas, and figures. For this, I thank the author for his hard work. I also believe that the Admeta framework provides an impressive perspective on the future work of the optimizer. Experiments demonstrate its good results in many tasks. All in all, it's an interesting piece of work that I'm sure would be of interest to researchers in related fields.\n\n[1] AdaBelief optimizer: adjust step size based on observed gradients", "strength_and_weaknesses": "Strength:\n1. The paper provides a point of view that is often overlooked by researchers.\n2. This paper provides experimental results on various tasks and datasets to demonstrate the advantages of the proposed method.\n3. The paper is easy to read.\n\nWeaknesses:\n\n1. The proof in the paper, although valid, is not sufficient to demonstrate the advantages of the proposed method, as it only guarantees convergence, which is the most basic requirement of an optimizer in deep learning.\n\n2. The paper can be improved by adding more theoretical explanations. What's more, the paper could add more and larger datasets.\n\n3. The improvements of the proposed method on some tasks are small in the experiments, so error bars are needed to verify that these improvements are not due to randomness. However, this paper does not provide error bars for the empirical results.\n\n4. AdaBelief's report in CIFAR seems bad. However, in the original paper by Adabelief ([1]), it achieves state-of-the-art results, albeit with a different backbone model. I don't know if there is a problem with the parameter settings of AdaBelief.\n\n5. SGDM in PyramidNet tried a learning rate of 0.5. Why not try it in ResNet110?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RK0Opiw_Gn1", "reviewer": "Reviewer_merB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a framework for stochastic optimizers named Admeta. Based on SGD and R-Adam, the authors provide two implementations, AdmetaS and AdmetaR. \n\nSpecifically, the author contributes a new gradient decent algorithm framework in the style of SGD and Adam optimizer. First, the author identifies the problem of the commonly used EMA method: the increasing lag time and overshoot problem. Then the author identifies the problem of the Lookahead optimizer: a slow stepsize around the early stage of training. The paper is well written.\n", "summary_of_the_review": "I believe that the paper has a lot of potentials but requires some additional discussion. My main concern lies in the results and innovation of the proposed optimizer. In general, this paper is worth publishing if concerns can be well addressed in the rebuttal stage.", "strength_and_weaknesses": "Pros:\n- The authors identify the problem of the traditional used EMA and Lookahead method and give improvements to modify them.\n- The introduction of the method is straightforward and easy to follow and the motivation is very interesting which comes from stock markets.\n- It is simple but effective. I am positively surprised by the results achieved in some tasks brought by the modifications. \n\nConcerns:\n- I notice that in the CIFAR-100 task trained on PyramidNet, the results of R-Adam, Ranger, and Adabelief are even worse than AdamW. It is very interesting. What hyper-parameters do you use? More discussion of this phenomenon is welcomed.\n- Can you explain why SGD-family achieves awful results in Finetune task? Additional experiments are needed to explore this point. How will AdmetaS perform in Finetune tasks? Will all SGD family optimizers behave like this? What if the model chooses the Deep RNN/CNN architecture instead of the Deep Transformer architecture? \n- The authors simply changed EMA to a DEMA variant and turn lookahead into a dynamic one. I think innovation is a bit incremental.\n- There are some problems in the organization of the paper. Many important parts have been placed in the appendix, which increases the difficulty for readers.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0y6hfWHTPzD", "reviewer": "Reviewer_dVGC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors give a clear introduction to current optimizers in deep neural networks and propose a new framework for improvement, which divides the optimizer into two parts, forward-looking and backward-looking. \n\nTo replace the traditional EMA method used in backward-looking part, a variant of double exponential moving average (DEMA) is used. The authors demonstrate its advantage over EMA by analyzing the structure of the formula and attaching a comparison figure, which visually presents the convergence trend in the appendix.\n\nAnd to further improve the Lookahead optimizer, an asymptotic weight is used, which achieves faster training than constant lookahead around the early stage and maintains the advantage of better convergence around the final stage. The authors also provide two alternatives to compute the asymptotic weight.\n", "summary_of_the_review": "The paper proposes a novel framework for optimizer. It seems that this work is quite original and the experimental results are relatively good. The main weaknesses are that some points are not made clear and extended experiments are needed to demonstrate the effectiveness of the proposed method. In general, the paper is well-written and comprehensible, and the idea is impressive.", "strength_and_weaknesses": "Strengths:\n\nThe ideas are reasonable, and this work gives a novel view in the optimizer field.\n\nThe authors conduct several experiments in Image Classification, Natural Language Understanding and Audio Classification, and experiments show the effectiveness of the proposed method. For all but a few tasks, this method shows advantages over other recent optimizers.\n\nThis paper follows a good logic and is easy to understand.\n\nConcerns:\n\nOn the other hand, I also have some concerns and questions toward this paper:\n\n (1) The theoretical explanation for the advantage of DEMA is insufficient. For example, the authors mention that the past gradients in EMA follow a fixed proportionality which is incompatible with actual optimization. Why?\n\n (2) In Image Classification, why the improvement for ResNet-110 is larger than PyramidNet?\n\n (3) The authors introduce a bias term in the DEMA method, but omit it in the rest of the paper and point out that this does not make difference. Why?\n\n(4) The choice of $\\mu$ and $\\kappa$ seems strange. What is the reason for constructing such a function? \n\n(5) How do you choose the hyper-parameter of the Admeta optimizer? It seems that the hyper-parameters chosen for each task are different.  \n\n (6) There are some deficiencies in the theoretical analysis, although I understand the space is limited. If more theoretical analysis is added, it will be more convincing and interesting. \n\n (7) Other works on optimizers like [1] tend to give an interval on experiment results in the appendix. I suggest that the results need to be tested for statistical significance.\n\n(8) Although training on large datasets is not necessary for the optimizer, if Admeta can be used in the large-scale dataset ImageNet or BERT pre training stage, the advantages of the proposed optimizer will be better demonstrated.\n\n(9) Though the related work is organized well and summarized into SGD Family, Adam Family, Stochastic Second Order Family and Other Optimizers, the authors should clearly explain the relation between the proposed method and related works.\n\n[1] Apollo: An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject