A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range
Authors: Guoqiang Zhang
Kenta Niwa
W. Bastiaan Kleijn
Venue: Accepted by TMLR ()

Abstract: We make contributions towards improving adaptive-optimizer performance. Our improvements are based on suppression of the range of adaptive stepsizes in the AdaBelief optimizer. Firstly,  we show that the particular placement of the parameter $\epsilon$ within the update expressions of AdaBelief reduces the range of the adaptive stepsizes, making AdaBelief closer to SGD with momentum.  Secondly, we extend AdaBelief by further suppressing the range of the adaptive stepsizes. To achieve the above goal, we perform mutual layerwise vector projections between the gradient $\boldsymbol{g}_t$ and its first momentum $\boldsymbol{m}_t$ before using them to estimate the second momentum.   The new optimization method is referred to as \emph{Aida}. Thirdly, extensive experimental results show that Aida outperforms nine optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image classification over CIAF10 and CIFAR100 while matching the best performance of the nine methods when training WGAN-GP models for image generation tasks. Furthermore, Aida produces higher validation accuracies than AdaBelief for training ResNet18 over ImageNet.

Published in Transactions on Machine Learning Research (08/2023)
A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range
Guoqiang Zhang
g.z.thang @ exeter.ac.uk
Department of Computer Science University of Exeter, UK
Kenta Niwa
kenta.niver @ ntt.com
NTT Communication Science Laboratories, Japan
W. Bastiaan Kleijn
bastiaan kleijn @ vww.ac..rz
School of Engineering and Computer Science
Victoria University of Wellington, New Zealand
Reviewed on OpenReview: http:////.rreeeeeee/////////////
Abstract
We make contributions towards improving cepptive-optimizee performance. Our improvements are based on suppression of the range of adaptive stepsizes in the AdaBelicf optimizer. Firstly, we show that the particular placement of the parameter within the update expressions of AdaBelief reduces the range of the adaptive stepsizes, making AdaBelief closer to SGD with momentum. Secondly, we extend AdaBelief by further suppressing the range of the adaptive stepsizes. To achieve the above goal, we perform mutual layerwise vector projections between the gradient Jr and its first momentum m, before using them to estimate the second momentum. The new optimization method is referred as Aida. Thirdly, extensive experimental results show that Aida outperforms nine optimizers when training transformers and LSTMs for NLP, and VGG and ResNet for image classification over CIAF10 and CIFAR 100 while matching the best performance of the nine methods when training WGAN-GP models for image generation tasks. Furthermore, Aida produces higher validation accuracies than AdaBelief for training ResNet18 over ImageNet. Our implementation is available at
htwp:ehuuu...omuuqqquuuuaaaaaaaa zhang x/Aida Opt imiizrir
Introduction
In the last decade, stochastic gradient descent (SGD) and its variants have been widely applied in deep learning LeCun ct al. (2015): Vaswani et al. (2017;; ; et al. (0066); Chen et al. (2020) due to their simplicity and effectiveness. In the
literature, SGD with momentum Sutskever et al. 2013 : Polyak (1964)) dominates over other optimizers for image classification tasks He et al. 2015)); Wilson et al. (2017). 1. Suppose the objective function f(0) : E Rd DNN
model is differentiable. Its update expression for minimising f() can be represented as
{
m. 3, +
JSG with momentum)
(1)
0: = 0, m,m,
where 9. vf(0--1) is the gradient at O., and 7): is the common stepsize for all the coordinates of 0. In practice, the above method is often combined with certain step-size scheduling method for 7)t when training DNNs.
To bring flexibility to SGD with momentum, an active research trend is to introduce elementwise adaptive stepsizes for all the coordinates of m: in (T). referred to as adaptive optimization Duchi et al. (20IT): Tieleman Hinton (2012); Kingma & Bal (2017). In the literature, Adam Kingma Ba (2017) is probably the most popular adaptive optimization method (e.g., Vaswani al. (017): Liu al. (2021): Zhang et (2019;; ; Devlin & Toutanoval (2018).. Its update
expression be written

---

Published in Transactions on Machine Learning Research (08/2023)
m, = -- + +11 (1- B1)g. V. = 82 + (1 - 321g;
AAamm
0, = 0,1 - 1/1 1-81 v vo.(01 -2)
where 9, V[(0,-1), 0 < 31.82- < 1, and 6 > 0. The two vector operations (-)2 and ./- are performed in
elementwise manner. The two exponential moving averages (EMAs) m, and is alternatively referred to the first and second momentum. The two quantities 1 - 34 and 1 3 are introduced to compensate for the estimation bias in m, and v,, respectively. T) is the common stepsize while 1/( 0/ /1((1 35) + E) E R represents the elementwise adaptive stepsizes.
Due to the great success of Adam in training DNNs, various extensions of Adam have been proposed, including AdamW [oshchilov & Hutter (2019), NAdam Dozai (2016), Yogi Zaheer al. (2018). MSVAG Balles & Hennig al. (2020). Is worth noting that in Liu et al. (2019). the authors found that better generalization could be achieved by reducing the variance of the adaptive stepsizes of Adam. In doing so, they suggested multiplying rectified scalar by m, when computing 0, in ( when the variance is large, which is referred to as RAdam. The AdaBound method Lf et al. (2019) is designed to avoid extremely large and small adaptive stepsizes of Adam, which has a similar effect as RAdam. In practice, AdaBound works as an adaptive method at the beginning of the training process and gradually transforms to SGD with momentum, where all the adaptive stepsizes tend to converge to a single value. Conceptually speaking, both RAdam and AdaBound aim to reduce the range of the adaptive stepsizes of Adam to mimic the convergence behavior of SGD with momentum certain extent.
(2017), Bronsage
Bernstein et al. 22020), and AdaBelief/Zuaan
al.
Inspired by the above work Liu et al. (2019); ; Luo et al. (2199), we consider suppressing the range of adaptive stepsizes of AdaBelief. It is noted that AdaBelief extends Adam by tracking the EMA of the squared prediction error (m, (m,-9:)² The update expressions of AdaBelief are given by Zhuang et al. (2000)
m: = -mm--1 + (1 31 (g)
[AdaBelief]
9: = 8 (1 52) (m: 91) +e
(3)
0. 01- 1/t 1- (1-)
We emphasise that the parameter is involved in the computation of both s, and 0 in AdaBelief, which is different from that of Adam. The work of Zhuang et al. (2020) does not study the impact of e in the computation of 8;, and mainly focuses on the motivation of the EMA of (m, - g. instead of the EMA of g being employed in Adam.
In this paper, we make three contributions. Firstly, explain why it is important to include the parameter e in the computation of s; in B which will be inherited by our new algorithm Aida as described later on. We show via Taylor expansion that the inclusion of e in the computation of essentially suppresses the range of the adaptive stepsizes of AdaBelief. The above property makes AdaBelief closer to SGD with momentum.
Secondly, we perform layerwise vector projections to further suppress the range of adaptive stepsizes of AdaBelief. Let us denote the subvectors of (m,, gt) for the Ith layer of a DNN model as (mi.t.. We perform K mutual- (0) (K) (K) for the (th layer starting from (m, gt.: (1,,), As extension (K) g1, ((K))² for the (th layer, where the resulting method is referred to Aida 3l The new method has the property that the adaptive stepsizes within each neural layer have smaller statistical variance, and the layerwise average of the adaptive stepsizes are more compact across all the neural layers than the reference method. A discussion on the benefit of the above property will be provided in Subsection 3.1 In addition, convergence analysis for Aida is conducted in Subsection 3.3 As an example, Fig. [ and 2 demonstrate that Aida indeed produces more compact range of adaptive stepsizes than AdaBelief and Adam for training VGG11 over CIFAR1O. Furthermore, the adaptive stepsizes of Aida become increasingly more compact as the iteration increases. It is worth noting that at the end of the training process, the 11 layerwise average stepsizes in Fig. [ do not converge to a single value, indicating the adaptability of Aida.
vector projections to obtain (m,, gi,t
of AdaBelief, we then track and employ the EMA (or equivalently the second momentum) of (m,
As example, considering Adam itcration t, the layerwise average of adaptive stepsizes for the Ith layer of VGG11 is computed as Dd, 1/(v pi,e [5/(11 - 85) + c), where vi,4 E Rd: the subvector of the 2nd momentum v. E Rd for the lth layer. 3; I11 named after an Italian opera by Verdi.
2

---

Published in Transactions on Machine Learning Research (08/2023)
Adam
AdaBellef
Alda (K-1)
Aida (K-2)
1000
1000
1000
1000
800
800
800
800
600
600
400
400
400
400
200
50
100
150
200
50
100
150 200
50
100
150
200
50
100
150
200
opoch
epoch
epoch
epoch
Figure 1: Comparison of layerwise average adaptive stepsizes for the 11 neural layers of VGG11 by training over CIFAR10 for 200 epochs. See Appendix C for the parameter setups of the three methods, where the optimal parameter for Adam was selected from discrete set to give the best validation performance. The jumps in the curves at 100 and 160 epochs are due to the change in the common stepsize. Aida has much more compact range of layerwise average stepsizes than Adam and AdaBelief, respectively.
400
Adam
400
AdaBelief
400
Aida (K-1)
400
Aida (K-2) 2)
300
300
300
200
200
100
100
100
50
100
150
200
100
150
200
50
100
150
200
50
100
150
200
epoch
epoch
epoch
epoch
Figure 2: Comparison layerwise standard deviations (stds) of adaptive stepsizes for the 11 neural layers by training VGG11 I over CIFARIO for 200 epochs. Aida has much smaller layerwise stds than Adam and AdaBelief, respectively,
Thirdly, extensive experimental results show that Aida with K = 2 yields considerably better performance than nine optimization methods for training transformer Vaswani et al. (2017) and LSTM Hochreiter & Schmidhuber (1997) models natural language processing (NIP) tasks, and VGGI1 Simonyan & Zisserman (2014) and ResNet34 in image classification tasks over CIFARIO and CIFARI00. It is also found that Aida matches the best performance of the nine methods when training WGAN-GP models in image generation tasks. Lastly, Aida outperforms AdaBelief when training 8 on the large ImageNet dataset.
Notations: We use small bold letters to denote vectors. The l2 and too norms of a vector y are denoted as || and | oc. respectively. Given an L-layer DNN model of dimension d, we use 0 of dimension dj to denote the subvector of 0 for the ith layer. Thus, there is Ete di d. The ith element of 0; is represented by (r]... The notation [4] stands for the set [L] = {l,..., L}. Finally, the angle between two vectors y and x of the same dimension is denoted by Lary.
2 Impact of e in computation of s; in AdaBelief
In this section, we study the impact of e in computing the second momentum 9; in (3), which is missing in Zhuang et al. (2020). By inspection of (3), one can see that the parameter appears twice in the update expressions, the first one for computing 8; and the second one for computing 6,. The impact of the second can be ignored due to the fact that ve((1 83) >> when is sufficiently small (e.g., e = le 8). a result, we only need to focus on the first when computing 8t.
Next, show that the first in the computation of s, helps suppress the range of adaptive stepsizes of AdaBelief. To this purpose, we reformulate the update expressions in 3
mi $1 m + (1 - B1)g:
&, = 828 + (1 B2) m: - Te = 82r=-1 +
(4)
0. 0-1 11 1-
((
3

---

Published in Transactions on Machine Learning Research (08/2023)
AdaBelief
AdaBelid"
1000
1000
800
800
600
400
400
200
200
50 100
150
200
50 100 150
200
epoch
epoch
Figure 3: Comparison of layerwise average of adaptive stepsizes for the 11 neural layers by training VGG11 over CIFARIO for 200 epochs. AdaBelief" is obtained by removing the first in the computation of 8, and only keeping the second e. See Appendix [A] and C for the update procedure of AdaBelief* and the parameter setups of the two optimization methods. The optimal for AdaBelief* is selected from a discrete set that gives the best validation accuracy.
where the second e is removed, and
8 + = 8, + c(1 - 115)((1 -B2),
(5)
where &0 = 0 and ro = 0. As result, the adaptive stepsizes 1
(8t 1 +r)) (1)) (35) 44 can be approximated as
1/ (6: rt)((1 (1/11/8))) (3)= 1/v 1/v8./( 8./(1
(3) e/(1 B2)
(6)
(7)
8,/(1 83)
c((1 32) 2 8,/ 83
1st term
2nd term
where in the last step. the Taylor approximation is applied to a function h(x) v around c/(1 B2) and = 8,/(11 (.).
0, where
We now investigate 7 Generally speaking, small elements of 8, lead to large adaptive stepsizes while large elements lead to small adaptive stepsizes due to the inverse operation 1/(-). is clear from (7) that for small elements of & the second term in the denominator is relatively large, implicitly penalizing large stepsizes. Furthermore, (6) indicates that those large stepsizes upper-bounded by the quantity 1/ve B2). In contrast, for large elements of & the second term is relatively small, thus avoiding extremely small adaptive stepsizes. In short, including in the computation of suppresses the range of adaptive stepsizes in AdaBelief by avoiding extremely small stepsizes.
Fig. demonstrates that when the first is removed from in AdaBelief, the resulting method AdaBelief* indeed has a broader range of adaptive stepsizes than AdaBelief. At epoch 200, the eleven layerwise average stepsizes in A are distributed in [190,1000] while ten out of cleven layerwise average stepsizes in AdaBelief are close to single value of 320. That is, the first in indeed makes the adaptive sizes of AdaBelief more compact.
3 Algorithmic Design
We showed the previous section that the particular placement of e in the update expressions of AdaBelicf suppresses the range of the adaptive stepsizes. In this section, we develop a new technique to further reduce the range of adaptive stepsizes of AdaBelief, which is referred as layerwise vector projections. The new method is named Aida. A convex convergence analysis is presented at the end of the section.
4

---

Published in Transactions on Machine Learning Research (08/2023)
Adam: {ofule 2 }}
mi, 5
(2)
AdaBelief: {(mi, 91,t) 2 }}
Aida: {ml) (K) (8) 2 20}
(2)
91.1
9t,
Figure 4: Computation of ((m
: -i,t (9t,))) [k [K]} by starting from the pair (mi,,, 91.4) via sequential and alternating
vector projections in Aida.
3.1 Motivation
Our aim is to design new adaptive optimization algorithm, in which the range of the adaptive stepsizes is smaller than that of AdaBelief. To achieve the above goal, we consider processing m, and Jr in layerwise manner at iteration before using them to estimate the second momentum Due to the nature of back-propagation when training DNN model 0, it is computationally more efficient to perform layerwise processing than operating on the entire vector m. and gt-
Intuitively speaking, when the variance of the adaptive stepsizes within particular neural layer is encouraged to be small by layerwise manipulation, the update for the model parameters within the same neural layer would become relatively robust to elementwise gradient outliers (exploding vanishing gradient elements across iterations). The recent work You et al. (2020) proposed the LARS and LAMB optimizers. They are, respectively, extensions of SGD with momentum and Adam that introduce layerwise normalisation in the update of DNN model per iteration. The purpose for introducing the layerwise normalization in You et al. (2020) to provide robustness to exploding layerwise gradients and plateaus for the scenario of utilizing extremely large batchsizes to train large DNN models. Differently from You et al. 2020), our work considers the scenario of normal training batchsizes.
We now study what kind of layerwise operation is desirable for reducing the layerwise variance of the adaptive step- sizes in AdaBelief. Firstly, we note that the parameter of (3) in AdaBelief essentially defines upper bound on the adaptive stepsizes and is independent of neural layer and iteration indices. By inspection of (4) (4). the upper bound can be expressed as
1
(8)
Vri/(1 35)++ Ve/(1 - 32) +
We use (m4,gg..)) to denote the subvectors of (m+.g) gt) for the ith neural layer. Suppose we track the EMA of (1,,,m,, :2 for the Ith layer, where 0 nt,, St, S 1 are functions of (mi,, gt,z), instead of the EMA of (mi, g1,²) being tracked in AdaBelief. If the scalars {n,,, 3 are sufficiently small in the extreme case, all the adaptive stepsizes of the new method tend to approach the upper bound in ( As result, the new method will have smaller range of adaptive stepsizes than AdaBelief either in a layerwise manner or globally.
We propose to compute the scalars (11,, of mi, and gt., mentioned above via K mutual-vector projections starting from (mi,,, .I.)) (see Fig. 4 for demonstration). In practice, it is found that K = is sufficient to produce small scalars (y,,,,1,c), leading parameter K of Aida in Fig. 2 was set to K E (1,2). In the following, the update expressions of Aida are presented in detail.
smaller range of adaptive stepsizes than those of AdaBelief and Adam. The
3.2 Aida as an extension of AdaBelief
Consider the Ith layer of a DNN model at iteration t. We perform a sequence of mutual-vector projections to obtain set of projected vectors {m
(k) (x2)) & [K]) starting from the initial pair (m 9 (00) (mi,) g1.)- Using
(0)
5

---

(R) Published in Transactions on Machine Learning Research (08/2023)
algebra, the two vectors iteration & can be represented
( mi, (R)
(K) (A))
(I+1)
14,
l + gt.t
(9)
(10)
(g
mi.))
(l++)
gt. | 112
(1c) 1,
(10)
where (.,)) denotes the inner product, and 5 > 0 is scalar parameter make sure the division operations are valid, The above two projections (I) ensure that the resulting projected vectors share the same vector-direction either MA, or Dt,t- See Fig. for visualisation.
Remark 1. We note that the K mutual-vector projections starting from the pair of gt. and ma, can, in principle, be realized by scaling the Iwo vectors via the Kth power of the cosine of L which would make the additional computational overhead roughly constant a function of K. In practice, we found that positive value of the param- eter in (9 and (70 of the mutual-vector projections improves the generalization performance, which, for simplicity, was set to le 20 across all the experiments in the paper. We refer the reader to Appendix []r supplementary experiments on the utilization of the Kth power of the cosine of m. to perform the scaling. The empirical study in Subsection 4.3 indicates that Aida with K = only introduces a small computational overhead compared to AdaBelief, which makes the formulation of the mutual-vector projections ( and (TO) practically affordable.
Once (m,, Ii, (K)) are obtained for the Ith layer, Aida tracks the EMA of the squared difference (m),) g..) given by
(k)) 2 (K) - -1. + te, V1,1 = 32 + (1 32) (mi
(11)
where > 82 O, and c 0 is added recommended by our earlier analysis. With 01,6, the model parameters or,, of the Ith layer can be updated accordingly. See Algorithm I for a summary of Aida.
Next, we consider the geometric properties of the set of projected vectors. It is not difficult to show that after projection, the resulting vectors have either shorter or equal length in comparison to the original vectors:
(k) 1|2 < |
|m -mff--1l
1l2
and
|| (k) 12 S l1
(k-1) 1 V122-
(12)
Using the fact that mutual projections of two vectors do not change the angle, we then have
| gi 2| < || (k-1) 1)
(k)
(k)
gi, (k-1) 1122
(13)
where the equality holds if mı, and gt.t are on the same line and & can be ignored in 10
For the extreme case that each neural layer has only one parameter (i.e., It. E IR, VI E [L]. it is easy to show that the projection operation has no effect. That is, (g mm(a))2 (gi, m4,)) for all k [K] if g is ignored in (10). In this case, Aida reduces AdaBelief.
From the above analysis, we can conclude that the EMA of (m (K) g2 2 for the Ith layer can be viewed as the
EMA of (fK...mii - $K.L.gI,)?, where the scalars "K..I. 3K,L E (0, 1]. In general, the angles mt, 2 0} would be non-zero due to randomness introduced by the minibatch training strategy in a typical DNN task. As result, increasing the number K of vector projections would cause the elements of {7K,,,, 8Ka.lt 2 0} to approach zero. In other words, the parameter K controls the range of the adaptive stepsizes of Aida. A larger K makes the adaptive stepsizes more compact.
Figs. D and 2 provide empirical evidence that Aida does indeed have a smaller range of adaptive stepsizes than Ad- aBelief and Adam. Furthermore, as K increases from to 2, the range of adaptive stepsizes of Aida becomes increas- ingly compact. Hence, Aida makes a bridge between SGD with momentum and AdaBelief. As will be demonstrated in the experiments, Aida improves the generalization of Adam and AdaBelief for several classical DNN tasks.

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm 1 Aida: Suppressing the range of adaptive stepsizes of AdaBelief by layerwise vector projections
Input: 31. 32, Th. > 0, 1e 20, K
Init.: 0g E my =0, 2 Du = 0 Rd
for 1,2,.... ................................ do
9: +- vf(0=-1)
m, - 8mm + +11 (1- 31)gc
for = 1,.... L do
m (0) m
(0) (0)
9:
9t
for k = 1, 1,... K do
(k) (m Pis (k-1)
mi.
llg. 1 91.t (k-1)
(k)
(m t-1
91,1
m (k 14-1) 11g4
1
end for
V1,1 - 32011111 (1 32) (m) gi.t )
(K) (K)
end for
v L
mi - 1-31
m
1- 1=1
0,+ 01-1-
1i: V
end for Output: 0r
3.3 Convergence analysis
In this paper, we study the convergence of Aida for convex optimization. Our analysis follows strategy similar to that used to analyse AdaBelief in Zhuang et al. (2020).
Theorem 1. Suppose {0. and (vr )T are the iterative updates obtained by either Aidd starting with (mo. vo) (0, 0). Let 0 S S1t = A < 1,0 < 32 < 1, and fpp = 3 Assume (I): f(0) is a differentiable convex function
with lIge -c S Gz/2 (hence |m g ( S Gxx for all E [T]; (2): the updates {0a)T= and the optimal o. I 3 for all
solution 0" are bounded by hyper-sphere, i.e., ||0|12 D and | ||2 S D; (3): 0
E (l....,d) d} and E [T]. Denote Or = + - following bound on regret:
O2 and g = ((g) [i) ...., (gTl))) E R² We then have the
11( -11((1 - B2)T + D2d + vC)
D2 + Vc)
(1+ B1) n/1 + log T
1(0)) ((0))
+
2(1 B,)T Ve(1 B,) (
VTn(1 - 8:)(1 32)
D2 + Vc)
(14)
T(1 - 31)(1 (2))n(1 - 2)²²
Proof. See Appendix for proof.
We note that the major difference between our analysis for Aida and that for AdaBelief in Zhuang et al. (2020) that we assume ë, S 6 1] while the analysis in Zhuang et al. (2020) essentially utilizes the condition é 2 0, j 1] for convergence analysis. Our motivation for the condition is that the iteration index increases, the gradient ge tends to approach zero, which makes ún approach to zero.
4 Experiments
We evaluated Aida on three types of DNN tasks: (1) natural language processing (NLP) on training transformer and LSTM models; (2) image classification on training VGG and ResNet He et al. (2015) models; (3) image generation
48 in Algorithm is generalized 81.. 0 facilitate convergence analysis.
7

---

Published in Transactions on Machine Learning Research (08/2023)
training WGAN-GP Gulrajani et al. (2017). Two open- repositories were used for the above DNN training tasks. The first repository was adopted for the task of training a transformer. 'he second on was used for all the remaining tasks, which includes the original implementation of AdaBelief. The second repository was used to compare AdaBelief with SGD with momentum and seven adaptive optimization algorithms from the literature, namely Yogi Zaheer et al. (2018), RAdam Liu et al. (2019), VSVAG Balles& & Hennig (2017), Fromage Bernstein et al. (2020). Adam Kingma & Ba (2017), AdaBound Luo et al. (2019), and AdamW Loshchilov & Hutter (2019). We compared our Aida algorithm with all the above optimization methods.
We now briefly explain the selection of syper-parameters. for the tested optimizers, of which the detailed information was provided in Appendix D For our experiments on using the second repository. the tested hyper- parameters were inherited from the source code for AdaBelief. To be specific, in the source code of AdaBelief, the optimal parameter of AdaBelief was, in general, set differently for different DNN tasks. The setting for the parameter in Aida follows that of the Adabelief setup in most cases and hence also was configured differently for different tasks. To make a fair comparison with Aida and AdaBelief, the optimal parameter c was searched for the other adaptive optimizers for each DNN task. For each optimizer, we stopped searching when its performance dropped significantly, indicating the value of e was either too large or too small. Lastly, when training LSTM models in the open-source of AdaBelief, the initial learning rate 1/u of the tested optimizers was searched together with c. Consequently, in our work, we also searched the initial learning rate for those optimizers when training LSTMs.
It was found that Aida with K = 2 outperforms the nine reference methods for training transformer, LSTM, VGGIL, and ResNet34 models while it matches the best performance of the nine methods for training WGAN-GP. Furthermore, experiments on training ResNet18 8 on the large ImageNet dataset show that Aida outperforms AdaBelief.
The time complexity of Aida evaluated for training VGG11 and ResNet34 2080 Ti GPU. In brief, Aida with K = 2 consumed 25% more time per epoch compared to AdaBelief.
4.1 On training transformer
In this task, we consider the training of a transformer for WMT16: multimodal translation by using the first open- source as indicated in the footnote. In the training process, we retained almost all of the default hyper-parameters provided in the open-source except for the batch size. Due to limited GPU memory, we changed the batch size from 256 to 200. The parameters of Aida were set to (No, $1 $2, c) (0.001, 0.9, 0.98, 1e 16). The parameter-setups for other for each optimizer were searched over some discrete sets to optimize the validation performance. For example, the parameter e of Adam was searched over the set (1e-6, 6, 1 1e-1e- 16} while the remaining parameters were set to (7o, $1, 32) = (0.001, 0.9, 0.98) as in Aida. Once the optimal parameter-configuratie for each optimizer was obtained by searching, three experimental repetitions were then performed to alleviate the effect of the randomness.
optimizers be found in Table 7 of Appendix D where certain
It is clear from Table T and Fig. 5 that Aida significantly outperforms all other methods. We emphasize that the maximum number of epochs was set to 400 for each optimizer by following the default setup in the open source, and no epoch cutoff is performed in favor of Aida. When K increases from I to 2 in Aida, the method converges considerably faster and produces better validation performance, which may be due to the fact that Aida with K = 2 has a more compact range of adaptive stepsizes. On the other hand, the non-adaptive method SGD with momentum produces a performance that is inferior to all adaptive methods except Fromage and MSVAG.
4.2 On training LSTMs
In this experiment, we consider training LSTMs with a different number of layers over the Penn TreeBank dataset Marcus et al. (1933) by utilizing the second open-source repository for AdaBelief. As explained earlier, when training LSTMs in the original source code, both the initial learning rate тo and the parameter were searched in those adaptive optimizers. In this work, we followed a similar procedure. See Table g in Appendix D for a summary of the fixed and free parameters for each optimizer. An example is Adam for which y)o E {0.01, 0.001] and c E {1e-6, le-7, le- were tested to find the optimal configuration that produces the best validation performance.
httpp///ghhhoooooooreelllllaoooo pytorch
8

---

Published in Transactions on Machine Learning Research (08/2023)
Table 1: Performance comparison for training the transformer
SGD
55.58=0.34
(non-adaptive)
AdaBound
55.90:10.21
Yogi
60.47:10.61
RAdam
64.47:10.11
MSVAG
53.79:10.13
Fromage
35.57:1:019
Adam
64.71++0.57
AdamW
64.49:10.24
AdaBelief
66.90:10.77
Aida(K=1)
68.77-+0.16 16
Aida(K=2)
68.96-10006
(a): train perplexity epoch
(b): val. epoch
10
Adam
AcaBeliel
65
Aisa(K-11
Aida/K=2)
60
10
50
10
40
Adam
AdaBeriel
35
Aida/K-11 Aids/K>22)
30
25
100
200
300
400
100
200
300
400
epoch
epoch
Figure Performance visualisation of Aida, Adam, and AdaBelief for the training of the transformer. See Fig, in Appendix F for the plots of the associated learning curves versus runtime.
Table 2: Validation perplexity on Penn Treebank for 1, 2, 3-layer LSTM. lower is better.
Aida(K=1)
AdaBelief
AdamW
Adam
Yogi
AdaBound
layer
82.27
84.21
88.36
84.28
86. 66.78
84.52
layer
66.16
66.29
73.18
66.86
71.56
67.01
layer
61.98
61.23
70.08
64.28
67.83
63.16
SGD
Aida(K=2)
RAdam
MSVAG
Fromage
(non-adaptive)
layer
81.53
85.52
88.76
84.75
85.20
layer
65.04
67.44
74.12
68.91
72.22
layer
60.18
63.68
70.41
65.04
67.37
1-layer LSTM
(b): 2aaayer LSTM
3-layer
105
AdeRaia! Aicaak-2)
85
ActaBeliaf AdatK=a)
AdsRiie AkcatK=2)
100
lest perplexity
test perplexity
80
1est Serpiaan
75
70
90
70
65
60
150
200
opoch
epoch
opoch
Figure 6: Performance visualisation of Aida, AdaBelief, and Adam Table of the associated learning curves versus runtime.
See Fig.
in Appendix for the plots

---

Published in Transactions on Machine Learning Research (08/2023)
Table 3: Validation accuracies (in percentage) and time complexity in seconds per epoch (referred c.) of nine methods for training VGG11 and ResNet34 over CIFARIO and CIFARI00. The best result is highlighted in each column.
CIFAR10
CIFAR100
VGGII
ResNet34
VGGI1
ResNet34
optimizers
val. acc
t. c.
val. acc
t. c.
val. acc
t.
val. acc
C.
SGD
91.36=0.07
5.83
+0.1
30.45
0..0
5.85
18.10:0.11
30.92
(non-adaptive)
Yogi
90.7410.16
6.49
9.29
31.74
65.57=0.17
6.42
77.17=0.12
32.20
RAdam
89.1
6.28
94.64++0.18
31.21
63.62++0.00
6.29
74.87+10.13
31.58
MSVAG
90.04=0.22
7.08
94.65
33.78
62.67=00.3
7.19
75.57=0.14
33.80
Fromage
89.72=10225
6.66
94.6411.07
35.19
0..93=0.53
6.56
74.84+0.27
35,50
Adam
91.20+0.21
6.15
95.09=10118
31.28
0.113
6.20
77.31
31.47
AdamW
89.46=0.08
6.25
94.48=0.18
31.71
63.50=#0223
6.31
74.29+0.200
31.80
AdaBound
90.4810.12
6.71
94.7310.16
33.75
64.80=10.42
6.73
76.15+0.10
33.78
AdaBelief
0..5511.13
6.47
95.15:1.111
31.66
68.05
6.49
77.32:10.37
31.74
Aida(K=l)
91.52110005
7.27
95.31
35.25
68.89-0.09
7.32
77.50:0.12
35,46
Aida(K=2)
91.6810.116
7.95
95.57=0.13
39.53
69.02=0011
8.01
78.86+
39.64
Table Best FID obtained for each optimizer (lower better)
Aida(K=2)
Aida(K=1)
AdaBelief
Adam
RAdam
AdaBound
best FIDs
55.7
55.65
56.73
66.71
69.14
61.65
AdamW
MSVAG
SGD
Yogi
Fromage
best FIDs
63.76
69.47
90.61
68.34
78.47
We emphasize that the parameters of Aida were set to (No, B1, 82, e) (0.001, 0.9, 0.999, le 16), whereas yoo and remain the same for all different numbers of neural layers in LSTMs.
Table. Suble.2aar summarises the obtained validation perplexities of the ten methods for training 1, 2, and 3-layer LSTMs. It was found that for each optimizer, independent experimental repetitions lead almost the same validation perplexity value. Therefore, we only keep the average of the validation perplexity values from three independent experimental repetitions for each experimental setup in the table and ignore the standard deviations.
It is clear from Table. that Aida again outperforms all other methods in all three scenarios. Fig. 6 further visualises the validation performance of Aida compared to AdaBelicf and Adam. The performance gain of Aida is considerable in all three scenarios. One observes that the initial training stage, Aida converges slower than AdaBelief only for the I-layer LSTM. This because the optimal parameters (No, e) in AdaBelief are different for different numbers of neural layers. In particular, the optimal setups AdaBelief are: (No.e) = (0.001, 1e 16) for 1-layer LSTM, and (mo, e) (0.01, le 12) for 2-layer and 3-layer LSTMs. When the parameters of AdaBelief for the 2-layer and l-yayer LSTMs also set to (7o. (0.001, le 16) be in line with the setup of Aida, it is found that Aida also converges slower than AdaBelief in the beginning of the training process.
4.3
On training VGG11 and ResNet34 over CIFAR10 and CIFAR100
In this task, the ten optimizers were evaluated by following similar experimental setup in /Zhuang et al. (2020). In particular, the batch size and epoch were set to 128 and 200, respectively. The common stepsize vic is reduced by multiplying by 0. at 100 and 160 epochs. The detailed parameter-setups for the optimizers can be found in Table T0 Appendix Three experimental repetitions were conducted for each optimizer to alleviate the effect of randomness.
Both the validation performance and the algorithmic complexity are summarised in Table 3 clear that Aida with K 2 consistently outperforms the nine reference methods in terms of validation accuracies at the cost of additional
10

---

Published in Transactions on Machine Learning Research (08/2023)
computational time. This demonstrates that the compact range of adaptive stepsizes in Aida does indeed improve the generalization performance.
We can also conclude from the table that SGD with momentum is the most computationally efficient method. On the other hand, due to the layerwise vector projections, Aida with 2 consumed an additional 25% time per epoch compared AdaBelief.
4.4 On training WGAN-GP over CIFAR10
This task focuses on training WGAN-GP, where the setup 31 = 0.5 follows the original setting in training AdaBelief.
The parameters of Aida were set to (Mt, 31, 32, e) = (0.0002, 0.5, 0.999, 1e 12)1 in agreement with the recommended 12), setup of AdaBelief in the original open-source. The other eight optimizers have both fixed and free parameters, details of which can be found in Table of Appendix D As an example, the free parameter of Adam is {le 4, le performed due
6,...; le to the relatively unstable Frechet inception distance (FID) scores in training WGAN-GP.
14). For each pwrameter-confiuraaae of optimizer, three experimental repetitions
Table shows the best FID for each method. can be seen from the table that Aida with K E {1,2) provides better performance than AdaBelief, while the other methods perform significantly worse. It is worth noting that the FID for AdaBelief is better than the reported results in Zhuang et al. (2020) which may be due to different versions of python packages.
4.5 On training ResNet18 over ImageNet
In the last experiment, we investigated the performance gain of Aida compared to AdaBelief for training ResNet18 on the large ImageNet dataset. The maximum epoch and minibatch size were set to 90 and 256, respectively. The common stepsize T) dropped by a factor of 0.1 70 and 80 epochs. The parameter setup for the two optimizers can be found in Table 6 of Appendix D To make a fair comparison, the parameter of AdaBelief was searched over the discrete set {le 8, le 9, le 10} instead of using the recommended setup of e 1e - 8 in the original repository for AdaBelief. Three experimental repetitions were conducted for each optimizer to mitigate the effect of randomness.
It is seen from 5ahat for the large ImageNet dataset, Aida performs slightly better than AdaBelief, indicating that the performance gain Aida is robust against different sizes of datasets. We note that the performance of AdaBelicf is slightly lower than the reported result in [Zuang et al. (2020), where no standard error is reported. This might be either because we repeated the experiments three times per optimizer or due the fact that different versions of the Python packages were used, which difficult track.
From an overall perspective, all five experiments in our paper demonstrate that Aida consistently outperforms Ad- aBelief. The performance gain is significant for certain tasks and small for other tasks. Our results suggest that proper manipulation of the range of adaptive stepsizes of an adaptive optimizer improves generalization performance. We hope our work will lead to the development of related techniques to improve the performance of other existing optimizers.
Table 5: Validation accuracies (in percentage) of AdaBelicf and Aida for training ResNet18 over ImageNet.
optimizers
AdaBelief
Aida( K = 2)
val. acc.
69.65+0000
69.70=0008
5 Conclusions
In this paper, have shown that the range of the adaptive stepsizes of DNN optimizers has significant impact on performance. The proposed Aida optimizer suppresses the range of the adaptive stepsizes of AdaBelief making it closer to SGD with momentum. Our experimental results indicate that Aida will be able to produce better performance across a wide range of DNN-based applications.
In the design of the Aida optimizer, we track the EMA (or equivalently the second momentum) of (11, Mi, - I1.) for the 1th layer of a DNN model as opposed (m,,--0 g..))² used in AdaBelief, where Nt.. Br. E (0, 1] are obtained by vector projections. Consequently, the adaptive stepsizes of Aida have more compact range than those of AdaBelief.
11

---

Published in Transactions on Machine Learning Research (08/2023)
Our empirical study shows that Aida with K 2 outperforms nine optimizers including Adam and AdaBelief for training transformer, LSTM, VGGH1, and ResNer34 models while at the same time, it matches the best performance of the nine methods for training WGAN-GP models. In addition, experiments on training ResNet18 over the large ImageNet dataset show that Aida performs better than AdaBelief. On the other hand, it was found that the non- adaptive method SGD with momentum only produces good performance when training VGG and ResNet models, This suggests that the adaptivity of Aida important, allowing the method to effectively train different types of DNN models.
References
L. Balles and P. Hennig. Dissecting adam: The sign, magnitude and variance of stochastic gradients. arXiv preprint DXXiv:175.007720 2017.
J. Bernstein, A. Vahdat, Y. Yue, and Ming- Yu Liu. On the distance between two neural networks and the stability of learning. arXiv preprint arXiv:2002.03432. 2020.
N. Chen, Y. Zhang, H. Zen, R. J. Weiss, M. Norouzi, and W. Chan, WaveGrad: Estimating Gradients for Waveform Generation. arXiv:2009.00711 September 2020.
T. Dozat. Incorporating Nesterov Momentum into Adam. In International conference on Learning Representations (ICLR), 2016.
J. Duchi, E. Hazan, and Y. Singer. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.
D. Silver et al. "Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):884 2016.
Guirajani, F. Ahmed, M. Arjovsky, Dumoulin, and A. C. Courville. Improved training of wasserstein gans. Advances neural information processing systems, pp. 5767-57777 2017.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2015.
S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation, 98): 1997.
K. Lee Devlin, M-W. Chang and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv: S.V:1880000400. 2018.
D. P. Kingma and J. L. Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: 2017.
Y. LeCun, Y. Bengio, and G. Hinton. Deep Learning. Nature, 521:436-444, 2015.
L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arxi 1rXiv:1908.03220 032665..5, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In International Conference on Computer Vision (ICCV), 2021.
Loshchilov and F, Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019.
L. Luo, Yuanhao Xiong, Y. Liu, and X. sun. Adaptive Gradient Methods with Dynamic Bound Learning Rate. ICLR, 2019.
M. Marcus, B. Santorini, and M. A. Marcinkiewiez. Building large annotated corpus of english: The penn treebank, 1993. http:tttt///aataaaa alto ldc. www.n.eeee.m.m...m......e httpea/ncedu//bocs//oo/ss// http://cc955///// 1tt
B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4:1-17, 1964.
12

---

Published in Transactions on Machine Learning Research (08/2023)
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
H. Sutskever, Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning, In International conference Machine Learning (ICML., 2013.
T. Tieleman and G. Hinton. Lecture 6.5-RMSProp: Divide The Gradient by a Running Average of Its Recent Magni- tude. COURSERA: Neural networks for machine learning, 2012.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. N. Gomez, L.. Kaiser, and I. Polosukhin. Attention is all you need,
[cs, CLJ. 2017.
A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The Marginal Value of Adaptive Gradient Methods in Machine Learning. In 31st Conference Neural Information Processing Systems (NIPS), 2017.
Y. You, J. Li, S. Reddi, Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh. Large Batch Optimisation for Deep Learning: Training Bert in 76 Minutes. In ICLR, 2020.
M. Zaheer, S. Reddi, D. Sachan, S. Kale, and S. Kumar. Adaptive methods for nonconvex optimization. In Advances in neural information processing systems (NeurIPS), pp. 9793-9803, 2018.
J. Zhang, S. P. Karimireddy, A. Veit, S. Kim, S. J. Reddi, S. Kumar, and S. Sra. Why ADAM Beats SGD for Attention Models. In submitted for review by ICLR, 2019.
Zhuang, T. Tang, S. Tatikonda Ding, N. Dvornek, X. Papademetris, and J. Duncan. AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients. In NeurIPS, 2020.
13

---

Published in Transactions on Machine Learning Research (08/2023)
A
Update procedure of AdaBelief*
The first e removed in to verify if AdaBelief* has a broad range of adaptive stepsizes than AdaBelief.
Algorithm 2 AdaBelief*
Input: 31, 32, Th,
Init.: E my = 0, to 0 Rd
3: for L = .,,, , do
4:
()(
5: m: - B1m=-1 + (1 B1 gt
6: qt + 829--1 + (1 B2) m, - 9:)²
m. - I- qt I-2
9t
0.4 01 Vacte
end for
10: Output: Or
B Convex Convergence Analysis
Firstly, we note that the bias term 1 - 8i in the update expressions of Aida in Algorithm can be absorbed into the common stepsize 1)- Therefore, we will ignore the bias term in the following proof. Suppose 0 is the optimal solution for solving the convex optimization problem, i.e., 0 arg mine, f(0). Using the fact that 0₁ 0:-1 -, 11/2 we have
m.:
10 /4 (0, 00)1
[|0|/4(0,
17: 1/2 m. - 00
01)1 + n/ 0; 1/111 11 2m + (1 Bre]g 0,- -0")
m.
[10//((0,-
001 n ", m.
1/- 112
2n 11-) (gt, 0. - - 11 0.1--
s |10]/4(0, 0')13 + n |, 1/4 m. + 11.111 1/4 m 31e118,/
-
-1) (g) 0,-1 0")
((:-1 -0:-1-0"113 0²)
(15)
where the above inequality uses the Cauchy-Schwarde inequality 2(a, b) S ||a + [b]2 Note that 15 corresponds to (2) in the appendix of Zhuang et al. (2020) for AdaBelief.
Summing (15) from until & T, rearranging the quantities, and exploiting the property that g: v.((0.1))
and f(-) being convex gives
f(OT) f(0")
T-1
f
Σ
0: -(00)
T
(f(0.-) - f(0)))
T
T
(g. 01-1--1: -")
1
(162/1(0--
00) | //, 00 )0))
T
2n 318)
14

---

Published in Transactions on Machine Learning Research (08/2023)
+
7/
00
1/4
m.
11
1r.31
01 m.
1--1 1/1
m,-1
11 +
B1e
10, (0.- 00)
1/4
2(1 Bie)
2(1-B1r)
21/( (1 - Bix)
91=01
|0] 1/1/4/1 (00- -0")13
21y(1 B.
1
11 10 (0. - -0")1²
1
10: 1/4 (0. 00)
20/++1 (1 B1((+1))
2/p (1 B1o))
1 T
Σ (=1
1)t 2(1-311)
|,,, 1/1
mm112 + 2(1-
1/:111
10, 1/4 m1-1112
311
|1 1/4 10/ 00)1
217(11 $11)
00 0 S [0] for all i 1,........... I....,d.) d, ) {
condition:
0ss B1(t+1) S Bi <1
100//(00 -0')11
211(1 - B1)T
1
1
1
1
1/4
(0, - 00)1
7/++1
2(1
31c)
1=1
711
1/0.1/1m 1//mm11/++
1/4
1/
|,,-11 1/4
m, m1
B1 10,/((0, 0* 00]
T t=1
2(1 (1)
2(1-81)
21,((1 31)
(condition: T/2 = i-n//Vi S
31111 S11 < 10 =0)
T-1
100 0')1
1
vt vt+1= - vi 110 -0)
$ 211(1 B1)T
+
2Tn(1 - B1)
1=1
1 (1 + 31)
Σ
10,
1/4
mmrl3 +
1
Bie 2/1 /2(0001111 0²)
T
2(1 -1)
T(1 -8)
21te
t=1
(condition:
S 100
DD
D2
D2
T-1
d vt+1 Vt
(81 1/2
(014110))1/2
n - T
Tn(1 - B1) 1=1
1=1
1=1
T
Te( 31))+
|0,
1/4
m + T(1 - )
D22
T
31
( (1))
2(1-
1=1
i=1
(0) Dx + VC)
D'd(Goo + v
B1
-1 1/1
123idd(Gs + Vc)
|0
m:
111 (1-31)(1-327
(T((L=B1((-82) 32) t=1
2(1 2(1-31)
T0188)11-3)>11-17
(16)
where both step (a) and (b) use the property of f(-) being convex, and step (c) uses the following conditions
Igg. 20 GGx/22 /2
|m
(K)
(K)
Il <ox = 11/211>0 S(G vOP2(11 -B2)
(17)
Bu d1
$ Σt X-15 S --ー
where the 1st condition in is derived by using the expression
17
01.
T1.
81.2
+
where 81, = - 11-38)81,-11 + (2
(K) - -i, (K) y²
1
32
Next consider the quantity Σt= 1/1 |0, 1/4 m.
|1 in 16),
the upper bound of which is given in Zhuang et al. (0220)
Lemma I (Equ. (4) in the appendix of Zhuang et al. 22020). three assumptions given in the theorem, we have
Let ( ? ..... ()) E RR.. Under the
T
7 + log T
Σ
1/4 mill
<
Ve(1 ) (r)
(18)
15

---

Published in Transactions on Machine Learning Research (08/2023)
Finally, plugging (88 into (66 produces the upper-bound regret in the theorem. The proof is complete.
C Parameter Setups for Optimization Methods in Fig. 1-3
The common stepsize tit is dropped by factor 0.1 at 100 and 160 epochs. The optimal parameter is searched over the set (10-?, 110-3,10-3 3 ....,,00' 10- 10 for Adam and AdaBelief".
optimizer
fixed parameters
searched parameters
(mo. B1, $2)
Adam
{10-2,10* 2,10
10 10-15}
-0.001, 0.9, 0.999)
(no. (82)
AdaBelief"
{10-2,11 3,10 10-3 10 16
(0.001, 0.9, 0.999)
No. 31. ())
AdaBelief
(0.001, 0.9, 0.999, 10 10-*)
Aida Aida
No. 11, ()
(K 1)
(0.001, 0.9, 0.999, 10 10-
No, 31. 62,
(K 2)
(0.001, 0.001,009.000 0.9, 0.999, 10 10-*) *)
D Parameter-setups for training different DNN models
Table 6: Parameter-setups for training ResNet over ImageNet. The weight decay was set to 10-2. searched parameters
optimizer
fixed parameters
(no. 31, 52)
AdaBelief
{10-5 10-9, 10.10-10)
(0.001, 0.9, 0.999)
Aida KK 2)
(No, (1, (0.001, 0.9, 0.999, 10 10-9)
Table 7: Parameter-setups for training Transformer. The weight decay for AdamW was 5e 4 while the weight decay for all other algorithms was set to 0.0. The setup B3 0.98 follows directly from the first open-source repository. One observes that the search grids of e are different for different optimizers. This is because we stopped searching for a particular optimizer when the performance dropped significantly as e is either too large or too small. optimizer
fixed parameters
searched parameters
c (lc 6, le 7, le 16)
AdaBound
(no, 31, (2, 7) = (0.001, 0.9, 0.98, 0.001)
final stelistep E (0.1, 0.01, 0.001}
Yogi
(No, 31, (2) (0.001, 0.9, 0.98)
CE {1e 2, le 3. 3..... le 16)
SGD
momentum=0.9
1Jo E (1.0, 0.01, 0.001)
RAdam
(No, (1, (32) (0.001, 0.9, 0.98)
CE (le 6, le 7 7...., le 16)
MSVAG
(No, 31, [82] (0.001, 0.9, 0.98)
C [le le 7...., le 16)
Fromage
Noo E (0.1,0001 0.001, 0.0001
Adam
(No, 31. (32) (0.001, 0.9, 0.98)
CE [lc 6, le 7..... le 16)
AdamW
(No, 31, [2) (0.001, 0.9, 0.98)
e [le le 7.... le 16)
AdaBelief
(No, 31, (32) = (0.001, 0.98)
CE [le 8, le 9..... le 16)
Aidr(K=1)our)
(no, 31, B2, c)
(0.001, 0.9, 0.98, le 16)
(odr(K=2)our)
(Nu, B1, B2, c)
(0.001, 0.98, 1e 16)
16

---

Published in Transactions on Machine Learning Research (08/2023)
Table 8: Parameter-setups for training LSTMs. The weight decay for every algorithm searched parameters
1.2e 6.
optimizer
fixed parameters
?00 {0.01, 0.001)
AdaBound
72, = 00.9, 0.001)
C {le 6, le - 17ee-7.... -,...., le 16)
final_stepsize E (0.1,3, 30)
?)0 {0.01, 0.001}
Yogi
(B1,82) (0.9, (0.990))000.
CE (1e-2. le 3, le 4, le 16)
SGD
momentum=0.9
7o E (30, 3, 1, 0.11
?/o E {0.01, 0.001]
RAdam
(31, = (0.9, 0.999)
6 {le 6, le - 7,..., le- 16]
Yo E (30, 1, 0.00,
MSVAG
(01,/8) (0.9, 0.999)
e {le 6, le 7,..., le 163
Fromage
flu E (0.1. 0.001
?0 (0.01,0.001)
Adam
(81, (2) (0.9, 0.999)
6E {le 6, le 7,:1---777... 7,.... le 16)
7/0 {0.01, 0.0013
AdamW
(81, (2) (0.9, 0.999)
C {le 6, 1c 7..... le 16)
700 (0.01, 0.001
AdaBelief
(B1, (2) (0.9, 0.999)
CE fle 8, le 9,
le 16)
Aida(K=1) (our)
(Mo, 01, 6) = (0.001, 0.999, 1e 16)
Aida(K=2) (our)
(Mo, $1, 12, c) = (0.001, 0.9, 0.999, le 16)
Table 9: Parameter-setups for training WGAN-GP. The weight decay for AdamW decay for all other algorithms set to 0.0. We stopped searching the parameter for particular adaptive optimizer when the FID score dropped significantly for either too large or too small values. optimizer
5e - 4 while the weight
fixed parameters
searched parameters
e [le - 2, le .,..., le 14)
AdaBound
(no, B1, B2, 7) = (0.0002, 0.5, 0.999, 0.001)
final stelsste E (0.1,0.01)
Yogi
(no, 31, (2) (0.0002, 0.5, 0.999)
6 E - 2, le 3, le 4, 14)
momentum (0.3, 0.5,
SGD
1/ E {0.1, 0.02, 0.002, 0.002,0.0002
RAdam
[0000==00000000000000 (No, 11. = (0.0002,0) 0.5, 0.999)
c [le 4, 1e 6 le 14)
1o E {0.1, 0.02, 0.0.2,.00.200000 0.0002]
MSVAG
(B1, B2) (0.5, 0.999)
6 (le 4, le 6.....
14]
Fromage
?)o (0.1, 0.01, 0.0,,00000001 0.001
Adam
(No, 31, (2) (0.0002, 0.5, 0.999)
CE {le 4, le
le 14)
AdamW
(No, (3) (0.0002, 0.5, 0.999)
6 6 {le 4, le
14)
AdaBelief
(Jo, (1, <) = (0.0002, 0.5, 0.999, le 12)
Aida(K=1) (our)
(Jo, B1, (1, e)
(0.0002, 0.5, 0.999, le 12)
Aida(K=2) (our)
(Jo, 81, 01, <)
(0.0002, 0.5, 0.999,
12)
17

---

Published in Transactions on Machine Learning Research (08/2023)
Table 10: Parameter-settps for training VGG and ResNet models CIFARIO and CIFARIO0. The weight decay for AdamW set 0.01 while the weight decay for all other algorithms to 5e 4. As mentioned in the main paper, we stopped searching the parameter when the validation performance dropped significantly for either too large or too small values.
optimizer
fixed- parammerssssters
searched-parametne
E (le - 2, le 3, 1c 9)
AdaBound
(no, 110.31,88 32, 7) = (0.001, 0.9, 0.999,
final stepsize E {0.1, 0.01]
Yogi
(no. B1, B2. = (0.001, 0.9, 0.999)
EE {le 1, le 2,... le 9)
SGD
(momentum, 77)) (0.9, 0.1)
RAdam
(1,.31, 32) = (0.001, 0.9, 0.999)
EE (le - 2, - 3
9}
MSVAG
(T), 31, 32) 1, 0.9, 0.999)
e (le - 2, - 2
9}
Fromage
1/ [0.1, 0.01, 0.001)
Adam
(No. 31, (82) (0.001, 0.9, 0.999)
C (le - 2, 3,
9}
AdamW
(no. 31, (32 - (0.001, 0.9, 0.999)
e (le - 2, le 3
9}
AdaBelief
(mn, ,1, (32) = (0.001, 0.999)
6 [le 8, 9,
10}
Aida(K=1) (our)
(No: B1, $2, c) (0.001, 0.9, 0.999, le 9)
Aida(K=2) (our)
(N), B1, $2, e) = (0.001, 0.9, 0.999, le 9)
Algorithm Aida": Scaling (mi...... via power cos Lm,
Input: 31 82, 1t. 0, le 20, =
Init.: 0, RR my = =0, 8o 0u = 0 Rd
for t= 1.2,.... do
9: + Vf(0:-1
m, - 31m,-17 + (1 31)g(
for = -,..., do
N1. = (mi,e. (1.1) m/. gt.ll 5
Ut. < f201,1-1
(2) mi, ?
[the Kthh power of nh is employed for scaling]
end for
m, f m.
v t1, +
1-91
1- /// 1=1
0:- + 0 nam
end for Output: 0r
E Performance Investigation by Scaling (mi,,, gi,)) via power of cos Zmy,gi,
In this section, we consider scaling It and my, in the computation of the second momentum in AdaBelief by utilizing the power of cos Lg,, which is computed
cos Lu, = 7/, = (gt,,, mm..mm..///.(l. mmmll + E),
(19)
where the parameter ş is introduced to avoid division by zero. The modified optimizer is referred to as Aida" (see Alg. 3 above for the complete update procedure). The only difference between Aida and Aida" the scaling of the layerwise vectors (mr,. (m.c..
We performed additional experiments by applying Aida* to train ResNet34 and LSTMs. The performance results are provided in Table 1T and 12 respectively. It is seen that the validation accuracies (the higher the better) for training ResNet34 over both CIFARIO and CIFAR100 are degraded slightly in comparison to Aida. Also, the test perplexities (the lower the better) for training 1-layer and 3-layer LSTMs are slightly degraded. The above experiments suggest that the parameter& in (10 and (9) in the mutual-vece vector projections does indeed make a difference in performance. our work, we recommend the implementation in Alg. [
18

---

Published in Transactions on Machine Learning Research (08/2023)
Table 11:
Performance comparison of Aida and Aida* for training ResNet34. The values in the table represent validation accuracies (the higher the better).
optimizers
CIFARIO
CIFARIO0
Aida (K = 2)
90.57+0..1
78.16
Aida" (K 2)
95.43:10.12
78.44410.34
Table 12:
Performance comparison of Aida and Aida" for training LSTMs. perplexity (the lower the better).
The values in the table represent test
optimizers
I-layer
2-layer
3-layer
Aida (K 2)
81.53
65.04
60.18
Aida" (K 81.58
64.85
60.43
F
Additional Plots of Learning Curves Versus Runtime
The two figures Fig. 5 and only plot the learning curves of Adam, AdaBelief, and Aida over epochs. Since Aida needs additional computational overhead compared to AdaBelief, it is interesting to also study the learning curves of Adam, AdaBelief, and Aida over runtime. Relevant results are shown in Fig. 7 and g It is clear that even though Aida requires a slightly longer learning time, the method produces significant validation performance gain for training both transformers and LSTMs.
(e): train perplexity runtime
(b): val. arc. ~ runtime
10
70
Adam
AdeRallel
65
10
AlcalI--2)
60
briin PPL
vol. 80c.
55
50
10a
45
40
Adam
AddBelal
10
35
Aktx(K=1)
Akin/K=2]
30
200
time (min)
time (min)
Figure 7: Visualisation of the learning curves versus runtime for Aida, AdaBelief, and Adam when training the Transformer. The maximum number of epochs is set to 400 in all three methods. The execution time for each epoch takes into account both the training and validation time.
1-layer LSTM
2-layer LSTM
3-layer LSTM
Adam
86
Adde|::91
100
test perplexay
TestPo perplextyy
80
Poxardin 1eet
75
06
75
20
70
86
66
80
0
500
1000
1500
2000
1000 2000 3000 4000 5000 time (a)
2000
4000
8000
8000
time
time
Figure& 8: Visualisation of the learning curves versus runtime for Aida, AdaBelief, and Adam when training LSTMs. The maximum number of epochs is set to 200 in all three methods. The execution time for each epoch takes into account both the training and validation time.
19


Reviews:
[{"review_id": "77sS1n322f", "reviewer": "Reviewer_CKQE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe proposed method is very easy to follow and I think the main difference is from the projection of g and m. \nThe idea about vector projections between the gradient g_t and first momentum m_t is also very interesting. If related work exists, the authors could provide some discussion in the revision.\nThe paper also evaluates the proposed in some different tasks, such as computer vision (image classification, image generalization) and NLP. \n\nWeakness:\n\nThe motivation is not very clear for me. For example, why we need to suppress the range of adaptive stepsizes. As you mentioned, Adbelief tried to solve this problem, and why you also try to solve it. What is the main problem in Adabelief you mainly focus on and try to solve.   \nSome claim need more explanation. For example, “The new method has the nice property that the adaptive stepsizes within each neural layer have smaller statistical variance” My question is why we hope the adaptive stepsizes in each layer has a smaller variance. As the adaptive stepsize for different parameters in the same layer are naturally different.\nSome related layer-wise optimizers have been proposed, and the authors could discuss these methods in the revision.\nThe improvement of Aida on ImageNet is not significant, and it's unclear whether the proposed method can work well in large-scale datasets. The authors could provide some discussion on this. Additionally, the reported accuracy of AdaBelief in this paper is lower than the result in the original paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel optimizer to reduce the range of adaptive stepsizes in each layer using a layerwise vector projection-based method. The authors evaluate the proposed method on various tasks, such as image classification, image generalization, and NLP, and demonstrate that it outperforms some popular optimizers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Typo: \nThe definition of g after Eq 2 : g = \\nabla f(\\theta)\n", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8STsR7LV7Y", "reviewer": "Reviewer_cU6d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed optimizer achieves better generalization and faster convergence in some cases.\n- The analysis of AdaBelief is interesting.\n\nWeakness:\n- The motivation behind the specific approach of iterative projection, proposed by the authors for their method Aida, is unclear. Specifically, after making the observation that the epsilon term in AdaBelief helps keep the adaptive step size range small (which helps generalization), the authors propose to perform this iterative projection of the g and m (gradient and the moving average of gradient respectively) onto each other. The resulting g and m are in the same direction as the original g or m (depending on the number of iterative projection steps). Importantly, the result of this iterative projection is that the scale of m and g becomes smaller in magnitude, which the authors also discuss in section 3.1. In fact, the iterative projections are computationally redundant, since the cosine of the angle between m and g during this iterative process remains unchanged. Therefore, I wonder why the authors chose to propose this iterative projection approach, as opposed to (say) scaling g and m using a scalar hyper-parameter multiplied with the cosine of the angle between g and m. That would've been both simpler and computationally cheaper.\n\n- My above point is further strengthened by the fact that the authors find that K=2 already yields good results (where K is the number of iterative projection steps). I understand that once advantage of using this iterative approach instead of directly specifying a scaling hyper-parameter is that K=2 is a much simpler choice over choosing a floating number as a hyper-parameter. If the authors choose to use this justification, then I suggest the following changes: \n 1. replace the dot product in the iterative projection step with the equivalent vector norms times cosine, which does not require redundant computation.\n 2. Explicitly mention the reasoning for proposing this approach as described above.\n Note that while a convergence analysis is provided for the proposed approach, it merely shows sufficiency, and does not imply that the proposed iterative approach is a necessary one.\n\n- In almost all plots, Aida converges faster than other methods. However, in figure 6a Aida has a much slower convergence initially. This was a bit concerning to me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the AdaBelief algorithm, specifically how a small constant added to the squared gradient moving average equation (middle eq in Eq 3) in AdaBelief significantly impacts optimization and generalization. Based on the gathered insight, the authors then propose an additional change to the AdaBelief algorithm, and call the new algorithm Aida. A convergence proof is shown for the proposed algorithm. Finally, the authors show experiments using Aida and previous optimizers and claim improved optimization and generalization performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see above\n\nIt would also be helpful to have a discussion on why reducing the adaptive step size range improves optimization and generalization.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OsdziB3x32", "reviewer": "Reviewer_17x9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The proposed modification to AdaBelief is well motivated, directly following the observation from previous work, that a compact adaptive step size range helps with improving the generalization.\n2. Empirical results across different tasks, models, and datasets show that the resulting method achieves consistent gain over existing optimizers, and also speedup the convergence. Yet I have some doubts regarding the hyperparameter choice of those experiments.\n\nWeakness:\n1. Lack of justification for the choice of hyperparameters for each optimizer in the experiments. While the hyper-parameters for each optimizer are provided in the appendix, it is not always clear to me why they are set that way, how they are selected, and whether their choice proposes fair comparison. For example, the epsilon of Aida for each task is different.; sometimes the epsilon for baselines are searched while the learning rates are fixed; The first momentum in WGAN training is different than others; The choice of which hyperparameters to fix and which hyperparameters to search misses explanation; The search grid of hyperparameters are different across different tasks. I would appreciate more elaboration on that.\n2. Non-negligible computational overhead. Since the computational overhead of Aida over Adabelief is 25%, it might be worthwhile to also plot the learning curve against runtime to account for that.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims at improving adaptive step size optimizers. Inspired by previous work that links compact adaptive step size range to better generalization, the paper proposes a modification to the AdaBelief that suppressed the range of its adaptive step size. It is achieved by adding a scaling term to both the momentum and gradient term when updating the state variable for the adaptive factor (denominator); the scaling term is computed using mutual projection between two vectors, which gradually reduces the length of each vector while preserving their angle. A convergence analysis is provided to support this change. Empirical evaluation is conducted on a range of tasks and models: image classification (VGG ResNet), generation (WGAN), and machine translation (Transformer, LSTM), where the proposed optimizer Aida achieves consistent improvement over baseline optimizers, with 25% computational overhead (on some tasks).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Include a discussion on the criteria of hyperparameter choice for each experiment.\n2. Also plot the learning curve against runtime to account for the 25% computation overhead.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was appreciated by the reviewers that still raised several questions. The authors replied to those questions well and the reviewer are now leaning toward an acceptance. But the authors discussed several interesting things with the reviewers that were not reported in the revision of the document so the final decision is "accept with minor revision" so that the authors can integrate those discussions (and references) in the final version of the paper (and supp).

For the next revision the authors must take into account all the comments from the reviewers and integrate to the paper the very interesting discussions raised by the reviewers. More details below:
+ (CKQE.1) the motivation detailed in the reply is very clear and should appear in the introduction.
+ (CKQE.2 and cU6d.3) This discussion about the variance is also important and should be in the paper, so should be the new reference used by teh authors in the discussion.
+ (CKQE.3 and 17x9.1) The authors must discuss more in details in the paper the choices of hyper-parameters and why they might not have the same performances as in the original papers. 
+ (cU6d.2) discussion about K=2 and cosine between the vector should be in the text and the new results should be provide in the supplementary.
+ (17x9.2) add the learning VS runtime curve as stated in the reply.



Decision: Accept (poster)