SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Test PPL and BLEU score for LM and NMT tasks. † is reported in AdaHessian [34].
Dataset	PTB	IWSLT14
Metric	PPL, lower is better	BLEU, higher is better
Model	1-layer LSTM	2-layer LSTM	3-layer LSTM	Transformer
SGD	85.36 ± .34 (−4.13)	67.26 ± .17 (−1.42)	63.68 ± .17 (−2.79)	28.57 ± .15†(+7.37)
Adam	84.50 ± .16 (−3.27)	67.01 ± .11 (−1.17)	64.45 ± .26 (−3.56)	32.93 ± .26 (+3.01)
AdamW	88.16 ± .19 (−6.93)	95.25 ± 1.33 (−29.41)	102.61 ± 1.13 (−41.72)	35.82 ± .06 (+0.12)
AdaBelief	84.40 ± .21 (−3.17)	66.69 ± .23 (−0.85)	61.34 ± .11 (−0.45)	35.93 ± .08 (+0.01)
AdaHessian	88.62 ± .15 (−7.39)	73.37 ± .22 (−7.53)	69.51 ± .19 (−8.62)	35.79 ± .06†(+0.15)
AGD	81.23 ± .17	65.84 ± .18	60.89 ± .09	35.94 ± .11
(a) 1-layer
(b) 2-layer
(c) 3-layer
Figure 4: Test PPL  $([\mu \pm \sigma ])$  on Penn Treebank for 1,2,3-layer LSTM.
(a) ResNet20 on Cifar10
Figure 5: Test accuracy  $([\mu \pm \sigma ])$  of different optimizers for ResNet20/32 on Cifar10 and ResNet18 on ImageNet.
(b) ResNet32 on Cifar10
(c) ResNet18 on ImageNet
ImageNet are illustrated in Figure 5. Notice that the numbers of SGD and AdaHessian on ImageNet are lower than the numbers reported in original papers [7, 34], which were run only once (we average multiple trials here). AdaHessian can achieve  $70.08\%$  top-1 accuracy in Yao et al. [34] while we report  $69.57 \pm 0.12\%$ . Due to the limited training details provided in Yao et al. [34], it is difficult for us to explain the discrepancy. However, regardless of which result of AdaHessian is taken, AGD outperforms AdaHessian significantly. Our reported top-1 accuracy of SGD is  $69.94 \pm 0.10\%$ , slightly lower than  $70.23\%$  reported in Chen et al. [7]. We find that the differences in training epochs, learning rate scheduler and weight decay rate are the main reasons. We also run the experiment using the same configuration as in Chen et al. [7], and AGD can achieve  $70.45\%$  accuracy at  $\mathrm{lr} = 4\mathrm{e} - 4$  and  $\delta = 1\mathrm{e} - 5$ , which is still better than the  $70.23\%$  result reported in Chen et al. [7].
We also report the accuracy of AGD for ResNet18 on Cifar10 for comparing with the SOTA results  $^{3}$ , which is listed in Appendix A.5. Here we clarify again the ResNet naming confusion. The test accuracy of ResNet18 on Cifar10 training with AGD is above  $95\%$ , while ResNet32 is about  $93\%$  since ResNet18 is much more complex than ResNet32.
4.4 RecSys
To evaluate the accuracy of CTR estimation, we have adopted the Area Under the receiver-operator Curve (AUC) as our evaluation criterion, which is widely recognized as a reliable measure [15]. As stated in Cheng et al. [10], Wang et al. [32], Ling et al. [20], Zhu et al. [38], even an absolute
<sup>3</sup>https://paperswithcode.com/sota/stochastic-optimization-on-cifar-10-resnet-18
Table 4: Top-1 accuracy for different optimizers when trained on Cifar10 and ImageNet.
Dataset	Cifar10	ImageNet
Model	ResNet20	ResNet32	ResNet18
SGD	92.14 ± .14 (+0.21)	93.10 ± .07 (+0.02)	69.94 ± .10 (+0.41)
Adam	90.46 ± .20 (+1.89)	91.54 ± .12 (+1.58)	64.03 ± .16 (+6.32)
AdamW	92.12 ± .14 (+0.23)	92.72 ± .01 (+0.40)	69.11 ± .17 (+1.24)
AdaBelief	92.19 ± .15 (+0.16)	92.90 ± .13 (+0.22)	70.20 ± .03 (+0.15)
AdaHessian	92.27 ± .27 (+0.08)	92.91 ± .14 (+0.21)	69.57 ± .12 (+0.78)
AGD	92.35 ± .24	93.12 ± .18	70.35 ± .17
Table 5: Test AUC for different optimizers when trained on Avazu and Criteo.
Dataset	Avazu	Criteo
Model	MLP	DCN
SGD	0.7463 ± .0005 (+1.7%)	0.7296 ± .0067 (+72.7%)
Adam	0.7458 ± .0010 (+2.2%)	0.8023 ± .0002 (+0.0%)
AdaBelief	0.7467 ± .0009 (+1.3%)	0.8022 ± .0002 (+0.1%)
AdaHessian	0.7434 ± .0006 (+4.6%)	0.8004 ± .0005 (+1.9%)
AGD	0.7480 ± .0008	0.8023 ± .0004
improvement of  $1 \text{‰}$ in AUC can be considered practically significant given the difficulty of improving CTR prediction. Our experimental results in Table 5 indicate that AGD can achieve highly competitive or significantly better performance when compared to other optimizers. In particular, on the Avazu task, AGD outperforms all other optimizers by more than  $1 \text{‰}$ . On the Criteo task, AGD performs better than SGD and AdaHessian, and achieves comparable performance to Adam and AdaBelief.
4.5 The effect of  $\delta$
(a) ResNet18 on ImageNet
Figure 6: The distribution of  $\hat{b}_t$  on different epochs/steps. The colored area denotes the ratio of  $\hat{b}_t$  in the corresponding interval. The values of  $\delta$  for ResNet18 on ImageNet, Transformer on IWSLT14, 2-layer LSTM on PTB, and DCN on Criteo are 1e-5, 1e-14, 1e-5 and 1e-8, respectively.
(b) Transformer on IWSLT14
(c) 2-layer LSTM on PTB
(d) DCN on Criteo
In this section, we aim to provide a comprehensive analysis of the impact of  $\delta$  on the training process by precisely determining the percentage of  $\hat{b}_t$  that Algorithm 1 truncates. To this end, Figure 6 shows the distribution of  $\hat{b}_t$  across various tasks under the optimal configuration that we have identified. The black dot on the figure provides the precise percentage of  $\hat{b}_t$  that  $\delta$  truncates during the training process. Notably, a lower percentage indicates a higher degree of SGD-like updates compared to adaptive steps, which can be adjusted through  $\delta$ .
As SGD with momentum generally outperforms adaptive optimizers on CNN tasks [34, 39], we confirm this observation as illustrated in Figure 6a: AGD behaves more like SGD during the initial stages of training (before the first learning rate decay at the 30th epoch) and switches to adaptive optimization for fine-tuning. Figure 6b indicates that the parameters taking adaptive updates are dominant, as expected because adaptive optimizers such as AdamW are preferred in transformers. Figure 6c demonstrates that most parameters update stochastically, which explains why AGD has a similar curve to SGD in Figure 4b before the 100th epoch. The proportion of parameters taking adaptive updates grows from  $3\%$  to  $5\%$  afterward, resulting in a better PPL in the fine-tuning stage. Concerning Figure 6d, the model of the RecSys task trains for only one epoch, and AGD gradually switches to adaptive updates for a better fit to the data.
4.6 Computational cost
We train a Transformer small model for IWSLT14 on a single NVIDIA P100 GPU. AGD is comparable to the widely used AdamW optimizer, while significantly outperforms AdaHessian in terms of memory footprint and training speed. As a result, AGD can be a drop-in replacement for AdamW with similar computation cost and better generalization performance.
Table 6: Computational cost for Transformer small.
Optimizer	Memory	Time per Epoch	Relative time to AdamW
SGD	5119 MB	230 s	0.88×
AdamW	5413 MB	260 s	1.00×
AdaHessian	8943 MB	750 s	2.88×
AGD	5409 MB	278 s	1.07×
Using the framework developed in Reddi et al. [28], Yang et al. [33], Chen et al. [9], Zhou et al. [37], we have the following theorems that provide the convergence in non-convex and convex settings. Particularly, we use  $\beta_{1,t}$  to replace  $\beta_{1}$ , where  $\beta_{1,t}$  is non-increasing with respect to  $t$ .
Theorem 1. (Convergence in non-convex settings) Suppose that the following assumptions are satisfied:
1.  $f$  is differential and lower bounded, i.e.,  $f(\pmb{w}^{*}) > -\infty$  where  $\pmb{w}^{*}$  is an optimal solution.  $f$  is also  $L$ -smooth, i.e.,  $\forall \pmb{u}, \pmb{v} \in \mathbb{R}^n$ , we have  $f(\pmb{u}) \leq f(\pmb{v}) + \langle \nabla f(\pmb{v}), \pmb{u} - \pmb{v} \rangle + \frac{L}{2} \| \pmb{u} - \pmb{v} \|^2$ .
2. At step  $t$ , the algorithm can access a bounded noisy gradient and the true gradient is bounded, i.e.,  $\| \pmb{g}_t\|_\infty \leq G_\infty$ ,  $\| \nabla f(\pmb{w}_t)\|_\infty \leq G_\infty$ ,  $\forall t \in [T]$ . Without loss of generality, we assume  $G_\infty \geq \delta$ .
3. The noisy gradient is unbiased and the noise is independent, i.e.,  $\pmb{g}_t = \nabla f(\pmb{w}_t) + \pmb{\zeta}_t$ ,  $\mathbf{E}[\pmb{\zeta}_t] = \mathbf{0}$  and  $\pmb{\zeta}_i$  is independent of  $\pmb{\zeta}_j$  if  $i \neq j$ .
4.  $\alpha_{t} = \alpha /\sqrt{t},\beta_{1,t}$  is non-increasing satisfying  $\beta_{1,t}\leq \beta_1\in [0,1),\beta_2\in [0,1)$  and  $b_{t,i}\leq$ $b_{t + 1,i}\forall i\in [n].$
Then Algorithm 1 yields
$$
\min  _ {t \in [ T ]} \mathbf {E} [ \| \nabla f (\boldsymbol {w} _ {t}) \| ^ {2} ] <   C _ {3} \frac {1}{\sqrt {T} - \sqrt {2}} + C _ {4} \frac {\log T}{\sqrt {T} - \sqrt {2}} + C _ {5} \frac {\sum_ {t = 1} ^ {T} \hat {\alpha} _ {t} \left(\beta_ {1 , t} - \beta_ {1 , t + 1}\right)}{\sqrt {T} - \sqrt {2}}, \tag {5}
$$
where  $C_3, C_4$  and  $C_5$  are defined as follows:
$$
C _ {3} = \frac {G _ {\infty}}{\alpha (1 - \beta_ {1}) ^ {2} (1 - \beta_ {2}) ^ {2}} \left(f (\boldsymbol {w} _ {1}) - f (\boldsymbol {w} ^ {*}) + \frac {n G _ {\infty} ^ {2} \alpha}{(1 - \beta_ {1}) ^ {8} \delta^ {2}} (\delta + 8 L \alpha) + \frac {\alpha \beta_ {1} n G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {3} \delta}\right),
$$
$$
C _ {4} = \frac {1 5 L n G _ {\infty} ^ {3} \alpha}{2 (1 - \beta_ {2}) ^ {2} (1 - \beta_ {1}) ^ {1 0} \delta^ {2}}, C _ {5} = \frac {n G _ {\infty} ^ {3}}{\alpha (1 - \beta_ {1}) ^ {5} (1 - \beta_ {2}) ^ {2} \delta}.
$$
The proof of Theorem 1 is presented in Appendix B. There are two important points that should be noted: Firstly, in assumption 2, we can employ the gradient norm clipping technique to ensure the upper bound of the gradients. Secondly, in assumption 4,  $b_{t,i} \leq b_{t+1,i} \forall i \in [n]$ , which is necessary for the validity of Theorems 1 and 2, may not always hold. To address this issue, we can implement the AMSGrad condition [28] by setting  $\boldsymbol{b}_{t+1} = \max(\boldsymbol{b}_{t+1}, \boldsymbol{b}_t)$ . However, this may lead to a potential decrease in the algorithm's performance in practice. The more detailed analysis is provided in Appendix A.4. From Theorem 1, we have the following corollaries.
Corollary 1. Suppose  $\beta_{1,t} = \beta_1 / \sqrt{t}$ , we have
$$
\min  _ {t \in [ T ]} \mathbf {E} \left[ \| \nabla f (\boldsymbol {w} _ {t}) \| ^ {2} \right] <   C _ {3} \frac {1}{\sqrt {T} - \sqrt {2}} + C _ {4} \frac {\log T}{\sqrt {T} - \sqrt {2}} + \frac {C _ {5} \alpha}{1 - \beta_ {1}} \frac {\log T + 1}{\sqrt {T} - \sqrt {2}},
$$
where  $C_3, C_4$  and  $C_5$  are the same with Theorem 1.
The proof of Corollary 1 can be found in Appendix C.
Corollary 2. Suppose  $\beta_{1,t} = \beta_1$ ,  $\forall t\in [T]$ , we have
$$
\min  _ {t \in [ T ]} \mathbf {E} \left[ \| \nabla f (\boldsymbol {w} _ {t}) \| ^ {2} \right] <   C _ {3} \frac {1}{\sqrt {T} - \sqrt {2}} + C _ {4} \frac {\log T}{\sqrt {T} - \sqrt {2}},
$$
where  $C_3$  and  $C_4$  are the same with Theorem 1.
Corollaries 1 and 2 imply the convergence (to the stationary point) rate for AGD is  $O(\log T / \sqrt{T})$  in non-convex settings.
Theorem 2. (Convergence in convex settings) Let  $\{\pmb{w}_t\}$  be the sequence obtained by AGD (Algorithm 1),  $\alpha_{t} = \alpha /\sqrt{t}$ ,  $\beta_{1,t}$  is non-increasing satisfying  $\beta_{1,t}\leq \beta_1\in [0,1)$ ,  $\beta_{2}\in [0,1)$ ,  $b_{t,i}\leq b_{t + 1,i}\forall i\in [n]$  and  $\| \pmb {g}_t\|_\infty \leq G_\infty ,\forall t\in [T]$ . Suppose  $f_{t}(\pmb {w})$  is convex for all  $t\in [T]$ ,  $\pmb{w}^{*}$  is an optimal solution of  $\sum_{t = 1}^{T}f_{t}(\pmb {w})$ , i.e.,  $\pmb{w}^{*} = \arg \min_{\pmb {w}\in \mathbb{R}^{n}}\sum_{t = 1}^{T}f_{t}(\pmb {w})$  and there exists the constant  $D_{\infty}$  such that  $\max_{t\in [T]}\| \pmb {w}_t - \pmb {w}^*\|_\infty \leq D_\infty$ . Then we have the following bound on the regret
$$
\sum_ {t = 1} ^ {T} \left(f _ {t} \left(\boldsymbol {w} _ {t}\right) - f _ {t} \left(\boldsymbol {w} ^ {*}\right)\right) <   \frac {1}{1 - \beta_ {1}} \left(C _ {1} \sqrt {T} + \sum_ {t = 1} ^ {T} \frac {\beta_ {1 , t}}{2 \hat {\alpha} _ {t}} n D _ {\infty} ^ {2} + C _ {2} \sqrt {T}\right),
$$
where  $C_1$  and  $C_2$  are defined as follows:
$$
C _ {1} = \frac {n (2 G _ {\infty} + \delta) D _ {\infty} ^ {2}}{2 \alpha \sqrt {1 - \beta_ {2}} (1 - \beta_ {1}) ^ {2}}, \quad C _ {2} = \frac {n \alpha G _ {\infty} ^ {2}}{(1 - \beta_ {1}) ^ {3}} \left(1 + \frac {1}{\delta \sqrt {1 - \beta_ {2}}}\right).
$$
The proof of Theorem 2 is given in Appendix D. To ensure that the condition  $\max_{t\in [T]}\| \pmb{w}_t - \pmb{w}^*\|_\infty \leq D_\infty$  holds, we can assume that the domain  $\mathcal{W} \subseteq \mathbb{R}^n$  is bounded and project the sequence  $\{\pmb{w}_t\}$  onto  $\mathcal{W}$  by setting  $\pmb{w}_{t + 1} = \Pi_{\mathcal{W}}\left(\pmb{w}_t - \alpha_t\frac{\sqrt{1 - \beta_2^t}}{1 - \beta_1^t}\frac{\pmb{m}_t}{\max(\sqrt{\pmb{b}_t},\delta\sqrt{1 - \beta_2^t})}\right)$ . From Theorem 2, we have the following corollary.
Corollary 3. Suppose  $\beta_{1,t} = \beta_1 / t$ , we have
$$
\sum_ {t = 1} ^ {T} \left(f _ {t} \left(\boldsymbol {w} _ {t}\right) - f _ {t} \left(\boldsymbol {w} ^ {*}\right)\right) <   \frac {1}{1 - \beta_ {1}} \left(C _ {1} \sqrt {T} + \frac {n D _ {\infty} ^ {2} \beta_ {1}}{\alpha \sqrt {1 - \beta_ {2}}} \sqrt {T} + C _ {2} \sqrt {T}\right),
$$
where  $C_1$  and  $C_2$  are the same with Theorem 2.
The proof of Corollary 3 is given in Appendix E. Corollary 3 implies the regret is  $O(\sqrt{T})$  and can achieve the convergence rate  $O(1 / \sqrt{T})$  in convex settings.
In this paper, we introduce a novel optimizer, AGD, which incorporates the Hessian information into the preconditioning matrix and allows seamless switching between SGD and the adaptive optimizer. We provide theoretical convergence rate proofs for both non-convex and convex stochastic settings and conduct extensive empirical evaluations on various real-world datasets. The results demonstrate that AGD outperforms other optimizers in most cases, resulting in significant performance improvements. Additionally, we analyze the mechanism that enables AGD to automatically switch between stochastic and adaptive optimization and investigate the impact of the hyperparameter  $\delta$  for this process.
We thank Yin Lou for his meticulous review of our manuscript and for offering numerous valuable suggestions.
[1] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In Kimberly Keeton and Timothy Roscoe, editors, 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016, pages 265-283. USENIX Association, 2016.
[2] Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Comput., 10(2):251-276, 1998. doi: 10.1162/089976698300017746.
[38] Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, and Xiuqiang He. Open benchmarking for click-through rate prediction. In Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong, editors, CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, pages 2759-2769. ACM, 2021. doi: 10.1145/3459637.3482486. URL https://doi.org/10.1145/3459637.3482486.
[39] Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C. Tatikonda, Nicha C. Dvornek, Xenophon Papademetris, and James S. Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
A Details of experiments
A.1 Configuration of optimizers
In this section, we provide a thorough description of the hyperparameters used for different optimizers across various tasks. For optimizers other than AGD, we adopt the recommended parameters for the identical experimental setup as indicated in the literature of AdaHessian [34] and AdaBelief [39]. In cases where these recommendations are not accessible, we perform a hyperparameter search to determine the optimal hyperparameters.
The weight decay is set to 1e-4 (1.2e-6) for all optimizers in the NMT (LM) task. For adaptive optimizers, we set  $(\beta_{1},\beta_{2})$  to (0.9, 0.98) in the NMT task and (0.9, 0.999) in the LM task. For the LM task, the general dropout rate is set to 0.4.
- SGD/Adam/AdamW: We adopt the same experimental setup in AdaHessian [34]. For SGD, the initial learning rate is 0.1 and the momentum is set to 0.9. For Adam, the initial learning rate is set to 0.001 and the epsilon is set to 1e-8. For AdamW, the initial learning rate is set to 0.005 and the epsilon is set to 1e-8.
- AdaBelief: We explore the best learning rate for ResNet20/32 on CIFar10 and ResNet18 on ImageNet, respectively. Finally, the initial learning rate is set to be 0.01 for ResNet20 on CIFar10 and 0.005 for ResNet32/ResNet18 on CIFar10/ImageNet. The epsilon is set to 1e-8.
- AdaHessian: We use the recommended configuration as much as possible from AdaHessian [34]. The Hessian power is set to 1. The initial learning rate is 0.15 when training on both CIFar10 and ImageNet, as recommended in Yao et al. [34]. The epsilon is set to 1e-4.
- AGD: We conduct a grid search of  $\delta$  and the learning rate. The choice of  $\delta$  is among {1e-8, 1e-7, 1e-6, 1e-5, 1e-4, 1e-3, 1e-2} and the search range for the learning rate is from 1e-4 to 1e-2. Finally, we choose the learning rate to 0.007 and  $\delta$  to 1e-2 for Cifar10 task, and learning rate to 0.0004 and  $\delta$  to 1e-5 for ImageNet task.
The weight decay for all optimizers is set to 0.0005 on Cifar10 and 0.0001 on ImageNet.  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$  are for all adaptive optimizers.
4https://github.com/juntang-zhuang/Adabelief-Optimizer
<sup>5</sup>https://github.com/amirgholami/adahessian
RecSys Note that we implement the optimizers for training on our internal distributed environment.
- SGD: We search for the learning rate among  $\{1\mathrm{e} - 4,1\mathrm{e} - 3,1\mathrm{e} - 2,0.1,1\}$  and choose the best results (0.1 for the Avazu task and 1e-3 for the Criteo task).
- Adam/AdaBelief: We search the learning rate among  $\{1\mathrm{e} - 5,1\mathrm{e} - 4,1\mathrm{e} - 3,1\mathrm{e} - 2\}$  and the epsilon among  $\{1\mathrm{e} - 16,$  1e-14, 1e-12, 1e-10, 1e-8, 1e-6\}. For the Avazu task, the best learning rate/epsilon is  $1\mathrm{e} - 4 / 1\mathrm{e} - 8$  for Adam and  $1\mathrm{e} - 4 / 1\mathrm{e} - 16$  for AdaBelief. For the Criteo task, the best learning rate/epsilon is  $1\mathrm{e} - 3 / 1\mathrm{e} - 8$  for Adam and  $1\mathrm{e} - 3 / 1\mathrm{e} - 16$  for AdaBelief.
- AdaHessian: We search the learning rate among  $\{1\mathrm{e} - 5,1\mathrm{e} - 4,1\mathrm{e} - 3,1\mathrm{e} - 2\}$  and the epsilon among  $\{1\mathrm{e} - 16,$ $1\mathrm{e} - 14,1\mathrm{e} - 12,1\mathrm{e} - 10,1\mathrm{e} - 8,1\mathrm{e} - 6\}$ . The best learning rate/epsilon is  $1\mathrm{e} - 4 / 1\mathrm{e} - 8$  for the Avazu task and  $1\mathrm{e} - 3 / 1\mathrm{e} - 6$  for the Criteo task. The block size and the Hessian power are set to 1.
- AGD: We search the learning rate among  $\{1\mathrm{e} - 5,1\mathrm{e} - 4,1\mathrm{e} - 3\}$  and  $\delta$  among  $\{1\mathrm{e} - 12,1\mathrm{e} - 10,1\mathrm{e} - 8,1\mathrm{e} - 6,1\mathrm{e} - 4,$ $1\mathrm{e} - 2\}$ . The best learning rate/  $\delta$  is  $1\mathrm{e} - 4 / 1\mathrm{e} - 4$  for the Avazu task and  $1\mathrm{e} - 4 / 1\mathrm{e} - 8$  for the Criteo task.
$\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$  are for all adaptive optimizers.
A.2 AGD vs. AdaBound
Table 7: The performance of AGD and AdaBound across different tasks.
Optimizer	3-Layer LSTM, test PPL (lower is better)	ResNet18 on ImageNet, Top-1 accuracy
AdaBound	63.60 [39]	68.13 (100 epochs) [8]
AGD	60.89 (better)	70.19 (90 epochs, still better)
A.3 Numerical Experiments
In our numerical experiments, we employ the same learning rate across optimizers. While a larger learning rate could accelerate convergence, as shown in Figures 7a and 7b, we also note that it could lead to unstable training. To investigate the largest learning rate that each optimizer could handle for the Beale function, we perform a search across the range of  $\{1\mathrm{e} - 5, 1\mathrm{e} - 4, \dots, 1, 10\}$ . The optimization trajectories are displayed in Figure 7c, and AGD demonstrates slightly superior performance. Nonetheless, we argue that the learning rate selection outlined in Section 3.3 presents a more appropriate representation.
(a) Adam
(b) AGD
Figure 7: Optimization trajectories on Beale function using various learning rates.
(c) Comparison
A.4 AGD with AMSGrad condition
Algorithm 2 summarizes the AGD optimizer with AMSGrad condition. The AMSGrad condition is usually used to ensure the convergence. We empirically show that adding AMSGrad condition to AGD will slightly degenerate AGD's performance, as listed in Table 8. For AGD with AMSGrad condition, we search for the best learning rate among  $\{0.001, 0.003, 0.005, 0.007, 0.01\}$  and best  $\delta$  within  $\{1e-2, 1e-4, 1e-6, 1e-8\}$ . We find the best learning rate is 0.007 and best  $\delta$  is  $1e-2$ , which are the same as AGD without AMSGrad condition.
A.5 AGD for ResNet18 on Cifar10
Since the SOTA accuracy  ${}^{6}$  for ResNet18 on Cifar10 is 95.55% using SGD optimizer with a cosine learning rate schedule, we also test the performance of AGD for ResNet18 on Cifar10. We find AGD can achieve 95.79%
$^{6}$ https://paperswithcode.com/sota/stochastic-optimization-on-cifar-10-resnet-18
Algorithm 2 AGD with AMSGrad condition
1: Input: parameters  $\beta_{1},\beta_{2},\delta ,w_{1}\in \mathbb{R}^{n}$  , step size  $\alpha_{t}$  , initialize  $m_0 = 0,b_0 = 0$
2: for  $t = 1$  to  $T$  do
3:  $\pmb{g}_t = \nabla f_t(\pmb{w}_t)$
4:  $\pmb{m}_t\gets \beta_1\pmb{m}_{t - 1} + (1 - \beta_1)\pmb{g}_t$
5:  $\mathbf{s}_t = \left\{ \begin{array}{ll} \mathbf{m}_1 / (1 - \beta_1) & t = 1 \\ \mathbf{m}_t / (1 - \beta_1^t) - \mathbf{m}_{t-1} / (1 - \beta_1^{t-1}) & t > 1 \end{array} \right.$
6:  $\pmb{b}_t \gets \beta_2 \pmb{b}_{t-1} + (1 - \beta_2) \pmb{s}_t^2$
7:  $\pmb{b}_t \gets \max(\pmb{b}_t, \pmb{b}_{t-1})$  // AMSGrad condition
8:  $\pmb{w}_{t+1} = \pmb{w}_t - \alpha_t \frac{\sqrt{1 - \beta_2^t}}{1 - \beta_1^t} \frac{\pmb{m}_t}{\max(\sqrt{\pmb{b}_t}, \delta \sqrt{1 - \beta_2^t})}$
9: end for
Table 8: Top-1 accuracy for AGD with and without AMSGrad condition when trained with ResNet20 on Cifar10.
Optimizer	AGD	AGD + AMSGrad
Accuracy	92.35 ± .24	92.25 ± 0.11
accuracy at  $\mathrm{lr} = 0.001$  and  $\delta = 1\mathrm{e - }2$  when using the same training configuration as the experiment of SGD in Moreau et al. [24], which exceeds the current SOTA result.
A.6 Robustness to hyperparameters
We test the performance of AGD and Adam with respect to  $\delta$  (or  $\epsilon$ ) and learning rate. The experiments are performed with ResNet20 on Cifar10 and the results are shown in Figure 8. Compared to Adam, AGD shows better robustness to the change of hyperparameters.
(a) AGD
(b) Adam
(c) AGD
(d) Adam
Figure 8: Test accuracy of ResNet20 on Cifar10, trained with AGD and Adam using different  $\delta$  (or  $\epsilon$ ) and learning rate. For (a) and (b), we choose learning rate as 0.007 and 0.001, respectively. For (c) and (d), we set  $\delta$  (or  $\epsilon$ ) to be 1e-2 and 1e-8, respectively.
B Proof of Theorem 1
Proof. Denote  $\hat{\alpha}_t = \alpha_t\frac{\sqrt{1 - \beta_2^t}}{1 - \beta_{1,t}^t}$  and  $\pmb{v}_{t} = \max (\sqrt{\pmb{b}_{t}},\delta \sqrt{1 - \beta_{2}^{t}})$ , we have the following lemmas.
Lemma 1. For the parameter settings and assumptions in Theorem 1 or Theorem 2, we have
$$
\hat {\alpha} _ {t} > \hat {\alpha} _ {t + 1}, t \in [ T ],
$$
where  $\hat{\alpha}_t = \alpha_t\frac{\sqrt{1 - \beta_2^t}}{1 - \beta_{1,t}^t}$