SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yun Yue *
Ant Group
Hangzhou, Zhejiang, China
yueyun. yy@antgroup.com
Zhiling Ye *
yezhiling.yzl@antgroup.com
Jiadi Jiang *
jiadi.jjd@antgroup.com
Yongchao Liu
yongchao.ly@antgroup.com
Ke Zhang
yingzi.zk@antgroup.com
Adaptive optimizers, such as Adam, have achieved remarkable success in deep learning. A key component of these optimizers is the so-called preconditioning matrix, providing enhanced gradient information and regulating the step size of each gradient direction. In this paper, we propose a novel approach to designing the preconditioning matrix by utilizing the gradient difference between two successive steps as the diagonal elements. These diagonal elements are closely related to the Hessian and can be perceived as an approximation of the inner product between the Hessian row vectors and difference of the adjacent parameter vectors. Additionally, we introduce an auto-switching function that enables the preconditioning matrix to switch dynamically between Stochastic Gradient Descent (SGD) and the adaptive optimizer. Based on these two techniques, we develop a new optimizer named AGD that enhances the generalization performance. We evaluate AGD on public datasets of Natural Language Processing (NLP), Computer Vision (CV), and Recommendation Systems (RecSys). Our experimental results demonstrate that AGD outperforms the state-of-the-art (SOTA) optimizers, achieving highly competitive or significantly better predictive performance. Furthermore, we analyze how AGD is able to switch automatically between SGD and the adaptive optimizer and its actual effects on various scenarios. The code is available at this link<sup>2</sup>.
Consider the following empirical risk minimization problems:
$$
\min  _ {\boldsymbol {w} \in \mathbb {R} ^ {n}} f (\boldsymbol {w}) := \frac {1}{M} \sum_ {k = 1} ^ {M} \ell (\boldsymbol {w}; \boldsymbol {x} _ {k}), \tag {1}
$$
where  $\pmb{w} \in \mathbb{R}^n$  is the parameter vector to optimize,  $\{\pmb{x}_1, \dots, \pmb{x}_M\}$  is the training set, and  $\ell(\pmb{w}; \pmb{x})$  is the loss function measuring the predictive performance of the parameter  $\pmb{w}$  on the example  $\pmb{x}$ . Since it is expensive to calculate the full batch gradient in each optimization iteration when  $M$  is large, the
*Co-first authors with equal contributions.
$^{2}$ https://github.com/intelligent-machine-learning/dlrover/tree/master/atorch/atorch/optimizers
standard approach is to adopt a mini-batched stochastic gradient, i.e.,
$$
\boldsymbol {g} (\boldsymbol {w}) = \frac {1}{| \mathcal {B} |} \sum_ {k \in \mathcal {B}} \nabla \ell (\boldsymbol {w}; \boldsymbol {x} _ {k}),
$$
where  $\mathcal{B} \subset \{1, \ldots, M\}$  is the sample set of size  $|\mathcal{B}| \ll M$ . Obviously, we have  $\mathbf{E}_{p(\boldsymbol{x})}[g(\boldsymbol{w})] = \nabla f(\boldsymbol{w})$  where  $p(\boldsymbol{x})$  is the distribution of the training data. Equation (1) is usually solved iteratively. Assume  $\boldsymbol{w}_t$  is already known and let  $\Delta \boldsymbol{w} = \boldsymbol{w}_{t+1} - \boldsymbol{w}_t$ , we have
$$
\begin{array}{l} \operatorname * {a r g   m i n} _ {\boldsymbol {w} _ {t + 1} \in \mathbb {R} ^ {n}} f (\boldsymbol {w} _ {t + 1}) = \operatorname * {a r g   m i n} _ {\Delta \boldsymbol {w} \in \mathbb {R} ^ {n}} f (\Delta \boldsymbol {w} + \boldsymbol {w} _ {t}) \\ \approx \underset {\Delta \boldsymbol {w} \in \mathbb {R} ^ {n}} {\arg \min } f (\boldsymbol {w} _ {t}) + (\Delta \boldsymbol {w}) ^ {T} \nabla f (\boldsymbol {w} _ {t}) + \frac {1}{2} (\Delta \boldsymbol {w}) ^ {T} \nabla^ {2} f (\boldsymbol {w} _ {t}) \Delta \boldsymbol {w} \tag {2} \\ \approx \underset {\Delta \pmb {w} \in \mathbb {R} ^ {n}} {\arg \min} f (\pmb {w} _ {t}) + (\Delta \pmb {w}) ^ {T} \pmb {m} _ {t} + \frac {1}{2 \alpha_ {t}} (\Delta \pmb {w}) ^ {T} B _ {t} \Delta \pmb {w}, \\ \end{array}
$$
where the first approximation is from Taylor expansion, and the second approximation are from  $\pmb{m}_t\approx \nabla f(\pmb {w}_t)$  ( $\pmb{m}_{t}$  denotes the weighted average of gradient  $\pmb{g}_t$ ) and  $\alpha_{t}\approx \frac{(\Delta\pmb{w})^{T}B_{t}\Delta\pmb{w}}{(\Delta\pmb{w})^{T}\nabla^{2}f(\pmb{w}_{t})\Delta\pmb{w}}$  ( $\alpha_{t}$  denotes the step size). By solving Equation (2), the general update formula is
$$
\boldsymbol {w} _ {t + 1} = \boldsymbol {w} _ {t} - \alpha_ {t} B _ {t} ^ {- 1} \boldsymbol {m} _ {t}, \quad t \in \{1, 2, \dots , T \}, \tag {3}
$$
where  $B_{t}$  is the so-called preconditioning matrix that adjusts updated velocity of variable  $\boldsymbol{w}_{t}$  in each direction. The majority of gradient descent algorithms can be succinctly encapsulated by Equation (3), ranging from the conventional second order optimizer, Gauss-Newton method, to the standard first-order optimizer, SGD, via different combinations of  $B_{t}$  and  $m_{t}$ . Table 1 summarizes different implementations of popular optimizers.
Intuitively, the closer  $B_{t}$  approximates the Hessian, the faster convergence rate the optimizer can achieve in terms of number of iterations, since the Gauss-Hessian method enjoys a quadratic rate, whereas the gradient descent converges linearly under certain conditions (Theorems 1.2.4, 1.2.5 in Nesterov [25]). However, computing the Hessian is computationally expensive for large models. Thus, it is essential to strike a balance between the degree of Hessian approximation and computational efficiency when designing the preconditioning matrix.
Table 1: Different optimizers by choosing different  $B_{t}$
Bt	Optimizer
Bt = H	GAUSS-HESSIAN
Bt ≈ H	BFGS [4, 13, 14, 31], LBFGS [5]
Bt ≈ diag(H)	ADAHESSIAN [34]
Bt = F	NATURAL GRADIENT [2]
Bt2≈ Femp	SHAMPOO [16]
Bt2≈ diag(Femp)	ADAGRAD [12], ADADELTA [35], ADAM [18], ADAMW [21], AMSGRAD [28]
Bt2≈ diag(Var(gt))	ADABELIEF [39]
Bt = Π	SGD [29], MOMENTUM [27]
$\mathbf{H}$  is the Hessian.  $\mathbf{F}$  is the Fisher information matrix.  $\mathbf{F}_{emp}$  is the empirical Fisher information matrix.
In this paper, we propose the AGD (Auto-switchable optimizer with Gradient Difference of adjacent steps) optimizer based on the idea of efficiently and effectively acquiring the information of the Hessian. The diagonal entries of AGD's preconditioning matrix are computed as the difference of gradients between two successive iterations, serving as an approximation of the inner product between the Hessian row vectors and difference of parameter vectors. In addition, AGD is equipped with an adaptive switching mechanism that automatically toggles its preconditioning matrix between SGD and the adaptive optimizer, governed by a threshold hyperparameter  $\delta$  which enables AGD adaptive to various scenarios. Our contributions can be summarized as follows.
- We present a novel optimizer called AGD, which efficiently and effectively integrates the information of the Hessian into the preconditioning matrix and switches dynamically between SGD and the adaptive optimizer. We establish theoretical results of convergence guarantees for both non-convex and convex stochastic settings.
- We validate AGD on six public datasets: two from NLP (IWSLT14 [6] and PTB [23]), two from CV (Cifar10 [19] and ImageNet [30]), and the rest two from RecSys (Criteo [11] and Avazu [3]). The experimental results suggest that AGD is on par with or outperforms the SOTA optimizers.
- We analyze how AGD is able to switch automatically between SGD and the adaptive optimizer, and assess the effect of hyperparameter  $\delta$  which controls the auto-switch process in different scenarios.
We use lowercase letters to denote scalars, boldface lowercase to denote vectors, and uppercase letters to denote matrices. We employ subscripts to denote a sequence of vectors, e.g.,  $\boldsymbol{x}_1, \ldots, \boldsymbol{x}_t$  where  $t \in [T] := \{1, 2, \ldots, T\}$ , and one additional subscript is used for specific entry of a vector, e.g.,  $x_{t,i}$  denotes  $i$ -th element of  $\boldsymbol{x}_t$ . For any vectors  $\boldsymbol{x}, \boldsymbol{y} \in \mathbb{R}^n$ , we write  $\boldsymbol{x}^T \boldsymbol{y}$  or  $\boldsymbol{x} \cdot \boldsymbol{y}$  for the standard inner product,  $\boldsymbol{x} \boldsymbol{y}$  for element-wise multiplication,  $\boldsymbol{x} / \boldsymbol{y}$  for element-wise division,  $\sqrt{\boldsymbol{x}}$  for element-wise square root,  $\boldsymbol{x}^2$  for element-wise square, and  $\max(\boldsymbol{x}, \boldsymbol{y})$  for element-wise maximum. Similarly, any operator performed between a vector  $\boldsymbol{x} \in \mathbb{R}^n$  and a scalar  $c \in \mathbb{R}$ , such as  $\max(\boldsymbol{x}, c)$ , is also element-wise. We denote  $\| \boldsymbol{x} \| = \| \boldsymbol{x} \|_2 = \sqrt{\langle \boldsymbol{x}, \boldsymbol{x} \rangle}$  for the standard Euclidean norm,  $\| \boldsymbol{x} \|_1 = \sum_i |x_i|$  for the  $\ell_1$  norm, and  $\| \boldsymbol{x} \|_{\infty} = \max_i |x_i|$  for the  $\ell_{\infty}$ -norm, where  $x_i$  is the  $i$ -th element of  $\boldsymbol{x}$ .
Let  $f_{t}(\pmb{w})$  be the loss function of the model at  $t$ -step where  $\pmb{w} \in \mathbb{R}^n$ . We consider  $\pmb{m}_{t}$  as Exponential Moving Averages (EMA) of  $\pmb{g}_{t}$  throughout this paper, i.e.,
$$
\boldsymbol {m} _ {t} = \beta_ {1} \boldsymbol {m} _ {t - 1} + (1 - \beta_ {1}) \boldsymbol {g} _ {t} = (1 - \beta_ {1}) \sum_ {i = 1} ^ {t} \boldsymbol {g} _ {t - i + 1} \beta_ {1} ^ {i - 1}, t \geq 1, \tag {4}
$$
where  $\beta_{1}\in [0,1)$  is the exponential decay rate.
ASGD [36] leverages Taylor expansion to estimate the gradient at the global step in situations where the local worker's gradient is delayed, by analyzing the relationship between the gradient difference and Hessian. To approximate the Hessian, the authors utilize the diagonal elements of empirical Fisher information due to the high computational and spatial overhead of Hessian. ADABELIEF [39] employs the EMA of the gradient as the predicted gradient and adapts the step size by scaling it with the difference between predicted and observed gradients, which can be considered as the variance of the gradient.
Hybrid optimization methods, including ADABOUND [22] and SWATS [17], have been proposed to enhance generalization performance by switching an adaptive optimizer to SGD. ADABOUND utilizes learning rate clipping on ADAM, with upper and lower bounds that are non-increasing and non-decreasing functions, respectively. One can show that it ultimately converges to the learning rate of SGD. Similarly, SWATS also employs the clipping method, but with constant upper and lower bounds.
3 Algorithm
3.1 Details of AGD optimizer
Algorithm 1 AGD
1: Input: parameters  $\beta_{1},\beta_{2},\delta ,\pmb{w}_{1}\in \mathbb{R}^{n}$  , step size  $\alpha_{t}$  , initialize  $\pmb {m}_0 = \mathbf{0},\pmb {b}_0 = \mathbf{0}$
2: for  $t = 1$  to  $T$  do
3:  $\pmb{g}_t = \nabla f_t(\pmb{w}_t)$
4:  $\pmb{m}_t\gets \beta_1\pmb{m}_{t - 1} + (1 - \beta_1)\pmb{g}_t$
5:  $\mathbf{s}_t = \left\{ \begin{array}{ll} \frac{\mathbf{m}_1}{1 - \beta_1} & t = 1 \\ \frac{\mathbf{m}_t}{1 - \beta_1^t} - \frac{\mathbf{m}_{t-1}}{1 - \beta_1^{t-1}} & t > 1 \end{array} \right.$
6:  $\pmb{b}_{t} \gets \beta_{2}\pmb{b}_{t-1} + (1 - \beta_{2})\pmb{s}_{t}^{2}$
7:  $\pmb{w}_{t + 1} = \pmb{w}_t - \alpha_t\frac{\sqrt{1 - \beta_2^t}}{1 - \beta_1^t}\frac{\pmb{m}_t}{\max(\sqrt{\pmb{b}_t},\delta\sqrt{1 - \beta_2^t})}$
8: end for
Figure 1: Trajectories of AGD and Adam in the Beale function.
Algorithm 1 summarizes our AGD algorithm. The design of AGD comes from two parts: gradient difference and auto switch for faster convergence and better generalization performance across tasks.
Gradient difference Our motivation stems from how to efficiently and effectively integrate the information of the Hessian into the preconditioning matrix. Let  $\Delta \pmb{w} = \pmb{w}_t - \pmb{w}_{t-1}$  and  $\nabla_i f$  denote the  $i$ -th element of  $\nabla f$ . From Taylor expansion or the mean value theorem, when  $\|\Delta \pmb{w}\|$  is small, we have the following approximation,
$$
\nabla_ {i} f (\boldsymbol {w} _ {t}) - \nabla_ {i} f (\boldsymbol {w} _ {t - 1}) \approx \nabla \nabla_ {i} f (\boldsymbol {w} _ {t}) \cdot \Delta \boldsymbol {w}.
$$
It means the difference of gradients between adjacent steps can be an approximation of the inner product between the Hessian row vectors and the difference of two successive parameter vectors. To illustrate the effectiveness of gradient difference in utilizing Hessian information, we compare the convergence trajectories of AGD and Adam on the Beale function. As shown in Figure 1, we see that AGD converges much faster than Adam; when AGD reaches the optimal point, Adam has only covered about half of the distance. We select the two most representative points on the AGD trajectory in the figure, the maximum and minimum points of  $\| \nabla f \|_1 / \| \mathrm{diag}(H) \|_1$ , to illustrate how AGD accelerates convergence by utilizing Hessian information. At the maximum point (A), where the gradient is relatively large and the curvature is relatively small ( $\| \nabla f \|_1 = 22.3$ ,  $\| \mathrm{diag}(H) \|_1 = 25.3$ ), the step size of AGD is 1.89 times that of Adam. At the minimum point (B), where the gradient is relatively small and the curvature is relatively large ( $\| \nabla f \|_1 = 0.2$ ,  $\| \mathrm{diag}(H) \|_1 = 34.8$ ), the step size decreases to prevent it from missing the optimal point during the final convergence phase.
To approximate  $\nabla f(\pmb{w}_t)$ , we utilize  $\pmb{m}_t / (1 - \beta_1^t)$  instead of  $\pmb{g}_t$ , as the former provides an unbiased estimation of  $\nabla f(\pmb{w}_t)$  with lower variance. According to Kingma and Ba [18], we have  $\mathbf{E}\left[\frac{\pmb{m}_t}{1 - \beta_1^t}\right] \cong \mathbf{E}[\pmb{g}_t]$ , where the equality is satisfied if  $\{\pmb{g}_t\}$  is stationary. Additionally, assuming  $\{\pmb{g}_t\}$  is strictly stationary and  $\mathbf{Cov}(\pmb{g}_i, \pmb{g}_j) = 0$  if  $i \neq j$  for simplicity and  $\beta_1 \in (0,1)$ , we observe that
$$
\mathbf {V a r} \left[ \frac {\boldsymbol {m} _ {t}}{1 - \beta_ {1} ^ {t}} \right] = \frac {1}{(1 - \beta_ {1} ^ {t}) ^ {2}} \mathbf {V a r} \left[ (1 - \beta_ {1}) \sum_ {i = 1} ^ {t} \beta_ {1} ^ {t - i} \boldsymbol {g} _ {i} \right] = \frac {(1 + \beta_ {1} ^ {t}) (1 - \beta_ {1})}{(1 - \beta_ {1} ^ {t}) (1 + \beta_ {1})} \mathbf {V a r} [ \boldsymbol {g} _ {t} ] <   \mathbf {V a r} [ \boldsymbol {g} _ {t} ].
$$
Now, we denote
$$
\boldsymbol {s} _ {t} = \left\{ \begin{array}{l l} \boldsymbol {m} _ {1} / (1 - \beta_ {1}) & t = 1, \\ \boldsymbol {m} _ {t} / (1 - \beta_ {1} ^ {t}) - \boldsymbol {m} _ {t - 1} / (1 - \beta_ {1} ^ {t - 1}) & t > 1, \end{array} \right.
$$
and design the preconditioning matrix  $B_{t}$  satisfying
$$
B _ {t} ^ {2} = \mathrm {d i a g} (\mathrm {E M A} (\pmb {s} _ {1} \pmb {s} _ {1} ^ {T}, \pmb {s} _ {2} \pmb {s} _ {2} ^ {T}, \dots , \pmb {s} _ {t} \pmb {s} _ {t} ^ {T})) / (1 - \beta_ {2} ^ {t}),
$$
where  $\beta_{2}$  represents the parameter of EMA and bias correction is achieved via the denominator.
Note that previous research, such as the one discussed in Section 2 by Zheng et al. [36], has acknowledged the correlation between the difference of two adjacent gradients and the Hessian. However, the key difference is that they did not employ this relationship to construct an optimizer. In contrast, our approach presented in this paper leverages this relationship to develop an optimizer, and its effectiveness has been validated in the experiments detailed in Section 4.
Auto switch Typically a small value is added to  $B_{t}$  for numerical stability, resulting in  $B_{t} + \epsilon \mathbb{I}$ . However, in this work we propose to replace this with  $\max(B_{t}, \delta \mathbb{I})$ , where we use a different notation  $\delta$  to emphasize its crucial role in auto-switch mechanism. In contrast to  $\epsilon$ ,  $\delta$  can be a relatively large (such as 1e-2). If the element of  $\hat{\boldsymbol{b}}_t \coloneqq \sqrt{\boldsymbol{b}_t / (1 - \beta_2^t)}$  exceeds  $\delta$ , AGD (Line 7 of Algorithm 1) takes a confident adaptive step. Otherwise, the update is performed using EMA, i.e.,  $m_t$ , with a constant scale of  $\alpha_t / (1 - \beta_1^t)$ , similar to SGD with momentum. It's worth noting that, AGD can automatically switch modes on a per-parameter basis as the training progresses.
Compared to the commonly used additive method, AGD effectively eliminates the noise generated by  $\epsilon$  during adaptive updates. In addition, AGD offers an inherent advantage of being able to generalize across different tasks by tuning the value of  $\delta$ , obviating the need for empirical choices among a plethora of optimizers.
3.2 Comparison with other optimizers
Comparison with AdaBound As noted in Section 2, the auto-switch bears similarities to AdaBound [22] in its objective to enhance the generalization performance by switching to SGD using the clipping method. Nonetheless, the auto-switch's design differs significantly from AdaBound. Rather than relying solely on adaptive optimization in the early stages, AGD has the flexibility to switch seamlessly between stochastic and adaptive methods, as we will demonstrate in Section 4.5. In addition, AGD outperforms AdaBound's across various tasks, as we will show in Appendix A.2.
Figure 2: Comparison of stability between AGD and AdaBelief relative to the parameter  $\delta$  (or  $\epsilon$ ) for ResNet32 on Cifar10. AGD shows better stability over a wide range of  $\delta$  (or  $\epsilon$ ) variations than AdaBelief.
Comparison with AdaBelief While in principle our design is fundamentally different from that of AdaBelief [39], which approximates gradient variance with its preconditioning matrix, we do see some similarities in our final forms. Compared with the denominator of AGD, the denominator of AdaBelief  $s_t = g_t - m_t = \frac{\beta_1}{1 - \beta_1} (m_t - m_{t-1})$  lacks bias correction for the subtracted terms and includes a multiplication factor of  $\frac{\beta_1}{1 - \beta_1}$ . In addition, we observe that AGD exhibits superior stability compared to AdaBelief. As shown in Figure 2, when the value of  $\epsilon$  deviates from 1e-8 by orders of magnitude, the performance of AdaBelief degrades significantly; in contrast, AGD maintains good stability over a wide range of  $\delta$  variations.
3.3 Numerical analysis
In this section, we present a comparison between AGD and several SOTA optimizers on three test functions. We use the parameter settings from Zhuang et al. [39], where the learning rate is set to 1e-3 for all adaptive optimizers, along with the same default val
ues of  $\epsilon$  (or  $\delta$ ) (1e-8) and betas  $(\beta_{1} = 0.9, \beta_{2} = 0.999)$ . For SGD, we set the momentum to 0.9 and the learn
ing rate to 1e-6 to ensure numerical stability. As shown in Figure 3, AGD exhibits promising results by firstly reaching the optimal points in all experiments before other competitors. In addition, we conduct a search of the largest learning rate for each optimizer with respect to the Beale function, and AGD once again stands out as the strongest method. More details on the search process can be found in Appendix A.3.
(a)  $f(x,y)$
(b) Beale
(c) Rosenbrock
Figure 3: Trajectories of different optimizers in three test functions, where  $f(x,y) = (x + y)^2 + (x - y)^2 / 10$ . We also provide animated versions at https://youtu.be/Qv5X3v5YUw0.
4.1 Experiment setup
We extensively compared the performance of various optimizers on diverse learning tasks in NLP, CV, and RecSys; we only vary the settings for the optimizers and keep the other settings consistent in this evaluation. To offer a comprehensive analysis, we provide a detailed description of each task and the optimizers' efficacy in different application domains.
NLP: We conduct experiments using Language Modeling (LM) on Penn TreeBank [23] and Neural Machine Translation (NMT) on IWSLT14 German-to-English (De-En) [6] datasets. For the LM task,
we train 1, 2, and 3-layer LSTM models with a batch size of 20 for 200 epochs. For the NMT task, we implement the Transformer small architecture, and employ the same pre-processing method and settings as AdaHessian [34], including a length penalty of 1.0, beam size of 5, and max tokens of 4096. We train the model for 55 epochs and average the last 5 checkpoints for inference. We maintain consistency in our learning rate scheduler and warm-up steps. Table 2 provides complete details of the experimental setup.
CV: We conduct experiments using ResNet20 and ResNet32 on the Cifar10 [19] dataset, and ResNet18 on the ImageNet [30] dataset, as detailed in Table 2. It is worth noting that the number of parameters of ResNet18 is significantly larger
Table 2: Experiments setup.
Task	Dataset	Model	Train	Val/Test	Params
NLP-LM	PTB	1-layer LSTM	0.93M	730K/82K	5.3M
2-layer LSTM	13.6M
3-layer LSTM	24.2M
NLP-NMT	IWSLT14 De-En	Transformer small1	153K	7K/7K	36.7M
CV	Cifar10	ResNet20/ResNet32	50K	10K	0.27M/0.47M
ImageNet	ResNet18	1.28M	50K	11.69M
RecSys	Avazu	MLP	36.2M	4.2M	151M
Criteo	DCN	39.4M	6.6M	270M
than that of ResNet20/32, stemming from inconsistencies in ResNet's naming conventions. Within the ResNet architecture, the consistency in filter sizes, feature maps, and blocks is maintained only within specific datasets. Originally proposed for ImageNet, ResNet18 is more complex compared to ResNet20 and ResNet32, which were tailored for the less demanding Cifar10 dataset. Our training process involves 160 epochs with a learning rate decay at epochs 80 and 120 by a factor of 10 for Cifar10, and 90 epochs with a learning rate decay every 30 epochs by a factor of 10 for ImageNet. The batch size for both datasets is set to 256.
RecSys: We conduct experiments on two widely used datasets, Avazu [3] and Criteo [11], which contain logs of display ads. The goal is to predict the Click Through Rate (CTR). We use the samples from the first nine days of Avazu for training and the remaining samples for testing. We employ the Multilayer Perceptron (MLP) structure (a fundamental architecture used in most deep CTR models). The model maps each categorical feature into a 16-dimensional embedding vector, followed by four fully connected layers of dimensions 64, 32, 16, and 1, respectively. For Criteo, we use the first 6/7 of all samples as the training set and last 1/7 as the test set. We adopt the Deep & Cross Network (DCN) [32] with an embedding size of 8, along with two deep layers of size 64 and two cross layers. Detailed summary of the specifications can be found in Table 2. For both datasets, we train them for one epoch using a batch size of 512.
Optimizers to compare include SGD [29], Adam [18], AdamW [21], AdaBelief [39] and AdaHessian [34]. To determine each optimizer's hyperparameters, we adopt the parameters suggested in the literature of AdaHessian and AdaBelief when the experimental settings are identical. Otherwise, we perform hyperparameter searches for optimal settings. A detailed description of this process can be found in Appendix A.1. For our NLP and CV experiments, we utilize GPUs with the PyTorch framework [26], while our RecSys experiments are conducted with three parameter servers and five workers in the TensorFlow framework [1]. To ensure the reliability of our results, we execute each experiment five times with different random seeds and calculate statistical results.
4.2 NLP
We report the perplexity (PPL, lower is better) and case-insensitive BiLinguual Evaluation Understudy (BLEU, higher is better) score on test set for LM and NMT tasks, respectively. The results are shown in Table 3. For the LM task on PTB, AGD achieves the lowest PPL in all 1,2,3-layer LSTM experiments, as demonstrated in Figure 4. For the NMT task on IWSLT14, AGD is on par with AdaBelief, but outperforms the other optimizers.
4.3 CV
Table 4 reports the top-1 accuracy for different optimizers when trained on Cifar10 and ImageNet. It is remarkable that AGD outperforms other optimizers on both Cifar10 and ImageNet. The test accuracy  $([\mu \pm \sigma])$  curves of different optimizers for ResNet20/32 on Cifar10 and ResNet18 on