Title: AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix
Venue: NeurIPS 2023
Accepted: yes
Keywords: adaptive optimizer

gradient difference

auto switch

AGD
Abstract: Adaptive optimizers, such as Adam, have achieved remarkable success in deep learning. A key component of these optimizers is the so-called preconditioning matrix, providing enhanced gradient information and regulating the step size of each gradient direction. In this paper, we propose a novel approach to designing the preconditioning matrix by utilizing the gradient difference between two successive steps as the diagonal elements. These diagonal elements are closely related to the Hessian and can be perceived as an approximation of the inner product between the Hessian row vectors and difference of the adjacent parameter vectors. Additionally, we introduce an auto-switching function that enables the preconditioning matrix to switch dynamically between Stochastic Gradient Descent (SGD) and the adaptive optimizer. Based on these two techniques, we develop a new optimizer named AGD that enhances the generalization performance. We evaluate AGD on public datasets of Natural Language Processing (NLP), Computer Vision (CV), and Recommendation Systems (RecSys). Our experimental results demonstrate that AGD outperforms the state-of-the-art (SOTA) optimizers, achieving highly competitive or significantly better predictive performance. Furthermore, we analyze how AGD is able to switch automatically between SGD and the adaptive optimizer and its actual effects on various scenarios. The code is available at https://github.com/intelligent-machine-learning/dlrover/tree/master/atorch/atorch/optimizers.

AGD: an Auto-switchable Optimizer using Stepwise Gradient Difference for Preconditioning Matrix

Abstract

Adaptive optimizers, such as Adam, have achieved remarkable success in deep learning. A key component of these optimizers is the so-called preconditioning matrix, providing enhanced gradient information and regulating the step size of each gradient direction. In this paper, we propose a novel approach to designing the preconditioning matrix by utilizing the gradient difference between two successive steps as the diagonal elements. These diagonal elements are closely related to the Hessian and can be perceived as an approximation of the inner product between the Hessian row vectors and difference of the adjacent parameter vectors. Additionally, we introduce an auto-switching function that enables the preconditioning matrix to switch dynamically between Stochastic Gradient Descent (SGD) and the adaptive optimizer. Based on these two techniques, we develop a new optimizer named AGD that enhances the generalization performance. We evaluate AGD on public datasets of Natural Language Processing (NLP), Computer Vision (CV), and Recommendation Systems (RecSys). Our experimental results demonstrate that AGD outperforms the state-of-the-art (SOTA) optimizers, achieving highly competitive or significantly better predictive performance. Furthermore, we analyze how AGD is able to switch automatically between SGD and the adaptive optimizer and its actual effects on various scenarios. The code is available at this link 2 .

1 Introduction

Consider the following empirical risk minimization problems:

where w ∈ R n is the parameter vector to optimize, { x 1 , . . . , x M } is the training set, and glyph[lscript] ( w ; x ) is the loss function measuring the predictive performance of the parameter w on the example x . Since it is expensive to calculate the full batch gradient in each optimization iteration when M is large, the

standard approach is to adopt a mini-batched stochastic gradient, i.e.,

where B ⊂ { 1 , . . . , M } is the sample set of size |B| glyph[lessmuch] M . Obviously, we have E p ( x ) [ g ( w )] = ∇ f ( w ) where p ( x ) is the distribution of the training data. Equation (1) is usually solved iteratively. Assume w t is already known and let ∆ w = w t +1 -w t , we have

where the first approximation is from Taylor expansion, and the second approximation are from m t ≈ ∇ f ( w t ) ( m t denotes the weighted average of gradient g t ) and α t ≈ (∆ w ) T B t ∆ w (∆ w ) T ∇ 2 f ( w t )∆ w ( α t denotes the step size). By solving Equation (2), the general update formula is

where B t is the so-called preconditioning matrix that adjusts updated velocity of variable w t in each direction. The majority of gradient descent algorithms can be succinctly encapsulated by Equation (3), ranging from the conventional second order optimizer, Gauss-Newton method, to the standard first-order optimizer, SGD, via different combinations of B t and m t . Table 1 summarizes different implementations of popular optimizers.

Intuitively, the closer B t approximates the Hessian, the faster convergence rate the optimizer can achieve in terms of number of iterations, since the Gauss-Hessian method enjoys a quadratic rate, whereas the gradient descent converges linearly under certain conditions (Theorems 1.2.4, 1.2.5 in Nesterov [25]). However, computing the Hessian is computationally expensive for large models. Thus, it is essential to strike a balance between the degree of Hessian approximation and computational efficiency when designing the preconditioning matrix.

H is the Hessian. F is the Fisher information matrix. F emp is the empirical Fisher information matrix.

In this paper, we propose the AGD ( A uto-switchable optimizer with G radient D ifference of adjacent steps) optimizer based on the idea of efficiently and effectively acquiring the information of the Hessian. The diagonal entries of AGD's preconditioning matrix are computed as the difference of gradients between two successive iterations, serving as an approximation of the inner product between the Hessian row vectors and difference of parameter vectors. In addition, AGD is equipped with an adaptive switching mechanism that automatically toggles its preconditioning matrix between SGD and the adaptive optimizer, governed by a threshold hyperparameter δ which enables AGD adaptive to various scenarios. Our contributions can be summarized as follows.

We present a novel optimizer called AGD, which efficiently and effectively integrates the information of the Hessian into the preconditioning matrix and switches dynamically between SGD and the adaptive optimizer. We establish theoretical results of convergence guarantees for both non-convex and convex stochastic settings.

We validate AGD on six public datasets: two from NLP (IWSLT14 [6] and PTB [23]), two from CV (Cifar10 [19] and ImageNet [30]), and the rest two from RecSys (Criteo [11] and Avazu [3]). The experimental results suggest that AGD is on par with or outperforms the SOTA optimizers.

We analyze how AGD is able to switch automatically between SGD and the adaptive optimizer, and assess the effect of hyperparameter δ which controls the auto-switch process in different scenarios.

Notation

We use lowercase letters to denote scalars, boldface lowercase to denote vectors, and uppercase letters to denote matrices. We employ subscripts to denote a sequence of vectors, e.g., x 1 , . . . , x t where t ∈ [ T ] := { 1 , 2 , . . . , T } , and one additional subscript is used for specific entry of a vector, e.g., x t,i denotes i -th element of x t . For any vectors x , y ∈ R n , we write x T y or x · y for the standard inner product, xy for element-wise multiplication, x / y for element-wise division, √ x for element-wise square root, x 2 for element-wise square, and max( x , y ) for element-wise maximum. Similarly, any operator performed between a vector x ∈ R n and a scalar c ∈ R , such as max( x , c ) , is also element-wise. We denote ‖ x ‖ = ‖ x ‖ 2 = √ 〈 x , x 〉 for the standard Euclidean norm, ‖ x ‖ 1 = ∑ i | x i | for the glyph[lscript] 1 norm, and ‖ x ‖ ∞ = max i | x i | for the glyph[lscript] ∞ -norm, where x i is the i -th element of x .

Let f t ( w ) be the loss function of the model at t -step where w ∈ R n . We consider m t as Exponential Moving Averages (EMA) of g t throughout this paper, i.e.,

where β 1 ∈ [0 , 1) is the exponential decay rate.

2 Related work

ASGD [36] leverages Taylor expansion to estimate the gradient at the global step in situations where the local worker's gradient is delayed, by analyzing the relationship between the gradient difference and Hessian. To approximate the Hessian, the authors utilize the diagonal elements of empirical Fisher information due to the high computational and spatial overhead of Hessian. ADABELIEF [39] employs the EMA of the gradient as the predicted gradient and adapts the step size by scaling it with the difference between predicted and observed gradients, which can be considered as the variance of the gradient.

Hybrid optimization methods, including ADABOUND [22] and SWATS [17], have been proposed to enhance generalization performance by switching an adaptive optimizer to SGD. ADABOUND utilizes learning rate clipping on ADAM, with upper and lower bounds that are non-increasing and non-decreasing functions, respectively. One can show that it ultimately converges to the learning rate of SGD. Similarly, SWATS also employs the clipping method, but with constant upper and lower bounds.

3 Algorithm

3.1 Details of AGD optimizer

Algorithm 1 AGD

1: Input: parameters β 1 , β 2 , δ , w 1 ∈ R n , step size α t , initialize m 0 = 0 , b 0 = 0

2: for t = 1 to T do

3: g t = ∇ f t ( w t )

4:

m

t

β

1

m

t

-

1

+(1

-

β

1

)

g

t

6:

b

t

7:

w

t

+1

8: end for

4

2

0

2

4

4

3

2

1

0

1

2

3

4

A

B

Optimal

Adam

AGD

Algorithm 1 summarizes our AGD algorithm. The design of AGD comes from two parts: gradient difference and auto switch for faster convergence and better generalization performance across tasks.

β

2

=

b

t

-

1

+(1

-

β

√

α

-

1

1

2

β

-

β

t

1

)

t

2

m

t

b

t

,δ

s

2

t

w

t

-

t

√

√

-

1

β

t

2

)

Gradient difference Our motivation stems from how to efficiently and effectively integrate the information of the Hessian into the preconditioning matrix. Let ∆ w = w t -w t -1 and ∇ i f denote the i -th element of ∇ f . From Taylor expansion or the mean value theorem, when ‖ ∆ w ‖ is small, we have the following approximation,

It means the difference of gradients between adjacent steps can be an approximation of the inner product between the Hessian row vectors and the difference of two successive parameter vectors. To illustrate the effectiveness of gradient difference in utilizing Hessian information, we compare the convergence trajectories of AGD and Adam on the Beale function. As shown in Figure 1, we see that AGD converges much faster than Adam; when AGD reaches the optimal point, Adam has only covered about half of the distance. We select the two most representative points on the AGD trajectory in the figure, the maximum and minimum points of ‖∇ f ‖ 1 / ‖ diag ( H ) ‖ 1 , to illustrate how AGDaccelerates convergence by utilizing Hessian information. At the maximum point (A), where the gradient is relatively large and the curvature is relatively small ( ‖∇ f ‖ 1 = 22 . 3 , ‖ diag ( H ) ‖ 1 = 25 . 3 ), the step size of AGD is 1.89 times that of Adam. At the minimum point (B), where the gradient is relatively small and the curvature is relatively large ( ‖∇ f ‖ 1 = 0 . 2 , ‖ diag ( H ) ‖ 1 = 34 . 8 ), the step size decreases to prevent it from missing the optimal point during the final convergence phase.

glyph[negationslash]

To approximate ∇ f ( w t ) , we utilize m t / (1 -β t 1 ) instead of g t , as the former provides an unbiased estimation of ∇ f ( w t ) with lower variance. According to Kingma and Ba [18], we have E [ m t 1 -β t 1 ] glyph[approxorequal] E [ g t ] , where the equality is satisfied if { g t } is stationary. Additionally, assuming { g t } is strictly stationary and Cov ( g i , g j ) = 0 if i = j for simplicity and β 1 ∈ (0 , 1) , we observe that

Now, we denote

and design the preconditioning matrix B t satisfying

where β 2 represents the parameter of EMA and bias correction is achieved via the denominator.

Note that previous research, such as the one discussed in Section 2 by Zheng et al. [36], has acknowledged the correlation between the difference of two adjacent gradients and the Hessian. However, the key difference is that they did not employ this relationship to construct an optimizer. In contrast, our approach presented in this paper leverages this relationship to develop an optimizer, and its effectiveness has been validated in the experiments detailed in Section 4.

Auto switch Typically a small value is added to B t for numerical stability, resulting in B t + glyph[epsilon1] I . However, in this work we propose to replace this with max( B t , δ I ) , where we use a different notation δ to emphasize its crucial role in auto-switch mechanism. In contrast to glyph[epsilon1] , δ can be a relatively large (such as 1e-2). If the element of ˆ b t := √ b t / (1 -β t 2 ) exceeds δ , AGD (Line 7 of Algorithm 1) takes a confident adaptive step. Otherwise, the update is performed using EMA, i.e., m t , with a constant scale of α t / (1 -β t 1 ) , similar to SGD with momentum. It's worth noting that, AGD can automatically switch modes on a per-parameter basis as the training progresses.

Compared to the commonly used additive method, AGD effectively eliminates the noise generated by glyph[epsilon1] during adaptive updates. In addition, AGD offers an inherent advantage of being able to generalize across different tasks by tuning the value of δ , obviating the need for empirical choices among a plethora of optimizers.

3.2 Comparison with other optimizers

Comparison with AdaBound As noted in Section 2, the auto-switch bears similarities to AdaBound [22] in its objective to enhance the generalization performance by switching to SGD using the clipping method. Nonetheless, the auto-switch's design differs significantly from AdaBound. Rather than relying solely on adaptive optimization in the early stages, AGD has the flexibility to switch seamlessly between stochastic and adaptive methods, as we will demonstrate in Section 4.5. In addition, AGD outperforms AdaBound's across various tasks, as we will show in Appendix A.2.

10

-15

10

-13

10

-11

10

-9

10

-7

10

-5

δ

(or

/epsilon1

)

60

65

70

75

80

85

90

95

100

Test accuracy (%)

AGD, lr=0.001

AGD, lr=0.005

AGD, lr=0.01

AdaBelief, lr=0.001

AdaBelief, lr=0.005

AdaBelief, lr=0.01

3.3 Numerical analysis

In this section, we present a comparison between AGD and several SOTA optimizers on three test functions. We use the parameter settings from Zhuang et al. [39], where the learning rate is set to 1e-3 for all adaptive optimizers, along with the same default values of glyph[epsilon1] ( or δ ) (1e-8) and betas ( β 1 = 0 . 9 , β 2 = 0 . 999 ). For SGD, we set the momentum to 0.9 and the learn-

Comparison with AdaBelief While in principle our design is fundamentally different from that of AdaBelief [39], which approximates gradient variance with its preconditioning matrix, we do see some similarities in our final forms. Compared with the denominator of AGD, the denominator of AdaBelief s t = g t -m t = β 1 1 -β 1 ( m t -m t -1 ) lacks bias correction for the subtracted terms and includes a multiplication factor of β 1 1 -β 1 . In addition, we observe that AGD exhibits superior stability compared to AdaBelief. As shown in Figure 2, when the value of glyph[epsilon1] deviates from 1e-8 by orders of magnitude, the performance of AdaBelief degrades significantly; in contrast, AGD maintains good stability over a wide range of δ variations.

3

2

1

0

1

2

3

3

2

1

0

1

2

3

Optimal

AGD

SGD + Momentum

Adam

AdaBelief

AdaHessian

(a)

f

(

x, y

)

4

2

0

2

4

4

3

2

1

0

1

2

3

4

Optimal

AGD

SGD + Momentum

Adam

AdaBelief

AdaHessian

(b) Beale

6

4

2

0

2

4

6

6

4

2

0

2

4

6

Optimal

AGD

SGD + Momentum

Adam

AdaBelief

AdaHessian

(c) Rosenbrock

ing rate to 1e-6 to ensure numerical stability. As shown in Figure 3, AGD exhibits promising results by firstly reaching the optimal points in all experiments before other competitors. In addition, we conduct a search of the largest learning rate for each optimizer with respect to the Beale function, and AGD once again stands out as the strongest method. More details on the search process can be found in Appendix A.3.

4 Experiments

4.1 Experiment setup

We extensively compared the performance of various optimizers on diverse learning tasks in NLP, CV, and RecSys; we only vary the settings for the optimizers and keep the other settings consistent in this evaluation. To offer a comprehensive analysis, we provide a detailed description of each task and the optimizers' efficacy in different application domains.

NLP: We conduct experiments using Language Modeling (LM) on Penn TreeBank [23] and Neural Machine Translation (NMT) on IWSLT14 German-to-English (De-En) [6] datasets. For the LM task,

we train 1, 2, and 3-layer LSTM models with a batch size of 20 for 200 epochs. For the NMT task, we implement the Transformer small architecture, and employ the same pre-processing method and settings as AdaHessian [34], including a length penalty of 1.0, beam size of 5, and max tokens of 4096. We train the model for 55 epochs and average the last 5 checkpoints for inference. We maintain consistency in our learning rate scheduler and warm-up steps. Table 2 provides complete details of the experimental setup.

CV: We conduct experiments using ResNet20 and ResNet32 on the Cifar10 [19] dataset, and ResNet18 on the ImageNet [30] dataset, as detailed in Table 2. It is worth noting that the number of parameters of ResNet18 is significantly larger

than that of ResNet20/32, stemming from inconsistencies in ResNet's naming conventions. Within the ResNet architecture, the consistency in filter sizes, feature maps, and blocks is maintained only within specific datasets. Originally proposed for ImageNet, ResNet18 is more complex compared to ResNet20 and ResNet32, which were tailored for the less demanding Cifar10 dataset. Our training process involves 160 epochs with a learning rate decay at epochs 80 and 120 by a factor of 10 for Cifar10, and 90 epochs with a learning rate decay every 30 epochs by a factor of 10 for ImageNet. The batch size for both datasets is set to 256.

RecSys: We conduct experiments on two widely used datasets, Avazu [3] and Criteo [11], which contain logs of display ads. The goal is to predict the Click Through Rate (CTR). We use the samples from the first nine days of Avazu for training and the remaining samples for testing. We employ the Multilayer Perceptron (MLP) structure (a fundamental architecture used in most deep CTR models). The model maps each categorical feature into a 16-dimensional embedding vector, followed by four fully connected layers of dimensions 64, 32, 16, and 1, respectively. For Criteo, we use the first 6/7 of all samples as the training set and last 1/7 as the test set. We adopt the Deep & Cross Network (DCN) [32] with an embedding size of 8, along with two deep layers of size 64 and two cross layers. Detailed summary of the specifications can be found in Table 2. For both datasets, we train them for one epoch using a batch size of 512.

Optimizers to compare include SGD [29], Adam [18], AdamW [21], AdaBelief [39] and AdaHessian [34]. To determine each optimizer's hyperparameters, we adopt the parameters suggested in the literature of AdaHessian and AdaBelief when the experimental settings are identical. Otherwise, we perform hyperparameter searches for optimal settings. A detailed description of this process can be found in Appendix A.1. For our NLP and CV experiments, we utilize GPUs with the PyTorch framework [26], while our RecSys experiments are conducted with three parameter servers and five workers in the TensorFlow framework [1]. To ensure the reliability of our results, we execute each experiment five times with different random seeds and calculate statistical results.

4.2 NLP

We report the perplexity (PPL, lower is better) and case-insensitive BiLingual Evaluation Understudy (BLEU, higher is better) score on test set for LM and NMT tasks, respectively. The results are shown in Table 3. For the LM task on PTB, AGD achieves the lowest PPL in all 1,2,3-layer LSTM experiments, as demonstrated in Figure 4. For the NMT task on IWSLT14, AGD is on par with AdaBelief, but outperforms the other optimizers.

4.3 CV

Table 4 reports the top-1 accuracy for different optimizers when trained on Cifar10 and ImageNet. It is remarkable that AGD outperforms other optimizers on both Cifar10 and ImageNet. The test accuracy ( [ µ ± σ ] ) curves of different optimizers for ResNet20/32 on Cifar10 and ResNet18 on

Dataset

PTB

IWSLT14

Metric

PPL, lower is better

BLEU, higher is better

Model

1-layer LSTM

2-layer LSTM

3-layer LSTM

Transformer

SGD

85

.

36

±

.

34

(

-

4

.

13

)

67

.

26

±

.

17

(

-

1

.

42

)

63

.

68

±

.

17

(

-

2

.

79

)

28

.

57

±

.

15

†(

+7

.

37

)

Adam

84

.

50

±

.

16

(

-

3

.

27

)

67

.

01

±

.

11

(

-

1

.

17

)

64

.

45

±

.

26

(

-

3

.

56

)

32

.

93

±

.

26

(

+3

.

01

)

AdamW

88

.

16

±

.

19

(

-

6

.

93

)

95

.

25

±

1

.

33

(

-

29

.

41

)

102

.

61

±

1

.

13

(

-

41

.

72

)

35

.

82

±

.

06

(

+0

.

12

)

AdaBelief

84

.

40

±

.

21

(

-

3

.

17

)

66

.

69

±

.

23

(

-

0

.

85

)

61

.

34

±

.

11

(

-

0

.

45

)

35

.

93

±

.

08

(

+0

.

01

)

AdaHessian

88

.

62

±

.

15

(

-

7

.

39

)

73

.

37

±

.

22

(

-

7

.

53

)

69

.

51

±

.

19

(

-

8

.

62

)

35

.

79

±

.

06

†(

+0

.

15

)

AGD

81

.

23

±

.

17

65

.

84

±

.

18

60

.

89

±

.

09

35

.

94

±

.

11

0

25

50

75

100

125

150

175

200

Epoch

80

85

90

95

100

105

Test PPL

SGD

Adam

AdamW

AdaBelief

AdaHessian

AGD

(a) 1-layer

0

25

50

75

100

125

150

175

200

Epoch

65

70

75

80

85

90

95

100

Test PPL

SGD

Adam

AdamW

AdaBelief

AdaHessian

AGD

(b) 2-layer

0

25

50

75

100

125

150

175

200

Epoch

60

65

70

75

80

85

90

95

100

105

Test PPL

SGD

Adam

AdamW

AdaBelief

AdaHessian

AGD

(c) 3-layer

0

20

40

60

80

100

120

140

160

Epoch

0.4

0.5

0.6

0.7

0.8

0.9

Test accuracy

120

140

160

0.915

0.920

0.925

SGD

Adam

AdamW

AdaBelief

AdaHessian

AGD

(a) ResNet20 on Cifar10

0

20

40

60

80

100

120

140

160

Epoch

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test accuracy

120

140

160

0.925

0.930

SGD

Adam

AdamW

AdaBelief

AdaHessian

AGD

(b) ResNet32 on Cifar10

0

20

40

60

80

Epoch

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Test accuracy

60

80

0.68

0.70

SGD

Adam

AdamW

AdaHessian

AdaBelief

AGD

(c) ResNet18 on ImageNet

ImageNet are illustrated in Figure 5. Notice that the numbers of SGD and AdaHessian on ImageNet are lower than the numbers reported in original papers [7, 34], which were run only once (we average multiple trials here). AdaHessian can achieve 70 . 08% top-1 accuracy in Yao et al. [34] while we report 69 . 57 ± 0 . 12% . Due to the limited training details provided in Yao et al. [34], it is difficult for us to explain the discrepancy. However, regardless of which result of AdaHessian is taken, AGD outperforms AdaHessian significantly. Our reported top-1 accuracy of SGD is 69 . 94 ± 0 . 10% , slightly lower than 70 . 23% reported in Chen et al. [7]. We find that the differences in training epochs, learning rate scheduler and weight decay rate are the main reasons. We also run the experiment using the same configuration as in Chen et al. [7], and AGD can achieve 70 . 45% accuracy at lr = 4e-4 and δ = 1e-5, which is still better than the 70 . 23% result reported in Chen et al. [7].

We also report the accuracy of AGD for ResNet18 on Cifar10 for comparing with the SOTA results 3 , which is listed in Appendix A.5. Here we clarify again the ResNet naming confusion. The test accuracy of ResNet18 on Cifar10 training with AGD is above 95%, while ResNet32 is about 93% since ResNet18 is much more complex than ResNet32.

4.4 RecSys

To evaluate the accuracy of CTR estimation, we have adopted the Area Under the receiver-operator Curve (AUC) as our evaluation criterion, which is widely recognized as a reliable measure [15]. As stated in Cheng et al. [10], Wang et al. [32], Ling et al. [20], Zhu et al. [38], even an absolute

Dataset

Cifar10

ImageNet

Model

ResNet20

ResNet32

ResNet18

SGD

92

.

14

±

.

14

(

+0

.

21

)

93

.

10

±

.

07

(

+0

.

02

)

69

.

94

±

.

10

(

+0

.

41

)

Adam

90

.

46

±

.

20

(

+1

.

89

)

91

.

54

±

.

12

(

+1

.

58

)

64

.

03

±

.

16

(

+6

.

32

)

AdamW

92

.

12

±

.

14

(

+0

.

23

)

92

.

72

±

.

01

(

+0

.

40

)

69

.

11

±

.

17

(

+1

.

24

)

AdaBelief

92

.

19

±

.

15

(

+0

.

16

)

92

.

90

±

.

13

(

+0

.

22

)

70

.

20

±

.

03

(

+0

.

15

)

AdaHessian

92

.

27

±

.

27

(

+0

.

08

)

92

.

91

±

.

14

(

+0

.

21

)

69

.

57

±

.

12

(

+0

.

78

)

AGD

92

.

35

±

.

24

93

.

12

±

.

18

70

.

35

±

.

17

Dataset

Avazu

Criteo

Model

MLP

DCN

SGD

0

.

7463

±

.

0005

(

+1

.

7

‰)

0

.

7296

±

.

0067

(

+72

.

7

‰)

Adam

0

.

7458

±

.

0010

(

+2

.

2

‰)

0

.

8023

±

.

0002

(

+0

.

0

‰)

AdaBelief

0

.

7467

±

.

0009

(

+1

.

3

‰)

0

.

8022

±

.

0002

(

+0

.

1

‰)

AdaHessian

0

.

7434

±

.

0006

(

+4

.

6

‰)

0

.

8004

±

.

0005

(

+1

.

9

‰)

AGD

0

.

7480

±

.

0008

0

.

8023

±

.

0004

improvement of 1‰ in AUC can be considered practically significant given the difficulty of improving CTRprediction. Our experimental results in Table 5 indicate that AGD can achieve highly competitive or significantly better performance when compared to other optimizers. In particular, on the Avazu task, AGD outperforms all other optimizers by more than 1‰. On the Criteo task, AGD performs better than SGD and AdaHessian, and achieves comparable performance to Adam and AdaBelief.

4.5 The effect of δ

0

10

20

30

40

50

60

70

80

90

Epoch

0

20

40

60

80

100

Ratio (%)

86.95

82.2

80.93

80.57

22.68

5.03

2.63

1.58

1.07

0.91

[0, 1e-8)

[1e-8, 1e-5)

[1e-5, 1e-4)

[1e-4, +

)

(a) ResNet18 on ImageNet

0

10

20

30

40

50

Epoch

0

20

40

60

80

100

Ratio (%)

0.0

0.05

0.21

0.29

0.32

0.34

0.34

0.35

0.35

0.35

0.35

[0, 1e-14)

[1e-14, 1e-6)

[1e-6, 1e-5)

[1e-5,+

)

(b) Transformer on IWSLT14

0

25

50

75

100

125

150

175

200

Epoch

0

20

40

60

80

100

Ratio (%)

97.05

97.06

97.01

96.97

96.39

95.27

95.12

[0, 1e-6)

[1e-6, 1e-5)

[1e-5, 1e-4)

[1e-4,+

)

(c) 2-layer LSTM on PTB

0

10000

20000

30000

40000

50000

60000

70000

Global Step

0

20

40

60

80

100

Ratio (%)

100.0

84.03

71.52

60.16

49.51

39.09

29.09

20.15

14.7

[0, 1e-10)

[1e-10, 1e-8)

[1e-8, 1e-6)

[1e-6, 1e-4)

[1e-4, +

)

(d) DCN on Criteo

In this section, we aim to provide a comprehensive analysis of the impact of δ on the training process by precisely determining the percentage of ˆ b t that Algorithm 1 truncates. To this end, Figure 6 shows the distribution of ˆ b t across various tasks under the optimal configuration that we have identified. The black dot on the figure provides the precise percentage of ˆ b t that δ truncates during the training process. Notably, a lower percentage indicates a higher degree of SGD-like updates compared to adaptive steps, which can be adjusted through δ .

As SGD with momentum generally outperforms adaptive optimizers on CNN tasks [34, 39], we confirm this observation as illustrated in Figure 6a: AGD behaves more like SGD during the initial stages of training (before the first learning rate decay at the 30th epoch) and switches to adaptive optimization for fine-tuning. Figure 6b indicates that the parameters taking adaptive updates are dominant, as expected because adaptive optimizers such as AdamW are preferred in transformers. Figure 6c demonstrates that most parameters update stochastically, which explains why AGD has a similar curve to SGD in Figure 4b before the 100th epoch. The proportion of parameters taking adaptive updates grows from 3% to 5% afterward, resulting in a better PPL in the fine-tuning stage. Concerning Figure 6d, the model of the RecSys task trains for only one epoch, and AGD gradually switches to adaptive updates for a better fit to the data.

4.6 Computational cost

We train a Transformer small model for IWSLT14 on a single NVIDIA P100 GPU. AGD is comparable to the widely used AdamW optimizer, while significantly outperforms AdaHessian in terms of memory footprint and training speed. As a result, AGD can be a drop-in replacement for AdamW with similar computation cost and better generalization performance.

5 Theoretical analysis

Using the framework developed in Reddi et al. [28], Yang et al. [33], Chen et al. [9], Zhou et al. [37], we have the following theorems that provide the convergence in non-convex and convex settings. Particularly, we use β 1 ,t to replace β 1 , where β 1 ,t is non-increasing with respect to t .

Theorem 1. (Convergence in non-convex settings) Suppose that the following assumptions are satisfied:

f is differential and lower bounded, i.e., f ( w ∗ ) > -∞ where w ∗ is an optimal solution. f is also L -smooth, i.e., ∀ u , v ∈ R n , we have f ( u ) ≤ f ( v ) + 〈∇ f ( v ) , u -v 〉 + L 2 ‖ u -v ‖ 2 .

At step t , the algorithm can access a bounded noisy gradient and the true gradient is bounded, i.e., ‖ g t ‖ ∞ ≤ G ∞ , ‖∇ f ( w t ) ‖ ∞ ≤ G ∞ , ∀ t ∈ [ T ] . Without loss of generality, we assume G ∞ ≥ δ .

glyph[negationslash]

The noisy gradient is unbiased and the noise is independent, i.e., g t = ∇ f ( w t ) + ζ t , E [ ζ t ] = 0 and ζ i is independent of ζ j if i = j .

α t = α/ √ t , β 1 ,t is non-increasing satisfying β 1 ,t ≤ β 1 ∈ [0 , 1) , β 2 ∈ [0 , 1) and b t,i ≤ b t +1 ,i ∀ i ∈ [ n ] .

Then Algorithm 1 yields

where C , C and C

The proof of Theorem 1 is presented in Appendix B. There are two important points that should be noted: Firstly, in assumption 2, we can employ the gradient norm clipping technique to ensure the upper bound of the gradients. Secondly, in assumption 4, b t,i ≤ b t +1 ,i ∀ i ∈ [ n ] , which is necessary for the validity of Theorems 1 and 2, may not always hold. To address this issue, we can implement the AMSGrad condition [28] by setting b t +1 = max( b t +1 , b t ) . However, this may lead to a potential decrease in the algorithm's performance in practice. The more detailed analysis is provided in Appendix A.4. From Theorem 1, we have the following corollaries. √

Corollary 1. Suppose β 1 ,t = β 1 / t , we have

where C 3 , C 4 and C 5 are the same with Theorem 1.

The proof of Corollary 1 can be found in Appendix C.

Corollary 2. Suppose β 1 ,t = β 1 , ∀ t ∈ [ T ] , we have

where C 3 and C 4 are the same with Theorem 1.

Corollaries 1 and 2 imply the convergence (to the stationary point) rate for AGD is O (log T/ √ T ) in non-convex settings.

Theorem 2. (Convergence in convex settings) Let { w t } be the sequence obtained by AGD (Algorithm 1), α t = α/ √ t , β 1 ,t is non-increasing satisfying β 1 ,t ≤ β 1 ∈ [0 , 1) , β 2 ∈ [0 , 1) , b t,i ≤ b t +1 ,i ∀ i ∈ [ n ] and ‖ g t ‖ ∞ ≤ G ∞ , ∀ t ∈ [ T ] . Suppose f t ( w ) is convex for all t ∈ [ T ] , w ∗ is an optimal solution of ∑ T t =1 f t ( w ) , i.e., w ∗ = arg min w ∈ R n ∑ T t =1 f t ( w ) and there exists the constant D ∞ such that max t ∈ [ T ] ‖ w t -w ∗ ‖ ∞ ≤ D ∞ . Then we have the following bound on the regret

where C 1 and C 2 are defined as follows:

The proof of Theorem 2 is given in Appendix D. To ensure that the condition max t ∈ [ T ] ‖ w t -w ∗ ‖ ∞ ≤ D ∞ holds, we can assume that the domain W ⊆ R n is bounded and project the sequence { w t } onto W by setting w t +1 = Π W ( w t -α t √ 1 -β t 2 1 -β t 1 m t max( √ b t ,δ √ 1 -β t 2 ) ) . From Theorem 2, we have the following corollary.

Corollary 3. Suppose β 1 ,t = β 1 /t , we have

where C 1 and C 2 are the same with Theorem 2.

The proof of Corollary 3 is given in Appendix E. Corollary 3 implies the regret is O ( √ T ) and can achieve the convergence rate O (1 / √ T ) in convex settings.

6 Conclusion

In this paper, we introduce a novel optimizer, AGD, which incorporates the Hessian information into the preconditioning matrix and allows seamless switching between SGD and the adaptive optimizer. Weprovide theoretical convergence rate proofs for both non-convex and convex stochastic settings and conduct extensive empirical evaluations on various real-world datasets. The results demonstrate that AGD outperforms other optimizers in most cases, resulting in significant performance improvements. Additionally, we analyze the mechanism that enables AGD to automatically switch between stochastic and adaptive optimization and investigate the impact of the hyperparameter δ for this process.

References

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In Kimberly Keeton and Timothy Roscoe, editors, 12th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2016, Savannah, GA, USA, November 2-4, 2016 , pages 265-283. USENIX Association, 2016.

Shun-ichi Amari. Natural gradient works efficiently in learning. Neural Comput. , 10(2):251-276, 1998. doi: 10 . 1162/089976698300017746.

Avazu. Avazu click-through rate prediction. https://www . kaggle . com/c/avazu-ctr-prediction/ data , 2015.

Charles G. Broyden. The convergence of a class of double-rank minimization algorithms. Journal of the Institute of Mathematics and Its Applications , 6:6-90, 1970. doi: 10 . 1093/imamat/6 . 1 . 76.

Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput. , 16(5):1190-1208, 1995. doi: 10 . 1137/0916069.

Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign , pages 2-17, Lake Tahoe, California, December 4-5 2014.

Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. pages 3239-3247, 07 2020. doi: 10 . 24963/ijcai . 2020/448.

Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the generalization gap of adaptive gradient methods in training deep neural networks. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020 , pages 3267-3275. ijcai.org, 2020. doi: 10 . 24963/ijcai . 2020/452.

Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of A class of adam-type algorithms for non-convex optimization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. Wide & deep learning for recommender systems. In Alexandros Karatzoglou, Balázs Hidasi, Domonkos Tikk, Oren Sar Shalom, Haggai Roitman, Bracha Shapira, and Lior Rokach, editors, Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, DLRS@RecSys 2016, Boston, MA, USA, September 15, 2016 , pages 7-10. ACM, 2016. doi: 10 . 1145/ 2988450 . 2988454.

Criteo. Criteo display ad challenge. http://labs . criteo . com/2014/02/kaggle-displayadvertising-challenge-dataset , 2014.

John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12:2121-2159, 2011. doi: 10 . 5555/ 1953048 . 2021068.

R. Fletcher. A new approach to variable metric algorithms. Comput. J. , 13(3):317-322, 1970. doi: 10 . 1093/comjnl/13 . 3 . 317.

Donald Goldfarb. A family of variable metric updates derived by variational means. Mathematics of Computation , 24(109):23-26, 1970. doi: 10 . 1090/S0025-5718-1970-0258249-6.

Thore Graepel, Joaquin Quiñonero Candela, Thomas Borchert, and Ralf Herbrich. Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoft's bing search engine. In Johannes Fürnkranz and Thorsten Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel , pages 13-20. Omnipress, 2010.

Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 1837-1845. PMLR, 2018.

Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from adam to SGD. CoRR , abs/1712.07628, 2017.

Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations , ICLR '15, San Diego, CA, USA, 2015.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). http://www . cs . toronto . edu/~kriz/cifar . html , 2009.

Xiaoliang Ling, Weiwei Deng, Chen Gu, Hucheng Zhou, Cui Li, and Feng Sun. Model ensemble for click prediction in bing search ads. In Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich, editors, Proceedings of the 26th International Conference on World Wide Web Companion, Perth, Australia, April 3-7, 2017 , pages 689-698. ACM, 2017. doi: 10 . 1145/3041021 . 3054192.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn treebank, June 1993.

Thomas Moreau, Mathurin Massias, Alexandre Gramfort, Pierre Ablin, Pierre-Antoine Bannier, Benjamin Charlier, Mathieu Dagréou, Tom Dupré la Tour, Ghislain Durif, Cassio F. Dantas, Quentin Klopfenstein, Johan Larsson, En Lai, Tanguy Lefort, Benoit Malézieux, Badr Moufad, Binh T. Nguyen, Alain Rakotomamonjy, Zaccharie Ramzi, Joseph Salmon, and Samuel Vaiter. Benchopt: Reproducible, efficient and collaborative optimization benchmarks. 2022.

Yurii E. Nesterov. Introductory Lectures on Convex Optimization - A Basic Course , volume 87 of Applied Optimization . Springer, 2004. ISBN 978-1-4613-4691-3. doi: 10 . 1007/978-1-4419-8853-9. URL https://doi . org/10 . 1007/978-1-4419-8853-9 .

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 8024-8035, 2019.

Boris T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964. doi: 10 . 1016/0041-5553(64)90137-5.

Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In Proceedings of the 6th International Conference on Learning Representations , ICLR '18, Vancouver, BC, Canada, 2018. OpenReview.net.

Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics , pages 400-407, 1951.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. doi: 10 . 1007/s11263-015-0816-y.

David F. Shanno. Conditioning of quasi-newton methods for function minimization. Mathematics of Computation , 24(111):647-656, 1970. doi: 10 . 1090/S0025-5718-1970-0274029-X.

Ruoxi Wang, Bin Fu, Gang Fu, and Mingliang Wang. Deep & cross network for ad click predictions. In Proceedings of the ADKDD'17, Halifax, NS, Canada, August 13 - 17, 2017 , pages 12:1-12:7. ACM, 2017. doi: 10 . 1145/3124749 . 3124754.

Tianbao Yang, Qihang Lin, and Zhe Li. Unified convergence analysis of stochastic momentum methods for convex and non-convex optimization. CoRR , abs/1604.03257v2, 2016.

Zhewei Yao, Amir Gholami, Sheng Shen, Kurt Keutzer, and Michael W. Mahoney. ADAHESSIAN: an adaptive second order optimizer for machine learning. CoRR , abs/2006.00719, 2020.

Matthew D. Zeiler. Adadelta: An adaptive learning rate method. CoRR , abs/1212.5701, 2012.

Shuxin Zheng, Qi Meng, Taifeng Wang, Wei Chen, Nenghai Yu, Zhiming Ma, and Tie-Yan Liu. Asynchronous stochastic gradient descent with delay compensation. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , volume 70 of Proceedings of Machine Learning Research , pages 4120-4129. PMLR, 2017.

Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. On the convergence of adaptive gradient methods for nonconvex optimization. CoRR , abs/1808.05671, 2018.

Jieming Zhu, Jinyang Liu, Shuai Yang, Qi Zhang, and Xiuqiang He. Open benchmarking for click-through rate prediction. In Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong, editors, CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021 , pages 2759-2769. ACM, 2021. doi: 10 . 1145/3459637 . 3482486. URL https://doi . org/10 . 1145/3459637 . 3482486 .

Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C. Tatikonda, Nicha C. Dvornek, Xenophon Papademetris, and James S. Duncan. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

Appendix

A Details of experiments

A.1 Configuration of optimizers

In this section, we provide a thorough description of the hyperparameters used for different optimizers across various tasks. For optimizers other than AGD, we adopt the recommended parameters for the identical experimental setup as indicated in the literature of AdaHessian [34] and AdaBelief [39]. In cases where these recommendations are not accessible, we perform a hyperparameter search to determine the optimal hyperparameters.

NLP

SGD/Adam/AdamW: For the NMT task, we report the results of SGD from AdaHessian [34], and search learning rate among {5e-5, 1e-4, 5e-4, 1e-3} and epsilon in {1e-12, 1e-10, 1e-8, 1e-6, 1e-4} for Adam/AdamW, and for both optimizers the optimal learning rate/epsilon is 5e-4/1e-12. For the LM task, we follow the settings from AdaBelief [39], setting learning rate to 30 for SGD and 0.001 for Adam/AdamW while epsilon to 1e-12 for Adam/AdamW when training 1-layer LSTM. For 2-layer LSTM, we conduct a similar search with learning rate among {1e-4, 5e-4, 1e-3, 1e-2} and epsilon within {1e-12, 1e-10, 1e-8, 1e-6, 1e-4}, and the best parameters are learning rate = 1e-2 and epsilon = 1e-8/1e-4 for Adam/AdamW. For 3-layer LSTM, learning rate = 1e-2 and epsilon = 1e-8 are used for Adam/AdamW.

AdaBelief: For the NMT task we use the recommended configuration from the latest implementation 4 for transformer. We search learning rate in {5e-5, 1e-4, 5e-4, 1e-3} and epsilon in {1e-16, 1e-14, 1e-12, 1e-10, 1e-8}, and the best configuration is to set learning rate as 5e-4 and epsilon as 1e-16. We adopt the same LSTM experimental setup for the LM task and reuse the optimal settings provided by AdaBelief [39], except for 2-layer LSTM, where we search for the optimial learning rate in {1e-4, 5e-4, 1e-3, 1e-2} and epsilon in {1e-16, 1e-14, 1e-12, 1e-10, 1e-8}. However, the best configuration is identical to the recommended.

AdaHessian: For the NMT task, we adopt the same experimental setup as in the official implementation. 5 For LMtask, we search the learning rate among {1e-3, 1e-2, 0.1, 1} and hessian power among {0.5, 1, 2}. We finally select 0.1 for learning rate and 0.5 for hessian power for 1-layer LSTM, and 1.0 for learning rate and 0.5 for for hessian power for 2,3-layer LSTM. Note that AdaHessian appears to overfit when using learning rate 1.0. Accordingly, we also try to decay its learning rate at the 50th/90th epoch, but it achieves a similar PPL.

AGD: For the NMT task, we search learning rate among {5e-5, 1e-4, 5e-4, 1e-3} and δ among {1e-16, 1e-14, 1e-12, 1e-10, 1e-8}. We report the best result with learning rate 5e-5 and δ as 1e-14 for AGD. For the LM task, we search learning rate among {1e-4, 5e-4, 1e-3, 5e-3, 1e-2} and δ from 1e-16 to 1e-4, and the best settings for learning rate ( δ ) is 5e-4 (1e-10) and 1e-3 (1e-5) for 1-layer LSTM (2,3-layer LSTM).

The weight decay is set to 1e-4 (1.2e-6) for all optimizers in the NMT (LM) task. For adaptive optimizers, we set ( β 1 , β 2 ) to (0.9, 0.98) in the NMT task and (0.9, 0.999) in the LM task. For the LM task, the general dropout rate is set to 0.4.

CV

SGD/Adam/AdamW: We adopt the same experimental setup in AdaHessian [34]. For SGD, the initial learning rate is 0.1 and the momentum is set to 0.9. For Adam, the initial learning rate is set to 0.001 and the epsilon is set to 1e-8. For AdamW, the initial learning rate is set to 0.005 and the epsilon is set to 1e-8.

AdaBelief: We explore the best learning rate for ResNet20/32 on Cifar10 and ResNet18 on ImageNet, respectively. Finally, the initial learning rate is set to be 0.01 for ResNet20 on Cifar10 and 0.005 for ResNet32/ResNet18 on Cifar10/ImageNet. The epsilon is set to 1e-8.

AdaHessian: We use the recommended configuration as much as possible from AdaHessian [34]. The Hessian power is set to 1. The initial learning rate is 0.15 when training on both Cifar10 and ImageNet, as recommended in Yao et al. [34]. The epsilon is set to 1e-4.

AGD: We conduct a grid search of δ and the learning rate. The choice of δ is among {1e-8, 1e-7, 1e-6, 1e-5, 1e-4, 1e-3, 1e-2} and the search range for the learning rate is from 1e-4 to 1e-2. Finally, we choose the learning rate to 0.007 and δ to 1e-2 for Cifar10 task, and learning rate to 0.0004 and δ to 1e-5 for ImageNet task.

The weight decay for all optimizers is set to 0.0005 on Cifar10 and 0.0001 on ImageNet. β 1 = 0 . 9 and β 2 = 0 . 999 are for all adaptive optimizers.

RecSys Note that we implement the optimizers for training on our internal distributed environment.

SGD: We search for the learning rate among {1e-4, 1e-3, 1e-2, 0.1, 1} and choose the best results (0.1 for the Avazu task and 1e-3 for the Criteo task).

Adam/AdaBelief: We search the learning rate among {1e-5, 1e-4, 1e-3, 1e-2} and the epsilon among {1e-16, 1e-14, 1e-12, 1e-10, 1e-8, 1e-6}. For the Avazu task, the best learning rate/epsilon is 1e-4/1e-8 for Adam and 1e-4/1e-16 for AdaBelief. For the Criteo task, the best learning rate/epsilon is 1e-3/1e-8 for Adam and 1e-3/1e-16 for AdaBelief.

AdaHessian: We search the learning rate among {1e-5, 1e-4, 1e-3, 1e-2} and the epsilon among {1e-16, 1e-14, 1e-12, 1e-10, 1e-8, 1e-6}. The best learning rate/epsilon is 1e-4/1e-8 for the Avazu task and 1e-3/1e-6 for the Criteo task. The block size and the Hessian power are set to 1.

AGD: We search the learning rate among {1e-5, 1e-4, 1e-3} and δ among {1e-12, 1e-10, 1e-8, 1e-6, 1e-4, 1e-2}. The best learning rate/ δ is 1e-4/1e-4 for the Avazu task and 1e-4/1e-8 for the Criteo task.

β 1 = 0 . 9 and β 2 = 0 . 999 are for all adaptive optimizers.

A.2 AGD vs. AdaBound

A.3 Numerical Experiments

In our numerical experiments, we employ the same learning rate across optimizers. While a larger learning rate could accelerate convergence, as shown in Figures 7a and 7b, we also note that it could lead to unstable training. To investigate the largest learning rate that each optimizer could handle for the Beale function, we perform a search across the range of {1e-5, 1e-4, ..., 1, 10}. The optimization trajectories are displayed in Figure 7c, and AGD demonstrates slightly superior performance. Nonetheless, we argue that the learning rate selection outlined in Section 3.3 presents a more appropriate representation.

4

3

2

1

0

1

2

3

4

4

3

2

1

0

1

2

3

4

Optimal

Adam, lr=1e-3

Adam, lr=1e-4

(a) Adam

4

3

2

1

0

1

2

3

4

4

3

2

1

0

1

2

3

4

Optimal

AGD, lr=1

AGD, lr=1e-1

AGD, lr=1e-2

AGD, lr=1e-3

AGD, lr=1e-4

(b) AGD

4

3

2

1

0

1

2

3

4

4

3

2

1

0

1

2

3

4

Optimal

SGDM, lr=1e-5

AdaBelief

AGD, lr=1

AdaHessian, lr=1e-1

(c) Comparison

A.4 AGD with AMSGrad condition

Algorithm 2 summarizes the AGD optimizer with AMSGrad condition. The AMSGrad condition is usually used to ensure the convergence. We empirically show that adding AMSGrad condition to AGD will slightly degenerate AGD's performance, as listed in Table 8. For AGD with AMSGrad condition, we search for the best learning rate among {0.001, 0.003, 0.005, 0.007, 0.01} and best δ within {1e-2, 1e-4, 1e-6, 1e-8}.We find the best learning rate is 0.007 and best δ is 1e-2, which are the same as AGD without AMSGrad condition.

A.5 AGD for ResNet18 on Cifar10

Since the SOTA accuracy 6 for ResNet18 on Cifar10 is 95 . 55% using SGD optimizer with a cosine learning rate schedule, we also test the performance of AGD for ResNet18 on Cifar10. We find AGD can achieve 95 . 79%

Algorithm 2 AGD with AMSGrad condition

1: Input: parameters β 1 , β 2 , δ , w 1 ∈ R n , step size α t , initialize m 0 = 0 , b 0 = 0

2: for t = 1 to T do

3: g t = ∇ f t ( w t )

4: m t ← β 1 m t -1 +(1 -β 1 ) g t

6: b t ← β 2 b t -1 +(1 -β 2 ) s 2 t

7: b ← max( b , b

t t t -1 ) // AMSGrad condtion

8: w t +1 = w t -α t √ 1 -β t 2 1 -β t 1 m t max( √ b t ,δ √ 1 -β t 2 )

9: end for

accuracy at lr = 0.001 and δ = 1e-2 when using the same training configuration as the experiment of SGD in Moreau et al. [24], which exceeds the current SOTA result.

A.6 Robustness to hyperparameters

We test the performance of AGD and Adam with respect to δ (or glyph[epsilon1] ) and learning rate. The experiments are performed with ResNet20 on Cifar10 and the results are shown in Figure 8. Compared to Adam, AGD shows better robustness to the change of hyperparameters.

0

20

40

60

80

100

120

140

160

Epoch

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test accuracy

lr=0.0001

lr=0.0005

lr=0.0007

lr=0.001

lr=0.005

lr=0.007

lr=0.01

(a) AGD

0

20

40

60

80

100

120

140

160

Epoch

0.4

0.5

0.6

0.7

0.8

0.9

Test accuracy

lr=0.0001

lr=0.0005

lr=0.0007

lr=0.001

lr=0.005

lr=0.007

lr=0.01

(b) Adam

0

20

40

60

80

100

120

140

160

Epoch

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Test accuracy

=1e-2

=1e-3

=1e-4

=1e-5

=1e-6

=1e-7

=1e-8

(c) AGD

0

20

40

60

80

100

120

140

160

Epoch

0.4

0.5

0.6

0.7

0.8

0.9

Test accuracy

=1e-2

=1e-3

=1e-4

=1e-5

=1e-6

=1e-7

=1e-8

(d) Adam

B Proof of Theorem 1

Proof. Denote ˆ α t = α t √ 1 -β t 2 1 -β t 1 ,t and v t = max( √ b t , δ √ 1 -β t 2 ) , we have the following lemmas.

Lemma 1. For the parameter settings and assumptions in Theorem 1 or Theorem 2, we have

where ˆ α t = α t √ 1 -β t 2 1 -β t 1 ,t .

Proof. Since β 1 ,t is non-increasing with respect to t , we have β t +1 1 ,t +1 ≤ β t 1 ,t +1 ≤ β t 1 ,t . Hence, β t 1 ,t is nonincreasing and 1 1 -β t 1 ,t is non-increasing. Thus, we only need to prove φ ( t ) = α t √ 1 -β t 2 is decreasing. Since the proof is trivial when β 2 = 0 , we only need to consider the case where β 2 ∈ (0 , 1) . Taking the derivative of φ ( t ) , we have

Since

we have ψ ′ ( t ) > 0 when t > 0 and ψ ′ ( t ) = 0 when t = 0 . Combining ψ (0) = 0 , we get ψ ( t ) > 0 when t > 0 . Thus, we have φ ′ ( t ) < 0 , which completes the proof.

Lemma 2. For the parameter settings and assumptions in Theorem 1 or Theorem 2, we have

where v t = max( √ b t , δ √ 1 -β t 2 ) .

Proof.

By assumptions 2, 4, Lemma 1 and Lemma 2, ∀ t ∈ [ T ] , we have

Following Yang et al. [33], Chen et al. [9], Zhou et al. [37], we define an auxiliary sequence { u t } : ∀ t ≥ 2 ,

Hence, we have

By assumption 1 and Equation (8), we have

Rearranging Equation (9) and taking expectation both sides, by assumption 3 and Equation (6), we get

To further bound Equation (10), we need the following lemma.

Lemma 3. For the sequence { u t } defined as Equation (7), ∀ t ≥ 2 , we have

Proof. Since ∀ t ∈ [ T ] , ∀ i ∈ [ n ] , 1 / (1 -β 1 ,t ) ≥ 1 / (1 -β 1 ,t +1 ) , v t ≥ δ √ 1 -β 2 , ˆ α t -1 /v t -1 ,i > ˆ α t /v t,i . By Equation (8), we have

where the last inequality follows from Cauchy-Schwartz inequality. This completes the proof.

Now we bound P 1 , P 2 , P 3 and P 4 of Equation (10), separately. By assumptions 1, 2, Equation (6) and Lemma 3, we have

Replacing P 1 , P 2 , P 3 and P 4 of Equation (10) with Equation (11) and telescoping Equation (10) for t = 2 to T , we have

Since

substituting Equation (13) into Equation (12), we finish the proof.

C Proof of Corollary 1

Proof. Since β 1 ,t = β 1 / √ t , we have

Substituting Equation (14) into Equation (5), we have

This completes the proof.

D Proof of Theorem 2

Rearranging Equation (15), we have

where the first inequality follows from Cauchy-Schwartz inequality and ab ≤ 1 2 ( a 2 + b 2 ) . Hence, the regret

where the first inequality follows from the convexity of f t ( w ) . For further bounding Equation (16), we need the following lemma.

Lemma 4. For the parameter settings and conditions assumed in Theorem 2, we have

Proof. From Equation (4), we have

where the second inequality follows from Cauchy-Schwartz inequality. Therefore,

where the last inequality follows from

This completes the proof.

Now we return to the proof of Theorem 2. Let ˆ α 0 := ˆ α 1 . By Lemma 1, Lemma 2, Lemma 4, Equation (16) and the third inequality of Equation (6), we have

This completes the proof.

E Proof of Corollary 3

Proof. Since β 1 ,t = β 1 /t , we have

This completes the proof.

Peer reviews:
comments: SUMMARY:
The paper proposes a new gradient based optimization called AGD. The idea is to utilize gradient difference between the current and previous steps in the preconditioning matrix which is related to Hessian. Also, an autoswitching function is proposed to switch between SGD and the adaptive optimizer. As mentioned in the paper, both ideas were previously brought up but the attempt to make an explicit optimizer is first done in the paper.

STRENGTHS:
- Well written and motivated
- Conducting number of downstream experiments
- Presenting extra studies to examine the method

WEAKNESSES:
-The idea is not fully new
- The details of experiments could be more explained

QUESTIONS:
- Figure 3 suggests AGD is faster to converge, and Table 3 suggests AGD results in better performance. Could we make such a conclusion that it should be always used instead of SOTA optimization methods?

- What is the stop criteria of different optimization techniques in Table 3? I am wondering if the improvement obtained by AGD is accurately examined or for example all methods go for the same number of iterations and AGD converges faster? You argued this issue in section 4.3 for CV task.

- The performance of AGD in CV task is quite marginal. How about the convergence time? does AGD maybe converges faster? 

- I would have also liked to see more about autoswitching and why and how it happens in a simple toy example.

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel optimizer, namely, Auto-switchable optimizer with Gradient Difference of adjacent steps. The authors propose a different way of approximating information from the Hessian for faster and better convergence in a way that a preconditioning matrix is computed using successive optimization steps in the parameter steps. The authors evaluate their optimization algorithm across different tasks and including some widely used architectures such as Transformers and RNNs. Moreover, the authors provide a theoretical analysis of the convergence of their algorithm and some bounds.\n\nSTRENGTHS:\n- The paper is well written in terms of giving the appropriate context of the most commonly used optimization algorithms, nice figures for toy experiments to understand how the algorithm works.\n- The experimental framework idea is well designed in the sense that a good range of optimization tasks is considered to show the importance of the algorithm.\n- The idea of using the successive steps seems novel to me to approximate Hessian information for faster convergence.\n\nWEAKNESSES:\n- In my humble opinion, the issue with the current idea lies in terms of the number/width of the optimization steps and how far the solution lies compared to the initialization point in the parameter space. The information that one can obtain by gradient differentials in the optimization space is only getting less and less useful while we increase the dimensionality of the optimization space. That means that for smaller optimization problems where the optimal solution would lie close to the initialization point, then I can really see the benefit of AGD compared to less Hessian well-informed optimizers like Adam. One can also see this emergent problem even from the theoretical analysis of the authors. For instance, in Theorem 2, the authors bound the approximation of the function using $D_\\infty$ which effectively is the upper bound in terms of successive optimization steps in the parameter space and also an upper bound of each weight vector compared to its distance from the optimal solution. That means that for real-world multi-dimensional optimization problems (e.g. billions of parameters) those bounds and probably the algorithm itself could be called into question. Having said that, there are other works like NTK that have shown that neural networks stay pretty close in general near the initialization regime, so, I would not consider it appropriate to ask the authors to prove these things but they could put more effort into making their experiments more convincing to ameliorate this huge concern.\n- Building upon my previous argument, my belief about the problem of scaling of the algorithm to bigger optimization problems is also displayed in the experiments where AGD scores significantly higher than all the other optimizers in smaller parameter spaces (e.g. 1-2 layers of LSTM) but becomes almost negligible with slightly bigger networks (e.g. compare Adam vs Proposed in Table 4, ResNEt 20, 32, Table 5 MLP, DCN, compare AdamW vs AGD in Table 1 Transformer). Having said that, the authors can include experiments with larger and state-of-the-art networks (e.g. ResNet 100+ layers on ImageNet, include some experiments with some networks with 500+ million parameters transformers) to prove me wrong and compare against state-of-the-art performance reported numbers with SGD and/or Adam. \n\nI am willing to increase my score if most of the most important above concerns are addressed by the authors since I truly believe that the paper has a great potential to offer a more robust general optimization strategy to replace Adam.\n\nQUESTIONS:\nWould you consider including experiments with some harder tasks such as estimation tasks, e.g. audio source separation instead of detection and retrieval tasks?\n\nLIMITATIONS:\nThe authors should include my first point under the weaknesses of the paper and include a small paragraph to address this potential limitation of their algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper proposes a new optimizer based on finite difference approximation to obtain the inner product between the Hessian row vector and the parameter vector difference from gradients of succeeding steps and an auto-switching function to switch between SGD and the adaptive optimizer. It uses an exponential moving average of gradient instead of gradient for lower variance. Experimental results show that the proposed method performs better than existing methods in most cases.

STRENGTHS:
The proposed algorithm can efficiently acquire the information of the Hessian.

Experimental results show performance improvements.

WEAKNESSES:
The explanation of related works is minimum. It needs to be more concretely clarified what problem of existing second-order methods it addresses. 

The experiments in section 4 only report model performance and do not report computational costs. 

I need help finding Figures 5 and 6 in the paper. Are they missing?

QUESTIONS:
What is the reason for the performance improvement? Is it due to its ability to find better local optima or the faster convergence in a limited computation budget?

LIMITATIONS:
There is no numerical explanation for the computational cost.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new optimizer called AGD that integrates the information of the Hessian into the preconditioning matrix and switches dynamically between SGD and the adaptive optimizer. The authors establish theoretically proven convergence guarantees in both non-convex and convex stochastic settings. The authors validate AGD on a total of six public datasets: two from NLP (IWSLT14 and PTB), two from CV (Cifar10 and ImageNet), and the rest from RecSys (Criteo and Avazu). The experimental results reveal that AGD can outperform or be on a par with the SOTA optimizers. Overall, the paper makes a significant contribution to the field of deep learning optimization. AGD is a promising new optimizer that has the potential to improve the performance of deep learning models on a variety of tasks.\n\nSTRENGTHS:\n• AGD can achieve a faster convergence rate over SGD\n\n• AGD can automatically switch between stochastic and adaptive optimization, depending on the progress of the optimization. This allows AGD to achieve the best of both worlds, i.e., the fast convergence of adaptive optimizers and the robustness of SGD.\n\n• AGD has a few flexible hyperparameters, making tuning for different tasks and datasets easy.\n\n• AGD was evaluated on various datasets across different domains, and it outperformed other optimizers in most cases.\n\nWEAKNESSES:\n• The convergence rate of AGD depends on the hyperparameters δ and β1. If these hyperparameters are not chosen carefully, AGD may not converge to the optimal solution. Looking forward to seeing more comprehensive robustness evaluation using different network structures and larger datasets.\n\n• AGD requires more computation than SGD. How is its efficiency for large-scale problems, e.g., large language models? I think performance on large-scale generative model optimization is a very interesting direction. Does the author have plans to try on such a task?\n\nQUESTIONS:\n•\tPlease see weaknesses.\n\nLIMITATIONS:\nThere is no limitation discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed AGD, a novel optimiser that can dynamically switch between an adaptive optimiser and SGD. To achieve this, an auto-switching function is introduced to change the diagonal elements of the preconditioning matrix based on the gradients of adjacent steps. Both theoretical analysis on the convergence of the optimiser, and experimental results on six public CV, NLP and RecSys setups are provided, demonstrating AGD is a promising method both in theory and practice.\n\nSTRENGTHS:\nIt is often noticed in practice that SGD with proper hyperparameters (learning rate, momentum, and weight decay) can outperform the adaptive optimisers with better local optimum found, while the adaptive optimisers require less effort in hyperparameter tuning and sometimes faster convergence. The auto-switching strategy may help combine the advantages of them, and experimental results and analysis demonstrate the soundness and robustness of the chosen auto-switching function.\n\nWEAKNESSES:\n1. Optimisers can perform differently on large-scale and small-scale experiments. Although good PTB results are given in the paper, a larger-scale language modelling experiment may still be useful.\n2. Some hyperparameter settings, such as those for PTB, are missing in the paper and the supplementary materials. The effect of regularisation methods, in particular L2 and dropout, are not given. This often makes a key difference to the generalisation ability of the test set and the performance of SGD and could be at least considered in the numerical analysis session. \n3. The proof of the faster convergence speed of AGD using the case studies in Sec. 3.2 may have resulted in some questions, as the comparisons of optimisers with different learning rates and other hyperparameters may not be fair enough. \n4. Extra memory cost, computational cost and possible limitations of the method are not discussed.\n\nQUESTIONS:\nWhat's the trajectory of Adam at the minimum point B in Figure 1?\n\nLIMITATIONS:\nDiscussions of regularisation methods (at least with SGD) and large-scale experiments with complex model structures (e.g. a Transformer with both encoder and decoder) are missing. \nThe analysis of the convergence speed in the numerical analysis session may not be completely fair. \nWould be useful to include the analysis of computation and storage costs.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}