Title: SING: A Plug-and-Play DNN Learning Technique
Venue: NeurIPS 2023
Accepted: no
Keywords: optimization

normalization

standardization

centralization

gradient descent

adam

stabilization

plug-and-play
Abstract: We propose SING (StabIlized and Normalized Gradient), a plug-and-play technique that improves the stability and generalization of the Adam(W) optimizer. SING is straightforward to implement and has minimal computational overhead, requiring only a layer-wise standardization of the gradients fed to Adam(W) without introducing additional hyper-parameters. We support the effectiveness and practicality of the proposed approach by showing improved results on a wide range of architectures, problems (such as image classification, depth estimation, and natural language processing), and in combination with other optimizers. We provide a theoretical analysis of the convergence of the method, and we show that by virtue of the standardization, SING can escape local minima narrower than a threshold that is inversely proportional to the network's depth.

1

Abstract

We propose SING (StabIlized and Normalized Gradient), a plug-and-play technique that improves the stability and generalization of the Adam(W) optimizer. SING is straightforward to implement and has minimal computational overhead, requiring only a layer-wise standardization of the gradients fed to Adam(W) without introducing additional hyper-parameters. We support the effectiveness and practicality of the proposed approach by showing improved results on a wide range of architectures, problems (such as image classification, depth estimation, and natural language processing), and in combination with other optimizers. We provide a theoretical analysis of the convergence of the method, and we show that by virtue of the standardization, SING can escape local minima narrower than a threshold that is inversely proportional to the network's depth.

1 Introduction

Neural network training is a highly non-convex and stochastic optimization problem, complicated by 13 hidden dynamics between the optimization algorithm and the network architecture. Several common 14 pitfalls have been identified, such as bad initialization, vanishing and exploding gradients [3, 32], 15 abrupt shifts in the distribution of layer inputs (the so-called internal covariate shift [19]). Significant 16 progress has been made by tackling these issues either by architectural improvements [1, 17] or with 17 better optimizers [21, 28, 39]. 18

The Adam(W) optimizer [21, 26] is widely adopted for neural network training due to its ability to 19 combine first and second-order moments of the gradient, mitigating the sensitivity to the learning 20 rate, and providing adaptability to gradient updates of different magnitude or sparsity. It is applicable 21 to widely different architectures, from convolutional to transformers, and application domains. 22 Nonetheless, it has shown instabilities in specific scenarios, such as large-scale problems [5, 29] or, 23 as we demonstrate in this work, some image-to-image tasks. These instabilities manifest as spikes in 24 the training loss which might involve a prolonged recovery periods - if it recovers. 25

26

27

28

29

30

31

Contributions. In this work we propose a simple layer-wise gradient standardization as a technique to improve the stability of existing optimizers. Our technique, SING, is plug-and-play: by simply changing the gradient fed to AdamW (or any other 'host' optimizer) it integrates seamlessly without introducing any additional hyperparameters. As such, it does not require any additional fine-tuning apart from that of the host optimization framework. In this way, SING preserves the desirable properties of the host optimizer but with increased stability.

32

33

34

35

We notably theoretically show that the optimizer is capable to escape narrow local minima single step

within a

, given a sufficiently high learning rate (see Theorem 3.1). Moreover, the magnitude of this learning rate is

inversely proportional to the depth of the network

i.e.

for a fixed learning rate, the higher the number of layers of the network, the lower the learning rate must be to escape local

minima. This highlights the compatibility of our technique with deep neural networks. Since narrow 36

local minima are often associated with poor generalization in non-convex optimization landscapes 37

50

51

-10

-5

0

5

0

1

2

3

F(x)

Initial point

Iterates of SGD

Iterates of SING + SGD

Final point of SGD

Final point of SING + SGD

x

[6, 15, 16, 20, 33], it is crucial for an optimizer to avoid such pitfalls. We capitalize on this theoretical 38

result and stabilize the gradient using several techniques [18, 57, 40] to reach a step size as large as 39

possible, and thus escape from as many local minima as possible. 40

Although our initial focus was to address Adam's limitations, our technique demonstrates robust gen41

eralization even in tasks where Adam performs well. Notably, we show improvements over AdamW 42

on several tasks such as image classification, depth estimation and natural language processing. 43

The paper is organized as follows. Section 2 describes the proposed method, followed by its theoretical 44 analysis in Section 3 where the properties of SING are presented. We discuss related work in Section 45

In Section 5 we provide extensive numerical experiments which demonstrate the effectiveness of 46

the proposed method on a variety of tasks in natural language processing and computer vision. 47

2 Algorithm 48

We seek to approximate the solution to the following optimization problem, 49

We assume there exists a random function f : R p → R such that E [ ∇ f ( x )] = ∇ F ( x ) for all x ∈ R p , and that we have access to an oracle providing i.i.d. samples ( f n ) n ∈ N [9].

In the case of neural networks, the optimization variable x represents the parameters of the network, 52 F is the oracle loss and f is the empirical loss evaluated on a random mini-batch. The parameters 53 of a neural network have a specific structure. They are made of the concatenation of the parameter 54 tensors from each layer of the network. We use D ∈ N to denote the number of these parameters 55 tensors and define ( I k ) k ∈ [ [1 ,D ] ] such that x I k = { x i : i ∈ I k } represents the k -th parameter tensor. As 56 an example, let's consider the neural network N ( · ; A,b ) = A · + b . In this case, the network has two 57 parameter tensors, hence D = 2 . The first parameter tensor is x I 1 = A and the second is x I 2 = b . 58

Our objective is to endow an optimizer with several key properties: 1) stability 2) capacity to 59 escape narrow local minima 3) adaptability to geometry of the energy landscape, and 4) convergence. 60

61

Importantly, we want to achieve all of these properties without adding any additional hyper-parameters

62

63

and with minimal computational overhead. We achieve the first property - stability - by dividing the gradient by its norm. This prevents vanishing and exploding gradients, which can lead to unstable

training. This also allows to use fixed gradient steps, which enables the algorithm to move past 64

narrow local minima, as we show in the next section. 65

Algorithm 2: PYTORCH implementation of our algorithm. The Γ operator is implemented by NORMALIZE and ϕ by CENTRALIZE. Our technique can be used within any existing first order method i.e. the OPTIMIZER function can be any optimizer (see Table 3 for a comparison).

We define the steps taken by our optimizer by 66

where ϕ is the gradient centralization operation [49] and the division is applied element-wise. The 67 operator Γ corresponds to the parameter-wise normalization i.e. 68

In theory, there could be a division by zero in (2). To avoid this we can add ϵ = 10 -8 to the 69 denominator although it is not strictly necessary because the gradient norm is large in practice. 70 This parametrization naturally arises in usual Deep Learning frameworks, see Algorithm 2 for the 71 PYTORCH implementation. 72

Our setting differs from regular normalized gradient descent in two ways: we center the gradients 73 before normalizing and we perform the normalization on a parameter-wise basis. This is particularly 74 important for large networks where the norm of the full gradient can be very large, making it nearly 75 impossible to train the network effectively. 76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

3 Theoretical Analysis

This section analyzes the key properties of our technique. Theorem 3.1 demonstrates how normalization techniques aid in escaping local minima. Theorem 3.2 establishes stability results, including several invariance properties of the algorithm. Moreover, Theorems 3.3 and 3.4 provide insights into the rate of convergence of our algorithm in a stochastic setting, under mild assumptions. For complete proofs and technical details, please refer to Appendix A.

3.1 Escaping from narrow local minima

One of the key properties of our algorithm is its ability to escape from narrow local minima. This is crucial because the stochasticity of the optimization landscape often leads to the creation of artificial local minima, generally associated with poor generalization performance [6, 15, 16, 20, 33]. To achieve this we normalize the gradient to take fixed-size steps during training, where the learning rate controls the step size. Doing so allows the escape from narrow local minima provided the steps are large enough. This property is central to our algorithm and leads to better generalization performance.

For simplicity, we assume a deterministic setting in this section. We show that the normalization procedure helps the optimizer to escape narrow local minima. To formalize this observation, we first define the basin of attraction of a critical point of F .

Definition 3.1. Let x ∗ be a critical point of F . The basin of atraction of x ∗ is defined to be the set 93 W ( x ∗ ) such that 94

Moreover, we write B ( x ∗ ) to be the largest ball contained within W ( x ∗ ) , and r its radius. 95

In the previous definition, if x ∗ is a saddle point, A ( x ∗ ) = { x ∗ } and r = 0 . 96

Theorem 3.1 (Escaping from narrow local minima) . Let x t be the sequence of iterates defined by (2) 97 and y t the sequence of iterates of gradient descent, 98

̸

Assume that x t ∈ B ( x ∗ ) (resp. y t ∈ B ( x ∗ ) ) i.e. the ball contained in the basin of attraction of x ∗ , defined in Definition 3.1. Also, assume that x t (resp. y t ) is not a critical point i.e. ∇ F ( x t ) = 0 (resp. ∇ F ( y t ) = 0 ). If the stepsize is sufficiently large,

then the iterates x t +1 (resp. y t +1 ) is outside the set B ( x ∗ ) . See Figure 1 for an illustration.

We see that GD struggles to escape local minima: under mild assumptions on ∇ F , the closer y t is to x ∗ the higher the learning rate must be to escape from A ( x ∗ ) . Indeed, for GD there is no finite step-size η GD that guarantees escaping A ( x ∗ ) . In contrast, Theorem 3.1 tells us that our algorithm escapes A ( x ∗ ) in a single step, provided the learning rate is sufficiently large. Furthermore, as the number of parameter tensors in the model increases, it becomes easier to escape from A ( x ∗ ) . This is an important advantage of our algorithm over GD, especially for large models where the optimization landscape can be highly complex and difficult to navigate.

When the Hessian at x ∗ is well conditioned, escaping from A ( x ∗ ) is roughly equivalent to escaping from the local minimum. Therefore, it is crucial to use the highest possible learning rate. However, using a high learning rate can be problematic as the gradients are unstable and tend to oscillate leading to suboptimal convergence. To address this issue, several methods have been proposed to stabilize the gradients and allow for larger learning rates. Such methods include gradient centralization, LookAhead [54], different momentum strategies such as Adam [21], AdaBelief [57] or even AdaFactor [37] and larger batch sizes, among others. For this reason, the final implementation of our algorithm incorporated within AdamW features LookAhead and softplus calibration [40]. Note however that it does not introduce any additional hyper-parameters as the parameters of these stabilization methods are fixed once and for all.

3.2 Invariance properties

In this section, the setting is considered deterministic for simplicity. This section examines the invariance properties of the technique.

Firstly, we show that a rescaling of the objective function,

does not affect the updates. This property is desirable as the network's performance is unaffected by a scaling of the loss. A similar invariance property applies to changes during training that cause a rescaling of the gradients of a layer. If during training, the output of one layer of the network is rescaled, it won't affect the update of the previous layers, thus allievating part of the problem of internal covariate shift [19].

Second, the algorithm presented in this paper preserves the mean i.e.

where [ x ] i corresponds to the i -th component of the vector x .

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

Theorem 3.2. The iterates defined by (2) are invariant w.r.t. transformation (6), and preserve the mean (7).

The property of preserving the mean has been demonstrated to improves the stability of the optimiza133 tion process in deep neural networks [49]. Moreover, it is motivated by the observation that many 134 non-linear layers demonstrate a mean-shift behavior [19], which alters their behavior based on the 135 sign of input values. This mean-shift behavior is mitigated by the presence of normalization layers, 136 that re-scale and shift the weights. Preserving the mean enhances the stability of the optimization 137 dynamics when normalization layers are present. 138

̸

Furthermore, normalizing the centered gradients mitigates a potential pathological scenario where the 139 gradient signal is diminished. Indeed, the mean of the gradient can hinder the important signal when 140 the mean is too large compared to the centered gradient [49]. However, in such case the amplitude of 141 the centered gradient can be relatively small, preventing efficient updates. Normalizing the gradient 142 solves this issue by preserving its amplitude. 143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

3.3 Convergence

In this section, two theorems of convergence are provided. In the first one, the normalization is studied without the centralization. Under mild assumptions, we show the ℓ 2 -norm of the gradient can be reduced to any desired precision. In the second one, we consider the full setting and show the same result for the ϕ -norm (which is a pseudo-norm). We assume that the stochastic gradient has a σ -bounded variance ( σ > 0 ) i.e.

and the objective function F is positive and L -smooth,

Theorem 3.3 (Convergence without gradient centralization) . Let assumptions (8) and (9) hold. Assume the gradient is computed across a mini-batch of size B = σ 2 ϵ 2 . Let x t be the sequence of iterates (2) with ϕ = I . Then, we have

If we set τ ∼ U ([ [0 , T -1] ]) , η = 2 ϵ L and T = LF ( x 0 ) 2 ϵ 2 , we obtain E [ ∥∇ F ( x τ ) ∥ 2 ] ≤ (2 + √ D + D ) ϵ . Therefore, the iteration complexity and computation complexity to achieve an ϵ -stationary point are O (1 /ϵ 2 ) and O (1 /ϵ 4 ) , respectively.

Theorem 3.4 (Convergence with gradient centralization) . Let assumptions (8) and (9) hold. Assume the gradient is computed across a mini-batch of size B = σ 2 ϵ 2 . Let x t be the sequence of iterates (2). Then we have

where ∥ · ∥ 2 ϕ = ⟨· , ϕ ( · ) ⟩ 2 is a pseudo-norm. If we set τ ∼ U ([ [0 , T -1] ]) , η = 2 ϵ L and T = LF ( x 0 ) 2 ϵ 2 , we obtain E [ ∥∇ F ( x τ ) ∥ ϕ ] ≤ (2 + √ D + D ) ϵ . Therefore, the iteration complexity and computation complexity to achieve an ( ϵ, ϕ ) -stationary point are O (1 /ϵ 2 ) and O (1 /ϵ 4 ) , respectively.

Note that Theorem 3.4 only gives us an ( ϵ, ϕ ) -stationary point i.e. in the limit ϵ → 0 , ∇ F ( x τ ) converges to a point in Ker ( ϕ ) = Span ((1 , . . . , 1) T ) . Indeed, applying ϕ to the gradients amounts to do a projected gradient descent onto the set of weights with the same mean as the initial weights.

We argue that this 'weaker' convergence result it is not problematic. Reaching a point in Ker ( ϕ ) means that the optimization process cannot go any further without violating the constraint. However, since neural networks have lots of parameters, adding one constraint to the solution is not likely to lead to worse performance [49].

4 Related Work

The most-used optimizer nowadays is Adam [21], which computes the entrywise first and second order moments of the gradient, and uses them to adaptively normalize the gradient. In contrast, SING first removes to the gradient its mean and divides it by its norm (standardization) prior to any further computations at the layer level . Furthermore, in Adam the first and second orders are averaged temporally while ours is not. Numerous variants of Adam have been proposed, mainly focusing on stabilizing the iterations of Adam. Notably, the popular AdamW [26] optimizer corrects how weight decay is applied in Adam, yielding a more robust method training larger models, for instance for

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

0

20

40

60

80

Epochs

10

20

30

40

50

60

70

Top-1 Accuracy

W+SING

AdamW

0

20

40

60

80

100

Epochs

10 -4

10 -3

10 -2

10 -1

10 0

10 1

Train loss

W+SING, ViT-S

W+SING, ViT-B

AdamW, ViT-S

AdamW, ViT-B

training (visual) Transformers in practice [41]. Also in the panel of corrections to Adam, RAdam [24] proposes to fix the variance of adaptive learning rate by rewriting the update term, AdaBound [27] and AdaMod [12] clip the update, and AdaNorm [14] directly clips the gradient instead. Conversely, EAdam [52] and SAdam [40] target improving the ϵ term in the denominator. AdaBelief [57] is another variant of Adam. It computes an estimate of the standard deviation instead of the second order moment. AdaFactor [37] factorizes the elements of the gradient to reduce the memory consumption of the optimizer. The authors propose as well an analysis of the instabilities of Adam, and fixes. In this work, we also target reducing Adam's instabilities via gradient standardization.

The works most closely related to ours are LARS [50] and LAMB [51]. Indeed, both optimizers normalize the gradient in a layer-wise fashion like us. However, both methods multiply the normalized gradient by the weight norm. This multiplication is undesired in our case as it would tame down our main theoretical result in Section 3 (Theorem 3.1) which is central to our work. Indeed, this theorem is the keystone to building a stable optimizer able to escape from narrow local minima using larger learning rates, whereas these methods leverage very large batch size to improve performance. Additionally, our method is hyperparameter-free in contrast to those of [50, 51]. Furthermore, these methods are new optimizers to be used as a replacement for Adam(W) whereas SING is a technique that can be used within any optimizer.

Other approaches leverage standardization to better train neural networks: Weight Standardiza195 tion [34] and Weight Normalization [18, 36] parameterize the weights of the network to allow for a 196 smoother training. While this affects the gradients, this approach is orthogonal to ours and could be 197 used with our technique. 198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

Table 2: Ablation study of the different components of SING using a ViT-S on RDE. The reported metric is the accuracy. Each component allow for a higher learning rate. For AdamW, we added the component and studied the convergence. For AdamW + SING, we removed it. As the theory suggests, the higher the learning rate, the higher the final performance. NC stands for no convergence i.e. the loss could not be stabilized throughout the iterations. The maximum LR reported corresponds to the one for AdamW + SING. As displayed in Figure 4, the training of ViT-B spiked resulting in irrelevant performance. Note that for AdamW, the very unstable nature of the training largely widens the gaps in performance when adding a component. In all cases, the best learning rate using AdamW alone was 10 -3 .

Another part of the literature focuses on improving the stability of training processes to ensure smoother convergence. Notably, techniques such as LookAhead [54] adopt an approach where weights computed over the previous k iterations are averaged. Similarly, Gradient Centralization [49] involves subtracting the mean of the gradient, effectively reducing its ℓ 2 norm. In our work, we draw upon these techniques, but it is important to highlight that our approach is distinct and independent from this line of research.

Lastly, it is common in non-convex optimization to normalize the gradient descent algorithm [8, 30, 56]. This line of work supports that the standardization strategies is a simple way to find a better minimizer. In this work, we translate this strategy to deep learning.

5 Experiments

In this section, we evaluate SING on classification, depth estimation and natural language processing. We run all the experiments on a single Tesla V100 GPU with 32GB of VRAM. The code to reproduce the results will be made available upon publication.

5.1 Image classification

We evaluate our technique on the large-scale ImageNet-1K dataset [10] which consists of 1.28 million images for training and 50K images for validation from 1000 categories. We use the FFCV library [23] and its recipe: the data augmentation consists in random horizontal flips and random resized crops. Notably, the downsampling layers are replaced by BlurPool [55]. The size of the images is 192 × 192 during training and 224 × 224 at evaluation [42]. Our networks are trained for 88 epochs with a batch size of 1024. The loss to be minimized is the cross-entropy with a label smoothing [38] of 0 . 1 . For all networks, there is a 5-epoch linear warmup and a cosine decaying schedule afterward.

We carefully design our hyper-parameter tuning strategy to ensure a fair comparison. First, we tune the learning rate among limited values: { 5 × 10 -4 , 10 -3 , 5 × 10 -3 , 10 -2 } for AdamW and { 5 × 10 -3 , 10 -2 , 5 × 10 -2 , 10 -1 } for SING used within AdamW. In the rare cases where the best learning rate found is one of the extreme value of the set, additional learning rates were tested. For all networks and optimizers the best learning rate found is the last one before the training explodes. Then, we tune the weight decay using the best learning rate found. The values assessed for the weight decay are { 5 × 10 -4 , 5 × 10 -3 , 5 × 10 -2 , 5 × 10 -1 } . Finally, the results are reported in Table 1. We notice that SING combined with AdamW systematically outperforms AdamW. The evolution of the accuracy throughout training can be seen in Figure 3. SING seems to outperform AdamW during the entire training, but seem to loose its edge at the end of the training. We leave the study of this phenomena for future works.

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

Additionally, we trained a ResNet18 on CIFAR100 [22], which consists of 50K training images and 10K testing images from 100 classes. The images are of size 32 × 32 . The network was trained for 300 epochs using a batch size of 128. The learning rate scheduler and the tuning strategy are the same than for ImageNet. The results are visible in Table 1. We see that even in this challenging setting, the combination of AdamW and SING outperforms AdamW and SGD.

5.2 Depth Estimation

In this section, we investigate the performance of our optimizer on a depth estimation task using a synthetic dataset. The RDE dataset [7] consists of 50K 128 × 128 images of rectangles randomly placed within an image. Depth naturally arises as rectangles are placed on top of each other. The goal is to predict the depth for each pixel in the image, depending on which rectangle it belongs too. This task is interesting because although there exists a simple algorithm that computes the desired depth with 100% accuracy, neural networks struggle to get good performance. Notably, we found training a ViT-small [13] on this task in an image-to-image fashion to be particularly challenging using AdamW. For usual learning rates, the loss spikes randomly during training, largely lowering the final performance. See Figure 4 for more details. For very small learning rates, the training loss doesn't decrease fast enough to get results in a reasonable amount of time. In this case, we found using SING with AdamW to be a good choice as the normalization prevents the gradient from exploding during training. As a result, the combination of AdamW and SING outperformed AdamW by a large margin. The larger the assessed model, the worse the instabilities. ViT-big [13] does not converge when using AdamW. We tried several sets of hyper-parameters to draw this conclusion.

We used the same hyper-parameter tuning strategy and learning rate scheduler as for ImageNet-1K. The network was trained for 100 epochs using a batch size of 512. The loss we minimized was the MSE. See Table 2 for the results and an ablation study. The ablation study shows that each component of SING helps to achieve a higher learning rate and therefore higher performance. Notably, softplus [40] seems to largely help SING while it is detrimental for AdamW. The normalization seems to be a determining factor for reaching convergence although it does not fully explain the success of SING. We also studied the impact of SING when combined with other optimizers. The results are visible in Table 3. We used all methods with their default hyper-parameters except for SGD where we tried different values for the momentum. We see that for three out of the four assessed optimizers, the variant with SING significantly outperforms its counterpart. For AdaFactor [37] there is barely any performance gain. We claim this is due to the numerous tweaks within the optimizer that have been tailored for a gradient descent without SING.

5.3 Natural language processing

In this section, we evaluate the performance of our optimizer on natural language processing tasks. First, we trained a Transformer with pre-norm convention [45] on the IWSLT14 German-to-English (De-En) dataset [4] using the FAIRSEQ [31] library. We used the code of AdaHessian [48] as is but surprisingly we were not able to reproduce the results reported for AdamW. Instead, we used the hyper-parameters reported in [46] and found them to be better, but still below the announced results. Then, we used Hugging Face TRANSFORMERS library [44] to fine-tune Bert [11] on the SQuAD dataset [35] and RoBERTa on SWAG [53]. The results are reported in Table 4. In all cases, the combination of AdamW and SING outperforms a well-tuned AdamW.

We noticed that the gradient norm was increasing throughout training. After investigation, it turned 272 out the culprits were the weights and biases of the Layer Normalization [2] layers. We decided to 273 disable the learning of these weights and found the performance of both optimizers to be improved. 274 We claim doing so is not problematic in practice as disabling the learning of these parameters have 275 been pointed out as beneficial in the literature [47]. 276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

6 Limitations

We tried SING on other tasks such as image denoising, where the models trained with SING attained the same performance of AdamW, but did not result in improved results. This suggests SING's effectiveness can vary depending on the task and architecture. Additionally, we found our optimizer to not work well when used in conjunction with LayerNorm [2] or LayerScale [41]. While a simple fix is to disable the learning of these weights, it raises the question of why and calls for a better solution. Finally, we propose a convergence proof of the iterates defined in 2, which does not incorporate AdamW even though we mainly used their combination in the paper.

7 Conclusion

We introduced SING, a plug-and-play technique that improves the stability and generalization of the Adam(W) optimizer in deep learning, and can be used with any optimizer. By leveraging layer-wise gradient standardization, SING enhances the performance of the optimizer without introducing additional hyperparameters. Extensive experimentation across various tasks demonstrates its effectiveness compared to the original AdamW optimizer. Theoretical analysis reveals that SING enables the optimizer to escape narrow local minima within a single step, with the required learning rate inversely proportional to the network's depth. This compatibility with deep neural networks highlights its practicality. The analysis also provides valuable insights into the behavior of SING, such as its convergence rate or stability, and its advantages over traditional optimization techniques.

In conclusion, our proposed SING technique offers a practical and effective upgrade to the Adam(W) optimizer in deep learning. By enhancing stability and generalization, it contributes to the improvement of optimization algorithms in neural network training. The results of our research, combined with the theoretical analysis, open avenues for further exploration, including investigating the compatibility of SING with other optimization frameworks and addressing the challenges associated with specific normalization techniques.

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

References

Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375 , 2018.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Sunitha Basodi, Chunyan Ji, Haiping Zhang, and Yi Pan. Gradient amplification: An efficient way to train deep neural networks. Big Data Mining and Analytics , 3(3):196-207, 2020.

Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign , pages 2-17, 2014.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Yaim Cooper. The loss landscape of overparameterized neural networks. arXiv preprint arXiv:1804.10200 , 2018.

Adrien Courtois, Jean-Michel Morel, and Pablo Arias. Investigating neural architectures by synthetic dataset design. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4890-4899, 2022.

Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning , pages 2260-2268. PMLR, 2020.

Alexandre Défossez, Léon Bottou, Francis Bach, and Nicolas Usunier. A simple convergence proof of adam and adagrad. arXiv preprint arXiv:2003.02395 , 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Jianbang Ding, Xuancheng Ren, Ruixuan Luo, and Xu Sun. An adaptive and momental bound method for stochastic learning. arXiv preprint arXiv:1910.12249 , 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Shiv Ram Dubey, Satish Kumar Singh, and Bidyut Baran Chaudhuri. Adanorm: Adaptive gradient norm correction based optimizer for cnns. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 5284-5293, 2023.

Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.6544 , 2014.

Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. Advances in neural information processing systems , 32, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Lei Huang, Xianglong Liu, Yang Liu, Bo Lang, and Dacheng Tao. Centered weight normalization in accelerating training of deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 2803-2811, 2017.

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 , 2016.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, Int. Conf. on Learning Representations , 2015.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.

Guillaume Leclerc, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. FFCV: Accelerating training by removing data bottlenecks. https: //github.com/libffcv/ffcv/ , 2022. commit 2544abd.

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 , 2019.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. Adaptive gradient methods with dynamic bound of learning rate. arXiv preprint arXiv:1902.09843 , 2019.

Agnes Lydia and Sagayaraj Francis. Adagrad-an optimizer for stochastic gradient descent. Int. J. Inf. Comput. Sci , 6(5):566-568, 2019.

Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, et al. A theory on adam instability in large-scale machine learning. arXiv preprint arXiv:2304.09871 , 2023.

Ryan Murray, Brian Swenson, and Soummya Kar. Revisiting normalized gradient descent: Fast evasion of saddle points. IEEE Transactions on Automatic Control , 64(11):4818-4824, 2019.

Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038 , 2019.

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning , pages 1310-1318. Pmlr, 2013.

Jeffrey Pennington and Yasaman Bahri. Geometry of neural network loss surfaces via random matrix theory. In International Conference on Machine Learning , pages 2798-2806. PMLR, 2017.

Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch-channel normalization and weight standardization. arXiv preprint arXiv:1903.10520 , 2019.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 , 2016.

Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems , 29, 2016.

Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory 395 cost. In International Conference on Machine Learning , pages 4596-4604. PMLR, 2018. 396

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re397 thinking the inception architecture for computer vision. In Proceedings of the IEEE conference 398 on computer vision and pattern recognition , pages 2818-2826, 2016. 399

400

401

T. Tieleman and G. Hinton. Lecture 6.5 - rmsprop, coursera: Neural networks for machine learning. Technical report, 2012.

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

Qianqian Tong, Guannan Liang, and Jinbo Bi. Calibrating the adaptive learning rate to improve convergence of adam. Neurocomputing , 481:333-356, 2022.

Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 32-42, 2021.

Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. Advances in neural information processing systems , 32, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-theart natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics.

Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning , pages 10524-10533. PMLR, 2020.

Haoran Xu, Benjamin Van Durme, and Kenton Murray. Bert, mbert, or bibert? a study on contextualized embeddings for neural machine translation. arXiv preprint arXiv:2109.04588 , 2021.

Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in Neural Information Processing Systems , 32, 2019.

Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence , volume 35, pages 10665-10673, 2021.

Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. Gradient centralization: A new optimization technique for deep neural networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 635-652. Springer, 2020.

Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.03888 , 2017.

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962 , 2019.

Wei Yuan and Kai-Xin Gao. Eadam optimizer: How ϵ impact adam. arXiv preprint arXiv:2011.02150 , 2020.

Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326 , 2018.

Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: k steps 443 forward, 1 step back. Advances in neural information processing systems , 32, 2019. 444

Richard Zhang. Making convolutional networks shift-invariant again. In International confer445 ence on machine learning , pages 7324-7334. PMLR, 2019. 446

Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic 447 normalized gradient descent. Science China Information Sciences , 64:1-13, 2021. 448

Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon 449 Papademetris, and James Duncan. Adabelief optimizer: Adapting stepsizes by the belief in 450 observed gradients. Advances in neural information processing systems , 33:18795-18806, 451 2020. 452

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SING, a simple gradient preprocessing technique that, combined with any optimizer of choice, argues for improved stability and generalization. The paper further provides a theoretical convergence analysis of the approach\n\nSTRENGTHS:\n1. The paper is clear\n2. The technique is simple and easy to implement.\n\nWEAKNESSES:\nOverall the paper proposes a straight forward extension to the gradient centralization method, where the gradients are also normalized in a pointwise fashion as in other adaptive techniques. The main weaknesses of the paper are:\n\n1. Incremental - i do not think the contribution of this paper merits publication due to its incremental nature. Adaptive optimizers already normalize gradients in a similar way and it is not clear what is added in the proposed method. \n\n2. Non-convincing experiments - Only 1 experiment compares gradient centralization + AdamW (GC + AdamW), which was proposed in [1], which is the closest method to the one proposed in the paper.  By that experiment GC + AdamW already achieves approximately the same performance, hence stripping SING from any practical significance. I do not understand why GC + AdamW is not used as a baseline for other experiments as it clearly shows strong performance. At it stands, it is not clear whether the apparent improvement of SING stems from the GC part of SING, or the added normalization which constitutes it novelty. I would encourage the authors to add this ablation study to the paper to make it more convincing. Finally, results in the paper do not include standard deviation which is be a must for an empirical paper.\n\n3. The theory can be equally applied to other adaptive optimizers, hence it is not special to SING\n\n\n\n\n[1] - Yong et al - \"Gradient Centralization: A New Optimization\nTechnique for Deep Neural Networks\"\n\nQUESTIONS:\nI would appreciate if the authors could clarify what is the motivation to normalize the gradients element-wise when this is already done in any adaptive technique (in various forms that might not match SING exactly)\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a simple and hyper-parameters-free way to improve the stabilization and generalization properties of optimizers used in deep-learning scenarios. They show that with gradient centralization and gradient normalization methods, SING can escape the local minima with large step sizes theoretically. The authors provide several experiments on datasets like ImageNet-1K, RDE, and some NLP tasks to show the superiority of SING together with popular optimizers like AdamW. They also give out some other theoretical results like convergence and invariance properties.\n\nSTRENGTHS:\n1. The experiments on real datasets show that SING+AdamW performs significantly better than other baselines at image classification, depth estimation, and NLP tasks. This efficient method is also simple and not requires additional hyper-parameters.\n2. The authors show that SING can escape the basin of attraction of the critical point when the step size is sufficiently large, and the stepsize threshold is inversely proportional to the network's depth, while GD cannot. And the experiments result (Figure 4) show that SING can stabilize the performance of the optimizers like AdamW.\n3. The paper is well-writen and easy to follow.\n\nWEAKNESSES:\n1. Although the empirical results are remarkable, the novelty of this paper is limited. As mentioned in the paper, gradient centralization[1] and gradient normalization[2,3,4] are common methods in the previous works, and this paper combines these two methods and systematically investigates the properties of SING.\n2. The theoretical analysis just focuses on the gradient rather than combining it with momentum and scheduler, but since the gradient is normalized and the step size is large, the momentum and learning rate scheduler is critical for the global convergence, like in Thm 3.3, 3.4, $\\eta$ is small, but $\\eta$ needs to be large to escape local minima from Thm 3.1.\n3. Thm 3.3, 3.4 requires that the mini-batch size B be some concrete value, this is too strict and it's better to relax this assumption.\n\n[1] Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. Gradient centralization: A new optimization technique for deep neural networks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16, pages 635–652. Springer, 2020\n[2] Ashok Cutkosky and Harsh Mehta. Momentum improves normalized sgd. In International conference on machine learning, pages 2260–2268. PMLR, 2020.\n[3] Ryan Murray, Brian Swenson, and Soummya Kar. Revisiting normalized gradient descent: Fast evasion of saddle points. IEEE Transactions on Automatic Control, 64(11):4818–4824, 2019.\n[4] Shen-Yi Zhao, Yin-Peng Xie, and Wu-Jun Li. On the convergence and improvement of stochastic normalized gradient descent. Science China Information Sciences, 64:1–13, 2021.\n\nQUESTIONS:\n1. The results in figure 3 seem not fully converged, can the author propose the results with larger total training epochs?\n2. If the author can relax the assumption of the mini-batch size B in Thm 3.3 and 3.4, similar to weakness-3? \n3. The W(x) in definition 3.1 should be A(x) in the later paper?\n\nLIMITATIONS:\nThe author mentions that their method cannot be used together with LayerNorm or LayerScale, and the theoretical results not incorperate with AdamW.\nI don't find ethical or immediate negative societal consequences in this work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, authors have proposed a method (called SING) for stabilizing the optimization algorithms used in training of deep models. The proposed method is based on only a layer-wise standardization of the gradients without introducing any additional hyper-parameters. In addition, a theoretical analysis for convergence to a stationary point is provided. Extensive empirical simulations show improvement of the training performance when the existing optimization algorithms use the proposed approach on various tasks.\n\nSTRENGTHS:\nIn general, the paper is well-written, and concepts have been presented in an accessible way. Providing theoretical results, including the convergence and invariance properties provide more credibility to the proposed method. Furthermore, experiments on different architectures and on various datasets is another strength point of this paper.\n\nWEAKNESSES:\nI need some clarifications on the followings:\n\n1 - As mentioned in the theorems 3.3 and 3.4, convergence is guaranteed only to a stationary point. On the other hand, Theorem 3.1 states that the algorithm can escape from a narrow local minimum. How can SING guarantee that the stationary point is a local minimum (what happens if the algorithm converges to a saddle point or even local maximum) ?\n\n2 - What defines the narrow local minimum and the wide local minimum. There is no curvature information/notion in Theorem 3.1 to distinguish local flat minimum from the sharp one.\n\n3 - In Theorem 3.3, $\\epsilon^2$ is given by $\\sigma^2/B$, so to have an arbitrary small error on the expectation of the gradient at some stationary point, $\\sigma$ should scale as $\\mathcal{O}(\\frac{\\sqrt{B}}{D})$. My question is for a very large model, (i.e., $D$ is huge), does the assumption (8) hold for every $x\\in\\mathbb{R}^p$? I am not sure how the assumption holds for a highly non-convex loss in a large deep model? This assumption is stronger assumption than other approaches. Typically, ADAM, and other optimization in deep learning either assume some level of convexity or use somehow reasonable assumptions like small gradient, or bounded sequence of estimates, etc.,\n\n4 - Regrading the previous point, it is a good idea to run an experiment to illustrate the effect of $D$ on the training performance with SING.\n\n5 - Do the results in experiment section (Table 1, 2, and 3) show the validation accuracy or the training accuracy ? Please clarify this.\n\n6 -The convergence result doesn't provide any insight for the generalization to the unseen data. It is a purely an optimization perspective.\n\nQUESTIONS:\nThe largest ball contained within the basin of attraction in Definition 1.1 is denoted by $\\mathcal{B}$; however, in other places, authors use $A()$, am I right?\n\nI couldn't find the definition of $(\\epsilon, \\phi)$-stationary point used in Theorem 3.4.\n\nLIMITATIONS:\nPlease see my comments for Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents SING (StabIlized and Normalized Gradient), a new method designed to enhance the stability and generalization capabilities of the Adam(W) optimizer. SING involves a layer-wise standardization of the gradients that are input into Adam(W), and does not require the introduction of additional hyper-parameters. This makes it straightforward to implement and computationally efficient.\n\nThe authors demonstrate the effectiveness and practicality of SING through improved results across a broad range of architectures and problems, including image classification, depth estimation, and natural language processing. It also works well in combination with other optimizers.\n\nIn addition to these experimental results, a theoretical analysis of the convergence of the SING method is provided. The authors argue that due to the standardization process, SING has the ability to escape local minima narrower than a certain threshold, which is inversely proportional to the depth of the network. This suggests that SING may offer significant advantages in training deep neural networks.\n\nSTRENGTHS:\n1. As the authors have claimed, the proposed method can be applied in a plug-and-play style with impressive applicability to a lot of tasks, datasets and optimizers. Without additional hyperparameters introduced, I think this work has great potential to become a standardized training technique with big impact in the community. And the authors did provide the elegantly implemented source code in PyTorch which I think is already close to ready to be included in the standard PyTorch library.\n\n2. Empirical performance is impressive with huge improvements over baseline optimizers in many settings.\n\n3. Mostly the paper is written in quality and easy to follow with only a few ambiguities, which I will mention in the weaknesses part.\n\nWEAKNESSES:\n1. Firstly, I believe there is a misalignment between the theoretical analysis and practical method. To be specific, the analysis in Theorem 3.1 compares the learning rate needed for escaping local minima for SING and SGD. However, as the authors claimed previously, the SING algorithm is proposed to overcome the limitations of Adam(W). Therefore, it would be better to directly analyze SING against Adam(W), which is also mainly compared against in the experiment part.\n\n2. Some issues in terms of writing. One is that the authors did not formally formulate the centralize operation in math equations but only in codes, which could be confusing for readers who are not familiar with PyTorch framework. I strongly recommend the authors to provide strict math formulations instead of ambiguous codes only. For example, at least I am still confused the mean operation is executed over which dimension and what the meaning is for that averaging. A second issue in writing is that it seems the authors interchangeably use the terms \"learning rate\" and \"step size\" in Section 1 but treat them as different things in Section 3. I hope the authors could clarify the differences or use one term consistently.\n\nQUESTIONS:\n1. According Figure 4, the spikes occur once during one training process. Do the authors have any comments on what causes the spikes exactly?\n\n2. The second row of Table 2 seems to be wrongly presented.\n\n3. What if the momentum also comes into the picture to be combined with SING?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes SING, a plug-and-play approach to enhance optimizers without introducing additional hyperparameters.\n\nThe idea consists of standardizing gradients in a layer-wise manner prior to the host optimizer’s execution, and is motivated by factors such as easier escaping of narrow minima and invariance properties.\n\nExperiments on image classification, depth estimation, and NLP tasks such as NMT and QA are used to assess the proposed method’s performance and measure improvements over host optimizers.\n\nSTRENGTHS:\nThe paper is written clearly and well-organized.\n\nThe proposed method is easy to implement and works in a plug-and-play fashion. The layer-wise gradient standardization can be viewed as a gradient pre-processing step and hence completely agnostic to the host optimizer, making the approach general and flexible.\n\nThe authors provide theoretical analyses to motivate and better understand SING. The convergence analysis considers the smooth non-convex bounded variance setting, which I believe to be a good balance between assumptions and how well it captures network training.\n\nExperiments include different tasks from two domains and distinct network architectures, including ResNets and Transformer-based models.\n\nWEAKNESSES:\nThe theoretical analysis is not helpful in motivating or better understanding SING’s benefits.\n\nWhile SING adopts layer-wise normalization, it seems that the analysis holds given any partition of the $p$ many parameters in $D$ many sets – i.e. the fact that each of the $D$ tensors is assumed to correspond to a different layer is not necessary nor used anywhere in the analysis. We can then consider the effect of variable $D$ for a fixed $p$ (grouping the parameters in larger or smaller sets, say filter-wise, kernel-wise, or even parameter-wise), and we recover Normalized Gradient Descent (NGD) with $D=1$ and a form of sign SGD with $D=p$ (this has been studied in previous works to understand how normalizing gradients of coarse/fine-grained parameter sets can affect performance and convergence).\n\nThis has two concerning implications:\n\n- For Theorem 3.1\n\nSince $||\\frac{g}{\\Gamma (g)}|| = \\sqrt{D}$, it follows that the post-processed gradients will scale (in norm) as $\\sqrt{D}$, and hence $\\eta_{SING}$ in Theorem 3.1 is fundamentally ‘undoing’ this scaling, hence claiming that ‘SING can escape local minima narrower than a threshold that is inversely proportional to the network’s depth’ is not very meaningful.\n\nA similar argument would be to pre-process the gradient by scaling it up by 100 and claiming that now we can use a 100x smaller learning rate, which, although technically correct, is not useful.\n\nWhile I’m aware that the layer-wise normalization will actually change the update direction and not just scale it, it seems that this change in direction does not play any positive role in the presented analysis.\n\nFinally, one could simply set $D=p$ (i.e. artificially view each parameter as an independent layer) and Theorem 3.1 would state that the sufficient learning rate to escape narrow minima would actually decrease way more aggressively (as 1/sqrt(# parameters) instead of 1/sqrt(# layers)) – this is clearly not actually useful since we’re just scaling up the (norm of) post-processed gradients (compared to the layer-wise case) and compensating by scaling down the learning rate.\n\n\n- For Theorems 3.3. and 3.4\n\nTo achieve stationarity $\\delta$ independent of $D$ (i.e. $\\epsilon \\propto \\frac{\\delta}{D}$) we would set $\\eta = \\Theta(\\frac{\\delta}{D})$, $T = \\Theta(\\frac{D^2}{\\delta^2})$, $B = \\Theta(\\frac{D^2}{\\delta^2})$, where we are ignoring dependencies on $F(x_0)$ and $L$.\n\nThis means that, in the original case where $D =$ # layers, the guarantee requires both the number of iterations and the batch size to increase quadratically with the depth of the model, which is concerning.\n\nMoreover, if we set $D=1$ (i.e. NGD) we actually minimize the required number of iterations and batch size. Therefore, these results do not motivate or support layer-wise normalization, and actually question this design choice by offering significantly better guarantees for NGD.\n\n- Other points\n\nAlthough the theoretical analysis considers updates following Eq. 2, the experimental studies heavily focus on AdamW + SING, which is not well-discussed. My main concern in this case is that the normalization from SING affects both $m_t$ and $v_t$ in the numerator and denominator of AdamW, respectively. It is unclear what is really happening in this case.\n\n\nIt seems that for long enough training time windows, if the layer-wise gradient norms remain roughly constant then the normalization effect would cancel out, reducing to AdamW’s updates (that is, if the $\\epsilon$ term in the denominator of AdamW is negligible). Accounting for $\\epsilon$, on the other hand, yields AdamW’s updates except with different values for $\\epsilon$ for each layer, each scaled by the layer’s gradient norm.\n\nAlthough it is unlikely that layer-wise gradient norms remain roughly constant for many enough iterations, this hints that SING’s normalization might be affecting the size of $\\sqrt(v_t)$ compared to $\\epsilon$ differently for each layer. This could result in confounding effects since the value of $\\epsilon$ (compared to $\\sqrt(v_t)$) can play a major role in the behavior of Adam-like methods, potentially improving the performance in multiple settings.\n\nThe experimental analysis could also be substantially improved. The hyperparameter tuning strategy (choosing best learning rate, then fixing it to choose best weight decay) can easily lead to suboptimal values, especially for SGD and any adaptive method that does not inherently incorporate AdamW’s weight decay decoupling (see Fig. 1 and 2 of Loshchilov & Hutter).\n\nThis can lead to an unfair advantage to AdamW and AdamW + SING over all other methods. The cosine schedule is also known to improve AdamW’s performance and more often than not harm SGD (compared to step-wise), hence it would be valuable to also collect results with a step-wise schedule for a more comprehensive and clear comparison.\n\nThere is also some loss in novelty from the fact that the actual method that plays the key role in the experiments also adopts LookAhead and softplus calibration. In particular, centering is not novel although it has been explored more extensively for the 2nd moment estimate (centered RMSProp, AdaBelief, SDProp, ACProp, etc), and layer-wise gradient normalization for adaptive methods has also been studied (AdaShift & AvaGrad – none of the two are cited or discussed). These methods should be included in the comparison to have a clear picture that would allow a proper assessment of SING.\n\nFinally, there are also concerns regarding the other vision tasks. It is unclear what was the exact ResNet-18 model used for CIFAR-100: if it is a ~11M param model, then it is a wider version (DeVries ResNet-18) which differs from the one originally proposed by He et al. and achieves over 77% acc. on CIFAR-100 when trained with SGD (see LookAhead’s paper and DeVries&Taylor).\n\nThis would indicate issues with the CIFAR-100 results in Table 1, since results with SGD would be ~1.4% worse even with additional augmentation and 100 extra training epochs. As for depth estimation, the dataset is synthetic and not well studied, hindering a proper assessment of its results. Nonetheless, ViT’s are typically well-trainable with SGD if warmup and gradient clipping are employed (which is common practice for these models).\n\nThe fact that SGD achieved 0.25% accuracy suggests that the experimental setup should be revised – warmup and grad clipping should be adopted for SGD since they are common practice, especially if SGD only achieves 0.25% accuracy without them.\n\nQUESTIONS:\nSee my points above regarding points for improvement.\n\nLIMITATIONS:\nThe authors discuss limitations satisfactorily.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-17"}