Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks
Authors: Matteo Tucat
Anirbit Mukherjee
Mingfei Sun
Procheta Sen
Omar Rivasplata
Venue: Accepted by TMLR ()

Abstract: We present and analyze a novel regularized form of the gradient clipping algorithm, proving that it converges to global minima of the loss surface of deep neural networks under the squared loss, provided that the layers are of sufficient width. The algorithm presented here, dubbed $\delta-$GClip, introduces a modification to gradient clipping that leads to a first-of-its-kind example of a step size scheduling for gradient descent that provably minimizes training losses of deep neural nets. We also present empirical evidence that our theoretically founded $\delta-$GClip algorithm is competitive with the state-of-the-art deep learning heuristics on various neural architectures including modern transformer based architectures. The modification we do to standard gradient clipping is designed to leverage the PL* condition, a variant of the Polyak-Łojasiewicz inequality which was recently proven to be true for sufficiently wide neural networks at any depth within a neighbourhood of the initialization.

Published in Transactions on Machine Learning Research (06/2025)
Regularized Gradient Clipping Provably Trains Wide and Deep Neural Networks
Matteo Tucat"
mattestucat@@maallcomm com
Deportment of Computer Science The University of Manchester
Anirbit Mukherjee
Deportment Computer Science The University of Manchester
Mingfei Sun
wiinget..su.@mannchseerrr.....
Department of Computer Science The University of Manchester
Procheta Sen
pracheta.senn@licerpooo..
Department Computer Science Universily of Liverpool
Omar Rivasplata
Department of Computer Science The University of Manchester
Reviewed OpenReview: https:// openrreveee,.. net/ pormm? id= ABT1XQLbOm
Abstract
We present and analyze novel regularized form of the gradient clipping algorithm, proving that it converges to global minima of the loss surface of deep neural networks under the squared loss, provided that the layers are of sufficient width. The algorithm presented here, dubbed 6-GClip, introduces modification to gradient clipping that leads to first-of-its- kind example of a step size scheduling for gradient descent that provably minimizes training losses deep neural nets. We also present empirical evidence that this theoretically founded 8- 8-GClip algorithm competitive with the satto-of-thh-art deep learning heuristics on various neural architectures including modern transformer based architectures. The modification do to standard gradient clipping is designed to leverage the PI. condition, a variant of the Dollykk-Lojasiwwice inequality which was recently proven to be true for sufficiently wide neural networks at any depth within neighbourhood of the initialization.
1 Introduction
In various disciplines, from control theory to machine learning theory, there has boen long history of trying to understand the nature of convergence
non-convex objectives for first-order optinization algorithms, i.e. algorithms which only have access to (an estimate of) the gradient of the objective (Maryak & Chin, 2001; Fang et al., 1997). The new incarnation of this question in optimization problems in high dimensions, which arise in modern machine learning applications including neural network training, motivate the need for finite-time analysis of such algorithms. However, a challenging aspect of these modern use cases is their reliance on fine-tuning of some byper-parameters, such as the step size, momentum, and batch size. the wake of this, the "adaptive gradient" algorithms, such as Adam (Kingma & Ba, 2014) have become essentially indispensable for deep learning (Sun & Spall, 2019; Melis et al., 2018; Bahar et al., 2017).
"ork done while student at the Department of Computer Science, University of Manchester. 1 Coorrsppondigg author.

---

Published in Transactions on Machine Learning Research (06/2025)
The widespread popularity of adaptive gradient methods in deep learning, like Adam, arguably stems from the fact that it seems easy to find lask-specific and useful neural nets for which the default settings of these algorithms work well out of the box. Adam-like methods use as their update direction a vector which is the image of a linear combination of some (or all) of the gradients seen until the current iterate, under a linear Lransformation often called the "diagonal pre-conditioner" conatiioooo" out of the history of the gradients. It generally believed that this "pre-conditioning makes these algorithms much less sensitive the selection of its hyper-parameters An important precursor Adam was the AdaGrad algorithm (Duchi et al., 2011). The far-reaching usefulness of adaptive gradient methods has motivated significant attempts at their theoretical justifications in the non-convex setting.
However, to the best of our knowledge, there has not been so far a theoretical guarantee for any adaptive gradient algorithm to converge to the global minima of deep neural network loss surfaces.
On the other hand, in recent times a number of motivations have come to light to consider training algorithms beyond these conventional adaptive gradient algorithms (Bernstein et al., 2018). In works like Simsekli et al. (2019) and Zhang et al. (2020b), a number of reasons have been pointed out as to how gradient clipping based adaptivity is better suited for deep learning. In this kind of adaptivity, the primary interest is on mechanisms to prevent the algorithm from using arbitrarily large gradients. Gradient clipping has been successfully deployed in a wide range of problems, particularly in natural language processing tasks such as GPTss (Brown et al., 2020) and LSTMs (Merity et al., 2018), and more recently in computer vision tasks (Brock et al., 2021). Clipping the gradient is also known to alleviate the problem of exploding gradients in recurrent neural networks (Pascanu et al., 2012; 2013), as well as helping to provide privacy guarantees in differentially private machine learning (Abadi et al., 2016; Ma et al., 2023). However, as for Adam, gradient clipping, too, has no known theoretical guarantee of training neural nets.
Inspired by the above, in this work we introduce form of gradient clipping which in experiments we demonstrate to be competitive with Adam, vanilla stochastic gradient descent (SGD) and standard gradient clipping, while also being guaranteed to train neural nets of arbitrary depth when training on the squared loss and when the layers are sufficiently wide. Our proof crucially leverages the novel PI, condition that was proven for such nets in Liu et al. (2022). Thus, we give a first-of-its frst-of-its-kind deep learning algorithm that is of practical benefit while being rigorously provable to minimize loss functions of deep neural nets.
Summary of Results
In Zhang et al. (2020a), the following specific form of gradient clipping (which from here onwards we will refer to "standard gradient clipping" or "GClip") is studiod.
Definition 1 (GClip). For any 7,72 > 0, the standard gradient clipping (GClip) algorithm for a differentiable objective function f is defined as
Øt+1 F. -x: -((x)))))))))))))))))()) v V (x:),
with h(x;)= =1 min
[v)()))]
The term y acts as the gradient norm threshold. To the best of our knowledge, this algorithm has no known convergence guarantees for deep learning, motivating us to present a modification of GClip, which we refer to as d-Regularized-GGlin or 8-GClip for short.
Definition (d-Regularizet-GCli)) For 8 (0. 1) and n,7 > O, the G-Regularize--GGIipp (8-GClip) algorithm for differentiable objective function is defined
xt+1 x; h(xx)) vf(x;),
with h(xx)) = n min
max
7
[v
The critical max(6,...) term ensures h(x;) 2 nō, thus preventing h(x;)) from vanishing as [V](x)] + 00. It
is important to note that, due to this modification, the distance between any two iterates [z~1 2/|| is not bounded the gradient norm grows
2

---

Published in Transactions on Machine Learning Research (06/2025)
Written alternatively, 8- 6-GClip for å (0, implements GD with gradient norm dependent step-length i.c
#4+1 Bt he vV((2))
7
if [V 5y
with
h
il y < V (())) s
(1)
[+x)
no if v
Note that setting = 0 in the definition of 8-GClip above recovers standard GClip (Definition 1). And 8 = 1, 8-GClip is the standard GD at constant step-length. Thus 5-GClip be viewed particular choice of contimuously interpolating algorithms between gradient clipping and gradient descent and we will prove and demonstrate that E (0,1) provides for wanted deep-learning properties not known to be obtainable al either extremities.
In the experiments given in Section 3 we shall see that, in practice, ô has to be chosen very small; though its presence is critical for the convergence guarantee that we shall establish in Theorem 2.1, which the main theoretical contribution for our algorithm 8-GClip.
The informal description of our key result is follows: Given deep neural network that is sufficiently wide (parametric in 8), 8-GClip will minimize the squared loss to find a zero-loss solution al exponential convergence rale, for any training data. To the best of our knowledge, 8-GClip is the first instance of an adaptive gradient algorithm or GD with a non-trivial learning rate schedule, that provably minimizes the empirical loss of neural nets at any depth. Additionally, our experiments show that 6-GClip offers competitive performance when compared against state-of-the-ar deep learning optimizers for architectures (like transformers) and losses far outside the ambit of the proof given.
6=
8=1
5-GClip
Standard Gradient Clipping Not Proven for Neural Nets Works Well Practice
Gradient Descent
Provable Trainer Large Widths Competitive in Practice
Provable Trainer at Large Widths Doesn't Work Well Practice
Figure 1: A Summarized View of 8-GClip.
A stochastic version of d-GClip will also be considered in Section 2 (cf. Definition 5), which is relevant when under a certain noisy gradient setup, and we state convergence result for it in Theorem 2.2.
Organization
In Section 2 we give the formal statements of our convergence theorems for our adaptive gradient method, 5-GClip. In Section 3 demonstrate empirically that our algorithm can compete with various popular deep learning heuristics benchmark tests with VAE, ResNet-18, Vision Transformer and BERT based model. In Section 4 we review the curentt literature on provable deep learning optimization algorithms, focusing on adaptive gradient algorithms. In Section 5 we give the full proof of our main theorem on provable minimization of the squared loss for deep nots by our 5-GClip when using full gradients. We conclude, in Section 6, discussing possible future directions of research. Appendix contains the proof convergence of stochastic 5-GClip.
Notation The Euclidean ball centered wy e I of radius R B( Wp. R) (w R | w|12 s R).
Unless otherwise stated, |-||| denotes the f--orm for vectors and the spectral norm for matrices.
3

---

Published in Transactions on Machine Learning Research (06/2025)
2 Theory for ò- d-Regularized-GGii
This section contains theory results for 6-GClip (Theorem 2.1) and for its stochastic version (Theorem 2.2). Towards the theory results, we first outline technical definitions.
Definition 3 (μ-PL* Condition). non-negative loss function L is said to satisfy j-PI. on a set R if 3u > 0 such that Vw é S: [V[(w)]' pf(w).
Next, the I-hidden layer forwrd neural network architectures are recalled, and their loss setups, which were within the ambit considerations in Liu et al. (2022).
Definition A neural network is given by
a =0 ( wo for le [1, 1].
f(w;)) = a(++))
(0)) xz
where W(O) m is the matrix of connection weights for the I-th layer, my is the width of the Ith layer with mæ+1 1, and ()) is the output from the I-th layer, after the activation is applied. Also, the "weight vector' w encompasses all weights across all layers. We assume that the last layer activation 0r4 is Lo Li-isscchtz contimuuus and -g-Lipschitz smooth (8,-smooth), and satisfies | p>0.
We train the weights of f(w,-) using an n-sample training dataset, {z, (z;, vi) I....,n.. We write F(w) (/(w:: f(w; a.,) I for the vector of outputs for all training samples, and y (y),
We use the squared loss [(w)= = [ - -
Now we have all the requisite background to state the key theorem pertaining our algorithm 8-GClip.
Theorem 2.1 (o-Regularizee-GClii Provably Trains Wide and Deep Neural Nets). Suppose overparametrized neural network f is being trained using the square loss L(w), specified in Definition 4. Then 3 Ao > 0 s.t for any p. > 0 appropriately small enough and 5 1, if the minimum width of the network layers satisfies,
= 52 nRbL+2
TV 23 ((wo)
with R=
(2)
(No 100-Hpp 2)2
178p
then one can initialize the weights s.t. w.h.p over initialization, the above loss is p-PL" in the ball B(wo, R) around initialization wo. Furthermore, let 87 be s. F(w) is locally 85-smooth in B(wo, R). Then, training such network using ò-R d-Reeularized-GGC GClip with n min{ 1.) results in geometric convergence global minimizer w. B(wo, R) such that f(w.)= and at rate given as,
((w:) s [(wp) (1- (1-1-761))
(3)
Remark.
The assumptions of 1 < 1/p and 8 imply (1 1 nohu) (2, hence lim L(w;) = 0.
t-000
This theorem fulfils the theoretical guarantee for our 8-GClip algorithm. The proof is deferred to Section 5.
Next, consider stochastic version of our 8-GClip algorithm, defined as follows.
Definition (Stochastic d-Rogularized-GCllip)) We define the stochastic &-Rogularizod-GGih algo-
rithm for differentiable function as
W w: - hhgi)-)) gr.
where
5, and E[g. w.] v v(w.)
h(g.) min 1, 1,
Igg
(4)
for any n, 0, 5 (0,1) and arbitrary choice of wi, the initial point.
4

---

Published in Transactions on Machine Learning Research (06/2025)
Towards analyzing the stochastic version of 6-GClip just defined, we make the following assumptions. Assumption 1. 0 s.t. Yw, [ s 0.
Assumption 2. L is non-negatively lower bounded i.e. min L= L. 2.
Assumption 3. L is 8-smooth.
The following theorem holds for stochastic 8-GClip.
Theorem 2.2 (Convergence of Stochastic b-Regularized-
Given Assumptions 1, and 3, and for an arbitrary choice of é > 0, let e' := e/0. Then, for 3 = 1, d = ( ((1+))(+++"""" +3)²), and 1= (2,62)// + +))
stochastic Srnguullazze-GCClij iterates satisfy the following inequality:
8t e4'
min E =
for T2
The proof of this theorem is given in Appendix A, where we first prove a slightly more general result.
We note that albeit &- 8-GClip only partially clips the gradient, the convergence guarantee above does not need the gradient norms to be bounded as was also the case for standard stochastic gradient clipping, cf. Theorem 7 in Zhang et al. (2020a). Nerl, we note that the convergence guarantee for standard stochastic clipping does not immediately hold as stated in Zhang et al. (2020a) for the standard smoothness assumption that is used here. Lastly, unlike standard gradient clipping, the work presented here offers convergence guarantees in the deterministic ("full gradient") sctting (Thoorem 2.1) as well as the stochastic ("noisy gradicnt") setting (Theorem 2.2), for the same clipping algorithm.
3 Experimental Evidence for The Performance of d-Regularized-GCIi
This section contains experimental evidence for d-GClip. We split our experimental demonstrations into two segments. In the next subsection we focus on certain conventional deep learning models and in the later subsection we focus on transformer based models. The experiments cover text as well as image data. We note that all the models considered here are outside the scope of the theory proven previously and hence the success of 8-GClip in such varied scenarios can be seen to robustly demonstrate its practical abilities as a deep learning algorithm. The code for all our experiments can be found in the GitHub repository."
3.1 Experiments for d-Regularized-GGlip on a ResNet and a VAE
We demonstrate here that the regularization term in d-Regularized-GGlii helps improve the performance of standard gradient clipping-wii which anyway outperforms stochastic gradient descent (SGD)-and and in fact competitive when compared against the most popular optimizers such as Adam, even surpassing it in some cases. We test in supervised classification as well as unsupervisod distribution lcarning settings.
We perform four sets of experiments. The first set is on the standard ResNet-18 (He et al., 2016) being trained on the benchmark CIFAR-10 (Krizhevsky, 2009) dataset, which we recall is a 10-class image classification task with 50, 000 training images and 10, 000 test images. The second set of experiments is training VAE model on the Fashion- MNIST dataset, with 60, 000 training samples and 10, 000 for testing. Further, each of these is done both with learning rate scheduling- whereby the learning rate 7) is reduced at certain points in the training and without such scheduling (constant throughout).
In the (supervised) classification experiments, the training is done using the cross-entropy loss and ReLU gate nets, and using weight-decay (of 5c-4). Whereas the VAE setup docs not have a loss function in the same conventional sense as considered in the theorems in Section 2. Hence, these experiments demonstrate the efficacy of regularized gradient clipping (8-GClip) beyond the ambit of the current theory.
We ran all experiments of this segment using a standard desktop with a GeForce RTX 2060 graphics card. We built custom implementations of Regularizedd-Glli and standard GClip and, importantly, used the standard Pytorch optimizers for SGD and Adam, which we recall is highly optimized. Hence, would be
5

---

Published in Transactions on Machine Learning Research (06/2025)
demonstrating performance of our modification in competitions which are priori skewed in favour of the existing benchmarks.
In the legends of the figures, the notation 'SGD (0.1)' stands for stochastic gradient descent with learning rate = 0.1, notation '6-GClip (1:1; stands for S-RReglarized GClip with 1j 1, = 1, = 1e-8, and 'GClip (5; for standard gradient clipping with 7 5, 7 = The notation 'Adam (1)' stands for Adam with 7) and similarly for other hyperparameter choices.
CIFAR-10. The ResNet-18 of size 512. We tested all the following hyperparameter combinations:
ResNet-18
trained using the full training set using mini batthba
{0.0001, 0.001, 0.1, 1,5), y {0.25, 1, 5, 10} and 8 {1c-3, 1c-8} for cach optimizer. For Adam, only the learning rate (n) was modified, the rest were left at the PyTorch defaults (31 0.9, A2 0.999, & le-8). In the case with scheduling the value quoted the legend denotes the 7 value at epoch 0, ie. before any reductions by the scheduling algorithm are done.
Experiments Without Learning Rate Scheduling. In Figure only plot the best cest-performing (in terms of test accuracy) hyperparameter selection for each algorithm.
ResNet-18 on CIFAR-10 (No scheduling)
95
Training Loss (log)
102
Test Accuracy (%)
90
101
85
100
80
150
200
50
100
150
200
Epochs
Epochs
Test Accuracy (%)
Training Loss
90
2
88
86
150
160
170
180
190
200
150
160
170
180
190
200
Epochs (last 50)
Epochs (last 50)
SGD (0.1)
6-GClip (1,0.25;1e-03)
6-GClip (1:0.25:1e-08)
GClip (1;0.25)
Adam (0.01)
Figure 2: CClp (6-GClip) is competitive against SOTA heuristics for training ResNet-18 CIFAR-10 without ceariing-rtt scheduling.
Experiments With Learning Rate Scheduling. In Figure we show a repeat of the above experiments and again plot the best-performing hyperparameters. In here, we start at larger values and divide by 10 at epochs 100 and 150, following the setup from Zhang al. (2020a).
For completeness, in Figure present version of the experiment above but without weight-decay for any of the algorithms considered.
We note that the performance of the gradient clipping based algorithms, well as Adam, do not show significant changes with the removal of weight decay; however, SGD performs significantly worse
We draw two primary conclusions from these results.
Firstly, that very small value of 8 in 8- Regulearizdd does not seem to have significant effect either for loss minimization test accuracy. The results for Sengullarzed-Gllii and standard GClip, set to similar n

---

Published in Transactions on Machine Learning Research (06/2025)
ResNet-18 18 on CIFAR-10 (LR scheduling)
Training Loss (log)
102
Test Accuracy (%)
95
10
90
10a
85
10
80
50
100
150
200
50
100
150
200
Epochs
Epochs
0.4
Test Accuracy (%)
Training Loss
0.3
94
93
92
0.1
91
150
170
190
200
160
180
200
Epochs (last 50)
Epochs (last 50)
SGD (1)
G-GClip (5:0.25:1e-03)
6-GClip (5:0.25;1e-08)
GClip (5;0.25)
Adam (0.0001)
Figure 3: -Regularized-GClin (6-GClip) outperforms other optimizers for training ResNet-18 on CIFAR-10
with learning-rate scheduling.
ResNet- 18 on CIFAR-10 (LR scheduling, no weight-decay)
Training Loss (log)
10a
Test Accuracy (%)
95
101
100
85
10
80
Epochs
Epochs
0.15
Test Accuracy (%)
Training Loss
0.10
91
0.05
90
89
150
160
170
190
200
150
160
170
180
200
Epochs (last 50)
Epochs (last 50)
SGD (1)
6-GClip (5:0.25;1e-03)
6-GClip (5;0.25;1e-08)
GClip (5;0.25)
Adam (0.0001)
Figure 4: -Regularized-GCli (6-GCLip) matches the best heuristics for training a ResNet-18 on CIFAR-10 with learning-rate scheduling, but no weight-decay.
and 7 values, practically identical in either scenarios for all small enough values of 8 tried. As alluded to

---

Published in Transactions on Machine Learning Research (06/2025)
in the previous sections, the gradient norm would have to be larger than 77/6 for the lower bound on h(w,) to be attained, and even for the larger setting of å 1e-3 and typical 0.25 setting requires gradient norm of over 250, which is only infrequently seen along the optimization trajectory.
Secondly, though Adam attained the best test accuracy without learning rate scheduling by margin of about ~ 1 percentage point compared to both B-Regularized--CClip and standard gradient clipping, but all other optimizers surpassed it by ~ 3 percentage points when learning rate scheduling was used. The best performance with scheduling (which is by our regularized gradient clipping) is beller than for any algorithm (ours or not) without scheduling. Interestingly, with learning rate scheduling Adam performed the best in terms of minimizing the training loss while SGD performed the worst, even though SGD's solution seems generalize significantly better (as shown by the ~ percentage point higher test accuracy).
The significant ability of d-regularized gradient clipping to exploit learning rate scheduling motivates an interesting direction for future exploration in theory.
VAE on Fashion-MNIITT. We performed the VAE training experiment both with and without scheduling when training on the Fashion-MNIST dataset. We tested the following hyperparmeter grid choices: {le-5, 1e-4, 1e-3, le-2), {10, 50, 200, 500), 8 e (01,,,0.111) We utilize the same scheduling as in the ResNet experiment (17 division by 10 at epochs 100 and 150), and the results are given in Figure 5.
VAE on Fashion-MNIST (LR scheduling)
27
27
Training Loss
26
26
Test Loss
25
25
24
24
50
150
50
150
200
Epochs
Epochs
26.0
26.0
25.5
25.5
Training Loss
Test Loss
25.0
25.0
24.5
24.5
24.0
24.0
23.5
23.5
150
160
170
180
190
200
150
160
170
180
190
200
Epochs (last 50)
Epochs (last 50)
SGD (0.0001)
6-GClip (0.001:200;0.1)
GCliip (0.001:200)
Adam (0.001)
Figure 5: d-Regularized-GGIi (6-GClip) is competitive against SOTA heuristics for training VAE on the Fashion-MNIST dataset with lcarning-ran rate scheduling.
The VAE results with (and without, though not shown here) learning rate scheduling supports our carlier observations that the added regularization term of 8 helps the performance w.r.t that of GClip the same values of step-length and clipping threshold, which anyway outperforms SGD. And it is only mildly underperforming with respect to Adam.
We therefore conclude from our experiments that &-Regularized -Clilarriie-GGlipppppp clipping remains competitive with current optimizers, while offering the significant benefit of provable deep neural network training.

---

Published in Transactions on Machine Learning Research (06/2025)
3.2 Evidence for The Performance of 6- -eegularized-GChae Transformers
In this section we focus benchmark state-of-the-art transformer-based architectures, that are not known to satisfy the p-PI,* condition as in the examples of the previous section. Yet, we show that our 8-GClip proposal continues to be competitive against the widely used heuristic of Adam.
In Figure 6 we show that starting from random initialization Adam's ability to train a Vision Transformer (Dosovitskiy et al., 2020) for the classification task on the CTFAR-10 dataset is matched in train and test loss and accuracy by our theoretically grounded proposal of 6-GClip 1j 0.2, 7 and 8 = 10-
Loss (ViT Tiny)
Accuracy (ViT Tiny)
2.0
Adam-train
Accuracy
0.6
Loss
-tlaa-G Adam-test delta- Celta-GClip-test
1.5
0.4
1.0
0.2
25
50
75
100
0
25
50
75
100
Epoch
Epoch
(a) VIT Tiny: embed dim=192, depth=12, num heads=3, MLP ratio=4
Loss (ViT Small)
Accuracy (ViT Small)
0.75
2
Accuracy
Adam-train delta- Seita-GClip-triin
Loss
0.50
Adam-test
Delta-GClip-tert
1
0.25
25
50
75
100
25
50
75
100
Epoch
Epoch
(b) ViT Small: embed dim=384, depth=12, num heads=6, MLP ratio=4
Figure d-GClip can be seen be competitive against. the SOTA heuristic of using Adam for training a Vision Transformer on the CIFAR-10 dataset.
Venturing further, we consider a fine-tuing task on a variant of the BERT (Devlin et al., 2019) model DisilBerEoosqqooooCaaaaiieennnnn on the datasct of sst2".
On this benchmark do a hyperparameter search over v and y values for 8-GClip to decide on the setting of 7= ==0.1, 0.1, 0.1 and as before set = 10-6, As shown in Figure 7, metric of test accuracy, yet again 5-GClip matches the standard heuristic which is a mix of standard gradicnt clipping and Adam. Hence in this experiment we see that our proposed 8-GClip can compele the blend of adaptive gradient algorithms that is usually deployed to train a BERT.
that the crucial performance
In Table we present further performance comparison between Adam and 8-GClip at different ós are pro- vided, for training the same ViTs studied in Figure 6. Similar comparisons are presented in Table 2 for the same BERT variant studied in Figure 7. We note that in all these studies the performance met- rics reported give the averaged results with the respective standard deviations over multiple runs of the experiment. Thus, from these tables we can robustly conclude that 5-GClip is competitive against Adam in transformer models where similar convergence guarantees are not yet available for 8-GClip, as was presented carlier. Lastly, from these tables we also note that the performance of 5-GClip, is extremely stable with orders of magnitude of variation in 8, in the ranges we consider. Thus the experiments lend themselves to
9

---

Published in Transactions on Machine Learning Research (06/2025)
109
0.90
0.85
10-1
Training Loss
Test. Accuracy
0.80
10
0.75
10-3
0.70
Average Training Loss (Adam GClip)
0.65
Average Test Accuracy (Adam CClip)
10
Average Training Loss (delta-GClip)
Average Test Accuracy (delta-GClip)
Epoch
Epoch
Figure 7: 6-GClip (6 = 6) can be seen to be competitive against the SOTA heuristic of using combination of Adam and GClip for training variant of the BERT model. (The shaded area shows the variation about mean performance over multiple runs.)
Transformer Experiments on the CIFAR-10 Datasel
Test Accuracy (avg.
Train Error (avg.)
Model
Algorithm
Last Iterate
Last Iterate
(6 0.01)
0.69740.002
0.700=0.003
Vision Transformer Tiny embed dim-192, 12pth-12, num heads=3, MLP ratio=4
(5 0.001)
0.701 #0.001
0.705
(5 0.0001)
0.701 70.000
0.706
(5 0.00001)
20.0440.0041
0. 0.703=0.000
(Adam)
0. 7.778800003
0.848=0.003
(5 = 0.01)
0. 0.751 20.003
0.697
Vision Transformer Small embed dim=384, depth=12, num heads=6, MLP ratio-4
(8 0.001)
0. 70.000
0. 0.693#0.008
(5 0.0001)
0. +0.009
00.05
66 0.00001)
0.754400.00
0.699400.00
(Adam)
0.672+0.004
0.975
Table 1: Comparison of Performance between 8- R-Regularized (at different 8) and Adam for experiments Vision Transformers.
the interpretation that the introduction of the small ó parameter in standard GClip is theoretical tool that "stabilizes" standard clipping in ways that lend it to provable guarantees without making any material difference to its performance.
Transformer Experiments on the sst2 Classification Datasel
Test Accuracy (avg.)
Train Error (avg.)
Model
Algorithm
Last Iterate
Last Iterate
(8 8 10-") (5 10-9) (5 - 1.25 10-7)
0.879*0.018
0.045
0.879+0.018
0.045+0.003
DasillBBertfsSqqqeeeeollssssaaii
0.879*0.018
0.045#0.003
(Adam)
0.871 0.871+0.004 40.004
0.003
Table 2: Comparison of Performance between 6-Regularized--Gllp (at different 8) and Adam for experiments
BERT variant.
10

---

Published in Transactions on Machine Learning Research (06/2025)
4 Related Works
In this section, we will summarize the state-of-the-art literature on standard clipping and provable deep learning optimization algorithms that are adaptive. At the very outset, we note that the Neural Tangent Kernel (NTK) approach to provable deep learning at large widths Jacot et al. (2018); Du et al. (2018); Chizat et al. (2019); Allen-Zhu et al. (2019b;a); Zou et al. (2020) is not within the scope of our review since, in general, they do not use slep-length scheduling, Furthermore, il is also known that NTK based predictors are outperformed by standard deep learning architectures (Chen et al., 2020b; Arora et al., 2019).
Literature Review of Theory for Adam. Adam was proposed in Kingma & Ba (2014) as an adaptive algorithm which requires hyperparamcters $1, E [0,1) to control the decay rates of exponential moving average estimates for the gradients and the squared gradients, respectively. In Reddi el al. (2018) it was proved that, for common hyperparameter choices (31 < v32),, there exists a stochastic convex optimization problem where Adam does not converge. They presented a modification to Adam that provably converges for online convex optimization. In De et al. (2018) for the first time the convergence of Adam was established in the deterministic case, without the use of convexity, but leveraging Lipschitz smoothness and a bounded gradient norm.
For the same optimization target above, in Chen et al. (2019), convergence rate of OlogT//VT)
shown for Adam-like adaptive gradient algorithms under the assumption of a bounded gradient oracle. Later, a burn-in stage was added in Staib et al. (2019) to prove O(1/VT) convergence rate, In Chen et al. (2020a); a partial adaptive parameter was introduced, and they proved convergence to criticality for a class of adaptive gradient algorithms, which does not include RMSProp. It was shown in Zou et al. (2019) that generic Adam (including RMSProp) converges with high probability under certain decaying conditions on Bn and step size, in contrast to the usual implementations. In Ward et al. (2019), similar convergence results were proved for AdaGrad, which is a special case of RMSProp.
Review of Theory for Adaptive Gradient Methods Training Neural Nets, In contrast to the convergence to criticality results mentioned above, there have also been works providing guarantees of con- vergence to a global minima for adaptive methods in shallow neural net training scenarios. Wu et al. (2019) provides a proof of the convergence of the AdaLoss adaptive algorithm to global minima on two-layer neural nets, under widths large enough to be in the NTK regime. Zou et al. (2023) provides a proof of Adam's global convergence two-layer convolutional neural networks to a zero-error solution whilst utilising weight decay regularization. Further, Zou et al. (2023) provide evidence that although both GD and Adam converge to zero-error solutions, GD's s solution generalises significantly better. In the context of Generative Adversar- ial Networks (GANs), Dou & Li (2021) analysed the performance of Adam-like algorithms and proved the convergence of Extra Gradient AMSGrad to an E-stationary point under novel assumptions they motivated.
Literature Review of Gradient Clipping. In the smooth non-convex case, Zhang al. (2020a) proved the convergence of deterministic gradient clipping to an e-stationary point under a new smoothness notion that is strictly weaker than standard Lipschitz smoothness. Their provided iteration complexity implies that gradient clipping can converge faster than gradient descent (in constants), while achieving ((s) -(s)ccriiccllity in ()= steps. They provide a similar analysis in the stochastic case, with the additional assumption of either a bound on the noise of the stochastic gradient or its distribution being symmetric sub-Gaussian. It is important to note that the provided stochastic iteration complexity does not supersede that of SGD in the general case. They had pointed out, possibly for the first time, that gradient clipping can converge, in deterministic as well as noisy settings, on smooth functions without the need for gradients to be bounded.
In Zhang et al. (2020b), Lipschitz smoothness is utilized while working with non-convex otimization targets and heavy-tailed gradient stochasticity to achieve (1/tt) close convergence to criticality in t steps, which matches that of SGD in the non-heavy-tailed setting. The same work gave lower bound in the same sctting, which matches up to conslants the run-time given above and thus proving that their convergence rate is worst-case optimal. Furthermore, the said work also considered non-smooth but strongly convex functions with a bound on the expected norm of the stochastic gradients- Shamir& & Zhang (2013) for heavy tailed settings- and achieve the same convergence, implying that the convergence rate is optimal even in the Lipschitz smooth and strongly convex setting.
we recall had appeared carlier in
11

---

Published in Transactions on Machine Learning Research (06/2025)
We posit that from above kinds of analysis of adaptive algorithms (including GClip), either for depth-2 neural networks or in the more general (non-)convex settings, there is no obvious path towards provable convergence guarantees in deep neural network training for adaptive gradient algorithms. However, in the recent work of Liu et al. (2022), convergence guarantees were proven for (S)GD for sufficiently wide, and arbitrarily deep neural worksrks, by leveraging the novel PI, condition that the samne paper proved to be true for squared losses for such neural nets. Next we will briefly review those results.
Review of the PI. Condition. Our motivation for studying the convergence characteristies of algorithms under the PL* condition comes from Liu et al. (2022), where it is proved that overparametrized feedforward, convolutional and residual (ResNet) neural networks can all satisfy the PL* condition within a finite radius of the initialization, if they are sufficiently wide. In particular, Liu et al. (2022) showed the following.
Theorem 4.1. Consider any neural network of the form described in Definition 4. If weight matrices are randomly initialized s.t. Wh ~ (O, (mxm--)) ) for le [0, L 1], and defining Ào == Amin (aainKK(uun (Kr(w))) 0 where then for any μ & Aop²) and the minimum layer width of the network being
n.R61+2
= Ω
(5)
(No 180-440 (20-Mp-2210
the p-PL," condition holds for the square loss in the ball B(wo: R) where is a finite radius.
Therefore, a path opened up, that by proving that the iterates of our --Regllarize-GGCin algorithm never leave a ball of finite radius, and proving the convergence of d-Regularized-GGIl on locally smooth μ-PI.' functions, we can argue for the algorithm's convergence to the loss global minima in such neural nets.
5 Proof of the Main Theory Result for d-Regularized-GClin
Towards giving the proof of the main theorem for 8-GClip algorithm (Theorem 2.1), we begin with listing some lemmas that needed for the proof.
5.1
Preparatory Lemmas
Lemma 5.1. Corresponding to constants a, b> > 0 and oj < suppose loss function L is 8-smooth, min C 0, and satisfies the p-PL" condition within a Euclidean ball B(wo. R), with R2
by 23 ((w)) Then there
ap exists a global minimizer w. B(w,, R) of L, such that L(w.) Furthermorc, given first-order adaptive step size algorithm of the form
wi-1 = w. w.- - h(w))) vL(w)),
(6)
where h(w:)) is time//ttrraee-dpeedent function such that 0 < h(w;) s b min 4), then the algorithm will converge with rate
((w,) s C(wo)( -1 ap)" app'
Lemma 5.2. The G-Regularized-GGlipp step size h(w) is bounded np h(w) 7, given that 0< 8 1.
Lemma 5.3. (6-Regularized-GGCi) Converges on smooth PL* functions.) Corresponding to positive con- stants 1,,8, f. such that 17 min 1/p} and 0 < 8 1, suppose there exists loss function L that is 8-smooth, lower-bounded by 0, and satisfies the f-PI." condition within an Euclidean ball B(wo, R) where R² NV 2V/23/E(wo) Then there exists global minimizer w, B(wo, R) of C, such that L(w,) = 0. Further- 1- -pp
more, 8- Ceegullrized-GCli will converge at rate
1 [(w,) s ((wo) T)6p)².
(8)
The proofs for Lemmas 5.1, 5.2 and 5.3 can be found in Subsection 5.3.
12

---

Published in Transactions on Machine Learning Research (06/2025)
5.2 Proof of Theorem 2.1
Proof. Firstly, we invoke the assumption that the initialization is s.t that the conditions of Theorem 4.1 apply, which we know from therein to be high-probability event. In particular we conclude that L satisfies p-PL* within a finite ball B(wy R) for some R and that the tangent kernel at initialization is positive definite.
If La and 3o are the Lipschitz constant and the Lipechitz smoothness coefficients for the activation o, then it was shown in Liu et al. (2022) that for the prediction map F we have its Lipschitz constant,
L s La (V]K>(wb)]- R/ā O(R()/mm))
//m)).
well as its smoothness constant,
35 s B Lo (Vl)= - O(F4) Ly O(H2L /Vm)
where Ky is the neural tangent kernel (recall that K5 DF(w))D((w))
By plugging in the lower bound on specified in the theorem, we get that both Ly and By are upper bounded by a constant and thus m-independent. If He is the Hessian of the loss function, then by Liu et al. (2022) we also have that
BC
sup |Hc(u)) 1 By (FF yl-
wEB( It)
By Jacot et al. (2018), have that [F(ww)) y is also m-indepeudent with high probability for the given size of the net. Therefore, L can be said to be 3c-smooth within B(wo, R), where Bc is m-independent and thus R-independent. Hence, we can say that for every R > O, for some width which satisfies the given condition, the loss function is f-smooth (and by Theorem 4.1, p-PL") in B(w,, R) with high probability.
Thus far the argument above was parametric in R. But given that we satisfy all the conditions invoke Lemma 5.3 we can compute from it the minimum R value required such that the iterates of regularized gradient clipping never leave B(w,, R), ie
R= IV 2 ((w))
nnn
and conclude that d-Regullrizee--GClip converges to a zero-loss solution within B(wo, R) at a convergence rate of [(w.)= s [1 - -1=))'")
5.3 Proofs of the Lemmas
Proof. (of Lemma 5.1) We shall prove the theorem by induction and our hypothesis is that, up to step 4, w, e B(w,, R) for the given R, L(w:) [(wo)(1 au)² and thus up to t the algorithm explored region where the y-PL." condition holds. The base case is trivial, when t = 0 then wp e B(w,, R). Now we set out to prove that these contimue to hold at t too.
From the assumptions that, L is 3-smooth, we have
3 ((w.+)) - L(wx) - VL(w:) - wi) s 11 - -wzl??
(9)
As h(w;) min ( },, we have that M(me) B, hence we relax the above inequality get, -(wr1)--L(u VZ( (C(w))] (w -,;) s sh(w.) [ - u
Using the definition of the algorithm, that wi-1 w: = - VL(w)), we can rearrange the above to get
[(w..1) L(w:) S h(w:)
vL(w))1
(10)
2
13

---

Published in Transactions on Machine Learning Research (06/2025)
Next, use the induction hypothesis for the p-PL" condition at the current iterate, [V[(w;)]2 /
pl(ww).
to get
((w;-1) [(w,) h(ws)
h(w)) ((ure).
[VL(w;))] S 2
2
And the above can be rearranged to
1 ((w++1 & (1 - h M(n)))))((n.)
(11)
Note that for the convergence rate to hold, h(w;) must be bounded such that Vt, (1 2 h(w))p) e (0, 1)
and this follows from the bounds on a, b. We then unroll the recursion to get
[[ h(w,))
((wr+1) s L(wo)
i=0
2
(12)
1
$ L((w)
(41)
2
where the last inequality comes from 0 a h(w,). Therefore, assuming that the convergence rale holds till time & implies that it also holds till ++1.
Next we embark on proving that Wi-1 e B(w,, R). From the algorithm's update equation, the triangle inequality, and recalling that h(w))= s b, we get
| wol Σ [((w:)) v[(w))) s 2 [v[(w;)
(13)
1=0
We can rearrange the 3-smoothness inequality from equation (9) and apply Cauchy-Schwand get
B 0s 2 W W
w.,l? 2 [V [| We+1
will + -(w:)) - L(w++1)
(14)
We can relax the above inequality by dropping the [(w++1) term and treating the above as a quadratic in [W++1 wel, to then conclude that the inequality only holds the discriminant is non-positive, [V[(w))] / (23L(w:). Substituting this inlo equation (13) we get
[w-1 woll s Σ /2.((w.).
(15)
1=0
Using the assumed convergence rate till the current iterale we gel
( i-0
1 2 h(w,))))11
1wi+1-uu s
(16)
Since a h(w,)) 1/2, we have, 0< 0- h(w:)(() < 1 aaac 1. Thus get
W wol s / ((wp)
E ap))*2)
(17)
Upper-bounding the above by the closed-form expression for the infinite geometric series, we get
|wa. - wwol < 1/2 ((wa) R.
(18)
ap
The last inequality follows by the definition of and hence we have proven that w E B(wo R), and hence up to time the algorithm is still exploring the region within which the μ-PL' condition holds.
Thus induction follows and we have that Vt, w. B(wo, R) and L(w.) L(wn)(1 ap))..
14

---

Published in Transactions on Machine Learning Research (06/2025)
Proof of B-ReeularizeedGGCipp Having a Bounded Step Size
Proof. (of Lemma 5.2) Utilising d-RegularizeddGGGlii definition for h, we get that if [V 2 7/6, then h(w;) = min(1, 17 n6}.. Otherwise, if || (w:))l < </6, then h(w,)= = min {n, m//V[(()))> The smallest possible h for the above would be [v)(w)) [ was as large as could be, which would result in h(w,) = min (n, n63. As << 1, we conclude 0 < n & h(m))) n.
Proof of d Clil Convergence on Smooth PL* Functions
Proof. (of Lemma 5.3) From Lemma 5.2, we know that d-Reeuularized-GCllppp satisfies the condition 0 < nj & hu h(w:)) s v7 Therefore, by setting of < min 3. } and 8 < 1, we can apply Lemma 5.1 and obtain the convergence rate
[(w,) s ((w)) 1
2 noō))"
(19)
nV 23 ((w))
as well as that the PI, condition must hold within a ball B(wo, R), where R
1 V m6p
6 Conclusion
In this work, we have presented a new adaptive gradient algorithm, d-Reeularized-GGin that provably trains deep neural networks (at any depth) with arbitrary data and while training on the squared loss. To the best of our knowledge, such guarantec does not exist for any previously known adaptive gradient method. Additionally, we have also given experimental evidence that our algorithm is competitive with the deep learning algorithms in current use, and sometimes outperforming them. Our proof critically hinges on the interplay between the modification we do to standard gradient clipping and the μ-PL* condition that has previously been shown to be true for squared losses on deep ncural nets of sufficient width.
Our work suggests an immediate direction of future research into establishing convergence guarantees for regularized gradient clipping on other standard losses in use, such as cross-entropy and for nets with ReLU activation. Further, the demonstrated success of 8-GClip on certain transformer architectures motivates study of the possible validity of the μ=PI," condition for these models too.
Lastly, we note that recently reported heuristics which are particularly good for LLM training, cf. Liu et al. (2024), can also be seen as modifications of the clipping algorithm. We envisage exciting lines of investigation that could open up in trying to explore the efficacy of these new developments crossed with the provably performant modification of gradient clipping that we have instantiated here.
References
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang, Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communécations Security. ACM, Oct 2016. URL https: org/
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. In Advances in Neural Informalion Processing Systems, pp. 6155 61556166, 2019a.
Zeyuan Allen-Zhn Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference Machine Learning, pp. 242-252 2019b.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Lı, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, volume 32, 2019.
Parnia Bahar, Tamer Alkhouli, Jan-Thorster Peter, Christopher Jan-Steffen Brix, and Hermann Ney. Em- pirical investigation of optimization algorithms in neural machine translation. The Prague Bulletin of Mathematical Linguistics, (08(1):13-22) 25, 2017.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar. signSGD: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp. 560 569. PMLR, 2018. URL https://proceediin mlr. .press//t00/berrsteeinaaaa htt.....nntteeeaaa.....l..... httpp//eciigssttt.e.eee//////snn
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-performance large-scale image recog- nition without normalization. In International Conference on Machine Learning, pp. 1059-1071 PMLR, 2021. URL
preess/139/brock/22 html.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Iya Sutskever, and Dario Amodci. Language models are few-shot learners. In Advances in Nearal Information Processing Systems, 2020.
Jinghui Chen, Dongruo Zhou, Yiqi Tang, Ziyan Yang, Yuan Cao, and Quanquan Gu. Closing the general- ization gap adaptive gradient methods training deep neural networks. In 29th International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3267-3275, 2020a.
Minshuo Chen, Yu Bai, Jason D Lee, Tuo Zhao, Huan Wang, Caiming Xiong, and Richard Socher. Towards understanding hierarchical learning: Benefits of neural representations. In Advances in Neural Information Processing Systems, volume 33, pp. 22134 22145, 2020b.
Xiangyi Chen, Sijia Liu, Ruoyu Sun, and Mingyi Hong. On the convergence of class of Adam-type algorithms for non-convex optimization. In International Conference on Learning Representations, 2019. net/forummideeeeeeeeeee x309tm. arXiv preprint arXiv:1808.02941.
URL
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, volume 32, 2019. URL 1812.07956.
Soham De, Anirbit Mukherjee, and Enayat Ullah. Convergence guarantees for RMSProp and ADAM in non-convex optimization and empirical comparison to Nesterov acceleration. arXiv preprint arXiv:1807.06766, 2018. URL https://arxis: /tt////xiiv..r/////////////// .07666.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina N. Toutanova. BERT: Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 2019, 2019. URL
Pre-training of Deep
048000000000
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, G Heigold, Gelly, al. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. In International Conference on Learning Represen- tations, 2020. URL https:/ /tt///axvvv../ wwtprvv..rr//bb//1/000 11929.
Zehao Dou and Yuanzhi Li. On the One-sided Convergence of Adam-type Algorithms in Non-convex Non- concave Min-max Optimization. arXiv preprint arXiv:2109.14213, 2021. URL https///ppnrrriew.nett forum?id- Herumiid-NKShhymmegz..
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over- parameterized neural networks. In International Conference on Learning Representations, 2018. URL
https://openreviiee
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 222111222, 2011.
Haitao Fang, Guanglu Gong, and Minping Qian. Anncaling of iterative stochastic schemes. SIAM journal on control and optimization, 35(6):1886-1100 1997.
16

---

Published in Transactions on Machine Learning Research (06/2025)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pallern Recognilion, pp- 770-778, 2016. URL https://araiv.orggaa///55..// 03385.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and Generaliza- tion in Neural Networks. In Advances in Newral Information Processing Systems, volume 31, 2018. URL
http:// fttp:///axivv htt///aaivv....aaba//// 2og//bb/188060077.22
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint 1rXiv01412.6980, 2014. URL attps://anin wwtp:/rs//aaa.......... 6980..
Alex Krizhevsky. Learning multiple layers of features from tiny images. Masters thesis, University of Toronto, Toronto, Canada, 2009.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-paramelerized non- linear systems and neural networks. Applied and Computational Harmonic Analysis, 19:85-116, 2022.
Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International Conference on Learning Repre- sentations, 2024. URL https: //pp://///e/eeeeee/eeem wttp:arwwwieeeeeeeeteeeeeee hett/foruumi.eexxxiieeBB....
Chunmei Ma, Xiangshan Kong, and Baogui Huang. Image Classification Based Layered Gradient Clipping Under Differential Privacy. JEEE Access, 11:20150-20155 2023. doi: 10.1109/ACCCI 2023.3249577
John L Maryak and Daniel C Chin. Global random optimization by simultaneous perturbation stochastic approximation. In Proceedings of the 2001 American Control Conference. (Cat. No. 01CH37148), volume 2, pp. 756-762. IEFF, 2001. doi: 10.1109) 201.2000..44806.
Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. Tn International Conference on Learning Representations, 2018. URL https://arxiv http://axxvvv.ro//aaaa argg/bbs/700/ 11007. 05589.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and Optimizing LSTM Language Models. In International Conference on Leurning Representations, 2018. URL http://arxiv.. owwp:xiwwi.rrr.eooo /bs// 02182.
Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. Understanding the exploding gradient problem. arXiv preprint 1rX6v.1211.5063, 2012. URL http:://axiis http:///axxivoorggabbbe htt///bb///1111.31
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International Conference on Machine Learning, pp. html.
PMLR, 2013. URL
https::/prooeeedngg: www. wtt.:peeee////seeeaeeeeea
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of Adam and beyond. In International Conference on Learning Represenlations, 2018. URL https://openreeee wtt.:ento//ooommdddddm/////
Ohad Shamir and Tong Zhang. Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes. In Proceedings of the 30th International Conference on Machine Learning, pp. 71-79. PMLR, 2013. URL https://proceeeding: mlr, wttp:iiinn.iirr..es/eeeeeea/ matsss//28/shamr11/ shamir 1hmll..........
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In Proceedings of the 36th International Conference Machine Learning. PMLR, 2019. URL https://proceedding mml. htwpeedddwg..........eee////eeas wtwppess//77//iimeee111110/ html.
Matthew Staib, Sashank J Reddi, Satyen Kale, Sanjiv Kumar, and Suvrit Sra. Escaping saddle points with adaptive gradient methods. In International Conference Machine Learning, pp. 5956-5965. PMLR, 2019. URL
mlr. pttpss//9/sttailll0... html.
Shiqing Sun and James C Spall. SPSA Method Using Diagonalized Hessian Estimate, Proceedings of the IEFE Conference Decision and Control, pp. 4922-4922 2019. doi: 10.100//DDC4002420000000022
17

---

Published in Transactions on Machine Learning Research (06/2025)
Rachel Ward, Xiaoxia Wu, and Leon Bottou. AdaGrad Stepsizes: Sharp Convergence Over Nonconvex
Landscapes. In Proceedings of the 36th International Conference on Machine Learning, pp. 6677-6686. PMLR, 2019. URL https://proceedingg peweoeeedinggmmr...... mtt.pprees///97/rrddd... html.
Xiaoxia. Wu, Simon S. Du, and Rachel Ward. Global convergence of adaptive gradient methods for parameterized ncural network. arXiv preprint arXiv:1902.07111, 2019. URL http://arxiv.. org/ abs/ 1902.07111.
Jingzhao Zhang, Tianxing Hc, Suvrit Sra, and Ali Jadbabaic. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference Learning Represenlations, 2020a. URL https: httt::::///pperevieeee .wwp://orrunndddd.ggnppppppp///p
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems, volune 33, 2020b. URL https: //aaxiv....ie/ orgooo////a htt///baa1111/// 03194.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over-parameterized deep ReLU networks. Machine Learning, (09:467-492) 2020. URL
s10994-019-05839-6
Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the generalization of adam in learning neural networks with proper regularization. In International Conference on Learning Representations, 2023. URL
net (tte//oorrmmmdd-1111111a....
Fangyu Zou, Li Shen, Zequn Jie, Weizhong Zhang, and Wei Liu. A Sufficient Condition for Convergences of Adam and RMSProp. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), 2019. URL https://arxis: wwwp:ii.orrg/bb//111100038
18

---

Published in Transactions on Machine Learning Research (06/2025)
A A Proof of Convergence for Stochastic d-Regularized-GGCig
We start by proving a more general result as follows,
Theorem A.1. Given Assumptions 1, and 3, and for an arbitrary choice of > 0, consider
1>d> (1+ (a))
(1+3())
and
8 [1+3 (1)) (1+
0 < 17
23(1 + ($)3
Then, stochastic d-Regularizedd-GClip satisfies the following inequality over any T> 0 iterations:
min E [[vv[(u:)))
s T (36 B11²)
[(wi)
(20)
It's clear from above that can choose any >> 0 howsoever small and T> 0 howsoever large and have the minimum value over iterates of the expected gradient norm be similarly small. To prove Theorem A.1 we need the following two lemmas.
Lemma A.2.
E (.:) (.W,), VL((m:)) [m.] s s v[
(21)
Proof. We begin by employing Canchy-Schwarz and Assumption 1 to get {(h(g:)²
vL(w))ll wi] VV
v[(w:)]
s <k(h(g:)² w. VC(w:)11 <n"8|[[((w))]
(22)
where in the last inequality invoked the fact that h(g)) 5n
Lemma A.3.
E E[(-h(g))) VL(w,) w.] (7) 18)-66 8- [-|]v[(w))]]
Proof. Because is an unbiased gradient estimate we have, B((-h(g:)) (g. vL(w)), VL(w:)) E[(n h(:))) (g. vL(w.), v((:))))) Noting that n ((g:) 176, we get
E[ (-h)):))) (g. VL(w,,, s (n - no) 0 ( [V[(w))]
which establishes the claimed inequality.
A.1 Proof of Theorem A.1
Proof. We parameterize the line from to wi+1 as k(t) tw, + (1 t) and applying the Taylor's expansion and then Cauchy-Schwarz formula for the loss evaluated at its end-point we get
E[C(w(1) w.] E [(w;) h(9:) vL(w:))
1 + 2
{ (1 w. (' - ds wi
[(w,) E[h (g., vL(w.)) w.]
E 1 - w, 12 w. [- [(C((ss))
+
19

---

Published in Transactions on Machine Learning Research (06/2025)
Invoking | wil = =(gg)l9e and [[v))(((s)) 3 we have
2z h(g))² 1 wi
B[L(w(+1) wi] & ((w;) - (((())))))) we]
2
Substituting V((w)) +g. - VVL(w:) for gr. the second and the third term above, we gel
[[L((w+1) | we] $ - VV(() [w.) [..]-B[[ae ((((g:)) un wx]]V[((nn)11
8
h(gi) ((V (f (V((()))))))V(((
vL(ww))) wi
Recalling that n7 s h(g))) 77 and given that de (0,1),
[[[((w++1) mn] s (( (w:)) ([h(g:) - V.(w), v vL(ww) (w:))) wt]
3n [VC(w,,)) 3n
BE (gg V[(w))) [w,].
Now we invoke Lemma A.3 on the second term above and Lemma A.2 on the last term of the RHS above and take total expectations to get,
3r7 E [[[[(())))
0n
(1(1-600- [[[((w)))]]] (1())++((1111001111111 8)0 + (1/20) B[[v[(w))]]] n7
Given a T summing the above over all t = 1,.... T and recalling that wi is an arbitrary non-random initialization, we have
3n
T Σ E
T ( (7(1 0)0 3n Σ (7(1
3r220 T.
no
[[]v[(w))]]+
t=1
2
Invoking the incquality, 0 [[V[(())) (6² IV + and that L 2 we get
(n8
3n² 2 1=1
Σ [[v [(ur)) {1((1 - 5) Br1²)
Σ
F |v[(w))
2
[=1
B17² + n(1 - + By72
8²T.
The above implies
3y²
1) ((1-5)
8) (8721
Σ E [ sk(w))) + (28r²² 17(1 5)
n1
02T.
2
2
2
(1+())²)
Invoking the assumption that 5>
and n 28(1+())7)
38-1 23
we get
(1+3())²)
min E
[V[(w))]
+ e [[V(((())
L(wi)
2312² + (n(1
03
T. (3 Bmp)
2. (36 5 1) :3722)
20

---

Published in Transactions on Machine Learning Research (06/2025)
Now for arbitrary e> O, solve the inequality
1 - (1 2 (c) {=)
- 2,317 (8)
and thus we get,
8 31+3 (0) -11- (9)*
7 E 0,
2 + (4)
Note that the above upper bound on is the range of chosen in the statement. Thus we have established the claimed theorem statement.
A.2 Proof of Theorem 2.2
Proof. With the given choices of n, and B as in the theorem statement, we have
16(1+ (()+33)
34 1 Bn)
e3e + + +4)
17
" 51e4
11+
O
16
Substituting the above into the guarantee Theorem A.1, meaning the RHS of (20), along with using T 1/ we get the result claimed.
21


Reviews:
[{"review_id": "Nip586uUiN", "reviewer": "Reviewer_ftSa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has several strengths:\n\n- It is well written and it presents clearly its contributions.\n- It provides an optimization result in the case of a non-trivial adaptive algorithm. The authors also consider the stochastic case. The proofs appear to be correct.\n- It contains an extensive experimental study with neural networks applied in several applications.\n\nHowever, I identified two weakness:\n\n- Admittedly, the technical contribution of the paper is limited; the main tool of the paper comes from the PL* condition for wide networks.\n- The paper claims competitiveness of the proposed algorithm with SGD and Adam, yet all the results are with respect to one random seed. I would recommend being more cautious with some claims in the paper regarding the experiments, as limited conclusions can be drawn from single-seed experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies convergence of a gradient-clipped version of SGD in wide neural networks for the squared loss. The proof of the main result is based on the PL* condition, which has been proven itself by Liu et al recently, utilising the NTK of the model. Furthermore, experiments are presented that complement the theoretical analysis and demonstrate the applicability of the algorithm for training neural networks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the paper is interesting and ready for publication at its current form. However, the authors could consider improving the papers along the following axes:\n\n- I believe the statement of Theorem 2.1 and the subsequent discussion could be improved. In particular, the last two sentences of the Theorem's statement appear to be misphrased. Also, the authors could perhaps discuss the smoothness assumptions for F(w).\n- Similarly, the smoothness of L is assumed for Theorem 2.2, yet the authors do not explicitly comment on it.\n- On the bottom of pg. 4, there is a typo: \"deep-learnning\"\n- In the experiments section: In Figure 2, the clipped algorithms display a non-monotonic behaviour in the loss, before epoch 150 when an additional learning rate reduction is applied. Can you comment on it? Do you have any intuition on why it happens?\n- More random seeds for the experimental study.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0PHOyrwJkS", "reviewer": "Reviewer_rGm3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is clearly written and easy to follow.\n2. Analyzing the convergence behavior is a significant area of interest in optimization, particularly in the context of scalable deep neural networks. \n3. The authors offer a rigorous theoretical analysis of the convergence properties of the proposed delta-gradient clipping algorithm, providing a solid foundation for understanding its behavior.\n\n**Weakness**\n\n1. In my view, the introduction does not well articulate the motivation and significance of this work. For example, it remains unclear why the authors emphasize the convergence analysis of Adam optimization, given that the paper does not provide further theoretical contributions on this topic. Additionally, the problem that the proposed delta-gradient clipping method aims to address should be explicitly stated. In other words, what motivates the authors to propose this method? I could understand what the authors would like to contribute, but the writing here is not quite clear.\n\n    And by the way, several papers have focused on analyzing the convergence of Adam under certain assumptions. The authors should properly acknowledge and discuss these existing results rather than claiming, \"However, to the best of our knowledge, there has not been so far a theoretical guarantee for any adaptive gradient algorithm to converge to the global minima of deep neural network loss surfaces.\" This statement requires revision to accurately reflect the literature.\n\n    I recommend that the authors revise the introduction to more clearly articulate the motivation and objectives of the paper, as well as the significance of the proposed approach. At present, it lacks clarity regarding the core contributions and the specific problems the authors aim to solve.\n\n2. The core of the proposed method, i.e. delta-gradient clipping, is to add a lower bound during the gradient clipping process, as compared to conventional approaches. In my view, incorporating such a lower bound does not appear to offer substantial practical benefits for training. Furthermore, the convergence analysis largely follows the framework established in the paper \"Loss landscapes and optimization in over-parameterized non-linear systems and neural networks\", and thus the results presented are largely anticipated.\n\n3. The presentation of empirical results should be especially improved. At present, the figures are difficult to interpret, and the visual clarity is insufficient. At a minimum, a table regarding the comparisons between the proposed method and the baselines should be provided. Also, the experiments should involve several different architectures on Cifar-{10, 100}, and ImageNet.\n\n    Moreover, an ablation study w.r.t the extra hyper-parameters delta should be provided. Currently, delta is set to a very small value, and it remains unclear whether the gradient norm actually reaches this lower bound during training. The authors should provide evidence or analysis to clarify whether the proposed lower bound is actively engaged and contributes to training dynamics.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors strive to introduce delta-gradient clipping, a regularized gradient clipping method that guarantees convergence to global minima in deep neural networks with squared loss, given sufficient layer width. Compared to conventional approaches, the core of the proposed method is to add a lower bound during the gradient clipping process. Some empirical results show delta-gradient clipping competes with current methods, including transformer architectures.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weakness", "broader_impact_concerns": "I have not found any discussions about the limitations and potential negative societal impact. But in my opinion, this may not be a problem, since the work only focuses on the optimization in deep learning. Still, it is highly encouraged to add corresponding discussions.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vDRgl8JUF4", "reviewer": "Reviewer_xetK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper offers a clean, unified view of gradient clipping approaches through the lens of regularized optimization, which is both theoretically grounded and intuitive.\n- The method is simple and incurs little overhead, which makes it likely to be adopted in practice.\n- Strong experimental evidence shows that  $\\delta$-GClip improves convergence speed and robustness across a range of settings: ViTs, diffusion models, and policy gradients in RL.\n\n## Weaknesses\n- The paper could better contextualize its theory in relation to other robust optimization or clipping strategies, including, for example, connections to trust region methods, proximal point methods, or more modern noise-aware optimizers could further clarify where $\\delta$-GClip fits in the broader optimization landscape.\n- While the method is said to be robust, the paper could more clearly communicate how sensitive $\\delta$-GClip is to the choice of the regularization coefficient.\n- In the reinforcement learning experiments, the comparison set is relatively narrow. Many RL algorithms already include sophisticated stabilization mechanisms (e.g., the clip function in PPO, trust regions in TRPO, or gradient penalty methods). Without direct comparisons to these or an integration of $\\delta$-GClip into such methods, it's unclear how much of the improvement is due to $\\delta$-GClip versus general tuning or architectural choices.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces $\\delta$-GClip, an algorithm for stabilizing training by combining gradient clipping with a regularization term. The key idea is to allow gradients to be clipped more smoothly, rather than abruptly, by framing clipping as a regularized optimization problem. The authors present a unified perspective on common clipping methods (like norm-based and adaptive clipping), show theoretical stability properties of $\\delta$-GClip, and demonstrate improved performance on several tasks including large-scale vision models and reinforcement learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More systematic analysis or guidance on how to set the regularization parameter across tasks will strengthen the case for the proposed method as a practical, low-effort improvement.\n\nIt would also be helpful to add more comparisons to recent clipping or optimization strategies in large-scale training and reinforcement learning to reinforce the empirical claims.\n\nOptional: \n- More intuitive explanations or diagrams to clarify the optimization interpretation of $\\delta$-GClip, making it more accessible.\n- Show whether $\\delta$-GClip also affects generalization at late training stages or primarily helps early convergence.\n- Include more ablation studies to separate the effects of $\\delta$-GClip from other training details (e.g., learning rates or model sizes).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)