How iteration composition influences convergence and stability in deep learning
Authors: Benoit Dherin
Benny Avelin
Anders Karlsson
Hanna Mazzawi
Javier Gonzalvo
Michael Munn
Venue: Accepted by TMLR ()

Abstract: Despite exceptional achievements, training neural networks remains computationally expen- sive and is often plagued by instabilities that can degrade convergence. While learning rate schedules can help mitigate these issues, finding optimal schedules is time-consuming and resource-intensive. This work explores theoretical issues concerning training stability in the constant-learning-rate (i.e., without schedule) and small-batch-size regime. Surprisingly, we show that the composition order of gradient updates affects stability and convergence in gradient-based optimizers. We illustrate this new line of thinking using backward-SGD, which produces parameter iterates at each step by reverting the usual forward composition order of batch gradients. Our theoretical analysis shows that in contractive regions (e.g., around minima) backward-SGD converges to a point while the standard forward-SGD generally only converges to a distribution. This leads to improved stability and convergence which we demonstrate experimentally. While full backward-SGD is computationally intensive in practice, it highlights that the extra freedom of modifying the usual iteration composition by reusing creatively previous batches at each optimization step may have important beneficial effects in improving training. To our knowledge, this represents a new and unexplored avenue in deep learning optimization.

Published in Transactions on Machine Learning Research (12/2025)
How iteration composition influences convergence and sta- bility in deep learning
Benoit Dherin'
dherin eeoogle@googe.coom
Google Research
Benny Avelin" Department of Mathematics, Uppsala University
bermy.avekin&@mathh
Anders Karlsson"
anders. wwr.soooonssnngnn.....
Deportment of Mathematics, University of Geneva and Uppsala University
Hanna Mazzawi
mazzoui@gooolee.cm
Google Research
Javier Gonzalvo Google Research
Michael Munn Google Research
munn@goooge.coo com
Reviewed on OpenReview: https: //tp://ppppe openreview.
net/foruse
G2CBM2Yo3a
Abstract
Despite exceptional achievements, training neural networks remains computationally expen- sive and is often plagued by instabilities that can degrade convergence. While learning rate schedules can help mitigate these issucs, finding optimal schedules is time-consuming and resource-intensive. This work explores theoretical issues concerning training stability in the constant-learning-rate (i.e., without schedule) and small-batch-size regime. Surprisingly, we show that the composition order of gradient updates affects stability and convergence in gradient- based optimizers. We illustrate this new line thinking using backward-SGD, which produces parameter iterates at each step by reverting the usual forward composition order of batch gradients. Our thcoretical analysis shows that contractive regions (c.g., around minima) backward-SGD converges to point while the standard forward-SGD generally only converges to distribution. This leads to improved stability and convergence which we demonstrate experimentally. While full backward-SGD is computationally intensive in practice, it highlights that the extra freedom of modifying the usual iteration composition by reusing creatively previous batches at each optimization step may have important beneficial effects in improving training. Our experiments provide proof concept supporting this phenomenon. To our knowledge, this represents new and unexplored avenue in deep learning optinization.
1 Introduction
In recent years, neural networks have achieved remarkable success across diverse domains from text generation Gemini (0024 2023): Brown et al. (2020) and image creation Ramesh et al. (2022 2021); Saharia et. al. (2022) to applications protein folding Jumpper al. (2021a b) and ma-
terial discovery Merchant al. (2033) However, their training remains challenging and compu-
"These authors contributed equally this work

---

Published in Transactions on Machine Learning Research (12/2025)
tationally expensive. One of the reasons for this is due to training instabilities which often oc- cur Morchdi et al. (2023); Li et al. 2019); Cohen et al. (2021); Chen et al. (2018) and which produces hard to interpret loss curves, wasted computation time, and potentially failed experiments. One way to view this challenge is as a trade-off between stability and performance: hyperparameter settings that often yicld better test performance, such as higher learning rates and smaller batch sizes, tend to exacerbate these instabilities.
As a basic example, consider the training batch size. For many common optimizers, smaller batch sizes often lead to improved test performance. In fact, recent rescarch has shown that small batches induce form of implicit regularization Novack et al. (2023); (mmith et al. (2021) |s Dherin et al. (2022;; |: Keskar et al. (017); Ali ct al. (020) which benefits generalization. On the other hand, the greater variability of small batches tends to exacerbate oscillations of the training loss, prolonging time to convergence. In this work, we demonstrate that these instability and convergence issues associated with small batch sizes can be mitigated without loss of generalization power by reversing the composition order used to produce each iterate. The general line of thinking behind our approach consists of loveraging not only the current batch, but also the previous batches to produce the current. parameter iterate. The creative reuse of previous batches at a given step has been already explored in Choi et al. (2019) for instance in order to speed up training by composing previous batch gradient updates to the current iterate while waiting for the data pipeline to deliver new batch in case of I0 bound situations. In this work, we reverse the composition order of all the batches received so far from initialization and show theoretically and practically that the sequence of iterates produced this way enjoys better convergence and stability properties than the usual composition order of the batch gradient updates. We now describe below this reverse composition procedure in details.
Namely, standard training algorithms, such stochastic gradient descent (SGD), Adam, and other gradient- based optimizers, are iterative processes. At each step, these algorithms update the network parameters 0 using a randomly sampled data batch B;. This update can be formalized as a transformation, 0' = (?),,, where 0' represents the new parameter value. Because at each step i the batch B; is randomly sampled, the update operator T; can be formalized as a random operator. For instance, in the case of SGD, the random operator is defined by the update rule T0( = 0 - LV, (0), where LB, is the loss function evaluated on batch B, and h is the learning rate. The sequence of parameter updates generated by these iterations, starting from initial paramcter value 0, defines the standard learning trajectory, which we will refer to as the forward trajectory:
0o = 0, 01 = Ti))), = (TT,,
On - -TTT-T-- Ti)).
Notation: To conserve notation, we write TiT; to denote the mapping composition T TJ; ie., (0) (O).
Moderate or large learning rates and small batches, when used with the standard forward trajectory for training, tend to destabilize common gradient-basse basee optimizers, causing convergence and stability issues. Consequently, learming rate schedules have become essential as an attempt to encourage proper convergence of the loss during training. Our key contribution, which we support with both theoretical analysis and empirical results (sce Figure 2 and Appendix A), is in demonstrating that reversing the batch composition order to produce an iterate at a given step what we call the backward trajectory leads to siguificantly more stable convergence. Specifically, the backward trajectory consists of iterates generated by composing the training batches in reverse order; iee,
0p 0, 0, =T 0g = TTT2(0,,
- ..
T.
For the sake of clarity, let us work out the first two iterates both for forward and backward. For instance for SGD, if we receive first the batch B1 and then the batch Bg in sequence in the training loop, then at the first. step both the backward and forward iterates coincides with Ti))) initialized parameter value. However at the second step the forward iterate becomes
LVLB (0), where is the randomly
(0) (0) - LT
0 - AVLB, (0) AV (0 LVLB (0).
2

---

Published in Transactions on Machine Learning Research (12/2025)
Forward iterations
Backward iterations
previously consumed
consumees
Bi
B
B1
B
B,
B1
B--1
B.
B+41
B1
B
B1
B141
B
BL-1
B,
B141
8+2
Be+2
B1
B:-1
Bg
Br+1
Heta
Figure 1: Naive implementation the backward dynamics: Forward iterations (left) and backward iterations (right). The training steps are represented by Pac-men consuming batches. Forward iterations maintain a training state and consume a new batch each step, while backward iterations restart the training and consume all the batches received far in reverse order.
On the other hand, the second step of the backward iterate is
(T) (0) hv hVLB, (T2(0)) 0 - h AV WVL,, (VI,,
A naive (and computationally intensive) implementation of the backward optimization depicted Figure where all the batches received so far each step are re-processed from scratch in the reverse order in which they were received from the initialization point.
Main contributions: The main contributions of this paper are show theoretically (Theorems 2.2 and (.6) and experimentally (Figure 2 and Appendix (A] that backward optimization has better convergence and stability properties than the standard forward optimization. As already known (see related work in Section 1.1 forward trajectories do not generally converge toward points but rather toward probability distribution of the iterates (in the fixed learning-rate regime).
We argue that the advantage of backward trajectories comes from their convergence toward actual points (see Theorem 2.2 sampled from the forward distribution (see Theorem 2.6). We prove this using a generalization of the Banach fixed point theorem when the random maps become contractions. Note that backward optimization theoretically be applied to any gradient-basod optimizer (not only to SGD) to improve stability and convergence, since our theoretical statements hold at the level of the Tí's, maller what their actual forms are. To our knowledge, these concepts are new in deep learning optimization. The main goal of this paper is to expose how this phenomenon manifests in deep learning. We defer engineering applications leveraging this phenomenon (like efficient implementations the backward SGD) to future work, while outlining a few potential directions the paper's 's conclusion.
Overall, this work should be understood as a proof of concept showing that reversing the iteration composition order can significantly improve stability and convergence in stochastic optimization.
1.1 Related work
Convergence of SGD: A number of works prove under different assumptions that the (forward) iterates
of SGD with constant learning rate do not converge toward points but rather toward a stationary probability
distribution: Merad & Gaiffas (2023) and Dieuleveut et al. (2020) show this for the strongly convex case; Shi
rokoff & Zaleski (2024) treats the non-convex case with separable loss; Babichev & Bach (0188 focus
3

---

Published in Transactions on Machine Learning Research (12/2025)
losses coming from exponential family models; Cheng al. (2020) quantify the rate of convergence of SGD to its stationary distribution in non-convex optimization problems; see also Dieuleveut & Bach (2016) and Meyn & Tweedie (193)). In Huang et al. (2017), the authors leverage the fact that SGD with fixed large learning rate oscillates between different solutions in order to create a cheap average of models by saving the explored parameters along the way. For convergence to a particular solution, the forward order for SGD needs an extra decaying learning rate schedule as shown in Robbins Monro (1951) or Mertikopoulos al. (2020).
Contractions in deep lcarning: From our point of view, one important feature that leads to convergence for the backward trajectory is the contraction property of the random operators. This notion (which we believe is under-exploited in deep learning) has surfaced in different contexts in deep learning: see Bergomi et al. 2019), Qian & Wegman (2019), and Avelin & Karlsson (2022).
In many Markov Chain Monte Carlo (MCMC) algorithms the goal is to sample from a distribution p. The idea is to construct a Markov chain with stationary distribution μ and then run the chain for a long time to get samples from μ. The Propp-Wilson algorithm uses a form of backward iterations to accelerate the convergence toward samples from the distribution (see Propp & Wilson ). More generally, the idea of backward dynamics is hidden in many constructions in Markov chain theory when the Markov chain is given by itcrations of random operators as illustrated in Diaconis & Freedman (1999). In particular, they prove a general result concerning the convergence in dist distribution of these types of iterated Markov chains using the backward dynamics.
Markov chains, iterated functions, and MCMC:
Stability in deep learning: Already instability issues appear in the full-batch regime, and number of theoretical works have studied it under the heading edge of stabilily Cheng el al. (2020); ; Wu et al. [2024 Cai
ct al. (2024). Other works have also studied stability in the large batch regime after a batch size saturation takes place using the implicit conditioning ratio Lee et al.] (2022) Agarwala & Pennington (2024). In this work we focus on the stochastic or small batch setting. Our findings do not really matter for the full-batch setting since backward and forward iterates then coincide. In the context of physics-informed neural networks, it has been observed that the gradient field is stiff, producing instabilities in learning trajectories [Wang] et al. 2020). To remedy this Li et al. (2023) propose a backward Euler scheme to stabilize training in this context. However the backward Euler method is implicit Runge- using iteration backward.
Runge-Kutta Kutta method, which is different from
Sample order: Curriculum learning Soviany el al. (2022) leverages the impact of sample order for generalization Mange (2019) by organizing training examples in a meaningful sequence, typically starting with simpler examples and gradually introducing harder oncs, thereby optimizing the learning process and improving model performance. The backward optimization can be viewed as an automated form of curriculum, mitigating the forgetting of previous examples as new examples are added; a phenomenon which also related catastrophic forgetting and the stability gap in continual learning (see Lange al. (2023) for instance).
2
A backward contraction principle
The contraction mapping principle, also called the Banach fixed point theorem, is cornerstone result. in mathematics and science, in particular for finding solutions of equations (Newton's method) or of differential equations (Picard's method of successive approximation). It is also behind Google's PageRank algorithm Page
et al. 19988 It concerns the existence of fixed point of maps which are uniform contractions of a complete metric space In detail, let (12 d) be a complete metric space and T Ω + Ω be a continuous map. Tf there exists < < such that
(T(O)),T(())) S k- d(0,,02),
Recall that complete metric space (9, d) is set Ω equipped with distance metric d(01, 02) for which all Cauchy sequences (Le., sequences 6. such that d(On, Bm.) + as n, m + 0) converge to points in the space; typically Rd with the Euclidean distance a complete metric space.
4

---

Published in Transactions on Machine Learning Research (12/2025)
for all 01,82 12, then T is called uniform contraction. The fixed point theorem now states that T has fixed point, that is, 0o E Q such that ((0)) 0o- This point can be found by iterating the map:
T 0g
for any 0 E S2 as + oc. For example, in PageRank Page et al. 1998), one iterates the PageRank matrix from 50 to 100 times and the result is a very good approximation of the PageRank vector dp. Another application the standard proof of convergence for (full-batch) gradient descent around a minimum, which can be interpreted as an application of the Banach fixed point theorem detailed the next example:
Example 2.1. (Full-batch gradient descent convergence.) In this case the operator is T(0) 0 - -V for loss function L. The idea of the proof is to choose the learning rate h small enough so that T becomes a contraction. More precisely, around a minimum l" the operator T can be approximated using a first-order expansion of the gradient around the minimum as T(Ø) = 0" + (I - hH)(0 - 6*), where H V(L(0"). Now we have that
IT((1)) - T(22) S || hH 1op111 -2ll.
(1)
where ||T - hHF lp the operator norm of hH, that is the operator maximum eigenvalue: max; |1 h (here the A cigenvalues of H). Now it is casy to verify that - hH olop 1 if and only if the learning rale is strictly smaller than 2/A 2/Amax where Amax is the largest eigenvalue of H. Convergence for that setting follows from the Banach fixed point theorem.
The generalization of this convergence argument to Stochastic Gradient Descent (SGD) is actually problematic. The main reason is that now do not have a single opcrator but sequence of them T2, ..... cach computing the loss gradient turns out that usual forward iterations will not converge to a point in general as illustrated in Example 2.4 Theorem 2.2. however, establishes the convergence of the backward iterations.
different batch of data. Even if each of the operators is contraction it
Theorem 2.2. (Backward contraction mappings principle) Let T, be sequence of continuous self-maps of complete metric space. Assume T; 's are uniform contractions, with cerlain ks < 1 in common, and for some 0 there is a constont D such that,
((O)).(0)) < D for all
(2)
Then for any 0 e Ω the backward iterates
0, T
converge to a point 0* as n - oo. 00. depending on 8 such that
Moreover, the converyence rate is exponential: i.e, there a constant C'
d(0', -.. -(0,,T133---T.0000 TT(()) C
Proof. Condition 22 expresses that the distance d(0, (0)) is uniformly bounded by constant D for all T/s for point A. Let us show that if this happens for single point 0 this happens for all points, provided we change the constant D. To see this, take another point 0. We will compute another constant Ó such that d(0 (0) < D. Namely, using the triangle inequality and the fact that T. are uniform contractions, we obtain that
(0))
D (1+)))
Now, we want to prove that 0, is Cauchy sequence, i.e, that d(On, Om.) tends to zero for m as -> Since Q is assumed to be complete, this will mean that the backward iterates O,, converge toward point 0*. The idea is to bound the quantity
A d(On. 0m)
d(Ti TT ..
5

---

Published in Transactions on Machine Learning Research (12/2025)
Because of the backward order (note: the forward order would not allow that), apply the contraction property n times (since m > n), yielding:
A km d(0, Tat1- (0))
Now a simple application of the triangle inequality produces
A I k" [(0,, (m+1(()))))
+d(Tn+110), In+1In+2(00)
+ d(Int1 10), Tnt1 (mmt))))m(()))) (m)()
At this point, we can use the condition in ( for the first term (0) < and in conjunction with the contraction property for the subsequent terms in the sum, yielding:
A S kn D + D k + ... + D p>
1 nn=
< D km
1 1-k
D k"
1-k
where we used the sum of geometric scries. Now, this yields that A 0 as n + oo, meaning that the sequence On is a Cauchy sequence and thus converges to certain point 0* since Qt complete. Then, taking the limit we obtain
D. = k.
d(On,, )) 1- k
with C= D I-E: which shows the exponential convergence rate.
Remark 2.3. The contractivity assumption for the operators Ti's is in general too restrictive for deep-learning settings. However, the proof of Theorem 2.2 still works in more generalized setting that encompasses the non-convexity of deep-learning loss-landscapes. The price to pay though is the introduction of the more abstract notion pseudo-metrics for which the argument in the proof Theorem 2.2 still works verbatim.
We give the details of this more complicated approach Appendix G
As for the condition in (2), let verify that it is satisfied for SGD in the case of an overparametrized model like a neural network. Namely, in this setting, we generally consider losses of the form L(0) = 3/2 Σ L,)0,, where L is the loss computed on batch B, with (0) being non-negative. Because the over poor-parameterization, there generally exists points 0" where the full loss vanishes. This in turns implies that Le(0") 0 for all i. For SGD, the operators are (0) = 0 (0). Now it is easy to see that (2) is satisfied: ((0,T,((0)) | - - (V))( = 0, since the global zero 0 is critical point for all the L,'s.
Now here is a major point: The contraction mapping principle fails with the forward order. As the next example demonstrates, it is not enough for the operator Ti's to be uniform contractions for the forward sequence of iterates to converge, while this is always true for the backward sequence because of Theorem 2.2
Example 2.4. (Forward iterations counter-example) Here is an extreme example illustrating the convergence failure for the forward sequence of iterates (even when the maps are uniform contractions). while the backward sequence converges to a single point under the conditions of Theorem 2.2 Consider the two constant maps S()) æo, and (0) yo, which are contractions with k = 0. Assume that 20 + 3o, now also assume that each T; is either equal to S or U with equal probability to be selected. Then, independently of 0, the forward iterates
T-Tn-1--.. ... (1))
will jump between dn and yo. according to whether T, is S or U at that particular n. Thus, no convergence to a single point is possible, although the sequence converges to a probability uniformly distributed in the two outcomes, since for cach forward itcrate either outcome has probability 1/2. On the contrary, the backward trajectory
TiT2 1.2...T,ll (.)

---

Published in Transactions on Machine Learning Research (12/2025)
will always converge to single point determined by the first element in the sequence: cither to T; = or to yo if 7 = U.
2.1 A fundamental example of a contraction
We now describe how the SGD updates are uniform contractions when the batch losses are strongly convex near & minimum. First, consider strongly convex smooth function L Rd + R. Recall that is strongly convez if there is constant > 0 such that for all 01, 02 E R
(V)(0)) (L((2), 02) 2 110 8211
(3)
where (a, b) = a²'b is the inner product between two vectors. Furthermore, we assume that there is a constant M m such that
-
- M 10: - -211
(4)
This is sometimes denoted as S1.1 m, (Rd), see Nesterov (013 The following lemma (whose proof is in Appendix F) shows that for strongly convex functions we can always find a learning rate small enough so that the corresponding gradient descent update is a contraction.
Lemma 2.5. Let L(0) satisfy and and define the map T(0) LVL Then
[T((1) - T(2))l S V1 2hm + 12 110 - 021l
In particular, for small enough h E (0. 1) (depending on m and M) the map T is a uniform contraction.
Now if we suppose that batch losses L,'s are strongly convex (possibly near global minima), then Lemma tells us that the SGD maps T:(0) 0 - (O) are uniform contractions. Thus, we can apply Theorem and see that the sequence backward iterates, 0₂ = TTT. ... .T,(), converges to a point.
2.5
2.2
2.2 Relation between forward, backward, and generalization
In this section address the question of the generalization power of backward iterates, which we argue is the same that of the forward iterates. The relationship with generalization is implicit through Theorem 2.6 below which states that backward convergence points are distributed according to the forward stationary distribution. follows that the best test-performance forward-sohutios can be reached by the backward trajectories with the same probability. This good thing as shows that the backward trajectories are more stable but without tost-performance reduction on average, which is often not the case as for instance the increased stability obtained by increasing the learning rate or decreasing the batch-size (see Dherin et al. (2022)). In particular, the bias of backward SGD is statistically equivalent to that of standard SGD. Consequently, the convergence point of the backward iterates cannot outperform the best solutions attainable by the forward iterates.
The goal of this work aims to demonstrate that the composition order creates two different processes, which we could described intuitively as follows:
The forward iterate process converges toward a distribution, "forgetting" the initial batches and being most influenced by the recent ones. This makes this process robust to bad start but leaves it perpetually noisy.
The backward itcrate process converges toward a point, "remembering" the initial batches but "forgetting" the recent stochastic noise.
Theorem 2.6 formalizes this intuition and essentially generalizes the situation already seen in Example 2.4 an actual realization of forward trajectory converges to a distribution which uniformly distributed between the two outcomes. On the other hand, any realization of the backward trajectory always converges to a point sampled from this distribution. Let's formalize this for the general case.
7

---

Published in Transactions on Machine Learning Research (12/2025)
First of all, recall that a map T Ω + Z produces corresponding push-forward map T. P(2)) - P(Ω) at the level of probability measures P(Ω) on f. Recall that a probability measure P(Ω) associates to each subset A C St a mumber μaA) modelling the probability ( E A) of 0 being in the subset A if 0 is sampled from J.. Now, if ş1 is a probability mcasure on Ω then the push forward (A = Ju(T (A)) for A C Ω computes the probability p E A) of T(0) being in A 0 is sampled according to p..
Therefore, the forward sequence .T..... T:TT: 12 + Q induces a push-forward initial distribution Mo to the probability mcasure Hn, = (T,, T2 modeling the probability distribution of the nthh forward iterate 0, = T ... T2I (6o) when the initial point dg is randomly sampled from pe. Observe that the forward iterates 9, form sequence of random variables, which is Markov chain (since for the forward iterates we have that On = Tn (0)-1)). Many previous works (see related work in Section 1.1] have shown under different assumptions that this sequence of forward iterates do not converge to a point when using fixed learning rates, but rather their probability distributions In converge to a limiting probability distribution, called the stationary distribution of the Markov chain. In other words, starting at initial point On the distribution yo, = (Tn). ... (-,, of the forward iterate 0n converges to stationary dist por- Recall that dø, is the Dirac measure concentrated at 0g (ie., 6R, (A) is 1 if A contains O and 0 otherwise).
-.T.)) that maps an
On the contrary, we have seen in Theorem 2.2. the backward iterates converge to actual points in regions where the maps Ti's become contractions (which we will call contractive regions). The following theorem (whose proof in Appendix E shows the relation between the point-wist wise convergence of the backward iterates and the distributional convergence of the forward iterates:
Thcorem 2.6. Consider sequence {T,], i == 1,2,... of independent and identically distributed random operators. Suppose that for 0g 1 Rd the backward iterates converge to a random point (randomness is due to the sampling of the random operators T; 's):
T:T - Tw, - 0" as
n
+ 0o.
Then the probability distribution of the forward iterates from Og converge (in distribution) to a stationary probability measure ,ar Moreover, the random point H is distributed according the same forward iterate stationary distribution par
3 Two explicit examples stochastic gradient descent
We now present two examples that provide intuition on why backward SGD converges to point rather than to a distribution forward SGD does, when the learning rate is fixed. The reason in both examples is that the stochastic noise added to the parameter iterates at each step due to the randomization of the batch diminishes to zero in back ward SGD as the training progress even with fixed learning rales, while this stochastic noise stays unchanged at each step of forward SGD, requiring a learning rate decay for convergence.
3.1 Example: Quadratic loss
Consider the quadratic loss function L(0)= = 62/2 with 0 E R. Its gradient is A. We want to find the minimum of L (which of course is 0 at 0 = 0) by a stochastic gradient descent. To model the batch noise, we add an i.id random error e; to the gradient at step i, yielding the iteration procedure:
Ti(0) = 0 - h (VL(0) - -s) = (1 h)0 + ho:
where h is the learning rale. These maps are uniform contractions provided that E (0, 1) since
[T((i)) T.(02) = [(1 () 02) s h| 02)-
Now we would like to compare the behavior of the forward trajectory to that of the backward trajectory for the Tss. s. In order to do this we begin by calculating the forward iterates:
(0)
(1 - 1-))((1- h)0 he1) + hez
(1 --)30 hez (1 h)he1.
8

---

Published in Transactions on Machine Learning Research (12/2025)
One sees easily how this continues:
-. = (1- + hen +... h(1 - h
By switching indices, for the backward iterales we obtain:
T ... (0) = - 0 + hej + ... h(1 100-hh)- cn.
We see that the forward iterates receive at each step n an additional error hen which stays constant during the whole trajectory preventing coneerenn,,,,n,e,, while for backward iterates that same error h(1 h)" 1i)mlen en is decaying to zero the trajectory progresses.
3.2 Example: Learning dynamics close to minima
The previous example was a warm up for the following more realistic example. However, the key idea is identical. Consider loss function
L( Σ L,
where Lj is the loss computed on the random batch B, The SGD update is
Ti(0) = (0)
where h is again the learning rate. We now compare forward and backward SGD dynamics sufficiently close minimum 0 of the loss function so that we can approximate well the full loss gradient g() (l) using the loss Hessian H0) (g)(): i.e., first order Taylor's expansion of the gradient the minimum yields
g g(0 (00) - 0*) H(0)) 8*),
55)
since at a minimum g(0") = To simplify the notation, we will denote by (i) the gradient of the batch loss (0), which now a random vector (because the batch is random). We can model this stochastic gradient as follows: g((x) = g(x) E,, where we assume that the differences e = g(x) - gx(x) are i.dd random vectors with zero mcan. Using gradient approximation (5) we have that single gradicnt update is given in form very similar to that of the previous example:
Ti(() = 8 - (9(0)) - = + (1- 111--h1 hH) - 8*) + he,
where H. denotes the Hessian evaluated at the minimum Since
(T,( T1(l))l = || hH 02ll
and since || hH|| < 1 for h strictly smaller than 2/ 2/Amax (as in Example 2.1 Theorem 2.2 tells us that the backward trajectory converges. Now let us have a look at the form of the forward and backward iterates in this particular case to give us an intuition about why this is true. In a similar way than in the previous example, the th forward SGD iterate given by
T, (0) 0" (1 hO 0²) + he, + h(1 hHH))--1
+ --1-h
While the term (1- h0 8*) converges to zero (when h < 2/Amar), random vector he, is added to the iterate, making the whole sequence converge to a distribution rather than to point. On the other hand the backward iterate is obtained by reversing the indices:
that at each step fresh
T T... T.()) = 0* + - hHy - 0*) + hej + h(1 - hH)e2 ........4 1(1 - hH)"-1e en- Cn-
This expression converges as n + do since the terms are decreasing at an exponential rate as n grows. Thus the iterates converge to point now rather than a distribution. Looking the actual form of the backward iterates, is clear that they are distributed by the probability measure given by the forward iterates prescribed by Theorem 2.6
9

---

Published in Transactions on Machine Learning Research (12/2025)
4 Experiments
Note on plotting multiple seeds: We are interested in the variability per rcalization of the backward and forward trajectories. The backward trajectories are more stable individually along their own path, but these paths can be very different from seed to seed (because the convergence toward different points). Therefore the phenomenon is much clearer when the seeds are plotted individually rather than averaged, which creates artificially more variability in the backward trajectories as there really on each individual realization. This is why we reporting the various seed plots individually and not as single averaged plot with error bars. Note that the increased stability and point convergence is visible in each of the seeds, which are added to the Appendix B
0.50
Accuracy
Train Loss
2.0
0.35
0.30
0.25
tteraion Order
1.6
0.20
forward
backward
0.10
500
1000
1500
2000
2.4
0.45
0.40
Accuracy
0.35
Test Loss
2.0
0.30
0.25
1.6
0.20
0.15
0.10
1000
1750
500
1000
1500
2000
Iberation
Iteration
Figure 2: Backward SGD exhibits decreased variance and increased stability compared to forward SGD for a RosNet-18 model trained on CIFAR-10. The additional seeds are in Appondix B.1
Increased convergence stability for backward SGD: We trained a ResNet-18 with stochastic gradient descent and no regularization on the CIFAR-10 dataset Krizhevsky (2009). We used learning rate of 0.025 and batch-size of 8. Tn Figure 2 we recorded the learning curves at each gradient update for both the forward and backward iterations for 2000 steps. The additional seeds are in Appendix [..] We observe that the training loss for the backward trajectory is more stable, converges faster, and has less variability than the forward trajectory, and similarly for all the other learning curves. In Appendix A we repeat the experiment with different datasets (synthetic dataset, MashionMNIST, and CIFAR-100) as well as different architectures (ResNet-50, VGG, and MLP) and using different base optimizers (AdamW). Each time we observe the same phenomenon for all 5 seeds.
Convergence toward points v.s. distributions: We trained a MLP with 5 layers of 500 neurons each with stochastic gradient descent with no regularization to learn FashionMNIST Xiao et al. (2017). The batch-size set to 8 while the learning rate 0.001. In Figure 3 report the test accuracy at every single step for 2000 steps. At step 1000, we reset the point from which we perform backward SGD to that of the parameter at this step. As a result we see that the backward trajectory from that reset point seems to converge again, but to a different point (with different test accuracy), while forward SGD seems to oscillate between these two points. This is line with the theoretical prodiction that backward trajectories SGD converge toward points distributed according to the distribution induced by forward trajectories (sce Theorem 2.6 The additional seeds are in Appendix B.2
10

---

Published in Transactions on Machine Learning Research (12/2025)
0.85
0.85
0.80
0.80
Accuracy
Test Accuracy
0.75
0.75
0.70
0.70
Iteration Order forward
Iteration Order
0.65
forward intermittent backward
intermittent backward
0.60
0.60
500
1000
1500
2000
500
1000
1500
2000
Iteration
Iteration
Figure Backward SGD converges toward a different minimum after resctting the initialization point at step 1000 ("intermittent backward") while forward SGD oscillates between them for MLP trained on FaShiOn Top: On the first seed, backward changes from higher test-performance trajectory to lower fest-performance trajectory at the reset step 1000. Bottom: On the second seed, backward changes this time from a trajectory converging to lower test test-performance point to trajectory converging to a higher lest-performance point. The other seeds can be found in Appendix B.2 including all the learning curves.
5 Conclusion and future work
In this paper, we analyzed the impact of two different composition orders (to produce each iterate) the convergence and stability of stochastic lcarning trajectories. Although the very idea of backward trajectories may seem strange and counterintuilive al first, we show that this approach of backward optimization has clear advantages. Namely, we show theoretically that the backward trajectories converge toward actual points while the forward trajectories converge toward probability distributions in regions where the optimization maps become contractions, leading to improved stability and convergence. We observed this phenomenon experimentally on number of datasets (synthetic, FashionMNIST,, CIFAR-10, and CTFAR-100) and neural network architectures (MIP, VGG-19, ResNet-18, and ResNet-50), making it relevant to deep lcarning. This phenomenon points toward the fact that the ordering in which the data examples are consumed to produce each parameter iterate impacts the properties of both the learning trajectories and their convergence points. In particular the last examples used to produce a parameter iterate seem to have great importance. Indeod, the backward order is the only order in which the sequence of the last examples used to produce a parameter iterate remains consistent at each training step. As we saw, this yields convergence toward a point rather than oscillations between solutions of varied performances.
However, realistic applications of the full backward order challenging due to the prohibitive computation time that grows quadratically with the number of batches consumed. We now point toward conjectural workarounds and applications to showcase possible exploitation of this phenomenon. This is outside of the scope of this paper; nevertheless, we give supporting evidence in the appendix when possible.
One possible strategy to mitigate the intensive computational requirements of the full backward order is apply it only partially on fixed window of iterations from point that is reset periodically. This way the extra computation time is capped by the window size. We see that this approach leads to convergence and increased stability in Figure 3 however the backward trajectory may change and converge to different point at each resot. We can imagine strategics to initiate this reset only when the performance of the new backward trajectory increases, leading lo stable convergence to higher test. performance points. Another possibility is to start the backward order only later in training to force convergence toward a single solution after the forward trajectory has done enough exploration of the space as demonstrated Figure [1 in Appendix [A] We imagine that this approach may possibly lead toward more reliable early-stopping criteria. Similarly, application of the backward order at the very beginning of training may give accurate idea of whether
11

---

Published in Transactions on Machine Learning Research (12/2025)
specific hyperparameter setting is promising needs to be aborted by creating stabler trajectories whose learning curves are easier to interpret early on.
An orthogonal approach to deal with the computational challenges of backward optimization is to approximate the backward trajectory. In Appendix [
we show how to compute a backward iterate from a cheaper forward iterate to which we add a correction that. can be computed completely independently. We compute these corrections up to order 2 in Theorem
C.2 Theorem
C.2
namely:
T .1... Tn = Tx Ti + h
VL, VL,1+0(11)
ISKKYE
is the Lie bracket between the vector fields VL, and Vl.g. Even though the approximation we provide is not immediately useful because the higher order terms seem matter, it probably will be useful if higher order corrections are included. As proof of the usefulness of these type of corrections added to the forward iterates to emulate alternate orderings, we show in Appendix D that the corrections we computed at second order are alrcady enough to approximate possible ileration orders on a batch window. Adding these corrections to the forward iterates produces a beneficial regularizer that mimics small batch training regularization. In fact, a training step can be seen as a parameter average of mixture of models where each model is trained not from a different seed but from a different batch order. We hope that this paper will bring awareness and foster more research toward understanding and exploiting the role of iteration order in the production of models with more consistent. properties along more stable learning trajectories.
where [,, VL,10) = (,
average between all the
Broader Impact Statement
This work focused the thcoretical aspect of learning algorithms, espocially stability. We do not foresce any negative impact from this theoretical work.
Acknowledgments
The third-listed author was supported by the Swiss NSF Grants 200020-200400 and 200021-212864, and by the Swedish Research Council Grant 104651320. We are grateful to Massimo Horvath for several insightful comments on an earlier version of the manuscript. We would like to thank Michael Wunder, Peter Bartlett, George Dahl, Atish Agarwala, and Scott Yak for helpful discussion and feedback.
References
Atish Agarwala and Jeffrey Pennington. High dimensional analysis reveals conservative sharpening and a stochastic edge of stability. arXiv preprint orXiv:2404. 19261, Apr 2024.
Aamir Ali, Edgar Dobriban, and Ryan Tibshirani. The implicit regularization of stochastic gradient flow for least squares. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 233 233-244. PMLR, 2020.
B. Avelin and A. Karlsson. Deep limits and cut-off phenomena for neural networks. J. Mach. Learn. Res., (191):29, 2022.
Dmitry Babichev and Francis Bach. Constant step size stocbastic gradient descent for probabilistic modeling, 2018. URL https:://axxvv.or//eea http//aas/111..55565 05567
Racf Bassily, Mikhail Belkin, and Siyuan Ma. On exponential convergence of sgd in non-convex over- parametrized learning. ar preprint arXi::1811.0956,, 2018.
Mattia G. Bergomi, Patrizio Frosini, Daniela Giorgi, and Nicola Quercioli. Towards a aaalogical-goomotricall theory of group equivariant non-expansive operators for data analysis and machine learning. Nature Machine Intelligence, 1:423-433 2019.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877- 1901,, 2020.
Yuhang Cai, Jingfeng Wu, Song Mei, Michael Lindsey, and Peter Bartlett. Large stepsize gradient descent for non-homogeneous two-layer networks: Margin improvement and fast optimization. In The Thirly-eighth Annual Conference Neural Information Processing Systems, 2024. URL https:///ppenrevier.nett
forum?id==hlolUinnai
Yuansi Chen, Chi Jin, and Bin Yu. Stability and convergence trade-off of iterative optimization algorithms. arXiv preprint arxiv:1800 1ene1804.000 01619, 2018.
Xiang Cheng, Dong Yin, Peter Bartlett, and Michael Jordan. Stochastic gradient and langevin processes. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1810-1819, 2020.
Dami Choi, Alexandre Passos, Christopher J. Shallue, and George E. Dahl. Faster neural network training with data echoing. Xiv preprint XrXi::1907. 055505550,000 2019. URL
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability, In International Conference on Learning Representations, 2021. URL httpps://pperrevies net wttp//foruniid-h--rtrrraee
Benoit Dherin, Michael Munn, Mihaela Rosca, and David GT Barrett. Why neural networks find simple solutions: The many regularizers of geometric complexity. In NewrIPS, 2022.
Persi Diaconis and David Freedman. Tterated random functions. STAM Review, 41(1):45-70 1999. doi: 01088//$00331445583333
URL
Aymeric Dieuleveut and Francis Bach. Nonparamctric stochastic approximation with large step-sizes. The Annals of Slatislics, 44(4):1363 1399, 2016.
Aymeric Diculeveut, Alain Durmus, and Francis Bach. Bridging the gap between constant step size stochastic gradient descent and markov chains. Ann. Stalist., 48(3):1348 1382, 06 2020.
T. Team Gemini. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint (tl:2403.05500 2024.
Team Gemini. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.1180, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get m for free. In International Conference on Learning Representations, 2017. URL htttss:/openreevie net www..ntttooomm/ddddd//////
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunkool,, Russ Bates, Augustin Zídck, Anna Potapenko, ct al. Highly accurate protein structure prediction with alphafold. Nature, 596(7833):583 589, 2021a.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589, 2021b.
Nitish Shirish Keskar, Dheovatsa Mudigere, Jorge Nocodal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint 1rXiv::600 1009.69944400 069604486,, 2017.
13

---

Published in Transactions on Machine Learning Research (12/2025)
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Matthias De Lange, Gido M van de Ven, and Tinne Tuytelaars. Continual evaluation for lifelong learning: Identifying the stability gap. In ICLR, 2023.
Kiwon Lee, Andrew N. Cheng, Courtney Paquette, and Elliot Paquette. Trajectory of batci-batch momentum: Batch size saturation and convergence in high dimensions. In Advances in Neural Information Processing Systems, volume 35, pp. 72437-22444 2022.
Xiang Li, Wenhai Huang, Zheng Li, and Lei Zhou. On the variance of the adaptive learning rate and beyond. arviv preprint Xrx:::198.. 03265, 2019.
Ye Li, Song-Cas Can Chen, and Sheng-Jun Huang. Implicit stochastic gradient descent for training physics- informed neural networks, 2023. URL hatpp::/aarriv.oor///a///// 01767
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Appl. Comput. Harmon. Anal., 59:85-116, 2022.
Πya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint ar arXis:1111.0510. 2017.
Jeremy Mange. Effect of training data order for machine learning. In 2019 International Conference Compulational Science and Compulational Intelligence (CSCT), pp. 1006-007, 2019. doi: 10.1 10.1109/CSC14923 CCCCT9977 2019.00078.
Ibrahim Merad and Stéphane Gaiffas. Convergence and concentration properties of constant step-size sgd through markov chains. arXie preprint arXi::2306.114 1497, 2023.
Amil Merchant, Simon Batzner, Samuel S Schoenholz, Muratahan Aykol, Gowoon Cheon, and Ekin Dogus Cubuk. Scaling deep learning for materials discovery. Nature, (24(7790):800883 85, 2023.
Panayotis Mertikopoulos, Nadav Hallak, Ali Kavis, and Volkan Cevher. On the almost sure convergence of stochastic gradient descent in non-convex problems. In Advonces in Neural Information Processing Systems, 2020.
Sean P. Meyn and Richard L. Tweedie. Markoe Chains and Stochastic Stability. Communications and Control Engineering, Springer London, 1993.
Chodi Morchdi, Yi Zhou, Jie Ding, and Bei Wang. Exploring gradient oscillation in deep neural network training. Xi preprint Xrx:222306. 20055, 2023.
Yurii Nesterov. Introduclory lectures on conver oplimization: A basic course, volume 87. Springer Science & Business Media, 2013.
Zachary Novack, Simran Kaur, Tanya Marwah, Saurabh Garg, and Zachary Lipton. Disentangling the mechanisms behind implicit regularization in sgd. In International Conference on Learning Representations, 2023.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab, January 1998. URL http: Hillpubs.stanfon edu 80900422//
James Gary Propp and David Bruce Wilson. Exact sampling with coupled markov chains and applications to statistical mechanics. Random Structures & Algorithms, 9(1-2):223-22)) 252, 1996.
H. Qian and M.N. Wegman. L2-nonexpansive neural networks. In International Conference Learning Representations (ICLR), 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint ar Xrx:211002 12092, 2021.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:220400612 2022.
Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3))400- 407, 1951.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Joshua Baldridge, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:37117-371388 2022.
David ShirokofT and Philip Zaleski. Convergence of markov chains for constant step-size stochastic gradient descent with separable functions, 2024. URL http:://arxxvv.or///b//////// 12243.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXi:: 2014.
Samuel Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't decay the learning rate, increase the batch size. arXiv preprint arX:::1111.004 2017.
Samuel L Smith, Benoit Dherin, David G.T. Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In ICLR, 2021.
Petru Soviany, Radu Tudor Jonescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey, 2022. URL https://arxile httpaa///aaxiv.orgaabaaaa//
Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologics
physics-informed neural networks. ArXiu, abs/2001.04523 2020. URL https://api. semanticccholar
DTr/CoorpusDD:214473555
Jingfeng Wu, Peter L. Bartlett, Matus Telgarsky, and Bin Yu. Large stepsize gradient descent for logistic loss: Non-monotonicity of the loss improves optimization efficiency. In Proceedings of Thirty Seventh Conference on Learning Theory, volume 247 of Proceedings of Machine Learning Reseurch, 2024.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 708.07747 2017.
15

---

Published in Transactions on Machine Learning Research (12/2025)
A Additional Experiments
Note on plotting multiple seeds: We are interested in the variability per rcalization of the backward and forward Irajectories. The backward trajectories are more stable individually along their own paths, but these paths can be very different from seed to seed (because of the convergence toward different points). Therefore, the phenomenon is much clearer when the seeds are plotted individually rather than averaged, which creates artificially more variability in the backward trajectories as there really on each individual realization. This is why reporting the various seed plots individually and not as single averaged plot with error bars. Note that the increased stability and point convergence is visible in each of the seeds.
16

---

Published in Transactions on Machine Learning Research (12/2025)
A.1 Regression on synthetic datasets
In this experiment, to verify the increased stability of backward SGD over the standard forward version, we trained with both algorithms a neural network with 3 layers of 300 neurons with no regularization each to regress 100 points uniformly sampled from function graphs D = {(2;, yi) : y = f(r;) with xq = -1 + 100
2 i
O..... 100} using the functions f(x) = x² (Figure 4 f(z)= = (10x) (Figure 3). and f(z) = x3 (Figure
In all cases, we can observe a much higher stability for backward SGD over the forward version.
Terword SGD
0:00
Seed
0:02
0.00
220
1230
Tia
Farward 500
Terword 500
0.00
0:02
0.00
1230
Tha
Farward 500
Terwerd 5CD
0.00
0:02
0.10
1000
750
1003
1230
Tram
Farnard 560
Terwerd 550
0.00
0.44
0.02
0.00
1230
Tha
Farward 500
Peowerd 550
0.00
0.00
0.02
1230
Figure 4: Decreased variance and increased stability in train (left) and test (right) losses for backward SGD compared to forward SGD for all sceds. The data was sampled from f(x) = x² and the training performed with batch size 1 and learning rate 0.05 for 1400 steps.
17

---

Published in Transactions on Machine Learning Research (12/2025)
loos
Famaadd SGD
Ferwar 500
300
192
1000
250
2000
1250
Suckwardd 500
Backword 500
Seee
3.0
Buckwarrd SG0
04
S4a52
Last
Farmand 5CD
Torwart 500
Send3
250
300
353
1000
-250
220
200
3000
1250
Farwadd
Forward 5G0
Buchward 500
Backwerd 500
Seed4
Figure 5: Decreased variance and increased stability in train (left) and test (right) losses for backward SGD compared to forward SGD for all 5 seeds. The data was sampled from f(x) cos(10r) and the training performed with batch size 1 and learning rate 0.02 for 1400 steps.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Fanure S60
Tenwer SGD
0.00
Seed
0.09
01/
0.02
0.00
1230
Farvard 500
Tenwor 5C0
0.00
0.02
0.00
1230
Farward 500
Tenword SCD
0.00
0.02
0.00
3230
Farnurd 500
Terword SCD
0.00
0.02
1230
Farmard 500
Perwerd $50
0.00
0.02
1350
Figure 6: Decreased variance and increased stability in train (left) and test (right) losses for backward SGD compared to forward SGD for all sceds. The data was sampled from f(x) = x and the training performed with batch size 1 and learning rate 0.02 for 1400 steps.
19

---

Published in Transactions on Machine Learning Research (12/2025)
A.2 MLP trained on Fashion-MNIST
In this experiment, to verify the increased stability of backward SGD over the standard forward version, trained a MLP with 5 layers of 500 neurons each with no regularization using both forward and backward stochastic gradient descent with no regularization on the Fashion-MNIST dataset Xiao et al. (2017). We repeated the experiment for 5 different random seeds. For all seeds, we used a learning rate of 0.001 and a batch size of In Figure 7 we recorded the training loss at each gradient update for both the forward end backward iteration and plotted the 5 seeds separately (one seed for each row). For each seed, we observe that the training loss for the backward iterations is more stable and converges faster with less variability when compared to the forward iterations. We also observe stability improvements for all other learning curves for the backward trajectories.
3.2
Iteration Order Furwa
badkward
12
1.1
0.00
0.80
3.4
0.78
0.70
0.71
0.7
0.6
0.65
0.72
500
1000
1500
2000
500
1000
1500
2000
500
1000
3500
2000
0.70
500
1000
1900
2000
Iteration Order
forward
1.
0.00
badlward
1.
0.75
0.8
0.74
0.7
0.6
0.65
0.72
502
1000
1300
2000
500
1000
1.500
2000
320
1000
2502
2000
2000
1.3
0.85
0.82
Iteracion Order
forward
1.2
0.00
bacloaad
0.75
0.8
0.74
0.r
0.05
0.72
1.3
0.85
0.82
ftemation Order
Torward
1.2
baclwward
0.75
0.9
0.76
0.8
0.04
0.65
1.3
0.85
0.82
Iteration Order
forward
0.75
0.0
0.76
0.8
0.70
0.74
1000
1500 2000
1000
1500
2000
1000
1500
2500
2000
Benabion
Iteration
Berition
Figure 7: MLP trained on Mashion-MNIST. Training with backward SGD significantly reduces the variance and increases the stability of both the train and test loss well as the train and test and accuracy compared to forward SGD. This behavior is consistent across all seeds.
20

---

Published in Transactions on Machine Learning Research (12/2025)
A.3 VGG-19 trained CIFAR-10
In this experiment, to verify the increased stability of backward SGD over the standard forward version, trained VGG-19 model [Simonyan & Zisserman (2014) using both forward and backward stochastic gradient descent with no regularization on the CIFAR-10 10 dataset Krizhovsky (2009). We repeated the experiment for different random seeds. In all seeds, we used a learning rate of 0.001 and a batch-size of In Figure S we plot the training and test loss at each gradient update for both the forward and backward iteration. We plotted the 5 seeds separately, represented by each row. Note that for each seed, we observe that the training loss for the backward iterations is again more stable, converges faster, and has less variability than the forward iterations.
Thein Lowe
Test Leves
Then Accurecy
Test ACCurICT
0.5
2.2
2.2
0.6
1.5
0.3
1:2
0.2
2500
2000
2002
2000
2.4
Iteration
0.0
2.2
0.4
1.6
0.3
0.3
1.4
1.2
300
1050
1500
2000
500
1000
1500
2000
0.5
300
2000
13000
2030
300
2002
1500
2000
2.4
Iteration Order
0.5
baclorars
0.4
0.3
0.2
1.4
1.4
0.1
500
1000
1500
2003
1.2
500
1300
1500
2002
0.0
500
1000
1500
2030
0.0
500
1003
1500
2305
Itoration Order
2.4
farward
Lockward
2.0
0.4
1.4
1.5
0.1
300
2000
2500
2000
1.2
500
1000
2500
2000
0.0
300
2000
1300
2000
0.0
200
2003
1500
2000
Rteratio Ordder
2.4
forward
0.5
tachorar2
0.5
1.8
2,2
1.6
0.2
0.2
0.1
503
1000
1500
2000
500
1350 1500
2000
1500 2020
1002 1500
2303
Figure 8: VGG-19 trained on CIFAR-10. Training with backward SGD significantly reduces the variance and increases the stability of both the train and test. loss as well as the train and test and accuracy compared to forward SGD. This behavior is consistent across all seeds.
21

---

Published in Transactions on Machine Learning Research (12/2025)
A.4 ResNet-50 trained CIFAR-100
In this experiment, to verify the increased stability of backward SGD over the standard forward version, we trained ResNet-50 model He et al. (2016) using both forward and backward stocbastic gradient descent with no regularization on the CIFAR-100 dataset Krizhovsky) (2009). We repeated the experiment for 5 different random seeds. In all seeds, we used a learning rate of 0.001 and a batch-size of 16. In Figure 9 we plot the training and test loss at each gradient update for both the forward and backward iteration. We plotted the seeds separately, represented by each row. Note that for each seed, we observe that the training loss for the backward iterations is again more stable, converges faster, and has less variability than the forward iterations.
Thei
Test
Thei Accipacy
Teat Accurnsy
Perstion Order
0.00
5.32
fonward
backoord
3.20
0.05
5:25
5.25
0.05
0.04
3.50
0.04
0.03
4.50
4.50
0.02
0.02
4.25
4.20
0.01
4.00
500
1000
1500
2030
8.30
500
1000
1503
2003
500
1000
1500
2010
500
1900
2500
2300
6.02
9.50
5.75
Keration Order
forward
5.75
0.06
5.53
backward
5.50
0.04
7y /
4.75
4.70
4.50
4.30
5:81
2005
2038
6.00
6.00
5.75
5.75
0.05
5.22
3.20
1p 1/ 1//1//// /1rl
5.00
5.50
0.04
0.85
4.75
4.75
0.02
4.50
4.25
4.25
4.00
500
1090
1500
6.00
500
1000
1503
2007
0.00
1000
1:500
2010
8.00
500
1000
1503
2309
5.75
5.50
keration Order
forward
backword
5.75
5.50
0.07
0.02
0.00
5:25
5.25
0.03
0.04
5.00
3.00
0.04
0.03
5.02
4.52
4.50
4.25
4.20
4.02
4.20
300
1000
1500
2030
502
1000
1503
2003
500
1000
1550
2010
502
1000
1500
2000
6.02
9.20
5.75
Reration Order
foownrd
3.75
0.06
9.50
backward
3.50
5.00
5.50
0.58
4.75
4.75
4.50
4.30
4.25
5.23
Ttarsal km
Lerxilon
Figure 9: ResNet-50 trained on CIFAR-100. Training with backward SGD significantly reduces the variance and increases the stability of both the train and test loss well as the train and test and accuracy compared to forward SGD. This behavior is consistent across all seeds.
22

---

Published in Transactions on Machine Learning Research (12/2025)
A.5 ResNet-18 trained CIFAR-10 with AdamW
In this experiment we verify improved stability of backward ilerates for base oplimizer other than vanilla SGD. AdamW Loshchilov & Hutter (2017) combines momentum from Adam with weight decay regularization and improves upon the standard Adam by proventing the adaptive learning rates from distorting the intended regularization strength.
Here, we use AdamW for both forward and backward gradient updates, only altering the order in which batches are consumed. We train ResNet-18 model He al. (2016) and do not apply any regularization on the CIFAR-10 dataset Krizhevsky 2009). We repeated the experiment for 5 different random seeds. In all seeds, we used a learning rate of 0l00025 and batch-size of 8. In Figure TO we plot the training and test loss at each gradient update for both the forward and backward iteration. We plotted the 5 seeds separately, represented by each row. Note that for each seed, we observe that the training loss for the backward iterations is again more stable, converges faster, and has less variability than the forward iterations.
Thils Loss
Lost
Train Accurscy
Teat Accurccy
Beration Order
tackovers
2.2
0.5
0.4
1.2
0.1
bockward
2.0
0.4
0.3
0.2
505
1050
1500
2000
500
1:500
1500
2000
500
1000
1500
2030
1500
2303
2.4
2.4
0.5
2.0
0.3
0.2
1.4
2.4
300
1000
2503
2000
500
1000
1500
2000
500
1000
1300
2020
200
1000
1500
2009
Koration Order bockward
0.5
forward
2.3
0.4
0.3
2.4
0.1
0.1
553
10:00
3503
2300
500
3300
3500
2000
530
3000
15:00
2030
500
1000
1503
2355
2.4
2.4
Reration Ordeer tecleaars
Torward
2.0
0.5
0.4
1.4
1.5
0.2
12
0.1
0.1
553
2020
2503
2003
500
3500 1500 2002
520
3000
1500
2030
500
3000
1503
2303
Figure 10: ResNet-18 trained on CIFAR-10 with AdamW. Training with backward AdamW significantly roduces the variance and increases the stability of both the train and test loss as well as the train and test and accuracy compared forward AdamW. This behavior is consistent across all seeds.
23

---

Published in Transactions on Machine Learning Research (12/2025)
A.6 Backward stabilization after forward iterations
In this experiment, to verify the increased stability and convergence after turning on backward iterations after a number of forward iterations, we trained MLP with 5 layers of 500 neurons each with stochastic gradient descent with no regularization to learn the 10 classes of Fashion-MNIST datasct Xiao et al. (2017). We repeated the experiments for seeds. We used a learning rate 0.001 and a batch-size 8. In Figure we recorded the learning curves each gradient update for both the forward iteration and the backward iteration switched on after step 1000, Each seed plotted independently. We observe that the training loss for the backward iterations has again more stable convergence once backward iterations are switched on after step 1000 than the forward iterations.
Loss
Accuracy
Accuracy
Paeabion Deder
forward
0.850
0.650
farwant badioaadd
0.8
0.8
0.000
0.000
0.775
0.775
0.750
A7750
0.5
500
1000
15:0
2000
0.4
50D
1000
1500
2000
700
500
2000
2500
2000
700
500
1000
150D
2000
0.9
Eoration Order
fenward
0.9
5.050
0.050
wwwant_bbciwadd
0.925
n.825
0.8
0.8
0.7/5
0.775
0.750
0.150
0.5
0.5
500
1000
1500
2000
500
1000
15500
2000
500
3000
3500
2000
500
1000
1500
2000
1.0
0.0
heration Order
forward
0.0
2.250
0.850
forward bacioaard
0.825
0.823
0.771
0.775
0.6
0.750
0.750
0.5
0.5
0.725
1000
1500
2000
2200
1.500
500
25000
3.0
1.0
0.9
Lan aliin Ordder
fonward
0.9
p.a50
0.850
forward baccwward
0.925
0.775
0.775
0.6
0.6
0.750
0.750
0.5
0.5
0.725
0.725
2000
3.0
1.0
Peration Order
0.850
0.850
forward
ferward bacioward
0.825
0.625
2.000
A.no
0.8
0.6
0.750
5.725
0.725
2006
2000
Boratton
Iterat
Iteration
toration
Figure 11: Decreased variance and increased stability for all learning curves and all seeds once backward SGD is switched on at step 1000 after forward SGD iterations for an MLP trained on Fashion MNIST.
24

---

Published in Transactions on Machine Learning Research (12/2025)
A.7 Continued stability of backward SGD throughout training.
In this experiment, to verify that the behavior of increased stability and convergence for backward iterates continues throughout training, we trained a ResNet-18 model on the CIFAR-10 dataset using the same hyperparametors as in Figure 2 (i.c., no regularization, a lcarning rate of 0.025 and batch size of 8) but training for 25000 steps instead of 2000. In Figure [2] we record the learning curves for both forward and backward SGD, performing model evaluation every 100 steps. Throughout training, we continue to notice that the training loss (as well as all the other learning curves) for the backward iterations is again more stable, converges faster, and has less variability than the forward iterations. Note, here only performed this experiment for one seed because because of the high compulational requirements for backward SGD with so many training steps.
222
2.2
2.00
2.0
1.75
1.50
1.6
3.4
1.00
1.2
0.8
5.00
1.0000
15000
20003
25000
10000
15000
20000
25000
Iteration
Iteration
0.9
0.7
0.8
Agcuracy
0.6
0.5
0.5
0.4
0.3
Itoration Order
forward
0.2
backward
0.1
0.1
10000
35:00
20000
25000
10000
15000
20000
Iteration
Reeation
Figure 12: ResNet-18 trained on CIFAR-10 for 25000 steps. Decreased variance and increased stability throughout model training.
25

---

Published in Transactions on Machine Learning Research (12/2025)
B
Additional seeds to plots in the main paper
We are interested in the variability per rcalization of the backward and forward Irajectories. The backward trajectories are more stable individually along their own paths, but these paths can be very different from seed to seed (because the convergence toward different points). Therefore the phenomenon is much clearer when the seeds are plotted individually rather than averaged, which creates artificially more variability in the backward trajectories as there really on each individual realization. This is why reporting the various seed plots individually and not single averaged plot with error bars. Note that the increased stability and point convergence is visible in each of the seeds.
Note on plotting multiple seeds:
B.1
All 5-seeds for ResNet-18 18 trained on CIFAR-10; cf. Figure 2
In Figure below, we plot the learning curves for all five seeds in Figure of the main paper.
Train Loos
Test Loss
Trai Accuracy
Test Accuracy
2.0
2.2
1.0
1.5
1.4
1.4
Iteration Cindtr
1.4
Backoaard
503
1050
1500
2000
500
1:00
1500
2000
500
2000
1500
2030
2002
1500
2003
0.4
1.8
1.6
1.5
Ittrattion Order
forward
0.1
1.4
1.4
backward
300
1000
1500
2000
590
1000
1500
2000
0.0
500
1000
1500
2000
0.0
1003
1500
2309
0.4
0.2
Itenation Order ferward
0.1
backward
353
1000
2500
2000
500
3300
1500
2002
0.0
500
3000
1500
2000
0.0
500
3003
1500
2005
0.4
0.2
Iteration Order
forward
0.1
backward
500 1000 1500
2000
590
1000
1500
2000
500
1000
1500
2000
500
1003
1500
2000
0.4
0.3
0.2
Ileration Otider ferward
baccnard
303 2500
2000
500
1500 2000
2020
2303
Itrrtton
Beration
Perraton
Iteratiion
Figure 13: All 5-seeds plot of Figure
26

---

Published in Transactions on Machine Learning Research (12/2025)
B.2 All 5-seeds for MLP trained FashionMNIST; cf. Figure 3
In Figure
14
below, we plot the learning curves for all five seeds as in Figure of the main paper.
Eoration Order
M
foroard
Intemmetent bacoward
0.0
0.80
0.8
0.70
0.70
0.5
0.5
0.85
500
1000
1500
2000
500
1000
1500
2000
520
1000
1500
2000
500
1000
1500
2000
hecation Order
0.4
Forward
0.0
0.00
0.00
Intermattent beciosadd
0.75
0./0
0.3
0.5
0.05
0.80
1000
1500
1000
500
3.5
1.0
0.85
Torward
Benation Dricler
0.9
0.60
eitemneliens bacleward
0.75
0.75
0.70
0.6
0.6
0.65
1.0
1.0
0.01
Forward
heration Order
0.00
Wn AAAA
Internittent bacioward
0.75
0.75
0.8
0.6
0.65
3.5
1.0
0.85
Eenration
0.60
0.75
0.7
0.7
0.6
0.65
Beretion
herre
(teretion
(tererion
Figure 14: All 3-seeds plot of Figure 3
27

---

Published in Transactions on Machine Learning Research (12/2025)
C Toward an approximate backward SGD
Backward SGD has increased stability and convergence over forward SGD but naive implementations are computationally intensive. In this section, we propose an approximation of the backward itcrates by modifying the forward iterates with Lie bracket term of order (112) the learning rate (Theorem C.2p. This comes an extra cost, but which can be made smaller than evaluating the backward iterates from the initial point at every step (Corollary C.4 Our approximation is valid
for any optimizer of the form T)(0) = A (+), where Vi(0) is a random vector field on the parameter space depending on the randomly sampled batch data B, (Of course, in the case of SGD we have Vi(0) (0) where L, LB, the loss function evaluated on batch Bz.)
up an error of order (hh)) in the learning rate.
Unfortumately, it seems that in real-life cases higher orders beyond (1)²) play a significant role in the backward dynamics, and therefore can not be neglected for reasonable ranges of the learning rate. Nevertheless, we include our second order approximation here, since we believe it gives a reasonable path on how to produce approximations of the backward trajectories using the forward trajectorics. Namely, while this is out-of-scope for this paper and lopic of further research, believe that adding higher order corrections may produce useful approximations that could be used stabilizers for the forward trajectories. The main idea is to expand generic k term iterate
Tú (0) (1 MV,) (1
(6)
in Taylor's series in the learning rate. This is what the next lemma gives. (We demonstrate the usefulness of such expansions Appendix D where we use Lemma C.1 to produce a beneficial second order regularizer that emulates an iteration order average.)
Observe also that Lemma C.1 tells that all possible iteration orders coincide order h but start differing at order ((h)):
Lemma C.1. Consider sequence (Titiso of operalors of the form T:(0) = 0 + NV.(0), where V.(0) is vector field on the parameter space. Then for any choice of indices i,, in we have that
T, ... T = 1
VR + 12
Σ ESacuck
V. +0h3)
1=1
Proof. We proceed by induction. For the base case, k = 1, this is trivial. Suppose now that this is true for any composition of k - operators. By definition of Ti we have that
T. ... Tú, (0) = T; (T; ... T., = X + AV, (X) with = Ti.... T., (0). Now by induction hypothesis we have that
(7)
&
x > V4,
(0)
+12
Σ (0)
(8)
1=2
25ucoSk
Therefore, taking Taylor series for the second term of 7 we obtain
AV. VV, (0) + 12
(0)
1=2
Summing up in 7 the expressions we have found for X and hV, (X) above, obtain that the composition Th ...... T,, (0) has the form
+ h
(0) A2
Σ
(LV..). O(12),
1=1
Isucesk
28

---

Published in Transactions on Machine Learning Research (12/2025)
which completes the proof.
The next theorem gives way to approximate the backward iterates up to order ( by correcting the forward iterates with term keeping track of their difference:
Theorem C.2. Consider sequence (Ti of operators of the form T;(0) 0 hV,(0), where V:(0) is vector field the parameter space. The bockward and forward iterates of the sequence related by the following identity:
T TT...Ta(()) = =T, ... (i(0) + +12
E
(VV11()) + 0
1SKJJSN
where the [V, (.) V((0) - V((O)V((0) -V(0)(( is the Lie bracket belween the vector fields V and Vj.
Proof. By Lemma
C.1 we have that the backward iterate
T T,
Σ V + h
Σ
WV + 0
Igacegn
while the forward iterate is obtained by reversing the indices:
Ta ... -1
Σ
Vi 2
Σ
V. + 0(h))
1=1
ISucUSh
We now see that the difference
D(0) = Ti...... ... (0)
........(0)) - Tú (i(
between the backward and forward iterates is of the form
D(0)
h² Σ (O)
((00)
ISuCUSD
1
Σ
(.) - (117).
IsuceSn
which completes the proof.
We now consider the case of SGD where the vector fields Vi are given by the gradients of the loss evaluated at the current batch. We give below a definition of the approximate backward iterate in that particular case, although a similar definition can be given for any vector fields.
Definition C.3. Consider the SGD operators T,(0) 0 - LV),(() obtained by taking the gradient of loss function on batch B; of data at step i. We denote by 0,, T, Ti (0,) = 0,-1 hv (0n-1) the forward SGD iterate starting at initial point d, and by 0 T (o,))())) the corresponding backward iterate starting at the same initial point. Motivated by Theorem C.2 we introduce the approzimate backward iterate 6, as follows:
0, = 8n +
VL, (L))
(10)
Isicisn
Theorem C.2 tells us that the true backward iterate and its approximation are within O(h²) of each other:
110, - 0z 0(h²)
(11)
which allows us to use the approximation for learning rates small enough so that the terms in O(h3) can be neglected. The following corollary tells us how the approximate backward iterates can be computed in an iterative fashion by keeping in memory additional variable C, of the same size the network parameters:
29

---

Published in Transactions on Machine Learning Research (12/2025)
Corollary C.4. In the notation above, con obtain the approrimate backward SGD iterate 6, from the forward SGD iterate On slarting at Og in ileralive fashion as follows:
On On-1 L,
9n. = 9n-1 + V 1
H. = Hn-1 + V² ((0)
C Cn-1 + Hn V In (0o) (200)--22mp (m
en On + h
with 90 = 91 =0 0, Hn H1 = 0, and cn = c1 =0
Proof. By Definition C.3 of the approximate backward ilerates we have that 0,, - 0n + 1² with
CN
E
[L. VL,1(00).
ISiesS"
First observe that we can split Cn into two parts
On
E
VLg (00) +
Σ 1sisn-1
VLi (l))
IsKjJS-11
Cn-1
+[
Vhg, (On)
Kisn-1
Cn-1+
Σ
b2I((0)
(L.(
Isisn-1
Ln
Σ
(o,(
1sign-1
C.-1+ (o, - Ln
where Hn and 9n are expressed recursively as in the theorem statement.
D Approximate implicit regularization of smaller batches
Dherin et al. 2022 for instance, small batches have an implicit regularization effect, producing solutions with higher test accuracy as the batch size decreases. In this section, we show that approximations the type given by Lemma C.I are useful to understand this implicit regularization effect. We also see how we can go beyond this small batch effect and produce explicit regularizers with even higher test performance based the idea. of iteration order average the small batches, leveraging Lemma C.1
As shown in Keskar el al. 2017 Smith et al. 2017 2021
Throughout the section we will write (i()) for the negative batch loss gradient - L.,((f) computed on batch B, in order to keep the notation simple.
D.1 The effect of small batches
The idea is to use Lemma to understand the implicit regularization effect of small batches, very much in line with the findings in Smith et al. (2021) First, consider two settings. In the first setting, we perform a single gradient update
Tirgge (0) = hVB(O),
with one large batch and learning rate h. In the second setting, we split the large batch B into e small batches of equal size: B1, Bz,..., B. (i.e, each of size Bl/c). Then we apply SGD sequentially times for
30

---

Published in Transactions on Machine Learning Research (12/2025)
each of the smaller batch and with learning rate h h/c:
Tmmall (0) (1+ +V
Then Lemma C.1 gives us immediately second order approximation for the second setting:
T smaal (0) = + 11' Σ
V:(0) 2
+(0))1(0)+000)))
==1
Now it is easy to see that
(V 1/ Σ V((0).
Therefore, we can extract the added implicit regularization that is induced from smaller batches. Namely, we obtain that the composition of the smaller batches the same as a single gradient step with the larger batch using learning rate h = ch' plus an additional second-order regularization term:
smmall (0) Targe (0) + h Σ (1()), + O(11
Smaller Batch Regularization
So we can attribute the second order regularization effect of small batch training to this additional term, in line with the computations in Smith al. 2021
D.2 Explicit regularization through iteration order averages
In the previous section, we applied the small batches in some chosen order. Another order may have worked well, although producing different second order regularization term. This begs the question of whether can instead take the average over all possible iteration orders so as to produce an iterate for which no particular order is preferred:
T peru (0) - 1 Σ
(1+ (1 +1+ NNV (0)))
(12)
oCPerm.
where Perm, is the permutations of a elements. Again Lemma C.1 for this new update rule:
gives us the second order approximation
T perm (0) = 7 Targe (0)
h²
Σ +(00)
i
In the next section, we show that the iteration- ceniiioo-orde---vergiinn term, that we can extract from the computation above, namely,
((()),
produces more powerful regularizer than the obtained from single ordering of the small batches. In way, this new regularizer emulates mixture of models, each of which is produced by different ordering of the iterations in window of batches (rather than from different random seed). The next section shows the benefits of this new "ordering-Tree" regularizers experimentally.
31

---

Published in Transactions on Machine Learning Research (12/2025)
D.2.1 Experiments
We trained a MLP with 5 layers of 500 neurons each with stochastic gradient descent with no regularization to learn the 10 classes of Pashion-MNIST dataset Xiao et al. (2017). We used a learning rate of = 0.001 (and h'= h/c), and a batch-size of [ 204 We examine three different shrinking factors E {2, 4). In each setting, test the values of X E (0.5h² h? 2h 4h²). The following figures prosent the train and test loss over training compared to training with large-batch |B| 2048 and learning rate h' = h/c, and training with small-batch 2048/c c with learning rate h'. The experiment shows that our method can outperform vanilla training both for large and small batches with minimal tuning of A for this specific experiment.
0.6
Than
0.60
legg Jutch
lage Jurch
1emiida-2..
bumboda-4..
0.50
00.3
0.42
0.
0.1
0.0
1000
2000
$420
4000
3000
5003
7500
0.35
1000 2000
3000
Shop
.003
5500
5000
1003
Figure 15: Train curves for injecting regulariza- tion resulting from splitting the batch in two (i.c., c= 2).
Figure 16: Test curves for injecting regularization resulting from splitting the batch in two (i.e., 2). As you can see, the performance improves as we increase the X.
0.6
Tran Loss
bege,
0.60
Tex: Los
0.5
lombda-5.
batch
0.55
lambbda-4.0
0.
8
0.45
0.40
0.0
1000
2300
2000
4000
5000
6002
7000
Shep
Figure 17: Train curves for injecting regulariza- tion resulting from splitting the batch in three (i.e., c. 3).
Figure 18: Test curves for injecting regularization resulting from splitting the batch in three (i.e., c = 3). Similar to the case where c = 2 above, the performance improves as we increase the A.
2Qr the closest. integer divisible by 6; e.g., 2049 for c= 3.
32

---

Published in Transactions on Machine Learning Research (12/2025)
0.6
Then Loss
0.50
Rest LOGS
paten
0.5
lambda--..
0.55
lambdaa-1.
0.4
0.33
0.40
0.0
1000
2000
3030
Steo
1000
5550
6003
7000
0.35
Step
4003
0000
Figure 19: Train curves for injecting regulariza- tion resulting from splitting the batch in four (i.e., c 4).
Figure 20: Test curves for injecting regularization resulting from splitting the batch in four (i.c., = 4). Similar to the cases above, the trend continues and even more visible; Performance improves we increase the A.
E Proof of Theorem 2.6
E.1 Definitions
Before we dive into the proof, let us define the following three different notions of convergence of random variables. First we have the strongest notion, which is convergence almost surely, That is, sequence of random variables X, converges to a limiting random variable X almost surely if
P( lim X, Xx)= 1.
This can be thought as sort of point-wise convergence. weaker notion of convergence which implied by almost sure convergence (see Theorem 2.7 (i) in Van der Vaari (2000)) is convergence in probability. The sequence Xn of random variables converges to the random variable Xxx in probability if for every >> 0 the following holds
lim P( X >x)2))) c)= 2=300
That is, for large it becomes more and more likely that X,, is close X The weakest notion of convergence is convergence in distribution, which is implied by convergence in probability (see Theorem 2.7 (ii) in Van der Vaart (2000))- There many equivalent ways to define this, but the most concrete way is the following: Let (x) P(X,, <)) be the cunulative distribution function for Xn and let Fxo (x) be the corresponding cumulative distribution function for Xxo: then we say that the sequence X,, converges in distribution to X if for every continuity point x of the cumulative distribution function Foo we have the following
lim Fz(z) = (x).
If we define n4, dFR and Moo oF then when say that pnn - Moo convergence in distribution in the above sense.
E.2 Proof
Consider a sequence The probability distribution of the forward iterates 0, T,Tn-1 ... Ti (00) given by
independent and identically distributed random operators.
... Ti) where da is the delta distribution concentrated at 0g (i.e. S0, (A) if 0g A and 0 otherwise).
33

---

Published in Transactions on Machine Learning Research (12/2025)
We want to show that when the backward iterates converge to random point (randomness is due to the sampling of the random operators T; 8), i.e., when
TT ... (.66)) _ 0" as n - o0,
then this implies that
the probability distribution of the forward iterates from 0, converge to stationary probability measure Pn + pa-
the random point 0 is distributed according to the same forward iterate stationary distribution pe..
To see that let y denote the forward iterates n
yo (
and let X denote the backward iterates
x
00) E Rd as Define the cumulative distribution function (CDF) for the random vector y and (0), (0),....0')
F == PP S s"..... aa S 8d).
By independence of the random operators Ti' we know by symmetry that (f) is also the CDF of x
Under our assumption that the backward iterates converge random point l" (almost sure convergence) will define the CDF of that random point as, (0) and for the sake of the proof we denote that point with X instead of 8 to highlight that it is random vector.
Now, our assumption that the backward iterates converge to point implies that the random vector X converges to X% almost surely, in the sense defined above. As alluded to in the previous section above, the almost sure convergence implies convergence in distribution, i.c., the convergence of the CDF lim (0) = (0), for every where Fly continuous.
Since X and Y have the same distribution (same CDF) we have immediately that Y also converges in distribution to X The limiting stationary distribution μlg- is simply the probability measure corresponding to F%, i.e. p4g- = dF%
F Proof of Lemma 2.5
Consider a smooth function I.: IRd + R satisfying 13 and 4
We want show the following inequality
[T T(02)l < V1 2/m + 12 |10
02ll
for the gradient operator T(0) 0 - L().
We can see that by first expanding the square of the operator difference:
[[T(0:) - 12 = ||0 0211
2h (LL()) (L())), 01 02)
+ V - VL(02)
Now applying the strict convexity conditions, we get
[TT(0,) -(22)] < 100 - 021 2/m - 0211
+ 2 00 02 12
(1 2hm h² (2) 8211.
which ends the proof by taking the square root both sides.
34

---

Published in Transactions on Machine Learning Research (12/2025)
G Discussion on the assumption of contractions.
The following discussion demonstrates that the contraction hypothesis required in Theorem 2.2 can be satisfied under standard conditions that frequently arise in deep learning. Specifically, we show that gradient descent. transformations are contractions with respect to a natural loss-based pseudo-metric metric when the loss function satisfics both Lipschitz and Polyak-Eojasieviez conditions.
Setup and Assumptions Let S be domain in RM representing the parameter space, and let S R20 be loss function to be minimized, having the form
M
((O)
Σ 100)) m
We make the following standard assumptions:
1. Each individual loss 4 2 O and the global minimum value of L is O. This interpolation assumption is reasonable for over-parametrized systems where the model can perfectly fit the training data.
2. All functions differentiable.
3. There exists a constant C' such that all gradients V4 (and therefore L)) are C-Lipschitz continuous in the relevant parameter space S.
The Polyak-Lojasieviez Condition Additionally, we assume that L satisfies the Polyak-Lojasiees (PL) condition: there exists constant such that
1
2VL()))22111 2
for all 0 S. This condition, while satisfied by strongly convex functions, is particularly important because it also holds for por-parametrized neural networks (which are typically non-convex). as demonstrated in Liu et al. (2022). The PL condition essentially ensures that the gradient magnitude provides a lower bound on the distance to optimality.
Loss Pscudo-Metric natural petric- based on the loss function.
To analyze the contraction properties of gradient-based updates, we introduce a
Definition. For the parameter space S, define the loss pseudo-metric S S:Sx
S S
Rz by:
[(x) + L(y) if + y.
d(x, v)
ifx y.
Properties.
This function satisfies the standard metric axioms:
Symmetry: d(z.y) v) L(x) L(y) L(v) L(x) d(y,
Triangle inequality: For any x, y,
d(z, v) L(x) L(y) < L(x) + L(2) + L(z)+ -(y) d(x, + d(z,)),
which follows from the non-negativity of L.
Pscudo-metric property: d(x, y) = for x # y if and only if x, E Z, where Z = (0 S: L(0)
0} denotes the set of global minima.
The key insight is that measures the "total suboptimality" of two points. Unlike true metric, distinct points in the zero-loss set Z have distance zero from each other, hence the term pseudo-metric.
35

---

Published in Transactions on Machine Learning Research (12/2025)
Contraction Analysis for Full-Batch Gradient Descent We now establish that gradient descent is contraction in the loss mseudo-metric d. The Lipschitz condition on VL immediately implies that for any y E S:
- J/ VL(x + -y--((y))--))))))))))))))))>>)() (y-
- x) & (VL(x), - x) + C ly x1|² 2
Consider the full-batch gradient descent transformation T(x) (V with learning rate h > 0. Applying the above inequality with T(x), we obtain:
L(T())) L(x) L()) (VL(T), C [T(x)) =12²
-VVL(O), + C
|~))((
- --m(1- Ch |V((T)]]
Combining this with the PL condition and requiring 0 < h < 2/C (a standard step-size constraint), we get
L(T())) -h (1- (1-1111)) Ch
Ch
| - -2hu (1- 2 L(x).
This yields the key contraction result:
L(T(x)) s (1 - hu(2 - Ch)) L(x)
for sufficiently small h. The factor == 1 1-11pp2 h2 Ch) points y S, have
represents the contraction rate. That is, for
dT),TTy) = L(Tx) + L(TV) S kL(x) + kL(y) = kd(x,
establishing that is a contraction in the loss pseudo- metric d.
Extension Stochastic Mini-Bate Updates For stochastic gradient descent with mini-batches, we consider transformations of the form
Txxx) = AV
(x),
JEL
where 1 denotes a mini-batde (a subset indices) of size m. Through a similar to above but more technical argument (sce (Bassily et al., 2018 p. 3-4)), one can show that
E [L((T(()) s kL(x),
for some contraction factor 0 < k < 1, provided the hyperparameters are chosen appropriately. The key insight is that while individual mini-batch updates may not be contractions, they are contractions in expectation.
Application to the Backward Contraction Principle The contraction analysis established above directly enables the application of our backward contraction principle (Theorem 2.2 to gradient-based optimization. Several technical points ensure the theory applies seamlessly:
Pseudo-metric compatibility: The contraction principle extends naturally from true metrics pseudo-metrics. In our loss pseudo-metris framework, the zero-loss set Z effectively becomes single point when passing to the metric completion, serving the unique fixed point of both T and any T,,
The required condition 2 can be satisfied by restricting the parameter space S. For instance, imposing L(0) < B for some bound B creates an invariant set since gradient descent decreases I. (and T; decreases B[[.])-
Bounded displacement condition:
36

---

Published in Transactions on Machine Learning Research (12/2025)
Stochastic contractions: For mini-batch updates that are only contractions in expectation, more sophisticated analysis techniques are available (see Diaconis & Freedman (999) and references therein).
This framework provides the rigorous foundation needed to apply our backward iteration theory to practical deep learning optimization algorithms.
37


Reviews:
[{"review_id": "KXLWTfClyv", "reviewer": "Reviewer_gbTF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates how the composition order of maps affects convergence and stability in optimization. The authors introduce and study a training variant—backward-SGD—in which each iterate is obtained by applying all past batch updates in reverse order rather than the standard forward order. They prove that, under uniform contraction conditions, backward iterations converge to a single point at an exponential rate (Theorem 2.2). Moreover, they establish that the limiting backward points are distributed according to the stationary distribution of the forward process (Theorem 2.7).\n\nStrengths:\n\n1. The idea is conceptually novel. It could have important implications for training stability, early stopping strategies, and hyperparameter tuning.\n\n2. The authors support their claims with clear toy examples, analytical derivations, and extensive empirical validation on real-world applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The theoretical results are supported by formal proofs and validated through experiments.", "audience_yn": "Yes", "audience_explanation": "Yes. The paper’s findings on how iteration order affects convergence and stability would likely interest TMLR readers working on optimization, training dynamics, and algorithmic stability in deep learning.", "requested_changes": "I have the following concerns and questions:  \n\n1. Backward SGD is significantly more expensive than forward SGD in computational and memory costs. Even with the approximation proposed in Section C, it still requires computing Hessians.  \n\n2. While Theorem 2.2 proves convergence to a point, it does not guarantee that this point is meaningful. In the SGD context, Examples 3.1, 3.2, and 2.4 suggest that the first map $T_1$ has a large influence — the initial error term $h \\epsilon_1$ persists and does not decay. By contrast, forward SGD with a diminishing step size can be shown to converge to the true minimizer. For backward SGD, if the first batch is unrepresentative or poor, the algorithm may still converge, but to a suboptimal point. In line with Theorem 2.7, this means that, in terms of performance, backward SGD is not inherently better than forward SGD.  \n\n3. Applying Theorem 2.2 to SGD may require restrictive conditions, as noted in Remark 2.3: namely, that each per-batch gradient descent step is a uniform contraction, and that there exists a $\\theta^*$ which is a critical point for all $L_i$.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6g5OHnzGmK", "reviewer": "Reviewer_rt2N", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies how the *order* of composing stochastic update maps affects optimization. It defines a “backward” trajectory that applies the per-batch update maps in reverse order, and proves a *backward contraction principle* showing pointwise convergence of the backward iterates under uniform contraction assumptions (Theorem 2.2). It also relates the backward limit to the stationary distribution of the forward Markov chain (Theorem 2.7). Simple models illustrate why noise decays in the backward order, and experiments on CIFAR-10/100 and Fashion-MNIST with small batches and fixed learning rates suggest improved stability of the backward trajectory. The paper discusses the prohibitive cost of the naive backward scheme and proposes an analytical approximation using Lie-bracket corrections (Appendix C).", "claims_and_evidence_yn": "No", "claims_explanation": "The paper’s core theoretical claims (e.g., the Backward Contraction Mappings Principle, Thm. 2.2) are stated clearly and proved correctly under their assumptions, and the link between backward limits and the forward stationary distribution (Thm. 2.7) is sound. \n\nHowever, the assumptions (uniform contraction; i.i.d. random operators; bounded displacement) are strong and not validated in realistic deep-net settings, and the empirical section, while suggestive, does not control for the large extra compute of naive backward recomposition, which confounds claims of “faster” or “more stable” convergence relative to forward SGD. Finally, the Lie-bracket approximation is presented theoretically but its practicality is not demonstrated. Overall, the evidence is insightful but not yet convincing for the broad claims about stability/convergence advantages in practice.", "audience_yn": "Yes", "audience_explanation": "Iteration order and stability are timely topics for the optimization and generalization communities. The paper offers a crisp theoretical lens (backward vs. forward compositions) and empirical patterns (reduced variance across seeds on CIFAR/Fashion-MNIST under small-batch, constant-LR SGD) that many readers will find thought-provoking even if not yet practice-ready.", "requested_changes": "1. **Compute-normalized experimental comparisons.**\n   Compare backward and forward under equal FLOPs/wall-clock (or report TFLOPs/energy) to separate algorithmic stability from extra compute due to recomposition. Include windowed/backward-late variants at matched budgets;\n\n2. **Quantitative stability metrics and statistics.**\n   Beyond per-seed plots, report per-seed variance of loss/accuracy, oscillation amplitude, integrated path length, and confidence intervals. Summarize test accuracy distributions and run basic statistical tests across seeds;\n\n3. **Baseline controls that use similar extra compute.**\n   Add cheap, compute-matched baselines: (i) tail-replay of last *M* batches; (ii) EMA/SWA; (iii) gradient accumulation / multiple passes on the same mini-batch; (iv) intermittent short-window reversal;\n\n4. **Clarify scope/assumptions and provide diagnostics.**\n   Discuss when contraction-like conditions plausibly hold in deep nets; include empirical diagnostics (e.g., local Lipschitz/contractivity estimates around late-training iterates) and comment on i.i.d. vs. epoch-shuffle dependence.;\n\n5. **Practicality of the Lie-bracket approximation.**\n   Provide complexity/memory analysis and an implementation using Hessian-vector products; show accuracy vs. cost and sensitivity to learning rate (since higher-order terms appear to matter). Include ablations of order-2 vs. higher-order corrections, or clearly delimit why higher order is needed.\n\n6. **Coverage of optimizers and settings.**\n   Add results with momentum/AdamW and at least one non-vision model (e.g., a small transformer) to test generality; report behavior with LR schedules and standard augmentations.\n\n7. **Reproducibility and clarity.**\n   Release code and add concise pseudocode for: (a) naive backward; (b) windowed/reset/backward-late; (c) approximate backward update (caching & Hessian-vec usage). Specify data-loader policy (with/without replacement), seeds, and evaluation cadence.\n\n\n8. **Positioning vs. related ideas.**\n   Briefly connect the approximation to BCH-type expansions; relate the empirical stability narrative to “edge of stability” and curriculum/order effects discussed in the intro.\n\n9. **Minor edits.**\n   Clean up typos and unify notation (e.g., identity element formatting) and figure captions. (Examples occur around App. C and the figure appendix.)", "broader_impact_concerns": "The submission states it does **not** foresee negative impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sHKQT02GL9", "reviewer": "Reviewer_Pyai", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis paper studies how reversing the order of samples in stochastic gradient methods affects the algorithm's convergence behavior. The authors show that their backward iteration has better stability properties compared to the standard SGD update by analyzing the contraction of the associated random operators. Some numerical experiments validate the effect of the proposed method.", "claims_and_evidence_yn": "No", "claims_explanation": "**Weaknesses**\n\nAlthough the perspective is interesting, I have several concerns regarding its relevance to real neural network training, and more broadly, stochastic optimization.\n\n1. Theoretical result\n\n   One of the main results of the paper is the claim that \"backward update\" has better stability properties compared to forward update. The idea of the paper can be summarized as follows: since the backward update injects randomness to the beginning instead of the end of the trajectory, the algorithm will finally be dominated by the initial samples, say $\\{B_1, B_2, \\ldots, B_{M}\\}$. Given the assumptions in this paper on the random operators, the \"convergence of iterates\" follows almost immediately. \n\n   However, I have reservations about its significance/soundness.\n\n   First, I feel it's unreasonable to treat each \"backward iteration\" as an algorithm update, since generating \"backward iteration $k$\" involves running SGD through the entire algorithm trajectory over $k$ samples; computations from all the previous iterations are discarded. Therefore, the paper essentially studies the relation between the last iterates of different SGD trajectories, given that the finally encountered samples -- $\\{B_1, B_2, \\ldots, B_{M}\\}$ are the same. In my opinion, this approach is not directly comparable to SGD. \n\n   Besides, I also find that the claimed \"stability\" result unconvincing. For example,\n\n   - The paper uses operator notation throughout and applies a *local* second-order Taylor expansion $T(\\theta) = \\theta^\\star + (1 - h \\nabla^2 L (\\theta^\\star))(\\theta - \\theta^\\star)$ in several places. In other words, several results are discussed almost in convex quadratic case. However, real neural networks are typically not smooth or convex. \n   - Even under twice differentiablity assumption, to ensure that the local expansion (uniform contraction) holds, the algorithm has to stay in a small neighbourhood around $\\theta^\\star$. In the context of SGD, it imposes several additional restrictions: the action of previous operators need to ensure that the iterates stay locally close to $\\theta^\\star$ so that uniform contraction still holds in later iterations. It contrasts the claimed stability argument.\n\n   Finally, from an optimization perspective, the cost of *the last* \"backward iterate\" is exactly the same as SGD with reversed sampling order. When the stochastic noise is i.i.d., the backward iteration would give  no gain in terms of convergence rate. This can be seen from the quadratic loss **Example 3.1**.\n\n2. Practicality of the result and numerical experiments \n\n   The authors mentioned in the paper that a backward iteration is very expensive. As mentioned in the previous point, this is because each backward iteration is simply running SGD from scratch and generating a new trajectory. Therefore, given the independence between backward iterates, the experiment seems to suggest that \"if we run SGD with reversed sampling order, then its performance is better\", which is counterintuitive to me.\n\nBesides, I also find the structure of the paper slightly confusing. For example, there are several examples in the paper (**Section 2.1, 3.1, 3.2**), but they are either quadratic or a local approximation to quadratic. There are unnecessary high-level discussions on the extensions, such as **Remark 2.6**.\n\nGiven the restrictions above, I believe this paper should be largely rewritten to improve its presentation and to clarify its contribution.", "audience_yn": "Yes", "audience_explanation": "**Strength**\n\nThis paper is, in general, easy to follow. The perspective of running backward iteration looks interesting, and some experiments show its effect. I believe this paper would be of interest to the community if my concerns are addressed.", "requested_changes": "**Questions**\n\n1. Could you clarify why each backward iteration should be viewed as an iteration, instead of the last iterate of a new optimization trajectory?\n2. Can the theory be extended beyond the smooth convex case? How could you ensure that the local assumptions in the paper are satisfied by the whole iteration trajectory when there is noise? How to choose $h$ in practice?\n3. The current description of backward iteration is highly inefficient since most of the information from previous iterations is discarded. Could you find a way to reuse them?\n4. Could you add an experiment where the initial samples ($\\{B_1, B_2, \\ldots, B_{M}\\}$) have higher variance than later samples and see whether the backward method still performs well?\n\n**Minor issues**\n\n1. Page 2\n\n   \"aiming to reduce the training loss on batch $B_i$\" is not accurate\n\n2. Page 2\n\n   $\\theta$ in the randomly ...  => $\\theta$ is the randomly ...\n\n3. Page 2\n\n   Computation intensive => Computationally intensive\n\n4. Page 4\n\n   Please use $I$ instead of $1$ for identity matrix\n\n5. Page 6\n\n   strictly convex => strongly convex\n\n6. Page 6\n\n   Remark 2.6 looks irrelevant to the topic of the paper\n\n7. Page 8\n\n   a i.i.d. => an i.i.d.\n\n8. Page 9\n\n   $\\lambda_{max}$ =>  $\\lambda_\\max$\n\n9. Page 10, 11,\n\n   Please align the figures in a row\n\n**Requested changes**\n\nSee questions and minor issues", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)