Dual Perspectives on Non-Contrastive Self-Supervised Learning
Authors: Jean Ponce
Basile Terver
Martial Hebert
Michael Arbel
Venue: ICLR 2026 Poster (2026)
Keywords: Deep learning
representation learning
self-supervised learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: A theoretical study of stop gradient and exponential moving average training procedures in self-supervised learning from the dual viewpoints of optimization and dynamical systems

Abstract: The {\em stop gradient} and {\em exponential moving average} iterative procedures are commonly used in non-contrastive approaches to self-supervised learning to avoid representation collapse, with excellent performance in downstream applications in practice. This presentation investigates these procedures from the dual viewpoints of optimization and dynamical systems. We show that, in general, although they {\em do not} optimize the original objective, or {\em any} other smooth function, they {\em do} avoid collapse. Following [Tian et al. 2021], but without any of the extra assumptions used in their proofs, we then show using a dynamical system perspective that, in the linear case, minimizing the original objective function without the use of a stop gradient or exponential moving average {\em always} leads to collapse. Conversely, we characterize explicitly the equilibria of the dynamical systems associated with these two procedures in this linear setting as algebraic varieties in their parameter space, and show that they are, in general, {\em asymptotically stable}. Our theoretical findings are illustrated by empirical experiments with real and synthetic data.

Published as a conference paper at ICLR 2026
DUAL PERSPECTIVES ON NON-CONTRASTIVE SELF-SUPERVISED LEARNING
Jean Ponce
Basile Terver
Ecole normale supérieure/PSI.
Meta FAIR
New York University
INRIA Paris
jean.ponceeen fr
baasileterv@meta.ccom
Martial Hebert
Michael Arbel
Gemegie-Mellowe University
Univ. Grenoble Alpes. Inria
ma rt ial. hebert @cs cmu edu
CNRS, Grenoble INP, LJK
mi chae arbe iinria... fr
ABSTRACT
The stop gradient and exponential moving average iterative procedures are com- monly used in non-contrastive approaches self-supervised learning to avoid rep- resentation collapse, with excellent performance in downstream applications practice. This presentation investigates these procedures from the dual viewpoints of optimization and dynamical systems. We show that, in general, although they do not optimize the original objective, any other smooth function, they do avoid collapse. Following Tian et al. (202T). but without any of the extra assumptions used in their proofs, we then show using a dynamical system perspective that, in the linear case, minimizing the original objective function without the use of a stop gradient or exponential moving average always leads to collapse. Conversely, we characterize explicitly the equilibria of the dynamical systems associated with these two procedures in this linear setting as algebraic varieties in their parameter space, and show that they are, in general, asymptotically stable. Our theoretical findings are illustrated by empirical experiments with real and synthetic data.
INTRODUCTION
Self-supervised learning (or SSL. is an approach to representation learning that exploits the inter- nal consistency of training data without requiring expensive annotations. It has proven to be an effective alternative to traditional supervised technology, for applications in natural language pro-
cessing (Mikolov et al. 2013] Vaswani et al. 2017). image analysis (Chen et al., 2020 Grill et al.
Caron et al., 2021] Chen and He [02 Radford et al., 2021 Zbontar et al., [202]] Bardes 2022) and video understanding (Bardes al. 2024). Early SSL approaches, e.g., (Mikolov et al.
2020
al.
2013 Chen et al. 2020; He et al., 2000) Radford et al. 2021). were contrastive: models are learned from training pairs that can be either negative, when one data point is representative of the target population and the other one is not, or positive, when both data points are representative of the target. The training consists in pushing negative pairs apart while pulling positive ones together. Non-contrastive approaches to SSL have emerged as powerful alternative often outperforming
contrastive ones empirically (Grill et al., 2020 Chen and He, 202T Bardes al.) 2024; Assran et al. 2023, Bardes et al., 2022, Caron et al., 2021). These techniques compute representations of different views of the same data learn predict one from another, thus avoiding the need for mining negative data. They are, however, susceptible to representational collapse where constant embedding is learned (LeCun) (022)).
Preventing representation collapse in non-contrastive approaches has thus become key focus in SSL, leading to two principal strategies: feature decorrelation and enforcing asymmetry between the two views. The first strategy addresses representational collapse by explicitly enforcing decor- relation among the learned features. For instance, Bardes et al. (2022) introduce a regularizer Ω designed to avoid collapse by keeping the variance of the codes of the views of samples above

---

Published as a conference paper at ICLR 2026
fixed threshold while encouraging the codes associated with the same sample to be similar. More recently, Sansone et al. (2025) proposed an auxiliary classification task with randomly assigned la- bels, providing theoretical guarantees against collapse. Despite their conceptual simplicity, feature- decorrelation methods are often empirically outperformed by techniques that introduce asymmetries teacher/student between the views during training to avoid collapse. Specifically, these rely architecture, where the student computes source view as the composition of encoder and predictor networks and aims predict a targer view obtained using the teacher network. The latter is either frozen copy (through a stop gradient operation or SG) or a delayed version (through an exponential moving average or EMA) of the student encoder (Oquab et al. 2024 Bardes et al. [024] |Assra et al. 2023 Grill et al. 2020; Chen and He 2021). SG and EMA have shown strong empirical performance and remain standard components of state-of-the-ar SSL models (Assran et al. 2023
Oquab et al. 2024).
Problem statement. Despite the empirical success of SG and EMA, there is no obvious link be- tween these methods and the optimization of a well-defined objective function. This motivates gaining a theoretical understanding of their behavior, including: (a) Do SG and EMA solve an op- timization problem and, if they do, which one? (b) Do they converge and, when and if they do, they guaranteed to avoid collapse? (c) Seen as dynamical systems, are their stationary points, any, stable, so there is no risk of drifting from them to some trivial solution? These the problems we address in the rest of this presentation, from dual perspectives: an optimization perspective for (a) and (b), and dynamical system one for (c), following the work of (Littwin al. 2024 Tian al., 2021 Wang et al. (2021) in the linear case.
Main contributions.
(1) We prove with Proposition 3.1 that, in general | neither the SG algorithm nor its EMA counterpart minimizes the objective they are derived from and that, if they converge, they both avoid collapse (Figure D..
(2) In the case where the loss the squared Euclidean distance, we then prove with Proposi- tion 1o2 the conjecture given in (Grill et al. (2200) that the SG and EMA algorithms do not optimize any well defined function.
(3) We confirm (1) empirically (Section 3.3 on an action classification task from video data, further finding that the SG and EMA algorithms do not appear to converge, although their downstream performance increases momentarily in training.
Following Littwin et al. (2024); Tian et al. (202 system perspective in the case where the encoder and predictor are both linear operators.
;; Wang et al. 2022), we then switch to dynamical
(4) We characterize in Proposition 4.5 and Corollary 4.6 the equilibria of the dynamical sys- tems associated with the SG and EMA algorithms as a finite set of algebraic varieties.
(5) We prove with Proposition 4. that these equilibria are, in general, asymptoticully sta- ble dAmol'd 1992). In particular, when started close to them, the two procedures guaranteed to converge there and stay there.
(6) We run some simulations in the simplified setting where the input space is scalar (m =
1) and show that the two algorithms converge in general in these experiments, although possibly to trivial minima.
All proofs are relegated to the appendix for conciseness. In the linear case, these proofs leverage the equations' structure (Petersen and Pedersen| 2012) to avoid cumbersome tensor manipulations. On minor note, this notably allows us to rederive, for completeness, several results about the dynamics of the SG and EMA algorithms (Lemmas without assumptions from their original proofs, such as that the two views of the data be drawn from the same distribution conditioned on the data, or that the cigenvalues of certain PSD matrices be bounded away from zero, whose validity is difficult to guarantee in practice.
4.44 already known from (Tian al., 2021), but
Whenever we state that some property holds in general, this means that, although they may not hold for certain data satisfying specific equations, they do hold, in practice. for all generic data, in the standard mathematical sense, following the common notion of genericity in dynamical systems and algebraic geometry, e.g., Hirsch al. 2013).
2

---

Published as a conference paper at ICLR 2026
Figure 1: A (toy) illustration of the optimization landscape for the objective function E(0, v)) of Eq. (T) Here the global minimum of E0, 1)) (shown as negative instead of zero for readability) associated with col- lapse of the training process; B is nontrivial local minimum one may reach using an appropriate regularization to avoid collapse; and A is limit point of the stop gradient (SG) training procedure associated with parameters o" and 1= convergence. In general, is not minimum of E and thus does not correspond collapse of the training process,
minimum with respect to & E(O", v)) See text for details.
1.1 RELATED WORK
Several works have explored why SSI. methods learn effective representations, with feature- decorrelation methods being specifically investigated in (Balestriero and LeCun] 2022, Weng et al.,
2022 Ziyin et al. 2023; Jing et al.) 2022). Balestriero and LeCun (2022) attempt to unify VI- CReg (Bardes al. 2022 SimCLR (Chen et al. 2020) and Barlow Twins (Zbontar et al. 2021) I in
a single framework using global and local spectral embedding methods. [Weng et al. (2022) investi-
gate how whitening- baattnnigg-bseed losses avoid collapse. Ziyin et al. (2023) develop analytically tractable theory of SSL loss landscapes for both contrastive and feature cisture-decorrelation methods, analyzing factors that affect SSL robustness data imbalance.
Experimental studies of asymmetry-based methods (Tao et al. 2021 1 Zhang et al. 2022 Liu et al, 2022) have investigated their properties. Notably, UniGrad TTo al.) 2021) systematically com- pares contrastive and non-contrastive SSL approaches, concluding that a momentum encoder is key improved performance. Zhang al. (2022) use gradient analysis of 12-normalized representations to study why SimSiam avoids collapse. Liu et al. (2022) experimentally show that asymmetry- based methods like BYOL and SimSiam implicitly enforce feature decorrelation, linking them methods such Barlow Twins and VICReg.
Theoretical investigations of asymmetry based methods have been proposed, notably by Tian et al. (021), showing that, under isotropic data and specific optimization trajectory assumptions, the predictor and stop-gradient are essential to prevent collapse
linear BYOL and SimSiam. This study has inspired algorithms where the predictor is an explicit function of the encoder, with proven collapse-avoiding properties (Wang et al. 2021 Jing et al. 2022; Halvagal et al.) 2023) Tang et al. 2023). Wen and Li (022)) investigate the encoder's role in avoiding collapse using two-layo neural network, albeit under a simplified data tata-generating process. Finally, [ittwin et al.] (2024) study the effect of depth learned representations in deep linear models, showing that Ambed-dmg Predictive Architectures (or JEPA) models prioritize "influential" features (features which are most informative in prediction), with derivations relying on diagonal covariance matrices and orthogonal initialization.
2 PROBLEM SETTING
Given a parametric encoder fo R + R" and a parametric predictor R" + R" with parameters R and & in RV, it is possible (LeCun) 2022, Bardes et al.) (022) to learn & and z from data embedded in R" without outside supervision by minimizing with respect to these parameters the objective function
(I) Here, and y different views of some data point (e.g.. different crops of the same image), Ez,y, is the mean of the function over the (unknown) distribution of the views conditioned on the corresponding data, approximated in practice by
E(O, v)) Ex E00 v. x,,), where E(O, v. y) = llgv o fe(x), fe( 200,
finite number of data samples,
3

---

Published as a conference paper at ICLR 2026
1 X R" R+ is some loss and Q R x R R- is some regularizer. The corresponding architecture is that of a Siamese network (Bromley et al., 1994) whose branches correspond to the encoders of the two views compared, with shared parameters, while the predictor sits top of the first branch. In this setting, E(O, w) can be minimized with respect to these parameters by using, for example, stochastic gradient descent. With proper learning rates, the training procedure will converge to some critical point of E' where both gradients are zero (Figure D. A difficulty, however, is how to prevent from collapsing by converging the degenerate zero global minimum corresponding to fo being constant and 9v. being the identity, or fo being zero and Jv being function such that (0) 0. To address this difficulty, BYOL (Grill et al. (020) and SimSiam (Chen) and He 2021 propose to use exponential moving averuge and slop grudient training procedures, as defined in the rest of this section, as alternative minimization procedures. To properly define these procedures, let us introduce an objective function with an additional argument in IRP:
F(0, <) F (0, v, E, v), where F(0, v. E, x. y) lggv fz) + 2(0,, v), (2)
and consider instead the exponential moving average (EMA) procedure (Grill et al. 2020).
EMA algorithm: Initialize Gg to and Eo to some values and t to 1, then repeat until convergence or you run out of patience:
(a) 0, - 0t-1 1e V F(1-1 11 &c-1,
(b) v/ - 1h 14 V nt 11 11
(c) &, + a1 + (1 a1)
(d) t t+1
This the procedure used to train BYOL in (Grill et al. 2020) and V-JEPA in (Bardes et al, 2024 The corresponding architecture is no longer a true Siamese network because the encoders in its two branches have different parameters, so fe and fe are sometimes respectively called the online (or student) and target (or teacher) networks. An alternative is to consider the stop gradient procedure (Chen and He 2021) used to train SimSiam, which is a true Siamese architecture with identical encoders in its two branches but uses &&F proxy for E when updating 8.
SG algorithm: Initialize Ou and th some values and to 1, then repeat until convergence you run out of patience:
(a) 8, + 01-1 14 V i 1h
1/ 1/-11 U1 (0,-,,,,-,,,0-11
(c) t+1
Note that E(0. (b) F0, vi, 0) for any values of 0 and v but it is priori unclear whether any limit point the SG procedure related to the critical points of E. Note also that the EMA algorithm reduces to the SG one when a; = 0 and &o 0p. Indeed, Chen and He (2021) refer to SimSiam as "BYOL without the momentum encoder", but in practice a₁ is taken very close to 1, e.g. ag varies between 0.996 and in BYOL (Grill et al. 2020
3 AN OPTIMIZATION PERSPECTIVE
We assume from now on for simplicity that 04 is constant with # or converges to such constant as t tends to infinity. Under this assumption, if and when the EMA and SG procedures converge, we have by continuity 0 at limit point because of step (c), with the same gradient values in 0 and 3 as in SG. The dynamics may be different, and lead to different limit points, but these will obey the same zero gradient conditions at the equilibria of the corresponding dynamical systems.
2BYOL Grill et al. 2020) assumes that what we call an encoder is proper encoder followed by projection operator and that the loss acts normalized versions of its inputs. Both BYOL and SimSiam (Chen and He 2021) make the loss symmetric by having each view predict the other. This is subsumed by the framework presented here and does not change the conclusions of our analysis
4

---

Published as a conference paper at ICLR 2026
3.1 EMA AND SG OBJECTIVES
The motivation for the EMA algorithm in BYOL (Grill et al. 2020 Sec. 3.1, Eq. (2)) is to minimize over 6, & and & the mean over pairs of views z and y of the data the squared Euclidean distance between the prediction g ff (z) obtained from the representation of z and the targer encoding fz(y) of y, where & the moving average defined recursively by step (c) of EMA. Note that, although & is well defined at each i, an explicit definition of is missing, except in the limit case if and when
EMA converges. It is thus a priori unclear whether its iterations minimize a well-defined function. Similarly, the motivation for the SG training procedure in (Chen and He 2021 Sec. 3, Eq. (1)) is to minimize the mean squared Euclidean distance between the prediction gv fo(x) obtained from æ and the target fo(y), using Vo as a proxy for the true gradient Vo E of the corresponding objective function. It is therefore again priori unclear whether it minimizes well-defined function.
3.2 SG AND EMA ALGORITHMS VS OPTIMIZATION PROBLEMS
It is important for fairness to clearly state that none of the papers we are aware of that use the SG or EMA procedure claims to actually minimize such functions in practice. In fact, the BYOL authors write (Grill et al. 2020): "More generally, we hypothesize that there is no loss Lø.g such that B BYOL's dynamics is a gradient descent on L jointly over 0, g." One of our main contributions is actually prove this conjecture (see Proposition 3.2 below). Let us first state simple but important result (see the appendix for its proof).
Proposition 3.1. The SG and EMA algorithuss do not, in general, minimize the original objective function E of Eq. {7]. If and when they converge, the corresponding solution is, in general, not degenerate one corresponding to zero global minimum of that function.
A harder question to answer is whether these algorithms optimize any objective function. Let us assume from now on for simplicity that I is the (half) squared Euclidean distance and (2 v) X[100|2 + 1/2 With this choice, we have Val(u, v) = u - v, Vos2 = 0, v = & and
V F(F 1..F00,00000 0,0,)) Jou(O, vi, x) [u( x. v(0, y)] + X8, = Jou(O, x) (1, x)) v(0, v) + Anh.
(3)
V (0,
where u(0, x)) Dv fo(x) and v(0,y) Se(y)- We wish understand whether the vector fields Eyy Vg F(0, the x, v)] and Re (V (10, 4/, y)) are the gradients of some well defined scalar function which the SG algorithm (and its EMA counterpart) would presumably minimize. The answer is negative:
Proposition 3.2. Under the (mild) assumption that fa and gu are smooth neural networks whose last layer is linear and that they are not identically i.e. there exists Go, th, No and zq so that fee (aro) + 0 and 9 (Zo) + O, then the vector fields Ex.y ( v. y)l and Ez.g V v F(6, tb x. v)l are not, in general, the gradient fields of any smooth function.
Proof sketch. The full proof, given in the appendix, follows from Eq. (3) and Schwarz's 's integrability theorem, according to which, necessary condition for these vector fields to be the gradient field of a smooth scalar function is that their second-order cross derivatives be the transposes of each other. By expressing these cross derivatives we find that their difference (taking into account the transposition operation) does not vanish in a generic sense. To establish genericity, we show that there exists arbitrarily small perturbations to the data distribution for which the corresponding cross derivative difference cannot be identically zero
L
3.3 EXPERIMENTS WITH REAL DATA
We investigate in this section three fundamental questions about the SG and EMA algorithms: (1) Although they do not, in theory, minimize the original objective E(0, v), do they minimize it in practice? (2) Do they converge, and in particular do 0, 0,- and vù - both tend toward zero as the number training steps increases? (3) Does the classification accuracy increase with training time? We address these three questions in realistic setting with experiments a video classifica- tion task using the code of Bardes et al. 2024), on the Kinetics710 10 and SSv2 benchmarks (Goyal et al. 2017 Smaira al. 2020).
5

---

Published as a conference paper at ICLR 2026
EA ml
25.0
213
20.
375
386
123
100
51.C
LO0K
353
200
53.00
100.0K
256.6
200
258.6
200
SOOK
226.K
250.K
253.0K
3003
67
44
9.5
30
02
$2.0K 200.KK J.SLEC 200.0K 250.K 338.6
50.0K 200.0
153EK 200.KK
250.06
300
Figure 2: Evolution over 300,000 iterations of (left) the E objective (plus F for EMA) without the squared Euclidean norm regularizer, (middle) the norms of the parameter increments 0,- 0--10. 1, -hh-1 and - and (right) the classification accuracy on the K400 and SSv2 benchmarks for SG (top) and EMA (bottom). All curves here are smoothed using a moving average, and the squared Euclidean norm regularizer is ignored to emphasize the main loss.
Experimental setting. To provide a realistic setting, we address the challenging problem of self- supervised learning for action classification in videos, task for which VICReg (Bardes et al.) 2022) and regularization-
methods such as SimCLR (Chen et al., 2020) and Barlow Twins (Zbontar 2021 have not proven successful far. Indeed, the state-of-the-art V-JEPA model of Bardes et al.
iet al.
2024) uses the EMA algorithm for training, and we use its code, kindly provided by its authors, in our experiments, with & loss for VIT-S encoder and ViT-T predictor (Dosovitskiy al., and SSv2 training datasets (Goyal et al.) [017]] Smaira 2020) for learning the representation. We also learn in supervised manner an attentive pooling classifier as in (Bardes et al. (024), with the train splits of each dataset. We report the top-1 accuracy on the classification task for each dataset on its validation splits.
2021)
We use the unions of the Kinetics710 10
al.
We slightly modify the training setup compared to (Bardes et al. (2024) to make it simpler and faster. We run the SG and EMA algorithms on both datasets our experiments, with 1000 training epochs (300 000 iterations), instead of 300 epochs in V-JEPA. We also restrict the train set to Kinetics7 10 and SSv2, discarding fowto100M Miech al. (201)). We use a ViT-Base encoder instead of a ViT-L encoder and reduce the batch size to 1024. Instead of the cosine annealing with warmup schedulers, we fix the learning rate to 0.0001 and the weight decay to 0.1. Following (Bardes et al.) 2024), the variable a increases from 0.998 to 0.9996 across iterations. As expected, with these simplifications, the performance of the trained models on the downstream K400 and on SSv2 video classification tasks (Figure 2 bottom) is lower than the accuracies of 72.9 and 67.4 reported in (Bardes et al., 2024) for the V-JEPA ViT-L model trained on K710 and SSv2 for a total of 900K samples seen during training. Yet we believe that this simple setting is sufficient to run realistic experiments and give preliminary yet meaningful answers to the questions addressed in this section.
Results and conclusions. Figure (top, left) shows the evolution of E(O,, thxa) for the SG procedure as function of the number of iterations. As expected, the algorithm does not minimize the function, its smallest value being reached early in the iterations. Figure 2 (bottom, left) shows the evolution of F(O,, th,, {.) and ElO,, w)) for the EMA procedure, and similar conclusions can be reached. As shown in Figure 2 (top, middle), 0 and 1/ do not converge in our experiments either since the norms of 92 8 1 and th w do not tend to zero. As shown in Figure (bottom, middle), a similar conclusion can be reached for 0, se and & and the EMA algorithm. Finally, classification accuracy does increase initially for both the SG and EMA algorithms before decreasing late in the process for SG, and reaching plateau for EMA (Figure 2 right). confirming the commonly accepted fact that they learn something, although it is still unclear exactly what they learn. In this particular setting, EMA gives better classification results than SG. General conclusions about other downstream tasks should not be drawn from this of course.

---

Published as a conference paper at ICLR 2026
4 A DYNAMICAL SYSTEM PERSPECTIVE IN THE LINEAR CASE
Let us now consider, following (Littwin et al. 2024 Tian et al. 2021 Wang et al. 2021) 1. the linear case where fa( Ar, fe = Cy. and (z) Bz, where E R A and C are n m matrices, with n > m, 2 E IR", B is an n x n matrix, and the vectors 0, 5 and sb store row after row the coefficients of A, and B. We will from now on write [xx" for Ez [xx²) and [y="] for Ez.y [yz"]] Recall that we consider, here, the (half) squared Euclidean distance for and (00 1b) = >(110] + 1/2
4.1
THE SG AND EMA ALGORITIIMS AS DYNAMICAL SYSTEMS
An algorithm such as the SG and EMA procedures can also be viewed as a discrete dynamical system. In this case, the dynamics are driven by the gradients of the function F with respect to the parameters being optimized, i.e., the matrices A, B and C. Note: We assume from now on that on is constant * 1. For completeness, we rederive in the rest of this section several results about the dynamics of the SG and EMA algorithms (Lemmas 4. to 4.4) already known from (Tian al.] 2021]. As noted earlier, however, required by the original ones, for example that the two views of the data be drawn from the same distribution conditioned the data, or that the eigenvalues of certain PSD matrices be bounded away from zero, whose validity is difficult to guarantee in practice.
proofs, given in appendix. do not rely the assumptions
Lemma 4.1. The discrete dynamics of the EMA algorithm in the linear case are given by
{
A = At-1 - - (B R( 1 Bt-1, Ct-1) + AA(-1), B, = Bi- - (v([(A, Br-1, C-1)) ABI-I),
(4)
C. = aCr-1 ( A)A,,
where R(A, C) def BA(xxT) Clyx7'
The dynamics for the SG algorithm are obtained by taking C = A in Eq. (4) and not using the C update. The stationary points of SG must satisfy the p q equations in p + q unknowns defined by P = 0 and Q = 0, with = nm and g = n², whose solutions include A = 0 and B 0. In particular, unlike the nonlinear case, it is priori possible in the linear setting for limit point of the SG or EMA algorithm to be the degenerate global minimum associated with = and B = o The following lemma follows easily from Eq. (4) and is reminiscent of prior results on gradient descent for deep linear networks (Baldi and Hornik 1989).
Lemma 4.2. When > 0, the limit points of the SG algorithm, they exist, satisfy B B AA".
We switch now to a continuous dynamical system perspective, following (Littwin et al. 2024 Tian
et al. 2021 Wang et al. 20221) to simplify the analysis. The continuous version of Eq. 4 is
{
À R(A, B, B,C) + AA), B (R(A, B, AT +AB),
(5)
C (1 () C).
where R(A, B, C) is the same as in Lemma 4.1 At a limit point, we have in addition A = 0.
B = 0 and multiplying the first equation on the right by AT and the second one on the left by BT" then subtracting the results yields B² = AA". Given Eq. [5) we can now rederive two results from (Tian et al. (021).
Lemma 4.3. Given the dynamical system associated with the gradient flow the original objective function of Eq. (7) the matrix A always converges to zero.
This is Theorem 2 in Tian al. 2021) but without the assumptions made in the original proof. This result implies that there is no other limit point than the global trivial minima where A = O and B can assume any value. |Tan et al. (2021) (and (Littwin et al., 2024 Wang al.) 202T) interpret Eq. (5) as defining a gradient flow. Note that it might be more appropriate to see these equations as defining
Thhe degenerate solution where fo is nonzero constant and go is the identity does not occur in the linear case since Ar constant implies A
7

---

Published as a conference paper at ICLR 2026
general flow instead, that is, first-order ordinary equation whose integral curves the output of the algorithm since we have shown that their corresponding vector fields are not the gradients of well defined function by Proposition 3.2 We can now state in our setting Theorem from (Tian
et al.
2021) as follows.
Lemma 4.4. Under the SG or EMA dynamics, the difference of the two matrices B and AAT lends zero as t. tends to infinity.
4.2 CHARACTERIZING THE EQUILIBRIA OF THE SG AND EMA ALGORITHMS
Proposition 4.5. Assuming that the matrix A has maximal rank algorithm, these equilibri are the (A, B) pairs such that
equilibria of the SG
([xx") (id)(S(xxx) Ad) [ay"")((yx") where S= SAATA A.
(6)
and
B Aly="1A" where W A[Tx"|A² + XId.
(7)
Assuming again that the matrix A has maximal rank m n the equilibria of the EMA algorithm, these equilibria are the (A, B, C) triples where A verifies Eq. (6). B is given by Eq. 7 and C = A. In both cases, the (A, B) pairs associated with equilibria also verify B AAT
Equation (6) is a system of m(m + 1)/2 quadratic equations the m(m + 1)/2 independent entries of the symmetric matrix S, which is positive definite since A is assumed to have maximal rank. Such a system admits in general at most 2mm(m-1)/2 solutions, and we obtain the following result.
Corollary 4.6. Let K denote the number of distinct real solutions Sk (k = l...., of Eq. ( such that Sy is positive definite and, for k = I....,,, K. let VS, denote the unique positive definite square root of Sy. The equilibria of the SG algorithm can be decomposed into K sub-varieties of Raxm x I mmxmm formed by pairs (A, B) such that A belongs to
A {A (AERuXM, A S = (UV5K, where U E RR and U Id),
(8)
and satisfies Eq. (7). The equilibria (A. B, of the EMA algorithm can be characterized similar way to A in At. B satisfying Eq. 0 and C A.
One might ask whether such maximal rank solutions exist in the first place. The following propo- sition, whose proof is provided in the appendix, leverages Brouwer's fixed point theorem and the implicit function theorem to establish that, in general, they do exist.
Proposition 4.7. Let Po be some data distribution. For any positive there always exists a per- turbed data distribution (āt, 9) P. that c-close (e.g., in the Wasseesseei distance sense), so that there exist equilibria of the SG and EMA algorithms with A having maximal rank m for X small enough.
Using classical results on the dynamics of differential equations will now allow us prove our last result. Let us first define the stable equilibria of such a dynamical system (Arool's 'd 1992).
Definition 4.8. An equilibrium for the dynamical system á = v(2), where is smooth field over Rd is a point where v(e) O. An equilibrium e is called (Lyapunov) stable when solutions of the differential equation with initial values close to converge uniformly nearby point. A stable equilibrium said to be asymptotically stable when any solution started close to e converges
Theorem 4.9. Arrol d 1992 Chap. 3) Consider a dynamical system = v(x)) whose dynamics can be approximated by the linear operator J: v(x) = Jr + O [[] A sufficient condition for an equilibrium be asymptotically stable that all eigenvalues of J have a negative real part.
Armed with this classical result, prove in the appendix the following proposition.
Proposition 4.10. The equilibria of the dynamical system associated with the SG the EMA pro- cedure, if any, are, in general, asymptotically stable.
Note that Proposition 4.10 does not imply that a nontrivial equilibrium exists or that the dynamics converge to such equilibrium. It is, however, valid even in the case where 1.
8

---

Published as a conference paper at ICLR 2026
Figure 3: Integration paths for the SG (top) and EMA (bottom) procedures. In the top (resp. bottom) part of the figure, the left diagram shows sample path for the (resp. 6) coefficients of B, (resp. a, B), the central one shows a path where all parameters converge toward 0, and the right diagram shows trajectories the (resp. a, c) plane, starting from random locations. One the trajectories has origin as a limit point.
4.3 EXPERIMENTS WITH SYNTHETIC DATA
For these experiments, consider the case of scalar input space (m 1) since now vector, which simplifies visualizations and numerical simulations, and denote by and T the scalars [rx"") and [g To emphasize that these quantities are vectors, let use a for A and for C. In this simplified setting, equilibria are characterized by the following proposition.
Proposition 4.11. In the case m 1, a necessary and sufficient condition for the existence of -pp 2 When this condition
nonzero equilibria of the SG and EMA algorithms is that A satisfied, these equilibria are the pairs (a, B) such that lies either of the hyperspheres S and S2 of Rh centered at the origin with radii B verifies Eq. (7). The equilibria associated with S2 are asymptotically stable, but those associated with Sn saddle points. The equilibria (a, B,c) of the EMA algorithm and their stability can be characterized in a similar fashion, with the additional condition that c a.
([T) VA)/2p and T2 ([T /A)/2p. and
Proposition shows that the case m = both illustrative of the general case because the two spheres Si and S2 are just the algebraic varieties identified in Proposition 4.5 and Corollary 4.6 and extremely nongeneric since the equilibria associated with S are saddle points, which never happens when m See the proof Proposition 4.11 in the appendix for an explanation of this phenomenon.
Results and conclusions. Figure shows sample trajectories obtained by numerical simulations for 2, so and are points in the plane, and equilibria are located at the origin and on two circles centered the origin with radii rı < r2. We have (arbitrarily) taken p 3. r 2 and 0.1 with T = 300 time steps. For the SG algorithm (Figure 3 top), we show on the left the evolution of a (red and blue lines) and B (in other colors), and the middle a case where all coefficients converge to zero. Fifteen trajectories initialized from random positions and drawn in the plane are shown on the right. Figure (bottom) illustrates the EMA algorithm. We use a; = 0.9 + 0.1t /T in this experiment, so = I at t T. following Grill et al. (2020). Although is not guaranteed to converge to in this case, it has done so in all our trials. This illustrated on the left, where the parameters are shown solid red and blue curves, and the parameters are shown dashed red and blue curves (the other 4 curves correspond to B). The center part of the figure is an example where all parameters converge zero. Sample trajectories starting from various random positions are shown on the right as red curves for a and blue curves for Although we have not been able to prove the convergence of the EMA and SG algorithms so far, Figure 3 typical of observations: In 10,000 trials with parameters drawn at random the two algorithms always converge, 92.8%
Values for and À are respectively drawn uniformly in [0, 3). [-1,1] and [0.01, 0.1). The coordinates of initial points drawn from normal distributions with zero and unit variance.

---

Published as a conference paper at ICLR 2026
O
10
Figure 4: Top: Two trajectories for the SG algorithm plotted in the plane (left) and the evolution of E(a, values along these trajectories (right). The error appears to decrease steadily with time the first case, but not the second one. Bottom: similar diagram for the EMA algorithm. In this case we show in red the curve E(a, B) and in blue the curve Fa, B, c).
limit point the outer circle S2 for EMA (resp. SG), and the rest of the time to the origin (as noted in Section 4.1 contrary to the generic nonlinear case, A and B 0 are equilibria in the linear case). As expected. we have never observed convergence to the saddle points of Sj. As in the nonlinear case, the SG and EMA procedures do not appear to minimize E or F. This is illustrated by Figure 4 where we plot the values of E(a, B) as function of time for some trajectories of the SG and EMA algorithms. Although (a, B) appears to steadily decrease in some cases, definitely does not in other cases.
(resp. 82.0%)
CONCLUSIONS
We have shown that SG and EMA algorithms are not, general, proper optimization procedures: In particular, they do not minimize any well defined objective function. On the other hand, they do not lead in general to collapse when they converge, and they enjoy interesting properties as dynamical systems since in the linear case any nontrivial limit point is asymptotically stable thus will not devolve into a trivial one by longer training. This point is important in practice since the SG and EMA training procedures empirically give good results, shown for example in (Bardes al., 2024 Grill et al. 2020; Chen and He 2021), and appear, in the general nonlinear case, to prevent falling into the degenerate global minima they designed avoid. But then, what is it they really learn in the classical sense of the word? Much work remains to be done.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work explores the theoretical foundations of two widely used methods in self-supervised learn- ing, the stop-gradient and exponential moving average algorithms. Self-supervised representation learning has already had a significant impact in various applications, including natural language processing, computer vision, and robotics. We do not anticipate particular risks of this work, but a deeper understanding of self-supervised learning may allow the development of more performant and scalable algorithms, with potential increased impact on the aforementioned fields, which are not exempt of risks of misuse.
REPRODUCIBILITY STATEMENT
We have included in the main text all the necessary details to understand and reproduce our ex- perimental results, both for the real-world data experiments, in Section [3.3] and for the synthetic data in Section 4.3 For the real-world data experiments, we use the publicly available code of V-JEPA (Bardes 2024), which we slightly modify explained in Section 3.3
ACKNOWLEDGMENTS
This work was supported in part by the French government under management of Agence Nationale de Recherche as part the "France 2030" program, PR[AIJRIE-PSAI projet, reference ANR23- IACL-0008. JP was supported in part by the Louis Vuitton/ENS chair in artificial intelligence, the Institute of Information Communications Technology Planning & Evaluation (IITP) grant funded by the Korean Government (MSIT) (No. RS RS-2024-00457882, National AI Research Lab Project), and Global Distinguished Professorship at the Courant Institute of Mathematical Sciences and the Center for Data Science New York University.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
V.I. d. Ordinary differential equations. Springer, 1992.
M. Assran, Duval, Misra, Bojanowski, P. Vincent, M.G. Rabbat, Y. LeCun, and N. Ballas. Self-supervised learning from images with a joint-emeedding predictive architecture. In CVPR, 2023.
Pierre Baldi and Kurt Hormik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58, 1989.
R. Balestriero and Y. LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In NeurIPS, 2022.
A. Bardes, J. Ponce, and Y. LeCun. VICReg: Variane-iiwariance-ovvvaaann regularization for self-supervised learning. In ICIR, 2022.
A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, LeCun, M. Assran, and N. Ballas. Revis- iting feature prediction for learning visual representations from video. Transactions Machine Learning, 2024.
J. Bromley, Guyon, LeCun, E. Säckinger, and R. Shah. Signature verification using "Siamese" time delay neural network. In Proc. NIPS, 1994.
M. Caron, H. Touvron, I. Mishra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties self-supervised vision transformers. In ICCV, 2021.
T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
X. Chen and K. He. Exploring simple siamese representation learning. In Proc. CVPR, 2021.
A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
R. Goyal, S.E. Kahou, V. Michalski, J. Materzyúska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, Hoppe. C. Thurau, I. Bax, and Memisevic. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
J.-B. Grill, F. Strub, F. Altché, C. Tallec, P.H. Richemond, Buchatskaya, Doersch, B.A. Pires, Z.D. guo, M.G. Azar, B. Piot, K. Kavukcuoglu. R. Munos, and M. Valko. Bootstrap your own latent a new approach to self-supervised learning. In Proc. NeurIPS, 2020.
Manu Srinath Halvagal, Axel Laborieux, and Friedemann Zenke. Implicit variance regularization in non-contrastive SSL. In Thirth-seventh Conference on Neural Information Processing Systems, 2023. URL htw.ltttss..m /ttp:///ee.e.eeeee...o.. net./ fo rum? d= 92 zs1
K. He, H. Fan, Wu, S. Xie, and R. Girschick. Momentum contrast for unsupervised visual representation learning. In Proc. CVPR, 2020.
Morris W Hirsch, Stephen Smale, and Robert I. Devaney. Differential equations, dynamical systems, and an introduction to chaos. Academic press, 2013.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Representations, 2022. URL
n http://www..ee.rmm/eeeeeeeeeeeee Fow.mmmne.....YsSSSEEEE
Y. LeCun. A path towards autonomous machine intelligence. Open Review, Jun 2022.
E. Littwin, O. Saremi, M. Advani, V. Thilak, P. Nakkiran, C. Huang, and J.M. Susskind. How JEPA avoids noisy features: The implicit bias of deep linear self distillation networks. In NeurIPS, 2024.
12

---

Published as a conference paper at ICLR 2026
Kang-Jun Liu, Masanori Suganuma, and Takayuki Okatani. Bridging the gap from asymmetry tricks to decorrelation principles in con-contrastive superviuer learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Pro- cessing Systems, 2022. URL btLhtts:: /openreviee net/ furmm? J2
A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, Laptev, and Sivic. Howtol 100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019.
T. Mikolov, I. Sutskever, K. Chen, G.S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. NIPS, 2013.
M. Oquab, T. Darcet, T. Moutakanni, H.V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, H.Wewss P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G2 Synnaeve, H. Xu, H. Jégou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
K. B. Petersen and M. S. Pedersen. The matrix cookbook, 2012.
A. Radford, J.W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Emanuele Sansone, Tim Lebailly, and Tinne Tuytelaars.
Collapse-pros non-contrastive self- supervised In Forty-second International Conference on Machine Learning. 2025, URL
ew net. forum? iiwammiiiiiii fI 8PX60p
L. Smaira, J. Carreira, E. Noland, E. Clancy, A. Wu, and A. Zisserman. A short note on the kinetics- 700-2020 human action dataset, 2020.
Yunhao Tang, Zhaohan Daniel Guo, Pierre Harvey Richemond, Bernardo Ávila Pires, Yash Chan- dak, Rémi Munos, Mark Rowland, Mohammad Gheshlaghi Azar, Charline Le Lan, Clare Lyle, András György, Shantanu Thakoor, Will Dabney, Bilal Piot, Daniele Calandriello, and Michal Valko. Understanding self- uiffpprdditive learning for reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org. 2023.
Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self-supervised learning via a unified gradient framework. 2022 IEEE/CVF Conference Computer Vision and Pattern Recogmition (CVPR), pages 14411- 14420, 2021. hatt::
semant wecholsccllcccc htt.a//ooo...ao.ooooooooooooooa.
Y. Tian, X. Chen, and Ganguli. Understanding self-supervised learning dynamics without con- trastive pairs. In Proc. ICML, 2021.
A. Vaswani, N. Shazeer, N. Parmar, J. Uzskoreit, L. Jones, A.N. GOmez, and L. Kaiser. Attention is all you need. In Proc. NeurIPS, 2017.
X. Wang, X. Chen, S.S. Du, and Y. Tian. Towards demystifying representation learning with non- contrastive self-supervision, 2021.
Zixin Wen and Yuanzhi Li. The mechanism of prediction head in non-contrastive self-supervised learning. In NewrIPS, 2022.
Xi Weng, Lei Huang. Lei Zhao, Rao Muhammad Anwer, Salman Khan, and Fahad Shahbaz Khan. An investigation into whitening loss for self-supervised learning. In Proceedings of the 36th International Conference on Neural Information P Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 1088.
J. Zbontar, I.. Jing, Misra, Y. LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X. Pham, Chang D. Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? unified understanding with self-supervised contrastive learning. In International Conference on Learning Representa- fions, 2022. [ hhttps://openreviieett www./eeenurrnee wttp:r/wmniiddowwowwwwwws
13

---

Published as a conference paper at ICLR 2026
Liu Ziyin, Ekdeep Singh Lubana, Masahito Ueda, and Hidenori Tanaka. What shapes the loss landscape of self supervised learning? In The Eleventh International Conference on Learning Representations, 2023. URL https:: httppp/wwwweeeeeeeeee///eeaeaaaa
14

---

Published as a conference paper at ICLR 2026
6 PROOFS
We will use, when convenient, the notation f(0,z) 2) for fo(x) and g 2) for ge(2). For given matrix A we denote by vec(A) the vector collecting all the components of the matrix A.
PROOF OF PROPOSITION 3.1
As noted before, E )b) F(0, tb, 8) for any values of & and th. However, there is no a priori reason for a limit point of SG (if and when it converges) to be critical point of E. Let write
F(O, v., E, y) = U[((,, v, æ), v(5. y)] + 24),
where u(0,, di, x) - bv o fa(x) and v(E. v) - Se(w), and denote by Jou(O, v x) and J (0, th, x) the n p and n x Jacobians of u(, v. v.x) Iv fa( with respect and qb. We also introduce 1(0, v, g, x, y) = lfu(0, sb x), v(5, We have
v F = Jau(O, 1h
+ Vo2(0, V F Jon(O, v. x) V,0,, 10, v, + (000 v)
(10)
and the means of these two gradients (also gradients of F) should vanish at any limit point of SG. But
V.E10,6) = VoF(0, 0,0) + (Jo [ ")² v.'V V.d[u(O, 1 x), v(0, a, v)ll. V(,00))) F0,, 4, 0),
(11)
where Jef(0, Jev(0, y) is the n x Jacobian of f with respect to 0. There is a priori no reason why the second term of the gradient of with respect to 6, which depends on the data, should be zero at critical point of F and thus at limit point of SG (and thus of EMA) if such point exists.
To establish this, we consider a counter example the linear case where fo(x) = Ax and De = B matrix and B & f vec( Anticipating the result of Proposition 4.7 we know that we can always find an ar- bitrarily small perturbation the data distribution, call it P. so that there exists a matrix A of maximal rank and a matrix B*, that are critical points of the dynamics. Set 0" vec(A*) and di" vee( B*). Hence, fe- (z) and g (2) cannot be degenerate solutions of the EMA or SG dy- namics. Now, wish to further establish that Vo Fip (0*, a/*) is not 0 in generic sense. Let us first express VoEp, (0, v) == E( V tb, T, 7)] in terms of the matrices A and B and v Fe, (8*, "."""""" 0") = PP. F
where A is
matrix with > m. Here 0 vec(A) and
(0, v., 0, 2. f)]:
Vo Br, (0", x²) = Vr. (0" 1/* 0*) A [00 1
B* A A
= (vū | B A i
Note that V Fr. (0, 10 9) and Fr. (0, v/ 0) are independent of the matrix [00 by virtue of the expression:
Vole. (0. = B (BA AljJ DD + AA
V V.lr.. (0, 0, = BA JAT Al JAT + AB.
Therefore, the critical points (A*, B independent of the matrix [Vū We treat two cases: either EoEz, (8*, vb)) + 0, and there is nothing to prove, or vg (#", 10*) = this case, we
will construct second perturbed data distribution Q from IP, follows:
(12) where (ā. y) are samples from P. and is standard centered gaussian independent of ē and g. is easy to check that (e(xx)]] [ and that [z'(z")
=' y = ÿ + et2,
] so that the critical points A and + cld. follows, under such perturbed
B' remain unchanged. On the other hand (1())) distribution Q. that:
V (0*, (b)) = VF.. (0',"", 0*) + A AV B² (1((
= A A" 1 B" A A - +
= =AA- # 0.
Hence, we have established that can always find small perturbation Q. to the data distribution so that the equilibrium not degenerate and does not correspond to a critical point of the objective Eeee Q. [F(0, (F(0,, v1 x', v-)]
15

---

Published as a conference paper at ICLR 2026
PROOF OF PROPOSITION 3.2
Define for simplicity, the following vector fields:
P(O, v. V F (0, Q v. - F 1/, 0, x. x.
(13)
(14)
Let D be some probability distribution over data (α,g) and define PO == (aa))>> [P(0, 3,,,, x)] and Q2 .) == (z [Q v. z, y)]. According to Schwarz's inte- grability theorem, a necessary condition for Po and Qr to be the gradient field of a smooth sealar function is that their second-order cross derivatives be the transposes of each other. We have
ap
(0, v. x, y) Ha vs x))((0, b, - v00 v) Jou(O, 1b x²² J v,
Oib iQ
00 (0, V. x. v) = Hw, vi Iu) 15 - v00, v)1 + (0, v; x) [Jou(O, d, - Jnv(0, v)
(15)
where Jov(O,y) 3) denotes the n x p Jacobian of with respect to 0, which is equal to the Jacobian (0 y), and Ha. (resp. Hy.su) is the p x q x n (resp. q x p x n) tensor associated with the second partial derivative of u with respect to I and e (resp. & and 6). The products of these tensors by the vector u - yield p x q and X matrices that are transposes of each other, and we obtain
1b) def PPr
aQu
Oub (0, v) (0,
00 (0, v² = Ez.9 (t..nJoo[00 (o, (ogo)))TTTggl... 1(0,
(16)
A(9.E.).)))
We wish to establish that Á + 0 in generic sense over all possible data distributions. This means that even if for some data distribution Dy we have AD. O, we can always construct a perturbed data distribution D. that gets arbitrarily close to Dy approaches O and for which ÅD, * 0. To this end, let's consider a case for which Ån, = for some Do. otherwise there nothing to prove.
By Proposition stated stated below, we know that the integrand A appearing in the expression of Åp, is a function that not identically Hence, there exists 0, &, Fo and yo so that || E, Bo).../)lll Bo)ll || 2 2n for some positive 7). Since both function f and 9 are assumed to be continuously differentiable, we know that A must be a continuous map. Hence, there exists a positive radius r > small enough that |0, &, 2. v/ A0,, &, zu, yo)llæ I 17 for any (z. y) in ball B centered around (Io, yo) of radius r. Consider uniform distribution over such ball. It is then easy to see that (0, E, 2, y)] A0, E. Zo. ya) I 17, so that:
[E (x.)))/ (0,,
n H (1, &, ro, yoll (=(()))> (A, E, x. v)] A(0, &, zu. vo)ll..
(17)
Now, define the following perturbed distribution by convex mixture of DD, and g:
BB = e)Ilo + Eq.
(18)
We will show that || ll 2 en.. To see this, we first notice that:
Åp. (1 Ao. +cB(ex)) (0,, E, v))]
(19)
Hence, by direct application of the triangle inequality, we have that:
on S c (1)A(0, To. yolllr [E< ((B, x) A0,, E, No, 30)llr) S |ÂD, ll.
(20)
We have, therefore shown an arbitrarily small perturbation the data distribution suffices to ensure that Ay. * O, which in turn implies that that a the corresponding vector fields Pb, and Qn, are not the gradient of any scalar function.
The interested reader may wonder how the integrability condition of Proposition 3.2 translates in the linear case. As shown below, necessary condition for P and Q to be the gradient field of smooth function for linear encoders and predictors is that A|ry"] = 0. This condition depends on the data and is not in general satisfied by A, as expected.
The proof goes as follows: Given any s X t matrix U with rows u, (i = 1,..., s) and vector v in R!. let (Ur); un v denote the ith entry of Uv. We have (UU).//u, and (Uv)); /)://u
16

---

Published as a conference paper at ICLR 2026
for any } i, and it follows that the two Jacobians (Ay) and (B are respectively the x p
and n x matrices
0
xx AT
0
y
and
(21)
0
0"
x² A
Substituting these values in Eq. 16 shows that (0,, 1) a q block diagonal matrix whose n (m x n) diagonal blocks are all equal to [yr' JAA". Transposing A v) = 0 concludes the proof.
We end this section with the main proposition used in the proof above.
Proposition 6.1. Assume that the parametric encoder and predictor are not identically O, i.e. there exists Oo, vo, Zo and za so that (0) + O and g( 2o) + O. Furthermore, assume both f and g have final linear layer, In other words, they can be expressed in the following form:
x(0 := x))
g(b;2) == 2)(
(22)
some where A and B are n. x k and n. X d matrices for some positive integers is and d. while U and V are parameters so that vee( A, U) and s/- veel B. V). Here, & and h are differentiable functions
with values in Be and TRd. Furthermore, consider the quantity
A00, 1b x. == Je y) Jug(V, f(0,x)),
(23)
where Jef and Jug denote the Jacobians of f and g w.n.. and vb. Then A00,0, cannot be identically O.
Proof. It suffices to show that the components Á of A corresponding to the partial derivatives w.r.t. the linear parameters A and B are not identically O. Hence, by contradiction, we will assume that Á vanishes everywhere. Hence, Á must vanish, when applied to any arbitrary perturbation matrices 5.A and 8B of A and B, that the following holds:
(24) The above identity is obtained by standard calculus. Furthermore, since f and g are non vanish- ing, there exists parameter values Un and Vo so that + ((U,, 2) and + h(Vo. 2) are not identically 0. Let yo be a vector in RT so that (UV, yo) * 0. Moreover, fix any arbitrary vec- SAd(U,, yo) (simply take
T (0, ar, () 8B) = (6AA(U, u)) 8Bh(V, Aδ(U, x)) 0.
tor 2 R". There must exist perturbation matrix 5.A that
5AA ou 20( Bo) )) Therefore, by Equation 24 it follows:
(25) for any z in R" and any matrix 8B. This directly implies that h(V, AQ x)) = 0 for any A, x and
z ZT (6 x))) =0,
V. Furthermore, by choosing V = Vo. x yo and A = 3o (Un, yo) for any arbitrary R", get that: h(Vo, 2) 0. This contradicts the fact that + h(Vo, 2) is not identically
0. Hence, have shown that Á is not identically 0 which, a fortiori, implies that A is itself not identically null.
PROOF OF LEMMA
We can rewrite F in this case F(A,
1 Cyll2 + 2 CIAIE + [BI)))
= A BAr - TT BAx + 1 CT + CH + -I[I)
(26)
We know from Eqs. [70] and [82] in the matrix cookbook (Petersen and Pedersen 2012 2 that
a T (bb) abT.
au
(27)
-VTUbe" + VTUUbT+VUUR -UOLT.
00
follows that:
0F
(A, =B - Cy)x + XA,
0A
(28)
OF
(A,,,Crz.y) = (BAx - -Cy)=T AT + XB.
0B
17

---

Published as a conference paper at ICLR 2026
PROOF OF LEMMA 4.2
Multiplying the first equation in 44) by A on the right and subtracting the second one multiplied by Bd on the left immediately yields B² B AA at limit point since both derivatives P(A, C) and B,C) are zero at such a point.
PROOF OF LEMMA
Let us first write the derivative of with respect to A. We have
dE
0F
+ Ayu" BAry" = Bd (BAx - Ay)x - (BAx - Ay)y + XA,
(29)
04
0A
and thus A = -BB" R(A, - + where S(A, 13) = BA(zy") - Substituting the temporal derivative A/2 we obtain
a A = ((r (At A)] = -tr B" R(A. + A' AATA)
-tr [IT A B" + T AT - xx" A Ay y" AA + AATA)
-EE,, |BA - - )
S A|
(30)
which implies the exponential convergence of || toward zero.
PROOF OF LEMMA 4.4
Multiplying on both sides the first equality Eq. left by Bd and taking the difference yields
5 on the right by AT and the second one on the
BT (B + XB) = (À + AA)A
(31)
Adding this equation its transpose and multiplying both sides by 22 now yields
2M
2 B + BT B
2AB"
B) e "AA" + +AA 2AAA"),
(32)
from which we conclude that
- 1e
d 2M B" B]
1
d 2 AAT], AA
(33)
di
di
and obtain AAT B + e K, where K is constant independent of the data. This implies in particular that B AAT -
as t + t00.
PROOF OF PROPOSITION 4.5
We only give there the proof for the SG algorithm since the proof for the EMA algorithm follows the exact same reasoning with the extra parameter C' known to be equal to A at an equilibrium.
The equilibria (A, B) of the SG algorithm are characterized by the two equations
{ 0 BT BT (BA[r="" Alyx XA,
0
(34)
0 = (BA[z=') Alyæ DA + AB = BW - Alyx""A' O.
The fact that B = AAT follows immediately from multiplying both sides of the first condition by AT on the right and both sides of the second one by 8" on the left, then taking the difference. Equation ( follows immediately from the second condition, the inverse being well defined when since W symmetric positive definite in this case. Substituting this value the first equality
of Eq. 34 now yields
W-tA[Ey"11
A(yx²1) AA=
(35)
and equilibria of the SG algorithm are exactly the pairs (A, B) where A satisfies this condition and B is given by Eq. (7). Assuming that A has full rank m and multiplying both sides of this equation on the left by W and on the right by AT yields equivalent condition
A J (Alya") ATW WW- Ald) Alyx²) JA²) AW AAT =0
(36)
18

---

Published as a conference paper at ICLR 2026
Multiplying both sides of this equation the right by W, we obtain the condition
AQU -U-V"V) 0, where {
U A A A|xx") ] + Ald,
(37)
V Alyx
which is equivalent to UTU VTV since we have assumed that has full rank, and thus Eq. as well.
(6)
Note that Tian et al. (2021) give in Appendix D of their paper an alternative characterization of equilibria in the case where À = 0, which essentially corresponds to the condition of Eq. 7 in this case, without the characterization of S = A" A by Eq. 66.
PROOF OF COROLLARY 4.6
The proof is textbook material and included for completeness. When U is column orthogo- nal and A UVSR, we obviously have A A Sk. Conversely, when ATA Sg. let us take U = AV5; (the inverse is guaranteed to exist since Ss is positive definite). We have UTU = V S ld.
PROOF PROPOSITION
;= [yx' + 81 and R == [xx 51 are Fix O. There exist 0 < 8 S. positive so that T' :=
both invertible. To see this, it suffices to notice that + det((yz' + ô1) a non-zero polynomial, thus vanishes for finite number of values 8. Therefore, det([yz + 5I) * O, so that is invertible. Furthemore, is necessarily invertible since it is the sum of the PSD matrix [xe: J] and the PD matrix Q1. The matrices R and T correspond to the covariances of the following perturbed variables:
always find 5 < for which
-=
ÿ うーy+8
(38)
where is standard isotropic gaussian. Thus we have R = [T and T = 1 show that there exists positive Ao small enough that the following equation always admits PD solution for any 0 S À < Ap:
We will
IS + (i "
XId) = [y JS
(39)
We will first establish existence of the solution for À = 0, then show that the property still holds for small enough.
Case À = 0. In this case, the equation simplifies to:
R ST,
(40)
where we used the notation R and T for simplicity. Since, R is invertible, we multiply both sides by R- 1 (left and right), which yields:
S2 (
(41)
Since the matrix TR-1 is invertible, we can directly apply the technical Lemma 6.2 stated below, which guarantees the existence PD solution S to the above equation.
Case À > O. We will apply the implicit function theorem to show the existence of solutions for small enough. Consider the matrix valued map ( S) defined as:
G(, 8) = 52 + A(SR + R 1) + A (TRR"))T((RRR
(42)
We have already established that the equation ( S) = 0 admits solution St, It suffices to prove that the partial differential (0, S*) at (0, S*)) is invertible. Since ds S*) is linear map from the set of m m matrices to itself, it suffices to establish its injectivity. Direct calculations show that (0, (H) is given by:
(43) Using again the technical Lemma 6.2 we know that the only solution to the equation (O is H O. Hence, (O, is injective. Therefore, by the implicit function theorem, there exists a positive A so that for any < < Ags the equation G(A, S) = admits a solution S*.
do (H) = H - "
19

---

Published as a conference paper at ICLR 2026
Lemma 6.2. Let C be an invertible m matrix. There exists symmetric PD solution X* the following equation:
x = =C_
Moreover, consider the following linear system X + -H* C HC with unknown H. The only solution to such system is H 0.
Proof. Existence. We will apply Brouwer's fixed point theorem to a suitable operator. Denote by μ and p its smallest and largest eigenvalues of C C which are positive. Define the following continuous map G over the set of S of symmetric PSD matrices of size m X m:
g(X) = (C XC)
where the square root denotes the unique PSD square root PSD matrix. Note that ((X) is well defined for any X - Consider the following set of matrices:
M E
x
Then MI is a convex compact subset of vector space of matrices. We will show that G(M) M, which will allow us to apply Brouwer fixed point theorem. Indeed, for any X M, simple matrix inequalities yield:
p S CT XC S PC
Recalling that the symmetric square root preserves the matrix order, we directly get:
(CTC) s ((X) S p (C
However, by definition of and we have that μI < CTC S pl. Consequently, it follows that ul s 9(X) & pl. Hence, we have established that MI is stable set of the map G. Since the map G(X) is continuous and M. is convex compact subset of the space of square matrices, it follows by Brouwer's fixed point theorem that there exists X satisfying the equation g(X*) X'. After taking the square of such equation, we get that X is solution to X C XC.
Uniqueness of the solution to the linear system. Let be an m x m matrix solution to the linear system:
X + HX* CT HC O.
We wish to show that = 0. can multiply such equation (left and right) by (X+)- to get:
(X (x*)) (X"))+ - ( HC(X*)
By direct calculation and using the definition of X* (i.e. the solution to the equation X = CT XC), we have that E satisfies EE = X Now, consider the change of variables H = (x*) (X+)) 3. We can thus express the above equation in terms of il and E and X as follows:
Define E -
x + Ax - EÉE T = 0.
Since E is invertible, we can further multiply the equation by E-1 1 on the left and by E on the right and use the identity EET = X* to get:
N.
The above equation directly implies that и must be symmetric. Furthermore, by direct calculation, obtain the following expression for / [|
+ E HE) (E -T-T + FE-HAE) || | = Tr ( #E
tr(EE H
2tr(E HE E (E) ttr(E H²
2 M|
The above identity can only be true if H | 0 which directly implies that H since X* is invertible.
20

---

Published as a conference paper at ICLR 2026
PROOF OF PROPOSITION 4.10
We only present here the proof for the EMA procedure. The SG case is similar and slightly simpler, and thus omitted for conciseness. Let (A, B) be an equilibrium point of our dynamical system and (A + D, B + E. C + F) point in its neighborhood, where, like A, B and C, the matrices D, E and F are respectively of size nu x m, n x n and n X m. To first order, we have
Á(A+ D, C F) R -B* BD) rx 1 Flyx*) E" R(A, AD, B(A+ D, B F,C P) R ((EA BD) Je 1 Flyx D)A" R(A,B,C)D -AE, CIA+D,B+E,C+ = (1- P).
(44)
The eigenvectors of the corresponding linear operator in D, E and F and the corresponding eigen- values f are thus characterized by
0 = Bd ((EA+ + BD) [xr Flyx B() A) { 0 (EEA+ BD) pxx [x-1 -
+ ( EEE
Flyal D B(A,B, (1 - a = a F.
(45)
Let us first consider the trivial equilibrium where = and B = 0. Substituting these values in Eq. (45) shows that for any triplet (D, F. satisfying this equation
must have either = -À < O if D or E is nonzero, or f. a 1 D and E are both zero But, as noted before, normally taken smaller than equal to 1 and we assume here that a: + (the moving average would not make much sense otherwise since & would be constant in that case), so < 0 in that case as well. It follows that trivial equilibria are asymptotically stable
Let us now consider the case of nontrivial equilibria where, in particular, A +f O. Note that for any eigenvector triplet (D, E, F)) and associated eigenvalue μ satisfying Eq. (45). either 1 a + f is zero, in which case F can take any value and, as just observed, < 0, or it is not, in which case F 3D with (1 a1 a+ + p) so we can focus on the first two equations.
Multiplying the first one on the right by A and the second one on the left by 8", subtracting the two and using Eq. (34) yields
+ -MM where M (DAT - B))..
(46)
Now, any matrix U such that UT aU for some scalar also verifies U = aUT by taking the transpose on both sides, and thus = aUT = a(at)) = a²U, which means that, either U = 0 or U + and a² = with either a and U symmetric or a 1 and U skew-symmetric. In our case, when M + 0, it is either symmetric with μ - 2X, or skew-symmetrie with = 0. In the first case, any equilibrium asymptotically stable according to Theorem 4.9 while, in the second one, all eigenvalues vanish and nothing can be said to first order, which should not happen generically.
Let us now prove that μ < 0 in the slightly more complicated case M Using B²E DAT, multiplying again the first equation in 445 on the right by A" (or the second one by B" on the left) and using Eq. (34) now yields
(TEEAA+ BD) Fly + ET R(A,
(++D)AA
(DAT AA" [xx BT Flyx" A - XAD +A*+))DA",
(47)
= N + P "DAT,
where
{ N AAT D NT LA
3BB D(yo0")A AAT XADT P P=DATA X [ + ATAT -DA" Alxx"|AA + Ald).
(48)
Here, N is an n x n matrix of rank at most m with n > m. therefore singular with kernel of dimension m. So is its transpose. Let us pick u in Ker(NT) such that AD u + O. Generically this is always possible since there no reason for Ker(N"') and Ker(AD")) to coincide.
Multiplying the second equation in 48) on the left by uf and on the right by AD u + 0 now yields
[|]=="A'' + lld)e + s||lul12 = 0,
(49)
and since A(x=T)A² Ald is positive definite, we conclude that μ < 0.
21

---

Published as a conference paper at ICLR 2026
PROOF OF PROPOSITION 4.11
When m = 1, S = ||a|l and Eq. can be rewritten as
(pllall + X)² = =
(50)
or equivalently
- X = 0 where #1.
(51)
A necessary and sufficient condition for real solutions of this quadratic equation in ||a|| to exist is that its discriminant A be nonnegative and they are nonnegative when ET is itself nonnegative, i.c., sign(r), so
p||a|| |r| ||l|| -
(52)
These solutions indeed correspond to the two hyperspheres S and S2 defined above, which con- cludes the proof of the first part of the proposition. These correspond exactly to the varieties A and A2 associated with the two positive roots rî and rå of the quadratic equation Eq. (6) in ||a|? of
course. Now, let a be an element of (i = 1, 2). According to Eqs. 77 and (2) we have
1 siin(r)aaa.
B = = W-
2||a||2 an
(53)
ri
This concludes the first part of the proof of the proposition.
Le to its second part, assume A 2 and consider equilibrium with in S.- As shown in the proof of Prop. 4.10 generically, all eigenvalues are negative unless M = da 8² Substituting the value of B in this equation shows that da" = (sign(7)/r.)aa")) eigenvectors are only defined up to scale we can pick d = and E = sign(T)rae (note that there exists an infinity of n x n matrices verifying this equality, including E B). Substituting in Eq. (45 and using Eqs. 11 and (22) now yields
and since
0 = ( - 2sign(r)r; + A p)a- = (μ (μ- X+ pr²)a = (u 2A I7|r:)a
(54)
and thus, when , 0, 14 = 2A |r|:; (note that with this value for μ., that E sign(r) satisfy Eq. (45) and are thus indeed the (d, E) part of the corresponding
and any matrix such
eigenvector). Now, r; (|Tl 7/a A) 2/p where 7 -1 and 1)2 = so we have
1
A - 2p
-
== 2\
(|T| +
(A
(v + ml
(55)
2p
2p
In particular, a necessary and sufficient condition for μ to be positive that 7/1 = 1, corresponding to the hypersphere of radius T1, and that
(56)
which is always true.
does not since the case where a is vector is non generie: in this case = x is a scalar (and thus commutes with all matrices involved) and B (e/ [e/aal]aaa", and
Proposition 4.11
appears to contradict Proposition 4.10
rewrite NT
N = [p
ET (d (((d-a)a-a a)a Adla" - ||a|]
|a|12 |aal2aaa
1 aa + Id)da --( aa + Id)aa (57)
||al22
In particular, since ((1/|aa||3aat) and we cannot conclude that is negative in the last part of the proof of Proposition 4.10 This is generically not the case for >> so there is no contradiction.
Id) is positive definite, Ker(NT)) Ker(aaT) Ker(ad
22


Reviews:
[{"review_id": "hfPrOwaZyk", "reviewer": "Reviewer_PrY7", "summary": "This work studies the role of two key components: stop gradient and Exponential Moving Average (EMA) for non-contrasitve from both optimization and dynamical systems perspectives. From the optimization perspective, they consider a general setting and prove that neither SG nor EMA can be interpreted as minimizing a well-defined smooth loss function, but they can avoid the representation collapse. From the dynamical systems perspective, they prove with SG and EMA that the dynamic system is asymptotically stable (assuming a linear setting). They also conduct real-world and simulation experiments to support their discussion.", "questions": "Question: \n\nQ1: Why does the proof of this work require weaker assumptions compared with Tian et al?\n\nMinor Comments: It would be better to use more formal writing, such as lines 247-248, and some experimental settings can be moved to the appendix. Then, there is more space for the proof sketch.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "* This work clearly shows the influence of SG and EMA from the theoretical perspective and conducts experiments to support their results.", "weaknesses": "* It would be better to provide a proof sketch for Proposition 2.2, which makes it easier for readers to understand why with SG and EMA, the $\\bar{P}$ and $\\bar{Q}$ are not the gradient fields of any smooth function.\n\n* We know that the analysis for the nonlinear setting is hard. However, it would be better to discuss how to extend to a nonlinear NN (even a 2-layer Softmax NN).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oS1kDnCCAQ", "reviewer": "Reviewer_dqoh", "summary": "In this paper, paper provides a theoretical study of two core techniques used in non-contrastive self-supervised learning (SSL) methods - stopping Gradient (SG) and Exponential moving Average (EMA) - that are essential to prevent representation collapse in algorithms such as BYOL and SimSiam. From a dual perspective of optimization theory and dynamic systems, the authors investigate why these methods have been empirically successful in the absence of negative samples and explicit regularization.", "questions": "1. In proving that P, Q is not a gradient field, the author uses the term \"in general\" several times, but does not explicitly state the specific conditions under which it applies. As a reader, it is difficult to judge whether there are exceptions to this conclusion. If there are such special cases, can you give an intuitive example or a theoretical explanation?\n2. The analysis in Chapter 3 assumes that both encoder and predictor are linear maps, which leads to conclusions about the asymptotic stability of the dynamical system under this condition. But in real networks are usually nonlinear. Then, can this stability analysis obtained in the linear case be extended to nonlinear models?\n3. Since α is assumed to be constant in the analysis, do the authors explore the impact of dynamic changes in α on stability or convergence?\n4. After all the proofs and discoveries, how can this be implemented technically (like a solution or new method the authors propose based on the observation)?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The topic of the paper is good. It tries to explain non-contrastive self-supervised methods like BYOL/SimSiam from both optimization theory and dynamical systems theory, which \"learn good representations without seeming to have an objective function.\" This combination is innovative in the research of SSL theory. In particular, the perspective of dynamical systems (continuous-time analysis, equilibrium points, and stability proofs) provides new insights into the dynamics of self-supervised training. The authors verify that the non-convergence properties of SG and EMA are consistent with the theory on video classification tasks. Authors also conduct toy experiments to illustrate the trajectory of the dynamical system. Although the experiment is small-scale, it serves as an intuitive verification of the theoretical behavior.", "weaknesses": "1. The paper introduces several symbols on the same page, and these symbols represent different meanings in different contexts. This high density of symbolic definitions makes readers need to frequently backtrack and compare with the previous texts during reading, which increases the burden of understanding and is not conducive to quickly grasping the core deduction logic. \n2. I understand the authors’ intention to introduce examples of SG and EMA directly in the introduction. However, such a structure can be risky if not carefully managed. The current presentation makes it challenging for readers to grasp the main content of the paper, the problem being addressed, and its practical significance.\n3. The experiments only include two video classification tasks with low-dimensional toy simulations. There is a lack of comparison with other theoretical assumptions and a lack of quantitative metrics showing the extent to which the original objective function is not minimized.\n4. In Proposition 2.1 and 2.2, \"in general\", \"in practice\", \"it is unclear whether...\" appear several times. The authors lack clarity on which cases are valid and which are exceptions.\n5. Authors must not assume that all readers are familiar with technical terms, so authors need to approach the introduction of their work from the general ML audience perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bWsR39tFZb", "reviewer": "Reviewer_f64b", "summary": "This paper investigates the common self-supervised learning procedures of **stop gradient** and **exponential moving average**, which are used to prevent representation collapse. \n\nThe authors show that while these iterative procedures do not optimize the original or any other smooth objective function, they are effective at avoiding collapse. \n\nUsing a dynamical systems perspective in the linear case, the study demonstrates that minimizing the original objective *without* these procedures always leads to collapse, and it explicitly characterizes the **asymptotically stable equilibria** for both procedures as algebraic varieties, which are illustrated with empirical experiments.", "questions": "See weaknesses", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Provides a clear theoretical foundation explaining why stop-gradient and EMA prevent representation collapse.\n2. Offers rigorous analysis from both optimization and dynamical systems perspectives.\n3. Empirical experiments effectively support the theoretical findings on real and synthetic data.", "weaknesses": "1. The statement in Lines 156–157 (“The SG and EMA training procedures have been designed to avoid collapse in self-supervised learning”) is not entirely accurate. Prior work, such as SimSiam, has already shown that Stop-Gradient alone can prevent collapse without EMA. Moreover, BYOL’s EMA inherently includes a Stop-Gradient operation, and the true collapse-prevention factor lies in the asymmetric *predictor* component — without it, SG or EMA alone would fail.\n\n2. While the paper provides some theoretical analysis of non-contrastive learning, the discussed methods are now outdated compared to more recent and stronger approaches such as MoCov3, DINO, iBOT, and MAE-based self-supervised models. Several studies [1–3] have already offered deeper theoretical insights into both contrastive and non-contrastive frameworks, making the contribution of this paper less timely and impactful.\n\n3. The experimental validation is weak, relying heavily on limited and synthetic data without robust quantitative evaluation on standard benchmarks. This weakens the credibility of the theoretical claims — more comprehensive and realistic experiments are needed to meet the expectations for publication.\n\n[1] *How Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-Supervised Contrastive Learning*, ICLR 2022\n\n[2] *Understanding Collapse in Non-Contrastive Siamese Representation Learning*, ECCV 2022\n\n[3] *Contrasting the Landscape of Contrastive and Non-Contrastive Learning*, AISTATS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqwontLAUL", "reviewer": "Reviewer_sCRT", "summary": "The paper investigates two popular optimization approaches in non-contrastive learning: exponential moving average (EMA) and stop-grad (SG), both of which are employed in practice to prevent the model from feature collapse (that is, learning the trivial solution where model output is always zero). This question has been previously studied, starting from the work of Tian et al (2021), but the present work weakens some assumptions by relying on classic results from dynamical systems.\n\nThe main contributions of this paper are:\n- to show EMA/SG does not optimize the original loss function and may not optimise any well defined function\n- assuming the models are linear, the paper characterises the equilbria of the dynamical systems associated with EMA and SG, and proves that these equilibria are asymptotically stable", "questions": "The main questions are in point-2 of weakness. Several other minor comments are listed below:\n\n1. Description of SG: I think the SimSiam paper eventually uses a symmetrized version of the loss, and hence, SG gets applied to both models. This is not considered in the present paper, where SG is applied only on $\\xi$. It would be good to clarify this. Also line 99 (SG step-(a)) is confusing if one directly replaces $\\xi_{t-1}$ by $\\theta_{t-1}$ (because the update of second model is not clearly specified). The description of EMA with $\\alpha_t=0$ is more clear.\n\n2. lines 227-234: u,v are not defined and $\\lambda$ is missing for inline expressions for $\\nable \\Omega$.\n\n3. Proposition 3.5: Assumption $A$ is full rank is incorrect because it is a rectangular matrix. It is better to assume $S$ is strictly positive definite. By the way, is it possible to characterise when (if at all) is $S$ strictly positive at equilibria? This also may depend on the data, and may not hold generally.\n\n4. Line 368: \"Such a system admits $2^{m(m+1)/2}$ solutions.\"  It is not clear to me why this should hold. Please provide a reference or proof.\n\n5. Theorem 3.8: Replace $A$ with a different notation (since $A$ is used in the model)\n\n6. Lemma 3.4: The result that $B^\\top B = AA^\\top$ asymptotically is reminiscent of several results on gradient descent for deep linear networks (dating back to 1990s). It would be good to include a short discussion on this. Perhaps the asymptotic equality is a consequence of EMA/SG in this setting. \n  \n7. Line 665: \"There is a priori no reason ..\" I don't believe this is a formal way to prove a statement. You need to provide a concrete counter-example. Consider a simple case of a data and loss function to argue that a critical point of $F$ may not be a critical point of $E$. \n\n8. Line 687-688: Same issue as above. Use the linear case (discussed after the proof ends) and assume some simple fixed data to make your point. Perhaps assuming the augmentation is $y = x + \\epsilon$ could work here.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "EMA and SG are often deployed in optimising deep learning models to prevent collapse. While the focus of this paper is on non-contrastive models (where these techniques are always used), there is a general need for understanding EMA/SG, and how these techniques can lead to desirable/undesirable solutions.\n\nIn particular, this paper provides theoretical/empirical evidence that EMA/SG do not minimise the original objective and the parameters need not necessarily converge, but the equilibria can be characterised as an (asymptotically stable) algebraic variety. These characterisations can provide a more informed understanding of when to (not) apply EMA/SG in deep learning or derive theoretically sound fixes.", "weaknesses": "1. The interesting results (Section 3) assume linear models: I note this first because it's a typical grudge about many theory papers. However, it's important to make two observations:\n- The linearity is assumed with respect to trainable parameters. Notably, one could replace the linear non-contrastive models $f_\\theta, f_\\xi$ in Section 3 by random feature models or deep networks in lazy-training regime. This can be easily fixed in the paper by redefining $[xx^\\top], [xy^\\top]$. I believe this would not change any analysis. So, in principle, the analysis holds for models that are nonlinear with respect to input data.\n- The linearity with respect to training parameters is necessary with our current theoretical understanding of EMA/SG. It is evident from the paper that interesting conclusions, which were not previously known, can still be derived assuming linearity. In summary, this \"weakness\" is just to explicitly refute any other possible review that criticises the linearity assumption in the paper\n\n2. Vagueness of some proofs and assumptions. The presentation of some of the proofs is not formal enough (a few less-critical concerns are listed in questions). A bit more critical concern is the lack of clarity of loss in Section 3. Although not clearly stated, the loss is assumed to be a squared distance between the representations from the two models. Many of the proof steps rely on this fact. Here are a few observations:\n- While the analysis of EMA/SG is motivated by papers on BYOL/SimSiam, both consider cosine-based loss functions and not exactly squared distance between (linear) model outputs. The normalisation plays a crucial role in these works, which cannot be explained by the current analysis.\n- Some of the claims in Section 3 cannot extend to other losses. A simple example is the following case of SG: $\\theta(0) = \\xi(0) = 0; \\dot{\\theta}(t) = \\cos(t); \\dot{\\xi}(t) = \\theta(t) - \\xi(t)$. In this case, WolframAlpha gives the solution: $\\theta(t) = \\sin(t)$ but $\\xi(t) = (e^{-t} + \\sin(t) - \\cos(t))/2$. Hence, in this case, the continuous-time dynamics suggested in Equation (5) is an incorrect approximation to SG in discrete time. Similar examples can be constructed for EMA as well, raising doubt for the continuous-time dynamics.\n- The proofs of Lemma 3.2 and Proposition 3.5 assume that the parameter $C$ follows $A$ (since it holds for SG in discrete time), but shouldn't these proofs include the dynamcis of $C$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)