Implicit variance regularization in non-contrastive SSL
Authors: Manu Srinath Halvagal
Axel Laborieux
Friedemann Zenke
Venue: NeurIPS 2023 poster (2023)
Keywords: Self-supervised learning
Non-contrastive learning
Learning dynamics

Abstract: Non-contrastive SSL methods like BYOL and SimSiam rely on asymmetric predictor networks to avoid representational collapse without negative samples. Yet, how predictor networks facilitate stable learning is not fully understood. While previous theoretical analyses assumed Euclidean losses, most practical implementations rely on cosine similarity. To gain further theoretical insight into non-contrastive SSL, we analytically study learning dynamics in conjunction with Euclidean and cosine similarity in the eigenspace of closed-form linear predictor networks. We show that both avoid collapse through implicit variance regularization albeit through different dynamical mechanisms. Moreover, we find that the eigenvalues act as effective learning rate multipliers and propose a family of isotropic loss functions (IsoLoss) that equalize convergence rates across eigenmodes. Empirically, IsoLoss speeds up the initial learning dynamics and increases robustness, thereby allowing us to dispense with the EMA target network typically used with non-contrastive methods. Our analysis sheds light on the variance regularization mechanisms of non-contrastive SSL and lays the theoretical grounds for crafting novel loss functions that shape the learning dynamics of the predictor's spectrum.

Implicit variance regularization in non-contrastive SSL
Manu Srinath Halvagal 1.2,+
Axel Laborieux !.
Friedemann Zenke 1.2
(firstname lastnaaae @aiiaammm@@mm...mmmmmmm ch
Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland Faculty of Science, University of Basel, Basel, Switzerland These authors contributed equally.
Abstract
Non-contrastive self-supervised learning (SSL) methods like BYOL and SimSiam rely on asymmetric predictor networks to avoid representational collapse without negative samples. Yet, how predictor networks facilitate stable learning is not fully understood. While previous theoretical analyses assumed Euclidean losses, most practical implementations rely on cosine similarity. To gain further theoretical insight into non-contrastive SSL, we analytically study learning dynamics con- junction with Euclidean and cosine similarity in the eigenspace of closed-fome linear predictor networks. We show that both avoid collapse through implicit variance regularization albeit through different dynamical mechanisms, Moreover, we find that the eigenvalues act as effective learning rate multipliers and propose a family of isotropic loss functions (IsoLoss) that equalize convergence rates across cigenmodes. Empirically, IsoLoss speeds up the initial learning dynamics and increases robustness, thereby allowing us dispense with the exponential moving average (EMA) target network typically used with non-contrastive methods. Our analysis sheds light the variance regularization mechanisms of conttraraeteve SSL and lays the theoretical grounds for crafting novel loss functions that shape the learning dynamics of the predictor's spectrum.
Introduction
SSL has emerged as a powerful method to learn useful representations from vast quantities of unlabeled data [1-8]. In SSL, the network's objective is to "pull" together its outputs for two differently augmented versions of the same input, so that they learn representations that are predictive across randomized transformations [9]. To avoid the trivial solution whereby the network output becomes constant, also called representational collapse, SSL methods use either a contrastive objective to "push" apart representations of unrelated images [2. 3. 10-13] or other non-contrastive strategics. Non-contrastive methods comprise explicit variance regularization techniques [6, 14], whitening approaches [15, 16], and asymmetric losses in Bootstrap Your Own Latent YBYOL) [1] and SimSiam [5]. Asymmetric losses break symmetry between the two branches by passing one of the representations through predictor network and stopping gradients from flowing through the other "target" branch. How this architectural modification prevents representational collapse is not obvious and has been the focus of several theoretical [17-21] and empirical studies [22-24]. A significant advance was provided by Tian et al. [17] who showed that linear predictors align with the correlation matrix of the embeddings, and proposed the closed-form predictor DirectPred based on this insight. However, previous analyses assumed a Euclidean loss at the output [17-19, 21J except [20], whereas practical implementations typically use the cosine loss [1. 5] which yields superior performance on downstream tasks. This difference raises the question whether analysis based on the Euclidean loss provides an accurate account of the learning dynamics under the cosine loss.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this work, we provide a comparative analysis of the learning dynamics for the Euclidean and cosine-based asymmetric losses in the eigenspace of the closed-form predictor DirectPred. Our analysis shows how both losses implicitly regularize the variance of the representations, revealing connection between asymmetric losses and explicit variance regularization in VICReg [7]. Yet, the learning dynamics induced by the two losses are markedly different. While the learning dynamics of different eigenmodes decouple in the Euclidean case, dynamics remain coupled for the cosine loss.
Moreover, our analysis shows that for both losses, the predictor's eigenvalues act as learning rate multipliers, thereby slowing down learning for modes with small eigenvalues. Based on our analysis, we craft an isotropic loss function (IsoLoss) for each case that resolves this problem and speeds up the initial learning dynamics. Furthermore, IsoLoss works without an EMA target network possibly because it boosts small eigenvalues, the purported role of the EMA in DirectPred [17]. In summary, our main contributions are the following:
We analyze the SSL dynamics in the eigenspace of closed-fome linear predictors for asym- metric Euclidean and cosine losses and show that both perform implicit variance regulariza- tion, but with markedly different learning dynamics.
Our analysis shows that predictor eigenvalues act as learning rate multipliers which slows down learning for small eigenvalues.
We propose isotropic loss functions for both cases that equalize the dynamics across eigen- modes and improve robustness, thereby allowing to learn without an EMA target network.
2 Eigenspace analysis of the learning dynamics
To gain a better analytic understanding of the SSL dynamics underlying con-contratt methods such BYL and SimSiam 51, we analyze them in the predictor's ' eigenspace. Specifically we proceed in three steps. First, building on DirectPred, we invoke the neural tangent kernel (NTK) to derive simple dynamic expressions of the predictor's eigenmodes for Fuclidean and cosine loss. This formulation uncovers the implicit variance regularization mechanisms that prevent representational collapse. Using the eigenspace framework, we illustrate how removing the predictor or the stop- gradient results in collapse or run-away dynamics. Finally, we find that predictor eigenvalues act as learning rate multipliers for their associated mode, thereby slowing down learning for small eigenvalues. We derive modified isotropic loss function (IsoLoss) that provides more equalized learning dynamics across modes, which showcases how our analytic insights help to design novel loss functions that actively shape the predictor spectrum. However, before we start our analysis, we will briefly review DirectPred [17] and the NTK [25], a powerful theoretical tool linking representational changes and parameter updates. We will rely both concepts for our analysis.
2.1 Background and problem setup
We begin by reviewing DirectPred [17] and defining our notation. In the following, we consider a Siamese neural network = [2x; 0) with output E RMM input E RN and parameters O. We further assume a linear predictor network Wp E M and use the same parameters for the online and target branches as in SimSiam [5]. We denote pairs of representations as 21)) z22) corresponding to pairs of inputs (1)), (2) related through augmentation and implicitly assume that all losses are averaged over many augmented pairs. The asymmetric loss function (Fig. la), introduced in BYOL [1], is then given by:
L d (W)z(), (Wpz(),SG((()) ,SG(=()))
where SG denotes the stop- grodigradentt operation, and d is either the Euclidean distance metric d(a, b) = llla - b or the cosine distance metric d(a, b) a [6] functions as Laue and Coos respectively.
a b We refer to the corresponding loss
DirectPred. Tian et al. [17] showed that a linear predictor in the BYOL setting aligns during learning with the correlation matrix of representations C. := E,, [zz T. where the expectation is taken over the data distribution. Since the correlation matrix is real symmetric matrix, one can diagonalize over IR: C- = UD where U is an orthogonal matrix whose columns are the cigenvectors of C2 and De is the real-valued diagonal matrix of the eigenvalues s, with E [1, MM..

---

a
da under gradient descent on C d (We2")
b Euclidean distance
Cosine distance
Loss
t >1
X
(P2
0.00
Stop-grad
2 (2)
Predictor
good 2()
(2)
9900
Wp2
A2 <1
Online net
Target
0
0
(a) Schematic of Siamese network with predictor network and a stop-gradient on the target network branch. The target network can copy (SimSiam [5]) or moving average (BYOL [1]) of the online network. In either case, the target network is not optimized with gradient descent. (b) Visualization of learning dynamics under the Euclidean distance metric showing learning update directions along two eigenmodes, with the light cloud representing the distribution of the representations 2, the darker cloud representing the predictor outputs Wpz, and the dotted circle indicates the steady state \12
Figure 1:
reached during learning. All eigenvalues converge to one. (c) Same as (b), but for the cosine distance. The dotted line indicates the steady state A X2.
Given this eigendecomposition, the authors proposed DirectPred, in which the predictor learned via gradient descent but directly set to:
Wp = fa (C=)
(I)
where is a positive constant exponent applied element to De. The eigenvalues An of the predictor matrix Wp are then Am = S We use D to denote the diagonal matrix containing the eigenvalues Am- While DirectPred used a = 0.5, the follow-up study DirectCopy [18] showed that = is also effective while avoiding the expensive diagonalization step. While Tian et al. [17] based their analysis on the Fuclidean loss Leue, most practical models, including Tian al.'s large-scale experiments, relied on the cosine similarity loss _ This discrepancy raises the question to what extent setting the predictor to the above expression is justified for the cosine loss. Empirically, we find that trainable linear predictor does align its eigenspace with that of the representation correlation matrix also for the cosine loss (see Fig. in Appendix A).
Neural tangent kernel (NTK). The NTK powerful analytical tool characterizing the learning dynamics of neural networks [25, 26). Here, we recall the definition of the empirical NTK [26] corresponding to a single instantiation of the network's parameters If [] denotes the size of the training dataset, L IR + IR an arbitrary loss function, X. the training data concatenated into one
vector size N [D], and Z = = (x)), the concatenated output of size MDD, then empirical NTK
the (M|D) M D matrix:
(x(xx)) = VoZVo2
and the continuous-ting gradient-descent dynamics [26] of the representations z are given by: =
dz
dt
In other words, the empirical NTK links the representational dynamics dr the parameters 0, and the "representational gradient" ZzL.
2.2 Implicit variance regularization non-contrastive SSL
As starting point for our analysis, we first express the relevant loss functions in the eigenbasis of the predictor network. We do this using a closed-form linear predictor as prescribed by DirectPred. In the following, we use & U z to denote the representation expressed in the eigenbasis.

---

Lemma 1. (Euclidean and cosine loss in the predictor eigenspace) Let Wp be a linear predictor set according to DirectPred with eigenvalues Am and the representations expressed in the predictor's eigenbasis. Then the asymmetric Euclidean loss L and cosine loss L can be expressed as:
M
C
Σ
Am
Am
s(1)
SG
(3)
m
M
Am a(1)
SG 2
(4)
[D [G)
for which we defer the simple proof to Appendix B. Rewriting the losses in the eigenbasis makes clear that the asymmetric loss with DirectPred can be viewed as an implicit loss function in the predictor's eigenspace, where the variance of each mode naturally appears through the Am terms. In the following analysis, we will show how the learning dynamics implicitly regularize these variances Am From Eq. (3) we directly that L is a sum of M terms, one for each eigenmode, which decouples the learning dynamics, fact first noted by Tian et al. [17]. In contrast, the form of L
yields coupled dynamics due the || | V (Aw term in the denominator. This coupling arises from the normalization of the representation vectors the unit hypersphere when calculating the cosine distance. The normalization effectively removes one degree of freedom and, in the process, adds a dependence between all the representation dimensions (Fig. lb and Ic).
To get an analytic handle on the evolution of the rrgennrepresentations & as the encoder learns, we first note that if training were update the representations directly, instead of indirectly through updating the weights 0, they would evolve along the following "representational gradients".
20; Leuc -
-
(5)
(2
(D2()))T()))) || (2
V 200
(6)
D&
| 1
In practice, however, representations of different samples do not evolve independently along these gradients, but influence each other through parameter changes in 0. This interdependence of rep- resentations and parameters are captured by the empirical NTK O, X') (cf. Eq. (2)). Because the NTK is positive semi-definite, loosely speaking, gradient descent on the parameters changes representations "in the direction" of the above representational gradients.
To see this link more formally, we express the NTK in the eigenbasis as (x, = VeŻVeŹT where 2 = (x) U (A)). Since are concerned with the learning dynamics in this rotated basis, we will rewrite Eq. (2) for continuous-time gradient descent for a generic loss function C as:
d (z, X'V
dt
Note, that structurally these dynamics are the same as the embedding space dynamics in Eq. (2) but merely expressed in the predictor eigenbasis (see Lemma 2 in Appendix B for derivation). Although Ô, changes over time and is generally intractable in finte-widthh networks, it is positive semidefinite. This property guarantees that the cosine angle between the representational training dynamics under the parameter-sper spacee optimization of a neural network d 2 x - 2 & and the dynamics that would result from optimizing the representations de 2 x
dt
z V is non nog-tegative:
C, d2
n ( 3 C. 20.
In other words, the representational updates due to network training lie within a 180-degree cone of the dynamics preseribed by Eqs. (5) and (6). This guarantee makes possible to draw qualitative conclusions about asymptotic collective behavior, e.g., whether a network is bound to collapse or not, from analyzing the more tractable dynamics that follow the representational gradients di & - -V of the transformed BYOL/SimSiam loss. For ease of analysis, we now consider linear networks with Gaussian i..d inputs, important limiting case amenable for theoretical analysis [27]. In this setting

---

the empirical NTK becomes the identity and the simplified representational dynamics are exact, allowing us to fully characterize the representational dynamics for L and C in the following two theorems. In the proofs for these theorems, we show that the assumption of Gaussian inputs can be relaxed further.
Theorem (Representational dynamics under L For a linear network with i.i.d Gaussian imputs learning with _ the representational dynamics of each mode m independently follow the gradient of the loss Veee More specifically, the dynamics uncouple and follow M independent
differential equations:
d
CCeuc
(1)
1)Am
(((2) =m
Am s(!))
(8)
di
dam
which, after taking the expectation over augmentations yields the dynamics:
d&m
= 1m (1 Xm) 2m
(9)
dt
We provide the proof Appendix and appreciate that d &m has the same sign as &m whenever Am < 1 and the opposite sign whenever Am > 1. These dynamics are convergent and approach an eigenvalue Am of one, thereby preventing collapse of mode m. Since the eigenmodes are orthogonal and uncorrelated, and the condition simultaneously holds for all modes, this ultimately prevents both representational and dimensional collapse [28]. Since the eigenvalues also correspond to the variance of the representations, the underlying mechanism constitutes an implicit form of variance regularization. Finally, we note that the above decoupling of the dynamics for the Euclidean loss has been described previously in Tian et al. [17].
Nevertheless, the representational dynamics are different for the commonly used cosine loss Leos
Theorem 2. (Representational dynamics under ( For a linear network with i.i.d Gaussian inputs trained with C the dynamics follow a system of M coupled differential equations:
d dt
11 D
Σ Nu (2x(2 knfm
Am a(1) (1 ((2)
(10)
and reach regime in which the eigenvalues comparable in magnitude. In this regime, the expected update over augmentations is well approximated by:
dâm dt
RJ 11
E
22
m
E
2m
Σ kegtmm
Ax (At Am))
(11)
D2
where we have assumed averages over augmentations. See Appendix B for the proof. Theorem 2 de &m has the same or different sign as 2m depending on the sign of the aggregate sum Extm \x(A >m)). This relation suggests that a steady state only reached through mutual agreement when the non-zero eigenvalues are all equal. In contrast the Euclidean case, there is no pre-specified target value (see Fig. in Appendix A). Thus, the cosine loss also induces implicit variance regularization, but through a markedly different mechanism in which eigenmodes cooperate.
states that d
2.3 Stop-grad and predictor network are essential for implicit variance regularization.
We now extend our analysis to explain the known failure modes due to ablating the predictor or the grodgraadent for each distance metric. When we omit the stop-grate -gad operator from Leue we have:
dêm dt
nOSG (1W 2201
11 >mm)) &
(12)
so that d &m and &m always have opposite signs (see Appendix C for the derivation). This drives the representations toward zero with exponentially decaying eigenvalues, causing the notorious representational collapse [5]. Omitting the stop-grad hoopgrad operator from Less yields nontrivial expression for the dynamics causing the largest eigenmode to diverge (see Appendix C) Interestingly, this different from the collapse to zero inferred for the Euclidean distance.
de
5

---

Similarly, when removing the predictor network in the Euclidean loss case, the dynamics read:
d 0
Loue noPred
111=00)
(13)
dt
meaning that no learning updates occur. When the predictor is removed in the cosine loss case, the dynamics are:
2o08 noPeed
(2()) (2))
dam
Σ Reffm
E
E
2,
E
2.
&
E
|20)]||((((111
dt
||
(14)
As show in Appendix C, these dynamics also avoid collapse. However, the effective learning rates become impractically small without the eigenvalue factors from Eq. (11). We summarized the predicted dynamics of all settings in Table Thus, our analysis provides mechanistic explanations for why stop-grad and predictor networks are required for avoiding collapse in non-contrastive SSL.
Table Summary of eigendynamics as predicted by our analysis for linear networks.
Loss
dêm/ / x
Predicted dynamics
Leue
Am Am)
As converge large ones faster.
C
- (1 - Am)
All As collapse.
poPred
none
0
No learning updates.
Liso
(1 - 2m)
As converge to at homogencous rates.
L
Am Enzi A Am)
As converge equal values.
Appendix C
All As diverge.
poos
Appendix C
As converge to equal values at low rates.
ngos
Exyo de (Ax - Am)
As converge to equal values at homogeneous rates.
2.4 Isotropic losses that equalize convergence across eigenmodes
In Eqs. (9) and (11) the eigenvalues appear as multiplicative learning rate modifiers front of the difference terms that determine the fixed point. Hence, modes with larger eigenvalues converge faster than modes with smaller eigenvalues, reminiscent of previous theoretical work on supervised learning [27]. We hypothesized that the anisotropy in learning dynamics could lead to slow con- vergence for small eigenvalue modes or instability for large eigenvalues. To alleviate this issue, we designed alternative isotropic loss functions that equalize relaxation dynamics for all eigenmodes by exploiting the stop-grad function. Put simply, this involves taking the dynamics from Eqs. (8) and (10), removing the leading Am term, and deriving the loss function that would result in the desired dynamics. One such isotropic "IsoLoss" function for the Euclidean distance is:
Cove x SG) + z (1) - Wp2
(15)
We note that this IsoLoss has the same numerical value as Leue, but the gradient flow is modified by placing the prediction inside the stop-grad and also adding and subtracting 211) inside and outside of the stop-grate grad. The associated idealized learning dynamics in our analytic framework are given by:
d
17 (1 Am) 2m:
(16)
dt
where the Am factor (cf. Eq. (9)) disappeared (Table 1). Similarly, for the cosine distance,
(2)
+ SG ( (W)z(")) (2)
) |[ 1/2 20p (17)
2 = -( SG
|| [|
is one possible Lsososs, in which W//²2 U with the square-root applied element- to the diagonal matrix D. While this IsoLoss does not preserve numerical equality with the original loss goos it achieves the desired effect of removing the leading Am learning- rate modifier (cf. Table l).

---

a
L
b
L Luue noPred
Lec
d
L 1s0
10"
10-1
1000
100000
1000
1000
Iteration
Iteration
Iteration
Iteration
oos nnSG
cos Poorred Pred
cos
pos C isa
10
10-1
1000
100000
1000 Iteration
1000 Iteration
Iteration
Iteration
Figure 2: Evolution of representations (top) and eigenvalues (below) of Wp throughout training with different loss functions. The representational trajectories correspond training with M 2 for visualization and the points signify the final network outputs. The cigenvalues were computed with dimensions N Euclidean case (top), and diverging eigenvalues for the cosine case (bottom).
and . 10. (a) Omitting the stop-grad leads to representational collapse the
(b) No learning occurs without the predictor with the Euclidean distance, but learning does occur with the cosine distance, although low rates. Note the change in scale of the time-axis. (c) Optimizing the BYOL/SimSiam loss leads to isotropic representations under both distance metries. (d) Optimizing IsoLoss has same effect, but with uniform convergence dynamics for all eigenvalues for both distance metrics.
3 Numerical experiments
To validate theoretical findings (cf. Table 1), first simulated small linear Siamese neural network as shown Fig. la, for which Theorems and hold exactly. We fed the network with independent standard Gaussian inputs, and generated pairs of augmentations using isotropic Gaussian perturbations of standard deviation a 0.1. We then trained the linear encoder with each config- uration described above. Training the network with fess resulted in collapse with exponentially decaying eigenvalues, whereas L succumbed to diverging eigenvalues as predicted (Fig. 2a).
[oos
Training without the predictor caused vanishing updates for Leue RoPPeed and slow learning for C PoPred in line with our analysis (Fig. 2b). Optimizing L the representations become increasingly isotropic with all the eigenvalues Ann converging to one (Fig. 2c, top), whereas optimizing _eos also resulted in the eigenvalues converging to the same value, but different from one (Fig. 2c, bottom). The anisotropy in the dynamics of different eigenmodes noted above is particularly striking in the case of the Eu- clidean distance (Fig. 2c). Training with C and Cons resulted in similar convergence properties as their non-isotropic counterparts, but the eigenmodes converged more homogeneous rates (Fig. 2d). Finally, we confirmed that these findings were qualitatively similar in the corresponding nonlinear networks with ReL.U activation (see Fig. 6 Appendix A). Thus, our theoretical findings hold up in simple Siamese networks.
7

---

Cenc Iso
L BYOL
10a
75
50
10a
25
Iso
1
COs Iso
_ BYOL
d
10a
75
10"
50
25
Iso
10-3
500
1000
0
500
1000
0
500
1000
500
Epoch
Epoch
Epoch
Epoch
Figure 3: Learning dynamics for ResNet- 18 network trained with different loss functions. (a) Evo- lution the eigenvalues of the representation correlation matrix during training for closed-form predictors as prescribed by DirectPred (left) and IsoLoss (center). Right: Standard BYOL with the nonlinear trainable predictor. For clarity, we plot only one in ten eigenvalues. Both L and fiss drive the eigenvalues converge quickly and remain constant thereafter with relatively small fluctuations (note the logarithmic scale). BYOL results in the eigenvalues being spread across large range of magnitudes.
(b) Linear readout validation accuracy for Lewe and Ciue during the first 500 training epochs. IsoLoss accelerates the initial learning dynamics as predicted by the theory. (c) Same as in (a) but for the cosine distance. oos recruits few large eigenvalues, but drives them gradually to the same magnitude, whereas Coos quickly recruits all eigenvalues and causing them converge an isotropic solution. contrast, BYOL recruits eigenvalues in a step-wise manner. (d) Same as (b) but for the cosine distance.
3.1 Theory qualitatively captures dynamics in nonlinear networks and real-world datasets.
To investigate how well our theoretical analysis holds up in non-toy settings, we performed several self-supervised learning experiments 110AR-10, CIFAR-100 [29], STL-10 [30], and Tinylma- geNet [31]. We based our implementation" on the Solo-learn library [32], and used a ResNet- 18 backbone [33] as the encoder and the cosine loss, unless mentioned otherwise (see Appendix D for details). As baselines for comparison, we trained the same backbone using B BOL with the nonlinear predictor and DirectPred with the closed-fore form linear predictor. We recorded the online readout accuracy of a linear classifier trained on frozen features following standard practice, evaluated either on the held-out validation test set where available.
We found that the eigenvalue dynamics of the representational correlation matrix the ResNet- 18 closely mirrored the analytical predictions for the closed-form predictor. For Euclidean distances (Fig. 3a), the eigenvalues for DirectPred and IsoLoss converged to a small range of values around one. However, the dynamics for BYOL with a learnable nonlinear predictor deviated significantly with the eigenvalues distributed over a larger range. Consistent with our analysis, IsoLoss had faster initial dynamics for the cigenvalues which also resulted in faster initial improvements in model performance (Fig. 3b). The faster learning with IsoLoss was even more evident for the cosine distance (Fig. 3c). Surprisingly, BYOL, which dynamics in the case of the cosine distance. Furthermore, the dynamics showed stepwise learning phenomenon wherein eigenvalues findings for other SSL methods [34]. Finally, IsoLoss exhibited faster initial learning (Fig. 3d), in agreement with our theoretical analysis. Thus, our theoretical analysis accurately predicts key properties of the eigenvalue dynamics in nonlinear networks trained on real-world datasets.
nonlinear predictor also closely matched the predicted
progressively recruited one-by-one, consistent with recent
'Code is available https: htpps:///ithull www./mii-aase///eeccccoooaea http://ww....aammmmmmmcta.a.rrre

---

3.2 IsoLoss promotes eigenvalue recruitment and works without an EMA target network.
To further investigate the impact of IsoLoss on learning, we first verified that it does not have any adverse effects on downstream classification performance. We found that IsoLoss matched or outperformed DirectPred on all benchmarks (Table 2) when trained with an EMA target network as used in the original studies. Yet, it performed slightly worse than BYOL, which uses nonlinear predictor and an EMA target network. Because EMA target networks are thought to amplify small eigenvalues [17], we speculated that LsoLoss may work without it. We repeated training for the closed-form predictor losses without EMA to test this idea. We found that 2 was indeed robust to EMA removal. However, it caused slight drop in performance (Table 2) and a notable reduction in the recruitment of small eigenvalues (see Fig. in Appendix A). In contrast, optimizing the standard B EYOL/SimSiam loss _ess with the symmetric linear predictor was unstable, as reported previously [17]. Finally, we confirmed the above findings also hold for a = (cf. Eq. (1)) as prescribed by DirectCopy [18] (see Table 3 in Appendix A). Thus, IsoLoss allows training without an EMA target network.
Table 2: Linear readout validation accuracy in % + stddev over five random seeds. The i denotes crashed runs, known to occur with symmetric predictors like DirectPred [17]. Starred values were taken from the Solo-learn library [32].
Model
EMA
CIFAR-10
CIFAR-100
STL-10
TinylmageNet
BYOL
Yes
92.6*
70.5*
91.7 + 0.1
38.3 = 1.5
SimSiam
No
90.7 + 0.2
66.3 + 0.4
87.5 + 0.7
39.8 = 0.6
Yes
92.0 0.2
66.6 0.5
88.8 + 0.3
40.1 = 0.5
DirectPred (a 0.5)
No
12.1 = 1.3
1.6 0.6*
10.4 0.1
1.3 + 0.21
Yes
91.5 = 0.2
69.0 0.2
89.0 + 0.3
44.8 0.4
IsoLoss (ours)
No
91.5 + 0.2
64.3 0.3
87.4 + 0.1
40.4 0.4
The above result suggests that IsoLoss promotes the recruitment of small eigenvalues in closed-form predictors. Another factor that has been implicated in suppressing recruitment is weight decay [18]. To probe how weight decay and IsoLoss affect small eigenvalue recruitment, we repeated the above simulations with EMA and different amounts of weight decay. Indeed, we observed less eigenvalue recruitment with increasing weight decay for DirectPred (Appendix A, Fig. 8a), but not for oss loss (Fig, 8b). However, for IsoLoss larger weight decay resulted in lower magnitudes of all eigenvalues. Hence, IsoLoss reduces the impact of weight decay on eigenvalue recruitment.
4 Discussion
We provided a comprehensive analysis of the SSL representational dynamics in the eigenspace of closed-form linear predictor networks (i.e., DirectPred and DirectCopy) for both the Euclidean loss and the more commonly used cosine similarity. Our analysis revealed how asymmetric losses prevent representational and dimensional collapse through implicit variance regularization along orthogonal eigenmodes, thereby formally linking predictor- based SSL with explicit variance regularization approaches [6, 7, 14]. Our work provides theory framework which further complements the growing body of work linking contrastive and non-contrastive SSL [24, 35-38J.
We empirically validated the key predictions of our analysis in linear and nonlinear network models on several datasets, including CIFAR-10/100 STL-10, and TinyImageNe Moreover, we found that the cigenvalues of the predictor network act as learning rate multipliers, causing anisotropic learning dynamics. We derived Euclidean and cosine IsoLosses, which counteract this anisotropy and enable closed-fomt linear predictor methods work without an EMA target network, thereby further consolidating its presumed role in boosting small eigenvalues [17].
To our knowledge, this is the first work to comprehensively characterize asymmetric SSL learning dynamics for the cosine distance metric widely used practice. However, our analysis rests on several assumptions. First, the analytic link through the NTK between gradient descent on parameters and the representational changes is an approximation nonlinear networks. Moreover, we assumed Gaussian i.i.d inputs for proving Theorems and 2. Although these assumptions generally do not

---

hold in nonlinear networks, our analysis qualitatively captures their overall learning behavior and predicts how networks respond changes in the stop-grad placement.
In summary, we have provided a simple theoretical explanation of how asymmetric loss configurations prevent representational collapse in SSL and elucidate their inherent dependence on the placement of the stop-grad operation. We further demonstrated how the eigenspace framework allows crafting new loss functions with distinct impact on the SSL learning dynamics. We provided one specific example of such loss functions, IsoLoss, which equalizes the learning dynamics in the predictor's eigenspace, resulting in faster initial learning and improved stability. In contrast DirectPred, IsoLoss learns stably without an EMA target network. Our work thus lays out an effective framework for analyzing and developing new SSL loss functions.
Acknowledgments and Disclosure of Funding
This project supported by the Swiss National Science Foundation [grant numbers SCEFP3_20298 and TMPFP3_211 __P_____,, by EU's Horizon Europe Research and Innovation Pro- gramme (grant agreement number 101070374) funded through SERI (ref 1131-52302), and the Novartis Research Foundation. The authors declare no competing interests.
References
[1] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo. Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.
[2] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances Neural 2020.
Information Processing Systems,
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[5] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[6] Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing. pages 123 12300-2 PMLR, 2021.
[7] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invaaiance-oooo covaaraiicc regular- ization for self-supervised learning. arXiv preprint arXiv:2105.049000 2021.
[8] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, pages 456 Springer, 2022.
[9] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.
[10] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
10

---

[11] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1XX::1807.03774, 2018.
[12] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
[13] Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34:1 1834-11845, 2021.
[14] Manu Srinath Halvagal and Friedemann Zenke. The combination of Hebbian and predictive plasticity learns invariant object representations in deep sensory networks. bioRxiv, accepte, 2022. doi: 10.1 1.1100022..03.1..4888.. 1101/2022. 03. 10..17..880 484712v2.
URL https://www. wwwpwwwwwww...............ww.ww. wwwpe//wi...oo/i/oeooooo content/10.
[15] Alcksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International Conference on Machine Learning, pages 2015-3024 PMLR, 2021.
[16] Xi Weng, Lei Huang, Lei Zhao, Rao Anwer, Salman H Khan, and Fahad Shahbaz Khan. An investigation into whitening loss for self-superviser learning. Advances in Neural Information Processing Systems, 35:29748- 2276000 2022.
[17] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268-10278. PMLR, 2021.
[18) Xiang Wang, Xinlei Chen, Simon S Du, and Yuandong Tian. Towards demystifying repre- sentation learning with non-contrastive self-supervision. arXiv preprint 11.04:2110.04947, 2021.
[19] Kang-Jun Liu, Masanori Suganuma, and Takayuki Okatani. Bridging the gap from asymmetry tricks decorrelation principles in non-contrastive supersued learning. Advances in Neural Information Processing Systems, 15:19824-19835 2022.
[20] Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self superviie learning via a unified gradient framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14431-14440 2022.
[21] Pierre H Richemond, Allison Tam, Yunhao Tang, Florian Strub, Bilal Piot, and Felix Hill. The edge of orthogonality: A simple view of what makes byol tick. arXiv preprint arXiv:2302.0482 048177 2023.
[22] Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, al. Byol works even without batch statistics. arXiv preprint arXiv:2010. 10241, 2020.
[23] Xiao Wang, Haoqi Fan, Yuandong Tian, Daisuke Kihara, and Xinlei Chen, On the importance of asymmetry for siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16570-16579, 2022.
[24] Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung Pham, Chang D Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. arXiv preprint arXiv:2203. 16262, 2022.
[25] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl:: Convergence and generalization in neural networks. Advances in newral information processing systems, 31, 2018.
[26] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. Advances in Neural Information Processing Syy- tems, 32:8572-8583, 2019. URL https: https:h//t/s:stteeeeeeee http///ppoceen...//////////
wwapiingg..eeuiippe newrins www.iip..cc/ppppe//0000... wwtpccppppppeee///aaaaeae
4111 www. www.:e445552saa-aaarrrrtttt..tee Abstract.htm
11

---

[27] Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning deep linear neural networks. arXiv:1312.612. [cond-mat, q-bio, stat]. December 2013. URL. http:/ttp//ee bectt://aarie /arpivv///xvvv.e//eeeeeee org/ abs//31// 6120.
[28] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:2110.09348, 2021.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images, 2009.
[30] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[31] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 7(7):3, 2015.
[32] Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of self superviedd methods for visual representation learning. Journal Ma- chine Learning Research, 23(56):1-66 2022. htLh:tp://jmll httpp//...g/pppppp///////1111111 html.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[34] James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. arXiv preprint arXiv:2303. 15438, 2023.
[35] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprint arXiv:2206.0233 02574, 2022.
[36] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205.11500 2022.
[37] Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang. Improving self- supervised learning by characterizing idealized representations. Advances in Neural Information Processing Systems, 15:11279-11296 2022.
[38] Ashwini Pokle, Jinjin Tian, Yuchen Li, and Andrej Risteski. Contrasting the landscape of contrastive and non-contrastive learning. arXiv preprint arXiv:2203. 15702, 2022.
12

---

A
Supplementary Material
a
Louc
b
J COS
Alignment
1
1000
Epoch
500
-1
100 200
100
200
Sorted index
Sorted index
Figure 4: Eigenspace alignment between C₂ E.g zz and linear predictor Wp trained with gradient descent. Following [1], we measure eigenspace alignment as the cosine between u, and Wpu, for every cigenvector u, of Cz. (a) Measured alignment for every eigenvector of C₂ ordered by sorted eigenvalue indices over training epochs, when the network is trained with L (b) Same (a) but for training with Ce08
a
b
Lene
c08
A Aa
1
$ 1
0
0
1000
1000
Steps
Steps
Figure 5: Comparison between the dynamics under Euclidean and cosine asymmetric losses for different initializations in network with M 2 output neurons. (a) Observed dynamics of the eigenvalues in the two-neuron toy network under three different initializations. Both eigenvalues always converge to regardless of the initialization. Under different initializations, the two eigenvalues converge to arbitrary, but equal, values.
(b) Same as (a), but for the cosine distance.
Table 3: Linear readout validation accuracy in % + stddev over five random seeds. The t denotes crashed runs, known to occur with symmetric predictors like DirectPred [1].
Model
EMA
CIFAR-I0
CIFAR-100
STL-10
TinylmageNet
Yes
91.3 0.2
68.7 +
89.3 0.2
45.3 0.3
DirectCopy
No
12.1 = 0.6*
1.5 0.5f
10.3 + 0.1*
0.6 0.1
Yes
92.0 + 0.2
66.6 + 0.5
88.8 + 0.3
40.1 + 0.5
DirectPred
0.5
No
12.1 + 1.3
1.6 0.61
10.4 + 0.1
= 0.2
Yes
91.5 0.2
69.3 +
89.1 0.3
45.6
No
91.4 0.2
63.0 - 0.3
0.
38.5
Isol.oss (ours)
Yes
91.5
69.0
89.0
4..8
0.5
No
91.5 0.2
64.3
8.14
4.4
13

---

C nnSG
b
noPred
L
d
onu iso
10a
10-1
1000
100000
1000
1000
Iteration
Iteration
Iteration
Iteration
e
f
C noPPred
g
C
h
Liso
10a
10-1
1000 Iteration
100000
1000 Iteration
1000 Iteration
Iteration
Figure 6: Same as Fig. 2 but with ReLU nonlinearity on the embeddings. We observe learning dynamics qualitatively similar to the linear network.
with EMA
w/o EMA
104
102
10a
10-2
0
500
1000
500
1000
Epoch
Epoch
Figure 7: Eigenvalue dynamics for learning under IsoLoss (Coos) with and without the EMA target network. Removing the EMA results in markedly different dynamics with fewer eigenmodes recruited during training.
14

---

nwd = 4 x 10-4
Nwd = 10
nwd 10-6
10a
10
10-3
103
100
10-3
0
500
1000
500
1000
0
500
1000
Epoch
Epoch
Epoch
Figure 8: Effect of weight decay eigenvalue recruitment for DirectPred and IsoLoss. (a) Evolution of eigenvalues during learning under DirectPred (Loos) with EMA and different amounts weight decay. Decreasing weight decay correlates with the number of eigenvalues recruited during learning. (b) Same as (a) but for IsoLoss (Coo)). All eigenvalues are recruited independent the strength of weight decay. However, the magnitude of the eigenvalues inversely correlates with the magnitude of weight-decay.
15

---

B Proofs
Lemma 1. (Euclidean and cosine loss in the predictor eigenspace) Let Wp. be a linear predictor set according to DirectPred with eigenvalues Ams and the representations expressed in the predictor's eigenbasis. Then the asymmetric Euclidean loss L and cosine loss L can be expressed as:
M
Leue
Am
=(1) m
)G
M
Am cm
(
Leos
(4)
(1) |||SG(
T
Proof. Under DirectPred, the predictor is a symmetric matrix with eigendecomposition Wp UDU Since U is an orthogonal matrix, we also have UU T = I so that we can simplify the losses follows:
L
21 - SGG
DU 2()) (GGUUT=(2)))
-11D2W--G(( - SSG
1 Σ Am 2(1) )
(W) T SG(2(2)
[Wpz(>]I[SG
(z(1) TUDU (G
(SG (G
Am 2m SG
(1)
D (G
where used the fact that U is orthogonal and therefore does change the Euclidean norm. -U z is the representation rotated into the eigenbasis.
16

---

Lemma 2. (Learning dynamics in a rotated basis) Assuming that given loss L is optimized by gradient descent on parameters of a neural network with network outpus 2, a given orthogonal transformation 2 = U z and learning rale d, then the rotated representations 2 evolve according to the dynamics:
d -= &f
dt is the empirical NTK expressed in the rotated basis.
where
Proof. Let 0 be the parameters the neural network. Then we obtain the representational dynamies using the chain rule in the continuous cattinuous-time gradient-flor -floww setting [2]:
dž
de df = Voй (-1)o
V
dt
= Vož (- 2C)
(x, zf
The above is a reiteration of the derivation of Eq. (2) given by Lee et al. [2], with an additional orthogonal transformation on the network outputs.
We proceed by proving the following Lemma which we will use in our proofs of Theorems and 2.
Lemma The NTK for a linear network is invariant under orthogonal transformations of the network oulput.
Proof. We first note that for a linear network, the parameters are just the feedforward weights W. Therefore, for any orthogonal transformation U of the network output:
= f(x) Wa
= V Vw (U))
& T
(18)
where & the Kronecker product resulting from the fact that every input vector component appears in the update once for each output component.
We now study (x, X),, the transformed empirical NTK (cf. Lemma 2). The (M x M) diagonal blocks in the full (M D x [DI) empirical NTK (X X) correspond to single samples and the off-diagonal blocks are cross-terms between samples, where D] denotes the size of the training dataset and M the dimension of the outputs. We develop generic expression for each (M x M) block ,, æ2,) corresponding to the interactions between samples i and as:
θ. x.)) = V 2, V W
(z) @ U
(x) & TUT)
(z. @ U
(m, @ U)
(xTx,) @ (UTU)
(z]zz) @ IM
(19) where we have used the fact that (A T) = A & BT and (A () D)= AC & BD. Here, Ix is the identity matrix of size M. Noting that Eq. (19) is unchanged when U is just the identity matrix completes the proof.
(z. xz) IM-
17

---

Theorem 1. (Representational dynamics under L For a linear network with i.i.d Gaussian inputs learning with L the representational dynamics of each mode m independently follow the gradient of the loss Lo More specifically, the dynamics uncouple and follow M independent differential equations:
dam
(1)
(5(2) 2m
Am 2(1)
(8)
dt
d2m
which, after taking the expectation over augmentations yields the dynamics:
dmm
dt 17 (1 Am) 2m
(9)
Proof. For a linear network with weights W E RMXN, we have from Lemma 3 that the empirical NTK θ(X, X') in the orthogonal eigenbasis is equal to the empirical NTK @ ([''A Xt) in the original
basis. Furthermore from the proof for the lemma (see Eq. (19) above), each (M M) block of the full (M|D) x M (D]) empirical NTK is given by:
( 24) (a. Φ;) IM-
(20)
where Ja E MxM is the identity. Eq. (20) gives the total effective interaction between the samples i and j from the dataset. For high-dimensional inputs æ drawn from an i.i.d standard Gaussian distribution, we have E; 2j s dij by the central limit theorem. Therefore, in the special case of linear network with Gaussian i.i.d inputs, the representational dynamics (Lemma simplify as follows:
d
- XV zf
0 x; V Vz,C Σ (z V
jti
= - x3 22, C 17
Σ
x x3j)
Vz
-
(21)
While the assumption of Gaussian i.i.d inputs is quite restrictive, offer generalizing interpretation here. Specifically, the above argument also holds when the inputs æ are not all mutually orthogonal, but fall into P orthogonal clusters in the input dataset. Then, we would have xx 2t 6p:=p, where p: is the "label" of the cluster corresponding to sample i. P; is the number of all the samples with
(e(a) &,,
the same label then Eq. (21) would simply be scaled to give dt
For brevity, proceed with the simplest Eq. (21) in which every input is orthogonal. For Lowe, the representational gradient v,, L is then given by:
L
D (1) - -2(2) D.
Noting that D, is just a diagonal matrix containing the eigenvalues Am and dropping the sample subscript for notational ease, obtain for the m-th component of Lz,Leu,
deuuc
Am (Am =(1) (2) 9y
m.m
Substituting this result in Eq. (21) gives us Eq. (8), the expression we were looking for. Finally, introducing 2m E 1 = E as the expectation over augmentations, we find that each eigenmode evolves independently in expectation value as:
dêm
=1-m [E(3(2] - Am (.=(((
dt
dt
1)mm (1 - Am 8m
18

---

Theorem 2. (Representational dynamics under ) For a linear network with i.idd Gaussian inputs trained with _ the dynamics follow system of M coupled differential equations:
d dt
11
Σ A (Ae(E kfm
Am 2(1 x() =11)
(10)
I 1312
and reach a regime in which the eigenvalues are comparable in magnitude. In this regime, the expected update over augmentations well upproximated by:
d2m dt
2
Rs 11
E
E
Σ kemm
Au (Ax Am),
(11)
DE
Proof. We can retrace the steps from the proof for Theorem until Eq. (21):
d (1)
VV
dt
Vg, C is vector of dimension M. Ignoring the sample subscript for simplicity, and focusing on the m-th component of V,, C, we get:
Am (1) SG (2)
C
D&
|||| [||SGG(22))
7c
Am. 2m
(2)
Σn 1 (1)) ((2)
X 2(1)
0:l1
|D (1) 22)
DE 12 113111111111
)
Am
112 (1) 1122210) -
Am =(1)
Σ Ao s(2)
D (1) 13 (2)
mm
]
Am
( 212)
Am²(l))
z(1)
Σ Na 212)
|D 1311
Am
Σ
Av (2=()()))2200) - Am
(1) (2))
ID (1)
1/ 12)
2k 2k
d
Oc
1)
dt
=(1) =(1)((1)) ((1) Σ A (Na() Am 2/
D2 (1)
13
proving Eq. (10). Assuming sufficiently small augmentations, 2 and =(? carry the same sign, and the net sign of both terms inside the parenthesis fully determined by Tm = sign ( Hence, we may write:
d
Mm I'm
Σ kmm
(A2(=())) - Am 11
dt
| (1) 11312(221
useful to separate out Ym in this manner because every other term in the expression is now de sign(2m. dém. dt tells us whether én tends to increase decrease
non-negative. Then sign( de in magnitude, as we have argued in the main text.
Asymptotic analysis. To get a handle on how the different eigenvalues influence each other, we consider two important limiting cases. First, we consider the asymptotic regime dominated by one eigenvalue, and show that tends towards more symmetric solution which the gap between different eigenvalues decreases. Second, we derive asymptotic expressions for the near-uniform regime which all eigenvalues are comparable in size and show that this solution tends toward the uniform solution (cf. Eq. (11)).
19

---

To facilitate our analysis, we define each mode's 's relative contribution Xm Eq. (10) taking the expectation value over augmentations:
Em and evaluate
E
dam
= 11Am Σ
(# [
Am Ax
E
2m 2K
2 (1) (2))
2A
dt
I1 (1) 1 (2)
d
28
2m ||
2m &e |
2x
17 17
Σ
Att E
IE
EE Am )k
IE
dt
|| ||3
[|
Neptom
|
||
mpdmemn Ymm
2
E
X
| 13
E [Xm]
Amdx
E
Xm.x
| ||
IE
[xx]
kemm
(22)
In the second equality used the fact that the expectation value taken over augmentations is conditioned on the input sample, which makes them conditionally independent.
One dominant eigenvalue. First, we consider the low-rank regime in which one eigenvalue dominates. Without loss of generality, we assume >
d Vk: + 1. We then have:
X1 1
X r (0<E<1) 4 #
Plugging these values into Eq. (22) gives the following dynamics for the dominant eigenmode:
|
||
E
c2
[
A1
[
dt
|
1 1
Ag
||
||
1)
m
Σ
Agee
[1] 2 B
||
||
= =
Σ Angf
Au OA A1)
||
These updates are always opposite in sign to the representation component, which corresponds to decaying dynamics for the leading cigenmode because 71 da, For all other modes we have:
dt 0.
d Tm
|| | 3
||
Σ
Age E
[E
Amddee? E
TE. [c]
dt
| 1|
Aod [E(m,)}
||
||
+ mmmm Ym
x7 E
E [c] Amdre E
E [1]
|| ||
| 3
||
-mmmmm
A1 Am)
Σ
du(Ae Am)
||
leg [mm1))
||
To Ame E
(A1 Am
ID
dém so that Ym de 0, i.e. the updates have the same sign as the representation component, which corresponds to growth dynamics. In other words: The dominant eigenvalue "pulls all the other eigenvalues up." a form of implicit cooperation between the eigenmodes. We also that the non-dominant eigenmodes increase at rate proportional to e, whereas the dominant eigenmode decreases at slower rate proportional to e Thus, for sensible initializations with at least one large and many small eigenvalues, the modes will tend toward an equilibrium at some non-zero intermediate value, without a dominant mode. Next we study this other limiting case in which all cigenvalues are of similar size.
20

---

Near-uniform regime. To study the dynamics in near-uniform regime, we note that all Xm are of order (1) 2m. whereas the eigenvalues Am are of order ( ). In this setting, the effect of the eigenvalue terms Am on the dynamics is stronger than the Xm terms which are bounded between O and With sufficiently high-dimensional representation, all Xm terms will be centered around 1/v M. Based on these observations, we may make the simplifying assumption that the contributions are all approximately equal, i.e. Xi x for all i. Substituting this value in Eq (22) gives:
d&m
2 1 |
77 Tm E
E [x]
Σ Aepform
du (Ae Am)
(23)
di
Finally, substituting for x, which by assumption are approximately equal:
2m
X
Xm
and absorbing back the sign from Yim we obtain the approximate dynamics in Eq (11):
dêm
2m
20 1mm E
m
E
Σ
d
(2x
Am)
dt
|
||
21

---

C Derivation of idealized learning dynamics for different loss variations
C.1 Removing the stop-grad from the Euclidean loss Le
Omitting the stop-grad operator from L gives:
BoSG
>/IWpz(1) - =2(2)11
1 M
(mm= 2(1)
2
=m
Tracing the steps to prove Theorem and assuming Gaussian i.i.d inputs for a linear network, we write:
a P&m
Ö noSG
@Lass
d2l1)
0212)
Am 2(1)
2(22)
Am (mm=
s(1)
=((2)
(mm
((1)
2(2)
(Am - 1)
dåm
eL
+
n
dt
a2m
-1 (Am B [2(0011 - (Am 1)
=-n (1 >mm)² 2m
which results in decaying representations and thus collapse.
C.2
Removing the stop-grad from the cosine loss Coos
Following the same arguments as above, omitting the stop-grad operator from cos gives:
(1) (2)
[|p2()>|||=22]]
AmAx
((1) c11) 2k 2k
+ ( Au 2(2) (2)
21
F2m
1 1101111111111
+ Am
(2 (=()))a
Xmm (=(2)))22211
[ 11
so that, when taking the expectation value over augmentations, the dynamics follow:
dêm dt
-nE
Fam
||
||
mm Yen
Do IE
X
|
E [Xm]] - Am
XmxX
|D
IE [xx]]
kmmm
||
||*|||
mm Tm
Σ k
Auc
X
1
IE
E
Xk
|>
IE [x
xx]l==]]]]]
|
||
+ +7722 7m
12 E
Xm
3
D
E
|
E
Xm
[D
E
[ =1]]]]
22

---

In the asymptotic regime with dominant eigenvalue Å1, we get the dynamics:
daP df
||
||
Σ Ak
[ID=|1
1
|| |
|
+ 1 7
Σ
dk
Am
2
IE
eE
IE
[II
22 3
||
| ||
||
||
+ -m)iin
11
EI
18 [1) [1]
||
||
|| |
=0 nppn1
IE
||
||3
||
d2mØ1
||
||
= 17 7m
d
Ame
dt
|1
k
||
||
Am +
Am Al
AiiE
AmeE
|
||D2||
||
||
T'm um +
A
AmndRe
E
[1] [1|
DD
3
||
22
kg(m.))
||
||
+ 7m Tm A1
A1
<E
E ||| 11
||
||
||
||
1 ||
||
+ nm Ym
A
IE
é'
E [||
|1
|| |
2 7 7m A7
EE
E
||
||
Thus, all eigenmodes diverge because 7/m de
dem. 0.
Similarly, find divergent dynamics when starting in the near-uniform regime:
d
|
mm im
E
X
d (A Am))
D&
||
+ n Ym
E
[1
||
mm Tm
X
ID
IE
[x3
Σ
Ax
||
1|
Rd
17 M
x
Σ
X2
||
selecting the terms with the highest power in the eigenvalues.
Thus, omission of stop-grad precludes successful representation learning for both the Euclidean and the cosine loss, but due to different mechanisms. Euclidean loss yields collapse, whereas the cosine loss succumbs to run-away activity.
C.3 Removing the predictor from the Euclidean loss Lucc
To analyze the representational dynamics in the absence of the predictor network, consider Loue
ceue
ブタ -1 (2))
M
Σ
(1())
SG(
2
23

---

The dynamics resulting from this loss function are special case of the dynamics derived in Theorem 1 with all the eigenvalues equal one (Ag 1). particular Eq. (8) becomes:
d
a Pred (t)
[=(2) mm mm
=(1)
7
7)
di
02m
which evaluates to 0 under expectation over augmentations. Hence there is no learning without the predictor.
C.4 Removing the predictor from the cosine loss Leos
Similarly, when we remove the predictor from Cess it yields:
2m SSSSG (
[11)
CCUS noPred
|||SG(
so that Eq. (10) becomes:
d
7
Σ kytt
2 (3 (2)))))1))))
= 21))) (2)
(24)
dt
(2)
Here, the near-uniform approximation (Eq. (11) of ignoring the differences in X between different eigenmodes is not valid. This is because the X-terms are longer present, and the effects of the x-terms on the dynamics cannot be treated as negligible. In particular, setting Wp = / in the dynamics derived in Theorem 2 would yield zero dynamics. However taking the expectation of Eq. (24) over augmentations yields the non-zero dynamics:
dgm dt
Σ A ma
E
2m
EE
2m
2k
(25)
||
||
1
|
which consists of terms of order O
(11) and ((1) in 2m. hinting at slower dynamies compared 3pm
to Eq. (10). To study these granular effects, we would need to explicitly model the effect of the augmentations, for which we not have a good statistical model. In lieu of deriving these dynamics analytically, we make an observation which restricts the possible dynamical behavior. Specifically, the sum of the eigenvalues remains constant throughout training. To show this, we begin by writing out the expression for the derivative over time of the sum of all the eigenvalues:
d
Σ Am
Σ
ddm dt
Σ
d
Edata
[
dt
dt
d
Edata
dt
2m d.Em
EEaaaa
dt
We can derive the term inside the expectation by adding up the dynamics given by Eq. (25) for all the different eigenmodes:
a(1 (1)
(2(0)22(0)0)
(2 (2)
mm dt
|| (2)
22)
A
d=(
(1)
"
Σ Σ
(2(1)2 (2()))2(1)) mm
(2)
m
dt
|| 1313 (2)
Easta
Eang
Σ TD
(() m
( Edata
Σ T7C
2m
d&m
0
dt
dt
proving that d dynamics where all eigenvalues go to zero as well as diverging dynamics where least one eigenvalue diverges.
de Σm Am O. i.e, the sum of the eigenvalues is conserved. This precludes collapsing
24

---

C.5 Isotropic losses for equalized convergence rates
In Expressions (9) and (11) we see that the overall learning dynamics have quadratic dependence on the eigenvalues with a root near collapsed solutions, which causes these modes to learn slower. We reasoned that this anisotropy could be detrimental for learning. To address this issue, we sought to derive alternative loss functions that encourage isotropic learning dynamics for all modes.
C.5.1 Euclidean IsoLoss.
We start by deriving IsoLoss function for the Euclidean case L To avoid the unwanted quadratic dependence, we first note that we would like to arrive at the following expression for the dynamics:
dam dt 7 (1 Am) Em
By recalling the Euclidean loss and corresponding dynamics:
M
+ d&m
CUC
=(0) m - SG Am
= 17 (1 - Am) 8m
dt
771
we note that the leading Am term has no influence on the overall sign of the dynamics, and is introduced by the second step in the chain rule:
dL
(mm - 2(2)
21) 10n=0)))000
Based on this realization we see that this second step needs to be modified. To that end, we start with the desired derivative:
OC 02l11
a
(Am - =((2)
(a(1) =2(2)
aem
and see that several loss funetions are possible. The one we have reported in Eq. (15) we derived by applying an appropriate stop-grad while integrating:
dL
a
(2(1) + xm))()) - =((2
2(0))
(=(1)
=(2))
02(1)
to give:
Ceue
(00) m
SG
5(1) Am
2(1))22
Another alternative loss with the same desired isotropic learning dynamics, but using different placement of the stop-gradient operators, is given by:
M
reuLc 180
Σ
SG (Am 2(2)
=(2)
(009)
SGG (G(())
C.5.2 Cosine Similarity IsoLoss.
Since most practical SSL approaches rely on cosine similarity, which suffers from similar anisotropy of the learning dynamics, we sought to find IsoLosses this setting, With the same goal as above, we would like to arrive at the dynamics:
dåm
(2)
Σx A
(2) 2
.(1)
7)
Am
dt
21
starting from the cosine loss and corresponding dynamics:
Am 2m (
al))
CODS
(26)
D || (2(2))
dêm
17
Am x(2
17
Σx 2x
(1) (2)
2/
x11
s(1)
(27)
dt
(1) (1 11
| 131
25

---

The IsoLoss in this case can be derived by noting how Am arises in each of the two terms in Eq. (27), and engineering an alternative loss function corresponding to each term separately.
the first term, Am arises from the partial derivative the numerator Am 2m SG (=M)) in the original loss (Eq. (26)). This can be remediated by using a(1) (G as the numerator instead.
In the second term Eq. (27), X arises from the partial derivative of [ V (2 in the denominator. We can reduce X to Am by instead taking the partial derivative of (1
V E 2m 2m))2
Putting these insights together, we arrive at the desired partial derivative:
a (1) (()) 2mm 2m
Σx A (2)
am (2
0:
|
19
IIDE
(2
ash
-1
0(2 (1) (2)
Σx
A4 2/ (1) (2))
a
/1 |
|| (110)1)
aam
|
(2) 131 (2)
2
asm
and the integrated Lss in eigenspace:
C -
2(2)
1 + SG
(
|
[
(
||
1381
Rotating all terms back the original space gives the desired IsoLoss for cosine similarity reported (Eq. (17)):
Ciso = - TSG
z 22))
)
1 SG
(Wpx(1)
[|
2
[|
26

---

D Experimental methods
Self-supervised pretraining. We used the CIFAR-IO, CIFAR-I00 [3], STL- [4], and Tinylma- geNet [5] datasets for self- learning with ResNet-18 18 [6] encoder and the SimCLR set of transformations [7]. We also adopted several modifications of ResNet-18 which have been proposed to deal with the low resolution of the images in these datasets [7]. The ResNet modifications comprise using 3 3 convolutional kernels instead of 7 x 7 kernels and skipping the first max- max-pooling operation. The Solo-learn library [8] also provides specialized sets of augmentations that work well for these datasets, which we adopted as well. The configurations we used for each dataset are summarized in Table 4. We used BatchNorm in the backbone and the projector multi-layer perceptron (MLP) in the hidden layer for all methods. For B BYOL, we included BatchNorm also the hidden layer the predictor MLP.
We used a projection dimension of 256 for the projection MLP using one hidden layer with 4096 units, and the same architecture for the nonlinear predictor for the BOLL baseline. For networks using EMA target networks, we used the LARS optimizer with learning rate 1.0 whereas for networks without the EMA, we used stochastic gradient descent with momentum 0.9 and learning rate 0.1. Furthermore, we used a warmup period
epochs for the learning rate followed by a cosine decay schedule and batch size of 256. We also used weight decay 4 x 10-1 for the closed-form predictor models and 10-5 for the nonlinear predictor models. For evaluation, we removed the projection MILP and used the embeddings the pooled output of the ResNet convolutional layers following standard practice. For the EMA, we started with Toase = 0.99 and increased TEMA to with a cosine schedule exactly following the configuration reported in [9]. For DirectPred, we used 0.5 and T 0.998 for the moving average estimate of the correlation matrix updated at every step.
Table 4:
CIFAR-10
(TFAR-100
STL-10
TinylmageNet
Resolution
32 x 32
32 x 32
96 96
64 64
Kernel size
3 x 3 No
3 x
7 x 7 Yes
3 x 3 Yes
First max-pool
No
Blur
No
No
Yes
No
Linear evaluation protocol. We reported the held-out classification accuracy on the test sets for 10/10- and STL-10, and the validation set for TinyImageNer after online training of the gradient-isolated linear classifier on each labeled example in the training set during pretraining.
Compute resources All simulations were run in-house cluster consisting of 5 nodes with V100 NVIDIA GPUs each, one node with 4 A 100 NVIDIA GPUs, and one node with 8 A40 NVIDIA GPUs. Runs on CIFAR- CIFAR-10/100 took about hours each, and the STL-10 10 and Tinylmagenet runs took about 24 hours each.
27

---

References
[1] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-super learning dynamics without contrastive pairs. In International Conference Machine Learning, pages 10268-10278 10278. PMLR, 2021.
[2] Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent. Advances in Neural Information Processing Sys- tems, 32:8572-8583, 2019. URL https: httpa////cpo/oco..../.o.. htt.::44444488881111888888888888 Aat.aaa---btttttt.tcct
[3] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images, 2009.
[4] Adam Coates, Andrew Ng, and Honglak Lee. An analysis single-layer networks unsuper- vised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011
[5] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning. pages 1597-1607. PMLR, 2020.
[8] Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library self-super methods for visual representation learning. Journal of Machine Learning Research, 23(56):1 6, 2022. URL /attp://jmll org/ papers/ /vtrsr/vv3321111 1155. html.
[9] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 1222224444 2020.
28


Reviews:
[{"review_id": "P7Oz5uINzO", "reviewer": "Reviewer_ZNsZ", "summary": "This work analyzes the learning dynamics of non-contrastive SSL approaches such as BYOL and Simsiam. Based on the proposed theory, the authors analyze the how the stop-grad and predictor module affect the learning dynamics. Importantly, the authors design a theoretically inspired loss and gain improvement on classification tasks.", "questions": "1. What does \"Gaussian i.i.d inputs\" in line 133 refer to specifically? To what extent does this setting fit the real-world scenario?\n\nOthers:\nLine 68: Replace z^{(1/2)} with z^{(1)}, z^{(2)} to avoid ambiguity\nLine 146: typo: \"lammata\"\n$\\ell_{iso}$ first appears in Fig 2. and Tab 1., which is before where it is defined.\n\nIf the author solves my concerns, then I would be willing to raise the rating.\n\n[1] https://github.com/vturrisi/solo-learn", "limitations": "See weaknesses", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work provides solid analysis and discussion on the learning dynamics of non-contrastive SSL algorithm, which contributes to our understanding of the algorithm. Especially, they consider the widely used cosine similarity loss, while existing works only discuss Euclidean loss. They also propose an isotropic loss which is inspired by the theory.\n2. The simulation experiments helps understanding the proposed theory.", "weaknesses": "1. Lack of basic introduction to the used techniques, e.g., the neural tangent kernal (NTK).\n\n2. Lack of awareness of dividing the article into paragraphs. The preliminary, theoretical analysis and the simulation experiment are all mixed in chapter 2, which might be confusing for readers.\n\n3. The accuracy for the baseline model (BYOL) is too low. In the well-known SSL repository, solo-learn [1], the top-1 accuracy of BYOL (R-18, 1000ep) is 92.58 on CIFAR-10, and 70.46 on CIFAR-100. However, in this paper the baseline is only 89.4 and 61.1. I am afraid that this experiment cannot justify the effectiveness of the proposed isotropic loss. \n\n4. This paper mainly provides theory to understand how non-contrastive SSL algorithms work (4 pages), and the design of the isotropic loss is inspired by the theory (only half a page). I think the title of this paper is not accurate enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback. It helped us significantly improve our numerical results. Please find below our point-by-point response.\n\n> *“Lack of basic introduction to the used techniques, e.g., the neural tangent kernel (NTK).”*\n\nWe are sorry this was missing in the initial submission. We will restructure the paper and add a dedicated “Background” section with an introduction to the NTK.\n\n> *“The preliminary, theoretical analysis and the simulation experiment are all mixed in chapter 2, which might be confusing for readers.”*\n\nThanks for pointing this out. We will restructure the paper to separate the background, theory, and experiments. Specifically, we will split Section 2 into a “Background” section containing introductions to both DirectPred and the NTK, a “Theoretical results” section, and an “Experimental validation” section. We believe the paper will be clearer after these changes.\n\n> *“However, in this paper the baseline is only 89.4 and 61.1. I am afraid that this experiment cannot justify the effectiveness of the proposed isotropic loss.”*\n\nThanks for this comment and pointing us to the solo-learn repository. We reimplemented all our simulations within this framework. As it contains several optimized choices of hyperparameters and augmentations, we found that it significantly improved the performance of all baselines and our own method (Rebuttal Table 1). While most claims remain justified with these new improved performance values, we can no longer hold the claim that our method outperforms nonlinear predictor methods. We will therefore remove this claim from the abstract and the relevant sections in the paper. IsoLoss still serves as an empirical validation of our theory since it correctly predicts faster initial convergences than DirectPred (see Rebuttal Fig.1d,e). Since IsoLoss was only a small application of our theory in the first place, we agree that it will be best to edit the abstract and article to focus on the analysis following your suggestion.\n\n> *“I think the title of this paper is not accurate enough.”*\n\nThanks for making this point. We agree. and suggest the following new title: “Uncovering implicit variance regularization mechanisms in non-contrastive SSL”.\n\n> *“What does \"Gaussian i.i.d inputs\" in line 133 refer to specifically? To what extent does this setting fit the real-world scenario?”*\n\nApologies for not specifying this properly. We refer to high-dimensional data in which every input component is drawn from a standard normal distribution. In the proof for Theorem 1, we further made the argument that the less restrictive assumption of input data distributed among a finite set of mutually near-orthogonal clusters would also suffice. We will move this generalizing argument to the main text. While these assumptions are violated by real-world data, our numerical simulations show that our theory makes qualitatively accurate predictions on ResNet-18 (Rebuttal Fig. 1) despite the Gaussian input assumption. \n\n> *“Others: Line 68: Replace z^{(1/2)} with z^{(1)}, z^{(2)} to avoid ambiguity Line 146: typo: \"lammata\". $l_{iso}$ first appears in Fig 2. and Tab 1., which is before where it is defined.”*\n\nThanks for reading carefully. We will fix the notation and typos in the revised version of the manuscript."}, {"review_id": "BBJQsWc629", "reviewer": "Reviewer_qE19", "summary": "The learning dynamics of non-contrastive self-supervised learning is an important problem to understand how these methods avoid collapse without using negative samples. In this paper, the authors provide a rigorous analysis of this problem on a simple linear network with Gaussian inputs, especially the difference between Euclidean and cosine loss functions. It is demonstrated that both losses have implicit regularization effects on the variance of representation, and the role of predictor and stop-grad operations are thoroughly investigated. Based on these insights, the authors propose an isotropic loss to equalize the convergence rate and lead to a better performance in various settings.  ", "questions": "1. In Table 1, it is predicted that $L_{noPred}$ converges slower than $L$, however, in Figure 2 there is an eigenvalue in $L$ that converge much slower, there seems to be a discrepancy between theory and experiments.\n\n2. In theorem 1 and 2, the dynamics are taken to be the expectation of original dynamics over the distribution of augmentation. It seems that after taking expectations, it is essentially equivalent to inputting the original images without augmentations, which may lose an essential part of contrastive learning.\n\n3. In Theorem 2, it is assumed that the eigenvalues are of comparable magnitude, is this assumption required only on the initial value, or on the whole trajectory? In the latter case, it seems not appropriate to draw conclusions on the behavior of eigenvalues.", "limitations": "No certain limits.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The theoretical analysis in this paper is novel and clearly improves the understanding of how loss function, stop grad and projector affect the performance of non-contrastive self-supervise learning. Based on the mathematical understanding, the authors propose a new loss function that could beat the baseline method on different datasets, which illustrates the validity and power of the proposed theory in practice.", "weaknesses": "The major weakness of this paper is the strong assumptions that require the network to be linear and the input to be iid Gaussian. For real-world datasets and practical network structures, it is not clear if those insights are still valid.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for enabling a further discussion of the finer points of our theoretical analysis. Please find below our point-by-point response.\n\n> *“For real-world datasets and practical network structures, it is not clear if those insights are still valid.”*\n\nThanks for the question. Our empirical results suggest that our theory makes accurate qualitative predictions that do apply to real world data and deep neural networks such as ResNet-18 (see Fig.3 and eigenvalue dynamics in Rebuttal Fig. 1a,b).\nIn the proof for Theorem 1, we further make the point that the less restrictive assumption of input data distributed among a finite set of mutually near-orthogonal clusters would also suffice. We will move this generalizing argument to the main text.\n\n> *“In Table 1, it is predicted that $L_{noPred}$ converges slower than $L$, however, in Figure 2 there is an eigenvalue in $L$ that converge much slower, there seems to be a discrepancy between theory and experiments.”*\n\nThanks for this feedback. Perhaps it was not clear from the figure, but the x axis for the plots of $L\\_{noPred}$ and $L\\_{noPred}^{euc}$ are on a larger scale than the other plots to appreciate the slower convergence. We realize that this may cause confusion easily. To make it more clear, we will use the same axis-range in the revised version of the manuscript and add the current longer-timescale plot in an inset.\n\n> *“In theorem 1 and 2, the dynamics are taken to be the expectation of original dynamics over the distribution of augmentation. It seems that after taking expectations, it is essentially equivalent to inputting the original images without augmentations, which may lose an essential part of contrastive learning.”*\n\nThanks for raising this important point. Indeed after taking the expectation, we lose the error component arising from the different augmentations. The main objective of our analysis was to study the macroscopic learning dynamics underlying collapse and divergence. These dynamics are dominated by the eigenvalues of the representational covariance matrix, and thus can be studied after taking the expectation, as we have demonstrated. Our theory does not provide a complete description of the microscopic dynamics underlying representation learning. Such a theory would require a powerful statistical model of the augmentations and nonlinear transformations in deep neural networks which are difficult to come by. However, Reviewer qXm9 pointed out that Wang et. al (DirectCopy) have made some exciting headway in this direction and we hope to develop our approach further in future work by integrating similar insights.\n\n\n> *“In Theorem 2, it is assumed that the eigenvalues are of comparable magnitude, is this assumption required only on the initial value, or on the whole trajectory? In the latter case, it seems not appropriate to draw conclusions on the behavior of eigenvalues.”*\n\nThank you for the question. The assumption is not required. This information is a bit hidden, but can be found in the Appendix with the proof of Theorem 2. Specifically, we show that, even from an initial configuration where one eigenvalue is much larger than the others, the dynamics result in all the eigenvalues (including the single large one) becoming more equal over learning, leading to the comparable magnitude configuration. This is further supported by our simulation results (Rebuttal Fig. 1a). We realize that we should have made this point clear in the statement of the Theorem, and we will add this sentence in our revised manuscript."}, {"review_id": "Z1sXwcm8e5", "reviewer": "Reviewer_qXm9", "summary": "This article builds on the theory of non-constrastive (nc) self-supervised learning (SSL), with methods such as BYOL or SimSiam. Contrary to the existing litterature, they study directly the cosine similarity loss used rather than a Euclidian loss on the eigenspace, using NTK dynamics. They show that collapse is avoided in this case with implicit variance reduction. With their analysis, they find an undesirable anisotropy which they fix with a new family of losses that outperforms the state-of-the-art, especially without needing an Exponiential Moving Average (EMA) target network.", "questions": "If would define $\\hat z = U z$ a bit more clearly maybe before Lemma 1.\n\nEquation 3: There is a dimensional issue with the diagonal matrix on the right with the product \"$zD$\" if I am not mistaken. Shouldn't $D$ be on the left?\n\nDefinition 1: I find the different notations often a bit unclear, but most importantly is the sudden usage of the time $t$ index which is not consistent. It is present at line 114 but not at line 111, and appears only on part of the variables. Similarly for lemma 2, where $\\hat z_t$ loses the time index again. The use of bold variables is also often inconsistent.\n\nTable 2: I would have expected the inclusion of DirectCopy and of SimSiam in the comparison.", "limitations": "The authors address adequately the limits of the theory they established, although they do not talk about the effect of EMA and weight decay in their theory or their hypothesis of DirectPred which was established with an Euclidian loss.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The theory provides strong results in the linear framework, explaining the eigenvalues dynamics of both euclidian and cosine losses with and without projectors and stop gradients. The use of the NTK dynamics and the analysis of the cosine similarity is novel to me.\n\nThe findings that the eigenvalues affect learning as a learning rate provide a good justification for the new IsoLosses, which gives state-of-the-art results. In particular, the removal of the EMA and the increased dimensionality of the losses ensures their consistency.", "weaknesses": "One of the main claims of the article is the analysis of the cosine similarity loss. However in that case, if the dynamics are quite different as claimed by the authors, I find the use of the result of Tian et al. that the predictor eigenspace aligns with the one of the correlation matrix of the representations surprising since it was established with a Euclidian loss. I would have hoped atleast a comment on this, if not an analysis of the alignment as done in Tian et al.\n\nI was awaiting an analysis of the eigenvalues similar to Figure 2 for a real network, and not only a comparison with other losses. As such, the analysis of the IsoLoss feels a bit disconnected from the theoretical analysis. Why do the dimensionality increase? Is it because small eigenvalues do not collapse due to their low learning rates ? Do the eigenvalues evolve during training similarly to the linear case, for BYOL/SimSiam and IsoLoss ? (and their euclidian counterparts)\n\nTwo important notions in the theory of NC-SSL are the EMA and the weight decay. Since an important improvement of the method is the removal of the need of the EMA of the target network, I would find natural to mention the effect of the EMA in the dynamic. \nSimilarly, the results of Wang et al. indicate the importance of the role of weight decay to collapse unwanted eigenvalues. I am surprised that the role of weight decay is not mentioned anywhere, especially despite its use in the implementation.\n\nThe formulation of the Isoloss (in particular the cosine similarity one) is very hard to interpret. However, it is logical due to the way it was found.\n\nThis is an interesting theoretical article despite some flaws and questions raised, and I am ready to raise my score if my questions are answered.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your questions and suggestions. It resulted in an exciting new figure showing that our theory does apply to general settings. Please find below our point-by-point response\n\n> *“I find the use of the result of Tian et al. that the predictor eigenspace aligns with the one of the correlation matrix of the representations surprising since it was established with a Euclidian loss. I would have hoped at least a comment on this, if not an analysis of the alignment as done in Tian et al.”*\n\nThanks for making an important point. The predictor eigenspace in fact still aligns with the correlation matrix of representation under the cosine loss. This can be understood with a timescale argument. If the linear predictor is trained until convergence on a faster timescale than the encoder, then the converged predictor is the same regardless of the distance metric used, because both L2 and cosine distances have the same optimal solution for the linear predictor. For a cosine distance metric, an optimal linear reconstructor is still the PCA matrix. We propose to add the following comment in the manuscript where DirectPred is introduced (line 83), as well as an additional Appendix section detailing our reasoning:\n\n*“Although Tian et al derived the optimal predictor in the Euclidean loss case, we show in Appendix that the optimal predictor is the same for the cosine loss”*\n\n> *“Why do the dimensionality increase? Is it because small eigenvalues do not collapse due to their low learning rates ? Do the eigenvalues evolve during training similarly to the linear case, for BYOL/SimSiam and IsoLoss ? (and their euclidean counterparts)”*\n\nYes, we believe the dimensionality increases because IsoLoss recruits more small eigenvalues. We now have numerical support for this hypothesis which we will add in a new Figure in the revised text showing the eigenvalue evolution during training for the different cases (Rebuttal Fig. 1). Many thanks for this question. Looking into this matter revealed quite different eigenvalue dynamics for direct predictor approaches and nonlinear predictor networks like BYOL. This observation raises a host of new questions that we will communicate in the final paper and whose origins we will study in more detail in the future.\n\n> *“Since an important improvement of the method is the removal of the need of the EMA of the target network, I would find natural to mention the effect of the EMA in the dynamic.”*\n\nThanks for raising this point. We should have highlighted it in the submission. It is true that our theory does not directly include the effect of the EMA. However, the need for an EMA in DirectPred (Tian et al) was justified by its “automatic curriculum” effect on small eigenvalues. Since IsoLoss provides an alternative way to accelerate learning of those small eigenvalues, it makes sense that it is robust to removal. We will add this remark in the revised version of the manuscript.\n\n> *“I am surprised that the role of weight decay is not mentioned anywhere, especially despite its use in the implementation.”*\n\nThank you for pointing this out. The work by Wang et. al. indeed is relevant to our discussion on eigenvalue recruitment. However, their work is largely orthogonal to ours since they discuss the threshold set by weight decay on across-augmentation variance, whereas we are primarily concerned with across-data variances. We do not include it in our analysis to avoid complication, but use it in our simulations in order to get good performance. We will include this in our discussion in the updated version. Revisiting this paper also suggested to us an interesting experiment, wherein lower weight decay could increase the number of eigenvalues potentially recruited for all losses and might complement the effect of IsoLoss. We have not looked carefully into this matter, but we do provide some initial plots in Rebuttal Fig. 1c showing the effect of weight decay on the dynamics. We will study this further in future work.\n\n> *“If would define $\\hat{z} = Uz$ a bit more clearly maybe before Lemma 1.”*\n\nThank you for the suggestion. It would indeed add to the clarity of the paper. We will include this change in the final version.\n\n> *“Equation 3: There is a dimensional issue with the diagonal matrix on the right with the product \"$zD$\" if I am not mistaken. Shouldn't $D$ be on the left?”*\n\nThank you for reading carefully and pointing out this mistake. It should indeed be on the left to be consistent with our column-vector convention, and will be corrected. Since D is a diagonal square matrix, its position doesn’t affect the final product and the rest of our analysis.\n\n> *“Definition 1: I find the different notations often a bit unclear, but most importantly is the sudden usage of the time  index which is not consistent. It is present at line 114 but not at line 111, and appears only on part of the variables. Similarly for lemma 2, where $\\hat{z}_t$ loses the time index again. The use of bold variables is also often inconsistent.”*\n\nThank you for your careful feedback. Apologies for the mistakes. Further, we agree that the time index is not consistently used. Specifically, it should always be included as a subscript on the representations z starting from line 111 until line 140, after which we explicitly omit it for clarity. We will also double-check our use of boldsymbol for all vector quantities. These inaccuracies and mistakes will be corrected.\n\n> *“Table 2: I would have expected the inclusion of DirectCopy and of SimSiam in the comparison.”*\n\nThese would be useful inclusions. We have included single runs of these methods in a new table in the Rebuttal PDF, and will update the main text with a full set of runs. "}]

Author rebuttal:
We thank the reviewers for the detailed and insightful feedback, which allowed us to further improve the article. Importantly, we improved our numerical results. The comments further motivated us to reconsider the scope of our article by focusing more on the theory part.  

A common concern raised by the reviewers was whether our theoretical analysis still applies to practical settings with real-world data and large architectures. We provide new plots of the eigenvalue dynamics for learning in ResNet-18 that show empirically that our analysis holds to a surprising extent (Rebuttal Fig. 1a,b). 

One reviewer raised concerns about model performance compared to SOTA implementations in the solo-learn package. To address these concerns, we re-implemented our simulations with solo-learn to take advantage of their optimized hyperparameter choices. This change improved the overall performance of our method to 69.4% on CIFAR-100 and 92.2% on CIFAR-10 (Rebuttal Table 1).
We confirmed that our theory’s qualitative predictions about isotropy remain correct. Specifically, the new implementation also shows that IsoLoss works without EMA unlike DirectPred, which fails to learn without EMA. Further, the learning curves (see Rebuttal Fig. 1d,e) demonstrate IsoLoss’ faster initial convergence as predicted by our theory. 
The new implementation also enabled DirectPred to reach the same performance level as IsoLoss. However, this final accuracy does not match the best performance of BYOL (92.6% on CIFAR-10 and 70.5% on CIFAR-100). Therefore, we cannot defend the claim that IsoLoss outperforms BYOL/nonlinear predictor methods.

To accommodate these new findings, we will moderate the claims in the abstract and relevant passages and focus on the advances in theoretical understanding of the learning dynamics. We will use IsoLoss to demonstrate the merits of the analysis and remove the claim of outperforming nonlinear predictor methods. Despite this change, we believe that by shining a light on the learning dynamics our approach may help design improvements for negative-sample-free SSL. Consequently, we also propose to change the paper title to “Uncovering implicit variance regularization mechanisms in non-contrastive SSL” to reflect the shifted focus toward understanding.  

Meta-review: This paper provides a rigorous analysis of non-contrastive self-supervised learning on a simple linear network with Gaussian inputs, especially the difference between Euclidean and cosine loss functions. It is demonstrated that both losses have implicit regularization effects on the variance of representation, and the role of predictor and stop-grad operations are thoroughly investigated. Based on these insights, the authors propose an isotropic loss to equalize the convergence rate and lead to a better performance in various settings.

All reviewers believe this paper makes valid contributions. The AC agrees and recommends acceptance.
Decision: Accept (poster)