Representation Learning Dynamics of Self-Supervised Models
Authors: Pascal Esser
Satyaki Mukherjee
Debarghya Ghoshdastidar
Venue: Accepted by TMLR ()

Abstract: Self-Supervised Learning (SSL) is an important paradigm for learning representations from unlabelled data, and SSL with neural networks has been highly successful in practice. However current theoretical analysis of SSL is mostly restricted to generalisation error bounds. In contrast, learning dynamics often provide a precise characterisation of the behaviour of neural networks based models but, so far, are mainly known in supervised settings. In this paper, we study the learning dynamics of SSL models, specifically representations obtained by minimising contrastive and non-contrastive losses. We show that a naive extension of the dymanics of multivariate regression to SSL leads to learning trivial scalar representations that demonstrates dimension collapse in SSL. Consequently, we formulate SSL objectives with orthogonality constraints on the weights, and derive the exact (network width independent) learning dynamics of the SSL models trained using gradient descent on the Grassmannian manifold. We also argue that the infinite width approximation of SSL models significantly deviate from the neural tangent kernel approximations of supervised models. We numerically illustrate the validity of our theoretical findings, and discuss how the presented results provide a framework for further theoretical analysis of contrastive and non-contrastive SSL.

Published in Transactions on Machine Learning Research (06/2024)
Representation Learning Dynamics of Self-Supervised Models
Pascal M. Esser*
esser www.liiii.uum...cm de
Technical University of Munich, Germany
Satyaki Mukherjec'
paglosatyaki@mmaal..om com
National University of Singapore, Singapore
Debarghya Ghoshdastidar
ghoshdas@cit.tume
Technical University of Munich, Germany
Reviewed on OpenReview: https: //
// openreveww. net/ forum? id= QXLKnrymE1
Abstract
Self-Supervised Learning (SSL) an important paradigm for learning representations from unlabelled data, and SSL with neural networks has been highly successful in practice. How- current theoretical analysis of SSL is mostly restricted to generalisation error bounds. In contrast, learning dynamics often provide precise characterisation of the behaviour of neural networks based models but, so far, are mainly known in supervised settings. In this paper, we study the learning dynamics of SSL models, specifically representations obtained by minimising contrastive and non-contrastive losses. We show that näive ex- tension of the dymanics of multivariate regression SSL leads to lcarning trivial scalar representations that demonstrates dimension collapse in SSL. Consequently, we formulate SST. objectíves with orthogonality constraints the weights, and derive the exact. (network width independent) learning dynamics of the SSL models trained using gradient descent on the Grassmannian manifold. We also argue that the infinite width approximation of SSL models significantly deviale from the neural langent. kernel approximations of supervised models. We numerically illustrate the validity of our theoretical findings, and discuss how the presented results provide framework for further theoretical analysis of contrastive and non-contrastive SSL.
1 Introduction
A common way to distinguish between learning approaches to categorize them into unsupervised learning, which relics input data consisting of feature vector (z), and supervised lcarning which relics feature vectors and corresponding labels (I, However, in recent years, Self-Supervised Learning (SSL) has been established as an important paradigm between supervised and unsupervised learning as it does not require explicit. labels but relies on implicit knowledge of what makes some samples semantically close to others. Therefore SSL builds on inputs and inter-sample relations (x,x*), where x is often constructed through data-augmenttaions of known to preserve input semantics such as additive noise or horizontal flip for an image Kanazawa al. (2016); Novotny et al. (2018); Gidaris al. (2018). While the idea of SSL is not new Bromley et al. (1993), recent deep SSL models have been highly successful in computer vision Chen et al.
(2020 Caron et al. (2021)) ;s Jing & Tian (2019) natural language processing Misra & Maaten (2020): Devlin ct al.
2019h. speech recognition Steffen et al. 20199: Mohamed et al. (002)). Since the carly works Bromley 1993), methods for SSL have predominantly relied on neural netwsrks however with a strong focus on
et al. model design with only little theoretical backing.
"P. M. Esser and S. Mukherjee equally contributed this work. The work carried while S. Mukherjee was the Technical University Munich.

---

Published in Transactions on Machine Learning Research (06/2024)
While there are theoretical works addressing SSL specific question such as Wen & Li
(021) analyzing the
importance of data augmentation and Pokle et al. (2022) characterizing 'dimension collapse' through the loss landscape, the main focus of the theory literature on SSL has been either on providing generalization
crror bounds for downstream tasks on embeddings obtained by SSL (Arora et al. 2019b: Ge et al. 2023
Bansal al. 2021 Lee et al.
2021 Saunshi el al.) 2021: Tosh el al. 2021] Wei et al. 2021 Bao et al.. 2022
Chen ct al. 2022), or analysing the spectral and isoperimetric properties of data augmentation (Balestricro
& LeCun 2022; Han et al. 2023 Zhuo et al. 2023). The latter approach also result in novel bounds on the generalisation error (HaoChen et al. 2021 Zhai et al. 2023). While generalisation theory remains one of the fundamental tools to characterise the statistical performance, it. has been already established for supervised learning that classical generalisation error bounds do not provide a complete theoretical understanding and can become trivial in the conlext of neural network models (Zhang et al.) 2017 Neyshabur el al., 2017). Therefore a key focus in modern deep learning theory is to understand the learning dynamics of models, often under gradient descent, as they provide a more tractable expression of the problem that can be an essential Lool to undersland the loss landscape and convergence (Fukumizu 1998 [axx al. 2014 Pretorius 2018
et al. 2018), early stopping (Li et al., 2021), linearised (kernel) approximations (Jacot et al.
Du et al.
2019) and, mostly importantly, generalisation and inductive biases (Soudry et al. 2018
Luo et al. 2019
Heckel & Yilmaz 2021).
In this paper, we analyze the learning dynamics of SSL models under contrastive and non-contrastive losses (Arora et al. 220196 Chen et al. 2020), which we show to be signifieantly different from the dynamics of supervised models. This gives a simple and precise characterization of the dynamics that can provide the foundation for future theoretical analysis of SSL models.
Contrastive Learning. Contrastive SSL has its roots in the work of Bromley et al. learning based contrastive SSL show great empirical success in computer vision [Chen et al. 2020 Caron
19933.
Recent deep
et al. 2021 Jing & Tian 2019 video data (Fernando et al. 2017 Sermanet et al. 2018), natural language
tasks Misra & Maaten 2020 Devlin et al.) 2019) and speech (Steffen et al. 2019 Mohamed et al. 2022).
In general a rastivetive loss is defined by considering an anchor image, x E Rd, positive samples (a+ C Rd
generated using data augmentation techniques as well as independent negative samples {x-} C Rd. The hcuristic goal is to align the anchor more with the positive samples than the negative ones, which is rooted in the idea of maximizing mutual information between similar samples of the data. In this work, we consider
simple contrastive loss minimisat ion problem along the lines of Arora et al. (2019b), assuming exactly one positive sample x;; and one negative sample x; for each anchor x4
min
u (xx:) (u(0,)- u u(x;)),
(1)
where u= [u (o..)))....(.0))))) Rd + R is the embedding function, parameterized by O.
The objective of this paper to derive the evolution dynamics of the learned embedding u = u(.0)) under gradient flow for the contrastive [ More specifically we show the following with proofs provided in the supplementary material:
Contributions.
We express the learning dynamics contrastive learning and show that, the evolution dynamics is same across dimensions. This explains why SSL is naturally prone to dimension collapse.
Assuming a 2-layer linear network, we show that dimension collapse cannot be avoided by adding standard Frobenius norm regularisation, but by adding orthogonality or L2 norm constraints.
We further show that at initialization, the dynamics of 2-layer network with nonlincar activation is close to their linear, width independent counterparts (Theorem D. We also provide empirical evidence that the evolution of the infinite width non-linear nerworks are close to their linear coun- terparts, under certain conditions on the non- lineeiiy (that hold for tanh).
It. straightforward to extend our analysis to multiple positive and negative samples, but the expressions become cumber- some, without providing additional insights.
2

---

Published in Transactions on Machine Learning Research (06/2024)
Going beyond the 2-layer setting, derive the learning dynamics of SSL for deep linear networks, under orthogonality constraints (Theorem 2 We further show the convergence of the learning dynamics for the one dimensional embeddings 12=
We numerically show, on the MNIST dataset, that our derived SSL learning dynamics can be solved significantly faster than training nonlinear networks, and yet provide comparable accuracy on downstream tasks.
Finally we show that the presented results extend to simple non-contrastive losses as well.
Related works. Our focus is on the evolution of the learned representations, and hence, considerably different from the aforementioned literature on generalisation theory and spectral analysis of SSL. From an optimisation perspective, Liu et al. (2023) derive the loss landscape of contrastive SSL with linear models, u(xx = Wx, under InfoNCE loss van den Oord et al. (2018). Although the contrastive loss in (1) seems simpler than InfoNCE, they are structurally similar under linear models (Liu el al. 2023 see Equs. 4 6). Similarly [boontar et al.] (2021) maximizes the variability of the embeddings by decorrelating the components of the embeddings vectors. Training dynamics for contrastive SSL with linear shallow models (with only one hidden layer, u(x))= Wx) have been derived in Simon el al. (2023) under simplified Barlow twins loss and deep linear models have been partially investigated by Tian [2022), who show an equivalence with principal component analysis, and by Jing et al. (2022), who establish that dimension collapse occurs for over-pan porr-paammttisedd linear contrastive models. Theorem 2 provides a more precise characterisation and convergence criterion of the evolution dynamics than previous works. Furthermore, none of prior works consider non-linear models or orthogonality constraints as studied in this work.
We also distinguish our contributions (and discussions on neural tangent kernel connections) with the kernel equivalents of SSL. studied in Kiani et al. (2022); Johnson et. al. (2023); ; Shah et al. (2022) Cabannes ct al. 2023). While Shah ct al. (2022); Cabannes et al. (2023) specifically pose SSL objectives using kernel models, Kiani et al. (2022); Johnson el al. (2023) show that contrastive SSL objectives induce specific kernels. Importantly, these works neither study the learning dynamics nor consider the neural tangent kernel regime.
Towards analysing deep, non-linear networks. Analyzing doep, non-lincar networks presents a complex challenge and no unilying approach has been established even in the more studied supervised setting. Existing works usually consider one (or several) of the following assumptions: (a) linear networks. This allows for
exact characterization of the network (Saxe et a..) 2014 Ziyin et al. 2022 Basu et al. 2019 however the considered proof techniques do not extend to the non non-l linear setting. (b) strong data. / initialization
assumptions. Taacht et al. (2018); Mei & Montanari (022)) are able to derive exact solutions in the non- linear setting but the proof structure breaks down the assumptions (which might not be used in practice) are violated. (c) strong architecture assumptions. [Jacot et al. (2018): Arora al. (2019b) derive dynamics for deep, non-linear networks, however need strong assumptions on the initialization and the (infinite) width of the network. How exactly the behaviour of finite and infinite wide networks relate is still an open question.
In this work we aim to analyze settings that are relevant in practice but still allow for exact theoretical analysis. We address the challenge of analyzing non-linearities and depth by first showing that linear and non-lincar networks are close in Theorem | and secondly by deriving dynamics for deep linear networks in Theorem 2 In addition our results only have mild initialisation (orthogonality) and data assumptions.
Notation. Let In, be an X n identity matrix. For a matrix let F| and ||A|]| be the standard frobenious norm and the L2-operator norm respectively. The machine output is denoted by u(-). While u is time dependent and should be more accurately denoted as u, we suppress the subscript where obvious. For any time dependent function, for instance u, denote û to be its time derivative i.e. du, our non-linear activation function and we abuse notation to also denote its co-ordinate-wise application on a vector by (()) (..)) is used to denote the standard dot product.
de & is used to denote
2 Learning Dynamics of Regression and its Näive Extension to SSL
In the context of regression, Jacot et al. (2088) show that the evolution dynamics of (infinite width) neural networks, trained using gradient descent under a squared loss, is equivalent to that of specific kernel machines,
3

---

Published in Transactions on Machine Learning Research (06/2024)
known the neural tangent kernels (NTK). The analysis has been extended to wide range of models, including convolutional norks (Arora et al. 2019a), recurrent networks (Alemohammad et al. 2021) overparametrised autoencoders Nguyen et al. 2021), graph neural networks (Du et al., 2019 Sabanayagam ict al. 2022) among others. However, these works are mostly restricted to squared losses, with few results for margin loss (Chen et al. 2021), but derivation of such kernel machines are still open for contrastive or non-contrastive losses, or broadly, the context of SSL. To illustrate the differences between regression and SSL, we outline the learning dynamics of multivariate regression with squared loss, and discuss how a näive extension to SSL is inadequate.
2.1 Learning Dynamics of Multivariate Regression
Given training feature matrix X [. xx]" E Rexd and corresponding z-dimensional labels Y == [y,,... yn yn]" E R consider the regression problem of learning a noural network function u(x) [un (1r, θ) ... (2 O)]T, parameterized by 6, by minimising the squared loss function (()) -ー= ll((zz) wll.. Under gradient flow, the evolution dynamics of the parameter during training is θ VeL and, consequently, the evolution the L-th component of network output u(x), for any input z, follows the differential equation
(x))) = (V)u(=),)) = ΣΣ (Vou(z), (eu)( 9(.))
(2)
While the above dynamics apparently involve interaction between the different dimensions of the output u(xx) through (V)u(x)) (x;), is easy to observe that this interaction does not contribute to the dynamics of linear or kernel models. We formalise this in the following lemma.
Lemma 1 (No interaction across output dimensions). Let Rd - RR be either linear model u(x)= = Oz, or a kernel machine u(x) Oυ(x), where vo corresponds to the implicit feature map of a kernel k, that is, b(x,x') = (b(x), (xx'))
Then in the infinite width limit (h + xo) the inner products belween the gradients are given by
{ 0 TTx
l + j j (lineor case), if l j (kernel case).
(Veu(()), Vou,
k(x,x')
For infinite width neural networks, whose weights are randomly initialised with appropriate scaling, Jacot let al. (2018) show that at, initialisation, Lemma holds with being the neural tangent kernel. Approxi- mations for wide neural networks further imply the kernel remains same during training (Liu al. 2020
and Lemma. continues to hold through training.
Remark (Multivariate regression independent univariate regressions). A consequence of Lemma is that the learning dynamics simplifies to
i=(x)= =
(euu()), Vou(z=) (u((x;) Vr,)),
that is, each component of the output uy evolves independently from other Uj.J Fl. Hence, one may solve a z-variale squared regression problem as z independent univariate problems. We discuss below that similar phenomenon is true SSL dynamics with disnstrous consequences.
2.2 Dynamics of Näive SSL has Trivial Solution
We now present the learning dynamics of SSL with contrastive losses function .dd + R is paramcterised by θ the gradient of the loss L()) Di u(x;)) (u(x,) =(xx))
Assuming that the network
is
Vol(O)= - ΣΣ
ug(x;) (ou((at) + u,(x;t) Vouy(x)) - uz(x:) (ous(a7) - -1((5) Vous(I;)
i=1 1=1
4

---

Published in Transactions on Machine Learning Research (06/2024)
Hence, under gradient descent é -Ve,, the evolution of each component of u(z), given by ú((x)
(Vou(x)) 6) is expressed by
(x)x)= = Vou((), (ou)(I.)) uy(a,) + Vou uy
1=1
(Vou(()), (our(I:)) my(x; (Veu(r), (ou,(()) us (z())
(3)
We note Lemma [ depends only on the model and not the loss function, and hence, it is applicable for the SSL dynamics in (3). However, there are no multivariate training labels y E R in SSL (i.e. y = 0) that can drive the dynamics of the different components M1y...; U2 in different directions, which leads to dimension collapse.
Proposition 1 (Dimension collapse in SSL dynamcis). Under the conditions of Lemma 4 every component of the network oulpul Rd - R² has identical dynamics. As consequence, the oulpul collapses to one dimension at convergence. For linear model, u(x) = Or, the dynamics of u(x) is given by
(xx) = (x (u)l)tt - (((x)) + (TTxt at - -x xxxx )u(()
for controstive model. For kernel models, the dynamcis is similarly obtained by replacing each xf x' by k(z,z".
By the extension of Lemma I to neural network and NTK dynamics, one can conclude that Proposition and dimension collapse also happen for wide neural networks, when trained for the contrastive loss
Remark 2 (SSL dynamics for other losses). One may argue that the above dimension collapse is consequence of loss definition in ( and may not erist for other losses. We note that Liu et al. (2023) analyse contrastive learning with linear model under InfoNCE, and the simplified loss closely resembles I. which implies decoupling of output dimensions (and hence, dimension collapse) would also happen for InfoNCE. The same argument also holds for non-construstive loss in Chen al. (2020). However, for the spectral contrastive loss of HaoChen et al.) (2021), the output dimensions remain coupled in the SSL dynamics due to eristing interactions u(x:) u(x;) on the braining dala.
Remark 3 (Projections cannot overcome dimension collapse). Jing et al. 2022) propose to project the representation learned by SSL model into much smaller dimension, and show that fixed (non trainable) projectors may suffice. For a linear model, this implies u(x) AOz, where A E Bex=, r & z is fized. It is struightforward to adapt the dynamics and Proposition r components of u(x) have identical learning dynamics, and hence, collapse at convergence.
7 to this case, and observe that for any > 1, all the
3 SSL with (Orthogonality) Constraints
For this section, we assume that the SSL model u Rd + RR corresponds to 2-layer neural network of the form
u(x) = W.J ((11)) RR
where h is the size of the hidden layer and θ = (W1, W:T) are trainable matrices. Whenever needed, we use R for the output to emphasize the nonlinear activation o, and contrast it with 2-layer linear network (x(x) = W, Wi.. The optimization problem can now be restated as a trace minimization problem which has been previously observed Balestriero et al. (2023 function as
Assume the linear setting, we can write our loss
c = Σ Tr [ Wi2 (I xx) W.T Tr (W: Wnöw, W:) Tr WWW.W.CWIIN Tw)
(4)
5

---

Published in Transactions on Machine Learning Research (06/2024)
with
Č 2
and Č= Σ 2 (z
(5)
Furthermore 1 can easily be extended to the p positive and q negative sample setting where we then obtain C=E; Σ ( (x)) =1 (1,)( (1z))
Based on the discussion in the previous section, it natural to ask how can the SSL problem be rephrased to avoid dimension collapse. An obvious approach
add regularisation or constraints (Bardes et al. 2021 Ermolov al., 2021 Caron et al. 2020). The most obvious regularisation or constraint on W1, W2 is entry-wise, such as Frobenius norm. While there has been little study on various regularisations in SSL literature, plethora of variants for Frobenius
be found for autoencoders, such sum-regularsiation, |Will + [Walep or product regularisation | Will (Kunin et al. 2019).
regularisations
It is known in the optimisation literature that regularised loss minimisation can be equivalently expressed as constrained optimisation problems. In this paper, we use the latter formulation for convenience of the subsequent analysis. The following result shows that Frobenius norm constraints do not prevent the output dimensions from decoupling, and hence, it is still prone to dimension collapse.
Proposition (Frobenius norm constraint does not prevent dimension collapse). Consider linear embedding function (x) W, o(Wix) and let (WW,WW) be given by (1. then the optimisation problem
min L(W,, W2) s.l. [W < e1, Walll 02 W,
with constants 61 and e2 has a global solution u(x) [u(x)) 0... O0 E TR
The above result preciscly shows dimension collapse for lincar networks u even with Frobenius norm con- straints. Intuitively we note that the Frobenius norm is defined as the sum of the norms of each column. Critically it therefore does not have any control over the inner products between the columns and has no control over the rank. As such there is nothing preventing all the columns from being the same. Therefore we consider to constrain the I2-operator norm as na alternative to Frobenius norm constraint. To this end, the following result shows that, for linear nelworks, the operator norm constraint can be realised in multiple equivalent ways.
Proposition (Equivalence of loss under operator norm and orthogonality constraints). Consider a linear embedding function u (x) W2 WIx, and let the loss L(W1, W2) be given by (1) whose general form is given in (4) Let C be defined by 5 and has alleast one negalive eigenvalue. Then there exists a W1, W₂ that is optimal for all following optimization problems:
L(W1, W2)
1. min W..W
||all22 |W
2. min
C(Wi, W2)
8.1.
[[a]|2 $, ||i < 1;
Wi.Wa
3. min L(Wi, W2)
s.l.
[[W W ||2 1
W..Ws,
4. min L(W,, W2) s.l. W2 W2 T., WT W,WW- Taa
W.,Ws,
Avoidance of dimensional collapse is also heuristically evident in the orthogonality constraint W W2 Iz, W.W. W, = IJ, which we focus on in the subsequent sections. In particular we observe from the proof of Prop 3 that this regularization extracts the eigenvectors of corresponding to its "most-negative" eigenvalues Example 1 (SSL dynamics on half moons). We numerically illustrate the importance of constroints in SSL. We consider contrastive setting with the loss in (T) and u(x)) W.J tanh(W,,) (z) for the dataset shown in Figure la where x= an independent sample from the dataset and x x+e where ~ NO, 0.1I). Let us now compore the dynamics of L. (no constraints) and Lorth, the scaling loss that corresponds to orthogonality

---

Published in Transactions on Machine Learning Research (06/2024)
Data Set
Loss Curve
With Orth. Constraints
Constraints
-100
-200
-300
200
400
Epoch
(a) Dataset and loss. (left) Illustration of the dataset in R.. The considered test point is marked with the blue cross.
(right) Loss curve (mean over 100 initializations) for the network with and without orthogonal constraint.
At initialization
No Constraints
With Orth. Constraints
x
1500
Wi Wir+ +++
1000
0.0
500
-0.5
500
-1.0
1000
-1.5
-1000
1000
2000
-1
22
22
(b) Embedding into R². (left) Embedding initialization. (middle) Embedding for the network without constraints after 30 epochs. (right) Embedding for the network with orthogonal constraints after 500 epochs.
Figure 1: Illustration of dimension collapse. We consider the Dataset in Figure la and plot the loss for constraint and unconstrained models. In Figure lb we furthermore plot the embedding at initialization and during training for both models.
constraints, and present the results in Figure 1 We firstly observe in Figure la that the unconstruined loss does not converge. Secondly considering the embeddings as plotted in Figure lb we observe dimension collapse for the unconstmined loss function (middle) but not for the one with orthogonal constraints (right).
3.1 Non-Linear SSL Models are Almost Linear
While the above discussion pertains only linear models, now show that the network, with nonlinear activation & and orthognality constraints,
"ig (z) W. ((W1)) s.t. W, W2 = I,, W. W1 = Ml..
& is almost linear. For this discussion, we explicitly mention the time dependence as subscript "(y We begin by arguing theoretically that in the infinite width limit at initialization there is very little difference between the output of the non-linear machine u and that of its linear counterpart wfo-
Theorem 1 (Comparison of Linear and Non-linear Network). Recall that use provides the output of the machine at time t and therefore consider the linear and non-linear setting at inétialization as
(0) = W. WA (0)
s.t. W, W, = I,, W. W = Lai
(6)
WT (Wiz)
s.t. W, W2 = L., W W1 = La.

---

Published in Transactions on Machine Learning Research (06/2024)
Tanh
RelU
Sigmoid
10
100
1000
0.0
80
80
80
Epoch
Epoch
Epoch
Figure 2: Difference between the non-linear output and the linear output under various conditions on the activation function. Change of the difference while training for hidden layer size 10 1000.
Let (6-) be un activation function, such that (00) = 0, d'(0) = 1, and 10" S c. Then at inilialization as uniformly rondom orthogonal matrices
[ // s Ke Ke||=|| di log' h
where universal constant Φ, d is the feature dimension and h the width of the hidden layer.
We furthermore conjecture that the same behaviour holds during evolution.
Conjecture 1 (Evolution of Non-linear Networks). Consider the setup of Theorem with the linear (u(s)) and non-linear machine (1())) as defined in and an optimization of the general
min Tr (MT (M() s.t. W W2 T1, W, W1 Id.
W,W;
Again assume is an activation function, such that (00) 0 and (0) 1. Then
| - utoo| + Vt as + xc.
Numerical justification of the above conjecture is presented in the following section.
3.2 Numerical Evaluation
We now illustrate the findings of of Theorem 1 and Conjecture D numerically For evaluation we use the following experimental setup: We train a network with contrastive loss as defined in ① using gradient descent with learning rate 0.01 for 100 epochs and hidden layer size from to 1000. We consider the following three loss functions: (1) sigmoid, (2) ReLU (ø(x) max(x,0}) and (3) tanh. The results are shown in Figure 2 where the plot shows the average
10 initializations. We note that tanh fulfills the conditions on & and we see that with increasing layer size the difference between linear and non-linear goes to zero. While ReLU only fulfills ((0) = 0 the overall picture still is consistent with tanh but with slower convergence. Finally the results on sigmoid (which has a linear drift consistent with its value at 0) indicate that the conditions on ø are necessary as we observe the opposite picture: with increased layer width the difference between linear and non-lincar increases.
This overall picture (with increasing width linear and non-linear models get closer to each other) is well established in the NTK setting for supervised models. However such results are stated in terms of squared loss while the above findings show that the same behaviour for the SSL setting and under orthogonality constraints.
2This last assumption also weakened say that d continuous O. See the proof of the theorem for details.

---

Published in Transactions on Machine Learning Research (06/2024)
4 Learning Dynamics of Deep Linear SSL Models
Having showed that the non-linear dynamics are close to the linear ones we now derive the learning dynamics numerically evaluate the theoretical
and discuss the evolution of the differential equation. Furthermore results and show that the dynamics coincide with learning the general loss function under gradient decent. Importantly we now consider deep neural networks of depth I such that we now analyze the following trace minimization problem.
Definition 1 (General Loss Function). Consider the following loss function for layer deep linear network
min Tr (W, Wi-1 ... WiCW. .......... ...W. s.L. W.W. =1, ViE (1.
(7)
Wi. +E[[
where II is an upproprialely sized idenlily matria, Wi E Rixd, Wi E R W; E Rixh vi E 1,...I-1 are the trainable weight matrices. Ce &.dx is symmetric, data dependent matria.
With the general optimization problem set up we can analyze 7 by deriving the dynamics under orthogo- nality constraints on the weights, which constitutes gradient descent on the Grassmannian manifold. While orthogonality constraints easy to initialize the main mathematical complexity arises from ensuring that the constraint is preserved over time. Following Lai et al. 2020 we do so by ensuring that the gradients li in the tangent bundle of orthogonal matrices.
4.1 Theoretical Analysis
In the following we present the dynamics in Theorem 2 followed by the analysis of the evolution of the dynamics in Theorem 3
Theorem 2 (Learning Dynamics in the Deep Lincar Setting). Assume the general linear trace minimization problem stated in 7 and assume that C' E Raxd is symmetric, data dependent matrices, such that C VAVT with V := [u....., val. Then with q = [u'(.,).. u [va)] where represents the machine function i.e. (z) w, Wi-1 WII, the learning dynamics of q, the machine outputs are given by
i = 22 Agg 9 9q Tg_
(8)
Similar differential equations to have been analysed in Yan al. 1994 and Fukumizu 19988.. The typical way to find stable solutions to such equations involve converting it to differential equation on gyT. This gives us matrix riccati type equation. For brevity's sake we write below complete solution when ==1. 1.
Evolution of the differential equation. While the above differential equation doesn't seem to have a simple closed form, few critical observations can still be made about it particularly about what this differential equation converges to. As observed in Figure 3 (right), independent of initialisation we converge to either of two points. In the following formalise this observation.
Theorem 3 (Evolution of learning dynamics in for z = 1). Let then our update rule simplifies to
i = - q. A) - (II - q9 Ag.
2
We can distinguish two cases:
Assume all the eigenvalues of A are strictly positive then g converges to 0 with probability 1.
Assume there is alleast negalive eigenvaaue of A, then g becomes the smallest eigenvector, V1.
9

---

Published in Transactions on Machine Learning Research (06/2024)
Loss
Dynamics of the Machine Output
0-
0.4
Comparison
-100
SGD, h=
0.2
Convergence Points
-200
SGD. h- 100
Initialisation
5GD, 1000
0.0
SGDD
-300
Differential Equation
Differential Equation
-400
0
100
200
300
400
500
1.0
0.5
0.0
0.5
1.0
utv;)
Figure 3: Comparison of gradient decent optimization and differential equation. (left) comparison of the loss function (right) comparison of the outputs.
The requirement of negative cigenvalues of C' for non-trivial convergence might be surprising however we can observe this when considering C' in expectation. Let us assume C is constructed by (5) and note that E[Ć] E Σx 24 (.. (x= =) While this already gives a heuristic of what is going on, for some more precise mathematical calculations, can specialise to the situation where at is given by an independent sample and 2 is given by adding noise value sampled from (0, ol),,,, i.e. x =
= = e. Then
B Σ
E|,: 1 BBaa | -nB[ax
Thus EE[]] is in fact negative definite. Extending Theorem 3 to a general k >> 1 we conjecture the following to hold.
Conjecture 2 (Evolution of learning dynamics in for 2 > 1). If C has atleast lo strictly negative eigenealues and u1.... UK are the eigenveclors correspouding the most negalive eigenvalues then the space spanned by U, and that by 91.... .s are the same in limit where q, denotes the ''th colurn of q.
We note that to prove the conjecture it enough to show in limit u Uk are each contained in the space spanned by Q1....9 ... As the norm of Us is 1, it is enough in turn to show that the norm of the projection of t2 onto the space of 01,..., I converges to 1. We experimentally verify and illustrate this in Figure
New Datapoint. While the above dynamics provide the setting during training we can furthermore in- vestigate what happens we input a new datapoint or a testpoint to the machine. Because u is a linear function and because v1, Ua basis this quite trivial. So & new point, let a (01,,.)aa)))) be the co-ordinates of i, iee. at- = Ee 0400% or a = V Then
d
(+) = un Σ 0,4% = Σ (w;) = 9. a V
4.2 Numerical Evaluation
We can now further illustrate the above derived theoretical results empirically. Let us first consider the setting of 2-layer networks.
Leaning dynamics (Theorem 2) and new Datapoint. We can now illustrate that the derived dynamics in 8 do indeed behave similar to learning 7 using gradient decent updates. To analyze the learning dynamics we consider the gradient decent update of
((++1) w((h) 1.2 7/ w wwo
(10)
where (((t) (((() are the weights at time step and n is the learning rate as reference. Practically the constraints in enforced by projecting the weights back onto W, W = πl and W, W, I after each
10

---

Published in Transactions on Machine Learning Research (06/2024)
2
2
2
1.0
|Proipte
0.6
0.4
0.0
200
250
50
100
200
250
100
150
Epoch
Epoch
Epoch
Figure 4: Consider the dataset presented in Figure la and embedding into = (1, 3). Plotted is the norm of the projection of 01....03 onto the space spanned by the columns of q.
Runtime
Accuracy
0.6
Neural Network
1.00
0.5
Differential Equation
Runtime
0.4
0.90
0.3
0.85
0.2
0.1
0.75
0.0
Hidden Layer Size
Hioden Layer Size
Figure 5: (left) Run-time comparison between rumning differential equation and SGD iteration for different hidden layer width. (right) Downstream task: Accuracy comparison for SVM on embedding obtained by SGD optimization and running the differential equation.
gradient step. Secondly we consider a discretized version of 1
4++1 = - :m[2Aq Aquel qi 9:9 Aq]]
(11)
where de is the machine outputs at time step t.
We now illustrate the comparison through in Figure where consider different width of the network (h E {10, 100, 1000)) and 0.01. We can firstly observe on the left, that the loss function of the trained network and the dynamics and observe while the decay is slightly slower in the dynamics setting both converge to the same final loss value. Secondly we can compare the function outputs during training Figure 3 (right): We initialize the NN randomly and use this initial machine output as fo- We observe that during the evolution using (10) to the same final outputs.
for a given initialization the are stay close to each other and converge
Numerical Evaluation of Theorem 3 We can again il- lustrate that the behaviour stated in Theorem g can indeed be observed empirically. This is shown in Figure (right); a setting where has negative eigenvalues. We observe that eventually the machine outputs converge the smallest eigenvector.
Dynamics of the Machine Output
0.1
Convergence
D.0
Hidden Layers
Runtime and downstream task. Before going into the il- lustration of the dynamics furthermore note that update step using 11 is significantly faster then a SGD step using [10). For this illustration we now consider two classes with 200 datapoints each from the MNIST dataset Deng (2012). This is illustrated in Figure 5 (Ieft) where we compare
Hidden Layers
-0.5
Figure 6: Numerical illustration of the con- vergence of deep linear models.
runtime over different layer width (of which (11) is independent of). Expec- fantly (10) scales linearly with h and overall [1 has shorter
11

---

Published in Transactions on Machine Learning Research (06/2024)
runtime per timestep. While throughout the paper we focus the obtained embeddings we can furthermore consider the performance of rownstream tasks on top of the embeddings. We illustrate this in the setting above where we apply a linear SVM on top of the embeddings. The results are shown in Figure 5 (right) where we observe that overall the performance of the downstream task for both the SGD optimization and the different ial equation coincide.
Deep Linear Networks. for several hidden layer sizes. In Figure O we observe that for one, two and three hidden layers the dynamics behave very similarly and all converge to the same final embedding, the one derived in Theorem 3
Numerically we illustrate that the learning dynamics as derived in Theorem 2
5 Non-Contrastive Loss Functions
In the previous sections analyzed contrastive loss functions however we show in this section the derived results extend beyond this setting and are also applicable to con-contrastive losses. Non-contrastive losses emerged from the observation that negative samples (or pairs) in contrastive SSL are not necessary in
practice, and suffices to maximise only alignment between positive pairs Chen & He (2021) Chen et al.
(2020 Grill et al. (2020). Considering a simplified version of the sctup in Chen et al. (2020) one learns a representation by minimising the loss
min
u(xx))((x;]) u(2,))
(12)
The embedding u [uz (-. u.( Rd R. parameterised by O typically comprises of a base encoder network and a projection head in practice (Chen et al. 2020) We now shortly outline how the previously presented results apply to this setting as well.
Dimension collapse. Analogous to the contrastive setting earlier we can now discuss dimension collapse for the non-contrastive setting. Assuming that the network function u : Rd + R is parameterised by O the gradient of the loss L(O) = Σ "=u(x)TT(x(?) is Vel(9) - - Ej (z(x)) Veu,(=;) ) + (z()))
"
1=1
Veu,(zi). Hence, under gradient descent è = - VeC, the evolution of each component of u(x), given by (x(x) (Veu(z), 6) d((x) E. E (V))(x), Vau, (z;)) u, + (Veu(z), (at (z())) For linear model, u(x) = Ox, the dynamics of u(x) is given by ún(x) = En (=T))()) (z)u(x;)) + (x x (x;) From there Proposition [ extends to the non-contrastive setting. Under the conditions of Lemma | every component of the network output u. Rd + R= has identical dynamics. As a consequence, the output collapses to dimension convergence.
Learning dynamics of linear non-contrastive models. In addition we can also frame the previously considered non con-contrastive model in 12) in the simple linear setting by considering the general loss function
with C = =. xy (x+) Therefore the results from Theorem 2 & extend to models under 12) as well.
Limitations and trivial solutions. For arbitrary functions this loss can have a trivial minimizer that is constant function. However in the linear setting mainly consider in this paper the only constant function is the zero function which would not be optimal.
6 Conclusion
The study of lcarning dynamics of (infinite-width) neural networks has led to important results for the supervised setting such as understanding the loss landscape and convergence, early stopping, linearised (kernel) approximations and, mostly importantly, generalisation and inductive biases. The analysis of similar quantities is of interest in the SSL setting as well, however, there is little understanding of SSL dynamics.
We simplify Chen ct al. 2020) by replacing the cosine similarity with the standard dot product and also by replacing additional positive sample x; by anchor a; for convenience.
4It. has previously been observed that one can unify contrastive and non-contrastive losses in more general framework Garrido al. 2022
12

---

Published in Transactions on Machine Learning Research (06/2024)
Our initial steps towards analysing SSL dynamics encounters hurdle: standard SSL training has drastic dimension collapse (Proposition D. unless there are suitable constraints. We consider a general formulation of linear SSL under orthogonality constraints 7 and derive its learning dynamics (Theorem 2 We also
show that the derived dynamics can approximate the SSL dynamics using wide neural networks (Thoorem T under some conditions on activation Φ. We not only provide a Iramework for analysis of SSL dynamics, but
also shows how the analysis can critically differ from the supervised setting. As we numerically demonstrate, our derived dynamics can be used an efficient computational tool to approximate SSL models. In particular, the equivalence in Proposition 3 ensures that the orthogonality constraints scaled loss, which is easy to implement in practice. We conclude with limitation and open problem. Our analysis relies on a linear approximation of wide networks, but more precise characterisation in terms of kernel approximation (Jacol et. al. 2018 Liu et al. 2020) may be possible, which can better explain the dynamics of deep SSL models. However, integrating orthogonality or operator norm constraints in the NTK regime remains an open question.
be equivalently imposed using
Acknowledgements
This work has been supported by the German Research Foundation (DFG) through the Priority Program SPP-2298 (project GH-257/2-1) and DFG-ANR PRCI "ASCAr (GH 257/3-1).
References
Sina Alemohammad, Zichao Wang, Randall Balestriero, and Richard G. Baraniuk. The recurrent neural tangent. kernel. In International Conference Learning Represenlations, 2021.
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Ruslan Salakhutdinov, and Ruosong Wang. On exact com- putation with an infinitely wide neural net. In International Conference on Neural Information Processing Systems, 2019a.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theorel- ical analysis of castiae unsupervised representation learning. In International Conference on Machine Learning, 2019b.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In Advances in Newral Information Processing Systems, 2022.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamod Pirsiavash, Yann LeCun, and Micah Goldblun. A cookbook of self-supervised learning, 2023.
Yamini Bansal, Gal Kaplun, and Boaz Barak. For sele-supe learning, rationality implies generalization, provably. In 9th International Conference on Learning Representations, 2021.
Han Bao, Yoshihiro Nagano, and Kento Nozawa. On the surrogate gap between contrastive and supervised losses. In International Conference on Machine Learning, 2022.
Adricn Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varaacc-imarrancc-cooaaan regularization for self-supervised learning. arXiv preprinl arXio:2105. 04906, 2021.
Saurav Basu, Koyel Mukherjee, and Shrihari Vasudevan. Layer dynamics of linearised neural nets. arxiv preprint XrXix:1904. 10689, 2019.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using siamese' time delay neural network. Advances in neural information processing systems, 1993.
Vivien Cabannes, Bobak Toussi Kiani, Randall Balestriero, Yann LeCun, and Alberto Bictti. The SSL interplay: Augmentations, inductive bias, and generalization. CoRR, abs/2302.02774, 2023.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Aduances in neural information processing systems, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, 2021.
Changyou Chen, Jianyi Zhang. Yi Xu, Liqun Chen, Jiali Duan, Yiran Chen, Son Tran, Belinda Zeng, and Trishul Chilimbi. Why do we need large batchsizes in contrastive learning? A gradient-bias perspective. In Advances in Newral Information Processing Systems, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In JEEE Conference on Computer Vision and Pattern Recognition, 2021.
Yilan Chen, Wei Huang, Lam M. Nguyen, and Tsui-Wei Weng. On the equivalence between neural network and support vector machine. In Advances in Neural Information Processing Systems 34, 2021.
Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 2012.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Simon S Du, Kangcheng Hou, Russ R Salakhutdinov, Barnabas Poczos, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. In Advances in Neural Information Processing Systems, volume 32, 2019.
Alan Edelman, Tomás A Arias, and Steven T Smith. The geometry of algorithms with orthogonality constraints. STAM journal on Matriz Analysis and Applications, 1998.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International Conference on Machine Learning, 2021.
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-onc-out networks. In JEEE conference on computer vision and pattern recognition, 2017.
Kenji Fukumizu. Dynamics of batch learning in multilayer neural networks. In ICANN 98: Proceedings of the 8th International Conference on Artificial Neural Networks. Springer, 1998.
Quentin Garrido, Yubci Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. arXiv preprinl urXiw:2206. 025774, 2022.
Jiawei Gc, Shange Tang, Jianqing Fan, and Chi Jin. On the provable advantage of unsupervised protraining. arXiv preprint, abss/3303.01. 00666, 2023.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint WrXiv:1803.07728 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self- supervised learning. In Advances in Neural Information Processing Systems, 2020.
Lu Han, Han-Jia Ye, and De- Chuan Zhan. Augmentation component analysis: Modeling similarity via the augmentation overlaps. In The Fleventh International Conference on Learning Representations, 2023.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In Advances in newral information processing systems, 2021.
Reinhard Heckel and Fatih Furkan Yilmaz. Early stopping in deep networks: Double descent and how to eliminate it. In International Conference on Learning Representations, 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In International Conference Neural Information Processing Systems, 2018.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In The Tenth International Conference on Learning Represenlations, 2022.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
Daniel D. Johnson, Ayoub El Hanchi, and Chris J. Maddison. Contrastive learning can find an optimal basis for approximately view-invariant functions. In International Conference on Learning Representations, 2023.
Angjoo Kanazawa, David W Jacobs, and Manmohan Chandraker. Warpnet: for single-view reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Weakly supervised matching
Bobak Toussi Kiani, Randall Balestriero, Yubei Chen, Seth Lloyd, and Yann LeCun. Joint embedding self- supervised learning in the kernel regime. CoRR, abs/2200.1488% 2022. doi: 10.48550/aa /tee///rXiiv////// 2209.14884
Daniel Kunin, Jonathan Bloom, Alcksandrina Goeva, and Cotton Seed. Loss landscapes of regularized linear autoencoders. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Zehua Lai, Heng Lim, and Ke Ye. Simpler grassmannian optimization, 2020. URL https://aare
org/ aa//abbs//0000 13502
Jason D. Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning, In Advances in Neural Information Processing Systems, 2021.
Jiangyuan Li, Thanh V Nguyen, Chinmay Hegde, and Raymond K. W. Wong. Implicit sparse regularization: The impact of depth and early stopping. In Advances in Neurol Information Processing Systems, 2021.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin, On the linearity of large non-linear models: when and why the tangent kernel is constant. In Advances in Neural Information Processing Systems 39, 2020.
Ziyin Liu, Ekdeop Singh Lubana, Masahito Ueda, and Hidcnori Tanaka. What shapes the loss landscape of self supervised learning? International Conference on Leurning Representations, 2023.
Ping Luo, Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. In International Conference on Learning Representations, 2019.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics
and the double descent curve. Communications on Pure and Applied Mathematics, 2022.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of protext-invariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognitiinn, 2020.
Abdel-Rahman Mohamed, Hung yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang- Wen Li, Karen Livescu, Lars Maalće, Tara N. Sainath, and Shinji Watanabe. Self-supervised speech representation learning: A review. IEEE JSTSP Special Issue on Self-Superciser Learning for Speech and Audio Processing, 2022.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in
deep learning. Tn Advances in Neural Information Processing Systems, 2017.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Thanh V. Nguyen, Raymond K. W. Wong, and Chinmay Hegde. Benefits of jointly training autoencoders: An improved neural langent kernel analysis. IEEE Transactions on Information Theory, 2021.
David Novolny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. Self-supervised learning of geometrically stable features through probabilistic introspection. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Ashwini Pokle, Jinjin Tian, Yuchon Li, and Andrej Risteski. Contrasting the landscape of contrastive and non-contrastive learning. arXiv preprial arXiv:2203. 15702, 2022.
Arnu Pretorius, Steve Kroon, and Herman Kamper. Learning dynamics of linear denoising autoencoders. In International Conference on Machine Learning, 2018.
Mahalakshmi Sabanayagam, Pascal Esser, and Debarghya Ghoshdastidar, Representation power graph convolutions: Neural tangent kernel analysis, 2022. URL https://arxiv.orrt http:/oo///abb////eeee 09809
Nikunj Saunshi, Sadhika Malladi, and Sanjeev Arora. A mathematical exploration of why language models help solve downstream tasks. In International Conference on Learning Representations, 2021.
Andrew M. Saxe, James L. McClelland, and Surya. Ganguli. Fxact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Brain. Time-contrastive networks: Self-supervised learning from video. In JEEE internalional conference on robotics and outomation, 2018.
Anshul Shah, Suvrit Sra, Rama Chellappa, and Anoop Cherian. Max-maraea margin contrastive learning. In AAAI Conference on Artificial Intelligence, 2022.
James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning, In International Conference on Machine Learning. PMLR, 2023.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data, Journal of Machine Learning Research, 2018.
Steffen Steffen, Alexei Baevski, Ronan Collobert, and Michael Auli, wav2vec: Unsupervised pre-training for speech recognition, 2019.
Remi Tachet, Mohammad Pezeshki, Samira Shabanian, Aaron Courville, and Yoshua Bengio. On the learning dynamics of deep neural networks, arXiv preprint orXic:1809.068// 2018.
Yuandong Tian, Understanding deep contrastive learning via coordinate optimization. In Advances in Neural Information Processing Systems, 2022.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory, Proceedings of Machine Learning Research, 2021.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, has/1800.03747 2018. URL
037488
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. Advances in Neural Information Processing Systems, 2021.
Zixin Wen and Yuanzhi Li. Toward understanding the feature lcarning process of self-supervisod contrastive learning. In International Conference on Machine Learning. PMLR, 2021.
Wei- Yong Yan, U. Helmke, and J.B. Moore. Global analysis of oja's flow for neural networks. JEEE Transactions on Neural Networks, 1994.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference machine learning. PMTR, 2021.
Runtian Zhai, Bingbin Liu, Andrej Risteski, Zico Kolter, and Pradeep Ravikumar. Understand- ing augmentation-based self-supervised representation learning via RKHS approximation. CoRR, abs/2306.00788, 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Zhijian Zhuo, Yifei Wang, Jinwen Ma, and Yisen Wang. Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. In The Eleventh International Conference on Learning Representations, 2023.
Liu Ziyin, Botao Li, and Xiangming Meng. Exact solutions of a deep linear network. Advances in Neural Information Processing Systems, 2022.
17

---

Published in Transactions on Machine Learning Research (06/2024)
A
Appendix
In the appendix we provide the following additional proofs
Proof Lemma
Proof of Proposition
Proof of Proposition
3
Proof of Theorem
1
Proof of Theorem
2
Proof of Theorem 0
A.1 Proof of Lemmal
Proof. Let the collumms of W2 be denoted by W1, W2, Wg. Then we note that each component of u, uj is given by uj(x)) = wo ((Wi)). Thus if + j. uz(x) has no dependence with wi ie. Vw, = 0. Thus we get that when + J.
(Vou(v), Veuz(E)) = (Vw,u(r), Vw.y(T1))
We can use Liu et al. 2020) (for instance its Lemma 3.1) which basically concludes that no training happens at the penultimate or prior layers. In limit all positive gradients arise only from the final layer. As such
(Wnyu(()), Vw, = 0.
By the same token, for /
(Vou(w), Veny(x')) (Vw, (1),, Vmnu(=')) + +V (V)), Vuyu((d)) (6(W1x)) o(W1z))
Finally again using the fact that W1 does not change training and that W1 initialized from a normal- ized gaussian when á is the identity map, it is well known that the above converges x TTa' (as there (o(Wiz)) (WW1z")
(W, W1)æ + x²) and otherwise to deterministic kernel & (see e.g. LLu et al. Arora et al. 2019bb).
2020
A.2 Proof of Proposition
Proof. For simplicity of the proof we begin by reformulating the loss function in both contrastive and noncontrastive setting a more general form. In particular it trivial check that we can generalize by writing
- Tr (W,
where X denotes the collection of all the relevant data (i.e. V 1 < S I,, as well as of and x- where applicable), and f(X, Wi) = En (Wix;)) (#(W1z77) - o(W1x;)) in the contrastive setting (equation while f(X, Wi) - Σi= (W (W112))((112)) in the non-contrastive set setting (equalion 12
Then decompase
k W.W, Σ
o uu
18

---

Published in Transactions on Machine Learning Research (06/2024)
Note then that ||2 Tr (W2W22) = Ei= o₂. Thus the optimization target,
& = Tr (1)2w)) = Tr (S(X, Σ of
L(W,, W2) = Tr (W22
h -o;= 1 min to f(X, Wi)e ;= k
Σ पं f(x, Wi Wi)u:
min tuz f(x, W1) W) i=
t=1
Thus when the Frobenius norm is restricted (i.e. bounded between 0 and c). if f(X, ((X,W) has atleast one negative eigenvalue the loss is minimized when i1 is the eigenvector corresponding to the most negative cigenvalue of f(X, W1) with 01 = ||Wl|p, with no other non-zero singular value. On the other hand if f(X, Wi) has no negative eigenvalue then the loss is minimized when W2 0.
A.3 Proof of Proposition 3
Proof. We begin by quickly observing that (1)
(2). This is simply done by defining W. Wr for [
= 2. Then have
Tr (W2 W W1CWTW, W2)
Tr (w, WCCWT
argmin Wa
argmin
W| |W212
w. w. wi |
Using the fact that atleast one eigenvalue of C is strictly negative (this rules out the case that the optimal is achieved when W. 0 as that would have prevented division by norm) then we quickly get that
Tr (W. W.CWT W/) W. s W,, W1
Tr (w. WWTTTWWW (.C.TW.).
argmin
argmin
For (2) = (3), begin by observing that by submultiplicatiiity of norm, any W1. W2 such that [|ill22 < 1 and |Wall2 I automatically falls is the optimization space given by [[ W2 < thus giving one direction of the optimization equivalence for free. For the other side we note that given any W1, W2 such that || Wz 12 [[W WoW, Wi S 1, we can construct Wi, W2 such that [w S and
W,TWww WoW, W, W W W. This follows from considering the singular values decomposition of W, W2. getting W,W,- = U ΣV. As the norm of the product is smaller than 1, all the entries of the singular value matrix E are less than 1. Thus depending upon which among or 2 is larger we consider either the matrices SU and V or the matrices and VV to be our candidate W, and W2 respectively. To complete we will simply have to add zero rows our choice i.c. say U and EV to match the dimensions (i.c. to get n d matrix from x d one).
Finally for (3) < (4) we begin by defining W = WTT W2. Then the optimization problem in (3) becomes,
min Tr (WTCW) Willi
min Tr TCWWTT).
We then prove that we are done if we can prove the claim at optimal of (3) (i.e. the above optimization problem) all the eigenvalues of WWT are 1 Given this claim the singular value decomposition of W becomes only W = V. where if k rank(W), U is & x d matrix and V & x matrix. Additionally by property of SVD, the collumns of U and V are orthonormal. Finally as
k rank(W) I mankrr..WWi) rank(W2)) <min(d, 2) Sn. n,
we can add a bunch of zero rows to U and V to get our n and n matrices which will be our corresponding W₁ and Wz.
It remains to prove that Tr (CWWT) is minimized when all the eigenvalues of WWT are 1 or 0. To do this simply decompose
WWT = Σ u
19

---

Published in Transactions on Machine Learning Research (06/2024)
where 22 is the set of orthonormal eigenvectors of WWT alternatively non-zero singular values of W) Then
corresponding non-zero eigenvalues of WWT (or
Tr (CWWT) =Tr
C oin
t=1
0 Trr (Cv.u))
0 Cus. 1=1
Thus if C has I many strictly negative eigenvalues \ı S S A with corresponding eigenvectors , and of is positive the above quantity is minimized by choosing many of these possible i.c. 01 C.....: (min(d 2 and setting the corresponding to be 1 while every setting all other eigen- values to 0.
We then also note by consequence the above proof that we avoid dimension collapse when possible i.e. when C has multiple strictly negative eigenvalues (which what one should expect if the data is not one dimensional as E[C] -Ezr I))
A.4 Proof of Theorem
Proof. Let us start by defining some properties for the non-linearity: Assune the non-linear function ø is continuously twice differentiable near and has no bias i.e. (00) Then via scaling we can assume WLOG that (0( 1. As ( S c, get that P
l6(x) x| cx2
(13)
2
Recall that the mapping of the first weight matrix is given by W1 Rd RA, h> under the constraint that W.TW- I. Under uniformly random initialization by Lemma (see proof below) then with probability asymptotically going to we have that
log h max (Wi) s C
h
Thus the norm of each row of Wi we get with a.w.h.p.
row; (Wi)l2 = E(wni, s c
dloog
J=1
From there we can now write the value of each node in the layer using Cauchy-Schwarz inequality
| d log?
[wo,(WW)) x < |roww (WaII? ||x|| s C
(14)
h
We now apply the non-linearity to this quantity and denote the output of the first layer after the non-linearity as
0 (row; (W,)= ..)
Define the vector e e IRA, where
9 = 0; row; (Wi) .
We can actually also use the weaker assumption that (0) continuous Thus there some bounded (compact) set containing constant such that Vzr E A, [d(x)
x| < m
20

---

Published in Transactions on Machine Learning Research (06/2024)
Then we have for h large enough f
|e||
h
Σ
(1: row; (W1). x)2
(row; (W1) xx)
by equation 15
?
d'log" h
C [1
by equation 14
h
hal log'
d² log') h = K²c² ||/]||
= k2e
|
h?
h
where K is the universal constant outputs of the two networks as
C.
Combining this with the second layer we get the difference of the
W[ W2 W
||W (v - wi
WW'll ||||| |||||
as W
log"
s Ke||ue| dv
0.
Lemma 2. Given any d p, Let Q be uniformly random d semi-orthonormal matria. L.e. Q is the first d columns of an uniformly random h x h orthonormal matrin. Then there are constants L and sequence Ep converging to 0 as h goes to infinity such that
Mlgh') Vh.
max
S En
Proof. We note that it is enough to prove the claim when d = h, i.c. Q is uniformly random hxh orthonormal matrix. Then as our distribution uniform, the densily at any particular Q is same as the density at any UQ where U some other fixed orthogonal matrix. Thus if qı is the first column of Q, the marginal distribution of qı has the property that its density at any 4 is same that of Un for any orthogonal matrix U. In other words the marginal distribution for any column of Q is simply that of the uniform unit sphere.
Consider then the following random variable which has the same distribution as that of a fixed column of Q i.c. uniform unit h-sphere. Lot X (X,,.., X₁)) be iid random variables from (0, Then we know X that X ~~ Ix) From the rotational symmetry property of standard gaussian then we have that eX] is distributed as an uniform sample from the unit sphere in h dimensions. By union bound then, we have
Plng"
P
max Isshh
X 2 (log h
< he V2T
22 Low?
P
max
[X|] < & log h
/2m he
ISSA
Note that for the weaker assumption we still use equation O and thus row; A in limit.
13
This because by equation 14 w.h.p. row:(W1) x goes
21

---

Published in Transactions on Machine Learning Research (06/2024)
As each X; is tid normal, X3 is iid Chi-square with E[X?] 1, thus by Chernoff there exists constants C², such that
h iill
X
P
2 1 21 C'e
e 21-C'e-dhs"
Since maxigigh\x [X,] I tlog h and
E
1-1
x2
< (1 + 8) implics that maxisish
logh Vi(l=))
we get that
h
|X]|
max Isish
X
tlog h
2
2' he
+ Ing
C'e -'eo
X
/h(1 s)
max ISish
X
2
tlog h
he
logg2
+ C'e--hhs
X
h[( s)
2
From the argument before that any j'th column of Q is distributed as X. Using the above and another union bound then get us
max max Qi,
2
tlog h
S
1
he
lux h
+ C'e-ch
Isish iSih
Vh(1- Vh(1-s)
v
2m
=
max max [Qi
1 log
hre
2 loc?
+ C''h-'
ISyS ISish
Vh(1 8)
22T
We note that for any constants c that as h goes infinity, both h&e 2ngh and he e'hs? goes to zero. The proof is then finished by choosing some appropriate constants s, t 0.
A.5 Proof of Theorem
Proof. To simplify notation we are dropping the superscript. from ((e) The u in the following proof is already presumed to be linear. For the same reason we are also dropping the symbol of time, t, from u, Wa, W2, W, even though all of them are indoed time dependent. Finally for any time dependent function f, denote 0f by f.
Using this and recalling that the loss in Eq. Consider the following loss function for L layer deep linear network
min Tr (W, Wi-1 ..... Wi s.t. W W,W, W, w. kE
=1 vi [T].
where I is an appropriately sized identity matrix, Wi Raxd, W T W, E Ra Vi E 2.... - are the trainable weight matrices. Ce e Iddd is a symmetric, data dependent matrix.
To simplify the proof we now consider a three hidden layer network and we will observe that the obtained results exlends to arbitrarily deep networks. Let the embedding function therefore be
u(x) = w3 W2 s.t. W.T, = 1, W.J W3 = and W., W2 =.
From (Edelman et al. by left-multiplying
1998
we get that the derivative of a function 7 restricted to grassmanian is derived
>y to the "free" unrestricted derivative of y. We therefore can write W,, W, and
Wa as
(t(() (I WoW,) VW,C= -2 2 WoW,T) (WWWCWW W.W.CW,TWTWW.......... Ws), W2(t) = - (I W2W,) Vw = -2 (1- W2W,)) (WaWJ WoWiCw,T),
(i()) (I- w.WWT) Vwn -2 W1W1)) (W. WaW, ([TWW)))....
22

---

Published in Transactions on Machine Learning Research (06/2024)
Thus obtain
i (()
(C) (2W1 + Wa(1) TWa
11
= -(11- -(1- (- T WaWi(()
+ W (L-WWW (-WaW3 W2W1CW;)) Wi(t)z
+ +Ws(1) TWs(t)
W.Wa (W,,,,,, W2W1C)
-22 W2W1Cr)
+2(W3 W2W,CW] W W W2W11 + W.J W2 WT WTJ Wa W W2W1Cx)
= -2W, W2W,Cr W,W,CW, W. W,W, w2W1x
Σ W W2W W1 W Wa W3 W2W.Cr))
where obtain the second equality by expanding the terms, laking advantage of W,T,, W. 1, i E [3]. and Ia = Σ. p...T Now setting a as and using the fact that they are eigenvectors for C and using C= Σx Naai gives us:
i(w;) 22)u(s(1)))) - Σ Xauro (1))))))) (u)(())))))))))))))))))))) - X3 E (o(())))))))) (uc(())))
Let's rewrite this in matrix notation. First define q q= := [u(v1), u(va)] thus obtaining:
q 22Ag Agg q qq Ag].
Finally note that the presented argument would apply similarly for any additional hidden layer, through the same formulation as W2, therefore extending the result to deeper networks well which concludes the proof.
A.6 Proof of Theorem 3
Proof. For instance first suppose that all the eigenvalues of A are strictly positive and thus TAAq > 0. Then
d(q Tg q)
dt 24 T = 4[-44111100 (1-44
TT 4)q T Aq - 4 TT(II -
(I-qq' qq )Ag]
= =-8(1 1-- - Aq
Observing that because of orthonormality of our weight matrices, TT |lo|2 1 (as ||사|| * initialization with probability 1) we get that the derivative of l|qll is always negative and thus q converges to 0.
Now suppose the other hand there is atleast one negative eigenvalue. WLOG let e denote the eigenvector with the smallest eigenvalue (which is negative). Then
d(e q) = =T = 2 1 601-11- (1- gT A) A qq T AAq
dt
2[[(1 - (((1-qTa)) ()(->eeeT + (q Aq >1)eT g)]
We now note that qT > 2 0 A is the smallest. eigenvalue. Thus ー>1 is positive, the derivative
of eT q is always positive unless 1 - q T Aq A1 0, which only happens at q e1. In other words. eventually q becomes the smallest eigenvector El.
J
23


Reviews:
[{"review_id": "bE6xVRoRTV", "reviewer": "Reviewer_xEdX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. SSL learning dynamics is an important topic.\n2. The authors show that the learning dynamics of naïve SSL setting leads to trivial solution, which provides insights to the dimension collapse.\n3. The study is extended to the setting with orthogonal constraint, and the author has provide numerical evidence to support the claims.\n\n\nPotential Weaknesses:\nAn important reference [1] on SSL learning dynamics is missing. The authors may want to look into this paper and discuss the connection. [1] Simon et al., On the Stepwise Nature of Self-Supervised Learning\n\nProposition 1 shows that SSL learning objective leads to trivial solution without regularization. And, then the rest of the paper is established upon the orthogonality constraints. While this is a reasonable setting, the results from this paper still has a large gap to the practice in deep SSL. Could you author provide more connection to the deep SSL methods and show how the results in this paper can (significantly) help us in the practice of deep SSL. I would really appreciate if a deep insight can be provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the learning dynamics of SSL methods, which is an important direction. The paper is well-written and makes valid points under the target settings There are a few points, I would appreciate if the authors can address in the revision.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. An in depth discussion of the related reference on SSL learning dynamics.\n2. To provide further insight between the settings used in this paper and the deep SSL practice.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R01ADlONKL", "reviewer": "Reviewer_nAXY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nI appreciate studying the problem through the trace minimization formulation. I like the connection the authors make with previous work on the dynamics of the eigenvectors of the data-dependent matrix. I think the work is a fresh perspective of the problem, and it will be useful to practitioners.\n\nWeaknesses:\n\nSome of the motivations of the paper need to come earlier in the text. For example, there is not a concrete definition of dimensional collapse. At many points in the work, there are introductions of some objectives/ dynamics without motivated dynamics. E.g., the trace formulation appears before the authors introduce it later in the text; it is not clear how Lemma 1 connects to Section 2.2 (i.e. what do you mean by SSL not having labels, implies $y=0$, and then how exactly that implies dimension collapse is unclear from the main text); there is not enough intuition presented why Frobenius norm doesn't work for avoiding collapse, while the L2-operator norm works. \n\nI think the work lacks a conclusion that could discuss the impact of the work on SSL as a whole. Currently, there is only a mention of regularization in NTK as an open problem. We need more practical guidance, and high-level insights about the dynamics of SSL in the conclusion.\n\nnit: there are also a few minor typos, e.g. \"atleast\" <- \"at least\"; one sumation starts from \"i=n\" and goes to \"n\"; and a few others in the text.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the dynamics of SSL in a novel way. The authors first demonstrate a dimensional collapse issue in SSL without orthogonal regularization. Then they focus on a reformulation of the SSL loss a trace minimization problem with a suitable data-dependent symmetric matrix in the trace operator. The authors show that under certain assumptions (and one conjecture) that the non-linear and linear dynamics settings are close to each other. Then they continue studying the linear setting. The authors accurately understand the dynamics through the eigenvectors of the data-dependent matrix. Numerical evidence is given for the conjectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please, take a look at the Weaknesses discussion above, and propose changes that can alleviate my concerns. Thanks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0hSJTkB7oK", "reviewer": "Reviewer_w2XZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Questions:\n- I wonder why the paper considers this simplified form of the non-contrastive loss, which has a trivial minimizer that is constant function. The authors cite previous papers to argue that \"it suffices to maximise only alignment between positive pairs\". However, this misses a crucial point that the success of non-contrastive loss relies on regularizations (e.g. requiring the feature covariance to be close to the identity matrix) or algorithmic tricks (e.g. normalization layers) to preserve the rank of the features.\n- Prop 3: I'm not sure why 1&2 can avoid dimensional collapse. For example, if $C$ only has 1 strictly negative eigenvalue, then $W_1,W_2$ only need to be of rank 1. This also means that the optimization problems e.g. 2&4 are not equivalent: the loss functions have the same minimal value, but the set of minimizers are different (e.g. rank-deficient solutions are allowed in loss 2 but not loss 4).\n- Conjecture 1: what's the basis for this conjecture? Sec 3.2 provides numerical evaluation, but I don't understand what this adds beyond the intuition from NTK.\n- Sec 3.2 / Fig 2: could you clarify the takeaway here please? For example, the results on tanh and ReLU are supportive of a NTK / lazy training point of view (i.e. increasing the layer width makes the linear and non-linear closer to each other), and I don't see how it's specific to SSL.\n\n\nRelations to prior work should be discussed further, and there are missing citations:\n- For theoretical analyses of optimization of contrastive and non-contrastive methods:\n  - Wen and Li 21: Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning\n  - Pokle et al. 22: Contrasting the landscape of contrastive and non-contrastive learning\n- Barlow Twins, a non-contrastive method whose objective is very similar to the one in Conjecture 1. Notably, the minimizer for the Barlow Twins objective is for the network outputs to have an identity covariance.\n- In Section 4, about viewing contrastive and non-contrastive objectives as a \"general trace minimization problem\": the connection between non-contrastive objectives and trace minimization has been pointed out in prior work, such as Balestriero & LecCun 22. This can also be connected to contrastive methods, e.g. Garrido et al. 22 has observed the duality between contrastive and non-contrastive methods.\n  -  Garrido et al. 22: On the duality between contrastive and non-contrastive self-supervised learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies dimensional collapse in self-supervised methods, where dimensional collapse is defined as having rank-deficient representations.\nThis paper analyses the contrastive learning and non-contrastive losses and shows that they both suffer from dimension collapse, and proposes an orthogonality constraint that is meant to prevent dimensional collapse.\n\nTheoretical analyses in this paper are under the infinite width limit.\nThe paper first shows that where the nonlinear dynamics and the linear dynamics are close to each other: consider a linear network $u^\\mathbb{I}(x) = W_1^\\top W_1x$ and a nonlinear network $u^\\phi(x) = W_2^\\top \\phi(W_1)x$.\n- Theorem 1 states that at initialization, the difference in outputs of a linear and a non-linear network is bounded by $O(\\|x\\|^2 \\cdot \\frac{d\\log^2 h}{\\sqrt{h}})$, where $d$ is the feature dimension, and $h$ is the network's hidden dimension.\n- There's no formal guarantee for the full training dynamics, but the paper conjectures (Conjecture 1) that the two outputs will remain close throughout training, approaching 0 as $h \\rightarrow \\infty$.\n\nGiven the closeness between the linear and nonlinear case, the training dynamics focuses on the linear case, i.e. $u(x) = W_2^\\top W_1x$. The contrastive and non-contrastive objectives are rephrased and unified to a general trace minimization problem with some matrix $C$. The analyses track the evolution of the representations for eigenvectors of $C$  (Theorem 2) and a 1-dimensional special case (Theorem 3).\n\nThe paper corroborates the theoretical results with experiments on half-moon and MNIST.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "One concern I have is that the theoretical results are based on 2-layer linear network, with limited implications to other setups or practice. Please comment more on the implications, or provide results from more complex empirical setups as further evidence.\n\nPlease add more thorough discussions to the prior work; please see the comments above for some examples.\n\nI'd also suggest a careful proofread of the manuscript as there are many typos. Here are some examples:\n- the section title of Sec 2.2 should be \"a trivial solution\" or \"trivial solutions\".\n- Thm 1, \"a universal constant\". Thm 2 contains a typo too.\n- Sec 3.2,  \"three activation functions\" (rather than loss functions)?\n- sentence breaks, e.g. in the paragraph below eq (12).\n- Prop 2: has $a(x)$ been defined?", "broader_impact_concerns": "There's no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the training dynamics of self-supervised models, deriving exact learning dynamics under a simplified form of the loss function and with orthogonality constraints on the weights. While the reviewers are in agreement that understanding the dynamics of self-supervised learning is an important question, there was some discussion as to whether the simplifications made here faithfully preserve the important aspects of the problem, especially because there were doubts as to whether the empirical analysis fully closed this gap. On the other hand, the paper clearly describes its limited approach in analyzing simplified settings as a step towards building understanding of more practical configurations. Since the paper does provide sound and convincing evidence supporting this more limited analysis, it seems to fall well within the criteria for publication at TMLR. I recommend acceptance.
Decision: Accept (poster)