Collapse-Proof Non-Contrastive Self-Supervised Learning
Authors: Emanuele Sansone
Tim Lebailly
Tinne Tuytelaars
Venue: ICML 2025 poster (2025)
Keywords: self-supervised learning
hyperdimensional computing
feature decorrelation
clustering
collapses
Primary area: deep_learning->selfsupervised_learning
TL;DR: Principled and simplified design of the projector and loss function for non-contrastive self-supervised learning based on hyperdimensional computing

Abstract: We present a principled and simplified design of the projector and loss function for non-contrastive self-supervised learning based on hyperdimensional computing. We theoretically demonstrate that this design introduces an inductive bias that encourages representations to be simultaneously decorrelated and clustered, without explicitly enforcing these properties. This bias provably enhances generalization and suffices to avoid known training failure modes, such as representation, dimensional, cluster, and intracluster collapses. We validate our theoretical findings on image datasets, including SVHN, CIFAR-10, CIFAR-100, and ImageNet-100. Our approach effectively combines the strengths of feature decorrelation and cluster-based self-supervised learning methods, overcoming training failure modes while achieving strong generalization in clustering and linear classification tasks.

Collapse-Proof Non-Contrastive Self-Supervised Learning
Emanuele Sansone
Tim Lebailly'
Tinne Tuytelarrs
Abstract
D
We present a principled and simplified design of the projector and loss function for non-contrastive self- sufe-sueeeieee learning based hyperdimen- sional computing. We theoretically demonstrate that this design introduces an inductive bias that encourages representations to be simultaneously decorrelated and clustered, without explicitly en- foreing these properties. This bias provably en- hances generalization and suffices avoid known training failure modes, such representation, di- mensional, cluster, and intracluster collapses. We validate our theoretical findings on image datasets, including SVHN, CIFAR-10, CIFAR-100, 100, and ImageNet-100 100. Our approach effectively com- bines the strengths of feature decorrelation and basse supervieed learning methods, overcoming training failure modes while achiev- ing strong generalization in clustering and linear classification tasks.
Figure 1: Top: Venn diagram relationships between types of collapses, viz. dimensional collapse (ID), cluster collapse (C), intracluster collapse (I) and representation (full) col- lapse of embeddings. Bottom: on the left, collapse-pree proof data embeddings (each blue dot correspond to an embedding of a data point, e.g. an image) and on the right, examples of collapses for each single type (collapsed clusters are high- lighted in red).
In this work, we focus on the family of non-contrastive SSL approaches, aiming to distill the essential principles to guarantee the avoidance of known failure modes. More concretely, identify sufficient conditions avoid repre- sentation, dimensional, cluster, and intracluster collapses, exemplified in Figure 1, and correspondingly devise a pro- jector and loss function enforcing them by design. In par- ticular, we demonstrate that minimizing invariance to data augmentations while matching priors suffices to avoid repre- sentation and cluster collapses, whereas orthogonal frozen weights based on hyperdimensional computing, and large prediction outputs in the projector are key to avoiding di- prove mensional and intracluster collapses. Moreover, that these principles are sufficient to guarantee (i) the decor- relation of embeddings. without any explicit computation of their covariance matrix, and (ii) the clustering of embed- dings, without the use of specialized clustering layers. experimentally validate the theory on four image datasets, including SVHN, CIFAR-10, CIFAR-100 and ImageNet- 100, showcasing training robustness to failure modes and strong generalization to downstream clustering and classifi- cation tasks compared to popular feature decorrelation and cluster-based SSL.
1. Introduction
Self-supervised learning (SSL) has unlocked the poten- tial of learning general-purpost representations from large amounts of unlabeled data. Despite its successes, important challenges remain, hindering the applicability of SSL to broader spectrum of real-world tasks and its widespread adoption and democratization. One such challenge is the presence of failure modes occurring during the training of SSL models. Several heuristic strategies have been proposed and analyzed the literature, such as momentum encoder, stop gradient and asymmetric projector heads (Chen al., 2022; He et al., 2020; Grill et al., 2020; Tao al., 2022; Chen & He, 2021; Tian et al., 2021; Halvagal et al., 2023; Wang et al., 2022). However, these heuristics do not always come with universal guarantees, making it unclear whether failure modes can be avoided in all situations.
'Department of Electrical Engineering (ESAT)) KU Leuven, Belgium *CSAIL, MIT, US. Correspondence to: Emanuele San- sone sanaanulee.sasonee kuleuven.be>.
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To summarize, our key contributions are:

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
At conceptual level, we identify sufficient conditions to avoid known failure modes and accordingly devise CPLearn, the first Collapse-Proof Proof non-constrastive self-supprvised Learning approach.
et al., 2023b; Balestriero & LeCun, 2022; Huang al., 2023) aiming towards unifying different families of SSL. Our work complements these efforts by providing a princi- pled solution and design to bring together cluster-based SSL and feature decorrelation methods from the non-contrastive family.
These conditions are shown to be sufficient to jointly decorrelate and cluster embeddings, thus providing evidence on the feasibility of unifying non-contrastive and fluster-based SSL families of approaches.
Failure modes in SSL. SSL can be affected by four un- desired failure modes, namely representation. dimensional, cluster and intracluster collapses, ure 1. Representation collapse refers to the case where neural representations collapse to an identical constant vec- tor, irrespectively of their input. Different strategies have been proposed to avoid the issue, such as leveraging con- trastive objectives to maximize mutual information between data and representations (den Oord et al., 2018; O. Henaff, 2020; Chen et al., 2020; Lee, 2022: Linsker, 1988; Becker & Hinton, 1992; McAllester & Stratos, 2020; Barber & Agakov, 2004; Belghazi et al., 2018; Poole et al., 2019; Tschannen et al., 2019; Song & Ermon, 2020), introducing heuristics such as momentum encoder, stop gradient and asymmetric projector heads (Chen et al., 2022: He et al., 2020; Grill et al., 2020; Tao al., 2022; Chen & He, 2021; Tian al., 2021: Halvagal et al., 2023; Wang et al., 2022), regularizing the objective by introducing a generative term to reconstruct or estimate the data density (Hendrycks et al., 2019; Winkens et al., 2020; Mohseni et al., 2020; Kim & Ye, 2022; Gatopoulos & Tomczak, 2020; Zhue et al., 2020; San- sone & Manhaeve, 2022; Wu ct al., 2023; Nakamura et al., 2023; Sansone, 2023; Sansone & Manhaeve, 2023; 2024) and leveraging predictive models to identify masked data such as image patches text tokens (Assran al., 2022: Zhou et al., 2022). Dimensional collapse occurs when em- beddings span a subspace of the whole vector space. Several methods (Zbontar et al., 2021; Zhang et al., 2021; Ermolov et al., 2021; Li et al., 2022b; Liu et al., 2022; Bardes et al., 2022a:b; Ozsoy et al., 2022) propose to mitigate the issue by whitening the feature embeddings (Hua et al., 2021). Dimensional collapse has been recently linked to reduced performance of downstream tasks (He & Ozay, 2022; Li et al., 2022a; Garrido et al., 2023a), and different evaluation metrics have been consequently derived, such as the compu- tation of the entropy of the singular value distribution for the covariance matrix (Jing et al., 2022), the rank estimator (Gar- rido et al., 2023a), the computation of the AUC (Li et al., 2022a) or a power law approximation (Ghosh et al., 2022) of the singular value distribution. Cluster collapse is phe- nomenon observed in cluster- based SSL., where data points are assigned to a subset of available prototypes (Caron et al., 2018; Asano et al., 2020; Caron et al., 2020: Li et al., 2021; Caron et al., 2021; Govindarajan et al., 2023). The issue is typically mitigated by introducing regularizers in the objec- tive, such as the Koleo regularizer (Sablayrolles et al., 2019; Oquab et al., 2024; Govindarajan et al., 2024) explic- itly enforcing uniform cluster assignments (Amrani et al.,
exemplified in Fig-
We establish the first connection between SSL and hy- perdimensional computing, thus supporting training and the systematic exploitation of large projector out- puts.
At practical and computational level, we simplify the design and training of non-contrastive SSL. and provide a proof-of-concept demonstration of the properties of CPLearn
The structure of the paper is organized as follows: In $2, we relate our work to theory of SSL, clarify the relation between different failure modes and review recent efforts aiming to address them. In $3, we provide design principles for the loss and projector head of CPLearn. Subsequently, we provide the theory supporting its design. In $4, we compare our solution to existing feature decorrelation and cluster- based SSL strategies and analyze the different properties highlighted by the theory. Finally, we conclude with $5 by summarizing the main findings and discussing future work.
2. Related Work
We frame this work within the context of theoretical studies of SSL and solutions aimed at mitigating collapses.
Theory and relations among different faamilies of SSL. Several works have theoretically investigated con- trastive (Saunshi et al., 2019; Wang & Isola, 2020; Zim- mermann et al., 2021; Tosh et al., 2021; HaoChen et al., 2021; Saunshi et al., 2022; Wang et al., 2024) and non- contrastive SSL methods (Tian et al., 2021; Kang-Jun et al., 2022; Weng et al., 2022; Wen & Li, 2022; Shwartz-Ziv et al., 2023), to improve our understanding and provide more principled or simplified solutions. There have been works identifying key properties of SSL objectives and inductive biases (Wang& & Isola, 2020; Dubois et al., 2022), generaliz- ing SSL to incorporate data augmentation graphs (HaoChen et al., 2021; Wang et al., 2024). deriving generalization error bounds (Saunshi et al., 2022: Bao et al., 2022; Shwartz-Ziv et al., 2023), understanding SSL objectives from an opti- mization perspective (Tian et al., 2021; Tian, 2022; 2023) as well as understanding the role of asymmetries and pro- jector heads (Kang-Jun et al., 2022; Wen & Li, 2022). A recent line of studies has focused on identifying connections between contrastive and non-contrastive methods (Garrido
2

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
2022; Assran al., 2023). Last but not least in terms of importance, intracluster collapse, similarly to the notion of neural collapse observed in supervised learning (Papyan et al., 2020; Fang et al., 2021; Yang et al.. 2022; Chen et al., 2022; Kothapalli, 2023; Dhuliawala et al., 2023) occurs whenever the variability of the embeddings within some clusters is infinitesimally small. Intracluster collapse can be mitigated by enforeing representation equivariance (rather than invariance) to data augmentations (Dangovski et al., 2022; Komodakis & Gidaris, 2018; 8; Scherr et al., 2022; Park et al., 2022) or by splitting the embeddings into content and style parts, while using only content for the self-supervision task (Louizos et al., 2016; Kügelgen et al., 2021; Garrido et al., 2023c). In contrast, this work provides principled yet simple solution based on an objective function and projector head that avoid all forms of collapses.
[c], whereas
Volog 1-e(c-1)
with an arbitrarily small positive scalar. 2 Notably, we will provide theoretical justification for the design choice of the CPLearn projector, demonstrating good properties for the embeddings, as being both well clustered and having their features decorrelated. Moreover, we suggest to choose > f to avoid dimensional and intracluster collapses, as we will show in $3.2 and $3.3. The CPLearn loss consists of two terms, including one to promote invariance to data augmentations and one for prior matching, namely:
B
LCPLeam (D)
Pig
log
3. Method and Properties
n1
Notation. We denote matrices using capital bold letters, e.g. P. their elements using lowercase letters with subscript indices, e.g. Pij. their row and column vectors using lower- case bold letters, e.g. P, and Py. Additionally, we denote sets using capital letters, e.g. S and use squared brackets when dealing with sets of integers, e.g. [n] (l,...,7) ,n. Finally, we use lowercase letters for functions, scalars, inte- gers and constants, e.g. n. Whenever evaluating functions on matrices, we always assume that the function is applied row-wise.
qj log Σ
Σ
Psf
(2)
72
with q= l.......... 9c] E Q corresponding to a prior prob- ability vector, chosen uniformly for all e codes in all our experiments, viz. 93 = 1/c for all j E [c], B > is a weight hyperparameter to balance the relative importance of the two loss terms and Pip. Ply are elements of P, P', respec- tively. We will prove in $3.1 that, when the two loss terms are minimized, the proposed loss function is guaranteed to avoid representation and cluster collapses and therefore allows to train both the backbone and the projector networks through backpropagation without requiring any additional heuristics, such as stop gradient, momentum encoder or clustering operations typically introduced in non-contrastive learning (Chen al., 2022; He et al., 2020; Grill et al., 2020; Tao et al., 2022: Chen & He, 2021: Tian et al., 2021; Hal- vagal et al., 2023; Wang et al., 2022). Taken altogether, the CPLearn projector and loss functions guarantee to train the backbone network in a robust manner, preventing all known forms of collapses. Consequently, this represents the first callapse-prooy non-contrastive SSL solution and we summarize the method together with its PyTorch-like pseudo-code in Figure 2.
Overview of CPLearn. Given an unlabeled batch of data D [XX. X')} containing n pairs of augmented images, so that X, X' Raxd, we propose to train backbone encoder g Rd + I using the CPLearn projector and loss functions.' The projector takes the representations (Z, = (x)), ()),,((X)) (X)), with Z, Z' E RTxs, and per- forms two operations. Firstly, it computes embeddings H, H' E / for the corresponding representations and then it computes probabilities P, P,P' gaxe for assign- ing embeddings to codes available from frozen dictionary W E 1] More precisely, the projector is defined by the following two layers:
H Vf/n- L2-nom((n(((iierr(())
P Softmax (HW /T)
3.1. Minima of the Loss Function
In the following, we assume that the backbone has infinite capacity. Further discussion about the relaxation of this assumption is left to Appendix C. Therefore, we decouple the study of the objective and its minima from the neural network. In Appendix D, we prove that
where the embeddings are obtained from representations through the composition of linear, batch norm and I.2 nor- malization layers, and 7 is the temperature parameter of the softmax layer. Each element Wiy of W is drawn indepen- dently and identically distributed according to a Rademacher distribution, i.e. Wij ~ Rademacher for all i E [f] and
For images d is the product of the width, height and color hands.
26 le throughout the paper.
3

---

Collapse-Prool Non-Contrastive Self-Supervised Learning
L CPLeam
Algorithm 1 Pseudocode for CPLear
g:
encoder network
batch size
Frozznn
enbedding dimensionality
Adfacency (Clustering)
dictionary size
Embedding
8 sps: 1e-6
bn: batch normalization
Representations
normalization activation
Learnable
softmax: softmax activation with temperature # bete: weight for the invariance leas
Encoder
ce: crossentropy loss
compute non-learnable dictionary codea
Covariance (Decorelattonn
W = 2 randint sizen(t, el) 1 f-by-o
for x loader: load batch with samples
Augmented Images
X
randonly augnented versions X_o
x, x augment (x of
$ compute cepresentations
2 g # n-by-f g(X)) n-by-f
extract enbeddings (blue square block)
(bn (linear
aqrt / n-by-E (+) (Linear (2'))) aget / o-by-f (+)
Original Imsges
X
compute probabiiities (gray blook)
tau = / (#get (n) 1og((1 eps (o 1)) epe))
softmax (H W, tau] n-by-c
softmax
tau) n-by-o
In CPLearn, minimizing the proposed objective together with the corresponding projector ensures that the embedding representations are clustered and at the same time that their features are decorrelated. This guarantees that the representations are collapse-rroof, meaning that di- mensional, cluster, intra-cluster and representation collapses prevented.
Figure 2:
compute losses
loss_ piroprior = ce / (.))
loss inv ce P))mmen(()
prior
optimization step
loss backwarr)
optimizer. step
+ See Practical Considerations for simpler alternative to L2- i2-normalization
Lemma 1 (Minima). Vi [m] and E [c), S Pij S
explained by observing that the first two conditions (in- variance and extrema) and the last one (matched prior) are mainly a by-product of the invariance and the matching prior losses in Eq. 2, respectively. Indeed, note that the invariance loss can be equivalently expressed as a cross- entropy loss. Therefore, it can be decomposed into the sum of an entropy term for p: and a Kullback-Leiin gence term between p: and p/, thus enforcing the extrema condition through the entropy term and the invariance con- dition through the minimization of the KL one. Minimizing the matching prior loss is equivalent to minimize the KL between zj and 1/n E Pij. thus enforcing the matched prior condition. It is also important to specify that while the results of the Lemma are general and valid for any prior distribution q, our focus is mainly on the uniform setting, We leave the study of the non-uniform case (Assran et al., 2023) to future work.
1-e(c-1) and S 93 S 1-) with 0 S E 1/c,
then the global minima of the loss function in Eq. 2 jointly satisfy the following conditions:
IIvariance Vi [n] and j E [c]. Pis ply
(KL) diver-
* Extrema Vi E [n], Eij E [c] such that Pij 1-c(c-1) and Ak: E [c] with lo + j. Pik = e.
Here, the word extrema refers to the extrema of probability simplex.
Matched prior Vj E [cl, 1 En Pij = qz-
More-
over, Vj E [c] define Tmaz (j) = 1i [m] Pij
1- c-c (.ee(cc)) 1)}. then |max (j)
0i di-e
1-ox n.
The global minimum value of Eq. 2 is bounded by
An important implication of the Lemma is that the global minima of the CPLearn objective guarantee to avoid rep- resentation and cluster collapses, consequently reducing the need for heuristics, such as stop gradients, momentum encoder and/or specialized clustering layers typically in- troduced in non-contrastive settings. Indeed, observe that all data points indexed by [mx] are assigned in hard way to one of the codes available in the dictionary due to the extrema condition. Moreover, the distribution of the assignments follows the result of the matched prior
LCPLeam (D) 2 - -B(1 c(c - log(1 - c(c 1)) 1))
- -Be(c log (())
(3)
being equal to H() whenevere = 0, where H(g) is the entropy of q.
The assumptions in the Lemma can always be met by properly choosing arbitrarily small e to satisfy the relation < <<//// 1/c. The results of the Lemma can intuitively
4

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
condition, specifically me (i) n(g,-e)/(1-ce) For uniform prior 9j = 1/c, we have that ma (i) n/c for all codes j. meaning that data points are partitioned and equally assigned to all available codes. Representation collapse is prevented because data points are assigned in a hard fashion to different codes, whereas cluster collapse is avoided because all codes contribute to the partitioning of the data. Moreover, attaining the lower bound value in Eq. gives certificate for the avoidance of these collapses.
same direction of a single code. If azy = - 20112////// VT. the contribution of the spurious term becomes non-zero, scaled by factor of 2/c. This notion of alignment is important to achieve decorrelated and clustered features, as we will see shortly. The key assumptions to the theorem are the orthogonality of W, whose codes define a basis for the embedding space and consequently each embedding can be expressed as a linear combination of the dictionary codes, and normalized embeddings, which allow constrain the possible values of coefficients for this linear combination. It is important to specify that, for the sake of generality, the theorem considers an orthogonal dictionary matrix, which deviates from the specific choice made in Eq. 1. We will elaborate this detail about W in the next subsection.
A similar guarantee result has recently appeared in another work (Sansone, 2023), where the same training objective genera-
to Eq. 2 is used in addition to a likelihood- tive term. Their analysis studies each loss term separately, demonstrating their different properties. That is, the gener- ative term prevents representation collapse, the invariance term enforces smoothness and adherence to the cluster as- sumption, while the matching prior loss prevents cluster collapse. Differently from (Sansone, 2023), we study the objective where the invariance and the matching prior losses are jointly optimized and show through Lemma that they are sufficient to avoid the two collapses without the need of additional terms, like the generative one used in (Sansone, 2023).
The theorem has three important consequences that are dis- tilled in the following corollaries:
Corollary 1 (Perfect alignment). Given the assumptions in
Theorem 1, if + sc. then Vi E [n]. e [c] such that ai vt
1 is unique and h, ]
vn w
Proof. The result in Theorem states that Vi E [m], Alj e [c]. the coefficients aiy E {1/V (1-2/c) /vn) Taking - oc, forces all admissible values of azy coincide with unique value 1/ 1/vn. Substituting this result into Eq. 4 completes the proof.
3.2. Properties of the Projector
We now turn the analysis to the design of the projector and state the main theorem of this work for the case of (the proof is provided in Appendix E). We will later see how to generalize these results to * f. Importantly, the theorem sets the stage for the two key properties of CPLearn, that is of learning decorrelated and clustered features and avoiding dimensional and intracluster collapses.
This means that the orthogonality of codes, the large size of the dictionary and the normalization of the embeddings are important inductive biases that are sufficient to guar- antee perfect alignment the codes. Indeed, for large dictionary, each embedding is assigned to only one of the available codes, avoiding spurious terms. We also prove in Appendix F that
Theorem 1 (Embedding). Given the projector defined in Eq. 1 with f > 2 and a dictionary matrix W satisfying the condition W W = fI. if the optimal- ity conditions of Lemma I are met, then the embed- dings H satisfy the following relation
Corollary 2 (Diagonal covariance). Given Eq. 4 in Theo- rem and uniform q, assume that Vi E [n], 3ij E [c] such that oij = Va then the covariance of the embeddings is equal to the identity matrix, i.e. H H I.
Vie E [n] E [c] s.T.
The assumptions of the corollary can be satisfied by simply choosing a large dictionary along with the other inductive biases discussed for Corollary 1. These biases are suffi- cient to ensure that the embeddings are decorrelated and span the entire embedding space, thus avoiding dimensional collapse. Moreover, the importance of decorrelated em- beddings translates into the property of having both the embedding and representation matrices with full rank. This ensures improved generalization to supervised linear down- stream tasks, as shown in Appendix G. Finally, we prove in Appendix H that
h; = 0i + aij
Σ light
wk (4)
vn
with auj E of 1/2 (1 -1-2 2) Vn
The theorem tells that at optimality embeddings align with the orthogonal codes from the dictionary. More concretely, each embedding aligns with one code up to some spurious additive term, i.c. the second addend in Eq. 4, whose contri- bution depends on the admissible values of the coefficient aij and the remaining codes. Notably, if aij = 1/Vn, the spurious term disappears and the embedding shares the
Corollary 3 (Block-diagonal adjacency). Given Eq. 4 in Theorem I and uniform q, assume that Vi E [n] e [c] such that aig -n
1. then the adjacency matrix for the
5

---

Collapse-Prool Non-Contrastive Self-Supervised Learning
embeddings, i.e. H is a block-diagondl matrix with blocks of equal size and their size being equal to n.
tional properties of HC is beyond the scope of the current work and left to future work.
In Eq. I we have a linear and batch normalization layer to ensure that the representations have well-behaved first- order and second-order statistics throughout training, thus speeding up the training convergence. Some examples are provided in Appendix A. Additionally, we replace the L2- normalization activation in the projector with the hyperbolic tangent one, without affecting the results of the theory. The reason for such choice is to facilitate training: Instead of constraining embeddings to lie on a hypersphere and consequently reduce their dimensionality, we constrain them to lie inside a hypercube while preserving their dimensions. We also provide an experimental comparison between the two activations in terms of generalization performance in Appendix B.
The orthogonality of the codes and the normalization of the embeddings are therefore sufficient conditions to enforce clustered embeddings. The large size of the dictionary (c > 1) contributes to decreasing the block size of the adjacency matrix, consequently reducing the effect of intra-cluster collapse.
3.3. Practical Considerations
So far, the analysis has focussed on the case of c = f. demonstrating that by choosing large с (also large f) leads to decorrelated and clustered embeddings and the preven- tion of dimensional and intracluster collapses. However, in practice, we rarely have control over the size of the rep- resentation f. The typical learning setting of SSL takes backbone network with fixed f and uses a projector to train it. Therefore, it is natural to ask whether our previous results hold with the increase of e when / is fixed. At a first glance, the answer to this question is negative. Indeed, note that all previous results rely on the assumption of orthogonal W, that the codes span the whole embedding space and also act as orthogonal basis. Since f is fixed, we can have only f orthogonal codes. We can go beyond such limitation and provide an affirmative answer to the above question by probabilistically relaxing the notion of orthogonal W and leveraging principles from hyperdimensional comput- ing (HC) (Kanerva, 2009). More concretely, we can choose codes as in Eq. that W E {-1,1) with elements drawn i...d from Rademacher distribution, to obtain quasi-orthogonal dictionary matrix. Indeed, we observe that all columns of W have fixed norm, namely ||æ, = vF. and that the expected cosine similarity between two codes satisfies the following properties:
4. Experiments
The experimental analysis is divided into four main parts. Firstly, we compare CPLearn against non- contrastive approaches from the families of feature decor- relation and cluster-based methods on three image datasets, i.e. SVHN (Netzer et al., 2011), CIFAR-IO,, CIFAR- 100 (Krizhevsky et al., 2009). Secondly, we demonstrate the effects of increasing the dictionary size to validate the results in Corollary 2 and 3 and their implication to general- ization on downstream tasks, including clustering and linear probe evaluation. Thirdly, we scale the analysis using a larger backbone on CIFAR-IO. Finally, we run the analysis on ImageNet- 100. We use ResNet-8 backbone network with 128 for SVHN and CIFAR1O, and with f 256 for CIFAR- 100, following the methodology from (Sansone, 2023). The scaled analysis, leverages a ResNet- 18 backbone with ʃ = 512. For ImageNet-100, we use a standard small VIT with f = 384, following the methodology from (Caron et al., 2021). The 3 parameter in Eq. 7 is chosen from the range {0.01, 0.05, 0.1, 0.25, 0.5, 1, 2,2,5 5, 10), so that both terms in the objective are minimized. More details are avail- able in Appendix 1). We use the repository from (da Costa et al., 2022) for SVHN and CIFAR experiments, and the one from (Caron et al.. 2021) for ImageNet- 100 experiments. Further details are available in Appendices I, L,, M and N.
1
j=
Ew (cos(ur,. )
and
0
j
j
1
(orw(cos(wu) wy
Vj.j" e [c]
(5)
Therefore, the codes are orthogonal to each other aver- age with the variance being inversely proportional to the size of the representation. In other words, W W = fI holds on average independently of the choice of and all assumptions for Theorem and its corollaries are still satis- fied. We illustrate the concept in Figure 3. There are several ways define random code vectors in HC. We chose to use the multiply-ade aedd-ee encoding, which leverages Rademacher distribution. We refer the reader to recent survey on HC for more details (Kleyko et al., 2022). This is a form of encoding equipped with simple element wise addition and multiplication operations to perform algebraic compositions. Notably, the exploitation of the composi-
Generalization on downstream tasks. We compare CPLearn with Barlow Twins (Zbontar et al., 2021), forc- ing diagonalization of the embedding cross-covariance, SWAV (Caron et al., 2020), using Sinkhorn- based clus- tering layer in the projector, Self- flaf-Classifier, using a similar loss (Amrani et al., 2022), and GEDI (Sansone & Manhaeve, 2024), using our loss function in conjunction with a multi-
At optimality, embeddings are aligned and all results from the corollaries are still valid.
6

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
6
%.5
9..5
%0.
9.5
ca) f 50
(b) C 2f
(c) - 5f
(d) C
100
(e) 2f
(1) C 5f
Figure 3: Top: Illustration of W W obtained by randomly sampling W. Bottom: Normalized histograms of the elements of W W. Figs. 3a-3e have fixed 50, whereas Figs. 3d-3f have fixed f 100. w W has diagonal values and random off-diagonal values centered around zero. For larger and fixed variance remains constant and quasi-orthogonaliyy preserved.
Table 1: Test generalization on downstream tasks including clustering and supervised linear probing. Performance are measured terms of normalized mutual information (NMJ), accuracy (Acc.) and are averaged over 5 training runs obtained from random initialization seeds. We test CPLearn for undercomplete (c 10), complete (c = f) and overcomplete (c = 16384) dictionaries. For the Self-Classifier, we test recommended size for the dictionary (c = k, with & being the number of ground truth classes) and the overeomplete case (c 16384).
Clustering (NMI)
Supervised Linear Probing (Acc.)
Method
SVHN
CIFAR-10
CIFAR-100
SVHN
CIFAR-10
CIFAR-100
Barlow
0.06::0.02
0.05+0001
0.10=0001
0.76:10.01
0.63=10.00
0.28=00.00
SwAV
0.03:10.01
0.2911.02
0.12:1.07
0.40
0.56
0.10+10.06
GEDI no
0.0710002
0.3310002
0.2811000
0.63110.2
0.66
0.39.10.00
GEDI
0.07110.00
0.2910001
0.2510.01
0.5810.00
0.64
0.38110.00
Self-Classifier (c- k)
0.0710002
0.2810.01
0.26-10.00
0.58100.01
0.5910.01
0.15
Self-Classifier (16384)
0.2510001
0.1410.10
0.1710.21
0.70.10.01
0.34
0.16:10.00
CPLearn (10)
0.11+00.01
0.28+0.02
0.15+0000
0.60
0.59
0.15++0.00
CPLeam (c f)
0.16++0.02
0.25+0001
0.33+0000
0.60
0.59+0001
0.18++0.01
CPLeam (16384)
0.29110.00
0.35
0.59
0.75110.00
0.67
0.40
(a) = 10
(b) c = 128
(c) c 16384
(d) 10
(e) = 128
(f) C 16384
Figure 4: Realization of embedding covariance (left) and adjacency matrices (right) for the whole CIFAR-10 100 test dataset. Increasing e reduces the value of the off-diagonal elements of the covariance, thus contributing to increase the decorrelation of features (cf. Corollary 2). Moreover, increasing has the effect to reduce the block sizes of the adjacency matrix (cf. Corollary 3).

---

Collapse- Propp Non-Contrastive Self-Supervised Learning
Accuracy
Loa of Sinuuaar Values
Setchaiite %ell
Extiiatee
65
25
60
Enerayy
8
512
1014
2040
4096
0152
16384
40
8096
2132
16384
Se Fclauireer
Burfow
Dictonary
Dictinaay
40
80
100
120
16
10
10
Figure 5: Downstream generalization on CIFAR-10 test
Singular Value Rand Index
Number of Misture Components
dataset, clustering (left) and linear evaluation results (right).
(a) Dimensional
(b) Intracluster
Figure 6: Analysis of dimensional and intracluster collapses. For Self-Classifier, we choose c =
layer perceptron projector as in Barlow Twins.* 4 We test the downstream performance on clustering using normalized mutual information (NMI) computed between the projector predictions and ground truth labels and supervised linear probing on the representations using accuracy (Acc.). In Table we report all results and include three different set- tings for CPLearn, each corresponding to the undercomplete f), complete (c=f) f) and overcomplete (c>f) f) dictionary case. We observe that Barlow Twins performs well on linear probe evaluation compared to the other baselines, thanks to the connection between feature decorrelation and gener- alization (cf. Appendix G), whereas SwAV, Self-Classifier and GEDI perform better on clustering tasks than Barlow Twins. We also observe that CPLear in the undercomplete case performs comparably well to the other cluster-basee based baselines despite its difference in the design of the projector. However, CPLearn is the only method to systematically leverage larger dictionaries to improve both the clustering and classification performance, as predicted by our theory.
study intra-cluster collapse by estimating the entropy of the embedding distribution. This is done by: (i) fitting Gaussian mixture model with diagonal covariance to the embeddings; (ii) estimating the entropy of the resulting dis- tribution via Monte Carlo sampling using 10k samples; and (iii) repeating the analysis for different numbers mixture components, i.e., 10, 20, 50, 100, 200, 500, 1000. Higher entropy values indicate lower degree of intra-cluster col- lapse. In Appendix K, (i) qualitatively demonstrate how the loss function helps avoid representation and cluster collapse, and we (ii) quantitatively show how large dictio- naries prevent dimensional and intracluster collapse, thus confirming that more structural information is preserved in the projector. Finally, we repeat the analysis of collapses by comparing CPLearn to competing methods. In Fig. 6a, observe that CPLearn achieves performance comparable to the best baseline Barlow Twins. This is due to feature decorrelation property induced by our design. All other approaches face dimensional collapse. In Fig.6b, we further demonstrate that CPLearn shows increased robustness to intra-cluster collapse compared to its competitors.
Effects of increasing the dictionary size. We provide addi- tional insights the benefits of increasing the size of the dictionary. Specifically, we show some examples of em- bedding covariances and adjacency matrices computed on CIFAR- 10 for different values of e in Fig. 4, thus demonstrat- ing that larger dictionary sizes contribute to implicitly diag- onalize the covariance as well as to reduce the block sizes in the adjacency, as predicted by our corollaries. Moreover, provide a quantitative evaluation the downstream tasks in Fig. 5, where we observe a monotonic increase in clus- tering and classification performance with large values of c. Similar results hold for other datasets, cf. Appendix Interestingly, the monotonic increase clustering perfor- mance suggests that more and more structured information preserved in the projector.
Method
Clustering (NMI)
Linear (Acc.)
Barlow
29.1
92.2
SwAV
18.9
89.6
GEDI no gen
44.6
80.0
Self-Classitier (c f)
36.9
84,8
Self-Clussifier (16384)
33.9
64.9
CPLearn (c f)
47.4
91.6
CPLeam (16384)
48.2
91.3
Table 2: Downstream generalization on Resnet-188
Analysis with larger backbone. We perform experiments with a larger backbone, i.e. ResNet-18, 18, trained for 1000 epochs on CIFAR-10 comparing all methods in terms of linear classification and clustering. Further details about hyperparameters given in Appendix L. Results are shown in Table 2. Overall, the table highlights similar observations to the ones from experiments on ResNet-8, with CPLearn achieving comparable performance to Barlow Twins and significantly outperforming all other approaches in terms of clustering. Additional analysis is provided Appendix L.
Additionally, we investigate dimensional collapse following the methodology proposed in previous work (Jing et al., 2022), by computing the singular value distribution of the covariance matrix of the embeddings. We also propose to
"The work in (Sansone & Manhaeve, 2024) proposes two so- lutions, one adding generative term to our objective, named GEDI, and one without, named GEDI no gen. Both versions use standard MIP network for the projector.
8

---

Collapsc-Proof Non-Contrastive Self-Supervised Learning
Table 3: Test generalization on downstream tasks including clustering and supervised linear probing. Performance are measured in terms of normalized mutual information (NMI), top-1 accuracy (Acc.) Models are trained on ImageNet- 100 for 300 epochs. oom stands for out-of-memory. Obtaining this table roughly costs 2k euros ( 48 GPU hours per simulation * 40 simulations * 1.35 euros per GPU hour on A100 GPUs). In bold, the best performance for each method.
Clustering (NMI)
Supervised Linear Probing (Acc.)
Small Projector
Large Projector
Small Projector
Large Projector
Method
128
1024
2048
32768
65536
131072
128
1024
2048
32768
65536
131072
Barlow
30.0%
53.6%
59.4%
oom
oom
00m
66.9%
77.2% %
77.2% %
oom
00m
00m
SWAV
28.0%
47.9%
51.2%
60.7%
60.5%
62.8%
76.6%
77.7%
78.0%
76.6%
77.7%
77.3%
DINO
46.0%
53.4%
55.2%
63.1%
64.7%
64.3%
71.8%
73.6%
73.9%
75.1%
76.2%
75.8%
GRDI no gen
24.5%
36.3%
38.8%
32.7%
32.9%
33.3%
71.8%
73.2%
72.8%
72.9%
72.7%
72.8%
CPLearn 34.1%
54.3%
57.9%
69.3%
69.6%
70.9%
70.8%
72.8%
70.1% 1%
74.5%
74.7%
78.0%
Analysis on ImageNet-100. We analyze the effect of the dictionary size ImageNet-100 100 using ViT-small back- bone and compare CPLearn against Barlow Twins (Zbontar et al., 2021), Sw AV (Caron et al., 2020), GEDI (Sansone & Manhaeve, 2024) and DINO (Caron et al., 2021). We use the original DINO codebase for the experiments and train all models for 300 epochs. Table 3 summarizes the results in terms of clustering and linear probing evaluation. While Barlow Twins and SwAV achieve good generalization on the linear probing task, they underperform compared to DINO in terms of clustering performance, demonstrating that no method is capable to tackle both tasks well. In contrast, CPLearn is able to make effective use of large dictionary size to achieve good generalization on both clustering and linear probing tasks. This further demonstrates the value of the theoretical guarantees of CPLearn, translating into simplified and principled design of the projector and loss function compared to DINO (such as avoiding the use of asymmetric operations like stop gradient, centering opera- tion for the teacher network, use of different temperature parameters for student and teacher networks and exponential moving average update of the teacher parameters), while ensuring good properties like decorrelated and clustered features.
the future, we plan to leverage the connection with hyperdi- mensional computing and leverage its algebraic properties for tackling learning and reasoning tasks.
Impact Statement
While the results of this work are foundational in nature, several positive impacts can be anticipated. Specifically, this work (i) enhances the robustness of non-contrastive self-supervised learning against training failures and (ii) simplifies the training design, taking a meaningful step to- ward more trustworthy representation learning strategies and democratizing access to their solutions.
Author Contributions
E.S. had the original idea, developed the theory, wrote the code, ran the experiments, wrote the paper. T. L. imple- mented the initial distributed code and helped with scaling results on ResNet-18 and ImageNet. T. T. provided the computational resources to run the experiments in Table 1, provided feedback and support in proof-reading the article.
Acknowledgments
This work received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement n 101021347) and under the Horizon Europe research and innovation programme (MSCA-GF grant agreement n 101 149800). The computational resources and services used in this work were provided by the VSC (Flemish Supercom- puter Center), funded by the Research Foundation Flanders (FWO) and the Flemish Government department EWI. Moreover, we acknowledge Euro CC Belgium for awarding this project (n 465001811 11700 access to the LUMI supercom- puter, owned by the EPr Joint Undertaking and hosted by CSC (Finland) and the LUMI consortium. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of our sponsors.
5. Conclusions
We have distilled the essential principles of non-contrastive self-supervised learning into the design of a projector and loss function that prevent known failure modes, including representation, cluster, dimensional, and intracluster col- lapses. This approach enables robust training of backbone network, achieving representations that are both decorre- lated and clustered. We have also demonstrated that the resulting solutions improve generalization to supervised downstream tasks when large dictionaries are used in the projector. A future direction is improve scalability when storing and using large dictionaries. We could leverage the bipolar nature of the codes, instead of treating them as floating-poin vectors, to significantly enhance efficiency. In
9

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
References
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations. In ICML, 2020.
Amrani, E., Karlinsky, L., and Bronstein, A. Self- Supervised Classification Network. In ECCV, 2022.
Chen, X. and He, K. Exploring Simple Siamese Represen- tation Learning. In CVPR, 2021.
Asano, Y. M., Rupprecht, C., and Vedaldi, A. Self-Labelling via Simultaneous Clustering and Representation Learning. In ICLR, 2020.
da Costa, V. G. T., Fini, E., Nabi, M., Sebe, N., and Ricci, E. Solo-learn: A Library of Self-supervised Methods for Visual Representation Learning. JMLR, 23(56):1-6, 2022.
Assran, M. al. Masked Siamese Networks for Label- Efficient Learning. In ECCV, 2022.
Assran, et The Hidden Uniform Cluster Prior in Self-Supervised Learning. In ICLR, 2023.
Dangovski, R., Jing, L., Loh, C., Han, S., Srivastava, A., Cheung, B., Agrawal, P., and Soljacie, M. Equivariant Contrastive Learning. ICLR, 2022.
Balestricro, R. and LeCun, Contrastive and Non- Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods. In NeurIPS, 2022.
den Oord, A. V., Li, Y., and Vinyals, O. Representation Learning with Contrastive Predictive Coding. In arXiv, 2018.
Bao, H., Nagano, Y., and Nozawa, K. the Surrogate Gap between Contrastive and Supervised Losses. ICML, 2022.
Dhuliawala, S., Sachan, M., and Allen, C. Variational Clas- sification. TMLR, 2023.
Barber, D. and Agakov, F. The IM Algorithm: Variational Approach to Information Maximization. In NeurIPS, 2004.
Dubois, Y., Hashimoto, T., Ermon, S., and Liang, P. Improv- ing Self-Supervised Learning by Characterizing Idealized Representations. In NeurlPS, 2022.
Bardes, A., Ponce, J., and Lecun, Y. VICReg:
Ermolov, A., Siarohin, A., Sangineto, E., and Sebe, N.
Regularization for Self- Supervised Learning. ICLR, 2022a.
Whitening for Self- Self-Supervised Representation Learning. In ICML, 2021.
Bardes, A., Ponce, J., and LeCun, Y. VICREGL: Self- Supervised Learning of Local Visual Features. NeurlPS, 2022b.
Fang, C., He, H., Long, Q., and Su, W. J. Exploring Deep Neural Networks via Layer-Peeled Model: Minority Col- lapse in Imbalanced Training. PNAS, 2021.
Becker, S. and Hinton, G. Self-Organizing Neural Network that Discovers Surfaces in Random-Dof Stereograms. Na- ture, 1992.
Garrido, Q., Balestriero, R., Najman, L., and LeCun, RankMe: Assessing the Downstream Performance of Pre- trained Self-Supervised Representations by Their Rank. In ICML, 2023a.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Ben- gio, Y., Courville, A., and Hjelm, D. Mutual Information Neural Estimation. ICML., 2018.
Garrido, Q. Chen, Y., Bardes, A., Najman, L., and Lecun, Y. On the Duality Between Contrastive and conCootrasttve Self-Supervised Learning. ICLR, 2023b.
Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep Clustering for Unsupervised Learning of Visual Features. In ECCV, 2018
Garrido, Q., Najman, I.,, and LeCun, Y. Seff-supervised learning of Split Invariant Equivariant representations. In ICML, 2023c.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. NearIPS, 2020.
Gatopoulos, and Tomezak, J. M. Self-Supervised Vari- ational Auto-Encoders. In Symposium on Advances in Approximate Bayesian Inference, 2020.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging Properties in Self-Supervised Vision Transformers. In ICCV, 2021.
Ghosh, A., Mondal, A. K., Agrawal, K. K., and Richards, B. A. Investigating Power laws in Deep Representation Learning, In arXiv, 2022.
Chen, M., Fu, D. Y., Narayan, A., Zhang, M., Song, Z., Fatahalian, K., and Ré, C. Perfectly Balanced: Improv- ing Transfer and Robustness Supervised Contrastive Learning. ICMI. 2022.
Govindarajan, H., Sidén, P., Roll, J., and Lindsten, F. DINO as von Mises-Fishe mixture model. In ICLR, 2023.
10

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
Govindarajan, H., Sidén, P., Roll, J., and Lindsten, F. On Partial Prototype Collapse in Clustering-Basee Self- Supervised Learning. In arXiv, 2024.
Komodakis, N. and Gidaris, S. Unsupervised Representa- tion Learning by Predicting Image Rotations. In ICLR, 2018.
Grill, B., Strub, F., Altché, F., Tallec, C.,, Richemond, P. H., Buchatskaya, E., Doersch, C., Pires, B. A., Guo, Z. D., Azar, M. G., et al. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. In NewrIPS, 2020.
Kothapalli, V. Neural Collapse: A Review on Modelling Principles and Generalization. TMLR, 2023.
Krizhevsky, A., Hinton, G., et al. Learning Multiple Layers of Features from Tiny Images. Technical report, Univer- sity of Toronto, 2009.
Halvagal, M. S., Laborieux, A., and Zenke, F. Implicit Vari- ance Regularization in Non-Contrastive SSL. In NeurlPS, 2023.
Kügelgen, J. V., Sharma, Y., Gresele, L., Brendel, W., Schölkopf, B., Besserve, M., and Locatello, F. Self- Supervised Learning with Data Augmentations Provably Isolates Content from Style. In NeurIPS, 2021.
HaoChen, J. Z.. Wei, C., Gaidon, A., and Ma, T. Prov- able Guarantees for Self-Supervieed Deep Learning with Spectral Contrastive Loss. In NeurlPS, 2021.
Lee, K. Prototypical Contrastive Predictive Coding. In ICLR, 2022.
He, B. and Ozay, M. Exploring the Gap between Collapsed & Whitened Features in Self-Supervised Learning. In ICML, 2022.
Li, A. C., Efros, A. A., and Pathak, D. Understanding Col- lapse in Non-contrastive Siamese Representation Learn- ing. In ECCV, 2022a.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Momen- tum Contrast for Unsupervised Visual Representation Learning. In CVPR, 2020.
Li, Y., Hu, P., Liu, Z., Peng, D., Zhou, T., and Peng, X. Contrastive Clustering. In AAAI, 2021.
Hendryeks, D., Mazeika, M., Kadavath, S., and Song, D. Using Self-Supervised Learning Can Improve Model Ro- bustness and Uncertainty. In NeurlPS, 2019.
Li, Z.,, Chen, Y., LeCun, Y., and Sommer, F. T. Neural Manifold Clustering and Embedding. In arXiv, 2022b.
Linsker, R. Self-Organization in a Perceptual Network.
Hua, T., Wang, W., Xue, Z., Ren, S., Wang, Y., and Zhao, H. On Feature Decorrelation in SeperSupe Learning. In ICCV, 2021.
Computer, 1988.
Liu, X., Wang, Z., Li, Y., and Wang, S. Self-Supervised Learning via Maximum Entropy Coding. In NeurIPS, 2022.
Huang, W., Yi, M., Zhao, X., and Jiang, Z. Towards the generalization of contrastive sulf-super learning. In ICLR, 2023.
Louizos, C., Swersky, K., Li, Y., Welling, M., and Zemel, R. S. The Variational Fair Autoencoder. In ICLR, 2016.
Jing, L.,, Vincent, P., LeCun, Y., and Tian, Y. Understand- ing Dimensional Collapse in Contrastive Self-supervised Learning. ICLR, 2022.
McAllester, D. and Stratos, K. Formal Limitations on the Measurement of Mutual Information. AISTATS, 2020.
Kanerva, P. Hyperdimensional Computing: An Introduction to Computing in Distributed Representation with High- Dimensional Random Vectors. Cognitive Computation, 2009.
Mohseni, S., Pitale, M., Yadawa, J. B. S., and Wang, Z. Self-Supervised Learning for Generalizable Out-of- Distribution Detection. In AAAI, 2020.
Kung-Jun, L., Masanori, S., and Takayuki, O. Bridging the Gap from Asymmetry Tricks to Decorrelation Principles in Non-contrastive Seff-Supervised Learning. In NeurIPS, 2022.
Nakamura, H., Okada, M., and Taniguchi, T. Representation Uncertainty in Self-Supervised Learning as Variational Inference. ICCV, 2023.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A., al. Reading Digits in Natural Images with Unsupervised Feature Learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Kim, B. and Ye, J. C. Energy-Barte Contrastive Learning of Visual Representations. In NewrIPS, 2022.
Kleyko, D., Rachkovskij, D. A., Osipov, E., and Rahimi, A. A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Trans- formations. ACM Computing Surveys, 2022.
O. Henaff, O. Data-Efficient Image Recognition with Con- trastive Predictive Coding. In ICMI., 2020.

---

Poole, B., Ozair, S., Oord, A. V. D., Alemi, A., and Tucker, Collapse-Proof Non-Contrastive Self-Supervised Learning
Oquab, M. et al. DINOv2: Learning Robust Visual Features without Supervision. TMLRR 2024.
Song, J. and Ermon, S. Understanding the Limitations of Variational Mutual Information Estimators. ICLR, 2020.
Ozsoy, S., Hamdan, S., Arik, S., Yuret, D., and Erdogan, A. Self-Suuee Learning with an Information Max- imization Criterion. NeurlPS, 2022.
Tao, C., Wang, H., Zhu, X., Dong, J., Song. S., Huang, G., and Dai, J. Exploring the Equivalence of Siamese Self- Supervised Learning via Unified Gradient Framework. In CVPR, 2022.
Papyan, V., Han, X. Y., and Donoho, D. L. Prevalence of Neural Collapse During the Terminal Phase of Deep Learning Training. PNAS, 2020.
Tian, Y. Understanding Deep Contrastive Learning via Coordinate- Sonddn Optimization. In NeurlPS, 2022.
Park, Y., Biza, O., Zhao, I.., van de Meent, J., and Walters, R. Learning Symmetric Embeddings for Equivariant World Models. In ICML, 2022.
Tian, Y. Understanding the Role of Nonlinearity in Training Dynamics of Contrastive Learning. In ICLR, 2023.
Tian, Y., Chen, X., and Ganguli, S. Understanding Self- Supervised Learning Dynamics Without Contrastive Pairs. In /CML, 2021.
G. On Variational Bounds of Mutual Information. In ICML, 2019.
Tosh, C., Krishnamurthy, A., and Hsu, D. Contrastive Es- timation Reveals Topic Posterior Information to Linear Models. JMLR, 2021.
Sablayrolles, A., Douze, M., Schmid, C., and Jégou, H. Spreading Vectors for Similarity Search. In ICLR, 2019.
Sansone, E. The Triad of Failure Modes and a Possible Way Out. In NeurlPS SSL Theory and Practice Workshop, 2023.
Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lucic, M. On Mutual Information Maximization for Representation Learning. In ICLR, 2019.
Sansone, E. and Manhaeve, R. GEDI: GEnerative and Discriminative Training for Self superviee Learning. arXiv, 2022.
Wang, T. and Isola, P. Understanding Contrastive Represen- tation Learning Through Alignment and Uniformity on the Hypersphere. ICMI,, 2020.
Sansone, F. and Manhaeve, R. Learning Symbolic Repre- sentations Through Joint GEnerative and DIscriminative Training. In ICLR NeSy-GeMs Workshop, 2023.
Wang, X., Fan, H., Tian, Y., Kihara, D., and Chen, X. On the Importance of Asymmetry for Siamese Representation Learning. In CVPR, 2022.
Sansone, E. and Manhaeve, R. A Bayesian Unification of Self-Supervised Clustering and Energy-Based Models. arXiv, 2024.
Wang, Y., Zhang, Q., Guo, Y., and Wang, Y. Non-Negative Contrastive Learning. In ICLR, 2024.
Wen, Z. and Li, Y. The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning. NeurIPS, 2022.
Saunshi, N., Plevrakis, O.. Arora, S., Khodak, M., and Khandeparkar, H. A Theoretical Analysis of Contrastive Unsupervised Representation Learning. In ICML, 2019.
Weng. X.,
Huang, L., Zhao, L., Anwer, R. M., Khan, S., and Khan, F. S. An Investigation into Whitening Loss for Supe-super Learning. In NeurIPS, 2022.
Saunshi, N., Ash, J., Goel, S., Misra, D., Zhang, C., Arora, S., Kakade, S., and Krishnamurthy, A. Understanding Contrastive Learning Requires Incorporating Inductive Biases. In ICML, 2022.
Winkens, J., Bunel, R., Roy, A. G., Stanforth, R., Natarajan, V., Ledsam, J. R., Mac Milliims, P., Kohli, P., Karthike- salingam, A., Kohl, S., et al. Contrastive Training for Improved Out-of-Distribuun
Scherr, F., Guo, Q., and Moraitis, T. Self-Supervised Learn- ing Through Efference Copies. In NeurlPS, 2022.
Detection. In arXiv, 2020.
Shwartz-Ziv, R., Balestriero, R., Kawaguchi, K., J.Rudner, T. G., and LeCun, An Information Theory Perspective on Variance-Invarianne Invariance-Covarian. Regularization. In NeurlPS, 2023.
Wu, C., Pfrommer, J., Zhou, M., and Beyerer, J. Generative- Contrastive Learning for Self-Supervised Latent Repre- sentations of 3D Shapes from Multi-Modal Euclidean Input. arXiv, 2023.
Shwartz-Ziv. R., Balestriero, R., Kawaguchi, K., Rudner, T. G. J., and LeCun, Y. An Information- Perspec- Regularization.
Yang, Y., Chen, S., Li, X., Xie, L., Lin, Z., and Tao, D. Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Leamable Classifier at the End of Deep Neural Network? In NewrIPS, 2022.
tive on In NewrIPS, 2023.
12

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
Zbontar, J., Jing, L.,, Misra, LeCun, Y., and Deny, S. Barlow Twins: Self-Supervised Learning via Redundancy Reduction, In ICML, 2021.
Zhang. S., Zhu, F., Yan, Zhao, R., and Yang, X. Zero- CL: Instance and Feature Decorrelation for Negative-Free Symmetric Contrastive Learning. In ICLR, 2021.
Zhou, J., Wei, C., Wang, H., Shen, W., Xie, C., Yuille, A. I.., and Kong. iBOT: Image BERT Pre-Training with Online Tokenizer. ICLR, 2022.
Zhue, Y., Rengiang, R. M. M., Kadav, A., and Graf, H. P. S3VAE: Self-Supervised Sequential VAE for Represen- tation Disentanglement and Data Generation. In CVPR, 2020.
Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. Contrastive Learning Inverts the Data Generating Process. In ICML, 2021.
13

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
Appendix
A. Training Statisties
provide some examples demonstrating the effectiveness introducing linear and batch normalization layers in the projector Fig. 7. This contributes to prevent excessing increase of mean or variance. An alternative and promising approach to the linear and batch normalization layers is to penalize the norm of the representations directly in the objective. We leave this to future investigation.
75-
0-
-25
-50
-79
100
120
20
80
100
120
-00
Represencation
(a) checkpoint 50 epochs
checkpoint 200 epochs
Figure 7: Example of mean and standard deviation statisties for the representation features obtained by the backbone network trained on SVHN data. Statistics are computed for batch of size of 100 samples. Results corresponds to checkpoints for the projector (c 4096) without (top) and with (bottom) linear and batch normalization layers. Linear and batch normalization layers contribute to stabilize the training by avoiding mean or variance increase.
B. Comparison of Different Activations for the Projector
We provide a comparison between activation functions on SVHN, CIFAR- 10 and CIFAR-100 Table 4. For both activations, we use 3 according to Table
C. Discussion on Finite Capacity
It is important to mention that the global minima for the CPLearn objective might not be reached when using backbone network of finite and small capacity. In this case, the avoidance of representation and cluster collapses can still be guaranteed when the invariance and the matching prior losses are both minimized. Indeed, we observe that for representation collapse Pi Pj for E [m], j [c] (i.e. outputs of the overall network are constant with respect to their inputs) and that the corresponding mimimum value of the objective is given by the following formula
LCPLeam (D) = (9H(p) + CF(q.P)
14

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
Table 4: Test generalization on downstream tasks including clustering and supervised linear probing. Performance are measured terms of normalized mutual information (NMI,, accuracy (Ace.) and are averaged over 5 training runs obtained from random initialization seeds. We test CPLear using L.2-Norm and Tanh activations.
Clustering (NMI)
Supervised Linear Probing (Acc.)
Activation
SVHN
CIFAR-10
CIFAR-100
SVHN
CIFAR-10
CIFAR-100
L2-Norm
0.14=0.02
0.30=0.01
0.16#1.01
0.64+0.02
0.60+
0.15
10
Tanh 0.11=0.01
0.28
0.15+0.00
0.60#0.00
0.59
0.15
L2-Norm 0.16=0.00
0.26-1.00
0.3310.01
0.59
0.60
0.20=10011
f
Tanh 0.16=0.02
0.25
0.33+0.00
0.60
0.59+00.01
0.18
L2-Norm
0.31=0.00
0.35
0.58+0.00
0.78
0.68+10.00
0.41
16384
Tanh
0.29
0.00
0.59+00.00
0.75++0.00
0.68++0.00
0.40+00.00
where the first addend arises from the invariance loss, whereas the second one arises from the matching prior one. Notably, the terms cannot be minimized the same time due their competitive nature. For instance, in the case of uniform q. the solution of p q minimum for the matching prior loss but not for the invariance one (this is actually saddle point, as corresponding to the maximum for the entropy term in the above equation).
Cluster collapse occurs whenever 3j, k # E [c] such that for all E [r]. Pt S Pik. The minimization of the invariance loss forces the whole network to make low entropy predictions, whereas the minimization of the matching prior loss forces to distribute these predictions across all codes according to q. Hence, when both losses are minimized cluster collapse avoided.
D. Minima of the CPLearn Loss
Proof. We recall here the loss
LCPLeorn D
Pij log -
Σ 1=1
Lj
log
Pi
1=1 1=1
[=1
and prove all optimality conditions. Before doing that, we observe that the loss convex w.r.t. P when P' fixed, as the first addend a sum of linear terms, whereas the second addend sum of convex terms. Similarly, we observe that convexity holds w.r.t. P' when P fixed by exploiting the same reasoning. However, is important to mention that the loss not convex globally. This can be shown firstly by computing the Hessian of the first addend w.r.t. both P and P and secondly by observing that the Hessian is not positive semi-definite (we skip the tedious calculation of the Hessian).
Invariance. We observe that P appears only in the first addend of LcPLeam and that this addend can be equivalently rewritten in the following way:
B
3
P/j
Pi log =
log
Pij
log
pij
j=
i=1
i=1
1=1
8
H (p:)
KL(p:llp!)
(6)
where (.), KL.) are the entropy and Kullback-Leibler divergence, respectively. Therefore minimizing LCPLeom w.r.t. P equivalent to minimizing Eq. 6. The solution is given by p. = p/c Vi E [n], thus proving the invariance condition.
Extrema. We first leverage the invariance condition, p. P/. Vi E [n). and rewrite LCPLeam accordingly:
3 Pij log Pij
Lepteam
Σ
Lj log
Σ
Pij
(7)
11
11
==
1=1
15

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
We observe that the loss in Eq. 7 convex w.r.t. P. Therefore, closed-form solutions for the minima of the second addend in Eq. 7, and then constraining the optimization of the first addend with these solutions and deriving the corresponding minima.
obtain its optimality conditions, by deriving the
Let's start by considering the following constrained convex minimization problem, obtained from the first addend in Eq. 7, with n, 3 being dropped as being constant for the optimization:
min P
Pig
logPij
i=1
j=l
Σ Pij = L, Vi [n]
s.t.
j=1
c PS Pij S de - 1), ( [c)
(8)
and the corresponding Lagrangian with multipliers A, A R R is:
Píj log Pij + Σ
1/8
Σ 1=1
Pij - -1
1
j=1
1=1
Dayle - Pij) + 5ig
clc 1))1
(9)
We observe that the Lagrangian is constructed so to satisfy the following relation
Ptj log Pıj 2 (P::A.
(10)
Let's maximize L1 w.r.t. P by setting P.y L 0. This leads the following closed-form expression:
Pig = e to Vi E [m]..j E [c]
(11)
By evaluating L1 at the solutions in Eq. 11, obtain the Lagrange dual function
= =n+
Σ
1i +
Aie 81₃(1 - c(c 1))]
(12)
i=1
The Lagrange multipliers in Eq. 12 depend on the values of P through the Karush-Kuhn- Tucker (KKT) conditions. We distinguish two main cases for P*, each leading to different evaluation of the Lagrange dual function:
Case 1. When all probability values touch their extrema, such as
Vi E [n], e [c], [c] with + s.t. pij = clc c(c-1) and Pic =e
By the KKT conditions (i.e. complementary slackness), we have that Aij = 0 and 8,z = O, whereas Ask 2 0, dzg 2 0.
By substituting these conditions in Eq. 12, we obtain that
Σ
- 81 e(c 1)) + Σ Inme
(13)
kgj
16

---

Collapse-Proof Non-Contrastive Seetpperr Learning
By taking into account also Eq. 11, we have that vi E [m], Bij E [c], Ay [c]
iGg P4 + log(1 <(cc 1)) and λsk -1 11+ 14 loge
(14)
And by substituting Eq. 14 into Eq. 13, we obtain that
-(c (c-1-110 1)) ()) and Eq 14 n(1 E(c -l
ne(c - 1) loge
(15)
Case 2. When all probability values never touch the highest extrema, such
Vi E [m].] E [c], s.t. Pij < 1 e(c 1)
By KKT conditions, we have that dig = 0. By substituting these conditions in Eq. 12, we obtain that
L A.P': =
U4 +
Xipe
(16)
j=1
which always satisfies the inequality
G/P":AAA....//100mm02222222 A,
( (A0=
(17)
and therefore also
L (.
A, A, ,,, and Eq.
(18)
Finally, we observe that the objective of the optimization problem of Eq. 8 evaluated at the solutions of Case / is
= (.P.
(19)
Ei.
1=1
And by leveraging also the result in Eq. 18, we can state that the solutions of Case are the global minima of the objective in Eq. 8. Thus concluding the proof for the extrema condition.
Matched prior. We consider the minimization of the second addend in Eq. 7 subject to the extrema condition
min
Lj log
Psj
Σ Pij = 1, Vi E [n]
s.t.
#=1
Pij E
- e(c -e,1-e(ee
1)}. Vie [r],j e [c].
(20)
Let's define βú Σn Ptj for all j [c] and observe that Σラ Pj and < ps 5 -e(c 1). Therefore, can rewrite the problem in Eq. 20 equivalently
min
9i
log
Ø=1
Σ p0j = 1,
s.t.
]=1
Pj
E - 1),
Vj E [c],
(21)
17

---

Collapse-Prool Non-Contrastive Self-Supervised Learning
Now, we observe that the optimization objective satisfies the following equality
lo (q) + kL
(22)
The minimum for Eq. 22 is obtained q = p and this solution satisfies the constraints in Eq. 21 because S qj S 1--(cc1) for all j [c] (indeed have that 1.2 - Pij = 4j for all j E [c]-
always choose to satisfy the inequality), thus being the global optimum. In other words,
Finally, recall that maz (i) = {i E [m] Pij = 1 1-e 1)J. Vj E [c]. which identifies all elements having the highest possible value of probability in P. We observe that
Σ
Σ
Péj
Pij +
Psj
iElman() (f)
igLman (f)
Σ
(1 6(c 1))
Σ
(by Extrema condition)
iElmao(]) (j)
i (
Mmaz (j)l(1 Cε) + me
By the condition #/2 Ei Pij = Oj and the above relation we have that
max (i) ce) + ngj:
vjj E
equivalently that
mm (i) = 1 αC
(23)
Now, for the case of uniform prior, Eq. 23 becomes
n
Ima (i)==
Vj E [c]
(24)
This concludes the proof for the matching prior condition.
Finally the global minimum value of the CPLearn objective can be obtained by dividing Eq. by n and adding the entropy term (as for the result obtained by the matched prior condition). This concludes the proof of the Lemma.
E. Embedding Theorem
Proof. Recall the extrema condition from Lemma 1, that
Vi E [m), Alj e [c], Vk E
[c] with j.t. Piy = =1 1-6(cc- 1) and Pij =
Moreover, due to orthogonality of W Ogy R, This fact leads to the following equation
express the Span condition, i.e. h, ES= wy for all i E [m] with
3 hu/T
as5l
Pij
=1 f/7
Vi E [n], E [c]
(25)
Dj'
Spon
Combining the extrema condition with Eq. 25 gives us a system of equations for each i [n]
culte
= -1-(cc- c
Ejsre e19777 TIgI Salle
=
Vk +
18

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
By taking the logarithm both sides of the equations and resolving the above system, the solution is equal
T aij f
[1<(c--1
1-
Oik
log
-rij"
4k j
(26)
Vn
where the last equality holds due the choice 5/(VV log(11 -e(c- 1))/e))- Using Eq. 26 in the Span condition gives us the following result
vi E [n], [c] s.t. h; OnjWy + (iij
wk
(27)
Note that the oij could potentially take any value in dij E R. This is not allowed as embeddings are normalized by design choice (cf. Eq. 1). Indeed, the norm of the embeddings can be rewritten to exploit Eq. 27
1 h h
2(c- - 1)5 ai + f (c- 1)
cfa
(28)
Vn
Eq.2
and by equating Eq. 28 to the fact that embeddings are normalized | L for all i E [n] we obtain the following quadratic equation
2(c B(c-1)5
1)5
aij + L {c (c- - 1)
L
af
Vn
n2
n
whose solutions are given by
Ohij
n
2)
V
This concludes the proof.
F. Diagonal Covariance
Proof. Recall from Theorem that
Vi E [n], e [c] s.t. h; = 0i +
Oij
Σ
wk
Vn
By assumption Oaj- = vn and therefore
1
E [nj. E [c] s.t. h;
w n
(29)
meaning that the rows H are equal up to a constant to the codes the dictionary and that they span the same space the columns of W. namely the whole embedding space. We can therefore express H as linear combination of W.
Without loss of generality, dictionary. Therefore, by combining this with Eq. 29 we have that
always define H
ensure that nearby rows associated to the same codes in the
H AT
19

---

Collapse-Prool Non-Contrastive Selleuprrrsdd Learning
with
T
n/c
A
vn
o/c
RRex
1
Vn n/c/
where 1n/e is vector containing n/cc ones (whose size follows due the assumption on uniformity of q). Importantly matrix A satisfies the following property
AA" I
(30)
Therefore, we have that
WWT
Eq.
where the last equality simply follows by the orthogonality condition T W = =fI and the fact that W is square matrix (c = f). Indeed, we have that
WTW = =I
WWWW W |II
(WW (
WWT =I
thus concluding the proof.
G. Generalization to Supervised Linear Downstream Task
We first observe that by the results of Theorem and the uniformity of q, H has full rank. Moreover, considering that H is function of Z through the first layer of the projector in Eq. 1, Z must be also full rank. As consequence,
z Z has full rank. (Full Rank Property)
(31)
Now, we recall existing result for generalization to supervised downstream tasks from (Shwartz-Ziv al., 2023) (Section 6.1) and demonstrate that the Full Rank Property reduces the generalization error.
Indeed, consider a classification problem with classes. Given an unlabeled dataset DD used for training CPLearn with the corresponding unknown ground truth labels YD E RRax and a supervised dataset S = {(x,, ) with yi being the rows the label matrix Ys x define Z E g and Rmxf the representations obtained by feeding datasets D and S, respectively, through the backbone network g. Moreover, define
Pb = I I- Z Z)
Ps = I-2 2
where symbol . denotes the pseudo-inverse. Now, suppose we train linear classifier with parameters U R/xr on the latent representations obtained from dataset through the following supervised loss
(U) [lg(x) 211 |U||
Then, we can state the following theorem
20

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
Th. 1 (restated from (Shwartz-Ziv al., 2023)). vg with probability at least - 6, we have that
2
Ex Ex.yl/a.y
-
(1 - 2( + ED
sup Σ Eill((z.) - s(æi)lle) }+
2 Vn
PoYpll=+ - - PsYsl + consi(n, m)
(32)
Vm
where& & is vector of ii.d. Rademacher random variables.
Therefore, the expected supervised loss in Eq. 32 can be reduced by minimizing its upper bound. Note that the first addend Eq. is minimized by the CPLearn loss, whereas the second addend is also statistically minimized when n large. The third addend refers the contribution term for the classification the unlabeled data. While ground truth Yo unknown, this addend can be minimized by exploiting the following relation
PoYYlle & (Pole
Indeed, note that in order to minimize the left-hand side of the inequality, suffices to minimize the term |Pol r. which occurs when Z has maximum rank. This is our case due to the Full Rank Property. Finally, by the same argument used for the third term in Eq. 32, we can minimize the fourth one by having 2 Z with maximum rank. This condition holds because z Z and Z Z concentrate to each other by concentration inequalitites (cf. (Shwartz-Ziv et al., 2023) for more details).
To summarize, minimizing the CPLearn loss ensures that we reduce the invariance of representations to data augmentations and increase the rank of the representation covariance. This leads to decrease of the generalization error as from the result of Theorem 1.
H. Block- Diagonll Adjacency
Proof. The proof follows step step the one for the diagonal covariance except for the fact that
HH A
fA" A= JBA
WTW f1
where
1ax
0
0
0
11
0
BA = A
IR
0
0
11
and 1 is a matrix of ones. This concludes the proof.
Experimental Details on SVHN, CIFARIO and CIFAR100
Training. We used a ResNet-8 (details are provided in Table 5. We consider the hyperparameters in Table 6 for training. Beta is chosen to ensure both losses are minimized, ef. Table 7.
Evaluation. For linear probe evaluation, we followed standard practice by removing the projector head and train a linear classifier on the backbone representation. We train the classifier with Adam optimizer for 100 epochs and learning rate equal 2.
J. A Aditional Results on Dictionary Size
We provide additional visualization results for the covariance and adjacency matrices on SVHN and CIFAR- 10, ef. Figs. 8, 9. Moreover, we add the analysis of generalization on downstream tasks on SVHN and 1IFAR-100 varying the size of the dictionary in Figs 10, 11.
K. Qualitative and Quantitative Analysis of Collapses
In Fig. 12, we provide some qualitative evidence on the avoidance of representation and cluster collapses. Indeed, when
21

---

Collapse-Proof Non-Contrastive Sel--Supervised Learning
Table 5: Resnet architecture. Conv2D(A,B,C) applies a 2d convolution to input with B channels and produces an output
with C channels using stride (1, 1), padding (1, 1) and kernel size (A, A).
Name
Layer
Res. Layer
Conv2D(3,3.F)
AvgPool2D(2)
LeakyREL.U(0.2)
Conv2D(3.F.FF)
Block
Conv2D(1,,,F) no padding
AvgPool2D(22)
Sum
LeakyRELU(0.2)
Conv2D(3.F.F)
Block 2
LeakyRELU(0.2)
Conv2D(3.F.F)
AvgPool2D(22)
LeakyRELU(0.2)
Conv2D(3,F.F)
Block 3
LeakyREL.U(0.2)
Conv2D(3.F.F)
LeakyRELU(0.2)
Conv21((,FFF))
Block 4
LeakyRELU(O.2))
Conv2D(3.F.F)
AvgPool2D(all)
both losses in the CPLearn objective are minimized these two failure modes are avoided. Additionally, we investigate dimensional collapse following the methodology proposed in previous work (Jing et al., 2022) by computing the singular value distribution of the covariance matrix for the embeddings. In Fig. 13a, we observe that for the undercomplete setting only 10 singular values have large values. This is explained by the fact that embeddings align with the 10 codes in the dictionary, as predicted by Theorem When e increases, more and more singular values increase in their value. This provides evidence that the loss function in conjuction with the proposed projector allows exploit the whole embedding space and avoid dimensional collapse. Finally, we propose to study intracluster collapse by estimating the entropy of the distribution for the representations. We do so by (i) fitting a Gaussian mixture model with diagonal covariance on the representation from the backbone, (ii) estimating the entropy of the distribution through Monte Carlo using 10k samples and (iii) repeating the analysis for different number of mixture components, i.e. (10, 20, 50, 100, 500, 1000). Intra-cluster collapse avoided when achieving higher values of entropy. We illustrate this in Fig. 13b, showcasing improved performance for larger values of dictionary size. We provide additional results for the collapses on SVHN and CIFAR100. Specifically, in Fig. 14 show the analysis of dimensional collapses, whereas in Fig. 15 we show the one for intracluster collapse.
L. Experimental Details and Additional Analysis with ResNet-18
Experimental details for ResNet-18. We used a ResNet-18 backbone network on CIFAR-10 10 and train it for 1000 epochs with Adam optimizer, learning rate equal to 1e - 3 and batch-size equal to 64 on 1 A100 GPU. Beta is selected from a smaller subset of values 10.1, 0.25, 0.5, 1} (given the more expensive nature of the experiments) to ensure both losses are minimized and chosen being equal to
Training convergence. We provide comparison of the linear probing performance over training on the experiments for ResNet-18 18 on CIFAR-10. 10. We visualize the topl accuracy over training epochs both linear and logarithmic scales. We observe that CPLearn achieves better convergence rate compared to the other approaches, thanks to the quasi-orthogonality condition of the code matrix W. Results are shown in Fig. 16.
Time/storage analysis. Table we compare all methods in terms storage (MB) and training time (minutes) using a ResNet-13 18 trained for 1000 epochs on CIFAR-10. We also provide the linear classification and clustering results for the sake of completeness. The training time does not seem to be impacted much by the larger projector, The size the checkpoints
22

---

Collapse-Prool Non-Contrastive Self-Supervised Learning
Table 6: Hyperparameters (in terms of optimizer and data augmentation) used in SVHN, CIFAR-10 and CIFAR-100 experiments.
Class
Name param.
SVHN
CIFAR-10
CIFAR-100
Color jitter prob.
0.1
0.1
0.1
Gray scale prob.
0.1
0.1
0.1
Data augment.
Random crop
Yes
Yes
Yes
Additive Gauss. noise (std)
0.03
0.03
0.03
Random horizontal flip
No
Yes
Yes
Batch size
64
64
64
Epochs
20
200
200
Adam $1
0.9
0.9
0.9
Optimizer
Adam B3
0.999
0.999
0.999
Learning rate
le -
le
1e 1e-4 4
Table Values of 3 hyperparameter. This is chosen from the range {0.01, 0.05, 0.1, 0.25, 0.5, 2.5, 103 ensure that both losses minimized.
Dictionary Size
128
256
512
1024
2048
4096
8192
16384
SVHN
0.5
0.5
0.25
0.25
0.1
0.1
0.1
0.05
0.05
CIFAR-10
0.5
0.5
0.25
0.25
0.1
0.1
0.1
0.05
0.05
CIFAR- 100
0.5
0.5
0.25
0.25
0.1
0.1
0.1
0.05
0.05
Method
Clustering
Linear
State dict size [MB]
Full checkpoint size [MB]
Training time [min]
Barlow
29.1
92.2
79
157
356
SwAV
18.9
89.6
102
405
GEDI no gen
44.6
80.0
47
140
353
Self-Classifier (c f)
36.9
84.8
51
144
353
Self- ClffsClassifier (16384)
33.9
64.9
175
268
355
CPLear (c = f)
47.4
91.6
51
144
357
CPLear (16384)
48.2
91.3
358
Table 8: Comparison of methods clustering and linear evaluation, with model sizes and training times.
do increase significantly. However, the relative difference would be much smaller with larger networks such as ViT-S/B/L or larger ResNet architectures as the backbone itself would be much larger compared to the additional parameters in the projector.
Linear
2.0
10.0 (cluster collapse)
1.5
10.0 (cluster collapse)
1.25
10.0 (cluster collapse)
1.0
91.6
0.75
90.2
0.5
88.1
0.25
82.7
0.1
75.1
Table 9: Effect of varying 8 on linear evaluation performance.
Sensitivity analysis (on 3. We conducted additional experiments evaluate the performance of CPLearn across a finer
23

---

Collapse-Prool Non-Contrastive Self-Supervised Learning
(a) c 10
(b) 128
(c) c 16384
cd) 10
(e) C 128
(f) C 16384
Figure 8: Realization of embedding covariance (left) and adjacency matrices (right) for the whole SVHN test dataset. Increasing reduces the value of the off-diagonal elements of the covariance, thus contributing to increase the decorrelation of features (ef. Corollary 2). Moreover, increasing has the effect to reduce the block sizes of the adjacency matrix (cf. Corollary 3).
c 10
(b) = 256
(c) c 16384
(d) c 10
(c) = 256
(f) c 16384
Figure 9: Realization of embedding covariance (left) and adjacency matrices (right) for the whole CIFAR- 100 test dataset. Increasing e reduces the value of the off-diagonal elements of the covariance, thus contributing to increase the decorrelation of features (cf. Corollary 2). Moreover, increasing has the effect reduce the block sizes of the adjacency matrix (cf. Corollary 3).
NMI
Accuracy
30
70
20
60
10
50
10
128
256
512
1024
2048
4096
8192
16384
10
128
256
512
1024
2048
4096
8192
16384
Dictionary size
Dictionary size
Figure 10: Analysis of downstream generalization for different values dictionary size on SVHN dataset.
NMI
Accuracy
60
40
40
20
20
10
128
256
512
1024
2048
4096
8192
16384
10
128
256
512
1024
2048
4096
8192
16384
Dictionary size
Dictionary size
Figure 11: Analysis of downstream generalization for different values of dictionary size CIFAR-100 dataset.
24

---

Collapse-Prool Non-Contrastive Sel--Supervised Learning
Mutringg Proo
EnVra000
S
0.2
A0000 19000 180000
8000
19000 10300
Muadiing Praor
Inveriace
4.00
3.28
Sandbooa
Mitching
Inastance
0.50
0.00
0.30
Hensone
Farakrr
(a) Training losses
(b) Adjacency matrix
(c) t-SNE visualization
Figure 12: Example of bad (top with 8 = 0 and middle with 3 = 10) and well-behaved (bottom, 3 = 0.1) training loss dynamics on CIFAR-10 10 with dictionary size 8192. When only one term is minimized, the model faces cluster collapse, as demonstrated by the adjacency plots in the top and middle rows (corresponding to and clusters, respectively). However, when both losses are minimized the collapse avoided. Interestingly, the visualization of the representations reveals the absence of representation collapse in all cases (colors are used to denote different ground truth classes).
25

---

Collapse-Proof Non-Contrastive Seleeppeeriid Learning
10
512
4096
10
512
4096
128
1024
8192
128
1024
8192
Log of Singular Values
256
2048
16384
256
2048
16384
Entropy Estimate
-10
220
-5
-250
-10
-300-
-15
20
40
60
80
100
120
10
102
10
Singular Value Rank Index
Number of Mixture Components
(a) Dimensional collapse
(h) Intracluster collapse
Figure 13: Collapse analysis on CIFAR- 10 test data for different dictionary sizes c. Results are averaged over 5 training runs obtained from random initialization seeds. Left: The singular values the embedding covariance are sorted order and logarithmic scale. The curve rises with very large values of c, avoiding zero singular values. Right: The number mixture components are in logarithmic scale. The curve rises with very large values of for all number of mixture components.
range of values, using ResNet-18 CIFAR-1O. The results are presented in Table We observed that performance improves 3 increases. However, excessively large values of / lead to cluster collapse. This occurs because minimizing only the invariance loss ensures invariance and low-entropy predictions, but does not guarantee satisfaction of the matched prior condition described in Lemma 1. In other words, the network tends to use only a subset of codes, resulting in cluster collapse.
M. Experimental Details on ImageNet-1000
Training. We used ViT-small backbone network and train it for 100 epochs with learning equal 5e 4 and batch-size per GPU equal to 64 on node with 8 NVIDIA A100 GPUs. Beta is selected from smaller subset of values {0.1, 0.25, 0.5, 1} (given the more expensive nature of the experiments) to ensure both losses are minimized and chosen being equal to 0.25.
Evaluation. For linear probe evaluation, we use the DINO codebase and train the classifier with Adam optimizer (Caron al., 2021).
N. Practical Implementation of the Loss
We observed training instability when using the larger backbone ImageNet- 100. The issue is due some dictionary codes being unused during the initial training phase (cluster collapse), making the KL matching prior term infinity. Indeed, we have that
LCP (D) (CF(p CFFg..P)
x xBCF(pp.p) + KT(q.p)
In practice, the reverse KL term is sufficient to avoid the issue:
LCPLeam (D) =BCE(p.p)) KL(p.9)
26

---

Collapse-Proof Non-Contrastive Self-Supervised Learning
10
4096
128
1024
8192
128
1024
8192
Log of Singular Values
256
2048
16384
Log of Singular Values
256
2048
16384
0
-5
-5
-10
-10
-15
-15
20
40
60
80
100
120
50
100
150
200
250
Singular Value Rank Index
Singular Value Rank Index
(a) SVHN
(b) CIFAR-100
Figure 14: Dimensional collapse analysis on test data for different size of dictionary. Results are averaged over training runs obtained from random initialization seeds.
10
512
4096
512
4096
128
1024
8192
128
1024
8192
256
2048
16384
256
2048
16384
Entropy Estimate
-150
Entropy Estimate
-300
200
-40
-250
-500
-300
-350
-600
10
102
103
10
102
103
Number of Mixture Components
Number of Mixture Components
(a) SVHN
(b) CIFAR-100
Figure 15: Intracluster collapse analysis on test data for different size dictionary. Results are averaged over training runs obtained from random initialization seeds.
27

---

Collapse-Prool Non-Contrastive Sel--Supervised Learning
90
90
Self-Classifier
GEDI gen
80
80
SwAV
top-1 accuracy [%]]
top-1 accuracy [%]]
Barlow
70
70
CPLeam
60
60
50
Self-Classifier
50
GEDI no gen
40
40
SwAV
30
Barlow
30
CPLearn
20
20
200
400
600
800
1000
100
101
102
103
epoch
epoch (log-scale)
Figure 16: Analysis of training convergence.
28


Reviews:
[{"review_id": "UHGt1CMq32", "reviewer": "Reviewer_KMvq", "summary": "The paper introduces CPLearn, a non-contrastive self-supervised learning method designed to avoid common failure modes—namely, representation, dimensional, cluster, and intracluster collapses. The authors propose a simple projector design and loss function, leveraging ideas from hyperdimensional computing, that naturally encourage both decorrelation and clustering of embeddings without relying on heuristics like stop gradient or momentum encoders. The method is theoretically analyzed, and its properties are established through several results. Experimental results are conducted on datasets such as SVHN, CIFAR-10, CIFAR-100, and ImageNet-100 to test for robust generalization in clustering and linear classification tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating our work and the constructive review.\n\n**Validation of quasi-orthogonality** $W^TW=fI$ holds in probabilistic terms, formally governed by Eq. 5 in the paper, namely in expectation we have $E_W[cos(w_i,w_j)]=\\delta(i-j)$, with $\\delta$ being a Kronecker delta function (equal to 1 iff $i=j$ and zero otherwise), and its variance is determined by $Var_W[cos(w_i,w_j)]=1/f$. For large $f$, the variance quickly approaches zero and the equality holds almost surely.\n\n**Baselines on ImageNet-100** Please find below the table of results with the addition of GEDI no gen and SwAV. The best performance are highlighted in bold for each method. Overall, we observe that CPLearn achieves linear probing performance comparable to the best methods, such as SwAV and Barlow. Moreover, CPLearn significantly outperforms all other approaches in terms of clustering performance. This stems from the unique capability of the model to perform both clustering and feature decorrelation.\n\n| | **Clust** |  |  |  |  |  | **Linear** |  |  |  |  |  |\n| - | - | - | - | - | - | - | - | - | - | - | - | - |\n| **Projector** | **Small** | | | **Large** | | | **Small** | | | **Large** | | |\n| **Method**                | **128**   | **1024**  | **2048**  | **32768** | **65536** | **131072** | **128**   | **1024**  | **2048**  | **32768** | **65536** | **131072** |\n| Barlow           | 30.0% | 53.6% | **59.4%** | oom | oom | oom | 66.9% | **77.2%** | **77.2%** | oom | oom | oom |\n| SwAV             | 28.0% | 47.9% | 51.2% | 60.7% | 60.5% | **62.8%** | 76.6% | 77.7% | **78.0%** | 76.6% | 77.7% | 77.3% |\n| DINO             | 46.0% | 53.4% | 55.2% | 63.1% | **64.7%** | 64.3% | 71.8% | 73.6% | 73.9% | 75.1% | **76.2%** | 75.8% |\n| GEDI no gen    | 24.5% | 36.3% | **38.8%** | 32.7% | 32.9% | 33.3% | 71.8% | **73.2%** | 72.8% | 72.9% | 72.7% | 72.8% |\n| CPLearn           | 34.1% | 54.3% | 57.9% | 69.3% | 69.6% | **70.9%** | 70.8% | 72.8% | 70.1% | 74.5% | 74.7% | **78.0%** |\n\n**Experiments on Resnet-18** Please find below the table with the results on CIFAR10 using ResNet-18. We used the [solo-learn](https://github.com/vturrisi/solo-learn) codebase to run the analysis with the same suggested hyperparameter configurations. Note that we reproduced all baselines ourselves and that their performance are slightly higher than what reported in [solo-learn](https://github.com/vturrisi/solo-learn). Also please note that NMI is a standard and well-established metric to measure clustering performance. Overall, the table highlights similar observations to the ones from experiments on ResNet-8, with CPLearn achieving comparable performance to Barlow Twins and significantly outperforming all other approaches in terms of clustering. Interestingly, both Self-Classifier and CPLearn achieve the highest clustering performance for $c=10$, corresponding to the ground truth number of classes.\n\n| Method                   | Clustering | Linear |\n| - | - | - |\n| Barlow                    | 29.1       | **92.2**   |\n| SwAV                      | 18.9       | 89.6   |\n| GEDI no gen               | 44.6       | 80.0     |\n| Self-Classifier (c=10)    | 54.2       | 78.0     |\n| Self-Classifier (c=512)   | 36.9       | 84.8   |\n| Self-Classifier (c=16384) | 33.9       | 64.9   |\n| CPLearn (c=10)            | **62.1**       | 85.4   |\n| CPLearn (c=512)           | 47.4       | **91.6**   |\n| CPLearn (c=16384)         | 48.2       | 91.3   |\n\n**Clarity about collapses** We are going to move Figure 1 (showcasing all collapses) in the 1st page and then reference it at line 15 to introduce it early on.\n\n**Design choices**. The activation function in $H$ is chosen in a way to ensure the validity of Theorem 1. We have an extra linear layer with batch norm in $H$ to ensure well-behaved statistics throughout training (as shown in Appendix A). $P$ is a standard linear layer with softmax activation. \\\nThere are several ways to define random code vectors in hyperdimensional computing (HC). We chose to use the multiply-add-permute encoding, which leverages a Rademacher distribution. We refer you to a recent survey on HC for more details [1]. This is the simplest form of encoding equipped with simple element-wise addition and multiplication operations to perform algebraic compositions. We are going to add this in the paper and reference the survey. It is also important to mention that the exploitation of the compositional properties of HC is beyond the scope of the paper, but we are actively working on this direction. \\\nWe are going to explicitly mention that Eq. 2 consists of a sum of two cross-entropy losses (this can be also seen from Algorithm 1 in the main paper).\n\nWe hope we've effectively addressed your concerns and encouraged a reconsideration of your rating.\n\n**Reference** \\\n[1] A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations. ACM Comp. Surv. 2022", "claims_and_evidence": "The claims are supported by theoretical proofs (Lemma 1, Theorem 1) and empirical validation. The proofs leverage convex optimization and quasi-orthogonality assumptions, and experiments demonstrate improved generalization and collapse avoidance. However, the assumption that $W^TW\\approx fI$ for large $c$ (practical quasi-orthogonality) is not rigorously validated beyond empirical histograms (Fig. 3). Further, the experiments use a ResNet-8 backbone on CIFAR datasets, resulting in lower overall performance compared to more standard architectures (e.g., ResNet-18/ResNet-50).", "methods_and_evaluation_criteria": "The methods are novel and interesting: the projector design and loss terms directly address collapse modes, and the use of large dictionaries aligns with theoretical insights. Evaluation on standard SSL benchmarks (CIFAR, ImageNet-100) is appropriate, and metrics (NMI, linear probe accuracy) are widely accepted. Nonetheless, including experiments with more standard backbones like ResNet-18 and additional baselines on ImageNet-100 would improve the empirical analysis", "theoretical_claims": "Lemma 1’s derivation via Lagrangian multipliers and KKT conditions is sound. Theorem 1 assumes an orthogonal $W$, which is relaxed via hyperdimensional computing principles.", "experimental_designs_or_analyses": "The experimental design is generally sound, with evaluations conducted on several datasets and extensive ablation studies. The analysis of dictionary size effects (using different settings for c) and the comparisons against multiple baselines are strengths. However, the use of a ResNet-8 backbone on CIFAR datasets leads to overall low performance compared to more common architectures, making it unclear whether the improvements would hold in more standard settings. Including experiments with ResNet-18 and adding baselines for larger-scale settings like ImageNet-100 would provide a more convincing picture.", "relation_to_broader_scientific_literature": "The work bridges non-contrastive SSL (e.g., Barlow Twins) and cluster-based SSL (e.g., SwAV), addressing gaps in collapse prevention. It extends Sansone (2023) by removing the generative term and formalizing guarantees. The hyperdimensional computing connection is novel but underdeveloped. The related works section is extensive and well-contextualizes the contributions within a broad range of literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- The paper provides an in-depth analysis of clustering-related collapse, extending prior work that mainly focused on dimensional collapse.\n\n- It offers a principled, unified design that tackles multiple collapse modes simultaneously.\n\n- The related works section is comprehensive and well-informed, covering a broad range of literature.\n\nWeaknesses:\n\n- The experiments on CIFAR-10/100 use a ResNet-8 backbone, which leads to performance levels that are significantly lower (about 25%) than what is typically observed with architectures like ResNet-18. Its unclear how these gains translate to standard and more widely used architectures. \n\n- The paper omits baseline comparisons on ImageNet-100 with methods such as GEDI and SwAV, leaving it unclear how CPLearn performs in larger-scale settings compared to the closest baselines.\n\n- Based on the ImageNet-100 results, CPLearn does not consistently outperform DINO or Barlow Twins. The projection sizes at which CPLearn shows an advantage are not reported for some of these baselines, as many values are missing due to out-of-memory errors.\n\n- The paper is very hard to follow. For instance, the introduction doesn't even define representation collapse and isn't well-written and motivated. It gives a very shallow background. \n\n- In the methods section, the authors don't give any insights into their design choices corresponding to $H$ and $P$ matrices. Why is $W$ being sampled from the Rademacher distribution? Similarly, there are no insights around the final loss term. Some of the ways in which the first term promotes invariance can be easily mentioned since it minimizes the effective cross-entropy loss.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BPICy7HJd1", "reviewer": "Reviewer_nCR1", "summary": "This paper introduces CPLearn, a novel self-supervised non contrastive approach that avoids heuristics such as stop gradient or momentum encoders for feature collapse. CPLearn does this by utilizing a projector module and a special loss function which minimizes the invariance between augmented views while enforcing a prior distribution (uniform) on the features. The authors evaluated their approach on multiple different datasets, such as SVHN, CIFAR-10, CIFAR-100, and ImageNet100.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating the theoretical nature of our paper and for the time dedicated to review it. Please find below the answers to the major concerns.\n\n**Experiments on ResNet-18** Thank you for the suggestion. Please find below the table with the results on CIFAR10 using ResNet-18. We used the [solo-learn](https://github.com/vturrisi/solo-learn) codebase to run the analysis with the same suggested hyperparameter configurations. Note that we reproduced all baselines ourselves and that their performance are slightly higher than what is reported in [solo-learn](https://github.com/vturrisi/solo-learn). Overall, the table highlights similar observations to the ones from experiments on ResNet-8, with CPLearn achieving comparable performance to Barlow Twins and significantly outperforming all other approaches in terms of clustering. Interestingly, both Self-Classifier and CPLearn achieve the highest clustering performance for $c=10$, corresponding to the ground truth number of classes.\n\n| Method                    | Clustering | Linear |\n| - | - | - |\n| Barlow                    | 29.1       | **92.2**   |\n| SwAV                      | 18.9       | 89.6   |\n| GEDI no gen               | 44.6       | 80.0     |\n| Self-Classifier (c=10)    | 54.2       | 78.0     |\n| Self-Classifier (c=512)   | 36.9       | 84.8   |\n| Self-Classifier (c=16384) | 33.9       | 64.9   |\n| CPLearn (c=10)            | **62.1**       | 85.4   |\n| CPLearn (c=512)           | 47.4       | **91.6**   |\n| CPLearn (c=16384)         | 48.2       | 91.3   |\n\n**Experiments with ViT on ImageNet-100** We respectfully disagree with the reviewer. ViT is a standard backbone for benchmarking self-supervised learning algorithms, see for instance [DINO](https://github.com/facebookresearch/dino).\n\n**Typos** Thank you for spotting the typos. Indeed, the sentence “whereas SwAV, Self-Classifier and GEDI achieve overall perform better on clustering tasks” should be rephrased to “​​whereas SwAV, Self-Classifier and GEDI perform better on clustering tasks than Barlow Twins.”\n\nWe hope we've effectively addressed your concerns and encouraged a reconsideration of your rating.", "claims_and_evidence": "The experiments are conducted in a non-standard manner, thus hardly justifying if the experimental results well support the claims.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense for this application.", "theoretical_claims": "I did not check the correctness of any proofs.", "experimental_designs_or_analyses": "The experiments are sound but some of the metrics and training configurations do not necessarily match those of previous works. Specifically, previous works typically do not use ResNet 8 or ViT models as the backbone. NMI is also not a metric typically used which makes comparison with other SSL models difficult.", "relation_to_broader_scientific_literature": "As self-supervised models have been increasingly researched, many have utilized heuristics to address problems of collapse. This paper introduces a new approach to circumvent this issue and provides mathematical justification to support it can address collapse.", "essential_references_not_discussed": "There are not related works that are essential to understanding the key contributions which are missing.", "other_strengths_and_weaknesses": "This paper is well-written, clearly organized, and effectively structured, with figures and algorithmic descriptions that enhance the understanding of CPLearn. The theoretical foundations are solid, and the approach is rigorously validated across multiple datasets. However, since CPLearn leverages hyperdimensional computing, a discussion on computational resource usage and potential scalability concerns would strengthen the paper since the instability observed when using larger backbones raises concerns about the practicality of the method in large-scale settings. Providing a justification for why CPLearn remains valuable despite this limitation, or exploring possible solutions, would improve the overall discussion.", "other_comments_or_suggestions": "The paper was well written overall, but there seems to be a grammatical issue at the end of the “Generalization on downstream tasks” subsection in the sentence “…Self-Classifier and GEDI achieve overall perform better on clustering tasks.”", "questions_for_authors": "No questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o8JO67a4Gr", "reviewer": "Reviewer_vWmQ", "summary": "This paper theoretically demonstrate the conditions of avoiding four kinds of collapse in non-contrastive self-supervised learning based on the CPLearn projector design. Specifically, the authors prove that minimizing invariance to data augmentations while matching priors suffices to avoid representation and cluster collapses, whereas orthogonal frozen weights based on hyperdimensional computing, and large prediction outputs in the projector are key to avoiding dimensional and intracluster collapses. The theoretical claims are validated through experiments on benchmark datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating the theoretical nature of our paper and providing constructive suggestions.\n\n**Baselines for Table 1 and Table 2** Thank you for the suggestion. We did an extra effort in the limited time available to provide additional baselines and make Table 2 more complete. Please find below the table of results with the addition of GEDI no gen (the version without the generative model) and SwAV. The best performance is highlighted in bold for each method. Overall, we observe that CPLearn achieves linear probing performance comparable to the best methods, such as SwAV and Barlow. Moreover, CPLearn significantly outperforms all other approaches in terms of clustering performance. This stems from the unique capability of the model to perform both clustering and feature decorrelation.\n\n| | **Clust** |  |  |  |  |  | **Linear** |  |  |  |  |  |\n| - | - | - | - | - | - | - | - | - | - | - | - | - |\n| **Projector** | **Small** | | | **Large** | | | **Small** | | | **Large** | | |\n| **Method**                | **128**   | **1024**  | **2048**  | **32768** | **65536** | **131072** | **128**   | **1024**  | **2048**  | **32768** | **65536** | **131072** |\n| Barlow           | 30.0% | 53.6% | **59.4%** | oom | oom | oom | 66.9% | **77.2%** | **77.2%** | oom | oom | oom |\n| SwAV             | 28.0% | 47.9% | 51.2% | 60.7% | 60.5% | **62.8%** | 76.6% | 77.7% | **78.0%** | 76.6% | 77.7% | 77.3% |\n| DINO             | 46.0% | 53.4% | 55.2% | 63.1% | **64.7%** | 64.3% | 71.8% | 73.6% | 73.9% | 75.1% | **76.2%** | 75.8% |\n| GEDI no gen    | 24.5% | 36.3% | **38.8%** | 32.7% | 32.9% | 33.3% | 71.8% | **73.2%** | 72.8% | 72.9% | 72.7% | 72.8% |\n| CPLearn           | 34.1% | 54.3% | 57.9% | 69.3% | 69.6% | **70.9%** | 70.8% | 72.8% | 70.1% | 74.5% | 74.7% | **78.0%** |\n\n**Clarity: choice of Rademacher distribution** Thank you for the curious question. There are several ways to define random code vectors in hyperdimensional computing (HC). We chose to use the multiply-add-permute encoding, which leverages Rademacher distribution to sample them. We refer you to a recent survey on HC for more details [1]. This is the simplest form of encoding equipped with simple element-wise addition and multiplication operations to perform algebraic compositions. We are going to add this in the paper and reference the survey. It is also important to mention that the exploitation of the compositional properties of HC is beyond the scope of the paper, but we are actively working on this direction. \\\nAlso, it is important to clarify that the code matrix W is fixed throughout training. This is necessary to achieve the collapse-proof property as demonstrated by the theory in the paper (cf. Theorem 1 and related Corollaries).\n \n**Refer**  \\\n[1] Kleyko et al. A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations. ACM Comp. Surv. 2022\n\n**Time/storage analysis** We provide a table for the experiments with ResNet-18 trained for 1000 epochs on CIFAR10 comparing all methods in terms of storage (MB) and training time (minutes). We also provide the linear classification and clustering results for the sake of completeness. The training time does not seem to be impacted much by the larger projector. The size of the checkpoints do increase significantly. However, the relative difference would be much smaller with larger networks such as ViT-S/B/L or larger ResNet architectures as the backbone itself would be much larger compared to the additional parameters in the projector.\n\n| Method                    | Clust | Linear | State dict size [MB] | Full checkpoint size [MB] | Train time [min] |\n| - | - | - | - | - | - |\n| Barlow                    | 29.1       | 92.2   | 79                   | 157                       | 356                 |\n| SwAV                      | 18.9       | 89.6   | 53                   | 102                       | 405                 |\n| GEDI no gen               | 44.6       | 80     | 47                   | 140                       | 353                 |\n| Self-Classifier (c=10)    | 54.2       | 78     | 47                   | 140                       | 347                 |\n| Self-Classifier (c=512)   | 36.9       | 84.8   | 51                   | 144                       | 353                 |\n| Self-Classifier (c=16384) | 33.9       | 64.9   | 175                  | 268                       | 355                 |\n| CPLearn (c=10)            | 62.1       | 85.4   | 47                   | 140                       | 350                 |\n| CPLearn (c=512)           | 47.4       | 91.6   | 51                   | 144                       | 357                 |\n| CPLearn (c=16384)         | 48.2       | 91.3   | 175                  | 268                       | 358                 |\n\n**Typo Lemma 1** Indeed that is a typo. Thank you for spotting it !\n\nWe hope we've effectively addressed your concerns and encouraged you to reconsider your rating.", "claims_and_evidence": "Yes, the claims are well supported.", "methods_and_evaluation_criteria": "- The NMI and accuracy of the proposed method are tested on several benchmark datasets, whereas in Tables 1 and 2 the selection of compared methods are different. I think the authors should explain why.\n- The experiments are conducted on small benchmark datasets. Additional results on e.g. full ImageNet would be more convincing.\n- The accuracy improvements in Table seem marginal.", "theoretical_claims": "- I skimmed through the proofs but did not check the details. They seem to make sense.\n- Line 155 (left). The matrix W is drawn from the Rademacher distribution. Is there any explanations on this assumption? I suppose in practice the W matrix is learned through optimization instead of drawn from a certain distribution.", "experimental_designs_or_analyses": "Please refer to *Methods And Evaluation Criteria*.", "relation_to_broader_scientific_literature": "NA.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "**Strengths**\n- The paper is overall well written with the claims well supported.\n- This paper established the sufficient condition for avoiding collapse through sound theoretical analysis.\n\n**Weaknesses**\n- The intuitive explanations of the theoretical settings and results could be enhanced, e.g. why is the W matrix assumed to follow the Rademacher distribution?\n- The experimental are relatively small scale, but this should not be over criticized since this paper mainly focus on the theoretical perspectives.", "other_comments_or_suggestions": "1. The study of optimal $\\beta$ parameter is lacking.\n2. Typo in Lemma 1. $q_{ij}$ should be $q_j$?", "questions_for_authors": "1. Why is W assumed to follow the Rademacher distribution?\n2. Why do Table 1 and Table 2 use different compared methods?\n3. How much extra computational burden does the large projector bring? e.g. extra computing time/storage?\n4. How do the dimensional and intracluster collapses of CPLearn compared with related methods? (Figures 12-14)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zsw9ZZsSHC", "reviewer": "Reviewer_MTDo", "summary": "This work studies non-contrastive self-supervised representation learning. The authors identify known collapse modes in non-contrastive SSL and propose a collapse-proof approach (CPLearn). They show that CPLearn jointly decorrelates and clusters embeddings, avoiding common collapse modes without the need for heuristics as in previous literature.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the appreciation of our work and the time dedicated to review our paper. Please find below the answers to your questions:\n\n**Comparison with VicReg** We agree that VicReg is another non-contrastive self-supervised method. Given the limited amount of time available for the rebuttal, we have decided to provide additional results including experiments with a ResNet-18 on CIFAR10 and add additional baselines, such as SwAV and GEDI no gen, on ImageNet-100 to answer the requests from other reviewers (please refer to the answer to Reviewer KMvq). While we didn’t perform an explicit comparison with VicReg on ResNet-8 and ImageNet-100, it is still possible to observe that VicReg achieves similar performance to Barlow Twins when using a Resnet-18 on CIFAR-10 by looking at the results reported in the [solo-learn](https://github.com/vturrisi/solo-learn) repository, which is used in our experiments (e.g. Barlow Twins 92.1% vs. VicReg 92.1% acc@1 on CIFAR-10). This is reasonable as VicReg belongs to the family of feature decorrelation approaches like Barlow Twins.\n\n**Training Full ImageNet** The main bottleneck arises from the fact that we don’t have access to large computational resources to scale up to full ImageNet.\n\n**Training convergence** We provide a comparison of the linear probing performance over training on the experiments for ResNet-18 on CIFAR10. We visualize the top1 accuracy over training epochs both on linear and logarithmic scales. We observe that CPLearn achieves better convergence rate compared to the other approaches, thanks to the quasi-orthogonality condition of the code matrix W. The plots are available at the following anonymous [link](https://aquamarine-lula-78.tiiny.site) and will be added in the Supplementary Material together with the additional experiments done in this rebuttal (that is experiments with a ResNet-18 on CIFAR10 and additional baselines, such as SwAV and GEDI no gen, on ImageNet-100).\n\nPlease let us know if you have any additional questions.", "claims_and_evidence": "To support the claims, the paper provides mathematical proofs to specific design choices such as the loss function and the projector, and provides experiment results on four classic SSL image datasets. The proposed method (CPLearn) achieves competitive performance when the dictionary size gets large.", "methods_and_evaluation_criteria": "The method essentially projects and soft assigns learned representations to a code book and computes the loss w.r.t the codes. The design intuitively makes sense and is supported by the provided theoretical justifications.\n\nThe evaluation criteria adopt standard SSL evaluation. Experiments are conducted on four classic SSL image datasets and compared with representative baselines in the field. The practical choice of a critical hyperparameter (c) is also discussed and experimented. Additionally, the performance range for multiple runs is provided in the tables, making the evaluation solid.", "theoretical_claims": "I did not rigorously check the correctness of the proofs.", "experimental_designs_or_analyses": "Experimental designs and analyses are sound. Besides the experiments, visualization on important aspects of the learning process are also provided in Figure 3 and 4, which are informative with detailed explanations.", "relation_to_broader_scientific_literature": "The paper provides sufficient literature discussion.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n1. Theories rather than heuristics are presented to support the design choices and are backed by experiment results and visualization.\n2. The proposed method seems straightforward to implement. The provided pseudocode is clear.\n\nWeaknesses:\nSee questions.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Vicreg is also a mainstream non-contrastive SSL method, why was it not compared with in the experiments?\n\nResults on ImageNet are not provided. Are there any difficulties in applying the method to a larger scale of SSL training, for example, on ImageNet?\n\nHow fast is the training convergence compared with the baselines?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This submission received ratings of 4,3,3,2. The weak reject reviewer likes the overall paper but complains about larger scale experiments. Since the other reviewers are generally positive about the paper, it will be accepted.
Decision: Accept (poster)