On the duality between contrastive and non-contrastive self-supervised learning
Authors: Quentin Garrido
Yubei Chen
Adrien Bardes
Laurent Najman
Yann LeCun
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Self-supervised learning
contrastive
non-contrastive

Abstract: Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches. While differences between the two families have been thoroughly discussed to motivate new approaches, we focus more on the theoretical similarities between them. By designing contrastive and covariance based non-contrastive criteria that can be related algebraically and shown to be equivalent under limited assumptions, we show how close those families can be. We further study popular methods and introduce variations of them, allowing us to relate this theoretical result to current practices and show the influence (or lack thereof) of design choices on downstream performance. Motivated by our equivalence result, we investigate the low performance of SimCLR and show how it can match VICReg's with careful hyperparameter tuning, improving significantly over known baselines. We also challenge the popular assumption that non-contrastive methods need large output dimensions. Our theoretical and quantitative results suggest that the numerical gaps between contrastive and non-contrastive methods in certain regimes can be closed given better network design choices and hyperparameter tuning. The evidence shows that unifying different SOTA methods is an important direction to build a better understanding of self-supervised learning.

Published as a conference paper at ICLR 2023
ON THE DUALITY BETWEEN CONTRASTIVE AND NON- CONTRASTIVE SELF-SUPERVISED LEARNING
Quentin Garrido 1.2+
Yubei Chen 1,5
Adrien Bardes"
Laurent Najman
Yann LeCun 1.4.5
Meta AI FAIR
Univ Gustave Eiffel, CNRS, LIGM, F-77454 Mamn-la-Vallee France
Inria, École normale supérieure, CNRS, PSL Research University
*Courant Institute, New York University
"Center for Data Science, New York University
ABSTRACT
Recent approaches in self-supervised learning of image representations can be categorized into different families of methods and, in particular, can be divided into contrastive and non-contrastive approaches. While differences between the two families have been thoroughly discussed to motivate new approaches, we fo- cus more the theoretical similarities between them. By designing contrastive and covariance based con-oottrattive criteria that can be related algebraically and shown be equivalent under limited assumptions, we show how close those fam- ilies can be. We further study popular methods and introduce variations of them, allowing us to relate this theoretical result to current practices and show the influ- ence (or lack thereof) of design choices on downstream performance. Motivated by our equivalence result, we investigate the low performance of SimCLR and show how it can match VICReg's 's with careful hyperparameter tuning, improving significantly over known baselines. We also challenge the popular assumption that con-contrasttvv methods need large output dimensions. Our theoretical and quantitative results suggest that the numerical gaps between contrastive and non- contrastive methods in certain regimes can be closed given better network design choices and hyperparameter tuning. The evidence shows that unifying different SOTA methods is important direction build a better understanding of self- supervised learning.
INTRODUCTION
Self-supervised learning (SSL) of image representations has shown significant progress in the last
few years (Chen al. 2020a He et al. 20200
Chen et al.) 2020b
Grill et al. 2020 Lee et al.]
2021b Caron et al.] 2020 Zbontar et al.
2021
Bardes et al. 2021
Tomasev et al. 2022 C Caron
al. 2021 Chen et al. 2021b et al.
2022a
Zhou et al., 2022a b HaoChen et al.) 202 ap-
proaching, and sometime even surpassing, the performance of supervised baselines on many down- stream tasks. Most recent approaches are based on the joint-embedding framework with siamese network architecture (Bromley et al. 1994) which are divided into two main categories, contrastive and non-contrastive methods. Contrastive methods bring together embeddings different views of the same image while pushing away the embeddings from different images. Non-contrastive meth- ods also attract embeddings of views from the same image but remove the need for explicit negative pairs, either by architectural design (Grill al. 2020 Chen & Hel 2020) by regularization of the covariance of the embeddings (Zbontar et al. 2021 Bardes et al.
2021 Li et al. 2022b).
While contrastive and non-contrastive approaches seem very different and have been described as
such Zbontar et al.] 2021 Bardes et al. 2021 Ermolov et al. 2021 Grill et al. 2020). we pro-
"Correspondence ga dog@meta.comm

---

Published as a conference paper at ICLR 2023
pose to take closer look the similarities between the two, both from theoretical and empirical point of view and show that there exists a close relationship between them. We focus on covariance regularization-aased
non-contrastive methods (Zbontar al. 2021; Ermolov et al., 2021 Bardes et al. 2021) and demonstrate that these methods can be seen as contrastive hetween the dimensions of the embeddings instead of contrastive between the samples. We, therefore, introduce the term dimension-cconrasste
methods which we believe is better suited for them, and refer to the original contrastive methods as sample-contrastive methods. To show the similarities between the two, we define contrastive and non-contrastive criteria based on the Frobenius norm of the Gram and co- variance matrices of the embeddings, respectively, and show the equivalence between the two under assumptions on the normalization of the embeddings. We then relate popular methods to these cri- teria, highlighting the links between them and further motivating the use of the sample-contrusivve and diieension-contraviii contrastive method (VICReg), and a sample-contrasie equivalence empirically and improve both VICReg and SimCLR through this lens. Our contributions can be summarized as follows:
nomenclature. Finally, we introduce variations of an existing dimension-
one (SimCLR), This allows us to verify this
We make significant effort to unify several SOTA sample-contrastive and dimension- contrastive methods and show that empirical performance gaps can be closed completely. By pinpointing its source, we consolidate our understanding of SSL methods
We introduce two criteria that serve as representatives for sample- and dimension- contrastive methods. We show that they are equivalent for doubly normalized embeddings, and then relate popular methods them, highlighting their theoretical similarities.
We introduce methods that interpolate between VICReg and SimCLR to study the practical impact of precise components of their loss functions. This allows us to validate empirically our theoretical result by isolating the sample- and wwwpnniiiionoontstiiiii.. nature of methods.
Motivated by the equivalence, we show that advantages attributed to one family can be transferred the other. We improve SimCLR's performance to match VICReg's s, and improve VICReg to make it as robust to embedding dimension as SimCLR.
2 RELATED WORK
Sample-contrastive methods. In self seff-supervised learning of image representations, contrastive methods pull together embeddings of distorted views of a single image while pushing away embed- dings coming from different images. Many works in this direction have recently flourished (Chen
al., 2020a He et al. 2020 Chen et al. 2020b 202Tb Yeh et al. 2021), most of them using the Info CE criterion (Oord et al. 2018), except HaoChen et al. (2021). ). that uses squared similarities
between the samples. Clustering-
methods (Caron et al. 2018 2020 2021 can be seen as
contrastive between prototypes, clusters, instead of samples.
Non-contrastive methods. Recently, methods that deviate from contrastive learning have emerged and eliminate the use of negative samples in different ways. Distillation-basee methods such as BYOL (Grill et al., 2020). SimSiam (Chen & Hc 2020) or DINO (Caron et al., (02TT) use archi- tectural tricks inspired by distillation avoid the collapse problem. Information maximization methods (Bardes et al., 2021 Zbootar et al.) 2021] Ermolov et al.) [021]] [L et al.) (0026) maximize the informational content of the representations and have also had significant success. They rely on regularizing the empirical covariance matrix of the embeddings so that their informational con- tent is maximized. Our study of dimension-cootrrative learning focuses on these covariance-based methods.
Understanding contrastive and non-contrastive learning. Recent works tackle the task of un- derstanding and characterizing methods. The fact that a method like SimSiam does not collapse is studied in Tian et al. (202T). The loss landscape of SimSiam is also compared to SimCLR's 's in PPokle et al.] (2022), which shows that it learns bad minima. In Wang & Isola (2020), the optimal solutions of the InfoNCE criterion are characterized, giving a better understanding of the embedding distri- butions. A spectral graph point of view is taken in HaoChen et al. (2022 2021); [Shen et al. (2022) to analyze self-ssuerviised learning methods. Practical properties of contrastive methods have been studied in Chen et al. (202 In Huang et al. (2021) Barlow twins criterion is shown to be related criterion. We go further and exactly quantify the gap be-
to an upper bound of a sample-contrase tween the criterion, which allows to use the link between methods in practical scenarios. Barlow
2

---

Published as a conference paper at ICLR 2023
Twins' criterion is also linked to HSIC in Tsai et al. (2021) The use of data augmentation in sample- contrastive learning has also been studied from a theoretical standpoint in Huang et al. (2021) I) Wen
& (202I). In [alestriiroo & LeCun (2022), popular self-supervised methods are linked to spectral methods, providing a unifying framework that highlights their differences. The gradient of various methods is also studied in [Tao et al.] (202T), where they show links and differences between them. In Lee et al. (2021a), a link is made between CCA and SCL by showing similar error bounds on linear classifiers.
3 EQUIVALENCE OF THE CONTRASTIVE AND NON-CONTRASTIVE CRITERION
While our results only depend on the embeddings and not the architecture used to obtain them, nor do they depend on the data modality, all the studied methods are placed in joint embedding framework and applied on images. Given dataset D with individual datum d; E Rexkxw, this datum augmented to obtaín two views 2; and x/.- These two views are then each fed through a pair of neural networks Jo and fa. We obtain the representations fo(xs) and fn (1(), which are fed through a pair of projectors Pe and pby such that embeddings are defined as po Po (xx)) and par (x{)). We denote the matrices of embeddings K. and K' such that K.,, ( (z+)), and similarly for K", we have K. E R MMN,, with A the embedding size and N the batch size, and similarly for K". These embedding matrices are the primary object of our study. In practice, we use fo = f-g and pa = pl.. While most self sele-super learning approaches and negative pairs {vj.j + i, (z),
(fa((z)))
positive pairs (z,, æ²) * i, (Iz, x2)3 for a given view I,, we focus on the simpler scenario where negative samples are just [vj.j * i, (Ix, kT)]- There is no fundamental difference when 0 = 8' and when the same distribution of augmentations is used for both branches, and we therefore make these simplifications to make the analysis less convoluted.
We start by defining precisely which contrastive and non-contrastive criteria we will be studying throughout this work. These criteria will be used to classify methods in two classes, sample- contrastive, which corresponds to what is traditionally thought of as contrastive, and dimension- contrastive, which will encompass non-contrastive methods relying on regularizing the covariance matrix of embeddings,
Definition 3.1. Given a matrix E R"x*, We define its extracted diagonal diag (A) E R as:
{0, Ac,ár i
diag (A),,
otherwise.
Definition 3.2. A method is said to be sample-contrustive if it minimizes the contrastive criterion L. ||'T K diaiiaKK." XC))( I.) Similarly, a method said be dimension-coouraativve if minimizes the non-contrastive criterion Lnc = ||CCT diag(KCKC)
The sample-contrastive criterion can be seen as penalizing the similarity between different pairs of images, whereas the dimenssion-contrastivve criterion can be seen as penalizing the off-diagonal terms of the covariance matrix of the embeddings. These criteria respectively try to make pairs of samples or dimensions orthogonal.
Invariance criterion. While Lee and Tine focus on regularizing the embedding space, they are not optimized alone. They are usually combined with an invariance criterion that aims at producing the same embedding for two views of the same image. As such, a complete self-supervised loss would look like LSSL = Line + Lreg with Lreg being either I L,c for our prototypical sample- methods. This invariance criterion is generally a similarity
contrastive and measure, such as the cosine similarity or the mean squared error of the difference between a pos- itive pair of samples. Both are equivalent from an optimization point of view if using normalized embeddings, hence we focus on the regularization part which the souree of differences between these methods.
Proposition 3.1. Considering an infinite amount of available negative samples, SimCLR and DCL's criteria lead to embeddings where for negative pairs (I,x x E R M we have
1
EE
0 and Var x
[ M
SimCLR and DCL cannot be easily linked to L. since they rely on cosine similarities instead of their square absolute value. Indeed, while Tu, aims at making pairs of embeddings dimensions
3

---

Published as a conference paper at ICLR 2023
orthogonal, SimCLR and DCL's criteria go step further and aim at making them opposite. Both cannot be satisfied perfectly in practice, as we would need as many dimensions as samples for Le to have all negative pairs be orthogonal, and more than two vectors cannot be pairwise opposite for SimCLR and DCL's criterion. Nonetheless, as shown by Proposition 3.1 SimCLR and DCL's criteria will lead to dot products of negative pairs with null mean, which is exactly the aim of Te- This shows that while the original formulations of DCL and SimCLR do not fit perfectly into our theoretical framework, they will still lead results similar to other methods that we study. In order to complement this result, we introduce SimCLR-sp sq and SimCLR-abs as variations of SimCLR, which respectively use square or absolute values of cosine similarities. We define DCL-sq and DCL-abs similarly. We provide study of SimCLR-sq and SimCLR-ai in supplementary section G where we compare them to SimCLR. The main conclusion that the distribution of off-diagonal terms of the Gram matrix is similar between all studied methods, with high concentration of values around zero, as predicted by Proposition 3.1 We also see that changing SimCLR into these variations does not impact performance. We even see small increase in top-1 accuracy ImageNet (Deng al. 2009) with linear evaluation when using SimCLR-abs, where we reach 68. 18.71% % top-1 accuracy, compared 68.61% with our improved reproduction of SimCLR. Both of these theoretical and practical arguments reinforce the proximity of SimCLR to our framework.
Proposition 3.2. MunCCR-abbs/se DCL-sq/abs, and Spectral Contrastive Loss (HaoChen et al, 2021 are sample-contrastive methods. Barlow Twins (Zbontar et al. 2021), VICReg (Bardes et al. 2021
and TCR (Li et al. 2022b) are dimension-comtraativee methods.
Even though they do not fit perfectly in our framework, we discuss how methods such as DINO, SimSiam, or MoCo can be linked to Le and Luc in supplementary section B From proposition 3.2 we can see that sample-contrastive and together by Le and Lnc- This alone not enough to show the link between those two families of methods and we will now discuss the link between Le and Lune to show how close those families are.
methods can respectively be linked
Theorem 3.3. The sample-contrastive and dimension-contrastie criteria Le and Line are equivalent up to row and column normalization of the embedding matrix K. Consider batch size of N and an embedding dimension of M. We have:
M Luc II La +
Theorem 3.3 is similar to lemma 3.2 from Le et al. (201)), where we consider matrices that are not doubly stochastic. It is worth noting that our result does not rely on any assumption about the embeddings themselves. A similar result was also used recently in HaoChen et al. (2022), where they relate the spectral contrastive loss to Tine-
The proof of theorem 3.3 hinges on the fact that the squared Frobenius norm of the Gram and Covari- ance matrix of the embeddings are equal, i.c., |||]/ = ||KC 1|2 This means that penalizing all the terms of the Gram matrix (i.e.. pairwise similarities) is the same penalizing all of the terms of the Covariance matrix. While this gives an intuition for the similarity between the contrastive and non-contrastive criteria, it is not representative of the criteria used in practice as L. and Lne are. While theorem [.3 shows that sample-contrastive and dimension-contrasiivv approaches minimize similar criteria, for none of these methods we conclude that both criteria can be used in- terchangeably. However, if both rows and columns of IC were L2 normalized, we would have Line Le N M. In this case, both criteria would be equivalent from an optimization point of view, and we could conclude that sample-contrastive and dimension-contrasee methods are all minimizing the same criterion.
Influence of normalization. The difference between the two criteria then lies in the embedding matrix row and column norms, and most approaches do normalize it in one direction. Since SimCLR relies on the cosine distance as similarity measure between embeddings, we can effectively say that uses normalized embeddings. Similarly, Spectral Contrastive Loss projects the embeddings on a ball of radius VT. with μ tuned parameter, meaning that the embeddings are normalized before the computation the loss function.
Barlow Twins normalizes dimensions such that they have null mean and unit variance, all dimensions will have norm of VN. VICReg takes similar approach where dimensions centered, but their variance is regularized by the variance criterion. This is similar to what is done for Barlow Twins and thus leads to dimensions with constant norm. However, for TCR, the embeddings
4

---

Published as a conference paper at ICLR 2023
normalized and not the dimensions, contrasting with other dimension-coniee contrastive methods. One of the main differences between normalizing embeddings or dimensions is that in the former embeddings are projected on M dimensional hypersphere, and in the latter, they not constrained on a particular manifold; instead, their spread the ambient space is limited.
Nonetheless, constraint the norm of the embeddings also constrains the norm of the dimensions indirectly, and vice versa, as illustrated lemma 3.4
Lemma 3.4. If embeddings normalized such that Vi, || /22 have
N M
|C 112 S 22a
(4)
Conversely, dimensions are normalized such that Vj, || = a we have
M
| S M2a
(5)
Following the proof of lemma 3.4 the lower bounds can be constructed with a constant embedding matrix and the upper bounds with an embedding matrix where either the rows or columns contain only one non-zero element. Both correspond to collapsed representations and will thus not at- tained in practice. While it is impossible to characterize non-collapsed embedding matrices and, as such, derive better practical bounds, these bounds can still be useful to derive the following corol- lary. We study how close methods are to these bounds in practice in section of the supplementary material. The main conclusion is that all practical scenarios, the sum norms will be very close to the lower bounds, deviating by single-digit factor.
Corollary 3.4.1. embeddingg are L2 norrmalizzdd we have
N2
Inn N S
The S Tine N + N
(6)
M
Similarly, if dimensions are L2- normalizee we have
M
La M
< Lnc S L. M + 2
N
Lemma 3.4 applied to Theorem 3.3 directly gives us corollary 3.4.1 which means that in practical scenarios, even when we compare methods where the embeddings are not doubly normalized, the contrastive and non-contrastive criteria can't anntti be arbitrarily far apart. We further show experimentally in section [..] that the normalization strategy does not matter from a performance point of view on SimCLR, reinforcing this argument. Considering the previous discussions, we thus argue that the main differences between sample-contrastive and dimension-contrasti methods come from optimization process as well as the implementation details.
Disguising VICReg as contrastive method. To illustrate theorem we can rewrite VICReg's criterion to make Le appear. We first recall the different components of VICReg's criterion. The variance criterion v hinge loss that aims at making the variance along every dimension greater than 1, and the covariance criterion is exactly defined L,ee applied to centered embeddings. For more details, confer Bardes et al. (202I). To make Le appear, we will still apply the invariance and variance criterion the embeddings, but the covariance criterion will be applied to the transposed emheddings, effectively making it contrastive since we have:
c(K3²) ||TT (KT)² diag (KT (ICT)) l = | dia = L.()).
(8)
We then just need to add regularization term the norms of embeddings and dimensions as follows:
N
M
Lreg(K) (K. Σ
IAC l1..
(9)
and VICReg's loss function can then be written as
N
LVICRey = IK., KK,, (o(K) + +(XC))++ (Lo(K) + LKC + Lc(K') + Lreg (K')).
(10)
5

---

Published as a conference paper at ICLR 2023
This rewriting be seen variation of SCL which is added Lyey and that uses the variance loss for normalization. Being able to make VICReg's criterion sample-contrasie highlights the close relationship between existing sample-contrastive and dimension-contern ther shows that the difference in the behavior of different methods is not mainly due to whether they contrastive not.
methods and fur-
INTERPOLATING BETWEEN METHODS: IMPACT OF THE LOSS FUNCTION.
While we have discussed the link between the contrastive and non-contrastive criteria, der how the design differences in popular criteria manifest themselves practice. To do so we start by introducing variations on VICReg that will allow us interpolate between VICReg and SimCLR while isolating precise components of the loss function. While our focus will be on performance, we provide an analysis of the optimization quality in supplementary section [ The conclusion is that while some design choices negatively impact the optimization process on the embeddings, there are no easily visible differences the representations which are used in practice.
won-
VICReg variations. We introduce two variants of VICReg, one that is non-contrastive but inspired by the InfoNCE criterion and one that contrastive and also inspired by the InfoNCE criterion. The former is motivated by one of the main differences between methods, which is the use of the LogSumExp (LSE) for the repulsive force (e.g., SimCLR) or the use of the sum of squares (e.g., SCL., VICreg, BT). The latter is motivated by the wish to design contrastive methods, where im- plementation details such as the negative pair sampling are as close as possible to another method. This way, comparing VICReg either of those methods will yield comparison that truly isolates specific components of the loss function. These two methods can also be seen as a transformation from VICReg SimCLR, which allows
when the behavior of VICReg becomes akin to SimCLR's s, as illustrated in the following diagram:
Exp
Contrastive,
VICReg-ctr Neg. sampling
VICReg
VICReg-exp
SimCLR
The first variant that we will introduce is VICReg-exp, which uses a repulsive force inspired by the InfoNCE criterion. We first define the exponential covariance regularization as:
1 Σ log d
Σ cCNILTTT
Ceep (1.)
(11)
VICReg-exp is then VICReg where replace the covariance criterion by this exponential covari- ance criterion, giving an overall criterion of
N LVICReg-eap = Σ
IKC, + + =(K')) + v (Cezp(K) + Gee)(X')))
(12)
We then define VICReg-ctr, which VICReg-exp where we transpose the embedding matrix before applying the variance and covariance regularization. This means that the variance regularization will regularize the norm of the embeddings, and the covariance criterion now penalizes the Gram matrix, with the same repulsive force as in DCL. Transposing the embedding matrix for the variance criterion leads to more stable training and enables the use of mixed precision. We thus have the following criterion:
N
LVICReg-ctr = A Σ KC -C 1., (O(K)T) + u(X22) + (Cezp(X(*) + (kz (13)
This way, VICReg-exp will allow us to study the influence of the use of the LogSumExp opera- tor in the repulsive force, and VICReg-ctr to study the difference between sample-contrastive and dimension- contrassinee methods when comparing to VICReg-exp We will now be able to study the optimization of the two criteria and how different design choices affect it.

---

Published as a conference paper at ICLR 2023
VTC Roug
VICRog-ncp
VICRog-etr
SimCLR-D079
SIm R-Tueed
65-
64
00
66
66
94
top
61-
60
64-
10
02
02:
02
02-
10
206 612 20242068
8152
206 552 10242048
3152
256 512 1024
3192
256
512
10242048
8152
255 512 1024
8102
Iimbedding
limbeeddnng
Figure I: VICReg, VICReg-exp: and VICReg-ctr perform similarly in 100 epochs training, vali- dating empirically our theoretical result. While the original implementation of SimCLR performs significantly worse - which is unexpected per our theory - we are able to improve its performance to VICReg's level. This further validates our findings. While different projector architectures impact performance, behaviors are similar across methods. Confer supplementary section K for numerical values and hyperparameters.
PRACTICAL DIFFERENCES BETWEEN SAMPLE-CONTRASTIVE AND DIMENSION-CONTR METHODS
While we have discussed how close sample and dimension contrastive methods are in theory, one of the primary considerations when choosing or designing a method the performance on downstream tasks. Linear classification ImageNet has been the main focus in most SSL methods, focus on this task. We will consider the two following aspects, which are responsible for most of the discrepancies between methods.
will
Loss implementation. Thanks to VICReg-exp, we are able to study the difference between penal- izing the Frobenius norm directly and using LogSumExp to penalize it. Similarly, for VICReg-ctr we are able to study the practical differences between the contrastive and non-contrastive criteria. Finally, with SimCLR we will be able to see how the last details between VICReg-ctt and it can impact performance.
Projector architecture. One of the main differences in methods is how the projector is designed.
To describe projector architectures we use the following notation: X Y - Z means that we use linear layers of dimensions X, then Y and Z. Each layer is followed by a ReLU activation and a batch normalization layer. The last layer has no activation, batch normalization, or bias.
In order to study the impact that this has on performance with respect to embedding size, we study three scenarios. First, d - d d, which is the scenario used for VICReg and BT, then 2048 d which was originally used for SimCLR, and finally 8192 8192 d which was optimal for large embeddings with ViCReg.
Due to the extensive nature of the following experiments, we use proxy of the classical linear evaluation ImageNet, where the classifier is trained alongside the backbone and projector. Rep- resentations are fed to a linear classifier while keeping the gradient this classifier's criterion from flowing back through the backbone. The addition of this linear classifier is extremely cheap and avoids a costly linear evaluation after training. The performance of this online classifier correlates almost perfectly with its offline counterpart, so can rely on it discuss the general behaviors of various methods. This evaluation was briefly mentioned in Chen et al. (2020a) but without experi- mental support. We discuss the correlation between the two further in supplementary section [E]
Empirical validation. The first takeaway from figure is that the transition VICReg + VICReg-exp via the addition of the LogSumExp did not alter overall performance or behav- ior. While small performance differences are visible between the two when using light projec- tors, especially at low embedding dimension, as soon as we use a larger projector these differ- ences disappear with them achieving 18%13% and 68.00% respectively. Focusing on the transition VICReg-exp + VICReg-ctr, we can see that there is no noticeable gap in performance in a setting where we were able to isolate the sample-contrastiv and dimension-contrrativv nature of the meth- ods. This validates empirically our theoretical findings on the equivalence of sample-contrastive and dimensiooncontrastii methods. When comparing VICReg-etr to reproduction of SimCRR

---

Published as a conference paper at ICLR 2023
Table 1: Normalisation strategy used by different methods. Scenarios A and B SimCLR enable fairer comparison to VICReg-ctr and VICReg respectively.
SimCLR
Method
VICReg
VICReg-exp
VICReg-ctt
Classical
B
Dimension centering
X
/
V
Embedding
1
Dimension norm
VN
VN/M
using the original hyperparameters, we can see that VICReg-ctr performs significantly better than SimCLR, achieving 67.92% top-1 accuracy compared to 66.33%. This is surprising since the main difference between the two is that VICReg-ctr uses fewer negative pairs, which should not improve performance. As such we will focus on showing that the previously known performance of SimCLR suboptimal and then fix In supplementary section we further validate our results with k-nn classification accuracy and also show that features correlate extremely well between methods.
Improving SimCLR's performance. To the best of our knowledge, the highest top-1 accuracies reported ImageNet with SimCLR in 100 epochs are around 66.8% (Chen et al. 2021a). While much higher than the 64.7% originally reported, this is still significantly lower than VICReg. Moti- vated by the performance of VICReg-ctr, hyperparameters, allowing us to find that a temperature of 0.15 and base learning rate of 0.5 can lead to a top-1 accuracy of 68.6%, matching VICReg's ' performance in Bardes al. (202I). This reinforces our theoretical insights and highlights the contribution of precise engineering in recent superviue advances. As stands, SimCLR can still serve as a strong baseline.
used the same projector VICReg and heavily tuned
A larger projector increases performance. From figure we can see that for every studied method, going from projector with architecture 2048 to 8192 8192 in performance, especially for VICReg and VICReg-ctr, both gaining 3.5 4 points. The projector d d is in between the two depending on the embedding dimension but also shows similar trend, the performance increases with the number of parameters for every method. While out of the scope of this work, the study of the importance the projector's capacity is an exciting line of work that should help gain deeper understanding of its role in self-supervised learning. We provide preliminary discussion in the supplementary section !|
yielded significant boost
Clearing up misconceptions. While contrastive methods often thought of as sample inefficient, thus requiring large batch sizes, and non-contrastive methods as dimension inefficient, thus projec- tors with large output dimensions, all of these apparent issues can be alleviated with some care, Most notably, the need for large batch sizes of contrastive methods has been studied in Yeh et al. (2021) and Zhang al. (022) where the main conclusions are that with more tuning of the InfoNCE parameters, the robustness of SimCLR and MoCo to small batches can be improved. Regarding the robustness of non-contrastive methods to embedding dimension, our experiments show that with more adequate projector architecture and with careful hyperparameter tuning, the drop performance at low embedding dimension not as present as initially reported (Zbontar et al., [2021] Bardes et al. 2021). With 256-dimensional embeddings, we were able to achieve 61.36% top-1 accuracy by tuning VICReg's hyperparameters, compared to the 55.9% that were initially reported in Bardes al. (022)). This can be further im- proved to 65.0 01 with a bigger projector. While a drop is still present, we are able to reach peak performance at 1024 dimensions, which is lower than the representation's 's dimension of 2048 and shows that large embedding dimension not a deciding factor in downstream performance.
argue that both of these assumptions
misleading and that
5.1 INFLUENCE OF THE NORMALIZATION STRATEGY
While we have shown that the performance gap between sample-contrastive and dimension- contrastive methods can be closed with careful hyperparameter tuning, in the studied settings not
Popular Torch implementations of SimCLR that are compatible with DDP use wrong gather her opera- tor. which when combined with codebase is correct and should be used. This change had significant impact on performance and allowed us reach VICReg's performance.
DDP divides the gradients by the world size. The implementation in VICReg's
8

---

Published as a conference paper at ICLR 2023
SimCLR
SimCLR w/ centering
SimCLR w/ dim. std.
66-
66
00-
top.
d-l-d
84
64
64
2018
2018 2048 d
62
256
512
1024
2048
8192
62
256
512
1024
2048
6192
62
256
512
1624
2048
8192
Imbbeddtig dimonston
Embedding dimondtoon
Emboodiing dimenston
Figure 2: The performance of SimCLR is unchanged when introducing centering dimension standardization, highlighting the lack of importance of normalization on peak performance.
all details are equal. This is especially true regarding the normalization strategies that are used, and we illustrate the different ones in table [ In order to show that these differences do not impact performance, we will introduce two variations of SimCLR. First, we will look SimCLR with the centering of the dimensions, and then at SimCLR with the centering of the dimensions as well as normalization along the dimensions instead of the embeddings. This last strategy is in essence a standardization of the dimensions and the same scheme used by VICReg, More precisely the dimension standardization can be written as
R.
N V
1
N Σ
vi E (I...,,, M) K.
with R.,, = K..
(14)
||C,1122
M
N
These variations will allow us to compare VICReg and SimCLR when both adopt the same normal- ization strategy, resulting in comparison that will more closely fit our theoretical framework.
As we can see in figure 2 the centering and dimension standardization do not impact performance at all and we are able to achieve the same peak performance before. The performance is slightly lower with a shallow projector 2048 d, but in all the other scenarios we retrieve the same perfor- mance as the original SimCLR. This performance on par with VICReg and its variations which reinforces our theoretical result in practice. This was further confirmed in a 1000 epoch run, where SimCLR with dimension standardization was able to reach 72.6% top-1 accuracy, compared to 73.3% for VICReg, While a small difference persists, hyperparameter tuning is very expensive in this setting and is most likely the cause of this gap. From these results, we can conclude that while the normalization strategy can be theoretically mo- tivated or can ease the optimization process, is not a deciding factor the performance of self- supervised methods and that the normalization strategy that should be used is the one that is the easiest to work with for given method.
6 CONCLUSION
Through an analysis of their criteria, we were able to show that sample-contrastive and dimension- contrastive methods have learning objectives that are closely related, as they are effectively minimiz- ing criteria that are equivalent up row and column normalization of the embedding matrix. This suggests a certain duality in the behavior of such methods, which we studied empirically. Through the lens of variations of VICReg, we were able study popular design choices in self-supervised loss functions and show their lack of impact on performance, significantly improving the robust- ness to embedding dimension of VICReg along the way. Motivated by our theoretical findings, we performed ample hyperparameter tuning on SimCLR and were able to close its performance gap with VICReg. We also showed that the normalization strategy does not play an important role in performance. This further reinforces the similarities between methods as predicted by our theoreti- cal results. We expect that our results will help extend theoretical works self-supervised learning to wider family of methods, as well as help analyses by deriving criteria that are easier to work with. We also expect that our findings will help alleviate preconceived ideas on contrastive and non-contrastive learning. If one thing must be remembered from this work, it is that dimension- contrastive and sample-contraste most important message of this work is to show that different SOTA SSL methods can be unified. Pinpointing the source the advancements is an important direction to consolidate our understand- ing.
methods are two sides of the same coin. Finally, perhaps the

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGMENTS
The authors wish to thank Randall Balestrierio, Li Jing, Grégoire Mialon, Nicolas Ballas, Surya Ganguli, and Pascal Vincent, in no particular order, for insightful discussions. We also thank Florian Bordes for the efficient implementations that were used for our experiments.
8 REPRODUCIBILITY STATEMENT
While our pretrainings are very costly, each taking around a day with 8 V100 GPUs, we provide complete hyperparameter values in table S6 They are compatible with official implementations of the losses, and for VICReg-ctr and VICReg-exp we also provide Py PyTorh pseudocode in supple- mentary section L In order reproduce our main figure, we also give the numerical performance in table All of this should make our results reproducible, and, more importantly, should make it so that practitioners can benefit from the improved performance that we introduce.
REFERENCES
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning re- cover global and local spectral embedding methods. arXiv preprint /0xxiv:2005./5088 2022.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance- hwwpauoee-iiarinnnnccovvvaaaaei reg.a//eeoooaaaiananeegaaaaaitoi for solf-super learning. arXiv preprint arXiv:2105.04906, 2021.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Sackinger, and Roopak Shah. Signature verifi- cation using "siamese" time delay neural network. In NeurlPS, 1994.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, and Julien Mairal Piotr Bojanowski Ar- mand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Ting Chen, Simon Korbliihh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Calvin Luo, and Lala Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34:1 1834-11845, 2021a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04220 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- self-supervised vision transformers. In ICCV, 202
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning, 2021.
Pierre Fernandez, Alexandre Sablayrolles, Teddy Furon, Hervé Jégou, and Matthijs Douze. Water- marking images in self-supervised latent spaces. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical de- pendence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
10

---

Published as a conference paper at ICLR 2023
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your latent: A new approach to self-supervised learning. In NewrlPS, 2020.
Matthew Gwilliam and Abhinav Shrivastava. Beyond supervised vs. unsupervised: Representative benchmarking and analysis of image representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9642-9652, 2022.
Manu Srinath Halvagal, Axel Laborieux, and Friedemann Zenke. Predictor networks and stop-grads provide implicit variance regularization in byol/simsiam. arXiv preprint arXiv:2212.0488. 2022.
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. NeurlPS, 34, 2021.
Jeff HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: Analyzing the linear transferability of contrastive representations to related subpopulations. arXiv preprint arXiv:2204.02668, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self- supervised learning. arXiv preprint arXiv:2111.00743, 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-super learning. In International Conference on Learning Representations, 2022.
eww. net /ttp:/orrmmm...eeees0200
Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Quoe Le, Alexandre Karpenko, Jiquan Ngiam, and Andrew Ng. Ica with reconstruction cost for efficient overcomplete feature learning. NeurlPS, 24, 2011.
Jason Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps:
Provable self-supervised learning. Advances in Neural Information Processing Systems, 34:309- 323, 2021a
Kuang-Hue Huei Lee, Anurag Arnab, Sergio Guadarrama, John Canny, and Ian Fischer. Compressive visual representations. In NeurlPS, 2021b.
Chunyuan Li, Jianwei Yang. Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. In ICLR, 2022a.
Shengqiao Li. Concise formulas for the and volume of hyperspherical cap. Asian Journal of Mathematics and Statistics, 4(1 4(1):66-70, 2011.
Zengyi Li, Yubei Chen, Yann LeCun, and Friedrich T Sommer. Neural manifold clustering and embedding. arXiv preprini arXiv:2201.10000 2022b.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. XrXiv preprint arXiv: 2018
Ashwini Pokle, Jinjin Tian, Yuchen Li, and Andrej Risteski. Contrasting the landscape of contrastive and non-contrastive learning. arXiv preprint arXiv:2203. /5702, 2022.
Kendrick Shen, Robbie Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. arXiv preprint arXiv: mrXiv:2204005570 2022.

---

Published as a conference paper at ICLR 2023
Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self-supervised learning via unified gradient framework. arXiv preprint arXhv:2112.051411 2021.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. arXiv preprini vrXiv:2102.068111 2021.
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of sulf--upervised resnets: Can we outperform supervised learning without labels imagenet? arXiv preprint arXiv:2201.051119 2022.
Yao-Hung Hubert Tsai, Shaojie Bai, Louis-Philippe Morency, and Ruslan Salakhutdinov. A note on connecting barlow twins with negative-sammpl-free contrastive learning. arXiv preprint arXiv:2104. 13712, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In ICML, pp. 9929-9939, PMLR, 2020.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning. pp- 11112-11122.
PMLR, 2021.
Chun-Hsiao Yeh, Cheng-
Cae Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. arXiv preprint arXiv:21 10.06848, 2021.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint 1708 100.00888.08888 2017.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, pp. 12310-12320. PMLR, 2021.
Chaoning Zhang, Kang Zhang, Trung X Pham, Axi Niu, Zhinan Qiao, Chang D Yoo, and In So Kweon. Dual temperature helps contrastive learning without many negative samples: Towards understanding and simplifying moco. arXiv preprint 220.:::2003.72455 2022.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. In ICLR, 2022a.
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi-granular supe-vueee learning framework, arXiv preprint arXiv:2203. 240.:2253.44 2022b.
12

---

Published as a conference paper at ICLR 2023
A BACKGROUND
In this section, we will recall the loss functions of all the methods considering throughout our theoretical analysis.
DCL: We first take look DCL's criterion. We consider that i 12 normalized column-wise, i.c. embeddings are normalized. We have
N
K. K.
DCL
log
log
Σ
(15)
Ejjt C.R
jzi
SimCLR:
We now take look at SimCLR's criterion. We consider that IC is normalized column- wise, ie. embeddings are normalized. We have
N
SSICLL
log
KT + +E EE (T.K.slT)
(16)
1=1
KT KT
K..Kialt + Eddiieeutt)
+ log
(17)
Spectral Constrastive Loss: Spectral Contastive Loss is defined as
N LSCL = -2 Σ K. ( =
Σ Ex,K,, )
N
(18)
1=1
The normalization that is employed is to project all embeddings ball of radius μ. This means that if their norm is lower than μ,, nothing will happen to them.
Barlow Twins: We consider that K. is 12 normalized row-wise, i.e. dimensions are normalized. This gives us:
M
M (KCTT))
M
CBT Σ (1 (1- (K-KK))))))) +A E 1=1
- (1 - (ICKT) +AIIICC + -dig diagg(CX
(ACC )|
j=1
(19)
VICReg: VICReg's criterion is defined
LVTEReg > Σ K. + (c(K) c(K')
(20)
With criterion that penalizes the off-diagonal terms of the covariance matrix
<(KC) Σ Cov(K)), = ||C. diag(KCK. )B = Iucs
(21)
itj
and v a criterion that aims at normalizing dimensions, i.c. rows of K.
TCR: TCR's cost function is defined
- log det (T + aCov(KC)) log det (I akk?)= - D log (1 + ao?),
LTCR -
(22)
where is the i-th singular value of K.
13

---

Published as a conference paper at ICLR 2023
B LINKS BETWEEN METHODS AND OUR CRITERIA
While we focus on methods for which the regularization obtained through the criterion, several other methods can be linked informally to our results. The difficulty in linking them to Le or Lice can also come from choices that are motivated by practical limitations, such as the use of a memory bank, and which do not change methods fundamentally.
One of the most surprising lines of works, BYOL Grill et al. 2020 and SimSiam (Chen & He 2020), showed that using stop-gradient on one side the encoder and using predictor network to create asymmetry was enough to avoid collapse and learn good representations. Even though they do not avoid collapse explicitly through their criteria, recent works such as [Halvaaal al. (2022) or Theorem 3 from Tian et al. (202I) have shown links between the training dynamics of SimSiam and variance and covariance regularization, akin to what Lue would lead to. While these analyses require assumptions such as the linearity of the encoder, they still help shine a light on SimSiam and BYOL's behavior and enable us to see how they can be related to our results.
Due the popularity of sample-contrastive methods, several variants have emerged to improve their sample efficiency or their performance in general. One such modification is illustrated in MoCo (He et al., 2020) Chen et al. 2020b) 202Tb) where a memory bank of sample is combined with an exponential moving average (EMA) of the encoder to provide better negative pairs and thus improve training. While this makes it hard to relate MoCo our framework, it still relies on an InfoNCE criterion like SimCLR and thus leads to similar representations. SimCLR and MoCo become especially close near convergence since the online network and the EMA one will be very similar and thus the two methods also become more alike.
Clustering methods such as DeepCluster (Caron et al.] 2018). SWAV (Caron al., 2020) or DINO et al. (2021) can also be related informally to cample-contrativve approaches. Similarly to MoCo, the main difference lies in the construction of the negative pairs, which are constructed us- ing cluster centers here. The embeddings are then contrasted with these clustering prototypes using losses akin to InfoNCE. In DINO, the clustering aspect is more subtle as is done online, thanks to the last linear layer of the projector which can be thought of as the bank of cluster prototypes, and the embeddings are then the outputs of the penultimate layer. Its projector can thus be decomposed into two parts, the first being the classical projector which is followed by L2 normalization, and the last layer which acts as a clustering layer thanks to the softmax that follows it. As such, while clustering methods cannot be clearly linked is still present, even if only informally.
framework, link cample-contrastive methods
Overall, while not all methods can fit clearly into our results, we are still able relate most of them to cample-contrastive or reinforces the similarity between methods.
methods, even it is with less rigor. This further
C PROOFS
Lemma C.1. Let X.Y D-1 two i..d. random variables corresponding to vectors uniformly distributed on SD-1 Their dot product follows the following distribution
X +1
D DD-1 D D-1)
Beta
2
2 2
Proof. A similar result was proved in Fernandez et al. (2022). though we go one step further and derive the distribution of X"y-1, We follow more geometrical argument and invite the reader to confer Fernandez et al. (002 for an alternative approach.
By the symmetry of the hypersphere. the distribution of ' is the same as the one of (1, 0), which corresponds rotating the reference frame. The cumulative distribution function then corresponds to the surface of the hyperspherical cap of angle cos (X1). Using the formulas for the area of a spherical cap on S derived in Li (2011). as well as the fact than sin (cos- (xx) 1 - x² we directly obtain that for XTY > 0 (i.e. cos -l ((x) I }) we
have 1 (XT))2 Beta P 1)
14

---

Published as a conference paper at ICLR 2023
Since the density of the Beta distribution has reflectional symmetry, that (X²
Beta 2: 2
D-1)
By substituting in u= XTy+1 if follows directly that
D D-1
Beta
(23)
2
concluding the proof.
Proposition C.2. Considering infinite amount of available negative samples, SimCLR and DCL's criteria lead to embeddings where for negative pairs (I, x E RM we have
1
and
Var [ x
(24)
M
Proof. The proof hinges Theorem from Wang Isola 2020), which states that the number of negative samples goes to infinity, optimizing the repulsive force of the InfoNCE criterion leads uniformly distributed embeddings on the . -ypepsesspeere.
This uniform distribution allows us to leverage Lemma CT1
saying that as the number of neg-
ative samples goes to infinity, for any pair of random embeddings X, Y,Y, we have xTy-1 ~ Beta ( (M-1, M-1).
We can directly obtain the two following properties
X
+1
M
E
2
Y = 0,
E
(25)
2
2
2
Var
XTY
+1
M-1
2
X
M-1
Var X
1
(26)
2
M
M-
M
M-1
+1)
4M
M
2
2
concluding the proof.
Proposition C.3. SimCLR-abs/su, DCL-ss/abs, as well as Spectral Contrastive Loss are sample- contrastive methods. Barlow Twins, VICReg, and TCR are dimension-conntaativvv methods.
Proof. DCL-sq/abs: We first take a look at DCL-sq/abs's 's criteria. We consider that K. is 12 normal- ized column-wise, i.e. embeddings are normalized. Let f + R+ either defined as f(x) = x² for DCL-sq as fxx |a|| for DCL-abs. We have
N
(KTT,K)))) Ejjre ((TT)
N
f (KT,K,,)
LDCL.
log
+ log
Σ Jipl
1=1
(27)
The first part of this criterion is the invariance criterion and the second part is the LogSumEXp(LLSE) of embeddings' similarity. We know that this smooth approximation the max operator with the following bounds:
Σ Eet(ETk.))11 s max (1V) jti
max (fvj ii, f (KC,K.,)))) < log
(28)
We can thus say that using either
N
N or Σ
Σ log
max f (K. KcT, 371
(29)
15

---

Published as a conference paper at ICLR 2023
repulsive force will lead the same result, diagonal Gram matrix. Since this is the same goal as for our sumple-contrastvv criterion, DCL-sq and DCL-abs are sample-contrastivg methods.
The link Le is more visible with the right term, which corresponds to only penalizing value per row/column of the Gram matrix. While this less effective than penalizing all of them at once, given sufficient training iterations will converge the same solution.
SimCL R-sg/bbs We now take a look at SimCLR-abs/sq" ' criteria. We consider that is 12 normal- ized column-wise, i.e embeddings normalized. Let f IR + R+ be either defined f(x) = x² for SimCLR-sq or as f(xx) = |a|| for SimCLR-abs. We have
N
(KT
LSiMLLR =
Σ log ((T,
(30)
Ef
1=1
N
((C.,. + E (1T,
+ log
(31)
j
Due to the presence of the positive pair in the repulsive force (right term), we cannot use the same reasoning with the max operator as for DCL-sq/abs which gave clear intuition.
Nonetheless one can clearly see that to minimize this criterion, all the similarities between the nega- tive pairs, i.e. Vi, Vj o f (C.,,...))) need be minimized. this will result in diagonal Gram matrix, we can say that minimizing this criterion will also minimize our sample-contrasiiee one. We can thus conclude that SimCLR-ss and SimCI.R-abs are sample-contrastive methods.
Spectral Constrastive Loss: We will now consider Spectral Constrastive Learning's criterion. We have
N Σ [ + - (32)
LSCL -2
K.
Σ [(CK.))'' -2
This means that Spectral Contrastive Loss also falls in the sample-contrastive cateory.
Barlow Twins: Looking at Barlow Twin's criterion have
A
M
M
CBT = (1 - (KK )a)
(KCT)
E (1 (1- (I-KKTT)))))
" +A|IIC -iag (XCC (I..
1=1
(33)
Since distribution of augmentations is the same for both views of the images, and the backbone is shared, taking a negative pair from K. or K' is the same. Barlow Twins' criterion can then be rewritten as
M
LBT (1- ),s) + AIICKTT diag(ICK7)
(34)
f=1
As such the right part of Barlow Twins' criterion is indeed the dimenssion-comtrastvve criterion, making Barlow Twins cuimension-conrastti method.
VICReg: VICReg's criterion is defined as
LVICReg = A > KK., -K, (u(K) o(K')) (c(K) c(K'))
(35)
Recall that e is a criterion that penalizes the off-diagonal terms of the covariance matrix as follows:
c(K) Σ
Cov(K)) = ||CK diag(XCK M) Lnc-
(36)
in
This means that VICReg is a dimension commensioo-cootrastivv method.
16

---

Published as a conference paper at ICLR 2023
TCR: TCR's cost function is defined
Lren - log det (7 aCoo((C)) - log det (7 + akc!
E10g log (1 + ao?)
(37)
where o; is the i-th singular value of K. As discussed in Li al. (022b), this criterion leads to diagonal covariance matrix, similarly to the non-contrastive criterion. We can thus say using either
Σ log (1 + taa? or |KCK" - diag(KCK"))l
(38)
will lead to diagonal covariance matrices, or similarly, null off-diagonal terms the Covariance matrix. This means that TCR also falls in the category cimeension-contrastive methods.
Theorem C.4. The sample-contrastive and dimension-cootrastive criteria L and Line are equivalent up row and column normalization of the embedding matrix K. Consider batch size of N and an embedding dimension of M, We have:
Tine Σ 12, The
Σ
KKC,,11
(39)
Proof. This iroof is heavily inspired by the proof of Lemma 3.2 from a similar result for doubly stochastic matrices.
Leetal. 2011
which provides
We have
Tune IIKK diag diag(KKK))
(40)
tr [[KK" diag(Kk")) - diag(K(X))))
(41)
= Br(KKTKKTT - 2tr(KCK" diag(KCK")) + (r(diag(XCX") diag(KCK)))
(42)
= =tr(KCT - tr (KCK" diag(ICK)))
(43)
tr trCK"" diag(ICK("))
(44)
Similarly for Lee, obtain
L. = =I - diag((X"xC)IIE
(45)
= =Ir(KTTTTT)) Kiagg K.).
(46)
Since (KC 1C=)),,, = 1|1., we deduce that tr(K IC diag((( K.)) = EN' |1,. Similarly, we obtain that tr (ACKT diag(KCK.)) = Σ Plugging this back in, we finally deduce that
j=1 1
Lnc La
Σ
|
12
Σ |C l1.
(47)
concluding the proof.
Lemma C.5. If embeddings are normalized such that Vi, [|C., = a we have
N2
|C 11 < 24
(48)
M
Conversely, if dimensions normalized such that Vj. | we have
M
N
4
<
|C 1 < a
(49)
N
1=1
17

---

Published as a conference paper at ICLR 2023
Proof. We start with the first set of inequalities. Since vi, 122 2 0 have
2 =|C|1 = Nºa
Σ IC
Σ
(50)
Which gives upper bound. For the lower bound, using the convexity of the function f x² we obtain
M
M
N2 M a²
Σ
11,
KC 112
=
(51)
M
M
3=1
Combining those two inequalities gives us the desired bounds.
For the second set of inequalities, we follow the same reasoning and use the fact that in this scenario ||A]|] M giving us the aforementioned bounds and concluding the proof.
D TRAINING PROCEDURE
For training, we follow common procedure and use ResNet-50 backbone (He al. 2016), with the LARS (You et al. 2017) optimizer. We use by default base learning rate of 0.3 and compute the effective learning rate as Ir base Ir batch_size
256 We also use momentum of 0.9 and
weight decay of 10 The learning rate follows cosine annealing schedule after a 10-epoch linear warmup. We train for 100 epochs in all of our experiments.
For data augmentation, we follow the protocol of BYO (Grill et al. 2020) which as follows
Table S1: Image augmentation parameters, taken from Grill ct al. 2020).
Parameter
View 1
View 2
Random crop probability
1.0
1.0
Horizontal flip probability
0.5
0.5
Color jittering probability
0.8
0.8
Brightness adjustment max intensity
0.4
0.4
Contrast adjustment max intensity
0.4
0.4
Saturation adjustment max intensity
0.2
0.2
Hue adjustment max intensity
0.1
0.1
Grayscale probability
0.2
0.2
Gaussian blurring probability
1.0
0.1
Solarization probability.
0.0
0.2
Each experiment was run on Nvidia V100 GPUs, with 32GB of memory each, and took around 24 hours to complete.
While this was our base experimental protocol, was adapted for each method, mostly by changing method- -srehdd-ppecficc hyperparameters well as the learning rate, confer supplementary section K for the exact hyperparameters used for each experiment. The Pytorch pseudocode for VICReg-cx) and VICReg-ctr is also available in supplementary section
ONLINE LINEAR PROBE
As previously discussed, to evaluate our experiments, we relied on the use of linear classifier that trained jointly with our main network. This means that it is trained suboptimal representations and stronger augmentations compared to what is typically done for linear evaluation. Even though these two approaches seem closely related, interested in finding how well they are correlated.
18

---

Published as a conference paper at ICLR 2023
To do so, trained linear evaluation on VICReg and VICreg-exp with a projector architecture of 8192 8192 d. E [256, 512, 1024, 2048, 8192] using the following protocol. We train the linear classifier frozen representations for 100 epochs with batch size of 1024 using the SGD optimizer with base learning rate 0.25 (for VICReg) or 1.4 (for VICReg-exp); momentum 0.9, weight decay 10-6 and using cosine annealing learning rate scheduler. We compute the learning rate as Ir = base_rr x
botch_size For augmentations, we follow standard procedure and use random 236
cropping with a scale between 0.08 and with an image size of 224 224 and horizontal flip with probability 0.5 during training. For evaluation, we do a center crop.
Table S2: Relationship in performance between the online linear probe and the offline linear classi- fier. We used VICReg and an expander with architecture 8192 8192 d.
Embedding dimension
256
512
1024
2048
8192
Online top-1 Offline top-1
65.01
66.72
68.06
68.06
68.13
65.11
66.64
67.96
68.00
68.02
Table S3: Relationship in performance between the online linear probe and the offline linear classi-
fier. We used eip and an expander with architecture 8192
8192
Embedding dimension
256
512
1024
2048
8192
Online top-1 Offline top-1
65.24
66.71
67.86
68.00
67.93
65.30
66.58
67.83
67.89
68.18
As we can see in table S2 and S3 the performance achieved by the offline classifier extremely close the performance of the online classifier. While the online classifier cost in compute is negligible, the linear evaluation is almost as long as the pretraining due to data loading bottlenecks and it requires significant amount of learning rate tuning. This makes this online classifier very appealing alternative since it demonstrates very correlated performances for a fraction of the computing cost.
Training a linear regression on those two sets evaluations gives a model with a slope of 0.97, an intercept of 2.1, and R² of 1.0. It is worth noting that since most values are close 68, the fitting of linear regression on this data is sensitive to noise. Nonetheless, the low intercept, as well as the closeness of the slope to 1. confirm the negligible gap between the two evaluation methods that we previously intuited.
F ADDITIONAL EVIDENCE OF THE SIMILARITY OF LEARNED REPRESENTATIONS
The goal of this section is to provide additional empirical evidence of the similar properties of repre- sentations learned by sumple-contrastive and evaluate representations with k-nn classifier and compare their similarities with CKA (Kornblith ct al. 2019).
methods. To this effect, we will
if our previous results only validated similar performance in linear classification setting, we will look at performance with k-nn classifiers which evaluate how well metric is preserved instead of linear separability. We rely on the protocol of Bardes et al. (202T), and use values of l: in [1, 5, 10, 20, 50, 200), with temperatures in [0.05, 0.07, 0.1, 0.2, 0.5, for the weighting of the classifiers. We then look at the best performance achieved by all methods to give a comparison that is as fair as possible.
k-nn evaluation. In order
As we can see in figure ST we are able to retrieve behaviors similar to figure although results ap- pear less stable for VICReg-exp and VICReg-ctr. Nonetheless, looking at the transition VICReg-exp - VICReg-ctr we can see that the peak performance is still preserved, further validation our results for these methods were the dimensson-conttrasivv and sample-contraste ilarly as for linear evaluation, the original implementation of SimCLR performs significantly worse than other methods, but tuned SimCLP CLR recover the performance of VICReg with 5.5 point
natures isolated. Sim-
19

---

Published as a conference paper at ICLR 2023
VICReg
VICRog-ncp
VICReg-e1r
SimCLR-Dr1g
SimCLR-Tuued
60
101
55-
50
3.
38
54
54
02
12
12
52
60-
4.-
68
68
68
8192
6-5-d
4192
2085-1
206 012
20242048
8150
206 512
10242048
8192
512
1064
256 512 2024
8162
256
512
1024
2018
8182
Imbbedinn dmaneus
Renbedding dim-nnoo
Emmbedding dic--rro
Aimbedding dis-etatoo
2imbesdtng dimsstaim
Figure S1: Reproduction of figure with k-nn classifier. We notice the same pattern previously, where going from dimension-contrastivv to sample-contrastive does not lead to significant drop in performance.
increase in performance. This highlights how the practical implications of our results extend beyond linear classification, while further validating our theory.
CKA. CKA (Centered Kernel Alignment) (Kornblith al. 2019 powerful tool to study the similarities between representations, which relies on HSIC (Hilbert-cchmic Independance Crite- rion) (Gretton et al., 2005) with a given kernel. We will use a linear kernel for simplicity. For each method, we will study three different experiments that reached the same level of performance to measure both intra- and inter-method correlation between representtions. We also consider random network to give lower bound of what we can expect.
1.0
VICReg-1024
0.93
0.91
0.91
0.89
0.09
0.14
VICReg-2018
0.93
0.93
0.9
0.91
0.91
0.9
0.9
0.9
0.89
0.09
0.89
0.15
VIC8eg-8192
0.93
0.93
1
0.9
0.91
0.91
0.9
0.9
0.9
0.89
0.89
0.80
0.15
VIC exp-1024 124
0.9
0.9
0.9
0.92
0.92
0.9
0.9
0.9
0.87
0.87
0.87
0.13
VICReg--ep-2048
0.91
0.91
0.92
0.92
0.9
0.13
0.6
VICRop-exp-8122
0.91
0.89
0.88
0.13
VICReg--tr-110
ViCRep-ctr-00BB
0.89
0.9
0.89
0.89
0.14
-0.
VICIeeg-tr-8192
0.9
0.9
0.9
0.54
0.89
0.09
0.89
0.14
.....a.***..
SimCLR-1024
0.89
0.89
0.89
0.87
0.88
0.88
0.89
0.89
0.89
0.96
0.96
0.14
SmmCLR-2010
0.09
0.09 0.89
0.87 0.80
0.00
0.89
0.89
0.89
0.96
0.96
0.14
0.2
SmCLR-8192
0.89
0.89
0.89
0.87 0.88 0.88
0.80
0.89
0.80
0.96
0.96
0.14
Random 0.14
0.15
0.13
0.13
0.13
0.34 0.14
0.14
VICRop 2048
WiCRap 1152
ViCReg eap- 122
VICleg 2019
VicRey 982
VIRR otz 1024
VIC VZCRop otr 2044
SinCLR 1024
Sim LR 20d
Sinc .190
Random
0.0
ViCRep-
Figure S2: CKA of the learned representations on all considered methods. For a given method, experiments with different output dimensions (1024,2048,8) that achieved equivalent performance.
As we can see in figure all of the learned representations are highly correlated, where intra- and inter-methods CKA are very similar. This both shows that different self-supervised methods, whether dimension-contrastiin or sample-contrastive, provide consistent representations over differ- ent runs and also all learn similar representations. These results contrast with the findings in Figure from Gwilliam & Shrivastava (2022) where they found that different methods lead to representations with low CKA. We believe that their findings can be explained by different training setups between methods since the models used were trained with different projectors and data augmentation.
20

---

Published as a conference paper at ICLR 2023
Both of classification, k-nn classification, and CKA, all studied self-superrvsed methods produce extremely similar representations.
these analyses help cement results, where say that through the lens of linear
G IMPACT OF THE SIMILARITY MEASURE ON SIMCLR
While SimCLR uses cosine similarity to push away negative pairs, we will look at what happens when use the square absolute value of cosine similarities, in SimCLR-sq SimCLR-abe
StmCLRRsg
SimC1R-abs
SimCLR
68
68
68
Orerre top 30/2ro/y
66
66
top.1
top-1
64
0-2
62-
622
d:d
d:d-d
dddid
81522-8 81 92
8192-0182-4
0192-411822 92-4-4
60
2048-4
60
20488d
2048-1
236
Embeddinng
Embeddiing
Figure S3: Influence of using squared or absolute values of the cosine similarities for VICReg, with different projector architectures.
As we can see in figure S3 the use of the squared or absolute values the similarities did not impact the performance on image classification, it even improved slightly with large projector when using absolute values, achieving 68. 7% top-1 accuracy.
1e8
SimCLR-abs
1e6
SimCLR-sq
1e7
SimCLR
1.2
6-
1.50
1.0
1.25
0.8
4
0.6
3
0.75
0.4
2
0.50
0.0
0-
0.00
0.4
0.2
0.0
0.2
0.4
0.4
0.2
0.0
0.2
0.4
0.4
0.2
0.0
0.2
0.4
Figure S4: Histogram of cosine similarities for negative pairs in SimCLR-abs, SimCLR-sq and SimCLR.
As we can see in figure S4 for all three methods we obtain distribution of cosine similarities that centered but they all have very different standard deviations. The main culprit of this differ- ence is dimensional collapse, as studied extensively in Jing et al. (2022). We study this behavior in figure [S5] where we see that the three methods show different levels of collapse. While SimCLR- abs appears to have an almost full rank embedding matrix, we can see some collapse at around 256 dimensions for SimCLR, and 64 for SimCLR R-s.-q. Per proposition 3.1 we know that with a perfect optimization of SimCLR's criterion, we should observe variance of T/ D for the cosine similarities, we have D-dimensional embeddings. However this is not the ambient dimension but the embed- dings' dimension, and when combining this result with the dimensional collapse, we clearly see that SimCLR-ate should have less variance as it has the least amount of collapse, and SimCLR --s the highest variance as it has the most amount of collapse. Since this is what observe in practice, these results are coherent with the three methods producing similar cosine similarities distributions, albeit with different standard deviations depending the amount of dimensional collapse.
21

---

Published as a conference paper at ICLR 2023
Singular value representations
Singular value spedtrum of emboddings
SimCLB-abu
LegiSingtter vater)
-2
LegiSinguhh
-10
SimCL8 abs
StmCLR SinCLR
1000
1500
100
200
Singslar
Singslar
Representations
(b) Embeddings
Figure S5: Singular value distribution of the embeddings and representations computed on the train- ing set of ImageNet for SimCLR, SimCLR-abs and SimCLR-sg. All methods use 512 dimensional embeddings.
H Row AND COLUMN NORMS INTERPLAY
While provided bounds that apply any matrix in lemma 3.4 in practice embedding matrices have a particular structure and one can wonder where the norms are in between the relatively distant bounds.
To study this we took 1024 images from ImageNet, computed the corresponding embedding matri- ces, and then 12-normalized the rows or columns.
Table S4: The empirical interplay between embedding matrix norms under row- column-wise 12-normalization for different methods and projector architectures. We abbreviate thousands with and millions with M. The experiment "Random" indicates randomly initialized network.
Colum normalization
Row normalization
Experiment
Projector
N² 37
|AC,,-13
N
M
Σ |
M
VICReg
8192 8192 8192
128
128.19
1M
65k
83k
67M
VICReg-exp
8192
8192 8192
128
128.26
1M
65k
95k
67M
VICReg-ctr
8192
8192 512
2048
2078
1M
256
287
262k
8192 8192 512
2048
2061
1M
256
433.54
262k
SimCLR
8192 8192 8192
128
129.43
1M
65k
113k
67M
Random
8192 8192 8192
128
361.34
1M
65k
75k
67M
As we can see table S4] for every method, in any expansion or projection scenario, we are always close the lower bound, deviating by factor of 3 at most. This significantly smaller than the factors N or M in lemma 3.4maa..4which which are tight when making no assumptions on the embedding matrix K. As previously discussed these extreme cases consist respectively of constant matrix and one with only one non-zero element per row/column. It is logical that the embedding matrices that we have practice are closer constant matrix, with a uniform spread of information, even though they still present some sparsity.
As such, for all practical concerns, the bounds are much closer in practice than they theoretically are. This means that the sample-contrastive and dimensionn-contrativve criteria will also be closer in practice.
22

---

Published as a conference paper at ICLR 2023
1
IMPACT OF THE PROJECTOR CAPACITY
VICReg-eep
VICRog-etr
SimCIR--Oiig
R.Tuued
65-
60
06-
06
04
80
62-
622
0182-0182-4
2005-d
2048
2068
256 512 20241204
0199
2516 512 1024 Emmbedding dimmno
2061
0132
256 512 1024 2003
0192
255 512 1034 2011
0152
255
512
1034 2011
0192
Eimbedding amanoo
Embeeding dimeeoo
1mmbeddtng aimattioo
Smibediing dimanattm
Figure S6: Online performance on ImageNet for VICReg, VICReg-exp, VICReg-ctr, and SimCLR with respect to embedding dimensions when changing the projector's architecture.
As discussed in section 5 the design of the projector plays a significant role in downstream perfor- mance. In figure S6 we also overlay the results for a projector with architecture 2048 2048 - on top of the previously discussed ones. Such projector offers similar behavior 8192 8192 one, but with a bit lower performance. The drop in performance is especially noticeable in dimension-comtraattie methods.
Online Tep-] accurwcy
66
VICReg
ViCRgg-ogg
80
VICRog-etz
SimCLR
10'
10'
10"
Number of paramelers
Figure S7: Online performance ImageNet for VICReg, VICReg-exp, VICReg-ctr, and SimCLR with respect to the number of parameters in their projector.
As we can see in figure S7 if we take a look at the performance with respect to the number of parameters of the projector when increasing the number of parameters of the projector. This conclusion holds for all methods though there are some scenarios that are clear outliers. For example, for VICReg and VICReg exp we can see that with a 2048 256 projector, the performance is significantly lower than expected.
can see a clear trend that indicates that performance is increased
While would be interesting to see if this increase performance saturates at some point, our largest projectors already have 151 million parameters. Increasing it further quickly starts to become impractical due to memory constraints during training, and as such, we leave this study to future work.
Another aspect worth mentioning is that the increase in performance when increasing the number of parameters is not automatic. For example for VICReg, the scenario 2048 2048 1024 achieves 66.68% top-1 1 for 10 million parameters, but the scenario 8192 8192 - 256 only achieves 65.01% % though it has 86 million parameters. This drastic difference suggests that some must be taken when designing the projector and that even though the number parameters important, the architecture in itself also
23

---

Published as a conference paper at ICLR 2023
J INFLUENCE OF LOSS FUNCTION DESIGN ON OPTIMIZATION QUALITY
As previously discussed, the introduction of VICReg-exp allows us to study the influence of the use of the LogSumExp operator in the repulsive force, and VICReg-ctr to study the difference between sample-contrastive and dimension-coontrastive methods when comparing it to VICReg-exp. This enables us to quantify the impact of these design choices on the quality of the optimization process.
While a perfect optimization of the aforementioned criteria would lead to embeddings with similar properties for the covariance and Gram matrix, one can wonder how well they are optimized in practice and whether design choices have a significant impact. To this effect we will look at the Gram and Covariance matrices after optimization, both on the embeddings of the optimization process and on the representations to study the transferability of this process to the representations since they are used for downstream tasks. For the embeddings, we use the same normalization process as is used during training, and we center the representations alleviate the fact that the last ReLU layer constrains them to the positive orthant. This centering on the representations is only done make the visualization more interpretable.
study the quality
As we can see in figure S8 while VICReg penalizes the off-diagonal terms of the covariance matrix and not the Gram matrix, both matrices have off-diagonal terms that significantly smaller than their diagonal counterparts. Similarly for VICReg-exp. we can see that both the Gram and covari- ance matrices are dominated by their diagonal in the embedding space, though there noise in the off-diagonal terms. This is due to the use of the LogSumExp, which as a smooth approximation of the max operator, will mostly penalize the largest values. On the other hand, using squared values will make them penalized by their absolute and not relative value.
We also observe the same behavior for VICReg-ctr and SimCLR, leading Gram and covariance matrices that are dominated by their diagonal but are overall noisier than for VICReg and VICReg- exp. This suggests that the main culprit of this noise is indeed the LogSumExp but that the sample- contrastive nature of VICReg-ctr and SimCLR also played role in creating it.
Looking at the representations, the differences between the methods start to fade. They all still produce Gram and covariance matrices that are dominated by their diagonal, but with some off- diagonal noise. Even though we could see a clear difference the quality of the optimization in the embedding space, the similarity in the representation space makes it harder to interpret for practical scenarios. Indeed, we saw that all methods can be made to perform the same when evaluating the representations.
24

---

Published as a conference paper at ICLR 2023
Covariance matrix
Gram matrix
Covariance matrix
Gram matrix
On the embeddings
(ii) On the representations
(a) VICReg
Covariance matrix
Gram matrix
Covariance matrix
Gram matrix
(i) On the embeddings
(ii) On the representations
(b) VICReg-exp
Covariance matrix
Gram matrix
Covariance matrix
Gram matrix
embeddings
(ii) the representations
(c) VICReg-ctr
Covariance matrix
Gram matrix
Covariance matrix
Gram matrix
(i) On the embeddings
(ii) On the representations
(d) SimCLR- -uned
Figure S8: Covariance and similarity matrices on random part of the ImageNet training set, using VICReg, VICReg-exp, VICReg-ctr, and SimCLR pretrained on ImageNet for 100 epochs. The covariance matrix is limited to the first 64 dimensions, while the Gram matrix is limited to the first 64 samples. In all cases, we used a projector with an output dimension of 2048, the same as the representation dimension.
25

---

Published as a conference paper at ICLR 2023
K COMPLETE PERFORMANCE AND HYPERPARAMETER TABLES
Table S3: Top-1 accuracy on ImageNet using the online linear classifier, including all performances
for figures
and S6
Experiment
Projector
256
512
1024
2048
8192
d d
61.36
63.50
65.35
66.74
68.13
8192 - 8192 d
65.01
66.72
68.06
68.07
68.13
VICReg
2048 d
59.57
64.51
64.64
64.15
64.06
2048 2048 d
64.16
65.87
66.68
66.74
66.78
d - d d
61.02
63.90
66.33
66.82
67.93
8192 8192 d
65.24
66.71
67.86
68.00
67.93
VICReg-exp
2048 d
61.53
63.59
65.06
64.31
64.34
2048 2048 d
64.66
65.77
66.99
66.82
66.88
d - d
62.87
64.95
66.21
66.93
67.73
8192 8192 d
67.72
67.86
67.84
67.92
67.73
VICReg-ctr
2048 d
64.79
64.83
65.00
64.81
64.44
2048 2048 d
66.95
66.91
66.98
66.93
67.18
- d
63.42
64.35
65.05
65.61
66.08
8192 8192 d
66.11
66.33
66.00
66.02
66.08
SimCLR-Orig
2048 - d
64.78
64.69
64.83
64.66
64.73
2048 2048 d
65.36
65.75
65.61
65.61
65.70
d
63.42
65.35
66.36
67.62
68.68
8192 - 8192 d
68.45
68.61
68.49
68.48
68.68
SimCLR-Tune
2048
66.13
66.04
66.19
66.42
66.27
2048 2048 d
67.48
67.50
67.62
67.62
67.65
Table S6: Hyperparameters used for the results in table S5 Sim., Var. and Cov. indicate the weights of the criteria VICReg and its variations. r indicates the temperature used for LogSumFxxp-aaed methods. The hyperparameters for VICReg and SimCLR usable with the official implementa- tions. For VICReg-exp and VICReg-ctr, the hyperparameters are compatible with the pseudocode in section
VICReg
Experiment
Projector
Batch size
base Ir
Sim.
Var.
Cov.
256
1024
0.3
25
25
512
1024
0.3
25
25
2
VICReg
1024
1024
0.3
25
25
2
2048
1024
0.3
25
25
2
8192
1024
0.3
25
25
0.5
& 8192
1024
0.5
2
0.1
VICReg-exp
8192
1024
0.8
1
2
0.1
VICReg-ctr
All
1024
0.6
0.15
SimCLR-Tune
All
2048
0.5
0.15
26

---

Published as a conference paper at ICLR 2023
L VICREG VARIATIONS PSEUDOCODE
Algorithm 1: VICReg-exp PyTorch pseudocode.
f: encoder network,p, p: projector network, lambda, mu, nu: coefficients of the invariance, variance and covariance losses, N: batch size, D: dimension of the representations, tau: temperature mse_loss: Mean square error loss function, relu: ReLU activation function, cut_out_diags remove the diagonal of matrix,
for x in loader: load batch with N samples
# two randomly augmented versions of x x_a, x_b augment (x)
# compute embeddings a_a = (x
# N x D
k_b (x N
invariance loss
sim_loss = ass__los ak_a, b_b)
# variance loss
std_k_a = torch. sqrt (k_a. aa (dim=0) + 1e-04)
std_k_b = torch. sqrt (k b. var (dim=0) 1e-04)
soss = torch. mean (relu std_k_a)) /2 + Corch.meat (
relu(1 std_k_b))/ /2
# covariance loss
k_a k_a - k_a. mean (dim=0)
k_b k_b
mean (dim=0)
cov _ov_k_a = (k_a.I @ k_ / (N 1)
cov kov_k_b (k_b.T C
kbb) / (N 1)
cov loss = torch. logsumexp (cut_out diag (oo___diagoooo___aaaaaaeo 1)
mean () /2
torch. logsumexp (cut_out_ (cov (cov_k_bbaaa) 1)
mean () /2
# loss loss lambda sim loss + mu std loss + nu cov_loss
# optimization step loss baskward ()
optimizer. step() ()
27

---

Published as a conference paper at ICLR 2023
Algorithm 2: VICReg-ctt ctr PyTorch pseudocode.
f: encoder network, p: projector network, lambda, mu, nu: coefficients of the invariance, variance and covariance losses, N: batch size, D: dimension of the representations, tau: temperature mse_loss: Mean square error loss function, relu: ReLU activation function, aut_out_diag: remove the diagonal of a matrix,
for x in loader: # load batch with N samples
# two randomly augmented versions of x_a, x_b = augment (x)
# compute embeddings
k_ (x # N x
k (x # x D
# invariance loss sim_loss = mse_loss (k_a, k_b)
#Maee the method contrastive
k_ a_a.T kb k_b.T
# variance loss std___ = torch. (k_a. var var (dim=0) 1e-04)
stb = torch. sqrt (k_b. var (dim=0) + 1e-04)
std_loss torch. mean (relu std_k_a) + Norch.mean (
relu(1 std_k_b))/2 /2
# covariance loss
k_a k_a - a_a.meat (dim=0)
b k k_bbmeen (din=0)
cov cov_k_a = (k_a.I @ k_a) / (N 1)
cov_k_b (k_b.I @ k_b)
cov_lose = torch. logsumexp mean ()/2
11)
torch. logsumexp
(cov_ 1)
mean () /2
# loss
loss lambda sim_loss + mu std_loss + cov_loss
# optimization step loss. basswbackward optimizer. step
28


Reviews:
[{"review_id": "tgN_xHyzpG", "reviewer": "Reviewer_87vt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper links two different views of self-supervised learning, namely contrastive and non-contrastive self-supervised learning. This is done by interpreting the non-contrastive methods as dimension contrastive ones, and regular contrastive methods as sample contrastive. The authors use this insight to demonstrate relationships between the two types of self supervised loss, and demonstrate how one can derive one type of loss from the other.", "summary_of_the_review": "I believe that this is a solid work, and barring some minor issues I have with the presentation I believe that it should be accepted in the conference. The framework proposed by the authors can be used to unify several different methods used in self-supervision, and this insight can be potentially used to further improve upon existing methods, by easily considering both contrastive and non-contrastive variations.", "strength_and_weaknesses": "Strengths:\n\n- This paper promotes an interesting link between two variants of self-supervised loss which are often used in practice. Linking these two variants is a very original and interesting line of research, and the fact that the authors demonstrate that the two are very closely related is very useful to provide a better understanding of these techniques. \n\n- The theoretical part of the paper is useful, in that it shows that each of the two variants of self-supervised loss can be used to derive bounds for the other one. This is done in Equations 6 and 7 in the main paper. While the upper bounds in these equations are quite loose (for example, in Equation 6 the upper bound has a difference of the order $N^2 - N$), I believe that it should be argued that the lower bounds are more important. In Equation 6, normalizing the embeddings means we are optimizing $L_c$, and the lower bound tells us that $L_{nc}$ cannot be arbitrarily large either (similarly in Equation 7). I think this is an important connection between the two styles of self-supervised learning.\n\n- The experimental part of the paper is also insightful, in that it supports the proposition of the authors that the two styles of self-supervised learning are closely related, so with sufficient tuning both should achieve similar performance. This is in line with the author's theoretical results. Moreover, in the Appendix the authors also compare the relationship between the losses in practice. These experiments serve to fully support the close connection between the two variants of self-supervised learning.\n\nWeaknesses:\n- The authors demonstrate how several well-known self-supervised learning methods fall in their framework of sample contrastive and dimension contrastive methods. To fully complete this comparison, I believe that methods such as MoCo, BYOL and DINO should also be included in this discussion (in Proposition 3.2), since they are also very well-known. Knowing how these methods align in this framework would be useful (mainly because BYOL and DINO also make use of a second teacher modle during training).\n\n- As a minor issue, for the sake of clarity I believe that in their definitions of $L_c$ and $L_{nc}$, the authors should include the alignment term between the representations, even if it is the same across the two. This would make it easier for the reader to recognize that the main difference across methods is the regularization term used in each.\n\nMinor issues:\nThere is a typo in Section 5.1, third line: “strateoes” -> “strategies”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ouWGcfSNZkY", "reviewer": "Reviewer_mAZP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1. This paper interprets the covariance regularization-based non-contrastive methods as dimension-contrastive methods and interprets the traditional contrastive methods as sample-contrastive methods. It then proves that the dimension-contrastive criterion is equivalent to the sample-contrastive criterion when both the rows and columns of the embedding matrix are normalized. It also bounds the difference between these two criteria when the embedding matrix is not doubly normalized.  \n2. In the experiments, this paper shows that the performance of several SOTA sample-contrastive and dimension-contrastive methods can be closed. This paper showed several design choices have a negligible impact on performance by studying the interpolation between VICReg and SimCLR. Along the way, it also improved the robustness to embedding dimensions of VICReg and improved the performance of SimCLR to match VICReg. It also showed that the normalization strategy does not have a significant influence on performance. ", "summary_of_the_review": "This is a very insightful paper that provides a unifying view of contrastive and non-contrastive methods. It interprets them as sample-contrastive methods and dimension-contrastive methods and proved they are equivalent when the embedding matrix is doubly normalized. In the experiments, they showed that the performance of several SOTA sample-contrastive and dimension-contrastive methods can be closed and many techniques originally developed for one family can be transferred to the other. I believe this unifying view can help us have a deeper understanding of self-supervised learning and guide the community to develop better algorithms. ", "strength_and_weaknesses": "Strengths:\nThis is a very insightful paper that provides a unifying view of the contrastive and non-contrastive self-supervised learning methods. From a theoretical perspective, they proved that the sample-contrastive criterion is equivalent to the dimension-contrastive criterion up to the normalization of the embedding matrices. In the experiments, they further empirically verified that the performance of several SOTA sample-contrastive and dimension-contrastive methods can be closed. I believe this unifying view of contrastive and non-contrastive self-supervised learning can help us gain a deeper understanding of these methods. And many techniques and advantages of one family can be transferred to the other, as demonstrated in this paper. It might also be possible to improve the performance by designing a self-supervised learning algorithm that exploits both sample-contrastive and dimension-contrastive criteria. \n\nWeaknesses:\nFor the theoretical part, it's proved that the sample-contrastive criterion is equivalent to the dimension-contrastive criterion when the embedding matrix is doubly normalized. However, when the embedding matrix is not doubly normalized, the bounds in Lemma 3.4 and Corollary 3.4.1 is pretty loose. So I wonder if it's possible to prove tighter bounds in some more restricted theoretical settings. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZJ_qLE0CZwX", "reviewer": "Reviewer_Yvpx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper makes an attempt at bridging the two broadly classified approaches towards self supervised learning, namely contrastive and non-contrastive. Recognizing that the methods can be viewed as sample contrastive and dimension contrastive respectively, the work categorizes popular contributions in the field under these regimes and attempts to consolidate the approaches by drawing\nequivalence. Towards this aim, the paper (i) introduces two criteria characteristic of the regimes and demonstrates the\nequivalence between them, (ii) attempts to empirically close the performance gaps between popular approaches under the regimes, given the equivalence, (iii) introduces methods that interpolate between the two regimes to isolate components of the loss functions for targeted study, (iv) attempts to validate the theory by improving the performance of SimCLR to match VICReg's and the robustness of VICReg to embedding dimensions like SimCLR.", "summary_of_the_review": "The paper attempts to answer the fundamental question of the parallels between sample and dimension contrastive approaches towards self-supervised learning. In that regard, the authors conduct many experiments supporting their theoretical motivations and provide a reproducibility statement for verification. However, the presentation of this paper is rather unclear, and the paper lacks some experiments that may serve to strengthen the claims, and may contain some erroneous in statements regarding the sensitivity of these approaches to batch size or feature dimensions. ", "strength_and_weaknesses": "Strength:\nThe paper conducts many experiments towards substantiating the claims. The contribution is novel, to the extent of my knowledge, making a better understanding of the self-supervised learning methods. The work has potential for further theoretical research. \n\nWeaknesses:\n1. The presentation is not clear, which needs significant revision. More details are given in the Clarity section.\n\n2. It seems this paper only considers SimCLR and VICReg, while the title tries to cover the whole contrastive and non-contrastive learning methods.\n\n3. Lack of focus on invariance criteria.\nAlthough the claim is made that different similarity criteria are equivalent from an optimization point of view, experimental evidence in support of the claim is missing. I am not convinced that the differences between contrastive and non-contrastive learning methods only rely on the objective in Definition 3.2. I don't think different contrastive learning methods can be simply represented using definition 3.2, e.g., how can you show SimCLR can be written with Definition 3.2? Thus, I think there are many other factors in different contrastive learning methods, an using the designs in this paper cannot cover these.\n\n4. Lack of KNN evaluation.\nThe paper speaks of the equivalence of the two contrastive regimes and illustrates the equivalence through the gram and covariance matrices, besides empirical evidence with downstream performance. We see that the off diagonal elements in specifically the gram matrix are significantly lower than their diagonal counterparts. But this begs the question of whether the performance of dimension-contrastive methods are at par with sample-contrastive methods for KNN based evaluation of features. KNN performance based experiments will be helpful towards reinforcing the claim of the equivalence of the two regimes.\n\n5. Claim that the small batch size issues with contrastive methods are misleading (Section 5, Clearing up misconceptions).\nThe paper adequately demonstrates the merits of hyperparameter tuning and use of a better design choice for the projectors may alleviate the problems with small batch sizes, but it is in contrast with the work of [1] where they identify a systematic bias in the updates in SimCLR. This leads us to believe that a better landscape of hyperparameters or projector architectures may be available for large batches as well, reinstantiating the problem with smaller batch sizes in sample contrastive methods, which may require more experiments in those settings. Furthermore, keeping the architecture consistent for fair comparison, we still witness a drop in VICReg performance as the embedding\ndimensions decrease (Figures 1 and S4, Table S5). \n\n[1] Yuan, Zhuoning, et al. \"Provable stochastic optimization for global contrastive learning: Small batch does not harm performance.\" International Conference on Machine Learning. PMLR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VD7IUMxy56", "reviewer": "Reviewer_6jNg", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"On the duality between contrastive and non-contrastive self-supervised learning\" suggests a unifying perspective on two families of self-supervised learning. One family is the contrastive methods (such as SimCLR) that repel samples from each other through negative pairs. Another family is the covariance-regularizing methods (such as VICReg) that repel samples from each other by enforcing a unit spherical covariance matrix of the embedding. The two competing families have so far been seen as conceptually distinct. This paper argues mathematically and shows empirically that they are approximately equivalent.", "summary_of_the_review": "Very strong paper suggesting a unifying perspective on various self-supervised learning methods.", "strength_and_weaknesses": "I am a big fan of unification papers. There are several new self-supervised methods appearing every year each claiming SOTA performance, but every method makes its own design choices and so conceptual comparisons can be difficult. This paper shows that many design choices are not essential, and is able to transform SimCLR into VICReg and vice versa while using the same network architecture. The authors also show that simply tuning some of the SimCLR hyper-parameters increases its performance on ImageNet to the SOTA level. I think this is a very welcome contribution and a very strong accept.\n\nAll the experiments were run on ImageNet for 100 epochs (only two final runs were made for 1000 epochs). Given that ImageNet optimization is costly, I was wondering if the authors could also demonstrate some of the results e.g. on CIFAR-10 where running 1000 epochs is not a problem. Or is CIFAR-10 \"too simple\" for the purpose of this study?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%