VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning
Authors: Adrien Bardes
Jean Ponce
Yann LeCun
Venue: NeurIPS 2021 Submitted (2021)
Keywords: self-supervised learning

Abstract: Recent self-supervised methods for image representation learning maximize the agreement between embedding vectors produced by encoders fed with different views of the same image. The main challenge is to prevent a collapse in which the encoders produce constant or non-informative vectors. We introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with two regularizations terms applied to both embeddings separately: (1) a term that maintains the variance of each embedding dimension above a threshold, (2) a term that decorrelates each pair of variables. Unlike most other approaches to the same problem, VICReg does not require techniques such as: weight sharing between the branches, batch normalization, feature-wise normalization, output quantization, stop gradient, memory banks, etc., and achieves results on par with the state of the art on several downstream tasks. In addition, we show that our variance regularization term stabilizes the training of other methods and leads to performance improvements.

VICReg: Variane-livarrane--Cvaainee Regularization for Self-Supervised Learning
Anonymous Author(s)
Affiliation
Address
email
Abstract
Recent sulf-super methods for image representation learning maximize the agreement between embedding vectors produced by encoders fed with different views of the same image. The main challenge is prevent collapse which the encoders produce constant or innirnformetive vectors. We introduce VICReg Regularization), method that explicitly avoids the collapse problem with two regularizations terms applied both embeddings separately: (1) a term that maintains the variance of each embedding dimension above threshold, (2) term that decorrelates each pair of variables. Unlike most other approaches to the same problem, VICReg does not require techniques such as: weight sharing between the branches, batch normalization, feature-wis wise normalization, output quantization, stop gradient, memory banks, etc., and achieves results on par with the state of the on several downstream tasks. In addition, we show that our variance regularization term stabilizes the training of other methods and leads to performance improvements.
4
6
10
12
13
Introduction
15
16
Self-supervised representation learning has made significant progress over the last years, almost
reaching the performance of supervised baselines on many downstream tasks [1, 2, 5, 6, 7, 9]. Several recent approaches rely on a joins embedding architecture in which two networks are trained to produce similar embeddings for different views of the same image. A popular instance is the Siamese network architecture [10], where the two networks share the same weights. The main challenge with joint embedding architectures is to prevent a collapse which the two branches ignore the inputs and produce identical output vectors. There are two main approaches to preventing collapse: contrastive methods and information maximization methods. Contrastive methods [3, 11, 12] use loss that explicitly pushes the embeddings of dissimilar images away from each other. They often require mining procedure search for offending dissimilar samples from memory bank [3] or from the current batch [12]. Contrastive methods tend be costly, require large batch sizes or memory banks, and do not seem to scale well with the dimension of the embedding. Quantization- based approaches [5. 13] force the embeddings of different samples to belong to different clusters the unit sphere. Collapse is prevented by ensuring that the assignment samples to clusters is as uniform as possible. 30 A similarity term encourages the cluster assignment score vectors from the two branches to be similar. More recently, few methods have appeared that do not rely contrastive samples vector quantization, yet produce high-quality representations, for example BYOL [6] and SimSiam [7]. 33 They exploit several tricks: batch-wer wise or feature-wisg normalization, "momentum encoder" in version of the parameter vector the
17
18
20
21
23
24
26
27
29
32
34 which the parameter vector of one branch is low-pass-filtens other branch [6, 14], or a stop-gradde gradient operation in one of the branches [7]. The dynamics of learning
35
36 in these methods, and how they avoid collapse, is not fully understood, although theoretical and feature-wise normalization [14, 15). 38 Finally, an alternative class collapse prevention methods relies on maximizing the information content of the embedding [9, 16]. These methods prevent informational collapse by decorrelating
37 empirical studies point the crucial importance batch-wise
Submitted to 35th Conference Neural Information Processing Systems (NeurIPS 2021). Do distribute.

---

40 every pair of variables of the embedding vectors. This indirectly maximizes the information content of the embedding vectors. The Barlow Twins method drives the normalized cross-correlation matrix unintain varimce
biing
fo
hΦ
ot2)
t~ T
X
ditrbbtion transormatioss
random translecmazioss
fe enoeders
W ongendors
hasdh images
f'e:
Y'
h' Φ0
X.X'
lundics of views
Y.Y"
hanches of iepresentviions
lunchee of ofcnholliigs
joint embedding architecture with variance, invariance and covariance regularization. Given a batch of images I, two batches of different views X and X' are produced and are then encoded into representations Y and Y'. The representations are fed to an expander producing the embeddings Z and Z'. The distance between two embeddings from the same image is minimized, the variance of each embedding variable over batch is maintained above a threshold, and the covariance between pairs of embedding variables over batch are attracted zero, decorrelating the variables from each other. Although the two branches do not require identical architectures nor share weights, in most of our experiments, they are Siamese with shared weights: the encoders are ResNet-50 backbones with output dimension 2048. The expanders have fully-connected layers of size 8192.
Figure VICReg:
42 of the two embeddings towards the identity [9], while the Whitening-MSS method whitens and
43 spreads out the embedding vectors on the unit sphere [16].
2 VICReg: intuition
44
45 We introduce VICReg training joint embedding architectures based on the principle of preserving the information content of the embeddings. The basic idea is to use a loss function with three terms:
Covvariine Regularization), a self-supervised method for
47
48
Invariance: mean square distance between the embedding vectors.
49
Variance: a hinge loss to maintain the standard deviation (over batch) of each variable of the embedding above given threshold. This term forces the embedding vectors of samples within a batch to be different.
50
51
Covariance: term that attracts the covariances (over batch) between every pair of (centered) embedding variables towards zero. This term decorrelates the variables of each embedding and prevents an informational collapse in which the variables would vary together or be highly correlated.
53
54
Variance and Covariance terms are applied both branches of the architecture separately, thereby preserving the information content each embedding at a certain level and preventing informational collapse independently for the two branches. The main contribution of this paper is the Variance preservation term, which explicitly prevents collapse due a shrinkage of the embedding vectors towards zero. The Covariance criterion is borrowed from the Barlow Twins method and prevents informational collapse due to redundancy between the embedding variables [9]. VICReg is more generally applicable than most of the aforementioned methods because of fewer constraints on the
57
59
60
61
62
63
architecture. In particular, VICReg:
64
does not require that the weights of the two branches be shared, not that the architectures be identical, nor that the inputs be of the same nature;
86
does not require memory bank, nor contrastive samples, nor a large batch size;
feature-wiss normalization; and does not require vector quantization nor a predictor module.
does not require batch-wist
68
69 Other methods require asymmetric stop gradient operations, as in SimSiam [7], weight sharing between the two branches as in classical Siamese nets, weight sharing through exponential moving

---

71 average dampening with stop gradient in one branch, as in BYOL and MoCo [3, 6, 17]. large batches
72 of contrastive samples, as in SimCLR [12], or batch-wise and/or feature-wise normalization [5, 6,
73 9, 16]. One of the most interesting feature of VICReg is the fact that the two branches are not required
74
to share the same parameters, architecture, or input modality. This opens the door to the use of
75 non-contrastive self-supervised joint-embedding for multi-modal signals, such video and audio. We
76 demonstrate the effectiveness of the proposed approach by evaluating the representations learned with
77 VICReg several downstream image recognition tasks including linear head and semi-supervised evaluation protocols for image classification on ImageNet [18]. and other classification, detection and instance segmentation tasks. Furthermore, we show that incorporating variance preservation into other self-supervised joint-embedding methods yields better training stability and performance improvement on downstream tasks. More generally, we show that VICReg is an explicit and effective. 82 yet simple method for preventing collapse in self-supervised joint-embedding learning.
78
79
81
3 Related work
83
84
Contrastive learning. In contrastive SSL methods applied to joint embedding architectures, the output embeddings for sample and its distorted version are brought close to each other, while other samples and their distortions are pushed away. The method is most often applied to Siamese architectures in which the two branches have identical architectures and share weights [2. 3. 10, 11,
85
86
88
12, 17, 19, 20, 21, 22, 23]. Many authors use the InfoNCE loss [22] in which the repulsive force is larger for contrastive samples that are closer to the reference. While these methods yield good performance, they require large amounts of contrastive pairs in order to work well. These contrastive pairs can be sampled from a memory bank as in MoCo [3], or given by the current batch of data as in SimCLR [12], with a significant memory footprint. This downside of contrastive methods motivates search for alternatives.
89
91
92
94
Clustering methods. Instead of viewing each sample as its own class, clustering- -based methods
95
group them into clusters based on some similarity measure [5. 13, 24, 25, 26, 27, 28, 29, 30, 31]. DeepCluster [13] uses k-means assignments of representations from previous iterations as pseudo- labels for the new representations, which requires an expensive clustering phase done asynchronously. and makes the method hard to scale up. SWAV [5] mitigates this issue by learning the clusters online while maintaining balanced partition of the assignments through the Sinkhorn-K Knopp transform [32]. These clustering approaches can be viewed as contrastive learning at the level of clusters which still requires a lot negative comparisons to work well.
96
98
89
100
101
102
Distillation methods. Recent proposals such as BYOL, SimSiam, OBoW and variants [6, 8, 14, 33J have shown that collapse can be avoided by using architectural tricks inspired by knowledge distillation [34]. These methods train a student network to predict the representations of teacher network, for which the weights are a running average of the student network's weights [6], or are shared with the student network. but no gradient is nack-propagated through the teacher [7]. These methods are effective, but there is no clear understanding of how but suffer from lack of explainability regarding the way collapsing solutions are avoided. A Alternatively, the images can be represented as bags of word over a dictionary of visual features, which effectively prevents collapse. In [33] and [8] the dictionary is obtained by off-line or on-line clustering. By contrast, our method explicitly prevents collapse in the two branches independently, which removes the requirement for
103
105
108
108
109
110
111
112
shared weights and identical architecture, opening the door to the application of maint-embedding SSL to multi-modal signals.
113
Information maximization methods. A principle to prevent collapse is to maximize the information content of the embeddings. Two such methods were recently proposed: W-MSE [16] and Barlow Twins [9]. In W-MSE, an extra module transforms the embeddings into the eigenspace of their covariance matrix (whitening
115
116
117
Sarhunen-Loève transform). and forces the vectors thereby obtained to be uniformly distributed on the unit sphere. In Barlow Twins, a loss term attempts to make the normalized cross-correlation matrix of the embedding vectors from the two branches to be close to the identity. Both methods attempt to produce embedding variables that are decorrelated from each other, thus preventing an informational collapse in which the variables carry redundant information.
118
119
120
121
122
Because all variables are normalized over batch, there is no incentive for them to shrink nor expand.
123
This seems to sufficient to prevent collapse. Our method borrows the decorrelation mechanism of Barlow Twins. But it includes an explicit variance-prest preservation term for each variable of the two embeddings and thus does not require any normalization.
124
3

---

128
We describe here the variance, invariance and covariance terms that compose our loss function. The images are processed in batches, and we denote Z = [a]....: žn and Z' = - 2/x1] the two batches composed of n vectors of dimension d, of embeddings coming out of the two branches of VICReg: detailed description
4
127
VICReg follows recent trends in self-supervised learning [5, 6, 7, 9. 12] and is based on joins embedding architecture. Contrary to many previous approaches, our architecture may be completely symmetric completely asymmetric with shared structure parameters between the two branches. In most of our experiments, we use a Siamese net architecture in which the two branches are identical and share weights. Each branch consists of an encoder fo that outputs the representations (used for downstream tasks), followed by an expander h& that maps the representations into embedding space where the loss function will be computed. The role of the expander is twofold: (1) eliminate the information by which the two representations differ, (2) expand the dimension in non-linead fashion so that decorrelating the embedding variables will reduce the dependencies (not just the correlations) between the variables of the representation vector. The loss function uses a term s that learns invariance to data transformations and is regularized with a variance term v that prevents norm collapse and a covariance term e that prevents informational collapse by decorrelating the different dimensions of the vectors. After pretraining, the expander is discarded and the representations of the encoder are used for downstream tasks.
128
129
130
132
135
134
135
136
137
138
139
4.1
Method
142
Given an image i sampled from a dataset DD two transformations t. and t' are sampled from a distribution T to produce two different views x t(i) and x' = +i) of i. These transformations are random crops of the image, followed by color distortions. The distribution TT is described in Appendix C. The views x and x' are first encoded by fe into their representations y = (2) and
144
145
fe(x'), which are then mapped by the expander h& onto the embeddings = hs(y) and has (y'). The loss computed at the embedding level on 2 and z'.
147
148
149
150
151
the siamese architecture. We denote by 2 the vector composed of each value at dimension j in
152
all vectors in Z. We define the variance regularization term hinge function on the standard deviation of the embeddings along the batch dimension:
153
(Z))
m
j=1
154
where S is the regularized standard deviation defined by:
S(x, - 100=-Var(p Var(()++ +
(2)
155 is a constant target value for the standard deviation, fixed to 1 in our experiments, e is small scalar preventing numerical instabilities. This criterion encourages the variance inside the current batch to be equal to ry along each dimension, preventing collapse with all the inputs mapped on the same vector. Using the standard deviation and not directly the variance is crucial. Indeed, if we take
156
157
S(z) Var(x) in the hinge function, the gradient of S with respect to x becomes close 0 when 159
160
is close to æ. In this case, the gradient of u also becomes close to 0 and the embeddings collapse. We define the covariance matrix of Z as:
1
n
1
a
C(Z)
22 2)(2
=" where
Z
(3)
n
n
162 Inspired by Barlow Twins [9], we can then define the covariance regularization term c as the sum of the squared off-diagonal coefficients of C(Z), with a factor 1/d that scales the criterion function
164 of the dimension:
LIC(Z) c(Z) = d 1
(4)
165 This term encourages the off-diagonal coefficients of C(Z) to be close to 0, decorrelating the different
186 dimensions of the embeddings and preventing them from encoding similar information. Decorrelation
167 at the embedding level ultimately has a decorrelation effect at the representation level, which is a
non trivial phenomenon that study in Appendix D. We finally define the invariance criterion

---

189 between Z and Z' as the mean -qeared euclidean distance between each pair of vectors, without any normalization:
170
s (Z, - Σ | - 2/11
(5)
The overall loss function is a weighted average of the invariance, variance and covariance terms: 2Z = >s + plu( + =(Z') +
(6)
172
where A, lr and V are experiments, we set V = 1 and perform grid search on the values of À and with the base condition
controlling the importance of each term the loss. In our
173
A 1 1. The overall objective function taken on all images over an unlabelled dataset D is given
175 by:
f= e(z'.2"). TEDLP'TT
where T and Z" are the batches of embeddings corresponding to the batch of images T transformed by and t'. The objective is minimized for several epochs, over the encoder parameters 0 and expander parameters c. We illustrate the architecture and loss function of VICReg in Figure 1.
176
177
178
180
4.2
Implementation details
Implementation details for pretraining with VI-
CReg on the 1000-classes ImagetNet' dataset without labels are as follows. Coefficients À and
Algorithm 1: VICReg pseudocode.
182
183
tratatiy my cieees four and cesariancs seaens in Feoeh alse, esaenssze reperercacion 'concciio
184
I are 25 and v is 1 in Eq. (6), and e 0.0001 in Eq. (1). The encoder network fe is stan-
185
the x to Scecees a load beces with samples
186
dard ResNet-50 backbone [35] with 2048 output
187
units. The expander hg is composed of two fully-connected layers with batch normalization (BN) [36] and ReI.U, and a third linear layer.
compete represstationn
AtaLSc a00 1.. 5,21
189
190
The sizes of all layers were set to 8192. As with Barlow Twins, performance improves when the size of the expander layers is larger than the dimension of the representation. The impact of the expander dimension on performance is
SHALOAAE Conchunesr tretuct HHUGGH
ALA 2. Caneees 1010013
192
193
off off Cook (o06, (000
194
195
studied in Appendix D. The training protocol fol-
contnnsscion step
196
lows those of B YOL. and Barlow Twins: LARS optimizer [37, 38] run for 1000 epochs with
198
weight decay of 10 and learning rate Ir batch. batch_siieee200 size/256 x base Ir. where batch size is set
199
to 2048 by default and base_lr is a base learning rate set to 0.2. The learning rate follows a cosine
decay schedule [39], starting from 0 with 10 warmup epochs and with final value of 0.002.
5 Results
202 In this section, we evaluate the representations obtained after self-supervised pretraining of a ResNet-
203 50 [35] backbone with VICReg during 1000 epochs, on the training set of ImageNet, using the training protocol described in section 4.
5.1
Evaluation on ImageNet
208
Following the ImageNet [18] linear evaluation protocol, we train linear classifier on top of the frozen representations of the ResNet-50 backbone pretrained with VICReg. We also evaluate the performance of the backbone when fine-tuned with linear classifier on subset of ImageNet's training set using 1% 10% of the labels, using the split of [12]. We give implementation details about the optimization procedure for these tasks in Appendix C. We have applied the training
207
208
209
210
procedure described in section 4 with three different random initialization. The numbers reported in
212
Table for VICReg are the mean scores, and we have observed that the difference between worse and best run is lower than 0.1 % accuracy for linear classification, which shows that VICReg is a
213
ImmageNet is free to for research purpose and non-commercial use only.
5

---

Table 1:
Evaluation on ImageNet. Evaluation of the representations obtained with ResNet-50 backbone pretrained with VICReg on: (1) linear classification on top of the frozen representations from ImageNet; (2) semi-supervised classification on top of the fine-tuned representations from I %% and 10% of ImageNet samples. We report Top-1 and Top-5 accuracies (in %). Top-3 best self-supervised methods are underlined.
Linear Classification
Semi cemi-supervised Classification
Method
Top-1
Top-5
Top-1
Top-5
1%
10%
1%
10%
Supervised
76.5
25.4
56.4
48.4
80.4
MoCo [3]
60.6
PIRL [2]
63.6
57.2
83.8
CPC v2 [40]
63.8
CMC [41]
66.2
SimCLR [12]
69.3
89.0
48.3
65.6
75.5
87.8
MoCo v2 [17]
71.1
SimSiam [7]
71.3
SWAV [5]
71.8
InfoMin Aug [4]
73.0
91.1
OBoW [8]
73.8
82.9
90.7
BYOL [6]
74.3
91.6
53.2
68.8
78.4
89.0
SWAV (w/ multi-crop) [5]
75.3
53.9
70.2
78.5
89.9
Barlow Twins [9]
73.2
91.0
55.0
69.7
79.2
89.3
VICReg (ours)
73.2
91.1
54.8
69.5
79.4
89.5
214
very stable algorithm. Lack of time has prevented us from doing the same for the semi-supervised classification experiments, and the experiments of section 5.2 and 6, but expect similar conclusion to hold. We compare Table our results on both tasks against other methods on the validation set of ImageNet. The performance of VICReg is on par with the state of the art without using the negative pairs of SimCLR, the clusters of SwAV, the bag-of-words representations of OBoW, or
216
217
219
any asymmetric networks architectural tricks such as the momentum encoder of B BOL and the stop-gradient operation of SimSiam. The performance is comparable to that of Barlow Twins, which
shows that VICReg's more explicit way of constraining the variance and comparing views has the same power than maximizing cross-correlations between pairs of twin dimensions. The main advantage of VICReg is the modularity of its objective function and the potential applicability to multi-modal setups.
222
224
5.2
Transfer to other downstream tasks
226
Following the setup from [2], we train a linear classifier on top of the frozen representations learnt by our pretrained ResNet-50 backbone on a variety of different datasets: the Places205 [42] scene classi- fication dataset, the VOCO7 [43] multi-label image classification dataset and the iNaturalis12018 [44] fine-grained image classification d We then evaluate the quality of the representations by transferring to other vision tasks including VOC07+12 [43] object detection using Faster R-CNN [45] with a R50-C4 backbone, and COCO [46] instance segmentation using Mask-R-CNN [47] with 350-FPN backbone. We give implementation details in Appendix C. We report the performance in Table 2, VICReg performs on par with most concurrent methods, and better than Barlow Twins, across classification tasks, but is slightly behind the top-3 on detection tasks. This could be
227
229
231
232
234
235
explained by the fact that VICReg learns representations that are more invariant to transformation, but eliminates more low-level information about the images than the other methods.
236
6
Ablation study
238
In this section we study how the different components of our method contribute its performance. as well as how they interact with components from other self-supervised methods. All reported
239
2 Places205 was released under the CC-BY license. Pascal VOC, iNaturalist18 18 and COCO are free to use for research purposes and non-commercial only.

---

Table 2: Transfer learning on downstream tasks. Evaluation of the representations from ResNet- 50 backbone pretrained with VICReg on: (1) linear classification tasks on top of frozen representa- tions, we report Top-1 accuracy (in %) for Places20S [42] and iNat18 [44], and mAP for VOCO7 [43]; (2) object detection with fine-tunning, we report SP for VOCO7+12 12 using Faster R-CNN with C4 backbone [45]; (3) object detection and instance segmentation, we report AP for COCO [46] using Mask R-CNN with FPN backbone [47]. We use to denote the experiments run by Top-3 best self-supervised methods are underlined.
Linear Classification
Object Detection VOCO7+12 COCO det COCO seg
Method
Places205
VOCO7 iNat18
Supervised
53.2
87.5
46.7
81.3
39.0
35.4
MoCo [3]
46.9
79.8
31.5
PIRL [2]
49.8
81.1
34.1
SimCLR [12]
52.5
85.5
37.2
MoCo v2 [17]
51.8
86.4
38.6
82.5
39.8
36.1
SimSiam [7]
82.4
BYOL [6]
54.0
86.6
47.6
40.4
37.0
SwAV (w/ multi-crop) [5]
56.7
88.9
48.6
82.6
41.6
37.8
OBoW [8]
56.8
89.3
82.9
Barlow Twins [6]
54.1
86.2
46.5
82.6
40.0
36.71
VICReg (ours)
54.3
86.6
47.0
82.4
39.4
36.4
results are obtained on the linear evaluation protocol using ResNet-50 backbone and 100 epochs of pretraining, which gives results consistent with those obtained with 1000 epochs of pretraining. The optimization setting used for each experiment is described Appendix
242
243
Asymmetric networks. We study the impact of different components used asymmetric architec- tures and the effects of adding variance and covariance regularization, in terms of performance and training stability. Starting from a simple symmetric architecture with encoder and an expander without batch normalization, which correspond to VICReg without batch normalization in the ex- pander, we progressively add batch normalization in the inner layers of the expander, predictor, stop-gradient operation and momentum encoder. We use the training protocol and architecture of SimSiam [7] when stop-graden gradient used and the training protocol and architecture
246
249
B BYOL [6] when momentum encoder is used. The predictor as used in SimSiam and BYOL is a learnable module 9v that predicts the embedding of view given the embedding of the other view of the same image. and v are the embeddings of two views of an image, then = (w) and p' = (=')') the predictions of each view. The invariance loss function of Eq. (5) is now computed between batch embeddings Z .2.....22 2nl and the corresponding batch of predictions P Ip'..... phl. then symmetrized:
250
252
255
1 Σ D( pc), 2n
8 12 (Z,Z,,,P') 2n D(a--)
-1)++
256
where a distance function that depends on the method used. BYOL uses the mean square error between (2- n-normalized vectors, SimSiam uses the negative cosine similarity loss and VICReg the mean square error without Iz-normalization. The variance and covariance terms are regularizing the output Z and Z' of the expander, which we empirically found to work better than regularizing the output of the predictor. We compare different settings in Table 3, based on the default data augmentation, optimization and architecture settings of the original BYO,, SimSiam and VICReg methods. In all settings, the absence of BN indicates that BN is also removed in the predictor when is used.
258
259
261
282
We analyse first the impact of variance regularization (VR) the different settings. When using VR. adding a predictor (PR) to VICReg does not lead to significant change the performance, which indicates that PR is redundant with VR. In comparison, without VR, the representations collapse, and both stop-gradient (SG) and PR are necessary. Batch normalization in the inner layers of the expander (BN) VICReg leads to a 1.0% increase in the performance, which is not a big improvement
285
287
268
considering that SG and PR without BN is performing very poorly at 35.1%
7

---

Table 3: Effect of incorporating variance and covariance regularization in different methods. Top-1 ImageNet accuracy with the linear evaluation protocol after 100 pretraining epochs. For all methods, pretraining follows the architecture, the optimization and the data augmentation protocol of the original method using our reimplementation. ME: Momentum Encoder. SG: stop-gradient. PR: predictor. BN: Batch normalization layers after input and inner linear layers in the expander. No Reg: No additional regularization. Var Reg: Variance regularization. Var/Cov Reg: Variance and Covariance regularization. Unmodified original setups are marked by a a.
Method
ME
SG
PR
BN
No Reg
Var Reg
Var/Cov Reg
BYOL
69.3
70.2
69.5
SimSiam
67.9
68.1
67.6
SimSiam
35.1
67.3
67.1
SimSiam
collapse
56.8
66.1
VICReg
collapse
56.2
67.3
VICReg
/
collapse
57.1
68.7
VICReg
collapse
57.5
68.61
VICReg
collapse
56.5
67.4
270
Finally, incorporating VR with SG or ME further improves the performance by small margins of respectively 0.2% and 0.9%, which might be explained by the fact that these architectural tricks that prevent collapse are not perfectly maintaining the variance of the representations, i.e. very slow collapse is happening with these methods. We explain this intuition by studying the evolution of the standard deviation of the representations during pretraining for BOLL and SimSiam in Appendix D. then analyse the impact of adding additional covariance regularization (CR) the different settings, along with variance regularization. We found that optimization with SG and CR is hard,
272
273
275
278
277
even if analysis of the average correlation coefficient of the representations during pretraining in Appendix shows that both fulfill the same objective. The performance BYO and SimSiam slightly drops compared to VR only, except when PR is removed, where SG becomes useless.
278
279
280
BN is still useful and improves the performance by 1.3%. Finally with CR, PR does not harm the performance and even improves it by a very small margin. VICReg+PR with 1000 epochs of pretraining exactly matches the score of VICReg (73.2% on linear classification).
281
Weight sharing. Contrary most self- supervisd learn- ing approaches based on Siamese architectures. VICReg has several unique properties: (1) weights do not need to be shared between the branches, each branch's 's weights are updated independently of the other branch's weights; (2) the branches are regularized independently, the variance and covariance terms computed each branch indi- vidually; (3) no predictor is necessary unlike with methods
Table 4: Impact of sharing weights or not between branches. Top-1 ac- curacy linear classification with 100 pretraining epochs. In all settings, the encoder and expander of both branches share the same architecture, but either share weights (V), or have different weights in the two branches.
204
285
287
288
290
where one branch predicts outputs of the other branch. Ta- ble 4 shows results ImageNet using the standard linear protocol for situations where the weights of the encoder and the expander are shared or not. In all settings, there collapse and the performance slight drop in accuracy without sharing is likely due to the increased number of parameters. Importantly, the ability of VICReg to function with different parameters, chitectures, and input modalities for the branches widens the applicability to joint-embedding SSL to many applications, including multi-modal signals.
Encoder Expander Top-
293
66.5
294
competitive. The
67.3
67.8
296
68.6
298
299
Loss function coefficients. Table 5 reports the performance for various values of the loss term coefficients in Eq. (6). Without variance regularization the representations immediatcly collapse to a single vector and the covariance term, which has no repulsive effect preventing collapse, has no impact. The invariance term is absolutely necessary and without it the network can not learn any good representations. By simply using the invariance term and variance regularization, which is
302
303
305
306
very simple baseline, VICReg still reaches an accuracy of 57.5%. These results show that variance and covariance regularizations have complementary effects.
8

---

Table 5: Impact of variance-covariance regu- larization. Inv: invariance loss is used, À > 0, variance regularization, μ > O, Cov: covari- ance regularization, V > 0, in Eq. (6).
Impact of normalization. Std: vari- ables are centered and divided by their standard deviation over the batch. This is applied or not to the embedding and the expander hidden layers. 1z: the embedding vectors are 12 n-rroaalized.
Table 6:
Var:
Method
A
Top-1
Expander
Emhedding
Top-1
Inv
1
0
0
collapse
Std
None
68.6
Inv Cov
25
0
I
collapse
Std
Std
68.4
Inv Cov
0
25
collapse
None
None
67.4
Inv Var
1
1
0
57.5
None
Std
67.2
Inv Var Cov (VICReg)
25
25
1
68.6
Std
l2
65.1
Normalizations. VICReg is the first self--upervvsed method for joint-embedding architectures 309 we are aware of that does not require normalization. Contrary to SimSiam, W-MSE, AwAV and B BOL, and others, the embedding vectors are not projected on the unit sphere. Contrary to Barlow Twins, they are not standardized (equivalent batch normalization without the adaptive parameters). Table 6 shows that the best settings do not involve any normalization of the embeddings, whether is batch-we weicc or feature-wise (as in lا normalization). Whenever the embeddings are standardized (lines 3 and 5 in the table) the covariance matrix of Eq. (3) becomes the normalized auto-correlation
310
312
313
314
315
matrix with coefficients between- -1 and 1. This hurts the accuracy by 1.1%. We observe that when unconstrained, the coefficients in the covariance matrix take values in wider range, which seems to facilitate the training process. Standardization is still important component that helps stabilize the training when used in the hidden layers of the expander, and the performance drops by 1.2% when it is removed. Projecting the embeddings on the unit sphere implicitly constrains their standard deviation along the batch dimension to be 1/Vd, where d is the dimension of the vectors. We change the invariance term of Eq. (5) be the mean square error between (2 nz-ormallieed vectors, and the target y in the variance term of Eq. (1) is set to 1 1/Vd instead of 1, forcing the standard deviation to get closer to 1/ /d, and the vectors to be spread out on the unit sphere. This puts lot more
318
317
318
320
321
322
323
324
constraints on the network and the performance drops by 3.5%.
325
7
Discussion
We introduced VICReg, simple approach to self-supervised learning based triple objective: learning invariance different views with invariance term, avoiding collapse of the representations with a variance preservation term, and maximizing the information content of the representation with covariance regularization term. VICReg achieves results on par with the state of the art many downstream tasks, but is not subject to the same limitations as most other methods, particularly because it does not require the embedding branches to be identical or even similar.
327
328
330
331
332 I Limitations. The time and memory costs of VICReg are dominated by the computation of the covariance matrix for each processed batch, which is quadratic in the dimension of the embeddings. Our experimental analysis, which corroborates the analysis of [9], shows that increasing the dimension of the embeddings significantly improves performance. Future work will explore how this quadratic bottleneck can be overcome by different approximation techniques, as well as completely new information maximization approaches based on higher-order statistics, and whether large expander networks are required.
333
334
336
337
SWAV [5] introduced multi-crop. data autlaaammmetaiiion protocol where more than two views produced for each image, which improves considerably the performance on downstream tasks. Using crop with VICReg did not yield any performance improvement and showed signs of overfitting. More generally, multi-crop does not seem to help VICReg, Barlow Twins [9], SimSiam [7] nor BYOL (6). but yields performance improvements with SWAV [5], SimCLR [12] and MoCo [3], which might be related to the fact that these methods are contrastive.
340
341
342
343
344
345
Broader impact. This work increases the domain of applicability of self- learning, and may improve the performance on tasks for which labeled data is scarce, visual or otherwise, such as healthcare, environmental protection, material science, and the understanding and translation of rare languages. Using the method described here is not likely to mitigate the usual issues with machine-learning systems due biases in the data or the model architecture
346
347
319

---

References
350
[1] P. Bachman, R. D. Hjelm, and W. Buchwalter, "Learning representations by maximizing mutual
352 information across views," " in NeurIPS, 2019.
[2] I. Misra and L. d. Maaten, "Self-supervised learning of pretext-invariant representations"" in
354
CVPR, 2020. 1, 3, 6, 7, 16
355
[3] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum contrast for unsupervised visual
356
representation learning," " in CVPR, 2020. 3, 6, 7, 7, 16
[4] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, "What makes for good views
358
for contrastive learning." " in NeurIPS, 2020. 1,6
[5] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, "Unsupervised learning of visual features by contrasting cluster assignments," in NeurIPS, 2020. 1, 3, 4, 6, 7, 9, 14, 15,
360
361
16, 17, 18
362
[6] J. Grill, F. Strub, F, Altché, C. Tallec, H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, "Bootstrap
363
364
your own latent: A new approach self-supervised learning," in NeurIPS, 2020. 1, 4, 6, 7,
365
14, 15, 16, 17, 18
386
[7] X. Chen and K. He, "Exploring simple siamese representation learning," in CVPR, 2020. 1, 2,
367
3, 3,4, 6, 7, 9, 14, 15, 17, 18
388
[8] S. Gidaris, A. Bursuc, G. Puy, N. Komodakis, M. Cord, and P. Pérez, "Online bag-of-visual-
389
words generation for unsupervised representation learning," in CVPR, 2021. 1, 3, 6,
370
99 Zbontar. L. Jing, Misra, Y. LeCun, and S. Deny, "Barlow twins: Self-supervised learning via redundancy reduction," arXiv preprint arxiv: (03.0220 2021. 1, 2, 3, 4, 6, 9, 14, 15, 16,
371
17, 18
373
[10] J. Bromley, I. Guyon, Y. LeCun, E. Sackinger, and R. Shah, "Signature verification using a "siamese" time delay neural network," in NeurlPS, 1994. 1.
375
[11] R. D. Hjelm, A. Fedorov, S. Aavoie-Marchildone K. Grewal, A. Trischler, and Y. Bengio, "Learning deep representations by mutual information estimation and maximization," in ICLR, 2019. 1.3
377
[12] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, "A simple framework for contrastive
379
learning of visual representations," 2020. 1, 4, 5, 6, 7, 9, 14, 15, 17, 18
[13] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, "Deep clustering for unsupervised learning,"
381
in ECCV, 2018. 1,3
[14] P. H. Richemond, J.-B. Grill, F. Altché, C. Tallec, F. Strub, A. Brock, S. Smith, S. De. R. Pascanu, B. Piot, and M. Valko, "Byol works even without batch statisties," arXiv preprint Stf::2010.10241,, 2020. 3,3
383
384
385
[15] Y. Tian, X. Chen, and S. Ganguli, "Understanding self-supervised learning dynamics without
386
contrastive pairs." arXiv preprint 1xXiv:2102.06810 2021. 1
387
[16] A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe, "Whitening for self-supervised representa-
388
tion learning," 2021. 1, 3, 14
389
[17] X. Chen, H. Fan, R. Girshick, and K. He, "Improved baselines with momentum contrastive
390
learning." arXiv preprint arXiv:2003. 042997 2020. 3, 6,
391
[18] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale
392
hierarchical image database," 2009. 3,
393
[19] R. Hadsell, S. Chopra, and Y. LeCun, "Dimensionality reduction by learning an invariant
mapping." in CVPR, 2006. 3
395
20 M. Ye, X. Zhang. P. C. Yuen, and S.-F. Chang, "Unsupervised embedding learning via invariant
and spreading instance feature," in CVPR, 2019. 3
397
[2]] Z. Wu, Y. Xiong, S. Yu, and D. Lin, "Unsupervised feature learning via non-parametric instance discrimination," in CVPR, 2018. 3, 18
399
[22] A. van den Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive 2018. 3
coding," arXiv preprint
401
[23] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton, "Big self-supervised models are strong semi-supervised learners." in NewrlPS, 2020. 3
10

---

403
[24] M. A. Bautista, A, Sanakoyeu, E. Sutter, and B. Ommer, "Cliqueenn: Deep unsupervised exemplar learning," in NeurIPS, 2016. 3
404
405
[25] J. Yang, D. Parikh, and D. Batra, "Joint unsupervised learning of deep representations and image clusters," in CVPR, 2016. 3
407
[26] J. Xic, R. Girshick, and A. Farhadi, "Unsupervised deep embedding for clustering analysis," in ICML., 2016. 3
408
409
[27] Huang. Q. D. andShaogang Gong, and X. Zhu, "Unsupervised deep learning by neighbour- hood discovery," in ICML, 2019. 3
410
411
[28] C. Zhuang, A. L. Zhai, and D. Yamins, "Local aggregation for unsupervised learning of visual embeddings." " in ICCV, 2019. 3, 17, 18
412
[29] M. Caron, P. Bojanowski, J. Mairal, and Joulin, "Unsupervised pre-training of image features
414
on non-curated data," in ICCV, 2019. 3
[30] Y. M. Asano, C. Rupprecht, and A. Vedaldi, "Self-labbeling via simultaneous clustering and representation learning." " in ICLR, 2020. 3
416
417
33 X. Yan, I. Misra, A. Gupta, D. Ghadiyaram, and D. Mahajan, "Clusterii: Improving generaliza- tion of visual representations," in CVPR, 2020. 3
419
[32]
M. Cuturi, "Sinkhorn distances: Lightspeed computation of optimal transport," in NeurlPS, 2013. 3
420
421
[33] S. Gidaris, A. Bursue, N. Komodakis, P. Pérez, and M. Cord, "Learning representations by
422
predicting bags of visual words," in CVPR. 2020. 3
423
[34] G. Hinton, O. Vinyals, and Dean, "Distilling the knowledge in a neural network," in NIPS Deep Learning and Representation Learning Workshop. 2015.
424
[35] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in CVPR,
426
2016. 5, 15
427
(36] S. loffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," in ICML, 2015. 5, 15
428
429
[37] Y. You, I. Gitman, and B. Ginsburg, "Large batch training of convolutional networks," arXiv
430
preprint arXiv: 7708 03888, 2017. 5, 15, 17
431
[38] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, "Accurate, large minibatch sgd: Training imagenet in I hour," arXiv preprint
433
a7Xiv: 02677, 2017. 5, 15
434
[39] I. Loshchilov and F. Hutter, "Sgdr: stochastic gradient descent with warm restarts," in ICLR,
2017. 5, 15
436
[40] O. J. Hénaff, A. Srinivas, J. D. Fauw, A. Razavi, C. Doersch, S. M. A. Eslami, and A. van den Oord, "Data-efficient image recognition with contrastive predictive coding," in ICML, 2019. 6
438
[41] Y. Tian, D. Krishnan, and P. Isola, "Contrastive multiview coding," " arXiv preprint
439
1106::99000844444 2019. 6
440
[42] B. Zhou, A. Lapedriza, Xiao, A. Torralba, and A. Oliva, "Learning deep features for scene
441
recognition using places database," in NeurIPS, 2014. 6, 7, 16
[43] M. Everingham, L.. V. Gool, J. W. Christopher K. I. Williams, and A. Zisserman, "The pascal visual object classes (voc) challenge," " IJCV, 2010. 6, 7, 16
443
[44] G. V. Horn, O. M. Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and Belongie, "The inaturalist species classification and detection dataset," in CVPR, 2018. 6, 7, 16
445
446
447
[45] Ren, K. He, R. Girshick, and J. Sun, "Faster r-enn: Towards real-time object detection with
448
region proposal networks," in NewrIPS, 2015. 6.7
[46] T.-Y. Lin, M. Maire, S. Belongie. L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan,
450
C. L. Zitnick, and P. Dollár, "Microsoft coco: Common objects in context," in ECCV, 2014. 6,7
[47]
K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-cnn," in ICCV, 2017. 6,7
452
[48] V. Nair and G. E. Hinton, "Rectified linear units improve restricted boltzmann machines," in
453
ICML, 2010. 15
454
[49] Goyal, Q. Duval, Reizenstein, M. Leavitt, M. Xu, B. Lefaudeux, M. Singh, V. Reis. M. Caron, P. Bojanowski, A. Joulin, and I. Misra, "Vissl." https:// thub. com/
455
facebookresearchhrr 2021. 16
11

---

457
[50] Y. Wu, A. Kirillov, F. Massa, Y. Lo, and R. Girshick, "Detectron2." https:///gtthut
com/ http://ooooeeeerrccceeeeeeeeeeee wotprrcrccceeeeeteeeeoo.... 2019. 16
459
[51] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, "Liblinear: A library for large linear classification," " JMLR, 2008. 16
460
461
[52] S. Zagoruyko and N. Komodakis, "Wide residual networks," arXiv preprint arXiv:
2016. 17
463
[53] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. Hc, "Aggregated residual transformations for deep neural networks," in CVPR, 2017. 17
485
[54] J. Li, P. Zhou, C. Xiong, and S. C. Hoi, "Prototypical contrastive learning of unsupervised representations." in ICLR, 2021. 18
12

---

Checklist
467
1. For all authors...
469
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7.
472
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
473
Section 7.
474
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
475
478
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
478
(b) Did you include complete proofs of all theoretical results? [N/A]
479
3. If you ran experiments..
480
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See zip archive in supplementary material.
481
(b) Did you specify all the training details (e.g., data splits, hyperparameters. how they were chosen)? [Yes] For all experiments, training details are included in Section 4.2 and Appendix C.
484
485
486
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We discuss running multiple seeds in Section 5.1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix E.
489
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
491
(a) If your work uses existing assets, did you cite the creators? [Yes]
492
(b) Did you mention the license of the assets? [Yes] See footnotes on pages and 6.
493
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Code associated to the main results in the paper.
494
495
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A] All used dataset are opensource.
486
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] All used datasets are publicly available and have been widely used in the research community for many years.
498
499
500
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
502
503
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
504
505
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
506
13


Reviews:
[{"review_id": "u5bUJB8SpfV", "reviewer": "Reviewer_W5JX", "summary": "The authors propose a self-supervision technique with a more flexible structure requirements such as  not requiring batch norm, normalization, weight sharing or quantization.  Their model uses 2 encoders and a loss function with 3 variance and covariance regularization terms.  The authors mention that compared with the previous works, their main novelty lies in one of their variance terms which adjusts the variance of each input dimensions and, in return expected to mitigate collapsing problem. The authors claim that not sharing weights and having regularization terms separate on each encoder branch make their model better suited for multimodal inputs and tasks. They show results mainly on ImageNet classification, object detection tasks, with an ablation study on their and other baseline methods by varying regularization terms.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##### *Originality:*\n\n- The idea of trying to prevent collapsing and using decorrelations, also working on a more flexible model exists in the recent works. The authors' main contribution lies in the details of the formulation of their loss. For example, if we compare with Barlow twins method, we can say that authors' invariance term corresponds to changing the cosine metric to euclidean for the diagonal covariance terms(so the same idea but different metric here). They kept covariance regularization separate for each embedding model as another difference. Exactly using hinge loss to control standard deviation dimension-wise is novel as of my knowledge. \n- Overall, the work is novel in the sense that they collected all these details in one model, but I see these modifications as training tricks rather than a new loss function for self supervision, due to a lack of any theoretical analysis or at least a proof of concept with simpler cases(My opinions about the intuition part are below). \n\n- The  authors did a great job in the supplementary to compare and separate themselves from the previous works.(Also Figure 2 is very informative) I believe they could be mentioned in the main paper in a better, more organized way. \n\n- The related work was adequate, figures and comparisons in the supplementary section could be a good part of a survey paper in my opinion. \n\n- I appreciate the authors efforts for their detailed and transparent ablation analysis (Table 3,5,6 ), a good example for our field.\n\n&nbsp;\n##### *Quality:*\n\n- The authors both present their weaknesses and strengths in their analysis with sufficient experimental detail, it is admirable, but they could provide more intuition why other methods do better than theirs. \n\n- The claims could be better supported. Some examples and questions(if I did not miss out anything)\n     - Why using normalization is a problem for a network or a task (it can be thought as a part of cosine distance)? How would Barlow Twins perform if their invariance term is replaced with a euclidean distance? \n\n     - Your method still uses 2048 as the batch size, I would not consider it as small. For example, Simclr uses examples in the same batch and its batch size changes between 256-8192. Most of the methods you mentioned need even much lower batch size. \n\n     - You mentioned not sharing weights as an advantage, but you have shared weights in your results, except Table 4 in which the results degraded as you mentioned. What stops the other methods from using different weights? It should be possible even though they have covariance term between the embeddings, how much their performance would be affected compared with yours? \nMy intuition is that a proper design might be sufficient rather than separating variance terms. \n\n     - Do you have a demonstration or result related to your model collapsing less than other methods? In line 159, you mentioned gradients become 0 and collapse; it was a good point, is it commonly encountered, did you observe it in your experiments?\n\n     - I am also not convinced to the idea that the images and their augmentations need to be treated separately, they can be interchangeable. \n\n    - Variances of the results could be included to show the stability of the algorithms since it was another claim in the paper(although \"collapsing\" shows it partly, it is a biased criteria since the other methods are not designed for var/cov terms).\n\n    - How hard is it to balance these 3 terms?\n\n    - When someone thinks about gathering two batches from two networks and calculate the global batch covariance in this way; it includes both your terms and Barlow Twins terms. Can anything be said based on this observation, about which one is better and why?\n   \n&nbsp;\n\n##### *Significance:*\n\n- Currently, the paper needs more solid intuition or analysis  or better results to make an impact in my opinion. The changes compared with the prior work are minimal. Most of the ideas and problems in the paper are important, but they are already known.\n\n- The comparisons with the previous work is valuable to the field, they could maybe extend their experiments to the more of the mentioned methods or other variants.  \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did a great job in presenting their work's limitations, their results in general not being better than the previous works and their extensive analysis(tables). If they did a better job in explaining the reasons/intuitions in a more solid way, or include some theory if there is any, I would be inclined to give an accept.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "oJ0CVkupg0Q", "reviewer": "Reviewer_cVf4", "summary": "This paper proposes a simple and effective method(VICReg) that prevents the collapse in self-supervised learning with two regularizations terms. The variance term maintains the variance of the embedded vector along each dimension independently. The covariance term prevents the network encode similar information to the same dimension in the embedded space.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nVICReg achieves comparable results with the SOTA on several downstream tasks.\nVICReg does not require some training constraints on the architecture, which is more applicable.\nThe paper is clearly written and well organized.\n\nCons:\nThe covariance term is borrowed from the Barlow Twins method, which decreases the originality and significance.\nThe improvement of Variance Reg and covariance Reg on other self-supervised methods is marginal. It would be better if VICReg can be plugged into different methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12hours"}, "extra_text": {}}, {"review_id": "ag-lOAO8boX", "reviewer": "Reviewer_a9mx", "summary": "Summary:\nThis paper proposes a new self-supervised learning method. It makes self-supervised learning very elegant: it does not require techniques such as weight sharing between the branches, batch normalization, feature-wise normalization, output quantization, stop gradient, memory banks, etc., and achieves results on par with the state-of-the-art results on several downstream tasks. It does not require that the inputs be of the same nature. This opens the door to the use of non-contrastive self-supervised joint-embedding for multi-modal signals, such as video and audio. I really like this paper. Still, I have some concerns about this paper.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n(Positive) This paper correctly understands the critical problem in self-supervised learning, i.e., the main challenge is to prevent a collapse in which the encoders produce constant or non-informative vectors.\n\n(Positive) This paper makes self-supervised learning very elegant: it does not require techniques such as weight sharing between the branches, batch normalization, feature-wise normalization, output quantization, stop gradient, memory banks, etc., and achieves results on par with the state-of-the-art results on several downstream tasks.\n\n(Positive) This paper correctly understands the critical problem in BYOL, i.e., the dynamics of learning and how they avoid collapse, is not fully understood.\n\n(Negative) There is a tradeoff for Variance, Invariance, and Covariance. I am afraid that different hyper-parameter will result in very different performances. If so, the proposed method will not be robust. Especially when I see Coefficients λ and μ are 25. Could the authors provide some dense results/visualization curves beyond Table 5?\n\n(Positive) The following property is fascinating: it does not require that the inputs be of the same nature. This opens the door to the use of non-contrastive self-supervised joint-embedding for multi-modal signals, such as video and audio. I really like this property very much!\nImportantly, this good property also indicates: The ability of VICReg to function with different parameters, architectures, and input modalities for the branches widens the applicability of joint-embedding SSL to many applications, including multi-modal signals.\n\n(Negative) Unfortunately, in most of our experiments, this paper uses a Siamese net architecture in which the two branches are identical and share weights.\n\n(Negative) Although I like the above property very much, finding a real-world case meeting the property is difficult.  Because if we cross-modal learning, this is supervised learning but NOT unsupervised learning. Please refer to OpenAI's CILP. Could the authors provide a real-world case meeting the above property with an experimental result?\n\n(Negative) In the overall paper, the authors consider BYOL and MOCO very different. Specifically, BYOL is regarded as a distillation method, while MoCo is considered as a contrastive learning method. I am not convinced by this categorization. If the authors think deeply, they will find that BYOL and MoCo are almost the same, except that MoCo has negative samples. The EMA networks (or mean teachers)  in MoCo and BYOL are exactly the same. If BYOL is categorized into knowledge distillation, MoCo could also be considered knowledge distillation, distilling knowledge from both positive and negative samples. Actually, I tend not to categorize methods like BYOL into knowledge distillation because the teacher is just a mean teacher with dynamic changes during the training. Specifically, SimSiam even has no mean teacher, and the learning target is just a copy of the student networks without BP.\n\n(Negative) Is the following sentence correctly written?\nLine 107-108: These methods are effective, but there is no clear understanding of how t but suffer from a lack of explainability regarding the way collapsing solutions are avoided.\n\n (Neutral) I think this paper may over-attribute some contributions from Barlow Twins. Actually, it borrows the decorrelation mechanism of the Barlow Twins. But it includes an explicit variance-preservation term for each variable of the two embeddings and thus does not require any normalization.\n\n(Positive) The description of both motivation and the method are clear and easy to follow.\n\n(Negative) Would you please provide an empirical analysis of gamma in Eqn (1), which is a constant target value for the standard deviation?\n\n(Positive) The following observation is good:\n\"Using the standard deviation and not directly the variance is crucial. Indeed, if we take S(x) = Var(x) in the hinge function, the gradient of S with respect to x becomes close to 0 when x is close to x ̄. In this case, the gradient of v also becomes close to 0, and the embeddings collapse.\"\n\n(Negative) The following results are not good: \"We report the performance in Table 2, VICReg performs on par with most concurrent methods, and better than Barlow Twins, across all classification tasks, but is slightly behind the top-3 on detection tasks. \"\nThe explanation is not persuasive: \"This could be explained by the fact that VICReg learns representations that are more invariant to transformation, but eliminates more low-level information about the images than the other methods.\"\n\n(Neutral) According to Table 3, the proposed method should have used symmetric loss (i.e., Eqn. (8)). Please clarify.\n\n(Negative) The following sentence is confusing:\nLine 315: This hurts the accuracy by 1.1%. Who is compared with who?\n\n(Negative) Two online networks increase both the memory and the running time.\n\n(Positive) I strongly agree with the following observation (in my experiments, I also have these observations): \"Using multi-crop with VICReg did not yield any performance improvement and showed signs of overfitting. More generally, multi-crop does not seem to help VICReg, Barlow Twins [9], SimSiam [7] nor BYOL [6], but yields performance improvements with SwAV [5], SimCLR [12] and MoCo [3], which might be related to the fact that these methods are contrastive.\"\n\n(Negative) Provide evidence for the following claim: \"Self-supervised learning is also trained on data, which may also have a bias.\nUsing the method described here is not likely to mitigate the usual issues with machine-learning systems due to biases in the data or the model architecture.\"\n\n(Negative) The authors claim that multi-cropping yields performance improvements with SwAV [5], SimCLR [12] and MoCo [3], which might be related to the fact that these methods are contrastive. This is not true. For example, DINO and the Triplet loss method are not in line with this claim. They don't belong to contrastive learning, but Multi-crop learning does help them. Maybe the gain is related to loss function but not contrastive learning.\nDINO: Emerging Properties in Self-Supervised Vision Transformers\nTriplet Loss: Towards Solving Inefficiency of Self-supervised Representation Learning\n\n(Negative) Because the proposed method can handle two different networks with different architectures and different weights. I would like to see such an example. Especially when one network is ResNet-50 and another network is a larger network than ResNet-50 (e.g., ResNet-101 or a strong ViT). Could the linear evaluation result of ResNet-50 be very high in these cases? It would be interesting to see such a result because it implicitly contains a knowledge distillation effect that distills the knowledge from the larger network.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\n\nPlease refer to and respond to the \"negative\" items in the above main review.\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "\nNo.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "24"}, "extra_text": {}}, {"review_id": "Z05AqVP2PrT", "reviewer": "Reviewer_oH4g", "summary": "This paper proposes a Variance-Invariance-Covariance regularization (VICReg) for self-supervised learning. The total loss function is the weighted sum of three terms: 1. distance between two views of the same input batch; 2. hinge loss on the variance of the embedding variables, encouraging them to stay away from zero; 3. loss on the covariance between different embedding variables, encouraging them to be close to zero, and hence preventing a representation collapse. The proposed model does not require the embedding branches to be identical, nor does it require mining of contrastive pairs. Experiments are conducted to demonstrate the effectiveness of the proposed regularization.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "** Pros **\n1. The paper is well written and well organized. It is easy to read.\n2. The proposed variance term is easy to understand and implement.\n3. Experiments and ablation study has been conducted to verify the effectiveness of each component of the proposed VICReg.\n\n** Cons **\n1. The novelty and contribution of the proposed work seems to be limited: the authors have stressed multiple times that VICReg does not require the two branches to be identical. But, if I understand correctly, the prior work Barlow Twins also does not have such restriction.\n2. Also, the work seems to be built on the idea of Barlow Twins, with the only addition of the variance term in the proposed VICReg.\n3. The fact that the proposed work does not require batch-wise or feature-wise normalization does not strike me as extremely important. After all, in table 6, the accuracy drop after standardizing the embedding is only 0.2%\n4. The choice of hyperparameter also seems a bit willy-nilly -- why is $\\lambda = \\mu$? Is there any specific reason for such choice?\n5. The method has only comparable accuracy compared to Barlow Twins, which makes me wonder what is the major gain from using variance regularization.\n6. In 5.1, the authors mentioned that the performance of VICReg is on par with state of the art without using asymmetric networks architectural tricks such as the momentum encoder of BYOL. Is there going to be performance gain when such asymmetric network tricks are used? After all, the fact that the proposed architecture does not need to be symmetric is a highlight of the work.\n\n**Review after author response**\nI appreciate author's response addressing my concerns (2-6). The preliminary experiment on comparing Barlow Twins and the proposed work is also promising, but more study on the performance gain is warranted. I am raising my rating to 5.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 3 of the 4 reviewers voted to reject this paper based on the lack of significant differences from prior work, especially Barlow Twins, and the lack of SOTA numerical results. One concern I share with a number of the reviewers is that what is "simple" is somewhat subjective. E.g. it is not obvious that normalization is especially complex or replacing it with a (co)-variance regularizer is necessarily "simpler". While one reviewer points to potential significant gains with multi-modal data, this is a hypothesis not supported by specific claims in the paper as it stands, and therefore that cannot be considered in the current review process. As a result, I am recommending rejection. However, the approach will be a good paper with some revisions (e.g. additional results on multi-modal data).
Decision: Reject