Zero-CL: Instance and Feature decorrelation for negative-free symmetric contrastive learning
Authors: Shaofeng Zhang
Feng Zhu
Junchi Yan
Rui Zhao
Xiaokang Yang
Venue: ICLR 2022 Poster (2022)
Keywords: Self supervised learning
representation learning

Abstract: For self-supervised contrastive learning, models can easily collapse and generate trivial constant solutions. The issue has been mitigated by recent improvement on objective design, which however often requires square complexity either for the size of instances ($\mathcal{O}(N^{2})$) or feature dimensions ($\mathcal{O}(d)^2$). To prevent such collapse, we develop two novel methods by decorrelating on different dimensions on the instance embedding stacking matrix, i.e., \textbf{I}nstance-wise (ICL) and \textbf{F}eature-wise (FCL) \textbf{C}ontrastive \textbf{L}earning. The proposed two methods (FCL, ICL) can be combined synthetically, called Zero-CL, where ``Zero'' means negative samples are \textbf{zero} relevant, which allows Zero-CL to completely discard negative pairs i.e., with \textbf{zero} negative samples. Compared with previous methods, Zero-CL mainly enjoys three advantages: 1) Negative free in symmetric architecture. 2) By whitening transformation, the correlation of the different features is equal to zero, alleviating information redundancy. 3) Zero-CL remains original information to a great extent after transformation, which improves the accuracy against other whitening transformation techniques. Extensive experimental results on CIFAR-10/100 and ImageNet show that Zero-CL outperforms or is on par with state-of-the-art symmetric contrastive learning methods.

Published as a conference paper at ICLR 2022
ZERO-CL: INSTANCE AND FEATURE DECORRELA- TION FOR NEGATIVE-FREE SYMMETRIC CONTRASTIVE LEARNING
Shaofeng Zhang', Feng Zhu", Junchi ;an'; Rui Zhao 1.2, Xiaokang Yang' Shanghai Jiao Tong University, "Sense'Time Research [sherrylone, yan.jucchi, xkyang) @ jtu edu. on (zhufeng, zhaorui]esensetimee com
ABSTRACT
For self-super contrastive learning, models can easily collapse and generate trivial constant solutions. The issue has been mitigated by recent improvement on objective design, which however often requires square complexity either for the size of instances (O(N2) or feature dimensions (O))). To prevent such collapse, we develop two novel methods by decorrelating on different dimensions on the instance embedding stacking matrix, i.e., Instance-wise (ICL) and Feature- wise (FCL) Contrastive Learning. The proposed two methods (FCL, ICL) can be combined synthetically, called Zero-CL, where "Zero" means negative samples are zero relevant, which allows Zero-CL to completely discard negative pairs i.c., with zero negative samples. Compared with previous methods, Zero-Cl CI mainly enjoys three advantages: 1) Negative free in symmetric architecture. 2) By whiten- ing transformation, the correlation of the different features is equal zero, allevi- ating information redundancy. 3) Zero-CL remains original information to a great extent after transformation, which improves the accuracy against other whiten- ing transformation techniques. Extensive experimental results on CIFAR-10/100 and ImageNet show that Zero-CL outperforms or is on par with state-of-the-arr symmetric contrastive learning methods.
INTRODUCTION
One of the current main bottlenecks in deep network training is the dependence heavy anno- tated training data, and this motivates the recent surge of interests in unsupervised (Donahue & Simonyan, 2019) and self-supervised (Chen & He, 2021; Chen et al., 2020) methods. Specifically, in self-supervised representation learning (SSL), network is pre-trained without any form of man- ual annotation, thus providing means to extract information from unlabeled data sources (e.g., text corpora, videos, images from the Internet, etc.). In self-supervision, label-based information is re- placed by a prediction problem using a certain context or using pretext task. Pretext task in SSL can mainly be divided into three categories: 1) Generative based approaches (Donahue & Simonyan, 2019) learn to generate or otherwise model pixels in the input space. However, pixel-level generation is computationally expensive and may not be necessary for representation learning. 2) Contextual based methods (Vincent et al., 2008; Pathak et al., 2016; Ye et al., 2019) design pretext tasks (de- noising auto-encoders (Vincent et al., 2008), context auto encoders (Zhang et al., 2016; 2017), etc). 3) Contrastive based methods (Chen et al., 2020; Grill et al., 2020; Caron et al., 2020; Asano et al., 2019) take augmented views of the same image positive pair and others negative pairs. Gener- ally, one positive sample corresponds to lots of negative samples. In recent works, contrastive based methods have shown great promise, achieving state-of-the-ar results in image classification (Chen et al., 2020), video classification (Han et al., 2020) and other downstream tasks (Chen & He, 2021).
However, trivial constant solutions (different samples get the same representation) is easily happen- ing without the proper design of architecture and objective function. The well-known solutions to
Junchi Yan is the correspondence author. Shaofeng Zhang, Junchi Yan. Xiaokang Yang are also with MoE Key Lab of Artificial Intelligence, Artificial Intelligence Institute, Shanghai Jiao Tong University. Rui Zhao is also with Qing Yuan Research Institute, Shanghai Jiao Tong University.

---

Published as a conference paper at ICLR 2022
avoid this problem can be summarized into two parts: asymmetric model architecture and proper objective function. 1) Model architecture: MoCo (He et al., 2020), BYOL (Grill et al., 2020) up- date encoders separately and stopping gradient operation is adopted to avoid such problem. Then, BOI and SimSiam (Chen & He, 2021) introduce a predictor module to avoid collapse, which is composed by MIP (Goodfellow et al., 2016). Current mainstream interpretation is using the predic- lor to construct an asymmetric structure, which is useful to alleviate trivial solutions. 2) Objective function: SimCLR uses symmetric framework in contrastive learning. They prevent trivial solutions by using negative pairs and InfoNCE, where InfoNCE can be divided into an alignment term and a uniformity term (Arora al., 2019). The uniformity term pulls different samples to a hyper-sphere uniformly, forcing obtaining different representations and avoiding trivial solutions. Recently, Bar- low Twins (Zbontar et al., 2021) designs a new objective function from the information redundancy perspective, which also has two terms (an invariance term and redundancy reduction term). The invariance term maximizes the correlation of the same feature across different views, and the redun- dancy term reduces information redundancy. However, for the symmetrie framework, both SimCLR and Barlow Twins require square order complexity in objective functions, and the main complexity comes from the uniformity and redundancy term. In this paper, we propose two methods, named Zero-ICL and Zero-FCL, where Zero-ICL. discards the uniformity term and only requires O(N) complexity by instance-wise whitening. Correspondingly, Zero-FCL discards the redundancy term by feature-wiss whitening and only requires (dd complexity. Our contributions are:
1) We propose new contrastive learning framework prevent trivial solutions. Zero-CL, which includes two parts, i.c., Zero-ICL (instance-wise) and Zero-FCL (feature-wise). either of which can work independently and only requires linear order complexity (objective function).
2) To our best knowledge, Zero-ICL is the first attempt of instance-wise whitening, which is con- ceptually comprehensible for preventing collapses contrastive learning. Note that most previous methods (including other domains beyond vision) e.g. (Eldar & Oppenheim, 2003; Kessy et al,, 2018) only use whitening transformation to reduce the information redundancy on feature-wise.
3) We give empirical analysis on the relationship between previous methods and our Zero-CI,, where previous negative sample consuming methods (Zbontar et al., 2021; Chen et al., 2020) can be re- garded as our method with Lagrangian transformation. Then, we theoretically introduce ZCA-based whitening from the maximal correlation (Kessy et al., 2018) perspective.
4) Experimental results on standard image benchmarks (CIFAR-10/100 and ImageNet- 100ge/) show our method achieves new statc-of-the-arr results for symmetric contrastive learning compared with (Chen et al., 2020; Zbontar et al., 2021), especially for small hidden dimension and batch size.
2 RELATED WORK
We review recent contrastive learning works. We mainly divide the previous contrastive methods into feature-wise and instance-wise according to different contrastive dimensions. We particularly discuss the similarity and differences of some most related methods in Section 5 in detail.
instance-wise contrastive learning. instance-wise wise contrastive learning aims to attract positive pairs and repulse negative pairs (Chen et al., 2020; He et al., 2020), where each pair is composed of two instance views. One of the widely used objective functions in instance-wise learning is InfoNCE, and extensive experiments in (Oord et al., 2018) show that InfoNCE requires negative pairs to avoid trivial solutions. SimCLR (Chen et al., 2020) regards views augmented from different images in a mini-batch as negative pairs. However, SimCLR requires a large batch size improve the accuracy, which is GPU intensive. To address this issue, MoCo (He et al., 2020) proposes storing negative samples in the memory bank and updates the bank by the frst-in-first-on principle. MoCo further proposes two operations to prevent trivial solutions, i.c., momentum update key encoder and stop gradient. In recent works (Chen et al., 2020; Grill et al., 2020), the mainstream explanation of the two operations is they construct an asymmetric framework, which can avoid trivial solutions. Inspired by this, BYOL (Grill et al., 2020) adopts stop gradient operation and EMA algorithm to update target encoder. Further, BYOL introduces a predictor module to build more asymmetric framework. Then, SimSiam (Chen & He, 2021) empirically shows that stop gradient and the predictor is the key component to avoid trivial solutions. Another approach to preventing such solutions is introduced in W-MSE (Ermolov et al., 2021). Different from previous works (Chen et al., 2020; Chen & He, 2021; He et al., 2020), W-MSE requires multiple augmented views to improve the accuracy. Besides, W-
2

---

Published as a conference paper at ICLR 2022
Shared
Lms
X4
Z
Images
H AA
Encoder
MLP g
Whitening
On-diag only
H
Lrea
BB
H
Figure 1: Framework of the proposed Zero-Cle The illustrated embedding Z 8 means there are eight samples with six features. Zero-CL includes two branches, i.e., feature-wise branch (Zero-FCL, bottom) and instance-wise branch (Zero-ICL, top). In instance-wise branch, whiten- ing transformation is conducted on instance dimension. Then, the trace of cross affinity matrix tr((HA)) HD) maximized. Note feature-wise branch is similar instance-wise branch.
MSE simply borrows Cholesky-basee whitening transformation in SSL and the multi-view objective requires (NNM) complexity, where M the number of views.
feature-wise contrastive learning. Different from instance-wise learning, feature-wise methods contrast on feature dimension. Barlow Twins (Zhontar al., 2021) the first feature-wise method, which pushes the cosrelcorel matrix to identity matrix I. Its objective includes both invariance term and redundancy reduction term, where the latter term can avoid trivial solutions. Inspired by Barlow Twins, VICReg (Bardes al., 2021) proposes covariance on feature dimension, and achieves similar results to Barlow Twins. However, both Barlow Twins and VICReg require large contrastive dimensions and the covariance regularization requires (dd) complexity, which could be also regarded as requiring negative pairs on feature dimension (each negative pair is composed of features in different dimensions across images).
regularization term named invariance-
Whitening transformation. Whitening, or sphering, is a common pre-processing step to transform random variables orthogonality. However, due rotational freedom, there are infinitely many possible whitening procedures. There is diverse range of sphering methods, e.g. principal compo- nent analysis (PCA) (Jégou Chum, 2012), Cholesky matrix decomposition (Higham, 1990) and pero-phase component analysis (ZCA) (Bell & Sejnowski, 1997). Compared with other whitening methods, ZCA remains maximal correlation original data, which we will clarify in Sec. 3.6.
3 THE PROPOSED ZERO-CL
In this section, we present the framework of Zero-Cir CL in detail, starting with the whitening transfor- mation procedure, followed by the model's 's framework as well as the objective function. At last, we provide theoretical analysis on whitening from maximal correlation perspective.
3.1 WHITENING TRANSFORMATION
Given set of images {x] extract embedding in contrastive space z, g(f(x;, 0). using an encoder network (He et al., 2016) f(, f) and MLP module g(: where 2; E Rixdd Denote the set of embeddings as Z E xdd whitening transformation can be formulated as:
= h2,..... hx) WZ
(1)
where H E d the whitened embedding matrix. The square matrix W Rdxd xd is the so- called whitening matrix. Since var(H) I. follows that WEW = I and thus W W) W, which is fulfilled if W satisfies T E However, the constraint does not uniquely determine the whitening matrix W. Specifically, for any W RW with orthogonal matrix, R will also be a whitening transformation matrix. Here we present ZCA whitening matrix in detail.
ZCA-based whitening takes E (stacks together eigenvectors of the covariance matrix) the thogonal matrix R. Then ZCA whitening matrix can be written as:
ZCA EA 1/2E
(2)
By rotation matrix E, the whitened data will be as close possible to original data (see Theorem 1). Note that in low-rank conditions, i.e., some eigenvalues equal common approach replacing A -1 112 with (A + AI) 1122 22 to prevent zero division, where À le as default.
3

---

Published as a conference paper at ICLR 2022
7um
Zao Comcanic
KREEK
Poethve
Negatee CAAA
Whitening
Whilening
Enooder & MEP
Encoder & MLP
Encoder MLP
bacoder MLP
XA
B
X A
X
X
X
X
X
T-T
T-T
T-T
T-T
X
X
X
(3) SimCLR
Darlow Twins
(c) Zem-IICL
(d) Zero-FCL
Figure 2: Frameworks of SimCLR (a), Barlow Twins (b), Zero-ICL (c) and Zero-FCL (d), where row vector means embedding of instance, and column vector means vector composed by i- th dimension feature values among all instance. The solid red line means negative pairs and the corresponding objective function includes such negative term. The dash red lines mean the two vectors are orthogonal and thus, there is no need for negative term in objectives.
3.2 SYMMETRIC OBJECTIVE FUNCTION BACKGROUND
For instance sistncce-wiseeeee contrastive learning, avoid trivial solutions in the symmetric framework, the objective function usually can be divided into alignment and uniformity terms (Arora et al., 2019). We now review the loss formulated in SimCLR (Chen et al., 2020)
Linfo (xxxx Dxppo I- f(x) f(x))(((()))
Too Eella
(x.at preppon Oo N MMddad
slignment
uniformity
(3)
where is the temperature hyper- pppemeearameter. x+ and x- are positive and negative samples, respec- tively. The first term is the so-called alignment, which maximizes the similarity between positive pairs. The second term is the key to prevent trivial solution. pushing all the data points on unit hyper-sphere uniformly. Then, we rewrite the feature-wise objective of Barlow Twins as:
LBT = -1- ( Ca)
cΣ 1
(4)
invarianee
redundancy reduction
where C E Raxd is the covariance matrix. The first term encourages correlation of the same fea- ture across different views, while the second term minimizes correlation different features. We observe that the objective of both instanc--wise and feature-wis methods requires square order calculation complexity, where the first term (alignment, invariance) requires linear order complex- ity and the second term (uniformity, redundancy reduction) requires square order complexity. We wonder if we can discard the second term, i.c., completely discard negative pairs.
3.3 INSTANCE-WISE AND FEATURE-WISE WHITENING
Inspired by W-MSE (Ermolov al., 2021), we propose two efficient methods in SSL, which named Zero-ICL (instance-wise wise) and Zero-FCL. (feat. Zero-ICI. can replace the uniformity term in Eq. 3, while Zero-FCL. can replace the redundancy reduction term in Eq. 4.
Instance-wise whitening, Given a set of centered embeddings Z E IR Nxd different from traditional whitening methods, we calculate the affinity matrix by S ZZ" where E NxN Then the ZCA whitening matrix WI-2CA can be calculated by:
w EA 1/²ET
(5)
where E IR N stacks cigenvector of the affinity matrix and Ag is the diagonal variance matrix.
One question is how this transformation replaces the uniformity term. We denote the whitened embedding matrix as H. For rotation orthogonal matrix E i.e E E EE I, the following

---

Published as a conference paper at ICLR 2022
formula becomes I E R NxN. It means embeddings of different samples strictly orthogonal.
HH' E (EA5 1 (EA,
(EAA
Then, we can replace the second part in InfoNCE with instance-wise whitening transformation. Feature-wise whitening, Similar to instance-wise whitening transformation, it is calculated by:
WF- EA ET
where C is the covariance matrix, Ac and E E Rdxd the diagonal variance matrix and eigen- matrix of C, respectively. Denote whitened embedding matrix as e Nxxd, we have TE(H H I, i.e., the correlation of the different features is equal O. Hence, redundancy reduction term in Eq. can be replaced with feature-wise whitening transformation.
3.4 FRAMEWORK AND OBJECTIVE FUNCTION
Like other methods in SSL (Chen et al., 2020; Zbontar et al., 2021; Chen & He, 2021), given batch [X,) sampled from a dataset, we first augment data via distribution of data augmentation T. We term the two views as X and X The two batches are then fed to an encoder (He et al., 2016; Kipf & Welling, 2016) f(- 8), producing batches of embeddings T and T Then, a projector module 7) is used to map representation to contrastive space. The two batches of embeddings are then fed into projector g. producing Z K and Z R Since whitening transformation requires centered vector, we use standard scaling method to each hatch of embeddings. Note that calculate instance-wise and feature-wise whitening transformation matrix, we must center embeddings on feature dimension and instance dimension, respectively. We denote feature-wis and instance-wiss -wie centered embeddings as Z Fea and Z Z as follow:
Z
Z, p4 1siSK Vo
Z Ins
b4 isisd, Vo +
Fea Z
(8)
where Z, is the i-th vector in Z and Z.; is the vector composed of each value at dimension in all embeddings in Z. μ and a are mean and variance of one vector. e is set lc-4 as default.
Instance-wise objective. Now we have two batches of instance-wise centered embeddings AA.ms and Z B,lns By Eq. 5, we can obtain instance-wis wise whitened embeddings A4,nn and H 8,Ins Then, the instance- objective be formulated as:
K Lins Σ
1-I
H A.Ins
H. B.Ins
where Hi,d represents the d-th feature value of the i-th instance.
Feature-wise objective. Similar to Eq. 9, we can obtain feature-wist wise whitened embeddings AACC ZCA and H B.C ZCA via Eq. 7. Then the feature-wise objective be formulated as:
Frea Σ(1- 1
1
H.A. I,
A.Fea -...Feea
(10)
where K denotes the batch size. The overall loss can be formulated as:
min J Lins + X Lrea 8,7
(11)
We the hyper-parameter À in this paper. Note that both losses can work independently and we term Lins only as Zero-ICL (instance-wise) and Leea only as Zero-FCL (feature-wise).
3.5 EMPIRICAL ANALYSIS
Here for simplicity, discuss feature-wise whitening. From the redundancy reduction perspective, high-quality self-supervised embeddings require that: 1) embeddings are in non-trivial constant distribution, where trivial distribution means all the embeddings collapse to single point; 2) posi- tive image pairs share similar semantics; and 3) correlation of the different feature should be zero. Considering feature- wise SSL, for simplicity, denote i-th feature in Z Z, where Z E R. and we can formulate this problem follows:
Z., I N11
min E (dist(Z.,, z.0]] s.t. cov(Z c) 8,7
(12)
5

---

Published as a conference paper at ICLR 2022
where dist(:--) can be quantified any similarity metric. To solve the conditional optimization problem, previous works (Zhang et al., 202 la; Zbontar et al., 2021) transform the hard condition to soft condition via the Lagrangian multiplier. The objective becomes:
minE [Z z + A (IIZ)) Z 1| (1 z I 8,y
(13)
However, the second term of the above objective function requires square order complexity. Hence, propose another approach solve this hard conditional optimization problem, i.e., whitening transform the representations before calculating loss. Then the objective of Eq. 12 is equal to
min E disl (H H.)
(14)
where is the whitened embeddings. Finally, our objective only uses O(d) complexity to solve the conditional problem, which is similar to the instance-wise branch.
3.6 THEORETICAL ANALYSIS ON WHITENING
By Eq. 1. for any W = RW, which satisfies R R = Wrot can be a whitening matrix. For SSL, whitening transformation aims make different instances features orthogonal, and also remains as much original information as possible, The problem can be formulated as:
E Z) T (H 2)] s.t. H = R R R W
(15)
where R. E are the rotation matrix and covariance matrix. Z and H are embeddings before and after whitening, respectively. Since H and are two centered scaling matrices, we can rewrite the constraint follows, and thus minimizing the variance between whitened data and original data equals to maximizing the second term:
E [H Z)
(H
Z)] -I 2tr CREA-1122
(16)
Theorem Maximization of tr (REA 1/2E uniquely determines the rotation matrix R L
Proof 1 Since A is a diagonal matrix, write Eq. 16 as:
tr (REA - 1/22 = 1r( 1/2
RE - A
(17)
where A E RE. Since R. and E are both orthogonal, A is also an orthogonal matrix. So we have Au 1. Note that E a rotation matrix, i.e., E EE I. Thus, and only if I, we can obluin the muximum of Eq. 17. Then, we can complete the proof.
4 EXPERIMENTS
We the following datasets. The implementation and protocol details given in Appendix.
1) Two small-scale datasets CIFAR-10, CIFAR-
Table 1: Accuracy on ImageNet with ResNet-50.
100 (Krizhevsky et al., 2009), are composed of
100 eps
400 eps
32 x 32 images with 10 and 100 classes.
Method
acc I
acc@
acc I
acc @
2) ImageNet-100 and ImageNet-Ik (Deng et al., 2009) sets include 100 and 1k classes, respectively. The datasets are well-balanced in class distribution and the images contain an iconic view of objects, which are the commonly used benchmark in SSL (Zbontar et al., 2021).
SimCLR
66.5
86.31
69.2
89.0
Moco v2
67.1
87.59
71.1
90.1
BYOL
66.3
87.66
73.2
91.3
SwAV
66.5
87.71
70.7
SimSiam
68.1
88.48
70.8
Barlow Twins
67.7
73.1
91.0
Zero-CI.
68.9
88.74
72.6
90.5
Main comparison. We mainly conduct experiments on CIFAR-10/100 and ImageNet-100/11 datasets. For fair comparison, we set batch size as 128 ImageNet-1000 1024 on ImageNet-Ik Ik, and 256 on CIFAR- 10/10.. Table and 2 show the best results on different datasets across different methods, where method outperforms most of the prior arts with symmetric architecture and less complexity. We modify our code based on Barlow Twins, and the reported results are strictly fol- lowed official code. Note that CIFAR-10 and CIFAR- 100 datasets, remove the first maxpool

---

Published as a conference paper at ICLR 2022
Table 2: Main comparison on CIFAR and ImageNet-100 Proj. and Pred. mean the hidden dimen- sion in projector and predictor. Negs means negatives (both feature-wise and instance-wise). All methods are trained 1000 epochs on CIFAR-10/100 (batch size 256) and 400 epochs on ImageNet- 100 (batch size 128). N means number of samples, d means hidden dimensions, M is the number of views and is the clustering classes. The complexity only considers objective function in line with peer works e.g. (Zhang et al., 2021a), since other parts like feedforward computing share the same overhead. Note that some results are directly quoted from solo-learn (da Costa et al., 2021). 1).
CIFAR-10
CIFAR-100
ImageNet- 100
Method
Proj.
Pred.
Negs.
Complexity
Acc@1
Acc@5
Acc@1
A
Acc@1 Acc@5
BYOL
4096
8192
X
((N)
92.61
99.82
70.18
91.36
80.09
94.99
DINO
2048
X
O(N)
89.19
99.31
66.38
90.18
74.84
92.92
SimSium
2048
512
X
OIN)
90.51
99.72
65.86
89.48
77.04
94.02
MoCo V2
2048
X
O(NK)
92.94
99.79
69.54
91.49
78.2
95.5
ReSSL
2048
X
O(N*)
90.63
99.62
65.83
89.51
76.59
94.41
VICReg
2048
X
/
O(N dP)
90.07
99.71
68.54
90.83
79.22
95.06
SWAV
2048
X
O(NC)
89.17
99.68
64.67
88.52
74.28
92.84
W-MSE
256
X
O(NMP)
88.18
99.61
61.29
87.11
69.06
91.22
SimCLR
2048
O(N2)
90.74
99.75
65.39
88.58
77.48
93.42
Barlow Twins
256
X
O(d)
87.39
99.42
57.92
85.23
67.21
90.64
Barlow Twins
2048
X
O(P)
89.57
99.73
69.18
91.19
78.62
94.72
Zero-FCL.
256
X
X
Old)
89.77
99.73
66.81
89.71
75.67
93.63
Zero-FCI.
2048
X
O(d)
90.51
99.76
70.25
91.96
79.32
94.94
Zero-ICL
256
X
O(N)
90.47
99.76
69.33
91.62
78.02
95.61
Zero-CL.
2048
X
O(N + d)
90.81
99.77
70.33
92.05
79.26
94.98
Table 3: Convergence rate. Cholesky-FCL and Table 4: Analysis on negatives. The hidden di- Cholesky-ICL mean replace ZCA whitening with mension of Zero-ICL and -FCL set 256 and Cholesky whitening on feature and instance- wise, 2048. SimCLR w/o negatives means directly us- respectively. For fairness, we the dimension of ing MSE loss after l2 normalization between pos- projection head 128-12 128-128-117 128 and batch size 256.
itive embeddings, while the reproduced Barlow Twins w/o negatives means only using the invari- ance term in Eq. 4.
100 eps
1000 eps
Method
Acc@1
5
Acc@
Acc
Barlow Twins
80.65
99.10
87.27
99.46
w/o negatives
w/ negatives
SimCLR
78. 28.39
98.84
90.16
99.66
Method
Acc@1
Acc@5
Acc@1
Acc
Cholesky-ICI.
Zero-ICL
84.59
76.51
99.28
98.75
90.15
85.24
99.16
99.71
Barlow Twins
16.98
41.26
65.81
90.18
SimCI.
6.92
22.36
64.98
89.91
Zero-FCL
83.2
99.11
88.25
99.41
Cholesky-FCL
82.83
99.02
87.63
99.37
Zero-PCL
Zero-ICL
67.31
67.54
90.85
90.76
62.61
65.95
87.89
90.15
Zero-CI. 85.01
99.36
90.24
99.70
layer and modify the first convolutional layer with kernel size 3 and strides 1 in ResNet-18, which are commonly used tricks in low-resolution datasets (Chen et al., 2020). The MLP (Projector) in methods is quantified as three linear layers with two BNs (loffe & Szegedy, 2015) and ReLU activation function. Table shows the results on CIFAR and ImageNet-100 datasets. We compare our Zero-FCL with Barlow Twins with the same hidden dimension. With 256 hidden dimension, Zero-FCL outperforms Barlow Twins by 8.89% and 8.46 % in top-1 accuracy on CIFAR-100 and ImageNet- respectively. For hidden dimension 2048, Zero-FCL outperforms Barlow Twins 1.07% and 0.7% top-1 accuracy on CIFAR- 100 and ImageNet- 100 datasets. Note that although Zero-ICL is an instance-wiss method and the batch size is set only 128 or 256. Zero-ICL still gets 69.33% top-1 accuracy on CIFAR- 100, which is 3.94% higher than SimCLR.
Convergence rate. Many experiments (Gutmann Hyvärinen, 2010; Chen al., 2020) have shown that contrasting on high dimension sphere converge faster than low dimension sphere. Hence, to explore our methods can converge fast on low contrastive dimension, we conduct extensive experiments with 100 and 1000 epochs on CIFAR-10 dataset. For fair comparisons, we choose two symmetric methods, i.c., SimCLR finstance-wise and Barlow Twins (feature-wise) as baselines. Table illustrates method outperforms the mentioned two methods. Specifically, Zero-FCL outperforms Barlow Twins and SimCLR by a large range (4.36% % 6.63% top-1 accuracy) with 100 epochs, while the improvement decreases after 1000 epochs. We conjecture the reason may be that
7

---

Published as a conference paper at ICLR 2022
AcoRt
bidkien
hadden
batcch
Bottch
(a) Top-1 (hidden)
(b) Top-5 (hidden)
(c) Top-1 (batch)
(d) Top-5 (batch)
Figure 3: Classification accuracy on CIFAR- 100 under different hidden dimensions in (a), (b) and batch size in (c), (d). For (a) and (b), the batch size of Zero-ICL and Zero-FCL are set 256 and 1024. For (c) and (d), the hidden dimension of Zero-ICL and Zero-FCI. is set 256 and 2048, respectively.
Barlow Twins and SimCLR have two terms to optimize, where the first term is the key to learn good embeddings (Arora et al., 2019), and the second term (uniformity or redundancy reduction) is the key to prevent trivial solutions. However, the second term is hard to converge (Wang & Isola, 2020). In contrast, our methods exactly discard the second term via whitening transformation. Thus, our method converges at faster speed. We also report the results of different whitening transformation methods, i.c., Cholesky and ZCA. We find that for instance-wise contrastive learning, Cholesky- based decomposition gets much lower accuracy scores rather than the ZCA-based method. For feature-wise contrastive learning, Cholesky gets a slightly lower accuracy than conjecture this is because ZCA-based whitening remains maximal information of original data (see Theorem 1), which is important for instance-wise learning (slight rotation on instance dimension will heavily influence linear evaluation), while for feature- wise learning, when the features are permuted or rotated, it may not cause serious influence for linear evaluation (Mitra et al., 2002).
We
Sensitivities to batch size. To explore our method's sensitiveness to batch size, we conduct experi- ments with different batch sizes from 32 ~ 1024. For a fair comparison, each method with a different batch size trained with 50K iterations. Fig. 3(c) shows the results on CIFAR- We can ob- serve SimCLR and Zero-ICL get lower accuracy when batch size is small and this phenomenon is not surprising since instance-wise contrastive learning methods usually require large batch size to create negative pairs (SimCLR) or exact statistics (Zero-ICL). Another observation is the accuracy of Zero-FCL saturates when BatchSize = 256, which is consistent with the results in (Zbontar et al., 2021). Note that under 1024 batch size, Zero-ICL outperforms SimCLR by about 3.1% top-1 accuracy and Zero-FCL. outperforms Barlow Twins 3.8% top-1 accuracy. We guess that's because of the diagonal property of whitened embeddings, which makes our methods robust to batch size.
Sensitivities to hidden dimensions. We also explore the sensitivity of our method to hidden dimen- sions, which are set from 32- 4096. Note that the three linear layers in projection MLP g are set as the same input and output dimension. For feature-wise methods (Barlow Twins, Zero-FCL), the batch size set as 256 (best hyper-parameter in Fig. 3(c).. For instance- wise methods (SimCLR, Zero-ICL), we set batch size as 1024. All the results are get by 50K iterations. Fig. 3(a) shows clas- sification accuracy with different hidden dimensions. For Barlow Twins and Zero-FCL, both two methods are heavily influenced by hidden dimensions and Zero-FCL outperforms Barlow Twins under all hidden dimensions settings. Oppositely, because the objective functions of SimCLR and Zero-ICL are instance wise, making them more robust to hidden dimensions than Barlow Twins.
Ablation to negatives. To show Zero-CLi is indeed a negative- -free method, we conduct an ablation study w/ and w/o negative samples and compare our methods with SimCLR, Barlow Twins. For our methods w/ negatives, we add the off-diag entry of cross-correlation matrix in our objective function, which is followed Barlow Twins (Zhontar et al., 2021). Table 4 shows results of four symmetric methods w/ and w/o negatives pairs. Performance of Barlow Twins significantly decreases and SimCLR directly crashes w/o negative pairs. Zero-FCL and Zero-ICL do not require negatives and perform the best without negatives. After adding negatives, both Zero-ICL and Zero-FCL. lose some accuracy. We analyze that is because whitening transformation can be regarded as a constraint, while using negatives in the objective function is also constraint. Such two constraints will trade off the weight of alignment part (Arora et al., 2019), where the alignment part is the key to learn representations in contrastive learning (Grill et al., 2020; Wang & Isola, 2020). Thus, the accuracy of Zero-ICL and Zero-FCL with negatives is lower than those without negatives.
Breaking Symmetry. In line with Barlow Twins (Zbontar et al., 2021), conduct extensive periments with different asymmetric methods, i.c., stop- atop-gradient and predictor. For Zero-ICL, the objective only has alignment part, which has been shown easily to collapse without the asym-
8

---

Published as a conference paper at ICLR 2022
metric framework (Grill al., 2020; Chen & He, 2021). while the proposed instance whitening can solve this problem well. Table 5 gives classification accuracy on ImageNet- 100 with ResNet-18 backbone. In our experiments, the predictor module is composed of two linear layers with batch normalization. Similar to Barlow Twins, we find these asymmetries slightly decrease the perfor- mance of network. For stop gradient, it is easy understand the accuracy dropping, i.e., with- out considering collapse, the information provided by two views will be more precise than single view (Caron al., 2020; Ermolov al., 2021), while for predictor, we guess that's because overfit- ting by the non non-lineat linear transformation power of deep MLP (Goodfellow et al., 2016).
5 FURTHER DISCUSSION
We further compare our work with existing decorrelation methods and highlight our differences.
Relation to CCA-SSG (Zhang et al., 2021a). Inspired by classical Canonical Correlation Analysis, CCA-SSG proposes a new loss function for graph data learning, which similar to Barlow Twins.
LOCA Z 2 +1
[(4A) z - 1|12 (1 Z
1||
(18)
invariance
decorrelation
where the first term is similar to LBT. However, the decorrelation term of LCCA aims to minimize the intra-correlation of different features, while the redundancy term of LB minimizes cross- correlation of different features. Our feature-wise whitening with Fe only requires the invariance term of CCA-SSG or Barlow Twins, while the other term of LCC is replaced by intra-feature- wise whitening, i.e., whitening transformation is conducted inside each batch view.
Relation to W-MSE (Ermolov et al., 2021). W-MSE firstly combines classical whitening transformation with contrastive learning. Compared with this whitening method, Zero-CL mainly has three differences. 1) Method. W-MSE directly borrows whitening transformation on feature-wise, which is widely used in other domains (Huang et al., 2018; Zhang al., 2021b). However, after feature-wise whitening, the instance- wise objective function is adopted, which is groundless and also limits their performance. Depart from W-MSE, Zero-
Table 5: Ablation study of asym- metric/symmetrrc architectures as generated by applying stop gra- dient ('SG' //rediccrtcctoo ('Pred') one of the branches, by follow-
ing (Zbontar al., 2021).
Method
SG
Pred.
Top-1
Top-5
x
x
79.32
94.94
CL aims replace the second term of InfoNCE and CBT. Thus, Zero-ICL uses instance-wise objective function with instance-win wise whitening, and Zero-FCI. uses feature-wise ob- jective function with feature-wist whitening, which is mean- ingful and more comprehensible. Such methods also leverage the performance (See Table 2). 2) Technology. Instead of directly using Cholesky or PCA-based whitening, we first an- alyze which whitening transformation method is really need. 3) SSL settings. W-MSE requires multiple views to boost the accuracy, while Zero-CL can get higher accuracy with only two views.
Vero-FCL.
76.57
94.18
77.81
94.22
74.81
92.81
x
x
78.02
94.11
Zero-ICL
75.12
93.18
74.87
92.72
72.19
91.95
6 CONCLUSION
We have proposed two new methods to prevent degenerate solutions the symmetric architecture of SSL. We first analyze existing objective functions SSL (Cinfor Let and LCCA). Then, we show why Zero-FCL and Zero-ICL can replace the uniformity term of Linfo and redundancy term of LBT, respectively. We further give theoretical analysis on how ZCA based whitening remains maximal information rather than other whitening transformation methods. Finally, we conduct ex- periments on three image benchmarks and results show that our methods can outperform or be on par with prior arts. Further, to the best of our knowledge, Zero-ICI. is the first attempt for instance- wise whitening and we hope it can bring more inspiration to other tasks.
ACKNOWLEDGMENTS
This work in part supported by National Key Research and Development Program of China (200AAAA007700), Shanghai Municipal Science, Technology Major Project (2021SHZDZX0102) and BenseTime Collaborative Research Grant.

---

Published as a conference paper at ICLR 2022
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229 2019.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. arXiv preprint arXiv:1911.0537/ 2019. 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varinne-iivaranne--vviiae regularizationnn for supersuue learning. arXiv preprint arXiv:2105.04996, 2021.
Anthony J Bell and Terrence J Sejnowski. The "independent components" of natural scenes are edge filters. Vision research, 37(23):3327-3338 1997.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09888 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning. pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Victor G. Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self-supervised methods for visual representation learning. https: / htggtt...uuu.... com/ vturriss i solloloooooo lea r,, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Jeff Donahue and Karen Simonyan, Large scale adversarial representation learning. arXiv preprint 1rX1::1907.02544, 2019.
Yonina Eldar and Alan V Oppenheim. Mmse whitening and subspace whitening. IEEE Transac- fions Information Theory, 49(7):1846-1811, 200.,
Alcksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In International Conference on Machine Learning, pp. 3015- 3024. PMLR, 2021.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-sueerviedd learning. arXiv preprint arXiv:2006.000 07733, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-con estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings. 2010.
Tengda Han, Weidi Xie, and Andrew Zisserman. Self-supervised co-training for video representa- tion learning. arXiv preprint arXiv:2010.09709 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Nicholas Higham. Analysis of the Cholesky decomposition of a semi-defiitte matrix. Oxford University Press, 1990.
Lei Huang, Dawei Yang, Bo Lang, and Jia Deng. Decorrelated batch normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 791-800, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Hervé Jégou and Ondiej Chum. Negative evidences and co-occurences image retrieval: The benefit of pea and whitening. In European conference on computer vision, pp. 774-787. Springer, 2012.
Agnan Kessy, Alex Lewin, and Korbinian Strimmer. Optimal whitening and decorrelation. The American Statistician, 72(4):309 314, 2018.
Thomas you2017 Kipf and Max Welling, Semi-supervised classification with graph convo- lutional networks. arXiv preprint 160X::660900290000 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608. 100.:03808..883,, 2016.
Pabitra Mitra, CA Murthy, and Sankar K. Pal. Unsupervised feature selection using feature similar- ity. IEEE transactions on pattern analysis and machine intelligence, 2413)300-312, 2002.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: arXiv:1807.03744: 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue. Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544 2016.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9939. PMLR, 2020.
Mang Yc, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via in- variant and spreading instance feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6210-6219, 2019.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708 03888, 2017.
Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.032..4 2021.
Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S Yu. From canonical correlation
analysis to self-supervised graph neural networks. arXiv preprint arXiv:2106.1244 2021a.
Richard Zhang, Phillip Isola, and Alexci A Efros. Colorful image colorization. In European confer- on computer vision, pp. 649-666 Springer, 2016.

---

Published as a conference paper at ICLR 2022
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
Shengdong Zhang, Ehsan Nezhadarya, Homa Fashandi, Jiayi Liu, Darin Graham, and Mohak Shah. Stochastic whitening batch normalization. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 10978-10988 2021b.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 IMPLEMENTATION IN VISION
Image augmentations. Each input image transformed twice to generate two different views mentioned before. The image augmentation pipeline follows Barlow Twins (Zbontar et al., 2021), composed of: random cropping, resizing to 224 224 (32 x 32 for CIFAR), horizontal flipping, color jittering, converting to gray-scale, Gaussian blurring, and solarization. The last five are applied randomly on two views with different probabilities.
Architecture. Followed by recent works (Chen et al., 2020; Zbontar et al., 2021), the encoder consists of ResNet-50 and ResNet-18 (He et al., 2016) (without the final classification layer, 2048, 512 output units, respectively) followed by MLP module. In line with Barlow Twins (Zbontar et al., 2021), the MLP module contains three linear layers, each with the same output units. The first two layers of the MLP are followed by BatchNorm layer (loffe & Szegedy, 2015).
Optimization. Similar to previous works (Grill et al., 2020; Zbontar et al., 2021), we use the LARS optimizer (You al., 2017) on both three image datasets. We use a learning rate of 0.2 for the weights and 0.005 for the biases and batch normalization parameters. We multiply the learning rate by batch size and divide by 256. We use learning rate warm-up period of the first 10 epochs, after which we reduce the learning rate by factor of 1000 using cosine decay scheduler (Loshchilov Hutter, 2016). For CIFAR-10 10 and CIFAR- 100, we use single 1080 GPU. For ImageNet- 100, the batch size is set as 128 as default, and we use 8 Tesla V100 16G GPUs. For ImageNet, we use 64 1080Ti GPUs.
Evaluation. We train linear classifier on three vision datasets on top of fixed representations of ResNets pre-trained by Zero-CL. Specifically, the linear classifier is trained for 100 epochs with learning rate of 0.3 and cosine learning rate scheduler. We minimize the cross-entropy loss with SGD optimizer with momentum 0.9 and weight decay 1c-6. In line with previous arts (Zbontar et al., 2021: Chen et al., 2020), set batch size 256. At the inference stage, resize the image to 256 x 256 and center crop it to a size of 224 x 224.
13


Reviews:
[{"review_id": "mGzn80xGLCO", "reviewer": "Reviewer_1y87", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ This paper dedicates to whitening transformation with a ZCA-based solution. It is intuitive to apply the whitening to both instance and feature dimensions, which makes the objective function of ICL and FCL consistent.\n+ Paper is well presented and easy to follow.\n\n- The key concerns came from the symmetric framework. Despite the fancy and consistent solution, the problem is down to the different properties of feature dimensions and instances. For image classification, we believe the ideal feature representation should have decorrelated non-redundant dimensions. However, the correlation between instances is inevitable - the inner-class correlation should be encouraged. Is a pure orthogonal matrix that pushes every instance unique the best solution? Should it be a \"block-based\" matrix with ones filled in the same class?\n- Another big concern is about the claimed contribution in complexity. It is argued the second term in Eq(13) results in square order complexity. And the improvement is made by the replacement of the whitened embedding. I am not super clear why this can reduce the computational complexity.\n\n\n- Eq (9) seems an error: should it be Z_{i,d}^{A,Ins}\\dot Z_{i,d}^{B,Ins}?\n- Decorrelating for redundancy removal can often result in low-rank effects (it is equivalent to implicit feature selection or sampling) in the transformed matrix. It would be good to discuss and potential properties and problems resulting from the low-rank issues.\n- In the literature review, the background, properties and advantages of ZCA compared to PCA and Cholesky should be extended to clarify the motivation of using ZCA.\n- Eq(4) the \"1\" should be in bold to denote an identity matrix?\n", "summary_of_the_paper": "This paper addresses the collapse problem during self-supervised contrastive learning. ICL and FCL methods are proposed to decorrelate instances and features. Zero-Contrastive Learning can discard negative pairs with advantages in negative free, reducing correlation of different features to zero and retaining information during transformation. Promising experimental results are demonstrated on CIFAR and ImageNet.", "summary_of_the_review": "Overall, it is a good paper. I am keen to understand why the symmetric framework can be consistently applied to features and instances given they have different properties in the contexts of image classification. I am also not clear about the complexity reduction, which seems to be claimed as a very important contribution in this paper.\n\nMinor issues include some typos in equations and the discussion of low-rank property and ZCA background.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XxP2_zzb4lv", "reviewer": "Reviewer_kq6q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This loss function is somewhat new. However, the justification is relatively weak. The work is good, though at this stage, the authors have only been able to show it gives satisfactory results. It is unclear whether this is just an incremental work among the recent advances.\n\nThis formulation is still using other samples as negative samples. The negative samples are unrelated to the positive samples  in the loss function. This is nice but in theory it is still quite similar to the recent contrastive learning approaches.\n\nTypo: eq 9 Z -> H.", "summary_of_the_paper": "This paper proposes a novel contrastive loss by whitening the embedding vectors in two ways: along the instance dimension and along the feature dimension. The results are comparable to recent works.\n\n", "summary_of_the_review": "The paper is satisfactory.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IOdyh2GAiI3", "reviewer": "Reviewer_AaoY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n+ The proposed formulation can be integrated into various methods as a plug-and-play component.\n+ The approach is clean and well backed-up both theoretically and experimentally.\n+ In-depth ablation study\n\nCons/Quest:\n- How frequent is the mode collapse to trivial solutions in practice for the competing methods? Most of this approaches are quite stable in practice even without using negative samples (i.e. SimSiam).\n- What is the actual advantage in wall-clock training time? Does the lower complexity in theory translates well in practice?\n- In Table 1, what is the explanation behind the fact that the proposed approach lags behind as the number of iteration increases?\n- Can this approach scale well to larger models too?\nMinor: \n- Section 2, a few paragraphs don't start with a capital letter.", "summary_of_the_paper": "The paper is concerned with preventing the model collapse in the self-supervised learning scenario using contrastive losses. The main addition is an adaptation to the existing formulations that performs instance or feature whitening, avoiding the use of negative examples than can be expensive to store and to compute the similarity. Experiments on the selected datasets show that the method generally matches the current top-performing methods.", "summary_of_the_review": "Overall the idea looks clear and efficient, it's not clear however how problematic for the current approaches the collapse is and some additional clarifications could strengthen the work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-ivbKlyGZsc", "reviewer": "Reviewer_GfmZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work presents a good extension on the basis of the previous regularization-based SSL method [1], and further, proposes Zero-CL from the instance and feature-wise aspects. The empirical experiment shows only incremental performance gain over the prior arts like Barlow Twins, yet the proposed Zero-CL has an obvious complexity advantage as it's not dependent on the feature dimension. \n\nSome technical doubts regard the Zero-CL:\n1. It seems the parameter \\lambda regulates the weight of feature/instance terms, is there any intuition behind the setting of lambda=1?\n\n2.  According to what is shown in Figure 3, L_{fea} has a consistent Acc@1 trend with Barlow Twins (larger dimension yields better results) but there does have a seemingly optimal hidden-size (1024 as shown in the Figure). Does this indicate that there's a conflict between the instance/feature-wise objective? In such a case, how could one decide the best-hidden size besides empirical experiences?\n\n[1] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021.", "summary_of_the_paper": "This paper presents a new self-supervised learning objective that aims to further adapt the previous decorrelation-based [1] contrastive learning method [Bardes, 2021], which largely alleviates the trivial solutions in SSL. Compared with prior arts, the proposed Zero-ICL/FCL are constructed from mainly the aspects of instance and feature-wise. In particular, Zero-CL requires no negative samples; feature-wise FCL discards the redundancy term by feature-wise whitening, and the proposed ICL prevents the collapse of contrastive learning effectively. Quantitative evaluations validate that Zero-CL leads to on-par performances with previous state-of-the-art results.\n\n", "summary_of_the_review": "- The paper is very constructed and has a clear motivation, comprehensive ablative studies, and good analytic discussions.\n\n- The only concern from the reviewer is whether the complexity advantage is a sufficient contribution over the previous regularization method [1], given that Zero-CL shows only incremental performance gain. \n\n- The exhibited theoretical analysis of Zero-CL with previous efforts gives direct and clear comparisons. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews for this paper were somewhat diverging, however the paper did not receive any significant negative criticism to push it towards below the acceptance threshold. The reviewers have found some minor issues about the paper. Following the reviewer recommendations, the meta reviewer recommends acceptance.
Decision: Accept (Poster)