Understanding Dimensional Collapse in Contrastive Self-supervised Learning
Authors: Li Jing
Pascal Vincent
Yann LeCun
Yuandong Tian
Venue: ICLR 2022 Poster (2022)
Keywords: self-supervised learning
contrastive learning
implicit regularization
dimensional collapse

Abstract: Self-supervised visual representation learning aims to learn useful representations without relying on human annotations. Joint embedding approach bases on maximizing the agreement between embedding vectors from different views of the same image. Various methods have been proposed to solve the collapsing problem where all embedding vectors collapse to a trivial constant solution. Among these methods, contrastive learning prevents collapse via negative sample pairs. It has been shown that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse, whereby the embedding vectors end up spanning a lower-dimensional subspace instead of the entire available embedding space. Here, we show that dimensional collapse also happens in contrastive learning. In this paper, we shed light on the dynamics at play in contrastive learning that leads to dimensional collapse. Inspired by our theory,  we propose a novel contrastive learning method, called DirectCLR, which directly optimizes the representation space without relying on a trainable projector. Experiments show that DirectCLR  outperforms SimCLR with a trainable linear projector on ImageNet. 

Published as a conference paper at ICLR 2022
UNDERSTANDING DIMENSIONAL COLLAPSE IN CON- TRASTIVE SELF-SUPERVISED LEARNING
Li Jing, Pascal Vincent, Yann LeCun, Yuandong Tian
Facehook Al Research (1 [1n,,, pascal, yann, yuandong) ffb. com
ABSTRACT
Self-supervised visual representation learning aims to learn useful representations without relying on human annotations. Joint embedding approach bases on max- imizing the agreement between embedding vectors from different views of the same image. Various methods have been proposed to solve the collapsing problem where all embedding vectors collapse to trivial constant solution. Among these methods, contrastive learning prevents collapse via negative sample pairs. has been shown that non-contrastive methods suffer from a lesser collapse problem of a different nature: dimensional collapse, whereby the embedding vectors end up spanning lower-dimensional subspace instead of the entire available embedding space. Here, we show that dimensional collapse also happens in contrastive learn- ing. In this paper, shed light on the dynamics at play in contrastive learning that leads to dimensional collapse. Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which directly optimizes the rep- resentation space without relying on a trainable projector. Experiments show that DirectCLR outperforms SimCLR with a trainable linear projector on ImageNet.
INTRODUCTION
Self-supervised learning aims learn useful representations of the input data without relying on hu-
man annotations. Recent advances in self-supervised visual representation learning based on joint
embedding methods (Misra& & Maaten 20206 He et al. 2020 Chen et al.) 2020a Chen & He 2020
Grill al. (2020; Zbontar al.) 2021: Bardes et al. [222 Chen et al. 2020b Dwibedi et al.
2021
Lict al. | 202T|Misrr & Maaten 2020a HaoChen et al. 2021 Assran et al.] 2021] Caron et al. 2021
show that self-supervised representations have competitive performances compared with supervised ones. These methods generally aim to learn representations invariant to data augmentations by max-
imizing the agreement between embedding vectors from different distortions of the same images.
As there are trivial solutions where the model maps all input to the same constant vector, known the collapsing problem, various methods have been proposed to solve this problem that rely on different mechanisms. Contrastive methods like (ceeCChe&& et al. (2020a) and He et al. (2016) define 'pos- itive' and 'negative" sample pairs which are treated differently the loss function. Non-contrastive methbods like Grill et al. (0220) and Chen & He (2020) use stop-gradient, and an extra predictor to prevent collapse without negative pairs; Caron al. (2018) 2020) use an additional clustering step; and Zbontar et al. (2021) minimize the redundant information between two branches.
These self suf-ruueee learning methods are successful preventing complete collapse whereby all representation vectors shrink into a single point. However, it has been observed empirically in non- contrastive learning methods (Hua et al. (02TT) Tian et al. (202T) that while embedding vectors do not completely collapse; they collapse along certain dimensions. This is known as dimensional collapse (Hua et al. 2021), whereby the embedding vectors only span a lower-dimensional subspace.
In contrastive methods that explicitly use positive and negative pairs in the loss function, it seems intuitive to speculate that the repulsive effect of negative examples should prevent this kind of di- mensional collapse and make full use of all dimensions. However, contrary to intuition, contrastive learning methods still suffer from dimensional collapse (See Fig. 7.. In this work. we theoretically study the dynamics behind this phenomenon. We show there two different mechanisms that

---

Published as a conference paper at ICLR 2022
cause collapsing: (1) along the feature direction where the variance caused by the data augmenta- tion is larger than the variance caused by the data distribution, the weight collapses. Moreover, (2) even if the covariance of data augmentation has smaller magnitude than the data variance along all dimensions, the weight will still collapse due to the interplay of weight matrices at different lay- known implicit regularization. This kind of collapsing happens only in networks where the network has more than one layer.
Inspired by our theory, we propose a novel contrastive learning method, called DirectCLR, which directly optimizes the encoder (i.e., representation space) without relying on a trainable projector. DirectCLR outperforms SimCLR with linear trainable projector on ImageNet.
We summarize our contributions as follows:
We empirically show that contrastive self-supervised learning suffers from dimensional collapse whereby all the embedding vectors fall into lower-dimensional subspace instead of the entire available embedding space.
We showed that there are two mechanisms causing the dimensional collapse in contrastive learning: (1) strong augmentation along feature dimensions (2) implicit regularization driv- ing models toward low-rank solutions,
We propose DirectCLR, a novel contrastive learning method that directly optimizes the rep- resentation space without relying trainable projector. DirectCLR outperforms SimCLR with linear trainable projector.
RELATED WORKS
Self-supervised Learning Methods Joint embedding methods are a promising approach in self- supervised learning, whose principle to match the embedding vectors of augmented views of a training instance. Contrastive methods (Chen et al. 2020a He al. 2016) directly compare train- ing samples by effectively viewing each sample as its own class, typically based on the InfoNCE contrastive loss (van den Oord al. 2018) which encourages representations from positive pairs of examples to be close in the embedding space while representations from negative pairs are pushed away from each other. In practice, contrastive methods are known to require large number of negative samples. Non-contrastive methods do not directly rely on explicit negative samples. These include clustering- based methods (Caron et al. 2018 2020p. redundancy reduction methods (Zbon- tar et al. 2021 Bardes et al. (021)) and methods using special architecture design (Grill et al. 2020 Chen He 2020).
Theoretical Understanding of Self-supervised Learning Although self-supervised learning mod- els have shown success in learning useful representations and have outperformed their supervised counterpart in several downstream transfer learning benchmarks (Chen et al.] (020a), the under- lying dynamics of these methods remains somewhat mysterious and poorly understood. Several theoretical works have attempted to understand it. Arora et al. (019b)) Lee et al. (2020); "Tosh ct al. (202I) theoretically proved that the learned representations via contrastive learning are use- ful for downstream tasks. Tian et al. (2021) explained why non-contrastive learning methods like BYOL (Grill et al. 2020) and SimSiam (Chen He. 2020) work: the dynamics of the alignment of eigenspaces between the predictor and its input correlation matrix play key role in preventing complete collapse.
Implicit Regularization It has been theoretically explained that gradient descent will drive adjacent matrices aligned in linear neural network setting (Ji & Telgarsky 2019). Under the aligned matrix assumption, Gunasekar et (2018) prove that gradient descent can derive minimal nuclear norm solution. |Aror et al. (2019a) extend this concept to the deep linear network case by theoretically and empirically demonstrating that a deep linear network can derive low-rank solutions. In general,
over- over-parane mee--parametrized neural networks tend to find flatter local minima (Saxe et al. 2019 Neyshabur
al. 2019 Soudry al. 2018 Barrett Dherin 202T).
DIMENSIONAL COLLAPSE
Self-supervised learning methods learn useful representation by minimizing the distances between emhedding vectors from augmented images (Figure Tap. On its own, this would result in collapsed
2

---

Published as a conference paper at ICLR 2022
Embeddings
Input
Augmentation
InfoNCE loss
(a) embedding space
(b) complete collapse
(c) dimensional collapse
Figure Illustration of the collapsing problem. complete collapse, the embedding vectors collapse to
same point. For dimensional collapse, the embedding vectors only span a lower dimensional space.
solution where the produced representation becomes constant (Figure Tb
Contrastive methods
prevent complete collapse via the negative term that pushes embedding vectors of different input images away from each other. In this section, we show that while they prevent complete collapse, contrastive methods still experience a dimensional collapse in which the embedding vectors occupy a lower-dimensional subspace than their dimension (Figure
We train SimCLR model (Chen et al. (2020a))
with a two-layer MLP projector. We followed the standard recipe and trained the model ImageNet for 100 epoch. We evaluate the di-
Log of singuiar values
mensionality by collecting the embedding tors on the validation set. Each embedding vec-
-10-
tor has size of d 128. We compute the co-
variance matrix C E Radd the embedding
layer (here z Ex1 i=1 //N and N is the total number samples):
20
40 60
60
100
320
N C
Singular Value Rank Index
Σ - Z)T 1=1
(1)
Figure 2: Singular value spectrum of the embedding space. The embedding vectors are computed from a pretrained SimCLR model on the validation set of Ima- geNet. Each embedding vector has dimension of 128. The spectrum contains the singular values the covari- ance matrix of these embedding vectors sorted or- der and logarithmic scale. A number of singular values drop zero, indicating collapsed dimensions.
Figure 2 shows singular value decomposition on this matrix (C = USV sorted order and logarithmic scale ((log(a*)) (o0)(())))) })). We observe that a number of singular values collapse to zero, thus representing collapsed di- mensions.
diag( o*)) )))).. in
4
DIMENSIONAL COLLAPSE CAUSED BY STRONG AUGMENTATION
4.1 LINEAR MODEL
In this section, explain scenario for contrastive learning to have collapsed embedding dimen- sions, where the augmentation surpasses the input information. We focus on a simple linear network setting. We denote the input vector and the augmentation an additive noise. The network is single linear layer with weight matrix is W. Hence, the embedding vector is z Wx. We focus on typical contrastive loss, InfoNCE (van den Oord et al. 2018):
exp( 1z z/12/2)
L L- Σ log Σjtr exp(-lz; z, 2/2) exp( 7, 7 2/2)
where z; and z/ are pair of embedding vectors from the branches. z, indicates the negative samples within the minibatch. When all z; and z/ are normalized to be unit vector, the negative distance -I2; /2 can be replaced by inner products z z/. The model is trained with a basic stochastic gradient descent without momentum weight decay.
3

---

Published as a conference paper at ICLR 2022
4.2 GRADIENT FLOW DYNAMICS
We study the dynamics via gradient flow, i.e. gradient descent with an infinitesimally small learning rate.
Lemma 1. The weight matrix in a linear contrastive sulf-siee learning model evolves by:
W --
where G= E,E,,,, +82(x() and 82, is the gradient the embedding vector z; (similarly g::
This can be easily proven based on the chain rule. See proof in Appendix B.1 For InfoNCE loss defined in EqnΩ the gradient of the embedding vector for each branch can be written as
Sz. =
Eaylz Z)) + Σ Lajilzy - z:), 336
S
025(2 2s)
(4)
J
where {ai } are the softmax of similarity of between 2, and {2₂}. defined by otj exp( 1 124
(///22/Zn, Ozi cxp(-Ie z/2 /2///2/Zz, and Z; EÿZs exp( lz: 2,1² (2/1/22)222 + z, 2 2/22 Hence, 2jy Since 2, Wx;, have
G
(5)
where
X Σ Σ ais EE
-x)x x,) aasixx- -x,)
x ---0-
[1-aa)(xx) -x))xx
x,)x,"
(6)
i&i
Lemma 2. X is a difference of two PSD matrices:
-
(7)
Here Eo- = Eig axx(x; - xx)(xx x²)² is a weighted data distribution covariance matrix and = (1- - (x, - xx,)" is a weighted augmentation distribution covariance matrix.
See proof in Appendix B.2 Therefore, the amplitude of augmentation determines whether X is a positive definite matrix. Similar to Theorem 3-4 in Tian et al. (2020). Lemma ( also models the time derivative of weight W as a product of W and a symmetric and/or PSD matrices. However, Lemma 2 is much more general: it applies to InfoNCE with multiple negative contrastive terms, remains true when 0; varies with sample pair (i. and holds with finite batch size N. In contrast, Theorem 4 in Tian al. (2020) only works for one negative term in InfoNCE, holds only in the population sense (i.e., N + +0o), and the formulation has residual terms, if asj are not constants.
Next, we look into the dynamics of weight matrix W given property of X. Theorem 1. With fixed matrix X (defined in Eqn 6) and strong augmentation such that X has negative eigenvalues, the weight matrix W has vanishing singular values.
See proof in Appendix B.3
Corollary (Dimensional Collapse Caused by Strong Augmentation). With strong augmentation, the embedding space covariance matrix becomes low-rank.
The embedding space is identified by the singular value spectrum of the covariance matrix on the embedding (Eqn. [b.. C = E, Z)(2; /) = E. W(x x)(x; - -x) "W" /N. Since W has vanishing singular values, C' is also low-rank, indicating collapsed dimensions.
Numerical simulation verifies our theory. We choice input data as isotropic Gaussian with covari- ance matrix Σi, (x, - xx)(x; x
I. We set the augmentation additive Gaussian with
covariance matrix equal to Ex(x x, (x, x// N block diagonal (0, A * I), where the block has the size of 8x8. We plot the weight matrix singular value spectrum in Figure 3 with various augmentation amplitude k. This proves that under linear network setting, strong augmentation leads to dimensional collapse in embedding space.
4

---

Published as a conference paper at ICLR 2022
Our theory in this section is limited to linear network settings. For more complex nonlinear networks, the collapsing condition will still depend on "strong augmentation" but interpreted differently. A strong augmentation will be determined by more compli- cated properties of the augmentation (higher-order statisties of augmentation, manifold property of aug- mentation vs. data distribution) conditioned on the capacity of the networks.
14-
14
Singular
Singular Value Rank Index
DIMENSIONAL COLLAPSE
CAUSED BY IMPLICIT REGULARIZATION
Figure 3:
Weight matrix singular value spectrum with different augmentation amplitude &. The set- ting is single layer lincar toy model with each weight matrix of the size of 16x16, where the block has the size 8x8. Strong augmentation results in vanishing singular values in weight ma- trices.
5.1 TWO-LAYER LINEAR MODEL
With strong augmentation, linear model under In- foNCE loss will have dimensional collapse. How- ever, such scenarios rely on the condition that the network has a limited capacity which may not hold for real cases. On the other hand, when there no strong augmentation (Σ1 < Eo) and thus X matrix remains PSD, single linear model won't have dimensional collapsing. However, interestingly. for deep networks, dimensional collapsing still happens in practice. In the following, regularization, where over-vovraae wver-parametrized linear networks tend to find low-rank solutions.
will show that it stems from different nature: implicit
To understand this counter counter-intuitive phenomena, start with the simplest over-parametie wer-ppaaaetrized setting by choosing the network
two-layer linear MLP without bias. The weight matrices of these two lay- ers are denoted by W1 Rd and W2 E Ra Similar to the setting in Sec the input vector is denoted as x and the augmentation noise. The embedding vector from each branch is z W2W x, hence
x
Ws
W
Input
InfoNCE loss
an additive
W
IR", do not normalize See Figure 4 We use InfoNCE loss defined in Eqn []
Figure 4: Two-layer Linear Model
The model trained with a basic stochastic gradient descent without momentum weight decay.
5.2 GRADIENT FLOW DYNAMICS
Similar to Lemma D we derive the gradient flow on the two weight matrices W, and W2.
Lemma 3. The weight matrices of the two layer linear contrastive supersupe learning model
evolves by (G (8 x +zz xx) is defined in Lemmal 7
Wi = WW"G,
W. = GWT
(8)
This can be casily proven based on the chain rule. See proof in Appendix B.4 For the two layer case, similar to Equ 5 we have the specific form of G:
= -=
(9)
where X is defined in Eqn 6 According Lemma
2 we know that with small augmentation,
X E. E > is positive-definite matrix.
5.3 WEIGHT ALIGNMENT
Since have two matrices W1 and W2. the first question is how they interact with each other. We apply singular value decomposition on both matrices W, and W2, i.., W, = U,S, VTT, W, = U2S2V2 and S diag((of()) S2 diag((o))). The alignment now governed by the interaction
5

---

Published as a conference paper at ICLR 2022
between the adjacent orthonormal matrices V/2 == [ve] and U1 [uf]. This can be characterized by the alignment matrix A = V/2 U2, whose (k, -enttyy represents the alignment between the k-th right singular vector v22 of W2 and the w'-th left singular vector uf of Wi. The following shows that indeed W1 and W2 aligns.
Theorem 2 (Weight matrices align). for all t. (t(t)Wi(t) * O. X(t) is positive-definit and Wi(+-o), W2( (+oo0 have distinctive singular values, then the aligument matrix A VV2 U1 -
See proof in Appendix B.5 Here, we also empirically demonstrate that under InfoNCE loss, the absolute value of the alignment matrix A converges to identity matrix. See Figure 5
The alignment effect has been studied in other scenarios (Ji & Telgarsky 2019 Radhakrishnan et al,, 2020). In the real case, when some of our assumptions singular values in weight matrices, we will not observe perfect alignment. This can be easily un- derstood by the fact that the singular decomposition is no longer unique given degenerate singular values. In our toy experiment, we specifically initialize the weight matrices to have non-degenerate singular values. serve the alignment matrix to converge to block-diagonal matrix, with each block representing group of degenerate singular values.
satisfied, e.g., there are degenerate
real scenario, when weight matrices are randomly initialized, we will only ob-
Given the fact that singular vectors corresponding the same singular value align, we can now study the dynamics of the singular values of each weight ma- trix W1 and W2.
Theorem 3. If W2 and W1 are aligned (i.e., V2 UT), ), then the singular values of the weight matrices W1 and W2 under InfoNCE loss evolve by:
ot ( Xv
(10)
22 = (1) 6T X
Figure 5: Visualization of the alignment matrix A V U1 after training. The setting 2-layer lincar toy model with cach weight matrix the size of 16x16. The alignment matrix converges to an identity matrix.
(11)
See proof in Appendix B.6 According to Eqn. 10
(of)² = (a C. We solve the singular
value dynamics analytically: of = ({a))) +
C')(+ Xvt). This shows that a pair of singular values (singular values with same ranking from the other matrix) have gradients proportional to themselves. Notice that X is positive definite ma-
trix, the term vf T Xvf is always non-negative This explains why we observe that the smallest group of singular values grow significantly slower. See demonstrative experiment results in Figure 6a and
6b
W1 Spectrum
W2 Spectrum
Embedding Space Spectrum
top singuar values
300
1880
180 1088 AiB
400
208 1800
1200
2008
1000
1200 Iterations
201
800
Beratiors
nerations
(a) W
(b) Wa
(c) Embedding Space
Figure 6: Evolution of the singular values of the weight matrices and the embedding space covariance matrix. The setting is 2-layer linear toy model with each weight matrix of the size of 16x The lowest few singular values of cach weight matrix remain significantly smaller.
Corollary (Dimensional Collapse Caused by Implicit Regularization). With small augmentation and over-parametrized linear networks, the embedding space covariance matrix becomes low-rank.
The embedding space is identified by the singular value spectrum of the covariance matrix on the embedding vectors, C (z ( z)=/N= - (x x)(x x) W/W? /N. As

---

Published as a conference paper at ICLR 2022
W2W1 evolves to be low-rank, C' is low-rank, indicating collapsed dimensions. See Figure 6c for experimental verification.
Our theory can also be extended to multilayer networks and nonlinear setting. Please see Appendix C
6 DIRECTCLR
6.1 MOTIVATION
We now leverage our theoretical finding to design novel algorithms. Here we are targeting the projector component in contrastive learning.
Empirically, adding projector substantially improves the quality the learned representation and downstream performance (Chen al., (2200a). Checking the spectrum of the representation layer also reveals a difference with/without projector. To see this, we train two SimCLR models with and without a projector. The representation space spectrum are shown in Figure 7b The dimensional collapse in representation space happens when the model is trained without a projector. Thus, the projector prevents the collapse in the representation space.
Represantations
Embeddings
Log of sinqular volues
Input
-38-
x
Augmentation
InfoNCE loss
Encoder
Projector
-2
1991
1168
Singular Value Rank Index
representation and emhedding
(b) Representation space spectrum
Figure 7: (a) Definition of representation and the embedding space; (h) Singular value spectrums of the representation space of pretrained contrastive learning models (pretrained with without projector). The representation vectors are output from ResNet50 encoder and directly used for downstream tasks. Each representation vector has dimension of 2048. Without projector, SimCLR suffers from dimensional collapse in the representation space.
The projector in contrastive learning is essential to prevent dimensional collapse in the representa- tion space. We claim the following propositions regarding a linear projector in contrastive learning models.
Proposition 1. A linear projector weight matrix only needs to be diagonal. Proposition 2. A linear projector weight matrix only needs to be low-rank.
Based on our theory on implicit regularization dynamics, expect to see adjacent layers W1 (= V/)) and W2(= V(") to be aligned such that the overall dynamics is only governed by their singular values S1 and S2. And the orthogonal matrices VTT and U, are redundant as they will evolve to V U1 I, given S and S2.
Now, let's consider the linear projector SimCLR model and only focus on the channel dimension. W, is the last layer in the encoder, and W2 is the projector weight matrix. Our propositions claim that for this projector matrix W2, the orthogonal component V/2 can omitted. Because the previous layer W, is fully trainable, its orthogonal component (U;) will always evolve to satisfy V. V:TU, I. Therefore, the final behavior of the projector is only determined by the singular values (S2 of the projector weight matrix. This motivates Proposition the orthogonal component of the weight matrix doesn't matter. So we can set the projector matrix as a diagonal matrix.
Also, according to our theory, the weight matrix will always converge to the low-rank. The singular value diagonal matrix naturally becomes low low-rank, so why not just set it low-rank directly? This is the motivation of Proposition

---

Published as a conference paper at ICLR 2022
These propositions are verified via ablation studies in Sec 6.3 Given these two propositions, propose DirectCLR, which is effectively using low-rank diagonal projector.
6.2 MAIN IDEA
We propose to remove the projector in contrastive learning by directly sending sub-vector of the representation vector to the loss function. We call our method DirectCLR. In contrast to all re- cent state-of-the-ar self-supervised learning meth- ods, our method directly optimizes the representa- tion space. See Figure 8 DirectCLR picks a subvec- tor of the representation hyperparameter. Then, it applies standard In- foNCE loss on this normalized subvector z/ /z).
Representations
Input
Augmentation
InfoNCC loss
r(0 dol. where do
X'
Encoder
L L= E; log Σ; exp(&::2,))
exp(ž, K.)
Figure 8: DirectCL.R trainable projector, sim- ply apply InfoNCE loss on the a fixed sub-vector of the representations
We train DirectCLR with a standard recipe of Sim- CLR for 100 epochs ImageNet. The backbone encoder is a ResNet50. More implementation details can be found in the Appendix [D DirectCLR demonstrates better performance compared to SimCLR with trainable linear projector on ImageNet. The linear probe accuracies for each model are listed
Table
Loss function
Projector
Accuracy
SimCLR
2ayer nonlinear projector
66.5
SimCLR
1- I-layer linear projector
61.1
SimCLR
no projector projector
51.5
DirectCLR
62.7
Table Lincar probe accuracy on ImageNct. Each model trained on ImageNet for 100 epochs with standard training recipe. The backbone encoder is a ResNet50. DirectCLR outperforms SimCLR with 1-layer linear projector.
We visualize the learnt representation space spectrum Figure DirectCLR prevents dimensional collapse in the representation space similar to the functionality of trainable projector in SimCLR.
Aayar ieni propeter DinssCUP
residuel
singular vvalues
low-rank
InfoNCE
loss
full-rank
low-renk
-15-
layer (full-rank)
nonlinear conv block
representations
Why is the whole representation vector meaningful in DirectCLR while only part of it receives gradient? It takes advantage of the residual connection the backbone. Thus, the gradient passing through representation vector is low-rank where only the first de channel dimensions enters the ResNet backbone and passes through the last nonlinear conv block. becomes full rank. Therefore, this hidden layer h receives gradients on all channels. During forward pass, h is directly fod to the representa- tion vectors via the residual connection. Therefore, the entire representation vector meaningful.
Figure 10:
Singular Value Rank Index
Figure 9: Representation space spectrum of Di- rectCLR compared to SimCLR (a) with 2-layer nonlinear projector (b) with 1-layer linear pro- jector (c) without projector. The spectrums are computed based on the output from the backbone. using ImgacNet validation set. Similar to Sim- CLR with projectors, DirectCLR able prevent dimensional collapse the representation space.
non-zero. When the gradient
8

---

Published as a conference paper at ICLR 2022
One may suspect that the contrastive loss in DirectCLR does not apply gradient the rest part of the representation vector r[do :]. then why these dimensions would contain useful information?
Here, show that the entire representation vector contains useful information. See Figure T0 First, the gradient backpropagating through the representation vector is low-rank, where only the first dg channel dimensions are non-zero. When the gradient enters the ResNet backbone and passes through the last nonlinear conv block, becomes full rank. Therefore, this hidden layer h receives gradients on all channels. Note that h and have same channel dimension of 2048. Next, consider the forward pass. This hidden layer h is directly fed to the representation vectors via the residual connection. As result, the rest part of the representation vector r(d, addition, we run an ablation study in Sec | to test the linear probe accuracy based only on the "directly" optimized vector. This verifies that the whole representation vector is meaningful.
trivial. In
6.3
ABLATION STUDY
Projector
diagonal
low-rank
Top-1 Accuracy
no projector
51.5
orthogonal projector
52.2
trainable projector
61.1
trainable diagonal projector
60.2
fixed low-rank projector
62.3
fixed low-rank diagonal projector
62.7
Table 2: Ablation study: top-1 accuracies on ImageNet by SimCLR model with different projector settings.
To further verify our hypothesis, we have perform ablation studies.
Proposition (aatioonmmmmaahhe the fact that: (a) an orthogonal constrained projector performs the same as the non-projector setting; (b) fixed low-rank projector performs the same fixed diagonal projector; (c) trainable linear projector performs the same as a trainable diagonal projector.
2 Proposition matches the observation that low-rank projector has the highest accuracy.
Please see more detailed ablation study discuss and additional ablation experiments in Appendix
7 CONCLUSIONS
In this work, we showed that contrastive self-supervised learning suffers from dimensional collapse, where the embedding vectors only span a lower-dimensional subspace. We provided the theoretical understanding of this phenomenon and showed that there are two mechanisms causing dimensional collapse: strong augmentation and implicit regularization. Inspired by our theory, we proposed novel contrastive self- supervised learning method DirectCLR that directly optimizes the representa- tion space without relying on trainable projector. DirectCLR outperforms SimCLR with linear projector ImageNet.
ACKNOWLEDGEMENT
We thank Yubei Chen, Jiachen Zhu, Adrien Bardes, Nicolas Ballas, Randall Balestriero, Quentin Garrido for useful discussions.
REPRODUCIBILITY STATEMENT
We provide detailed proof for all the lemmas and theorems in the Appendices. Code (in Py PoTch) is at / http://hhhhh..cc..ccccaaeeeeeerr rectclr

---

Published as a conference paper at ICLR 2022
REFERENCES
Sanjeev Arora, Nadav Cohen, W. Hu, and Yuping Luo. Implicit regularization in deep matrix fac- torization. In NeurIPS, 2019a.
Sanjeev Arora, H. Khandeparkar, M. Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical
analysis of contrastive unsupervised representation learning. In ICML, 2019b.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Bal- las, and Michael G. Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. ArXiv, abs/2104. 13963, 2021.
Adrien Bardes, J. Ponce, and LeCun. Vicreg: Yarinace-ivarrancc-covannna regularization for self-supervised learning. ArXiv, abs/2105.04900 2021.
D. Barrett and B. Dherin. Implicit gradient regularization. ArXiv, abs/2099.11162 2021.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e e J' J'egou, J. Mairal, Piotr Bojanowski, and Ar- mand Joulin. Emerging properties in self-supervised vision transformers. ArXiv, abs/2104.14294, 2021.
Mario Lezcano Casado and David Martinez-Rubio Cheap orthogonal constraints in neural net- works: A simple parametrization of the orthogonal and unitary group. ArXiv, abs/1901 0t//1901.000 (ts/1991.08428, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2020.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, ArXiv, 000/2003.042977 2020b.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. ArXiv, abs/2104. 14548, 2021.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurlPS, 2020.
Suriya Gunasekar, Blake E. Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro. Implicit regularization in matrix factorization. 2018 Information Theory and Applications Workshop (ITA), pp. 1-10, 2018
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. ArXiv, 0ass22066041566 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu. Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEF/CVF Conference on Computer Vision 2020.
and Pattern Recognition (CVPR), pp.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Yue Wang. Sucheng Ren, and Hang Zhao. On feature decorrelation in self-supervised leaming. ArXiv, abs/2105.00470) 2021.
10

---

Published as a conference paper at ICLR 2022
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. ArXiv, abs/1810.02032,, 2019.
Jing. J. Zbontar, and Y. LeCun. Implicit rank-minimizing autoencoder. ArXiv, abs/2010.00679, 2020.
J. Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Prov- able self-supervised learning. ArXiv, abs/2008.0106/ 2020.
Junnan Li, Pan Zhou, Caiming Xiong, R. Socher, and S. Hoi. Prototypical contrastive learning of unsupervised representations. ArXiv, abs/2005.04966, 2021.
Ishan Misra and L. V. D. Maaten. Self-supervised learning of pretext-invarian invariant representations. 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (C (CVPR), pp. 6706-6716, 2020a.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In CVPR, 2020b.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Y. LeCun, and Nathan Srebro. Towards understanding the role of werrparaaeetrizatoon in generalization of neural networks. ArXiv, abs/1805. 2019.
Adityanarayanan Radhakrishnan, Eshaan Nichani, D. Bernstein, and Caroline Uhler. On alignment in deep linear neural networks. arXiv: Learning, 2020.
Andrew M. Saxe, James L. McClelland, and S. Ganguli. mathematical theory of semantic devel- opment in deep neural networks. Proceedings of the National Academy of Sciences, 116:11537 11546, 2019.
Daniel Soudry, E. Hoffer, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. ArXiv, abs/1710.103455, 2018.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks. arXiv preprint arXiv:2010.0057,, 2020.
Yuandong Tian, Xinlei Chen, and S. Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. ArXiv, abs/2102.06810, 2021.
Christopher Tosh, A. Krishnamurthy, and Daniel J. Hsu. Contrastive learning, multi-vies view redun- dancy, and linear models. ArXiv, abs/2008. 10150, 2021.
Aäron van den Oord, Y. Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/1 807 807.00748, 2018.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint arxiv:2103.03230, 2021.
A
USEFUL LEMMAS
We adapt two useful lemmas from Arora et al. (2019)).
Lemma 4. Given a matrix W and the dynamics that W evolves by W, the singular values of this matrix evolve by:
a " Wy
(12)
where x and x are singular value o² corresponding left and right singular vectors. i.e. the k-th column of matrices U and V respectively.
11

---

Published as a conference paper at ICLR 2022
Proof. Given matrix W and its singular value decomposition W USVT. We have the dynamics of the matrix
W ÜSVT +USVT + USVT
Multiplying U from the left and multiplying V from the right, considering U and V are orthogonal matrices, we have
UW = U $ SVVV
Since diag(a*)) is diagonal matrix, we have
22 uroak -² phppt 7 g² xT Wv Again, considering a and v have unit-norm, we have u k² u and vh T derive
p
vb 0. Therefore,
uu Wyk
Lemma 5. Given matrix W and the dynamics that W evolves by W, the singular vectors of this malrix evolve by:
0 = U( @
(13)
V = @ U
(14)
where represents Hadamard element-wiss wise multiplication. H is a skew-symmetric matrix
[ 11/(0t2 xk
Hk.K"
ifk #
(15)
00
i
Proof. Same as proof for Lemmal we start from the following equation
UFWV U'US+ + $ +VTVV
Considering the fact that UTÚ and TTV are skew-symmetric matrices, whose diagonal terms all zero, we Hadamard-multipyy i to both sides of the equation. Here, has all diagonal values equal zeros and all off off-diagonal values equal to one, have
TOU'VV
(16)
Taking transpose, we have
To WW = U V'
(17)
Right-multiplying Eqn and left mult-yyli Eqn then adding them up, we have
- T WV
Therefore, we have
Ú = ---UH OU' SV
where
H
i
if
Similar proof applies to Eqn 14
Lemma (Alignment matrix dynamics). The alignment matrix defined by = Vi7 U, evolves
by:
-= - @ (A A)) + (11, @ (AF + FA
(18)
where & represents Hadamard (element-wise) multiplication. Hi is a skew-symmetric matrix, whose (k,W) k.) We-eetrry is given by
1/(ot2 - o
H.K.K
ik
(19)
i
and defined by
F = S2U GV.S
(20)
12

---

Published as a conference paper at ICLR 2022
Proof. According to Lemma.
5 have
U (1 (
122 V2 U.
Plugging the above two equations and Eqn 8 be written as
the dynamics of the alignment matrix A V2" U1
can
Á V + U
UTH, O -
-A(Hi @ (1
-A(H 000155504001555555777775655331
+(H₂ (551000111(/11333300555444444)1
A(H @ (11'500000v555555VV77"333
+(1, (55,
A(HI O F+ (112 @ (AFT + )AA))A
where
F= = S2U GV
Lemma (Singular value dynamics). The singular values of the weight matrices Wi and Wa evolve
by:
of = Σ
R Gv > (ut (ut? vojoo' (ub² Gr
(r Jo (u
(21)
o
(22)
Proof. According Lemma
6j = WiI
Plugging in Eqn 8 we have
ot W3 G
= V2S2UZ Gr
Σ (v2 uf) (ub Gr.)
Similar proof applies to Eqn 22
B DELAYED PROOFS
B.1 PROOF OF LEMMA 1
The gradient on matrix is
dL
ØL Oz. 0L Oz dz OW
Σ
dW
Oz. OW
dz. We denote the gradient on z; and x as 8,, and Bx; respectively. Since aw
= x; and 0z.
= x/, we
get
W - " y Σ
w
Bg x + ggx, 7)
13

---

Published as a conference paper at ICLR 2022
B.2 PROOF OF LEMMA 2
Proof. X is defined in Eqn 6
X Σ an (x)- xx,)+ Σ
aj((x; x,)xx - -a-aww( an)( - x
i71
ΣΣ
any x
aijxjx?
azi(x; xx)((x x,)²
iti
jpi
ΣΣ
Σ (1 Oji(x; - xx)x3}
ax))(x - -x)((x - xi)² - Σ
(1 - -x) x.)x,"
Given the fact that Ejjs Oij = 1 dis we have E, Eyy x -1 ax)x(xf Also,
since E, Ejfs jtz iterates all pairs of i,j, can replace the index between i and j. we have
Σ, En4 axxxx = E. Eset az:x.x..
Therefore
x= = ΣΣ
O5:(x; - xx))(x; - xx)(xxxxx)* -x,)= - -o))ax)xx -x)(xxxxxxx)))) -x)²
##
B.3 PROOF OF THEOREM !
Proof. According to Lemma we have
d W WX
(23)
dt
For a fixed X, we solve this equation analyically,
((t) = (0)) exp(Xt)
Apply eigen-decompossitinn X, X UAUT. Then we have exp(Xt) U exx((AA)UT. There- fore,
W W(O) exp(Ar)U"
Because X has negative eigenvalues, i.e., A has negative terms, we have for / - oo, exp(AI) is rank deficient. Therefore, know that W ((oo) is also rank deficient, the weight matrix W has vanishing singular values.
B.4 PROOF OF LEMMA 3
Proof.
The gradient on matrix W2 is
dL
aL iz
ØL δz
+
(24)
dW₂
Dz. aw dz oW,
dz. We denote the gradient on z; and 2 as g., and ex. respectively. Since aw, Wix, we get
Bx Wix, and
DW;
dl. )r = Σ 8g, x. +ggx WT
W2 - dW2
(25)
Similar proof applies to W.
14

---

Published as a conference paper at ICLR 2022
B.5 PROOF OF THEOREM!
2
Here, we prove that under the assumption that singular values are non-degenerate, the alignment matrix A V/2 Ui converges identity matrix.
Proof. According to Lemma
we have
d
(W WiWT)
WGGG" W GWT
dt
d
(W2 (2) WW' W1 W,G"W,
dt
therefore,
d
(W:W W2) W W W2 =
dt
or
Next, we show that the Frobenius norm each weight matrix grow infinitely.
d
d fr(W, = ttr( tr(W,
W 11
dt
dt
According to Eqn
9 G = W, we have
-tr(W? GWT) = (WW,T WWW1 =
W?)
Because X is positive definite matrix and for all I, W2(()W1()) + 0, we know B
W₂ W W is positive semi-definite and B * O. Therefore, tr(B) = Σ. AB(B)= > 0 since not all eigenvalues of B are zero.
Therefore, we know ||iile + +oo (similarly 1|2 + +oo). In the limit t- > to0, we have WTW? W W2
Plug in the singular value decomposition of W1 and W2, we have UT = V.. Assuming W1 and W2 have non-degenerate singular values, due to the uniqueness of eigen-decomposition, we have
U 1/2
therefore,
Vu+U; =
Remark. Note that when the non-degenerate singular value assumption does not hold, the corre- sponding singular vectors are not unique and we will not observe the corresponding dimensions becoming aligned.
B.6 PROOF OF THEOREM 3
Proof. According to Theorem 2 for of and of with same index, the corresponding singular vector pairs v and ut will get aligned, i.e., v u + 8i.j- Therefore, Eqn and Eqn can be simplified
to
ot (u Gvf) tt + (u Gvf)
Insert Eqn and considering the alignment, derive
ot + o Xv
t - x
15

---

Published as a conference paper at ICLR 2022
C EFFECT OF MORE LAYERS AND NONLINEARITY
In our toy model, we focused on a two-layer linear MLP setting. Here, we empirically show that our theory extends multilayer and nonlinear cases, shown in Figure Ila
Stronger over parametrization leads to stronger collapsing effect, which has been shown theoreti- cally (Arora al. 2019a Barrett & Dherin 202T) and empirically (Jing et al.) 2020). This can be explained by the fact that more adjacent matrices getting aligned, and the collapsing in the product matrix gets amplified. Note that for single-layer case, 1, there no dimensional collapse in the embedding space, which is consistent with our analysis.
singular
singular
to
13
10
Singular Value Rank Index
Singular Value Rank Index
(a) multiple layers
(b) nonlinear
Figure 11:
Embedding space singular value spectrum with different layers (a) linear and (b) nonlinear networks. All models use weight matrices with size of 16x16. Adding more layers the network leads more collapsed dimensions. Adding nonlinearity leads a similar collapsing effect.
We empirically show that the collapsing effect also applies the nonlinear scenario. We insert ReLU between linear layers and observe a similar singular value collapse compared to the linear case. See Figure 11b
D IMPLEMENTATION DETAIL.
D.1 AUGMENTATIONS
Each input image is transformed twice to produce the two distorted views for contrastive loss. The image augmentation pipeline includes random cropping, resizing to 224x224, random horizontal flipping. color jittering, grayscale, Gaussian blurring, and solarization.
D.2 NETWORK
Throughout the ImageNet experiments in this paper, we use a ResNet-50 (He et al. 2016) as an encoder. This network has output of dimension 2048, which is called representation vector.
D.3 OPTIMIZATION
We LARS optimizer and train all models for 100 epochs. The batch size is 4096, which fits into 32 GPUs during training. The learning rate is 4.8 as in SimCLR (Chen et al. (020a), which goes through 10 epoch of warming up and then cosine decay schedule.
E HYPERPARAMETER TUNING ON dg
Here, we list the ImageNet accuracy with various dy value in Figure 12 It's easy to see that when du - 0, there's 's too little gradient information coming from the loss, the performance drops. When dn + 2048, the model converges to standard SimCLR without projector, which we know suffers from dimensional collapse in representation space.
16

---

Published as a conference paper at ICLR 2022
Top-1 Imagehet Accuracy
04
0
250
500
750
1040
1250
1500
1750
2000
0,0
Figure 12: Hyperparameter tuning on do based on ImageNet linear probe Top- accuracy.
F ABLATION STUDY DETAIL
Fixed low-rank projector vs Fixed low-rank diagonal projector: DirectCLR is equivalent to SimCLR with fixed low-rank diagoanl projector. performs the same as a SimCLR with fixed low-rank projector, which achieves 62.3% linear probe accuracy. Specifically, the singular values of this low-rank matrix are set to have do numbers of and for the rest, then left- and right- multiply fixed orthogonal matrix. Therefore, their only difference is that this fixed projector has extra fixed orthogonal matrix in between.
Trainable projector trainable diagonal projector: We trained SimCLR model with trainable projector that constrained be diagonal. The model achieves 60.2% linear probe accuracy on ImageNet, which is close to SimCLR with 1-layer linear projector.
Orthogonal projector vs no projector: We train a single layer projector SimCLR model with orthogonal constraint using ExpM parametrization (Casado & Martínez-Rubio 2019). Therefore, the projector weight matrix has all singular values fixed to be 1. This model reaches 52.2% accuracy on ImageNet which is close a SimCLR without projector.
These ablation studies verify the propostion that the SimCLR projector only needs to be diago- nal. Also, according to Table 2 find that low-rank projector setting consistently improves the performance, which verifies proposition 2.
Linear probe on subvector instead of the entire vector: For DirectCLR, we perform a linear probe only on the sub-vector and get 47.9% accuracy on ImageNet. This shows that the rest of r still contains useful information even though it does not see gradient directly coming from the loss function.
Random dropout instead of fixed subvector: Since DirectCLR drops out a number of dimensions for the loss function, it would be natural to ask whether random dropping out can reach the same performance. We train SimCLR model without a projector and randomly feed de number of features to InfoNCE loss every iteration. This model reaches only 43.0% accuracy on ImageNet. This demonstrates the importance applying a fixed subvector, which allows the alignment effect to happen.
17


Reviews:
[{"review_id": "fdkNCzKim30", "reviewer": "Reviewer_z9oW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In 4.2 GRADIENT FLOW DYNAMICS, what is the meaning of $X$ in Eq. 6? How amplitude of augmentation is reflected in Eq. 7? The proof is based on simple linear network setting. I wonder how it extends to more complex nonlinear networks. Also, how 'strong’ is strong augmentation? With more complex structures, the capacity of the network also increases, and I expect it would be more tolerable to `strong’ augmentation.\n\nThe propositions regarding the role of projector are lack of justification. I am not sure whether the claims are correct or not.\n\nDirectCLR picks a subvector of the representation z = r[0 : d_0]. Is there any assumption on the order of feature dimensions? Why do we expect the subvector based approach helps? How it compares to random dropout of features?\n\nThe experimental validation is quite limited. There are a few recent work addressing the problem and propose to leverage only positive pairs for learning. The authors may want to compare with them. \n[1] BYOL: Bootstrap Your Own Latent A New Approach to Self-Supervised Learning, NIPS 2020\n[2] SimSiam: Exploring Simple Siamese Representation Learning, CVPR 2021\n[3] WMSE: Whitening for Self-Supervised Representation Learning, ICML 2021", "summary_of_the_paper": "This paper investigates the collapsing problem of contrastive learning. It attempts to attribute the collapsing phenomena to strong augmentation and implicit regularization, using simple linear network models. Based on the above analysis, the paper then propose a simple sub-vector based CL method called DirectCLR.", "summary_of_the_review": "This paper investigate an important and interesting question for CL. The analysis performed is interesting but more justification may be needed for some propositions. However, the experimental validation is quite limited. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bDV_wmbujO", "reviewer": "Reviewer_t7MP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strength\n\nThe paper presents and studies an important problem in all joint-embedding, comparison-based, self-supervised learning techniques: how to avoid degenerative solutions when representations are collapsed. This problem was previously widely known for the “completely collapse” case, where every representation is mapped to a single point, hence the introduction of explicit negative pairs in contrastive methods to avoid this trivial solution. Recently, some methods have been shown to be able to learn without explicitly negative samples, but these methods suffer from a milder collapsing problem, only spanning a subspace of the given capacity. This paper observes that naive contrastive methods without an embedding projector also suffer from this “dimension collapse” problem. The motivation, problem and current state of understanding are clearly described in Section 2, 3 and intuitively illustrated in Fig.1 and Fig.2.\n\nBy analyzing the gradient flows in a simplified setting with shallow and linear network, Section 4 and 5 prove two different scenarios of why some dimension collapse and embedding space converge to low-rank solution. Both analysis maps well to intuition regarding the problem.\n- The cause in Section 4 can be intuitively understood as: when augmentation is too strong for a given model capacity, that destroys too much similarity information, the representations span a subspace of the full capacity to keep the similarity score high.\n- The causes in Section 5 are studied from the implicit regularization point of view, where multi-layers (linear) networks are implicitly biased to learn a low rank solution.\n\nThe projector is an important heuristic design choice of many state-of-the-art contrastive methods, with or without using explicit negative pairs. Previously the projector component was only motivated by its computation efficiency when computing similarity scores in a lower-dimensional space, however there was no explanation for the performance loss without it. In the light of the dimensionality collapse phenomenon, the projector now shields the dimension of the representation layer from collapsing, thus allowing it to retain more useful information for transfer tasks.\n\n### Concerns\n\nThe theoretical analysis in Section 4 and 5 both relies on the linearity property in the simplified setting. Especially in section 5, where the study of weights alignment seems to only be applicable between two linear layers. The relevance to modern deep nonlinear networks are made only through some empirical evidence from section 5.4. This seems to be a general limitation in the toolbox for theoretical analysis as a field, not just for this paper, so in Figure 6, it would be nice to see more empirical evidence (plots) for multi-layer networks with non-linearity at the same time.\n\nThe proposed DirectCLR training method is equivalent to 1-layer linear projector (SimCLR v1 [1]), so it only explained some benefits of the projector components. In case of multi-layer non-linear projector (SimCLRv2 [2]), dimensionality collapse as visualized using singular value in Fig.9 can not explain the performance gain completely. While this was shown in the first line of Table 1, it was not discussed anywhere in the text. A small discussion of other potential effects of non-linear projector (see review paper [3]) for future research would be helpful.\n\nI am a little confused about Fig.10 and its caption. The caption says “The rest part the representation ... is copied from the previous layer via residual connection, which experiences full rank gradient passed through the last convolution block.” However in the figure, the gradient vector through the residual connection was annotated with “low-rank”. It would be great if the paragraph before Fig.10 can be made better to understand.\n", "summary_of_the_paper": "This paper shows that contrastive methods also suffer from the “dimensionality collapse” phenomenon, a milder version of the “total collapse” problem that originally motivated the development of using explicit negative pairs to prevent learning trivial solutions. Two underlying causes for this problem, i.e too strong data augmentation and implicit low-rank regularization, are proved in simplified settings of shallow, linear networks. The dimensionality collapse problem is then related to the embedding projector component of SimCLR, and theoretical analyses are used to motivate an alternative training technique without the need for the embedding projector.\n", "summary_of_the_review": "This paper empirically demonstrates and theoretically investigates an important problem of representation learning in general and contrastive methods in particular. The paper provides useful insights, related it to a current heuristic architectural design (projection layers) and proposes a novel training technique to get rid of that heuristic.\nWhile the theoretical analysis is performed under simplified settings and its applicability to deep networks are only shown empirically, its limitation is inline with other theoretical works of the field.\nFor the reason above I believe this is a useful contribution to the field and hope to see this get accepted at ICLR.\n\n##### References\n[1] Chen, Ting, et al. \"A simple framework for contrastive learning of visual representations.\" International conference on machine learning. PMLR, 2020.\n\n[2] Chen, Ting, et al. \"Big self-supervised models are strong semi-supervised learners.\" arXiv preprint arXiv:2006.10029 (2020).\n\n[3] Le-Khac, Phuc H., Graham Healy, and Alan F. Smeaton. \"Contrastive representation learning: A framework and review.\" IEEE Access (2020).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "WjKJr82cd7", "reviewer": "Reviewer_HfRQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is interesting. The proposed DirectCLR dropping projector is a novel idea. The experiments are also exciting, which shows that the projector may not be necessary for contrastive learning. I would like to vote for a “weak accept”, because I also have the following concerns:\n \n1). The motivation is not completely solid. I agree that dimensional collapse is an essential issue that should be avoided in training. However, directly increasing the dimensionality is also not good for the model generalization, which is usually regarded as the curse of dimensionality. Sometimes, the low-rank/low-dimensional data may help us find out the intrinsic low-dimensional manifold, and that is why we need the dimensionality reduction technique.\n \n2). The proposed method seems to be contrary to motivation. If we want to avoid dimensional collapse, why do we further reduce the fitting ability of the network? Do fewer features incur the more heavily dimensional collapse?\n \n3). Directly selecting the features from 0 to d0 is simple, but how can we ensure that the useful features are just located in [0, d0]?", "summary_of_the_paper": "This paper firstly studies the dimensional collapse problem in existing contrastive learning (CL) methods. The authors provided both empirical and theoretical results to reveal that the popular SimCLR method may incur dimensional collapse by showing the low-rank property of the covariance. They thus propose a new framework called DirectCLR to solve the dimensional collapse in CL. Experiments on ImageNet demonstrate the effectiveness of the proposed method.\n", "summary_of_the_review": "See above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UItjZiRQq8x", "reviewer": "Reviewer_9VqB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper focuses on the dimensional collapse problem faced by contrastive learning methods, which is a valuable problem to be discussed.\n2. Authors analyze the collapsed problem by analyzing the magnitude of different singular values of the covariance matrix of the embeddings. Furthermore, they analyze the two reasons which may cause the dimensional collapse. The analysis in toy model (e.g. linear layer, additive noise) is convincing.\n\nWeakness:\n1. In sec.4, authors analyze the dynamics of the gradient flow with a linear model, and conclude that strong augmentation will derive low-rank covariance matrix, through lemma 2. However, the analysis lacks experimental validations. Authors do not answer the question that what kinds of augmentations are \"strong\", and only show the toy model that augmentation is additive noises. I believe the question is important to the contrastive learning.\n2. It seems that the analysis in Sec 5 is the properties of all deep models? So I am puzzled that if the Theorem 3 only means the gradients vanishing problem in deep models, I am looking forward to the reply of authors to give a more clear explanation.\n3. About the DirectCLR, I thought d_0 is an important hyper-parameters. However there lacks the ablations on d_0. ", "summary_of_the_paper": "The paper discusses the dimensional collapsed problem in contrastive learning, and analyze two reasons to cause the dimensional collapses solutions. According to the analysis, they provide a simple method named DirectCLR to solve the dimensional collapse problem.", "summary_of_the_review": "The analysis is insightful for some toy settings, but lacks some extensions to the real settings in Sec4. Also experiments are not adequent to validate the points.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3rf6KmsWR2e", "reviewer": "Reviewer_fU6b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n* The mathematical analysis of the representation space is insightful.\n* The claims are theoretically well described and the process of proof is easy to follow with the supplementary proofs in the appendix.\n\n\n### Weaknesses\n\n* No empirical results regarding the strength of augmentation\n\n  Though it is mathematically claimed that the strong augmentation evokes the dimensional collapse of the embedding space, there is no such experimental results that advocate this idea. Besides, how can we find the strength of augmentation that results in the row-rank covariance matrix?\n\n* The gap between the simple linear matrix and the deep model\n\n  Is it reasonable to make analogy between the raw data in the proof with the features extracted from the convolutional network? Wouldn't deep models (CNN encoder) find an optimal way to differentiate two augmented samples in the training process?\n\n* Lack of analysis of the results\n\n  * It seems that the residual connection has the strongest impact on the result, so I feel that more analysis on this factor is needed. Such as, \n       * How this  result interpreted with one of the claims?\n       * How about the result when the residual connection is applied on SimCLR?\n  * There is no explicit advantage of proposed method in terms of the classification performance and the conservation of ranks of the covariance matrix. If SimCLR-based models with projectors are doing well, what is the main advantage of the proposed model?\n\n* Lack of details\n\n  * How do you decide the values of d_0 for the contrastive loss?\n  * What is the dimension of the h, z and r in the Figure 10?\n  * Specification of Conv Block in the Figure 10?\n\n\n", "summary_of_the_paper": "This paper theoretically proves that the contrastive learning results in the dimensional collapse in the feature representation space.\nBy analyzing the covariance matrix of the embedding space, the authors claims that it is specifically arise from the strong augmentation and the implicit regularization.\nTo circumvent this issue, the authors propose a novel method of contrastive learning without using the projector at the end of an encoder.", "summary_of_the_review": "The theoretical analysis of the embedding space are insightful, but it seems not well connected with the experimental results.\nAlso, the gap between the model in the proof process and the ones in the experiment part are quite large.\nI think this paper need some additional work on the experimental part.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The theory and results presented in this paper provide a new method to avoid collapse in contrastive learning.  All but one reviewer recommend acceptance.  The lone negative reviewer is concerned with the limited experiments, but the other reviewers, and the AC, find the experimentation convincing enough to warrant acceptance.
Decision: Accept (Poster)