Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the collapsing problem of contrastive learning. It attempts to attribute the collapsing phenomena to strong augmentation and implicit regularization, using simple linear network models. Based on the above analysis, the paper then propose a simple sub-vector based CL method called DirectCLR.
main_review: In 4.2 GRADIENT FLOW DYNAMICS, what is the meaning of $X$ in Eq. 6? How amplitude of augmentation is reflected in Eq. 7? The proof is based on simple linear network setting. I wonder how it extends to more complex nonlinear networks. Also, how 'strong’ is strong augmentation? With more complex structures, the capacity of the network also increases, and I expect it would be more tolerable to `strong’ augmentation.

The propositions regarding the role of projector are lack of justification. I am not sure whether the claims are correct or not.

DirectCLR picks a subvector of the representation z = r[0 : d_0]. Is there any assumption on the order of feature dimensions? Why do we expect the subvector based approach helps? How it compares to random dropout of features?

The experimental validation is quite limited. There are a few recent work addressing the problem and propose to leverage only positive pairs for learning. The authors may want to compare with them. 
[1] BYOL: Bootstrap Your Own Latent A New Approach to Self-Supervised Learning, NIPS 2020
[2] SimSiam: Exploring Simple Siamese Representation Learning, CVPR 2021
[3] WMSE: Whitening for Self-Supervised Representation Learning, ICML 2021
summary_of_the_review: This paper investigate an important and interesting question for CL. The analysis performed is interesting but more justification may be needed for some propositions. However, the experimental validation is quite limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper firstly studies the dimensional collapse problem in existing contrastive learning (CL) methods. The authors provided both empirical and theoretical results to reveal that the popular SimCLR method may incur dimensional collapse by showing the low-rank property of the covariance. They thus propose a new framework called DirectCLR to solve the dimensional collapse in CL. Experiments on ImageNet demonstrate the effectiveness of the proposed method.

main_review: This paper is interesting. The proposed DirectCLR dropping projector is a novel idea. The experiments are also exciting, which shows that the projector may not be necessary for contrastive learning. I would like to vote for a “weak accept”, because I also have the following concerns:
 
1). The motivation is not completely solid. I agree that dimensional collapse is an essential issue that should be avoided in training. However, directly increasing the dimensionality is also not good for the model generalization, which is usually regarded as the curse of dimensionality. Sometimes, the low-rank/low-dimensional data may help us find out the intrinsic low-dimensional manifold, and that is why we need the dimensionality reduction technique.
 
2). The proposed method seems to be contrary to motivation. If we want to avoid dimensional collapse, why do we further reduce the fitting ability of the network? Do fewer features incur the more heavily dimensional collapse?
 
3). Directly selecting the features from 0 to d0 is simple, but how can we ensure that the useful features are just located in [0, d0]?
summary_of_the_review: See above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper shows that contrastive methods also suffer from the “dimensionality collapse” phenomenon, a milder version of the “total collapse” problem that originally motivated the development of using explicit negative pairs to prevent learning trivial solutions. Two underlying causes for this problem, i.e too strong data augmentation and implicit low-rank regularization, are proved in simplified settings of shallow, linear networks. The dimensionality collapse problem is then related to the embedding projector component of SimCLR, and theoretical analyses are used to motivate an alternative training technique without the need for the embedding projector.

main_review: ### Strength

The paper presents and studies an important problem in all joint-embedding, comparison-based, self-supervised learning techniques: how to avoid degenerative solutions when representations are collapsed. This problem was previously widely known for the “completely collapse” case, where every representation is mapped to a single point, hence the introduction of explicit negative pairs in contrastive methods to avoid this trivial solution. Recently, some methods have been shown to be able to learn without explicitly negative samples, but these methods suffer from a milder collapsing problem, only spanning a subspace of the given capacity. This paper observes that naive contrastive methods without an embedding projector also suffer from this “dimension collapse” problem. The motivation, problem and current state of understanding are clearly described in Section 2, 3 and intuitively illustrated in Fig.1 and Fig.2.

By analyzing the gradient flows in a simplified setting with shallow and linear network, Section 4 and 5 prove two different scenarios of why some dimension collapse and embedding space converge to low-rank solution. Both analysis maps well to intuition regarding the problem.
- The cause in Section 4 can be intuitively understood as: when augmentation is too strong for a given model capacity, that destroys too much similarity information, the representations span a subspace of the full capacity to keep the similarity score high.
- The causes in Section 5 are studied from the implicit regularization point of view, where multi-layers (linear) networks are implicitly biased to learn a low rank solution.

The projector is an important heuristic design choice of many state-of-the-art contrastive methods, with or without using explicit negative pairs. Previously the projector component was only motivated by its computation efficiency when computing similarity scores in a lower-dimensional space, however there was no explanation for the performance loss without it. In the light of the dimensionality collapse phenomenon, the projector now shields the dimension of the representation layer from collapsing, thus allowing it to retain more useful information for transfer tasks.

### Concerns

The theoretical analysis in Section 4 and 5 both relies on the linearity property in the simplified setting. Especially in section 5, where the study of weights alignment seems to only be applicable between two linear layers. The relevance to modern deep nonlinear networks are made only through some empirical evidence from section 5.4. This seems to be a general limitation in the toolbox for theoretical analysis as a field, not just for this paper, so in Figure 6, it would be nice to see more empirical evidence (plots) for multi-layer networks with non-linearity at the same time.

The proposed DirectCLR training method is equivalent to 1-layer linear projector (SimCLR v1 [1]), so it only explained some benefits of the projector components. In case of multi-layer non-linear projector (SimCLRv2 [2]), dimensionality collapse as visualized using singular value in Fig.9 can not explain the performance gain completely. While this was shown in the first line of Table 1, it was not discussed anywhere in the text. A small discussion of other potential effects of non-linear projector (see review paper [3]) for future research would be helpful.

I am a little confused about Fig.10 and its caption. The caption says “The rest part the representation ... is copied from the previous layer via residual connection, which experiences full rank gradient passed through the last convolution block.” However in the figure, the gradient vector through the residual connection was annotated with “low-rank”. It would be great if the paragraph before Fig.10 can be made better to understand.

summary_of_the_review: This paper empirically demonstrates and theoretically investigates an important problem of representation learning in general and contrastive methods in particular. The paper provides useful insights, related it to a current heuristic architectural design (projection layers) and proposes a novel training technique to get rid of that heuristic.
While the theoretical analysis is performed under simplified settings and its applicability to deep networks are only shown empirically, its limitation is inline with other theoretical works of the field.
For the reason above I believe this is a useful contribution to the field and hope to see this get accepted at ICLR.

##### References
[1] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[2] Chen, Ting, et al. "Big self-supervised models are strong semi-supervised learners." arXiv preprint arXiv:2006.10029 (2020).

[3] Le-Khac, Phuc H., Graham Healy, and Alan F. Smeaton. "Contrastive representation learning: A framework and review." IEEE Access (2020).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper theoretically proves that the contrastive learning results in the dimensional collapse in the feature representation space.
By analyzing the covariance matrix of the embedding space, the authors claims that it is specifically arise from the strong augmentation and the implicit regularization.
To circumvent this issue, the authors propose a novel method of contrastive learning without using the projector at the end of an encoder.
main_review: ### Strengths

* The mathematical analysis of the representation space is insightful.
* The claims are theoretically well described and the process of proof is easy to follow with the supplementary proofs in the appendix.


### Weaknesses

* No empirical results regarding the strength of augmentation

  Though it is mathematically claimed that the strong augmentation evokes the dimensional collapse of the embedding space, there is no such experimental results that advocate this idea. Besides, how can we find the strength of augmentation that results in the row-rank covariance matrix?

* The gap between the simple linear matrix and the deep model

  Is it reasonable to make analogy between the raw data in the proof with the features extracted from the convolutional network? Wouldn't deep models (CNN encoder) find an optimal way to differentiate two augmented samples in the training process?

* Lack of analysis of the results

  * It seems that the residual connection has the strongest impact on the result, so I feel that more analysis on this factor is needed. Such as, 
       * How this  result interpreted with one of the claims?
       * How about the result when the residual connection is applied on SimCLR?
  * There is no explicit advantage of proposed method in terms of the classification performance and the conservation of ranks of the covariance matrix. If SimCLR-based models with projectors are doing well, what is the main advantage of the proposed model?

* Lack of details

  * How do you decide the values of d_0 for the contrastive loss?
  * What is the dimension of the h, z and r in the Figure 10?
  * Specification of Conv Block in the Figure 10?



summary_of_the_review: The theoretical analysis of the embedding space are insightful, but it seems not well connected with the experimental results.
Also, the gap between the model in the proof process and the ones in the experiment part are quite large.
I think this paper need some additional work on the experimental part.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses the dimensional collapsed problem in contrastive learning, and analyze two reasons to cause the dimensional collapses solutions. According to the analysis, they provide a simple method named DirectCLR to solve the dimensional collapse problem.
main_review: Strengths:
1. The paper focuses on the dimensional collapse problem faced by contrastive learning methods, which is a valuable problem to be discussed.
2. Authors analyze the collapsed problem by analyzing the magnitude of different singular values of the covariance matrix of the embeddings. Furthermore, they analyze the two reasons which may cause the dimensional collapse. The analysis in toy model (e.g. linear layer, additive noise) is convincing.

Weakness:
1. In sec.4, authors analyze the dynamics of the gradient flow with a linear model, and conclude that strong augmentation will derive low-rank covariance matrix, through lemma 2. However, the analysis lacks experimental validations. Authors do not answer the question that what kinds of augmentations are "strong", and only show the toy model that augmentation is additive noises. I believe the question is important to the contrastive learning.
2. It seems that the analysis in Sec 5 is the properties of all deep models? So I am puzzled that if the Theorem 3 only means the gradients vanishing problem in deep models, I am looking forward to the reply of authors to give a more clear explanation.
3. About the DirectCLR, I thought d_0 is an important hyper-parameters. However there lacks the ablations on d_0. 
summary_of_the_review: The analysis is insightful for some toy settings, but lacks some extensions to the real settings in Sec4. Also experiments are not adequent to validate the points.
review_title: null

Meta-review: The theory and results presented in this paper provide a new method to avoid collapse in contrastive learning.  All but one reviewer recommend acceptance.  The lone negative reviewer is concerned with the limited experiments, but the other reviewers, and the AC, find the experimentation convincing enough to warrant acceptance.
Decision: Accept (Poster)