Preventing Collapse in Contrastive Learning with Orthonormal Prototypes (CLOP)

Authors: Huanran Li, Manh Nguyen, Daniel L. Pimentel-Alarcón
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Deep Learning, Contrastive Learning, Neural Collapse, Image Classification

Abstract:
Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, dimensional collapse, where embeddings converge into a lower-dimensional space, poses a significant challenge, especially in semi-supervised and self-supervised setups. In this paper, we first theoretically analyze the effect of large learning rates on contrastive losses that solely rely on the cosine similarity metric, and derive a theoretical bound to mitigate this collapse. Building on these insights, we propose CLOP, a novel semi-supervised loss function designed to prevent dimensional collapse by promoting the formation of orthogonal linear subspaces among class embeddings. Unlike prior approaches that enforce a simplex ETF structure, CLOP focuses on subspace separation, leading to more distinguishable embeddings. Through extensive experiments on real and synthetic datasets, we demonstrate that CLOP enhances performance, providing greater stability across different learning rates and batch sizes.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730251125699

thread_type:
official_review

<thread_object>
creator: reviewer_tw63

<content>
summary:
The paper introduces a contrastive learning method called CLOP, which aims to address the issue of neural collapse by leveraging orthonormal prototypes. The authors theoretically analyze the collapse phenomenon in self-supervised and semi-supervised contrastive learning setups and propose a novel loss function to mitigate collapse by enforcing orthogonal subspace constraints. Experiments on CIFAR-100 and Tiny-ImageNet suggest that the proposed method stabilizes performance under varying batch sizes and learning rates.

strengths:
*Theoretical Analysis:* The paper provides a theoretical analysis of neural collapse, focusing on the impact of large learning rates in contrastive learning settings. The theoretical insights could offer useful guidance for practitioners concerned with collapse issues in representation learning.

*New Loss Design:* The proposal of CLOP, a loss function to enforce orthogonality among embeddings, is a potentially valuable direction for addressing neural collapse. It could add diversity to the current strategies used in self-supervised and semi-supervised learning frameworks.

weaknesses:
*Motivation:* The primary motivation of CLOP—to prevent neural collapse and enhance feature diversity—bears significant similarity to VICReg, which also aims to mitigate collapse by enforcing variance, invariance, and covariance constraints on representations. Although CLOP takes an orthogonal approach, it does not sufficiently differentiate itself from VICReg in either theoretical justification or practical application. The theoretical analysis mainly emphasizes the necessity for spatial separation, an objective already addressed by VICReg, leading to questions about the novelty of the approach.

*Insufficient Comparative Analysis:* While VICReg is briefly acknowledged, the paper lacks a robust empirical comparison with VICReg to establish CLOP’s superiority or complementary advantages. In particular, the experiments fail to demonstrate a distinct performance edge over VICReg, which diminishes the contribution of the proposed method. For this paper to make a convincing case for CLOP, direct comparisons, especially in scenarios prone to collapse, would be necessary.

*Experimental Design Issues:* The choice of CIFAR-100 and Tiny-ImageNet as benchmark datasets provides limited insights into the scalability and effectiveness of CLOP in larger and more complex real-world scenarios. Furthermore, while CLOP is tested under different batch sizes and learning rates, the experiments do not include variations in data augmentation or pretext tasks, which are known to impact the success of contrastive learning methods.

questions:
Please see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730482214041

thread_type:
official_review

<thread_object>
creator: reviewer_wfg9

<content>
summary:
This paper considers the problem of neural collapse in contrastive learning. The authors theoretically investigate the effect of large learning rate in contrastive learning with cosine similarity and provide a bound for the learning rate to avoid the collapse. The authors also propose a semi-supervised contrastive loss. Experimental results on small image datasets support the claim.

strengths:
A theoretical upper bound of the learning rate for collapse is provided.

weaknesses:
- The definition of neural collapse is incorrect. Neural collapse refers to the phenomenon that within-class variance becomes zero. Although it would result in low-rank representations, "embeddings converge into a lower-dimensional space" is insufficient to properly describe neural collapse. Please refer to [Papyan et al.] that mainly discuss neural collapse, which is not cited in this paper.

[Papyan et al.] Prevalence of Neural Collapse during the terminal phase of deep learning training. PNAS 2021.

- The bound seems to be not practical. 2 is too large learning rate for contrastive learning.

- While experimental results are sensitive to the batch size, there is no discussion about the batch size throughout the theoretical analysis. 

- Experimental setting is limited to small image datasets without transfer learning, so its generalizability is questionable.

- No ablation study on the choice of the similarity function, while the authors claim that the proposed method is better than the one with consine similarity.

- Does Lemma 2 still hold when Eq. (3) is used?

questions:
Please address concerns above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605942008

thread_type:
official_review

<thread_object>
creator: reviewer_rdbf

<content>
summary:
This paper addresses the dimensional collapse problems within the semi-supervised contrastive learning paradigm. This paper first provides theoretical insights into the influence of large learning rates and cosine similarities on dimensional collapse. To mitigate this issue, the authors introduce a method, termed CLOP, aimed at enhancing subspace separation.  Experimental results on CIFAR and Tiny-ImageNet demonstrate the performance improvements of the algorithm.

strengths:
1.	The paper tackles the critical issue of dimensional collapse within semi-supervised learning scenarios.
2.	This paper provides some theoretical clues to support their claims.
3.	The paper is generally well structured and easy to follow.

weaknesses:
1.	The paper appears to conflate two distinct concepts: dimensional collapse and neural collapse. Dimensional collapse in semi-supervised learning (SSL) typically refers to the limited dimensionality of learned SSL features, while neural collapse is associated with a desirable state in supervised training marked by several good qualities such as intra-class alignment and inter-class separation (properties NC1-NC4).
2.	The claim regarding orthogonal structures achieving the most distinguishable classes relative to the Simplex ETF structure is unclear. Simplex ETF theoretically maximizes angular separation, reaching a pairwise cosine value of -1/(k-1), whereas orthogonal structures attain pairwise cosine values of zero.
3.	The issue of dimensional collapse is primarily addressed by the SSL community; however, the paper's focus is on semi-supervised methods, yet it lacks comparison with other semi-supervised baselines.
4.	The proposed method should be compared with ETF-based(or uniform variants) methods[1] to adequately demonstrate its effectiveness. Moreover, to comprehensively address dimensional collapse, additional self-supervised experiments are recommended, including comparisons with ETF-based SSL methods [2] and other methods targeting dimensional collapse [3,4].
5.	Experiments are limited to small-scale datasets, such as CIFAR and Tiny-ImageNet, which may restrict the generalizability of the findings.
6.	The performance improvements are not significant on larger batch sizes.

[1] Targeted supervised contrastive learning for long-tailed recognition.

[2] Combating Representation Learning Disparity with Geometric Harmonization.

[3] Understanding dimensional collapse in contrastive self-supervised learning.

[4] Variance-invariance-covariance regularization for self-supervised learning.

questions:
please refer to the weakness part.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the problem of dimensional collapse in semi-supervised contrastive learning (SSL) by introducing a novel method termed CLOP. The authors provide a theoretical analysis of collapse in SSL, focusing on the effects of large learning rates and cosine similarities. CLOP is designed to enhance subspace separation using orthonormal constraints. The experimental results, primarily conducted on CIFAR-100 and Tiny-ImageNet, show some performance improvements, but significant concerns remain regarding the scope, novelty, and experimental rigor of the work.

The distinction between CLOP and existing methods, such as VICReg, is not sufficiently clear. Both approaches aim to mitigate collapse, but CLOP does not convincingly differentiate itself in terms of theoretical justification or practical utility. The conflation of dimensional collapse and neural collapse in the initial manuscript weakens the theoretical foundation, though this was partially addressed in the rebuttal.

The experiments are limited to small-scale datasets (CIFAR-100, Tiny-ImageNet) and do not convincingly demonstrate the method’s generalizability to larger datasets or other domains. While the rebuttal included results on ImageNet-1K, these additions do not fully address the concerns about scalability or robustness.