Title: Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: contrastive learning

self-supervised learning

neural collapse
Abstract: Despite its empirical success, the theoretical foundations of self-supervised contrastive learning (CL) are not yet fully established. In this work, we address this gap by showing that standard CL objectives implicitly approximate a supervised variant we call the negatives-only supervised contrastive loss (NSCL), which excludes same-class contrasts. We prove that the gap between the CL and NSCL losses vanishes as the number of semantic classes increases, under a bound that is both label-agnostic and architecture-independent.

We characterize the geometric structure of the global minimizers of the NSCL loss: the learned representations exhibit augmentation collapse, within-class collapse, and class centers that form a simplex equiangular tight frame. We further introduce a new bound on the few-shot error of linear-probing. This bound depends on two measures of feature variability—within-class dispersion and variation along the line between class centers. We show that directional variation dominates the bound and that the within-class dispersion's effect diminishes as the number of labeled samples increases. These properties enable CL and NSCL-trained representations to support accurate few-shot label recovery using simple linear probes.

Finally, we empirically validate our theoretical findings: the gap between CL and NSCL losses decays at a rate of $\mathcal{O}(\frac{1}{\#\text{classes}})$; the two losses are highly correlated; minimizing the CL loss implicitly brings the NSCL loss close to the value achieved by direct minimization; and the proposed few-shot error bound provides a tight estimate of probing performance in practice.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a duality between contrastive learning (CL, self-supervised) and negatives-only supervised contrastive learning (NSCL, where samples belonging to the same class as the positive sample are removed from the denominator). \nFirst, it shows that the standard contrastive learning approximates the NSCL and that the gap between loss values of CL and NSCL decreases with the number of classes with $\\frac{1}{C-1}$ (for a balanced dataset).\nNext, it introduces a new bound on few-shot error of linear probing. Compared to previous work, this bound depends not only on class-distance-normalized variance (CDNV) between classes, but also on its *directinal* variant, i.e., on *directional* CDNV, which measures the variation along the line between centers of the respective classes. Moreover, the authors show that this newly introduced directional CDNV dominates the bound on few-shot error of linear probing.\nFinally, the findings presented in this paper are empirically validated on four datasets (CIFAR10, CIFAR100, mini-ImageNet, SVHN) using SimCLR or MoCov2 algorithm (with DCL loss).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is well written with a clear structure and story.\n- The findings reported in this paper are valuable in the steps towards a better theoretical understanding of contrastive learning.\n- Good visualizations and plots that support the findings.\n- The findings are well documented and commented on. The notation of the equations is clear. \n- The reported contributions are empirically validated on four datasets using either SimCLR or MoCov2 algorithm with a nice level of detail given about the experiments.\n\n**Weaknesses**\n- The closseness of the two loss values (for DCL and NSCL) does not directly imply the same behavior and structure of the learned representation. I am grateful that this is also commented on by the authors.\n- (minor) Sometimes, the generally-used variables (e.g., *K* on L220) are introduced earlier in the text and it can be a bit confusing.\n- Experiments on a larger dataset (e.g., ImageNet) are missing.\n- It is not explained what \"upper bound\" in Fig 2 (top) stands for?\n- Claims on L298-299 are not discussed. What can be the reason for the gap between the losses to increase?\n\nQUESTIONS:\n- How do you define the \"same point\" in the augmentation collapse on L66? Is it some small neighborhood?\n- What does \"upper bound\" in Fig. 2 (top) represent?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide theoretical foundations for self-supervised contrastive learning (CL) by investigating the gap between CL loss and a supervised negatives-only supervised contrastive loss (NSCL). The loss gap is proved to vanish as the number of classes increases. The paper also discusses that the NSCL-trained representations achieve accurate linear probing performance because the perfectly clustered representations lead to zero CDNV and directional CDNV. Experiments on benchmark image datasets are conducted to validate the theoretical findings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength.**\n\n1. The paper provides theoretical foundations for self-supervised contrastive learning from a new perspective.\n2. The theoretical claims are validated through experiments.\n\n**Weakness.**\n1. There is a mismatch between the main theorem and its proof. Theorem 1 states that the bound is for DCL, but according to its proof, the bound should be for CL. Also, the claims in Sections 3 and 4 are for DCL, whereas in the abstract, the gap is claimed to be between standard CL and NSCL losses.\n2. Several related theoretical works are not mentioned, e.g., https://openreview.net/forum?id=XDJwuEYHhme, https://openreview.net/forum?id=c5K7gcd2Rw, https://openreview.net/forum?id=cN9GkuBo4w. The authors should discuss the connections and differences compared with these papers.\n3. The concept of directional CDNV needs intuitive explanations. Some examples could help.\n4. The theoretical claims show that DCL approximates NSCL when the number of classes is large, and that NSCL leads to perfectly clustered representations and accordingly good few-shot classification performance. However, the theoretical discussions fail to address why DCL leads to good representations. Specifically, do the theoretical claims mean that when the number of classes $C \\to \\infty$, the DCL representations are perfectly clustered? This doesn't seem to be right under my knowledge.\n5. The experimental validation of diminishing loss gap is weak (Fig. 2). It only verifies $C=10$ and $C=100$. More results on Tinyimagenet (C=200) or Imagenet1k (C=1000) could further strengthen this claim.\n\nQUESTIONS:\n1. Figure 3 shows that the gap slightly increases over training. Is there any explanation?\n2. In Figure 5, the CDNV of DCL increases at the end of training. Is there any explanation?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the author aims to establish a connection between contrastive learning and supervised learning. Through theoretical analysis, the author demonstrates how self-supervised contrastive learning can be mapped to a supervised contrastive loss that uses only negative samples. Both the upper and lower bounds of the proposed model are provided. Experiments involving few-shot classification show that as the number of classes increases, the behavior of the two paradigms becomes increasingly similar.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is well written and easy to follow.\n\n- The two theorems proposed in the paper are mathematically well-formulated and rigorously proved.\n\n- The experimental design effectively validates the theoretical claims made by the authors.\n\nWeakness:\n- The experiment conducted by the author is quite small in scale, as CIFAR-10, CIFAR-100, and mini-ImageNet each contain only 50,000 images. The results could be easily influenced by overfitting, especially in the presence of noisy samples. I am curious whether this phenomenon still holds when the number of samples per class increases or when the data becomes noisier. Nowadays, pretraining typically requires massive and often noisy datasets. Further experiments should be conducted to verify the author’s claims.\n\n- The central claim that standard contrastive learning approximates a supervised variant under a growing number of classes is conceptually intuitive and has been suggested in prior works. While Theorem 1 formalizes this intuition with a clean bound, the assumption that the number of classes tends to infinity makes the result less surprising and arguably less practical. The paper ultimately confirms that contrastive objectives yield clustering behavior similar to supervised objectives, but this is already an implicit assumption in much of the literature. Thus, the theoretical contribution, though correct, feels incremental rather than insightful.\n\n- This paper does not provide practical guidance for designing CL systems. For example, it does not address important  questions such as how augmentation affects learning, the robustness of  the theoretical results under incomplete augmentation, or whether CL can outperform supervised learning in all cases. Although the authors  introduce an upper bound on the few-shot error, it relies on quantities that are difficult to explicitly control during learning (e.g.,  class-wise variance) and has limited practical impact on accuracy.  Moreover, minimizing loss does not directly translate into subsequent accuracy improvements, weakening the practical relevance of the results.\n\nQUESTIONS:\nPlease refer to the weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical and empirical bridge between self-supervised contrastive learning (CL) and a supervised variant called negatives-only supervised contrastive learning (NSCL). The authors show that as the number of classes increases, CL implicitly approximates NSCL, even without label supervision. They prove that NSCL leads to representations exhibiting neural collapse structure, and they propose a new bound for few-shot linear probing error based on feature variability. Experiments on image classification benchmarks support the theoretical claims, showing that CL-trained models, despite lacking labels, learn semantically meaningful and transferable representations.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a compelling and well-founded theoretical connection between unsupervised and supervised contrastive losses. The experimental results consistently support the theoretical claims.\nHowever, despite the clear structure and explanations, several questions remain—such as the performance gap between DCL and NSCL, comparisons with standard CL, and the degree of equivalence between the learned representations.\n\nQUESTIONS:\n1. The experiments show that the gap between DCL and NSCL decreases as the number of classes increases, which aligns well with the theoretical analysis. However, the performance difference between the two methods remains noticeable. More discussion on this observation could help clarify how much of the theoretical alignment translates into practical outcomes.\n2. In addition to DCL and NSCL, it would be helpful to include comparisons with standard CL methods. Based on the paper’s analysis, the gap between CL and NSCL is expected to be even larger, which could further support the theoretical claims.\n3. Can the authors quantify the similarity of the learned representations—not just the loss values—between CL/DCL and NSCL-trained models? For example, using clustering metrics and others.\n\nLIMITATIONS:\nWhile the paper presents clear analysis and supporting experiments, further evaluation on a wider range of models and larger-scale datasets would be helpful. This could clarify how well the findings generalize beyond the current experimental setting.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}