Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
Authors: Achleshwar Luthra
Tianbao Yang
Tomer Galanti
Venue: NeurIPS 2025 poster ()
Keywords: contrastive learning
self-supervised learning
neural collapse
Primary area: deep_learning
TL;DR: A theoretical bound showing that SSL approximates supervised contrastive learning as a function of #classes in the training dataset. Studied the connection between the representation geometry and their downstream transferability.

Abstract: Despite its empirical success, the theoretical foundations of self-supervised contrastive learning (CL) are not yet fully established. In this work, we address this gap by showing that standard CL objectives implicitly approximate a supervised variant we call the negatives-only supervised contrastive loss (NSCL), which excludes same-class contrasts. We prove that the gap between the CL and NSCL losses vanishes as the number of semantic classes increases, under a bound that is both label-agnostic and architecture-independent.

We characterize the geometric structure of the global minimizers of the NSCL loss: the learned representations exhibit augmentation collapse, within-class collapse, and class centers that form a simplex equiangular tight frame. We further introduce a new bound on the few-shot error of linear-probing. This bound depends on two measures of feature variability—within-class dispersion and variation along the line between class centers. We show that directional variation dominates the bound and that the within-class dispersion's effect diminishes as the number of labeled samples increases. These properties enable CL and NSCL-trained representations to support accurate few-shot label recovery using simple linear probes.

Finally, we empirically validate our theoretical findings: the gap between CL and NSCL losses decays at a rate of $\mathcal{O}(\frac{1}{\#\text{classes}})$; the two losses are highly correlated; minimizing the CL loss implicitly brings the NSCL loss close to the value achieved by direct minimization; and the proposed few-shot error bound provides a tight estimate of probing performance in practice.

Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning
Achleshwar Luthra
Tianbao Yang
Tomer Galanti
(luthra, tianbao-yang, galanti]@tamm edu Department of Computer Science and Engineering Texas A&M University
Abstract
Despite its empirical success, the theoretical foundations of self-supervised con- trastive learning (CL) are not yet fully established. In this work, we address this gap by showing that standard CL objectives implicitly approximate supervised variant we call the negatives-only supervised contrastive loss (NSCL), which excludes same-class contrasts. We prove that the gap between the CL and NSCL losses vanishes as the number of semantic classes increases, under bound that is both label-agnostic and aechitecture-indepnniit
We characterize the geometric structure of the global minimizers of the NSCL loss: the learned representations exhibit augmentation collapse, within-class collapse, and class centers that form simplex equiangular tight frame. We further introduce a new bound on the few-shot error of linear-probing. This bound depends on two measures of feature variability-wihlee within-class dispersion and variation along the line between class centers. We show that directional variation dominates the bound and that the within-class dispersion's ''s effect diminishes as the number of labeled samples increases. These properties enable CL and NSCL-trained representations support accurate few-shot label recovery using simple linear probes.
Finally, we empirically validate our theoretical findings: the gap between CL and NSCL. losses decays at rate of O #clusses ); the two losses are highly correlated; minimizing the CL loss implicitly brings the NSCL loss close to the value achieved by direct minimization; and the proposed few-shot error bound provides a tight estimate of probing performance in practice. The code and project page of the paper are available at [code, project page].
Introduction
Unsupervised representation learning refers class of algorithmic approaches designed to discover meaningful representations of complex data without relying on explicit supervision signals. The goal is to learn representations that preserve and expose semantic information, allowing them be effectively leveraged in downstream supervised tasks. In recent years, these methods have proven to be effective in pre-training models on unlabeled data, enabling strong generalization on downstream
computer vision tasks [ 2,
3
5
6
7
8
9 10
12 13
14
15
16 and in natural language
processing 18
20
21
One of the most successful paradigms for unsupervised learning is self-supervised contrastive learning (CL), where models are trained distinguish different augmented views of the same image (positives) from views of other images (negatives), typically by minimizing the InfoNCE loss [26] 2 Notable methods in this category include SimCLR (5), MoCo (27 6 77) and CPC These approaches have
led to representations that generalize remarkably well to downstream tasks, often competing with or
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Initialization
Epoch 10
Epoch 100
Epach 500
Epoch 1000
Epoch 2000
Figure 1: DCL. forms semantic clusters without label supervision, while NSCI. yields tighter, more separable clusters, despite not explicitly pulling same-class samples together. We plot UMAP visualizations for (top) decoupled contrastive learning (DCL) [25] and (bottom) negatives-only supervised contrastive learning (NSCL) training on mini-ImageNet. See Appendix A for details.
surpassing supervised learning. For example, ImageNet- 1M pre-trained MoCo achieves 81.5 50 on PASCAL VOC (28), while its supervised counterpart achieves 81.3 AP50-
Although the representations learned in supervised classification problems are relatively well understood-thame to characterizations such as neural collapse [29 30), which occurs under a variety of loss functions including cross- cross-entropy [3] 32 33], mean squared error (MSE) [30 34 35), and supervised contrastive loss [33 36 37 (8) the theoretical understanding of SSL remains limited. Surprisingly, despite the absence of labels, SSL-trained models often produce representations that closely resemble those learned via supervised training: they tend to cluster nicely with respect to se- (see also Fig. T1 and support downstream tasks such as linear classification
mantic classes (39 40 41
and clustering
5 27
2 This raises the following question:
How does self-supervised contrastive learning learn representations similar to those learned through supervised learning, despite the lack of explicit supervision?
Contributions. We provide theoretical framework that connects CL to it supervised counterpart. Our key contributions are:
Duality between self-supervised and supervised CL: In Thm. we formally establish con- nection between decoupled contrastive learning (DCL) 251 and supervised variant call negatives-only supervised contrastive loss (NSCL), in which negative pairs are drawn from dif- ferent classes. Our main insight is that, although DCL does explicitly exclude same-class samples from the loss denominator, the probability of treating them as negatives vanishes as the number of classes increases. We prove that the gap between the two losses shrinks with more classes. Unlike prior analyses [2 43 36 45). our result holds without assumptions on model architecture, data distribution, correlated, their gap decreases as the number of classes increases, and minimizing the DCL loss implicitly minimizes the NSCL loss.
augmentations. Figs. 3 show that the two losses are highly
Estimating the few-shot error downstream tasks: Prop. introduuess bound on the m--sho classification error based on geometric properties of the learned representation. Specifically, the bound implies that lower directional CDNV (see Sec. (.2) lead to lower m-shot error, with the impact of the regular CDNV diminishing as the number m of per-class labeled examples increases. Empirically,
variiance (CDNV) (see (46) and Sec. 3.2) and Sec.3.2)
verify this behavior for DCL-trained models (Fig. 7 and find the bound to be tight in practice (Fig 6).
Characterizing the global minima the NSCL loss: In Thm. 2 we show that any global minimizer of the NSCL loss satisfies: (i) Augmentation collapse all augmented views of sample map to same point; (ii) Wahin-class collapse-aa all samples from the same class share representation; and (iii Simplex ETF stracture- class centers form maximally separated, symmetric configuration. These solutions coincide with the optimal solutions certain supervised classification settings, including those using cross-entropy loss [3] 32 33 mean squared error

---

(MSE) [30 35|, and supervised contrastive loss [33. Combined with Prop. [ these properties
imply that label recovery via linear probing guaranteed with few labels.
2 Related Work
Theoretical analyses of CL. A growing body of work aims to explain the effectiveness of CL. Early studies attributed CL's success to maximizing mutual information between augmentations of the same image 3 but later work showed that enforcing strict bounds on mutual information can hurt Another influential line of work focuses on the alignment and uniformity of learned representations [49, [50] [5]] For example, [49] showed that same- sample augmentations are aligned, while negatives are uniformly distributed on sphere. [5T] extended this wider family of losses, showing how alignment and uniformity can be modulated by hyperparameter.
performance [47 48
While these important properties, they do not characterize how CL algorithms organize samples from different classes in the embedding space. As a result, several papers [42] 44 [52 [53] [54] [55] [56]
57
studied CL's ability recover meaningful clusters and latent variables the data, However, these results often rely on strong assumptions about the data and augmentations, such as assuming augmentations of the same sample are conditionally independent given their cluster identity (see, e.g., 42 43
58
36
59
36]). In order to avoid such assumptions, [60] studies function classes that induce similar bias towards preserving clusters without any assumption on the connectedness of augmentation sets. However, their analysis is focused on minimizing a spectral contrastive loss which serves as a surrogate for the more practically used InfoNCE loss. [45) instead proves that InfoNCE learns cluster-presd juuster-rrserrvng embeddings by capping the representation class's 's capacity so that any function that tries to carve semantic cluster must also split an image from its own augmentation set, making cluster integrity the loss mossmminii choice.
Another way to understand SSL is by examining its connection supervised learning. For example, [6]] showed that, in linear models, certain SSL objectives resembling VicReg are equivalent to supervised quadratic losses. Building on this perspective, we establish bound between the contrastive loss and supervised variant of it. Unlike previous work, our bound is both architecture-ing inneeeeeter--ddeeeeeeennttt and label-agnostic.
Other theoretical studies have analyzed SSL from different angles, such as feature learning dynamics in linear models and two-layer networks 622
64 65]. the role of augmentations [12 66), the
projection head [67] 68 69 70], CL's sample complexity [71 and how to relax the dependence on
large mini-batches [72]. Other work explores connections between self-supervised contrastive and
non-contrastive learning 73
74
75
76
77
Neural collapse. Neural collapse (NC) [29 30 is phenomenon that occurs in the final stages in training of overparameterized networks for classification. In this regime, we observe: (i) within- collapse, where the embeddings of samples within each class converge
single vector, (ii) class separation, where these class means spread out and often form a Simplex Equiangular Tight Frame (ETF); and a form of (iii) alignment between the feature space and the classifier's weights.
To understand the emergence of neural collapse, several papers studied the emergence of these properties in different learning settings. In 78 79 they introduced the "unconstrained features model" or "layer peeled model"- each sample's 's embedding is treated as a free parameter in Rd. Under such relaxations, research has demonstrated that neural collapse is characteristic of
of global
minimizers for many commonly used supervised loss functions [80), including cross-entropy [3T
32 [33, mean squared error (MSE) [30 34 35). and supervised contrastive loss [33 36 37 38 In
Thm. 2 we use a similar approach in order to characterize the global minima of the CL and NSCL
losses, connecting them to downstream performance using tools from 46 81
3 Theoretical Analysis
In this section, we explore why CL-ddspptt being agnostic to class labels- to organize data by class and support few-shot transfer. We first relate the self-supervised loss to a label-aware loss function, then describe the geometric conditions under which representation supports few-shot transfer, and finally show how this loss function induces these properties.
3

---

Setup. We consider training embedding function f E FFCC {f f: X supervised contrastive learning, using a dataset S [(. X [C], where the algorithm only sees inputs E X' (e.g., images), but not the corresponding class labels Vi E [C]. We assume that each class contains at most M'max samples. We wish to learn a "meaningful" (see Sec. 3.2) function f X' + Rd that maps samples to d-dimensional embedding vectors.
f: - Rd) via self-
Concretely, for each sample Ii, we also construct K augmented versions x (x;) (via data augmentations oq...., a, e.g. identity, random cropping, jitter, etc.). Then, we define 2/ ((()( as the embeddings of the augmented samples.
We focus on the global decoupled contrastive loss (DCL) 22 which is given by
(DCL(f) = K 11.2==1
1
Σ
Σ log Eis=1 ESENNYO) exp(sim(2"), =2-1)
exp(sim(=1), 2(2))
(I)
where sim(.,) : IRd x Rd + R is the cosine similarity function, which is defined as follows:
sim(x1,
| (x1 .22)
We focus on the DCL loss since its a slight improvement to the original
lan lla"
global CL loss 2
[
227
72 where Σÿ= is replaced with E the denominator. While here
we focus on the global 72 49,,
for completeness, in Appendix B.I extend Thm. T to mini-batch CL losses as done in practice.
Despite the lack of label supervision, contrastive models often learn class-aware features. This is surprising: Why should label-agnostic loss promote semantic structure?
3.1 Self-Supervised vs. Supervised Contrastive Learning
To answer this, we compare DCL with a supervised contrastive loss that removes same-class contrasts. Namely, we consider the negatives-only supervised contrastive loss (NSCL), CNSCL(()), (f), which is defined the same as DCL in I but with [jE[M)(A) in the denominator replaced by Σjustee
When comparing the two losses, the unsupervised denominator includes at most extra K [nmmx 1) terms corresponding to data from the same class. Therefore, if maxx small relative to the number of classes C' (i.e., when C' is large), the extra K (T 1) terms are expected to be negligible compared to the total number of negatives 2 K(N mmmax), and the two objectives, CDCL( (f) and LNSCL (f). become nearly equivalent. For instance, assume for simplicity that exp (sim (zi, 2j)) a y for all * j (i.e., the similarity values are approximately constant). Then the unsupervised denominator is approximately K K(N - 1)> while the supervised denominator (excluding same-class data) becomes approximately (NN- - 7 ), Overall, we obtain that [CDCL()) CNSCL (f) Rd
log 1
Panaxe
Tmmax
Hence, the gap between the two losses shrinks as grows
N Piana
Tl Plaaa
In essence, this example shows that if exp(sim(z,), =/+)) is nearly constant, removing same-class negatives has little effect on the loss. The following theorem shows that one achieve slightly worse bound on CCDCL. (f) CNSCL((f) without assuming that exp(sim((e) 2/ 2/2)) is constant.
Theorem 1. Let S = (1 y C X x [C] be a labeled dataset with C' classes, each containing al most max distinct samples. Let ʃ X RRd be any function. Then, we have
CNSC ()) & DDDLL) < L (SSC + log (1
Tmmae N Tomax
S NSCL
(
+
Bmaace N N-Tmaxx
where denotes Euler's constant. For balanced classification problem,
Pamax
The above theorem gives a justification for why contrastive SSL yields class-aware features without ever seeing the labels. Interestingly, since the right-hand hand side depends only on the number of samples N and the maximal number of samples per class nm -not not on the specific assignment Y -he inequality is label-agnostic. Namely, for any labeling Y with Remax samples per class, we have,
0 < CCL (f) CN (f) < log(1 +
(2)
N-
where CNSCL (f) is the NSCL loss under that labeling. In particular, C +
+ oo the term log(1
-imaxx vanishes, the probability of drawing same-class contrasting drops to zero, and DCL effectively collapses onto the NSCL for all labelings simultaneously.

---

3.2 Characterizing Good Representations for Few-Shot Learning
The primary goal of SSL is to learn representations that are easily adaptable to down- stream tasks. We now describe the conditions under which an SSL-trained representation can be adapted to specific downstream task. For this purpose. we consider the re- coverability of labels from learned representation f: X + IRd. We call ʃ "good" if supports accurate downstream classification with only a few labeled samples per class.
We formalize this via the expected m- shot classification error. Suppose we have set of class-conditional distri- butions .......D'' and define D of Ee, De as their uniform mixture, For example, Di,..., Do can represent sub- of C" C' classes from the pre- training task, where all D; are either train- ing datasets or data distributions corre- sponding to their respective classes. Given an m-shot training set S; D for each class i, we define: (ff == Es (err)(h,,3)], where h is a classifier trained using only the small la- beled support $ = u See and ( is
Adai
(a) CDNV
(b) dir-CDNV
Illustration of CDNV and directional CDNV. CDNV compares how tightly samples within a class cluster to how far apart class centers are; lower values indicate tighter clusters and larger gaps between classes. The latter measures variability only along the line connecting two class centers, highlighting the com- ponent most relevant for distinguishing those classes.
Figure 2:
the probability that h misclassifies ran-
dom test point drawn from the task distribution D. Specifically, we denote linear probing (LP) and nearest-class-center classification (NCCC) errors by enrthn (m) (J) and err))(()) which refer to e. (f) with y,3 being linear classifier that minimizes e and the nearest-clas--center
classifier arg [C [= (S.
Prior work 46 g1 shows that this error can be bounded in terms of how clustered the represen-
tations are. Specifically, for each class i, let μ4; Ez~D, [f (z)] and of EnnD, [l]((z) - [قll
denote the mean and variance of the class embeddings, and let Vy D2) = of /l/m= 143 !1? be the
llas-distance-nnrmalized variance [46 81 82
between classes i and j.
These works (see, e.g., Prop. 7 in showed that
LP (f) enrm (J) & (C'- 1)(1 A (D, D2)]-
NCC
(3)
In supervised learning, representations tend become tightly clustered, causing CDNV to decrease, and thus reducing few-shot error 22 46 83
However, in the self-supervised case, labels are not available during training. As result, there is no mechanism to explicitly encourage intra-class similarity, and thus we cannot generally expect that
V Avg [V, (D,,,, D,)] will be small.
To better capture settings where label information is implicit, we introduce a refinement of (3) that prioritizes directional variability (see Fig. 2 for visual illustration). For each pair i,j, let:
oi = Varz~D, [([(x) Múi, wir)), where unj -
Then the directional CDNV, V,,
Avg (D,, Dj)], is defined as: (D,,, Dý) o2//1le m3llž that measures the variation along the line between the class centers. This leads to the following stronger bound:
Proposition 1. Lei C' 2 2 and m 2 10 be integers. Fix a feature map f : X + Rdd and
class-conditional distributions Dı,..., De over X. Let Vj = Avg. IV,, (D,, D2)]. We have:
eTTTD S NCG (f) S (C' -0'-1) (8V,
8 V7 + mm
V, + AV 8 vm
(4)
Unlike 33. which depends the full CDNV. Prop. I depends primarily on the directional CDNV, (and on - Vy), which is always smaller than the CDNV (V, Vy) and in isotropic distributions fo a D, it even scales as V, Vy. Thus, the new bound can explain low m-shot error even when the regular CDNV is large. In essence, the bound predicts that SSL-trained models are effective
5

---

for downstream tasks when their directional CDNV is very low and their CDNV remains moderate.
Fig. 7 shows that this prediction holds in practice.
Finally, we note that the coefficients (14 and 28.5) in the bound are sub-optimal. In (33) (Appendix
give more general form of the coefficients, parameterized by a variable In Cor. (Appendix C
we give an optimized version of the bound using a near-optimal choice of a. In Fig. bottom), we
show that the bound in Cor. is fairly tight in practice.
3.3 Characterizing the representations learned with CNSCL
Prop. essabbisshs that if the directional CDNV is small and the overall CDNV remains bounded, then the linear probing error will likewise be small. Since Thm. T the DCL loss also drives down the NSCL loss, we use NSCL as stand-in for DCL. Accordingly, in this section we analyze representations learned under the NSCL objective and evaluate how well they recover class labels through linear probing.
guarantees that minimizing
Because minimizing over all neural networks f intractable, we follow prior work 78 79 and adopt the unconstrained features model. Specifically, we treat f as an arbitrary function that given an augmented sample z selects an embedding 2 as a arre learnable vector in F. The following theorem characterizes the glohal minimizers of the NSCL. loss, showing that they satisfy the NC1, NC2, and NC4 properties of NC (29, 30), and thus learn representations similar those of minimizers of other classification losses, such as cross-entrope [31 32
33). MSE
30
34 35), and SCL. (33)
Theorem 2. Let d 2 C 1 and let S = {( 1:) X [C] be balanced labeled dataset with C classes. Suppose f is a global minimizer of the supervised contrastive loss CNSCL (f) (over all functions f X + )d). Then, the representations satisfy the following properties:
1. Augmentation Collapse: For each i E [N] and for every pair 11,2 E [K] we huve 2 2/2
2. Within-Class Collapse: For any two samples I. and x; with the same label (y: Uÿ). their representations coincide: 2, = 2z. Namely, each class has a unique class embedding.
3. Simplex Equiangular Tight Frame: Let {Mi...., Hc} denote the set of class-center embeddings.
These vectors form a simplex ETF in Rd;; specifically, they satisfy E He =0, ||Mell2 = ||
11u-l122 and (pe: (pe:He) = =- C-1 for all c + d' E [C].
Thm. 2 implies that any global minimizer of ( yields perfectly clustered representations. As such, the CDNV and the directional CDNV are zero and by Prop. the 1-shot errors vanish: err (f) S err (f) 0. In CL on the other hand, cannot achieve such collapse, since it would imply that the representations encode only label information-despea agnostic. Still, the DCL-NSCL duality invites a more nuanced question: Does minimizing CL still promote weak forms of clustering? While zero CDNV is tied to a specific labeling, the few-shot error m.S (f) and the directional CDNV can be small under many possible labelings. In Sec. we empirically explore the extent to which minimizing the DCL loss induces such structure.
the loss being label-
Experiments
4.1 Experimental Setup
Datasets. We experiment with the following datasets CIFARIO and CIFAR100
85 mini-
ImageNet 86), Tmny-Ima [87] SVHN [88 and ImageNet- 89
Methods, architectures and optimizers. We trained our models with the SimCLR rithm. We use ResNet-50 [90] encoder with width-multiplier factor of 2 and ViT-Base [91]. Both are followed by a projection head with a standard two-layer MLP architecture composed of: Linear (2048 2048) - ReLU Linear (2048 + 128). For additional experiments and setup details about Vision Transformers 91 and MoCo v2 [6, see Appendix[A] A
5 algo-
Instead of training our models with the standard InfoNCE loss that is used in SimCLR, we use the DCL loss that avoids positive-negative coupling during training 255. In order to minimize the loss,
we adopt the LARS optimizer [92] which has been shown in [5] to be effective for training with large

---

Controt ve 105
5.75
upper 20083
Contregt ve to4
RVG m95
Uppe louad
Contreet ue toke
7.40
U230 board
DNC1 bix
Contoott logo
7.40
2.20
Usoer bound
100
10
7.75
0.0
80
6.7-
6.60
307
30
sprets Hop
10
Thinkin aocems
12
10
10"
Taiinng spocns scale)
30
12
17
2aniing
10
12
30'
0.20
DC,
DCL
DC.
modd
7.40
rada
radd
Con Conrecttee 005
7.20
Contrrtion pane
Corgastto 600
1.0
1.00
100
6.00
100
20
6.00-
and Jusk 2n0
aastiand 1
Triniig spects lop scal))
Thaining azechs dog scate)
Taaing specna fog scalel
Taning spocns Dog scalet
(a) SVHN
(b) CIFARIO
(c) CIFAR100
(d) mini- mini--mageNNer
Figure 3: (Top) We train the model minimize the DCL loss, tracking the DCL loss, the NSCL loss, and the bound NSCL+log(1 + N- on both the training and test sets throughout training. All three quantities are highly correlated. (Bottom) We compare the NSCL loss of two models: one trained with the DCL loss and the other with the NSCL loss. The resulting NSCL losses are comparable, regardless of the training objective. In both the top and bottom plots, correlations are computed between the DCL and NSCL losses on the train and test data.
Timaxe"
batch sizes. For LARS, we set the momentum to 0.9 and the weight decay to 1e-6. All experiments are carried out with a batch size of 1024. The base learning rate is scaled with batch size as 0.3 | B/256]] following standard practice [5]. We employ warm-up phase (93] for the first 10 epochs, followed by a cosine learning rate schedule without restarts [94] for the remaining epochs. All models were trained on a single node with two 94 GB NVIDIA H100 GPUs.
Evaluation metries. In several experiments we monitor CDC (see IT) and CNSCL (see 3.1). To calculate each one of these loss functions, we replace the sum over samples and their K augmentations in the denominator with a sum over a random batch of B 1024 random samples and one random augmentation for each sample as in SimCLR (see Appendix B.I for details).
To evaluate the quality of learned representations, we Classification (NCCC) accuracy 833, and linear probing accuracy [95] 96 97). Suppose we have a feature map f, classification task with C classes, and dataset S _ St (either training or test data). We estimate (ff and (CC with h y.3 being an NCC classifier a linear classifier. To train the linear classifier, we use cross-entropy minimization for 500 epochs, with batch size min( (1, 256), learning rate -4, weight decay 5e-4, and momentum 0.9. The expectation over the selection of S , is estimated by averaging the error over selections of S from S.
methods: the Nearest Class-Center
We also estimate the bound in Prop. As mentioned above, the coefficients in ( are sub-optimal, so we instead use the refined bound from Cor. [ (see Appendix (). To assess its ability to predict downstream performance, we compute the bound on both the tramnnn and test data by treating the per-class train/test subsets S; as the distributions D,, The CDNV and the directional CDNV are calculated exactly as described in Sec. 3.2
4.2 Experimental Results
Validating Thm. I during training. We train models using SimCLR minimize the DCL loss. We evaluate both the DCL and NSCL losses on both training and test sets. Additionally, we evaluate the proposed upper bound, given by CNSCL()) log(1 "maxt" ), where N is the total number of samples and Rmma is the maximal number of samples per class (see Thm. [].
In Fig. ktopp we observe that the DCL loss consistently upper bounds the NSCL loss, and that the two losses become closer for tasks with a larger number of classes (e.g., Cl- FARTOO and mini-ImageNNe compared to CIFARTO and SVHN). In addition, when C is large (e.g., C 100), the bound becomes very tight, closely matching the losses. Al-
7

---

Tan
Cascha
Eparh
Coech-
tarchit
tpodh
Eaech
(yyaa Dog deb (oor
1.0
Looe gap Blog scale)
1,0
1.0
Loss gap flog
A.P maR. C0R- 8.00
AP (Faas
4 trak) Gol,
0.1
0.0
0.0
15
35.0
30
63
105
2530
40
10
100
Number of classes (C)
Number classes (C)
Number of classes (C)
(a) CIFARIO
(b) CIFARIOO
(c) maii--mageeNe
Figure 4: The gap between the DCL and NSCL losses shrinks as the number of classes C' grows. Models were trained to minimize the DCL loss, and at several training epochs we plot the empirical
difference _DCL CSCCL alongside the bound log(1 + (-1) function of C. We also report correlation between the loss gap at epoch 300 and the bound.
c²
though we notice a slight divergence between the self-supervised and supervised loss metrics as training progresses, the overall trend continues to align well with the inequality stated in Thm. For similar experiments with SimCLR-VED VIT and MoCo-v2, see Fig. and Fig. 10
While this result shows that for large C, the NSCL loss decreases alongside the DCL loss, it does not imply that minimizing the DCL loss leads to the same solu- tion as directly minimizing the NSCL loss. To investigate this, we conducted an experiment in which we trained two models to minimize the DCL and NSCL losses (respectively) and compared their resulting NSCL values. As shown in Fig. [botomm) the gap between the NSCL losses of the two models is fairly small. This indicates that, in practice, optimizing the DCL loss leads to representations that are fairly clustered in comparison with those obtained by explicitly optimizing the NSCL loss.
Dataset
CIFARIO
CIFARI00
mini ImageNet
NCCC
NCCC
NCCC
DCL NSCL
85.3 + 26
86.3
Se
57.3 le"
61.7 = 5e
69.0 2e 79.8
72,9 1 2e
95.7 + 46
95.6
70.8 + le-
73.7 = 20
1e-1
81.3 + 26
Table 1: NCCC and LP 100-shot test-time accuracy rates (and their standard deviations) of DCL and NSCL-trained models.
Validating Thm. with varying C. In the next experiment, we analyze how the gap CDCT( (f) CNSCL(J) scales with the number classes C. Specifically, we empirically validate that this gap is upper-bounded by log(1 + Amaxe? large C, and that it is highly correlated with the actual gap between the losses.
N-Rimax log(1 1 + C-1 ) (see Thm. I that the gap becomes tighter for
For each value of C, randomly sample C classes from the full dataset, train self-supervised model from scratch on data from these classes only, and compute CDCL(S) - CNSCL( (J') at various training epochs. To account for variability in class selection, we report the averaged value of this quantity over five independent random selections of C' classes along with error bars. Fig. [rreeens the empirical results, showing that the gap CDCL(F) CNSCL(() (f) slightly increases over the course of training.
As shown in (5), the loss gap is given by log (1+ Dyar,gyyy esp(ein(sz.25)) Ex,tn, exp(ein(2..2p))
At initialization, the
embeddings 2 are randomly distributed in the representation space. During training, the emboddings cluster with respect to their classes thereby increasing sim(2,, 2;) for y; y: while reducing it for yo * yi- This leads to gradual increase in the loss gap as observed in Fig. 4 but it remains
consistently bounded by log(1 + e all epochs. Moreover, the magnitude of this gap decreases with C and is highly correlated with our bound.
Comparing downstream performance. To evaluate the quality of the learned representations, we measure the few-shot downstream performance of models trained with DCL and NSCL. Specifically, we report the NCCC error and linear probing error (see Sec. 4. lp. Fig. performance across all classes as function of the number of shots per class (mn). As can be seen, although the NCCC and linear probing errors of the DCL-trained models are, as expected, higher than those of the NSCL-traine model, they remain fairly low (see Table [for numeric comparison). This
F (top) shows train and test
8

---

indicates that, despite not being explicitly optimized to align with class labels, the representations learned by DCL still exhibit strong clustering behavior.
Fig. 6 (bottom) compares the bound from Cor. [ with the NCCC and linear-probe errors of
DCI-trained models on 2-way downstream tasks (averaged over 10 tasks), for both train and test data. Unlike Prop. 7 in [82). and despite the constants in our bound, it is fairly tight as m increases; for instance, m7 500 for CI indicates that the few-shot errors are below 0.89, At 10%,
the test-time bound (red horizontal line) predicts an error of at most 0.26.
To complement experiments, take a publicly available
-e model pretrained on IM-1K and perform 2- 2-way m-shot NCCC evaluation similar to Fig. 6 (bottom). In Fig. 5 we verify our error bound with pretrained weights mini -mageeet and IM-IK.
Fig- shows that DCL training only
203.20
108.00
Figure 5: The bound in Cor. [ is fairly tight for ImageNet pre-trained models. We reproduced Fig. 6 with ResNet- pre- trained on IM-1K.
fourd
und bound
modestly lowers
taroe maaia
Groee main
CDNV but reduces directional CDNV
2.55
by about an order
of magnitude during training. Because
Numborr ahott
Prop. liiks few-shot
(a) mini-mmanchie ImageNet
(b) IM-1K
error primarily the directional CDNV, this sharp drop explains why DCL-trained models can recover labels (see Fig. (6 with limited supervision. contrast, NSCL reduces both variances substantially, which accounts for its lower m-shot error.
Augmentation collapse. We verified augmentation collapse by training DCL ViT models for 2k epochs and measuring cosine similarity (0 indicates collapse: indicates perfect collapse). For each dataset, we computed the mean cosine similarity between augmentations of the same sample and between augmentations of different samples, using five augmentations per pair across the full dataset.
As shown in Tab. same- sample augmentations achieve mean cosine similarity of approximately 0.90, whereas different-sample pairs remain be- low 0.15. This consistent gap across CIFAR- 10/100 and mini- ImageNet rules out trivial global collapse and indicates that DCL promotes strong augmentation invariance while preserving inter-sample separability.
Dataset
CIFAR10
CIFAR100
mini-ImageNet diff
same
diff
same
diff
same
DCL
0.92
0.13
0.91 0.11
0.89
0.07
Table Cosine similarity between embeddings of two augmen- tations of the same sample ("positives") versus different sam- ples ("negatives")) for ViT trained with DCL. Positives exhibit. markedly higher similarity than negatives.
Representation Alignment. Thm. Ishows that the DCL. and NSCL objectives have similar values, but this does not guarantee that the embeddings learned by minimizing the two objectives will have identical geometry. To better understand the structure of representations learned by models trained with DCL and NSCL, we run experiments that quantify representation similarity using Centered Kernel Alignment (CKA) (98] and Representation Similarity Analysis (RSA) 199] When embeddings are highly aligned, RSA and CKA are close to 1; with no alignment, they approach
We train DCL and NSCL models with the exact same initialization, same mini-batches, and same hy- perparameter configurations as de- scribed in Sec. 4. for 300 epochs.
Encoder
CIFARI0 CKA
CIFARI00 CKA
mini ImageNt RSA
RSA
RSA
CKA
ResNet-50 ViT-Base 0.86
0.83
0.81
0.91
0.91
0.87
0.90
0.82
0.91
0.91
0.89
0.89
As shown in Tab. 3 the models achieve very high RSA and CKA values that confirm alignment at the representation level.
Table 3: DCL and NSCL models trained for 300 epochs with matched initialization and minibatch order show very high representation similarity.

---

Thsn
MSCL LP
USC.
Trsh Tett
MGCL
MSCL NCCC
15C. NCCC
1SCL NCOC
DCL
DEL
DEL
DCL MCOC
DCL NOCC
DCL NOCC
0.7
0.2
50
10
200
150
Namber of shots
Number of shots
Number of shots
100.00
100.00
100.00
Tran
DCL NCCC
Thah
DCL NCOC
7an
DCL
DCL
DCL.
10.00
10.00
Ereo 100 scale
(log
1.20
0.50
Error (tog
0.30
0.10
0.10
0.01
MFT: C30 6.00
P.P/ L.0
10 20
150
300
1000
0.01
30 150 200 505
0.01
20
200
500
(a) CIFARIO
(b) CIFARIO0
(c) mini Minii-ImaeeNet
Figure 6: (Top) We compare the m- error (Linear Probing (LP) and Nearest Class-Centere Classifier (NCCC)) for C-way (all-class) classification. Although the NCCC and LP errors of the DCL-trained models are, as expected, higher than those of the NSCL-trained model, they remain fairly low. (Bottom) We compare the two-way m-shot NCCC and LP errors with the bound in Cor. [[] The errors are bounded by our bound, which decreases with m. The dashed red 'Lim bound' line specifies the bound m 10, providing a tight test-time error estimate for large m. We also report correlation between the error bound and the errors err mmD(f) (f) and NCC (f) (on the test data).
10.0
Normalized variance
Normalized variance
Normalized
1.0
0.1
DCL (CONV)
DCL
CDNV
0.1
DCL (CDNV)
(Dir CONV
0.1
DCL (CDM)
CDNV
Train Test
(CDMV)
Thain
CDNV)
NSCL [Dir CONV)
CONVO
0.0
10'
10
10
10
0.0
10"
10
10
10
0.0
10"
10'
10"
10
Training epochs (log scale)
Training epochs (lo scale)
Training epachs (log scale)
(a) CIFARIO
(b) CIFAR100
(c) mini maae-et
Figure DCL training yields a moderate reduction in CDNV and a significant reduction in directional CDNV, whereas NSCL training significantly decreases both. We plot the CDNV and directional CDNV for both train and test data for DCL and NSCLetrained models over 2k epochs.
5 Discussion, Limitations, and Future Work
CL is at the forefront of modern pre-trae pre-training techniques. Our work takes several steps toward a better understanding of CL by: establishing a duality between CL. and NSCL,, analyzing the minimizers of the NSCL. loss, and linking the downstream emor of CL-trained models to directional CDNV.
Nevertheless, our work has several important limitations that remain open for future investigation. While we show that the DCL and NSCL losses are close in value, this does not directly imply that their respective minimizers are close in parameter space or yield similar representations. It would be interesting to explore this connection both theoretically and empirically-for example, by monitoring the distance between two models trained using DCL and NSCL on the same batches. Another limitation is that the bound in Prop. [[and Cor. I involves large constants and scales linearly with C". which may limit its practical utility. Tightening this bound by relaxing these dependencies would enhance its predictive power. Finally, extending these ideas to multimodal models such CLIP [100, or to LLMS trained with analogous loss functions, presents an exciting direction for future research.
10

---

References
[1] R Devon Hjelm, Alex Fedorov, Samuel aavoie-Marccildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
[2] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.
[3] Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[4] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 99 12-9924. Curran Associates, Inc., 2020.
[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations, In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020.
[6] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, 2020.
[7] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training sulf-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9640-9649, October 2021.
[8] Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding, 2020.
[9] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations, 2021.
[10] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations, 2019.
[11] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI, page 776-794, Berlin, Heidelberg, 2020. Springer-Verlag.
[12] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6827-6839 Curran Associates, Inc., 2020.
[13] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via via redundancy reduction. In International conference on machine learning. pages 12310-13 12320. PMLR, 2021.
[14] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-ing Variance-invarianee eewariance-covarianee regular- ization for self-supervised learning. In International Conference on Learning Representations, 2022.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
11

---

[16] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbar, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024.
[17] Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cert: Contrastive self-supervised learning for language understanding, 2020.
[18] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[19] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Farie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 202/ Conference on Empirical Methods in Natu- ral Language Processing, pages 6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[20] Yixuan Su, Fangyu Liu, Zaiqiao Meng, Tian Lan, Lei Shu, Ehsan Shareghi, and Nigel Collier. TaCL: Improving BERT pre-training with token-aware contrastive learning. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Findings of the Association for Computational Linguistics: NAACL 2022, pages 2497-2507, Seattle, United States, July 2022. Association for Computational Linguistics.
[21] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors. Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 3982-3992, Hong Kong, China, November 2019, Association for Computational Linguistics.
[22] Dejiao Zhang. Shang- Wen Li, Wei Xiao, Henghui Zhu, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. Pairwise supervised contrastive learning of sentence representations. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5786-5798, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[23] John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 879-895, Linguistics.
895, Online, August 2021. Association for Computational
[24] Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. ConSERT: A contrastive framework for self-supervised sentence representation transfer. In Chengqing Zong. Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5065-5075, Online, August 2021. Association for Computational Linguistics.
[25] Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI, page 668-684, Berlin, Heidelberg, 2022. Springer-Veriie
12

---

[26] Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Stafistics, volume 9 of Proceedings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
[27] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[28] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. Int. Comput. Vis., 88(2):303- 338, 2010.
[29] Vardan Papyan, X. Y. Han, and David L. Donoho, Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, (T(44)):2465-24466)) 2020.
[30] X.Y. Han, Vardan Papyan, and David L. Donoho. Neural collapse under MSE loss: Proximity and dynamics on the central path. In International Conference on Learning Representations, 2022.
[31] Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59:224-241, 2022. Special Issue on Harmonic Analysis and Machine Learning.
[32] Hien Dang, Tho Tran, Tan Nguyen, and Nhat Ho. Neural collapse for cross-entropy class- imbalanced learning with unconstrained relu features model. In Proceedings of the 41st International Conference on Machine Learning, 24. JMLR.org, 2024.
[33] Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting supervised contrastive learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 3821-3830. PMLR, 18-24 Jul 2021.
[34] Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse, 2022.
[35] Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features, 2022.
[36] Pranjal Awasthi, Nishanth Dikkala, and Pritish Kamath. Do more negative samples necessarily hurt in contrastive learning? In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1 1101-1116. PMIR, 17-23 Jul 2022.
[37] Jaidev Gill, Vala Vakilian, and Christos Thrampoulidis. Engineering the neural collapse geometry of supervised-cootrastiie loss, 2023.
[38] Ganesh Ramachandra Kini, Vala Vakilian, Tina Behnia, Jaidev Gill, and Christos Thram- poulidis. Symmetric neural-collapse representations with supervised contrastive loss: The impact of reLU and batching. In The Twelfth International Conference Learning Represen- tations, 2024.
[39] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. ECCVW What Motion For?, 2022.
[40] Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse engineering self-supervised learning. Teirth- Conference on Neural Information Processing Systems, 2023.
13

---

[41] Xi Weng, Jianing An, Xudong Ma, Binhang Qi, Jie Luo, Xi Yang, Jin Song Dong, and Lei Huang. Clustering properties of supervvee learning, 2025.
[42] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning, 2019.
[43] Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy, Understanding contrastive learning requires incorporating inductive biases. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19250-19286. PMLR, 17-23 Jul 2022.
[44] Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. In Algorithmic Learning Theory, pages 1179-1206. PMLR,
2021.
[45] Advait Parulekar, Liam Collins, Karthikeyan Shanmugam, Aryan Mokhtari, and Sanjay Shakkottai. Infonce loss provably learns cluster-preserving representations. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pages 1914-1961. PMLR, 12-15 Jul 2023.
[46] Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. In International Conference on Learning Represemtations, 2022.
[47] David McAllester and Karl Stratos. Formal limitations on the measurement of mutual infor- mation. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 875-884. PMLR, 26-28 Aug 2020.
[48] Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2020.
[49] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939, PMLR, 2020.
[50] Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2404-2504, 2021.
[51] Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 14::11834-1188 2021.
[52] Roland Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pages 12979-12990.. PMLR, 2021.
[53] Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning. In Gustau Camps- Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 1 of Proceedings of Machine Learning Research, pages 7187-7209. PMLR, 28-30 Mar 2022.
[54] Kento Nozawa and Issei Sato. Understanding negative samples instance discriminative self-supervised representation learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5784-5797. Curran Associates, Inc., 2021.
14

---

[55] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 5000-5011.
Curran Associates, Inc., 2021.
[56] Jeff Z. HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: analyzing the linear transferability of contrastive representations to related subpopulations. In Proceed- ings of the 36th International Conference Neural Information Processing Systems, NIPS 222, Red Hook, NY, USA, 2022. Curran Associates Inc.
[57] Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z. Haochen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In Kamalika Chaudhuri, Stefanie Jegelka. Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19847-19878. PMLR, 17-23 Jul 2022.
[58] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a lad- der: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations, 2022.
[59] Han Bao, Yoshihiro Nagano, and Kento Nozawa. On the surrogate gap between contrastive and supervised losses. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 1585-1606. PMLR, 17-23 Jul 2022.
[60] Jeff Z. HaoChen and Tengyu Ma. A theoretical study of inductive biases in contrastive learning. In The Eleventh International Conference on Learning Representations, 2023.
[61] Randall Balestriero and Yann LeCun. The birth of self supervised learning: A supervised theory. In NeurlPS 2024 Workshop: Self- Self-Supervised Learning Theory and Practice, 2024.
[62] Yuandong Tian. Understanding deep contrastive learning via coordinate-wist optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 19512- Curran Associates, Inc., 2022.
[63] Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: a theoretical analysis. J. Mach. Learn. Res., 24(1), January 2023.
[64] Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self- supervised contrastive learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11112-11122. PMLR, 18-24 Jul 2021.
[65] Yuandong Tian. Understanding the role of nonlinearity in training dynamics of contrastive learning. In The Eleventh International Conference on Learning Representations, 2023.
[66] Shlomo Libo Feigin, Maximilian Fleissner, and Debarghya Ghoshdastidar. A theoretical characterization of optimal data augmentations in self-supervised learning, 2025.
[67] Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. Under- standing and improving the role of projection head in self-supervised learning, 2022.
[68] Yu Gui, Cong Ma, and Yiqiao Zhong. Unraveling projection heads in contrastive learning: Insights from expansion and shrinkage, 2023.
[69] Yihao Xue, Eric Gan, Jiayi Ni, Siddharth Joshi, and Baharan Mirzasoleiman. Investigating the benefits of projection head for representation learning. In The Twelfth International Conference Learning Representations, 2024.
15

---

[70] Zhuo Ouyang, Kaiwen Hu, Qi Zhang, Yifei Wang, and Yisen Wang. Projection head is secretly an information bottleneck. In The Thirteenth International Conference on Learning Representations, 2025.
[71] Noga Alon, Dmitrii Avdiukhin, Dor Elboim, Orr Fischer, and Grigory Yaroslavtsev. Optimal sample complexity of contrastive learning. In The Tweifth International Conference on Learning Representations, 2024.
[72] Zhuoning Yuan, Yuexin Wu, Zi-Hao Qiu, Xianzhi Du, Lijun Zhang, Denny Zhou, and Tianbao Yang. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 Proceedings of Machine Learning Research, pages 25760-25788 PMLR, 2022.
[73] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self- training with deep networks on unlabeled data. In International Conference on Learning Representations, 2021.
[74] Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-super learning recover global and local spectral embedding methods. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[75] Jason D Lee, Qi Lei, Nikunj Saunshi, and JIACHENG ZHUO. Predicting what you already know helps: Provable self-supe learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 309-323. Curran Associates, Inc., 2021.
[76] Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In The Eleventh International Conference on Learning Representations, 2023.
[77] Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim G. J. Rudner, and Yann LeCun. An information theory perspective on variance-invarance-cvaaaaa regularization. In seirth- Conference on Neural Information Processing Systems, 2023.
[78] Dustin G. Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features, 2020.
[79] Cong Fang, Hangfeng He, Qi Long, and Weijie J. Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. Proceedings of the National Academy of Sciences, (18(4))e2103091 18, 2021.
[80] Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 31697-31710. Curran Associates, Inc., 2022.
[81] Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. In First Workshop on Pre-training: Perspectives, Pirfalls, and Paths Forward at ICML 2022, 2022.
[82] Tomer Galanti, András György, and Marcus Hutter. Generalization bounds for few-shot transfer learning with pretrained classifiers, 2023.
[83] Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. Comparative generalization bounds for deep neural networks. Transactions on Machine Learning Research, 2023.
[84] Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. In Alice H. Oh Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
16

---

[85] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[86] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[87] Liu Han. Tiny imagenet challenge. Seep-learning thu-2020 2020 2020. Kaggle.
[88] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011.
[89] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[90] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[91] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[92] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708.08
2017.
[93] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.0266 2017.
[94] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1rXiv:1608.03983, 2016.
[95] Arne Köhn. What's in an embedding? analyzing word embeddings through multilingual evaluation. In Lluís Mårquez, Chris Callison-Burch Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2067-2073, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
[96] Abhijeet Gupta, Gemma Boleda, Marco Baroni, and Sebastian padó. Distributional vectors encode referential attributes. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 12-21, 2015.
[97] Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 2801):007-2 2022.
[98] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3519-3529. PMLR, 09-15 Jun 2019.
[99] Nikolaus Kriegeskorte, Marieke Mur, and Peter A. Bandettini. Representational similarity analysis: connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4, November 2008
[100] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1748-87733 PMLR, 18-24 Jul 2021.
17

---

[101] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary ViVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[102] L. McInnes, J. Healy, and J. Melville. UMAP: Uniform Manifold Approximation and Projec- tion for Dimension Reduction. ArXiv e-prints, February 2018.
[103] Kentaro Yoshioka. (vit) and related models cifar-10. 10.
rision-raafformen-ccfa11000:: Training vision transformers
betps://githut cow.uu/.cco/eettaoo7777..
vision www.strrraoroeesiiieiieeee cifar10 2024.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract outlines both the theoretical and empirical contributions, well as the main assumptions made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide limitations section in the main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

The paper provides the full set of assumptions for each theoretical result, along with complete and correct proofs the appendix. All key logical steps in the derivations are clearly presented.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe the experiments and their setup in detail in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
20

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide well-doeumented code in the supplemental material reproduce our experimental results. We use publicly available datasets in our work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
Sublic/guides (wdeeSubiissiiooooll......
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
cttpss.cccpubblci gwwdes/ideeeee www.oo/wwwwswooooooooooooooo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we provide full details required to reproduce our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We include error bars when appropriate and feasible, given computational budget. For instance, in Tab. I and Fig. + where the experiments were averaged over different sample selections class subsets, report the results using means and standard deviations or error bars. Due to limited computational resources, we could not afford to repeat the more computationally intensive experiments. Instead, we prioritized running them on multiple datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide information about our compute resources, and memory.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is theoretical paper on self-supervised learning. While it analyzes algorithms that significantly impact widely used applications, the contributions of this work have no direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
As theoretical paper, it does not pose risks of this kind.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available datasets and code with appropriate attribution and in accordance with their respective licenses. Specifically:
Datasets: CIFAR-10 and CIFAR1O0 85 mini-ImageNNet 86 and SVHN XS that are explained in Appendix A
Frameworks: Our models are implemented in PyTorch [101. which is released under the BSD 3 License.
We build on official or open-source implementations of SimCLR and MoCo, distributed under Apache 2.0 and CC-BY-NC 4.0 Licenses, respectively.
Codebases:
Guidelines:
The answer NA means that the paper does not existing assets.
23

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets..paerssttt paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not create any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We LLMs for editing (e.g., grammar, spelling, word choice).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy for what should or should not be described.
25

---

A Additional Experiments
Datasets. We experiment with the following standard vision classification datasets CIFARIO
and CIFARI00 [85] mini-ImageNet [86]. Tiny-ImageNet [87]. SVHN [88] and ImageNet-IK [89]
CI and CIFAR100 both consist of 50000 training images and 10000 validation images with 10
classes and 100 classes, respectively. uniformly distributed across the dataset, i.e., CIFARI 10 has 5000
samples per class and CIFAR100 has 500 samples per class. mini- ImageNet also has 5000 images on top of 50000 train and 10000 validation images, with 100 of 1000 classes from ImageNet-IK [89] (at the original resolution). total 200 classes from IM-IK. Each class has 500 training, 50 validation, and 50 test images. SVHN consists of digit classification data with 10 classes from real-world images, organized into "train" (73,257 samples), "test" (26,032 samples), and "extra" (531,131 samples) splits. For scalability verification, we combine the "train" and "extra" splits during training for validating Thm. T (shown in Fig. B). ImageNet- IK spans 1000 object classes and contains 1,281,161 training images, 50,000 validation images and 100,000 test images. We use "train" split for training and "validation" split for reporting test results.
contains 100000 images downsampled to 64 x 64, with
CIFARIO
CIFAR 100
mini- ImageNet
(a) Initialization
(b) DCL
(c) NSCL
Figure 8: UMAP visualizations of representations from models trained with different objectives (Random init, DCL, NSCL) on CIFAR10, 10, CIFAR 100, and mini-ImageNer. Each point corresponds an image embedding colored by class. Better clustering and separation are evident as move from Random to NSCL.
We use the same augmentations as SimCLR [5). For experiments on mini- Mini-lmaeeNet. and IM-IK, use the following pipeline: random resized cropping to 224 x 224,
Data augmentations.
26

---

1000
OCL
7.40
V-
1-1
DWC1 bix
6.80
Bec1 box
Coupagg 1o5.
Contrott ve toee
1.00
1230 2oun6
Contrret un toee
U2ON bound
60
Uzper bound
Contoggt
100
60
0.20
6:40
10
30
31
10
3V Taiingg (og scale)
10
221
12
Trath Dop
Taahing
Taiiing epocns (og
(a) CIFARIO
(b) CIFAR100
(c) mini- Muni-ImmgeNet
(d) Tiny-ImageNer
Figure 9: We reproduced Fig. 3 (top) for SimCLR with ViT-Base architecture. The loss-curves
remain tightly bound per Thm independent of the encoder network architecture.
7.60
8.00
Train
DCL loss
Train
DCL loss
Test
NSCL loss
Test
NSCL loss
Contrastive loss
7.75
Upper bound
Contrastive loss
7.40
Upper bound
7.50
7.20
7.25
7.00
7.00
6.80
6.75
p. p-val (train): 1.00, 0.00
A. p-val [train]: 1.00, 0.00
p. p-val (test): 1.00, 0.00
A. p-val (test): 1.00, 0.00
6.50
10
10
10a
103
100
10
10
10
Training epochs (log scale)
Training epochs (log scale)
Train
DCL model
7.60
Train
DCL model
7.40
Test
NSCL model
Test
NSCL model
Contrastive loss
Contrastive loss
7.40
7.20
7.20
7.00
7.00
6.80
6.80
6.60
p-vai (train):
0.88, 0.00
p. p-val (train): 0.87, 0.00
p val (test):
0.82, 0.00
p. p-val (test): 0.75, 0.00
6.60
100
10
10
10a
100
10a
102
10a
Training epochs (log scale)
Training epochs (log scale)
(a) CIFARIO
(b) CIFARI00
Figure 10: The DCL and NSCL losses, along with our bound, for MoCo. Same as Fig. 3 with
MoCo training in place of SimCLR.
random horizontal flipping, color jittering (brightness, contrast, saturation: 0.8; hue: 0.2), random grayscale conversion (p = 0.2), and Gaussian blur (applied with probability 0.1 using a 3 x 3 kernel and a = 1.5). For CIFAR datasets and SVHN, we adopt a similar pipeline with appropriately scaled parameters. The crop size is adjusted to 32 32, and the color jitter parameters are scaled to saturation 0.4, and hue 0. For Tiny- ImageNe,, we use the same saturation and hue values as CIFAR datasets with the crop size is adjusted to 64 x 64.
UMAP visualizations of clustering behaviors. In Fig. T we visualize the clustering behavior of DCL and NSCL-trained representations learned on mini- mini-ImageNet using 2D UMAP [102] projections. We randomly selected classes from mini-ImageeNe and sampled 200 images per class, projecting
their corresponding embeddings into 2D space using UMAP. As shown in Fig. I training with the
NSCL loss results in tight and clearly separable clusters, as predicted Thm. 2 Although DCI. label-agnostic, as shown in Thm. and Fig. DCL training implicitly minimizes the NSCL loss,
27

---

0.9
Train
NSCL, NCCC
0.9
Train
NSCL, NCCC
0.8
Test
DCL, NCCC
0.7
0.7
0.6
0.6
Error
0.5
0.4
0.3
0.2
0.1
0.1
0.0
0.0
10
100
200
500
20
50
100
200
Number of shots
Number of shots
(a) CIFARIO
(b) CIFARI00
Figure 11: m-shot error (Nearest Ceast-e Classifier (NCCC) for C-way (all-class) classifica-
tion.
100.00
MOCC
100.00
Tran
100.00 DCL
DCL NCOC
Tan
MCCG
Ereor scaee)
10.00
Erroe bound Lim bound
Error tlgg scale)
10.00
Emter Loundd
(1m sound
10.00
1.00
1.00
1,00
0.50
0.50
0.50
0.30
0.10
0.10
MODCC ESS.
ISCCO: $6.
PHRIINOO0: Cd5. 3.C
0.01
mu 3P1
PL
50
10
100
500
200
Number shots
Number of shots
Number shots
(a) CIFARIO
(b) CIFAR100
(c) mini mmaie-mmaeeNet
Figure 12: Two-way m-shot NCCC and errors with ViT-Base to verify the bound in Cor. as shown in Fig. 6 (bottom).
which explains the visible clustering behavior seen in Fig. I In Fig. g we demonstrate that the behavior generalizes across different datasets, i.e., CIFARIO, CIFARIOO, and mini-ImageNet.
A.1 Extension to SimCLR (w/ ViT) and MoCo
To verify the generality of our empirical findings in the main text, we repeat some primary experiments with Vision Transformer (ViT) 91 SimCLR, and also using the Momentum Contrast method, specifically MoCo v2 (6. This section summarizes empirical results of SimCLR-Ve RVV and MoCo, mirroring our analyses for MumCLR-ResNNt--S which we presented earlier.
SimCLR-ViT setup. We adopt the ViT-Base (ViT-B/16) architecture, which consists of 12 trans- former layers, each with 12 attention heads and hidden dimension of 768. For input images of size 224 x 224. we use a patch size of 16 x 16, resulting in 196 tokens (plus [CLS] token). The MLP hidden dimension 3072, and layer normalization is applied before each attention and MLP block. For CIFARIO and CIFAR we follow the scaling procedure described in [103]. adapting patch size and positional embeddings accordingly.
MoCo setup. We use the same architecture as with SimCLR. To train our model, we use SGD as optimizer. We set momentum 0.9 and the weight decay to 1c-4, All experiments out with a batch size of B = 256. The base learning rate is set to 0.03 (27). Similar to our SimCLR training strategy, we employ a warm-up phase 993 for the first 10 epochs, followed by cosine learning rate schedule [94].
carried
A.1.1 Results
Validating Thm. dduring training. We train SimCLR-ViT and MoCo using the DCL loss for 2k epochs, and evaluate both the DCL and NSCL losses training and test datascts, along with the
28

---

Normalized variance
10
10"
10
Norallzzdd
10
10
(CDNV)
10-1
DCL (Dir CDNV)
10
DCL (Dir CONV!
NSCL (CONV)
Than
NSCL (CDMV
Train
NSCL
Test
CONV
Training epochs tlgg scale)
Training epochs (log scale)
(a) CIFARIO
(b) CIFARI00
10.0
10.0
10.0
Normali zed variance
Nomalized variance
Normalized
0.1
DCL (CDNV)
0.3
DCL (CDNV
0.1
DCL (CDNV)
CONV)
(CDNV)
Train Test
NSCL
Train Test
(CDNV)
NSCL (Dir CONVI
NSCL (Dir CONV)
NSCL (Dir CDNV)
Test
10a
"104
10p0
18>
0.0
10a
Yoa
100
10"
0.0
10
10a
10
10
Training epochs (log scale)
Training epochs Dlg scale)
Training epochs (ogg scale)
(c) CIFARIO
(d) CIFARI00
(e) mini ImageNet
Figure 13: CDNV and directional CDNV analysis. Top Row: Results for MoCo-trained ResNet-50 on CIFARIO and CIFARIO0 100. Bottom Row: Results for SimCLR-trainee ViT-Base on CIFARIO. 110 CIFAR100, and mini- mini-ImageNNt.
N N-nmas We repeat the experiments illustrated in Fig. 3 top). In Fig. 9 and Fig. 10 top). we notice similar trends as we have shown for SimCLR. The DCL loss consistently upper bounds the NSCL loss, with narrowing gap with increase in number of classes C. The empirical CCL and L NSCL values remain tightly bounded by our proposed bound throughout training. For MoCo, further validate that minimizing the DCL loss implicitly leads to low NSCL loss by repeating the experiments in Fig, 3 (bottom). As shown Fig. 10 boottomm)) NSCCL for both DCL and NSCL-trained models remain close. This confirms our earlirr conclusion that optimizing DCL implicitly reduces the NSCL. loss.
theoretical upper bound C NSCT. + log(1
Comparing downstream performance. We evaluate the quality of the learned representations from MoCo models via few-shot error analysis as earlier shown in Fig. [6] Specifically, we report the NCCC error both train and test datasets. In Fig. topp) for MoCo, evaluate NCCC error the complete dataset as a function number of shots per class (m) and show a comparison between DCL and NSCL-based models. In Fig. 12 for SimCLR-VE
perform 2-way classification (averaged
over 10 tasks) and compare the NCCC error with the bound from Cor. We notice that the theoretical
bound remains tight for increasing
Similarly to our previous analysis, we also evaluate CDNV and directional CDNV in Fig. 13 for both DCL and NSCL-based MoCo and SimCLR- R-V models. We observe that while CDNV initially decreases for both models, the NSCL-trained models achieve substantially lower CDNV values as training progresses, whereas DCL-trained models maintain higher CDNV values. For directional CDNV, both the models achieve substantial reductions (an order magnitude). For CIFARIO, directional CDNV for NSCL drops an additional order of magnitude compared to DCL.
B Proof of Thm.
Theorem Let S ((1,, y:)) C. X [C] be labeled dataset with C classes, each containing most Druaxe distinct samples. Let X d be any function. Then, we have
(SCC < CCCCL()) < c NSCL log [1+
NSCL
Omage
N-
where denotes Euler's constant. For balanced classification problem,
29

---

Proof. First, we show that L CNSCT(() (f) S CDCL.() (f). For each anchor (1, (with i E (I...., , and [I.....K) define
N
K
neg
exp(sim
22))),
pos
exp(sim(22) 22/))))))/
1==1
j=l
j=1 Wy
and pos
di ;
V: P
By the definitions of the decoupled contrastive loss (DC and the negatives-only supervised contrastive loss (f), we obtain
DDCL (f) - C NSCL, (f)
N
(sp(sim(2/,2//1))
exp(sim(ef 22/2))
Σ
log
+ log
K2N
Zing + 2id1
pool/self
Zuig
1=1
N
K
K
Zudli + Z
pos
(5)
Σ
log
K
Zidg
1=1
11
sost
log
1
Zidg
Since log(1 + x) 0 for all 2 0, it follows that NSCL (FCL.(). A similar argument
shows that CDCL (n) < (
Next, by using the same reasoning as in (5) but replacing z
postsell with pos we have
(oL NSCL (f)
1
log
+
Zid pos tids neg
N
2 For fixed anchor (i,4). note that there are at most Knmax positive
Next, we bound the ratio
2
terms (since each class contains most mmax samples) and least K(N mmaxx negative terms. Moreover, because sim(z, x') E [-1, 1]. every term satisfies
exp(-1) I exp(sim(2 (2 22))) s exp(1).
Thus, we obtain
Ziu pos
<
K nmmax cxp(1) and Zudt 2 K(N mmmax exp( (-1).
Hence,
Zi,
K Kn Rmmax cxp(1)
Bmaa
pas
Zidz
K (N mmxx) exp(-1)
N 7max
follows that for each anchor,
Mmmax 2 nmaa
log
+
S log
1
Zid
Since this bound is uniform in i, 11, and lz, we conclude that
(C-(1)) - - (SCL(()) < log 1 N
T
1 emaae
Amax
N -maxx
where the last inequality follows from log(1 + z) S for all 2 0.
30

---

B.1 Batch-Based Contrastive Losses
In practice, contrastive learning objectives are implemented using batches samples rather than full-dataset sums [5]. To formalize this, let B E N denote a chosen batch size. We assume access dataset S {z yt)] X [C], where 2; E X are inputs (e.g., images) and y; [C] are their class labels. For each sample Ti,, let x ~ a((x)) be an independently generated augmentation from distribution of augmentations a(z;). Given batch B = {{2}, 4 H5))e1 sampled with replacement from S, we define a per-sample contrastive loss for an anchor (24, y:) as:
f (f,(()) - log EjEB (xxpsii((((=:))(())))) +
(6)
The self-supervised contrastive loss then defined as the expectation over randomly chosen anchors, their augmentations and batches:
() E,..e Es (c.,s()))
In contrast, the negatives-only supervised contrastive loss restricts the negatives in each batch to only include examples from different classes. Specifically, for each anchor (ar4, Ui), define B {[Tj. Z/,, Má. ) as a batch of size B, drawn (with replacement) from S \ {(zj, My)
Mi
Then, the loss
LB C2)O - E2,zEE: (4,, (f)
With these notations are ready to describe bound the gap between these two loss functions.
Theorem 3. Let S = (( v1) C X' x [C] be labeled dataset with C classes, each containing n distinct samples (N = Cn). Let B E N be the batch size, <> O be a positive number and
B = [B(1 - - []] Let f X + Rd be any function. Then, the difference between the self-supervised contrastive loss fo and the decoupled supervised contrastive loss CNSCL (f) satisfies
L
22(og(2BB++2 exp())2xxxp((2 < - (ʃ) (1)-111))) CB
CSCC
e c(1-)-- 2((og(2B)+ exp( exp(-2BB2) 2)
Proof. Fix an arbitrary anchor sample (x,, E S and let ny be an augmentation of mq. Denote
2; f(x;) and 2 f(z()
the corresponding embeddings. Next, let B {( Ub. ) be random batch of B samples (with replacement) from along with their augmentations. For the anchor (xi, define the per-sample contrastive loss by
exp (sim( 2(D)
(,,(S) log EjEB eexp (sim(z), 2j)) + exp (2s: 25))
Thus, the overall self-supervised contrastive loss is
C (f) = E, Es [(,
For each anchor i, consider instead a batch
B [(I 2/y, ,3.) _
of B = [B(1 - - c)1 samples drawn (with replacement) from S {(z),,;) y = yı}. i.e., only
from the negatives relative The decoupled supervised contrastive loss defined as
CNSCL( C (f)
E. R5, [C.,x (f)]
For each anchor i, define the event
A; {B # E B: yj # vi B}.
When A; holds, the distribution of random subset ByK of B negatives from B coincides with that of BY. By the law of total expectation.
CC)
NSCL
(f) E, [P[AA.] E BA [(, E,, (f]] + EE, [F[A]] (E A. [.),(())] Es; [c,,r (f)])
31

---

To bound the difference on the event A,, define
Σ
(exp [sim(2s, 2,)) + exp (sim 2ÿ))]
JEB
Then, one may write
Z
(, (1) log
log
1+
Zr,
Z,,
Since log(1 + u) S for u 2 0, it follows that
Zi, Zi,
( (f)
Observe that ByB above by Similarly, sy is of B terms, each bounded below by e Therefore,
By is sum of B B terms and, since sim(:.) E 1-1, 1], each term is bounded
cBB B)
(B B)
c² B(1
C 6))
c² + c1 cC)
B
B 0 -6)
- C(1-6)-
Thus, we deduce
(1++
EA [(, 1, (f) $
C(1-6)-1
On the complement A, the batch B contains fewer than B negatives. In this case one may bound the losses uniformly. In fact, using
B
log( exp(ay)) & maxfai...... ag} + log
and the fact that sim(...) E 1-1, 1]. we obtain
((,(01 < [sim(z,, =1) + log and similarly,
Σ JEB
exp((sim(zzzz)) zy)) + exp(sim(zs: xf)))
< log(2B) + 2,
|4,,5 (f) S log log(2B) +2 < log(2B) + 2. so that [Y
Define the indicator variable Y, [ * yil
J. By Hoeffding's inequality,
B
P Σ Y, < [B(1 ( Be] = P
Σ
Y, < B(1 - ) - Be
S eexp(22Be"
Hence, P.A.]] S exp (-Be²) and the contribution of the event A, is bounded by
E2 [P[[A.] (EB)A [()] EBy [Cr,sy (f)))
< F2 [P[A.] (E [[(,s())] + EBy [lC.,yy (fI)
< 2 (log(2B) + 2) exp (-Bc²).
Combining the bounds on A; and A., we conclude that
CEL ((0)-NNSH CNCCL (S) < c2(1+c0)
C(1 (11-c)-1 + 2 (log(2B) + 2) exp (-2Be2).
Finally, we note that under A,, (i,(()) > l..r (f). In addition, similar to the above:
E, [P[].]] (BBJA, [f) - EBy [4,,5; (f)]) 2 -2 (2Bg(2 + 2) exp ( -Be?)
yields the corresponding lower bound:
(C)(() L (NSCL() 2 -2 (log(2B) + 2) exp (-2B(2)
This completes the proof.
32

---

C Proof of Prop. |
Proposition 1. Let C' 2 2 and m 2 10 be integers. Fix a feature map f : :: + R and
class-conditioaal distributions Dı,..., Do over X. Let Vi = Avgi [V, (D;, Dj)], We have:
en (f) s (f) < (C' (1' (1) + vm v7
V + 4 Vi] 3m vm
(4)
Proof. We adapt Prop. 7 of 82 Assume dıj == 1|ا - 1اll > 0 for all i # so that Wij == (1y p4)/ds day is well defined. Fix i + j. Draw independent samples S, ( Fi,m } DTmh ~ DJ", independent also of a fresh 2, ~ D,. Let uü = m Ex= ((c,o) and Me = [x](() | x~ DJ] for E {i,j). and write 2; = f(z)). Set de == me Me. For C E {i,3]
and S = {2y.1...,,
define f(ze) with Ie De. let Ee Cov(ze) and Ue tr(Ee), and put
Vy(D,, D,) = EnyL
Vj(D,, == U/ + Uj
dr
dy
Let D = Wij - Mi: d [|||, and A 2; Mitus A direct expansion gives
|| 10 |||: ph 1 = d² 2dX
2A (6, 5.) + D (5, +,) + (11,, [18:11))
DT,
where X = (z: - m:) T Usg. Hence NCC predicts j iff
2 T + S +
1 Q
2x 2 d
T A ( (6,-b:), s T (6()++6)) Q: = -1,111--16111- (8)
d
d
Fix a 2 and m 2 6, and set T(a,
;= 1 2
23/2
E (0, 1/2). sol 1 2F + 2/2 If
2
am
2
[S]
Q
S 2r,
d2
d²
then 8 implies X 2 rd, hence
{NCC predicts j} S (X 2 rd]
U
2(T)
+
181
+ 12
2T
de
24
2
Since E[x] = 0 and Var(X) = U/j J EuUjs
Var(X)
Pr[X 2 rd] s
dd
=
22 VV (D, Dj).
(10)
By Markov and Cauchy-Schward
2 + E|S| + E|QI
Pr
>1 1 2T
(11)
d=
de
11-27 (d
Using independence of (zig 81, 6,) and B[d.] = -,
B[T2] 1 m
tr (E, + E,))
da T 4 U (E, + Eg, E.)0us))
B[sa] d² m "T + Exjuij-
Since tr tr(B)) S tr(A) (r) for PSD A, B and T Eu str(E),
tr (E, + Σ, S 3/ (v. + uy))
uT/(E, + E,)us S 2/ 22.
Hence, by Cauchy-Schwarl and V vab < (a + b)/2.
2
2
d
va(u; U2)) _ (v; (z))
d
Vm d²
1
U/ + Ujj
+ Uj
- +V%),
1
2
m
d2
d2
33

---

and
E S
+ vj
1
Viji
B[Q
Vig d2
d2
m
d
mm
m
m
Therefore
2
2
+ E|S|
+
B
1
(2/Vis + 2Vis) + m
Vij-
(12)
d2
Combining the union bound,
TTO and 12
Pr(NCC predicts j) S r(a, m)--m/ +
( Vij +
2
Vij + - Vis
1 - 2r(a, m)
(/m
Vm
Averaging over and summing over j # yields
errm (J) 1 (C' 1) ((a,m)= + 1- m) (f)
1
2
2
- V ]
Vj
+
Vj
V
vm
m
Since (1-27)-1 v a/4, we obtain the simplified bound
err m.D (f) < (C' 1) inf a25
(: a
22/2 am
-2
Vj
+
2 m
V7
Vf
1 V m
(13)
4
vm
By picking a = 16 and using the assumption that m > 10 we get the desired bound. Finally,
err m.D I NCC since the Euclidean nearest-centroid rule is representable by multiclass linear
classifier.
Corollary 1. Let C' 2 2 and m 2 10. Fix a feature map f : X + R and class-conditional distributions Di...., Dc over X. Write V, = Avgiz (V, D,)], V, = Avg [V)(D,,D))
and Vj = Avgiyj [V, Dj)]. Set
23/2
( 2
2 vm
-
F == 2 A
A = 2
B = 4 m
v
+
V1
m
m
B
with the understanding that if B = O then the bound below reduces to (C" 1) 4V, (achieved in
the limit a + 20). Define yf to be the unique positive real root of y² 8Fy - 16F A 0, namely
8F
A 2
- 3² = 8F A+
1
8F 27
8F
A V 12
8F
27
27
8F
2F
3 8F
A
COs
arccos
3AV
27
3
Let a" == -2A+ y" and define the constrained optimizer Copt == max a*}. Then
M.D.
NCC S (C' 1)) E(aopt),
E(a)
11 2 2
23/2
-2
Vj B
am
Proof. Starting from
NCC(f) S (C" 1) inf
(1 (1-22
23/2
-2
Vj
+
a
(
Vi
+ 2 V, 2,,, Vm
+ =v,))
a25
am
4
7m
write E(a) = 2V, Ba with c(a) 1
2-2A 24
A 2
2/2
and B as above. On
m
(2A, oc) one has
24,A A a²c(a)3
Ea(a) = 32A (A + a)
E'(a)
+ B,
0.
(2A-o) a)
so strictly convex with unique stationary point a* 2A solving
2²,A A
a²c(aa²²
F.
34

---

2A)3
y²
Let - 2A. Since aLc(a)s
8(y + 24) the stationarity condition is y²
8a
8Fy 16F A 0. Cardano's formula gives the stated y" > 0 and a' = 24 + y" Ata"
B 4A a (a + 2A)
E(a")
by substituting the stationarity equation into E. Because m 2 10 we have 2A < 5, so the constrained minimizer [5, oo) is Copt max a²} and infa>s E(a) E(aopt), which yields the claim
and two cases. B O, then E(a) = c(a) -V, decreases to 4V, as a + oo, giving the stated simplification.
D Proof of Thm.
Theorem 2. Let d 2 C C and let S Wi C X x XC] be a balanced labeled dataset with C classes. Suppose f a global minimizer of the supervised contrastive loss CNSCL (f) (over all functions f: + Rd). Then, the representations satisfy the following properties:
1. Augmentation Collapse: For each i E [N] and for every pair 1112 E [K] we have 2 2/2
2. Within-Class Collapse: For any two samples and x, with the same label (y: Fÿ), their representations coincide: 2, = 2z. Namely, each class has a unique class embedding.
3. Simplex Equiangular Tight Frame: Let (uI)...,He) denote the set of class-center embeddings.
These vectors form a simplex ETF in R4; specifically, they satisfy Ee He = 0, ||Hell2 = ||
||e-112 and (Po:Pe) C-1 for all e + d' E (C).
Proof. We divide the proof into three parts. the first part, we restate and relax the problem. In the second part, show that the embedding vectors of augmentations of the same sample collapse to the same vector. In the third part, we demonstrate that the embeddings of samples from the same class collapse to the same vectors, and the class vectors form Simplex ETF.
Restating the problem. Since we focus on the unconstrained features model rewrite L NSCL (f) in the following way:
78
79
we can
exp(sim(z)) x/2)) Eー exp(sim(22 a/ .2/1))
(2)
Σ K2N 4
Σ log
Eb=1
#
where Z= (2))iE(N],te(H denotes a collection of learnable representations, where each of is the embedding of sample z; under augmentation 1.
Since one may either normalize the vectors externally inside the cosine similarity and because unit norm constraint is stronger than a norm upper bound constraint, we have:
exp(sim(=** 2²²))
min
K
log
K 21z=1 Ex fz. exp (ii ( 2/11)
K
N
exp
min
log
s.t. Vi [N], [K] ||
K
K E1z=1 En सr. exp (z+ 2/11)
1=1
K
exp , 2221)
2 min
log
s.L. e L [K] ||=/|| $
KºN
K Eiz=1 Enn tu. expl (2/1 (211,
Q2)
We will show that the global minimum of the last optimization problem is obtained when we have augmentation collapse, within-class collapse, and a simplex ETF behavior. In particular, this will give us the result that the solution to the latter problems is achieved in the same way.
35

---

Augmentation Collapse. We begin by proving that for every io E [M], the vectors 2/ 200 are identical at any global minimum. Fix some index do [N] and suppose for contradiction that there exist two distinct augmentations * 12 with 2 * 2/ Define the averaged vector 2ho * Σ % and let Z be the collection obtained by replacing 2/ by 2i, for all I E [K]. We will show that Q(Z) Q(Z).
k
Consider the loss function:
exp 21
2 K2N f1,2=11
Σ log
Eis=1
Ej exp( 2131)
Now, fix the index is and split the sum over i into the contribution from in and the contributions from all other indices + in. In addition, let de exp((tt) 2/2)). With this separation, we obtain
(
K
exp(1at), .2:))
2
log
K
exp 22/1))
11.1
km
1g
Σ
(2f) 2(0))
K
11.12=1
11 Eyfio
K
N
K
log
Σ
Σ
exp 22/11)
KON
log
Σ
exp((z 2/1
KN
11
f:lyy #aip
Similarly,
K
2(2) N
(2101 200)
(2f1 2
K²
11.1==1 17/10
K
log
Σ
Σ
exp((xt) 25/1)) K Kexp((et' ((d))))))))
KaN
h1.2=1 sy6ig
11-1
JEIN]
K log
+ N
exp ((āro 2 1/1))
lg
By definition, since 2g
k Er i.g its squared norm is given by
1
K
(ãio ) = 11
K2
11.2==1
Since z for every [N] and E [K] (otherwise the cosine similarity would be undefined),
the function
(x) (z)) exp((=),,.))
convex æ. Consequently, by Jensen's inequality, for any subset and any collection of vectors 2 j E S} we have
(4+
Σ expl (+ 2/))) 2 exp
JS
jES
jES
with equality if and only if all vectors 2// in the subset are identical.
Applying this inequality to the relevant subsets yields
K
log Σ
Σ
exp (Ep(( (39)) > log Σ
Σ
exp((='
,xp((=(+,//)) + K (o0(()),zz)))
1==1i:
J: My #
i
36

---

where the strict inequality follows from the assumption that for some indices 1 and 12 (with 1 * 12).
11 we have =/0 + kg
Next, define the function
K Σ
G(x)
log
Σ
(ap((a,22)))
9: M
is well known that G(r)) is convex in æ. Therefore, by Jensen's incquality we obtain
1 Σ (())t)
G K
Σ
20
K
which can be written equivalently as
K
log
Σ
Σ My
exp((= 2//11
2 log Σ
Σ My=1 Mi
exp((Et0) =/311)
K
1=1
Combining this inequality with the previous expansion, we deduce that Q(2) < Q(Z), which contradicts the assumption that Z is global minimizer of the loss. Consequently, for each in E [N],
must that all vectors = 2 :
2K are equal a global minimum.
Within-class collapse and Simplex ETF. Since the augmentations collapse, with no loss of general- ity we assume that K = 1. We now analyze the embeddings 2; by applying the mean (AGM) inequality to the n(C' - 1) positive numbers
{exp (exp((zs, =5)) 1 UJ + vi).
for each fixed E [N]. The AGM inequality yields
NTC
1
Σ
exp((z,,))
II exp((zn,2))) jツリテリ
(14)
(CC
Taking natural logarithms both sides gives
log
Σ
exp((zz, (())))))) =g)1) 2 log(n(C-1))
1
Σ
(5)
n(C 1)
ド:
Averaging this incquality over i E [N] yields
N
1
Σ
log
Σ exp((z,, (())))))) log(n(C-1))+ Nn(C 1)
Σ
Σ
(,,, (15)
N
jon
Notice that the double sum of inner products can rewritten as
Σ (z),2))
(z zz)
1=1 fu
1=1
f:ly =
1=1
tiy= =c
= n
mmpeell
c=1
where we define the class-mean embeddings as Me Eisgime Ri. Substituting this identity into yields
15p
c
C
log
Σ
(<p(((r,2))))
2 log(n(C- -1)) C 1)
Σ
Me
Σ
llml
(16)
N
#1:
c=l
37

---

By Jensen's 's inequality.
Moll
=
Σ
2
S
1
Σ
|
2
1,
(17)
=:
it follows that
Σ (2))2))
n²
Σ
Me
2
n²C
(18)
i=1
Substituting this estimate into 15 gives
Σ exp(( ()))
2 log(n(C' - + C(C - 1) 1 Hol -c).
log
N
Substituting the bound (16) into the expression for Q(2) (the loss) results in
(
1
2 + log(n(C + C(C - 1)
1
Σ Me c).
(19)
N
Since
12il2 2 -1 and
He 2
2
(20)
we obtain the lower bound
Q 2 log log(n(C 1)) - -1 1
(21)
C-1
Equality in (1 is achieved only if all the preceding inequalities hold equalities. In particular,
equality in requires _ Me and Vi [N] ||2/||2 Similarly, equality in (17) (used in
the derivation) forces
=g = which, via Jensen's inequality, implies that My: for all i E [N] Furthermore, equality in (1) is attained if and only if
| >0
Vi exp (zi, zz)) is constant w.r.t. j such that yj + y/- In particular, by applying natural logarithm in both sides, and considering that fact that z; Myy
Ve E [C] (Mes pe) is constant for all c' + c. In particular, (44, Mb) = (Ur: Mb) = (pr and we have the more general equation:
(Mer constant for all + d' [C].
Together, these conditions exactly characterize simplex equiangular tight frame (ETF); in particular, they imply that
C
Melll 1,
Σne pc = o, and (Per (po)De)) = C-
forall c c e [C].
straightforward calculation shows that
2 = -1 + 1 Σ log (C- 1) (opp(tt)) = log(C 1)
C-1
so that the lower bound (19) is attained. Consequently, a set vectors Z is global minimizer of (Z)) if and only if it forms simplex equiangular tight frame.
38


Reviews:
[{"review_id": "1o5M0uu7xg", "reviewer": "Reviewer_8fbK", "summary": "This paper provides a theoretical and empirical bridge between self-supervised contrastive learning (CL) and a supervised variant called negatives-only supervised contrastive learning (NSCL). The authors show that as the number of classes increases, CL implicitly approximates NSCL, even without label supervision. They prove that NSCL leads to representations exhibiting neural collapse structure, and they propose a new bound for few-shot linear probing error based on feature variability. Experiments on image classification benchmarks support the theoretical claims, showing that CL-trained models, despite lacking labels, learn semantically meaningful and transferable representations.", "questions": "1. The experiments show that the gap between DCL and NSCL decreases as the number of classes increases, which aligns well with the theoretical analysis. However, the performance difference between the two methods remains noticeable. More discussion on this observation could help clarify how much of the theoretical alignment translates into practical outcomes.\n2. In addition to DCL and NSCL, it would be helpful to include comparisons with standard CL methods. Based on the paper’s analysis, the gap between CL and NSCL is expected to be even larger, which could further support the theoretical claims.\n3. Can the authors quantify the similarity of the learned representations—not just the loss values—between CL/DCL and NSCL-trained models? For example, using clustering metrics and others.", "limitations": "While the paper presents clear analysis and supporting experiments, further evaluation on a wider range of models and larger-scale datasets would be helpful. This could clarify how well the findings generalize beyond the current experimental setting.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper presents a compelling and well-founded theoretical connection between unsupervised and supervised contrastive losses. The experimental results consistently support the theoretical claims.\nHowever, despite the clear structure and explanations, several questions remain—such as the performance gap between DCL and NSCL, comparisons with standard CL, and the degree of equivalence between the learned representations.", "was_revised": true, "final_justification": "The authors have provided a rebuttal, including extensive new experiments across multiple datasets and architectures. These results support the theoretical claims and address the concerns raised in my initial review. I also agree with reviewer 8nSm that clearly distinguishing the roles of CL and DCL in the revised version would help improve the clarity and impact of the paper.", "rebuttal": "**Note:** We thank the reviewer for their very thoughtful feedback and comments. Following the reviewers’ suggestions, we conducted multiple experiments to strengthen our work. Since there is no dedicated common space for experiments, **we present them at the beginning of the responses to reviewers 8fbK (part 1) and J3ay (part 2)**.\n\n---\n---\n\n## Summary of experiments (part 1)\n\n\n**Experiment (1)** We reproduced Fig. 2 (top) with SimCLR-v2 with the ResNet-50 architecture for **ImageNet-1k (1000 classes)**.\n\n| **Epochs** | **CL** | **DCL** | **NSCL** | **Upper Bound** |\n| ---------- | ------ | ------- | -------- | --------------- |\n| **0** | 6.9156 | 6.9146 | 6.9136 | 6.9210 |\n| **10** | 6.1298 | 6.1276 | 6.1265 | 6.1338 |\n| **50** | 6.0995 | 6.0972 | 6.0960 | 6.1034 |\n| **100** | 6.1165 | 6.1142 | 6.1130 | 6.1203 |\n\n**Experiment (2)** We reproduced the experiments in Fig. 2 (top) for SimCLR-v2 with the **ViT_base architecture**. The experiments were conducted for **TinyImageNet (200 classes)**, CIFAR10, CIFAR100, and mini-ImageNet. \n\n### *CIFAR10*\n\n| **Epochs** | **CL**   | **DCL**  | **NSCL** | **Upper Bound** |\n|------------|----------|----------|----------|-----------------|\n| **0**      | 7.4213   | 7.4206   | 7.3130   | 7.9125          |\n| **10**     | 6.8764   | 6.8753   | 6.7488   | 7.3483          |\n| **100**    | 6.7430   | 6.7418   | 6.6040   | 7.2034          |\n| **1000**   | 6.7128   | 6.7114   | 6.5706   | 7.1699          |\n\n### *CIFAR100*\n\n| **Epochs** | **CL**   | **DCL**  | **NSCL** | **Upper Bound** |\n|------------|----------|----------|----------|-----------------|\n| **0**      | 7.3633   | 7.3627   | 7.3522   | 7.4241          |\n| **10**     | 6.9615   | 6.9605   | 6.9494   | 7.0213          |\n| **100**    | 6.8088   | 6.8077   | 6.7964   | 6.8684          |\n| **1000**   | 6.7659   | 6.7648   | 6.7534   | 6.8254          |\n\n### *Mini-imagenet*\n\n| **Epochs** | **CL**   | **DCL**  | **NSCL** | **Upper Bound** |\n|------------|----------|----------|----------|-----------------|\n| **0**      | 7.1455   | 7.1447   | 7.1345   | 7.2065          |\n| **10**     | 6.6273   | 6.6259   | 6.6145   | 6.6865          |\n| **100**    | 6.5006   | 6.4990   | 6.4875   | 6.5595          |\n| **1000**   | 6.3979   | 6.3962   | 6.3839   | 6.4559          |\n\n### *Tiny-imagenet (C=200)*\n\n| **Epochs** | **CL**   | **DCL**  | **NSCL** | **Upper Bound** |\n|------------|----------|----------|----------|-----------------|\n| **0**      | 6.7862   | 6.7851   | 6.7800   | 6.8165          |\n| **10**     | 6.2234   | 6.2213   | 6.2158   | 6.2523          |\n| **100**    | 6.0748   | 6.0725   | 6.0669   | 6.1034          |\n| **500**    | 6.0342   | 6.0317   | 6.0261   | 6.0625          |\n\n**Experiment (3)** We reproduced the experiment in Fig. 4 (bottom) with a pre-trained ResNet-50 that was **trained on ImageNet-1k** and **evaluated the model on both ImageNet-1k and TinyImageNet**.\n\n### *mini-imagenet*\n\n|      Num-shots >         | **1**    | **5**    | **10**   | **20**   | **50**   | **100**  | **200**  | **500**  |\n|---------------|----------|----------|----------|----------|----------|----------|----------|----------|\n| **err_nccc**  | 0.2446 | 0.0866 | 0.0593 | 0.0526 | 0.0506 | 0.0413 | 0.0400 | 0.0400 |\n| **err_bound** | 55.3146 | 9.4475 | 5.3099 | 3.2047 | 1.7806 | 1.2012 | 0.8969 | 0.6788 |\n\n\n### *Full-imagenet*\n\n|      Num-shots >         | **1**    | **5**    | **10**   | **20**   | **50**   | **100**  | **200**  | **500**  |\n|---------------|----------|----------|----------|----------|----------|----------|----------|----------|\n| **err_nccc**  | 0.2813 | 0.1000 | 0.0566 | 0.0420 | 0.0400 | 0.0340 | 0.0360 | 0.0340 |\n| **err_bound** | 56.9036 | 9.5940 | 5.3628 | 3.2189 | 1.7749 | 1.1896 | 0.8687 | 0.6502 |\n\n\n**Experiment (4)** We reproduced the experiment in Fig. 4 (bottom) with the **ViT_base architecture** for **CIFAR10, CIFAR100, mini-ImageNet**. \n\n### *CIFAR10*\n\n|       Num-shots >       | **1**    | **5**    | **10**   | **20**   | **50**   | **100**  | **300**  | **1000** |\n|---------------|----------|----------|----------|----------|----------|----------|----------|----------|\n| **err_nccc**  | 0.3306   | 0.1357   | 0.0713   | 0.0360   | 0.0195   | 0.0176   | 0.0159   | 0.0153   |\n| **err_lp**    | 0.1661   | 0.0727   | 0.0357   | 0.0463   | 0.0347   | 0.0266   | 0.0160   | 0.0058   |\n| **err_bound** | 37.2999  | 6.1490   | 3.4042   | 2.0238   | 1.1013   | 0.7299   | 0.4423   | 0.3141   |\n\n### *CIFAR100*\n\n|      Num-shots >        | **1**    | **5**    | **10**   | **20**   | **50**   | **100**  | **200**  | **500**  |\n|---------------|----------|----------|----------|----------|----------|----------|----------|----------|\n| **err_nccc**  | 0.2973   | 0.1457   | 0.1052   | 0.0782   | 0.0622   | 0.0583   | 0.0537   | 0.0509   |\n| **err_lp**    | 0.2239   | 0.0970   | 0.0749   | 0.0599   | 0.0509   | 0.0420   | 0.0350   | 0.0290   |\n| **err_bound** | 32.6582  | 5.5778   | 3.1349   | 1.8920   | 1.0512   | 0.7091   | 0.5295   | 0.4007   |\n\n### *Mini-imagenet*\n\n|      Num-shots >         | **1**    | **5**    | **10**   | **20**   | **50**   | **100**  | **200**  | **500**  |\n|---------------|----------|----------|----------|----------|----------|----------|----------|----------|\n| **err_nccc**  | 0.2210   | 0.0922   | 0.0679   | 0.0651   | 0.0530   | 0.0517   | 0.0488   | 0.0460   |\n| **err_lp**    | 0.2500   | 0.0820   | 0.0540   | 0.0459   | 0.0450   | 0.0500   | 0.0379   | 0.0240   |\n| **err_bound** | 65.0692  | 10.0063  | 5.3739   | 3.0990   | 1.6178   | 1.0365   | 0.6889   | 0.4433   |\n\n\n---\n---\n\n## Response to 8fbK\n\n> **Reviewer:** The experiments show that the gap between DCL and NSCL decreases as the number of classes increases, which aligns well with the theoretical analysis. However, the performance difference between the two methods remains noticeable. More discussion on this observation could help clarify how much of the theoretical alignment translates into practical outcomes.\n\nThe loss‑gap bound in Thm. 1 is label‑agnostic: it shows how DCL incentivizes the embeddings to support any labeling (see lines 155–161), but it does not imply that the learned embedding geometry will match a specific labeling, as NSCL does. In fact, under the unconstrained features model and assuming $d > N$, any global minimizer of DCL forms an $N$‑way equiangular tight frame (ETF). This configuration drives directional‑CDNV to 0 even when the full CDNV remains non‑zero.\n\nAs observed in Figs. 5, DCL effectively drives the directional‑CDNV to low values, while Thm. 2 for NSCL implies that both directional and regular CDNVs converge to low values. As predicted by Prop. 1 (and verified in Fig. 4 (bottom)), this difference explains why DCL achieves reasonable downstream performance, but NSCL—by driving both CDNV measures lower—ultimately performs even better.\n\nIn conclusion, we do not expect DCL and NSCL to learn the exact same representations. However, our theory characterizes their relationship, clarifies the types of representations produced by DCL training, and explains why these representations remain effective for downstream tasks using linear probes.\n\n> **Reviewer:** In addition to DCL and NSCL, it would be helpful to include comparisons with standard CL methods. Based on the paper’s analysis, the gap between CL and NSCL is expected to be even larger, which could further support the theoretical claims.\n\nFollowing the reviews, we conducted an experiment to reproduce Fig. 2 (top) for CL-training (see experiment (6) in the response to J3ay). In addition, we evaluated CL, DCL and NSCL and the bound for DCL/CL-trained models and observed negligible differences in the values of CL and DCL (see experiments (1), (2) and (6) in the responses to 8fbK and J3ay). \n\nTheoretically, CL and DCL differ only by removing the anchor sample from the denominator, and the proof of Thm. 1 applies to both. Specifically:\n$$\n\\mathcal L_{\\text{NSCL}} \\le \\mathcal L_{\\text{DCL}} \\le \\mathcal L_{\\text{CL}}\n\\quad\\text{and}\\quad\n\\mathcal L_{\\text{CL}}-\\mathcal L_{\\text{NSCL}}\n\\le \\log\\Bigl(1+\\tfrac{e^{2}}{C-1}\\Bigr).\n$$\nThis implies\n$$\n0 \\le \\mathcal L_{\\text{DCL}}-\\mathcal L_{\\text{NSCL}}\n\\le \\mathcal L_{\\text{CL}}-\\mathcal L_{\\text{NSCL}}\n\\le \\log\\Bigl(1+\\tfrac{e^{2}}{C-1}\\Bigr).\n$$\n\n> **Reviewer:** Can the authors quantify the similarity of the learned representations—not just the loss values—between CL/DCL and NSCL-trained models? For example, using clustering metrics and others.\n\nWe conducted new experiments to quantify the representation similarity between CL/DCL- and NSCL-trained ViT models (see experiment (7) in the response to J3ay).\n\n> **Reviewer:** While the paper presents clear analysis and supporting experiments, further evaluation on a wider range of models and larger-scale datasets would be helpful. This could clarify how well the findings generalize beyond the current experimental setting.\n\nThank you for the suggestion. Following the reviews, we conducted extensive experiments with the ViT architecture, as well as the ImageNet-1k (1000 classes) and Tiny-ImageNet (200 classes) datasets (see experiments (1)-(5) in the responses to 8fbK and J3ay)."}, {"review_id": "WYcUI1qXGS", "reviewer": "Reviewer_WmN5", "summary": "In this paper, the author aims to establish a connection between contrastive learning and supervised learning. Through theoretical analysis, the author demonstrates how self-supervised contrastive learning can be mapped to a supervised contrastive loss that uses only negative samples. Both the upper and lower bounds of the proposed model are provided. Experiments involving few-shot classification show that as the number of classes increases, the behavior of the two paradigms becomes increasingly similar.", "questions": "Please refer to the weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and easy to follow.\n\n- The two theorems proposed in the paper are mathematically well-formulated and rigorously proved.\n\n- The experimental design effectively validates the theoretical claims made by the authors.\n\nWeakness:\n- The experiment conducted by the author is quite small in scale, as CIFAR-10, CIFAR-100, and mini-ImageNet each contain only 50,000 images. The results could be easily influenced by overfitting, especially in the presence of noisy samples. I am curious whether this phenomenon still holds when the number of samples per class increases or when the data becomes noisier. Nowadays, pretraining typically requires massive and often noisy datasets. Further experiments should be conducted to verify the author’s claims.\n\n- The central claim that standard contrastive learning approximates a supervised variant under a growing number of classes is conceptually intuitive and has been suggested in prior works. While Theorem 1 formalizes this intuition with a clean bound, the assumption that the number of classes tends to infinity makes the result less surprising and arguably less practical. The paper ultimately confirms that contrastive objectives yield clustering behavior similar to supervised objectives, but this is already an implicit assumption in much of the literature. Thus, the theoretical contribution, though correct, feels incremental rather than insightful.\n\n- This paper does not provide practical guidance for designing CL systems. For example, it does not address important  questions such as how augmentation affects learning, the robustness of  the theoretical results under incomplete augmentation, or whether CL can outperform supervised learning in all cases. Although the authors  introduce an upper bound on the few-shot error, it relies on quantities that are difficult to explicitly control during learning (e.g.,  class-wise variance) and has limited practical impact on accuracy.  Moreover, minimizing loss does not directly translate into subsequent accuracy improvements, weakening the practical relevance of the results.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns. The method proposed in this paper offers theoretical insights into contrastive learning and presents a potential framework for evaluating and guiding pretraining strategies. I believe these contributions can benefit the community and support the development of more effective self-supervised learning methods. Therefore, I consider this paper to be above the acceptance threshold.", "rebuttal": "**Note:** We thank the reviewer for their very thoughtful feedback and comments. Following the reviewers’ suggestions, we conducted multiple experiments to strengthen our work. Since there is no dedicated common space for experiments, **we present them at the beginning of the responses to reviewers 8fbK (part 1) and J3ay (part 2)**.\n\n---\n---\n\n## Response to WmN5\n\n> **Reviewer:** The experiment conducted by the author is quite small in scale, as CIFAR-10, CIFAR-100, and mini-ImageNet each contain only 50,000 images. The results could be easily influenced by overfitting, especially in the presence of noisy samples. I am curious whether this phenomenon still holds when the number of samples per class increases or when the data becomes noisier. Nowadays, pretraining typically requires massive and often noisy datasets. Further experiments should be conducted to verify the author’s claims.\n\nThanks for raising this point. We believe that the reported results are not due to overfitting: \n\n1. We conducted a wide range of experiments across multiple datasets, and as shown in Tab. 1 and Fig.  4, DCL‑trained models transfer well to downstream tasks with simple adaptation methods (e.g., LP, NCCC). \n\n2. Beyond CIFAR and mini‑ImageNet, we also used SVHN, which contains 600 k images, and observed the same trends (Fig. 1).\n\n3. Following the reviewer’s suggestion to scale up, we added experiments on Tiny‑ImageNet (200 classes) with ViT backbone (see experiment (2) in the response to 8fbK) and ImageNet‑1k (1000 classes) with ResNet50 backbone (see experiment (1) in the response to 8fbK). These datasets not only have a significantly larger number of classes but also naturally contain more label noise and sample‑level ambiguity than CIFAR‑like datasets. \n\nWe would also be happy to explore specific experiments with noisier data if the reviewer has a concrete suggestion.\n\n> **Reviewer:** The central claim that standard contrastive learning approximates a supervised variant under a growing number of classes is conceptually intuitive and has been suggested in prior works. While Theorem 1 formalizes this intuition with a clean bound, the assumption that the number of classes tends to infinity makes the result less surprising and arguably less practical. The paper ultimately confirms that contrastive objectives yield clustering behavior similar to supervised objectives, but this is already an implicit assumption in much of the literature. Thus, the theoretical contribution, though correct, feels incremental rather than insightful.\n\nWe appreciate the reviewer’s perspective and the opportunity to clarify the novelty and practical relevance of our results.\n\n1. Incremental contribution: We agree that it is intuitive to view self-supervised learning (SSL) and supervised learning (SL) as closely related, and we acknowledge that the literature often assumes clustering properties. However, these are very strong and largely unsubstantiated assumptions. **Our work provides concrete theoretical guarantees that explicitly establish the relationship between SSL and SL and explain not only why clustering arises, but also what kind of clustering is expected.**\n\nThm. 1 provides a label‑agnostic bound that directly relates CL and SL. It informs us what kind of supervised loss is most related to CL (e.g., NSCL), it holds for all possible labelings simultaneously and does not rely on data‑distributional, augmentation, or architectural assumptions, is substantially more general than previous heuristic arguments (see lines 80–91) that depend on strong, often unrealistic conditions. In response to reviewer feedback, we further substantiated this result by showing in (experiment (7) in the response to J3ay) that the learned representations from DCL and NSCL are highly aligned.\n\nProp. 1 and its associated experiments make an equally important contribution: they (i) characterize the specific clustering properties that hold in practice for CL‑trained models, and (ii) demonstrate why these properties benefit downstream tasks. Since clustering is often *assumed* rather than analyzed in the literature, these results fill a crucial gap in our understanding of SSL.\n\n2. Beyond the $C \\to \\infty$ regime: While the gap in Thm. 1 tightens as the number of classes increases, the theorem does not require $C \\to \\infty$. As shown in Figs. 2-3, the bound is already tight and correlates strongly with the observed loss gap for moderate class counts (e.g., 10–100 classes in CIFAR‑10/100 and mini‑ImageNet).\n\nIn summary, our contributions (i) formalize a widely cited but previously speculated intuition, (ii) validate it in realistic settings, and (iii) link it directly to downstream performance. We therefore believe the work goes well beyond incremental progress, offering genuinely new theoretical and empirical insights into why SSL methods succeed.\n\n> **Reviewer:** This paper does not provide practical guidance for designing CL systems. For example, it does not address important questions such as how augmentation affects learning, the robustness of the theoretical results under incomplete augmentation, or whether CL can outperform supervised learning in all cases. Although the authors introduce an upper bound on the few-shot error, it relies on quantities that are difficult to explicitly control during learning (e.g., class-wise variance) and has limited practical impact on accuracy. Moreover, minimizing loss does not directly translate into subsequent accuracy improvements, weakening the practical relevance of the results.\n\nWe thank the reviewer for raising important questions about the practical implications of our work. While our primary focus is not on proposing new practical training techniques, our work provides a foundational framework for explaining why existing self-supervised methods, such as CL and DCL, are theoretically sound, what aspects make them so successful, and what kinds of representations they capture. While these insights may not directly tell practitioners how to choose training hyperparameters, they offer a more informed way to analyze and ultimately improve CL systems in practice.\n\n* **Relationship with supervised learning:** There is a growing intuition that self-supervised learning and supervised learning are closely related, but it is only through the bound in Theorem 1 and the discussion in lines 155–161 that we identify the mechanism behind this relationship. Our bound quantifies the gap between SSL and supervised learning and clarifies which supervised method (e.g., NSCL) is most closely related to CL/DCL training. In fact, our new empirical results (experiment (7) in the response to J3ay) verify that by showing a very strong alignment between the learned embeddings of CL/DCL and NSCL.\n\n* **Practical impact of the few-shot error bound:** The reviewer is correct that our bound depends on quantities not directly controlled during training and that it is not a direct estimator of accuracy. However, we argue this is a feature, not a limitation, because its purpose is to explain rather than serve as a new training objective.\n\nA useful analogy is the bias–variance tradeoff. The formula decomposing error into bias, variance, and irreducible error is one of the most important concepts for practitioners. Although one cannot directly control bias or variance during a single training run, the decomposition provides a powerful framework for diagnosing failures (e.g., high variance implies overfitting). This understanding then guides practical decisions such as increasing regularization or gathering more data.\n\nSimilarly, our bound in Prop. 1 is an analytical tool for understanding contrastive learning. Its primary value lies not in the final number it produces but in how it decomposes error and introduces new vocabulary for analyzing performance. Specifically, it distinguishes between two different quantities: within-class variance (CDNV) and the variance along the directions separating class centers (**directional CDNV**). This analytical lens leads to a core finding presented in Fig. 5: successful CL training should significantly reduce directional CDNV while maintaining a relatively stable CDNV. This is a non-obvious insight that explains how label-agnostic methods learn linearly separable features."}, {"review_id": "Uc2uRqeAXu", "reviewer": "Reviewer_8nSm", "summary": "The authors provide theoretical foundations for self-supervised contrastive learning (CL) by investigating the gap between CL loss and a supervised negatives-only supervised contrastive loss (NSCL). The loss gap is proved to vanish as the number of classes increases. The paper also discusses that the NSCL-trained representations achieve accurate linear probing performance because the perfectly clustered representations lead to zero CDNV and directional CDNV. Experiments on benchmark image datasets are conducted to validate the theoretical findings.", "questions": "1. Figure 3 shows that the gap slightly increases over training. Is there any explanation?\n2. In Figure 5, the CDNV of DCL increases at the end of training. Is there any explanation?", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength.**\n\n1. The paper provides theoretical foundations for self-supervised contrastive learning from a new perspective.\n2. The theoretical claims are validated through experiments.\n\n**Weakness.**\n1. There is a mismatch between the main theorem and its proof. Theorem 1 states that the bound is for DCL, but according to its proof, the bound should be for CL. Also, the claims in Sections 3 and 4 are for DCL, whereas in the abstract, the gap is claimed to be between standard CL and NSCL losses.\n2. Several related theoretical works are not mentioned, e.g., https://openreview.net/forum?id=XDJwuEYHhme, https://openreview.net/forum?id=c5K7gcd2Rw, https://openreview.net/forum?id=cN9GkuBo4w. The authors should discuss the connections and differences compared with these papers.\n3. The concept of directional CDNV needs intuitive explanations. Some examples could help.\n4. The theoretical claims show that DCL approximates NSCL when the number of classes is large, and that NSCL leads to perfectly clustered representations and accordingly good few-shot classification performance. However, the theoretical discussions fail to address why DCL leads to good representations. Specifically, do the theoretical claims mean that when the number of classes $C \\to \\infty$, the DCL representations are perfectly clustered? This doesn't seem to be right under my knowledge.\n5. The experimental validation of diminishing loss gap is weak (Fig. 2). It only verifies $C=10$ and $C=100$. More results on Tinyimagenet (C=200) or Imagenet1k (C=1000) could further strengthen this claim.", "was_revised": true, "final_justification": "The rebuttal solved my concerns and I have raised my recommendation score to 4. However, I cannot give a higher rating because this paper investigates the geometry of the representation learned by NSCL instead of CL or DCL, and despite the loss values approximating each other, the behaviors of their representations are not alike.", "rebuttal": "**Note:** We thank the reviewer for their very thoughtful feedback and comments. Following the reviewers’ suggestions, we conducted multiple experiments to strengthen our work. Since there is no dedicated common space for experiments, **we present them at the beginning of the responses to reviewers 8fbK (part 1) and J3ay (part 2)**.\n\n---\n---\n\n## Response to 8nSm\n\n> **Reviewer:** There is a mismatch between the main theorem and its proof. Theorem 1 states that the bound is for DCL, but according to its proof, the bound should be for CL. Also, the claims in Sections 3 and 4 are for DCL, whereas in the abstract, the gap is claimed to be between standard CL and NSCL losses.\n\n1. The proof actually covers both objectives; we stated Thm. 1 in terms of DCL only to keep notation light. The proof in fact covers both objectives, as shown by the chain of inequalities (that appear in the proof):\n$$\n\\mathcal L_{\\text{NSCL}} \\le \\mathcal L_{\\text{DCL}} \\le \\mathcal L_{\\text{CL}}\n\\quad\\text{and}\\quad\n\\mathcal L_{\\text{CL}}-\\mathcal L_{\\text{NSCL}} \\le \\log\\bigl(1+\\tfrac{e^{2}}{C-1}\\bigr),\n$$\nwhich yields\n$$\n0 \\le \\mathcal L_{\\text{DCL}}-\\mathcal L_{\\text{NSCL}}\n\\le \\mathcal L_{\\text{CL}}-\\mathcal L_{\\text{NSCL}}\n\\le \\log\\bigl(1+\\tfrac{e^{2}}{C-1}\\bigr),\n$$\nThus the same bound holds for both CL and DCL. \n\n2. We focused on DCL in the main text since it is an improved variant of the original CL objective (as reported in the original DCL paper). The abstract mentions standard CL because that term is more familiar and one can think of CL as a family of algorithms rather than necessarily a very specific implementation; we will revise it to read \"(decoupled) contrastive loss\" and add a sentence clarifying that Thm. 1 applies to both CL and DCL. We will also update the theorem statement and proof header to make this dual coverage more explicit.\n\n3. Empirically, we conducted an experiment to reproduce Fig. 2 (top) for CL-training, showing the same behaviour as before (see experiment (6) in the response to J3ay). In addition, we evaluated CL, DCL and NSCL and the bound for DCL-trained models and observed negligible differences in the values of CL and DCL (see experiments (1) and (2) in the response to 8fbK). \n\n> **Reviewer:** Several related theoretical works are not mentioned, e.g., https://openreview.net/forum?id=XDJwuEYHhme, https://openreview.net/forum?id=c5K7gcd2Rw, https://openreview.net/forum?id=cN9GkuBo4w. The authors should discuss the connections and differences compared with these papers.\n\nWe appreciate the reviewer highlighting these additional works. In Sec. 2 (lines 80–91) we already review 13 related SSL theory papers, but we are happy to add the suggested works and clarify their differences from ours.\n\nBoth papers derive bounds on supervised terms (classification error in the first, supervised loss in the second), but just like the works mentioned in lines 80-91, their analyses rely on very strong assumptions that we avoid:\n\n(1) Thms. 1 and 6 in the first paper rely on Definition 1, the $(\\varepsilon,\\delta)$-augmentation condition. It effectively assumes that, for almost every pair of images in the same semantic class, there exists a sequence of the model’s augmentations that brings their augmented versions arbitrarily close. For a class such as “cat images”, that would mean crops, colour‑jitter, blurs, etc’, can morph one cat image into another, a claim that is clearly idealized and rarely satisfied in practice.\n\n(2) The bounds in the second paper (Thms. 2.1, 2.3, 2.4) depend on terms such as:\n$$\n\\mathbb{E}\\_{c}\\mathbb{E}\\_{\\bar{x},\\bar{x}’ \\sim \\rho\\_{c}}\\mathbb{E}\\_{a}\\min\\_{a’}|| f(a(\\bar{x})) - f(a’(\\bar{x}’))||\n$$ \nand \n$$\n\\mathbb{E}\\_{c}\\mathbb{E}\\_{\\bar{x}’ \\sim \\rho\\_{c}}\\max\\_{a,a’}|| f(a(\\bar{x}’)) - f(a’(\\bar{x}’))||\n$$\nwhich together essentially require embeddings of augmented samples from the same class to be close. While the second term can be somewhat controlled through optimization, the first term is far more restrictive: (i) it implicitly depends on class labels (which are unavailable to SSL algorithms) and (ii) it presupposes intra-class consistency across augmentations, a property that may not hold in practice. In effect, the analysis builds in the very property it seeks to prove (low intra-class variation in embeddings) in order to derive a bound on the supervised loss.\n\n> **Reviewer:** The concept of directional CDNV needs intuitive explanations. Some examples could help.\n\nThank you for the suggestion. In order to illustrate the CDNV and directional CDNV (and to understand their difference), we created two plots for each one of those. The first plot includes two classes in 2‑D. We illustrate how CDNV measures the total spread of each class relative to the distance between their means—visualised as the radius of two equal‑radius \"balls\" around the class centres. For the directional CDNV, we keep the same denominator (inter‑centre distance) but now measure the spread only along the line that joins the two centres. In the figure this is visualized as the variance of the 1‑D projections of the samples onto that connecting line.\n\nThis visual makes it clear why directional CDNV can be much smaller than CDNV—e.g., if the clouds are elongated orthogonally to the centre‑to‑centre axis, yet still dominates few‑shot error (Prop. 1). We will include these plots with an explanation in the final version.\n\n> **Reviewer:** The theoretical claims show that DCL approximates NSCL when the number of classes is large, and that NSCL leads to perfectly clustered representations and accordingly good few-shot classification performance. However, the theoretical discussions fail to address why DCL leads to good representations. Specifically, do the theoretical claims mean that when the number of classes $C \\to \\infty$, the DCL representations are perfectly clustered? This doesn't seem to be right under my knowledge.\n\nTo clarify, we do not argue that DCL leads to perfectly clustered representations as $C \\to \\infty$. When $C \\to \\infty$ then we are able to perfectly minimize the NSCL loss (which leads to perfect clustering) only if the embedding dimension $d \\to \\infty$. Thus, in any realistic setting where $d$ is finite we will always have a noticeable gap between the two losses. Instead, we show that the DCL and NSCL losses are close which does not directly lead to the same level of clustering in the representations. In fact, similar to the proof of Thm. 2, it is possible to show that the global minimizer of DCL (which is only possible when the embedding dimension $d$ exceeds the number of samples $N$), the embeddings of training samples form an equiangular tight frame (ETF) of $N$ vectors. In that configuration, the directional‑CDNV is exactly zero (and the CDNV to be proportional to the number of samples per class), which is a weaker notion of clustering than the one given by minimizing the NSCL loss.\n\n> **Reviewer:** The experimental validation of diminishing loss gap is weak (Fig. 2). It only verifies $C=10$ and $C=100$. More results on Tinyimagenet (C=200) or Imagenet1k (C=1000) could further strengthen this claim.\n\nFollowing the reviews, we conducted additional experiments on both ImageNet-1k and TinyImageNet, as well as further experiments with the ViT architecture (see experiments (1)-(5) in the responses to 8fbK and J3ay).\n\n> **Reviewer:** Figure 3 shows that the gap slightly increases over training. Is there any explanation?\n\nWe can express the loss difference as\n$$\nL_{\\text{DCL}} - L_{\\text{NSCL}} = \\log\\left(1 + \\frac{A}{B}\\right),\n$$\nwhere\n* $A = \\sum_{j \\neq i, y_j = y_i} e^{\\text{sim}(z_i, z_j)}$ is the sum over same‑class negatives, and\n* $B = \\sum_{y_j \\neq y_i} e^{\\text{sim}(z_i, z_j)}$ is the sum over all cross‑class negatives.\nEarly in training, similarities between all samples (same or different class) are roughly equal, so $A/B$ is small—dominated by the fact that there are far fewer same‑class terms than cross‑class ones. As a result, the loss gap is negligible.\n\nAs training progresses, same‑class samples become more similar while cross‑class similarities decrease. This increases $A$ relative to $B$, so the gap $\\log(1 + A/B)$ grows slightly. The increase remains small because the same‑class terms are still a small fraction of the total. This explains the gradual upward trend observed in Fig. 3.\n\n> **Reviewer:** In Figure 5, the CDNV of DCL increases at the end of training. Is there any explanation?\n\nThanks for bringing this up. Throughout the experiments we have consistently observed a phenomenon we refer to as \"over‑training\" in CL/DCL. At the first stage of training, the directional‑CDNV decays rapidly while the CDNV remains fairly stable (and may even decrease slightly). As training progresses, because the model continues to push samples apart (while still enforcing consistency within augmented views), the CDNV eventually starts to grow again, while the directional‑CDNV continues to decay (or saturates).\n\nSimilar to the proof of Thm. 2, it is possible to show that if the embedding dimension $d$ exceeds the number of samples $N$ (an unrealistic but illustrative case), the global minimizer of CL/DCL organizes the embeddings of training samples as an equiangular tight frame (ETF) of $N$ vectors. In that configuration, the directional‑CDNV is exactly zero but the CDNV is proportional to the number of training samples per class.\n\nWe will add discussion of this behavior and its origins in the next version of the paper."}, {"review_id": "9xo8AG6GkC", "reviewer": "Reviewer_J3ay", "summary": "This paper studies a duality between contrastive learning (CL, self-supervised) and negatives-only supervised contrastive learning (NSCL, where samples belonging to the same class as the positive sample are removed from the denominator). \nFirst, it shows that the standard contrastive learning approximates the NSCL and that the gap between loss values of CL and NSCL decreases with the number of classes with $\\frac{1}{C-1}$ (for a balanced dataset).\nNext, it introduces a new bound on few-shot error of linear probing. Compared to previous work, this bound depends not only on class-distance-normalized variance (CDNV) between classes, but also on its *directinal* variant, i.e., on *directional* CDNV, which measures the variation along the line between centers of the respective classes. Moreover, the authors show that this newly introduced directional CDNV dominates the bound on few-shot error of linear probing.\nFinally, the findings presented in this paper are empirically validated on four datasets (CIFAR10, CIFAR100, mini-ImageNet, SVHN) using SimCLR or MoCov2 algorithm (with DCL loss).", "questions": "- How do you define the \"same point\" in the augmentation collapse on L66? Is it some small neighborhood?\n- What does \"upper bound\" in Fig. 2 (top) represent?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is well written with a clear structure and story.\n- The findings reported in this paper are valuable in the steps towards a better theoretical understanding of contrastive learning.\n- Good visualizations and plots that support the findings.\n- The findings are well documented and commented on. The notation of the equations is clear. \n- The reported contributions are empirically validated on four datasets using either SimCLR or MoCov2 algorithm with a nice level of detail given about the experiments.\n\n**Weaknesses**\n- The closseness of the two loss values (for DCL and NSCL) does not directly imply the same behavior and structure of the learned representation. I am grateful that this is also commented on by the authors.\n- (minor) Sometimes, the generally-used variables (e.g., *K* on L220) are introduced earlier in the text and it can be a bit confusing.\n- Experiments on a larger dataset (e.g., ImageNet) are missing.\n- It is not explained what \"upper bound\" in Fig 2 (top) stands for?\n- Claims on L298-299 are not discussed. What can be the reason for the gap between the losses to increase?", "was_revised": true, "final_justification": "I've read the rebuttal and appreciate the insights and new experiments you presented. It helped me to understand the paper better and answered my questions. I see it important to include these in the final version of the paper, also with the discussion of why the gap in Figure 3 slightly increases (this was a shared question with another reviewer).\n\nI maintain my original rating of 5 - Accept.", "rebuttal": "**Note:** We thank the reviewer for their very thoughtful feedback and comments. Following the reviewers’ suggestions, we conducted multiple experiments to strengthen our work. Since there is no dedicated common space for experiments, we **present them at the beginning of the responses to reviewers 8fbK (part 1) and J3ay (part 2)**.\n\n---\n---\n\n## Summary of experiments (part 2)\n\n**Experiment (5)** We reproduced the experiment in Fig. 5 with the **ViT architecture** on **CIFAR10, CIFAR100, mini-ImageNet and Tiny-ImageNet**.\n\n### *CIFAR10*\n\n\n|       **Epochs >**            | **0**   | **10**  | **100** | **1000** | **2000** |\n|-------------------|---------|---------|---------|----------|----------|\n| **DCL CDNV**      | 11.39   | 2.90    | 1.67    | 1.42     | 2.00     |\n| **NSCL CDNV**     | 10.31   | 2.38    | 0.43    | 0.06     | 0.05     |\n| **DCL Dir‑CDNV**  | 1.88    | 0.27    | 0.10    | 0.05     | 0.05     |\n| **NSCL Dir‑CDNV** | 1.99    | 0.25    | 0.05    | 0.02     | 0.02     |\n\n### *CIFAR100*\n\n\n|       **Epochs >**            | **0**   | **10**  | **100** | **1000** | **2000** |\n|-------------------|---------|---------|---------|----------|----------|\n| **DCL CDNV**      | 6.90    | 3.85    | 3.35    | 3.015    | 3.25     |\n| **NSCL CDNV**     | 6.14    | 3.91    | 2.73    | 1.24     | 0.83     |\n| **DCL Dir‑CDNV**  | 1.23    | 0.14    | 0.07    | 0.06     | 0.06     |\n| **NSCL Dir‑CDNV** | 1.01    | 0.15    | 0.07    | 0.06     | 0.05     |\n\n### *Mini-imagenet*\n\n\n|       **Epochs >**            | **0**   | **10**  | **100** | **1000** | **2000** |\n|-------------------|---------|---------|---------|----------|----------|\n| **DCL CDNV**      | 5.78    | 2.75    | 2.51    | 1.69     | 1.63     |\n| **NSCL CDNV**     | 6.12    | 2.73    | 2.34    | 1.06     | 0.64     |\n| **DCL Dir‑CDNV**  | 1.03    | 0.20    | 0.09    | 0.05     | 0.04     |\n| **NSCL Dir‑CDNV** | 1.04    | 0.20    | 0.09    | 0.04     | 0.04     |\n\n### *Tiny-Imagenet*\n\n| **Epochs >** | **0** | **10** | **100** | **500** |\n| ----------------- | ----- | ------ | ------- | ------- |\n| **DCL CDNV** | 6.49 | 4.06 | 3.69 | 3.08 |\n| **NSCL CDNV** | 6.29 | 4.13 | 3.47 | 2.58 |\n| **DCL Dir-CDNV** | 1.03 | 0.14 | 0.08 | 0.06 |\n| **NSCL Dir-CDNV** | 0.87 | 0.14 | 0.08 | 0.06 |\n\n**Experiment (6)** We reproduced the experiment in Fig. 2 (top) with a ResNet-50 trained with the CL loss instead of the DCL loss:\n### **CIFAR10**\n| Epochs | CL     | DCL    | NSCL   | Upper Bound |\n|--------|--------|--------|--------|-------------|\n| 0      | 8.2898 | 8.2895 | 8.1843 | 8.7837      |\n| 10     | 7.8307 | 7.8303 | 7.7079 | 8.3073      |\n| 100    | 7.6074 | 7.6068 | 7.4649 | 8.0643      |\n| 1000   | 7.5065 | 7.5060 | 7.3545 | 7.9539      |\n### **Mini-imagenet (C=100)**\n| Epochs | CL     | DCL    | NSCL   | Upper Bound |\n|--------|--------|--------|--------|-------------|\n| 0      | 7.3899 | 7.3892 | 7.3794 | 7.4515      |\n| 10     | 6.7831 | 6.7819 | 6.7703 | 6.8422      |\n| 100    | 6.5832 | 6.5817 | 6.5693 | 6.6413      |\n| 1000   | 6.5592 | 6.5578 | 6.5451 | 6.6171      |\n\n**Experiment (7)** We trained pairs of CL or DCL and NSCL-trained models starting from the same initialization, trained using the same batches at each iteration. In order to evaluate the similarity/alignment between the pairs of models we used the Centered Kernel Alignment (CKA) (https://proceedings.mlr.press/v97/kornblith19a/kornblith19a.pdf) and Representation Similarity Analysis (RSA) (https://www.frontiersin.org/journals/systems-neuroscience/articles/10.3389/neuro.06.004.2008/full) metrics (0 = no alignment, 1 = perfect alignment). As can be seen, these paired models achieved very high CKA and RSA values, demonstrating that the learned representations are strongly aligned across the two training objectives.\n\n|                | **CIFAR10** |        | **CIFAR100** |        | **tiny-ImageNet** |        |\n|----------------|-------------|--------|--------------|--------|-------------------|--------|\n|                | **RSA**     | **CKA**| **RSA**      | **CKA**| **RSA**           | **CKA**|\n| **ViT (DCL)**  | 0.8599        | 0.8202   | 0.9073       | 0.9099 | 0.8918            | 0.8945 |\n| **ResNet50 (CL)** | 0.8338   | 0.8102 | 0.9104         | 0.9113  | 0.8864            | 0.8974 |\n\n**Experiment (8)** We verified augmentation collapse by training DCL ViT models and measuring cosine similarity (0 = no collapse, 1 = perfect collapse). We measured the averaged cosine similarity between the embeddings of augmentations of the same sample and the cosine similarity between augmentations of different samples (over the full dataset and 5 augmentation for each pair of samples). Same‑sample augmentations averaged ~0.90, while different‑sample augmentations averaged < 0.15, confirming highly aligned embeddings of augmentations of the same sample.\n\n| **CIFAR10 (same)** | **CIFAR10 (different)** | **CIFAR100 (same)** | **CIFAR100 (different)** | **mini-ImageNet (same)** | **mini-ImageNet (different)** |\n|------------------|-----------------------|-------------------|------------------------|------------------------|---------------------------|\n| 0.9155             | 0.1321                  | 0.9091              | 0.1077                   | 0.8875                   | 0.0650                     |\n\n---\n---\n\n## Response to J3ay\n\n> **Reviewer:** The closseness of the two loss values (for DCL and NSCL) does not directly imply the same behavior and structure of the learned representation. I am grateful that this is also commented on by the authors.\n\nYou are absolutely right: a small difference in objective values does not by itself guarantee identical embedding geometry. The loss‑gap bound is only the first step. To better understand representation‑level similarity we: (i) Introduce directional CDNV (Sec. 3.2): a geometry‑aware variance measure that both DCL and NSCL implicitly drive to low values during training and (ii) link directional CDNV to downstream accuracy (Prop. 1, Sec. 3.2). \n\nTo further substantiate the alignment between CL/DCL‑ and NSCL‑trained models, we ran dedicated experiments quantifying representation similarity. We trained matched CL/DCL and NSCL models from the same initialization with identical batches at each iteration and measured alignment using Centered Kernel Alignment (CKA) and Representational Similarity Analysis (RSA)—both standard embedding‑similarity metrics (ranging between 0–1). As shown in experiment (7) (in the response to J3ay), the paired models achieve very high CKA and RSA values, indicating that their learned representations are strongly aligned.\n\n> **Reviewer:** (minor) Sometimes, the generally-used variables (e.g., K on L220) are introduced earlier in the text and it can be a bit confusing.\n\nThanks for pointing this out. We will revisit these widely used variables later in the text as a reminder.\n\n> **Reviewer:** Experiments on a larger dataset (e.g., ImageNet) are missing.\n\nThank you for the suggestion. To scale-up our experiments, we conducted plenty of experiments with the ViT architecture, as well as the ImageNet-1k (1000 classes) and Tiny-ImageNet (200 classes) datasets. The experiments reproduce Fig. 2, 4 and 5 (see experiments (1)–(5) in the response to J3ay).\n\n> **Reviewer:** It is not explained what \"upper bound\" in Fig 2 (top) stands for?\n\nThe \"upper bound\" in Figure 2 refers to $_{\\text{NSCL}} + \\log(1 + e^2/(C − 1))$, as noted in the caption. We will clarify this further in the next version of the paper.\n\n> **Reviewer:** Claims on L298-299 are not discussed. What can be the reason for the gap between the losses to increase?\n\nWe can express the loss difference as\n$$\nL_{\\text{DCL}} - L_{\\text{NSCL}} = \\log\\left(1 + \\frac{A}{B}\\right),\n$$\nwhere\n* $A = \\sum_{j \\neq i, y_j = y_i} e^{\\text{sim}(z_i, z_j)}$ is the sum over same‑class negatives, and\n* $B = \\sum_{y_j \\neq y_i} e^{\\text{sim}(z_i, z_j)}$ is the sum over all cross‑class negatives.\n\nEarly in training, similarities between all samples (same or different class) are roughly equal, so $A/B$ is small—dominated by the fact that there are far fewer same‑class terms than cross‑class ones. As a result, the loss gap is negligible.\n\nAs training progresses, same‑class samples become more similar while cross‑class similarities decrease. This increases $A$ relative to $B$, so the gap $\\log(1 + A/B)$ grows slightly. The increase remains small because the same‑class terms are still a small fraction of the total. This explains the gradual upward trend observed in Fig. 3.\n\nThe gap is still small because the same‑class negatives make up only a tiny fraction of all negatives, and the label‑agnostic bound keeps the difference tightly controlled.\n\n> **Reviewer:** How do you define the \"same point\" in the augmentation collapse on L66? Is it some small neighborhood?\n\nBy \"same point,\" we mean that for two augmentations $x'$ and $x''$ of the same sample, the embeddings satisfy $f(x')=f(x'')$; i.e., they are exactly equal. This definition arises from our analysis of global minimizers under the unconstrained features model, which is an idealized abstraction of the learning setting. \n\nIn practice, exact equality is not achievable. Instead, augmentations of the same sample tend to produce very similar embeddings. Empirically, we observe high cosine similarity between such pairs: in experiment (8) (in the response to J3ay) the average cosine similarity between augmentations of the same sample was ~0.9, compared to <0.15 for augmentations of different samples (averaged over 5 augmentations for each pair of samples)."}]

Author rebuttal:

Meta-review: This paper tackles the issue of insufficient theoretical foundations for self-supervised contrastive learning, in particular decoupled contrastive learning (DCL). This is done by approximating the DCL loss function with NSCL (negatives-only supervised contrastive loss). This approximation is shown to be justified when the number of classes increases. Given this, the geometry of the representation learned by NSCL is studied to demonstrate properties such as augmentation collapse, within-class collapse, etc. The paper then shows convincing empirical evaluation.

Several positives were identified during the review process. For example, the paper provides a new perspective on the theoretical foundations for self-supervised contrastive learning. This was noted by all the reviewers and the AC. Another important aspect is the strong empirical evaluation on multiple datasets.

After the reviewer-author and the reviewers-AC discussion phases, two questions remain regarding this paper: (1) the need to use DCL instead of CL for better clarity, and (2) the fact that the paper analyzes "the geometry of the representation learned by NSCL instead of (D)CL", as noted by reviewer 8nSm. The first point can be addressed relatively easily by the authors (which they agreed to take note of), and while the second point is indeed very relevant, the paper has sufficient merit in presenting a theoretical analysis of self-supervised (decoupled) contrastive learning. Given the overall positive comments from all the reviewers, the recommendation is to accept this paper. The final version should include all the clarifications provided in the rebuttal and the discussions that followed.
Decision: Accept (poster)