On the Alignment Between Supervised and Self-Supervised Contrastive Learning
Authors: Achleshwar Luthra
Priyadarsi Mishra
Tomer Galanti
Venue: ICLR 2026 Poster (2026)
Keywords: Representation alignment
self-supervised learning
contrastive learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Self-supervised contrastive learning (CL) has achieved remarkable empirical success, often producing representations that rival supervised pre-training on downstream tasks. Recent theory explains this by showing that the CL loss closely approximates a supervised surrogate, Negatives-Only Supervised Contrastive Learning (NSCL), as the number of classes grows. Yet this loss-level similarity leaves an open question: {\em Do CL and NSCL also remain aligned at the representation level throughout training, not just in their objectives?}

We address this by analyzing the representation alignment of CL and NSCL models trained under shared randomness (same initialization, batches, and augmentations). First, we show that their induced representations remain similar: specifically, we prove that the similarity matrices of CL and NSCL stay close under realistic conditions. Our bounds provide high-probability guarantees on alignment metrics such as centered kernel alignment (CKA) and representational similarity analysis (RSA), and they clarify how alignment improves with more classes, higher temperatures, and its dependence on batch size. In contrast, we demonstrate that parameter-space coupling is inherently unstable: divergence between CL and NSCL weights can grow exponentially with training time.

Finally, we validate these predictions empirically, showing that CL–NSCL alignment strengthens with scale and temperature, and that NSCL tracks CL more closely than other supervised objectives. This positions NSCL as a principled bridge between self-supervised and supervised learning.

Published as a conference paper at ICLR 2026
ON THE ALIGNMENT BETWEEN SUPERVISED AND SELF-SUPERVISER CONTRASTIVE LEARNING
Achleshwar Luthra
Priyadarsi Mishra Tomer Galanti
Department of Computer Science and Engineering
Texas A&M University
(luthra, pri priyadarsimishra. ga anlanni @tamu..eeee edu
ABSTRACT
Self-supervised contrastive learning (CL) has achieved remarkable empirical suc- cess, often producing representations that rival supervised pre-training on down- stream tasks. Recent theory explains this by showing that the CL loss closely ap- proximates a supervised surrogate, Negatives-Oni. Supervised Contrastive Learn- ing (NSCL) loss, as the number of classes grows. Yet this loss-level similarity leaves an open question: Do CL and NSCL also remain aligned at the representa- tion level throughout training, not just in their objectives?
We address this by analyzing the representation alignment of CL and NSCL models trained under shared randomness (same initialization, batches, and angmentations). First, we show that their induced representations remain similar: specifically, we prove that the similarity matrices of CL and NSCL stay close under realistic condi- tions. Our bounds provide high-pun high-probability guarantees on alignment metrics such centered kernel alignment (CKA) and representational similarity analysis (RSA), and they clarify how alignment improves with more classes, higher temperatures, and its dependence on batch size. In contrast, we demonstrate that parameter-space coupling is inherently unstable: divergence between CL and NSCL weights can grow exponentially with training time. Finally, we validate these predictions empir- ically. showing that CL-NSCL alignment strengthens with scale and temperature, and that NSCL. tracks CL more closely than other supervised objectives. This posi- tions NSCL as principled bridge between self-supervised and supervised learning. Our code is available at dlfundamentals
INTRODUCTION
Self-supervised learning (SSL) has become the dominant approach for extracting transferable repre- sentations from large-scale unlabeled data. By leveraging training signals derived directly from the data, SSL methods avoid costly annotation while producing features that generalize across modalities, from vision (Chen et al., 2020; He et al., 2020; Zbontar et al., 2021; He et al., 2022; Oquab et al., 2024) to language (Gao al., 2021; Reimers & Gurevych, 2019), speech (Schneider et al., 2019; Baevski et al., 2020; Hsu et al., 2021; Baevski et al., 2022), and vision-language (Radford et al., 2021; Jia al., 2021; Zhai et al., 2023; Tschannen al., 2025). Among SSL approaches, contrastive learning (CL) has been particularly successful: methods such as SimCLR (Chen et al., 2020), MoCo (He et al., 2020; Chen al., 2021b), and CPC (van den Oord et al., 2019) train encoders by pulling together augmented views of the same input while pushing apart other samples. This simple principle has yielded state-of-the the performance, often rivaling or surpassing supervised pre-training.
Despite this empirical success, a central puzzle remains: why does CL recover features so well aligned with semantic class boundaries? CL models often support nearly supervised- downstream perfor- mance (Amir et al., 2022; Ben-Shaul et al., 2023; Weng et al., 2025), suggesting that supervision is somehow implicit in the objective. Recent theoretical progress sheds light on this: Luthra al. (2025) showed that the CL objective closely approximates supervised variant, Negatives-Onle Supervised Contrastive Learning (NSCL), where same-class samples are excluded from the denominator, Their analysis established that the CL-NSCL losses converge as the number of classes grows, and further characterized the geometry of NSCL minimizers and their linear probe performance. These results indicate that CL carries supervised-like signal at the loss level.

---

Published as a conference paper at ICLR 2026
Yet this view leaves crucial question unresolved:
Do contrastive and supervised contrastive models remain
aligned throughout training, not just at the level of their objectives?
Loss-level similarity does not guarantee that opti- mization paths coincide. In principle, differences in curvature, gradient noise, or learning rate schedules could amplify small loss discrepancies, causing stochastic gradient descent (SGD) trajecto- ries to diverge. Thus, it remains unclear whether CL merely converges to solution similar to NSCL, whether their parameter and representations remain coupled across training. While some preliminary empirical results (Grigg al., 2021) provide evidence that supervised and self-supervised models learn fairly well-aligned representations geometrically, is not clear to what extent this alignment holds, under what conditions it arises, and what factors control the alignment between the two regimes.
0.00
107
Thanting spochs
(a) Weight Space
(b) Representation Space
(c) SSiilarity Metrics
Figure 1: Comparison of learning dynamics for CL and NSCL models.
(a) Weight space vectors show divergent paths (85.7" 7 apart). (b) contrast, representation space vectors for target class show high alignment (27.8 apart). (c) This is confirmed over training epochs, where representational similarity (CKA, RSA) remains high while the weight gap increases (see figure details in App. B).
Contributions. In this work, we theoretically and systematically study the alignment between CL and NSCL under shared randomness (same initialization, mini-batches, and augmentations):
From drift to metrics. The similarity control yields explicit, high-probability lower bounds on linear CKA and RSA at every epoch, showing that CL. and NSCL. representations remain nontrivially aligned and that the certified alignment tightens C and B grow and as increases (Cors. 1-2). For completeness, we also bound parameter drift under 3-smoothness (Thm. 2), which can grow exponentially even when representations remain aligned.
Conceptual contribution. Our results provide a conceptual framework for what CL optimizes during training. We (i) identify NSCL as the supervised objective whose representations and training trajectories are most tightly coupled to those CL-without claiming that NSCL the strongest supervised baseline in terms of top-1 accuracy and (ii) shift the focus from guarantees on downstream classification accuracy to geometric alignment between supervised and self-supervised representations. Whereas prior work shows that minimizing self- self-supervised losses can yield good downstream classifiers under generative assumptions (e.g., (Arora et al., 2019; Tosh et al., 2021; Saunshi al., 2022; Awasthi al., 2022; HaoChen & Ma, 2023)). our analysis instead characterizes when CL and NSCL induce similar similarity structures, perspective that is particularly relevant for tasks that depend representation geometry, such as interpretability and image segmentation.
Empirical validation. We validate our theory with experiments on RIFAR-10/100, Tiny- Lnn-lmmgeNet, mini- ImageNet,, and ImageNet- K. We find that (i) CL-NSCL alignment strengthens with more classes and higher temperatures as well as correlates with the bound's dependence on batch size; and (ii) NSCL aligns with CL. more strongly than other supervised learning methods (such as cross-entropy minimization and supervised contrastive learning (SCL) (Khosla et al., 2020)).
2 RELATED WORK
A large body of work has sought to explain the success of contrastive learning (CL) from different perspectives. Early accounts linked CL to mutual information maximization between views of the same input (Bachman et al., 2019), though subsequent analyses showed that enforcing mutual information constraints too strongly can degrade downstream performance (McAllester & Stratos, 2020; Tschannen et al., 2020). A different line of work formalizes CL. in terms of alignment and
2

---

Published as a conference paper at ICLR 2026
uniformity properties of the representation space (Wang & Isola, 2020; Wang & Liu, 2021; Chen et al., 2021a), capturing how positives concentrate while negatives spread across the sphere. These geometric criteria, while intuitive, do not fully explain how samples from different semantic classes are organized under CL. training.
To address this, several papers have studied the ability of CL recover latent clusters and semantic structures (Arora et al., 2019; Tosh et al., 2021; Zimmermann et al., 2021; Ash et al., 2022; Nozawa & Sato, 2021; HaoChen et al., 2021; 2022: Shen al., 2022; Wang et al., 2022; Awasthi et al., 2022; Bao et al., 2022). Most of these results rely on restrictive assumptions, such as conditional independence of augmentations given cluster identity (Arora et al., 2019; Tosh et al., 2021; Saunshi al., 2022; Awasthi et al.. 2022). To weaken such assumptions, HaoChen & Ma (2023) proposed analyzing spectral contrastive objectives that encourage cluster preservation without requiring augmentation connectivity, while Parulekar et al. (2023) showed that InfoNCE itself learns cluster-presse cusser--reserving embeddings when the hypothesis class is capacity-limites
Another perspective comes from linking CL to supervised learning. For instance, Balestriero & LeCun (2024) showed that linear models, self-supervised objectives such as VicReg coincide with supervised quadratic losses. In addition, Luthra et al. (2025) established explicit coupling between the InfoNCE contrastive loss and a supervised variant that removes positives from the denominator. In contrast prior results, these bounds are label-agnostic throughout optimization. In a related vein, Lee (2025) formulate self-supervised contrastive learning as approximation to supervised prototype- based objectives, deriving balanced contrastive loss closely related to InfoNCE. On the representation-lewe alignment side, Grigg et al. (2021) provided empirical evidence that supervised and self-supervised trained models learn fairly geometrically aligned representations.
architecture-indepemte
and hold uniformly
Beyond clustering and supervision, other theoretical studies have examined different aspects of CL: feature learning dynamics in linear and shallow nonlinear networks (Tian, 2022; Ji et al., 2023; Wen & Li, 2021: Tian, 2023), the role and optimality of augmentations (Tian et al., 2020; Feigin et al., 2025), the projection head (Gupta et al., 2022; Gui et al., 2023; Xue et al., 2024; Ouyang et al., 2025), sample complexity (Alon et al., 2024), and strategies reduce batch-size requirements (Yuan et al., 2022). Finally, several works explore connections between contrastive and non-contrastive SSL paradigms (Wei et al., 2021; Balestriero & LeCun, 2022; Lee et al., 2021; Garrido al., 2023; Shwartz-Ziv et al., 2023).
3 PROBLEM SETUP
We work with a dataset = (a Wi C X' x [C] (with C classes), where [C] = (l....,C] C} and each class contributes nee examples. Here, N Ee Rh is the total number of samples and let Te Re/N. An encoder fi X + Rd with parameters E Rº maps inputs to embeddings. Similarity is measured by a bounded function sim Rd x Rd - [-1, 1]; throughout our experiments we use cosine similarity on fz-normalized embeddings, sim u, v) = (u, u) ()/([ [[]l]])]))
Data augmentations are modeled by a Markov kernel ax. (x) on X:: given x, we draw an independent view x' ~ axx). Unless stated otherwise, augmentation draws independent across samples, across repeated views of the same sample, and across training steps. We write s/ ~ a(x;) for single view and (x!" (1) xx (2)) iid.
(xx;) for two views of the same input.
Fix a batch size B E N. A batch is multiset B [(r,, s/. f sampled with replacement from S, with independent augmentations zi ~ a(x.). ). For each element in the batch, define 2, := (xx)) and 2/- := fo (xf). For any anchor triple (xi, 2/1, y4) E B, define the per-anchor CL loss and the CL batch loss as
exp(sim(z,, =1))/
Ch (w B) = - -log
exp(sim(zz x))T) + exp(sim(22, 2t)/7)
B B
X3 (w)
(w; B3)
3

---

Published as a conference paper at ICLR 2026
For the same realized batch B, define the negative index set 1, (j fl...., the corresponding negative subset B, = {(z 22j, U/) 1, The NSCL ver-anchor and batch losses are
......,,,, B) yy ye and
exp(sim(z, <1/r)
NNCL (w; (w; = -log
exp(sim(21,2)) 2,)/T) exp(sim(2s, (3))/1)))))))))))))))))))) 23)/T))
Jer
B
ENSCL
88
(w)
NSCL
(w; B,
B
Prior work (Luthra et al., 2025) shows that the CL-NSCL loss gap is uniformly O(1/C), but what we ultimately care about whether the embeddings align. To quantity representation similarity we use linear Centered Kernel Alignment (CKA) and Representation Similarity Analysis (RSA) (Kornblith al., 2019; Kriegeskorte et al., 2008) defined on posine-similarity matrices: for N common inputs with embeddings Z { and Z' = {<0) let (()) = cos(2;, 2j) and H = - 111;
linear CKA is
CKA z') = (HE(Z)H, HE(Z)))F
||H [HE
and RSA is the Pearson correlation between the (upper-triangular) off-diagonal entries of the dissimilarity matrices RDM(Z) = Σ(Z) and RDM(Z') 11 E("'):
RSA Z Corr (vec2 (RDM(()), voe2 (RDM(Z')),
where veca stacks the upper-triangull entries (i < column-wise.
This raises the following question: Beyond a small objective gap, does training CL and NSCL actually lead to similar representations (e.g., high CKA/RSA)?
In the spirit of Thm. of Luthra et al. (2025), we prove that when two runs use shared randomness and augmentations), the per-step gradient mismatch is uniformly
(same initialization, bounded (Lem. 7). Similarly, we show that the CL and NSCL similarity matrices remain close throughout training (Thm. 1), which yields explicit CKA/RSA lower bounds (Cors. 1-2).
THEORY
We examine how contrastive learning (CI.) and negatives-only supervised contrastive learning (NSCL) co-evolve when initialized identically and trained with the same mini-batches and augmentations. While one might first attempt to study their trajectories in parameter space, such an approach quickly breaks down: without strong assumptions on the loss landscape (e.g., convexity strong convexity), small reparameterizations can distort distances, and nonconvex dynamics cause parameter drift to grow uncontrollably over time (see App. C). For this reason, we weight-space coupling aside and turn instead to the aspect that directly shapes downstream hehavior-the iitreseenatiins-anallee their alignment in similarity space.
4.1 COUPLING IN REPRESENTATION (SIMILARITY) SPACE
Let Σ, E [[11, N denote the pairwise similarity matrix of a fixed reference set at step (cosine similarity of normalized embeddings; diagonals are 1). We analyze the coupled evolution of the CL and NSCL similarities, ΣCL and Σ ENSCL [-1, N under identical mini-batches and augmentations. This representation-space view is invariant to reparameterization and directly tracks representational geometry.
Surrogate similarity dynamics. To make the analysis explicit, we work with a "similarity- surrogate that updates only those entries touched by the current batch. For a realized mini-batch B; = {(T), 2'y. y5) ) (with z/ a(xx)), let (2)) and B. (NS be the usual InfoNCE-tye losses written as functions of the relevant similarity entries batch-gradient maps
temperature 0). Define the
G.L Vs CCL (ICL),
G NSCCL
NSCL NNNCC V
B. (5
4

---

Published as a conference paper at ICLR 2026
setting all untouched entries to zero. The surrogate updates
E Ex_ G
NSCL 211
E T/e G GNSCL
with shared initialization and shared randomness (same B3 and augmentations).
In App. D we show that these surrogate dynamics faithfully track the similarity evolution induced by parameter-space SGD. Intuitively, for the similarity map [(w) and corresponding batch loss ((w) (either for CL or NSCL), one SGD step W++1 = wy 7)t V a ((wr) induces E(w++1) E(wr) =
-n,P.G RR, G. (T(e)(, P. = J.Jy J. :- 0E/Ow|w.. up to second-order remainder Rg. Under the regularity assumptions (22 I Ly and quadratic Taylor bound on E 5(w++A AAw), together with bounded gradients and learning-rait grate schedule with bounded E, m ()) and E. nž. App. D shows that the induced trajectory E, = E(w;) and the similarity-descent trajectory remain uniformly close. In particular, for small step sizes, sufficiently large batch B, and moderate temperature т, parameter-space SGD moves similarities almost as if performed gradient descent directly in similarity space, so the surrogate dynamics faithfully track the evolution of CL and NSCL representations. We now formalize the coupling bound.
A Additional notation for high-probability factors. Fix a training horizon T E N, confidence
level 8 E (0, 1), and a temperature T > O. For later use, define 6B, = 2B
log
TB
and
8
A T) == 2e2/- (smww (where Tmax = max, Tel, and assume BB. <
C so the
1-Tmax-cn en,
denominator is positive.
Theorem 1 (Similarity-space coupling). Fix E,, 1), and lemperature T > O. Consider the coupled similarity-descent recursions equation 1 for CL and NSCL with shared initialization and shared seti--btthhee/agmetttteee mini-batches and augmentations, for any stepsizes ())))
Then, with probability at least 1 - 8 over the draws of the
||
ENSCLI| LT
I| S
exp 2T
T
7h
An (B;T)
t=0
t=0
The above bound makes explicit how standard CL. design choices control the discrepancy between CL and NSCL in similarity space. In partieular, both the prefactor and the exponential term in equation are monotone in the usual hyperparameters, so that regimes in which CL "behaves like" NSCL correspond precisely to regimes where the right-hand side of equation is small. First, assuming balanced classes, larger number of classes C' reduces the 1 1/C contribution inside A [B; r), hence decreasing the overall bound and shrinking the CL-NSCL gap. Second, increasing the batch size B simultancously reduces the concentration error ε and the factor 1/VB, and also shrinks the coefficient b.ン in the exponential, all of which act to decrease the right-hand side of equation (see Fig. 5(d)). Third, increasing the temperature т reduces the factors _ and + appearing in the prefactor and exponent, again decreasing the upper bound in equation 2. consistent with the empirical trend in Fig. 4 that higher temperatures bring CL closer to NSCL. Finally, smaller learning rates T) (or, more generally, a smaller total step size Σ, ni) reduce both the prefactor A/B Σ. Th and the
exponent exp 2,28 E, m)), so more conservative optimization schedules yield a tighter coupling between CL. and NSCL (see Fig. 5). Overall, Thm. shows that large batches, high temperatures. and small effective step sizes-are are precisely the regimes in which the similarity dynamics of CL and NSCL nearly align.
As final note, the result in Thm. 1 is stated in terms of similarity descent, whereas in practice we use gradient descent on the network's trainable parameters. To obtain an explicit bound on the gap between the CL and NSCL similarity matrices under standard parameter-space stochastic gradient descent, we can combine Thm. with twice the bound cquation 9, applying that bound once to CL and once to NSCL.
From similarity drift to CKA/RSA guarantees. We translate the gigh-probability control on the similarity drift from Thm. 1, into bounds two standard representational metrics.
CKA. Recall from Sec. 3 that linear CKA (Kornblihh et al., 2019) is the normalized Frobenius inner product between centered similarity matrices. H T 11 be the centering projector and
5

---

Published as a conference paper at ICLR 2026
define centered Gram matrices KL (KG
HIG" 1 H and Ky := HET H. The (linear) CKA ,
at step T is CKAT IF
E [0. 1/. Because ||XX||| S || any bound on | K For convenience, introduce the relative deviation
T
D ENSCL F controls
2T PT |KŞ |
Corollary (CKA lower bound). In the setting of Thm. 1. Assume least 6,
| 1 > 0. With probability
exp 2T3B 2i= me)
Et me) (r,o(B:r))
CKAT 2
1- PT 1+ pT
PT
TVB |K
RSA. Recall from Sec. 3 that RSA (Kriegeskorte et al., 2008) is the Pearson correlation between the off-diagonal entries of representational dissimilarity matrices (RDMs). Let M = (2) and define off- diagonal RDM vectors ar, by E R by ar(u, v) 1 (T ") and ( v) = 1 ENSCL (u, v) for u < v. Write p,T > for the empirical standard deviation of the entries of or- The RSA score is the Pearson correlation RSAT= Corr (ar, br). Zeroing the diagonal does not increase Frobenius norms, so ||6T oT l12 I | |ENSCL 27 [[.. will be useful to measure the relative discrepancy
[|br-arll VM ap.r
PT
Corollary (RSA lower bound). In the setting of Thm.. 1. Assume DDT > O. With probability at least 1 - 8.
exp 2T 2t=o nt)
TVB (ET ni) (B;T(B:T))
1
-- 1+
RSAT 2
These results complement the parameter-space analysis. While parameter trajectories may diverge exponentially (in the non-convex setting), the induced similarities-and hence representational metrics such as CKA and RSS-remain tightly controlled by class count, batch size, learning rate, and temperature т. The key quantity is the similarity-nn -mttrxx drift | -- ENSCC which Thm. 1 bounds in two stabilizing ways.
First, the exponential factor is moderated by the マoロ term in the exponent. Unlike parameter space, where the growth rate scales with 8, the "instability rate" in similarity space is only अ, and is therefore negligible for typical batch sizes (c.g., B 102-103).
Second, the prefactor .VB (E. (B,)(B;T) decreases rapidly with batch size and class count (note A (B:T) shrinks with smaller Mmax and grows with smaller through e//r). In practical regimes (C ~ 103, B ~ 103-103), this prefactor is small, making the total Frobenius gap negligible relative to the scale of the similarity matrices.
Together, these effects yield high-probabilitg guarantees CKAr 2 (1- pr) Pr) and RSAT > (1 (1- (1 +T) with pr, Pr, < in realistic conditions. Thus, even if parameters drift, the induced representations evolve in coupled and stable manner-consisterr with empirical findings that CL and NSCL remain closely aligned in practice.
Proof idea. We begin with high-probability batch-composition guarantee (Cor. 3): with prob- ability at least d-d, d, every anchor's denominator contains the expected proportion of negatives up to an éB, fluctuation. This rules out positive-heavy batches that would otherwise cause the NSCL renormalization to deviate substantially from CL. Conditioning on this event, the CL-NSCL batch-gradient gap decomposes into (i) reweighting error, bounded in total variation by & (Lem. 6), and (ii) a stability term from the dependence on the current similarities, controlled by the 2-ンル -ipscchipssss of the batch-grann match-gradient map in Frobenius norm (Lem. 2 at temperature T). Using biock-orthogonality across anchors (Lem. 1), the reweighting contributions combine in quadrature, giving the per-step estimate (Lem. 8),
< 1 (r:T(B:T)) +
| - CNNCL
|
NNSCL
VB
2r2B
Consequently, the similarity drift satisfies the recurrence
|| |>
NSCL 11
1p < (1+
n) Br²B
| - NNSCLL
+ m
Ar.
BB

---

Published as a conference paper at ICLR 2026
10FAR-10
CIFAR-100
Mini-ImageNet
Tiny-ImageNet
NCCC
LP
NCCC
LP
NCCC
LP
NCCC
LP
CL
88.37
90.16
54.62
65.65
60.78
65.30
40.59
44.61
NSCL
94.47
94.09
60.14
68.38
63.92
72.60
40.76
45.79
SCL 94.93
94.67
64.06
69.52
74.78
76.00
48.63
48.73
CE 92.97
93.39
67.35
68.04
75.20
74.00
48.28
52.57
Table 1: Nearest Class-Center Classifier (NCCC) and Linear Probe (LP) test accuracies (%). We report the accuracies against the all-way classification task in each dataset. The models (also used in Fig. 2) were pre-trained on their respective datasets.
where the first term propagates existing enror and the second injects the new discrepancy introduced at step Unrolling this recurrence (discrete Grönwall) yields
EEC
NNSCL LT
F
S exp 2r
Σ T1 (
TVB
Finally, centering contracts Frobenius norms, so this control transfers directly the centered Gram matrices, and applying standard (1 p1pp/(11+)) and (1 (-r)/(1 r1+(1r)) comparisons yields the claimed CKA/RSA lower bounds.
5 EXPERIMENTS
Datasets and augmentations. We experiment with the following standard vision classification datasets CIFAR1O and CIFARI00 (Krizhevsky, 2009), Mini-ImageNet (Vinyals al., 2016), Imny-lma (Han, 2020), and ImageNet- 1K (Deng et al., 2009). (See App. B for details.)
Methods, architectures, and optimizers. For all our experiments, we have followed the Sim- CLR (Chen et al., 2020) algorithm. We use ResNet-50 (He et al., 2016) encoder with width- multiplier factor of The projection head follows a standard two-layer MLP architecture composed of: Linear (2048 2048) + ReLU + Linear (2048 - 128). For cross-entropy training, we attach an additional classification head Linear (128 + C) where the number of classes.
For contrastive learning, use the DCL loss that avoids positive-neeative coupling during train- ing (Yeh et al., 2022). For supervised learning, we use the following variants: Supervised Contrastive Supervised Contrastive Loss (Luthra al., 2025), and
Loss (Khosla et al., 2020), Negatives-Onin Cross-Entropy Loss (Shannon, 1948). To minimize the loss, we adopt the LARS optimizer (You et al., 2017) which has been shown in (Chen al., 2020) be effective for training with large batch sizes. For LARS, we set the momentum to 0.9 and the weight decay to 1c-6. 6 All experiments are carried with batch size of B 1024. The base learning rate is scaled with batch size 0.3 1B/256], following standard practice (Chen et al., 2020). We employ a warm-up phase (Goyal et al., 2017) for the first 10 epochs, followed by cosine learning rate schedule without restarts (Loshchilov Hutter, 2016) for the remaining epochs. All models were trained on single node with one 94 GB NVIDIA H100 GPU.
Evaluation metrics. To quantitatively measure the alignment between the learned representation spaces of different models, we monitor línear CKA and RSA (check Sec. for details) during training. Both CKA and RSA range from to 1, where 1 indicates identical similarity structures. To manage the significant memory requirements N N matrices (Gram matrices for CKA, RDMs for RSA), we use memory-efficient, chunk- wise computation strategy.
5.1 EXPERIMENTAL RESULTS
Alignment analysis function of epochs. To understand how representational similarity evolves, we trained a model with CL objective and monitored its alignment (via CKA/RSA) against supervised models trained with NSCL, CE, and SCL.. We find that NSCL consistently achieves the
7

---

Published as a conference paper at ICLR 2026
100
1.00
1.00
2.00
8
80
104L
154L
155L
3pe
Tran
Treh
0.50
10
10
10
10
10
107
10
13
Pxiiinn
Taning
0.000
0
0.60
R0CL
0.40
5544
0.40
SCCL
1521
0.00
Traning
Taning
Taning
(a) CIFARIO
(b) CIFAR100
(c) M.MiiiIImaane Miti-emaeeNet
(d) Tiny-ImageNe
Figure Alignment during training. We train ResNet-50 models with decoupled CI., SCL,, NSCL, and CE. For the first 1,000 epochs, the Cl-trained model is substantially more aligned with the NSCL-trained model than with the others. However, alignment declines when training continues much longer.
No1000 12
P 52 26660
*88229 4 6
752248924 8
522
88
A 5 3.
*5 0. 0 38
CIFAR-10 10
CIFAR-100
Mini-ImageNett
Tiny-ImageNet
ImageNet- 1k
Figure CL-NSCL alignment (linear CKA) increases with the number of training classes. The heatmaps show the linear CKA between CL and NSCL models. We visualize alignment on the training (top row, green) and test (bottom row, purple) sets. The y-axis indicates the number of classes (N) used for training, and the x-axis represents the training epoch. While alignment is consistently higher for larger N, it also tends to decrease as training progresses for any fixed N.
highest alignment with CL throughout training across multiple datasets compared to CE and SCL (see Fig. 2). For example, after Ik epochs on Tiny-ImageNee ImageNet, the CL-NSCL alignment reaches CKA of 0.87, in contrast to just 0.043 for CL-SCL.
Intuitively, these alignment patterns follow from how each loss shapes representation geometry. All three methods incentivize neural collapse (Papyan et al., 2020; Han et al., 2022; Zhou et al., 2022: Lu & Steinerberger, 2022: Dang al.. 2024: Graf al., 2021: Awasthi et al., 2022; Gill al.. 2023; Kini et al., 2024; Luthra et al., 2025), but differ in how directly and how quickly they drive it. NSCL structurally closest to CL. both attract a single positive toward an anchor and repel negatives, primarily enforcing instance-level discrimination and thus inducing similar geometry. SCL, by contrast, imposes a stronger class-level constraint, explicitly pulling together augmentations of same-class samples and pushing apart different-class samples, which rapidly reduces intra-class variance and forms tight class clusters that depart from CL's instance-level structure. Cross-entropy (CE) lies between these extremes, promoting collapse more indirectly via error minimization with regularization. the self-supervised setting, CL representations need not collapse as tightly as supervised ones, since they are learned without labels. As training enters the 10- -0000hh range,
8

---

Published as a conference paper at ICLR 2026
1.00
1.00
0.5
2 3.80
2 0.00
02
40
20.00
24:25
14:25
0.80
24:25
2
270
301
0.00
216
330
-06
350
200
302
Trining spocns
Taining
Taining
1.00
1.00
1.00
1.00
0.40
8 0.87
8
8
0.75
0.80
1-:31
0.00
0-63
350
200
301
200
200
330
206
200
530
200
200
507
(a) CIFARIO
(b) CIFAR100
(c) Mini-ImageNNt
(d) Tiny-ImageNet
Figure Higher increases the CL.NNSLL alignment. The plots show RSA (top row) and CKA (bottom row) over 300 epochs. We trained CL and NSCL models with varying temperatures
(r E {0.1, 0.5, I.0J) on four datasets. Across all datasets, a higher temperature T 1.0 (shown in purple) evidently results in the highest alignment.
SCL and CE move closer the neural collapse regime, while NSCL continues mimic the CL label-free optimization for a longer duration, producing the evolving alignment dynamics in Fig. 2.
For completeness, along with CKA and RSA, also report downstream performance via Nearest Class Center Classifier (Galanti et al., 2022) and Linear Probe accuracies in Tab. 1.
Validating Thm. 1 as a function of class count. Thm. predicts that using more classes yields stronger CL-NSCL alignment. We test this via C"-way training: for each C' E [2, CJ. we train CL and NSCL on random C'-class subsets for 1,000 epochs (except 100 epochs for IM- IK). As shown in Fig. 3, representation similarity (RSA/CKA) increases with C" across all datasets.
Effect of temperature on alignment. As per Thm. I and Cors. 1-2, CL-NSCL alignment improves with higher values of temperature (T). We empirically verify this claim by training CL and NSCL models for 300 epochs, over three different values of 7 E {0.1, 0.5, 1.0]. Both models- CL and NSCL-are trained with same in each run. As shown in Fig. 4, models trained with 1.0 achieve higher alignment compared to models trained with lower temperatures.
Effect of batch size alignment. Thm. links alignment B depending on how the learning rate scales. To investigate this, we vary 7 with across four 0B and 0.3. Under ((B) scaling, CL-NSCL alignment decreases as grows, matching the theorem's is implication for that scaling; for the other three cases, alignment increases with B, again consistent with the bound under those dependencies (see Fig. 5).
bound that may rise or fall with
033
03/B 256 1
cases: 7 256 1
256
Weight-space coupling. We next study whether the observed alignment between representations of contrastive and supervised models is also reflected directly in their parameters. For this, measure the average weight difference between a contrastive model and two supervised counterparts
liwbe Wi
as follows: E: 0. + where w and w/up are weights corresponding to 1th layer of
self-supervised and supervised models respectively, and || || denotes Frobenius norm. As we show in Fig. for each dataset, we observe a significant divergence in weight space: both supervised models (NSCL and SCL) increasingly separate from the contrastive model as training progresses.
CONCLUSIONS, LIMITATIONS AND FUTURE WORK
Conclusions. We studied the dynamic alignment between contrastive learning (CL) and its super- vised counterpart (NSCL). By analyzing coupled SGD under shared randomness, we showed that while parameter-space trajectories may diverge exponentially, representation-spat dynamics are far more stable: the similarity matrices induced by CL and NSCL remain close throughout training. This

---

Published as a conference paper at ICLR 2026
1.00
1.00
1.003
1.000
3.975
0.902
0.93
0424
0.00
1-2424
0.557
2
20000-
0.00
0850
0.802
0.000
105
203
302
307
100
202
1.000
1.002
1.000
1.000
3.802
6.4
2.242
8000
000
800
0.002
0.73
2--25
0385
0593
0.700
30
105 153
200
102
230
Trariing
107
200
30.
100
201
(a) (B)
(b) O(B)2)
(c) O(B)/)
(d) (11)
Figure 5: Effect of batch size with scaled learning rates. We trained CL., and NSCL models for 300
epochs with varying batch-sizes (B E {256, 512, 1024)). For each experiment, the learning rate nj is
scaled as function of batch-size, as mentioned under each panel. For instance, the results shown in
panel (b) use learning rate of 0.3
200
1.00
1.20
8..00
8.000
0.00
00
0.60
00
0.40
0.00
3.40
0.
Vaining spicts
Taining spochs
Taning dpocts
Taning spacts
(a) CIFARIO
(b) CIFARIO0
(c) M.Miiiimmaa Mini-ImmgeNet
(d) Inyyeee
Figure 6: Weight-space alignment quickly deteriorates. Using the same ResNet-50 instances as in Fig. 2, we plot the average weight gap between CL and the supervised models (NSCL and SCL)
across training epochs. Both supervised variants diverge from the CL model, with SCL showing wider separation.
yields high- probability lower bounds on alignment metries such as CKA and RSA, directly certifying representational coupling. Empirically, our experiments confirmed these trends across datasets and architectures. Together, our results highlight that the implicit supervised signal CL not confined to its loss function but extends throughout the entire optimization trajectory.
Limitations. Our theoretical bounds are structurally informative but not expected to be tight in large-scale or long-horizon regimes. As is common in machine learning theory, the guarantees are conservative worst-case bounds derived from uniform high- ceab-bbbbbbbitty arguments, favoring generality numerical sharpness. Many influential results in optimization and stability theory for deep learning similarly rely on loose worst-case analyses-e.. (Bousquet & Elisseeff, 2002; Hardt et al., 2016; Mou al., 2018; Kuzborskij & Lampert, 2017>-yet still provide useful conceptual guidance. In our setting, without additional structural assumptions (such as stronger curvature or smoothness conditions), one cannot generally expect qualitatively sharper dependence than the scaled exponential factors appearing in Thm. and equation 9, Thus, while in practice the bounds are quite loose, they achieve their intended goal of identifying which parameters govern the CL-NSCL similarity gap and explaining how this gap scales with them.
Future directions. We view our results as a first step toward more refined theory of self suf-suueeeeeed alignment. Future work could (i) derive tighter constants by exploiting data-dependent structure rather than worst-case bounds, and (ii) extend the framework to other SSL paradigms (e.g.. non-contrastive methods). Improving these guarantees while retaining their stability properties would provide an even stronger theoretical bridge between supervised and self-supervised learning.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
We have taken several steps to ensure the reproducibility of our results. All datasets used in this work (CIFAR-10/100, Tiny-ImageNes and MiageemmaeNee) are publicly available, and we describe the data processing and augmentation pipelines in Section 3 and App. B. The theoretical results are supported by detailed proofs in App. C, D. E. where all assumptions are explicitly stated. Experimental details, including architectures, optimizers, hyperparameters, and training schedules, are reported in Section 3, with additional clarifications in the appendix. To facilitate further verification, we provide an anonymous code repository in the supplementary material that contains implementations of the CL, NSCL, and baseline objectives, along with scripts to reproduce all figures and tables in the paper. Together, these resources intended to make both the theoretical and empirical findings fully reproducible.
REFERENCES
Noga Alon, Dmitrii Avdiukhin, Dor Elboim, Orr Fischer, and Grigory Yaroslavtsev. Optimal sample complexity of contrastive learning. In The Twelfth International Conference on Learning Representations, 2024. URL https:/: /ttp::/s///////eveeeeeee
net/ forumrrrueioooo (tt.rummdd-NNUARAWV...
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. ECCVW What is Motion For?, 2022.
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi.
A theoretical analysis of contrastive unsupervised representation learning, 2019. URL https: //aaxiiv.orr//////e//aaa..a...aa
19022 09229.
Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning. In Gustau Camps- Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 7187-7209. PMLR, 28-30 Mar 2022. URL https: / /ttp//eeeccees...eeee/e
mlr. press / /w1.//111//1/11 2sh5s////h22 .htll.
Pranjal Awasthi, Nishanth Dikkala, and Pritish Kamath. Do more negative samples necessarily hurt in contrastive learning? Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1101-1116. PMLR, 17- 23 Jul 2022. URL http:: http:///occceeengce/e/ mlr press / /1/vvv162222 62/ awasthi22b html.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- Associates, Inc., 2019. URL https: pesttss:/pproeee http:///ooccdddeiceoc ngg ies cc/paper- files/ paper/2019/ fil e 2at.3332229aaa11111oo00000000ea0
Alexei Baevski, Henry Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In Kama- lika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1298-1312. PMLR, 17-23 Jul 2022. URL mllr. attpps:/se http:://prcceedinge www.m.rrrsss.....mmmmmmm wwwpssewelvieeee...e.peeppppppws wwwpskaiaaaaaaaaaaaaaaaaaaaaaaaa html.
Randall Balestriero and Yann LeCun. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttps://oppereeeeeeet
net forrforruutt d= S0s57555575

---

Published as a conference paper at ICLR 2026
Randall Balestriero and Yann LeCun. The birth of self supervised learning: A supervised theory. In NeurIPS 2024 Workshop: Superruue Learning- Theory and Practice, 2024. URL https:
//openreview met/forrmmiiii (ot.rrmmmdd-iMAAAAAa
Han Bao, Yoshihiro Nagano, and Kento Nozawa. On the surrogate gap between contrastive and supervised losses. Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1585-1606. PMLR, 17-23 Jul 2022. URL https: : httpp//poceedccceeceoocooo mlr press / v 62 / bbw./2aaeeeeeeeeeeeeeeee html.
do Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse engineering self-supervised learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: opwnreveeeeeeeeeeeeeeeee net/ fott//foumm?mc wwwooorummiiiisio. V-NsVE
Olivier Bousquet and André Elisseeff. Stability and generalization. J. Mach. Learn. Res., 2: 499-526, March 2002. ISSN 1532-4435, doi: 10.1 10.1162/133244307700000000 //doi. org/ 10. 11..1.11...0 /2
URL https:
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020. URL https: / /proceed ngg. mlr. www.ssresssveeeee /assssvlll.. 9/ cchen201 html.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34:1 1834 1845, 2021a.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9640-9649, October 2021b.
Hien Dang, Tho Tran, Tan Nguyen, and Nhat Ho. Neural collapse for cross-entropy class-imbalanced learning with unconstrained relu features model. In Proceedings of the 41st International Confer- ence on Machine Learning, ICML'24. JMLR.org, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 244.-255,200 2009. doi: 10. 20W.CCCVRR.20095206663333
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https: / openneevie....... net/ bet/fozze Eorum? id= wiccbaatbTyyyy
Shlomo Libo Feigin, Maximilian Fleissner, and Debarghya Ghoshdastidar. A theoretical char- acterization of optimal data augmentations in self-supervised learning, 2025. URL https:
//arxiv.org./eoooe...t...... abs awws//4//00 22112241..000 01411.0767..
Tomer Galanti, András György, and Mareus Hutter. On the role of neural collapse in transfer learning. In International Conference on Learning Representations, 2022. URL https: http:///ppeeeeeeeeeeeee net/forune id SwIp-wIBpa4000
Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE:
Simple contrastive learning of sentence embeddings. In Mlarie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methody in Natural Language Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. htRtttss:
Association for Computational Linguisties. doi: 10. 0665/////021.emmlpmmm //aclanthology org/2021. connp-mmaii 552/.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann LeCun. On the duality between contrastive and non-contrastive self-supervised learning. In The Eleventh International Conference on Learning Representations, 2023. URL http::://peeeeediiiee /oopnneviev net/ forum? id=kDRL.91
12

---

Published as a conference paper at ICLR 2026
Jaidev Gill, Vala Vakilian, and Christos Thrampoulidis. Engineering the neural collapse geometry of supervised-comee contraastde loss, 2023. URL https::sp... pttpp:://arxive wwwexiiv..rrgaooooooo abs /2310 00893.
Priya Goyal, Piotr Dollár, Ross Girshick. Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint
2017.
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised con- trastive learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 Proceedings of Machine Learning Research, pp. 3821-3830. PMLR, 24 Jul 2021. graf2la.htm.
URL https: httpt:://ttcceeeeeeeeeee http:///ooceeeieeeee mlr. htt.eeeiiin..m.....ee.eeeeeeeeee www.m...p..........
Tom George Grigg, Dan Busbridge, Jason Ramapuram, and Russ Webb. Do self-supervised and supervised methods learn similar visual representations?, 2021. URL
abs/21l3 00528.
Yu Gui, Cong Ma, and Yiqiao Zhong. Unraveling projection heads in contrastive learning: Insights from expansion and shrinkage, 2023. URL https: arrvv. orggabs//306 03335.
Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. Understanding and improving the role of projection head in self-supervised learning, 2022. URL https: //arxiv. //arxiv.org/abb. org/abs/
2212. 11491.
Liu Han. Tiny imagenet challenge. 2020, 2020. Kaggle.
https: http://ppppeecommmmmmmttiiiooooo
deep- Seep-leernin--tee
X.Y. Han, Vardan Papyan, and David I.. Donoho. Neural collapse under MSE loss: Proximity to and
dynamics on the central path. In International Conference on Learning Representations, 2022. URL https://seeetooc http:://ppeeevvieeee net wtt.w/wwwwe/eetttddddieee...
R3.
Jeff Z. HaoChen and Tengyu Ma. A theoretical study of inductive biases in contrastive learning. In The Eleventh International Conference on Learning Representations, 2023. URL https:
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 5000-5011. Curran Associates, Inc., 2021. URL https:: /ptpe/////ooccccoegeeoeeo neurips cwturips.c/pappeeee wwtppppreeppiiieceeeeee fiwefflllsapeeeee stlles/ppper/222111 file /ttp:///666222226666922222222222
pdf.
Jeff Z. HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: analyzing the linear transferability of contrastive representations related subpopulations. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook,
NY, USA, 2022. Curran Associates Inc. ISBN 978171387100
Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: stability of stochastic gradient descent. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 1.ML'16,pR pp. 1225-1234. JMLR.org, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
13

---

Published as a conference paper at ICLR 2026
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Trans. Audio, Speech and Lang. Proc., 29:3451-3460, October 2021. ISSN 2329-9290. doi: 10.1 109/TAS1..22221111222 org/10 1109/TASLP 2222221222222
URL https: / /doi.
Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, and Linjun Zhang. The power of contrast for feature learning: a theoretical analysis. J. Mach. Learn. Res., 24(1), January 2023. ISSN 1532-4435.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and rision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4904 4916. PMLR, 18-24 Jul 2021. URL https: http:///oceecddeeeeee
press vv399
jia2lb. html.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 18661-18673. Curran Associates, Inc., 2020. URL https: http://://///ooedeen/e/e neurips wwwrips.cc/pappee...p.eee cc/paper...........c.coooeeee..e files/paaprr22200 aal.aacccccco2llllbbabbaaaaaaaaa
Ganesh Ramachandra Kini, Vala Vakilian, Tina Behnia, Jaidev Gill, and Christos Thrampoulidis. Symmetric neural-collapse representations with supervised contrastive loss: The impact of reLU and batching. The Twelfth International Conference on Learning Representations, 2024. URL
.nt/ hwtp:/erteeeteeeeeeeeeeeee (oo.umtid-AyXXIDEVVSS
Simon Korblithh Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3519-3529, PMLR, 09-15 Jun 2019. URL https:// proceedings mlr www............................. www.pppessssveee / htt.//ooorbthhheecm
html.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A. Bandettini. Representational similarity analy- sis: connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:4, November 2008. doi: 10.3388/neuro.006.004.00000
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical re- port, University of Toronto, 2009. URL https:/ http:///wwwccs.oootoooo.oo edu/zkriz/
Learning-featuress 20..
Ilja Kuzborskij and Christoph H. Lampert. Data-dependent stability of stochastic gradient descent. arXiv preprint arXiv:1703.0167 2017.
Byeongchan Lee. Understanding self-supervised contrastive learning through supervised objec-
tives. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL https:
//openreview.co net/ 2tteeddd=mm7775000
Jason D Lee, Qi Lei, Nikunj Saunshi, and JIACHENG ZHUO. Predicting what you already know helps: Provable self-supervised learning. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 309-323. Curran Associates, Inc., 2021. URL httpp::/pproceeinngs neurips cttppssec/pppprrreeeeee fwwee/iee/ppppe http//pppper//11//////// file/
-ddf.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
preprint arXiv:/608. 03983, 2016.
Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Compu- rational Harmonic Analysis, 19:224-241, 2022. ISSN 1063-5203. doi: https://doi.org// w0wg//..././.ooo.0 1016/ j.acha.2021112.01 URL https: / www.. sciencedi rect com/ sccseccceeeeeccccccceoccc art /cle/ p1i//11635203220000 123. Special Issue Harmonic Analysis and Machine Learning.
14

---

Published as a conference paper at ICLR 2026
Achleshwar Luthra, Tianbao Yang, and Tomer Galanti. Self-supervised contrastive learning is abgpg///bb/256..... /5006.
approximately supervised contrastive learning, 2025. CRLhttps:/ 04411.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In Silvia Chiappa and Roberto Calandra (eds.). Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 875-884, PMLR, 26-28 Aug 2020. URL htttps:://procceedings. mlr. pte.spres//eee ve//v11121/ wwee1//mmaaaleeeeeeee 2ht.:2...............
Wenlong Mou, Liwei Wang, Xiyu Zhai, and Kai Zheng. Generalization bounds of sgld for non-convex learning: Two theoretical viewpoints. In Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet (eds.). Proceedings of the 31st Conference On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pp. 605 638. PMLR, 06-09 Jul 2018. URL pettps:/prrceeeeee wtt.:.ss........r.....m //15//////o/// 8a. html.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning, In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http: //tp:://flllla/// wtw.f//.....aa..dd....o/ wtw.ford.edd.oooo.o.oooooo. edu/ pdf.
Kento Nozawa and Issei Sato. Understanding negative samples in instance discrim- inative self-supervised representation learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, volume 34, pp. 5784-5797, Curran Associates, Inc., wwtpppprpppppiieeeeeeeee www.ffillsppaeeeee /ttpe////pp///////// 2021. URL https: Paper Pwp.papper........ pdf.
neurips wwweerrps.ccppeeeeeeee
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaacidin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Yao Huang. Shang-Wer Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024. URL https: / / arxivv org/ 2304 07193.
Zhuo Ouyang, Kaiwen Hu, Qi Zhang, Yifei Wang. and Yisen Wang. Projection head is secretly an information bottleneck. In The Thirteenth International Conference on Learning Representations, 2025. URL https: htttp:://ppereeeee
net
Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020. doi: 10. htt73//maa.2015509900 1ttp//1..17////mm. ppa.s./p//./../ 2. 5509117.
URL https: hwwp pnas /ttg//i////oooooooooo///////o/oo
Advait Parulekar, Liam Collins, Karthikeyan Shanmugam, Aryan Mokhtari, and Sanjay Shakkottai. Infonce loss provably learns cluster puuster-preserving representations. In Gergely Neu and Lorenzo Rosasco (eds.), Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pp. 1914-1961. PMLR, 12-15 Jul 2023. URL walr.pressee vvtsss/vvv1/// 95 httpp/aaaalaaaaaaaaaaaaaaaa html.
htlps::/pprceeeeee
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763 PMLR, 18-24 Jul 2021. URL https: : / proceed ineeedinggggg ngs. mlm. ..l..sprsssss.. / v1 39/ r forddrrrdd111 html.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pp. 3982-3992. Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1/D19/ 1410.
URL htttps://aaee anthol org/D1 1410/.
15

---

Published as a conference paper at ICLR 2026
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari. Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 19250-19286. PMLR, 17-23 Jul 2022. URL https: http//ppooceeeeneeeeeeee/eeee/ee mlr press / 121 sat.s/ianshh22aa. html.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition, 2019. URL ht http/axxivvv.aa/aaa//aa org/
http//aww.....a///a/aaoaaa////// absp///ss///99// 05862.
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3): 473, 1948. doi: 10. 111111138777798....1
Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z. Haochen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 19847-19878, PMLR, 17-23 Jul 2022. URL https: httpppss/prroeeeeeeeee /tpppppooocnnngeeeee mww. www.sppeesvvll.... 62 http//hhnn2ddmmdm
Ravid Shwartz-Ziv, Randall Balestriero, Kenji Kawaguchi, Tim G. J. Rudner, and Yann LeCun. An regularization, In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL https:/ /ttp/r//opeeeeieeeeee net/forumt
information theory perspective on inrianceiivarrince--ooo
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6827- 6839. Curran Associates, Inc., 2020. httttttss////tceeeette files Files/paper/ /ttpeeer//2///11// 2ttp0////0//22222222200002221130
Yuandong Tian. Understanding deep contrastive learning via coordinate optimization. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 19511-19522. Curran Asso- ciates, Inc., 2022. URL httpa///ee.pppoeeee.........e..p 2022/fil /ttp////000000000000000001111110
www.iiseesppp..ppe cc Pa----apper-oooeeeneeeeeeee pdf.
Yuandong Tian. Understanding the role of nonlinearity in training dynamics of contrastive learning. In The Eleventh International Conference on Learning Representations, 2023. URL https:
/ ///ppreeveew..c wtwp://www.et//oommieeeeeee /tt///mmmmmd-11111133333
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory, pp. 1179-1206. PMLR, 2021.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2020. URL https: /ttp////ppp/eeeeeeeeeeep/eeeaeee
wew.wiewwweeeeeeeeeeeeee...e.eee www.:/eee/oommmmeeemmm forummid= 4FPH.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features, 2025. URL https://arxise wttp////////////oo/////////// 14786.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019. URL https: :
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In D. Lee, M. Sugiyama. U. Luxburg, I. Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Asso- ciates, Inc., 2016 URL https: UtL nwwri oc httpppppp.ppppppppepppeep /ppper files/paper// 0000
16

---

Published as a conference paper at ICLR 2026
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2495 2954, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939 PMLR, 2020.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations, 2022. URL https: htt.tt::/opeeeeeriiiii www.reppeeeeee.eeeeeee net/ forum? id=ECvgmYVyeUz.
Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data, International Conference on Learning Representations, 2021. URL httph:tp///c/m
http//ppeeeeviieeeeeeeeeeeeeeeee net/ forum? id=rC8sJ41 66aH.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Researck, pp. 11112-11122. PMLR, 18-24 Jul 2021. URL https: http:///ooeeeeeeeeeeeeeeeeececee mal..press. /viessss/v//9 v139/
@en2lc.html.
Xi Weng, Jianing An, Xudong Ma, Binhang Qi, Jie Luo, Xi Yang, Jin Song Dong, and Lei Huang. Clustering properties of self-supervised learning, 2025. URL https: //axxiv.//////////aa//a/aa org abs/
2501. 18452.
Yihao Xue, Eric Gan, Jiayi Ni, Siddharth Joshi, and Baharan Mirzasoleiman. Investigating the benefits of projection head for representation learning. In The Twelfth International Conference Learning Representations, 2024. URL https: //::://ooooo oppnnevieeeeeeeeeeeeeeeeeee net / forum? ? iid-idd-g-n Ldd-GEEAAYYPA,
Chun-Hsiao Yeh, Cheng-Yas Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. Decoupled contrastive learning. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23 23-27, 2022, Proceedings, Part XXVI. pp. 668 684, Berlin, Heidelberg. doi: 10.1007/978- 3-0-----3---- 19809-0 38. URL
2022. Springer- Verlag. ISBN 978-3-031 110---3--31-1188----11
https: : / /ooi .rgg 0. 007 / 978- 3 031 9809-0 0_...
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 1008.00888, 2017.
Zhuoning Yuan, Yuexin Wu, Zi-Hao Qiu, Xianzhi Du, Lijun Zhang, Denny Zhou, and Tianbao Yang. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári. Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17- 23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 25760-25782. PMLR, 2022. URL https: v162 2062//uan225 html.
httptps:/pprcceeeangggg http///pcoeeenng.eeeee///oe mlr mlw.presst
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310- 12320. PMLR, 2021.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings the the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11975-11986, October 2023.
Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. In Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 31697-31710. Curran Associates, Inc., 2022. URL https: /tttps://prooee http://pooeeeennn.eeee wwwpooeeddingg.nuiiiieeeeee neurips httpepceeppppppeeeeeeeeeeeeeeeee www.s//pppppee/////// 41c aaaeee/c444..ffaaoooot77722222a2 wwwaapp--ooffeeneee..o .df..
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International Conference on Machine Learning, pp. 12979-12990. PMLR, 2021.
17

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
Large Language Models (LLMs) were used solely an assistive tool for improving the clarity and presentation of the manuseript (e.g.. editing grammar, refining phrasing). All technical content, including theoretical derivations, proofs, experimental design, and analysis, was developed entirely by the authors. No parts the paper were written or ideated by an LLM in a way that would constitute substantive scientific contribution, and no LLM was used to generate fabricate results.
B ADDITIONAL EXPERIMENTS
Datasets and augmentations. CIFARIO and CIFARI00 both consist 50000 training images and 10000 validation images with 10 classes and 100 classes, respectively, uniformly distributed across the dataset, i.e., CIFARIO 10 has 5000 samples per class and CIFAR 1100 has 500 samples per class. Mini-ImageNet also has 5000 test images on top of 50000 train and 10000 validation images, with 100 of 1000 classes from ImageNet-Ik (Deng et al., 2009) (at the original resolution). Tiny-ImageNer contains 100000 images downsampled to 64 x 64, with total 200 classes from IM-1K. Each class has 500 training, 50 validation, and 50 test images.
We use standard augmentations as proposed in SimCLR (Chen et al., 2020). For experiments on Mini- Met we use the following pipeline: random resized cropping to 224 x 224, random horizontal flipping. color jittering (brightness, contrast, saturation: 0.8; hue: 0.2), random grayscale conversion (p = 0.2), and Gaussian blur (applied with probability 0.1 using a 3 x 3 kernel and a 1.5). For Niage we drop saturation to 0.4 and hue to 0.1 due to low resolution images. For CIFAR datasets, we adopt similar pipeline with appropriately scaled parameters. The crop size is adjusted to 32 x 32, and the color jitter parameters are scaled to saturation 0.4, and hue 0.1.
B. EXPERIMENTS WITH THE VIT ARCHITECTURE
To further support the claims made in the main text, we reproduce the experiment from Fig. using the ViT-Base architecture (Dosovitskiy al., 2021). Throughout these experiments, use the same training hyperparameters and augmentations for each dataset as in the ResNet-50 experiments. As shown in Fig. the alignment between CL and supervised models exhibits the same qualitative trends observed for the ResNet-50 architecture in Fig. 2, demonstrating that the relationship between training dynamics and representational alignment is consistent across both convolutional and transformer-based models.
In addition, we repeat the experiments in Figs. and 5 for the ViT-Base architecture. The correspond- ing results, shown in Figs. 8 and 9, closely match those obtained with ResNet-50, further reinforcing the robustness of our findings across architectures.
B.2 EFFECT OF NUMBER OF CLASSES ON ALIGNMENT
In addition to the linear CKA results reported in the main text (Fig. 3), we also evaluate representa- tional similarity using RSA. The corresponding RSA values presented in Fig. 10, providing complementary perspective on alignment across varying numbers of classes. In addition, we also reproduced the results with RSA for the ViT models (Fig. 11).
B.3 PERFORMANCE, ALIGGNNEET TRADEOFF
The bound Thm. predicts that alignment increases with larger r. Moreover, when v)- = OBB) suggests that alignment should decrease as B grows, whereas under пソ = O(B1/4) it instead predicts higher alignment for larger B. In this experiment, we examine whether higher alignment in fact corresponds to more similar downstream accuracies. Specifically, Figs. 12-13 we vary the parameters and B (respectively) and plot the gap between the accuracies of the CL and NSCL models against their RSA alignment values. To obtain the accuracy measures, we perform full-shot linear probing both the CL- and NSCL-trained models and report their test accuracies. As can be seen from the results, we consistently observe that higher alignment corresponds to a smaller gap between the accuracy rates of the CL- and NSCL- models. This suggests that the alignment
18

---

Published as a conference paper at ICLR 2026
1.00
1.00
1.00
0.80
0.0
5
2.00
154L
104L
554L
154L
0.20
Tya
Treh
10
12
17
10
107
10
107
7aiiing
7sning
Taning
1.00
8 0.800
8
4.60
I5CL
5CCL
55C4
1521
0.0-
0.20
350
Taxt
500
Raaiing
Raninn
Taaing
CIFAR-10
CIFAR-100
Mini-ImageNet
Timy-ImageNet
Figure 7: Alignment during training for ViT. We train ViT-base model with CL. NSCL,, SCL. and CE objectives. The alignment between CL and supervised models follow similar trends shown for ResNet-50 in Fig. 2.
Tran
Trx
T-03
0.55
8.00
0.00
3 3.00
5 90
0.60
0.0
0.60
0.50
no spocts
7aniigg spochs
Taning spachs
Tsning sporns
Taaa
100
1.00
10
1.00
0.55
8
8 6/
3020
0.60
350
200
300
300
200
330
08
306
206
300
0.60
200
200
302
Traiing spocts
Taning spochs
Taning spocns
Taning spacts
CIFAR-10
CIFAR-100
Mini-ImageNet
Tiny-ImageNNc
Figure 8: Effect of the temperature (7) on CL-NSCL alignment. We train ViT-Base models with decoupled CL and NSCL objectives using different temperature values т. All models are trained for 300 epochs. Across all datasets, alignment consistently increases as 7 becomes larger.
between CL and NSCL models translates into concrete predictions about how close the models are in their performance. For completeness, summarize these accuracy values in Tab. 2.
CIFAR-100
Mini-ImageNNt
Tiny-ImageNet
0.1 T= 0.5
== 1.0
0.1
0.5
1.0
7
0.1
T
0.5
T 1.0
CL
65.18
61.62
58.60
70.30
70.55
68.21
44.50
40.41
35.40
NSCL
68.25
62.44
59.02
73.93
71.88
67.76
46.51
39.95
35.29
Table 2: Linear Probe (LP) test accuracies (%) for varying r. We train CL and NSCL ResNet-50 models for 300 epochs, and observe that the accuracy gap decreases with higher alignment between CL and NSCL models (also shown Fig. 12).
B.4 EXPERIMENTS WITH
DATA
19

---

Published as a conference paper at ICLR 2026
1.00
1.002
1.003
1.000
1-20
1924
1.003
0.00
3.00
2.93
2 3.772
090
3347
0.00
1.000
0.57
200
302
303
202
1,00
Tel
Tea:
0-232
Teat
Tel
3.902
0.002
1.00
3.99
677
0997
2.00
0.007
20
10
90
100
Thairing
102
230
Trairing
100
200
30 100
202
(a) (
(b) O(B
(c) O(B²/*
(d) O(1)
Figure 9: Effect of batch size (B) on CL-NSCL alignment. We follow the same learning-rate scaling strategy as for ResNet-50. The alignment trends observed when varying the batch size similar to those for ResNet-50.
00000
CIFAR-10
CIFAR-100
Mini-ImageNet
Imageeta
ImageNet-Ik -k
Figure 10: CL-NSCL alignment (RSA) increases with the number of training classes. Sec. 5.1 and Fig. 3 for experimental details.
See
Since our theory is tighter for relatively balanced classes, but does not require perfectly balanced data, also evaluate it the SVHN dataset (Netzer et al., 201 1), which is well known for its pronounced class imbalance. In Fig. 14. we plot the RSA and CKA metrics between coupled CL and NSCL models trained for 300 epochs. The training hyperparameters and the data augmentations are the same as in our CIFAR- 100 experiments to facilitate direct comparison.
Despite the class imbalance in SVHN, we observe that the alignment between the two models is consistently high sigh-dndeed, it is even stronger than what we typically obtain after 1,000 epochs on CIFAR-100, which has the same number classes. This finding suggests that substantial class imbalance does not hinder strong representational alignment from emerging between coupled CL and NSCL models, and further supports the robustness of our theoretical predictions beyond the approximately balanced setting.
B.5 ATTENTION MAPS ALIGNMENT
Methodology. To analyze the self-attention maps from the frozen Vision Transformer encoder, look into the Multi-Hess Self Attention (MHSA) mechanism the final transformer layer (L = 12 for ViT-Base).
20

---

Published as a conference paper at ICLR 2026
CIFAR-10
CIFAR- 100
Mini-ImageNet
Imag-em
ImageNet- Ik
Figure 11: CL-NSCL alignment (RSA) increases with the number of training classes for ViT- Base models. The alignment increases with number of classes, and is consistent with trends observed for ResNet-50 models.
2.000
3,000
1.750
2,500
3.000
Acur33 655
2.000
560
500
1250
0000
0.750
0250
0.
0.825
0.650
0.525
0.50
Alignment (RSA)
Aignment (ASA)
Aligmment (RSA)
74.
$8.000
73.000
44,000
72.000
4240000
64.000
71.000
70.000
60.000
68.000
35,000
0.800 0.800
0.003
0.920 0.940
5.600
0800
0.850
0.903
0.950
0.525 0.850 0.575 9,500 0.525 0.50
Allgnment (RSA)
Algnment (RSA)
Alignment (RSA)
C1FAR-100
Mini-ImageNet
Tiny-ImageNet
Figure 12: Performance vs. alignment varying temperatures. (Top) The gap between the linear prohe accuracies of CL. and NSCL ResNet-50 models (trained for 300 epochs) decreases as their alignment increases with higher temperature (7) values. (Bottom) Although the accuracy gap between CL and NSCL models is correlated with the alignment of their representations, higher alignment does not necessarily imply better downstream performance, as performance remains sensitive to the choice of hyperparameters.
Let A E R denote the attention weights, where H is the number of heads and N is the number of tokens. We first average weights across all attention heads. We then extract the row corresponding to [CLS] token, specifically focusing on its attention to N - image patch tokens. This vector is reshaped into 2D grid (14 x 14 for ViT-Base) to match the spatial arrangement of image patches. Finally, upscale the low-resolution grid to original image resolution, normalize to the range [0, 1]. and overlay on the input image.
Analysis. To quantify the structural similarity between representations of VIT models trained with decoupled CL and supervised objectives, we calculate the cosine similarity between their attention maps. As shown in Fig. 15, track this metrie across training epochs and show that NSCL consistently maintains the highest alignment with DCL compared to NSCL and CE. To strengthen argument, further visually illustrate this alignment in Fig. 16. The qualitative analysis align
21

---

Published as a conference paper at ICLR 2026
4.500
B-512
4.000
2.000
3.500
25
Accuracy gap
3.000
Accuracy gap
2.500
1.000
8-128 8-256
B-012
1.000
B-128
0.500
b0202
0.500
0.860 0.880 0.900 0.920 0.940
0.960
0.980
0.986
0.988
0.990
0.992
0.994
Alignment (RSA)
Alignment (RSA)
(a) ((B)
(b) O(B(1)
Figure 13: Performance vs. alignment over varying batch sizes. The gap between the linear probe accuracies of CL and NSCL ResNet-50 models (trained for 300 epochs) varies systematically with the batch size (B) and their RSA alignment: when training with ?t O(B), larger batch sizes tend to reduce alignment and increase the accuracy gap, whereas with 1) O(B /4) larger batch sizes tend to increase alignment and reduce the gap.
1.00
1.00
0.90
0.90
RSA
0.80
CKA
0.80
0.70
0.70
Train
Train
0.60
Test
NSCL
0.60
Test
NSCL
10
101
10a
10
102
10a
Training epochs
Training epochs
Figure 14: CL-NSCL alignment for class-imbalanced data. We train ResNet-50 models on SVHN (Netzer et al., 2011) with decoupled CL and NSCL objectives to analyse alignment when the classes are not uniformly distributed. The RSA and CKA values are comparable to class-balanced datasets (shown in Fig. 2- 7).
with cosine similarity trends, confirming that NSCL preserves the spatial attention structure of CL more faithfully than other supervised methods.
B.6 FIG. METHODOLOGY
We explain how to generate the plots comparing alignment in weight-space and representation The two plots on the left visualize the direction of learning for each model. Each vector represents the change in model's state from initialization (epoch 0) epoch 1000.
Model states. We consider CL and NSCL models trained on CIFARI00, corresponding to epoch and epoch 1000-a total of four models.
Weight space. This plot shows how the raw parameters evolve during training. For all four models, first flatten all the weights into massive vector which gives us four points in very high dimensional space (order of 107). To visualize these points, we perform Principal Component Analysis (PCA) all four vectors combined and fit them to 3D space. This creates shared 3D)
22

---

Published as a conference paper at ICLR 2026
1.00
1.00
1.00
0.00
Cosnee Smibee
0.00
0.00
10CL
0.50-
55CL
2.40
0.40
0.40
10
10
10
T0
10
70
T0
Taning
CIFAR-10
CIFAR-100
Mini-Image ImageNet
Liny-ImaeeNet
Figure 15: Alignment in attention maps. We evaluate cosine similarity between attention maps of decoupled CL and supervised models, and observe similar trends as for RSA/CKA values. NSCL remains the most aligned hinting at a deeper structural similarity between representations of CL and NSCL models.
1000 coordinate system. We transform all four points into this space and we get PCL. PCI. PKSCLI PNSCI. Using these points, create two vectors: (VCL, UNSCL), and create polar plot using the final vectors and the calculated angle between them (85.7%).
1000
Representation space. This plot shows how model's alignment for specific class evolved. We pick one class from our dataset (CIFAR 100) and randomly sample 100 images. We use the same samples for all four models to extract their corresponding features, say Z E R R100xd, where d is the projection dimension. We concatenate total 400 representations (100 from each model) and perform PCA to learn shared 3D coordinate system. The representations are transformed to this shared space (R 100xd + I 100x3) and averaged to single 3D point for each model. Just like before, polar plot is created using the vectors and angle between them (27.8%).
Similarity metrics. We report RSA and CKA values computed between DCL and NSCL models trained on CIFAR100. Additionally, we show their average weight gap detailed in Sec. 5.1. Is evident that models stay aligned in representation space but diverge in weight space.
B.7 MODEL MERGING
In addition to our main analysis, we also conduct simple experiment that merges models directly in representation space. Specifically, we interpolate between the learned embeddings of CL encoder trained on the full dataset and an NSCL encoder trained on only 30% of the dataset. This merged representation already surpasses both the full-data CL. model and the small-data NSCL model, reinforcing that NSCL and CL remain geometrically compatible in practice.
Concretely, given input let fei (z) and fsect.o (x) denote the representations from the CL encoder and the NSCL encoder trained on 30% of the dataset, respectively. We merge them via simple linear interpolation:
fmergea (z) + (1
fxsctan (x).
We then perform NCCC and LP evaluations using the same 30% subset from the training split and report accuracy the full mini-ImageNet test split Fig. 17.
As shown in the figure, for all values of a the merged model outperforms the NSCL baseline, and for [0.7, 1) it also outperforms the CL baseline on the mini- mini-ImageNet downstream classification task. This suggests that the CL and NSCL representations are well aligned, making it possible to effectively merge them directly in representation space.
C PARAMETER-STE SPACE COUPLING
To complement the analysis in Sec. 4, compare the two trajectories in parameter space. Let e, = ||w w w. NSCL. denote the parameter drift at step t. We would like to bound it as a function of the number of training iterations, batch size, and learning rate scheduling. We classic techniques that can be found at (Bousquet & Elisseeff, 2002; Hardt et al., 2016; Mou et al., 2018; Kuzborskij & Lampert, 2017).
23

---

Published as a conference paper at ICLR 2026
Input
DCL
NSCL
SCL
CE
Figure 16: Visualization of attention maps. We visualize the self-attention of the [CLS) token from the last layer the frozen ViT encoder. Beyond a high cosine similarity between attention maps, these visualizations reveal strong structural similarity between CL and NSCL.
24

---

Published as a conference paper at ICLR 2026
62.00
66.00
60.00
64.00
Accuracy
58.00
Accuracy
62.00
56.00
60.00
NCCC
54.00
NSCL Baseline
58.00
NSCL Baseline
Baseline
CL Baseline
52.00
56.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
a
a
NCCC
LP
Figure 17: Model merging in representation space: We report NCCC and LP scores mini- ImageNet using CL encoder trained on full dataset and NSCL encoder trained on 30% the dataset. The performance gains obtained using merged representations illustrate the compatibility of CL and NSCL models and further support our main finding that CL and NSCL maintain closely aligned embedding geometries throughout training.
Optimization. In order to isolate the effect of the loss, we optimize both objectives (CL and NSCL) with standard mini-batch SGD under single coupled protocol: at step f we draw batch B, {(r5, aly, y5) 11) with replacement, where each n'j ~ a(x,) (e.g., random crop/resize, horizontal flip, color jitter, Gaussian blur); we average per-anchor terms to form either (w)) or ZNS (w)) using cosine similarity (optionally temperature-scaled), hence bounded in [-1,1]; and we update W Wt 7/ Vlg (w)) with prescribed 1/2 O. We then run coupled SGD trajectories from the initialization w wi NSCL that share the same batches and augmentations (B:) and differ only by NSCL's exclusion of same-class negatives:
CL. CL. Th wi+1 wt
CL B.
(T))
NSCI. NSCI. NSCL IJ V W++1
B. (wi (SCT). L= ,,...,T--1 1
(w:
wi
Throughout the analysis, make standard assumptions the smoothness of the loss functions and the scale of gradients.
Assumptions. To control the dynamics, impose two standard conditions on the geometry of the batch objectives and the scale of pairwise gradients.
Assumption 1 (Uniform smoothness). For every batch B, the functions (w) and ZNSCL (w) 8-smooth with the same constant 3 >> 0:
[V - Vo(u)ll < 1| -wl for all v. w E RRP, OE ( PNSCI)..
Assumption 2 (Bounded pairwise gradients). There exists G > O, independent of B and t, such that for all and all pairs (u, v) appearing in any denominator term,
| w sim (fw fm (u))l < G.
We quantify drift between the coupled trajectories under shared randomness in the nonconvex B- smooth regime. Throughout, the only data- term is (B, which decreases with more classes and larger batches.
Theorem 2. Fix B, E N, 8 (0, 1), and temperature Then, with probability at least 1 8,
0. Suppose Assumptions 1-2 hold.
G G- AB,
Σ 11 -1)
er
exp
BT
The bound scales linearly with G and A (B; (B:T). but crucially it is amplified by the exponential factor exp(33 E. n.). Unless the step sizes are aggressively annealed, this term grows rapidly with
25

---

Published as a conference paper at ICLR 2026
training time. Even though Ax, (B;)) improves with C' and B (e.g., for C=1000, B=512, 8=0.01, we obtain A = 0.01 so that the reweightings of the steps differ by about one percent), the exponential accumulation still overwhelm this small per-step gap.
In other words, parameter-space coupling guarantees only that the two runs do not drift apart too quickly in weight space. But because the weights may follow very different trajectories even when representations remain similar, this control is too weak to yield meaningful statements about representational alignment. This motivates our next step: shifting the analysis to similarity space, where we can obtain bounds that remain stable throughout training and translate directly into guarantees on metrics such as CKA and RSA.
Proof idea. With high probability over batches (Cor. 3), every anchor's denominator is dominated by negatives up EB, fluctuations. This keeps the (temperature-7) softmax reweighting gap between CL and NSCL small. In particular, Lem. 7 shows that the per-batch parameter gradients differ uniformly as
G (B;T(
V CL B.
NSCL (w)) (w) - Veg.
<
By B-smoothness each batch loss, each step can expand distances by at most factor (1 + 3m.) Combining this smoothness expansion with the uniform gradient-gap bound yields the following recurrence:
G e1+1 S (1 Bmb) Et + Th A (.,.s(B:)).
where the first term propagates the previous eтror (with amplification controlled by curvature), and the second injects the new discrepancy introduced by the CL-NSCL gap at temperature т.
Unrolling over T steps and applying the discrete Grönwall inequality gives the exponential-tyy typpe bound
G er & 37 (AA.((B:T)
T-1
Σ Th
-1)
exp
t=0
Thus, cumulative drift scales with the reweighting gap and is amplified exponentially with the total step size; smaller tightens the softmax and increases the constants (via both 1/7 and e²/r inside Ar,,), so keeping E. 7/ moderate especially important.
D WHY GRADIENT DESCENT IN SIMILARITY SPACE IS A FAITHFUL SURROGATE
We now explain why running gradient descent directly in similarity space closely tracks the dynamics induced by gradient descent parameter space.
When parameters move from w, to W1+1, the induced change in the similarity matrix can be approxi- mated by linear expansion:
E(w++)) 1) E(wr) Je(wee1 wx), J. = J(w,),
where J(w) 05/0w is the Jacobian. The error in this expansion, denoted RG quadratic in the step size:
[(w++)) (ww) Je(we+1 w.) Rs.
By the chain rule, the gradient in parameter space can be written as follows:
Vol(w,) J Vz((V((:)) J,G., where G. = V>r((((w)).. Substituting this into the update rule gives
((w++)) (w,) - P.G, + Re, Pt = JrJ, > 0.
(5)
Thus, parameter descent acts like similarity descent, but with a preconditioning matrix P., plus the remainder Rr.
Assume there exist constants Ls, My; > O such that
My 1 | 2
[[(w]l22-2 & Lz,
| + Aw) E(w) A(w)Awl) -
26

---

Published as a conference paper at ICLR 2026
Then ||P|l2--2 L3 and, with Awy -ni V ((we))))
Mr
My 2
| R &
2 2 [0
=:
=
Let Σ, := E(we) be the similarity trajectory induced by parameter descent, Define Ee as the trajectory of explicit similarity descent:
E+t Gn, G, V)((E,),
with ΣS = Let E, [Σ, E,11 and Cj; == sup, II 2 2 S L + Using equation
adding and subtracting -m, Gr, and applying the temperature-n bounds equation 11 and equation 6, one obtains
E++1 - (1 + 2T E. mCCI|GGull + My =
(7)
Unrolling this recursion from Eg 0 and using II., (1 aw) explEx au) yields
T
T-1
Mx
T-1
E5 Er S exp 2r
Σ
mt)
C3
Σ
me Ilell +
Σ
th
E
(8)
By bounding ||@.|l via equation 13, namely ||@.|l S + v 3/ this simplifies to
[5 - Erlls Exlls S exp
Σ m 2r2B
1
V2C₂ TVB
Σ 1=
Mr
Σ
n t=
11
2
To understand when this bound is conceptually reasonable, suppose (we(())) G for all t, so that E, I G The right-hand side of equation is then controlled by two quantities: the cumulative step size E. Th which appears both inside the exponential and in the linear prefactor (V2Cs/(TVB)) E, Mrs and the term Σ, rk.
A simple sufficient regime is to assume that Et=o T/ S c and Et=o mi S C2 for fixed constants 61, C2 independent of T. Under these conditions, the exponential factor bounded by exp ((1/(2 B)) E, nh) S cxp(e1/2), the linear prefactor by (V2 C/T//BB)) E. 7) I V2 Ci B (a fixed constant for given (T, B) and moderate c1). and the quadratic remainder by (Ms E, = S
oz. In particular, when Σ, Mt/t/ / B) and E, n/ are both bounded by constants independent of T, the bound guarantees that [パET - ErIl remains controlled (and small whenever Cz, Mg, G are moderate).
To summarize, the similarity and parameter trajectories stay close whenever the normalized cumulative step size () bounded and the learning-rate schedule is sufficiently decaying so that Ex 2/ remains bounded. For fixed learning-rate schedule, large batch size B and moderate temperature T act as stabilizing factors via the 1/(rvB) dependence in equation 9, while very small extremely large, non-decaying step sizes can make the coupling poor, reflected by the bound.
E TECHNICAL TOOLS AND PROOFS
E.1 NOTATION AND BASIC SOFTMAX FACTS
Let S (II; ) be dataset with C classes (each class e has n. points, with E n = N, and we do not assume the ne equal). For parameters w. let 2; fw (2;,) and define the bounded similarity matrix
E(w); = sim (2n E [-1,1].
At step f, draw a mini-batch B, = (( 214,,,,/,,,, Vy. 8 with replacement, using independent augmen- tations 2²j a(I,,). For an anchor E (ji,...,jB) let D; be its denominator index set, and let Dueg := (k+ E D;: y) # yr} (and similarly D denote the subset restricted to negatives (c.g., in two-view SimCLR, D; consists of all 2B views except the anchor itself).
27

---

Published as a conference paper at ICLR 2026
Define the anchor's logit vector (w)) (E(w)a,))eD and the corresponding softmax distributions with temperature 0 (default 1):
Pt softmax(s.(w))1)
q: =0t)m
Let i' denote the positive (augmented) index for anchor
For contrastive learning (CL) and negatives-only supervised contrastive learning (NSCL), the per- anchor and batch losses are
(x1())
llogogga Pi
ENSC.(s) (8:)
log 94,5'
CCL B
Σ
(s)))
NNSCL B.
Σ
NSCL (8:)
B
B
iE(i
Since [(w), E [-1, 1], each exponential term inside the softmax lies
exp(E(w))//)) E (e- ,0/r,
a fact used below to control softmax mass ratios.
Lemma aachor-blocks orthogonality). Fix step and batch Bi. For each anchor E B₁, let D
be the set of indices appearing in i's denominator and define the per-anchor gradient 9: E R by
Valli placed on the coordinates {(i,k) k E D.} C It.
with zeros elsewhere (here Ia is the of all coordinates touched at step t... Ifi + j. then Pi and gj have disjoint supports, and hence
(gi. 9j) = 0. B
Consequently, for the batch gradient
+ 9t.
||G|
Σ B
11..1...
(10)
sEB
Proof. By construction, 9i is supported only on coordinates k(t E D;}. while is supported only k() k E D. ). For * j these sets disjoint, every coordinatewise product zero yielding (gi. (3)r = 0. Expanding the square for G,
LIGH
1 Σ
96 B Σ 9)) F
1 B
Σ
19
+
Σ 1l B
1
B
B
if
where the cross terms vanish by orthogonality.
Lemma 2 (Softmax Hessian and gradient Lipschitzness). Fix a step and batch Br. Let Ie be the set of coordinates (i,k) k) that appear in any anchor's 's denominator at step and view LB. (either CL. or NSCL) as a function of the restricted similarity entries E R.Z. For each anchor i, write 8; (E(i,k) ki, I and ps coftmax(s:/1) Then:
1 -/2 Jss), (ss) Diag(p:) P.P..
(6), 272B
Consequently, for all E, 5 E R.I.
Vv (2) V <
|| E11
(11)
2r²
Proof. With temperature F > 0, for an anchor we have p. = noftmax(s:/7) and
1
V.Pi = 1
(ss))))) - (p. -er) = (2l((s)))
72 [(s)..
=
72
28

---

Published as a conference paper at ICLR 2026
where (s.) Diag(p:) pip. Bound [J(st))l2-2 via the infinity norm:
[J((x)lla--2 s [1J(s;)]ls oo
max Σ
Jrel
max (1.p(1-Ptt))
-Pir)+ + E Pi. +PrisDe)
Egr
max 2p,,r Pir)) S 1/2
since x=1 x)) s 1/4 for x E [0.
The batch loss an average over anchors, so its Hessian is block-diagonal across anchors with prefactor 1/B:
BBB (E)
iEB 72 blkdiag ((si)) )EB,
1 blkdiag (ssi)) ies
Hence
v2 Ls, (Σ) 2- (2) By the mean-value (integral) form for vector fields,
72B max [8(8)]l12-22 S 2r²
(2) Vxls,(5) = //' (S+ -E)) - Σ d0,
and therefore
1
(I) Vx(,,(E) I
E11e S V (Eo) 12) 11-
[ |--E||r 2T
sup #E(0,1
as claimed.
Lemma 3 (Per-anchor gradient norm and batch average). For an anchor let 8; be the vector of logits in its denominator and p: = softmax(ss/ (st/a) Let denote the (umique) positive index (for NSCL, if il is not in the denominator, set ps,y == in the display below). Then
V 4112 = - (1- +
2
Pi.k
<
(12)
7
hence Vell 22/T. Moreover, by block orthogonality across anchors,
3 =
Σ SEB
412 <
2
1
FV
1
2
(13)
B FEB:
B
B2
B iEB
Proof. For CL., the loss is log par entropy derivative with temperature.
P with ps softmax(s:/7). By the standard softmax-cross-
Voli = 1 cr),
-p:-),),
So
IV 412 = 7 (1 pi.s)) + Σ
Pi
S 72
(1 - --.pi +
Σ
Pi,k
2
=
72
(1
(1-pis))
2 < 72
Risht
since pl a probability vector and Eker Pi,k 1 Pt
For NSCL, two cases. If E D,, the same computation applies (the target index is present), hence the same bound holds. If 1' & D, (negatives-only denominator), then the loss is - llg fr,r" with = softmax((ss)0 supported only on D7, and
1 91 on neg (and 0 on D,!o),
V = 7
29

---

Published as a conference paper at ICLR 2026
1V,, = 7 ΣΣ R S
Σ JEDr
1 7
2 7
=
5
JEDT
Thus in all cases vV S v2/T, establishing equation 12.
For the batch bound equation 13, gradients from different anchors have disjoint supports
coordinates {(t, k) l E Di so they are orthogonal Frobenius inner product (Lem. 1). Therefore,
EB
Σ SEB
2
F
B
B
7
T²
BEB
which also implies |1 EieB V.GAIFF S 2
Lemma (Bounded logits imply bounded softmax masses). Fix step and an anchor Suppose all active logits satisfy k) E [-1,]] For any index subset S in the anchor's denominator, define
Es == Σ exp(((i,A))
with temperature > 0.
RES
Then
|// 1/T
< Zs <
|S
In particular, if Spos and Snog are the positive and negative index sets with sizes Apos and negg and ZSnee then
pos
Zueg S
Tpos
S 7spos S
Tipos e e//
Uneg e
1/7
Znog S Oneg e//T
and hence
Zpon
< 2/T
Ppos
and
Zpos Zneg
2 22T Ppos
Zueg
nneg
Thng
Proof. Since k(,,k) [1,, have exp(EI(i,k)/)) (e-1/__e2/r] for every active k. Sum- ming over S yields |S/e-1/r < Us s e/. Apply this with S Spos and S Sneg and take ratios to obtain the stated bounds.
E.2 HIGH-PROBABILITY BATCH COMPOSITION
Fix T, N and e 0. For step and anchor i B., let Y 1 # v1 for I...., B. Lemma 5 (Batch-composition event). For population with C classes and class priors The ne. Ne/N. (i) the te are i.i.d. Bernoulli with mean - Ty: For any > 0,
B Σ s=1
P 3tt
1- 1 Ty. -
S TBe 2B:² -
Equivalently, with probability TBc 2B22 -B22 every anchor sees at least B(1 Ty. c) negatives.
Proof. Fix any step and anchor i. Because batches drawn with replacement from a population with class priors म. = Te/N,, for each position E {I...., ...,,,) the indicator y: 1 + y1 is Bernoulli with mean ( for any e > 0,
Ty.: and (y:(0)) Ism1 are i.i.d. across s. By Hoeffding's inequality,
P
1
<1 n. -
M
r EY -c < exp(-2Be²).
B
B
There are at most TB anchor secchor-teeppee pairs (t, i) over 0, ..... T - and B, union bound gives
B M < Tyz
P ((,
< TBe 2Bc² -2c2
Equivalently, with probability at least
TBe e 2n every anchor in every step has at least B(1 Ty. -)) negatives in its denominator.
30

---

Published as a conference paper at ICLR 2026
Corollary 3. For 8 E 10, set EB, == v 28 log( ( and let ne. Ne/N be the class priors every anchor has least B Ry: 6 B,6) negatives and at most ( + =BB,6) positives in its denominator. In particular,
and Taax == max we- With probability
D 2 B(1 max - EB,6);
|D! < (mTxxxx + +B,6).
Using bounded logits, the ratio of total positive negative softmax mass (at temperature satisfies, for all anchors and steps.
0)
2,00
2/T (Tmax + CB,,)
2 An
(14)
1 Tmax CB,5
where
A (B;T 202/r (Tmax + (,B,))
Tmax - BB,
2B log( ( and A (B;T) = 2/ (Toa ten,))
Proof. Set BB, =
Apply Lem. 5 with
Tmax c00.
this e: with probability least 6, for every step and every anchor i,
D
2 B(1 Ty. - EB,6),
D I
B BTT + CB,6).
In particular,
D 1
B(1 max CB,6).
DDPuM
B maax + CB,6).
In two-view SimCLR, each sampled point contributes two denominator entries, so the denominator contains at least " negative entries and most D.os| positive entries; the factor 2 cancels in the ratio below.
Because similarities are bounded in [-1, 1]. each logit lies in [-1, 1] and hence each exponential 1/T,
term temperature lies [e-1/T, ,e//r] Therefore, for any anchor and step,
Z,008 S 1//T (2)
Z 2 o-1 1/7 (2)
and thus
7os
DDp
2/7 mmax EB.8
<
2//r
<
1/2 (B;))
D
1 Tmax BB,
The bound is meaningful whenever CB,6 < Tmax so that the denominator is positive. This proves the corollary.
Lemma 6 (Per-anchor reweighting gap). On the event of Cor 3, let p be the CL softmax (temperature T > 0) over an anchor's 's full denominator, and the NSCL softmax (same T) that removes same-class entries and renormalizes over negatives. Then
lp - all1 S (B;T(
dll <
1lp 1p--lli r/l1 S (B;T)
Proof. Fix an anchor and let D. Drog be its positive and negative index sets in the CL denomi- nator. Write = E(i, k)) and define
pos
2P00
exp(s)/T),
ZuNe
Σ exp(s//T). jED
=
Z,00 + 2,
DF
Let be the CL softmax D.000 U D.oeg and let g be the NSCI softmax that zeros positive entries and renormalizes on negatives: gkk) = for & E Do and q(i) = p(j)/(1 a) for D.eg. Then
22pos
1p - 411 =
Σ
Pk +
Σ JED
Pj
1 1-a
Dj
a + (1 a)
| 1-a
a 2a 5
Z.
KE
DO
On the high-probability event of Cor. 3, since
E [-1,]] + exp(s/T) E 10-1/5
e 1/T
Z > e 117 D.
31

---

Published as a conference paper at ICLR 2026
Moreover, by Cor. 3,
D
I 2B max + cB,6)
2B
Tmax
EB,6)
(each sampled point contributes two keys, so the factor 2 cancels in the ratio). Hence
27,p00
D neg
202/T Tmax + CB,5)
Z S 202/T
S
(T;6(B:T).
mmax CB,
Therefore llp - 4111 I n F). Finally, ||p alle S |lp - 411 yields the second claim,
E.3 PARAMETER-SPAN COUPLING: SUPPORTING LEMMAS AND PROOFS
Lemma (Per-batch serameter-gradeeee gap). On the event of Cor. 3, for any step and any w.
G An,T(B;T).
V (gg((w) (w) V (SC
Proof. Fix L and For an anchor B, let D; be its denominator index set, split as D; pos, L neg,, where pos, collects all same-class indices (including the designated positive i') and neg, the rest. Write the logits 8 [(i, the CL softmax Pı exp(su//1) LeED, exp($ie/T). and the NSCL softmax
negatives 4ú Pij/ a:) for neg,, with Ok 0 for k E pos,, V - (f fw (xx)); by Assumption 2, where O₁ Exepos, Pik- Define Via V Sik
||us&| I G for all (i,k).
For the per-anchor losses,
CL B.B.
Pik UsA U.p) KED.
Σ
NSCL B,B,
E jEncg,
Qiy Uij - Us8
Hence the per-anchor gradient difference is
Ag. =
CL 1,8
NNSCL
4.B,
1
Σ
Pik
Uik +
Σ
(pij - 9i5) Usj
kEpos
jeneR;
(A)
(B)
By the triangle inequality and ||o SG
| S G
Σ
Pik +
Pii 9sl)
7
REpOR,
jEneR:
Since dú P1sp(11 a:) for j E nog;
Σ pij - 4:i =
Σ jEneg.
Pij - O6i
aç.
jEueg,
Therefore ||Agil| < ((;;; + az)
20 os.
Writing
Zyos with Zpos Zneg
Exepoe, exp(si/TT), Zneg = Ejeneg, exp(s (), we have 147, so
hence 20;
2ri
< 2r,,
1t+;
2G Zpos
A <
Zueg
On the high- probability event of Cor. 3, for every anchor
Zpon
2/7 Tmax + EB,8)
(B:T)
2
Tmax CB,5
so Agill S C (B; for all anchors i.
32

---

Published as a conference paper at ICLR 2026
Finally, the batch gradients are averages anchors:
FNSCL
Σ
Agi.
B
EE
hence
V
CL B.
V
V
B
S
B
1
Σ
Aall
S
B
1
Σ
Ar, = - (B:iB)
G
G
iEB
iEB
Theorem 2. Fix B.T E N. ô E (0, 1), and lemperature T 0. Suppose Assumptions 1-2 hold.
Then, with probability at least
G BT
A (B:r) (exp B
T-1
Σ 11 1).
GT I
t=
Proof. Let (w)
CLL B,
(w) and (N)CL(w)) = : (w)) Assume each TCL is B-smooth. Set
e, |000 w ||.
Write
6++1 = | NSCL [T( - (w, T/t V0, NSCL (wNNsC))) wi+1
S [T((w CL.) - Ti (w [ + m (w, - V (wi (w:
NSCC.)
Bounding (I). Using the integral Hessian representation,
(1) He(u, u) (u v), He(o,u) = / (u (o) dr)
and B-smoothness gives [H, || 2-12 S. Hence
[T((u) Te(w)]
= =I(I - -n (1, u)) - w)
S ||| - [V, w))l2=2
2-2 u wl|
< (1 + nr:3) ||u v/|.
Thus, (I) S + +m,3)e
Bounding (II). On the brob-biobyyylity event of Cor. 3, Lem. yields
G (II) s (Bo(B:T).
Combining the bounds,
ex11 S (1+ 11+) e + n (B;(B:
(15)
Iterating equation 15 from ey = 0 gives
er <
T
G A (B:r)
II s=t+1
T-1
T-1 (1 s Ar,
G
Σ 1=
n exp (8
Σ n.).
T-1
t=
s=t+1
where we used 1 + x < et Let St = T- y/s so that St 11 + St+1- Then for each 1,
The exp(3S < ( (exp(3S)) exp(
3
since Bn - 3r,. Summing over t = 0, .
T - telescopes to
ET < BT (no(B:T) (exp(
G
T-1
Σ mh) 1).
t=0
This holds with probability least - 8.
33

---

Published as a conference paper at ICLR 2026
E.4 SIMILARITY-SPACE ANALYSIS AND COUPLING
Lemma 8 (Per-step gradient gap in similarity space), On the event of Cor. 3, for any step I,
1 ( (iTT)
NNSL GG G. (M) (SNSCL)ll Ilp
ECL
NSCL
VB
2T B
reweigghtin (Mock-orrh.)
Lipachitz in Σ
Proof. Add and subtract GNSCL (N and apply the triangle inequality:
||G (E G (I) (ENsCL)) |||-
G G(L((EE))) G GNSCL (2C G
+ Ge (NSC
G2
GNSCL (S.NCC.) G
(16)
(B)
Term (B): Lipschitz in E. By the temperature-T softmax-Hessian bound equation 11,
(B) S 27
ECL ENSCL
Term (A): reweighting gap fixed I Decompose the batch gradient into anchor blocks:
1 Σ (E), (E = B
OE NSCL),
iEB.
where each 96,4 has support only the coordinates of anchor i. For anchor i, with temperature т, g.. (2 = (I/T)(p: er), gi. (XCC)) = (1/7) - ey), so 91,3 (2(T)) gi, (2CL)))))) = (I/T)(P- q:) on that block. By block orthogonality (Lem. 1),
Σ
1 (p:
40)
(A)
|lp: 1:
B
7B
TEB.
iEB.
On the event of Cor. 3, Lem. 6 gives |pp 4:ll2 S A ) for every anchor, hence
(A) < 7B V (B:) 1 (B;((:iT)
1
VB
Combining the bounds (A) and (B) yields the claim.
Theorem 1 (Similarity-spacee coupling). Fix B,T E N, 8 E (0, 1), and temperature T >> Consider the coupled mimillarttydescenn recursions equation I for CL and NSCL with shared initialization and shared mini-batches/aun baw-bbeheeeugmmettttoonns Then, with probability at least - over the draws of the and augmentations, for any stepsizes (m))
EASCLII
S exp
Σ f=
7h
Ans(B:T).
(2)
2r
TVB
t=
Proof. Condition the event of Cor. 3 (which holds with probability least 1 5). Let Dr |
NNSCL|| p. From the coupled updates equation 1,
CL -11 1+1
NSCL
(I -NSCLL- nt (GFL CNNCCL
hence
D41 S Dr + me
G.L (ENCCL
Add and subtract (FCCLL(CC) and apply Lem. 8 (reweighting gap + Lipschitz with temperature r):
A (B:T)
||
(XNSCLL <
+
2- B D.-
B
34

---

Published as a conference paper at ICLR 2026
Therefore,
D+41= S 1
11t D. + 1
A (B;T(
2r
vB
(B.T(
Let aq 2r²B and 7 nt V/product form gives
With Du 0 (shared initialization), the discrete Grönwal-
TVB
T-1
T-1 (1+aa)) S exp
DT S
Σ ==0
Ta II
Σ
au
T
a=++1
a=0
==0
using [1 au))
a.) S exp(E, au). Substituting az; yields
T-1
T-1 Σ 1=0
Dr & exp 2r
To
Ar.((:
1=0
desired.
Consequences for CKA and RSA.
Corollary (CKA lower bound). In the setting of Thm. 1. Assume |K > 0. With probability at least 1 6,
exp 2 0 m))
(Et= m1)
CKAT 2
1- PT 14 pr
PT
|KCL
TVB
Proof.
Let Ar |KTT||| 0 and AKT |K K KNSCI. ||,, where all norms Frobenius.
Then
(KGL, KASCLL- = ( K - (KAYSCL - -KG))
(17)
|| + (K KANSLL K(T) A Ar AK,T;
by Cauchy- Cauchy-Schwarz. Schwarz. By the triangle inequality, ||
|KNSCLLI| 1/00 S Ar AKT. Hence
(KFL KNCC CKAT |KG
A - AT
1- A-AKT/AT
K KASCLIIF
I|F
AT(AT AK,T)
1 1+AKTT/Ar
Next, K HEgH H with the centering projector H 1 $ 11 so AKT ||
NNSCL ) S |5 .n ENSCL ET 8,
|| because || 2- = By Thm. with probability at least
DCCL E ENSCLI & exp
Σ
(B;T)
Br
t=
1=
Combining the last two equations yields the lower bound CKAT with probability least ô.
O
Corollary 2 (RSA lower bound). In the setting of Thm. 1. Assume DDT > 0. With probability at least 8.
1- 1+rT
exp exp 2FB Et Ht)
TVB EE' m) (B;)(BiT))
2 RSAT
rT <
VOOT
Proof. Let M 2 and let == 11 be the centering projector in R.M. Write a, Car
and b. = CbT, Then
(ac: be)
RSA
||aell2 lz libell2"
For any nonzero u and any u in an inner-product space,
(u, v) (u, + (o u))
|ull2 (u, u) 2 |/ul12 [uul2 ||e - ull2.
and || ||ll22 + ||l u/|2- Therefore.
(u, v)
1 1 u / 21122
1 /u/12 ||1//2
1
u u
35

---

Published as a conference paper at ICLR 2026
Apply this with u dg and be to obtain
1 |bb - ael12/ l
RSAT 2 1 - acll2/ |lccl2
Since C' is an orthogonal projector, l - acll2 = |br - ar)ll2 - |by arl22. By construction of the RDM vectors,
bt 0T vec (off (2, 2T1)),
so |IbT - aT|2 = [ NNNLL NSCL EgH)|| I ||SSSLL Eg ||.. Finally, by Thm. 1, with probability at least 1 - Ó,
NSCL DT - exp 2r
T1
Σ 1=
(T;r(
TVB
1=
Combining the last three displays yields the stated (1 (1 + r) lower bound on RSAr after substituting ||ell2 VM DDT.
36


Reviews:
[{"review_id": "GJmMouu39Z", "reviewer": "Reviewer_SXMf", "summary": "While contrastive learning and negatives-only supervised contrastive learning have similar objectives, this does not necessarily imply that they will learn similar representations through training. Using CKA and RSA, the authors analyze this alignment's emergence in various conditions to provide a better understanding of the similarities between such methods.\nThis is complemented by clear theoretical results that provide intuition on each parameter's influence on this alignment.", "questions": "1) In Figure 2 there seems to be a dynamics change around 10-100 epochs when looking at SCL and CE, any intuition as to why ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Beyond training objective similarity, the authors provide an analysis of representation similarity through training, providing better insights in similarities between CL and NSCL.\n\n- On top of the thorough experiments, the authors prove clear theoretical results that help guide empirical analysis.\n\n- The detailed explanation of the bound obtained in Theorem 1 lines 246-262 is greatly appreciated to provide more practical intuitions in the result.\n\n- The overall work strengthens our understanding of similarities and differences between CL and related supervised learning algorithms.\n\n- Experiments are done at a good practical scale (up to ImageNet using a resnet 50) which helps yield more practically relevant insights.", "weaknesses": "1) My main issue is that it is unclear whether alignment is beneficial in practice. Having CL be more aligned with a **fixed** high performing NSCL model is most likely good in practice. However in the scenarios where alignment increases, it is possible that performance suffers as well. It would be interesting to have more data points regarding alignment vs performance potential tradeoffs.\n\n2) While alignment is studied in the general case, it seems that it can get worse in practical settings.\nLine 415-146 “Alignment increases with higher values of $\\tau$”. Usually for CL temperatures used tend to be low, with performance decreasing as it is increases, suggesting that increasing alignment through temperature comes at a practical cost.\n\n3) Line 079: $C$ is discussed but not introduced yet (I assume it is the number of classes from line 125). It would be good to have its meaning be clear when first discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xdoXlURSyB", "reviewer": "Reviewer_Dkwy", "summary": "This work provides an in-depth theoretical analysis of the alignment between Self-supervised Contrastive Learning (CL) and Negative-only Supervised Contrastive Learning (NSCL). In particular, a bound is derived coupling the CL and NSCL similarity dynamics, demonstrating the CL-NSCL behavioral alignment. Several experiments are also conducted to empirically validate the findings.", "questions": "In line 1054 (Appendix D), the authors stated that \"the difference between the two trajectories stays small when the step sizes are not too aggressive.\" However, Eq. (9) contains an exponential term on cumulative learning rate—does this really guarantee that the trajectories stay close even if the step sizes are not aggressive? Even so, how is this effect accounted for in Thm. 1, while practical algorithms perform parameter-GD instead of similarity-GD?\n\nIntuitively, the similarity-GD and parameter-GD differ significantly as shown in Eq. (5) ($P_t := J_t J_t^\\top$). The update delta of each similarity entry, when trained via parameter-GD, can vary depending on the network gradient evaluated at different input points. I find it somewhat counter-intuitive that the results on similarity-GD can be applied to algorithms utilizing parameter-GD. Therefore, It would be glad to hear the authors' responses on this matter (or please correct me if I misunderstood anything).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Leveraging similarity-space dynamics, this paper bridges CL and NSCL behavior in a theoretically rigorous manner. Empirical verifications are also done thoroughly, with near real-world scale datasets like tiny-ImageNet. Empirical results support theoretical findings.", "weaknesses": "As the authors stated in the conclusion/limitation section, the proposed bounds can be loose under large-scale settings due to the exponential factors on cumulative step size.\n\nMore importantly, please see my question below regarding the discrepancy between similarity-based dynamics and parameter-based dynamics, which I think is a very important step in the derivation of the proposed bound. It perhaps deserves a more detailed explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zSZ1ixxVFp", "reviewer": "Reviewer_DFRi", "summary": "The paper asks whether the objective-level affinity between contrastive learning (CL) and negatives-only supervised contrastive learning (NSCL) holds to the representation trajectories. Specifically, the authors bound the discrepancy between the cosine similarity matrices induced by CL and NSCL under shared randomness, and then derive bounds for CKA and RSA. Experiments with ResNet-50 support the predicted dependencies on the number of classes $C$, batch size $B$, and temperature $\\tau$. In particular, CL aligns much more closely with NSCL than with SCL or cross-entropy.", "questions": "1. How does the analysis extend to class-imbalanced sampling? \n1. Is the bound for $\\|\\Sigma^{\\mathrm{CL}}-\\Sigma^{\\mathrm{NSCL}}\\|_F$ tight?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. A technically sound similarity-space analysis of representations. This is an incremental contribution over recent connections between CL and NSCL [2].\n1. For large $B$ and $C$, the main high-probability result is simplified to\n    $$\n        \\|\\Sigma^{\\mathrm{CL}}-\\Sigma^{\\mathrm{NSCL}}\\|_F =  \\tilde O\\left(\\frac{1}{\\sqrt{B}}\\left(\\frac{1}{C} + \\frac{1}{\\sqrt{B}}\\right)\\right),\n    $$\n    ignoring the explicit dependence on $\\tau$ and $T$. This in turn gives bounds on representation similarity for CL and NSCL. The empirical trends in $C$, $B$, and $\\tau$ match the theory, and the CL-NSCL pair consistently shows stronger alignment than CL-SCL or CL-CE.", "weaknesses": "1. Appendix D explains why the proposed \"similarity-descent\" mirrors small-step gradient descent on parameters. Additional remark of that argument in the main text would help follow the logic from SGD updates to similarity dynamics.\n1. The theory assumes class-balanced sampling. It would be useful to discuss how class imbalance alters the bound, for example by replacing $1/C$ with empirical class priors and indicating the resulting rate.\n1. Prior work reports a non-trivial CKA between SimCLR and supervised ResNet-50 trained independently [1]. It would be great to have additional discussions on the extension of the current theoretical results to a broader class of supervised learning algorithms.\n1. A comment on the tightness of the upper bound for $\\|\\Sigma^{\\mathrm{CL}} - \\Sigma^{\\mathrm{NSCL}}\\|_F$ would be helpful.\n\n**Minor Comments**\n\n1. Line 308: replace \"GPUs\" with \"GPU\".\n1.  \"ImageNet-1K\" (line 84) vs \"ImageNet-1k\" (Figure 3 lower right)\n1. The per-anchor CL and NSCL loss definitions would be improved for clarity.\n1. Move the ``Additional notation for high-probability factors'' so it appears immediately before Theorem 1.\n\n[1] Grigg, T. G., Busbridge, D., Ramapuram, J., \\& Webb, R. (2021). Do self-supervised and supervised methods learn similar visual representations? arXiv:2110.00528.\n\n[2] Luthra, A., Yang, T., \\& Galanti, T. (2025). Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning. arXiv:2506.04411.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XMGc6uHf5m", "reviewer": "Reviewer_nMEv", "summary": "This paper studies whether self-supervised contrastive learning (CL) and a supervised counterpart, Negatives-Only Supervised Contrastive Learning (NSCL), produce similar representations during the training process. The authors theoretically prove that CL and NSCL maintain close alignment in their representation spaces under realistic conditions. They provide probabilistic bounds for alignment metrics such as centered kernel alignment (CKA) and representational similarity analysis (RSA), showing that representation similarity improves with larger numbers of classes, higher temperature, and certain batch size conditions. Empirical results confirm the theory: CL and NSCL representations become increasingly aligned as model scale and temperature grow, with NSCL tracking CL more closely than other supervised objectives.", "questions": "See Weaknesses.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe theoretical proofs and analysis are thorough and well-founded, providing a clear explanation of the differences between contrastive learning and negative-only supervised learning.\n2.\tThere is a strong alignment between the theoretical and empirical results.\n3.\tThe paper is well-written, with a clear and accessible logic flow that is easy to follow", "weaknesses": "1.\tMy primary concern is the lack of clear practical insights derived from the theoretical analysis in this paper. For instance, [1] identifies that downstream classification performance depends on labeling errors and the connectivity of the augmentation graph, while [2] and its follow-up works analyze the role of negative samples in contrastive learning. However, in this paper, what practical benefits can we gain from the theoretical relationship between CL and NSCL, especially considering that we have no access to labeled data in self-supervised learning? In other words, what real-world applications or improvements in contrastive learning can be derived from the theoretical framework presented here?\n2.\tI'm uncertain whether the theoretical analysis offers additional advantages over existing work. Specifically, [2] also characterizes the training process of contrastive learning and establishes guarantees between contrastive loss (even when it is not optimal) and supervised downstream loss. The authors note that prior works often rely on restrictive assumptions, but several follow-up works have relaxed these assumptions, including the conditional independence assumption. It would be helpful to further discuss how the theoretical analysis in this paper improves to the existing body of work.\n3.\tAnother weakness is the lack of connection between the terms in the theoretical bounds and the design choices in contrastive learning. For example, while the paper shows there is still a gap between CL and NSCL, it is unclear which specific terms in the bounds contribute to this gap. Additionally, how might we modify these terms to close the performance gap between the two methods?\n4.\tThe empirical results show that the performance of CL and NSCL is still significantly below that of supervised contrastive learning, particularly on datasets like ImageNet. This raises the question of whether focusing only on supervised information in negative pairs limits the potential of the analysis. For instance, [3] directly compares the gap between contrastive learning and supervised contrastive learning. Besides, it is impractical to only use NSCL if we have access to supervised information.\n\n[1] HaoChen J Z, Wei C, Gaidon A, et al. Provable guarantees for self-supervised deep learning with spectral contrastive loss[J]. Advances in neural information processing systems, 2021, 34: 5000-5011.\n\n[2] Saunshi N, Plevrakis O, Arora S, et al. A theoretical analysis of contrastive unsupervised representation learning[C]//International conference on machine learning. PMLR, 2019: 5628-5637.\n\n[3] Cui J, Huang W, Wang Y, et al. Rethinking weak supervision in helping contrastive learning[C]//International Conference on Machine Learning. PMLR, 2023: 6448-6467.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)