Hard-Negative Sampling for Contrastive Learning: Optimal Representation Geometry and Neural- vs Dimensional-Collapse
Authors: Ruijie Jiang
Thuan Nguyen
Shuchin Aeron
Prakash Ishwar
Venue: Accepted by TMLR ()

Abstract: For a widely-studied data model and general loss and sample-hardening functions we prove that the losses of Supervised Contrastive Learning (SCL), Hard-SCL (HSCL), and Unsupervised Contrastive Learning (UCL) are minimized by representations that exhibit Neural-Collapse (NC), i.e., the class means form an Equiangular Tight Frame (ETF) and data from the same class are mapped to the same representation. We also prove that for any representation mapping, the HSCL and Hard-UCL (HUCL) losses are lower bounded by the corresponding SCL and UCL losses. In contrast to existing literature, our theoretical results for SCL do not require class-conditional independence of augmented views and work for a general loss function class that includes the widely used InfoNCE loss function. Moreover, our proofs are simpler, compact, and transparent. Similar to existing literature, our theoretical claims also hold for the practical scenario where batching is used for optimization. We empirically demonstrate, for the first time, that Adam optimization (with batching) of HSCL and HUCL losses with random initialization and suitable hardness levels can indeed converge to the NC-geometry if we incorporate unit-ball or unit-sphere feature normalization. Without incorporating hard-negatives or feature normalization, however, the representations learned via Adam suffer from Dimensional-Collapse (DC) and fail to attain the NC-geometry. These results exemplify the role of hard-negative sampling in contrastive representation learning and we conclude with several open theoretical problems for future work. The code can be found at https://github.com/rjiang03/HCL/tree/main

Published in Transactions on Machine Learning Research (04/2025)
Hard-Negative Sampling for Contrastive Learning: Opti- mal Representation Geometry and Neural- vs Dimensional- Collapse
Ruijie Jiang"
Rudjje. edu
Deportment of Electrical and Computer Engineering Tufts University, Medford, MA
Thuan Nguyen"
ngugent1 1 1@etru edu
Department of Engineering, Engineering Technology East Tennessee Stale Universily, Johmson City, TN
Shuchin Acron
shuchin tufts. edu
Department of Electricol and Computer Engineering Tufts University, Medford, MA
Prakash Ishwar
piltuu edu
Department of Electricol and Computer Engineering Boston Universiby, Boston, MA
Reviewed on OpenReview: https: // openreview. net/forunts id- J5
Abstract
For a widely-studied data model and general loss and sample-hardening functions we prove that the losses of Supervised Contrastive Learning (SCL.) Hard-SCL, (HSCL), and Un- supervised Contrastive Learning (UCL) are minimized by representations that exhibit Neural-Collapse (NC), i..., the class means form Equiangular Tight Frame (ETF) and data from the same class are mapped to the same representation. We also prove that for any representation mapping, the HSCL and Hard-UCT. (HUCL) losses are lower bounded by the corresponding SCL and UCL losses. In contrast to existing literature, our theoretical results for SCL do not require consiionnaiiinal independence of augmented views and work for a general loss function class that includes the widely used InfoNCE loss function. Moreover, our proofs are simpler, compact, and transparent. Similar existing literature, our theoretical claims also hold for the practical scenario where batching is used for optimization. We empirically demonstrate, for the first time, that Adam optimization (with batching) of HSCL and HUCL losses with random initialization and suitable hardness levels can indeed converge to the NC-geometry if we incorporate unit-ball or unit-sphere feature normalization. Without incorporating nard-negatives or feature normalization, however, the representations learned via Adam suffer from Dimensional Collapse (DC) and fail to attain the NC-geometry. These results exemplify the role of hard-ncgative sampling contrastive represcntation learning and conclude with several open theoretical problems for future work. The code be found at https: /ttpttttttttt///////////tttttt com Hatpia/n///////////aaaaaaaa
Keywords:
Contrastive Learning, Hard-Negative Sampling, Neural- Colaal-C Authors with equal contribution.

---

Published in Transactions on Machine Learning Research (04/2025)
1 Introduction
Contrastive representation learning (CL) methods learn a mapping that embeds data into a Euclidean space such that similar examples retain close proximity each other and dissimilar examples are pushed apart. CL, and in particular unsupervised CL, has gained prominence in the last decade with notable success in Natural Language Processing (NLP), Computer Vision (CV), time-scries, and other applications. Recent surveys Balestriero et al. (2023), Rethmeier & Augenstein (2023) and the references therein provide a comprehensive view of these applications.
The characteristics and utility of the learned representation depend on the joint distribution of similar (positive samples) and dissimilar data points (negalive samples) and the downstream learning task. In this paper we are not interested in the downstream analysis, but in characterizing the geometric structure and other properties of global minima of the contrastive learning loss under general latent data model. In this context, we focus on understanding the impact and utility of hard-negative sampling in the UCL and SCL settings. When carefully designed, hard-negative sampling improves downstream classification performance of representations learned via CL as demonstrated in Robinson et al. (2020), Jiang et al. (2023), and Long et al. (2023). While is known that hard-negative sampling can be performed implicitly by adjusting what is referred as the temperature parameter in the CL loss, in this paper we set this parameter to unity and explicitly model hard-negative sampling through general "hardening" function that can tilt the sampling distribution to generate negative samples that are more similar to (and therefore harder to distinguish from) the positive and anchor samples. We also numerically study the impact of feature normalization on the learned representation geometry.
1.1 Main contributions
Our main theorelical contributions are Theorems 1, 2, and 3, which, under a widely-studied latent data model, hold for any convex, argument-wise non-decreasing contrastive loss function, any non-negative and argument- wise non-decreasing hardening function to generate hard-negative samples, and norm-bounded representations of dimension least C' - where C is the number of classes in the dataset.
Theorem 1 ostablishes that the HSCL loss dominates the SCL loss and similarly the HUCL loss dominates the UCL loss. This is accomplished by creatively utilizing the Harris comparison inequality. In this context we note that Theorem 3.1 in Wu et al. (2020) is a somewhat similar result for the UCL setting, but for a special loss function and it does not address bard-negatives directly and in the generality that we do.
Theorem 2 is a novel result which states that the globally optimal representation geometry for both SCL and HSCL corresponds to Neural-Collapse (NC) (see Definition 6) with the same optimal loss value. In rontrast to existing works (see related works section 2), we show that achieving NC in SCL and HSCL does not require class-conditional independence of the positive samples.
Similarly, Theorem 3 establishes the optimality of NC-geometry for UCL if the representation dimension is sufficiently large compared to the number of latent classes which, in turn, is implicitly determined by the joint distribution of the positive examples that corresponds to the augmentation mechanism.
A comprehensive set of experimental results on one synthetic and three real datasets are detailed See. 5 and Appendix C. These include experiments that study effects of two initialization methods, three different feature normalization methods, three different batch sizes, two very different families of hardening functions, and two different CL sampling strategies. Empirical results show that when using the Adam optimizer with random initialization, the matrix of class means for SCL is badly conditioned and effectively low-rank, i.e., it exhibits DimennionnllCollapee (DC). In contrast, the use of hard-negatives at appropriate hardness levels mitigates DC and enables convergence to the global optima. A similar phenomenon observed in the unsupervised settings. We also show that feature normalization is critical for mitigating DC in these sottings. Results are qualitatively similar across different datasets, a range of batch sizes, hardening functions, and CL sampling strategies.
2

---

Published in Transactions on Machine Learning Research (04/2025)
2
Related work
2.1 Supervised Contrastive Learning
The theoretical results for our SCL setting, where in contrast to Graf et al. (2021) and Khosla et al. (2020) we make use of label information in positive as well as negative sampling, are novel. The debiased SCL loss in Chuang et al. (2020) corresponds to our SCL loss, but no analysis pertaining to optimal representation geometry was considered in Chuang et al. (2020). Feeney & Hughes (2023) considers using label information for negative sampling in SCL with InfoNCE. loss, calling it the SINCERE, loss. Our theoretical results prove that NC is the optimal geometry for the SINCERE loss. Furthermore, our results and analysis also apply to hard-negative sampling as well, a scenario not considered thus far for SCL.
We would like to note that our theoretical set-up for UCL under the sampling mechanism of Figure 1, can be scen to be aligned with the SCL analysis that makes use of label information only for positive samples as done in Khosla et al. (2020) and Graf et al. (2021). Therefore, our theoretical results provide an alternative proof of optimality of NC based on simple, compact, and transparent probabilistic arguments complementing the proof of a similar result in Graf et al. (2021). We note that similarly to Graf et al. (2021), our arguments also hold for the case when one approximates the loss using batches.
We want to point out that in all key papers that conduct a theoretical analysis of contrastive learning e.g.. Saunshi et al. (2019); Graf et al. (2021); Robinson et al. (2021), the positive samples are assumed to be conditionally i.i.d., conditioned on the label. However, this conditional independence may not hold in practice when using augmentation mechanisms typically considered in CL. settings e.g., Oh Song et al. (2016); Oord et al. (2018); Wu et al. (2018); Chen et al. (2020).
Unlike the recent work of Zhou et al. (2022) which shows that the optimization landscape of supervised learning with least-squares loss is benign, iee. all critical points other than the global optima are strict saddle points, in Sec. 5 we demonstrate that the optimization landscape of SCL is more complicated. Specifically, not only may the global optima not. be reached by SGD-like methods with random initialization, but also the local optima exhibit the Aimensional-Coolapee (DC) phenomenon. However, our experiments demonstrate that these issues are remedied via HSCL whose global optimization landscape may be better. Here we note that Yaras et al. (2022) show that with unit-sphere normalization, Riemannian gradient descent methods can achieve the global optima for SCL, underscoring the importance of optimization methods and constraints for training in CL.
2.2 Unsupervised Contrastive Learning
Wang & Tsola (2020) argue that asymptotically (in the number of negative samples) the InfoNCE. loss for UCL optimizes for a trade-off between the alignment of positive pairs while ensuring uniformity of features the hypersphere. However, non-asymptotic and global analysis of the optimal solution is still lacking. In contrast, for UCL in Theorem 3, we show that as long as the embedding dimension is larger than the number of latent classes, which in turn is determined by the distribution of the similar samples, the optimal representations in UCL exhibit NC-geometry.
Our results also complement several recent papers, e.g., Parulekar et al. (2023), Wen & Li (2021), Wang et al. (2021), that study the role of augmentations in UCL. Similar to thcoretical works analyzing UCL, e.g. Saunshi et al. (2019) and Lee et al. (2021), our results also assumee conditional independence of positive pairs given the label. This assumption may may not be satisfied in practice.
We demonstrate that recent result, viz., Theorem 4 in Jing et al. (2021), that attempts to explain DC in UCL is limited in that under a suitable inilialization, the UCL loss trained with Adam does not exhibit DC (see Sec. 5). Furthermore, we demonstrate empirically, for the first time, that HUCL mitigates DC in UCL at moderate hardness levels. For CL (without hard-negative sampling), Ziyin et al. (2022) characterize local solutions that correspond to DC but leave open the analysis of training dynamics leading to collapsed solutions.
3

---

Published in Transactions on Machine Learning Research (04/2025)
A geometrical analysis of HUCL is carried out in Robinson et al. (2020), but the optimal solutions are only characterized asymptotically (in the number of negative samples) and for the case when hardness also goes to infinity, the analysis seems to require knowledge of supports of class conditional distributions. In contrast, we show that the geometry of the optimal solution for HUCL depends on the hardness level and is, in general, different compared to UCL due the possibility of class collision.
3 Contrastive Learning Framework
3.1 Mathematical model
Notation:
L, 11.... -1, Rdz. C For i,j e iiiiiiiii == 1,6.1,........ ,...,,, and "ty" 0, 1."1+11-.. Ifi j,i:j and ".y are "null".
Let f X Z denote a (deterministic) representation mapping from data space X' to representation space Z C Rdz Tet F denote family of such representation mappings. Contrastive Learning (CL) selects reprosentation from the family by minimizing expected loss function that penalizes "misalignment" between the representation of an anchor sample [(x) and the representation of posilive sample zt f(x*) and simultaneously penalizes "alignment" between and the representations of k negative samples 2,- = f(x,), ), 1;;==1111 =::==:::
We consider CL loss function E, of the following general form. Definition 1 (Generalized Contrastive Loss).
eg (,,,2",=TIN) = 1/ (=- (= ==)))
(1)
where 2/ IR* + is a conver function that is also argument-wise non-decreasing (i.e., non non-decreasing with respect to each argument when the other arguments are held fized) throughout TR.
This subsumes and generalizes popular CL loss functions such as InfoNCE and triplet-loss with sphere- normalized representations. InfoNCE corresponds to dh (t = log(a Ex_ e²t) with a > 0' and (tr.) =
E max(t, + a, 0), > 0, the triplct-loss with spherc-normalized representations. However, some CL losses such as the spectral contrastive loss of HaoChen et al. (2021) are not of this form.
The CL loss is the expected value of the CL loss function:
L (f) == E (o 14 14, (f(x), f(x)), (()))))) ((
where Pcz (z, at xxTh) the joint probability distribution of the anchor, positive, and & negative samples and is designed differently within the supervised and unsupervised settings as described below.
Supervised CL (SCL): Here, all samples have class labels: a common class label y E y for the anchor and positive sample and one class label for each negative sample denoted by y, E y for the i-th negative sample, 1,...,k. k. The joint distribution of all samples and their labels is described in the following equation:
P PSCl (y, v, xt, Mi 2 ( = X g(2,, x =l1))))))))))(>)) II rl,, (v) s(x, 1, 1=1
(2)
r(y, (v) = 10,, + y)
(3)
(1- X₂)
where A, E (0, 1) for all y y the marginal distribution of the anchor's label and (y-) the conditional
probability distribution of any negative sample x given its class y
This joint distribution may be interpreted from a sample generation perspective as follows: first, common
class label E y for the anchor and positive sample is sampled from a class marginal probability distribution X.
This is the bog-summ-xpponntaaa function which convex over R for all 2 and strictly convex over R if 0.
4

---

Published in Transactions on Machine Learning Research (04/2025)
Then, the anchor and positive samples are generated by sampling from the conditional distribution q(a x [y). Then, given x, rt and their common class label y. the l negative samples and their labels are generated in conditionally HID manner. The sampling of v . for each i, can be interpreted as first sampling a class label V, different from y in a manner consistent with the class marginal probability distribution À (sampling from distribution r(y, 1y)) and then sampling 27 from the conditional probability distribution s(-l-) of negative samples given class v,- Thus SCL, the & negative samples are conditionally ITD and independent of the anchor and positive sample given the anchor's label.
In the typical supervised setting, the anchor, positive, and negative samples all share the same common conditional probability distribution s(-l-) within each class given their respective labels.
We denote the CL loss in the supervised setting by L (R) L0
Unsupervised CL (UCL): Here samples do not have labels or rather they are latent (unobserved) and the k negative samples are IID and independent of the anchor and positive samples.
Latent labels (classes) in UCL can be interpreted as indexing latent clusters. Suppose that there are C latent classes from which the anchor, positive, and the k negative samples can be drawn from. Then the joint distribution of all samples and their latent labels can be described by the following equation:
PUCL PUCI(y, (,, , Wis TTx) = X, g00, at (V) II rr( ((
(4)
where X is the marginal distribution of the anchor's latent label, r(-) the marginal distribution of the latent labels of negative samples, and s(>=( ) is the conditional probability distribution of any negative sample x given its latent label y- We have used the same notation as in SCL (and slightly abused it for r(-)) in order to make the similarities and differences between the SCL and UCL distribution structures transparent.
In the typical UCL setting, r r= x and the conditional distribution of x given its label y and the conditional distribution of x- given its label y are both s(-ly) which is the conditional distribution of any negative sample given its label.
Independence relationships between anchor, positive and k negatives in the typical UCL setting: To preclude potential confusion, we note that the joint distribution given by (4) and the typical UCL setting described above imply the following independence relationships between samples. The anchor-positive-pe independent of the k negatives. The & negatives are IID. But the anchor sample and the positive sample are dependent through their common shared latent class label. In the typical UCL setting, the anchor and positive samples are conditionally IID given their shared label, but they are not (unconditionally) ITD. Tn fact, their dependence through their sharod label is crucial. This remains true even in the balanced scenario where all classes are equally likely, i.e,, X; = 1/C for all i. Overall, the (k 2) samples (anchor, positive, k negatives) are not IID.
is
We also note that the number of latent classes, C, in the UCL setting, is an unknown intrinsic property of the underlying sampling mechanisms generating the data and is not a tunable parameter of a neural network model used to learn mapping from data space to representation space.
We denote the CL loss in the unsupervised setting by (k) (()
Anchor and positive samples: For the SCL scenario, we will consider sampling mechanisms in which the representations of the anchor and the positive sample z have the same conditional probability distribution s(-ly) given their common label (sce (2)). We will not, however, assume that x and xt are conditionally independent given y. This is compatible with settings where x and x are generated via IID augmentations of a common reference sample gref as in SimCLR Chen et al. (2020) (see Fig. for the model and Appendix B for a proof of compatibility of this model).
For UCL, we assume the same mechanism for sampling the positive samples as that for SCL, but the latent label y unobserved. Further, the negative samples are generated independently of the positive pairs using
5

---

Published in Transactions on Machine Learning Research (04/2025)
x²
y
ree
x
U
rref-
Figure 1: Graphical model for augmentation and negative sampling for SCL and UCL scttings used in practical implementations such as Sim-CLR.
the same mechanism, e.g,, IID augmentations of an w-e/- chosen independently of xre/ (see Fig. 1). Thus, the anchor, positive, and negative samples will all have the same marginal distribution given by Σ X,x(110).
3.2 Hard-negative sampling
Hard-negative sampling aims to generate negative samples whose representations are "more aligned" with that of the anchor (making them harder to distinguish from the anchor) compared to given reference negative sampling distribution (whether unsupervised or supervised). We consider a very general class of "hardening" mechanisms that include several classical approaches as special cases. To this end, we define a hardening function as follows.
Definition 2 (Hardening function). 7 R - IR hardening function if it is non-negative and argument- wise non non-decrgg throughoul R
As an example, 1(( = -T e B > 0, is an exponential tilting hardening function employed in Robinson et al. (2021) and Jiang et al. (2023).
Hand-negative SCL (HSCL): From (2) it follows that in SCL, pa (r, Ey-err(e r(y (g) st>
p(
PSCl (x] is the reference negative sampling distribution for one negative
(1) sample and p( xx,xx, +t, .y) p(
P'sc (2, ly) is the reference negative sampling distribution P
for k negative samples. Let 17 be hardening function such that for all X and all y y,
7(, y. f) = E -,, II ( (n((()) f(x]).... f(x) f(x ) E (0).
Then we define the harder negative sampling distribution for SCL as follows. Definition 3 (r-harder negatives for SCL).
P PHSCL (1 (2 xt f).f) = n(((s) T((()) ((T))....... ((x) ((TT) ((TT)
k II 1=1
PSCL (2,71)) ly).
(5)
f)
Observe that negative samples which are more aligned with the anchor in the representation space, i.e., f (x) fx;7) is large, are sampled relatively more often in Piseer than in the reference Pser. because is argument-wise wise non-decreasing throughout #
In HSCL, xI are conditionally independent of x given and y but they are not conditionally independent of given y (unlike in SCL). Moreover, "I may not be conditionally IID given (x, if the hardening function is not (multiplicatively) separable. We also note that unlike in SCL, PHSCL depends on the representation function f-
We denote the joint probability distribution of samples and their labels the hard-negative SCL setting by PHscl and the corresponding CL loss by (k) (f).
Hard-negative UCL (HUCL): From (4) follows that in UCL, p(u |a, x .xtt)) v) p(x-) Ey-ey r4y )s(r (s(x_y11) (g)) =: Pucu (x is the reference negative sampling distribution for one negative sample

---

Published in Transactions on Machine Learning Research (04/2025)
and p(= (z, xt .2) p(x = II. Pucc (2,) is the reference negative sampling distribution for k negative samples. Let be a hardening function such that for all x E X,
(a, f) = B " IDDeeL
[n(((z) f(27).... f() ((g))) (0)
(6)
Then we define the harder negative sampling probability distribution for UCL as follows. Definition 4 (1-harder negatives for UCT...
Phucl (T xx f) n(((1) ((T)) M(TT)
(oc)(
(7)
(x,
Again, observe that negative samples which are more aligned with the anchor in representation space, i.c., [<)) (xx)) is large, are sampled relatively more often in PHUCe than in the reference PUCL because v) is argument-wis wise non-decreasing throughout g
In HUCL, FI& are conditionally independent of x+ given æ, but they are not independent of x (unlike in UCL.. Moreover, Th may not be conditionally ITD given if the hardening function is not (multiplicatively) scparable. We also note that unlike in UCL, Puuci. depends the represcntation function f.
We denote the joint probability distribution of all samples and their (latent labels) in the hard-negative UCL setting by PICCi and the corresponding CL loss by [(k) [UCLL(f).
4 Theoretical results
In this section, we present all our theoretical results using the notation and mathematical framework for CL described the previous section. We will focus on general CL. loss Section 4.1 and Section 4.2 and then extend results to sample-based empirical and batched empirical losses in Section 4.3.
4.1 Hard-negative CL loss is not smaller than CL loss
Theorem 1 (Hard-negative CL versus CL losses). Let v/r in (1) be argumenl-wise non-decreasing over T& and assume that all expectations associated with (CC((f) L (k) finite. Then, for all f and all ki
(H) Seg (f), L HS erist and are
2 L and (ik) (
(6) (f).
We note that convexity of bh is not needed in Theorem 1. The proof Theorem is based on the generalized (multivariate) association inequality due Harris, Theorem 2.15 Boucheron et al. (2013) and its corollary which are stated below.
Lemma (Harris-ineqqualtyy, Theorem 2.15 in Boucheron et al. (2013)). Let g R + R and h R - IR be argument-ute wise non-decreasing throughout R If u1 IIDp p then
E U1 IID ( MLu E " HD www.iiiiplu..... [ ui IID (h(
whenever the expectations exist and are finite.
Corollary 1. Let R TR be non-negative and argument-wise non- non-decreasing throughout TR such that :=
E - - (n E (0, oo). Let Pu (1r)) = v((nn)) I p(u,). If g R2* + R is argument-wise non-decreasing throughout && such that "na IID, pnn (g(u,))) exists and is finite, then
B "La-Da [ "La HD
l (g((
Proof
E u1 [

---

Published in Transactions on Machine Learning Research (04/2025)
E u 1p g(u mu
ui HID (
2 II
" HD l
where the inequality in the second step follows from the Harris-inequality (see Lemma 1).
Proof of Theorem I. The proof essentially follows from Corollary by defining u := u- f(=) for 1: k defining 92 22 (# := 6,(4, - f(x) (()) f(x*),...,)
"* - [())T((("))) noting that "1.8 are conditionally IID given (x, x) in the UCL setting and conditionally IID given (a, xt v) in the SCL setting, and verifying that the conditions of Corollary hold.
[ For clarity, we provide a detailed proof of the incquality L'sc (f) 2 inequality L LHver(f 2 (f) parallels that for the (more intricate) supervised setting and is omitted.
I (f). The detailed proof of the
(k) (f) =
( (I(=T) ((s)))))(((t)..,..... (1(x)) ((x [(x+)))) [(ot)))
= =E(ntt
E
La Pusc (T Taat wi)
psc (f(x)) -(x=)))(x=)))))))))) f(x*)((+++)))))))) )).... f(x) (f(x))) f(x"1))
n(f()) f(x f((., ffxl)((())))
(8)
(1,111)
2 Bizatt w) E UD
soon Glv) (T((f())((((>)) - f(xx))) ))...............). f(x) (f(x)) -f(x))))(+)))
f(x+1)))
E "
ID (DD (1v) ((((x)) f(zT).... {((T))
(9)
v1..v.f)
E( w)) [BrTa- P3oz (12) thx (a(())
- (f(=)) f(x s(x))))
(10)
-LSO((1 (f)
(k)
where (8) follows from (5) which defines PHSCL (9) follows from the application of the Harris-incuulity (sce
Lemma 1) to the inner expectation where and x are held fixed, and (10) follows from the definition of (0, V. J' in (6).
Theorem 1 is used in Theorem to show that both SCL and HSCL achieve NC.
4.2 Lower bound for SCL loss and Neural-Collapse
Consider the SCL model with anchor, positive, and negative samples generated as described in Sec. 3.1. Within this selting, we have the following lower bound for the SCL loss and conditions for equality.
Theorem 2 (Lower bound for SCL loss and conditions for equality with unit-ball representations and equiprobable classes). In the SCL model with anchor, positive, and negative samples generated as described in (2) and (3), let (a) X , 1/ for all y y (equiprobable classes), (b) {z Rd ||=||| S 1} (unit-ball representations), and (c) the anchor, positive and negative samples have a common conditional probability distribution s(.1-) within each class given their respective labels. If v/ is conver function that is also argument-wise non-decreasing throughout Rek then for all f X Z,
L (0) 2 1/b (T-1))... (-1)
(11)

---

Published in Transactions on Machine Learning Research (04/2025)
For given f F and all y y, let
yy 1- E >>e()))) [f(z)]
(12)
If a given f E satisfies the following additional condition:
Equal inner-produed class means: Vj
==
(13)
then equality will hold in (11), i.e., additional condition (13) sufficient for equality in (11). Additional condition (13) also implies the following properties:
(i) zero-sum class means: Σjey 1 = 0,
(8) unit-nomy class means: Vj E 1111,111=11
(ti) dg C-1.
(tv) zero within-class variance: for all j E y and all 1: k, f(xt) = =1
Pr(f(=,) = P,,,, = =}), and
(0) The support sets of s(-ly) for all y E y must be disjoint and the anchor, positive and negative samples must share common determinislic labeling function defined by the support sets.
(vi) equality of HSCL and SCL losses:
(k) L (H)
(b( (.)C Un (1- (o.......))
If 2/ is strictly convex function that is also argument-wise strictly increasing throughout R$, then additional condition (13) is also necessary for equality to hold in (11).
Theorem 2 has a necessary part and a sufficient part. The sufficient part states that if there is an f such that the equal angles condition (13) holds (and we note that this is a property of the family of representation maps and the dataset, but the loss function), then the lower bound (11) will be achieved for any general th (convex and argument-werr non-decreasing) and we will have other properties (i)-(v)) as well. The necessary part of Thcorem 2 (the last sentence) states that the only way can get equality in the lower bound is to have the equal angles condition (13).
We note that the results of Theorem 2 hold for any loss function that is convex and argument-wise non- decreasing (see Definition 1), which includes the triplet loss, and is not confined to just the InfoNCE and similar loss functions. In the "Empirical SCL loss" paragraph of Section 4.3, we discuss how condition (13) can be guaranteed provided the family of representation mappings (defined by neural network) has sufficiently high capacity. This holds irrespective of the loss function.
Proof We have
T, T
(f)
E aa -T. (1(z)) - (()))))).))))))))))))))))))))))) - ((=))))))
E
aaa IT [1) (f))) [(x
(xx) 11]
(14)
th (E,
2, [s(x) ((x ((()){(())))
I,...,, 2 [f(x) [(x - -
(15)
the (E
E x {f()) T f(x 1b, v - 1,..,, E
[f(x) [_,,_[((x))
E
[f(x) f(. 1)
(16)
v.Ui
2.
vie Σ
a pr
I,.... Σ
(17)
C(C-1)
CIC-13
j.
1112 E |1u1|
1E asll"
-111²-E Eney 11M
1/e
-1
(18)
C(C-1)
C(C-1)
2 1/
0-C - 1
0-C 0-C-1
-1)
(19)
c[C-T]
c(C-1)
= 2/
(0-1)..... [0-1]
-C
CC
(20)

---

Published in Transactions on Machine Learning Research (04/2025)
which is the lower bound in (11). Inequality (14) is because 2/2 is argument- non-decreasing and I (a)TT((*)) S 1 by the Cauchy-Schward incquality since [If( (I(+t)) I (unit-ball representations). Inequality (15) is Jensen's inequality applied to the convex function dx- Equality (16) is due to the law of iterated expectations. Equality (17) follows from (2). (3), and the assumption of equiprobable classes. Equality (18) is because Σjtey H P Ey.ey,.fe , Me ΣEey 1la,11? Inequality (19) is because 2/2 is argument-wise non-decreasing and the smallest possible value of || j|E 12,11 is zero and the largest, possible value of 1|,, 112 one (unit-ball representations): Jensen's inequality for the strictly convex function || ||2 together with [=])) < 1 (unit-ball representations) imply that for all y E V, we have
II 11 |I (>)((())))) [x))]]] E am (( (z) 1.
(21)
Finally the oquality (20) because C(C-1) (C-1)'
(1) and (ii) Proof that additional condition (13) implies zero-sum and unit-norm class means: Inequality (21) together with condition (13) implies that
0 0
Σ Hell = Σn Me=
Dnpme=
Σ
Ss He +
Σ
| S
C(C-1), -
1==
C-1
JEY
JAEY
jEY
JEY
Thus Ejey M 112 = 0 and for all j jED V, 11,,11=11
(iit) Let M := (e E RC Then from (13) and (ii), the gram matrix M M = - where Lo is the C' x C' identity matrix and Lc is the C' x 1 column vector of all ones. From this it follows that M M has one eigenvalue of zero corresponding to eigenvector 1 and C - 1 eigenvalues all equal to T corresponding to (C - 1) orthogonal eigenvectors spanning the orthogonal complement of l Thus, M
has C - 1 nonzero singular values all equal to / c and rank equal to C C-11 c-15dg-
(iv) Proof that additional condition (18) implies zero within-class variance: We just proved that additional condition (13) together with the unit-ball representation constraint implies unit-norm class means. This, together with (21) implies that for all y E y,
1 11, 1|2 = ||E (m)()))) [[())]] < E (- (1) (0)
< 1.
(22)
This implics that we have equality in Jensen's incquality applied to the strictly convex function g(z) = | with 2 = fz), and the push-forward conditional probability measure f (s(xly)). Equality can occur iff, with probability one given y,, we have fxx = M (since || 1| is strictly convex).
Since the anchor, positive and negative samples all have a common conditional probability distribution s(I-) within each class given their respective labels, it follows that for all j E y and all i k, 1ylb, j) = 1. Moreover, since the
Pr(f(x) nyly 5) Pr(f(=*)= Myly j) Pr(f(=,) anchor and positive samples have the same label, for all j E V, with probability one given y = j, we have [(x)= -(x+))+)) 11.
Proof that additional condition (13) is sufficient for equality to hold in (11): From the proofs of (1), (ii), and (ir) above, additional condition (13) holds, then we showed that with probability one given y = we have f(x) = f(x+) = Ht (see the para below (22)). This equality of f(x) and f(*+) is a conditional equality given the class. Since this is true for all classes, it implies equality (with probability one) of f(x) and f(x)) without conditioning on the class:
Pr(f(x)= = f(x))) = Σ Pe(s(o)=((ot) v=t) = 1.
(23)
C
JEY
From (23) we get
Pr(f()TT(<>)) =) Pr(1][())11 = =1)= Σ
Pr( Pe]ld(a)|=-115))) 1
(24)
Jey
10

---

Published in Transactions on Machine Learning Research (04/2025)
since f(x)= = f(xx))==))))))))))))))) M with probability one given y = j and I|u, 1. Equality in (14) then follows from (23) and (24). Moreover, due to zero within-class variance we will have
with probability one, for all i == = f(x) f(17)= f(x)) Ho My = -1 C-1'
(25)
and then we will have equality in (15) and (19). Therefore additional condition (13) is a sufficient condition for equality to hold in (11).
(v) Proof that support sets of s(-1y), y E y are disjoint: From part (ie), all samples belonging to the support set of s(-ly), y E V, are mapped to "y by S. From part (ii) and condition (13), distinct labels have distinct representation means: for all v. E )/ if y' # y, then Hy * Hyy. Therefore the support sets of s(-ly) for all y E y' must be disjoint. Since the anchor, posilive, and negalive samples all share a common conditional probability distribution s(-|-) and the same marginal label distribution X, it follows that they share common conditional distribution of label given sample (labeling function). Since the support sets of s(Ily) for all y E y are disjoint, the labeling function is deterministic and is defined by the support set to which a sample belongs.
(wi) Proof of equality of HSCL and SCL losses under additional condition (13): under the equal inner- product class means condition, with probability one f(x) [(=)) (())
simultaneously for all i = 1 k and = n -- :.). a constant. Consequently, for all x, y and the given J,
must have 71. f) Ka-...... 2 which would imply that (see Equation 5) P Pcs (T |a, a.t,..73 V. f) ( (1=fI)...... (.))).)..))))))))) where the last equality is because additional condition (13) is sufficient for equality to hold in (11).
IT, Pscr (I.1y) (v) and therefore L / [H)c 1.
Proof that additional condition (13) is necessary for equality in (11) 1 is strictly conver and argument- wise strictly increasing over R* If equality holds in (11), then it must also hold in (14), (15), and (19). If 1i is argument-wing wise strictly increasing, then equality in (19) can only occur if all class means have unit norms. Then, from (22) and the reasoning in the paragraph below it, we would have zero within-class variance and equations (23) and (24). This would imply equality in (14). If ty is strictly convex then equality in (15), which is Jensen's 's inequality, can only occur if for all i = 1 k, Pr(sT (x))(()) = =,) = 1 for some constants 3 _1 Since (z, 27) has the same distribution for all i, it follows that for all i, 3, = B for some constant B. Since we have already proved zero within-class variance and the labels of negative samples are always distinct from that of the anchor, it follows that for all j # f, we must have 11 11g- = B. Since we have equality in (14) and v, is argument-wise strictly increasing, we must have B = which implies that additional condition (13) must hold (it is a necessary condition).
Remark 1. We note that Theorem 2 also holds if we have unil-sphere representations (a stronger constraint) as opposed to ball representations, i.e., if Z {2 E Rdz : || = =}: the lower bound (11) holds since the unit sphere is a subset of the unit ball and equality can be attained with unit-sphere representations in Theorem 2.
Remark 2. Interestingly, we note that inequalily (19) and therefore the lower bound of Theorem also holds if we replace the unit-ball constraint on representations [[f(())] s with the weaker requirement I - 11,
Definition 5 (ETF). The equal inner-product, zero-sum, and unil-norm conditions on class means in Theorem 2 define a (normalized) Equiangular Tight Frame (ETF) (see Malozemov & Pevnyi (2009)).
Definition 6 (CL. Nolral-Collapse (NC)). We will say representation map f(-) exhibits CL Neural- Colaase if it has zero within-class variance as in condition (iv) of Theorem 2 and the class means in representation space form a normalized ETF as in Definition 5.
Remark 3. The term "Neural- Collapse" was originally used for representation mappings implemented by deep classifier neural networks (see Papyan et al. (2020)). However, here we use the term more broadly for any family of representation mappings and within the contect of CL instead of classifier training.
The following corollary is partial restatement of Theorem 2 in terms of CL Neural-
11

---

Published in Transactions on Machine Learning Research (04/2025)
Corollary 2. Under the conditions of Theorem 2, equality in (11) is attained by any representation map f that exhibits CL Neural- Colla-S Morcover, if V is striclly convee and argument-wist strielly increasing over R&, then equality in (11) is allained by a represenlation map J, if, and only if, it exhibits CL Newral- Colla-Ce
4.3 Empirical and batched empirical SCL losses
All results presented till this point pertain to the expected loss and not sample-based based empirical loss or batched empirical loss. In this section we demonstrate that results that were established in previous sections for expected loss also hold for the empirical and batched empirical losses.
Empirical SCL loss: Theorem 2 also holds for empirical SCL loss because simple averages over samples can be expressed as expectations with suitable uniform distributions over the samples. If the family of representation mappings F has sufficiently high capacity (e.g., the family of mappings implemented by sufficiently deep and wide feod-forward neural network) and Vy E V, s(-ly) is discrete probability mass function (pmf) over finite set (e.g., uniform pmf over training samples within each class) with support-sets that disjoint across different classes, then the equal inner-product condition (13) in Theorem 2 can be satisfied for a suitable ʃ in the family If either convexity or monotonicity of 4y is not strict, e.g., 14 (1.)) = E max(t, + a, 0}, a > O, then it may be possible for representation map to attain the lower bound without exhibiting CL Neural-Collapse.
Batched empirical SCL loss: We will now show that representations that exhibit CL Neural-Collapse will also minimize batched empirical SCL loss under the conditions of Theorem 2. Here, the full data set has balanced classos (equal number of samples in each class) but is partitioned into B disjoint nonempty batches of possibly unequal size. Lct b denote the batch index and n, the number of samples in batch b. Let E denote the empirical SCL loss in batch b and
a.t lb F
L (f)
Σ
E'me E b=1 n
TT 16 -f())))()) ((())))).((())))))))) (f(x)))) f(x+)(())))
a.
the overall batched empirical SCL loss. Note that in a given batch the data may not be balanced across classes and therefore we cannot simply use Theorem 2, which assumes balanced classes, to deduce the optimality of CL Nelral- representations.
We lower bound the batched empirical SCL loss as follows:
B
L
(f)
E
(f(z5) f((+)))
a
6m
1
- 1, f(x))- f(x) f(xx)) - 11]
E
0 (fxx)
(26)
n
B
B
2
Σ
E
10 [f(x) f(x - 1....,
Σ
1
E [f(x) f(x
(27)
n,
Tb
2,07
1=1
b=1
2/2
E
[[(x)]((x))) )
- )
1,.... E 2
[s(x) [(x)) -1 11)
(28)
2/a [--I.....
-C -
(29)
where inequality (26) holds for the same reason as in (14), inequality (27) is Jensen's inequality applied to dh which is convex, and equality (28) is due to the law of iterated (empirical) expectation. The right side (27) is precisely the right side of (15) and therefore (29) follows from (15) (20). From the above analysis it follows that the arguments used to prove Theorem can be applied again to prove that the conclusions of Theorem 2 and Corollary 2 also hold for the batched empirical SCL loss.
12

---

Published in Transactions on Machine Learning Research (04/2025)
4.4 Lower bound for UCL loss with latent labels and Neural-Collapse
Consider the UCL model with anchor, positive, and & negative samples generated as described in Sec. 3.1. Within this setting, we have the following lower bound for the UCL loss and conditions for equality.
Theorem 3 (Lower bound for UCL loss with latent labels and conditions for equality with unit- ball representations and cquiprobable classes). In the UCL model with anchor, positive, and negative samples generated as described in (4), lel (a) X, - è for all y E y (equéprobable classes), (b) = {= E R : || S 1} (unit-ball representations), (c) the anchor, positive and negative samples have a common conditional probability distribution s(-l-) within each latent closs given their respective labels, (d) = X in (4), and (e) the anchor and positive samples are conditionally independent given their common label, i.e., q(x, x+ (y) = ((zl))s((* in (4). 1f 1/ is a convex function that is also argument wise non- non-decrg throughout IR* then for all f x + 3.
L
(k) (C)
Σ 1/ (
(CC1(N #2) (C-1)
-0 #v) (C-
(30)
Ck+I
where 1() is the indicator function. For a given J E and all y E E, let y be as defined in (12). If a given f E F satisfies additional condition (13), then equality will hold in (30), i.e,, additional condition (13) is sufficient for equality in (30). Additional condition (13) also implies the following properlies:
(i) zero-sum class means: Ejey 1 = 0,
(11) unil-norm class means: Vj E V ||, 11
(tt) dz C-1.
(fv) zero willhin-class variance: for all j y and all i= 11 k, (at) =1
Pr(f(=;) - M,,0, =},, and (v) The support sets of s(-ly) for all y E y must be disjoint and the anchor, positive and negative samples must shore common deterministic (latent) labeling function defined by the support sets.
If 1/2 is strictly convex function that is also argument-wise strictly increasing throughout R. then additional condition (13) is also necessary for equalily to hold in (30).
Proof For i = 1 k; we define the following indicator random variables b. 10, # v) and note that for all ==1: :11ke k, b, is deterministic function of (y.87). Since Il (yr&) and V1.1 ~ IID Uniform()), it follows that bi ~ IID and independent of We then have the following sequence of incqualities:
(16) L (f) = IE
n,at TTA [v, (f(x) (f(x7)- f( f(x+)) (()) f(x+)))]
2 R
MMT [Wa. (B "2, ( (xx) T [xx]]] - B
2, (o(o))) [ [(x*)],....
E o (((p)) 10, [1xx) [(x7))
E - (((()T((()))))
(31)
= E =E
(e) (1) Mop My My My M My - (/
(32)
M.
2 B [1% (B(12), Mon 11 11210 Muh.- ... Blue Maj - -I1,,,1 11,b 1
(33)
R [ett (1((2 (1nn,,1,, "0, ||m,11 11 ...... Elu, M, -Ile,, 11,,,11110, 112)
(34)
= -En, [eta (b, Blu Mo - || 1110, = 1]. ....... by E(u My -11u 1116, = 11)]
(35)
2 B [1ba (b, Blu, Poj 1|6 = =1], 1l................... .a....... 62 E(e4 T u
1|b, = 11)]
(36)
=E [ b1 Ens, (1 11
by Er (1] me-1)
(37)
C(C-1)
C(C-1)
11 E, My 1s
|1 -CC--1)
ba E, ~11l"-- [1a,,111 -1-))-
= =B [xaa]
(38)
C(C-1)
C(C-1)
2 As discussed in Sec. all existing works that conduct a theoretical analysis of UCL make this assumption.
13

---

Published in Transactions on Machine Learning Research (04/2025)
2 E, [t) -10-C--CC-I)
dy b(00---ic-1))
(39)
bii a(C-1)
Q(C-1)
= -Bene (//(( CIC-1) y (0000111
(1000-0)
Bis [ (0-1)
Cb.
Cbb
(40)
[C-1]
Σ 1
Ci(y, Fv)
Clly #l)
(41)
-OFFT
(C-1)
(C-1)
M-llick EY
where the validity of each numbered step in the above sequence of inequalities is explained below.
Incquality (31) is Jensen's incquality conditioned on y, f applicd to the convex function V/g. Equality (32) holds because for every i, we have x and x, are conditionally independent given y and y, (per the UCL model (4)), z and x are conditionally independent given their common label (assumption (d) in the theorem statement), and the class means in reprosentation space are as defined in (12). Incquality (33) is Jensen's 's inequality conditioned on b1s applied to the convex function 4/2 Equality (34) holds because for all i = 1: k (U.1T) 11 and ug T fy, - 11u, y 11 = O. Therefore the expressions to the right of the equality symbols in (34) and (35) match when b, =0 0 and when b; =1. Inequality (36) is because 1/2 is non-decreasing and IIr,,1| S for all y because all representations are in the unit closed ball. Equality (37) holds because for each V. ~ IID Uniform(J)) and * v,5 when b2 1. Equality (38) follows from elementary linear algebraic operations. Incquality (39) holds because 2 is argument-wist wise non-decreasing, the smallest possible value for || 1E,,, 1111 is zero and the largest possible value for I|u,,1?, for all j. is one. Equality (41) follows from the definition of the indicator variables in terms of y, Vis and because y, Vi. are IID Unilorm()).
(b1r l + 1b, Equality (35) holds because b, only takes values 0, 1 and if b, 0, then 19,- = 14
1 k,
Similarly to the proof of Theorem 2, if additional condition (13) holds for some f, then properties (i)-(r) in Theorem 3 hold. Morcover, then (23), (24), and (25) also hold and then we will have equality in (31), (33), (36) and (39). Thus additional condition (13) is sufficient for equality to hold in (30).
Proof that additional condition (13) is necessary for equality in (30) if v is strictly conver and argument- wise strictly increasing over R² If equality holds in (30), then must also hold in (31), (33), (36), and (39). If the is argument-wise strictly increasing, then equality in (39) can only occur if all class means have unit norms (which would also imply equality in (36)). Then, from (22) and the reasoning in the paragraph below it, we would have zero within-class variance and (24), which would imply that with probability one for all i, f(=) = Pydn, and f(x) f(x+)= = IIp,11 112 which would imply equality in (31) as well. Equality in (33) together with strict convexity of dx and I|u,112 = 1 would imply that with probability one, for all i, given b, we must have ", = some deterministic function of b, and if ey is also argument-wise strictly increasing, then this function must be such that My ,0 1 = (c-1)
-Ob, due to (40). This would imply that
for all i, we must have n pg 1 Ch. [C1]] =1 City tv) Thus for all y # yr, i.e., b,=1 =1 (and this has (C-1)
nonzero probability), My ",- =1 - c-I T-1 which is the additional condition (13). Thus the additional condition (13) must hold (it necessary condition).
Counterparts of Corollary 2 and results for empirical SCL loss and batched empirical SCL loss can be stated and derived for UCL.
A novel perspective offered by Theorem 3, vis-å-vis existing theoretical works on UCL, is that it relates the structure of the underlying sampling mechanism to the optimal value of the contrastive loss, albeit under certain restrictive assumptions which may not be possible to verify in practice for given sampling mechanism (especially the latent-olass-condiiionaa independence of positive pairs).
One important difference between results for SCL and UCL is that Theorem 3 is missing the counterpart of property (vi) in Theorem 2. Unlike in Theorem 2, we cannot assert that if the lower bound is attained, then we will have I (k) (SC = (1k) (V))()). This is because in the UCL and HUCL settings, the
14

---

Published in Transactions on Machine Learning Research (04/2025)
negative sample can come from the same latent class the anchor (latent class collision) with a positive probability (oz). Then for a representation f that exhibits Neural-Collapse, we cammot conclude that = ()) for some constant B.
with probability one we must have n(f(x) ((TT))...,{()) Deriving a tight lower bound for HUCL and determining whether it can be attained iff there is Veural-Collapse in UCL (under suitable conditions), are open problems.
Neural-Collapse in SCL or UCL requires that the representation space dimension dz - (see part (iti) of Theorems and 3). This can be ensured in practical implementations of SCL since labels are available and the number of classes is known. In UCL, however, not just latent labels, but even the number of latent classes is unknown. Thus even if it was possible to attain the global minimum of the empirical UCL loss with an f exhibiting Neural-Collapse, we may not observe Newral-Collapse with an argument-wise strictly increasing and strictly convex V/2 unless dz is chosen to be sufficiently large.
In practice, even without knowledge of latent labels, it is possible structure that is compatible with (4) and conditions (c) and (d) of Theorem 3, e-g, via IID augnentations of reference sample as in SimCLR illustrated in Fig. 1. However, it is impossible to ensure that the equiprobable latent class condition (a) or the anchor-positive conditional independence condition (e) in Theorem 3 hold or that the supports of s(-l-) determined by the sample augmentation mechanism will be disjoint across all latent, classes. Thus a representatio ation minimizing UCL loss may not exhibit Neural-Collapse even if 4 is strictly convex and argument-winn wise strictly increasing or it might exhibit zero within-class variance, but the class means may not form an ETF (see Nguyen et al. (2024)).
design sampling distribution having a
A second important difference between icalretical results for SCL and UCL is that unlike in Theorem 2, conditional independence of the anchor and positive samples given their common label is assumed in Theorem 3 (condition (d) in the theorem). It is unclear whether the results of Theorem 3 for the UCL setting will continue to hold in entirety without conditional independence and we leave this as an open problem.
Finally, unlike the SCL loss lower bound given by Equation (11) which is a non-decrcasing function of the number of classes C (since bi is argument-wise non-decreasing), the UCL loss lower bound given by Equation (30) is a non-increasing function of the number of latent classes C. We prove this monotonicity property of the UCL loss lower bound in Appendix A for the case when the mumber of negative samples is k= 1. We also provide empirical evidence of the monotonicity property for k = 2,2,3,4,5.
5 Practical achievability of global optima
We first verify our theoretical results for UCL, SCL, HUCL, and HSCL using synthetic data and then investigate the achievability of global-optima three real-world image datasets.
Datasets:
Synthetic Dataset: This comprises three classes with 100 data points per class. The points within each class are IID with a 3072-dimensional Gaussian distribution having an identity covariance matrix and randomly generated mcan vector having IID components that are uniformly distributed over [-1, 1]. The data dimension of 3072 allows for reshaping the vector into 32 32 tensor which can be processed by ResNet.
Real-world Image Datasets: This comprises CIFAR10, CIFAR100 (Krizhevsky et al., 2009), and
(Te & Yang, 2015). These datasets consist of 32 x 32 x 3 images across 10 classes (CIFAR10), 100 classes (CIFAR100), and 200 classes (TinylmageNee)) respectively. Similar phenomena are observed in all three datasets. We present results for CIFAR100 here and results for CIFAR10 and TinylmageNet in Appendix C.
Anchor-positive pairs: We explored the following three strategies for constructing positiv pairs:
Using Label Information: For each anchor sample, a positive sample is chosen uniformly from among all samples having the same label the anchor (including the anchor). All samples that
15

---

Published in Transactions on Machine Learning Research (04/2025)
share the same label with the anchor are used to form the positive pairs. Note that the anchor and the positive can be identical.
Additive Gaussian Noise Augmentation Mechanism: For each (reference) sample, we generate the anchor sample by adding IID zero-mean Gaussian noise of varianco 0.01 to all dimensions of the reference sample. The corresponding positive sample is generated in the same way from the reference sample using noise that is independent of that used to generate the anchor.
SimCLR framework: We also report additional results with the SimCLR framework in Ap- pendix C.5 where a positive pair is generated using two independent augmentations from one reference sample.
We note that anchor-positive pair sampling using label information satisfies the assumptions on the sampling mechanism described in Theorem 3. In particular, the positive samples are conditionally IID given their latent. class label (but not unconditionally IID). In sampling via independent additive Gaussian noise augmentations of a reference sample, if the noise level not high, the anchor and positive samples are likely to have the same latent class label as the reference, but they are not guaranteed to be conditionally IID given the label. In the SimCLR framework, the assumptions of Theorem 3 cannot be guaranteed.
Negatives: In all the supervised settings, for a given positive pair, we select k negative samples independently and uniformly at random from all the data within a mini- and all their augmentations (if used), but having labels different from that of the positve pair. In all the unsupervised settings, for given positive pair, we select k negative samples independently and uniformly at random from all the data within a mini-batch and all their augmentations (if used), including possibly the anchor and/or the positive sample. Thus, we do not use label information sample negatives in all the unsupervised settings. We call this random negative sampling.
Hard Negatives: We utilize the InfoNCE loss with the exponential tilting hardening function described in Sec. 3.1. Within each minibatch, when we sample negatives, we use the hardeming function to increase the probability of selecting negative samples that have a higher value of f(x) f(x by a multiplicative factor equal to the hardening function value at family of polynomial hardening functions.
In Appendix C.4 we report additional results for a
Representation family, normalization, k values: We used ResNet-50, He al. (2016), as the family of representation mappings F and set the representation dimension to d = C 1 to observe Neural- Collapee (Definition 5). We normalized representations to be within a unit ball as detailed in Algorithm 1, lines 5-12, in Appendix C. We only report results for k = 256 negative samples, but observed that results change only slightly for all E [32, 512].
Other parametors: We chose hyper-parameter of the hardening function from the set (0, 10, 30, 50) for synthetic data and the set {0,2, 2,5, 10, 30} for real data and trained each model for E = 400 epochs with a batch size of B 512 using the Adam optimizer at learning rate of 10-3, We did not use weight decay. Computations were performed an NVIDIA A100 32 GB GPU.
5.1 Results for synthetic data
Figure 2 summarizes the results for synthetic data. For a representation function f that achieves Neural- Collapse, the values of ( (f*) and L 1 (f*) across all 3 values and the value of 1 are 0.2014, 0.2014, and 0.3935, respectively. These values are obtained by numerically evaluating the lower bounds in Theorem 2 and Theorem 3.
The first row in Figure 2 shows the result using label information for positive pair construction.
We note that label information is used for sampling positive pairs even in the unsupervised settings empirically validate the results of Theorem 3. Sampling positive pairs with label information guarantecs the conditional independence assumption (e) in Theorem 3.
16

---

Published in Transactions on Machine Learning Research (04/2025)
The values of the minimum loss in different settings displayed at the top of Fig. 2. Our simulation results are consistent with our theoretical results. After 200 training epochs, we observed that L LsCL (f) and I. (256)
"HSCL (f) across all 3 values and I. (556) (UCL()) (J) converged to their minimum values. From the figure, we can visually confirm that the representations exhibit Neural-Collapse in SCL, HSCL, and UCL. However, Neural-Collapse was not observed in HUCL as the class means deviate significantly from the ETF geometry.
The second row in Figure 2 shows the results using the additive Gaussian noise augmentation mechanism, where label information not used in constructing positive examples.
We note that the conditional independence assumption (e) in Theorem 3 is not guaranteed in this method for sampling positive pairs. In accordance with our theoretical results, NC is observed in SCL and HSCL (recall that the conditional independence assumption is not needed in Theorem 2). However, NC is not observed in UCL and HUCL, likely due to the lack of ionditional independence. Moreover, the degree of deviation from NC increases progressively with increasing hardness levels.
Contrary to a widely held belief that hard-negative sampling is beneficial in both supervised and unsupervisod contrastive learning, results on this simple syuthetic dataset suggest that not only may SCL not benefit from hard negatives, but UCL may suffer from it. To investigate whether these conclusions hold true more generally or whether there are practical benefits of hard-negatives for SCL and UCL, we turn to real-world datasets next.
Inibelization
HSCL, B-10
HSCL 8-30
HSCL, B 0.2014
HUCL 8-10
HUCL 8-30 0.6531
HUCL, 8-50 50 0,6931
0.2014
0,2014
0,2014
0.3535
0.6931
Irilialication
SCI 0,2014
HSCL B- 0.2019
HSCL 8 30
HSCL, B 0.2028
LCL
HUCL 5-10 0.6925
HUCL 8 30 0.6589
HUCL, 3-99 50 0.6999
0,2027
0,3643
Figure 2: Synthetic dataset results using label information (top row figures) or additive Gaussian noise augmentation mechanism (bottom row figures) for generating anchor-positive pairs: Initial twoo-dimensional representations (left), post-training SCL and HSCL representations and losses at different hardness levels (middle), post-training UCL and HUCL representations and losses at different hardness levels (right).
5.2 Results for real data
To validate the theoretical results on real data, in all four CL settings (UCL, SCT,, HUCT, and HSCL),, for a given anchor x, we randomly sample (without augmentation) the positive sample xt from the class conditional distribution corresponding to the class of z.
We first show that hard-negative sampling improves downstream classification performance. From the first row of Fig. 3, we see that with negative sampling at moderate hardness levels (3 = 5, 10), the classification accuracies of HSCL and HUCL are more than 40% points higher than those of SCL and UCL respectively.
Achievability of Neural-Collapse: We investigate whether SCL and UCL losses trained using Adam can altain the globally optimal solution exhibiting NC. To test this, in line with properties (i), (ii) and condition (13) in Theorems 2 and 3, we employ the following metrics which are plotted in Fig. 3 in rows two through four.
E5 2. Unit-norm metric: 1 EjEr ||0y11 11
1. Zero-sum metric:
17

---

Published in Transactions on Machine Learning Research (04/2025)
3. Equal inner-product metric: c(C-1)
Iu5 kk + o-Ll kIEE kØj
Σ
We note that even though the equal inner-product class mcans condition together with unit-ball normalization implies the zero-sum and unil-norm conditions, we report these three metrics separately lo gain more insight.
According to Theorems 2 and 3, the optimal solutions for UCL, SCL, and HSCL are anticipated to manifest NC. However, our experimental findings reveal a gap between the thcoretical expectations and the observed outcomes. Specifically, in both supervised and unsupervised settings, when leveraging the random negative sampling method, Le., when the negative samples are sampled uniformly from the whole data. in a mini-batch which may include the anchor and positive samples, NC is not exactly achieved: the zero-sum and equal unner-product metrics do not approach zero for all hardness levels (rows 2 and 4 in Fig. 3). This is also supported by the results in Table 1 which shows the theoretical minimum SCT loss value and the practically observed SCL loss values after 400 epochs for different hardness levels. While our theoretical results posit that both SCL and HSCL should have the same minimum loss value for all values of B > O, the practically observed loss value for SCL deviates noticeably from theory. On the other hand, increased hardness in HSCL, especially at B 5, 10, brings the observed loss value close to the thcorctical minimum.
Theory
Empirical
SCL
HSCL
0.3105
B = 0
B = 2
8 5
8 = 10 B = 30
0. 0.3384
0.3106
0.3107 0.3222
Table 1: Comparison of minimum theoretical loss and practically observed loss values after 400 epochs for CIFAR100.
In addition, the manner in which the final values of NC metrics change with increasing hardness levels is qualitatively different in the supervised and unsupervised settings. Specifically, in the supervised settings, increased hardness invariably leads to improved results, and the model tends to approach NC, notably at f = 5, 10, 30. However, in the unsupervised settings, there seems to be just single optimal hardness level (8 = 5 is best among the choices tested).
5.3 Dimensional-Collan Collapse
To gain further insights, we investigate the phenomenon of Dimensionnll-oollpse (DC) that is known occur in contrastive learning (see Jing et al. (2021)).
Definition 7. ( Collels (DC) We say that the class meons p1...... suffer from DC if their empirical covariance matriz has one more singular values that zero orders of magnitude smaller than the largest singulor value.
If dg = 1-,, then under Neural-Collapse (NC), the class mean vectors would have full rank C - in representation space since they form an ETF (see Definition 5). Thus when d, = C - NC + -DC. However, we note that -DC A NC because, for example, the class means could have full rank and satisfy the equal inner-product and unit-norm conditions in Theorem 2 without satisfying the zero-sum condition.
We numerically assess DC by plotting the singular values of the empirical covariance matrix of the class means (at the end of training) normalized by the largest singular volue in decreasing order on log-scale. Results for UCL, SCI., HUCL, and HSCL are shown in Fig. 4. In the supervised settings, (first row and first column of Fig. 4), the results align with our previous observations from Fig. 3. However, in the unsupervised settings (first. row and second column of Fig. 4), while HUCL with high hardness values deviates more from NC compared to UCL in Fig. 3, in Fig. 4 we see that HUCL suffers less from DC.
We must emphasize that observing NC on the training data does not necessarily imply better performance in downstream tasks or improved generalization ability. Our experiments show that using hard-negative
18

---

Published in Transactions on Machine Learning Research (04/2025)
sampling aids the attainment of NC when using iterative methods starting at random initialization. But more significantly, even if hard-negative sampling may not. completely ensure the attainment of the NC solution (which is the optimal solution minimizing the training loss), it could prevent or reduce DC.
5.4 Role of initialization
To gain further insights into the DC phenomenon, we trained model using HSCL with 3 = 10 for 400 epochs until it nearly attains NC as measured by the three NC metrics (zero-sum, unit-norm, and equal inner-product) shown in the second row of Table 2. We call this representation mapping (or pre-trained model) the "near-NC" representation mapping (pre-trained model).
Next, with the near-NC representation mapping as initialization, we continue train the model for an additional 400 epochs under 10 different. settings corresponding to hard supervised and hard unsupervised contrastive learning with different hardness levels. Rows 3-12 in Table 2 show the final values of the three NC metrics for the 10 settings. The resulting normalized singular value plots are shown in the second row of Fig. 4.
Setting
Zero-
Unit-norm
Equal
sum
inner-product
near-NC model
0.012
1.8 x 10-8
0.004
SCL
0.006
2.0 x 10
0.007
HSCL, 5 = 2
0.005
2.2 x 10
0.004
HSCL, 8 = 5
0.007
1.9 x 10
0.002
HSCL, B = 10
0.007
2.0 x 10
0.002
HSCL, 3 = 30
0.004
1.7 x 10
0.001
UCL
0.005
3.9 x 10
0.005
HUCL, _ = 2
0.005
6.1 x 10
0.003
HUCL, 8 = 5
0.004
1.4 X 10
0.002
HUCL, $ = 10
0.093
1.5 X 10 10-3
0.047
HUCL, 3 = 30
0.761
7.9 x 10
0.586
Table 2: Post-training NC metrics for near-NC initialization in different settings.
From Table 2 we note that in all 5 supervised settings, the final representation mappings have NC metrics that are very similar to those initial near-NC mapping. However, the unsupervised settings, especially HUCL for = 10, 30, the unit-norm and equal inner-product metrics of the final representation mappings are significantly larger than those of the initial near-NC mapping. This shows that mini-batch Adam optimization of CL losses exhibit dynamics that are different in the supervised and unsupervised settings and are impacted by the hardness level of the negative samples.
From the second row of Fig. we make the following observations:
SCL and HSCL trained with near-NC initialization and Adam do not exhibit DC or DC is negligible (second row and first column of Fig. 4).
UCL trained with near-NC initialization and Adam also does not exhibit DC, but the behavior of HUCL depends the hardness level B. A larger 3 value appears to make DC more pronounced. This could be explained by the fact that a higher 3 value increases the odds of latent-class collision.
5.5 Role of normalization
Feature normalization also plays an important role in alleviating DC. To demonstrate this, we test three normalization conditions during training: (1) unit-ball normalization, (2) unit-sphere normalization, and (3) no normalization. The resulting normalized singular value plots are shown in Fig. 4 (rows 1, 3, and 4). As be observed, the behavior of unit-sphere normalization is close to that of unit-ball normalization, and
19

---

Published in Transactions on Machine Learning Research (04/2025)
with hard-negative sampling, both SCL and UCL achieve NC (for suitable hardness levels). Without normalization, neither random-negative nor hard-negative training methods attain NC and they suffer from DC. We also observe that for SCL and UCL, absence of normalization leads to less DC (compare blue curves in rows 1 and of Fig. 4). However, feature normalization could potentially reduce DC in hard-negative sampling for a range of hardness levels.
As we discussed in the paragraph following Definition 7, one way to avoid DC to achieve NC. As shown in Theorem 2 and Theorem 3 (and Remarks and 2 following Theorem 2), one condition for NC is unit-ball unit-sphere normalization of feature vectors. This may partially explain why feature normalization helps prevent or mitigate DC. On the other hand, as discussed in the paragraph following Definition 7, it is not necessary to achieve NC in order to avoid DC. We leave it to future work to study more general conditions that help avoid DC.
5.6 Impact of batch size
In Appendix C.3, we report results using different. batch sizes.
In Section 4.3 we showed that NC can occur in Contrastive Learning with any value of batch size certain conditions are satisfied. Thus, in theory, NC could be observed regardless of the value of the batch size, practice, however, it may be harder to achieve NC with a very small or a very large batch size as we explain below.
Large batch size effects: When the batch size is too large, the neural network may require greater representation capacity to map a larger number of samples from the same class into the same point (class- collapse). This could be solved by increasing the capacity of the neural network by expanding its size. In our experimental results, however, we only used standard architectures and observed that NC is consistently achieved for all batch sizes ranging from 64 to 512.
Small batch size effects: When the batch size decreased 64, NC is still evident in HSCL and HUCL for certain values of B. However, when the batch size is further reduced to 32, NC is no longer observed. One potential reason for this is the presence of 100 distinct classes in our dataset. With batch size of 32, the anchor can be compared with samples only from a very limited number of classes, reducing the diversity of negative pairs. This suggests that using larger batch sizes can contribute to more stable and robust results in the context of high class-cardinality.
5.7 Role of hardening function
We investigated the impact of hardening functions, using family of polynomial functions detailed in Appendix C.4. Results shown in Figs. 12 and 13 of Appendix C.4 confirm that hard-negative sampling helps prevent DC and achieve NC.
5.8 Experiments using the SimCLR framework
In Appendix C.5 we report results using the state-of-the-ar SimCLR framework for contrastive learning which uses augmentations to generate samples. The results of Figs. 14 and 15 in Appendix C.5 are somewhat similar to those in Figs. 3 and 4, respectively, but a key difference is the failure to altain NC in the SimCLR sampling framework for both supervised and unsupervised settings at all hardness levels. This is primarily because the SimCLR sampling framework does not utilize label information and cannot guarantee property (v) in Theorem 2 and Theorem nor guarantee (for UCL) the conditional independence of anchor and positive samples given their label.
6 Conclusion and open questions
We proved the theoretical optimality of the NC-gcometry for SCL, UCL, and notably (for the first time) HSCL losses for a very general family of CL losses and hardening functions that subsume popular choices. We empirically demonstrated the ability of hard-negative sampling achieve global optima for CL and mitigate
20

---

Published in Transactions on Machine Learning Research (04/2025)
dimenstonal-collapse, in both supervised and unsupervised settings. Our theoretical and empirical results motivate number of open questions. Firstly, tight lower bound for HUCL remains open due to latent-class collision. It is also unclear whether the HUCL loss is minimized iff there is Neural-Collapse.
If the number classes more than the dimension of the latent space plus one, then ETF is no longer the optimal solution, and the lower bound in Theorem 3 is not achievable. In this case; we may need other approaches based on convex optimization described in Nguyen et al. (2024).
Our theoretical results for the SCL setting did not require conditional independence of anchor and positive samples given their label, but our results for the UCL setting did. A theoretical characterization of the geometry of optimal solutions for UCL in the absence of conditional independence remains open. A difficulty with empirically observing NC in UCL and HUCL is that the number of latent. classes is not. known because it is, in general, implicitly tied to the properties of the sampling distribution, and this requires one to choose a sufficiently large representation dimension. Another open question to unravel preciscly how and why hard-negatives alter the optimization landscape enabling the training dynamics of Adam with random initialization to converge to the global optimum for suitable hardness levels and what are optimum hardness levels.
7 Acknowledgements
This research was supported by NSF 1931978.
21

---

Published in Transactions on Machine Learning Research (04/2025)
52
UCL
40
HUCL, 8=
HUCL, =
30
30
HUCL, 10
HSCL,
20
20
HUCL, B 30
HSCL, =
10
HSCL 10
10
HSCL, 30
50
100
150
200
150 200
250
350
400
1.0
SCL
1.0
HSCL, B
0.8
HSCL, f =5
HSCL, f = 10
HUCL, $ =
HSCL, f= 30
HUCL, $ =
HUCL, $ 10
HUCL 8 30
MM
50
100
150
200
250
300
150
400
50
100
150
200
250
300
350
400
UCL
0.8
0.0
HSCL, ==2
HUCL, $ =2
HSCL, =
HUCL, 8
0.6
HSCL, 10
0.6
HUCL, 8 10
HSCL, B 30
HUCL, 8 = 30
0.4
0.2
100
350
SCL
UCL
HSCL,
HUCL, 8
1.0
1.0
HSCL, p=
HUCL, $ 5
0.8
HSCL, ==10 10
HUCL, 8
HSCL, 30
HUCL, 8 30
0.4
0.2
50
100
150
200
250
300
350
50
100
150
200
250
300
400
Figure 3: Results for CIFAR100 under supervised settings (SCL, HSCL, left column) and unsupervised settings (UCL, HUCL, right column) with mail-ball normalization and random initialization. From top bottom: Downstream Test Accuracy, Zero-sum metric, Unit-norm metric, and Equal inner-produet metric, all plotted against the number of epochs.
22

---

Published in Transactions on Machine Learning Research (04/2025)
-10
-10
-15
-15
SCL
UCL
HSCL, 2
HUCL, 8=2
-25
HSCL, =
-25
HUCL, =
8
8
-30
-30
HSCL, / = 30
HUCL, 3
20
60
100
40
100
-5
-10
SCL
SCL
-20
-20
HSCL,
HSCL, 5
HSCL, =
25
HSCL. 8 10
-25
HSCL, 8 == 10
8
HSCL,
near-NC model
near-NC model
-35
-30
60
-5
-5
-10
SCL
UCL
-30
-20
HSCL, B ==
HUCL, p=2
HSCL, 5=
HUCL, $ =5
HSCL, 8 ==100 10
HUCL, 8 10
== 30
HUCL, 30
-35
-35
-5
-5
SCL
UCL
-20
-20
HSCL, ==
HUCL, B ==2
HSCL, 8
-25
HUCL,
HSCL, B 10 10
HUCL, B 10
HSCL, 8 30
HUCL, 3
-35
-35
Figure 4: Normalized singular values of the empirical covariance matrix of class means (in representation space) plotted in log-scale in decrcasing order for CIFAR100 under supervised (left column) and unsupervised (right column) settings. The horizontal axis is the sorted index of the singular values. From top to bottom: Unit-ball normalization with random initialization, Unit-ball normalization with near-NC initialization, Unit-sphere normalization with random initialization, and un-normalized representation with random initialization.
23

---

Published in Transactions on Machine Learning Research (04/2025)
A Monotonicity of SCL and UCL loss lower bounds
In this section, we discuss certain monotonicity properties of the SCL and UCL loss lower bounds in Theorem 2 and Theorem 3, respectively.
The SCL loss lower bound is given by (11), specifically,
I "SCL (1) 2 1h (ToSTT
(
(42)
Since (.) argument-wise non-decreasing, the SCL loss lower bound is a non-decreasing function of the number of classes C.
Interestingly, the UCL loss lower bound given by (30), i.c.,
Lee (f)
Σ U/ ( -01(( (C-1)
-0' Ky グv)
(43)
Ck+1
(C-1)
2- Ey
is also monotonic in C, but contrast to the SCT. loss lower bound, it is a monotonically non-increasing function of C. This conclusion not transparent since although the outer term 1., is decreasing in C, the inner term ΣyDriEE ED 1/ (-C #v) (C-1)
-C zv) is a non-decreasing function in C.
(C-1)
For the case 1t 1, f.e., when there is a single negative sample, we prove below that the UCL loss lower bound is, indeed, a non-increasing function of C. We then provide numerical evidence to support our conjecture that this result is true for all k >>> 1.
When k = 1, we can rewrite the UCL loss lower bound follows:
Lower-bound UCL. (C)
1
Σ
-CIIy Fv)
-C1(y zv)
(44)
Ck+1
(C-1)
(C-1)
Σ
411
-Cisy #V) C-1
(45)
C
V. Ey
Σ
-O1(y Sv) C-1
(46)
yEy 27 Ey
C
Σ 1/
-CI(y
+
Σ 1/1
(47)
91 $2
31 =
1 (IC 1)21,
-C C-1
+ 1/ (0))
(48)
Next, we show that Lower-bound UCL (C) is non-increasing in C by showing that Lower-bound UCL (C)
Lower-bound (C
Power-boundder (C) - Lower-bound uct, (C (C+1) ucu + 1) for all C O. Indeed,
(49)
((C-1)))) 1)th, (x)) + 1/, (0)) - (Ot) ( (-(CtL) 1², (0)(())
(50)
(C 10+1111 +)+1)(C 1)1; (-4) (C C 1) (0) C2,,, (-ctt) Cs (0)
a(+11
(51)
(02 - 1)l1 ( + ()) (0) C24, (11+11
(52)
[(C+1]
+ + (0) -1 (((++)
(53)
C+1
0-1
c
z G+1
C
0--1
@2 x -c + 02 x x0) e
0-1
(((++))
c
(54)
C
[(++1) - (11+11)
(55)
c+1
24

---

Published in Transactions on Machine Learning Research (04/2025)
(56)
where the inequality (54) is due to Jensen's inequality applied to the convex function 16 (.), and other
rows are just algebraic transformations, Since Lower-bound UCL (C) 2 nower-bound,e UCL (C + 1), it follows that uo (C) is a non-increasing function of C:
Extending this result to larger values of k is algebraically more complicated. Therefore, instead of proving it in full generality, numerically plot the UCL. loss lower bound for values of C' ranging from 2 through 20 and values of & ranging from 1 through 5. Figure shows plots of the UCL loss lower bound for the InfoNCE loss function whereas Fig. 6 shows plots for the triplct loss function.
Recall that the InfoNCE loss function corresponds to v/ (1)))) = log(a + e²1), with a > 0, and the triplet loss function (with sphere-normalized representations) is given by v/ (110) Eizl max , + a, 0},
> 0 We set 1 in both the InfoNCE and the triplet loss functions.
These plots show that the UCL loss lower bound always decreases with C' for any given value of k and for any given value of it always increases with k.
UCL lower bound with InfoNCE loss
1.4
ke4
1.2
Values UCL lower bound
0.8
0.6
0.4
0.2
10
12
14
16
18
20
Number of classes (C)
Figure 5: The values of the UCL loss lower bound in Theorem for the InfoNCE loss function, for k 1,2,3,4, 4,5, negative samples and C 2,3,..., 20, latent classes.
B Compatibility of sampling model with SimCLR-like augmentations
The following generative model captures the manner in which many data augmentation mechanisms, including SimCLR, generate a pair anchor and positive samples. First, a label y is sampled. The label may represent an observed class in the supervised setting or a latent (implicit, unobserved) class index in the unsupervised setting. Then given y, reference sample re is sampled with conditional distribution (-er(-ly). Then a pair
of samples (z, x+) are generated given (xref, y) via two independent calls to augmentation mechanism whose behavior can be statistically described by a conditional probability distribution Pang (-Ix-ef, .y), i.c., p(r, "1 ,v) Paug(x/x"' (x|x"e1, y) Paug ref, y). Finally, the representations = f(x) and = = f(x*) computed via mapping f()), e.g., neural network. Under the setting just described, it follows that
25

---

Published in Transactions on Machine Learning Research (04/2025)
UCL lower bound with triplet loss
k-1
20
Values UCL lower bound
1.59
0.59
0
10
12
14 16 18
20
Number of classes (C)
Figure 6: The values of the UCL loss lower-bound in Theorem 3 for the triplet loss function, for k 1,2,3,4, negative samples and C' 2,3,..... .... 20, latent classes.
both z|y and 2 2y have identical conditional distributions given y which we denote by s--ly). This can be verified by checking that both zly and x+ have the same conditional distribution given y equal
Paug(111" p(-ly) [_-,, Paug (-1xre) .y)p (x (x)t/ly)dx le
where the integrals will be sums in the discrete (probability mass function) setting. Note that although or, x+ are conditionally IID given (xref,)) yy) they need not be conditionally IID given just y.
C Additional experiments
We replicated the same experiments conducted in Sec. 5 on CIFARIO. The results are plotted in Fig. and Fig. 8. In contrast to CIFAR10 and CIFAR100 whore the numerical results are provided under three different settings, namely, unit-ball normalization with random initialization, unit-ball normalization with Neural-Collapse initialization, and unit-sphere normalization with random initialization, for Tmag-Nma,,Net, only conduct experiments under unit-ball normalization with random initialization. This is because the size of the Tmag-emageNet dataset (120000 images) is much larger than the sizes of both CIFAR10 and CIFAR100 datasets (50000 images per dataset) which results in significantly longer processing time. The results for Imageet are plotted in Fig. 9.
C.1 Neural-Collapse and Dimensional-Collapse
For CIFAR10, from Fig. 7 and Fig. 8, we observe similar phenomena as those for CIFAR100. As before, we note that while Theorems 2 and 3 suggest. that Neural-Collapse should occur in both the supervised and unsupervised settings when using the random negative sampling method, one may not be able to observe Neural-Cole Collas- in unsupervisod settings in practice. For the supervised case CIFAR10, any degree of hardness propels the representation towards Neural Neural-Collapse. This may be due to the small number of classes in CIFAR10.
26

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm Contrastive Learning Algorithm
Require:
Batch size N, data X, label V, neural-net parameters of representation function f. Algorithm: SCL/ UCL/HSCL/HUCL, normalization type: http://uii-phhere//oooonnaaaaaaa the hardening
function n == II , 0.
1: Define negative distribution p (2 based on the chosen Algorithm, see Sec. 3 for details.
2: for each sampled minibatch {xr,),= do
3: for all i E [I,...,NN do
4: Compute f(x,)
5: if unit-ball normalization then
if f Il < then
2, - (1xx)
else
f(a.) |f/x
2,
10:
end if
11:
else if unit-sphere normalization then
f(z.)
12:
2, [[f(z,)]
13:
else if no-normalization then
2, f(x))
14:
va
15:
end if
16:
end for for all
17:
(I,...,NN N) do for all E (L....,Ny N) do
18:
19:
if v(x,)) u( then
20:
Draw { from
(2 12.22
21:
{U (' Im = [2 am 2/ 2g m=1
22:
EGs log
[ 1 D*
1: Em== em
23:
else
24:
f.j =0 end if
25:
26:
end for
27:
Compute the average loss of sample x 4, Hlgz0H Ej= fes
28:
end for
29: Compute the average loss of minibatch: L 3/2 Σn= l
30:
Take one stochastic gradient step using Adam
31:
end for
return Encoder network f(-)
27

---

Published in Transactions on Machine Learning Research (04/2025)
MnnA A
HSCL, B 2
HUCL, 8 =
HSCL, B
HUCL, 8 ==
HSCL, 8 10
HUCL, 8 = 10
HSCL, B= 30
HUCL, B 30
50
100
150
200
250
300
350
400
100
150
200
250
300
350
400
SCL
0.8
HSCL, =
HSCL, =
UCL
HSCL, =0
HUCL, 8 = 2
HUCL, B 5
0.4
HUCL, B
HUCL,
0.2
0.0
50
100
150
250
300
350
150
200
250
300
400
1.0
SCL
UCL
HSCL,
HUCL, 8
HUCL, 8
HSCL, β= 10
HUCL, 8 = 10
0.6
HSCL, 8 30
HUCL, 8 30
0.1
0.4
0.0
1.4
SCL HSCL, ==2
UCL
HUCL, 8 =
HSCL, 8
HUCL, 8
0.8
HSCL, 10
HUCL, P 10
HSCL, $= 30
HUCL, B 30
0.6
MMM
0.2
0.0
100
150
250
300
350
400
150
200
250
300
400
Figure 7: Results for CIFAR10 under supervised settings (SCL, HSCL, left column) and unsupervised settings (UCL, HUCL, right column) with unit-ball normalization and random initialization. From top to bottom: Downstream Test Accuracy, Zero-sum metric, Unit-norm metric, and Equal inne-produie metric, all plotted against the number of epochs.
28

---

Published in Transactions on Machine Learning Research (04/2025)
-5
-5
-15
-15
SCL
UCL
-20
-20
HSCL, 8
HUCL, 8
HSCL, 85
-25
HUCL, 8
HSCL, 8 10
HUCL,
-30
-30
HSCL, / 30
HUCL, B 30
-10
-15
-15
SCL
SCL
-20
HSCL, ==
-20
HSCL, =
HSCL, 8
HSCL, p
HSCL, # 10
HSCL, 30
-30
HSCL, =
near-NC model
near NC model
-35
-35
-10
-10
-15
-15
SCL
UCL
HSCL, 8 =
HUCL, $ ==2
-25
HSCL, #5
-25
HUCL, 0 =
HSCL, 10
HUCL, P = 10
HSCL, 8 30 30
HUCL, B
-35
-35
-5
-5
-15
-15
SCL
UCL
-20
-20
HUCL, 8
HSCL, 8
-25
HUCL, 8
HSCL, 8 10
HUCL, $
-30
-30
HSCL, A 30
HUCL, $ 30
Figure 8: Sorted normalized singular values of the empirical covariance matrix of class means (in representation space) in the last epoch plotted in log-scale for CIFAR10 under supervised (left column) and unsupervised (right column) settings. From top to bottom: Unit- ball normalization with random initialization, Unit-bat ball normalization with ncar-NC initialization, Unit-sphere normalization with random initialization, and un- normalized representation with random initialization.
29

---

Published in Transactions on Machine Learning Research (04/2025)
For Iiny-ImageNet, from Fig. 9, we observe that when 3 = 5, 10, 30, the geometry of the learned representation closely aligns with that of Neural-Collapse. However, in HUCT, high degree of hardness can be harmful. At 3 = 5, the geometry most closely approximates Neural-Collapse for both CIFARIO and Tiny-Imagell However, increasing the degree of hardness further, for example at 3 = 30, causes the center of class mcans to deviale from the origin and the equal inner-product condition is heavily violated.
Furthermore, from the the normalized singular values for CIFARIO in Fig. 8 and for Tiny--mageNNe in Fig. 9, we observe that random negative sampling without any hardening (3 0) suffers from DC whereas hard-negative sampling consistently mitigates DC. The supervised case benefits more from higher degree of hardness, since in the unsupervised cases there are higher chances of (latent) class collisions.
C.2 Effects of initialization and normalization
The normalized singular value plots for CIFARIO are shown in Fig. 8. Compared to CIFAR 100 (see Fig. 4), the phenomenon of DC in CIFAR10 is far less pronounced. This may be because CIFAR10 has a smaller number of classes compared to CIFAR100 (10 us. 100). However, the effects of initialization and normalization on the learned reprosentation geometry are similar to that for CIFAR100:
Effects of initialization: 1) SCL and HSCL trained with near-NC initialization and Adam do not exhibit DC and 2) UCL trained with near-NC initialization and Adam also does not exhibit DC, but the behavior of HUCL depends on the hardness level B.
Effects of normalization: The behavior of unit-sphere normalization is close to that of unit-ball normaliza- tion, and with hard-negative sampling (and suitable hardness levels), both SCL and UCL can achieve NC. Without normalization, neither regular nor hard-negative training methods attain NC and they suffer from DC. We also observe that with random-negative sampling, un- normormalized representations lead to reduced DC in both SCL and UCL. However, hard-nogative sampling benefits more from feature normalization and its absence leads to more severe DC.
C.3 Experiments with different batch sizes
To investigate the effect of batch size on the outcomes, we conducted experiments with varying batch sizes. All previous experiments were performed with a batch size of 512. In this section, we present results for batch sizes of 64 and 32 in Fig. 10 and Fig. 11, respectively.
We observe that when the batch size is reduced to 64, Neural-Collapse is still nearly achieved in both HSCL (8 = 5, 10,30) and HUCL (3 = 5). However, with a further reduction of batch size to 32, Neural-Collapse is only achieved in HSCL (3 = =0),, and it fails to occur in HUCL for any value of 3.
C.4 Experiments with a different hardening function
To explore whether similar results can be achieved with a different hardening function, we investigated the impact that changing the hardening function has on NC and DC under unit-ball normalization. We conducted experiments using the CIFAR10 and CIFARI00 datasets adopting the setup consistent with previous experiments but used new family of hardening functions having the following polynomial form: n((,,))== = II = (max((, + 1,0))), 0, for the following set of hardness values e 3, 5, 10, 20. We note that this family of hardening functions decays at significantly slower rate compared to the exponential hardening function we used in all our previous experiments.
Results for CIFAR100 and CIFARIO are plotted in Figs. 12 and 13, respectively. We observe phenomena similar to those in Figs. 3 and 7. By selecting an appropriate hardening parameter e, we can achieve, or nearly achieve, NC in both supervised and unsupervisod sottings. Consequently, we can draw conclusions that are qualitatively similar those in Sec. C.1. Specifically, hard-negative sampling in both supervised and unsupervised settings can mitigate DC while achieving NC.
30

---

Published in Transactions on Machine Learning Research (04/2025)
C.5 Experiments with the SimCLR framework
We conducted additional experiments using the state-of-the-art SimCLR framework for Contrastive Learning. Instead of sampling positive samples from the same class directly, we follow the setting in SimCLR which uses two independent augmentations of a reference sample to create the positive pair. No label information is used to generate the anchor-positivy pair.
Figure 14 shows results for SimCLR sampling using the loss function proposed in SimCLR which is the large-k asymptotic form of the InfoNCE loss:
L SimoLB(() E p(x,a+) log 1 pla (o) loje-oo
QE
[es(e) 5(c ef(a) ((tt)
I
(57)
where Q is weighting hyper-parameter that is set to batch size minus two. Compared to our previous experiments, all the NC metrics deviate significantly away from zero in both supervised and unsupervised settings and all hardness levels. Still, the high-level conclusions for DC are qualitatively similar to those from our previous experiments, specifically that hard-negative sampling can mitigale DC in both supervised and unsupervised settings.
Figure 15 shows results for SimCLR sampling using the InfoNCE loss with k 256 and positive distribution that only relies on the augmentation method. The results are very similar to those in Fig. 14. Since the results in Figs. 14 and 15 share the same SimCLR sampling framework but different losses, it follows that the failure to attain NC is not due to the particular loss function used, but the SimCLR sampling framework itself which does not utilize label information to generate samples. From the unit-norm metric plots in both figures it is clear that the final representations are mostly distributed within the unit-ball than on its surface.
References
Randall Balestricro, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv preprint 1rX1v:2304.12210, 2023.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration Inequalities Nonasymplolic
Theory of Independence. Oxford University Press, 2013.
Ting Chen, Simon Korublith, Mohammad Noroudi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMT.R, 2020.
Ching-Yar Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning, Advances in neural information processing systems, 33:8765 8775, 2020.
Patrick Feeney and Michael C Hughes. Sincere: Supervised information noise-contrastive estimation revisited. arXiv preprint 1477 2023.
Florian Graf, Christoph Hofer, Mare Niethammer, and Roland Kwitt. Dissecting supervised contrastive
learning. In International Conference on Machine Learning, pp. 3821-3830. PMLR, 2021.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances Neural Information Processing Systems, 34:5000-5011 2021.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Compuler Vision and Pallern Recogrition (CVPR), pp. 770 778, 2016.
Ruijic Jiang, Prakash Ishwar, and Shuchin Acron Hard negative sampling via regularized optimal transport for contrastive representation learning. In 2028 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEFE, 2023.
31

---

Published in Transactions on Machine Learning Research (04/2025)
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Represenlations, 2021.
Prannay Khosla, Piotr Telerwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:1 166111661,,77 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Jason D Lee, Qi Lei, Nikunj Saunshi, and Jiacheng Zhuo Predicting what you already know helps: Provable self-supervised learning. Advances in Newral Information Processing Systems, 34:309-322 2021.
Zijun Long, George Killick, Richard McCreadie, Gerardo Aragon Camarasa, and Zaiqiao Meng. When hard negative sampling meets supervised contrastive learning. or preprint ar XrXiv:2308. 14893, 2023.
Vassili Nikolaevich Malozemov and Aleksandr Borisovich Pevnyi. Equiangular tight. frames. Journal of Mathematical Sciences, 157(6):788 111,, 2009.
Thuan Nguyen, Ruijie Jiang, Shuchin Aeron, Prakash Ishwar, and D Richard Brown. On neural collapse in contrastive learning with imbalanced datasets. In 2024 IEEE 94th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2024.
Hyun Oh Song, Yu Xiang, Stefanic Jegelka, and Silvio Savaresc. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on compuler vision and patlern recognition, pp- 4004-4012, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint (cXiv:88070077/ 2018
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(44):24652-22 24652-263333, 2020.
Advait. Parulekar, Liam Collins, Karthikeyan Shanmugam, Aryan Mokhtari, and Sanjay Shakkottai. Infonce loss provably learns cuustrrppreeervin representations. In The Thirty Sizth Annual Conference on Learning Theory, pp. 1914 1961. PMLR, 2023.
Nils Rethmeier and Isabelle Augenstein. A primer on contrastive pretraining in language processing: Methods, lessons learned, and perspectives. ACM Computing Surveys, 55(10):1-17, 2023.
Joshua David Robinson, Ching-Yas Yao Chuang, Suvrit Sra, and Stefanic Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2020.
Joshua David Robinson, Ching-Yas Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Represenlations, 2021.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Conference Machine Tearning, pp. 5628-56666660 5637. PMTR, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference Machine Learning, pp. 99299992999999 9939. PMLR, 2020.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations, 2021.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference on Machine Learning, pp. 11112-11122. 11122. PMLR, 2021.
32

---

Published in Transactions on Machine Learning Research (04/2025)
Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman. Conditional negative sampling for cont rastive learning of visual representations. In International Conference on Learning Representalions, 2020.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3733 3742, 2018.
Can Yaras, Peng Wang, Zhihui Zhu, Laura Balzano, and Qing Qu Neural collapse with normalized features: A geometric analysis the riemannian manifold. Advances in neural informalion processing systems, 35: 11547-11560, 2022.
Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under mse loss: Global optimality with unconstrained features. In International Conference on Machine Learning, pp. 27179 27202. PMLR, 2022.
Liu Ziyin, Ekdeep Singh Lubana, Masahito Ceda, and Hidenori Tanaka. What shapes the loss landscape of self supervised learning? In The Eleventh International Conference on Learning Representations, 2022.
33

---

Published in Transactions on Machine Learning Research (04/2025)
-5
-5
SCL
-10
HSCL, 8 2
-15
-15
UCL
-20
HSCL, 10
-20
HUCL, B=
HSCL, B 30
HUCL,
HUCL,
-30
-30
HUCL, =
100
175
200
HSCL, /
0.8
HSCL, 8 5
1m00 Mu 154 UCL
ma M mm
HSCL, 8= 10
HUCL, $ =
HSCL, 8 30
HUCL, B = 5
HUCL, B 10
HUCL, B = 30
0.2
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
SCL
UCL
0.8
HSCL, B
HUCL, $ 8=2
HSCL, =
HUCL, $ =
HSCL, p=
HUCL, B
HSCL,
HUCL, 8
0.4
0.2
0.2
0.0
750
1.0
SCL
1.0
UCL
HSCL, =
HUCL, 8
HSCL, $
HUCL,
HSCL, 10
HUCL, 8 10
Mn A, www
0.6
HSCL, B= 30
0.6
HUCL, 0 3:30
0.2
0.0
8.0
Figure 9: Results for Tiny-ImageNet under supervised settings (SCL, HSCL, left column) and unsupervised settings (UCL, HUCL, right column) with unit-ball normalization and random initialization. Top row: sorted normalized singular values of the empirical covariance matrix of class means (in representation space) in the last epoch plotted in log-scale. Rows 2-4: Zero-sum metric, Unit-norm metric, and Equal inner-product metric, all plotted against the number of epochs.
34

---

Published in Transactions on Machine Learning Research (04/2025)
-5
-10
-10
SCL
-20
UCL
HSCL, 8
HUCL, $ =
-25
HSCL, 5
-25
HUCL, =
HSCL, 10
HUCL, 10
-30
-30
HSCL, / 300
HUCL, 8 3
00
100
SCL
UCL
0.8
0.8
HSCL, $ =2
HUCL, =
HSCL, =
HUCL
0.5
HSCL, 8 10
HUCL. 10
HSCL, B 30
HUCL, B 30
0.4
0.
0.2
02
0.0
30
100
150
200
250
300
350
000
50
100
150
200
250
300
350
100
1.0
SCL
1.0
HSCL, 8 =
0.11
HSCL, 8 =5
0.5
HSCL, 8 10
UCL
0.6
HSCL, 0 30
0.5
HUCL, HUCL, A HUCL, 8 = 10
HUCL, 30
0.13
20
130
150
200
250
300
350
400
30
100
150
200
250
300
350
400
1.2
SCL
3.2
UCL
HSCL, =2
HSCL, 8 =5
HUCL, =
HSCL, 8 10
HUCL. = 10
0.0
0.8
HSCL, 30
0.-
20
200
250
200
250
300
350
000
50 100
150
200
250
300
250
100
Figure 10: Results for CIFAR100 under supervised settings (SCL, HSCL, left column) and unsupervised settings (UCL, HUCL, right column) with unit-ball normalization and random initialization when batch size is equal to 64. Top row: sorted normalized singular values of the empirical covariance matrix of class means (in representation space) in the last epoch plotted in log-scale. Rows 2 4: Zero-sum metric, Unit-norm metric, and Equal inner-produc metric, all plotted against the number of epochs.
35

---

Published in Transactions on Machine Learning Research (04/2025)
-5
-10
-10
SCL
-20
UCL
HSCL, 2
HUCL, $ =
-25
HSCL, 5
-25
HUCL, =
HSCL, 10
HUCL, 10
-30
-30
HSCL, 8 300
HUCL, 8 3
00
100
SCL
UCL
0.8
0.8
HSCL, $ =
HUCL, 8
HSCL, =
HUCL =
0.5
HSCL, 8 10
HUCL. 10
HSCL, B 30
HUCL, B 30
0.4
0.
0.2
02
0.0
30
100
150
200
250
300
350
000
50
100
150
200
250
300
350
100
1.0
SCL
1.0
HSCL, =
0.11
HSCL, 8 =5
0.5
HSCL, 8 10
UCL
0.6
HSCL, 0 30
0.5
HUCL,
HUCL, ==
HUCL, 8 10
HUCL, 30
0.13
20
130
150
200
250
300
350
400
50
100
150
200
250
300
350
400
1.2
SCL
3.2
UCL
HSCL, 8
HSCL, 8 =5
HUCL, =
HSCL, 8 10
HUCL. = 10
0.0
0.8
HSCL, 30
0.2
20
200
150
200
250
300
350
000
50 100
150
200
250
300
250
100
Figure Results for CIFAR100 under supervised settings (SCL, HSCL, left column) and unsupervised settings (UCL, HUCL, right column) with unit-ball normalization and random initialization when batch size is equal to 32. Top row: sorted normalized singular values of the empirical covariance matrix of class means (in representation space) in the last epoch plotted in log-scale. Rows 2 4: Zero-sum metric, Unit-norm metric, and Equal inner-produc metric, all plotted against the number of epochs.
36

---

Published in Transactions on Machine Learning Research (04/2025)
-5
-5
-10
-15
-15
-20
SCL
-20
UCL
HSCL, =
HUCL, 6
HSCL,
HUCL, =
HSCL, ==
HUCL, E
-30
-30
HSCL,
HUCL,
-35
UCL
0.0
HSCL, ==
HUCL, E =
0.8
HSCL, ==5
HUCL,
HSCL, = 10
HUCL, E == 10
HSCL, = 20
HUCL, 20
0.2
50
100
150
200
250
300
350
400
0
50
100
150
200
250
300
350
400
SCL
UCL
0.8
HSCL, ==3
HUCL, E
HSCL, e
HUCL, =5
HSCL, = 10
HUCL, & 10
HSCL, 20
HUCL,
0.4
40.4 // An Apma
0.4
0.2
0.0
350
1.4
SCL
UCL
HSCL, ==
HUCL, ==3
HSCL, == HSCL,
HUCL, E =
HSCL, E 20
HUCL, E = 20
0.6
0.0
Figure 12: Results for CIFAR100 with polynomial hardening function, under supervised settings (SCL, HSCL, left column) and unsupervised settings (UCI,, HUCL, right column) with unit-ball normalization and random initialization. Top row: sorted normalized singular values of the empirical covariance matrix of class means (in representation space) in the last. epoch plotted in log-scale. Rows 2-4: Zero-sum metric, Unit-norm metric, and Equal inner-produe metric, all plotted against the number of epochs.
37

---

Published in Transactions on Machine Learning Research (04/2025)
-5
-10
-15
-15
-20
SCL
-20
UCL
HSCL, =
HUCL, 6
HSCL,
HUCL, =
HSCL, ==
HUCL, E
-30
-30
HSCL,
HUCL,
-35
UCL
0.0
HSCL, ==
HUCL, & =
0.8
HSCL, 5-5
HUCL,
HSCL, == 10
HUCL, E 10
HSCL, = 20
HUCL,
0.2
50
100
150
200
250
300
350
400
0
50
100
150
200
250
300
350
400
SCL
UCL
0.8
HSCL, ==3
HUCL, E
HSCL, =
HUCL,
HSCL, = 10
HUCL, E 10
HSCL, 20
HUCL,
0.4
0.4
0.2
0.0
350
1.4
SCL
UCL
HSCL, ==
HUCL, ==
HSCL, = HSCL, HSCL, E 20
HUCL, E =
HUCL,
HUCL, E = 20
0.6
0.0
Figure 13: Results for CIFAR10 with polynomial hardening function, under supervised settings (SCL, HSCL, left column) and unsupervised settings (UCI,, HUCL, right column) with unit-ball normalization and random initialization. Top row: sorted normalized singular values of the empirical covariance matrix of class means (in representation space) in the last. epoch plotted in log-scale. Rows 2-4: Zero-sum metric, Unit-norm metric, and Equal innerpprodie metric, all plotted against the number of epochs.
38

---

Published in Transactions on Machine Learning Research (04/2025)
-5
-10
-10
SCL
-20
UCL
HSCL, =
HUCL, $ =
-25
HSCL, 5
-25
HUCL, =
HSCL, 10
HUCL, 10
-30
-30
HSCL, 8 =30
HUCL, 3
60
100
SCL
HSCL, $ =2
0.8
HSCL, 8 =5
0.8
HSCL, 8 10
UCL
0.8
HSCL, B 30
0.6
HUCL, B
HUCL,
0.4
HUCL 10
HUCL, 30
50
120
150
200
250
300
350
100
30
100
150
200
250
300
350
100
0.81
0.6
0.5
0.1
SCL
UCL
HSCL, 8 m
HUCL,
0.2
HSCL, 5
0.2
HUCL,
HSCL, $ == 10
HUCL, = 10
HSC 8
HUCL, == 30
00
0.0
20
100
150
200
250
350
400
50
100
150
200
250
300
350
400
1.2
SCL
1.0
1.0
HSCL, 8
HSCL, 8 =5
0.8
0.11
HSCL, 8 10
UCL
HSCL, 8 30
0.6
HUCL, =
HUCL, 8 10
HUCL.
20
200
150
200
250
300
350
000
50 100 200
250
300
250
100
Figure 14: Results for CIFAR100 with SimCLR sampling with SimCLR loss (Eq.44), under supervisod scttings (SCL, HSCL, left. column) and unsupervised settings (UCL, HUCL, right column) with unit-ball normalization and random initialization. Top row: sorted normalized singular values of the empirical covariance matrix of class means (in representation space) in the last epoch plotted in log-scale. Rows 2 4: Zero-sum metric, Unit-norm metric, and Equal inner-product metric, all plotted against the number of epochs.
39

---

Published in Transactions on Machine Learning Research (04/2025)
-5
-10
-10
SCL
-20
UCL
HSCL, 2
HUCL, $ ==2
-25
HSCL, 5
-25
HUCL, =
HSCL, 10
HUCL,
-30
-30
HSCL, 8 =30
HUCL, 3
50
100
SCL
HSCL, $ =2
0.8
HSCL, 8 =5
0.8
HSCL, 8 10
UCL
0.8
HSCL, B 30
0.6
HUCL, =
HUCL,
0.4
HUCL 10
HUCL. 30
50
120
150
200
250
300
350
100
30
100
150
200
230
300
350
100
SCL
UCL
0.81
HSCL, 8
HUCL,
HSCL, 8 =5
HUCL, =5
0.6
HSCL, 8 10
0.5
HUCL = 10
HSCL, 0 30
HUCL,
0.1
0.2
0.
00
0.0
20
100
150
200
250
350
400
50
100
150
200
250
300
350
400
1.2
SCL
1.2
UCL
HSCL, 8
HSCL, 8 ==
HUCL, =5
HSCL, 10
HUCL, 10
0.8
0.8
HSCL, 30
0.2
20
200
150
200
250
300
350
000
50 100 200
250
300
250
100
Figure 15: Results for CIFAR100 with SimCLR sampling using InfoNCE loss, under supervised settings (SCL, HSCL, left column) and unsupervised settings (UCL, HUCL, right column) with unit-ball normalization and random initialization. Top row: sorted normalized singular values of the empirical covariance matrix of class means (in representation space) in the last epoch plotted in log-scale. Rows 2 4: Zero-sum metric, Unit-norm metric, and Equal inner-preed uneer-product metric, all plotted against the number of epochs.
40


Reviews:
[{"review_id": "o5uHpIWUu9", "reviewer": "Reviewer_PVUi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**：\n1. Theoretical Contributions：The paper provides formal proofs that hard-negative contrastive learning converges to Neural-Collapse, extending existing results beyond the class-conditional independence assumption. Moreover, It establishes that HSCL and HUCL losses are greater than or equal to SCL and UCL losses, respectively. Finally, The results apply to a broad class of loss functions, including InfoNCE and triplet loss.\n2. Practical Relevance: The study provides insights into choosing the hardness level to achieve better feature separation.The results highlight the necessity of feature normalization and hard-negative sampling to avoid Dimensional-Collapse.\n\n**Weakness**:\n1. Clarity: The section 3.2 \"Hard-negative sampling\" is unclear for the reader to understand. For the harding function (definition 2), while an exponential tilting hardening function is provided as example, however, it is unclear how it is used in the practice. Moreover, HSCL and HUCL is hard to understand. It would be better to provide some examples for intuition. \n2. Strong assumption: For theorem 2, the authors assume that the function f satisfy the equal inner-product class, which actually directly assume the simple ETF structure of the class-mean centers, which might  holds some some special case of losses function, such as InfoNCE based on previous literature, however for this assumption holds for other losses such as triple loss is unknow.  \n\n**Questions**:\n1. Hardness Selection Heuristics: The study empirically explores various hardness levels but does not provide a principled method for selecting the optimal hardness level in practical settings. Is there a principled way to determine the optimal hardness level for hard-negative sampling in contrastive learning? Could an adaptive hardness function be developed?\n2. Role of Feature Normalization: Why is feature normalization so crucial for avoiding Dimensional-Collapse? Could there be other approaches to address Dimensional-Collapse without normalization?\n3. How does the choice of batch size affect the convergence to Neural-Collapse? Would using larger batches inherently lead to more stable results?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper provides a theoretical and empirical investigation into the impact of hard-negative sampling in contrastive learning (CL). The authors establish that supervised and unsupervised contrastive learning losses are minimized by representations that exhibit Neural-Collapse (NC), where class means form an Equiangular Tight Frame (ETF). The paper introduces Hard-Supervised Contrastive Learning (HSCL) and Hard-Unsupervised Contrastive Learning (HUCL), proving that their losses are lower-bounded by the corresponding standard CL losses. Through rigorous mathematical proofs and experiments on synthetic and real datasets, the paper demonstrates that feature normalization and hard-negative sampling mitigate Dimensional-Collapse (DC), a phenomenon where learned representations degenerate to a low-rank subspace.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. While the paper is generally well-written, some definition(especially for the hard-negative sampling part), statements and notations could be streamlined for better readability.\n2. Equal inner-product class assumption directly enforces Neural Collapse on class means. While it simplifies theoretical analysis, it bypasses the question of whether this collapse naturally occurs during training in contrastive learning settings. A more relaxed assumption would allow class means to emerge as an ETF rather than explicitly requiring it.", "broader_impact_concerns": "No Broader Impact Concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mHUNBFOYwQ", "reviewer": "Reviewer_BYJa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The assumption that each data sample belongs to one class is very restrictive for application. For instance, real-world data samples often follow a hierarchical class attributes and state-of-the-art contrastive learning models such as CLIP [1] are often considered as learning such hierarachical structure.\n\n- In SimCLR, the InfoNCE loss is adopted with $\\psi_k(t_{1:k}) = \\log(\\alpha + \\sum_{i=1}^k e^{t_i})$ and $\\alpha = 0$. This could be mentioned in Section 5.8 to explain why SimCLR does not exhitbit Neural-Collaspe, as it determines the strict convexity of $\\psi$ which is a condition considered in Theorem 3.\n\n- In page 9 `(iv) Proof that additional condition (13) implies zero within-class variance`, it is not clear why (22) $\\Leftrightarrow f(x) = \\mu_y$ given $y$. For instance, it is possible that for $\\bar{x},\\tilde{x} \\sim s(x | y)$, (22) $\\Rightarrow \\| f(\\bar{x}) \\| = \\| f(\\tilde{x}) \\| = 1$ but $f(\\bar{x}) \\neq f(\\tilde{x})$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a lower bound analysis for a family of contrastive learning loss function, for which these lower bounds are attainable by the Neural-Collaspe solution. Their analysis relies on the assumption that each data sample is assigned to one latent class label. An `equal inner-product class means` property (13) is proposed and the anaylsis studies its connection to the global optimum of contrastive losses, for example claiming (13) as a necessary condition of optimality for strictly convex loss function $\\psi$. Experimental results are provided as an attempt to observe the Neural-Collaspe phenomenon under supervised/unsupervised losses with different hard-negative sampling strategies.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Typos\n- $f(x)_1^-$ should be $f(x_1^-)$ above equation (14).\n- Typo in Section 5: `all samples that share the same label ...`. \"all\" should be capitalized.\n\n### Presentation\n- The losses in Theorem 1 are only implicitly defined through the definition of $L_{CL}^{(k)}(f)$ in Definition 1. I suggest the authors to generalize the notation after Definition 1 and define $L_{HUCL}, L_{UCL}, L_{HSCL}, L_{SCL}$ properly, for example denote $L_{HUCL}^{(k)}(f) = L_{CL}^{(k)}(f, p_{HUCL})$.\n- The result of Theorem 1 is not discussed. What are the implications from to the proved inequalities?\n- The experiment results can be presented more clearly. I suggest the authors to pick out some representative figures to be shown nearby in the main text, rather than showing numerous figures in a single page that is far from the discussion of those results.\n\n\n\n### References\n\n- I suggest the authors to include a reference to a recent work on hard-negative sampling [2]. Since it is mentioned in Section 6 that the training dynamic of hard-negative sampling is an open question, we can see [2] explains the optimization trajectory of hard-negative samples as obtaining unbiased gradient estimation of the global contrastive loss. On the other hand, it would also be interesting to see if the minimizer of the global contrastive loss also exhibits Neural-Collaspe by a similar experiment setup in Section 5.\n\n[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. ICML, 2021.\n\n[2] Chung-Yiu Yau, Hoi-To Wai, Parameswaran Raman, Soumajyoti Sarkar and Mingyi Hong. EMC$^2$: Efficient MCMC Negative Sampling for Contrastive Learning with Global Convergence. ICML, 2024.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iTCJqEe9Cz", "reviewer": "Reviewer_C1Bz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper provides a very nice and neat unifying theory for both SCL and UCL and include the analysis for hard negative sampling. This can be regarded as a strong unified theory on neural collapse in contrastive learning. \n\n- The experimental evaluation is structured and convincing and nicely complements the theoretical findings while also inviting a further research on the topic (such as deeper explanation on why hard negative sampling helps preventing the dimensional collapse in some cases). \n\n- The paper is clearly written and easy to understand in most cases. \n\n**Weaknesses:**\n\n- The theoretical contribution, while unifying and structured, is not particularly novel. Theorems 1 and 3 are already partially covered by the related work with only minor improvements. The statement of theorem 2 is novel (to the best of my knowledge), however the proof techniques of both theorem 2 and 3 (such as applying the Jensen inequality to obtain the optimality of equidistant configurations) represent a rather routine way of proving NC both in CL, but also in standard training, because the considered loss can be understood as an easier version of the unconstrained features model used to explain NC in standard training (see for instance [1,2, 3]). \n\n- Another issue is that the authors do not lift the conditional independence assumption in the UCL, which in my opinion is literally impossible to satisfy in practice. \n\n- On the experimental front, I think it is not insightful that the hard negative sampling helps improve collapse because in this setting to speak about collapse is the same as to speak about training loss and thus if the related literature already established that hard negative sampling improves the training loss and/or the overall performance of the networks, this empirical observation doesn't bring any new insight.\n\n- It is not entirely clear whether the the number of latent classes in UCL influences the loss function and if yes, what does the model do about it? In particular, it is the choice of the model how many latent classes will there be as long as it is at least the number of true classes. Won’t this produce some degenerate behavior form the model? \n\n[1] Zhu, Zhihui, et al. \"A geometric analysis of neural collapse with unconstrained features.\" Advances in Neural Information Processing Systems 34 (2021): 29820-29834.\n\n[2] Zhou, Jinxin, et al. \"Are all losses created equal: A neural collapse perspective.\" Advances in Neural Information Processing Systems 35 (2022): 31697-31710.\n\n[3] Ji, Wenlong, et al. \"An unconstrained layer-peeled perspective on neural collapse.\" arXiv preprint arXiv:2110.02796 (2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the geometry of the loss landscape of contrastive learning with normalized features in both supervised and unsupervised setting as well as both hard negative sampling and standard negative sampling setting. On the theoretical front, authors first show that hard negative sampling is always at least as hard as the standard negative sampling in terms of the loss. Then they show, both in supervised and unsupervised setting that the loss of CL is lower-bounded by a value that is achieved by neural collapse. In case of strictly increasing and convex loss function neural collapse is the only solution achieving that loss. In SCL, the authors also show equivalence with the HSCL loss. The authors lift the class-conditional independence assumption from related work in their SCL analysis. On the empirical front, the authors first empirically back up their theoretical findings on synthetic dataset. Then, they analyze the convergence to NC solutions of CL in practice and show that the models sometimes exhibit dimensional collapse, which, however, can be partially avoided/remedied by using the hard negative sampling. They also study the effect of several design parameters and hyper-parameters on the emergence of NC and/or DC in CL.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Questions:** \n\n-\tIn the paper you claim that the number of latent classes in UCL depends on the sampling procedure of the similar samples. How?\n\n-\tHow is the sampling in hard negative sampling performed in practice? It seems to be computationally expensive to always evaluate the network on all the samples.\n\n-\tHow does the model decide how many latent classes will there be in UCL and will it influence the optimal loss function? \n\n-\tIn page 14 you write that you sample the negative samples from the entire batch and potentially including the anchors, positive examples and/or same-class examples. Doesn’t this undermine the difference between SCL and UCL and make all experiments effectively UCL? This question is further motivated by what you write on page 16 where you say that the NC is not exactly achieved when you use this procedure. Why using this procedure in the first place then?\n\n-\tDo you use weight decay in your practical experiments? \n\n**Requested changes:**\n \n-\tPlease add a detailed discussion on the effect of the number of latent classes on the loss in UCL and what does that mean for the incentives of the model and whether or not it introduces some biases, e.g. towards small or large number of latent classes. \n\n-\tPlease add the discussion in the beginning of section 4.3 about the usage of your data distribution and loss as empirical distribution and empirical loss already to the point where you introduce those concepts. Furthermore, please always contextualize each theorem and argument to whether it refers to empirical or theoretical loss or both, to make the discussion more transparent. \n\n-\tFix the typo in the beginning of page 14 where you refer to property “(d)” but it should be “(e)”.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This long submission received extensive reviewer feedback regarding its presentation, experiments, and theoretical contributions—including its improvements over prior work and claims about the number of latent classes. The authors addressed these comments through additional discussions and experiments.  With thorough back-and-forth discussions, all comments have been satisfactorily addressed, and all three reviewers now recommend acceptance.
Decision: Accept (poster)