On the Similarities of Embeddings in Contrastive Learning
Authors: Chungpa Lee
Sehee Lim
Kibok Lee
Jy-yong Sohn
Venue: ICML 2025 poster (2025)
Keywords: contrastive learning
representation learning
embedding
similarity
negative pair
positive pair
variance
Primary area: theory
TL;DR: We propose a unified framework for understanding contrastive learning through the lens of cosine similarity, and present two key theoretical insights derived from this framework.

Abstract: Contrastive learning operates on a simple yet effective principle: Embeddings of positive pairs are pulled together, while those of negative pairs are pushed apart. In this paper, we propose a unified framework for understanding contrastive learning through the lens of cosine similarity, and present two key theoretical insights derived from this framework. First, in full-batch settings, we show that perfect alignment of positive pairs is unattainable when negative-pair similarities fall below a threshold, and this misalignment can be mitigated by incorporating within-view negative pairs into the objective. Second, in mini-batch settings, smaller batch sizes induce stronger separation among negative pairs in the embedding space, i.e., higher variance in their similarities, which in turn degrades the quality of learned representations compared to full-batch settings. To address this, we propose an auxiliary loss that reduces the variance of negative-pair similarities in mini-batch settings. Empirical results show that incorporating the proposed loss improves performance in small-batch settings.

On the Similarities of Embeddings in Contrastive Learning
Chungpa Lee" Sehee Lim
Kibok Lee
Jy-yong Sohn
Abstract
ing negative pairs, achieve remarkable performance across various downstream tasks (Wang & Isola, 2020).
Contrastive learning operates simple yet ef- fective principle: Embeddings of positive pairs are pulled together, while those of negative pairs are pushed apart. In this paper, we propose a unified framework for understanding contrastive learning through the lens of cosine similarity, and present two key theoretical insights derived from this framework. First, in full-batch settings, show that perfect alignment of positive pairs is unattainable when negative-pair similarities fall below a threshold, and this misalignment can be mitigated by incorporating within-view negative pairs into the objective. Second, in mini-batch settings, smaller batch sizes induce stronger sep- aration among negative pairs in the embedding space, i.e., higher variance in their similarities, which in turn degrades the quality of learned rep- resentations compared to full-batch settings. To address this, we propose an auxiliary loss that re- duces the variance of negative-pan -air similarities in mini-batch settings. Empirical results show that incorporating the proposed loss improves perfor- mance in small-batch settings.
In parallel, several theoretical studies have analyzed the embeddings obtained by CL in full-batch settings. Lu & Steinerberger (2022) showed that the widely used InfoNCE loss (Oord et al., 2018) achieves its minimum value when positive pairs are perfectly aligned, and negative pairs are uniformly separated with the cosine similarity of n-1 where is the size of the training dataset. Lee et al. (2024) further extended this optimality analysis other contrastive losses, including the SigLIP loss (Zhai et al., 2023).
Due to computational constraints, CL methods are typically implemented using mini-batches rather than full batches in practical scenarios. This has motivated recent studies understanding whether the property of the optimal state observed in full-batch settings, i.e., perfect alignment of positive pairs and uniform separation of negative pairs with the similarity of - - also holds under mini-batch settings. Cho et al. (2024) partially addressed this by showing that, for the InfoNCE loss, the optimal embeddings of mini-batch set- tings are identical those of full-batch settings, only when the sum of all possible mini-batch losses is minimized. Ko- romilas al. (2024) explored embeddings learned through kernel-based contrastive losses (Li et al., 2021; Waida et al., 2023) in mini-batch settings. While these studies provide valuable insights into understanding CL, their analyses are limited to specific forms of contrastive losses.
1. Introduction
Contrastive learning (CL.) has emerged a powerful ap- proach to representation learning (Chen & He, 2021; Chen et al., 2020; He et al., 2020; Radford et al., 2021; Zhai et al., 2023; Khosla et al., 2020). In CL, an embedding model is trained to produce similar representations for two different views of the same data instance- referred to as a positive aair-and dissimilar representations for views from differ- ent data instances- empirical studies have shown that representations learned by these objectives, i.e., aligning positive pairs while separal-
To address this limitation, we propose unified framework for analyzing the embeddings obtained by both full-batch and mini-batch settings. Our analysis centers on the cosine similarity of embeddings of both positive and negative pairs. By characterizing the statistical properties of similarities of these embeddings, we reveal how different contrastive losses influence the structure of the learned embedding space under various training conditions.
to as negative pairs. Recent
Our key contributions are summarized follows:
'Department of Statistics and Data Science, Yonsei Uni- versity, Scoul, Korea. Correspondence to: Jy-yong Sohn <jysohnl 1008@yosei.accrr>>>
In full-batch settings, identify fundamental trade-off between the alignment of positive pairs and the separation of negative pairs. Specifically, we show that the perfect alignment of positive pairs is not feasible when the av- erage similarity negative pairs falls below the certain threshold n-T- We demonstrate that such misalignment
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

On the Similarities of Embeddings in Contrastive Learning
arises in class of existing contrastive losses, which can be mitigated by incorporating within-view negative pairs into the contrastive loss.
embeddings that minimize the InfoNCE loss form a simplex Equiangular Tight Frame (ETH) (Papyan et al., 2020; Sustik et al., 2007), where each positive pair is perfectly aligned and negative pairs are equally separated at the same angle, resulting in maximal separation among representations. Lee et al. (2024) further showed that the sigmoid- con- trastive loss, a variant of the softmax- InfoNCE loss, also achieves the same simplex ETF optimum when the temperature parameter of the loss is sufficiently large. This optimal simplex ETF structure still holds even in mini-batch settings, provided that optimization is performed over all possible mini-batch combinations, rather than a single batch at a time (Cho et al., 2024). Building on these findings, we introduce a unified theoretical analysis of the similarities of embedding pairs.
In mini-batch settings, we demonstrate that negative pairs within the same batch exhibit stronger separation com- pared to those from different batches. As a result, we show that smaller batch sizes induce higher variance in the similarities of negative pairs the embedding space. We identify this increased variance as a distinctive feature of mini- settings that is absent in full-batch settings.
Motivated by prior studies that full-batch settings often outperform their mini-batch counterparts, we hypothesize that the increased variance may underlie this performance gap. To explore this, we propose an auxiliary loss that can be integrated into contrastive losses to reduce this variance. Empirical results show that incorporating the proposed term improves the performance of CL methods, especially in small-batch settings.
Effect of Batch Size in CL. CL shows outstanding performance, particularly when trained with large batch sizes (Chen et al., 2020; Radford et al., 2021; Pham et al., 2023; Tian et al., 2020b; Jia et al., 2021). However, large batch sizes require substantial memory resources, which poses practical challenges and often necessitates the use of smaller batches. This compromise in batch size typically leads to performance degradation, motivating several the- oretical studies to investigate its causes (Cho et al., 2024; Koromilas et al., 2024). For example, Yuan al. (2022) demonstrate that the optimization error in SimCLR (Chen et al., 2020) is upper bounded by a function inversely propor- tional to the batch size, indicating that smaller batches yield larger optimization errors. Additionally, Chen et al. (2022) show that contrastive losses exhibit increasing discrepan- cies between the true gradients and those estimated during training as the batch size decreases. While previous studies have primarily focused on optimization error and gradient estimation, we prove that training with small batch sizes leads to increased variance in the similarities of negative pairs in learned embeddings.
2. Related Work
Contrastive Loss. The InfoNCE loss (Gutmann & Hyvärimen, 2010; Oord et al., 2018) is widely adopted con- trastive loss that has been applied to various tasks (Wu et al., 2018; Hjelm et al., 2019; Bachman et al., 2019; Chi et al., 2021; Gao et al., 2021; Qian et al., 2021). SimCLR (Chen et al., 2020) modifies the InfoNCE loss to improve robust- ness to augmentations by treating different augmented views of the same instance positives and all other augmented instances in the batch as negatives. However, SimCLR si- multancously optimizes both positive and negative pairs in the normalization, which can introduce confficts during optimization. To address this issue, Decoupled Contrastive Loss (DCL) (Yeh et al., 2022) modifies the normalization so that the selection of negative pairs is restricted. Building on DCL, Decoupled Hyperspherical Energy Loss (DHEL) (Ko- romilas et al., 2024) further refines the selection by focusing on negative pairs between augmented views of the same instance, which are more challenging than those between different instances. Figure 5 visualizes which pairs are in- cluded as positives and negatives for each method.
3. Problem Setup
Let (ar, y) denote a pair of data points used for model train- ing, where and y correspond to two distinct views of instances. This formulation provides a unified framework for CL. in both unimodal (Chen et al., 2020) and multi- modal (Radford et al., 2021) settings. In the unimodal case, x and y are two randomly augmented views. the mul- timodal case, x and y are views from different modalities. For clarity, we present our analysis in the unimodal setting, but the findings also apply to the multimodal case.
Meanwhile, Zhai al. (2023) raised concerns about the softmax function in the InfoNCE loss, noting that it causes all instances to be dependent on each other through normal- ization. To address this limitation, they propose replacing the softmax with a sigmoid function, which allows each in- stance to be processed independently in an additive manner.
Understanding CL Through Embedding Structures. Several studies have examined how optimal embeddings should be structured to minimize contrastive loss (Lee et al., 2025). Lu & Steinerberger (2022) showed that the optimal
In CL, an encoder f(-) E Rd is trained to map inputs into d-dimensional embedding vectors, thereby representing the data. The encoder is assumed to produce normalized em- beddings such that ||/u||2 = for all embeddings u. This
2

---

On the Similarities of Embeddings in Contrastive Learning
normalization is commonly adopted in related works for mathematical simplicity (Wang et al., 2017; Wu et al., 2018; Tian et al., 2020a; Wang & Isola, 2020; Zimmermann et al., 2021; Cho et al., 2024; Lee et al.. 2024), and is widely used in practice, as experimental results consistently demonstrate its effectiivness in improving performance (Chen et al., 2020; Chen & He, 2021; Xue et al., 2024).
they are the encoder outputs of randomly augmented views.
For simplicity, the subscript is omitted for the set of all pairs, i.e., (U, V) == (n). Vin)).
Let f be the optimal encoder that minimizes the expected contrastive loss, given by
f == arg min E (U. (.. f
(1)
The encoder produces outputs which together form an embedding pair (u, v). When the embedding pair is generated from augmented views of the same instance, it is called a positive embedding pair and is encouraged to be similar. In contrast, a negative embedding pair, where each embedding comes from different instances, encouraged to be dissimilar. For simplicity, we refer to positive embedding pairs as positive pairs and similarly to negative pairs, when there is no risk of confusion.
f(x) and v = f(y),
where L (U, V) denotes the contrastive loss for given sample (U, V). In this work, we focus on two specific forms of contrastive losses used in practice:
Definition 3.1 (InfoNCE-Bar) -ased Contrastive Loss). For a given index set I S [m]. the contrastive loss Linfossym (Uz, V1) is defined in its symmetric form as
1 (Un, Vi)+ Linto UT),
Linfo-sym (U,, Vr) = 2
Formally, (x, y) denotes the distribution of positive pairs, and Pueg (2. y) represents the distribution of nega- tive pairs. As in prior work (Wang & Isola, 2020), the marginal distribution of augmented view æ denoted by Pz, and similarly use Py to denote the marginal distribution for y. These marginals satisfy the following conditions: (=)))) = [ Ppos (ar, yy = Pueg yd for all æ, and (y) = [ Ppes (x, d) = f Pneg (v, y y)dæ for all y. Note that the randomness in these distributions arises from the data augmentation process used to generate different views.
(2)
where the asymmetric component Liuto (U1, V1) is
Linto (U,, v1) =
1
Σ
@ ((vg- - -v)) T uu))
iE!
SENVO
+ C2
Σ
& ((u--b;) - -b;) u:)
JEN(A
for some constants (2) E ((0, (1,0), (), (1, and some convex and increasing functions o, + R
Let n be the size of the training dataset. For any positive integers and with < b n, define the index sets (a : : {a, a + 1,... ,b and [a] := [1 : a). where index E [n] refers to the i-th instance in the dataset. For i E [m]. let Poos yx denote the empirical distribution of positive pairs derived from the i-th instance. We assume that the supports of Pioce and Ppc.. are disjoint for き j. as each instance is distinct. Moreover, we assume that each instance is used equally for training. Therefore, the empirical distribution of all positive pairs, denoted Ppos (x. (x. y), can be written as (x, y) = Eie [m] Pipca (x. y). Similarly, the empirical distribution of all negative pairs is denoted by Pneg (x, y).
Definition 3.2 (Independently Additive Contrastive Loss). For a given index set C [n]. the contrastive loss Lind-ada (U,, Vr) is defined as
Lind-ada (Ur, VT) = I
Σ
Mu vi)
(3)
IEI
+
C1
(u, uz)
[
iEjET
C2
(v(u, ut) + (0, ((TTo,)) v,))
2)/1(144 - 1) iøjEI
Following the notations introduced in Koromilas et al. (2024), we denote the element- pushforward measures induced by the encoder f as frP2, J1p Jap and f SaPueg- For example, faPneg (u, v) represents the empirical distri- bution of negative embedding pairs, i.e., the distribution of (u, v) = f(y) where (x. y) comes from Pueg.
for some constants (21, E {(0. 1). (1, 0), (1, where & IR - R is a differentiable, concave, and increasing function, and 1p : R + RR is a differentiable. convex, and increasing function. Here, i A j E is simplified notation representing i E I and j E I 1 {i].
The constants (c),02) E {(0, 1). (1, 0), (1, in both loss formulations determine which types of negative pairs are included in the contrastive loss. The cross-view negatives re- fer to pairs of embeddings from different views of different instances, i.e., (u,, Dj) with + j, whereas within-view neg- atives are pairs from the same view but different instances, i.e.. (u, uz) or v,) with i + j (Shen et al., 2016). Set- ting C = includes cross-vies view negatives, while e2 = 1
Contrastive Loss. For any a I b with a, b E [n]], let (Ua),,, Visss) := {(,,, v,) : E [a : b]} be the set of embedding pairs for instances whose indices range from a to b. The notation (U[az): M(saj) ~ fa indicates that each positive pair (u,, v,) is sampled from falpece for all i E (a : b). Note that u, and v, are random variables, as
3

---

On the Similarities of Embeddings in Contrastive Learning
12g
V1
11g 12g
Un V1 V2
V1
41
U11
U2
122
112
neg.
poss.
pos.
11
neg.
11
11
neg.
V1
V1
V1
nog.
nog
10g
mag
V2
2
Y2
pos.
pos.
po6.
Vn
ncg
V
meg
1008.
(a) (2) = (1,0)
(b) (c1, 02) (0, 1)
(c) (cI, 02) = (1,
Figure 1. Illustration of negative pair considered in the loss formulations defined in Def. 3.1 and Def. 3.2, which depends on the choice of (cr, c2). Each grid shows all possible pairs embeddings Uin) and Vin), and each cell represents one pair. Green regions represent positive pairs, and blue-striped regions indicate which negative pairs are included in the loss.
includes within-view negatives. Figure 1 summarizes which types of negatives are incorporated for each configuration of (cI, c2). Further discussion on the distinction between cross-view and within-view negatives in the single-modal case are provided in Appendix B, emphasizing that their key difference lies in how they are structurally incorporated into the loss, not in how they are generated.
similarity of embeddings of a positive pair is defined as
s( Ppoe) : f(x)² f(y) for (z, y) ~ Ppos
dubbed as the positive-pair similarity for the encoder S.
Similarly, the similarity of embeddings of a negative pair defined
s == f(y) for (x, ~ Pneg.
Remark 3.3. The first form of losses in Def. 3.1 encom- passes a variety of contrastive losses such as InfoNCE (Oord et al., 2018; Radford al., 2021), SimCLR (Chen al., 2020), DCL (Yeh al., 2022), and DHEL (Koromilas et al., 2024), see Appendix A. The second form in Def. 3.2 includes contrastive losses such as SigLIP (Zhai et al., 2023) and Spectral CL (HaoChen et al., 2021), see Appendix A.2.
dubbed as the negative-pair similarity for the encoder f. We call both similarities as embedding similarities.
Note that the similarities of embeddings in Def. 4.1, de- noted by s(f: ppos) and (neg), are random variables, where randomness arises from data sampling and augmenta- tion. Specifically, positive pair is generated by selecting an instance from a dataset of size n and applying two ran- dom augmentations. Similarly, a negative pair is generated by randomly selecting an instance pair from the n(n - 1) possible combinations and independently applying random augmentations to each instance.
The difference between the two forms of losses lies in com- putational efficiency. The first form in Def. 3.1 necessitates simultaneous computation based pairwise similarities across the entire set of embeddings due to the need for nor- malization, which becomes impractical for extremely large batch sizes. In contrast, the second form in Def. 3.2 is in- dependently additive, allowing it to compute components of each pairwise similarity individually and aggregate them, making it applicable to larger datasets.
Expectation & Variance of Negative-pair Similarities. Recall that the negative-pair similarity s(fipneg) is random variable. Here we investigate how the expec- tation and the variance of embeddings. First, as the expectation E [s([: Pueg)] in- creases, the negative pairs are less separated, which typ- ically degrades the representation quality. Second, the high variance Var [s(f: Poeg)] implies that some negative pairs are mapped unusually close, while others mapped much farther apart. Figure 2 shows the effect of the variance Var [s(f] of the negative-pat similarities. Here, we have three different cases of learned embeddings [u, veeee in three dimensional space. While all three cases have same expectation E (f:]))))), the geometry
affect the learned
4. Similarities of Embedding Pairs
In CL, the encoder is trained to bring positive embedding pairs closer together while pushing negative embedding pairs further apart. Accordingly, the cosine similarities of embedding pairs provide a straightforward way to evalu- ate how well representation achieves its objective. These similarities are formally defined as follows.
Definition 4.1 (Similarities of Positive/Negativee Pairs), The
4

---

On the Similarities of Embeddings in Contrastive Learning
Mini-batch loss: arg min {c (U), (ua)) + & (U/a.4), Va:4))
Full-batch loss: arg min (c (14)) VIsa))
U1
V/
2/2
U3
44 14
12 V2
U1 V1
21
U3 3/
U4 V
14 11
U2 = V/
U4
(a) Bussetulu 0,)= Variasey [u] [u]e]]] (b) Bspsetyllu ey) = Varanseggll (epesseglluu]]]
(0) Eiazeeaa (4] my)
3' Vari [M] 13) 1=]=0
Figure Visualization three different cases of eight embeddings on the three-dimensional unit sphere. Positive embedding pairs are represented in the same color and share the same subscript, while negative pairs refer to any two embeddings with different subscripts. (a) and (h), embeddings minimize the fixed mini-hatch contrastive loss described Theorem 5.5, with the batches partitioned as (ui, v1, u2, 02) and {us, vs, u4, va). In (c). the embeddings minimize the full- contrastive loss described in Theorem 5.1. In all cases, the expectation of negative- -aiir similarities, Exyselt [u, dy}. remains the same. However, the variance of negative- -piir similarities, Var [u," v, increases mini-bateh settings, indicating that some negative pairs are much more similar to each other while others more dissimilar.
of embeddings significantly changes depending on the vari- ance Var (f Pueg)]. One can confirm that the rightmost case having equi-distant embeddings (u, (e(4) achieves the zero variance of negative-pal -airr similarities.
5. Behavior of Learned Embeddings
In this section, we interpret the behavior of embeddings trained by contrastive learning, through the lens of similar- ities of positive/negative pairs, denoted by s( Ppos) and s( sf; ; Pueg), defined in Sec. 4. We begin by examining the full-batch CL and subsequently extend our analysis to the mini-batch CL. Proofs for all statements in Sec. 5.1 and Sec. 5.2 are provided in Appendix C.3 and C.4, respectively.
Comparison with Existing Metrics. The similarities of embeddings in Def. 4.1 are related with metrics proposed in previous work. For example, the alignment metric used in (Wang& & Isola, 2020) can be represented as
5.1. Full-Batch Contrastive Learning
(( [1l-- ain-v1122 01[]] 2E [s Ppus)] + 2, (4)
Recall that we consider various contrastive losses which can be categorized into two parts: the InfoNCE-based con- trastive loss in Def. 3.1 and the independently additive con- trastive loss in Def. 3.2. Our first main result below provides the similarities of positive/negative pairs for the two types of contrastive losses, when the encoder is full-batch trained.
which is related with the positive-pair similarity ppppp)))-
Here, the higher alignment value indicates that representa- tions are largely invariant to random noise factors introduced by augmentations.
On the other hand, the uniformity metric used in (Wang & Isola, 2020) is related with the negative-pain similarity s(f; Pueg), since it defined by the logarithm of the Gaus- sian potential function (Cohn & Kumar, 2007)
Theorem 5.1. Suppose d 2 n - 1. Let the contrastive loss L(U.V) be one of the following forms:
i. Linforsym (U.V) in Def. 3.1.
log Eu- exp (- ||u =13)]
ñi. Lind (U,V) in Def. 3.2, where (c,, (1,1)).
(5)
w f18
iii Lina-sda in Def. 3.2, where (c1,02) = (1,0) and
~ 2 (E [f: pueg)] + Var [s( Pueg)] 1), (6)
&' (1) > 2n- T/
n-2
T--1
where the approximation in (6) is detailed in Appendix C. The uniformity metric measures how representations are uniformly distributed on the unit hypersphere, and thus a lower uniformity value indicates that the representations preserve more information from the data.
Then, in full-batch settings, the embedding similarities for the optimal encoder f in (1) are
1
8(f" = ((f*;Peej)
n2
5

---

On the Similarities of Embeddings in Contrastive Learning
According Theorem 5.1, all positive pairs achieve the perfect alignment, while all negative pairs are uniformly separated with the cosine similarity of - This is generalized version of previous studies (Lu & Steinerberger, 2022; Cho et al., 2024; Lee et al., 2024; Koromilas et al., 2024), where we extend in two directions. First, the CL loss formulations (Def. 3.1 and Def. 3.2) we considered are a much broader class of losses compared with existing work. Second, consider the randomness of n embedding pairs in the optimization, where this randomness is introduced through augmentations. Now we analyze the behavior of embeddings for arbitary encoder f. that is not necessarily in the optimal status f*. The below result provides the relationship between the positive-pair similarity and the negative-pair similarity.
increases if the positive-pair similarity drops below 1. When the condition in (8) is satisfied, ignoring the scaling factor, the loss reduction from separating negative pairs beyond the similarity threshold of - can be greater than the loss increase from reducing positive-pair similarity below 1. As a result, the optimization process favors pushing negative pairs even further apart, leading to the excessive separation. We provide a specific loss where this issue arises:
Example 5.4. Consider the sigmoid contrastive loss Csig( (U, V) (Zhai et al., 2023), defined as
Caig UU V): Σ
log (1 + exp (-tu, v.) exp(b))
71
:E|n]
+ n
log (1 +exp (Iu, 05) eexp(-b))
Theorem 5.2. For any normalized encoder f.
(9)
E [s (f: s + [E[s (fiPne)) + n 1
) (7)
where >> O and b E IR are hyperparameters. This loss follows the form of Lind-ada (U, V) in Def. 3.2, where (c1, - 0), Φ(x) = - log(1 + exp( + b)), and bxz)= (n log(1 + exp(tx - b)). If hyperparameter
where the equality in (7) holds if and only if tr (fa [u vi)) = 0 and Eur/
lu lu+ vj O.
and b are chosen such that
Theorem 5.2 highlights the relationship between the expec- tation of positive-pate similarities, IE [s (f: Ppus)], and that of negative-pair similarities, E (J: Pueg)]. ). When the av- erage of negative-pait similarities drops below T- the positive pairs cannot be fully aligned, which is not desired. We call such phenomenon as the excessive separation of negative pairs in full-butch Cl., since the average of negative- pair similarities drops below _ when negative pairs are more separated compared with the optimal status specified in Theorem 5.1. This issue may arise when certain losses are used, as shown in the following theorem.
1+ exp R-1 bb)
n -
1 expxp(1 b) <
(10)
2
then embedding similarities for the full- full-batch optimal en- coder f in (1) satisfy
1
n-T
:Pos) < 1, s(f":
n2 1
Note that (10) is just a rephrase of (8) by plugging in e and e for the sigmoid contrastive loss. When the hyperparame- ter b of the sigmoid contrastive loss sufficiently small, the condition in (10) is satisfied, thus the learned embedding suffers from the excessive separation issue. This can be also explained by the sigmoid contrastive loss formula. In (9), the relative weight of second term (compared to the first term) increases as decreases. In such case, minimizing the negative-pal pair similarity becomes more important. Con- sequently, decreasing b induces the excessive separation of negative pairs.
Theorem 5.3 (Excessive Separation in Full-Batch CL). Sup- pose that d 2 n. Consider the contrastive loss L (U, V) in the form of aid (U, V) in Def. 3.2, where (2), = (1, 0) and
n1 2
1 - n 1
o' (1)
2(n
Then, under full-batch settings, the embedding similarities for the optimal encoder f* in satisfy
s(f": < 1,
:Pog)
<
n2
Mitigating Excessive Separation. A natural question that arises is whether the excessive separation of negative pairs in full-batch CL can be mitigated. According to Theorem 5. and Theorem 5.3, this issue depends the specific form of the contrastive loss. In particular, under the independently additive loss Lind-ada (U, V). the optimal embeddings do not suffer from excessive separation when C2 = 1 (i.e., case (ii) of Theorem 5.1), whereas the issue arises when C2 = 0 and condition (8) holds.
The inequality condition on & and vb in (8) explains why the loss Lind-ada (U, V) in Def. 3.2 causes the excessive separation of negative pairs. Specifically, Lind-ada(U) V) is formulated as the sum of v-()) over negative pairs and - over positive pairs. Therefore, v'(- 1 n indicates how much the loss decreases if the negative-pait similarity falls below - while (1(1) measures how much the loss
6

---

On the Similarities of Embeddings in Contrastive Learning
This observation suggests two potential solutions the excessive separation problem. The first is to set 02 = in the loss, effectively incorporating within-view negative pairs. The second is to tune hyperparameters such that condi- tion (8) does not hold, i.e., case (iii) in Theorem 5.1. While both approaches are theoretically valid, the former offers a more principled and practical remedy. In contrast, the latter lacks clear guidance for selecting suitable hyperparameters and may incur significant computational overhead. There- fore, we advocate including within-view negative pairs as straightforward and effective strategy to avoid excessive separation in full-batch CL.
negative-pair similarity is not uniform across the pairs, i.e, the variance is positive when the mini-batch size m is strictly less than the sample size n. Thus, the effect of using mini- batch (compared with using full-batch) is in the increased variance of negative-pale similarities. Throughout the paper, we call such phenomenon as the excessive separation of negative pairs in mini-batch CL.
Figure 2 visualizes the effect of using nini-batches, com- pared with full-batch settings, when full-batch settings, shown in (c) of Figure 2, the variance of negative- serattive-pirraae similarities is zero, indicating that all negative pairs are equi-distant, In contrast, (a) and (b) of Figure 2 illustrate the embeddings learned by mini-batch settings, where the fixed mini-batches are specified as (U[12)), V1:22) and (U(a.4, Via.4)-- For both (a) and (b), the variance of negative-pair similarities is positive, where (a) represents the case that achieves the highest variance in (II), and (b) corresponds the case that achieves the lowest variance.
and m 2. For
5.2. Mini-Batch Contrastive Learning
In Sec. 5.1, we show that when a proper contrastive loss is chosen for full-batch settings, the learned embedding satisfies the following behavior: all negative pairs have the cosine similarity of -- - and all positive pairs are perfectly aligned. What about the practical scenarios when we use mini-batches for training? Suppose n training samples are partitioned into b mini-batches where each batch contains m := nn/b samples. Consider training the embeddings by under the fixed mini-batch configuration, where k-th mini- batch contains the samples with indices in I = {mik 1) + 1, mmk - 1) + 2, ..... mkk. Under this scenario, the below theorem analyzes the behavior of embeddings learned by mini-batch training.
Effect of Batch Size. Note that the variance of negative- pair similarities in (11) depends the batch size m. For example, in the full-batch case where m the upper bound in (11) is zero, which is consistent with Theorem 5.1. One can confirm that the upper and lower bounds on the variance is a monotonically decreasing function of m, which implies that smaller batch sizes inherently exacerbates the excessive separation of negative pairs in mini-batch settings.
Theorem 5.5 (Excessive Separation in Mini-Batch CL).
The below theorem analyzes the effect of batch size on the training dynamics, when a popular CL loss is used:
Suppose d 2 m - 1. Let the contrastive loss L (U, V) be one of the forms in Theorem 5.1. Define fhstch as the optimal encoder that minimizes the fixed mini-batch loss, given by
Theorem 5.6. Consider the InfoNCE loss LItOONCE (U.V) (Oord et al., 2018), which corre- sponds to the loss Linto-sym (U, in Def. 3.1 where o(z) = exp(x/t) for some >> 0, (zz)) = log(1 + z). and (C1, (1,0). For any two integers mi, m E [n] satisfying m1 S m2, the gradient of the InfoNCE loss with respect to the negative-pair similarity satisfies
Sbatch == arg min E (U. (abll)
Σ LU Vin) kE|b
f
where Is := [m - 1) + mk] for K E [b). Then, em-
bedding similarities for the optimal encoder Siarch satisfy
s( Ffrachhi (hos) = 1,
LIntONCE (Umma)i Vinal)
Ō (u. U₁)
1
E (Sbatthhi pmeg)] = n 1¹
2 (u, v,) LInfONCE (Umaa) Vim.))
(12)
Var [s(Seatchi Pueg)]
E (m-
n-m (m) [](n
1](n
1721
(11)
for any distinct indices i + j j.EE. [m]] Moreover, the equality in (12) holds if and only if mm = 1/2.
A necessary condition for attaining the minimum variance of negative-pan pair similarities in (11) is 2 b(m - (1)
According to Theorem 5.6, the gradient of the loss with respect to the negative- pair similarity is always non-negative, implying that gradient descent decreases the similarities of negative pairs, thereby pushing them further apart. Notably, the magnitude of this gradient increases as the batch size gets smaller. As a result, negative pairs within each batch exhibit greater separation the batch size gets smaller.
According to Theorem 5.5, the embeddings learned by mini-batch settings have the following behaviors. First, the positive-pair similarity is equal to 1, i.e., all positive pairs are fully aligned. Second, the expectation of negative- pair similarities is equal to - which happens for full- batch settings as well. Third, unlike full-batch settings, the
7

---

On the Similarities of Embeddings in Contrastive Learning
Proposed Variance Reduction Method. Prior empirical studies on CL have shown that mini-batch settings often underperform compared to full-batch settings (Chen et al., 2020; Radford et al., 2021). This naturally raises question: What is the main factor contributing to this performance degradation in mini-batch settings, and how can be ad- dressed? According to Theorem 5.5, from the perspective of cosine similarity of embeddings, the key difference intro- duced by mini-batch settings lies in the increased variance of negative-pai similarities. Motivated by this theoretical insight, we propose an approach to improve mini-batch contrastive learning by introducing an auxiliary loss term, LVaNs (U, V), which explicitly reduces the variance of negative-pair similarities:
Table I. Variance of the similarities of embeddings of negative pairs, obtained from models trained with diffcrent batch sizes, Each model is trained using either the SimCLR loss alone jointly with our auxiliary loss in Def. 5.7, which is proposed to reduce this variance. One can confirm that the variance is effectively reduced by using the proposed auxiliary loss.
Variance of negative- -airi similarities
Batch size
SimCLR
SimCLR + Ours
32
0.1649
0.1008
64
0,1505
0.0952
128
0.1444
0.0929
256
0.1404
0,0921
512
0.1396
0.0917
Definition 5.7 (Reducing Variance of Negative-Paii Pair Simi- larities). Let m be the mini-batch size. Define
Koromilas et al., 2024), we use a ResNet-18 18 encoder (He et al., 2016) followed by two-layer projection head. The models pretrained on CIFAR-100 (Krizhevsky al., 2009) by minimizing the SimCLR loss with the temperature parameter of t = 0.2. Five models are trained with mini- batches sampled uniformly at random, with batch sizes of 32, 64, 128, 256, and 512, respectively. Additional details of the experimental setup are provided in Appendix D.
2
(URNS(U(m) [m] V(m) Vmm): = m(m- - 1),
Σ
u
n- 1
(fjE(m)
as the auxiliary loss for reducing the variance of negative- pair similarities.
One can combine arbitrary conventional mini-batch loss C (Umm). Vmmi) with the proposed auxiliary loss to get the modified loss, given by
Based on these pretrained models, we generate 5,000 posi- tive embedding pairs by applying random augmentations to the training data and extracting the corresponding outputs from the projection head. We then compute the cosine sim- ilarities of negative pairs, and report the variance of these similarities in Table 1. As shown in the table, training with smaller batch sizes leads to higher variance in begative-pain similarities, which aligns with the result in Theorem 5.5.
L (m)) Vim)) + A CVRNS (U[m), Vimi)
where À > is hyperparameter. By including the proposed term into the mini-batch loss, we encourage all negative-pair similarities to be close to - the ideal value achieved in full-batch settings in Theorem 5. As a result, the proposed loss controls the variance of negative-pair similarities.
To evaluate the effectiveness of our proposed auxiliary loss, we train five models for each batch size by min- imizing the SimCLR loss combined with the auxiliary loss LVNNS UU V) in Def. 5.7 with the hyperparameter of = 30. The variances of negative-pair similarities from these additional models are shown in the last column of Table 1, and are consistently reduced across all batch sizes. This indicates that our proposed loss effectively mitigates excessive separation of negative pairs in mini-batch settings.
6. Empirical Validation
In this section, we empirically validate the impact of our theoretical results discussed in Sec. 5, especially for the practical scenarios of mini-batch settings. First, we empiri- cally observe that the excessive separation of negative pairs (proven in Theorem 5.5) actually happens in experiments on benchmark datasets. Second, we empirically confirm that such excessive separation issue can be mitigated by using the proposed loss in Def. 5.7 which reduces the variance of the negative-pair similarities. Third, we observe such variance reduction improves the quality of learned represen- tations in various real-world experiments.
6.2. Effect of Variance Reduction on Performance
We further investigate whether reducing the variance of negative-pair similarities improves the quality of learned representations in terms of the downstream performances.
Experimental Setup. We pretrain models on CIFAR- 10, 110 (Krizhevsky et al., 2009), and ImageNet (Deng et al., 2009) using various contrastive losses that follow the formulation in Def. 3.1, including SimCLR, DCL, and DHEL. For all methods, we compare models trained with and without incorporating the auxiliary loss LVRNS V)
6.1. Excessive Separation of Negative Pairs
To investigate the excessive separation of negative pairs in CL. we evaluate the variance of the negative- -paii similarities of embeddings learned by real-world experiments. Follow- ing prior works on contrastive learning (Chen et al., 2020;
8

---

On the Similarities of Embeddings in Contrastive Learning
SimCLR
89
CIFAR-10
80
87
60
50
SimCUR-+ + Ours (CIFAR 10)
accuracy
SimCLR (CIFAR 10)
256
512
SimCUR + Ours (CIFAR 100)
40
SimCLR (CIFAR 100)
CIFAR-100
0.07
0.10
0.50
1.00
2.00
temperature parameter
60
59
Figure 3. Classification accuracy CIFAR datasets. Models are trained by minimizing the SimCLR loss with and without the auxiliary loss proposed in Def. 5.7, using various temperature parameters in the SimCLR loss.
57
56
64
128
256
512
batch
in Def. 5.7. The hyperparameter À for the proposed loss tuned over {0.1, 0.3, 1, 3, 10, 30, 100). Unless otherwise specified, the other settings follow those in Sec. 6.1.
Figure 4. Effect of the auxiliary loss proposed in Def. 5.7 top-1 classification accuracy when combined with various baseline meth- ods (SimCLR, DCL, and DHEL)) on CIFAR datasets. The gray bars highlight the performance gains achieved by incorporating the proposed term across various batch sizes. The proposed auxiliary loss consistently improves the model performance.
Performance Gains from Variance Reduction. The qual- ity of the representations learned through CL is known to sensitive to the choice of the temperature parameter in con- trastive losses, as influences the distribution of similarities among embeddings (Wang & Liu, 2021). To investigate this sensitivity, train models using the SimCLR loss
7. Conclusion
with temperature values ranging from 0.07 to 2.00. We compare the standard SimCLR loss against our proposed variant, which incorporates the auxiliary loss introduced in Def. 5.7 to reduce the variance of negative-pair similarities. As shown in Figure 3, incorporating the proposed term leads consistently higher and more stable classification accu- racy across all temperature settings, alleviating the need for careful temperature tuning.
To understand contrastive learning (CL), we mathematically analyze the distributions of similarities of embeddings, mea- sured for positive pairs and negative pairs. Our theoretical results in full-batch settings demonstrate that misalignment of positive pairs becomes inevitable when the average sim- ilarity of negative pairs falls below its optimal value, situation that can arise with existing contrastive losses. In mini- Mati-bbtch settings, we prove that the variance of negative- pair similarities increases as the batch size decreases distinctive characteristic absent in full-batch settings and potential contributor to the performance degradation ob- served in mini-batch settings. To address this, propose an auxiliary loss that explicitly reduces the variance of negative- pair similarities. Empirical results show that incorporating the proposed loss improves the performance of CL methods, especially in small-batch settings.
We further evaluate the auxiliary loss in Def. 5.7 on existing CL methods, including DCL and DHEL. Experiments are conducted on the CIFAR datasets with various batch sizes. As shown Figure 4, incorporating the proposed term leads improved classification accuracy, with the effect being more pronounced at smaller batch sizes. Additional results on the ImageNet dataset are presented in Appendix E.
Caveats of Variance Reduction. While auxiliary loss proposed in Def. 5.7 effectively reduces the variance of negative-pair similarities, this reduction can influence both desirable and undesirable sources of variance. On the posi- tive side, it helps reduce variance introduced by mini-batch sampling, which can degrade the representation quality. However, it may also suppress the variance that captures meaningful structure in the data. Further discussion of the limitations the proposed loss is provided in Appendix F.
Promising directions for future work include extending this work in two directions. First, disentangling the variance of negative-pair similarities that reflects intrinsic data structure from that caused by mini-batching could enable targeting only the variance that degrades representation quality. Sec- ond, analyzing the behavior of embedding similarities not only during pretraining but also during fine-tuning may pro- vide deeper insights into its dynamics throughout different stages of training.

---

On the Similarities of Embeddings in Contrastive Learning
Acknowledgements
Cohn, H. and Kumar, A. Universally optimal distribution of points on spheres. Journal of the American Mathematical Society, 20(1):99-148, 2007.
This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Ministry of Science and ICT (MSIT) of the Korean government (RS- 2024-00341749, RS-2024-0034533
da Costa, V. G. T., Fini, E., Nabi, M., Sebe, N., and Ricci, E. solo-learm: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23 2022.
AS-2024-004080003
under the ICT Challenge and Advanced Network of HRD
(ICAN) support program 40S-2023-00259934, RS-2025-
02283048), supervised by the Institute for Information & Communications Technology Planning & Evaluation (IIP)..
Deng, J.. Dong, W., Socher, R., Li, L.-.,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 248-255. leee, 2009.
This research was also supported by the Yonsei University Research Fund (2025-22-0025).
We sincerely thank the anonymous reviewers for their criti- cal reading and constructive feedback enhancing this paper.
Gao, T., Yao, X., and Chen, D. SimCSE: Simple contrastive learning of sentence embeddings. In Conference on Em- pirical Methods in Natural Language Processing. 2021.
Impact Statement
This paper presents work whose goal is to theoretically understand CL through embedding similarities. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Gutmann, M. and Hyvärinen, A. Noise-contrastive esti- mation: A new estimation principle for unnormalized statistical models. In International Conference on Artifi- cial Intelligence and Statistics, pp. 297-304, 2010.
References
HaoChen, J. Z., Wei, C., Gaidon, A., and Ma, T. Provable guarantees for self-supervised deep learning with spec- tral contrastive loss. In Advances in Neural Information Processing Systems, volume 34, pp. 5000-5011, 2021.
Bachman, P., Hjelm, R. D., and Buchwalter, W. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, volume 32, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Chen, C., Zhang, J., Xu, Y., Chen, L., Duan, J., Chen, Y., Tran, S. D., Zeng, B., and Chilimbi, T. Why do we need large batchsizes in contrastive learning? gradient- bias perspective. In Advances Neural Information Processing Systems, 2022.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0009997299999 977899978,,111 2020.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Hjelm, R. D., Fedorov, A.,
S., Grewal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. In International Conference on Learn- ing Representattons, 2019.
Chen, X. and He, K. Exploring simple siamese represen- tation learning. In IEEE/CVF conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Horn, R. A. and Johnson, C. R. Matrix analysis. Cambridge university press, 2012.
Chi, Z., Dong, L., Wei, F., Yang, N., Singhal, S., Wang, W., Song, X., Mao, X.-L., Huang, H., and Zhou, M. framework for cross- lingual language model pre-training. In Conference of the North American Chapter of the Association for Com- putational Linguistics, June 2021.
Jia, C., Yang. Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q., Sung, Y.-H., Li, Z., and Duerig, T. Scaling up visual and lasion-languuge representation learning with noisy text supervision. In International Conference on Machine Learning, 2021.
InfoXLM: An information-theoree
Cho, J., Sreenivasan, K., Lee, K., Mun, K., Yi, S. Lee, J.-G., Lee, A., yong Sohn, J., Papailiopoulos, D., and Lee, K. Mini-batch optimization contrastive loss. Transactions on Machine Learning Research, 2024.
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pp.
2020.
10

---

On the Similarities of Embeddings in Contrastive Learning
Korblithh, S., Shlens, J., and Le, Q. V. Do better imagenet models transfer better? In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Koromilas, P., Bouritsas, G., Ginnakopouloss, T., Nicolaou, M., and Panagakis, Y. Bridging mini-batch and asymp- totic analysis in contrastive learning: From InfoNCE to kernel-based losses. In International Conference on Ma- chine Learning, 2024.
Shen, X., Sun, Q--., and Yuan, Y.H.. Semi-paired hashing for cross view retrieval. Neurocomputiin, 213:14-23, 2016.
Sustik, M. A., Tropp, J., Dhillon, S., and Heath Jr, R. W. On the existence of equiangular tight frames. Linear Algebra and its applications, 426(2-3):619-636 2007.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. In European Conference on Computer Vision, 2020a.
Lee, C., Chang, J., and Sohn, J.-y. Analysis of using sigmoid loss for contrastive learning. In International Conference on Artificial Intelligence and Statistics, 2024.
Tian, Y., Sun, C.,, Poole, B., Krishnan, D., Schmid, C.,, and Isola, P. What makes for good views for contrastive learning? In Advances in Neural Information Processing Systems, volume 33, pp. 6827-6839, 2020b.
Lee, C., Oh, J., Lee, K., and Sohn, J.-y. A theoretical framework for preventing class collapse in supervised contrastive learning. In International Conference on Arti- ficial Intelligence and Statistics, 2025.
Waida, H., Wada, Y., Andéol, L., Nakagawa, T., Zhang, Y., and Kanamori, T. Towards understanding the mechanism of contrastive learning via similarity structure: A theoret- ical analysis. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, 2023.
Lee, H., Lee, K., Lee, K., Lee, H., and Shin, J. Improving transferabilitty of representations via augmentation-awe self-supervision. In Advances in Neural Information Pro- cessing Systems, volume 34, pp. 17710-17722, 2021.
Wang, F. and Liu, H. Understanding the behaviour of con- trastive loss. IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2495-2500 2021.
Li, Y., Pogodin, R., Sutherland, D. J., and Gretton, A. Self- supervised learning with kernel dependence maximiza- tion. In Advances in Neural Information Processing Sys- tems, 2021.
Wang. F,, Xiang, X., Cheng, J., and Yuille, A. L. Normface: L.2 hypersphere embedding for face verification. In ACM international conference on Multimedia, 2017.
Lu, J. and Steinerberger, S. Neural collapse under cross- entropy loss. Applied and Computational Harmonic Anal- ysis, 59:224 1911, 2022.
Wang, T. and Isola, P. Understanding contrastive represen- tation learning through aligmment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929 9939. PMLR, 2020.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation leamn ing with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018
Wu, Z., Xiong, Y.. Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance discrimina- tion. In IEEE/CVF Conference Computer Vision and
Papyan, V., Han, X., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learn- ing training. Proceedings of the National Academy of Sciences, 1 (17(40):24652-2466) 2020.
Pattern Recognition, pp- 3733-3742, 2018.
Xue, Y., Gan, E., Ni, J., Joshi, S., and Mirzasoleiman, B. Investigating the benefits of projection head for represen- tation learning. In International Conference on Learning Representations, 2024.
Pham, H., Dai, Z., Ghiasi, G., Kawaguchi, K., Liu, H., Yu, A. W., Yu, J., Chen, Y.-T., Luong, M.-T., Wu, Y., et al. Combined scaling for zero-shot transfer learning. Neurocomputing, 555:126658, 2023.
Yeh, C.-H., Hong, C.-Y., Hsu, Y.-C., Liu, T.-L., Chen, Y., and LeCun, Y. Decoupled contrastive learning. In Euro- pean Conference on Computer Vision, 2022.
Qian, R., Meng, T., Gong, B., Yang, M.-H., Wang, H., Belongie, S., and Cui, Y. Spatiotemporal contrastive video representation learning. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021.
Yuan, Z., WU, Y., Qiu, Z.-H., Du, X., Zhang, L.,, Zhou, D., and Yang, T. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In International Conference on Machine Learning, pp. 15760-25782. PMLR, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J.,

---

On the Similarities of Embeddings in Contrastive Learning
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In IEEE/CVF International Conference Computer Vision, 2023.
Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. Contrastive learning inverts the data generating process. In International Conference Machine Learning, pp. 12979-12990. PMLR, 2021.
12

---

On the Similarities of Embeddings in Contrastive Learning
A. Contrastive Losses
outline how various losses commonly used in CL can be instantiated by the general formulation provided in Def. 3.1 and Def. 3.2. For each case, we specify the corresponding choices of functions and parameters.
A.1. Contrastive Losses Following Def. 3.1
11 Ug
U1 V1 V2
Vn.
Ug
1p V1 V2
Va
11
U1
U2
Un
V1
205
V1
V2
V2
InfoNCE
SimCLR
11 U2
Up V1
Vn
11 U12
V1 V2
U1
112
U2
11,
11,
205
V1
V2
V2
V
Vn,
DCL
DHEL
Figure Illustration comparing different contrastive losses, following the form of 3.1. The green area represents the positive pair, while the blue-striped regions indicate the negative pairs that are normalized together with the positive pair for each loss.
Table 2. Function and parameter selections Def. 3.1 that correspond contrastive losses.
((x)
sbxx)
Ci
C2
InfoNCE (Oord et al., 2018)
exp(x/t)
log(1 + x)
0
SimCLR (Chen et al., 2020)
exp(x/t)
log(1 + x) log(2)
1
DCL (Yeh et al., 2022)
exp(x/l)
1
DHEL (Koromilas et al., 2024)
exp(x/t)
log(x)
13

---

On the Similarities of Embeddings in Contrastive Learning
1. InfoNCE (Oord et al., 2018), CLIP (Radford al., 2021):
LInfONCE(U (U, V)
1
Σ
log
exp (u. v./l)
log
exp (ue 0z/l)
(13)
2n (E(n
Σje(m) exp (u; 0//6)
2n sE/ml
Eje(m] exp ( vz/t)
log
1+
Σ exp ((V, vz) vz) TT)T(itt) /t//tt
2n
iE(n)
(E
Σ exp ((2) -u;) To/tt) jE(1/
log
1+
2n
RE[n]
where t > 0 is the temperature parameter.
2. SimCLR (Chen et al., 2020):
CSmLLL (U, V) = 2n kE(M)
1
Σ log
exp (2, 0s/t)
EsElnJOX exp (u 2//t) + EyEnn]M exp (2, ui/tt//t)))
exp (u, 0,/t)
log
Ejeln exp [u) 01/t) + Σje(E}( exp (a] v,ttt
iE(n]
log
1 Σ exp ((D) v.) uuitt) + E exp ((u, - vs) Tueettt
2n
(E(N)
JEMN)((
jEMMMM
log
1+
Σ exp ((u, + Σ JE(m]\(0
exp ((D, - u:) To//tt)
2n
JE(m/\(
where & > 0 is the temperature parameter.
3. DCL (Yeh et al., 2022):
exp (u, v,/t) Ejen/o exp (u. vj/t) + EsENNM exp (u. uz/t)/
(U. = 2n
Σ log
sE(m]
exp (u. wi/t)
Σ log
2n iE(n)
EjE(NNO exp (u) v./t) + ExEM)) exp (u] wz/t)
Σ log
(exp ((13 Tuitt) exp ((w) vz) (u:n//l) auiit))
2n
iE(m]
+ 2n
log
(exp ((u, u.) Tv,t) + exp ((V, us) wr/t))
ie(n)
JEINMM
where t > 0 is the temperature parameter.
4. DHEL (Koromilas et al., 2024):
SUHEL(U.V) 2n
Σ
log
exp cxp vilt)
1
log
exp (u. v//t)
Ejemnno exp (ui uz//t)
2n
Eje(n exp (1e) us/t)
VE(n]
sEin]
Σ log
Σ exp (lu, - v;) uu.tt)
Σ
log Σ exp ((o, - u.) To))Tull vsst) wn/t)
+
2n VEIn)
2n
jE
iE(ml
jE
where 1 the temperature parameter.
14

---

On the Similarities of Embeddings in Contrastive Learning
A.2. Contrastive Losses Following Def. 3.2
Table 3. Function and parameter selections Def. 3.2 that correspond contrastive losses.
<(x)
(x))
C1
02
SigLIP (Zhai et al., 2023)
log( +exp (-tx+b))
(n- 1) looloo (1+exp (Lx b))
0
Spectral Contrastive I Loss (HaoChen et al., 2021)
x²²
0
1. SigLIP (Zhai et al., 2023):
SSigLIP(U,V)
Σ
log (1 + exp (-tu, vs + +b) +
Σ
Σ
log (1 + texp (tx vj - b))
inin
sE/mi jEfn/Vt
Σ kE(M
(- log (1+exp (-lu, v. bb))
1 n(n 1)
Σ E (n 1)-log (1+exp (lu, 05 (1- by-b)).
xE(m] JE(n/((
where t the temperature parameter and b R is the bias term.
2. Spectral Contrastive Loss (HaoChen et al., 2021):
Sppectral (U, (U,V) = 1
Σ
u Ug
Σ iE[n] jE(m]M{
(u uz)
n(n
EE(m)
B. Distinction Between Cross- View and Within-View Negative Pairs
L1
11
cross-view negative pair
U
12
V2
12
V2
1
V2
within-view negative pair
u3
V3
u3
V3
113
V3
11
V3
u
u
V1
11
V1
u
V
V
2
U12
u2
V2
11
V2
u2
V2
u3
L
u
V
u3
u
V
(a) (c1, (2) = (1,0)
(b) (c1, 02) = (0.
(c) (c1, 02) = (1,1)
Figure 6. Graphs illustrating different loss configurations for 7 3.
The key distinction between cross-view and within-view negative pairs our analysis lies their structural incorporation within the contrastive loss, rather than in the manner of their generation. To demonstrate that cross-vits view and within-view negatives are not equivalent, we present graph- representation Figure 6, which reframes Figure In these graphs, each node corresponds to an embedding, and each edge indicates negative pair considered the loss.
In the unimodal CL, the distinction between the two views, u4 and for i [3], is not semantically meaningful. Thus, W; and e; may be interchanged without affecting the results. This implies that the four graphs depicted in Figure 6a are equivalent under permutation of views, and the same reasoning applies to Figure 6b. Nevertheless, the overall graph structures in Figure 6a and Figure 6b remain fundamentally different. One can confirm that cross-view graphs are fully connected bipartite, whereas within-view graphs consist of disconnected subgraphs. This topological difference highlights their non-equivalence.
15

---

On the Similarities of Embeddings in Contrastive Learning
C. Proofs
C.I. Approximation of Uniformity Metric
Under the normality assumption on u v. Proposition C.I gives
log R exp (2u v)] = (TE [u=T)) Var [u v))
Accordingly, the uniformity metric be approximated
exp (- || 1 m1 Ps log E. (u, Baor exp 1-1lu= m113)
log fsp vー
Rs 2 (E (8]:: Pura)) + Var (s(] Prea)) - 1).
by Proposition C.2, as goes to infinity.
Proposition C.1. Assume that the random variable u follows the normal distribution, Then,
log E (exp (2u' [2uTv]]]] (EE [u v) + Var [u (T)))
Proof. Let u² v. Since X follows the normal distribution, define μ [[ and o² = Var[X]. Note that the moment generating function of normal distribution is given by
E [exp(tXX] exp (ut 0212
Substituting = have
[exp(2X)) = exp (2 202) exp (2 ] 2Vrr(X]),
which is equal to
log E [exp(2X)) = 2 [E + Var (ar(X])-
Since X u² v, conclude
log E exp (2u [u]] (E [uJ + Var [ Te)=)
C.2. Proofs for Relation Between Positive and Negative Pairs
Proposition C.2. The distribution of negative pairs satisfies Pnog (x, y) = (xx))), for all æ and y. However, for training dataset of size n, the empirical distribution of negative pairs is given by
Puag (xe 9) n (2x(2)pb()) n
Ppan ( (x.y),
for all x and y.
Proof. The empirical distribution of positive pairs, Ppos. is defined under the assumption that all instances are equally weighted with the probability Pr(T = i} = 1 for all E [n], where a random variable representing the index. Under this assumption, the probability that a randomly selected pair is positive is
1
1
Pr/pos} Pr(i = = Σ Pr(i= i) Pr(i *E(m)
n
n
Then, the empirical distribution of negative pairs, Pneg. is subsequently derived
P= (x)) = Pposs (x, y) Pr(pos} + Pneg (2t y) Pr (pos))
1
Ppos (x, y) Poog (x. y)
(14)
n
16

---

On the Similarities of Embeddings in Contrastive Learning
which leads
Pueg (x. y) nt n- Pr (2(
Ppos (2, y)
n
Moreover, as oc, the above result implies that Pneg( (0, y) Pn (= (x)
Lemma C.3, Assume that the encoder f(-) satisfies [[f(2))]1 = for all a. For any distribution p. the following holds.
tr (s) lu - 01) + 1 2
(i [u' TuTur u] + Eu fap w- IpPs
u TuT
() [u + w
Proof. Note that
Biu [u w1l12
( faPpo
[u + will2 -4E (,) SeDos [2]] SiPpoo [0]
4 fan [2x) Bowsssy [0),
(15)
where the equality in (15) follows from the assumption of matching marginals. From the definition variance, we have
tr (Var faPpon lu -a-00 v)) = S)
tr (r((uu00) -)( SaPone u)) ((u-o) E)( [u 2i) b1)')]
[| - v) [u S:P 2i [
B(
u 0112 [[(u,))~ /ePoos [u - 21 112
B((,o)>
E(u,v))
[2 u v) [B [u + m|112 4E frD, [2a] T Fow
21 (1)
[u u] -I (u + + 41E [w] Eew |0].
(16)
By rearranging (16) and dividing by 2, we have
2 ir (Var [u - v)) + || E (sp (u w| - E(u SsPpo
[u T v] + En [2s] Euw
B3 1u v Euw fop [u TeT
Lemma C.4. Assume that the encoder f(-) satisfies [] = 1 for all æ. For any empirical distribution f with a sample size of n, the following holds.
1 [u To] tr (s) vl) + (E [u a
E (e), [u ul
71
n
Proof. Using Proposition C.2, the expectation over the empirical distribution can be decomposed into the expectations of positive and negative pairs as follows.
n2 - = [u' v0
n E B), [u]
Applying Lemma C.3 the empirical distribution 6. we have
1
tr (Varr (u- + +111E(uu, (u, (u, (u [u+ E E(u,v)~
[u [T] + Fu 1u
2
w~ 11/0
E ((
(u [uuv)
n
n
m
17

---

On the Similarities of Embeddings in Contrastive Learning
Theorem C.5. Assume that the encoder (() satisfies [I](2))] 1 for all For any empirical distribution with a sample size of n, the following inequality holds.
1 where equality holds if and only if tr (1boo lu vl) and Bu /ee] + Bowsie [o] O.
E B(u,o)-ffPoo [u uu] s1+ [E(u,)) [u [u'e] n n
Proof. From Lemma C.4, and variance and norm non non-negative, have
E ftBpos [u tu]]]=
n + fe)Brnn [u vl
2(n (nn-) 1) tr (e) ju u|) 2(n 1) | (1 [u + uil12
n + fiu [u [uTol,
(17)
where equality in (17) holds if and only if tr (Va lu -2) 0 and ((,,w)> [u u| 0. Moreover,
the condition of [)) u 2) 2 0 is equal to E Fsb (u)) + Evp sap, [w]] = O. result, the following holds:
E ([(,,)) [u [u]e] + 1 - [u [u]]]= I++
E ((u,))
C.3. Proofs for Full-Batch CL
Lemma C.6 (Restatement of Lemma in Lee et al. (2024)), Let U1 U1 U2 U2 ... Uh, Un be 2n vectors, satisfying u u, = u. 0; for all E [n]. Then, the following inequality holds.
1
Σ
u
n - 2n(n 1)
Σ
u. 8;
n
(18)
n(n 1)
2(n 1)
se|m JEIm/((()
sE(m)
where the equality conditions are
u: cor all i E [n], for some constant vector c, Σ Eienn] u. + Eie(n) vi = O.
Proof. By using Jensen's 's inequality, we have
u
u:
01:)
72
1 xE
EE|n]
(n]
2
v.))
Σ
ui
Σ
71
72
iE(n)
iE(m)
se
[m]
> -4
Σ ui
Σ vi
ig|n]
where the equality conditions are
u; V; c for all i [n], for some constant vector c,
(19)
Σ Eienn U4 - Σ iE(m) U, O.
18

---

On the Similarities of Embeddings in Contrastive Learning
From the above, it follows that
Σ Ui n kE(n)
Σ Vi (m(
2
Σ
|| Uu - v:l2
(20)
4n
EE
1
+ 2n
Σ
u1
Us:
(21)
2
iE(m)
where the last equality uses ||u, Note that the inner product of centroids is
2uF on for all i E [n].
Σ us sE(m]
- n
Σ vE(n)
Σ
Σ
Σ
u 0. Uj.
vi
=
u
U.
n
n²
cE(m)
jE|](()
Combining this with (21), we have
E
1
Σ Ui
1 Σ Us
Σ
u V;
11
REIN] SE
VE ml]
sEim]
sE(n)
1
Σ
Σ
+
U
U2
U
Us
2
2n 3E|n]
n2
KE(N)
n 2
1
21
11
2n
2
<E(M)
The inequality follows from (20), with equality achieved under the conditions specified in (19).
Lemma C.7. Let u1 U1 U2 U2 .. Un Un be 2n vectors, satisfying u. u; = u. 0; = for all i E [n]. Then, the following inequality holds.
Σ
(u. Uj o0 0./000 Uj + 2u v2) 2 n(nn 1)
2
Σ
u 0;
2
(22)
nnn 1)
1
iE(n]
where equality holds if and only if Eieln) (u, + +v,) = O.
Proof. Note that
2
Σ kE(x)]
(u
+
0z)
Σ ui
Σ ui
+
Σ 1
Σ Di
+2
Σ ui
Σ Vi
sE/ml
iE ml]
FE|m
/E(x]
iE/ml
ic/n
2
Σ
Σ
u ú
Σ
Σ
2
Σ
us T Us + 2
Σ
Σ
u. Vj
(e(m) jE [E]]
VE[N]
((
VE(x)
2m + Σ
( Uj + 0. U3 + 24 03) + Σ
41 U2.
iE
m
(E
ic(m)
Rearranging terms, we have
1
1
2
(n. u, + v. u, + 2u, v;) = n(n -
Σ 12 + 12)
u
n(n
n(n 1) iE(n)
1
(E(n] [33)
|E|m|
2
Σ
u U.
n(n - 1)
n
iE|m]
where the equality condition is (u + +1) = O.
19

---

On the Similarities of Embeddings in Contrastive Learning
Lemma C.8. Let u1, U1, U2, V2, .. Uy Un be 2n vectors, satisfying u, u, 0; v; for all i [n]. Then, the following inequality holds.
2
(u u + 0. 0₁) 2
(23)
n(n iEin) jE(n]
12 -
where equality holds if and only Lig[n] u, Cie(m] U1 0.
Proof. Since u, u, = v.a = 1 for all i E [n], we have
(u, uj + uz 0z) n(n 1) 1)
Σ
u; Uj
Σ
U. U + Σ
Σ
2 Uj
Σ
vz
n(n A JE (0)
JE(N]
(E(n)
iE(m) jE(n]
kE(m)
2
Σ Ut
Σ fj jE[n
+
n(n - 1)
kE[m)
2
71 1
where the equality condition is Σie(n) u, LiE[m] v.=0.
0.
Theorem C.9. Suppose that d 2 n - 1. Let the contrastive loss L (U, V) be one of the following forms.
Linfo-sym (U, V) in Def. 3.1.
i. Lind-add (U, V) in Def. 3.2, where (c1, {(0,, (1, (1)
üi. Lind-add (U, V) in Def. 3.2, where (ci, (2) (1,0) and d' (1) 2 v (-+-T)
Then, the embedding similarities for the full-batch optimal encoder I in (1) satisfy
=(f";Ppod) = s(f"; :(og)
-
Theorem C.10. Suppose that d n. Let the contrastive loss (U, V) be the form of Lind-ada (U, () in Def. 3.2, where (cr, (2) = (1, and ø' (1) I
2-2
vb' (-x-T)- Then, embedding similarities for the full-batch optimal encoder f in
(I) satisfy
s(f*: Ppos) < 1, s(ft: Pneg) <
Proof of Theorem. C.9 and Theorem. C.I0. We prove for each category of loss, Linfo-sym (U,V) in Def. 3.1 and Lind-add (U, V) in Def. 3.2, separately.
First, consider the case of Linfo-ymm (U, V) in Def. 3.1. By using Jensen's inequality, we have
Linto (U,V)
Σ
GI
Σ
((v, - vi) Tu)T), + Σ @ ((u; - 0,) Tu.) ui)
n2
iE|n]
jE(m]\(A
jE(m)\(}
V
(cI (2) - (2)+c2(n-1)) 11
Σ
(c( - 0;) Tu, + c2(u, -o;) Ti)Jui)) Tu:)
12
EE|m]
JE(n]Mt
(24)
1
Σ
(c1 + (2)n 11).6
Σ
(cru) u, + +cuujuu u: -aa + c2)o, ui) )
[ci c2)(n (2) jE(n][(
72
1E|m]
20

---

On the Similarities of Embeddings in Contrastive Learning
where equality in (24) holds if the argument of is constant for all j [n] {i).
Let define (2) (z) (((c + (n) [))(2)) which is convex and increasing function. Then, we have
Linto-sym (U.V) = 7 Cinto (U.V) VV+++ + 2
Cinto (V.U)
2
1
Σ
(a)+c2)(-1)) -
Σ
(cro) uu czujur us - (c1 02)10 us)
2n
(a (2) jE(n]([8)
iE|ml
Σ vb
(2) 1)
Σ
(cru 2i + en0o vs - (a1 c2) 2i.)
2n
(cl (2)
kE[n]
2 v (c1 + cz) - 1)
Σ
Σ
(crw u + - (ci + c2)0, us)
2n
(a) + (2)
kc(n)
jE(n]\(}
Σ
Σ
(eu7 vs + oz0, vs - (a + o2)u,
2n
(al + c2) - 1)
iE|n]
jEIN/
(25)
= oz) 2(e1 + c2)m((n 1)
Σ
Σ
(a.23J + c2( u; + v.Ju)) v.) 2(c1 + o2)uu v.)
where the inequality in (25) holds for Jensen's 's inequality. Equality in (25) holds if the arguments of h(c) constant for all i E [m]. Moreover, from Jensen's 's inequality, we have
E u.. se (info-sym (U,V)) V)I
(q2u) u, + c2(u] u, + vuJv)) - 2(c + t2)2 vi) )
2 E (u.Vaaapppll
Marcos)
Σ
Σ
2(c + c2)n(n - 1)
(e(n) je(m)((()
(c12u] v: + c2(u] u, + 0.) 2(cr +o2) vi)) )
2 (x)) 2(c1 + o2)
U. W.
nin-
Σ
Σ
KE[N] [E
Therefore, only have to minimize
Σ
Σ
(c)2 J vs + cz u. + v.) -((1 + o2)u wi)
Sealull
n - 1)
Now, consider each case of (21, E {(0, 1), (1, 0). (1,
For the case of (cI, c2) 11, by using Lemma C.7, have
(.l n(n
Σ
Σ
(2u) + uu u; + v. wz
Au wi)
iE(m] jE(n]Mo
2 E (.V)~-af
2
u 02
4
Σ
T
U;
n(n 1)
m
sE[n
vE(n]
For the case of (c1, (2) = (0, 1), by using Lemma C.8, we have
Σ
Σ
(u] us + 0 v2 2u, [o:)
2 F
2
2
Σ
4.
0/
n(n 1)
71
n
sE(n] jE(
vE(n]
21

---

On the Similarities of Embeddings in Contrastive Learning
For the case of (2) = (1, by using Lemma C.6, have
(2u3 - 2u, vi)
n(n
iE(m) jC(m])(()
2 B (1
n - 2
12
Σ velni jE(m]M{
2n(n 1) iE|ml
2(n 1)
n(n - 1)
2
n1 + 122 +1
n(n +
E
Σ [EIM)
ui Vi
2n(n - -1)
Therefore, for every cases (C1, C2), u4 v2 = 1 holds for all positive pairs (ui, 0z) ~ Ig ipos and i E [n]] To achieve the all equality conditions, u v must hold for all negative pairs (u, v) fü Pueg. Therefore, embedding similarities for the full-batch optimal encoder f* in (1) satisfy
:(f":Ppus) Ppos) 1,
s(f:
Second, consider the case of Lind-add (U, V) in Def. 3.2. From Jensen's inequality, we have
Lind-add (U.V)
Σ
pu vi) +
C)
Σ
(u 0z) + 2n(n - 1)
02
Σ
(W(uT usy) + (0, vj))
n(n - 1)
iE(n]
iøjE/nl
itjEiul
U: Yis
+ n(n 1)
C1
(bu 03) + 2n(n - 1)
C2
(w(u, uz) + (1VT
π1
iE(n)
i*jE(n]
itjE(n)
(2cru, Uj czu uj + c20. v,))
u v;
+ v
71
(2c1 02 e 1)
iEIn]
sjE|n]
Equality conditions for both inequality is e and va are applied to constant argument. Now, consider each case of (21 E {(0, 1), (1, 0), (1, For the case of (c1, 02) = 11, by using Lemma C.7, we have
Lind-ada (ind-adaUU,V) (U,V) -
Σ VE[n)
u
+ 1b
(2u, v, + u, u, + v. 0,)
3n(n - 1)
-
u T
+
2
u T
2
3n(n
1)
3(n
iE(m)
iE(n]
Note that 6 and z are increasing functions. Therefore, by using the similar manner in the proof of Linfo-sym (U, V) case above, embedding similarities in Def. 4.1 of the full-batch optimal encoder I in (1) satisfy
s ppoo) =
s(I::
1-1
For the case of (c1, 02) = (0, 1), by using Lemma C.8, we have
Lind-add (U, V) 2
-
1
u
v;
+
1
Σ
(u, us + vj 0,))
11
ninn
1)
ml
iøje[ml
Φ
1
Σ
u Di
+ 1/
2
n
n
3E(m]
22

---

On the Similarities of Embeddings in Contrastive Learning
Note that &i and increasing functions. Therefore, by using the similar manner in the case of Linfo-sym (U, V) in Def. 3.1, embedding similarities in Def. 4.1 of the full-batch optimal encoder f in satisfy
s sff Ppos) L,
:(eg))))))
n
For the case of (c1.02) (1, by using Lemma C.6, have
Lid (U.)) 2 -
U
U;
+ vb
u Dj
n(n 1)
[E(n]
ifjE[n]
2 -
Σ
+ 3
12 2
u
n
2n(n - 1) 1=1
2 1)
KE(M)
h
Σ
u Us
=c(n)
where the function i(-) is defined as
==))==-(04)) + n 2
72
T
2(n 1)
2(n
Note that both --(-) and (-) are differentiable and convex functions, therefore h(-) is also differentiable and convex function. Therefore, attain the minimum at L. (1) 0 holds. Then,
0 h(11= (1)
n - 2 n - 2
n- 2 2nn - 1) ) - (1) -0)(11
n
w/
1
2(n - 2(n - 1)
2(n 1) n 1
which is equal to
n- 2 2(n - 1/ (-
Φ' (1)
>
Therefore, by using the similar manner in the case of Linfo-sym (U, V) in Def. 3.1, embedding similarities in Def. 4.1 of the full-batch optimal encoder f in (1) satisfy
s(f": = 1,
:(ff:
if Φ' (1) -ーーー
3/
On the other hand, if (1) <
n rb
embedding similarities in Def. 4.1 of
the bull-batch optimal encoder f in (1) satisfy
< 1,
s(f" Pueg) <
Moreover, the existence of the embedding for d 2 n can be shown in Proposition in Lee et al. (2024) Example C.11. Consider the sigmoid contrastive loss Levg. (U, V) (Zhai et al., 2023), defined as
Laig (U,V):= Σ log (1 exp (-tuy v.) exp(b)) Σ log (1+exp (tu, ((u]),)) -ex((bb)).
iEim]
where and E R are hyperparamettrs. This loss follows the form of Lind-ada (U, V) in Def. 3.2, where (2) (1, (), d(x) log(1 + exp(-tx + b)), and (x)) = (n - 1). log(1 + exp(tx b)).
23

---

On the Similarities of Embeddings in Contrastive Learning
hyperparameters and b are chosen such that
1 1+ exp n-1 +b)
71 - 2 1 exp(1 b) < 2
embedding similarities of the full-batch optimal encoder S in (1) satisfy
Ppos)
s(f":
Proof. The sigmoid contrastive loss Ssg (U, V) follows the loss form in Def. 3.2, see Appendix A.2. Consequently, by Theorem C.10, it suffices to verify the condition of
Ø' (1) < 11 n
(26)
2(n 100-1 1)
where d(x) log(1 exp(-tx + b)) and (xx) (n 1) + exp(tr 6)). Taking the derivative of 6(x). obtain
texp( -tx
d'(x) 1 expp to + b) 1 1 exp - b)
and differentiating (x) yields
Lexp(1x - b)
(x()) (n 1).
exp(tx b)
By plugging the derivative values into (26), we get
(n 1)xexp
(---1 b)
- 2
+ exp(t - b) 2(n - 1)
1 + exp
exxp
(- - b)
which simplifies to
(n - 1)exp (-n---1 -b)
11 - 2
11 1
+ exp(t b)
2(n - 1)
1 + exp
-- -11 b)
1
2
exp
n-1 + b
1
+
Therefore, if hyperparameters and b satisfy
1 + exp
1 +b)
n1 1
(27)
+ exp(t
2
following from Theorem 10 the similarities of the full-batch optimal encoder f* in (1) satisfy
s(f*: Pposs < 1,
:(f=)
C.4. Proofs for Mini-Batch CL
Definition C.12 (Simplex ETF), A set of vectors U on the d-dimensional unit sphere is called (n 1) simplex ETF, if
u 1 = 1 and u T
Vu NUFEEU
1'
Note that (n- 1)-sippeex ETF exists when
24

---

On the Similarities of Embeddings in Contrastive Learning
Lemma C.13. Let vectors U be (n s)-sipleex ETF on the d-dimensional unit sphere with 2 Then, the following holds:
Σ WEU
u 2=0
Proof. By the definition of a (n 1)-simplex ETF, each vector U satisfies ||u|| = 1 and the pairwise inner product n- Then,
for any u, v E U with u + v is u
Σu
Σ uEU
u
Σ AEU
Σ EEU
u u
Σ uANEU
u u'v= n n(n 1) (1-)--11) 11-1
0.
MEU
Since the squared norm is zero, we conclude Euey 4
Lemma C.14. Let a set of n vectors U be a (n - 1)-simplex ETF, and a set of m vectors V be a (m - 1)-simllex ETF where all vectors are on the d-dimensional unit sphere with max(n, m) Then, the following holds:
u uv== u 1
Σ
UUV
m-+111
UFFEULV
Here, UUV V denotes the concatenation of the two sets, representing the full collection of all n + m vectors from and V.
Proof. From Def C.12, we have
1
Σ
1
u T e
u
1
m(m 1)
uUEUU
uVVEV
Moreover, Lemma C.13 implies that
Σ u mEU
Σ 1 = 0 VEV
WEU DEV
The total sum of pairwise inner products for the combined set U U V can be written as
Σ u To= Σ
Σ
TT
UEDEUUV
WEULV
(M))
Σ
Σ
u VU Σ
Σ u Tv
WEU DEUVV (u)
WeV vEULV
Σ
Σ
Σ
Σ
u T U
Σ
Jut Σ
Σ T.
W U
uEU (E)
uEU VEV
WeV DEU
W (u))
Σ uT uTo
Σ
u +
Σ
u
u TTv v
UFEEU
WEV
VEV
VAVVV
Σ uAVEU
uTo +
Σ
u
(28)
V
1
nín - 1)
+ +m(m= 1))
m,
where the equality in (28) holds because the total pairwise inner product sums within U and within V are zero.
Therefore, we obtain
Σ
(n mn m (-n m)
1
U
n +
UVVEUUV
25

---

On the Similarities of Embeddings in Contrastive Learning
Lemma C.15. Suppose d 2 n 1. Let V and W be d matrices whose columns form (n 1)-impleex ETF on the d-dimensional unit sphere. Then, there exist orthogonal matrix P E Raxd such that V PW.
Proof. From the definition of simplex ETF in Def. C.12, the Gram matrices satisfy
vv== W
where each diagonal entry and each off-diagonal entry -n-T. Then, from Theorem 7.3.11 in Horn & Johnson (2012) there exist an orthogonal matrix P E Rdxd such that V = PW.
Theorem C.16. Suppose d 2 m - 1. Let the contrastive loss L (U, V) be one of the forms in Theorem 5.1. Define fatch as the optimal encoder that minimizes the fixed mini-batch loss, given by
Batch = arg min E
L(U Vin)
AE
where Is ==:= [m(k - +1 mk) for k E [b].
Then, embedding similarities for the mini-batch optimal encoder fhanch satisfy
s ffbarchi Ppos) 1,
1
E [ ('aahhh DroR)]
Var [Sacchh Preg)) E
n m
(n m7)
(29)
(m
- () 1)2' (m - 1)(n - 1)2
necessary condition for attaining the minimum variance of negative-pats similarities in (11) is d 2 b(m 1).
Proof. Note that
E U Saptaa
Σ C(U VL.)
Σ E (..) [C(U,, V1,), kE[b]
KEW
by applying Theorem 5.1 to each batch, m random vectors each batch are degenerated to construct the (m 1) simsixplex ETFin in Def. C.12. Therefore, for E [b]. we have:
u e = 1
v(u, v) ~ f-
u v
V(u, v) ~ Sapiiig
(30)
m
In what follows, for all positive pairs, we have
w 2
vuu, v) ~ f Ppos:
which is equal to
8 8fbatchi Ppox) =
For & [6], let (k) and (k) denote d m random matrices, where the columns represent the vectors in the k-th batch. Additionally, define U and V x mb random matrices formed by concatenation of all corresponding batch matrices, i.e., U : U U12) ( and V = [1), ( .... ( (b)).
Let W be a d x matrix whose columns form (m - 1)-simplex ETF in Def. C. From Lemma C.15, for E [b], there exist orthogonal matrices ((k) E Rdxd such that (k) (()) W. Moreover, based on the singular value decomposition, let W = WWW,, where W1 is a d x orthogonal matrix, W2 is an m x m orthogonal matrix, and Σ is a m rectangular diagonal matrix with non-negative values of σ1, T2, " .*. on on the diagonal.
26

---

On the Similarities of Embeddings in Contrastive Learning
For all & [6]. we have
((vr)) T
v=00l2=110 -I((r)) vrool = - =m- m(m -1) (-
m
1).
171
For all h1 # h2 E [b], we have
((rtkr) v(ke))
= (vidi)) T v(ke) -WT
WT (plk.)) p(ki 2
0,
where the minimum value of zero is achieved if (P(R))) T p(ke) is the d x consisting entirely of zero elements. On the other hand,
(orkhi) v(laa 2 = (Vihii) vrossl
WT (plbi)) T p(kt) W W2EwT (pin))
2 EWT (p(t.) plk) EPT P.E
where P1 p(h)) Wi and :2:= = p(ks)W W are x m orthogonal matrices, and each Pi; and Ps is column vector of Pi and P2, respectively. Since P1 and P3 are orthogonal matrices, their columns orthonormal vectors, respectively. Then,
(Dilki) ((k2) = EP P
m
Σ PP 2 oo==
ΣΣ 2
W
W
m 1
im(m)
iE(m)
where the maximum value of m attained Pı; Par for all E [m], i.e. P1 P2. From Lemma C.14, the expectation of the negative-pair similarity is
E [sfoaach Pueg)] () Boeweh pung [u" T vl
Note that
[Bs[fatthh (neg))
E () fharch frnn
[(uT))²
0) []
(1UT VI - -
n
1
(Brlkt) v (Az) 2
1
2
n1
((rrkt) T
(k2))
Σ ke(
(Ur))) T
v(k)
+
12.
k1 of Eb
F
n²
n-1
27

---

On the Similarities of Embeddings in Contrastive Learning
(orth))
((22)
+
1
b
m²
1
n n²
F
n1
11
m
n
1
((rrki) ((2)
+
122
(m
1) 1)
ky A
F
m²
E 0 + (m 1) - 1)² n22
b(b - 1).
+
1
(m 1)(n 1)
where the range is equal Therefore, the variance of the negative-pair similarity is
(m- Im-In[n-I) m(b-1)+t (m-1)(n-11)
Var (fhatch Pueg]] = EE s (fffbacch Pueg)
E ifhatch Peeg)) )
E s (fbacchi Pueg)
(n 1)2
n (m (mn-))(n- 1)2 (m - 1) - 1)2
17 (n ml)
Lemma C.17. Let my and my be natural numbers with my S m2. For any positive values {cr > 0) the following inequality holds:
m1 Eie(ma) Ci
$ 1,
m2 Eie(ma) Ci
where the equality condition is mi m2-
Proof.
Since m2
0 and m2
1771 2 0, we have
m2
C4 m
= m2
Σ
+ + (m2 mm)
Σ
G 2 0,
iE [m2]
EE m1)
ie [e(m=]\[mm]
AC [mi]
where the equality condition is my = m2, Note that my Eie(mi) Cs > 0. Rearranging the above yields
m. Σ Exelma) Ci
< 1.
m2 Evelma) Ci
Theorem C.18. Consider the InfoNCE loss CIaONNCE (U, V) (Oord et al., 2018), which corresponds to the loss Linfo-sym (U, V) in Def. 3.1 where (bx)) = exp(x/t) for some >> 16(x) log(1 + x), and (c1, (1,0).
For any two integers m m, E [n] such that my S m2, the gradient of the InfoNCE loss with respect a negative-pair similarity satisfies the following inequalities for any distinct indices i # [m].
[-
a (u. v,) LIntANCEUU (U, v3
i a (u. 1z)
((U.V))-
LinGONCE (U, V)
5.
Moreover, the equality condition of the first inequality is m22-
Proof. Without loss of generality, suppose there are positive pairs in (U, V). Then, the InfoNCE loss is given follows.
C(U, V) Σ log 1+
exp(u, (v, - u.)/t)
Σ log mm(m]
1+
exp((u; u.) (o/ott))))))))))))) )
mm[m]
jE(m]](A
28

---

On the Similarities of Embeddings in Contrastive Learning
Following the gradients analysis in Wang & Liu (2021), the partial derivatives of the loss with respect to negative pair derived as follows. In particular, for all # [mm) have
a d u 03)
C(U, V)
1 a m
log 1+
Σ
exp(u (uy ub)/t)
a (u v3))
J'EmIMM
+
log 1+
Σ exp((ur - " ws/t)
a u U3)
exp(u (u, vz)lt)/6
1
exp((u; uz) [o,])//()/
m
(e/m))(()) exp(u, - w:)/t) m 1+E PEmml((s) exp((ur - [t,/t// exp(u; Uj/L) E Eye(m] exp(1 u, 0y,/t) mt Eve(m) exp exp(u,
1
1 explu, w;/l)
mt
1
exp(u, /
exp(u, Uz/t)
ml
EjE(m) exp(24 u, w,/t)
Eve(m) exp D1/t)
0.
Then, for any S m2 and + j E [mm]]. the following inequality holds:
1
exp(u T Dj/t)
exp(u,,"u) Uj
C(U, v)
0 Ouu V3))
m2l
Eme(me) exp(2 uu 0,/t)
Ee(mm)) expl uu vs/t)
exp(11, _ Uÿ/()
mi jElmil exp(u, u₁
(UUV))-fi ma
mnt
Σ Se(mm) exp(11 (ue 0z/l)
m2
Eje(ma] exp(u 1;
exp(u, U₁
mi Eve(mi) exp (2 v2/t)
+
mnt
E
[U Si
Evelmal exp(us, wz/t)
m2 Evelmal exp(u,. w:/t)
exp(u, u;/t)
1
mil V.
Σ JElm.) exp(1 u u₁/t)
exp(u, T v;
+
E UUVV
(31)
mnt
Σ Cre(ma) exp(u, v1/t)
1 mnt
Σ Eje(mi) exp(11 /z/l) + exp(u,To,/ 0//t)
exp(u, vz/t)
(32)
Eve(mi) (spp(u,///1)
((U.v)
EU
1 (u. vz))
where the inequality in (31) follows from Lemma C.17, since the exponential function strictly positive. The equality condition in (31) is m1 m2.
Moreover, equality in (32), where fappa is replaced with ftppu holds because the expectation involves only embeddings (U, V) with indices in [m,]. This concludes the proof.
29

---

On the Similarities of Embeddings in Contrastive Learning
D. Experiment Details
In this section, we provide the details the experiment setup mentioned in See. 6. Our implementation based on the open-source library solo-lear (da Costa al., 2022) for superviie learning. The source code is available at https: : / ////tttttet github. com/ lee Leechungpa/ wwwpuaapaabbeeediinggggeeee
D.1. Architecture and Training Details
For all experiments, we use modified R8 (He al., 2016; Chen al., 2020) as the backbone for CIFAR datasets and ResNet-50 for ImageNet-100 100. For CIFAR datasets, we modify ResNet-18 by replacing the first convolutional layer with 3x3 kernel stride of and removing the initial max pooling step. In contrast, architecture for ImageNet- 100 without any modifications. Regardless of the backbone used, we attach 2-layer MLP the projection head, which projects representations to 128-dimensional latent space. Batch normalization is applied to the fully connected layers, with the hidden layer dimension set to 2048 for ImageNet- 100 and 512 for the CIFAR datasets.
the standard ResNet-50
We follow the data augmentation strategy used in SimCLR (Chen et al., 2020). Specifically, we apply random resized cropping, horizontal flipping, color jittering, and Gaussian blurring. For the CIFAR datasets, the crop size is set 32, while for ImageNet-100, we use crop size of 224. These augmentations are applied consistently across all experiments.
For the optimizer, we stochastic gradient descent (SGD) for 200 epochs. The learning rate is scaled linearly with the batch size as Ir x BatchSize/255 where the base learning rate is set to 0.3 for the CIFAR datasets and 0. for ImageNet- A cosine decay schedule is applied, with a weight decay of 0.0001 and SGD momentum set to 0.9. Additionally, we use linear warmup for the first 10 epochs.
We tune the temperature parameter for baseline methods, SimCLR, DCL, and DHEL,, by performing a grid search over the range of 0.1 to 0.5 in increments of 0.1 and selecting the temperature value that yielded the best performance for each method. For tuning the proposed loss LVRNS(U, V) in Def. 5.7, conducted a grid search for À from the set {0.1, 0.3, 3, 10, 30, 1003.
All experiments were conducted using single NVIDIA RTX 4090 GPU.
D.2. Evaluation Details
For the linear evaluation protocol, we remove the projector head and using the pretrained encoder for downstream classification tasks. Specifically, we extract the encoder outputs from the trained model without applying any augmentations. These feature vectors are then normalized and used to train a linear classifier. Following prior works (Komblith et al., 2019; Lee et al., 2021; Koromilas et al., 2024), we report top I accuracy on the downstream dataset. We use SGD, setting the learning rate to 0.1 without weight decay. The classifier is trained for 200 epochs with a batch size of 256.
E. Additional Experiments on the ImageNet Dataset
Table 4. Effect of our proposed loss (when combined with SimCLR) on the top-1 and top-5 classification accuracies (%). Bold entries indicate the highest accuracy.
Dataset
Temperature
SimCLR
SimCLR + Ours
Top 1
Top 5
Top
Top 5
t 0.1
70.14
91.14
69.50
90.80
[ 0.2
73.80
93.14
73.94
93.08
ImageNet- 100
t 0.3
72.30
92.94
74.04
93.24
f 0.4
69.92
92.10
73.12
92.98
t = 0.5
68.60
90.90
72.88
92.84
We further validate the effectiveness of the proposed loss through experiments on the ImageNet dataset. Table 4 reports the top-1 and t classification accuracies on ImageNet-100 for various temperature values. For each temperature, we compare SimCLR with and without the proposed loss. The results demonstrate that incorporating our loss generally improves
30

---

On the Similarities of Embeddings in Contrastive Learning
Table Top-1 accuracy (%) the full ImageNet dataset.
Method
Top-1 accuracy (%)
SimCLR
51.79
SimCLR Ours
52.48
performance across a range of temperatures, with the largest gain observed at 0.3.
We also report results from 100-epoch training on the full ImageNet dataset. Using the same experimental setup as for ImageNet- 100, compare SimCLR with method (SimCLR the proposed loss) using 0.2 and X 40. As shown in Table 5, our method outperforms the baseline.
F. Discussion on the Proposed Loss for Variance Reduction
The auxiliary loss (U,,s(U, V) proposed in Def. 5.7 shows effectiveness in the following scenarios:
Small Batch Training: As established in Theorem 5.5, the variance of negative- parr similarities increases as batch size decreases. The proposed loss in Def. 5.7 directly penalizes this variance, making it particularly ad advantageous when training with small batch sizes. This effect empirically validated in Figure 4.
Temperature Robustness: The performance of CL methods is often sensitive to the choice of the temperature parameter, which affects the distribution of similarities among embedding pairs (Wang & Liu, 2021). By explicitly encouraging negative-paii similarities towards the optimal value of -1/(n 1). the proposed loss reduces this sensitivity and stabilizes performance across a wide range of temperature settings, as shown in Figure 3.
Despite its advantages, the proposed loss also presents several limitations:
Suppression of Semantically Meaningful Variance: In some cases, variance negative-pait similarities may capture meaningful semantic differences between instances. Enforcing uniform similarity can potentially suppress this informative structure, adversely affecting representation quality.
Reduced Impact with Large Batch Sizes: The variance-reducing effect of proposed loss diminishes as batch size increases, since the variance negative-pair similarities naturally decreases in larger batches.
Hyperparameter Sensitivity: The proposed loss introduces an additional hyperparameter, A, which necessitates careful tuning to achieve optimal performance.
31


Reviews:
[{"review_id": "68JSFtYMnd", "reviewer": "Reviewer_BhMK", "summary": "The paper analyzes the distribution of positive and negative pairs in contrastive learning and shows that perfect alignment becomes impossible when expecting negative pairs to fall below the optimal threshold. The paper also proposes variance reduction for negative-pair similarity loss to reduce the variance of negative pairs when using a small batch size. Furthermore, they experiment on different datasets such as CIFAR-100 and ImageNet-100 to demonstrate that this addition can improve contrastive learning methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive feedback, including that our paper is **well-written**, provides **a clear explanation of results**, and may be **useful for future CL research**. Below, we address each of the reviewer’s comments in detail.\n\n---\n### W1. When VRN loss term downgrades the performance in Fig.2?\n\nFig. 2 in our paper has been updated (see below) with improved hyperparameter tuning:\n\n[`Fig.1` Effectiveness of VRN.](https://osf.io/mejs6?view_only=066d766d57914710810f46ab5f849bf9)\n\nIn earlier results, VRN degraded performance on CIFAR-100 (e.g., DCL, DHEL), likely due to a fixed weight $\\lambda=30$. After tuning over a wider range of {0.1, 0.3, 1, 3, 10, 30, 100}, we observe consistent improvements across all methods.\n\nWe also note:\n* VRN operates on cross-view negatives,\n* SimCLR includes such pairs, DHEL does not,\n* and gains from VRN are larger when the base loss includes cross-view negatives (e.g., SimCLR).\n\nThis suggests VRN is most effective when complementing losses that already involve cross-view negatives.\n\n---\n### W2. Limitations on using VRN and scenarios where it is benefitial to use it.\n\nWe summarize below the scenarios where VRN is most effective, followed by its main limitations.\n\n1. When VRN is beneficial\n* Small batch training:\nFrom Theorem 5.5, small batches lead to higher variance in negative-pair similarities. Since VRN minimizes this variance, it yields stronger gains in low-batch regimes. Empirical results in `Fig.1` support this.\n* Robustness to temperature:\nContrastive loss is sensitive to the temperature parameter, which affects embedding similarity distributions `[R1]`. VRN encourages negative similarities toward the optimal $-1/(n-1)$, reducing performance fluctuation.\nBelow, we show SimCLR with VRN ($\\lambda=30$) yields more stable accuracy across temperature values on CIFAR-10/100:\n\n[`Fig.2` Robustness to temperature.](https://osf.io/cp3yn?view_only=066d766d57914710810f46ab5f849bf9)\n\n`[R1]` *Wang, et al. Understanding the behaviour of contrastive loss. CVPR2021.*\n\n2. Limitations of VRN\n* Loss of meaningful structure:\nVariance in negative similarities can reflect semantic diversity. Forcing uniform similarity may suppress this, as discussed in Sec. 4 (L233–245) and Sec. 7 (L433–436).\n* Diminished effect with large batches:\nAs batch size increases, the natural variance stabilizes, reducing the marginal benefit of VRN.\n* Hyperparameter tuning:\nVRN introduces an additional weight $\\lambda$, which requires tuning per setting.\n\n---\n### E1. How many negative pairs fall above the optimal threshold?\n\nWe evaluated the ratio of negative pairs whose cosine similarity falls below the theoretical threshold of $-1/(n-1)$, using models pretrained with SimCLR (ResNet-18, CIFAR-100, $t=0.2$).\n\nWe generated 5,000 negative pairs from each model by applying random augmentations and measuring cosine similarity from the projector output. The table below summarizes the results:\n\n|Batch size|Ratio below threshold (%)| Var. of negative similarity | Top-1 acc. (%)|\n|:-:|:-:|:-:|:-:|\n|32|58.09|0.1649|56.34|\n|64|58.10|0.1505|58.40|\n|128|57.79|0.1444|58.80|\n|256|57.61|0.1404|59.72|\n|512|57.38|0.1396|59.69|\n\n`Table 1` *Ratio of excessively separated negative pairs and associated statistics.*\n\nWe find that over 57% of negative pairs fall below the optimal similarity across all batch sizes. Smaller batches show greater variance and slightly lower accuracy, aligning with Theorem 5.5, which predicts increased variance in negative similarity with reduced batch size.\n\n---\n### Q1. How many of negative pairs does not satisfy the optimal thershold if a method only uses negative pairs into the loss function?\n\nWe interpret your question in two possible ways:  \n(1) How many negative pairs fall below the optimal similarity threshold when training only with the VRN loss, or  \n(2) when using a contrastive loss with only cross-view negatives.\n\nRegarding (1), the VRN loss is not intended to serve as a standalone training objective. It is designed to regularize standard contrastive losses, and cannot learn meaningful representations on its own.\n\nRegarding (2), we report statistics from models trained with sigmoid loss (in Example 5.4), which uses only cross-view negatives. In this case, a large fraction of negative pairs have similarities below the threshold of $-\\frac{1}{n-1}$, where $n$ is the sample size. This indicates over-separation. When the loss is modified to include both cross-view and within-view negatives, this issue is mitigated, as shown below:\n\n\n|Negative-pair type|Ratio below threshold (%)|\n|:-:|:-:|\n|cross-view|69.30|\n|cross-view & within-view|66.88|\n\n`Table 2` *Negative pair similarity in models pretrained with sigmoid loss.*\n\nPlease let us know if this interpretation differs from your intent — we would be glad to clarify further.\n\n---\n### Q2. Is there any toy experiments?\n\nPlease see `E1` in our response to Reviewer qfkv.", "claims_and_evidence": "The papers claims that", "methods_and_evaluation_criteria": "The paper provides comparison on CIFAR-10, CIFAR-100 and ImageNet-100 datasets using 4 methods. The numerical results show that using VRN increase the accuraccy in most cases.", "theoretical_claims": "I checked proofs for Proposition B.1 and B.2. and they were correct.", "experimental_designs_or_analyses": "All the experimental section. I am wondering how many negative pairs falles above the optimal threshold in the experimnts.", "relation_to_broader_scientific_literature": "The paper showed that pervect alignmet becomes unreachable if we expect all the negative pairs to fall bellow the optimal threshold. Instead, considering the negative pairs in the loss function can be more benefitioal. These findings are useful in the future direction research in Contrastive Learning.", "essential_references_not_discussed": "The papers mentiones related works adequantly.", "other_strengths_and_weaknesses": "**Strengths**:\n\n1.  The paper is well-written and the results are clearly presented.\n\n**Weaknesses**:\n1. The authors did not discussed in which scenarios adding VRN loss could downgrade the performance. For example in Figure 2, I observe that combining VRN with DCL results decrease (Downwards arrows) in most cases.\n\n2. I expect more investigation on the limitations on using VRN and scenarios where it is benefitial to use it.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "I have the following quesitons from the authors:\n\n1. Could authors provide that how many of negative pairs does not satisfy the optimal thershold if a method only uses negative pairs into the loss function?\n\n2. Is there any toy experiments that we can observe the effect of using VRN loss? (Probably something similar to Figure 1 in [1])\n\n\n[1] Liu et al., Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap, ICLR 2023", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SJcPGQlxcR", "reviewer": "Reviewer_qfkv", "summary": "This paper mathematically analyzes the geometric properties of the positive pairs’ embeddings as well as negative pairs’ embeddings in different contrastive learning objectives. The authors mathematically find the optimal threshold for the expected negative pair similarities that results in preventing a misalignment of positive pairs in the embedding space. \n\nThey further provide an analysis of how the variance of similarity of the negative pairs can also affect the misalignment of positive samples. This variance is especially evident in the mini-batch setting. \n\nUsing the optimal threshold of the similarities of negative samples, the authors propose a comprehensive variance reduction loss function, namely VRN, for the negative pairs that can be added to any existing contrastive loss function. They show its effectiveness in classification tasks using various contrastive learning loss functions on the CIFAR and ImageNet datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that our mathematical analysis **provides important insights for contrastive learning in both unimodal and multimodal settings**. We are also encouraged by the positive evaluation of our experimental claims and the effectiveness of our proposed VRN approach on classification tasks. Following the reviewer’s constructive suggestions to strengthen our paper, we provide additional experiments as follows.\n\n---\n### E1. More in-depth analysis of embeddings and the structure of the embedding space after the proposed VRN approach.\n\nTo analyze the effect of the proposed VRN loss on the embedding space, we conduct experiments on both synthetic and real-world datasets.\n\n**[Experiment on Synthetic Data]** \n\nWe use a synthetic dataset of 4 samples, each augmented twice (8 embeddings in 3D). Embeddings are optimized directly using SGD (lr=0.5, 100 steps, mini-batch size=2). We compare two setups: (1) SimCLR with temperature $t=0.5$, and (2) SimCLR + VRN loss with $\\lambda=3$.\n\n[`Fig.3`. Visualization of learned embeddings.](https://osf.io/phte3?view_only=066d766d57914710810f46ab5f849bf9)\n\nAs shown in the figure above, both methods successfully align positive pairs. However, with SimCLR+VRN, the negative pairs are more evenly distributed in cosine similarity, centering around the theoretical optimum of $-1/3$. Specifically:\n\n- SimCLR: mean = -0.3201, std = 0.2051\n- SimCLR+VRN: mean = -0.3327, std = 0.0207\n\nGiven the absence of semantic structure in the synthetic samples, a uniform separation among negative embeddings is desirable. The VRN loss seems to facilitate such balanced distribution.\n\n**[Experiment on Real Data]**\n\nWe further assess how VRN affects negative pair distribution in realistic scenarios. Using ResNet encoders trained on CIFAR100 with either SimCLR or SimCLR+VRN, we sample 5,000 negative pairs from augmented training images and compute their cosine similarities.\n\nWe repeat this across different batch sizes and report the variance of negative-pair similarities:\n\n| Batch size | Variance of the negative-pair similarity (SimCLR) | Variance of the negative-pair similarity (SimCLR+Ours) |\n|:-:|:-:|:-:|\n|32 |0.1649|0.1008|\n|64 |0.1505|0.0952|\n|128|0.1444|0.0929|\n|256|0.1404|0.0921|\n|512|0.1396|0.0917|\n\n`Table 3` *The effect of VRN loss on the variance of negative-pair similarity, where embeddings are generated from models pretrained with different batch sizes.*\n\nWe observe that the variance of negative-pair similarity consistently decreases when VRN is used, across all batch sizes. We will include these experimental results and their discussion in the revised version of the manuscript.\n\n---\n### S1. The notation d in lines 242, 283 and 313 seem to not have been introduced in the paper.\n\nThank you for pointing this out. The notation $d$ refers to the dimension of embedding vectors, as mentioned in line 96 in Sec. 3 (Problem Setup). We will clarify this in the relevant lines.", "claims_and_evidence": "Yes, the claims seem convincing to me.", "methods_and_evaluation_criteria": "Yes, the evaluation makes sense. The authors also provide a link to the source code.", "theoretical_claims": "I did not attempt to check the proofs.", "experimental_designs_or_analyses": "Yes. The paper provides experiments on 3 classification datasets and examines them with regard to the classification accuracy. \n\nHowever, the paper could benefit from more in-depth analysis of embeddings and the structure of the embedding space after the proposed VRN approach. E.g., analysis of the distribution of cosine similarities of the positive and negative samples and PCA/DOSNES visualizations based on classes could be provided. These analysis would provide more insights on the geometric structure of the embedding space.\n\n**Update after author response:** The additional experiment analysing the embedding space seems helpful, but additional empirical analysis of typical real-world models, which are based on much larger training runs, would be a welcome addition, especially also with further insights regarding the effects on the semantics in the vector space.", "relation_to_broader_scientific_literature": "The findings appear to provide an important insight for contrastive learning in unimodal and multimodal settings. One of the pioneering works in this direction is [1], and since then, many efforts have been done to understand the alignment and separation of embeddings in the embedding space from a geometrical point of view and their results on downstream tasks.\n\n[1] Wang, Tongzhou, and Phillip Isola. \"Understanding contrastive representation learning through alignment and uniformity on the hypersphere.\" International conference on machine learning. PMLR, 2020.", "essential_references_not_discussed": "None I am aware of.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "The notation d in lines 242, 283 and 313 seem to not have been introduced in the paper.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WSuw223lQx", "reviewer": "Reviewer_Zp1i", "summary": "This paper investigates the geometry of embeddings learned by contrastive learning. This paper first extends the geometry of optimal embeddings (perfectly aligned positives and negatives with cosine similarity $-1/(n-1)$) to an inclusive form of contrastive loss. Then it proves that over-separated negatives with similarity less than $-1/(n-1)$ harm the perfect alignment of positives, and that in fixed mini-batch training, the same-batch negatives are over-separated, especially when the batch size is small. To address this problem, the authors propose a VRN loss term to regularize the similarity of negative pairs. They conduct experiments on benchmark datasets to validate the effect of the proposed VRN.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive review, especially the effort to understand and verify our theoretical results. We are glad the reviewer saw our analysis as a meaningful **extension of existing CL theory** and appreciated its **new perspective** on the role of negative pair similarity. Below, we provide detailed responses to your constructive comments.\n\n---\n### Q1 & C1-1. The difference between cross-view and within-view negatives in the single-modal scenario?\n\nThank you for the insightful question. The figure below illustrates the structural difference between cross-view and within-view negatives in the single-modal case. Each graph has six nodes for three instances, with edges denoting negative pairs: $(u_i, v_j)$ for cross-view, and $(u_i, u_j)$ for within-view, where $i\\neq j$.\n\n[`Fig.4` Illustration of cross-view and within-view negatives.](https://osf.io/ejbm3?view_only=066d766d57914710810f46ab5f849bf9)\n\nOne can confirm that cross-view graphs are **fully connected bipartite**, whereas within-view graphs consist of **disconnected subgraphs**. This topological difference highlights their non-equivalence, even in the unimodal setting.\n\nWe will clarify this distinction in the revision.\n\n---\n### Q2 & C2. Is excessive separation still a problem under the random mini-batch scenarios?\n\nYes, as confirmed by additional experiments. In `Table 1` of our response to Reviewer BhMK, excessive separation still emerges under random mini-batching and correlates with performance degradation when it occurs more frequently.\n\n---\n### Q3 & C3. The negative effect of imperfect alignment?\n\nImperfect alignment degrades representation quality. Using the sigmoid loss (in Example 5.4) with varying bias $b$, we observe in the figure below that lower positive similarities correlate with reduced top-1 accuracy in linear probing. This demonstrates the performance sensitivity to alignment quality.\n\n[`Fig.5` Negative effect of imperfect alignment.](https://osf.io/57knr?view_only=066d766d57914710810f46ab5f849bf9)\n\n---\n### Q4. VRN performance under additive contrastive loss and full ImageNet?\n\nWe evaluated VRN loss in the linear probing setting on CIFAR-10. Using Sigmoid loss alone ($t=1$, $b=-1$) yields 81.10% accuracy, while combining it with VRN loss ($\\lambda=30$) improves performance to 87.81%, under the same training setup as in our paper. This demonstrates a significant gain from incorporating VRN, under additive contrastive loss.\n\nWe applied VRN to SimCLR on full ImageNet, using the same setup as in our ImageNet-100 experiments. After 100-epoch training with $t=0.2$ and $\\lambda=40$, our method (SimCLR+VRN) achieved 52.48% top-1 accuracy, outperforming the SimCLR baseline (51.79%). This confirms the scalability of VRN to large-scale datasets.\n\n---\n### C1-2. The contribution, that within-view negative pairs can mitigate the excessive separation of negative pairs, is supported by neither specific theorems nor experiments.\n\nThe mitigation effect of within-view negatives is theoretically supported by Theorems 5.1 and 5.3 (see final paragraph of Sec. 5.1). Theorem 5.3, based solely on cross-view negatives $(c_1, c_2) = (1, 0)$, leads to excessive separation due to the independently additive form of the loss. In contrast, the loss in Theorem 5.1 includes both cross-view and within-view negatives $(c_1, c_2) = (1, 1)$, which alleviates this issue.\n\nTo empirically validate this, we evaluated the sigmoid loss (as in Example 5.4) with and without within-view negatives. As shown in `Table 2` of our response to Reviewer BhMK, adding within-view negatives reduces the proportion of excessively separated negative pairs, confirming the theoretical insight.\n\n---\n### C4. Accuracy drops after incorporating VRN?\n\nThe initial drop was due to using a fixed VRN weight ($\\lambda=30$) at submission time. After tuning $\\lambda$, we observed consistent performance gains across all methods and datasets. Updated results are provided in `Fig.1` of our response to Reviewer BhMK.\n\n---\n### S1. Are $(x,y)$ always positive pairs? Later parts seem to allow negative pairs as well.\n\nThere was a notational oversight in Sec. 3. The notation $(x,y)$ should represent either positive or negative pairs depending on the sampling. From Sec. 4, we clarify this with $(x,y)\\sim p_{pos}$ or $p_{neg}$. We’ll revise Sec. 3 to reflect this properly.\n\n---\n### S2. The notation of $Var$.\n\nThank you for pointing this out. The standard form is $Var[X]=\\mathbb{E}[(X-\\mathbb{E}[X])(X-\\mathbb{E}[X])^\\top]$, while we wrote $\\mathbb{E}[(X-\\mathbb{E}[X])^\\top(X-\\mathbb{E}[X])]$. We’ll revise the appendix to follow the standard or express it as an expectation.", "claims_and_evidence": "- In the first contribution (line 55), the authors claim that within-view negative pairs can mitigate the excessive separation of negative pairs under full-batch scenarios, whereas this point is supported by neither specific theorems nor experiments. Moreover, in the single-modal case, the cross-view and within-view pairs seem to have no difference, because they are both generated by the same combination of random augmentations. Do I understand this correctly? If so, how can more negative pairs help reduce the excessive separation?\n- To address the excessive separation problem of mini-batch training, the authors derive the theorems under the *fixed* mini-batch assumption. However, many contrastive learning methods (e.g. SimCLR, MoCo, etc.) empirically support the use of *random* mini-batches, i.e., the loader has the data reshuffled at every epoch. Is excessive separation still a problem under the *random* mini-batch scenarios? Or at least the authors should demonstrate the significance of the *fixed* mini-batch setting.\n- Proposition 5.2 demonstrates that excessive separation harms perfect alignment, but I think there lack demonstrations of the negative effect of imperfect alignment, because the essential goal is not to minimize the contrastive loss but to achieve good embeddings. What if the excessive separation brings some advantages to the downstream tasks? To demonstrate the negative effect of imperfect alignment, I think perhaps a worse error bound of the downstream generalization (e.g. linear probing) is necessary.\n- The authors claim that the proposed VRN loss term improves the accuracy of contrastive losses. However, according to Table 1 and Figure 2, there are cases where the accuracy drops after incorporating VRN, especially in Figure 2 CIFAR-100 DCL, the top 1 accuracy drops in 4 of the 5 cases.", "methods_and_evaluation_criteria": "- The experiments are conducted based on the InfoNCE-based contrastive losses. Additional validations on the independently additive contrastive loss (Definition 3.2) could be helpful.\n- I understand the authors might have limited computational resources, but additional experiments on the full ImageNet can be more convincing.", "theoretical_claims": "I've checked the proofs of Theorem 5.1, Proposition 5.2, and Theorem 5.6. They seem to be correct.", "experimental_designs_or_analyses": "Please refer to *Methods And Evaluation Criteria*.", "relation_to_broader_scientific_literature": "NA.", "essential_references_not_discussed": "The related works are properly cited.", "other_strengths_and_weaknesses": "**Strengths**\n1. This paper extends the theoretical analysis of contrastive losses with proper assumptions on the generation of positive and negative samples.\n2. This paper studies from a new perspective and proves that the similarity of negative pairs could affect the alignment of positive pairs.\n\n**Weaknesses**\n1. The fixed mini-batch assumption is somewhat less realistic. More evidence is needed to show if the excessive separation of negative pairs is indeed a real problem in contrastive learning.\n2. The experimental results are less convincing.", "other_comments_or_suggestions": "The notations need refinement or specification. \n1. At the beginning of Section 3, $x$ and $y$ are denoted as two distinct augmentations of the same instance (positive pair). Yet in the following parts, they can also represent negative pairs. \n2. The variance of a vector typically means the covariance matrix, but the notation $Var$ in this paper indicates the deviation w.r.t. the l2 norm. This needs specification to improve the readability.", "questions_for_authors": "1. What is the difference between cross-view and within-view negatives in the single-modal scenario?\n2. Is excessive separation still a problem under the *random* mini-batch scenarios?\n3. What is the negative effect of imperfect alignment?\n4. How does VRN perform under additive contrastive loss and full ImageNet?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper theoretically studies contrastive learning when negative pair are too close to each other, and mitigates the issue of the suboptimality by the variance reduction. Though a few reviewers initially had a concern about how the theoretical/empirical findings will be affected when it comes to the mini-batch setting, the authors have adequately addressed their concerns. Thus, I recommend this submission for acceptance.
Decision: Accept (poster)