Global Minimizers of Sigmoid Contrastive Loss
Authors: Kiril Bangachev
Guy Bresler
Iliyas Noman
Yury Polyanskiy
Venue: NeurIPS 2025 poster ()
Keywords: Representation Learning
Contrastive Learning
Multimodal Encoders
Sigmoid Loss
Modality Gap
Primary area: theory

Abstract: The meta-task of obtaining and aligning representations through contrastive pretraining is steadily gaining importance since its introduction in CLIP and ALIGN. In this paper we theoretically explain the advantages of synchronizing with trainable inverse temperature and bias under the sigmoid loss, as implemented in the recent SigLIP and SigLIP2 models of Google DeepMind. Temperature and bias can drive the loss function to zero for a rich class of configurations that we call $(\mathsf{m}, \mathsf{br})$ -Constellations. 
$(\mathsf{m}, \mathsf{br})$ -Constellations are a novel combinatorial object related to spherical codes and are parametrized 
by a margin $\mathsf{m}$ 
and relative bias $\mathsf{br}$. 
We use our characterization of constellations to theoretically justify the success of SigLIP on retrieval, to explain the modality gap present in SigLIP, and to identify the necessary dimension for producing high-quality representations. Finally, we propose a reparameterization of the sigmoid loss with explicit relative bias, which improves training dynamics in experiments with synthetic data.

Global Minimizers of Sigmoid Contrastive Loss
Kiril Bangachev kirilbemit. edu
Guy Bresler guy@mit.ed edu
Iliyas Noman iliyasmmit edu
Yury Polyanskiy ypemit edu
Department of Electrical Engineering and Computer Science Massachusetts Institute of Technology Cambridge, MA, 02139
Abstract
The meta-task of obtaining and aligning representations through contrastive pre- training is steadily gaining importance since its introduction in CLIP and ALIGN. this paper we theoretically explain the advantages of synchronizing with train- able inverse temperature and bias under the sigmoid loss, implemented in the recent SigLIP and SigLIP2 models of Google DeepMind. Temperature and bias can drive the loss function to zero for rich class of configurations that we call (m, See--Coostellationss (m, b -Conssellalass are novel combinatorial object related to spherical codes and are parametrized by margin and relative bias bret- We use our characterization of constellations to theoretically justify the success of SigLIP retrieval, to explain the modality gap present in SigLIP, and iden- tify the necessary dimension for producing high-quality representations. Finally, we propose reparameterization of the sigmoid loss with explicit relative bias, which improves training dynamics in experiments with synthetic data. All code is available Representationl wttpesettatiooeeeannnnnniieeeeee TerringTheory/Sigg....
Introduction
Background. Synchronizing representations' is an increasingly important meta-task in modern machine learning, appearing in several qualitatively different contexts. Models that operate jointly on visual and language data necessitate synchronization of the representations of images and text 21, DKAJ21, SRC+21, CSDS21, 2Y, LLXH22, SBV+22, BPK ZWM+22, HGW+22, 2YH+ 23C+ ZMKB23b, TGW+25b] and sometimes additional modalities well such audio, thermal data, and others [GENL" +23]. oftte-of-thhear vision models based on self-distillation rely on aligning the representations of augmentations the same image [CKNH20a, HFW+ 20, CTM 21J. Likewise, aligning the representations of data produced by teacher and student networks [TK120h, GS25) has been proposed as a method for distillation. Similarly to the teacher- student setup, the field of cacppald-compatible learning aims synchronize the features produced by new models with features of already trained old models [RAVF*22 22, BPBDB23, SXXS20, JFF 23].
To mathematically formalize the task of synchronizing two representations, suppose that there are N data pairs {(,,, YJ) (X' x )') For concreteness, one can think of as the space of images, y as the space of text, and (X,, Y:) satisfy correspondence relation such as the fact that they are true image-caption pair. The goal is to train neural network encoders fo X I&d and Is : y - Rd in such a way that the embeddings produced by them capture the correspondence relation. Such synchronization is usually achieved via minimizing a certain contrastive loss.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite the prevalence of the task of synchronizing representations, there is still limited understand- ing of what loss function to use, how to choose its
and what properties of the synchronized embeddings are desirable. Theoretical results focus mostly on two loss functions - the InfoNCE loss [W120, CRI++20, RCS121, EW22, L.222, PHD200 2R] and the Sigmoid Loss [LCS24, LS22], which both depend on temperature and bias hyper-parametee have yielded useful insights, they leave important gaps in our understanding:
While prior works
1. Currently understood regimes for the number of represented objects N compared to the dimension of representations d do not reflect practice. To the best of our knowledge, in all prior theoretical works, either d 2 N, or N approaches +00 for fixed value of d. As a comparison, the SigLIP2 model embeds text and images in d = 10 dimensions and operates with a dataset of size N z 10 [TGW 22bb.. Thus the practically relevant regime- - in which d < N < 2d - is not captured by prior work. The different regimes exhibit crucially different behaviors: practically relevant phenomena such as the modality gap [L.ZK+22] only arise when N > d, as we show in Theorem 3.6.
2. The optimal configurations identified by prior works are too rigid. For example, works in the regime N I d typically suggest a simplex structure of the embeddings of each modality [EW22, LS22, LCS24]. This does not explain what the minimizing configurations are when one modality is pretrained and locked. In the regime N + +oo, existing results typically suggest a perfect alignment between different representations. Again, this may be too stringent since it has been proposed that "different modalities may contain different information" [HCWI24]. In fact, empirical work suggests that even after synchronization, representations of text and images are completely disjoint, phenomenon known as the modality gap [LZZK FMF25].
Our Contributions. In the current work, we address these gaps by analyzing the sigmoid loss with trainable inverse lemperature and bias parameters, as used in Google's 's SigLIP models [ZMKB23b, TGW+ 25b] and Gemma [TKF*25), Making bias and temperature trainable a key departure from prior theoretical work [WI20, LS22. EW22, LCS24] and leads to novel theoretical guarantees and practical recommendations, We first introduce the sigmoid loss and then describe our contributions.
The sigmoid loss for U+- = fo(X;) and V = (Yi) and inverse temperature' I. and bias b is:
N
CSI (0,00,; d;t,b) = Σ log (1 (1+ cxp((p V3) + b)) + +Σ log (1+ cxp(ttt, VS) - (I) iipll
b)).
The first part of the loss encourages the embedding of an image and its caption to be similar, while the second part encourages mismatched image-caption pairs to be dissimilar.
1. The Geometry of Loro-Loss Configurations. Our work is the first to rigorously characterize global minima in representation synchronization tasks in the practical regime N > d.
Products Between
We show that the SigLIP loss-with trainable temperature and bias can be driven to zero by a rich family of solutions, which we fully characterize in terms of two novel geometric quantities- - the margin m 2 O and the relative bias b Formally, (d, m, (he))-Conttelatiin {(U:, V.)) E gd-1 is defined by the follow- ing inequalities:
Text and Image Embeodinng
Motching Labet
Figure 1: Distribution inner products hetween image and text embeddings from the ImageNet validation set using the B/16 224 x 224 SigLIP model available at (2) HuggingFace. The experimental details for all plots as well as further experiments are in Appendix D.
(U,, V)) 2 m + byel (U,, V2) -m+ + bee
Vi, Vi Aj.
The existence of such m. bys,, which one can observe is equivalent to the inner product separation (U, V₂) maxizy(U; (U,, VJ), is necessary and sufficient condition for {(U,, V.) N be global minima of the sigmoid loss with trainable inverse temperature and bias. We show that this nearly satisfied in practice for the SigLIP model trained on real images and text See Figure I. Surprisingly, any configuration satisfying this condition is also a global minimum for the triplet loss, see Observation 4. We interpret the margin and relative bias in Section 3.1.
'Previous works on synchronizing with sigmoid loss such as [ZMKB23b, TGW 25b. LCS24J call the temperature. We call inverse temperature to he more consistent with statistical physics terminology. We usually omit the parameter d since it is clear from the context and only write "(m, bonn-coonsellattoon.

---

In practice, one needs to choose dimension for the encoders which has large enough "capacity" to hold the embeddings of great many pairs U,, V,, However, despite intuitive notion that capacity should increase with dimension, to the best of our knowledge no such quantitative characterization was available before our work. Formally, we deñine the following combinatorial problem and make partial progress in Section 3.2 via connection to spherical codes.
Problem 1. For a given m 2 O, brel E [-1, find the largest number of points N = NMRB (d, m, brel) such that there exist 2N vectors {(U,, Vi)l E Sd-1 satisfying (2).
2. Success of Loso-Loss Configurations on Downstream Tasks. In Corollary I, we use the characterization of zero-loss configurations to show that a standard nearest neighbor search on any (m, -oonsCtCntllllation gives per- fect retrieval, even though typically there is no perfect alignment between the two representations. Increasing the margin of constellation makes retrieval robust to larger approximation errors. This is important in prac- tice since retrieval is often performed via an approxi- mate nearest neighbor search for computational efficiency [XXL+21, KZ20, MT21].
No onviditteo ctir
Medallr Oup
Vogre
of exppaaarati and keat
Coustiiiie
Rebave Blai Roc.
Region of possible (m, b Constellations. In red is the impossible re- gion, in which no large configurations are possible (Theorem 3.4). In green is the re- gion where constellations of exponential size exist (Theorem 3.3 and Theorem 3.5), In the shaded region we prove that a modality gap exists (Theorem 3.6).
Figure 2:
3. The Modality Gap: Synchronize, do not Align. The analysis of [WI20] suggests alignment between represen- tations when training via the InfoNCE loss the represen-
tations of the word "cat" and the image of cat should (nearly) coincide. Yet, has been empirically observed that there is a modality gap [LZK 22, FMF25]. The representations of images and text- synchronized via the InfoNCE loss in CLIP- do not align, but rather belong to fully disjoint, linearly separable regions. Furthermore, this is not caused by the difference between architecture of image and text encoders, as initially thought, but rather directly by virtue of (approximately) minimizing InfoNCE loss [FMF25].
We shed light on this empirical discovery and prove in Theorem 3.6 that linear separability between modalities holds for any zero-loss con- figuration of the sigmoid loss in the practically relevant regime N > d when breil < m (Figure 2). We verify our findings by performing experiments with 8 different SigLIP models from Hugging Face on the ImageNet dataset (models given in Table 5). We observe perfect linear separability of image and text embeddings for all models. From a philo- sophical point of view, as "different modalities may contain different information" [HCWI24], it only natural that they be represented in disjoint parts of the space.
hyparplane
With Zero Clecuitcation on Erres
Iaga
Modality gap in SigLIP on ImageNet data with the B/16 model with resolution. We find perfect linear separator using the per- ceptron algorithm.
Figure 3:
We leverage the modality gap to build linear adapter which can be used towards synchronizing representations when one encoder is locked. This is the reason why we use the name representation synchronization rather than representation alignment: alignment between modalities is neither achieved nor desired.
4. Implications of The Solution Geometry in Practice: Relative Bias Parameterization of Sigmoid Loss. We propose parametrization of the sigmoid loss that depends on the relative bias rather than the bias in Definition 1. The relative bias parametrization has the following advantages:
1. Locked Representation: For example, in LiT [ZWM++22], the image encoder is already trained and locked and we want to synchronize the text encoder with it. The sigmoid loss with trainable parameters in the relative bias parametrization allows us to find a zero-loss configuration for text and images regardless of the image encoder. In Observation 1, we show that trainable relative bias and inverse temperature provide mechanism implicitly add linear adapters on top of the two encoders as in Figure 4. The adapters we propose in Observations I and 2 extend the Double- Constant Embedding Model of [LCS24].
3

---

2. More than Two Modalities: The framework of training with the relative bias parameterization also leads to theoretical guarantees for the global minima of synchronizing more than two modalities via the sigmoid loss. In Observation 2 we show that the parameterization implicitly captures the addition of modality-dependeer linear adapter to each encoder.
lcacaer)
Shored Panancterd
Signrsi (%4
Truieable Eucodo stndest)
Adpper A
Figure 4: Implicit adapter in relative bias parame- terization of sigmoid loss with a locked representa- tion. The paramcters σ, 6, ò in green blocks are trainable. Parameter @ is locked.
Guiding Relative Bias: Relative bias and margin, which are related by inequalities that we fully char-
acterize Theorems 3.3 and 3.4, control important properties of the synchronized representations such as retrieval robustness and the presence of modality gap. We observe empirically that in the usual sigmoid loss parameterization, Adam [KB15) 11] finds configurations with a zero relative bias, thus limiting the set of trained representations. By adding relative bias parameter and locking it, we can provably guide the zero loss configuration more diverse set of solutions. See Appendix D.4.
2 Background and Prior Work
Representation Learning And Synchronization. A key insight in 1RKH+21, JYX+ is that training model to simultaneously operate on multiple modalities (such image and text) enables SOTA performance on individual modalities as well - "if you want to train the best vision model, you should train not just on N images but also on M sentences" [HCWI24]. Several empirical approaches towards synchronizing multiple representations have been proposed, including CLIP [RKH*21] 21J, BLIP [LLXH22], ALIGN [JYX+21]] 21], LiT [2WM+22], and SigLIP [ZMKB23b, TGW 2GbJ. The task of synchronizing representations goes beyond synchronizing across different modalities such as image and text, but also includes synchronizing the representations of student model to teacher model with the purpose of distillation [TK1220, GS25] self distilaattatonnn [CTM* 21], and aligning the representations of data augmentations [ZIE+ 16, NF16, GSK 18, CKNH20a, HFW+ 20, CTM+21J.
Formalizing Representation Synchronization. In this paper, we consider unit-norm encoders fg : - gd-1 I4 y - gd-1 (unit norm representations are predominant in practice, e.g. [SKP15, PVZI5, LWY+ WXCY17, CKNH20b, HFW+ TKI20a, ZMKB23b, TGW+25b) and others). Synchronizing the representations produced by g6 is usually achieved via minimizing certain loss function L over the kernel produced by the embeddings:
()) L( ((((fo())) (ol,))))))))))))
where I' is a set of sypper-paramters, typically involving (inverse) temperature and bias. The optimization is performed via batch first-order optimization methods such as SGD or Adam [KB15].
Depending on the targeted representations, one may choose the parameters over which the opti- mization is performed. If both fa. 9o are untrained, one may perform gradient descent on both d, 0 in (3) as in CLIP [RKH*21]
On the other hand, if one of the models say So is al- ready trained and trusted (for example, because
the teacher model that we are trying to distill [CTM+21, GS25J), we do not update its parameters or only update a small adapter on top of it as in [ZZF* 22, LXGY23, GGZ 24, YZWX24, LHY 24, EANP24]. Likewise, this is the case when one of the modalities has already been trained and is locked [ZWM* NWWM+22, 22, RNP 22, LLXH22, LLSH23).
Besides choosing which parameters to update, one also needs to choose conerete loss function C. The choices depend on two factors: 1) What is the geometry of the desired minimizing configurations of representations? 2) How efficient the computation of the loss terms of the batch size?
We focus on two different loss functions - InfoNCE and sigmoid - and now survey previous works on them. In order to understand the solution geometry of the minimizers, typical assumption is that the underlying networks fo, Is are sufficiently expressive and can encode any embedding {(U,, V = ((fo(X,), go (Y.)) [WI20, EW22, LS22, LCS24]. We also adopt this approach.
Solution Geometry with InfoNCE. The InfoNCE loss [vdOLV19] with inverse temperature / 0 and bias is a special case of (3) and takes the following form
CInfoNCE CInfoNCE(((UU, (1(U,, VOH =
1
log E, cuUU.NS)
log
((U,,V
(4)
E, e(UU,,v3
4

---

It effectively takes a soft-max over all the rows and columns of the matrix V an interpretation that yields connection to the InfoMax principle [HFLM* 19] as well as to maximizing point-wise mutual information [TKI20a] and approximate sufficient statistics [OLCM25, LM25].
The solution geometry has been characterized in the case d 2 N 1. The global minimum loss is achieved when Uj- = Vú for each i and U1, U Un form simplex [EW22, LS22]. When N - the minimizing measures converge perfectly aligned (U; V; for all i) and uniform (the discrete measure corresponding to {U+ converges weakly to the uniform measure). Several works including [SPA* + 19, EG24] take a different direction and analyze the global minima of the InfoNCE loss (and its symmetrization SimCLR) in terms of performance on downstream (linear) classification tasks instead. While such geometric characterization is appealing from practical point of view, these works also do not address the aforementioned gaps in our understanding. The results of [SPA+ 19] hold the regime N _> +00 and [FG24) points out that "temperature scaling in the SimCLR loss remains challenging."
In very different direction, recently was also rigorously shown that the InfoNCE yields an optimal dimensionality reduction with input data from a Gaussian Mixture Model [BKS25].
Solution Geometry with Sigmoid Loss. An alternative loss function used towards alignment is the sigmoid loss [ZMKB23b, TGW+25b] defined in (I), One advantage of the sigmoid loss over InfoNCE is that it does not have batch normalization term such as Σ exp(t(U,, V2) b) and, thus, every pair (U,, Vj) be processed separately. This allows for parallel computation.
The solution geometry of configurations achieving global minimum loss has been characterized when d 2 N [LCS24]. For simplex {W.) in $d-2 and some ó E [0, 1J, it holds that U; the relationship
(oW, vI- 8²), Vi (ÓW, -VI 8²) for each i, where the value of 5 depends between t and b. In most cases, either the representations collapse to perfectly aligned (5 = 1 and for all i). The construction of [LCS24] is the basis for several of our results, including the adapters proposed in Observations 1, 2.
U, = V, for all or antipodal (5 0 and U, = V, (1, 0,........ ..... 0)
Other loss functions. Loss functions such as the triplet loss [SKP15] and f-MICI (LZS 23] have also been considered. We further discuss the triplet loss in Appendix A.3.
3 Main Results
3.1 Geometric Characterization of Zero Loss Representations
In (1), L Vo) b) 2 0 holds for any inputs because log(1 e*) 2 0 for any K E R. Hence, global minimizers are any choice of representations and parameters {U,, Vo) E (8 x (Sd-1)0N, x [0, +oo] [-o0, co] leading to zero loss. We characterize such configu- rations fully in the following theorems. The proofs are simple and delayed to Appendix
Theorem 3.1 (All Global Minima are (m, bral) Jonnteosttllltionns.... Suppose that any iterative algorithm
produces a sequence (U(()
Ji=1 (V((+) Ji=1 ((0) 0,6(*) for 1,,,, such that
N
lim (SU) +0000
(4)
1 ( ( = 0.
Then, there exists some subsequence indexed by (s, ) such that
lim U( =U,, lim vier) = V: for all i, lim b(e.) U.
+o0 ((ar) = brel,
(5)
and there exists some m 2 such that {(U,, Vo) m, bral satisfy (2). Theorem 3.2 (All (m, brei) Consstelllllss Are Global Minimizers). Suppose that ((U,, N gd-1 satisfies (2) for some m > 0. If we set b = brel X t, then
lim
+00 L(U.)) (V.) brel x () =
Moreover. for m (min, (U,, V1) (1)-mash max V2)), bia = (min, (Us, Vi) + maxi4 (U6, V2)), inf C ((U:) [Vi) 6) = = -oom)
and is achieved when b = bü x t ot).
5

---

This characterization is for global minima in the case of zero loss, which may seem too idealistic. However, turns out that practically trained models are (up to a small error) also Constellations. In Table 5 in Appendix, we provide the optimal relative bias and margin after removing 5% outliers from the positive and negative pairs.
It turns out that (m, constellllss are also global minimizers for the triplet loss, which is another popular contrastive training objective [SKP15], see Appendix A. Theorem 3.2 shows not only that any (m, bral) -constellllnn is a global minimizer, but also that the optimal margin m² characterizes the speed of convergence of the loss zero.
Any (m, Thee)-Costellattin yields perfect retrieval follows corollary of Theorem 3.1. In the image-text retrieval task, one is given an image (respectively text) and has to produce the text (respectively image) that best matches it. In mathematical model, this corresponds to producing U; on input V; (and V2 on input U.).
Corollary (Nearest Neighbor Search Yields Perfect Retrieval). Suppose that {(U,, V.)a is zero loss configuration. Then, nearest neighbor of U2 among (V,) is Vi. If, furthermore, the margin m is strictly positive, this neighhor is unique.
In practice, retrieval is often performed via approximate nearest neighbor search [XXI. 21, K7200, MT21] this approach has significant computational efficiency advantages. Hence, representations more robust to approximation errors are more desirable. Since min.(U, Vi) max (U;, Vz)> 2 2m when (2) satisfied, representations with a larger margin are more robust. The importance of margin on retrieval has been empirically observed and exploited in several empirical works [LWYY16, DGY+22). Corollary is for perfect zero-loss constellations. more robust version also holds, which is closer to practice due to the fact that practical models are not trained to zero loss (and cannot be, both due computational limitations and mislabeled data). We note that in the basic version of the proposition, one can ignore batch size and take B = N. We also include the effect of batch size since this is how models are trained in practice.
Proposition 1 (Robustness of Retrieval via Nearest Neighbor Search). Let the embedded dataset be {(U,, Vo) Suppose thatfor inverse temperature and bias t > 0, b and some E [0, 1], and some batch size N > B VN, it holds that.3
IE [N
Σ log exxp((tt(Ue, Vo,) + +b) j=1
+ Σ log (1 e1+exp(t Vaj) - bi)] s & logg? 2. ktfj
Then, for at least 1 returns V, (respectively, U.).
B(B-1 fraction of the values Ug (respectively, V4), nearest neighbor search
NE
The proof is delayed to Appendix A.2. Note that while N >> B > VN is restrictive, it is relevant to models trained with massive compute. For example, in [ZMKB23b], the authors run models with batch sizes up to 64000 which makes the statement meaningful for datasets of size as large as 10
3.2 Constructions of (d, m. b -onntteltllla And Cardinality Bounds
Our results so far are vacuous no (m, haa--oontellations exist. this section, we show generic construction which is largely motivated by the Double-Constant Embedding Model of [LCS24] but replaces the simplex with a spherical code. For a E [-1, and d e N, a (d, a a--spherical code is a collection of vectors X1, X2,..., ..... X, E 5d-1 such that (X,, X,) S for all + j [CSB+ 13, (52)). In particular, any (d, a) code a (d, 2 bre! + >constellation and vice-vert versa. This implies that any construction of spherical codes immediately implies a construction of (m, brel) Connstellllls when m + b = 1. Spherical codes are a well-studied object in combinatorics and many constructions exist depending on (see [CSB* 13] and references therein). The following construction shows that we can extend to the case when m + b 7 1.
We denote by x the uniform distribution over B- B-tuples in [N] of distinct indices.

---

Construction (Construction of (m, Constelonlllll Consider any (d-2, 2, m, bra)) constellati
{(U,, = Then, for any E [0, 1) such that 8² + x² < 1, the following vectors form
(d, m', b/al constellaaio with m' = 5 m and by brel + o²² (1 8² )²):
U = (OU;, 0. V 12
and
V = (oV, 0, V 82 02)
(6)
This construction shows that (m, Maansoooselllasnns not only exist but constitute a rich family. One can in fact construct them from any locked (d - 2)-dimensional embedding {Xx)]== as long as X; * X, for i + j which is the basis of our algorithm for synchronizing with a locked encoder in Observation 1. Recall that the margin impacts the robustness of the representation for retrieval. Thus, it is of both practical and theoretical interest to analyze how large the margin could be for a given dimension and sample size N. Construction immediately gives a recipe for this based on spherical code bounds.
Figure 5: 3D visualiza- tion of configurations in Example 3. obtained by minimizing sigmoid loss with Adam; each (U, Vi) pair is reflection across hyperplane.
That is, let Nsc (d, a) be the largest possible size of a (d, a-spheriiall code. Let Esc(a) = lim infff++++ log (sc(d.a) Determining Nsc, Esc is 13). We similarly define the numbers (M (d, m, brel) and EMRB (m, b for the largest (m, b constellations in dimension d. Construction 1, together with the classical bound Esc(a) 2 log2 VI- due to Shannon [Sha59] and Wyner [Wyn68] implies:
d a well studied problem in coding theory [CSB+ 13].
Theorem 3.3 (Lower Bound on the Size of Constellations), Suppose that m 2 O, brel E [-1,1]
satisfy m + b 1 and 3m- < brel- Then, there exist (m, brel) constelontlls of size exponential in dimension and furthermore
1 + brel 3m
1 brel 3m 3mm 1 log2 (1- 2
)²
EMRB(M, bre!) 2 Esc
2
1 + brel + m
+ brel + m
Proof. Let = 1+bw-3m 3m I+b
E [0, Let X1, X2...., .N be an a-spherical code in dimension
d - 2 of size exp (d (E) + o(1)) = exp (d(Esc(a) + ((1)) Choose Φ, 6 as follows:
2m 82 i- and 3m < + brel imply that these values are well-defined in the sense that 82 > 0, Φ > 0 and, furthermore, 52 + x² < 1. Now, we can apply Construction with U, V; X, and 6, Φb and conclude the desired result.
/ 2b.itt 2- One can easily check that the inequalities m + brel S
The conditions m + beel S and 3m < 1 + bret are not virtue of our construction, but turn be necessary via an argument resembling Rankin's 's proof that among any k + vectors in Sd-1 1, there exist two with inner product least - 1/2 [Ran55]. We actually manage to prove a lower bound for more general set of configurations than constellations.
Theorem 3.4 (Upper Bounds on Margin via Relative Bias). Suppose that {(U,, V satisfy that 1
N E,U, V.) 2 m + b and N(N-T] Ei Vz) S -m + brel (in particular, this holds for any (m, conttelacnnsttllliiinn)) Then, it also holds that
m bral 1 and 3m S 1 brel + o().
Proof. The inequality b I 1 is trivial since m S max Vi) S max | 1 [ < by (2) and Cauchy-Schward For the second inequality, we use the following fact from [LCS24]. For any unit vectors (U, v.)) it holds that N= Eijy (Ui, V₁) 2 2N (U, V.) 2. Using (U, V2) < brel m, (U, V.) 2 bral + m gives (3 )m S brel-
N-2
We also provide upper bounds on the size of constellation given the margin m. This can be used to inform the size of the embedding space given the number pairs (U,, V.) we want to embed in
Theorem 3.5 (Upper Bound on the Size of Constellations). Suppose that (U,, K))N is (m. bra))-
constellation for some m 2 O, b E [-11. 1] which satisfy m brab S 1 and 3m S b Then, S
exp d log
14b -3m 3m I+ +
+ o(d) Equivalently, EMRE (m, byel) S - log (1
I+ +
7

---

The proof as well as an illustration of the bounds is delayed to Appendix B. In Appendix E, we note that the proof also illustrates a connection with the linear representation hypothesis, e.g. [PCV24].
We note that in [RCSJ21, Theorem 4), the authors find a different connection between spherical codes and minimiz- ers of the InfoNCE loss. In the recent work [WBNL25), the authors show different dimension lower-bound for existence of vector embeddings for top-k retrieval.
Magie
We end with zooming into the Figure in Appendix and plotting the performance of several trained SigLIP models com-
Relaive Boss
from Hugging Face. We observe two clusters posed of the larger s0400m models (around 1B parameters) and another of smaller models (up .B)..
Figure 6: Margin and relative bias with re- spect to mean inner products of positive pairs (U, Vi) and negative pairs (U, V2) from the ImageNet validation dataset.
3.3 The Modality Gap in SigLIP
The construction in Example satisfies the modality gap property - when å > O, the representations of the two modalities are separated by hyperplane (orthogonal to the last coordinate). This phenomenon has been observed empirically on synchronized text and image embeddings in CLIP [LZK 22, FMF25] and in SigLIP by in Figure 3. We show rigorous justification for this.
Theorem 3.6 (Modality Gap in Zoso-Loss Configurations). Suppose that N 2 d and {(U,, v are such that (Ue, V,) > 0 for all i, (U, Vj) 0 for all i + j. This happens for example, when > |brell in a (m, Pre)-CConsellltion. Then, there exists some E gd-1 such that:
U4) 0 (h, V2) <0 0
for all i,
for at least N - values of
(8)
The fact that the condition (8) is satisfied for at least N - d values of j instead of all N is rather minor in practice. As mentioned, in SigLIP2 TTG++25b) the modality gap holds for all but .0000001% of the text embeddings. We note that the theorem is essentially tight- - in Example 3, we show an example in which d of the vectors Vú cannot be separated from the vectors U,. We also that m breil is also plausible as practically trained models such as SigLIP2 have small relative bias of magnitude less than 0.1 [ZMKB23a].
zs 1010 and d R. 10 Thus, our result shows
Proof Sketch. The full proof in Appendix C, where we also analyze further properties of configura- tions satisfying (7) and (8). Here we give sketch. First, we use Helly's 's theorem (Theorem C. to show that the convex sets {a: (ac U.) > 0) have a non-empty intersection and, hence, there exists some h E gd-1 such that (h, U,) (Theorem C.3) to show that the projection h of h on the convex cone defined by U1, U2...., UN also has this property. Finally, we use Caratheodory's theorem (Theorem C.2) to show that h has positive inner product with all the vectors U, and is in the convex cone of at most of the vectors Uy. This implies that has a negative inner product with all the other N - d vectors Vi.
for each Then, we use the hyperplane separation theorem
3.4 Experiments: Sigmoid Loss with Explicit Relative Bias Parameterization
Due the importance of the relative bias parameters for global minima of sigmoid loss, propose a parameterization that explicitly captures this dependence.
Definition (Parameterization with Explicit Relative Bias). The relative bias purametrization of the sigmoid loss for encoder fe, 94 over data pairs {(X,, Y.) with U, fo(X.), V, =9 is
RBB CRB-SiE(A, dz brel)
Σ log (1 (1te- ((wk)(00)))tbb))
+ Σ log ((0,
Clearly, CRB-SiE(B, d; brel) (S t. brel t) the loss functions the same. However, we show that running Adam [KB15] on CRB-Sis yields faster convergence see Figures 7 and 8. It also provides the additional flexibility to freeze the relative bias to a desired value and only train inverse temperature. This may be important since observe that in practice relative bias converges

---

to O when not frozen. For example, in SigLIP2 [ZMKB23a] with the B/16 model with resolution 384 x 384, we have learned parameters 117.8,b = -12.9, so b -0111. We give further experimental evidence for this in Appendix D.4. Thus, we propose using our parameterization Sig in practice over [isg.
Fixed Relative Bias. One functionality that this new parameterization gives us is to train models with fixed relative bias. As expected from Fig. 2. this in turn has an effect on the margin of the configurations. We plot in Fig. 10 in Appendix the evolution of margins (computed as (min, (U,, Vi)- maxizs (U,, /2)) for different fixed relative biases and in Fig. 9 the final optimal relative biases (computed as (mine (U;, Vi) + maxi/y (U;, v2))
Conuergence Zero Loss With Different Sigmold Implerr
Locked Encoder. In particular, this gives a concrete recipe for training via the sig- moid loss with a fixed representation: one just adds a simple adapter Alocked that trans- forms X; - (6X;, v - 82) for the locked representation and A Traiaable that transforms X; + (ōX;, -vI-vvii 8²) for the trainable representation. It turns out that the relative bias parametrization cap- tures this transformation without explicitly adding an adapter.
Rorotton
Figure 7: Synchronizing a locked representation with differ- ent sigmoid loss functions on synthetic data. On the left, we have the evolution of the loss function. On the right, we show the distributions of non-matching inner products (U,, V2) for A in red and matching inner products (Us, Vi) in blue for each model.
Observation 1. For any {(U,, Vi)) and 5, bret, t, it is the case that
CRB-SiE (E((fockk A (V.) brd) CRB- Sis (I(UU, VOYE t5 bral + (1 - 6)3,
X2
As we can see in Figure 7, the mod- els with trainable b (respectively t, bie)) significantly outperform the model with fixed temperature and bias. Furthermore, the convergence to zero loss faster for
Convergence of the Looa With pifferent Sigmolel implementations
Loas flg scaldl
CRB-Sig Sig than CSie. Thus, recommend synchronizing with CRB Sig and trainable
t, beel-
Iooration
Figure 8: Inner product distributions between k 4 modali- tics synchronized with different implementations of sigmoid loss. We plot the same data as in
More Modalities and A New Perspec- tive on Simplex Embeddings. Our dis- cussion so far has been predominantly in the case of two modalities, To synchronize the representations {(U.9)
(8) N. of k > 2 modalities, one typically minimizes the sum of several pairwise losses [TKI20a, GENL More formally, if G = (V, E) is the synchronization graph on vertex set V = {1, 2,... k} the different modalities, one minimizes
Σ CRB- (RE-SS((((U())) "ubonn bal).
(1EE)EE
Common instances are when G is the complete graph and one sums over all pairwise losses [TKI20a], and when G is a star graph with one central modality [GENL+23], Since the loss function [is is non-negative, a configuration ((U!") ".......")))) )((')) is zero-loss if and only if there exist some m, b such that ((U,5)) ) m, b is zero loss for any (J1,j2) E E. In particular, ((U ((())) IN is zero loss if there exist some m. brel such that (((()) ) m, b is zero loss for all J1 * jz. This leads us to the following construction.
Construction 2 (Construction of Constellations). Consider any (d - k + 1, a--code (X,) Let W1, W2..... Wh E g be the vertices of regular k-simplex. Then, for any 8 E [0, 1), the following configuration is zero loss for any synchronization graph:
(1
5² - a)
1 1 82
0(6) = (8X,, V1 -8w,),
m
brel
(10)
2
1
9

---

We can enforce this structure with an adapter which appends a modality-dependennt suffix. Observation 2. For any ((U))) U ..... U,!4, ) and ó, brels 4 CRB-SiE (IA (AA( (AOOS"((AA""").....CCC. ( Birii bw) cRB- -is (IU!)............... ..... )3 16 brat + (1-)
AL
3.5
Ablation Studies
Training the temperature and bias is the key mechanism that drives the loss to zero for a wide range of configurations which we described (m, biel) Doontteostllltionss We compared our proposal of training CRB-Sig with trainable bret; parameters against several alternatives. We concretely focus the inner product separation condition min, (U,, Vi) 2 maxiæj (U; Vj) and corresponding margin. This a key property of interest since, as explained, determines the success of the model on retrieval via (approximate) nearest neighbor search. We also analyze the convergence of the loss to zero, which is an indicator for how many epochs of training a model needs till convergence.
1. Training with fixed low inverse temperature (f S 10) and bias as in the analysis of [LCS24] is first natural alternative. We observed that in the contexts of synchronizing multiple embeddings (Figure 8) and synchronizing with a locked encoder (Figure 7), the resulting embeddings fail to satisfy the inner product separation condition do with much smaller margin than models with trainable inverse temperature and (relative) bias.
2. Training with fixed high inverse temperature (1 > 10) and bias. Any (m, -oont-cottllllttionn is nearly a global minimum the regime of large 6, so one may expect similar performance to the trainable inverse temperature and bias model. This approach fails in practice since it does not allow the algorithm to gradually find the synchronized representations. The embeddings discovered are not useful towards retrieval the inner- product separation fails and the loss does not approach zero (Figures and 8), even though representations with nearly-zero loss exist due to the low temperature.
3. Training with bias parameterization. Finally, we compared against the bias parameterization _i used in [ZMKB23b, TGW+ While this model also generally led (m, ConsttCloss which can be used for perfect retrieval, we observed slower convergence of the loss function and smaller margin due tendency towards zero relative bias (Figures 7 and 8).
4 Limitations and Future Directions
We provide the first theoretical analysis of synchronizing representations in the practically relevant regime d < N < 2d. A theoretical limitation is that while we identify global minimizers and empirically show that first-order methods such as Adam find them, we do not prove rigorous performance guarantees for first-order methods. Another theoretical limitation is that we do not fully resolve the combinatorial Problem 1, which as we point out is practically relevant for choosing the embedding dimension of encoders. Finally, we show that the parametrization of sigmoid loss with relative bias leads to more flexibility and faster convergence on synthetic data, but do not perform experiments with it on real data. We believe that all of these are exciting directions for future research.
Acknowledgments
KB was supported in part by an NAE Grand Challenge Vest Fellowship. GB was supported by NSF award 2428619. The research of YP was supported, in part, by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number FA8750-19 118750-19-2-0 1000-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
References
[BKS25] Parikshit Bansal, Ali Kavis, and Sujay Sanghavi. Understanding self-supervised learning via gaussian mixture models, 2025.
10

---

[BMS12] Vladimir Boltyanski, Horst Martini, and P.S Soltan. Excursions into Combinatorial Geometry. Universitext. Springer Nature, Netherlands, 2012.
[BPBDB23] Niccolò Biondi, Federico Pernici, Matteo Bruni, and Alberto Del Bimbo. Cores: Compatible representations via stationarity. IEEE Trans. Pattern Anal. Mach. Intell., 45(8):9577-9588 August 2023.
[BPK*22] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https://gi thtgg com/ Taakaobrain/ccoo-daaassss 2022.
[Carl1] Constantin Carathéodory. Über den variabilitätssereich der fourier' schen konstanten von positiven harmonischen funktionen. Rendiconti del Circolo Matematico di Palermo (1884- 1940), 32:193-2111 1911.
[CKNH20a]
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020.
[CKNH20b] Ting Chen, Simon Korbliihh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
(CRL*20) Ching- Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. Advances in neural information processing systems, 33:8765-8775. 2020.
[CSB+13] J.H Conway, N.J.A Sloane, E Bannai, R.E Borcherds, Leech, S.P Norton, A.M Odlyzko, R.A Parker, L Queen, and B.B Venkov. Sphere packings, lattices and groups, volume 290 of Grundlehren der mathematischen Wissenschaften. Springer, third edition. edition, 2013.
[CSDS21] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3557-3567, 2021.
[CTM*21] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bo- janowski, and Armand Joulin. Emerging properties in self- suff-super vision transform- ers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, 2021.
[3WC+23] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver. Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal. James Bradbury, Weicheng Kuo, Mojtaba Seyedhos- seini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly- scaled multilingual language-iman immge model. In The Eleventh International Conference on Learning Representations, 2023.
[DGY+22] Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 44(10. (0:000))::6222550001 5979, October 2022.
[DKAJ21] Karan Desai, Gaurav Kaul, Zubin Trivadi Aysola, and Justin Johnson. Redcaps: Web- curated image-text data created by the people, for the people. In Thitty-ffthh Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Round 1),
2.21.
11

---

[EANP24] Sayna Ebrahimi, Sercan O. Arik, Tejas Nama, and Tomas Pfister. Crome: Cross-modar adapters for efficient multimodal I1m, 2024.
[EG24] Anna Van Elst and Debarghya Ghoshdastidar. Tight pac-bayesian risk certificates for contrastive learning. CoRR, abs/2412.03386,, 2024.
[BW22] Weinan E and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova, editors, Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, volume 145 of Proceedings of Machine Learning Research, pages 270-290. PMLR, 16-19 Aug 2022.
[FMF25] Abrar Fahim, Alex Murphy, and Alona Fyshe. It's not modality gap: Characterizing and addressing the contrastive gap, 2025.
[GENL+23] 233] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind one embedding space to bind them all. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15180-15 15900, 2023.
[GGZ+24] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International
jnternationalouraall journal of computer vision, 132(2):581-595, 2024.
[GRI++24] Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structuring representation geometry with rotationally equivariant contrastive learning. In The Twelfth International Conference Learning Representations, 2024.
[GS25] Nikolaos Giakoumoglou and Tania Stathaki. Discriminative and consistent representa- tion distillation, 2025.
[GSKI8] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, 2018.
[HCWI24]
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 20642220642. PMLR, 21-27 Jul 2024.
[Hel23] Ed. Helly. Über mengen konvexer körper mit gemeinschaftlichen punkte. Jahresbericht der Deutschen Mathematiker- Verernittkeeereeeeeiiuinnngg. 12:175-176, 1923.
(9FLM+19] Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildonn Karan Grewal, Phil Bach- man, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
[HFW+20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726-9735. 2020.
[HGW+22] Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pretraining for image captioning. In 2022 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17959 2022.
[JFF+23] Florian Jaeckle, Fartash Faghri, Ali Farhadi, Oncel Tuzel, and Hadi Pouransari. Fastfill: Efficient compatible model update. In The Eleventh International Conference on Learning Representations, 2023.
12

---

[JX Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuad Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, vol- ume 139 of Proceedings of Machine Learning Research, pages 4904-4911 PMLR, 18-24 Jul 2021.
[KB15] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[KZ20] Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 39 48, New York, NY, USA, 2020. Association for Computing Machinery.
[LCS24] Chungpa Lee, Joonhwan Chang, and Jy-yong Sohn. Analysis of using sigmoid loss for contrastive learning. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 1747-1755. PMLR, 02-04 04 May 2024.
[LHY+24] Haoyu Lu, Yuqi Huo, Guoxing Yang. Zhiwu Lu, Wei Zhan, Masayoshi Tomizuka, and Mingyu Ding. Uniadapter: Unified parameter-efficien transfer learning for cross- modal modeling. In The Twelfth International Conference on Learning Representations. 2024.
[LLSH23] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language- image pre-training with frozen image encoders and large language models. In Proceed- ings of the 40th International Conference on Machine Learning, ICML'23, JMLR.org. 2023.
[LLXH22] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language- image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12888- 12900. PMLR, 17-23 Jul 2022.
[LM25] Licong Lin and Song Mei. statistical theory of contrastive learning via approximate sufficient statistics, 2025.
[LS22] Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59:224 241, 2022. Special Issue on Harmonic Analysis and Machine Learning.
[LWY+17] 17] Weiyang Liu, Yandong Wen. Zhiding Yu, Ming Li, Bhiksha Raj. and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In 2017 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages
[LWYY16] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In Proceedings of the 33rd International Conference International Conference Machine Learning Volume 48, ICML'16 16. page 507-516. JMLR.org. 2016
[LXGY23] Hongye Liu, Xianhai Xie, Yang Gao, and Zhou Yu. Parameter-efficient transfer learning for audio-visualllanguae tasks. In Proceedings of the 31st ACM International Conference on Multimedia, MM '23, page 387-396, New York, NY, USA, 2023. Association for Computing Machinery.
13

---

[LZK Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Understanding the modality gap in modal contrastive representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
[LZS*23] Yiwei Lu, Guojun Zhang, Sun Sun, Hongyu Guo, and Yaoliang Yu. SIS-MICL: Understanding and generalizing infoNCE-based contrastive learning. Transactions on Machine Learning Research, 2023.
[MCCD13] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space, 2013.
[M] H. (Hermann) Minkowski. Geometrie der Zahlen. Teubner, Leipzig, 1910.
[MT21] Craig Macdonald and Nicola Tonellotto. On approximate nearest neighbour selection for multi-stage dense retrieval. In Proceedings the 30th ACM International Confer- ence on Information & Knowledge Management, CIKM '21, page 3318-3322, New York, NY, USA, 2021. Association for Computing Machinery.
[NF16] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, edi- tors, Computer Vision ECCV 2016, pages 69-84, Cham, 2016. Springer International Publishing.
[OLCM25] Kazusato Oko, Licong Lin, Yuhang Cai, and Song Mei. A statistical theory of con- trastive pre-training and multimodal generative ai, 2025.
[PCV24) Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference Machine Learning, ICML'24. JMLR.org, 2024.
[PHD20] Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 1 117(40):4652-244) 24663, 2020.
[PVZ15] Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In British Machine Vision Conference, 2015.
[Ran55] R. A. Rankin. The closest packing of spherical caps in n dimensions. Proceedings of the Glasgow Mathematical Association, 2(3):139-144) 1955.
[RAVF*22] 22] Vivek Ramanujan, Pavan Kumar Anasosalu Vasu, Ali Farhadi, Oncel Tuzel, and Hadi Pouransari. Forward compatible training for large-scale embedding retrieval systems. In 2022 IEEC/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19364-19373, 2022.
[RCS121] Joshua David Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Con- trastive learning with hard negative samples. In International Conference on Learning Representations, 2021.
[RKH+21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th Inter- national Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-8763. PMLR, 18-24 Jul 2021.
[RNP+22]
Elan Rosenfeld, Preetum Nakkiran, Hadi Pouransari, Oncel Tuzel, and Fartash Faghri. APE: Aligning pretrained encoders to quickly learn aligned multimodal representations. In Has it Trained Yet? NeurIPS 2022 Workshop, 2022.
14

---

[SBV+22] 22] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[Sha59] Claude E. Shannon. Probability of error for optimal codes in a gaussian channel. Bell System Technical Journal, 38(3):61 1-6666 1959.
[SKP15] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: unified embed- ding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
SPA+119 Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learn- ing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages
PMLR, 2019.
[SRC*21] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. Wit: Vikipedia-basee image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 2443 2449, New York, NY, USA, 2021. Association for Computing Machinery.
[Ste13] Ernst Steinitz. Bedingt konvergente reihen und konvexe systeme. Journal für die reine und angewandte Mathematik, 143: 443:128-11 1266176 1913.
[SXXS20] Yantao Shen. Yuanjun Xiong, Wei Xia, and Stefano Soatto. Towards backward- compatible representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6367-6376. 2020.
[TC Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunning- ham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Transformer Circuits Thread, 2024.
[25 Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Nacem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip Multilingual vision-lange saiiun- encoders with improved seman- tic understanding, localization, and dense features. arXiv preprint arXiv:2502. 14786. 2025.
[2GW+ Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Nacem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. Siglip 2: Multilingual vision-language encoders with improved semantic understand- ing, localization, and dense features, 2025.
[5]++55] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ra- mona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai Anton Tsitsulin, Robert Busa- Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, lain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorster Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas
15

---

Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Ben- debury. Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Char- CJ Carey, Cormac Brick, Daniel Deutsch,
line Le Lan, Christopher A. Choquette- Danielle Eisenbud, Dee Cattle, Derek Cheng. Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eu- gene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Pluccnn
Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hus- sein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vo- drahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy. Nathan Byrd, Nick Roy. Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Pöder. Sijal Bhatnagar, Sindhu Raghuram Panyam. Sivan Figer, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evei, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gle- icher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean- Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025.
[TKI20a]
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, pages 776-794, Cham, 2020. Springer International Publishing.
[TKI20b] Yonglong Tian, Dilip Krishnan, and Phillip Isola Contrastive representation distillation. In International Conference on Learning Representattios, 2020.
[vdOLV19] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.
[WBNL25] Orion Weller, Michael Boratko, Iftekhar Naim, and Jinhyuk Lee. On the theoretical limitations of embedding based retrieval. arXiv preprint arXiv:2508.21038 2025.
[WI20] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
[WXCY17] Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hyper- sphere embedding for face verification. In Proceedings of the 25th ACM International Conference on Multimedia, MM 17, page 1041-1049, New York, NY, USA, 2017. Association for Computing Machinery.
[WYH+22] Jianfeng Wang, Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang, GIT: A generative image-to-text transformer for vision and language. CoRR, 222.22255.141000 2022.
[Wyn68] A. D. Wyner. Communication of analog data from gaussian source over a noisy channel. Bell System Technical Journal, 47(5):801-812, 1968.
16

---

[XX Lee Xiong, Chenyan Xiong. Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Amold Overwijk. Approximate nearest neighbor negative con- trastive learning for dense text retrieval. In International Conference on Learning Representations, 2021.
[ZWX244] Lingxiao Yang. Ru- Yuan Zhang. Yanchen Wang. and Xiaohua Xie. Mma: Multi-modal adapter for
models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 23826-238377 June 2024.
[ZIE#16] 16] Richard Zhang, Phillip Isola, Alexei A. Efros, Nicu Sebe, Jiri Matas, Max Welling, and Bastian Leibe. Colorful image colorization. In Computer Vision ECCV 2016, volume 9907 of Lecture Notes in Computer Science, pages 649-666. Springer International Publishing AG, Switzerland, 2016.
[ZMKB23a] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Siglip demo experiments by. 2023.
[ZMKB23b] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 19911-11952, 2023.
[2WM+22) Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In 2022 /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 18102-18111 12, 2022.
IZZF* 22] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li. Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classifica- tion. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXV, page 493-510, Berlin, Heidelberg. 2022. Springer-Verlag.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In Theorems 3, and 3.2 we characterize fully the global minima of sigmoid loss with trainable temperature and bias. We address the modality gap in 3.6, the success on retrieval in Corollary I, the dimension recommendations Theorem 3.3, and practical recommendations based relative bias parameterization in Section 3.4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 4. We identify an point out both theoretical and empirical limitations and further directions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
18

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All proof' are written rigorously either immediately after the statements or referenced in the relevant appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The section Appendix D contains thorough description of all experiments in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes].
Justification:
In repo linked in the abstract. Represenationloomming/eerrrrryr
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
onoooicy))yy)) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. cc/ ctc/public/// httliic/ggii/////e/ee (edeee//odSuumiii Sissuubbssssss aassonPPlllo onnolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental results are explained in the relevant captions. More details is given in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte:://///u...//eeeeeeeee cc/ htte/uublcc/Ehiicccc ctCuideellleeeeeeeecccccc ines?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We are not aware of any negative or direct impacts on society of our work. The work can have indirect societal impact as the findings are relevant to modern large-scale machine learning systems.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode..ooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
23

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: httprrppsscc//oofeeeeeeeee////// for what should or should not be described.
24

---

A Omitted Proofs From Section 3.1
A.1 Global Minimizers The Sigmoid Loss are (m, Consteelntllsssn
We will repeatedly use the following fact which follows from log(1 + exp(x)) 2 0 for any R E R.
Observation For any {(V,, vi) and i, b, holds that
max log (1 11+exx exp(-t(U,, Vt) + b)) max log (1+ ([=exp(((U))))) (3) -b1) (1
max
(11)
iF)
max log (1 + exxep=((UUs Vi) + b)) max log (1 (1+exp(((U)) Vz) - -1)) (12) if
S N2 x max
Proof of Theorem 3.1. Suppose that lim,-++o ( (S) )N 1i=1: (V.(0) }=mi ((0) b(o)) 0 indeed
holds. By, (11), this means that
+ +b(1)) = vi, lim log (1 + e1xp(_<66) 1100) )
+oc
lim log [ (1+exp((6)) v((0) - -b(0)) =0 +oo
Vi jj.
Equivalently,
lim
(1())(())) v((0)) + ((o) 000 11,
+00
lim
(()) v(()) b(o) = 00
iA *
+00
Equivalently,
((a)
lim τ60 (1)(()) ((())
= +00
Vi, 00
+000
{(x)
b(t)
{(0) (()(+)) v((0)
lim
Vi
+000
t(a)
In particular, as (()) > 0 always, this means that for all large enough s, the quantity b(4) satisfies that
8(0)
rloj
(U p((a)) bba) 2 0
Vi,
(13)
(U(+), v(0)) ((o) 00
Vi + j.
(14)
However, as all (+) (( are unit vectors, (U ((()) Hence, for all large enough s, (13) and (14) imply that
S 1, ( vv(a) 2 -1 holds for any i, s.
-1 S b(d) 1.
Now, observe that (U(*), U/()) "o......,
U69) v(6) v(()
(+() b E (S4-1)82N [-1, 1]. As
rel
(8d-1)02N x [-1, is compact set, (U(0), u.(o) ..... U(a) v (()
..... N .66) J=z has
(1)
to)
convergent subsequence. Suppose that it converges to {U1, U2..... UN. V1, V2...., VN, b By (13) and (14), we have that
(Us, V2) brel 2 0
Vi,
(15)
(U,, V,) brel S 0
Vi サ
(16)
Setting
min min(U,, Vi) bret min bret (U, V3)) igf
gives the desired result.
25

---

Margin m
0.9
0.6
0.5
0.4
7/9
5/9
1/9
5/9
Relative Blus Bit
Figure 9: Achieved optimal relative bias and margin when training with fixed relative bias. The annotations correspond to the fixed relative bias.
Proof of Theorem 3.2. Suppose that {(U,, V:) are (m, bre)--Conttlllation with some m Then, for any it holds that
0.
{(UV, V,) brr) 2 m X Vi,
(17)
t(UU, V2) brel)
x Vi
(18)
In particular, by Observation (12), it follows that S ((U:) [Vi]i= (bre))
s 2 x log exp(-m t)
s N2 exp(-m x f) exp(-m x + 21og N/t) exp( m x + o())
which proves the convergence to zero. Choosing (m", bid) in this argument, where
1 m² ] (min Vi) max(U,, V2)) 1 (m + brel) (-m + brel)) m 0, 2
by 2 (mmin (U,, Vi) + max(U,, (U,, Vi)),
#
also proves that inf,, Sie (U (V, [i) S e --tm"++0 +o(t), All that is left to show is that
inf L ((U,) ltt,b) 2 +m(t)
Equivalently, we can show that inf, Sis ((U)) (K)) > log (1 e-tmm=+0( since log(1 +>) = + 0(7) as - 0. However, by (11), for the last inequality, is enough to show that
max (1(U), Vz) - :) 2 -t -txm* m² 153
max - t(U,. Vi) +bb)
max
Equivalently
min min (Uz, Vi) - b ) min ( t(U Vz) (-
+2) !) < m²
Suppose, for the sake of contradiction, that for some b, t, we have that
min min (U,, Vi) )))
t(U, t(U,., V,) + ;1) m' m
min ipf
Then,
b min Vi) max(U,, Vj) m' (6 -mmm) 2m' 2m"
which is a contradiction with the definition of m
26

---

Merg Exension
Figure 10: Evolution of margins when training with diffcrent fixed relative biases, average over 100 iterations.
A.2
Robustness of Nearest-Neighhor Retrieval: Proof of Proposition
Suppose that
E (ay)
Σ
log
11ep(-tt(Ua, Vog) + +b)
/=1
Σ log (1 (1 exxex(((UUa) Va,) - b)] S log 2. itfj
holds. Again,as the function
log(1 + e*) is non-negative, this implies that for some 0 < S E
B
Vag) + bi)]
xlog
IF
log
(a, (NNjKE
j=1
log 1 exp(-11U1]) V2) + +b))
IE B
However, whenever t(U,, Vi) b then log (1 exe(---t(11) Vi) + b)) 2 log 2. By Markov's inequality, it follows that
IP [eU,, V2) s - x
[n]]
B
Hence, for all but at most / fraction of the data indices j. it follows that (U,, Vÿ) b/t.
In the exact same way, for some y 2 such that x + y S &. it follows that
IP LU, ijj~MNj= [U,,, Vn) b 0] < B(B - 1)
Note that for every fixed if IP J [e V₁) b 0] 0, then IP (o [L Vy) > 0] 1/(N 1). Hence, one can similarly argue that for all
but most -(N-1) fraction of the data indices i, follows that (G), < b/t for all Thus, for least a
B(B-1)
VNN
B B(B 1)
fraction of the indices i, it follows that for any + i,
V.) b/t VU),
Clearly, for these indices, nearest neighbor search succeeds. Optimizing over 0 S x, g. S. we reach the conclusion.
A.3
Triplet Loss
In the context of synchronizing embeddings, the triplet loss function [SKP15] with hyperparameter margin takes form
L
Triplet (((U,, V)) = Σ ma v.1l3 ||, VVll2 0).
(19)
27

---

Observation 4. Suppose that {(U,, VoJ is a (m, brel) Connstellll,,,. Then, for any I 4m, it is also the case that
C T [Trilet(((U Vil 0.
Proof.
Suppose that {(U,, Vi is (m, brd) Consttllllll Then, for any i, i J. we have that
|U; Villă ||U; Vjllă
= 2 - 2,U,, Vi) - (2 2 Vi)) 2((U,, Vi)) (U,, Vi))
$ 2( 2 bre m brel) = -4m.
This finishes the proof.
B Proof of Theorem 3.5: Dimension vs Size tradeoff
Proof of Theorem 3.5. Let
~ Unil(S*-)),
C(H) = (i: (c,, H) 8), N' = IC((),
where G (U, + V22/2 and (0, 1) will be chosen later.
||= 1 (U4, Vi)
1+m- bu 1]. so a|||c/12 2 (11 + + b
2
Then the inner product satisfies
Pr[ H3 > 6) -Ta 5 Tall
(1
2 &v
I+m+b
where Ta(x) = Pr[H, > 2] is strictly decreasing in By linearity of expectation,
E[N' 2 (T ba
so there exists a realization of H with
N' 2 NT 8 I+m++b.l
(20)
Define for the index set C = C(H) the sums
Uc Σ
Un Ve = Σ
Vi,
44 = U, - V,,
DC =
Σ Ti
TEC
IEC
IEC
and set
Ac = E(U,, Vi), Bc Σ (U.V))
&c N
Σ
||23|13)
N7 x0 2c 0.
IEC
DEC
iE0
41
direct expansion shows
-cc) 2(N 22 Ac + Be.
(21)
On the other hand,
Σ KEC
UC + Voll 2 Σ (c, H) > N'5,
=
2
IEC
| + Vell2 > 222
(22)
Combining (21) and (22), and using Ag 2 (m + N' and Be < (brel (N - -1), yields
22 < 22 Ec) 2(N' - 2) (m + D + -4)(bba-m)) -1)]
28

---

Which means N' N3 (3m - -1+ 28² - -brel I 4m.
Where in the last reduction we dropped the & 2 0
term. Hence, whenever 20² > 1 -3m + brel-
4m
N' s
(23)
20 (1
3m + brel)
Combine (20) and (23) to get
4m
N 1
Te 6v
2
28t (1 3m + brel)
1+m+b
Recalling Shannon's asymptotic lower-bound I' (cos 0) exp sin og (d)) [Sha59, (ID)
with cos 2/ m bret) and corresponding sin 0 v1 cos
-bi the I+m+bal
bound is optimized by choosing
-ba This results in the claimed bound
2
N S exp( d sin 0 on (d))-
Bounds vs lb, rel = 0.01
Bouncs berel Im 0.1)
35
Lover Bound
Usper Round
3.0
Boued
0.0
3.00
0.5
0.10
0.15
0.29
0.25
0.30
0.6
0.5
02
0.6
Figure 11: Upper and Lower Bounds from Theorem 3.5 and Theorem 3.3 for fixed brel m 0.1.
C
Omitted Proofs from Section 3.3: Combinatories of The Modality Gap
Here, we analyze configurations {(V,, Vi) E (Sd-1 x gd-1)xN with the following property:
(U, Vi) 0 Vi,
(24)
(U; Vÿ) $j.
Ultimately, we aim to prove Theorem 3.6. We also prove several other facts on the way. Our proofs are based on simple facts from convex geometry which we introduce now. One can find more, for example, in the excellent book [BMS12).
C.1 Preliminaries from Convex Geometry
set K C Rdd is called conver if for any a [0, 1] and any p,q K, it is also the that ap + (1 - a)q K. In particular, for any points PI.P2....,PP. Rd the following two sets convex. The convex hull defined by
conv(pi, Pe..i,................... P..) = { (a1pi, 02p2 + ... + OAPP a 2
Vi and Σ 04 1
and the conver cone defined by
cone(p1. P2.... Pn)) = { (aipl, ... 0xPk a, 0
Vi and
a =
= 0).
29

---

We also introduce the dual cone. For set S S IR", the dual cone is given by
dualcone(S) = {or E R" (v, 20 0 Vx E S}.
We will the following classic theorems from convex geometry. Theorem C.I (Helly [Hel23]). Let X1, X2...,,, be finite collection of convex sets in R If the intersection of every d + 1 of these sets is nonempry, then the intersection of all the sets is nonempty. Formally,
If x for all C (1,2,...,, with [4] = + then n xx+ #00 0.
KEI
Theorem C.2 (Carathéodory [Carl Stel3)). Let A C Rd If E Comv(A), then there exists set B C such that [B] S d ++ and x E (B).
Theorem C.3 (Hyperplane Separation Theorem [Min10]). Let X and Y be two nonempty, disjoint convex sets in R Then there exists nonzero vector E R and a scalar b such that
(a, E for all x E X, (a. v) for all y Y.
C.2 Combinatorics of Modality Gap
Proposition 2. If (24) hold and N 2 d + 2, then there exists some gd-1 such that (h, U4) > for all i.
Proof. For each 1,.... N, define the open half-space
H; = E (U,,I) > 0}.
Each H, convex. We first show that any subcollection of + 1 of these half-spaces has nonempty intersection. Indeed, pick distinct indices ..,..., ii41; since N 2 d + 2, there is an index j g (i1,...,444l) By (24), for each h 11...,d=
(Use, Vi)) < <0, so (Ueu: -V₁)) >0,
and VV dd+1 Hi. # Ø.
Since every d + of the H, intersect and 2 + 2, Helly's theorem implics
N
H + Ø.
Choose any ho E n H4. Then (ho, UA) for all i. Setting h = ho/ e preserves these strict incqualities.
Propasition 3. If (24) hold and N 2 d d+2 2, then there exists some h E Rd such that (h, U2) > 0rror all and h E conv(U1, U2, UN).
Proof. Let C cOnv(U ...... Un) C RRd, compact convex set. Take the unit vector given by Proposition 2 and denote by H its projection onto C. This implies the fact that that the hyperplane
H = { gd (h 1h-N', h²) = 0}
supports the set C' at the point W/: all points of C' (hence each U;) lie in the closed half-space
H = EReR: (h- h', h1) < 0}. whereas h itself belongs to the opposite open half-space H+ = {x (h -1, h²) 0). Choose an orthonormal basis (e,,..., E2) with
(25)
h
61 =
11 h'
30

---

In these coordinates
N1' + 061, a: |h 11 >>0 0, = 0-e1 N11
while every U, has a decomposition U; = U2,1 e1 + Ui, with U1 O. For each
(K',U,) - U,) = (h' h, Ui) = -a 2 0.
Because (h, U2) > O, we conclude that
Uk' 0 (i (i=1, 1,..,..., N).
(26)
Proof of Theorem 3.6. Let h be the vector provided by Proposition 3, so h conv(U,..... UN) and (h, U.} > for every i. Set the cone
N
C' == cone (U...... UN) = Σ
a.U. az 2 2
1=1
Because each U; has positive dot product with h, define the affine hyperplane
f E Fd (x,h) = 1}. Every ray (AU: >> 0} meets I once, namely
U. E H. Qi = hU
Consequently
C' = conv Ox].
(27)
The the vector ht in H which is parallel to h. (h h) 1, so lit is h rescaled by a positive scalar. Since ht also lies in conv(U,,... UN) C C". we have ht E C' H. The hyperplane H is (d - di-dimensional. By Carathéodory's theorem in Rd-1, there is subset C {1,...,NN N} with |S| < d and weights A 2 0, Eies A 1, such that
U (U),
ht = Σ M Σn
(28)
iES
iES
Fix & S. Using Theorem C.2 and (24),
(U,, Vk) (U4, h)
(ht, Vk)
A
0,
because each numerator (U,, Vi² is negative and each denominator (U,, is positive. Hence (hl, Vi)) 0 for every k S. Only the (at most) d indices in S may give non-negative value. Note that ht is already the unit circle.
Now we prove that there is a construction for which this bound almost tight and we can separate all but least d vectors with hyperplane.
Construction (Tightness of Theorem 3.6). There exists a set of vectors {(U,, Vi) such that (U.. V.) > 0 for each i, (U,, Vi) < 0 for each * j, and for any h E gd-1 at least d - 1 values if holds that (h, U.) and (h, Vi) have the same sign.
Proof. For the construction for d = 3, note that we can take two parallel k-gons equally far from the equator of the sphere such that the zeniths of their points for one of them is 7/4 + 8 and for the other one is 3/4 - 5 and by taking 5 + 0 we can ensure that the dot product between corresponding pairs positive and the dot product between non-matching pairs negative. Now note that by taking sufficiently large and the 5 sufficiently small. The intersection of that configuration with the wedge with dihedral angle a contains least N points from one the k-gons (the U's) and N points from the other (the V's) for any value of a. See Fig. 12.
31

---

Figure 12: Construction 3D.
Label those
UI....,U UN-2 and Vi...., VN- Finally place more pairs the equator:
UN-I = VN-1. Un = VN
chosen so that (UN-1, Un) and both make negative dot-products with each wedge boundary ray. This completes the d = 3 example.
For the general case (d 3). Let wh.... wa+1 be the vertices of regular simplex in TR.. Set
U: V1 Wi, i= 1, d 3.
These occupy a (d (d-33 3) dimensional subspace. the orthogonal complement (which dimen- sional), embed the d = 3 construction, obtaining pairs (Ua- Va Va-2), (UN, VN). Finally, pick small & and renormalize:
U Eae22 V2 U,
V'=ewa-2 VI 22 V,, i= d .. N.
For sufficiently small, all the required dot-product signs are preserved, and since the configuration was orthogonal to U, for = 1, ........ d 3,
1 (Us Vi) = - < 0, = 1,2 1-,,,...d3... = d - 2, I,
as needed.
Proposition If (24) hold then U, & cone( (one((Ug))4s)) for i.
Proof. Suppose, to the contrary, that for some fixed there exist scalars aj 2 0 for j * i such that
U, = Σ U.
iti
Taking the inner product with Vi2 gives
(V.
Ev, iti
Since by (24) we have (V,, U)) < 0 for all j + and each a, 2 0. the right-hand side is nonpositive. But the left-hand side is strictly positive, contradiction. Therefore U, & cone({U,) (one({U,)iyi)) O
Proposition 5. Ifd and N > 4, there does not exist configuration satisfying (24).
32

---

Proof. Because N 2 d +++ 2, Proposition 3 gives unit vector h with (h, U,} > 0 for every i. Rotate
so h (1, 0): all U, now lie in the open right half- half-plane= O. Write their polar angles (- ) as
< 01 < <... Now note that U2 E CUne(U1, U3) which is a contradiction by Proposition 4. Therefore configuration (24) cannot exist when d = 2 and N 2 4.
ON < 2
D Further Experiments and Experimental Details
For the experiments in Appendix D.I, we used single A100 GPU. All other experiments are done on a standard CPU and take at most several minutes.
D.1 Experiments on ImageNet
In Figures and 3, we performed experiments on real data with the SigLIP implementation. While next gen- eration vision-language encoder was introduced with the SigLIP paper [TGW+ 2, we opted to use the original SigLIP model rather than SigLIP 2 because SigLIP 2's hanced training recipe- - incorporating auxiliary decoder, self-distillation, and masked- prediction masked-prediction losses would confound our ability to isolate the impact of the core Sig- moid Contrastive Loss the embeddings. The data we used is the validation dataset of ImageNet which contains 50000 captioned images with 1000 distinct captions. We used 8 trained models listed Table which can all be downloaded from Hugging Face.
Figure 13: "African chameleon" on the right and "American chameleon" the left from the ImageNet validation dataset. The B/16 model representation the image "Amer- ican chameleon" was closer to the represen- tation of the word African chameleon than that of American chameleon
We embedded all images and labels in the validation set using the B/16 model. We used PIL to resize all images to 224x24. Figure I, we show in red the inner products between wrong image-caption pairs and in blue between correct image-caption pairs.
As we point out, the inner product separation is nearly satisfied. There are some errors, but such are expected. For example, we discovered that the best matching image embeddings picture of the word "African chameleon" was "American chameleon". Both are species of chameleon and, hence, the images similar, such errors are to be expected in practice. For large models, the reported accuracy on ImageNet in [ZMKB23b] is 84.5%.
D.2 Experiments with Locked Representation
In Figure 4, performed experiments in which one modality is fixed. Namely, first draw (U uniformly on the sphere and then fix them. Then, we try to synchronize with (V,) by running gradient descent on the respective loss function. Specifically, we have experiments on:
Fixed Low Temperature = 200 and bias b = 0. We fix t = 200, b = 0 and run Adam on tt b) and initial learning rate 0.01.
(V:) for the loss (SU.) [V.).V
Fixed High Temperature = 10 and bias b = 0. We fix L = 10, b = and run Adam on 1, and initial learning rate 0.01.
(V:) for the loss (SU.( (V:).N
Trainable Temperature and Bias. We initialize
10 = et' (V.) f', b for the loss S ((U.), (V.) e" b) and initial learning rate 0.01. We note that all of our trainable experiments are with the parametrization e" which ensures positive temperature in [ZMKB23b].
and run Adam with on
Trainable Temperature and Relative Bias. We initialize at 10 et 0 and run Adam ,V. bre for the loss CRB- Siz ((U.) (V.) el bra)) and initial learning rate 0.01. We note that all of our trainable experiments are with the parametrization e' which ensures positive temperature in [ZMKB23b].
33

---

The specific experiment in Fig. 7 is for d = N = 100. We also note that we did one more comparison, which is not reported in the main paper with an explicit adapter from Figure Namely:
10 e" 0, Itet with 1/2 and run Adam on {V Drel. x for the loss 1188--e8((
Trainable Temperature and Relative Bias with Explicit Adapter. We initialize
CRB ( ([A (U,))== (Atrainable [Vo) e' brei) and initial learning rate 0,01. Since the adapter is an invertible transformation on the representations, we reported the inner products both with the adapter and without it (that is, invert by removing the last coordinate and dividing by 6.)
Relectve Bios And Adspte" Trainable But
Convergence to Zero Loss With Different Sigmoid Loss implementations
Fiedd - 200,
Reed 10,
Loss (log scale)
10-1
Bias Porametrization hemablle L
as Parametertzation Inirabll & By
Helarive
Adspter Removed
tretrativ Eut
Matching Non Matching
10
Relavve Blas Farametrizaoon
Bia Faramateriaation
Raad - 10 b-
Read - 200 and
Relative Bies Tenametraation
With Adapter
Iteration
Inner Product Va 02
Figure 14: Inner-product separation and loss convergence under six sigmoid- parameterizations. Left: Log-density histograms of inner-product scores for matching (blue) versus non-matching (red) pairs, evaluated under fixed inverse temperature b. our relative-bias parameterization (trainable bre)), and the same two schemes with the adapter removed; only the trainable-bing bias models show clear separation. Right: Sigmoid- loss trajectories (log scale) over 10,000 iterations for the same six settings; only those variants that learn both bias and inverse temperature reach zero loss, and our relative-bias parameterization (with and without adapter) converges most rapidly.
200, = O, fixed = 10, O, trainable bias
We can overall see that the performance of RRB-S algorithm with an adapter and without is rather comparable and the inner product separations similar. One difference is that the training with adapter seems less stable. Thus, we believe that in practice not using the adapter might be the better approach.
D.3 Experiments with Multiple Modalities
In Figure 8, we performed experiments with = ((T)) u((2 17(3 ( J==1 by running gradient descent on the sums of all pairwise loss func- tions between the modalities. Specifically, we have experiments on:
modalities. Namely, we synchronize
Fixed Low Temperature = 200 and bias b = 0. We fix t = 200, b = 0 and run Adam on ( for the loss (SU,) (Vi),N 1: b) and initial learning rate 0.01.
Fixed High Temperature (V for the loss ((U NV:),
10 and bias 6 0. We fix t 10, b and run Adam 1); 1, b and initial learning rate 0.01.
Trainable Temperature and Bias. We initialize at t 10 e 0 and run Adam with on [V.)N l', b for the loss [S (V)), e"" b) and initial learning rate 0.01. We which ensures
note that all of our trainable experiments are with the parametrization positive temperature as in [ZMKB23b].
Trainable Temperature and Relative Bias. We initialize at 10 = et 0 and run Adam NV t', bre for the loss CRB-SiE NV brei) and initial learning rate
34

---

0.01. We note that all of our trainable experiments are with the parametrization et which ensures positive temperature in [ZMKB23b].
The specific experiment in Fig. 8 is for d = 10, N = 100.
We ran additional experiments to investigate how increasing the number of modalities k affects the final separation margin. With trainable temperature and relative bias, we observe that the margin generally increases as we synchronize more modalities, as summarized in Table 1. This suggests that training with more modalities may lead to more robust representations, as larger margin implies better separation between matching and non-matching pairs.
Table 1: Final margin as function of the number of modalities being synchronized. The experiment was run with N 100 and d
10.
Number of Modalities
Final Margin
0.471241 0.427528
4
0.472571
8
0.595576
0.610853
20
0.61 1314
D.4 Bias Parameterization Leads to Zero Relative Bias
Finally, do experiments to show that training with S leads to in [ZMKB23a]. We compare with CRB-Sig Sig Concretely, we run experiments with N 100 points {U, Vi)l initialized at random and run Adam ([ Vi) ii b), respectively _RB Sig (((U,, Vi) brel), for 10000 epochs starting at 10 and varying biases.
relative bias, as
We compare the evolution relative biases, inverse temperature, loss function, and margins of the final configuration.
Temperature Fvolution
Inttal
Blas
Intall Rixs
Inttll Blas
Intall Baa
Figure 15: Evolution of the inverse temperature parameter during the training process.
Relative Bias Evolution
Initel Bas
Inttal Blas -
Initlal Bas
118 Sias
Inft Blas - 10
30
80
Relative bias is in general smaller when training with S parameterization. In general, converges to zero and is significantly smaller than the relative bias of the RBB Sis
Figure 16:
Finally, we also compare the margins. The reason is that as we know from Theorem 3.4, there is an important relationship between relative bias and margin. The fact that embeddings trained with CRB-Sig have larger relative bias also impacts the margin.
35

---

Loss Evolution
Inttal Blas
Intlal
Intea Bls
Intell Blus
Intial Blas
Figure 17: In general, the loss converges faster to zero when trained with the LRB-Sit parameterization than when trained with L
Final Margins
Bias Parameterization
Relative Bias Parameterization
Final Margin
2250
3226-
1350
50
Initial Bias
Figure 18: general, margin much larger representations trained with the CRB-Sic parameterization. As we know from 1, this means that they are more robust on retrieval tasks.
D.5 Initializing Fixed Relative Bias
We verify this with experiment where we initialize representations uniformly random, fix the relative bias bret. and train the representations and inverse temperature / using Adam. As shown in Table 2, choosing beel 0.7 yields the largest final margin, while other choices result in smaller margins. This confirms that the relative bias parameter can effectively steer the optimization towards configurations with desirable properties.
D.6 Initializing Learnable Temperature and Relative Bias
We investigated the effect of initial temperature and relative bias bral the final margin. We ran hyperparameter search and found that the final margin is best for small initial temperature (t < 3) intermediate temperature
10) with a relatively large initial relative bias (b,at 0.6). The results, summarized in Table 3, show that while the optimization is robust to a range of imitializations, poor choice (c.g., high initial and low bye)) can lead to suboptimal final representations with small or even negative margin.
E
Connection to Linear Representation Hypothesis Across Modalities
has been observed by many authors that modern dense embedding spaces acquire correspondence between linear-algebrate operations and real-world concepts. This has been immortalized as "King Man + Woman Queen" word2vec [MCCD13] and is also observed in modern LLMs well [PCV24, TCM+24 Curiously, we find that contrastive pretraining with sigmoid loss also leads to special case of LRH: there emerges a direction such that adding recovers the embedding of a matching text caption. Indeed, looking at the optimal embeddings in Fig. we can see that U, - V; does not depend on which we take as manifestation of LRH in this context (the concept being "shift text to image" or more generally one modality to another). Furthermore, both our upper and lower bounds on the cardinality of the embeddings Section 3.2 require Cross-Moo Mooos-Modality-LAn -LRH satisfying configurations to be tight.
an image embedding (almost)
36

---

Table 2: Final margin and loss for different fixed values of relative bias b Training performed on the representations and inverse temperature t. The largest margin is achieved for byel = 0.7.
Fixed Relative Bias
Final Temperature Achieved Margin
Final Loss
-1.00
6.961601
-0.000001
0.693150
-0.90
56.188858
-0.000000
0.009245
-0.80
23.300198
-0.000000
0,014105
-0.70
162.664841
0.092437
0.000005
-0.60
125.656731
0.122326
0.000003
-0.50
104.095329
0.152893
0.000003
-0.40
90.788622
0.182992
0,000002
-0.30
81.079422
0.213618
0.00000
-0.20
75.061546
0.242438
0.00000
-0.10
71.254242
0.273600
0.000000
0.00
69.018265
0.301340
0,000000
0.10
69.534515
0.329022
0.000000
0.20
67.996437
0.353406
0,000000
0.30
61.796310
0.390087
0.000000
0.40
55.452553
0.430921
0.000000
0.50
48.406261
0.466707
0.000000
0.60
44.391388
0.498564
0.000000
0.70
42.265457
0.527834
0.000000
0.80
37.361767
0.539749
0.000001
0.90
33. 33.167175
0.483351
0.000036
1.00
23.817665
0.513416
0.000693
Table 3: The final margin achieved for different initializations of temperature (Temp) and relative bias (brat). The best results (bolded) are obtained with low-to-intermediaat temperature and high relative bias.
Temp
-1.0
-0.8
-0.6
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.567
0.567
0.566
0.564
0.566
0.566
0.565
0.568
0.570
0.574
0.573
3
0.545
0.543
0.526
0.499
0.483
0.488
0.536
0.563
0.570
0.574
0.573
10
0.439
0.425
0.415
0.406
0.402
0.410
0.429
0.524
0.566
0.547
0.562
30
0.301
0.297
0.294
0.315
0.343
-0.942
-0.915
-0.935
-1.171
-1.051
-1.145
100
-0.774
-0.679
-0.483
-0.740
-0,878
-0.956
-0.978
-1.109
-1.186
-1.880
-1.499
In the proof of Theorem 3.5 in Appendix we defined the following quantity characterizing an arbitrary constellation
{ = Σ [E.11))
1 N
| 2 0.
where x; U, and a # (In the proof was defined for carefully chosen sub- constellation). We note that the upper bound in that Theorem 3.5 could only possibly be tight if & In this section we will further show that & can be used quantitative measure of the degree to which Linear Representation Hypothesis is satisfied.
First, let establish that when is small (as & 2 is used in the proof of Theorem 3.5 this means that the bounds that proof are tight). More importantly, a small & implies something significant about our representations: suggests that the difference vector, U, V,, is nearly identical for all indices i. Think of it this way: if you have two sets learned representations, say U; for images and V, for their corresponding text descriptions, small & means that you can apply consistent shift (a single vector) to all the V; vectors to transform them into their corresponding U; vectors. So, by shifting text representation, you could get its corresponding image representation.
i1I2 o(1) and all puirs of representations align the sense that U, V, UU- U,- V; for all i,j. In particular, the U are obtained from the V's by adding a vector Σ, which thus serves concept shift.
Proposition 6. !f = o(1) then I
& EN |
37

---

Proof. A simple algebra shows that & has the following two equivalent expressions:
| x
2N
1,j=1
Thus, the statement "5 = o(1)" is equivalent to
0 S
I. ø1 0.
N
1=1
which in ur implies that U, V, se T simultaneously for all i. The argument for U; - V2 Rd Uj similar.
Corollary 2. =f 0, then U: V, are all identical for all indices
Indeed, in Fig. 19 when training directly the representations, we can observe the & value converging to for range of different dimensions.
Evolution over iterations 20 different runs with N 1100, d
Evolution over iterations
Evolution É over iterations
20 different runs with =100, d 20
20 different runs with =100, d 100
Log-10 of
Log Log-10 of
33
Log-10 of
355
60
183
Iteration
Iteration
Iteration
Convergence of the value of & to zero during an experiment with d = 10 and 100 U, V; pairs trained with SigLIP.
Figure 19:
However, the value of for real models is far from 0 on the ImageNet validation dataset. Our intuition for this is that the dimension used zv 1000 is far from optimal, hence & = O not required. is interesting open direction whether g + O in that case. For example, one can explicitly add & in the loss function.
train models in lower dimension utilizing the fact that
Table 4: & for different SigLIP models in ImageNet validation. We plot respectively &, & Ex ||::13))
as mean of norms, | norm of mean, 1 EΣ ||U V= random mean of is a uniformly random permutation. We can see that the mean of norms is closer to the random mean of norms (i.e., random pairing of text-images, not corresponding the ground truth) rather than the norm of means, which would imply & - 0.
where
Model
g
Mean Norms Norm of Mean Random Mean Norms
higlip-so440m-patchh44-3333
0.6086
1.7249
1.1162
2.0029
geelip-base-patchl-22
0,5880
1,8100
1.2221
2.0609
Seeli--bse-patcch6--1 6-384
0.5908
1.8068
1.2160
2.0631
segii--large-pacch16 6-256
0.5535
1.7955
1.2420
2.0711
1atpp--o4400-pptchhh 14-224
0.6207
1.7270
1.1063
2.0038
ggllip-base-pathhee
6-266 6-256
0.5767
1.7991
1.2225
2.0588
Saapp-bba--ppach16--111
0.5908
1,8059
1.2151
2.0644
sigliplaarge-pacch16-38.
0.5744
1.8084
1.2340
2.0762
38

---

Table 5: Margin and Relative bias corresponding to 5th-percentile positive and 95-th percentile negative pairs for different SigLIP models. We highlight that the two largest so400m models have a substantially different relative bias than the rest of the models. Likewise, the margin is perfectly correlated with the embedding dimension bigger models have bigger margin.
Model
5% Positive Pairs 95% Negative Pairs Margin
Relative Bias
Dimension
bglii--so400m-pptchhh 14-384
14-334
0.0769
0.0486
0.0142
0.0627
1152
gggii-soo400m-patchh-222200 14-224
0.0747
0.0483
0.0132
0.0615
1152
Siglip-large-patchh6--555 1622h
0.0400
0.0151
0.0124
0.0276
1024
siglip-large-pucchl6-BB.
0.0353
0,0120
0.0117
0.0237
1024
siglip-base-patch16--111
0.0409
0.0170
0.0120
0.0289
768
dglli--bse-ppacch665 1--34
0.0408
0.0173
0.0118
0.0290
768
siglip-base
caap--aas-pacchhlleeeee p-tch11-2566
0,0413
0,0200
0.0106
0.0306
768
siglip-base patch1 2-24
0.0383
0.0181
0.0101
0.0282
768
Table 6: Mean cosine similarities, margin, and relative bias for different SigLIP models. It is interesting to consider that the so400m are exactly on the boundary where the modality gap is guaranteed (all 8 models do satisfy the modality gap with zero misclassification error). The difference in margins is partly explained by dimensionality - larger dimensions correspond to larger margins, The Pearson correlation coefficient between dimension and margin .948 and the Spearman coefficient is .926.
Model
Mean Pairs Mean Neg, Pairs Margin Relative Bias Dimension
diglip-so400m> 1a::000mm-pachh---00
0.1376
-0.0015
0.0695
0.0680
1152
siglin-so0400m--patch4433
0.1365
-0.0022
0.0694
0.0672
1152
siglip-large-pacch16-55 6-256 6-226
0.1023
-0.0359
0.0691
0.0332
1024
siglip-large-patch hetpacchh11668.. 6-384
0.0958
-0.0384
0.0671
0.0287
1024
siglip-base-paan maase-patch16-2555
0.1004
-0.0294
0.0649
0.0355
768
hglip-bbse-patte
satii-bbase-ptch1--5112 patch16-512
0.0971
-0.0322
0.0646
0.0324
siglip-base hatp--baasppchhh--oe
0.0966
-0.0319
0.0642
0.0324
768
siglip- base-pacch1162222
0.0950
-0.0305
0.0627
0.0322
768
39


Reviews:
[{"review_id": "F07blCq0Bj", "reviewer": "Reviewer_HUkA", "summary": "This paper provides a rigorous theoretical analysis of the sigmoid contrastive loss, a key component in modern large-scale vision-language models like Google's SigLIP. The authors identify a gap between the practical success of these models and the theoretical understanding of their underlying mechanisms, particularly in the realistic regime where the number of training examples (N) far exceeds the embedding dimension (d) but is not infinite.\nThe paper characterize the global minima of the sigmoid using (m, b_rel)-costellations, geometrical comfigurations which correspon to the spherical codes on a hypersphere.\nSpecifically, the Relative Bias Parameterization is proposed to address the problem of the optimization dynamics which naturally force the effective relative bias (b/t) to zero as t grows large. \nUsing this framework, the paper offers theoretical explanations for the modality gap phenomenon and sunchronization of multiple modalities.", "questions": "1. To facilitate the adoption of L_RB-Sig, could you provide practical guidance on initializing the learnable parameters? Specifically, what initial values for t and b_rel would you recommend for stable large-scale training? \n2. For the case of a fixed b_rel, how sensitive is the model's performance to this choice? What is the impact of choosing a suboptimal value compared to both the learned b_rel and the standard L_Sig baseline?\n3. Your theory is based on ideal zero-loss configurations. How do you expect the geometric properties of the (m, b_rel)-constellations to behave in a realistic, non-zero loss regime typical of large-scale training?", "limitations": "Yes, in Sec 4.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n- The paper's greatest strength is its formal approach. By introducing concepts from geometry and coding theory like (m, b_rel)-constellations and spherical codes, it brings mathematical precision to a field often driven by empirical results. This provides a solid foundation for understanding why these models work.\n- The work is not just abstract theory. It is directly motivated by and analyzes state-of-the-art models (SigLIP). It successfully explains real-world phenomena like the modality gap, which had previously been an empirical observation without a strong theoretical justification.\n- The proposed L_RB-Sig is a simple, practical, and elegant solution to the identified problem. It's a minor, well-justified change to the loss function that has significant theoretical and practical benefits. The paper also provides clear guidance on how to use it (either learning b_rel or freezing it to guide the learned geometry of the space).\n\n\n\n**Weaknesses**:\n\nWhile theoretically strong, the paper's practical impact is limited by several key weaknesses. It often priorites the ``what'' over the ``why'' letting to the reader the effort to concretically understand the meaning of the theory:\n\n- The paper overlooks the related field of backward-compatible representation learning (eg [1,2,3,4]) is missing in the synchronization of representations even though the aim is the same\n- The theoretical analysis primarily characterizes the geometry of \"global minimizers,\" which are configurations that achieve zero loss. In practice, training on massive, noisy web-scale data never results in zero loss. While the principles likely still hold, the analysis doesn't fully address how these geometric structures are affected in the more realistic, non-zero loss regime. Are the constellations easy to constryct also in this case? \n- The paper concludes that the larger margins achieved by L_RB-Sig mean the representations are \"more robust on retrieval tasks.\" This is a well-accepted and highly plausible heuristic in the field, but without a direct evaluation of a downstream retrieval task, this remains an assumed (though likely correct) consequence rather than a demonstrated fact of the paper.\n- Adapters are proposed and never evaluated. Furthermore, a comparison with a baseline with a standard trainable adapter (e.g. [3,4,5]) should be included.\n- The paper lacks crucial implementation details that would help practitioners adopt its findings. For example, it provides no pseudocode or guidance on how to initialize the learnable t and b_rel parameters, which is critical for stable training.\n\n[1] Shen, Y., Xiong, Y., Xia, W., & Soatto, S. (2020). Towards backward-compatible representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6368-6377). \n\n[2] Biondi, N., Pernici, F., Bruni, M., & Del Bimbo, A. (2023). Cores: Compatible representations via stationarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8), 9567-9582.\n\n[3] Ramanujan, V., Vasu, P. K. A., Farhadi, A., Tuzel, O., & Pouransari, H. (2022). Forward compatible training for large-scale embedding retrieval systems. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19386-19395).\n\n[4] Jaeckle, F., Faghri, F., Farhadi, A., Tuzel, O., & Pouransari, H. (2023). Fastfill: Efficient compatible model update. ICLR2023.\n\n[5] Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Locatello, F., & Rodolà, E. (2022). Relative representations enable zero-shot latent space communication. ICLR2023.", "was_revised": true, "final_justification": "The authors have clarified all my points and considering also the other reviews I suggest the acceptance for this paper.", "rebuttal": ">**Q1:** To facilitate the adoption of $\\mathcal{L}^{RB-Sig}$, could you provide practical guidance on initializing the learnable parameters? Specifically, what initial values for t and $b_\\text{rel}$ would you recommend for stable large-scale training?\n\n**A:** Our theoretical analysis currently does not yield any initialization recommendations. In experiments with synthetic data (where we optimize directly over the representations rather than a given network), it seems like the initial choice of $b_\\text{rel}$ makes little difference when it is trainable. \n\nWe point out that in the original SigLIP paper ``Sigmoid Loss for Language Image Pre-Training'' by Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer, there is a recommendation for $t = 10, b = -10$ (corresponding to $b_\\text{rel} = -1$). When using these initial values for our experiment with trainable relative bias we always converge to $0$ loss and stable training. The setting we used is $N=1024$ and $d=20$ using Adam with both $t$ and $b_\\text{rel}$ trainable with the representations initialized i.i.d. uniformly on the sphere. We ran a hyperparameter search to find the best initialization for $t \\in \\{ 1,3,10,30,100\\}$ and $b_\\text{rel} \\in \\{ -1.0, -0.8, \\dots, 0.8, 1.0\\}$ and found that in general the best representations (measured by the final margin $\\min_{i} (U_i, V_i) - \\max_{i\\neq j}(U_i, V_j)$) is achieved for small $t  \\lesssim 3$ or intermediate $t\\approx 10$ and  $b_\\text{rel} \\gtrsim 0.6$. In those settings the final relative bias was $\\approx 0.7$. \n\nA possible intuition justifying starting with small $t$ is that the representations can move relatively freely when $t$ is small, and only once they arrive at a configuration attaining small loss does the temperature parameter start to increase, further driving down the loss.\n\nThe margins of final configuration are in the following table, with rows indexed by initial t and columns by initial $b_\\text{rel}$:\n\\\\begin{array}{c|rrrrrrrrrrr}\n\\\\text{Temp} & -1.0 & -0.8 & -0.6 & -0.4 & -0.2 & 0.0 & 0.2 & 0.4 & 0.6 & 0.8 & 1.0 \\\\\\\\\n\\\\hline\n1   & 0.567 & 0.567 & 0.566 & 0.564 & 0.566 & 0.566 & 0.565 & 0.568 & 0.570 & \\\\mathbf{0.574} & 0.573 \\\\\\\\\n3   & 0.545 & 0.543 & 0.526 & 0.499 & 0.483 & 0.488 & 0.536 & 0.563 & 0.570 & 0.573 & \\\\mathbf{0.574} \\\\\\\\\n10  & 0.439 & 0.425 & 0.415 & 0.406 & 0.402 & 0.410 & 0.429 & 0.524 & 0.566 & 0.547 & 0.562 \\\\\\\\\n30  & 0.301 & 0.297 & 0.294 & 0.315 & 0.343 & -0.942 & -0.915 & -0.935 & -1.171 & -1.051 & -1.145 \\\\\\\\\n100 & -0.774 & -0.679 & -0.483 & -0.740 & -0.878 & -0.956 & -0.978 & -1.109 & -1.186 & -1.080 & -1.449 \\\\\\\\\n\\\\end{array}\n\n>**Q2:** For the case of a fixed $b_\\text{rel}$, how sensitive is the model's performance to this choice? What is the impact of choosing a suboptimal value compared to both the learned $b_\\text{rel}$ and the standard $L^{Sig}$ baseline?\n\n**A:** \nIn our experiments given below in Table 2, we investigated the impact of the fixed value of $b_\\text{rel}$. We initialized the $N=1024$ points i.i.d. uniformly at random on the sphere in $d=20$ dimensions, and for the initial temperature we chose $t = 10$. Runs were stopped whenever gradients of all representations dropped below $1e-10$ or after $10k$ epochs.  We see that a range of $b_\\text{rel}$ between $-0.2$ and $0.7$ yield nearly zero loss when training stabilizes. The size of the margin does, however, seem to depend fairly significantly on the value of $b_\\text{rel}$, as seen in the table. \n\nChoosing $b_\\text{rel}\\approx 0.7$ yields the largest final margin, and this matches the final value for trainable $b_\\text{rel}$ in the experiments described above in response to Q1. Other choices for fixed $b_\\text{rel}$ result in smaller final margins (and in the extremes, larger final loss) as indicated in the following table. \n\n**Table 2**: Rows: Fixed Relative Bias\n\n| Fixed Relative Bias | Final Temperature | Achieved Margin | Final Loss |\n|---------------------|-------------------|------------------|-------------|\n| -1.000000           | 6.961601          | -0.000001        | 0.693150    |\n| -0.900000           | 56.188858         | -0.000000        | 0.009245    |\n| -0.800000           | 23.300198         | -0.000000        | 0.014105    |\n| -0.700000           | 162.664841        | 0.092437         | 0.000005    |\n| -0.600000           | 125.656731        | 0.122326         | 0.000003    |\n| -0.500000           | 104.095329        | 0.152893         | 0.000003    |\n| -0.400000           | 90.788620         | 0.182992         | 0.000002    |\n| -0.300000           | 81.079422         | 0.213618         | 0.000001    |\n| -0.200000           | 75.061546         | 0.242438         | 0.000001    |\n| -0.100000           | 71.254242         | 0.273600         | 0.000000    |\n| 0.000000            | 69.018265         | 0.301340         | 0.000000    |\n| 0.100000            | 69.534515         | 0.329022         | 0.000000    |\n| 0.200000            | 67.996437         | 0.353406         | 0.000000    |\n| 0.300000            | 61.796310         | 0.390087         | 0.000000    |\n| 0.400000            | 55.452553         | 0.430921         | 0.000000    |\n| 0.500000            | 48.406261         | 0.466707         | 0.000000    |\n| 0.600000            | 44.391388         | 0.498564         | 0.000000    |\n| 0.700000            | 42.265457         | 0.527834         | 0.000000    |\n| 0.800000            | 37.361767         | 0.539749         | 0.000001    |\n| 0.900000            | 33.167175         | 0.483351         | 0.000036    |\n| 1.000000            | 23.817665         | 0.513416         | 0.000693    |\n\n>**Q3:** Your theory is based on ideal zero-loss configurations. How do you expect the geometric properties of the $(m, b_\\text{rel})$-constellations to behave in a realistic, non-zero loss regime typical of large-scale training?\n\n**A:** We find that representations of practically trained models  (at non-infinite temperature) share multiple key geometric aspects with zero loss configurations at infinite temperature parameter. We elaborate on this in a moment.\n\nWe first remark that while our main result is for zero-loss configurations in the limit of infinite temperature, our experiments (e.g. in Table 2 above) do result in nearly zero loss. Additionally, practically trained models end up with very high temperatures: e.g., in trained SigLIP models by Google Research, the final temperature parameter $t$ is on the order of $100$. \nBecause this is in the exponent of the loss, the resulting configurations are well-approximated by infinite temperature zero-loss configurations: pairs $\\langle U_i, V_j\\rangle$ leading to non-zero loss at infinite temperature will have a large contribution also for $t=100$. \n\n\nWe next discuss how Figures 1 and 3 illustrate that representations of practically trained models share key geometric aspects with zero loss configurations.\nFigure 1 shows that a trained SigLIP model yields  near perfect separability between inner products of correctly matched pairs and inner products of mismatched pairs. That is, \n$\\langle U_i, V_i \\rangle \\ge \\langle U_{i'}, V_{j'}\\rangle$ holds for almost all $i, i'\\neq j',$ which is exactly the definition of constellations (with $m = (\\min_i \\langle U_i, V_i\\rangle - \\max_{i\\neq j} \\langle U_i, V_j\\rangle)/2$ and \n$b_\\text{rel} = (\\min_i \\langle U_i, V_i\\rangle + \\max_{i\\neq j} \\langle U_i, V_j\\rangle)/2$). This is one crucial aspect in which the geometry of actual representations resembles the geometry of zero-loss configurations. \n\nFigure 3 shows that trained SigLIP models exhibit the modality gap, a property shared by zero-loss configurations as described in Theorem 3.5.\n\nThus, these figures and experiments demonstrate that our theoretical insights indeed carry over to practically trained models.\n\n\n>**W1 and W3 (paraphrased):** The paper overlooks previous work on backward-compatible representation learning. \n\n**A:** Thank you for pointing out the line of work on  backward-compatible representation learning. This direction is highly relevant to our paper and aligns with the broader set of applications we mention, such as modality synchronization and distillation. We have added the appropriate references to acknowledge this connection. \n\nBriefly, our work fits into that literature as follows. Our primary focus is not on benchmarking specific applications of contrastive learning. Instead, our work aims for a theoretical understanding of phenomena appearing from training with the sigmoid contrastive loss. As far as we are aware, such theoretical analysis does not exist for any contrastive learning objective in the $N\\gg d$ regime of interest. We hope to offer practitioners tools, such as the relative bias parameter, to modulate their behavior. We view backward-compatible representation learning as a particularly promising application of our findings and an exciting avenue for future research.\n\n\n>**W2:** The theoretical analysis primarily characterizes the geometry of ``global minimizers,\" which are configurations that achieve zero loss. In practice, training on massive, noisy web-scale data never results in zero loss. While the principles likely still hold, the analysis doesn't fully address how these geometric structures are affected in the more realistic, non-zero loss regime. Are the constellations easy to construct also in this case?\n\n**A:** Please refer to the answer of Q3."}, {"review_id": "DamThUUXOA", "reviewer": "Reviewer_HL7g", "summary": "This work theoretically analyzes global minimizers of the Sigmoid loss used to train SigLIP models, and find that resulting embeddings follow a structure termed (m,b_rel)-constellation, which achieves  perfect retrieval and is robust to approximation errors when using approximate nearest neighbor search as the constellation impose a margin between matching and non-matching pairs. \n\nThe paper also proves that the modality gap exist in global minimizers and thus show that modality gap does not prevent perfect retrieval.  Based on the theoretical results, the paper proposes a new parametrization of the Sigmoid loss yielding faster training convergence", "questions": "Please refer to Strengths And Weaknesses", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well written and insightful. Previous theoretical works only considered cases where embedding dimension d is higher than number of points N, or number of points N goes to infinity, both cases are impractical, whereas here they consider a more practical setup  d << N << 2^d. Important for practitioners is measuring the evolution of margin during training  as larger margin means more robustness to errors in approximate nearest neighbor search.\n\nThe relative bias parametrization is only shown to be faster convergence than original Sigmoid loss on a toy example with N=100 and d=10 and a single run with fixed hyper-parameters, which makes the conclusion limited. How does it behave on a real world example ? and does the faster convergence also translate to better performance on downstream tasks (e.g., zero-shot classification) ?", "was_revised": true, "final_justification": "The paper is technically solid and the rebuttals/discussions answered my questions.", "rebuttal": "> **Q:** The relative bias parametrization is only shown to be faster convergence than original Sigmoid loss on a toy example with N=100 and d=10 and a single run with fixed hyper-parameters, which makes the conclusion limited. How does it behave on a real world example ? and does the faster convergence also translate to better performance on downstream tasks (e.g., zero-shot classification) ?\n\n**A:** To address the question about scaling, we ran a larger experiment with $N = 50,000$ synthetic examples ($10^5$ data points total) and dimension $d = 100$ the results are qualitatively the same as the ones described in the paper which is good evidence that the empirical results are scalable.\n\nBeyond our empirical results concerning relative bias, we emphasize that our main contributions are to carry out theoretical analysis characterizing the minimizers of sigmoid contrastive loss in the practically relevant parameter regime. Prior to our work it was not at all clear what sorts of configurations this loss seeks. We further determined a number of interesting properties both of theoretical and practical interest of the minimizing configurations, including modality gap and perfect retrieval via 1-top nearest neighbor search.\n\nWhile we have not performed experiments confirming this hypothesis, based on prior work on representation synchronization (albeit with different contrastive loss functions), it is to be expected that larger margins, which are achieved by the relative bias parameterization, lead to better performance in various downstream tasks. For example:\n\n1.  \"Understanding Contrastive Representation Learning through\nAlignment and Uniformity on the Hypersphere\" by Tongzhou Wang and Phillip Isola (e.g. Tables 1 an 2) show how directly optimizing the representation properties of alignment (which corresponds to a smaller modality gap) and uniformity (which corresponds to higher margin because higher margin means more uniform packing of the sphere with caps of that margin) lead to better downstream performance on classification (CIFAR-10 and STL-10) and indoor scene understanding (NYU-Depth-V2).\n\n2. \"ArcFace: Additive Angular Margin Loss for Deep Face Recognition\" which shows that directly optimizing the margin in the representations leads to better downstream performance on face recognition tasks.\n\n3. \"CoReS: Compatible Representations via Stationarity\" which was pointed out to us by Reviewer HUkA also uses the heuristic that larger margin means better performance by embedding the classes to a regular polytope. While their work requires the dimension of the representation space to be very high (one lower than the number of classes) it still shows us that larger margins lead to better performance."}, {"review_id": "MBVayRIE46", "reviewer": "Reviewer_szUN", "summary": "The paper mathematically formalized the synchronization of two representations, mainly discussing when and how the sigmoid-based contrastive objective can drive its loss to zero. Specifically, this paper introduces a new combinatorial structure, the (m,b_{rel}​)-Constellations to show that SigLIP-style losses admit zero-loss solutions precisely when the representations come from one of these constellations. From there they draw out practical implications for retrieval performance, the “modality gap,” embedding dimension requirements, and even propose a cleaner reparameterization of the sigmoid loss.", "questions": "Q1: How does “synchronization” differ from “alignment” in latent clustering, and why do synchronization embeddings occupy disjoint regions that will hamper the model performance?\n\n\nQ2: In the proof of Theorem 3.1, why from 660 to 661, the -\\infty becomes the +\\infty in the second term when just retrieving t(s)?\n\n\nQ3: Theorem 3.2 identifies $(m,b_{\\rm rel})$-Constellations as all global minima in the t \\to\\infty, but real training uses finite tt and stochastic optimization. Under finite temperature and sampling noise, do approximate $(m,b_{\\rm rel})$-Constellations still govern near-optimal minima, and can we quantify their attraction basins?\n\nQ4: In section 3.4, as the number of modalities k grows or the synchronization graph sparsifiers, how does learning relative biases impact convergence speed and final separation?\n\n\nQ5: Do you have any performance guarantee for your method in the paper? Like when we set up the hyperparameters which arrive at the upper bound or lower bound as your paper mentioned? Do you have any valid metrics to show the global minimizers improve the embedding?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "For strengths across the dimensions of originality, quality, clarity, and significance:\n\n\n\n\n\nQuality:\n\n\n\n\nTheoretical insights are rigorously developed, providing a solid foundation for the proposed connections and methodologies. \n\n\nClarity:\n\n\n\n\nThe manuscript is well structured, with definitions and proofs presented in a logical progression. However, the duplication of “Background” material in both the Introduction and Section 2 could be streamlined to avoid redundancy and sharpen the narrative flow.\n\n\nSignificance:\n\n\n\n\nWhile the work delivers a compelling geometric explanation for tuning contrastive-loss hyperparameters and formalizes the roots of the “modality gap,” its practical impact hinges on demonstrating how these insights translate into improved performance or design guidelines for real-world multi-modal systems. Clarifying this bridge from theory to real system gains would greatly strengthen the paper’s broader relevance.\n\n\nOriginality:\n\n\n\n\nThe paper presents an innovative setup involved the combinatorial characterization of all zero-loss solutions for the sigmoid contrastive objective in the N≫d regime, and showing that these constellations exactly capture when and how one can drive the loss to zero by tuning temperature and bias.\n\n\n\nWakeness:\n\n\nW1: Although the authors frame their work as a new theory of contrastive losses, the core result—characterizing minimizers of InfoNCE and sigmoid objectives through temperatures and bias—overlaps heavily with existing analyses. It isn’t clear what truly novel insight this paper adds beyond re-deriving known minimizer behavior in a different combinatorial language.\n\n\nW2:  ImageNet experiments in this paper mainly for visualization of the modality gap through inner‐product histograms, but they are short of measuring end‐to‐end retrieval performance (e.g. top-1/top-5 accuracy) on large-scale benchmarks. It remains unclear whether the theory actually informs practical model design or improvement.\n\n\n\nW3: The paper repeatedly invokes the N≫d setting, yet fails to explain why that specific regime generates a modality gap in practice, or why it differs fundamentally from previous high-dimensional results. As a result, the significance of both the regime and the modality gap itself feels under-justified.\n\n\n\nW4: While the paper’s ablation study (Section 3.5) systematically compares fixed vs.\\ trainable temperature/bias and measures inner‐product margins and loss convergence, some key corollaries—such as “increasing margin m of a constellation makes retrieval robust to larger approximation errors”—are never empirically tested. There are no ablations showing how retrieval performance degrades under quantifiable perturbations or approximate search budgets, leaving theoretical robustness claims unverified.", "was_revised": true, "final_justification": "This work importance is about the \"modality gap\". I am still trying to understand weather the modality gap will hurt the downstream performance.", "rebuttal": ">**W1:** Although the authors frame their work as a new theory of contrastive losses, the core result—characterizing minimizers of InfoNCE and sigmoid objectives through temperatures and bias—overlaps heavily with existing analyses. It isn’t clear what truly novel insight this paper adds beyond re-deriving known minimizer behavior in a different combinatorial language.\n\n**A:**\nWe did our best to cite all related work, and are not aware of any theoretical analyses that characterize minimizers in any related setting in the parameter regime of interest. If we have missed any key references, we would be very grateful to the reviewer for a list of papers so that we can appropriately cite them and place our work in better context. \n\nWe thoroughly cite prior theoretical work in the field on page 2 (lines 39 - 52). We emphasize that to the best of our knowledge, prior work does not address the practically relevant regime of parameters $d\\ll N\\ll 2^d,$ and thus, does not explain important qualitative phenomena such as the modality gap (which does not have exist for zero-loss configurations when $N\\leq d$). Both our techniques (for example, the use of convex geometry in the proof of Theorem 3.5) and our results are quite different from the ones in prior works. We elaborate when responding to W3 below.\n\n\n>**W2:** ImageNet experiments in this paper mainly for visualization of the modality gap through inner‐product histograms, but they are short of measuring end‐to‐end retrieval performance (e.g. top-1/top-5 accuracy) on large-scale benchmarks. It remains unclear whether the theory actually informs practical model design or improvement.\n\n**A:**\nIn Corollary  1 (lines 212 - lines 218), we show that the the inner-product separation condition (i.e., positive margin) of constellations directly implies the success of top-1 retrieval. In this light, our Figure 1 illustrating the inner products in ImageNet is a direct demonstration  of the high success rate of SigLIP on top-1 retrieval. The inner products with matching labels (in blue), which are above all the inner products with mismatched labels (in red), are all correctly classified.\n\n>**W3:**  The paper repeatedly invokes the $N\n\\gg d$ setting, yet fails to explain why that specific regime generates a modality gap in practice, or why it differs fundamentally from previous high-dimensional results. As a result, the significance of both the regime and the modality gap itself feels under-justified.\n\n\n**A:** We are not sure which previous high-dimensional results are being referred to; it would be very helpful if the reviewer can provide specific references in order that we can cite and discuss any related work we might have missed. \n\nThe practical regime for $N$ and $d$ is $d\\ll N \\ll 2^d$, since modern machine learning models operate on huge sample sizes ($N$ on the order of 1B-10B) and embed them in a much lower dimension ($d$ is typically between 1k and 10k).\nTo the best of our knowledge our work is the first theoretical analysis of sigmoid contrastive loss in the regime $d\\ll N \\ll 2^d$. \n\nWe next describe why the regime $d\\ll N$ is closely tied to modality gap.\nIndeed, one of our main theorems (Thm 3.5) demonstrates that it is the SigLIP loss itself that drives configurations to have a modality gap in the parameter regime of interest: zero loss configurations have a modality gap. \nThis theorem only holds when $N\\ge d + 2$. Indeed, the theorem’s conclusion does not hold otherwise: When $d\\ge N+1,$ the configuration $U_i = V_i = X_i$, where $X_1, X_2, \\ldots, X_N$ is a simplex, satisfies that $\\langle U_i, V_i \\rangle > 0 $ for all $i$ and $\\langle U_i, V_j \\rangle <0$ for all $i\\neq j$ (and is therefore a zero-loss configuration), but certainly does not exhibit the modality gap. \n\nOn the mathematical side, \nour analysis is substantially different from prior literature, including works demonstrating optimality of simplex configurations (all of which we cite).\nNote that simplices with $N$ points in $d$ dimensions do not even exist when $N>d+1$, and indeed, the minimizing configurations in our setting are very different from a simplex.\n\n\n>**W4:** While the paper’s ablation study (Section 3.5) systematically compares fixed vs.\\ trainable temperature/bias and measures inner‐product margins and loss convergence, some key corollaries — such as ``increasing margin m of a constellation makes retrieval robust to larger approximation errors\" — are never empirically tested. There are no ablations showing how retrieval performance degrades under quantifiable perturbations or approximate search budgets, leaving theoretical robustness claims unverified.\n\n**A:** Refer to the answer of Reviewer HL7g's question where we explain why larger margin leads to better performance of downstream tasks.\n\n>**Q1:** How does “synchronization” differ from “alignment” in latent clustering, and why do synchronization embeddings occupy disjoint regions that will hamper the model performance?\n\n**A:** We choose to use the term synchronization instead of alignment since perfect alignment between modalities, meaning that $U_i=V_i$ for all $i$, is neither achieved nor necessarily desired. That is, the representations of an image of a cat and the word cat are synchronized, so the corresponding vectors are close, but they are not aligned -- the vectors are still distinct.\n\nThe synchronization property holds for virtually all state-of-the-art multimodal models. \nNote that modality gap cannot occur if alignment occurs; in other words, modality gap implies that synchronization rather than alignment occurs. \nModality gap has been observed empirically by us in SigLIP and modality gap is also observed in CLIP. \n\nWe remark that the fact that models are synchronized rather than aligned might actually improve (rather than hamper) performance, since different modalities often contain different information which is best captured by distinct representations. \nAn example scenario supporting this intuition was considered in the paper \"The Platonic Representation Hypothesis\" by Huh et al. \n\n>**Q2:** In the proof of Theorem 3.1, why from 660 to 661, the $-\\infty$ becomes the $+\\infty$ in the second term when just retrieving t(s)?\n\n**A:** Thank you for catching this typo! It should remain $-\\infty.$ This agrees with equation 15 where \n$\\langle U_i^{(s)}, V_j^{(s)}\\rangle - \\text{b}_{\\text{rel}}^{(s)}\\le 0.$\n\n>**Q3:** Theorem 3.2 identifies $(m, b_\\text{rel})$-Constellations as all global minima in the t $\\to\\infty$, but real training uses finite tt and stochastic optimization. Under finite temperature and sampling noise, do approximate -Constellations still govern near-optimal minima, and can we quantify their attraction basins?\n\n**A:** \nWe expect our theoretical insights at infinite temperature to carry over to practically trained models at finite temperature, as we discuss next.\n\nWhile our main result on the shape of minimizing representation configurations holds in the limit of infinite temperature, we note that practically trained models do have extremely high temperatures: for example, in trained SigLIP models by Google Research, the temperature parameter $t$ is on the order of $100$. As this is inside the exponent defining the loss, a value of 100 is very large, and thus forces behavior on most pairs $\\langle U_i, V_j\\rangle$ that is well-approximated by infinite temperature. \n\nEmpirically, Figs 1 and 2 illustrate that representations of practically trained models share some key geometric aspects with zero loss configurations. We describe these next.\n\nIn Fig 1, we show that a trained SigLIP model yields  near perfect separability between inner products of correctly matched pairs and inner products of mismatched pairs. That is, \n$\\langle U_i, V_i \\rangle \\ge \\langle U_{i'}, V_{j'}\\rangle$ holds for almost all $i, i'\\neq j',$ which is exactly the definition of constellations (with $m = (\\min_i \\langle U_i, V_i\\rangle - \\max_{i\\neq j} \\langle U_i, V_j\\rangle)/2$ and \n$b_\\text{rel} = (\\min_i \\langle U_i, V_i\\rangle + \\max_{i\\neq j} \\langle U_i, V_j\\rangle)/2$). This is one crucial aspect in which the geometry of actual representations resembles the geometry of zero-loss configurations. \nIn Fig 2, we also showed that trained SigLIP models exhibit the modality gap, a key property of zero-loss configurations.\n> **Q4:** In section 3.4, as the number of modalities $k$ grows or the synchronization graph sparsifiers, how does learning relative biases impact convergence speed and final separation?\n\nWe ran additional experiments to investigate how changing the number of modalities affects the margin in the representations (i.e., the final separation) with $N=100$ and $d=10$ and trainable temperature and relative bias. The results are summarized in the table below, more we observe that the margin increases as we increase the number of modalities.\n\n**Final Margins by Number of Modalities**\n\n| Number of Modalities | Final Margin |\n|----------------------|--------------|\n| 2                    | 0.471241     |\n| 4                    | 0.427528     |\n| 6                    | 0.472571     |\n| 8                    | 0.595576     |\n| 14                   | 0.610853     |\n| 20                   | 0.611314     |\n\n>**Q5:** Do you have any performance guarantee for your method in the paper? Like when we set up the hyperparameters which arrive at the upper bound or lower bound as your paper mentioned? Do you have any valid metrics to show the global minimizers improve the embedding?\n\n**A:** A bit of clarification regarding this question would be helpful, we are not sure that we fully understand it.\nAs discussed in the paper, we use the margin of a constellation as a proxy for end-to-end retrieval performance. At finite temperature, we have carried out experiments demonstrating that margin increases monotonically as loss decreases. Thus, configurations with lower loss have a larger margin."}, {"review_id": "F2BJ4XKD1a", "reviewer": "Reviewer_9MsS", "summary": "This paper provides a theoretical analysis of representation synchronization under the sigmoid contrastive loss with trainable inverse temperature and bias, as used in recent SigLIP models. The authors introduce the novel concept of (m,rb)-Constellations, a class of configurations related to spherical codes and characterized by a margin and a relative bias. The paper leverages this formulation to explain the empirical success of SigLIP in retrieval tasks, interpret the presence of the modality gap, and identify the necessary dimension for producing high-quality representations. Additionally, the authors propose a reparameterization of the sigmoid loss using explicit relative bias, enabling practical improvements in scenarios such as synchronization with a locked encoder or across more than two modalities.", "questions": "In practice, over-eliminating the modality gap would not improve or even degrade the performance of the multimodal model. It would be helpful if the authors could provide a theoretical perspective on the optimal degree of separation between modalities in the context of the sigmoid contrastive loss. Is this gap also generalizable to InfoNCE contrastive loss ?\n\n\nCan the authors analyze how the model's performance on standard benchmarks evolves as the sigmoid contrastive loss progressively approaches zero during training?\n\nIt would be valuable for the authors to offer insights into whether minimizing the sigmoid contrastive loss to zero might adversely affect the model’s generalization ability, e.g. zero-shot retrieval or under query distribution shifts.", "limitations": "The authors have discussed the limitations in Section 4.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\nThe paper provides the first characterization of global minimizers for sigmoid contrastive loss, with the number of data points N greatly exceeding the embedding dimension d.\n\nThe proposed reparameterization of the sigmoid loss with relative bias provides both theoretical insights and practical benefits, such as faster convergence and better performance in locked encoder scenarios.\n\nWeakness\n\nIn practice, over-eliminating the modality gap would not improve or even degrade the performance of the multimodal model. It would be helpful if the authors could provide a theoretical perspective on the optimal degree of separation between modalities in the context of the sigmoid contrastive loss. Is this gap also generalizable to InfoNCE contrastive loss ?\n\n\nCan the authors analyze how the model's performance on standard benchmarks evolves as the sigmoid contrastive loss progressively approaches zero during training?\n\nIt would be valuable for the authors to offer insights into whether minimizing the sigmoid contrastive loss to zero might adversely affect the model’s generalization ability, e.g. zero-shot retrieval or under query distribution shifts.", "was_revised": true, "final_justification": "The authors' rebuttal effectively addresses my concerns.\n\nTherefore, I tend to maintain my score.", "rebuttal": "> **Q1:** In practice, over-eliminating the modality gap would not improve or even degrade the performance of the multimodal model. It would be helpful if the authors could provide a theoretical perspective on the optimal degree of separation between modalities in the context of the sigmoid contrastive loss. Is this gap also generalizable to InfoNCE contrastive loss ?\n\n**А:** We thank the reviewer for suggesting the direction of theoretically understanding what the optimal modality gap is.\n\nCurrently, our results are more qualitative and are aimed at justifying why multimodal models trained with sigmoid loss exhibit a modality gap. We do not, however, suggest that one should aim to eliminate the modality gap -- in fact, as written on page 3, \"we leverage the modality gap to build a linear adapter which can be\nused towards synchronizing representations when one encoder is locked.\" Thus, we view the modality gap not necessarily as a drawback of multimodal models, but rather as a necessity for state of the art performance. Again, as explained on page 3, as \"different modalities\nmay contain different information it is only natural that\nthey be represented in disjoint parts of the space.\"\n\nWe note that in taking this perspective, we crucially depart from previous theoretical work such as \"Understanding Contrastive Representation Learning through\nAlignment and Uniformity on the Hypersphere\" by Tongzhou Wang and Phillip Isola. Prior work in the literature aims to show convergence to \"aligned configurations\". Such may not be optimal and are also not realistic. Both models trained with sigmoid loss (such as SigLIP) and models trained with InfoNCE loss (such as CLIP) exhibit a modality gap -- for CLIP, see Liang, Victor Weixin, et al. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning\" Advances in Neural Information Processing Systems 35 (2022): 17612-17625.\n\n> **Q2:** Can the authors analyze how the model's performance on standard benchmarks evolves as the sigmoid contrastive loss progressively approaches zero during training?\n\n\n\n**А:** We set up the following experiment on synthetic data $N = 1000, d=20$ with both trainable relative bias and temperature and optimization using Adam. To track the performance of the optimization we used the margin and the number misclassified pairs (such that $U_i$ is not the closest point to $V_i$ out of the U's or vice verse) of the representations as a proxy (for why this translates to better performance on downstream tasks refer to the answer of Reviewer HL7g's question). This experiment confirmed that the margin continues to improve throughout training and the number of missclassified pairs drops to zero after less than $200$ epochs.\n\n\n> **Q3:** It would be valuable for the authors to offer insights into whether minimizing the sigmoid contrastive loss to zero might adversely affect the model’s generalization ability, e.g. zero-shot retrieval or under query distribution shifts.\n\n**А:**  We don't think this is a big concern for the following two reasons:\n1.  The loss in SigLIP is already very small. For more on this see the comment under the first question of Reviwer HUkA.\n2. While over-optimization in deep learning often leads to poor generalization, this is not necessarily the case in our setting. When the sigmoid contrastive loss converges to zero, the resulting representations form an $(m,b_\\text{rel})$-constellation—an analytically desirable configuration. Therefore, analyzing the global minima of the loss function is not merely of theoretical interest, but also of practical relevance, as it helps us understand whether the learned representations exhibit the intended structural properties."}]

Author rebuttal:

Meta-review: This paper presents a theoretical analysis of the sigmoid contrastive loss, particularly in the practical regime where the number of data points (N) is much larger than the embedding dimension (d). The authors introduce (m, rb)-Constellations, a geometric structure, to characterize the set of global minimizers for this loss function when temperature and bias are trainable. The paper's primary strength, as noted by all reviewers, is its rigorous theoretical approach to understanding a widely used loss function. It provides a solid mathematical foundation for empirical phenomena observed in state-of-the-art models like SigLIP, most notably offering a formal explanation for the "modality gap." The work is well-motivated and directly connects its theoretical findings to practical aspects of representation learning, such as retrieval robustness and multi-modality synchronization.

The main weaknesses, raised consistently by reviewers HUkA, HL7g, and szUN, center on the limited empirical validation of the paper's practical claims. For instance, the proposed re-parameterization of the sigmoid loss (L_RB-Sig) is shown to have faster convergence, but only on small-scale synthetic data. Claims about improved retrieval robustness due to larger margins are theoretically justified but not demonstrated on downstream tasks. During the rebuttal period, the authors addressed these points by arguing that their zero-loss analysis is a reasonable approximation for real-world models which train to very high temperatures. They also clarified the novelty of their work by highlighting its focus on the d << N regime, which distinguishes it from prior analyses and led reviewer szUN to raise their score. The authors acknowledged the limited scope of their experiments but provided additional synthetic results and cited relevant literature to bridge the gap between their theoretical contributions and expected practical outcomes.

Despite the borderline scores, the paper's theoretical contributions are significant enough to warrant acceptance. The characterization of global minimizers via (m, rb)-Constellations and the principled explanation of the modality gap are novel insights that will be of considerable interest to the representation learning community. The discussion period clarified the scope and significance of the work. Therefore, I recommend this paper for acceptance. The authors are strongly encouraged to incorporate the feedback from the reviewers, particularly by better contextualizing the practical implications of their theoretical results and acknowledging the limitations of their experimental validation in the final version.
Decision: Accept (poster)