UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels
Authors: Hangke Sui
Yuqing Wang
Minh N. Do
Venue: ICLR 2026 Poster (2026)
Keywords: contrastive learning
representation learning
multimodal alignment
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper introduces a unified and theoretically grounded perspective for contrastive alignment through RKHS, providing closed-form solutions via a contrastive similarity weight matrix.

Abstract: Contrastive objectives power state-of-the-art multimodal models, but their training remains slow, relying on long stochastic optimization.
We propose a Unified Framework for Efficient Contrastive Alignment via Kernels (UniCon), which spans linear and nonlinear encoders as well as one-to-one and many-to-many alignments.
At its core, UniCon introduces the contrastive similarity weight matrix $S(\gamma)$, which enables closed-form global solutions that provably replace minibatch back-propagation with exact updates. Through the lens of reproducing kernel Hilbert spaces (RKHS), UniCon provides a kernelized perspective that unifies contrastive alignment and reveals its connection to spectral methods.
To validate the theory, we conduct experiments on synthetic, unimodal, multimodal, and zero-shot tasks, demonstrating that UniCon achieves substantial efficiency gains while preserving generality and strong empirical performance.

Published as a conference paper at ICLR 2026
UNICON: UNIFIED FRAMEWORK FOR EFFICIENT CON- TRASTIVE ALIGNMENT VIA KERNELS
Minh Hangke Sui 1,3+ Yuging Wang?. Minh N. Do 1.23.4
* Equal Contribution.
Department of Electrical & Computer Engineering, The Grainger College of Engineering, UITC
2 Siebel School of Computing and Data Science, The Grainger College of Engineering, UIUC
3 Coordinated Science Laboratory, University Illinois Urhana-Champaign (UTUC)
4 VinUni-Illinoni Smart Health Center (hangkes2, yuging14, minhdo @illinois. .du
ABSTRACT
Contrastive objectives power state-of-the-aa multimodal models, but their training remains slow, relying long stochastic optimization. We propose a Unified Framework for Efficient Contrastive Alignment via Kernels (UniCon), which spans linear and nonlinear encoders well as one-to-one and many-to-many alignments. At its core, UniCon introduces the contrastive similarity weight matrix S(?), which enables closed-form global solutions that provably replace minibatch back- propagation with exact updates. Through the lens of reproducing kernel Hilbert spaces (RKHS), UniCon provides a kernelized perspective that unifies contrastive alignment and reveals its connection to spectral methods. To validate the theory, we conduct experiments on synthetie, unimodal, multimodal, and zero-shot tasks, demonstrating that UniCon achieves substantial efficiency gains while preserving generality and strong empirical performance.
INTRODUCTION
Learning semantically aligned representations across different modalities, such as vision and language. has long been a central goal in machine learning (Ngiam et | | Srivastava & Salakhutdinov 2012). particular, Multimodal Contrastive Learning (MMCL)(Haam) et al. (2024) has recently achieved remarkable success in zero-shot classification (Radford et al. 2021 Ja et al. (021). cross- modal retrieval (Mu et al., 2022] Goel al.] (022)) and general visual understanding (Surís et al.. 2023 Lin et al.) (023). These models typically train modality encoders, e.g., vision encoder and a language encoder, such that paired inputs are mapped to nearby representations in a shared space, while unpaired inputs mapped far apart. At the heart of MMCL lies contrastive representation learning (Chopra et al., 2005; Gutmman Hyvärinen] 2010 Sohn 2016: Oord et al. 2018 Chen
et al. 2020 Radford et | 2021). Its versatility has made core component across diverse domains, including NLP (Gao et al. [022 Izacard et al., (021). bioimaging (Sanchez-Fernande et al. 2203)
Taleb al. 2022 Han et al.) (2022). recommendation (Xie al. 2022 Yu et 2023 Jing al. 2023 Yang et al. 2023), and graph learning (Kipf et al. 2019 You et al. 2020). The typical pipeline involves feature extraction via deep encoders and the optimization of contrastive loss.
Despite the impressive empirical performance of contrastive learning across vision, language, and multimodal domains. the theoretical foundations underlying its success remain only partially under-
stood. There are works on the analysis of loss and training dynamics(Wanm & Liu [022]] Tian 2022
HaoChen Ma 2022), provably guarantee of the model generalization (HaoChen et al. 2021 2022
Tosh et al. 2021 Parulekar et al.] 2023), duality between contrastive and non-contrastive method ( Tian
ct al. 2021 Balestriero & LeCun 2022). A growing body of theoretical work has sought to formalize
contrastive learning (Saunshi al. 2019 Tian et al. 2021 Jing et al. 202 Wen & Li 2021 often
by simplifying the problem to single modality settings. Recent advancements in contrastive learning
have introduced novel loss functions and analytical frameworks to enhance representation quality
and training efficiency. (Xu et al., 2023) Wang et al.) 2024 |Schuhmann et al. (022). Analytical
studies have examined contrastive learning from different perspectives. For example, Shi al.

---

Published as a conference paper at ICLR 2026
(2024) interpret the CLIP loss through the lens of optimal transport; while Tian 2022); Nakada et al. (2023) analyze multimodal contrastive learning using SVD- and PCA- based formulations. showing that, under linear encoders, contrastive loss minimization reduces to calculating weighted covariance matrix. Yet, this insight has not been translated into nonlinear encoder settings and practical implementations.
We introduce UniCon, a Unified Framework for Efficient Contrastive Aligument via Kernels, which leverages structured contrastive similarity weight matrix S()) directly solve contrastive objectives. As illustrated in Figure T UniCon departs from gradient-based training and instead: (i) in the linear setting(Nakada al. 2023). solves a single spectral decomposition yielding optimal encoder matrices in closed form (ii) in general nonlinear setting, provides a unified kernelized framework that enables fast alignment via implicit representation inference.
Our key contributions are follows:
Theoretically, we provide a kemel-based perspective that unifies linear and nonlinear en- coders, showing that minimizing contrastive loss is equivalent to spectral update. This leads to a provably optimal solution in closed form and connects contrastive learning to spectral methods.
Beyond one-to-one matching, our framework generalizes to many-to-many alignment, broadening the applicability contrastive alignment.
Empirically, we demonstrate that UniCon converges fast and achieves competitive or superior performance across synthetic, unimodal (CIFAR-10), and multimodal (Flickr33K,MSSCOCC) and zero-shot transfer (image-text retrieval), offering up 461x speed-up over minimizing CLIP loss by stochastic gradient descent.
Nanlinear (with kernels)
Lissar (with SVD)
6
WITO CHITOT FOLD
(,Or))
(,9g)
(,0g)
5,92)
72
T
Prs
Pt
4,00
7e Xa
Hyper spherical Sumilarity
S(y)
Figure 1: Unified Framework for Efficient Contrastive Alignment via Kernels (UniCon). Starting from paired inputs, UniCon builds contrastive similarity weight matrix 5(y) using hyper-spherical similarities, then computes either (i) a closed-form spectral update in the linear case (orange) or (ii) kernelized solution in the nonlinear case (blue).
BACKGROUND
Contrastive Representation. Contrastive learning (Chopra et al. 2005 Gutmann & Hyvärinen 2010; [Sohn] 2016) Oord et al., 2018 Chen et al.) 2020 Radford et al. 2021) leverages paired inputs as a form of supervision. The central goal to learn a representation space where positive (matching) pairs are mapped to nearby embeddings, while negative (non-matching) pairs are pushed apart. Learning representations on the hypersphere leads to better performance than in Euclidean space(Wang al. 2017). it avoids conflicting forces between attractive and repulsive gradients. (Wang & Isola 2020) further shows that the distribution of representations on the unit hypersphere is encouraged to be uniform.
2

---

Published as a conference paper at ICLR 2026
Contrastive Loss Contrastive loss (Hadsell et al., 2006) first proposed in Siamese networks to pull together positive pairs and push apart negatives. The formulation was later unified under the InfoNCE loss (Oord et al. 2018), which serves as the basis for many self-supervised methods, including SimCIR (Chen et al. 2020). Supervised Contrastive (SupCon) loss (Khosla et al.] (020) extended contrastive learning the supervised setting, where each anchor have multiple positive samples from the same class. Building on the general family of contrastive losses introduced by (Tian) 2022). (Nakada et al. 2023) showed that these multimodal contrastive objectives can be connected to singular value decomposition (SVD) under linear setting.
Kernel Method Given positive finite measure l over parameter space θ define kernel k X' x X' + R by k(2, œ) = (Ø(x; 0), (i;()) := So o(x; (z; 0) du(0), which induces a Reproducing Kernel Hilbert Space (RKHS). Any function f in this space admits the representation
f(x Σ
175
== Wi k(x,,) 22), W IR. Kernels are commonly used to learn representations (Korbblihh)
et al. 2019 Klabunde et al. 2025), as they capture the relative structure among samples, critical for
many learning algorithms (Aronszajn 1950: Hofmann et al., 2008 Müller et al., 2018 Gong et al.)
2025
3 METHODOLOGY
We observe N paired samples ((x, where x; E R and y, E R The objective of contrastive learning is to learn encoders fg, RRd _ IR" and føz RR - R' with modality-specini parameters 01 and 82, such that paired inputs are mapped to similar representations in a shared r- dimensional embedding space, while non-paired inputs remain dissimilar.
In the sections that follow, we first formalize the general contrastive learning framework, then analyze it under a linear representation setting, and finally unify our analysis spanning nonlinear encoders in RKHS. The proof can be found in the Appendix [
Definition 1 (Hyper-spherical similarity). Define the similarity between x; and y: as the inner product on the hyper-sphere:
(x:) llfo, (xx)l12 [[ (12))l2/ g-
(y,(y,)
fg] (x)
(1) Definition 2 (Generalized contrastive loss). Given N paired samples ((x;,))) we write the similarity matrix [sy]- With o, 1b weights Egg E [0, 11. the bidirectional general contrastive loss
8i = (x),,, for (Vy (y)))s-cce
= llfo, (x+) l12 ll (r,)
1 R-
monotonically increasing, scaling factor v 2 1, and
1
L(01,02) 02)
Σ ())
&( Σ jEP.()
exy - v8a)) + eavb(su - vSs)))
(2)
2n
Pa
1=1
1
Σ
ei vSKs) + cacb(sk usKi)) + R(O,, 02) (PP,(0
Σ
2n
kE),
where Pz (i) denotes the index set of all samples in {yr) paired with x, while Po(i) denotes the index set of all samples in {2r} paired with yy. (P)( is the cardinality of the set Pz(i;; and R is an optional regularizer.
This formulation naturally extends from one-to-one alignment Tian 2022 to many-to- align-
ment: for example, a single image may correspond to multiple valid captions, and data augmentation can be viewed creating diverse positive pairs. The scaling factor v 2 1 adjusts the relative influence of positive pairs, while <ij 2 O controls which pairs contribute to the loss (often Gjj = 1 for all negatives). The functions ø and & are typically convex and monotonie, shaping the loss for optimization. By choosing specific forms for &b and Φ, we can recover familiar losses. More detailed examples can be found in Appendix A
Definition (Contrastive similarity weight matrix), Consider the general contrastive loss L(O,, 02), and a batch of paired samples {(x;, Denote {e, ye the elementary basis vectors of TR.. The contrastive similarity weight is then defined as:
7y P2(1)
T3s [Pa())
S(7) n
e,e,
(3)
3

---

Published as a conference paper at ICLR 2026
with weight coefficients
- v Emmp Cm Cm (sim v8s))) ExEP 0b (Ei))) - v8ss)),
Pyj (ss(1-
ifj E E
7ij -
(4)
ifi $ P.(1)
By (c:x111))/11) - ()sj()) - ".Emer Emel vsji))
ifj P,())
(5)
(kEP)(1) Hi (191 - =sss)),
ifj & P.(1)
where
1y = - - -))8s)) +
Σ Cim -V8i)),
(6)
mpP.(0)
81 -1
- -V)ass +
Σ
(me - (Mmss)))
mpP,(0
The contrastive similarity weight matrix S(7) weighs pairwise interactions. We now show that minimizing the contrastive loss is equivalent to maximizing new objective function with the constructed 5()).
Lemma (Gradient equivalence). Consider minimizing the general contrastive loss (see Equation 2). the gradient of the contrastive loss with respect encoder parameters satisfies:
dc
åtr (Fe, (X)S())F 8g (Y))
aR (01.02)
KE (1.2)
002
00p
80
(01.09))))))
where
X x=(x1, [x1, X,]] E di xn
"",,"",,,, E d2d2x
Fo.(x [Co, (x.)... fo (xn)) R Fo, Y) [Co, (vi) (yg B
This result reveals that the gradient of the contrastive loss in Equation 2 is the negative of the gradient of the proposed objective function in Equation [8] Hence, minimizing the contrastive loss is equivalent to maximizing the objective with contrastive similarity weight matrix S(y):
tr (Fe, (X) R (0),0)).
(8)
3.1 LINEAR REPRESENTATION SETTING
We first specialize the general framework to the linear representation case. Here the encoders are parameterized as matrix multiplications: fo, (x) Fix, fo (y) F2y, where Fj E Brxd F2 E grxd2 are learnable projection matrices.
Definition (Weighted contrastive covariance). Define the weighted contrastive covariance as:
1
I1 5,5
7ife
C(r) = X =
x.y
P
(P(())
with coefficients "ij share the same definition with Definition 3
Remark 6. Note that the definition of C(7) exactly matches the definition of 8(3) Nakada al. (2023) under one-to-one alignment setting, which is proved in details in Appendix B.1 The structure Clr) captures positive and negative pairs relationships, weighted appropriately. Our expression keeps the diagonal correction (an+ 5z)/2 /2 that prior work reduced to 1 by assuming that & and i are identity functions. This modification improves both theoretical generality and empirical performance.
Proposition 7. Under the linear setting, the Lemma 4 is specialized
ac 0tr (F(C(T)F2 OF,
ÖR (F) OFr
KE (1,2)
(10)
OF
(F..F)
To solve the optimization problem induced by our reformulated objective, characterize its maxi- mizer the linear setting. We arrive at the following theorem, which establishes that the convergence of the contrastive loss be replaced by closed-form spectral update.
4

---

Published as a conference paper at ICLR 2026
Theorem (Spectral characterization (Nakada al. 2023)). loss function C (F1. F2), with F2) = F F2l F. Then,
Consider minimizing the contrastive
arg min& (2)
arg max FiERexd,, F P2ER
tr (FIC())
P
F2
(11)
F2
((F1,FF2) E E R Rxxd2 F F2
Σ P ouui
1
(12)
where (0r, Ui, vi)= are the top-r singular values and vectors of C(r) according to the Eckart-Yuug-Mirrly theorem.
Consequently, in linear case, the global minimum is achieved by projecting the contrastive covariance C(r) onto its top-r singular components. Thus, gradient descent on any loss in the contrastive family (Ø, 15) merely tracks the dominant singular subspace of C(r). Our algorithm UniCon performs this update in closed form, replacing thousands of SGD steps by one spectral factorization.
3.2 KERNELIZED REPRESENTATION SETTING
Why leave the linear world? The linear setting reveals clean spectral structure for contrastive alignment, but modal relations (e.g., vision <+ language) are typically nonlinear. Moreover, with frozen or partially frozen pretrained encoders, the residual alignment is rarely captured by mere linear heads. We therefore lift the analysis to nonlinear encoders while keeping the output space R" shared across modalities. Kernliization provides tractable route with explicit spectral solutions that reduce to the linear case.
RKHS representation.
Let (Hx, kx) and (yy,ky) be RKHSs with canonical feature maps
xx(x) = kx (x(x)) x) E Hxx Φy (y) = k. E Hy,
(13)
satisfying the reproducing property s(x) = (S. (x)xxx)))Hx)))))))))))))))))))) Hx for all f Hx (and analogously for Hy-). For r-dimensional outputs, the a-th coordinate (a (a=1...... I,...,r) admits the representer form
n
1/0 = Σ Asa (xx:); ==1
((0 = Σ
Bja ky
(14)
j=1
with A, B RaXT Let Kx [xx (x)) and Ky = -ky (y.,,,,,))), The batch embeddings are
Fo, (X) A Kx R
Fo, =B B R**.
(15)
The contrastive trace term becomes
- -
S(7) Ky)
(16)
With the RKHS parameterization above and the kernelized trace form in equation 16 the entire objective can be written purely in terms of the Gram matrices. Under this notation, the optimizer is governed by the principal singular structure of M := S(>) Theorem 9 (Kernelized spectral characterization (unified form)). Letp > and define the regularizer R(A, B) || (K A)T (K3/² B) 1| Then minimizing the contrastive loss is equivalent to the kernelized maximization
1/2 as formalized below.
tr(A Kx S(y) KyB) P
[K 1/2 A) (K B)
max B.
(17)
2
Let A' A K B' BT 1/2 M x S(7) K22²2 Then equation 17 rewrites
(BAM - 1 B'1?.
max N.B'
(18)
M U is an SVD and M, El "iuni its best rank-r approximation by Eckart-Young Mokarr-Young-Mirsee Mirsky theorem, then all maximizers satisfy the relation
TB' M,
ABT
1 1/2
(19)
P
If Kx or Ky is singular, replace inverse square roots by Moore-Penrose pseudo-inver inverse square root..
5

---

Published as a conference paper at ICLR 2026
One explicit optimal choice is = U U, and B* Ky 1/2,, Er/p, where U, [u],-... .......: Ur] V, = Ur], , E, diag(o),-..,,00 .r) are SVD of M.
Corollary (Kernel inference (out-of-sample)). Let D = {(xis):))i=1 be the reference batch used to build contrastive similarity weight S(r)) and let l be positive definite ker- nel. For a new pair (x,,y) set xx(xx)=
.......x. kx (X x²)]] and Ky (y") =
(y1, ................................ ky y*)" With optimal (A", B²) from Theorem 9
lo (x*) = (A*)Txx(x"))
lo - (B*) (
(20)
and the similarity
Kx x ky()")
sxx","")
(21)
|| Rx (x" )1 B+ Ky (y
In practice we observe that simple angular kernel k(u, v) = |||||||b||| (sin 0 + (5 c) yields the best trade-off between speed and accuracy.
arccos T+i|اaa
3.3 UNIFIED SPECTRAL VIEW (LINEAR AS A SPECIAL CASE OF KERNEL)
Our kernel formulation strictly generalizes the linear setting. For linear kernels kx (x, x') (x, r²).
ky (v. = (y. v'). the Gram matrices reduce to Kx X and Ky Y Y. Setting Fi A and F/2 = B Y yields R (A. B) = | F2 ? = tr(FiFi F2F2 ), exactly matching the penalty used in the linear section. In the linear analysis we considered the weighted contrastive
covariance matrix C(r) = X S(7) Y while in the RKHS analysis the central operator is M
K S(y) KYL2. When the kernels are lincar, M (X
T
Let reduced SVDs be X Ex and Y = Uv Vy, and define T E with SVD T = U Then we obtain
C(7) UxTUY (UXUx)Ex(U
M V (VXUT) (22)
Hence left/right multiplication by orthonormal matrices maps C(7) and M to the same nonzero singular values Σn. In particular, their best rank-r approximations select the same spectrum (Eckart-Young-Mirse
When X or Y is rank-deficient (or for general kernels where Kx. Ky may be singular), all statements hold on the effective subspace using
square roots.
Thus the kernel SVD of M is the exact RKHS analogue of the linear SVD of C(y). and the linear setting is recovered the special case of the linear kernel.
Unified consequence. T alignment:
unified spectral view for understanding contrastive
contrastive loss minimization
best rank-r approximation with RKHS.
with 51)) encoding the particular contrastive objective (e.g.. InfoNCE, CLIP, triplet) and the kernel selecting the nonlinear feature space in which the alignment performed.
3.4 SCALABLE TRAINING
Batch aggregation. In practical scenarios, particularly in large-scale vision-language or multimodal applications, N can be substantial, leading to prohibitive computational and memory demands. We therefore aggregate mini-batch contrastive similarity weight matrix 5(b)(z) by our closed-form solution. The final S(y) by taking quality weighted sum.
Numerical stability. If Kx or Ky is ill-conditioned or singular, form square roots with a Tikhonov regularization when needed, replacing K by K + A/ with À > 0 k/2 This enhances robustness to near-singular Gram matrices and stabilizes the closed-form update. For low rank approximation step, one can use randomized SVD (Halko et al. 201T) on M or Nyström approximations of Kx, Ky to reduce both memory and time.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
During implementation, UniCon directly leverages the contrastive similarity weight S(7) for con- trastive alignment. The full computation of S(7) provided Appendix C.I We start with synthetic data and then extend evaluate the practical utility of UniCon unimodal and multimodal datasets. In the unimodal setting, we test on CIFAR-100, 100, where the goal is image-to-image alignment. In the multimodal setting, test FLICKR33K and MS-COCO for image-text retrieval. Across both settings, we compare UniCon against standard CLIP-style contrastive learning trained with stochastic gradient descent (SGD-CLIP) All experiments and wall-clock times are reported for reference.
single NVIDIA L40S GPU,
4.1 SYNTHETIC DATA
To verify our theoretical results in fully controlled environment, conduct synthetic experiments in both linear and nonlinear regimes. Details of setup could be found in AppendixC.2
Linear Latent-Factor Model. We generate synthetic data from latent vectors z E R" that are sampled around K 3 cluster centers in latent space. We compare our method UniCon, which performs a spectral update using the closed-forg SVD of the weighted covariance C(y), against a standard baseline trained via stochastic gradient descent (SGD) on the CLIP loss with AdamW optimizer (Ir = 2 10-$). As shown in Figure 2 UniCon achieves 100% matching accuracy after just 0.02 seconds. CLIP-SGD requires 400 epochs (0.32 seconds) to reach the same accuracy. This demonstrates that UniCon not only preserves structure in the latent space but also converges faster than gradient-based methods.
()
Dato
LIKCAndoouRRCy 1.0,
Figure 2: Visualization of cross-modal alignment using t-SNE embeddings of the shared represen- tation space. Modality 1 (cross) and modality 2 (circle) projected from different spaces into shared representation space IR.. Colors indicate ground-truth clusters, and lines connect matched image-text pairs. Both SGD-CLIP (left) and UniCon (right) successfully align paired samples while preserving cluster structure. The visual similarity between the two plots is expected: UniCon achieves a comparable aligned representation to SGD-CLIP with substantially less training time.
Nonlinear Latent-Factor Model. We further evaluate the method under nonlinear transformation of the latent space. The baseline model trains nonlinear MIP encoder via SGD on CLIP loss and AdamW optimizer. Our method UniCon calculates sequence of (Fe, (x) (y)) each for one training batch. Then we apply batch aggregation on validation data to calculate the weight for each training batch. The performance is evaluated by the correctly matched pairs of test data using the kernel-weighted generalization. UniCon converges in 2 epochs (0.04 seconds), achieving 86% matching accuracy, while CLIP-SGD reaches 84% after 500 epochs (0.65 seconds).
Summary. both linear and nonlinear settings, UniCon demonstrates rapid convergence and strong alignment performance, validating the thcoretical claims that contrastive learning objectives can be solved via single spectral step. eegufirm that UniCon achieves consistent cross-mode modal alignment.
7

---

Published as a conference paper at ICLR 2026
(a) Initial S(7)
(b) After epoch
(c) After epoch 2 (converged)
Figure 3: Evolution of the contrastive similarity weight matrix S(y) in the nonlinear latent-factes model across training.
4.2 IMAGE-IMAGE ALIGNMENT ON CIFAR- 10 (UNIMODAL)
Setup. We evaluate UniCon unimodal image alignment task using CIFAR-10. benchmark dataset containing 10 object categories. Following the convention of SimCLR (Chen et al. 2020)), treat augmentations derived from the same image positive pair, while augmentation pairs from different images are treated as negative. Feature embeddings are extracted using a frozen 18 encoder. The objective is to align these embeddings such that images from the same class are pulled closer together in the shared feature space, thereby facilitating unimodal classification.
We conduct this experiment under nonlinear setting. For baseline, train lightweight projection encoder g(-) on frozen ResNet-18 features with bidirectional InfoNCE optimized by SGD. The encoder a two-layer MLP encoder. We optimize with SGD for 300 epochs. The trained encoder is then frozen for linear probing with a small classifier on the 128-dimensional embeddings. UniCon learns a kernelized projection from frozen ResNet-18 features by a spectral closed-form solution. Given two augmented views (21, =2), we compute an angular kernel between features and iteratively estimate a batch-wise feature map A. After learning an average of A, we freeze it and train a small linear classifier on the 128-dimensional embeddings for linear probing, mirroring the SGD setup for fair comparison.
piter Seacee
(a) Contrastive classification
(b) UniCon
(c) SGD-CLIP
Figure 4: Visualizations of unimodal alignment on CIFAR-10. (a) Self-supervised contrastive learning clusters semantically similar images and uniformly distributes clusters on the hypersphere. (b-c) Unimodal confusion matrices for UniCon and SGD-CLIP, showing predicted vs. true class accuracy. The pear-identity structure and visual similarity of both matrices indicate that UniCon and SGD-CLIP achieve comparable discriminative performance in unimodal contrastive alignment.
Results. To evaluate classification performance, we report the confusion matrix Figure (b-)), which summarizes the number of correct and incorrect predictions for each class. Each row of the matrix corresponds to the predicted class, while each column represents the ground truth. Specifically, the diagonal entries indicate the number of correctly classified samples, and the off-diagonal entries capture misclassifications between classes. This allows a detailed analysis model behavior beyond overall accuracy. Numerically, UniCon achieve the average accuracy of 61. 8.8%% with 23.38s while
8

---

Published as a conference paper at ICLR 2026
SGD can achieve the average accuracy of 62.21% with 41.98s. Empirically, UniCon converge within 2 epochs while SGD requires various iterations to converge to a comparable optimal point.
4.3 IMAGE-TER TEXT RETRIEVAL AND ZERO-SHOT SHOT TRANSFER (MULTIMODAL)
Setup. To evaluate UniCon in multimodal setting, benchmark on the standard image-text retrieval tasks from FLICKR30K and MSCOCO. We consider three backbone choices for image x: and text y: (a) ResNet-18 (He al. 2016) for images with Sentence-BERT (all well-mppet--ee -base-v2) (Reimers & Gurevych] 2019) for text; (b) ResNet-50 + Centence-BERTY (c) the pretrained CLIP VIT-B/32 model as frozen visual-textual feature extractor. UniCon is compared against an SGD- optimized CLIP baseline (GD-CLIP) under matched training/ovaluuaionn settings; both are trained to convergence.
Results. Table summarizes top-1/10 recall in both directions. Across all backbones, UniCon attains competitive or superior accuracy while reducing training time by 25-50x. With CLIP ViT- B/32 features, UniCon further improves accuracy despite requiring only single spectral update. Notably, UniCon on Resnet50+SBERT backbone achieves comparable averaged top-10 10 retrieval accuracy with CLIP ViT-B/32 backbone aligned SGD-CLIP. These findings are consistent with our theory: the spectral step efficiently recovers the dominant cross-modal structure that iterative SGD approximates over many epochs.
Table 1: Image-text retrieval on FLICKR30K. report Recall@) and Recall 10 for both image- ttxt and text->image directions.
Image Text
Text-IImag
Average
Backbone
Method
Train time
R@1
R@ 10
R@D
R@I0 10
R@1
R@10 10
SGD-CLIP
45.6 s
.043
.221
.041
.217
.042
.219
RN-18 SBERT
UniCon
1.7
.000
.145
.087
.361
.054
.253
SGD-CLIP UniCon
45.0 s
.043
.221
.041
.217
.042
.219
RN-50 SBERT
0.81 s
.134
.644
.18
.567
.161
.515
SGD-CLIP
45.3 s
.231
.595
.241
.600
.236
.597
CLIP VIT-B/32
UniCon
0.76
.284
.636
.421
.777
.353
.701
Table 2: Retrieval on MSCOCO and zero-shot transfer to FLICKR30K. All models are trained on MSCOCO. We report image to text (I+T) and text to image (T-+I) on MSCOCO and zero-shot on FLICKR30K (no fine-tuning).
MSCOCO
FLICKR30K (zero-shot)
Backhone
Method
Train (s)
Dir.
R@1 1
R@ 10
R@5
R@ 10
SGD-CLIP
5121.72
IT
.053
.253
T+1
.060
.286
RN-50 + SBERT
UniCon
11.11
I-T
.105
.388
.171
.261
T-x1
.129
.439
.249
.353
1066.60
FT
.128
.455
T-11
.123
.47
CLIP ViT-B/32
UniCon
11.15
HT
.329
.685
.808
.879
T-1
.292
.644
.766
.848
Table 2 augmeets our results with MSCOCO retrieval and zero-shot transfer to FLICKR3OK. Our training follows the standard retrieval protocol on MSCOCO with each image paired with 5 captions, and report test retrieval accuracy on 5,000 held-out pairs. UniCon achieves higher accuracy than SGD-CLIP MSCOCO while being 96-461x x faster. Beyond scalability, the learned alignment transfers robustly: models trained on MSCOCO maintain strong performance on FLICKR30K without any adaptation. Despite distribution shifts in both image and text domains, UniCon maintains

---

Published as a conference paper at ICLR 2026
strong retrieval accuracy. These results underscore UniCon's scalability, generality, and cross-dataset transfer, reavealing its potential in real world tasks.
5 DISCUSSION
Computation Efficiency. UniCon achieves rapid stabilization of the alignment subspace through derived spectral updates, which bypass many small gradient steps, demonstrating computational efficiency. Details can be found in Appendix C.T Empirically, that M (or C(7) in the linear case) converges in or a few steps. We provide an intuitive explanation: Unlike gradient-based based methods that take small local steps, each spectral update directly jumps the global maximizer of the surrogate objective, making the update much more informative.
observe an interesting phenomenon
Data Efficiency. Additionally, on MSCOCO, using only 200 images (0.24% of the dataset), with each image paired with 5 captions, already yields meaningful retrieval alignment (66.45% avg @@ 10), demonstrating both subspace convergence and data efficiency. As we discussed in Section 3 alignment is r-rank discovery problem, which gives an intuition that we don't 't need massive datasets to find the principal axes.
Static vs. Evolving Input Spaces. The theoretical optimality results with r-rank aproximation is derived under the assumption that the input space of UniCon is static. It includes two cases: (a) Input space data space (raw modalities), where UniCon itself performs end-to-end aligmment. (b) Input space is embedding space from frozen encoders. In both cases, UniCon provides globally optimal spectral solution to contrastive loss minimization from the perspective of r-rank approximation. When encoders are trainable (non-static input space), UniCon is applied during jointly optimizing encoders, the spectral update becomes conditionally optimal subproblem, i.e., optimal for the current encoder outputs.
6 CONCLUSION
We propose UniCon, theoretically grounded and computationally efficient framework for contrastive alignment that unifies linear and nonlinear encoders through kernels. We show that minimizing contrastive loss is equivalent maximizing kernelized trace objective, which in turn reduces to a best rank-r spectral approximation in an RKHS. The closed form update is driven by explicitly constructing a contrastive similarity weight matrix S(7). In the linear reduction, UniCon recovers projection onto the top-r singular directions of the weighted contrastive cross-covariance. This yields clear spectral lens on contrastive learning, interpreting alignment as r-rank structure discovery in nigh-dimensional feature spaces.
We see three conerete future directions: (i) structure-exploiting kernels, for example random features, to reduce K's 's rank and cost; (ii) hybrid spectral Srectral-SGD strategy or warm-start strategy when the input space is non-static (e.g. finetuning for domain adaptation):(ii) mini-batch aggregation and streaming variants that balance analytical fidelity with scalability, including distribution-shift-vaaaa weighting, online updates of the aggregated operator, and robust selection of batches to maintain stable global estimates.
With the theoretical grounding and competitive empirical results, UniCon advances understanding of contrastive learning for unimodal representation learning, multimodal alignment and beyond.
ACKNOWLEDGMENT
This work was supported in part by the Advanced Research Projects Agency for Health (ARPA-H) and the IBM AMMIllinnis Discovery Accelerator Institute.
ETHICS STATEMENT
We affirm that this work complies with the ICLR Code of Ethics. Our study does not involve human subjects, sensitive personal data, potentially harmful applications. All datasets used are publicly
10

---

Published as a conference paper at ICLR 2026
available (e.g., CIFAR-10) and contain personally identifiable information. We acknowledge the importance of faireess and responsible Al development and have taken care to ensure that our methods do not propagate bias or cause unintended harm.
REPRODUCIBILITY STATEMENT
We are committed ensuring the reproducibility our work. All implementation details, including model architectures, hyperparameters, and training procedures, are described in the Appendix C. We provide pseudocode for our algorithm in Appendix C and include complete proofs of theoretical results in Appendix B. All datasets used in this study are publicly available, and we will release source code and experiment scripts as supplementary materials later to facilitate replication of our results.
REFERENCES
Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society, 68(3): 337 404, 1950.
Randall Balestriero and Yann LeCun. Contrastive and non non-contrastive self-supervised learning recover global and local spectral embedding methods. Advances in Neural Information Processing Systems, 35:26671-26685, 2022.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1661 1607. PmLR, 2020.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively. with application to face verification. In 2005 JEEE computer society conference computer vision and pattern recognition (CVPR'OS) volume pp. 539-546. IEEE, 2005.
Konstantinos Drossos, Samuel Lipping, Tuomas Virtanen. Clotho: An audio captioning dataset, 2019. URL 09387
sttp:://arxxlv.org/aaa
Tianyu Gao, Xingcheng Yao, and Dangi Chen. Simese: Simple contrastive learning of sentence embeddings. 2022. [ttps:::://axx http://aww.vv..ora//////// hb/p//ass/2200 08821
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704-6719, 2022.
Shizhan Gong, Yankai Jiang, Qi Dou, and Farzan Famnia. Kernel-based unsupervised embedding alignment for enhanced visual representation in vision-language models. arXiv preprint arXiv:2506.00 02557, 2025.
Michael Gutmann and Aapo Hyvärinen. coise-contrastive estimation: A new estimation principle for unnormal- ized statistical models. In Proceedings of the thirieenth international conference artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognation (CVPR 06), volume pp- 1735-1742, 2006. doi: 10. 10.1109C22....006....
Nathan Halko, Per-Gunnar Martinsson, and Joel Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM review, 53(2):217-288, 2011.
Wenkai Han, Yuqi Cheng, Jiayang Chen, Huawen Zhong, Zhihang Hu, Siyuan Chen, Licheng Zong, I Liang Hong, Ting-Fung Chan, Irwin King, et al. Self-supervised contrastive learning for integrative single cell ma-seq data analysis. Briefings in Bioinformatics, 23b5):bb 2022.
Jeff Z HaoChen and Tengyu Ma. theoretical study of inductive biases contrastive learning. arXiv preprint arXiv:2211.14699, 2022.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in neural information processing systems, 34:5000-5011, 2021.
11

---

Published as a conference paper at ICLR 2026
Jeff Z HaoChen, Colin Wei, Ananya Kumar, and Tengyu Ma. Beyond separability: Analyzing the linear transferability contrastive representations related subpopulations. Advances in neural information processing systems, 35:26889 26902, 2022.
Kaiming He, Xiangyu 7hang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 770-778, 2016.
Thomas Hofmann, Bernhard Schölkopf, and Alexander J Smola. Kernel methods in machine learning. 2008.
Wei Huang. Andi Han, Yongqiang Chen, Yuan Cao, Zhiqiang Xu, and Taiji Suzuki. On the comparison between multi-modal and single-modal contrastive learning. Advances in Neural Information Processing Systems, 37: 81549-81605, 2024.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoe Le, Yun- Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint a 2021.
Mengyuan Jing, Yanmin Zhu, Tianzi Zang, and Ke Wang. Contrastive self-supervised learning in recommender systems: A survey, ACM Transactions Information Systems, 42(2):1-39, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances neural information processing systems, 23:18661-18673, 2020.
Thomas Kipf. Elise Van der Pol, and Max Welling. Contrastive learning of structured world models, arXiv preprint ar 1911 2019.
Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: A survey of functional and representational measures. ACM Computing Surveys, 57(9):1-52, 2025.
Simon Korbllthh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pp. 3519-3529, PMLR, 2019.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. 2023.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-mmag pre-training. In European conference on computer vision, pp. 529-544. Springer, 2022.
Klaus-Robert Müller. Sebastian Mika, Koji Tsuda. and Koji Schölkopf. An introduction to kernel-based learning algorithms. In Handbook of neural network signal processing, pp. 4-1. CRC Press, 2018
Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics, pp. 4348-4380. PMLR, 2023.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y Ng. et al. Multimodal deep learning. In ICML, volume 11, pp. 689.696.75 2011.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: h0w::8807.335 837.04488 2018
Advait Parulekar, Liam Collins, Karthikeyan Shanmugam, Aryan Mokhtari, and Sanjay Shakkottai. Infonce loss provably leams cluster-preserving representations. In Gergely Neu and Lorenzo Rosasco (eds.). Pro- ceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pp. 1914-1961, PMLR, 12-15 Jul 2023. URL parulekar23a html
mml. www.s/resssee... ptess///ss////. v195/ htwpp//g........ee..eeeee.e.eeee
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmI.R, 2021.
12

---

Published as a conference paper at ICLR 2026
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arxivv preprint arXiv:1908. 10084, 2019.
Ana Ferhhnzzz,,,,, Elisabeth Rumetshofer, Sepp Hochreiter, and Günter Klambauer. Cloome: contrastive learning unlocks bioimaging databases for queries with chemical structures. Nature Communications, 14(1): 7339, 2023.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In International Conference Machine Learning, pp. 5628-5637. PMLR, 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp-
315-823, 2015.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset training next generation image-text models. Advances in neural information processing systems, 35: 25278-25294, 2022.
Liangliang Shi, Jack Fan, and Junchi Yan. Ot-clip: Understanding and generalizing clip via optimal transport. In Forty-first International Conference on Machine Learning, 2024.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Nitish Srivastava and Russ Salakhutdinov. Multimodal learning with deep boltzmann machines. Advances in neural information processing systems, 25, 2012.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11888-11899, 11898, 2023.
Aiham Taleb, Matthias Kirchler, Remo Monti, and Christoph Lippert. Contig: Self-supervised multimodal contrastive learning for medical imaging with genetics. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 20908 20921, 2022.
Yuandong Tian. Understanding deep contrastive learning via coordinate-wise optimization. Advances in Neural Information Processing Systems, 15:19511-19522, 2022.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervieed lcarning dynamics without contrastive pairs. In International Conference on Machine Learning, pp. 10268-10278. PMLR, 2021.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy. and linear models. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato (eds.), Proceedings of the 32nd International Conference on Algorithmic Learning Theory, volume 132 Proceedings of Machine Learning Research, pp. 1179-1206. PMLR, 16-19 Mar 2021. URL ht tettss: http://pooeceiccccccooo/ooo www.mm.innnmmmmmmm...mmmmmmmmm.. m... r.pees///// v132/tosh21a. htm
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2495-2504, 2021.
Feng Wang, Xiang Xiang. Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. Proceedings of the 25th ACM international conference on Multimedia, pp. 1041-1049, 2017.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International conference on machine learning, pp. 9929-9939, PMLR, 2020.
Yiping Wang, Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamieson, and Simon $ Du. Cliploss and norm-based data selection methods for multimodal contrastive learning, Advances in Neural Information Processing Systems, 777115028-1069) 2024.
Zixin Wen and Yuanzhi Li. Toward understanding the feature learning process of self-supervised contrastive learning. In International Conference Muchine Learning, pp. 11112-11122. PMLR, 2021.
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2elip: Learning robust audio representations from clip, 2022. URI.
11499
13

---

Published as a conference paper at ICLR 2026
Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang. Bolin Ding, and Bin Cui. Contrastive learning for sequential recommendation, 2022 JEEE 38th international conference on data engineering (ICDE), pp. 1259-1274 IEEE, 2022.
Jie Shuo Chen, Yazhou Ren, Xiaoshuang Shi, Hengtao Shen, Gang Niu, Xiaofeng Zhu. Self-weighted contrastive learning among multiple views for mitigating representation degencration. Advances in neural information processing systems, 36:1 19-1131, 2023.
Haoran Yang, Xiangyu Zhao, Yicong Li, Hongxu Chen, and Guandong Xu. An empirical study towards prompt- tuning for graph contrastive pre-training in recommendations. Advances in neural information processing systems, 36:62853-62868, 2023.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812 15:5812-5824. 5823, 2020.
Junliang Yu, Xin Xia, Tong Chen, Lizhen Cui. Nguyen Quoe Viet Hung, and Hongzhi Yin. Xsimgel: Towards extremely simple graph contrastive learning for recommendation. IEER Transactions Knowledge and Data Engineering, 36(2):91-9926, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference computer vision, pp. 11975-11986, 2023.
14

---

Published as a conference paper at ICLR 2026
LLM Usage We used a large language model (LLM) solely writing aid to polish wording, and improve grammar/clarity. All technical content (definitions, theorems, proofs, experiments, figures, and tables) was authored and verified by the paper's authors.
The complete code for all experiments will be made publicly available on GitHub https///gttubbccommuuuugggeeCCod
CONTENTS
Contrastive Loss
15
Theoretical Proofs
17
B.1
Linear Representation Setting
19
B.2
Kernelized contrastive alignment
22
Implementation
25
C.1
Computation of 8(7)
25
................................
C.2
Experiment details and Convergence visualizations
................................
27
C.3
Sensitivity Analysis
30
C,4
Numerical stabilization for spectral updates in UNICON
................................
31
C.5
Experiments with other modalities
32
C
Kernel
32
Loss Variations
33
A
CONTRASTIVE LOSs
Under the assumption of one-to-one alignment, let Es, X and Egg -' R. denote two modality-specifff encoders with trainable parameters 01, 02. Given N paired samples {(x, y.) ) we form the similarity matrix S [s;g]. All contrastive objectives used in practice can be written in Nakada et al. 2023
the bidirectional general form Tian 2022
N
1
0(),,02)
E
(5 (oy- + + eu (5 ( vss)) R((0,,0)) (23)
2N
with
v Φ: R RR increasing (shape of the loss),
v : relative weight on the positive pair,
Eg E [0, 1] which pairs used.
R: optional regulariser (e.g. weight decay).
Remark Because the embeddings are length Sty E[-1, and all geometry lives on the unit hypersphere.
By choosing specific forms for v and d, we can recover familiar losses. For example, choosing Ø(x) = 7 log(z), (xxx)= exp(x/r) and including positive pairs in the normalization (O;j = for positive pair (i,j)), recovers the RadPoRadford et al. 2021 loss, and the InfoNCE(Oord et al. 2018 loss is the same instantiation appears
simplified variant focusing only one direction. And choosing d(x) = x. (xx) = [-2 c] gives triplet Schroff et al. 2015). Equation 2 thus unifies wide spectrum of contrastive objectives via variant choices of (Ø) 2/ e), providing common lens for analysing and extending multimodal representation learning. This formulation allows for distinct temperature scaling of positive and negative similarities.
15

---

Published as a conference paper at ICLR 2026
Derivation for CLIP (Radford al. 2021 InfoNCE (Oord et al., 2018 Loss.
Set v(z)
e"/T, ((x) log x; 1, Gjj 1 dij in equation 23 and omit R. We define
1, if =
6ij
(24)
0, if
This gives the loss
L
1
Σ Φ
Σ sE(N]
CG (sig v88))
Σ Φ Σ
GGS
(s)s vsss)
R (01. 02)
2n
2n
jE/m]
(25)
T
Σ log
Σ exp
Sij -
+
7
Σ log
Σ exp
8jz
(26)
2n
2n
JE(N
5E(N]
T
Σ
log
Σ -jE(n]
exp
r
+
T
Σ Σ
log
jE[m)]
exp
$33
(27)
2n
cxp
(1s))
2n
exp
T
exp
exp
log
log
(28)
2n
Σje(n] exp
Σje(ni
exp
LCLIP
(29)
For InfoNCE loss, we keep the first term (i.e. only one-directional loss), then
LInfONCE = = Σ
exp St
log
Σ, exp 8ij
Derivation for triplet loss (Schroff et al., 2015).
With a margin é > 0, choose (xz) = [e
a] (0x)) = x. 1, 6zj
dig:
2 1 Σ Φ Σ
c (sis - vsS))
+
Σ
Φ
Σ
Gge ($1s vssu
+ R (01000) 02)
2n
2n
i
(E(n)
jEln]
(30)
1
Siij
s;)]
+ Σ
1=17 iu
[ (s3: sx)]+
(31)
2n
We can also only keep one direction:
12
2
n
L
(si 84) s1]]4]
max 88 s
sij +e]
(32)
j=1
ff1
where sij captures distance between negative pairs, and 8a captures distance between positive pairs.
Equation equation 23 thus provides a general form that captures various contrastive loss, making possible to analyze them collectively and design new variants with principled control over positive / negative balance, temperature, and weighting.
Many-to-mane aligment contrastive loss(Khosla et al. 2020) Note that the loss in Equation equa- tion 23 is defined with (Iz, y/) being the positive pairs for all i. However, in many cases, single I; may have multiple positive pairs. Khosla et al. (220)) extended contrastive learning to the supervised setting with the Supervised Contrastive (SupCon) loss, which is not restricted to one-to-one pairs. This loss encourages embeddings from the same class to be pulled together while pushing apart
16

---

Published as a conference paper at ICLR 2026
embeddings from different classes. Formally, given minibatch of normalized embeddings with labels ys, the SupCon loss is defined as
exp(z; /p/T) Ene A(8) exp(2; 22 za/T)
L out
Σ iE1
out Σ
log
(33)
PEP())
where P(i) = {D E A(i) Up = } is the set of indices of all positives in the multiviewed batch distinct from and [P()] is its cardinality.
Therefore, we extend the general form of the contrastive loss in equation 23 to handle many-to-man many aligmment scenarios. We define the unified form
L(01,02)
Σ
o jEP.(0)
Σ
cijbissi - -8Sa)) + eavlss v8sx))
2n
(
1 2n
Σ
Σ
eip((sH vSNs) Eav((SR vski)) + R(0 02)
(E) (PP,(0
(34)
where (i) and Pj(j) denote the index sets of all samples in {8r J. {ar paired with æ; and V/+ respectively. The term (i) denotes the cardinality of (i) and R is an optional regularization term. We incorporate this loss function in the following sections.
B THEORETICAL PROOFS
Theory Road-map
Definition 3 Contrastive Similarity Weight Matrix S(7)
Linear
Non-linear
Theorem Kernelized Spectral
Prop. 7
Lemma Gradient bridge
(linear specialization)
characterization (unified form)
Theorem
Spectral Characterization
Cor. 10 Kernel inference (out-of-sample)
Nakada et al. (2023)
Definition 3 (Contrastive similarity weight matrix) Consider the general contrastive loss L(O, 02) in Equation equation 34 with choice of (ó, e, and a batch of samples {(x,, yx y Denote He as the elementary basis vectors of RR. The contrastive similarity weight matrix is then defined as:
S(7) = Σ
Kk
Rzs
e,e,
(35)
TP,(i
with weight coefficients
Ø'1 (ss(1 (c)(1-v))((1-2)) Empgr Cim ('
-uss))
ifj E Px(1)
(36)
EbEP2(9 Hi (e1 vssk))
f
17

---

Published as a conference paper at ICLR 2026
8'j (eH(1-
- Empp,( mmil' -ss))))))
ifj E Pa())
Tig
(37)
ExEP,(S) Ri. (s - vsss)).
where we define
4 (0mm voss)
13 =
+
Σ
(38)
m²P=(0)
13 =Φ' C100(1- ())jj
(Smss vSJs)
(39)
Lemma 4 (Gradient Equivalence) Consider minimizing the general contrastive loss (see Equa- tion equation 2. the gradient of the contrastive loss with respect to encoder parameters satisfies:
Oc Øtr (Fe, (X) (Y))
2R (01,02)
E (1,2)
(40)
00
00
00₂
(01.8)))))
where
X [x,, xx,] E 1d xn Y --y.. -y. , ynl Rd
Fo, (X) = (x1) fo, (X2) ... fo, (xx)) E Rrxn
FY) [Cog (y1) fez (Y2) ... (y))) E IR"x".
Proof. Let Ou.e be the 8-th component of Oz. We have
00y,1 L
1
á cuv m)ss))
Σ
th (sim v8ss))
2n
(i)
kEP
mpPa(()
+
1
Σ
Σ
cmoo((1 v)sss) +
Cmith (ams vsss))
12 (0., 02)
2n
[P(()]
LEPp(0)
mgPy(()
(41)
Σ o' (cvi((1- (11+) +
Σ
cimeth (sim vsek)
2n
(o(
kEPz())
m (gP
E ((1 w)sth (1 v)
Eimill (8im v8ik) (Do Aim vūo 8it))
Σ &
FLERi((1 - v)BE))
Σ
Emi (8mi MAEE)
2n i=1
Pi(
kEFy(i)
mEPa(1)
CNEc ((1 - v) v)ssss(( (1 v) 10 +
emait' (smi vski) (Bnr sms - vdne. vūne, skie
So R
mgPp(()
(42)
Σ
Dnc, 8im
2n
(ili
kEPs()
PgPPa(t
1
1
Σ
Fando Shi Ski +
Fum On,, 8mi
Sn.
(43)
2n
P((
(t)
mgP
3st
Oep. Stj
(44)
2n
(.(
P.O
18

---

Published as a conference paper at ICLR 2026
Define
Φ/ss (cts(1 (101--))'(111))00)) - - Empps(o) Cim Cim - vSss)..
if E P=(1)
(45)
ExePs()) Pie (s5) v8ss)),
if P.(1)
Øi (5x(l (c)((1-)ab>(( - -))sfi) E (mp Emoli -mm)))))
if P₂(1)
(46)
Exerp Pie (esna) vsNs)),
if $ Pa(1)
where we define
Bi =
61 (v))))
E
Cim (/8m mmis)
(47)
meP.(()
Kig =
c5r8(11 (100(1 -2)sj
Σ
Cmeth(8ms v88j))
(48)
m mpP,
To simplify the notation, assume that the encoded representations Le, (Irs) and fo, (Ms) are already 4 normalized. Then the gradient follows that
3o 6
e0 (o, (U,)) + do R
(49)
2m
P=(1)
(P(
=00.
S(7) (x)
do,,.
(50)
where (F%, (X) f (Ir, Lo, (w;) denotes the similarity between sample æ; and yy, and S(>) denotes the entry in the i-th row and j-th column of matrix S(x), which defined Equation equation
Note that
tr(A B)
(A Dau Bug = - A Bas
(51)
ims
Therefore have
0C
(tr(S(n) Fg Fg,(()) (X)FFo (Y))
aR(01, 02)
(52)
00
00
00
atr(Fo, (X)S(z)FF 02
(Y))
OR(01,02) 0z)
(53)
00
00
where S(7) is defined Equation equation 35
B.1 LINEAR REPRESENTATION SETTING
In this setting, the hyper-spherical similarity between a pair 1x) is computed as the inner product of their normmalleee embeddings:
| (I [Fy,,ll2
Fix:
F2N,
x3 FT Fay;
Sij (Fix,, Fays)sr AC
(54)
1[ F2y,
R
Proposition 7 Under the linear setting, the Lemma 4 is specialized
ac =
0tr (F,XS(n) FT)
aR (F1, Fb) +
k [1,2]
(55)
0Fr
OF
OF
(=3(F).F))
0tr (FIC())
RR (F F2) FFk
k: [1,2]
(56)
0F
B=B(F)
19

---

Published as a conference paper at ICLR 2026
where C(v) = XT())
To solve the optimization problem induced by our reformulated objective, we characterize its maxi- mizer in the linear setting. We arrive the following theorem, which establishes that the convergence of the contrastive loss can be replaced by a closed-fort update.
Theorem 8 (Spectral Characterization Nakada al. 2023 Consider minimizing the contrastive loss function C F2), with R F1 F2)
F F2
Then,
min PIRTT F 4
(FF.
(57)
arg max FiERrxe), FEERRK4
(RCC())) (0/2) Fell
(58)
- (F), E Rexd, F F2
1 Σ iinio
(59)
where (o,, Wi, vi are the top-r singular values and vectors of C(7) according the Sekart-Yungg-irsh theorem.
Proof. Observe that
Ir (F(CCG)))) - (0/2) Foll
(60)
-tr(R((()))) tr [F] Fi FiFT F2)
(61)
=(r(F(C()))) tr (F. FiFT F2)
tr (c())) + tr (CCG)T(())
(62)
2p
2p
(CC()()))))
p tr [FT (())
F. F2 _.(m))
(63)
2p
2
||(
- - F. F2
(64)
2p
The first term is constant for fixed C(7). and the second term is minimized at FTT F2 C)()). Since FL Rexdi F2 E Rrxd2, F,T F2 has rank most Thus, the minimization be achieved Fl F. = Ei= Juu.y by Eckart- Youngrt-oun---irsk theorem for low rank matrix approximation. Here (o,, ui, U are the top-r singular values and vectors of S.
In summary, linear case, the global minimum is attained by projecting the contrastive covariance C(r) onto its top-r singular components. Thus, gradient descent on any loss in the contrastive family (é, w) merely tracks the dominant singular subspace of C(v). UniCon performs this update in closed form, replacing thousands of SGD steps with one spectral factorization.
Relationship with (Nakada et al. 2023) Furthermore, the formulation of (Nakada et al. 2023)
can be seen as a special case of our contrastive similarity weight matrix S(y). which arises under the specific assumptions of one-to-oms alignment in a linear representation setting and with further restrictions on the functions v and c. Similar to the paper (Nakada al. (023), we define
aij 2 Egp
Σ meini
Cim (sim vsS)) v/ (sig vsu),
(65)
āij A egp
Σ mE[m]
Cim (8mi vsss)) 8// (893 vSs))
(66)
20

---

Published as a conference paper at ICLR 2026
Then derive that when (P) [Py())) 1, for positive pairs (2s, VU) with = j.
(7h + %+s)== =t
<v1i((1 (wvi((1-usss)
v)8+) +
Σ
cmm (Wam vs(s)
myfli
x
6:(1 - 1) <u(1-v)0((1 - -))s)) - v Σ
Ci (sim vssi)
mikhs
0 6110(11 - -v)su) +
+
Σ
Eme - v8ss)
x
6u(1 - (o(1-v)e'(0) - v)sss) - v
E
Emit/((sms - vsús)
(67)
mgti
Ci fmm - vsas)
(u 1/85s) - 1/ Σ
Cim (sm - 1351s)
m=1
m==1
(Gall(su - vsis) - Σ
+ ø'
Σ m=1
Cmi (smi - vsss)
(smi
(8mi - USss)
m=1
(68)
=0. + Olge
(aim Gam)
(69)
For negative pairs (Is; Uj)
with i + j.
Yig Tzi =o
c v)si +
Σ Cam vsui) (Gu)(( vss))
+ of
9g1b(1 - 1)855 +
Emjth v8js) (s1) v833))
(70)
mfj
Cim 188m vssss)
w ('(si) - vssi)
m=
+ egO
Emy (Smm - -183)) (s - vsss)
(71)
m=
-0i + āji
(72)
Then we can define
Oij + āji
Oij + āig
au
Big
3,
(73)
2
Thus have
n
C(7) X
(my + 3,)0,1]
Σ Braay
Σ Bejorey
(74)
2n
Nakada et al. (2023). they define the contrastive cross-covariance S(3) as:
S(B)
1
Σ Bijayy]
(75)
Cn
Cn
it)
21

---

Published as a conference paper at ICLR 2026
+ Ogj
Oij + Oāj
Bij = 2
B =
1.
(76)
2
jE[n]
Therefore, with Cn our C(t) is equivalent to S( in Nakada et al. (2023 corresponds to the special case of our definition with identity functions for @ and u.
where their S
Understanding C(y):
Consider the trace objective function
1
Tii + T3 (F)(x.), F2(y,))
u (())))
(77)
Every similarity inside batch is multiplied by scalar 26y Pu1,
Sia on the diagonal strengthens the attractive force for the positive pair Yi):
2
(i j) on the off-diagonals weights the repulsive force for negative pairs.
where:
Via + T3s
Ozg + Caj
Yi +
Oijj
+ O4j
Ogi
+ az
(78)
2
2
2
2
Oij and αij encode the bidirectional importance of the pair (i,j).
v adjusts the influence of positive pairs relative to negatives.
The plus/minus pattern makes the "pull" vs. "push" behaviour of contrastive learning transparent.
B.2 KERNELIZED CONTRASTIVE ALIGNMENT
RKHS parameterization. Let (Hxx kx) and (Hy, ky) be RKHSs with kernels xx, by and canon- ical feature maps
xx(xx) == xx(xx),
Øy: by:Y4 Hr. oy kr(x)).
(79)
They satisfy the reproducing property:
Af E Hx, Vx X:
f(x) (f, (x))
(80)
Ag E Hy, vy EY:
gly) (g, dry (y))
In particular,
(ox (x:), (x (x))xx kx (xi, x),
by by kv
(81)
By the representer theorem, for each output coordinate I,...,r.
1/ (a)(() = Σ kx f/0
Σ j=1
Bja ky
(82)
and we stack coefficients into matrices A, B E Raxr (column stores the coefficients of coordinate a).
Define
xx xx(x) = |x (x1, x).... kx (x,,, (x,,,,
x)] E RR"
ky = V/.
ER..
(83)
Then the r-dimensional encoder outputs at a point are
fo (x) = (x), (x)=(x))..., (x)) A (x)
fy, BJ BT PT ky) E RR.
(84)
22

---

Published as a conference paper at ICLR 2026
Theoretically, in infinite- -inntt--admeniionlll RKHS,
(x)
Aia xxx (Xi, ) ox (x))
(85)
Hx
Computationally, in finite n-dimensional representation,
16. (x) = A xix (x).
(86)
Let the Gram matrices be
Kx (2x E R=
= (, 8,)Ri=1 E R=x"
(87)
Stacking the n samples columns, the batch embeddings are
Fa, A A Kx ER
(Y) BB B Kr =.
(88)
Inner products and induced norms. For any c, e, R.
E (x(xx))) Hx
Kxd,
D [ c Kxc.
(89)
and similarly with Ky.
Therefore, the total RKHS norm of the r output coordinates is
Σ E
Σ AxaxA -ax
KxA),
(90)
and analogously for B with Ky.
Similarity. For samples Y;) the predicted similarity is
(fa, (x:), fo, (v;)) = =xx(x;) AB (y,),
(91)
which is exactly the (i, entry of
Spred == Fo, (X) (Y) Kx ABT Ky E gux..
(92)
with entry [Spredlis = (Fo, (Ii), (o, (
Definition (Kernel cross-covariance regularizer). Define
R (A, = L
Ky = 1| /2 AB K 11?
(93)
The second equality in equation 93 follows from the identity | B' tr A B') with
A' = AK 1/2 and B' = BK3/2
In the linear case, fo, (X) = FiX. Let wa) be the a-th row vector of Fi, then f/. (x) = w(a)) x.
thus
((a)
(1()) (la)) ) THx
7x
(a)
(a))
(94)
( 112
23

---

Published as a conference paper at ICLR 2026
Proposition 12 (Linear-keree reduction). Suppose kx (2, x') (a. and ky (v. 1 (y.v"), and let the sample matrices be X = [Ti.....200 Enl Kx X X and Ky Y Y. and with
E R Rd xn Y = Iir..... ynl E R2 Then
F1 = A E RRx
: B B E Brxdo
(95)
we have
R B) = (r Kx (A - - F.B.
(96)
Proof.
With Kx X X and Ky = Y Y we compute
FFT =AA (XXA) x)A A'KXA,
FET =BBT =(BTT)(YB)) = (YB
Therefore,
= -AA = (A,
Theorem 13 (Kemelized spetral charaterization (unified form)). Let > and define the regularizer
R(A, B) 2 A)T (K3/² B) [| Then minimizing the contrastive loss is equivalent to the kernelized maximization
tr(A Kx Syn)KyB) P 11 (K
12 A) ( B)
max A,BER"x*
(97)
Let
A' = A ER" B²: B K2 E R M = x2 S(7) K2/² E ROx
(98)
Then equation 97 rewrites
max Lr (T(A'MBT) - IIATT B A'B
(99)
M UEVT is an SVD and Mr Ei- Jrney its best rank-r approximation by Eckart-Youg- theorem, then all maximizers satisfy the relation
BA' + AB
K M, 1/2
(100)
P
(If Kx or Ky is singular, replace inverse square roots by roots.)
pseudo inverse square
Proof. Insert K 1//2 Kx and k Ky-² Ky into the trace term and set A' AT k22
B' = B K M k/) to obtain
tr (A Kx5(1) - (p/2) (K A) B)
(101)
=tr ( - (p/2) (4 k (B k2/2)
(102)
= -tr (A'MB'') TT
(103)
Now same steps in proof of Theorem complete the square in A B' to see that the maximizer aligns the column spaces A', B' with the top singular vectors of M, yielding AT B' = p-'M, Undo the change of variables to get equation T00
By Theorem 8 the optimal solution satisfies
(A, AT B' = -1- M,
(104)
(A, : (AT (AT T (BT kス} [
(105)
(A, B): K k²= 1 k k (()
=
(106)
1 1 k K 5k
(A, ABT
(107)
If Kx Ky is singular, Moore-Penross pseudoinverses K K
24

---

Published as a conference paper at ICLR 2026
One explicit optimal choice is A Kx U, and B* Ky 1/2,, Ex/p, where U, -u Ur] E, = diag(o),...,.0 .r) are SVD of M.
V, [r),..., Ur],
Corollary 14 (Kernel inference (out-of-sample)). Lei {(x, y:) the reference batch used build contrastive similarity S()) and let k be positive-deffiite kernel. For a new pair (x" yy)) set xx xxx) = [ (I1, "011,x"),, xz"...... b....,,ky xx Tn x²) and Ky (y)) = [ky
(1yy(/1. (11.8")) )......, by (Nn. y²)] With
an optimal (A*, B²) from Theorem
lo (x="
lo2(y)) (B*) (oYxY(")))
(108)
and the similarity
(x(x") A* (x(
s(x",8")
(109)
||A xx (x M12l|BB* bY (2^)122
C IMPLEMENTATION
The complete code for all experiments will https//gghhbb.ooom/siiiann///u/t
be made publicly available on GitHub
C.1 COMPUTATION OF S(n)
In this section, we provide the PyTorch implementation of the contrastive similarity weight matrix S(7) computation used UniCon. The function below computes the matrix S(7) in one-to-one paired settings, and generalized many-to-many settings, where similarity values Sij used to form the weighting terms Tij. We have also provided a complete pseudocode outlining the training pipeline of UniCon in Algorithm 2 which offers step-by-step description of our method's implementation to facilitate reproducibility and deeper understanding.
Algorithm Discussion. In UniCon, efficiency claim refers the rapid stabilization of the mapping ((A, B)) parameters toward the desired alignment subspace through derived-fomm spectral updates, which bypass many small gradient steps. Specifically,, we reformulate the minimization of the contrastive loss as an equivalent maximization problem, using the proposed contrastive similarity matrix S(7). We will state in the unified nonlinear form, as the notations can be reduced to linear case analysis in Section 3.3. The general objective is max (r(A") K KxS()) Ky B)
[]I(K> ()T(K/ B)I.?. The optimal solutions satisfy ABT 1/ [()) K/ where [,,, denotes the best rank-r approximation. Note that S(7) is itself function of (A, B). Therefore, the overall optimization becomes fixed-point problem. We solve via an iterative procedure, such as,
J, 1, ky
A(+1), B(t-1)
P 1 S(7; B()Kk/1),K
In this process, we observe rapid convergence of (A, B) to a stable solution. This rapid stabilization, subspace convergence, what refer to efficiency.
def compute_$_gamma
8, datal_batchh data2_batch, tau=1.0, nu=0.1,
psi-torch.exp, phi-torch..ogl. pailicn_ij=11 epsilon_sioni. di diff_psi-torch.exp diff_phi-lambae
eps-1e-8: 1.0 / (1.0 + eps)
):
n = datal_batch. (0)
10
Build epsilon mask for weighting epsilen epsilon_ii torch.eye (n, device=s.device epsilen ipsilon_ij (1 terch. (n, devico=s.
12
13
14
15
Row-wise similarity terms: a11 nu siif s_diag_rov torch (s) unsqueeze (1). .expand (-1, s_diag_rea
s_nu_rew 17
25

---

Published as a conference paper at ICLR 2026
psi terms psi row)
19
sum_psi_terms torch sum (epsilen psi_terms, dim=1, keepdin=True)
20
diff_phi_terme aiff_phi (sun_psi_terms)
diff_psi_tore sttms - diff_psi (s_nu_row) alpha = epsilon diff_phi_terms diff_psi_tenne "ermss
24
Column-wise similarity terms: 5_ff nu s_ii
25
s_diag_col torch. (s) exxand n) s_nu_col = nu s_diag_col
26
psi_terme bar (s_nn_coll)
28
sun_psi_terms__at torch strch..summeee. (epsil psi psi_terms_bar.1 dim=l, keepdim-1rue)
29
diff_ehi_terms_bne diff_phi [sun_psi_terms_aat
diff _eff_psi_tt__ terms bar = diff_pst s_nu____________________ee___ pu__uuucoll_ col.T)
diff_psi_terms bar alpha bar epsilon diff_phi terms
32
33
Compute g_gamaa woights S_ij = (alpha alp.a_bar.t / 2
34
torch. (alpha + alpha_bar) dim=1) torch. alpha alphe_bar) bar)
$_i
36
37
38
gamma = ii /
gamma range (n), range (n) =S_
40
41
return S_gamma
Listing 1: Contrastive Smilarity Weight Matrix Computation (one-to-one case)
def Laaputtt____aama_geeerellll (s, pos jos_mask,
batch,
nu=1.5, tau=1.0, www-torcccc.cccccccmmmm...cccc exp, phi-torch. loglp, apsilon_ij=l, apsilon_ii=l, diff_psi-torch. diff_phi= lambda x, eps-le-8: 1.0 (1.0 eps))
normalized__dtta__bee
n shap device s. deveice
Create epsilon matrix epsilon torch. ones (n, n, device=device) epsilen_ij apsilen. fill
Create masks neg wws.mask___aaaa base "pos mask
11
12
Expand tensors for broadcasting: (n, n) where dim 0 is i, dim I is j, dim 2 is m
s_i_m s.unsqueeze (1) expand n, $ s[i,m]
14
s_i_j s.unqqueeze (2) expand n, n) $ s11,f]
15
epsilen_i_n ____n____________ epsilon.unsqueeze
unsqueeze (1) expand (n, n, n) $ epsilon[i,m]
16.
mask_nct_i = has_mask_baseeee unsquezese (1). expand (n, n, n) mask for
not in [i]
$11,311 psi_terms terms epsilon i psi i nu s_i_)) mask not
Compute psi terms: epsiion/i, psi (s/i,m/
18
19
sum psi = psi terms. sum (dim=2) # (n, n), sum over m
20
Compute diff_psi terms
diff_psi_ternt
epsilon_i_m diff (s
sil_))
mask_not_i
22
sum_diff_pss - diff_psi_teeteeeeeee stttemsssumm/ae (dim=2) (n, n)
23
Phi arguments
phi Lai_args- = epsilon psi ((1 - nu)
+ sum psi (n, n)
Gamma for positive samples
26
gamma_pes pes diff_phi (phi_args)
27
epsilon + psi ((1 nu) + 5) (1 nu) nu sum_diff_ps
28
30
diff diff phi
unsqueeze (1) exxand (n, D, n)
26

---

Published as a conference paper at ICLR 2026
gamma neg diff phi (epsilon diff psi (s_
32
pos_mask_k pos_mask. unsqueeze (1) expand (n, n, n)
33
Cemma_neg ( pos_mask_k] sum (dim=2)
34
For gamma bar positive samples
36
s_m_i ..I.unqquezz (1) expand (n, n, n) a i]
37
s_i_i s.T. unsqueezee d expand (n, ny n) s[j,1]
38
epsilon_me mi epsilon. w.wloonT.uunaa. wnsqueezee (1) expand (n, n, n) epsilon 1/ Compute psi terms for gamma Barma_aar
psi terms bar epsilon sum_psi_bar psi_terms_bar (0 Compute di diff_psi terms for gamma_bas (no epsilon in second sum) di diff_psi_terms_ sum ciff psi bar = diff pllpppsi_srp__e terms bar. Sur. (dim=2)
psils m
i_ $ (n, n), sum over
mask_not__
42
43
diff_psi (s_m_i i- - nu s_l_l mask_not_i
(n, m)
Phi arguments for gamna bar phi arga bar epsilon. psi (11 nu) S.T) + sum psi_ bar (n,
46
47
gamma_bar for positive samples gamma_bart Lacla_bbr_poo_ diff phi (phi_args_bar
epsilen. diff_psi ((1 nu) s.T) (1 nu) nu sum fum_diff_psi_____
50
51
diff i diff phi (phi args bar) unsqueeze (1) expand n, n)
53
gamma bar neg diff phi k (epsilon.I diff psi (s_i_1 mu
8 s_m_))
54
pos pos_mask_k pos_mask.unsqueezell (1) .xxpand (n, n, n) pos_mask_k) sum (din=2)
Lamma_bar_nne (gamma_bar_
57
Combine positive and negative samples
58
gamma = torch, where (pos_ (pos_mask, gamma_pos, gamma_ney
59
gamma_bar = torch. forch.where (pos_mask, ganma_bar pos, gar_a_bar__egi
61
pos_mask_rov_
(dim=1)
62
pos pos_mask_col sum pos pos__ask.
mws.maskkkkk musk.aumddimo0)
63
pos Baa___ss__oo_ss__e_______
unsqueeze (1) expand (n,
pos_ _aa_a___c__u___eepddee pos_mask_col Baa_aakK_o_ssumuuuueeeeeee unsqueeee (0) expand (n,
65
S_ gamma - (gamma pos mask rew sun sun_expanded + ganna bar. pos _a__mass_cco
67
68
C_n n # Normalizaticn constant
69
S g_mama S_ganma / C_n
return 8_gamma
Listing 2: Contrastive Similarity Weight Matrix Computation (generalized, support many-to-many matching)
C.2 EXPERIMENT DETAILS AND CONVERGENCE VISUALIZATIOONS
We present additional plots illustrate the convergence behavior of the SGD-CLIP bascline across three experimental settings: synthetic latent factor models, unimodal image clustering, and multimodal image-text retrieval.
These visualizations confirm that our reported SGD-CLI performance is after sufficient training. providing a fair comparison against our proposed Unicon method. While SGD-CLIP ultimately achieves high accuracy. it requires many iterations to converge-underssonee underscoring the computational inefficiency of iterative optimization when compared the efficient closed form update of Unicon.
27

---

Published as a conference paper at ICLR 2026
Algorithm Training Pipeline for Linear Case
Require: Initial P1; F2. dataset {(X,, Y=)
Ensure: Trained F and F2
Initialize Fi - FP. F2 F2
2: while not converged do
3: (F1 F2)__sun t- 0
|accumulator for batch-wist wise F1 F2)
4: for each batch (X,, Yi) do
5: similarity - X.) (P2Y;)
6: S(1) compute S gamma cammaasimmlarityy X,, Yi)
(11 F2): x (,))
7:
8:
weight, validation( F2)x)
9:
(F) F2) sum +- (FT F2) sum (FT (F). weight,
10: end for
11: Update F2 based aggregated (FI F2)_ sum
12: decompose Fi and F2
13: end while=0
Algorithm 2 Training Pipeline for Nonlinear Case
Require: Initial Ag, Bu, dataset ((X,, Y))
Ensure: Trained A and B
1: Initialize + Ao. +- Bo
2: while not converged do
3: AB AB_sum= <
[accumulator for batch- wise AB
4: for each batch (X, Yi) do
5: Kx Ky, < kernel matrices of X,, Y;
6: similarity +- (A (B Ky,) S(1) + compute_S gaaaaaaammiaannn
7:
X,, Y,) K A(n: B(t) bK) Ky.
8:
(ABT). x
x;
1/2
1/2]
1/2
9:
weight, +- validation (AB )r)
10: AB_sum < AB_sum + (AB ); weight,
end for
12: Update A, B based on aggregated AB sum
decompose A and B
14: end while=0
28

---

Published as a conference paper at ICLR 2026
Synthetic Setting: Linear Latent-Factor Model. We generate synthetic data from latent vectors z E IR' that are sampled around K cluster centers in latent space.
The observed pairs y) are linearly projected from using orthogonal matrices, followed by additive Gaussian noise:
x Uiz E1,
y U2 E2.
(110)
Here, U1 E Rd and U2 Rd orthogonal projections sampled from the Haar measure Od, and Og respectively, with d1 = 40, d2 = 30, r 10. Noise vectors Ç1, &2 are sampled from
with SNR = 0.3. Each of the N 600 training samples is drawn from one of K clusters in latent space. C the accuracy score of SGD-CLIP method converges across training epochs.
Synthetic Setting: Nonlinear Latent-Factor Model. nonlinear transformation of the latent space:
We further evaluate the method under
I = tanh(U + E1),
y= tanh((UUz +2))
(111)
where U1, U2 are again uniformly sampled from orthogonal groups Oa, and Odz. taking the first columns. The additive Gaussian noise is drawn from (0, SNR²) with SNR 0.3.
(a) Linear latent factor model
(b) Nonlinear latent factor model
Figure C1: Convergence of SGD-CLIP. Training accuracy over epochs for linear and nonlinear synthetic settings.
Unimodal Classification (CIFAR-10). Figure C2 reports the training loss of SGD-CLIP in the CIFAR-10 10 image clustering task.
Figure C2: Training loss SGD-CLIP on CIFAR-10. Unimodal image clustering task with frozen ResNet-18 features.
Multimodal Retrieval. On Flickr30K, we shuffle the dataset into to 25426 (80%), 3178 (10%), and 3179(10%), for train/validation/test. Every image æ and text y is embedded once with a) ResNet-18 (He et al. 2016) for images + Sentence-BERT (all (all-mpnet- -mm-mmmtaaa-eee2222---- (Reimers & Gurevych 2019); b) ResNet-50- + Sentence-BERT: c) the CLIP VIT-B/32 model for visual-textual feature extraction as frozen backbone. Matching is performed in shared embedding space of dimension r = 128 with 5 = 1. SGD-CLIP runs for 50 epochs, and run-time is measured wall-clock. Figure shows loss curves for SGD-CLIP trained on Flickr30K across three backbones: ResNet-18, ResNet- 50, and CLIP ViT-B/32. Despite faster convergence with stronger backbones, all variants require many epochs reach stable loss values, whereas UniCon completes alignment more efficiently.
29

---

Published as a conference paper at ICLR 2026
(a) ResNet- 18+ SBERT
(b) ResNet- 50 SBERT
(c) CLIP VIT VIT-B/32
Figure C3: Training loss SGD-CLIP on Flickr30K. Loss curves for three backbone architectures in multimodal alignment.
On MSCOCO, follow the standard retrieval protocol on MSCOCO. The training set contains 82,783 images, each paired with 5 human captions. We validate on 40,504 image-tert -ext pairs and report test results on 5,000 held-out pairs. We report Recall and Recall 10 for both directions.
To further ensure sufficiency of training, we re-evaluated validation accuracies on MSCOCO with backbone Resnet 50 SBERT in FigureC4] it even begin to decline after the peak, suggesting potential overfitting with additional training. Specifically, on MSCOCO, we trained SGD) for 1000 epochs and observed that the best validation performance is already reached around epoch 300. Beyond this point, the model does not improve further. For comparison, extended UniCon to 20 iterations across all batches. We observed that model norms stabilize after just iterations, with only minimal fluctuations thereafter.
Table 3: Image-text retrieval on MSCOCO. We report Recall @1 1 and Recall @ 10 for both image- -txx and text- directions. UniCon achieves superior accuracy to SGD-CLIP with --96-661 x faster training.
Image TTeg
Text-+Iage
Average
Backbone
Method
Train time
R@D 1
R@ 10
R@1
R@ 10
R@l
R@ 10
SGD-CLIP
5121.72
.053
.253
.060
.286
.057
.270
RN-50 SBERT
UniCon
11.11 s
.105
.388
.129
.439
.117
.414
SGD-CLIP
1066.60
.128
.415
.123
.427
.126
.421
CLIP VYT-B/32
UniCon
11.15 s
.329
.685
.292
.644
.311
.665
Table 4: Zero-shot image-text retrieval FLICKR30K (trained on MSCOCO, fine-tuning). We report Recall@5 and Recall 10 for both directions; higher is better.
R@S
R@ 10
Backbone
I+T
T-1
Avg
I+T
T-+l
Avg
RN-50 SBERT
.249
.261
.353
.307
CLIP VIT-B/32
.808
.766
.787
.879
.848
.863
C.3 SENSITIVITY ANALYSIS
Robustness to batch size. We discuss that the batch size n does not significantly affect the total model performance. We extensively evaluated the effect of batch sizes on all the experiment settings, showing robustness of UniCon to batch size variations. For multimodal alignment, we experimented retrieval task using Flickr30k and MSCOCO varying the batch size across [100, 500, 1000, 10000, 20000]. For unimodal alignment, we experimented clustering using CIFAR-10 with batch size [200,300,400]. We observed that performance metrics remained nearly identical across
30

---

Published as a conference paper at ICLR 2026
(a) Loss
(b) Validation Accuracy
Figure C4: Training loss and validation accuracy of SGD-CLIP on MSCOCO trained for 1000 epochs to check convergence.
these ranges. Importantly, the retrieval performance was robust to batch size variations, which implies data efficiency.
Batch aggregation strategy To reduce memory and computation overhead, we adopt a batch-wise training strategy. We evaluate several strategies to aggregate multiple batch-level models into a global predictor, including:
Accuracy-weighted fusion: Normalize validation accuracies a, to weights wy Σ, az
and
linearly combine predictions.
Noftmax-accuracy weighted fusion: Apply softmax over {a:} smooth weights.
Majority voting: Select the most frequent prediction across batch models.
From our experiments on both unimodal and multimodal settings, we observe that different aggrega- tion strategies yield similar performance, with variations within a 1-2% gap. Its performance under extremely biased imbalanced data distributions remains an open question.
We also evaluate the impact of statistical differences between training batches on CIFAR-10. For each batch, we form paired inputs datal and data2 under two settings: (1) Random: For each sample pair, we independently sample classes for datal and data2, then select an image from each chosen class. This independent sampling creates varying class distributions across batches and introduces inter-bated batch differences.
(2) Balanced: We iterate through all 10 classes within per batch, sampling two images per class for datal and data2, ensuring balanced and identical class distributions within and across batches. From experiments unimodal settings CIFARIO, find that the random and balanced sampling strategies yield similar performance, with differences within 1 1-2%.
C.4 NUMERICAL STABILIZATION FOR SPECTRAL UPDATES IN UNICON
Stabilized SVD. We analyze the numerical profile of C(7) Rd xde On large, high high-dimensional tasks, the raw ((r) often exhibits rapid spectral decay, small singular-value gaps, and large effective condition numbers. To stabilize the closed-fome spectral step, we tested the following techniques:
Tikhonov regularization. Add Ala to C(7) to improve conditioning and stabilize SVD.
Randomized SVD with power iterations (Halko et al. 2011). Use randomized SVD with
power iterations to efficiently extract the top-r components.
Unit-hypersphere normalization. Before forming similarities/covariancee, project embed- dings onto the unit sphere, matching the contrastive geometry.
Symmetric case (unimodal). When a unimodal subproblem reduces to estimating a sym- metric target (e.g. solving FT F in single modality), use the symmetrized and ridge-shifted matrixand then apply eigendecomposition.
On the large-scale MSCOCO retrieval benchmark, compare baseline that uses plain truncated SVD on C(y) against our stabilized pipeline above. The latter yields higher recall with negligible overhead.
31

---

Published as a conference paper at ICLR 2026
Table Effect of stabilization on MSCOCO (image-text retrieval). Stabilized SVD regulariza- tion randomized SVD (with power iterations) + unit-sphere normalization.
SVD method
Train time (s)
R @
R@5
R@ 10
Standard truncated SVD Stabilized SVD (UniCon)
32.47
0.2235
0.4486
0,5649
33.28
0.2601
0.4990
0.6149
C.5 EXPERIMENTS WITH OTHER MODALITIES
To further substantiate the general applicability of our approach with other complex modalities, we additionally evaluate UniCon on an audio text alignment task using the Clotho (roosos pre-trained CLIP and Wav2CLIP Wu et al. 2022) encoders
et al. 2019). In this experiment, to extract features from text and audio inputs respectively, followed by a linear projection layer for cross- modal alignment. The results show that without explicit alignment, the original feature structures exhibit a significant modality gap, while both UniCon and SGD achieve comparable and effective alignment performance after training. This additional experiment provides further evidence of our method's effectiveness in diverse modality alignment scenarios.
We believe this experiment with different modalities provide further support for UniCon's scalability and robustness.
Table 6: Audio-Text Alignment Results on the Clotho Dataset.
Method
R@1 a2t
R@1 1 t2a
R@5 a2t
R@5 12a
R@ 10 a2t
R@ 10 t2a
Time
No alignment
0,0010
0.0000
0,0048
0,0029
0.0096
0.0096
UniCon
0.0335
0,0249
0.1311
0.1110
0.1943
0.1789
13.45s
SGD-CLIP
0.0373
0.0278
0.1244
0.1139
0.1923
0.2077
347.48s
C.6 KERNEL
Why kernel? The kernel-based formulation is essential in nonlinear contrastive alignment, as enables an implicit mapping of feature representations into (potentially anfnite-dimensional) Reproducing Kernel Hilbert Spaces (RKHS). This implicit lifting significantly enhances expressivity, allowing UniCon to capture complex cross-modal relationships that cannot be represented by linear projections alone, without explicitly constructing high-dimensional coordinates. Moreover, the kernel mapping effectively unfolds nonlinear manifolds (e.g., spherical or curved distributions) into linearized feature space, where the spectral alignment mechanism can operate directly via rank-r approximation.
Why Angular Kernel? We adopt angular kernels because features are normalized on the unit hypersphere, which is an effective practice contrastive learning. Prior work work/Wang al.) (017)) has shown that learning representations on the hypersphere leads to better performance than in Euclidean space, as avoids the conflicting forces between attractive and repulsive gradients. Angular kernels are particularly well-suited for this geometry: they are theoretically sound, and simple to implement. In our view, this simplicity not a limitation but an advantage. For comparison, we also experimented with the RBF kernel. The results confirmed our hypothesis: angular kernels consistently outperform RBF when embeddings lie on the hypersphere. still worth to explore kernel approximation methods for memory efficient computation.
Kernel Selection Study. To further demonstrate the impact of kernel choice alignment perfor- mance, we have included an empirical study in Table 7 The results below show the performance variance across different kernel types:
32

---

Published as a conference paper at ICLR 2026
Table 7: Ablation study kernel selection. Results demonstrate that kernels with stronger geometric expressivity (e.g., Angular and Arc-Cosine) yield superior alignment performance.
Kernel Type
Synthetic Accuracy
CIFAR- 10 Accuracy
RBF
.66
.11
Matérn
.73
.44
Cosine
.81
.63
Exponential Cosine
.73
.63
Arc-Cosine
885
.63
Angular
.86
.63
C.7 LOss VARIATIONS
Support for Non-Smooth Losses (e-g., Triplet Loss). Our generalized contrastive loss formulation accommodates both smooth and non-smooth cases. For losses such the hinge-based triplet loss, classical gradients are not defined everywhere. yet their optimization well-defined using Clarke subgradients. The rank-r spectral characterization of UniCon still applies in this generalized subdifferential setting. To support this, we performed a synthetic nonlinear experiment (same setup Sec. 4.1, replacing CLIP loss with triplet loss), where UniCon achieved 90 % alignment accuracy, confirming its compatibility with margin-based losses.
Sigmoid- blased Contrastive Losses. We further evaluate UniCon under the Sigmoid contrastive loss used in SigLIP et al. 2023). The results show that UniCon achieves performance comparable to SGD-SigLIP, demonstrating that UniCon is not limited to softmax- based contrastive objectives.
Method
12T R
T2I R@1
12T R@5
T2I R@5
I2T R@ 10
T2I R R@ 10
UniCon (SigLIP) 0.3340 0.2852
0.2862 0.2816
0.5816 0.5610
0.5334 0.5538
0.6852 0.6924
0.6394
SGD-SigLIP
0.6704
Table 8: Comparison of UniCon and SGD-SigLAA under Sigmoid contrastive loss.
33


Reviews:
[{"review_id": "RWHbSDS6E6", "reviewer": "Reviewer_GRPF", "summary": "This paper presents UniCon, a novel and theoretically grounded framework that reformulates contrastive learning as a spectral decomposition problem. The core contribution is the introduction of a contrastive similarity weight matrix, which allows for the replacement of iterative, gradient-based optimization with a closed-form spectral update. The framework unifies linear and nonlinear (via RKHS kernelization) encoders and extends from one-to-one to many-to-many alignment settings. The empirical results across synthetic, unimodal, and multimodal tasks are compelling, demonstrating performance competitive with or superior to SGD-based baselines.", "questions": "- Add more baselines other than SGD-CLIP.\n- Provide more comparative results between linear and nonlinear; between UniCon and SGD-CLIP.\n- Add more in-depth analysis.\n- Add a detailed example to illustrate why the nonlinear setting is better than linear.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Theoretical Unification and Novelty\n- Comprehensive Empirical Validation\n- The paper is generally well-structured, with a clear road-map of the theoretical contributions.", "weaknesses": "- Breadth of Baseline Comparison: The paper mainly compares to SGD-CLIP, without other recent strong baselines.\n- The paper provides a novel kernel implementation, but provides limited ablation or intuition as to why. \n- The introduction provides limited hints about the shortcomings of previous works and why the current study is necessary. \n- The introduction claims that the linear and nonlinear settings are unified. However, this claim is questionable since the linear and nonlinear methodologies are almost always stated separately. \n- The superiority of the nonlinear setting compared to the linear one needs to be stated clearly, e.g., by providing an intuitive example.\n- The Experiment section lacks the direct comparison of performances between linear and nonlinear methods, either by Figures or quantitative results (Table 1 and 2).\n- The visualization difference between UniCon and SGD-CLIP (Figure 2 (mid vs right), Figure 4 (b) vs (c)). Consider adjusting the visualization to make the difference more evident.\n- The Discussion section simply provides some observations and conclusions, which makes it difficult to differentiate from the Conclusion. Is it possible to add some in-depth analysis that supports these observations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqGTuuG7lP", "reviewer": "Reviewer_5pMu", "summary": "The paper proposes UniCon, a kernel-based framework for efficient contrastive alignment that yields closed-form spectral updates instead of gradient-based training. The method introduces a contrastive similarity weight matrix and shows that minimizing a broad family of contrastive losses is equivalent to maximizing a trace objective whose optimizer is characterized by a spectral decomposition, both for linear encoders and for nonlinear encoders via RKHS kernels.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1) Motivation and scope are clear.\n(2) Unified kernelized theory and shows gradient equivalence from general contrastive losses to a trace objective, extends linear SVD view to RKHS with an explicit optimizer.\n(3) The method achieves outstanding performance with synthetic-linear scaling and 100% matching in 0.02 s (compared to 0.32 s for SGD), reaching similar accuracy in ~2× less time and with only ~2 epochs on the CIFAR-10 dataset.", "weaknesses": "(1) The kernel selection and the corresponding ablation are not explained in detail. The paper recommends an angular kernel and notes it outperforms RBF when embeddings are on the hypersphere but quantitative kernel ablations are not shown beyond a qualitative statement.\n(2) Limited experimental scope. All experiments use frozen or very shallow trainable encoders and comparison baselines are limited to vanilla SGD-CLIP without comparing against other efficient contrastive learning methods.\n(3) Theorem 9 contains a typo in the title (\"characterization\") and I think the author may optimize the notation in the article.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tcjI2d5yjk", "reviewer": "Reviewer_WMSJ", "summary": "To address the slow training problem of the state-of-the-art multimodal models, the paper proposes a Unified Framework for Efficient Contrastive Alignment via Kernels (UniCon). Specifically, it uses the contrastive similarity weight matrix to find the closed-form global solutions, instead of minibatch backpropagation.", "questions": "1. The paper claims faster convergence than the CLIP objective, but no convergence curves are shown. Please add plots comparison (e.g., epoch/time vs. validation accuracy) and report CLIP results with other optimizers (SGD/Adam/AdamW). How were hyperparameters (optimizer, learning rate, etc.) tuned for CLIP and for your method? Is the comparison fair?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. UniCon provides a kernel-based perspective analysis showing the connection between contrastive loss and the spectral update.\n\n2. UniCon shows faster convergence than the CLIP-SGD.", "weaknesses": "1. In Lemma 4, it states $\\beta=\\beta(\\theta_1,\\theta_2)$. What is $\\beta$ here? It is not explained, and $\\beta$ does not appear in the equation. Please clarify (likely $\\gamma$).\n\n2. Typo (line 729): “Equation equation 2” → “Equation 2”.\n\n3. In line 134 you write “scaling factor $\\nu \\ge 1$,” while line 148 says “$\\nu > 0$.” For rigor, please use a consistent and correct domain for $\\nu$.\n\n4. $\\gamma$ depends on $\\phi'$ and $\\psi'$ (Def.~3), but choices like the triplet loss that is mentioned multiple times are non-smooth. \n\n\n5. The statement “UniCon is the first implementation that directly leverages the contrastive similarity weight $S(\\gamma)$ for contrastive alignment” is too strong. For example, [1] leverages similarity weights via optimal transport, and [2] uses kernel-based alignment for multimodal contrastive learning.\n\n\n6. Figure 2 is also not clear to me. The two T-SNE figures for SGD-CLIP and UniCon look identical to me. What benefit beyond training time is demonstrated? Please clarify or add quantitative metrics.\n\n7. Since efficiency is a claimed contribution, a convergence analysis (or at least a theoretical discussion) is needed to justify why the method should be more efficient.\n\n8. Experiments are limited to small projection heads (e.g., two-layer MLPs). Real multimodal setups are larger. It is unclear how the method scales to larger/complex heads, partial/full CLIP fine-tuning, or training from scratch. Some baselines appear underpowered; e.g., [3] shows that with appropriate adapters/architectures, SGD-CLIP can perform well (including zero-shot). Given the abstract’s opening claim about slow training of state-of-the-art models, it would be fair to evaluate efficiency at that scale; current comparisons are not yet convincing.\n\n\n[1]. Shi L, Fan J, Yan J. Ot-clip: Understanding and generalizing clip via optimal transport[C]//Forty-first International Conference on Machine Learning. 2024.\n\n[2]. Gong S, Jiang Y, Dou Q, et al. Kernel-based unsupervised embedding alignment for enhanced visual representation in vision-language models[J]. ICML, 2025.\n\n[3]. Gao P, Geng S, Zhang R, et al. Clip-adapter: Better vision-language models with feature adapters[J]. International Journal of Computer Vision, 2024, 132(2): 581-595.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5eTmxvKY0v", "reviewer": "Reviewer_tNkS", "summary": "This paper proposes UniCon (Unified Framework for Efficient Contrastive Alignment via Kernels), a theoretically grounded and computationally efficient alternative to stochastic contrastive learning. Unlike traditional contrastive methods such as CLIP or SimCLR that perform pairwise optimization between positive and negative samples through minibatch-based SGD, UniCon introduces a global contrastive similarity weight matrix $S(\\gamma)$ that captures the pairwise interactions among all samples simultaneously.\n\n## Technical Approach\n\nUniCon formulates the contrastive objective in matrix form $C(\\gamma) = X S(\\gamma) Y^\\top$ (or its kernelized version $M = K_X^{1/2} S(\\gamma) K_Y^{1/2}$) and converts contrastive alignment into a spectral problem. Through rigorous derivations, the paper proves that minimizing the contrastive loss is equivalent to a rank-$r$ spectral approximation of $C(\\gamma)$ in either linear or reproducing-kernel Hilbert space (RKHS) settings, yielding a closed-form global solution via singular value decomposition (SVD)—replacing thousands of gradient steps with a single spectral update. The authors further extend this formulation to nonlinear and many-to-many alignments through kernelization, demonstrating theoretical unification across linear, nonlinear, unimodal, and multimodal regimes.\n\n## Empirical Results\n\nUniCon achieves substantial speedups (100–400× faster) across synthetic and real datasets while maintaining or surpassing the performance of traditional SGD-based contrastive methods.\n\n## Overall Impact\n\nUniCon reframes contrastive learning as global spectral alignment rather than local pairwise optimization, providing both a unified theoretical interpretation and a dramatically accelerated training paradigm.", "questions": "## 1. Optimality Under End-to-End Training\n\nFor end-to-end trainable encoders (non-frozen backbones), does the optimality of the spectral closed-form solution still hold? If the encoder weights evolve during training, would $S(\\gamma)$ need to be recomputed and re-factorized frequently to maintain optimal alignment?\n\n## 2. Low-Frequency Update Motivation\n\nWhy do the authors choose to update the global $C(\\gamma)$ or $S(\\gamma)$ at a low frequency? Is this design mainly motivated by computational efficiency, numerical stability, or to preserve the consistency of the learned spectral subspace as the encoder outputs evolve?\n\n## 3. Distributional Shift Robustness\n\nWhen the encoder undergoes significant distributional shifts (e.g., during fine-tuning or domain adaptation), could the low-frequency update scheme cause a mismatch between $S(\\gamma)$ and the current feature distribution?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "## Conceptual Novelty and Breakthrough\n\nThis paper presents a genuinely novel and impactful perspective on contrastive learning by reformulating it from a local, pairwise optimization problem into a global spectral alignment framework. The introduction of the contrastive similarity weight matrix $S(\\gamma)$ and its associated closed-form spectral solution represents a conceptual breakthrough—showing that contrastive objectives, long thought to require large-batch SGD, can instead be solved analytically through singular value decomposition.\n\n## Theoretical Rigor and Practical Impact\n\nThe problem addressed is fundamental and timely, given the enormous training cost and inefficiency of existing multimodal models like CLIP. By providing a mathematically rigorous unification of linear and nonlinear (kernelized) encoders, UniCon bridges theoretical understanding and practical acceleration, delivering both interpretability and efficiency within one framework.\n\n## Empirical Validation\n\nEmpirically, the results are striking: UniCon achieves the same or better alignment performance while being up to hundreds of times faster than SGD-based CLIP training, across both synthetic and real-world multimodal datasets. The experiments are carefully designed to validate each theoretical claim—showing consistent convergence of $S(\\gamma)$, clear geometric interpretability, and strong zero-shot transfer.\n\n## Overall Assessment\n\nThis paper stands out for its original idea, theoretical depth, and practical relevance. It redefines how we think about contrastive learning optimization, offering a theoretically elegant and computationally transformative solution.", "weaknesses": "The paper lacks validation on large-scale datasets (such as LAION-400M or Conceptual-12M) and complex modalities including multilingual and video data. This limits our understanding of whether UniCon's spectral alignment approach scales effectively to real-world production settings and remains efficient when dealing with high-dimensional, heterogeneous modal representations.\n\nWhile the theoretical formulation of $C(\\gamma) = X S(\\gamma) Y^\\top$ is elegant, the paper would benefit from a clearer algorithmic description or pseudocode detailing how $C(\\gamma)$ (or its kernelized version) is computed, aggregated, and updated in practice. This would improve reproducibility and help practitioners understand the computational flow beyond the mathematical abstraction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)