Learning Shared Representations from Unpaired Data
Authors: Amitai Yacobi
Nir Ben-Ari
Ronen Talmon
Uri Shaham
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Learning
Unpaired data
Spectral Embedding
universal embedding
Primary area: other
TL;DR: Learning shared representations almost exculsively from unpaired data.

Abstract: Learning shared representations is a primary area of multimodal representation learning. The current approaches to achieve a shared embedding space rely heavily on paired samples from each modality, which are significantly harder to obtain than unpaired ones. In this work, we demonstrate that shared representations can be learned almost exclusively from unpaired data. Our arguments are grounded in the spectral embeddings of the random walk matrices constructed independently from each unimodal representation. Empirical results in computer vision and natural language processing domains support its potential, revealing the effectiveness of unpaired data in capturing meaningful cross-modal relations, demonstrating high capabilities in retrieval tasks, generation, arithmetics, zero-shot, and cross-domain classification. This work, to the best of our knowledge, is the first to demonstrate these capabilities almost exclusively from unpaired samples, giving rise to a cross-modal embedding that could be viewed as universal, i.e., independent of the specific modalities of the data. Our project page: https://shaham-lab.github.io/SUE_page.

Learning Shared Representations from Unpaired Data
Amitai Yacobi
Nir Ben-Ari" Department Computer Science
Department of Computer Science
Bar-Ilan University
Bar-Ilan University
Ramat-Gan, Israel
Ramat-Gan, Israel
amitaalyacobi@@mal.cccmmm
nirnirba@gmail. com
Ronen Talmon
Uri Shaham
Electrical and Computer Engineering Technion Haifa, Israel
Department of Computer Science
Bar-Ilan University
Ramat-Gan, Israel
tneeniee.technion ac.il
uri shaham@biu ac.
Abstract
Learning shared representations is primary area of multimodal representation learning. The current approaches to achieve a shared embedding space rely heavily paired samples from each modality, which significantly harder to obtain than unpaired ones. In this work, we demonstrate that shared representations can be learned almost exclusively from unpaired data. Our arguments grounded in the spectral embeddings of the random walk matrices constructed independently from each unimodal representation. Empirical results in computer vision and natural language processing domains support its potential, revealing the effectiveness unpaired data capturing meaningful cross-modal relations, demonstrating high capabilities in retrieval tasks, generation, arithmetics, zero-shot, and cross-domain classification. This work, the best of our knowledge, is the first to demonstrate these capabilities almost exclusively from unpaired samples, giving rise to cross- modal embedding that could be viewed as universal, i.e., independent of the specific modalities of the data. Our project page: https: //sr//ashhaa.. lab. lab.llll.giiioo. aab..ithub........... io/ SUE_page
1 Introduction
The great success of unimodal models 19
2 in the last decade, and the increasing
demand for multimodal applications, have shifted large research attention into the cross-modal domain. Multimodal models present impressive performance on various cross-modal tasks, including image-text (60] 46 47). speech-text [22] vidco-text [80] and medical-image-text [84). to name few.
primary task in multimodal representation learning the learning of shared representations, i.c.. representation spaces to which instances from different modalities can be mapped, and in which such multimodal instances can compared. Currently, models learning shared representations typically rely on vast amounts of paired data for training. For example, CLIP (60] was trained on 400 million pairs of images and their captions (i.e., corresponding texts). This kind of supervision (i.e., pairing) is often costly, difficult, and might be even impossible to obtain in many domains and applications. In medical domains, for instance, obtaining large number of samples is often challenging, as typically depends on expert annotations. This supervision becomes both costly and time-intensive.
"Equal contribution, random order.
Also at Moodify.ai, Kefar Saba, Israel
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

G Ginages
Ctexts
Corresponding Handom Walks
Pers
Demsity
200
Arbitrary Random Walks
Density
1.5
Non-pan non-pars
1.0
100
0.5
0.75
0.80
0.85
0.90
0.0
0.5
1.0
1.5
2.0
Frobenius
(8)
(b)
Figure I: Empirical demonstration of universality. (a) Distances between corresponding random walks on image and text graphs from MSCOCO, compared to distances to randomly shuffled (non- matching) walks. Although constructed independently from unimodal features, corresponding walks exhibit significantly greater similarity. (b) Distances between paired and unpaired points in the shared space of aligned 2D spectral embeddings (SEs). Paired points consistently closer, indicating that the independently learned SEs capture analogous structure across modalities (see App. A.7
or requires rare and specialized tests 32 Unlike paired data, unpaired data significantly more accessible and available.
In this paper, we demonstrate that is possi- ble to learn shared representation spaces while relying almost exclusively on unpaired data. This statement might sound counterintuitive at first, as the supervisory signal from pairing in- stances from different modalities is crucial for contrastive learning, which the standard tool used for learning shared representations. How- ever, we argue that this signal can replaced by concept we name "universal embeddings". Specifically, we argue that is possible to lever- age unpaired data to learn embedding functions and such that if (x, x')
man surfing"
"A pizza on table"
"Playing in the snow"
) is a paired data in- stance (that is, and x' correspond to specific different modalities), V (x: = ((x'). More- over, argue that practical means learn such maps and is to compute the leading eigenfunctions of the diffusion operators corre- sponding to the unpaired unimodal instances.
Figure Almost exclusively unpaired image retrieval. Retrieved images by SUE for custom captions on MSCOCO, trained with 100 pairs and 10k non-pairs. Despite minimal paired data, the results semantically align closely with text queries.
Specifically, modern pre- preitrained unimodal foun- dation models have proven ability to represent semantics [36] For example, two given images have close embeddings if their semantic meaning is similar, and far otherwise. These similarities can be captured by a random walk process on the samples' representations, suggesting that a random walk process defined on such unimodal representations should largely correspond semantic similarity. Therefore, we can expect random walks defined on different unimodal representations that capture semantics well to be highly similar (see Fig. [ This assumption is supported by substantial empirical evidence from recent works 336] 25
Random walk processes are finite analogs of diffusion operators. Thereby, the similarity of ran- dom walks that are constructed from different, modality-dependent representations implics that the eigenfunctions of the corresponding diffusion operators will have universality properties (i.e., modality-invand invariance) [3]. Therefore, constructing spectral embedding (SE) based on the leading eigenvectors of random walks, which are viewed as discrete approximations of the leading eigenfunc- tions of diffusion operators [ (6],, enables us to take advantage of this concept even in the absence of paired samples. That is, these eigenfunctions can be learned independently from each modality using only unpaired data.
We demonstrate our arguments via a pipeline we call SUE: Spectral Universal Embedding, designed to uncover universal embedding in reakly-paire scenario, i.e., when large amounts of unpaired data are available alongside only very small number of paired samples. To the best of our knowledge.

---

this work is the first to learn shared representations from almost exclusively unpaired data. SUE consists of three steps (see Fig. 3): (i) SE, to extract universal features, (ii) Canonical Correlation Analysis (CCA) on very few samples, to align those features, and (iii) MMDD-eet to provide additional non-linear alignment. We demonstrate SUE on vision and language domains and show that with only a small number of paired examples, unpaired data can be leveraged to learn rich multimodal representations. For instance, on the MSCOCO dataset, SUE improves retrieval performance by over 250% on average compared to contrastive learning trained with the same number of pairs. Fig. demonstrates SUE's ability to learn cross-modal properties with almost no paired data.
Our contributions can be summarized as follows: (i) we demonstrate that learning of shared rep- resentation spaces can be done based on unpaired data, (ii) we show that spectral embeddings are an effective means to obtain universality, and (iii) we establish a pipeline for shared representation learning, utilizing primarily unpaired data, along with a very small number of paired instances.
2 Related Work
Paired-trained models. Multimodal models have recently seen great improvements in rich-data modalities, such as text with image [60, 46, 447,, speech [22 and video [80]. Current state-of- the-art multimodal models rely on contrastive learning between corresponding pairs of unimodal representations. While these methods yield high-quality results, they require extremely large amounts of paired data, limiting their applicability to arbitrary domains. Alternative methods, such as CSA [48], slightly reduce the number of pairs required but still operate exclusively on paired data. Unlike these, the universal embedding offers an integration of any two domains, uncovering a shared space primarily from unpaired samples, complemented by only small number of pairs. Moreover, following recent research on the modality gap in multimodal learning [49 23 65). contrastive- methods fail to be universal due to modalities residing on different manifolds. contrast, universal embedding should not exhibit this gap (see App. B for details).
Incorporating Unpaired Data. Unpaired samples have been used to replace certain applications
of multimodal models without constructing shared representations, including medical image segmen-
tation [82 775,, image-to-speedh translation [52], managetto-image translation [87 88
12). bilingual
word embeddings [1. and domain confusion [26] 27 72 85). Other methods incorporate unpaired
data to complement additional signals, such as large paired datasets [56] 71
18
28
78
83 or
classification labels 774
79 45 In contrast, SUE uncovers a latent universal space between two
unimodal domains that share semantic content, using unpaired data and only a small number of pairs.
Several works explore shared representation learning from limited Wolf 34 proposed an unpaired variant of CCA, but as noted in their work, the method is unstable and requires many runs to yield satisfactory results, limiting its practicality. MACK (35), while targeting image-text alignment, depends on a segmentation-to--er limiting its scope. Closer our direction are [10 [44, which align modalities by referencing a set of paired examples. However, they overlook local structure in the unimodal embedding space- an important cue, since non-immediate neighbors are often uninformative in high-dimensional settings. In contrast, SUE leverages local unimodal neighborhoods and few weak pairs to construct a unified cross-modal representation.
supervision. Hoshen and
model trained with paired data,
Spectral Embedding (SE) in multimodal learning. SE is known for its global structure preserva- tion properties of unimodal manifolds [15) 55) 73). One perspective, for instance, is that the SE is the space in which the Euclidean distances represent the diffusion distances on the original manifold [44 The diffusion distance is manifold-dependen metric, offering advantages over traditional metrics (e.g., Euclidean, cosine), especially in high- bigg--iimnsional spaces [[5]. Recent works investigated the global structure preservation abilities of SE in the cross-modal scenario 24 44 81). Specifically, they showed that joint SE extracts meaningful representations from each modality. The universal embedding is inspired by these findings, and the analysis we provide extends this idea. Particularly, we show that independent SEs capture similar semantic properties across different modalities, such as images and texts.
3

---

Preliminaries
Spectral Embeddings.
SE 4 14
is a popular method, used across various domains 55
86
21 42
43
68
8 89
In particular, let X' S Rd denote a collection of data points sampled from R=** be a positive symmetric graph affinity matrix, with nodes corresponding
manifold M. Let to X, and let D be the corresponding diagonal degree matrix (i.e., Dü: ンド=1 Wiz). The random
walk matrix defined as P D- and the random walk graph Laplacian as
P. where
is the identity matrix. The eigenvalues of P can be sorted to satisfy 1 = X1 2 X 2 ... 2 An with corresponding eigenvectors V1, 22, Un 76).
For a given target dimension k, the leading non non-trivial eigenvectors provide a non-linear low- dimensional embedding of the random walk process, known Spectral Embedding (SE). Specifically, the SE representation of a sample x; E X is given by yi (i).
Laplace- Belttrcm and Diffusson Operators. Given a Riemannian manifold M the Laplace- Beltrami operator A acts on functions f: M:M-> + R by Af = div(V.), where div is the divergence and the gradient. The diffusion operator (heat kernel) time t is then defined by P. -tA For more details please see [14]. Importantly to this work, the Laplacian and random walk matrices converge to the Suplace-Beltrane and diffusion operators. respectively [4 69
Parametric SE. Recent works proposed parametric SE implementations, such as SpectralNet 667 (]. These are deep-learning methods, designed to tackle the scalability and generalizability drawbacks of SE. Specifically, they learn parametric map f R + Rt, which minimizes the Rayleigh quotient
1 Lxpectralnet (f) n²
Trace (S(X)*L(())
while enforcing orthogonality (i.e., f IR), where is the random walk Laplacian corresponding to the data matrix E Raxd
Maximum Mean Discrepancy (MMD). MMD 29 30 is distance measure between two proba- bility distributions p, q. is defined with respect to a function class F by
MMD(F, p.9) sup (Ex=p
f(
JEF
When F is a reproducing kernel Hilbert space with kernel A (e.g., RBF kernel), the squared MMD can be written as
MMD2 9) a' x²) 2T y) Eyya kyy, kl,, yy)
where and x' are independent, and and y'. We use empirical variant of the squared MMD (see Sec.
4 Spectral Universal Embedding
4.1 Rationale
Here we detail the mathematical motivation for each component of SUE and describe an overview of a pipeline to reveal universal embedding out of given modalities (see Fig. for an overview).
Mathematical Motivation. We formalize assumption follows. Let M be a latent, underlying semantic manifold, and let f.g be two transformations, such that f(M) and g(M) represent the two modalities from which we observe samples. There is body of work specifying conditions under which the spectral properties of M are preserved under f. For example, if f. have bounded distortion and bounded Ricci curvature, the corresponding eigenfunctions of the Laplace- Beltrami operator, as well as of the diffusion operator, on (M) and g(M) are similar in the L sense 7 (see App. D)
Laplace-Beltrami
Intuitively, our assumption states that the diffusion operators defined on each modality are relatively similar. This assumption is empirically supported by the results in Sec. 5 well as in recent

---

our epproach
Modality
unobserved distribution
Spectral Embodding
CCA on samples
MMD
Pre-trained embeddings
Figure 3: SUE's overview. The modalities (represented by their unimodal embeddings) represent an unobserved universal (semantic) distribution; the SE is capable of retrieving this universal structure, up to rotations; CCA on a minimal number of pairs enable linear alignment between the modalities, but not sufficient for joint universal embedding; the MMD then fixes the misalignment between the modalities, integrating them into the universal embedding space.
works [36 25
Then, universality is enabled through the eigenfunction preservation properties of the similar diffusion operators. Namely, the cigenfunctions of these operators will be universal, in the sense of moddalit-invarianne operators' eigenfunctions obtained via SpectralNet [67] while trained to compute the eigenvectors of the random walk matrix of its training data, being generalizable parametric map makes it practical means to compute the eigenfunctions of the diffusion operator, viewing the eigenvectors as discretization of the eigenfunctions [5] [69] Crucially, we train SpectralNet unimodal data only; hence, no paired data is needed to learn the Laplacian eigenfunctions, i.e., our universal embedding functions.
(sce Sec. Dp. In practice, the ability to learn the diffusion
Overview. SUE consists of three steps: SE, CCA and MMD. First, it maps each pre-trained unimodal embedding space into its corresponding eigenspace, retrieve the global structure each modality 44 55 701. Using SpectralNet [67]. this is done parametrically, allowing generalization test data. Noteworthy, SE is not unique, as eigenvalues with multiplicity p can yield any basis spanning the p-dimensional eigenspace and even single eigenvectors may differ by sign.
To resolve the SE ambiguity, and provide additional linear alignment, number of paired samples. However, as the CCA purposefully considers a limited number of samples, and the SEs differ by than orthogonal transformation, alignment using a Maximum Mean Discrepancy (MMD) residual network [66]. This kind of network architecture originally designed for batch-effect removal, by minimizing the empirical MMD) value of two distributions. Namely, we view the two low-dimensional representations as similar distributions and learn (close identity) non-linear shift align the distributions. The MMD serves as the last step to fine-tune the alignment. Notably, MMD loss does not require paired data, which enables the utilization the full unpaired dataset.
CCA minimal
strengthen the cross-modal
4.2 Computing SUE
In this section, we describe the computation of SUE, roughly described in Sec 4.1 in more detail. A summary of the steps of the SUE algorithm is outlined in Algorithm 1.
Notations.
Throughout this section, we will use the following notations. Let X C R4, C d be sets of unpaired pre-trained unimodal embeddings of sizes n1, n22, resp. Accordingly, denote X, {æ1, Fm C K, (yI,... Dm S y to be sets of paired embeddings. Importantly, m & ni, n2. Let le 2 r be two pre-chosen dimensions for the SE and final universal representations.
Approach. Given X' Y. we train two independent SpectralNet models Sx X - R. Sy y T approximate the k-dimensional SE of each modality. Due to the non-uniqueness of the SE, Sx and Sy might differ by sign and basis of each eigenspace.
5

---

To address this ambiguity we utilize Xp and y/p Specifically, we employ CCA on (Sx(X)), ( Sy()))) to obtain the projections Qx Qy E Rx These projections are used to align Sx (X) and Sy()). The linearly aligned SEs approximations can be written as Sx Qx Sx, Sy == Qy Sy.
Then, we learn a residual neural network Fg : IR" + R' bring the distribution of the linearly aligned SEs as close as possible. Specifically, we minimize the squared MMD between the two empirical distributions
LMMD
1 mi
E xz) JEZEEX
E (1) EY
Σ MENEEY
k(Ut, Viy)
(1)
m2m2
m
where m.: are the corresponding batch sizes, K is a universal kernel (e.g., RBF kernel), and J; Sx(x;), y = Sy(v). The final functions can be written as fx == Sx, Jy == Fo o Sy.
Given a new test point ye. sampled from the same distribution as V, we simply propagate it through fy. and similarly test point sampled from the X' distribution.
Algorithm 1: Spectral Universal Embedding (SUE)
Input: Unpaired sets of pre-trained unimodal embeddings X' E 12' xd1 and y E R and paired sets Xp and y%P of size 2 0
Output: Maps fx dd + IR", fy dR + H" approximating the universal embedding from each modality
Train Sy
2 Perform CCA on (Sx( (y)) to obtain projections Qx, Qy E Rx
3 Train residual neural network Fo : I²" -> IR" to minimize the MMD loss LMMD (Eq.
4 Return the maps:
fx = Qx Sx, fy := F9 o Qy Sy
At inference time, propagate the sample y through the appropriatc map fx( or fy(v)
Experiments
In this section, we provide empirical evidence for the effectiveness of unpaired data for constructing shared representation by evaluating SUE's ability to uncover it. We assess SUE's effectiveness in cross-modal settings using both quantitative and qualitative analyses. Our experiments cover a diverse set of tasks, including retrieval, generation, semantic arithmetic, zero-shot learning, and cross-domain classification. Notably, this is the first work, the best of our knowledge, to demonstrate such versatility across tasks and modalities while relying almost exclusively on unpaired data.
The remainder of this section is organized as follows. Sec. 5.] evaluates retrieval performance. Sec. 5.2 presents a range of applications universality. To better understand the contribution of each component in SUE, the impact of incorporating additional paired and unpaired samples, revealing that unpaired data holds greater potential than previously recognized. Sec. [5.4] presents ablation study assessing the roles of SUE's components (SE, CCA, and MMD), demonstrating that the novel integration of these techniques is central SUE's effectiveness. Due space constraints, refer to App. for additional results. Further details on the experimental setup, implementation, and training procedures provided in Appendix
investigate the effects of various components. In Sec. 5.3 examine
Datasets. To evaluate SUE's performance, we use several paired datasets. To provide informative qualitative results and facilitate an intuitive understanding of the universal embedding concept, we use three vision- languala datasets (mm): Flickr30k [59], MSCOCO [50], and Polyvore [31]: as well as vision-vision dataset (mm): Edges2Shoes [37]: and a tabular-to-tabula dataset m): Handwritten 20 For the generation task, we use another vision- dataset (mm): Caption-FFHQ [40]. See App. E. for further details on the datasets and preprocessing (e.g., the unimodal models used and
adaptation to weakly-paine pairedd settings).
Baselines. To the best of our knowledge, this one of the very first works to exploit weakly-paired data (i.e., very few pairs). As such, no direct basclines exist for this setting. To contextualize results,

---

Table 1: Retrieval results. Results with few paired samples vision-language
vision- vision (mm), and tabular-tabular (mm) datasets from each modality to another: image-to-te to-text (I2T), text-to-image (T2I), edges-to-shoes (E2S), shoes-to-edges (S2E), KL coefficients-to-pixel averages (K2P), pixel-to-KL. (P2K); by SUE, Contrastive, and CSA. The Imp. column states the relative mean improvement of SUE over Contrastive learning. Using the same small number of pairs, SUE significantly outperforms the popular paired method. SUE substantially relies on unpaired data.
SUE (ours)
Contrustive
CSA
#paired
Imp.
R@1 1
R@5
R@ 10
R@1
R@5
R@ 10
R@1
R@5
R@10
MSCOCO
100
12T
5.75
21.50
34.25
1.50
8,50
13.00
0.00
1.25
3.00
+257.20%
T21
5,25
18.25
33.25
0.80
5.80
12.20
0.00
1.00
2.25
Flickr30k
12T
4.25
19.75
32.00
3.00
9.50
16.20
0.25
1.25
2.50
500
+103.32%
T21
5.75
22.00
32.75
2:50
9.80
15.00
0.25
0.75
2.75
volyvore
12T
6.00
22.75
32.25
3.20
13.8
22.5
0.25
1.25
2.25
500
+55.67%
T21
4.75
20.75
32.00
4.00
11.50
23.00
0.25
1.00
3.25
2Sges2Shoes
50
E2S
4,00
16.00
25.25
1.00
5.50
14.00
0.25
1.50
2.75
+200.51 %
S2E
3.50
17.00
27.00
0.80
6.00
12.80
0.25
1.50
3.00
Handwritten
K2P
25.50
62,00
79.00
4.80
17.00
28.00
4.25
12.25
17,50
100
+283.60 %
E8
P2K
25.00
61.75
78.00
4.80
17.80
30.50
3.50
9.00
15.00
we compare SUE with the contrastive method designed for fully paired data [60 80 22] (training an MLP on pre-trained unimodal features using the same number of pairs), and the CSA method
48 designed for small paired datasets. In addition, we examine domain confusion (App. and
(App. A.2)) representative approach for incorporating unpaired data alongside paired data; both perform notably worse in these scenarios.
SAIL [83
5.1 Retrieval
Retrieval. Tab. I reports retrieval scores for SUE and contrastive learning several datasets using
a randomly sampled test set of 400 examples. Following prior work [60] [22], we use the Recall @ metric. The results demonstrate SUE's ability to capture cross moosll semanties with few 100 pairs for MSCOCO, 500 for Flickr30k and Polyvore, and just 50 for Edges2Shoes. Remarkably, these results are also comparable with the ones reported by Huang et al. [35] on the Flickr30k T2I t despite MACK using a paired segmentation-to-tet mean average precision (mAP) results, reinforcing the same conclusions. In App. A.6 we further analyze how different unimodal encoders affect SUE's retrieval performance.
model (see App. A). In App. A.5 we report
These retrieval results introduce significant advance in the ability to learn meaningful representations in the practical scenario of limited pairs. Specifically, when comparing SUE, trained on the MSCOCO dataset with only 100 pairs (and additional unpaired data), to a network trained with CLIP loss solely on the same 100 pairs, SUE achieves better performance by more than 250%. This highlights that SUE derives its strength from the unpaired samples.
Fig. 5 analyzes the power of the contrastive method with small number of pairs, compared to SUE. Remarkably, to get the results SUE achieves with 100 pairs (and 10k non-pairs), the contrastive method requires an order of magnitude more pairs. This shows the utility of SUE in limited pairs scenarios.
In Fig. 4h we supplement the quantitative results with several qualitative examples. Additional qualitative results can be found in App. A.9 These results illustrate how SUE successfully captures semantic relationships across modalities, even in cases where the exact pair not among the first neighbors (leading to zero Recall @ score).
In Fig. 2 we consider a custom caption scenario, where we retrieve images of more general query sentence that cannot be found in the dataset. These examples further support SUE's ability to capture these cross-modal semantics. These qualitative examples also depict the challenge of reliably measuring semantic coherence across modalities in retrieval tasks using the recall@ k metric. Therefore, App. C
we consider new soft variant of Recall@ k, which follows previous works 33 and uses CLIP score as a measure of semantic alignment between image-text pairs.
PR @ 1 10.3; R@S 25.1: R@ 34.0.
7

---

glr with blandd hair
boy happy"
"a
with sunglasmee"
"an
mountaii skis and ctrok out vioe"
digg sketma
(b)
"a woman"
glasses"
"with sunglasses"
"is smiling"
"an old man"
"Is smilinn"
(4)
(c)
Figure 4: (a) Images retrieval examples. SUE captures cross-modal semantic structure. Top four retrieved images for text and shoe-edge queries from Flickr30k and edges2shoes. True pairs are excluded, yet results remain semantically aligned. (b) (Almost-) Text-free text-to-image generation examples. Images generated from various text queries using a generator and converter trained exclusively on images. (c) (Almost-) Text-free arithmetics examples. Images generated from sums of text and image queries for example, "with sunglasses" man/woman image yields corresponding result with sunglasses.
These examples show that despite minimal paired training data, SUE effectively reveals meaningful cross-modal semantic connections, suggesting its utility in real-world scenarios where large paired datasets are unavailable or impractical to obtain.
5.2 Applications of Universality
(Almost-) Text-free text-to-image generation. Fig. 4 depicts several examples of text-to-image generation with minimal text-image correspondence. In particular, we use a pre-trained GAN inversion model (77) trained on the FFHQ dataset [40) and train converter model that maps from the universal image embeddings to the GAN's latent space. During inference, we pass universal text embeddings to the converter and generate face images from its output using the GAN decoder. Noteworthy, both the GAN and the converter were trained exclusively on images, with few text-image pairs available only during SUE's training. For more details on the training process of the converter model, please refer to App.
(Almost-) Text-free arithmeties. Fig. presents the results of emergent arithmetic operations on the universal embedding. Notably, these examples require no additional training we simply perform vector summation of text and image embeddings and generate the image using the pre-treeee pre-trained converter and GAN. This approach aligns with techniques demonstrated using CLIP [6]] but with limited text-image pairing. This summation in the vector space translates into an intuitive integration of semantics, making it, in sense, a semantic vector space, while almost no text-image pairs were available during the construction of the shared embedding. This showcases the SUE's cross-modality capturing abilities, even in the absence of large paired datasets.
Zero-shot. In App A.3 we further demonstrate SUE's ability to capture high-level tross,modal, using a zero-shot scenario acting as a classifier without being explicitly trained for classification.
Emergent cross-modal classification. In App A.4 demonstrate the effectiveness of universality for the classification of an unlabeled domain with minimal correspondence to a labeled domain.
5.3 Effect of Paired and Unpaired Samples
Unpaired samples. Fig. 5p shows the impact of additional unpaired samples. This experiment is of significant interest, as unpaired samples are usually considered unusable the multimodal setting for point-to-point matching. The results indicate that additional unpaired data significantly enhances

---

Recallig1 (86)
Reccallee 10 196
30-
Reralg @10
28
Concrssvve
10
100
200
300
400
500
600
700
000
1000
5000
10000
15000
20000
25000
10
50
120
500
2000
5000
# Paired Samples
Unpaired Samples
Paired Samples
(a)
(b)
(c)
Figure 5: (a) Contrastive requires order of magnitude more pairs to achieve similar results as SUE in the weakly-paired regime. Recall 10 results on MSCOCO by SUE (with 100 pairs) and Contrastive with various numbers of pairs. SUE exploits unpaired data outperform contrastive learning when limited pairs are available, An order of magnitude more pairs are required to achieve similar results with contrastive learning: (b-c) Effect #unpaired and #paired samples on Recall @ 10 results on image retrieval on the Flickr30k dataset. (b) SUE improves as the amount of unpaired data increased. (c) SUE relies on non-pairs instead of pairs. SUE relies minimally on paired data, while substantially on unpaired data, enabling it to enhance its performance with additional unpaired samples, which are much easier obtain.
retrieval results. This opens the door for new regime of multimodal learning using unpaired data with only a minimal number of available pairs.
Paired samples. Fig. 5 depicts the results of an analogous experiment examining the effect of the number of paired samples required for the CCA step. with the unpaired samples held constant. As expected, a minimal number of paired samples are required for good results (~500 in this case of Flickr30k). However, SUE does not rely additional pairs, increasing their number above the minimum required is redundant. This outcome highlights the potential for learning significant cross-modal embeddings while focusing on unpaired data, which is much easier obtain.
5.4 Ablation Study
Tab. presents the results of various ablations of SUE, from the original unimodal representations to the complete SUE pipeline. Alongside these empirical results, we provide UMAP visualizations of SUE's three steps in Fig. [3] in App. AA.IT The main outcome from these results that has pivotal role in revealing the universal embedding. Attempts to align the original unimodal representations yield mostly trivial retrieval results (see left column of each dataset in Tab. (2). However, applying the same aligmment approach (i.c., CCA alongside MMD) significantly improves the retrieval performance. This underscores SE's essential role in extracting the modality-invarian representations. This is further supported by the results in Tab. TO in App. A. where replacing the SE step with an Auto-Encoder significantly decreases the performance. These results support the existence of the universal embedding. SUE successfully lears meaningful cross-modal semantic representation with only minimal number of paired samples, used solely to facilitate linear cross-modal alignment. See extended ablation study in App. A.10
the SE space,
Table 2: Ablation study results. SE is pivotal. Text-to-iman image R @ 10 results on the Flickr30k and MSCOCO datasets. The highlighted numbers are SUE's full pipeline. Notably, SE is necessary for the success of CCA (and CCA with MMD).
Flickr30k +SE
MSCOCO
+SE
2.25
8.75
1.50
4.25
+MMD
3.75
5.50
2.00
3.75
+CCA
4.50
30.25
7.75
31.50
+CCA +MMD
4.75
32.75
9.75
33.25

---

Conclusion and Future Work
This paper demonstrates the ability to learn shared representations almost exclusively from unpaired data. Specifically, we showed that a shared structure can be captured through a random walk process on each modality--peeiific representation, with Spectral Embedding (SE) the practical foundation for uncovering it, revealing its universal properties.
In doing so, challenge the prevailing approach in multimodal learning, which relies heavily on extremely large amounts of paired samples. SUE, operating with weakly-paine data, in the sense of very few pairs, yields meaningful results across various applications, especially highlighted by the comparison to contrastive learning.
Limitations. Despite these contributions, several limitations remain. While SUE provides com- pelling evidence for the hidden potential of unpaired data in learning a shared representation space, does not yet match the performance of state-of-the afr-the-rrt art models trained with massive paired datasets. Furthermore, our evaluation has been limited to a small set of modalities and tasks, leaving open questions about its generalizability more complex domains such video, speech, scientific data. Addressing these challenges offers a promising avenue for advancing the framework.
Importantly, our work does not yet aim to replace cxisting multimodal models trained with large sets of paired data. Rather, introduces a new concept for settings where such supervision is unavailable. Interestingly, the observation in App. A.6 along with empirical results from Huh et al. [36]. Fan et al. 255, suggests that increasing the capacity unimodal models improves the alignment between unimodal representations, leading to better SUE performance without requiring additional paired data. Finally, this work establishes a foundation for fully unpaired multimodal learning framework across arbitrary modalities, which consider intriguing and promising avenue for future work.
References
[1] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451-462, 2017.
[2] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianceinnvriine--ovarr ization for self-supervised learning. arXiv preprint arXiv:2105.049900 2021.
regular-
[3] Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Directional graph networks. In International Conference on Machine Learning, pages 748-758. PMLR, 2021.
[4] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15()):1333-336, 2003.
[5] Mikhail Belkin and Partha Niyogi. Convergence of laplacian eigenmaps. Advances in neural information processing systems, 19, 2006.
[6] Nir Ben-Ari, Amitai Yacobi, and Uri Shaham. Generalizable spectral embedding with an application to UMAP. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL htt.itttpp://opeeeevieeee http:/eeee/ooooriiddduuuueuere
[7] Pierre Bérard, Gérard Besson, and Sylvain Gallot. Embedding riemannian manifolds by their heat kernel. Geometric & Functional Analysis GAFA, 4:373-398, 1994.
[8] Kieran Campbell, Chris P Ponting, and Caleb Webber. Laplacian eigenmaps and principal curves for high resolution pseudotemporal ordering of single-cell rna-seq profiles. bioRxiv, page 027219, 2015.
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[10] Chi Chen, Peng Li, Maosong Sun, and Yang Liu. Weakly supervised vision-and- -nd---agaaggee pre-training with relative representations, arXiv preprint arXiv:2305.15440 2023.
10

---

[11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[12] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8789-8797, 2018.
[13] Ronald R Coifman. Machine common sense: The darpa perspective. YouTube, 2020. URL bttps://wwit http:///wwwt.bbbbee.mmmemme com http///atcch/=vvA333mmmpma Accessed: 2024-11-11.
[14] Ronald R Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21(1):5-30, -00, 2006.
[15] Ronald R Coifman, Stephane Lafon, Ann B Lee, Mauro Maggioni, Boaz Nadler, Frederick Warner, and Steven W Zucker. Geometric diffusions tool for harmonic analysis and structure definition of data: Diffusion maps. Proceedings of the national academy of sciences, 102(21): 1436,7431, 2005.
[16] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
[18] Benjamin Devillers, Léopold Maytié, and Rufin VanRullen. Semi supervisd multimodal representation learning through a global workspace. IEEE Transactions on Neural Networks and Learning Systems, 2024.
[19] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini arXiv:1810.04800 2018
[20] Robert Duin. Multiple Features.
UCI Machine Learning Repository, 1998. DOI:
[21] Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699, 2020.
[22] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. JEEE, 2023.
[23] Sedigheh Eslami and Gerard de Melo. Mitigate the gap: Improving cross-modal alignment in CLIPP In The l'hirteenth International Conference on Learning Representations, 2025. URL
httpps://ppenreviee..o www.:nne/oooummdaaaammmmem
[24] Davide Eynard, Artiom Kovnatsky, Michael M Bronstein, Klaus Glashoff, and Alexander M Bronstein. Multimodal manifold analysis by simultancous diagonalization of laplacians. IEEE transactions on pattern analysis and machine intelligence, 37(12):505-2517, 2015.
[25] David Fan, Shengbang Tong, Jiachen Zhu Koustuv Sinha, Zhuang Liu, Xinlei Chen, Michael Rabbat, Nicolas Ballas, Yann LeCun, Amir Bar, et al. Scaling language-free visual representa- tion learning. arXiv preprint arXiv:2504.01017 2025.
[26] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1 1180-1189. PMLR, 2015.
[27] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. aomain-adversariil training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.

---

[28] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180-15190, 2023.
[29] A Gretton, K Borgwardt, M Rasch, B Schölkopf, and A Smola. A kernel method for the two-sample-problee advances in neural information processing systems. MIT Press, 2006.
[30] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[31] Xintong Han, Zuxuan Wu, Yu-Gang Jiang, and Larry Davis. Learning fashion compatibility with bidirectional Istms. In ACM Multimedia, 2017.
[32] Alvaro S Hervella, José Rouco, Jorge Novo, and Marcos Ortega. Deep multimodal reconstruc- tion of retinal images using paired or unpaired data. In 2019 International Joint Conference Neural Networks (JCNN), pages 1-8. IEEE, 2019.
[33] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021.
[34] Yedid Hoshen and Lior Wolf. Unsupervised correlation analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3319-3328, 2018.
[35] Yan Huang, Yuming Wang, Yunan Zeng, and Liang Wang. Mack: multimodal aligned conceptual knowledge for unpaired image-text matching. Advances in Neural Information Processing Systems, 35:7892-77999959 7904, 2022.
[36] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Forty-first International Conference on Machine Learning, 2024.
[37] Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
[38] Qian Jiang, Changyou Chen, Han Zhao, Liqun Chen, Qing Ping, Son Dinh Tran, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Understanding and constructing latent modality structures in multi-modal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7661-7677 2023.
[39] Ting Jiang, Minghui Song, Zihan Zhang. Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580, 2024.
[40] Vahid Kazemi and Josephine Sullivan. One millisecond face alignment with an ensemble of regression trees. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1867-1874, 2014.
[41] Ryan King, Tianbao Yang, and Bobak J Mortazavi. Multimodal pretraining of medical time series and notes. In Machine Learning for Health (ML4H), pages 244-255. PMLR, 2023.
[42] Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létournau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Pro- cessing Systems, 34:21618-21622 2021.
[43] Sibsankar Kundu, Dan C Sorensen, and George N Phillips Jr. Automatic domain decomposition of proteins by gaussian network model. Proteins: Structure, Function, and Bioinformatics, 57 (4):725-733, 2004.
[44] Roy Lederman and Ronen Talmon. Learning the geometry common latent variables using alternating diffussinni Applied and Computational Harmonic Analysis, 44(3):509-536, 2018.
12

---

[45] Felix Leeb, Satoshi Hayakawa, Yuhta Takida, and Yuki Mitsufuji. Partial alignment of represen- tations via interventional consistency. In Second Workshop on Representational Alignment at ICLR 2025.
[46] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-
image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888-12900. PMIR, 2022.
[47] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-imag pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[48] Po-han Li, Sandeep P Chinchali, and Ufuk Topeu. Csa: Data-efficient mapping of unimodal features multimodal features. arXiv preprint arXiv:2410.07610, 2024.
[49] Victor Weixin Liang. Yuhui Zhang. Yongchan Kwon, Serena Yeung. and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35: 15:177 16112117625 2022.
[50] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014.
[51] Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 364, 2019.
[52] Shuang Ma, Daniel McDuff, and Yale Song. Unpaired image-to-speech synthesis with multi- modal information bottleneck. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7598-7607, 2019.
[53] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.0344 2018
[54] Sangwoo Mo, Minkyu Kim, Kyungmin Lee, and Jinwoo Shin. S-clip: Semi- Semi-supervised vision- language learning using few specialist captions. Advances in Neural Information Processing Systems, 36:61 187-61222 2023.
[55] Boaz Nadler, Stéphane Lafon, Ronald R Coifiman, and Ioannis G Kevrekidis. Diffusion maps, spectral clustering and reaction coordinates of dynamical systems. Applied and Computational Harmonic Analysis, 21(1):1131277 2006.
[56] Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji, James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference on Artificial Intelligence and Statistics, pages 4348-4380. PMLR, 2023.
[57] Jianmo Ni. Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Y Zhao, Yi Luan, Keith B Hall, Ming- Wei Chang, et al. Large dual encoders are generalizable retrievers. arXiv preprint arXiv:21/22 02.999 2021.
[58] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 07/0009999 2023.
[59] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting cartoonto--hraase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
13

---

[61] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 06X15 1(2):3 2022.
[62] N Reimers. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
[63] Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. htLpsttp:://rret org/ abs/1908 10084
[64] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Vision-ECCV 2010 11th European Conference Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pages 213-226. Springer, 2010.
[65] Simon Schrodi, David T. Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: On the modality gap, object bias, and information imbalance in contrastive vision- language models. In The Thirleenth International Conference Learning Representations, 2025. URL htt.shhtps//// http::///pperevvieee net/ www.mnidduuAPHHRaa....
[66] Uri Shaham, Kelly Stanton, Jun Zhao, Huamin Li, Khadir Raddassi, Ruth Montgomery, and Yuval Kluger. Removal of batch effects using distribution distribution-matcing residual networks. Bioinformatics, 33(16):2539-25 2546, 2017.
[67] Uri Shaham, Kelly Stanton, Henry Li, Boaz Nadler, Ronen Basri, and Yuval Kluger. Spectralnet: Spectral clustering using deep neural networks. arXiv preprint (KKi:18001.1588 2018.
[68] SJ Shepherd. CB Beggs, and Sue Jones. Amino acid partitioning using a fiedler vector model. European Biophysics Journal, 37:105-109, 2007.
[69] Zuoqiang Shi. Convergence of laplacian spectra from random samples. arXiv preprint arXiv:1507.00151. 2015.
[70] Amit Singer and Ronald R Coifiman Non-linear independent component analysis with diffusion maps. Applied and Computational Harmonic Analysis, 25(2):226-239, 2008.
[71] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEF/CVF conference on computer vision and paltern recognition, pages 55638-156518 2022.
[72] Stefan G Stark, Joanna Ficek, Francesco Locatello, Ximena Bonilla, Stéphane Chevrier, Franziska Singer, Gunnar Rätsch, and Kjong- Van Lehmann. Scim: universal single-cell matching with unpaired feature sets. Bioinformatics, 36(Supplement 2):1919-i921 2020.
[73] Ronen Talmon, Stéphane Mallat, Hitten Zaveri, and Ronald R Coifman. Manifold learning for latent variable inference in dynamical systems. IEEE Transactions on Signal Processing, 63 (15):3843-3856, 2015.
[74] Subash Timilsina, Sagar Shrestha. and Xiao Fu. Identifiable shared component analysis of unpaired multimodal mixtures. arXiv preprint arXiv:2409. 194222 2024.
[75] Vanya V Valindria, Nick Pawlowski, Martin Rajehl, Ioannis Lavdas, Eric O Aboagye, Andrea G Rockall, Daniel Rueckert, and Ben Glocker. Multi-modal learning from unpaired images: Application to multi-organ segmentation in and mri. In 2018 IEEE winter conference on applications of computer vision (WACV), pages 547-556. IEEE, 2018.
[76] Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing, 17:395-416. 2007.
[77] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity gan inversion for image attribute editing. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
14

---

[78] Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Hengshuang Zhao, and Zhou Zhao. Omnibind: Large-scale omni multimodal representation binding spaces. arXiv preprint arXiv:2407. 1895, 2024.
[79] Johnny Xi, Jana Osea, Zuheng Xu, and Jason S Hartford. Propensity score alignment of unpaired multimodal data. Advances in Neural Information Processing Systems, 37:141103-144 1111111--111,,,11111111111111111 2024.
[80] Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. arXiv preprint arXiv:2109. 140.04444 2021.
[81] Amitai Yacobi, Ofir Lindenbaum, and Uri Shaham. Generalizable and robust spectral method for multi-view representation learning. Transactions on Machine Learning Research, 2025.
ISSN 2835-8856 URL httpp:://openrrvveer. wttp://wwww.e.eoeoioieeoeoa. (tt.:/orrmmm1110111011111111
[82] Jie Yang. Ye Zhu, Chaoqun Wang. Zhen Li, and Ruimao Zhang. Toward unpaired multi-modal medical image segmentation via learning structured semantic consistency. arXiv preprint arXiv:2206. 10571, 2022.
[83] Le Zhang, Qian Yang, and Aishwarya Agrawal. Assessing and learning alignment of unimodal vision and language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2025.
[84] Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2-25. PMLR, 2022.
[85] Ziqi Zhang, Chengkai Yang, and Xiuwei Zhang. scdart: integrating unmatched scrna-seq and scatac-seq data and learning cross-modality relationship simultaneously. Genome biology, 23 (1):139, 2022.
[86] Ziwei Zhang, Peng Cui, Jian Pei, Xin Wang, and Wenwu Zhu. Eigen-gnn: A graph structure preserving plug-in in for gnns. IEEE Transactions on Knowledge and Data Engineering. 35(3): 2544 2555, 2021.
[87] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[88] Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. Advances in neural information processing systems, 30, 2017.
[89] Qiyao Zhu and Tamar Schlick. A fiedler vector scoring approach for novel ma motif selection. The Journal of Physical Chemistry B, 125(4): 144-1155,,41 2021.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions and accurately reflect its scope and results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
No new theoretical results are presented in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: See App.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
Our code will be publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpipp/.ccppullc//uuuii wwwtc//ggui////eeeeceoocc CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See App.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars are provided for relevant results in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See App.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and confirm that our work fully complies with its guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is foundational and not tied specific applications with foresceable societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve high-risk models or datasets, so no safeguards were necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See App.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not research human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not research human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
21

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This work does not include LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

A Additional Results
Code availability. Our code, including SUE implementation and all experiments, available at https://githut wwt.ahw.......ae..eeeeeeee..eeee wtt.ccooo/ssaaaee Baiiaammaaa//aa lab/SUE
Note about MACK. While the Flickr30k T21 task results are reported in the paper by Huang et al. the absence of public implementation prevents more comprehensive comparison with MACK.
35
A.1
Domain Confusion results
Tab. shows the results of MMD (as representative of the domain confusion methods) and SUE on all retrieval tasks considered in Tab. [ MMD does not necessitate any paired data for training. However, the results show that it significantly underperforms SUE, which uses only very few pairs.
Table MMD Retrieval results. Retrieval results vision-language ( and vision-vision ( datasets from each modality to another: image-to-text (I2T), text-to-image (T2I), edges-to-shoes (E2S), shoes-to-edges (S2E), note-to-measuremmntt (N2M), measureement-to-nntee (M2N). MMD significantly underperforms SUE.
SUE (ours)
MMD
R@1
R@s
R@ 10
R@D 1
R@5
R@ 10
Flickr30k
12T
4.25
19.75
32.00
0.50
1.25
4.00
5.75
22.00
32.75
0.50
1.75
3.75
MSCOCO
5.75
21.50
34.25
0.00
1.50
3.00
T21
5.25
18.25
33.25
0.50
1.75
2.00
Polyvore 12T
6.00
22.75
32.25
0.00
1.25
2.50
T21
4.75
20.75
32.00
0.00
1.25
2.50
Edges2Shoes
E2S
3.75
18.00
26.75
0.50
1.25
4.00
S2E
2.75
15.25
25.50
0.50
1.75
3.75
A.2 Comparison with SAIL
Tab. presents comparison between SUE and SAIL. 33
representative method for incorporating
large unpaired data alongside paired data. The comparison is on vision-language datasets ( m)) under the same settings Tab. The results demonstrate that SUE is more effective in the weakly paired scenario.
Table 4: SAIL comparison results. Results with few paired samples vision-language ( from each modality to another: image-to-text (I2T), eext-to-image (T2I); by SUE and SAIL. Using the same small number of pairs, SUE significantly outperforms SAIL.
SUE (ours)
SAIL R@5
#paired
R@1
R@5
R@ 10
R@
R@10 10
MSCOCO
12T
5.75
21.50
34.25
0.75
4.75
7.25
100
T21
5.25
18.25
33.25
0.75
4.50
7.00
Flickr30k
12T
4.25
19.75
32.00
1.00
3.75
6.50
500
T21
5.75
22.00
32.75
0.50
3.00
6.75
Polyvore
12T
6.00
22.75
32,25
0.50
4.25
500
T21
4.75
20.75
32.00
0.75
4.00
7.50
A.3 Zero-shot
To demonstrate SUE's ability to capture high-level cross-modal semantics, we consider the zero-shot scenario (60]. This is by-product of SUE, with no further adjustment. We use the zero-shot scenario
23

---

to demonstrate the universality properties of SUE. Specifically, we train SUE on Flickr30k and use encode new ImageNet images [77] and their labels. We then compute the cosine similarity between each image and all sentence labels, selecting the highest-scoring label. As shown in Fig. 6 SUE effectively labels images from the unobserved dataset, despite having only minimal image- text correspondence available during training. Tab. | further quantifies these results with small
50-sample test set, compared to CLIP [60), trained on 400 million pairs. (see App. E.3 for more details)..
alfor snow
Day
women
Mounstaii Mlore Bnow
Cer
Day
Mon
Womant
Moontate
shore
Srow
Car
Dog
Man
Womon
Mountai
dhore
Snow
Figure 6: Zero-shot examples. SUE captures high-level cross-modal semanties. Classification probabilities by cosine similarity between an ImageNet's 's images and seven labels, on SUE trained on Flickr30k with 500 pairs. Notably, the images are labeled correctly with only minimal available correspondence during training.
Table 5: Zero-shot results. Accuracy results (%) for the zero-shot task. Classification was done using cosine similarity between the ImageNet's images and seven labels, on SUE trained on Flickr30k with 500 pairs. SUE achieves high performance with only very few pairs available during training.
Accuracy
CLIP
99.26
SUE (ours)
88.31
A.4 Emergent Cross-domain Classification
In this experiment, we demonstrate additional application of universality: emergent cross-domain classification. Specifically, we address the scenario where one domain or modality labeled, while the other is not, and only very few pairs between them are available. Given two modalities or domains, we train a classifier on the universal embeddings of the labeled domain and apply it to the unlabeled domain, enabling effective knowledge transfer.
Tab. presents classification accuracy results (%) on the Office31 dataset 64 which contains 31 classes and three domains. We used the following two domains: Amazon images and DSLR images. In one experiment, we trained a classifier on the embeddings from the Amazon domain and used to classify the DSLR domain, and vice versa. SUE achieves high performance with relatively few paired samples, outperforming the MMD baseline (see App. A.1 for more details on this baseline). For additional information on the dataset, its preparation, and classifier, please refer to App. E.4
Table 6: Emergent cross-domain classification accuracy results (%). "Amazon-E tortroonntt-D-LRR' indi- cates that DSLR images were classified using a classifier trained on embeddings from the Amazon domain, and vice versa. SUE significantly outperforms the MMD baseline.
Amazon-to-DSLR
DSLR-to-Amazon
MMD
36.78
26.33
SUE (ours)
66.02
47.40

---

A.5 mAP Results
In Tab. 7 we provide the mean Average Precision (mAP) results, in the same settings in Tab. I SUE outperforms both contrastive and CSA under this ranking-sensitive metric as well, further validating its effectiveness.
Table 7: mAP results. Results with few paired samples vision-language vision-vision (=m), and tabular-tabular (mm) datasets from each modality to another: image-to-text (12T), text-to-image (T2D), edges-to-shoes (E2S), shoes-to-edges (S2E), KL coefficients-to- averages (K2P), pixel-to- KL (P2K); by SUE, Contrastive, and CSA. Using the same small number of pairs, SUE significantly outperforms the popular paired method.
SUE (ours) mAP
Contrastive mAP
CSA mAP
#paired
MSCOCO
12T
9.72
5.00
1.48
100
T21
9.80
4.90
1.38
Flickr30k
12T
8.91
5.80
1.67
500
T21
8.07
5.80
1.67
Polyvore
12T
9.87
8.8
1.54
500
T21
9.35
9.2
1.68
Edges2Shoes
F2S
7.03
5.7
1.77
50
S2E
6.99
5.1
1.88
Handwritten
K2P
14.38
12.60
9.42
100
P2K
14.03
12.90
7.96
A.6 Effect of Different Unimodal Encoders
Tab. 8 analyzes the effect of different pre-trained unimodal encoders the quality of the shared space. We focus on the MSCOCO dataset, and use text-to-image retrieval results to quantify the quality of the space. In the table, provide the R@ 10 result MSCOCO for all four combinations of four different unimodal encoders. To Dinov2 (ViT-B/14) and SB (MiniLM--L6, which were used in the paper, add smaller version of each one Dinov2 (ViT-s/14 14) and SB (MiniLM-LL3). As expected, higher-capacity unimodal encoders yield better results.
To account for recent advances in text encoders, we additionally evaluate GTR [77 larger and stronger text encoder, in Tab. g Interestingly, its absolute performance is slightly lower. We hypoth- esize that the Mini-LM or alternatively, that GTR less aligned with the image representations. either case, the core effectiveness of method appears robust the choice of text encoder.
already sufficient given the strength of the vision encoder (DINOv2),
Table 8: Encoders comparison. Text-to-image R@ 10 results on the MSCOCO dataset using different pre-traine rre-trained unimodal models. Larger models provide better results.
Encoders
R@ 10
Dinov2 (ViT-s/14) SB (MiniLM-L3)
30.00
Dinov2 (ViT-B/14) + SB (MiniLM-L3)
31.50
Dinov2 (ViT-s/14) (MiniLM-L6)
32.25
Dinov2 (ViT-B/14) SB (MiniLM-L6))
34.25
Dinov2 (ViT-B/14) GTR (t5-large)
31.25
A.7
Fig. details
This section outlines the experiment corresponding to the results presented in Fig.
25

---

A.7.1 Similarity of Random Walks on Corresponding Modalities (Fig, T4)
The objective of this experiment to evaluate whether the structural relationships among image samples are reflected similarly in their associated text descriptions. To this end, we sampled 1,000 sample batches of image-text pairs from the Flickr30K dataset and constructed the respective random walk matrices for both modalities, Pr, Pr. The similarity between random walks was quantified using the normalized Frobenius distance between the corresponding matrices,
Pr Prile
d(P), Pr)
| + Pri|F
The blue histogram shows the distribution of distances computed between aligned image-text pairs, while the red histogram reflects distances between randomly shuffled (i.e., unpaired) image and text sets. Although the shuffled matrices preserve statistical properties, they break semantic alignment. The results demonstrate that the distances between corresponding random walks are significantly smaller than those between arbitrary ones, indicating strong structural similarity. These findings suggest that random walks derived independently from images and texts exhibit a form universality.
The images and their corresponding captions, used for constructing the demonstration in the figure are depicted in Fig.
goalio the puck while holding white hockey
group hockey players slide along the during game
stands in front the gosl hockee game
defending the goal in hockay game
stick kneeling in front of goalie
Figure 7: Images and texts used for the construction of the demonstration in Fig. T Colors correspond to the nodes' colors.
A.7.2 Universal Properties of Spectral Embedding (Fig. (b)
In this experiment, we investigated whether the eigenfunctions of the Laplace-Beltrane operator exhibit universal properties across modalities. Specifically, we asked whether the eigenfunctions of saplace-Beltrami independently constructed Laplace- Beltrami operators each unimodal representation, e and o. are similar. To this, we computed the first three eigenfunctions for each modality in the MSCOCO dataset using SpectralNet [67], and aligned them using CCA on a paired set. Fig. DP presents the cosine distances between pairs compared to the cosine distances between non-pairs. shows the cosine distances between aligned pairs compared to randomly matched non-pairs. Notably, paired samples are significantly closer the aligned space, indicating that for a pair (æ, x'), (xx) ~ ((x').
A.8 Extended Comparison Contrastive with Various #paired
Fig. extends Fig. 5l by including results for the remaining datasets from Tab. [ Flickr30k. Polyvore, and Edges2Shoes. The figure highlights that SUE effectively leverages unpaired data, enabling reduction in the number of required paired samples by more than factor of four on Flickr30k and Edges2Shoes, and by factor of two on Polyvore.
A.9 Qualitative Results
(almost-) text-free text-to-image generation. Fig. 9 extends Fig. 4p with additional text-to-image generation results, achieved with almost no text-image correspondence.
Image retrieval.
Fig. includes additional qualitative image retrieval examples.
Text retrieval.
Fig. includes qualitative texts retrieval examples.
Edges retrieval.
Fig. 12 includes qualitative edges retrieval examples the Edges2Shoes dataset.
26

---

34
Reccllte 10 (%)
Recalli81n (%0
Recailgh 10 (8)
30
22.5
20.0
23
13.5
15.0
500
730
1000
1250
1500
1750
2000 2256
500
600
700
000
900
1020
50
73
100
125
150
175
200
Paired Samples
Paired Samples
Paired Samples
Flickr30k
Polyvore
Edges2Shoes
Figure 8: Contrastive requires much more pairs to achieve similar results SUE in the weakly- paired regime. Extension Fig. 5a. Recall 10 results on Flickr30k, Polyvore, and Edges2Shoes by SUE and Contrastive with various numbers of pairs. SUE exploits unpaired data outperform contrastive learning when limited pairs are available. Much more pairs are required to achieve similar results with contrastive learning.
"a glasses
"a smilling"
baby"
"a smillnng
with
smiling"
Figure 9: text-to-image generation with minimal text-image correspondence. Additional exam- ples.
'iichen' hore leather over
"I bus that some nocky"
Jesn"
Figure 10: Additional image retrieval examples. Right: Polyvore; Left: MSCOCO.
bay a rod and white ihiriss on swing
givercty akinny Jean
ehitar being ner wast anseher child waseens..
girl cursmeottdd plays
spriscter park
Scpshop moto leigh oold wosh akinny joans
battyy palaping at parts
knowboarder vito
odtens and boandiess
sunglasses
baing
soter inovent stmeste pmepy round sungtasees
fore alter makting thas way mountsis
battage acatore teather sunglansee
ice
teather circte sunglasses
Figure 11: Text retrieval examples. Right: Flickr30k; Left: Polyvore.
Figure 12: Edges retrieval examples. Retrieved edges to shoe queries from the Edges2Shoes dataset.
27

---

A.10 Extended Ablation Study
It includes the retrieval results these corresponding ablations CCA (with the same number of paired samples as in SUE), and SE + MMD. The CCA results further support the key role of SE the universal embedding concept and SUE. In addition, the SE + MMD results show the necessity of CCA in the current version of SUE.
Tab. 2 extends the CCA and SE + MMD cells in Tab. 2
Table 9: Extended Ablation. R@@ 10 retrieval results on vision-language (mM) and vision-vision (mM) datasets from each modality to another: image-to-text (I2T), text-to-image (T2I), edges-to-shoes (E2S), tooeseto- (S2E), monesto-mmaurrmmen (N2M), measurement- toaeeennttttoooooo to-note (M2N). SUE results are significantly better, supporting the key role of SE in the universal embedding concept and SUE, as well the necessity the CCA step in the current implementation.
SUE (ours)
CCA
SE
MMD
Flickr30k
12T
32.00
3.75
4.75
T21
32.75
4.50
5,50
MSCOCO
12T
34,25
7.75
3.00
33.25
7.75
3.75
Polyvore
12T
32.25
4.00
3.75
T21
32.00
4.75
4.50
Edges2Shoes
E2S
26.75
4.00
1.75
S2E
25.50
3.25
2.50
Tab. 10 shows the results of another ablation experiment, where the SE step was replaced with an Auto-En. Notably, the new pipeline considered performs significantly worse compared than SUE, achieving near random results (e.g.. 2.75 in R @ 10, where random guess would achieve, on mean, 2.5). This again shows the pivotal role of SE in uncovering the universal embedding, as using different dimensionality reduction method results in incomparably worse performance.
Table 10: Spectral Embedding is a key component of universality. Image-to-text (I2T) and text-to-image (T2I) retrieval results on the Flickr30k dataset with 500 paired samples available during training, using SUE (SE CCA MMD), and AE + CCA MMD.
SUE (ours)
AE CCA MMD
#paired
500
500
R@1
R@S
R@
R@1
R@5
R@10
12T
5.00
20.75
32.50
0.50
1.50
2.75
T2I
5.75
23.00
33.50
0.50
1.50
2.75
A.11 Ablation Visualization
Fig. 13 visually demonstrates the three steps of SUE (SE, CCA, and MMD) and their combination achieve universal embedding. Specifically, the SEs of the two modalities (computed with no paired samples) are globally similar, but not aligned. The CCA step helps linearly align the two SEs. Then, the MMD step fine-tunes the aligmment non-linearly (and with no paired samples). Aligned with Tab. 2 this supports the key role of SE SUE's pipeline, the alignment achieved by the CCA and MMD steps is direct outcome of the SE, and cannot be achieved without it.
B
Modality Gap
Recent research involved the modality gap in multimodal learning 49 38 65). That is, the embed- dings of the two modalities of contrastive-based methods are located in two completely separate regions of the embedding space. Some efforts have been made to mitigate this gap 3g 23 In particular, the modality gap means that the embeddings are not universal, as the modalities lay on different intrinsic manifolds. In contrast, universal embedding should have no modality gap.
28

---

SE
+CCA
+MMD
Figure 13: SUE's UMAP ablation on Flickr30k. Images in red and texts in blue. The plots represent
each SUE's steps: SE, CCA, and MMD.
CLIP
SUE
Figure 14: SUE contains no modality gap. UMAP embeddings of Flickr30k text and image embeddings by CLIP and SUE. Images in red and texts in blue. Notably, while in the CLIP embedding texts and images lay on two distinct manifolds, SUE places the two modalities on the same manifold.
As visualized Fig. 14 SUE indeed does not contain modality gap. The embedding discovered by SUE is shared manifold of the two modalities (unlike CLIP, for instance). This arises from two core components our method: CCA produces whitened outputs, ensuring that the representations for each modality are centered at the origin, which implies zero modality gap by definition; and the MMD objective penalizes distributional discrepancies, further reducing any residual mismatch between modalities.
C Relative CLIP Score
lays bench while his dog sits by him
brown dog running through neck-ddep weter tennis bell
dog the water from sprinitter on tawn
showing hor painting and signing paper
Figure 15: Relative CLIP Score examples. The relative CL.P score matches our human intuition of semantic similarity.
The qualitative examples in Fig. and Fig. 4h depict the retrieval weakness in reliably measuring semantic coherence across modalities. We address this by presenting soft variant of Recall k for the text-image domain, which name Relative CLIP Score (RCS). CLIP score (CS) is popular measure of similarity between images [33], but is difficult to interpret. Following this idea, we define the RCS between pair of an image query and retrieved text the CS between them, relative to the CS between the image and its true pair (the RCS between a text query and retrieved image is defined analogously). This results in measurement that spans (approximately) between 0 to 1.
Fig. 15 demonstrates RCS in several examples. RCS means that the given image and text are pairs, and as the RCS decreases to 0 the image and pairs become gradually less similar semantically. To enable evaluating the RCS of CLIP, we use a later version of CLIP (ViT-L/14) as the evaluator of RCS.
29

---

Tab. [1 shows the RCS of SUE and CLIP on two image-text datasets. Notably, compared to CLIP (trained only on a huge amount of paired samples), SUE achieves about two-thirds performance with no more than 500 paired samples.
Table 11: Additional Relative CLIP Score results. RCS results by SUE and CLIP (ViT-L/32). SUE reliably captures global semantic structure while having seen less than 500 pairs during training.
Flickr30k
#paired
12T
T2I
SUE (ours)
500
0.667
0.638
CLIP
400M
0.971
0.973
D Theoretical Support
In this section, we cite Thm. 21 from 7 to support the claims made in Sec.
Theorem 1. (Bérard et al. 171, Thm. 21) Let he be any meiric M such that (1-e)g S h S (1+c),
e < en- assume furthermore that the metrics under consideration have their Ricci curvatures bounded from below by -(n 1)K² for some constant K. There exists constants 1g. x x(e), 1 S s No. which go to zero with c, such that to any orthogonal basis {v) of eigenfunction of D2 one can associate orthonormal basis { of eigenfanctions of Ag satisfying || -balooo S Mh..K (c) for i i No, where ol is the sup-norm.
Intuitively, the result shows that if embedders f.g g of common manifold .M have bounded distortion and bounded Ricci curvature, then the eigenfunctions of the associated Laplace-Belle Beltrami (or diffusion) operators (MM and g(M) are close in the L sense. For further discussion and explanations we refer to
E
Technical Details
E.1
Datasets
We use several datasets of varying modalities to evaluate SUE's performance and the existence of universal embedding. Tab. details the properties of each dataset.
E,2
Unimodal Models
As extensively discussed in Sec. and Sec. 4 our first step is extracting meaningful (unimodal) embeddings that capture semantic similarity, using pre-trained unimodal models. The unimodal models used for the different datasets include: DINOv2 (ViT-B/14 distilled) (58), VICReg 22. SBERT [63), the Barro-FashionSigL..I image encode and the MPMTS Measurements
encoder [1 Tab. [2 details the models used for each dataset. Tab. 13 details the key properties of each unimodal encoder.
Feature extraction details. For the text modality, which produce fixed-length sentence embeddings via mean pooling as implemented the library. For the image modality, we extract features using DINOv2, where the [CLS] token serves as the image embedding vector by default.
Centence-Transtommee (e.g., all all-MiniLM),
E.3
Generation and zero-shot
Image Generation. After obtaining the universal image embeddings from the caption-FFHQ dataset using SUE, we encoded the entire dataset of images with a pre-trained GAN inversion model to obtain the latent representation for all images. Next, we trained a converter model to map the
7
*This dataset consists of two handcrafted feature sets derived from images, and therefore can be viewed as tabular domain.
https: :/e:://wwwe //www/ marqo att/bloggsearcc--ooeel-- fashion
30

---

Table 12: Datasets' properties. #unpaired indicates the number of unpaired samples used by SUE; #paired indicates the number of paired samples used by SUE. Model refers to the unimodal encoder for the first modality, and Model 2 corresponds to the second modality. More details on the Unimodal encoders can found in App. E.2
Dataset
Modality
Modality
Model
Model 2
#unpaired
#paired
Flickr30k
Images
Captions
DINOv2
SBERT
27794
500
Polyvore
Images
Captions
Fighi
SBERT
17190
500
MSCOCO
Images
Captions
DINOv2
SBERT
8550
Edges2Shoes
Images
Edges (images)
DINOv2
VICReg
12690
50
caption-FFHQ
Images
Captions
DINOv2
SBERT
16958
1000
Office31
Images
Images
VICReg
DINOv2
8000
1000
Handwritter
Tabular
Tabular
NA
NA
1900
100
Table 13: Encoders' properties. For each encoder, we report the backbone architecture, the output dimension, and the number of parameters.
Model
Backbone
Output Dim.
Params
DINOv2 ViT-B/14
ViT-B (patch 14) ResNet-50
768-dim, 257 tokens
866.6
VICReg ResNet-50
7x7x2048 feature maps
~23M
FashionSigLIP
SigLIP-based vision-language MiniLM (6-layer Transformer)
768-dim (vision)
150M
SBERT (MiniLM-L6-v22)
384-dim sentence
22.7M
universal image embedding an input image to its corresponding latent. achieve this, we used an MSE loss to align the converter's output with the latent obtained from the GAN's encoder.
Once the converter model trained the universal image embeddings, leveraged the uni- versality property as follows: we mapped the text queries to their universal embeddings using SUE, passed them to the converter (which was trained solely on image embeddings), and then fed the output into the GAN's decoder to generate face image. As shown in Fig. 4p and Fig. 4c), the converter successfully mapped the universal text embeddings to the same latent space as the images, resulting in meaningful image generation.
For the converter, used neural network with hidden layers, each of size 2048. The training process is configured with a learning rate of 10-4, a batch size of 512, and an Adam optimizer, running for 600 epochs.
Zero-shot. For this experiment, collected random images of 7 object categories from the ImageNet dataset [17] cat, dog, man, woman, mountain, shore, snow. Fach label was converted into the following sentence: "A photo of []". For each input image, embedding using SUE, which was trained on Flickr30k and then calculated the cosine similarity between the image embedding and the universal embeddings of all label texts. The label for the image was predicted by selecting the text label that had the highest cosine similarity to the image's universal embedding.
computed its universal
E.4
Cross-domain Classification
For this experiment, we adjusted the data preparation process. Since the Office31 dataset is relatively small, augmented the data for each domain using four types of transformations: rotation, flipping, brightness adjustment, and contrast adjustment. This resulted approximately 30k samples for the Amazon domain and 9k samples for the DSLR domain.
We split the data into a training set (85% and test set (15%), ensuring classes were represented in both splits. For the training set, we created 1 paired samples, consisting of image pairs from the same class, while the remaining training samples were left unpaired, with varying numbers of samples per class across domains.
31

---

For unimodal models, we used VICReg 2 encoder for the Amazon domain and DINOv2 [58] for the DSLR domain. After obtaining universal embeddings from SUE the test set, we constructed KNN classifier using 5 neighbors on one domain and evaluated it on the other, as explained in App.
E.5
Graph Construction
As discussed in Sec. [ to calculate the Spectral Embedding (SE), graph affinity matrix is first constructed. Here we detail the graph construction used in SUE.
Given distance measure d between points, we first compute the k-nearest neighbors of each point z,. Let ( Tiy:.... Dig represent the k-nearest neighbors of æ; under d. For each point, we define:
Ps = min d( Di,); as = median (d( 1 [
Next, we use modified RBF kernel compute the affinity matrix:
W = exp (d(m).a.) a.)
if E {E Dirh otherwise
0
Finally, to ensure symmetry, we update W as:
W
+ W
W
2
As unimodal models are often trained using cosine similarity (c.g., 63
58
we analogously defined
d to be the cosine distance:
d(I;,I;)= =
| 15g
E.6 Training procedure
For the parametric computation of the SE, we adopted the training process presented [67]. and trained neural network capable of approximating the SE on the training set. Each modality was trained independently, and upon completion, the resulting parametric maps were used to compute the SE for the test points of each modality. For further details the training process for this parametric map, we refer the reader to [67].
To train the residual network for minimizing the squared MMD loss, used publicly available implementation the MMD loss in PyTorch 6 The network architecture consists of a single residual connection from the input to the output, without any inner residual connections.
E.7 Data split and preparation
Data Split. For each dataset, we excluded 400 paired samples for evaluation, using the remaining samples for training. To train the parametric SE model, the training set was further divided into 90% training subset and 10% validation subset. Similarly, during the training of the MMD network, the training set was partitioned into 90% training subset and 10% validation subset.
Data Preparation. After projecting the data using the unimodal models, we manipulated the data to include only a few paired samples, resulting in teakkly-paires data. Given n training samples, we split them into two portions: a paired portion where the samples remain perfectly aligned across modalities, and an unpaired portion where we introduce controlled noise. For the unpaired portion, we randomly remove 10% of the samples from each modality independently and shuffle their order. This process creates realistic scenario where only minimal number of corresponding pairs are available. while the rest are missing or misaligned. Notably, this data preparation procedure is only applied to the training set, as the test set requires properly paired samples to accurately compute cross-modal retrieval metrics and evaluate the model's performance.
32

---

E.8 Hyper-parameters
In this paragraph. we detail all the technical details regarding the different components of our implementation. This includes output dimensions of the different components, network architectures, and syper-parameters
Numeric SE. For the numeric SE, we constructed the graph as outlined in Sec. E.S using A = 100 for each point. The Laplacian matrix used is the random walk Laplacian, and we selected 10 eigenvectors, which correspond to the output dimension of the SE.
Parametrie SE. Computing the SE using the parametric approach involves training neural network. The network architecture consists of an MLP with hidden layers of sizes 4096, 4096, and 1024 for both networks (one per modality). We set the batch size to 4096, and the learning rate to 10-t with decay factor of 0. and the training was run for 100 epochs. The optimizer used is Adam, and the learning rate is adjusted using the PyTorch ReduceL.ROnPlatenn scheduler with a patience of 10. We used the same graph construction used for the numeric SE, outlined in Sec. E.5
As discussed in Sec. 4.2 the CCA projections are calculated using a small subset of paired samples, with the number of pairs fixed at 600 for all datasets. Fig. 5 presents an experiment demonstrating the impact of the amount of paired data on the results. For these projections, we utilized the CCA implementation from scikit-learr with the number of components set to 8 across all datasets.
CCA.
Residual Network (MMD). To train the residual network, we employed MLP architecture with hidden layers of size 128, 128, and 128, incorporating a residual connection from the input to the output. The optimizer used is AdamW, with a learning rate set to 10 The network was trained for 100 epochs.
Auto-Enee Euco-Encoder. To train the Auto-Encodes network, employed MLP encoder architecture identical to those of the parametric SE MLP, with corresponding decoder architecture. The optimizer used is AdamW, with learning rate set to 10 10-3 3, The network was trained for 100 epochs.
E.9
OS and Hardware
The training procedures were executed on Rocky Linux 9.3, utilizing Nvidia GPUs including GeForce GTX 1080 Ti and A A100 80GB PCIe.
(CA.html
33


Reviews:
[{"review_id": "xDoqSyKkvO", "reviewer": "Reviewer_bSLV", "summary": "This paper introduces a method for learning a shared embedding for multiple modalities, such as text and image, using primarily unpaired data. The core idea is to first learn unimodal representations for each modality separately using spectral embeddings, which are argued to effectively preserve the global manifold structure. Then these independently learned manifolds are aligned into a shared space using MMD and a small set of paired samples via CCA. The primary contribution is a framework that significantly reduces dependency on large-scale paired data.", "questions": "1.  **Impact of Pre-trained Encoders:** Could you please clarify exactly how features were extracted from the pre-trained backbones (e.g., `[CLS]` token vs. averaged features)? This introduces a hidden confounder that may \"pre-homogenize\" the features between modalities. E.g. filter out features from the backgrounds in images (which also tend not appear in the captions). Please discuss the potential confounding effect of using representations that were part of a pre-training objective. \nI would expect representations of images/videos to have a higher information density than text (especially captions), in which case the global manifold structure across modalities may not be as alignable - does this undercut the proposed method?\n2.  **Baseline Comparisons:** Could you provide comparisons against established semi-supervised or weakly-supervised multimodal learning methods? This is critical for assessing the paper's contribution, and my score would increase significantly if SUE proves competitive.\n3.  **Robustness of \"No Modality Gap\" Claim:** The claim that SUE has no modality gap (Appendix B) is quite interesting. Could it be highlighted more in the main paper? Is this observation robust across different datasets and embedding dimensions, or is it specific to the UMAP visualization shown in Figure 14? Generally, there should be more (of the extensive) results in the main paper, rather than relegated to the appendix.", "limitations": "No. The limitations should be discussed more thoroughly - e.g. inherent potential trade-offs between preserving global structure and fidelity.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n**Originality and Significance:** The paper tackles the highly significant problem of reducing the need for large-scale paired data in multimodal representation learning.\n\n**Methodological Soundness:** The design of the method is well-motivated, and explained clearly. The ablation studies presented in the appendix are a particular strength, clearly demonstrating how each component helps. Extensive experiments across multiple datasets and tasks provide strong empirical support for the method's effectiveness.\n\n### Weaknesses\n\n**Insufficient Baseline Comparisons:** The most critical weakness is the lack of adequate baseline comparisons. The comparison only comparison (in the main paper), is a naive contrastive baseline, making it impossible to judge the practical utility and trade-offs of the method. While the authors argue no direct baselines exist for this setting, several semi- or weakly-supervised methods (mentioned in the related work), would help contextualize the proposed method. Some discussion of existing methods for aligning representations such as https://openreview.net/forum?id=eimAJqoIWt&noteId=eimAJqoIWt or https://openreview.net/forum?id=l2izo0z7gu\nOn a related note, the choice of the autoencoder as a non-spectral baseline in the ablation study is not well-justified - why not a contrastive method (e.g. SimCLR)?\n\n**Coarse-Grained Alignment:** The qualitative results, particularly in the concept arithmetic experiments (Figure 4c), suggest that the alignment between modalities is rather coarse, failing to preserve fine-grained details like facial identity. This may be an inherent trade-off of the chosen approach; by prioritizing global manifold structure via spectral embeddings, the method may naturally lose local, fine-grained information. This trade-off should be investigated more thoroughly.\n\n### Minor Issues\n\n- l. 69: the abbreviation CCA is never defined, unlike MMD and SE.\n- l. 129: \"kenrnel\" -> \"kernel\"", "was_revised": true, "final_justification": "I have raised my score to reflect the authors' strong rebuttal, which directly addressed several of my concerns by adding a several new empirical results and discussion. However, I believe fully integrating the additional discussion, clarifications, and results may benefit from a resubmission. Nevertheless, the work is very promising.", "rebuttal": "We thank the Reviewer for their thoughtful and positive feedback. In particular, we are encouraged by the acknowledgment of the **importance and originality** of the problem we tackle, and the **clarity and motivation** underlying our methodology. We also appreciate your recognition of the **extensive empirical validation** of our approach, as well as the **insightful ablation studies** that demonstrate the contribution of each component. Below, we address your comments in detail, add new results, and clarify the points you raised.\n\n**W.1**\n\n**Baseline comparison.** Following the Reviewer’s suggestion, we have compared SUE to an additional baseline below. Specifically, we have added a comparison to CSA (proposed by Reviewer M2P3). CSA [1] is a recently introduced method designed to match CLIP performance while reducing reliance on paired data. The table below extends Tab. 1 in the main paper and compares our method (SUE), contrastive training, and CSA under the same limited pairing regime. Importantly, SUE significantly outperforms this new baseline in the extremely low-pair regime we focus on. We hope that the additional comparison provides better context for SUE's performance.\n\nNotably, while CSA effectively reduces the number of pairs needed to match CLIP, it is unable to operate in the extreme low-pair regime addressed by our paper. Specifically, when the number of pairs $n$ falls below the input dimension $d = \\min\\{d_1, d_2\\}$, CSA collapses. To enable a fair comparison, we use the minimal feasible number of pairs for CSA ($n=d$), which is still larger than the number used by our method (SUE). Even in this more favorable setting for CSA, our method (SUE) significantly outperforms it, highlighting the strength of our approach in this regime, and in leveraging unpaired data.\n\nWe will include these results in the revised version of the manuscript.\n\n| Dataset      | #paired | Direction | SUE R@1 | SUE R@5 | SUE R@10 | Contrastive R@1 | Contrastive R@5 | Contrastive R@10 | CSA R@1 | CSA R@5 | CSA R@10 |\n|--------------|---------|-----------|---------|---------|----------|------------------|------------------|-------------------|---------|---------|----------|\n| MSCOCO       | 100     | I2T       | 5.75    | 21.50   | 34.25    | 1.50             | 8.50             | 13.00             | 0.00    | 1.25    | 3.00      |\n|              |         | T2I       | 5.25    | 18.25   | 33.25    | 0.80             | 5.80             | 12.20             | 0.00    | 1.00    | 2.25      |\n| Flickr30k    | 500     | I2T       | 4.25    | 19.75   | 32.00    | 3.00             | 9.50             | 16.20             | 0.25    | 1.25    | 2.50      |\n|              |         | T2I       | 5.75    | 22.00   | 32.75    | 2.50             | 9.80             | 15.00             | 0.25    | 0.75    | 2.75      |\n| Polyvore     | 500     | I2T       | 6.00    | 22.75   | 32.25    | 3.20             | 13.80            | 22.50             | 0.25    | 1.25    | 2.25      |\n|              |         | T2I       | 4.75    | 20.75   | 32.00    | 4.00             | 11.50            | 23.00             | 0.25    | 1.00    | 3.25      |\n| Edges2Shoes  | 50      | E2S       | 4.00    | 16.00   | 25.25    | 1.00             | 5.50             | 14.00             | 0.25    | 1.50    | 2.75      |\n|              |         | S2E       | 3.50    | 17.00   | 27.00    | 0.80             | 6.00             | 12.80             | 0.25    | 1.50    | 3.00      |\n\n**References.** We thank the reviewer for pointing us to relevant references. Both will be cited and discussed in the revised related work section. However, similar to the other methods already mentioned in the related work, these approaches assume access to additional supervision or large-scale paired training data. For example, [2] incorporates class-level labels into the alignment process, whereas our setting is label-free. [3] depends on multimodal foundation models trained on hundreds of millions of paired examples, which falls outside the minimal-pair regime we consider. As such, while informative, these methods are not directly comparable to SUE.\n\n**AE ablation.** Our framework operates on pre-computed semantic embeddings (e.g., from Sentence-Transformers or DINOv2), not raw input data. In this setting, applying SimCLR or related contrastive methods is non-trivial: these methods are designed for raw data and rely heavily on stochastic augmentations to define positive/negative pairs, an approach that does not translate naturally to pre-extracted embedding spaces. Moreover, the contrastive paradigm assumes access to meaningful augmentations or label structure to guide the loss. In contrast, the autoencoder serves as a natural and neutral baseline in the embedding space, offering a non-spectral method that reconstructs input embeddings without introducing alignment or graph structure. This allows us to isolate and highlight the specific contribution of our spectral learning component.\n\n**W.2. Coarse-Grained Alignment:** We thank the reviewer for this insightful observation. Our work addresses the question: \"What can be achieved when only a small number of cross-modal pairs are available alongside large unpaired datasets?\" In this low-pair regime, standard paired models are inapplicable or ineffective. SUE provides a principled alternative that leverages the intrinsic structure of each modality for alignment, achieving downstream performance way beyond what was previously possible.\n\nWe agree with the Reviewer that our approach, by prioritizing global manifold structure through spectral embeddings, may naturally trade off some fine-grained detail, in favor of broader semantic alignment. We view this as an inherent and interesting property of our method, and believe that a deeper investigation of this trade-off between global alignment and local fidelity is a valuable direction for future work. We will revise the manuscript to clarify this point and provide better context for interpreting these qualitative results.\n\n**Q.1**\n\n**Feature extraction details.** For the text modality, we use Sentence-Transformers (e.g., all-MiniLM), which produce fixed-length sentence embeddings via mean pooling as implemented in the library. For the image modality, we extract features using DINOv2, where the [CLS] token serves as the image embedding vector by default. We will add these to the technical details appendix.\n\n**On the potential confounding effect.** We acknowledge that pre-trained encoders introduce semantic biases that may facilitate cross-modal alignment, as reflected in Fig. 2. However, rather than viewing this as a hidden confounder, we consider this semantic preprocessing a deliberate and essential design choice. By embedding modalities into meaningful feature spaces, SUE can effectively leverage unpaired data for alignment. Attempting alignment from raw pixels and tokens without such semantic abstraction would demand far more paired supervision, contradicting our goal of minimal pairing.\n\n**Information density and alignability.** Regarding the comment that images contain higher information density than captions, our method capitalizes on the semantic abstraction of these encoders. Both DINOv2 and Sentence-Transformers distill inputs into embeddings, emphasizing relevant semantic content while reducing irrelevant details such as background noise. This abstraction actually enhances alignability, as supported by the similar graph structures observed in Fig. 2, demonstrating that both modalities capture analogous semantic relationships conducive to spectral alignment.\n\n**Q.2.** Please see our response to **W.1**.\n\n**Q.3 Modality gap.** Thank you for your interest in our discussion of the modality gap. This finding is not limited to the UMAP visualization in Fig. 14, it holds consistently across datasets and embedding dimensions. It arises from two core components of our method: (1) CCA produces whitened outputs, ensuring that the representations for each modality are centered at the origin, which implies zero modality gap by definition; and (2) the MMD objective penalizes distributional discrepancies, further reducing any residual mismatch between modalities. We have verified this behavior across all datasets and will include the corresponding visualizations in the revised version. Due to space constraints, many of our extensive experiments were placed in the appendix; we will highlight key results, including the modality gap analysis, more prominently in the main paper.\n\nFinally, thank you for pointing out these minor issues. We will address them in the revised version.\n\n[1] Li et al. Csa: Data-efficient mapping of unimodal features to multimodal features. ICLR 2025.\n\n[2] Leeb et al. Partial alignment of representations via interventional consistency. ICLR 2025.\n\n[3] Wang et al. Omnibind: Large-scale omni multimodal representation via binding spaces. ICLR 2025."}, {"review_id": "hWWjKAlcxU", "reviewer": "Reviewer_MCZ3", "summary": "The paper aims to learn multi-modal representations from unpaired data. Current methods are trained on large amounts of paired data, which is often difficult to obtain. This work learns shared embeddings largely from unpaired data using spectral embeddings of random walk matrices built separately for each modality. The method is validated on vision-language and vision-vision datasets.", "questions": "1. The method is validated primarily on vision-language tasks with datasets that are carefully annotated paired data. It is unclear whether the approach would generalize to other modality pairs with weaker or noisier alignment. Further, the method is used for vision-language datasets which already have large amounts of paired data. What would be interesting is if this could be validated in domains where such aligned pairs are hard to obtain such as medical imaging, etc. \n2. The final MMD-based alignment step assumes that the distributions are already well-aligned after CCA. If this initial alignment is poor, MMD alone may not be sufficient to close the gap between modalities without paired supervision. How would this be mitigated? \n3. More details about how these unpaired datasets were constructed are needed. \n4. For retrieval, mAP should also be reported which considers the ranking of the retrieved examples. \n5. The paper claims ‘An order of magnitude more pairs are required to achieve similar results with contrastive learning’. Was any example mining for contrastive loss done? Would the result still hold if informative sampling is done?", "limitations": "Several important limitations as highlighted in detail in Weaknesses and Questions sections. \n1. Mainly related to crucial assumptions that the method is based on \n2. Need more extensive evaluation, comparison with the state-of-the-art and ablation studies", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and addresses an important topic. The motivation behind the problem is explained clearly. \n2. Provided ablation studies and visualization for validation on vision-language and vision-vision datasets. \n\nWeaknesses: \n1. The approach assumes that semantic similarity graphs (for example as shown in Fig 2), derived from unimodal foundation models are sufficiently aligned across different modalities. In practice, text and images can emphasize different semantic aspects. Also, the text caption is subjective to the annotator. This crucial assumption may not always hold.\n2. On a similar note, in lines 148-150, the approach assumes the existence of a single unified latent semantic manifold shared across modalities. Realistically, the semantic structure captured by different modalities (e.g., images vs. text) may differ significantly, violating this assumption.\n3. The argument that similar random walks imply aligned eigenfunctions is presented without rigorous guarantees. Stronger theoretical analysis is required to validate this critical assumption.\n4. The empirical evidence for the approach is drawn largely from vision-language data and some vision-vision data with curated and strongly paired datasets. It is unclear how well the method would generalize to other modality pairs with weaker or noisier semantic alignment. Further, the method is used for vision-language datasets which already have large amounts of paired data. What would be interesting is if this could be validated in domains where such aligned pairs are hard to obtain such as medical imaging, etc. \n5. My major concern is the lack of extensive evaluation. The method is only compared against Contrastive loss and no other state-of-the art methods or losses. \n6. An important ablation is needed. How much % of unpaired data is needed to roughly get similar performance as compared to fully paired data. Performance plotted across % of unpaired data would give us better insights about how well this method works.", "was_revised": true, "final_justification": "The rebuttal addressed most of my concerns and included additional evaluation. So I have raised my scores. However, I am still not convinced about W1, W2. While this may work for certain modalities that are semantically aligned, many modalities such as audio-video may capture complementary information, and this method may not work.", "rebuttal": "We thank the Reviewer for their insightful review. We are pleased that you found the paper to be **well-written** and addressing an **important topic**. We appreciate your recognition of the **clear motivation** behind our problem setting, as well as the value of our **ablation studies and visualizations**, which aim to validate our approach across both vision-language and vision-vision tasks. Below, we respond to your comments and provide further results and clarifications where needed.\n\n**W.1.** The assumption made in the paper, that similarity graphs, derived from unimodal foundation models are sufficiently aligned across different modalities, is supported in the paper with extensive experiments in the vision-language and vision-vision domains. As shown in Fig. 2, the random walks and spectral embeddings exhibit strong cross-modal similarity. Furthermore, Sec. 5 empirically validates this assumption by demonstrating that SUE performs effectively across multiple tasks. This assumption is also supported by recent works that report consistent structural alignment between modalities, including text-image pairs [1, 2, 3, 4], as well as more diverse combinations such as audio-text, LiDAR-text, and timeseries-text [5]. Notably, while text and images may each capture modality-specific nuances, prior work demonstrates that their global semantic structures remain well-aligned.\n\n**W.2.** Similarly to our answer above, perhaps surprisingly, we show that such a unified latent semantic manifold exists to some extent. While text and images may each capture modality-specific nuances, they still describe the same semantic objects, and thus share their global semantic structure. This assumption is supported by previous work [1, 2, 3, 4, 5].\n\n**W.3.** Thank you for this comment. In Section 4.1, we support the argument that similar random walks imply aligned eigenfunctions by referencing Theorem 21 from [6]. Specifically, the result shows that if two embeddings $f,g$ of a common manifold $\\mathcal{M}$ have bounded distortion and bounded Ricci curvature, then the eigenfunctions of the associated Laplace-Beltrami (or diffusion) operators on $f(\\mathcal{M})$ and  $g(\\mathcal{M})$ are close in the $L_{\\infty}$ sense. For completeness, we will add the theorem to the appendix.\n\nThis provides a rigorous justification for our assumption in the setting where the observed data lie on geometrically similar manifolds. While these assumptions may not hold perfectly in all practical cases, they offer a strong theoretical foundation for why alignment via spectral structure is both plausible and effective, especially when supported by empirical evidence such as the graph similarities shown in Fig. 2.\n\n**W.4.** We appreciate the Reviewer’s interest in the broader applicability of our method. In response, we have added results on a new multiview benchmark, the Handwritten dataset [7], which can be viewed as a tabular domain. This dataset consists of two handcrafted feature sets derived from images: (1) Karhunen–Loève coefficients (K); and (2) pixel averages over 2x3 windows (P), without the use of foundation models or pretrained embeddings. These experiments are thus fundamentally different from those in the main paper, and they provide evidence that SUE generalizes to non-vision-language settings. The results (see table in W.5 below) show that SUE significantly outperforms the contrastive baseline on this dataset as well.\n\nNotably, while our experiments mainly focus on vision-language and vision-vision settings, these domains were selected for their publicly available benchmarks, controllable pairing levels, and intuitive visualization, which support clear comparisons and understanding. We agree that extending SUE to other domains such as medical imaging is a compelling future direction. This requires further and deeper analysis, and we anticipate that SUE's ability to leverage unpaired data will provide clear advantages in those settings.\n\n**W.5.** Following the Reviewer’s suggestion, we have compared SUE to an additional baseline below. Specifically, we have added a comparison to CSA (proposed by Reviewer M2P3). CSA [5] is a recently introduced method designed to match CLIP performance while reducing reliance on paired data. The table below extends Tab. 1 in the main paper and compares our method (SUE), contrastive training, and CSA under the same limited pairing regime. Importantly, SUE significantly outperforms this new baseline in the extremely low-pair regime we focus on. We hope that the additional comparison provides better context for SUE's performance.\n\nNotably, while CSA effectively reduces the number of pairs needed to match CLIP, it is unable to operate in the extreme low-pair regime addressed by our paper. Specifically, when the number of pairs $n$ falls below the input dimension $d = \\min\\{d_1, d_2\\}$, CSA collapses. To enable a fair comparison, we use the minimal feasible number of pairs for CSA ($n=d$), which is still larger than the number used by our method (SUE). Even in this more favorable setting for CSA, our method (SUE) significantly outperforms it, highlighting the strength of our approach in this regime, and in leveraging unpaired data.\n\n| Dataset     | #   | | SUE R@1 |SUE R@5  |SUE R@10 | Contrastive R@1 |Contrastive  R@5  |Contrastive  R@10 | CSA R@1 | CSA R@5  | CSA R@10 |\n|----------|-----|-----|------|-----|-----|------|-----|-----|--------|-----|-----|\n| MSCOCO   |100  | I2T | 5.75 |21.5 |34.3| 1.50 | 8.5 |13.0 |   0.00 |1.25 |3.00 |\n|          |     | T2I | 5.25 |18.3 |33.3| 0.80 | 5.8 |12.2 |   0.00 |1.00 |2.25 |\n| Flickr30k|500  | I2T | 4.25 |19.8 |32.0| 3.00 | 9.5 |16.2 |   0.25 |1.25 |2.50 |\n|          |     | T2I | 5.75 |22.0 |32.8| 2.50 | 9.8 |15.0 |   0.25 |0.75 |2.75 |\n| Polyvore |500  | I2T | 6.00 |22.8 |32.3| 3.20 |13.8 |22.5 |   0.25 |1.25 |2.25 |\n|          |     | T2I | 4.75 |20.8 |32.0| 4.00 |11.5 |23.0 |   0.25 |1.00 |3.25 |\n| E2Shoes  | 50  | E2S | 4.00 |16.0 |25.3| 1.00 | 5.5 |14.0 |   0.25 |1.50 |2.75 |\n|          |     | S2E | 3.50 |17.0 |27.0| 0.80 | 6.0 |12.8 |   0.25 |1.50 |3.00 |\n| Handwritten|100  | K2P |25.50 |62.0 |79.0| 4.80 |17.0 |28.0 |   4.25 |12.2 |17.5 |\n|          |     | P2K |25.00 |61.8 |78.0| 4.80 |17.8 |30.5 |   3.50 |9.00 |15.0 |\n\n\n**W.6.** We thank the reviewer for raising this important point. The tradeoff between the amount of unpaired and paired data may be extracted from in Fig. 5b and Tab. 1. For instance, on the Flickr30k image retrieval task, the contrastive baseline achieves an R@10 of approximately 15 using 500 paired examples. SUE reaches comparable performance using around 15,000 unpaired samples. Importantly, as the amount of unpaired data increases, SUE's performance continues to improve, without requiring additional paired supervision. We will clarify this insight in the revised manuscript.\n\n**Q.1.** Please see our response to **W.4**.\n\n**Q.2.** We agree with the reviewer that the MMD-based alignment step is most effective when the modality-specific embeddings are already roughly aligned. As shown in our experiments (e.g., Tab. 1 and Tab. 2), this initial alignment is relatively strong, which allows the final MMD step to refine rather than create alignment. We also confirm the reviewer’s intuition that MMD alone is insufficient to bridge the modalities without prior alignment: our ablation study in Tab. 2 demonstrates that using MMD without the spectral embedding and CCA steps results in significantly worse performance.\n\n**Q.3.** Please see \"Data Preparation\" paragraph in App. D.7.\n\n**Q.4.** We thank the reviewer for the suggestion. We have computed mean Average Precision (mAP) results, and summarized them in the table below. SUE outperforms the baselines under this ranking-sensitive metric as well, further validating the effectiveness of our approach.\n\n| Dataset       | # | | **SUE** mAP  | Contrastive mAP| CSA mAP|\n|------------|----|-----|------|------|------|\n| MSCOCO     |100 | I2T | **9.72** | 5.00 | 1.48 |\n|            |    | T2I | **9.80** | 4.90 | 1.38 |\n| Flickr30k  |500 | I2T | **8.91** | 5.80 | 1.67 |\n|            |    | T2I |**8.07**| 5.80 | 1.67 |\n| Polyvore   |500 | I2T |**9.87** | 8.80 | 1.54 |\n|            |    | T2I |**9.35**| 9.20 | 1.68 |\n| E2Shoes    | 50 | E2S |**7.03** | 5.70 | 1.77 |\n|            |    | S2E |**6.99** | 5.10 | 1.88 |\n| Handwritten  |100 | K2P |**14.38** |12.60 | 9.42 |\n|            |    | P2K |**14.03**|12.90 | 7.96 |\n\n\n**Q.5.** We thank the reviewer for raising this point. If we understand correctly, the question refers to the use of informative example mining to improve the performance of contrastive learning. While this is a powerful technique in fully supervised settings, it is not applicable in our scenario. Our work explicitly focuses on the low-pair regime, where only a small, fixed set of aligned pairs is available, and additional pair selection or mining is not possible.\n\nThis setting reflects real-world constraints in weakly supervised multimodal learning, where collecting or identifying informative pairs is either expensive or infeasible. Therefore, the contrastive baselines we compare against operate under the same limited supervision, and our claim that contrastive learning requires an order of magnitude more pairs holds under this constraint.\n\n[1] Huh et al. Position: The platonic representation hypothesis. ICML 2024.\n\n[2] Fan et al. Scaling language-free visual representation learning. arXiv 2025.\n\n[3] Moschella, et al. Relative representations enable zero-shot latent space communication. ICLR 2023.\n\n[4] Norelli et al. Asif: Coupled data turns unimodal models to multimodal without training. NeurIPS 2023.\n\n[5] Li et al. Csa: Data-efficient mapping of unimodal features to multimodal features. ICLR 2025.\n\n[6] Bérard et al. Embedding riemannian manifolds by their heat kernel. GAFA 1994.\n\n[7] Duin. Multiple Features. UCI Machine Learning Repository, 1998."}, {"review_id": "6syM6BUvxB", "reviewer": "Reviewer_M2P3", "summary": "This paper focuses on a cross-modal representation learning problem with limited paired multimodal data instances and a relatively larger amount of unpaired unimodal data. The proposed method combines CCA and MMD, two existing methods. CCA first alignment between the representations from unimodal encoders, and MMD further refines the alignment.", "questions": "1.\tWeakness 1: I think a more comprehensive comparison to the referred works is needed, as this paper does not really compare the proposed method to any baselines. \n2.\tSee Weakness 2.\n3.\tI cannot find the number of parameters of the MMD network. I am skeptical of how the authors trained an MMD network using such a small amount of paired data, especially given that the contrastive baseline does not work.\nOverall, this is a strong paper. However, I have concerns about the lack of baseline comparisons and the unresolved issue in Q3. If these are addressed, I will consider updating my score.", "limitations": "No. \n\nLimitations:\nThe authors mentioned that the limitations are listed in Section 6. However, the section mainly focuses on future works and conclusion, where the discussion on limitations is sparse. I suggest that the authors separate the limitations to another section for clarity.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.\tThe paper provides very intuitive explanations to justify why to separate the methodology into 3 stages, especially to justify the random walk stage and the CCA stage.\n2.\tThe paper visualizes the results for the embedding space for a better understanding of the problem and method.\n3.\tThe authors ran an excessive number of experiments on various domains to demonstrate the capability of the proposed method.\nWeaknesses:\n1.\tI believe that the authors missed comparisons and reference to previous works on the same problem: “**when large amounts of unpaired data are available alongside only a very small number of paired samples**”, which contradicts the authors’ claim: “**To the best of our knowledge, this work is the first to learn shared representations from almost exclusively unpaired data.**”\nThe authors should compare their method with at least the most recent work cited below. [3] also leverages CCA to find a multimodal representation space with limited paired data, which I think is the same setting and similar methodology as this paper. \n\nReference:\n[1]  Moschella, Luca, et al. \"Relative representations enable zero-shot latent space communication.\" International Conference on Learning Representations (2023)\n[2] Norelli, Antonio, et al. \"Asif: Coupled data turns unimodal models to multimodal without training.\" Advances in Neural Information Processing Systems 36 (2023).\n[3] Li, Po-han, et al. \"CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features\". International Conference on Learning Representations (2025)\n\n2.\tThere is no ablation study on how changing the unimodal encoders affects the performance on the various tasks and the intuition of when to use which encoder.\nMinor Weaknesses:\n1.\tThe caption of Fig. 1 does not indicate that the retrieval result is by SUE.\n2.\t“he Marqo-FashionSigLIP image encoder” in Section D.\n3.\tThe figures are not sorted in the given numbers in the appendix (Fig. 14 comes earlier than Fig. 12 and Fig. 13).", "was_revised": true, "final_justification": "Per my comment.", "rebuttal": "We thank the Reviewer for their detailed and constructive review. We are glad that you appreciated the **intuitive justifications** provided for our three-stage methodology. We also value your positive remarks on our **visualizations**, which aim to offer clearer insight into the embedding space, and on our **extensive experimental evaluation** across diverse domains, which supports the effectiveness of our approach. Below, we address your comments, add new results, and clarify specific points.\n\n**Q.1.** We thank the reviewer for pointing out these relevant and timely works. In response, we have incorporated a comparison to CSA [3], which builds upon and improves over ASIF [2], itself based on [1]. Accordingly, we include all three works in the related work section and compare directly against CSA, the strongest among them.\n\nAs shown in our updated results below, SUE significantly outperforms CSA in the extremely low-pair regime we focus on. This highlights a key distinction: while CSA leverages limited paired data, it fails to operate in the setting we address, where the number of available pairs is extremely small, and unpaired data plays a central role.\n\nTherefore, although these prior works explore related goals, to the best of our knowledge, ours is the first to demonstrate successful shared representation learning from almost exclusively unpaired data. We will emphasize this distinction and the expanded comparison in the revised version.\n\nThe table below extends Tab. 1 in the main paper and compares our method (SUE), contrastive training, and CSA under the same limited pairing regime. Notably, while CSA effectively reduces the number of pairs needed to match CLIP, it is unable to operate in the extreme low-pair regime addressed by our paper. Specifically, when the number of pairs $n$ falls below the input dimension $d = \\min\\{d_1, d_2\\}$, CSA collapses. To enable a fair comparison, we use the minimal feasible number of pairs for CSA ($n=d$), which is still larger than the number used by our method (SUE). Even in this more favorable setting for CSA, our method (SUE) significantly outperforms it, highlighting the strength of our approach in this regime, and in leveraging unpaired data.\n\nWe will include these results in the revised version of the manuscript.\n\n| Dataset      | #paired | Direction | SUE R@1 | SUE R@5 | SUE R@10 | Contrastive R@1 | Contrastive R@5 | Contrastive R@10 | CSA R@1 | CSA R@5 | CSA R@10 |\n|--------------|---------|-----------|---------|---------|----------|------------------|------------------|-------------------|---------|---------|----------|\n| MSCOCO       | 100     | I2T       | 5.75    | 21.50   | 34.25    | 1.50             | 8.50             | 13.00             | 0.00    | 1.25    | 3.00      |\n|              |         | T2I       | 5.25    | 18.25   | 33.25    | 0.80             | 5.80             | 12.20             | 0.00    | 1.00    | 2.25      |\n| Flickr30k    | 500     | I2T       | 4.25    | 19.75   | 32.00    | 3.00             | 9.50             | 16.20             | 0.25    | 1.25    | 2.50      |\n|              |         | T2I       | 5.75    | 22.00   | 32.75    | 2.50             | 9.80             | 15.00             | 0.25    | 0.75    | 2.75      |\n| Polyvore     | 500     | I2T       | 6.00    | 22.75   | 32.25    | 3.20             | 13.80            | 22.50             | 0.25    | 1.25    | 2.25      |\n|              |         | T2I       | 4.75    | 20.75   | 32.00    | 4.00             | 11.50            | 23.00             | 0.25    | 1.00    | 3.25      |\n| Edges2Shoes  | 50      | E2S       | 4.00    | 16.00   | 25.25    | 1.00             | 5.50             | 14.00             | 0.25    | 1.50    | 2.75      |\n|              |         | S2E       | 3.50    | 17.00   | 27.00    | 0.80             | 6.00             | 12.80             | 0.25    | 1.50    | 3.00      |\n\n**Q.2.** The paper includes an ablation study on how changing the unimodal encoders affects the retrieval performance, in App. A.4. It is referred to from Sec. 5.1.\n\n**Q.3.** The architecture details for all components, including the MMD network, are provided in Appendix D.8. As discussed in Section 4.1, the MMD network is trained entirely on unpaired data, which is available in large quantities. It does not rely on paired supervision, so the limited number of pairs is not a limiting factor here. However, MMD alignment (especially using a residual network) assumes that the input distributions are already roughly aligned. For this reason, the MMD step acts only as a final fine-tuning stage, building on the alignment achieved by the previous pipeline components. As shown in Table 2 and Appendix A.1, MMD alone fails without this prior structure, reinforcing its role as a lightweight refinement rather than a standalone solution.\n\nFinally, thank you for pointing out these minor issues. We will address them in the revised version.\n\n[1] Moschella, et al. Relative representations enable zero-shot latent space communication. ICLR 2023.\n\n[2] Norelli et al. Asif: Coupled data turns unimodal models to multimodal without training. NeurIPS 2023.\n\n[3] Li et al. Csa: Data-efficient mapping of unimodal features to multimodal features. ICLR 2025."}, {"review_id": "ZMHbJhfWNP", "reviewer": "Reviewer_bd36", "summary": "The authors demonstrate that shared representations can be learned almost exclusively from unpaired data. First, they use pre-trained unimodal encoders to extract feature embeddings from each modality. Then, they use parametric spectral embeddings to reveal universal structures. Finally, they align the modalities linearly via canonical correlation analysis and correct any misalignment using maximum mean discrepancy.", "questions": "* The paper would greatly benefit from additional experiments concerning scaling and additional modalities.\n* The text encoder seems weak and outdated compared to the vision model, which should be changed.\n* The results for other downstream tasks do not seem as convincing and should be better contextualized.\n* It is worrying that (Fig. 5c) the method does not scale with additional labels; if the scaling from unpaired data also plateaus, then the proposed approach has reached its limits, while others like CLIP would still scale.", "limitations": "Limitations stated in the paper.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n* The authors address an important problem in multimodal learning, supporting their claims with compelling evidence.\n* The theory is clearly explained and well-motivated, which helps readers better understand the proposed method.\n* The ablation study complements the main experiments and connects the theoretical and empirical parts of the paper.\n\n**Weakness**\n* The experiments were conducted using only image and text modalities. Ideally, more pairs should be included.\n* The scaling experiments in Figure 5 should be extended to see if and when the performance plateaus, as in (c).\n* The authors use a SOTA vision encoder (DINOv2), but the text encoder (Mini-LM) seems small and outdated.", "was_revised": true, "final_justification": "The proposed method is very interesting and has a lot of potential. The additional experiments support the claim that SUE can be applied to other settings. However, as the other reviewers noted, the experimental section is a bit lacking. I suggested to the authors to take a look at the paper \"Assessing and Learning Alignment of Unimodal Vision and Language Models\" by Zhang et al. (CVPR 2025). Like SUE, SAIL uses two pre-trained encoders from each modality and aligns them. Since both methods use contrastive learning, SAIL may require more paired data than SUE but this needs to be evaluated. Nevertheless, the SAIL paper's experiments are very comprehensive, involving image and text encoders of various sizes and model types which is not given here. The other reviewers also share the same concern as me that the alignment heavily depends on the type of modalities with image and text having strong correlations. Modalities from other domains such as single-cell images and single-cell expressions might not fulfill this requirement. This should be better contextualized in the paper. Overall, I increased my score from 4 to 5 but 4.5 would better reflect my assessment.", "rebuttal": "We thank the Reviewer for their thoughtful and encouraging feedback. We are pleased that you found the **problem setting important** and the **theoretical motivation clear and well-explained**. We also appreciate your recognition of the **ablation study** as a meaningful bridge between theory and empirical results. Below, we respond in detail to your comments and suggestions.\n\n**Q.1a: Additional modalities.** We appreciate the Reviewer’s interest in the broader applicability of our method. In response, we have added results on a new multiview benchmark, the Handwritten dataset [1], which can be viewed as a tabular domain. This dataset consists of two handcrafted feature sets derived from images: (1) Karhunen–Loève coefficients (K); and (2) pixel averages over 2x3 windows (P), without the use of foundation models or pretrained embeddings. These experiments are thus fundamentally different from those in the main paper, and they provide evidence that SUE generalizes to non-vision-language settings. The table below extends Tab. 1 from the main paper and shows that SUE significantly outperforms the contrastive baseline on this dataset as well.\n\n| Dataset     | #paired | Direction | SUE R@1 | SUE R@5 | SUE R@10 | Contrastive R@1 | Contrastive R@5 | Contrastive R@10 |\n|-------------|---------|-----------|---------|---------|----------|------------------|------------------|-------------------|\n| Handwritten | 100     | K2P       | 25.50   | 62.00   | 79.00    | 4.80             | 17.00            | 28.00             |\n|             |         | P2K       | 25.00   | 61.75   | 78.00    | 4.80             | 17.80            | 30.50             |\n\nNotably, while our experiments mainly focus on vision-language and vision-vision settings, these domains were selected for their publicly available benchmarks, controllable pairing levels, and intuitive visualization, which support clear comparisons and understanding. We agree that extending SUE to other domains such as medical imaging is a compelling future direction. This requires further and deeper analysis, and we anticipate that SUE's ability to leverage unpaired data will provide clear advantages in those settings.\n\n**Q.1b: Unpaired scaling.** We appreciate the reviewer’s suggestion to extend the scaling experiment in Fig. 5 to observe whether and when performance plateaus. However, Fig. 5b already uses the full available dataset, and unfortunately, no additional unpaired samples are available for this benchmark. Importantly, even at this point, SUE already significantly outperforms the baseline. Notably, performance continues to improve with the amount of unpaired data throughout the observed range. We will clarify this observation in the revised manuscript and view investigating the saturation regime as an important direction for future work with larger or synthetic datasets.\n\n**Q.2: Text encoder.** Thank you for the observation. To address this point, we extended our encoder ablation study in App. A.4 to include a stronger and more recent text encoder, GTR [2]. The results are provided in the table below, extending Tab. 7 from the paper.. Interestingly, absolute performance is slightly lower. We hypothesize that the Mini-LM was already sufficient given the strength of the vision encoder (DINOv2), or alternatively, that GTR is less aligned with the image representations. In either case, the core effectiveness of our method appears robust to the choice of text encoder.\n\n| Encoders                             | R@10  |\n|--------------------------------------|-------|\n| Dinov2 (ViT-B/14) + GTR (t5-large)   | 31.25 |\n\n**Q.3: Downstream task results.** We thank the reviewer for this observation. This paper addresses the question: \"What can be achieved when only a small number of pairs are available, alongside large unpaired datasets?\" In this setting, standard paired models fall short. While SUE is not intended to compete with models such as CLIP trained on 400M image-text pairs, it offers a compelling alternative for low-pair regimes where such resources are unavailable. In this context, the downstream task performance achieved by SUE is significantly beyond what was previously possible. We will revise the manuscript to better highlight this setting and provide additional context for interpreting downstream results.\n\n**Q.4: Scaling.** We thank the Reviewer for raising this important point. As discussed in previous responses, our work specifically addresses the underexplored and practically relevant setting where only a small number of paired samples are available alongside large amounts of unpaired data. In this scenario, existing contrastive methods like CLIP, which require hundreds of millions of pairs, are not applicable. Instead, our goal is to ask: What is achievable in the absence of large-scale paired datasets? In this regime, SUE offers a strong solution and significantly outperforms existing baselines (see additional baseline in the general comment). We agree that investigating whether performance continues to improve with more unpaired data is an important direction for future work.\n\n**Q.5: $k \\geq r$ assumption.** In our method, the original data from each modality lies in $\\mathbb{R}^{d_1}$ and $\\mathbb{R}^{d_2}$, respectively. These are first reduced to a common dimension $k$ via Spectral Embedding. Then, CCA is applied to project the data from $\\mathbb{R}^d$ to a shared latent space of dimension $r$. Since CCA performs a projection onto the top $r$ canonical directions, we require $k \\geq r$ to ensure the projection is well-defined. This assumption is standard in CCA-based methods.\n\n**Q.6: MMD network.** The MMD residual network is intended as a final refinement step between two approximately aligned, equi-dimensional distributions. For simplicity and efficiency, we chose to apply the residual network only to $Y$, projecting it to better match $X$. An alternative would be to symmetrically transform both $X$ and $Y$ into a shared space that minimizes the MMD between them. While this could potentially improve alignment, we found the simpler asymmetric approach sufficient in our setting and left such architectural variations for future exploration.\n\n**Q.7: Universal manifold.** The central assumption in our paper is that similarity graphs constructed from unimodal foundation models are sufficiently aligned across different modalities. We support this through extensive experiments in both vision-language and vision-vision domains. As shown in Fig. 2, the random walks and spectral embeddings exhibit strong cross-modal similarity. Furthermore, Sec. 5 empirically validates this assumption by demonstrating that SUE performs effectively across multiple tasks. This assumption is also supported by recent works that report consistent structural alignment between modalities, including text-image pairs [3, 4, 5, 6], as well as more diverse combinations such as audio-text, LiDAR-text, and timeseries-text [7]. While our results affirm the utility of this alignment in current domains, extending SUE to additional modalities is a promising direction for future research and will require further analysis.\n\n**Q.8: proven ability.** Thank you for pointing this out. We will add a citation to [3], which directly demonstrates and discusses the semantic representation capabilities of unimodal models, supporting our claim.\n\nFinally, thank you for pointing out these paper formatting improvements. We will include them in the revised version.\n\n[1] Duin. Multiple Features. UCI Machine Learning Repository, 1998.\n\n[2] Ni et al. Large dual encoders are generalizable retrievers. ACL 2022.\n\n[3] Huh et al. Position: The platonic representation hypothesis. ICML 2024.\n\n[4] Fan et al. Scaling language-free visual representation learning. arXiv 2025.\n\n[5] Moschella, et al. Relative representations enable zero-shot latent space communication. ICLR 2023.\n\n[6] Norelli et al. Asif: Coupled data turns unimodal models to multimodal without training. NeurIPS 2023.\n\n[7] Li et al. Csa: Data-efficient mapping of unimodal features to multimodal features. ICLR 2025."}]

Author rebuttal:

Meta-review: All reviewers suggest accepting the paper, so will be accepted. Please address the remaining concerns in the camera ready version.
Decision: Accept (poster)