Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models
Authors: Sharut Gupta
Shobhita Sundaram
Chenyu Wang
Stefanie Jegelka
Phillip Isola
Venue: ICLR 2026 Poster (2026)
Keywords: Unpaired Multimodal Representation Learning
Cross-modal Learning; Multimodal Learning from Unpaired Data
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We show that incorporating auxiliary unpaired multimodal data can significantly improve performance on an individual modality.

Abstract: Traditional multimodal learners find unified representations for tasks like visual question answering, but rely heavily on large paired datasets. However, an overlooked yet potentially powerful question is: can one leverage auxiliary $\textit{unpaired}$ multimodal data to directly enhance representation learning in a $\textit{target}$ modality? We introduce $\textbf{UML}$: $\textbf{U}$npaired $\textbf{M}$ultimodal $\textbf{L}$earner, a modality-agnostic training paradigm in which a single model alternately processes inputs from different modalities while sharing parameters across them. This design exploits the assumption that different modalities are projections of a shared underlying reality, allowing the model to benefit from cross-modal structure without requiring explicit pairs. Theoretically, under linear data-generating assumptions, we show that unpaired auxiliary data can yield representations strictly more informative about the world than unimodal training. Empirically, we show that incorporating unpaired data that share underlying semantic information from auxiliary modalities—such as text, audio, or images—consistently improves downstream performance across diverse unimodal targets such as image and audio. Our project page: https://unpaired-multimodal.github.io/

Published as a conference paper at ICLR 2026
BETTER TOGETHER: LEVERAGING UNPAIRED MULTI- MODAL DATA FOR STRONGER UNIMODAL MODELS
Sharut Gupta
Shobhita Sundaram
Chenyu Wang
MIT CSAIL sharut @mit edu
MIT CSAIL
MIT CSAIL
shobhita@mit.eem
wangchyemit edu
Stefanie Jegelka TU Munich, MIT CSAIL stefje@mit.edu
Phillip Isola
MIT CSAIL
phillipi@mit. edu
Unpaired Multimodal Representation Learning
Downstream Task
Image-only
Image w/ unpairedi text
Images
Image Modeling
Classifir Logits
Objective
Weight sharing
Viceroy
Homarch
Unpaired Text
"Monarch Butterfly has orange with black veins"
Language Modeling
bushy stalks silently at down"
Objective
Viccooy resembles Monarch but
has 1ine across hindwing"
Figure 1: Text provides complementary information beyond images, even when not paired directly; We introduce UNPAIRED MULTIMMOAL LEARNER (UML) whereby sharing model weights across modalities (e.g., image and text) extracts synergies and enhances unimodal representations, outperforming methods that rely only on single modality (such images above).
ABSTRACT
Traditional multimodal learners find unified representations for tasks like visual question answering, but rely heavily on large paired datasets. However, an overlooked yet potentially powerful question can one leverage auxiliary unpaired multimodal data to directly enhance representation learning in a targel modality? We introduce UML: UNPAIRED MULTIMODAL LEARNER, a modality-agnostic training paradigm in which single model alternately processes inputs from different modalitics while sharing parameters across them. This design exploits the assumption that different modalities are projections of a shared underlying reality, allowing the model to benefit from cross-modal structure without requiring explicit pairs. Theoretically, under linear data-generating assumptions, we show that unpaired auxiliary data can yield representations strictly more informative about the world than unimodal training. Empirically, we show that using unpaired data that share underlying semantic information from auxiliary modalities suchh as text, audio, or mongges-cossssenttee improves downstream performance across diverse unimodal targets such image and audio. Our project page: https: //uppataat red-mul moda g thub. io/.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
It is often taken for granted that to model modality well, one must train on data from that modality, For instance, if one wants an accurate image classifier, one trains on images; if one wants language model, one trains on text. Recent advances in multimodal learning. however, suggest that multiple modalities can benefit one another: in particular, using text captions paired with images yields richer representations, often surpassing their unimodal counterparts in zero-shot transfer, cross-modal retrieval, and downstream classification (Radford et al., 2021; Singh et al., 2022; Mizrahi et al., 2023; Girdhar et al., 2023a; Bachmann et al., 2022; Li et al., 2023; Bachmann et al., 2024; Jia et al., 2021). However, these gains have been realized largely through paired data, where one has access to aligned examples (z., y) ~ Px,y, such as an image x and its corresponding caption y. Such paired supervision allows aligning modalities into shared latent space, where cross-modal correlations can be captured and transferred to downstream tasks.
This reliance paired corpora also poses bottleneck. Collecting and curating aligned datasets is expensive and domain-limited, whereas unpaired data-innepeenden samples from px and py Py-are naturally abundant. For example, vast image collections and vast text corpora exist independently, but without explicit alignment. This raises more fundamental question:
Can unpaired data from a secondary modality Y enhance representations of the target modality X. even the absence of (z. y) correspondences?
There is growing evidence that the answer may be yes. Recent work posits the existence of shared statistical model of reality, an empirical parallel to Plato's concept of ideal Forms, suggesting that as deep networks scale, their embeddings across modalities converge toward unified representation of the underlying world (Huh et al., 2024; Huang et al., 2021). This convergence implies that when paired supervision is available, model can leverage natural co-occurrences between modalities and thus more accurately capture shared semantics. Critically, however, achieving convergence does not necessarily require explicit pairs; as long as each modality samples from the same underlying ground- truth latent space, it may be sufficient to align their marginal distributions to uncover the common semantic structure (Timilsina et al., 2024; Sturma al., 2023). Intuitively, even unpaired data from another modality can provide complementary cues to better estimate the underlying reality (Figure 1).
In this work, we formalize this idea through Unpaired Multimodal Representation Learning, framework for improving unimodal representations by leveraging unpaired data across modalities. Theoretically, under linear assumptions, we derive conditions under which incorporating unpaired samples yields strictly more informative representations than unimodal training alone. Strikingly, in some regimes, a single sample from Y provides greater per-sample value than a sample from X itself when the goal is to model X. Building on these insights, we introduce UML: UNPAIRED MULTIMODAL LEARNER, a shared-network framework that applies the same set of parameters to in- puts from different modalities. By nothing more than weight sharing i.e. without surrogate objectives or inferred alignments, the model learns modality-agnose tion across modalities (Sutskever, 2023). Importantly, this does not imply that UML or unpaired multimodal training more broadly is immune to the optimization challenges due to modality conflicts documented in prior works for paired learning (Huang et al., 2021; Wang al., 2020; Ma et al., 2025; Du et al., 2021; Xu et al., 2023; Fan et al., 2023; Sun et al., 2021: Zhang et al., 2024; Zhou et al., 2023; Wu al., 2022); rather, it provides an existence proof of regimes where unpaired multimodal data can help, without guaranteeing such benefits in all multimodal training settings. Empirically, we evaluate UML on diverse image-text tasks in healtheare, and affective computing, as well 10 standard visual benchmarks. Unpaired data from auxiliary modalities consistently improves unimodal representations across eef-suuprviied and supervised regimes, in both few-shot and full-data regimes, and with diverse encoders including CLIP, DINOv2, OpenLLaMA, and others. These benefits compound when moving from two to three modalities, with audio, vision, and text each adding complementary signals. We further demonstrate effective cross-modal transfer without paired data by initializing vision models with pretrained language-monl model weights. Finally, we quantify the exchange rate between modalities, mapping how many words equate to image (and vice versa) for optimal performance. These performance gains are clearest when there is shared semantic information between modalities, and do not necessarily apply to every setting (e.g., semantically unrelated modalities).
features and naturally transfers informa-
2

---

Published as a conference paper at ICLR 2026
To summarize, the key contributions of our work are:
We introduce UMI., modality-agnostic framework that leverages unpaired data to improve unimodal models. We test it across self-supervised and supervised encoders (CLIP, DINOv2, OpenLLaMA, and others), in both few-shot and full-data regimes, showing consistent gains over range of image-text benchmarks and extensions to audio.
We theoretically characterize, under linear assumptions, the conditions where unpaired data yield strictly more informative representations than unimodal training. Remarkably, the conversion ratio between modalities can fall below one: in certain regimes, single sample from Y contributes more to modeling X than an additional sample from X itself.
We quantify conversion ratios between images and text; i.e., how many words is an image worth for training vision models. We also show that unpaired multimodal data systematically widens inter-class margins and aligns modalities in weights.
PAIRED MULTIMODAL REPRESENTATION LEARNING
A useful way to conceptualize learning across modalities is to posit shared ground-truth reality, denoted Z", which manifests through multiple projections such as images, text, or audio record- ings (Huh et al., 2024; Timilsina et al., 2024; Sturma al., 2023). The goal of representation learning to learn an embedding space that captures the structure of 2", whether from single modality or from several jointly. Thus, unimodal representations alone can reveal: a single camera view may contain occlusions; an audio recording lacks visual details; textual descriptions may lack layout information.
inherently limited by what one projection
These limitations motivate multimodal representation learning, which extends the unimodal setting learn from multiple modalities together. For brevity, consider two modalities with observable data denoted by X' (c.g., images) and y (e.g., text) and samples æ E X and E V. Mathematically, Multimodal representation learning seeks encoders fe. X and fo, y + Z mapping each modality into shared embedding space Z.
Different frameworks optimize for desirable properties of 2. Contrastive methods encourage instances of the same concept (such as an image and its text label) to be close together (Radford et al., 2021; Zhai et al., 2023; Jia et al., 2021). Fusion approaches learn shared layers for multimodal inputs with reconstruction (Singh et al., 2022; Li et al., 2019; Lu et al., 2019; Bachmann et al., 2022) or contrastive objectives (Roy et al., 2025). Generative methods often structure Z to enable translation between modalities(Li et al., 2022; Rombach et al., 2022). Other methods disentangle common factors from coodality-speciffe factors (Wang et al., 2024; Liang et al., 2023). Further works have also explored the limitations of paired multimodal learning. including paired classification networks, multimodal LLMs, and theoretical multimodal studies (Zhang et al., 2025; Ma et al., 2025; Wang al., 2020; Javaloy al., 2022). These works show that in such settings, and without special treatment, modality conflict can give rise to hallucination, optimization, and convergence challenges.
(a) Paired Data
(b) Unpaired Data (w/ labels)
(c) Unpaired Data
"A red panda Flokks at the comera"
"A guinea pig nests on grass" Aig
pig grass"
parttli
"Two dogs play hapoily"
raboit the flowers" Rabibk
"A nabest the #1 Flowenss
Dog
"A guires on grass"
elephant is large nanmal"
elephant large mimmal"
Guinea Pig
Eupphett
Figure 2: (a) Paired learning uses (I,, 91) with known correspondences. We instead study Unpaired learning: (b) with labels, using (x,, q)) and (His C,), where e; and é, denote labels for 2; and yy, but cross- modal correspondences; and (c) without any labels correspondences, using {2;} and (u₂}, )-
3

---

Published as a conference paper at ICLR 2026
Common all of these prior work, however, is requirement for pre-existing knowledge of one-to- one correspondences between the modalities, ie., samples (z, y) Px.y (as shown in Figure 2(a)), which are assumed be aligned views of the same underlying entity in Z We ask whether it is possible to recover this shared structure without such correspondences-nanme samples from the marginals Px and Py are available, as shown in Figure when only unpaired affects downstream performance individually on the primary modality X.
if so, how this
3 UNPAIRED MULTIMODAL. REPRESENTATION LEARNING
{x;) and Dn = {y, 2 be datasets of Ng and N, samples respectively, drawn from marginal distributions Px and Py as shown in Figure 2(c). The joint distribution Px,y is unknown. Critically, in contrast to paired multimodal learning, for any given æ; E Dxx and y/ E Dy, there no assumption that v, and y, are projections of the same underlying entity E 2.
Let Dx = Dx===x=-
Our objective is to learn mappings fx X - Z and fy : y + 2 from each modality to a common embedding space that captures the shared structures between the unpaired datasets. We refer to this as Unpaired Multimodal Representation Learning. In contrast to previous works (Lin et al., 2023; Lee Yoon, 2025; Girdhar et al., 2022; Chada et al., 2023) (discussed in Appendix A), we leverage unpaired data without inferring alignment, incorporating paired data, or assuming pre-aligned embeddings. Intuitively, without known correspondences, even unpaired data can be useful if modalities capture different axes of information pertaining to the underlying 2". We formalize this intuition in Sec- tion 3.1, within the popularly studied framework of linear models, and introduce a modality-agnostic algorithm UML in Section 3.2 for learning representations from unpaired data. Our analysis is purely iiformttion-thooreti and does not characterize optimization or convergence behaviors.
3.1 THEORETICAL PERSPECTIVES
We adopt the widely used modeling choice that each modality can be expressed as a linear combination of shared and modality-specifff components (Sturma al., 2023; Timilsina al., 2024; Huang et al., 2021). This formulation inherently accounts for informational differences across modalities by incorporating modality-specific features (Wang et al., 2024). Prior works (Sturma et al., 2023; Timilsina et al., 2024) have developed sufficient conditions for recovering joint distributions and the shared causal structure under this lincar model. Building on this framework, we ask how joint training with unpaired modalities can still enhance representations within a single modality.
Data Generating Process. Assume that all factors of variation in reality live in single d-dimensional space, Z², i.e., A E Rd, modeled using linear data- generating pipeline. This parameter can further be decomposed as 0 = [0., 0z 6y] where 0g Rdd. 02 E I2 By Rd and de + dN + d, Here, O. captures the common (shared) parameters that affect both modalities, 0₂ denotes the parameters that only affect modality X, and f, denotes the parameters that only affect modality Y. We observe two datasets, one from each modality (X.),"i E RT and ( E IR", each reflecting partial measurements of the ground truth latent space Z":
X; A On + A On + EX,
ex, ((0, mm Y Bag Og + By. ey.h ey. N(0,, InIn).
Here, Ac,i Ar, Beg By.j are known design blocks capturing how each sample probes the latent factors and EX,,, Ey.j represent the independent measurement noise.
In our linear setting, estimating the true latent state 0 and hence the underlying reality Z governed by the Fisher information matrix [(0) = -E V ((0) (Fisher, 1922; 1925), which measures how sharply the likelihood "curves" around the true A. Because X and Y samples are independent given 2', their curvature contributions add pointwise, resulting in the joint Fisher information being simply the sum of the unimodal blocks. Thus, loosely speaking, any nonzero contribution from the unpaired Yssaplles strictly increases curvature and thus strictly tightens the variance bound along those directions in f... We formalize these insights in Theorem and Theorem 2. Our results also generalize naturally to more than two modalities since the total contribution of all auxiliary modalities (exeluding the primary modality X) can be obtained by summing their individual Fisher information matrices. Complete proofs and additional background are provided in Appendix C.
4

---

Published as a conference paper at ICLR 2026
Theorem (Variance Reduction with Unpaired Multimodal Data). Let 0x,0y by be the least-squares
estimators for 0 using only {X;} and only (Y₂) and let 0x,y be the joint estimator using both unpaired datasets. Then, under the assumption that at least one Be.j, where j E (1,2....Ngh has full rank, the common-factor covariance satisfies the strict Loewner ordering i.e. Var()x,))o.. 0c,0c K
Var(Ox) 9.,0" equivalently, the Fisher information on Oe stricily increases when combining both modalities, despite not having sample-wise pairing: [Ix + )0..0. > (Ix)0..9.
Theorem says that although there is no sample-wise pairing, simply adding an unpaired modality Y, that carries non-degenerate information about every direction in the shared subspace, can only tighten our uncertainty about the shared parameters 9e. Geometrically, the uncertainty ellipsoid for Og shrinks in every direction where Y contributes any curvature, and never expands elsewhere. In practice however, no single data sample covers every latent axis. In these settings, while global shrinkage (Theorem 1) longer applies, we still get directional gains. We capture this in Theorem 2.
Theorem (Directional Variance Reduction with Unpaired Multimodal Data). Let all notation be as in Theorem I, and let E Rdd \ {0}. If there exists at least index j E {1, (.,,,..N.)) such that Beg # O, then v (Ix + le... direction u is strictly reduced if range((Tx)(...
v (Ix)e,,e Further, the variance of the estimator in
Building on Theorem 2, Corollary and Corollary 2 quantify the precise variance contraction factor and analyze special cases where second modality resolves otherwise ill-posed directions. Thus far, we have studied incorporating data from an auxiliary modality without considering sample size constraints. A natural next question is how to allocate a fixed budget of N additional samples: it better to collect them from the same modality X, or from complementary modality Y? We formally address this in Theorem 3 below.
Theorem (Data from Auxiliary Modality Can Outperform More of the Same). Define for any m, (m))
- A A Acji and 1\/l
=Éi=1 BT Be.. If rraagee (mm)) g range x then there
x
exists nonzero Rdd such that I(mm) v 1 (m)
Unsalred Y)-datx X-data
Theorem 3 shows that if the second modality covers "blind spot" of the first, adding additional samples from the first modality does not increase the Fisher information in that direction; however, unpaired samples from the second modality provide strictly positive information along that axis.
Reconstruction Error
3.0
2.5
1.5
To empirically validate the above result, we design an illus- trative Gaussian experiment (Appendix E.14) in which both modalities are generated as noisy linear projections of the same underlying Gaussian latent variable 0.. We allocate a fixed budget of samples, splitting evenly between X and Y (details in Appendix E. 14). Joint training with UML significantly im- proves reconstruction fidelity on X (Figure 3) compared to training on X alone. Strikingly, this also reveals that the effec- tive exchange rate between modalities is below one, meaning single additional sample from Y is worth more than an additional sample from X, even when the test task is on X. In Section 4.4, we extend this idea by quantifying the exchange rate between images and text on real world benchmarks.
1.0
200
Training Steps
Figure 3: Adding unpaired Y samples boosts X reconstruction more than adding extra X samples.
3.2
UML: UNPAIRED MULTIMODAL LEARNER
The theory developed above establishes that adding unpaired auxiliary modality strictly increases Fisher information along shared directions, thereby reducing estimator variance. We now translate these insights into a practical framework: UML (UnPAIRED MULTIMODAL LEARNER). The central idea is to share parameters across modalities: since both and Y are projections of the same underlying reality Z", forcing them through shared weights can extract synergies by accumulating training gradients on the same parameters. Under the assumptions in Proposition 1, this accumulation additivity on the shared parameters, ensuring that even without paired
yields exact
5

---

Published as a conference paper at ICLR 2026
Tokenization and Embed
Unpaired Multimodal Learner (UML)
Shared Weights
Information Flow
Petch embd.
image Enizader
8x
gy
"Butterfly"
"eagle"
11
CLS/meen embe.
h
h
golden eagle"
Text Encoder
Token embos.
fx
fx
fy
(a) Self-Supervised Setting
(b) Supervised Setting
Figure 4: (Left) Inputs from different modalities (e.g., images or text) are tokenized into patch or token embeddings using pretrained encoders processed features; (Right) UMI. can be trained under two settings: (a) Self-supervised, where patch/token embeddings are passed through shared network and modality-specifff decoders perform next-token/patch prediction; (b) Supervised, where mean/CLS embeddings are fed through the shared classifier to predict labels within each modality.
samples, auxiliary modalities tighten estimates of shared structure. While prior work has explored shared weights in partially paired settings (Sturma et al., 2023), for related visual streams such as video and depth (Girdhar et al., 2022), or within already-aligned spaces like CLIP (Lin et al., 2023), we study and show their pronounced impact in the most general case of unpaired modalities with unaligned encoders.
Consider unpaired samples preprocessed features embeddings zx = fx (x) and zy (y) as shown in Figure 4. These embeddings are passed through shared network h, producing 7' h(=x) and ry h(2y). ). We consider two regimes to train these networks: (a) self-supervised learning for fully unpaired data (Figure 2(c)) and (b) supervised learning for unpaired data with labels shown in Figure 2(b). In the self-supervised setting, each modality has its own decoder (yx and gy) trained to reconstruct the input or predict the next token/patch depending on the form of embeddings zx and zy (refer Figure 4(a)). The training objective (Pseudocode in the appendix: Algorithm 1) is
Px and Py. each encoded into
LUML-SSL- EE (gx z) Ey-Py (gr y).
where é is the mean squared enror for continuous targets or cross cross-entropy for discrete tokens. In the supervised setting, when labels are available, ex for and cy for y, a shared classifier c(-) is trained on top of r'x and ry. Since both e(-) and h(.)) are shared across modalities, they can be viewed as single shared head h (refer Figure 4(b)). The supervised (Pseudocode the appendix: Algorithm 2)
LUML-Sup (xxex) lcE (ce(c(rx), ex) (y.cy) (cE(c(ry), ev).
In both scenarios, although supervision is codality-speeifee the shared head receives updates from both modalities. Consequently, gradients from also flow into fx, effectively transferring information from fy and thus y even without paired samples. At inference, we discard the auxiliary pathway and use r' as the representation for the target modality, training a linear probe top for downstream tasks. UML. also naturally extends to more than two modalities by incorporating additional modality-speciine encoder/decoder heads or classification layers.
EXPERIMENTAL RESULTS
In this section, we empirically evaluate UML across diverse benchmarks and configurations, leading to three main takeaways: (i) auxiliary modalities consistently boost target image representations in both self-supervised (Section 4.1) and supervised regimes (Section 4.2), with particularly strong gains in fine- grained and low-shot tasks; (ii) the benefits compound as we move from two to three modalities, with audio, vision, and text each contributing complementary signals (Section 4.2); and

---

Published as a conference paper at ICLR 2026
(iii) weight sharing across modalities generalizes beyond co-training, pretrained language model weights transfer effectively to vision (Section 4.3). Finally, we quantify a marginal rate of substitution between modalities, asking how many text samples equate to image (Section 4.4). While our benchmarks include samples without corresponding data in other modalities (e.g.. classes that appear only in images only in text) and thus touch the missing-modality regime (Dai et al., 2025; Kim & Kim, 2024), they are not constructed as large-scale mussing-mmdaale combining independent unimodal datasets. Our theoretical framework nonetheless subsumes this setting, as it does not require every class or instance to appear in every modality.
benchmarks formed by simply
4.1 UML IN SELE-SUPERVISED SETTING
Table I: UML (Ours) achieves higher top-I linear probe accuracy than unimodal baselines on learned representations, averaged over three seeds, across both MultiBench and vision-text benchmarks.
Datasel
MultiBench
Standard vision-text
MUSTARO
UR-FUNNY
Oxford Pess
MIMIC
MOSEI
UCF101
MOSI
D1D
Mothod
Unimodal
59.66
55.16
70.62
56.17
56.99
85.04
79.86
78.13
Ours 63.28
57.10
71.98
58.16
57.34
86.32
80.98
78.49
To study UML in the self-supervised setting, we use the real-world multimodal benchmark from MultiBench (Liang et al., 2021), which includes curated datasets such as text and images, with downstream labels covering a variety of domains, including healthcare, affective computing, and multimedia research and three standard classification benchmarks (Parkhi et al., 2012; Cimpoi et al., 2014; Soomro et al., 2012). We follow the same setting (dataset splitting, encoder architecture, pre-extracted features) as (Liang et al., 2023; 2021).
For training (refer to Figure 4(a)), we use linear encoders (fx and fy) to project each modality to a shared embedding space. Projected embeddings are then processed by a shared network (h): an autoregressive 5-layer 5-head transformer. Linear decoders (ex and gy) then project back to the original modality dimension. We train with a inference, we average the transformer output embeddings and use the resulting embedding for linear probing on the downstream classification tasks.
prediction objective. At
We report test accuracy using embeddings from the primary modality (image), and ensure fair comparisons, we perform rigorous hyperparameter tuning for all methods and repeat each experiment with three random seeds. For more details, refer Appendix B.3.1 and Appendix B. As shown in Table 1, UML significantly outperforms its unimodal counterpart across all benchmarks, particularly on MUSTARD where unique information from the text modality expresses sarcasm, such as ironic tone of voice.
4.2 UML IN SUPERVISED SETTING
We evaluate UML 9 widely-used visual classification benchmarks (Fei-Fei et al., 2004; Parkhi et al., 2012; Krause et al., 2013; Nilsback & Zisserman, 2008; Bossard et al., 2014; Xiao et al., 2010: Cimpoi et al., 2014; Soomro et al., 2012) in three settings: (1) Full fine-tuning: initializing from pretrained vision backbone and updating all parameters on the target dataset. (2) Few-shot linear probing: freezing the vision backbone and training linear classifier
& labeled samples per class (k 1,2,4, 8, 16). (3) Full-dataset linear probing: freezing the vision backbone and training a linear classifier on the entire training dataset, discussed in Appendix E. 1.2.2. In all cases, we enrich image representations with unpaired text embeddings, using ViT-S/14 DINOV2 as the vision encoder and OpenLLaMA-3B the frozen text encoder. To construct conceptually related yet unpaired text data, we generate text templates with varying amounts of semantic information about the dataset. For further details and specific prompts, refer to Appendix B.3. To train UML, we initialize the classifier with the average text embedding of each class, giving a strong prior for image-class alignment (refer to Appendix E.1 for ablation).
7

---

Published as a conference paper at ICLR 2026
Table UML (Ours) outperforms unimodal baseline on image classification with ViT-S/14 DINOv2 and La-33 in both settings: (i) Full finetuning and (ii) Few-shot linear probing (k = 1,2, 4).
Dataset
FOVC Aireraft
Oxford Flowers
Stanford Cars
Oxford Pets
Caltech101
SUN397
UCP101
Food101
Average
DTD
Shot
Method
Full finetuninnn
Unimodal
79.45
66.20
66.99
72.16
83.18
80.65
90.67
99.18
95,45
81.54
Ours
86,39
66.03
73.44
74.27
84.69
81.97
91.72
99.82
97.60 f
83,99
Few-shot Linear Probing
Unimodal
13.18
34.15
14.09
36.60
46.74
35.18
63.51
89.62
76.66
45.52
Ours
16.49
41.79
15.63 1
42.04
52.33
42.27
73.59
93.64
84.52 f
51.36
Unimodal
24.68
47.88 f
23.09
47.75
56.81
48.54
75.32
96.02
86.90
56.33
Ours
28.65
24.78
53.25
63.86
$4.44
97.63
90.55
Unimodal
38.76
57.51
32.10
59,69
67.75
60,79
98.59
93.48
Ours
43.17
60.89
33.86
62.43
71.13
63.88
87.36
99.17
94.96
68.53
Unpaired Textual Data Improves Visual Classification. As shown in Table 2, across both full fine-tuning and few-shot linear probing, UML consistently improves over unimodal baselines across all datasets, with the largest gains on fine-grained tasks (c.g.. Stanford Cars, FGVC Aircraft) where unpaired text sharpens class boundaries, and in low-shot regimes where textual cues help disambiguate visually similar classes. Additional results other shots, datasets, model scales, and prompt variants are reported in Appendix E.
We also evaluate the robustness of UMI--trained models under test-time distribution shifts. A 16- shot linear probe with DINOv2 is trained on ImageNet and tested across four distribution-shifeed target datasets: ImageNet-V2. ImageNet-Sketchl ImageNet-A, and ImageNet-R. UMI consistently outperforms the unimodal baseline (Figure 5), showing that language priors yield more transferable features. Additional robustness results are provided in Appendix E.3.
Finally, for all these settings, we also replace the independent vision (DINOv2) and text encoders (OpenLLaMA) with those of CLIP; since CLIP embeddings are already aligned, the gains from UML are even stronger (Appendix E.2.1, Appendix E.2.2, Appendix E.2.3.).
Additional Ablations. For all experiments, we keep the encoder frozen: Unfreezing the text encoder yields slightly weaker gains but still outperforms the unimodal baseline (Appendix E.12). Swapping in different text encoders yields consistent gains (Appendix E.6), while richer, more diverse captions provide especially strong boosts in few-shot settings (Appendix E.7). Further, training with semantically unrelated modalities (Appendix E.9) does not improve over the unimodal counterpart, confirming that gains stem from semantic correlations rather than spurious ones (Appendix E.9). Finally, unpaired multimodal data systematically widens inter-class margins and aligns modalities in weights (Appendix F).
Unimodal
Ours
70
61
38
25
52
180
60
37
24
+6.3%
23
59
36
22
68
58
35
21
imageNet (Source)
57
ImageNetv2
34
Imagefiet-Sletch
20
ImageNet-A
46
ImageNet-RR
Figure 5: Our approach UML much more robust than unimodal counterpart across four test time distribution shifted target test sets, All results averaged across three random seeds.
Unpaired Image and Textual Data Improve Audio classification. We extend UML ceaiiovvssiontter setting using the ImageNet-ESC benchmark (Lin et al., 2023), which links ImageNet objects and captions with ESC-50 environmental sounds. The benchmark has two
8

---

Published as a conference paper at ICLR 2026
versions: ImageNet-ESC-22 and SrnggeNet-ESC-19 For audio encoding, AudioCLIP with an ES- backbone (Guzhov et al.. 2021), while images and text are encoded by DINOv2 and OpenLLaMA-3B encoders. For further details, refer to Appendix B.3.
Unpaired image and text data consistently improve audio classification (Figure 6), with larger gains from CLIP's aligned encoders (Appendix E.13.3). Conversely, both audio and text also enhance image classification, showing that transfer works in both directions (Appendix E.13).
Finally, study the full three- modee-modality case, where treat two modalities as auxiliaries and as the target-for example, using both image and text as auxiliaries for audio classification. As shown in Appendix E. 10, performance improves monotonically with each added auxiliary modality, with the strongest gains achieved when all three modalities (audio, vision, and text) are used together.
Audio On
Ours (Audio Image)
Ours (Audio
Text)
52
+15.8%
58
16.0%
15 0%
135.39%
+2.4%
29
+3.0%
56
+2.4%
50
36
49
ImageNet-ESC-19
ImageNet-ESC-22
Figure 6: UML (Ours) improves audio classification using unpaired image and text samples on ImageNet-ESC-I 19 and ImageNet-ESC-27. All results are averaged across three random seeds.
4.3 TRANSFER LEARNING
Thus far, have explored how sharing model weights while co-training with multiple un- paired modalities improves the learned representation. But weight sharing need not be re- stricted to co-training: if modalities capture the same underlying latents, then pretrained weights from one should also serve as a useful initialization for another. We therefore study if transferring knowledge from by initializing the transformer layers of modality enhance performance in another ViT (Dosovitskiy al., 2020) with pretrained BERT (Devlin et al., 2019) weights and evaluat- ing on ImageNet (details in Appendix B.4.4).
20.0
+42.7%
2.59
Patch embeddings are learned from scratch through a linear layer, augmented by a CLS token and positional embeddings. As shown Figure initializing with BERT boosts perfor- mance for both frozen and unfrozen backbones. Our results indicate that the semantic knowl- edge of language models provides strong ini- tialization for vision compared to training from scratch.
Accurscy
58
15.0
57
12.5
5666
10.0
55
Scratch from BERT
Scratch from BERT
Figure 7: Image classifier trained from BERT initialization outperforms training from scratch for (left) full fine-tuning; (right) frozen backbone
4.4 MARGINAL RATE-OF-SUBSTITUTIO BETWEEN MODALITIES
Having established that unpaired modalities boost representation learning, robustness, and transfer, we now ask a more fundamental question: what is the relative value of each modality? both images and text provide views of the same semantic space, can we quantify their exchange now many words is an image worth? Figure 8 and Figure 9 visualize image-text conversion ratios using test accuracy isolines equivalent to an image for the same performance. Aligned CLIP encoder (1 image 228 words) is more efficient than non-aligned DINOv2 and Ope encoders (1 image 1034 words). Indced, in some cases, an image may quite literally be worth a thousand words. We also observe little additional benefit from adding more text beyond few samples. This is likely because
Oxford-Ped -Pets (Parkhi et al., 2012) dataset. These map the number of texts

---

Published as a conference paper at ICLR 2026
do not control for increasing complexity. adding sentences does not guarantee extra information. Further results on additional datasets and key details, refer to Appendix E.4.
0.44
045
Captions per class ('n (m-shot)
Captions per clsss (m-shot)
643
822
w
050
654
8.5
Sccuracy
050
0.65
647
672
0.50
6:11
0:59
643
68
6:8
667
Images per class (n-shot)
Images per class (n-shot)
Figure 8: img RJ 228 words (CLIP)
Figure 9: img ~0 1034 words (DINOv2)
CONCLUSIONS AND LIMITATIONS
Conclusions. We propose and investigate Unpaired Multimodal Representation Learning for hancing unimodal representations with unpaired multimodal data. Under linear assumptions, we theoretically show that unpaired data from multiple modalities strictly increases Fisher information along shared directions, resulting in a more accurate representation the underlying world. Mecha- nistically, UNPAIRED MULTIMODAL LEARNER (UML) achieves this by accumulating gradients from different modalities on shared weights, which can be viewed as an operational analogue of the Fisher information gain. Empirically, we show performance gains across vision, text and audio benchmarks, and estimate conversion ratios between modalities. UML provides a new perspective harness abundance unpaired data to learn better representations. This maybe especially useful in domains such medical imaging, scientific data, and robotics, which contain rich auxiliary modalities like text, audio, or metadata that are not often paired with every instance of the primary modality.
how to
Limitations and Future Work. While experiments study learning from unpaired multimodal data under both the self-supervised and supervised settings, downstream evaluations are conducted primarily for classification. A natural extension of UMI. is to combine separate dingle-modllle datasets to explicitly model missing modalities and partial overlap across modalities; this regime already subsumed by theoretical framework, and our experiments probe it in controlled settings, but a systematic empirical study remains an important direction for future work. Furthermore, we evaluate how multimodal data enhances image and audio classification; it remains show if they can, in turn, offer useful information for textual tasks. Finally, our results do not necessarily extend every possible setup of unpaired multimodal learning. In particular, do explicitly model or control gradient interference, modality collapse, or confiicing supervisory signals, which are known to induce optimization instability, negative transfer, and modality collapse paired multimodal systems. Systematically characterizing these phenomena in the unpaired setting remains an important direction for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was sponsored by the Department of the Air Force Artificial Intelligence Accelerator under Cooperative Agreement Number (NSF CCF-21 12665) and the Alexander von Humboldt Foundation. This work was also supported by a Packard Fellowship to P.I., and by ONR MURI grant N00014-22-1-1 -221-2740. S.G. is supported by the MathWorks Engineering Fellowship. S.S. is supported by an NSF GRFP fellowship. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Govermment purposes, notwithstanding any copyright notation herein.
in part by the NSF AI Institute TILOS

---

Published as a conference paper at ICLR 2026
REFERENCES
Amjad Almahairi, Sai Rajeshwar, Alessandro Sordoni, Philip Bachman, and Aaron Courville. Aug- mented cyclegan: Learning many-to-many mappings from unpaired data. In International confer- ence on machine learning, pp. 195-204. PMLR, 2018.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi- task masked autoencoders. In European Conference Computer Vision, pp. 348-367. Springer, 2022.
Roman Bachmann, Oğuzhan F Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir. 4m-21: An any-to-any vision model for tens of tasks and modalities. Advances in Neural Information Processing Systems, 37:61872-61911, 2024.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. moid-in discriminative compo- nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pp. 446-461. Springer, 2014.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an _obviously- perfect paper). arXiv preprint arXiv:1906. 01815, 2019.
Rakesh Chada, Zhaoheng Zheng, and Pradeep Natarajan. Momo: A shared encoder model for text, image and multi-modal representations. arXiv preprint arXiv:2304. 0.5523,, 2023.
Abhra Chaudhuri, Anjan Dutta, Tu Bui, and Serban Georgescu. A closer look at multimodal representation collapse. arXiv preprint arXiv:2505.22488; 2025.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos. Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Ruiting Dai, Chenxi Li, Yandong Yan, Lisi Mo, Ke Qin, and Tao He. Unbiased missing-modality multimodal learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 24507-24517, 2025.
Pinar Demetci, Rebecca Santorella, Björn Sandstede, William Stafford Noble, and Ritambhara Singh. Scot: Single-cell multi-omics alignment with optimal transport. Journal of Computational Biology, 29(1);3-18, 2022. doi: 10. 10.10899/mbbb2 2021.0446 URL https: //attss://oi.. http::///oooooo/.../////o 1089 cmb.
2021. 0446. 6. PMID: 35050714.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, and Hang Zhao. Improving multi-modal learning with uni-modal teachers. arXiv preprint arXiv:2106. 10059, 2021.
Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20029-20038, 2023.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178-178. IEFE, 2004.
12

---

Published as a conference paper at ICLR 2026
Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, 222(594-604)3309 368, 1922.
Ronald Aylmer Fisher. Theory of statistical estimation. In Mathematical proceedings of the Cam- bridge philosophical society, volume 22, pp. 700-725. Cambridge University Press, 1925.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better rision-language models with feature adapters. International Journal of Computer Vision, 132(2):581-595 2024.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autocncoders learn transferable representations. arXiv preprint arXiv:2205.14204 2022.
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens Van Der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16102-16112, 2022.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15180-15190, 2023a.
Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and
Ishan Misra. Omnimac: Single model masked pretraining on images and videos. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10406-10417, 2023b.
Andrey Guzhov, Federico Raue, Jömn Hees, and Andreas Dengel. Esresne (x) t-fbsp: Learning robust time-frrquency transformation of audio. In 2021 International Joint Conference on Neural Networks (IJCNN, pp. 1-8. IEEE, 2021.
Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis- Philippe Morency, et al. Ur-funny: A multimodal language dataset for understanding humor. arXiv preprint arXiv: 1904.9444400 06618, 2019.
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). Advances in Neural Information Processing Systems, 34:10944-10999 2021.
Minyoung Huh, Brian Cheung. Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024.
Adrián Javaloy, Maryam Meghdadi, and Isabel Valera. Mitigating modality collapse in multimodal vaes via impartial optimization. In International Conference on Machine Learning, pp. 9938-999999 198-9964. PMLR, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Mare Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, et al. Dinov2 meets text: A unified framework for image-and pixel-level vision- rision-language alignment. arXiv preprint ar 4rXh::2412.16334, 2024.
Donggeun Kim and Taesup Kim. Missing modality prediction for unpaired multimodal learning via joint embedding of unimodal models. In European Conference on Computer Vision, pp. 171-187. Springer, 2024.
13

---

Published as a conference paper at ICLR 2026
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In International Conference on Machine Learning, pp. 17283- 17300. PMLR, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fineggrained categorization. In Proceedings of the IEEE international conference computer vision workshops, pp. 554-561, 2013.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Phrase- based & neural unsupervised machine translation. arXiv preprint Rkkiv:8804.077555 2018.
Jae-Jun Lee and Sung Whan Yoon. Can modality model synergize training of other modality models? In The Thirteenth International Conference on Learning Representations, 2025. URL https: /eppeepeeeeeeeeeeeeeeeeeeeeeeeee net Forum?ide d
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learming, pp. 12888-12900. PMLR, 2022.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv: 2019.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language- image pre- pre-training via masking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23390-23400, 2023.
Paul Pu Liang Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems, 2021 (DB1)DBBB))) 2021.
Paul Pu Liang, Zihao Deng, Martin Q Ma, James Y Zou, Louis Philippe Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 06:32971-32998 2023.
Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: mroal few-shot learning with multimodal models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19325-19337, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. Advances in neural information processing systems, 30, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X1v:111..05001, 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019.
Xiaoyu Ma. Hao Chen, and Yongjian Deng. Improving multimodal learning balance and sufficiency through data remixing. ICML, abs/2506. 15500, 2025. URL https://api. semanticscholas
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515// 2013.
Mayug Maniparambil, Raiymbek Akshulakov, Yasser Abdelaziz Dahou Djilali, Mohamed El Amine Seddik, Sanath Narayan, Karttikeya Mangalam, and Noel E O'Conoor. Do vision and language encoders represent the world similarly? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14334, 2024.
14

---

Published as a conference paper at ICLR 2026
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint arXiv:2210.07183, 2022.
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. arXiv preprint arXi::2209 /5/62, 2022.
David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling. Advances in Neural Information Processing Systems, 36:58363-58408, 2023.
Luca Moschella, Valentino Maiorea, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. arXiv preprint Xrvi:22009 15430, 2022.
Maria-Elema Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference computer vision, graphics & image processing, pp. 722-729. IEFF, 2008.
Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodola, and Francesco Locatello. Asif: Coupled data turns unimodal models to multimodal without training. Advances in Neural Information Processing Systems. 13333-1335,,, 2023.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pp. 1498-3505. IEEE, 2012.
Sarah Pratt, lan Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? gener- ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15691-15701, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- 8748 8763. PmLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Karsten Roth, Jae Myung Kim, A Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15746-15757, 2023.
Shuvendu Roy, Franklin Ogidi, Ali Etemad, Elham Dolatabadi, and Arash Afkanpour. A shared encoder approach multimodal representation learning. arXiv preprint arXiv:2503.01654, 2025.
Jayoung Ryu, Charlotte Bunne, Luca Pinello, Aviv Regev, and Romain Lopez.. Cross-modality matching and prediction of perturbation responses with labeled gromov-wasserstein optimal transport. arXiv preprint arXiv:2405. 000388 2024.
Yuyang Shi, Valentin De Bortoli, Andrew Campbell, and Arnaud Doucet. Diffusion schrödinger bridge matching. Advances in Neural Information Processing Systems, 36:62183-6222 2023.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15638- 15650, 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0440 2012.
Siddharth Srivastava and Gaurav Sharma. Omnivec: Learning robust representations with modal sharing. 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1225- 1237, 2023. URL htw.s:tpp//... : /api. semante nt www.ooal...oroo/ooooeeoo htw.e/oooooooooooooooooooos/////
15

---

Published as a conference paper at ICLR 2026
Siddharth Srivastava and Gaurav Sharma. 0mnivec2 novel transformer based network for large scale multimodal and multitask learning. In CVPR, pp. 27402-27414, 2024. URL https:
wod.i..oggg...1 111.:/...111/////2122/3 2024 02588.
Nils Sturma, Chandler Squires, Mathias Drton, and Caroline Uhler. Unpaired multi multi-domain causal representation learning. Advances in Neural Information Processing Systems, 36:34465-34492. 2023.
Ya Sun, Sijie Mai, and Haifeng Hu. Learning to balance the learning rates between various modalities via adaptive tracking factor. IEEE Signal Processing Letters, 28:1650-1654, 2021. doi: 10.1109/
LSP:2021.3101422
Iya Sutskever. "an observation on generalization". YouTube video, Aug 2023, URL https:
// yout www/watchh. AKMuA_TV23.. Accessed: 2025-09-24.
Subash Timilsina, Sagar Shrestha, and Xiao Fu. Identifiable shared component analysis of unpaired multimodal mixtures. arXiv preprint arXiv:2409.19422 2024.
Chenyu Wang, Sharut Gupta, Xinyi Zhang, Sana Tonekaboni, Stefanie Jegelka, Tommi Jaakkola, and Caroline Uhler. An information criterion for controlled disentanglement of multimodal data. arXiv preprint MoXiv:2410.239000 2024.
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pp. 12695-12705, 2020.
Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof J Geras. Characterizing and overcoming the greedy nature learning in multi-modal deep neural networks. In International Conference on Machine Learning, pp. 24043-24055. PMLR, 2022.
Zhenbang Wu, Anant Dadu, Nicholas Tustison, Brian Avants, Mike Nalls, Jimeng Sun, and Faraz Faghri. Multimodal patient representation learning with missing modalities and labels. In The Twelfth International Conference Learning Representations, 2024.
Johnny Xi, Jana Osea, Zuheng Xu, and Jason S Hartford. Propensity score alignment of unpaired multimodal data. Advances in Neural Information Processing Systems, 37:141103-141123, 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and puttern recognition, pp. 3485-3492. IEEE, 2010.
Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang, and Di Hu. Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine- mee-grained learning. In ICASSP 2023 2023 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2023. doi: 10.1109/
ICASSP49357.20D) 2022. 20..5...2....00000 10096655.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal cor- pus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint 1r0vvv/606.0220 2016.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18123-18133 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-1 1986, 2023.
Xiaohui Zhang, Jachong Yoon, Mohit Bansal, and Huaxiu Yao. Multimodal representation learning by alternating unimodal adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 27456-27466, 2024.
16

---

Published as a conference paper at ICLR 2026
Yiyuan Zhang, Kaixiong Gong. Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. teta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802, 2023.
Zongmeng Zhang, Wengang Zhou, Jie Zhao, and Houqiang Li. Robust multimodal large language models against modality conflict. ICML, 2025.
Ying Zhou, Xuefeng Liang, Shiquan Zheng, Huijun Xuan, and Takatsune Kumada. Adaptive mask co-optimization for modal dependence in multimodal learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
17

---

Published as a conference paper at ICLR 2026
APPENDIX
A Further Related Works
20
B Supplementary Experimental Details and Assets Disclosure
21
B.1
Assets
21
B.2 Hardware and setup
................................
B.3 Datasets
21
B.3.1
MultiBench
................................
B.3.2
Image Classification Benchmarks
21
B.3.3
Constructing text templates
B.3.4
ImageNet-ESC Dataset
22
B.4
Training Protocol
22
B.4.1
Unpaired Multimodal Representation Learning under Self- Seff-Superviiion
22
B.4.2
Image Classification using Image and Unpaired Texts
23
B.4.3
Evaluation ImageNet-ESCC
24
B.4.4 Transfer Learning from Language to Vision
24
C Proofs of Theoretical Results
25
C.I Background and Definitions
25
C.2 Maximum Likelihood Estimators and Fisher Contributions
25
Theorems and Proofs
27
C.4 Connection to Sharing Parameters Across Modalities
31
D UML: Algorithm Pseudocode
32
E
Additional Experiments
34
E.1
Improving Image Classification using Unpaired Texts (Unaligned encoders)
34
E.1.1
Supervised Finetuning (across architectures)
34
E.1.2
Linear Probing (across architectures)
35
E.1.3
Few-shot Linear Probing (across architectures)
38
E.2
Improving Image Classification using Unpaired Texts (Aligned encoders)
40
E.2.1
Supervised Finetuning
40
E.2.2
Linear Probing
41
E.2.3
Few-shot linear Probing (across architectures)
E.3
Improving Visual Robustness Using Unpaired Texts
41
E,4
Marginal Rate-of-Subssitttion Between Modalities
43
E.4.1
Unaligned Encoders
43
E.4.2 Aligned Encoders (CLIP)
46
E.5
Impact of Scaling Vision Backbone
47
18

---

Published as a conference paper at ICLR 2026
E.6 Impact of Varying Text Encoders
47
E.7 Learning with Coarse-Grainge vs, Fine-Grained Textual Cues
48
E.8 Impact on Performance with Increasing Unpaired Text Prompts
48
E,9 Effect of Unrelated Auxiliary Modalities
49
E.
Extension to More Than Two Modalities
49
E.11 Effect of Ratio of Modality Batches
50
E.12 Effect of Freezing vs. Unfreezing the Text Encoder
51
E.13 Additional Experiments for Audio Visual Setting
E.13.1 Improving Image Classification with Unpaired Audio and Text (Unaligned
encoders)
51
E.13.2
Improving Image Classification with Unpaired Audio and Text (Aligned
encoders)
52
E.13.3
Improving Audio Classification with Unpaired Image and Text (Aligned encoders)
52
E.14 Gaussian Experiments
52
Analysis of the Learned Classifier
53
F.1 Change in Decision Boundaries with Unpaired Data from Another Modality
53
F22 What do models learn from unpaired data? .1
54
19

---

Published as a conference paper at ICLR 2026
A FURTHER RELATED WORKS
Unpaired Multimodal Learning. Unpaired data has long been used for image-to-iman (Zhu et al., 2017; Liu et al., 2017; Almahairi et al., 2018; Shi et al., 2023) and text-to-text translation (Lample et al., 2018) . More recently, several works have also proposed learning from unpaired data by inferring coarse- or fine-grained alignments through distribution matching or optimal transport objectives (Xi et al., 2024; Demetci et al., 2022; Ryu et al., 2024). In contrast, we leverage unpaired data for learning representations without the need for explicit inferred alignment. (Timilsina et al., 2024; Sturma et al., 2023) theoretically analyze the problem of identifying shared latent components and causal structures in unaligned multimodal mixtures. Most closely related to our work is (Lin et al., 2023), which leverages coarse- coarse-graines grained text data such as class names to improve image classification on CLIP using shared linear head. Another related line of works (Roth et al., 2023; Pratt et al., 2023; Menon & Vondrick, 2022; Gao et al., 2024) leverage prompting templates and pretrained LLMs to generate descriptive class captions, showing improved image classification performance with CLIP. Nonetheless, these methods operate on CLIP with pre-aligned representation spaces, whereas our approach also learns from unpaired data without assuming prior alignment. Several works have also proposed learning large multitask multimodal models with joint encoders and unified embedding spaces (Srivastava & Sharma, 2024; 2023; Zhang et al., 2023; Girdhar et al., 2022; Geng et al., 2022), often using joint training over separate tasks and/or masked prediction objectives. In a similar vein, (Chada et al., 2023) uses a stage-wise training strategy with both unpaired and paired data, and (Girdhar et al., 2023b) trains a single model across visual modalities. However, most of these methods rely on some amount of paired data for preliminary alignment and then leverage abundant modalityyspeciii unpaired data for further improvement. In contrast, our approach demonstrates that a model can implicitly learn mooss correlations from purely unpaired data, without requiring explicit alignment as a prerequisite.
Multimodal Representation Alignment. Our method relies on the notion of shared information and structure between unaligned modalities. Closely related to this are works demonstrating that unimodal representations trained without multimodal data are nevertheless converging. Huh et al. (2024) presents evidence that better-performing language models exhibit increased alignment to self-supervised vision models. Similarly, (Maniparambil et al., 2024) shows latent space alignment between vision and text encoders across backbones and training paradigms, and uses the CKA metric to connect unaligned encoders zero-shot. Earlier works also note alignment between models trained with different datasets and modalities (Moschella et al., 2022; Norelli et al., 2023). Several works have also shown that linear projection MLP is sufficient to stitch together the latent spaces of pretrained vision and language models (Merullo et al., 2022; Liu et al., 2023; Koh et al., 2023), Zhai al. (2022) extends this training a text encoder align to a frozen pretrained image model: this method was in turn used to integrate DINOv2, a large self-supervised vision model, with text encoder (Jose et al., 2024).
Modality Conflicts and Imbalance. A complementary line of work studies when joint multimodal training fails due to conflicts between modalities (Huang et al., 2021; Wang et al., 2020; Ma et al., 2025; Du et al., 2021; Xu et al., 2023; Fan et al., 2023; Sun et al., 2021; Zhang et al., 2024; Zhou et al., 2023; Wu et al., 2022). Here, modality competition typically refers to the phenomenon that one modality (often the easier or higher-SNR one) dominates optimization, causing the other modality to be under-utilized or even harmful to final performance. For example, Wang et al. (2020) shows that multimodal classifiers trained on paired inputs can underperform strong unimodal baselines because modalities converge and generalize at different rates, and propose gradient reweighting to address this. Fan et al. (2023) and related works explicitly treat modality imbalance as a competition for decision boundaries and introduce prototypical rebalancing to prevent a dominant modality from suppressing weaker ones. Ma et al. (2025) further attributes such conflicts to batch-level dynamics and proposes data remixing strategies to decouple and rebalance multimodal batches. All of these works focus on joint training with paired data (often training from scratch) and design mechanisms to mitigate conflict in that regime. A related line of work analyzes modality collapse, where representations degenerate to rely on a single modality, a phenomenon that can be especially severe under missing- modality patterns (Kim & Kim, 2024: Wu et al., 2024; Chaudhuri et al., 2025; Dai et al., 2025; Javaloy et al., 2022). These methods consider paired or partially paired training with systematically missing inputs, and design objectives that either correct the bias induced by absent modalities or predict
20

---

Published as a conference paper at ICLR 2026
the representation of the missing modality. By contrast, setting uses features from pretrained encoders or backbones initialized from pretrained models, together with fully unpaired data. this regime, do not observe the suboptimal modality competition that these works seek to mitigate.
B SUPPLEMENTARY EXPERIMENTAL DETAILS AND ASSETS DISCLOSURE
B.1 ASSETS
We do not introduce new data in the course of this work. Instead, we use publicly available, widely used image datasets for the purposes of benchmarking and comparison.
B.2 HARDWARE AND SETUP
Each experiment conducted NVIDIA Tesla V100 GPUs, each with 32GB of accelerator RAM. The CPUs used were Intel Xcon E5-2698 v4 processors with 20 cores and 384GB of RAM. All experiments were implemented using the PyTorch deep learning framework.
B.3 DATASETS
B.3.1 MULTIBENCH
We evaluate our approach on a diverse of multimodal fusion datasets from MultiBench (Liang et al., 2021) spanning healthcare, sentiment, and humor detection:
CMU-MOSEI (Moschella et al., 2022): The largest sentence-level multimodal sentiment and emotion dataset, containing 23,000 annotated monologue videos (over 65 hours of content from more than 1,000 speakers across 250 topics). Each video is labeled with sentiment intensity in the range [-3, 3). which we cast into binary positive/neeaatie sentiment classification.
CMU-MOSI (Zadeh al., 2016): A related multimodal sentiment dataset with 2,199 YouTube video clips, reflecting real-world opinionated speech. Sentiment intensities are annotated in the range [-3, 33, and we again formulate the task binary sentiment classifi- cation.
UR-FUNNY (Hasan et al., 2019): A large-scale humor detection dataset derived from over 16,000 TED talk videos. It includes 8.257 humorous punchlines identified by laughter markers, paired with 8,257 negative examples drawn from non-humorous contexts, forming balanced binary humor classification task.
MUSTARD (Castro et al., 2019): A multimodal sarcasm detection dataset with 690 video clips from TV shows. Each sample is annotated for the presence or absence of sarcasm, yielding challenging binary classification task.
MIMIC-II (Johnson al., 2016); A large-scale clinical dataset with records of over 40,000 ICU patients. combines time-series physiological measurements (recorded hourly over 24-hour window) with statie demographic and tabular features. We use it for binary classification of patients into group of ICD-9 codes.
B.3.2 IMAGE CLASSIFICATION BENCHMARKS
We evaluate on the following widely-used classification benchmarks: ImageNet (Deng et al., 2009), StanfordCars (Krause et al., 2013), UCF101 (Soomro et al., 2012), Caltech101 (Fei-Fei et al., 2004), Oxford Flowers (Nilsback & Zisserman, 2008), SUN397 (Xiao et al., 2010), DTD (Cimpoi et al., 2014), FGVCAircraft (Maji et al., 2013), OxfordPets (Parkhi al., 2012), and Food101 (Bossard et al., 2014). More details about the dataset and splits is provided in Table 3.
B.3.3 CONSTRUCTING TEXT TEMPLATES
To construct conceptually related yet unpaired text data, we generate text templates that capture varying granularities of information about the dataset. Our first approach (Vanilla) uses the
21

---

Published as a conference paper at ICLR 2026
Table 3: Detailed statistics of the 10 datasets for image classification.
Dataset
Classes
Train
Val
Test
Caltech 101 (Fei-Fei et al., 2004)
100
4,128
1,649
2,465
OxfordPets (Parkhi et al., 2012)
37
2,944
736
3,669
StanfordCars (Krause et al., 2013)
196
6,509
1,635
8,041
Oxford Flowers (Nilsback & Zisserman, 2008)
102
4,093
1,633
2,463
Food101 (Bossard et al., 2014)
101
50,500
20,200
30,300
FGVCAircraft (Maji et al., 2013)
100
3,334
3,333
3,333
SUN397 (Xiao et al., 2010)
397
15,880
3,970
19,850
DTD (Cimpoi al., 2014)
47
2,820
1,128
1,692
UCF101 (Soomro et al., 2012)
101
7,639
1,898
3,783
ImageNet (Deng et al., 2009)
1,000
1.28M
N/A
50,000
straightforward template Va photo of ()" with natural language label for each category, resulting in a basic text description for each class. However, this simple textual corpus lacks fine-grained information necessary distinguish between visually similar subcategories or to resolve contextually ambiguous terms. To address this, for the second template, we draw from the extensive literature on improving text prompts for zero-shot classification in CLIP (Gao al., 2024; Menon & Vondrick, 2022; Pratt et al., 2023; Roth et al., 2023). Specifically, for the second approach (GPT-3 Descriptions), we adopt the text prompt generation strategy developed by Pratt et al. (2023), using large language models such GPT-3 to generate diverse and contextually rich prompts for each image category. We use three generic hand-written sentences across the datasets:
Describe what a/the 0 looks like:
Describe a/the 0 What are the identifying characteristics of a/the ()?
The blank portion of each template is populated with the category name, along with the category type for specialized datasets (e.g., "pet" + + for Oxford Pets or "aircraft" { for FGVC Aircraft). The type specification important for disambiguating categories with multiple interpretations. Some examples of these descriptions are provided in Table for the Oxford Pets dataset.
B.3.4 IMAGENET-ESC DATASET
Experimental Setup. We extend our results beyond vision and language to an audiovisual- langiua dataset: the Ees benchmark (Lin et al., 2023). This benchmark combines ImageNet (1000 object categories) and ESC-50 (50 environmental sound classes) by matching classes that logically correspond. For example, the dog (barking) class from ESC-50 aligns with various dog breeds from ImageNet, while the clock-alarm sound maps both analog clock and digital clock. This alignment captures the relationship between visual objects, their sounds, and their textual descriptions. The benchmark consists two versions: 1) ImageNet-ESC-211 A broader set including loosely matched visual-audio pairs (e.g., drinking-sipping to water bottle); 2) ImageNet-ESC-11: A more precise subset containing only accurate visual-audio matches.
B.4 TRAINING PROTOCOL
B.4.1 UNPAIRED MULTIMODAL REPRESENTATION LEARNING UNDER SELF-SUPERVISION
MultiBench. For the MIMIC dataset, which contains tabular and medical time-series inputs, we train models directly on the raw modality inputs. For the four video datasets (MOSEI, MOSI, UR- FUNNY, and MUSTARD), we train models on standard pre-extracted features from text, video, and audio modalities (Liang et al., 2021). All models are trained for 100 epochs, with hyperparameter search over learning rates {10 10--0033000 3, 10 1104).. We report the best performance across learning rates, averaged over three random seeds. To train UML. each modality is first projected into shared embedding space of dimension d E {10, 40, 150, 300) via a learned linear transformation. The projected inputs are processed by shared 5-layer, 5-head autoregressive Transformer encoder,
22

---

Published as a conference paper at ICLR 2026
Table 4: Sample text descriptions per class for Oxford Pets dataset
Class
Examples
Wheaten Terrier
wheaten terrier a small, shaggy dog with a soft, silky coat.
A wheaten terrier has soft, wheat-colored coat that is low-shedding and hypoallergenic.
The wheaten terrier is a medium-sized, hypoallergenic dog breed.
A pet Wheaten Terrier usually has intelligent expression and soft, wheat-colored coat.
Great Pyrenees
A great pyrenees is large, white, shaggy-coate dog.
A Great Pyrenees is large, fluffy dog with calm, gentle disposition.
The great pyrenees was originally bred protect livestock from predators.
Great Pyrenees are known for being very large, white dogs with thick fur.
Sphynx
A pet Sphynx typically has small, wrinkled head and a hairless body.
Sphynx is hairless cat breed known for its soft, warm skin.
A Sphynx often displays large cars, pronounced checkbones, and no fur. Sphynx are unique cats characterized by their lack of coat and wrinkled skin.
Birman
A Birman is long-haired, color-pointed cat with a "mask" of darker fur on its face. A Birman has silky, pale cream ivory fur with deep seal- or lilac-colored points. Birman cats possess striking blue eyes and contrasting white "gloves" on their paws. They are known for being gentle, affectionate, and amooth-coater companions.
Pomeranian
Pomeranian a small, fluffy dog with a thick double coat.
Pomeranians are toy-sized, alert dogs with fox-like faces and plumed tails. A pet Pomeranian often comes in orange, black, white, or mixed coat colors.
They are lively, outgoing, and known for their bold, friendly personalities.
followed by a linear projection back to the original modality dimension. Training uses a next- token/patch embedding prediction objective, with both modalities sharing the Transformer backbone to encourage cross-modat synergies in the latent space. At inference time, we average the Transformer outputs across sequence length and use the resulting embeddings for linear probing on downstream classification tasks. All models (Unimodal baseline and UMI.) are trained for 100 epochs, with hyperparameter search over learning rates (10-2 10-3 10-1). ). For UML. in addition, we perform hyperparameter search over a curriculum parameter step E {0, 30, 50, 70). This parameter controls whether training begins on X alone for the first step epochs before switching joint training, with step ( corresponding to joint training from the start. For each dataset, we select the best-performing model on the validation set and report test accuracy averaged over three random seeds.
Standard Vixion-Text Benchmarks. We extract image and text embeddings using ViT-B/14 DINOv2 and OpenI.LaMA-3BB respectively. As in MultiBench, patch and token embeddings are projected to a shared 256-dimensional space via modality-specifft linear layers. A 4-layer, 4-head transformer serves as the shared encoder, with outputs projected back to the original embedding dimensions using modality
linear projections. We perform rigorous hyperparameter tuning for both the unimodal baseline and UML, and report average test accuracy of the best model across three seeds. Full hyperparameter ranges are listed in Table
B.4.2 IMAGE CLASSIHICATION USING IMAGE AND UNPAIRED TEXTS
For text, we use OpenLLaMA- as our default encoder and ablate against BERT-Large. RoBERTa- Large, GPT-2 I Large, and the pre-aligned CLIP text encoder, keeping the text encoder frozen. For images, main backbone is VIT-S/14 DINOv2, with ablations across other DINOy2 variants and the CLIP vision encoder. the linear- linear-probe setting, all encoder weights stay fixed and we train only single linear classification head; in full fine-tuning, jointly update the image backbone and that head, while still freezing the text encoder.
We optimize cross-entropy loss via AdamW (Loshchilov & Hutter, 2017) and perform an extensive grid search over learning rate, weight decay, cosine learning rate scheduling with linear warmup, dropout, and learnable, modality-specifie scaling on the logits. The results reported for the
23

---

Published as a conference paper at ICLR 2026
best-performing model on the validation dataset. We report results for the model achieving highest validation accuracy; the full hyperparameter ranges are in Table 5.
For full tine-tuning, we jointly update the image backbone and classification head with a fixed learning rate 5 x 10-5, batch size 64, and omit learnable modality-speciie scaling, since it showed no benefit in this setting.
Table 5: Hyperparameter grid for linear probing.
Hyperparameter
Values
Optimizer
adamw
Learning rate
{0.001, le-4)
Weight decay
{0.0, 0.01, 0.001)
LR scheduler
cosine
Batch size
{8. 32}
Max iterations
12,800
Warmup iterations
50
Warmup type
1 linear
Warmup min L.R
le-5
Dropout
(0.0)
Modality-specific learnable scaling
False, True)
Early-stop patience
10
B.4.3 EVALUATION ON IMAGENET-ESC
Similar to our vision-language experiments, we perform few-shot evaluation using the 5-fold splits defined in the benchmark. Each fold contains 8 samples per class, with one fold used for training and validation and the remaining four for testing. We repeat the process over random splits and report the average performance. For audio encoding, we use AudioCLIP with an ESS ES-ResNeXTT backbone (Guzhov et al., 2021). AudioCLIP is pretrained on AudioSet and generates audio em- beddings in the same representation space CLIP. Following the instructions in (Guzhov et al., 2021; Lin et al., 2023), we use train () mode in Pytorch to extract the features since eval () mode yields suboptimal embeddings. We evaluate
models two image classification glassfication-ccmmaiing the unimodal baseline against two multimodal variants in which the primary modality is each time augmented by one of the other modalities.
classification and
B.4.4 TRANSFER LEARNING FROM LANGUAGE TO VISION
To adapt language model image classification, we embed image patches using linear projection and add positional encodings to capture spatial structure. We then use transformer layers initialized from pretrained BERT, and finally, a 2-layer MLP classification head. Specifically, we split each image of size 224 x 224 into patches of size 16 x 16 with 196 patch tokens. Each patch is then projected into the model's embedding space of dimension d(e.g. d=768 for GPT-2,
1024 for BERT) via learned linear layer. We then prepend learnable " [CLS]" " token, add learned positional embeddings of shape (N + 1) X d, and apply dropout with probability p 0.1. This (N 11) ))x))))) d sequence is passed into the pretrained transformer stack (either GPT-2 or BERT). using full bidirectional attention mask over all patch tokens and the CLS token. We extract the final hidden state corresponding to the CLS token and feed it through a two-layer MLP classification head.
During training, we evaluate two scenarios: 1) one where the pretrained backbone frozen and only the patch embedding and linear head are trained, and 2) another where the backbone is initially frozen to align the trainable layers (patch embedding and head) with the pretrained language backbone, and then unfrozen after 2000 steps for end-to-end training. This approach allows to test whether the semantic richness captured by language models provides strong initialization, leading to better convergence and performance compared training ViT from scratch.
24

---

Published as a conference paper at ICLR 2026
C PROOFS OF THEORETICAL RESULTS
In this section, we present complete derivations and proofs of the main theoretical claims. Ap- pendix C.I gathers all definitions and background required for our arguments. Appendix C.2 formalizes the linear data-generating model, derives closed-fon form each modality and their joint estimator, and computes the corresponding block-wise Fisher informa- tion. Finally, Appendix C.3 provides the detailed proofs of our variance-reduccion claims, showing rigorously how unpaired multimodal estimation strictly lowers estimator variance.
Baaximum-lkkellhon estimators for
C.I BACKGROUND AND DEFINITIONS
In this section we revisit the mathematical definitions used in our theoretical analysis, including nodrrxsosd,,,ngs, characterization of symmetric matrices and Fisher information.
Definition 1 (Positive Semidefinite Matrix). A real symmetric matrix A e R2 is positive semidefi- nite if for all vectors v IR v Au 2 0. Equivalently, all cigenvalues of are nonnegative. We denote the set of all d x d symmetric, positive-semiddefiit matrices Szu
Definition 2 (Positive Definite Matrix). A real symmetric matrix A Rd is positive definite if for every nonzero v E IRd, v Av > 0. Equivalently, all eigenvalues of A are strictly positive. We denote the of all d x d symmetric, positive definite matrices as S
Definition 3 (Loewner Order). For two real symmetric matrices A, E Rdxd, we write A N
B < B positive semidefinite and < < a partial order on the cone of symmetric matrices.
B A is positive definite. This defines
Definition 4 (Fisher Information Matrix), Given a parametric family of densities p(1; data æ, the Fisher information matrix at parameter 0 is
I(0) Ez-p(=:) V log p 0) Vo p(:; 0);
Equivalently, for regular models, I(0) = - V pog
C.2 MAXIMUM LIKELIHOOD ESTIMATORS AND FISHER CONTRIBUTIONS
In this section revisit our linear data-generating model, introduce notations for the X-only, Y-only and joint likelihoods, derive the closed-form MLEs 0x, By and x,. and formalize their information contributions towards estimating the ground truth parameters
[0z, oz. By]
Data Generating Process. Recall our linear uata-generating process: Assume that all factors of variation in reality live single d-dimensional space Z = E gd modeled using a linear data-generating pipeline. This parameter can further be decomposed
0 = [0c, 0,, 0,] where 0 E R 0₂ E Rd= 6, E Idd, and de + d₂ + dy = d. Here, 6g captures the common (shared) parameters that affect both modalities, 0, denotes the parameters that only affect modality X, and 0g denotes the parameters that only affect modality Y. We observe two independent datasets, one from each modality { R and (Y1) E IR, each reflecting partial measurements of the ground truth latent space Z":
X, Ac, + Ar, O2 + cx, cx, ~ N(,0 Tm.
(3)
Y, Be By, 0, + exis EY. ~ N(0, Ing)
Here, Ac.i, Ae,i Bogi By are known design blocks capturing how each sample probes the latent factors and Ex,,Ey, represent the independent measurement noise.
In our linear setting, estimating the true latent state 6 and hence the underlying reality z*-is is governed by the Fisher information matrix I(0) -E V E(O)). which measures how sharply the likelihood "curves" around the true 6. High curvature along a particular axis means the data tightly constrain that component, driving down estimator variance there.
25

---

Published as a conference paper at ICLR 2026
Unimodal Estimators. We first estimate using only the Xatateset. Stacking {X:) yields design matrix A with block rows [Ac, Ag,s, OJ. The least-squares solution
No
xx arg min
Σ X.
Act Ag, 0z
8
omits fy entirely. Consequently, the Fisher information on Oy vanishes, making it unidentifiable.
Analogously, stacking {Yj)
Ny defines B with block rows [Ba.i O, Byys and yields
No E BBe 0. BBy 0y11²
Dy arg min
==1
This estimator doesn't depend on Oz. providing zero coverage for that component. Thus, each unimodal estimator entirely fails to recover the parameters exclusive the omitted modality.
Multimodal Estimators. Despite the lack of one-to-one pairing, both {X,} and (Y;} share common parameters 0z. Since the two distributions conditionally independent, the joint likelihood factorizes as
N₂ I
N
P(,, Go, (2)
X
11 1=1
p(y 0c,0,) 0y)
1=1
Maximizing this yields the combined estimator
N₂ |x - Ac,i 0. Az,i 0zl1 +
N₂
0x.y = arg min 0..0..0z
Σ
Σ j=1
YY Ba, 0. Byy 6y 0y1
Intuitively, there is no requirement to match up individual (X,, Yj) pairs. Instead, the estimate for Ag improved by both modalities while remaining unpaired.
Fisher Information. In our linear model, each dataset contributes alock-struetured Fisher information. For the datasset::
N₂
AT
Aci
AAA Az,
0 B
Ix
A
Ae,
0 0
and for the Y dataset:
Ny
B T B
0
BTB
Bud
ly
Σ
0
j=1
B
viJ
Bai
Byg
Because X and Y samples independent, their curvature contributions add pointwise, resulting in the joint Fisher information being simply the sum the unimodal blocks.
Σ, AT + E, BT Bes
Ix, = + +y
Σ, A
0
,
E, B Bus
where "*" denotes the cross-modal blocks. In particular, we have the
block as
N
N
(Ix.))_.0 =
B
Be.j
For clarity of exposition, we assume that the parameterisation is chosen such that the common factors 8_ are Fisher-otthogonal to the modality-speciiff factors 0₂ and 0, i.e. the cross Fisher blocks vanish. Under this assumption, the f block of the Fisher matrix inverts blockwise to yield the covariance on f..
26

---

Published as a conference paper at ICLR 2026
C.3 THEOREMS AND PROOFS
The aim of this section is to detail the proofs of the theoretical results presented in the main manuscript The key theoretical tools driving our analysis are already prepared in Appendix C.I and Appendix C.2.
Core to our theoretical analysis are a few lemmas around the Loewner-order monotonicity result for inverses that we prove below.
Lemma (Loewner Order reversal for inverses). Ler M,N E 8 with M < (or M N N). Then N-1 < M-1 (or N-1 3 M-1).
Proof. Since > 0, N/2 exists and is nonsingular. Define C N=2NN12 Because congruence with an invertible matrix preserves pesitive-definiteness, C > 0; hence C-1 is well defined and C-1 I (the scalar map x + x x= is strictly decreasing on (0, oo)). Undoing the congruence gives
1/2
M = NN N 12 N-1
Lemma 2 sowerse-monotonniity of the Moore-Penrose pseudoinverse). Let M, E s satisfy M < N and ker M = ker N =: K. Then their pseudoinverses obey Nt < Mt.
Proof. Set == K- and let P Ps be the orthogonal projector onto S. Because M and N vanish on K, we have the decompositions M PMP and N PNP. Restricted to S both matrices are positive-definit:
: PMP,
Ñ = PNP $40
M < N.
Apply Lemma to M, N to obtain N-1 M-1 on S. The Moore-Penrose pseudoinverse equals the ordinary inverse S and is zero on K:
M PMM Therefore N1 PN-1P
N PÑ-1P.
Mt.
Lemma 3 (Directional Loewner Order reversal), Let M, E 8 with M 3 N. If a non-zero vector v satisfies vJ Mu v. Nu, then
1. For the vector v, it holds that TT M- 2 vT N- with strict inequality ot M- > u N N- if and only (N MMM #0.
2. There exists a non-zero vector u E Rd such that aJ - u uu.
Proof. Denote the Loewner gap A == N - M 0. Then, the assumption v N > v. Me is
equivalent to eT Av > O. Introduce the congruence cenerueecc--ivariann invariant normalisation C: 1/2 A 1/2 = 0. Now, using A = MI/CCM//2 and properties of inverse,
N MI CCM
N M M-
since I + C > 0 (because C and > 0). Thus,
- N 1 - (I + (I+C)-1 M-
= =M M - + -
because (1 -1- (4-(+()-)[(+))--
(1) Directional inequality and strietness.
Evaluating in the direction v, have,
v (M-I-N- 1
1/2
=1((4t) (where u - -1/2
since (I+C)- E S-o and C E S2o- This proves that xJ 1 v 2 vt 1-1 strict if and only if ker(C). Since ker( + C kcr ker(C(I + -1-1) -kr(CC), this is equivalent to Cu o O. This can be further written as
v. The inequality is
Cu M 1/2
1/2
M 1/1/2
27

---

Published as a conference paper at ICLR 2026
As 1/2 is invertible, this holds if and only if AM- o O, which proves 1.
(2) Existence of a direction with strict reversal.
Now, from the premise v Av > it follows that + 0. Since M > O, 1 is invertible, C is also not the zero matrix. Since C ž this means that C' must have at least one strictly positive eigenvalue, Let À > 0 be such an eigenvalue, and let 2 + 0 be a corresponding eigenvector. Define,
x = M1 + 0. Thus, we have x (M N x = 2 C(I+
I+x |2|| > 0, showing
the existence of a non-zero vector such that x M 2 > x N-1x.
Lemma (Fisher information is positive semidefinite). Let {p(x; 0) 0 e c Rd] be a regular parametric family with score S(0) V log (X;;0); and Fisher information T(0) E ((((())(0)T]) Then v I(0)u 2 equivalently I(0) E _
Proof. For any E Rd
==))==
B[(u S(O)) 2 0.
Thus I(0) is positive semidefinite.
Theorem 1 (Restatement of Theorem 1). Let 0x,00 0y be the least-squares estimators for using only } and only {Yg} and let 0x, be the joint estimator using both unpaired datasets. Then, has full rank, the common-
under the assumption that at least one Be, where j E Var(0x,y) 9, x Var(0x) 8. or factor covariance satisfies the strict Loewner ordering i.e. Var
equivalently, the Fisher information Og strictly increases when combining both modalities, despite not having sample-wist pairing: + ly]o..0. > (Ix))0.9e.
Proof. Let Mx (Tx))..0.. My (Ty)o... 0...0.. and Mx,y (Tx,Y)o..0, be the Fisher information submatrices for the common parameters 0... From the definitions provided and Lemma 4, we have
and My -E, B Be By the theorem's assumption, at least one Be. has full column rank, then for any u E IR \ {0). v B Bug = || |ull > O. Now, since My sum of positive semidefinite matrices, least one of which is positive definite, My must be positive definite. Thus, My > 0.
Mx -; Ac,
Now since the joint Fisher information Mx,y Mx My and My > we have Mx, -x My > 0. This immediately proves the strict increase in Fisher information:
(Ix + lY)e..0. Mx,y > Mx (Ix))_.8.
In our Gaussian linear model, the beast-squares estimators coincide with the maximum likelihood estimators, and their asymptotic covariance on f, the inverse the Fisher information, up to the usual 1/n scaling, which is common to all three estimators and irrelevant for comparisons. Thus, whenever Mx is positive definite, Var(0x), )a_.. M and Var(fx,y) p_.0 - Since Mx, Mx 0, from Lemma 1, we have M Mx 4 MMx, i.c.
Var(Ox,x)
Var(Ox)e.e
If Mx is only positive semidefinite. interpret covariances via the Moore Penrose pseudoinverse. Along directions v E ker[Mx). the X-only estimator has infinite variance (no information), whereas Mx,y > 0 yields finite variance for the joint estimator. Hence adding the unpaired Y-modality strictly reduces the variance (or, equivalently, strictly increases the Fisher information) on the common factors 0..
Theorem 2 (Restatement of Theorem 2). Let all notation be in Theorem 1, and define Mx := (1x))_0. My := (IY) 0. and Mxy == Mx + My. Let v e Rd \ {0}. If there exists at least one index E {1, .,.yN such that Be + 0, then the following hold:
1. The Fisher information strictly increases in direction i.e. t Mxy v > o Mxv.
28

---

Published as a conference paper at ICLR 2026
2. The variance of the estimator in direction v is strictly reduced ie v Var x ) v Var(0x) 0.00 v, o & range(Mxx. For range(Mx) this strict inequality holds for under an additional invertibility condition and is always guaranteed for some u range(Ms, i.e 3u s.1. u. Var Var(Ox,y)e.. 0c.0e u (ar((x)g 90.80 8, u.
Proof. By assumption, 3j such that Be, + 0. Thus:
No
My Σ Bayull BB O.
j=1
Hence My is positive-definite in direction v, implying Mx,y > Mx in this direction, thus proving the first part of the theorem:
Mxy u Mx + T Myu > T Mxv,
For part (2), we break the analysis into two cases:
Case 1: v & (xx). Then decompose v = + UK, where Us E Range(MM and Ux kcr(Mxx with рк * O. The linear combination of parameters o 0. vg 0g U2 Since 2x ker(Mx), the component e 0 is not identifiable by the X X-only model. Consequently, the asymptotic variance of an unbiased estimator for v 0g using only the X -ddaaset is infinite. We denote this e Var(fx) 00.00. oc.
On the other hand, we already showed that n. "''Mxx'' > O, so & ker(Mxy). For symmetric positive semidefinite matrix Mxy, its Moore-Penrose pseudoinverse M is also symmetric positive semidefinite and satisfies ker(M = ker ker(Mxy). Hence v & ker(M and
< v M < co.
Since the asymptotic covariance of 0x,y y in direction e is governed by M we obtain
0 u (ar((xxy)) 0. <0= = T Var(0x)e
showing strict variance reduction along v this case.
Case 2: u E Rango ). Let Rango (Mx) and let Ps be the orthogonal projector onto Because Mx = Mx Ps and Mxy Mx + My, the restrictions
Mx := PsMxPs.
Mxy == Ps Ps = Mx + PsMyPs
are positive-defintt on S; To see this, take any non-zero w S. Since w range(M MM)), Psw = w; hence
w Mx = w Mxw > 0 (Ps is identity when restricted to S) Thus Mx > Because Ps Ps 2 0. adding preserves positive definitenessi,,sss
Mxy = Mx + Ps z Mx > on S.
Applying Lemma 3(1) Mx and Mxx on S gives us T JTM S u MM Striet inequality
v M Myre < T Mxx' holds if and only the condition C₂ == ((Mxy - Mx M o 0) is met. Therefore, if condition C, holds, the directional variance along this constrained space is strictly reduced:
w Var((xx.))
M < M
Var
Further, from Lemma 3(2), there exists some vector E S such that u M M u M u. Thus we have,
Te <TVar(0xx)) ). TVar(0x)o_...
Thus, completing the proof.
We note that true asymptotic variance defined v Var(Ox,y)a..e. o MMMYy w. My M-1 if S is an invariant subspace of Mxy and MxY is block-diagonal with respect to S and S (i.e,, Mx P&s which implies My Ps1
29

---

Published as a conference paper at ICLR 2026
Corollary 1. Assume direction E Rd (0) with v ([x))oo > and b v (ly 0, 0. 0 where is the common eigenvector of (1x)e,,0 and (Ly 0..0. Then the variance in direction v contracts by the factor
v Var(Ox.x)
1/(a +b)
a+b 1,
V Var(Ox)
1/a
So the joint estimator achieves strictly lower error along v.
Proof. Let Mx (Ix)e,,0. and My (Iy)e,,e) 0e..0. By assumption, is common eigenvector of Mx and My. Thus, Mxx = Axu and My = Xy for some eigenvalues Ax and λy. From the assumptions, have Ax a/ w/ > and Xy b/|lu||2 > O. Since Mx is symmetric and Mxu Ax with Ax > the pseudoinverse acts M Xx Therefore, the variance in direction for the X-only estimator is
v Var((xx) v u
X ||ul| a "|ol|
Since is common cigenvector, it is also eigenvector of Mxy Mxx My:
(Mx + My)c = Mx + My Axe Ayu (Ax + Ay
The corresponding eigenvalue is Ax = Ax + Ay Since Ax > and xy > 0, Axy > 0. Thus, (Mx + u (Ax +
The variance in direction for the joint estimator is
Var(Oxx
VT (Mx + My) = (Ax + xx) Xy) ||ol = (a + b) "|u|||
Now, we form the ratio of these variances:
T Var(0xx Var(Ox)ooee
Ax a
Ax + Ay a a+ <1
Assume a direction w E Rd \ [0} with v Then v Var(Ox) oo and v. Var(0x,y) 0o i.e. direction unidentifiable from X alone becomes well pell-posed with even umpaired data from Y.
Corollary 2.
and v TT (Iv)o v >
Proof. This corollary follows directly from Case of Theorem 2. The condition 8. (1xx),,, v for * 0 implies e E ker( )o.,0)) and thus & range( ([xx )o....). Given the additional condition O, the conclusions of Case 1 of the theorem apply directly.
T (Iy ).
Corollary 3 (Variance Reduction for Eigenvectors of Mx). Let E RRd \ (0} be an eigenvector of Mx =
=xx=(xx).... (Ix)e.,.. with a corresponding eigenvalue Ax > 0. If the -dtaaset provides information in this direction e (i.e., vT My > O, where My = (Iy)).,0.). then the variance in direction is strictly reduced by incorporating the Y dataset::
v. T Warr(fxyy)ar 2 < w (ar((xx)xx.. v.
Specifically, o [Var(xx)__ Xx 1|013
1011
Proof. Let Mx = (Ix)a )a... and My = (Ty),,, Since u is an eigenvector of Mx with a positive eigenvalue Ax > follows that v E Range(Mxx) Let Range(Mxx. The variance using only the X -dattset in direction v is given by
(Var(x)),.oo 00,0000000 M
Because v an eigenvector of Mx with xx > 0, Ma = Ax n. Thus,
Var v. (Nx'v) >x'1111?
This scenario falls under Case of Theorem 2, specifically its conclusion regarding S. According to that theorem, strict variance reduction Var(Ox,y)o, < T Var(Ox)oo_0 occurs if the
regaduionn
30

---

Published as a conference paper at ICLR 2026
condition C. ((PsMy MM) holds. Here, Ps is the orthogonal projector onto S, and Mxls is the restriction of Mx to S, so (Mxls) Xx V.
The condition Cg thus becomes (PsMy (A u) o O. Since Ax O, this is equivalent to Ps Psu * O. We are given that v My 0. u S. Psu v. Therefore, v My V PsMMy Psv > 0. Let As Ps Ps restricted to As is a positive semidefinite operator on S.. The condition T Agg > 0 for S, v + 0 implies that Agu # (because if Agt = then T AAu = 0, which contradicts Ag > 0). Thus, My Psu + O, which means the condition C, satisfied.
Since V E S and the condition C, for strict inequality is met, by Theorem 2, it follows that I Var(Ox,y))o.k < v Var(Ox))..e v.
Theorem 3 (Restatement of Theorem 3). Define for any I E Bi Be.j If frangee ((m) g rango(1x")) then there exists a nonzero v E Rd such that (m)
- Ar, and I\mh)
mm) 0 ( v.
Proof. Let Rx range(1X ((m)) Ry == range((I"))) By the assumption Ry & Rx, choose vector Ry Rx. Since Rd. is finite dimensional inner product space and Rx its finite dimensional subspace, we can decompose w l|| + with W| E Rx and E Rx. Because w & Rx, the orthogonal component e is non-zero.
(i) Term from (m) From the Fundamental Theorem of Linear Algebra, for any symmetric matrix S, kor S = range(S) (S) L; hence R = kcr ((m) Thus, f(m)) = 0.
(ii) Term from I/m) Because contradiction, that 1 v O. Then v E ker 1 Rr, so 1 w. But w v== [i] + 2)
Ry rango(Ty (T(m)), there exists with T(m)).. u. Suppose, for (mm)
u ||/|| = ||aa|2 > 0 because e 1. Wp while + 0. This contradicts v w; therefore ((T)) + 0 and, by positive semidefiniteness,
oJ Fmm > 0.
Combining the above inequalities yields v (m) > o (T( 0, with v * 0, which is the desired inequality.
C.4 CONNECTION TO SHARING PARAMETERS ACROSS MODALITIES
The results Section 3.1 establish that, in the linear model, the two modalities contribute additively to the Fisher information associated with the shared latent block A... This additivity arises solely from the structural fact that both observation channels depend on a common set of shared latents. To connect this to UML, consider likelihood model in which the parameters are partitioned into shared block & and moodality-specifii blocks, and the two modalities are conditionally independent given these parameters. Proposition shows that, in any such model, the Fisher information on Φe is exactly the sum of the unimodal Fisher blocks. Thus, the shared trunk in UML implements the same structural condition: the parameters dee are common to both losses, and therefore receive curvature contributions from both modalities. In this sense, weight sharing in UMI. is the architectural mechanism that reproduces the shared- -aramm-trreeter structure responsible for Fisher-information addition in our linearized theory. Below, we now formally state this result.
Proposition 1 (Fisher Information Additivity for Shared Parameters). Let de E Iede be a shared parameter block and let de, By be modality-speciipy parameters. Suppose
p(z,y) y des da, Φy) = -x( des (n) (y
des Un);
(5)
so that X and Y are independent given (de, Pe. py). Let (x,NN, and (Y, N be i.i.d. samples from the induced marginals. Denote by (x (be)) Ey (dbe) and f(oe) the log- likelihoods in for the X X-only,
31

---

Published as a conference paper at ICLR 2026
Yonty and joint experiments, respectively, and by Ix (x()),, 1y (6c) and (x,) the corresponding Fisher information matrices for dc. Then, under standard regularity conditions,
(oc) Ix((e) + Iy (rc))..
(6)
In particular, for any u E Rd {(0}.
v Ix,y (Ōe)u = v Ix( + T (6
(7)
Proof. For fixed Φr, oy the log-likelihood contributions are
Lx (x( =
logpx (X, pc: 0a),
E=
Ny
bv (r(()) =
Σ j=1
log (Y, 0y),
and ((o.) (x(e)) (dr) Let
gx (xx) vx.ex (o( fo
g( = vo.(()) gx(()) + gy (Ø:)
be the corresponding score functions. The Fisher information for de in the joint experiment is
(x.) E[s().)o())')
where the expectation is taken under the joint law of { and (rg) Since gx depends only on (X,) and gy only on {Yj), conditional independence of X and Y implies that 9x and gy independent random vectors. Moreover, under standard regularity assumptions, each score has zero mean, [[gx (6.) Elgy ( 0. Hence
(x.r(0.) [[[xx + gr (9x + +gr
= <[[xxgX] + Elgrgy| + Elgxgyl + [[grgk]
-E[gxgX] + Blgrgř]
[x(x)) + (bb),
which is the desired identity. The directional statement follows by left- and right-multiplinng with V.
Remark 1 (Connection to Section 3.1 and UML). The factorization
p(z, y de, o2, ④y) Px (a de, Φ2) pY (y Pc.
the standard conditional-indepeneeer parameters, the two observation channels are driven by independent noise. This is exactly the structure of the linear-Gaussian model in Section 3.1 well. There, 0g plays the role of &e- least-squares estimators coincide with maximum likellhood estimators, and their covariance is asymptotically given by the inverse Fisher information, recovering Theorems parameter structure by employing common trunk (parameters 6c) updated from loss of the form L= Lx + Ly, so that unpaired modalities contribute additively to the curvature experienced by the shared block.
assumption that, given the shared and modality-specific
3. UML mirrors this shared-
D UML: ALGORITHM PSEUDOCODE
In this section we present the full pseudocode for UML for both the self-supervised and supervised settings as shown in Algorithm 1 and Algorithm 2 respectively.
32

---

Published as a conference paper at ICLR 2026
Algorithm 1 Pytorch Pseudocode for UML in the self-supervised setting
f_img, f_text: image encoder, text encoder
g_img, g_text: image decoder, text decoder
h: shared backbone
while not converged: # training lcep
x_img fetch_next (image_loader) # image minibatch
x_text fetch_next
# text minibatch (random/ung 1 igned)
emoed (x img) # input image patch embeddings tete tokenize and embed (x text) input text taken embeddings
img = patchity end_e
2_img = f (x_ng()_img) image patch embeddings
2_text = f_text (x_text) text teken embeddings
y_img (__nng(h(z_mgg
predict image patch embeddings y_text g_text (h (h(2teext) predict text patch embeddings
Next Token/Patch Embedding Prediction Loss loss img MSE (y (y_imyggiigaao :-, x_imgl;,;;; loss text MSE ttyttetllee. :-,,;l, 1;, loss img 1ambda * loss_text # total 1oss
loss backward back-propagate update (h, f ing, t_xext g img, g_text) SGD update
Define Mean Squared Error loss def MSE (pred, target): return (preed target) ** 2) ()an
Algorithm 2 Pytorch Pseudocode for UML in the supervised setting
f img: image encoder (frozen er trainable)
f text:: text encoder (frozen)
is trainable: True f_img trainable else False h: classification hoad
while not converged: # training lcep
x_img fetch_nextt (image_loader # image minibatch
x_text fetch_next (text_loader) # text minibatch (random/unaligneel
z_img = s iinnglx nogix_ime img) # image enbeddings z_text f_text text) text embeddings
Logits img h(2 higgimg) predict image labels logits_text h(a_eext) predict text labels
loss_img (logits_img, labels_img) image classification loss loss text CE (logitss_tei text, labels labels_text) # text classification loss loss lcss_img lambda * loss_text # total loss
loss backward()))) back-propagate update (h, fmigg) if is_trainable else update (h) # SGD update
Define Cross-Entropy loss def CE (logits, labels) return -sum (labels 1og_ (logits, dim=l)) / 1en (1abels)
33

---

Published as a conference paper at ICLR 2026
E ADDITIONAL EXPERIMENTS
E.1 IMPROVING IMAGE CLASSIFICATION USING UNPAIRED TEXTS (UNALIGNED ENCODERS)
In this section we report image-classfficttion results on benchmarks (see Appendix B.3), covering three settings:
1. Full-dataset fine-tuning: train both the vision backbone and classification head (Ap-
pendix E.1.1).
2. Full-dataset linear probe: train only the classification head (Appendix E. E.1.2).
3. Few-shot linear probe: train only the classification head under few-shot conditions (Ap- pendix E.1.3).
In each setting, we compare UML with baselines across all datasets and multiple DINO-initialized vision backbones. Our method has two variants: Ours (UML), where we alternately train with both image and unpaired text data (see Algorithm 2), and Ours (init) where initialize the classifier with the average text embedding of each class, providing a strong prior to align image and class level information.
E.1.1 SUPERVISED FINETUNING (ACROSS ARCHITECTURES)
In this section, we fine-tune both the vision backbone and the linear classifier on ten downstream tasks, comparing UML against strong image-only baselines. evaluate four DINO-initialized backbones:
ViT-B/16 in Table
VIT-B/8 Table 7
DINOv2 ViT-S/14 in Table
DINOv2 ViT-B/14 in Table 9
Results for DINOv2 VIT-L/14 omitted due computational constraints. Across all backbones, UML consistently improves over the image-only baseline by leveraging unpaired text embeddings. For some backbones such as DINOv2 VIT-B/16, our head- read-initialization variant (Ours (init)) outperforms training using unpaired multimodal data from scratch (Ours), while in others it does not.
Table 6: Full finetuning classification with ViT-B/16 DINO and our proposed approach with the image-only baseline when fine-tuning on the target dataset. All vision encoders are initialized from DINO weights, and our approach leverages unpaired text data using Ope3LLaMA-3B embeddings.
We compare
Dataset
FGVC Aircraft
Oxford Flowers
Stanford Cars
Oxford Pets
Caltech 101
SUN397
UCFI01
Food101
Average
DID
Method
Unimodal
78.41
63.99
62.12
74.17
81.43
82.38
92.00
98.24
96.31
81.01
Ours
82.56
67.04
67.38
76.42
84.06
81.79
93.20
98.98
97.04
83.16
Ours (init)
81.95
67.12
68.29
73.84
84.31
81.12
92.60
98.73
96.84
82.76
34

---

Published as a conference paper at ICLR 2026
Table Full finetuning on classification with VIT-B/8 DINO and OpenLLaMA-3B. We compare our proposed approach with the image-only baseline when fine-tuning the target dataset. All vision encoders are initialized from DINO weights, and our approach leverages unpaired text data using Ope3LL embeddings.
Dataset
FGVC Aireraft
Oxford Flowers
Stanford Cars
Oxford Pets
Caltech 96.82
SUNEY
UCF101
Food101
Average
DTD
Method
Unimodal
85.67
68.04
72.60
76.65
83.94
85.32
93.06
99.22
84.59
Ours 87.95
70.28
75.31
77.19
85.59
84.83
93.05
99.43
97.12
85.64
Ours (init) 87.44
70.03
76.09
76.24
86.49
84.71
93.81
99.27
97.16
85.69
Table 8: Full finetuning on classification with VIT-S/14 DINOv2 and OpenLLaMA-33. We
compare proposed approach with the image-only baseline when fine-tuning the target dataset. All vision encoders are initialized from DINOv2 weights, and our approach leverages unpaired text data using OpenLLaMA-33 embeddings.
Dataset
FGVC Aircraft
Oxford Flowers
Stanford Cars
Oxford Pets
Caltech101
SUN397
UCF101
Food101
Average
DID
Method
Unimodal
79.45
66.20
66.99
72.16
83.18
80.65
90.67
99.18
95.45
81.54
Ours 84.87
66.72
71.54
74.14
84.77
81.16
91.87
99.55
97.03
83.52
Ours (init)
86.39
66.03
73.44
74.27
84.69
81.97
91.72
99.82
97.60
83.99
Table 9: Full finetuning on classification with VIT-B/14 DINOv2 and OpenLLaMA-33. We
compare our proposed approach with the image-only baseline when fine-tuning on the target dataset. All vision encoders are initialized from DINOv2 weights, and our approach leverages unpaired text data using Oaen.LaMMA33 embeddings.
Dataset
FGVC Aircraft
Oxford Flowers
Stanford Cars
Oxford Pets
Caltech10 101
SUN397
UCF101
Food101
Average
DTD
Method
Unimodal
89.62
71.45
77.29
73.88
88.00
82.94
94.55
99.88
97.69
86.14
Ours
90.93
70.97
80.02
75.83
87.52
86.25
94.74
99.88
97.57
87.08
Ours (init)
90.73
70.92
80.23
75.87
87.60
83.43
94.47
99.80
97.93
86.77
E. LINEAR PROBING (ACROSS ARCHITECTURES)
In this section, train only the linear classifier, top of the frozen vision and language backbone, on ten downstream tasks, comparing UML against strong image-only baselines. We evaluate five DINO-initialized backbones:
35

---

Published as a conference paper at ICLR 2026
ViT-B/16 in Table 10
ViT-B/8 in Table 11
DINOv2 ViT-S/14 in Table
DINOv2 ViT-B/14 in Table 13
DINOV2 ViT-L/14 in Table
Across all backbones, UML consistently improves over the image-only baseline by leveraging unpaired text embeddings. For backbones, our head-initialization variant (Ours (init)) outperforms training using unpaired multimodal data from scratch (Ours).
Table 10: Full linear probing classification with VIT-B/16 DINO and OpenLLaMA-3B We compare our proposed approach with the image-only baseline when training a linear probe on the target dataset. All vision encoders are initialized from DINO weights, and our approach leverages unpaired text data using OpenLLaMA-33 embeddings.
Dataset
FGVC Aireraft
Oxford Flowers
Stanford Cars
Oxford Pets
Caltech 95.01
SUN397
UCF101
Food101
Average
DTD
Method
Unimodal
67.10
64.63
56.02
72.42
81.27
74.96
93.07
98.32
78.08
Ours 68.71
65.14
57.42
72.95
82.06
75.30
93.18
98.46
96.19
78.82
Ours (init)
68.60
65.59
57.98
73.11
82.40
75.73
93.62
98.42
96.35
79.09
Table 11: Full linear probing on classification with VIT-B/8 DINO and OpenLLaMA-3B. We compare our proposed approach with the image-only baseline when training a linear probe on the target dataset. All vision encoders are initialized from DINO weights, and our approach leverages unpaired text data using OpenLLaMM-331 embeddings.
Dataset
FGVC Aireraft
Oxford Flowers
Stanford Cars
Oxford Pets
Caltech 96.23
SUN391
UCF101
Food101
Average
DTD
Method
Unimodal
72.01
67.19
62.02
76.18
82.95
78.57
91.99
98.78
80.66
Ours
72.93
68.17
63.49
77.13
83.16
79.87
92.59
98.50
96.47
81.37
Ours (init)
72.81
68.36
64.09
76.48
83.72
80.01
92.50
98.74
96.43
81.46
36

---

Published as a conference paper at ICLR 2026
Table 12:
Full linear probing on classification with VIT-S/14 DINOv2 and OpenLLaMA-3B. We compare our proposed approach with the image-only bascline when training a linear probe on the target dataset. All vision encoders are initialized from DINOv2 weights, and our approach leverages unpaired text data using OpenLLaMA-3B embeddings.
Dataset
FGVC Aircraft
Oxford Flowcrs
Stanford Cars
Oxford Pets
Caltech 97.00
SUN397
UCF101
Food101
Average
DTD
Method
Unimodal
77.48
70.72
66.28
78.25
82.64
84.39
94.29
99.62
83.40
Ours 78.45
71.53
67.33
78.70
83.51
84.67
94.70
99.82
97.11
83.98
Ours (init)
78.58
72.24
67.50
79.51
83.57
84.74
94.78
99.89
97.15
84.22
Table 13: Full linear probing classification with ViT-B/14 DINOv2 and OpenLLaMM--3. We compare our proposed approach with the image-only baseline when training a linear probe on the target dataset. All vision encoders initialized from DINOv2 weights, and our approach leverages unpaired text data using
embeddings.
Dataset
FGVC Aireraft
Oxford Flowcrs
Stanford Cars
Oxford Pets
Caltech
SUN391
UCFI01
Food101
Average
DTD
Method
Unimodal
85,46
75.42
72.34
79.73
87.26
88.70
95.56
99.76
97.81
86.89
Ours
85,40
75.22
75.22
80.73
87.21
89.02
95.83
99.88
97.85
87.37
Ours (init)
85.74
75.70
74.17
81.32
87.26
88.78
95.78
99.88
97.93
87.40
Table 14: Full linear probing on classification with ViT-L/14 DINOY2 and OpenL.LaM--3B. We
compare our proposed approach with the image-only baseline when training a linear probe on the target dataset. All vision encoders are initialized from DINOv2 weights, and our approach leverages unpaired text data using OpenL.LaMA-3B embeddings.
Dataset
FGVC Aircraft
Oxford Flowers
Stanford Cars
Oxford Pets
Caltech101
SUN397
UCF101
Food101
Average
DTD
Method
Unimodal
88.16
77.26
74.32
81.56
89.82
90.95
96.27
99.84
97.97
88.46 88.97
Ours
88.45
77.20
76.93
82.39
90.19
91.09
96.51
99.92
98.01
Ours (init)
87.99
77.75
77.20
82.51
90.17
91.29
96.32
99.92
97.93
89.01
37

---

Published as a conference paper at ICLR 2026
E.1.3 FEW-SHOT LINEAR PROBING (ACROSS ARCHITECTURES
In this section, we train only the linear classifier, on top the frozen vision and language backbone, for few-shot classification ten downstream tasks, comparing UML against strong image-only baselines. We evaluate five DINO-initialized backbones: ViT-B/16 in Table 16, ViT-B/8 in Table 15, DINOv2 ViT-S/14 in Table 17, DINOv2 ViT-B/14 in Table 19, DINOv2 ViT-L/14 in Table 19. Across all backbones, UML consistently improves over the image-only baseline by leveraging unpaired text embeddings. For all backbones, our head-initiee sead-initialization variant (Ours (init)) outperforms training using unpaired multimodal data from scratch (Ours).
Table 15: Linear evaluation of frozen features on fine-grained benchmarks for few-shot learning. We compare proposed approach with the image-only baseline by training linear classifier on top of frozen VIT-B/8 DINO features. Our method leverages unpaired text data using OpenLLaMA-3B
Datasel
Oxford Flowers
Stanford Cars
Feve Airenft
Oxford Pots
Calteeb
Sun397
Ucf101
Imagenet
Avarage
Did
Train Shot
Method
Unimodal
7.40
26.37
12.16
28.62
39.75
19.23
42.81
54.97
58.22
74.13
36.37
Ours
7.71
28.01
13.56
33.22
42.08
21.13
43.27
55.85
58.61
77.51
38.10
Ours (init)
9.24
34.23
14.49
36.27
47.55
24.81
46.75
60.09
61.59
80.23
41.52
Unimodal
14.43
37.96
20.28
79.80
53.03
30.62
$4.75
68.12
77.59
81.91
47.85
Ours
15.71
40.74
43.
58
33.52
54.49
86
18
84.52
49.67
Ours (init)
16.94
45.16
22.17
45.43
59.02
35.89
56.78
71.57
77.94
86.06
51.70
Unimodal
25.67
49.23
29.39
52.52
64,27
43.82
61.64
75.85
87.41
90.36
$8.02
Ours
27.30
51.23
54.31
66.72
45.58
77.51
87.96
91.36
Ours (init)
28.54
53.68
31.31
56.13
67.47
47.40
62.84
79.10
88.29
91.98
60.67
Unimodal
41.04
56.86
40.03
61.15
72.39
54.47
66.10
82,30
93.95
92.28
66.06
Ours
43.76
42.56
73.13
56.30
67.46
Ours (init)
44.16
59.80
42.30
64.46
74.30
57.07
67.18
84,85
94.00
93.24
68.14
16
Unimodal
57.72
61.74
52.63
67.69
76.18
62.63
68.87
87.31
96.41
94.27
72.54
63.21
54.53
78.13
63.74
69
87.73
94.54
73.76
Ours (init)
60,36
64.26
54.81
70.27
78.76
64.13
70.05
88.23
96.63
94.73
74.22
Table 16: Linear evaluation of frozen features on 11 fine-grained benchmarks for few-shot learning. We compare our proposed approach with the image-only baseline by training linear classifier on top of frozen VIT-B/16 DINO features. Our method leverages unpaired text data using Open..LaMA-3B
Dataset
Oxford Flowers
Stanford Cars
Tgve Aiteralt
Oxford Per
Caltech101
Sun397
Lif1111
Food101
Imagones
Avenge
Did
Train Shot
Method
Unimodal
6.28
22.43
9.72
29.22
37.85
15.40
38.67
60.12
54.62
73.25
34.76
Ours
7.89
26.08
10.41
32.45
40.27
18.14
39.28
80.88
58.32
75.66
36.94
Ours (init)
8.96
31.34
34.22
44.32
42.68
66.39
60.37
79.74
40.16
2
Unimodal
12.64
35.64
14.98
38.93
51.14
26.05
50.34
70.84
75.61
83.16
45.93
Ours
14.38
38.62
17.00
40.37
54.28
29.24
50.83
72.88
77.14
85.95
48.07
Ours
15.99
42.31
17.65
42.89
56.46
32.15
52.90
74.82
77.32
87.34
49.98
4
Unimedal
22.60
45.95
24.27
50.30
63.00
38.51
57.99
80.14
85,60
89.67
55.80
Ours
24.83
48.62
25.76
52.64
64.39
40,74
57.96
80.92
87.20
91.17
57.42
Ours (init)
25.83
51.01
26.35
55.06
65.86
42.69
59.32
82.23
87.83
91.99
58.82
8
Unimedal
37.68
52.94
33.67
59.18 8
70.62
49, .48
62.97
85. 26
92.83
93.17
63.78
Ours
39.31
55.31
35.56
60.48
71.88
50.46
63.08 03.08
86.25
93,23
93.47
64.90
Ours (init)
40.50
57.03
35.64
62.27
73.18
51.50
64.09
86.93
93,59
93.71
65.84
16
Unimodal
52.48
58.27
45.34 14
64.81
75.72
56.24
66.36
88.57
95.90
94.27
69.80
Ours
55.88
60.57
47.70
66.21
76.81
96.55
95.12
71.31
Ours (init)
55.82
61.73
48.14
67.02
77.39
58.76
67.08
90.53
96,62
94.98
71.81
38

---

Published as a conference paper at ICLR 2026
Table 17: Linear evaluation of frozen features on 11 fine-grained benchmarks for few-shot learning. We compare our proposed approach with the image-only baseline by training linear classifier on top of frozen VIT-S/14 DINOv2 features. Our method leverages unpaired text data using OpenLLaMA-3B
Dataset
Oxford Flowers
Stanford Cars
Feve Aircraft
Oxford Pets
Caltech10r
San397
Ucf101
Food101
Imagenet
Average
Did
Train Shot
Method
Unimodal
13.18
34.15
14.09
36.60
46.74
35.18
36.48
63.51
89.62
76.66
44.62
Ours
14.95
37.25
14.88
38.93
49.18
37.91
38.35
68.92
91.42
84.04
47.58
Ours (init)
16.49
41.79
15.63
42.04
52.33
42.27
42.69
73.59
93.64
84.52
50.50
Unimodal
24.68
47.88
23.09
47.75
56,81
48.54
50.41
75.32
96.02
86,90
55.73
Ours
26.93
49.65
24.29
50.99
61.67
51.77
51.31
79.44
96.90
89.80
58.28
Ours (init)
28.65
53.15
24.78
53.25
63.86
$4.44
54.21
81.41
97.63
90.55
60.19
Unimodal
38.76
57.51
32.10
59.69
67.75
60.79
58.73
83.89
98.59
93.48
65.12
Ours
41.69
58.87
33.38
61.58
69.60
62.69
59.69
86.27
98.84
94.56
66.71
Ours (imit)
43.17
60.89
33.86
62.43
71.13
63.88
61.38
87.36
99.17
94.96
67.82
Unimodal
54.56
63,00
45,05
64.78
74.19
68.06
64.53
88.68
99.27
94,35
71.65
8
Ours
56.27
64.57
45.98
66.31
75.19
69.22
65. 14
89.78
99.27
95.42
72.71
Ours (init)
57.91
65.82
47.40
67.81
75.99
69.71
66.40
90.29
99.54
95.84
73.67
Unimodal
67.96
67.35
55.89
71.36
77.92
73.24
68. 14
90.73
99.63
96.43
76.22
16
Ours
69.42
68.50
58.54
72.24
78.69
73.80
68.70
91.87
99.72
96.63
77.80
Ours (init)
70.32
69.19
58.74
73.17
79.58
74.51
69.44
92.47
99.82
96.80
78.81
Table 18: Linear evaluation of frozen features 10 fine-grained benchmarks for few-shot learning with DINOv2 VIT-B/14. We compare our proposed approach with the image-only baseline by training linear classifier top of frozen VIT-B/14 DINOv2 features. Our method leverages unpaired text data using OpenLLaMA-3B
Datuset
Oxford Flowers
Stanford Cars
Fgve Aireraft
Oxford Pets
Cattec1011
Sun397
Ucf101
Food101
Imagenet
Average
Did
Train Shot
Method
Unimodal
22.42
43.03
15.79
38.85
58.57
48.71
52.26
76.47
97.12
83.64
53.69
Ours
23.10
45.12
16.
42.69
61.05
51.30
52.45
78.14
98.08
87.68
55.58
Ours (init)
25.47
48.56
16.83
45.31
63.53
54.16
55.56
81.08
97.94
88. 13
$7.66
Unimodal
35.17
55.41
25.54
51.16
69.49
62.13
62.35
84.31
99.58
89.55
63.47
Ours
37.38
56.98
25.88
54.65
70.61
63.89
63.21
85.50
99.70
92.02
64.98
Ours (init)
38.78
59.81
26.00
55.61
71.38
66.54
65.06
86.49
99.62
92.79
66.21
Unimodal
$1.40
63,68
34,25
61.25
76.32
71.60
68, 86
89.05
99.76
94,51
71.07
Ours
54.26
64.65
35.52
62.63
76.87
72.33
69. 14
90.00
99.70
95.51
72.06
Ours (init)
55.01
66.55
35.
63.97
77.57
73.25
70.30
90.31
99.57
95.65
72.73
Unimodal
66.01
68.88
48.17
66.67
79.92
76.26
72.48
90.97
99.80
95.54
76.47
Ours
68.53
69.75
50.88
68.46
81.44
77.34
73.12
92,39
99.70
96.20
77.78
Ours (init)
67.91
70.66
51.26
69.56
81.85
77.95
73.75
92.50
99.68
96.51
78.16
16
Unimodal
77.31
72.17
62.38
73.76
83.80
80.74
75.15
93.34
99.81
97.40
81.59
Ours
78.92
72.80
64.51
75.16
84.62
81.00
75.46
92.92
99.59
97.38
82.24
Ours (init)
78.52
73.18
65.81
75.65
84.77
81.18
75.82
93.28
99.78
97.57
82.56
39

---

Published as a conference paper at ICLR 2026
Table 19: Linear evaluation of frozen features on 10 fine-grained benchmarks for few-shot learning with DINOv2 ViT-L/14. We compare our proposed approach with the image-only baseline by training linear classifier on top of frozen VIT-L/14 DINOv2 features. Our method leverages unpaired text data using Ope3LL
Dataset
Oxford Flowers
Stanford Cars
Feve Aircraft
Oxford Pets
Caltech101
San397
Ucf101
Food101
Imagenet
Average
Did
Train Shot
Method
Unimodal
24.89
48.36
17.69
38.77
66.46
59.27
57.50
79.83
98.13
82.96
$7.39
Ours
25.88
49.63
18.08
42,93
69.18
60.12
58.37
83.51
98.42
86,23
59.24
Ours (init)
27.90
52.86
18.95
43.18
70.98
63.17
60.80
83.86
98.59
88. 17
60.85
Unimodal
39.95
58.95
26.87
50.18
75.79
70.74
67.14 14
84.71
99.74
89.82
66.39
Ours
41.22
60.82
27.15
53.01
76.61
72.07
67.
86.07
99.72
91.95
67. 67.65
Ours (init)
42.93
63.36
28. 14
$4.96
77.72
73.87
69.20
87.13
99.81
91.71
68.88
Unimodal
56.49
66.37
38.59
59.08
80.84
77.39
72.41
89.90
99.73
94.44
73.52
Ours
58.19
67.36
39.57
61.78
81.36
78.19 19
72.82
90.99
99.76
95.27
74.53
Ours (imit)
58.60
68.84
39.19
62.77
81.50
78.99
73.63
90.74
99.88
96.02
75.02
8
Unimodal
70.00
70.71
51.57
66.47
83,84
81.69
76.02
93.53
99.89
95,55
78.93
Ours
71.63
71.59
55.13
67.91
84.47
82.12
76.43
93.62
99.88
96.36
79.91
Ours (init)
72.02
72.51
55.49
69.03
84.57
82.52
76.78
93.80
99.89
96.73
80.33
16
Unimodal
80.84
73.83
64.13
73.96
87.43
84.58
77.78
94.69
99.91
97.36
83.45
Ours
81.85
74.39
69.45
74.70
87.35
84.58
78.35
94.59
99.89
97.61
84.28
Ours (init)
82.76
74,80
69.42
74.88
87.65
84.96
78.58
94.42
99.81
97.62
84.49
E.2
IMPROVING IMAGE CLASSIFICATION USING UNPAIRED TEXTS (ALIGNED ENCODERS)
E.2.1 SUPERVISED FINETUNING
In this section, we fine-tune both the vision backbone and the linear classifier on nine downstream tasks, comparing UML against strong image-only baselines. We evaluate two different backbones: ResNet-50 and VIT-B/16.
As shown in Table 20, across all backbones, UML consistently improves over the image-only baseline by leveraging unpaired text embeddings. Further, our ieaddiniiializatoon variant (Ours (init) outperforms training using unpaired multimodal data from seratch (Ours).
Table 20: Supervised finetuning on fine-grained classification benchmarks with CLIP. We compare proposed approach with the image-only baseline All vision encoders are initialized from CLIP ResNet50 weights, and our approach leverages unpaired text data using the corresponding CLIP text encoder.
when fine-tuning the target dataset.
Dataset
Oxford Flowers
Stanford Cars
Fgve Aireraft
Oxford Pets
Caltech
Sum397
Uef101
Food101
Average
Dtd
Method
Unimodal
36.12
25.93
37.70
51.06
52.49
69.24
63.17
88.42
83.61
56.42
Ours
37.00
24.05
41.34
55.67
60.48
69.77
74.49
92.57
84.79
60.02
Ours (init)
72.75
62.33
66.58
56.50
67.54
76.95
86.97
94.80
87.95
74.71
40

---

Published as a conference paper at ICLR 2026
Table 21: Full linear probing on classification with CLIP ResNet-50 Image Encoder and Text encoder. We compare our proposed approach with the image-only baseline when training linear probe on the target dataset. All vision encoders are initialized from ResNet-50 weights, and our approach leverages unpaired text data using the corresponding CLIP text embeddings.
Dataset
FGVC Aireraft
Oxford Flowers
Stanford Cars
Oxford Pets
Caltech
SUNEY
UCF101
Food101
Average
DID
Method
Unimodal
76.36
70.97
41.88
72.81
81.23
81.60
88.39
97.89
92.78
78.21
Ours
77.23
71.18
42.66
71.81
81.81
81.51
87.84
97.65
93.01
78.30
Ours (init)
79.14
73.83
42.81
73.76
82.13
82.44
90.90
97.69
94.19
79.65
E.2.2 LINEAR PROBING
In this section, we train only the linear classifier, on top of the frozen vision and language backbone from CLIP, on ten downstream tasks, comparing UML against strong image-only baselines.
As shown in Table 21, UMI. consistently improves over the image-only baseline by leveraging unpaired text embeddings. Further, our head-initialization variant (Ours (init]) outperforms training using unpaired multimodal data from scratch (Ours).
E.2.3 FEW-SHOT LINEAR PROBING (ACROSS ARCHITRCTURES)
In this section, we train only the linear classifier, on top of the frozen vision and language backbone from CLIP, for few-shot classification on ten downstream tasks, comparing UML against strong image-only baselines. We evaluate two different backbones: ResNer-50 and VIT-B/16.
As shown in Table 22 and Table 23, across both backbones, UML consistently improves over the image-only baseline by leveraging unpaired text embeddings. Further, our head-initialization variant (Ours (init)) outperforms training using unpaired multimodal data from scratch (Ours).
E.3 IMPROVING VISUAL ROBUSTNESS USING UNPAIRED TEXTS
In this section, we evaluate the robustness models trained with UML to test-time distribution shifts. We train k-shot linear probe (where {1, 2, 4, 83) with DINOv2 on ImageNet and evaluate across four distribution-shifted target datasets: ImageNet-V2? V2, ImageNet-Sketch, ImageNet-A. A, and ImageNet-R. Our method consistently improves robustness over the unimodal baseline (Figure 10, Figure 11, Figure 12 and Figure 13) across different training shots, indicating that language priors help capture more transferable features.
Unimodal
Ours
Ours (init)
+16.7%
+18
24
+36.5%
+55 9%
+37.5%
36
Accuracy
42
36
22
34
40
+5,2%
+6.88
20
+13.5%
14
+24.3%
32
30
+11.6%
3.8
32
18
12
28
ImageNet (Source)
mageNetv2
16
mageNet-Sketch
ImageNet-A
26
ImageNet-R
Figure 10: Robustness under test-time distribution shifts. Our approach (trained on 1-shot) is much more robust than its unimodal counterpart across four distribution-shuffed target sets.
41

---

Published as a conference paper at ICLR 2026
Table 22: Linear evaluation of frozen features on 10 fine-grained benchmarks for few-shot learning. We compare our proposed approach with the image-only baseline by training a linear classifier top of frozen CLIP ResNet50 features. Our method leverages unpaired text data using corresponding CLIP text encoder
Datasct
Oxford Flowers
Stanford Cars
Fgve Aireraft
Oxford Pets
Caltech101
Sun397
Uef101
Food101
Imagenet
Average
Did
Train Shot
Method
Unimodal
23.24
29.14
12.38
30.24
37.55
27.26
34.61
21.36
59.07
66.52
34.14
Ours
36.32
45,40
16.84
40.92
53.19
49.76
53.03
36.48
68.55
76.80
47.73
Ours (init)
57.88
64.59
22.23
50.85
65.99
76.73
86.59
60.92
81.08
83.79
65.06
Unimodal
38.37
43.83
18.63
40.33
$3.25
44.60
47.75
32.62
75.03
78.90
47.33
Ours
46.64
53.53
20.81
48.35
62.01
56.67
60.64
42.21
77.97
84.58
55.34
Ours (init)
61.86
65.90
24.19
55.30
70.39
77.07
87.40
61.40
86.20
85,94
67.57
Unimodal
$1.34
54.38
23.08
$2.07
64.06
$7.29
61.32
41.72
86.16
85.41
$7.68
55.21
59.48
24.77
56.78
67.65
62.68
67.31
47.04
86.46
87.23
61.46
Ours (init)
65.80
68.11
27.49
60.13
73.62
77.79
86.54
62.37
91.60
87.57
70.10
Unimodal
61.74
61.47
30.22
60.15
70.16
64.63
94
49.48
92.20
89.14
64.81
Ours
62.75
63.70
30.69
61.84
70.74
67.73
73.62
52.14
92.31
89.89
66.54
Ours (init)
69.78
69.61
31.62
64.13
77.24
78.58
89.07
63.34
94.21
91.58
72.92
Unimodal
70.94
65.53
35,91
64.30
75.13
70.67
78.49
55.07
95.21
91.26
70.25
Ours
71.58
67.08
36.23
65.62
76.09
71.63
79.52
56.92
95.44
91.94
71.20
Ours (init)
74.56
71.33
37.13
68.09
78.66
79.06
89.71
64.31
96.17
93.31
75.23
Table 23: Linear evaluation of frozen features on 10 fine-grained benchmarks for few-shot learning. We compare our proposed approach with the image-only baseline by training a linear classifier top of frozen CLIP VIT-B/16 features. Our method leverages unpaired text data using the corresponding CLIP text encoder
Dataset
Oxford Flowers
Stanford Cars
Fgve Aireraft
Oxford Pets
Caltech101
Sun397
Uef101
Food101
Imagenet
Average
Did
Train Shot
Method
Unimodal
31.53
33.51
17.76
31.72
43.64
39.40
37.43
27.65
67.95
71.68
40.23
Ours
48.28
53.44
22.06
47.04
63.40
63.92
60.95
47.35
77.82
83.14
14
$6.74
(init)
67.76
70.13
32.26
55.16
75.02
84.25
90.91
69.50
87.58
88.87
72.14
Unimodal
48.45
48:70
23.38
42.04
60.08
58.30
53.56
41.68
82.01
83,20
54.14
Ours
57.89
59.95
27.19
52.27
69.60
71.18
66.78
54.24
87.43
90.20
63.67
(init)
70.75
71.52
33,99
60.17
78.37
85,39
90.67
70.19
92.18
90.09
74.33
Unimodal
61.64
60.66
31.01
54.37
70.49
71.91
69.35
$2. 15
90.99
91.08
65.36
66.24
65,56
32.98
$9.95
74.16
76.19
75.92
$8.
91.32
93.23
69.40
Ours (init)
74.58
73.54
37.38
64.30
81.10
86.05
91.64
70.89
94.80
93.70
76.80
Unimodal
71.76
66.67
38.47
61.96
77.11
78.16
78.25
59.90
95.20
92.98
72.05
72.77
69,50
39.09
64.89
79.01
80.07
80.85
62.63
94.98
94.36
73.82
Ours (init)
78.43
75.07
41.77
68.50
83.41
86.87
92.55
71.97
96.94
95.27
79.08
Unimodal
78.76
71.49
44.74
68.79
80.43
82.08
85.16
63.87
96.97
94.54
76.68
79.40
72.19
45.06
69.41
81.97
82.12
85.92
64.93
96.49
95.28
77.28
Ours (init)
82.38
76.51
47.14
72.13
84.66
86.60
92.68
72.79
97.70
96.08
80.87
42

---

Published as a conference paper at ICLR 2026
Unimodal
Ours
Ours (init)
+7.1%
+16.7%
+24 7%
+15.7%
54
Accuracy
42
45
52
+1.6%
44
+1.95
+4.8%%
40
+3.7%
50
ImageNet (Source)
ImageNetv2
ImageNet-Sketch
ImageNet-A
ImageNat-R
Figure 11: Robustness under test-time distribution shifts. Our approach (trained on 2-shots) is much more robust than its unimodal counterpart across four Iestribution-shuflee target test sets.
Unimoda
Ours
Ours (init)
62
53
+12.2%
Accuracy (N6)
+5.4%
34
22
61
33
21
60
51
+2.0%
32
20
55.5%
59
50
19
18
49
29
ImageNet (Source)
ImageNetV2
ImageNet Sketch
ImageNet-A
ImageNet-R
Figure 12: Robustness under test-time distribution shifts. Our approach (trained 4-shots) is much more robust than its unimodal counterpart across four distribution-shufii target test sets.
Unimodal
Ours
Ours (init)
58
+3 8%
+10.8%
24
50
+8 7%
57
Accuracy
66
56
+1.5%
34
+4 3%
22
48
65
47
64
20
46
54
45
ImageNet (Source)
ImageNetv2
ImageNet-Sketch
ImageNet-A
ImageNet R
Figure 13: Robustness under test-time distribution shifts. Our approach (trained on 8-shots) is
much more robust than unimodal counterpart across four distribution-shuffles target test sets.
E.4 MARGINAL RATE-OF-SUBSTITUTIIN BETWEEN MODALITIES
How many words is an image worth? In this section, extend our results to evaluate image-text conversion ratios using test accuracy isolines on the remaining eight datasets. We measure these global equivalence ratios by fitting plane to the accuracy values given the number of image and text shots. Figures 14 to demonstrate the conversion ratios for DINOv2 VIT-S/14 as the vision backbone and Openl LaMa-33 to 29 show the same ratios for CLIP ResNet-50 as the vision and text encoders (aligned encoders). As expected, with the fully aligned CLIP backbone, each image equates to far fewer text prompts than under the unaligned DINO setting, showing the higher efficiency of aligned embeddings.
the text backbone (unaligned encoders). Analogously, Figures 22
E.4.1 UNALIGNED ENCODERS
43

---

Published as a conference paper at ICLR 2026
14
0.1
6440
14
120
class (m-shot)
1.5
Texts per class (m-shot)
1
1..4
0.5
5.46
6
1.41
0.11
0
Images per class (n-shot)
Images per class (n-shot)
Figure 14: SUN397.
img 1568 words
Figure 15: Caltech101. img 1248 words
14
14
Texts (m-shot)
12
1.99
Texts per class (m-shot)
12
035
10
1.25
1.00
0.42
6
1.20
4
2
E..7
0.45
Images per class (n-shot)
Images per class (n-shot)
Figure 16: Stanford Cars. img 1799 words
Figure 17: DTD. img = 2309 words
1.39
2607
(m-shot)
12
020
Texts per class (m-shot)
10
0.2
8
0.94
4
2
-.12
2
2
Images per class (n-shot)
Images per (n-shot)
Figure 18: FGVC Aireraft. img R: 3220 words Figure 19: Oxford Flowers. img R: 1895 words
44

---

Published as a conference paper at ICLR 2026
0.6
100
033
0.3
Texts per class (m-shod
I
038
0.45
0.13
6.56
0.35
0.50
0.44
per class
8
0.53
0.43
0.35
2
0.16
000
Images class (n-shot)
Images class (n-shot)
Figure 20: Food101.
img As 2608 words
Figure 21: UCF101.
img R 2617 words
45

---

Published as a conference paper at ICLR 2026
E.4.2
ALIGNED ENCODERS (CLIP)
16
061
16
0.09
14-
14-
Texts class (m shot)
12-
036
(m-shot)
12-
005
10
150
10-
8-
6-
4-
2-
0.54
0.79
1
Images per class [n-shot]
Images class (n-shot)
Figure 22: SUN397. img 221 words
Figure 23: Caltech101. 1 img R 256 words
16
14
Texts class (m shot)
12
032
Texts per class (m-shot)
12
031
10
10
6.45
8
6.42
0.16
4-
0.9
0.39
Images per class (n-shot)
Images per class (n-shot)
Figure 24: Stanford Cars. img > 649 words
Figure 25: DTD. img R 228 words
16
16
14
14-
0.24
Texts class (m-shod)
12.
(m-shot)
12:
10-
10
8
8
6
4
-118
2
0.5
0
000
Images per class (n-shot)
Images per class (n-shot)
Figure 26: FGVC Aircraft. img ~ 691 words
Figure 27: Oxford Flowers. img x 851 words
46

---

Published as a conference paper at ICLR 2026
16
049
16
14
042
14
Texts class (m shot)
12
0.58
(m-shot)
12
10
0.55
25
10
0.33
8
6:48
6-
6:44
4
644
0.50
6:52
0
6.18
0.64
Images per class (n-shot)
Images per class (n-shot)
Figure 28: Food101. img R 202 words
Figure 29: UCF101. img R 393 words
E.5 IMPACT OF SCALING VISION BACKBONE
In this section, we study how our method's 's performance scales with the size and architecture of the vision backbone. In addition ViT-S/14 DINOv2, extend our analysis range of ViT-based architectures, including ViT-B/14 and ViT-L/14 DINOv2 and ViT-B/16 and ViT-B/8 DINO models. To ensure fair comparison, we follow the same training protocol as previous experiments. Our method consistently outperforms the unimodal baselines in every setting. In few-shot linear probing across ViT-B/8, ViT-B/16, DINOv2- ViT's and ViT-L/14 backbones (Tables 15 19), we see clear gains. The same holds for full-dataset end-to-end fine-tuning of both encoder and head (Tables 6 9), and even when only the linear classifier is trained on the full splits (Tables 10 14).
E.6 IMPACT OF VARYING TEXT ENCODERS
In this section, we study how our method's performance varies with different language models used for generating text embeddings. Through this experiment, aim understand how differences in embedding quality and model capacity affect the integration textual information in our multimodal setup. Specifically,
cover L.MMs with diverse architectures and scales, including BERT-Large, RoBERTa-Large and GPT-2 Large. As shown in Figure 30, adding unpaired text embeddings shows significant boost in J-shot accuracy and still decent gains at 16 shots on SUN397 dataset. Overall, OpenL.LMMA-3B outperforms all other language models.
70
65
Test Accuracy (%)
60
55
50
45
Unimodal
+BERT
40
+GPT2
+RoBERTa
35
+OpenLLaMA-38
1
16
Training shots
Figure 30: Few-shot classification accuracy on SUN397 using UML with unpaired, frozen embeddings from various pretrained language models.
47

---

Published as a conference paper at ICLR 2026
E.7 LEARNING WITH COARSE- GRAINED VS. FINE- GRAIIERD TEXTUAL CUES
Understanding the type of information extracted from textual cues is crucial to assessing the effec- tiveness of our multimodal approach. key question whether the model merely utilizes class names or goes beyond to capture richer, more descriptive features. To investigate this, we compare the performance of our method using two types of text templates: vanilla template that consists solely of the class name (c.g., "a photo of a [class]") and descriptive templates generated from GPT-3, as detailed Section Appendix B. As shown Figure and Figure 32, both multimodal approaches consistently outperform the unimodal baseline, with descriptions from GPT-3 offering more substantial performance gain. This shows that leveraging richer, contextually diverse text cues can significantly enhance model performance, even in low-shot learning scenarios.
65
Test Accuracy (%)
60
55
50
45
45
40
Unimodal
40
Unimodal
Vanilla Descriptions
Descriptions
35
Der3-Dessciptions
35
GPT3-Descriptions
8 Training shots
16
2
8
16
Training shots
Figure 31: Few-shot SUN397 accuracy with UMI.
Figure 32: Few-shot SUN397 accuracy with
using two levels of textual granularity: (a) vanilla class descriptions and (b) GPT-3--enerated fine- grained descriptions.
UML (init) using two levels of textual granular- ity: (a) vanilla class descriptions and (b) GPT- 3-generated fine-grained descriptions.
E.8
IMPACT ON PERFORMANCE WITH INCREASING UNPAIRED TEXT PROMPTS
augment each image with increasing number unpaired text prompts. Figure 33 shows these accuracy curves as we vary the number of unpaired text prompts per image shot across five image-shot budgets. In every regime, our multimodal initialization ("Ours (init)") outperforms training the head from scratch, with most of the gain coming from the first few prompts and gains tapering off thereafter. Note that we do enforce diversity or novelty in the unpaired text pompts--imppl adding more sentences does not guarantee additional information.
Here, we investigate how classification accuracy evolves
48

---

Published as a conference paper at ICLR 2026
Image shots: 1
Image shots:
Image shots: 4
42
61
Test Accuracy (%)
Test Accuracy (%)
Test Accuracy
40
52
60
38
50
59
Ours
Ours
Ours
58
Ours (init)
Ours (init)
Ours (init)
48
7
12
5
10
12
0
10
15
Text shots
Text shots
shots
Image shots:
Image shots:
66
Test Accuracy (%)
Test Accuracy (9%
69.0
65
68.5
64
68.0
(init)
67.5
Ours (init)
1
5 7 10 10125 shots
2 5 1012
shots
Figure 33: Classification accuracy as a function of the number of text prompts per image shot for the SUN397 Dataset.
E.9 EFFECT OF UNRELATED AUXILIARY MODALITIES
While our main experiments investigate the benefits of incorporating semantically related auxiliary modalities, an important question what happens when the auxiliary data is unrelated or even adversarial. In our unpaired setting, this requires reasoning not about individual mismatched pairs, but about the relationship between entire data distributions. To meaningfully study the adversarial case, one would need a principled definition of "negative correlation" between modality distributions, as well as metrics for quantifying We leave such a rigorous framework to future work.
Instead, here, we evaluate simpler but informative scenario: the independent case, where the auxiliary modality semantically unrelated to the target modality. We use our few-shot learning setup to train UML with image data from SUN397 and text data from Stanford-Cars unrelated datasets. Table reports results for few-shot image classification on the SUN397 dataset. When the auxiliary text is unrelated the image domain, performance does not improve the unimodal baseline. In contrast, semantically related text provides consistent gains across all shot counts.
semantically
Table 24: Training UML with unrelated auxiliary text from the Stanford Cars dataset does not yield performance gains image classification the SUN397 dataset. However, semantically correlated text (from SUN397) consistently improves accuracy across few-shot settings.
Method
1-shot
2-shot
4-shot
8-shot
16-shot
Unimodal (Image)
34.15
47.88
57.51
63.00
67.35
UML (Image + Unrelated Text)
35.27
47.12
57.50
62.45
67.25
UML (Image + Related Text)
41.79
53.15
60.89
65,82
69.19
E. EXTENSION TO MORE THAN TWO MODALITIES
Our framework naturally extends beyond two modalities. We validate this by extending our image and audio classification experiments ICS all three modalities: image, audio, and
49

---

Published as a conference paper at ICLR 2026
text. Training alternates batches from each modality while applying modality classification losses, consistent with our woo-modality setup.
Tables 25 and 26 summarize results audio and image classification. In both cases, incorporating additional unpaired modalities consistently improves performance over unimodal or pairwise settings. These findings demonstrate that the performance benefits of UML extend robustly more than two modalities.
From theoretical perspective, results also generalize directly. Since modality observa- tions are conditionally independent given the ground-truth latent 2", their joint contribution to the Fisher information reduces the sum of the unimodal blocks. Consequently, the total contribution of all auxiliary modalities (excluding the primary modality X) can be obtained by summing their individual Fisher information matrices.
Table 25: Training UML with all three modalities from ImageNet-ESC outperforms unimodal or pairwise training audio classification. Numbers in parentheses denote relative improvements over the unimodal baseline.
Dataset
Method
I-shot
2-shot
4-shot
ESC-19
Audio-Only
28.78
39.85
52.22
Audio Image
34.59
44.13
50.00
Audio + Text
35.47
52.19
52.90
Audio Image + Text
44.46 (+54.4%)
51.48
(+29.2%)
56.57 (+8.3%)
ESC-27
Audio-Only
25.65
35.99
44.79
Audio Image
37.15
42.86
51.15
Audio Text
41.97
47.02
53.85
Audio Image + Text
44.68 (+74.2%)
48.03 (+33.4%)
54.16 (+20.9%)
Table 26: Training UML with all three modalities from ImageNet-ESC outperforms unimodal or pairwise training on image classification. Numbers in parentheses denote relative improvements over the unimodal baseline.
Dataset
Method
1-shot
2-shot
4-shot
ESC-19
Image-Only
60.28
74.10
78.70
Image Audio
64.663
76.17
85.02
Image Text
88.84
89.71
92.07
Image + Audio + Text
90.55 (+50.2%)
91.08 (+22.9%)
91.72 (+16.5%)
ESC-27
Image-Only
55.33
65.60
77.85
Image Audio
59.75
70.93
78.14
Image Text
86.39
88.91
90.14
Image Audio + Text
88.22 (+59.5%)
88.96 (+35.6%)
91.78 (+17.9%)
E.11 EFFECT OF RATIO MODALITY BATCHES
In our main experiments, UML was trained with simple 1:1 alternation of batches across modalities. To study the effect of this schedule, we ablate the ratio of text to image batches, denoted by r, on SUN397 using VIT-S/14 DINOv2 (vision) and LaMA3B (text). We evaluate both in the (a) linear probe setting and the (b) full finetuning setting.
Across both settings and for both UML and UML (init), observe that the choice of has little impact on performance. The gains primarily arise from the presence auxiliary information rather than the exact frequency of its appearance during training.
50

---

SUN397 Published as a conference paper at ICLR 2026
Table 27: Few-shot linear probing with different ratios of text-to-image batches on SUN397 using ViT-S/14 DINOv2 and OpenL OpenLLaMM-3BB. Performance remains stable across ratios indicating robustness of UML to batch scheduling.
Method
Shot
0.25
r 0.5
1.0
2.0
4.0
UML
2-shot
49.03
49.23
49.65
49.56
49.97
4-shot
59.05
59.06
58.87
58.92
58.70
8-shot
64.63
65.24
64.57
64.98
65.29
16-shot
68.55
68.79
68.50
68.86
68.36
UML (init)
1-shot
42.60
42.13
41.79
41.95
42.04
2-shot
52.83
53.01
53.15
53.04
52.81
4-shot
61.10
61.09
60.89
60.63
60.99
8-shot 65.29
65.32
65.82
65.15
65.21
16-shot
69.19
68.91
69.19
68.51
68.71
Table 28: Full finetuning of the vision encoder and linear head with different text-to-image batch ratios SUN397. Results show that performance largely insensitive
Method
0.25
0.5
1.0
2.0
r= 4.0
UMI. 66.44
66.80
66.72
67.60
66.44
UML (init)
66.58
65.41
66.03
65.86
64.25
E.12 EFFECT FREEZING vs. UNFREEZING THE TEXT ENCODER
In all our main experiments, we freeze the text encoder. This design choice allows to isolate the role of the auxiliary modality, ensuring that improvements in the primary modality (e.g., vision) arise from cross-modal transfer rather than joint training of both encoders.
In principle, however, one can also unfreeze the text encoder and update it during training. To study this, we ablate freezing versus unfreezing on Stanford Cars and SUN397 using VIT-S/14 DINOv2 (vision) and OpenLLaMM-33 (text). As shown Table 29, unfreezing the text encoder improves performance Stanford Cars, but slightly reduces performance SUN397- due the larger number trainable parameters and increased optimization complexity.
Table 29: Effect of freezing versus unfreezing the text encoder when training with UML.. Freezing stabilizes training and often yields slightly stronger gains.
Method
Stanford Cars
Unimodal (Image Only)
79.45
66.20
UML (Unfrozen Text Encoder)
84.23
65.80
UML (Frozen Text Encoder)
84.87
66.72
E. ADDITIONAL EXPERIMENTS FOR AUDIO- VISUAL SETTING
In this section, extend unpaired multimodal framework to the tri-modal ImageNet-FSC benchmark, examining how unpaired audio and text signals can enhance image classification under both aligned (Appendix 13.2) and unaligned encoders(Appendix E. 13.1). We then reverse the setting, showing that unpaired visual and textual context likewise improves audio classification (Appendix 13.33)
E.1 IMPROVING IMAGE CLASSIFICATION WITH UNPAIRED AUDIO AND TEXT (UNALIGNED ENCODERS)
51

---

Published as a conference paper at ICLR 2026
mage-Only
Ours (Image + Audio)
Ours (Image + Text)
80
825
Accurryy
75
+10 10%
9%
15.5%
665
0%
80
+4.3% 85
+3 5%
+1.1%
70
85
2 600
75
72.5
65
(a) ImageNet-ESC-19
(b) ImageNet-ESCC22
Figure 34: UML improves image classification using unpaired audio and text samples on both ImageNet-ESC 19 and ImageNet-ESC-27 benchmarks when trained on top of DINOv2 VIT-S/14 and OpenLLaMa-3B.
E. 13.2 IMPROVING IMAGE CLASSIFICATION WITH UNPAIRED AUDIO AND TEXT (ALIGNED ENCODERS)
Image-Only
Ours ( mage + Audio)
Ours (Image + Text)
Text Accuracy
80
90
+21
90
Tedt Rrcuu Rocurayy 251
80
-35.5%
90
+15.0%
30
80
85
+8.0%
70
80
85
75
+2 8%
80
80
+8.0%
70
+0 4%
80
1-hhot oiassiiccaionn
2-0hot class/icatiin
4-thot cssiifiationn
1-ahot dlseeficaio
2vshol classification
Asihol ficatio
(a) ImageNettESC-199
(b) ImageNet-ESC-27
Figure 35: UML improves image classification using unpaired audio and text samples on both ImageNet-ESC-199 and ISSC benchmarks when trained on top of CLIP ResNet-50 image and text encoders
E. 13.3
IMPROVING AUDIO CLASSIFICATION WITH UNPAIRED IMAGE AND TEXT (ALIGNED ENCODERS)
Audio Only
Ours (Audio + Imade)
Ours (Audio + Text)
575
45
Accuracy 196)
35
50
+31.0%
35.0
+1.3%
800
4464 0%%
45
+30.0%
-20.2%
52
291
45
+10 7%
30
30
47.
25
ohot c assification
35
shot dlsss cotion
43
shot desssficatioon
shot catsn
2 shot dassificationn
shot classiication
(a) ImageNet-ESC-19
(b) ImageNet-E5C-27
Figure 36: UML improves audio classification using unpaired image and text samples both ImageNet- and ImageNet-ESC-27 benchmarks when trained on top of CLIP ResNet-50 image and text encoders
E. GAUSSIAN EXPERIMENTS
Here, we shift our attention to a more nuanced and intriguing question: can incorporating unpaired multimodal data actually improve the reconstruction quality of a single modality? At first glance, this seems unlikely-why would adding data from a different modality make X reconstruction better than training with X Moreover, we push this question further: can incorporating data from different modality, while keeping the total dataset size fixed, still improve the reconstruction of compared to using the same number of samples X dataset alone? This setup isolates the importance of multimodal information from mere data scaling, and surprisingly, our experiments show that this improvement is indeed possible
52

---

Published as a conference paper at ICLR 2026
To investigate this, design synthetic experiment inspired by theoretical framework in Sec- tion 3.1. We generate data from two partially overlapping modalities, X and Y, derived from a shared latent space de, while also containing unique components (0z and 0y). The observations follow the same linear structure as in our theory:
X; = Ac,tte + A2, + cx,i
y, Be By. CY.j
The training data are generated from Gaussian latents with dimensions d0m( = 10, dim (0xz 5, and dim(0,)) = 5. For X, only the first 10% of the shared components in 0g are retained full strength, while the rest are downscaled by 0.05; Y observes all shared components at full strength. This asymmetry makes informative about structure that is only weakly present X. The validation set is constructed from the same projections but without attenuation, g.. Observations are 50-dimensional with Gaussian noise cx, cy ~ (O, 0.09I). In the unimodal setting, training X alone uses 10,000 samples from X. When training UML on unpaired X and Y, we instead use 5,000 samples from each modality to keep the total sample budget fixed, ensuring a fair comparison.
both modalities fully observe
Our architecture is a shared autoencoder. Each modality X E 1250 Y E R250 is projected into common space of dimension 128 through mppelit linear layers. shared encoder (two linear layers with ReLU) maps into a latent space of dimension 10, followed by a shared decoder (two linear layers) that expands back dimension 128. Finally, podality-sppecifn heads reconstruct the original inputs. The shared pathway enforces cross-modal alignment, while the separate adapters preserve modality fidelity.
As shown in Figure 37, the surprising outcome is that training on both modalities, even when they are unpaired, consistently improves the reconstruction of X compared to training solely on X. More strikingly, this improvement holds even when the total number of training samples is fixed, with half the data coming from X and half from Y; showing that the model not just benefiting from increased data quantity but from the diversity and complementary information provided by the second modality.
3.5
Reconstruction Error on
Uncaired (X Y)-data
3.0
X-data only
2.5
2.0
1.5
1.0
200
400
Training Steps
Figure 37: Training on N/2 samples from X and N/2 unpaired samples from Y improves test reconstruction on X, more than training on N samples from X.
F ANALYSIS OF THE LEARNED CLASSIFIER
CHANGE DECISION BOUNDARIES WITH UNPAIRED DATA FROM ANOTHER MODALITY
Our decision boundary visualizations constructed by projecting the nigh-dimensional embedding space of a given classifier to 2D plane. Axis is computed as the normalized difference between the classifier weights of the two selected elasses, representing the primary decision direction. Axis
53

---

Published as a conference paper at ICLR 2026
2 is chosen to be orthogonal to Axis 1, constructed from the difference between the class mean embeddings after removing the component parallel to Axis 1. This orthogonalization ensures that the two capture complementary aspects: Axis 1 reflects the primary model decision boundary, while Axis 2 captures the variation orthogonal to that decision. The final 2D projection matrix combines these vectors as columns, and embedding vectors
then mapped to this plane using simple dot product. Figure 38, Figure 39 and Figure 40 show the change in decision boundary when adding unpaired textual information for 2-shot classification on top of frozen CLIP ResNet-50 features for Oxford Pets, DTD and Oxford Flowers datasets.
Train Image
Test Image
Russian Blue
Abyssinian
Decision Boundary
pet typically has blue-gray coat,
eyes. auscular body.
A pet Abyssinian looks Like small, brown and black ears.
(a) Unimodal
(b) Ours
Figure 38: Impact of umpaired text on decision boundaries (CLIP ResNet50.) (Left) Visual features alone learn ambiguous class boundaries between Russian Blue and Abyssinian cats. (Right) Adding unpaired text sharpens the boundary, leveraging semantic cues to better distinguish similar categories.
Train Image
Test Image
Knitted
Cobwebbed
Decision Boundary
The knitted texture looks like series interconnected loops.
Cabwebbed texture looks Like spiderweb.
(a) Unimodal
(b) Ours
Figure 39: Impact of unpaired text on decision boundaries (CLIP ResNet50). (Left) Visual
features alone learn ambiguous class boundaries between knitted and cobwebbed. (Right) Adding unpaired text sharpens the boundary, leveraging semantic cues to better distinguish similar categories
F.2 WHAT DO MODELS LEARN FROM UNPAIRED DATA?
To understand what the model is truly learning and how its weights evolve, develop and analyze three key metries: functional margin, silhouette score, and class-prototype vectors. These metrics inform how well the model distinguishes between classes and how text information influences the structure of feature-space
Functional margin. This quantifies how confidently model separates a given sample from the decision boundary. For sample belonging to class y, we calculate the margin relative to the next highest competing class. Specifically, identify the second- highest logit among the incorrect
54

---

Published as a conference paper at ICLR 2026
Train Image
Test Image
Ball Moss
Passion Flower
Decision Boundary
The flower flower purple and white flover.
Ball greentsh. brown plant
(a) Unimodal
(b) Ours
Figure 40: Impact unpaired text on decision boundaries (CLIP ResNet50). (Left) Visual features alone learn ambiguous class boundaries between ball moss and passion flower. (Right) Adding unpaired text sharpens the boundary, leveraging semantic cues to better distinguish similar categories
classes, denoted as class j", and compute the functional margin as
w T; w..
71 Wyy W l|2
(8)
where / xy represents the logit for the true class, while wf. 2n represents the highest logit among the competing classes. Larger margins indicate more confident and robust classification, while smaller margins imply that the sample lies closer to misclassification boundary. As shown in Figure 41, both Ours and Ours (init) exhibit substantially larger classification margins than the unimodal baseline, demonstrating that augmenting primary-modality mooality training with unpaired multimodal data improves confidence in predictions over the primary modality.
Unimodal
Ours
Ours (init)
Cassification Margin (tb)
+70.7%
12
+49.90
+43,9%
8
58.7%
7
+25.1%
10
20..1%
6
+10.0%
+10.9%
+5.3%
5
8
classification
classification
classification
classification
Figure 41: Functional margin of the linear head trained SUN397 dataset for few-shot classification significantly increases when training with both UMI. and UML with linear head initialization.
Silhouette Score and DB-Index. The Silhouette Score indicates how well-separated the clusters are, while the DB-Index measures intra-class compactness versus inter-class separation. Higher silhouette and lower DB-Index values mean better-defined clusters, indicating that text helps tighten intra-class spread and widen inter-class gaps. As shown in Figure 42 and Figure 43, both Ours and Ours (init) exhibit reduced intra-class distances and increased inter-class separations, further confirming improved class separability.
Class-Prototype Vectors. These vectors are the rows of the final linear layer's weight matrix, representing the class centroids in the shared embedding space. We compute heatmap of inner products between class prototypes and average text embeddings of the corresponding class to assess how well text features align with class centers. This helps reveal how the model organizes multimodal information. Figure 44 shows a pronounced diagonal structure, indicating that each class's text embedding aligns closely with the learned weights of the model.
55

---

Published as a conference paper at ICLR 2026
Unimodal
Ours
Ours (init)
0.020
0.05
0.05
0.06
Slinuuete Score (1)
-140.0%
+60 0%
+16.7%
+33.455
+15.4%
+17.%%
+25 1%
+23.1%
0.03
0.03
0.010
0.02
0.005
0.01
0.01
Figure 42: Silhouette Score of the linear head trained on SUN397 dataset for few-shot classification significantly increases when training with both UML and UML with linear head initialization.
Unimodal
Ours
Ours (init)
1.40
2.100
2.70
3.10
1:38
-0.9%
1,8%
2.075
2.65
3.05
1.36
2.050
2.60
3.00
1.34
-1
-2.0%
3,2%
2.2%
-4.1%
1.32
1.30
2.shat classification
2.000
shoht cassification
2.50
B Bishot classitication
2.90
16 shot classrfication
Figure 43: DB-Index of the linear head trained on SUN397 dataset for few-shot classification significantly improves when training with both UML and UML with linear head initialization.
Caea
Ceel
8
Cre2
3
Cas a
Cascs
Casc6
Classifeer weights (per class)
Cass 7
Can O
Can
Cas t0
Can
2
Similarity
Cee
Cae
Cae
Can
can
Cas
nox
nocs
Cus
Ose
Cas
owe
ose
Cue
oue 23
one
One
oss ow0 P
Average class descriptions
Figure 44: Inner products between each linear-head weight vector and its class's mean text embedding.
demonstrating that text features align well with class prototypes.
56


Reviews:
[{"review_id": "Ry1t6am6MR", "reviewer": "Reviewer_8Sv3", "summary": "The authors propose to augment unimodal models with unpaired training data from other modalities. Their approach shares weights across all modalities while optimizing for the downstream task on a joint, unpaired, multimodal training dataset. They evaluate their approach on several supervised, self-supervised, and transfer learning settings.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is very well written - ideas and motivations are expressed clearly and in easy-to-understand terms; experimental results are stated and discussed in a well-organized manner.\n\n2. The experiment in Section 4.4 on marginal rate-of-substitution between modalities is quite interesting and the results are insightful.\n\n3. The authors empirically show that in the scenarios considered, it is indeed possible to achieve some practical performance improvement through joint/pre-training with data from modalities other than the target.", "weaknesses": "1. The idea that data from multiple modalities can be used for unified pretraining without any special consideration for the nature of the modalities being integrated is a fundamentally flawed premise. It is well known in literature that modalities can often have conflicting information, and even multimodal models, which often operate on explicitly paired data and are trained with the objective of aligning the modalities, struggle with this integration. There are plenty of works dedicated to addressing specifically this problem [a, b]. At the low level, for instance, modalities can often have different convergence rates [c] and provide conflicting gradients to the model [d], none of which can be thought of as being helpful from a training perspective without special treatment [d]. In fact, [c] specifically established the general impossibility of what the authors propose in this work.\n\n2. Although the experiments show performance some improvements, the generality with which the paper is presented is misleading. Attempting to perform joint multimodal training using the proposed approach in the settings and datasets used in [a, b, c, d], is likely to falsify the findings reported, since special adjustments to cope with the challenges of integrating multiple modalities is needed to deal with those settings.\n\n3. No ablation or analytical studies have been reported for the proposed approach, which makes it difficult to evaluate the contribution of the various design choices, for instance, the proposed UML objectives, the contribution of unpaired samples from a different modality, convergence in the uni-modal vs multi-modal setting, etc.\n\n4. Finally, neither the idea of sharing model weights, nor doing pretraining with data from multiple modalities can be regarded as novel, since they have been around in the multimodal learning community for a long time [e, f, g]. In fact, this is also the problem that works on domain generalization attempt to solve, albeit framed in a different manner [h].\n\nMinor:\\\nLine 449: \"Further results on\" -> \"For further results on\"\n\nReferences:\\\n[a] Zhang et al., \"Robust Multimodal Large Language Models Against Modality Conflict\", ICML 2025.\\\n[b] Ma et al., \"Improving Multimodal Learning Balance and Sufficiency through Data Remixing\", ICML 2025.\\\n[c] Wang et al., \"What makes training multi-modal classification networks hard?\", CVPR 2020.\\\n[d] Javaloy et al., \"Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization\", ICML 2022.\\\n[e] Ngiam et al., \"Multimodal Deep Learning\", ICML 2011.\\\n[f] Hu et al., \"Towards Unsupervised Sketch-based Image Retrieval\", BMVC 2022.\\\n[g] Rastegar et al., \"MDL-CW: A Multimodal Deep Learning Framework with Cross Weights\", CVPR 2016.\\\n[h] Gulrajani et al., \"In Search of Lost Domain Generalization\", ICLR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Af2KkY0ZFO", "reviewer": "Reviewer_EcBF", "summary": "This paper proposes a novel method, **UML** (**U**npaired **M**ultimodal **L**earner), to improve the performance of unimodal models by leveraging abundant, unpaired data from other modalities.\nThe core idea is that even without direct pairings (like an image and its specific caption), data from an auxiliary modality (e.g., text) can provide complementary information to enhance a model focused on a target modality (e.g., images).\nThe **UML** method works by having a single model with shared parameters (weight sharing) alternately process inputs from the different modalities. This design allows the model to capture shared underlying concepts and structures from both datasets, even though they are not explicitly linked.\nMoreover, the paper theoretically demonstrates that this approach strictly increases the Fisher information under linear assumption of generating data, resulting in more informative and robust representations for the target modality than training on that modality alone.", "questions": "- What is the main difference from prior works, and what constitutes the key novelty of this paper? In my view, as the authors mention in Section 3 and Appendix A.1, UML might seem slightly incremental, as it combines concepts from previous studies, such as shared model parameters (e.g., [1]), and the use of unpaired datasets (e.g.,[2]).\n- What happens if the text or data (modality $\\text{Y}$) are randomly generated? For instance, what if the text description is entirely unrelated to the image (e.g., the image depicts a dog, but the text describes playing sports)? Would such mismatched modalities disrupt the increase in Fisher Information?\n- Yet the authors note in the limitations section that most experiments are conducted on classification tasks, I still raise a concern regarding the applicability of the proposed method to other tasks, such as image–text retrieval.\n\n\n[1]  Chada, et al. \"Momo: A shared encoder model for text, image and multi-modal representations.\" arXiv preprint 2023\\\n[2] Lee, Jae-Jun, and Sung Whan Yoon. \"Can One Modality Model Synergize Training of Other Modality Models?.\" ICLR 2025\n\n=======================================================\n\n**Note**: I acknowledge that I may have partially misunderstood certain aspects of the paper. Therefore, I am willing to raise my rating score if these questions and concerns are adequately addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper presents strong theoretical approaches demonstrating how unpaired multimodal datasets, or datasets with missing modalities, can be effectively synergized with existing datasets. It shows that these datasets can be linearly combined, leading to an increase in Fisher Information.\n- The paper provides extensive experiments on various benchmark datasets that support its theoretical framework, validating the proposed methods and concepts.\n- The paper is well written, particularly in the Introduction and the section explaining the main concept.", "weaknesses": "**Major**\n\nThe paper’s effort to ground its empirical findings in theoretical analysis is commendable; however, the theoretical assumptions appear too restrictive to be fully explanatory. The authors provide valuable intuition for UML by presenting theorems (Sec. 3.1) derived under a linear data-generating process.\nNonetheless, this represents a significant simplification of the highly non-linear dynamics of the large Transformer-based models (e.g., DINOv2, OpenLLaMA) used in the experiments. While the idea that shared weights act as a practical analogue of *Fisher information linear combination* is intriguing, it remains more of an intuitive analogy than a formal justification.\nIn my view, this creates a potential gap between theory and practice. The empirical results are undeniably strong and well-presented, yet they seem to be motivated by the theoretical framework rather than explained by it.\n\n**Minor**\n\n- Although the theoretical motivation is clearly written, the derivations of the theorems are somewhat verbose and occasionally redundant, which reduces readability. A more concise and streamlined presentation could enhance the clarity of the theoretical section.\n- While the experimental validation is extensive, its presentation in Section 4 and the Appendix seem overly dense. For example, much of the dataset and model information is relegated to the Appendix and only briefly mentioned in Section 4.1, where it hinders readability. Additionally, the color scheme used in Table 1 and Table 2 (blue and pink) is identical, even though the categories differ (Table 1: datasets; Table 2: settings).\nAs a suggestion (not a requirement), reorganizing the experimental sections might improve clarity. (e.g., focusing on the supervised setting in Section 4.1 and moving the self-supervised setting to Section 4.2, and removing the color scheme from Tables 1 and 2 to avoid confusion)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IehNigMq5n", "reviewer": "Reviewer_mJFa", "summary": "The authors propose the Unpaired Multimodal Learner (UML), which leverages auxiliary unpaired multimodal data to enhance representation learning in a target modality. Specifically, the authors present two frameworks, one for self-supervised learning and one for supervised learning, both of which utilize multiple encoders and a shared network. Theoretical and empirical results indicate that adding unpaired data of modality Y can lead to better reconstruction in modality X compared to adding additional data from X itself.", "questions": "1. The experiments do not include comparisons with related works. Do the authors think it is necessary to compare against other approaches mentioned in the Further Related Works section?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The topic addressed in the paper is interesting.\n2. The paper is clearly written and easy to follow.\n3. The theoretical analysis is sound, and the proposed method is simple yet effective.", "weaknesses": "1. The primary concern lies in the motivation: unpaired multimodal data is relatively uncommon, whereas missing-modality data is more frequently encountered. If the data is only unpaired, why not use existing models (e.g., CLIP) to align them and create paired data? Therefore, a more practical setting might be using images from one dataset alongside text from another dataset.\n2. It would strengthen the paper to include experiments using various encoders and various shared networks to validate the general effectiveness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)