Can One Modality Model Synergize Training of Other Modality Models?
Authors: Jae-Jun Lee
Sung Whan Yoon
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal learning
Representation learning
learning theory
Primary area: learning theory
TL;DR: We conducted theoratical and empirical frameworks that one model training can be promoted by the other model across modalities without exactly paired labels.

Abstract: Learning with multiple modalities has recently demonstrated significant gains in many domains by maximizing the shared information across modalities. However, the current approaches strongly rely on high-quality paired datasets, which allow co-training from the paired labels from different modalities. In this context, we raise a pivotal question: Can a model with one modality synergize the training of other models with the different modalities, even without the paired multimodal labels? Our answer is 'Yes'. As a figurative description, we argue that a writer, i.e., a language model, can promote the training of a painter, i.e., a visual model, even without the paired ground truth of text and image. We theoretically argue that a superior representation can be achieved by the synergy between two different modalities without paired supervision. As proofs of concept, we broadly confirm the considerable performance gains from the synergy among visual, language, and audio models. From a theoretical viewpoint, we first establish a mathematical foundation of the synergy between two different modality models, where each one is trained with its own modality. From a practical viewpoint, our work aims to broaden the scope of multimodal learning to encompass the synergistic usage of single-modality models, relieving a strong limitation of paired supervision. The code is available at https://github.com/johnjaejunlee95/synergistic-multimodal.

Published as a conference paper at ICLR 2025
CAN ONE MODALITY MODEL SYNERGIZE TRAINING OF OTHER MODALITY MODELS?
Jae-Jun Lee', Sung Whan Yoon!.2.
Graduate School of Artificial Intelligence and *Department of Electrical Engineering Ulsan National Institute of Science and Technology (UNIST) {john jae 959 (ayoon88@unn.. att.oss..utsst......m.oo. at.ac.kr
ABSTRACT
Learning with multiple modalities has recently demonstrated significant gains many domains by maximizing the shared information across modalities. How- ever, the current approaches strongly rely on high-quality paired datasets, which allow co-training from the paired labels from different modalities. In this context, we raise pivotal question: Can a model with one modality synergize the train- ing of other models with the different modalities, even without the paired multi- modal supervision? Our answer is 'Yes'. As a figurative description, we argue that writer, i.e., a language model, can promote the training of painter, i.e., a visual model, even without the paired ground truth of text and image. We theoret- ically show that superior representation can be achieved by the synergy between two different modalities. without paired supervision. As proofs of concept, we broadly confirm the considerable gains from the synergy across visual, language. and audio models. From a theoretical viewpoint, we first establish mathemati- cal foundation of the synergy between two different modality models, where each one is trained with its own modality. From practical viewpoint, work aims broaden the scope of multimodal learning to encompass the synergistic usage single-modality models, relieving strong limitation of paired supervision. The code is available at http:///hhub.ooo///jjjjjuueeee// maannniiiiiimmmmmmdaa.. multimodal.
INTRODUCTION
In recent years, multimodal learning, which aims train the joint information across different modalities, is changing the paradigm of cutting-edge deep models from the conventional one modal- ity training to the joint training of multiple modalities. Based on the recent success of large mod- els, such as residual architectures (He et al., 2016), Transformers (Vaswani et al., 2017; Devlin al., 2018; Liu et al., 2019) and Vision Transformers (ViT) (Dosovitskiy et al., 2021; Steiner al., 2022), has spurred the emergence of numerous multimodal learning methods, mainly focusing on the vision-language domain.
Notable practices include CLIP (Radford et al., 2021) and ALBEF (Li et al., 2021), which pursue to align the representations from the vision and language modalities. Also, as another branch, CoOp (Zhou et al., 2022b) and CoCoOp (Zhou al., 2022a) are based on adjustments of the prompt tokens of pretrained languae models. More recently, ViT+LLaMA (Pang et al., 2024) attempt to concatenate the pretrained language models with vision models, hypothesizing that the language models would filter significant information from the features extracted by the vision encoder.
This practical success relies the presumption that the correlated modalities create synergy when jointly trained on multimodal data samples, e.g., a synergy of the visual information and the textual description of image, synergy of the RGB-basee camera images and the LIDAR-based sens- ing signals of autonomous vehicles. As a theoretical foundation of the practices, researchers claim the existence of the true latent representations. which able encompass multiple modalities (Huang et al., 2021; Huh et al., 2024). This implies that when a paired supervision from multiple modalities is given, a model has the potential to more accurately represent the shared semantics by incorporating additional information from other modality models.
Corresponding Author

---

Published as a conference paper at ICLR 2025
We want to raise critical limitation of the prior works in both practical and theoretical sides. First, the existing multimodal learning methods mainly require the perfectly paired datasets, leading to the immense efforts in building multimodal datasets of high quality and restricting the usage of descent single-modality models in unleashing the synergy between modalities. Second, the current theory only explains how multiple modalities promote better representations when paired labels across modalities are given.
In this context, we start by raising a pivotal question: Can one modality model synergize training of other modality model, even without matched supervisions across modalities? As an informal de- scription of the question: Can writer without true visual supervision (a language model), promote the training of a painter without true textual supervision, (a visual model)? Our answer is 'Yes'.
To showcase how one modality model pro- motes the training of the other models, we here provide a preliminary experiment, which is simple yet intuitive. We consider case that a language model [L] synergizes the training of visual model [V]; denoting it as [L-+V] case. Specifically, a pretrained BERT model is used to promote the training of a ResNet model for the CIFAR-10 classification task (Krizhevsky et al., 2009). We used the text prompt "This is about Class #. ". providing imper-
(a) [V] (Ace: 96,01) (b) [L-V] (Ace: 97.09)
Figure I: t-SNE visualizations between single (left) and multimodalityy (right) on CIFAR-10 feet supervision loosely associated with the image. Herein, # indicates the Arabic class index, which does not provide semantic information of a given image. When utilizing this imperfect tex- tual representation in training of the visual model, we surprisingly observe meaningful performance gains. As shown in Figure I, when compared with the dingl-modality [V] case, our case [L-V] shows the well-clustered feature representations and accuracy gain'. Noteworthy, demonstrates that language models help visual models, even with imperfect prompts, i.e., writer indeed helps painter. Our work aims to provide a theoretical foundation for understanding how happens, and further proof-of-concepts in variety modalities, architectures, and tasks.
In this paper, we establish a theory that makes us envision how one modality promotes others with- out paired supervision. The key of this claim is that there exists an interpolated representation of two single-mo modalltyy representations that outperforms the two (referred to Theorem 1). Furthermore, can be well-approximated without paired supervisions from the given modalities (referred to Theo- rem 2). As the proof of concepts of our claim, we select the most widely-used modalities, i.e., vision [V], language [L., and audio [A], to empirically evaluate the synergies between the three modalities. When using notation, [M, - M,J. indicates that modality M, promotes the training of modality Mj. we mainly confirm that language models promote the training of visual models, i.e., [L-V]. Moreover, we find that visual models or language models aid the training of audio models, and vice versa, i.e., [V-A], [A-VV], [L=+] and [A L. Noteworthy, our theoretical claim not limited to particular modalities, which broadens the fundamental understanding of the synergy between differ- ent modalities. Also, our work offers the opportunity to utilize the descent single-modalit models to enhance other modality models, which strongly relieves the crucial demands of paired supervision of the current multimodal learning.
2 RELATED WORKS
2.1 VISION LANGUAGE MODEL
Vision and Language are among the most common modalities in deep learning, driving advance- ments in both empirical and theoretical perspectives. These developments led to foundation models like Transformers for language (Vaswani et al., 2017: Devlin et al., 2018: Liu et al., 2019) and Vision Transformer (ViT) for vision (Dosovitskiy et al.. 2021; Steiner al., 2022), enabling co-training with aligned datasets. A key example is CLIP (Radford et al., 2021), which contrastively learns latent features by maximizing similarity between matching vision-Languaan pairs while increasing separation for non-matching pairs. Building on this, methods have emerged that incorporate label
Full training details are descrihed in Section 4
2

---

Published as a conference paper at ICLR 2025
spaces (Yang et al., 2022) or hard negative samples (Robinson et al., 2021; Li et al., 2021) to improve representation learning. However, these approaches require high-quality, perfectly aligned datasets. Our work overcomes this limitation by showing that even simplified or loosely aligned information can significantly enhance performance.
2.2 TRANSLATION OF LANGUAGE INTO VISION MODALITY
Recent advancements in vision- language foundation models have extended their applications across modalities. Large Language Models (LLMss), such as GPT-3 (Brown et al., 2020), and the LLaMA series (Touvron et al., 2023a;b; Dubey et al., 2024), have achieved otate-of-the-art results in tasks like reasoning (Talmor et al., 2019) and knowledge retrieval (Kwiatkowski et al., 2019), primarily through prompt engineering. Building on this, several approaches have employed LLMs trained solely on language data to assist in vision tasks (Sharma et al., 2024), using their capabilities to generate new prompts or serve as auxiliary tools. Conversely, our approach focuses on utilizing latent features derived from language modality, even when imperfect data, without relying on the extensive capabilities of LLMs to perform vision-related tasks. This enables a more targeted and efficient use of cross-mode guidance.
2.3 MULTIMODAL LEARNING
Several approaches have moved beyond vision-language paradigms address wider range of multimodal tasks, both empirically and theoretically. On the practical side, methods like MFAS and Multibench (Liang al., 2021; Pérez-Rúa al., 2019) employ fusion networks (Zadeh al., 2017; Tsai et al., 2019) to combine inputs or latent features from multiple modalities, enhancing multimodal learning. Additionally, related research has explored approaches for handling missing or unpaired sets in multimodal learning (Girdhar et al., 2023; Mizrahi et al., 2024). These methods include training separate classifiers for each modality (Kim & Kim, 2025), training prompts (Lee et al., 2023), or utilizing a shared encoder (Wang et al., 2023) to address missing modalities.
However, prior works also contain few key limitation. Recent methods in fully supervised or unpaired settings often aim to improve multimodal model performance, typically by training or fine- tuning both modalities (Shukor al., 2023). contrast, our approach utilizes latent representations from a well-trained modality model to enhance the training of another modality model from scratch. Moreover, synergy between modality models remains simultaneously underexplored from both em- pirical and theoretical sight. Thus, our approach addresses this gap by enabling single-modality models to support the training of others empirically, and provide theoretical framework based on interpolated representations to explain their synergy.
THEORETICAL PERSPECTIVE AND METHODS
3.1 BASIC NOTATIONS AND A SKETCH OF MATHEMATICAL CLAIMS
Let us denote M; and My as two different modalities. For inputs, let x and x indicate the paired inputs from modalties M; and Mj respectively. In additon, 9: X - 2 and g, X'' + 2 are two representation models that map an input to latent space, for the respective modalities. Also, h; Z' + y is the hypothesis from embedding space to the label space y for the respective modaltiy. Here, representations and hypothesis are independently trained on each modaltiy, without joint training.
In the context of multimodal learning, it widely ac- cepted to assume the existence of true latent space Z*, which is the optimal representation across both modalities M; and M, (Huang et al., 2021). As brief sketch, we show that there exists an interpolated latent representation space z that shows smaller distance Z², leading to outperform the two Z' and Z (referred to Theorem 1). It implies that we can find better representation by interpolating two rep- resentations from different modalities. However, the first
dog Gnage Scmeacicaced
representations, i.e.,
Figure 2: A conceptual sketch of our claims in [] case
3

---

Published as a conference paper at ICLR 2025
claim is limited in assuming perrectly-paireed imputs z' and æ² which draws extensive costs in tations of multimodal learning. To this end, we argue that even given with an imperfect or restricted embedding & E 20 (not correctly-paired with 2²), there exists an interpolated representation 2 between 2 and 21, which is closer to the true representation, leading to surpass the two, i.e., Z and 2 (referred to Theorem 2). Figure illustrates brief sketch of our claims. It means that imperfect samples from modality M, can promote the training of the superior representation that promotes the performance of modality M,; enabling the [M, + M,] case. In the following part, provide the formal description.
3.2 How ONE MODALITY MODEL SYNERGIZE THE TRAINING OTHERS?
We begin by introducing the distance metric between two distributions Pi and P2 as follows:
Definition (2- (2-Wasserstein Distance (Villani al., 2009)). 2- Wasserstein distance between 2 dis- tribution P1 and P22 is defined as:
inf TEr(Pi,P) Pa) Ve d(=1, 22) (2),
1/2
W2(P1, P2)
where I(P1, P2) is all set of distribution (couplings) of Pi and Pz.
We use 2-Wasserstein Distance to represent the distance between the probability density functions of the extracted features from different representations, described below:
Definition 2. Let zt E Z', z Z', and =* 2 denote the lalent spaces of the i.j-modality, and the true latent space, where z" represents the optimal representation across both modalities. z² and z are assumed be correctly-paired, which means that they represent the same underlying concept but in different modalities (e.g., i.for image and for text). Furthermore, let P.. Pj. and P. are their corresponding distributions, i.e., x² ~ Pi, z² ~ Pa, and z" ~ P.. The distances between the distributions are then defined as follows:
= W2(P,, P.), A = W2(P..P;) and Dij := We(Pe.P)).
where A, Dj. and Dij are positive real numbers.
Let us define interpolated representation between the representations of two different modalities: Definition 3. Let latent space 2² z is interpolated representation between the latent repre- sentation spaces of the i,j ij-modalities with interpolation coefficient a E [0, 1]. as follows:
z = {2t (1 (1- a)= + azz 2' E Z 2 E Z').
(3)
In addition, let Ps denote the distribution of 2t, and 2- Wassersteii distance between Pa and P. is A₁ == W2(P., Px).
Assumption 1. Let Aij not converge to O, i.e. Aij >
Assumption 1 implies that even if modalities M, and My are similarly distant from the true latent space, the distance between M, and M, remains significant. Intuitively, while two modalities may represent similar information certain contexts, their overall representations can differ markedly. Furthermore, they may contain distinct information. For example, modality [L] can effectively convey representations about "questions," whereas modality [V] may hard to provide such represen- tations. Thus, assuming that Aij > () is reasonable and justifiable consideration.
In the following theorem, we present the closed form solution of the interpolation coefficient a", which makes 2 be closest to Z
Theorem 1. The optimal a² that minimizes A is formulated as follows:
A
A + A2
(4)
2
Moreover, the resulting interpolated representation satisfies De A; and Ae Aj.

---

Published as a conference paper at ICLR 2025
Corollary 1.1. The optimal of is bounded as follows:
A
iA
A + A
a²
(5)
4 A
0,
otherwise
A
For the proofs of Theorem and Corollary 1.1, please see Appendix A.I.
Remark 1.1. (The behavior of x²) a" is strongly influenced by the quality of modalities. For an instance, let us discuss what happens when A changes while fixing A, Specifically, 1) when A, A.. Le., modaltiy Mj is superior modality M,, the optimal ox will be greater than 0.5, i.e. a' > 0.5, implying that Pk tends to be closer to P... Conversely, 2) if A, < Aj.. a² < 0.5, indicating that Ps shifts closer Pt. Finally, 3) when D. Aj, the optimal a* equals 0.5. is convincing that the behavior of o" shows that the superior interpolated representation forms near to the better modality. Further discussions are provided at Appendix A.I.
Remark 1.2. (The condition of the synergy) For a scenario of utilizing Mj training M,, is crucial to judge how beneficial Mj is in promoting the training of the Mj. If modality Mj does not bear information on the true latent space, where D; > D,, this suggests that the M, much far from both the true latent space and the M,. Thus, leveraging M, is not effective in training M., and makes to x²* close O, which leads to the single-modality training of M,. Thus, Assumption is critical to ensure that information from the Mj can be effectively utilized.
As aforementioned, obtaining exactly paired datasets in real-world scenarios is challenging. Theo- rem assumes the perfectly matched pairing of 2; and 2, with common concept. Let us draws setting of 'imperfect' or 'restricted' pairing between modalities into our theoretical framework.
Definition 4. Let 3) 2i the imperfect latent space of the j-modality, with Po as correspond- ing distribution. The gap 8 between 2 and 2 defined as:
8 == W2(3, P.).
(6)
Additionally, the distance from P, to the true latent space Pa and Pi are denoted as respectively:
Â = (2,,,,P,) and Å. == Pz)
Based on Definition of imperfect representation, we rephrase the Theorem to provide the fol- lowing theorem:
Theorem 2.
Let z 2* be the interpolated latent space between the Mj and the restricted M defined by the interpolation coefficient a as: 2¹ = = = (1-a)z' + azt 1² E z 2 E 2),, where PR represents its distribution. Let the 2- 2-Wasserstein distance between Pk and Pa be denoted as (PP,., Âr. Then the optimal 6 that minimizes Åu is formulated follows: Å
A
A + A
(8)
2
Moreover, the resulting interpolated representation satisfies Å D. and Å s A, Corollary 2.1. The optimal & is bounded as follows:
A
A + A
+
8
â'
A
0,
otherwise
A
A
For the proofs of Theorem 2 and Corollary 2.1, please see Appendix A.2.
5

---

Published as a conference paper at ICLR 2025
Remark 2.1. (6 does not hinder the synergy) Although 23 ~P deviates from Pj. it still represents the latent space of the M3. We can extract an imperfect feature representation from Pj by giving imperfect input to the modality Mj. This allows 2 exist in the distribution P, Consequently, 2 is closer to part of the latent space of the Mj than to that of the M, or the true latent space. Therefore, additional gap 6 in Equation 6 is unlikely to significantly impact the determination of the optimal â", will be generally much smaller than both A and Aj. It stems for ö does not hinder the synergy between two modalities, i.e,, the modality Mj can promote the training of M,, even with an imperfect representation.
Remark 2.2. (M, indeed helps the training of M2) In training Mj. we can find a superior repre- sentation 2 by utilizing imperfect feature representation from M,. When revisifing the conceptual sketch in Figure 2 and the preliminary experiments on CIFAR-IO shown in the Introduction, an imperfect textual description, i.e., "This is about Class $. works as a² promotes the training of visual models by finding the interpolated representation.
3.3 METHODOLOGY: TRAINING MODALITY M, BY LEVERAGING MODALITY Mj
Based on our theory, we thus propose training method for one modality M, by leveraging modality Mj with sampling imperfect 2j from the representation space of M,, Let us consider the scenario for training modality M, by leveraging modality Mj. i.e. [M, - M,,, without loss of generality.
We extend our notation by introducing uppercase superscript to denote the data modality. Specifi- cally, let St {(xnn ym ) representing the dataset for the M, to be learned, where x denotes the input data and Vin its corresponding label, where M is the number of data points sampled. Sim- M
ilarly, let S = (>e, m=1 represent the sampled set of imperfect samples of the My. This yields the latent vectors 2/1 = 9, (x2)) E 21, as introduced in Definition 4, where 9j(-) is the pretrained model function associated with the My. St and S are sampled subsets from their respective full modality datasets, D' and Di. In last, the primary objective is to minimize risk through Empiri- cal Risk Minimization (ERM). We then define two associated empirical risks by the following loss functions:
Classification Loss: Lota
E (xe), Mh ):
[Sor(h g(x()) wmn)]
Latent Loss: C2
E (xc% Min Six 20
[11g(((m)) 2/ [l
In this approach, the classification loss Lete is utilized to optimize the learning of the M, while latent loss L. is treated as regularization term, enforcing alignment of features from the M,.
We here propose straightforward loss formulation, which allows find the interpolated repre- sentations between two modalities. We linearly combine the classification and latent loss terms: Frotal (1 Cets + aLz,, to find optimal gx g(x')) larg min Ciotal A psuedocode of the learn- ing procedures is given in Appendix B.2. Lastly, describe how to obtain imperfect feature 2 for each modality in the following section, including vision, language, and audio.
4 EXPERIMENTAL RESULTS
In this section, we provide an overview of the experimental results, along with detailed descriptions of the datasets, models and additional experimental settings.
4.1 EXPERIMENTAL. SETTINGS
Datasets For the main experiments, we test on the ImageNet-1K IK dataset (Krizhevsky et al., 2012) for visual tasks as the case of [L- For further experiments in the multimodal setting, we employ the IEMOCAP (Busso et al., 2008) and AVMNIST (Liang et al., 2021; Li et al., 2023) datasets. IEMOCAP includes [A+L+Video] modalities, where we specifically focus the [A+L] subset for our experiments. We performed experiemnts on both direction, [L~A] and [A-L]. For AVM- NIST, which contains [A+V] modalities, we followed the preprocessing steps outlined in CentralNet
We fully descrihe how can he done in the empirical testing in Section and Appendix B.

---

Published as a conference paper at ICLR 2025
Table Classification results on ImageNet- IK and evaluation benchmarks (OOD and robustness)
Model [L-
IN
V2
Rend.
Sketch
A
Style.
C (4)
ResNett-50 (reproduced)
77.83
66.20
39.28
27.35
6.44
8.59
66.01
BERT (Devlin al., 2018)
78.41
67.10
40.38
28.19
8.47
9.64
64.96
RoBERTa (Liu et al.. 2019)
78.54
67.30
40.92
28.78
8.25
9.19
65.32
ViT-B/32 (reproduced)
75.04
62.02
40,31
27.34
9.23
16.56
55.45
BERT (Devlin et al.. 2018)
76.59
63.37
41.28
28.53
11.31
18.11
53.28
RoBERTa (Liu al., 2019)
76.75
64.00
41.81
29.50
11.55
18.75
52.95
ViT-B/16 (reproduced)
80.07
68.60
44.72
31.22
24.20
18.81
51.21
BERT (Devlin et al., 2018)
81.62
70.07
45.72
33.13
25.12
20.31
49.27
RoBERTa (Liu et al., 2019)
81.90
70.55
45,41
33.19
26.89
19.93
48.51
Table 2: Classification results IEMOCAP and AVMNIST datasets each cases of [M, + M,
Datasets
Model [L-+A]
Accuracy
Model [A L+
Accuracy
Wav2Vec2f (Ravanelli et al., 2021)
$9.46
BERT (Devlin et al., 2018)
55.81
IEMOCAPIT
+ BERTTB (Devlin el al., 2018)
60.44
WWayNeee-B (Baevski et al., 2020)
56.49
BERT-L (Devlin al., 2018)
61.20
Wav2Vec2-L (Baevski et al., 2020)
56.05
Datasets
Model (V-A]
Accuracy
Model [A=+V]"
Accuracy
Audio Model (Li et al., 2023)
41.28
Vision Model (Li al., 2023)
65.18
AVMNIST
ResNeet-18 (He al., 2016)
42.08
Wav2Vec2-B (Baevski al., 2020)
66.37
ResNet-34 (He al., 2016)
42.44
War2Vec2-L. (Baevski et al., 2020)
66.69
SpeechBrain (Ravanelli et al., 2021) experimented with 4 out of 6 labels; used the all labels. tt; Owing transformer-type model requires numerous data, we fine-tuned the pretrained model.
: Since the audio data in AVMNIST is based on spectrograms, we use the original raw audio data prior to its conversion into spectrogram.
(Vielzeuf et al., 2018), transforming raw audio into 112 x 112 spectrograms and utilizing 28 x 28 PCA-projected MNIST images. Similar IEMOCAP datasets, we conducted experiments both directions, [V A and [A-V].
Models For the [[-+V] case with the ImageNet- dataset, we employed modern architectures,
including ResNet50 (He et al., 2016), ViT-B/32, and ViT-B/16 for [V] modality. To incorporate additional information from text prompts, we utilized pretrained BERT (Devlin et al., 2018) and
RoBERTa (Liu et al., 2019), two extensively used transformer encoders for embedding the [L] modality. We use the "large" size versions on both encoders. For the [[I+A] and [A-I.] cases with IEMOCAP datasets,
employed Wav2Vec2 (Baevski et al., 2020) model with SpeechBrain (Ravanelli et al., 2021) configurations for [A] modality, alongside pretrained BERT-B and BERT-L for the [L] modality. For the [V-A] case with AVMNIST experiments, we used the audio encoder from AGM (Li et al., 2023), enhanced with an additional classification layer for the [A] modality, and pretrained ResNet- 18 for the [V] modality. In addition, for the [A- case with AVMNIST, we used the original pretrained Wav2Vec2-B/L. model for the [A] modality. Especially, in Table 2, "-B" refers "base" size of model, while "-I." refers to "large" size configuration. For [M, - M,). all model parameters of M, modality are frozen.
Additional Settings (how to get 23) For the [L-V] case with the ImageNet-IK IK dataset, we ob- tain z by employing simple prompt engineering that provides imperfect and restricted information for the language modality. Specifically, prompts like "This s is about Class #. are used, where # random number unrelated the actual class label, ensuring imperfect supervision for the language modality. For the [L~A] case with IEMOCAP, & is generated using prompts like "This is about Emot i on #. ", where # is also a random number unrelated actual labels, helping audio classification on the IEMOCAP dataset. We exclude the paired [L.] modality data in IEMOCAP. In the [A case with IEMOCAP, we added Gaussian noise to audio data and randomly shuffle to represent É) thereby promoting sentiment classification in the [L] modality. For the [V-A] case with AVMNIST, we use randomly shuffled images from AVMNIST as 2 in audio classification tasks. For the [A case in AVMNIST, 2 is generated by adding Gaussian noise to the audio data and apply shuffling, similar to the [A case, to assist the classification task on PCA-projected MNIST images. Detail description of implementing ş is provided on Table 5.
7

---

Published as a conference paper at ICLR 2025
IEMO. [L-A]]
WD
TEMO. [A LL]
WD
AVMN. (V -AJ
WD
AVMN. VV+
WD
Pw)
0.494
P.)
0.965
W2(PA.Pa)
0.025
Wa(Pv, Px)
0.908
Ws(P1,Pe)
0.141
Wa(PA, Px)
0.460
(e)PvPPe)
0.754
Wa(Pa, Px)
0.502
PL)
0.977
PA)
1.005
Wi(PA: Pv)
0.790
W2(Pv, Pa)
0.954
Figure 3: t-SNE visualizations and Wasserstein Distances (WD) across multimodal datasets
Implementation For the [I.-vV case in ImageNNe-IK K classification task, we followed hyperpa- rameter settings from the AugReg- ViT (Steiner et al., 2022) in training ResNet50, ViT-B/32, and ViT
VIT-B/16. For multimodal datasets, i.c. IEMOCAP and AVMNIST, particularly in the cases of [L-A], [A-L], [V-A] and [A- we employed customized hyperparameter settings for each We consistently used 30 training epochs with Adam optimizer (Kingma & Ba, 2015). We omitted any data augmentations. Lastly, we applied a = 0.5 for the [L-V] case, and a 0.3 for the other cases. Additional details are provided in Appendix B.
4.2 RESULTS
Main Results Our main results are two parts: Table with ImageNet- 1K and Table 2 with multi- modal datasets, i.e., IEMOCAP and AVMNIST.
According to Table 1, the results for the multimodal learning case, especially [L-V] case, high- lights the outstanding performance of our approach on ImageNet- 1K (IN). where achieved im- provements between approximately + +1.5% and +2.0%. Notably, our approach also demonstrates significant improvements across additional evaluation datasets, including ImageNet-V2s V2, out-of- distribution (OOD) datasets such ImageNet-Rend. (Hendrycks et al., 2020), ImageNet-Sketch (Wang et al., 2019), and ImageNet-Style. (Geirhos et al., 2019), achieving performance gains be- tween +1.0% and +2.0% Furthermore, our approach also excels in robustmess benchmarks, achiev- ing improvements up to +2.6% on adversarial examples from ImageNet-A (Hendrycks et al., 2020) and 2.54 on corrupted images from ImageNet-C (Hendrycks & Dietterich, 2019). Surprisingly, language models clearly promote the training of visual models, i.e., writers indeed help painters. Notably, the synergy shows consistent gains even OOD, adversarial, and corrupted samples. It emphasizes that language models also facilitate visual models to acquire representations, which are well-generalized on wide range of visual data distributions.
For more cases with IEMOCAP and AVMNIST in Table 2, our approach also leads to performance gains across various tests. the IEMOCAP experiments, shows considerable gain of approxi- mately +1.6% in the [L-+A] case and +0.6% gain in the [A-L] case. For the AVMNIST testing, performance improvement is around + 1.2%% in the [A [A] case and + 1.44% the [A+L]] case.
Consequently, we confirm that the three modalities--iinn mutually enhance each other's 's training in ways that are not easily anticipated. This underscores the potential for significant performance improvements when multiple modalities are effectively integrated and demonstrates the broad applicability of our claims across different types of modalities. Moreover, emphasize that claims have been validated across numerous existing deep model architectures, where this validation highlights the model aagnosic nature our approach, making applicable to broad spectrum of scenarios.
Language, and Audio are are shown to
ANALYSIS
5.1 ANALYSIS AND REPRESENTATION VISUALIZATIONS
Wasserstein Distance Between Modalities Our hypothesis suggests that the Wasserstein distance between the latent feature distributions of individual modalities should exceed that of an adaptively
8

---

Published as a conference paper at ICLR 2025
VIT-8932 BERTTL
VET-WVEz RARERTES
BERT + Ward
Viioo Model WardVee3-L
Latent Lows
VIT-WIIs BERT--
Latent Lors
WardYe2 HERT-B
Latent Lass
Audio Medel ResNeet-5
VIT-RVI6 ROBERTE
WardWe22
Audio RoNNc134
0.5
100
Egoch
Epoch (b) TEMOCAP L.
Epoch
(a) ImageNet-1K Cs
(c) AVMNIST Lz
Figure 4: Validation Latent loss L3 across the entire datasets
Table 3: Evaluation across the different value of
Table 4: Comparison: 31 vs. 27
Accuracy
Model [L-V]
Datasets
0.1
a 0.3
0.5
a
ResNet-50
78.54
78.61 (+0.07)
RoBERTa
JEMOCAP [L->A]
59,46
60.34
61.20
59.87
59.93
ViT-B/32
IEMOCAP [A [A+
55.81
55.51
56.49
54.19
55.90
76.75
76.99 (+0.24)
+ RoBERTa
AVMNIST [V-A]
41.28
42.03
42.44
41.77
41.79
VIT-B/16
AVMNIST [A -V]
65.18
65.03
66.69
64.77
64.76
+ RoBERTa
81.90
82.54 (10.64)
trained distribution, PR. which ideally lies between the latent spaces of the given modalities. Figure 3 illustrates this concept. In the test of [M, - M,] scenario, the denote latent vectors =f from the M, modality,
represent M, modality latent vectors z!. As expected, t-SNE visualizations largely represent that &* resides between the latent spaces of M, and Mj. with asmaller Wasserstein distance to each modality (see W2(-, Pk) terms), while individ- ual modalities are shown to be distant to each other. These findings visually illustrate our hypothesis, coinciding with the claim of 'interpolated' representations. It further confirms that our Liotal em- pirically serves as an adequate objective function to acquire the superior interpolate representation in between individual modalities.
represent {** 1M m= and
The Convergence of Latent Loss We here show how the latent loss, i.e., Lz, which reflects how much the interpolated representation moves close to the modality M,, behaves during the training. As shown in Figure 4, the loss consistently decreases throughout all cases. This result highlights the narrowing gap between the latent vectors from modality M, and the representation vectors derived from the input data. By minimizing this gap, models successfully exploits simplified yet informative latent features 2) thereby improving overall performance. As Mean Squared Error (MSE) is em- ployed, the magnitude of the gap may vary across experiments due to dimensions of representation vectors. To account for this, we applied normalized latent loss to observe clear convergence for all cases. Despite some fluctuations in certain cases, the loss consistently decreases and converges to a value close to, but not exactly, zero due to interpolation.
5.2 ABLATION STUDIES
The Effect of a Due to the fact that a plays a crucial role in finding optimal latent space, we evaluated how variations in influence performance on multimodal datasets. We tested various values of a = {0.0, 0.1, 0.3, 0.5, 0.7), and the corresponding results are shown in Table 3. We observe clear performance degradation when biased with too small or large values is used. It demonstrates that the biased cases, which tend to strongly rely on one single modality, do not show synergy. Among candidates, importance of balancing the contribution of each modality. When reminding t-SNE visualizations in Figure 3, with moderate value coincides with the spatial position of the representation the middle of two individual representations.
0.3 consistently shows the best accuracies, highlighting the
The Usage of Paired Supervision zd vs. Z': Rather than using imperfect 84, we investigate how much gains would be further achieved when using the perfectly matched 21, In Table 4, we tested the ImageNet- IK (IN) cases, where the column with '3j' refers to the numbers Table I of the main experiments, and the column with "z'" is done with perfectly matched supervision in the [T.]

---

Published as a conference paper at ICLR 2025
modality. As shown in Table 4, although the performance slightly improves, the gains mini- mal. This result aligns with our hypothesis, indicating that 6 from Theorem has a limited effect on performance, supporting
hypothesis. Also, the result emphasizes that approach signifi- cantly relieves the cost of pairing perfectly matched supervision across modalities while achieving comparative performance with the ideal case with paired supervision.
FURTHER DISCUSSIONS AND LIMITATIONS
Innovating Multimodal Learning: In prior works, many multimodal learning methods have relied on paired-datasets for training. This contrasts with human learning, which often occurs without the need for precisely paired object from different modality. Similar to human's perspective, our work seek to overcome this limitation, demonstrating that minimal or even imperfect supervision from different modalities can still enhance learning in the primary modality. Consequently, our approach suggest further advancements in technical perspective, enabling the effective utilization of multimodal settings even when only a single modality is available.
Discovery of Unexpected Synergy Between Modalities: It is commonly assumed that not every modality can effectively assist another. The prevailing notion is that unrelated modalities may not provide meaningful assistance to one another. However, our experiments on [V-A]] and [A-VV] reveal that cross-modality interactions, even between seemingly unrelated domains, can lead to sig- nificant performance improvements. These results demonstrate that integrating seemingly unrelated modalities can still yield benefits, uncovering hidden correlations and unexpected synergies between them. This approach opens new possibilities for exploring multimodal combinations that were pre- viously considered non-beneficial, such as integrating language with signal- bggnal-based sensory data. Ex- panding the scope of multimodal learning may reveal hidden synergies between multiple modalities, leading to improved overall performance.
Broad Impacts on Wide Range of Tasks: Recent works has focused on addressing the general- ization problem, such as -iitt--distriuution (OOD), adverserial, and robustness. We validated that our method not only achieves strong performance on the ImageNet- IK but also shows significant improvements in OOD, adversarial, and robustness scenarios, ings suggest that our approach could be extended to further tackle the generalization problem by incorporating additional guidance from imperfect data across other modalities.
shown in Table 1. These find-
While our experiments demonstrate the effectiveness the proposed approach, certain limitations remain. One key challenge is scalability, both in terms of computational feasibility, and the other is theoretical extensions to multiple modalities.
Limitations:
Primarily, due resource constraints, it is limited to conduct large-scale evaluations larger archi- tecture size, leaving empirical model scalability of our method an open challenge for future research. However, is important to emphasize that our theoretical framework is not inherently restricted by model size. We expect the core principles of our approach to generalize effectively to larger models.
The other limitation, as mentioned on the previous discussion, is that our method has been evaluated only on the paired modalities. Extending it to incorporate more than two modalities could unlock further performance gains for individual modalities. Nonetheless. aligning latent vector dimensions across multiple modalities introduces additional complexity, which may impact performance and requires further investigation.
7 CONCLUSIONS
We presented both theoretical and empirical frameworks demonstrating that one modality with im- perfect representation can still enhance learning in the other modality. Our results and extensive analyses support proposed hypotheses and reinforce theoretical foundations of this approach. No- tably, showed that paired supervision between datasets is unnecessary, as weakly related super- vision or even mismatched setting across modalities can still lead to improvements. In last, we then propose future research explore more complex multimodal settings, such as leveraging more than two modalities or scaling to larger models, which may drive significant advancements in this field.
Additional details providing more precise information are included in the Appendix. B.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2020-11201336, Ar- tificial Intelligence Graduate School Program (UNIST)), (No. novative Human Resource Development for Local Intellectualization program), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS- 2024-00459023).
In-
REFERENCES
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 12449-12460, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 1877-1901. Cur- ran Associates, Inc., 2020. URL https: http:s::///ocpeepeeeeeee http://cceeeeeeeee..e..e.......e www.eeewwww..eeeeeeeeeeeee paper/2020/ file//ill/ /tt.a///////00000000060110011110
files/
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. IEMOCAP: Interactive Emotional dyadie MOtion CAPture database. Language Resources and Evaluation, volume 42, pp. 335- 359. Springer, 2008.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings the IEEE/CVF conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshps), pp. 702-703, 2020.
Jacob Devlin, Ming-Wey Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume i (Long and Short Papers) (ACL), pp. 4171-4186, Minneapolis, Minnesota, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N19-- 14233333. org/Ni /N///1/6 9- 423.
URL https: http://aaaaaaohhloooooo
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference Learning Representations (ICLR), 2021. URL https: //pp/ppeevviiieee net /ttpeeef///////ee/eeee/eeeeeetee
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The LLaMA 3 herd of models. In arXiv preprint arXiv:2407.21788. 2024.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019. URL https: https:///peeeeeeeeeeeeee net
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15180-15190, 2023.
11

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A crit- ical analysis of out-of-distribution generalization. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), volume 2, 2020.
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 10944-10956, 2021.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. Position: The platonic repre- sentation hypothesis. In International Conference on Machine Learning (ICLR), 2024. URL httpps://ope //tppre///peereiieww net http://nnnneeeee..eooooooooottaa /oo.mmmdd---tTTyo0em
Zohar Jackson, César Souza, Jason Flaks, Yuxin Pan, Hereman Nicolas, and Adhish Thite. dakcoosskfffre-ppkkerddiiidaaaed vl. 0.8, 2018.
Donggeun Kim and Taesup Kim. Missing modality prediction for unpaired multimodal learning via joint embedding of unimodal models. In European Conference on Computer Vision (ECCV), pp. 171-187. Springer, 2025.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, (ICLR), 2015. URL http: / arxiv.. org/ abs/ / 1112. 6980.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Advances Neural Information Processing Systems (NIPS), vol- ume 25, 2012.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. In Transactions of the Association for Computational Linguistics (TACL), volume 7, pp. 453-466, 2019.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basse learning applied to document recog- nition. Proceedings of the IEEE, volume 86, pp. 2278-2324, 1998. doi: 10. 100000
Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen Yu Lee. Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14943-14952, 2023.
Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou. Boosting multi-modal model performance with adaptive gradient modulation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 22214-22224, 2023.
Jian Li, Yong Liu, Rong Yin, Hua Zhang, Lizhong Ding, and Weiping Wang. Multi-class learning: From theory to algorithm. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances Neural Information Processing Systems (NeurIPS), volume 34, pp. 9694-9705, 2021.
12

---

Published as a conference paper at ICLR 2025
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal represen- tation learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 2021, pp. 1. NIH Public Access, 2021.
Haotian Liu, Chumyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2024.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining 2019.
approach. In arXiv preprint
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations (ICLR), 2017. URL https: //pp///v/e//......./ommome. net / forum? 8 9Scxx.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Con-
ference on Learning Representations (ICLR), 2019. URL https:///peerrevien
forum?id-Bkg661CCq(
Guillaume Mahey, Laetitia Chapel, Gilles Gasso, Clément Bonet, and Nicolas Courty. Fast optimal transport through sliced generalized wasserstein geodesics. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2024.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksäi Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. International Conference on Learning Representations (ICLR), 2018. URL https::/openrewe
net.
David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2024.
Ziqi Pang, Ziyang Xie, Yunze Man, and Yu-Xiong Wang. Frozen transformers in language models are effective visual encoder layers. In International Conference on Learning Representations (ICLR), 2024. URL hitps: htwttps://pperevviieeee net
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-perfore regehperformance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Juan-Manuel Pérez-Ráa, Valentin Vielzeuf, Stéphane Pateux, Moez Baccouche, and Frédéric Jurie. Mfas: Multimodal fusion architecture search. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6966-6975, 2019.
Gabriel Peyré and Marco Cuturi. Computational optimal transport. In Foundations and Trends in Machine Learning, volume 11, pp. 355-607, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning (ICML), pp. 8748-8763. PMLRR 2021.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju- Ju-Chieh Chou. Sung- Lin Yeh, Szu- Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. SpeechBrain: general-purpose speech toolkit, 2021. arXiv:2106.0444
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning (ICML), pp. 5389- 5400. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2025
Joshua David Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations (ICLR), 2021. URL https: http:////wpp.eee...eeeeeee http://www...tttrrrroaaaaooooCOo
Pratyusha Sharma, Tamar Rott Shaham, Manel Baradad, Stephanie Fu, Adrian Rodriguez-Mungn Munnz, Shivam Duggal, Phillip Isola, and Antonio Torralba. A vision check-up for language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR),
pp. 14410-14419, 2024.
Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. UnIVAL: Unified model for image, video, audio and language tasks. In Transactions on Machine Learning Research (TMLR), 2023. URL https: / opwnreview..w....///ee net/ forummiddddaa whopppp..pppppp.oo
Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. In Transactions on Machine Learning Research (TMLR), 2022. URL https: /tpp/reeeeeee...eeeeeeeee net /artm?Errm i 4nd 1Kc..
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA:: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4149-4158, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/N1- 1421. URL https: ttttps://aclaathhttoe halaaaaannntoogoyoooooaaaoooollt org N1 1421/.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint urXiv:2302. /3971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. LLaMA 2: Open foun- dation and fine-tuned chat models. In arXiv preprint arXiv:2307.09220 2023b.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philipee Morency, and Ruslan Salakhut- dinov. Learning factorized multimodal representations. In International Conference on Learning Representations (ICLR), 2019. URL rуςеQsд9ΚX.
net wwwrmoormm ?id=
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems (NIPS), volume 30. Curran Associates, Inc., 2017. URL
http://ccceednnnn.....ooeoeeeeee nwwrips........... cc ftsppr__fllesssppee///111 fitp://Eee2255766611110333322213
Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie. Centralnet: multilayer approach for multimodal fusion. In Proceedings of the European Conference on Computer Vision
(ECCV) Workshops. pp. 0-0, 2018
Cédric Villani al. Optimal transport: old and new, volume 338, Springer, 2009.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Systems (NeurIPS). pp. 10506-10518, 2019.
Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, and Gustavo Carneiro. Multi- modal learning with missing modality via shared-specific feature modelling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15878-15887, 2023.
Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR). pp. 12695-12705, 2020.
14

---

Published as a conference paper at ICLR 2025
Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Uni- fied contrastive learning in image-text- teageeeettllbell space. In Proceedings of the IEEE/CVE Conference Computer Viston and Pattemn Recognition (CVPR), pp. 19163-19173, 2022.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Ten- sor fusion network for multimodal sentiment analysis. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference on Empirical Methods in Nar- ural Language Processing, pp. 1103-1114, Copenhagen, Denmark, September 2017. Asso- ciation for Computational Linguisties (ACL). doi: 10. 1115111111111 / //aclanthology orgg/17--11111//
URL https:
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empir- ical risk minimization. In International Conference on Learning Representations (ICLR), 2018.
URL htwps::pp/.................tt / /wppnreeeeeeeeeeeeeeeeeeeeeveeee
net./ forum? d=r1Ddp1-Rb.
Xue Zhang, Ming-Jiang Wang, and Xing-Da Guo. Multi-modal emotion recognition based on deep learning in speech, video and text. In 2020 IEEE 5th International Conference on Signal and Image Processing (ICSIP). pp. 328- 328-333, 2020. doi: ALLL:///SSII19999200000001933330
Andy Zhou, Jindong Wang, Yu-Xiong Wang, and Haohan Wang. Distilling out-of-distribution ro- bustness from vision-language foundation models. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https: wpw.revieneeeeeeeeeeeee net http://eee///oooeeeee/eeeeeeee forrmm? iidiiiiii idwwwp3BBSS....
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on Computer Vision and Patlern Recognition (CVPR), pp. 1681 16816-16825, 2022a
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. In International booktitle of Computer Vision (IJCV), volume 130, pp. 2337- 2348. Springer, 2022b.
15

---

Published as a conference paper at ICLR 2025
A
MATHEMATICAL DETAILS FOR THEORETICAL FRAMEWORKS
A.1 PROOF OF THEOREM 1 AND COROLLARY 1.1
Theorem 1, The optimal a' that minimizes As is formulated as follows:
4 + A
24
Moreover, the resulting interpolated representation satisfies As S A. and As SG
Proof. Given that Wasserstein distance is symmetric and satisfies triangle inequality (Villani et al., 2009; Peyré & Cuturi, 2019), define the relationship each distribution of latent spaces:
W2(P:, W2(P., P:) + (2,,.,
(10)
and this could be simplified via notations provided on Definitions at Section 3.2:
A - A +
(11)
According to prior works (Villani et al., 2009; Peyré & Cuturi, 2019; Mahey et al., 2024), we can define a new equation as follows about the relation between wasserstein distances:
Definition (Generalized Geodesies (Mahey et al., 2024)). According to the convexity property of 2- Wasserstein distance, squared 2- Wasserstein distance between Pk and P, is bounded by:
(P,,, P.) < (1 a (1 ( a) + +a4 - a(1 a))
a(1 (P))
(12)
(13)
Considering that our primary goal is determine the optimal a, we can express Equation 13 the quadratic function of a,, i.e., f(a) = (1 - aa + a ax11 a) The optimal value of a can then be determined by taking the derivative of the function f(a):
As(a)
A + 3 - (1 - 4 = 0
(14)
da
Changing into the term of a is then:
A7 A3
1+C 2
where C AA)+A,))A - -A)
α- 1
(15)
22
where a² arg (a). This optimal a" remains valid under the Assumption that Dij is bigger than 0, ensuring that a* does not diverge.
still remains to be proven that the optimal value of a" indeed minimizes A and satisfies the condition that A is smaller than both A, and Aj. To establish this, we begin by verifying if As is truly minimized. Let the minimized value of A be denoted as A Assume that 2* ~ Ps lies on the geodesic between z' ~ P: and zÁ P,, representing a projection onto the true latent space.
To demonstrate this, we can apply the Pythagorean theorem. Under the above assumption, A can be expressed as:
A 2
A? (a A ((1 - a))))
(16)
(17)
We can further rearrange the above equations as:
A?
A3 (2a 1)
(18)
"This concept is detailed in the min-SWDD paper (Mahey al., 2024).
16

---

Published as a conference paper at ICLR 2025
From this, solve for optimal a, denoted as a*:
a" 1 A7 Ag 2
(19)
This expression corresponds to the optimal value a" from Equation 4 and satisfies the Pythagorean theorem, which we assumed to hold when minimizing As. This minimized value of A represents the distance between P. and the distribution Pk. projected directly onto the geodesic connecting P. and P.. Moreover, we conclude that the minimized A indeed smaller than both A, and A where At & A as related to Equation 16 and A v A3 as related to Equation 17. Therefore, result is formally expressed as:
A S min A A3)
(20)
where it satisfies the Theorem 1.
Corollary 1.1 The optimal a" is bounded as follows:
A
iA
A +
A
4
0,
otherwise
A
+ A
Proof. To establish the validity of the optimal a' under different conditions for A and Aj, begin by examining the relationship between these two quantities.
Condition 1.1: (A; > Aj) This scenario implies that the difference between A, and A, is
positive, meaning C exhibits a positive value. In this case, the constant C' is bounded by the inequality:
A Aj C A
(21)
+
Substituting this bound into Equation 15, we have the following inequality for the optimal value
of a":
1 A -
(22)
2
A; + A
A
(23)
A. A++A, Aj
As we know that a" is constrained by 0 S a* < 1, this inequality confirms that a* satisfies the first condition of optimality, given by Equation 5. This ensures that when A, A the chosen value of a" lies within the allowable range and maintains the necessary relationship between the distances.
Condition 1.2: (A, S A₂) In this case, the difference between A; and A5 is non non-positive, and consequently, the value of C' becomes negative. Thus, the constant C' is bounded by the inequality:
A A
<
(24)
A + +1;
By applying the same process as in Condition 1, we substitute this bound into the formula for a". yielding:
4
<
(25)
A +
Given that a must satisfy 0 < a < 1, this condition is similarly met. Moreover, as a' 2 0, this value conforms to the second condition of optimality in Equation 5. Hence, when A Aj. the optimal value a continues to fulfill the constraints of the inequality, ensuring that the geometric relationship between the distances is preserved.
17

---

Published as a conference paper at ICLR 2025
Thus, the optimal value a' satisfies both conditions for the two possible relationships between A and Ay, concluding the proof.
A.2 PROOF OF THEOREM 2 AND COROLLARY 2.1
Theorem 2. Let 2h 2* be the interpolated latent space between the M; and the restricted Mj. defined by the interpolation coefficient a as: 2 = { (1 a'2²* + a²d 2² E z', E 2 where Pk represents its distribution. Let the 2-Wasserstein distance between Pk and P. be denoted as W2 (P, = AR. Then the optimal â that minimizes As is formulated as follows: Ås:
A A + A
2
Moreover, the resulting interpolated representation satisfies A S D. and D S S
Proof. Similar to the approach in Appendix A. Ax can be defined with following Definition
(P,.., Pk) _ (1 W3 a - - a(1-a)W3(PPPP) - a) + a44 - a)11-));,
(26)
(27)
To determine the optimal a that minimizes the left-hand side of Equation 27, we can utilize the derivative of the function (() (1 - A + aوه - a(1 - و)ه and set it equal to zero, as
demonstrated in Equation 14:
0f(a) = A
: + A - (1 22) = O.
(28)
da
This can be rephrased in terms of a, analogous to Equation 15:
1
A
1 C'
where C" (A, + A;) A
a"
(29)
2
A3
Additionally, the optimal â" is valid, similar to the validity of a*, under Assumption and the property of Âij, where Aiy S A.g..
To achieve the goal of minimizing Åg. we need to demonstrate that applying the optimal & results in value of AS smaller than both A, and Å,. Following the previous steps outlined in the proof of Theorem 1, we can proceed as follows.
Let the minimized distance be denoted Åj. with the optimal coefficient being a*. Using the Pythagorean Theorrm, relate the distances A, Áj, and Áij through the following expressions:
A = A (à" Á1,)
(30)
A ((1 à' A
(31)
These two equations can rearranged terms of â", leading to the expression:
A 33 (26* - 1)
(32)
From this, solving for d yields:
A A
14
(33)
2
This expression is equivalent to Equation Theorem 2. The assumption that the optimal â" minimizes Åg. which denotes the geodesic projection of the distribution Ps. between P; and Po,, aligns with the closed-form solution for d" given in Definition and Equation 29.
18

---

Published as a conference paper at ICLR 2025
Furthermore, by substituting into Equations 30 and 31, the minimization of Å implies:
A < min( (.). A3) Å min A Σ,)
(34)
(35)
Based on Definition 4 and the triangle inequality, Å, can be further bounded as A, و 4y However, discussed in Remark 2.1, the term 8 is expected to have negligible effect the Wasserstein distance between the distributions, resulting in only minor constant offset. Therefore, this does significantly impact the Wasserstein distance between the distributions, which Equation 35 will approximately similar to Equation 20.
Corollary 2.1 The optimal à" is bounded as follows:
A A
if > ر
+
à'
A
0, A A
otherwise
Proof. We analyze the bounds on â" by considering conditions based A, and Пy:
Condition 2.1 (A; > A :: In this case, C' is positive. From the triangle inequality for the Wassersttei distance, we obtain the following lower bound:
A A
(36)
A
Substituting this bound into Equation 29, we derive:
A
â'
(37)
A + A
A
(38)
A + A +
where a small constant, accounting for minor deviations between F, and Aj. as discussed in Remark 2.1. Since â* < 1, this satisfies the first condition in Equation 9.
Condition 2.2 (A; < A In this scenario, C' becomes negative. Using similar approach as Case 1, the upper bound for C' is:
C' A
(39)
A Å
Substituting this into Equation 29, we get:
A
â
(40)
A
Å
Since A 2 Aj. we can substitute A, with Aj. resulting in looser bound:
A;
(41)
A
Thus, this bound satisfies the second condition Equation 9.
Therefore, find that the derived bounds on 6* fulfill the both conditions stated in Equation 9.
n
19

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS
B.1 DATASETS
ImageNet-IK and Evaluation Benchmarks ImageNet-IK (Krizhevsky et al., 2012) the image datasets that contains 1000 classes with 1,281,167 training images and 50,000 validation images. ImageNet has been widely used in image classification benchmarks in various methoods, especially in computer vision task (Dosovitskiy et al., 2021; Steiner et al., 2022; Zhou et al., 2023; Pang et al., 2024). In our evaluation, we also assessed ImageNet-related validation benchmarks focusing on out-o--distribution (OOD) and robustness scenarios. These benchmarks can be categorized into three types: in-domain, OOD, and robustness.
For the in-domain category, utilized Ima (Recht et al., 2019), which consists of 10 images per class from the original ImageNet- IK, with total 10,000 images. The OOD benchmarks contains ImageNet-Rendition (Hendrycks et al., 2020), ImageNet-Sketch (Wang al., 2019), and ImageNet-Stylized (Geirhos et al., 2019). ImageNet- of 30,000 images, while ImageNet-Sketce contains approximately 50 images per class, totaling 50,889 images sourced from Google image queries labeled as "sketch of {class name). Lastly, the robustness scenarios encompass ImageNet-A (Hendrycks et al., 2020) and ImageNet- C (Hendrycks & Dietterich, 2019), ImageNet-A consists of images misclassified by the ResNet-50 model. ImageNet-C features variety of generated corruptions, such as Gaussian noise and blurring, and is commonly used in adversarial training approaches.
features 200 classes with a total
IEMOCAP The IEMOCAP dataset (Busso al., 2008) contains video, language and audio modal- ities. It consists of 151 recorded dialogue videos featuring two speakers per session, resulting in a total of 302 videos. Each segment is annotated for nine emotions: angry, excited, fear, sad, sur- prised, frustrated, happy, disappointed, and neutral. Recorded across five sessions with five pairs of speakers. also contains the audio and script for each video data. IEMOCAP dataset is a valuable resource for research in multimodal emotion recognition and has been widely employed in various multimodal methods (Zhang et al., 2020; Li et al., 2018) for the emotion sentiment classification.
As mentioned in Section 4, our implementation focused solely the script and audio components, emphasizing the language and audio modalities. For convenience, we narrowed our analysis to six specific emotions: neutral, happy, angry, sad, excited, and frustrated, employing the official PyTorch framework (Paszke et al., 2019) for dataloader.
AVMNIST The AVMNIST dataset (Vielzeuf et al., 2018) contains digit images (0 to 9) from the MNIST dataset (Lecun et al., 1998). where each image has dimensions of 28 x 28 x 1. These images have been subjected PCA projection, resulting in a reduced information representation compared to the original MNIST dataset. In addition to the visual modality, the dataset includes an audio modality from the Free Spoken Digits Dataset (FSDD) (Jackson et al., 2018). The audio data has been preprocessed into mel- freeueeyy spectrograms, sampled at resolution of 1 112 x 112 x 1.
In the context of the audio modality [A], we approach the two cases, [V-A] and [A-V]] using distinct strategies. For the [V=A] case, the audio modality is processed using convolutional neural networks (CNNs), outlined in AGM (Li et al., 2023). In contrast, for the [A-V]] case, we utilize raw audio data from the FSDD dataset rather than mel-spectrogram images, pretrained models specifically trained on spectrogram data are not readily available for this task.
Representation of ž In Section 5.2, we discussed the representation of z As ImageNet- 1K does not contains any supervision related to the [L] modality, except class label, we employed a new description for each image generated by LLaVA (Liu et al., 2024).
Specifically, we input images from ImageNet along with the following prompt: "USER: <image>>n (nWht does this image represent? Explain in a sentence. \nASSISTANT: which provides description that is closer to the true supervision of the language modality, 23, compared to our original prompt, "This is a class about #.", $1. As demonstrated in Table 4, this change does not result in significant performance differences, where it aligns to our theoretical perspective in Theorem 2 and Remark 2.1 that 5 does not substantially affect the distance between the distributions of each modality.
LLaVA generates a descriptive prompt for each image,
20

---

Published as a conference paper at ICLR 2025
B.2 EXPERIMENTAL DETAILS
Pseudo Code and Detail Implementation of 2² We provide the pseudo-code on Algorithm 1 and implementation details for %/ in all cases in Table 5 as follows:
Table 5: Implementation details of ay across different datasets and cases
Datasets & Cases
Implementation of 2
ImageNet- Ik [L-V]
[L] +
This is about Class #.*
IEMOCAP [[-+A]
[1]] + This 8 is about Emot ion #.1
IEMOCAP [A-L] [A] > Add Gaussian Noise: &~ N/ AT)tt Random Shuffling
AVMNIST [V-A]] [V] + Random Shuffled Image (mismatch paired sets)
AVMNIST [A-v]
[A] = Add Gaussian Noise: { Ar)tt & Random Shuffling
# is a random numher that does not directly correspond to the actual label. t1: A is parameter that controls the variance of the Gaussian noise. We applied À = 10-3
Algorithm Traininig Procedures for [M, M,]
Hyperparameter: a: interpolate coefficients, B: batch size
Input: (f y M St: input data from M, modality sampled in batch size B for each, { LM
1mml ~ SS:: imperfect input data from M, modality sampled batch size B for each Loss, 05: M; modality model parameters
Required: LcE: Cross-Entuen
Function: ((::0,), latent feature mapping function of Mj modality, 9j (:;0)): latent feature map- ping function of M, modality, h4(:: 0,): hypothesis function of M, modality
1: while not done do
2: for m 1,.... . do
3:
2/ 9, (
4:
L.ts LoF (he (g: (x 0;);;:);:00 vmn)
5:
C. E ||lg (x : 0,) 2/ 12
6:
Liotal (1 Leds + oL2
7:
0, 0,+0,- 0, - V Lrotal
8: end for
9: end while
Hyperparameters Settings In the [L- case for the ImageNet-IK classification task, we adhered to the hyperparameter settings established by AugReg-Vii (Steiner et al., 2022) for training models, specifically ResNet-50, ViT-B/32, and ViT-B/16. For the baseline model, trained for 300 epochs with a batch size 1024, utilizing learning rate of 1 x 10-3 3 and a weight decay of 5 10-2 We employed the AdamW optimizer (Loshchilov & Hutter, 2019) with cosine learning rate scheduling (Loshchilov & Hutter, 2017) and implemented a linear warmup for 20 epochs. Data augmentations, including MixUp (Zhang et al., 2018), Rand-Aggeett (Cubuk et al., 2020), and Au- gReg (Steiner et al., 2022), were applied throughout the training process. Furthermore, we utilized Automatic Mixed Precision (Micikevicius al., 2018) in conjunction with 4 A6000 GPUs
For the multimodal datasets, we customized two hyperparameter settings configuration based on the datasets: IEMOCAP dataset for fine-tuning and AVMNIST dataset for training from scratch. In the fine-tuning approach, particularly for the cases of [L-A] and [A-L] on IEMOCAP dataset, we conducted training for 30 epochs with a batch size of 4. Given that the learning procedure is fine-tuning, we adjusted the learning rate to 5 10-5 and the weight decay to 5 X 10-4, while also applying cosine learning rate scheduling (Loshchilov & Hutter, 2017). The Adam optimizer (Kingma & Ba, 2015) was employed, and no data augmentations were utilized. For training from scratch, specifically in the cases of [[ and [A-V] on the AVMNIST dataset, we again trained for 30 epochs with batch size of 32. The learning rate was set 1 X 10-3, with reduction factor of 0.1 applied at 25 epochs. The Adam optimizer (Kingma & Ba, 2015) was used, and data augmentations were omitted during this training phase well.
21

---

Published as a conference paper at ICLR 2025
C ADDITIONAL CASE STUDIES
C.I DESIGN CHOICES IN LATENT LOSS FUNCTION L:
In this section, we conduct an ablation study assess the impact different latent loss functions. Our primary approach employs mean squared error (MSE) loss to directly align latent representation vectors, we also explore the use of cosine embedding loss: L. E [1 cos (g ), ) For comparison, we consider the best results of each cases from Table I and Table 2. As summarized Table 6, performance remains consistent across both loss functions, with variations ranging from -0.1% to +0.7%. These results suggest that adhering to the representation distribution of the M, modality effectively supports training of M, modality, regardless of the choice of loss function.
Table 6: Performance comparison of different L; across various datasets and cases
Datasets & Cases
MSE
Cosine Embedding Loss
ImageNet-1K 1K [L->V]
81.90
81.97
IEMOCAP [L-A]
61.20
61.20
IEMOCAP [A LL]
56.49
56.77
AVMNIST [V [V+
42.44
42.67
AVMNIST [A-V]
66.69
66.25
C.2 ADDITIONAL. RESULTS IN THE LARGER MODEL.
We also conducted experiments with larger model, focusing the [L-V] case the ImageNet- 1K dataset. Specifically, we utilized ViT-L/16, which requires approximately 5x FLOPs compared to ViT-B/16. As the results are provied on Table 7, it demonstrate that approach achieves significant improvements even with this larger model, where it shows similar gap compare to Table 1. This indicates that our method does not rely on the scalability of model size; instead, leverages the qualitative representations of the M, modality to effectively synergize with or enhance the training of the Mj modality.
Table 7: Large model classification results on ImageNet- IK and evaluation benchmarks
Model [L-V]
IN
V2
Rend.
Sketch
A
Style.
C (4)
ViT-L/16 (reproduced)
80.04
70.75
51.33
39.73 40.87
31.01
25.63
44.85
RoBERTa (Liu et al., 2019)
81.63
71.76
53.00
32.33
28.50
41.61
C.3 DESIGN CHOICES FOR IMPERFECT SUPERVISION 2
In this section, we present additional experiments for the [I-+] case ImageNet-IK K classifica- tion to evaluate how varying cases of imperfect supervision impact performance. We define three different cases of imperfection setting to analyze effects:
Case 1 Completely Imperfect Supervision: At this case, we generated 1,000 random sen- tences using ChatGPT-4o and randomly matched them to the training data of the M, modality model. This represents a highly imperfect supervision setting, introducing significant noise into the supervision of the M, modality.
Case 2 (Recap)- "This is about Class # #: This case corresponds to our main ap- proach, where supervision is based on random number # assigned to the data.
Case (Recap)- Perfect Supervision: As described Section 5.2 and Appendix B.1, this case employs high-quality supervision generated by LLaVA for each image in the dataset.
22

---

Published as a conference paper at ICLR 2025
The results in Table 8 reveal that even when & is entirely unrelated, representing completely imperfect paired dataset, it can still synergize effectively with the M2 modality. Notably, this setup achieves an improvement of nearly 1.5% compared to training with single modality and Case supervision. These findings underscore the strength our approach and strongly align with our theoretical perspectives, in Theorem 2.
Table ImageNet- IK classification results with ViT-B/32 + RoBERTa under different cases of gd
Model
[L-V]
Single
Case
Case 2
Case
Accuracy
75.04
75.52
76.75
76.99
T: No RoBERTa, ViT-B/32 only.
C.4 ADDITIONAL ABLATION APPROACHES CONSTRUCTING 2x
Here, we conducted ablation studies to construct 2, using alternative multimodal learning ap- proaches. While our method leverages 2- to enhance representation learning from the M, modality, we also evaluated traditional strategies such concatenation and addition. For concatenation (late fusion), we followed the standard late fusion approach (Wang et al., 2020), where the representations from both modalities concatenated into single vector for joint training. Before concatenating, we applied an interpolation term a, combining (1 - a) z* and a S. For addition, we directly combined the interpolated representations with the coefficient by adding them.
As shown in Table 9, traditional approaches result significant performance degradation compared method, which utilizes the latent loss function L2. Similarly, Table 10 further demonstrates how our approach uses to interpolate the representation space 2* between 2' and 20, effectively reducing the latent representation gap. In contrast, concatenation and addition primarily introduce bias from the M, modality into the M2 modality without effectively reducing the latent represen- tation gap. where there are no additional loss term that reduces its gap. As the result, these con- ventional methods fail to construct an optimal 2t, which, according to our theoretical hypothesis, should be embedded in the interpolated space between Z' and Ê1, Therefore, these findings under- score the limitations of traditional multimodal learning approaches our framework.
Table 9: Performance comparison traditional multimodal learning approaches and applying E
Table 10: Gap (MSE Loss) between &h and & across different approaches
Datasets Cases
f,
Concatenation
Addition
Datasets & Cases
Concatenation
Addition
IEMOCAP [L-A]
61.20
60.68
59.68
IEMOCAP [L-+A]
0.129
0.892
0.906
IEMOCAP [A-L]
56.49
55.90
55.34
TEMOCAP [A-L]
0.270
0.516
0.762
AVMNIST [V->A]
42.44
42.07
41.49
AVMNIST [V-A]
0.017
0.453
0.774
AVMNIST [A-+V]
66.69
65.44
65.59
AVMNIST [A-+]]
0.033
0.481
0.360
23


Reviews:
[{"review_id": "PjdmNbEduK", "reviewer": "Reviewer_hzXr", "summary": "The paper aims to answer whether imperfectly aligned paired data from other modalities can help learning in a multimodal setting.\nSpecifically, the authors propose an additional latent loss, to directly align the target modalities' latent representation with that of the output of a pre-trained encoder of the secondary (supportive) modality. The authors introduce and study a theoretical framework and show that even imperfect paired data can help approximate a hypothetical, perfectly aligned representation. They further demonstrate empirically that the additional latent loss led to stronger performance of the target modalities' encoder across various tasks and modalities.", "questions": "Regarding Remark 2.1. \"δ does not hinder the synergy\", you say that\n\"We can extract an imperfect feature representation from Pj by giving\nimperfect input to the modality Mj . This allows ˆzj exist in the distribution Pj 2. Consequently, ˆzj\nis closer to or part of the latent space of the Mj than to that of the Mi or the true latent space.\"\n\nWouldn't this quite directly depend on how far \\hat P_j is from P_j, as well as how aligned P_i is to P_j to begin with? Surely one could construct counter examples with adversarially chosen δ? This may not be much of a practical concern for reasonably close \\hat P_j but is this statement not a bit strong in the general case?\nExpanding on this, perhaps this could be empirically verified by exploring different levels of noise to introduce in \\hat z^j, particularly in the L -> V task as suggested above. Have you perhaps already considered / explored different noising functions and compared their impact?\n\nFor the V -> A case in AVMNIST, you say \"For the [V→A] case with AVMNIST, we use randomly shuffled images from AVMNIST as ˆzj in audio classification tasks\". Could you clarify the random sampling in this case? Is it a random image from the entire dataset or a random image from the samples of the same target class? If it is a random image (i.e. unrelated to the paired modality at all), this seems significantly more \"noise\" than in other settings, it'd be great to understand a bit better to understand the motivation for this choice and perhaps similar experiments for other modalities.\n\nOn a general level, if we assume that the target distribution for a modality encoder g_i is similar to the one of a pre-trained encoder g_j of a different modality, the proposed latent alignment loss has some similarity to knowledge distillation. In this field there's been some notable prior work that suggests that the success of KD is partially attributable not only to a superior knowledge of the teacher but also to benefits of the training strategy itself. Notably, Born Again Networks (Furlanello et al., 2018) suggests a simple strategy of self-distillation can improve performance. Yuan Li et a., 2020 explores this further in \"Revisiting Knowledge Distillation via Label Smoothing Regularization\". This is relevant to this work since it could suggest a different mechanism leading to the empirically observed improvements that is less about multimodal transfer and perhaps more about a sort of regularization effect of the added latent loss. Has this been considered?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work considers a range of relevant modalities and conducts experiments across language, vision, and audio in various cross-combinations.\nThe mathematical framework introduced is intuitive and easy to follow.", "weaknesses": "The paper asserts in multiple locations that prevailing multimodal learning methods require \"perfectly paired datasets\" (quote from the introduction) between modalities. This, in my opinion, is not accurate an accurate representation of the thinking in the field. The authors cite CLIP as a notable multimodal model, which famously is trained on noisy web-scale paired data in the form of image alt text. Training multimodal models on noisy labels such as alt-text from web-scraped images is common practice and widely established (Radford et al. 2021, Dosovitskiy et al. 2020.,...). While improving the alignment of the training modalities is generally seen as desirable (e.g. Fang et al., 2023), \"perfection\" seems not a requirement.\nThe paper does not cite and / or discuss other works in the space of aligning multiple modalities without direct paired supervision data, including popular works such as ImageBind (Girdhar, et al., 2023) or 4M (Mizrahi et al., 2024). These works do not (solely) rely on paired multimodal data, seemingly directly addressing the limitations discussed by this work in section 2.3.\nThis, together with arguably understating how much alignment on noisily paired data has been previously studied in the field, arguably limits the novelty of this work.\n\nOne of the main statements of this work is that the label does not need to be perfectly paired / can be noisy. However, the transformations to introduce this noise studied in the work may not be sufficiently realistic. For example for the L -> V case, the noisy label \\hat z^j is constructed by embedding the text \"This is about (Class|Emotion) #.\" as per table 5. In terms of the measured downstream task, which is classification, this label is arguably not noisy, but perfectly represents the target task. In table 4 it is shown that changing this supervision signal with a caption produced by LLaVA leads to only minor improvement, which may not be surprising in this setting. (See the Questions section for suggestions around this.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3dy1ChC5p9", "reviewer": "Reviewer_E3nE", "summary": "This paper investigates whether one modality model can enhance the training of another modality model without requiring paired multimodal supervision. The authors propose both theoretical and empirical evidence that imperfect supervision from one modality (e.g., language) can improve the performance of another modality (e.g., vision). They establish mathematical foundations showing that an interpolated representation between two modalities can outperform single-modality representations, even with imperfect cross-modal supervision. The work is validated through extensive experiments across vision, language and audio modalities, demonstrating consistent performance improvements. For example, in the vision domain, they show improvements of 1.5-2.5% on ImageNet classification and similar gains on robustness benchmarks by leveraging simple language prompts during training.", "questions": "- Figure 2 right, $Z_i$ should be $Z_j$.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Novel insights: The work challenges common assumptions about requiring paired supervision for multimodal learning and demonstrates unexpected improvements using single-modality pre-training and synergies between seemingly unrelated modalities.\n\n- Strong theoretical foundation: The paper provides rigorous mathematical proofs for how and why cross-modal learning can work without paired supervision, establishing bounds on the interpolation coefficient $\\alpha$ and showing the existence of superior interpolated representations.\n\n- Comprehensive empirical validation: The authors demonstrate their approach across multiple modality pairs (Vision-Language, Vision-Audio, Language-Audio) and various architectures, showing consistent improvements across different settings and tasks. Results include not just standard classification metrics but also out-of-distribution generalization and robustness benchmarks, showing broad improvements across different evaluation criteria.", "weaknesses": "- Other language supervision: The language prompts used (e.g., \"This is about Class #\") are quite basic. It would be interesting to see how the method performs with more complex or varied language supervision.\n\n- Theoretical assumptions: Some theoretical assumptions (e.g., Assumption 1 about $\\Delta_{ij} \\ge  0$) could benefit from more empirical validation or discussion of when they might not hold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2GZwQTagml", "reviewer": "Reviewer_Mtx8", "summary": "The paper demonstrates that performance on a target modality can be improved by leveraging another modality, even without paired samples. Both theoretical and empirical evidence are provided to support this claim, showcasing the method's effectiveness.", "questions": "Please check the weaknesses section, in particular the question about how leveraging other modalities is done.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The paper tackles an important problem: obtaining paired data is often challenging in many setups.\n* The approach is backed by both theoretical analysis and empirical results, showing clear improvements over baseline methods.\n* The paper is well-structured, with illustrations that help clarify key messages.", "weaknesses": "* The main contribution—showing that using an unpaired modality can improve performance—has already been explored in prior works. For instance, some studies demonstrate leveraging unpaired modalities through pretraining on one modality and fine-tuning on another, like from image to video to audio [1], or from text to image-text [2]. Additionally, the issue of handling unpaired or missing modalities has been addressed before, yet the paper does not discuss relevant works in this domain [3,4,5]. Including this discussion would better position the paper.\n\n* It is not clear why the author decides to leverage other modality through an l2 loss between the features spaces. Other design choices can  be explored. For example, concatenation, addition or other multimodal features fusion techniques.\n\n* The experiments use relatively small models on classification tasks. It remains unclear whether the proposed method would be effective on larger, more complex, maybe generative models (e.g., Multimodal LLMs, CLIP).\n\n[1] Shukor, Mustafa, et al. \"Unified model for image, video, audio and language tasks.\" TMLR (2023).\n\n[2] Liu, Haotian, et al. \"Improved baselines with visual instruction tuning.\" CVPR 2024.\n\n[3] Kim, Donggeun, and Taesup Kim. \"Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models.\" ECCV (2024).\n\n[4] Lee, Yi-Lun, et al. \"Multimodal prompting with missing modalities for visual recognition.\" CVPR. 2023.\n\n[5] Wang, Hu, et al. \"Multi-modal learning with missing modality via shared-specific feature modelling.\" CVPR. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)