Towards Out-of-Modal Generalization without Instance-level Modal Correspondence
Authors: Zhuo Huang
Gang Niu
Bo Han
Masashi Sugiyama
Tongliang Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Multi-Modal Learning; Generalization
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We study Out-of-Modal Generalization, which aims to learn novel modalities using knowledge from existing ones.

Abstract: The world is understood from various modalities, such as appearance, sound, language, etc. Since each modality only partially represents objects in a certain physical meaning, leveraging additional ones is beneficial in both theory and practice. However, exploiting novel modalities normally requires cross-modal pairs corresponding to the same instance, which is extremely resource-consuming and sometimes even impossible, making knowledge exploration of novel modalities largely restricted. To seek practical multi-modal learning, here we study Out-of-Modal (OOM) Generalization as an initial attempt to generalize to an unknown modality without given instance-level modal correspondence. Specifically, we consider Semi-Supervised and Unsupervised scenarios of OOM Generalization, where the first has scarce correspondences and the second has none, and propose connect & explore (COX) to solve these problems. COX first connects OOM data and known In-Modal (IM) data through a variational information bottleneck framework to extract shared information. Then, COX leverages the shared knowledge to create emergent correspondences, which is theoretically justified from an information-theoretic perspective. As a result, the label information on OOM data emerges along with the correspondences, which help explore the OOM data with unknown knowledge, thus benefiting generalization results. We carefully evaluate the proposed COX method under various OOM generalization scenarios, verifying its effectiveness and extensibility.

Published as a conference paper at ICLR 2025
TOWARDS OUT-OF-MODAL GENERALIZATION WITH- OUT INSTANCE-LEVEL MODAL CORRESPONDENCE
Zhuo Huang', Gang Niu?, Bo Han Masashi Sugiyama 2:4, Tongliang Liu 1
1 SSydnyy AI Centre, The University of Sydney 'RIKEN Center for Advanced Intelligence Project
á Hong Kong Baptist University "The University of Tokyo
ABSTRACT
The world is understood from various modalities, such as appearance, sound, and language. Since each modality only partially represents objects in cer- tain meaning, leveraging additional ones is beneficial in both theory and prac- tice. However, exploiting novel modalities normally requires cross-modal pairs corresponding to the same instance, which is extremely resource-consuminn sometimes even impossible, making knowledge exploration of novel modalities largely restricted. To seek practical multi-modal learning, here we study Out of-Modal (OOM) Generalization as an initial attempt to generalize to an un- known modality without given instance-level modal correspondence. Specifi- cally, we consider Semi- Suni-Supervised and Unsupervised scenarios of OOM Gen- eralization, where the first has scarce correspondences and the second has none, and propose Connect& Connec&EExplore (COX) to solve these problems. COX first con- nects OOM data and known In-Modal (IM) data through variational information bottleneck framework to extract shared information. Then, COX leverages the shared knowledge create emergent correspondences, which is theoretically jus- tified from an information- thhooreticc perspective. As a result, the label information on OOM data emerges along with the correspondences, which helps explore the OOM data with unknown knowledge, thus benefiting generalization results. We carefully evaluate the proposed COX method under various OOM generalization scenarios, verifying its effectiveness and extensibility. The code is available at ICLR..OO
and
htsps:///gthhie
INTRODUCTION
To understand the world, we use various data modalities, such im- age (He et al., 2016; 2017; Ren et al., 2015) and text (Devlin et al., 2018; Vaswani et al., 2017). Each modality describes objects through a certain physical perspective, thus contributing to understanding ob- jects. Therefore, multi-modal learning (MML) (Alayrac et al., 2022; Ngiam et al., 2011; Radford et al., 2021: Socher et al., 2013) which learns from multiple modality data has been a core research topic in AI. Thanks to the utilization of various modalities, the learning per- formance has shown to be beneficial on various tasks compared to uni-modal learning (Huang et al., 2021; Lu, 2024; Radford et al., 2021; Sun et al., 2020), such as cross-modal retrieval and genera- tion (Yasunaga et al., 2023; Zhang et al., 2021; Zhen et al., 2019), human-computer interaction (Pantic & Rothkrantz, 2003; Rahman et al., 2022), and robotics (Jiang et al., 2023; Yu et al., 2023).
Figure 1:
AI is enhanced as more modalities are in- corporated, so how can we teach AI to learn from novel modalities based on the ones it already know?
However, existing states of the art are not satisfactory, and emerging modalities need to be leveraged effectively just like the relatively new data modalities of geomagnetic fields (Hashimoto, 1926), sound waves (Harley et al., 2003), and electromagnetic waves (Weinstein, 1988). Therefore, emerg- ing technologies have constantly leveraged new sensors enhance their performance. For example, Embodied AIs (Savva et al., 2019) already possess abilities like 3D vision and language, but they still exploring novel skills, such tactile and bio-sensing. Since it is hard leverage such un- common and inexperienced skills in practice, adapting the knowledge from common modalities to
"Corresponding Tongliang Liu (tong] ang. 1 u@aaddey. au).

---

Published as a conference paper at ICLR 2025
understand the novel ones could be beneficial, as shown in Figure 1. In practice, most existing MML investigations (Radford et al., 2021; Girdhar et al., 2023; Wang et al., 2024d: Zhu et al., 2023) re- quire instance-level modal correspondence, i.e., multi-modal data are paired with the same instance, which is often hard to satisfy in real-world scenarios when facing novel modalities (Liang et al,, 2023; 2021; Sun al., 2020; Xia et al., 2024). For robotic example, some modalities common and easy to acquire, e.g., vision and language. However, others like tactility need special sensors to resample from the same objects seen spoken. Unfortunately, the resample could no longer be accessible in practice. As result, the new modalities usually have incomplete or even no cor- respondence, which could seriously block the knowledge interaction across modalities and hinder the benefits brought by MML. Hence, question naturally occurs: Do we really need instance-level modal correspondence to explore novel modalities?
This paper studies practical yet unexplored problem named Out-of-Modal (OOM) Generalization. Particularly, given several modalities, i.e. In-Modal (IM) data, the goal to generalize to an un- known modality without sometimes only with scarce correspondence. Such setting implies the real-world utilization of novel modalities: Even though human perception is limited to certain modalities, e.g., touch, sight, sound, and smell, magnetism by utilizing cahherentlyposseessee senses, e.g. feel the force when pulling two magnets together; or see the magnetic field by observing the alignment of iron filings around magnet.
still understand unperceivable ones such as
Based on this insight, we utilize IM perceptors that contain prior knowledge to encode known IM data, which can be implemented using existing MML models (Radford et al., 2021; Girdhar al., 2023; Zhu et al., 2023; Wang et al., 2024d), and an OOM learner which learns novel modalities without any prior knowledge. By analyzing the interactions between latent features, we show the- oretically and empirically that the OOM learner can be trained to gradually discover the OOM knowledge, as shown in Figure First, we consider aupi-supee OOM generalization where few correspondences are given. Based on the correspondence, we can capture the prior probability distribution and learn mappings that connect OOM data and IM data. Through an information- theoretic perspective, we propose Connect& Connect&Fxplore (COX), which encourages the agreement on mappings across modalities, further sharing the cross-moda knowledge and exploring OOM knowl- edge. Then, we extend COX to an unsupervised OOM generalization scenario where there is no instance-level correspondence at all. To tackle such a challenge, we enhance the OOM-IM connec- tions by maximizing cross- modal interaction. First, we select data pairs from cross modal mappings according to feature similarity. By assuming that the data pairs closing to OOM mappings can be considered as correspondence, we can create emerging correspondence and solve the unsuper- vised case via the semi-supervised solution. To validate the proposed COX, we carefully design experiments using various multi-mode modal datasets to validate its effectiveness. Moreover, we provide extensive analyses in various scenarios to understand our method and inspire future research. To sum up, our contributions are foree-fold:
We discover novel and practical problem named Out-of-Modal Generalization, which aims to explore a novel modality using the knowledge from known modalities.
We consider two typical situations: Semi-Superrised OOM generalization and Unsuper- vised 0OM generalization, and propose Exploet&Explore framework to tackle both problems from an information-theoretii perspective.
We conduct extensive experiments to tackle the OOM generalization on various datasets and provide intuitive insights to help inspire future research.
RELATED WORK
Modality Generalization (Liu et al., 2024) generally focuses on leveraging the knowledge from some modalities and generalizing to another one. Existing studies are conducted in different set- tings and with various tasks. Cross-modal fine-tuning mimics transfer learning by adapting the distribution of IM data to OOM data using the same model. Shen et al. (2023) proposed to conduct distribution alignment to achieve this goal which requires both pre-trained knowledge and labeled target modality data. Based on a similar problem setting, Cai et al. (2024c) designed a gradual modality generation scheme that selects the top-k active feature patches from target modalities, and replaces them with source modalities patches. Such progressive strategy data to ensure generalization. Tross-Modal. Generalization uses separate encoders and focus on gen- eralizing to different modality data from the same instance. Liang et al. (2021) used meta-learning
align target modal
2

---

Published as a conference paper at ICLR 2025
Table 1: A comparison of different MML problems and their corresponding settings.
Problem
References
Knowledge
Knowledge
Correspondence
Cross-Modal Fine- Tuning
Shen (2023); Cai et al. (2024c) pre-trained labeled
labeled
Cross-Modal Gencralization
Liong ct al. (2021)
pre-trained & labeled
pro-trained
Xia ot al. (2024)
pre-trained & labeled pre-trainod & labeled
labeled Mudii-Modal Data Liang et (2023)
parially labeled
partially labels
0OM Generalization
Semi Supervised case (Section 3.3) pre-trained labeled
scarcely labeled
few
Unsupervised case (Section 3.4)
pre trained labeled
X
align OOM data to IM space and generalize to OOM tasks dynamically. Xia et al. (2024) studied different setting where IM and OOM data are both known during training. Then, unified represen- tation space is learned help with the downstream generalization of OOM data. Some other stud- ies consider generalization when all modalities are available, Ma et al. (2019) studied cross-modal generalization without paired data, Wang et al. (2023) applied the information bottleneck to CLIPP training. Fang et al. (2024) conducted multi-modal fusion under limited clinical data, and Dong al. (2023) considered domain generalization with fully-paired multi-modal data. A recent study MML without Labeled Multi-Modal Data (Liang al., 2023) proposed different setting where both IM and OOM data have labels, but they are not paired. Instead, additional unlabeled paired multi- data is given for learning the interaction between modalities. Moreover, Xue et al. (2022) understood the interactions and applied it to knowledge distillation. Except for cross-modal fine-tuning which follows transfer learning, existing MML works mostly require instance-level correspondence, This work proposes OOM Generalization, where there is no correspondence and the OOM knowledge is barely provided. The comparison of related works is shown in Table
Modality Binding aims to learn a joint embedding space across different modalities. Contrastive Language-Image Image Pre-training CLIP (Radford et al., 2021) is the first work that aligns image with language data. Then, ImageBind (Girdhar et al., 2023) proposed to use vision modalities to bind various modalities into the same representation space. Further, LanguageBind (Zhu et al., 2023) proposed using language as an alternative solution, which binds various modalities similarly. Re- cently, FreeBind (Wang et al., 2024d) extended the existing unified space into an additional expert space. Specifically, two types of binding were considered, namely space displacement bond and space combination bind. Since modality binding often requires a large amount of data with cor- respondence, the selected modalities are often quite common. Therefore, the OOM generalization problem take advantage of the development of modality binding by leveraging the encoders as our IM perceptors to learn novel modalities.
OOM GENERALIZATION
In this section, we first formalize the OOM generalization setting. Then, we demonstrate the pro- posed method. Further, we consider a Semi-Supervised case where few correspondences avail- able and an Unsupervised scenario where there is no correspondence, showing that the proposed method can successfully tackle both settings and effectively leverage unpaired OOM data.
3.1 PROBLEM SETTING
In OOM generalization, we given set of known modalities {M) [M!...., where MI (k).....KK)
oh,,), E X' x }} is composed of N number of labeled IM exam- ples with its subscript denoting the correspondence across different modalities. Moreover, we have an unknown modality M = [(x)) ) containing M unlabeled OOM examples. In some cases, possible to obtain few correspondences with IM data, then our OOM data could be M = (x v H ((())))) 11 where L < M and the subscript i traces the corresponding IM data instance and label.
To tackle OOM generalization, we propose a learning framework as shown in Figure/ 2. Particularly, we use a set of IM perceptors (of...., to perceive IM data, which can be realized by many ex- isting modality-bindimg models, such as ImageBind (Girdhar et al., 2023) and LanguageBind (Zhu al., 2023).
2
gi
*
z
*
gk
zkz
Figure 2: Learning framework of our OOM generalization.
3

---

Published as a conference paper at ICLR 2025
Then, the features of IM data are obtained via z 9 (z.). Morcover, we use an OOM learner 9 to learn features 2 from OOM data through = (x Our goal is to effectively generalize to OOM data by exploring the relationships between the OOM feature 20 and IM features {z) {=k R= Note that we only focus the generalization performance of OOM data, the improvement of learn- ing IM data is not the goal of this paper. Therefore, we freeze the parameters of all IM perceptors and only train the OOM learner during experiments. On top of the above models, we further define classifiers 10 (x*)= == (x ; and hxxr))) == hu (1t gt)) that make predictions.
3.2 METHODOLOGY: CONNECT& (COX)
Here we elucidate the proposed method based on the interactive relationship between modalities (Liang et al., 2023; Williams & Beer, 2010), Specifically, the total information of two modali- ties under a certain task is decomposed into 1) commonalityy which indicates common attributes across modalities, 2) uniqueness that is only presented in each modality, and 3) synergy denoting the emerging information when modalities are presented together. Note that we do not consider 3) in this paper our goal is generalizing to OOM data.
To generalize to an unknown modality based on common ones, we aim to extract the commonality that can help partially comprehend OOM data based on IM data. Then, we model the posterior distribution of OOM data by selecting anchor points with minimum uniqueness. To this end, the OOM generalization can be successfully established. The proposed COX method comprises two steps: 1) learning connections by mapping IM data to OOM data to extract commonality, and 2) exploring high uniqueness OOM data by matching their posterior to high-commonality OOM data.
Connection through Commonality aims to capture common knowledge across modalities using generative models (Lu, 2024). Here we follow the variational information bottleneck (VIB) frame- work (Alemi et al., 2016) to achieve this goal. We assume that given IM data X and OOM data X the latent variable V extracted from X12, and label Y, the joint distribution is factorized
(1) where we assume pVV, X X²) ((XIX)) corresponding the Markov chains V ++ X ++ x and X + Y 4 X Such an assumption means that V is not related to X (Alemi et al., 2016) and the given label Y is not directly connected X under our OOM setting. Intuitively, given an IM datum, i.e., dog image, it is sufficient to infer the label "dog". and the same for inferring from an unknown OOM datum. i.e., dog bark. Thus, in common multi-modal settings, the label prediction using IM information dog image is not further conditioned on OOM knowledge dog bark, because here the OOM knowledge redundant when IM data is given.
P(X!, X XX, VVY) = p(V, Y x (x))
Our goal is to extract valuable knowledge from IM data to leverage OOM data by maximizing the information commonality (Liang et al., 2023; Williams & Beer, 2010):
(2) where I(XO; X4; denotes the mutual information between X and X regarding the task Y. i.e. the label; and start with the first term:
max ; = I(xx; x²) - x
indicates the conditional mutual information irrelevant to Y. We
T(XO; x!) - do do p(00 ..)) log plo0aat) =
ff deppe'ppaa x²) log p(z0)=
px)
(3)
p(x0)p(x')
can be approximated via a de- coder 9(2%)) Since the Kullback Leibler (KL) divergence is always nen-nvga,ive, we have KL[p(XxVV ll q(X*))) 2 0 = { dx p(x (l)) p( 2 dx pppx (v) log (vx and leveraging Jensen's inequity, we can have
where p(2) = (upp(0,0/a))
= de
du dx' xx") log J dogg (1))(ot)))))
[1x;x'13
log
(4)
p(x0)
dodtttppp xd) log ((x (p)p()l))))))))))))))))))))))))) H(X)),
H(
(5)
is originally termed "redundancy" which is negative. However, such property is quite positive for tackling our problem, and hence we rename it "commonality". Note that the latent variable V here is different from the feature representation 2' and 20
4

---

Published as a conference paper at ICLR 2025
where the last term is independent of our optimization process. Further, rewrite p(x I dup(x>), x² .)) = dep(2' )p(x)) (z))p(u(x"), " Then, we have the following lower bound:
T(X;;
dx dx'ddp((>> p(( (x"")//))))) [=")p(ula? log ( ( ())((l>²),
which is realized by sampling from the joint data distribution, the latent variable from our encoder p(ulzt), and the tractable variational approximation g( (v).
Similarly, we can upper-bound the second term
(details shown in Appendix A):
(20 J dr'dy(( x. y) log p(ul )p(z (a)) )p(x') log h (ylz²),
where 11 ( is our classifier model for predicting OOM data. To this end, we can lower-bound our objective by combining Eqs. 6 and 7:
1(x0; f de'htepp) p((
log q( (v))((pp(())) ( (p(l=") log
(8)
- da dx'dyp(a ,²,8) log p(yl= 1(" (xt ((<x))++++/))))) +log)) h ( (ylx
Coon
The above lower bound contains two parts: 1) OOM data reconstruction where reconstruct X using the latent V and 2) OOM data label prediction where we model the label distribution Y. In practice, we can approximate p(x x²1 y) using empirical samples from IM and OOM data. More- over, we use encoder p(u|==") without any prior assumptions because we can leverage the feature distribution from the pre-trained IM perceptors. Additionally, a classifier h(ylz is optimized to categorize OOM data based on given labels. Empirically, we can minimize
M
Luon
Σ
log h (3:11:))
(9)
M
where we use the reconstruction error || 112s to realize the log-likelihood (u)p(()))) as sim- ilarly done by Kingma & Welling (2013). After building the connections, we can ensure the task- relevant information shared across modalities is learned, which helps partially understand OOM data regarding its commonality. However, note that the second term in Eq. 8 is not fully lever- aged which contains p(yl=!) modeled by the IM perceptors. Take a step further, we can obtain -fdeotetuyp(("" ( _.. y) log plyla' )m(aa")(=: (alna"laa))) mma)) Since pla:0/z ((())) is fixed in label prediction, (g(g)=")
we can derive (p(y (=t) || no (yyx²)) which implies that the label information related IM data can be harnessed to explore commonality. Next, we demonstrate how the commonality helps OOM generalization, and provide a solution to explore uniqueness.
Exploration of Uniqueness
be achieved via selecting and exploring the OOM data with high uniqueness. To identify these data, we can leverage the agreement and disagreement achieved by the optimal classifiers from various IM data. Our final goal is to optimize via
min KL O (u)==)) ko
O (ul==)) where x.d E DD, 2. A,
(10)
in which ht and hī denote the optimal classifiers found in two IM data of and z1 respectively, and 22 and x are selected from OOM data with modality disagreement D = {ar hi (1(x)) + hz())}} and agreement A := {x (x = h hz respectively. Here we use two in- modalities for simplicity, but the conclusion can be extended to multiple modalities. Moreover, the data with agreement is considered anchor points that guide the exploration of those with disagree- ment. This objective aims to match the posterior of OOM data with uniqueness f = to the one of anchor points h0 (g)==2). To justify this, we first define modality disagreement:
Definition 1 (Modality disagreement). Given X1, and target Y, as well as their corre- sponding optimal classifiers h and hī, their modality disagreement is defined as ahhi, hī) = Ep(ay.-r)) [d(h;, hī)] where d y y R+ is distance function in the label space scoring the disagreement between hi and hī-
TThough training generative models in input space is computationally inefficient, propose to connect modalities in the feature space experiments. Therefore, the data is replaced by latent feature
5

---

Published as a conference paper at ICLR 2025
Theorem 1. Given two Bayes' optimal classifiers h and h2 from two in in-modalities, under relaxed triangle inequality, inverse Lipschitz condition, and classifier optimality assumptions (Sridharan & Kakade, 2008), the modalities disagreement is upper- pouurr-bunded by (see details in Appendix B)
athi h2) < T(XO, X2, Y|X)) T(X X1, Y xx) 2T(XO, y|X X2)-
(11)
Finally, based on the decomposition of the task-related mutual information of X I(X .Y T(XO; X2, Y|X+) I(X", Xt, YIX)) + I(XO, Y|X,, X3) + I(X", X1, X2, Y), shown in Figure 3, we can achieve
a(hi, 2 (xx Y) - 1(x x x) + Y xx))
(12)
where the first term denotes the overall information, the second term indicates the commonality shared be- tween all modalities, and the third term stands for the uniqueness only preserved in OOM data. Intuitively, when we try to increase the modality disagreement, the
x
100 AVo.xI.VIN vIK3)
commonality is decreased and OOM uniqueness is in- creased, which successfully justifies our learning objec- tive: In order to explore the uniqueness of OOM data, we can explore the ones with high modality disagree- ment; conversely, the OOM data with high common- ality and low uniqueness is found where agreement is achieved among hi and h2z Therefore, select such data as anchor points that provide informative guidance to help explore uniqueness.
X
X2
Figure 3: Decomposition of T(x0,)) YY
Next, we consider realistic scenarios of OOM generalization and demonstrate how the proposed COX method can tackle them.
3.3 SEMI-SUPERVISED OOM GENERALIZATION
We start with semi-supervised case where a few correspondences are available in OOM data, as shown in Figure (a). Based on the VIB framework proposed in Sec- tion 3.2. we first leverage the OOM data {(x) y, y corresponding to IM data {( 90,)) Vk K} to build K connections using additional generative models that can be trained via a point-to- point mapping. As result, the map- pings on the OOM feature space can successfully match the OOM feature distribution, which allows us directly apply IM data posteriors to select and explore the uniqueness of OOM data. Hence, we formulate our objective as
z
20
zk
2/
20
(8)
(b)
Figure 4: Two scenarios: (a) Semi-Supervised OOM Gen- cralization and (b) Unsupervised OOM Generlaizaiton.
min Last
Σ i=1
CE(N (aP)))))))))))) + L+ Fd. 2
ΣΣ
(13)
where the first term exploits labeled OOM data with correspondence and the second term explores OOM data D with modality disagreement by minimizing its KL divergence from the label posterior. Through the above objective, we maximally exploit the uniqueness of OOM data to achieve effective OOM generalization.
3.4 UNSUPERVISED OOM GENERALIZATION
As for the unsupervised case, propose two- two-phase training: 1) we first conduct warm-up training to initialize the OOM feature space and the connection, and 2) then, we enhance the connection by creating emergent correspondence and further exploring OOM data.

---

Published as a conference paper at ICLR 2025
Specifically, we select anchor points from OOM data by directly applying modality agreement among all Bayes' optimal classifiers from IM data via
K
Asorted = SORTT(A (A,
max hig (xx (x))). where A = E =M.
(14)
where the SORTT(-;-) a sort function, which ranks each element x0 in A based on the value of R Ex= max hig (x from large to small. Here, we select anchor points with the top- largest likelihood averaged over all K IM classifiers. Then, we warm up the OOM learner via minimizing arr
cross-entropy loss min T E CB(hO(x:)) arg max hz(x))). Additionally, we also warm up the connection by leveraging caas-wise information. Specifically, we compute the cluster cen- troids for each modality via C,, 2n =c,, (x")) =f and pair them to each IM centroid correspondingly. To this end, can build up initial connections by following the VIB framework.
After the warm-up, we aim to further enhance both our connection and OOM exploration by creating emergent correspondence, as shown in Figure 4 (b). To tackle this, we map all IM data into the OOM feature space. If an OOM feature is close to all mappings Uk,i, Vk = (l, K J. then they can form a strong correspondence. Further, we select such OOM data as anchor points, which is further labeled the same as the corresponding IM data. Formally, we optimize OOM learners via
min Cuns NO
E CE(h (z(),) v)+ LAHD
1
E
EELL(N kL (2)) 11 (2));,i,h)) (15)
T
MIAA
>PEDaPEA
where A denotes the updated anchor points which are realized by sorting the Fuclidean distance: 1 E 11 - where the first term computes A == SORT ((()) ..))) 1- min.e( K
the cross-entropy loss from the anchor points, and the second term calculates the KL divergence between the OOM data with modality disagreement and the OOM anchor points.
After these two steps, we can effectively tackle the unsupervised OOM generalization. In practice, connect modalities and select anchor points in the feature space, and hence application to both two scenarios can be efficient. In the next section, we carefully conduct extensive experiments to justify the effectiveness and extendibility of the proposed COX method under various settings.
EXPERIMENTS
In our experiments, we first elucidate the experimental details. Then, we provide performance com- parisons to various baseline methods on different datasets. Finally, we conduct empirical analyses to provide an intuitive understanding of the proposed method.
4.1 IMPLEMENTATION DETAILS
Datasets. We consider datasets with at least three modalities: 1) TVL dataset (Fu al., 2024) contains tactile sensing, RGB image, and class name which can be transformed into language; 2) LLVIP (Jia al., 2021) dataset has infrared thermal data, RGB image, and annotations for pedestrian detection. We follow Zhu et al. (2023) to crop the pedestrian and background which stand for two classes. Further, we use the OpenAl template (Radford et al., 2021) to create language description; 3) NYU-D dataset (Silberman et al., 2012) contains RGB image, depth data, and class name that can be transformed into language description as well; 4) VGGS dataset (Chen et al., 2020a) includes video data, corresponding sound, and the language description; 5) MSR-VTT (Xu et al., 2016) includes videos and text description, we break down the videos into video frames and the audio data; 6) MOSEI dataset (Zadeh et al., 2018) contains videos from 7 classes of emotions, we extract audio data from the videos and use the emotion type to create language descriptions.
Models. We employ two types of IM perceptors, namely ImageBind (Girdhar et al., 2023) and LanguageBind (Zhu et al., 2023) which correspondingly contain 6 and 5 encoders to process differ- ent modalities. We select one modality for each experiment as OOM and then choose the rest as IM. For IM data, we use the existing encoders to extract their features. As for OOM data, we conduct preprocessing to ensure its compatibility. Then, we initialize an OOM learner from scratch using VIT-T/16 to learn from the OOM data using the guidance from IM perceptors. Note that for the TVL dataset, there are no existing encoders to process tactile modality. Therefore, when the tactile modality is chosen as IM data, fine-tune the encoder using contrastive learning on the training
7

---

Published as a conference paper at ICLR 2025
Table 2: Classification performance comparison of different methods across multiple datasets with different OOM modalities.
TVL
LLVIP
NYU-D
VGGS
Setting
IM Perceptor
Method
RGB
Lan
Tac
RGB
Lan
The
RGB
Dep
Lan
Aud
Vid
Lan
Random
0.4
0.3
0.2
48.2
47.3
51.0
10.2
11.3
10.2
0.3
0.3
0.3
HRM
23.1
19.5
22.7
54.6
53.1
54.1
45.2
44.5
38.1
9.3
10.2
8.4
Semi Semi--Sperrised
ImageBind
EatMin
24.0
21.8
23.6
56.7
57.0
55.4
48.0
46.3
39.3
10.5
13.3
8.9
31.2
25.3
26.5
59.2
58.3
50.7
44.2
16.8
18.4
11.7
aligned
79.5
29.8
35.8
65.4
61.8
63.4
61.8
54.0
52.7
27.8
29.3
19.1
Rundom
0.4
0.3
0.2
48.2
47.3
51.0
10.2
11.3
10.2
0.3
0.3
0.3
ERM
23.6
20.1
22.6
56.5
54.9
58.3
44,8
44.5
39.9
9.8
13.7
9.9
LanguageBind
EatMin
25.7
23.1
25.1
49.4
47.3
42.7
14.5
12.8
COX
33,5
26.3
27.3
61.2
62.3
66.4
58.8
53.5
48.4
18.3
22.1
13.4
aligned
81.6
31.2
38.3
74.1
73.2
87.2
68.6
65.1
57.7
38.6
32.5
20.9
Random
0.4
0.3
0:2
48.2
47.3
$1.0
10.2
11.3
10.2
0.3
0.3
0.3
Unsupervised
ImageBind
COX
SSL
18.9
6.3
15.4
4.3
17.1
5.1
54.8
52.3
57.2
56.1
53.8
52.4
21.7
14.6
22.0
13.6
19.5
18.9
2.5
9.3
10.2
6.9
10.5
3.8
aligned
79.5
29.8
35.8
65.4
61.8
63.4
61.8
54.0
52.7
27.8
29.3
19.1
Random
0..4
0.3
0:2
48.2
47.3
$1.0
10.2
11.3
10.2
0.3
0,3
0.3
LanguageBind
SSL
6.8
6.5
5.1
54.6
57.8
53.8
16.9
18.1
16.3
7.2
5.6
4.8
COX
19.3
19.2
18.6
55.0
56.4
55.7
24.5
23,1
20.4
10.0
11.6
10.4
aligned
81.6
31.2
38.3
74.1
73.2
87.2
68.6
65.1
57.7
38.6
32.5
20.9
set. For ImageBind, the tactile encoder is aligned with the image encoder, and for LanguageBind, it aligned with the language encoder, which is the same as the original training process. For training the connection between modalities, employ multi-laye perceptrons to realize the variational in- formation bottleneck framework. Moreover, to obtain the optimal classifier from each in-modality, utilize the extracted features and train a linear layer as classification heads.
Setup. We consider two scenarios of OOM generalization: For the semi- supervisd case, sam- ple 10% of the training data as labeled data with each class having a balanced number of labels. For the unsupervised case, we have no labels at all. For selecting the number of anchor points, we choose the same number of examples for the warm-up and training phases, which is 10% of the total training set. To train the OOM learner, we use the Adam optimizer with initial learning rate of le 3 with weight decay 1e 5, and train the model for 50 epochs.
Baseline methods. Since there is no existing baseline method to compare with under our setting, we implement four methods for comparison, namely: Random where the model is randomly initial- ized, ERM where only labeled data is used to minimize the empirical risk, EntMin (Grandvalet & Bengio, 2004) which minimize the entropy of unlabeled data meanwhile conduct ERM, SSL which conducts eeff-supervised learning using Gaussian noise perturbation on the input, and MoCo He et al. (2020) which updates model parameters with ensembling and meanwhile conducts contrastive learning. Note that we use MoCo for comparison for the retrieval task in Table 3 because it is not for classification, and it is combined with EntMin the semi-supervised case. Moreover, we use a pre- trained encoder as upper- limit baseline "aligned". Next, we carefully compare the performance of our COX to these bascline methods
4.2 PERFORMANCE COMPARISON
For performance comparisons, we conduct classification and cross-modal retrieval to validate the proposed COX. There are seven modalities are considered, namely RGB image, language, tactile, thermal, depth, audio, and video which are simplified as RGB, Lan, Tac, The, Dep, Aud, and Vid, re- spectively. For each column, we choose one modality as OOM data, the rest modalities are selected IM data. For the retrieval task, we report the recall rate in both top 1 (R@) I) and top 5 (R@5). The results are shown in Tables 2 and We can see that the proposed COX clearly shows the best per- formance in both scenarios. Specifically, COX can achieve more than 5% performance improvement for most of the OOM setting, which justifies that leveraging the knowledge from IM perceptors can indeed help OOM generalization compared to using OOM data alone. Moreover, even though the performance is relatively limited compared to the fully pre-trained baseline under the unsupervised case, considering it is an extremely challenging setting, for over 10% compared to the Random baseline, which demonstrates that the unsupervised OOM generalization is indeed learnable further leads to novel research direction for improving the gen-
still largely improve the performance
8

---

Published as a conference paper at ICLR 2025
Table 3: Cross modal retrieval performance comparison of different methods across multiple datasets with different OOM modalities.
MSR-VTT
MOSEI
Setting
IM Perceptor
Method
And
Lan
Vid
Aud
Lan
Vid
R@1
R@5
R@1
R@5
R@
I
R@5
R@1
R@5
R@
R@5
R@1 I
R@8
Random
25.1
25.4
5.4
24.2
42.5
14.4
42.8
14.1
ERM
15.6
30.3
16.1
35.2
18.5
38.2
28.0
45.3
29.3
47.1
33.4
48.2
EmtMin
32.4
19.2
21.0
39.4
46.7
32.0
48.7
35.4
50.5
ImageBind
MuCo
20.5
33.9
21.1
38.9
23.4
43.5
30.1
47.3
32.7
50.1
36.2
51.0
35.8
23.4
26.5
48.0
38.8
Semi Supervied
Aligned
35.5
51.5
32.3
52.4
36.8
61.8
42,9
66.4
48.2
69.4
50.5
71.6
Random
5.2
24.3
5.4
25.1
5.0
25.6
13.5
43.1
14.2
42.7
14.6
41.9
ERM
16.3
31.1
16.5
36.2
18.7
37.9
27.3
45.5
28.4
47.6
33.4
49.3
EatMin
19.6
33.4
19.8
38.6
22.4
37.9
30.2
45.5
33.5
49.0
36.0
49.7
LangungeBind
MoCo
21.1
34.8
20.9
39.2
24.5
38.6
31.1
46.7
34.5
50.5
37.0
51.7
COX
25.2
36.0
24.1
40.0
28.7
49.5
34.6
49,8
34.6
50.2
39.2
55.4
Aligned
42.0
53.6
38.8
58.6
44.8
70.0
44.6
68.9
49.5
67.4
51.1
68.3
Random
5,4
25.1
5.0
25.4
5.4
24.2
14.3
42.5
14.4
42.8
14.1
42.1
SSL
28.4
28.1
10.1
29.5
48.8
16.2
45.2
16.0
45.0
MoCo
9.2
28.9
9.5
28.4
10.6
30.0
17.8
50.3
16.6
45.8
17.1
44.4
ImageBind
COX
30.4
16.5
15.2
34.8
53.7
18.2
Unsupersin viee
Aligned
35,5
51.5
33..3
52.4
36.8
61.8
42,9
66.4
48.2
69.4
50.5
71.6
Random
24.3
5,4
25.1
5.0
25.6
13.5
43.1
14.2
42.7
14.6
41,9
SST.
9.2
28.9
11.0
28.8
10.3
28.7
18.0
48.9
18.4
45.0
17.8
45.6
MoCo
9.6
29.4
11.1
28.5
11,0
29.3
18.8
50.7
18.5
45.2
18.0
45.5
LangungeBind
COX
14.8
31.1
18.4
34.4
15.4
35.0
23.1
52.8
19.4
47.2
20.4
49.9
Aligned
42.0 53.6
38.8
58.6
44.8
70.0
44.6
68.9
49.5
67.4
51.1
68.3
eralization performance. Additionally, note that the performance of COX is affected by the quality of IM perceptors, as using LanguageBind shows relatively higher performance compared to using ImageBind. Thus, it would be potentially helpful to leverage sophisticated IM perceptors benefit the generalization performance.
4.3 EMPIRICAL ANALYSIS
To provide an intuitive justification for the proposed method, here we conduct empirical analyses using the MSR-VTT dataset on various OOM scenarios and modalities.
0OM genera Video
Generalization Audio
OOM generalization
A2V MMD HNN
A2V
V2A
150
MND MMR
$003>
T2V ACC
T2A HND
12A A2C.AC
1955
ACC
366
$002
MND
MMD
50027
315
0.00
335
302
M015
352
A12
5000
0.00
Training Epoch
Training
Training Epoch
14
Figure 5: Connection effect on maximum mean discrepancy and accuracy across modalities.
Connection mitigates modality gap. To understand the performance of our VIB-based connec- tion learning, here we show its effect on generalization out-of-modal. Specifically, during connec- tion training, we compute the maximum mean discrepancy (MMD) between the mapping of each IM data and the OOM data. Mcanwhile, the IM mappings which are used to learn a classification head as the optimal classifier. Based on our theoretical result, we apply the classifiers to OOM data and compute their accuracies, Figure 5. We can see that as training goes on, the MMD between each IM mapping and OOM data decreasing and the corresponding accuracy is increasing, which shows that: 1) our connection can indeed close the modality gap between their features and 2) as the mappings of IM data getting close OOM data, the optimal classifier shows better classification results on OOM data, which benefits the knowledge transfer from known modalities to unknown ones.
evenly select 6 points during the training and extract
shown in
Modality disagreement identifies uncertainty. To understand the effect of modality disagree- ment, conduct analysis under the semi-supervisee scenario by training the OOM learner to use only labeled data for 10 epochs. Then, we leverage the modality disagreement criteria to sepa- rate OOM data into those with disagreement and agreement and show their prediction accuracies in

---

Published as a conference paper at ICLR 2025
Figure 6 (a). We see that the accuracy for OOM data with disagreement is significantly lower than those with agreement, meaning that the prediction uncertainty, i.e, data with low accuracy, is effectively identified by the proposed modality disagreement.
Modality agreement alleviates uncertainty. Further, we con- duct training by following the procedure proposed Section 3.3 and again show the accu- racies of OOM data with dis- agreement and agreement in Figure 6 (b). We can see that the performance gap be- tween the two types of data is largely mitigated, which justi- fies the methodology of explor- ing OOM data using the guid- ance of modality agreement. As result, we can achieve almost comparable performance on both types of data, benefiting the overall generalization performance.
Prediction Accuracy
Prediction Accuracy
Dbergremmnt:
Divegnamment
Agresmmntt
Mi
AuE
Lan
Vid
Aud
Lan
O0M
OOM Data
Figure 6: Prediction accuracy of OOM data with modality dis- agreement and modalities agreement, respectively. (a) Before ex- ploration. (b) After exploration.
Ablation study. Additionally, we conduct an abla- tion study to justify the effect of our methodology. Specifically, consider three ablations: 1) "w/o connection" where we remove the connection and directly apply the modality disagreement criteria on the original features of IM data and OOM data, 2) "wwo exploration" where we only leverage the OOM data with agreement for training, 3) For unsuper- vised scenario, we consider "w/o warm-up" where do not conduct the warm-up phase and directly training the model. The results in Table show that all modules are essential for achieving effective OOM generalization. Specifically, the connection is vital for the knowledge transduction of IM data to OOM data, without which the generalization performance is largely degraded. The conclusion consistent with the connection analysis where directly applying optimal classifiers across modal- ities leads to poor accuracy. Morcover, removing exploration also hinders the performance because the uniqueness of OOM data is largely ignored, Additionally, we find that the warm-up phase is essential for the unsupervised case. As initialized models have no classification capability, we need pre-training form basic feature clusters that are consistent with IM data, further enabling effective OOM generalization.
Table 4: Ablation study various settings.
MSR- VTT R@1
Setting
Ablation
Aud
Lan
Vid
w/o connection
7.9
10.3
Semi
w/o exploration
16.4
16.5
18.8
COX
25.2
24.1
40.0
w/o warm-up
7.4
11.5
10.5
Unsup.
COX
14.8
18.4
15.4
Discussion on computational efficiency. Note that we conduct the feature connection mostly on the feature space, the computational cost of training VIB framework work quite acceptable. The main cost is training the OOM learner which is the basic training with cross-entropy loss optimiza- tion and can be implemented on single NVIDIA 3090/4090 GPU.
5 CONCLUSION AND LIMITATION
In this paper, we study a novel and promising research direction dubbed Out-of-Modal (OOM) Gen- eralization which aims leverage knowledge from existing modalities generalize an unknown modality without instance-level correspondence. We consider two scenarios where there are a few correspondences and there is no correspondence, i.e.. semi-supervised and unsupervised cases, re- spectively. To tackle these problems, we propose Connect&Explore (COX) method which first learns connections across modalities knowledge of OOM data based on modality disagreement. Extensive experiments are conducted to justify the proposed method and intuitive insights are provided to inspire future studies. However, our research is limited to several aspects which we hope to address in the future. First, although challenging as it is, the performance is relatively limited compared to fully-aligned models, which requires more investigations to enhance generalization. Second, our OOM generalization is mostly conducted within the modalities from the same dataset. In the future, we hope to discover scenarios where the OOM data from different dataset with large modality gap.
extract common knowledge and then explores the unique
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENTTS
Bo Han was supported by RGC Young Collaborative Research Grant No. C2005-24Y and RIKEN Collaborative Research Fund. Masashi Sugiyama was supported by JST ASPIRE Grant Number JPMJAP24055 Tongliang Liu was partially supported by the following Australian Research Council projects: FT2201003 DP220102121, LP220100527, LP220200949, and IC190100031, he was also supported by JST ASPIRE Grant Number JPMIAP2405.
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, volume 35, pp. 1-7336,, 2022.
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.004// 2016.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In NeurIPS, volume 32, 2019.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2 07258, 2021.
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Bayesian-guider label mapping for visual reprogramming, In NewrlPS, 2024a.
Chengyi Cai, Zesheng Ye, Lei Feng. Jianzhong Qi, and Feng Liu. Sample-specific masks for visual reprogramming based prompting. In ICML, 2024b.
Lincan Cai, Shuang Li, Wenxuan Ma, Jingxuan Kang, Binhui Xie, Zixun Sun, and Chengwei Zhu. Enhancing cross-modal fine-tuning with gradually intermediate modality generation. arXiv preprint ar arXiv:2406.099
2024c.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020a.
Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistically-inforned interpolation hidden space for semi-supervised text classification. arXiv preprint arXiv:2004. /2239, 2020b.
Runnan Chen, Youquan Liu, Lingdong Kong, Nenglun Chen, Xinge Zhu, Yuexin Ma, Tongliang Liu, and Wenping Wang. Towards label-free scene understanding by vision foundation models. In NeurlPS, volume 36, pp. 75896-75910, 2023a.
Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In CVPR. pp. 7020-7030, 2023b.
Shiming Chen, Ziming Hong, Wenjin Hou, Guo-Sen Xie, Yibing Song, Jian Zhao, Xinge You, Shuicheng Yan, and Ling Shao. Transzerot+: Cross attribute-guin -uiteedggudddd transformer for zero-shot learning. IEEE transactions pattern analysis and machine intelligence, 12844)::2444-22811 2022a.
Shiming Chen, Ziming Hong, Yang Liu, Guo-Sen Xie, Baigui Sun, Hao Li, Qinmu Peng. Ke Lu, and Xinge You. Transzero: Attribut-guiddd transformer for zero-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 330-338, 2022b.
Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Understanding and improving feature learning for distt--iistiitution generalization. In NeurIPS, volume 36, pp. 68221-68275. 2023c.
11

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprins 1eXiv:18100048800 2018.
Hao Dong. Ismail Nejjar, Han Sun, Eleni Chatzi, and Olga Fink. Simmmdg: A simple and effective framework for multi-modal domain generalization. NeurIPS, 36.78674-78695, 2023.
Yingying Fang, Shuang Wu, Sheng Zhang, Chaoyan Huang, Tieyong Zeng, Xiaodan Xing, Simon Walsh, and Guang Yang. Dynamic multimodal information bottleneck for multimodality classifi- cation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 7696-7706, 2024.
Letian Fu, Gaurav Datta, Huang Huang, William Chung- Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, and Ken Goldberg. A touch, vision, and language dataset for multimodal alignment. In ICML, 2024. URL https::/p/// / //ppprpeeeeeeeee//////eeeee
mert/ooum? F=FFOOO09eH0.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023.
Madelyn Glymour, Judea Pearl, and Nicholas Jewell. Causal inference in statistics: primer. John Wiley & Sons, 2016.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. NeurIPS, 17, 2004.
Heidi E Harley, Erika A Putman, and Herbert L Roitblat. Bottlenose dolphins perceive object fea- tures through echolocation. Nature, 424(6949))667-669 2003.
Masukichi Hashimoto. Origin of the compass. Memoirs of the Research Department of the Toyo Bunko (The Oriental Library), 1:69-92 92, 1926.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020.
Rundong He, Yicong Dong, Lan-Zhe Guo, Yilong Yin, and Tailin Wu. Re-evaluating the impact of unseen-class unlabeled data on semi-supervised learning model. In ICLR, 2025.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of dut-of-distribution generalization. In ICCV, pp. 8340-8349, 2021.
Ziming Hong, Shiming Chen, Guo-Sen Xie, Wenhan Yang. Jian Zhao, Yuanjie Shao, Qinmu Peng. and Xinge You. Semantic compression embedding for generative zero-shot learning. IJCAI, pp. 956-963, 2022.
Ziming Hong, Zhenyi Wang, Li Shen, Yu Yao, Zhuo Huang. Shiming Chen, Chuanwu Yang, Ming- ming Gong, and Tongliang Liu. Improving non-tran -ontrransferabee representation learning by harnessing content and style. In ICLR, 2024.
Yu Huang, Chenzhuang Du Zihui Xue, Xuanyao Chen, Hang Zhao, and Longbo Huang. What makes multi-modal learning better than single (provably). In NeurIPS, volume 34, pp. 10944- 10956, 2021.
Zhuo Huang, Li Shen, Jun Yu, Bo Han, and Tongliang Liu. Flatmatch: Bridging labeled data and unlabeled data with cross-sharpness for semi cemi-supervised learning. In NeurIPS, volume 36, pp. 18474-18494, 2023a.
12

---

Published as a conference paper at ICLR 2025
Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing ousto--distiiuution examples via augmenting content and style. In ICLR, 2023b.
Zhuo Huang, Muyang Li, Li Shen, Jun Yu, Chen Gong, Bo Han, and Tongliang Liu. Winning prize comes from losing tickets: Improve invariant learning by exploring variant parameters for cut-of-distribution generalization. International Journal of Computer Vision, pp. 1-19, 2024.
Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. Livip: A visible-infrared paired dataset for low-light vision. In ICCV, pp. 1496-35044 2021.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: Robot manipulation with multimodal prompts. In ICMI., pp. 14975-15022. PMLR, 2023.
Diederik P Kingma and Max Welling. arXiv:1312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
Mingyu Li, Tao Zhou, Zhuo Huang, Jian Yang. Jie Yang, and Chen Gong. Dynamic weighted
adversarial learning for semi-supervised classification under intersectional class mismatch. ACM Transactions on Multimedia Computing, Communications and Applications, 20(4): 1-2,, 2024a.
Muyang Li, Runze Wu, Haoyu Liu, Jun Yu, Xun Yang, Bo Han, and Tongliang Liu. Instant: Semi- supervised learning with instance-dependent thresholds. In NeurlPS, volume 36, pp. 2922-2938, 2023.
Xiu-Chuan Li and Tongliang Liu. Efficient and trustworthy causal discovery with latent variables and complex relations. In ICLR, 2025.
Xiu-Chuan Li, Kun Zhang, and Tongliang Liu. Causal structure recovery with latent variables under milder distributional and graphical assumptions. In ICLR, 2024b.
Xiu-Chuan Li, Jun Wang, and Tongliang Liu. Recovery of causal graph involving latent variables via homologous surrogates. In ICLR, 2025.
Paul Pu Liang, Peter Wu, Liu Ziyin, Louis-Philippe Philippe Morency, and Ruslan Salakhutdinov. Cross- modal generalization: Learning in low resource modalities via meta-alignment. In Proceedings of the 29th ACM International Conference Multimedia, pp. 2680-2689 2021.
Paul Pu Liang, Chun Kai Ling, Yun Cheng, Alex Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, Louis-Philippee Morency, and Ruslan Salakhutdinov. Multimodal learning without labeled multimodal data: Guarantees and applications. arXiv preprint 2xXx:233060055399 2023.
Runqi Lin, Chaojian Yu, Bo Han, and Tongliang Liu. On the overomeimiinttion during natural, robust and catastrophic overfitting. arXiv preprint arXiv:2310.08847, 2023a.
Runqi Lin, Chaojian Yu, and Tongliang Liu. Eliminating catastrophic overfitting via abnormal adversarial examples regularization. NeurlPS, 36:67866-678888 67885, 2023b.
Runqi Lin, Chaojian Yu, Bo Han, Hang Su, and Tongliang Liu. Layer-aware analysis of catastrophic overfitting: revealing the pseudo-robust shortcut dependency. arXiv preprint arXiv:2405. 16262, 2024.
Yexiong Lin, Yu Yao, Xiaolong Shi, Mingming Gong, Xu Shen, Dong Xu, and Tongliang Liu. Cs- isolate: Extracting hard confident examples by content and style isolation. In NeurlPS, volume 36, pp. 58556-58576, 2023c.
Yexiong Lin, Yu Yao, and Tongliang Liu. Learning the latent causal structure for modeling label 2025.
noise. In NeurlPS, volume 37, pp.
Xiaohao Liu, Xiaobo Xia, Zhuo Huang, and Tat-Seng Chua. Towards modality generalization: A benchmark and prospective analysis. arXiv preprint 12. 2024.
Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei Liu. Segment any point cloud sequences by distilling vision foundation models. In NewrlPS, volume 36, pp. 37193-37229, 2023.
13

---

Published as a conference paper at ICLR 2025
Zhou Lu. A theory of multimodal learning. NeurIPS, 36, 2024.
Shuang Ma, Daniel McDuff, and Yale Song. Unpaired image-to-speed synthesis with multimodal information bottleneck. In ICCV, pp. 7598-7607, 2019.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim. Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In ICML, pp. 689-696, 2011.
Maja Pantic and Leon JM Rothkrantz. Toward an sffect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91(9):1370-133 2003.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763. PMLR, 2021.
Muhammad Arifur Rahman, David Brown, Nicholas Shopland, Andrew Burton, and Mufti Mah- mud. Explainable multimodal machine learning for engagement analysis by continuous perfor- mance test. In International Conference on Human-Compuler Interaction, pp. 386-399. Springer, 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurlPS, 28, 2015.
Leslic Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In ICML, pp. 8093-8104. PMLR, 2020.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In ICCV, pp. 9339-93477 2019.
Junhong Shen, Liam Li. Lucio M Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-modal fine-tuning: Align then refine. In ICML, pp. 31030-31055 PMLR, 2023.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In Computer Vision-ECCC 2012: 12th European Conference Computer Vision, Florence, Itll,, October 7-13, 2012, Proceedings, Part V 12, pp. 746-760. Springer, 2012.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. NeurlPS, 26, 2013.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In NeurlPS, volume 33, pp. 596-608, 2020.
Karthik Sridharan and Sham Kakade, An information theoretic framework for multi-view learn- ing. In COLT, number 114, pp. 403-414, 2008
Xinwei Sun, Yilun Xu, Peng Cao, Yuqing Kong, Lingjing Hu, Shanghang Zhang, and Yizhou -temmation-thhorttic
Wang. Tegm: An information theoretic framework for semi-supervised multi-modality learn- ing. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 171-188. Springer, 2020.
Hugo Touvron, Thibaut Lavril Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23771 2023.
Weijie Tu, Weijian Deng, and Tom Gedeon. A closer look at the robustness of contrastive language- image pre-training (CLIP). In Thirty-seventh Conference on Neural Information Processing Sys- tems, 2023. URL https:/ wpwpoeview.wwee////ppppw wwtp:/nttttttt///eeeeeee Sooouoiiid--e wwt.mm/iiddwiMMeeeedeeM
Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, and Tom Gedeon. An empirical study into what matters for calibrating lanion-language models. In JCML, 2024a. URL https:// openreview net forum? wtwoormmddggouueeeee dd-goxusshh7s
14

---

Published as a conference paper at ICLR 2025
Weijie Tu, Weijian Deng, and Tom Gedeon. Toward a holistic evaluation of robustness in clip models. arXiv preprint arXiv:2410.01534, 2024b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. NewrIPS, 30, 2017.
Haoyu Wang, Zhuo Huang, Zhiwei Lin, and Tongliang Liu. Noisegpt: Label noise detection and rectification through probability curvature. In NeurlPS, 2024a.
Qikai Wang, Rundong He, Yongshun Gong, Chunxiao Ren, Haoliang Sun, Xiaoshui Huang, and Yi- long Yin. Diverse teacher-students for deep safe semi superviie learning under class mismatch. arXiv preprint arXiv: 16093, 2024b.
Qizhou Wang, Yong Lin, Yongqiang Chen, Ludwig Schmidt, Bo Han, and Tong Zhang. A sober look at the robustness of clips to spurious features. In NeurlPS, 2024c.
Qizhou Wang, Bo Han, Yang Liu, Chen Gong, Tongliang Liu, and Jiming Liu. W-doe: Wasser- stein distrobution-annosiic outlier exposure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, al. Freematch: Self-adaptive thresholding for semi-supervised learning. arXiv preprint arXiv:2055.07246, 2022.
Ying Wang, Tim GJ Rudner, and Andrew G Wilson. Visual explanations of image-text representa- tions via multi-modal information bottleneck attribution. NeurlPS, 36:16099-16027, 2023.
Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, et al. Freebind: Free lunch in unified multimodal space via knowledge fusion. In ICML, 2024d.
LA Weinstein. Electromagnetic waves. Radio i svyazz' Moscow, 1988.
Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint 15X1v:1004.2515, 2010.
Yuhao Wu, Jiangchao Yao, Bo Han, Lina Yao, and Tongliang Liu. Unraveling the impact of het- erophilic structures graph positive-unlabeled learning. In ICML, 2024a.
Yuhao Wu, Jiangehao Yao, Xiaobo Xia, Jun Yu, Ruxin Wang, Bo Han, and Tongliang Liu. Mitigat- ing label noise on graphs via topological sample selection. In ICML, 2024b.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurlPS, volume 32, 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurlPS, volume 33, pp. 7597-7610, 2020.
Xiaobo Xia, Bo Han, Nannan Wang, Jiankang Deng, Jiatong Li, Yinian Mao, and Tongliang Liu. Extended t: Learning with mixed closed-set and open-set noisy labels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Yan Xia, Hai Huang, Jieming Zhu, and Zhou Zhao. Achieving cross modal generalization with multimodal unified representation. NeurIPS, 36, 2024.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
Zihui Xue, Zhengqi Gao, Sucheng Ren, and Hang Zhao. The modality focusing hypothesis: Towards understanding crossmodal knowledge distillation. arXiv preprint arXiv:2206. 06487, 2022.
15

---

Published as a conference paper at ICLR 2025
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. NeurlPS, 13:7260-7271, 2020.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under a structural causal model. In NeurIPS, volume 34, pp. 4409-4420, 2021.
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-augmented multimodal language modeling. In ICML, 2023.
Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, Jae Sung Park, Ximing Lu, Rowan Zellers, Prithviraj Ammanabrolu, Ronan Le Bras, Gunhee Kim, al. Fusing pre-trained language models with multimodal prompts through reinforcement learning. In CVPR, pp. 10845-10856, 2023.
Suqin Yuan, Lei Feng, and Tongliang Liu. Late stopping: Avoiding confidently learning from mislabeled examples. In ICCV, pp. 16079-16088, 2023.
Suqin Yuan, Lei Feng, and Tongliang Liu. Early stopping against label noise without validation data. In ICLR, 2024.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 2236-2246, 2018.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. CVPR, pp. 833-842, 2021.
Liangl Zhen, Peng Hu, Xu Wang, and Dezhong Peng. Deep supervised cross-modal retrieval. In CVPR. pp. 10394-10400 10493, 2019.
Jiyang Zheng, Weihao Li, Jie Hong, Lars Petersson, and Nick Barnes. Towards open-set object detection and discovery. In CVPR, pp. 3961-3970, 2022.
Jiyang Zheng, Yu Yao, Bo Han, Dadong Wang, and Tongliang Liu. Enhancing contrastive learning for ordinal regression via ordinal content preserved data augmentation. In ICLR, 2024.
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n- modality by language-based semantic alignment. arXiv preprint arXlv:2310.01852, 2023.
16

---

Published as a conference paper at ICLR 2025
In the appendix, first provide a detailed derivation of our VIB framework. Then, carefully prove the modality disagreement in Theorem 1, Further, we provide additional experiments to eval- uate our method. Finally, give discussion the relationship between existing studies and provide a prospective outlook for future research.
A LOWER BOUND OF OUR VIB FRAMEWORK
Recall that we have the following factorization:
p(X!, XX V,Y) PVV , X')x(XXXXX)((X)
(16)
with Markov chains V X X and X Y 44 X
Our goal is maximize the information
redundancy (Liang et al., 2023; Williams & Beer, 2010):
max ;(x=; x1; I(xx; 1(xxx
(17)
where the first term lower-bounded by:
;x0xx1)> /
du de det p( (x))p((l log glx (u)p(ul?))
(18)
Then, consider the second term
1(x"; f de da'dypp ,at v) log p(x01y)p(=*1))
p(x ,
(19)
= =f de dz'dypl x², v) log plylx0)
pl x2 HYY HXX
(20)
Note that we use the factorization p(x v) = p(y|=' ((=0)))( and further ignore the en- tropy terms*, then we have:
T(XO; (XI:))M))) da ddddddyypllla" (x (=")pp"")logpllll"" 9 (()) log h(y) (21)
which is based on the positivity of KI. divergence between our classifier (y)x::) and p(y) [ 0).
To this end, we can lower-bound our objective by combining Eqs. and 21:
(x))e(/>>)))))) 1(xxxx)))> fdx da dz p(a (p(el=") log q(x (o)p(((x")
(22)
- duoddttyppul (1) )p(x)) logplul ((ppp((()ppp(((a))))))))))())))) 1 pp=)) log h(gl Lron
(23)
B PROOF OF THEOREM
Proof.
Assumption 1 (Relaxed triangle inequality). For the distance function d y x y - R+, there exists ca 2 such that Vd,, 92. U3 e Yd(0i, 02) S ca(d(01, 03)) + d(02. Da)).
Assumption (Inverse Lipschitz condition). For the function d. it holds that Yh, E[d(h(x1, 22), h² (x1, 2, and I2; and B[d(h(x)) (x)) < [(h) L(h")), where h" is the Bayes optimal classifier on r.
x2)) < [(h) L(h*)), where h" is the Bayes optimal classifier both
Assumption 3 (Classifier optimality). For any classifiers h in comparison to the Bayes' optimal classifier h², there exists constants > 0 such that [C(h) L(h*)) <.
"We focus the optimization of (Y xx), and p(Y (|) is given and frozen in our setting.
17

---

Published as a conference paper at ICLR 2025
To bridge hī and h2. we use h and h to denote the Bayes' optimal classifier on both IM data and all data, respectively. Then, we capture the relationship between the uniqueness of OOM data given both IM data and the difference in their Bayes' optimal prediction errors:
(2(, _(h*))2 Ex E [.H.xxxxoo00 (at, 22, ).
(() (#) X2), v)
(24)
By xx E(h (0). an x ), -
Rxxx|xx xx),
(25)
KL(p(yl=) my 2 =(y)
(26)
< ExxLL((pvll di ) p(llal x/))
(27)
= 1(x,, ,),
(28)
Then, we first capture the redundancy between one IM data and OOM data given another IM data:
((h)) - 2(h*))²= Ey xl.x (o" (2), 2l. x)) ). y) Ex ((
(29)
S Xx E (a) 22, By (x)))))))))))))
(30)
s (L(p(ula) ay x p())=())
(31)
s Ex ExxL(p(vlu), x2, x p(yl=)))
(32)
1xx X2 )(x)
(33)
Further leveraging triangle inequality through the Bayes' optimal classifier h² and the inverse Lips- chitz condition, we have:
E =(
(d [d(h) h1,2) S E p Ep al [d(h",hi,2)
(34)
s (c())) - ( - 2)
(35)
< (XX,,XX,, x|x1) +
(36)
Symmetrically, we have [C(h)) - 2( < 1(X x1.Y and further obtain
Ep(r) w/ [d(hs, hi)) & 1(X X ((x)) + (xx, X2). Then combining with Eq. 36:
E Epte 1 [d(,, hD) - I(X" X +
./. y|x!) + 2 Y|X],
(37)
based on the decomposition of the task-related mutual information of x 1(x Y)
F .,,, X|X + I(x", X X|) + I(x*, Y Y X²) + [(X", Xt, X Y), as shown in
Figure 3, can achieve:
a(hi, (5) = pp(e). (d(; k9) < 1(x Y) - 1(x", x x Y) + 1(x , (38)
C ADDITIONAL EXPERIMENTS
We conduct additional experiments to further justify the proposed COX. First, we study the per- formance benefits brought by COX under various correspondence rates in OOM data. Specifically, we choose MSR-VTI and NYU-D datasets and use Vid and Dep as OOM modalities, respectively, and show the result in Figure 7. First of all, we observe that COX brings more benefits when correspondence is more scarce. This because sufficient correspondence can maximally uncover the knowledge of OOM data. As correspondence gets less, the knowledge that be explored from correspondence decreases. However, COX leverages the knowledge from IM data which brings more benefits with less correspondence. Thus, the increased benefits of COX under low-correspondence scenarios demonstrate its effectiveness in tackling OOM generalization with- out correspondence.
Moreover, we testify how varied performance levels IM perceptors could affect the OOM perfor- mance. To achieve this, we change the number of IM data in each dataset 10%, 40%, 70%, and
18

---

Published as a conference paper at ICLR 2025
MSR-VTT with Various Correspondence Rates
NYU-D with Various Correspondence Rates
EntMas
Performance Improvement
Performance improvementt
Performance
COM Performance
56-
45
30%
60%
30%
10%
30%
30% Correspandence Rate
60%
10%
Correspandence Rate
Figure 7: Performance benefits brought by COX under various correspondence rate in OOM data.
Perceptor Performance
VTT
Perceptor
IM: IM: Lan
IM Lan
COM Performance
COM Porformancc
Performance
GDN: Vid
DOM: Dep
15%
40r
16%
4%
Perceptor Level
IM Perceptor Level
Figure Effect of Varying IM Perceptor Performance Level.
MSR-VTT
Vision
NYU-D
Language
Setting
Method
MSR-VTT
NYU-D
FreeMatch
45.2
MixText
21.2 23.4
MoCo
30.0
15.7
Unsup
COX
52.3
COX
MoCo+COX
35.4
23.8
Table Comparison with competitive uni- modal methods from Vision and Language.
Combining COX with MoCo for knowledge extraction from OOM to IM data.
Table
100%, and test the OOM performance of COX, shown in Figure 8. We that the OOM performance is significantly affected by the accuracy level of IM perceptors. When the performance of IM perceptors improves, the OOM performance of COX is also enhanced. Therefore, improving the performance of IM perceptors is vital for OOM generalization using COX.
Further, to understand the contribution of COX uni-modal study, conduct comparison and combination with uni-modal methods. First, we consider two uni-modalities vision and language from MSR- VTT and NYU-D datasets, respectively. By comparing to FreeMatch (Wang et al., 2022) and MixText (Chen et al., 2020b), two competitive semi-supervised learning methods that correspondingly deal with vision and language data, we show the performance of COX in Table 5. Even though the two baselines were effective under their original setting, their performance is still limited when applied to challenging multi-modal datasets with scarce knowledge. As we can see, COX still shows very effective performance compared to them, again justifying the benefits from COX by leveraging IM data.
Then, consider combining COX with the well-known unsupervised method MoCo (He et al., 2020) and show the performance benefits brought by COX for enhancing unknown modality. We show the result in Table 6. Delightfully, observe significant performance improvement on both
19

---

Published as a conference paper at ICLR 2025
Vid from MSR-VTT and RGB from the NYU-D dataset. This is because COX unleashes the po- tential label information from IM data to enhance label prediction of OOM data, i.e. Vid and RGB here. Such finding implies that COX can extract knowledge from other modalities to enhance new ones, which is the main goal of this study.
D RETROSPECTIVE AND PROSPECTIVE DISCUSSION
OOM generalization without instance-level correspondence presents challenges that could be related to existing studies but differentiates from them. Here we identify the relationship between OOM generalization and existing fields and discuss how we can be inspired to solve OOM generalization future studies.
Out-of-Distii Distr--Dittiittioon (OOD) Generalization (Hendrycks et al., 2021; Chen et al., 2023c; Huang et al., 2023b; 2024; Wang et al., 2025) where the goal is generalizing to unknown data distributions given several existing ones. The difference between OOM and OOD generalization lies in the data gap: the former faces modality gap while the latter deals with distribution gap. Intuitively, the modality gap demonstrates the change of learning space, thus it is more challenging than the distribution gap where data is still sampled within the same learning space, i.e., same dimension and formats. To solve OOM generalization, comprehensive understanding of how modalities are generated is required. For example, the surface of an object affects the reflection of light, thus deciding its visual information. Meanwhile, the texture of the surface also influences its tactile sensing. Therefore, we hypothesize that some modalities are commonly influenced by a hidden factor (Glymour et al., 2016; Li et al., 2024b; 2025; Li & Liu, 2025; Lin et al., 2023e; 2025; Zheng et al., 2024). By identifying such a factor, the shared information across modalities could be discovered, further benefiting OOM generalization.
Trustworthy Machine Learning aims to develop reliable, robust, explainable models under realistic scenarios, which considers various types of problems, such as Semi-Supervised Learning (SSL) (Berthelot et al., 2019; Sohn et al., 2020; Huang et al., 2023a; He et al., 2025; Li et al., 2023; 2024a; Wang et al., 2024b), Learning with Noisy Labels (LNL) (Xia et al., 2019; 2020; 2022; Yao et al., 2021; 2020; Yuan al., 2023; 2024; Wang et al., 2024a; Wu et al., 2024b;a), and Robustness (Rice et al., 2020; Lin et al., 2023a;b; 2024; Hong et al., 2024). Compared to the existing studies, OOM generalization without instance- level correspondence requires linking between data across modalities. During this process, unpaired data and noisy correspondence could occur, further hindering the generalization performance. To solve this problem, progressively generating pseudo correspondence and denoising the noisy ones as done by SSL and LNL might be helpful to complement the correspondence. Therefore, future studies for pairing multi-modal data through trustworthy machine learning techniques could be a promising direction.
Foundation Models (Chen al., 2023b;a; Radford et al., 2021; Touvron al., 2023; Bom- masani et al., 2021; Liu et al., 2023; Cai et al., 2024b;a) are pre-trained on large-scale data possess powerful zero-shot generalization abilities (Wang al., 2024c; Hong al., 2022; Chen et al., 2022b:a; Tu et al., 2023; 2024a;b; Zheng et al., 2022) have been widely used in practice nowadays. In OOM generalization, we rely on existing foundation models to encode modalities into a shared latent space. Therefore, the performance of our study is highly related to the capabilities of the foundation models. Common strategies either fuse all modalities into a uni-modal model or separately encode and align. Because of the cost multi-modie correspondence, the former multi-modal fusion strategy might be impracti- cal for large-scale applications. Therefore, alignment between pre-trained encoders could be a promising direction. To effectively organize multiple foundation models to achieve 0OM generalization, more comprehensive studies on addressing the modality gap, modal- ity imbalance, and alignment strategy are needed in future research.
20


Reviews:
[{"review_id": "ORmpRSEP58", "reviewer": "Reviewer_2YMo", "summary": "This paper proposes a novel method for Out-of-Modal (OOM) generalization, which uses a COX framework to achieve cross-modal knowledge transfer and generalization to unknown modalities. At the same time, this paper considers both semi-supervised and unsupervised scenarios. Experiments show that this method has a certain level of effectiveness.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The motivation behind this approach is clear, and studying out-of-model (OOM) generalization is meaningful.\n2. The authors provide a detailed description of the proposed method, making it easy to follow and reproduce.", "weaknesses": "1. In the theoretical analysis, the paper uses $h^*$ to represent ideal optimal classifiers. However, in practice, it is difficult to find the optimal classifier. Although the authors compared the performance of ImageBind and LanguageBind, they should conduct more experiments to demonstrate the impact of using classifiers with different levels of accuracy on the final results.\n2. The paper does not theoretically or experimentally explain why using the IM Perceptor is more effective than directly performing semi-supervised or unsupervised training.\n3. If using the IM Perceptor indeed improves training results as mentioned in the issues above, then for the same task, would knowing more modalities enhance the generalization of OOM?\n4. The paper contains numerous writing errors, such as \"OMM\" in line 494. Additionally, the logic of symbol definitions in the mathematical proof section is confusing, for example, using $X^O$ to represent IM data and $X^I$ to represent OOM data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dhp7jpMKAE", "reviewer": "Reviewer_Zg1q", "summary": "This paper explores a novel model generalization problem, specifically focusing on out-of-modal generalization in the absence of label and modality correspondence. The authors employ a two-stage generalization approach, first constructing modality-shared information and then focusing on modality-specific information. They propose different training losses for both semi-supervised and completely unsupervised scenarios.", "questions": "1.Does the inclusion of $L_{con}$ with label $ y $ imply that exploring connections across modalities requires the use of samples with correspondence for training?\n\n2.In a completely unsupervised setting, the feature distributions of different modalities may differ significantly. Is the choice of the anchor method reasonable in this context?\n\n3.Could you provide a more detailed proof regarding the upper and lower bounds in Sec 3.2? \n\n4.In the ablation experiments, the results of removing any module are significantly lower than those of EntMin. Would applying these two losses individually on EntMin lead to improvements?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The problem addressed in the paper is quite novel, and the semi-supervised scenario provides valuable insights.\"\n\n2.Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.", "weaknesses": "1.The paper contains several typos that affect readability and need to be corrected. Regarding line 195, it seems that the symbols for IM data $ X^O$ and OOM data $ X^I$ are used incorrectly. Also in Eq.3, why is there an integral sign before $p ( X^O, X^I ) $\n\n2.The proof in the paper lacks rigor. How is the lower bound derived from Equation 3 to Equation 4? Although the proof draws on VIB, there are still significant discrepancies.\n\n3.Similarly, the proof in Appendix A.1 for Equation 20, which relies on the non-negativity of the KL divergence, is not sufficiently rigorous.\n\n4.In Section 3.2, there is a lack of necessary descriptions regarding the trained model, such as the input and output of the decoder $q( X^O, X^I)$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OcoFp1MCEi", "reviewer": "Reviewer_PuJV", "summary": "This work focuses on the semi-supervised and  unsupervised scenarios of *Out-of-Model* (OOM) Generalization without given instance-level model correspondence. This work proposes an OOM generalization method based on the interactive relationship between modalities, *connect & explore* (COX), in an attempt to extract the commonality that can help partially comprehend OOM data based on IM data. It introduce a variational information bottleneck framework to connect OOM data and In-Model (IM) data and extract shared information. Finally, several sets of experiments demonstrate the effectiveness and extensibility of the proposed method.", "questions": "Please see 'weakness', which simply can be summarised as:\n\n(i) What the advantages of this work vs. previous works?\n\n(ii) What is \"prediction uncertainty\" described in the article and how is it reflected? \n\n(iii) Can the conclusion that COX performance is affected by quality of IM perceptors be fully explained based on the final performance results of only two IM perceptors, ImageBind and LanguageBind? What are the quality differences between ImageBind and LanguageBind reflected?\n\n(iv) How do the data in Table 2 and Table 3 reflect that leveraging the knowledge from IM perceptors can indeed help OOM generalization compared to using OOM data alone?\n\n(v) There are some grammatical errors and confusing notations in the article. Can it be further polished and revised?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work focuses on the novel and practical issues in the field of *Out-of-Model* (OOM) generalization.\n\nThis work proposes an approach that attempts to combine common knowledge based on connections across modalities with unique knowledge of OOM data based on modality disagreement.\n\nThis work conducts multiple sets of experiments to verify the effectiveness and extensibility of the proposed method.", "weaknesses": "(i) The OOM problem has also been discussed in other works, such as IB for modality missing, cross-modal generalization, etc., ([1-5] for example) but not mentioned in the paper (nor experiments). Please explain the difference between this paper and these works. [1] Unpaired image-to-speech synthesis with multimodal information bottleneck. [2] Visual explanations of image-text representations via multi-modal information bottleneck attribution. [3] Dynamic Multimodal Information Bottleneck for Multimodality Classification. [4] SimMMDG: A simple and effective framework for multi-modal domain generalization\n\n(ii) The work mentions \"prediction uncertainty\" in section 4.2, but after carefully reviewing all the contents, it is hard to find the definition or description of prediction uncertainty, while without the direct description of this concept in the experiment section for verification. \n\n(iii)The work mentions \"the performance of COX is affected by the quality of IM perceptors\", but only contrast the performance between ImageBind and LanguageBind in the experiment section can not fully illustrate this conclusion.\n\n(iv) Has some loopholes, e.g.,\n\n  - Tense and grammatical errors, e.g., \"OMM data\" -> \"OOM data\"\n  - Confusing notations, e.g.,\"given IM data *X<sup> O </sup>*and OOM data *X<sup> I </sup>*\"\n\nPlease correct the grammatical mistakes and polish them if possible.\n\n(v) In the experimental part, the results in Table 2 and Table 3 can not adequately illustrate this conclusion \"leveraging the knowledge from IM perceptors can indeed help OOM generalization compared to using OOM data alone\". How do the results in Table 2 and Table 3 show the performance improvement of the proposed method compared to the methods using only OOM data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "thkVLqMUmv", "reviewer": "Reviewer_oDE6", "summary": "This paper addresses the problem of out-of-model (OOM) generation and proposes a novel technique, COX. The authors perform experiments on five datasets, comparing their method with several baseline approaches.", "questions": "From my understanding, the authors consider a scenario where the training data consists of two parts: IM, where both inputs and labels are available, and OOM, where only inputs are provided without any labels (unsupervised case), and only a limited amount of labeled data is present. Please correct me if I’m mistaken, but it seems that no data contains both IM and OOM inputs. In other words, there is no data where the pair (x1, x2)—representing two modalities—is known.\n\nIf the above assumption is correct, how do we sample from the joint distribution p(xO | xI)p(xI) in Equation (6)?\n\nI find the assumption stated below Equation (1) too strong. It implies that the proposed method only applies when xO can be directly inferred from xI. For example, let’s assume a1 = a2 = a, representing the sound of a dog barking, and b1 and b2 represent images of a golden retriever and a border collie, respectively. V is a constant vector, and Y represents the class \"golden retriever.\" While it is true that P(V|xI=a1)P(Y|xI=a1) = P(V|xI=a2)P(Y|xI=a2), given that a1 = a2, we see that P(V,Y=is_golden_retriever|xO=b1, xI=a1) does not equal P(V,Y=is_golden_retriever|xO=b2, xI=a2).\n\nIn Line 255, do h1* and h2* refer to the classifiers corresponding to modality 1 and modality 2, respectively? If so, how can they be applied to xO, which belongs to a different modality?\n\nThe baselines should be reconsidered, especially by including stronger baselines. For instance, there are many state-of-the-art semi-supervised learning (SSL) approaches, and simply adding Gaussian perturbations to the input may not be sufficient. Stronger SSL baselines are crucial, as otherwise, it could be argued that using SSL alone is enough, which raises the question, why COX? Additionally, the 'Random' baseline is too simplistic, effectively leaving the unsupervised setting in Table 2 with only one baseline. \n\nFurthermore, for the semi-supervised case, it is not surprising that the baseline performs worse, given the very limited amount of labeled data used for training.\n\nI would like to bring [1] to the authors' attention, which could be helpful for better understanding multimodality:\n\n[1] Zihui Xue et al., The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation, ICLR 2022.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The problem statement is clear.", "weaknesses": "Insufficient and weak baselines: I am especially concerned the proposed approach's performance compared with SSL. \n\nSome notations should be improved, and a few detailed explanations should be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YbzsFvfZh2", "reviewer": "Reviewer_KNt4", "summary": "This paper propose a novel out-of-modal generalization problem that focuses on generalizing learned knowledge to unknown modalities. In addition, it considers both semi-supervised and unsupervised scenarios. The proposed connect & explore (COX) scheme tries to explore the shared and unique between modalities via variational information bottleneck. The proposed method can build connections between known and unknown modalities, further generating correspondences to explore more knowledge. The expleriments verify the effectiveness of COX.", "questions": "see in Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel OOM problem, which is valuable because of the importance on generalizing exsting knowledge to unknown modalities without paired instances. \n2. The proposed COX method sounds reasonable and explores the connections between different modalities without priori knowledge. The motivation and method are clear to me. The results are convincing to me.", "weaknesses": "1. I acknowledge the novelty and value of the proposed OOM scenario. However, since the method lacks paired samples, it only shows limited effectiveness in learning shared knowledge, making it significantly inefficient in exploring unknown modalities. This is evident in the experimental results, which show a large gap compared to \"aligned\". Although the compared methods contain ERM and SSL, I question whether the proposed approach is better than SOTA unsupervised uni-modal solutions. Only surpassing or enhancing existing uni-modal methods with the proposed approach can make it really valuable. \n2. As I said in 1, the comparison with existing uni-modal methods is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)