Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning
Authors: Divyam Madaan
Taro Makino
Sumit Chopra
Kyunghyun Cho
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-modal learning
deep learning
Primary area: optimization_for_deep_networks
TL;DR: We distinguish between different modeling paradigms for multi-modal learning from the perspective of generative models and offer a general recipe for designing models that efficiently leverage multi-modal data, leading to more accurate predictions.

Abstract: Supervised multi-modal learning involves mapping multiple modalities to a target label. Previous studies in this field have concentrated on capturing in isolation either the inter-modality dependencies (the relationships between different modalities and the label) or the intra-modality dependencies (the relationships within a single modality and the label). We argue that these conventional approaches that rely solely on either inter- or intra-modality dependencies may not be optimal in general. We view the multi-modal learning problem from the lens of generative models where we consider the target as a source of multiple modalities and the interaction between them. Towards that end, we propose inter- \& intra-modality modeling (I2M2) framework, which captures and integrates both the inter- and intra-modality dependencies, leading to more accurate predictions. We evaluate our approach using real-world healthcare and vision-and-language datasets with state-of-the-art models, demonstrating superior performance over traditional methods focusing only on one type of modality dependency. The code is available at https://github.com/divyam3897/I2M2.

Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning
Divyam Madaan Taro Makino
Sumit Chopra 1,2.3
Kyunghyun Cho 122445
Courantt Institute of Mathematical Sciences, New York University
Center for Data Science, New York University
Grossman School of Medicine, New York University
4 Prescient Design, Genentech CIIAR LMB
Abstract
Supervised multi-modal learning involves mapping multiple modalities target label. Previous studies this field have concentrated on capturing in isolation either the inter-modality dependencies (the relationships between different modali- ties and the label) or the intra-modality dependencies (the relationships within a single modality and the label). We argue that these conventional approaches that rely solely on either inter- or intra-modality dependencies may not be optimal in general. We view the multi-modal learning problem from the lens of generative models where we consider the target as a source of multiple modalities and the interaction between them. Towards that end, we propose inter- & modlltyyy modeling (12M2) framework, which captures and integrates both the inter- and nttaa-odallity dependencies, leading to more accurate predictions. We evaluate our approach using real-world healthcare and state-of-the-art models, demonstrating superior performance over traditional meth- ods focusing only on one type of modality dependency. The code is available at
datasets with
Introduction
Supervised multi-modaa learning involves mapping input data to a target label, where the data is derived from multiple modalities and information about the boundaries between different modalities available. This problem has garnered interest in numerous applications, such as autonomous driving [18, 9, 40), healthcare [28, 67], robotics [55, 76, 14], name few. We encourage readers to refer to the latest survey papers [79, 36, 42] for recent developments in this field.
Despite multi-modal learning being key paradigm in machine learning, its effectiveness varies across different applications. In some cases, a multi-modal learner outperforms a uni-man miimodall Icarner [8, 78], while in others, it may not be effective as individual uni-modal learners [74, 16] or simple combination of uni-modal learners [53]. These differing results beg for principled framework that explain such discrepancies in multi-modal model performance and provide a general recipe for designing models that can leverage multi-modd data more efficiently and without such shortcomings.
In this work, we aim to uncover the underlying factors behind such discrepancies and introduce a more principled approach to multi-modal learning to resolve them. We view the supervised multi-modal learning problem from probabilistic lens and define the underlying data-generating process. More formally, the proposed generaenng process is shown Figure la. Without loss
"Correspondence hatryammmmaannnyu.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

y
x'
X
(a)
(b)
(c)
Figure 1: Data generating process for various scenarios with two modalities x, x and output y. In the context of multi-modal learning a), the label modulates the individual modalities (referred to as intra-modality dependencies) and the interaction between them (referred to as inter-modality dependency) through the selection variable v. In contrast, conventional approaches assume the graphical model in b) or c). In the graphical model shown in b), the dependency between each individual modality and the label only modulates through the selection variable v. On the other hand, the graph e) assumes that the dependency between two modalities independent of the label.
of generality, consider the output y generating data and x' for the two modalities. We define the statistical dependency between the set of modalities and the label using a selection variable {0, 13. Formally,
x =)= p(y)p((x y)p(x' yy)p(v 1
This selection variable is always set to one because it is the mechanism that induces the dependencies between the modalities and the label. The strength of this selection mechanism varies across datasets. When the selection effect is strong, inter-modality dependencies (dependencies between the modalities and label) become more significant. Conversely, when the selection effect is weak, intra-modality dependencies (dependencies between the individual modalities and label) are more crucial.
At high-level, our framework assumes that the output label generates the data associated with individual modalities. In addition, it also defines the relationship among different modalities and the label with the selection mechanism. The extent which the output is dependent on the data from individual modalities and on the cross-modality relationships, differs from use-case to use-case. Given the lack of prior knowledge about the strength of these dependencies on the final task, multi-modal system must model both the inter- and intra- intra-modality dependencies. We achieve this by building a classifier for each modality to capture the intr-moodality dependencies and another classifier to capture the dependencies between the output label and the inter-modality interactions. These classifiers are then combined by constructing a product or log-ensemble of their outputs. We name this approach inter- & intra-modality modeling (12M2), stemming directly from the above multi-modal generative model.
The proposed framework can be used to categorize all the previous works on multi molti- learning into two categories. The first category correspond the inter-modality modeling methods [4, 6, 19, 27, 45, 43, 8, 78, 2, 32, 75, 78, 56, 57, 15, 83]. This includes methods that predominantly focus on capturing the dependencies between the modalities to predict the target. From our graphical model's perspective these methods are based on the assumption that there are no direct edges from to x, (see Figure 1b). Although these methods can technically capture both inter- and intra- modalityy dependencies, they often fail to do so effectively [74, 78, 15]. This ineffectiveness stems from their reliance on incomplete underlying assumptions about the generative model for multi-modal learning. The second category correspond to the intra-modaliy modeling methods [54, 33, 23, 39, 66). This include approaches that consider the interactions between different modalities that occur only through the label (see Figure lc). These approaches do not capture the relationship between the modalities for prediction, which contradicts the objective of multi-modal learning. Inter- modality approaches excel when modalities share significant information to predict the label, while intra- aitaatto approaches are effective when cross-modalin information is sparse or absent. Often times, such information is not provided to when building the multi-modal models.
The proposed 12M2 framework addresses this shortcoming by not requiring prior knowledge of the strength of these dependencies. It explicitly models both inter- and intra-modality dependencies, making it adaptable and effective across various conditions. We validate our claims on multiple datasets, demonstrating the benefits of 12M2 over both inter- and intra-modalle methods. We apply

---

our method to multiple tasks in healthcare, including automatic diagnosis using knee MRI exams [8] ]]] and for mortality and ICD-9 code prediction in the MIMIC-III dataset [31]. We also demonstrate the benefits of 12M2 in multiple vision-and-lange and-language tasks such as VQA [3, 65] and NLVR2 [69]. Our evaluation shows the varying strength of dependencies across datasets; intra-modality dependencies are more beneficial for fastMRI dataset, while inter- anter-modality dependencies are more relevant for NLVR2 dataset. Both dependencies are pertinent for the AV-MNIST, MIMIC-III and VQA datasets. 12M2 excels across the board, ensuring robust performance regardless of which dependencies are most significant.
2
What is Multi-modal Learning?
Multi-modal learning refers the problem setup where the input is expressed set of observations from different modalities. Unlike conventional learning involving data set from a single modality, multi-modal learning can and should exploit the information from all the provided modalities for the purpose of prediction. In this work, we are interested in supervised multi-modal learning, where the goal is to map the inputs from multiple modalities to the targets.
We begin with the dataset D = {(x;, x's, y;)) with n examples. Without loss of generality, y; is the label and E X S Rd and x'; E X' C CR² represent data from the two modalities. To define multi-modal learning more formally, we define multi-modal data generating process in which label gives rise to both the modalities x and x' and the interaction between them (see Figure la). The variable v represents selection variable that captures the statistical dependencies across the modalities given the label. This selection variable is a binary random variable that is conditioned on all the input modalities and the target. As mentioned earlier, this variable is always present (i.e., 1), but its influence varies across datasets. The joint probability in this case can be written as:
p x, v = 1) = p(y)p(x y)p(x' yyp(v= = 1 x,)
(1)
While this might appear similar to the use of selection variables in modeling selection bias [26, 10, 5], in our context, selection does not refer to selecting examples, but to the mechanism that induces the dependencies between the modalities and the label. Particularly, we use this mechanism to break the conditional independence among the input modalities given the label, which is often referred to the 'explaining away' phenomenon. The challenge that, prior to analysis, the relative importance of inter- and intra-modality dependencies for classification is often unknown. Therefore, a multi-modal classifier needs to account for both inter- and intra- modaltty dependencies.
Our data generating process is commonly observed in many real-world scenarios. As concrete example, consider VQA [3], task that involves answering an open-ended question using information from an associated image. Each individual modality either the image or question can independently provide clues towards the correct answer [21, 8, 11, 65], yet these hints alone are often not sufficient to predict the answer. It is only by examining both modalities together that we can accurately infer the correct answer. This combination is captured our generative model by the selection variable v. Thus, this task requires building separate models capture the image and text specific information conditioned on the answer (intra-modality dependencies) and the dependency between the image, text modality given the answer (inter-modality dependency) to make the best prediction. This underlines the importance of a modeling approach that not only considers the interaction between the modalities but also uses each modality independently to predict the correct label.
3 Three Ways to Capture Modality Dependencies
Traditional approaches in multi-modal learning model the interaction between different modalities in order to predict the target, primarily by building novel architectures [54, 33, 64, 46, 2, 19, 75, 45,39, 39, 66]. Although these approaches occasionally outperform uni-modal models, there are cases where they fall short and are less effective than either the uni-modal learners [21, 8, 78] or their ensemble [53] counterparts. Furthermore, to the best of our knowledge, no prior work exists that sheds light on the reasons behind this discrepancy in model performance and provides a solution for the same. In this work we move our focus away from the question of model parameterization given multi-modal data. Instead, we focus towards uncovering the probabilistic assumptions required to study multi-modal learning.
3

---

We describe our 12M2 approach that incorporates the modality grouping information by considering models trained on individual modalities, while simultaneously capturing the interaction between the modalities. Next, we group existing studies into inter-modality modeling [4, 6, 19, 27, 45, 43, 2, 32, 75, 78, 56], which only uses the interaction between different modalities to predict the correct label, and intra-modality modeling [54, 33, 64, 23, 39, 66], which assumes the conditional independence between the modalities given the target.
3.1 Inter- & Intra-Modality Modeling (12M2)
Starting from Equation (1), we can write the conditional probability over labels as the product of four terms as follows:
p(y)p (x y) p (x' yy p0ppvv1llxx3 Ex p(y') (x, x y') = x
( x, x v
(2)
where p (x y) P (x' y) and p (v = =11x y x') are functions that map inputs (x, y), (x', y and (x, x', y) to a positive scalar. For clarity, we will use qx (y x). Ox (y x'), and Ox.x (y X, x') in
lieu of (x (y) pp (x') yy) and (vv = 1 x,x,x) x') and rewrite the above equation as follows:
ply x, x,X', V = 1) x ply) 1x (y x) Ixe (y xx) 1x.x (y x, x'),
Unimodal predictors
Multimodal predictor
where (y x) captures the conditional probability of the target given x, 4x x') the conditional probability of the target given the other modality x', @x,x' (y) x, x'x the conditional probability of the target given both the modalities. We omit v from the right hand side for brevity.
This suggests that we should build a separate predictive model for each modality and a model that takes as input the concatenated pair of both modalities. We combine these modality-speciif and multi-modal classifiers by building product of experts (or an additive ensemble in the log- pg-probability space). In this approach, we separately capture the intra-modality dependency within each modality and inter-modality interaction across the modality boundary, to predict the target. This modeling paradigm captures the influence of both individual modalities on the label, as well as the combined impact of both modalities.
12M2 closely aligns with the mutual information framework proposed in the multi-view frame- work [71, 44] and captures the three different types of mutual information. We motivate and explain this perspective for multi-modal learning from the principles of probabilsitic graphical models by explaining the underlying graphical models that give rise to such factorization of mutual information.
3.2 Inter-Modality Modeling
Current research on multi-modal learning mainly focuses on how to parameterize the conditional distribution over the label given multiple modalities. It has seen the development of various ar- chitectural strategies, which can be categorized into two main types: The first type is the early fusion [4, 6, 19, 27, 45, 43] which combines the input modalities at an initial stage and processes them using a single, shared model. The second type the intermediate fusion [2. 32, 75, 78, 56] which uses distinct models for each modality, linked together by fusion modules at different layers. We collectively call these modeling paradigms as inter-modale modality modeling, as it considers all the modalities together and does not explicitly use the information about modality boundaries, beyond parameterization. Thus, the predictive probability over y can be written as:
ply x, v 1) p(y) Ix, (y xx)').
(4)
This is derived from Equation (3), where gx (y x) and 4x (y x') are set to constant functions. This corresponds removing the direct edges y + x and y + x' in the graphical model depicted in Figure la, resulting in the graphical model shown in Figure 1b. Consider the example of NLVR [69, 70], which involves determining whether sentence accurately describes pair of images or not. This task requires model to possess joint understanding of visual and textual information to determine the accuracy of the statement. The dataset was curated to explicitly avoid visual or language bias [68]. To achieve this, each sentence was associated with multiple examples with conflicting labels. This was accomplished by showing workers a set of visually similar yet distinct images and asking them to write sentences that are true for some of the images but not for others.

---

Prior studies for inter-modane modality modeling [4, 6, 19, 27, 45, 43, 8, 78, 2, 32, 75, 78, 56, 57, 15, 83]
mostly focus on emphasizing inter-modality dependencies and identifying an issue with under utilizing these dependencies. While these methods are capable of capturing both inter and intra-modality dependencies, they often fail to do so effectively, especially when inter-modality information is absent or sparse [15]. Additionally, many of these approaches are developed for specific applications, such as person re-identification [35], multimedia recommendation [48] and sentiment analysis [47]. Unlike these specialized approaches. 12M2 is agnostic to the types of modalities and captures both the inter and intra-modality dependencies explicitly.
3.3 Infra-Modality Modeling
Intra-modality modeling [54, 33, 64, 23, 39, 66] processes each input modality through separate encoders. It then uses product of experts model consisting of these uni-modal predictors, where the correlation between the modalities does not predict the target. Consider an example of the tiger detection task, where y = 1 indicates the presence of tiger. The first modality is shape information, and the second modality is texture information. When O, the first modality has random shape, and the second modality a random texture. On the contrary, when the label is "tiger", i.e., y = the first modality (shape) has a tiger-like shape, regardless of the second modality (texture) and the same applies the texture modality. This implies that all the statistical dependencies between modalities manifest themselves via the label (y).
We refer to the predictive model for this generative process as intra-modality modeling, where assume conditional independence among the modalities given the label y. In this case the predictive distribution can be written as:
(5) Similar to Equation (4), this model stems from Equation (3), where xx.x' (y x, x') ) is treated as a constant function. This corresponds to eliminating the directed edge y y- in the graphical model in Figure 1a, leading to the graphical model shown in Figure Ic. We build a classifier for each modality and combine them by building a product (or log-ensemble) of these classifiers, ignoring higher-order interactions among the modalities.
P (y X,X' 1) x (y) 9x (y x) 9x (y xx')
Experiments
To differentiate various methods for multi-modal learning, we use audio-vision MNIST (AV-MNIST)
dataset [73], healtheare datasets, such fastMRI [81] and MIMIC-III [31], iisionaand-languall datasets, including VQA-VS [65] (a recently introduced version of VQA), and NLVR2 [70] (the latest version of NLVR). We consider kate-of-the-an art models for all the datasets. We defer the details of datasets to Appendix A and model sarammerrsesters to Appendix B.
4.1 AV-MNIST
Table 1: AV-MNIST ac- curacy comparison be- tween various methods.. Best results are high- lighted in bold.
Experimental setup. AV-MNIST combines audio and visual modal- ities for MNIST digit (0-9) recognition task. We use LeNet [37] with the top-performing methods from recent studies. Specifically, we em- ploy late fusion (LF), which concatenates the uni-modal feature repre- sentations followed by classification head, and low-rank multimodal fusion (LRTF) [50] from the recent multi-modal benchmark [41] for this dataset.
ACCURACY
IMAGF-ONLY AUDIO ONLY
L1 6.0 39,59 L1: 1.4
INTRA-MUDALITY
68 68.63
Results. Table shows that moder-y approaches outperform both intra-modality and single-modality approaches, underscoring the signifi- cance of mnter-modal interactions as anticipated by the data construction. 12M2 improves the performance by 1%, supporting our claim that both intra- and modeltty interactions are crucial for this task. Moreover, as expected from the dataset construction, the model trained with the visual modality more effectively predicts the label than the audio component. The performance is enhanced by 4% with intra-modalit modeling compared to using only the visual modality. 12M2 eliminates the need to pre-determine which specific dependencies should be modeled, offering more flexible and effective modeling approach.
INTER-MODALITY
LF
71.68
LRTF
71.49 ie
12M2
LRTF
72.38 8.77
5

---

Table Accuracy on MIMIC-III for mortality and ICD code prediction. 12M2 obtains higher performance in comparison to static (S) and time- series (T) uni-modal models, intra- iitra-modality mod- eling and inter-modality modeling.
90
AUROC (96)
INTRA INTER MORTALITY
ICD-9
T
140 239
460-519
85
/
x
X
76.32 0000
91.42 0:0000
55.99 (..0
x
77.04
iz 0.
83,36 (t 023)
68.15 (a: 0413
V
77.65 033)
91.42 080)
67.98 (s 0.42)
80
X
77.86
(=
91.54 0.12)
68.59 (#: 0.40
V
78.10 0.17)
91.58 (2: 0.10)
68.88 (:: 0.24)
75
ACL
Meniscus Cartilage Others
Table 3: Accuracy and VQA score for NLVR2 and VQA-VS respectively. 12M2 obtains com- parable performance to inter modality method for NLVR2, while outperforming it for VQA-VS. I and T denote the image and text modalities respec- tively. Best results are highlighted in bold.
Pathologies
Figure Results on fastMRI dataset. We compare coot-sum-ff-quaree magnitude and phase unimodal models, intra-modality model- ing, inter-modality modeling, and 12M2 mod- els (bars are in the same order). 12M2 obtains comparable performance to the intra-modality model by ignoring the inter-modality depen- dency, because comparatively to intra-modality, contributes less to predicting the label.
INTRA
INTER
NLVR2
VQA-VS
IID
OOD
$2,05 091
25.92
7.37 0.15)
/
X
52.97
(= 0.3)
43.78
(= 0.07)
22.03 (s 0.35)
54.31
da 0.3)
57.59
(: 0.00)
40.15 cs 0.28)
85.29
1.00)
68.04 0.00)
46.04 (= 0.46)
85,36
68.63
48.74
4.2 FastMRI
Experimental setup. This dataset [81, 82] consists of MR scans that include DICOM images and the corresponding raw measurements in the frequency domain (also known as k-space in the MR community), along with slice-level labels. We dissect the complex k-space data into magnitude and phase components, treating them as two distinct modalities for identifying the most significant knee pathologies: 1) anterior cruciate ligament (ACL), 2) meniscus tear, 3) cartilage and others grouping all the other pathologies. We use PreactResNet-118 [25] following Madaan et al. [52], the only study targeting diagnosis for this task.
Results. Consistent with our previous experiments, 12M2 outperforms the unimodal magnitude and phase models, inter-modality and intra-modality methods, shown in Figure 2. In this experiment, unlike the AV-MNIST dataset, inter-modality modeling degrades the performance compared to intra-modality modeling. Despite the opposite trend, I2M2 generally worked better than either of them across all pathologies. The superior performance of 12M2 highlights its ability to perform well even when one type of modality dependency is missing by effectively capturing the other.
Even when compared with root-sum-of-squares (RSS) - facto standard way of representing MR images in deep learning that benefits from the enhanced signal-to-noise ratio (SNR) offered by multi-coil data, 12M2 achieves performance superior to the model trained with RSS images across all pathologies. This finding is notable, given the higher SNR of RSS images due to the use of multi-coil data compared to our alternative representations synthesized by simulated single-coil output, resulting in lower SNR. This is useful because, until this point, we could not benefit from multiple modalities in complex MR images relative to RSS.
Lastly, the acquisition of MRI data is often a combination of multiple signals, which introduces background noise. Such background noise is easily impacted by the acquisition environments. is thus important to evaluate the generalization of our methodology while varying the SNR ratios during inference. We show the effectiveness of 12M2 over inter-modality modeling for varying SNR levels by increasing levels of Rician noise [61, 22] in Appendix C.

---

4.3 MIMIC-III
Experimental setup. The MIMIC-III dataset [31] encompasses ten years of intensive care unit (ICU) patient data from Beth Israel Deaconess Medical Center. We divide the dataset into two modalities [60, 41]: 1) the time-series modality, which includes hourly medical measurements over 24 hours, and 2) the static modality, capturing patient's medical information. We consider three tasks, namely a) mortality prediction of a patient within one day, two days, three days, one week, one year and beyond, and b) two binary classification tasks for ICD-9 codes, one to assess if a patient falls under group 1 (codes 140-239; neoplasms) and another for group (codes 460-519; diseases of the respiratory system). We adopt the best-performing models from the recent multi-modal learning benchmarks [60, 41] for this dataset (see Appendix B for more details).
Results. Table 2 shows that 12M2 enhances performance across all tasks when compared to methods that focus solely on either inter- modar-mo or intra-modality dependencies. Both modalities are predictive of mortality, but their effectiveness varies across different groups in predicting ICD-9 codes. Specifically, for ICD-9 codes 140-239, which are associated with neoplasms, the static modality proved more effective. This is likely because it includes factors such as the patient's s advanced age and the presence of chronic diseases, which are known to increase the risk of neoplasms. For ICD-9 codes 460-519, the model trained with time- modality showed better performance, as hourly measurements are useful to identify minor changes in respiratory functions, potentially signaling the onset or exacerbation of the disease.
For both mortality and ICD-9 prediction, we found that the intra-modality model obtains comparable performance to the inter- anter-modality model, highlighting that both the individual modalities and their interaction are predictive of the target for these tasks. As expected from our generative model, which formed the basis for our 12M2, the importance of these dependencies is contingent on the specific task label, and our method uses them effectively.
4.4 Natural Language Visual Reasoning
Experimental setup. NLVR2 [68] represents a binary classification task which the goal is to determine whether the text description correctly describes pair of two images. The model takes as imput two images and a text statement describing those images and predicts whether the text describes both images correctly. As discussed in Section 3.2, this dataset was constructed to minimize unimodal bias. For this dataset, we use the state-of-the- FIBER model [13, 53), which takes a pair of images and associated text as input and produces binary label its output. We fine-tune the full model consisting of Swin Transformer [51] for the image backbone, and RoBERTa [49] for the text backbone and MLP classifier on top of the encoder for five seeds.
Results. Table 3 shows that inter-modality modeling and 12M2 yield similar performance, which is substantially higher than the unimodal models and intra-modality model. This is because each of the image-only and the text-only models attains a chance-level accuracy for this dataset. This shows that neither the isolated text nor the image alone can make meaningful predictions in this problem. This can be attributed to the careful construction of the dataset, which eliminates language and visual biases and underscores the importance of inter- interaction within this dataset [68]. It demonstrates the ability of 12M2 to effectively disregard the uninformative antra-modality dependencies when predicting the target for this dataset. This aligns with observations from fastMRI, where 12M2 disregarded the anter-modality dependencies.
4.5 Visual Question Answering
Experimental setup. The objective of VQA is to answer questions about images, as detailed in Section 2. The labels comprise 3, 129 of the most common answers in the training and validation sets. The evaluation comprises IID and report the average OOD accuracy across nine OOD test sets. Additional details on the OOD test sets are provided in Appendix A, along with a detailed performance breakdown in Appendix C. Similar to NLVR2, we use the state-of-the-arr FIBER model [13] for training on VQA-VS dataset. We use the pre-trained model weights- Swin- Base [51] and RoBERTa-Base [49] for vision and text encoders and train an MI.P classifier on top of the pre-trained encoders following Makino et al. [53]. We use the VQA score metric across five random seeds to compare model performance.
(OOD) test sets released by VQA-VS [65]. We
7

---

96
96
90
90
AUROC (96
AUROC (%)
85
80
75
ACL
Meniscus Cartilage Pathologies
Others
75
ACL
Meniscus Cartilage
Others
Pathologies
Figure 3: AUROC performance for models with identical parameter count. We compare an ensemble of three magnitude-only, an ensem- ble of three phase-only models, an ensemble of magnitude and phase, ensemble of magni- tude, phase, and inter-modality model with 12M2. Despite having the same number of parameters, our proposed model consistently out- performs the ensemble models.
Figure 4: AUROC comparison with WideRes- Net models. We compare magnitude-only. phase-only models, inter-modality model trained with architecture WideResNett20-3 to our 12M2 trained with PreactResNet-11 18 across various knee pathologies in the fastMRI dataset (bars are in the same order). 12M2 obtains higher performance across all pathologies in compari- son the wider models.
Results. Table 3 shows the IID accuracy for the VQA-VS dataset. is evident that 12M2 surpasses inter-modality modeling and unimodal models in performance, emphasizing the importance of using both inter-modality and intra-modality dependencies. This improved performance is achieved by leveraging the dependencies between the modalities and individual modalities to predict the target. While the image-only model does not effectively predict the final task, the text-only model obtains 17.86% higher performance. This improvement can be primarily attributed to the language bias present within this dataset, as highlighted in previous works [21, 1, 11, 65]. While models suffer drop in performance for the OOD test-sets, 12M2 achieves relative improvement of 5.86% and 19.35% comparison to inter-modality modeling and intra-modalig modeling respectively. This highlights that addressing distribution shifts involves not only improving the individual experts but the robustness can also be enhanced through redundancy.
4.6 Further Analysis: Beyond Aggregate Metrics
While aggregate metrics provide broad overview, they often miss crucial details. We show that 12M2 surpasses other ensemble and wider models, even with a fixed parameter budget. Moreover, 12M2 avoids spurious dependencies between modalities and labels, common issue other models.
Comparison between models with identical parameter counts. To determine whether the perfor- mance improvement by 12M2 is due to the additional parameters or the specific manner in which the experts are combined, compare it with various mixture of expert models. Particularly, we compare with an ensemble three magnitude models, an ensemble three phase models, an ensemble of magnitude and phase models and ensemble of magnitude, phase and inter-modality model in Figure 3. We observe that across all pathologies, the ensemble outperforms individual unimodal models but 12M2 obtains better performance than all the ensemble models. This demonstrates that our 12M2 is more effective, even when constrained with a fixed parameter budget. We further contrast 12M2 using PreactResNet-11 18 with unimodal and inter-modalle Figure 4. Across various pathologies, 12M2 outperforms in terms of AUROC, reinforcing our view that the effectiveness stems more from our training approach and the integration of individual experts rather than merely from expanding the number of model parameters.
models using WideResNeet20-- in
Analysis of common mistakes. The training, validation, and test sets of VQA-VS contain a range of inter- and intra-modality spurious dependencies that are conditioned on specific labels. For example, the label "tennis" often correlates with words like "what", "sport", and "is" in the question, whereas the label "kite" is linked with "kite" in the image. Similarly, the words "how" and "many" in

---

What the sport?
What sport this?
What this?
What is the man doing?
many horses?
many elephants?
What sport this?
What this?
What this?
they playing?
How many birds?
How many sheep?
(a) OOD examples for questions
(b) OOD examples for images
(c) Multi-modal OOD examples
Figure 5: Visualization of samples from VQA-VS OOD test-sets. We visualize random instances without text, image and multi multi-modal spurious dependencies. Specifically, words like "what", "sport", and "is" in questions, the presence of "kite" in the image, and combination of "how" and "many" in the question with animals in the image are spuriously correlated with the labels "tennis", "kite", and "one", "two", "three" respectively in the IID sets. 12M2 using product of experts correctly predicts the target even when the spurious dependencies are absent, while individual expert models do not.
questions about animals or birds are typically associated with the labels "one", "two",, "three". In Figure 5, we purposefully chose examples where such spurious dependencies are absent from the OOD test-sets [65] demonstrate that the 12M2 can accurately predict in these scenarios, unlike the other models. We observe that most models overlook the question or image content in such instances, defaulting "kiting". "tennis". and "one". "two", or "three" as answers. accurately predict in these scenarios in comparison to these models. We find that in over 30% of cases where each expert fails individually, 12M2 succeeds in making the correct prediction.
the contrary, 12M2 can
Limitations and Future Work
Linear relationship between model size and number of modalities. The implementation 12M2 leads to a corresponding growth in model size with each added modality, which, although effective, results in higher computational costs. Specifically, the computational complexity of the model's forward pass increases linearly with the addition of modalities. For cases with small number of modalities, such as three or four, we can directly apply our modeling paradigm and iterate through all possible combinations of these modalities to construct product of experts model. This method straightforward due to the manageable number of combinations. For scenarios involving a larger number of modalities, employing a single network that receives all modalities as input. If any modality is absent for a specific example, this network receives null token instead. For each example, randomly select subset of combinations of conditional probabilities. The model can then constructed based on either the product the sum of logarithms of these conditional probabilities. This approach will keep the number of parameters linear, thereby managing complexity effectively.
propose the following approach for future investigation. We envision
Table 5: Effect of pre-training.
Challenges in model initialization. We found
ACI.
MENISCUS
CARTIL AGE
OTHERS
it more effective to train models for each modal- ity separately before fine-tuning them jointly, as opposed training them jointly from scratch, as shown in Table 5. This trend was consistent across all the datasets. This aligns with the findings of many previous works [38, 17, 77, 30], which have attributed this phenomenon to model-dominance effect [77], where single high performing model tends to dominate, and learner collusion [30], wherein learners bias their predictive distributions opposing directions, canceling each other out when aggregated. This suggests that there are optimization challenges in training multi-modal models from scratch that are not yet fully understood. We believe that investigating these challenges and developing end-to-end training methods is promising area for future research.
X
90.49 it: 0.39
11.95cm
029)
87.49 c= 0.40
80.42 (a: 0.40
92.62 6.30
93.79 c= 0.4
89.59 cs 641)
81.32 =:: 0.84)

---

6 Conclusion
In this paper, we proposed inter- & intra-modality modeling (12M2) for multi-modal learning, capturing both inter-modality and motralmydality dependencies. Applied to real-world datasets in healthcare and vision-language domains, 12M2 consistently outperformed conventional methods that rely solely on inter- or intra-modality dependencies and excelled in both in-distribution and cut-of--iitribution scenarios. Its versatility and modality nature make 12M2 a valuable tool, providing solid foundation for future research and applications in multi-modal learning.
7 Societal Impact
Contents posted on social media and other internet platforms are becoming increasingly more complex. They are no longer in just one modality, such as text-only, image-only or audio-only. These online contents are often multi-modal, requiring one to consider multiple modalities simultaneously the true meaning of these contents. As was pointed out earlier by Kiela et al. [34] in their hateful meme challenge, for instance, many harmful contents online require a holistic understanding of the text and the associated images, and text-only or image-only interpretation may miss the harmful nature of those contents. Advances in multi-modal learning, such as this work, will help us build a more effective content understanding system that can enable us to build better automated filtering system to keep online platforms less harmful. We also acknowledge, however, that better capability of multi-modal understanding can be used to build more advanced recommendation systems that may negatively contribute to the consumption of media and news by users.
grasp
Acknowledgement
This research was supported by Samsung Advanced Institute of Technology (Next Generation Deep Learning: from pattern recognition to AI), Hyundai NGV (Uncertainty in Neural Sequence Modeling in the Context of Dialogue Modeling), NSF Award 1922658, the Center for Advanced Imaging Innovation and Research (CAI2R), National Center for Biomedical Imaging and Bioengineering operated by NYU Langone Health, National Institute of Biomedical Imaging and Bioengineering through award number P41EB017183. The computational requirements for this work were supported in part by NYU IT High Performance Computing resources, services, and staff expertise and NYU Langone High Performance Computing Core's s resources and personnel. This content is solely the responsibility of the authors and does not represent the views of the funding agencies.
References
[1] A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[2] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom up and top-down attention for image captioning and visual question answering, In Proceedings of the IEEE International Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
[3] S. Antol, A. Agrawal, Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. Vqa: Visual question answering. In Proceedings of the International Conference Computer Vision (ICCV), 2015.
[4] T. Baltrušaitis, C. Ahuja, and L.-P. Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
[5] E. Bareinboim and J. Pearl. Causal inference and the data-fusion problem. Proceedings of the National Academy of Sciences, 2016.
[6] G. Barnum, S. Talukder, and Y. Yue. On the benefits of early fusion in multimodal representation learning. arXiv preprint arXiv:2011.0719. 2020.
[7] A. P. Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recognition, 1997.
10

---

[8] R. Cadene, C. Dancette, M. Cord, D. Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances in Neural Information Processing Systems (NeurIPS), 2019.
[9] X. Chen, H. Ma, Wan, Li, and T. Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[10] C. Cortes, M. Mohri, M. Riley, and A. Rostamizadeh. Sample selection bias correction theory. In Algorithmic Learning Theory: 19th International Conference, 2008.
[11] C. Dancette, R. Cadene, D. Teney, and M. Cord. Beyond question- based biases: Assessing multimodal shorteut learning visual question answering. In Proceedings of the International Conference Computer Vision (ICCV), 2021.
[12] S. U. H. Dar, M. Özbey, A. B. Çath, and T. Çukur. A transfer-learming approach for accelerated mri using deep neural networks. Magnetic resonance in medicine, 2020.
[13] Z.-Y. Dou, A. Kamath, Z. Gan, Zhang, Wang, L. Li, Z. Liu, Liu, LeCun, N. Peng. et al. Coarse-to-fine vision-language pre-training with fusion in the backbone. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[14] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
[15] C. Du, J. Teng, T. Li, Y. Liu, T. Yuan, Y. Wang. Yuan, and H. Zhao. On uni-modal feature learning in supervised multi-modal learning. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
[16] C. Du, J. Teng, T. Li, Y. Liu, T. Yuan, Y. Wang, Yuan, and H. Zhao. On uni-modal feature learning in supervised multi-modal learning. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
[17] A. Dutt, D. Pellerin, and G. Quénot. Coupled ensembles of neural networks. Neurocomputing, 2020.
[18] M. Enzweiler and D. M. Gavrila. A multilevel mixture-of-experts framework for pedestrian classification. IEEE Transactions on Image Processing, 2011.
[19] K. Gadzicki, R. Khamsehashari, and C. Zetzsche. Early vs late fusion in multimodal convo- lutional neural networks. In 2020 IEEF 23rd international conference information fusion (FUSION), 2020.
[20] V. Gareía, R. A. Mollineda, and S. Sánchez. Index of balanced accuracy: A performance measure for skewed class distributions. In IbPRIA, 2009.
[21] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[22] H. Gudbjartsson and S. Patz. The rician distribution of noisy mri data. Magnetic resonance in medicine, 1995.
[23] C. Gulcehre, O. Firat, K. Xu, Cho, and Y. Bengio. On integrating a language model into neural machine translation. Computer Speech and Language, 2017.
[24] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[25] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2016.
[26] M. Hernán, Hernández-Diaz, and J. M. Robins. A structural approach to selection bias. Epidemiology, 2004.

---

[27] J. Hessel and L. Lee. Does my multimodal model learn cross-modal interactions? it's harder to tell than you might think! In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
[28] S.-C. Huang, A. Pareek, S. Seyyedi, I. Banerjee, and M. P. Lungren. Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines. Nature digital medicine, 2020.
[29] Z. Jackson, C. Souza, J. Flaks, Y. Pan, H. Nicolas, and A. Thite. Jatcbboskkifrree-pkkeenniigg dataset: v1.0.8, 2018.
[30] A. Jeffares, T. Liu, J. Crabbé, and M. van der Schaar. Joint training of deep ensembles fails due to learner collusion. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[31] A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. Anthony Celi, and R. G. Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 2016.
[32] H. R. V. Joze, A. Shaban, M. L.. Iuzzolino, and K. Koishida. Mmtm: Multimodal transfer module for cnn fusion. In Proceedings of the IEEE International Conference Computer Vision and Puttern Recognition (CVPR), 2020.
[33] D. Kiela and S. Clark. Multi- and cross-modal semantics beyond vision: Grounding in auditory perception. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
[34] D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh, P. Ringshia, and D. Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in Neural Information Processing Systems (NeurIPS), 2020.
[35] M. Kim, Kim, J. Park, S. Park, and K. Sohn. Partmix: Regularization strategy to learn part discovery for visible-infrared person re- ee-ientffication. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[36] A. Kline, H. Wang, Y. Li, S. Dennis, M. Hutch, Z. Xu, F. Wang, F. Cheng, and Y. Luo. Multimodal machine learning in precision health: A scoping review. Nature Digital Medicine, 2022.
[37] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
[38] S. Lee, S. Purushwalkam, M. Cogswell, D. Crandall, and D. Batra. Why m heads are better than one: Training diverse ensemble of deep networks. arXiv preprint arXiv:1511.063/ 2015.
[39] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[40] J. Li, H. Dai, H. Han, and Y. Ding. Mseg3d: Multi-modal 3d semantic segmentation for autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[41] Liang, Y. Lyu, X. Fan. Z. Wu, Y. Cheng, J. Wu, L. Chen, P. Wu, M. Lee, Y. Zhu, al. Multibench: Multiscale benchmarks for multimodal representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[42] P. P. Liang, A. Zadeh, and L.-P. Morency. Foundations and trends in multimodal machine learning: Principles, challenges, and open questions. arXiv preprint 2xXiv:2209003330 2022.
[43] P. Liang, Y. Lyu, X. Fan, J. Tsaw, Y. Liu, S. Mo, D. Yogatama, L.-P. Morency, and R. Salakhut- dinov. High-modality multimodal transformer: Quantifying modality & interaction heterogene- ity for high-modality representation learning. Transactions on Machine Learning Research (TMLR), 2023.
12

---

[44] P. P. Liang, Z. Deng, M. Q. Ma, J. Y. Zou, L.-P. Morency, and R. Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[45] V. Likhosherstov, A. Arnb, K. M. Choromanski, M. Lucic, Y. Tay, and M. Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. Transactions on Machine Learning Research (TMLR), 2023.
[46] D. Lin, G. Chen, D. Cohen-Or, P.-A. Heng, and H. Huang. Cascaded feature network for semantic segmentation of rgb-d images. In Proceedings of the International Conference on Computer Vision (ICCV), 2017.
[47] Z. Lin, B. Liang, Y. Long, Y. Dang, M. Yang, M. Zhang, and R. Xu. Modeling intra- and inter-modal relations: Hierarchical graph contrastive learning for multimodal sentiment analysis. In Proceedings of the 29th International Conference on Computational Linguistics, 2022.
[48] Z. Lin, Y. Tan, Y. Zhan, W. Liu, F. Wang, C. Chen, S. Wang, and C. Yang. Contrastive intra-and inter-modality generation for enhancing incomplete multimedia recommendation. In Proceedings of the 31st ACM International Conference on Multimedia, 2023.
[49] Y. Liu, M. Ott, N. Goyal, Du, M. Joshi, D. Chen, O. Levy, M. Lewis. L. Zettlemoyer. and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. / 1692, 2019.
[50] Z. Liu, Shen, V. B. Lakshminarasimhan, P. P. Liang, A. B. Zadeh, and L.-P. Morency. Efficient low-rank multimodal fusion with modality-specifft factors. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2018.
[51] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[52] D. Madaan, D. Sodickson, K. Cho, and S. Chopra. On sensitivity and robustness of normalization schemes to input distribution shifts in automatic MR image diagnosis. In Medical Imaging with Deep Learning (MIDL), 2023.
[53] T. Makino, Y. Wang, K. J. Geras, and K. Cho. Detecting incidental correlation in multimodal learning via latent variable modeling. Transactions on Machine Learning Research (TMLR), 2023.
[54] E. Morvant, A. Habrard, and S. Ayache. Majority vote of diverse classifiers for late fusion. In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshop. Springer, 2014.
[55] K. Noda, H. Arie, Y. Suga, and T. Ogata. Multimodal integration learning of robot behavior using deep neural networks. Robotics and Autonomous Systems, 2014.
[56] H. Pan, Z. Lin, P. Fu, Y. Qi, and W. Wang. Modeling intra and inter-modality incongruity for mudti-modal sarcasm detection. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
[57] X. Peng, Y. Wei, A. Deng, D. Wang, and D. Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[58] J.-M. Pérez-Rúa, V. Vielzeuf, S. Pateux, M. Baccouche, and F. Jurie. Mfas: Multimodal fusion architecture search. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[59] K. J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, 2015.
[60] S. Purushotham, C. Meng, 7. Che, and Y. Liu. Benchmarking deep learning models on large healthcare datasets. Journal of biomedical informatics, 2018.
13

---

[61] S. Rice. Mathematical analysis of random noise. The Bell System Technical Journal, 1944.
[62] P. B. Roemer, W. A. Edelstein, C. E. Hayes, S. P. Souza, and O. M. Mueller. The nmr phased array. Magnetic resonance in medicine, 1990.
[63] O. Shitrit and T. Riklin-Raviv. Accelerated magnetic resonance imaging by adversarial neural network. In WWM.MMMMM-CSSSMMMCCCCCMC 2017.
[64] E. Shutova, D. Kiela, and J. Maillard. Black holes and white rabbits: Metaphor identification with visual features. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2016.
[65] Q. Si, F. Meng, M. Zheng, Z. Lin, Y. Liu, P. Fu, Y. Cao, W. Wang, and J. Zhou. Language prior is not the only shortcut: A benchmark for shortcut learning in vqa. In Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP), 2022.
[66] A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[67] L. R. Soenksen, Y. Ma, C. Zeng, L.. Boussioux, K. Villalobos Carballo, L. Na, H. M. Wiberg, M. L. Li, I. Fuentes, and D. Bertsimas. Integrated multimodal artificial intelligence framework for healtheare applications. Nature Digital Medicine, 2022.
[68] A. Suhr and Y. Artzi Nlvr2 visual bias analysis. arXiv preprint 19X9v:1909.100 10411, 2019.
[69] A. Suhr, M. Lewis, J. Yeh, and Y. Artzi A corpus of natural language for visual reasoning. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics, 2017.
[70] A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv: 1.0091, 2018
[71] Y.-H. H. Tsai, Y. Wu, R. Salakhutdinov, and L..P. Morency. Self-supervised learning from a multi-view perspective. In Proceedings of the International Conference on Learning Represen- tations (ICLR), 2020.
[72] M. Tygert and Zbontar. Simulating single-coil mri from the responses of multiple coils. Communications in Applied Mathematics and Computational Science, 2020.
[73] V. Vielzeuf, A. Lechervy, S. Pateux, and F. Jurie. Centralnet: multilayer approach for multimodal fusion, 2018
[74] W. Wang, D. Tran, and M. Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[75] Y. Wang, W. Huang, F. Sun, T. Xu, Y. Rong, and J. Huang. Deep multimodal fusion by channel exchanging. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[76] Z. Wang, Li, B. Wang, and H. Liu. Robot grasp detection using multimodal deep convolutional neural networks. Advances in Mechanical Engineering, 2016
[77] A. Webb, C. Reynolds, W. Chen, H. Reeve, D. Iliescu, M. Lujan, and G. Brown. To ensemble or not ensemble: When does end-to-end training fail? In Machine Learning and Knowledge Discovery in Databases: European Conference (ECML PKDD), 2021.
[78] N. Wu, S. Jastrzebski, K. Cho, and K. J. Geras. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. Proceedings of the International Conference on Machine Learning (ICML), 2022.
[79] Y. Xiao, F. Codevilla, A. Gurram, O. Urfalioglu, and A. M. López. Multimodal end-to-end autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 2020.
[80] Y. Yang, J. Sun, Li, and Z.. Xu. Admm-net: A deep learning approach for compressive sensing mri. arXiv preprint 17X.v:1705.06669. 2017.
14

---

[81] J. Zbontar, F. Knoll, A. Sriram, T. Murrell, Z. Huang, M. J. Muckley, A. Defazio, R. Stern, Johnson, M. Bruno, al. fastmri: An open dataset and benchmarks for accelerated mri. arXiv preprini arXiv: /8/ 1.08839, 2018.
[82] R. Zhao, B. Yaman, Y. Zhang, R. Stewart, A. Dixon, F. Knoll, Z. Huang, Y. W. Lui, M. S. Hansen, and M. P. Lungren. fastmri+: Clinical pathology annotations for knee and brain fully sampled multi-coil mri data. arXiv preprint
2021.
[83] Y. Zhou, X. Wang, H. Chen, X. Duan, and W. Zhu. Intra-and inter-modal curriculum for multimodal learning. In Proceedings of the st ACM International Conference on Multimedia, 2023.
15

---

Organization. In the supplementary material, we provide description of the datasets in Appendix A, implementation details in Appendix and additional results in Appendix C.
A Datasets
A.1 Audio-Vision MNIST
Audio-Vision MNIST (AV-MNIST) has been a popular toy benchmark [58, 73, 41] evaluate the performance of existing models for multi-modal learning. The images are collected from the MNIST dataset [37] and audio from the free spoken digit dataset (FSDD) [29] containing human-spoken digits. 75% of energy is removed from the visual modality with PCA and noise is added to the audio modality from the ESC-50 dataset [59]. We use 55000 examples for training, 5000 validation, and 10000 testing examples for evaluation.
A.2 FastMRI
Problem setup. The process of MR data acquisition involves exposing the human subject to varying magnetic field and radio- aadio-frequenee pulses and capturing the resulting electromagnetic responses from the human body. These electromagnetic responses measured by devices called receiver coils that are positioned in the vicinity of the tissue or organ being imaged. The measurements captured by the receiver coils are in the Fourier domain (a.k.a., k-space in the MR community), where each coil produces partial image with different spatial sensitivity across different parts of the volume.
The generation of ground truth images using the data from multiple coils consists of two steps. The first step involves an Inverse Fourier Transform of the k-space data from an individual coil to generate coil specific images. In the second step, generate single image by combining the magnitudes of the individual coil images voxel by voxel using the root-sum-of -f-quurrss (RSS) method [62]]
1/2
RSS
Σ C
x
where n are the k-space measurements acquired by the e-th coil and and ne is the number of receiver coils. The images generated by RSS method does not contain the phase information and the raw- data is often not openly released, which has slowed the progress in researching the task of disease identification using the raw measurements. Further, the phase of the acquired data is often discarded for diagnosis. To this end, even for MR image reconstruction majority of the methods used synthetic k-space data obtained from the ground-trute -ruth spatially reconstructed images [63, 80, 12].
In this research, we argue that blindly constraining the inputs of the DNN models to be the same what human would raw measurements as input to the DNNs, without regards to its human interpretability. will allow us build systems that automatically extract the most informative diagnostic signals. Towards that end, we explore the utility of using magnitude and phase of the input signal as two modalities for the DNN models and understand the role of the phase component of the complex data acquired by the MR scanners for disease identification.
render a diagnosis is sub-optimal. We hypothesize that using the
accurate predicting diseases, by enabling the DNN models to
Dataset. The fastMRI dataset [81] was the first large-scale dataset that consisted of raw k-space data alongside anonymized clinical magnetic resonance (MR) images. In this work for the purpose of simplicity we aggregate the data from multiple coils and emulate it to be originating from a single coil using the method proposed in Tygert and Zbontar [72], Zbontar al. [81] to compute emulated single-coil data.
FastMRI+ [82] augmented this dataset with pathology annotations. These annotations motivated our exploration into the potential of the raw k-space data components for automated diagnosis. Specifically, focused knee pathologies, using emulated single-coil k-space data with llice-level labels. We dissect the complex k-space data into magnitude and phase components, treating them as two distinct modalities for identifying clinically important pathologies in MR scans.
We consider the most significant knee pathalogies as highlighted by the clinicians: 1) Anterior Cruciate ligament (ACL) with 1,443 annotations of 254 subjects, 2) Meniscus tear with 5,658
16

---

annotations of 663 subjects, and 3) Cartilage with 3,600 annotations of 710 subjects and 4) others containing the group of all the other pathologies. The slices were cropped to 320 x 320 and 15% of the dataset for used for validation and test sets.
A.3 MIMIC-III
The Medical Information Mart for Intensive Care (MIMIC-III) dataset is a popular medical benchmark consisting of Electronic Health Records (EHRS) of of over 40, 000 patients at Beth Israel Deaconess Medical Center from 2001-2012. The dataset is categorized into two modalities [60, 41]. First, consists of the static modality, which contains five medical features of the patient including chronic diseases, admission type and age. Second, it is composed of time-series modality, which consists of twelve different measurements recorded hourly over 24-hour period.
We consider the following two tasks for this dataset:
Mortality prediction. The objective of this task is to predict whether the patient dies in one day, two day, three day, one week, one year, or longer than one year.
ICD-9 code prediction. This task focuses on predicting the appropriate group of the International Statistical Classification of Diseases and Related Health Problems, Ninth Revision (ICD-9) diagnosis codes for each patient admission. The ICDD-9 system categorizes similar diseases into groups. In this work, the dataset is used determine whether patient's condition falls within either Group 1, which includes codes 140 239 related to neoplasms, or Group covering codes 460 519 associated with respiratory diseases.
Following Liang et al. [41], we split the dataset into 80% training. 10% for validation and 10% for testing. This results in 28, 970 training, 3, 621 validation, and 3, 621 examples for testing.
A.4 Natural Language Vision Reasoning
The evaluation of vision and language models' capability to reason about visual compositions has been an important direction of research in the field. Natural Language Vision Reasoning (NLVR) [69] played an important role with the release of synthetic images, text, or combinations of both for analysis. The dataset consists of pairs of images against natural language sentence. The task is to ascertain if the sentence accurately describes (True) or inaccurately describes (False) the image pair.
Expanding on this, the NLVR2 dataset [70] incorporated real-world photographs. The dataset was curated to avoid visual or language bias [68] by ensuring that each sentence was paired with multiple examples with distinct labels. This was achieved by showing workers collection of visually similar yet distinct images and asking them to write sentences that are true for some images but false for others. The dataset consists of 59, 677 examples, each with an image resolution of 384 pixels.
A.5 Visual Question Answering
Problem setup. The task of visual question answering (VQA) involves answering questions related to given image. This task has been widely adopted as benchmark evaluate models that integrate both the vision and language modalities. Following prior studies, we consider as multi-label classification problem consisting of 3,129 distinet labels.
the performance of
Background. Initial VQA datasets were found to exhibit significant issue: models were able to answer questions even without considering images. This issue led to the creation of the VQA-CP v2 dataset [1], which was constructed to emphasize out-of-distribution (OOD) robustness by deliberately varying the distribution of answers for identical types of questions between the training and testing phases. This dataset was designed to incentivize models that properly rely on images for answering questions.
Despite its intention, VQA-CP v2 still exhibited several issues [11, 65]. First, it utilized an OOD test set for the purpose of model selection. Second, the models had to be retrained to assess their performance under IID conditions. Si et al. [65] recently addressed these problems in VQA-VS by introducing new IID validation and test splits from the original VQA dataset, along with OOD test sets based on images, language, and multimodal shortcuts.
17

---

Dataset. VQA-VS [65] consolidated the training and validations sets from VQA v2 dataset. The combined dataset was split into 70% for training, 5% for validation and 25% for testing. Further, Si et al. [65] created a set of out-of-distributio test sets, which were crafted from the test-set into the following categories:
Text-based test sets. This category encompasses four subsets, each evaluating whether the model relies concepts from the question to predict the answer. The question-type (QT) test-set clusters the samples where the question prefixes are predictive of the answer. The key-word (KW) and key-word pair (KWP) test-sets focus instances that exhibit a high degree of mutual information with the answer. The last group considers the QT and KW concepts together to construct the combined set.
Image-based test sets. The image-baue based test sets comprises of two subsets. The key object (KO) subset consists of instances where certain visual features demonstrate strong correla- tion with the answer. The key-object pair (KOP), expands on the KO set by incorporating pairs of visual elements that have high correlation with answer.
Multi-modal test sets. These combine elements from both the textual and visual domains, resulting in three subsets. The QT+KO consists of the instances where question-pree and key visual objects are predictive of the answer. Similarly, KW+KO focuses on instances with keyword elements and visual objects. QT+KW+KO amalgamates the instances where question types, keywords, and key visual objects are predictive of the answer.
B Models and Hyperparameters
AV-MNIST. We use the LeNet [37] model for both the image and audio modality following Liang et al. [41]. For training all the models, we optimize using cross-entropy loss with SGD using learning rate and weight decay equal to 5 x 10-2 and 1 x 10-1 for 25 epochs. Low-Rank Tensor Fusion (LRTF) uses rank 40 and the output dimension equal to be 120. We report the mean and standard deviation across five runs with random seeds. All experiments were conducted on a single NVIDIA A100 GPU.
FastMRI. For our baseline model, we use PreactResNNt-118 18 [24] following Madaan et al. [52] and use early fusion to capture the inter-modality dependencies. We report the mean and standard deviation of AUROC [7] and balanced accuracy [20] for all our experimental analysis across five independent runs. We conduct a grid search on the learning rate and weight decay for this dataset and report the optimal values in Table B.6. We employ early stopping based on the average AUROC across all pathologies to store the best model, which then used for reporting the results.
Table B.6: Learning rates (LR) and weight decay (WD) for fastMRI dataset.
Method
LR
WD
MAGNITUDE-ONLY
1 10-5
1 10-3
PIIASE-ONLY
1 10-4
1 x 10-1
INTRA-MODALITY INTER-MODALITY
1 1x 10-6
1 10
1 x 10-1
1 10-2
I2M2
1x 10-6
1 10
MIMIC-III We follow the experimental setup used by Purushotham al. [60], Liang et al. [41] for this dataset. The static encoder and decoder is a two-layer MLP with LeakyReLU activation function. The time-series encoder and decoder is GRU with the hidden dimension equal to 30. For capturing the inter-modality dependencies for this dataset, we use late fusion. We use batch-size 40 and RMSProp with learning rate of 1 x 10 to train for twenty epochs across all the tasks. The mean and standard deviation are reported across five independent runs for all the models and tasks.
NLVR2 and VQA-VS. We use the FIBER model [13] for both the datasets. It consists of the Swin Transformer [51] for the image backbone, and RoBERTa [49] for the text backbone. We fine-tune a MIP classifier on top of the encoder with learning rate x 10-1 for VQA-VS. For NLVR2, we fine-tune the full-model with the learning rate 10-5 for five seeds. We report the VQA score for VQA-VS VS and accuracy for NLVR2 with mean and standard deviation.
18

---

(60
40
10
90
VQA
KMP
VT+
ROP
RW of
Image:
Mulimodai-baaee
Figure C.6: VQA score on various OOD VQA-VS test sets. We compare image, text unimodal models, intra-modality modeling, inter-modality modeling with I2M2 (bars are in the same order).
Across all test-sets, 12M2 demonstrates superior performance compared to the other models.
SNR
SNR
SNR
SNR
100
10
5
4
100
5
100
5
100
5
90
AUROC
90
AUROC
90
AUROC
AUROC
90
80
80
80
80
20 25
20 25
20
25
20 25 Noise
Noise
Noise
Noise
Menicus
Cartilage
Figure C.7: AUROC performance comparison for varying ricean-noise levels. Comparison of AUROC performance for PreactResNet- 18 trained with inter-modality modeling and 12M2 ACL, meniscus tear, cartilage and other pathologies with increasing levels of Ricean-noise. We observe that 12M2 obtains obtains equal or better performance for majority of the noise levels compared to inter-modality modeling.
C Additional Results
Results with distribution shifts with VQA. 12M2 relies both the inter- and intra-modality dependencies. In scenarios where models depend heavily on one type of modality dependency, they often become less reliable when faced with changes in data distribution. This phenomenon is evident in Figure C.6, where we present the results for various OOD image, text and multi-modal based test-sets. While all models suffer drop in performance for all the test-sets, it is noteworthy that across all these test-sets, 12M2 consistently achieves a relative improvement ranging from two to four percent when compared the performance of inter-modality modeling. Furthermore, it achieves performance gain of 20 to 25% compared to models trained solely on text inputs. This highlights that addressing distribution shifts effectively involves not only improving the individual experts but the robustness can also be enhanced through redundancy.
Results with distribution shifts with FastMRL The acquisition of MRI data is often a combination of multiple signals, which introduces background noise. In fact among various factors, noise is by-far the biggest factor that contributes towards image deterioration when going from a high-field scanner (scanner with high magnetic field strength) to a low-field scanner (scanner with low magnetic field strength). Despite being inexpensive these scanners have not seen widespread adoption because of the poor diagnostic quality of the images generated by them. Evaluation of any methodology on low signal-to-noise ratio (SNR) images is important for gives some insights into how the methodology will generalize to low-field scanners, potentially leading to transformative impact in healthcare by facilitating the use of these scanners which inexpensive and relatively accessible.
Due to the dearth of data collected from the low-SNR MR scanners, we simulate the low-SNR k-space data during inference. Particularly, we know that the noise in the acquired k-space is Gaussian distributed, which leads to the ground truth images having noise that has a Rician distribution [61, 22]. Thus, to degrade the signal, we add Gaussian noise to both the real and imaginary channels.
Figure C.7 shows the effects of varying noise levels, or alternatively, varying signal-to- ugnal-to-noise ratios. We notice that for all types of pathologies, 12M2 performs better in terms of AUROC at lower noise levels. This improvement persists for meniscus and cartilage pathologies even at higher noise levels. This enhanced performance is attributed to the noise addition impacting both real and imaginary channels, affecting the magnitude and phase modalities, respectively. 12M2 captures the interactions within and between these modalities, showing greater resilience this noise due to redundancy.
19

---

Entropy measurement. Table C.7 shows the entropy the label distribution across three dif- ferent datasets, and it compares these values with the average entropy of the predictive distribu- tion for both unimodal models and mnter-modality models, post fine-tuning from 12M2. In the cases of AV-MNIST and VQA-VS, consistent with results Section 4, we observe that both the unimodal experts and the inter-modality model exhibit lower entropy. On the other hand, in the NLVR2 dataset, while the unimodal experts demonstrate high average entropy, the inter-modality model exhibits significantly lower entropy in its predictive distribution. This results from the interaction between the modalities being predictive of the target, with no significant information found in the modalities when considered separately, as also illustrated in Figure C.6.
Table C.7: Entropy of individual experts. We compare the entropy of the label (y) distribution with the average entropy of the predictions (y) for individual experts.
AV-MNIST
VQA-VS
NLVR2
H(y)
2.30 (:00
6.86 (#:000
0.69
H x
1.61 (#
2.71 (#071)
0.68 omn
H x
2.24 0.0)
4.32 1.25)
0.67
H(y
x,x')
0.95 0.08.
5.33 (..0.30)
0.17 0.02)
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 5
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Dataset details and hyper-ppaaee peremparssstss are provided in the supplementary material.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code available http://gghhhbboommaaaoyyyyssmsim
6. Experimental Setting/Detalls
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix and Appendix B.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes].
Justification: The results accompanied with confidence intervals across five random
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes].
Justification: See Appendix B.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the httpr/psss..pppiccc/chcccceoeooo
NeurIPS Code of Ethics
Answer: [Yes]
10. Broader Impacts
21

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section 7.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See, code, Appendix A and Appendix B.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
22


Reviews:
[{"review_id": "WTLy11Obl2", "reviewer": "Reviewer_vide", "summary": "This paper studies supervised learning for multi-modal data. Previous works can be categorized into inter-modality learning and intro-modality learning. Inter-modality learning aims to learn multi-modal data jointly by techniques such as feature fusion. Intral-modality learning focuses on learning uni-modal data separately. This paper attempts to unify these two frameworks by building a probabilistic graphical model of the multi-modal data. The previous inter- and intra-modal learning frameworks can be regarded as special cases of this graphical model by assuming specific independence of data. The joint modeling framework is expected to automatically infer whether the inter- or intra-information is important for given tasks. Additionally, the resulting model is simple to implement, by combining the inter- and intra-learning models together.\n\nExperiments are conducted on several tasks, such as auto-vision, healthcare, and vision-language. The results show that the proposed combination can improve performances of each single model, intra-modality combinations and inter-modality feature fusions.", "questions": "Can the author explain more about the derivation of Eq. (3)? How to replace $p(y \\mid x)$ by $q(x \\mid y)$? The former is probability of $y$ while the latter is probability of $x$.", "limitations": "Yes, the authors discuss about limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of the joint modeling is convincing. The resulting model is simple to use by combining different models.\n\n2. The paper discusses an interesting unification of the two paradigms in multi-modal learning.\n\n3. Experiments are conducted on different domains such as audio, image and language.", "weaknesses": "1. For the proposed model, are the inter-modal classifier and intra-modal classifiers trained from scratch? Can we simply use pre-trained ones? If the model should be re-trained or fine-tuned, it might be computationally expensive.\n\n2. As the paper says, the proposed model is similar to a mixture of experts. It should be helpful to discuss more about this point and previous approaches of mixing experts in the related work part. The discussion could help to highlight the difference and contribution.\n\n3. In experiments, the details of intra-modality seem missing. Do the authors choose competitive baselines for this class of methods? Moreover, for inter-modal models, Late fusion seems to be a simple baseline. LRTF was proposed in 2018. My concern is whether the results hold for more recent and competitive models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your review and your thoughtful comments. We appreciate you finding our paper well-motivated and simple to use with thorough experiments across different domains. We clarify your concerns below.\n\n> For the proposed model, are the inter-modal classifier and intra-modal classifiers trained from scratch? Can we simply use pre-trained ones? If the model should be re-trained or fine-tuned, it might be computationally expensive.\n\nAs discussed in the limitations section and compared in Table 5, our approach benefits from pre-training the inter-modal and intra-modal classifiers. While fine-tuning these models jointly further improves performance, the computational expense is minimal, since the models converge within a few epochs.\n\n---\n\n> The proposed model is similar to a mixture of experts. It should be helpful to discuss more about this point and previous approaches of mixing experts in the related work part. The discussion could help to highlight the difference and contribution.\n\nThe proposed model is a product-of-experts [Hinton et al. 2013] and NOT a mixture-of-experts. We provide an empirical comparison showing benefits over the mixture of experts model in Figure 3. It is important to note that Mixture of Experts cannot capture all the interactions between modalities and the label. For example, consider the XOR problem with two variables, simply combining the outputs of two advanced neural networks—a method akin to a mixture of experts—cannot solve this task. We will further expand on the previous approaches in the related work section and make this distinction clearer.   \n\n---\n\n> In experiments, the details of intra-modality seem missing. Do the authors choose competitive baselines for this class of methods? Moreover, for inter-modal models, Late fusion seems to be a simple baseline. LRTF was proposed in 2018. My concern is whether the results hold for more recent and competitive models. My concern is whether the results hold for more recent and competitive models.\n\nWe provide details for the intra-modality model on lines 187-189. Essentially, it is a product of experts model similar to I2M2, but without the inter-modality expert, making it a competitive baseline for comparison.\n\nFor inter-modal models, we were careful in picking the most recent and the most competitive models as baselines for all our experiments. This is highlighted in the experimental setup for each dataset. Specifically, for AV-MNIST and MIMIC-III, we selected LRTF because it was the top-performing model from the MultiBench benchmark [1]. For FastMRI, we referenced the work by Madaan et al., [2] the only study we identified that focuses on diagnosis for this task. Additionally, for VQA and NLVR2, we used the FIBER model, which achieved state-of-the-art performance on both datasets, as demonstrated in the recent studies [3, 4].\n\n---\n\n> Can the author explain more about the derivation of Eq. (3)? How to replace 𝑝(𝑦∣𝑥) by  𝑞(𝑥∣𝑦)? The former is probability of 𝑦 while the latter is probability of 𝑥. \n\nAs explained in Section 3.1, our model generates a probabilistic score using four components derived from the joint distribution in Equation 1. In Equation 2, we multiply these components, with the denominator serving as a normalizing constant that does not depend on the label $ \\mathbf{y} $.\n\nThe notation $ q_{\\mathbf{x}}(\\mathbf{y} \\mid \\mathbf{x}) $ is used to denote the function that maps $ (\\mathbf{y}, \\mathbf{x}) $ to a positive scalar. This function is proportional to $ p(\\mathbf{x} \\mid \\mathbf{y}) $ because $ p(\\mathbf{y} \\mid \\mathbf{x}) = \\frac{p(\\mathbf{x} \\mid \\mathbf{y}) p(\\mathbf{y})}{p(\\mathbf{x})} $. Thus, by defining $ q_{\\mathbf{x}}(\\mathbf{y} \\mid \\mathbf{x}) $ as proportional to $ p(\\mathbf{x} \\mid \\mathbf{y}) $, we have simply renamed $ p(\\mathbf{y} \\mid \\mathbf{x}) $ for convenience. Specifically, $ q_{\\mathbf{x}}(\\mathbf{y} \\mid \\mathbf{x}) = c \\cdot p(\\mathbf{x} \\mid \\mathbf{y}) $, where $ c $ is a constant that normalizes the function representing the interaction between $ \\mathbf{y} $ and $ \\mathbf{x} $.\n\n---\n\nThank you again for your time and efforts in reviewing our paper, and we hope that you will consider raising your score if you find our response satisfactory.\n\nThank you,    \nAuthors\n\n---\n\nReferences.   \n[1] Liang et al., 21 MultiBench: Multiscale Benchmarks for Multimodal Representation Learning   \n[2] Madaan et al., 23 On Sensitivity and Robustness of Normalization Schemes to Input Distribution Shifts in Automatic MR Image Diagnosis.   \n[3] Dou et al., 22 Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone.    \n[4] Makino et al., 23 https://openreview.net/forum?id=QoRo9QmOAr.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aaSCR1fGkS", "reviewer": "Reviewer_Y8eL", "summary": "This paper proposes a framework for multi-modal learning called inter- & intra-modality modeling (I2M2). I2M2 can simultaneously capture inter-modality dependencies (relationships between different modalities) and intra-modality dependencies (relationships within a single modality). This approach aims to improve the accuracy of predictions by integrating both types of dependencies, rather than focusing on one. Experiments on health care data verify the advantage of utilizing both inter- and intra-modality.", "questions": "1. How is the performance compared to existing inter-modality modeling methods, such as that in ref.[15]?\n2. How do we deal with multi-modality when the number of modalities is greater than 2? Should we consider the conditional probabilities on any combination of modalities or just modalities between any pairs of modalities? Further, should all the conditional probabilities be equally treated?\n3. In the FastMRI experiment, the performance under low SNR k-space data is analyzed. How about the robustness when only one modality is affected? For example, only magnitude modality is disturbed by noise during signal transmission.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The motivation is clear that both intra- and inter-modality should be considered simultaneously. Experimental results verify this claim.", "weaknesses": "As multi-modality learning has been fully studied, it is questionable that there are few comparisons with the existing works. In the introduction, the author states that existing inter-modality modeling methods can technically capture both inter- and intra-modality\ndependencies (but with some ineffectiveness), however, this statement is not reflected by the experiments. Therefore, it is questionable how their 'ineffiective' performance compared with the proposed 'effective' method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your review and your thoughtful comments. We appreciate you finding our paper well-motivated and claims verified through the experiments. We address your concerns below.\n\n> In the introduction, the author states that existing inter-modality modeling methods can technically capture both inter- and intra-modality dependencies (but with some ineffectiveness), however, this statement is not reflected by the experiments. Therefore, it is questionable how their 'ineffiective' performance compared with the proposed 'effective' method.\n\nTo begin with, it's important to acknowledge that in the context of multimodal classification tasks, either the intermodal or intramodal approach may perform better. The challenge lies in the fact that, prior to analysis, the relative importance of these dependencies for classification purposes is often unknown. Our goal in this paper was to develop a principled framework that can better capture both intra- and inter-modal dependencies when solving any task involving multiple modalities. The fact that inter-modality modeling is not the best approach for all multimodal tasks tells us that the conventional way of doing it is not the most effective. The impact of our proposed framework is demonstrated through across-the-board improvements in aggregate metrics across multiple tasks with diverse characteristics. Specifically, both dependencies are pertinent for the AV-MNIST and MIMIC-III datasets, while intra-modality dependencies prove more beneficial for FastMRI, and only inter-modality dependencies are relevant for the NLVR2 dataset.\n\n---\n\n> How is the performance compared to existing inter-modality modeling methods, such as that in ref.[15]?\n\n\nWe provide a performance comparison with UME (mixture of uni-modal experts model) In Figure 3 of our main paper, we compare UME ref. [15] (mixture of uni-modal expert models) against the proposed I2M2 and show that I2M2 outperforms UME across all knee pathologies. \nAdditionally, we present a comparison with both UME and UMT [15] on AV-MNIST and MIMIC-III datasets below:\n\n|       Model       |  AV-MNIST        | Mortality       | ICD-9 (140 - 239) | ICD-9 (460 - 519)       |\n|-------------------|------------------|-----------------|-------------------|-----------------|\n| UME               | 68.97 ± 0.34     | 77.55 ± 0.26    | 91.42 ± 0.01      | 68.69 ± 0.38    |\n| UMT               | 71.72 ± 0.27     | 77.04 ± 0.59    | 91.33 ± 0.18      | 66.76 ± 0.82    |\n| I2M2 (proposed)              | **72.38 ± 0.17** | **78.10 ± 0.17**| **91.58 ± 0.10**  | **68.88 ± 0.34**|\n\nFurthermore, we point to a fundamental difference between I2M2 and the methods proposed in ref [15]. Methods in ref[15] either capture  intra-modality (UME) or inter-modality (UMT) dependencies. They do not address the existence or necessity of combining both types of dependencies. They suggest using one type of dependency over the other based on their varying strengths. As demonstrated in our paper, the strength of these dependencies varies across different datasets, which we do not have prior knowledge of. Hence, we argue that it is essential to capture both of them. \n\n\n---\n\n> How do we deal with multi-modality when the number of modalities is greater than 2? Should we consider the conditional probabilities on any combination of modalities or just modalities between any pairs of modalities? Further, should all the conditional probabilities be equally treated?\n\nThere is a straightforward way to extend our methodology to more than two modalities, which we discussed in the Limitations section. Specifically, we can process multiple modalities using a single encoder, with a null token to indicate the presence or absence of each modality. For each example, we randomly select a subset of $k$ combinations of conditional probabilities. The model is then constructed based on either the product or the sum of logarithms of these $k$ conditional probabilities. This approach will keep the number of parameters linear, thereby managing complexity effectively.\n\n---\n\n> In the FastMRI experiment, the performance under low SNR k-space data is analyzed. How about the robustness when only one modality is affected? For example, only magnitude modality is disturbed by noise during signal transmission.\n\nWe want to clarify that in MRI the data is acquired in the frequency space and the measurements are in complex domain. An inverse Fourier transform of these complex measurements produces complex images with real and imaginary channels. From these real and imaginary channels one gets the magnitude and the phase channels. Whenever there’s a degradation in SNR it affects both the real and imaginary channels. Hence both the mangitude and phase will be affected (albeit in different ways). Under no circumstances will only the magnitude or phase alone will be affected. It is not a realistic scenario and hence we did not experiment with it. \n\nWe highlight that we provide experiments with various unimodal OOD image and text-based test sets for VQA in Appendix C. Across all these test sets, I2M2 consistently achieves a relative improvement ranging from two to four percent compared to the performance of inter-modality modeling. \n\n---\n\nThank you again for your time and efforts in reviewing our paper, and we hope that you will consider raising your score if you find our response satisfactory.\n\nThank you,   \nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bX3R8frmot", "reviewer": "Reviewer_nGTV", "summary": "Previous supervised multi-modal learning involves mapping multiple modalities to a target label, with previous studies focusing separately on either inter-modality or intra-modality dependencies. This approach may not be optimal, so the proposed inter- & intra-modality modeling (I2M2) framework captures and integrates both dependencies for more accurate predictions. Evaluation using real-world healthcare and vision-and-language datasets shows that the proposed method outperforms traditional methods that focus on only one type of modality dependency.", "questions": "See above weakness.", "limitations": "No potential negative societal impact of their work observed.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is relatively well-written and easy to understand. \n\n2. The proposed method performs well on real-world healthcare and vision-and-language datasets, indicating its potential to be applied in a broader field.", "weaknesses": "1. It would be better if the authors could also report the computational complexity of the proposed method. \n\n2. For visual language modeling, one of the recent popular ways is to use CLIP or CLIP-like models for visual language contrastive pertaining to boost accuracy and performance, it would be interesting for the authors to compare their method against the CLIP model on their datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your review and thoughtful comments. We are glad that you found our paper well-written, easy to understand, and potentially applicable to the broader field. We address your concerns regarding computational complexity and comparison with CLIP below.\n\n> It would be better if the authors could also report the computational complexity of the proposed method.\n\nComputational complexity can refer to either the number of parameters or the training time. In either dimensions, I2M2 only moderately adds to the complexity of the underlying models. \n\nNumber of parameters: The total number of parameters in I2M2 is the sum of the paramters of the inter-modal and intra-modal models used, as discussed in the Limitations section and Section 4.6. Furthermore, we show (in Figure 3 and Figure 4 of the paper) that even when the inter and intra-modality models are allocated the same number of parameters as I2M2,, their performance is significantly worse in comparison to I2M2. .\n\nTraining time: Even for large-scale datasets and models such as VQAv2 and NLVR2, the additional training time on top of pre-trained models was 16 hours for VQA and 4 hours for NLVR2. For other datasets, the increase in training time was minimal, and I2M2 converged within a few epochs. We emphasize that we only added an additional MLP for large models to keep the computational cost in terms of parameters minimal.\n\n---\n\n> For visual language modeling, one of the recent popular ways is to use CLIP or CLIP-like models for visual language contrastive pertaining to boost accuracy and performance, it would be interesting for the authors to compare their method against the CLIP model on their datasets.\n\nWe agree that building multimodal visual language models has become a recent trend. Our proposed approach is compatible with these models, as they work with images, text, or both combined. Therefore, our methodology can be applied to these as well, offering a promising way to initialize the inter and intra-modality models. We leave this investigation for future work.\n\n---\n\nThank you again for your time and efforts in reviewing our paper, and we hope that you will consider raising your score if you find our response satisfactory.\n\nThank you,    \nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0h2TsYItqO", "reviewer": "Reviewer_bvvY", "summary": "The authors present a novel framework, I2M2 (Inter- & Intra-Modality Modeling), designed to enhance supervised multi-modal learning by effectively leveraging multiple modalities. This framework captures both the relationships between different modalities (inter-modality dependencies) and within a single modality (intra-modality dependencies) concerning a target label. The multi-modal learning problem is approached through the lens of generative models, with the target label influencing multiple modalities and their interactions.\nExtensive experimental results have validated the effectiveness of I2M2 using real-world datasets from the healthcare and vision-and-language domains, demonstrating its superiority over traditional methods that focus on a single type of modality dependency.", "questions": "1. I'm curious how the strength of the selection mechanism varies and impacts across datasets, and how this strength is determined?\n2. Why does I2M2 seem to have more significant superiority in OOD scenarios, and is there an effect of data distribution differences on I2M2?", "limitations": "The authors have mentioned some limitations, such as the increased computational cost with more modalities and challenges in model initialization. However, it would be beneficial if they could provide a more detailed discussion of these limitations, including any potential workarounds or future research directions.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel framework, I2M2, which marks a significant departure from traditional approaches in multi-modal learning that typically focus on either inter- or intra-modality dependencies in isolation. \n2. By adopting a generative model perspective to understand multi-modal data and proposing a new data-generating process, the authors offer a fresh outlook on a well-studied problem.", "weaknesses": "1. Further experimental analyses could be strengthened, especially for the selection variable and the strength of the selection mechanism across datasets.\n2. While the paper compares I2M2 with traditional methods, a more comprehensive comparison with the latest state-of-the-art methods in multi-modal learning would better situate the framework within the current research landscape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your review and your thoughtful comments. We appreciate your recognition of our paper as providing a fresh perspective on multi-modal learning. We address your concerns below:\n\n> Further experimental analyses could be strengthened, especially for the selection variable and the strength of the selection mechanism across datasets.\n\n> I'm curious how the strength of the selection mechanism varies and impacts across datasets, and how this strength is determined?\n\n> Why does I2M2 seem to have more significant superiority in OOD scenarios, and is there an effect of data distribution differences on I2M2?\n\nWe assume that the selection variable is always present but its strength varies among different datasets. It is important to note that the strength cannot be determined explicitly. However, if in a dataset the selection effect is strong, the relationships between different types of modalities and the label (inter-modality dependency) are more important. If the strength is weaker, the dependencies between the individual modalities and the label (intra-modality dependencies) are crucial. Each dataset in our paper exhibits different characteristics: Intra-modality dependencies are more beneficial for fastMRI dataset, while inter-modality dependencies are more relevant for NLVR2 dataset. Both dependencies are pertinent for the AV-MNIST, MIMIC-III and VQA datasets. \n\nA key point to note is that we typically do not have prior knowledge of the dependency strength between modalities for classification. Our proposed I2M2 framework considers both intra- and inter-modal dependencies and performs better than using either of them alone. This gap widens in the out-of-distribution (OOD) experiments  (see Appendix C), as the strength of the selection effect can often shift for OOD distributions. In scenarios where models depend heavily on one type of modality dependency, these models often become less reliable when faced with changes in data distribution. In contrast, we observed consistent improvements with I2M2 across all settings.\n\n---\n\n> While the paper compares I2M2 with traditional methods, a more comprehensive comparison with the latest state-of-the-art methods in multi-modal learning would better situate the framework within the current research landscape.\n\nWe would like to emphasize that we indeed considered the SOTA models for ALL the datasets analyzed (see experimental setups of all the datasets). Specifically, for AV-MNIST and MIMIC-III, we selected the top-performing models from MultiBench [1], which recommends these datasets as benchmarks for evaluating multimodal learning capabilities. For FastMRI, our reference is the work by Madaan et al., [2] the only study we identified that focuses on diagnosis for this task. Furthermore, for VQA and NLVR2, we utilized the FIBER model as used in the recent studies [3,4], which achieves state-of-the-art performance on both datasets. \n\n--- \n\n> The authors have mentioned some limitations, such as the increased computational cost with more modalities and challenges in model initialization. However, it would be beneficial if they could provide a more detailed discussion of these limitations, including any potential workarounds or future research directions.\n\nWhile we have discussed potential workarounds and future research directions in the limitations and future work section of our work, we reiterate them here and will elaborate on these points further in the final revision:\n\nFor a large number of modalities, we propose processing all modalities using a single encoder, with a null token to indicate the presence or absence of each modality. For each example, we randomly select a subset of $k$ combinations of conditional probabilities. The model is then constructed based on either the product or the sum of logarithms of these $k$ conditional probabilities. This approach aims to keep the number of parameters linear, thereby managing complexity effectively.\n\nFor model initialization, there are optimization challenges in training multi-modal models from scratch that are not yet fully understood. We believe that investigating these challenges and developing end-to-end training methods is a promising area for future research. \n\n---\n\nThank you again for your time and efforts in reviewing our paper, and we hope that you will consider raising your score if you find our response satisfactory.\n\nThank you,    \nAuthors\n\n---\n\n\nReferences.   \n[1] Liang et al., 21 MultiBench: Multiscale Benchmarks for Multimodal Representation Learning   \n[2] Madaan et al., 23 On Sensitivity and Robustness of Normalization Schemes to Input Distribution Shifts in Automatic MR Image Diagnosis.   \n[3] Dou et al., 22 Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone.    \n[4] Makino et al., 23 https://openreview.net/forum?id=QoRo9QmOAr.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lPfuvUxO8a", "reviewer": "Reviewer_qKeb", "summary": "The proposed inter- & intra-modality modeling (I2M2) framework addresses the limitations of conventional approaches in supervised multi-modal learning. By considering both inter-modality dependencies and intra-modality dependencies, it achieves superior performance in predicting labels. The I2M2 framework offers adaptability without prior knowledge of dependency strength and demonstrates improved accuracy compared to methods focusing solely on one type of modality dependency.", "questions": "1. I am confused about the implementation details of the proposed method, particularly the simulation of q_{x, x'}(y|x, x') or q_{x}(y|x) in Equation 4/5. It is important for the authors to provide a clear description of how these probabilities are computed and how they contribute to the overall model's functionality.\n\n2. A lots of multi-modality models have been proposed to explore inter-modal and intra-modal information. A classic framework involves learning to concatenate two embeddings: one for inter-modal information, trained jointly across modalities, and one for intra-modal information, trained separately. What is  the superiority of the proposed model compared to this approach?\n\n3. The experiment results indicate a consistent pattern across all three datasets, where inter-modality methods perform comparably to the proposed model, while intra-modal methods exhibit significantly worse performance. This raises the question of whether the intra-modal approach is meaningful in processing multi-modal data. The authors should provide a thorough analysis and discussion to address this finding.\n\n4. As is claim in INTRODUCTION part, the proposed method aims to uncover the factors behind performance differences between multi-modal and uni-modal learners. Additional empirical expriment and analysis are necessary to understand how the proposed method explains the observed disparities.", "limitations": "See questions", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The I2M2 framework captures and integrates both inter-modality and intra-modality dependencies, leading to better predictions and enhanced model performance.\n\n2. Without requiring prior knowledge of dependency strength, the I2M2 framework effectively models inter- and intra-modality dependencies, making it versatile and adaptable in different scenarios.\n\n3.  Experimental evaluations on real-world datasets demonstrate the I2M2 framework's superiority over traditional methods that focus on only one type of modality dependency. By considering both inter- and intra-modality dependencies, the I2M2 framework achieves higher accuracy in multi-modal learning tasks", "weaknesses": "See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your review and your thoughtful comments. We are glad that you find our method versatile and adaptable to different scenarios. We address your concerns below:\n\n> I am confused about the implementation details of the proposed method, particularly the simulation of q_{x, x'}(y|x, x') or q_{x}(y|x) in Equation 4/5. \n\nAs explained in section 3.1, our model generates a probabilistic score through four components derived from the joint distribution in equation 1: $p(\\mathbf{y})$, which reflects the softmax bias; the intermediate terms, which correspond to predictive models for each individual modality; and the final term, which represents the multimodal predictive model, incorporating combinations of modalities and the label. Each of these values is a positive scalar. In Equation 2, we multiply them with the denominator as a normalizing constant that does not depend on the label $\\mathbf{y}$. \n\nThe notation $q$ for instance $q_{\\mathbf{x}}(\\mathbf{y} \\mid \\mathbf{x})$ is used to denote the function that maps $ (\\mathbf{y}, \\mathbf{x}) $ to a positive scalar. This function is proportional to $p(\\mathbf{x} \\mid \\mathbf{y})$ because $ p(\\mathbf{y} \\mid \\mathbf{x}) = \\frac{p(\\mathbf{x} \\mid \\mathbf{y}) p(\\mathbf{y})}{p(\\mathbf{x})}$. Thus, by defining $ q_{\\mathbf{x}}(\\mathbf{y} \\mid \\mathbf{x})$ as proportional to $ p(\\mathbf{x} \\mid \\mathbf{y}) $, we have simply renamed $ p(\\mathbf{y} \\mid \\mathbf{x}) $ for convenience. Specifically, $ q_{\\mathbf{x}}(\\mathbf{y} \\mid \\mathbf{x}) = c \\cdot p(\\mathbf{x} \\mid \\mathbf{y})$, where $c$ is a constant that normalizes the function representing the interaction between $\\mathbf{y}$ and $ \\mathbf{x}$. We will further articulate this clearly in the revised manuscript. \n\n---\n\n> A lots of multi-modality models have been proposed to explore inter-modal and intra-modal information. A classic framework involves learning to concatenate two embeddings: one for inter-modal information, trained jointly across modalities, and one for intra-modal information, trained separately. What is the superiority of the proposed model compared to this approach?\n\nWe would appreciate it if the reviewer could provide specific references to the  papers that describe the framework described in your response. To the best of our knowledge, all prior work (most of which is cited in our paper) focuses on either intra- or inter-modality dependencies, but not both. The method described by the reviewer aligns with our approach, where the model output is $\\exp(W_x h(x) + W_x' h'(x) + W_{x, x'} h''(x, x'))$. This is essentially equivalent to the product (or a log-ensemble) of these individual components. We fine-tune this full model end-to-end, rather than simply concatenating the outputs of pre-trained models.\n\n---\n\n> The experiment results indicate a consistent pattern across all three datasets, where inter-modality methods perform comparably to the proposed model, while intra-modal methods exhibit significantly worse performance. This raises the question of whether the intra-modal approach is meaningful in processing multi-modal data. \n\nThe fact that inter-modality modeling is not the best approach for all multimodal tasks indicates that intra-modality dependencies are beneficial, and conventional methods are not the most effective. The impact of intra-modality dependencies is demonstrated through our approach, which leverages both inter- and intra-modality dependencies, resulting in across-the-board improvements in aggregate metrics across multiple tasks with diverse characteristics. Specifically, both dependencies are pertinent for the AV-MNIST and MIMIC-III datasets, intra-modality dependencies prove more beneficial for FastMRI, and only inter-modality dependencies are relevant for the NLVR2 dataset.\n\n---\n\n> As is claim in INTRODUCTION part, the proposed method aims to uncover the factors behind performance differences between multi-modal and uni-modal learners. Additional empirical expriment and analysis are necessary to understand how the proposed method explains the observed disparities.\n\nWe clarify that the goal of this work was to develop a principled approach to solving multi-modal problems that performs consistently well across multiple tasks. This is motivated by the inconsistent performance of conventional methods across different datasets.\n\nTowards that end, we used our data generative process (Figure 1) to explain the performance differences of conventional methods across various datasets. The key takeaway from our work is that if the cross-modality information to predict the label is strong, the relationships between different types of modalities and the label (inter-modality dependency) become more important. Conversely, if the cross-modality information is weak, the dependencies between the individual modalities and the label (intra-modality dependencies) are crucial. Without prior knowledge of the strengths of these dependencies, existing methods often make incomplete assumptions about the datasets, resulting in sub-optimal performance. Our proposed method effectively addresses this by capturing both inter- and intra-modality dependencies. We validated the efficacy of our framework using a battery of experiments involving modalities of completely different nature, including images, text, audio, and even signals in frequency space acquired in healthcare. We would really appreciate if you could  clarify what additional specific experiments and analysis is necessary to improve our work. \n\n---\n\nThank you again for your time and efforts in reviewing our paper, and we hope that you will consider raising your score if you find our response satisfactory.\n\nThank you,    \nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a novel framework, I2M2, designed to enhance supervised multi-modal learning by effectively leveraging multiple modalities. Experimental evaluations on real-world datasets demonstrate the I2M2 framework's superiority over traditional methods.  The method is novel and technically sounded. Some comments about the implementation details of the proposed method and analysis of the superiority compared to other approaches were successfully addressed by the authors’ rebuttal. Some descriptions in the paper are still not clear enough, which needs to be further improved in the final version.
Decision: Accept (poster)