Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning

Authors: divyam  madaan
taro  makino
sumit  chopra
kyunghyun  cho
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 17:17:01

Abstract: Supervised multi-modal learning involves mapping multiple modalities to a target label. Previous studies in this field have concentrated on capturing in isolation either the inter-modality dependencies (the relationships between different modalities and the label) or the intra-modality dependencies (the relationships within a single modality and the label). We argue that these conventional approaches that rely solely on either inter- or intra-modality dependencies may not be optimal in general. We view the multi-modal learning problem from the lens of generative models where we consider the target as a source of multiple modalities and the interaction between them. Towards that end, we propose inter- \& intra-modality modeling (I2M2) framework, which captures and integrates both the inter- and intra-modality dependencies, leading to more accurate predictions. We evaluate our approach using real-world healthcare and vision-and-language datasets with state-of-the-art models, demonstrating superior performance over traditional methods focusing only on one type of modality dependency. The code is available at https://github.com/divyam3897/I2M2.

Main paper: 
 introduction: 
 Supervised multi-modal learning involves mapping input data to a target label, where the data is derived from multiple modalities and information about the boundaries between different modalities is available. This problem has garnered interest in numerous applications, such as autonomous driving [18, 9, 40] , healthcare [28, 67] , robotics [55, 76, 14] , to name a few. We encourage readers to refer to the latest survey papers [79, 36, 42] for recent developments in this field.Despite multi-modal learning being a key paradigm in machine learning, its effectiveness varies across different applications. In some cases, a multi-modal learner outperforms a uni-modal learner [8, 78] , while in others, it may not be as effective as individual uni-modal learners [74, 16] or a simple combination of uni-modal learners [53] . These differing results beg for a principled framework that can explain such discrepancies in multi-modal model performance and provide a general recipe for designing models that can leverage multi-modal data more efficiently and without such shortcomings.In this work, we aim to uncover the underlying factors behind such discrepancies and introduce a more principled approach to multi-modal learning to resolve them. We view the supervised multi-modal learning problem from a probabilistic lens and define the underlying data-generating process. More formally, the proposed data-generating process is shown in Figure 1a . Without loss Figure 1 : Data generating process for various scenarios with two modalities x, x ′ and output y.In the context of multi-modal learning a), the label modulates the individual modalities (referred to as intra-modality dependencies) and the interaction between them (referred to as inter-modality dependency) through the selection variable v. In contrast, conventional approaches assume the graphical model in b) or c). In the graphical model shown in b), the dependency between each individual modality and the label only modulates through the selection variable v. On the other hand, the graph in c) assumes that the dependency between two modalities is independent of the label.of generality, we consider the output y generating data x and x ′ for the two modalities. We define the statistical dependency between the set of modalities and the label using a selection variable v ∈ {0, 1}. Formally,p(y, x, x ′ , v = 1) = p(y)p(x | y)p(x ′ | y)p(v = 1 | x, x ′ , y).This selection variable is always set to one because it is the mechanism that induces the dependencies between the modalities and the label. The strength of this selection mechanism varies across datasets.When the selection effect is strong, inter-modality dependencies (dependencies between the modalities and label) become more significant. Conversely, when the selection effect is weak, intra-modality dependencies (dependencies between the individual modalities and label) are more crucial.At a high-level, our framework assumes that the output label generates the data associated with individual modalities. In addition, it also defines the relationship among different modalities and the label with the selection mechanism. The extent to which the output is dependent on the data from individual modalities and on the cross-modality relationships, differs from use-case to use-case.Given the lack of prior knowledge about the strength of these dependencies on the final task, a multi-modal system must model both the inter-and intra-modality dependencies. We achieve this by building a classifier for each modality to capture the intra-modality dependencies and another classifier to capture the dependencies between the output label and the inter-modality interactions. These classifiers are then combined by constructing a product or log-ensemble of their outputs. We name this approach inter-& intra-modality modeling (I2M2), stemming directly from the above multi-modal generative model.The proposed framework can be used to categorize all the previous works on multi-modal learning into two categories. The first category correspond to the inter-modality modeling methods [4, 6, 19, 27, 45, 43, 8, 78, 2, 32, 75, 78, 56, 57, 15, 83] . This includes methods that predominantly focus on capturing the dependencies between the modalities to predict the target. From our graphical model's perspective these methods are based on the assumption that there are no direct edges from y to x, x ′ (see Figure 1b ). Although these methods can technically capture both inter-and intra-modality dependencies, they often fail to do so effectively [74, 78, 15] . This ineffectiveness stems from their reliance on incomplete underlying assumptions about the generative model for multi-modal learning.The second category correspond to the intra-modality modeling methods [54, 33, 23, 39, 66] . This include approaches that consider the interactions between different modalities that occur only through the label (see Figure 1c ). These approaches do not capture the relationship between the modalities for prediction, which contradicts the objective of multi-modal learning. Inter-modality approaches excel when modalities share significant information to predict the label, while intra-modality approaches are effective when cross-modality information is sparse or absent. Often times, such information is not provided to us when building the multi-modal models.The proposed I2M2 framework addresses this shortcoming by not requiring prior knowledge of the strength of these dependencies. It explicitly models both inter-and intra-modality dependencies, making it adaptable and effective across various conditions. We validate our claims on multiple datasets, demonstrating the benefits of I2M2 over both inter-and intra-modality methods. We apply our method to multiple tasks in healthcare, including automatic diagnosis using knee MRI exams [81] and for mortality and ICD-9 code prediction in the MIMIC-III dataset [31] . We also demonstrate the benefits of I2M2 in multiple vision-and-language tasks such as VQA [3, 65] and NLVR2 [69] . Our evaluation shows the varying strength of dependencies across datasets; intra-modality dependencies are more beneficial for fastMRI dataset, while inter-modality dependencies are more relevant for NLVR2 dataset. Both dependencies are pertinent for the AV-MNIST, MIMIC-III and VQA datasets. I2M2 excels across the board, ensuring robust performance regardless of which dependencies are most significant.2 What is Multi-modal Learning?Multi-modal learning refers to the problem setup where the input is expressed as a set of observations from different modalities. Unlike conventional learning involving data set from a single modality, multi-modal learning can and should exploit the information from all the provided modalities for the purpose of prediction. In this work, we are interested in supervised multi-modal learning, where the goal is to map the inputs from multiple modalities to the targets.We begin with the datasetD = {(x i , x ′ i , y i )} n i=1with n examples. Without loss of generality, y i is the label andx i ∈ X ⊆ R d and x ′ i ∈ X ′ ⊆ R d ′represent data from the two modalities. To define multi-modal learning more formally, we define a multi-modal data generating process in which label y gives rise to both the modalities x and x ′ and the interaction between them (see Figure 1a ). The variable v represents a selection variable that captures the statistical dependencies across the modalities given the label. This selection variable is a binary random variable that is conditioned on all the input modalities and the target. As mentioned earlier, this variable is always present (i.e., v = 1), but its influence varies across datasets. The joint probability in this case can be written as:EQUATIONWhile this might appear similar to the use of selection variables in modeling selection bias [26, 10, 5] , in our context, selection does not refer to selecting examples, but to the mechanism that induces the dependencies between the modalities and the label. Particularly, we use this mechanism to break the conditional independence among the input modalities given the label, which is often referred to as the 'explaining away' phenomenon. The challenge is that, prior to analysis, the relative importance of inter-and intra-modality dependencies for classification is often unknown. Therefore, a multi-modal classifier needs to account for both inter-and intra-modality dependencies.Our data generating process is commonly observed in many real-world scenarios. As a concrete example, consider VQA [3] , a task that involves answering an open-ended question using information from an associated image. Each individual modality -either the image or question -can independently provide clues towards the correct answer [21, 8, 11, 65] , yet these hints alone are often not sufficient to predict the answer. It is only by examining both modalities together that we can accurately infer the correct answer. This combination is captured in our generative model by the selection variable v. Thus, this task requires building separate models to capture the image and text specific information conditioned on the answer (intra-modality dependencies) and the dependency between the image, text modality given the answer (inter-modality dependency) to make the best prediction. This underlines the importance of a modeling approach that not only considers the interaction between the modalities but also uses each modality independently to predict the correct label. 
three ways to capture modality dependencies: 
 Traditional approaches in multi-modal learning model the interaction between different modalities in order to predict the target, primarily by building novel architectures [54, 33, 64, 46, 2, 19, 75, 45, 39, 66] . Although these approaches occasionally outperform uni-modal models, there are cases where they fall short and are less effective than either the uni-modal learners [21, 8, 78] or their ensemble [53] counterparts. Furthermore, to the best of our knowledge, no prior work exists that sheds light on the reasons behind this discrepancy in model performance and provides a solution for the same. In this work we move our focus away from the question of model parameterization given a multi-modal data. Instead, we focus towards uncovering the probabilistic assumptions required to study multi-modal learning.We describe our I2M2 approach that incorporates the modality grouping information by considering models trained on individual modalities, while simultaneously capturing the interaction between the modalities. Next, we group existing studies into inter-modality modeling [4, 6, 19, 27, 45, 43, 2, 32, 75, 78, 56] , which only uses the interaction between different modalities to predict the correct label, and intra-modality modeling [54, 33, 64, 23, 39, 66] , which assumes the conditional independence between the modalities given the target.Starting from Equation (1), we can write the conditional probability over labels as the product of four terms as follows:EQUATIONwhere p (x | y) , p (x ′ | y) and p (v = 1 | y, x, x ′ ) are functions that map inputs (x, y), (x ′ , y) and (x, x ′ , y) to a positive scalar. For clarity, we will use q x (y | x), q x ′ (y | x ′ ), andq x,x ′ (y | x, x ′ ) in lieu of p (x | y) , p (x ′ | y) and p (v = 1 | y, x, x ′) and rewrite the above equation as follows:EQUATIONwhere q x (y | x) captures the conditional probability of the target given x, q x ′ (y | x ′ ) the conditional probability of the target given the other modality x ′ , q x,x ′ (y | x, x ′ ) the conditional probability of the target given both the modalities. We omit v from the right hand side for brevity.This suggests that we should build a separate predictive model for each modality and a model that takes as input the concatenated pair of both modalities. We combine these modality-specific and multi-modal classifiers by building a product of experts (or an additive ensemble in the log-probability space). In this approach, we separately capture the intra-modality dependency within each modality and inter-modality interaction across the modality boundary, to predict the target. This modeling paradigm captures the influence of both individual modalities on the label, as well as the combined impact of both modalities.I2M2 closely aligns with the mutual information framework proposed in the multi-view framework [71, 44] and captures the three different types of mutual information. We motivate and explain this perspective for multi-modal learning from the principles of probabilsitic graphical models by explaining the underlying graphical models that give rise to such a factorization of mutual information.Current research on multi-modal learning mainly focuses on how to parameterize the conditional distribution over the label given multiple modalities. It has seen the development of various architectural strategies, which can be categorized into two main types: The first type is the early fusion [4, 6, 19, 27, 45, 43] which combines the input modalities at an initial stage and processes them using a single, shared model. The second type is the intermediate fusion [2, 32, 75, 78, 56] which uses distinct models for each modality, linked together by fusion modules at different layers. We collectively call these modeling paradigms as inter-modality modeling, as it considers all the modalities together and does not explicitly use the information about modality boundaries, beyond parameterization. Thus, the predictive probability over y can be written as:EQUATIONThis is derived from Equation (3), where q x (y | x) and q x ′ (y | x ′ ) are set to constant functions. This corresponds to removing the direct edges y → x and y → x ′ in the graphical model depicted in Figure 1a , resulting in the graphical model shown in Figure 1b . Consider the example of NLVR [69, 70] , which involves determining whether a sentence accurately describes a pair of images or not. This task requires a model to possess joint understanding of visual and textual information to determine the accuracy of the statement. The dataset was curated to explicitly avoid visual or language bias [68] .To achieve this, each sentence was associated with multiple examples with conflicting labels. This was accomplished by showing workers a set of visually similar yet distinct images and asking them to write sentences that are true for some of the images but not for others.Prior studies for inter-modality modeling [4, 6, 19, 27, 45, 43, 8, 78, 2, 32, 75, 78, 56, 57, 15, 83] mostly focus on emphasizing inter-modality dependencies and identifying an issue with under utilizing these dependencies. While these methods are capable of capturing both inter and intra-modality dependencies, they often fail to do so effectively, especially when inter-modality information is absent or sparse [15] . Additionally, many of these approaches are developed for specific applications, such as person re-identification [35] , multimedia recommendation [48] and sentiment analysis [47] . Unlike these specialized approaches, I2M2 is agnostic to the types of modalities and captures both the inter and intra-modality dependencies explicitly.Intra-modality modeling [54, 33, 64, 23, 39, 66] processes each input modality through separate encoders. It then uses a product of experts model consisting of these uni-modal predictors, where the correlation between the modalities does not predict the target. Consider an example of the tiger detection task, where y = 1 indicates the presence of a tiger. The first modality is shape information, and the second modality is texture information. When y = 0, the first modality has a random shape, and the second modality a random texture. On the contrary, when the label is "tiger", i.e., y = 1, the first modality (shape) has a tiger-like shape, regardless of the second modality (texture) and the same applies to the texture modality. This implies that all the statistical dependencies between modalities manifest themselves via the label (y).We refer to the predictive model for this generative process as intra-modality modeling, where we assume conditional independence among the modalities given the label y. In this case the predictive distribution can be written as:EQUATIONSimilar to Equation ( 4), this model stems from Equation (3), where q x,x ′ (y | x, x ′ ) is treated as a constant function. This corresponds to eliminating the directed edge y → v in the graphical model in Figure 1a , leading to the graphical model shown in Figure 1c . We build a classifier for each modality and combine them by building a product (or a log-ensemble) of these classifiers, ignoring higher-order interactions among the modalities. 
experiments: 
 To differentiate various methods for multi-modal learning, we use audio-vision MNIST (AV-MNIST) dataset [73] , healthcare datasets, such as fastMRI [81] and MIMIC-III [31] , vision-and-language datasets, including VQA-VS [65] (a recently introduced version of VQA), and NLVR2 [70] (the latest version of NLVR). We consider state-of-the-art models for all the datasets. We defer the details of datasets to Appendix A and model hyper-parameters to Appendix B. Experimental setup. AV-MNIST combines audio and visual modalities for MNIST digit (0-9) recognition task. We use LeNet [37] with the top-performing methods from recent studies. Specifically, we employ late fusion (LF), which concatenates the uni-modal feature representations followed by a classification head, and low-rank multimodal fusion (LRTF) [50] from the recent multi-modal benchmark [41] for this dataset.Results. Table 1 shows that inter-modality approaches outperform both intra-modality and single-modality approaches, underscoring the significance of inter-modal interactions as anticipated by the data construction. I2M2 improves the performance by 1%, supporting our claim that both intra-and inter-modality interactions are crucial for this task. Moreover, as expected from the dataset construction, the model trained with the visual modality more effectively predicts the label than the audio component. The performance is enhanced by 4% with intra-modality modeling compared to using only the visual modality. I2M2 eliminates the need to pre-determine which specific dependencies should be modeled, offering a more flexible and effective modeling approach. Experimental setup. This dataset [81, 82] consists of MR scans that include DICOM images and the corresponding raw measurements in the frequency domain (also known as k-space in the MR community), along with slice-level labels. We dissect the complex k-space data into magnitude and phase components, treating them as two distinct modalities for identifying the most significant knee pathologies: 1) anterior cruciate ligament (ACL), 2) meniscus tear, 3) cartilage and 4) others grouping all the other pathologies. We use PreactResNet-18 [25] following Madaan et al. [52] , the only study targeting diagnosis for this task.Results. Consistent with our previous experiments, I2M2 outperforms the unimodal magnitude and phase models, inter-modality and intra-modality methods, as shown in Figure 2 . In this experiment, unlike the AV-MNIST dataset, inter-modality modeling degrades the performance compared to intra-modality modeling. Despite the opposite trend, I2M2 generally worked better than either of them across all pathologies. The superior performance of I2M2 highlights its ability to perform well even when one type of modality dependency is missing by effectively capturing the other.Even when compared with root-sum-of-squares (RSS) -de facto standard way of representing MR images in deep learning that benefits from the enhanced signal-to-noise ratio (SNR) offered by multi-coil data, I2M2 achieves performance superior to the model trained with RSS images across all pathologies. This finding is notable, given the higher SNR of RSS images due to the use of multi-coil data compared to our alternative representations synthesized by simulated single-coil output, resulting in lower SNR. This is useful because, until this point, we could not benefit from multiple modalities in complex MR images relative to RSS.Lastly, the acquisition of MRI data is often a combination of multiple signals, which introduces background noise. Such background noise is easily impacted by the acquisition environments. It is thus important to evaluate the generalization of our methodology while varying the SNR ratios during inference. We show the effectiveness of I2M2 over inter-modality modeling for varying SNR levels by increasing levels of Rician noise [61, 22] in Appendix C.Experimental setup. The MIMIC-III dataset [31] encompasses ten years of intensive care unit (ICU) patient data from Beth Israel Deaconess Medical Center. We divide the dataset into two modalities [60, 41] : 1) the time-series modality, which includes hourly medical measurements over 24 hours, and 2) the static modality, capturing a patient's medical information. We consider three tasks, namely a) mortality prediction of a patient within one day, two days, three days, one week, one year and beyond, and b) two binary classification tasks for ICD-9 codes, one to assess if a patient falls under group 1 (codes 140-239; neoplasms) and another for group 7 (codes 460-519; diseases of the respiratory system). We adopt the best-performing models from the recent multi-modal learning benchmarks [60, 41] for this dataset (see Appendix B for more details).Results. Table 2 shows that I2M2 enhances performance across all tasks when compared to methods that focus solely on either inter-modality or intra-modality dependencies. Both modalities are predictive of mortality, but their effectiveness varies across different groups in predicting ICD-9 codes. Specifically, for ICD-9 codes 140-239, which are associated with neoplasms, the static modality proved more effective. This is likely because it includes factors such as the patient's advanced age and the presence of chronic diseases, which are known to increase the risk of neoplasms.For ICD-9 codes 460-519, the model trained with time-series modality showed better performance, as hourly measurements are useful to identify minor changes in respiratory functions, potentially signaling the onset or exacerbation of the disease.For both mortality and ICD-9 prediction, we found that the intra-modality model obtains comparable performance to the inter-modality model, highlighting that both the individual modalities and their interaction are predictive of the target for these tasks. As expected from our generative model, which formed the basis for our I2M2, the importance of these dependencies is contingent on the specific task label, and our method uses them effectively.Experimental setup. NLVR2 [68] represents a binary classification task in which the goal is to determine whether the text description correctly describes a pair of two images. The model takes as input two images and a text statement describing those images and predicts whether the text describes both images correctly. As discussed in Section 3.2, this dataset was constructed to minimize unimodal bias. For this dataset, we use the state-of-the-art FIBER model [13, 53] , which takes a pair of images and associated text as input and produces a binary label as its output. We fine-tune the full model consisting of Swin Transformer [51] for the image backbone, and RoBERTa [49] for the text backbone and an MLP classifier on top of the encoder for five seeds.Results. Table 3 shows that inter-modality modeling and I2M2 yield similar performance, which is substantially higher than the unimodal models and intra-modality model. This is because each of the image-only and the text-only models attains a chance-level accuracy for this dataset. This shows that neither the isolated text nor the image alone can make meaningful predictions in this problem. This can be attributed to the careful construction of the dataset, which eliminates language and visual biases and underscores the importance of inter-modal interaction within this dataset [68] . It demonstrates the ability of I2M2 to effectively disregard the uninformative intra-modality dependencies when predicting the target for this dataset. This aligns with our observations from fastMRI, where I2M2 disregarded the inter-modality dependencies.Experimental setup. The objective of VQA is to answer questions about images, as detailed in Section 2. The labels comprise 3, 129 of the most common answers in the training and validation sets.The evaluation comprises IID and out-of-distribution (OOD) test sets released by VQA-VS [65] . We report the average OOD accuracy across nine OOD test sets. Additional details on the OOD test sets are provided in Appendix A, along with a detailed performance breakdown in Appendix C. Similar to NLVR2, we use the state-of-the-art FIBER model [13] for training on VQA-VS dataset. We use the pre-trained model weights -Swin-Base [51] and RoBERTa-Base [49] for vision and text encoders and train an MLP classifier on top of the pre-trained encoders following Makino et al. [53] . We use the VQA score metric across five random seeds to compare model performance. Figure 4 : AUROC comparison with WideRes-Net models. We compare magnitude-only, phase-only models, inter-modality model trained with architecture WideResNet-20-3 to our I2M2 trained with PreactResNet-18 across various knee pathologies in the fastMRI dataset (bars are in the same order). I2M2 obtains higher performance across all pathologies in comparison to the wider models.Results. Table 3 shows the IID accuracy for the VQA-VS dataset. It is evident that I2M2 surpasses inter-modality modeling and unimodal models in performance, emphasizing the importance of using both inter-modality and intra-modality dependencies. This improved performance is achieved by leveraging the dependencies between the modalities and individual modalities to predict the target.While the image-only model does not effectively predict the final task, the text-only model obtains 17.86% higher performance. This improvement can be primarily attributed to the language bias present within this dataset, as highlighted in previous works [21, 1, 11, 65] . While all models suffer a drop in performance for the OOD test-sets, I2M2 achieves a relative improvement of 5.86% and 19.35% in comparison to inter-modality modeling and intra-modality modeling respectively. This highlights that addressing distribution shifts involves not only improving the individual experts but the robustness can also be enhanced through redundancy.While aggregate metrics provide a broad overview, they often miss crucial details. We show that I2M2 surpasses other ensemble and wider models, even with a fixed parameter budget. Moreover, I2M2 avoids spurious dependencies between modalities and labels, a common issue in other models.Comparison between models with identical parameter counts. To determine whether the performance improvement by I2M2 is due to the additional parameters or the specific manner in which the experts are combined, we compare it with various mixture of expert models. Particularly, we compare with an ensemble of three magnitude models, an ensemble of three phase models, an ensemble of magnitude and phase models and an ensemble of magnitude, phase and inter-modality model in Figure 3 . We observe that across all pathologies, the ensemble outperforms individual unimodal models but I2M2 obtains better performance than all the ensemble models. This demonstrates that our I2M2 is more effective, even when constrained with a fixed parameter budget. We further contrast I2M2 using PreactResNet-18 with unimodal and inter-modality models using WideResNet-20-3 in Figure 4 . Across various pathologies, I2M2 outperforms in terms of AUROC, reinforcing our view that the effectiveness stems more from our training approach and the integration of individual experts rather than merely from expanding the number of model parameters.Analysis of common mistakes. The training, validation, and test sets of VQA-VS contain a range of inter-and intra-modality spurious dependencies that are conditioned on specific labels. For example, the label "tennis" often correlates with words like "what", "sport", and "is" in the question, whereas the label "kite" is linked with a "kite" in the image. Similarly, the words "how" and "many" in We visualize random instances without text, image and multi-modal spurious dependencies. Specifically, words like "what", "sport", and "is" in questions, the presence of "kite" in the image, and a combination of "how" and "many" in the question with animals in the image are spuriously correlated with the labels "tennis", "kite", and "one", "two", "three" respectively in the IID sets. I2M2 using a product of experts correctly predicts the target even when the spurious dependencies are absent, while individual expert models do not.questions about animals or birds are typically associated with the labels "one", "two", or "three". In Figure 5 , we purposefully chose examples where such spurious dependencies are absent from the OOD test-sets [65] to demonstrate that the I2M2 can accurately predict in these scenarios, unlike the other models. We observe that most models overlook the question or image content in such instances, defaulting to "kiting", "tennis", and "one", "two", or "three" as answers. On the contrary, I2M2 can accurately predict in these scenarios in comparison to these models. We find that in over 30% of cases where each expert fails individually, I2M2 succeeds in making the correct prediction. 
limitations and future work: 
 Linear relationship between model size and number of modalities. The implementation of I2M2 leads to a corresponding growth in model size with each added modality, which, although effective, results in higher computational costs. Specifically, the computational complexity of the model's forward pass increases linearly with the addition of modalities. For cases with a small number of modalities, such as three or four, we can directly apply our modeling paradigm and iterate through all possible combinations of these modalities to construct our product of experts model. This method is straightforward due to the manageable number of combinations. For scenarios involving a larger number of modalities, we propose the following approach for future investigation. We envision employing a single network that receives all modalities as input. If any modality is absent for a specific example, this network receives a null token instead. For each example, we randomly select a subset of combinations of conditional probabilities. The model can then constructed based on either the product or the sum of logarithms of these conditional probabilities. This approach will keep the number of parameters linear, thereby managing complexity effectively. Challenges in model initialization. We found it more effective to train models for each modality separately before fine-tuning them jointly, as opposed to training them jointly from scratch, as shown in Table 5 . This trend was consistent across all the datasets. This aligns with the findings of many previous works [38, 17, 77, 30] , which have attributed this phenomenon to model-dominance effect [77] , where single high performing model tends to dominate, and learner collusion [30] , wherein learners bias their predictive distributions in opposing directions, canceling each other out when aggregated. This suggests that there are optimization challenges in training multi-modal models from scratch that are not yet fully understood. We believe that investigating these challenges and developing end-to-end training methods is a promising area for future research. 
conclusion: 
 In this paper, we proposed inter-& intra-modality modeling (I2M2) for multi-modal learning, capturing both inter-modality and intra-modality dependencies. Applied to real-world datasets in healthcare and vision-language domains, I2M2 consistently outperformed conventional methods that rely solely on inter-or intra-modality dependencies and excelled in both in-distribution and out-of-distribution scenarios. Its versatility and modality-agnostic nature make I2M2 a valuable tool, providing a solid foundation for future research and applications in multi-modal learning. 
societal impact: 
 Contents posted on social media and other internet platforms are becoming increasingly more complex. They are no longer in just one modality, such as text-only, image-only or audio-only. These online contents are often multi-modal, requiring one to consider multiple modalities simultaneously to grasp the true meaning of these contents. As was pointed out earlier by Kiela et al. [34] in their hateful meme challenge, for instance, many harmful contents online require a holistic understanding of the text and the associated images, and text-only or image-only interpretation may miss the harmful nature of those contents. Advances in multi-modal learning, such as this work, will help us build a more effective content understanding system that can enable us to build a better automated filtering system to keep online platforms less harmful. We also acknowledge, however, that a better capability of multi-modal understanding can be used to build more advanced recommendation systems that may negatively contribute to the consumption of media and news by users.Organization. In the supplementary material, we provide a description of the datasets in Appendix A, implementation details in Appendix B and additional results in Appendix C. 
 
 Appendix: A.1 Audio-Vision MNIST Audio-Vision MNIST (AV-MNIST) has been a popular toy benchmark [58, 73, 41] to evaluate the performance of existing models for multi-modal learning. The images are collected from the MNIST dataset [37] and audio from the free spoken digit dataset (FSDD) [29] containing human-spoken digits. 75% of energy is removed from the visual modality with PCA and noise is added to the audio modality from the ESC-50 dataset [59] . We use 55000 examples for training, 5000 validation, and 10000 testing examples for evaluation.Problem setup. The process of MR data acquisition involves exposing the human subject to varying magnetic field and radio-frequency pulses and capturing the resulting electromagnetic responses from the human body. These electromagnetic responses are measured by devices called receiver coils that are positioned in the vicinity of the tissue or organ being imaged. The measurements captured by the receiver coils are in the Fourier domain (a.k.a., k-space in the MR community), where each coil produces a partial image with different spatial sensitivity across different parts of the volume.The generation of ground truth images using the data from multiple coils consists of two steps. The first step involves an Inverse Fourier Transform of the k-space data from an individual coil to generate coil specific images. In the second step, we generate a single image by combining the magnitudes of the individual coil images voxel by voxel using the root-sum-of-squares (RSS) method [62] :EQUATIONwhere x c are the k-space measurements acquired by the c-th coil and and n c is the number of receiver coils. The images generated by RSS method does not contain the phase information and the rawdata is often not openly released, which has slowed the progress in researching the task of disease identification using the raw measurements. Further, the phase of the acquired data is often discarded for diagnosis. To this end, even for MR image reconstruction majority of the methods used synthetic k-space data obtained from the ground-truth spatially reconstructed images [63, 80, 12] .In this research, we argue that blindly constraining the inputs of the DNN models to be the same as what a human would use to render a diagnosis is sub-optimal. We hypothesize that using the raw measurements as input to the DNNs, without regards to its human interpretability, will allow us to build systems that are more accurate at predicting diseases, by enabling the DNN models to automatically extract the most informative diagnostic signals. Towards that end, we explore the utility of using magnitude and phase of the input signal as two modalities for the DNN models and understand the role of the phase component of the complex data acquired by the MR scanners for disease identification.Dataset. The fastMRI dataset [81] was the first large-scale dataset that consisted of raw k-space data alongside anonymized clinical magnetic resonance (MR) images. In this work for the purpose of simplicity we aggregate the data from multiple coils and emulate it to be originating from a single coil using the method proposed in Tygert and Zbontar [72] , Zbontar et al. [81] to compute emulated single-coil data.FastMRI+ [82] augmented this dataset with pathology annotations. These annotations motivated our exploration into the potential of the raw k-space data components for automated diagnosis. Specifically, we focused on knee pathologies, using emulated single-coil k-space data with slice-level labels. We dissect the complex k-space data into magnitude and phase components, treating them as two distinct modalities for identifying clinically important pathologies in MR scans.We consider the most significant knee pathalogies as highlighted by the clinicians: 1) Anterior Cruciate ligament (ACL) with 1,443 annotations of 254 subjects, 2) Meniscus tear with 5,658 annotations of 663 subjects, and 3) Cartilage with 3,600 annotations of 710 subjects and 4) others containing the group of all the other pathologies. The slices were cropped to 320 × 320 and 15% of the dataset for used for validation and test sets.The Medical Information Mart for Intensive Care (MIMIC-III) dataset is a popular medical benchmark consisting of Electronic Health Records (EHRs) of of over 40, 000 patients at Beth Israel Deaconess Medical Center from 2001-2012. The dataset is categorized into two modalities [60, 41] . First, it consists of the static modality, which contains five medical features of the patient including chronic diseases, admission type and age. Second, it is composed of time-series modality, which consists of twelve different measurements recorded hourly over a 24-hour period.We consider the following two tasks for this dataset:• Mortality prediction. The objective of this task is to predict whether the patient dies in one day, two day, three day, one week, one year, or longer than one year. The evaluation of vision and language models' capability to reason about visual compositions has been an important direction of research in the field. Natural Language Vision Reasoning (NLVR) [69] played an important role with the release of synthetic images, text, or combinations of both for analysis. The dataset consists of pairs of images against a natural language sentence. The task is to ascertain if the sentence accurately describes (True) or inaccurately describes (False) the image pair.Expanding on this, the NLVR2 dataset [70] incorporated real-world photographs. The dataset was curated to avoid visual or language bias [68] by ensuring that each sentence was paired with multiple examples with distinct labels. This was achieved by showing workers a collection of visually similar yet distinct images and asking them to write sentences that are true for some images but false for others. The dataset consists of 59, 677 examples, each with an image resolution of 384 pixels.Problem setup. The task of visual question answering (VQA) involves answering questions related to a given image. This task has been widely adopted as a benchmark to evaluate the performance of models that integrate both the vision and language modalities. Following prior studies, we consider it as a multi-label classification problem consisting of 3,129 distinct labels.Background. Initial VQA datasets were found to exhibit a significant issue: models were able to answer questions even without considering images. This issue led to the creation of the VQA-CP v2 dataset [1] , which was constructed to emphasize out-of-distribution (OOD) robustness by deliberately varying the distribution of answers for identical types of questions between the training and testing phases. This dataset was designed to incentivize models that properly rely on images for answering questions.Despite its intention, VQA-CP v2 still exhibited several issues [11, 65] . First, it utilized an OOD test set for the purpose of model selection. Second, the models had to be retrained to assess their performance under IID conditions. Si et al. [65] recently addressed these problems in VQA-VS by introducing new IID validation and test splits from the original VQA v2 dataset, along with OOD test sets based on images, language, and multimodal shortcuts.Dataset. VQA-VS [65] AV-MNIST. We use the LeNet [37] model for both the image and audio modality following Liang et al. [41] . For training all the models, we optimize using cross-entropy loss with SGD using learning rate and weight decay equal to 5 × 10 -2 and 1 × 10 -4 for 25 epochs. Low-Rank Tensor Fusion (LRTF) uses rank 40 and the output dimension is equal to be 120. We report the mean and standard deviation across five runs with random seeds. All experiments were conducted on a single NVIDIA A100 GPU.FastMRI. For our baseline model, we use PreactResNet-18 [24] following Madaan et al. [52] and use early fusion to capture the inter-modality dependencies. We report the mean and standard deviation of AUROC [7] and balanced accuracy [20] for all our experimental analysis across five independent runs. We conduct a grid search on the learning rate and weight decay for this dataset and report the optimal values in Table B .6. We employ early stopping based on the average AUROC across all pathologies to store the best model, which is then used for reporting the results. -ONLY 1 × 10 -5 1 × 10 -3 PHASE-ONLY 1 × 10 -4 1 × 10 -1 INTRA-MODALITY 1 × 10 -6 1 × 10 -3 INTER-MODALITY 1 × 10 -4 1 × 10 -2 I2M2 1 × 10 -6 1 × 10 -2MIMIC-III. We follow the experimental setup used by Purushotham et al. [60] , Liang et al. [41] for this dataset. The static encoder and decoder is a two-layer MLP with LeakyReLU activation function.The time-series encoder and decoder is a GRU with the hidden dimension equal to 30. For capturing the inter-modality dependencies for this dataset, we use late fusion. We use a batch-size of 40 and RMSProp with a learning rate of 1 × 10 -3 to train for twenty epochs across all the tasks. The mean and standard deviation are reported across five independent runs for all the models and tasks.NLVR2 and VQA-VS. We use the FIBER model [13] for both the datasets. It consists of the Swin Transformer [51] for the image backbone, and RoBERTa [49] for the text backbone. We fine-tune a MLP classifier on top of the encoder with learning rate 1 × 10 -4 for VQA-VS. For NLVR2, we fine-tune the full-model with the learning rate 1 × 10 -5 for five seeds. We report the VQA score for VQA-VS and accuracy for NLVR2 with mean and standard deviation. Comparison of AUROC performance for PreactResNet-18 trained with inter-modality modeling and I2M2 on ACL, meniscus tear, cartilage and other pathologies with increasing levels of Ricean-noise. We observe that I2M2 obtains obtains equal or better performance for majority of the noise levels compared to inter-modality modeling.Results with distribution shifts with VQA. I2M2 relies on both the inter-and intra-modality dependencies. In scenarios where models depend heavily on one type of modality dependency, they often become less reliable when faced with changes in data distribution. This phenomenon is evident in Figure C .6, where we present the results for various OOD image, text and multi-modal based test-sets. While all models suffer a drop in performance for all the test-sets, it is noteworthy that across all these test-sets, I2M2 consistently achieves a relative improvement ranging from two to four percent when compared to the performance of inter-modality modeling. Furthermore, it achieves a performance gain of 20 to 25% compared to models trained solely on text inputs. This highlights that addressing distribution shifts effectively involves not only improving the individual experts but the robustness can also be enhanced through redundancy.Results with distribution shifts with FastMRI. The acquisition of MRI data is often a combination of multiple signals, which introduces background noise. In fact among various factors, noise is by-far the biggest factor that contributes towards image deterioration when going from a high-field scanner (scanner with high magnetic field strength) to a low-field scanner (scanner with low magnetic field strength). Despite being inexpensive these scanners have not seen widespread adoption because of the poor diagnostic quality of the images generated by them. Evaluation of any methodology on low signal-to-noise ratio (SNR) images is important for it gives some insights into how the methodology will generalize to low-field scanners, potentially leading to transformative impact in healthcare by facilitating the use of these scanners which are inexpensive and relatively accessible.Due to the dearth of data collected from the low-SNR MR scanners, we simulate the low-SNR k-space data during inference. Particularly, we know that the noise in the acquired k-space is Gaussian distributed, which leads to the ground truth images having noise that has a Rician distribution [61, 22] . Thus, to degrade the signal, we add Gaussian noise to both the real and imaginary channels. We notice that for all types of pathologies, I2M2 performs better in terms of AUROC at lower noise levels. This improvement persists for meniscus and cartilage pathologies even at higher noise levels. This enhanced performance is attributed to the noise addition impacting both real and imaginary channels, affecting the magnitude and phase modalities, respectively. I2M2 captures the interactions within and between these modalities, showing greater resilience to this noise due to redundancy. 2.24 (± 0.02) 4.32 (± 1.28) 0.67 (± 0.01) H( y | x, x ′ ) 0.95 (± 0.08) 5.33 (± 0.36) 0.17 (± 0.02) Entropy measurement. Table C .7 shows the entropy of the label distribution across three different datasets, and it compares these values with the average entropy of the predictive distribution for both unimodal models and inter-modality models, post fine-tuning from I2M2. In the cases of AV-MNIST and VQA-VS, consistent with results in Section 4, we observe that both the unimodal experts and the inter-modality model exhibit lower entropy. On the other hand, in the NLVR2 dataset, while the unimodal experts demonstrate a high average entropy, the inter-modality model exhibits significantly lower entropy in its predictive distribution. This results from the interaction between the modalities being predictive of the target, with no significant information found in the modalities when considered separately, as also illustrated in Figure C.6. 


Decision: The paper presents a novel framework, I2M2, designed to enhance supervised multi-modal learning by effectively leveraging multiple modalities. Experimental evaluations on real-world datasets demonstrate the I2M2 framework's superiority over traditional methods.  The method is novel and technically sounded. Some comments about the implementation details of the proposed method and analysis of the superiority compared to other approaches were successfully addressed by the authors’ rebuttal. Some descriptions in the paper are still not clear enough, which needs to be further improved in the final version.

Reviews: review: paper_summary: paper_summary: This paper studies supervised learning for multi-modal data. Previous works can be categorized into inter-modality learning and intro-modality learning. Inter-modality learning aims to learn multi-modal data jointly by techniques such as feature fusion. Intral-modality learning focuses on learning uni-modal data separately. This paper attempts to unify these two frameworks by building a probabilistic graphical model of the multi-modal data. The previous inter- and intra-modal learning frameworks can be regarded as special cases of this graphical model by assuming specific independence of data. The joint modeling framework is expected to automatically infer whether the inter- or intra-information is important for given tasks. Additionally, the resulting model is simple to implement, by combining the inter- and intra-learning models together.

Experiments are conducted on several tasks, such as auto-vision, healthcare, and vision-language. The results show that the proposed combination can improve performances of each single model, intra-modality combinations and inter-modality feature fusions.
strength_weakness: strength_weakness: 1. For the proposed model, are the inter-modal classifier and intra-modal classifiers trained from scratch? Can we simply use pre-trained ones? If the model should be re-trained or fine-tuned, it might be computationally expensive.

2. As the paper says, the proposed model is similar to a mixture of experts. It should be helpful to discuss more about this point and previous approaches of mixing experts in the related work part. The discussion could help to highlight the difference and contribution.

3. In experiments, the details of intra-modality seem missing. Do the authors choose competitive baselines for this class of methods? Moreover, for inter-modal models, Late fusion seems to be a simple baseline. LRTF was proposed in 2018. My concern is whether the results hold for more recent and competitive models.
questions: questions: Can the author explain more about the derivation of Eq. (3)? How to replace $p(y \mid x)$ by $q(x \mid y)$? The former is probability of $y$ while the latter is probability of $x$.
limitations: limitations: Yes, the authors discuss about limitations.
paper_summary: paper_summary: This paper proposes a framework for multi-modal learning called inter- & intra-modality modeling (I2M2). I2M2 can simultaneously capture inter-modality dependencies (relationships between different modalities) and intra-modality dependencies (relationships within a single modality). This approach aims to improve the accuracy of predictions by integrating both types of dependencies, rather than focusing on one. Experiments on health care data verify the advantage of utilizing both inter- and intra-modality.
strength_weakness: strength_weakness: As multi-modality learning has been fully studied, it is questionable that there are few comparisons with the existing works. In the introduction, the author states that existing inter-modality modeling methods can technically capture both inter- and intra-modality
dependencies (but with some ineffectiveness), however, this statement is not reflected by the experiments. Therefore, it is questionable how their 'ineffiective' performance compared with the proposed 'effective' method.
questions: questions: 1. How is the performance compared to existing inter-modality modeling methods, such as that in ref.[15]?
2. How do we deal with multi-modality when the number of modalities is greater than 2? Should we consider the conditional probabilities on any combination of modalities or just modalities between any pairs of modalities? Further, should all the conditional probabilities be equally treated?
3. In the FastMRI experiment, the performance under low SNR k-space data is analyzed. How about the robustness when only one modality is affected? For example, only magnitude modality is disturbed by noise during signal transmission.
limitations: limitations: N/A
paper_summary: paper_summary: Previous supervised multi-modal learning involves mapping multiple modalities to a target label, with previous studies focusing separately on either inter-modality or intra-modality dependencies. This approach may not be optimal, so the proposed inter- & intra-modality modeling (I2M2) framework captures and integrates both dependencies for more accurate predictions. Evaluation using real-world healthcare and vision-and-language datasets shows that the proposed method outperforms traditional methods that focus on only one type of modality dependency.
strength_weakness: strength_weakness: 1. It would be better if the authors could also report the computational complexity of the proposed method. 

2. For visual language modeling, one of the recent popular ways is to use CLIP or CLIP-like models for visual language contrastive pertaining to boost accuracy and performance, it would be interesting for the authors to compare their method against the CLIP model on their datasets.
questions: questions: See above weakness.
limitations: limitations: No potential negative societal impact of their work observed.
paper_summary: paper_summary: The authors present a novel framework, I2M2 (Inter- & Intra-Modality Modeling), designed to enhance supervised multi-modal learning by effectively leveraging multiple modalities. This framework captures both the relationships between different modalities (inter-modality dependencies) and within a single modality (intra-modality dependencies) concerning a target label. The multi-modal learning problem is approached through the lens of generative models, with the target label influencing multiple modalities and their interactions.
Extensive experimental results have validated the effectiveness of I2M2 using real-world datasets from the healthcare and vision-and-language domains, demonstrating its superiority over traditional methods that focus on a single type of modality dependency.
strength_weakness: strength_weakness: 1. Further experimental analyses could be strengthened, especially for the selection variable and the strength of the selection mechanism across datasets.
2. While the paper compares I2M2 with traditional methods, a more comprehensive comparison with the latest state-of-the-art methods in multi-modal learning would better situate the framework within the current research landscape.
questions: questions: 1. I'm curious how the strength of the selection mechanism varies and impacts across datasets, and how this strength is determined?
2. Why does I2M2 seem to have more significant superiority in OOD scenarios, and is there an effect of data distribution differences on I2M2?
limitations: limitations: The authors have mentioned some limitations, such as the increased computational cost with more modalities and challenges in model initialization. However, it would be beneficial if they could provide a more detailed discussion of these limitations, including any potential workarounds or future research directions.
paper_summary: paper_summary: The proposed inter- & intra-modality modeling (I2M2) framework addresses the limitations of conventional approaches in supervised multi-modal learning. By considering both inter-modality dependencies and intra-modality dependencies, it achieves superior performance in predicting labels. The I2M2 framework offers adaptability without prior knowledge of dependency strength and demonstrates improved accuracy compared to methods focusing solely on one type of modality dependency.
strength_weakness: strength_weakness: See questions
questions: questions: 1. I am confused about the implementation details of the proposed method, particularly the simulation of q_{x, x'}(y|x, x') or q_{x}(y|x) in Equation 4/5. It is important for the authors to provide a clear description of how these probabilities are computed and how they contribute to the overall model's functionality.

2. A lots of multi-modality models have been proposed to explore inter-modal and intra-modal information. A classic framework involves learning to concatenate two embeddings: one for inter-modal information, trained jointly across modalities, and one for intra-modal information, trained separately. What is  the superiority of the proposed model compared to this approach?

3. The experiment results indicate a consistent pattern across all three datasets, where inter-modality methods perform comparably to the proposed model, while intra-modal methods exhibit significantly worse performance. This raises the question of whether the intra-modal approach is meaningful in processing multi-modal data. The authors should provide a thorough analysis and discussion to address this finding.

4. As is claim in INTRODUCTION part, the proposed method aims to uncover the factors behind performance differences between multi-modal and uni-modal learners. Additional empirical expriment and analysis are necessary to understand how the proposed method explains the observed disparities.
limitations: limitations: See questions

Comments: title: Official Comment of Submission7139 by Reviewer qKeb
Thank you for your response [2]
Thank you for your response [1]
Official Comment of Submission7139 by Reviewer qKeb
comment: Thanks for the authors' responses. I will keep the score.
Thanks to the reviewers for their response. I have increased my score.
Thanks for your detailed  response, most of my concerns have been solved, I will upgrade my scores. But I have to mention that some descriptions in the paper are not clear enough (other reviewers also have low confidence), and hope to see a more clear expression in your final version.
> Thanks for your reply, can I understand that your method can effectively measure the relationship between different modes and the dependence between modes? However, this property has not been reflected in subsequent experiments, and we hope to see some more valuable experimental results, rather than just the improvement of algorithm performance.

We've included additional experiments in the appendix, such as how the entropy changes and detailed performance comparisons of all methods under OOD distribution shifts. These may not have been fully evident in the main content and we will incorporate them into the main paper in the final revision. Do you have any further suggestions for other valuable experiments we should conduct?

---

Thank you,
Authors
Dear reviewer,

Thanks for your time and reply. We address your remaining concerns below:

> For the first question, the author may have misunderstood my question. My question is actually how do you estimate these conditional probabilities in your experiments or other real-world scenarios? I have re-checked the paper and it does not explain these specific experimental details clearly.

The conditional probabilities are estimated by training these predictive models using the maximum likelihood principle. These probabilities are obtained from the softmax of the logits produced by these predictive models (refer to line 134). 

---


> In fact, designing an intra-modal module and an inter-modal module to jointly solve multi-modal problems has been proposed in many fields, and I have simply listed some literature for the author's reference:

We emphasize that our method is fundamentally different from all the referenced methods. Based on the definitions in our paper, all these methods capture either inter- or intra-modality dependencies, but not both. We distinguish our methodology from all the papers you referenced below:

1. Deep-HOSeq: Deep higher order sequence fusion for multimodal sentiment analysis by Verma, Sunny et al., focuses on inter-modality modeling by our definition. Both Equation 2 and Equation 4 in their work focus on inter-modality, employing different parameterizations, as both equations apply non-linearity to the concatenated unimodal representation $T_{Val}​$.

2. In "Attention Bottlenecks for Multimodal Fusion" by Nagrani et al., referring to Figure 1, late fusion corresponds to intra-modality modeling. The other approaches depicted are simply different parameterizing schemes for inter-modality modeling within our framework. In contrast to their method, we propose that it is important to capture both inter and intra-modality dependencies. 

3. Equations 9, 10, and 11 are the most important in the paper "Improving what cross-modal retrieval models learn through object-oriented inter-and intra-modal attention networks" by Huang, Po-Yao, et al., but unfortunately, they are not well-formed and factually incorrect, making them difficult to interpret. It's unclear which variables $v$ and $t$ are being summed over in these equations. If summation is indeed intended over $v$ and $t$, then Equations 9 and 10 would not logically depend on $v$ and $t$, which introduces further confusion and undermines their correctness.

4. The paper "DPHANet: Discriminative Parallel and Hierarchical Attention Network for Natural Language Video Localization" by Chen, Ruihan, et al. clearly fits our definition of an inter-modality model. Figure 2 and the corresponding sections in the paper show that the concatenation of cross-modal interaction and intra-modal self-correlation is used as input to convolution layers with non-linearities and transformer blocks, thereby only focusing on inter-modality dependencies.

5. In the paper self-Supervised Multi-Modal Knowledge Graph Contrastive Hashing for Cross-Modal Search. by Liang, Meiyu, et al., there are separate labels for each modality as well as for the multi-modal case, which differs from our problem setup and methodology. They construct three distinct label sets based on textual similarity, image similarity, and a third set that combines these similarities (all derived from a knowledge graph). Their approach focuses on creating representations to solve three different problems independently, rather than using different types of dependencies that contribute to solve a single problem.

6. The paper "Self-Supervised Intra-Modal and Cross-Modal Contrastive Learning for Point Cloud Understanding" by Wu, Yue, et al. aligns with our definition of intra-modality modeling. The model does not capture any inter-modality dependencies, as defined in our framework. The contrastive loss function in Section III.D is a linear combination between the modality representations, as it simply evaluates the similarity between the two modality representations without capturing the non-linear interactions or dependencies between the modalities.

> I  have noticed the experimental results in the FastMRI dataset, But this experiment is in comparison to the intra-modality methods proposed by other people. As for the proposed intra-modal module in this paper, I have to mention that the results can only show the superiority of the inter-modal module in the ablation study of Tables 2 & 3.

Could you clarify what you mean by "intra-modality methods proposed by other people" in the context of the fastMRI dataset, and your statement that “results can only show the superiority of the inter-modal module in the ablation study of Tables 2 & 3”? 

In our experiments, we consistently use the intra-modality module from Section 3.3 across all datasets, including fastMRI, and our results demonstrate that incorporating this module with I2M2 leads to better performance.

---
Thanks for the author's response, I still have a few questions here:

1. For the first question, the author may have misunderstood my question. My question is actually how do you estimate these conditional probabilities in your experiments or other real-world scenarios? I have re-checked the paper and it does not explain these specific experimental details clearly.

2. In fact, designing an intra-modal module and an inter-modal module to jointly solve multi-modal problems has been proposed in many fields, and I have simply listed some literature for the author's reference:


```
[1] Verma, Sunny, et al. "Deep-HOSeq: Deep higher order sequence fusion for multimodal sentiment analysis." 2020 IEEE international conference on data mining (ICDM). IEEE, 2020.

[2] Nagrani, Arsha, et al. "Attention bottlenecks for multimodal fusion." Advances in neural information processing systems 34 (2021): 14200-14213.

[3] Huang, Po-Yao, et al. "Improving what cross-modal retrieval models learn through object-oriented inter-and intra-modal attention networks." Proceedings of the 2019 on International Conference on Multimedia Retrieval. 2019.

[4] Chen, Ruihan, et al. "DPHANet: Discriminative Parallel and Hierarchical Attention Network for Natural Language Video Localization." IEEE Transactions on Multimedia (2024).

[5] Liang, Meiyu, et al. "Self-Supervised Multi-Modal Knowledge Graph Contrastive Hashing for Cross-Modal Search." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 12. 2024.

[6] Wu, Yue, et al. "Self-supervised intra-modal and cross-modal contrastive learning for point cloud understanding." IEEE Transactions on Multimedia 26 (2023): 1626-1638.
```


3. I have noticed the experimental results in the FastMRI dataset, But this experiment is in comparison to the intra-modality methods proposed by other people. As for the proposed intra-modal module in this paper, I have to mention that the results can only show the superiority of the inter-modal module in the ablation study of Tables 2 & 3.

4. Thanks for your reply, can I understand that your method can effectively measure the relationship between different modes and the dependence between modes? However, this property has not been reflected in subsequent experiments, and we hope to see some more valuable experimental results, rather than just the improvement of algorithm performance.