CMoB: Modality Valuation via Causal Effect for Balanced Multimodal Learning
Authors: Jun Wang
Fuyuan CAO
ZhixinXue
Xingwang Zhao
Jiye Liang
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Learning
Causal Effect
Modality Valuation
Primary area: deep_learning
TL;DR: We propose CMoB, a novel causal-aware method that quantify dynamic variations of sample-level modality contributions for balanced multimodal learning.

Abstract: Existing early and late fusion frameworks in multimodal learning are confronted with the fundamental challenge of modality imbalance, wherein disparities in representational capacities induce inter-modal competition during training. Current research methodologies primarily rely on modality-level contribution assessments to measure gaps in representational capabilities and enhance poorly learned modalities, overlooking the dynamic variations of modality contributions across individual samples. To address this, we propose a Causal-aware Modality valuation approach for Balanced multimodal learning (CMoB). We define a benefit function based on Shannon's theory of informational uncertainty to evaluate the changes in the importance of samples across different stages of multimodal training. Inspired by human cognitive science, we propose a causal-aware modality contribution quantification method from a causal perspective to capture fine-grained changes in modality contribution degrees within samples. In the iterative training of multimodal learning, we develop targeted modal enhancement strategies that dynamically select and optimize modalities based on real-time evaluation of their contribution variations across training samples. Our method enhances the discriminative ability of key modalities and the learning capacity of weak modalities while achieving fine-grained balance in multimodal learning. Extensive experiments on benchmark multimodal datasets and multimodal frameworks demonstrate the superiority of our CMoB approach for balanced multimodal learning.

CMoB: Modality Valuation via Causal Effect for Balanced Multimodal Learning
Jun Wang', Fuyuan 2 Xue', Xingwang Zhao', Jiye Liang'
School of Computer and Information Technology, Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry Education, Shanxi University, Taiyuan, China
Shanxi Taihang Laboratory. Taiyuan, China waag85332ggmi11 com, cfy@sxu. edu. en, xuezhixin@sxu, edu, en, zhaoxw@sxu edu. cn, ljy@sxu. edu. cn
Abstract
Existing early and late fusion frameworks multimodal learning are confronted with the fundamental challenge of modality imbalance, wherein disparities in rep- resentational capacities induce inter- anter-modal competition during training. Current research methodologies primarily rely modality-leve contribution assessments to measure gaps in representational capabilities and enhance poorly learned modali- ties, overlooking the dynamic variations of modality contributions across individual samples. To address this, we propose Causal-awan Modality valuation approach for Balanced multimodal learning (CMoB). We define a benefit function based on Shannon's 's theory of informational uncertainty to evaluate the changes in the impor- tance of samples across different stages of multimodal training. Inspired by human cognitive science, we propose a causal-aware modality contribution quantification method from a causal perspective to capture fine-grained changes modality con- tribution degrees within samples. In the iterative training of multimodal learning, we develop targeted modal enhancement strategies that dynamically select and opti- mize modalities based on real-time evaluation of their contribution variations across training samples. Our method enhances the discriminative ability of key modalities and the learning capacity of weak modalities while achieving fine-graines balance in multimodal learning. Extensive experiments on benchmark multimodal datasets and multimodal frameworks demonstrate the superiority of our CMoB approach for balanced multimodal learning.
Introduction
Humans construct multi-dimennionn perception through multiple sensory modalities like vision, touch, hearing, and smell, processing information hierarchically to understand the real world 2 This biological cognitive mechanism has inspired the development of multimodal learning paradigms in the field of machine learning. These paradigms involve constructing collaborative learning frameworks across heterogeneous modal representation spaces (e.g., text, images, audio, and video) to simulate the cognitive process of human cross-modal information fusion 4 From a cognitive neuroscience perspective (5]
6 Z this paradigm aligns with the neural mechanisms of the human brain's multisensory integration, where different sensory cortices enable collaborative learning across heterogeneous modalities through synaptic plasticity.
In recent years, although extensive research on multimodal learning has made significant progress, some studies have found that most existing multimodal models encounter the challenge of modality
"Conesponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

collapse during joint training. This is because deep neural networks tend to prioritize modalities that are easy learn while neglecting other modalities. thus failing to effectively integrate heteroge- neous cross-modal information 8
9 10 This issue prevents the model from fully leveraging the complementary information across all modalities. Consequently, this information loss results in the performance of unimodal in joint multimodal training falling far short of their intrinsic performance ceilings. Researchers define this phenomenon as multimodal imbalance learning [ [S].
The primary cause of this phenomenon stems from the greedy nature of deep neural networks: these networks tend to rely on high-quality modality that is highly relevant to the target task, thereby inhibiting the optimization of other modalities. In the joint training process of multimodal learning, this greediness makes the modality dominant in the joint training, inhibiting the learning of other modalities and generating situation of modal competition. Researchers define this phenomenon as multimodal imbalance learning 16 17
TED that should be escited about shop opening.
TED
TED
a: What do you call a doer with no eye?
que devrais me réjouir l'ouverture d'un magasin
eye
(a)
(b)
(c)
Figure 1: A TED talk with audio, video, and text modalities.
Recent research has focused on mitigating model discrepancies and enhance modality-speciff data utilization during training to address the above challenge for improved multimodal model efficacy. Several methods have been proposed to identify and improve the training of poorly learned modalities by controlling gradient variations, modality contribution and incorporating loss functions [8 19
20 Gradient-based methods for assessing modality contribution typically rely on the assumption that larger gradients indicate higher modal importance. However, gradient values are susceptible to fluctuations from model parameters and training data, leading to unstable evaluations. Even though all modalities of a sample express the same concept (i.e., share identical label information), the amount of information related to the target object or target event varies across different modalities, which leads to differences in the data quality of individual modalities. These methods are only limited to modality- analysis and fail to capture contribution differences at the sample level. In many real-world datasets, not all samples of image modal data, video modal data are more informative than text modal data. Take TED talk video as example (as shown in Figure ID: for video comprehension, the speech stream (audio information) and subtitle stream (text information) of a TED talk convey consistent core information. Viewers typically adopt intuitive, information-rich audio information as their primary source, parsing the speaker's speech stream to understand the meaning of the conveyed content. But when the target language suddenly exceeds the receiver's comprehension scope ((c.g.. during a non-native language talk), the subtitle text replaces the auditory channel and becomes the core cognitive pathway for understanding the expression. Assessing the contribution degree of modalities at the modal level fails to reflect the true nature of the data. And although current encoders are effective in extracting multimodal features, the lack of interpretability 21 22 23 of the depth model makes it difficult to observe the role of each modality in the final prediction and to adjust unimodal training accordingly.
Inspired by the multimodal cognitive mechanisms of the human nervous system, we address the above problem by analyzing the learning process of the human brain's integrated processing of heterogeneous multimodal data 7 24 25 26]. We illustrate this concretely using a TED talk video example, as shown in Figure d). When the speaker utters "no eye deer"-a homophonic phrase identical in pronunciation to "no idea" -viewesss relying solely on the audio stream may struggle to grasp the intended meaning. In such cases, the cognitive system leverages instantaneous information entropy (27) to dynamically integrate subtitle data for assisted comprehension. During this process, the original cognitive system first extracts discriminative causal features (28] from audio data and based on experience to obtain the information entropy value of this modality. It then dynamically selects text modality data to assist comprehension, thereby continuously refining the cognitive system's 's understanding of these modalities. We analyze this process and find that it has two core characteristies: (1) modality contribution valuation based on causality. By acquiring discriminative feature information and then evaluating modality contribution according to its information entropy.

---

rather than simple signal quality comparison: (2) granular adjustment of the sample level. Even if a modality is overall reliable in the temporal dimension, the cognitive system will still implement cross-modality switching when an information bottleneck occurs in a single sample, and re-optimize its cognition of the modality. Inspired by this cognitive mechanism, in this paper we revisit the process of multimodal learning at the sample level from the perspective of causal learning. Causal learning aims to explore the essential causal relationship between things and reveal the real generation mechanism inside the data. We evaluate the degree of modality contribution by measuring the causal effect between samples, distinguishing the role of each modality in the final prediction and identifying low contributing modal samples. During the model training process, enhance and optimize the low contributing modal samples to selectively improve the learning direction of the multimodal model to alleviate the modality imbalance problem and improve the performance and interpretability of multimodal learning.
We propose causal-aware modality valuation method to evaluate the sample-level modality con- tribution in multimodal training. We design an optimization strategy for modality selection at the sample-level according to the contribution degree of each modality in order to mitigate the modality imbalance problem. In our experiments, we validate the effectiveness of our method by comparison and ablation experiments
publicly available data. In summary, our main contributions are as follows: (1) We evaluate the importance of sample in multimodal learning by means of a benefit function designed by information uncertainty theory. (2) We propose methods to quantify the degree of contribution of each modality from a causal perspective and represent the contribution of modalities in terms at the sample-level. (3) We propose a modality balancing approach from data perspective to improve the performance of multimodal learning by optimizing the selection of weak modalities from the sample level in terms of modality contribution. (4) We validate the effectiveness and benefits of our approach by conducting extensive experiments on three publicly available datasets for different modalities compared existing work.
2 Related work
2.1 Imbalance multimodal learning
The development of multimodal learning has been a great success, but many recent studies have found that the unimodal capabilities are not fully exploited in multimodal learning [I] 22] 30 B1]. Due to the imbalance in modality contributions during training, some unimodal capabilities are inhibited by the dominant modality and are unable to exert their upper limits 22 g 10 11 15). In some special cases, the performance of multimodal algorithms can even be lower than the performance of unimodal algorithms. Some researchers mitigate modality imbalance by analyzing the causes of modality imbalance and by modifying the process of forward propagation of model inference. For example, OPM algorithm []4]adopts a dynamic feature discarding approach for modality during multimodal training. OGM [17 Jachieves balanced multimodal learning by suppressing the dominant modality through an adaptive gradient adjustment strategy. PMR 12 accelerates the learning of weak modalities through category prototyping. GBlending (32 and MMpareto [16]reduce gradient conflicts during training by introducing unimodal assisted learning to control the direction of gradient update. Based on the above analysis. although these methods alleviate the modality imbalance to a certain extent, their approaches measure the degree of modality contribution through the modality- level and lack interpretability. Many scholars seek to investigate this issue from a sample-level perspective. SMLS (33] employs KL. divergence to align the factual contribution distribution with the utopia contribution distribution. PDF aims to reduce reliance on low-quality modalities via Relative Calibration (RC). The shapley value-based method (8, which enhances weak modalities by calculating combinations of all possible subsets. this paper, we propose an interpretable method to quantify the degree of contribution of each modality in multimodal training. We characterize the degree of modality contribution in terms at the sample-level to mitigate the modality imbalance issue.
2.2 Causal learning
Causal learning aims to discover causal relationships from data and utilize these relationships to make interventions, predictions and decisions (35). Causal learning focuses identifying causal relationships, which is different from traditional machine learning that only discovers correlations between variables. Causality strictly distinguishes between "cause" and "effect" variables, and
3

---

plays an irreplaceable role in revealing the mechanism of occurrence and guiding intervention
38 Causal inference has contributed to the development of artificial intelligence due to its ability to eliminate the harmful bias of confounders and to discover causal relationships among multiple variables. Causal effect are a central goal of causal inference, aiming to quantify the impact of interventions
behavior 36
266
39 40
outcomes by comparing the outcomes of treatment and control groups. Specifically, causal effect reveal causal relationships between variables by analyzing differences between potential outcomes across treatment conditions [28]. The central question is to answer: for a given individual or group, how do outcomes change when a certain intervention is applied compared to not applying that intervention? In this paper, revisit the challenge of modality imbalance in multimodal joint training from a causal learning perspective.
3 Methodology
This section presents our method. First, we formulate the multimodal learning problem and its representation paradigm. Then, we propose an information-theoreiic uncertainty measure for sample significance valuation and develop a causal-aware algorithm to quantify modalities contribution at the sample level. Finally, we propose a modality rebalancing approach from data perspective to improve multimodal learning by optimizing the selection of weak modalities from the sample-level modality contribution.
3.1 Preliminary
With any loss of generality, We formalize the general formalism for expressing multimodal learning. Given dataset D = {( yf")) l {1, .N),, m E {1.... M jis finite and nonempty set for all modalities with N² data samples and M modalities. Each sample x has M modality, and is the label of can refer to class, answering, etc. H; {+} 27 . xx J. y/ {1,2, K} denotes the corresponding class label from K classes. For ease of expression, we give examples with two modalities where and refer to the video and audio modalities, respectively. is worth noting that our method is not limited to two modalities and can be extended to more modalities.
For multimodal joint training methods, we use the mainstream M modality-specific encoder to extract features from the original space. We use h(()) to define the feature extraction of the multimodal model. For any sample, the feature extraction process can be formalized as:
h(@T.. 222
where z denotes the d-dimension feature vector 2!"h E RRd, 0th denote the parameters of modality- specific encoder 9T (8mh.) :).
We obtain joint representation of modality features by fusing them through a fusion function f(.). Then, we use (.) to denotes the output function of the multimodal model, which maps vectors into R This procedure can be formally formulated as:
Z f(]
2 M. Z R., f(W, 6, 2, W b.
where D denotes the dimension of the joint feature representation, W E R Kx denotes the weight of the last layer of the forward propagation process, denotes the number of categories the task, and b is the bias term, b E R
Finally, the cross-entropy loss is used for validation and the joint objective loss for multimodal learning can be formalized as:
N' N yi log(soollmxx f))-
Leross-enrrpp
When updating the gradient for model backpropagation, Wm and the parameters of modality-specific encoder θ are updated as:
2 eemossentropy 0mh 0th 6Th -11 N'
2 Leross Leross-entropy
a(W7 O
W
WM
11 N
0F(x))
OF(x:)
00m

---

Lerros-ennrpp (F)))) Σx=1 e 4.
e 1.th +bg.
19:=
where 7 is the learning rate, i/ as the classification result of zq. The phenomenon of modality imbalance can be formalized as: when modality prediction has better outcome, its contribution W TT dominates the logits output F(),,)). This reduces the magnitude of omanss-mory the loss 0f(z,).
Tenss entrupy already becomes smaller. The model is also optimized in the direction of smaller loss. Consequently, gradients for updating weaker modalities are suppressed, leading to under-optimized representations for them.
3.2 Sample benefit valuation
In multimodal learning, the multimodal model utilizes the different modalities of all samples for learning and inference. thereby enhancing the understanding of the data. For existing tasks of multimodal learning, all modalities are assumed to be predictive of the model [30]. The benefit function we designed is also based on this assumption. We mainly address the modality imbalance phenomenon of early fusion and late fusion in multimodal learning. We take the number of input modalities M as the value of the benefit function in multimodal joint learning when the prediction of the final network learning matched with the ground truth.
In multimodal learning, the importance of samples changes as the model is trained iteratively, Due to the heterogeneity of modalities, the modality data itself also contains noise, inter-modal redundant information. In the initial stage of training, this may prevent the model from learning the high-level semantic relationship information of different modalities. In real scenarios, the data quality of different modalities of sample is usually unstable, not all samples will have higher video modality than text modality [8. In reality, multimodal models rely on different modalities for different samples, rather than depending on the same modality for all samples. Moreover, for different tasks in the same dataset, the amount of information related to different target tasks in the same sample is different, so it is difficult to accurately define the "quality" of modality in each sample.
According Shannon's theory of information uncertainty [42], "the essence of information is to eliminate uncertainty" brings us new thinking. In other words, adding more modalities to a sample is accompanied by enhancing more information, which will bring less uncertainty the multimodal model. We can approximate this relationship as follow:
Proposition 1. Given a sample has M modalities denoted as (M(M) {xx,, ,x? ... MM. assume that there any two subsets .:(*), (x(C) of z (MM), and ( (C) C (M), then for any multimodal classifier F(-), it should be guaranteed that
Conf( (xx())) < <Conf((((x!0)) S Conf( (2x((())())))))
Current encoders and classifiers have achieved significant advancements. For any reliable multimodal classifier, when new modalities added, the prediction confidence of multimodal joint learning should increase and exhibit a positive correlation with the addition of modalities.
We defined benefit function to evaluate the importance of the samples in the multimodal model learning process as follow:
{ [M). == vi, Conf(F(z(())) < (F and C S M, 0, otherwise.
B(M)
(I)
When the multimodal model's prediction is correct and the confidence after adding new modalities is higher than that before their addition, in this case the benefit of this sample is defined as the number of input modalities.
3.3 Causal-aware quantification method
In this subsection, we propose a causal-aware modality contribution quantification method from a causal perspective to capture fine-grate fine-grained changes in modality contribution within samples. We dynamically evaluate the contribution degrees of different modalities in samples by calculating the changes in causal effects due to intervention modalities.
5

---

In the previous subsection, we evaluated the benefit value of the sample to the target task and measured the importance of the sample through the benefit function. Cognitive science research has revealed unique mechanism in multimodal information processing in which the human modality choice behavior has a significant causal inference property [5] This characteristic suggests that humans do not simply integrate all available information when faced with complex multimodal environments, but selectively focus on and utilize modality-speciffe information based on judgments of causality
and causal strength.
Intervention is used straightforward and effective method to assess the existence of direct causal relationships between events [28 35, 39]. Inspired by the concept of intervention in causality, we
assess the effect of one event (cause) leading the occurrence of another event (outcome) by means of intervention, and quantify the degree of contribution of modality in the sample. For multimodal deep neural networks, we observe the causal relationship between the variable 4 of the input set T and its corresponding output H(T) through an intervention denoted as do(t; = z), as shown in Eq 2
Φ = (T) - (T)ooo) = x))) V(H(T)) - (T))
Here, (T)) denotes control group, (T") denotes the treatment group. The difference between these two refers to the effect of the intervention. @ can be regarded as the ITE(IIdividual Treatment Effect) of & through the function H..) V(-) can be any function that valuates the effect of an outcome. T" denotes the intervened outcome.
For each sample which contains M heterogeneous modalities, S(x.) {at, aa denotes the set of modalities of the samples. We want to measure the ITE of modality j in sample i. The modality of the treatment variable is denoted as x/, the control group is S(x;), and the treatment group is (Zx))az then its individual causal effect can be expressed as follows:
MMM
ITE(x)) = dH p=|= = -(H(T)) - - ((H(S()))
v(H(S(z:)) -
B(F(S(z:))) ((F((()))
= B(F(S(z)))) - ((F((())) = 2())).
where (.) is the benefit function presented in the previous subsection. F(() denotes is the output function of the multimodal model. When quantifying the contribution of the modality j in the sample i, we must not only calculate the effect of the intervention modality j on the output but also consider its own impact on the output. Thus, its contribution can be expressed as follows:
+ (>x) ITE(()))))))))))))))))>))))))I + (HH(z))
(4)
-()(((()))) ((F(S(z)) = x())) + B(F(x))
Additionally, as specified in [ when the multimodal model makes an incorrect prediction, the control group is ((F(S())))= The minimum of treatment group ((F(((x)))do(( = x)))) is 0, and ((F(x))) )) <1. Then we have 2(a²) < 1. at which point we consider this treatment is not positive and the modality j in sample is weak modality. We conducted further discussions on ITE, at Appendix A.6
3.4 Dynamic modality optimization strategy
After quantifying the modality contribution at the sample level, we obtain the contribution of different modalities for each sample, and then distinguish the weaker modalities. We propose Algorithm| [ which implements a dynamic enhancement strategy to optimize weaker modalities during training iterations. To alleviate modality imbalance problem, we design a specific optimization function 55 selectively enhance the samples of weak modalities. The enhancement of weak modalities inevitably induces augmented modality-specifff data within samples, thereby increasing the likelihood of model overfitting during training. We employ adaptive masking to mitigate model overfitting. We mask localized features of specific modalities, thereby compelling the model to leverage cross-modal contextual dependencies to achieve optimization objectives. This method strengthens inter-modal semantic coherence while preventing over-reliance partial characteristics of individual modalities.

---

Algorithm 1 CModB Algorithm
Input: Dataset D {(= 1.)li (L....,N' J, m {I,..., M (train val test), device, method.
Output: Learned parameters 0 of multimodal model.
INIT: Optimized dataset D number of modalities M, initial parameters 6 maximum iterations E, learning rate 17, frrezee train epoch F.
to E do 2: if' < then
1: for
3:
Update model parameters 0 with dataset D;
4:
else
5:
Update D=p := D;
6:
for each sample (ar" v/) in do
7:
Encoder feature extraction i(();
8:
Feature fusion f());
9:
calculate the each unimodal ITE using Eq
calculate the each unimodal contribution & (xx() using using Eq
11:
Obtain optimize status using Eq
Update dataset by the optimize status to obtain an optimized dataset D 13: end for
Update model parameters with dataset DoP
15: end if
end for
By applying masking, we can simulate real-world scenarios where data may exhibit incompleteness or partial information loss. We employ Time Masking for audio modality and Spatial Masking for video modality. This approach enhances training data diversity while mitigating the model's sensitivity to localized noise. So that the multimodal model learns deal with incomplete data during training, and then has better adaptability and robustness when facing various complex situations in the real world.
[RC(1 (x.))) (a((k)
((x)) S 1, ((x)) > 1, <= E/2,
Re(x*)
0
(5)
a-Mask(1 - (xx)))
((x²) 1, E/2.
Here, (-) is monotonically increasing function and Mask(.) is a mask function, is the number of iterations, E is the maximum iterations of the multimodal model, a a hyperparameter. We mainly focus on optimizing the weaker modalities with low contribution. We maintain the original D inputs for the other modalities during the first half of multimodal training, while applying masking these during the latter phase. We employ time masking for audio modality and spatial masking for video modality We achieve directional and targeted modality rebalancing by the above method.
4 Experiments
4.1 Experimental setup
Datasets: We select five public innlsens CREMA-D(43), Kinetic Sounds UCF-101 45 CMU-MOSEI(46), and NVGesture[47] datasets to validate our proposed method. The description of the complete dataset is provided in Appendix(A.]
Implementation Details: Details of the implementation are given in the Appendix A.2
Baselines:
We compare a range of baseline methods. These contain traditional MML approaches and
fusion methods for balanced multimodal learning, namely, feature concatenation (Concat), prediction
summation (Sum), prediction weighting (Weight) [48]. MMCosine [49]. AGM [50), OGM [17]
GBlending 32). PMR [12, MMCooperation (8) Relearning [51]. MLA [0], MMPareto [66]
4.2 Comparison with imbalanced MML baselines
We conducted comparative experiments on multiple datasets with various modalities and compared with SOTA methods. The main results for all datasets presented in Table where "CMoB" denotes
7

---

our proposed approach. In T Unimodal-1 denotes training models using only the audio modality for datasets like CREMA-D, Kinetics Sounds, and CMU-MOSEL. For UCF-100 and NVGesture, this configuration corresponds to the RGB modality. Unimodal-2 infers training exclusively on the video modality for CREMA-D, Kinetics Sounds, and CMU-MOSEL, For UCF-101 and NVGesture, it corresponds to the optical flow modality. Unimodal-3 applies to CMU-MOSEI, where models are trained using the text modality, and to NVGesture, where the depth modality is utilized. It is worth noting that Concat refers to the baseline method commonly used in multimodal learning for mitigating modality imbalance problem, which employs concatenation fusion with a single multimodal cross- entropy loss function. Based the results, the following key observations proposed CMoB approach demonstrates strong generalization ability by showing excellent consistent performance across diverse heterogeneous multimodal datasets. Compared with PMR and OGM, which are rebalancing methods only applicable to two modalities, our method can handle datasets with more modalities and achieve the best results. (2) Each of the modality rehalancing frameworks significantly outperform the baseline Concat method relying on direct feature concatenation, with consistent performance improvements observed across evaluation metries. This empirical evidence substantiates the objective existence of multimodal imbalance challenges, necessitating dynamic -ppecit parameter adjustment during multimodal joint training. (3) In the CMU-MOSEI dataset, there is a clear phenomenon that the best unimodal performance (Unimodal-3) surpasses its multimodal learning counterpart and outperforms many well perfforrfnminn rebalancing methods on other datasets. In CREMA-D and CMU-MOSEI dataset, conventional fusion methods without modality balancing exhibit only minimal performance gains when compared the performance of the important unimodal (the dominant modality in multimodal training such as Unimodal-1 in CREMA-D, Unimodal-3 in CMU-MOSHI). We will discuss and visualize this in the next subsections.
be drawn: (1) Our
Table 1: Comparison with state-of-thed the-art (SOTA) multimodal learning algorithms. Bold and underlined results
denote the best and second- best performances, respectively.
Method
CREMA-D
KeneticsSonnds
UCF-101
CMU-MOSEI
NVGesture
ACC
F1
ACC
F1
ACC
F1
ACC
F1
ACC
F1
Unimodal-1
61.17
60.63
55.06
54.96
78.60
77.49
71.09
41.7
78.22
78.33
Unimodal-2
49.56
47.81
45.31
43.76
59.90
58.19
71.03
78.63
78.65
Unimodal-3
80.58
74.57
81.54
81.83
Concat
65.5
65.07
65.63
65.28
81.8
81.21
78.99
69.40
81.33
81.47
Sum
64.97
64.72
80.21
79.42
83.05
Weight
66.53
66.41
65.33
64.89
82.65
82.19
79.94
72.31
82.42
82.57
MMCosine
67.19
67.34
67.49
67.09
82.97
82.47
80.35 38
73.67
81.52
81,55
AGM
71.59
72.11
66.62
65.88
81.7
80.89
79.86
71.89
82.78
82.82
OGM
67.76
68.02
67.04
66.95
82.07
81.3
GBlending
71.59
71.72
68
66.43
85.01
84.5
79.64
73.29
82.33
82.91
PMR
67.19
67.20
67.11
66.87
81.93
81 48
MMCooperation
75.85
76.68
68.01
68.03
85.25
84.69
79.84
72.99
82.85
83.02
Relearning
71.02
71.46
65.92
65.48
82.87
82.15
78.75
770.22
82.87
82.94
ARL
74.63
MLA
79.43
79.90
69.05
68.75
85.38
84,84
78.65
70.02
83.73
83.87
MMPareto
76.87
77.35
74.55
74.21
85.3
84.89
81.18
74.64
83.82
84.24
CMoB
79.75
79.98
72.03
71.74
86.82
86.21
81.24
74.97
84.06
84.18
#0.27%
+00.8%%
+0,22%
+50.32%
+0,27%
+0.34%
0.19%
+00.26%
#0.14%
+0.21%
4.3 Comparison in scarcely informative modality case
In existing modal rebalancing methods, the mainstream view is to regard modalities with poor prediction performance as weak modalities and conduct additional training during the unimodal balancing. In practical applications, certain modalities may contain extremely limited label-relevant information and instead contain more noise. the multimodal model is forced to learn information from these modalities under such circumstances, it will memorize more noise, thereby Icading to negative outcomes. To simulate this scenario, we follow this paper [51] and choose to carry out experiments on the CREMA-D dataset. The video modality in the CREMA-D dataset consists of standardized emotional performances in a controlled laboratory environment, with its features primarily focusing facial details of the human face. Compared to its audio modality, it inherently contains less label-relevant information. We modified its audio modal by adding extra white Gaussian noise. The experimental results are visualized shown in Figure 2 and Appendix A..4
From the visualization results, we can observe that when the available information the dataset very limited, the performance of existing methods all declines. As shown in Figure 2 the Concat
8

---

method, the obvious cluster overlap in the video modality of the concat method indicates that it may be difficult to distinguish all categories in the high-dimensional space, and the categories in the audio modality are relatively dispersed. Through direct comparison with the concat method, it can be visually observed that the modality rebalancing method improves the model's discriminability to a certain extent. This further validates the necessity of modality imbalance learning. In the second row of Figure 2 and Figure the comparison the video modality shows that under such extreme conditions, all methods have drawbacks. Their data points fail to form distinct separated clusters in the low-dimensional space. Our method shows relatively better performance in the video modality. In the audio modality, method shows that the data points of the same category are more compact, while those of different categories are more separated. This indicates that our method makes easier to distinguish these emotional categories. This proves that our method can also exhibit good robustness in such extreme scenarios.
Audio Modality t:SNE Visualization
Audio Modality t-SNE Visuelization
35
Audio
1-SNE
1.SNE
-10
-20
-30
40
-10
10
t-SNE
1-SNE
Visual Modality 1 SNE Visualization
Visual Madality E-NNE Visualization
1000
Video
D-SNE
-10
-10
-20
HNE
t-SNE
Concat
CMoB
Figure 2: Each Unimodal representation visualization by t-SNE on the processed CREMA-D dataset. The six categories are indicated in different colors.
4.4 Ablation study
To comprehensively assess the effectiveness of our proposed method, we conduct experiments to study the influence of main components. Here, CQM represents our proposed causal-aware modality contribution quantification method, and RE denotes the dynamic modality optimization strategy. We conduct ablation study on CREMA-D and KineticsSounds datasets. The results are shown in The implementation of the dynamic modality optimization method (RE) is predicated on the
Table 2 causal- modality contribution quantification method (CQM). Consequently, RE is inapplicable in the absence of CQM. From Table we can see that both CQM and RE can boost performance in multimodal learning. Moreover, by integrating CQM with RE, the performance gap between audio modality and video modality is greatly reduced. Ablation studies clearly demonstrate the critical contribution of our proposed modules to overall method performance enhancement.
4.5 Further analysis
Analysis of Modality Gap: As mentioned in the paper [22, the existence of a geometric phe- nomenon in multimodal models: Modality Gap. It can denote regions in the shared space where the embeddings different modalities are significantly separated, and a larger modality gap indicates better performance. We make further comparisons by validating the modality gap between our method and the better performing methods (which outperforms CMoB method with some data). The results shown in Figure 3 In the concat method, the data points of the audio modality over-clustered.

---

Table 2: Results of ablation study on CREMA-D and KineticsSounds datasets
Module
Dataset
ACC
CQM
RE
X
X
65.50
CREMA-D
76.42
V
79.75
X
65.63
KineticsSounds
V
70.96
V
72.03
while those of the video modality are over-dispersed, and a smaller modality gap. This indicates that the audio modality is primarily leveraged for discrimination in the Concat method. As shown in Figure and Figure compared with MLA and MMPareto, our method has larger modality gap, with more compactly clustered modality representations and fewer outliers, demonstrating its ability to learn more discriminative features and further validating stability.
(a) Concal
(b) MMPareto
(c) MLA
(d) CMoB
Figure 3: Visualizations of the modality gap on CREMA-D dataset.
Visualization:
We employ Grad-CAM 53 to visualize the key regions focused by various rebal- ancing methods(Coneat, MMPareto, MLA) during training the CREMA-D dataset. Visualization results and analysis are provided the Appendix
A.3 A.3
Conclusion
In this paper, we first analyze the limitations in existing modality rebalancing methods that neglect the dynamic variations of modality contributions at the sample level during training. Inspired by the remarkable causal inference power the human modality choice behavior in cognitive science, we propose causal-aware modality validation approach for balanced multimodal learning. We employ intervention methods to evaluate the causal effect, quantifying changes in modality contributions at the sample level during multimodal learning. The fine-grained evaluation approach enables targeted optimizations across modalities at the sample level, effectively mitigating the issue of multimodal imbalance. Comparative experimental results in multiple datasets validate the effectiveness of the proposed algorithm.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (U24A20323 and 62376145), the Key Technologies Program of Taihang Laboratory Shanxi Province (THYF- JSZX-240107000 the Science and Technology Innovation Talent Team of Shanxi Province 2002040510020116 and the Taiyuan City "Double hundred Research action" of the first batch project about "Leading the Charge with Open Competition" (2024TYJB0127).
10

---

References
[1] N. Wu, S. Jastrzebski, K. Cho, and K. J. Geras, "Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks," in Proceedings of the International Conference Machine Learning, pp. 24 043-24 055, 2022.
[2] Z. Han, C. Zhang, H. Fu, and J. T. Zhou, "Trusted multi-view classification with dynamic evidential fusion," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 2, pp. 2551-2566, 2022.
[3] Xu, X. Zhu, and D. Clifton, "Multimodal learning with transformers: A survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 10, pp. 12 111-11111 132, 2023.
[4] J. N. Acosta, G. Falcone, P. Rajpurkar, and J. Topol, "Multimodal biomedical ai," Nature Medicine, vol. 28, no. 9, pp. 1773-1784, 2022.
[5] A. Occhipinti, S. Verma, C. Angione al., "Mechanism-aware and multimodal ai: Beyond model-agnostic interpretation," Trends in Cell Biology, vol. 34, no. 2, pp. 85-89, 2024.
[6] H. Cui, A. Tejada-Lapuerta, M. Brbić, J. Saez-Rodriguez, Cristea, Goodarzi, M. Lotfollahi, F. Theis, and B. Wang, "Towards multimodal foundation models in molecular cell biology," Nature, vol. 640, no. 8059, pp. 623-633, 2025.
[7] T. Zhong, J. Yu, Y. Pan, N. Zhang, Y. Qi, and Y. Huang, "Recent advances of platinum- based anticancer complexes in combinational multimodal therapy," Advanced Healthcare Materials, vol. 12, no. 22, p. 2300253, 2023.
[8] Y. Wei, R. Feng, Z. Wang, and D. Hu, "Enhancing multimodal cooperation via sample-level modality valuation,"' Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27 338-3322 347, 2024.
[9] Y. Zhou, X. Liang, S. Zheng, H. Xuan, and T. Kumada, "Adaptive mask co-optimization for modal dependence in multimodal learning," in Processing of the International Conference Acoustics, Speech and Signal, pp. 1-5 2023.
[10] X. Zhang, J. Yoon, M. Bansal, and H. Yao, "Multimodal representation learning by alternating unimodal adaptation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27 477--227 466 2024.
[11) Y. Yang, F. Wan, Q.-Y. Jiang. and Y. Xu, "Facilitating multimodal classification via dynamically learning modality gap." Advances Neural Information Processing Systems, vol. 37, pp.
62 (08-621) 122, 2024.
[12] Y. Fan, W. Xu, H. Wang, J. Wang, and S. Guo, "Pmr: Prototypical modal rebalance for multimodal learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 229 129-200338 2023.
[13] Y. Zhou, X. Wang, H. Chen, X. Duan, and W. Zhu, "Intra-and inter-modal curriculum for multimodal learning," in Proceedings of the ACM International Conference on Multimedia, pp- 3724-3733 2023.
[14] Y. Wei, D. Hu, H. Du, and J.-R. Wen, "On-the-fi fly modulation for balanced multimodal learning." IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[15] J. Fu, J. Gao, B.-K. Bao, and C. Xu, "Multimodal inbalance-aware gradient modulation for weakly-supervised audio-visual video parsing." IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 6, pp. 4843-4856, 2023.
[16] Y. Wei and D. Hu, "Mmpareto: boosting multimodal learning with innocent unimodal assistance," in Proceedings of the International Conference on Machine Learning. 2024.
[17] X. Peng, Y. Wei, A. Deng. D. Wang. and D. Hu, "Balanced multimodal learning via on-the-fly gradient modulation," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8238 8238-8247.... 2022.
[18] Y. Sun, S. Mai, and H. Hu, "Learning to balance the learning rates between various modalities via adaptive tracking factor," IEEE Signal Processing Letters, vol. 28, pp. 1650-1654, 2021.
[19] F. Xiao, J. Lee, K. Grauman, J. Malik, and C. Feichtenhofer, "Audiovisual slowfast networks for video recognition," in Proceedings of the IEEF/CVE International Conference on Computer Vision 2020.
11

---

[20] Y. Huang, J. Lin, C. Zhou, H. Yang, and L. Huang, "Modality competition: What makes joint training of multi-modal network fail in deep ?earning provablyy, " in Proceedings of the International conference
machine learning, pp. 9226-9259 2022.
[21] Z. You, Y.-H. Tsai, W.-C. Chiu, and G. Li, "Towards interpretable deep networks for monocular depth estimation," in Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 12 879-12 888. 2021.
[22] Barceló, M. Monet, J. Pérez, and B. Subercaseaux, "Model interpretability through the lens of computational complexity," Advances in Neural Information Processing Systems, vol. 33, pp. 487-1115 498, 2020.
[23] F.-L. Fan, J. Xiong, M. Li, and G. Wang, "On interpretability of artificial neural networks: A survey," IEEE Transactions on Radiation and Plasma Medical Sciences, vol. 5, no. 6, pp. 741-760, 2021.
[24] A. Khilkevich, M. Lohse, R. Low, I. Orsolic, T. Bozic, P. Windmill, and T. D. Mrsic-Flogel, "Brain-wide dynamics linking sensation to action during decision-making. Nature, vol. 634, no. 8035, pp. 890-900, 2024.
[25] R. Rideaux, K. R. Storrs, G. Maiello, and A. E. Welchman, "How multisensory neurons solve causal inference." " Proceedings of the National Academy of Sciences, vol. 118, no. 32. p. c2 c2106235118, 18, 2021.
[26] D. Mahajan, S. Tople, and A. Sharma, "Domain generalization using causal matching," in Proceedings of the International Conference on Machine Learning, pp. 7313-7324, 2021.
[27] C. E. Shannon, "A mathematical theory of communication," The Bell System Technical Journal, vol. 27, no. 3, pp. 379-423, 1948.
[28] J. Pearl, M. Glymour, and N. P. Jewell, Causal inference in statistics: A primer. Wiley, 2016.
[29] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, "Multimodal machine learning: A survey and taxonomy." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423-443, 2018.
[30] H. Ma, Q. Zhang, C. Zhang. B. Wu, H. Fu, J. T. Zhou, and Q. Hu, "Calibrating multimodal learning," in Proceedings of the International Conference on Machine Learning, pp. 23 429- 23 450 2023.
[31] J. Hu, Gu. S. Yu, F. Yu, Z. Li, Z. You, C. Lu, and Dong, "Interpreting low-level vision mod- els with causal effect maps," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[32] W. Wang, D. Tran, and M. Feiszli, "What makes training multi modal classification networks hard?" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 12 295-122 705 2020.
[33] Y. Zhou, X. Liang, Y. Xu, and X. Lin, "Dataset-aware utopia modality contribution for imbal- anced multimodal learning." " Information Fusion, vol. 124, p. 103383, 2025.
[34] B. Cao, Y. Xia, Y. Ding, C. Zhang, and Q. Hu, "Predictive dynamic fusion," in Proceedings of the International Conference on Machine Learning. 2024.
[35] K. Ahuja, D. Mahajan, Y. Wang, and Y. Bengio, "Interventional causal representation learning." in Proceedings of the International Conference on Machine Learning, pp. 372-407, 207233,,,0033,
[36] M. Yang, F. Liu, Z. Chen, X. Shen, J. Hao, and J. Wang. "Causalvae: Disentangled representationn learning via neural structural causal models." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9593-9602 2021.
[37] S. Deshpande, K. Wang, D. Sreenivas, Z. Li, and Kuleshov, "Deep multi-modal struc- tural equations for causal effect estimation with unstructured proxies," Advances in Neural Information Processing Systems, vol. 35, pp. 10931-10944, 2022.
[38] J. Zhang, E. Eldele, F. Cao, Y. Wang. X. Li, and J. Liang, "Counterfactual contrastive learning with normalizing flows for robust treatment effect estimation," in Proceedings of the Interna- tional Conference on Machine Learning. 2025.
[39] F. Lv, J. Liang, Li, B. Zang, Liu, Z. Wang, and D. Liu, "Causality inspired representation learning for domain generalization," in Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8046-8000000 8056, 2022.
12

---

[40] J. Brehmer, P. De Haan, P. Lippe, and T. S. Cohen, "Weakly supervised causal representation learning," Advances in Neural Information Processing Systems, vol. 35, pp. 38 319-38 331, 2022.
[41] F. Cao, X. Jing, K. Yu, and J. Liang. "Fwcec: An enhanced feature weighting method via causal effect for clustering," IEEE Transactions on Knowledge and Data Engineering, vol. 37, pp. 685-697, 2025.
[42] Soni and R. Goodman, mind at play: How Claude Shannon invented the information age. Simon and Schuster, 2017.
[43] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, "Crema- d: Crowd-sourced emotional multimodal actors dataset," IEEE Trunsactions on Affective Computing, vol. 5, no. 4, pp. 377-390, 2014.
[44] R. Arandjelovic and A. Zisserman, "Look, listen and learn," in Proceedings of the IEEE International Conference on Computer Vision, pp. 609-617, 2017.
[45] K. Soomro, A. R. Zamir, and M. Shah, "Ucf101: A dataset of 101 human actions classes from videos in the wild," arXiv preprint 12 2012.
[46] A. B. Zadeh, P. Liang. Poria, E. Cambria, and L.-P. Morency, "Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph," in Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 2236-2246, 2018.
[47] N. Wu, S. Jastrzebski, K. Cho, and K. Geras, "Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks," in Proceedings of the International Conference on Machine Learning, pp. 24 (043-240) 2022.
[48] Y. Yang, K.-T. Wang, D.-C. Zhan, H. Xiong, and Y. Jiang. "Comprehensive semi-supervised multi-modal learning." " in Proceedings of the International Joint Conference on Artificial Intelligence, pp. 4092-4098 2019.
[49] R. Xu, R. Feng. S.-X. Zhang, and D. Hu, "Mmcosine: Multi-modal cosine loss towards balanced audio-visual fine-grained learning," " in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5,2023.
[50] H. Li, X. Li, P. Hu, Y. Lei, C. Li, and Y. Zhou, "Boosting multi-modal model performance with adaptive gradient modulation," Proceedings of the IÉEE/CVF International Conference on Computer Vision, pp. 22 221-222 224 2023.
[51] Y. Wei, S. Li, R. Feng, and D. Hu, "Diagnosing and re-learning for balanced multimodal learning," " in Proceedings of the European Conference on Computer Vision, pp. 71-86, 2024.
[52] V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou, "Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning," Advances in Neural Information Processing Systems, vol. 35, pp. 17 612-17 625, 2022.
[53] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-cam: Visual explanations from deep networks via gradient based localization," Proceedings of the IEEE International Conference Computer Vision, pp. 618 018-626.0 2017.
[54] Z. Lu, "A theory of multimodal learning," Advances in Neural Informaion Processing Systems, vol. 36, pp. 57 244 57 255, 2023.
[55] P. Grecov, K. Bandara, C. Bergmeir, K. Ackermann, S. Campbell, D. Scott, and D. Lubman, "Causal inference using global forecasting models for counterfactual prediction," " in Proceedings of the Pucific-Asia Conference Knowledge Discovery and Data Mining, pp. 282-294 2021.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose a causal-award modality validation approach for balanced mul- timodal learning that captures the fine-graie fine-grained changes in modality contribution degrees within samples.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We discuss the limitation of proposed method, CMoB, at Sec. 4.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
All assumptions and proofs are clearly stated in Sec. 3 of our manuscript
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The experimental settings are detailed in Sec.4.1 and Appendix A.2 All experiments can be easily reproduced with our code.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We make our code publicly available at httts:///itthee com/ perpetual1859/ including data and detail documentation.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppulic//guue http:///cc/giiee////euuubbiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental settings and hyperparameter selection details are in Appendix
A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See the results with standard deviation in Table 1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We discuss the experiments compute resources at Appendix A.2 All experi- ments were NVIDIA DGX A100.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: Yes, this paper conform the the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This paper a foundational research and has no direct negative social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
18

---

Justification: No such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All data and methods are explicitly mentioned.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
This paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer:
[NA]
Justification:
This paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
The core methodology and experiments presented in this paper do not involve the use of large language models (LLMs) important, original, non-standard compo- nent. No LLMs were employed for data processing, model design, training. The research was conducted independently of any generative or foundation model technologies; hence, an LLM declaration is not applicable.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM for what should or should not be described.
20

---

A Supplementary Materials and Experimental Details
A.1 Dataset
CREMA-D is an emotion recognition dataset with two modalities, audio and video. The dataset includes 7,442 clips annotated in six emotions. Kinetic Sounds is an action recognition dataset with two modalities, audio and video. This dataset contains 19k 10-second video clips categorized into 31 human action classes, which are selected from the Kinetics dataset. UCF-101 is an action recognition dataset with two modalities, RGB and optical fow. This dataset contains 101 categories of human actions with 9,537 samples in the training set and 3,783 samples in the test set. CMU-MOSEI is a sentiment analysis dataset with three modalities, audio, video, and text. This dataset includes more than 1,000 online speakers segmented into 23,453 utterances and is annotated with utterance-level sentiment labels. NVGesture is a gesture recognition dataset designed for human-compuler interaction research and consists of three modalities, RGB, Depth, and optical flow. This dataset with 1,050 samples in the training set and 482 samples in the test set.
A.2 The implementation details
In our experiments, we use the raw data for experiments. Following [77 12 51, the architecture and initialization setup followed an unbalanced multimodal learning study for a fair comparison. the backbone for
For the CREMA-D and the Kinetic Sounds dataset, ResNet-18 is employed processing both audio and video data and trained from scratch. For the CMU-MOSEI dataset, we employ transformer-baase Encoders used for UCF-101 are ImageNet pre-trained. In term of video and optical flow modalities, we first select 10 frames from each clip and then uniformly sample three frames as input. We adjusted the input channels of ResNet18 from three to one to fit our data format. For audio modal data, we convert to 257x299 spectrograms for CREMA-D and a 257x1004 spectrograms for KinetiesSounds. For text- image datasets, our framework employs ResNet-50 as the image encoder and BERT for text processing, where images are resized 224x224 resolution and text sequences are truncated to a maximum length of 128 characters. During training, we use the SGD optimizer with momentum (0.9) and set the learning rate at 1 x 10-3 3 All models are trained on NVIDIA DGX A100.
networks as the backbone architecture, training the model from scratch.
A.3 Visualization
We employ Grad-CAM (53) to visualize the key regions focused on by various rebalancing meth- ods(Concat, MMPareto, MLA) during inference the CREMA-D dataset. By computing gradients of target class scores with respect to the last convolutional feature map and generating pixel-level im- portance weights, Grad-CAM can precisely localize visual regions that significantly contribute the decision-making process in the target task (emotion recognition). This visualization method not only helped us verify the effective an intuitive basis for us to analyze the allocation of visual attention in multimodal interactions. The visualization results are presented in Figure 4 where the first, second, third, fourth and last columns denote the results of the tenth, thirtieth, fiftieth and eightieth last epoch, respectively. This image keyframe extracted from the video modality of the category "NEU" in the CREMA-D dataset. Based on the heat map of the video modality, the following key observations can be drawn: (1) In the emotion recognition task, audio modality has richer information than the video modality, especially in the CREMA-D dataset. So we found that the video modality provides limited information at the early stage of training and relies mainly on the audio modality to dominate the feature extraction. At the 50th epoch of training, the video modality of the Concat method starts to focus on the feature extraction of facial expressions. However, the method is optimized by uniform joint loss function, at this point, the dominant modality (audio modality) leads to a modality imbalance problem that causes the subsequent optimization of the video model to be directionally biased. (2) At different stages of model training, compared to other modal rebalancing methods our method focuses on the features of a person's face that are most relevant to the emotion recognition task, verifying the effectiveness and stability of our method in weak modality learning.
of weak modality features by different models, but also provided
21

---

Concat
MMPareto
MLA
CMur(Ours)
epoch=10
epoch=30
epoch=50
epoch=80
epoch=100
Figure 4: Visualizations of various rebalancing methods on CREMA-D dataset.
A.4
Comparison scarcely informative modality case
Modality
Modality
30
30
Audio
10
-.SNE
-10
-20
-30
t-SNE
t-SNE
t-S Visualization
Modality SNE Visualization
40
30
20
Video
I-SNE
30
10
-10
-20
-20
L-SNE
I-SNE
MMParcto
MLA
Figure 5: Each Unimodal representation visualization by t-SNE on the processed CREMA-D dataset. The six categories are indicated in different colors.
A.5 Further discussion on ITE
further question we need to consider is: in multimodal learning, does our evaluation of the individual treatment effect (ITE) possess globality?
"Globality" generally refers to the property that an effect or conclusion holds universally among different individuals, groups, environments, emphasizing its generalizability and cross-scenario applicability (54 (5). Take classification tasks as an example: our goal to train robust multimodal deep neural network that constructs generalizable decision boundary, accurately mapping input
22

---

data into predefined classes. During training, all multimodal data also operate within the unified deep neural network. After multiple iterations, the resulting discriminative model can classify heterogeneous data from diverse modalities, and it has applicability in different modality in various samples. Therefore, we calculate that the individual causal effect of input modalities and output results processed by the multimodal deep network possesses globality.
A.6 Comparison with MML baselines on large-scale datasets
We conduct an experimental on the relatively large-scale dataset VGGSound. The VGGSound datasets consist of both audio and video modalities. The VGGSound dataset, which contains 310 classes and wide range of audio events in everyday life, is relatively large dataset. includes 168,618 videos for training and validation, and 13,954 videos for testing. The experimental results show the superiority of our method. The results of the comparative experiment are shown in Table
Table 3: Comparison with different methods on VGGSound dataset.
Method
MAP
ACC
AGM
51.98%
47.11%
MLA
54.73%
51.65%
ReconBoost 53.87%
50.97%
MMPareto 54.74%
51.25%
Ours
54.98%
51.74%
To further validate the effectiveness our method under scenarios with more modalities, we follow this paper and conduct further experiments on the Caltcch101-20 dataset [8]. We compare method with the Concat method and the Shapley value method. The experimental results confirm the effectiveness of method, even when using five modalities (views), as shown in Table 4
Table Accuracy of our methods on Caltech101-20) dataset.
Num of modalities
Concat
Shapley
Ours
82.91
83.47
83.71
3
87.71
87.99
88.22
93.64
94.07
94.35
5
94.63
94.73
94.86
23


Reviews:
[{"review_id": "OmFjxSujFf", "reviewer": "Reviewer_2axZ", "summary": "The authors discuss the limitations in existing modality rebalancing methods and point out they neglect the dynamic variations of modality contributions at the sample level during training. They propose a causal-aware modality validation approach for balanced multimodal learning. An intervention method is introduced to evaluate the causal effect, quantifying changes in modality contributions at the sample level. The fine-grained evaluation approach enables targeted optimizations across modalities at the sample level, effectively mitigating the issue of multimodal imbalance. Experimental results on multiple datasets show the effectiveness of the proposed algorithm.", "questions": "The authors should compare their work with some relatively new algorithms.", "limitations": "no", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe authors propose a causal-aware modality valuation method to evaluate the sample-level modality contribution in multimodal training. \nThe authors design an optimization strategy for modality selection at the sample-level according to the contribution degree of each modality in order to mitigate the modality imbalance problem. \nThe authors validate the effectiveness of the proposed method by comparison and ablation experiments on publicly available data.\n\nWeaknesses:\n1. Some important references are missing. Some related works about sample-level modality valuation and balanced learning, such as PDF (Cao, et al., ICML 2024), SMLS (Zhou, et al., Information Fusion 2025), and ARL (Wei, et al., ICML 2025), should be discussed in the related works.\n\n2. The authors should compare one or some relatively new algorithms, such as “Zhou, et al.: Dataset-aware Utopia modality contribution for imbalanced multimodal learning. Information Fusion 2025,” “Ma, et al.: Improving Multimodal Learning Balance and Sufficiency through Data Remixing. ICML 2025,” and “Zhou, et al.: Sample-level Self-paced Learning to Tackle Multimodal Imbalance Problem. ICASSP 2025.”\n\n3. The language quality of the submission is not very good. Some presentation issues should be addressed:\n(1) For example, “We” should be “we” in Line 143.\n(2) “we define a benefit function” should be “We define a benefit function” in Line 8 of the Abstract.\n(3) “we addresses the above problem” should be “we address the above problem” in Line 69 and Line 70.\n(4) “modality-specifice” should be “modality-specific,” which is used many times in this paper, such as in Line 157, Line 161, and Line 170.\n\nIn addition to the above issues, there are some confusing descriptions, such as “modality data”, I guess it should be “modality-specific data.” I truly suggest the authors carefully check the whole paper and improve the overall presentation quality.", "was_revised": true, "final_justification": "The authors have compared the proposed method with some recent methods, validating their effectiveness. I have accordingly raised my rating to 4. But the writing quality should be further improved.", "rebuttal": "$Q1:$ Some important references are missing. The sample-level modality contribution was discussed in \"Cao, et al: Predictive Dynamic Fusion. ICML 2024\".\n\n$R1:$ Thanks for your comment. Although both our method and the PDF[1] method are sample-level modality contribution quantification, the two methods focus on different aspects. Our method(CMoB), using causal learning and intervention-based contribution estimation, offers better interpretability. CMoB focuses on identifying dominant/ weak modalities at the sample-level during training, whereas PDF focuses on assessing quality disparities among modalities. CMOB emphasizes enhancing learning for weak modalities, whereas PDF aims to reduce reliance on low-quality modalities via Relative Calibration (RC). We conduct comparative experiments with the PDF method  on the CREMA-D dataset. The experimental results show that our proposed method exhibits strong robustness, as shown in the Table1. We will discuss Cao et al (ICML 2024) in the Related Work section and include experimental comparisons in the revision.\n\n$Q2:$ The authors should compare some relatively new algorithms.\n\n$R2:$ Following your suggestion, we conduct comparative experiments with the latest modality imbalance algorithms on the CREMA-D dataset, as shown in the Table 1. To further validate our approach, we add Gaussian noise on 50% modalities and ϵ presents the noise degree. The experimental results show the superiority of our method. In our manuscript , we also compared with relatively new algorithms, such as the MMPareto, MLA, Relearning, and MMCooperation methods were proposed in 2024, and the MMCosine, PMR, and AGM methods were proposed in 2023.\n\nTable1 Comparison with the latest algorithms on the CREMA-D dataset. The evaluation metric is Accuracy (ACC).\n| Method        | ε = 0.0       | ε = 5.0       |\n|--------------|---------------|-------------|\n| Concat        | 61.56±1.37     | 52.33±3.32    |\n| LATE FUSION    | 61.81±2.13     | 49.84±3.72    |\n| PDF[1]         | 63.31±1.11     | 57.85±2.04    |\n| Shapley[2]     | 77.82         | 70.44        |\n| OPM[3]     | 68.45        | 64.25        |\n| ERL-MR[4]     | 74.91      | 69.53        |\n| Utopia[5]     | 78.67      | 68.89        |\n| Ours(CMoB)    | 79.75       | 70.72        |\n\n$Q3:$ The language quality of the submission is not very good.\n\n$R3:$  Thanks for your comment. We will comprehensively revise the manuscript to improve its clarity, grammatical accuracy, and conciseness in the next version.\n\n[1] Cao B, Xia Y, Ding Y, et al. Predictive Dynamic Fusion[C].Proceedings of the International Conference on Machine Learning. PMLR, 2024: 5608-5628.\n\n[2] Wei Y, Feng R, Wang Z, et al. Enhancing multimodal cooperation via sample-level modality valuation[C]. Proceedings of the Conference on Computer Vision and Pattern Recognition. 2024: 27338-27347.\n\n[3]Wei Y, Hu D, Du H, et al. On-the-fly modulation for balanced multimodal learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025:47(1):469–485.\n\n[4]Han W, Cai C, Guo Y, et al. Erl-mr: Harnessing the power of euler feature representations for balanced multi-modal learning[C]. Proceedings of the 32nd ACM International Conference on Multimedia. 2024: 4591-4600.\n\n[5]Zhou Y, Liang X, Xu Y, et al. Dataset-aware Utopia modality contribution for imbalanced multimodal learning[J]. Information Fusion, 2025: 103383."}, {"review_id": "IhLjowiRJc", "reviewer": "Reviewer_WgAE", "summary": "The paper introduces a causal-aware modality valuation approach (CMoB) for addressing the problem of modality imbalance in multimodal learning. This method allows for the balancing of modality contributions at the sample level, helping to improve the performance of weak modalities while mitigating modality imbalance. The paper demonstrates the effectiveness of CMoB through experiments on several multimodal datasets.", "questions": "Please refer to the above section.", "limitations": "Please refer to the above section.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1. The method focuses an important issue in multimodal learning, modality imbalance, and provides a potential solution to improve the integration of diverse modalities in complex datasets.\n2. The paper validates its approach with experiments on multiple multimodal datasets, showcasing the effectiveness of the proposed method.\n\nWeakness:\n\n1. The motivation behind the paper is not clear. The authors mention being inspired by the human nervous system but then shift to a causal learning perspective without clearly explaining the connection between these two concepts. The introduction section lacks logical coherence in presenting the proposed approach.\n2. The proposed method claims to solve modality imbalance from a causal learning perspective. However, the actual operation involves comparing the effects of including or excluding a modality, which is conceptually similar to existing methods like Shapley value. What are advantages of this approach over Shapley value or other similar techniques?\n3. The paper's writing style needs improvement, especially in the introduction, which spends excessive time discussing related work but fails to clearly explain the motivation behind the proposed method. In addition, Section 3.1 is too detailed and spends considerable space describing concepts like cross-entropy, which do not directly contribute to the core argument of the paper. This section could be more concise and focused.\n4. In Section 4.3, the authors claim to replicate the experiment from [45] on the \"scarcely informative modality\" case but using the original CREMA-D dataset. However, [45] created such a case by adding significant noise to the audio modality of the CREMA-D dataset. The original CREMA-D dataset does not inherently contain \"scarcely informative modality\" cases. \n5. The quality of the images in the paper, particularly Figures 2 and 3, is poor. The legends in these figures are difficult to read, which diminishes the clarity of the data presented.", "was_revised": true, "final_justification": "After all the authors' responses, many of my concerns are addressed. Hence, I raise my score.", "rebuttal": "$Q1:$The authors mention being inspired by the human nervous system but then shift to a causal learning perspective without clearly explaining the connection between these two concepts. \n\n$R1:$ Thanks for your comment. Through analyzing the human cognitive system's processing of multimodal data, we observe that it first extracts discriminative causal features from raw multimodal inputs via experientially-guided mechanisms. Then, by using information entropy to dynamically quantify causal contributions across modalities, this process corresponds to the principle of \"**Causal  Effect**\"in causal learning. Therefore, our framework simulates the cognitive system's \" Dynamic Evaluation of Modality Contribution \" by quantifying the causal effects at the sample level .\n\n$Q2:$What are advantages of this approach over Shapley value or other similar techniques?\n\n$R2:$Thanks for your comment. Shapley value method inevitably leads to exponentially longer training times and heavier computational burdens. Our method (CMoB) reduces this burden substantially, while providing theoretical guarantees through causal intervention principles. For example, when evaluating the contribution of the $M^3$ modality in a dataset containing three modalities {$M^1$, $M^2$,$M^3$}, the Shapley value method is necessary to calculate the combinations of four subsets {∅, $M^1$, $M^2$, $M^1$+$M^2$}. Our causal learning framework only calculates the {$M^1$+$M^2$} coalition. The peak GPU memory consumption on the CMU-MOSEI dataset was 19030MiB for CMOB method and 24384MiB for Shapley value method . Comparison experiments with the baseline of Shapley values[1] further demonstrate the superiority of our method, as shown in the table below.\n\nTable1 Comparison with Shapley value algorithms. The evaluation metric is Accuracy (ACC).\n| Method   | CREMA-D | KineticsSounds | UCF-101 | CMU-MOSEI | NVGesture |\n|---------|---------|-----------|---------|---------|----------|\n| Concat  | 65.5     | 65.63    | 81.8   | 78.99   | 81.33   |\n| Shapley value | 77.82   | 68.01    | 85.25  | 79.87   | 82.87  |\n| Ours   | 79.75   | 72.03    | 86.82  | 81.24   | 84.06    |\n\n$Q3:$The paper's writing style needs improvement, especially in the introduction, which spends excessive time discussing related work but fails to clearly explain the motivation behind the proposed method. In addition, Section 3.1 is too detailed and spends considerable space describing concepts like cross-entropy, which do not directly contribute to the core argument of the paper. This section could be more concise and focused.\n\n$R3:$Thanks for your question. To address modality imbalance problem, current methods primarily rely on gradient variations to measure modality contribution. These methods primarily rely on modality-level contribution assessments to measure gaps in representational capabilities and enhance poorly learned modalities, overlooking the dynamic variations of modality contributions across individual samples. Although Shapley value-based methods can measure modality contributions at the sample level, they cause a sharp increase in computational overhead and training time on datasets with more than two modalities. Inspired by human cognitive science, we propose a causal-aware modality contribution quantification method from a causal perspective to capture fine-grained changes in modality contribution degrees within samples. And we dynamically select and optimize modalities based on real-time changes in their contributions.\n\nIn addition, In Section 3.1, we primarily formalize the modality imbalance problem. We will simplify section 3.1 and emphasize only components essential to modality imbalance understanding.\n\n$Q4:$ In Section 4.3, the authors claim to replicate the experiment from [45] on the \"scarcely informative modality\" case but using the original CREMA-D dataset. However, [45] created such a case by adding significant noise to the audio modality of the CREMA-D dataset. The original CREMA-D dataset does not inherently contain \"scarcely informative modality\" cases.\n\n$R4:$Thanks for your comment. Here we may not have provided a clear expression. We modify the audio data of the CREMA-D dataset, adding extra white Gaussian noise to make it noisier and scarcely discriminative. In this section, we conduct comparative experiments using the **processed** CREMA-D dataset.\n\n$Q5$:The quality of the images in the paper, particularly Figures 2 and 3, is poor. The legends in these figures are difficult to read, which diminishes the clarity of the data presented.\n\n$R5:$Thanks for your comment. We acknowledge that the current presentation does not adequately convey critical details. The legends in these figures are difficult to decipher because the annotation elements were not scaled appropriately. In the next version, we will use a higher resolution (600 dpi) and adjust the font size of all annotations to 10pt+, and increase the line thickness by 50% in Figures 2 and 3.\n\n[1]Wei Y, Feng R, Wang Z, et al. Enhancing multimodal cooperation via sample-level modality valuation[C]. Proceedings of the Conference on Computer Vision and Pattern Recognition. 2024: 27338-27347."}, {"review_id": "S8vx3AYC4Z", "reviewer": "Reviewer_iCFM", "summary": "This paper proposes the CMoB framework to address the \"modal imbalance\" problem in multimodal learning: first use the entropy-inspired sample benefit function to measure the confidence gain of each sample after adding different modalities, then treat each modality as a \"process\" to calculate the individual treatment effect (ITE), and finally dynamically shield and enhance the weak modality. Experiments on five datasets show that CMoB generally outperforms or approaches the latest balancing methods in terms of accuracy/F1, and its improvement effect on weak modalities is verified by t-SNE and Grad-CAM analysis.", "questions": "* How do the benefit‐function values and the final performance figures change if temperature-scaled logits or negative log-likelihood are used instead of the default soft-max confidence?\n* For each benchmark, what is the percentage increase in wall-clock training time and peak GPU memory consumption relative to the strongest baseline?\n* Please report an ablation study that quantifies the individual gains contributed by each module of the proposed framework.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n* This work is the first to incorporate causal Individual Treatment Effect (ITE) estimation into sample-level modality valuation, enabling a fine-grained quantification of each modality’s contribution. The introduction of causal inference (ITE estimation) into the modality rebalancing task is novel and insightful. It enriches the modality contribution evaluation beyond simple gradient or attention-based metrics.\n* The shift from global, modality-level contribution estimation to per-sample analysis allows CMoB to dynamically capture fine-grained imbalance and perform more targeted optimizations.\n* Extensive experiments across multiple benchmarks substantiate the effectiveness and robustness of the proposed approach.\n\n**Weaknesses:**\n* Per-sample ablation and dynamic masking are likely to incur substantial additional training time and computational overhead.\n* The paper lacks an ablation study that disentangles the respective contributions of the benefit function, the ITE computation, and the dynamic masking mechanism.\n* It is not recommended to share code via GitHub during the review process, as it may compromise the double-blind review principle. Instead, it is advisable to use an anonymous code-sharing platform such as https://anonymous.4open.science/.\nprocess, as it may compromise the double-blind review principle. Instead, it is advisable to use an anonymous code-sharing platform such as https://anonymous.4open.science/.\n* Typo: \"modality-spcifice\" should be \"modality-specific\" in line 157.", "was_revised": true, "final_justification": "The authors’ response has satisfactorily addressed my concerns. I keep my positive score.", "rebuttal": "$Q1:$How do the final performance figures change if temperature-scaled logits or negative log-likelihood are used instead of the default soft-max confidence?\n\n$R1:$Following your suggestion, we replaced the default softmax confidence with temperature-scaled logits. We set the temperature parameter T=2 and T=0.5. The evaluation metric is Accuracy (ACC). The experimental results demonstrate that using temperature-scaled logits(T=2) is helpful for the target task. In our method, we use cross-entropy loss, which is essentially softmax + negative log-likelihood. Our method(CMoB) isn't restricted to using cross - entropy loss and softmax confidence. We mainly use them for a fair comparison with other methods.\n\n| Method                | CREMA-D | KineticsSounds |\n|---------------------|---------|-------------|\n| CMoB(softmax)            | 79.75    | 72.03    |\n| CMoB(temperature-scaled logits(T=2)) | 79.92   | 72.41   |\n| CMoB(temperature-scaled logits(T=0.5)) | 79.04   | 71.88   |\n\n$Q2:$For each benchmark, what is the percentage increase in wall-clock training time and peak GPU memory consumption relative to the strongest baseline?\n\n$R2:$Thanks for your comment. Our modality selection and dynamic masking method incur higher computational overhead. We compare training time and computational overhead with the baseline model that performs best. Among them, the MLA method is the strongest baseline in the CREMA-D dataset, and the MMPareto method is the strongest baseline in the KineticsSounds dataset. Preliminary results show that our method increases GPU memory consumption by approximately 20% to35%. We consider that this overhead is acceptable given the improvement in the accuracy of the experimental results, as shown in the table below. Compared with methods that use data processing strategies (e.g., Shapley value) to address  modality imbalance, our method achieves lower computational overhead and reduced training time.\n\n| Method    | CREMA-D             |         |       | CMU-MOSEI       |       |         |\n|-----------|---------------------|----------------|----------------|-----------------|----------------|----------------|\n|           | training time       | GPU memory | ACC    | training time   | GPU memory   | ACC   |\n| MMPareto  | 4h35min      | 10672MiB | 76.87     | 7h29min         | 15598MiB  |  81.18   |\n| MLA       | 4h14min            | 9295MiB     |  79.43 | 6h53min         | 13841MiB       |78.65|\n| Shapley value  | 5h52min            | 13514MiB |     77.82 | 9h41min         | 24384MiB       |79.87|\n| CMoB      | 5h37min            | 13014MiB  |   79.75  | 8h34min         | 19030MiB       |81.24|\n\n$Q3:$Please report an ablation study that quantifies the individual gains contributed by each module of the proposed framework.\n\n$R3:$Thanks for your comment. We conduct an ablation study to demonstrate the effectiveness of the proposed method on CREMA-D and KineticsSounds datasets, and the results are shown in the table below. Here, CQM represents our proposed a causal-aware modality contribution quantification method, and RE denotes the dynamic modality optimization Strategy. Ablation studies clearly demonstrate the critical contribution of our proposed modules to overall method performance enhancement.\n\n| Dataset  | Module    |       | ACC    |       |\n|----------|-----------|-------|--------|-------|\n|          | CQM       | RE    |        |       |\n| CREMA-D  | ×         | ×     | 65.50  |       |\n|          | √         | ×     | 76.42  |       |\n|          | √         | √     | 79.75  |       |\n| KineticsSounds | ×    | ×     | 65.63  |       |\n|          | √         | ×     | 70.96  |       |\n|          | √         | √     | 72.03  |       |"}, {"review_id": "s0HZ9yoGKN", "reviewer": "Reviewer_DU6H", "summary": "This paper introduces Causal Modality Evaluation (CMoB) to address modality imbalance in multimodal learning. CMoB uses Shannon information uncertainty theory for a gain function, combines causal learning to measure fine - grained modality contributions within samples, and applies dynamic optimization to strengthen weak modalities.", "questions": "Can the quantification of causal effects explain the synergistic/competitive relationships between modalities? For example, the complementarity of audio-video modalities in emotion recognition.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros\n1. Combining causal learning with information theory to dynamically evaluate modality contributions at the sample level breaks through the limitations of traditional modality - level analysis.\n2. The benefit function is constructed based on Shannon information theory, and the causal effect derivation is complete.\n3. Compared with more than 10 baseline methods on 5 cross-modal datasets, the results show the best performance in all cases.\n\nCons\n1. The global nature of ITE is mentioned in the paper, but the consistency under different modality combinations is not sufficiently verified.\n2. Lacking ablation experiments to verify the effectiveness of each module.\n3. The specific implementation of the Mask(·) function in Equation 5 is not explicitly described.\n4. It is recommended to add efficiency tests on large-scale datasets and cross-modal number extension experiments.", "was_revised": true, "final_justification": "The authors' reply addresses most of my concerns. So I keep my initial score.", "rebuttal": "$Q1:$The global nature of ITE is mentioned in the paper, but the consistency under different modality combinations is not sufficiently verified.\n\n$R1:$Thank you for your comment. The \"global nature\" means our individual treatment effect (ITE) evaluation applies to different modality combinations, but the ITE values for each modality vary across these combinations. We conducted four different modality combinations ：{T+A}, {T+V}, {V+A}, { T+A+V } for experimental verification using three modality datasets (CMU-MOSEI). The Concat method is a baseline method.  The comparative experiments demonstrate the superiority and generalizability of our method, as shown in Table 1 .\n\nTable1 Comparison with different modality combinations on CMU-MOSEI datasets. The evaluation metric is Accuracy (ACC).\n| Dataset | Method  | T+A  | T+V  | V+A  | T+A+V |\n|---------|---------|------|------|------|-------|\n| CMU-  MOSEI  | Concat  | 0.745| 0.751| 0.656| 0.789 |\n|   CMU-  MOSEI | Ours    | 0.796| 0.773| 0.701|0.812|\n\n$Q2:$Lacking ablation experiments to verify the effectiveness of each module.\n\n$R2:$ Thanks for your comment. We conducted an ablation study to demonstrate the effectiveness of the proposed method on CREMA-D and  KineticsSounds datasets , and the results are shown in the Table 2. Here, CQM represents our proposed a causal-aware modality contribution quantification method, and RE denotes the dynamic modality optimization Strategy.  Ablation studies clearly demonstrated the critical contribution of our proposed modules to overall method performance enhancement.\n\nTable2 Ablation analysis of proposed modules.\n| Dataset  | Module    |       | ACC    |       |\n|----------|-----------|-------|--------|-------|\n|          | CQM       | RE    |        |       |\n| CREMA-D  | ×         | ×     | 65.50  |       |\n|          | √         | ×     | 76.42  |       |\n|          | √         | √     | 79.75  |       |\n| KineticsSounds | ×    | ×     | 65.63  |       |\n|          | √         | ×     | 70.96  |       |\n|          | √         | √     | 72.03  |       |\n\n$Q3:$The specific implementation of the Mask(·) function in Equation 5 is not explicitly described.\n\n$R3:$Thank you for your comment. We employed **Time Masking** for audio modality and **Spatial Masking** for video modality. \n\n$Q4:$It is recommended to add efficiency tests on large-scale datasets and cross-modal number extension experiments.\n\n$R4:$Following your suggestion, we conduct an experimental on large-scale dataset VGGSound. The VGGSound datasets consist of both audio and video modalities. The VGGSound dataset, which contains 310 classes and a wide range of audio events in everyday life, is a relatively large dataset. It includes 168,618 videos for training and validation, and 13,954 videos for testing.The experimental results show the superiority of our method. The results of the comparative experiment are shown in Table 3.\n\n\nTable3 Comparison with different methods on VGGSound dataset.\n| Method      | MAP     | ACC     |\n|-------------|---------|---------|\n| AGM         | 51.98%  | 47.11%  |\n| MLA         | 54.73%  | 51.65%  |\n| ReconBoost  | 53.87%  | 50.97%  |\n| MMPareto    | 54.74%  | 51.25%  |\n| Ours        | 54.98%  | 51.74%  |\n\nIn the manuscript we have already submitted, we have conducted experiments on more-than-two modalities datasets CMU-MOSEI and NVGesture. To further validate the effectiveness of our method under scenarios with more modalities, we follow this paper and conduct further experiments on the Caltech101-20 dataset [1]. We compare our method with the Concat method and the Shapley  value method. The experimental results demonstrate the effectiveness of our method, as shown in Table 4.\n\nTable4 Accuracy of our methods on Caltech101-20 dataset.\n| Num of modalities | Concat | Shapley | Ours   |\n|-------------------|--------|---------|--------|\n| 2                 | 82.91  | 83.47   | 83.71  |\n| 3                 | 87.71  | 87.99   | 88.22  |\n| 4                 | 93.64  | 94.07   | 94.35  |\n| 5                 | 94.63  | 94.73   | 94.86  |\n\n$Q5:$Can the quantification of causal effects explain the synergistic/competitive relationships between modalities? For example, the complementarity of audio-video modalities in emotion recognition.\n\n$R5:$Thank you for your question. We can indeed explain synergistic/ competitive relationships between modalities by quantifying causal effects of each modality. We treat each modality as a treatment variable and the emotion label as the outcome. Using our defined benefit function B(M), we compute the causal effect of each modality. For example, when processing with audio–video modalities in emotion recognition:\n\n• **Synergistic**: During iterative training, when a single modality cannot predict the emotion accurately, but adding the other modality achieves correct prediction, we have $ ITE(audio/video) $=$\n B\\bigl(\\hat{F}\\bigl(S(x)\\bigr)\\bigr) - B\\Bigl(\\hat{F}\\bigl(S(x) \\mid do(t = x^{(video/audio)})\\bigr)\\Bigr)$ = $B(multimodal) - B(video/audio)$ = $2$, indicating a synergistic relationship where audio and video are complementary.\n\n• **Competitive**: During iterative training, if the audio modality alone successfully predicts the emotion label, yet adding the video modality yields an accurate prediction with lower confidence than that achieved using the audio modality alone. At this point, $ITE(video)$=$\n B\\bigl(\\hat{F}\\bigl(S(x)\\bigr)\\bigr) - B\\Bigl(\\hat{F}\\bigl(S(x) \\mid do(t = x^{(audio)})\\bigr)\\Bigr)\n$\n=$-1$, and the relationship between the two modalities is competitive.\n\nWhen the causal effect of a modality is less than 1, it negatively impacts emotion prediction; if it is greater than 1, the impact is positive.\n\n[1]Wei Y, Feng R, Wang Z, et al. Enhancing multimodal cooperation via sample-level modality valuation[C]. Proceedings of the Conference on Computer Vision and Pattern Recognition. 2024: 27338-27347."}]

Author rebuttal:

Meta-review: The paper tackles the problem of modality imbalance in multimodal learning paradigm via causality-aware modality contribution quantification to capture more granular variations in modality contribution degrees in samples.
Some notable positive aspects mentioned by the reviewer are:
- the paper addresses an important issue in multimodal learning with causality perspective 
- first work to introduce causal individual treatment effect
- Extensive experimental results show effectiveness and robustness

Some concerns identified by the reviewers were:
- how different is to Shapley value techniques
- the motivation is not clear
- substantial cost due to per sample ablation
- improvements in writing at some places
- missing references to some relevant related works

After the post-rebuttal discussion period, the reviewers acknowledged that their major concerns have been resolved. For example, the comparison with latest methods, computational cost overhead compared to other aspects, cognitive science aspect, and new experimental results on accuracy comparison have been resolved. Therefore, the AC decides to recommend the acceptance of the paper and recommend authors to incorporate important reviewer's comments in the final version.
Decision: Accept (poster)