Multimodal Negative Learning
Authors: Baoquan Gong
Xiyuan Gao
Pengfei Zhu
Qinghua Hu
Bing Cao
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Negative Learning
Multimodal Imbalance
Robustness Lower Bound
Unimodal Confidence Margin
Primary area: general_machine_learning

Abstract: Multimodal learning systems often encounter challenges related to modality imbalance, where a dominant modality may overshadow others, thereby hindering the learning of weak modalities. Conventional approaches often force weak modalities to align with dominant ones in "Learning to be (the same)" (Positive Learning), which risks suppressing the unique information inherent in the weak modalities. To address this challenge, we offer a new learning paradigm: "Learning Not to be" (Negative Learning). Instead of enhancing weak modalities’ target-class predictions, the dominant modalities dynamically guide the weak modality to suppress non-target classes. This stabilizes the decision space and preserves modality-specific information, allowing weak modalities to preserve unique information without being over-aligned. We proceed to reveal the multimodal learning from a robustness perspective and theoretically derive the Multimodal Negative Learning (MNL) framework, which introduces a dynamic guidance mechanism tailored for negative learning. Our method provably tightens the robustness lower bound of multimodal learning by increasing the Unimodal Confidence Margin (UCoM) and reduces the empirical error of weak modalities, particularly under noisy and imbalanced scenarios. Extensive experiments across multiple benchmarks demonstrate the effectiveness and generalizability of our approach against the competing methods. The code will be available at: https://github.com/BaoquanGong/Multimodal-Negative-Learning.git

Multimodal Negative Learning
Baoquan Gong" Xiyuan Gao"
Pengfei Zhu Qinghua Hu Bing Cao School of Artificial Intelligence, Tianjin University, Tiajjin, China
Haihe Lab of ITAI {gongbaoquan, gaoxiyuan, zhupengfei, huqinghua, juu cn
Abstract
Multimodal learning systems often encounter challenges related to modality im- balance, where dominant modality may overshadow others, thereby hinder- ing the learning of weak modalities. Conventional approaches often force weak modalities align with dominant ones in "Learning to be (the same)" (Posi- tive Learning). which risks suppressing the unique information inherent the weak modalities. To address this challenge, offer "Learning Not to be" (Negative Learning). Instead of enhancing weak modal- ities' target-class predictions, the dominant modalities dynamically guide the weak modality to suppress non-target classes. This stabilizes the decision space and preserves modality-specie
learning paradigm:
information, allowing weak modalities pre- serve unique information without being over-aligned. We proceed to reveal the multimodal learning from a robustness perspective and theoretically derive the Multimodal Negative Learning (MNL) framework, which introduces a dynamic guidance mechanism tailored for negative learning. Our method provably tightens the robustness lower bound of multimodal learning by increasing the Unimodal Confidence Margin (UCoM) and reduces the empirical error weak modali- ties, particularly under noisy and imbalanced scenarios. Extensive experiments across multiple benchmarks demonstrate the effectiveness and generalizability our approach against the competing methods. The code is available at https: git
//github. com/
Introduction
Multimodal learning has become a cornerstone in many real-world applications, such as autonomous perception [1. medical diagnosis 22, and human-computer interaction ( By integrating infor- mation from multiple sources, such as vision, audio, and text, multimodal systems aim to improve performance and generalization. However, multimodal data often exhibit significant imbalance be- tween modalities due to noise, lack of information, or sensor heterogeneity 4. Unimodal prediction accuracy is widely used detect modality imbalance, which is simple and effective metric [5] 6 7), but it is inherently sensitive to noise and perturbations []. This vulnerability often leads to fragile performance in practice, limiting the reliability of such definitions in real-world deployments.
Extensive prior studies, especially those based on late fusion, also known as decision-level fusion strategies, have attempted to alleviate modality imbalance. Common approaches include enhancing the predictive performance of weak modalities through aggregating independently trained modality- weighting [5], adaptive ensembling [II], knowledge distillation [12] [3] from dominant modalities. While effective in certain scenarios, these methods often implicitly aim to align weak modalities with dominant ones in terms of prediction accuracy. This
specific classifiers [9] [0], confidence-bassee
"qqual contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

dominant-modallte prediction
wealemodalitt prediction
weak-modullity prediction
late fasion
hgggy
tagget
clars
forcefully allon accurccy for torget desss
weak-modallty
and
common specfic
onger
confusable closes
Doo aliqumentt legoo print
newtrol
Instable
Vite Erec Pate
dominant modality
Agppy
sai
stabilize deciion space
onger
suppresi
newrool
fear
Zon corgtorget
deses
robuntness
common specific
weak modality
(a)
(b)
Figure I: (a) Illustration of conventional (top) vs. our (bottom) strategies: instead of improving target prediction of weak modalities, we stabilize their decision space by suppressing non-target classes, which can preserve the moddlity-sspeciffe information and enhance robustness. (b) Empirical evidence: forced alignment can degrade weak modality (Video) predictions on certain samples, highlighting the risk of losing modality diversity. Where KL. Guidance stands for the forced alignment method.
wver-alinnment may lead to several issues: (i) suppressing the distinct and complementary information encoded in weak modalities; (ii) risking error propagation, as dominant modality errors are amplified when weak modalities follow them blindly [14 15 More importantly, we conduct statistical analysis illustrated Fig. (b), showing that traditional alignment- based strategies may even harm weak modalities. Specifically, we observe that samples originally predicted correctly by weak modalities (c.g., Video) but incorrectly by dominant modalities (c.g., Audio) become misclassified after being guided by fixed unidirectional KL guidance, as training progresses and eventually leads to what we term an over- over-alignnng alignment collapse point, where weak modalities lose their original predictive advantages due to excessive conformity [16].
In this paper, we propose novel perspective: rather than requiring weak modalities to select the correct class, we teach them to eliminate implausible ones, forming negative learning process. This paradigm shift is motivated by the intuition that it is often easier to rule out wrong answers than to select the right one [1 18] especially when data quality is limited. As shown in Fig. (a), unlike conventional methods that force weak modalities to identify the target class, they are often highly sensitive to noise and perturbations. Our approach enhances robustness by stabilizing the decision space, making the fusion process more resilient to uncertainty. Decision space instability refers to a model's 's high sensitivity to input perturbations near the decision boundary, where even minor changes may cause large shifts in prediction. By mitigating this instability, our method not only improves robustness but also preserves modality-spedn specific information.
By allowing dominant modalities to guide weak ones through negative learning, we gain two major advantages: (1) we stabilize the decision space, thereby improving the robustness and consistency of the final prediction, making it less sensitive to noise and better at resisting perturbations; (2) we reduce the performance gap between modalities, effectively mitigating modality imbalance. This climination- based view of weak modalities transforms them from noisy distractions into active agents of uncertainty reduction, which is especially useful in safety-critical or imbalanced scenarios.
To support this view, we introduce a learning not to be strategy for multimodal learning that shifts the focus from only improving weak modality predictions to reducing uncertainty over non-target classes. We establish a theoretical guarantee on the robustness lower bound of decision-level fusion, showing that this uncertainty suppression leads to more reliable performance. Furthermore, we demonstrate that the empirical error of weak modalities can be significantly reduced under this strategy, especially in noisy or imbalanced scenarios. In other words, our method not only enhances multimodal cooperation robustness under perturbations, but also narrows the performance gap between modalities. We summarize main contributions follows:
We provide an intuitive and rigorous multimodal learning paradigm from the perspective of robustness. Under the theoretical analysis, we propose new negative learning paradigm, stabilizing the decision space by instructing non-target classes of the weak modalities to learn from the dominant one.

---

Building on our theoretical finding, we derive new Multimodal Negative Learning (MNL) framework based on the Unimodal Confidence Margin (UCoM), This offers theoretical guarantees to tighten the robustness lower bound from multimodal learning, effectively mitigating modality imbalance and boosting robustness.
Our model flexible and compatible with a wide range of late fusion methods without introducing additional inference overhead. Extensive experiments confirm its practical effectiveness and generalizability in challenging multimodal scenarios.
2 Related Work
2.1 Imbalanced Multimodal Learning
Modality imbalance is a common challenge in multimodal learning, where different modalities vary in quality, completeness, and reliability. This issue is present across all fusion strategies: early fusion [19 20). intermediate fusion 66 21 (22), and late fusion [23 [23]24], 24). Among these, late fusion remains one of the most widely adopted paradigms due to its modular design, interpretability, and strong compatibility with unimodal pre- pre-trained models [25]. Due to the lack of feature-level compensation or cross-mon modsll interaction, late fusion tends to amplify the influence of strong modalities while weakening the role of lower-quality ones. Consequently, predictions from disadvantaged modalities often earry little weight or even introduce noise into the final decision [26]. To address this, a
variety of methods have been proposed to alleviate imbalance in late fusion settings. These include confidence-bent based weighting 27 28), modality dropout or gating [2],, and adaptive ensembling strategies [II). which aim to dynamically suppress or correct low-quality predictions. However, the prevailing philosophy in these works to improve the predictive accuracy of weak modalities for the target class, attempting to bring them closer to their stronger counterparts, which can easily cause modality-specifit information loss. Inspired by the intuition that "Learning not to be: ruling out wrong answers is often easier than identifying the correct one 18)", propose novel perspective: leveraging dominant modalities to assist weak modalities in identifying and suppressing non-target classes. By stabilizing the decision space of weak modalities in this way, reduce their exposure to noise and uncertainty, and enhance their utility in the final decision. This approach not only promotes alignment between weak and dominant modalities, but also preserves the distinctive information of weak modalities, enabling more diverse and robust decision fusion.
2.2 Robustness in Multimodal Cooperation
Robustness has long been a critical topic in multimodal learning due to the inherent sensitivity of cooperation to perturbations such data noise, label noise, and incomplete modality (30) 31..
These vulnerabilities stem from the heterogeneity of the modalities and their varying reliability under real-world scenarios. Existing studies address this by incorporating uncertainty modeling (21).
designing robustness- fusion strategies [5] Some works also attempt to quantify multimodal robustness through novel evaluation metrics [6]. connect it with generalization error bounds Collectively, these studies have highlighted a critical insight: the robustness of multimodal system can be bottlenecked by single weak modality [34]. revealing a strong link between modality imbalance and overall system reliability. However, most of these approaches emphasize prediction- consistency modeling, often overlooking the role of decision space instabiity--sppecially the impac of non-target categories. This leads to more complex architectures with limited generalizability across tasks with different class granularity 35 contrast, our method improves robustness from a decision perspective: by identifying and suppressing non-target categories, we stabilize the effective decision space and enhance resistance to noisy or conflicting predictions. This contributes to a tighter theoretical robustness lower bound for decision-level fusion, providing both practical generalization and theoretical justification.
32 33
3 Method
In this section, we first clarify the basic setup of the multimodal late fusion system and extend the multimodal robustness lower bound to the late fusion framework. We then introduce Multimodal Negative Learning (MNL), which reduces uncertainty in non-target classes and tightens the robustness
3

---

Cruss Entrupy
min (CB)phane CBBP ,91)
Unimodal Learner
Sad Anger
Sal
Anger
for
Multimodal Negative I Learning
Audio(a)
Nemel
Detach
MNL(Pin, pi
Ingits(a)
MNL. -B- piatan logl Pigl))
Late Fusion Strategy
Entrapy
Segativ Leuringg
Matisnddl
Carn fust by po
Multimodal Robustness Lower Bound
lugits(>)
Sin) mani < un 6
Haee
Herys
Unimodal
Learner
Degal
Dagas
Forwar
Visual(v)
aon-Saree class
Jahnsw Sheldll IE00
Figure 2: After cross entropy optimization, each unimodal learner updates its parameters based on ground-truth supervision, overlooking non-target class optimization, especially problematic for inferior modalities, which introduce noise during fusion. With the MNL, the robust dominant modality guides the inferior one by reducing uncertainty over non-target classes, thereby enhancing multimodal robustness with a larger UCoM.
lower bound in multimodal learning by enhancing the UCoM. Finally, we present the multimodal training strategy that incorporates the MNL.
3.1 Basic Setting
Given multimodal tasks, we denote the set of modalities by M, where |M represents the cardinality of M. Without loss of generality, the training data points are denoted as Dtrain = {(z,, v4) w))a C
I (lM))} represents the input for
X' x V. where N is the sample size of Derains 24 = {x, the i-th sample across all modalities, and y/ E Y denotes the corresponding label. In the late fusion framework, we define the logit output of the unimodal model is given by f(m). Depending on the specific fusion strategy employed in late fusion, the unimodal logits are either averaged or dynamically fused. The fusion result is represented as f(x),, which is formally defined as:
IM
fxx
Σ
(m)) p(m))
where (m)) 2 0 and Σ _m (m)) = 1. Static late fusion assigns equal weights to all modalities, whereas dynamic late fusion allows the weights to vary across samples. Overall, late fusion is favored for its flexibility, robustness, and interpretability 36
MM
3.2 Robust Lower Bound for Multimodal Learning with Late Fusion
Inspired by [16] and following their definition of multimodal robustness, we extend the lower bound of multimodal robustness to the case of late fusion. Specifically, when the multimodal model f correctly classifies the sample Ti, we introduce the multimodal robustness radius for the sample .;;
R(x)) = min || =1'l2 s. y: f(x)), = f(x();
(2)
where Vk + y, f(z;), f(x)),. f(r)), denotes the logit of class & and 2 denotes the adversarially perturbed sample. Given the ground- truth label yı and its nearest competing class j. this defines the smallest perturbation, denoted x; x/. Any perturbation smaller than this, i.e., within the multimodal robustness radius, can thus be reliably defended. For simplicity, for a unimodal logit output (mm) (m))),), we introduce the Unimodal Confidence Margin (UCoM) as follows:
<(m) = (m) (a. (m))
(m) (a: ((m))
where y is the target class and j is the most probable class among the non-target classes. Unless otherwise specified, we omit the explicit subscript of the competing class for simplicity, and denote compactly as E(m)- We analyze the UCoM. larger margin indicates that the unimodal modality
more reliable in distinguishing between these two classes.

---

For the competing class j in the final fused result, if there exists perturbed sample z of z4 such that f f(z)); (i.e., critical state), then for the original sample 24. the multimodal CoM between class y and the competing class is given by:
L
IM Σ
IM
[(xily (xx)i = Σ
w (m)) S(m)
wlm)
((m)
Σ (((). (f(m) - {(m))
(4)
where DMM (m) 1 and Dim (m)) <(m) O. Furthermore, we introduce the Lipschitz
constant T(m) 16 37]. which characterizes the minimal constant that bounds the local variation of a
function, to simplify:
[5(m) - {(m)) S Tim)]l - x!"') (m"" lla
(5) where (m)) denotes the UCoM of the sample u!(r)) after being perturbed a/m"", Finally, we can provide the lower bound of multimodal robustness late fusion framework. The proof details are provided in the Appendix A
Theorem 3.1. Multimodal Robustness of the Late Fusion Multimodal System. Given an input x; and a perturbed sample xi for the target class y and the closest competing class j # y, let E(m) denote the UCoM for the m-th modality under Lipschitz constraint T(m)). Let w(m)) represent the weight assigned to the m-th modality in a late fusion scheme. The lower bound of the perturbation radius in the late fusion framework when [M] can then be described as:
E())(1)) + ((2)
R(x;)= min |2, x2112
(6)
(ww())a))² + (1))2
Corollary 3.2. Larger Unimodal Confidence Margins lead to greater robustness in multimodal systems.
3.3 Multimodal Negative Learning
In multimodal tasks, significant disparities in modality quality and information capacity are common. Weak modalities often struggle produce accurate predictions independently, especially inter- modal imbalance increases, leading to greater uncertainty in the output space. Intuitively, it is easier for such modalities suppress incorrect predictions than to identify the correct one. Thus, leveraging high-confidence predictions from the dominant modality to suppress uncertainty in the weak modality over non-target classes is natural choice. On one hand, dominant modality that is more accurate on the target class typically exhibits lower uncertainty on non-target classes, enabling it to denoise the weak modality and enhance cross-modal consistency, On the other hand, restricting the guidance to non-target classes helps prevent the weak modality from over-aligning with the dominant one, thereby preserving its complementary information rather than being overwhelmed, Further details on preserving unique information of the weaker modality are provided in Appendix B.I
However, Theorem 3.1 and Corollary 3.2 show that relying only confidence in the ground-truth class to define modality roles can be risky. Guiding the weak modality using non- non-target signals from the strong one may reduce its margin and harm robustness. To avoid this, we redefine dominant and inferior modalities to ensure both lower uncertainty and preserved robustness.
Definition 3.3. A modality exhibiting higher confidence in the target class and a larger UCoM is regarded as the Robust Dominant Modality (RDM); the others are considered the Inferior Modality (IM).
Therefore, when the robust dominant modality exhibits lower uncertainty over non-target classes and larger UCoM, guiding the inferior modality using the non-target information from the dominant enables the inferior modality to eliminate more uncertain choices and enlarge its own margin. This ultimately enhances the overall robustness of the multimodal system. Based on this intuition, we propose Multimodal Negative Learning:
M MNL(P(RDM)) p(MM) . - p(RDM)) log((((M)))
where p(m) o (1,) ("))), o(.) denotes the softmax function and y indicates 0 at the ground-
truth class and 1 for all non-target classes. Most importantly, during the optimization of MNL, we detach the predictions from the robust dominant modality. Furthermore, an analysis of the MNL from
the perspective of empirical emror reduction is provided in the Appendix A.2 A.2
5

---

3.4 Dynamic Guidance and Training Strategy
is important to note that modality dominance is not a fixed property. Instead, emerges as dynamic phenomenon that varies across samples, tasks, and training iterations. Therefore, implement dynamic guidance between modalities based on the variation in modality predictions. Specifically, when [M] =
MNL(P(RDM), pimM)
- p()) log (P()) -y p(2) log((())
p < P E(2) PO < 222) E()) < E(2)
< E())
(8)
The dynamic guidance mechanism a critical component of MNL. Its core formalized in Definition 3.3, which jointly considers both Py and UCoM to identify the RDM and the IMs. If dydnmiic guidance absent in multimodal negative learning. an incorrect modality may guide correct one, and low UCoM modality might influence a high UCoM one, potentially weakening the multimodal system's ' robustness. Moreover, the motivation behind the MNL to reduce the uncertainty of the inferior modality, thereby enabling it to better focus the correct answer. Therefore, in addition to negative learning for the non-target classes, it is also intuitive to positive learning for the target class. For the target class, the standard cross-entropy loss naturally fulfills this objective. Consequently, in our training process, the overall loss is defined as:
2 L = C=CE(ptusion ,y) + Σ CE(PO) +A MMAMM((L((P)))) p(mm .7)
(9)
where -- denotes the cross-entropy loss, plusion = o((())) is the one-hot label, which equals at the ground truth class. The hyperparameter À controls the strength of MNL. The training process is divided into two stages. During Stage 1, we employ only the C'E(-,.) loss to optimize for the target class. In Stage 2, once the performance of both modalities stabilizes, we incorporate the MNL to better exploit the guidance from the dominant modality to the inferior modality. The warm-up setting primarily serves to reduce unnecessary computational cost the early phase of MNL training. The overall pseudocode of our model is provided in the Appendix C.6
4 Experiments
4.1 Experimental Setup
Datasets. We evaluate the proposed method on a variety of multimodal classification tasks, including (1) image-text classification using the UMPC Food-101 dataset (38), which contains approximately 100,000 image-text recipe pairs collected in uncontrolled environments, covering 101 food categories with varying levels of noise in both modalities; (2) sentiment analysis using the MVSA dataset 39]. which consists of paired user- user-generated images and texts annotated with sentiment polarity; (3) scene recognition based on the NYU Depth V2 dataset [40], which includes RGB-depth image pairs primarily focused on indoor scene understanding: and (4) emotion recognition using the CREMA-D dataset 41,, multimodal -iiuull dataset in which actors express six basic emotions (happiness, sadness, anger, fear, disgust, neutrality) through spoken utterances.
Evaluation metrics. To ensure consistency with prior work [33
42 43 we evaluate average model
performance under Gaussian and Salt noise for image modality. SNR-based noise for audio modality, and blank noise for the text modality. To reduce the variance introduced by stochastic factors, conduct evaluations using five independent random seeds.
Baselines and competing methods. In experiments, integrate the proposed MNL with both static late fusion method (LATE FUSION, LF) and advanced dynamic late fusion approaches. Specifically, we compare our method against representative static fusion strategies well as state-of-
the-art dynamic fusion techniques such as DynMM 44 TMC 43 QMF [42]. and PDF [33]. In
addition, establish unimodal baselines to provide comprehensive evaluation.
Implementation details. The network is trained using the original configuration of the corresponding method, with the MNI. directly incorporated. All experiments are conducted on NVIDIA TITAN GPU using Py Torch, with default settings applied across all methods. The warm-up time of Stage and additional details are provided in the Appendix

---

Table Performance of different methods under varying noise levels across four (MVSA, FOOD101, NYU Depth V2, and CREMA-D) datasets. We add noise (Gaussian noise for image-related modal- ities) on 50% modalities and e presents the noise degree. The red and blue represent the best and second-best result respectively. We used and + to illustrate the amount of increase or decrease our MNL method achieved. I.F denotes static late fusion. Full results with standard deviation are
reported in Appendix 14
MVSA
UMPC FOOD
101
NYU DEPTH V2
CREMA-D
Method
5=0.0
5.0
10.0
6=0.0
5.0
10.0
==0.0
5.0
10.0
6=0.0
5.0
10.0
Unimodal1
64.12
49.36
45.00
64.62
34.72
33.03
63.30
53.12
45.46
60.70
59.60
49.52
Unimoda12
75.61
69,50
47.41
86.46
67.38
43.88
62.65
50.95
44.13
56.23
52.47
38.17
I.F 76.88
63.46
55.16
90.69
68.49
57.99
70.03
64.37
60.55
68.04
64.25
52.39
DYNMM
79.07
67.96
59.21
92.59
74.74
59.68
65.50
54.31
46.79
63.27
62.01
51.43
TMC
74.87
66.72
60.35
89.86
73.93
61.37
70.40
59.33
50.61
63.63
62.68
57.97
QMF
78.07
73.85
61.28
92.90
76.03
62.21
69.54
64.10
60.18
66.13
64.27
50.77
PDF
79.94
74.40
63.09
93.32
76.47
62.83
71.37
65.72
62.56
67.07
64.57
53.33
LF+MNL
79.50
74.03
63.01
92.77
75.16
62.06
71.05
67.02
63.81
73.71
70.35
57.26
A
12.62
110.57
17.85
12.08
16.67
14.06
11.02
12.65
13.26
15.67
16.10
14.87
QMF+MNL
79.45
74.12
62.75
93.03
75.41
62.59
71.25
65.38
61.80
68.18
67.00
52.62
A 11.38
10.27
11.47
10.13
J0.62
10.38
11.71
11.28
11.62
12.05
12.73
11.85
PDF+MNL
80.54
74.07
63.78
93.33
76.65
63.16
71.52
67.01
63.07
69.18
66.94
55,43
A
10.60
J0.33
10.69
10.01
10.18
10.33
10.15
11.29
10.51
12.11
12.37
12.10
4.2 Results
Tables D andd present the main results across all datasets. For the MVSA and FOOD-101 datasets, Unimodall and Unimodal2 refer to the image and text modalities, respectively, the NYU Depth V2 dataset, they correspond to depth and RGB, while in the CREMA-D dataset, they denote the audio and visual modalities. Specifically, Gaussian noise is added to image-related modalities in Table [ Salt noise is added to image-relan related modalities in Table 2 blank (masking) noise is applied to text, and noise added to audio by adjusting the signal-to-noiss ratio (SNR) according to the noise level following 33 42 We standardize the noise levels across modalities
0, 5, and 10, allowing us to evaluate the robustness of multimodal models under varying degrees of noise. MNL enhances the model's robustness compared the baselines and competing methods. At multiple noise levels, MNL significantly improves the performance of the static late fusion method. In most cases, MNL. consistently outperforms both the PDF and QMF baselines.
Notably, the improvement brought by MNL is less pronounced when applied to dynamic fusion methods compared to static fusion strategy. As indicated in Theorem 3.1 and Corollary 3.2, the lower bound of multimodal robustness is jointly influenced by both the modality weights and the UCoM. Dynamic fusion strategies [33 42) generally assign higher weights to modalities that more confident in predicting the target class, while overlooking the potential synergy between modality weighting and UCoM enhancement. Specifically, MNL encourages weaker modalities increase their margins, but dynamic fusion tends to down-weight these weaker modalities, resulting a misalignment. This misalignment and may even lead to performance degradation relative to the baseline. In most cases, increasing the UCoM of the inferior modality leads higher confidence in predicting the ground truth. On the NYU Depth V2 dataset, MNL improves the performance of both static and dynamic fusion methods. However, the gains are relatively modest, likely because the dataset exhibits minimal imbalance between the two modalities. In other words, the gap between the robust dominant and inferior modalities is small. As result, the dominant modality has limited capacity to guide the inferior modality, reducing the effectiveness of the margin enhancement encouraged by MNL.
diminish the benefits of MNL in the dynamic fusion setting
Ablation Study
Table compares the performance of three guidance strategies- Prior, Confident, and Rohust-under different noise levels (e 0, 5, 10) on both dynamic PDF and static late fusion frameworks. Prior
7

---

Table 2: Performance of different methods under varying noise levels across four (MVSA, FOODI01, NYU Depth V2, and CREMA-D) datasets. We add noise (Salt noise for image-related modalities) on 50% modalities and e presents the noise degree. LF denotes static late fusion. Full results with
standard deviation are reported in Appendix I5
MVSA
UMPC FOODI01
NYU Depth V2
CREMA-D
Method
000
5.0
10.0
6=0.0
5.0
10.0
e=0.0
5.0
10.0
==0.0
5.0
10.0
Unimodall
64.12
56.72
50.71
64.62
50.75
36.83
63.30
50.99
38.56
60.70
59.60
49.52
Unimodal2
75.61
69.50
47.41
86.46
67.48
43.88
62.65
49.14
34.76
56.23
50.62
43.90
LF
76.88
67.88
55.43
90.69
77.99
58.75
70.03
62.05
$1.50
68.04
62.50
52.61
DYNMM
79.07
71.35
59.96
92.59
78.91
57.64
65.50
52.26
38.17
63.27
62.92
52.33
TMC
74.87
68.02
56.62
89.86
77.86
60.22
70.40
59.33
45.32
63.63
62.31
58.44
QMF
78.07
73.90
60.41
92.90
80.87
61.60
69.54
62.02
51.87
66.13
63.73
51.55
PDF
79.94
75.11
61.97
93.32
81.21
61,76
71.37
64.27
53.62
67.07
63.44
53.71
LF+NNI
79.50
74.68
62.31
92.77
80.87
61.41
71.05
64.88
54.08
73.71
68.06
58.41
A
12.62
16.80
16.88
12.08
12.88
12.66
11.02
12.83
12.58
15.67
15.56
15.80
QMF+MNL
79.45
75.14
64.68
93.03
81.14
62.47
71.25
63.62
53.30
68.18
64.05
52.32
A 11.38
11.24
14.27
10.13
10.27
10.87
11.71
11.60
11.43
12.05
10.32
10.77
PDF+MNL
80.54
75.76
64.93
93.33
81.52
62.95
71.52
64.32
54.08
69.18
63.77
54.30
A
10.60
10.65
12.96
10.01
10.31
11.19
10.15
10.05
10.46
12.11
10.33
10.59
uses fixed modality dominance from prior knowledge, Confident dynamically selects guidance based on each modality's 's confidence, and Robust incorporates UCoM into dynamic guidance.
Our proposed MNL corresponds to the Confident + Robust (V (VV) setting, which jointly considers modality-wise confidence and UCoM to dynamically determine the guidance direction. This dynamic guid- ance mechanism not only identifies which modality currently more trustworthy, but also detects and compensates for potential modality imbalance, en- hancing fusion stability. As shown in the results, method consistently outperforms all other vari- ants. For instance, on PDF at c 10, achieves 63.78% compared 63.24% for Confident-onj) and 61.02% for Prior-only. Similar trends are ob- served on LATE FUSION, with method reaching 63.01% %. significantly better than Prior-only (62.77%) and Confident-only (59.35%). These results demon- strate that our joint Confident + Robust guidance leads to more adaptive and balanced decision fusion, especially under noisy and imbalanced conditions where single-criterion strategies fail to generalize. In general, benefiting from the dynamic guidance mech- anism, MNL can improve the performance of most methods and achieve optimal results.
PCF CIOND
0.0
045
Table presents an ablation study comparing two guidance strategies: All-Class guidance, which aligns inferior modalities across the full label space, and our proposed Non- Target guidance, which focuses only on suppressing non-target categories. Across all per- turbation levels, Non-Target guidance consistently outperforms All-Class guidance in both the dynamic PDF framework and the static LATE FUSION base- line. For instance, under the highest noise level (e 10), Non- Mon-Ttt guidance improves accuracy from 61.56% to 63.78 % on PDF (+2.22%) and from 62.52% 63.01% on LATE FUSION (+0.49%). Even in the clean setting (e 0). it shows noticeable gains: +1.9% for PDF and +0.60% for LATE
Figure 3: Accuracy varies with Gaus- sian noise level on the CREMA-D dataset. "4" indicates models trained with the pro- posed Multimodal Negative Learning (MNL). Our method consistently improves accuracy across all noise levels.
8

---

Table 3: Performance comparison under differ- ent guidance strategies on MVSA dataset.
Table 4: Performance comparison under all-class and non-target guidance across different pertur- bation levels MVSA dataset.
Prior Confident Robust
10
LATE FUSION
76.88
63.46
55.16
All-Class Non- Non-Target
0 5
10
78.66
72.69
62.77
LATE FUSION
76.88
63.46
55.16
78.74
71.87
59.35
79.50
74.03
63.01
78.90
72,16
62.52
79.50
74.03
63.01
PDF
79.94
74.40
63.09
79.19
71.85
61.02
PDF
79.94
74.40
63.09
80.23 72.68
63.24
78.64
72,45
61.56
80.54
74.07
63.78
80.54
74.07
63.78
Logit tuteararee Clts
L22
0.3
0.2
C-NASA Licencice 2,00g
MiL-E-
Coy
0.1
HAL E..)
None
300
No Lond
(a)
(b)
Figure 4: (a) We visualize the average KL divergence between the two modality predictions and compare the performance gaps between our method and the baseline across varying noise levels. (b) and (c) show the evolution of UCoM on the CREMA-D and MVSA datasets, respectively.
FUSION. These consistent improvements verify that Non- Target guidance leads to better noise resilience and decision stability, by avoiding overfitting to uncertain target predictions and instead stabilizing the decision space through suppressing irrelevant classes. Further details regarding the settings of and the warm-up epochs can be found in Appendix D.2
4.4 Discussion
Analyzing the Sources of Performance Improvements. Fig. illustrates the performance of each modality and fusion model under increasing noise levels (e = 0 0), with and without the proposed MNL loss. We observe that adding the MNL loss (*+" curves) consistently improves accuracy across all settings. For example, under the static late fusion setting (top). Fusion+ outperforms Fusion by up to 3.2% at = 6, and maintains higher robustness as noise increases. Similarly, in the dynamic PDF setting (down), Fusion+ indicates that MNL not only improves clean performance but also significantly reduces performance degradation under perturbation. Moreover, both Audio+ and Video+ curves show that MNL benefits single-modality branches as well, with especially noticeable gains on the weak video modality. This confirms that MNL effectively guides weak modalities to suppress noisy predictions, stabilizing decision fusion and mitigating modality imbalance.
Analysis of Guidance Non-Target Classes. Fig- 4 aa) illustrates the optimization behavior of MNL. by comparing the average KI. divergence between dual-modal outputs and the corresponding performance gap under different guidance strategies. We use the average KL divergence between modality predictions to quantify modality discrepancy, and compare our Mor-Target class guidance with All-Class guidance under varying noise levels. Results show that although Non-Target guidance leads to higher KL divergence, it consistently achieves better multimodal performance across three noise levels, with gains of + +1.8%, +0.85%, and +4.64%, respectively. Notably, the weaker modality benefits the most from this strategy. This seemingly counterintuitive result highlights core strength of our method: unlike All-Class guidance, which enforces rigid alignment by pushing the weaker modality to imitate the stronger one, our Non-Target guidance selectively preserves discriminative cues in the weaker modality. This selective preservation maintains modality diversity and enhances cross- modal complementarity, ultimately improving system robustness.

---

Analysis of the Unimodal Confidence Margin. For presentation, UCoM is normalized via a softmax over the output logits. Fig. 4b) shows the UCoM of correctly classified validation samples during training for LATE FUSION and LATE FUSION+MNI. on CREMA-D. Typically, one modality quickly stabilizes while the weaker modality lags at a lower UCoM. As UCoM directly governs multimodal robustness (Theorem 3.1), this gap is critical. MNL consistently boosts the weak modality's s UCoM, reinforcing its decision boundary and improving overall robustness to perturbations. And Fig. Ce) shows the & values for all samples of each modality under different noise levels on the MVSA dataset. Notably, for weak modality (e.g., image), & is consistently higher with MNL, which ultimately leads to improved model performance under noisy conditions.
Furthermore, theoretical analysis reveals that the effectiveness of MNL depends on satisfying a robustness condition. As shown in representative case, when modality A exhibits higher predicted
probability than modality B, p(-4) > pf but lower UCoM {(A) E(B), directly using the non-target class signals from A modality to guide the B modality can actually reduce the UCoM of B. This phenomenon highlights that guidance strategies lacking pouustnsss-wwarr constraints can disrupt UCoM, ultimately degrading overall system performance and robustness.
Analysis of the extensibility of MNL. MNL. demonstrates strong scalability. In terms of the number of modalities, according to Definition 3.3, can be easily extended to multiple modalities. We conducted experiments on the CMU-MOSEI dataset (34) under varying levels of noise inter- ference, including the visual, textual, and audio modalities in T MNL consistently improves model performance across both static and dynamic fusion methods, particularly under noisy condi- tions. Moreover, MNL. can be further extended ILMs QA tasks and MLLMs VQA tasks. Specif- ically, we apply logit fusion to the answer token generated by two models, process we refer to as LATE FUSION. During optimization, MNL en- ables the more robust and accurate model to guide the weaker model eliminating incorrect answers, given that the models share the same vocabulary. We conducted experiments on the MathQA dataset (5) (QA task). For the QA task, Owen2.5-00.BB-Inttre
Table 5: Results of experiments conducted on the CMU-MOSEI dataset.
Method
e 0
5
=
10
LATE FUSION
66.42
61.71
45.80
+ MNL
67.44
63.36
55.09
PDF
66.14
63.54
42,47
MNI.
67.29
64.35
48.62
Table 6: Results of the QA task on MathQA dataset, where MI is Qwen2..5-0.B-IInttruc and M2 is Qwen2.5-1.5B.
Method
Fusion
M1
M2
LATE FUSION
50.89
42.85
49.41
+ -NNL
51.42
43.32
50.85
fine-tuned
We then compared the performance to validate the effectiveness of our approach in Table 6. Further details about the task setting and the
and Qwen2.5-1.55
[46] using LoRA 47
VQA task are provided in Appendix C.4
C.4
5 Conclusion
In this work, we address the challenge of modality imbalance and fragility in multimodal fusion by rethinking the role of weak modalities. Rather than forcing weak modalities to align with strong ones, we propose a novel Multimodal Negative Learning (MNL) framework that guides weak modalities to suppress non-target classes while preserving their unique contributions. Theoretically, we prove that this strategy tightens the robustness lower bound in decision-level fusion and reduces the empirical error of weak modalities. Our method can be flexibly integrated into existing late fusion architectures. Extensive experiments on noisy and imbalanced benchmarks demonstrate consistent improvements in both accuracy and robustness, confirming the effectiveness of our approach.
In future work, we plan to extend this framework more complex multimodal scenarios, such as lubti-label classification, open-set recognition, and temporal/sequential fusion. In addition, we are interested in developing more fine-grained uncertainty estimation methods to enhance dynamic guidance, and in exploring theoretical generalization bounds under adversarial or missing-modality settings. We believe that our perspective offers new and promising direction for building more robust, adaptive, and trustworthy multimodal systems.
10

---

Acknowledgements
This work was supported by the National Natural Science Foundation of China (Nos. 62222608. 62476198, U23B2049, 62436002), the National Natural Science Foundation of China (Grant No. 22527901) through the National Major Research Instrumentation Program, the Fundamen- tal and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (TYB2025XDXM505) the Natural Science Foundation of Tianjin (No. 25JCYBJC00950), and the CCF-Baide Open Fund. The authors thank the anonymous reviewers for their helpful remarks.
References
[1] Jamie Roche, Varuna De-Silva, and Ahmet Kondoz. A multimodal perception-driven self evolving autonomous ground vehicle. IEEE Transactions on Cybernetics, 52(9):9279-9289) 2021.
[2] Xia-an Bi, Xi Hu, Hao Wu, and Yang Wang, Multimodal data analysis of alzheimer's disease based on clustering evolutionary random forest. IEEE Journal of Biomedical and Health Informatics, 24:2973-2983, 2020.
[3] Tareq Mahmod Alzubi, Jafar Alzubi, Ashish Singh, Omar A Alzubi, and Murali Subramanian. A multimodal human-compuun interaction for smart learning system. International Journal of Comann-oomputer Interaction, 41(3):1718-177 1728, 2025.
[4] Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof Geras. Characterizing and overcoming the greedy nature of learning multi-modal deep neural networks. International Conference on Machine Learning, pages 14043-24055 PMLR, 2022.
[5] Yake Wei, Ruoxuan Feng, Zihe Wang, and Di Hu. Enhancing multimodal cooperation via sample-level modality valuation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27338 27347,, 2024.
[6] Xiyuan Gao, Bing Cao, Pengfei Zhu, Nannan Wang, and Qinghua Hu. Asymmetric reinforcing against multi-modal representation bias. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 17754-16722, 2025.
[7] Chengxiang Huang, Yake Wei, Zequn Yang, and Di Hu. Adaptive unimodal regulation for balanced multimodal information acquisition. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 25854-25863, 2025.
[8] Yake Wei, Siwei Li, Ruoxuan Feng. and Di Hu. Diagnosing and re-learning for balanced multimodal learning. In European Conference on Computer Vision, pages 71-86. Springer, 2024.
[9] Matthieu Guillaumin, Jakob Verbeek, and Cordelia Schmid. Multimodal semi-supervised learning for image classification. In 2010 IEEE Computer society conference computer vision and pattern recognition, pages 902-909. IEEE, 2010.
[10] Zirun Guo, Tao Jin, Jingyuan Chen, and Zhou Zhao Classifier-guided gradient modulation for enhanced multimodal learning. Advances in Neural Information Processing Systems, 37:133328- 133344, 2024.
[11] Michelle Livne, Jens Boldsen, Irene Mikkelsen, Jochen B Fiebach, Jan Sobesky, and Kim Mouridsen. Boosted tree model reforms multimodal magnetic resonance imaging infarct prediction acute stroke. Stroke, 49(4):912-918, 2018
[12] Mengxi Chen, Linyu Xing, Yu Wang, and Ya Zhang. Enhanced multimodal representation learning with cross-modal kd. Proceedings of the IEFF/VF Conference on Computer Vision and Pattern Recognition, pages 11766-11775. 2023.
[13] Wei Wei, Jiabin Tang, Lianghao Xia, Yangqin Jiang, and Chao Huang. Multi-modal knowledge distillation for recommendation with prompt-tuning. In The Web Conference 2024, 2024.
11

---

[14] Nuno C Garcia, Pietro Morerio, and Vittorio Murino. Modality distillation with multiple stream networks for action recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pages 103-118, 2018.
[15] Sungpil Woo, Muhammad Zubair, Sunhwan Lim, and Daeyoung Kim. Deep multimodal emotion recognition using modality aware attention network for unifying representations in neural models. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023.
[16] Zequn Yang, Yake Wei, Ce Liang, and Di Hu. Quantifying and enhancing multi-modal robustness with modality preference. In ICLR, 2024.
[17] Kieran Setiya. Knowing right from wrong. OUP Oxford, 2012.
[18] Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. NInl: Negative learning for noisy labels. In Proceedings of the IEEE/CVF international conference on computer vision, pages 101-110, 2019.
[19] Paul Pu Liang, Ziyin Liu, AmirAli Bagher Zadeh, and Louis-Philippe Philippe Morency. Multimodal language analysis with recurrent multistage fusion. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 150-161, 2018.
[20] Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis- Douis-Phllippe Morency. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7216-7223, 2019.
[21] Mingcheng Li, Dingkang Yang, Yang Liu, Shunli Wang, Jiawei Chen, Shuaibing Wang, Jinjie Wei, Yue Jiang, Qingyao Xu, Xiaolu Hou, et al. Toward robust incomplete multimodal sentiment analysis via hierarchical representation learning. Advances in Neural Information Processing Systems, 37:28515-2885 2024.
[22] Yang Yang, Fengqiang Wan, Qing- Yuan Jiang, and Yi Xu. Facilitating multimodal classification via dynamically learning modality gap. Advances in Neural Information Processing Systems, 37:62108 62122, 2024.
[23] Hedi Ben- Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2612-2620, 2017.
[24] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference, Association for computational linguistics. Meeting, volume 2019, page 6558, 2019.
(25) Hamid Reza Vaezi Joze, Amirreza Shaban, Michael I. Juzzolino, and Kazuhito Koishida. Mmtm: Multimodal transfer module for cnn fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13289-13299, 2020.
[26] Konrad Gadzicki, Razieh Khamsehashari, and Christoph Zetzsche. Early vs late fusion in multimodal convolutional neural networks. In 2020 IEEE 23rd international conference on information fusion (FUSION), pages 1-6. IEEE, 2020.
[27] Mohammad Rafiqul Alam, Mohammed Bennamoun, Roberto Togneri, and Ferdous Sohel. confidence-based late fusion framework for audio-visual biometric identification. Pattern Recognition Letters, 32:65-71, 2015.
[28) Yiqun Yao and Rada Mihalcea. Modality-specifee learning rates for effective multimodal additive late-fusion. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1824-1823 1834, 2022.
[29] Ahmed Hussen Abdelaziz, Barry-John Theobald, Paul Dixon, Reinhard Knothe, Nicholas Apostoloff, and Sachin Kajareker. Modality dropout for improved performance-driven talking faces. In Proceedings of the 2020 International Conference on Multimodal Interaction, pages
378-386, 2020.
12

---

[30] Yang Yang, De-Chuan Zhan, Xiang-Rong Sheng, and Yuan Jiang. Semi- Semi-superrvise multi learning with incomplete modalities. In IJCAI. pages 2998-3004 2018.
[31] Yang Yang, De-Chuan Zhan, Yi-Feng Wu, Zhi-Bin Liu, Hui Xiong, and Yuan Jiang. Semi- supervised multi-modal clustering and classification with incomplete modalities. IEEE Transac- lions on Knowledge and Data Engineering, 33(2):682-63) 695, 2019.
[32] Qingyang Zhang, Haitao Wu, Changqing Zhang. Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. In International conference on machine learning, pages 41753 41769. PMLR, 2023.
[33] Bing Cao, Yinan Xia, Yi Ding, Changqing Zhang, and Qinghua Hu. Predictive dynamic fusion. In Proceedings of the st International Conference on Machine Learning, pages 2024.
[34] Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslic Chen, Peter Wu, Michelle Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems, 2021(DBI)) 2021.
[35] Marco Federici, Anjan Dutta. Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In 8th International Conference on Learning Representations. OpenReview. net, 2020.
[36] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learn- ing: A survey and taxonomy. IEEE transactions on pallern analysis and machine intelligence, 41(2):423 443, 2018.
[37] Chris Finlay, Adam M Oberman, and Bilal Abbasi. Improved robustness to adversarial examples using lipschitz regularization of the loss. Openreview, 2018.
[38] Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pages 1-6. IEEE, 2015.
[39] Teng Niu, Shiai Zhu, Lei Pang, and Abdulmotaleb El Saddik. Sentiment analysis on multi-view social data. In MultiMedia Modeling: 22nd International Conference, MMM 2016, Miami, FL, USA, January 4-6, 2016, Proceedings, Part II 22, pages 15-27. Springer, 2016
[40] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pages 746-760. Springer, 2012.
[41] Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd- emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377-330, 2014.
[42] Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Jocy Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. In International conference on machine learning, pages 41753-41769 PMLR, 2023.
[43] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification with dynamic evidential fusion. IEEE transactions on paltern analysis and machine intelligence, 45(2):2551-256
2022.
[44] Zihui Xue and Radu Mareulescu. Dynamic multimodal fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2575 25844, 2023.
[45] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorse Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation based formalisms. In Proceedings of NAACL-HLT, pages 2357-2367, 2019.
[46] Qwen Team. Qwen2.5: A party of foundation models, September 2024.
13

---

[47] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 12):3, 2022.
[48] Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang. and Qingming Huang. Reconboost: Boosting can achieve modality reconcilement. In International Conference on Machine Learn- ing. pages
PMLR, 2024.
[49] Paul Pu Liang, Zihao Deng. Martin Q Ma, James Y Zou, Louis-Philippp Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36:32971-32998 2023.
[50] R. G. James, J. Ellison, and J. P. Crutchfield. dit: Python package for discrete information theory. The Journal of Open Source Software, 3(25):738, 2018.
[51] Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18177-18188 June 2022.
[52] Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Liveseu. Deep canonical correlation anal- ysis. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1247-1255, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
[53] Zixian Gao, Xun Jiang, Xing Xu, Fumin Shen, Yujie Li, and Heng Tao Shen. Embracing unimodal aleatoric uncertainty for robust multimodal fusion. Proceedings the Conference on Computer Vision and Pattern Recognition (CVPR), pages 26876-26885, June 2024.
the IEEE/CVF
[54] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[55] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255 Icce, 2009.
[56] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2019.
[57] Yake Wei and Di Hu. Mmpareto: Boosting multimodal learning with innocent unimodal assistance. In International Conference on Machine Learning, pages 2024.
PMLR,
[58] Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. CoRR, 2024.
[59] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995-5004, 2016.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main contributions and scope were reflected in the abstract and introduction (in the last paragraph of introduction, page 2).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of proposed method discussed the Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We provide short proof sketch in the main paper to offer intuition in section and include the full proof in the Appendix.
3.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experimental setup, including dataset, baseline, evaluation metric and implementation details, was provided in section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide code repository. All datasets used in this work publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
cublicc/uuddes/Coeeuubbiis cosssiiinplyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppppgguuuiaaeeeae
wtweeee//ooSSSeissss.ee.e www.ssssiicgcyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental details are presented in Section 4.1 For data splits, hyper- parameter selection, and the type of optimizer used, we follow the same settings as the baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We run the main experiments 5 times with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computer resource information for all experiments was provided in the section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips.. cwtpurrissscccppppeaeeo http:///wwrrpp.coo/iiiiicccccaaa
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We talk about the broader impacts Appendix E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not deal with this aspect of the problem.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets (datasets, code, models) used in the paper open source, and our use follows the relevant protocols.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

Appendix
A Proof
A.1 Robust Lower Bounds for Multimodal Learning with Late Fusion
In the decision-level late fusion framework, following the standard setting, let f(m) denote the logit output from modality m, where C' is the number of classes and B is the batch size. For simplicity, we illustrate the formulation using two modalities. The fusion weights satisfy w1 + w22 = with w1, O. The fused multimodal output is defined as:
-(x)= = ww (x))) ww)) (x(22)))
(10)
where x = {x(0, }. f(x) E BxC, According to the definition of UCoM, for given ground- truth class y and competing class 1s + y (the runner-up class), the UCoM is given by:
f(m) f((²) (a!!))) f(mm) (z((m))
(11)
In real-world scenarios, modality inputs are often subject to perturbations. Let the perturbed input be denoted as x' {a:11), a (2))), }. We consider a critical condition under which the perturbed multimodal margin becomes zero for some & + y.
2
f(d), f(x')
Σ (() f(m)
Σ m=l
w (m).
(x(m)
- -(mm) (z )x) =0 0 (12)
The perturbed multimodal logit output can then be expressed as:
f(x') (1) (x + (² (())
(13)
where x' = (~()) ,.2)) f(z') E R BxC, We now compute the multimodal CoM between class y and the competing class k:
f(x), f(x)x = w1)) E(1) + w(() E(2) (w()) EL(1) + wl ((()))
(14)
(E() {(1)) + n (x(2) {/2))
(15)
According to the Lipschitz continuity assumption 16 371, for modality m, there exists a constant T(m) such that:
(5(m) - 5(m)) I 7(m) [m)( z |2.
(16)
Combining Equation 15 and Equation [6] the total margin variation is upper bounded as:
f(z), f(x) I t0 Tw)la - ,00 1|2 + (0 re)) x (2)' 1|2.
(17)
Using the Cauchy-Schwarz inequality, define
= w( T()): b = == T()) u = (a, b), v= (1 - x (1) 1|,, [|x (2) (2)))
then
(u, v) < |1ul12 |1u||2 = Va² + + || - 2'112.
(18)
Therefore,
f(x), - f(x) < V (1o()) + (w2)) || 2'112.
(19)
Hence, the multimodal robustness radius is lower bounded as:
w w(( + 10 (12)
R(x) = min | x'112 >
(20)
((0(0))1))) + (wi()
22

---

Guidin Modally
Modally
Guiding Modalty
Modality
Guiding Modally
Modality
Guiding Modslly
Modality
(a)
(b)
(c)
(d)
Figure 5: During the training process, examples are provided for both the guiding and guided modalities, with the green bar representing the ground truth class. Specifically, (d) highlights the predicted probabilities for the ground truth class and the runner-up class.
A.2 Empirical Error
We incorporate Multimodal Negative Learning (MNL) into both traditional static late fusion methods and advanced dynamic fusion approaches. Within the MNL framework, we define one modality as the guiding modality (Robust Dominant Modality, RDM) and the other as the guided modality (Inferior Modality, IM). To analyze the impact of MNL on empirical error, we conduct case study based on different combinations of prediction correctness:
1. Both modalities predict correctly or both incorrectly: Since MNL operates independently of the ground truth label, it does not directly affect the empirical error in these cases, as illustrated in Fig. 5 a) and (b).
2. The guiding modality predicts correctly, while the guided modality predicts incorrectly: As shown in Fig. (Je). MNI. encourages the guided modality align with the guiding modality on non-target classes. This alignment helps steer the guided modality toward the correct prediction, thereby reducing the empirical error.
3. The guiding modality predicts incorrectly, while the guided modality predicts correctly: As illustrated in Fig. d),, from the perspective of empirical error, the guiding and guided roles assigned solely based prediction confidence, it is possible for the guiding modality to have a higher confidence score on the ground truth class (P guiding > P (uuidea) but lower UCoM (Eguiding < Equided). In such cases, the guiding modality may be making an incorrect prediction while the guided modality is actually correct. Therefore, in MNL, the assignment of guiding and guided roles not solely based prediction confidence or the probability of the ground truth class, Instead, it also considers the UCoM. As result, instances like the one shown in Fig. []d) where the guiding modality is incorrect despite high confidence, are excluded from contributing to the guidance.
Overall, integrating MNL into existing late fusion methods leads to a consistent reduction in empirical error.
B
More Analysis
B.1 Analysis of Unique Information in Modalities
In this subsection, we demonstrate the effectiveness of our approach from a mutual information perspective. Let the two modalities be represented by X1 and X2, and let y denote the ground-truth labels. Following the methodology in [48] 49], we decompose the multimodal mutual information ((K,,;;;)) into three conditional mutual information components:
(2)))
X2;3
S(X3 relevant info.
U(A))) relevant unique info, in X2 U(X2) ) relevvan unique 22
In the main paper, we analyze the relationship between the KL divergence of modality outputs and overall system performance to investigate the unique role of non-target class guidance in multimodal learning. Unlike traditional all-class guidance, the non-target strategy adopted by MNL. does not enforce strict alignment across modalities. Instead, selectively preserves discriminative features in the weaker modality. In other words, MNL's strength lies in the fact that the dominant modality is
23

---

more effective at excluding non-target classes, while the weaker modality can learn this exclusion ability without compromising own distinctive representation for target classes. This prevents the dominant modality from overwhelming the weaker one and helps maintain a healthy diversity between modalities.
Specifically, we employ the dit toolkit (50) to perform partial information decomposition, quantifying the unique information contributed by each modality. is important to note that this quantification reflects only the relative magnitudes of unique information within the multimodal system.
Fig. repoors the relative unique information of each modality under varying noise levels the CREMA-D dataset, measured as the ratio of unique information from the weaker modality to that from the stronger one. Under zero-noise conditions, methods that apply guidance across all classes tend to suppress the contribution of the weaker modality, resulting in limited benefits for fusion. In contrast, MNI., which applies guidance only non-target classes, consistently reduces the disparity between modalities and effectively preserves or enhances the unique information of the weaker modality.
10
FunionA
15.0
Lete
15.0
LOTe Fuzion MKL
Late Fuston NAL
3 dccureccy
12.5
12-
Procenten PN)
t0
2.5
2.5
0.0
2-
Prate Sovel DO
Figure 6: Results on the CREMA-D dataset under Gaussian noise (left) and Salt noise (right). denotes methods applying guidance across all classes. The lines show the relative modality-specifee unique information (Proportion), measured as the ratio of the weaker to the stronger modality. AAccuracy reflects the performance improvement brought by the MNL method over conventional all-class guidance approaches.
B.2 Extended Training for Improved Audio Modality Performance
In the main paper, based on previous methods, we compared the multimodal and unimodal perfor- mances of two strategies, Non-target Class Guidance (MNL) and All Class Guidance, after training for 100 epochs under different noise levels. The results show that at c = 0 and c = 5, the audio modality under the MNL method performs worse than that under All Class Guidance. This may be due to the dynamic guidance mechanism of MNL causing insufficient training of the audio modality. To this end, conducted two separate experiments: the other increasing the loss weight of the audio modality from 1.0 to 2.0.
extending the training to 200 epochs, and
Fig. and 7b illustrate the performance differences between the MNL method and the All-Class Guidance baseline under various levels of Gaussian noise, with extended training duration and increased loss weight for the audio modality, respectively. Notably, MNL continues to achieve superior multimodal fusion performance, yielding improvements (+0.83%, 14%,,, +6.45%) after training for 200 epochs, and (+1.88% +1.20% +5.58% when the loss weight for the audio modality increased to 2.0.
More importantly, MNL not only improves the overall multimodal performance but also consistently outperforms All-Class Guidance the audio modality alone, with gains of (+0.23%, +0.1 13%, +2.99%) in Fig. 7a and (+0.77%, +0. +0%, +5.81 ()) ) in Fig. 7b These results indicate that prolonged training or accelerating the learning of the stronger modality can be beneficial when adopting the MNL strategy.
24

---

Fuslan Audo Viocal
Fuslan Audo
Visual
Accuraos
mna
Noise love
Noise lovol5
Noki Nowel
Nokis lovel
Ncko loval
Noise lovel 10
(a) After training for 200 epochs
(b) Increase the loss weight of the audio modality
Figure 7: On the CREMA-D dataset with added Gaussian noise, compared the performance differences between the MNI. method and the approach guided across class under various noise levels.
B.3 Advantages of Late Fusion and Comparison with Latent Fusion
In multimodal learning, the advantages of late fusion lie its flexibility, robustness, and inter- pretability [36).
Flexibility. Latent fusion requires aligning and projecting heterogeneous modalities into a unified representation space, which is often challenging and prone to noise or information loss. In contrast, late fusion processes each modality independently and combines only the final predictions, significantly reducing the need for strict aligmment. As result, late fusion is more flexible and better suited for integrating diverse multimodal data types.
Robustness, In real-world scenarios, data from one modality may be missing entirely or severely degraded (e.g., due to noise). In latent fusion, such incomplete or low-quality modalities can corrupt the shared representation. leading to a significant drop perfor- mance [36 51]. In contrast, late fusion treats each modality independently and fuses only the final predictions. As result, if modality missing, its output can be simply ignored, and if one modality is noisy, it does not directly affect the others. This modular structure makes late fusion inherently more robust to missing or noisy modalities.
Interpretability. Latent fusion occurs at the intermediate stages of the model, where fused features are often highly abstract, making difficult to clearly understand the specific contribution of each modality to the final decision. In contrast, late fusion combines individual modality predictions, allowing inspection of each modality's 's output separately. This enables easier identification of which modality's prediction caused error, thereby offering superior interpretability.
We have further added comparisons with latent fusion methods the MVSA dataset. The experiments are conducted under varying levels of Gaussian noise, and all the competing methods share the same backbone. The compared methods described as follows:
we implement the CCA loss and combine with a cross-entropy loss, which performs learning using the shared latent representation.
Deep CCA: Following (52,
LFM 22 LFM integrates contrastive learning into multimodal learning to align features across modalities.
SUFA 53): SUFA aligns different modalities by minimizing the KL divergence between them. In addition, it constructs positive and negative sample pairs within each unimodal branch for contrastive learning.
25

---

Table Compared with latent fusion
Method
c
C
10
Deep CCA
76.49
63.58
53.94
LFM
77.26
66.47
57.80
SUFA
78.18
71.40
59.11
MNL
79.50
74.03
63.01
C More Details
C.1 Image-text Classification
For the image-text classification task, we conduct experiments on the MVSA and UMCP-FOODION
datasets, following the experimental settings described in 42 33 We use ResNet architecture (54) pretrained on ImageNet (55) as the backbone for the image modality, and a pretrained BERT model [56] for the text modality. The results of competing methods are reported based the implementations and settings from 42 33 All models are trained for 100 epochs with a batch size of 16, using the Adam optimizer. The learning rate is set to lc-4, with warmup proportion of 0.1.
C.2 Senses Recognition
For the sense recognition task on the NYU DEPTH V2 dataset, we compare the proposed method with several multimodal fusion approaches, following the experimental setting [42 33). We adopt the ResNet architecture [54], pretrained on ImageNet [555, as the backbone network for each modality. All models are trained for 100 epochs with batch size of 32, using the Adam optimizer. The learning rate is set to le-4, with a warmup proportion of 0.1.
C.3 Emotion Recognition
For the emotion recognition task, we conduct experiments on the CREMA-D dataset. implementing the baseline and comparison methods based on the settings in (57]. All models are reproduced under the same experimental conditions. We use ResNet-18 (54 as the backbone network, and all models are trained from scratch for the text and visual modalities. Optimizer settings, learning rates, and other hyperparameters follow those used in the respective baseline methods. Specifically, for LATE FUSION, QMF [22, PDF [3), and the variants incorporating MNL, we use the SGD optimizer with a batch size of 64, a learning rate of 0.002, and train for 100 epochs.
C.4 Multi-LLM Fusion
Recent MLLMs and LLMs [58 primarily rely feature-level fusion. To demonstrate the gen- eralizability of our approach, we extend MNL to large language models, thereby constructing a multi-LLM fusion setting. Specifically, for both QA and VQA tasks, each individual model, given a prompt, question, and the corresponding answer candidates, is required to predict only a single token representing the symbol of the chosen option. We collect the logits produced by each model (all models share the same vocabulary) and fuse them with fixed weights at the decision level. We refer to this procedure LATE FUSION, which allows MNL to be naturally adapted to the multi-LLM fusion setting. A similar strategy can also be applied to MLLMs.
For the QA task, we use the MathQA dataset (45), which contains math word problems annotated with gold answers and step-by-step solution programs. The problems span diverse domains such as arithmetic, algebra, geometry, and probability. MathQA is widely employed to evaluate and improve models' mathematical reasoning and problem-solvine capabilities.
For the VQA task, we conduct experiments on the Visual7W dataset (59] widely used benchmark for visual question answering. Each question Visual7W is annotated with gold answer and belongs to one of six categories: what, where, when, who, why, and how. This dataset is commonly used to study image understanding and multimodal reasoning. From Visual7W, we sample data 6993 examples for training and 1400 examples for inference. We fine-tune Oiee2-VL-2BB -L-2BB-nsstut and
26

---

Awen2.5-VL-3B using LoRA, and compare the performance of LATE FUSION and LATE FUSION MNL validate the effectiveness of our method. The results are summarized Table g
Table Performance comparison of VQA task.
Method
Fusion
Qwen2-
Qwen2.5-VL-3B-Inttee
LATE FUSION
84.87
83.71
84.76
LATE FUSION + MNL
85.21
84.57
84.82
C.5
Symbols Table
To avoid potential confusion, we provide table for main symbols in Table
Table 9: Summary of Symbols
Symbol
Description
M
Modality Set
[M]
Number of Modalities
R(x,)
Multimodal robustness radius of the sample 2;
The most probable class among the non-target classes Fusion logits output given sample I; Fusion logits output for class k given sample æ;
f(x.)
(sm) (m))))
Logits output from the m-th modality given modality-speciffe input Softmax function
a()
{(m)
UCoM for modality m as defined in Eq. (2), where j denotes the most probable class among the non-target classes
UCoM for modality m, its simplified form, denoted by </m) without explicitly specifying the competing class j
{(m)
((m)
UCoM for modality m after perturbation
T(m)
Lipschitz constant of modality m
(m)
Weight assigned to modality m in late fusion
p(m)
The predicted probability of modality m
pfm)
The predicted probability of the target class from modality
C.6 Pseudocode of MNL
To facilitate better understanding, we provide pseudocode in Algorithm (1).
Algorithm Multimodal Learning with MNL
Input: Training set Dirain = {(x,, H C X x y: Iteration number T; Warm-up epoch W Output: Trained multimodal model parameters 0
1: for = 1 to 7' do
2: Sample fresh mini-batch B, from Dtrain
3: Feed-forward B, to the model
4: Obtain modality-specific logits f(() and the fusion logits f(x) according to Eq.
5:
if W then
6:
Compute each modality {(m) (Eq. and identify RDM and IMs according Definition 3.3.
7:
Compute MNL loss (Eq. ? and the total loss (Eq. 9
8:
else
9:
Compute cross-entropy loss to warm up
end if
11:
Update model parameters 0
12:
end for
27

---

D Additional Results
D.I Additional Ablation Study
In the main paper, we conducted ablation studies on both the guidance scope and guidance conditions. To more intuitively demonstrate the effecttivneess of MNL, performed comprehensive ablation experiment on the aforementioned conditions, as shown in Table 13
D.2 Additional Experiments on Hyperparameters
For the hyperparameter À in Eq. | which controls the strength of MNL, our setup sums the MNL loss equally with the CE loss. To illustrate this, we report results on the MVSA dataset, where the CE weight is fixed while the MNL weight is varied from 0.2 to 2.0 under different noise levels as
Table 10
Table 10: Performance under different noise levels with varying MNL weights.
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
Gauss e = 0
76.88
78.03
79.19
78.03
80.54
79.38
79.00
81.02
80.35
79.77
Gauss 6 72.56
72.02
73.41
73.56
74.07
73.64
72.06
73.22
73.80
72.87
Gauss e = 10
62.46
62.36
62.66
63.38
63.78
63.46
62.12
61.19
63.74
63.70
Salt =5 74.95
74.57
74.37
74.95
75.76
75.14
75.72
74.76
74.37
75.26
Salt 10
62.27
63.09
63.46
63.27
64.93
65.06
64.08
63.50
61.96
63.01
Table 11: Effect of warm-up epochs on accuracy.
Warm-up Epoch Accuracy (%)
5
10
15
20
30
68.55
68.11
68.11
69.18
68.79
68.26
69.05
To further investigate the effect of warm-up epoch settings, we report the results on the CREMA-D dataset with noise level 0 in Table IT We observe that the hyperparameter warm-up epoch is not sensitive (we set it to 10). This may because the benefit of MNL is relatively small during the early stages of training and becomes more pronounced as training progresses and the modality gap widens, allowing for more meaningful guidance. Therefore, the warm-up primarily introduced to reduce unnecessary computational costs during the initial training phase.
D.3 Additional Overhead During Training
Integrating MNL with baseline methods incurs minimal computational overhead and requires no modification to the underlying network architecture. During training, MNL operates solely at the output level, identifying the Robust Dominant Modality (RDM) and the Inferior Modality (IM), determining the guidance direction, and computing the additional loss. At inference time, the procedure is identical to that of the baseline, introducing no extra cost. Specifically, we calculate the average time per training iteration (per batch) on MVSA, FOODI01, NYUDv2, and CREMAD, respectively. As shown Table the additional overhead introduced by MNL during training is negligible.
Table 12: Inference time (in milliseconds) of different methods various datasets.
Method
MVSA
FOOD101
NYUDv2
CREMA-D
LATE FUSION
20.96 ms
69.23 ms
32.27 ms
63.03 ms
LATE FUSION MNL.
31.15 ms
85.76 ms
49.19 ms
87.11 ms
28

---

D.4 Full Results with Standard Deviation
In this section, we present the full results with standard deviation in Table 14 and Table 15
Table 13: Performance comparison under different guidance scopes and conditions across various Gaussian perturbation levels the MVSA dataset, where red and blue indicates the best/runner-up performance.
All-Class
Non-target
Prior
Confident
Robust
c
6 10
78.77 + 1.52
72.77 # 0.36
62.36 + 2.09
78.47 0.87
72.74 0.62
62.62 1 1.40
78.90 0.29
72.16 0.48
62.52
78.66 0.87
72.69
1.07
62.77
78.74 0.79
71.87
1.93
59.35 0.16
79.50 1.70
74.03 1.11
63.01 +
E
Limitations and Broader Impacts
The limitations of MNL can be better understood by examining the performance gap between modalities across datasets, as shown in Fig. In the CREMA-D dataset, the audio modality outperforms the visual modality by a large margin, whereas in the NYU Depth V2 dataset. the RGB modality slightly outperforms the depth modality. The modality gap in CREMA-D is noticeably larger than in NYU Depth V2. Since MNL primarily relies leveraging the stronger modality to guide the weaker one, its benefits may be limited when the performance gap is small and both modalities improve in sync. These observations suggest that the effectiveness of MNL is closely tied to the degree of modality complementarity and asymmetry present in the data.
Reviewing Equation 20 the lower bound of multimodal robustness is determined by both the modality weights and the UCoM. MNL. aims improve the robustness of the multimodal system by enhancing the UCoM of weaker modalities. In contrast, dynamic late fusion methods typically assign higher weights to modalities that are more confident in predicting the ground truth class. This fundamental difference introduces degree of incompatibility between MNL and existing dynamic weight allocation strategies. As a result, the performance gains achieved by MNL are often smaller than those observed with statie late fusion methods. In future work, it would be valuable to explore whether dynamic weight allocation can be guided by UCoM, so that the enhancements MNL provides weaker modalities are not undermined by their lower assigned weights.
Furthermore, MNL can be directly integrated with most late fusion methods, with the exception of certain approaches such as evidence-basee late fusion. In contrast, how to effectively incorporate MNL into early fusion frameworks remains an open question, which we leave for future work.
Regarding the potential social impact, this paper proposes a method that leverages robust dominant modalities to guide inferior modalities suppressing non-target classes, effectively enhancing the robustness of multimodal systems under noisy conditions. This approach holds promise for safety- critical applications such and reliable in complex environments. However, due to the high uncertainty and uncontrollability of noise open environments, certain risks may still exist high-stakes scenarios like autonomous driving and medical diagnosis.
computer interaction, helping multimodal systems remain stable
29

---

Table 14: Performance of different methods under varying noise levels across four (MVSA, FOODI01, NYU Depth V2, and CREMA-D) datasets. We add Gaussian noise on 50% modalities and presents the noise degree.
DATASET
METHOD
= 0.0
c= 5.0
c= 10.0
IMG
64.12 = 1.23
49.36 = 2.02
45.00 + 2.63
TEXT
75.61 = 0.53
69.50 = 1.50
47.41 - 0.79
LATE FUSION
76.88 1.30
63.46 = 3.46
55.16 1 3.60
DYNMM
79.07 = 0.53
67.96 + 1.65
59.21 + 1.41
TMC
74.87 2.24
66.72 4.55
6..35
MVSA
QMF
78.07 = 1.10
73.85 = 1.42
2.12
PDF
79.94 = 0.95
74.40 + 1.51
63.09 1: 1.33
LATE FUS
79.50 = 1.70
74.03 +: 1.11
63.01 + 1.74
QMF+MNL
79.45 = 0.60
74.12 = 0.06
62.75 0.04
PDF+MNL
80.54 = 0.85
74.07 + 1.93
63.78 + 1.27
IMG
64.62 = 0.40
34.72 = 0.53
33.03 + 0.37
TEXT
86.46 = 0.05
67.38
0.19
43.88 0.32
LATE FUSION
90.69 = 0.12
68.49 = 3.37
57.99 + 1.59
DYNMM
92.59 = 0.07
74.74 0.19
59.68 0.20
UMPC FOOD 101
TMC
89.86 = 0.07
73.93 = 0.34
61.37 = 0.21
QMF
92.90 = 0.11
76.03 = 0.70
62.21 0.25
PDF
93.32 = 0.22
76.47 1 0.31
62.83 1 0.31
LATE MUNION+MNL
92.77 = 0.17
75.16 1 0.80
62.06 0.49
QMF+MNL
93.03 = 0.16
75.41 = 0.48
62.59 + 0.55
PDF+MNL
93.33 = 0.22
76.65 0.31
63.16 0.32
DEPTH
63.30 = 0.48
53.12 - 1.52
45.46 + 2.07
RGB
62.65 = 1.22
50.95 = 3.38
44.13 3.80
LATE FUSION
70.03 = 0.84
64.37 = 0.80
60.55 + 1.65
DYNMM
65.50 = 0.37
54.31 = 1.72
46.79 + 1.09
NYU DEPTH V2
TMC
70.40 = 0.31
59.33 - 2.19
50.61 + 2.87
QMF
69.54 = 1.06
64.10 1.42
60.18 - 1.23
PDF
71.37 = 0.76
65.72 1.72
62.56 1.84
LATE FUSION+MNL
71.05 = 1.06
67.02
0.56
63.81 + 1.63
QMF+MNL
71.25 = 0.59
65.38 + 1.09
61.80 + 1.52
PDF+MNL
71.52 = 0.89
67.01
1.72
63.07 0.50
AUDIO
60.70 = 0.94
59.60
1.40
49.52 2.91
VISUAL
56.23 = 1.67
52.47 = 2.06
38.17 3.74
LATE FUSION
68.04 = 3.92
64.25 - 6.68
52.39 7.40
DYNMM
63.27 = 0.68
62.01 - 1.27
51.43 1.24
TMC
63.63 = 1.16
62.68 1.17
57.97 2.14
CREMA-D
QMF
66.13 = 1.32
64.27 = 1.47
50.77 5.83
PDF
67.07 - 0.83
64.57
1.94
53.33
2.78
LATE FUSION+MNL
73.71 = 2.10
70.35 + 1.27
57.26 + 4.28
QMF+MNL
68.18 = 0.35
67.00
0.95
52.62
2.96
PDF+MNL
69.18 = 0.96
66.94 + 1.82
55.43 + 4.61
30

---

Table 15: Performance of different methods under varying noise levels across four (MVSA, FOOD101, NYU Depth V2, and CREMA-D) datasets. We add Salt noise on 50% modalities and presents the noise degree.
DATASET
METHOD
0.0
c 5.0
10.0
IMG
64.12 + 1.23
56.72 + 1.92
50.71 + 3.20
TEXT
75.61 + 0.53
69.50 + 1.50
47.41 + 0.79
LATE FUSION
76.88 + 1.30
67.88 + 1.87
55.43 + 1.94
DYNMM
79.07 + 0.53
71.35 1 0.97
59.96 1 1.31
TMC
74.87 2.24
68.02
56.62 3.07
MVSA
QMF
78.07 1.10
73.90 + 1.89
60.41 + 2.63
PDF
79.94 + 0.95
75.11 1 1.15
61.97 1: 1.14
LATE FUSION+MNL
79.50 1 1.70
74.68 1 1.43
62.31 1 1.71
QMF+MNL
79.45 - 0.60
75.14 1.19
64.68 + 1.94
PDF+MNL
80.54 + 0.85
75.76 1 0.48
64.93 + 1.37
IMG
64.62 + 0.40
50.75 + 0.44
36.83 + 0.92
TEXT
86.46 0.05
67.38 + 0.19
43.88 + 0.32
LATE FUSION
90.69 + 0.16
77.99 + 0.54
58.75 1 0.99
DYNMM
92.59 0.07
78.91 1 0.20
57.64 1 0.30
UMPC FOOD 101
TMC
89.86 + 0.07
77.86 0.41
60.22 + 0.43
QMF
92.90 0.13
80.87 + 0.40
61.60 1 0.20
PDF
93.32 1 0.22
81.21 1 0.34
61.76 1 0.33
LATE FUSION+MNL
92.77 0.17
80.87 1 0.57
61.41 1.49
QMF+MNL
93.03 + 0.16
81.14 0.52
62.47 + 0.38
PDF+MNL
93.33 0.22
81.52 1 0.27
62.95 1 0.19
DEPTH
63.30 - 0.48
50.99 1.41
38.56 + 2.16
RGB
62.65 + 1.22
1.41
34.76 + 1.59
LATE FUSION
70.03 + 0.84
62.05
51.50 + 1.81
DYNMM
65.50 + 0.37
1..46
3..17
NYU DEPTH V2
TMC
70.40 0.31
59.33
45.32 + 2.84
QMF
69.54 + 1.06
62.02
51.87 + 0.91
PDF
71.37 1 0.76
64.27 1 1.36
53.62 2.15
LATE FUsION+MNL
71.05 + 1.06
(4.88
54.08 2.34
QMF+MNL
71.25 + 0.59
63.62 + 1.62
53.30 + 2.60
PDF+MNL
71.52 1: 0.89
64.32 1 0.58
54.08 + 0.83
AUDIO
60.70 0.94
59.60 + 1.40
49.52 + 2.91
VISUAL
56.23 + 1.67
50.62 + 3.27
43.90 + 2.80
LATE FUSION
68.04 3.92
62.50 1 6.63
52.61 1 6.51
DYNMM
63.27 + 1.74
62.92 + 0.98
52.33 3.57
TMC
63.63 1.16
62.31 1.93
3.114
CREMA-D
QMF
66.13 + 1.32
63. + 1.11
51.55 5.31
PDF
67.07 0.83
63.44 1 2.39
53.71 2.40
LATE FUSION+MNL
73.71 + 2.10
68.06 + 1.25
58.41 + 4.70
QMF+MNL
68.18 0.35
64.05 + 2.93
52.32 2.67
PDF+MNL
69.18 1 0.96
63.77 1 2.24
54.30 1 4.84
31

---

Unimoci Accumey CREMADD D
Unimodal Accunary NYu Dupth V2
6.5
Opech
Epech
Figure 8: Unimodal performance the CREMA-D and NYU Depth V2 datasets
32


Reviews:
[{"review_id": "P4oSlBwvIj", "reviewer": "Reviewer_yEYi", "summary": "The paper addresses the modality imbalance problem, where one modality may dominate others due to noise, limited information, or differences in sensing methods. Rather than forcing weak modalities to align with stronger ones which risks suppressing modality specific information the proposed method aims to preserve the unique information in each modality while fixing the problem created due to modality imbalance.", "questions": "-  Robustness radius R(x_i) is greater means the perturbed x_i is far from x_i. You want this small right? If you get larger UCom it means I will have large R(x_i) which is not good right? How does this leads to greater robustness?\n\n- Please refer weakness for more questions.", "limitations": "Yes. In conclusion section as a future work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The idea of preserving the information in weaker modality seems to be novel.\n- The paper is driven by the theoretical findings.\n\nWeakness:\n- The paper is hard to parse and I have following comments:\n\n      - Abstract (What does it mean to enhance target class to supress non-target class in weak modalities? )\n\n      - Use same word either unimodal or uni-modal (line 25), similarly late fusion or late-fusion.\n\n      - What is late-fusion (line 29) state for readers?\n\n      - Fig 1b) video and guidance spelling wrong?\n\n      - Line 43-50 and Fig 1a) isn't increasing the probability of target class same as decreasing or suppressing the probabilities of non-target class?\n\n      - What does right side of block diagram represent Fig 1a?\n\n      - Fig 1b) Why is video error rate with KL guidance higher than guidance? This means there's no point in using KL guidance at all.\n\n      - Fig 1b) Does audio model also change? The figures are very hard to parse.\n\n- The setting of multimodal learning is not clear at the start. Is the goal to learn a separate classifiers on each modality and effectively combine them or transfer the discriminative power from strong modality to weaker one.\n- What are equations in Fig (2) ?\n- Theorem and notations are not clear and precise. Properly introduce notations.\n- Properly introduce late fusion, it is used everywhere.\n- What does decision space instability means?\n- What does uncertainty mean? Does it mean lower probability ? Is it same as confidence?\n- Hard to parse equation (5)?\n- What does $P_y^{(2)}$ means? No definitions provided.\n-  What are static and dynamic late fusion? (some intuition would help)\n-  Compare with positive learning.\n- Results on additional datasets for Table 3 and 4.\n- Experimental settings not properly provided with details on hyperparameters.\n- What is $\\epsilon$ in line 227, Is it gaussian noise variance ?\n- If one can segregate the RDM and IM, then why do we need the fusion itself. The RDM is responsible to correct the IM, but why not discard the IM rather than improving it? Probably need an experiment.", "was_revised": true, "final_justification": "I appreciate the authors for addressing the concerns regarding clarity and for incorporating the suggested details into the revised version of the paper and response.\n\nI would also encourage the authors to briefly motivate, within the main paper, why late fusion is beneficial compared to latent space-based methods.\n\nFollowing the above discussion, the authors have agreed to include additional experiments, an introduction to late fusion, a notation table, refined theorem and precise notations for improved clarity in the main paper. If these points are adequately addressed, I believe they will significantly enhance the overall clarity and quality of the work.\n\nAccordingly, I am raising my **clarity and quality scores to 2** each. I would also like to **update my overall score to 4 (Borderline Accept)**, contingent upon the authors incorporating the discussed revisions into the final version.", "rebuttal": "Thanks for your valuable feedback, which greatly contributed to improving the paper!\n\n> W1: Enhance target class to supress non-target class. & Compare with positive learning.\n\nThanks for your comment. There maybe some potential misunderstanding. \n**MNL aims to guide the weak modality’s predictions on non-target classes using the dominant modality’s predictions on non-target classes**, rather than constraining the weak modality’s prediction on the target class through the dominant modality’s prediction on the target class. We have added experiments on target class guidance (positive learning) and non-target classes guidance (MNL). As shown in the Table below, the positive learning is not effective enough.\n\n| Method                                | 𝜖 = 0 | 𝜖 = 5 | 𝜖 = 10 |\n|---------------------------------------|--------|--------|---------|\n| STATIC LATE FUSION                    | 76.88  | 63.46  | 55.16   |\n| STATIC LATE FUSION + target class guidance (positive learning) | 77.54  | 63.25  | 55.43   |\n| STATIC LATE FUSION + MNL              | **79.50**  | **74.03**  | **63.01**   |\n\n(1) **Guidance on target class:** Using the dominant modality's prediction on target class to guide that of the weak modality essentially performs single-class guidance, which is similar to applying a soft-label constraint on the target class. This overlaps with the supervision already provided by the cross-entropy loss, thus offering limited additional benefits.\n\n(2) **Guidance on non-target classes:** By leveraging the dominant modality’s predictions on non-target classes to guide those of the weaker modality, we provide the weak modality with richer semantic cues — namely, the dominant modality’s understanding of non-target categories. This effectively enhances the weak modality’s comprehension of non-target classes and improves its overall performance.\n\n\n> W3 & W9 & W12 & W17: Multimodal learning & Late-fusion & Static and dynamic late fusion?\n\n(1) **Multimodal Learning (MML)** refers to learning comprehensive information from multiple modalities and effectively integrating them [1-3], which can be mainly categorized into early, mid-level, and late fusion. \n\n(2) **Late fusion** typically refers to combining the outputs of different models as the final prediction [4-5]. Formally, each modality $x^{(m)}$ is passed through a modality-specific projection function $f^{(m)}$, resulting in an output $z^{(m)} = f^{(m)}(x^{(m)}) \\in \\mathbb{R}^C$, where $C$ is the number of classes. These outputs are then fused as:\n\n$\nz^{\\text{final}} = \\sum_{m=1}^M w^{(m)} \\cdot z^{(m)}\n$\n\nHere, $w^{(m)}$ is the fusion weight for the $m$-th modality, and $\\sum_{m=1}^M w^{(m)} = 1$.\n\n- **Static late fusion** uses fixed weights $w^{(m)}$ across all samples. It assumes the reliability of each modality is consistent.\n- **Dynamic late fusion** allows $w^{(m)}$ to vary across samples, often based on the confidence or uncertainty of each modality’s output. This enables the model to adaptively rely more on the high-quality modality for each input.\n\n> W5: Increasing the probability of target class and suppressing the probabilities of non-target class.\n\n**Using the dominant modality's target class probability to guide that of the weaker modality essentially performs single-class guidance, which is similar to applying a soft-label constraint on the target class.** This overlaps with the supervision already provided by the cross-entropy loss, thus offering limited benefit.\n\nIn contrast, **constraining the non-target class predictions of the weaker modality using those from the dominant modality provides richer semantic information**. This complements the label-based cross-entropy loss and effectively enhances the weaker modality’s understanding of non-target classes, thereby improving its overall performance.\n\nIt is worth noting that **we do not use the full output distribution of the dominant modality to guide the weaker one, which would risk overriding the unique information captured by the weaker modality**. Instead, we selectively guide only the non-target class predictions, which allows us to inject informative signals from the dominant modality while preserving the individuality of the weaker one.\n\n> W6: Right side of block diagram in Fig 1a.\n\nThe weak modality is aligned with the dominant modality across all classes. In contrast, the bottom part of Fig. 1(a) illustrates the dominant modality guiding the weak modality specifically on the non-target classes.\n\nIn our formulation, the **specific** information refers to the probability of the target class, while the **common** information refers to the probabilities of the non-target classes.\n\n- **Existing methods (top) tend to align both specific and common information simultaneously**, which may interfere with the unique information of the weak modality.\n\n- **Our method (bottom) constrains only the non-target class predictions**, thereby introducing guidance from the dominant modality while preserving the unique information of the weak modality.\n\n> W7: Video error rate. \n\nThe bar chart shows cases where the audio modality errs while video is correct. The line chart tracks how many previously correct video samples become incorrect in the next epoch (Video error rate). This demonstrates that aligning a modality (video) with another across all classes can lead to a  over-alignment collapse.\n\n\n>W8: Does audio model also change?\n\nThe audio model changes during training , as both the video and audio models are trained simultaneously.\n\n>W10 & W15: Equations in Fig. 2 & Equation (5).\n\n(1) In **Figure 2**, we illustrate the training losses (CE loss and MNL loss) and highlight MNL **lifts the multimodal robustness lower bound** during optimization.(W10)\n\n(2) Eq. 5 presents the specific form of the MNL loss. It uses the RDM’s predictions to guide IM on non-target classes.\n\n> W13 & W14 : Decision space instability&uncertainty \n\n**Decision space instability** refers to a model's high sensitivity to input perturbations near the decision boundary, where even small changes can lead to significant shifts in prediction. And uncertainty can be intuitively viewed as low confidence.\n\nThe weak modality itself is relatively unstable, which, according to Eq. (4), may reduce the overall robustness of the multimodal system after fusion. In contrast, MNL enhances the UCoM of the weak modality, thereby improving the overall robustness of the system.\n\n\n> W19: Results on additional datasets for Table 3 & 4.\n\nWe extended our approach and ablation studies to three modalities datasets, the CMU-MOSEI. The results are presented as follow:\n\n| prior | confident | robust | ε = 0  | ε = 5  | ε = 10 |\n|--------|--------|--------|--------|--------|--------|\n| √     |           |        | 65.91% | 60.76% | 41.01% |\n|       | √         |        | 67.07% | 63.87% | 48.36% |\n|       | √         | √      |**67.29%** | **64.35%** | **48.62%** |\n\n\n| all-class | non-target | ε = 0  | ε = 5  | ε = 10 |\n|-----------|------------|--------|--------|--------|\n| √         |            | 67.24% | 60.20% | 39.81% |\n|           | √          |**67.29%** | **64.35%** | **48.62%** |\n\n> W20: Details on hyperparameters.\n\nWe have provided some implementation details in Line 215–218 and Line 94-114 of **Supp. C**.\nFor the wiegits of CE and MNL losses, the CE weight is fixed at 1 and the MNL weight is varied from 0.2 to 2.0 under different noise levels, we set weight to 1.\n\n|weight    | 0.2    |  0.6    | 1.0        | 1.4    |  2.0    |\n| -------- | ------ |  ------ | ---------- | ------ |  ------ |\n| Gauss_0  | 0.7688 |  0.7919 | **0.8054**   | 0.7900 |  0.7977 |\n| Gauss_5  | 0.7256 |  0.7341 | **0.7407** | 0.7206 |  0.7287 |\n| Gauss_10 | 0.6246 |  0.6266 | **0.6378** | 0.6212 |  0.6370 |\n| Salt_5   | 0.7495 |  0.7437 | **0.7576** | 0.7572 |  0.7526 |\n| Salt_10  | 0.6227 |  0.6346 | **0.6493**     | 0.6408 |  0.6301 |\n\n> W21: $\\epsilon$.\n\nWe followed the setting of $\\epsilon$ in QMF [7] and TMC [8], and unified the noise levels to 0, 5, and 10 for clearer comparison. \n\nFor Gaussian noise, it is noise intensity; for salt-and-pepper noise, it is the probability of noise occurrence; for audio, it is the SNR-based noise level; and for text noise, it denotes the probability of blank noise. \n\n> W22: RDM and IM.\n\n- Even if the RDM and IM can be segregated, this does not imply that the performance of the unimodal RDM surpasses that of the fusion of RDM and IM. This is evident from the unimodal versus multimodal fusion results shown in Tables 1 and 2.\n\n- We utilize both RDM and IM to integrate complementary information. Extensive research has shown that multimodal fusion generally outperforms unimodal [6,7].\n\n>Q1: Robustness radius R(x_i).\n\nThere might be some misunderstanding here. Our model aims to increase the robustness radius $R(x)$, where a larger UCoM indicates a favorable phenomenon. Specifically, $R(x_i)$ is defined as the *minimum $\\ell_2$ perturbation required to flip the model’s prediction from the ground-truth class $y$ to any other class*. A **larger $R(x_i)$** signifies **higher robustness** to input perturbations.\n\nAccording to Eq. 4, a higher UCoM corresponds to a larger $R(x)$, which means stronger robustness of the multimodal system. MNL leverages the modality with high UCoM to guide the modality with low UCoM, thereby improving the latter’s UCoM and ultimately enhancing the overall system robustness.\n\n> Presentation issues.\n\nWe have standardized the use of the terms unimodal and late fusion, corrected the spelling error in Fig. 1(b), and refined Theorem and notations. Finally, we have carefully reviewed and revised all representation issues throughout the paper.\n\n[1] Multimodal machine learning: A survey and taxonomy. TPAMI2018\n\n[2] Multimodal learning with transformers: A survey. TPAMI2023\n\n[3] Multimodal deep learning. ICML2011\n\n[4] PDF. ICML2024\n\n[5] QMF. ICML2023.\n\n[6] DUA-Nets. AAAI2021\n\n[7] CRMT. ICLR2024\n\n[8] OGM-GE. CVPR2022\n\n[9] MMPareto. ICML2024"}, {"review_id": "SbknL1HBjd", "reviewer": "Reviewer_AePS", "summary": "This paper introduces Multimodal Negative Learning (MNL), a framework to address modality imbalance where dominant modalities often overshadow weaker ones. Traditional approaches attempt to force weak modalities to align their predictions with dominant ones, which can suppress the unique information present in the weak modality. In this paper, instead of enhancing the weak modality's ability to predict the correct class, the dominant modality guides it to suppress non-target classes. The authors claim that MNL helps to stabilize the decision space, preserve modality-specific information, and prevent the over-alignment.\n\nThe result shows the robustness improvement on multiple benchmarks, especially in noisy and imbalanced conditions.", "questions": "Do we have any ablations to the warm-up settings (e.g. steps)? This looks to affect the final model performance much.\nWhat is the additional efficiency cost for MNL training?\n\nminor: the paper’s name looks too general – “Multimodal Negative Learning” sounds to be able to cover the common MLLM tasks, which unfortunately are not verified in this paper \n\ntypo in the legend of Figure 1 (b): “vedio” → “video”", "limitations": "Limitations are not mentioned in this paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe concept of \"Learning Not to be\" is novel to the modality imbalance problem. Shifting the learning objective for weak modalities from identifying the right answer to avoiding wrong answers sounds to make sense and insightful.\n\nThe paper provides a rigorous theoretical justification.\nMNL is tested on four distinct datasets (MVSA, UPMC Food-101, NYU Depth V2, CREMA-D) spanning different task domains and modality combinations. The evaluations are also conducted under multiple noise types and levels.\n\nThis paper is well written and easy to read.\n\nWeaknesses\n\nThe authors claim modality imbalance is a problem, but there is lack of proof in which conditions such a problem exists and how serious it is. IMO, it’s natural to rely on one modality more than others, for example, self-driving often relies more on visual signals than others (e.g. audio). In MLLM, it’s also common to sometimes rely on one modality but sometimes rely on the other.\n\nThis paper also claims MNL can preserve specific information for weak modalities, but there is lack of such proof or analysis as well.\n\nThe tasks demonstrated in this paper are all for multimodal classification, which IMO is not commonly used in the real world. Instead, the most common scenario in the ML domain is the MLLMs (and also often “late fusion”, which fits this paper’s scope). It will be more convincing if this paper can show improvement on common MLLM tasks.\n\nLimitations are not mentioned in this paper.", "was_revised": true, "final_justification": "My previous major concern was the insufficient evals on common MLLMs. The authors conducted two additional evaluations during the rebuttal process, and the new result looked good to me. Thus I raised my score accordingly.", "rebuttal": "We sincerely appreciate the reviewers’ positive recognition of the novelty of our “Learn Not to Be” concept, the soundness of the theoretical justification. Below, we provide detailed responses to your concerns.\n\n> W1  lack of proof in which conditions modality imbalance exists and how serious it is. In MLLM, it’s also common to sometimes rely on one modality but sometimes rely on the other.\n\nThank you for your valuable comment.\n\n**Multimodal imbalance arises from the model’s limited ability to perceive and utilize weak modalities [1], rather than deficiencies in the input data itself [2].**\n\n- Multimodal imbalance often occurs during model optimization [3], where the learning of one modality suppresses the learning of others.\n- Multimodal imbalance is not related to the unbalanced usage of different modalities, nor does it imply that certain modalities provide less informative data.\n- Even if a modality itself is informative, the model may still fail to make accurate predictions if it cannot effectively exploit the modality’s useful information.\n\nPlease kindly note that this notion does not conflict with dynamic fusion strategies that utilize modalities based on data quality.\n\nIn our paper, using the CREMA-D dataset as an example, the audio modality alone achieves 60.70% accuracy, and the **visual modality 56.23%**.\n\n- With static late fusion and joint training, audio accuracy drops slightly to 59.35%, while visual accuracy decreases sharply to **41.12%**, demonstrating severe modality imbalance during optimization.\n- However, with MNL guidance, fusion accuracy improves to 73.71%, and the visual modality’s accuracy increases to **58.05%** (>56.23%), surpassing its standalone performance.\n\n> W2 This paper also claims MNL can preserve specific information for weak modalities, but there is lack of such proof or analysis as well.\n\nThanks for your valuable comments. We have discussed the specific information preservation for weak modalities in Line 298-313 and Figure 4 (a),  Line 51-72 and Figure 2 in Supp.\n\nAs shown in **Figure 4(a)**, we compare the average KL divergence between modality predictions and the corresponding performance under different guidance strategies and noise levels.\n\n- All-Class guidance reduces KL divergence by forcing the weaker modality to imitate the stronger one, but this leads to over-alignment and performance degradation.\n- In contrast, MNL applies guidance only on non-target classes, resulting in higher KL divergence and consistently better performance.\n\nMNL achieves improvements of 1.83%, 0.85%, and 4.64% under increasing noise levels. These results indicate that preserving the weaker modality’s distinct information improves robustness and cross-modal complementarity.\n\nIn **Supplementary B.1** (shown as pictures),Follow [4] we employ the *dit* toolkit [5] to perform partial information decomposition, aiming to quantify the unique information contributed by each modality. Furthermore, we present a quantitative analysis to support our findings:\n\nTable 1 under Gaussian noise\n\n\n| Method                   | Modality | ε = 0 | ε = 5 | ε = 10 |\n| ------------------------ | -------- | ------ | ------ | ------- |\n| STATIC LATE FUSION       | Audio    | 0.8986 | 0.7848 | 0.4853  |\n|                          | Visual   | 0.0000 | 0.0023 | 0.0063  |\n| STATIC LATE FUSION + MNL | Audio    | 0.4364 | 0.3556 | 0.1221  |\n|                          | Visual   | 0.0076 | 0.0149 | 0.0191  |\n\nTable 2 Under Salt Noise\n\n\n| Method                   | Modality | ε = 0 | ε = 5 | ε = 10 |\n| ------------------------ | -------- | ------ | ------ | ------- |\n| STATIC LATE FUSION       | Audio    | 0.8986 | 0.6150 | 0.4687  |\n|                          | Visual   | 0.0000 | 0.0028 | 0.0037  |\n| STATIC LATE FUSION + MNL | Audio    | 0.4364 | 0.2640 | 0.1073  |\n|                          | Visual   | 0.0076 | 0.0122 | 0.0196  |\n\nAs shown in the results, MNL enhances the consistency of weak modality (visual) in expressing its unique information and leads to improved performance. This provides direct evidence and analysis that MNL effectively preserves specific information for weak modalities.\n\n> W3 & Q2 this paper are all for multimodal classification. It will be more convincing if this paper can show improvement on common MLLM tasks. & the paper’s name looks too general\n\nThank you for your valuable comment.\n\n- Considering that most existing MLLM models [6] primarily perform   feature-level fusion, to demonstrate the generalizability of our approach,**we extend MNL    to large language models**, forming a  multi-LLM fusion setting.\n\n  Specifically, on the MathQA dataset [11], a choice math reasoning  benchmark, we fine-tune InternVL2_5-1B and Qwen2.5-1.5B (which share the same vocabulary) using LoRA. Furthermore, during training, we apply MNL by masking  the correct option and using $P_y$ and UCoM  to allow the stronger model to guide the weaker  one.\n\n  We compare the performance of STATIC LATE FUSION and STATIC LATE FUSION + MNL   to validate the effectiveness of our method. As shown in the table, applying  MNL consistently improves both the individual LLM and the fused result.\n\n\n| Method                   | Fusion | InternVL2_5-1B | Qwen2.5-1.5B |\n| ------------------------ | ------ | -------------- | ------------ |\n| STATIC LATE FUSION       | 50.89  | 42.85          | 49.41        |\n| STATIC LATE FUSION + MNL | 51.42  | 43.32          | 50.85        |\n\n- **MNL belongs to multimodal learning.** Multimodal learning is indeed a sophisticated and widely studied paradigm in the machine learning community and it refers to the integration and joint modeling of data from different modalities such as text, images, speech, video[7]. The aim is to improve a model's understanding capability and generalization performance[8]. This paradigm naturally covers a wide range of tasks, including multimodal classification [9,10], all of which fall under the broad definition of multimodal learning.\n\n  **the core of our method lies in establishing a negative learning paradigm within a multimodal system**. Specifically, the Robust Dominant Modality (RDM) provides supervision on non-target classes to guide the Inferior Modalities (IMs). This strategy not only preserves the unique information inherent to the IMs but also enhances the overall robustness of the multimodal system. Following the reviewers' suggestion, we further validated the effectiveness of MNL in the multiple-LLM setting.\n\n  Therefore, we believe that naming our method **Multimodal Negative Learning** is both appropriate and representative of its core contribution.\n\n> Q1 Do we have any ablations to the warm-up settings (e.g. steps)? This looks to affect the final model performance much. What is the additional efficiency cost for MNL training?\n\nThank you for your question.\n\n**(1) Warm-up Settings:**\n\nWe have provided the ablations on  warm-up epoch. According to the following table, the hyperparameter **warm-up epoch** is not sensitive in practice (we set it to 10). The reason is that the benefit of MNL is relatively limited during the early stages of training. As training progresses and the modality gap becomes more evident, MNL provides more effective guidance. Therefore, the warm-up setting primarily serves to reduce unnecessary computational cost in the early phase of MNL training.\n\n\n| Warm-up Epoch | 0     | 2     | 5     | 10    | 15    | 20    | 30    |\n| ------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Accuracy (%)  | 68.55 | 68.11 | 68.11 | 69.18 | 68.79 | 68.26 | 69.05 |\n\n**(2) Efficiency Cost:**\n\nWe provide a detailed efficiency analysis of MNL training’s extra cost by reporting average training iteration time on MVSA, FOOD101, NYUDv2, and CREMA-D, with no added overhead during inference.\n\n\n| Method                   | MVSA    | FOOD101 | NYUDv2  | CREMAD  |\n| ------------------------ | ------- | ------- | ------- | ------- |\n| STATIC LATE FUSION       | 20.96ms | 69.23ms | 32.27ms | 63.03ms |\n| STATIC LATE FUSION + MNL | 31.15ms | 85.76ms | 49.19ms | 87.11ms |\n\nThese results indicate that the additional overhead introduced by MNL is minimal.\n\n> Limitations are not mentioned in this paper.\n\nThanks for your comment.  we have provided a detailed discussion of the limitations Line 123-142 in **Supplementary E**. For example, the additional computational overhead introduced by MNL during training.\n\n> presentation issues\n\nThank you for pointing this out. We have corrected the typo in the legend of Figure 1(b) and carefully reviewed and revised all representation issues throughout the paper.\n\nAnd we sincerely hope that our responses have sufficiently addressed your concerns. Should any questions remain, we would be more than glad to provide further clarification. If you find our clarifications satisfactory, we would greatly appreciate your consideration in updating your score to support the acceptance of our paper.\n\n[1]Balanced multimodal learning via on-the-fly gradient modulation. CVPR2022\n\n[2]Missing modality imagination network for emotion recognition with uncertain missing modalities. ACL2021\n\n[3]Balanced Multimodal Learning via On-the-fly Gradient Modulation. CVPR2022\n\n[4]Reconboost:Boosting can achieve modality reconcilement. ICML2024\n\n[5]dit: a Python package for discrete information theory.\n\n[6]Mm-llms: Recent advances in multimodal large language models. ACL2024\n\n[7] Multimodal machine learning: A survey and taxonomy.TPAMI2018\n\n[8]A theory of multimodal learning. NeurIPS2023\n\n[9]Trusted multiview classification. ICLR2021\n\n[10]Multimodal dynamics: Dynamical fusion for trustworthy multimodal classification. CVPR2022\n\n[11]Mathqa: Towards interpretable math word problem solving with operation-based formalisms."}, {"review_id": "DlM5Xb4uoe", "reviewer": "Reviewer_9fVt", "summary": "This paper introduces a new paradigm for addressing modality imbalance in multimodal learning, called Multimodal Negative Learning (MNL). Unlike traditional positive learning methods that align weak modalities with dominant ones by enhancing target-class prediction, MNL proposes guiding weak modalities to suppress non-target classes, thus stabilizing their decision space. The framework builds on a formal robustness analysis using a newly defined Unimodal Confidence Margin (UCoM), which forms the basis of a provable lower bound on multimodal robustness. MNL is tested under various noise conditions across four datasets and consistently outperforms both static and dynamic late-fusion baselines. The paper is well written, with clear organization, logical flow, and the use of precise and professional terminology throughout.", "questions": "* Training complexity analysis.\n* Extension to the scenarios with three modalities.", "limitations": "yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strength:**\n* The concept of **learning not to be** is novel in the multimodal setting and effectively reframes weak modality alignment as negative class suppression rather than target class enhancement. Furthermore, the paper rigorously derives a robustness lower bound for late-fusion systems based on UCoM, providing strong theoretical motivation for the MNL design.\n* The proposed Robust Dominant Modality (RDM) definition is dynamic and data-driven, improving generalization and adaptability during training.\n* The paper conducts extensive evaluations across multiple datasets and under varying levels of perturbation (e.g., Gaussian, salt noise), with MNL consistently showing performance and robustness gains.\n* The paper demonstrates a high standard of academic writing, with coherent organization, a logical progression of ideas, and consistently precise use of domain-specific terminology.\n\n**Weaknesses**:\n* The two-stage training scheme (standard CE followed by MNL) adds some complexity to training dynamics and might not be ideal for scenarios requiring fast adaptation.\n* It is recommended that the authors provide a more detailed algorithmic workflow or pseudocode to facilitate understanding and reproducibility of the proposed method.\n* While MNL is presented for dual-modality cases, its extension to three or more modalities (especially when all are noisy) is not discussed or demonstrated.", "was_revised": true, "final_justification": "I am satisfied with the authors’ responses, and my concerns have been fully addressed.", "rebuttal": "We sincerely thank you for your valuable comments and appreciate your recognition of our proposed paradigm \"Learning not to be\" through Multimodal Negative Learning (MNL) and the theoretical foundation based on UCoM. We believe your constructive feedback will help improve the paper and enhance its potential impact on the multimodal learning community.\n\n> W1 & Q1 The two-stage training scheme (standard CE followed by MNL) adds some complexity to training dynamics and might not be ideal for scenarios requiring fast adaptation. & Training complexity analysis.\n\nThanks for your comment. Integrating MNL with baseline methods introduces minimal computational overhead and does not require any modification to the baseline network architecture. During training, MNL operates only at the output level by identifying the Robust Dominant Modality (RDM) and the Inferior Modality (IM), determining the direction of guidance, and computing the additional loss. During inference, the process remains identical to the baseline and does not introduce any extra cost.\n\nWe are glad to report the additional training cost. Specifically, we calculate the average time per training iteration (per batch) on MVSA, FOOD101, NYUDv2, and CREMAD, respectively.\n\n\n| Method                   | MVSA    | FOOD101 | NYUDv2  | CREMAD  |\n| ------------------------ | ------- | ------- | ------- | ------- |\n| STATIC LATE FUSION       | 20.96ms | 69.23ms | 32.27ms | 63.03ms |\n| STATIC LATE FUSION + MNL | 31.15ms | 85.76ms | 49.19ms | 87.11ms |\n\nThese results indicate that the additional overhead introduced by MNL is minimal.\n\n> W2 It is recommended that the authors provide a more detailed algorithmic workflow or pseudocode to facilitate understanding and reproducibility of the proposed method.\n\nThank you for the excellent suggestion. Below, we provide the pseudocode of the algorithm. The inclusion of this pseudocode will be reflected in the final version of the paper.\n\n```text\nAlgorithm 1: Multimodal Learning with MNL\n\nInput:\n Training set D_train = {(x_i, y_i)}_{i=1}^N ⊂ X × Y\n Iteration number T, warm-up epoch W\n\nfor t = 1 to T do\n    Sample a fresh mini-batch B_t from D_train\n    Feed-forward the batched data B_t to the model\n    Obtain fusion logits and modality-specific logits from the model\n\n    if t > W then\n        Identify RDM and IMs according to Definition 3.3\n        Compute MNL loss according to Eq. (5)\n        Compute total loss according to Eq. (7)\n    else\n        Compute cross-entropy loss\n    end if\n\n    Update model parameters\nend for\n```\n\n> W3 & Q2 While MNL is presented for dual-modality cases, its extension to three or more modalities (especially when all are noisy) is not discussed or demonstrated.\n\nWe conducted additional experiments on the CMU-MOSEI dataset[1], which contains three modalities: text, audio, and visual. In this setting, the Robust Dominant Modality(RDM) is used to guide the other two weaker modalities. The RDM is identified by comparing $P_y$ and $\\xi$ (UCoM) across modalities, in accordance with Definition 3.3. This strategy naturally generalizes to scenarios involving more than two modalities.\n\nTable 1 presents the experimental results conducted on the CMU-MOSEI dataset. Gaussian noise for visual modality, blank noise for the text modality, SNR-based noise for audio modality\n\n\n| Method                 | ε = 0 | ε = 5 | ε = 10 |\n| ---------------------- | ------ | ------ | ------- |\n| STATIC LATE FUSION     | 66.42% | 61.71% | 45.80%  |\n| STATIC LATE FUSION+MNL | 67.44% | 63.36% | 52.09%  |\n| PDF                    | 66.14% | 63.54% | 42.47%  |\n| PDF+MNL                | 67.29% | 64.35% | 48.62%  |\n\nTable 2 ablation experiments\n\n\n| prior | confident | robust | ε = 0     | ε = 5     | ε = 10    |\n| ----- | --------- | ------ | ---------- | ---------- | ---------- |\n| √    |           |        | 65.91%     | 60.76%     | 41.01%     |\n|       | √        |        | 67.07%     | 63.87%     | 48.36%     |\n|       | √        | √     | **67.29%** | **64.35%** | **48.62%** |\n\n\n| all-class | non-target | ε = 0     | ε = 5     | ε = 10    |\n| --------- | ---------- | ---------- | ---------- | ---------- |\n| √        |            | 67.24%     | 60.20%     | 39.81%     |\n|           | √         | **67.29%** | **64.35%** | **48.62%** |\n\nThank you again for this valuable suggestion. We have added the experiment to the manuscript, and it will be included in the final version.\n\n[1]Multibench: Multiscale benchmarks for multimodal representation learning. Advances in neural information processing systems 2021"}, {"review_id": "eiwNITvC8y", "reviewer": "Reviewer_d8o7", "summary": "This paper introduces a new paradigm for multimodal learning called Multimodal Negative Learning (MNL). The core idea is to address the \"modality imbalance\" problem, where a dominant modality can overwhelm a weaker one during training. Instead of traditional \"Positive Learning\" that forces the weak modality to mimic the strong one's predictions for the correct class, MNL uses the dominant modality to guide the weak modality to suppress non-target classes.", "questions": "1. What are the loss weights in Eq. 7 (is MNL summed equally with CE losses?) \n2. What are the criteria for transitioning Stage 1→Stage 2 training? \n3. Fig 1(b) shows forced alignment harms weak modalities. Does MNL mitigate such collapse? It will be beneficial to provide examples where MNL fails to preserve weak-modality advantages, and quantify how often this occurs.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength**: \n\n1. The paper addresses a well-known challenge in multimodal learning: modality imbalance and over-alignment issues.\n\n2. The paper attempts to theoretically justify its approach (lower bounds based on UCoM) by linking it to the robustness of the multimodal system. \n\n3. Experiments are conducted on four publicly available datasets with different modalities (vision, text, audio), providing a broad evaluation.\n\n4. Thorough ablation studies on different design choices. \n\n**Weakness**: \n\n1. In table 1 and table 2, the improvement from the baseline is not consistently strong. For instance, when MNL is applied to state-of-the-art dynamic fusion methods like PDF and QMF, the performance gains are often marginal, sometimes less than 1%. \n\n2. Theorem 3.1 assumes a statis fusion weights, yet experiments include dynamic fusion, creating a gap between theory and empirical results. \n\n3.  Equation 5 assembles knowledge distillation, where the \"soft labels\" from a teacher (RDM) are used to train a student (IM), diminishing the novelty of the paper.", "was_revised": true, "final_justification": "most of the questions have been addressed. However after the experiments of MNL (step1) + PDF (step2), the performance gain is still small when comparing with late fusion in table 1. Therefore, I'd like to maintain my score.", "rebuttal": "We'd like to thank the reviewer for the valuable comments and the recognition of our novel “Learn Not to Be” paradigm. We appreciate your support and constructive suggestions, and we address your concerns in detail as follows.\n\n> W1 In table 1 and table 2, the improvement from the baseline is not consistently strong. For instance, when MNL is applied to state-of-the-art dynamic fusion methods like PDF and QMF, the performance gains are often marginal, sometimes less than 1%.\n\nThank you for your valuable comment. Taking PDF [1] as an example, PDF aims to effectively leverage multimodal data, whereas **MNL focuses on enhancing the weak model's perceptual ability**. Therefore, when we follow the standard setting and directly integrate the two methods, their respective advantages may not be fully exploited. For dynamic fusion methods such as PDF, we suggest a two-step training procedure: First, use MNL to enhance the model’s recognition ability, especially for the weaker modality (Step 1), and then apply PDF to dynamically fuse informative content from different modalities (Step 2).\n\nSpecifically, during training:\n\n- **Step 1**: We perform multimodal fusion based on static late fusion and apply the MNL, while simultaneously training PDF to learn the ability to predict fusion weights $ w $.\n- **Step 2**: Based on the model obtained in Step 1, we utilize PDF to predict $ w $ and conduct dynamic fusion accordingly.\n\nTo validate this setting, we have conducted additional experiments on the MVSA dataset under Gaussian noise with varying levels of noise (ε = 0, ε = 5, ε = 10). The results are summarized in the following table (over 5 seeds).\n\n\n| Method                        | ε = 0    | ε = 5    | ε = 10   |\n| ----------------------------- | --------- | --------- | --------- |\n| PDF                           | 79.94     | 74.40     | 63.09     |\n| PDF + MNL                     | 80.54     | 74.07     | 63.78     |\n| **MNL (step1) + PDF (step2)** | **80.60** | **74.82** | **63.85** |\n\nThe most notable improvement occurs when ε = 5, where the performance of MNL (step1) + PDF (step2) surpasses that of PDF by 0.42%, in contrast to the 0.33% drop observed with PDF + MNL. The MNL (step1) + PDF (step2) configuration not only enhances the performance of the weaker modality but also dynamically allocates weights at inference based on data quality, thereby delivering a more effective boost to overall model performance.\n\n> W2 Theorem 3.1 assumes a statis fusion weights, yet experiments include dynamic fusion, creating a gap between theory and empirical results.\n\nThanks for your comment. **MNL supports dynamic fusion**, but according to Eq. (4), the dynamic weight $w$ is expected to be positively correlated with $\\xi$. However, since $\\xi$ is difficult to predict in practice, it becomes challenging to adjust the weights accordingly in a truly dynamic manner. Therefore, we adopt a static fusion assumption.\n\nAs indicated by Eq. (4), when $w$ is fixed, the robustness lower bound improves more steadily; when $w$ is dynamic, the robustness gain varies depending on how the weights are set, which may influence the overall performance improvement. This also explains why the benefit of combining our method with existing dynamic fusion approaches such as PDF is somewhat limited.\n\nTo mitigate the influence of dynamic weighting on MNL, we first training a static multimodal fusion system using MNL to enhance the model’s ability to perceive weak modalities, and then applying a dynamic fusion strategy like PDF to further boost overall performance (**MNL (step1) + PDF (step2)**).\n\n> W3 Equation 5 assembles knowledge distillation, where the \"soft labels\" from a teacher (RDM) are used to train a student (IM), diminishing the novelty of the paper.\n\nThanks for your valuable comment. We propose a new MNL learning paradigm inspired by knowledge distillation (KD).\n\nDifferently, KD focuses on training a student model under the guidance of a fixed teacher model, whereas our method focuses on dynamically enhancing the model’s ability to recognize non-target classes of the weak modality by leveraging the dominant modality.\n\nIn addition, beyond the commonly used KL divergence in KD, we also explore using L1 and L2 losses to implement MNL. We believe that our MNL has the potential to be compatible with more collaborative learning paradigms and adaptable to a wider range of tasks.\n\nTo explore this, we extended our experiments by modifying the MNL loss formulation (Eq. (5)) and implemented negative learning using  L1 or L2 losses.\nTable 1 MVSA dataset under Gaussian noise (over 5 seeds)\n\n\n| Method                       | ε = 0 | ε = 5 | ε = 10 |\n| ---------------------------- | ------ | ------ | ------- |\n| STATIC LATE FUSION           | 76.88  | 63.46  | 55.16   |\n| STATIC LATE FUSION + MNL     | 79.50  | 74.03  | 63.01   |\n| STATIC LATE FUSION + MNL(L2) | 79.48  | 73.48  | 62.96   |\n| STATIC LATE FUSION + MNL(L1) | 79.90  | 73.15  | 59.73   |\n\nTable 2 MVSA dataset under Salt noise (over 5 seeds)\n\n\n| Method                       | ε = 0 | ε = 5 | ε = 10 |\n| ---------------------------- | ------ | ------ | ------- |\n| STATIC LATE FUSION           | 76.88  | 67.88  | 55.43   |\n| STATIC LATE FUSION + MNL     | 79.50  | 74.68  | 62.31   |\n| STATIC LATE FUSION + MNL(L2) | 79.48  | 74.59  | 63.37   |\n| STATIC LATE FUSION + MNL(L1) | 79.90  | 73.73  | 59.02   |\n\n> Q1 What are the loss weights in Eq. 7 (is MNL summed equally with CE losses?)\n\nYes, according to our setup, the MNL loss is summed equally with the CE loss. To demonstrate this, we present our results on the MVSA dataset, where the CE weight is fixed at 1 and the MNL weight is varied from 0.2 to 2.0 under different noise levels.\n\n\n| weight   | 0.2    | 0.4    | 0.6    | 0.8    | 1.0        | 1.2        | 1.4    | 1.6        | 1.8    | 2.0    |\n| -------- | ------ | ------ | ------ | ------ | ---------- | ---------- | ------ | ---------- | ------ | ------ |\n| Gauss_0  | 0.7688 | 0.7803 | 0.7919 | 0.7803 | 0.8054     | 0.7938     | 0.7900 | **0.8102** | 0.8035 | 0.7977 |\n| Gauss_5  | 0.7256 | 0.7202 | 0.7341 | 0.7356 | **0.7407** | 0.7364     | 0.7206 | 0.7322     | 0.7380 | 0.7287 |\n| Gauss_10 | 0.6246 | 0.6236 | 0.6266 | 0.6338 | **0.6378** | 0.6346     | 0.6212 | 0.6119     | 0.6374 | 0.6370 |\n| Salt_5   | 0.7495 | 0.7457 | 0.7437 | 0.7495 | **0.7576** | 0.7514     | 0.7572 | 0.7476     | 0.7437 | 0.7526 |\n| Salt_10  | 0.6227 | 0.6309 | 0.6346 | 0.6327 | 0.6493     | **0.6506** | 0.6408 | 0.6350     | 0.6196 | 0.6301 |\n\nSimilar results are observed across multiple datasets. After careful consideration, we set the weights of both losses to be equal at 1.\n\n> Q2 What are the criteria for transitioning Stage 1→Stage 2 training?\n\nThank you for your question. We can provide the following table (CREMA-D dataset, Noise = 0).\n\n\n| Warm-up Epoch | 0     | 2     | 5     | 10    | 15    | 20    | 30    |\n| ------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Accuracy (%)  | 68.55 | 68.11 | 68.11 | 69.18 | 68.79 | 68.26 | 69.05 |\n\nWe observe that the hyperparameter warm-up epoch is not sensitive. We set it to 10.\n\n> Q3 Fig 1(b) shows forced alignment harms weak modalities. Does MNL mitigate such collapse? It will be beneficial to provide examples where MNL fails to preserve weak-modality advantages, and quantify how often this occurs.\n\nThank you for your question. First, regarding whether MNL can mitigate such collapse, the following table presents a comparison of MNL (warm-up epoch = 10) and KL Guidance in terms of the number of Audio-wrong but Video-right samples across different epochs.\n\n\n| Method      | 10  | 11  | 12  | 13  | 14  | 15  | 16  | 17  | 18 | 19 | 20 |\n| ----------- | --- | --- | --- | --- | --- | --- | --- | --- | -- | -- | -- |\n| KL Guidance | 446 | 385 | 324 | 252 | 212 | 137 | 158 | 122 | 77 | 36 | 29 |\n| MNL         | 392 | 300 | 266 | 145 | 128 | 119 | 94  | 28  | 7  | 4  | 2  |\n\nIt is evident that MNL consistently results in significantly fewer audio-wrong but video-right samples compared to KL Guidance at the same training epoch. \n- In KL Guidance, the audio modality (which generally performs better) is used to guide the video modality even when its own prediction is incorrect and the video prediction is correct. This misalignment introduces a conflict between the cross-entropy loss and the KL divergence, often leading to optimization collapse. \n\n- In contrast, MNL dynamically adjusts the guidance direction based on Definition 3.3, thereby avoiding such conflicts.\n\nAs for cases where MNL fails to preserve weak-modality advantages, we take the MVSA dataset as an example. When applying MNL, the guidance direction is determined dynamically based on $P_y$ and $\\xi$(UCoM), and according to Eq. (6), the guiding modality must have both higher $P_y$ and $\\xi$ than the guided one. However, this condition is not always satisfied for every sample. Out of **1,555** training samples in MVSA, we found **143** samples where MNL was inactive (**9.196%**). Among these cases, it is possible that weak-modality advantages existed but were not preserved.\n\n[1]Predictive dynamic fusion. ICML2024"}, {"review_id": "hck7GoXkOf", "reviewer": "Reviewer_v4mL", "summary": "Considering the multimodal imbalance of “Learning to be” in existing methods, this work proposes a new “Learning Not to be” paradigm, which leverages the dominant modality to dynamically guide the weak modality on non-target classes. The authors provided an interesting multimodal negative learning framework, which theoretically tightens the robustness lower bound of multimodal learning by increasing the Unimodal Confidence Margin (UCoM). In addition, the proposed MNL can be flexibly integrated with late fusion methods, such as QMF and PDF, which has the potential to be applied in broader scenarios. The comparisons on noisy data and ablated experiments have validated its effectiveness. Generally speaking, this work contributes new insights to the multimodal learning community.", "questions": "1. What’s the key effect of dynamic guidance on multimodal negative learning? Eq. 6 should be further explained.\n2. Is there any trade-off between multimodal positive learning and multimodal negative learning in Eq. 7?\n3. There are two stages during the training phase. How to determine stages 1 and 2 in practice? More details should be provided. \n4. Although the proposed method can be integrated with existing methods without introducing additional inference overhead, the additional training cost should be reported.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths: \n1. The proposed multimodal negative learning framework is novel and interesting, which changes the commonly used “Learning to be” to “Learning Not to be” paradigm. It allows weak modalities to preserve unique information without being over-aligned, stabilizing the decision space and preserving the specificity of individual modality.\n2. This work introduces a dynamic guidance mechanism from the robustness perspective. The proposed UCoM provably tightens the robustness lower bound of multimodal learning and reduces the empirical error of weak modalities.\n3. This work is verified on multiple benchmarks and has conducted experiments on extensive noisy and imbalanced scenarios, demonstrating its effectiveness and generalization.\n4. This work can be flexibly integrated with many existing methods without additional inference consumption and consistently improve their performance on multiple datasets. \n5. This work provides theoretical proof on robust lower bounds and empirical error in Supp. A, offering theoretical guarantees to mitigate modality imbalance and boost robustness.\n\n### Weaknesses:\n1. The authors conducted extensive experiments, however, the competing method “LATE FUSION” is confusing for me. QMF and PDF also belong to late fusion methods. Therefore, the claimed “LATE FUSION” is not inappropriate.\n2. The authors provide theoretical proof in supplementary material, while some key equations should be presented in the main paper to make it easier to follow. \n3. The presentation can be further improved, especially for the tables and figures. For instance, the font size of Tables 3 and 4 is too large, while the font is too small in Fig. 3. \n4. It is reasonable to consider the dynamic modality dominance in multimodal learning, while a more clear illustration will be helpful.\n5. Since this work can be flexibly integrated with existing methods, I suggest the authors present the pseudo code in supplementary material.", "was_revised": true, "final_justification": "After considering the authors' rebuttal and the other reviewers' comments, I maintain my overall rating of 5.", "rebuttal": "We sincerely thank the reviewer for the valuable comments, the recognition of our method’s novelty and theoretical contributions, and the acknowledgment of its potential benefits to broader multimodal learning research. We believe the constructive feedback will help us further improve the paper and enhance its impact on the community.\n\n> W1 The authors conducted extensive experiments, however, the competing method “LATE FUSION” is confusing for me. QMF and PDF also belong to late fusion methods. Therefore, the claimed “LATE FUSION” is not inappropriate.\n\nThanks for your comment. The \"LATE FUSION\" mentioned here specifically refers to \"STATIC LATE FUSION.\" We have corrected this in the manuscript and will ensure the final version reflects the change.\n\n> W4 & Q1 It is reasonable to consider the dynamic modality dominance in multimodal learning, while a more clear illustration will be helpful. & What’s the key effect of dynamic guidance on multimodal negative learning? Eq. 6 should be further explained.\n\nThanks for your question. The dynamic guidance mechanism is indeed a critical component of MNL. Its core is defined in Definition 3.3, which jointly considers both $P _y$ and the Unimodal Confidence Margin (UCoM) to identify the Robust Dominant Modality (RDM) and the Inferior Modalities (IMs). Specifically, **the RDM is selected such that its $P _y$ and UCoM are both greater than those of the IMs, thereby enabling the guidance direction to be determined dynamically.**\n\nRegarding Eq. (6), its key function is to determine two aspects: the guidance direction, which involves identifying the RDM and IMs (as described in Definition 3.3, with Eq. (6) focusing on the two-modality case), and the guidance scope, which is restricted to non-target classes.\n**If dynamic guidance is absent in multimodal negative learning:**\n\n- A modality that predicts incorrectly might guide a modality that predicts correctly. Even if the guidance is restricted to non-target classes, this can still disrupt the optimization process of the modalities.\n- A modality with a low Unimodal Confidence Margin (UCoM) might guide a modality with a high UCoM, which can potentially harm the robustness lower bound of the multimodal system.\n\n> W5 Since this work can be flexibly integrated with existing methods, I suggest the authors present the pseudo code in supplementary material.\n\nThank you for the excellent suggestion. Below, we provide the pseudocode of the algorithm. The inclusion of this pseudocode will be reflected in the final version of the paper.\n\n```text\nAlgorithm 1: Multimodal Learning with MNL\n\nInput:\n Training set D_train = {(x_i, y_i)}_{i=1}^N ⊂ X × Y\n Iteration number T, warm-up epoch W\n\nfor t = 1 to T do\n    Sample a fresh mini-batch B_t from D_train\n    Feed-forward the batched data B_t to the model\n    Obtain fusion logits and modality-specific logits from the model\n\n    if t > W then\n        Identify RDM and IMs according to Definition 3.3\n        Compute MNL loss according to Eq. (5)\n        Compute total loss according to Eq. (7)\n    else\n        Compute cross-entropy loss\n    end if\n\n    Update model parameters\nend for\n```\n\n> Q2 Is there any trade-off between multimodal positive learning and multimodal negative learning in Eq. 7?\n\nThanks for your question. We present our results on the MVSA dataset, where the CE weight is fixed at 1 and the MNL weight is varied from 0.2 to 2.0 under different noise levels.\n\n\n| weight   | 0.2    | 0.4    | 0.6    | 0.8    | 1.0        | 1.2        | 1.4    | 1.6        | 1.8    | 2.0    |\n| -------- | ------ | ------ | ------ | ------ | ---------- | ---------- | ------ | ---------- | ------ | ------ |\n| Gauss_0  | 0.7688 | 0.7803 | 0.7919 | 0.7803 | 0.8054     | 0.7938     | 0.7900 | **0.8102** | 0.8035 | 0.7977 |\n| Gauss_5  | 0.7256 | 0.7202 | 0.7341 | 0.7356 | **0.7407** | 0.7364     | 0.7206 | 0.7322     | 0.7380 | 0.7287 |\n| Gauss_10 | 0.6246 | 0.6236 | 0.6266 | 0.6338 | **0.6378** | 0.6346     | 0.6212 | 0.6119     | 0.6374 | 0.6370 |\n| Salt_5   | 0.7495 | 0.7457 | 0.7437 | 0.7495 | **0.7576** | 0.7514     | 0.7572 | 0.7476     | 0.7437 | 0.7526 |\n| Salt_10  | 0.6227 | 0.6309 | 0.6346 | 0.6327 | 0.6493     | **0.6506** | 0.6408 | 0.6350     | 0.6196 | 0.6301 |\n\nSimilar results are observed across multiple datasets. After careful consideration, we set the weights of both losses (CE and MNL) to be equal at 1.\n\n> Q3 There are two stages during the training phase. How to determine stages 1 and 2 in practice? More details should be provided.\n\nThank you for your question. We can provide the table (CREMA-D dataset, Noise = 0).\n\n\n| Warm-up Epoch | 0     | 2     | 5     | 10    | 15    | 20    | 30    |\n| ------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Accuracy (%)  | 68.55 | 68.11 | 68.11 | 69.18 | 68.79 | 68.26 | 69.05 |\n\nWe observe that the hyperparameter warm-up epoch is not sensitive (we set it to 10). This is because the benefit of MNL is relatively small in the early stages of training, and becomes more effective as training progresses and the modality gap becomes more pronounced, allowing for more meaningful guidance. Therefore, the warm-up is primarily introduced to reduce unnecessary computational cost during the initial training phase.\n\n> Q4 Although the proposed method can be integrated with existing methods without introducing additional inference overhead, the additional training cost should be reported.\n\nWe are glad to report the additional training cost. Specifically, we calculate the average time per training iteration (per batch), with batch sizes of 16, 16, 32, and 64 for MVSA, FOOD101, NYUDv2, and CREMAD, respectively.\n\n\n| Method                   | MVSA    | FOOD101 | NYUDv2  | CREMAD  |\n| ------------------------ | ------- | ------- | ------- | ------- |\n| STATIC LATE FUSION       | 20.96ms | 69.23ms | 32.27ms | 63.03ms |\n| STATIC LATE FUSION + MNL | 31.15ms | 85.76ms | 49.19ms | 87.11ms |\n\nThese results indicate that the additional overhead introduced by MNL is minimal.\n\n>presentation issues\n\nThank you for pointing out the typo. We have revised the description of “LATE FUSION,” improved the clarity of the figures and tables, and made the equations more comprehensible. Finally, we have carefully reviewed and corrected all representation issues throughout the paper."}]

Author rebuttal:

Meta-review: **Paper Summary:**\
This paper introduces a novel multimodal learning paradigm for addressing modality imbalance in multimodal learning. The concept of 'learning not to be' introduced in the paper is novel, and the experiments provide strong support for the authors’ claims. The paper received five peer review comments, and the authors engaged in several rounds of in-depth discussions with the reviewers. Based on these comments, most issues have been addressed, and the paper received unanimous recommendations for acceptance.

**Justification:**\
Given that the reviewers engaged in in-depth discussions with the authors and unanimously recommended acceptance, I concur with their assessment.

**Summary of Rebuttal Period:**\
Reviewers yEYi, 9fVt, and v4mL offered suggestions regarding the paper’s narrative style. Reviewer AePS raised questions about the experimental setup, while Reviewers AePS and 9fVt expressed concerns about scalability. Reviewer 9fVt also noted issues with complexity, and Reviewers d8o7 and v4mL raised questions about the theoretical foundation. Overall, the paper has been thoroughly discussed, and the reviewers’ concerns have been satisfactorily addressed.
Decision: Accept (poster)