Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion
Authors: Qing-Yuan Jiang
Longfei Huang
Yang Yang
Venue: NeurIPS 2025 oral ()
Keywords: Multimodal Leanring; Modality Imbalance.
Primary area: deep_learning

Abstract: Multimodal learning (MML) is significantly constrained by modality imbalance, leading to suboptimal performance in practice. While existing approaches primarily focus on balancing the learning of different modalities to address this issue, they fundamentally overlook the inherent disproportion in model classification ability, which serves as the primary cause of this phenomenon. In this paper, we propose a novel multimodal learning approach to dynamically balance the classification ability of weak and strong modalities by incorporating the principle of boosting. Concretely, we first propose a sustained boosting algorithm in multimodal learning by simultaneously optimizing the classification and residual errors. Subsequently, we introduce an adaptive classifier assignment strategy to dynamically facilitate the classification performance of the weak modality. Furthermore, we theoretically analyze the convergence property of the cross-modal gap function, ensuring the effectiveness of the proposed boosting scheme. To this end, the classification ability of strong and weak modalities is expected to be balanced, thereby mitigating the imbalance issue. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SOTA) multimodal learning baselines. The source code is available at https://github.com/njustkmg/NeurIPS25-AUG.

Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion
Qing- Yuan Jiang Longfei Huang', Yang Yang!
NNanjing University of Science and Technology
State Key Lab. for Novel Software Technology, Nanjing University, P.R. China {jiangay, hlf, yyang) edu.
Abstract
Multimodal learning (MML) significantly constrained by modality imbalance, leading to suboptimal performance in practice. While existing approaches primarily focus on balancing
learning of different modalities to address this issue, they fundamentally overlook the inherent disproportion in model classification ability, which serves as the primary cause of this phenomenon. this paper, we propose novel multimodal learning approach to dynamically balance the classification ability of weak and strong modalities by incorporating the principle of boosting. Concretely, we first propose a sustained boosting algorithm in multimodal learning by simultaneously optimizing the classification and residual errors. Subsequently, introduce adaptive classifier assignment strategy to dynamically facilitate the classification performance of the weak modality. Furthermore, we theoretically analyze the convergence property of the cross- modsl gap function, ensuring the effectiveness of the proposed boosting scheme. To this end, the classification ability of strong and weak modalities is expected to be balanced, thereby mitigating the imbalance issue. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-ard (SOTA) multimodal learning baselines. The source code is available at wtthttt////gg t/gtttb /eur IPS25-AUS
1 Introduction
In recent years, multimodal learning 33 48 43 47
21] has received growing attention for its ability to effectively integrate heterogencous information. As extra information from multimodal data can be utilized, multimodal learning expected to achieve better performance compared with unimodal approaches. However, contrary expectations, multimodal learning has been surprisingly shown to
underperform compared to unimodal ones in certain scenarios 42 35 46).
The root of this problem lies in the existence of the modality imbalance [42). Concretely, different modalities in a joint-training paradigm typically converge different speeds [35 45 The faster- converging modality, i.e., strong modality (49). tends to achieve higher performance, while the weak modality performs poorly. Subsequently, this disproportion in classification ability often leads to modality imbalance [42) ultimately resulting in lower performance.
Researchers have explored the modality imbalance issue from various perspectives in multimodal learning 42
35
52 Given the inconsistent learning progress between strong and weak modalities, a
50 is to manually intervene in their learning processes achieve rebalancing. Another type of method is to bridge the information gap between modality training phases and enhance the interaction between different modalities during training. To be specific. impressive works [52
natural idea
42
35
28
CCorresponiing author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

such as MLA [52] ReconBoost [2]] and DI-MML []] focus on bridging the learning gap of different modalities through injecting the optimization information between modalities.
Although the above methods can rehalance multimodal learning, they focus more on balancing the learning process while failing enhance the classification ability explicitly. Compared to weaker modalities, stronger modalities typically yield more robust classifiers due to their more sufficient information [49]. Is there a way directly improve the performance of weak classifiers to balance the classification performance between strong and weak modalities? A natural choice is boosting 13 which utilizes the ensemble technique to enhance the ability of the weak classifier. We conduct a toy experiment to illustrate this idea on CREMAD dataset [4) where the classifier of weak modality is enhanced by the gradient boosting [14. The results in Figure present the comparison among naive MML, model learning adjustment-baset MML approach (G-Blend (42)), and gradient boosting- based MML. (MML w/ GB). For MMI. w/ GB, we apply the gradient boosting algorithm to further improve the trained video model using naive MMI., while keeping the audio model fixed. We can find that the classification gap between video and audio modalities of naive MMI.and G-Blend is relatively large. More importantly, for MML w/ GB, the accuracy of audio modality remains unchanged. but the accuracy of video is greatly improved, leading to the improvement overall accuracy. This demonstrates the feasibility and effectiveness of using boosting to balance the classification ability of strong and weak modalities in mitigating modality imbalance.
According to the aforementioned observations, in this paper. we propose a novel multimodal learning approach by designing sustained boosting algorithm to facilitate the classifica- tion ability of the weaker modality. Concretely, we first propose a sustained boosting algorithm by jointly optimizing the classification loss and residual error, aiming to enhance the classifica- tion performance of the weak modality. The al- gorithm takes features extracted by the encoder input and provides predictions for the given data. Then, we employ the confident score [35] to monitor the learning status during joint train- ing, and further propose an adaptive classifier as- signment (ACA) strategy to adjust the classifier of weak modality. To this end, we can enhance the classification ability for the weak modality, thereby rebalancing the classification ability of strong and weak modalities. Meanwhile, we theoretically show that, under the boosting algo- rithm framework, the gap between different losses is provably convergent. In Figure I we present the classification enhancement results of our method (Ours). We can find that the performance of our method outperforms that of MML w/ GB thanks to the sustained boosting and adaptive classifier assignment strategy. Furthermore, it is worth mentioning that ReconBoost (21) also employs the gradient boosting algorithm for multimodal learning. However, unlike our approach, ReconBoost uses gradient boosting to iteratively learn complementary information across modalities. Our main contributions are outlined as follows:
Video;
Audio:
Multi
85.15%
0.8
72.044%
Accuracy
65.07%
64.63%
0.4
Ours Naive MML. G- MML w/ GB
Figure 1:
Comparison with naive MML., gra- dient boosting based MML (MML w/ GB), G- Blend [42), and Ours on CREMAD dataset. We find that enhancing the classification performance of the weak modality narrows the performance gap between the two modalities and improves overall performance.
A novel sustained boosting algorithm in MML is proposed. This algorithm aims to simulta- neously minimize the classification and residual errors to facilitate the classification ability of the weak modality.
A novel adaptive classifier assignment strategy is proposed to dynamically enhance the classification ability of weak modality based on the learning status, thus rebalancing the classification ability of all modalities.
We theoretically analyze the impact of the boosting algorithm on the loss gap between different modalities and prove its convergence.
Experiments reveal that our approach can outperform SOTA baselines to achieve the best performance by a large margin on widely used datasets.
2

---

2 Related Work
2.1 Rebalanced Multimodal Learning
The goal of multimodal learning 48
33 39 22 is to fuse the multimodal information from
diverse sensors. Compared to unimodal methods, multimodal learning can mine data information from different perspectives, thus the performance of multimodal learning should be better [30 38 20 16]
However, due to heterogeneity of multimodal data, multimodal learning often encounters imbalance
problems 42 23 in practice, leading to performance degeneration of multimodal learning.
Early pioneering works 42 35 12 17] focus more on adaptively adjusting the learning procedure for different modalities. Representative approaches in this category employ different learning strategies, c.g., gradient modulation (35) (8) and learning rate adjustment [50], to rebalance the learning of weak
and strong modalities. Other approaches including MLA [52] ReconBoost [1] and IGM 25] take a different path, focusing on enhancing the interaction between modalities to address the modality imbalance problem. For example, MLA [52] designs alternating algorithm to train different modalities iteratively. During the training phase, the interaction is enhanced by transferring the learning information between different modalities. ReconBoost [2]] balances modality learning by leveraging gradient boosting to capture information from other modalities during interactive learning. IGM [55 employs flat gradient modification strategy to enhance the interactive multimodal learning.
The aforementioned methods focus on rebalancing the learning process for weak and strong modalities while failing explicitly facilitate the classification ability of the weak modality. In this paper, we aim to address the modality imbalance issue from facilitating the classification ability of weak modality and rebalancing the classification ability of weak and strong modalities.
2.2 Boosting Method
Boosting algorithm 13
14
29
37 is one of the most important algorithms in ensemble learning.
The core idea of boosting to integrate multiple learners to create a strong learner. Adaboost [3], one of the earliest boosting algorithms, adjusts the weights of incorrectly classified data points, giving more attention to the harder-to-llassify examples each iteration. Gradient boosting [14), on the other hand, builds models in stage-wise fashion, minimizing loss function through gradient descent. iteratively refines the overall model by fitting the negative gradient of the loss function 32 with respect to the model's predictions.
The key advantage of boosting lies in its ability improve model accuracy without requiring complex individual models. Therefore, boosting becomes the natural choice for improving the performance of weak classifiers.
3 Methodology
3.1 Multimodal Learning
For simplicity, we use two modalities, i.e., audio and video, for illustration. It is worth mentioning that our method can be easily adapted to cases with more than two modalities.
Assume that we have N data points, each of which has audio and video modalities. Without loss of generality, we use X {(xf,, ax) }
to denote the multimodal data, where at and ai denote the i-th data point of audio and video, respectively. In addition, we are also given category labels set Y = y. y; E {0, 1 j where K denotes the number category labels. Given the above training information X and Y. the goal of multimodal learning is to train model to fuse the multimodal information and predict its category label as accurately as possible.
For the sake of simplicity, we use superscript to indicate the module corresponding to a specific modality in this section, where
{a, v). With the rapid growth of deep learning, representative MML approaches [33 42 I2 28 have adopted deep neural network (DNN) for multimodal learning. Following these methods, we also utilize DNN to construct our models. Specifically, we use (.) to denote encoders. Then the features can be calculated by u = (0xx;; 0"), where 0 denotes the encoder parameters. Then, the prediction of given data can be calculated by a classifier (+): pd = (u, (P), where O" denotes the parameters of the classifier. Based on p and its ground-
3

---

truth, the objective function can be written as:
( Y) = N Σ Epp.. y:) = N
Σ y log(p;),
(I)
where do _ 10", }r} denotes the parameters to be learned, X {mg)i=) and ((-) denotes the cross entropy loss.
3.2 Sustained Boosting
By training the model for each modality based on objective function ( we can obtain multiple individual classifiers. Due to the existence of strong and weak modalities [49], these classifiers exhibit different classification abilities. Hence, we can employ boosting technique 14] to improve the classification ability of weak modality.
Concretely, assuming the classification performance the o-modality requires improvement, we first apply the gradient boosting algorithm to train classifiers for the o-modality. Since feature extraction focuses on common patterns, we set the encoders of classifiers to shared. Then the j-th classifier can be defined as: Φ & (0", 0 E .1.... mn. ). In practice, we adopt multiple fully-connected layers and nonlinear activation rectified linear unit (ReLU) [ to construct our classification module. This module called the configurable classifier, relatively independent and can be adjusted based on the classification ability. Furthermore, adopt the shared head structure commonly used in to strengthen the interaction between weak and strong modalities during training.
MML 2
Inspired by gradient boosting [14), the classification ability can be facilitated through minimizing the residual error introduced by previous classifiers. Concretely, when learn t-th classifier, the residual labels are defined as:
Vi = y A
Σ y ④ Pij.
where À E [0, is used to soften hard labels utilize y, mask non ground-truth labels to ensure the non-negativity of residual labels. Then the objective function can be defined as follows:
40 o denotes the element-wise product, and we
<lC, (. (p 94),
where pf denotes the prediction obtained by t-th classifier for i-th data point. Since we utilize shared encoder, the encoder will be updated when training the t-th classifier. Therefore, other classifiers must be updated simultaneously to prevent performance degradation. The corresponding objective can be formed as:
t-1 pijs yi
call ( V..t) =
=
Σ j=1
ply ys
j=1
Meanwhile, we have to ensure the first following objective for 1 classifiers:
classifiers are well-trained. Hence, we define the
Pij. (
(pre(x), U,, =
(4)
By combining (2). 3 and (4, the objective can be defined as:
L( y. t) <(x)), vy. + <x ; De ( Epre (x) y/. f).
5
Unlike traditional gradient boosting [14] our method sustainedly minimizes classification and residual eтrors by optimizing (5). The loss function of sustained boosting can be formed as:
N
LSUB X YY,Y,n
Lax y. n),
(6)
where n denotes the number of classifier for o-modality.

---

Algorithm Learning algorithm of our proposed method.
Require: Training data X, category labels Y.
Ensure:
The learned DNN models for all modalities.
1: INIT Initialize the number of classifier n² = 1, n 1. Initialize iteration 1. Initialize
DNN parameters & and -
2: for tort-a 1 #iterations do
3: Sample a mini-batch X, = ((rf. ]
Learn MML models.
4:
Vaz, ". E X,, calculate features ut and u
5:
Calculate predictions {pl 1 and {p In 1j=1-
6:
Calculate loss in (6) based on predictions;
7:
Update DNN parameters t1 and tf based on SGD;
8:
if mt I, (v)
then Calculate confident score (58, " based on predictions;
Adaptive Classification assignment strategy.
9:
10:
if of as; then
11:
Add classifier for audio modality;
12:
n = n2 +1;
13:
else if st Js0 then
14:
Add classifier for video modality;
15:
n" = n" + 1;
16: end
17: end if
18:
end for
3.3 Adaptive Classifier Assignment
Thus far, we have defined a configurable classifier module and designed a sustained boosting in multimodal learning enhance the classification performance of weak modality. However, recent studies (35] have shown that differences between modalities evolve dynamically due to imbalance issues in multimodal learning. This implies the need to design strategy for enhancing classification ability that adapts to dynamic changes. Hence, we propose an adaptive classifier assignment strategy adjust the number of the weak classifier. For simplicity, we redefine the modality classifiers as: where n is the parameter be updated.
Φ (00, 07), L {1, n²),
Then, we utilize confident score to monitor the learning status. At t-th iteration, confident score can be calculated by:
Yo E {a.u). Σ yi E p
The confident score reflects the classification ability of the models. Hence, if st OS T, we assign a new configurable classifier for video modality at this iteration, where o 2 1 is the coefficient. т is the dead zone for fault tolerance. On the contrary, we also assign a new configurable classifier for audio modality if s asa <
Our algorithm is summarized in Algorithm (1). In practice, perform adaptive classification assignment strategy to determine if we need to adjust the classification ability every i iterations.
3.4 Theoretical Analysis
The sustained boosting algorithm introduced reduce the loss gap between different modalities. In this section, we theoretically analyze its effect minimizing this gap. We first define the gap function as follows:
(( = (d L (d)).
where d and 4 respectively denote the parameters audio and video modality, L and L" denote the overall loss function LsuR for audio and video, respectively. Without loss of generality, assume that L > L" the case where _ < L. can be analyzed analogously.
We derive the following conclusions:
5

---

Theorem (Convergence of Gap Loss, Informal) Under some assumptions for the loss function and the effectiveness of sustained boosting algorithm, we have:
1
( <
v2 1+ 2L, (9()(0)) ((*(0))
(8)
where U, Hi Lig and [3 are constant.
The results in Theorem 1 indicate that, by employing the gradient boosting algorithm, the loss gap between modalities converges at a rate of (I)TT). The proof can be found in the appendix.
4 Experiments
4.1 Experimental Setup
We carry out the experiments on six extensive multimodal datasets, i.e., CREMAD 4 KSounds 2) NVGesture [I], VGGSound [6], Twitter [51], and Sarcasm [3] datasets. The CREMAD, KSounds, and VGGSound datasets consist of audio and video modalities. NVGesture dataset contains three modalities, i.e., RGB, optical flow (OF), and Depth. Twitter and Sarcasm datasets consist of image and text modalities.
Dataset:
To ensure a fair comparison, we strictly follow the data partitioning strategy adopted by representative methods 42 35 522 (2)) Specifically, the CREMAD dataset contains 7,442 clips, which are divided into training set with 6,698 samples and testing set with 744 samples. For KSounds dataset, which contains 19,000 video clips, is divided into training set with 15,000 clips, validation set with 1, 1,900 clips, and testing set with 1, 1,900 clips. VGGSound dataset includes 168.618 videos for training and validation, and 13,954 videos for testing. The NVGesture dataset is divided into 1,050 samples for training and 482 samples for testing. Twitter dataset is divided into training set with 3,197 pairs, validation set with 1,122 pairs and testing set with 1,037 pairs. Sarcasm dataset includes 19,816 pairs for the training set, 2,410 pairs for the validation set, and 2,409 pairs for the testing set. More details are provided in the appendix.
We select two categories of methods for comparison, i.c.. traditional multimodal fusion methods and rebalanced multimodal learning methods. In detail, traditional multimodal fusion meth- ods include concatenation fusion (Concat), affine transformation fusion (Affine) [36], multi-
Baselines:
Istm fusion (ML-LSTM) [34), prediction summation fusion (Sum) 48 and prediction weighting
fusion (Weight) 48
Rebalanced multimodal learning methods include MSES [15]. G-blend [4)
MSLR 50
OGM 35), PMR [2), AGM 28 MMParato [44, SMV [4] MLA 52 DI-MML
LFM
47
ReconBoost
21
Evaluation Protocols:
Following the setting of MLA [52] and ReconBoost 21] we adopt accuracy.
mean average precision (MAP) and MacroFl as evaluation metrics. The accuracy measures the proportion of correct predictions of total predictions. MAP returns the average precision of all samples. And MacroFl calculates the average Fl across all categories.
Following OGM 33 we employ ResNet18 [19] as the backbone encode audio and video for CREMAD, KSounds and VGGSound datasets. All the parameters of the backbone are randomly initialized. For NVGesture dataset, we employ the I3D (5) unimodal branch following the setting of [45] We initialize the encoder with the pre-trained model trained on ImageNet. For the architecture of the configurable classifier, explore two-layer network, which be denoted as "Layerl (D) x 256) + ReI.U - Layer2 (256 x K)". Here, 1) denotes the output dimensions of encoders, "Layerl"/ Layerr2 are fully connected layer, and "ReLU" denotes the ReLU [T] activation layer. Furthermore, the Layer2 is utilized as shared head for all modalities as described in Section 3 Both Layerl and Layer2 are randomly initialized. In addition, all haper-parmmeters are selected by using the cross-validation strategy. Specifically, we use stochastic gradient descent (SGD) as the optimizer with momentum of 0.9 and weight decay of 1 x 10-4, The initial learning rate is set to be 1 x 10 for CREMAD, KSounds, VGGSound. and NVGesture datasets. During training, the learning rate progressively reduced by a factor of ten upon observing loss saturates. The batch size is set to be 64 for CREMAD and KSounds datasets, 16 for VGGSound dataset, and 2 for NVGesture dataset. We set the iteration y for checking whether to assign the classifier to 20 epochs for CREMAD, 5 for Twitter, I for Sarcasm, and 10 for VGGSound, KSounds, NVGesture datasets.
Implementation Details:

---

Table 1: Classification performance comparison with SOTA baselines. The best and second-best results are highlighted in bold and underline, respectively. The results with gray background denote the performance based on multimodal learning is inferior to that of the best unimodal approach.
CREMAD
KSounds
VGGSound
Twitter
Sarcasm
NVGesture
Method
Acc.
MAP
Acc.
MAP
Acc.
MAP
Acc.
F1
Acc.
F1
Acc.
F1
Unimodal-1
.4583
.5879
.5412
.5699
.4655
.4701
.5863
.4333
.7181
.7073
.7822
.7833
2
.6317
.6861
.5562
.5337
.3994
.3478
.7367
.6849
.1366
.8056
.7863
.7865
3
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
.8144
.1883
Concat
6361
.6841
.6455
.7130
.5116
.5352
.7011
.6386
.8286
.2400
.8237
.8270
Affine
66266
.7193
.6424
.6931
.5001
.5155
.7203
.5992
.2400
8188
.2788
.8281
ML-LSTM
6290
6673
.6394
.6902
.4966
.5399
7068
6564
.8777
8205
.8320
.8300
Sum
6344
69088
.6900
.7103
.1336
.5338
.3000
.6611
8294
.8447
.8050
80667
Weight
.6533
.7134
.6533
.7110
.5144
.5300
.7242
.6516
.8265
.8219
.7842
.7939
MSES
.6466
.7138
.6591
.7196
.4891
.5429
.7252
.6439
.8423
.8369
.8112
.8147
G-blend
.6465
.7392
.6722
.7274
.5866
.5555
.7309
.6799
.2866
.8215
.8299
.8305
MSLR
.6868
.7412
.6756
.7282
.4987
.5415
.7232
.6382
.8439
.8378
.8237
.2844
OGM
.6612
.7372
.6582
.7159
.8299
.4978
.7588
.6.35
.3660
.8293
PMR
66559
.7058
.7274
.4647
.4866
.7357
66.36
.2.56
AGM
.6333
.7807
.6791
.7388
.4711
.5198
.7261
6502
.3066
.8293
.8279
.8284
MMParato
.7487
.8355
.7000
.7850
.5125
.5473
.7358
.6229
.8348
.8284
.8382
.8424
SMV
.7872
8417
.6900
.7426
.5031
.5362
.7428
.68 17
.8418
.3368
.8352
.8341
MLA
.7433
.8722
.7004
.7445
.1165
.5473
.7352
.6113
.8426
.348
.8400
.8372
DI-MML
8158
8592
.7203
.7426
.5733
.5479
.7248
.6866
8411
8315
LFM
8362
9006
.2533
.7897
.5274
.5694
.7501
.7057
8497
.8457
.8436
.868
RecunBoost
.7577
8140
.6855
.7662
.5097
.5387
.7442
.6832
8437
.317
.8386
.834
.8515
.9103
.7263
.7901
.5301
.5826
.7512
.6622
.8510
.8458
.8501
.8533
Ours
200001
2:000004
20.0011
2:00063
1:00006
2.00017
-0,0088
1:000166
1:00054
1:00007
200005
20.011
For all datasets, we search À in (0.1,0.2, 0.33, 0.5, 1.0]. For all datasets, and T are set to be 1.0 and 0.01, respectively. For Twitter and Sareasm dataset, following [51 31 we adopt BERT ( as the text encoder and ResNer50 [19] as the image encoder. We use Adam (26 as the optimizer, with an initial learning rate of 2 x 10-3. The batch size set to 32 for Twitter and Sarcasm datasets. The other parameter settings are the same as audio-video datasets. For comparison methods, the source codes of all baselines are kindly provided by their authors. For fair comparison, all basclines also adopt the same backbone and initialization strategy for the experiment. All experiments are conducted on an NVIDIA GeForce RTX 4090 and all models are implemented with pytorch.
4.2 Main Results
Classification Performance Comparison: The classification results on all datasets reported in Table where denotes that corresponding methods cannot applied to the dataset with more than two modalities. And Unimodal- 1/2/3 is used denote the results based unimodal. Unimodal-1/2 respectively denote the video/audio for CREMAD and KSounds, and text/image for Twitter and Sarcasm. Unimodal-1/2/3 denotes the RGB/OF/Depth modality for NVGesture dataset, respectively. Furthermore, the results with a gray background indicate that the performance based on multimodal learning is inferior to that of the best unimodal approach. We can draw the following observations: (I). Compared with unimodal baselines, traditional multimodal fusion methods and rebalanced multimodal learning methods can achieve better performance in almost all cases; (2). Our method can outperform existing SOTA baselines to achieve the best accuracy in all cases for multimodal situations: (2). The accuracy on NVGesture dataset demonstrates that our method can extend the case with more than two modalities and achieve the best performance.
4.3 Sensitivity to Hyperparameter
We explore the influence of a and À on CREMAD and KSounds datasets this section, More results can be found in appendix.
7

---

0.
0.5
Aoousce
Prodatia AVlle
02
02
0-
125
1.5
1.76
0.102 0.8
0.5
Orpecoss
190
Figure 2: Sensitivity to (left) and X (right).
Figure 3: Performance comparison.
Table 2: The accuracy (Multi/Audio/Videe) for ab- lation study on CREMAD and KSounds datasets.
Table 3: The impact of weak classifier
selection strategy.
CREMAD
KSounds
#Classifter
Accuracy
Straregy
f
x
v
0.83330.646570.6334
0.694200.303300500
Audio
Viden
Multi
Audio
Video
X
v
x
O. 0.83200.657300.6700
0.719000.50330.5144
Fixed
10
0,8091
0,6774
0.6156
X
V
V
O. 0.8360066841/0.6677
0.7198/0.5261/0.523.
Fixed
0.8118
0.6519
0.6277
V V V
0.881550.6833/066622
0.776300.525770.573
Adaptive
0.8515
0.6828
Sensitivity to Threshold a: We study the influence of threshold a on CREMAD and KSounds datasets. The accuracy with different [1, 1.75) is shown in Figure We can find that our method is not sensitive to threshold a in a large range.
Sensitivity to Smoothing Factor X:
We explore the influence of smoothing factor À CREMAD and
KSounds datasets. The accuracy with different E [0.1, 1] is reported Figure 2
We can find that method is not sensitive to parameee smoothing factor À in large range.
4.4 Ablation Study
We investigate the effectiveness of our method by analyzing the influence of the key components
of our objectives Equation (2) 33. and 44), respectively denoted as e, co. and Ey. The accuracy results on CREMAD and KSounds datasets are reported in Table 2 From Table we can find that: (1). Both objectives in Equation (2). 3 and (4) can boost multimodal performance; (2). While the unimodal performance of the method using all objectives may not always reach the highest level, it achieves a more balanced classification performance across modalities. More results are reported in appendix.
We further investigate the impact of residual learning classification performance by comparing the performance of all f classifiers with that of the first classifiers during the training. The results are presented in Figure where the former accuracy is denoted as "Full Prediction" and the latter is denoted as "Prediction of t - CLS". In Figure 3 we also present the number of the video classifier. We observe that the number of classifiers for the video modality has increased, and the performance of all t classifiers is generally superior to that of the first t classifiers. This performance gain arises from our learning of the residual objective.
4.5 Further Analysis
Impact of Weak Classifier Assignment Strategy: We conduct experiment to study the influence of adaptive classifier assignment strategy. Specifically, we design fixed classifier assignment strategy for comparison. This approach allocates (f(x) classifiers for weak modality during the init stage. And we no longer dynamically adjust the number of classifiers during training for weak modality.
The results on CREMAD dataset are reported in Table 3 where (fxx) is set to be 10 and 12. The results in Table 3 demonstrate that our proposed adaptive classifier assignment strategy can boost performance compared with fixed classifier strategy. This is because our method dynamically adjusts modality classification performance in response to modality imbalance during training.
We further study the property embeddings through visualization. Specifi- cally, we illustrate the t-SNE 41 results on CREMAD dataset for naive multimodal learning (naive
Visualization Results:

---

Audio
Audio
Audio
(a). naive MML @ Audio. Video
(b). ReconBoost @ Audio.
(c). Ours @ Audio.
Video
Video
(d). naive MML @ Video.
(e). ReconBoost @ Video.
(I). Ours @ Video.
Figure 4: Visualization on CREMAD dataset. The video visualization highlights the need to improve
weak modality classification.
0.3
0.8
Ville Zans
01.5
Aocuney
0.7
Los wate:
0..6
*
Maltiple
0.5
Nate PARR
9.4
Aydio
Video
AGM
Cap Amonary
0.5
RevenBookt
Ouzz
0.4
100
150
200
50
100
160
200
Figure 5: Training time (hrs).
Figure 6: Loss visualization.
Figure 7: Accuracy change.
MML), ReconBoost (21), and our method in Figure From Figure 4 we can find that: (1). Com- pared to naive MML, our method and ReconBoost can Tearn more discriminative multimodal features, both approaches enhance the weak modality using information from the strong modality; (2). Compared to ReconBoost, our method demonstrates sigmificantly superior classification performance on the video modality, with several distinct categories highlighted by circle markers in Figure 4 (f). This improvement is primarily attributed to our explicit enhancement of the classification capabilities of the weaker modality.
Training Overhead: Considering that our method introduces additional computational overhead due to the use of the sustained boosting strategy, compare its training with that of competitive SOTA baselines, including Naive, PMR, AGM, MLA, and ReconBoost, through empirical experi- ments under the same setting. The results shown in Figure It can be observed that our method achieves the best accuracy while maintaining competitive training time.
Convergence of Gap Loss: To further validate the convergence of the gap function, conducted experiments on the CREMAD dataset. The change of gap function (( during training process is reported in Figure 6 where also report the unimodal loss and multimodal loss. It can be observed that as training progresses, the gap function gradually converges. Moreover, we report the accuracy changes during training in Figure 7 From Figure 7 can be seen that the accuracy exhibits similar convergence trend.
Impact of the Model Capacity: Since our method utilizes multiple designed classifiers for weak modality, it essentially utilizes more model parameters than baselines. Does our method benefit solely from having more model parameters? We conduct an experiment to verify this. Specifically,

---

Table 4: The impact of model capacity.
Table Performance comparison scenario with modality missing.
Arch.
#params.
Method
Ace.
Audio
Video
Audio
Video
Method
rate = 0%
rate = 20%
rate = 50%
Unimodal
R18
11.8M
0.4718
Naive MML
0.6507
0.5849
0.5242
Unimodal
23.3M
RoconBoost
0.7557
0.6321
0.5568
Naive
R18
RI8
11.8
11.8M
0.6507
MLA
0.7943
0.6935
0.5753
Naive
R18
R34
11.M
23.3M
0.6277
Ours
R18
R18
11.8M
12.8M
0,8515
Ours
0.8515
0.7540
0.6008
replace backbone ResNet18 8 (R18) of weak modality video a larger backbone, i.e., ResNet34 (R34). Then we run the baselines and our method on CREMAD dataset. The results are shown in Table From Table 4 we can find that our approach improves accuracy by nearly 20% with only an additional 1M parameters compared to naive MML with the same network architecture. Our method also outperforms the naive MML baseline with larger backbone. Furthermore, we observe an interesting and counterintuitive phenomenon. That is, the method with ResNet34 is worse than that with ResNet18. The reason behind this may be that the ResNet34-bar -bsee method more difficult to converge.
Stability under Modality Missing: Our method is adaptable scenarios involving missing modal- ities. We comprehensively evaluate its performance under test-time missing modality conditions. Specifically, test-time missing refers cases where the modalities are complete during training but missing during the testing phase. The experiments are conducted on CREMAD dataset with different missing rate (52). Naive MML, ReconBoost [2T]. and MLA [52] are selected as baselines for compar- ison, where MLA introduces specifically designed algorithms to address the corresponding challenges modality missing. We report the results with missing rate 20% and 50% in Table We can find that as the modality missing rate increases, the performance of all methods declines. Nevertheless, our method consistently achieves the best performance under all missing rates, demonstrating the effectiveness of our method in scenario with missing modality.
Conclusion
To address the modality imbalance issue, we propose a novel multimodal learning approach by designing a sustained boosting algorithm to dynamically enhance the classification ability of weak modality. Concretely, we first propose sustained boosting algorithm for multimodal learning by minimizing the classification and residual errors simultaneously. Then, we propose an adaptive classifier assignment strategy to dynamically facilitate the classification ability of weak modality. The effectiveness of the proposed boosting algorithm is theoretically guaranteed by analyzing the convergence properties of the cross-modal gap function. To this end, the classification ability can be rebalanced adaptively during the training procedure. Experiments on widely used datasets reveal that proposed method can achieve state-of-the-art performance compared with various baselines by large margin.
Limitations: Our proposed method mainly focuses on the classifier of each modality. For early fusion MML, our method can extend to balance the strong, weak, and fusion classification abilities. In addition, our theoretical analysis only examines the effect of the boosting algorithm the convergence of the cross-modal gap function. In the full iterative framework, the overall convergence behavior and its influence on the model's 's learning capability warrant further investigation. We leave a more comprehensive investigation as future work.
Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key R&D Program of China (2022YFF0712100) in part by the NSFC (62276131, 62506168), in part by the Natural Science Foundation of Jiangsu Province of China under Grant (BK20240081, BG2024042, BK20251431) and in part by the Fundamental Research Funds for the Central Universities (No.30925010205),
10

---

References
[1] A. F. Agarap, Deep learning using rectified linear units (rclu). CoRR, abs/ 80388833355555 2018.
[2] R. Arandjelovic and A. Zisserman. Look, listen and learn. In ICCV, pages 609-617. IEEE, 2017.
[3] Y. Cai, H. Cai, and X. Wan. Multi-modal sareasm detection twitter with hierarchical fusion model. In ACL, pages 2506-251 2515, 2019.
H. Cao, G. Cooper, M. K. Keutmann, Gur, Nenkova, and R. Verma. CREMA-D: crowd-sourced emotional multimodal actors dataset. TAC, 5(4):377-390, 2014.
[5] J. Carreira and A. Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR, pages 4724 4733. Computer Vision Foundation IEEE, 2017.
[6] H. Chen, W. Xic, A. Vodaldi, and A. Zisserman. Vggsound: A large-scale audio-visue dataset. In ICASSP, pages 721-725. IEEE, 2020.
[7] J. Chen. H. Mao. W. L. Woo, and X. Peng. Deep multiview clustering by contrasting cluster assignments. In ICCV, pages 16705-16715. IEEE, 2023.
[8] C. Cortes, M. Mohri, and U. Syed. Deep boosting. In ICMI, pages 11 1179-1189 187. PMIR, 2014.
[9] Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training deep bidirectional transformers for language understanding. In NAACL, pages 4171-4186. Association for Computational Linguistics, 2019.
[10] W. et al. What makes training multi-modal classification networks hard? In CVPR, 2020.
[11] Y. Fan, W. Xu, H. Wang, J. Liu, and S. Guo. Detached and interactive multimodal learning, In ACM MM. ACM, 2024.
[12] Y. Fan, W. Xu, H. Wang. J. Wang, and S. Guo. PMR: prototypical modal rebalance for multimodal lcarning. In CVPR, pages 20029-200000000000 20038. Computer Vision Foundation/ / IEEE, 2023.
[13] Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting, In EurCCOLT, volume 904 of Lecture Notes in Computer Science, pages 23-37. Springer. 1995.
[14] J. H. Friedman. Greedy function approximation: gradient boosting machine. AS, pages 1189 1239, 2001.
[15] N. Fujimori, R. Endo, Y. Kawai, Mochizuki. Modality-speciff learning rate control for multimodal classification. In ACPR, volume 12047, pages 412-422, 2019.
[16] R. Gao. Oh, K. Grauman, and L. Torresani. Listen look: Action recognition by previewing audio. In CVPR, pages 10454-10464, Computer Vision Foundation IEEE, 2020.
[17] Y. Ge, J. Ren, A. Gallagher, Y. Wang, M. Yang, H. Adam, L. Itti B. Lakshminarayanan, and J. Zhao. Improving zero-shot generalization and robustness of multi-modal models. In CVPR, pages 1010-11101. Computer Vision Foundation IEFF, 2023.
[18] L. Goncalves and C. Busso. Auxformer: Robust approach audiovisual emotion recognition. In ICASSP. pages 7357-7366 7361, IEEE, 2022.
[19] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages 770-778. Computer Vision Foundation IEEE, 2016.
[20] D. Hu, X. Li, and X. Lu. Temporal multimodal learning in audiovisual speech recognition. In CVPR, pages 3574-3582. Computer Vision Foundation / IEEE, 2016.
[21] C. Hua, Q. Xu, S. Bao, Z. Yang, and Q Huang. Reconboost: Boosting can achieve modality reconcilement. In ICML. PMLR, 2024.
[22] Y. Huang, C. Du, Z. Xue, X. Chen, H. Zhao, and L. Huang. What makes multi-modal learning better than single (provably). In NewrIPS, pages 10944-10956, 10956, 2021.
[23] Y. Huang, J. Lin, C. Zhou, H. Yang, and I.. Huang. Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably). In ICMI, volume 162 of Proceedings of Machine Learning Research, pages 9226-9259. PMLR, 2022.
[24] C. Jia, Y. Yang. Y. Xia, Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung. Z. Li, and T. Duerig. Scaling up visual and languae representation learning with noisy text supervision. In ICML, pages 4904 4916. PMLR, 2021.
11

---

[25] Q. Jiang, Z. Chi, and Y. Yang. Interactive multimodal learning via flat gradient modification. In IJCAI, pages 5489-5497. ijcai.orr. 2025.
[26] D. Kingma and J. Ba. Adam: method for stochastic optimization. In ICLR. OpenReview.net, 2015.
[27] K. Kontras, T. Strypsteen, C. Chatzichristos, P. P. Liang, M. B. Blaschko, and M. De Vos. Balancing multimodal training through game-theoretic regularization. NewrIPS, 2025.
[28] H. Li, X. Li, P. Hu, Y. Lei, C. Li, and Zhou. Boosting multi-modee modal model performance with adaptive gradient modulation. In ICCV, pages 22157-22167. IEEE, 2023.
[29] P. Liu. S. Han, Z. Meng, and Y. Tong. Facial expression recognition via boosted deep belief network. In CVPR, pages 1805-1812. Computer Vision Foundation / IEEE, 2014.
[30] F. Lv, X. Chen, Y. Huang, L. Duan, and G. Lin. Progressive modality reinforcement for human multimodal Computer Vision
emotion recognition from unaligned multimodal sequences. In CVPR, pages Foundation IEEE, 2021.
[31] P. Molchanov, X. Yang, S. Gupta, Kim, S. Tyree, and J. Kautz. Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural networks. In CVPR, pages 4207-4215. Computer Vision Foundation/ / IEEE, 2016.
[32] A. Natekin and A. C. Knoll, Gradient boosting machines, a tutorial. FINR, 7:21, 2013.
[33] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Loe, and A. Y. Ng. Multimodal deep learning. In ICMI, pages 689-696. Omnipress, 2011.
[34] W. Nie, Y. Yan, D. Song, and K. Wang. Multi-modal feature fusion based on multi-layers LSTM for video emotion recognition. MTA, 10211)::600-11614, 2021.
[35] X. Peng. Y. Wei. A. Deng. D. Wang. and D. Hu. Balanced multimodal learning via on-the-fly gradient modulation. In CVPR, pages 1228-8237 Computer Vision Foundation IEEE, 2022.
[36] E. Perez, F. Strub, H. Vries, Dumoulin, and A. C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, pages 3942-392299 3951, AAAI Press, 2018.
[37] L. O. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Dorogush, and A. Gulin. Catboost: unbiased boosting with categorical features. In NeurlPS, pages 6639-6649, 2018.
[38] K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in vidcos. In NeurIPS, pages 568 576, 2014.
[39] A. Singh, R. Hu, V. Goswami, G. Cousiron, W. Galuba, M. Rohrbach, and D. Kiela. FLAVA: A foundational language and vision alignment model. In CVPR, pages 15617-15629. Computer Vision Foundation / TERE, 2022.
[40] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Wojna. Rethinking the inception architecture for computer vision. In CVPR, pages 2818-2826. IEEE Computer Society, 2016.
[41] L.. Van der Maaten and G. Hinton. Visualizing data using t-sne. JMLR, 911)), 2008
[42] W. Wang, D. Tran, and M. Feiszli. What makes training multi-modal classification networks hard? In CVPR, pages 12692-12702. Computer Vision Foundation / IEEE, 2020.
[43] Y. Wei, R. Feng, 7. Wang, and D. Hu. Enhancing multimodal cooperation via sample-level modality valuation. In CVPR, pages 27338-27347 IEEE, 2024.
[44] Y. Wei and D. Hu. Mmpareto: Boosting multimodal learning with innocent unimodal assistance. In ICML, Proceedings of Machine Learning Research. PMLR, 2024.
[45] N. Wu, Jastrzebski, K. Cho, and Geras. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In ICML, volume 162, pages 24043-222 24043-24055. PMLR, 2022.
[46] Y. Yang, H. Pan, Q. Jiang. Y. Xu, and J. Tang. Learning to rebalance multi-modal optimization by adaptively masking subnetworks. TPAMI, 7(6))4553-4566 2025.
[47] Y. Yang. F. Wan. Q. Jiang, and Y. Xu. Facilitating multimodal classification via dynamically learning modality gap. In NewrIPS, 2024.
12

---

[48] Y. Yang. K. Wang, D. Zhan, H. Xiong, and Y. Jiang. Comprehensive semi-supervised multi-modal learning. In LJCAI, pages 4092-4098 ijeai.org, 2019.
[49] Y. Yang, H. Ye, D. Zhan, and Y. Jiang. Auxiliary information regularized machine for multiple modality feature learning. In IJCAI, pages 1033-1039, AAAI Press, 2015.
[50] Y. Yao and R. Mihalcea. Modality-specific learning rates for effective multimodal additive late-fusion. In ACL, pages 1824-1834, 2022.
[51] J. Yu and J. Jiang. Adapting BERT for target-oriented multimodal sentiment classification. In IJCAI, pages 5408-5415 5414. ijeai.org, 2019.
[52] X. Zhang, J. Yoon, M. Bansal, and H. Yao. Multimodal representation learning by alternating unimodal adaptation. In CVPR, pages 27456-27466. IEEE, 2024.
13

---

Reproducibility Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
Main contributions and scope were reflected in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the work was discussed in Section 3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

The theories and Corollaries presented in the paper are supported by proofs provided Appendix A
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
we provide multimodal datasets and introduce the baseline methods, evaluation metrics, and implementation details.
Justification: As detailed in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We release code at httttss: /ttttps::/github http//uuuuunn//euurSSSS2555 Furthermore, all datasets we used in this paper are available online based on their corre- sponding paper.
Neur IPS25- AUG
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc//ppp wttps/cccpccggguuddeeeeeeeeeee/e haabll/c/guidee/odeeuuubiii
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
All implementation details were provided in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results in this paper are obtained by conducting experiments on three random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resource information for all experiments was provided in Section
4.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https:
Answer: [Yes]
Justification:
We have checked the NeurIPS code ethics for our paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper does not deal with this aspect of the problem.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not deal with this aspect of the problem.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets (datasets, code, models) used in the paper open source, and our use follows the relevant protocols.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code we released in this paper contains related documents.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper doesn't 't involve the crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper doesn't 't involve the potential risks.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: Not used at all (you can then skip the rest).
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pecttp::/neurriie http///rriiicc//ooofeereeee////s cc//oonferencee//222//LL. for what should or should be described.
20

---

A Proof
During training process, the gradient boosting strategy will be applied when the performance gap larger than threshold at each iterations. Now we analyze the convergence rate of G with gradient boosting strategy.
Furthermore, we make the following assumption:
Assumption (Strongly Convexity) We assume that the function (2) j-strooggyy con- vex. That is, for any & we have:
Vo E fa, 2
2 ["((")++++++
Ho
| - 1011
(9)
2
and
Vo E (a [VC2 > (0())
(10)
Assumption (Smoothness) We assume that the functions (G(-), La (1), f.)()) are L-smooth. That is, for any T Φ,, have
Lo |1 12
Vo E {a, v). _ ( ("
(VL (0)) 4¹)
(11)
and
Lg . 3'112.
( 9 S (V9 ( $ Φ¹)
(12)
Assumption (Effectiveness of Weak Classifier) We assume that the gradient boosting strategy for weak modality is effective. That is, there exists v E (0, 1):
(VL" (*(t)) ((T(()))) - ~- L (I.
(13)
And there exists constant $ > 0:
[|((((t)| S V (o(t)) II.
(14)
where fi(-) denotes the function obtained by boosting algorithm. f(-)) aims to fit the negative gradient of weak modality model.
Based the definition of h(-), the updating rule can be formed as:
Φ + 1) (t) ()((((())
(15)
= (t) + (b(())t)
(16)
where 77 denotes the learning rate. Assumption (Optimal of Multimodels) We assume that there exists an optimal solution 4 so that we have: (L" VL"
Assumption 5 (Bounded Initial Value of G()(())) We assume that the initial value of G(b(()) is bounded, i.e., (0) I 2c.
Based on Assumption
we have the following lemma:
Lemma (Gap Bound) There exists constant such that the loss gap function g(-) and the gradient norm of the strong modality satisfy the following relationship:
VL) 2 klg(()))-
(17)
Proof (Proof of Lemma] I Since (-)- and L" (-) are strongly convex, there exists an constant p E (0, 1) satisfying:
[()(t) Φ- < (()) Φ
(18)
21

---

Since the smoothness, we have:
L
A (0))),
(t) 501
(19)
2
L
- 2
p²1 (0)) 111
(20)
Then we have:
((+(t)) - ((0))
(21)
(22)
S -=0( -000) ((((0)) (0))))
(23)
La + Le)
<
p 10)) - $'1?
(24)
2
2
La Le
[| - 2-12
(25)
2
By setting c c= La +
(Z
Iu)
[0) - & 1'1 we have:
(g( cc
(26)
Then we have:
(9(0(1)) 1 ((0(1)) c
(27)
According to Assumption
D. we have:
[VC pa (
(28)
Suppose that two modalities achieve the same optimal value, i.e., L (D6")) = (Tb". Then we have:
VVL' 2 ( (2'))
(29)
(Q(t) L (2')
(30)
2 (0(1) - --"((()))
(31)
2n(5(0(1))
(32)
2 24r (((()))
(33)
By setting K
2p we have:
(V 2 x()()(0)
(34)
We have the following theorem:
Theorem (Convergence of G with Gradient Boosting) Under assumption 2 and
3 if the
learning rate is set as 17+ L. we have:
G(+(T)) - 1+ dTG()(0) O
- -
((O(0)) =
(35)
where d u²²²² 2L.07
Proof 2 (Proof of Theorem
I Based on smooth assumption, we have:
Lu 11) (0)1,
(36)
2
Lan?
= + [|((((t))I?
(37)
2
13
Lani |I(((tt))1?
('(((t))- (V))
(38)
2
14
(m)t))))-mm//V + Lan 32
2 (()))())))
(39)
22

---

Considering that the parameters for each modality are independent, we have: (" + 1)) (*)(t)). Then we have:
Lonk
G(0(t+ = (ur
(
(40)
2
By setting of Leg we have:
v²
+)) 1()
V (L
(41)
2L 3a
By using lemma 1 we have:
x²x²
(( + =
2L (
(42)
Then we set d = 2L.32
g( + 2 ((((t)) d
(43)
z (( (1 dg(((())
(44)
g(((1)) d
(45)
By summing up for 1,... T. we have:
(( 0
((b(T)) < 1+ dTG(((())
(46)
B Notation Definition
We summarize the notation definition used in this paper in Table !
C Additional Experiments
C.1 Datasets
CREMAD: CREMAD 4 is audio-visual dataset designed for speech emotion recognition, It comprises 7,442 video clips of 2~33 seconds from 91 actors speaking several short words. which divided into 6,698 training samples and 744 testing samples. This dataset includes the six most common emotions: angry, happy, sad, neutral, discarding, disgust and fear.
KSounds 2 dataset is commonly used dataset containing 31 action categories that can be recognized visually and auditorily, which contains 19,000 10~second video clips from YouTube. This dataset is divided into training set with 15,000 clips, validation set with 1, 1,900 clips, and testing set with 1,900 clips.
KSounds:
NVGesture: NVGesture 31 dataset is a multimodal dataset specifically designed for gesture recognition, containing three types of data modalities, i.c., RGB, optical flow (OF), and Depth. This dataset includes 25 different gesture categories, covering variety of common gestures. This dataset divided into 1,050 samples for training and 482 samples for testing.
VGGSound: VGGSound 6 dataset is an audio-visual dataset in the wild, with nearly 200K 10- second video clips. Each sound-ene conddemitting object is also visible in the corresponding video clip in this dataset. After filtering out unavailable videos, 168,618 videos for training and validation, and 13,954 videos for testing in experimental settings.
Twitter: Twitter [5I dataset dataset designed for multilingual sentiment analysis, which divided into training set with 3,197 pairs, validation set with 1,1 pairs and testing set with 1,037
23

---

Table I: Notation Definition
Notation
Description
N K
The number of training data. The number of category labels. Audio/video data point. Training set.
x² /
X {E] x
E (0.
Category label of i-th data.
Y fy;
Category label set. o-modality.
Superscript
Superscript a/v
Audio/video modality.
()())
Encoder of modality.
0o
Parameters of (.).)
u
Feature of x
(.)
Classifier of o-modality.
Oo
Parameters of (-).
p
Prediction of x""
T
{00, 0m)
Parameters set.
((-)
Cross entropy loss.
(C
Cross entropy loss over training.
Alype-ppaaamete used to soften label.
Vi
Residual label of a. at iteration.
c(-)
Classification loss.
Eall()
Loss for classifiers.
Epre
Loss for 1 classifiers.
(())
Objective function.
LSUB())
Overall objective function.
n2
The number of classifier for o-modality.
sf
Confidence for o-modality at t-th iteration.
Coefficient of confidence score.
Tolerance score.
iN 9(-)
Adjustment frequency.
Gap function.
pairs. This dataset consists of tweets collected from Twitte positive, negative, and neutral.
with with three different sentiment labels:
Sarcasm: Sarcasm 3 dataset is specifically designed for sarcasm detection tasks, containing large amount of text data labeled as sarcastic or non-sarcastic, which typically draws from various sources, such as social media, news comments, and conversation data, with each entry clearly labeled for sarcasm. This dataset includes 19,816 16 pairs for the training set, 2,410 pairs for the validation set, and 2,409 pairs for the testing set.
C.2 Implementations of Toy Experiments
The toy experiment is designed to validate the motivation behind our method. Specifically, it is conducted on the CREMAD dataset, where the task is multimodal classification. Four methods are adopted for comparison: Naive MML, G-Blend [10] MML w/ GB, and Ours. Here, MML w/ GB refers to applying the gradient boosting algorithm to further improve the trained video model obtained from Naive MML, while keeping the audio model fixed. Ours refers to the method employing the sustained boosting algorithm proposed in this paper. Furthermore, ResNet- 18 used as the encoder for all methods, with its parameters randomly initialized. The selection of other hyperparameters is kept consistent with the main experiment.
http://twitter.com
24

---

C.3 More Results for Ablation Study
MAP Results on CREMAD and KSounds Datasets: We have added the MAP results of the ablation study on the CREMAD and KSounds datasets in Table II The MAP results in Table demonstrate that the objectives defined in Equation
3 and all lead to performance gains.
Table The MAP for ablation study on CREMAD and KSounds datasets.
CREMAD
KSounds
co
p
Multi
Audio
Video
Multi
Audio
Video
X
0.8967
0.7061
0.7523
0.7796
0.5465
0.5484
X
V
X
0.8895
0.7235
0.7535
0.7864
0.5482
0.5606
V
0,9014
0.7321
0.7335
0.7821
0.5474
0.5651
V
0.9103
0.7529
0.7612
0.7901
0.5461
0.6040
Ablation Study on Image-Text and Tri-Modal Datasets: an ablation study on image-text dataset, i.e., Twitter dataset, and the dataset with three modalities, i.e. NVGesture dataset. The accuracy and Fl are reported in Table and and Table [V The results demonstrate that the objectives defined in Equation (3) and (4) all lead to performance gains in terms of both accuracy and F1.
To further validate our method, we report
Table III: The results for ablation study on Twitter dataset.
Accuracy
Fl
E
Ep
Multi
Image
Text
Multi
Image
Text
X
0.7425
0.5400
0.7406
0.6830
0.3957
0.6868
X
V
X
0.7396
0.5429
0.7387
0.6787
0.4113
0.6806
V
0.7445
0.5265
0.7454
0.6767
0.3865
0.6794
V
V
0.7512
0.5434
0.7488
0.6962
0.3998
0.6911
Table IV: The results for ablation study on NVGesture dataset.
Accuracy
F1
Cp
Multi
RGB
OF
Depth
Multi
RGB
OF
Depth
X
0.8465
0.7573
0.7780
0.7946
0,8505
0.7649
0.7809
0.7968
V
X
0.8402
0.7427
0.7718
0.7759
0.8428
0.7604
0.7720
0.7812
V
0.8485
0.8008
0.7988
0.7801
0.8526
0.8085
0.8009
0.7881
v
0.8501
0.7988
0.8029
0.7967
0.8533
0.8076
0.8057
0.8033
C.4 Evaluation of Generalization in More Scenarios
To further validate the effectiveness of our method, employ a ietor-independent data partitioning strategy on the CREMAD dataset, ensuring that the training, validation, and test sets contain mutually exclusive subjects following the setting of [18 27 Specifically, we randomly select the audio-video data of 63 actors for training, 14 actors for validation, and 14 actors for testing, with no overlap of actors among the three splits.
We first investigate the influence of the hyperparameters a and on actor-independent CRE- MAD dataset. The results are shown in Figure]land Figure III The results demonstrate that our method is not sensitive to ayper-parameter, a and A. Furthermore, the ablation study on actor-independent CREMAD dataset is shown in Table V The results further verify the effectiveness of the objectives defined in Equation 2 and ④
Then, we compare our method with some representative methods actor-independent CRE- MAD dataset. The results are shown in Table From the results in Table we can find that our method can achieve the best performance compared with SOTA baselines.
25

---

0.8
0.6
0.4
0.4
Video Audio
Video
Audio
Multi
Multi
Figure Sensitivity to a.
Figure II: Sensitivity to
Table V: The results for ablation study on actor-independent CREMAD dataset.
Accuracy
MAP
Co
Multi
Image
Text
Multi
Image
Text
V
X
0.7443
0.6197
0.5191
0.8086
0.6517
0.5433
V
0.7333
0.6262
0.5344
0.8135
0.6508
0.5861
X
V
V
0.7388
0.6273
0.5366
0.8111
0.6496
0.5860
V
0.7639
0.6087
0.5388
0.8300
0.6557
0,5544
Table VI: Comparison with SOTA baselines actor-independent CREMAD dataset.
Method
Accuracy
MAP
Audio/Video
0.6219/0.5366
0.6588/0.528 5380
Naive MML
0.6153
0.6671
MLA
0.7421
0.8100
ReconBoost
0.7344
0.8027
Ours
0.7639
0.8300
C.5 Influence of Unstable Confidence Score
Since our method relies on confidence gap to determine whether to add a classifier, fluctuations in confidence scores indeed influential factor. We conduct an experiment explore the fluctuation of confidence score in scenario with modality noise on CREMAD dataset. Specifically, we add Gaussian noise to the video modality to introduce instability in confidence scores and investigate its impact on the algorithm. We design metric to evaluate the fluctuation of confidence score during training. We first compute the absolute difference in confidence scores between consecutive iterations, and then take the average over all training rounds:
Vo E vJ, 80
Σ t=
(47)
Here, sf denotes the confidence score of o-modality at t-th iteration, and n denotes the current iteration index in the training process. A higher value of go indicates more severe fluctuations in confidence scores, reflecting a sharper and less stable prediction behavior. Furthermore, we compare our method with naive MMI. and ReconBoost 21
We first compare the model performance in scenario with modality noise in Table VII where denotes the noise rate, "#CLS" denotes the number of classifier during training, and in the first column, we also report the fluctuation of confidence score. fact, as training progressed, we observe that in noisy scenarios, increasing the number of classifiers beyond certain point no longer led to improved performance on the validation set. This phenomenon became more pronounced as the noise level increased. In other words, the classifiers may overcompensate in scenarios involving modality noise. On the other hand, we also observe that after reaching its optimal performance, the model's performance gradually degraded without drastic drop. Therefore, in practical scenarios, early stopping strategy based on validation set performance can employed select the optimal model and avoid classifier overcompensation.
26

---

Table VII: The fluctuation of confidence score.
#CLS
Ç
0,
(5" 0.1428)
20, (5" 0.3920)
50, (8" = 0.4760)
0.1411
0.1411
0.2863
2
0.6559
0.6129
0.6398
0.7634
0.7608
0.7728
4
0,8038
0.7917
0,8185
0.8266 5
0.8118
0,8212
0.8306 6
0.8293
0.8253
0.8401
0.8333
0.8199
8 0.8441
0.8266
0,8239
0.8468
0.8293
0.8185
10
0.8515
0.8280
0.8145
Furthermore, we compare the performance with competitive baselines including naive MML. Re- conBoost [2T] The results are shown in Table VIIIl We can find that compared with the competitive method ReconBoost, our approach consistently achieves superior performance, demonstrating its effectiveness in scenario with modality noise.
Table VIII: Performance comparison under modality noise scenario.
Method
C -
Ç - 20
50
Naive MML
0.6507
0.6425
0.6331
ReconBoost
0.7557
0.7215
0.7031
Ours
0.8515
0.8333
0.8253
C.6 Computational and Memory Cost at Inference Stage
To demonstrate the practical upplicability of our approach, further provide comprehensive analy- sis of its computational and memory overhead at inference stage 3 Furthermore, we conduct an exper- iment to analyze the computational and memory cost during inference phase on CREMAD dataset. The baselines include naive MML,, PMR 12]. AGM 28 MLA [52] ReconBooss (11] The results
shown in Table XX We can find that method achieves superior performance while maintaining competitive inference time. Furthermore, compared with naive MMI., our method introduces only 1M additional parameters when 10 classifiers added. Given the total model size of 23.6M, this corresponds to a relatively small increase of approximately 4%.
Table IX: Computational cost comparison on CREMAD dataset.
Method
Accuracy
Inference time (s)
Naive MML
0.6507
5.29
PMR
0,6659
6,59
AGM
0.6733
5,58
MLA
0.7943
5.63
ReconBoost
0.7557
5,33
Ours
0.8515
5.62
C.7 Selection of Score Function
In our method, the score function is employed quantify the disparity in classification performance between different modalities, which serves as the basis for determining whether to introduce an additional classifier for the weak modality. Therefore, any metric that effectively captures the
For computational cost during training. the analysis have been posted in Section 4.5 of the original paper.
27

---

difference in classification capabilities across modality-specciff models can be adopted as a score function.
To some extent, entropy and loss can reflect the learning status of modality-specinn models and thereby serve proxies for their classification capabilities. To evaluate the effectiveness of using entropy and loss as score functions, we conducted experiments on the CREMAD dataset, where these two metrics were used guide the adaptive classifier assignment. The experimental results presented in Table X where т denotes the threshold. Please note that since we use the ratio between different modal metrics compare against the threshold т, the value of I remains consistent across different metric types such as confidence, loss, and entropy. These findings indicate that the proposed method is robust the choice of score function, consistently achieving comparable results regardless of whether confidence score, entropy, or loss is used.
Table X: Performance with different score function.
Score function
Accuracy
Threshold
#CLS
Confidence score (Ours)
0.8515
0.01
10
Entropy
0.8522
0.01
10
Loss
0.8562
0.01
10
C.8 Computational Cost vs. The Number of Classifiers
The computational wverhead-includee
both training and inference time-is indeed influenced by the number of classifiers. Since our method dynamically adds classifiers during training, we incorporate threshold to limit the maximum number of classifiers. This allows us to systematically investigate the trade-off between computational cost and model performance function of classifier quantity.
Specifically, conduct experiments the CREMAD dataset, where the maximum number of weak modality classifiers is constrained to not exceed a predefined threshold M. We vary M across the set {2, 4, 6, 8, 10), and for each setting, report the training time, inference time, and the corresponding classification performance.
The experimental results summarized in Table XI We observe that as the number of classifiers increases, both training time and inference time increase accordingly, while performance also improves certain extent. This indicates that incorporating more classifiers can indeed enhance model performance, but at the cost of increased computational overhead.
Table XI: Computational cost vs the number of classifiers.
#CLS
Accuracy
Training time (hrs)
Inference time (s)
2
0,8159
1.68
5.31
0,8387
1.72
5.38
6
0.8441
1.78
5,46
0.8468
1.86
5.53
10
0.8515
1.98
5.62
28


Reviews:
[{"review_id": "IvSJggdmpB", "reviewer": "Reviewer_Y2TP", "summary": "This paper addresses the often-overlooked issue of modality imbalance in multimodal learning (MML), particularly the disproportion in classification ability between strong and weak modalities. Instead of aligning the learning rates or optimizing joint features, the authors focus on directly enhancing the classification performance of the weaker modality. To achieve this, they propose a Sustained Boosting algorithm, which jointly minimizes classification and residual errors to progressively improve weak classifiers. Additionally, an Adaptive Classifier Assignment (ACA) strategy is introduced to dynamically assign more classifiers to underperforming modalities, guided by confidence scores. Theoretical analysis is provided to guarantee the convergence of the proposed method in terms of closing the loss gap between modalities. Empirical results across six multimodal datasets—including audio-video and image-text pairs—demonstrate that the proposed approach consistently outperforms a wide range of state-of-the-art baselines, including ReconBoost, LFM, and others. Ablation and sensitivity analyses further validate the contribution of each component.", "questions": "Adaptive Classifier Assignment: If the confidence score for a modality fluctuates due to noise or temporary degradation in input quality, does the algorithm risk overcompensating by adding unnecessary classifiers, potentially leading to overfitting? It would also be helpful to see results from a stress test on this adaptive strategy under noisy modality conditions.\n\nHow does your method behave when a modality is missing or corrupted? Given that the method is trained on full-modal input, would sustained boosting and classifier assignment degrade gracefully if a modality is unavailable during inference?\n\nA natural baseline to compare against would be modality-specific knowledge distillation, where weak modalities learn from stronger ones via teacher-student dynamics. Has this been explored or considered?", "limitations": "See questions", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper exhibits strong technical rigor and proposes a clearly motivated shift in focus—from aligning learning dynamics to balancing classification abilities across modalities. This perspective is refreshing and highlights a previously under-discussed source of performance degradation in MML. The use of a boosting-based framework is well-justified, and the idea of sustaining the optimization over both classification and residual errors is thoughtfully executed. Importantly, the inclusion of the Adaptive Classifier Assignment strategy provides a dynamic mechanism to address imbalance throughout training, rather than relying on fixed configurations.\n\nMoreover, the authors support their claims with extensive experiments across six datasets that cover diverse modalities and tasks. Theoretical convergence analysis is presented with full proofs, ensuring the robustness of the proposed approach. The empirical results are generally strong, often yielding the best performance on each benchmark.\n\nHowever, the work is not without its limitations. While the authors do touch on model overhead, the computational and memory cost introduced by using multiple classifiers—especially during inference—is not analyzed in detail. This raises concerns about the scalability and deployability of the approach in real-world applications, particularly those requiring low-latency or resource-constrained inference. Additionally, although the performance gain is consistent, the margin over strong baselines like ReconBoost or LFM is sometimes modest. Another limitation is that the approach is designed for cases where all modalities are fully present, and the framework’s behavior under partial or missing modality scenarios is not explored. Finally, while the adaptive strategy is a novel idea, there is a lack of discussion on its stability—e.g., how it reacts to noisy modality-specific confidence scores or abrupt changes in modality quality during training.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your conductive comments.\n\n**Answer for Question \"Influence of Unstable Confidence Score\"**: Since our method relies on confidence gap to determine whether to add a classifier, fluctuations in confidence scores are indeed an influential factor. We conduct an experiment to explore the fluctuation of confidence score in scenario with modality noise on CREMA-D dataset. Specifically, we add Gaussian noise to the video modality to introduce instability in confidence scores and investigate its impact on the algorithm. We design a metric to evaluate the fluctuation of confidence score during training. We first compute the absolute difference in confidence scores between two consecutive iterations, and then take the average over all training rounds, i.e., $\\bar{s}=\\frac{1}{n}\\sum_{t=2}^n|s_t-s_{t-1}|$. Here, $s_t$ denotes the confidence score at $t$-th iteration. A higher value of $\\bar{s}$ indicates more severe fluctuations in confidence scores, reflecting a sharper and less stable prediction behavior. Furthermore, we compare our method with naive MML and ReconBoost [1].\n\nWe first compare the model performance in scenario with modality noise in Table 1, where $\\mu$ denotes the noise rate [2], #CLS denotes the number of classifier during training, and in the first column, we also report the fluctuation of confidence score. In fact, as training progressed, we observe that in noisy scenarios, increasing the number of classifiers beyond a certain point no longer led to improved performance on the validation set. This phenomenon became more pronounced as the noise level increased. In other words, the classifiers *may overcompensate* in scenarios involving modality noise. On the other hand, we also observe that after reaching its optimal performance, the model's performance gradually degraded without a drastic drop. Therefore, in practical scenarios, an early stopping strategy based on validation set performance can be employed to select the optimal model and avoid classifier overcompensation.\n\n*Table 1. The fluctuation of confidence score.*\n\n| #CLS | &nbsp;&nbsp;&nbsp;&nbsp;$\\mu=0\\%$, ($\\bar{s}=0.1428$)&nbsp;&nbsp;&nbsp;&nbsp; | &nbsp;&nbsp;&nbsp;&nbsp;$\\mu=20\\%$, ($\\bar{s}=0.3920)$&nbsp;&nbsp;&nbsp;&nbsp; | &nbsp;&nbsp;&nbsp;&nbsp;$\\mu=50\\%$, ($\\bar{s}=0.4760)$&nbsp;&nbsp;&nbsp;&nbsp; |\n|:----:|:-----------------------------:|:-----------------------------:|:-----------------------------:|\n| 1 | 0.1411                  | 0.1411 | 0.2863 |\n| 2 | 0.6559                  | 0.6129 | 0.6398 |\n| 3 | 0.7634                  | 0.7608 | 0.7728 |\n| 4 | 0.8038                  | 0.7917 | 0.8185 |\n| 5 | 0.8266                  | 0.8118 | 0.8212 |\n| 6 | 0.8306                  | 0.8293 | **0.8253** |\n| 7 | 0.8401                  | **0.8333** | 0.8199 |\n| 8 | 0.8441                  | 0.8266 | 0.8239 |\n| 9 | 0.8468                  | 0.8293 | 0.8185 |\n| 10 | **0.8515**              | 0.8280 | 0.8145 |\n\nFurthermore, we compare the performance with competitive baselines including naive MML, ReconBoost [1]. The results are shown in Table 2. We can find that compared with the competitive method ReconBoost, our approach consistently achieves superior performance, demonstrating its effectiveness in scenario with modality noise. \n\n*Table 2. Performance comparison under modality noise scenario.*\n\n| Method  | $\\mu=0\\%$| $\\mu=20\\%$ | $\\mu=50\\%$ |\n|:----:|:----:|:----------:|:------------:|\n| Naive MML  | 0.6507   | 0.6425     | 0.6331     |\n| ReconBoost | 0.7557   | 0.7215     | 0.7031     |\n| Ours       |**0.8515**|**0.8333**  |**0.8253**  |\n\nRelevant experiments and discussion will be included in the final version.\n\n**Answer for Question \"Stability under Modality Missing\"**: Our method is adaptable to scenarios involving missing modalities. We comprehensively evaluate its performance under test-time missing modality conditions. Specifically, test-time missing refers to cases where the modalities are complete during training but missing during the testing phase. The experiments are conducted on CREMA-D dataset with different missing rate [3]. Naive MML, ReconBoost [1], and MLA [3] are selected as baselines for comparison, where MLA introduces specifically designed algorithms to address the corresponding challenges in modality missing. We report the results with missing rate 20% and 50% in Table 3. We can find that as the modality missing rate increases, the performance of all methods declines. Nevertheless, our method consistently achieves the best performance under all missing rates, demonstrating the effectiveness of our method in scenario with missing modality. \n\n*Table 3. Performance comparison in scenario with modality missing.*\n\n| Method  | $r=0\\%$  | $r=20\\%$ | $r=50\\%$ |\n|:----:|:----:|:----------:|:------------:|\n| Naive MML  | 0.6507   | 0.5849   | 0.5242   |\n| ReconBoost | 0.7557   | 0.6321   | 0.5568   |\n| MLA        | 0.7943   | 0.6935   | 0.5753   |\n| Ours       |**0.8515**|**0.7540**|**0.6008**|\n\nRelevant experiment and discussion will be included in the final version.\n\n**Answer for Question \"Modality-Specific Knowledge Distillation Strategy\"**: Using strong modalities to distill weaker ones is indeed a compelling and insightful idea. In our approach, we utilize performance gap to determine the relative strength of each modality and then independently enhance the classification capability of the weaker modalities. This can be viewed as an *implicit* strategy that leverages modality performance to boost the learning of weak modalities. In contrast, methods that *explicitly* distill knowledge from strong modalities aim to directly improve the classification performance of weaker ones through knowledge transfer. For instance, C2KD [4] is a representative method designed based on this idea by using the confidence scores from the strong modality to guide and refining the confidence of the weak modality. We conducted experiments to compare the effectiveness of our method and C2KD.\n\nWe compare this approach with our method on CREMA-D dataset under the same setting. The accuracy results are reported in Table 5, where the accuracy of C2KD is refered from its original paper. We can find that (1). Compared with naive MML, C2KD can achieve better performance, demonstrating the effectiveness of modality-specific knowledge distillation; (2). Our approach can achieve the best performance compared with naive MML and C2KD. This suggests that directly enhancing the performance of weak modality classifiers may be a more effective strategy than knowledge distillation.\n\n*Table 5. The effectiveness of cross-modal knowledge distillation.*\n\n| Method     | Audio    | Video    | Multi    |\n|:----:|:----:|:----------:|:------------:|\n| Naive MML    | 0.6075   | 0.2702   | 0.6507   |\n| C2KD*        | 0.614    | 0.628    | -        |\n| Ours         |**0.6835**|**0.6828**|**0.8515**|\n\nRelevant experiments and discussion will be included in the final version.\n\n**Answer for Question \"Computational and Memory Cost\"**: To demonstrate the practical applicability of our approach, we further provide a comprehensive analysis of its computational and memory overhead. For computational cost during training, the analysis have been posted in Section 4.5 of the original paper. Furthermore, we conduct an experiment to analyze the computational and memory cost during inference phase on CREMA-D dataset. The baselines include naive MML, PMR, AGM, MLA, ReconBoost. The results are shown in Table 6. We can find that our method achieves superior performance while maintaining competitive inference time. Furthermore, compared with naive MML, our method introduces only 1M additional parameters when 10 classifiers are added. Given the total model size of 23.6M, this corresponds to a relatively small increase of approximately 4%.\n\n*Table 6. Computational cost comparison on CREMA-D dataset.*\n\n|  Method     | Accuracy | Inference time (s) |\n|:----:|:----:|:----------:|\n| Naive MML    | 0.6507   | 5.29               |\n| PMR          | 0.6659   | 6.59               |\n| AGM          | 0.6733   | 5.58               |\n| MLA          | 0.7943   | 5.63               |\n| ReconBoost   | 0.7557   | 5.33               |\n| Ours         | **0.8515**   | 5.62               |\n\nRelevant experiments and discussion will be included in the final version.\n\n**Answer for Question \"Performance Comparison\"**: From the Table 1 of the original paper, we can find that our method achieves notable performance improvements on both video-audio and tri-modal datasets. Specifically, it improves accuracy by 1.53% on the CREMA-D dataset, 0.10% on the KSounds dataset, 0.27% on VGG dataset, and 0.36% on the NVGesture dataset. However, on the Twitter2015 and Sarcasm datasets, the performance gains are relatively modest, although our method still achieves the best results in most cases. This may be attributed to the limited complementary information between modalities in the text and image data. In fact, we observe that, compared to the best single-modal performance on these datasets, the improvements brought by all multimodal methods are generally minor. Notably, on the Twitter2015 dataset, most multimodal approaches even underperform the best single-modal baseline. Addressing this question is a promising direction for future research.\n\n**Reference:**\\\n[1]. Hua, Cong, et al. ReconBoost: Boosting can achieve modality reconcilement. *ICML*. 2024.\\\n[2]. Hu, Peng, et al. Learning cross-modal retrieval with noisy labels. *CVPR*. 2021.\\\n[3]. Zhang, Xiaohui, et al. Multimodal representation learning by alternating unimodal adaptation. *CVPR*. 2024.\\\n[4]. Huo, Fushuo, et al. C2kd: Bridging the modality gap for cross-modal knowledge distillation. *CVPR*. 2024."}, {"review_id": "4gids9FpoW", "reviewer": "Reviewer_5gtj", "summary": "This manuscript introduces a theoretically sound and empirically validated approach to address modality imbalance in multimodal learning. By crafting a specialized boosting algorithm, it enhances the classification ability of weaker modality-specific classifiers. Through an analysis of multimodal model capacity, the work sheds fresh light on the roots of modality imbalance. Comprehensive evaluations against state-of-the-art methods demonstrate the proposed approach’s clear advantages, and supplementary experiments further attest to its overall efficacy. However, the paper would benefit from deeper discussion and additional experiments regarding the selection of the boosting strategy and the broader applicability of the technique.", "questions": "1. Could other boosting paradigms yield comparable improvements, and what underlying mechanisms would support their success?\n2. In what ways could this boosting-based approach be adapted for or integrated into early fusion multimodal frameworks?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The authors reconceptualize modality imbalance as a disparity in classification capacities, an angle largely overlooked in previous studies, which prompts a reconsideration of the problem through the lens of model capacity.\n2. The boosting framework is effectively tailored for the multimodal context, marking a creative application of boosting techniques.\n3. Experiments conducted on six diverse datasets consistently show superior performance compared to leading baselines such as ReconBoost and DI‑MML.\n4. The work includes extensive analyses—hyperparameter sweeps, ablation trials, and visualizations—to underscore the proposed method’s strengths.\n5. Figure 4 clearly shows improved classification outcomes for weaker modalities, bolstering confidence in the approach.\n\n**Weaknesses:**\n\n1. Since the boosting algorithm is rooted in gradient boosting, it remains uncertain whether other boosting variants could match its performance.\n2. The current method is validated only for late fusion; its effectiveness with early fusion architectures is not explored.\n3. The annotation “Full Prediction vs. t−1 CLS” in Figure 3 is unclear—more precise labeling or explanation is needed.\n4. The design and parameters of the toy experiments are insufficiently described; supplying full experimental details would enhance reproducibility.", "was_revised": true, "final_justification": "I appreciate the author's response, which provided ample clarification and addressed my concerns. Therefore, I'm willing to maintain my rating.", "rebuttal": "Thank you for your conductive comments.\n\n**Answer for Question \"Adaptability for Other Boosting Strategy\"**: First, our algorithm is inspired by the gradient boosting paradigm. The motivation behind this approach lies in improving the classification performance of weak modalities during training, for which residual learning provides a natural solution. Second, other representative methods such as AdaBoost also enhance classifier performance by modifying the sampling strategy, which is another feasible direction. We believe that resampling misclassified samples from the weak modality can facilitate more effective learning, potentially bridging our motivation with AdaBoost-style algorithms.\n\nIt is worth noting that certain existing works have investigated sample-level imbalance. For example, SMV [1] achieves balanced learning by resampling modality-specific data that has not been sufficiently learned, and has demonstrated promising results. However, integrating this sample-level resampling strategy into the weak modality boosting framework may constitute a novel research direction. As such, we leave this exploration for future work.\n\nRelevant discussion will be included in the final version.\n\n**Answer for Question \"Effectiveness with Early Fusion Architectures\"**: Our method is built upon a late fusion architecture, with the core motivation of identifying and mitigating performance disparities across modalities, thereby achieving balanced classification capabilities among them. This implies that our method cannot be directly applied to architectures that rely solely on a single fusion classifier. In early fusion methods, only fused representations are used for prediction, and the classification process is not decomposable into modality-specific classifiers. As a result, our approach is not directly compatible with early fusion architectures.\n\nTo adapt our method to early fusion settings, it may be necessary to explicitly decouple the classification contributions of individual modalities and assess their performance differences. This would likely require a fundamentally new architectural design, which we consider a promising direction for future work.\n\n**Answer for Question \"Explation of Full Prediction vs $t-1$ CLS\"**: Our method adaptively adds new classifiers during the training process, and we denote the current number of classifiers for the weak modality as $t$. \"Prediction of $t-1$ CLS\" refers to the performance of the first $t-1$ classifiers, excluding the newly added one, while \"Full Prediction\" refers to the performance of all t classifiers. \n\nThe primary objective of this experiment is to verify the feasibility and necessity of residual learning by comparing the performance of the $t$-th classifier with that of the first $t–1$ classifiers. Essentially, the $t$-th classifier learns the residual—that is, the performance difference between itself and the previous classifiers. This observation is also supported by the results shown in Figure 3 of the original paper.\n\n**Answer for Question \"Implementations of Toy Experiments\"**: The toy experiment is designed to validate the motivation behind our method. Specifically, it is conducted on the CREMA-D dataset, where the task is multimodal classification. Four methods are adopted for comparison: *Naive MML*, *G-Blend* [2], *MML w/ GB*, and *Ours*. Here, MML w/ GB refers to applying the gradient boosting algorithm to further improve the trained video model obtained from Naive MML, while keeping the audio model fixed. Ours refers to the method employing the sustained boosting algorithm proposed in this paper. Furthermore, ResNet-18 is used as the encoder for all methods, with its parameters randomly initialized. The selection of other hyperparameters is kept consistent with the main experiment. \n\nRelevant implementation details will be included in the final version.\n\n**Reference:**\\\n[1]. Wei, Yake, et al. Enhancing multimodal cooperation via sample-level modality valuation. *CVPR*. 2024.\\\n[2]. Wang, Weiyao, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard?. *CVPR*. 2020."}, {"review_id": "OcRAYawlB2", "reviewer": "Reviewer_h6gC", "summary": "This paper proposes a promising and adaptable framework for improving multimodal learning in scenarios where certain modalities dominate others in classification performance. The proposed sustained boosting and adaptive classifier assignment modules offer an elegant way to dynamically rebalance classification capacity without heavily redesigning the architecture. Results show strong empirical gains across six datasets. However, a more thorough comparison and discussion of the similarities and differences with existing boosting-based method, i.e., ReconBoost, are necessary to better highlight the novelty of the proposed approach. Nevertheless, this is a meaningful and actionable contribution to the field of robust multimodal systems.", "questions": "1. In what ways is the proposed method innovative compared to ReconBoost?\n\n2. No ablation on the confidence scoring function—would other metrics (entropy, loss) work better?\n\n3. Although the authors demonstrate the effectiveness of their method on a trimodal dataset, relevant experimental details are lacking.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n1. Novelty of the Proposed Method: The proposed multimodal learning approach to address the modality imbalance issue is applicable to real-world multimodal systems where sensor strengths vary. And the adaptive classifier assignment strategy reduces the need for manual model tuning during training. \n\n2. Flexibility of the Proposed Method: Successful extension to 3 modalities (NVGesture) proves versatility beyond audio-visual/image-text tasks. Performance gains are consistent across tasks with more than two modalities, showing versatility.\n\n3. Effectiveness of the Experiments: Extensive experimental results demonstrate the effectiveness of the proposed method. Experiments include meaningful evaluation metrics (accuracy, MAP, MacroF1) suitable for practical deployment.\n\n4. Theoretical Proof: The convergence guarantees presented in Section 3.4 reinforce the credibility of the empirical results.\n\n## Weaknesses:\n1. Comparison with ReconBoost [1]: The authors propose a boosting algorithm to address the problem of modality imbalance. A detailed comparison with the ReconBoost method, including both similarities and differences, is necessary to clarify the contributions of the proposed approach.\n\n2. Selection of Scoring Function: The author employs the existing scoring method OGM [2]; however, it would be valuable to discuss whether more effective alternatives are available.\n\nReference:\n[1]. Hua, Cong, et al. Reconboost: Boosting can achieve modality reconcilement. ICML, 2024.\n[2]. Peng, Xiaokang, et al. Balanced multimodal learning via on-the-fly gradient modulation. CVPR, 2022.", "was_revised": true, "final_justification": "My concerns are addressed, and I will keep my positive rating.", "rebuttal": "Thank you for your conductive comments.\n\n**Answer for Question \"Comparison with ReconBoost\"**: Although both our method and ReconBoost [1] are inspired by the idea of gradient boosting, they differ fundamentally in motivation, implementation, and theoretical grounding:\n- The two methods are driven by distinct objectives. Our approach aims to improve the classification performance of weak modality classifiers using gradient boosting, whereas ReconBoost adopts the concept of gradient boosting to mitigate modality competition.\n- The utilization of gradient boosting differs significantly between these two methods. Our method learns the residual of a single-modality classifier to refine weak modality performance, while ReconBoost leverages the residual from another modality to address inter-modal competition.\n- The theoretical foundations of the two methods diverge. Our method establishes a link between the gradient boosting algorithm and a performance gap function, whereas ReconBoost demonstrates that, under certain conditions, introducing KL divergence as a regularization term results in a training objective equivalent to a specific form of gradient boosting.\n\nIn summary, despite both methods drawing inspiration from gradient boosting, they embody fundamentally different motivations, mechanisms, and theoretical interpretations.\n\nRelevant discussion will be included in the final version.\n\n**Answer for Question \"Selection of Score Function\"**: In our method, the score function is employed to quantify the disparity in classification performance between different modalities, which serves as the basis for determining whether to introduce an additional classifier for the weak modality. Therefore, any metric that effectively captures the difference in classification capabilities across modality-specific models can be adopted as a score function.\n\nTo some extent, entropy and loss can reflect the learning status of modality-specific models and thereby serve as proxies for their classification capabilities. To evaluate the effectiveness of using entropy and loss as score functions, we conducted experiments on the CREMA-D dataset, where these two metrics were used to guide the adaptive classifier assignment. The experimental results are presented in Table 1, where $\\tau$ denotes the threshold. Please note that since we use the ratio between different modal metrics to compare against the threshold $\\tau$, the value of $\\tau$ remains consistent across different metric types such as confidence, loss, and entropy. These findings indicate that the proposed method is robust to the choice of score function, consistently achieving comparable results regardless of whether confidence score, entropy, or loss is used.\n\n*Table 1. Performance with different score function.*\n\n| Score function | Accuracy | Threshold $\\tau$ | #CLS |\n|:----:|:----:|:----------:|:------------:|\n| Confidence score (Ours)  | 0.8515 | 0.01    | 10   |\n| Entropy        | 0.8522  | 0.01             | 10   |\n| Loss           | 0.8562  | 0.01             | 10   |\n\nRelevant experiments and discussion will be included in the final version.\n\n**Answer for Question \"Trimodal Experimental Details\"**: For the scenario with three modalities, a key aspect of the experiment lies in determining whether to add an additional classifier based on the confidence scores of the three modalities. Specifically, we compare the confidence score of each modality with the average of the confidence scores of the other two. If the difference exceeds a predefined threshold, a new classifier is assigned to the corresponding modality. \n\nRelevant implementation details will be included in the final version.\n\n**Reference:**\\\n[1]. Hua, Cong, et al. ReconBoost: Boosting can achieve modality reconcilement. *ICML*. 2024."}, {"review_id": "eM4149Ms1r", "reviewer": "Reviewer_3wbR", "summary": "This paper tackles the persistent challenge of modality imbalance in multimodal learning, proposing that the core issue stems from the inherent classification ability gap between modalities. The authors introduce a sustained boosting approach that incrementally strengthens weaker modality classifiers via residual error minimization. Additionally, an adaptive strategy dynamically assigns new classifiers based on modality-specific confidence scores. The theoretical convergence analysis is solid and bolstered by empirical results showing superior performance across diverse benchmarks. The results of visualizations, t-SNE plots, and convergence curves provides transparency into model behavior. However, despite the impressive results, the paper could benefit from further analysis on efficiency and more diverse ablation studies. Overall, this paper contributes a fresh viewpoint and a robust solution to the community.", "questions": "My key concerns about the paper are outlined below: \n\n(1). The relationship between training cost and the number of classifiers, as well as the corresponding performance, warrants further discussion.  \n\n(2) The authors are encouraged to report the results of ablation studies on additional datasets to validate the generality of the proposed method.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "This paper proposes a novel method to address the multimodal imbalance problem, with the following key advantages:\n(1)  the paper addresses a real and understudied issue in multimodal learning, i.e., disproportionate classification ability across modalities. It further identifies this problem as a key contributor to modality imbalance. Furthermore, it introduces a novel sustained boosting algorithm that explicitly enhances weaker modality classifiers;\n(2) theoretical analysis (Theorem 1) supports convergence guarantees of the proposed gap minimization framework; \n(3) consistent gains across metrics (Acc/MAP/F1) and modalities (video, audio, text) in Table 1 are impressive. Code release and exhaustive implementation details (Sec 4.1) align with open-science best practices; \n(4) the paper is clearly written, logically coherent, and well organized, making it easy to follow and understand.\n\nFurthermore, several weaknesses in the paper should be noted: (1). Lemma 1 only provides an abbreviated title, and it appears that the full statement or introduction may be missing; (2). Details on computational cost vs. performance tradeoff are limited and deserve deeper discussion. Training overhead (Fig 5) grows linearly with classifiers. This is problematic for resource-limited settings, yet mitigation strategies are absent; (3). The ablation study is primarily conducted on one dataset (CREMAD), which may not generalize well.", "was_revised": true, "final_justification": "The proposed method is novel, its convergence is theoretically guaranteed. During the rebuttle, the ahthous have well responsed my concerns. Besides, all reviewers consistently tends to accept it. So, I finnally give it 5:Accept.", "rebuttal": "Thank you for your conductive comments.\n\n**Answer for Question \"Lemma 1 Statement\"**: Thank you for pointing this out. This was indeed a typographical error on our part. The corrected statement of the lemma is as follows:\\\n**Lemma 1 (Gap Bound)** There exists a constant $\\kappa$ such that the loss gap function $\\mathcal{G}(\\cdot)$ and the gradient norm of the strong modality satisfy the following relationship:\n\n$$||\\nabla\\mathcal{L}^a(\\Phi_t)||\\ge\\kappa|\\mathcal{G}(\\Phi_t)|.$$\n\nWe will address this issue in the final version.\n\n**Answer for Question: \"Computational Cost vs The Number of Classifiers\"**: The computational overhead—including both training and inference time—is indeed influenced by the number of classifiers. Since our method dynamically adds classifiers during training, we incorporate a threshold to limit the maximum number of classifiers. This allows us to systematically investigate the trade-off between computational cost and model performance as a function of classifier quantity.\n\nSpecifically, we conduct experiments on the CREMA-D dataset, where the maximum number of weak modality classifiers is constrained to not exceed a predefined threshold $M$. We vary $M$ across the set {2, 4, 6, 8, 10}, and for each setting, we report the training time, inference time, and the corresponding classification performance.\n\nThe experimental results are summarized in the following Table. We observe that as the number of classifiers increases, both training time and inference time increase accordingly, while performance also improves to a certain extent. This indicates that incorporating more classifiers can indeed enhance model performance, but at the cost of increased computational overhead.\n\n*Table 1. Computational cost vs the number of classifiers.*\n\n| #CLS         | Accuracy | Training time (hrs) | Inference time (s) |\n|:----:|:----:|:----------:|:------------:|\n| 2            | 0.8159   | 1.68                | 5.31               |\n| 4            | 0.8387   | 1.72                | 5.38               |\n| 6            | 0.8441   | 1.78                | 5.46               |\n| 8            | 0.8468   | 1.86                | 5.53               |\n| 10           | 0.8515   | 1.96                | 5.62               |\n\nRelevant experiments and discussion will be included in the final version.\n\n**Answer for Question: \"Ablation Study on Other Dataset\"**: To further demonstrate the effectiveness of the key components of our method, we conduct the experiments for ablation study on trimodal dataset, i.e., NVGeasture dataset. Specifically, we analyze the effectiveness of the losses $\\epsilon$, $\\epsilon_o$, and $\\epsilon_p$ respectively defined in Eq.(2), (3), and (4) of the original paper. \n\nFor video-audio, image-text, and tri-modal scenarios, we selected three representative datasets for ablation experiments: the video-audio dataset CREMA-D, the image-text dataset Twitter2015, and the tri-modal dataset NVGesture. The result on CREMA-D dataset is presented in Table 2 of the original paper. And the results on Twitter2015 and NVGeasture dataset are respectively shown in Table 2 and Table 3. We can find that both objectives in Eq. (2), (3), and (4) can boost the multimodal classification performance in terms of accuracy.\n\n*Table 2. Ablation study on Twitter2015 dataset.*\n| $\\epsilon$ | $\\epsilon_o$ | $\\epsilon_p$ | Multi  | Image  | Text   |\n|:----:|:----:|:----------:|:------------:|:------------:|:------------:|\n|  yes       | no    | yes   | 0.7425 | 0.5400 | 0.7406 |\n|  no        | yes   | no    | 0.7396 | 0.5429     | 0.7387 |\n|  no        | yes   | yes   | 0.7445 | 0.5265 | 0.7454 |\n|  yes       | yes   | yes   | **0.7512** | **0.5434** | **0.7488** |\n\n*Table 3. Ablation study on NVGeasture dataset.*\n\n| $\\epsilon$ | $\\epsilon_o$ | $\\epsilon_p$  | Multi | RGB  | OF |  Depth |\n|:----:|:----:|:----------:|:------------:|:------------:|:------------:|:------------:|\n|  yes       | no    | yes        | 0.8465 |0.7573        |0.7780        |0.7946        |\n|  no        | yes   | no         | 0.8402 |0.7427        |0.7718        |0.7759        |\n|  no        | yes   | yes        | 0.8485 |**0.8008**        |0.7988        |0.7801        |\n|  yes       | yes   | yes        | **0.8501** |0.7988        |**0.8029**        |**0.7967**        |\n\n\nRelevant experiments and discussion will be included in the final version."}]

Author rebuttal:

Meta-review: This paper presents a novel and well-motivated approach to addressing modality imbalance in multimodal learning by viewing it as a disparity in classification ability and proposing a sustained boosting algorithm with adaptive classifier assignment. The method is theoretically grounded, empirically validated on six datasets (including tri-modal cases), and shows consistent improvements over strong baselines, with thorough ablation and sensitivity studies. While reviewers initially raised concerns about computational overhead, modest gains in some cases, and missing evaluations, the rebuttal effectively addressed these through additional experiments and clarifications. With these issues resolved, the reviewers reached a consensus to recommend acceptance, recognizing the work’s novelty, robustness, and potential impact on robust multimodal learning.
Decision: Accept (oral)