Classifier-guided Gradient Modulation for Enhanced Multimodal Learning
Authors: Zirun Guo
Tao Jin
Jingyuan Chen
Zhou Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: balanced multimodal learning
gradient modulation
Primary area: optimization_for_deep_networks

Abstract: Multimodal learning has developed very fast in recent years. However, during the multimodal training process, the model tends to rely on only one modality based on which it could learn faster, thus leading to inadequate use of other modalities. Existing methods to balance the training process always have some limitations on the loss functions, optimizers and the number of modalities and only consider modulating the magnitude of the gradients while ignoring the directions of the gradients. To solve these problems, in this paper, we present a novel method to balance multimodal learning with **C**lassifier-**G**uided **G**radient **M**odulation (CGGM), considering both the magnitude and directions of the gradients. We conduct extensive experiments on four multimodal datasets: UPMC-Food 101, CMU-MOSI, IEMOCAP and BraTS 2021, covering classification, regression and segmentation tasks. The results show that CGGM outperforms all the baselines and other state-of-the-art methods consistently, demonstrating its effectiveness and versatility. Our code is available at https://github.com/zrguo/CGGM.

Dassifier-guided Gradient Modulation for Enhanced Multimodal Learning
Zirun Guo 1.2, Tao Jin't, Jingyuan Chen', Zhou Zhao!:2 1.2
Zhejiang University, Shanghai AI Lab zrguo zrguo.cs@gman cswgmail.......mm.mmmmmm........ com
Abstract
Multimodal learning has developed very fast in recent years. However, during the multimodal training process, the model tends to rely on only one modality based which it could learn faster, thus leading to inadequate use of other modalities. Existing methods to balance the training process always have some limitations the loss functions, optimizers and the number of modalities and only consider modulating the magnitude of the gradients while ignoring the directions the gradients. To solve these problems, in this paper, present novel method to bal- ance multimodal learning with Classifier- Guided Gradient Modulation (CGGM), considering both the magnitude and directions of the gradients. We conduct tensive experiments on four multimodal datasets: UPMC-Food 101, CMU-MOSI, IEMOCAP and BraTS 2021, covering classification, regression and segmentation tasks. The results show that CGGM outperforms all the baselines and other state- of-the-art methods consistently, demonstrating its effectiveness and versatility. Our code is available at https: // gi github. com Zrguo/CGGM. CGGM.
Introduction
Humans perceive the world in multimodal way, such as sight, touch and sound. These multimodal features can provide comprehensive information to help us understand and explore the environ- ment. Recent years have witnessed great success in multimodal learning, such visual question answering [2]. multimodal sentiment analysis [18] and multimodal retrieval [26, 13].
Although multimodal learning has made significant progress in recent years, inadequate use of different modality information during training remains challenge. Theoretically, for example, Wu et al. [25] put forward the greedy learner hypothesis which states that multimodal model learns to rely on one of the input modalities, based on which it could learn faster, and does not continue to learn to use other modalities. Huang al. [12] find that during joint training, multiple modalities will compete with each other and some modalities will fail in the competition. Experimentallyy on some multimodal datasets, there is little improvement in accuracy between training with only one modality and training with all modalities [18,2] 21J. These theoretical analyses and experimental results demonstrate the inefficiency of multimodal learning to fully utilize and integrate information from different modalities.
To deal with this problem, recent studies [25, 17, 15, 8, 28, 9] investigate the training process of multimodal learning and propose gradient modulation strategies to better integrate the information of different modalities and balance the training process in some situations. However, all of these methods can not be applied easily for some limitations. For example, Wu et al. [25], Peng et al. [17], Li et al. [15] and Hua et al. [11] propose balancing methods based on cross-entropy loss for
*Coresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

classification tasks. For regression tasks or other tasks, we can not use these strategies. Besides, most of these methods can just deal with situations where there are only two modalities. For example, Wu et al. [25] propose the conditional learning speed which is difficult to calculate and employ if there are more than two modalities. For situations where there are more modalities, these methods can not be applied. Furthermore, most of these methods only consider modulating the magnitude of the gradients while ignoring the directions of the gradients.
Based on the above observations, in this paper, we propose a novel method to balance multimodal learning with Classifier- Guided Gradient Modulation (CGGM). In CGGM, we consider a more general situation with no limitations on the type of tasks, optimizers, the number of modalities, etc. Additionally, we consider both the magnitude and directions of the gradients fully boost the training process of multimodal learning. Specifically, we add classifiers to evaluate the utilization rate of each modality and obtain the unimodal gradients. Then, we leverage the utilization rate to adaptively modulate the magnitude of the gradients of encoders and use the unimodal gradients to instruct the model to optimize towards better direction.
We conduct extensive experiments on four multimodal datasets: UPMC-Food 101 [23], CMU- MOSI [27], IEMOCAP [3], and BraTS 2021 [1]. UPMC-Food 101 and IEMOCAP are classification tasks, CMU-MOSI is a regression task, and BraTS 2021 is segmentation task. CGGM outperforms all the baselines and other state-of- of-the-art methods, demonstrating its effectiveness and universality. In summary, our contributions are as follows:
We propose CGGM to balance multimodal learning by both considering the magnitude and direction of the gradients.
CGGM can be easily applied to many multimodal tasks and networks with no limitations on the type of tasks, optimizers, the number of modalities, etc. which indicates its versatility.
Our proposed CGGM brings consistent improvements to various tasks, including clas- sification, regression and segmentation tasks. Extensive experiments show that CGGM outperforms other state-of-the-an methods, demonstrating its effectiveness.
2 Related Work
Multimodal Learning. One of the main challenges of multimodal learning is how effectively utilize and integrate the information from different modalities to complement each other. According to the fusion strategies, there are three main multimodal fusion strategies: early fusion, intermediate fusion and late fusion. In early fusion methods [16, 24], raw data from different modalities is combined via concatenation or other methods at the input level before being fed into a model. Intermediate fusion [14] methods combine data from different modalities at various intermediate processing stages within a model architecture. Late fusion [2. 18) methods process data from each modality independently through separate models and combine them at later stage. In general, late fusion is the predominant method used in multimodal learning. The main reason [14] is that the architecture of cach unimodal stream has been carefully designed over the years to achieve state-of- the-art performance for each modality. Therefore, we can leverage these pre-trained models [5, 6] to achieve better results. Therefore, in this paper, our method is based on late fusion.
These fusion strategies are able to integrate information from different modalities effectively, but they have limited improvements to utilize information from different modalities to complement each other. In other words, they are not able to deal with the modality competition [12] or imbalanced multimodal learning. When the dominant modality is missing [10] or corrupted, the performance would degrade significantly. Different from these fusion strategies, our method aims to make relatively full use of the information of each modality and address the imbalanced multimodal learning.
Balanced Multimodal Learning. The inefficiency in fully utilizing and integrating information from multiple modalities poses a great challenge to the multimodal learning field. Some studies [18, 21] present that there is little improvement in accuracy between training with only one modality and training with all modalities. Wang et al. [22] show that multimodal models using multiple modalities can be even inferior to those using only one modality. To balance the multimodal learning process and fully utilize different modalities, series of balanced multimodal learning methods [25, 17, 15, 8, 28, 9, 7, 11] are proposed. Wu et al. [25] propose the conditional learning speed to capture the relative learning speed between modalities and balance the lcarning process. Peng et al. [17]

---

Encodas
Classifies
$1
f2
fM
(emy Van C)
(Emzs Var L)
(emxe: Vatae C)
d2
Fusion
Task Less
Direction
Magnitude
m
VgaL
max STID
Bmy Bays"
Busr
m
Foward
Backward
The overall architecture of CGGM. During the training stage, classifiers are introduced to calculate the directions of unimodal gradients and evaluation metries. During the inference stage, the classifiers are discarded.
Figure I:
propose gradient modulation strategy that adaptively controls the optimization of each modality via monitoring the discrepancy of their contribution towards the learning objective. More recently, Fan al. [8] propose the prototypical modal rebalance strategy to introduce different learning strategies for different modalities. Li al. [15] propose adaptive gradient modulation method that can boost the performance of multimodal models with various fusion strategies. Hua et [11] dynamically adjust the learning objective with reconcilement regularization against competition with the historical models.
However, all of these previous works have certain limitations and can only be used in some specific situations. For example, Wu et al. [25] propose conditional learning speed based intermediate fusion strategy which makes it hard to apply to situations where there are more than two modalities or where the network [9]. Li et al. [15] and Hua et al. [11] propose the balancing strategies with the assumption of the cross-entropy loss function mainly for classification. Particularly, Peng al. [17] employ the SGD optimizer. Different from these methods, we consider more general situation with no limitations on the number of modalities, the optimizer, the loss function and so on. Additionally, most of existing methods only consider the magnitude of the gradients and ignore the directions of the gradients. In contrast, we consider both them.
based on intermediate fusion. Peng al. [17]. Fan al. [8], Fu al.
Proposed Method
3.1 Problem Settings
Suppose there are modalities, referred as m1, m2, ... .m.- We denote the multimodal dataset as D {(x; J.) where N is the number of data in the dataset and a; (N/m1 , _.
, mM
We consider the most common structure (Figure 1) in multimodal models, where the inputs of different modalities are first fed into modality-specific encoders and then the representations of all modalities are inputted into fusion module. We denote the encoder of modality mi as Ø; where i = 1,2,, .M and the fusion module as Ω.
For the forward propagation, the features are first inputted into the encoder:
h4 00= (x:"))
where h; is the representation of modality m.. After obtaining the representations of all modalities, the fusion module is applied:
i = F(((()) h2, hwm))
where ý is the prediction, [ is the concatenation operation, and F(.) is the prediction head to predict the answer. 2(-)) fuses the multimodal representations and outputs the fused feature as the prediction token.

---

8.0
04
.00
(a)
(b)
(c)
Figure 2: (a) Accuracy of each modality and the fusion. (b) Gradient magnitude of each modality.
We use the Euclidean norm of the gradient vector to represent the gradient magnitude. (c) Gradient direction between each modality and their fusion. We use cosine similarity to represent the direction between two gradient vectors. We get all the results on the CMU-MOSI dataset.
3.2 Gradient Analysis
To introduce CGGM, first analyze the gradient updating process. We denote the loss function as (()) = Σx= (ii 3²) where 9 represents the parameters of the network, D the prediction and y² is the ground truth. For simplicity, we use ý' to represent the predictions in the following context. Different from previous methods which only consider cross-entropy loss [17, 8, 11], our L can be cross-entropy loss, loss or any other loss functions. With the Gradient Descent (GD) optimization method, the parameters of the fusion module Ω and encoders Φi can be updated as:
OF
OL (1" 34))
0 = =0- -v Von C(0,)) = of a
(3)
02
OF
OF On 8n 0Φ.
OE (5", 0F
04 = 0 Bo;
L(0 pu
where a is the learning rate, N is batch size, and f is the iteration. According to the chain rule used find the gradient in backpropagation, the update of Φ, will influence the update of S and vice versa. According to Figure 2(a) and 2(b). the gradient and the accuracy of the dominant modality will increase during the training process while the other two remain stable. Particularly, the gradient magnitude of the text modality increases very fast during the training process. This suggests the
encoder of the dominant modality will be updated much faster than others, which makes dp much larger. This phenomenon can also be validated by previous works [8, 17, 25J. Besides, in Figure 2(c), present the gradient direction between each modality and the fusion. We can observe that the similarity between audio modality and the multimodal fusion is less than 0, indicating that they optimize towards the opposite direction, thus hindering the gradient update for multimodal branch. Meanwhile, the similarity between text modality and the multimodal fusion is increasing, suggesting the optimization direction towards the dominant modality. With the progress of optimization, encoder of the dominant modality can make relatively accurate predictions, which makes the fusion module Ω only depend on this modality (both magnitude and direction mentioned above), leaving other encoders under-optimized.
di
3.3
Classifier-guided Gradient Modulation
3.3.1 Gradient Magnitude Modulation
As we discuss Section 3.2, the gradient magnitude of the dominant modality increases fast during the training while the other modalities remain stable, thus being under-optimized. To balance the training process and make the fusion module benefit from all the encoders simultaneously, we propose the classifier- Bassiffer-guidee guided gradient modulation. Specifically, we use modality classifier to make predictions of in Equation 1. We can write the process as:
(5) where fi is the classifier of modality m, and ÿm, is the prediction only using modality m,- The classifier fr consists of 1-2 multi-head self-attention (MSA) layers [20] and fully connected layer
Um, = [h:))

---

for classification and regression tasks, And for segmentation tasks, fi is light decoder. After h; is inputted into the fusion module 92, it becomes a more high-level representation. Therefore, we use several MSA layers to make h; more consistent with the output of the fusion module.
For specific task, we have some evaluation metrics such as accuracy and mean absolute error. Here, we choose one of the evaluation metrics (e.g. accuracy for classification tasks and mean absolute error for regression tasks) and denote it as E. For every iteration of training, we can get predictions from the classifiers. We denote the predictions as ø = (I Bmg....... ; where i is the current iteration. Furthermore, we evaluate the task using ÿ to get the evaluation metric ex (s,,, ex 'my '.' : Here, we use the difference between the two consecutive to denote the moodaltty-speciffe improvement for each iteration:
A = 5¹ (A
A m2
my
(6)
= (em m1 E m2
my?
max
where / 0, and T is the total iterations of training. Particularly, e0 is initialized to O. In some multimodal datasets, only using one of the modalities can achieve good results so we can not directly use to measure the utilization rate of different modalities. Therefore, it is reasonable to use the difference between E to denote the relative improvements for each itcration. Then, we define the gradient magnitude balancing term of modality m, for the t-th iteration follows:
Ek=LA A B - Σ ASt
where p is scaling hyperparameter and M is the number of modalities. According to Equation it is easy to find that when the performance of the model only using modality m, improves very fast (i.e. Aef, is large), Bh, will be small. Similarly, when the modality m, brings relatively limited improvements to the model (i.e. Ae'n, is small), " will be large. Therefore, Bú, is able to measure the relative utilization rate of these modalities and we use B modulate the magnitude of the gradient of the encoder Φi. So Equation 4 can be rewritten as:
0 0%
aubtt1y V
gh: L((()) Aett1 mi
Σn - 1. Σ At! Vem L(0
(8)
= 00 op
According to Equation 2, we know that the final predictions are closely related to h4 Therefore, after we modulate the gradient of the corresponding encoder Φi, it has an impact on the input of Ω which in turn helps the optimization of the fusion module f2
3.3.2 Gradient Direction Modulation
As Wu et al. [25] discover, when the model only depends on one modality to perform well, it does not contimue to learn to use other modalities. As discussed in Section 3.2, means that this modality dominates the updating of the model. Previous works [25, 17, 15] address this problem mainly by focusing on gradient magnitude modulation. However, in Section 3.2, we find that the model is optimized towards the dominant modality. Therefore, in this subsection, introduce method that could modulate the direction of the gradients to balance the training process.
In general, we want to balance the optimization direction of the model when the model only relies on one modality to make predictions. Therefore, we propose to enforce the gradient direction of the model as close as possible to the weighted average gradient direction of models only using one modality. We use Bi, in Equation 7 as the weight term. This ensures that when the model tends to optimize towards the dominant modality, Bi, can help the model use information from other modalities. Besides, since Bi,,, changes during the training process, this term can make dynamic adjustment to balance the optimization directions. Concretely, we can feed one modality into the model and drop other modalities by replacing them with O or other fixed values during training to calculate the gradient of this modality. By this method, we can calculate the unimodal gradients for all modalities. Then, we just enforce the gradient direction of the model as close as possible to the weighted average of these unimodal gradient directions. However, this method is very complex
5

---

Algorithm Classifier-guided gradient modulation
1: Input: Training dataset D {( Ma) iteration number T, the number of modalities M,
model F (Ør, S2, 2 F), classifiers fi. and hyperparameters.
2: Initiate: gP +- 0, E" + Empty List, fam t- O, classifier gradient list Lg-
3: for /. == to 7 do
4: Dj Sample D:
5: Forward propagation to get representations h = h2,,, has) in Equation 1;
6: for = to M do
7: Make predictions with h; using Equation 5;
8:
Calculate et and append ex:
9:
Append oe,. ((0t) to Ly:
10:
end for
11:
Ag - =et -
Calculate By ; 2, .M using Equation
13:
Calculate the loss L = Lrask + Xm and backward;
Calculate C using Equation 12;
ED & Ex, Lym f- p. g < Empty List;
16:
Update parameters using Equation 3 and 8.
17: end for
during training, because in every iteration we need to drop modalities to calculate the unimodal gradients, which is time-consuming with the increase in the number of modalities.
Therefore, we propose to use the gradients of the classifiers fi to represent the unimodal gradients. We will later demonstrate they are similar (Section 4.4 and Figure 4). Here, we take the gradient of regression tasks as an example where the output dimension is 1 so the gradient is an n-d vector. For classification tasks other tasks where the gradient is matrix, Appendix A for details. Concretely, we can calculate the gradient the classifier ft as:
Vor C(0%)) OC(OF)) 0f:
aC(ON) 00%
ac(OF:)
OC(05) aek
aek
where 05 represents the parameters of St. Different from 0f in Equation 3 and 4 where / is the iteration, 0f: here represents of the variables of @f. Similarly, we can calculate the gradient of the classifier F of the fusion module as:
AC(O7) 0F
[O((0T)
ac(OT)
aC(OT) 00.5
(10)
004
00
We regard Ver:C,i= 1,2, M as the unimodal gradients and Vor L as the fusion gradients. As mentioned before, we want to make Var L as close as possible to the weighted average direction I,
of C, 1,2, M.. Let sim(u, v) u v/l|u|ll|o| denote the dot product between l2 normalized and e (i.e. cosine similarity). We can enforce the gradient direction of the fusion module close as possible to the weighted average of these unimodal gradient directions by maximizing their cosine similarity:
M
maxx
B
sim (Voz L, Vor. L)
(11)
where is the current iteration. We rewrite Equation 11 as loss term:
Cym
BC
B sim (o V C))
(12)
M
The cosine similarity a number between -1 and 1. By adding [B( to the loss term, we can ensure that the loss L.m is always positive. As aforementioned, when modality m, has limited improvement, Bi, is large. Therefore, the corresponding term C will be large, making the optimization direction towards modality m,, which will balance the learning process.

---

Then our overall loss function can be written as:
C Ltask + AC gm
(13)
where Leas is the task loss function (e.g. cross-entropy loss, L1 loss) and is a trade-off between the two loss terms. We present our overall method in Algorithm 1.
4 Experiments
Table 1: The difference between the four datasets we
4.1 Datasets and Evaluation Metrics
use.
Dataset
Task type
No. of modality
We use four multimodal datasets: UPMC-
UPMC-Food 101
Classification Regression
2
Food 101 [23], CMU-MOSI [27], IEMO-
CMU-MOSI IEMOCAP
CAP [3], and BraTS 2021 [1]. Table 1 presents the difference between these four datasets.
Classification
3
BraTS 2021
Segmentation
4
UPMC-Food 101 [23] is food classification dataset, which contains about 100,000 recipes for a total of 101 food categories. Each item the dataset is represented by one image plus textual information. We use accuracy and F1 score to evaluate the performance of the model.
popular dataset for multimodal (audio, and video) sentiment analysis. Each video segment is manually annotated with sentiment score ranging from strongly negative strongly positive (-3 to +3). Following previous work [18, 10], we use binary accuracy (ACC-2), Fl score, 7-class accuracy (ACC-7), mean absolute error (MAE) and pearson correlation (Corr) to evaluate the performance of the model.
CMU-MOSI [27]
Table 2: Quantitative results on the UPMC-Food 101 dataset. Bold: best results. Underline: second best results.
Table 3: Results on BraTS 2021. WT, TC and ET denote the dice score of Whole Tumor, Tumor Core and Enhancing Tumor respectively.
Method
Acc
Fl
Method
WT
TC
ET
Avg.
Text only
84.77
84.72
flair only
70.42
51.41
45.27
55.70
Image only
68.24
68.23
tl only
50.73
36.13
38.77
41.88
Baseline
90.32
90.30
12 only
64.82
42.17
34.62
47.20
G-Blending [22]
90.42
90.38
tlee only
56.61
53.83
53.14
54.53
Greedy [25]
91.21
91.20
OGE [17]
91.08
91.08
Bascline
74.03
67.08
66.53
69.21
AGM [15]
91.49
91.48
MRD
74.02
68.04
67.28
69.78
PMR
92.01
91.98
MSLR
74.47
69.98
67.20
70.55
UMT [7]
91.82
91.82
UMT [7]
74.15
67.69
66.80
69.55
UME [7]
90.69
90.68
UME [7]
74.65
68.74
67.58
70.32
QMF [28]
92.87
92.85
QMF [28]
75.11
70.78
68.94
71.61
ReconBoost [1I]
92.52
92.51
ReconBoost [11]
75.21
70.18
70.01
71.80
CGGM
92.94
92.90
CGGM
76.94
72.75
72.14
73.94
IEMOCAP [3] a multimodal emotion recognition dataset, which contains recorded videos from ten actors in five dyadic conversation sessions. Following previous works [18, 24, 10], four emotions (happiness, anger, sadness and neutral state) are selected for emotion recognition. We use accuracy and F1 score to evaluate the performance of the model.
BraTS 2021 [1] a 3D multimodal brain tumor segmentation dataset, which has four modalities: flair, flce, 1/ and 12. The input image size is 240 240 x 155. The annotations are combined into three nested subregions: Whole Tumor (WT), Tumor Core (TC), and Enhancing Tumor (ET). We use Dice score of these three nested subregions and their average value to evaluate the performance.

---

Table 4: Quantitative results the CMU-MOSI and IEMOCAP datasets. Bold: best results.
Underline: second best results.
CMU-MOSI
IEMOCAP
Method
Acc-2
Acc-7
F1
MAE
Corr
Acc
Text only
76.83
28.24
76.80
1.016
0.664
65.35
64.30
Audio only
64.12
23.04
66.96
1.451
0.510
52.18
50.14
Video only
62.00
21.65
66.18
1.441
0.499
54.55
53.97
Baseline
81.23
29.26
81.23
0.952
0.710
70.74
69.53
MRD
80.78
31.44
80.75
0.975
0.688
71.81
71.08
MSLR
81.22
31.00
81.19
0.950
0.704
71.99
70.96
AGM [1S]
72.35
71.94
UMT [7]
81.78
31,98
81.77
0.942
0.712
70.75
69.81
UME [7]
80.83
30.94
80.78
0.969
0.701
71.53
70.94
QMF [28]
72.08
71.64
ReconBoost [11]
73.14
72.71
CGGM
82.84
33.73
82.74
0.915
0.717
75.38
74.97
9:30
8ox
(a)
(b)
(C)
Figure 3: Changes in (a) performance, (b) gradient magnitude and (c) direction during training with CGGM. We get the results on CMU-MOSI dataset.
4.2 Implementation Details
Input. For UPMC-Food 101, we use extracted features as inputs. Specifically, we use the pre-trained bert-base-uncased model [5] to extract text features and use pre-trained VIT [6] on ImageNet to extract image features. For CMU-MOSI and IEMOCAP, we follow Guo [10] to extract acoustic, visual and textual features. For BraTS 2021, we use the preprocessed raw images as inputs.
Backbone. For UPMC-Food 101, CMU-MOSI and IEMOCAP, use transformer encoders [20] as modality encoders and the fusion module. For the BraTS 2021 dataset, we use DeepLab v3+ [4] as the encoders and several convolution layers the fusion module.
Training Details. For images in UPMC-Food 101 and BraTS 2021, we implement data augmentation strategies, including random cropping, random flipping, color jitter, adding noise, etc. To save memory, we consider BraTS 2021 as a 2D segmentation task by randomly slicing an image from loss function. For UPMC-Food 101 and
the 3D image. For CMU-MOSI,
L1 loss IEMOCAP, we use cross-entropy loss. For BraTS 2021, we use the combination soft dice loss and cross-entropy loss. Besides, we use the Adam optimizer for CMU-MOSI, the AdamW optimizer for UPMC-Food 101 and IEMOCAP, and the SGD optimizer for BraTS 2021. Other hyperparameters described Appendix B in detail.
4.3 Main Results
Comparison with the state-of-the-arts. We compare our CGGM with other methods to demonstrate
the effectiveness of our proposed method. For these four datasets, we compare CGGM with the model
training only using one modality, multimodal joint training (Baseline), Modality Random Dropout
(MRD). and Modality-specifec Learning Rate (MSLR) methods. Additionally, we compare CGGM with SOTA methods including G-Blending [22], Greedy [25], OGM [17], AGM [15], PMR [8]. UMT [7]. UME [7]. QMF [28] and ReconBoost [1]]. Table 2, 3 and 4 present the results of

---

(a) UPMC-Food 101
(b) CMU-MOSI
(c) IEMOCAP
(d) BraTS 2021
1-SNE visualization of the gradients of classifiers and the unimodal gradients. Each point represents a gradient vector or matrix of batch of data.
Figure 4:
Table 5: Accuracy IEMOCAP. f1. f2 and fa represent the audio, video and text classifier, respec- tively. We train three separate models in unimodal training.
unimodal training
multimodal training
CGGM
audio
video
text
f1
f2
f3
f1
f2
f3
52.18
54.55
65.35
50.59
53.04
67.15
54.95
56.77
67.39
CGGM and its compared methods. Compared with the baseline method, CGGM brings significant improvements to the performance of the model, which demonstrates the effectiveness of our proposed method. Besides, CGGM takes both gradient magnitude and direction into consideration, thus making it outperform other gradient modulation methods consistently in all four datasets. Most importantly, CGGM can be easily applied various tasks and has good performance, including classification tasks, regression tasks, segmentation tasks, etc. Meanwhile, CGGM has no limitations on the optimizer, loss function and the number of modalities.
Effectiveness of CGGM. In Figure 3, we visualize the changes in accuracy, gradient magnitude and direction during training with CGGM. Compared with Figure 2(a), the accuracy of text modality in Figure 3(a) does not increase very fast with CGGM, which indicates that CGGM imposes constraints to the dominant modality during the optimization process. Besides, the accuracies of all the modalities and the fusion improves, indicating the effectiveness of CGGM. Additionally, in Figure 2(b), the dominant modality always has the largest gradient while in Figure 3(b). the gradient magnitude of the text modality decreases at first, indicating that CGGM slows down its optimization and accelerates other modalities' optimization, helping each modality learn sufficiently, thus improving the multimodal performance. In case of gradient direction, in Figure 2, the similarity between audio modality and the fusion is always less than 0 during the training process, indicating an opposite optimization direction between the unimodal and multimodal, thus hindering the optimization process. In Figure 3, we observe the multimodal direction consistent with all modalities, indicating that the multimodal branch utilizes unimodal information efficiently.
4.4 Classifier Performance and Gradient Direction
Classifier performance. In Table 5, we present the accuracy of classifiers in different situations. Unimodal training can be considered a baseline that fully utilizes the unimodal information. Compared with unimodal training, the accuracies of fi and f2 multimodal training drop slightly while the accuracy of fa increases slightly. This demonstrates that multimodal training can not fully utilize the information from audio and video, indicating that they are ontiiiopti The improvement of fa indicates that the text modality is fully exploited and learns some information from the other two modalities. In contrast, the accuracies of the three classifiers in CGGM all improve. This suggests that during the balancing process, the fusion can fully utilize the information from all the modalities, thus in turn making the encoders of three modalities fuse the information from other modalities during backpropagation. Therefore, the accuracies of all the three classifiers improve correspondingly. This also validates the effectiveness of CGGM.
Classifier gradient direction. In Section 3.3.2, we propose to use the gradients of classifiers to represent the unimodal gradients. In this subsection, we give a visualization of the gradients of classifiers and the unimodal gradients. Specifically, for every batch of data, we input them into

---

Accuradyy
Accuracy
soore
Improved perormacee
Impooved perforance
2.3
3.0
1.5
1.0
0.0
008
0.8
1.0
1.1
1.2
1,3
1.4
(a) Impact p
Impact of
Figure 5: The improved performance with different P and À compared to the joint training baseline.
the model get representations h2 which are then fed into the classifiers f; to get the gradients of classifiers. Then we input h; of only one modality into the fusion module S2 to get the unimodal gradients. We use t-SNE [19] to visualize the gradient vectors. Figure 4 shows the visualization results on the four datasets. As shown in the figure, for each modality, the unimodal gradient vectors and the gradient vectors of the corresponding classifier are very close to each other, demonstrating that it is meaningful to use the gradients of the classifiers to represent the unimodal gradients.
4.5 Ablation Study
Gradient magnitude and direction. To measure the contribution of gradient magnitude modulation and gradient direction modulation separately, we present our ablation results on IEMOCAP in Table 6. Compared with the baseline in the first row, modulating the magnitude of the gradients brings more improvements to the performance of the model than modulating the direction of the gradients. Compared with the CGGM performance in Table 4, the combination of modulating gradient magnitude and gradient directions furthermore enhances the performance of the model.
Scaling hyperparameter p. To explore the impacts of the scaling hyperparameter p on select seven different values of and present our results IEMOCAP in Figure 5(a). We discover that the accuracy improves with the increase of p before hitting the highest value when = 1.2. Then, the accuracy drops with the increase of p. Compared to the baseline, modulating the magnitude of the gradients brings consistent improvements regardless of how big p is taken. Intuitively, the larger the p. the larger the modification to the gradients. Therefore, the results in the table indicate that we need to carefully choose a p avoid modifications that are too large or too small.
Table 6: The benefits of modulating the magnitude of the gradients and the directions of the gradients.
the model's performance,
Model
Acc
F1
Baseline
70.74
69.53
CGGM (p = 1.0, X = 0)
72.35
71.56
CGGM (p None, x 0.1)
72.41
72.07
CGGM 1.0, À 0.1)
73.74
73.18
Loss trade-off X. X measures the strength we modulate the directions the gradients. We select six different values of À and present the results IEMOCAP in Figure 5(b). As shown in the figure, when À is 0.01 or 0.25, the accuracy will decrease slightly. When is too small, the modulation is insufficient and could influence the optimization process. When À is too large, the modulation is large and will influence the task loss, thus making optimization deviate from the task.
Conclusion
In this paper, we propose CGGM, novel strategy balance the multimodal training process. Compared to existing gradient modulation methods, CGGM has no limitations on the loss functions, the optimizer, the number of modalities, etc. Moreover, we consider both the magnitude and direction of the gradients with the guidance of the classifiers. Extensive experiments and ablation studies fully demonstrate the effectiveness and universality of CGGM. However, CGGM also has a limitation. CGGM needs to leverage extra classifiers to implement gradient modulation. Although these classifiers are lightweight, they still lead to more computational resources. We lead this challenging problem to future work.
10

---

Acknowledgement
This work was supported by National Key R&D Program of China under Grant No.20222D0162000
References
[1] Uijwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Crames Felipe C Kitamura, Sarthak Pati, et al. The rsna- asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02311 2021.
[2] Hedi Ben- Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2612-2620, 2017.
[3] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Ebrahim (Abe) Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth Narayanan. lemocap: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42: 335-359, 2008.
[4] Liang- Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018.
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1000000119999 1199,, 2020.
[7] Chenzhuang Du, Jiaye Teng, Tingle Li, Yichen Liu, Tianyuan Yuan, Yue Wang, Yang Yuan, and Hang Zhao. On uni-modal feature learning in supervised multi-modal learning. In International Conference on Machine Learning, pages 8632-8656. PMLR, 2023.
[8] Yunfeng Fan, Wenchao Xu, Haozhao Wang. Junxiao Wang, and Song Guo. Pmr: Prototypical modal rebalance for multimodal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[9] Jie Fu, Junyu Gao, Bing-Kun Kun Bao, and Changsheng Xu. Multimodal imbalance-aware gradient modulation for reekkl--supervisee audio-visse visual video parsing. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[10] Zirun Guo, Tao Jin, and Zhou Zhao. Multimodal prompt learning with missing modalities for sentiment analysis and emotion recognition. In Proceedings of the 62nd Annual Meeting the Association for Computational Linguistics (Volume Long Papers), pages 1726- 1736, 2024.
[11] Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boosting can achieve modality reconcilement. In The Forty-first International Conference on Machine Learning, 2024.
[12] Yu Huang. Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. Modality competi- tion: What makes joint training of multi-modal network fail in deep Marningg(provably In International Conference on Machine Learning, pages 9226-9259. PMLR, 2022.
[13] Tao Jin, Weicai Yan, Ye Wang. Sihang Cai, Shuaiqifan, and Zhou Zhao. Calibrating prompt from history for continual vision-language retrieval and grounding. In ACM Multimedia 2024, 2024.

---

[14] Hamid Reza Vaezi Joze. Amirreza Shaban, Michael L Iuzzolino, and Kazuhito Koishida. Mmtm: Multimodal transfer module for enn fusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 33289-13299 2020.
[15] Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou. Boosting multi-moda model performance with adaptive gradient modulation. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 22214-22221 2023.
[16] Paul Pu Liang, Ziyin Liu, AmirAli Bagher Zadeh, and Louis-Philippe Morency. Multimodal language analysis with recurrent multistage fusion. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun iich Tsujii, editors, Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 150-161, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/11DD18-000 URL
org/D18-110 1014.
[17] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. Balanced multimodal learning via on-the-fly gradient modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8238 8247, 2022.
[18] Yao-Hung Hubert Tsai, Shaojic Bai, Paul Pu Liang. J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Anna Korhonen, David Traum, and Lluís Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558-6569, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1/886///1/11111161 1656. URL https: / /aallalamm aallanholooooy org/P19-165 1656.
[19] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (86)) 2605, 2008.
[20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[21] Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, and Frédéric Jurie. Centralnet: multi- layer approach for multimodal fusion. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[22] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12695-12705, 2020.
[23] Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frédéric Precioso. Recipe recognition with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia Expo Workshops (ICMEW), pages 1-6, 2015.
doi: 10. 10.109//CMEW.2055716999
[24] Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis- Louis-Philipee Philippe Morency. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7216-7223, 2019.
[25] Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof Geras. Characterizing and overcoming the greedy nature of learning multi-modal deep neural networks. In International Conference on Machine Learning, pages 24043-24055 PMLR, 2022.
[26] Weicai Yan, Ye Wang, Wang Lin, Zirun Guo, Zhou Zhao, and Tao Jin. Low-rank prompt interaction for continual vision-language retrieval. In ACM Multimedia 2024, 2024.
[27] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31 (6):82-88, 2016. doi: 10.1009/MSS.006..90
[28] Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng. Provable dynamic fusion for low-quality multimodal data. In International conference machine learning, pages 41753 PMLR, 2023.
12

---

A Gradient Direction Modulation Details
For classification tasks, the classifier outputs more than one value. For example, for the UPMC- Food 101 dataset, Ji output 101 values for each piece of data. Therefore, we can define fi as fi = ( ,,?,,.,., ((")) where is the number of output. We calculate the gradients of the classifiers fü as:
aciot!
aciet
ac (m)
80%
86
80 ac(o
aC(o!!
ac
ae ((01)) aC(BF))
a0
00
00
(14)
0fi
ac
acie
@c (m)
a0
a6
a0
Similarly, the gradients of the fusion module classifier can be calculated as:
oc(ez())
@C(0x(2)
@c(
80
06
80.
(C(1=(1)
OC(O=(2)
ac())
V ((0 OL(O%)
802
06
805
(15)
0F
OC
aC(O
ac(O 80%
087
In order to calculate cosine similarity between these two terms, we flatten them into vectors and then Equation 12 calculate Lym-
B Implementation Details
Table 7: Main hyperparameters of the four datasets.
Hyperparameters
UMPC-Food 101
CMU-MOSI
IEMOCAP
BraTS 2021
batch size
128
32
64
optimizer
AdamW W
Adam
AdamW
SGD
base
1e-3
1e-3
le-3
1e-2
classifier Ir
5e-4
5e-4
5e-4
6e-3
weight decay gradient clip scheduler
3e-3
3e-4
0.8
0.8
0.8
StepLR 1.3
StepLR 1.3
StepLR 1.3
CosineLR
& warm-up epoch
1.2
0.15
0.20
0.15
0.10
10
epoch
80
30
30
150
Table 7 presents the main hyperparameters of the four datasets. Apart from the hyperparameters in the table, there are some task-specific hyperparameters.
For BraTS 2021, the start learning rate is set to 4e-4 with warm-up epochs to 1e-2 and the final learning rate is le-3. Besides, for the loss function, we use the combination of soft dice loss and cross-entropy loss, which can be represented as Ltast L \ LC We set À to 1. Particularly, we use a weighted cross-entropy loss function, where the weight 0.2, 0.3, 0.25 and 0.25 for the background, label 1, label 2 and label 3, respectively.
13

---

2.00
1.75
1.0
1.25
2 1.00
0.75
0.50
35:
2002
252
300
3.00
1.50
201
320
teratione
Berations
(a) w/o. CGGM
(b) gradient magnitude
1.75
3.75
3.25
3.00
3:25-
0.50
2.25
Eusion
25.33
2003
202
300
3.00
35.0
25.
2813
Derratine
Iterations
(c) gradient direction
(d) CGGM
Figure 6: Changes in loss during the training process.
video
Itorastons
Figure 7: Changes in balancing term during the training process.
C More Ablation Study
More visualizations of CGGM. We further visualize the loss changes in Figure 6. From the figure, we can observe that the loss of the dominant modality with CGGM implemented (b)-(d) will drop much slower than that in Figure 6(a). Besides, the losses of all modalities in (b)-(d) are smaller than those in (a), indicating the effectiveness of CGGM. Apart from the loss changes, we also visualize the changes in balancing term during the training process in Figure 7. When the value is higher than the red line, the modality is promoted. When the value lower than the red line, the modality is suppressed. In the first few iterations, the dominant modality is suppressed, ensuring that other modalities are fully optimized. During the optimization, balancing terms of three modalities turn up and down, ensuring each modality is sufficiently optimized.
Additional computational resources of classifiers. The additional classifiers will need more computational resources during training. However, during inference, the classifiers will be discarded. Therefore, they have no impact during the inference stage. We report the additional memory cost (MB) of the additional classifiers in Table 8. From the table, can observe that the additional
14

---

Table 8: Additional gpu memory cost (MB) of classifiers.
Setting
Food101
MOSI
IEMOCAP
BraTS
With classifiers
+8MB
+8MB
+8MB
+24MB
computational increase is low. There are two main reasons: (1) the classifiers or decoders are light with only a few parameters; (2) the classifiers only use the gradients to update themselves and do not pass the gradients to the modality encoders during backpropagation. Therefore, there is need to store the gradient for each parameter, thus reducing memory cost.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and Introduction
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The Conclusion Section
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The Method Section
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We present the detailed algorithm, codes and hyperparameters.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have released our codes.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detailed information in the implementation details.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: [NA]
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [No]
Justification: The experiments are not computationally expensive.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httilp::///nuuiippe http://pww....pppppoc/ccccccaaaa
16

---

Answer: [Yes]
Justification: [NA]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [NA]
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
17


Reviews:
[{"review_id": "CZQpY5H85H", "reviewer": "Reviewer_S6e7", "summary": "This paper proposes CGGM, a novel strategy to balance the multimodal training process. Compared with existing methods, it can deal with the unbalanced multimodal learning problem with different optimizers, takes,  and more than two modalities.", "questions": "see weakness", "limitations": "No", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation is sufficient and the experiments on different tasks and datasets prove that the proposed method solves the problem well.\n\nCGGM stands out by considering both the magnitude and direction of gradients for balancing multimodal learning. This combined approach effectively addresses the modality competition problem and ensures that all modalities contribute equally to the model’s performance.", "weaknesses": "1. The reviewer is curious about the computational complexity of the additional classifier or decoder. Is there any experimental result? \n2. As present in Line 149, the classifier fi consists of 1-2 multi-head self-attention (MSA) layers and a fully connected layer for classification and regression tasks. Does this apply to all models or classification tasks? Why is it set up like this? Why not just set it to the same classifier structure as the multimodal head?\n\n3. What's the light decoder used for segmentation tasks?\n\n4. The introduction of unimodal classification may limit the learning of multimodal tasks, such as gradient conflicts. How to deal with this problem?\n\n5. PMR has also discussed the problem of gradient direction and introduced unimodal loss to assist multimodal learning. What's the difference between CGGM and PMR?\n\n6. The reviewer is concerned about the accuracy of using the difference between the two consecutive ε to denote the modality-specific improvement for each iteration. According to my experience, the loss of the dominant modality will quickly drop to the magnitude of 1e-2 to 1e-3, while the magnitude of the weak modality is around 1e-1. At this time, the loss change of the weak modality will be larger, and according to the author, it will be regarded as the dominant modality. I \n\n7. What's the performance of the proposed method on CRAME-D and AVE datasets?  They are also widely used in previous studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable time and constructive comments.\n\n**(W1): Computational complexity of the additional classifiers experimental results.**\n\nThe additional classifiers will need more computational resources during training. However, during inference, the classifiers will be discarded. Therefore, they have no impact during the inference stage. We report the memory cost (MB) of the additional classifiers in the table below.\n\n|       Setting       | Food101 | MOSI | IEMOCAP | BraTS |\n| :-----------------: | :-----: | :--: | :-----: | :---: |\n|  With classifiers   |  8846   | 3902 |  4446   | 18072 |\n| Without classifiers |  8838   | 3894 |  4438   | 18048 |\n\nFrom the table, we can observe that the additional computational increase is low. There are two main reasons: (1) the classifiers or decoders are light with only a few parameters; (2) the classifiers only use the gradients to update themselves and do not pass the gradients to the modality encoders during backpropagation. Therefore, there is no need to store the gradient for each parameter, thus reducing memory cost.\n\n**(W2): About the additional classifiers setting.**\n\nInitially, we set the classifier the same as the multimodal head. However, according to the experiments, we observe that there is little performance gain compared with the classifiers with only a few layers. The results are shown below.\n\n|    Classifier     | $f_1$ | $f_2$ | $f_3$ | Multimodal |\n| :---------------: | :---: | :---: | :---: | :--------: |\n|  multimodal head  | 55.1  | 56.2  | 67.5  |    75.6    |\n| CGGM (1-2 layers) | 54.9  | 56.8  | 67.4  |    75.4    |\n\nIn the table, $f_1,f_2,f_3$ represents the accuracy of three modality classifiers. As we can see, more layers do not bring much improvement to the model and will add additional computational resources and parameters. From another perspective, the function of the classifiers in our paper is the reflection of the utilization rates of different modalities but not their own accuracies. Therefore, if these classifiers can reflect the utilization rates of each modality, we can design them as light as possible. Additionally, as discussed in W1, the additional decoders will not bring much computational costs.\n\n**(W3): Light decoder in segmentation task.**\n\nIn PyTorch, we implement the decoder with two Conv layers with concatenation of low-level features generated by the encoders and interpolation functions.\n\n**(W4): Unimodal branch may limit the learning of multimodal tasks. How to deal with this?**\n\nThe unimodal classifiers have no influence on the multimodal tasks because the unimodal classifiers only use the independent loss function to update themselves and do not pass the gradients to the multimodal branch. The reason is that the classifier is designed to reflect the utilization of each modality. Besides, this design can reduce the computational resources because there is no need to store the gradient for each parameter.\n\n**(W5): What's the difference between CGGM and PMR?**\n\nThere are several important differences between CGGM and PMR:\n\n- PMR needs to calculate a prototype for each class. This indicates that PMR can only be applied to pure classification tasks. In contrast, CGGM can be applied to various tasks.\n- The whole PMR and its formulations are based on cross-entropy loss. In contrast, CGGM has no limitations for this problem.\n- PMR discusses the gradient direction, but it does not balance the multimodal learning from the perspective of direction explicitly. It proposes two loss terms for modal acceleration and regularization. PMR adds this regularization term in the first several epochs and needs to delete it manually to avoid performance damage. In contrast, CGGM can modulate magnitude and direction dynamically with the training process.\n\n**(W6): Concerns about two consecutive $\\epsilon$**\n\nYes. At initial iterations, if no constraints are added, the loss of the dominant modality will drop quickly in our experiments (See Fig.2 (a) in the additional PDF in the general response). However, our method can prohibit this quick process. From iteration 0, CGGM will calculate the $\\epsilon$. $\\epsilon$ serves as the constraint to the dominant modality. If the dominant modality has much more improvements than that of weak modalities, CGGM will modulate its gradients to slow down its update and accelerate the weak modalities' update. Therefore, after CGGM is implemented, the loss of the dominant modality will not drop quickly (See Fig.2 (b) in the additional PDF, the loss of text drops much slower than (a)). Besides, CGGM can calculate the balancing term dynamically during the training process according to the optimization situations. Therefore, although text is the dominant modality in a task, during the training process, the \"dominant modality\" always changes according to the dynamic balancing term in CGGM. At some time, text is the dominant modality and at another time, audio may become the dominant modality (See Fig.3 in PDF). We can also reach this conclusion by comparing the accuracy change in Fig.1 in the PDF and Fig.2 in the paper.\n\n**(W7): Performance on CREMA-D and AVE**\n\nWe conduct these experiments for more evaluations of CGGM. For backbones, we use ResNet as encoders. For both datasets, we adopt SGD as optimizers. For additional classifiers, we use an MLP layer. The results are shown below.\n\n|      | CREMA-D | AVE  |\n| :--: | :-----: | :--: |\n| None |  61.5   | 64.8 |\n| CGGM |  79.2   | 73.6 |\n\nThanks for your suggestion. We will incorporate these details in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kyDlPIYdvh", "reviewer": "Reviewer_evvZ", "summary": "This paper focuses on the notorious modal imbalance problem in multi-modal learning. To alleviate the modality imbalance, the proposed method modulates gradient magnitude and the directions of the gradient simultaneously. Experiments on various multi-modal datasets demonstrate the efficiency.", "questions": "In section 3.2, $\\mathcal{L}$ denotes both the overall empirical loss and the loss of individual samples. It is recommended to denote the loss of individual samples as $\\ell$.\n\nEquations (3) and (4) are incorrect. The chain rule of differentiation for a scalar with respect to multiple vectors should be applied, rather than the chain rule for a scalar with respect to another scalar.\n$$\n\\frac{\\partial z}{\\partial x} = \\left(\\frac{\\partial y}{\\partial x}\\right)^{T}\\cdot \\frac{\\partial z}{\\partial y}\n$$", "limitations": "None.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper explores an interesting problem. In the current joint learning paradigm, the dominant modality overpowers the learning process and the resulting gradient prohibits further exploration of the features in the weak modality.\n- This paper is well-written and easy to follow.", "weaknesses": "- **The motivation is not stated clearly.** From Eq(3) and Eq(4), we can observe that the gradient magnitude can affect the update of a specific modality. However, the explanation of how the gradient direction between the specific modality and their fusion influences the modality update is unconvincing.\n- **The experiments are not convincing.** The authors ignore recent state-of-the-art methods, such as UMT/UME [1], QMF [2], and ReconBoost [3]. It is recommended that the authors compare these methods. Additionally, the authors should plot the gradient direction, accuracy curve, and gradient profile after using their method and compare these to Fig. 2 to better highlight the effectiveness of their approach.\n- **The related work section lacks discussion on recent research.** UMT [1] distills well-trained uni-modal features to assist multi-modal learning. QMF [2] provides a quality-aware multimodal fusion framework to mitigate the influence of low-quality multimodal data. ReconBoost [3] finds that the major issue arises from the current joint learning paradigm. They propose an alternating learning paradigm to fully harness the power of multi-modal learning.\n- Some notations are confusing. Please see the questions below.\n\nFor now, I recommend a borderline for this paper, leaning to reject. If the concerns in weakness can be addressed in the rebuttal phase, I am willing to raise my concern and accept this paper.\n\n[1]  On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.\n\n[2] Provable Dynamic Fusion for Low-Quality Multimodal Data. ICML2023\n\n[3] ReconBoost: Boosting Can Achieve Modality Reconcilement. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable time and constructive comments.\n\n**(W1): The explanation of how the gradient direction between the specific modality and their fusion influences the modality update.**\n\nIn Eq(3) and Eq(4), we know that the parameter $\\theta$ and the term $\\partial \\mathcal F, \\partial \\mathcal \\Omega. \\partial \\mathcal \\phi$ are all vectors not scalars. Therefore, these terms already include the direction information of the gradient. And just like the gradient magnitude, gradient direction can also affect the update of modalities from Eq(3) and Eq(4).\n\nAdditionally, we can use cosine similarity $\\cos(g_i,g_j)$ to represent the direction between two vectors. As shown in Fig2(c), $\\cos (g_a,g_{mil})<0$, indicating they optimize towards the opposite direction, thus hindering the gradient update for multimodal branch. When we implement CGGM, in Fig.1(c) in PDF, $\\cos(g_i,g_{mil}) > 0$ for all modalities, indicating they optimize towards the same direction.\n\n**(W2): Experiments.**\n\nThanks for the reminder. We will include these additional baselines into our papers. Since MOSI is a regression task where the label is a score and loss function is mean absolute error, methods such as QMF and ReconBoost where distributions are needed cannot be applied. Additionally, we implement a combination of Dice loss and entropy loss on BraTS dataset in our paper. Therefore, we modified it slightly to fit the methods.\n\n|            | UPMC-Food 101 | MOSI | IEMOCAP | BraTS |\n| :--------: | :-----------: | :--: | :-----: | :---: |\n|    None    |     90.3      | 81.2 |  70.7   | 69.2  |\n|    UMT     |     91.8      | 81.8 |  70.8   | 69.5  |\n|    UME     |     90.7      | 80.8 |  71.5   | 70.3  |\n|    QMF     |     92.9      |  -   |  72.1   | 71.6  |\n| ReconBoost |     92.5      |  -   |  73.1   | 71.8  |\n|    CGGM    |     92.9      | 82.8 |  75.4   | 73.9  |\n\nMeanwhile, we visualize the balancing process and the performance comparison in the PDF file which can be downloaded in the general response. Comparing the Fig.2 in the paper and Fig.1 in the additional PDF, we have several observations:\n\n- Performance: Compared with Fig.2 in the paper (around 0.7 after one epoch), the accuracy of text modality does not increase very fast with CGGM (around 0.55 after one epoch), which indicates that CGGM imposes constraints to the dominant modality during the optimization process (see Fig.3 in PDF). Besides, the accuracies of all the modalities and the fusion improves, indicating the effectiveness of CGGM.\n- Gradient magnitude: In Fig.2 in the paper, the dominant modality always has the largest gradient while in Fig.1 in the PDF, the gradient magnitude of the text modality decreases at first, indicating that CGGM slows down its optimization and accelerates other modalities' optimization, helping each modality learn sufficiently, thus improving the multimodal performance.\n- Gradient direction: In Fig.2 in the paper, $\\texttt{cos}(g_a, g_{mul})< 0$ during the training process, indicating an opposite optimization direction between the unimodal and multimodal, thus hindering the optimization process. In Fig.1 in the PDF, we observe that $\\texttt{cos}(g_{i}, g)> 0$ for all modalities, indicating that all modalities have the same direction with the multimodal fusion.\n\n**(W3): Recent work.**\n\nWe apologize for missing some recent work. We will add UMT/UME, QMF and ReconBoost methods in the related work section and incorporate the experimental comparisons with our methods.\n\n**(W4): Notations.**\n\nWe appreciate your feedback on the notation. We will update the $\\ell$ and equations for a better presentation. $(\\frac{\\partial\\mathcal F}{\\partial\\mathcal \\Omega})^\\top\\frac{\\partial \\ell(\\hat{y}^n,y^n)}{\\partial \\mathcal F}$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2z2bxE9rpm", "reviewer": "Reviewer_hVC3", "summary": "This paper proposed a balanced multi-modal learning method with Classifier-Guided Gradient Modulation (CGGM), considering both the magnitude and directions of the gradients, with no limitations on the type of tasks, optimizers, the number of modalities.", "questions": "The weaknesses above should be carefully considered.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Balanced multi-modal learning considering both the magnitude and directions of the gradients is a reasonable idea.\n3. The proposed method is easy to follow.", "weaknesses": "1. Balanced multi-modal learning considering the directions of the gradients is not novel. A previous work [1] have already analyze the issue of modality dominance caused by gradient conflicts. The difference and the comparison with this method should be considered in detail. Besides, the approach to controlling gradient magnitude is similar to the ideas of OGM[2] and PMR[3].\n2. This framework still does not explore the imbalance issue of multi-modal learning in more flexible task formats, such as the potential imbalance in tasks like AVQA and multi-modal generation. Expanding task formats to regression and segmentation tasks is only a minor improvement. Existing work can also be extended to these tasks with minor adjustments.\n\n[1] Wang, H., Luo, S., Hu, G. and Zhang, J., 2024, March. Gradient-Guided Modality Decoupling for Missing-Modality Robustness. In *Proceedings of the AAAI Conference on Artificial Intelligence* (Vol. 38, No. 14, pp. 15483-15491).\n[2] Peng, X., Wei, Y., Deng, A., Wang, D. and Hu, D., 2022. Balanced multimodal learning via on-the-fly gradient modulation. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition* (pp. 8238-8247).\n[3] Fan, Y., Xu, W., Wang, H., Wang, J. and Guo, S., 2023. Pmr: Prototypical modal rebalance for multimodal learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 20029-20038).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable time and constructive comments.\n\n**(W1): Difference and comparison with previous methods.**\n\nThe differences between CGGM and [1] are:\n\n- [1] considers a fixed loss term for direction during the training process while CGGM employs a dynamic loss term to balance the training. The fixed loss term in [1] might face a challenge during the training process because the gradient directions of different modalities are always changing. For example, PMR also employs a regularization loss term. However, this loss is only added in the first several epochs and needs to be deleted manually to avoid performance damage according to the optimization process. However, CGGM employs an adaptive loss function with the balancing term to train the model. It will dynamically adjust the loss according to the utilization rate of each modality during the training process.\n- [1] only considers the direction and overlooks the impact of magnitude while CGGM considers both.\n- [1] aims to address missing modality issues with direction loss while CGGM aims to address the imbalanced multimodal learning.\n\nWe compare the direction loss in [1] with the loss in CGGM and present the results on IEMOCAP below.\n\n|          | baseline | Loss in [1] | Dynamic loss in CGGM |\n| :------: | :------: | :---------: | :------------------: |\n| Accuracy |   70.7   |    71.6     |         73.3         |\n| F1 score |   69.5   |    70.9     |         72.8         |\n\nFrom the results, we can observe that our dynamic loss can adjust the directions of gradients better than the fixed loss term in [1].\n\nAdditionally, there are several important differences between CGGM and PMR:\n\n- PMR needs to calculate a prototype for each class. This indicates that PMR can only be applied to pure classification tasks. In contrast, CGGM can be applied to various tasks.\n- The whole PMR and its formulations are based on cross-entropy loss. In contrast, CGGM has no limitations for this problem.\n- PMR discusses the gradient direction, but it does not balance the multimodal learning from the perspective of direction explicitly. It proposes two loss terms for modal acceleration and regularization. PMR adds this regularization term in the first few epochs and needs to delete it manually to avoid performance damage. In contrast, CGGM can modulate magnitude and direction dynamically with the training process.\n\nBesides, there are also several important differences between OGM and CGGM:\n\n- OGM is based on cross-entropy loss and it can not applied to other tasks such as regression.\n- The balance term in OGM is designed for two modality situations. It calculates the discrepancy ratio between two modalities. However, when there are $M$ modalities, it needs to calculate a total of $M$ ratios for each modality. OGM does not consider how to use $M^2$ ratios to balance the training. In contrast, CGGM first calculates the term individually and then combines the term from different modalities to represent the utilization rate, indicating its universality.\n- OGM overlooks the influence of gradient direction.\n\n[1] 2024, March. Gradient-Guided Modality Decoupling for Missing-Modality Robustness. In *Proceedings of the AAAI Conference on Artificial Intelligence*\n\n**(W2): More flexible tasks.**\n\nTo show the universality and superiority of CGGM, we conduct experiments on more flexible tasks: multimodal retrieval tasks and video question answering. Specifically, we use MSRVTT and MSRVTT-QA datasets. For multimodal retrieval task, we use the common multimodal retrieval transformer MMT pre-trained on HowTo100M as the backbone. We used extracted features for the retrieval task. There are seven modalities in the extracted MSRVVT (motion, audio, scene, ocr, face, speech and appearance features). For video QA, we use the pre-trained VALOR-B as the backbone. For MMT and VALOR-B, they both consist of modality encoders and a multimodal fusion or decoder for downstream tasks. And we initialize the unimodal classifier with the multimodal fusion of the model. The results are shown below.\n\n|      |                 | Text$\\rightarrow$Video |                   |                 | Video$\\rightarrow$Text |                   |\n| :--: | :-------------: | :--------------------: | :---------------: | :-------------: | :--------------------: | :---------------: |\n|      | R@5($\\uparrow$) |    R@10($\\uparrow$)    | MnR($\\downarrow$) | R@5($\\uparrow$) |    R@10($\\uparrow$)    | MnR($\\downarrow$) |\n| None |      54.1       |          67.3          |       26.9        |      54.8       |          66.9          |       23.9        |\n| CGGM |      59.2       |          70.2          |       21.2        |      59.6       |          69.8          |       20.1        |\n\n|           | None | CGGM |\n| --------- | ---- | ---- |\n| MSRVTT-QA | 46.7 | 47.2 |\n\nFrom the results, we can observe that even in flexible tasks, CGGM can also improve the performance of the baseline model. Due to our idea of difference of evaluation metrics, CGGM can be easily applied to these flexible tasks, indicating its universality and effectiveness.\n\nThank you for your valuable feedback. We will incorporate these details in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GHe5eQsVah", "reviewer": "Reviewer_kP2D", "summary": "This paper proposes a balanced multimodal learning method. Compared to existing methods that only consider the gradient size, it also considers the direction of the gradient.", "questions": "1. It is interesting that the author calculates the improvement of each modality (such as the change in accuracy) instead of the current performance. However, it is hoped that more convincing theoretical proofs can be added.\n2. It is recommended to visualize changes in indicators before and after modulation (such as utilization rate, etc.), including only adjusting the gradient size, gradient direction, and both. It is best to also add the performance change process for each modality to avoid \"using the indicators proposed by yourself to measure your performance.\"\n3. Can it be implemented using only one classifier, instead of one classifier for each modality?\n4. Please add a brief introduction to the experimental comparison method, such as AGM, PMR, etc.\n5. The method in this paper is similar to the OGM method, but the original OGM also includes a part that uses Gaussian noise to improve generalization. Is this part of the method used in this paper? Can you add relevant discussions?", "limitations": "Yes, the authors have adequately addressed the limitations", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The experiment includes multiple data sets and multiple tasks.", "weaknesses": "There is less visualization analysis of the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable time and comments.\n\n**(W1):Theoretical analysis.**\n\nWe provide an analysis of why the combination of improvement can balance the training. In Sec 3.2, we know the dominant modality will be updated faster than others, which makes the gradient $\\partial\\Omega/ \\partial\\phi$ much larger. Therefore, according to Eq.(3,4), $\\Delta\\theta^{\\phi_i}$ will be larger than other modalities, indicating a larger step towards the optimal, which in turn influences $\\Delta\\epsilon$ (i.e. $\\partial\\Omega/ \\partial\\phi(\\uparrow)\\rightarrow\\Delta\\theta^{\\phi_i}(\\uparrow)\\rightarrow\\Delta\\epsilon(\\uparrow)$ ). Then, for modality $i$, change in balance term will be $\\frac{\\sum_{k=1,k\\ne i}^M\\Delta\\epsilon_k}{\\sum_{k=1}^M\\Delta\\epsilon_k}=\\frac{\\sum_{k=1}^M\\Delta\\epsilon_k-\\Delta\\epsilon_i}{\\sum_{k=1}^M\\Delta\\epsilon_k}=1-\\frac{\\Delta\\epsilon_i}{\\sum_{k=1}^M\\Delta\\epsilon_k}$. If $i$ is the dominant modality ($\\Delta\\epsilon_i(\\uparrow)$ increases), then $\\frac{\\Delta\\epsilon_i(\\uparrow)}{\\sum_{k=1}^M\\Delta\\epsilon_k(-)}$ will increase, which proves that the balance term $1-\\frac{\\Delta\\epsilon_i}{\\sum_{k=1}^M\\Delta\\epsilon_k}$ will decrease.  According to CGGM update rule ($\\theta^{\\phi_i}=\\theta^{\\phi_i}-\\rho (1-\\frac{\\Delta\\epsilon_i}{\\sum_{k=1}^M\\Delta\\epsilon_k})\\nabla g_i$), $\\Delta \\theta^{\\phi_i}$ will decrease, slowing down the optimization of the dominant modality. For other modalities, $1-\\frac{\\Delta\\epsilon_i}{\\sum_{k=1}^M\\Delta\\epsilon_k}$ will increase, thus accelerating their optimization. During training, the balancing term keeps changing according to the optimization, thus making all modalities sufficiently optimized.\n\n**(W2):Visualization.**\n\nWe have uploaded figures in PDF in the general response. We have made the following visualizations: Change in losses under four scenarios. Change in balancing term. Change in acc, gradient size and direction. Compared with Fig.2 in paper (0.7 after 1 epoch), the accuracy of text modality does not increase very fast with CGGM (0.55 after 1 epoch), indicating CGGM imposes constraints to it during optimization(see Fig3 in PDF). In Fig2 in paper, the dominant modality always has the largest gradient while in Fig.1 in PDF, the gradient magnitude of the text modality decreases at first, indicating that CGGM slows down its optimization. In Fig2 in paper, $\\texttt{cos}(g_a, g_{mul})< 0$ during training, indicating an opposite optimization direction between the unimodal and multimodal, thus hindering the optimization process. In Fig.1 in PDF, we observe that $\\texttt{cos}(g_{i}, g)> 0$ for all modalities, indicating that all modalities have the same direction with the multimodal. In Fig.2 in PDF, we can observe the loss of the dominant modality will drop much slower than that in Fig2a. Besides, the losses of all modalities in Fig2(bcd) are smaller than those in (a), indicating the effectiveness of CGGM. In Fig.3 in PDF, when the value is higher than the red line, the modality is promoted. When the value is lower than the red line, the modality is suppressed. In the first few iterations, the dominant modality is suppressed, ensuring that other modalities are fully optimized. During the optimization, balancing terms of three modalities turn up and down, ensuring each modality is sufficiently optimized.\n\n**(W3):Using only one classifier instead of one classifier for each modality**\n\nIt is difficult for only one classifier to catch the unimodal gradients and features accurately. During optimization, the classifier takes all modalities as input and absorbs multimodal information, thus failing to reflect unimodal utilization rates effectively. The comparison on IEMOCAP is shown below.\n\n|      | 3 classifiers | 1 classifier |\n| :--: | :-----------: | :----------: |\n| Acc  |     75.4      |     72.1     |\n\nBesides, additional classifiers does not require much memory, for classifiers do not pass gradients to modality encoders during backpropagation. \n\n|            | No classifier | 3 classifiers |\n| :--------: | :-----------: | :-----------: |\n| Memory(MB) |     4438      |     4446      |\n\n**(W4):Brief introduction to comparison methods.**\n\nThanks for your suggestion. G-Blending computes an optimal blending of modalities based on their overfitting behaviors to balance the learning. Greedy proposes conditional learning speed to capture the relative learning speed to balance the learning. OGM balances the multimodal learning by monitoring the discrepancy of their contribution to the learning objective with gradient enhancement. AGM proposes a metric built on mono-concept to represent competition state of a modality. PMR introduces prototypes for each class, accelerating the slow-learning modality by enhancing its clustering toward prototypes and reducing the inhibition from dominant modality with prototypical regularization.\n\n**(W5):Discussion on OGM and CGGM.**\n\nThere are crucial differences between OGM and CGGM:\n\n- OGM is based on cross-entropy loss and it cannot applied to other tasks such as regression.\n- OGM calculates the discrepancy ratio between two modalities. However, when there are $M$ modalities, it needs to calculate a total of $M$ ratios for each modality. OGM does not consider how to use $M^2$ ratios to balance the training. In contrast, CGGM first calculates the term individually and then combines them to represent utilization rate, indicating its universality.\n- OGM overlooks the influence of gradient direction.\n\nOGM also includes a part that uses Gaussian noise to improve generalization. We do not use this because this method is based on SGD generalization ability which follows a Gaussian distribution when the batch size $m$ is large enough. However, in many tasks, other optimizers may be chosen and batch size may be small. Therefore, adding Gaussian noise may hinder the optimization process. For example, we use AdamW on Food101 and when we add gaussian noise to the gradient, we find the accuracy drops from 92.9 to 92.5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their great effort and constructive comments on our manuscript. During the rebuttal period, we have been focusing on these beneficial suggestions from the reviewers and doing our best to add several experiments and revise our manuscript.

According to the reviewers' suggestions, we have included more recent baselines and more flexible tasks, which demonstrates the effectiveness and universality of CGGM. Besides, we have included several visualizations for a deeper analysis of CGGM. These visualizations include:

- Visualization of the changes in losses under four different scenarios.
- Visualization of the changes in the balancing term.
- Visualization of the changes in performance, gradient size, and direction.

These visualizations can be downloaded in this response. We believe these addition will make our experiments and methods more comprehensive and insightful. Additionally, we have made further clarifications about the setting of classifiers, the additional computational resources required by the classifiers, the differences between CGGM and other methods, as well as the theoretical analysis of CGGM. These modifications aim to make our manuscript more solid and convincing.

Meta-review: This paper was reviewed by four experts in the field. The paper received mixed reviews of Borderline Reject, Accept, Borderline Accept, and Borderline Accept after the rebuttal. During the rebuttal, two reviewers (evvZ, hVC3) raised their scores and the other two reviewers did not respond to the rebuttal.

The AC carefully read the paper, the reviews, and the authors' responses. Reviewer S6e7's concern is mainly about clarification of technical details. The AC believes that the authors' rebuttal should address the questions. Therefore, the decision is to recommend the paper for acceptance. The reviewers did raise some valuable suggestions in the discussion that should be incorporated in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.

Why not higher: Many concerns were raised by reviewers. The authors' rebuttal needs to be incorporated into the final paper to improve the work.
Decision: Accept (poster)