Facilitating Multimodal Classification via Dynamically Learning Modality Gap
Authors: Yang Yang
Fengqiang Wan
Qing-Yuan Jiang
Yi Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Learning
Modality Gap
Multimodal Imbalance
Primary area: deep_learning_architectures
TL;DR: A novel multimodal learning method integrates contrastive learning and supervised learning to address multimodal imbalance problem.

Abstract: Multimodal learning falls into the trap of the optimization dilemma due to the modality imbalance phenomenon, leading to unsatisfactory performance in real applications. A core reason for modality imbalance is that the models of each modality converge at different rates. Many attempts naturally focus on adjusting learning procedures adaptively. Essentially, the reason why models converge at different rates is because the difficulty of fitting category labels is inconsistent for each modality during learning. From the perspective of fitting labels, we find that appropriate positive intervention label fitting can correct this difference in learning ability. By exploiting the ability of contrastive learning to intervene in the learning of category label fitting, we propose a novel multimodal learning approach that dynamically integrates unsupervised contrastive learning and supervised multimodal learning to address the modality imbalance problem. We find that a simple yet heuristic integration strategy can significantly alleviate the modality imbalance phenomenon. Moreover, we design a learning-based integration strategy to integrate two losses dynamically, further improving the performance. Experiments on widely used datasets demonstrate the superiority of our method compared with state-of-the-art (SOTA) multimodal learning approaches. The code is available at https://github.com/njustkmg/NeurIPS24-LFM.

Facilitating Multimodal Classification via Dynamically Learning Modality Gap
Yang Yang', Fengqiang Wan', Qing-Yuan Jiang" Yi Xu²
(Nanjing University of Science and Technology
Daalan University of Technology
Lyyang, jiangay @njust. edu.cn, yxuddlut.edu..
Abstract
Multimodal learning falls into the trap of the optimization dilemma due to the modality imbalance phenomenon, leading unsatisfactory performance in real applications. A core reason for modality imbalance is that the models of each modality converge at different rates. Many attempts naturally focus on adjusting learning procedures adaptively. Essentially, the reason why models converge different rates is because the difficulty of fitting category labels is inconsistent for each modality during learning. From the perspective of fitting labels, find that appropriate positive intervention label fitting can correct this difference in learning ability. By exploiting the ability of contrastive learning to intervene in the learning category label fitting, we propose a novel multimodal learning approach that dy- namically integrates unsupervised contrastive learning and supervised multimodal learning to address the modality imbalance problem. We find that a simple yet heuristic integration strategy significantly alleviate the modality imbalance phenomenon. Moreover, we design a learning-basee integration strategy to inte- grate two losses dynamically, further improving the performance. Experiments on widely used datasets demonstrate the superiority of our method compared with state-of-the-art (SOTA) multimodal learning approaches. The code is available https: htmp httpa//.ooooiiiiiee//e////////--
Introduction
integrates heterogeneous information from different modalities to build an effective way to perceive the world. Over the past decades, multimodal learning has made incredible progress [I
Multimodal learning (MML) 3 38
31
[0
B and become a hot research topic with
wide range of real applications including image caption 6 14, cross-modal retrieval [2] 54 37]
vision reasoning 332 g
action recognition 23 and so on.
In multimodal learning, several recent studies
38 have revealed an interesting phenomenon, i.e., the performance of the multimodal model far from the upper bound or even inferior the uni- modal in certain situations. The root of this problem lies in the existence of the modality imbalance phenomenon B8 Coneretely, there commonly exists dominant modality and non-dominant modality in heterogeneous multimodal data. Multimodal learning usually adopts uniform objective. Due to greediness 442. the optimization tends to dominant modality while neglecting the non-dominant one during joint training, thus leading to unsatisfactory performance in real applications.
Recently, many impressive works 38 balance problem. Early pioneering approaches, such as gradient blending (G-Blend) 38 on-the-fly
24 have been proposed address the modality im-
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

gradient modulation (OGM) 31, adaptive gradient modulation (AGM) 24 and prototypical modal- ity rebalance (PMR) [3] focus on designing customized learning strategies for different modalities to adjust the optimization of dominant and non-dominant modality. These methods demonstrate that suppressing the optimization of the dominant modality can alleviate the modality imbalance problem to certain extent. Besides, several attempts, including uni-modal teachers (UMT) [[]] and balanced multimodal learning [42], try to introduce extra networks as an auxiliary module to facilitate multimodal learning.
Although the aforementioned approaches can boost performance in MML, these solutions are based on the phenomenon of inconsistent learn- ing speed itself and do not study the underlying
one-hot label
label free
66.92%
67.56%
causes of modality imbalance. We can't help but ask what is the essential reason behind this phenomenon. Is there a bias in the process of fitting category labels for different modalities? We carry out a simple experiment on Kinetic- sSounds dataset to seek answers. We adopt two types of labels to explore the influence of fitting labels. The first type is one-hot labels which indicate the category of each sample, where the loss is denoted as Ls. The second type is la- bel free, i.e., uniform label 1/c for all samples, where e denotes the number of categories. The second loss is defined as Lu- Furthermore, we define mixed loss 0.7L$ 0.3Lu, which is actually the label smoothing (35) strategy, by combining one-hot labels and uniform labels. The accuracy is reported in Figure | From Fig-
Accuracy
0.6
0.4
24.53%
10.36%
24.64%
0.2
2. 2.13 113
Ls
0.7 + 0.3Lu
Lu
Audio- Audio-Vidoo;
Audio:
Vidco
The influence of labels fitting on perfor- mance gaps (best view in color), where Ls and Lu denote the loss with one-hot labels and uniform labels (label free).
Figure 1:
ure II we can observe that with proper intervention by using uniform labels, the performance is slightly better than the model that fits one-hot labels. More importantly, the performance gap becomes smaller if we learn from uniform labels. This means that the difference between audio and video modalities is smaller in feature space when we reduce the weight of one-hot labels. The experiment inspires us that appropriate intervention label fitting can alleviate the difference in the learning ability of different modalities. In this way, we can reduce the modality performance gap and further mitigate the modality imbalance phenomenon as the smaller the difference in modality performance, the less severe the modality imbalance [39 20 [52 This also implies that fitting category labels is a core cause of modality imbalance multimodal learning.
How do we impose positive intervention in multimodal learning so that the impact of fitting labels on modality imbalance as low as possible without affecting the overall performance? For multimodal learning, although the models are learned from the heterogeneous data, we hope that multimodal data describing the same entity should be as close as possible in the feature space, which is usually modeled as contrastive learning [33]. Contrastive learning aims to learn similar representations for
data pairs of different modalities [2]. thus aligning the multimodal representations for all modalities.
Ideally, contrastive learning can also mitigate the effect of modality imbalance problem. Hence, we introduce contrastive learning to impose positive intervention in multimodal learning to alleviate the
impact of fitting labels.
In this paper, we propose novel multimodal learning approach by integrating unsupervised con- trastive learning and supervised multimodal learning dynamically. Specifically, after demonstrating the effectiveness of unsupervised contrastive learning in multimodal learning, we design two dynami- cal integration strategies, i.e., heuristic and learning- barning-based integration strategy. Our contributions are outlined as follows: (1). We observe a key phenomenon: fitting category labels leads to larger performance gap between different modalities. To the best of our knowledge, this is the first time that the modality imbalance problem has been analyzed from the perspective of category label fitting.
(2). We propose novel multimodal learning approach by integrating unsupervised contrastive learning and supervised multimodal learning. Two strategies are designed for dynamic integration. (3). Extensive experiments widely used datasets show that our proposed approach can significantly outperform other basclines to achieve ofate-of-the-ar performance.
2

---

2 Related Work
2.1 Multimodal Learning
Multimodal learning aims to leverage multimodal data from different sources to improve model per- formance. Based on the fusion strategy, multimodal learning approaches can be categorized into early fusion [38
52 50], late fusion 48 47 27]. and hybrid fusion [22 53 Early fusion methods aim to integrate multimodal features to study the interrelationship between different modalities with joint representations when features are extracted by encoders. Representative early fusion methods include G-Blend [38], association- -based fusion (AF) [26), and DOMFN [46). On the contrary, late fusion methods leverage the prediction of each model to make final decisions. Late fusion methods can be divided into two categories, i.e., soft late fusion and hard late fusion, where the former utilizes the confidence score to make decisions and the latter the category decision of each model. Pioneering late fusion methods include podality-sppccifie learning rate (MSLR) [47] Hybrid fusion methods try to amalgamate the advantages of both early and late fusion methods. Representative hybrid fusion methods include multimodal transfer module (MMTM) (22 and balanced multi-modal learning [42) Although these methods explore the algorithms and applications in multimodal learning, all of them assume that each modality can make sufficient contributions to achieve satisfactory performance during the training procedure.
46
2.2 Imbalanced Multimodal Learning
In reality, an obvious situation is that multimodal data and models diverse, which naturally leads to different contributions during the training procedure. Recent works (38) [1] 13 24 have
shown that modality imbalance is a ubiquitous phenomenon and often results unsatisfactory performance or even worse than unimodal algorithms in some cases. Considering the existence of dominant modality and non-dominant modality, early pionecring approaches [38
focus on adjusting learning speed for different modalities with customized learning strategies to balance the optimization of dominant and non non-dominant modality. For instance, G-Blend [3X proposes to minimize the overfitting- to- teeeetto-ggnnee to-generalization ratio (OGR) by using gradient blending technique based on the modality's 's overfitting behavior. OGM [3I] utilizes an on-the-fly fly gradient modulation strategy to control the modality's 's optimization procedure. To achieve the purpose of balanced multimodal learning, PMR [3 designs prototypical modal rebalance strategy to facilitate the learning of non-dominant modality. Other attempts 11 42
try to utilize extra networks to facilitate multimodal
learning. Concretely, UMT [I]] utilizes the teacher networks to distill the pretrained unimodal features to the multimodal network to tackle the modality imbalance problem. Balanced multimodal learning 42] utilizes the gradient norm and model parameters' norm to define conditional learning speed and uses guide the learning procedure. These methods alleviate the modality imbalance problem to a certain extent.
3 Methodology
We present proposed method in this section. Specifically, first present the problem definition of multimodal learning. Then, we introduce unsupervised contrastive learning to impose positive intervention in multimodal learning and propose two dynamical integration strategies to maximize the learning collaboration of unsupervised contrastive learning and supervised multimodal learning.
3.1 Preliminary
For the sake of simplicity, we use boldface lowercase letters like a and boldface uppercase letters like A to denote vectors and tensors, respectively. The i-th element of is denoted as a; Furthermore, we use || ||z to denote L2 norm of the vectors.
The goal of multimodal learning is to train a model to predict the category labels for given multimodal data. Without any loss of generality, we use X = {a:; ) to denote the training data points, where
each data point is with modalities, i.e., 2; (-, (3) pm j==1- The category labels are represented as Y {y, y. E {0, 1 Jez. where denotes the number of category labels.
3

---

For deep learning based multimodal approaches, we usually adopt deep neural network to extract representation from original space into feature space. We utilize (.)) to denote the feature extraction function for j-th modality. Given data point 2 the feature extraction be formed as:
2 = (x!")
where x, E Rd denotes the d-dimension feature vector of x!!" and Φ(3) denotes the parameters of j-th encoder. After vectors for modalities are extracted, we adopt a fusion function f(-) to fuse the different feature vectors, Then, we leverage fully-connected layer to map the vector into R This procedure can be formed as:
2: = f(z)
(m))
ÿ = sof tmax W z; + b).
Here, W E ReXD, b E R denote the weights and bias of the last fully-connected layer, respectively, and D denotes the dimension of 2,. Then, the objective function of multimodal learning can be formulated as:
Las(X,
Σ ==1
y log U,.
3.2 Integrating Unsupervised Contrastive Learning in MMI
To bridge the heterogeneous data in feature space, we utilize contrastive learning 33 multimodal learning. For a pair of data points {x]²) 2x J. we define the similarity as:
(00))
[z] (2(")122 (1) || l1 1121= 11
(1)
The modality matching objective function can be written as:
LMM(X)
1 log
exp(s(æ))
11) )/T) 1/T] + log exp(s(x) (1) x
)] }}
2n
Σn exp(s(x:)) (3) (1) 1TT)
Σn exp(s () 2, )/)
(1)
where the temperature parameter and n& denotes the batch size. By integrating the classification loss and modality matching loss, we can get the following objective function:
Liaal (1 - LCLs Y) + eLMM(().
where a denotes the weighted parameter between two losses.
3.3 Dynamic Integration
Although a fixed value of a allows the model to take into account both classification loss and modality matching loss, it cannot dynamically evaluate the weight of two losses during training. Hence, propose two strategies to adjust a dynamically to balance two losses.
Firstly, we utilize monotonically decreasing function to adjust the impact of category labels. The definition the function can be written as: at = w(),, where denotes the number of training epochs. this paper, we set ((t) = 1- c
Then, we further exploit learning based integration method by utilizing bi-level optimization strategy (36]. Specifically, while considering optimizing the multimodal classification loss LCLS, we use the minimum value of the total loss Toaal to restrict the feasible region of the parameters 0. In other words, we require the parameters not just to minimize classification loss but also to comply with a precisely defined constraint, i.c., simultancously minimize a composite loss function strategically engineered combination of modality matching loss and multimodal classification loss.
The specific formula is defined follows:
min LCLS(0" (a)) s.t. (a) E argmin {(1 - LcLs((( + (LMM
0SaSi
Here, 0 denotes the parameters of multimodal models, and a emerges as a key parameter, delicately balancing modality matching loss and multimodal classification loss to direct the model toward an

---

Algorithm 1: The Proposed Algorithm.
Input :Training set X, labels y/ method.
Output:I.caman parameters {0) of models.
INIT initialize parameters 0, parameter a, maximum iterations learning rale T/o-
for L to do
/* updating neural network parameters 0. */
for i i= to Inner_lies do
Calculate total Ioss Ltoui by forward phase.
Update parameters according to its gradient.
end
updating weighting parameters based on the chosen method.
method = 'learning Staamnnng-based then
Calculate gradient appriximation:
VLeis(((a)) = - V2 o Update according
L VoLas(XX
a To (o())(((a)).
Clip a into [0, 1|: max(O, min(1, o)).
clse if method = 'heuristic" then
Update a according to: a = 1 - 1/1
end end
optimal balance where both types of loss are effectively managed. The optimal parameter set, 8* (a), thus represents fine-tuned balance that, for any chosen a, strategically minimizes this composite loss. Within Equation (2). LCLS is pivotal for guiding classification accuracy, while LMM enhances
the model's ability establish meaningful connections across different modalities.
We utilize an approximation method proposed by [T6] solve bi-level optimization problem (2)
efficiently. Specifically, the gradient of (f(a)) with respect to a can be approximated by:
(o(a())(a)) = - L V2 XX, Y).
(3)
Based the approximation Equation in (3).
the gradient descent method to optimize a.
After defining the updating strategy for a, we utilize an alternating algorithm between model parameters 0 and to perform model learning. Specifically, algorithmic process iteratively refines the model parameters and the parameter a, employing nested loop structure where the inner loop focuses the currently given minimize total loss to update and the outer loop updates by function or bi-level policy to minimize classification losses. Through this structured optimization, the model achieves delicate balance between multimodal matching and classification losses. The overall algorithm of our model is outlined in Algorithm (T). where we utilize method to indicate the chosen updating strategy in practice. Moreover, the complexity of bi-level optimization (16) and algorithm in Algorithm (1) is On), which makes our approach highly practical.
Experiments
4.1 Experimental Setup
We select six widely used datasets, including KineticsSounds 22, CREMA-D (5). Sar- casm 4 Twitter2015 49). NVGesture (4), and VGGSound 77 datasets, validate our proposed method. Among these datasets, the KineticsSounds, CREMA-D and VGGSound datasets consist of both audio and video modalities. The KineticsSounds dataset, which contains 19,000 video clips categorized into 31 distinct actions, aims to advance video action recognition. It is divided into training set of 15,000 clips, a validation set of 1,900 clips, and test set of 1,900 clips. The CREMA-D dataset, encompassing 7, 7,442 clips, is divided into six emotional categories to enhance speech emotion analysis, with 6,698 clips in the training set and 744 clips in the test set. The VGGSound dataset, which contains 310 classes and a wide range of audio events in everyday life, is relatively large dataset. It includes 168,618 videos training and validation, and 13,954 videos for testing. Furthermore, the Sarcasm and Twitter2015 5 datasets consist of image and text modalities. The Sarcasm dataset offers a compilation of 24,635 text-image pairs, divided into 19,816 for the training set, 2,410 for the validation set, and 2,409 for the test set. The Twitter2015 dataset contains 5,338
Datasets:
5

---

Table 1: Comparison with SOTA multimodal learning methods. The best results are highlighted in bold. The underlining symbol denotes the second best performance. The results with gray background are based on MML but perform worse than the best unimodal approach.
KinetiesSounds
CREMA-D
Sarcasm
Twitter2015
NVGesture
Method
ACC
MAP
ACC
MAP
ACC
F1
ACC
F1
ACC
F1
Unimodal-1
54.12%
56.69%
63. 63.17%
68.61%
81.36%
80.65%
73.67%
68. 48.49%
78.22%
78.33%
Unimodal-2
55.62%
58.37%
45.83%
58. 79%
71.81%
70.73%
58.63%
43.33%
78.63%
78.65%
Unimodal-3
81.54%
81.83%
Concat
64.35%
71.31%
63.31%
68.41%
82.86%
82.43%
70.11%
63.86%
81.33%
81.47%
Affine
64.24%
69.31%
66.26%
71.93%
82.47%
81.88%
72.03%
59.92%
82.78%
82.81
Channel
63.51%
68.66%
66.13%
71.75%
81.54%
81.57%
ML-LSTM
63.84%
69.02%
62.94%
64.73%
82.05%
70.73%
70.68%
65.64%
83.20%
83.30%
Sum
64.97%
71.03%
63.44%
69.08%
82.94%
82.47%
73.12%
66.61%
82.99%
83.05%
Weight
65.33%
71.33%
66.53%
73.26%
82.65%
82.19%
72.42%
65.16%
83,42%
83.57%
ETMC
65.67%
71.19%
65.86%
71.34%
83.69%
83.23%
73.96%
67.39%
83.61 %
83.69%
MSES
64.71%
72.52%
61 01.56%
66.83%
84.18%
83.60%
71.84%
66.55%
81.12%
81.47%
G-Blend
67.12%
71.39%
64.65%
68.54%
83.35%
82.71%
74.35%
68.69%
82.99%
83.05%
OGM
66.06%
71.44%
66.94%
71.73%
83.23%
82.66%
74.92%
68. 18.74%
Greedy
66.52%
72.81%
66.64%
72.64%
82.74%
82.69%
DOMFN
66.25%
72.44%
67.34%
73.72%
83.56%
82.62%
74.45%
68.57%
MSLR
65.91% 71.96%
65.46%
71.38%
84.23%
83.69%
72.52%
64.39%
82.86%
82.92%
PMR
66.56%
71.93%
66.59%
70.36%
83.61%
82.49%
74.25%
68.62%
AGM
66.02%
72.52%
67.07%
73.58%
84.28%
83.44%
74.83%
69.11%
82.78%
82.82%
MLA
70.04%
74.13%
79.43%
85.72%
84.26%
83.48%
73.52%
67.13%
83.73%
83.87%
ReconBoost
70.85%
74.24%
74.84%
81.24%
84.37%
83.17%
74.42%
68.34%
84.13%
86.32% %
MMParcto
70.00%
78.50%
74.87%
75.15%
83.48%
82.84%
73.58%
67. 67.29%
83.82%
84.24%
69.05%
72.97%
72.15%
80.45%
84.12%
83.98%
73.87%
69.17%
83.24%
83.87%
Ours-H
+00.15%
+3.43%
+0.32%
+0.85%
+0.17%
+0.22%%
+0.35%
+0.26%
+0.07%
+0.18%
72.53%
78.38%
83.62%
90.06%
84.97%
84.57% %
75.01%
70.57% %
84.36%
84.68%
Ours-LB
1.0311%
1.0.37%
10.11%
1..0%%
10.27%
10.18%
10.16%
10.28%
10.14%
1..24%
text-image combinations from Twitter, with 3,179 in the training set, 1,122 in the validation set, and 1,037 in the test set. Lastly, the NVGesture dataset is used to construct research that goes beyond the limitation of two modalities. In this paper, we use RGB, Depth, and optical flow (OF) modalities for experiments, with 1,050 samples in the training set and 482 samples in the test set.
We select wide range of baselines for comparison. These baselines can be divided into two categories, i.e., traditional MML approaches and fusion methods with modal rebalancing strategies. The former category encompasses techniques like feature concatenation (CONCAT), affine transformation (Affine) [32]. channel-wise fusion (Channel) [22], multi-layer LSTM fusion (ML-
Baselines:
LSTM) (30), prediction summation (Sum), prediction weighting (Weight) [45] and enhanced trust modal combination (ETMC) [7]. And the latter category includes MSES [5),, G-Blend [38],
OGM 31. Greedy [1) DOMFN (46), MSLR [47). PMR [13, AGM 224 MLA [52]. Recon-
Boost 19 and MMPareto [40]. Other baselines including UMT [1 and QMF [51] are not adopted
as these two methods have been found to be outperformed by the adopted baseline ReconBoost.
Evaluation Metries: Following 331, we use accuracy (ACC) and mean Average Precision (MAP) as evaluation metrics for audio-video datasets. For text-image datasets, we adopt ACC and Macro Fl-score (F1) [4) ACC measures the proportion of correct predictions to total predictions, indicating the overall predictive accuracy. Macro Fl calculates the average of Fl scores across all categories, balancing precision and recall to evaluate performance evenly across classes. MAP represents the average precision across all categories, assessing the model's ranking ability for each category.
Implementation Details: In our experiments, we utilize raw data for experiments. Following BS) for the KineticsSounds and CREMA-D datasets, ResNet18 8 [8] serves as the foundational architecture for processing both audio and video data. For video analysis, we select 10 frames from each clip and subsequently sample three frames uniformly as inputs. We adapt ResNerl 8's input channels from three to one to accommodate our data format (7]. In terms of audio, we transform our sound recordings into spectrograms measuring 257 x 1004 for KineticsSounds and 257 x 299 for CREMA-D,

---

employing the librosa [29] library for conversion. For text-image datasets, our framework incorporates ResNet50 for images and BERT for text processing. We resize images to 224 x 224 and limit text sequences to a maximum length of 128 characters. Optimization for the audio-video datasets is conducted using stochastic gradient descent (SGD) with a momentum set to 0.9 and a weight decay parameter of 10 We initialize the learning rate to 10-2, progressively reducing it by factor of ten upon observing a plateau in loss reduction, with batch size of 256. For text-image datasets 44 49], we employ the Adam optimizer starting with learning rate of 10-4, with batch size of 128. For our methods, we run the experiment three times with different random seeds and present the detailed performance with mean and std. values to remove randomness. All models are trained on single RTX 3090 GPU.
4.2 Comparison with SOTA MML Baselines
The main results for all datasets, except VGGSound, are presented in Table where "Our-H" and "Ours-LB" de- note the proposed method based on heuristic strategy and learning-based strategy, respectively. In Table [ Unimodal-1 and Unimodal-2 refer to the audio and video modalities for audio-video datasets, and the image and text modalities for image-text datasets, respectively. For NVGesture dataset, Unimodal- respectively denote the RGB/OF/Depth.. From the results, we can derive the following observation: (1). Compared with all baselines including traditional multimodal learning approaches and fusion methods with modal rebalancing strategies, our pro- posed method with learning-basse strategy can achieve
Table 2: Results on VGGSound dataset.
Method
ACC
MAP
AGM
47.11%
51.98%
MLA
51.65%
54.73%
ReconBoost
50.97%
53.87%
MMPareto
51.25%
54.74%
Ours-H 50.42%
53.62%
Ours-LB
52.74%
55.98%
best performance by large margin in almost all cases. We can also find that the model with learning- based strategy can achieve better performance than that with heuristic strategy. (2). Across the 1witter2015 dataset, there is discernible trend where the optimal unimodal performance outstrips that of multimodal joint learning. Additionally, in other datasets, fusion methodologics devoid of rebalancing mechanisms manifest negligible enhancements relative to the foremost unimodal performance, notably on the CREMA-D and Sarcasm datasets. This shortfall originates from the prevalent challenge of modal imbalance. (3). Every modality rebalancing technique demonstrates significant improvements over traditional feature concatenation fusion. This finding not only un- derscores the detrimental impact of modal imbalance on performance but also corroborates the efficacy of the modality rebalancing approach. (4). For NVGesture dataset, differing from modal rebalancing methods restricted to scenarios with only two modalities, such Greedy, our approach with learning-based strategy can address challenges in scenarios involving more than two modalities and achieve best results. And our proposed method can outperform all baselines most cases.
For the relatively large dataset VGGSound, we select a set of recent algorithms, including AGM 24), MLA [52]. ReconBoost [19) and MMPareto [40), for experimental evaluation. The results shown in Table 2 From Table 2 we can see that our proposed methods can achieve the best performance in all cases compared with recent SOTA baselines on VGGSound dataset.
4.3 Ablation Study
To comprehensively assess the effectiveness of our proposed method, we conduct experiments to study the influence of main components, i.c., contrastive learning (CL) and dynamic integration (DI). The results are shown in Table 3, where the "CL" and "DI" denote that whether the contrastive learning and dynamic integration are applied during training. The unimodal MAP results are based on audio and video modalities for KineticsSounds and CREMA-D datasets, and unimodal FI results are based on image and text modalities for Sarcasm and Twitter2015 datasets. Please note that dynamic integration depends on the contrastive learning loss. Hence the method with dynamic integration but without contrastive learning cannot be performed. From T we can see that both contrastive learning and dynamic integration can boost performance in multimodal learning. Moreover, by integrating contrastive learning into multimodal learning, the performance gap between audio and video is greatly reduced.
7

---

Table 3: Results of ablation study. The symbols "CL" and "DI" denote that whether the contrastive learning and dynamic integration are applied during training.
Module
MAP/FI
Dataset
CL
DI
Multiple
Audio/Image
Video/Text
GAP
69.32%
48.82%
27.19%
21.63%
KineticsSounds
71. 71.76%
51.05%
47.05%
3.80%
78.97% % v
58.40% %
60.42%
2.02%
76.07%
70.97%
34.15%
36.82%
CREMA-D
X
86.32%
72.11%
52.51%
19.06%
90.06 %
75.27 %
67.36 %
7.91%
82.43%
62.81%
77.96%
15.15%
Sarcasm
83.10%
68.74%
80.72%
11.98%
84.57 %
74.53%
83.03
8.50%
X
X
63.86%
40.99%
68.38%
27.39%
Twitter2015
65.33%
46.84%
69.04%
22.20%
70.57% %
53.43 %
69.68 %
16.25 %
Table 4: Comparison of dynamic integration strategy on KineticsSounds and CREMA-D datasets.
Constant
Stepwise
Dynamic
Datasct
Modality
0.5
h(0)
h(1) h(0.05) h0.95)) Ours-H
Ours-La
Multiple 64.55%
64.70%
28.67%
65.17%
66.92%
66.01% 67.41% 69.32%
72.89%
KineticsSounds
Audio
49.17%
46.30% 34.11%
51.12%
52.34%
52.21% 53.41% 53.89%
$4.32%
Video
24.64%
44.02% 28.41%
41.21%
41.45%
42.31%
46.72%
49.18%
54.17%
Multiple
63.31%
70.45% 26.49%
66.45%
70.24%
69.11%
71.45% 72.39%
84.11%
CREMA-D
Audio
55.65%
60. 17% 33.15% 56. 19%
57.38%
58.099 09%
60.18% 61.89%
65.13%
Video 18.68% 42.54% 20.42%
45.14% 49.97%
46.41%
55.32%
57.14%
64.89%
4.4 Effectiveness of Integration Learning
We further study impact of different integration strategies for contrastive loss and classification loss. Specifically, we analyze three categories of integration strategy, i.e., constant, stepwise and dynamic strategy. For constant strategy, we assign a constant value to for the experiment and three sets of experiments
Analysis Integration Strategy:
0.5. and a = Here, "a = 0" denotes that we only perform supervised multimodal learning. Similarly, "a 1" denotes that only perform unsupervised contrastive lcarning. For stepwise strategy, define an indicator function h(p). where hu(p) denotes that a = p if the current epoch is less than half of the total epochs, otherwise strategy (Ours-H) H) and learning- based strategy (Ours-LB).
setting
O,
p. For dynamic strategy, assign value to heuristic
The experimental results are shown in Table From Table we draw the following observations: (1). Integrating multimodal learning and contrastive learning simultaneously with a constant ratio can slightly boost performance in some cases and greatly reduce the performance gap between audio and video. (2). general, the model with a stepwise strategy can outperform the model with constant strategy. Furthermore,
can find that the performance of the model with two-stage training, ie., ((0) or (1), is worse than that of the model which combines two losses with a constant value, ie.. ((0.05) or (00.95). (3). The overall performance of the model with the dynamic strategy is strategy can
better than that with the other strategy. Moreover, the model with the learning- achieve the best performance. And the performance gap of this model is nil negligible. The experimental results prove that the smaller the performance gap of the uni-modals, the better the overall performance of the model.
Change of for Learning- Strategy: To further observe the change of the optimal a during training, illustrate the change of a all datasets. The results are shown in Figure where

---

0.4
Bost
Image
Image
KinetiesSounds;
Sarcasm;
+w(t) =
0.2
CREMA
Twitte20115;
0.0
-02
.0.2
Gap Distance: 0.72
Distance:
0.9
0.8
(a). CONCAT
(b). G-Blend
0.6
0.3
Image
Audio
0.4
Audio
Image
0.4
0.2
0.2
0.3
0.1
-0.2
Distance: 0.98
Distance: 1.01
0.2 0.1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.4
.80
-90
#epoch//Hotal_.ee epochs
(c). MLA
(d). Ours
Figure 3: Change of on different datasets. We illustrate the value of the heuristic integration strategy for comparison.
Figure 2: Visualizations of the modality gap dis- tance on the CREMA-D dataset.
calculated by learning-bast based strategy, and wi() denotes the heuristic based strategy. As the total epochs for different datasets are different, we change the x-axis as the proportion of the current epoch, ie., #epoch/#total_ _eaoclss________eee From Figure 3 we can draw the following observations: (1). The general trend of the change for a is roughly the same on different datasets.
(2). The customized function wt)) close to the actual changes some extent, but there is still gap between the customized function and the actual changes. In practice, it is difficult to fit the change of parameter perfectly. Hence, we can see that our method has good adaptability in different scenarios. Furthermore, by comparing the trend of (t) with other curves, natural question arises: if the curve of the heuristic algorithm aligns more closely with the trend of the change curve based on the learning-based strategy, will the effect be better? The answer is yes. This issue can be easily verified by using third-order polynomial function to approximate the learning-basse bbaed curve. Specifically, by substituting the w(t) as the polynomial function (t) at + 6t2 + at + d, where we can achieve higher accuracy with 71.22% and MAP with 76.28% on KineticsSounds dataset compared with ours-H. However, this strategy only be applied once the actual changing trend of is observed.
1.5 x 10 10--_,,
665 10-3
3.2 x 10- 2--,, 1,
4.5 Further Analysis
Analysis of Modality Gap: As mentioned the paper 225 modality gap characterizes the corre- lation between different modalities in multimodal learning. And large modality gap leads to better performance some situations. We further illustrate the modality gap for CONCAT, G-Blend, MLA, and our method. The results shown in Figure From Figure we find that method can learn more discriminative representations and results in higher accuracy with large modality gap compared with other methods.
Robustness Analysis of the Pretrained Model: We further exploit the robustness of the large-scale language vision pretrained model CLIP 33 model Sarcasm and Twitter2015 datasets. We replace the encoders for image and text as the corresponding encoders pretrained by CLIP and fine-tune the model on Sarcasm and Twitter2015 datasets respectively. The results shown in Table 5 where "CLIP+MLA" and "CLIP+Ours" present that we apply the MLA's and ours algorithm, respectively. From Table 5
draw the following observations: (1). Both CLIP+MLA and CLIP+Ours can outperform CLIP all cases. (2). With the help dynamic integration, the performance of our method is better than that of MLA.
Visualization: We utilize GradCAM to showcase the visualization of image regions that attract the weak modality's focus during training. By using GradCAM, the importance scores assigned

---

Table 5: Results on the Sarcasm and Twitter2015 datasets achieved by using the CLIP pre-trained model as encoders.
Sarcasm
Twitter2015
Method
Image
Text
Multiple
Image
Text
Multiple
CLIP
74.82%
82.15% 83.19%
83.11% % 84.45%
54.48%
71.75%
72.52%
CLIP+MLA
77.45%
56.53%
72.37%
73.95%
CLiP+Ours
79.78%
83.67 %
85.42 %
64.67% %
72.59
74.43%
CONCAT
epoch
epoch =
epoch = 15
Ours-LB
epoch
epoch
epoch 15
Figure 4: Visualization on Twitter2015 dataset. Our proposed method tends to perform feature learning first and then fit the learned features to the category labels.
every pixel in each feature map, aiding in identifying the image regions critical for the model's predictions. We compare the visualization for CONCAT and our proposed method. The visualization results are presented in Figure where the second, the third, and the last columns denote the results of the first, the seventh, and the last epoch, respectively. The category label for this image is "Negative" and the corresponding text is "Crazy hair day ! CONCAT, we can see that our method focuses on the textual information from text modality, and then fits the learned features to the category labels.
contender.". By comparing our method with
5 Conclusion
In this paper, we discuss a core reason modality imbalance in multimodal learning, i.e., fitting category labels. We find that appropriate positive intervention label fitting can correct the difference in learning ability for different modalities, thus alleviating the modality imbalance phenomenon. Based on this observation, we propose novel multimodal learning approach overcome modality imbalance problem by dynamically integrating unsupervised contrastive learning and supervised multimodal learning. We design heuristic strategy and learning based strategy to perform integration dynamically. Experiments on various datasets demonstrate that our method can boost performance in multimodal learning.
Limitations: For the limitations our proposed method, the root cause of modality imbalance phenomenon caused by fitting category labels is worth discussing in depth. Does the specific category label contain attributes that are more suitable for fitting a certain modality? We leave it as a future work.
10

---

Acknowledgments
The authors thank the anonymous reviewers for their valuable comments. This work partially supported by National Key RD Program of China (2022YFF0712100) NSFC (62276131), Natural Science Foundation of Jiangsu Province of China under Grant (BK20240081), the Fundamental Research Funds for the Central Universities (No.30922010317).
References
[1] M. Saaro-Contreras, J. J. Valero-Mas, J. M. lñesta, and J. Calvo-Zaragoza, Late multimodal fusion for image and audio music transcription. ESWA, 216:1 19491, 2023.
[2] R. Arandjelovic and A. Zisserman. Look, listen and learn. In ICCV, pages 609-617. IEEE, 2017.
[3] T. Baltrusaitis, C. Ahuja, and L. Morency. Multimodal machine learning: A survey and taxonomy. TPAMI, 41(2):423 443, 2019.
[4] Y. Cai, H. Cai, and X. Wan. Multi-modal sarcasm detection in twitter with hierarchical fusion model. In ACL, pages 2506-2515. Association for Computational Linguistics, 2019.
[5] H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma. CREMA-D: crowd-soureed emotional multimodal actors dataset. TAC, 574)3377-390, 2014.
[6] A. X. Chang, W. Monroe, M. Savva, C. Potts, and C. Manning. Text to 3d scene generation with rich lexical grounding. In ACL, pages 53-62. The Association for Computer Linguistics, 2015.
[7] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman. Vggsound: A large-scale audio-visual dataset. In ICASSP, pages 721-725. IEEE, 2020.
[8] L. Chen, B. Li, S. Shen, Yang, C. Li, Keutzer, T. Darrell, and Liu. Large language models are visual reasoning coordinators. In NeurlPS, 2023.
[9] Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171-4186. Association for Com- putational Linguistics, 2019.
[10] C. Du, Teng, T. Li, Y. Liu, Wang, Y. Yuan, and Zhao. Modality laziness: Everybody's business is nobody's 's business, 2022.
[11] Du. J. Teng. T. Li, Y. Liu. T. Yuan, Y. Wang. Y. Yuan, and H. Zhao. On uni-modal feature learning in supervised multi-modal learning. In ICMI, volume 202 of Proceedings of Machine Learning Research, pages 8632-8656. PMLR, 2023.
[12] Y. Fan, W. Xu, H. Wang, Liu, and Guo. Detached and interactive multimodal learning. CoRR. hts/2407.11514 2024.
[13] Y. Fan, W. Xu, H. Wang. J. Wang, and S. Guo. PMR: prototypical modal rebalance for multimodal learning. In CVPR, pages 20029- 20038. Computer Vision Foundation IEEE, 2023.
[14] Z. Fu, K. Song, L. Zhou, and Y. Yang. Noise-aware image captioning with progressively exploring mismatched words. In AAAI, pages 12091-12099, AAAI Press, 2024.
[15] N. Fujimori, R. Endo, Y. Kawai, and T. Mochizuki. Modality-specific learning rate control for multimodal classification. In ACPR, volume 12047, pages 412-422, 2019.
[16] S. Ghadimi and M. Wang. Approximation methods for bilevel programming. CoRR, abs/1802.02246. 2018.
[17] Han, C. Zhang, H. Fu, and J. T. Zhou. Trusted multi-view classification with dynamic evidential fusion. TPAMI, 45(2):2551 -2666, 2023.

---

[18] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770-778. Computer Vision Foundation / IEEE, 2016.
[19] C. Hua, Q Xu, S. Bao, Z. Yang, and Q. Huang. Reconboost: Boosting can achieve modality reconcilement. In ICML. OpenReview.net, 2024.
[20] Y. Huang, J. Lin, C. Zhou, H. Yang, and L. Huang. Modality competition: training of multi-modal network fail in deep learning? (provably). In ICML, volume 162 of Proceedings of Machine Learning Research, pages 9226-9259. PMLR, 2022.
What makes joint
[21] Q. Jiang and W. Li. Deep eross cross-modal hashing. In CVPR, pages 3270-3278 (278. IEEE Computer Society, 2017.
[22] H. R. V. Joze, A. Shaban, M. L. Iuzzolino, and K. Koishida. MMTM: multimodal transfer module for CNN fusion. CVPR, pages 13286-13222 13296. Computer Vision Foundation IEEE, 2020.
[23] Z. Lan, L. Bao, S. Yu, W. Liu, and A. G. Hauptmann. Multimedia classification and event detection using double fusion. MTA, 71(1):333-344 2014.
[24] H. Li, X. Li, P. Hu, Y. Lei, C. Li, and Y. Zhou. Boosting multi-modal model performance with adaptive gradient modulation. In ICCV, pages 22157-22167. IEEE, 2023.
[25] W. Liang, Y. Zhang, Y. Kwon, Yeung, and J. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In NeurlPS, 2022.
[26] X. Liang, Y. Qian, Q. Guo, H. Cheng, and J. Liang. AF: association- fusion method for multi-modal classification. TPAMI, 44(12):9236995) -9544, 2022.
[27] B. Liu, L. He. Y. Xie, Y. Xiang, L. Zhu, and W. Ding. Minjot: Multimodal infusion joint training for noise learning in text and multimodal classification problems. INFFUS, 102:102071, 2024.
[28] W. Liu, I. W. Tsang, and K. Müller. An easy-to-hard learning paradigm for multiple classes and multiple labels. JMLR, 18:94:1-94: 2017.
[29] B. McFee, C. Raffel, D. Liang, D. P. W. Ellis, M. Vicar,r, E. Battenberg, and O. Nieto. librosa: Audio and music signal analysis python. In Python in Science Conference, pages 18-24. scipy.org, 2015.
[30] W. Nie, Y. Yan, D. Song, and K. Wang. Multi-modal feature fusion based on multi-layers LSTM for video emotion recognition. MTA, 80(1 (011)::6205-12244 2021.
[31] X. Peng, Y. Wei, A. Deng, D. Wang, and Hu. Balanced multimodal learning via on-the-ft fly gradient modulation. In CVPR, pages 2228-8231 Computer Vision Foundation / IEEE, 2022.
[32] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and C. Courville. Film: Visual reasoning with a general conditioning layer. In AAAI, pages
AAAI Press, 2018.
[33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139, pages 8748-8763. PMLR, 2021.
[34] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. IJCV, 128(2):33-335,, 2020.
[35] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architec- ture for computer vision. In CVPR, pages
IEEE Computer Society, 2016.
[36] L. N. Vicente and P. H. Calamai, Bilevel and multilevel programming: bibliography review. JGO, 5(3):291-306. 1994.
[37] Wan, W. Xiangyu, Z. Guan, and Y. Yang. Covir: Coordinating cross-mott modal consistency and intra-modal relations for vision-language retrieval. In ICME. ICME, 2024.
12

---

[38] W. Wang, D. Tran, and M. Feiszli. What makes training multi modal classification networks hard? In CVPR. pages 12622-12702. Computer Vision Foundation / IEEE, 2020.
[39] W. Wang and Zhou. Co-training with insufficient views. In ACML, volume 29 of JMLR Workshop and Conference Proceedings, pages 467-482. JMLR.org, 2013.
[40] Y. Wei and D. Hu. Mmpareto: Boosting multimodal learning with innocent unimodal assistance.
In ICML. OpenReview.net, 2024.
[41] N. Wu, S. Jastrzebski, K. Cho, and K. J. Geras. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In ICML, pages 24043-24055. PMLR, 2022.
[42] N. Wu, S. Jastrzebski, K. Cho, and K. J. Geras. Characterizing and overcoming the greedy nature of learning multi-modal deep neural networks. ICMI., volume 162, pages 24043-24055. PMLR, 2022.
[43] P. Xu, X. Zhu, and D. A. Clifton. Multimodal learning with transformers: A survey. TPAMI, (210)))121131221322 2023.
[44] Y. Yang, J. Guo, G. Li, L. Li, W. Li, and J. Yang. Alignment efficient image-sentence retrieval considering transferable cross-modal representation learning. FCS, 18(3)181335, 2024.
[45] Y. Yang, K. Wang, D. Zhan, H. Xiong, and Y. Jiang. Comprehensive semi-supervised multi- modal learning. In IJCAI, pages 4092-4098. ijcai.org, 2019.
[46] Y. Yang. J. Zhang, F. Gao, X. Gao, and H. Zhu. DOMEN: A divergence-orrentaaed multi-modal fusion network for resume assessment. In ACMMM, pages 1612-1620. ACM, 2022.
[47] Y. Yao and R. Mihalcea. -odality-specific learning rates for effective multimodal additive late-fusion In ACL, pages 1824-1834. Association for Computational Linguistics, 2022.
[48] Y. Ye, X. Liu, Q. Liu, X. Guo, and J. Yin. Incomplete multiview clustering via late fusion. CIN,
201 1866148846:116148455566 11, 2018.
[49] J. Yu and J. Jiang. Adapting BERT for target-oriented multimodal sentiment classification. In IJCAI, pages 5408-5414, ijcai.org, 2019.
[50] Y. Zeng, Yan, S. Mai, and Hu. Disentanglement translation network for multimodal sentiment analysis. INFFUS, 102:102031, 2024.
[51] Q. Zhang, H. Wu, C. Zhang, Q. Hu, H. Fu, J. T. Zhou, and X. Peng. Provable dynamic fusion for luw-quality multimodal data. ICML, volume 202 of Proceedings of Machine Learning Research, pages 41753-41766 PMLR, 2023.
[52] X. Zhang, J. Yoon, M. Bansal, and H. Yao. Multimodal representation learning by alternating unimodal adaptation. CoRR, abs/231 1223311.1100 10707 2023.
[53] X. Zheng, C. Tang, Z. Wan, C. Hu, and W. Zhang. Multi-level confidence learning for trustworthy multimodal classification. In AAAI, pages 11381-1 11389. AAAI Press, 2023.
[54] L. Zhu, T. Wang, F. Li, J. Li, Z. Zhang, and Shen. Cross-modal retrieval: systematic review of methods and future directions. CoRR, abs/2308 1 14263, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main contributions and scope were reflected the abstract and introduction (in the last paragraph of introduction, page 2).
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations of the work was discussed in section S
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper doesn't 't involve theoretical result.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The experimental setup, including dataset, baseline, evaluation metric and implementation details, was provided in section 4.1 In the implementation details part, we provide the network architecture, input data processing, optimization algorithm, the value of main parameters, hardware environment, and so on, to support the reproducing of the method.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We release our code httttttt//giiiaee http:::://ggitub...oooo///uuttt/ heam/njuuttkgg/Neuur IPS24- Furthermore, all datasets we used in this paper are available online based on their corre- sponding paper.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All implementation details provided in section 4.1
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We run the main experiments times with different random seeds.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
14

---

Answer: [Yes]
Justification: The computer resource information for all experiments was provided in the
section 4.1
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpppp/spppppppp...oooooa/alaaa
Answer: [Yes]
Justification: We have checked the NeurlPS code of ethics for paper.
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper does not deal with this aspect of the problem.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not deal with this aspect of the problem.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The assets (datasets, code, models) used the paper are open source, and our use follows the relevant protocols.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code we released in this paper contains related documents.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper doesn't involve the crowdsourcing experiments and research with human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper doesn't 't involve the potential risks.
15


Reviews:
[{"review_id": "VEsBsLYTsJ", "reviewer": "Reviewer_g7cu", "summary": "This paper focuses on the notorious multi-modal imbalance problem. The authors attribute this issue to different modality-specific convergence rates caused by the inconsistent difficulty of fitting class labels. Therefore, this paper adopts unsupervised contrastive learning to mitigate the modality gap. Moreover, the learning-based integration strategy is proposed to harmonize unsupervised and supervised optimization objectives. Experiments demonstrate the efficacy.", "questions": "Additionally, in table5, the authors compare the results using the CLIP pre-trained encoder. I am concerned about how to measure CLIP's performance on a single modality. Should a classifier head be fine-tuned for this purpose? The authors should give implementation details about Tab.5.", "limitations": "None", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper explores an interesting problem. In practice, the dominant modality usually overpowers the overall learning process, and how to pursue reconciliation between different modalities is critical but challenging in multi-modal learning.\n\nThe observation that introducing random noise into accurate one-hot labels can alleviate modality imbalance is novel and clearly illustrated.\n\nThe proposed method is presented clearly and easy to reproduce.", "weaknesses": "**The motivation for using contrastive learning to mitigate modality imbalance is not reasonable.** The authors' view that different patterns from the same sample should be as close as possible in the feature space seems incorrect. An ideal multi-modal representation should capture both shared and unique information relevant to the downstream task [1,2,3], **which are not necessarily identical**. The authors should explain in depth why introducing contrastive learning can mitigate modality imbalance and achieve better representation.\n\n**The related work section lacks discussion on recent SOTA research.** QMF [4] provides a quality-aware multimodal fusion framework to mitigate the influence of low-quality multimodal data. ReconBoost [3] finds that the major issue arises from the current joint learning paradigm. The proposed alternating learning paradigm pursues a reconciliation between different modalities and achieves significant improvements. MMPareto [5] resolves conflicts between multi-modal and uni-modal gradients under multi-modal scenarios and effectively alleviates modality imbalance.\n\n**The experiments lack comparison with the UMT, QMF, ReconBoost, and MMPareto.** Additionally, the authors should evaluate the performance of the uni-modal encoder to further demonstrate that the proposed method can alleviate modality imbalance.\n\n[1] On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.\n\n[2] Factorized Contrastive Learning: Going Beyond Multi-view Redundancy. NeurIPS 2023.\n\n[3] ReconBoost: Boosting Can Achieve Modality Reconcilement. ICML 2024.\n\n[4] Provable Dynamic Fusion for Low-Quality Multimodal Data. ICML2023\n\n[5] MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance. ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. \\\n**1. Answer for Question \"Contrastive Learning can Mitigate Modality Imbalance\"**: Our motivation aims to find a way to mitigate the modality performance gap caused by label fitting. As shown in literature [1], different modalities will compete with each other during the joint-training. Specifically, the neural network will not efficiently learn all features from different modalities, and only a subset of modality encoders will capture sufficient feature representations. Meanwhile, contrastive learning aims to learn similar representations for data pairs of different modalities [2], thus aligning the multimodal representations for all modalities. Hence, we utilize the contrastive learning to intervene the label fitting. Furthermore, this intervention is dynamic as the learning process. In the initial stage, the model primarily focuses on feature learning. As training progresses, label fitting becomes increasingly important. This phenomenon aligns with our expectations.\\\n**2. Answer for Question \"Lack Comparison with SOTA Method\"**: We conduct a comparison with existing state-of-the-art methods. The results in Table 1 demonstrate that our method achieves higher classification accuracy compared to these methods. Notably, our approach significantly narrows the modality performance gap, which contributes to an improvement in overall classification performance. We will add more detailed analysis in the final version.\n\nTable 1. Performance comparison between state-of-the-art methods and our method on Kinetics-Sounds\n|Method | Acc | Audio Acc | Video Acc | Modality Performance Gap |\n|---|---|---|---|---|\n|UMT[3] | 66.10 | 54.35 | 45.30 | 9.05 |\n|QMF[4] | 65.78 | 51.57 | 32.19 | 19.38 |\n|ReconBoost[5] | 70.85 | 53.79 | 52.12 | 1.67 |\n|MMPateto[6] | 70.13 | **56.40** | 53.05 | 3.35 |\n|Ours | **73.17** | 54.07 | **53.49** | **0.58** |\n\n**3. Answer for Question \"Results using CLIP Pre-trained Encoder\"**: In this case, following the setting of [7], we use the pre-trained weights in the CLIP as initialization. In the subsequent training process, the feature extractor and classification head are fine-tuned. We will add more details in the final version.\\\n**References:**\\\n[1]. Huang, Yu, et al. Modality competition: What makes joint training of multi-modal network fail in deep learning?(provably). ICML. 2022.\\\n[2]. Fan, Yunfeng, et al. Detached and Interactive Multimodal Learning. ACMMM, 2024.\\\n[3]. Du, Chenzhuang, et al. On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.\\\n[4]. Hua, Cong, et al. ReconBoost: Boosting Can Achieve Modality Reconcilement. ICML 2024.\\\n[5]. Zhang, Qingyang, et al. Provable Dynamic Fusion for Low-Quality Multimodal Data. ICML 2023.\\\n[6]. Wei, Yake, et al. MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance. ICML 2024.\\\n[7]. Zhang, Xiaohui, et al. Multimodal Representation Learning by Alternating Unimodal Adaptation. CVPR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b1Rc55d6vV", "reviewer": "Reviewer_hRBu", "summary": "This paper discusses a core reason for modality imbalance in multimodal learning, i.e., fitting category labels. It proposes a novel multimodal learning approach to overcome the modality imbalance problem by dynamically integrating unsupervised contrastive learning and supervised 319 multimodal learning.", "questions": "See weakness", "limitations": "Yes, they do.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper reveals the reason that models converge at different rates is the difficulty of fitting category labels is inconsistent for each modality during learning. This is a novel insight into imbalanced multimodal learning.\n\nThis paper further proposes a novel multimodal learning approach by integrating unsupervised contrastive learning and supervised multimodal learning.", "weaknesses": "1. The reviewer is curious about the performance of the proposed method on larger datasets, such as VGGsound.\n\n2. When the information of two modalities is unbalanced, it is optimal to rely more on the modality that contains more information. However,  the proposed method is forcibly narrowing the modality gap. Will this lead to the restriction of the learning of strong modality and lead to sub-optimal performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments.\\\n**1.Answer for Question \"Performance on Larger Dataset\"**: We compare our method with the state-of-the-art method MLA on the VGGSound dataset using the experimental settings described in the literature [1]. Experimental results in Table 1 demonstrate that our method achieves higher accuracy on larger datasets compared to state-of-the-art method MLA.\n\nTable 1. Performance comparison between state-of-the-art method and our method on VGGSound dataset.\n|Method|Acc|MAP|\n|---|---|---|\n|MLA|51.65|54.73|\n|Ours|**52.74**|**55.98**|\n\n**2. Answer for Question \"Restriction of Strong Modality\"**: Table 4 in the original paper shows that, compared to the standard joint training method, both the strong and weak modalities have shown improvement, with the weak modality improving more than the strong modality. Additionally, the single-modality performance of combined training is close to that of individual training.\\\n**References:**\\\n[1]. On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h5amkmhcjU", "reviewer": "Reviewer_CoWQ", "summary": "This paper proposes a novel multimodal learning approach by integrating supervised MML and unsupervised contrastive learning dynamically to address the multimodal imbalance problem. The authors begin by examining the differences in unimodal performance, analyze the reasons for these differences, and introduce unsupervised learning to eliminate them, thereby solving the problem of modality imbalance. Experiments on widely used datasets clearly demonstrate the superiority of the proposed method. The approach of analyzing modality imbalance from the label fitting perspective is particularly novel. The authors conducted comprehensive experiments to prove the effectiveness of the proposed method.", "questions": "Please refer to weakness.", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The approach of analyzing modality imbalance from the label-fitting perspective is particularly novel. Most of the existing multimodal learning approaches focus on designing adaptive learning strategies. This paper analyzes the causes of modality imbalance from another perspective and provides some inspiration for the development of multimodal learning.\n2. The paper is well organized and the writing is easy to follow. The overall logic of the paper is clear.\n3. To demonstrate the effectiveness of the proposed method, the authors provide detailed explanations from both methodological and experimental perspectives. Compared with SOTAs, we can find that the proposed method can eliminate the modality imbalance phenomenon and boost performance.\n4. The authors design a heuristic and learning-based strategy to integrate two losses. Furthermore, the authors analyze the effects of fixed value strategies (constant), heuristic strategy, and learning-based from an experimental perspective.", "weaknesses": "1. According to Figure 3, if a function aligns more closely with the trend of the learning strategy curve than \\omega(t), would it perform better than \\omega(t)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\\\n**1.Answer for Question \"Closely With the Learning Strategy Curve\"**: We use the polynomial function $f(t) = a t^3 + b t^2 + c t + d$, with coefficients $a = 1.5 \\times 10^{-4}$, $b = -6.5 \\times 10^{-3}$, $c = 3.2 \\times 10^{-2}$, $d = 1$ to fit the current learning curve and represent  $\\omega(t)$. The results in Table 1 indicate some degree of improvement. However, the functions fitted to different datasets are different, and it is difficult to find a unified function form, so we put forward the learning-based method.\n\nTable 1. Performance of different weights on Kinetics-Sounds.\n\n|Method|ACC|MAP|\n|:---|:---|:---|\n|Ours-H|\t69.05|\t72.97|\n|$a t^3 + b t^2 + c t + d$|\t71.22|\t76.82|\n|Ours-LB|\t**72.53**|\t**78.38**|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DzKEdZXd9F", "reviewer": "Reviewer_XdY4", "summary": "This paper proposes a novel multimodal learning method to address modality imbalance by dynamically integrating unsupervised contrastive learning and supervised multimodal learning. The authors design two dynamic integration strategies: heuristic and learning-based. Experimental results show that the proposed method significantly outperforms existing multimodal learning methods on several datasets.", "questions": "Please refer to weakness and address my concerns.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "- Clarity. The presentation of this paper is generally expectational. I enjoy the clarity of writing.\n- Important topics. Multimodal learning and modality imbalance have achieved increasing research interest recently.\n- Experimental results show that the proposed method significantly outperforms existing multimodal learning methods on several datasets.", "weaknesses": "**Weaknesses:**\n\n1. **Insufficient Analysis of Modality Imbalance**: The paper claims to be the first to analyze modality imbalance from the perspective of fitting category labels. However, only a simple experiment on the Kinetics Sounds dataset is presented, with an inadequate analysis. The experiment uses uniform labels and one-hot labels to obtain a mixed loss, finding that increasing the weight of LUL_ULU reduces the modality performance gap. This seems obvious, but how this phenomenon relates to the claim that \"this also implies that fitting category labels is a core cause of modality imbalance in multimodal learning\" is not clearly explained.\n2. **Difference Between Mixed Loss and Label Smoothing**: What is the difference between the so-called mixed loss and the label smoothing technique? The authors should clarify this distinction. Label smoothing first obtains mixed labels and then calculates the loss. In the context of cross-entropy, these should be equivalent. It would be more convincing if the authors' analysis could relate to the theoretical explanations of label smoothing.\n3. **Motivation and Method Inconsistency**: The paper's motivation is to alleviate multimodal imbalance by intervening in fitting category labels. However, it then introduces contrastive loss to intervene in the feature space. There is insufficient connection between the method and the motivation. The explanation \"Ideally, contrastive learning can also mitigate the effect of modality imbalance problem. Hence, we introduce contrastive learning to impose positive intervention in multimodal learning to alleviate the impact of fitting labels.\" is too superficial. The authors should provide a deeper and more solid analysis of how contrastive learning helps in intervening in label fitting.\n4. **Practicality of Dynamic Weight Methods**: Section 3.3 introduces heuristic and learning-based dynamic weights. The heuristic dynamic weights do not perform well in experiments; the learning-based dynamic weights involve bi-level optimization. Although the authors used an approximation method to derive Equation (3), it still involves calculating the Hessian matrix and its inverse. The authors did not discuss the computational cost or approximation techniques, making the practicality of this method questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments.\\\n**Overall Response:**  Our motivation is to explore the relationship between label fitting (one-hot label) and modality imbalance, thus to find a way to mitigate the impact of label fitting. We first utilize a toy noise label experiment to demonstrate that label fitting can lead to modality imbalance. Specifically, we compared the classification performance by using the original one-hot label loss ($L_s$), a label free loss ($L_u$) and their linear combination loss. One can image that $L_s$ is a supervised loss while $L_u$ is a unsupervised loss. The results in Figure 1 of the original paper show that even the overall classification performance of $L_u$ is worse than that of $L_s$, $L_u$ has the smallest modality performance gap, indicating that the unsupervised loss could mitigate the modality imbalance but lead to low classification performance. This inspires that there may be an optimal trade-off between the supervised loss $L_s$ and unsupervised loss $L_u$. To this end, one simple way is to tune the $\\alpha\\in [0,1]$ for $\\alpha L_s + (1-\\alpha)L_u$. We finally found that $0.7L_s+0.3L_u$ has the best overall classification performance with the middle modality performance gap, compared with $L_s$ and $L_u$. Based on the experimental findings, we then employ the contrastive unsupervised learning to align the modality representation of multimodal data [1], thereby mitigating the modality imbalance. In this way, overall classification performance of MML can be improved. It is worth to mention that the proposed method is a generic algorithm that can be integrated with many unsupervised learning.\\\n**1. Answer for Question \"Insufficient Analysis of Modality Imbalance\":** The overall classification performance of MML is a primary concern to us and is affected by the modality imbalance phenomenon. Literature [2,3,4] theoretically reveals the intrinsic relationship between modality performance gap and modality imbalance, i.e., the larger the modality performance gap, the more serious the modality imbalance. Based on these theoretical conclusions and our findings, we deeply explore the impact of label fitting on modality imbalance in this paper. The main logic of our paper is that label fitting leads to the modality performance gap, which in turn exacerbates the modality imbalance phenomenon. As you mentioned, increasing the weight of noise labels to reduce the impact of label fitting is positively correlated with a reduction in the modality performance gap. Meanwhile, as the modality performance gap narrows, the modality competition phenomenon will be alleviated [3], thus alleviating the modality imbalance. In summary, the overall performance of MML is improved through intervening the label fitting.\\\n**2. Answer for Question \"Difference Between Mixed Loss and Label Smoothing\":** The mixed loss is the label smoothing. As mentioned in the overall response part, the experiment of label noise aims to explore the relationship between label fitting and modality performance gap, thereby affecting overall performance. Therefore, we did not discuss the mixed loss and label smoothing in the paper. We will add appropriate discussion to the final version.\\\n**3. Answer for Question \"Practicality of Dynamic Weight Methods\":** Heuristic method is designed primarily to validate the motivation behind our methods. For example, by carefully adjusting the function form, such as using the polynomial function $f(t) = a t^3 + b t^2 + c t + d$, with coefficients $a = 1.5 \\times 10^{-4}$, $b = -6.5 \\times 10^{-3}$, $c = 3.2 \\times 10^{-2}$, $d = 1$, the classification accuracy achieves 71.22% compared to 70.04% with the MLA on the Kinetics-Sounds dataset. \\\nIn fact, the complexity of bi-level optimization [5] and our algorithm is O(n). More details of bi-level optimization can be referred to literature [5]. To further demonstrate the efficiency of our method, we adopt MLA and our method for experiments and report the training time cost on Kinetics-Sounds dataset. The average time costs for MLA and our method are 327s and 301s for an epoch, respectively. Therefore, we can demonstrate the practicality of our method. We will add some discussion and results in the final version.\\\n**References:**\\\n[1]. Fan, Yunfeng, et al. Detached and Interactive Multimodal Learning. ACMMM, 2024.\\\n[2]. Zhang, Xiaohui, et al. Multimodal Representation Learning by Alternating Unimodal Adaptation. CVPR, 2024.\\\n[3]. Huang, Yu, et al. Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably). ICML. 2022.\\\n[4]. Wang, Wei, and Zhi-Hua Zhou. Co-training with insufficient views. ACML. 2013.\\\n[5]. Vicente, Luis N., and Paul H. Calamai. Bilevel and multilevel programming: A bibliography review. JGO, 1994.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel multimodal learning approach to overcome the modality imbalance problem. It received overall positive reviews and the reviewers appreciated the problem addressed, proposed framework and writing. One of the reviewers still has concerns regarding the inconsistency between the motivation and the proposed framework. The authors have tried to explain it, but no theoretical justification has been provided. Based on the overall positive reviews, the recommendation is to accept the paper. The authors are strongly encouraged to address any remaining concerns in the camera-ready version.
Decision: Accept (poster)