Facilitating Multimodal Classification via Dynamically Learning Modality Gap

Authors: yang  yang
fengqiang  wan
qing-yuan  jiang
yi  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 03:08:23

Abstract: Multimodal learning falls into the trap of the optimization dilemma due to the modality imbalance phenomenon, leading to unsatisfactory performance in real applications. A core reason for modality imbalance is that the models of each modality converge at different rates. Many attempts naturally focus on adjusting learning procedures adaptively. Essentially, the reason why models converge at different rates is because the difficulty of fitting category labels is inconsistent for each modality during learning. From the perspective of fitting labels, we find that appropriate positive intervention label fitting can correct this difference in learning ability. By exploiting the ability of contrastive learning to intervene in the learning of category label fitting, we propose a novel multimodal learning approach that dynamically integrates unsupervised contrastive learning and supervised multimodal learning to address the modality imbalance problem. We find that a simple yet heuristic integration strategy can significantly alleviate the modality imbalance phenomenon. Moreover, we design a learning-based integration strategy to integrate two losses dynamically, further improving the performance. Experiments on widely used datasets demonstrate the superiority of our method compared with state-of-the-art (SOTA) multimodal learning approaches. The code is available at https://github.com/njustkmg/NeurIPS24-LFM.

Main paper: 
 introduction: 
 Multimodal learning (MML) [3, 38, 47, 41, 10, 26, 31, 43, 13] integrates heterogeneous information from different modalities to build an effective way to perceive the world. Over the past decades, multimodal learning has made incredible progress [31, 10, 13] and become a hot research topic with a wide range of real applications including image caption [6, 14] , cross-modal retrieval [21, 44, 54, 37] , vision reasoning [32, 8] , action recognition [23, 28] , and so on.In multimodal learning, several recent studies [38, 31] have revealed an interesting phenomenon, i.e., the performance of the multimodal model is far from the upper bound or even inferior to the unimodal in certain situations. The root of this problem lies in the existence of the modality imbalance phenomenon [38] . Concretely, there commonly exists dominant modality and non-dominant modality in heterogeneous multimodal data. Multimodal learning usually adopts a uniform objective. Due to greediness [42] , the optimization tends to dominant modality while neglecting the non-dominant one during joint training, thus leading to unsatisfactory performance in real applications.Recently, many impressive works [38, 11, 31, 13, 24] have been proposed to address the modality imbalance problem. Early pioneering approaches, such as gradient blending (G-Blend) [38] , on-the-fly gradient modulation (OGM) [31] , adaptive gradient modulation (AGM) [24] , and prototypical modality rebalance (PMR) [13] , focus on designing customized learning strategies for different modalities to adjust the optimization of dominant and non-dominant modality. These methods demonstrate that suppressing the optimization of the dominant modality can alleviate the modality imbalance problem to a certain extent. Besides, several attempts, including uni-modal teachers (UMT) [11] and balanced multimodal learning [42] , try to introduce extra networks as an auxiliary module to facilitate multimodal learning. The influence of labels fitting on performance gaps (best view in color), where L S and L U denote the loss with one-hot labels and uniform labels (label free).Although the aforementioned approaches can boost performance in MML, these solutions are based on the phenomenon of inconsistent learning speed itself and do not study the underlying causes of modality imbalance. We can't help but ask what is the essential reason behind this phenomenon. Is there a bias in the process of fitting category labels for different modalities?We carry out a simple experiment on Kinetic-sSounds dataset to seek answers. We adopt two types of labels to explore the influence of fitting labels. The first type is one-hot labels which indicate the category of each sample, where the loss is denoted as L S . The second type is label free, i.e., uniform label 1/c for all samples, where c denotes the number of categories. The second loss is defined as L U . Furthermore, we define a mixed loss 0.7L S + 0.3L U , which is actually the label smoothing [35] strategy, by combining one-hot labels and uniform labels.The accuracy is reported in Figure 1 . From Figure 1 , we can observe that with proper intervention by using uniform labels, the performance is slightly better than the model that fits one-hot labels. More importantly, the performance gap becomes smaller if we learn from uniform labels. This means that the difference between audio and video modalities is smaller in feature space when we reduce the weight of one-hot labels. The experiment inspires us that appropriate intervention label fitting can alleviate the difference in the learning ability of different modalities. In this way, we can reduce the modality performance gap and further mitigate the modality imbalance phenomenon as the smaller the difference in modality performance, the less severe the modality imbalance [39, 20, 52] . This also implies that fitting category labels is a core cause of modality imbalance in multimodal learning.How do we impose positive intervention in multimodal learning so that the impact of fitting labels on modality imbalance is as low as possible without affecting the overall performance? For multimodal learning, although the models are learned from the heterogeneous data, we hope that multimodal data describing the same entity should be as close as possible in the feature space, which is usually modeled as contrastive learning [33] . Contrastive learning aims to learn similar representations for data pairs of different modalities [12] , thus aligning the multimodal representations for all modalities. Ideally, contrastive learning can also mitigate the effect of modality imbalance problem. Hence, we introduce contrastive learning to impose positive intervention in multimodal learning to alleviate the impact of fitting labels.In this paper, we propose a novel multimodal learning approach by integrating unsupervised contrastive learning and supervised multimodal learning dynamically. Specifically, after demonstrating the effectiveness of unsupervised contrastive learning in multimodal learning, we design two dynamical integration strategies, i.e., a heuristic and a learning-based integration strategy. Our contributions are outlined as follows: (1) . We observe a key phenomenon: fitting category labels leads to a larger performance gap between different modalities. To the best of our knowledge, this is the first time that the modality imbalance problem has been analyzed from the perspective of category label fitting.(2). We propose a novel multimodal learning approach by integrating unsupervised contrastive learning and supervised multimodal learning. Two strategies are designed for dynamic integration.(3). Extensive experiments on widely used datasets show that our proposed approach can significantly outperform other baselines to achieve state-of-the-art performance.2 Related Work 
multimodal learning: 
 Multimodal learning aims to leverage multimodal data from different sources to improve model performance. Based on the fusion strategy, multimodal learning approaches can be categorized into early fusion [38, 46, 52, 50] , late fusion [48, 47, 1, 27] , and hybrid fusion [22, 53] . Early fusion methods aim to integrate multimodal features to study the interrelationship between different modalities with joint representations when features are extracted by encoders. Representative early fusion methods include G-Blend [38] , association-based fusion (AF) [26] , and DOMFN [46] . On the contrary, late fusion methods leverage the prediction of each model to make final decisions. Late fusion methods can be divided into two categories, i.e., soft late fusion and hard late fusion, where the former utilizes the confidence score to make decisions and the latter the category decision of each model. Pioneering late fusion methods include modality-specific learning rate (MSLR) [47] . Hybrid fusion methods try to amalgamate the advantages of both early and late fusion methods. Representative hybrid fusion methods include multimodal transfer module (MMTM) [22] and balanced multi-modal learning [42] .Although these methods explore the algorithms and applications in multimodal learning, all of them assume that each modality can make sufficient contributions to achieve satisfactory performance during the training procedure.In reality, an obvious situation is that multimodal data and models are diverse, which naturally leads to different contributions during the training procedure. Recent works [38, 31, 13, 24] have shown that modality imbalance is a ubiquitous phenomenon and often results in unsatisfactory performance or even worse than unimodal algorithms in some cases. Considering the existence of dominant modality and non-dominant modality, early pioneering approaches [38, 31, 13] focus on adjusting learning speed for different modalities with customized learning strategies to balance the optimization of dominant and non-dominant modality. For instance, G-Blend [38] proposes to minimize the overfitting-to-generalization ratio (OGR) by using a gradient blending technique based on the modality's overfitting behavior. OGM [31] utilizes an on-the-fly gradient modulation strategy to control the modality's optimization procedure. To achieve the purpose of balanced multimodal learning, PMR [13] designs a prototypical modal rebalance strategy to facilitate the learning of non-dominant modality. Other attempts [11, 42] try to utilize extra networks to facilitate multimodal learning. Concretely, UMT [11] utilizes the teacher networks to distill the pretrained unimodal features to the multimodal network to tackle the modality imbalance problem. Balanced multimodal learning [42] utilizes the gradient norm and model parameters' norm to define conditional learning speed and uses it to guide the learning procedure. These methods alleviate the modality imbalance problem to a certain extent. 
methodology: 
 We present our proposed method in this section. Specifically, we first present the problem definition of multimodal learning. Then, we introduce unsupervised contrastive learning to impose positive intervention in multimodal learning and propose two dynamical integration strategies to maximize the learning collaboration of unsupervised contrastive learning and supervised multimodal learning.For the sake of simplicity, we use boldface lowercase letters like a and boldface uppercase letters like A to denote vectors and tensors, respectively. The i-th element of a is denoted as a i . Furthermore, we use ∥ • ∥ 2 to denote L 2 norm of the vectors.The goal of multimodal learning is to train a model to predict the category labels for given multimodal data. Without any loss of generality, we use X = {x i } n i=1 to denote the training data points, where each data point is with m modalities, i.e., x i = {x(j) i } m j=1 .The category labels are represented asY = {y i | y i ∈ {0, 1} c } n i=1, where c denotes the number of category labels.For deep learning based multimodal approaches, we usually adopt a deep neural network to extract representation from original space into feature space. We utilize ϕ (j) (•) to denote the feature extraction function for j-th modality. Given data point x (j)i , the feature extraction can be formed as:z (j) i = ϕ (j) (x (j) i ; Φ (j) ), where z (j) i ∈ R d denotes the d-dimension feature vector of x (j)i , and Φ (j) denotes the parameters of j-th encoder. After vectors for all modalities are extracted, we adopt a fusion function f (•) to fuse the different feature vectors. Then, we leverage a fully-connected layer to map the vector into R c . This procedure can be formed as:z i = f (z (1) i , • • • , z (m) i ), ŷi = softmax(W z i + b).Here, W ∈ R c×D , b ∈ R c denote the weights and bias of the last fully-connected layer, respectively, and D denotes the dimension of z i . Then, the objective function of multimodal learning can be formulated as:L CLS (X, Y ) = - 1 n n i=1y ⊤ i log ŷi .To bridge the heterogeneous data in feature space, we utilize contrastive learning [33] in multimodal learning. For a pair of data points {xEQUATIONk }, we define the similarity as:s(x (j) i , x (l) k ) = [z (j) i ] ⊤ z (l) k ∥z (j) i ∥ 2 ∥z (l) k ∥ 2 .The modality matching objective function can be written as:L MM (X) = - 1 2n b n b i log exp(s(x (j) i , x (l) i )/τ ) k exp(s(x (j) i , x (l) k )/τ ) + log exp(s(x (j) i , x (l) i )/τ ) k exp(s(x (j) k , x (l) i )/τ ), where τ is the temperature parameter and n b denotes the batch size. By integrating the classification loss and modality matching loss, we can get the following objective function:EQUATIONwhere α denotes the weighted parameter between two losses.Although a fixed value of α allows the model to take into account both classification loss and modality matching loss, it cannot dynamically evaluate the weight of two losses during training. Hence, we propose two strategies to adjust α dynamically to balance two losses.Firstly, we utilize a monotonically decreasing function to adjust the impact of category labels. The definition of the function can be written as: α t = ω(t), where t denotes the number of training epochs. In this paper, we set ω(t) = 1 -e -1 t . Then, we further exploit a learning-based integration method by utilizing bi-level optimization strategy [36] . Specifically, while considering optimizing the multimodal classification loss L CLS , we use the minimum value of the total loss L Total to restrict the feasible region of the parameters θ. In other words, we require the parameters not just to minimize classification loss but also to comply with a precisely defined constraint, i.e., simultaneously minimize a composite loss function-a strategically engineered combination of modality matching loss and multimodal classification loss. The specific formula is defined as follows:EQUATIONHere, θ denotes the parameters of multimodal models, and α emerges as a key parameter, delicately balancing modality matching loss and multimodal classification loss to direct the model toward an Baselines: We select a wide range of baselines for comparison. These baselines can be divided into two categories, i.e., traditional MML approaches and fusion methods with modal rebalancing strategies. The former category encompasses techniques like feature concatenation (CONCAT), affine transformation (Affine) [32] , channel-wise fusion (Channel) [22] , multi-layer LSTM fusion (ML-LSTM) [30] , prediction summation (Sum), prediction weighting (Weight) [45] , and enhanced trust modal combination (ETMC) [17] . And the latter category includes MSES [15] , G-Blend [38] , OGM [31] , Greedy [41] , DOMFN [46] , MSLR [47] , PMR [13] , AGM [24] , MLA [52] , Recon-Boost [19] , and MMPareto [40] . Other baselines including UMT [11] and QMF [51] are not adopted as these two methods have been found to be outperformed by the adopted baseline ReconBoost.Evaluation Metrics: Following [31] , we use accuracy (ACC) and mean Average Precision (MAP) as evaluation metrics for audio-video datasets. For text-image datasets, we adopt ACC and Macro F1-score (F1) [4] . ACC measures the proportion of correct predictions to total predictions, indicating the overall predictive accuracy. Macro F1 calculates the average of F1 scores across all categories, balancing precision and recall to evaluate performance evenly across classes. MAP represents the average precision across all categories, assessing the model's ranking ability for each category.Implementation Details: In our experiments, we utilize raw data for experiments. Following [31, 13] , for the KineticsSounds and CREMA-D datasets, ResNet18 [18] serves as the foundational architecture for processing both audio and video data. For video analysis, we select 10 frames from each clip and subsequently sample three frames uniformly as inputs. We adapt ResNet18's input channels from three to one to accommodate our data format [7] . In terms of audio, we transform our sound recordings into spectrograms measuring 257×1004 for KineticsSounds and 257×299 for CREMA-D, employing the librosa [29] library for conversion. For text-image datasets, our framework incorporates ResNet50 for images and BERT [9] for text processing. We resize images to 224 × 224 and limit text sequences to a maximum length of 128 characters. Optimization for the audio-video datasets is conducted using stochastic gradient descent (SGD) with a momentum set to 0.9 and a weight decay parameter of 10 -1 . We initialize the learning rate to 10 -2 , progressively reducing it by a factor of ten upon observing a plateau in loss reduction, with a batch size of 256. For text-image datasets [4, 49] , we employ the Adam optimizer starting with a learning rate of 10 -4 , with a batch size of 128. For our methods, we run the experiment three times with different random seeds and present the detailed performance with mean and std. values to remove randomness. All models are trained on a single RTX 3090 GPU. The main results for all datasets, except VGGSound, are presented in Table 1 , where "Our-H" and "Ours-LB" denote the proposed method based on heuristic strategy and learning-based strategy, respectively. In Table 1 , Unimodal-1 and Unimodal-2 refer to the audio and video modalities for audio-video datasets, and the image and text modalities for image-text datasets, respectively. For NVGesture dataset, Unimodal-1/2/3 respectively denote the RGB/OF/Depth. From the results, we can derive the following observation: (1) . Compared with all baselines including traditional multimodal learning approaches and fusion methods with modal rebalancing strategies, our proposed method with learning-based strategy can achieve best performance by a large margin in almost all cases. We can also find that the model with learningbased strategy can achieve better performance than that with heuristic strategy. (2) . Across the Twitter2015 dataset, there is a discernible trend where the optimal unimodal performance outstrips that of multimodal joint learning. Additionally, in other datasets, fusion methodologies devoid of rebalancing mechanisms manifest negligible enhancements relative to the foremost unimodal performance, notably on the CREMA-D and Sarcasm datasets. This shortfall originates from the prevalent challenge of modal imbalance. (3) . Every modality rebalancing technique demonstrates significant improvements over traditional feature concatenation fusion. This finding not only underscores the detrimental impact of modal imbalance on performance but also corroborates the efficacy of the modality rebalancing approach. (4). For NVGesture dataset, differing from modal rebalancing methods restricted to scenarios with only two modalities, such as Greedy, our approach with learning-based strategy can address challenges in scenarios involving more than two modalities and achieve best results. And our proposed method can outperform all baselines in most cases. 
comparison with sota mml baselines: 
 For the relatively large dataset VGGSound, we select a set of recent algorithms, including AGM [24] , MLA [52] , ReconBoost [19] , and MMPareto [40] , for experimental evaluation. The results are shown in Table 2 . From Table 2 , we can see that our proposed methods can achieve the best performance in all cases compared with recent SOTA baselines on VGGSound dataset.To comprehensively assess the effectiveness of our proposed method, we conduct experiments to study the influence of main components, i.e., contrastive learning (CL) and dynamic integration (DI). The results are shown in Table 3 , where the "CL" and "DI" denote that whether the contrastive learning and dynamic integration are applied during training. The unimodal MAP results are based on audio and video modalities for KineticsSounds and CREMA-D datasets, and unimodal F1 results are based on image and text modalities for Sarcasm and Twitter2015 datasets. Please note that dynamic integration depends on the contrastive learning loss. Hence the method with dynamic integration but without contrastive learning cannot be performed. From Table 3 , we can see that both contrastive learning and dynamic integration can boost performance in multimodal learning. Moreover, by integrating contrastive learning into multimodal learning, the performance gap between audio and video is greatly reduced. Analysis of Integration Strategy: We further study the impact of different integration strategies for contrastive loss and classification loss. Specifically, we analyze three categories of integration strategy, i.e., constant, stepwise and dynamic strategy. For constant strategy, we assign a constant value to α for the experiment and run three sets of experiments by setting α = 0, α = 0.5, and α = 1. Here, "α = 0" denotes that we only perform supervised multimodal learning. Similarly, "α = 1" denotes that we only perform unsupervised contrastive learning. For stepwise strategy, we define an indicator function h(p), where h(p) denotes that α = p if the current epoch is less than half of the total epochs, otherwise α = 1 -p. For dynamic strategy, we assign value to α by heuristic strategy (Ours-H) and learning-based strategy (Ours-LB).The experimental results are shown in Table 4 . From Table 4 , we can draw the following observations:(1). Integrating multimodal learning and contrastive learning simultaneously with a constant ratio can slightly boost performance in some cases and greatly reduce the performance gap between audio and video. (2) . In general, the model with a stepwise strategy can outperform the model with a constant strategy. Furthermore, we can find that the performance of the model with two-stage training, i.e., h(0) or h (1) , is worse than that of the model which combines two losses with a constant value, i.e., h(0.05) or h(0.95). (3). The overall performance of the model with the dynamic strategy is better than that with the other strategy. Moreover, the model with the learning-based strategy can achieve the best performance. And the performance gap of this model is nil or negligible. The experimental results prove that the smaller the performance gap of the uni-modals, the better the overall performance of the model. is close to the actual changes to some extent, but there is still a gap between the customized function and the actual changes. In practice, it is difficult to fit the change of parameter α perfectly. Hence, we can see that our method has good adaptability in different scenarios. Furthermore, by comparing the trend of ω(t) with other curves, a natural question arises: if the curve of the heuristic algorithm aligns more closely with the trend of the change curve based on the learning-based strategy, will the effect be better? The answer is yes. This issue can be easily verified by using a third-order polynomial function to approximate the learning-based curve. Specifically, by substituting the ω(t) as the polynomial function ω(t) = at 3 + bt 2 + ct + d, where a = 1.5 × 10 -4 , b = -6.5 × 10 -3 , c = 3.2 × 10 -2 , d = 1, we can achieve higher accuracy with 71.22% and MAP with 76.28% on KineticsSounds dataset compared with ours-H. However, this strategy can only be applied once the actual changing trend of α is observed.Analysis of Modality Gap: As mentioned in the paper [25] , modality gap characterizes the correlation between different modalities in multimodal learning. And large modality gap leads to better performance in some situations. We further illustrate the modality gap for CONCAT, G-Blend, MLA, and our method. The results are shown in Figure 2 . From Figure 2 , we can find that our method can learn more discriminative representations and results in higher accuracy with a large modality gap compared with other methods.Robustness Analysis of the Pretrained Model: We further exploit the robustness of the large-scale language vision pretrained model CLIP [33] model on Sarcasm and Twitter2015 datasets. We replace the encoders for image and text as the corresponding encoders pretrained by CLIP and fine-tune the model on Sarcasm and Twitter2015 datasets respectively. The results are shown in Table 5 , where "CLIP+MLA" and "CLIP+Ours" present that we apply the MLA's and ours algorithm, respectively.From Table 5 , we can draw the following observations: (1) . Both CLIP+MLA and CLIP+Ours can outperform CLIP in all cases. (2) . With the help of dynamic integration, the performance of our method is better than that of MLA.Visualization: We utilize GradCAM [34] to showcase the visualization of image regions that attract the weak modality's focus during training. By using GradCAM, the importance scores are assigned to every pixel in each feature map, aiding in identifying the image regions critical for the model's predictions. We compare the visualization for CONCAT and our proposed method. The visualization results are presented in Figure 4 , where the second, the third, and the last columns denote the results of the first, the seventh, and the last epoch, respectively. The category label for this image is "Negative" and the corresponding text is "Crazy hair day ! T is a contender.". By comparing our method with CONCAT, we can see that our method focuses on the textual information from text modality, and then fits the learned features to the category labels. 
conclusion: 
 In this paper, we discuss a core reason for modality imbalance in multimodal learning, i.e., fitting category labels. We find that appropriate positive intervention label fitting can correct the difference in learning ability for different modalities, thus alleviating the modality imbalance phenomenon.Based on this observation, we propose a novel multimodal learning approach to overcome modality imbalance problem by dynamically integrating unsupervised contrastive learning and supervised multimodal learning. We design a heuristic strategy and a learning based strategy to perform integration dynamically. Experiments on various datasets demonstrate that our method can boost performance in multimodal learning.Limitations: For the limitations of our proposed method, the root cause of modality imbalance phenomenon caused by fitting category labels is worth discussing in depth. Does the specific category label contain attributes that are more suitable for fitting a certain modality? We leave it as a future work. 
 
 Appendix: Based on the approximation Equation in(3), we can use the gradient descent method to optimize α.After defining the updating strategy for α, we utilize an alternating algorithm between model parameters θ and α to perform model learning. Specifically, our algorithmic process iteratively refines the model parameters θ and the parameter α, employing a nested loop structure where the inner loop focuses on the currently given α to minimize total loss to update θ, and the outer loop updates α by function or bi-level policy to minimize classification losses. Through this structured optimization, the model achieves a delicate balance between multimodal matching and classification losses. The overall algorithm of our model is outlined in Algorithm(1), where we utilize method to indicate the chosen updating strategy in practice. Moreover, the complexity of bi-level optimization[16] and our algorithm in Algorithm (1) is O(n), which makes our approach highly practical.4 Experiments4.1 Experimental SetupDatasets: We select six widely used datasets, including KineticsSounds[2], CREMA-D[5], Sarcasm[4], Twitter2015[49], NVGesture[42], and VGGSound[7] datasets, to validate our proposed method. Among these datasets, the KineticsSounds, CREMA-D and VGGSound datasets consist of both audio and video modalities. The KineticsSounds dataset, which contains 19,000 video clips categorized into 31 distinct actions, aims to advance video action recognition. It is divided into a training set of 15,000 clips, a validation set of 1,900 clips, and a test set of 1,900 clips. The CREMA-D dataset, encompassing 7,442 clips, is divided into six emotional categories to enhance speech emotion analysis, with 6,698 clips in the training set and 744 clips in the test set. The VGGSound dataset, which contains 310 classes and a wide range of audio events in everyday life, is a relatively large dataset. It includes 168,618 videos for training and validation, and 13,954 videos for testing. Furthermore, the Sarcasm and Twitter2015 datasets consist of image and text modalities. The Sarcasm dataset offers a compilation of 24,635 text-image pairs, divided into 19,816 for the training set, 2,410 for the validation set, and 2,409 for the test set. The Twitter2015 dataset contains 5,338 


Decision: The paper proposes a novel multimodal learning approach to overcome the modality imbalance problem. It received overall positive reviews and the reviewers appreciated the problem addressed, proposed framework and writing. One of the reviewers still has concerns regarding the inconsistency between the motivation and the proposed framework. The authors have tried to explain it, but no theoretical justification has been provided. Based on the overall positive reviews, the recommendation is to accept the paper. The authors are strongly encouraged to address any remaining concerns in the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper focuses on the notorious multi-modal imbalance problem. The authors attribute this issue to different modality-specific convergence rates caused by the inconsistent difficulty of fitting class labels. Therefore, this paper adopts unsupervised contrastive learning to mitigate the modality gap. Moreover, the learning-based integration strategy is proposed to harmonize unsupervised and supervised optimization objectives. Experiments demonstrate the efficacy.
strength_weakness: strength_weakness: **The motivation for using contrastive learning to mitigate modality imbalance is not reasonable.** The authors' view that different patterns from the same sample should be as close as possible in the feature space seems incorrect. An ideal multi-modal representation should capture both shared and unique information relevant to the downstream task [1,2,3], **which are not necessarily identical**. The authors should explain in depth why introducing contrastive learning can mitigate modality imbalance and achieve better representation.

**The related work section lacks discussion on recent SOTA research.** QMF [4] provides a quality-aware multimodal fusion framework to mitigate the influence of low-quality multimodal data. ReconBoost [3] finds that the major issue arises from the current joint learning paradigm. The proposed alternating learning paradigm pursues a reconciliation between different modalities and achieves significant improvements. MMPareto [5] resolves conflicts between multi-modal and uni-modal gradients under multi-modal scenarios and effectively alleviates modality imbalance.

**The experiments lack comparison with the UMT, QMF, ReconBoost, and MMPareto.** Additionally, the authors should evaluate the performance of the uni-modal encoder to further demonstrate that the proposed method can alleviate modality imbalance.

[1] On Uni-Modal Feature Learning in Supervised Multi-Modal Learning. ICML 2023.

[2] Factorized Contrastive Learning: Going Beyond Multi-view Redundancy. NeurIPS 2023.

[3] ReconBoost: Boosting Can Achieve Modality Reconcilement. ICML 2024.

[4] Provable Dynamic Fusion for Low-Quality Multimodal Data. ICML2023

[5] MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance. ICML 2024
questions: questions: Additionally, in table5, the authors compare the results using the CLIP pre-trained encoder. I am concerned about how to measure CLIP's performance on a single modality. Should a classifier head be fine-tuned for this purpose? The authors should give implementation details about Tab.5.
limitations: limitations: None
paper_summary: paper_summary: This paper discusses a core reason for modality imbalance in multimodal learning, i.e., fitting category labels. It proposes a novel multimodal learning approach to overcome the modality imbalance problem by dynamically integrating unsupervised contrastive learning and supervised 319 multimodal learning.
strength_weakness: strength_weakness: 1. The reviewer is curious about the performance of the proposed method on larger datasets, such as VGGsound.

2. When the information of two modalities is unbalanced, it is optimal to rely more on the modality that contains more information. However,  the proposed method is forcibly narrowing the modality gap. Will this lead to the restriction of the learning of strong modality and lead to sub-optimal performance?
questions: questions: See weakness
limitations: limitations: Yes, they do.
paper_summary: paper_summary: This paper proposes a novel multimodal learning approach by integrating supervised MML and unsupervised contrastive learning dynamically to address the multimodal imbalance problem. The authors begin by examining the differences in unimodal performance, analyze the reasons for these differences, and introduce unsupervised learning to eliminate them, thereby solving the problem of modality imbalance. Experiments on widely used datasets clearly demonstrate the superiority of the proposed method. The approach of analyzing modality imbalance from the label fitting perspective is particularly novel. The authors conducted comprehensive experiments to prove the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. According to Figure 3, if a function aligns more closely with the trend of the learning strategy curve than \omega(t), would it perform better than \omega(t)?
questions: questions: Please refer to weakness.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a novel multimodal learning method to address modality imbalance by dynamically integrating unsupervised contrastive learning and supervised multimodal learning. The authors design two dynamic integration strategies: heuristic and learning-based. Experimental results show that the proposed method significantly outperforms existing multimodal learning methods on several datasets.
strength_weakness: strength_weakness: **Weaknesses:**

1. **Insufficient Analysis of Modality Imbalance**: The paper claims to be the first to analyze modality imbalance from the perspective of fitting category labels. However, only a simple experiment on the Kinetics Sounds dataset is presented, with an inadequate analysis. The experiment uses uniform labels and one-hot labels to obtain a mixed loss, finding that increasing the weight of LUL_ULU reduces the modality performance gap. This seems obvious, but how this phenomenon relates to the claim that "this also implies that fitting category labels is a core cause of modality imbalance in multimodal learning" is not clearly explained.
2. **Difference Between Mixed Loss and Label Smoothing**: What is the difference between the so-called mixed loss and the label smoothing technique? The authors should clarify this distinction. Label smoothing first obtains mixed labels and then calculates the loss. In the context of cross-entropy, these should be equivalent. It would be more convincing if the authors' analysis could relate to the theoretical explanations of label smoothing.
3. **Motivation and Method Inconsistency**: The paper's motivation is to alleviate multimodal imbalance by intervening in fitting category labels. However, it then introduces contrastive loss to intervene in the feature space. There is insufficient connection between the method and the motivation. The explanation "Ideally, contrastive learning can also mitigate the effect of modality imbalance problem. Hence, we introduce contrastive learning to impose positive intervention in multimodal learning to alleviate the impact of fitting labels." is too superficial. The authors should provide a deeper and more solid analysis of how contrastive learning helps in intervening in label fitting.
4. **Practicality of Dynamic Weight Methods**: Section 3.3 introduces heuristic and learning-based dynamic weights. The heuristic dynamic weights do not perform well in experiments; the learning-based dynamic weights involve bi-level optimization. Although the authors used an approximation method to derive Equation (3), it still involves calculating the Hessian matrix and its inverse. The authors did not discuss the computational cost or approximation techniques, making the practicality of this method questionable.
questions: questions: Please refer to weakness and address my concerns.
limitations: limitations: N/A

Comments: title: Response for Comments.
Please Review Rebuttal Response
Response for Rebuttals.
comment: Thank you very much for your positive feedback. I'm pleased that my response effectively addressed your concerns.
Thanks for your response.

We regret that your concerns have not been fully addressed. 
Our logic is not “intervening in label fitting can reduce the differences between modalities in the feature space. **Thus**, the authors introduced contrastive loss, claiming that it was intended to narrow the modality gap by intervening in the labels”. Intervening in label fitting is just one reason to introduce contrastive learning. More importantly, contrastive learning aims to learn similar representations for data pairs across different modalities [1], thereby aligning the multimodal representations. Hence, we utilize contrastive learning to intervene in label fitting, as we metioned in our response to reviewer g7cu. In other words, contrastive learning is chosen for two reasons, not just one.

Let us explain this from another perspective. Both noise label and contrastive learning can intervene in label fitting. However, we find that noise label leads to overall performance degradation, which contradicts our original motivation. In contrast, contrastive learning effectively achieves multimodal feature alignment [1]. Hence, we utilize contrastive learning to intervene the label fitting. Furthermore, this intervention is dynamic as the learning process. In the initial stage, the model primarily focuses on feature learning. As training progresses, label fitting becomes increasingly important. The experiments also verify this viewpoint.

Regarding the theoretical analysis you mentioned, we kindly request that you elaborate on the specific concerns or the particular aspect of theoretical analysis you would like to see. We believe this would greatly assist us in improving our work.

We look forward to hearing your valuable response to our comments.

[1]. Fan, Yunfeng, et al. Detached and Interactive Multimodal Learning. ACMMM, 2024.
I appreciate the efforts made by the authors during the rebuttal. However, my main concern remains with Question 3, i.e., Motivation and Method Inconsistency, has not been adequately addressed. The authors showed that intervening in label fitting leads to a reduction in the performance gap between modalities, but also results in overall performance degradation. Consequently, the authors concluded that intervening in label fitting can reduce the differences between modalities in the feature space. Thus, the authors introduced contrastive loss, claiming that it was intended to narrow the modality gap by intervening in the labels. However, this logic is hard for me to understand without solid theoretical analysis. I am keeping my original score.
We want to kindly remind you that our rebuttal is available for your review. If you have any further questions or require additional clarification, please feel free to reach out to us directly. Thank you very much for your time and consideration.
Thank you for the helpful response that addressed my concern. i will raise my score to weak accept
Thank you for the helpful response that addressed my concern.
Thank you for your valuable feedback and for acknowledging our work.
Thank you for your excellent efforts during the rebuttal. The authors have addressed most of my concerns, and now I believe this is a qualified manuscript for publication. I have raised my score to accept it.