Detached and Interactive Multimodal Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 11:28:35

Abstract: Recently, Multimodal Learning (MML) has gained significant interest as it compensates for single-modality limitations through comprehensive complementary information within multimodal data. However, traditional MML methods generally use the joint learning framework with a uniform learning objective that can lead to the modality competition issue, where feedback predominantly comes from certain modalities, limiting the full potential of others. In response to this challenge, this paper introduces DI-MML, a novel detached MML framework designed to learn complementary information across modalities under the premise of avoiding modality competition. Specifically, DI-MML addresses competition by separately training each modality encoder with isolated learning objectives. It further encourages cross-modal interaction via a shared classifier that defines a common feature space and employing a dimension-decoupled unidirectional contrastive (DUC) loss to facilitate modality-level knowledge transfer. Additionally, to account for varying reliability in sample pairs, we devise a certainty-aware logit weighting strategy to effectively leverage complementary information at the instance level during inference. Extensive experiments conducted on audio-visual, flow-image, and front-rear view datasets show the superior performance of our proposed method.

Main paper: 
 related work 2.1 modality competition in mml: 
 Multimodal learning is expected to outperform the unimodal learning scheme since multiple signals generally bring more information [14] . However, recent research [30] has observed that the multimodal joint training network underperforms the best unimodal counterpart. Besides, even if the multimodal network surpasses the performance of the unimodal network, the unimodal encoders from multimodal joint training perform worse than those from unimodal training [5, 32, 33] . This phenomenon is termed as "modality competition" [15] , which suggests that each modality cannot be fully learned especially for weak modalities since there exists inhibition between them. Researchers have proposed various methods to address this challenge, including gradient modulation [8, 22] , learning rate adjustment [27, 41] , knowledge distillation [6] , etc. Despite their improvement, the competition phenomenon still exists since they insist on leveraging joint training scheme with a uniform learning objective, which is the culprit for modality competition [8] . The preserved competition greatly limits the improvement of multimodal performance. In this paper, we aims to design a competition-free MML scheme which assigns isolated learning objectives to each modality without mutual inhibition, and guarantee the cross-modal interaction simultaneously.Contrastive learning (CL) [4] aims to learn an embedding space where positive samples are clustered together while negative samples are pushed apart. Traditionally, CL has been applied to unimodal scenarios, e.g., self-supervised learning [12, 16] , domain generalization [18, 40] and few-shot learning [21, 38] . In recent years, multimodal contrastive representation learning (MCRL) [19, 24] has been proposed to learn a shared feature space where the semantically aligned cross-modal representations are acquired. In MCRL, the paired multimodal samples are viewed as positive samples while the mismatched sample pairs are considered as negative samples. The cross-modal contrastive loss aims to pull the positive representations close in the instance level. MCRL has achieved great success yet. Multimodal pretrained models [10] emerged based on it, e.g., the vision-language models UniCL [37] , FILIP [39] , audio-text model CLAP [7] and audio-visual model CAV-MAE [11] . However, these methods are designed to align shared information in different modalities while overlooking the learning about the modality-specific and complementary features. In this paper, we aim to achieve crossmodal interaction during the unimodal learning process via the complementary knowledge transfer based on CL. 
methodology: 
 In this section, we analyze the modality competition problem and elaborate on the details of our proposed DI-MML. We mainly focus on a multi-class classification task with multimodal data.Let 𝑥 be a data sample and 𝑦 = [𝐾] be the corresponding label. Without loss of generality, we consider two input modalities 𝑥 = 𝑥 1 , 𝑥 2 . EQUATIONThis is a uniform learning objective for both modalities. MML is expected to exploit the complementary information of all modalities to outperform unimodal learning, but the modality competition phenomenon limits the performance improvement of MML since the dominant modality will inhibit the learning process of other modalities. As demonstrated in Table 1 , the unimodal performance from the traditional multimodal joint training severely underperforms the results from corresponding unimodal training. In particular, one of the two modalities could be severely suppressed, e.g., visual modality in CREMA-D and AVE, and the flow in UCF101.Although several methods [22, 27, 30] have been proposed to alleviate the modality competition, we find that the culprit behind, a uniform learning objective for both modalities, has not been resolved. According to the loss function Eq. 1, we can obtain the gradient of the softmax logits output with ground-truth label 𝑦:EQUATIONwhich is the gap between the predictive probability on ground truth with the value 1. If one modality performs better (i.e., the needed gradient strength should be low) and dominates the fusion feature, the strength of generated gradient with the uniform learning objective could be weak, which cannot satisfy the requirement of greater gradient strength for the weak modality, as illustrated in Figure 2 . Therefore, removing the uniform learning objective for encoder training is the key to eliminating modality competition. Intuitively, we can perform the detached unimodal learning for each encoder independently and then fuse their outputs (features or logits). As shown in Table 1 , we fix the pretrained unimodal learned networks and fuse their information in two ways: (1) MM Clf, train a multimodal linear classifier with the output features;(2) Preds Avg, average the prediction of each modality. It is clear that they can achieve impressive improvement compared with joint training despite the restricted cross-modal interactions, indicating the necessity to eliminate competition in MML. However, there still remain some challenges. Firstly, due to the heterogeneity between modalities, independent unimodal training may lead to disparate latent feature spaces. The correlations between modalities are ignored, making it difficult to fuse information effectively. For example, MM Clf on CREMA-D and UCF101 is worse than Preds Avg since the heterogeneous feature spaces hinder the feature fusion. Secondly, according to [6] , the cross-modal interactions in joint training can help to explore the complementary information that is hard to be learned with unimodal training. Independent encoder training blocks cross-modal interactions, thus, limiting the use of multimodal complementary knowledge. Here we apply naive cross-modal logit distillation in independently unimodal training, namely CM Dist, to achieve inter-modal knowledge transfer, enabling the multimodal interactions via prediction with multimodal data as in joint training. It can be seen that CM Dist is better than MM Clf and Preds Avg on CREMA-D and UCF101, showing the potential of cross-modal knowledge transfer for multimodal interactions. Nonetheless, the naive distillation does not consider the heterogeneity between the modalities so it does not work well always (perform worse on AVE), which motivates us to design more delicate cross-modal interactive behavior.We then present our method in next subsection, which not only solves all of the above challenges but achieves consistent improvement for various datasets on both multi-and uni-modal accuracy.According to the above discussion, we separately train each modality's encoder to avoid modality competition. Meanwhile, we enable cross-modal interactions during the encoder training and fusion process, as well as inference, to exploit the complementary information between different modalities. The details are given below and the overall framework is shown in Figure 3 . Detached unimodal training. The network of each modality is updated only according to its own data and learning objectives, and there is no fusion during the update of encoders. Encoders 𝜙 1 , 𝜙 2 are equipped with corresponding classifiers 𝜓 1 and 𝜓 2 . Therefore, the logit output of modality 𝑖 is 𝒛 𝑖 = 𝑓 𝑖 𝑥 𝑖 = 𝜓 𝑖 𝜙 𝑖 𝑥 𝑖 , 𝑖 ∈ {1, 2}. The classification loss L 𝑖 𝐶𝐸 𝑥 𝑖 of each modality is independent with each other, exploiting informative knowledge for classification. Interaction during encoder training. To address the disparate feature spaces, we use a shared linear classifier (S-Clf) for different modalities to regulate the consistent feature space. Given the extracted features 𝒉 𝑖 , the logit output through the shared classifier is𝒔𝒛 𝑖 = 𝑊 𝒉 𝑖 + 𝑏, where 𝑊 = [𝑊 1 , • • • ,𝑊 𝐾 ] ∈ 𝑹 𝑑 ×𝐾 , 𝑏 ∈ 𝑹 𝑑are the parameters of S-Clf and 𝑑 is the feature dimension. According to [20, 25] , the paired features 𝒉 1 , 𝒉 2 with label 𝑦 are optimized to maximize the similarity between them with the 𝑦-th vector 𝑊 𝑦 , and hence, S-Clf forces two modalities to locate at the same feature space using 𝑊 𝑦 as the anchor. The corresponding loss for each modality is denoted as L 𝑆𝑖 𝐶𝐸 𝑥 𝑖 . Then, we need to enable the cross-modal interaction to exploit the complementary information. According to the analysis in Section 3.1, cross-modal knowledge transfer is a promising way for interactions. Considering the gap between modalities [36] , we intend to transfer the modality-level complementarities for efficient knowledge transfer and importantly do not interfere with the learning of unimodal knowledge. To achieve this, we propose a novel Dimension-decoupled Unidirectional Contrastive (DUC) loss. Due to factors such as over-parameterization and implicit regularization [2, 42] , deep networks tend to learn low-rank and redundant features, which motivates us to compensate the ineffective information present in features with the effective cross-modal complementary information.First, we need to perform dimension separation to specify the effective and ineffective dimensions for each modality. We define the effective dimensions as dimensions with better discriminative knowledge. Therefore, we devise the dimension-wise prediction to evaluate the discrimination for each modality. With all the features from modality 𝑖, we can obtain the feature centroid of each class as:EQUATIONwhere 𝑁 is the number of all samples and 𝑁 𝑘 is the number of samples belong to 𝑘-th class. And then, we can make dimensionwise evaluation by comparing the distance for each dimension with its dimensional centroid:EQUATION𝑑 (•, •) is the distance function (Euclidean distance here). 𝑟 𝑖 𝑚 can be used to assess the effectiveness of dimension 𝑚 of modality 𝑖. Larger value indicates higher effectiveness on classification. Hence, the dimension separation principle is that the effective dimensions are represented with dimensions whose dimension-wise evaluation is greater than the mean value:EQUATIONwhere r𝑖 = 1 𝑑 𝑑 𝑚=1 𝑟 𝑖 𝑚 . Through this way, the feature dimensions of each modality are divided into effective group 𝑑 𝑖 𝑒 = 𝑚|𝑟 𝑖 𝑚 > r𝑖 and ineffective group 𝑑 𝑖 𝑛𝑒 = 𝑚|𝑟 𝑖 𝑚 < r𝑖 . The dimension separation is operated after some warmup epochs, see details in Algorithm 1 in Appendix.Due to the heterogeneity between modalities, they do not shared all the effective dimensions. Hence, we then propose to transfer the effective information in modality 1 to the corresponding ineffective dimensions in modality 2 and vice verse, as shown in Figure 3 . The knowledge transfer is performed by our proposed DUC loss: whereEQUATIONEQUATIONEQUATIONh1 𝑖 = ℎ 1 𝑖,𝑚 |𝑚 ∈ 𝑑 1 𝑛𝑒 ∩ 𝑑 2 𝑒 , h2 𝑖 = ℎ 2 𝑖,𝑚 |𝑚 ∈ 𝑑 1 𝑛𝑒 ∩ 𝑑 2 𝑒 , ĥ1 𝑖 = ℎ 1 𝑖,𝑚 |𝑚 ∈ 𝑑 1 𝑒 ∩ 𝑑 2 𝑛𝑒 and ĥ2 𝑖 = ℎ 2 𝑖,𝑚 |𝑚 ∈ 𝑑 1 𝑒 ∩ 𝑑 2 𝑛𝑒 .𝑇 is the temperature. Notably, the features of h2 𝑖 and ĥ1 𝑖 do not pass gradient backward, which means we only allow the ineffective dimensions of modality 1 (2) to learn toward the corresponding effective dimensions of modality 2 (1), and do not update the effective dimensions of modality 2 (1) with DUC to prevent damage on the unimodal learning process. Hence, we let the complementary knowledge between modalities transfer unidirectionally and use the integrated knowledge for prediction to enable cross-modal interaction.The final loss for modality 𝑖 can be calculated as:EQUATIONInteraction during co-prediction. The above training process does not directly utilize the multimodal data for completing tasks, therefore, in this stage we enable the interaction during the coprediction process via training a fusion module with multimodal objective Eq. 1 while fixing the learned encoders.In the training stage, we exploit the modality-level complementary information through DUC loss. However, the complementary capacities of the different modalities may also vary in different sample pairs [31] . Therefore, we propose a certainty-aware logit weighting strategy during inference to utilize the instance-level complementarities comprehensively, as demonstrated in Figure 4 . We use the absolute certainty to evaluate the 𝑗-th instance reliability for each modality and their fusion:EQUATIONsuperscript 𝑓 denotes the output of fusion module. Then, the final output is:EQUATIONwhere more reliable modalities are assigned with higher weights. Previous multimodal contrastive loss [24] pays attention to searching for the semantic alignment between modalities, hence, the learning strength is bidirectional on the whole dimensions, i.e. the positive samples of two modalities move toward each other. Nevertheless, the alignment objective is too 'hard' that may lead to information loss, since there may be noise in part of the dimensions for specific modalities and complete alignment would partially preserve the noise, as illustrated in Figure 5 . In contrast, our DUC loss is not intended to perform semantic alignment, but rather cross-modal transfer of complementary knowledge. Therefore, we decouple the feature dimensions and perform a unidirectional crossmodal knowledge transfer to enhance the dimensions with less informative knowledge while retaining effective information unique to the current modality. It can be seen that our DUC is more 'soft', and the dimensions in 𝑑 1 𝑒 ∩ 𝑑 2 𝑒 are not required to align with each other, preserving the specific characteristics of each modality. 
experiments 4.1 dataset: 
 We use four different multimodal datasets, i.e., CREMA-D [3] , AVE [28] , UCF101 [26] , and ModelNet40. CREMA-D is an audio-visual dataset for researching emotion recognition, comprising facial and vocal emotional expressions. Emotions are categorized into 6 types: happy, sad, angry, fear, disgust, and neutral. The dataset consists of 7442 segments, randomly divided into 6698 samples for training and 744 samples for testing. AVE is an audio-visual video dataset designed for audio-visual event localization, encompassing 28 event classes and 4,143 10-second videos. It includes both auditory and visual tracks along with secondary annotations. All videos are collected from YouTube. In our experiments, we extract frames from event-localized video segments and capture audio clips within the same segment, constructing a labeled multimodal classification dataset as in [8] . UCF101 is a dataset for action recognition comprising real action videos with 101 action categories, collected from YouTube. We treat the optical flow and images of the videos as two separate modalities. The dataset consists of 13,320 videos, with 9,537 used for training and 3,783 for testing. ModelNet40 is one of the Princeton ModelNet datasets [34] with 3D objects of 40 categories, consisting of 9,843 training samples and 2,468 testing samples. Following [33] , we treat the front view and the rear view as two modalities in our experiments.For the above four datasets, we used ResNet18 [13] as the backbone encoder network, mapping input data into 512-dimensional vectors. For the input data, for the CREMA-D and AVE datasets, audio modality data was transformed into spectrograms of size 257×1,004, and visual modality data consisted of 3(4 frames for AVE) randomly selected frames from 10-frame video clips, with image size of 224×224. For the UCF101 dataset, we randomly sampled contiguous 10-frame segments from videos during training, while testing, we sampled 10-frame segments from the middle of the videos. Optical flow modality data was of size 20×224×224, and visual modality data consisted of randomly sampled 1 frame. For the ModelNet40 dataset, we utilized front and back views as two modalities. For all visual modalities, we applied random cropping and random horizontal flipping as data augmentation during training; we resized images to 224×224 without any augmentation during testing. We trained all models with a batch size of 16, using SGD optimizer with momentum of 0.9 and weight decay of 1e-4, for a total of 150 epochs, with initial learning rate of 1e-3 decaying to 1e-4 after 70 epochs. For the training of fusion module, we trained for 20 epochs, with initial learning rate of 1e-3 decaying to 1e-4 after 10 epochs. All experiments were conducted on an NVIDIA GeForce RTX 3090 GPU and a 3.9-GHZ Intel Core i9-12900K CPU.We compare DI-MML with various baselines and analyze the DUC loss to validate the effectiveness of our method. Comparison with other baselines. The compared methods are divided into two groups: with and without the uniform objective for encoder training. Only the MM Clf, Preds Avg and our DI-MML do not utilize the uniform objective. The results are shown in Table 2 , we not only report the multimodal performance and also the unimodal accuracy. To ensure the fairness of the comparison, we fix the parameters of their unimodal encoder networks after multimodal training, and evaluate their unimodal performance by training a classifier independently. It can be that the methods with the uniform objective (joint training, MSES, MSLR, OGM-GE, PMR and UMT) are all suffered from severe modality competition as their unimodal performance is generally lower than the best unimodal training counterpart, especially on Visual in CREMA-D and AVE, Flow in UCF101 and Rear in ModelNet40. MSES, MSLR, OGM-GE and PMR regulate the learning progress of modalities by adjusting the learning rates or gradients of different modalities, which alleviates modality competition to some extent, but they are difficult to completely eradicate it. UMT maintains the unimodal performance better, but it requires pretrained unimodal models for distillation, which is expensive and impractical. In contrast, our method completely avoid the modality competition, resulting in comparable or even the best unimodal performance (improved by up to 3.11% and 3.44% on Flow and Image of UCF101) and the best multimodal performance (improved by up to 6.32% on CREMA-D) on all four datasets. Besides, we do not require additional computational cost for encoder training. Compared with MM Clf and Preds Avg, our DI-MML enables cross-modal interactions and complementary knowledge transfer during the encoder training. Therefore, our method can achieve both better multimodal and unimodal performance on these datasets. These results show that our approach is indeed competition-free, which is the key difference compared with previous methods. It also suggests that the proposed cross-modal interactions via knowledge transfer are effective.Ablation study. There are four main components in our method: two-stage training scheme (TS, i.e. encoders and fusion module are trained separately), shared classifier (S-Clf), dimension-decoupled unidirectional contrastive loss (DUC), and logit weighting (LW). Here, we perform an ablation study to explore the influence of various combinations of these components. The experiments are conducted on CREMA-D and AVE. As demonstrated in Table 3 , applying TS denotes the MM Clf method, which is better than Joint training because there is no modality competition. The shared classifier can align a feature space for different modalities and achieve considerable improvement on CREMA-D. The DUC loss facilitates cross-modal interaction and knowledge transfer, helping to achieve complementary knowledge utilisation at the modality level. Similarly, LW enables complementary knowledge integration at the instance level, both of them are important for multimodal performance enhancement. As discussed above, the four components are all essential in our method.Analysis on DUC loss. The DUC loss is the central technique in our method to enhance the cross-modal interaction during the encoder training stage. In Section 3.4, we compare the differences between DUC and traditional multimodal contrastive learning loss in terms of aim and formality. Here, we give more experimental results to show the superiority of our method. The results are shown 𝑖 ) move toward each other as traditional contrastive loss. It is clear that using traditional contrastive loss performs worst as it does not consider retaining the modality-wise complementary information. Applying DBC achieves improvement since the it does not affect the learning of effective dimensions shared by modalities (i.e., 𝑑 1 𝑒 ∩ 𝑑 2 𝑒 ). However, the noise information in the ineffective dimensions is preserved as illustrated in Figure 5 . Our DUC loss both preserves the unimodal knowledge and facilitates inter-modal cooperation through complementary knowledge transfer, resulting in the best multimodal results. In Figure 6 , we demonstrate the t-SNE [29] feature visualization for each modality on CREMA-D. Figure 6a showcases that there are no clear decision boundaries for visual features for joint training, consistent with its poor performance. As shown in Figure 6b , although applying contrastive loss in our method compensates for the gap between different modalities in feature space, the noise in visual modality is also preserved and transferred to audio modality to some extent, leading to worse multimodal performance. With the optimization of our DUC loss as shown in Figure 6c , the features of both modalities are more clearly clustered, besides, share a more similar distributional structure.Analysis on dimension separation. In this paper, we perform the dimension separation to divide dimensions into effective and ineffective parts. The separation results are displayed in Table 5 . The effective dimensions for both modalities take up about half or more (feature is a 512-dimensional vector), and their overlap also accounts for only about half of effective dimensions, indicating that there are enough dimensions to ensure cross-modal knowledge transfer. The performance of corresponding dimension sets of effectiveness and ineffectiveness is shown in Table 6 . When we evaluate the performance of effective dimension set, the values of ineffective dimensions are set to 0 and vice verse, removing its influence on the output prediction. The performance of effective dimensions is much better than that of ineffective dimensions, indicating that our dimension separation scheme is reasonable and effective.Effective dimension evaluation. In this paper, we devise the dimension-wise prediction as in Eq. 4 to evaluate the effectiveness of each dimension. Here, we compare our dimension-wise prediction with two other evaluation metrics: L2-norm and Shapley Value.According to [23] , the L2-norm of the features gives an indication of their information content, thus it can be used as a metric to measure the effectiveness of each dimension. And shapley value can also be used to identify important features (dimensions here) by removing specific content for prediction. As depicted in Table 7 , our proposed framework has significant enhancements with any evaluation method, showing the robustness of our DI-MML framework. Besides, among the three methods, our dimension-wise prediction performs the best on different datasets, indicating its validity for evaluating the dimensionally discriminative information. Hyperparameter sensitivity. In the calibration of our DI-MML, we encounter two hyperparameters to determine: 𝜆 𝑠 and 𝜆 𝐷 in Eq. 7, determining the strength for feature space alignment and crossmodal knowledge transfer respectively. Here, we explore the effects of them by varying their values as illustrated in Figure 7 . It is clear that the performance on DI-MML is marginally affected by 𝜆 𝑠 and 𝜆 𝐷 , suggesting the insensitivity of our method to hyperparameters. Despite some fluctuations in performance with hyperparameters, our method still demonstrates excellent effectiveness, i.e., being consistently better than joint training. In this paper, we select 𝜆 𝑠 = 1 and 𝜆 𝐷 = 1 for the best accuracy according to the obtained results. Robustness on Batch size. To analyze the effect of batch size of our method, we demonstrate the results with different batch sizes on CREMA-D and AVE, varying from 8 to 64. It can be seen that small batch size could lead to better performance on both joint training and DI-MML, and our method consistently outperforms joint training on all the batch sizes, indicating the robustness of our DI-MML with respect to batch size. In this paper, we set batch size to 8 to get the best results for the four datasets. 
conclusion: 
 In this paper, we analyze the multimodal joint training and argue that the modality competition problem comes from the uniform learning objective for different modalities. Therefore, we propose to train multimodel encoders separately to avoid modality competition. To facilitate the feature space alignment and cross-modal interaction, we devise a shared classifier and the dimension-decoupled unidirectional contrastive loss (DUC) to achieve modality-level complementarities utilization. And then, the learned encoders are frozen and a fusion module is updated for interaction during co-prediction. Considering the reliability differences on various sample pairs, we further propose the certainty-aware logit weighting strategy to exploit instance-level complementarities comprehensively. Through extensive experiments, our DI-MML outperforms all competing methods in four datasets. We also showcase that our method can further promote the unimodal performance instead of inhibiting them. In the future, we can investigate other types of cross-modal interactions and focus on multimodal tasks such as detection or generation instead of only classification. Besides, identifying the specific semantics in each dimension may be helpful to further evaluate the informative dimensions.  
: 
 modalities may contain distinctive and complementary knowledge, which allows MML outperforms unimodal learning [14] . Despite the advances in MML, fully exploiting the information from multimodal data still remains challenging.Recent studies [15, 30] have found that the unimodal encoder in MML underperforms its best unimodal counterpart trained independently. Huang et al. [15] attribute the cause of this phenomenon to modality competition, where the dominant modality hinders the learning of other weak modalities, resulting in imbalanced modalitywise performance. Existing solutions [9, 22, 41] mainly try to modulate and balance the learning paces of different modalities, which generally follow the joint training framework and a uniform learning objective is employed for all modalities, as shown in Figure 1 . However, according to [8] , the fused uniform learning objective is actually the reason for modality competition since the backward gradient predominantly comes from certain better modalities, hindering the learning of others, as illustrated in Figure 2 . Meanwhile, [6] has declared that despite the competition between modalities, the interactions in joint training can facilitate the exploitation of multimodal knowledge. Therefore, existing solutions are caught in the dilemma of mitigating competition and facilitating interactions, where the competition issue has not been eradicated, limiting further improvements in multimodal performance.In this paper, we empirically reveal that eliminating modality competition may be more critical for multimodal learning, which motivates us to design a competition-free training scheme for MML. Therefore, we decide to abandon the joint training framework and construct the detached learning process via assigning each modality with isolated learning objectives. Although the naive detached framework, i.e., performing unimodal training independently, could avoid modality competition, it still suffers from the following two challenges, limiting its further improvement.• Disparate feature spaces. The intrinsic heterogeneity between modalities usually requires different processing strategies as well as model structures, which may lead to disparate feature spaces based on independent unimodal training and then pose a great challenge on fusing the extracted multimodal knowledge. • Lack of cross-modal interactions. The cross-modal interactions can help to facilitate the exploitation of multimodal knowledge. However, independent unimodal training insulates the interactions for both encoder training and multimodal prediction process, limiting the learning and exploitation of multimodal complementary information.To address all above issues, we propose a novel DI-MML that achieves cross-modal Interactions under the Detached training scheme. Unlike independent unimodal training, we first apply an additional shared classifier to regulate a shared feature space for various modalities, alleviating the difficulty on fusion process. To encourage cross-modal interactions during encoder training, we propose a Dimension-decoupled Unidirectional Contrastive (DUC) loss to transfer the modality-level complementary knowledge. We introduce the dimension-wise prediction to evaluate the discriminative knowledge for each dimension and then divide feature dimensions into effective and ineffective groups, enabling the complementary knowledge transfer within modalities and maintaining the full learning of each modality itself. Further, to enhance interactions during multimodal prediction, we then freeze the learned encoders and train a fusion module. Considering that there may be reliability disparities between modalities in sample pairs, we devise a certainty-aware logit weighting strategy during inference so that we can fully utilize the complementarities at the instance level.Our main contributions can be summarized as follows:• To the best of our knowledge, this paper is the first to completely avoid modality competition while ensuring complementary cross-modal interactions in MML. We propose a novel DI-MML framework that trains each modality with isolated learning objectives. • We design a shared classifier to regulate a shared feature space and a Dimension-decoupled Unidirectional Contrastive (DUC) loss to enable sufficient cross-modal interactions, which exploits modality-level complementarities. • During inference, we utilize the instance-level complementarities via a certainty-aware logit weighting strategy. • We perform extensive experiments on four datasets with different modality combinations to validate superiority of DI-MML and its effectiveness on competition elimination. 
 
 Appendix:  


Decision: The final ratings of this paper are three borderline accept. After rebuttal, all reviewers agree that the paper's strengths warrant acceptance, and the AC agrees.

Reviews: review: paper_summary: paper_summary: This paper focuses on the problem of multimodal learning. The authors propose a framework named DI-MML, which avoids modality competition by separately training each modality encoder with isolated learning objectives. It enhances cross-modal interaction via a shared classifier and uses a dimension-decoupled unidirectional contrastive (DUC) loss for knowledge transfer. A certainty-aware logit weighting strategy is employed to leverage complementary information during inference. Experiments on various datasets show the superior performance of DI-MML.
strength_weakness: strength_weakness: The experimental evaluations are conducted on multiple different and challenging multimodal datasets, and the proposed solution demonstrates superior performance compared to existing methods. Additionally, detailed ablation experiments are performed to verify the influence of network structure design and hyperparameter selection on the accuracy of the proposed method.
limitations: limitations: 1) The authors have provided a dual-modal recognition scheme and experiments. Could the proposed method be extended to tasks involving three or more modalities, such as text-image-audio analysis?

2) The two modalities used have similar data and network structures. Can this method be extended to multimodal data with greater differences or different kind of network (2D ResNet and PointNet), such as text and images, images and point clouds, etc.?

3) The authors have studied the competitive relationships between modalities. However, when one modality is contaminated, such as noisy audio or blurred images, would the proposed method be affected?
4) Tables 1 and 2 use the UCF101 dataset, which has achieved approximately 99% classification accuracy. However, the accuracy presented in the tables differs significantly. Please provide some clarification on this discrepancy.
paper_summary: paper_summary: This paper introduces DI-MML, a detached Multimodal Learning (MML) framework that aims to address the modality competition issue inherent in traditional joint learning approaches. DI-MML employs a detached training strategy for each modality-specific encoder to avoid competition. The paper demonstrates DI-MML's superior performance across various datasets, including audio-visual, flow-image, and front-rear view pairs.
strength_weakness: strength_weakness: 1. This paper addresses a critical challenge in multimodal learning - modality competition. By introducing a novel competition-free training scheme for MML.

2. The authors propose a Detached Interactions MML (DI-MML) framework, which is an innovative method that combines detached training with cross-modal interactions, making full use of modality-level complementarities.

3. The paper proposes specific strategies, including a shared classifier, a Dimension-decoupled Unidirectional Contrastive (DUC) loss, and a certainty-aware logit weighting strategy. These strategies help tackle disparate feature spaces and the lack of cross-modal interactions.
limitations: limitations: 1.The paper has repeatedly stated "completely avoid modality competition while ensuring complementary cross-modal interactions in MML". How does the author substantiate the claim of completely avoiding modality competition rather than mitigating it to some extent? In the AVE dataset, the performance of the Audio modality in your proposed method doesn't surpass the performance of independently trained Audio modality.

2.What is the rationale behind choosing the CREMA-D, AVE, UCF101 datasets instead of a specific multi-modal dataset related to a particular task? Is there a correlation between the datasets chosen? To draw more reliable conclusions, we recommend selecting larger datasets, such as the VGG Sound and Kinetics-400 datasets used by the comparative methods discussed in this paper.

3.Is your method adapted only for classification tasks? If so, this might pose constraints on the practical application of the proposed method in the paper.

4.In the original paper of the comparative methods in this paper, the accuracy of the UCF101's RGB encoder and Opt-Flow encoder were 77.08% and 74.99% respectively. However, the accuracy reported in your paper is only around 50%.
paper_summary: paper_summary: This paper proposes Detached and Interactive MultiModal Learning (DI-MML) to address the modality competition problem in multimodal learning. In contrast to existing works that adjust modality-specific learning strength, DI-MML focuses on cross-modal detachment and interaction in the way of separated modality training, shared classifier, and dimension-decoupled unidirectional contrastive loss. Empirical results on multimodal classification datasets demonstrate the performance of DI-MML.
strength_weakness: strength_weakness: 1. The illustrated figures are clear and helpful for understanding.
2. The codebase attached in the supplementary material is well-organized.
3. The experimental results are relatively sufficient and verify the effectiveness of each module, such as DUC loss.
limitations: limitations: 1. The related works are not comprehensive. To my knowledge, a classical method named Gradient-Blending [1] and a recent method named I2MCL [2] are not mentioned or involved in this paper, both of which are proposed to solve the same problem, modality competition, as this work. Especially, Gradient-Blending also considers detaching multimodal learning, so I highly recommend comparing it empirically.
2. According to Table 3, it is observed that TS module (training separately) contributes most to solving the problem of modality competition, so it is necessary to explore whether a suitable reweighting of $L_{CE}^{i}$ is enough to improve the performance, that is, $L_{CE}^1+L_{CE}^2$ should be $\lambda_{CE}^{1}L_{CE}^{1}+\lambda_{CE}^{2}L_{CE}^{2}$.
3. The explanation on Figure 6 is confusing. Since this method aims to interact among multiple modalities, there should not be a clear boundary between the two modalities in Figure 6(c). Instead, the phenomenon in Figure 6(b) is more consistent with the expectation, that is, audio and visual features interact with each other to prevent modality competition. I think the reason why there is a clear boundary in Figure 6(c) may be because the authors discard the overlap dimensions $d_e^1\cap d_e^2$ when calculating effective dimensions in DUC loss, so some of the dimensions are interacted while others not, thereby showing a clear boundary.
4. A question is how to solve the problem with three or more modalities with this DI-MML method?

[1] Wang, Weiyao, Du Tran, and Matt Feiszli. "What makes training multi-modal classification networks hard?." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[2] Zhou, Yuwei, et al. "Intra-and Inter-Modal Curriculum for Multimodal Learning." Proceedings of the 31st ACM International Conference on Multimedia. 2023.