Ada2I: Enhancing Modality Balance for Multimodal Conversational Emotion Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 06:37:32

Abstract: Multimodal Emotion Recognition in Conversations (ERC) is a typical multimodal learning task in exploiting various data modalities concurrently. Prior studies on effective multimodal ERC encounter challenges in addressing modality imbalances and optimizing learning across modalities. Dealing with these problems, we present a novel framework named Ada2I, which consists of two inseparable modules namely Adaptive Feature Weighting (AFW) and Adaptive Modality Weighting (AMW) for feature-level and modality-level balancing respectively via leveraging both Inter- and Intra-modal interactions. Additionally, we introduce a refined disparity ratio as part of our training optimization strategy, a simple yet effective measure to assess the overall discrepancy of the model's learning process when handling multiple modalities simultaneously.  Experimental results validate the effectiveness of Ada2I with state-of-the-art performance compared against baselines on three benchmark datasets including IEMOCAP, MELD, and CMU-MOSEI, particularly in addressing modality imbalances.

Main paper: 
 introduction: 
 Multimodal learning is an approach to building models that can process and integrate information from multiple heterogeneous data modalities [2, 20, 21] , including image, text, audio, video, and table. Since numerous tasks in the real world involve multiple modalities, multimodal learning has become increasingly important and attracted widespread attention as an effective way to accomplish these tasks. In recent years, the field of Emotion Recognition in Conversations (ERC) has witnessed a surge in effective models [8, 26, 29] . Moving beyond unimodal recognition, the utilization of multimodal data offers a multidimensional perspective for more nuanced emotion discernment [9, 19, 24] . Consequently, the incorporation of multimodal data is a natural evolution for enhancing emotion D ,(02&$3 recognition in conversations. However, the widespread adoption of multimodal learning has revealed underlying challenges, with a primary focus on modality imbalances. These imbalances entail disparities in the contributions of individual modalities to the final decision-making process.As illustrated in Figure . 1, the text modality quickly addresses the overall model performance and the joint logit scores, whereas the visual and audio modalities remain under-optimized throughout the training process. In addressing modality imbalance, diverse terminologies have emerged to characterize this phenomenon and explore its underlying causes. Terms such as "greedy nature" [38] , "modality collapse" [15] , and "modality imbalance" [6, 22] have been employed in various studies. These terms are associated with factors such as the "suppression of dominant modalities" [25] , "different convergence rates" [35] , "diminishing modal marginal utility" [36] , or "modality competition" [14] . In essence, two primary perspectives emerge regarding this problem [36] : firstly, modalities exhibit varying levels of dominance, with models often overly reliant on a dominant modality with the highest convergence speed, thereby impeding the full utilization of other modalities with slower convergence speeds. Secondly, modal encoder optimization varies, necessitating the adoption of multiple strategies. Some approaches [7, 25] attempt to modulate the learning rates of different modalities based on the fusion modality. However, these approaches often overlook the impact of intra-modal data enhancement [44] . For instance, right from the initial representations through the modal encoder, the outputs can lead to misleading final results, resulting in its weakened position across all modalities. Hence, from the outset, it is crucial to enhance representations for each modality, regardless of whether they are weak or strong, as it can affect the imbalance in learning across modalities.Moreover, current methodologies primarily focus on interactions between pairs of modalities [6, 22, 25, 39] , resulting in complex computations and inadequate treatment across all modalities. These methods are commonly applied in tasks such as audio-visual learning [25, 39] and multimodal affective computing [44] , often using datasets related to sarcasm detection, sentiment analysis, or humor detection. However, there is a lack of methods explicitly tailored for multimodal emotion recognition in conversation, especially for well-known multimodal datasets like IEMOCAP [3] , MELD [27] , and CMU-MOSEI [1] . Additionally, in recent prominent studies [17, 32] , while overall performance for multimodal emotion recognition tasks has notably increased, a closer examination of the "importance of modality" reveals that pairwise modalities consistently fail to achieve satisfactory performance, creating a significant gap compared to leveraging all three modalities simultaneously. Therefore, it is crucial to simultaneously leverage learning from all modalities while also significantly enhancing the capabilities of weaker modalities to improve the overall learning performance of multimodal emotion recognition models in practical applications.In this paper, we propose a novel framework named Ada2I that addresses imbalances in learning across audio, text, and visual modalities for multimodal ERC. It consists of two primary modules including Adaptive Feature Weighting (AFW) and Adaptive Modality Weighting (AMW) for feature-level and modality-level balancing respectively in the consideration of Inter-and Intra-modal interactions. Focusing on feature-level balancing using Adaptive Feature Weighting (AFW), we apply tensor contraction to infer feature-aware attention weights for each modality, which aims to produce a feature-level balanced representation for each conversation. As an important component of AFW, Attention Mapping Network controls the balancing via maximizing the alignment between unimodal features and their corresponding attention coefficients. For modality-level balancing using Adaptive Modality Weighting (AMW), we further exploit feature-level balanced representations from the preceding AFW module to generate modality-level balanced ones through modality-wise normalization of features and learning weights before being used to enhance the emotion recognition. Additionally, we utilize the concept of disparity ratio, although with modifications compared to the study by Peng et al. [25] , called OGM-GE, as a value to supervise the training process and evaluate the model. Specifically, while OGM-GE [25] introduced gradient modulation for pairs of modalities, we refine it to handle all three modalities simultaneously-textual, visual, and audio-in the Multimodal emotion recognition in conversation task. This adjustment reduces model complexity and overall processing time, leading to enhanced efficiency.To summarize, our contributions are as follows:• We propose an end-to-end framework named Ada2I that addresses the issue of imbalance learning across modalities comprehensively for the multimodal ERC task. It not only considers modality-level imbalances but also leverages feature-level representations to contribute to the balancing step in the learning process.• With two modules intricately designed yet inseparable, Adaptive Feature Weighting (AFW) is crafted to enhance the representation of each conversation at the feature level, while Adaptive Modality Weighting (AMW) is proposed to optimize the modality-level learning weights during training. Additionally, we redefine the disparity ratio, a simple yet effective measure, to assess the overall discrepancy of the model's learning process when simultaneously handling multiple modalities, rather than just two as in the original approach from [25] . • Our empirical experiments illustrate the effectiveness and enhancements provided by our method in comparison to existing approaches across three prevalent multimodal ERC datasets: IEMOCAP [3] , MELD [27] , and CMU-MOSEI [1] . The paper is structured as follows: Section 2 presents related work while we descibe the proposed framework Ada2I in Section 3. Experiment settings are explained in Section 4, and Section 5 covers the experimental evaluation and results. Finally, Section 6 summarizes findings. 
related work 2.1 multimodal emotion recognition in conversation: 
 Multi-modal Emotion Recognition (ERC) has emerged as a focal point within the affective computing community, garnering significant attention in recent years. The integration of multimodal data provides a multidimensional perspective, enabling a more nuanced understanding of emotions. Moreover, researchers have increasingly turned to multimodal fusion techniques, combining text, audio, and visual cues to enhance multimodal ERC performance [9, 10, 16, 18, 24] . ICON [9] employs two Gated Recurrent Units (GRUs) to capture speaker information, supplemented by global GRUs to track changes in emotional states throughout conversations. Similarly, MMGCN [37] utilizes Graph Convolutional Networks (GCNs) to capture contextual information, effectively leveraging multimodal dependencies and speaker information. On the other hand, Multilogue-Net [30] introduces a solution utilizing a context-aware RNN and employing pairwise attention as a fusion mechanism. TBJE [4] , adopts a transformer-based architecture with modular co-attention to jointly encode multiple modalities. Additionally, COGMEN [16] is a multimodal context-based graph neural network that integrates both local (speaker information) and global (contextual information) aspects of conversation. Moreover, CORECT [24] employs relational temporal Graph Neural Networks (GNNs) with cross-modality interaction support, effectively capturing conversation-level interactions and utterance-level temporal relations. GraphMFT [18] utilizes multiple enhanced graph attention networks to capture intra-modal contextual information and inter-modal complementary information. More recently, DF-ERC [17] emphasizes both feature disentanglement and fusion while taking into account both multimodalities and conversational contexts. Moreover, AdaIGN [32] employs the Gumbel Softmax trick to adaptively select nodes and edges, enhancing intra-and cross-modal interactions. While these methods primarily focus on designing model structures, they overlook the challenges posed by modality imbalance during multimodal learning.Despite the suggestion by [13] that integrating multiple modalities could enhance the accuracy of latent space estimations, thereby improving the efficacy of multimodal models, our investigation within the multimodal ERC task reveals a phenomenon contradicting this notion. The problem of modality imbalance persists as a significant challenge in multimodal learning frameworks, particularly in tasks like multimodal Emotion Recognition in Conversations (ERC).Conventional methods often prioritize one modality over others, assuming that certain types of sensory data are more relevant for a given task. For example, textual cues may receive greater emphasis, while visual or audio cues alone might be prioritized [16, 24, 37] .Current methodologies for addressing imbalanced multimodal learning primarily focus on tasks such as audio-visual learning with a focus on optimizing pairwise modality learning [6, 25, 39] , sentiment analysis, and sarcasm detection [44] . However, these approaches often have task-specific limitations and framework restrictions, limiting their broader applicability. For instance, Wang et al. [35] identified that different modalities overfit and generalize at different rates, leading to suboptimal solutions when jointly trained using a unified optimization strategy. Peng et al. [25] proposed OGM-ME method where the better-performing modality dominates the gradient update, suppressing the learning process of other modalities. MMCosine [39] employs normalization techniques on features and weights to promote balanced and improved fine-grained learning across multiple modalities. Notably, there is a lack of specific approaches tailored for multimodal Emotion Recognition in Conversations (ERC) apart from the work by Wang et al. [36] . Recently, Wang et al. [36] observed a phenomenon referred to as "diminishing modal marginal utility" and proposed finegrained adaptive gradient modulation, which was applied to ERC, while I 2 MCL considers both data difficulty and modality balance for multimodal learning based on curriculum learning for affective computing, though not specifically for emotion recognition. To comprehensively address the challenge of modality imbalance in multimodal ERC, we propose an end-to-end model that ensures balance among text, audio, and visual modalities during training. 
methodology: 
 In the context of a conversation 𝐶 with 𝑁 utterances {𝑢 1 , 𝑢 2 , . . . , 𝑢 𝑁 }, the task of Emotion Recognition in Conversations (ERC) is to predict the emotion label for each utterance in the conversation from a predefined emotion category set E. Each utterance is associated with 𝑀 modalities, i.e. textual (t), audio (a), and visual (v) modalities, represented as:EQUATIONwhere 𝑢 𝑖 ∈ R 𝑀 ×𝑑 , 𝑑 signifies the dimension of modal features. For each modality 𝑚, we derive multimodal features {X 𝑚 } 𝑚∈ {𝑡,𝑎,𝑣 } ∈ R 𝑑 𝑚 ×𝑁 for the conversation 𝐶. Here, {𝑑 𝑚 } 𝑚∈ {𝑡,𝑎,𝑣 } is the feature dimension of each modality.In this section, we outline our proposed model Ada2I, including its main sub-modules: Modality Encoder, Adaptive Feature Weighting and Adaptive Modality Weighting. We also refine the disparity ratio metric as part of our Training Optimization Strategy. The architecture of our model is shown in Figure 2 .Given a conversation 𝐶, a Transformer [33] network is utilized as the encoder to generate a unimodal representation Z 𝑚 ∈ R 𝑁 ×𝑑 𝑚 respecting to the modality 𝑚 as:EQUATIONwhere the function 𝜙 (𝜃 (𝑚) ) is the Transformer network with learnable parameter 𝜃 (𝑚) .3.2 Adaptive Feature Weighting (AFW) We employ a tensor-ring-based generation function to retrieve the multi-interaction multimodal query tensor Q and key tensor K from the input modality presentations Z 𝑚 . Specifically, we compute Q and K as follows:Q = Tr{G (𝑡 ) 𝑄 , G (𝑎) 𝑄 , G (𝑣) 𝑄 } ∈ R 𝑑 𝑡 ×𝑑 𝑎 ×𝑑 𝑣 K = Tr{G (𝑡 ) 𝐾 , G (𝑎) 𝐾 , G (𝑣) 𝐾 } ∈ R 𝑑 𝑡 ×𝑑 𝑎 ×𝑑 𝑣 (3)Here, Tr{.} represents the tensor-ring decomposition function, which naturally provides the low-rank core tensor representations G 𝑚 𝑄 and G 𝑚 𝐾 for each modality. To perform multimodal attention in the tensor space, we need to compute the attention coefficient matrix, Θ, from the tensorized input. To achive this, we can first compute the Tensor-ring Key representation and Tensor-ring Query representation of input data, G 𝑚 𝑄 ∈ R 𝑑 𝑚 ×𝑟 𝑠 ×𝑟 𝑤 and G 𝑚 𝐾 ∈ R 𝑑 𝑚 ×𝑟 𝑠 ×𝑟 𝑤 , where 𝑚 ∈ {𝑡, 𝑎, 𝑣 }, the index 𝑠, 𝑤 ∈ {1, 2, 3}, and 𝑠 ≠ 𝑤. The attention coefficient matrix Θ of modality 𝑚 is formulated as follows:EQUATION)where ⊙ denotes the element-wise product, √︁ 𝑑 𝑘 is a scaling factor. More specifically, the modality 𝑚 core tensor G 𝐾 and G 𝑄 are expressed as follows:EQUATIONwhere 𝑚 ∈ {𝑡, 𝑎, 𝑣 }, 𝑊EQUATIONEQUATIONEQUATION𝐾 𝑚 ∈ R 𝑑 𝑚 ×𝑟 𝑤 are the linear transformation matrix; ⊗ 1 denotes the mode-1 Khatri-Rao product. . This module addresses the varying impact of each modality on inter-modality and intramodality interactions using attention mechanism. First, we calculate the attention pooling matrices A (𝑚) ∈ R 𝑟 𝑠 ×𝑟 𝑤 by averaging Θ (𝑚) across the modality dimension 𝑑 𝑚 , 𝑚 ∈ {𝑡, 𝑎, 𝑣 }. Inspired by MMT [31] , the feature-aware attention matrix 𝐴𝑡𝑡 𝑚 ∈ R 𝑁 ×𝑑 𝑚 for a given modality 𝑚 is computed as follows:EQUATIONwhere × 1 3 is the 𝑚𝑜𝑑𝑒 -( 13 ) tensor contraction. The feature-aware balanced representation Z 𝑓 -𝑎𝑑𝑎𝑝𝑡 𝑚 ∈ R 𝑁 ×𝑑 𝑚 of the conversation C for a given modality m is computed as:EQUATIONwhere 𝛽 ∈ [0, 1] is a balancing parameter to regulate the contribution of the original unimodal feature vector Z 𝑚 .Our key focus is to achieve balanced contributions from each modality during the training. Similar to [39] , we observe the imbalance problem in multimodal ERC through experiments analyzing the modality-wise weight in norm of each label during training. Apparently, the dominant unimodal encoder, e.g., text, tends to have its weight in norm increase much faster than the weaker modalities, i.e., audio and visual, leading to divergent unimodal logit scores and distorting the joint fusion representation. Inspired by [34, 43] , we propose to incorporate modality-wise L2 normalization to properly weight features, mitigating imbalances arising from differing data distributions and noise levels across modalities. This dynamic adjustment prevents any single modality from dominating the fusion process, thus enhancing overall performance. Therefore, the modality-level balanced representation Z 𝑚-𝑎𝑑𝑎𝑝𝑡 of the given conversation is calculated as follows:EQUATIONwhere 𝑊 𝑚 ∈ R 𝑑 𝑚 × | E | symbolizes the output matrix of the model pertaining to modality 𝑚, and E is the set of emotion classes.For emotion recognition, we feed Z 𝑚-𝑎𝑑𝑎𝑝𝑡 , into the mulilayer preceptron (MLP) with ReLU activation function to compute the output ŷ𝑖EQUATIONThe output ŷ𝑖 is utilized to predict emotion labels.First, we investigate the standard cross-entropy loss for this downstream task, i.e., mutilmodal ERC as:EQUATION)where 𝐵 is the batch size. Second, in order to align between the original unimodal representation of modality 𝑚 and its respective feature-aware attention weights as Eq (6), we employ Attention Mapping Network as follows:ÂEQUATIONwhere Φ 𝑚 (•) is a feed-forward neural network with the parameter 𝜓 (𝑚) , Â 𝑡𝑡 𝑚 ∈ R 𝑁 ×𝑑 𝑚 is the feature-aware self-attention weights of the modality 𝑚. To enhance feature-level balance across all modalities, we introduce a L1-norm loss L 𝑓 𝑒𝑎𝑡𝑢𝑟𝑒 as:EQUATIONAdditionally, we also consider the modality-level balance loss L 𝑚𝑜𝑑𝑎𝑙 , which is computed as:EQUATIONwhere Zrepresents the output of the 𝑗-th class for the 𝑖-th sample. Finally, we combine the all loss functions into a joint objective function, which is used to optimize all trainable parameters in an end-to-end manner.EQUATIONRecent studies have brought attention to the challenge of handling imbalanced optimization in joint learning models, particularly when dealing with multiple modalities. Peng et al. [25] introduce the OGM-GE method to address optimization imbalances encountered during the simultaneous training of dual-modal systems, i.e., visual and audio. However, directly applying the OGM-GE method to our framework is not practical as it only deals with two modalities. In contrast, our framework caters to more than two modalities across different domains, specifically tailored for the multimodal ERC task. Therefore, leanrable parameter of encoder layer is optimized during training process as the following strategy:EQUATION)where ĝ( 𝑡 ) denotes the gradient with respect to 𝐵 𝑡 .EQUATIONWe adjust the balance of modalities through gradient parameter adjustments. For each output at step 𝑡, we compute the discrepancy ratio for each modality using the softmax of the cosine similarity 𝑡 +1 = 𝜃 (𝑖 ) 𝑡 -𝜂 • ĝ(𝜃 (𝑖 ) 𝑡 ) • 𝑘 𝑖 𝑡 + 𝜂 • ℎ(𝜃 (𝑖 )𝑡 ) end for end for between the output weights and the corresponding feature vectors:EQUATION)where I 𝑘=𝑦 𝑗 equals 1 if 𝑘 = 𝑦 𝑗 and 0 otherwise, and softmax(.) estimates the unimodal performance of the multimodal model, 𝑀 denotes the count of modalities. Specifically, for the Multimodal ERC task under consideration, we delineate three modalities: text (𝑡), audio (𝑎), and visual (𝑣). The discrepancy ratio is calculated as:EQUATIONThe learnable parameters are updated according to:EQUATION)where the modulation coefficient 𝑘 𝑚 𝑡 is determined by 1tanh(𝛼 • 𝜌 𝑚 𝑡 ) if 𝜌 𝑚 𝑡 > 1, and 1 otherwise. Here, 𝛼 is a hyperparameter controlling the degree of modulation. Additionally, to enhance the adaptability of the modulation process, Gaussian noise ℎ(𝜃 (𝑖 ) 𝑡 ) sampled from a distribution N (0, 𝑠𝑔𝑑 (𝜃 (𝑖 ) 𝑡 )) is introduced after parameter updates:EQUATIONThe training process of Ada2I is illustrated in Algorithm 1.  
baselines and evaluation metrics: 
 Baselines: Ada2I is compared against several state-of-the-art (SOTA) baseline approaches for evaluating performance in multimodal ERC, particularly addressing modality imbalance problems. For the IEMOCAP and MELD datasets, we consider baseline models such as DialogueRNN [23] , DialogueGCN [8] , MMGCN [37] , BiD-DIN [40] , and MM-DFN [11] . We report the best results obtained from [36] , which enhanced these models to address modality imbalance. Additionally, we consider other SOTA models for multimodal ERC that do not explicitly address modality imbalance, including COGMEN [16] , CORECT [24] , GraphMFT [18] , DF-ERC [17] , and AdaIGN [32] . For the CMU-MOSEI dataset, we evaluated various baseline models for sentiment classification tasks, which include both 2-class sentiment, featuring only positive and negative sentiment, and 7-class sentiment, ranging from highly negative (-3) to highly positive (+3). These baseline models include Multilouge-Net [30] , TBJE [4] , COGMEN [16] , CORECT [24] , OGM-GE [25] , and I2 MCL [44] . Notably, OGM-GE and I 2 MCL specifically address the issue of imbalanced modalities in multimodal ERC, whereas the other baseline models do not.Evaluation Metrics: Similar to prior studies [23, 36, 37] , we evaluate the effectiveness of emotion recognition using Accuracy (Acc) and Weighted F1 Score (WF1) as our primary evalucation metrics.We derive multimodal features for each utterance from acoustic, lexical, and visual modalities using a combination of models and pre-trained models, as outlined in Table 2 . We employ PyTorch1 for training our architecture and Comet 2 for logging all experiments, leveraging its Bayesian optimizer for hyperparameter tuning. Additional parameters can be found in Table 2 . 3 , our model Ada2I performs better than the previous SOTA baselines in the context of balanced modality consideration on all modality combinations on both datasets. Indeed, in the AV modality pair on the MELD dataset, traditionally deemed the weakest, we observe a substantial performance boost in Multimodal ERC. Specifically, there is a noteworthy enhancement of 10.77% on WF1 and 6.98% on Accuracy compared to the previous SOTA model. This progress effectively reduces the performance discrepancy compared to modality pairs where text plays a dominant role. We also compare Ada2I with SOTA baseline models for multimodal ERC, particularly those focusing solely on multimodal fusion and architectural design without addressing modality imbalance. Figure 3b demonstrates that our proposed Ada2I significantly reduces the performance gap in WF1 between learning from all three modalities simultaneously (T+A+V) and pair-wise modality combinations on the MELD dataset. Most notably, with the weaker CMU-MOSEI dataset: Table 4 shows that Ada2I outperforms all baseline models. Specifically, when compared to OGM-GE and I 2 MCL, two models proposed for addressing modality imbalance during training, Ada2I demonstrates superior performance across all modality combinations. When compared to other baseline models that do not consider modality balancing, Ada2I also demonstrates significant balancing capabilities, reducing the performance gap between modality pairs. For instance, in the CORECT model, the gap between T+A+V and A+V is 15.09% for 2-class sentiment, and this figure increases to 21.76% for 7-class sentiment. However, with Ada2I, these gaps are significantly reduced to 10.32% and 13.07%, respectively, underscoring the effectiveness of Ada2I in addressing modality imbalances. 
ablation study: 
 5.2.1 Balancing Interpretation. We conduct ablation studies with the two main modules of the model, AMW and AFW, to assess their impact on the Ada2I model. Additionally, through the Discrepancy Ratio, we interpret the model's balancing by observing its changes. A smaller Discrepancy Ratio indicates a more balanced optimization process. Figure 4 shows that the discrepancy ratios 𝜌 𝑡 , 𝜌 𝑣 , and 𝜌 𝑎 significantly decrease when both AMW and AFW are combined within Ada2I, with all ratios approaching approximately 1 on the IEMOCAP dataset. In contrast, when one of the modules is ablated, Here, we provide a clearer visualization of these unimodal weights before imbalance processing (Only Encoder) and in the Ada2I model in Figure 5 for the IEMOCAP dataset. It is evident that with Only Encoder, the text encoder (dominant modality) weight in norm grows much faster than audio and visual. After balancing, our model exhibits a more balanced optimization process. 
conclusion: 
 In this work, we present Ada2I, a framework designed to address modality imbalances and optimize learning in multimodal ERC. We identify and analyze existing issues in current ERC models that overlook the imbalance problem. From there, we propose a solution comprising integral modules: Adaptive Feature Weighting (AFW) and Adaptive Modality Weighting (AMW). The former enhances intra-modal representations for feature-level balancing, while the latter optimizes inter-modal learning weights with the balancing at modality level. Furthermore, we introduce a refined disparity ratio to optimize training, offering a straightforward yet effective measure to evaluate the model's overall discrepancy when handling multiple modalities simultaneously. Extensive experiments on the IEMOCAP, MELD, and CMU-MOSEI datasets validate its effectiveness, showcasing SOTA performance. In the future, we anticipate enhancing the efficiency of the framework and maximizing the utilization of emotional cues. 
 
 Appendix: https://pytorch.org/https://comet.mlhttps://www.sbert.net/https://github.com/zengqunzhao/MA-Net 


Decision: Test-Time Adaptation for Multimodal Sentiment Analysis

Reviews: review: paper_summary: paper_summary: The paper introduces Ada2I, a novel framework specifically designed to tackle the challenge of modality imbalance in MERC. The authors address this challenge by implementing Adaptive Feature Weighting (AFW) for feature-level balance and Adaptive Modality Weighting (AMW) for modality-level balancing. Additionally, they introduce a refined disparity ratio to optimize the training process. The framework's effectiveness is demonstrated through extensive experiments on three benchmark datasets: IEMOCAP, MELD, and CMU-MOSEI.
strength_weakness: strength_weakness: - The paper designs two integral modules, AFW and AMW, which operate in tandem to ensure balanced learning across modalities.
- The paper validates the effectiveness of Ada2I against several baselines, showcasing improved performance on benchmark 
 datasets.
- The paper is well-written, with a clear presentation of methodology.
limitations: limitations: - Lack of novelty. The proposed improvements mainly integrate existing methods without demonstrating distinctiveness and their contribution is limited.
- The motivation is rather lacking for the modal imbalance problem. Some of the work maps other modalities to text space to solve the problem. Then the significance of the modal imbalance solution should be more fully explained in the text. Also, do the methods chosen in the experiments take into account the modal imbalance problem, and are there methods that do not take the above into account and still perform better?
- The ablation experiments need to be supplemented. Some of the parameters in the methods lack relevant discussion. Some of the hyperparameters may have a more significant effect on the final results.
paper_summary: paper_summary: This paper introduces a novel framework, Ada2I, designed to address the challenges of modality imbalances in multimodal emotion recognition in conversations (ERC). The authors propose a two-module system: Adaptive Feature Weighting (AFW) for feature-level balancing and Adaptive Modality Weighting (AMW) for modality-level balancing. They also introduce a refined disparity ratio as a training optimization strategy. The Ada2I framework is evaluated on three benchmark datasets: IEMOCAP, MELD, and CMU-MOSEI, demonstrating state-of-the-art performance and effective handling of modality imbalances.
strength_weakness: strength_weakness: (1) The paper tackles a significant problem in multimodal ERC, which is the imbalance in contributions from different modalities and proposes a comprehensive solution with the Ada2I framework.
(2) The introduction of the Adaptive Feature Weighting and Adaptive Modality Weighting modules provides a systematic approach to balancing feature-level and modality-level representations, respectively.
(3) The refined disparity ratio as a training optimization strategy is a novel contribution that offers a clear measure to evaluate the model's learning process across multiple modalities.
(4) The paper provides extensive experimental results, demonstrating the effectiveness of Ada2I against several state-of-the-art baselines on three well-known benchmark datasets.
(5) The ablation studies and visualizations of the discrepancy ratio and modality-wise weights provide insightful analysis of the model's performance and the impact of the proposed modules.
limitations: limitations: (1) The paper could benefit from a more detailed explanation of the tensor-ring decomposition method and its integration into the model, as this is a complex and less commonly used technique.
(2) While the paper mentions the use of pretrained models for feature extraction, it lacks specifics on how these models were chosen and whether their pretrained weights were fine-tuned for the task.
(3) The paper could provide more insight into the selection of hyperparameters and whether there were any sensitivity analyses conducted to understand their impact on model performance.
(4) Although the paper presents improvements over baseline models, it is not clear how Ada2I compares to models that do not focus on modality imbalance but have different architectural designs or fusion techniques.
paper_summary: paper_summary: This work presents a new approach for multimodal conversational emotion recognition (MER). It involves two attempts from both feature view and modality view. Meanwhile, an enhancement for training is also proposed.
strength_weakness: strength_weakness: 1. The work aims at improving MER task from an overall view, and constructs learnable weights from both feature and modality prospects.
2. The end-to-end framework of Ada2I is complete.
limitations: limitations: 1. The illustration of $\mathcal{G}$ is included in an cited article [42] implicitly, which is a basic component for the whole process. And it seems that $\mathcal{G}_{Q, K}^{(m)}$ is obtained from a linear transform via Fig.2. It's suggested to add a more detailed illustration about that. Besides, I'm confused about the relationship between $(d_a, d_v)$ in Line 320 and $(r_s, r_w)$ in Line 330. Does $(s, w) \in \{1, 2, 3\}$ means different modalities from $(t, a, v)$?
2. Only impacts of AFW and AMW modules are considered in ablation studies, which lack the analyze of training optimization strategy.
paper_summary: paper_summary: This paper aims to address the issue of modality imbalance in multimodal learning, commonly known as modality collapse. To tackle this problem, the authors employ Adaptive Feature Weighting (AFW) to achieve feature-level balance and Adaptive Modality Weighting (AMW) for modality-level balancing. In addition, the authors optimize the training optimization strategy.  Experiments were conducted on a wide range of datasets.
strength_weakness: strength_weakness: 1. The paper introduces a new framework called Ada2I, focusing on addressing modality imbalance in MERC. The design of AFW and AMW allows the model to balance at both the feature and modality levels.
2. Extensive experiments on three different benchmark datasets in the paper validate the effectiveness of the model.
3. The paper provides a novel training optimization strategy that helps to better understand and control the modality imbalance problem during the model training process.
limitations: limitations: 1. The proposed improvements, while addressing the problem from two perspectives, seem to lack clear innovation as they mainly integrate existing methods without showcasing distinctiveness. As a result, the contribution appears to be limited.

2. The main motivation is the modality imbalance problem in multimodal joint training. What would be the outcome if each modality underwent separate training, followed by integration of their results? Is the lack of significant improvement in multimodal performance due to individual modalities' inadequate performance or incomplete learning stemming from joint training? Many MLLMs involve mapping other modalities into the textual space for problem-solving, such as LLava and mini-GPT4. In such a context, what is the significance of the modality imbalance issue? What are its practical implications and application scenarios?

3. While the paper compares the performance of Ada2I with state-of-the-art models on the IEMOCA and MELD datasets, a direct performance comparison with SOTA is not given in Table 3. Although Figure 3 provides visualisation of the performance gap, these results alone do not convincingly demonstrate that solving the modal imbalance problem significantly improves performance. This shortcoming is also evident in Table 4, where the performance improvement of Ada2I on the MOSEI dataset is limited even after accounting for the modal imbalance problem.


4. The paper lacks certain experimental details, which may hinder result reproducibility. For example, the division of the IEMOCAP dataset is not explicitly outlined. Was it divided based on different sessions?

5. The explanations regarding how features and modality weights impact emotion recognition results in the ablation study are insufficient.

6. The parameter $\beta$ in Equation 7 seems significant, however, its influence on the final performance remains unclear. The ablation of some other hyperparameters is also lacking.

7. There are some issues in the writing and presentation. The motivation outlined in the introduction is not sufficiently clear, and discrepancies in Figure 1 undermine the credibility of the experiments. It's advisable for the author to verify the accuracy of the annotations. Furthermore, Section 2.1 of the related work enumerates numerous Graph-based MERC methods, which seem tangential to the approach proposed in this paper. How does the MERC scenario differ from other tasks? Is it inherently unique? Why does it necessitate specialized design, and what sets apart the method proposed in this paper from previous works? These aspects are not adequately addressed in the related work section. The methodology section suffers from symbol confusion, exemplified by the repeated use of 'M' in line 277 without clear distinction in meaning.