DQ-Former: Querying Transformer with Dynamic Modality Priority for Cognitive-aligned Multimodal Emotion Recognition in Conversation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 17:40:32

Abstract: Multimodal Emotion Recognition in Conversations aims to understand the human emotion of each utterance in a conversation from different types of data, such as speech and text. 
Previous works mainly focus on either complex unimodal feature extraction or sophisticated fusion techniques as general multimodal classification tasks do.
However, they ignore the process of human perception, neglecting various levels of emotional features within each modality and disregarding the unique contributions of different modalities for emotion recognition. 
To address these issues, we propose a more cognitive-aligned multimodal fusion framework, namely DQ-Former. 
Specifically, DQ-Former utilizes a small set of learnable query tokens to collate and condense various granularities of emotion cues embedded at different layers of pre-trained unimodal models. Subsequently, it integrates these emotional features from different modalities with dynamic modality priorities at each intermediate fusion layer. This process enables explicit and effective fusion of different levels of information from diverse modalities.
Extensive experiments on MELD and IEMOCAP datasets validate the effectiveness of DQ-Former. Our results show that the proposed method achieves a robust and interpretable multimodal representation for emotion recognition.

Main paper: 
 related work 2.1 multimodal fusion works: 
 The self-attention mechanism introduced in the transformer architecture [32] offers a versatile approach for modeling signals across diverse modalities. However, recent advanced models tend to specialize in particular modalities and are tailored for unimodal tasks [1, 3, 4, 20] . Hence, the dominant framework of previous MERC systems adopts a two-phase late-fusion pipeline: first extracting feature representations of each modality separately simply utilizing the final representations from pre-trained unimodal models [28, 41] ; and then employing sophisticated fusion techniques to integrate them [22, 29, 37, 38] . A typical late-fusion framework is MulT [31] , which uses directional pairwise cross-attention to align different modalities. MISA [8] projects each modality into a modality-specific subspace to capture their unique features and into a modality-invariant subspace to capture their shared characteristics. Sun et al. [29] employ unimodal transformer modules to model the representations of each modality and a multimodal transformer module to fuse all the modalities. These works fail to consider fine-grained modality-specific attributes.In addition to the two-phase late-fusion approach, there are efforts towards developing end-to-end early-fusion or middle-fusion MERC systems. One prevalent strategy involves directly combining different modalities into unified inputs for the transformer model [17] . This operation often ignores the heterogeneity among different modalities. On the other hand, Xu et al. [36] introduce a bridge-layer between the top layers of uni-modal encoders and each layer of the cross-modal encoder, so as to get a good visionlanguage representation with diverse levels of information. In addition, Nagrani et al. [25] propose a middle-fusion strategy MBT, which directs information exchange between modalities through bottleneck latent spaces, and achieves impressive performance on audio-visual benchmarks. Inspired by MBT, Wu et al. [35] introduce the multimodal recurrent intermediate-layer aggregation model for MERC task. Despite these advancements, all these works typically treat MERC as general multimodal tasks do. They lack consideration of different modality priorities for emotion recognition before fusion.Recent studies have shown that different modalities contribute uniquely to emotion recognition [7] . Existing works either rely on manually selected dominant modalities based on prior knowledge, or fail to fully exploit the dominant modalities and filter out the misleading signal. For example, Zhang et al. [39] propose a language-guided multimodal fusion strategy as they believe the language modality usually stands out as the dominant one among all modalities. Wang et al. [33] utilize modality-specific softmax functions to compute the attention weights in a single-stream Transformer. Hong et al. [9] propose a model-agnostic auxiliary network to select the fusion modality. Wang et al. [34] propose a new architecture that uses dynamic modality gating to determine a primary modality and hierarchically incorporates other auxiliary modalities. Although these methods consider the varying contributions of different modalities, they often treat multimodal inputs generically, ignoring the distinctive properties of each modality.Different from all these works, we propose a new middle-fusion framework. It not only considers various levels of emotion cues across multiple intermediate layers, but also fuses them with a dynamic modality priority. 
proposed framework: 
 In this section, we introduce a more cognitive-aligned multimodal middle-fusion framework, namely DQ-Former. We first state the problem. Following that, we provide details of the unimodal encoder, the proposed multimodal fusion framework, and dynamic modality priority learning. The overall architecture of DQ-Former is depicted in Figure 2 .Let 𝐶 = {𝐶 𝑖 } 𝑁 𝑖=1 denote a conversational dataset, where 𝑁 is the total number of conversations. Each conversation 𝐶 𝑖 consists of 𝑁 (𝑖 ) utterances. Let 𝑢 𝑖 𝑗 represents the 𝑗-th utterance in the 𝑖-th conversation 𝐶 𝑖 , where 𝑗 ∈ [0, 𝑁 (𝑖 ) emotion label of utterance 𝑢 𝑖 𝑗 , which belongs to 𝑘 emotion categories. This task aims to identify the emotion label 𝑦 𝑖 𝑗 of each utterance 𝑢 𝑖 𝑗 from multiple types of signals.Regarding the multimodal input, each utterance consists of text transcripts (𝑡) and audio (𝑎). As highlighted by Mao et al. [24] , textual modality performs much better in context-dependent settings, while acoustic modality always achieve better performance in context-free settings. Consequently, we structure our raw transcripts by incorporating dialogue history, as in "<cls> 𝑛 preceding (speaker: utterance) pairs <sep> the target (speaker: utterance)", while maintaining audio inputs as context-free.Formally, we adopt 𝑚 to represent a particular input modality, where 𝑚 ∈ {𝑡, 𝑎} can be either 𝑡 for textual modality or 𝑎 for acoustic modality. Let's denote the input sequence as 𝑋 𝑚 = {𝑥 1 , 𝑥 2 , . . . , 𝑥 𝑁 𝑚 }, where 𝑁 𝑚 signifies the length of the input sequence for modality 𝑚.To obtain modality-specific attributes and model intra-modal interactions, we encode the raw inputs from different modalities with separate unimodal transformer encoders. These unimodal encoders are initialized with pre-trained models, allowing them to inherit knowledge from pre-training stages. The textual encoder is initialized by BERT model [4] , while acoustic encoder is initialized using WavLM model [3] . The unimodal intermediate representations can be expressed as follows:EQUATIONHere, E is the vanilla Transformer encoder layer, comprising a multi-head self-attention mechanism and a feed-forward network [32] , depicted as the left portion of Figure 2 (b). The operator M signifies a linear projection that maps each token to R 𝑑 . 𝑯 0 𝑚 denotes the initial token embedding representations, while 𝑯 𝑖 𝑚 represents the hidden representation at the 𝑖-th encoder layer for modality 𝑚.Cognitive-related studies indicate that emotional expressions are better identified through specific attributes rather than broad semantic aspects [13] . Meanwhile, recent research on modality-specific models suggests that rich hierarchical information exists across different layers, with semantic features in higher layers and more modality-specific attributes in intermediate layers [3, 5] . Hence, we opt for a middle-fusion approach instead of traditional late-fusion methods to capture diverse levels of emotional information.Inspired by Nagrani et al. [25] and Li et al. [15] , we adopt a set of learnable queries to gather and condense emotion-relevant attributes from each modality within intermediate layers. 2 (b) provides a detailed illustration of the multimodal fusion encoder layer. Specifically, we randomly initialize the query tokens 𝑋 𝑞 = {𝑞 1 , 𝑞 2 , . . . , 𝑞 𝑙 }. And we intentionally set the length 𝑙 to be much shorter than that of unimodal inputs, which aims to condense information and mitigate computational complexity. These query tokens serve as the input of the multimodal fusion encoder. They firstly interact with each other through self-attention like unimodal encoders do:EQUATIONSubsequently, we employ cross-modal attention [31] to project information from various modalities into a unified query space, resembling a shared multimodal space. This approach aims to maximize synergy and minimize heterogeneity gaps among diverse modalities. Assume that the query representations Ĥ𝑖 𝑞 is the 𝑄, and the unimodal hidden representations 𝑯 𝑖 𝑚 is the 𝐾 and 𝑉 . The equation defining this process is:EQUATIONHere, C 𝑖 𝑚→𝑞 represents the cross-modal attention. The detailed calculation of C 𝑚→𝑞 is as follows:EQUATIONwhere LN denotes LayerNorm, and F is a FFN network. Finally, we aggregate 𝑯 𝑖 𝑞𝑡 and 𝑯 𝑖 𝑞𝑎 to obtain 𝑯 𝑖 𝑞 through dynamic modality priority learning, which will be detailed in Session 3.4. After multiple layers of inter-modal interaction and fusion, the final representation of query tokens can be regarded as the multimodal representation for emotion recognition.Intuitively, not all modalities play an equal role in emotion recognition. The goal of the instance-level dynamic modality priority learning is to mask unnecessary modalities before fusion and dynamically control the fusion degree of each modality.To enable dynamic modality fusion, we add a confidence estimation network (ConNet). The ConNet takes hidden states of the unimodal encoder at the 𝑖-th layer as inputs and predicts a single scalar between 0 and 1. The confidence calculation is as follows.EQUATIONEQUATIONEQUATIONwhere 𝑾 and 𝒃 are trainable parameters, 𝜎 (•) is the sigmoid function.For each instance, the ConNet outputs a high confidence score if the attribute of modality 𝑚 at 𝑖-th layer is a crucial emotion cue. Conversely, if this attribute is unimportant, it outputs a low confidence score. When the confidence score 𝑐 𝑖 𝑚 is close to 0, we can assume that the modality is unnecessary or even a noisy signal that may mislead the final emotion recognition result. Therefore, we set a mask threshold 𝜏 (i.e., 𝜏 = 0.1) to filter out unnecessary modalities. After applying the masking operation of misleading modalities, 𝑐 𝑖 𝑚 is updated as follows:EQUATIONThis ConNet is inserted in every fusion layer. Finally, the aggregation function of 𝑯 𝑖 𝑞 is as follows:𝑯 𝑖 𝑞 = ∑︁ 𝑐 𝑖 𝑚 • 𝑯 𝑖 𝑞𝑚 (9)Our DQ-Former framework comprises two key components: unimodal encoders facilitating intra-modal interaction, and a multimodal fusion encoder enabling inter-modal interaction. We denote the representations of the final layer as unimodal representations and multimodal representations:EQUATIONEQUATIONEQUATIONHere, 𝑎, 𝑡, and 𝑓 correspond to fusion, textual and acoustic, respectively. These representations are utilized to obtain the final unimodal and fusion predictions:EQUATIONwhere 𝑊 𝛼 ∈ R 𝑘 ×𝑑 , 𝒃 𝛼 ∈ R 𝑘 , 𝛼 ∈ {𝑎, 𝑡, 𝑓 } and 𝑘 represents the number of emotion labels.Our training object involves two parts: cross-entropy loss for unimodal predictions and multimodal predictions. The training optimization objective is expressed as:EQUATIONwhere 𝜆 𝑓 , 𝜆 𝑡 and 𝜆 𝑎 are the weighting factors. During inference, the fusion prediction 𝑦 𝑓 is chosen. 
experimental setting 4.1 experimental dataset: 
 To evaluate the efficacy of our model, we conduct experiments on two widely recognized MERC benchmark datasets: IEMOCAP [2] and MELD [27] . The IEMOCAP dataset is a multimodal emotion recognition dataset, which contains 151 conversations and 10,039 utterances. Each utterance is annotated with predefined emotion labels such as anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and others. Due to the data imbalance issue, we only focus on the first six main emotion categories, excluding the remaining labels. The MELD dataset is a multi-party conversational dataset collected from the Friends TV shows. This dataset contains 13,708 utterances and 1,433 conversations. Each utterance is annotated with one of the following seven labels: neutral, surprise, fear, sad, joy, disgust, and angry. Since the IEMOCAP dataset does not provide an official data split, we split all the conversations into training, testing, and validation sets according to a ratio of 8:1:1, ensuring that each dialogue appears in only one of the data divisions to avoid contextual content leakage. For MELD datasets, we follow the default data splitting. A concise summary of statistical values for all datasets is provided in Table 1 .We train two variants of DQ-Former, including base and large version. The DQ-Former base comprises 12 unimodal encoder layers, 4 multimodal fusion layers (fusion starts at the 9-th layer of the unimodal encoder), 8 attention heads, and hidden dimensions of 768. In the case of DQ-Former large, there are 24 unimodal encoder layers, and a hidden dimension of 1024. The textual and acoustic modality encoders are initialized using BERT [4] and WavLM [3] pretrained models, respectively. As for input of the multimodal fusion transformer, we set 16 learnable query tokens, which are randomly initialized. For the textual modality, we adapt a dialogue history window size of 5 utterances, restricting the input length to 512 tokens. For the acoustic modality, we set a limit of 10 seconds for the duration of audio input. Due to the imbalance across various emotions, we also perform up-sampling on minority class data while training. Besides, we set the loss weights 𝜆 𝑓 , 𝜆 𝑡 , and 𝜆 𝑎 to 1.All experiments are conducted using a single Tesla V100 GPU. To optimize trainable parameters, we use the AdamW optimization scheme [21] . The learning rate is set to 1e-5, and it follows a linear warm-up of 1% of total steps. We set a batch size of 8. The training process spans 20 epochs. To mitigate overfitting, we incorporate an early stopping mechanism with a patience of 7 evaluation steps. Subsequently, we employ the checkpoints that performed best on the validation set for testing. We run each experiment three times and report the average result.Consistent with previous studies, we utilize the Weighted Average F1 score (WAF1) as the main metric, considering the natural imbalance across various emotions. Weighted Average Accuracy (WAA) serves as the secondary evaluation metric.To validate the performance of DQ-Former, we conduct a comprehensive comparison with several state-of-the-art models. The comparative models are as follows: MMGCN [11] , DialogTRM [24] , MM-DFN [10] , EmoCaps [19] , FacialMMT [41] , GA2MIF [16] , Joyful [14] . All these models leverage textual, acoustic, and visual modalities as inputs. Additionally, we include CMCF-SRNet [40] in our baselines, as it is the advanced two-modality model incorporating textual and acoustic inputs.Since the choice of unimodal features seriously affects multimodal results, we also reproduce the following models under the same experimental settings as ours: MulT [31] (a conventional late-fusion framework for unaligned inputs), MBT [25] (a novel middle-fusion framework), and SA (it combines different modalities into unified inputs for the single-stream transformer). The unimodal encoders in these three models are also tuned. 
results and discussion: 
 To validate the efficacy of our approach, we first conduct comparative experiments with currently advanced systems. Following this, we perform ablation experiments. Finally, we discuss the interpretability of our model, examining how DQ-Former can extract necessary emotional cues across various layers within each modality and explicitly discern the relative importance of different modalities.We compare our proposed method with state-of-the-art approaches on the MELD and IEMOCAP datasets, and present the results in Table 2 . The results show that DQ-Former demonstrates competitive performance on both two datasets, particularly in the scenario involving only textual and acoustic modalities. Specifically, on the IEMOCAP dataset, DQ-Former (base) demonstrates the most promising performance, achieving a significant relative improvement of 1.43% WF1 compared to the second-best result obtained by EmoCaps. Similarly, on the MELD dataset, DQ-Former (large) outperforms FacialMMT, while DQ-Former (base) also achieves competitive results.Additionally, DQ-Former showcases outstanding performance in recognizing certain minority emotion categories. For example, both DQ-Former (base) and DQ-Former (large) demonstrate excellent emotion recognition capabilities in the fear and disgust categories on the MELD dataset. This highlights the effectiveness of masking misleading modalities to achieve more robust performance.It's worth noting that, under identical experimental conditions, the new middle-fusion framework DQ-Former consistently outperforms MulT and MBT. Compared to the typical late-fusion MERC system MulT, DQ-Former (base) achieves substantial improvements by 2.91% and 6.51% WF1 on the MELD and IEMOCAP datasets, respectively. Furthermore, integrating DMP results in significant performance enhancement compared to MBT, another middle-fusion method. This underscores the critical role of leveraging a dynamic modality priority fusion strategy to integrate diverse granularities of emotion cues from different modalities across various intermediate layers.Moreover, it's observed that DQ-Former (large) does not always outperform DQ-Former (base). This phenomenon may be attributed to the limited existing data combined with the high parameter count of the model, potentially leading to overfitting issues.We conduct experiments to shed light on the following reasonable doubts:• Does middle-fusion strategy outperform other fusion strategies? • Can dynamic modality priority learning (DMP) decide the relative importance between textual and acoustic modalities and can mask threshold effectively filter noising modalities? • How does DQ-Former perform under modality missing circumstance?In addition to validating on the IEMOCAP 6-way condition, we introduce the IEMOCAP 4-way condition, which concentrates solely on the four primary emotions: anger, happiness, sadness, and neutral. Ablation results are presented in Table 3 .Strategy. DQ-Former employs a middlefusion strategy to effectively leverage various levels of modalityspecific emotional features embedded in intermediate layers of pre-trained unimodal encoders. To delve deeper into the impact of these fusion strategies, we introduce two variants of DQ-Former: DQ-Former (late) which uses the final representation to fuse, and DQ-Former (early) which starts fusion at the first layer.Results in Table 3 consistently demonstrate the superiority of middle-fusion over both late-fusion and early-fusion approaches. Notably, the late-fusion strategy exhibits a clear performance decline. Specifically, DQ-Former (late) shows a decrease of 1.39%, 2.45% and 4.52% WF1 on MELD, IEMOCAP 6-way and IEMOCAP 4way, respectively. In contrast, DQ-Former (early) exhibits relatively less performance degradation. This emphasizes the importance of adaptively considering the various levels of emotional information across different layers within each modality. Moreover, it aligns with the findings of cognitive-related research, indicating that human emotions are more easily recognized through modality-specific attributes.To further investigate the impact of the DMP module in DQ-Former, we conduct a comprehensive ablation analysis. Specifically, we vary the mask threshold to 0.2 (𝜏=0.2) and 0 (w/o 𝜏), and also remove the DMP module entirely (w/o DMP). From the results, we observe:Firstly, the removal of DMP resultes in a significant performance decrease, with a drop of 1.7%, 3.07% and 1.37% WF1 on MELD, IEMO-CAP 6-way and IEMOCAP 4-way, respectively. This underscores the critical role of DMP in enhancing effective multimodal fusion. Secondly, masking misleading modalities effectively enhances the robustness of multimodal emotion recognition. DQ-Former (w/o 𝜏) experiences a performance decrease of 0.54% WF1 on IEMOCAP 6-way, 0.87% WF1 on IEMOCAP 4-way and 1.21% WF1 on MELD. Moreover, it is essential to carefully select the mask threshold. Setting 𝜏 to 0.2 leads to a performance decline on both two datasets. This decline may be attributed to the high mask threshold, which could potentially filter out useful signals, treating them as noise.To further investigate the role of each modality in MERC, we fine-tune the unimodal encoders on respective modality data, referred to as WavLM-FT and BERT-FT. Additionally, we assess the robustness of DQ-Former (base) under modality-missing scenarios. Specifically, we simulate the absence of a modality by masking it (i.e., setting the missing modality inputs to 0). From the results in Table 4 we find:Firstly, the multimodal fusion results of DQ-Former consistently outperform the unimodal setups across both the MELD and IEMO-CAP datasets, demonstrating its capability in leveraging complementary information from diverse modalities.Moreover, under modality-missing circumstances, all three models experience notable performance degradation, particularly evident when the textual modality is missing. For example, SA suffers a performance drop of over 50% when text inputs are masked, indicating a phenomenon of modality collapse during training. In contrast, DQ-Former demonstrates superior robustness to modality-missing, which maintains comparable performance to fine-tuned unimodal encoders. It is worth noting that if a multimodal system overly depends on the dominant modality, it will become less robust in scenarios where one modality is missing.In addition, not all datasets are suitable for MERC. On the MELD dataset, BERT-FT achieves 60.94% WF1, while WavLM-FT reaches 31.26%. Interestingly, all three models exhibit a slight performance To further investigate the priority of modalities across the layers, we examine the distribution of confidence scores across fusion layers of the DQ-Former (early). The result is depicted in Figure 3 .The result reveals several discernible patterns: 1) The textual modality predominantly asserts its influence in the lower layers, with confidence scores peaking between 0.8 and 0.9. 2) In contrast, the acoustic modality takes precedence in the middle layers, with confidence scores close to 1. 3) As we ascend to the higher layers, the confidence score distribution for both modalities becomes more widespread, indicating a delicate balance with a slight inclination towards the textual modality. This underscores that our model adaptively selects key acoustic attributes and textual semantic features for emotion recognition, which is aligned with human perception processes.The Modalities Priority of Different Instances. As introduced in section 3.4, the goal of DMP is to selectively mask misleading modalities before fusion, while dynamically managing modality fusion priorities for every certain instance. To further study the effectiveness of DMP, we examine the distribution of acoustic confidence scores across diverse instances.We select two distinct categories of samples from IEMOCAP, focusing on instances correctly classified by DQ-Former: 1) Text-Dominated: These samples are correctly classified by the textual modality but are misclassified by the acoustic modality. 2) Audio-Dominated: These samples are correctly classified by the acoustic modality but are misclassified by the textual modality. We analyze the confidence distribution in the penultimate layer of the multimodal encoder, considering that top layers primarily contain semantic and content-level information, with a trade-off between different modalities. Figure 4 presents the acoustic confidence distribution for both categories. Remarkably, for Audio-Dominated samples, confidence mainly exceeds 0.1, whereas for Text-Dominated ones, it peaks between 0 and 0.1, gradually decreasing with higher confidence. This trend mirrors human cognitive tendencies, indicating higher acoustic confidence in Audio-Dominated instances and lower acoustic confidence in Text-Dominated ones. Refer to Session 5.4 for additional practical examples.Additionally, we observe that when the acoustic confidence score is below the mask threshold (𝜏 = 0.1), the number of Text-Dominated samples significantly exceeds other confidence intervals, while the count of Audio-Dominated samples is notably lower in this interval compared to others. This observation underscores the necessity of filtering out misleading modalities.To better understand the main contributions of our work, we present four examples in Figure 5 . These examples are drawn from Audio-Dominated and Text-Dominated categories in the IEMOCAP dataset. The results delineate distinctive characteristics between Audio-Dominated and Text-Dominated samples. In the former, overt vocal cues are prominently apparent. For instance, in case 1, a waveform graph is characterized by heightened amplitude, indicative of palpable anger expressed by the speaker. Conversely, case 2 showcases subdued amplitude and specific inflection, reflective of frustrated emotions. In contrast, Text-Dominated samples usually contain emotional vocabulary that clearly conveys the speaker's emotions. These observations underscore the efficacy of the DMP module in DQ-Former, which adaptively captures and integrates necessary emotional cues from diverse modalities. 
conclusion: 
 In this paper, we introduce DQ-Former, a novel middle-fusion multimodal framework designed to enhance the modeling of diverse emotion cues for a robust MERC system. DQ-Former leverages various granularities of emotional information embedded in intermediate layers within each modality, and gathers them by learning the confidence levels associated with emotion recognition for complementary insights. Experiments on the IEMOCAP and MELD datasets demonstrate the effectiveness of DQ-Former in improving emotion recognition performance. Furthermore, our in-depth analysis demonstrates the ability of DQ-Former to extract key emotional cues within each modality and autonomously adjust the priority of each modality under different scenarios. This closely aligns with human cognitive processes and achieves robust and interpretable emotion recognition results. These findings offer valuable insights for leveraging advanced modality-specific pre-trained models from a cognitive perspective, thereby informing and potentially improving future research endeavors in multimodal emotion recognition.  
: 
 Figure 1 : Two examples from IEMOCAP. Not all modalities contribute equally to emotion recognition. In case (a), emotions can be readily inferred from the text, whereas weak pitch in the voice, conversely, tends to introduce disruptive noise. In case (b), a strongly intense tone of voice can clearly express the speaker's anger, whereas the long context in textual modality can be challenging to understand the emotion.cues [24] . The efficacy of MERC models heavily hinges on the quality of multimodal representations.Previous works mainly treat MERC as a general multimodal classification task. The dominant framework involves unimodal feature extraction, information interaction, and multimodal fusion. Some investigations underscore the pivotal role of high-quality unimodal features, emphasizing their direct impact on multimodal representations [28, 41] . Certain works delve into modeling intermodal or intra-modal interactions [23, 38] . Additionally, researchers develop sophisticated strategies for multimodal fusion [8, 31, 37] .However, such generalized frameworks tends to ignore the intricacies of human emotion expression and perception. Humans express emotions not only through semantics, but also through nuanced modality-specific attributes, such as pitch, loudness, and tempo in their voice [6, 13] . Existing late-fusion works, which rely on the final representations of unimodal models for multimodal fusion, fail to leverage the unique attributes of each modality [35] . Besides, the contribution of each modality vary under different circumstances for emotion recognition [7] . As illustrated in Figure 1 , while text emerges as the dominant modality for clearly conveying anger in one instance (case a), the loud voice signal could serve as the primary indicator of emotion in another (case b). Disregarding the relative importance of different modalities and fine-grained modality-specific attributes during multimodal fusion may result in suboptimal performance of MERC systems [30] .Recent studies indicate that pre-trained unimodal models contain rich hierarchical information across various layers [12, 18] . For example, interpretability studies in speech models [3, 18, 26] show that paralinguistic and linguistic properties are encoded in different layers. This phenomenon is mirrored in text models as well [12] . Generally, the final representations of unimodal models are mainly associated with semantic-level information, while various levels of modality-specific information are distributed across intermediate layers [3, 12] . These findings indicate the potential for the mid-layer fusion strategy to capture diverse levels of emotional cues within advanced modality-specific pre-trained models [35] .Motivated by the above observations, we propose a more cognitivealigned middle-fusion multimodal framework, namely DQ-Former, to enhance the modeling of emotional cues for a robust MERC system. Specifically, we first encode each modality using modalityspecific pre-trained models. Then we employ a small set of learnable queries to collate and condense emotion cues embedded in intermediate layers of each unimodal encoder through layer-wise cross-modal attention. Finally, we aggregate various granularities of emotion cues based on their confidence in emotion recognition. In this sense, the cognitive-aligned relationships between multiple modalities are explicitly reasoned, achieving a complementary and robust multimodal representation for emotion recognition.Our main contributions can be summarized as follows:• We present DQ-Former, a cognitive-aligned middle-fusion framework that leverages various granularities of emotion cues from each modality, since human emotions are conveyed not only through semantics but also through specific attributes.• We devise a novel dynamic modality priority learning module to aggregate multiple modalities, which explicitly manages the contribution of each modality for every instance. • DQ-Former achieves impressive performance on MELD and IEMOCAP datasets. Our analysis reveals that the proposed method establishes a robust and interpretable multimodal representation for emotion recognition. 
 
 Appendix:  


Decision: Test-Time Adaptation for Multimodal Sentiment Analysis

Reviews: review: paper_summary: paper_summary: The paper proposed a novel middle-fusion mechanism for multimodal attention
strength_weakness: strength_weakness: - The paper has very sound and robust experiments and ablation studies
- The proposed technique exhibits competitive performance on the MELD and IEMOCAP datasets
- The paper is very well-written and easy to follow. Clear motivation is provided for each of the decisions made, and has intuitive explanations for how to arrive at each of the ideas
limitations: limitations: - The proposed fusion technique only works with two modalities
- There is not enough clarity on the novelty of the proposed mechanism - i.e. the authors should aim to provide more information to better differentiate the proposed technique from existing mlddle and late fusion attention mechanisms.
paper_summary: paper_summary: The paper proposes DQ-Former, a multimodal fusion framework to perform Emotion Recogntion in Conversations (ERC). DQ-Former utilizes a middle-fusion approach for multimodal feature fusion incorporated with dynamic modality priority learning aimed to effectively fuse information across modalities. Audio and Text modalities are considered and experiments are conducted on the MELD and IEMOCAP datasets.

The effectiveness of the proposed middle-fusion, dynamic priority modality learning, and robustness of the proposed framework to missing modalities are studied.
strength_weakness: strength_weakness: The motivation/need for a middle-fusion technique to capture cross-modal information is well-grounded. The need to adaptively manage relative importance of modalities (towards tasks such as ERC) is well-explained. Identification that most existing methods treat MERC as general multimodal task is of importance. 

Ablation studies are provided to validate each component in the proposed method including scenario where modalities are missing.
limitations: limitations: While identifying the need to account for dynamic modality priority for multimodal ERC models, the proposed method utilizes only textual and acoustic modalities from MELD and IEMOCAP datasets while 3 modalities (visual, text and acoustic) are available. The reason behind omitting a modality is requested from the authors. Experiments with all 3 modalities would help demonstrate the significance of the proposed work. 

Section 3.4 discusses "Dynamic Modality Priority Learning". It is mentioned that for each instance (utterance), ConNet provides confidence estimates for each modality (m) at $i^\text{th}$-layer. An ablation study reporting the frequency of a modality (text or acoustic in this case) being dominant across utterances for MELD and IEMOCAP dataset can help identify patterns within the datasets if there is a dominating modality across most of the utterances (visual modality can be included in this study as well). This would help demonstrate the significance of the priority learning approach and its generalization ability across other multimodal tasks.
paper_summary: paper_summary: This paper proposes a new multimodal fusion framework for emotional recognition. Unlike prior works, it incorporates various levels of emotional features within each modality and considers the unique contributions of different modalities. Concretely, it suggests a new middle-fusion encoder with cross-modal attention and dynamic modality priority learning with threshold. The experimental results demonstrate the effectiveness of proposed methods on IEMOCAP and MELD datasets.
strength_weakness: strength_weakness: [S1. New methodology for middle-fusion] This paper suggests a new middle-fusion framework to both capture the diverse degree of emotional features and reflect the different contribution of each modality. Specifically, it opts for middle-fusion and adopts cross-attention with learnable query. It also dynamically aggregates the modalities at the instance level by utilizing ConNet. 

[S2. Effectiveness] The proposed method improves the emotion recognition performance in two conversational datasets. It also demonstrates robustness in modality-missing scenarios.
limitations: limitations: [W1. Unclear comparison to baselines] This paper does not clarify its contribution compared to baselines of the same intuitions. Specifically, this paper suggests two intuitions 1) incorporating the various levels of emotional features within each modality and 2) considering the different priorities of each modality. However, these intuitions are already introduced in baselines as described in Section 2. Thus, it is necessary to compare this paper and baselines in terms of each intuition, and show why the proposed framework has shown the enhancement. 


[W2. Insufficiently supported analysis] Experimental results are not supportive of the analysis. For example, in section 2.3.2, the distribution of audio-dominated samples seems not appropriate. This is because the DMP should have assigned higher acoustic confidence in the most of the audio-dominated samples. Also, the insufficient experiment in Table 2 does not support the statement that DQ-former shows recognition capability in minor categories (e.g. fear, disgust).


[W3. Weak baselines] Baselines are too weak. For example, in robustness analysis(5.2.3), the baselines are weak because most baselines are published before 2022. In the case of MELD datasets results, there are some models that show better performance, UniMSE[1], DF-ERC[2]. It is necessary to compare these models.


 
[1] Song, X., Huang, L., Xue, H., & Hu, S. (2022). Supervised prototypical contrastive learning for emotion recognition in conversation. arXiv preprint arXiv:2210.08713.
[2] Li, B., Fei, H., Liao, L., Zhao, Y., Teng, C., Chua, T. S., ... & Li, F. (2023, October). Revisiting disentanglement and fusion on modality and context in conversational multimodal emotion recognition. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 5923-5934).
paper_summary: paper_summary: - The author proposes a cognitive-aligned middle-fusion multimodal framework called DQ-Former to enhance the modeling of emotional cues for a robust MERC system. 
- This framework includes a dynamic modality priority learning module to aggregate text and audio inputs, managing the contribution of each modality for every instance. Experiments were conducted on the MELD and IEMOCAP datasets.
strength_weakness: strength_weakness: 1. **Methods**:

- The author opts for a middle-fusion approach instead of traditional late-fusion methods, capturing diverse levels of emotional information by initializing query tokens with a shorter length than unimodal inputs. These tokens serve as inputs to the multimodal fusion encoder, allowing them to interact through self-attention. Cross-modal attention is employed to project information from various modalities.
- Dynamic Modality Priority Learning aims to mask unnecessary modalities before fusion and dynamically control the fusion degree of each modality. 
2. **Experiments**:
- The experiments are extensive, with comparisons to various previous works.
- The ablation study sufficiently demonstrates the effectiveness of DQ-Former and its components. 
3. **Structure and Writing**:
The paper is well-written and easy to follow. The literature review is systematic and comprehensive.
limitations: limitations: **Limitations:**

1. **Methods:**
- The concept of middle-fusion needs clarification, especially in comparison to deep-fusion or layer-wise fusion in multimodal learning. Additional related works should be included to highlight the novelty of the proposed method.

2. **Experiments:**
- DQ-Former only explores text and audio modalities, comparing them to baseline models that use all three modalities, which is not entirely fair. Although audio+text provides strong results in MER, aligning all three modalities is challenging. The author should compare with SOTA baseline models that use only text and audio for fairness, citing and adding results from relevant papers (e.g., papers [1-4]).
- Additionally, for methods using all three modalities, comparison with more advanced SOTA models is needed (e.g., studies [5-8]).
- Table 2 compares the proposed model with baseline models. The author states that only SA, MulT, and MBT are reproduced, while others are from original papers. **However, the reported results do not match those in the original papers. This needs serious review as it affects the paper's reliability**.
- The experimental section lacks significance tests to verify the method's effectiveness further.

3. **Reproducibility:**
- The provided source code facilitates reproducibility.

4. **Typos, Grammar, Style, and Presentation:**
- Some quantities need clearer dimensionality specification for better model flow understanding.

5. **Questions:**
- Does the random creation of query tokens with varying lengths and continuous fusion increase computational complexity? Has the author measured this compared to simpler fusion methods? If so, measurable quantities like complexity, runtime per epoch, and parameter count should be provided.
- In Table 4, DQ-Former and SA results for the MELD dataset improve when audio input is masked. Can the author explain this phenomenon?

**References**

[1] Yoon, Seunghyun, et al. "Speech emotion recognition using multi-hop attention mechanism." ICASSP 2019-2019 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, 2019.

[2] Peng, Zixuan, et al. "Efficient speech emotion recognition using multi-scale cnn and attention." ICASSP 2021-2021 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2021.

[3] Feng, Lin, et al. "Multimodal speech emotion recognition based on multi-scale MFCCs and multi-view attention mechanism." Multimedia Tools and Applications 82.19 (2023): 28917-28935.

[4] Khan, Mustaqeem, et al. "MSER: Multimodal speech emotion recognition using cross-attention with deep fusion." Expert Systems with Applications 245 (2024): 122946.

[5] Hu, Guimin, et al. "UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

[6] Nguyen, Cam Van Thi, et al. "Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

[7] Shi, Tao, and Shao-Lun Huang. "MultiEMO: An attention-based correlation-aware multimodal fusion framework for emotion recognition in conversations." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

[8] Tu, Geng, et al. "Adaptive Graph Learning for Multimodal Conversational Emotion Detection." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.