# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 2 Related Work
[PERSON]'s study [12] focused on the interaction between modalities in multimodal emotion recognition. Text embeddings were generated using a Recurrent Text Encoder (TRE), capturing sentence-level information and sequential characteristics, while speech features were learned through an Audio Recurrent Encoder (ARE). The extracted text and speech vectors were concatenated into a unified vector, which was fed into an RNN-based final network for emotion classification. This approach achieved significant improvements in emotion recognition recall and precision by finely integrating the non-verbal elements of speech with the contextual information from text. [PERSON]'s study [13] proposed a method to enhance multimodal emotion recognition by applying an attention mechanism to a BERT and CNN architecture. The text modality utilized a BERT-based model to extract linguistic features, while the speech modality employed a CNN to analyze acoustic features. An attention mechanism was then used to effectively fuse information from both modalities for emotion classification. [PERSON] [14] introduced a model leveraging multi-scale MFCCs and a multi-view attention mechanism. In the speech modality, rich emotional features were extracted using MFCCs at different scales. For interaction with the text modality, emotional features were comprehensively fused across four aspects: speech self-attention, text self-attention, text-guided speech attention, and speech-guided text attention. Additionally, gender recognition was incorporated as an auxiliary task to enhance learning of gender-specific emotional cues. A combined loss function, integrating softmax cross-entropy loss and center loss, further improved emotion recognition accuracy. [PERSON]'s study [15] proposed the Cross-Modal RoBERTa (CM-RoBERTa) model for utterance-level emotion recognition. This model dynamically captured interactions between speech and text modalities using parallel self-attention and cross-attention mechanisms, effectively learning intra-modal and inter-modal features. The use of mid-level fusion and residual connections enabled modeling of long-term contextual dependencies and modality-specific patterns, achieving superior performance. [PERSON]'s study [16] suggested a multimodal emotion recognition model employing pre-trained transformer frameworks for self-supervised learning of speech and text representations, augmented by motion capture data. Speech features were extracted using wav2 vec 2.0, text features using BERT, and motion capture data added non-verbal behavioral cues. These features were fused at the feature level for emotion classification, demonstrating excellent results. [PERSON]'s study [17] proposed a novel approach for multimodal emotion recognition by leveraging label information. Representative label embeddings were generated for both text and speech modalities, and label-token and label-frame interactions were employed to learn label-enhanced representations for each utterance. A label-based attention fusion module was designed to integrate these enriched text and speech representations, achieving superior classification performance. [PERSON]'s study [18] conducted an empirical analysis of the impact of fusion strategies on the performance of audio and text modalities in multimodal emotion recognition. The study explored various fusion methods, identifying effective approaches while preserving the unique emotional expression characteristics of each modality. To further enhance model performance, a Perspective Loss function was introduced to maintain modality-specific emotional features during fusion. Recent multimodal emotion recognition studies have been progressing toward modeling interactions between modalities more precisely to improve the accuracy of emotion recognition. Initially, research focused on processing each modality independently or combining text and speech modalities through simple concatenation or ensemble techniques.