# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 1 Introduction
Similarly, [PERSON]'s study [9] combined text embeddings generated by KoBERT with CNN-based speech analysis, using a weighted average ensemble method to comprehensively understand user states. These multimodal approaches share the common goal of improving learning performance by integrating complementary information provided by text and speech. However, they often face limitations, such as restricted information exchange between modalities or the omission of critical cues, leading to reduced performance improvements. To overcome these challenges, this study proposes the HyFusER (Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention) model. Unlike previous studies that relied on simple concatenation or independent learning structures between modalities, the HyFusER model strengthens interactions between text and speech to effectively integrate information without losing key cues. Specifically, the Dual Cross Modal Attention mechanism focuses on maximizing the exchange of information by reinterpreting text and speech data in each other's contexts. This approach enables the model to fully exploit the complementary characteristics of the two modalities, paving the way for more accurate and reliable emotion recognition. The proposed model in this study begins by independently processing text and speech data through separate transformer encoders for each modality. Subsequently, the DualCross Modal Attention mechanism is employed to strengthen interactions between text and speech. In the first stage, text is set as the primary input (Query), while speech serves as the auxiliary input (Key and Value). This configuration allows text to effectively learn the acoustic characteristics and speech patterns embedded in the speech data. In the second stage, the roles are reversed. speech becomes the primary input (Query), and text serves as the auxiliary input (Key and Value). This enables speech to learn the linguistic context and semantic structure of the text data. This iterative process enhances the interaction between text and speech, facilitating efficient exchange and fusion of modality-specific information. As a result, the model maximizes integrated representation learning. Finally, the learned features of text and speech are combined using an average ensemble method to predict the final emotional state. The proposed HyFusER model leverages the strengths of both text and speech, significantly improving the accuracy and reliability of emotion recognition. It is anticipated to make meaningful contributions to various applications, including emotion-based conversational AI, healthcare, remote education, and emotional labor support. Future research aims to expand the scope by incorporating additional modalities, such as facial images, to enhance the performance of multimodal emotion recognition technologies capable of identifying more complex and diverse emotional states. The main strengths of the proposed HyFusER model are as follows:  
* Integration of Complementary Information: By employing Dual Cross Modal Attention, the model effectively combines information from text and speech, enabling recognition of complex emotional signals that are difficult to capture using a single modality. * High Learning Efficiency: The design allows each modality to learn features within the context of the other, ensuring that critical cues essential for emotion recognition are not missed. * Improved Prediction Accuracy and Reliability: By fully leveraging the strengths of both text and speech, the model significantly enhances the accuracy and reliability of emotion prediction. * Applicability to Various Domains: The model demonstrates high performance in practical applications requiring emotion recognition, including emotion-based conversational AI, healthcare, remote education, and emotional labor support. The structure of this paper is as follows. Section 2 defines and reviews the contents of previous studies related to multimodal emotion recognition.