# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 2 Related Work  
This chapter covers recent trends and key insights in multimodal emotion recognition studies utilizing text and speech. Methods based on single modalities are limited in capturing subtle emotional signals due to the inherent constraints of each modality. In contrast, multimodal approaches that integrate various modalities--such as linguistic, acoustic, or visual information--can significantly improve the accuracy of emotion recognition by leveraging the complementary strengths of different data types. Consequently, recent studies in emotion recognition have actively explored multimodal approaches to more precisely identify emotional states. Among these, this paper focuses on multimodal emotion recognition that combines text and speech. The complementary interaction between linguistic context and non-verbal acoustic signals allows for the recognition of complex emotional states that are difficult to capture with a single modality. [PERSON]'s study [7] proposed a simple concatenation approach for combining text and speech modalities. Acoustic features such as MFCC, energy, and pitch were extracted from speech data, while transcription data from text was processed using the Bag-of-Words method to extract key emotion-relevant keywords. The resulting feature vectors from both modalities were concatenated and fed into a multilayer perceptron (MLP)-based model for emotion classification. This study demonstrated higher accuracy in emotion recognition compared to single-modality approaches, highlighting the complementary relationship between speech and text. [PERSON]'s study [8] converted speech signals into spectrograms, which were processed using CNN to extract acoustic features. For the text modality, word embeddings were generated and fed into a CNN-LSTM architecture to capture sequential information. The final outputs of the text and speech models were combined using an average ensemble method for emotion prediction. This process, which considers both temporal-frequency domain features of speech and contextual information from text, showed that more refined emotion recognition is achievable compared to simple concatenation. [PERSON]'s study [9] proposed an algorithm combining text-based and speech-based emotion recognition to comprehensively understand user states. Text data underwent preprocessing steps such as tokenization, neutral data removal, and label mapping, followed by embedding using KoBERT. Speech data features were extracted using MFCC and analyzed with a CNN-based model. The results from both modalities were combined using a weighted average ensemble method, reflecting the importance of each modality. This integration significantly improved emotion recognition accuracy and provided higher reliability in assessing actual user states. [PERSON]'s study [10] utilized RoBERTa for sentence-level text embeddings to incorporate rich linguistic context and applied MFCC and HuBERT for deep acoustic feature extraction in the speech modality. These feature vectors were combined to construct multimodal representations, and the final emotion class was predicted. The study demonstrated superior performance compared to single-modality approaches by maximizing the interaction between text embeddings that convey diverse contextual meanings and speech embeddings that reflect speaker tone, pitch, and prosody. [PERSON]'s study [11] proposed Early Fusion, Late Fusion, and Hybrid Fusion methods for combining text and speech modalities in multimodal emotion recognition, comparing and analyzing their performance. Text embeddings were extracted using LLaMa2, and acoustic features were obtained using HuBERT. In the Late Fusion method, softmax probability values from modality-specific models were averaged to predict the final emotion. This method achieved over 11% higher accuracy than single-modality approaches, effectively combining information from text and speech to produce excellent results in complex emotion recognition scenarios.