## 3 Method and Model  
This study follows up on the previous research on the evaluation of emotion in audio by developing a method based on multimodal information fusion and emotion characterization for the emotion analysis and evaluation of audio and video of students' emotional expressions in their second language. First, the text and emotion labels in the IEMOCAP dataset are utilized to synthesize emotionally rich corresponding audio data by an LLM. Subsequently, the synthesized audio data and the audio in the IEMOCAP dataset are used to generate a speaker video that matches the corresponding audio through the speaker video generation model, and that is combined with the student audio and video to build a comprehensive dataset. Then, the emotion feature encoding network is introduced, which is based on multimodal information fusion, to extract the emotion features from multimodal data. Finally, a KAN-based network for evaluating linguistic emotional expressions is designed to achieve a regression analysis and accurate evaluation of students' second language emotional expressions by training on the emotion features extracted from multimodal information based on scoring by professional teachers. This chapter will subsequently detail the differences with citation [26], in that citation [26] mainly focuses on single audio information, while this paper integrates multimodal information. Figure 1 is an illustration of the overall framework, which has three parts. Step 1. Generating a Mixed Dataset: This paper utilizes an LLM, Typecast [27], to convert emotionally labeled text into audio data that conveys the corresponding emotion. We combined these audios with the audios from the IEMOCAP dataset through the speaker video generation model, Dream-talk [28], to generate speaker videos that match the corresponding audio and emotion. And we combined those with the collected student audio and video data to build a comprehensive dataset of student emotional expression evaluations. Step 2. Multimodal Emotion Feature Encoding Network: In this work, an emotion feature encoding network is introduced using multimodal information fusion to deeply extract the emotion spatial features from audio, video, and text. The data processing steps include the following: video data are processed through OpenFace's face detection and feature extraction; audio data are extracted and normalized with features using the ComParE_2016 method in OpenSmile; and text data are pre-processed to generate sentence embeddings using Sentence-BERT (SBERT) [29]. The network uses the Relational Temporal Graph Convolution Network (RT-GCN) [30] to extract global temporal features of multimodal information and Pairwise Cross-Modal Feature Interaction [30] to capture local features between different modalities. It effectively extracts emotion features from other factors of multimodal data (e.g., language style in text, background in video, accent in audio, etc.) to improve the evaluation accuracy. Step 3. Emotion Evaluation Network: For any input of students' language emotional expressions (with emotional labels), the process first generates TTS-synthesized audio through an LLM using the corresponding text and emotional labels. Subsequently, audio  
Figure 1: Architecture of the multimodal information fusion model for analyzing second language emotional expressions. is synthesized through TTS and the corresponding speaker video is generated using the speaker video generation model. Afterwards, the actual student audio and video data, along with the synthesized audio and video data and corresponding text, are fed into the corresponding emotion feature encoding network, where the relevant emotion features are extracted. The combination of the student data encoding and the generated data encoding is jointly inputted into the Kolmogorov-Arnold Network (KAN)-based emotion evaluation network [31]. This part enables the accurate scoring of students' emotional expressions by training on the emotion features extracted from both sets of data according to the teacher's scoring.