###### Abstract  
This study aims to develop an emotion evaluation method for second language learners, utilizing multimodal information to comprehensively evaluate students' emotional expressions. Addressing the limitations of existing emotion evaluation methods, which primarily focus on the acoustic features of speech (e.g., pronunciation, frequency, and rhythm) and often neglect the emotional expressions conveyed through voice and facial videos, this paper proposes an emotion evaluation method based on multimodal information. The method includes the following three main parts: (1) generating virtual data using a Large Language Model (LLM) and audio-driven facial video synthesis, as well as integrating the IEMOCAP dataset with self-recorded student videos and audios containing teacher ratings to construct a multimodal emotion evaluation dataset; (2) a graph convolution-based emotion feature encoding network to extract emotion features from multimodal information; and (3) an emotion evaluation network based on Kolmogorov-Arnold Networks (KAN) to compare students' emotion features with standard synthetic data for precise evaluation. The emotion recognition method achieves an unweighted accuracy (UA) of 68.02% and an F1 score of 67.11% in experiments with the IEMOCAP dataset and TTS data. The emotion evaluation model, using the KAN network, outperforms the MLP network, with a mean squared error (MSE) of 0.811 compared to 0.943, providing a reliable tool for evaluating language learners' emotional expressions. second language learning; data generation; multimodal emotion recognition; multimodal emotion evaluation; emotion features +
Footnote †: journal:facial videos. Emotion in language expression not only enhances expressiveness, but also improves the effectiveness of communication, making information exchange more accurate and vivid. Second language learners generally face challenges with emotional expression. Particularly in non-native environments, students often have difficulty in expressing emotions accurately and naturally, which may affect their performance in both oral exams and real-life communication. Traditional teaching and evaluation methods have not sufficiently focused on training emotional expression in speech. Therefore, incorporating the emotion evaluation of voice and video into the evaluation system of language expression will further enhance the comprehensiveness and accuracy of second language expression evaluation. Currently, emotion descriptions often use discrete emotion models, which describe emotions as independent labels with no correlation between them. Among them, [PERSON], an American psychologist, categorized discrete emotions into six fundamental emotions: anger, disgust, fear, happiness, sadness, and surprise. This form of categorization is one of the more frequently used research methods in the existing field of emotion research [5]. Meanwhile, some researchers use dimensional affective models to capture subtle changes in emotions through continuous variables [6]. Emotion recognition technology combines audio and video analysis to create a mapping relationship between data features and emotion features by extracting audio features such as intonation, speech rate, and volume, and video features such as facial expression, eyes, and muscle movement, so as to achieve an accurate emotion classification. Although emotion analysis using multimodal information has achieved remarkable results in the field of classification [7; 8; 9; 10], the quantitative evaluation of emotion is still limited. The complexity of multimodal information fusion also increases the difficulty of model design [11]. In view of this, this study focuses on multimodal information fusion technology to comprehensively evaluate language learners' emotional expressions. Through a more accurate quantitative emotion evaluation, effective feedback and guidance will be provided to learners to help them improve their emotional expressions.