# Collaborative Analysis of Learners' Emotional States Based on Cross-Modal Higher-Order Reasoning
## 4 Experiments
### Module Ablation Experiments  
In this study, a series of modality ablation experiments were designed to evaluate the impact of different modalities and their fusion methods on the model's prediction performance. In the experimental setup, the symbols T, V, and A represent scenarios where predictions were made using a single modality (text, visual, audio), respectively. Correspondingly, symbols TT, VV, AA indicate scenarios where self-attention mechanisms were applied to each modality to enhance the model's processing capability for single-modality information, followed by predictions.  
Additionally, this section also explores the complementarity between modalities, denoted by TA, TV, and VA, representing scenarios where two different types of modality data were fused through a cooperative attention module before predictions were made. Subsequently, the mean absolute error (MAE), correlation (Corr), accuracy (Acc), and F1 score were tested for each of nine scenarios. The specific experimental results are displayed in Figure 8.  
Based on the experimental data, we can draw three conclusions:  
(1) Comparing the single modalities (T, V, A) with their counterparts enhanced through attention mechanisms (TT, VV, AA), it was observed that the attention mechanism effectively improved the model's performance across all three modalities of text, visual, and audio. For instance, in the text modality, the enhancement with attention mechanism (TT) reduced MAE from 1.051 to 0.1 and increased the F1 score from 78.2 to 78.5, indicating that the attention mechanism more effectively captured important features and complex relationships within each modality;  
(2) When comparing the combined modalities (TA, TV, AV) with single modalities and their enhanced counterparts, we observed that the cooperative attention module further improved performance. Particularly in the text-audio (TA) and text-visual (TV) combinations, MAE, correlation (Corr), Acc-6, and F1 scores were all superior to the single modality T and the enhanced modality TT. This reflects the effectiveness of the cooperative attention module in integrating different modality information and extracting richer and complementary features;  
(3) The superiority of the text modality was evident in the data, particularly in several key performance indicators. The text modality had an MAE of 1.051 and an F1 score of 78.2, whereas the visual and audio modalities had MAE values of 1.502 and 1.313, respectively, and F1 scores of 71.3 and 70.2. The research results indicated that in emotion-recognition tasks, the text modality provided more accurate and relevant predictions.  
This is primarily because text data contains rich semantic and syntactic information, which are important factors for accurately capturing and understanding emotions. The richness of text enables the model to deeply analyze and interpret emotional states, which is often not achievable in visual or audio modalities. In the field of natural language  
Figure 8: Modality ablation experiment results.  
processing, a significant amount of research and technological advancements have focused on how to extract features from text, such as word embeddings, contextual embedding techniques, and pre-trained models like BERT in particular. Due to their ability to capture deep semantic relationships, text feature extraction has become more efficient with models like BERT.  
Additionally, in classroom streaming datasets, audio quality may be affected by background noise, student whispers, or speakers being too far from the microphone, resulting in unclear audio. Moreover, there may be modalities missing due to students not speaking during streaming sessions. Also, visual information may lose critical non-verbal cues due to scene restrictions or occlusion. In contrast, text data tends to be more complete due to collaborative organization through online teaching platforms and speech recognition. This explains why the performance of the text modality outperformed other modalities in modal ablation experiments.