# Collaborative Analysis of Learners' Emotional States Based on Cross-Modal Higher-Order Reasoning
## 3 Methods
### Modality Attention Extraction Module  
An attention mechanism can be utilized to enhance a model's focus on important parts of a time series. For instance, in a long video sequence, the model may only need to focus on a few key frames to determine the overall sentiment, while ignoring irrelevant or low-information parts. The application of attention mechanisms helps the model more accurately integrate information from different modalities such as visual, textual, and auditory, thereby improving performance. Through learning the correlations and importance between modalities, the model optimizes the use of multimodal information for decision making and prediction. Next, the principle of the module relying on the attention mechanism for feature extraction is introduced, as depicted in Figure 6.  
Firstly, through the aforementioned classroom streaming-media feature extraction module, we obtained three feature vectors with different dimensions. These feature vectors represented different modalities of information, namely textual, visual, and auditory information. In order to achieve dimensionality uniformity and facilitate subsequent processing, we input these feature vectors into one-dimensional convolutional layers and  
Figure 6: Modality attention extraction module structure diagram. batched normalization layers to stabilize the feature distribution. Finally, we obtained three-dimensionally consistent feature representations, which we respectively named as textual features \(V_{t}\), visual features \(V_{v}\) and auditory features \(V_{a}\). After obtaining standardized feature representations, the formula is as follows, where U represents the unprocessed feature vector, \(m\in\{t,v,a\}\), \(K\) corresponds to the convolution kernel size for each modality. The specific formulas are as shown in Equations (4) and (5):  
\[U_{m}^{t}=Conv1D(U_{m},K_{m}) \tag{4}\]  
\[V_{m}=BatchNorm(U_{m}^{t}) \tag{5}\]  
Next, we calculated positional embeddings for each position index in the sequence. Assuming the specific process followed Equations (4)-(6), the next step involved extracting self-attention mechanisms between the features of different modalities. This study utilized the superior performance of sine-based positional embedding functions as the positional encoding mechanism in the sequence model:  
\[\left\{ \begin{aligned} PE(i,2k)=sin\Big{(}i/10,000^{2k/d_{ model}}\Big{)}\\ PE(i,2k+1)=\cos\Big{(}i/10,000^{2k/d_{model}}\Big{)}\end{aligned} \right. \tag{6}\]  
Taking textual features \(V_{t}\) as an example, assuming that after being processed by a linear layer, the textual feature \(V_{t}=[X_{1},X_{2},\ldots\ldots X_{u}]\) is a matrix of dimension \(u\times d\), firstly, we mapped the input feature vectors \(V_{t}\) through linear transformations to obtain query (\(Q\)), key (\(K\)), and value (\(V\)) vectors. This mapping process was controlled by three weight matrices \(W_{Q}\), \(W_{K}\), \(W_{V}\), which respectively acted on the input feature vectors to yield query vectors \(Q\), key vectors \(K\), and value vectors \(V\). Through training, the neural network adjusted these weight matrices so that the model could better capture the correlations and importance between input features. The specific mathematical expressions are shown in Equations (7)-(9), where there are weight matrices for learnable linear mappings \(W_{Q}\in\mathbb{R}^{d\times u},W_{K}\in\mathbb{R}^{d\times u},W_{V}\in\mathbb{ R}^{d\times u}\):  
\[Q=W_{Q}V_{t} \tag{7}\]  
\[K=W_{K}V_{t} \tag{8}\]  
\[V=W_{V}V_{t} \tag{9}\]  
Next, we input the query (\(Q\)), key (\(K\)), and value (\(V\)) vectors into a multi-head self-attention mechanism. This mechanism aggregated information via computing attention weights. The multi-head mechanism enabled the model to learn different feature representations because each head learned different attention weights. Finally, we linearly transformed and concatenated the results from multiple heads using an output weight matrix \(W_{a}\), which served as the output of the multi-head attention mechanism.