# Collaborative Analysis of Learners' Emotional States Based on Cross-Modal Higher-Order Reasoning
## 3 Methods
### Cross-Modal Attention Cooperative Interaction Module  
The cross-modal self-attention interaction learning module is a key component aimed at effectively integrating multimodal information. Through modality interaction encoders, feature vectors from each modality are encoded to extract modality-specific information and weighted through self-attention mechanisms to highlight key information within each modality. This interactive learning strategy enables modalities to learn from each other, acquiring richer contextual information. This module not only integrates information from multimodal data but also enhances the correlations between modalities, thereby improving the model's performance and generalization capabilities.  
Let us denote the text modality \(F_{t}\in\mathbb{R}^{u\times\mathrm{d}}\) and the visual modality \(F_{v}\in\mathbb{R}^{u\times\mathrm{d}}\). The attention cooperative interaction module is illustrated in Figure 7.  
Firstly, we computed the correlation between features from the two different modalities. We applied linear transformations to \(F_{t}\) and \(F_{v}\) using weight matrices \(W_{a}\) and \(W_{b}\) respectively, and then calculated their transpose multiplication to obtain the interaction matrix \(C\).  
Then, we combined the text and visual features using the hyperbolic tangent function to obtain bidirectional attention weights. This step helped capture the complex relationships  
Figure 7: Collaborative attention interaction module schematic diagram.  
between text and visual features. The specific formulas were from Equations (14)-(16), where \(\text{W}_{a}\in\mathbb{R}^{d\times d}\), \(\text{W}_{b}\in\mathbb{R}^{d\times d}\), \(\text{W}_{v}\in\mathbb{R}^{u\times u}\), \(\text{W}_{q}\in\mathbb{R}^{u\times u}\) are learnable weight matrices:  
\[C=\tanh((F_{w}W_{a})^{T}(F_{p}W_{b})) \tag{14}\]  
\[H_{v}=\tanh(W_{q}\times F_{v}+W_{v}\times F_{t}\times C) \tag{15}\]  
\[H_{t}=\tanh(W_{q}\times F_{t}+W_{v}\times F_{v}\times C) \tag{16}\]  
Then, we normalized the bidirectional attention weights for text and visual features using the softmax function to obtain attention weights \(a_{v}\) and \(a_{t}\). This step allowed the network to focus attention on important feature information, thus improving feature representation. The specific formulas are shown in Equations (17) and (18), where \(w_{hp}\in\mathbb{R}^{d\times 1}\), \(w_{ht}\in\mathbb{R}^{d\times 1}\) is a learnable weight matrix.  
\[a_{v}=\text{softmax}(w_{ht}^{T}+H_{v}) \tag{17}\]  
\[a_{t}=\text{softmax}(w_{ht}^{T}+H_{t}) \tag{18}\]  
During the feature weighting stage, the model used attention weights to weight the text and visual features, obtaining the final weighted feature representations. Through applying attention weights \(a_{t}\) and \(a_{v}\) to the text features \(F_{t}\) and visual features \(F_{v}\), respectively, the model obtained the corresponding weighted feature vectors \(F_{t}^{\prime}\) and \(F_{v}^{\prime}\).These weighted feature vectors can be seen as projections of text and image in a common semantic space, reflecting the model's understanding and expression of semantic information from text and images. Finally, the two feature vectors were concatenated to obtain the bimodal attention interaction feature vector \(F_{vt}\in\mathbb{R}^{2\times m}\). The specific formulas were from Equations (19)-(21):  
\[F_{v}^{\prime}=a_{v}\times F_{v} \tag{19}\]  
\[F_{t}^{\prime}=a_{q}\times F_{t} \tag{20}\]  
\[F_{vt}=\text{concat}(F_{v}^{\prime},F_{t}^{\prime}) \tag{21}\]  
Based on the same operations, we obtained two more feature vectors \(F_{ta}\) and \(F_{va}\).