# Research on Personalized Course Resource Recommendation Method Based on GEMRec
## 3 Graph-Enhanced Multimodal Recommendation Method: GEMRec
### Multimodal Feature Fusion and Entity Extraction Strategy
#### 3.2.4 Entity and Relationship Extraction  
This section presents MEREF (Multimodal Entity-Relation Extraction Framework), a unified framework for extracting entities and relations from multimodal data. MEREF transcends the limitations of traditional text-only approaches by deeply integrating textual, visual, and audio features, enabling end-to-end optimization of entity and relation extraction. The framework's innovation lies in leveraging multimodal feature synergy to enhance entity recognition accuracy while establishing more precise entity relationship networks through graph-based reasoning. In the entity recognition phase, the framework extends the BiLSTM-CRF model with a Multimodal Feature Fusion Gate (MFFG) mechanism. Unlike traditional approaches, MEREF incorporates temporally aligned video and audio features alongside word embeddings. For pretrained word embeddings wi, the framework first maps them to dense vector representations through a feature alignment network, as follows:  
\[\mathrm{e_{i}}=\mathrm{MLP}([w_{i};v_{i};a_{i}]) \tag{15}\]  
where vi and ai represent the video and audio feature vectors corresponding to word token wi. This early fusion of multimodal features provides a richer information foundation for subsequent sequence modeling. The feature fusion process is implemented through a dynamic gating mechanism as follows:  
\[\mathrm{g_{t}}=\mathrm{sigmoid}\big{(}\mathrm{W_{g}[e_{i};v_{i};a_{i}]+b_{g} }\big{)}\mathrm{h_{t}}=\mathrm{g_{t}}\odot\mathrm{concat}(\mathrm{e_{i}},v_{i},a_{i}) \tag{16}\]  
The BiLSTM layer captures long-range dependencies in the sequence through bidirectional processing as follows:  
\[\overset{\rightarrow}{\mathrm{h_{i}}}=\mathrm{LSTM_{f}}\;(\mathrm{e_{i}}, \overset{\rightarrow}{\mathrm{h_{i-1}}}) \tag{17}\]  
\[\overset{\leftarrow}{\mathrm{h_{t}}}=\mathrm{LSTM_{b}}\;(\mathrm{e_{i}}, \overset{\leftarrow}{\mathrm{h_{i+1}}}) \tag{18}\]  
\[\mathrm{h_{i}}=[\overset{\rightarrow}{\mathrm{h_{i}}};\overset{\leftarrow}{ \mathrm{h_{i}}}] \tag{19}\]  
(GNN) to model and solve graph edit distance. The specific process is as depicted in Figure 3.  
Before discussing the graph edit distance calculation between the learner's knowledge state and course content, it is essential to explain how the learner's knowledge state graph \(G_{\text{learner}}\) is constructed. The knowledge state graph represents a learner's current understanding and mastery level of different concepts through a structured representation. The construction of \(G_{\text{learner}}\) involves analyzing learning interaction sequences and concept relationships. For each knowledge concept \(k_{i}\) in the learning process, we calculate its mastery level through  
\[s(k_{i})=\Sigma(\beta_{i}\cdot I(k_{i},b_{j})) \tag{26}\]  
where \(b_{j}\) represents learning behaviors related to concept \(k_{i}\) (such as video watching duration, exercise completion, and quiz performance), \(I(k_{i},b_{j})\) measures the contribution of learning behavior to concept mastery, and \(b_{j}\) is the weight coefficient determined by behavior importance. The relationship strength between concepts, \(r(k_{i},k_{j})\), is computed based on both course structure and learning sequence, as follows:  
\[r(k_{i},k_{j})=\lambda_{1}\cdot c(k_{i},k_{j})+\lambda_{2}\cdot t(k_{i},k_{j}) \tag{27}\]  
where \(c(k_{i},k_{j})\) indicates the correlation in course structure, \(t(k_{i},k_{j})\) represents temporal correlation strength, and \(\lambda_{1}\), \(\lambda_{2}\) are balancing parameters. The knowledge state graph \(G_{learner}\) is then formally defined as  
\[G_{\text{learner}}=(V_{l},E_{l},W_{l}) \tag{28}\]  
where \(V_{l}\) is the set of concept nodes, \(E_{l}\) represents the edge set, and \(W_{l}\) denotes the weight matrices for both nodes and edges. Given two graphs, \(G_{1}\) and \(G_{2}\), the graph edit distance GED(\(G_{1}\),\(G_{2}\))is defined as the minimum number of operations required to transform \(G_{1}\) into \(G_{2}\).