# Research on Personalized Course Resource Recommendation Method Based on GEMRec
## 4 Experiments and Results Analysis
### Recommendation Interpretability Experiment  
In this section, we focus on evaluating the interpretability of the recommendation system, including SHAP-based feature importance analysis, quality assessment of generated recommendation explanations, and the impact of interpretability on user experience.  
We used SHAP (SHapley Additive exPlanations) values to analyze the contribution of different features to the recommendation results. Figure 13 shows the global feature importance ranking.  
From Figure 13, we can see significant differences in the influence of various features on the recommendation system. The \(x\)-axis represents feature importance, and the \(y\)-axis shows the average impact of each feature. Features in the upper right corner have a substantial impact on the system's decisions, while those in the lower left have a relatively smaller impact. The size of each dot represents feature usage frequency, with larger dots indicating higher frequency. Among these, features like \"knowledge graph coverage\",  
Figure 12: Performance comparison of various methods under cold-start conditions.  
Figure 13: SHAP-based global feature importance.  
\"current learning goal\", and \"course rating\" have higher SHAP values, indicating their crucial role in the model's decision-making. For example, knowledge graph coverage effectively reflects the match between user learning needs and the course, thus enhancing recommendation accuracy.  
In contrast, features such as \"personalized recommendation score\", \"interactivity\", and \"available time\" have lower SHAP values, indicating that they have a relatively smaller impact on the final result. Although these features may be useful in specific user scenarios, their overall contribution to recommendation effectiveness is limited.  
We evaluated the generated recommendation explanations through both automated and manual assessments, as shown in Table 2.  
In the automated evaluation, the model-generated explanations show a certain degree of linguistic complexity, with a BLEU-4 score of 0.42, indicating moderate similarity between generated text and reference text in terms of syntax and content. Overall, the recommendation explanations perform well in language clarity and logical coherence, effectively persuading users. However, there is still room for improvement in matching the reference text, suggesting the model could be enhanced in generating more reasonable and cohesive explanations.  
To assess the impact of interpretability on user experience, we conducted an A/B test with 93 students. Figure 14 compares the results of recommendations with and without explanations.  
According to the A/B test results, adding interpretability significantly improved user experience. The explainable recommendation system (Group A) outperformed in explanation acceptance, need fulfillment, recommendation accuracy, and system trust. Especially in explanation acceptance and system trust, users' understanding and confidence in the recommendations were significantly enhanced, although the improvement in course click-through rate was limited.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Evaluation Method** & **Metric** & **Score** \\ \hline \multirow{3}{*}{Automated} & Perplexity & 15.3 \\  & BLEU-4 & 0.42 \\  & Readability & 4.2 \\ \multirow{3}{*}{Manual} & Relevance & 4.3 \\  & Persuasiveness & 4.1 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Quality assessment of recommendation explanations.  
Figure 14: Comparison of user experience with and without explanations.