# CKGAT: Collaborative Knowledge-Aware Graph Attention Network for Top-N Recommendation
## 5 Experiments
### Experimental Results  
#### 5.4.1 Recommendation Accuracy (RQ1)  
Like the related work [4,5,7,8] in the recommendation field, our experiments used four popular accuracy metrics [53], Precision@N, Recall@N, F1-measure@N, and NDCG@N, with N = {5, 10, 20, 50, 100}, to evaluate the accuracy of the recommendation methods. Figures 3-6 show the top-N recommendation accuracy comparisons between our CKGAT method and the nine comparison methods on the four experimental datasets. We have the following observations from the results:Figure 4: Top-N recommendation performance comparison on the Book-Crossing dataset. (**a**) Precession@N; (**b**) Recall@N; (**c**) F1-measure@N; (**d**) NDCG@N.  
Figure 3: Top-N recommendation performance comparison on the Last.FM dataset. (**a**) Precesion@N; (**b**) Recall@N; (**c**) F1-measure@N; (**d**) NDCG@N.  
Figure 5: Top-N recommendation performance comparison on the MovieLens 20M dataset. (**a**) Precession@N; (**b**) Recall@N; (**c**) F1-measure@N; (**d**) NDCG@N.  
Figure 6: Top-N recommendation performance comparison on the Dianping-Food dataset. (**a**) Precession@N; (**b**) Recall@N; (**c**) F1-measure@N; (**d**) NDCG@N.  
CKGAT achieves the best recommendation accuracy across all evaluation metrics on the four datasets, with the exception of the precision metric on the MovieLens 20M dataset. This result shows that on the basis of the heterogeneous propagation strategy, the user embeddings and the item embeddings generated by the knowledge-aware GAT-based attentive embedding layer and the attention aggregator enable CKGAT to accurately capture the users' potential interests and improve the accuracy of personalized recommendation.
2. In terms of the precision metric on the MovieLens 20M dataset, CKGAT is slightly lower than KGNN-LS, but is comparable with KGNN-LS in terms of Precision@20. Since N is usually set to 20-50 [54] in most practical application scenarios, it can be considered that CKGAT still has good recommendation accuracy on the MovieLens 20M dataset. The reason why the above exception exists may be that the average number of user-item interactions per user is so large that the propagation of knowledge in the knowledge graph is almost ineffective.
3. The recommendation accuracy of CKGAT on all the datasets is significantly better than CKAN. This result indicates that the topological proximity structures of entities in multi-hop ripple sets can effectively enrich the ripple set embeddings, thereby generating the refined user embeddings and item embeddings.
4. We have observed that the recommendation accuracy of CKGAT on all the datasets is overall better than all the comparison methods, and the recommendation accuracy of CKAN on all the datasets is overall better than other comparison methods except KGIN. These results show that both CKGAT and CKAN, which adopt the heterogeneous propagation strategy, can enhance the user embeddings and the item embeddings by effectively combining the collaborative signals in the user-item interactions and the knowledge associations in the knowledge graph, thereby improving the recommendation performance.
5. CKGAT, KGIN, CKAN, and KGAT outperform KGNN-LS, KGCN, and RippleNet in terms of most evaluation metrics on all the datasets. This result shows that CKGAT, KGIN, CKAN, and KGAT can use both the first-order user-item interaction information in the user-item interaction matrix and the knowledge associations in the knowledge graph to mine accurate user preferences.
6. The seven propagation-based recommendation methods, including CKGAT, are significantly better than the baseline methods BPRMF, CKE and KPRN across all evaluation metrics on all the datasets. This result shows that the propagation-based methods can effectively exploit the high-order relations in the knowledge graph to more accurately capture the users' potential interests in items.