# Neighborhood Aggregation Collaborative Filtering Based on Knowledge Graph
## 4 Experiments
### Performance Analysis  
1. Cold Start Problem  
In this subsection, we verify that introducing KG into the recommendation can effectively alleviate the cold start problem. We gradually reduced the size of the MovieLens-20M training set from \(r=100\%\) to \(r=20\%\) (evaluation set and test set unchanged) to study the performances of models in cold start scenario. The results of AUC are shown in Figure 5. When \(r=20\%\), the AUCs of the five baselines decreased by \(7.5\%\), \(5.9\%\), \(2.8\%\), \(4.8\%\) and \(1.9\%\) respectively. The performance of NACF decreased by only \(1.6\%\). This shows that NACF can achieve better result than baselines in a cold start scenario. At the same time, we observe that the recommendation models based on KG performed better on the cold start problem than the classic models. This phenomenon verifies the effectiveness of KG information to alleviate the cold start problem. 1. Effect of attention mechanism  
Next, we study the effectiveness of the proposed attention module. We have mentioned that the attention module calculates the weights of the two parts weights \(w_{(u,r)}\) and \(w_{(e_{i},e_{k})}\) respectively in Section 3. \(w_{(u,r)}\) represents the user's interest in the entities' current relation. \(w_{(e_{i},e_{k})}\) is the similarity between the current neighbor \(e_{k}\) and the recommended item \(e_{i}\). As shown in Figure 6, we observed the impact of different attention mechanisms on the Recall@\(20\) in three datasets. NACF-Mean means the NACF without attention module. In the sub-aggregation process, the calculation of \(V_{j}\) is to average the representation vector of neighbors. NACF-Rel and NACF-Item only use \(w_{(u,r)}\) or \(w_{(e_{i},e_{k})}\) respectively. NACF-Att represents the model that uses the full attention module. We observed that all models using attention mechanisms performed better than NACF-Mean. Moreover, the Recall@\(20\) performance of NACF-Item and NACF-Rel is lower than when them combined. We specifically observe the performance of the model in the MovieLens-20M. Figure 5: Model performances in cold start scenarios. Figure 6: Performance under different attention mechanisms. Compared with NACF-Mean, NACF-Item and NACF-Rel increased by 7.47% and 4.95% respectively, and NACF-Att had a performance improvement of 10.35%. These results validate the effectiveness of the proposed attention module. In neighborhood information aggregation of KG, it is helpful that consider the user's interest in different relations and the similarity between entities. Impacts of different embedded dimensions  
We analyzed the effects of different embedding dimensions of entity on NACF performance. The performances in the three datasets are shown in Table 5. When other parameters unchanged, the AUC of NACF will gradually rise with the increasing of the dimension \(d\) and achieve the optimal performance in 32-dimension or 64-dimension. But as the dimension gets larger, the performance gradually declines. That indicates that too large an embedding dimension may lead to over-fitting of NACF. Impact of Different Neighbor Samples  
We study the performance of NACF in different neighbor size by changing the number of neighboring samples \(K\). As shown in the following Table 6, NACF gets the best performance in all three datasets when sampling four neighbors. It shows that too small a neighboring size \(K\) will make the model unable to extract enough knowledge information, while too large a \(K\) will introduce too much noise and cause a slight decline in performance. Impact of different neighborhood layers  
We also analyze the effects of different sampling layers on the performance of the model. According to the NACF framework, more neighborhood layers will result in a geometric increase in the number of entities participating in calculation. The run time of NACF will also increase significantly. But the performance of the model does not increase gradually.