# Using Graph Neural Networks for Social Recommendations
## 4 Experimental Evaluation
### Experimental Results  
We first evaluate RelationalNet against four SOTA social RSs using RMSE, namely SocialMF [24], GraphRec [11], and Diffnet++ [13]. The results are shown in Table 2. It is worth mentioning that there are considerable variations between the metrics obtained by GraphRec and the other models. Specifically, while GraphRec predicts ratings on a scale of 1-5, the other models predict whether the user likes an item or not, with binary values of 1 or 0. Hence, comparing the performance of these models based on the same metric can be challenging due to significant differences in the rating scales. Nevertheless, RelationalNet and Diffnet++ clearly outperformed the other two systems, with the three-layer GNN architecture yielding slightly better results for both models. We therefore focused the rest of our experimental evaluation on those two models (four variations). To ensure fairness in our evaluation, we followed a similar evaluation methodology, using identical datasets and evaluation metrics as in Diffnet++ [13]. Specifically, the Yelp datasets were used, and the model was evaluated using HR and NDCG metrics. Table 3 presents the results for the HR metric of RelationalNet and Diffnet++ with various combinations of hyperparameters (number of layers). RelationalNet, consisting of a two-layer GNN, gives the best results. Both variations of the RelationalNet model outperform those of Diffnet++ up to 24% for the top-5 recommendations, with an average overall improvement of 11.8%. The analysis revealed that increasing the number of layers led to a decrease in the performance of the recommender system, a pattern that is observed both for the proposed RelationalNet model and the Diffnet++ model. Table 4 shows the results for the NDCG metrics for the different models, and again the RelationalNet model with two layers has the best performance. We observe that both RelationalNet variations perform better than those of Diffnet++ for all sizes of top-\(n\) recommendation lists with up to 10% improvement for top-5 recommendations and an average overall improvement of 6.2%. Figure 6(a),b illustrate the comparison of the model's training loss with different learning rates to analyze their performance under various hyperparameters. The visual representation of the loss function enables us to evaluate the model's convergence rate. These figures reveal that a learning rate of 0.0025 leads to a smoother decrease in training loss than a learning rate of 0.005, indicating that the former results offer better convergence. Moreover, the RelationalNet model outperforms the Diffnet++ model in terms of training loss for \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **NDCG (5)** & **NDCG (10)** & **NDCG (15)** \\ \hline Diffnet++ [13] (GNN Layers K = 2) & 0.4822 & 0.5167 & 0.5267 \\ \hline Diffnet++ [13] (GNN Layers K = 3) & 0.4573 & 0.4924 & 0.5044 \\ \hline RelationalNet Model (GNN Layers K = 2) & **0.5058** & **0.5347** & **0.5416** \\ \hline RelationalNet Model (GNN Layers K = 3) & 0.4972 & 0.5283 & 0.5376 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Normalized Discounted Cumulative Gain (NDCG) for top-5, top-10, and top-15 recommendations. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Model** & **HR (5)** & **HR (10)** & **HR (15)** \\ \hline Diffnet++ [13] (GNN Layers K = 2) & 0.2722 & 0.3136 & 0.3634 \\ \hline Diffnet++ [13] (GNN Layers K = 3) & 0.2376 & 0.2772 & 0.3320 \\ \hline RelationalNet Model (GNN Layers K = 2) & **0.2946** & **0.3288** & **0.3739** \\ \hline RelationalNet Model (GNN Layers K = 3) & 0.2840 & 0.3218 & 0.3694 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Hit rate (HR) for top-5, top-10, and top-15 recommendations. most learning rates, indicating its ability to learn from input data effectively and suggesting that it can achieve higher accuracy than the Diffnet++ model. Our findings illustrate the efficacy of the RelationalNet model in tackling the social recommendations generation problem.