# SR-GNN: A Signed Network Embedding Method Guided by Status Theory and a Reciprocity Relationship
## 4 Experimental Evaluation
### Experiment Results
#### 4.2.3 Sign Prediction  
The first task evaluates the ability of SR-GNN to predict the signs of edges. Each dataset is randomly split into an 80% training set and a 20% test set. The direction and sign of the links are derived from the node vector representations learned by the model, which aims to minimize the loss (loss function given by the previous section). Four standard metrics are used to measure the performance: AUC (Area Under Curve), Binary-F1, Macro-F1, and Micro-F1, with larger values indicating better performance. Among the baseline \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Models** & **Directed** & **Status Theory** & **Reciprocity** & **Deep Models** \\ \hline SGCN & No & No & No & Yes \\ SNEA & No & No & No & Yes \\ SDCNN & Yes & Yes & No & Yes \\ S-GNN & Yes & Yes & No & Yes \\ BESIDE & Yes & Yes & No & No \\ SIDE & Yes & Yes & No & No \\ SR-GNN & Yes & Yes & Yes & Yes \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Baseline features. algorithms, SGCN and SNEA are based on undirected signed network modeling, while the rest are based on directed signed networks. Table 3 summarizes the results across all datasets. Key findings:  
1. Directed signed models outperform undirected signed models, highlighting the direction of edges is a significant factor. 2. The SR-GNN model achieves the best results in sign prediction for links on datasets with a high percentage of reciprocal edges, suggesting that reciprocity is a distinct structural feature from edge direction. Disregarding reciprocity would result in the loss of key network structure information, which is also supported by the most recent study [24]. 3. We performed some ablation studies to discuss the effects of different objectives on the results. We controlled the loss function parameters \(\alpha\) and \(\beta\) in Equation (8) to control the weight of status theory and reciprocity, respectively. When \(\beta=0\), it degenerates to S-GNN, which only satisfies the status theory. In this situation, our results are comparable to those of the S-GNN model. We cannot set \(\alpha=0\) because the first term of the loss function \(\mathcal{L}_{1}\) includes information on the edges' direction and status theory. To achieve the best results, the weights of the two parameters are set differently on different datasets. When \(\alpha\) is at {0.75-0.9} and \(\beta\) is at {0.25-0.1}, the best results are obtained for each dataset. This fully demonstrates that the reciprocity feature is independent of the edge direction feature of the graph. The experiments also demonstrate that the impact of reciprocity depends on its proportion within the network. When the proportion of reciprocal edges is too low, such features are insufficient to guide the model in learning new information. When the proportion of reciprocal edges is too high (e.g., in the Bitcoin-OTC dataset, where reciprocal edges account for 78.2% of the total), the directed signed network will \"degenerate\" into an undirected signed network. In this case, modeling with reciprocity does not provide additional topological insights. Further research is needed to gain a better understanding of the