# SybilHP: Sybil Detection in Directed Social Networks with Adaptive Homophily Prediction
## 5 Experiment
### Comparative Experiments on Real-World Twitter Dataset  
In this section, we first focus on the model parameters' adaptation to the real-world Twitter datasets and then give a comparison study with other LBP-based methods. Model parameter adaptation: The initial homophily strength \(\overleftarrow{w},\overrightarrow{w}_{b},\overleftarrow{w}_{s}, \overleftarrow{w}_{b},\overleftarrow{w}_{s}\) for edge potential can be taken as adjustable parameters. We evaluate different configurations of these parameters by variable-controlling on the directed Twitter dataset. Figure 10 shows the variation of detection performance when we vary one of the parameters. We observe that there are points with a good trade-off between precision and recall, and we set them as the parameters for subsequent experiments. Figure 10: Detection performance under different parameter configurations. Note that the precision, recall, and accuracy are relative and have no reference value since the other parameters are fixed. Figure 9: Robustness against noisy labels. Overall classification and ranking performance: Table 4 shows the overall classification performance compared with the other state-of-the-art Sybil-detection methods. SybilHP achieved the highest precision and accuracy and second highest recall. Note that the SybilHP_basic in the table does not incorporate the adaptive homophily estimator mechanism, we see that, with the ablation of the adaptive homophily estimator enhancement, SybilHP still shows superiority over other methods. As LBP-based detection methods estimate the posterior probability for each node to be Sybil, we can rank the nodes by the posterior probabilities to produce a more thorough performance analysis. We take the Area Under the Receiver Operating Characteristic Curve (AUC) as the evaluation measure for ranking, which can be interpreted as the probability that a randomly sampled Sybil node is ranked higher than a randomly sampled benign node in the testing dataset. Figure 11 shows the overall ranking performance by AUC, and we make the following observations. First, we found that methods designed for directed social graphs substantially outperformed those methods for undirected graphs. To adapt to methods for undirected graphs, we only kept reciprocal edges in the original social graph (as the original papers suggest), which resulted in some isolated nodes that were unable to be involved in the computation. This is the main reason for the lower AUC performance compared with that reported in the original paper. However, even when we reevaluate these methods after excluding these isolated nodes, their performance is still limited by the loss of directed information, as SybilSCAR_re and SybilBelief_re shown in Figure 11.  
Second, we can see that complexity and more refined modeling facilitate the capturing of the characteristics of the data. SybilBelief, the basic version of GANG and SybilHP are more effective than their simplified versions because the computational efficiency can hardly be balanced with accuracy. Finally, we observed that, without an adaptive homophily estimator mechanism, SybilHP_basic still had decent performance. Furthermore, the performance improvement brought by adaptive homophily estimators is not significant, so the balance of performance and computational cost should be properly considered in practical applications. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Methods** & **Precision** & **Recall** & **Accuracy** \\ \hline SybilBelief [16] & 0.873 & 0.501 & 0.806 \\ SybilSCAR [17] & 0.905 & 0.508 & 0.815 \\ GANG\_matrix [20] & 0.798 & 0.446 & 0.74 \\ GANG\_basic [20] & 0.757 & **0.808** & 0.847 \\ SybilHP & **0.908** & 0.797 & **0.904** \\ SybilHP\_basic & 0.897 & 0.764 & 0.893 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Classification performance. Figure 11: AUCs of compared methods.