# An Attention-Based Graph Neural Network for Spam Bot Detection in Social Networks
## 4 Experiments
### Result Analysis
#### 4.5.2 Comparison with State-of-the-Art Methods  
The above baseline detection methods are all supervised models and can only use the social network's labeled part. Therefore, they have poor performance in the semi-supervised environment of the real world [19]. Accordingly, we choose GCN, GraphSAGE, and GAT, which belong to the graph neural network, to compare with our method.  
Because the dataset used in this paper pays more attention to the following relationship, the retweet subgraph extracted according to Section 3.2 is sparse. The effect is worse when used alone for spam bot classification than for embedding with the following relationship. Therefore, the advanced graph neural network algorithm for experimental comparison only considers the following relationship. The classification results are given in Table 3 regarding recall, precision, and F1-score on the test data.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Recall** & **Precision** & **F1-Score** \\ \hline RF & 0.66 & 0.88 & 0.76 \\ MLP & 0.73 & 0.81 & 0.77 \\ BP & 0.54 & 0.56 & 0.55 \\ Our Approach & 0.88 & 0.93 & 0.91 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Classification results of classical machine learning methods.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Method** & **Recall** & **Precision** & **F1-Score** \\ \hline GCN & 0.76 & 0.87 & 0.81 \\ GraphSAGE & 0.80 & 0.88 & 0.84 \\ GAT & 0.83 & 0.87 & 0.85 \\ Our Approach & 0.88 & 0.93 & 0.91 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of different algorithms on the dataset.  
In this part, due to the use of a more advanced graph neural network algorithm, the indicators have been significantly improved compared with the traditional machine learning algorithm, especially when the comprehensive evaluation metric of the F1 score, which is used to measure the algorithms' experimental performance, is relatively stable. Due to the introduction of GAT's self-attention mechanism, the correlation between node features is better integrated into the model. The node by node operation is free from the constraints of the Laplacian matrix. Therefore, the GAT algorithm shows a better performance in the experiment. Our method improves GAT and is superior to other algorithms in all metrics. The average increase of the three evaluation indicators is about 5%. In particular, the value of the F1 score is 6% higher than the second-ranked algorithm.  
From the experimental results, the three graph-based neural network algorithms' classification performances are relatively close, especially in the precision metric. It can be seen that this kind of method is better than the traditional method in both accuracy and robustness. Because it considers more neighbor types, our method achieves better classification results.  
Besides this, when evaluating the proposed GCNwithMRF method in the paper of [PERSON] et al. [19], considering the class imbalance problem in the detection of spam bots in the social network, the author uses the Area Under the Precision-Recall Curve (PRAUC) calculated from the prediction score as the evaluation metric. This is because the accurate recall curve is better in evaluating class imbalanced data sets' performances, while the ROC curve is deceptive in this case [36]. Therefore, we use Scikit-learn [37] to calculate the PRAUC value. Our method's PRAUC value is 0.91, which is better than that of the GCNwithMRF method (0.87).  
Through comparison, we can see that even in the imbalanced data, our method still maintains a relatively stable overall performance, and shows better robustness.