# Distributed Detection of Large-Scale Internet of Things Botnets Based on Graph Partitioning
## 5 Evaluation
### Effectiveness  
We evaluated the performance of our proposed method on three datasets: C2, P2P, and Chord. The evaluation metrics include precision, accuracy, F1 score, and recall. Additionally, we also considered the average runtime as a reference for comparison with the performance of GATv2, GraphSAGE, and GCN. The performance evaluation results are presented in Table 5. The performance evaluation results indicate that on the C2 dataset, GATv2, GraphSAGE, and our approach all demonstrate high precision, with all exceeding 99%. Among them, our approach achieves the highest precision of 99.66%, surpassing GraphSAGE by 0.1%. However, all three methods achieve an accuracy of over 99.95%. In comparison to precision, the difference in accuracy is approximately 1%. The trend of F1 score is similar to that of precision. While our approach does not reach the highest recall, the difference compared to the optimal \begin{table}
\begin{tabular}{c c} \hline \hline
**Hyperparameter** & **Values** \\ \hline Layers & [3, 4, 5, 6, 7, 8, 9, 10, 12, 16] \\ Hidden Channels & 128 \\ Clusters & [100, 1000] \\ Batch Size & [5, 10, 20, 25, 50] \\ Learning Rate & \(1\times 10^{-3}\) \\ Weight Decay & \(5\times 10^{-4}\) \\ Activation Function & ReLU \\ Optimizer & Adam \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Hyperparameter values used in our approach. solution is only around 0.04%. In terms of ave_time, our approach takes approximately 33% longer than GraphSAGE. On the other two P2P datasets, the performance of precision, accuracy, and F1 score remains consistent with what was mentioned earlier. In terms of recall, GATv2 and our method each achieve the best results on one dataset. However, GraphSAGE, which performed best on the C2 dataset, only achieves a recall rate of 76.16% on the Chord dataset. Apart from this, the recall rate fluctuates minimally across all four datasets. It is worth noting that our method achieves the minimum ave_time on the Chord dataset, but on the P2P datasets, GraphSAGE still outperforms it. We speculate that this may be related to the graph partitioning and storage method after METIS partitioning. In comparison, GraphSAGE performs fixed-neighbor sampling for each node in the graph, allowing it to efficiently capture local neighborhood information and reduce the training time. However, this sampling strategy is not conducive to distributed detection of botnets. Overall, the performance evaluation results indicate that our approach achieves the best precision and F1 score across all datasets, reaching 99.71% and 99.74%, respectively. For the Chord dataset, our approach outperforms other methods in all three aspects, even achieving an average runtime as low as 125.67 s.  
In addition, we compared the impact of clustering and non-clustering on the model, as shown in Figure 6. Under different numbers of model layers, the accuracy of clustering is higher than that of non-clustering. The impact of different cluster numbers on our approach was also assessed through 3-fold cross-validation, as shown in Table 6. We set the cluster numbers to 100 and 1000 during the METIS partitioning process and considered batch sizes of 10, 20, 25, and 5, 10, 20, 50, respectively. We separately discussed the influence of cluster numbers on the detection results when using batch sizes of 10 and 20, as well as the impact of different batch sizes on the detection results when the cluster numbers were the same. In Table 6, when the batch size is 10, the precision is 0.086% higher with a cluster number of 1000 compared to 100. However, as the cluster number increases, for batch sizes of both 10 and 20, there is a certain degree of decrease in accuracy, F1 score, and recall. When the cluster number is fixed, precision shows minimal fluctuations in both test groups. When the cluster number is 100, accuracy, F1 score, and recall exhibit an increasing trend with the increase in batch size, but the magnitude of improvement is not significant.