# Distributed Detection of Large-Scale Internet of Things Botnets Based on Graph Partitioning
## 5 Evaluation
### Effectiveness
When the cluster number is 1000, accuracy, F1 score, and recall reach their maximum values at a batch size of 50. To summarize, it can be observed that as the cluster number increases, the predicted precision slightly improves, but the remaining metrics show more or less decline. When the cluster number is fixed, accuracy, F1 score, and recall all increase with the increase in batch size, showing a positive correlation with it. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Method** & **Dataset** & **Precision** & **Accuracy** & **F1 Score** & **Recall** & **Avg\_Time (s)** \\ \hline \multirow{3}{*}{GATv2 [50]} & C2 & 0.9904 & 0.99950 & 0.9942 & 0.9982 & 310.50 \\  & P2P & 0.9920 & 0.99965 & 0.9959 & 0.9998 & 310.25 \\  & Chord & 0.9576 & 0.99239 & 0.9715 & 0.9864 & 287.40 \\ \hline \multirow{3}{*}{GraphSAGE [51]} & C2 & 0.9956 & 0.99975 & 0.9971 & 0.9987 & 133.00 \\  & P2P & 0.9963 & 0.99971 & 0.9966 & 0.9970 & 147.33 \\  & Chord & 0.9828 & 0.96691 & 0.8348 & 0.7616 & 152.00 \\ \hline \multirow{3}{*}{GCN [52]} & C2 & 0.9764 & 0.99875 & 0.9857 & 0.9956 & 152.00 \\  & P2P & 0.9827 & 0.99906 & 0.9890 & 0.9955 & 150.75 \\  & Chord & 0.9878 & 0.99789 & 0.9919 & 0.9961 & 152.38 \\ \hline \multirow{3}{*}{Our Approach} & C2 & 0.9966 & 0  
In Table 7, we evaluated the performance impact of different numbers of layers on our approach. A total of seven different numbers of layers were tested, including 6, 7, 8, 9, 10, 12, and 16, and even when the number of layers reached 16, our approach was still able to effectively distinguish between malicious and non-malicious nodes. Similar to Table 6, as the number of layers increases, we can observe in Table 7 that the precision increases from 0.04% to 0.29%, while the accuracy, F1 score, and recall decrease by 0.044%, 0.416%, and 1.176%, respectively. At the same time, the average time consumed per layer increased by 60.39%. Compared to the improvement in precision with an increase in the number of layers, the impact on accuracy, F1 score, recall, and time consumption becomes more prominent. To enhance the propagation of information from neighboring nodes and avoid gradient explosion, we utilize diagonal enhancement to process the embedding representation of each layer. As shown in Figure 7, the comparison between \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Number of Clusters** & **Batch Size** & **Precision** & **Accuracy** & **F1 Score** & **Recall** \\ \hline \multirow{3}{*}{100} & 10 & 0.99683 & 0.9997 & 0.99671 & 0.99659 \\  & 20 & 0.99694 & 0.9997 & 0.99673 & 0.99664 \\  & 25 & 0.99768 & 0.99981 & 0.99794 & 0.99821 \\ \hline \multirow{3}{*}{1000} & 5 & 0.99777 & 0.99916 & 0.99039 & 0.98326 \\  & 10 & 0.99769 & 0.99949 & 0.99411 & 0.99063 \\ \cline{1-1}  & 20 & 0.99778 & 0.99959 & 0.99548 & 0.99322 \\ \cline{1-1}  & 50 & 0.99726 & 0.99965 & 0.99607 & 0.99491 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Effect of different cluster numbers on our approach. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Number of Layers** & **Precision** & **Accuracy** & **F1 Score** & **Recall** & **Avg\_Time(s)** \\ \hline
6 & 0.99729 & 0.99980 & 0.99760 & 0.99792 & 157.43 \\
7 & 0.99600 & 0.99976 & 0.99717 & 0.99836 & 166.65 \\
8 & 0.99829 & 0.99972 & 0.99680 & 0.99532 & 175.64 \\
9 & 0.99819 & 0.99961 & 0.99550 & 0.99284 & 183.40 \\
10 & 0.99850 & 0.99954 & 0.99460 & 0.99077 & 194.00 \\
12 & 0.99846 & 0.99963 & 0.99574 & 0.99305 & 216.40 \\
16 & 0.99890 & 0.99936 & 0.99244 & 0.98616 & 252.50 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Effect of different layer numbers on our approach. Figure 6: Comparison of cluster and non-cluster methods on the C2 dataset. JumpingKnowledge in different layers shows that when the number of layers is six, the accuracy of the former is higher than that of the latter in the three datasets. Based on the above findings, when the number of layers of the graph convolution network is six, the proposed model has strong generalization ability.