# XAI-Based Framework for Protocol Anomaly Classification and Identification to 6G NTNs with Drones
## 7 Simulation Results
### Self-Evolving Classification  
To evaluate the effectiveness of the self-evolving classification method for uncovered attacks (SECs) with the proposed XAI detection framework, CNN-BiLSTM-based [35] and Hoeffding Anytime Tree (HATT)-based [43] detection methods are used for comparison. The CNN-BiLSTM method is configured according to Table 5, and the HATT method is set as described in [43]. The performance of the proposed SEC method is validated with the basic detection metrics defined in Equation (9). Further, the classification performance of uncovered attacks is also assessed, where the basic metrics of classification and detection are equal for rule detection methods. For example, the accuracy of classification equals the accuracy of detection for rule detection methods. Besides, to evaluate the abstraction of fingerprint \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Case** & \(n_{\text{UE}}^{\text{no}}\) & \(n_{\text{UE}}^{\text{n}}\) & \(n_{\text{CN}}\) & \(n_{\text{BS}}\) & **Frequency** \\ \hline Normal & 10 & 0 & 1 & 1 & 1/s \\ \hline UPF DoS & 10 & 50 & 1 & 1 & 100/s, 500/s and 1000/s \\ \hline Protocol sequence tempering & 10 & 0 & 1 & 1 & N/A \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Attack case parameters. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Category** & **Model** & **Parameter** & **Value** \\ \hline \multirow{8}{*}{Rule detection} & \multirow{4}{*}{Self-Attention} & Number of attention heads & 4 \\ \cline{3-4}  & & Attention dimension & 64 \\ \cline{3-4}  & & Hidden dimension & 256 \\ \cline{3-4}  & & Dropout rate & 0.1 \\ \cline{2-4}  & & Number of convolution kernels & 32 \\ \cline{3-4}  & & Convolution kernel size & 3 \\ \cline{2-4}  & & Number of LSTM units & 32 \\ \cline{2-4}  & & Activation function & Softmax \\ \hline \multirow{8}{*}{Baseline detection} & \multirow{4}{*}{AutoEncoder} & Loss function & MSE \\ \cline{3-4}  & & & \(\text{L1-norm}\), \(\text{L2-norm}\) \\ \cline{3-4}  & & Norm regularization & (\(\lambda=3\)) \\ \cline{3-4}  & & MSE threshold & 90\% \\ \cline{3-4}  & & Activation function & Sigmoid \\ \cline{3-4}  & & Interpreter & Kernel Explainer \\ \cline{3-4}  & SHapley Additive exPlanations & Error threshold & 0.5 \\ \cline{3-4}  & & Number of abnormal samples to be explained & 100 \\ \hline \multirow{2}{*}{Training} & \multirow{2}{*}{N/A} & Optimizer & Adam (lr = 0.001) \\ \cline{3-4}  & & Batch size & 128 \\ \cline{3-4}  & & Epochs & 100 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Configuration of models. features related to uncovered protocol attacks, the contributions of these features are represented by the waterfall figure of the SHAP method. To assess the adjustable capability, the attention of the features of the rule detection model is also provided. Comparing the attention figures before and after the adjustment, it can be seen that the rule detection model has enhanced the fingerprint features of uncovered protocol attacks after the adjustment. First, the detection performance of the SEC, CNN-BiLSTM, and HATT methods, namely, the average detection performance for uncovered attacks, is given in Table 6. For the detection of UPF DoS attacks, the precision, recall, and F1 score are used to evaluate performance. Here, the accuracy is omitted since the accuracy is equal to the precision if only uncovered attacks are considered. The CNN-BiLSTM model exhibits high precision (100%) and low recall (50.2%) for UPF DoS attacks, indicating that many of these uncovered attacks are missed, leading to many missed detections. The HATT model has low precision (75.3%) and high recall (78.2%) for UPF DoS attacks. In contrast, the proposed SEC method achieves high precision (100%) and recall rates (93.8%) for UPF DoS attacks. These findings indicate that the SEC method can accurately detect almost all uncovered attacks while maintaining a low false positive rate. This balance results in an F1 score of 96.6%, which is 44.6% and 25.7% higher than those of the CNN-BiLSTM and HATT methods on average, respectively.