# A Novel Method for Detecting Advanced Persistent Threat Attack Based on Belief Rule Base
## 4 Case Study
In Table 13, the recall of No Attack is 100%. This indicates that BRB detected all correct No Attack samples, but also detected some other samples incorrectly as No Attack. In Table 12, BRB has a precision of 95.2% between SVM and MLP for Normal Attack and 87.25% for APT. In Table 13, BRB has a recall rate of only 77.88% for Normal Attack, and a detection accuracy of 93.84% for APT between SVM and MLP. F1-score is defined as the harmonic mean between precision and recall as shown in Table 14. BRB has an F1-score of 96.03% for No Attack, an F1-score of 84.66% for Normal Attack, and an F1-score of 89.87% for APT between SVM and MLP. Accuracy is the ratio of the number of correctly classified samples to the total number of samples as shown in Table 15. The accuracy of BRB is 91.14%, which is lower than the 93.23% of SVM and 95.32% of MLP. Although the accuracy of BRB is not the best, it is acceptable and the results are interpretable. The high accuracy obtained by SVM and MLP with very few samples is difficult to explain. Multi-layer perceptron is initialized using random numbers. This approach implies that the starting point of the search results with suitable weights is random. The MLP model does not receive sufficient training with a limited sample size, so the final weights differ from the most suitable ones. However, in this case, the MLP achieved the highest accuracy with a small sample. This is difficult to interpret and does not provide valid suggestions for improving the defense plan. The SVM model with the \"rbf\" kernel has the same problem. In the case of small samples, there are no more samples to verify whether MLP and SVM really learn the appropriate parameters. MLP and SVM have high accuracy, but the reliability of the results is doubtful. BRB is different from SVM and MLP and is built from expert knowledge. Therefore, the initial values of the BRB model have received a large amount of expert knowledge constraints. This approach already allows BRB to fit functions close to the appropriate weights when BRB is not yet trained. BRB can achieve high accuracy after training with a small number of samples. The accuracy of the BRB model is very dependent on expert knowledge. If the expert knowledge is biased, the results of the model are not good. In this example, the BRB model has 91.14% accuracy, which is lower than SVM and MLP. However, the BRB model is interpretable. Based on the training process, we believe that it is caused by the bias of the expert knowledge. The biased expert knowledge makes the scope of the constraints shifted. This bias still exists after optimization and has an impact on the final \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Items**} & \multicolumn{3}{c}{**Recall**} \\ \cline{2-4}  & **No Attack** & **Normal Attack** & **APT** \\ \hline BRB & 100\% & 77.88\% & 93.84\% \\ SVM & 100\% & 89.84\% & 90.56\% \\ MLP & 100\% & 89.93\% & 96.14\% \\ \hline \hline \end{tabular}
\end{table} 
Table 13: The comparison of recall rate. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Items**} & \multicolumn{3}{c}{**F1-Score**} \\ \cline{2-4}  & **No Attack** & **Normal Attack** & **APT** \\ \hline BRB & 96.03\% & 84.66\% & 89.87\% \\ SVM & 100\% & 88.91\% & 89.26\% \\ MLP & 100\% & 92.23\% & 92.45\% \\ \hline \hline \end{tabular}
\end{table} 
Table 14: The comparison of F1-score. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Models**} & **BRB** & **SVM** & **MLP** \\ \hline Accuracy & 91.14\% & 93.23\% & 95.32\% \\ \hline \hline \end{tabular}
\end{table} 
Table 15: The comparison of accuracy. results. The results can help experts to validate and adjust their own knowledge, which is beneficial for updating BRB rules and correcting the defense plan. The results illustrate that BRB obtained a reliable acceptable accuracy of 91% in the small sample case. MLP and SVM have high accuracy but cannot be interpreted and are unreliable.