# AA-HMM: An Anti-Adversarial Hidden Markov Model for Network-Based Intrusion Detection
## 7 Experiments
### Experiment about Accuracy and Adaptivity: NSL-KDD
#### 7.3.2 Evaluation on NSL-KDD  
Data Pre-Processing  
We eliminated 13 features before running the AA-HMM because all the samples were concentrated on a single possible value of those features either before or after discretization, which should be treated uninformative features. As a result, 28 of 41 features were adopted by the AA-HMM and only one sub-was is assigned a greater weight during the ensemble procedure.  
Performance: Precision and Recall  
Tables 5-8 depict the performance of DNN, SMR, STL, and AA-HMM, respectively. Four points can be gained from the results: (1) AA-HMM outperformed the DNN on all metrics. (2) SMR is not a balanced model as its performance concentrates more on metrics precision (B) and recall (M). Also, since the SMR's precision (M) is very low, it achieves high recall (M) by predicting as many malicious records as possible. Deploying a SMR model in the real setting would delay the service (e.g., web service) response time, as it would block too many legitimate packets due to low recall (B), which causes the blocked data to be re-transmitted to the end users. A balanced model should control the differences in the four metrics within 10%, like AA-HMM (only 2.225%). (3) Although STL is better than the DNN and SMR, its precision (B) and recall (M) were much lower than the AA-HMM. Also, the metric recall (M) shows that the STL is not a reliable NIDS because it would miss too many attacks due to the low recall (M). (4) AA-HMM is a balanced and most accurate model, and would provide the highest security level to potential victims in the real settings.  
As the calculated efficiencies result from Equations (7)-(10), AA-HMM also outperformed DNN, SMR, and STL in terms of efficiency (security/protection level provided to users in practice).  
\[\begin{split}\textbf{Efficiency (DNN)}&=0.8300\times 10+0.7500 \times 15+0.6580\times 100+0.7420\times 150\\ &=8.3+11.25+65.8+111.3=\textbf{196.6500}\end{split} \tag{7}\]  
\[\begin{split}\textbf{Efficiency (SMR)}&=0.9656\times 10+0.6373 \times 15+0.6693\times 100+0.9700\times 150\\ &=9.656+8.9373+66.93+145.5=\textbf{231.0233}\end{split} \tag{8}\]  
\[\begin{split}\textbf{Efficiency (STL)}&=0.8544\times 10+0.9595 \times 15+0.9362\times 100+0.7841\times 150\\ &=8.544+14.3925+93.6+117.615=\textbf{234.1715}\end{split} \tag{9}\]  
\[\begin{split}\textbf{Efficiency (AA-HMM)}&=0.9337\times 10+0.9531 \times 15+0.9363\times 100+0.9106\times 150\\ &=9.337+14.2965+93.63+136.59=\textbf{253.8535}\end{split} \tag{10}\]  
Verifications: MD and DT  
After improving the performance, it was necessary to verify if the designed variables worked as expected and the performance improvement resulted from these variables. Taking one of the sub-models as an example (all the sub-models shared similar curves), its accuracy trajectory is shown in Figure 9. In the first five windows, the accuracies are very low because the model is adapting to the pattern from its initial state (BM). After fitting, the pattern starts to fluctuate, which lowers the accuracy again. However, the model rapidly fits the fluctuated pattern and maintains the accuracy at a high level until processing all the samples.  
Figure 10 shows the trajectories of MD and DT during evaluation; their values are very high in the early windows. Since the MD and DT act as the indicators of the variation and trend of the pattern in the current window, respectively, the two trajectories indicate that the accuracy would be low (high) when the \(\mid\!\!