## 5 Experimental Results
### Comparison and Discussion  
In this section, the proposed RL-based IDS design is compared with some other commonly used ML algorithms, including random forest, support vector machine, AdaBoost [43], FCN, LSTM, CNN and 1 DCNN on DDoS2019. Results are shown in Table 17, quantifying the performance of the proposed IDS for packet and flow level in comparison with other ML algorithms.  
Considering flow-level approaches, ensemble methods, including random forest and AdaBoost, achieve great performances on DDoS2019. SVM is the worst one among all models, and it takes the longest time to train. Neural networks, including fully connected neural networks and LSTM, also attain great performances. The proposed RL-CGAN flow-level approach still outperforms all other flow-level models listed in Table 17 without increasing computation time.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Exploration Rate** & **Traffic** & **Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline \multirow{8}{*}{0} & Normal & & 96.23\% & 95.52\% & 95.87\% \\  & PortMap & & 93.79\% & 93.96\% & 93.88\% \\  & NetBIOS & & 92.14\% & 96.12\% & 94.09\% \\  & LDAO & & 92.53\% & 92.72\% & 92.63\% \\  & MSSQL & & 91.48\% & 95.50\% & 93.44\% \\  & UDP & & 93.67\% & 89.98\% & 91.79\% \\  & UDP-Lag & & 93.92\% & 91.70\% & 92.79\% \\  & SYN & & 95.79\% & 93.82\% & 94.80\% \\ \hline \multirow{8}{*}{**0.5**} & Normal & & 98.11\% & 97.54\% & 97.82\% \\  & PortMap & & 96.45\% & 97.14\% & 96.79\% \\ \cline{1-1}  & NetBIOS & & 95.45\% & 98.20\% & 96.81\% \\ \cline{1-1}  & LDAO & & 95.98\% & 95.44\% & 95.71\% \\ \cline{1-1}  & MSSQL & & 94.86\% & 96.76\% & 95.80\% \\ \cline{1-1}  & UDP & & 96.52\% & 93.68\% & 95.08\% \\ \cline{1-1}  & UDP-Lag & & 96.75\% & 95.76\% & 96.25\% \\ \cline{1-1}  & SYN & & 97.29\% & 96.80\% & 97.04\% \\ \hline \hline \end{tabular}  
* The averaged computation time for detection is **0.059 s** with ‘0’ Exploration Rate. The averaged computation time for detection is **0.083 s** with ‘0.5’ Exploration Rate.  
\end{table}
Table 15: Performances (averaged over the last 10 episodes) on validation set. (a, b) denotes the search space for the Bayesian search algorithm.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \(\varepsilon\) & \(\gamma\) & **Mean Episode Reward** & **Mean Validation Accuracy** \\ \hline
**0.034 (0, 0.2)** & **0.066 (0, 0.2)** & **477.3** & **98.86\%** \\
0.062 (0, 0.2) & 0.529 (0.4, 0.6) & 462.9 & 96.82\% \\
0.102 (0, 0.2) & 0.804 (0.8, 1) & 40.9 & 54.22\% \\
0.596 (0.4, 0.6) & 0.015 (0, 0.2) & 472.6 & 98.49\% \\ \hline \hline \end{tabular}  
* In the second stage, we further strengthen the IDS with CGAN for exploration and data augmentation. Experiment results on the test set are shown in Table 16. Using CGAN, the IDS attains better performance, improved from 93.67% to 96.43%. It proves that CGAN can simulate a more realistic network environment, which leads to performance enhancement.  
\end{table}
Table 16: Performances on test set with RL-CGAN.  
For packet-level approaches, the comparison of the proposed approach with conventional 1D-CNN and CNN approaches (without RL) is conducted. It can be seen that conventional deep learning methods can also achieve relatively high performance on the test set, which suggests that image embedding is a useful data preprocessing approach for raw network traffic analysis. Most importantly, the proposed RL-based approach is significantly better in terms of generalizability when compared with these traditional deep learning approaches.  
Among all models (including both flow-level and packet-level), the proposed DQN-1 DCNN at the packet level is the best one for all performance measures. By comparing it to RL-CGAN at the flow level, it should be noted that flow information is incorporated by the DQN-1 DCNN agent through packet features, while RL-CGAN only utilizes flow features provided by the dataset. This is the main reason why DQN-1 DCNN outperforms RL-CGAN in this comparison study.