## 5 Experimental Results
### Results of the Proposed Packet-Level IDS  
To evaluate the proposed packet-level IDS, two agents based on DQN-CNN and DQN-1D-CNN are trained. Another two agents, PG-CNN and PG-1D-CNN, are trained with the policy gradient learning method for comparison. The dataset is split into a ratio of 4:1 for the training and validation set. During the partition, the training and validation set's session length distribution is consistent, thus, allowing the image ratio to be approximately 4:1. After the partition, the statistics of the dataset are shown in Table 9.  
In the experiment, we compare three reward discount values (\(\gamma\)), namely 0.1, 0.5 and 0.9, on four agents. Then by using the value of accuracy averaged over each class on the validation set to evaluate the performance and select the optimal discount value. The results are shown in Table 10. We notice that when \(\gamma\) equals 0.1, all of the four agents attain the highest performance. With regard to reinforcement learning algorithms, DQN-1D-CNN slightly outperforms PG-1D-CNN over all discount factors, and DQN-CNN also marginally outperforms PG-CNN over all discount factors. Meanwhile, it is observed that PG-based agents are more sensitive to discount factors. Overall results indicate that Deep Q-Learning is a relatively better algorithm than policy gradient methods in our experiments. Hence, in the following case studies, we assume that the discount factor value is fixed as 0.1 and mainly adopt Deep Q-Learning algorithms for the agent. Detailed experiment results of intrusion detection based on using DQN-1D-CNN with a discount value equal to 0.1 are given in Table 11. As shown in the table, our detection system can reach high accuracy at 98.78%. With respect to each class, 100% of normal traffic can be detected. The detection rate (Recall) of MSSQL is the least among all eight types, but still reaching 97.30%. Table 11 also lists the validation results of DQN-CNN. It is seen that the performance of DQN-CNN is slightly worse than that of DQN-1D-CNN, but still reaches a high accuracy of 96.07%. The experimental results prove that both DQN-1D-CNN and DQN-CNN can achieve high performances when dealing with image tasks. In addition, in our experiments, the image can also be readily trea \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Traffic** & **Training Images** & **Validation Images** & **Class** \\ \hline Normal & 25,012 & 6256 & 0 \\ PortMap & 20,200 & 5058 & 1 \\ NetBIOS & 23,652 & 5912 & 2 \\ LDAP & 18,019 & 5083 & 3 \\ MSSQL & 22,986 & 4709 & 4 \\ UDP & 19,159 & 6051 & 5 \\ UDP-Lag & 19,552 & 5198 & 6 \\ SYN & 22,640 & 4970 & 7 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Training and Validation Set Details. Figure 7: Images after image embedding for each class. data, and DQN-1D-CNN achieves better performance in this case. In addition, the average computation times for intrusion detection based on DQN-CNN and DQN-1D-CNN are also given. Afterward, we evaluate our DQN-1D-CNN agent on the test set. In this stage, we are required to set another important hyperparameter: \(\lambda\), the threshold that controls the detection performance of (unknown) anomaly traffic. We treat the eight known types of attacks as the negative class, and the anomaly traffic as the positive class. We use precision and recall to evaluate the performances and select the optimal \(\lambda\). Results are shown in Table 12. As \(\lambda\) is increasing, the precision is decreasing, which means that an increasing volume of network traffic is classified as the anomaly type. Conversely, as \(\lambda\) is increasing, the recall is also increasing, meaning that an increasing amount of anomaly traffic is detected. We can determine the value of \(\lambda\) based on the actual requirements of the intrusion detection system. In our experiments, we fix the \(\lambda\) as 0.7, which is a trade-off between 0.5 and 0.9.