## 6 Conclusions and Future Work  
In this paper, an RL-based IDS at the packet level and flow level is proposed. With the help of image embedding, network traffic is transformed into images processed by CNN and 1D-CNN for the detection task at the packet level. In addition, unknown network traffic can be detected through an anomaly detection module. An enhanced RL-based IDS with an exploration module at the flow level is also designed. The experimental results verify the superior performance of the proposed packet-level and flow-level IDS with the most commonly used state-of-the-art algorithms. However, further research works can be undertaken based on the following future directives.  
* GAN can be introduced for packet-level IDS to generate novel flows or sessions. Furthermore, we can use GAN to simulate a dynamic network environment for interaction.
* The idea of exploration can also be introduced into the packet-level reinforcement learning framework to employ the exploration policy for capturing more states in the interaction space.
* The proposed IDS at the packet level can detect the malignant image but can not identify the specific traffic session position embedded in the image. One possible solution is to use an N to N LSTM model for this problem.
* For flow-level IDS, to improve the generalizability of the RL agent, a new reward mechanism can be designed, where a more significant penalty can be assigned to the intrusion detection agent when some critical malignant traffics are not detected.
* Finally, to further validate the proposed IDS design, experiments can be conducted on different datasets with other types of attacks, and a thorough comparison can be performed with other RL-based IDS designs.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Algorithm** & **Level** & **Accuracy** & **Precision** & **Recall** & **F1 Score** & **Detection Time** \\ \hline Random Forest & Flow & 95.42\% & 96.72\% & 94.12\% & 95.40\% & 1.751 s \\ Adaboost & Flow & 94.79\% & 93.98\% & 95.11\% & 94.54\% & 2.610 s \\ SVM & Flow & 90.12\% & 92.15\% & 88.98\% & 90.54\% & 10.019 s \\
3-Layer FCN & Flow & 94.12\% & 93.09\% & 95.24\% & 94.15\% & 0.034 s \\
3-Layer LSTM & Flow & 93.90\% & 94.01\% & 91.99\% & 92.93\% & 0.192 s \\
**RL-CGAN** & Flow & 96.42\% & 96.43\% & 96.42\% & 96.41\% & 0.092 s \\
1D-CNN & Packet & 92.12\% & 94.12\% & 90.43\% & 92.29\% & 0.096 s \\ CNN & Packet & 90.68\% & 91.36\% & 93.10\% & 91.84\% & 0.084 s \\
**DQN-1 DCNN** & Packet & **97.69\%** & **98.10\%** & **96.65\%** & **97.14\%** & 0.101 s \\
**DQN-CNN** & Packet & 95.19\% & 95.94\% & 96.10\% & 96.01\% & **0.079 s** \\ \hline \hline \end{tabular}
\end{table}
Table 17: Comparison with different models.  
**Author Contributions:** Conceptualization, B.Y. and Q.Z.; methodology, B.Y.; software, B.Y. and M.H.A.; validation, B.Y. and Q.Z.; formal analysis, B.Y. and M.H.A.; investigation, B.Y.; resources, B.Y., M.H.A. and Q.Z.; data curation, B.Y.; writing--original draft preparation, B.Y.; writing--review and editing, M.H.A. and Q.Z.; visualization, B.Y. and M.H.A.; supervision, Q.Z.; project administration, Q.Z.; funding acquisition, Q.Z. All authors have read and agreed to the published version of the manuscript.  
This research was funded by the Natural Science and Engineering Research Council (NSERC), Canada.  
The authors declare no conflict of interest.