## 3 The Proposed IDS Framework at the Packet Level
### Reinforcement Learning Module  
Before developing a reinforcement learning module, it is necessary to transfer the intrusion detection problem into an RL-based problem. For this purpose, the intrusion detection problem has been compared to the Atari games studied in [38], considering intrusion detection as a special game. The comparison is shown in Table 3.  
The agent can take the following trajectory until it reaches the end (last packet) of a session:  
\[s_{0},a_{1},r_{1},s_{1},a_{2},r_{2},\ldots,s_{i},a_{i},r_{i},\ldots,s_{t}\]  
In the Atari game, the agent repeats until it reaches the terminated state (game over) of an episode. Similarly, the intrusion detection agent can also take the above trajectory until it reaches the session's end (last image). The state is the game screen in the Atari game, and in IDS, the state is also an image after image embedding on the sessions. The action space of the Atari game contains game operations, which can be expressed as discrete numbers,  
Figure 1: Procedure of the proposed image embedding, i.e., transforming packets to an image. Figure 2: Transform a session into a batch of images. The number of images \(M\) is calculated by: \([(N-1)/54]\ +\ 1\). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**RL** & **Symbol** & **Atari** & **IDS** & **IDS Space** \\ \hline State & \(s\) & Image(s) & Image & Images in dataset \\ Action & \(a\) & Game Operation & Prediction & Label Space \{0, 1, 2...\} \\ Reward & \(r\) & Game Feedback & Reward Mechanism & \{+1, \(-1\)\} \\ Episode & \(-\) & Game Round & Session & \(-\) \\ Agent & \(-\) & Game User & Classifier & \(-\) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison between Atari game and the intrusion detection game. e.g., 0 (one step left), 1 (one step right), 2 (one step up), etc. The action space of the IDS, however, contains the types of the traffic class, which can also be expressed as discrete numbers such as 0 (normal), 1 (attack 1), 2 (attack 2), etc. The reward system is different for the above two problems. For the IDS, the reward system has to be designed carefully. After conducting the research, it has been found that the reward scale significantly impacts the performance of reinforcement learning algorithms. Thus, inspired by the reward clip for the code-level optimization implemented in the Atari game agent [39], the following reward feedback rule has been designed: if the prediction made by the agent is correct, the reward is 1; otherwise, the reward is \(-1\). Table 4 shows the significance of the trajectory and the transformation between RL and IDS. DQN learning is an excellent choice for solving discrete problems and is much easier to implement compared to Proximal Policy Optimization (PPO). After identifying these essential reinforcement learning ingredients, we start designing our reinforcement learning module. Convolutional Neural Networks (CNNs) are chosen as the network structure of the IDS agent since the input states are images. It should be noted that 1D-CNN is also appropriate for the IDS. Each session's images are treated as a type of time-series data since packets embedded in the image are arranged in chronological order according to the capture time. Therefore, 1D-CNN is chosen as an appropriate choice for comparison during the evaluation of the proposed IDS. The main structure of the two RL agents, i.e., DQN-CNN and DQN-1D-CNN, are shown in Figure 3.  
The input states of the RL agents are a finite batch of images. CNN and 1D-CNN are applied for feature extraction on DQN-CNN and DQN-1D-CNN, respectively. An additional fully connected layer is deployed for final classification and detection. The output is Q-values of the current state, and the prediction/action is decided based on the following set of equations. \[Q^{new}(s_{i},a_{i})\gets Q^{curr}(s_{i},a_{i})+\alpha\Big{(}r_{i}+\gamma max _{a}Q(s_{i+1},a_{i+1})-Q(s_{i},a_{i})\Big{)} \tag{1}\]  
where \(\alpha\) is the learning rate and \(\gamma\) is the reward discount factor. The final action is calculated as