## 3 The Proposed IDS Framework at the Packet Level
### Reinforcement Learning Module
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**RL** & **Symbol** & **IDS** & **IDS Space** \\ \hline State & \(s\) & Image & Session Images \\ Action & \(a\) & Prediction & Label Space \{0, 1, 2,...\} \\ Reward & \(r\) & Reward Mechanism & [+1, \(-1\)] \\ Episode & \(-\) & Session & \(-\) \\ Agent & \(-\) & Classifier & \(-\) \\ Step & \(i\) & Packet Position in a session & \(-\) \\ Terminated & \(t\) & Last Packet in a session & \(-\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Conversion of IDS to RL at Packet-level. Figure 3: Main structure of two reinforcement learning agents DQN-CNN and DQN-1D-CNN. For DQN- CNN, the structure of the feature learning part is CNN. For DQN-1D-CNN, the structure of the feature learning part is 1D-CNN. FCN represents the fully connected network. \[a^{*}=argmax_{a}Q(s,a) \tag{2}\]  
As shown in Table 1, the RL module in the IDS has two sub-modules, the interaction module and the training module. The complete procedure of the interaction module is shown in Algorithm 1. The complete procedure of the training module is shown in Figure 4 (taking DQN as an example). Two networks (the Update and the Target) operate together to achieve the approximate regression. The functionality of the target network is to improve the training stability [40] by fixing the regression target in \(N\) steps. The structure of the update network is the same as that of the agent, while the target network copies the structure from the update network and is initialized with the same parameters of the update network. The update network is updated through back propagation (BP), and the target network is updated by copying the parameters from the update network every \(N\) times. ```
1:procedureStartInteracting
2:top:
3: Randomly sample a session and obtain its label
4: Take the first image in the session as current state \(s_{1}\)
5: Feed \(s_{1}\) into agent and obtain action/prediction \(a_{1}\)
6: Feed \(a_{1}\) and label into reward mechanism, obtain reward \(r_{1}\)
7:\(i\gets 0\)
8:loop:
9:\(i\gets i+1\)
10:ifLast image in the session then
11: Store \((s_{i},a_{i},r_{i},None)\) into replay buffer
12:break
13:else
14: Take next image in the session as next state \(s_{i+1}\)
15: Store \((s_{i},a_{i},r_{i},s_{i+1})\) into replay buffer
16:endif
17:gotoloop
18:gototop
19:endprocedure
```  
**Algorithm 1** Interaction Module  
Figure 4: Deep Q-Learning Training Module. The training module and the interaction module work alternately. For example, once the replay buffer is full, training for a specified number of epochs can start. After training, the new agent is set to interact with the environment and store new data into the replay buffer while removing the old data. The complete pack-level IDS algorithm, including both the interaction module and training module, is shown in Algorithm 2. ```
1:procedureDeep Q-Learning
2: Extract sessions from raw traffic file
3: Split sessions based on session-based rule
4: Conduct image embedding on each session
5: Initialize \(Q\) function for agent, set target \(Q\) function \(\widetilde{Q}=Q\)
6:for1... m iterations do
7: Start interacting until replay buffer is full
8:for1...n epochs do
9: Sample a batch data \((s_{i},a_{i},r_{i},s_{i+1})\) from replay buffer
10: Target \(y=r_{i}+\gamma\underset{a}{max}\widetilde{Q}(s_{i+1},a)\)
11: Update Q function through back propagation to make \(Q(s_{i},a_{i})\) close to \(y\)
12: Every \(C\) steps reset \(\widetilde{Q}=Q\)
13: Interact to replace old data
14:endfor
15:endfor
16:endprocedure
```  
**Algorithm 2** Deep Q-Learning Framework for Detection at Packet-level