## 4 The Proposed IDS Framework at Flow-Level
### Reinforcement Learning Module  
Table 8 describes the important elements of the RL module of flow-level IDS. For brevity, only the parts that are different from the RL module of packet-level IDS are detailed in this section. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Step** & **Input** & **Method** & **Output** \\ \hline
1 & Raw Dataset & Encoding & Encoded Dataset \\
2 & Encoded Dataset & Normalization & Applicable Dataset \\
3 & Applicable Dataset & Feature learning with SAE & SAE’s Encoder \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The Preprocessing Module at Flow-level. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Module** & **Sub-Module** & **Function** \\ \hline Preprocessing & None & Data Transformation and Feature Engineering \\ Reinforcement Learning & Interactive & Store batch data into replay buffer Train the agent \\ Exploration & Training & and the sample agent \\ None & None & A sub-module of Interaction Module \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Intrusion Detection Framework at Flow-level. The action represents the prediction performed by the agent. Furthermore, the flow level reward mechanism is the same as that of the packet level. However, the state and episode for the RL module, in the case of flow-level IDS, are defined differently. For the packet level, 'pcap' files are separated into different sessions; then, through image embedding, these sessions are transformed into images, which are considered as the states. However, network traffic features provided by the dataset are used directly as the states at the _flow level_. Moreover, at the packet level, packets embedded in the image and images in a session are arranged in chronological order (capture time). Thus, a session is viewed as an episode. Nonetheless, the data collected in the dataset has been shuffled at the _flow level_, so there is no apparent chronological order. The episode's length needs to be determined and is assumed to be fixed in the interaction process. Furthermore, the main structure of the update module is a Fully Connected Neural Network (FCN). In the proposed RL module of flow-level IDS, in addition to the detection agent, we also design a sample agent to facilitate the adversarial training. The agent performs the intrusion class's correct prediction (the action) by achieving maximum rewards. The sample agent provides guidance (the action) for the next class to be sampled from. To improve the variability, the sample agent tends to counteract the agent. It chooses a class that is most likely to be misclassified and suggests it as the class to be sampled from for the next state. For this reason, the reward feedback of the agent and the sample agent is the opposite. If the agent's prediction is wrong, the reward feedback of the sample agent is 1; otherwise, if the agent's prediction is correct, the reward feedback of the sample agent is \(-1\): this way, the sample agent functions as the adversarial training agent. The objective of the sample agent is to ensure that those state-action pairs with high classification error rates can be adequately trained. In the interaction stage, we should create a simulation environment. We take the preprocessed dataset as the simulation environment. In our experiments, we focus on episodic tasks; hence, we fix the length of an episode in advance. The complete interaction module is shown in Figure 5. All of the traffic features collected in the dataset can be considered as the states. In the beginning, a state is randomly sampled from the dataset. Feeding the state into the agent (classifier), the agent then outputs the prediction (action) of the current state. Feeding the action-label pair into the reward mechanism, we can obtain the reward. If the current state is not reaching the end of the present episode, we also feed the current state to the sample agent and obtain the next sample class. Afterward, we sample the next state that belongs to this class from the dataset.