## 3 The Proposed IDS Framework at the Packet Level
### Preprocessing Module  
Inspired by the adoption of natural language processing (NLP) techniques, such as word2 vec, in processing massive text documents and metadata files [36; 37], an embedding method, namely the image embedding of network packets for the RL-based IDS, is proposed in this section. Table 2 shows the main steps of the preprocessing module.  
In step 1, the network traffic recorded in the 'pcap' files is separated into small traffic files according to the session-based partition rules. One common rule is that if two packets share the same 5-tuple knowledge (source IP, source port, destination IP, destination port, transportation protocol), then they will be categorized in the same session. After partition, several separated sessions stored in 'pcap' files containing various packets recorded in the order of capture time are obtained.  
In step 2, the separated sessions are converted into images using the proposed image embedding method. Generally, a network packet in a session consists of an Ethernet header, a TCP or UDP header, an IP header and application messages. The total field length of the first three headers (except the application message) is 54 bytes. The application message field is dropped during image embedding because of its varying length. The 54-byte packet with selected fields is converted to a line of the image, with each byte representing one pixel. Using one session as an example, the whole procedure of image embedding is illustrated in Figure 1. Thus, a session is transformed into a standard image format with a fixed size of \(54\times 54\), an image consisting of only 54 packets, as shown in Figure 2. For sessions with more than 54 packets, additional images were created by repeating the embedding for the extra packets. Zero-padding is used if the number of remaining packets is less than 54 to ensure the fixed image size of \(54\times 54\). It is essential to embed packets in the order in which they appear in the session. This way, session information is implicitly attached to the packet-level experiments.  
In step 3, after image embedding, each session is labeled according to the time stamp given in the log file of the raw dataset. Finally, in the last step, all generated images' pixels are normalized into [0, 1] from [0, 255].  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Module** & **Sub-Module** & **Function** \\ \hline Preprocessing & None & Data Transformation and Feature Engineering \\ Reinforcement Learning & Interactive & Store batch data into replay buffer 1  \\  & Training & Train the agent \\ Anomaly Detection & None & Detect attacks blind to the training module \\ \hline \hline \end{tabular}
\end{table}
Table 1: Intrusion Detection Framework at Packet-level.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Step** & **Input** & **Method** & **Output** \\ \hline
1 & Raw Network Traffic & Session-based Rule & Segerated Sessions \\
2 & Segerated Sessions & Image Embedding & Session Images \\
3 & Session Images & Labeling with Log Files & Images and Label \\
4 & Images and Label & Normalization & Applicable Dataset \\ \hline \hline \end{tabular}
\end{table}
Table 2: The Preprocessing Module at Packet-level.