# Adaptive Semi-Supervised Algorithm for Intrusion Detection and Unknown Attack Identification
## 3 Methodology  
### Dataset and Preprocessing  
To validate the effectiveness of the proposed semi-supervised intrusion detection algorithm, two datasets were selected for experimental evaluation: CIC-IDS2017 and Kitsune. CIC-IDS2017 is a publicly available dataset widely used in network intrusion detection research, covering various types of normal traffic and malicious attack traffic, including DDoS, web attacks, brute force attacks, and more [21]. The dataset provides traffic records with over 70 features, offering a wide range of attack types and traffic behaviors. On the other hand, the Kitsune dataset offers 115 features and includes 9 distinct attack categories, such as OS scan, ARP MitM, and fuzzing. This provides a newer set of traffic patterns and behaviors for evaluation. Figure 1 below illustrates the distribution of different traffic types and the data distribution after filtering and combining selected categories from the CIC-IDS2017 dataset. Figure 1: DoS GoldenEye, DoS Hulk, DoS Slowhttptest, and DoS Slowloris in (**left**) were merged into a single category labeled “DoS”. Similarly, FTP-Patator and SSH-Patator were combined into the “Brute Force” category, and web attack—brute force, web attack—XSS, and web attack—SQL Injection were grouped under “Web Attack”. This resulted in six traffic types, as shown in (**right**). During the data preprocessing stage, the data were first cleaned by removing redundant and missing values, followed by the elimination of infinite values. These anomalies are typically caused by errors during dataset collection and can hinder the model's ability to converge effectively, weakening its generalization capacity and ultimately affecting its performance. Since the features in the dataset have varying scales and ranges, features with larger numerical ranges tend to dominate the learning process, overshadowing those with smaller values. When using gradient descent optimization, the lack of feature normalization can result in an unusually steep cost function surface, particularly when there are significant differences in gradient magnitudes across dimensions. This can slow down the convergence process or make it difficult to find the global optimum, preventing the model from converging properly. To address this, we applied min-max normalization to standardize the data features within the range of [0,1], improving the effectiveness of model training. The normalization formula is as follows:  
\[x^{\prime}=\frac{x-x_{\text{min}}}{x_{\text{max}}-x_{\text{min}}} \tag{1}\]  
Let \(x\) represent the original feature value of the traffic data in the dataset, and let \(x^{\prime}\) denote the normalized feature value. In this study, to fully leverage the self-attention mechanism of the Transformer for capturing long-range dependencies [22], we applied a specific transformation process to the preprocessed one-dimensional (1D) network traffic features. Each traffic record in the original dataset consists of a 78-dimensional 1D feature vector. While these features can be directly fed into traditional classification algorithms, models like Transformers, which excel at processing sequential and matrix data, would be limited in their ability to capture long-range dependencies if the 1D features were used directly. Therefore, we designed a feature transformation scheme to convert these 1D features into two-dimensional (2D) feature matrices, better suited for Transformer input formats and capable of revealing the intrinsic relationships within the network traffic features. Specifically, for each traffic record's 78-dimensional 1D feature vector, we transformed it into a 20 \(\times\) 20 2D feature matrix. Since the original feature length is less than 400, we replicated the 1D data five times to obtain a 390-length vector and then applied zero-padding to fill the remaining matrix positions.