# Evaluation of Machine Learning Intrusion Detection Systems Resiliency to Network Traffic Evolution

Florent Durécu, Maxime Puys, Gérard Chalhoub, Paul-Marie Grollemund

► To cite this version:

Florent Durécu, Maxime Puys, Gérard Chalhoub, Paul-Marie Grollemund. Evaluation of Machine Learning Intrusion Detection Systems Resiliency to Network Traffic Evolution. 33rd International Conference on Software, Telecommunications, and Computer Networks (SoftCOM 2025), Sep 2025, Split, Croatia, Croatia. hal-05191179

HAL Id: hal-05191179

https://uca.hal.science/hal-05191179v1

Submitted on 29 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Evaluation of Machine Learning Intrusion Detection Systems Resiliency to Network Traffic Evolution

1st Florent Durécu

LIMOS-CNRS

Université Clermont Auvergne

Clermont-Ferrand, France

2nd Maxime Puys

LIMOS-CNRS

Université Clermont Auvergne

Clermont-Ferrand, France

3rd Gérard Chalhoub

LIMOS-CNRS

Université Clermont Auvergne

Clermont-Ferrand, France

4th Paul-Marie Grollemund

LMBP-UMR 6620

Université Clermont Auvergne

Aubière, France

**Abstract**—Research in the Network-based Intrusion Detection Systems (NIDS) field is a major focus for academics and the emergence of reliable and applicable detection models quickly became imperative for the integrity of critical infrastructures. Thus, Machine Learning (ML) models—specifically Deep Learning (DL) models—have been developed to detect network attacks and multiple training datasets are available, which allows to reproduce experiments. Nevertheless, the problem of intrusion detection is challenging since generating a representative set of training data is difficult due to the large variety of applications, and the fact that network infrastructures evolve over time. Also, a key concern often ignored is the different performances of models between experiments on known and often curated datasets, and in reality when deployed. In this paper, we propose a methodology to highlight the shortcomings of Anomaly-based NIDS, showcasing significant performance drops using realistic scenarios of evolution in the datasets. We implement a Denoising AutoEncoder (DAE) as a NIDS model. Then, we present a method to test and interpret the NIDS results based on analyzing the DAE's Mean Squared Error (MSE) to provide insights into the model's flow classifications. We advocate for considering the natural evolution of infrastructures when designing NIDS and highlight further research to address this challenge.

**Index Terms**—network intrusion detection system, anomaly detection, network security, machine learning.

# I. INTRODUCTION

Intrusions pose important cybersecurity threats to infrastructures as they can lead to information leaks or service disruptions. These attacks can bypass static, layered defense composed of policies and controls. This is why cybersecurity experts rely on Intrusion Detection Systems (IDS). Network IDS (NIDS) are deployed on the network to protect the entire network from intrusions [1] in opposition to Host-based IDS that monitors activities on hosts to detect malicious behavior.

NIDS employ two main techniques: misuse-based/signature detection, which relies on known patterns to detect intrusions, and anomaly-based detection which uses algorithms to establish a baseline of normal usage patterns that can later be compared to abnormal behavior to detect intrusions. Both techniques can be used in a complementary manner [2]. Machine Learning (ML) models are widely used for anomaly detection in networking [3] for their reliable results. Most of these models are trained offline on large datasets representing networking activity. Many ML-based IDS have been proposed in the literature with exceptional performance results. Nevertheless, these models often fail to meet real-world

expectations in detecting intrusions on real-life networks. In 2017, Viegas, Santin and Oliveira [4], and Sommer and Paxson [5] highlighted the issues related to this methodology. Mainly, network traffic changes over time, making it very challenging to have comprehensive and representative dataset for training ML models for IDS. Currently, there is a lack of evaluation methods for offline models, particularly given the high variance in network traffic. In addition, ML models often assume that the classifier's accuracy with a test dataset corresponds to its accuracy in production. Moreover, proposed frameworks are overly simplified assuming an immutable network. This approach ignores the non-site-specific, continual changes in traffic behavior and the highly variable services found in real-world environment.

While online training approaches exist in the literature [6] to tackle these issues, offline approaches remain predominant. Both of these approaches suffer a lack of appropriate evaluation methods [4], [3].

**Contributions.** In this paper, we propose a methodology that measures the impact on an offline anomaly-based detection model when the network infrastructure evolves through multiple realistic scenarios. To showcase our methodology, we use a Denoising AutoEncoder (DAE) [7], as it is a cornerstone of multiple existing models [8] [9]. In our study, we focus on a self-supervised approach, using a DAE trained on benign flows. This model is then used to detect anomalies, considering that a flow poorly reconstructed by the model is a flow that is not normal, and is likely to be an attack. The reconstruction performance is here evaluated according to the Mean Square Error (MSE) value between an input flow and its reconstruction by the model. Moreover, we implement several scenarios designed to evaluate the DAE model performance on datasets with an evolving structure. In order to analyze the impact of such datasets on the performance of the model and its efficiency, we highlight the particular shapes in the Reconstruction Error Curves (REC) resulting from the particularity of the dataset.

We define multiple modification scenarios based on the CICIDS2017 dataset and evaluate their effects on detection performance. To summarize, our contributions are as follows:

1) We show the impact of realistic infrastructure changes on the performance of self-supervised offline models.
2) We implement and propose the use of MSE representations to detect and understand these impacts.

3) We leverage these representations to offer local interpretations of these scenarios, providing insights on how to generalize such findings.

**Outline.** The remainder of the paper is organized as follows. In Section II, we present a brief summary of existing contributions in the field of ML-based NDIS. In Section III, we describe the CICIDS2017 dataset and how we used it in our experiments. Section IV presents the model, the metrics and the scenarios. Section V presents the results of our experiments and cover the use of the REC to further interpret them. Section VI discusses the scenarios results and proposes further research perspectives.

## II. RELATED WORK

Anomaly detection in network infrastructures has been explored using various models [3] such as Support Vector Machine [10], Decision Trees [11], Naive Bayes classifiers [2], Evolutionary algorithms [3] and genetic approaches [12]. Notably, Deep Learning models have recently become predominant in the literature. AutoEncoders [13], Convolutional Neural Networks, Long Short-Term Memory (LSTM) [14], Transformers [15], combination of Autocoders and random forest [16] have also been applied to network anomaly detection [8].

Very little research has been conducted on studying the performance challenges of deploying pre-trained NIDS on-site. Sommer et al. [5] argue that intrusion detection using ML is fundamentally different from other tasks such as image recognition or detecting anomalies in a medical picture [7]. Authors list multiple challenges specific to NIDS such as the high cost of classification errors, the high variability of benign traffic, and the need to operate in an adversarial settings. Viegas, Santin and Oliveira [4] propose a new evaluation scheme for ML-based NIDS and demonstrate, using sixteen intrusion databases, that many assumptions in the NIDS literature do not hold when using their evaluation scheme. Their scheme is similar to the one we propose in this paper; however (i) they do not propose an interpretation method, (ii) they use a proprietary dataset rather than a public one, and (iii) their study focuses on supervised and non-deep learning models.

Accounting for the natural changes occurring over the lifecycle of a network when training data is a topic widely addressed in the machine learning community [17]. In the IDS field, one proposed solution to handle dataset evolution over time is Online Learning [15], [18]. For instance an Ensemble Learning combined with Stream Learning approach [6] measures data drift using Principal Component Analysis (PCA) and an Online Deep Neural Network (DNN). In this paper, we focus on offline implementations as they are widely adopted and considered into the literature.

The IDS literature primarily relies on public datasets to train and test anomaly detection algorithms. Each entry typically represents a network session, with features derived from session information such as duration or the number of

packets per second. NSL-KDD is a version of KDD-CUP'99, a dataset released by the DARPA, with duplicates removed and a reduced number of entries [19]. Both datasets are commonly used in NIDS research [20], [8], [11], [21]. Some studies consider this dataset unreliable and not representative of real-world network [22]. UNSW-NB15 [23] is another dataset used in various NIDS models [11], [20]. CICIDS2017 [24] was introduced by the Canadian Institute for Cybersecurity and defines eleven criteria for how IDS datasets should be built [25]. These include infrastructure heterogeneity, available protocols and a complete network setup. CICIDS2018 was later proposed by the same authors and includes more hosts and attack types. Both datasets use a B-Profile system to simulate real user behavior and are widely used in the literature [26], [8], [11]. CICIoT2023 is a recent IDS dataset developed by the same authors. It features 33 different attacks carried out within an IoT topology consisting of 105 devices.

To the best of our knowledge, existing contributions to NIDS using unsupervised ML do not explicitly evaluate the impact of evolving network traffic or topology on anomaly detection efficiency.

## III. DATASET USED AND NETWORK TRAFFIC DESCRIPTION

We aim to apply realistic topology or behavior modifications to network traffic in order to assess their impact on an unsupervised ML-based NIDS. We evaluate how these modifications affect the model's detection performance. To provide a reproducible setup that enables IDS designers to test the resilience of their models, we chose to rely on a curated version of the well-known CICIDS2017 public dataset, proposed by Engelen et al. [27].

### A. Brief Description of the Dataset

CICIDS2017 was generated using a realistic network topology that includes various types of equipment typically found in real-world networks. The setup features a diverse range of devices functioning as clients, generating outbound traffic to the Internet, as well as servers that receive inbound traffic from the Internet. The captured communications are varied and encompass many common protocols, including FTP, HTTP, HTTPS, SSH, SMTP, KERBEROS, NTP, NETBIOS, SMB, LDAP and DNS.

The dataset includes 10 clients, each running a distinct version of an operating system, a web server that supports access via SSH, FTP, HTTP and HTTPS, an Ubuntu server externally attacked using the Heartbleed vulnerability, in addition to internal DNS, NTP, LDAP, SMB, and Kerberos servers. Malicious traffic is generated by two attacker entities that perform bruteforce and SSH/FTP-Patator, Denial of Service (Hulk, GoldenEye, Slowloris, and DDOS), in addition to Web attacks (SQL and XSS injection, bruteforce), infiltration, and Portscanning. We chose to rely on this dataset for our experiments since it is widely used in research proposing N-IDS algorithms and its elements cover most of what is expected
from a basic infrastructure, and in addition, in includes many fundamental attacks.

## B. Distribution of Network Traffic on Client Hosts

In a given network, the traffic generated by each host depends on the services and applications running on that host. Introducing a new host on the network that generates previously unseen traffic – i.e. unseen during the training phase of the NIDS – can reduce the effectiveness of the detection process. Totally unseen traffic has a great chance of being misclassified, yet more subtle changes can still have an important impact. In this section, we analyse the network traffic associated with each host in the dataset. Each client host in the dataset has a different network traffic, characterized by different protocols and varying proportion of those protocols. Fig 1 displays the distribution of the three main protocols (DNS, HTTP, and HTTPS) on Ubuntu, Windows and the Mac host. We can note that all three protocols are present on each client in significant proportion, that the Mac host exhibits a relatively higher proportion of web traffic, and that no single client concentrates all occurrences of any one of the main protocols.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Representation of Protocols Distributions per host.</figcaption></figure>

# IV. MODEL DESCRIPTION AND EVALUATION
## METHODOLOGY

In this section, we describe the model used in our experiments, the proposed framework for experimentation, evaluation, and the scenarios considered.

### A. Model Characteristics and Data Preprocessing

The model used in this work is a Denoising AutoEncoder (DAE). In this section, we describe its architecture and the evaluation metrics.

**Data preprocessing.** The process used to train and test a DAE model is composed of four steps:

a) Data cleaning, performed in two phases: (i) first we remove highly correlated features using a correlation matrix. Our correlation matrix aligns with the findings

of Pelletier and Abualkibash [28], regarding the least important features that are dropped from the dataset. The dataset is reduced from 78 to 61 features. Then, (ii) only numerical features are retained for model input. Categorical features — such as Timestamps, source and destination IP address, and source and destination ports — are discarded.

b) We normalize the features and One-Hot-Encode the labels.

c) We split the dataset into 3 sets following standard ML practice [13]: A Train set, A Validation set, and a Test set. The Validation set is also used to determine the model’s classification threshold.

d) The DAE is trained on the prepared data and evaluated on the test set. We assume that we can isolate benign flows for the training.

The model is trained using data from the Train set, which contains 70% of the benign entries in the dataset. The remaining 30% benign entries are used to generate Validation and Test sets, each comprising 80% benign flows and 20% attack flows.

**Model Parameters.** The DAE consists of layers containing 512 nodes, making it overcomplete as the hidden layers have more nodes than the input layer. We used three configurations for the DAE varying the size of the latent space from 16, 32, and 512 — meaning without bottleneck — nodes. The models are denoted LAT16, LAT32 and LAT512 respectively.

The DAE is trained to reconstruct its input vector. Mean Squared Error is used as a loss function for the training process of the DAE by measuring the difference between the input vector Y and the reconstructed output vector O:

$$MSE = \frac{1}{n} \sum_{i=1}^{n} (Y_i - O_i)^2 \quad (1)$$

This MSE is computed for each entry and compared to a classification threshold to label each flow as either benign or abnormal. Hence, the model is a binary classifier.

**Metrics and Threshold Computation using F1-Score.** We use three performance metrics to evaluate the model: Accuracy, Recall, and F1-Score. These metrics are computed based on the predicted labels and ground truth True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN). In our case, a positive entry refers to malicious behavior within the network.

$$Accuracy = 2 \times \frac{TP + TN}{TP + TN + FN + FP} \quad (2)$$

Accuracy is defined as the proportion of correctly classified entries over the total number of entries.

$$Recall = 2 \times \frac{TP}{TP + FN} \quad (3)$$

Recall is the proportion of positive entries that are correctly classified. It is a key metric that we aim to maximise in order to detect as many anomaly as possible within the infrastructure.
$$F1 = 2 \times \frac{\text{accuracy} \times \text{recall}}{\text{accuracy} + \text{recall}} \qquad (4)$$

F1-Score is the harmonic mean of Accuracy and Recall, allowing us to balance the trade-off between detecting anomalies and minimizing false positives. We use this metric to determine the optimal MSE threshold. The selected threshold is the one that yields the highest weighted F1-score for the Validation Set. This threshold is then fixed for the model and used to compute the confusion matrix on the Test set.

## B. Experimentation Methodology

In what follows, we describe a methodology to simulate updates in network infrastructures by applying controlled modifications to the dataset. This approach involves training the model on a specific dataset, and then evaluating the trained model on a partially modified version that includes traffic flows from newly added hosts, new protocols or changes to network properties. In a real-world scenario, this emulates training an IDS on flows collected at a given time, followed by changes in the network — such as new hosts added, hosts removed, new network equipment or new services installed on a host — that were not present during the training phase.

Modifying CICIDS2017 to introduce new hosts, protocols or network changes is challenging, as the dataset simulates human-like behavior for each host [25]. To address this, we proceed by removing flows associated with specific hosts or protocols from the original dataset in the Train set and Validation set. Finally, we reintroduce the previously removed flows in the Test set, effectively testing on the original CICIDS2017 dataset. This procedure emulates the sudden appearance of specific hosts or protocols in the network after the model has been deployed.

To achieve this configuration, we follow the steps below:

1. For a specific scenario, we determine $E_i$, a set composed of elements (e.g. hosts or protocols), as newly added to the infrastructure.

2. We define two sets: $A_0$ the original dataset, and $A_i = A_0 \setminus E_i$, a subset of $A_0$, with $E_i$ removed.

3. We train the model using $A_i$ and denote this model $M(A_i)$.

4. We use the Validation set to define a classification threshold that maximizes the F1-score.

5. We use $A_0$, the original dataset, as the Test set. This emulate evaluating the model in an updated environment.

6. We evaluate $M(A_i)$ using the metrics defined in Section IV-A and compare it to a model trained with $A_0$.

7. For each scenario, we measure the evolution of MSE as new changes are introduced into the network.

We define several scenarios that reflect realistic changes that may occur in operational networks. We consider the following three scopes, **Scope 1** that simulates the addition of hosts, **Scope 2** that simulates newly introduced protocols not seen during training, **Scope 3** that simulates changes in throughput characteristics (e.g trained on a smaller rate of packets per seconds or bytes per seconds).

# V. RESULTS

In this section, we analyse the results in the aforementioned scenarios, demonstrate the use of the REC to visualize the evolution of the models, and highlight key findings in protocol classification to provide insight on the model's efficiency.

## A. Scope 1: Changing the number of hosts

Table I presents the different cases of hosts addition within the dataset following the actions described in Section IV-B. The last two columns show the performance of the model trained without element E, and when evaluated on the new E element. Since E elements is mainly composed of benign traffic, accuracy is a more appropriate metric for evaluating model performance.

As it can be seen, the models exhibit different behaviors: LAT512 outperforms both LAT16 and LAT32. The Windows case is the most predominant, where the performance of the two lower-dimensional latent space models drops to 16% and 9% accuracy, respectively, while LAT512 achieves 68%. LAT512 also performs better across all cases, particularly on Windows hosts. The different machines produce varying results. The score of the Ubuntu cluster can be explained by the presence of a Ubuntu server, which helps the model reconstruct flows from Ubuntu hosts. In all cases, the model's performance decreases on the corresponding element E that has been added.

Fig 2 shows the density of flow reconstruction errors. The solid blue line represents the benign entries used for training. The dash-dot orange line represents E — in this example the Mac host — and the red vertical dashed line indicates the threshold, computed as explained in Section IV-A, which corresponds to the reconstruction error that maximizes the F1 score on the validation set. Fig 2b shows that Mac traffic flows are correctly reconstructed and fall below the threshold. Fig 2a shows that the model trained without the Mac host exhibits lower performance, reconstructing a significant proportion of flows above the threshold.

## B. Scope 2: adding new protocols

Table II presents the different cases of protocol addition within the dataset. Ports 53 and 443 — DNS and HTTPS — are excluded because they are too fundamental to network

<table><caption>TABLE I: Scope 1: number of hosts.</caption><thead><tr><th rowspan="2">E</th><th rowspan="2">Model</th><th colspan="5">Performance Metrics</th></tr><tr><th>ACC</th><th>ACC</th><th>F1</th><th>FPR</th><th>FNR</th></tr></thead><tbody><tr><td rowspan="3">Baseline</td><td>LAT16</td><td>N/A</td><td>N/A</td><td>0.79</td><td>0.05</td><td>0.22</td></tr><tr><td>LAT32</td><td>N/A</td><td>N/A</td><td>0.81</td><td>0.05</td><td>0.19</td></tr><tr><td>LAT512</td><td>N/A</td><td>N/A</td><td>0.87</td><td>0.04</td><td>0.12</td></tr><tr><td rowspan="3">Ubuntu (4 hosts)</td><td>LAT16</td><td>0.96</td><td>0.95</td><td>0.68</td><td>0.1</td><td>0.2</td></tr><tr><td>LAT32</td><td>0.93</td><td>0.85</td><td>0.74</td><td>0.04</td><td>0.33</td></tr><tr><td>LAT512</td><td>0.99</td><td>0.94</td><td>0.79</td><td>0.05</td><td>0.21</td></tr><tr><td rowspan="3">Windows (5 hosts)</td><td>LAT16</td><td>0.94</td><td>0.16</td><td>0.57</td><td>0.1</td><td>0.45</td></tr><tr><td>LAT32</td><td>0.96</td><td>0.09</td><td>0.58</td><td>0.1</td><td>0.44</td></tr><tr><td>LAT512</td><td>0.96</td><td>0.68</td><td>0.71</td><td>0.04</td><td>0.38</td></tr><tr><td rowspan="3">Mac (1 host)</td><td>LAT16</td><td>0.94</td><td>0.57</td><td>0.74</td><td>0.05</td><td>0.31</td></tr><tr><td>LAT32</td><td>0.92</td><td>0.65</td><td>0.82</td><td>0.05</td><td>0.16</td></tr><tr><td>LAT512</td><td>0.97</td><td>0.76</td><td>0.89</td><td>0.01</td><td>0.16</td></tr></tbody></table>
<figure><img src="image_3.png" /><figcaption>Fig. 2: Mac host REC results.</figcaption></figure>

<table><caption>TABLE II: Scope 2: adding new protocols.</caption><thead><tr><th rowspan="2">E</th><th rowspan="2">Model</th><th colspan="5">Performance Metrics</th></tr><tr><th>ACC</th><th>M(E)</th><th>F1</th><th>FPR</th><th>FNR</th></tr></thead><tbody><tr><td rowspan="3">22 (SSH)</td><td>LAT16</td><td>0.97</td><td>0.34</td><td>0.79</td><td>0.05</td><td>0.24</td></tr><tr><td>LAT32</td><td>0.97</td><td>0.53</td><td>0.82</td><td>0.05</td><td>0.16</td></tr><tr><td>LAT512</td><td>0.96</td><td>0.51</td><td>0.9</td><td>0.01</td><td>0.15</td></tr><tr><td rowspan="3">389 (LDAP)</td><td>LAT16</td><td>0.75</td><td>0.37</td><td>0.8</td><td>0.05</td><td>0.21</td></tr><tr><td>LAT32</td><td>0.71</td><td>0.40</td><td>0.8</td><td>0.06</td><td>0.17</td></tr><tr><td>LAT512</td><td>0.78</td><td>0.56</td><td>0.88</td><td>0.04</td><td>0.09</td></tr><tr><td rowspan="3">137 (NetBIOS)</td><td>LAT16</td><td>0.90</td><td>0.28</td><td>0.75</td><td>0.05</td><td>0.3</td></tr><tr><td>LAT32</td><td>0.93</td><td>0.75</td><td>0.81</td><td>0.06</td><td>0.15</td></tr><tr><td>LAT512</td><td>0.94</td><td>0.73</td><td>0.86</td><td>0.03</td><td>0.14</td></tr><tr><td rowspan="3">138 (NetBIOS)</td><td>LAT16</td><td>0.90</td><td>0.80</td><td>0.75</td><td>0.05</td><td>0.29</td></tr><tr><td>LAT32</td><td>0.93</td><td>0.00</td><td>0.83</td><td>0.05</td><td>0.11</td></tr><tr><td>LAT512</td><td>0.96</td><td>0.66</td><td>0.89</td><td>0.03</td><td>0.1</td></tr><tr><td rowspan="3">139 (SMB)</td><td>LAT16</td><td>0.90</td><td>0.13</td><td>0.76</td><td>0.05</td><td>0.29</td></tr><tr><td>LAT32</td><td>0.93</td><td>0.09</td><td>0.82</td><td>0.05</td><td>0.15</td></tr><tr><td>LAT512</td><td>0.96</td><td>0.13</td><td>0.89</td><td>0.02</td><td>0.15</td></tr><tr><td rowspan="3">21 (FTP)</td><td>LAT16</td><td>0.90</td><td>0.63</td><td>0.77</td><td>0.05</td><td>0.26</td></tr><tr><td>LAT32</td><td>0.93</td><td>0.61</td><td>0.83</td><td>0.05</td><td>0.12</td></tr><tr><td>LAT512</td><td>0.95</td><td>0.62</td><td>0.86</td><td>0.04</td><td>0.11</td></tr><tr><td rowspan="3">80 (HTTP)</td><td>LAT16</td><td>0.86</td><td>0.79</td><td>0.63</td><td>0.1</td><td>0.31</td></tr><tr><td>LAT32</td><td>0.84</td><td>0.71</td><td>0.62</td><td>0.09</td><td>0.4</td></tr><tr><td>LAT512</td><td>0.83</td><td>0.71</td><td>0.63</td><td>0.07</td><td>0.47</td></tr><tr><td rowspan="3">123 (NTP)</td><td>LAT16</td><td>0.91</td><td>0.73</td><td>0.77</td><td>0.05</td><td>0.26</td></tr><tr><td>LAT32</td><td>0.92</td><td>0.54</td><td>0.82</td><td>0.05</td><td>0.21</td></tr><tr><td>LAT512</td><td>0.95</td><td>0.79</td><td>0.88</td><td>0.03</td><td>0.12</td></tr></tbody></table>

operations for their addition to be relevant or realistic. The destination ports used in the experiment correspond to potential new applications within an infrastructure.

Some of these protocols, notably LDAP and SMB, are poorly represented in the dataset — 0.09% and 0.25%, respectively —, making it difficult for the baseline model to recognize them accurately. However, a significant shift is still observable. The results indicate that the models need to be trained on new protocols before they can accurately reconstruct them and classify them as benign entries. In the case of SSH protocol, once it is learned by the model, it achieves an accuracy around 97%. However, if SSH is introduced into the infrastructure after the model has been trained, its classification accuracy drops to 34, 53 and 51% on LAT16, LAT32, and LAT512, respectively.

Fig 3 shows the REC corresponding to SSH. Here, the green dotted line represents the attacks in comparison to the threshold. The shift between Fig 3b and Fig 3a illustrates how the inclusion of SSH in the training set impacts the model

— LAT16 in this case. In a real-world setting, this anomaly detection model would be unable to correctly incorporate SSH traffic without a retraining.

### C. Scope 3: change in traffic throughput

Table III shows three cases involving changes to network parameters. The features affected are Bytes per second, Packets per second, Forward Packets per second, and Backward Packets per second.

Even if a change is observed, it is not a significant one. The Quantile Transformer normalization smooths out values that are of the same order of magnitude. This example shows that a simple change in few features does not have a major impact on flow reconstruction: the model is sensitive to more complex changes.

### D. Classification of Protocols disparities

When a model is evaluated with new hosts, as in Scope 1, the protocols from these new hosts are partially misclassified. However, the different protocols are not misclassified with the same manner. With LAT16, for the Mac host, the proportion of misclassified HTTPS flows is 77,2%, while misclassified DNS flows account for only 0.2%. This is due to the fact that HTTPS has varied usages, and its flow properties are highly dependent on the host.

To illustrate this, we compute — after applying the Quantile Transformer used in our models — the average Euclidian distance between Mac HTTPS and DNS flows and the rest of the HTTPS and DNS flows. For HTTPS, we obtain a distance of 752.16, as for DNS, the distance is 207.95. This result shows that HTTPS flows are more diverse than DNS flows, which explains the model’s lower generalization performance for HTTPS.

## VI. CONCLUSIONS

In this paper, we proposed a methodology to measure the impact of changes in a network infrastructure on an intrusion detection model based on an AutoEncoder. To simulate such changes, we defined a set of scenarios using a corrected version of CICIDS2017. These scenarios are constructed by introducing new elements to the test set that were not present during training. We evaluate the distribution of MSE to

<figure><img src="image_4.png" /><figcaption>Fig. 3: Results of SSH protocol on LAT16.</figcaption></figure>
<table><caption>TABLE III: Scope 3: change in traffic throughput.</caption><thead><tr><th>%of</th><th>Model</th><th>ACC</th><th>F1</th><th>FPR</th><th>FNR</th></tr></thead><tbody><tr><td rowspan="3">10%</td><td>LAT16</td><td>0.89</td><td>0.69</td><td>0.01</td><td>0.15</td></tr><tr><td>LAT32</td><td>0.93</td><td>0.82</td><td>0.05</td><td>0.16</td></tr><tr><td>LAT512</td><td>0.93</td><td>0.83</td><td>0.05</td><td>0.14</td></tr><tr><td rowspan="3">50%</td><td>LAT16</td><td>0.89</td><td>0.7</td><td>0.1</td><td>0.13</td></tr><tr><td>LAT32</td><td>0.93</td><td>0.82</td><td>0.05</td><td>0.15</td></tr><tr><td>LAT512</td><td>0.93</td><td>0.83</td><td>0.05</td><td>0.11</td></tr><tr><td rowspan="3">Baseline</td><td>LAT16</td><td>0.92</td><td>0.79</td><td>0.05</td><td>0.22</td></tr><tr><td>LAT32</td><td>0.93</td><td>0.81</td><td>0.05</td><td>0.19</td></tr><tr><td>LAT512</td><td>0.95</td><td>0.87</td><td>0.04</td><td>0.12</td></tr></tbody></table>

visualize the impact of these modifications on the model's detection performance. We further analyzed detection behavior by breaking down results per host or per protocol. Our analysis yields several key insights: Anomaly-based NIDS models can experience significant performance degradation when new benign elements are introduced into the network.

The model considered in our paper is a simple implementation of a DAE used for anomaly-based intrusion detection. In our future work, we plan to extend our methodology to more complex ML models based on ensemble learning [11]. Nonetheless, this study serves as a foundation for understanding the limitations of anomaly detection systems when network conditions evolve, and can, with other criteria, provide benchmarks to assess NIDS performance. Our approach can be adapted to assess the resilience of online models — specifically, how quickly they adapt to changes and re-stabilize. Finally, we intend on focusing on features characteristics and their impact on the efficiency of the anomaly detection.

## ACKNOWLEDGMENT

This research was financed by the French government IDEX-ISITE initiative 16-IDEX-0001 (CAP 20-25).

## REFERENCES

[1] Z. Ahmad, A. Shahid Khan, C. Wai Shiang, J. Abdullah, and F. Ahmad, "Network intrusion detection system: A systematic study of machine learning and deep learning approaches," *Transactions on Emerging Telecommunications Technologies*, vol. 32, no. 1, p. e4150, Jan. 2021.

[2] A. Patel, H. Alhussian, J. M. Pedersen, B. Bounabat, J. C. Júnior, and S. Katsikas, "A nifty collaborative intrusion detection and prevention architecture for Smart Grid ecosystems," *Computers & Security*, vol. 64, pp. 92-109, Jan. 2017.

[3] Z. Yang, X. Liu, T. Li, D. Wu, J. Wang, Y. Zhao, and H. Han, "A systematic literature review of methods and datasets for anomaly-based network intrusion detection," *Computers & Security*, vol. 116, p. 102675, May 2022.

[4] E. K. Viegas, A. O. Santin, and L. S. Oliveira, "Toward a reliable anomaly-based intrusion detection in real-world environments," *Computer Networks*, vol. 127, pp. 200-216, Nov. 2017.

[5] R. Sommer and V. Paxson, "Outside the Closed World: On Using Machine Learning For Network Intrusion Detection," 2010.

[6] O. A. Wahab, "Intrusion Detection in the IoT under Data and Concept Drifts: Online Deep Learning Approach," 2022.

[7] A. Kascenas, P. Sanchez, P. Schrempf, C. Wang, W. Clackett, S. S. Mikhael, J. P. Voisey, K. Goatman, A. Weir, N. Pugeault, S. A. Tsaftaris, and A. Q. O'Neil, "The role of noise in denoising models for anomaly detection in medical images," Jan. 2023.

[8] M. Arafah, I. Phillips, A. Adnane, W. Hadi, M. Alauthman, and A.-K. Al-Banna, "Anomaly-based network intrusion detection using denoising autoencoder and Wasserstein GAN synthetic attacks," *Applied Soft Computing*, vol. 168, p. 112455, Jan. 2025.

[9] F. S. Alrayes, M. Zakariah, S. U. Amin, Z. Iqbal Khan, and M. Helal, "Intrusion Detection in IoT Systems Using Denoising Autoencoder," *IEEE Access*, vol. 12, pp. 122401-122425, 2024.

[10] A. Özgür and H. Erdem, "A review of KDD99 dataset usage in intrusion detection and machine learning between 2010 and 2015," Apr. 2016.

[11] N. Thockchom, M. M. Singh, and U. Nandi, "A novel ensemble learning-based model for network intrusion detection," *Complex & Intelligent Systems*, vol. 9, no. 5, pp. 5693-5714, Oct. 2023.

[12] V. Hajisalem and S. Babaie, "A hybrid intrusion detection system based on ABC-AFS algorithm for misuse and anomaly detection," *Computer Networks*, vol. 136, pp. 37-50, May 2018.

[13] J. J. Shirley and M. Priya, "An Adaptive Intrusion Detection System for Evolving IoT Threats: An Autoencoder-FNN Fusion," *IEEE Access*, vol. 13, pp. 4201-4217, 2025.

[14] V. Hnamte and J. Hussain, "DCNNBiLSTM: An Efficient Hybrid Deep Learning-Based Intrusion Detection System," *Telematics and Informatics Reports*, vol. 10, p. 100053, Jun. 2023.

[15] J. Ghadermazi, S. Hore, A. Shah, and N. D. Bastian, "GTAE-IDS: Graph Transformer-Based Autoencoder Framework for Real-Time Network Intrusion Detection," *IEEE Transactions on Information Forensics and Security*, vol. 20, pp. 4026-4041, 2025.

[16] L. Mhamdi and M. M. Isa, "Securing SDN: Hybrid autoencoder-random forest for intrusion detection and attack mitigation," *Journal of Network and Computer Applications*, vol. 225, p. 103868, May 2024.

[17] A. Kumar and V. L. L. Thing, "Evaluating The Explainability of State-of-the-Art Deep Learning-based Network Intrusion Detection Systems," Feb. 2025.

[18] M. Soltani, K. Khajavi, M. Jafari Siavoshani, and A. H. Jahangir, "A multi-agent adaptive deep learning framework for online intrusion detection," *Cybersecurity*, vol. 7, no. 1, p. 9, May 2024.

[19] M. Tavallaei, E. Bagheri, W. Lu, and A. A. Ghorbani, "A detailed analysis of the KDD CUP 99 data set," in *Computational Intelligence for Security and Defense Applications*. IEEE, Jul. 2009, pp. 1-6.

[20] Y. Xue, C. Kang, and H. Yu, "HAE-HRL: A network intrusion detection system utilizing a novel autoencoder and a hybrid enhanced LSTM-CNN-based residual network," *Computers & Security*, vol. 151, p. 104328, Apr. 2025.

[21] E. Mushtaq, A. Zameer, M. Umer, and A. A. Abbasi, "A two-stage intrusion detection system with auto-encoder and LSTMs," *Applied Soft Computing*, vol. 121, p. 108768, May 2022.

[22] J. McHugh, "Testing Intrusion Detection Systems: A Critique of the 1998 and 1999 DARPA Intrusion Detection System Evaluations as Performed by Lincoln Laboratory," *ACM Transactions on Information and System Security*, vol. 3, no. 4.

[23] N. Moustafa and J. Slay, "UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set)," in *MilCIS*. IEEE, Nov. 2015, pp. 1-6.

[24] I. Sharafaldin, A. Habibi Lashkari, and A. A. Ghorbani, "Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization:," in *Information Systems Security and Privacy*, 2018, pp. 108-116.

[25] I. Sharafaldin, A. Gharib, A. H. Lashkari, and A. A. Ghorbani, "Towards a Reliable Intrusion Detection Benchmark Dataset," *Software Networking*, vol. 2017, no. 1, pp. 177-200, 2017.

[26] M. A. Talukder, M. M. Islam, M. A. Uddin, K. F. Hasan, S. Sharmin, S. A. Alyami, and M. A. Moni, "Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction," *Journal of Big Data*, vol. 11, no. 1, p. 33, Feb. 2024.

[27] G. Engelen, V. Rimmer, and W. Joosen, "Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study," in *2021 IEEE Security and Privacy Workshops (SPW)*. IEEE, May 2021, pp. 7-12.

[28] Z. Pelletier and M. Abualkibash, "Evaluating the CIC IDS-2017 Dataset Using Machine Learning Methods and Creating Multiple Predictive Models in the Statistical Computing Language R," vol. 5, no. 2.