# FREIDA: a concrete tool for reproducible evaluation of IDS using a data-driven approach

Solayman Ayoubi, Sébastien Tixeuil, Gregory Blanc, Houda Jmila

► To cite this version:

Solayman Ayoubi, Sébastien Tixeuil, Gregory Blanc, Houda Jmila. FREIDA: a concrete tool for reproducible evaluation of IDS using a data-driven approach. Risks and Security of Internet and Systems: 19th International Conference (CRISIS), Nov 2024, Aix-en-Provence, France. pp.40-55, 10.1007/978-3-031-89350-6_3 . hal-05048479

HAL Id: hal-05048479

https://hal.science/hal-05048479v1

Submitted on 28 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# FREIDA: A Concrete Tool for Reproducible Evaluation of IDS using a Data-driven Approach

Solayman Ayoubi¹[0000-0001-5711-4402], Sébastien Tixeuil¹⁴[0000-0002-0948-7172], Gregory Blanc²[0000-0001-8150-6617], and Houda Jmila³[0000-0002-4864-5380]

¹ Sorbonne University, CNRS, LIP6, F-75005 Paris, France

² SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, France

³ Institute LIST, CEA, Paris-Saclay University

⁴ Institut Universitaire de France, Paris, France

**Abstract.** Intrusion Detection Systems (IDS) are critical for securing future communication systems, yet existing evaluation methods lack standardization, resulting in incomplete and unreliable assessments. In particular, the evaluation of machine-learning (ML) based IDS often boils down to the demonstration that the IDS performs well on a given dataset, regardless of the dataset quality. Prior evaluation approaches lack formalization and disregard ML best practices. This paper addresses this challenge by presenting FREIDA, a concrete tool for ensuring completeness, reliability, and reproducibility of ML-based IDS evaluations. This tool emphasizes the relationship between evaluation choices and data selection, requiring the generation of purpose-specific datasets. In this paper, we present and provide a Python implementation of our evaluation tool, that is used to evaluate multiple models in a variety of settings. This research represents a crucial stride toward standardizing IDS evaluation methods. Indeed, the proposed tool facilitates the systematic evaluation of IDS by producing a generated configuration file, thereby enhancing the reproducibility of each evaluation.

**Keywords:** Intrusion Detection System · Machine learning · Data-driven Evaluation · Evaluation Framework

## 1 Introduction

The number of Internet subscribers (5.7 billion users in 2023) and the number of connected devices are expected to grow along the increasing adoption of the 5th generation of mobile communications (5G), resulting in 1.4 billion mobile devices being 5G capable [1]. The conjunction of the volume of exposed systems and their low security contributes to the number of breaches skyrocketing between the first halves of 2018 and 2019 (776% growth in attacks between 100 Gpbs and 400 Gbps). While the number of DDoS attacks has doubled, between 2018 and 2023, reaching 15.4 million, new technological trends also increased cyber risk leading to more diverse threats [1]. It is therefore crucial to detect these threats as early and accurately as possible, even more in mission-critical contexts.
Over the past two decades, the field of intrusion detection research has predominantly focused on the development of anomaly-based solutions backed with machine learning (ML). This focus contrasts with the prevalent use of signature-based solutions in the industry, thereby creating a paradox [14]. This can be attributed to the scarcity of high-quality datasets, the inadequacy of the evaluation methodology (especially in terms of metrics), and the limited explainability of the deep-learning approaches utilized. This observation is shared by other practitioners [6,16], and recommendations have been issued to help researchers design a pragmatic assessment methodology for machine learning-based network intrusion detection systems (NIDS) [2]. However, no methodology has been formally specified since that of Milenkoski et al. [11].

While several researchers have expressed interest in evaluating IDS in recent years, very few suggested expanding classical evaluation techniques to specifically address ML-based IDS. Among these techniques, Ayoubi et al. [4] designed a comprehensive yet abstract evaluation framework for machine learning-based intrusion detection systems (IDS). Their approach goes beyond evaluating intrusion detection performance and aims to propose standard data manipulation methods to tackle robustness and stability. The main downside of the approach of Ayoubi et al. [4] is its lack of practicality, as it merely presents an abstract methodology without offering a substantive proposal or solution. By contrast, this paper offers a precise and concrete implementation of their abstract framework. Specifically, we provide a Python implementation of the framework, facilitating the entire evaluation process of ML-based IDS models. The practicality of our implementation, called FREIDA, is validated through real experiments encompassing diverse models and datasets. Additionally, our tool facilitates the efficient and straightforward reproduction of any evaluation, streamlining comparisons among different models. With meticulous step-by-step configuration, the tool faithfully replicates processes while upholding best practices in model evaluation, a feature often lacking in existing evaluation methods from the literature.

The remainder of the paper is structured as follows: Sec. 2 navigates through prior methodologies employed for the evaluation of intrusion detection systems. Sec. 3 presents the practical implementation of the tool, followed in Sec. 4 by its validation through experimentation. Sec. 5 contrasts our contributions with respect to two works of reference. Finally, we conclude in Sec. 6.

## 2 Related Work

Over time, a multitude of IDS evaluation methodologies have been proposed by researchers. In this context, we will be describing a few of these approaches.

Ulvila and Gaffney [17] proposed an innovative methodology for evaluating Intrusion Detection Systems (IDS) that enhances the Receiver Operating Characteristic (ROC) curve through the application of cost analysis techniques. This approach considers both individual and combined IDS scenarios, providing a cost metric to ascertain the optimal operation of an IDS. The methodology facilitates
the computation of the optimal operating point, choosing the best parameters for the least expected cost, for one or multiple IDS. Furthermore, it clarifies the process of representing a composite IDS as equivalent to a single IDS and analyzes IDS using the expected cost metric. Overall, their method proposes to extend the ROC curves with a cost analysis to compute an expected cost. However, as Gu et al. [8] point out, cost measures in security are often subjective, which makes it impossible to evaluate IDSs objectively.

In 2006, Gu et al. [8] introduced a novel metric for evaluating IDS. The proposed metric, denoted as $C_{ID}$ (Intrusion Detection Capability), aims to quantify the effectiveness of an IDS in classifying events. Defined as the ratio of the mutual information between the IDS input and output to the entropy of the input, $C_{ID}$ encompasses all pertinent aspects of detection performance, including both false positives and true positives. This metric exhibits greater sensitivity compared to traditional measures, thereby enabling the detection of even marginal improvements in an IDS. The $C_{ID}$ metric simplifies the evaluation and comparison of IDSs by selecting the IDS with the highest value. Additionally, it facilitates the determination of the optimal operating point for an IDS. In this paper, the authors depict IDS input and output as binary values: 0 for normal traffic and 1 for intrusion. However, this binary representation lacks precision when calculating measurements for various attacks, particularly in scenarios where our IDS functions as a multiclass classifier.

Chua and Salam [7] primarily focus on the evaluation of the long-term performance of ML-based IDS. To achieve this, they suggest assessing the ML-based IDS using a test that was developed after the training set, thus eliciting concept drift in the data. Given that the test set encapsulates variations in the attack type and changes in the network infrastructure over time, the proposed method offers a more comprehensive evaluation of the long-term performance of an ML-based IDS. They put forth two experiments involving multiple datasets. In the first experiment, the CIC-IDS2017⁵ dataset is employed for training, while the CSE-CIC-IDS2018⁶ dataset is utilized for testing. In the second experiment, they use the LUFlow dataset, with data collected in July 2020 serving as the training dataset and data collected in January 2021 used as the testing dataset. In this method, both datasets must contain the same classes for effective model learning. Consequently, the authors opted to consolidate all attack types into a single malicious label. While this simplifies the process, it limits our ability to evaluate the models' classification capabilities comprehensively.

In 2019, Zoppi et al. [18] proposed RELOAD, a framework designed to facilitate the evaluation of anomaly detection algorithms. RELOAD simplifies the evaluation of anomaly detection algorithms through a structured process involving data preparation, where it can load data from various sources (e.g., text files, MySQL databases), apply feature selection techniques (e.g., variance, Pearson correlation), select algorithms to be evaluated (e.g., K-Means, SVM, kNN), and compute performance metrics (e.g., precision, recall, F1-score, ROC-AUC).

<footnote n="5">⁵ https://www.unb.ca/cic/datasets/ids-2017.html</footnote>

<footnote n="6">⁶ https://www.unb.ca/cic/datasets/ids-2018.html</footnote>
Overall, through RELOAD's intuitive GUI, users can efficiently run anomaly detectors with pre-selected features on a given dataset, and obtain performance metrics. However, similar to other tools, RELOAD focuses solely on assessing the effectiveness of IDS, and does not provide automated evaluation of various other relevant IDS properties.

In 2022, Ayoubi et al. [4] proposed a theoretical framework for evaluating IDS properties, drawing on Milenkoski et al.'s [11] approach. This framework emphasizes the critical role of selecting appropriate datasets and metrics to effectively assess various IDS properties. Their evaluation strategy aims to ensure high-confidence results free from experimental bias, aligning with Apruzzese et al.'s [2] definitions. Furthermore, Ayoubi et al. highlight the importance of meticulous dataset construction and the selection of metrics. They advocate for a refinement step to address potential dataset issues and propose several metric families to align with evaluation objectives. Their described testing and training pipeline underscores the need for explainability analysis in IDS evaluation, ensuring comprehensive and transparent assessments.

In summary, while the framework proposed by Ayoubi et al. [4] is arguably the most comprehensive for evaluating ML-based IDS, it remains largely theoretical and is defined informally. It lacks tangible workflow examples for assessing specific properties and does not provide a practical implementation to validate its applicability in real-world scenarios.

## 3 Implementation of the Evaluation Pipeline

In this section, we unveil our Python implementation<fnref n="7" /> of the tool. We organize the pipeline around three components (Dataset, Model, Evaluation) following Ayoubi et al.'s framework [4] (see Fig. 1). Our tool leverages libraries commonly used in the ML community such as Scikit-learn<fnref n="8" /> for preprocessing, data splitting, and similar tasks, while leveraging the Adversarial Robustness Toolbox (ART)<fnref n="9" /> for the generation of adversarial examples.

The tool generates and uses a configuration file (see Lst. 1.1). This file also guarantees evaluation reproducibility and is an outcome of our tool's evaluation process. Consequently, in addition to meticulously documenting each evaluation stage, this file enables precise replication of the evaluation by fellow researchers. Such fidelity allows for the validation of results on identical models or facilitates the application of the same evaluation methodology to diverse models, thereby enabling accurate comparisons of IDS.

In the configuration file, the *seed* field is the first entry. It is used to initialize the internal random number generator, essential for tasks like data set shuffling and partitioning into training and test sets and so on. Explicitly defining this value enables the faithful reproduction of random outcomes. We emphasize the importance of integrating the *seed* to ensure deterministic results when running

<footnote n="7">⁷ Available at https://gitlab.com/asolayman/freida/</footnote>

<footnote n="8">⁸ https://scikit-learn.org/</footnote>

<footnote n="9">⁹ https://github.com/Trusted-AI/adversarial-robustness-toolbox</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1: Equation pipeline for effectiveness evaluation</figcaption></figure>

algorithms that involve randomness, providing a reproducible evaluation. The configuration file also necessitates a property array that aligns with the desired evaluation properties. This array can encompass one or more properties, currently including effectiveness and robustness.

**Feature Selection.** At the outset, the configuration outlines the *features_selection* step, integral to the overarching **Dataset Construction** module. Within the configuration, the *features_selection* field permits the specification of the chosen selection method. In this example (see Lst. 1.1), we choose the **Lasso** feature selection method. By training a **Lasso** regression model with our data, we select only the most informative features based on a chosen threshold, defined by the field *alpha_lasso* in the configuration. By concentrating on the most significant features, this feature selection method improves model interpretability while reducing dimensionality.

**Data Preprocessing.** As part of our data preprocessing workflow, we carefully plan a sequence of transformations to get our datasets ready for machine learning. Several fields in the configuration file allow you to configure and fine-tune the preparation process. The *preprocessing* field defines the data scaling method used in the evaluation. For example, we can use the **StandardScaler** method defined in the example configuration file, which allows us to center the dis-
tribution of each feature around zero with a standard deviation of one. This approach diminishes the impact of outliers and facilitates the comparability of features on a standardized scale. Two additional fields are integral to the pre-processing phase. The first field, *drop_columns*, instructs the framework which data columns to exclude. This step is crucial, particularly when certain data, such as identifiers or labels, should not be utilized for training. The second field, *label_column*, specifies the location of the label within the dataset. Moreover, in datasets with multiple columns and labels for binary or multiclass classification, this field can also inherently indicate the type of task being evaluated. The *class_to_remove* field is used to remove all rows in the dataset where the class matches the specified value; the dataset must include a column with multiple classes. This field is used when we evaluate the effectiveness in an open-world scenario. Indeed, in this scenario, we train the model without this class and we evaluate with this unknown class in the test dataset. This step is also part of the **Dataset Construction** module.

***Adversarial Data Augmentation.*** When assessing the model’s robustness, it is imperative to incorporate all utilized attacks into the configuration file. Thus, the *attacks* field within the configuration file delineates the selected attacks employed during evaluation. Notably, this field allows for the specification of various types of evasion attacks. Furthermore, we advocate for the capability to assess both types of evasion attacks: white-box and black-box. But it is also possible to specify a poisoning attack. The key distinction lies in the nature of these attacks: evasion attacks alter the test dataset, whereas poisoning attacks manipulate the training dataset.

In scenarios where white-box compatibility is not guaranteed, we resort to training surrogate models to approximate the target model’s behavior, enabling the application of white-box attacks through the surrogate’s interface [10].

Additionally, for black-box attacks, we leverage model-agnostic attack techniques to probe model vulnerabilities without relying on the internal model structure. By systematically exploring a diverse range of attack strategies, we gain valuable insights into model weaknesses and fortify our defenses against potential adversarial threats. In our current implementation, we use the **Fast Gradient Sign Method (FGSM)** attack as a white-box attack and the **Zeroth-Order Optimization (ZOO)** attack as the black-box counterpart. These selections are strategic, aiming to comprehensively evaluate a model’s robustness against a spectrum of adversarial threats.

***Dataset Evaluation.*** This step sheds light on our approach to ensure data quality and integrity throughout our evaluation. This step is conducted on a non-split dataset, which the framework will split if necessary. However, if the dataset supplied is already split, the framework can still perform the evaluation. In the configuration file, the field *dataset_evaluation* defines the evaluation methods we wish to use. Currently, the sole dataset evaluation method we implement involves detecting null values within our dataset, a basic yet essential step towards adhering to best practices.
In the future, we plan to enhance dataset quality evaluation by integrating additional methods, such as the metrics proposed by Schoen et al. [13]. Their comprehensive approach for assessing synthetic legitimate network traffic quality will be particularly useful for properties requiring benign traffic data generation.

*Dataset Refinement.* Within our refinement method, we embark on a systematic process to address any identified problems. Similar to the *dataset_evaluation* step, this process is delineated in the configuration through the *dataset_refinement* array specifying the solution to be executed for each problem. Leveraging the evaluation results obtained from our prior dataset evaluation, we discern whether null values are present within our datasets. For now, our framework implemen- tation only addresses this aspect of dataset evaluation, but over time, we plan to incorporate more sophisticated evaluation methods. It might be for example related to the base rate fallacy [3]. Indeed, if the binary classes *benign* and *ma- licious* are too balanced, the refinement step can reduce the *malicious* class to better reflect the base rate.

*Dataset Splitting.* In the case the provided dataset is not already split, we can instruct the dataset splitting process through two fields. The first field, *splitting*, indicates the chosen split method. For example, we can choose an **imbalanced** split method to reproduce more realistic class proportions. Furthermore, other splits could be proposed, particularly to simulate concept drift, enabling the evaluation of several potential scenarios. The second field, *test_size*, simply allows defining the percentage of data to be used for testing.

*Training and Evaluation.* There is no need to detail the methods used for *Training* and *Evaluation* since there is no specific method. However, a pertinent field related to the training process exists. The *is_pretrained* field simply informs the framework whether the input model is pre-trained, enabling, in certain scenarios, direct evaluation without necessitating training. This feature facilitates the evaluation of deployed models, among other applications.

# 4 Experiments

## 4.1 Tool Description

In this section, we elaborate on the usage of the FREIDA evaluation tool. To illustrate this, we describe how to reproduce evaluations that are delineated in the subsequent section 4.2.

To utilize the tool, the first step is to clone the GitHub repository and install it as a Python module. Then, initiating an evaluation is straightforward. A new python file should be created and the module imported. Preparing the assessment is done following two steps. The first step is to create a dictionary containing the evaluation's configuration elements. This dictionary can either be sourced from a configuration file provided by another user, or defined directly
<figure><img src="image_3.png" /><figcaption>Code Listing 1.1: Example of config file</figcaption></figure>

in the Python code. For example, to reproduce the evaluation presented in this paper, the example configuration provided (see Lst.1.1) can be used. The second step is to load our data from a CSV file using, for example, Python's `pandas`<fnref n="10" /> library, which allows for efficient data manipulation and analysis, followed by instantiating our models using `scikit-learn` for traditional machine learning algorithms and/or `Keras` for deep learning models. This involves reading the CSV into a `Frame`, and then creating instances of the desired models with appropriate configurations. Then, we call `FREIDA` evaluation function, passing as parameters the list of models, the dataset, and the configuration specifying the evaluation steps.

After this initial evaluation, one has the flexibility to modify the evaluation process according to personal preferences, while maintaining a record of each evaluation result. For every evaluation conducted, a folder is generated containing both the results and the configuration file. The potential modifications to the evaluation process are vast: we might choose to assess only the effectiveness property, or focus solely on evaluating robustness against the FGSM attack, among other possibilities. Indeed, beyond evaluating specific properties or attacks, we might desire to adapt the data preprocessing method or even alter the classification type entirely. For instance, we could substitute the label column with a column containing multiclass labels, thereby customizing the evaluation process to suit our evolving requirements.

The objective remains to enable personalized assessments while adhering to best practices in model evaluation. Furthermore, in scenarios where applying the same method (e.g., preprocessing step) to all models is not appropriate, we can create multiple configuration dictionaries for each model. By doing so, we

<footnote n="10">¹⁰ https://pandas.pydata.org/</footnote>
can call the evaluation function on each model with the respective configuration,
ensuring that each model undergoes the appropriate evaluation process.

Technicalities about function names and parameter orders, as well as example
code, can be found in the companion GitHub repository.

## 4.2 Results

In this section, we will systematically compare and contrast multiple models to
ascertain their relative robustness and effectiveness through our implemented
tool.

We carefully examined the effectiveness and robustness of various machine
learning classifiers, evaluating their ability to withstand various attack scenarios
generated by adversarial examples created using the ZOO and FGSM techniques.
We additionally assess the robustness of the models against a basic poisoning
attack, involving the injection of noise into the training data.

Furthermore, we carried out these evaluations according to the implemen-
tation described above, using 10% of UNSW-NB15 [12] (randomly selected but
reproducible thanks to the seed), a portion of the CIC-IDS2017 dataset, and the
full NSL-KDD dataset. All results are easily reproducible from the tool and the
configuration file linked to this evaluation.

The first heatmap (see Fig. 2) shows the classifiers’ performance range on
the baseline, non-adversarial, dataset. Our evaluation reveals subtle differences
in classifier performance across the various metrics presented previously. In this
experimentation, we choose to assess several classifiers: AdaBoost, Bagging, De-
cision Tree, Gradient Boosting, Logistic Regression, Random Forest, Support
Vector Classifier, a Multilayer Perceptron, a Convolutional neural network, and
an LSTM model. These models have been implemented, with the Scikit-learn li-
brary for most, while the Deep Neural Network (MLP, CNN, and LSTM) stands
out with their implementation through the Keras<fnref n="11" /> library. This strategic selec-
tion of models not only underscores the versatility of FREIDA but also showcases
its proficiency in evaluating a diversity of models.

The initial results, without adversarial attacks, are as follows: for the UNSW-NB15, RF demonstrated the best performance with 90.1% accuracy, 91.2% balanced accuracy, and 0.792 MCC, particularly excelling on metrics for imbalanced datasets. LR and MLP achieved accuracies of 83.1% and 89.9%, respectively. AdaBoost underperformed with a score of 74.6%. In the case of NSL-KDD, AdaBoost achieved 82% accuracy, 82.2% F1-score and 0.671 MCC. LR performed comparably with an accuracy of 80.2%. MLP reached 80.0% accuracy, while RF had lower performance with 73.7% accuracy and a 71.0% F1-score. On the CIC-IDS dataset, All models showed exceptional results, with accuracy, ROC AUC, and balanced accuracy at 99.8%, and MCC at 0.995, across all metrics.

The tool’s added value concerning these results is twofold: firstly, they are
easily reproducible and the configuration used by the tool is also understandable
by the user, as the config file is interpretable. Secondly, they stem from an

<footnote n="11">¹¹ https://keras.io/</footnote>
evaluation adhering to best practices in IDS evaluation. This ensures a reliable
evaluation process without overlooking any crucial steps.

The second heatmap (see Fig. 3) shows the classifiers' performance in an
open-world scenario. In this scenario where we train our models without a se-
lected class, the results indicate that the Baseline models generally performed
better than the Open-World models.

In the case of UNSW-NB15, **RF** showed a slight accuracy drop to 89.3%, with precision improving to 97.2%. **LR** reached 82.6% accuracy and 77.2% recall, showing minor improvements. **Bagging** and **DT** saw significant accuracy declines, dropping to 58.1% and 59.5%, respectively, highlighting the challenges of the Open-World scenario. On NSL-KDD, **RF** accuracy dropped sharply to 61.6% with a precision of 96.3% but a low recall of 34.4%. **LR** maintained solid performance with 75.7% accuracy and 59.7% recall. **Bagging** and **DT** also declined, with accuracies of 60.5% and 71.6%, respectively. Models based on neural networks, like MLP, CNN, and LSTM, showed better adaptation to the Open-World scenario. For CIC-IDS2017, **LR** achieved 91.3% accuracy, with high precision (99.4%) and recall (76.1%). **MLP** performed best with 92.6% accuracy and a recall of 79.8%. **DT** followed closely with 90.0% accuracy. **AdaBoost** and **GB** had lower accuracies around 84%. Lower recall was observed in **AdaBoost** (55.8%) and **RF** (54.7%), indicating areas for improvement.

The robustness heatmap (see Fig. 4, 5) carefully evaluates the classifiers'
performance in the presence of adversarial instances, paying close attention to
how they react to FGSM and ZOO attacks. We are also evaluating the robustness
of the models against potential training data poisoning attacks. Our results (see
Fig. 6) show how these attacks affect the classifiers' performance metrics.

In the case of UNSW-NB15, **RF** maintained robustness under FGSM attacks with accuracy (81.7%), precision (89.8%), recall (82.3%), and MCC (0.602). **Bagging** also showed resilience, achieving 78.9% accuracy and MCC of 0.557. In contrast, models like **LR** and **SVC** exhibited significant performance drops under FGSM attacks. But for NSL-KDD, **RF** demonstrated moderate performance under adversarial conditions with 54.0% accuracy and MCC of 0.194, while **Bagging** and **GB** showed better robustness, with accuracies of 69.0% and 72.1%, respectively, and MCC values of 0.386 and 0.452. Models like **LR** (45.5% accuracy, MCC -0.163) and **LSTM** (32.2% accuracy, MCC -0.394) experienced severe performance drops. On CIC-IDS2017, all classifiers struggled under adversarial conditions. **RF** had 64.2% accuracy with an MCC of 0.0. **Bagging** and **AdaBoost** achieved around 62% accuracy but had extremely low F1-scores (0.003 and 0.0, respectively). **LR** performed the worst, with 10.7% accuracy and an MCC of -0.797. Neural models like **MLP** and **CNN** also struggled, with accuracies of 40.1% and 32.8%, revealing significant vulnerabilities to adversarial attacks.

Against the ZOO attack on the UNSW-NB15 dataset, **RF** accuracy dropped to 52.3%, while **AdaBoost** achieved the highest accuracy at 66.7%. The vulnerability of **RF** to this black-box attack is linked to the use of a surrogate model for generating adversarial examples, unlike in the FGSM attack, where **RF** showed
more resilience due to the transferability of adversarial examples [9]. While in the case of NSL-KDD, AdaBoost performed best with 78.8% accuracy and an MCC of 0.575. Bagging and DT also performed well, achieving around 72% accuracy and balanced F1-scores. In contrast, LR struggled with only 20.1% accuracy and an MCC of -0.637, highlighting its vulnerability. RF and SVC also showed weaker performance, with accuracies of 61.3% and 41.5%, respectively. For CIC-IDS2017, AdaBoost performed best with 67.9% accuracy. Bagging and DT struggled, achieving around 63% accuracy with low precision and recall. GB performed moderately (65.7% accuracy), but had only 12% recall. LR showed significant weakness with 39.3% accuracy and an MCC of -0.306, while RF had 64.5% accuracy but extremely low recall (1.4%), revealing substantial vulnerabilities.

Finally, against poisoning attacks on UNSW-NB15, several models, including AdaBoost, GB, LR, RF, SVC, and MLP, displayed resilience to perturbations. However, AdaBoost's accuracy dropped from 74.6% to 68%, a minor 8.8% decline. In contrast, DT suffered a drastic drop from 81% to 30%, reflecting its vulnerability. But for NSL-KDD the poisoning attacks significantly impacted most classifiers. AdaBoost, GB, LR, SVC, CNN, and LSTM all dropped to 42.6% accuracy, with no precision or recall. Bagging performed slightly better, achieving 55.5% accuracy, with 56.8% precision and 93.4% recall, though it struggled with low balanced accuracy (48.9%) and MCC (-0.047). RF achieved 44.3% accuracy but had very low recall (4.4%), highlighting the significant effects of poisoning attacks. In the case of CIC-IDS2017, MLP emerged as the top performer with 71.6% accuracy, 100% precision, and 20.7% recall, resulting in an F1-score of 34.3%. Other models, including AdaBoost, GB, LR, RF, SVC, CNN, and LSTM, all scored 64.2% accuracy but failed to recognize positive instances, achieving zero precision and recall. Bagging slightly improved with 69% accuracy and an F1-score of 39.1%, while DT showed the weakest performance, with 48.3% accuracy, 19.2% precision, and 13.8% recall.

Our study underscores the diverse responses of classifiers to adversarial attacks and poisoning, revealing a spectrum of robustness across models. This highlights the criticality of choosing suitable models tailored to security-sensitive applications like IDS. Notably, the RF emerged as a stalwart option, demonstrating resilience in both effectiveness and robustness assessments. Its versatility suggests potential deployment in scenarios demanding fortitude against adversarial incursions.

To conclude this phase of the experiment, we measure the execution time of each model's evaluation, along with the RAM consumption of each assessment, the results of the effectiveness evaluation and the robustness evaluation can be observed in Tab.1. Not surprisingly, the effectiveness evaluation proceeds swiftly as it derives insights from existing data and computes straightforward metrics. However, the robustness evaluation, involving the generation of model-based attacks, can be notably time-intensive.

In the previous sections, we meticulously outlined our implementation, FREIDA, of Ayoubi et al.'s abstract framework [4]. We also employed it to assess the effec-
<figure><img src="image_4.png" /><figcaption>Fig. 2: Evaluation results of the Baseline effectiveness</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 3: Evaluation results of the Open-World effectiveness</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>Fig. 4: Evaluation results of the robustness on FGSM</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 5: Evaluation results of the robustness on ZOO</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Fig. 6: Evaluation results of the robustness on poisoning</figcaption></figure>

<table><caption>Table 1: Hardware usage for the effectiveness and robustness evaluation on UNSW-NB15</caption><thead><tr><th rowspan="2">Metrics</th><th colspan="10">Effectiveness</th><th colspan="10">Robustness</th></tr><tr><th>AdaBoost</th><th>Bagging</th><th>DT</th><th>GB</th><th>LR</th><th>RF</th><th>SVC</th><th>MLP</th><th>CNN</th><th>LSTM</th><th>AdaBoost</th><th>Bagging</th><th>DT</th><th>GB</th><th>LR</th><th>RF</th><th>SVC</th><th>MLP</th><th>CNN</th><th>LSTM</th></tr></thead><tbody><tr><td>Time (sec)</td><td>0.12</td><td>0.05</td><td><strong>0.02</strong></td><td>0.07</td><td>0.03</td><td>0.26</td><td>2.6</td><td>0.20</td><td>6.6</td><td>13.4</td><td>1719</td><td>352</td><td><strong>90</strong></td><td>158</td><td>133</td><td>1871</td><td>607</td><td>897</td><td>1079</td><td>6497</td></tr><tr><td>RAM (MiB)</td><td>960</td><td>972</td><td>944</td><td>941</td><td>981</td><td>988</td><td>1100</td><td><strong>939</strong></td><td>1067</td><td>1033</td><td>983</td><td>981</td><td>1027</td><td>1026</td><td>949</td><td><strong>885</strong></td><td>1305</td><td>1085</td><td>1066</td><td>1664</td></tr></tbody></table>

tiveness and robustness of various models. This section discusses how FREIDA aligns with the research conducted by Apruzzese et al. [2].

In their paper, Apruzzese et al. [2] describe several criteria crucial for a pragmatic evaluation. Notably, the evaluation results must be devoid of experimental biases and engender a high level of confidence. Furthermore, evaluations should be conducted on testbeds mirroring real-world IDS operational scenarios. Lastly, comprehensive information required for model development must be furnished. In the case of FREIDA, we cannot guarantee to provide a dataset that corresponds to the network in which the IDS is deployed. Instead, FREIDA operates by processing an existing dataset and adjusting it to suit the specific property being evaluated. Consequently, while the dataset may originate from a testbed aligning with the conditions outlined by Apruzzese et al. [2], it may also diverge from these parameters. As we value reproducibility and comparability over practicality, we do not completely cover the conditions of Apruzzese et al. [2]. For a given use case, FREIDA allows to compare several IDS and several configura-
tions, making it easy to reproduce evaluations. Moreover, dataset manipulation
enables to escape the closed world [15].

FREIDA is designed to handle tasks such as binary classification, multiclass classification, and anomaly detection, which are crucial in IDS applications. Indeed, we emphasize that FREIDA is optimized for tabular data, which is typically used in network data for IDS. FREIDA enables the demonstration of the potential generalization of evaluated models. Specifically, it allows for testing a single model with a given configuration and across multiple datasets that share the same features. Similarly, multiple models can be tested on a single dataset while maintaining a consistent data split. By repeating these experiments, we can fine-tune the model's hyperparameters, bringing it closer to its optimal performance point, where it performs well across most datasets. FREIDA simplifies this process, making it less tedious.

We conclude that our tool, FREIDA, effectively satisfies several pragmatic
evaluation conditions outlined by Apruzzese et al. [2]. Its overarching objective
is continuous enhancement, enabling evaluations that consistently adhere to best
practices for IDS assessments.

6 Conclusion

There exists a gap in existing evaluation methodologies for ML-based intrusion
detection systems, where crucial aspects like unbiased, transparent, and repro-
ducible evaluation are often overlooked. To address this, we propose FREIDA,
a concrete tool for comprehensive assessments of ML-based IDS.

In this paper, we present the implementation of the framework defined by Ayoubi et al., exemplifying its applicability by evaluating IDS according to two interesting aspects: their detection effectiveness, and robustness to adversarial attacks. Furthermore, our Python implementation provides a practical resource for researchers engaged in IDS evaluations. It is noteworthy that our tool is designed for ongoing improvement. Notably, elements like explainability and data representation analysis, introduced in the original paper by Ayoubi et al. [4], are yet to be integrated into FREIDA. The implementation of our tool opens new research avenues, including evaluating state-of-the-art IDS, applying the framework to develop robust IDS solutions, and recently proposing an intuitive, user-friendly interface [5].

**Acknowledgements** This work is funded by the GRIFIN project (ANR-20-CE39-0011).

References

1. Cisco Annual Internet Report (2018-2023). Tech. rep., Cisco (2020)

2. Apruzzese, G., Laskov, P., Schneider, J.: Sok: Pragmatic assessment of machine learning for network intrusion detection. arXiv preprint arXiv:2305.00550 (2023)
3. Axelsson, S.: The base-rate fallacy and the difficulty of intrusion detection 3(3) (2000)

4. Ayoubi, S., Blanc, G., Jmila, H., Silverston, T., Tixeuil, S.: Data-driven evaluation of intrusion detectors: A methodological framework. In: Foundations and Practice of Security. pp. 142–157. Springer Nature Switzerland (2022)

5. Ayoubi, S., Tixeuil, S., Blanc, G., Jmila, H.: Demo: Towards reproducible evaluations of ml-based ids using data-driven approaches. In: Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security (CCS ’24). p. 3 (2024)

6. Chou, D., Jiang, M.: Data-driven network intrusion detection: A taxonomy of challenges and methods. ArXiv abs/2009.07352 (2020)

7. Chua, T.H., Salam, I.: Evaluation of machine learning algorithms in network-based intrusion detection system. ArXiv abs/2203.05232 (2022)

8. Gu, G., Fogla, P., Dagon, D., Lee, W., Skorić, B.: Measuring intrusion detection capability: An information-theoretic approach. In: Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security. p. 90–101. ASIACCS ’06, Association for Computing Machinery, New York, NY, USA (2006)

9. Gu, J., Jia, X., de Jorge, P., Yu, W., Liu, X., Ma, A., Xun, Y., Hu, A., Khakzar, A., Li, Z., Cao, X., Torr, P.: A survey on transferability of adversarial examples across deep neural networks. ArXiv abs/2310.17626 (2023)

10. Jmila, H., Khedher, M.I.: Adversarial machine learning for network intrusion detection: a comparative study. Computer Networks 214

11. Milenkoski, A., Vieira, M., Kounev, S., Avritzer, A., Payne, B.D.: Evaluating computer intrusion detection systems: A survey of common practices 48(1), 1–41 (2015), publisher: ACM New York, NY, USA

12. Moustafa, N., Slay, J.: Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In: 2015 Military Communications and Information Systems Conference (MilCIS). pp. 1–6 (2015)

13. Schoen, A., Blanc, G., Gimenez, P.F., Han, Y., Majorczyk, F., Me, L.: A tale of two methods: Unveiling the limitations of gan and the rise of bayesian networks for synthetic network traffic generation. Proceedings of the 9th International Workshop on Traffic Measurements for Cybersecurity (WTMC 2024) (2024)

14. Seng, S., Garcia-Alfaro, J., Laarouchi, Y.: Why anomaly-based intrusion detection systems have not yet conquered the industrial market? In: International Symposium on Foundations and Practice of Security. pp. 341–354. Springer (2021)

15. Sommer, R., Paxson, V.: Outside the closed world: On using machine learning for network intrusion detection. 2010 IEEE Symposium on Security and Privacy pp. 305–316 (2010)

16. Tavallaee, M., Stakhanova, N., Ghorbani, A.A.: Toward credible evaluation of anomaly-based intrusion-detection methods. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 40(5), 516–524 (2010)

17. Ulvila, J.W., Gaffney Jr, J.E.: Evaluation of intrusion detection systems. Journal of Research of the National Institute of Standards and Technology 108(6), 453 (2003)

18. Zoppi, T., Ceccarelli, A., Bondavalli, A.: Evaluation of anomaly detection algorithms made easy with reload. In: 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). pp. 446–455 (2019)