# Privacy benchmarking of intrusion detection sytems

Solayman Ayoubi, Gregory Blanc, Houda Jmila, Sébastien Tixeuil

► To cite this version:

Solayman Ayoubi, Gregory Blanc, Houda Jmila, Sébastien Tixeuil. Privacy benchmarking of intrusion detection sytems. 39th International Conference on Advanced Information Networking and Applications (AINA), Apr 2025, Barcelone, Spain. pp.406-417, 10.1007/978-3-031-87772-8_34. hal-05045414

HAL Id: hal-05045414

https://hal.science/hal-05045414v1

Submitted on 24 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Privacy Benchmarking of
Intrusion Detection Sytems

Solayman Ayoubi¹, Gregory Blanc², Houda Jmila³, and Sébastien Tixeuil¹,⁴

¹ Sorbonne University, CNRS, LIP6

² SAMOVAR, Télécom SudParis, Intitut Polytechnique de Paris

³ LIST, CEA, Paris Saclay University

⁴ Institut Universitaire de France

**Abstract.** Network-based Intrusion Detection Systems (NIDS) are crucial in safeguarding network security, especially as cyber threats continue to evolve in complexity and scope. Despite significant advancements in IDS development, the evaluation of these systems remains inconsistent and often inadequate, particularly concerning their resilience to privacy attacks. This paper addresses this critical gap by introducing a systematic approach to assess the privacy vulnerabilities of IDS. We implement and integrate our evaluation method into the FREIDA [4, 5] tool, which is specifically designed to ensure the completeness, reliability, and reproducibility of machine learning-based IDS evaluations. To validate our approach, we conduct extensive experiments using established datasets, demonstrating the effectiveness and reliability of our evaluation methodology.

# 1 Introduction

As Internet subscribers and connected devices increase, inadequate security has led to a surge in cyber breaches, further elevated by emerging technologies [1]. Early and accurate threat detection is crucial, especially in mission-critical contexts, to protect sensitive data, maintain operational integrity, and ensure system resilience against evolving cyber threats. In this context, significant progress has been made in developing more effective intrusion detection systems (IDS). However, a critical gap remains in the thorough and systematic evaluation of these systems. Recognizing this need, Ayoubi et al. [3] proposed a comprehensive framework for IDS evaluation, identifying several key properties that require rigorous assessment. Building on their work, we aim to focus on one crucial property: the privacy preservation of IDS. Our objective is to evaluate how robust IDS are against common privacy attacks.

While recent research [18] extensively explored the impact of privacy attacks on machine learning (ML) models, there remains a notable gap in examining how these attacks specifically target IDS. Indeed, the threat model for IDS is inherently different from that of general machine learning applications. In fact, IDS also typically handles sensitive data and may expose it should the attack be successful, it is thus crucial to rigorously assess their resilience to various
privacy threats. This paper aims to address this critical need by systematically analyzing the vulnerability of IDS to privacy attacks.

To provide a streamlined and reproducible evaluation method, we choose to integrate our evaluation method into the FREIDA toolchain [4, 5]. FREIDA is specifically designed to ensure the completeness, reliability, and reproducibility of machine learning-based IDS evaluations. By leveraging FREIDA, our solution not only simplifies the privacy robustness evaluation process, but also guarantees consistent and reliable assessments across different settings. Additionally, the code of our FREIDA extensions and the evaluation results presented in this article are made publicly available to the scientific community<fnref n="5" />.

The remainder of the paper is structured as follows: Sec. 2 provides a comprehensive review of existing research on privacy robustness related to IDS. Sec. 3 presents our evaluation methodology, followed in Sec. 4 by its validation through experimentation. Finally, we conclude in Sec. 5.

## 2 Related Work

In 2017, Niksefat et al. [15] proposed a study of privacy concerns in IDS by introducing a novel taxonomy that categorizes privacy issues into three main sources: IDS input data (such as network traffic and system logs), IDS built-in data (such as attack signatures), and IDS generated data (such as alerts and reports). The authors survey and evaluate existing privacy-preserving techniques, such as pseudonymization, cryptographic protocols, and homomorphic encryption, highlighting their impact on IDS performance and precision. The work emphasizes the need for a holistic approach to address privacy while ensuring effective intrusion detection, and it identifies challenges and future research directions, particularly in quantifying privacy preservation and protecting non-identity-related data.

Later, Arshad et al. [2] proposed a thorough review of IDS for Internet of Things (IoT) networks, focusing on performance, energy consumption, and privacy implications. The authors analyze the key challenges of implementing effective IDS in IoT environments, such as resource constraints, attack complexity, and the scarcity of relevant security data. The review categorizes existing IDS approaches by detection methods, emphasizing the need to balance detection accuracy with performance overhead and energy efficiency while preserving privacy. The study proposes a comprehensive attack model for IoT systems.

The two aforementioned articles focus on privacy concerns that arise when data is shared between entities, such as in collaborative IDS. In these systems, an adversary may compromise a node to access sensitive data. The studies emphasize protecting information through techniques such as anonymization as it flows between nodes. However, they primarily address privacy risks related to data sharing, leaving open the study of attacks that directly target the IDS models.

<footnote n="5">⁵ https://anonymous.4open.science/r/freida-90DB/ (Don't clone, use the website)</footnote>
Most of the works [6, 8, 10, 17, 24] about privacy preserving intrusion detection system are in the context of federated learning [13], where a global model is constructed out of many contributed local models by a server, so that the local data used to train the local model is not sent to the server that aggregates all local models into a global one.

As federated learning is typically seen as a privacy-enabling technology, attacks on IDS that use the federated learning framework are typically launched during the training phase, e.g. poisoning local training data or model to be sent to the aggregating server. Instead, we would like to focus on the privacy properties of the (global) final trained model itself, without any knowledge about the training process (be it federated or not).

A few recent works address privacy issues in the context of centralized IDS, using e.g. differential privacy [7, 12], homomorphic encryption [7], generative adversarial networks [21]. However, the evaluation of the approaches usually considers the effectiveness of the IDS model (that is, its ability to detect attacks) rather than the privacy properties that are guaranteed. Only proposals that are based on differential privacy evaluate the privacy gains of the approach, yet consider a single kind of privacy attack, namely membership inference.

Instead, we would like to consider privacy guarantees as a primary evaluation criterion for IDS, and consider such attacks in a global evaluation framework, so that it becomes possible to compare several IDS with respect to the impact of various privacy attacks.

# 3 Methodology

## 3.1 Threat Model

We first define the threat model that underpins our analysis of privacy attacks on ML-based IDS. Specifically, we consider two types of privacy attacks: **Membership Inference** and **Model Extraction**. These attacks exploit the vulnerabilities of the machine learning models used within IDS, potentially leading to significant privacy breaches for both the IDS itself and the network whose data is being analyzed.

**Membership Inference Attack.** This attack occurs when an adversary tries to determine whether a specific data instance was part of the training dataset used to develop the IDS model. Such attacks can reveal sensitive insights about the network. For instance, if traffic patterns are exposed, adversaries could infer which applications are being used, the volume of traffic, or even the number of users on the network. In some cases, revealing this data could expose operational details, such as peak usage times, preferred protocols, or even the identity of certain users. This makes the network more vulnerable to targeted attacks, as the adversary can exploit the revealed patterns to craft more precise and damaging strategies. For example, Yeom et al. [25] show that membership inference attacks can expose model overfitting, which attackers can exploit for adversarial purposes.
**Model Extraction Attack.** This attack involves an adversary attempting to reconstruct or approximate the underlying ML model used by the IDS. The attacker typically queries the IDS with a large number of inputs and analyzing the outputs, to learn the decision boundaries and potentially replicate the model. This can lead to intellectual property theft and can also enable more sophisticated attacks against the IDS. For example, Tramèr et al. [23] demonstrated that extracting a model and using it for a model inversion attack requires fewer queries and less time than performing a direct black-box inversion.

## 3.2 Formalism

In this subsection, we rigorously outline the methodology for constructing the dataset $D_{\text{attack}}$, this dataset is specifically designed to train the model for executing a privacy attack. For each type of privacy attack, we develop a tailored model trained on a corresponding dataset, which defines the nature of the attack. Our formalization outlines the methodology for constructing the dataset $D_{\text{attack}}$ for each specific attack presented in this study. This formalization for each selected attack scenario ensures consistency and reproducibility.

The formalism is divided into several subsections, each tailored to address specific aspects of the dataset construction process. These subsections not only provide a detailed conceptual framework for generating $D_{\text{attack}}$ but also highlight the flexibility in implementing various attack strategies. Additionally, each subsection explores the specific implementation strategy we adopted within the FREIDA framework, offering insights into our practical approach.

**Membership Inference** The first attack we formalize is the membership inference attack. As previously outlined, the objective of this attack is to ascertain whether a specific sample was included in the training dataset. The process of creating $D_{\text{attack}}$ is defined as follows.

**Definition 1 (Membership Inference Attacks).** Let $D := \{(x_i, y_i) | x_i \in \mathcal{X}^P, y_i \in \mathcal{Y}\}_{i=1}^N$ be a dataset with the same distribution as the target dataset, where $P$ represents the dimensionality of the vectors, $\mathcal{X}$ is the input space and $\mathcal{Y}$ the output space, $\mathcal{Y} = \{0, 1\}$ for binary classification. Let $\hat{h}$ be a non-trained model chosen according to attacker knowledge and used to label $D_{\text{attack}}$. Let $D_{\text{attack}}$ be the output dataset used to train our attack model. We define the process of creating a dataset to perform membership inference attacks as follows:

$$D_{\text{train}}, D_{\text{test}} = \text{Split}(D) \qquad (1)$$

$$D_{\text{train}}, D_{\text{test}} \subset D, \quad D_{\text{train}} \cap D_{\text{test}} = \emptyset \qquad (2)$$

Split is a generic function that takes a dataset as input and divides it into two separate datasets, typically used for creating training and testing sets in.

$$\text{Data labeling: } y_{\text{attack}}(x) = \begin{cases} 1 & \text{if } x \in D_{\text{train}} \\ 0 & \text{if } x \in D_{\text{test}} \end{cases} \qquad (3)$$
$$ \hat{h}_{\theta} = \text{Train}(D_{\text{train}}, \hat{h}) \qquad (4) $$

Train is a generic function that takes a training dataset as input and uses it to train a model. Let $\hat{h}_{\theta}$ be the model trained.

$$ D_{\text{attack}} := \{(\hat{h}_{\theta}(x_i), y_{\text{attack}}(x_i)) \mid \forall x_i \in D\}. \qquad (5) $$

$$ \text{MI}: (\mathcal{X}^P \times \mathcal{Y}) \times (\mathcal{X}^P \to \mathcal{Y}) \to (\mathcal{Y} \times \mathbb{B}) \qquad (6) $$

From this definition, we infer that to train an attack model capable of performing a membership inference attack, it is essential to possess a dataset $D_{\text{attack}}$ composed of the outputs from a model $\hat{h}_{\theta}$, along with labels indicating whether each output originates from the training set or not. This attack can be implemented through various approaches; however, we have chosen to adopt the shadow model technique proposed by Shokri et al. [20].

A shadow model approach is used to infer whether a specific data point was included in the training dataset of a target model. The process begins by creating a set of shadow models that are trained on datasets with similar distributions to the target model's training set. These shadow models are utilized to generate a dataset comprising both members (samples present in the training set) and non-members (samples absent from the training set), using an input dataset with the same distribution as the target dataset. For each shadow model, Shokri et al. [20] extract output probabilities for both member and non-member samples, capturing the behaviour of the model in response to various inputs. Subsequently, these outputs are used to train a classifier that distinguishes between the two classes based on the model's prediction confidence. Finally, the attack is executed against the target model by querying it with unknown samples, and the classifier predicts whether each queried sample is a member of the target model's training set based on the model's output probabilities.

**Model Extraction** The last attack we formalize is the model extraction attack. As previously stated, this attack aims to reconstruct or approximate the underlying machine learning model utilized by the IDS. The process of creating $D_{\text{attack}}$ for this attack is defined as follows.

**Definition 2 (Model Extraction Attacks).** Let $D$ be the initial dataset. We define the process of creating a dataset $D_{\text{attack}}$ to perform model extraction attacks on a target model $h_{\theta}$ (we can do queries to the target model) as follows:

Let $\mathcal{Q} \subseteq \mathcal{X}^P$ be the set of queries made to the target model $h_{\theta}$.

$$ \mathcal{Q} = D \cup D_{\text{synthetic}} \qquad (7) $$

$$ D_{\text{synthetic}} = \{x + \delta \mid x \in D\} \qquad (8) $$

where $\delta$ is a small perturbation.

$$ D_{\text{attack}} = \{(x, h_{\theta}(x)) \mid x \in \mathcal{Q}\} \qquad (9) $$

$$ \text{ME}: (\mathcal{X}^P \times \mathcal{Y}) \times (\mathcal{X}^P \to \mathcal{Y}) \to (\mathcal{X}^P \times \mathcal{Y}) \qquad (10) $$
From this definition, it is evident that the *D*<sub>attack</sub> dataset used to build the extracted model consists simply of the input samples and the corresponding outputs generated by the target model for those samples. In FREIDA, we chose to implement the Knockoff Nets attack as described by Orekondy et al. [16].

The KnockoffNets attack is a model extraction technique designed to repli-
cate the functionality of a black-box machine learning model. The attack involves
querying the target model, a black-box model where only the input-output be-
haviour is accessible with carefully selected input samples to obtain correspond-
ing predictions. These input-output pairs are then used to train a “knockoff”
model, which approximates the target model’s behaviour. The uniqueness of
KnockoffNets lies in its strategic selection of input queries, enabling the attacker
to approximate the target model with minimal queries efficiently. This attack
is particularly concerning as it allows adversaries to recreate high-performance
models with access only to the model’s outputs, bypassing the need for access
to the original training data or model architecture.

4 Experiments

In this section, we present a comprehensive evaluation of various machine learning-
based IDSs using the FREIDA tool. Our experiments aim to assess the resilience
of these systems against privacy attacks, specifically focusing on membership in-
ference [20] and model extraction attacks [16]. By leveraging the systematic
framework, we ensure that our evaluation is both rigorous and reproducible.
To achieve this, we selected a diverse set of IDS architectures that represent
the current state-of-the-art in machine learning applications for intrusion detec-
tion. Each architecture was evaluated under controlled conditions, allowing us
to analyze their performance, accuracy, and privacy implications effectively.

In addition to assessing the overall detection capabilities of the IDS, we also
measured their sensitivity to privacy attacks. The subsequent subsections detail
the experimental setup, the specific methodologies employed, and the results ob-
tained, providing a comprehensive overview of our findings and their implications
for the future of privacy-aware IDS development.

We choose to implement our evaluation within FREIDA because this frame-
work enables a comprehensive assessment of IDS without the need to reimple-
ment the entire ML pipeline. Additionally, FREIDA facilitates easy reproduction
and sharing of our evaluation through a straightforward configuration file (see
Lst.1.1).

4.1 Membership Inference

We first evaluated the models’ resilience to MI by measuring the accuracy of de-
termining membership status for both member and non-member data instances.

To evaluate the resilience of IDS against membership inference attacks, we
developed three distinct models based on widely used Deep Learning architec-
tures: Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), and
<figure><img src="image_2.png" /><figcaption>Listing 1.1: Configuration example for Model Extraction on UNSW-NB15</figcaption></figure>

<table><caption>Table 1: Results of Membership Inference attacks on all datasets</caption><thead><tr><th rowspan="2">Metrics</th><th colspan="3">UNSW-NB15</th><th colspan="3">NSL-KDD</th><th colspan="3">CIC-IDS2017</th></tr><tr><th>MLP</th><th>CNN</th><th>LSTM</th><th>MLP</th><th>CNN</th><th>LSTM</th><th>MLP</th><th>CNN</th><th>LSTM</th></tr></thead><tbody><tr><td>AUC</td><td>0.895</td><td><strong>0.918</strong></td><td>0.900</td><td><strong>1.0</strong></td><td>1.0</td><td>1.0</td><td><strong>0.655</strong></td><td>0.419</td><td>0.534</td></tr><tr><td>Advantage</td><td><strong>0.676</strong></td><td>0.666</td><td>0.615</td><td>0.991</td><td><strong>0.994</strong></td><td>0.990</td><td><strong>0.303</strong></td><td>0.231</td><td>0.188</td></tr><tr><td>PPV</td><td>0.999</td><td><strong>1.0</strong></td><td>1.0</td><td><strong>1.0</strong></td><td>1.0</td><td>1.0</td><td>0.873</td><td><strong>0.910</strong></td><td>0.845</td></tr></tbody></table>

Long Short-Term Memory (LSTM). We chose to focus on deep learning models as they seem most relevant for evaluation today. The models were trained and tested using three benchmark datasets–UNSW-NB15 [14], NSL-KDD[22], and CIC-IDS2017 [19]. Notably, our implementation allows for seamless adaptation to other datasets, enhancing its versatility. After training the models as multi-class classifiers, we implemented the attack following the methodology outlined in Sec. 3.2. Moreover, in this experiment, the key metrics considered include the Area Under the Curve (AUC); the Attacker Advantage, which reflects the relative improvement of the attacker model over a baseline, providing insights into its effectiveness in identifying targets; and Positive Predictive Value (PPV), which is simply the proportion of positive predictions that are true positives.

The results, summarized in Table 1, show that for the UNSW-NB15 dataset, the attack model performed best against CNN, achieving the highest AUC (0.918) and a PPV of 1.0, indicating significant success in identifying membership. However, the attack on MLP was also effective, with a comparable AUC of 0.895 and the highest Advantage score (0.676). LSTM was slightly more resistant, with a lower AUC (0.900) and Advantage (0.615). In contrast, for
the NSL-KDD dataset, the attacker models showed almost perfect effectiveness across all target models, with AUC values of 1.0 and PPV of 1.0, indicating that all three IDS models are highly vulnerable to MI. CNN slightly outperformed the others in terms of Advantage (0.994), indicating a marginally higher degree of vulnerability compared to MLP and LSTM.

The vulnerability to membership inference attacks in this scenario arises pri-
marily from the way the data is partitioned. Specifically, the NSL-KDD dataset
is pre-split into training and test sets. However, this split results in a mismatch
between the number of classes in the training and test sets—23 in the training set
and 38 in the test set. This discrepancy introduces a significant risk of informa-
tion leakage, especially in a multiclass classification context. Since the models
must be trained to accommodate the maximum number of classes across both
sets, but certain classes are absent from the training data, the model’s output
will inadvertently reveal which samples belong to the training set. This makes
it easier for an attacker to infer membership, exploiting the absence of certain
classes in training data as a piece of information.

Finally, for the CIC-IDS2017 dataset, the attack model struggled more, espe-
cially against MLP, which had the highest AUC (0.655) and Advantage (0.303),
but lower PPV (0.873). The attack on CNN was less successful, with an AUC
of 0.419 and Advantage of 0.231, while LSTM showed moderate resistance with
an AUC of 0.534 and the lowest Advantage (0.188).

In conclusion, the experimental results reveal significant variability in the
susceptibility of IDS models to MI attacks across different datasets and model
architectures. CNN-based IDS models were consistently the most vulnerable,
with high AUC and PPV scores indicating that attackers could effectively in-
fer membership in most cases, particularly for the UNSW-NB15 and NSL-KDD
datasets. Although MLP demonstrated higher resilience in certain metrics, espe-
cially for the CIC-IDS2017 dataset, it was still vulnerable to MI. LSTM models
generally exhibited moderate resistance but did not significantly outperform
other architectures in protecting against attacks. These findings highlight the
critical need to tailor privacy defenses, like Differential Privacy [9], on the model
architecture to mitigate the risks posed by MI in IDS systems.

## 4.2 Model Extraction

In this phase of the experiment, we evaluate the resilience of the same IDS
models-MLP, CNN, and LSTM-to model extraction attacks. These attacks aim
to replicate a target model by reconstructing it based on query outputs. Specif-
ically, we are interested in two key aspects: i) the performance of the extracted
model, which ideally mirrors the original model's accuracy and behaviour, and
ii) the number of queries required by the attacker to build a near-identical replica.
The evaluation uses the same three benchmark datasets (UNSW-NB15, NSL-
KDD, and CIC-IDS2017), but this time under a binary classification task to
simplify the attack scenario and focus on the extraction dynamics. By quanti-
fying the number of queries needed to achieve high-performance replication, we
can assess the models' susceptibility to extraction attacks and determine which
architectures present more resistance. The results provide insights into how effi-
ciently attackers can duplicate the functionality of the IDS models.

<figure><img src="image_3.png" /><figcaption>Fig. 1: Evolution of the Model Stolen Accuracy on UNSW-NB15, NSL-KDD and CIC-IDS2017</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2: Evaluation results of the Model Extraction attack on UNSW-NB15, NSL-KDD and CIC-IDS2017</figcaption></figure>
The outcomes of this evaluation are outlined in Fig. 1 and Fig. 2. The results clearly indicate that the attack successfully extracted models with performance levels identical to the original for all datasets and models tested. Specifically, in the UNSW-NB15 dataset, the highest accuracy was achieved with 1,500 queries, while the NSL-KDD dataset required 3,400 queries to obtain comparable results. In contrast, the CIC-IDS2017 dataset yielded similar performance after just 1,800 queries. In all study cases, the protection used (represented by the protected accuracy in Fig. 2) exhibits a strong capability to defend the model against ME attacks. Specifically, we employed a Reverse Sigmoid [11] protection mechanism, which alters the model's output in a way that obscures the decision boundaries. This transformation makes it significantly more challenging for attackers to accurately replicate the model's behaviour during extraction attempts. This finding underscores the effectiveness of the implemented protection strategies in enhancing model security across various datasets and architectures. By also applying this protection technique during the training, we can maintain the same performance as the non-protected model while safeguarding it against model extraction attacks.

Of course, the evaluated models are relatively simple, which facilitates the execution of these attacks. However, it is crucial to emphasize how easily model extraction attacks can succeed, especially considering that the nature of the data used (tabular data) further enhances the likelihood of successful extraction.

To conclude, these results underscore the trade-off between model performance and privacy, suggesting that even high-performing models can be susceptible to extraction, thereby compromising sensitive data and integrity in real-world deployments. This analysis highlights the necessity of implementing robust defenses to safeguard machine learning models against privacy breaches in IDS applications.

# 5 Conclusion

This paper has systematically examined the benchmarking of privacy properties of machine learning-based IDS, a topic left underexplored by previous research. By developing a comprehensive threat model and an attack formalism, then integrating those in the FREIDA framework, we laid the groundwork for a rigorous yet practical evaluation of IDS architectures against various privacy threats. Our approach not only streamlines the evaluation process, but also ensures the reliability and reproducibility of our findings across different settings. Through experimentation, we evaluated multiple machine learning-based IDS architectures, providing valuable insights about their vulnerabilities and resilience to membership inference and model extraction attacks.

This work deepens the understanding of intrusion detection and privacy, guiding future research to enhance IDS resilience against evolving cyber threats. While our paper focuses on privacy attacks that specifically target intrusion detection systems, our methodology can easily be extended to benchmark other privacy properties in various other contexts that make use of machine learning.
References

1. Cisco Annual Internet Report (2018-2023). Tech. rep., Cisco (2020)

2. Arshad, J., Azad, M.A., Salah, K., Jie, W., Iqbal, R., Alazab, M.: A review of performance, energy and privacy of intrusion detection systems for iot. ArXiv abs/1812.09160 (2018)

3. Ayoubi, S., Blanc, G., Jmila, H., Silverston, T., Tixeuil, S.: Data-driven evaluation of intrusion detectors: A methodological framework. In: Foundations and Practice of Security. pp. 142–157. Springer Nature Switzerland (2022)

4. Ayoubi, S., Tixeuil, S., Blanc, G., Jmila, H.: Demo: Towards reproducible evaluations of ml-based ids using data-driven approaches. In: Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security (CCS ’24). p. 3. ACM, Salt Lake City, UT, USA (October 2024). https://doi.org/10.1145/3658644.3691368

5. Ayoubi, S., Tixeuil, S., Jmila, H., Blanc, G.: Freida: A concrete tool for reproducible evaluation of ids using a data-driven approach. In: International Conference on Risks and Security of Internet and Systems. Springer (November 2024)

6. Bukhari, S.M.S., Zafar, M.H., Houran, M.A., Moosavi, S.K.R., Mansoor, M., Muaaz, M., Sanfilippo, F.: Secure and privacy-preserving intrusion detection in wireless sensor networks: Federated learning with scnn-bi-lstm for enhanced reliability. Ad Hoc Networks 155, 103407 (2024). https://doi.org/10.1016/j.adhoc.2024.103407, https://doi.org/10.1016/j.adhoc.2024.103407

7. Cui, J., Xiao, J., Zhong, H., Zhang, J., Wei, L., Bolodurina, I., He, D.: LH-IDS: lightweight hybrid intrusion detection system based on differential privacy in vanets. IEEE Trans. Mob. Comput. 23(12), 12195–12210 (2024). https://doi.org/10.1109/TMC.2024.3407709, https://doi.org/10.1109/TMC.2024.3407709

8. Du, J., Yang, K.: NIDS-FLGDP: network intrusion detection algorithm based on gaussian differential privacy federated learning. J. Circuits Syst. Comput. 33(3) (2024). https://doi.org/10.1142/S0218126624500488, https://doi.org/10.1142/S0218126624500488

9. Dwork, C., Roth, A.: The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9(3–4), 211–407 (2014). https://doi.org/10.1561/0400000042, http://dx.doi.org/10.1561/0400000042

10. Gupta, R., Alam, T.: An efficient federated learning based intrusion detection system using ls²dnn with PBKA based lightweight privacy preservation in cloud server. Multim. Tools Appl. 83(15), 44685–44697 (2024). https://doi.org/10.1007/S11042-023-17401-7, https://doi.org/10.1007/S11042-023-17401-7

11. Lee, T., Edwards, B., Molloy, I., Su, D.: Defending against machine learning model stealing attacks using deceptive perturbations. arXiv: Learning (2018)

12. Li, S., Han, M.: Priv-ids: A privacy protection and intrusion detection framework for in-vehicle network. In: Xu, Y., Yan, H., Teng, H., Cai, J., Li, J. (eds.) Machine Learning for Cyber Security - 4th International Conference, ML4CS 2022, Guangzhou, China, December 2-4, 2022, Proceedings, Part II. Lecture Notes in Computer Science, vol. 13656, pp. 165–179. Springer (2022). https://doi.org/10.1007/978-3-031-20099-1_14, https://doi.org/10.1007/978-3-031-20099-1_14
13. McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Singh, A., Zhu, X.J. (eds.) Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 20-22 April 2017, Fort Lauderdale, FL, USA. Proceedings of Machine Learning Research, vol. 54, pp. 1273-1282. PMLR (2017), http://proceedings.mlr.press/v54/mcmahan17a.html

14. Moustafa, N., Slay, J.: Unsw-nb15: a comprehensive data set for network intrusion detection systems (unsw-nb15 network data set). In: 2015 Military Communications and Information Systems Conference (MilCIS). pp. 1-6 (2015)

15. Niksefat, S., Kaghazgaran, P., Sadeghiyan, B.: Privacy issues in intrusion detection systems: A taxonomy, survey and future directions. Comput. Sci. Rev. 25, 69-78 (2017)

16. Orekondy, T., Schiele, B., Fritz, M.: Knockoff nets: Stealing functionality of black-box models. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 4949-4958 (2018)

17. Raza, M., Saeed, M.J., Riaz, M.B., Sattar, M.A.: Federated learning for privacy-preserving intrusion detection in software-defined networks. IEEE Access 12, 69551-69567 (2024). https://doi.org/10.1109/ACCESS.2024.3395997, https://doi.org/10.1109/ACCESS.2024.3395997

18. Rigaki, M., Garcia, S.: A survey of privacy attacks in machine learning. ACM Comput. Surv. 56(4) (Nov 2023). https://doi.org/10.1145/3624010, https://doi.org/10.1145/3624010

19. Sharafaldin, I., Lashkari, A.H., Ghorbani, A.A.: Toward generating a new intrusion detection dataset and intrusion traffic characterization. In: International Conference on Information Systems Security and Privacy (2018)

20. Shokri, R., Stronati, M., Song, C., Shmatikov, V.: Membership inference attacks against machine learning models. 2017 IEEE Symposium on Security and Privacy (SP) pp. 3-18 (2016)

21. Tabassum, A., Erbad, A., Lebda, W., Mohamed, A., Guizani, M.: FEDGAN-IDS: privacy-preserving IDS using GAN and federated learning. Comput. Commun. 192, 299-310 (2022). https://doi.org/10.1016/J.COMCOM.2022.06.015, https://doi.org/10.1016/j.comcom.2022.06.015

22. Tavallaee, M., Bagheri, E., Lu, W., Ghorbani, A.A.: A detailed analysis of the kdd cup 99 data set. In: 2009 IEEE Symposium on Computational Intelligence for Security and Defense Applications. pp. 1-6 (2009). https://doi.org/10.1109/CISDA.2009.5356528

23. Tramèr, F., Zhang, F., Juels, A., Reiter, M.K., Ristenpart, T.: Stealing machine learning models via prediction apis. In: USENIX Security Symposium (2016)

24. Vyas, A., Lin, P., Hwang, R., Tripathi, M.: Privacy-preserving federated learning for intrusion detection in iot environments: A survey. IEEE Access 12, 127018-127050 (2024). https://doi.org/10.1109/ACCESS.2024.3454211, https://doi.org/10.1109/ACCESS.2024.3454211

25. Yeom, S., Fredrikson, M., Jha, S.: The unintended consequences of overfitting: Training data inference attacks. ArXiv abs/1709.01604 (2017)