DiffuPac: Contextual Mimicry in Adversarial Packets Generation via Diffusion Model
Authors: Abdullah Bin Jasni
Akiko Manada
Kohei Watabe
Venue: NeurIPS 2024 poster (2024)
Keywords: Network Intrusion Detection System
Adversarial Machine Learning
Cybersecurity
Adversarial Sample Generation
Primary area: diffusion_based_models
TL;DR: DiffuPac, a first-of-its-kind model that utilized pre-trained BERT with diffusion model to generate adversarial packets

Abstract: In domains of cybersecurity, recent advancements in Machine Learning (ML) and Deep Learning (DL) have significantly enhanced Network Intrusion Detection Systems (NIDS), improving the effectiveness of cybersecurity operations. However, attackers have also leveraged ML/DL to develop sophisticated models that generate adversarial packets capable of evading NIDS detection. Consequently, defenders must study and analyze these models to prepare for the evasion attacks that exploit NIDS detection mechanisms. Unfortunately, conventional generation models often rely on unrealistic assumptions about attackers' knowledge of NIDS components, making them impractical for real-world scenarios. To address this issue, we present DiffuPac, a first-of-its-kind generation model designed to generate adversarial packets that evade detection without relying on specific NIDS components. DiffuPac integrates a pre-trained Bidirectional Encoder Representations from Transformers (BERT) with diffusion model, which, through its capability for conditional denoising and classifier-free guidance, effectively addresses the real-world constraint of limited attacker knowledge. By concatenating malicious packets with contextually relevant normal packets and applying targeted noising only to the malicious packets, DiffuPac seamlessly blends adversarial packets into genuine network traffic. Through evaluations on real-world datasets, we demonstrate that DiffuPac achieves strong evasion capabilities against sophisticated NIDS, outperforming conventional methods by an average of 6.69 percentage points, while preserving the functionality and practicality of the generated adversarial packets.

DiffuPac: Contextual Mimicry in Adversarial Packets Generation via Diffusion Model
Abdullah Bin Jasni
Akiko Manada
Graduate School of Engineering
Graduate School of Engineering
Nagaoka University of Technology Nagaoka, Japan s203108@stn 1at000000sttt.aaaaaatu.....
Nagaoka University of Technology
Nagaoka, Japan
bearada@@os.naggokke jc.
Kohei Watabe
Graduate School of Science and Engineering
Saitama University
Saitama, Japan
kwatabe@mail, sai tama-u.aa. ac.
Abstract
In domains of cybersecurity, recent advancements in Machine Learning (ML) and Deep Learning (DL) have significantly enhanced Network Intrusion Detection Sys- tems (NIDS), improving the effectiveness of cybersecurity operations. However, attackers have also leveraged ML/DL to develop sophisticated models that generate adversarial packets capable of evading NIDS detection. Consequently, defenders must study and analyze these models to prepare for the evasion attacks that exploit NIDS detection mechanisms. Unfortunately, conventional generation models often rely on unrealistic assumptions about attackers' knowledge of NIDS components, making them impractical for real-world scenarios. To address this issue, we present DiffuPac, first-of-its -of-itt-kind generation model designed to generate adversarial pack- ets that evade detection without relying on specific NIDS components. DiffuPac integrates a pre-trained Bidirectional Encoder Representations from Transformers (BERT) with diffusion model, which, through its capability for conditional denois- ing and classifier-free guidance, effectively addresses the real-world constraint of limited attacker knowledge. By concatenating malicious packets with contextually relevant normal packets and applying targeted noising only to the malicious pack- cts, DiffuPac seamlessly blends adversarial packets into genuine network traffic. Through evaluations on real-world datasets, demonstrate that DiffuPac achieves strong evasion capabilities against sophisticated NIDS, outperforming conventional methods by average of 6.69 percentage points, while preserving the functionality and practicality of the generated adversarial packets.
Introduction
Network Intrusion Detection Systems (NIDS) play a pivotal role in safeguarding the vast array of digital devices and infrastructures that permeate our lives. As of 2023, the global count of active IoT devices expected to reach approximately 15.14 billion, with projections suggesting a rise to 30 billion by 2030 (Statista [2023]). This explosive growth, fueled by applications spanning from consumer electronics to industrial automation and healthcare, presents formidable security challenges. To meet these challenges, advancements in Machine Learning (ML) and Deep Learning (DL) have significantly bolstered the efficacy NIDS monitoring IoT traffic and detecting malicious activities (Talaei Khoei and Kaabouch [2023]; Talaei Khoei et al. [2023]).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, the rapid evolution of generative AI technologies has ushered in a new era of cyberse- curity threats, notably through the creation of adversarial packets designed evade detection by even the most sophisticated NIDS. These Al-driven attacks can emulate and synthesize legitimate network behaviors, presenting an unprecedented challenge to existing security paradigms, Generative models, particularly those trained on extensive datasets of genuine network traffic, can generate adversarial packets that blend malicious functionalities within seemingly normal packet sequences, thus effectively camouflaging their malicious intents.
In response to these evolving threats, it critical for cybersecurity defenders to deepen their un- derstanding of these generative AI models. By scrutinizing the mechanisms through which these models generate adversarial packets, defenders can better anticipate and counteract adversarial tacties that compromise the detection capabilities of NIDS (Ibitoye et al. [2019]; Khazane et al. [2024]). The urgency to develop innovative solutions that can adapt to and preempt these adversarial tactics is paramount, ensuring the reliability and robustness of NIDS in an increasingly complex threat landscape.
Traditional methods for generating ad- versarial packets have primarily re- lied on direct engagement with NIDS or the use of surrogate classifiers. often assuming unrealistic levels of attacker access NIDS configura- tions. Table 1 summarizes recent literature on adversarial packet gen- eration, highlighting the limitations of these approaches. Studies using techniques such as Network Emula- tor (NetEM) and Metasploit (Homo- liak et al. [2018]) Generative Adver- sarial Network (GAN) and Particle Swarm Optimization (PSO) (Han et al. [2021]). and Reinforcement Learning (RL) (Hore et al. [2023]) demonstrate that while these methods successfully modified the packets behavior, their efficacy in evading detection in real-world conditions remains suboptimal. The reliance on detailed knowledge of NIDS models is flawed, as attackers typically operate with limited information about the underlying security infrastructure. This gap underscores the necessity for more practical and effective adversarial generation methods that align with the realistic constraints faced by attackers.
Table I: Summary of recent literature on adversarial packets generation.
Classifier- Techniques
Author
Year
Data Set
Algorithm
Homoliak et al. [2018]
2018
ASNM-
NBPO
Classlier
Suroogate
Metasploit
NetEM,
Hashemi et al. 12019]
2019
CICIDS-
2018
Clussiier
Surrogate
Trial and
Kappa et al. [2019]
2019
CICIDS-
2018
Classifier
Surrogute
Manifold
Approx.
Kitsune.
NIDS
GAN and
Han et [2021]
2021
CICIDS
feature
PSO
2017
extractor
Kitsune. CICIDS 2017
NIDS classifier
Sharon al. [2021]
2021
LSTM
CICIDS-
Hore al. [2023]
2023
CICIDS
2017.
Classifier
Surrogate
RL
2018
To address these challenges, we introduce DiffuPac: Contextual Mimicry in Adversarial Packet Generation via Diffission Model, novel solution that leverages the combined strengths of Bidirec- tional Encoder Representations from Transformers (BERT) and diffusion model. This innovative fusion not only promises high accuracy in generating adversarial packets but also operates under the realistic assumption that attackers lack direct access to NIDS models. DiffuPac leverages the extensive contextual understanding provided by BERT, which has been trained on diverse datasets representing a wide range of network behaviors, along with the generative capabilities of diffusion models. This fusion results in a sophisticated adversarial tacties where the elements of the attack are seamlessly integrated into the network traffic, making them indistinguishable from legitimate data. This capability represents a significant leap forward, offering a stealthy approach that outmaneuvers current NIDS through advanced mimicry rather than direct confrontation.
In summary, the principal contributions of this paper as follows: (a) we have pioneered the integration of BERT and diffusion models to create DiffuPac, marking a first in the cybersecurity domain. This novel methodology sets a precedent in the field by blending the advanced contextual comprehension of network traffic with sophisticated generative capabilities to produce adversarial packets that are both stealthy and indistinguishable from genuine traffic; (b) we introduce a unique concatenation strategy coupled with targeted noising techniques. These innovations ensure that the adversarial packets not not only blend seamlessly into the network environment but also dynamically adapt to evade modern detection systems; (c) DiffuPac advances a classifier-fre free approach to adversarial packet generation. This approach challenges traditional dependency on surrogate classifiers, offering

---

a new paradigm that more accurately reflects the constraints and capabilities of real-world attackers; (d) Lastly, our extensive experimental evaluations, conducted on real-world datasets, demonstrate that DiffuPac significantly outperforms existing methods in terms of evasion effectiveness, establishing new benchmarks for the generation of adversarial packets. These contributions collectively push the boundaries of what is possible in the realm of network security, paving the way for more resilient cybersecurity defenses and a deeper understanding of adversarial tactics in network environments.
2 Related Works
Adversarial Attacks on ML/DL-based NIDS. NIDS are crucial for protecting digital infrastructures by monitoring network traffic and identifying potential threats through signature- -bauud and anomaly- based detection paradigms. Signature- based NIDS use pattern matching against predefined threat databases, while anomaly-based NIDS employ machine learning models to detect deviations from benign traffic patterns, providing an advantage in detecting sophisticated threats. The transition to DL in anomaly-based NIDS, as highlighted by Ahmad et al. [2021], has enhanced threat detection due to the capabillity to learn abstract patterns, though this shift has introduced vulnerability to adversarial attacks. These attacks modify network data subtly to evade NIDS, a phenomenon first noted in computer vision (Szegedy et al. [2014]) and now challenging DL-based NIDS. Adversarial attack tactics vary based on attacker knowledge, ranging from white-box attacks with full system knowledge black-box attacks with no classifier knowledge, as outlined by McCarthy et al. [2022]. Black-box scenarios are common in real-world threats since attackers generally lack direct NIDS access, necessitating sophisticated modifications of data exploit DL model vulnerabilities and blur the distinction between normal and malicious traffic.
Ad Adversarial Attacks Evading NIDS. In the study of NIDS, researchers explore feature-level and packet-level attacks to enhance robustness. Feature-level attacks modify input network features using methods like GANs to mislead classifiers without direct knowledge of their mechanisms. For instance, Yang et al. [2018] 8] used transfer-based and score-based attacks to deceive a Deep Neural Network (DNN) model on the NSL-KDD dataset, while Sheatsley et al. [2022] developed an Augmented JSMA (AJSMA) to ensure realistic feature modifications. However, these still fail to generate executable malicious packets since they do not provide a method for converting modified features into packet sequences. Conversely, packet-level attacks, which modify network packets directly to maintain their malicious intent while evading detection, are particularly effective. As detailed in Table 1, studies such as Hashemi et al. [2019] show successful modifications using non-pa non-payloar payload based and mimicking operations. In the survey He et al. [2023], authors emphasized that the practicality and replayability of packet-level attacks make them more dominant, ensuring adversarial packets evade detection while remaining executable. Given this efficacy, our research model, DiffuPac, is designed to excel in this domain, outperforming previous models in robustness and detection evasion, and setting new standard for NIDS efficacy in adversarial cybersecurity.
Diffusion Models. Represent a breakthrough in generative modeling, offering a novel framework for understanding data as a dynamic and stochastic process. Based on the works of Sohl-Dickstein et al. [2015] and Ho et al. [2020]. these models conceptualize data points Zn E Hdd (where d is a positive integer) as the end result of reverse Markov chain. In the forward diffusion process, data points Zu are gradually transformed into noise distribution. This transformation is modeled by a Markov chain starting from the data distribution (Zo)) and ending in a Gaussian distribution 27 (0, I), where T represents the total number of timesteps in the forward diffusion process. The transitions are defined g (2,,/21)) = =N(z,; vT B,2,- B,I) for S < T. where ,3, as the variance scale. In the reverse process, the model fo (usually a U-Net or a Transformer) learns to reconstruct the original data Zo from the noised data øy. It does so by iteratively estimating the parameters pa (z (2t) = N (Zz-1i pe(z, t), If)), where μe and of predict the mean and variance of the distribution.
BERT and Diffusion Models Approaches in Cybersecurity and Other Fields. The application of pre-trained models, such as ET-BERT (Lin et al. [2022]) and NetGPT (Meng et al. (2023]), has revolutionized understanding of network traffic by capturing the intricacies of language and interpret- ing complex network patterns. Meanwhile, diffusion models have emerged as powerful generative tools capable of producing high-fidelity data for applications like image generation and notably, synthetic network traffic generation. These models leverage unique denoising training objective to closely mimic the original data distribution, making them promising for generating realistic network
3

---

Pcap Preprocessing
Pre-training
Fine-tuning with Diffusion Model
Normal
Malicaous
Same Sequence- Prediction
Musked Unidirectional
packet sequensze
packel
Flow Model
relevant pairingt
Ridirectional Flow based splinting
Misze
Packeet Seq. Concutonation
Session
Zo
S Normal (benign) Packet Seq.
S
Sre-hodas
BERT
Molicions Packet Sog
Dii-to-rre
splittine
fe-1
a(<elzz-1)
Py (Zz=11ze)
KASS
Hex number rxprssentation
Erear
Expra
prre
Embedding
St5ec16475085 43340500A000...
Torgeted Noising
F.
E
Fy
E
Farholding
Tokenization
ZT
Seyment
5056 COSA 7508
O&CO AMMO
Enhedding
See so-des.
Dst to sec
Forward
Process
Revene
Process
Guussian Noise
Figure 1: Proposed arhitecture divided into three phases: peap pre- pre-processing pre- pre-training and fine-tuning with diffusion models.
traffic for testing and analysis. Despite these advancements, combining the contextual understanding of pre-trained models with the generative prowess of diffusion models remains underexplored in adversarial packet generation. This integration offers a synergistic approach, merging the nuanced comprehension of network behaviors by pre-trained models with the high-quality data generation of diffusion models. By leveraging these technologies together, cybersecurity researchers can enhance the realism of simulated network environments and develop more advanced evasion tactics.
DiffuPac
In this section, introduce DiffuPac, a first-of-its kind adversarial packet generation model. Figure 1 illustrates the overall framework of the proposed model, consisting of three main phases: peap pre-processing, pre-training and fine-tuning with diffusion model.
3.1 Data pre-processing
In real network environments, traffic contains diverse flows from various applications, protocols, and services, complicating the learning of stable representations. Therefore, first split peap (packet capture) files into sessions (bidirectional flows) based on IP addresses, port addresses, and protocols. To refine the training of the BERT model, we further split sessions into unidirectional flows, categorizing them as either source-to-destination (sre-to-dst) packet sequence or destination-tor source (dst-to-sre) packet sequence. This categorization is crucial for the BERT model's pre-training. Network traffic varies due to diverse protocols and network services, resulting in different formats and patterns. To handle these variations and encoding requirements, we convert each byte to its corresponding hex number and tokenize using WordPiece (Wu et al. [2016]). Each token ranges from to 65535, with a dictionary size of 65536. We also incorporate special tokens [CLS]. [SEP], [PAD], and [MASK] for training tasks. [CLS] is used at the beginning of each sequence and helps in classification tasks. [SEP] separates different sequences or segments within the same input. [PAD] ensures sequences are of uniform length and satisfy the minimum length requirement. [MASK] is used pre-training task, where it temporarily substitutes tokens to be predicted.
3.2 Pre-training
As shown in Figure 1, the input tokens are processed using an embedding strategy that involves the sum of three types of embeddings: token embeddings. positional embeddings, and segment embeddings. Each embedding has dimension of 768. Token Embeddings high-dimensional

---

vectors that uniquely represent each token, acting as their exclusive identifiers. Positional Embeddings are used to capture the temporal relationships of tokens, ensuring that the model learns to focus on the order of data transmission. Segment Embeddings differentiate packets within a single flow, as packets may not inherently share semantic associations, and preserving the order of packets to maintain the temporal sequence of events in a session flow.
We design our pre-training tasks based on the approach described in Devlin et al. [2019]. Our two proposed pre-training tasks aim to capture the contextual relationships between traffic bytes. The first task involves predicting masked tokens learn the underlying patterns and dependencies within the traffic data. The second task predicts the transmission order by determining whether packets belong to sre-to-dst or dst-to-sre sre sequences, thereby capturing the directional flow of the network traffic.
Masked Unidirectional Flow Model. Inspired by BERT's Masked I Language Model (MI.) in natural language processing, adapt this approach for network traffic analysis through Masked Unidirectional Flow Model. This model is designed to understand and predict the semantic patterns within bidirectional network flows; that are stc-to-dst sequence and dst-to-src sequence. During pre-training, each token in the input sequence is masked with a probability of 15%. Amongst these masked tokens 80% are replaced with [MASK] token, 10% are replaced with a random token from the vocabulary, and 10% are left unchanged. This introduces variability that mimics real-world data inconsistencies. The training objective is to minimize the negative log- likelihooo of correctly predicting the original tokens at the masked positions. Formally, the loss function LMUM for this task defined as:
LMUM
(mm)) log P(my
Vinaskod 0):
(1)
where n is the total number of tokens in the sequence. 1 (m,)) is an indicator function that is 1 if the token iy was masked (0 otherwise). my indicates the masking status of the i-th token and Vinaked is the masked sequence. 2; is the actual token at position i and P(v; Vimaked; f) is the probability of predicting the original token for given masked sequence and model parameters O. The transformer encoder, characteristic of BERT, processes Vinsked to predict each masked token. This architecture leverages self-attentio mechanisms to capture dependencies and context effectively, which is crucial for understanding complex patterns in network traffic.
Same Sequence-origin Prediction. Inspired by Lin et al. [2022], this task employs a dedicated binary classifier to determine the directional origin of network traffic, specifically whether packets in a sequence originate from src-to src-to-dst or dst-to-sic. This classification enhances the model's s understanding of network flows and the contextual relationships between packets, which is crucial for recognizing communication patterns and dependencies in network traffic. For this task, each packet a, paired with another packet ag. 50% of the time, 0, is the next logical packet in the flow (src-to-dst or dst-to-sre); the other 50% of the time, it is randomly chosen packet from the opposite flow. The classifier then predicts whether a, follows a, in the correct directional sequence. This pairing strategy improves the model's 's capability to discern patterns in packet flows. Let A = {(a,, ag)} be the set of packet pairs, where each pair is labeled with bw E {0, 1 (1 if a, follows a, in the correct flow, otherwise). The loss function Lssp for this task can be formulated as:
LSsP Σ bw log Plb = Gr, as; 0) + (1 log(1 P(b.-
ar,, ar,aa,;;)) 0)),
(2)
where K is the total number of packet pairs and @ represents the trainable parameters of the classifier. p(b., = 1 a,, a,; O) is the probability predicted by the classifier that a, correctly follows a, in the given network flow direction.
In summary, the final pre-training objective is the sum of the above two losses, which can defined as:
L Lssp + TMUM-
Fu et al. [2021] mentioned that the number of packets within each flow can vary significantly. Given the constraints imposed by packet size and the potential volume of network traffic, computational efficiency is paramount concern. In addressing this problems, as demonstrated in Dai et al. [2023] that the initial packets in flow contain the most significant information, we limit our analysis to the first three packets of each heavy flow. This means that for each session, we analyze total of six packets-thi three from the src-to-dst flow and three from the dst-to-sre flow. This strategy ensures a comprehensive view of the session while maintaining efficiency.
5

---

3.3 Fine-tuning with Diffusion Models
Forward Process with Packet Sequences Concatenation Strategy. In the fine-tuning phase, the goal is to train the model so that malicious packets can mimic normal packets to bypass NIDS. This involves using packet sequences that are only from the sre-to-dst, reflecting the adversary's i control over the packets being sent (Hore et al. [2023]). The forward process of our diffusion model begins by embedding both normal (benign) packet sequences S Sben and malicious packet sequences Stmal This transformation of discrete packet data into a continuous feature spaces uses an embedding function adapted from Li et al. [2022]. Building upon the groundwork of DiffuSeq (Gong et al. [2023]) approach, which typically involves random merging, our model innovates by leveraging the deep contextual insights provided by the pre-trained BERT model (Details in A.2). This allows us to strategically pair normal and malicious sequences with strong contextual alignments that show similarity in network behavior patterns. These contextually aligned pairs are crucial for mimicking normal traffic and increasing the chances of bypassing NIDS. By integrating these contextually aligned pairs into our diffusion process, the model extends the original forward chain to a new Markov transition go (z Sben@mal)- = -V (EMBBBSSeeeeegi Bo B) where EMB (Sben@mal) symbolizes the embedding transformation and concatenation of normal and malicious packet sequences. While Sben@mal denotes the initial concatenated sequence, the forward process gradually perturbs this initial state Zo through a series of transitions, producing latent variables 2,, 22,..., Zz.
Targeted Noising. To enhance our diffusion model's capability for adversarial packet generation, we simplify the model's state transitions by defining z, = Xx @ yr, where x; and y, correspond to the portions of z, that belong to Shen and Small respectively. This setup allows us to strategically inject noise into only the malicious packet sequences (represented by yı), rather than the entire state ze during each forward step g(z 2(-1). This targeted noising, inspired by DiffuSeq. is pivotal for adapting conventional diffusion models for targeted modification.
Reverse Process With Normal Packet Guidance. Building upon the foundational principles of DiffuSeq, our model introduces an innovative reverse process tailored for more nuanced handling of network traffic data. This process distinctively utilizes normal packet sequences framework, enabling a sophisticated denoising technique that treats the concatenated sequences of normal and noise-added malicious packets as a unified unit. This approach effectively "teaches" the model to perceive these malicious elements as integral parts of the normal traffic pattern. A key aspect of this reverse process is the use of the pre-trained BERT model as the denoising engine during packet reconstruction. In this phase, the BERT model undergoes fine-tuning within the diffusion framework, ensuring that it is specifically adapted for reconstructing packets. As result, the BERT model is not only responsible for contextual understanding but also plays a central role in recovering malicious packets that have been integrated into normal traffic patterns. The conditional denoising process effectively employs Bayesian inference to parameterize the transition probabilities between states, ensuring precise control over each step in the reverse process. These parameterizations are mathematically articulated through the equations: po (Zo:T) := =p(zr) [[= Po(2 |2+) and Po(2,-1 (2+) = (o (Z,, (), to),) (2, t). where (=)()) and ao(-) model the mean and variance necessary for the stochastic reverse transition from Z₁ to Z4-1, respectively. To rigorously ensure that the malicious packets are not only recovered but also convincingly mimic normal packets, our model optimizes specially formulated variational lower bound LVLB. This objective underscores the model's 'ss effectiveness in blending malicious packets within normal packets:
guiding
min LVLB min
2 fo(z,, t)| + [|EMB(S hennmall- /E
fo(zi, 1)]1 log ps beod mai
Zu)
(4)
This objective is particularly focused on accurately reconstructing the initial state zo from the noised states, with distinct emphasis ensuring that the malicious components are scamlessly integrated into the normal packets pattern.
Preserving Packets' Integrity. We propose an innovative approach that utilizes a parallel data structure called "model dictionary" that clearly distinguishes between mutable and immutable fields within each packet. By categorizing fields into mutable (e.g. TCP flags, TTL, and window size) and immutable (including the critical 5-tuple information and payload), we ensure that modifications during training do not compromise the packet's 's integrity. The model dictionary serves dual purpose: it retains the original values of immutable fields and establishes a link to their mutable counterparts. This approach allows for dynamic modification of mutable fields during training without affecting

---

the core attributes of the packet. After training, any changes made to mutable fields are seamlessly integrated with the preserved immutable fields. This recombination ensures that the modified packets maintain their operational integrity and are indistinguishable from genuine traffic in real-world scenarios. By bypassing traditional domain constraints- -which typically complicate training and pose convergence challenges- our method simplifies the training process and obviates the need for additional compensatory loss functions (Letcher [2021]). The outcome is a highly effective generation of adversarial packets that are capable of evading detection without compromising the essential characteristics of the original packets.
4 Experiments
We conduct experiments to validate the performance of classifiers.
DiffuPac on types of attacks, against
4.1 Experimental Setup
Dataset. The datasets used for this model are Kitsune Dataset (Mirsky et al. [2018]) and CICIDS- 2017 Dataset (Sharafaldin et al. [2018)). Initially, pre-training of the BERT model utilizes large subset of unlabeled network traffic to leverage the model's capability to capture diverse traffic patterns, accounting for 60% of the total data. Fine-tuning phase, the focus shifts to a smaller, labeled dataset, which constitutes 20% of the total data. This dataset is distinctly partitioned into malicious and normal packets, which is crucial for training the model mimic malicious packets as normal. Training the classifier and NIDS then utilizes another 10% of the total data consisting of labeled portion. Testing phase is conducted with the remaining 10% of the data, reserved exclusively for evaluating the model's 's efficacy. This phase includes testing both original and mimicked malicious packets to rigorously assess the model's real- world applicability and its capability to generalize across unseen data.
Baseline Models. We evaluate two baseline models-Trrafic Manipulator (Han et al. [2021]) and TANTRA (Sharon et al. 2021])-bothh of which represent current strategies for evading NIDS but share significant limitation in their reliance on specific operational assumptions. Traffic Manipulator formulates the evasions as a bi-level optimization problem, where the first level uses GAN to identify adversarial features closely mimicking the original network features, and the second level utilizes PSO to modify packet characteristics to exploit vulnerabilities in NIDS. TANTRA employs a three-step process involving an LSTM trained to model benign traffic by predicting inter-pate satee- delays. This LSTM then adjusts malicious traffic to mimic these benign patterns before deployment. Both models, however, assume a degree of accessibility to NIDS configurations or feature extractors that is often unrealistic in practical scenarios. This reliance specific knowledge about NIDS configurations creates a critical dependency that can limit the deployment and scalability these evasion techniques in diverse operational environments. Conversely, DiffuPac operates under the assumption of zero prior knowledge about NIDS configurations or feature extractors.
Implementation Details. DiffuPac leverages BERT model with 12 transformer blocks, each featuring 12 attention heads and an embedding dimension of 768, to capture nuanced relationships in network traffic. This setup supports the maximum sequence length of 512. The diffusion model incorporates time step embedding to provide temporal context, enhancing the model's 's understanding of the reverse process. To manage the computational demands, particularly during the sampling phase, a projection layer reduces the embedding dimension to 128. Additionally, DiffuPac utilizes the FAISS library (Douze et al. (2024]), renowned for its capability to handle large-scale similarity search and clustering of dense vectors.
Experimental Details. Our experimental framework integrates advanced feature extraction tools and diverse set of machine learning classifiers, building on methodologies established in the Traffic Manipulator research. We employ 2 feature extractors: Afterlmage (Mirsky et al. [2018]), which provides detailed packet statistics, and CICFlowMeter (Draper- Gil et al. [2016]). which assesses connection- level metrics. Recognizing the documented shortcomings of the original CICFlowMeter, our research employs a revised version that integrates enhancements and corrections proposed in recent studies (Engelen et al. [2021], Liu et al. [2022]). These modifications are crucial in ensuring more accurate and reliable feature extraction for our analysis. Afterlmage and CICFlowMeter offer comprehensive view of network traffic, capturing both packet-level and flow-level data essential
7

---

Table 2: Comparative analysis of attack detection and evasion rates.
(a) Botnet
(b) MITM
Feature Extractor
Classifier
Detoction
Evasion Rate (MER)
Feature Extractor
Classifier
Detection
Evasion Rate (MER)
GAN
GAN
P
R
F1
&
LSTM Ours
P
R
FL
&
LSTM Ours
PSO
PSO
KINETT
0.84 0.94 0.92 37.24% 50.69% % 46.48%
KONNTT
0.92 0.94
0.91 38.22% 44.39%
5387%
CIC
DT
0.79
0.91 0,82 35,88% 49.62% 60.98%
CIC
DT
0.74
0.79
0.76 49.98%
57.77% 64.12%
FLowMeier
0.99
0.90 0.95 39.78%
42.93% 49.71%
FLowMeter
IF
0.99 0.92 0.94 26.74%
38.64% 52.99%
MLP
0.92 0.84 0.86 41.05%
53.87% 63.05%
MLP
0.77 0.72 0.74 52.07% 43.19%
73.21%
SVM
0.99 0.92 0.95
79%
91.95% 64.49%
SVM
0.74 0.79 0.78 42.11%
45.54% 60.46%
LR
0.84 0.91 0.89 24.72%
30.52% 42.08%
I.R
0.73 0.78 0.72 35.87% 50.04% 47.48%
KinNET
0.96 0.90 0.94 99.1 19.18%
99.79% 74.46%
KinNET
0.94 0.96 0.93 68.79%
79.29% 58.48%
DT
0.79 0.90 0.84 63.42% 67.30% 72.13%
DT
0.75 0.89 0.84 53.18% 58.15% 70.04%
Afterlmage
Afterimage
IF
0.99 0.90 0.94 31.48% 61.08% 52.79%
IF
0.81 0.83 0.86 26.53%
31.27% 45.71%
MLP
0.97 0.97 48.60%% 60.97% 64.92%
MLP
0.92 0.90
50.65%
59.30% 71.45%
SVM
0.99
0.90
0,94 40.311 51.24% 69.19%
SVM
0.99
0,90
0.94 63.51%
$7.51% 66.53%
LR
0.96 0.90 0.93 53.28% 50.70% 58.98%
LR
0.91 0.94 0.90 44.68% 46.39% 52.38%
for nuanced analysis. These feature extractors are integral to NIDS detection as they preprocess network data into structured features that are then analyzed by ML classifiers. This preprocessing step is crucial for transforming raw network traffic into a format that classifiers can effectively interpret, thereby enhancing the accuracy of the anomaly detection. We utilize a variety of machine learning classifiers for anomaly detection, including KiNET (Mirsky et al. ()), an ensemble of autoencoders; Multi-Layer Perceptron (MIP) for deep learning; Logistics Regression (I.R), Decision Tree (DT), and Support Vector Machine (SVM) for traditional approaches; and Isolation Forest (IF) for outlier detection. Our experiments cover 6 types of attacks-Marn -Man-in- Maaa-in-thh--aiee Middled (MITM), Botnet, Brute Force, DDoS, Port Scan, and Infiltration-utiine utilizing data from Kitsune and the CICIDS-2017 dataset.
Evaluation. To assess the effectiveness of the mimicked packets in evading the 6 classifiers, we employed a singular. highly illustrative metric from the Traffic Manipulator: the Malicious traffic Evasion Rate (MER). This metric is calculated using the formula: MER 1 - ( (Neddv /Nual), where Nrady and Nimal represent the number of detected adversarial and detected malicious packet sequences, respectively. This equation captures the percentage of adversarial packet sequences that goes undetected as compared to the original malicious packet sequences that was detected, effectively measuring the evasion capability of the adversarial packet sequences.
We employed the two-sample Colmogorov-Sminoo (K-S) test, a powerful parammaaettiic method used to determine the probabilistic differences between two data samples. Specifically, we compare the empirical cumulative distribution functions (cCDFs) of both the original malicious packets and the adversarial packets generated by our model. The K-S test calculates the maximum distance (K-S statistic, D) between these two eCDFs. This statistic measures the greatest deviation between the distribution of our adversarially modified packets and the original malicious packets (Hore et al. [2023]; Gretton et al. [2012]). A D value exceeding the critical threshold at chosen significance level suggests a rejection of the null hypothesis that is, the two samples are not derive from the same distribution. We demonstrated adversarial packets that were found to be -ustof-diitributtin (OOD) at 95% significance level.
Additionally, we used Wireshark, a widely recognized network protocol analyzer to ensure that during the modifications were made, the immutable fields crucial for the packet's 's attack functionality were preserved. This visual verification via Wireshark confirms that our adversarial packets maintain their structural integrity and functionality, successfully mimicking genuine network traffic while evading detection (Due to limited space, results and analysis are in Appendix B.
We also evaluated DiffuPac's 's efficacy in generating adversarial packets that maintain their malicious functionality in a controlled test environment using UTM hypervisor technology. This setup included two virtual machines: one running Kali Linux (attacker) and another running Ubuntu 24.04 (victim). The isolation ensured a realistic yet controlled network environment. We conducted the evaluation on two type of attacks: Port Scan and Brute Force. (Due to space constraints, results and analysis of the two attacks are in Appendix 1 and B.3.2).
8

---

4.2 Results and Analysis
Evasion Rate. We evaluated the evasion rates of the 6 attacks using DiffuPac and two baseline models, and compared the results in Table 2 (Due to space constraints, the other 4 attacks are demonstrated in Appendix B.2). As shown in Table 2, we conclude that DiffuPacc is capable of generating adversarial packets that achieves comparable or even higher in evasion rate compared with both Traffic Manipulator (GAN & PSO) and TANTRA (LSTM). Importantly, DiffuPac achieves this high performance without relying on NIDS components, surrogate classifiers or specific insights into the NIDS's ' feature extraction methods, highlighting its effectiivneess in realistic settings where attackers lack access to such insider information. The evasion performance varied significantly across different classifiers, reflecting the inherent variability in their robustness and detection capabilities. Flow-based NIDS exhibited greater resilience against our attacks compared to packet-based NIDS, likely due to Diffu- Pac's focus on packet-level modifications. Interestingly, the Traffic Manipulator model performed well with KitNet's Afterlmage feature extractor, likely because it was trained directly on these features, enhancing its capability to mod- ify them to evade detection. This specific alignment with NIDS features, while effective, does not typically reflect real-world attacker capabilities or constraints.
Table Percentage of successful ad- versarial samples found to be OOD.
Percentage of OOD Samples (%)
Attack Type
Botnet MITM
48.72
29.34
Port Scan
55.23
DDoS
78.14
TANTRA also demonstrated high evasion rates, particularly with AfterImage, thanks to its use of LSTTM models trained on benign traffic from the targeted network. This training allows TANTRA to reshape the timing between packets. This specific adjustment of interpacket delays directly impacts the time-based features extracted by AfterImage, making it more difficult for the classifiers to distinguish between normal and malicious packets.
Infiltration
58.06
Brute Force
69.67
Quantitatively, DiffuPac outperformed Traffic Manipulator by an average of 9.12 percentage points and TANTRA by average of 4.26 percentage points across all attack types. The overall average improvement of DiffuPac over both baselines is 6.69 percentage points, demonstrating its superior capability generate adversarial packets that effectively evade detection.
Based on the results of all the attacks, we can conclude that ML classifier are more robust than DNNs to adversarial attacks. Traditional ML models, like DT and IF, have simpler and more interpretable decision boundaries, making them less susceptible to subtle adversarial modifications (Sauka et al. [2022]). In contrast, DNNs with their complex and nigh-dimensional decision boundaries are more easily misled by the nuanced modifications introduced by DiffuPac.
Statistical Difference (K-S test). The percentage of successful adversarial packets that were found be OOD are demonstrated in Table From the result, we can obeserve that Brute Force and DDoS attacks show the highest percentage of OOD samples. The grounded reasons for these kind of attacks are due to the inherent nature of their traffic patterns. The nature of Brute Force attacks involves numerous failed login attempts, leading to highly irregular sequence numbers (Javed and Paxson (2013]). Even after modifications by DiffuPac, these attempts will stand out due to their frequency and pattern, resulting in significant deviations in the eCDFs. The sheer volume and repetitive nature of DDoS traffic (Haseeb-ur rehman et al. [2023]) make it difficult to disguise effectively. Despite efforts of intelligent mutation process, the high volume and distinctive traffic patterns will cause substantial deviations in the eCDFs as well. On the other hand, the MITM show the lowest percentage of OOD samples. Adjusting TTL values is straightforward they can be set to typical ranges seen in normal packets without significantly altering packet behavior, resulting in minimal deviation in the eCDFs.
5 Limitations
Our evaluations demonstrated DiffuPac's strong evasion capabilities, enabling the generation of adversarial packets that convincingly mimic legitimate network traffic, particularly in packet header fields. While these results highlight DiffuPac's ability to evade detection by NIDS, there remain areas for further study.

---

In particular, while we assessed the malicious functionality of generated adversarial packets in controlled environment using UTM hypervisor technology. this evaluation was limited to Port Scan and Brute Force attacks. Although DiffuPac successfully retained the malicious intent of these attacks, further evaluations are required to determine its effectiveness across broader range of attacks, especially those with more complex behaviors such as DDoS or Botnet attacks.
We also noted that DiffuPac's ' evasion performance not uniform across all scenarios, notably DDoS and Brute Force. This variability is due to the distinct nature of each attack's 's traffic patterns, which may not be fully captured by the model, especially in highly repetitive or anomalous behaviors, To enhance the model's s adaptability and generalizability, we plan to enrich the training dataset with a broader spectrum of real-world attack scenarios.
Moreover, the dual-use nature of adversarial generation models like DiffuPac presents significant ethical and legal challenges. While designed to improve security defenses, these technologies could be misused for malicious activities, such as facilitating cyber-attacks or disrupting services. To mitigate these risks, we have opted not to publicly release saved model checkpoints, aiming to prevent exploitation by malicious entities and ensure that our advancements in adversarial packet generation used responsibly and within ethical bounds.
6 Conclusion and Future Directions
In this study, we introduced DiffuPac, an intelligent generative model that successfully generates adversarial packets capable of evading advanced NIDS while maintaining attack functionality. Unlike previous research, which often assumes attacker access to NIDS, DiffuPac operates under constraints of limited attacker knowledge, reflecting more realistic scenarios. Here, we present the in-depths insights obtained from this study: (a) DiffuPac uniquely combines normal and malicious packet sequences using contextual alignments, ensuring scamless integration into genuine traffic while employing these normal packet sequences guide the denoising process. Through the performance evaluations with various NIDS, our model achieved an average improvement of approximately 6.69 percentage points in evasion rate compared to the two baselines across all attack types. (b) Evasion rates varied notably across attack types, with DDoS and Brute Force attacks showing higher probabilistic differences due to their complex, repetitive nature. This highlights potential areas for further refinement in DiffuPac's approach to handling voluminous attacks. (c) Simpler ML models like DT and IF displayed surprising resilience due their less complex decision boundaries, limiting the effectiveness of DiffuPac's modifications. In contrast, DNNs, with their intricate decision boundaries, more vulnerable, underscoring the complexities inherent in designing robust adversarial tactics. (d) DiffuPac's s capability to balance sophisticated attacks with operational stealth makes it especially suitable for environments where attackers lack comprehensive NIDS configurations, enhancing its utility in realistic defense testing.
As future studies, will compare DiffuPac against broader generative adversarial packets model and extend testing across more diverse NIDS to better understand its relative strengths and limitations. Next, we will further expand the range of attack types included in the malicious functionality evaluation. Moreover, adversarial defenses are a key focus of our ongoing research, particularly in relation DiffuPac's capabilities. One promising direction is the development of adversarial purification using diffusion models, a technique successfully applied image processing. This approach treats adversarial perturbations as noise, utilizing the diffusion model's reverse process to restore network packets to their original state before analysis by NIDS. We believe this method has significant potential for enhancing network security and aim to further develop and test it as a novel defense strategy.
Acknowledgements
This work has been partially supported by KAKENHI Grant Number JP23H03379.
10

---

References
Statista. Number of Internet of Things (IoT) connected devices worldwide from 2019 to 2023, with forecasts from 2022 to 2030. https: hopps://wwr /www: statista wtt.siistt.omstttiitttstse wtt.///taaisiiss//iieaeeaeee 1ot-connecten wow.ettedddiiics-woorllle
2023. Accessed: 2024-4-20.
Tala Talaei Khoei and Naima Kaabouch. Machine learning: Models, challenges, and research directions. Future Internet, 15(10):332, 2023. doi: 10.3390/h155100332
Tala Talaci Khoei, Hadjar Ould Slimane, and Naima Kaabouch. Deep learning: Systematic review, models, challenges, and research directions. Neural Computing and Applications, 35(31):23103- 23124, 2023. doi:
Olakunle Ibitoye, Rana Abou Khamis, Ashraf Matrawy, and M. Omair Shafiq. The threat of adversarial attacks on machine learning in network security - A survey. arXiv, 2019. URL
02621.
Hassan Khazane, Mohammed Ridouani, Fatima Salahdine, and Naima Kaabouch. A holistic review of machine learning adversarial attacks in IoT networks. Future Internet, 16(1):32, 2024. doi:
12.390/6160100333
Ivan Homoliak, Martin Teknøs, Martin Ochoa, Dominik Breitenbacher, Saeid Hosseini, and Petr Hanacek. Improving network intrusion detection classifiers by non-payload-band based exploit- independent obfuscations: An adversarial approach. EAI Endorsed Transactions on Security and Safety, 5(17), 12 2018. doi: 10.4108/eai. 11-1000-101011 1--2119.11154444
Mohammad J. Hashemi, Greg Cusack, and Eric Keller. Towards evaluation of nidss in adversarial setting. In Proceedings of the 3rd ACM CoNEXT Workshop on Big DAta, Muchine Learning and Artificial Intelligence for Data Communication Networks, 2019.
Aditya Kuppa, Slawomir Grzonkowski, Muhammad Rizwan Asghar, and Nhien-An Le-Khac. Black box attacks on deep anomaly detectors. Proceedings of the 14th International Conference on Availability, Reliability and Security (ARES 2019), 2019.
Dongqi Han, Zhiliang Wang, Ying Zhong, Wenqi Chen, Jiahai Yang, Shuqiang Lu, Xingang Shi, and Xia Yin. Evaluating and improving adversarial robustness of machine learning-based network intrusion detectors. IEEE Journal on Selected Areas in Communications, 396))2663-2267,, doi: 10.110//SAC.222133082224
Yam Sharon, David Berend, Yang Liu, Asaf Shabtai, and Yuval Elovici. TANTRA: Timing-based adversarial network traffic reshaping attack. IEEE Transactions on Information Forensics and doi:
Security, 17:3225-322 3237, 2021.
Soumyadeep Hore, Jalal Ghadermazi, Diwas Paudel, Ankit Shah. Tapas K. Das, and Nathaniel D. Bastian. Deep PackGen: A deep reinforcement learning framework for adversarial network packet generation. arXivv 2023. URL https: //trxiv.ivvv////////// org/ abs//b///55. 11039.
Zeeshan Ahmad, Adnan Shahid Khan, Cheah Wai Shiang, Johari Abdullah, and Farhan Ahmad. Network intrusion detection system: systematic study machine learning and deep learning approaches. Transactions Emerging Telecommunications Technologies, 32( ee4400, 2021. doi: https://doi.or//10. 10000/eett..15550
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun, editors, Proceedings of the 2nd International Conference Learning Representations (ICLR 2014), 2014.
Andrew McCarthy, Essam Ghadafi, Panagiotis Andriotis, and Phil Legg. Functionality- adversarial machine learning for robust classification in cybersecurity and intrusion detection domains: A survey. Journal of Cybersecurity and Privacy, 201):154-190 2022. doi: 10.3390/ jep2010010.
11

---

Kaichen Yang, Jianqing Liu, Chi Zhang, and Yuguang Fang. Adversarial examples against the deep learning based network intrusion detection systems. In Proceedings of 2018 IEEE Military Communications Conference (MILCOM 2018), pages 559-564, 2018. doi: 1.11099/IL.CMM 2018.8599759.
Ryan Sheatsley, Nicolas Papernot, Michael J. Weisman, Gunjan Verma, and Patrick McDaniel. Adversarial examples for network intrusion detection systems. Journal of Computer Security, 30 (5):727-752, 2022.
doi: 10.3233/JCS-210000
Ke He, Dan Dongseong Kim, and Muhammad Rizwan Asghar. Adversarial machine learning for network intrusion detection systems: A comprehensive survey, IEEE Communications Surveys & Tutorials, 25())533-566, 2023. doi: 10.1 1011099CCMSS12222.333399
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceed- ings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.
Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Proceedings of Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 6840-6855 2020.
Xinjie Lin, Gang Xiong, Gaopeng Gou, Zhen Li, Junzheng Shi, and Jing Yu. Et-bert: A contextualized datagram representation with pre-training transformers for encrypted traffic classification. In Proceedings of the ACM Web Conference 2022 (WWW 2022). ACM, 2022. doi: 10.1 10/1445/348 3512217.
Xuying Meng, Chungang Lin, Yequan Wang, and Yujum Zhang. Netgpt: Generative pretrained transformer for network traffic. arXiv, 2023. URL /sarxi///rriv...r www.ooggg///baaeeeeeoooeoe hat///ab///2..00./ 09513.
Yonghui Wu, Mike Schuster, Z. Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason R. Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv, 2016. URL https: // /arziv.ivva org/ abs 110//6669.000 08144.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), pages 4171-4186. ACL, 2019. doi: 10. 18653/v I/N19-1422
Chuanpu Fu, Qi Li, Meng Shen, and Ke Xu. Realtime robust malicious traffic detection via frequency domain analysis. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Commu- nications Security (CCS 2021), pages 3431-3446. ACM, 2021. doi: 10.1 10.1145/3400120.448555
Jianang Dai, Xiaolong Xu, and Fu Xiao. Glads: global-lowe local attention data selection model for multimodal multitask encrypted traffic classification of IoT. Computer Networks, 225: 109652, 2023. doi: https://doi.org// 10166/. (t6f)..ommet.222.100999
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- LM improves controllable text generation. In Proceedings of Advances in Neural Information Processing Systems 35 (NeurIPS 2022), pages 4328 4344,,333.11 2022.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong, DiffuSeq: Sequence to sequence text generation with diffusion models. In Proceedings of the 11th International Conference on Learning Representations (ICLR 2023), 2023.
Alistair Letcher. On the impossibility of global convergence multi-loss optimization. arXiv, 2021.
httpstts::/ http://axvvvvv/eeeeeeeeeeeaaa orw. wttp///////...o../////////////o 12649.
12

---

Yisroel Mirsky, Tomer Doitshman, Yuval Elovici, and Asaf Shabtai. Kitsune: An ensemble of autoencoders for online network intrusion detection. In Proceedings 2018 Network and Distributed System Security Symposium (NDSS 2018), 2018.
Iman Sharafaldin, Arash Habibi Lashkari, and Ali A. Ghorbani. Toward generating new intrusion detection dataset and intrusion traffic characterization. In Proceedings of the 4th International Conference Information Systems Security and Privacy (ICISSP 2018), pages 108-116. INSTICC, SciTePress, 2018.
doi: 10.520000066399000801111
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. arXiv, 2024. URL https:///axii: org/ aba/ 2201. .82281.
Gerard Draper- Arash Habibi Lashkari, Mohammad Saiful Islam Mamun, and Ali A. Ghor- bani. Characterization of enerypted and vpn traffic using time-related features. In Interna- tional Conference on Information Systems Security and Privacy (ICISSP 2016), 2016. doi:
10.52200005440000004444.
Gints Engelen, Vera Rimmer, and Wouter Joosen. Troubleshooting an intrusion detection dataset: the cicids2017 case study, In 2021 IEEE Security and Privacy Workshops (SPW), pages 7-12, 2021. doi: 10.1 10.1109/SPW53761.22222 20021000066
Lisa Liu, Gints Engelen, Timothy Lynar, Daryl Essam, and Wouter Joosen. Error prevalence in nids datasets: A case study cic-ids-2017 and cse-cic-ids-2018. In 2022 IEEE Conference Communications and Network Security (CNS), pages 254-262, 2022. doi: 10.1 10.1199//NS5611 14.
2022.9947235.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723-773, 2012.
Kudzai Sauka, Gun Gun-Yoo Shin, Dong-Wood Wook Kim, and Myung-Mook Han. Adversarial robust and explainable network intrusion detection systems based on deep learning. Applied Sciences, 12(13), 2022. doi: (0.3390/app1213645
Mobin Javed and Vern Paxson. Detecting stealthy, distributed SSH brute-forcing. In Proceedings of the 2013 ACM SIGSAC Conference on Computer and Communications Security (CCS 2013), pages 85-96, New York, NY, USA, 2013. ACM. doi: 10. 0t555/2508859.2511777
Rana M. Abdul Haseeb-ur rehman, Azana Hafizah Mohd Aman, Mohammad Kamrul Hasan, Khairul Akram Zainol Ariffin, Abdallah Namoun, Ali Tufail, and Ki-Hyung Kim. High-speed network ddos attack detection: A survey, Sensors, 23(15), 2023. doi: 10.3390/s2315681
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems (NIPS 2017), pages 6000-6010, 2017.
Zhe Zhao, Hui Chen, Jinbin Zhang, Xin Zhao, Tao Liu, Wei Lu, Chen, Haotang Deng, Qi Ju, and Xiaoyong Du. Uer: An open-source toolkit for pre-training models. EMNLP-IJCNLP 2019, page 241, 2019.
13

---

A Model Architecture
A.1 BERT
The core of DiffuPac is the BERT model, specifically adapted to serve the denoising component in the diffusion process. The architecture consists of 12 bidirectional Transformer blocks and each block incorporates 12 attention heads within its self-attention layers (Vaswani et al. [2017]). These layers are crucial for capturing the intricate and implicit relationships between traffic bytes. Each input token has a dimension of 768, and the model can process sequences with up to 512 tokens. This setup allows the BERT model to effectively handle extensive sequences of traffic data, making well-suited for the task of denoising during the diffusion process.
To implement of our pre-trained BERT model, we utilized the codebase from the UER (Universal Encoder Representations))-yy project (Zhao et al. [2019]). UFR-py is versatile toolkit designed for pre-training and fine-tuning various NLP models, including BERT. In UER-py, the MLM designed to predict masked words within a sentence. We adapted this approach to our masked unidirectional flow model, which focuses on predicting masked tokens within network traffic sequences. Instead of textual tokens, our model deals with traffic units from src-to-dst and dst-to-src sequences. We modified the MLM implementation to handle these traffic tokens, ensuring that each token is masked with probability of 15%. Our model then predicts the original token from the masked sequence, capturing semantic patterns within bidirectional network flows.
The Next Sentence Prediction (NSP) task in UER-py py involves prediction step to determine whether one sentence follows another. We transformed this concept into the same sequence-origin prediction task to determine the origin of directional network traffic. We adapted the binary classifier used in NSP to classify whether packets are originated from src-to-dst or dst-to-sre. By pairing packets and predicting the directional flow, model learns the contextual relationships and patterns in network traffic, similar to how NSP learns sentence relationships.
Table 4 lists the parameters are set to run our pre-trained BERT model. The remaining parameters were set to their default values.
We also conducted a comprehensive evaluation different BERT model configurations to identify the most efficient option that still maintains high performance. We tested four BERT model variants: Tiny, Small, Medium, and I Large, comparing their specifications, training time, loss, and accuracy. The results summarized in Table 5. The Medium BERT model was chosen the optimal configuration for DiffuPac because it offers a balanced trade-off between accuracy and loss metrics.
Table 4: Parameters used for pre-training the BERT model.
Parameter
Value
Embedding Size
768
Feedforward Size
3072
Hidden Size
768
Hidden Activation Function
GELU
Number of Attention Heads
12
Number of Transformer Layers
12
Max Sequence Length
512
Dropout Rate
0.1
Batch Size
64
Total Steps
500000
Learning Rate
le-4
A.2 Fine-tuning with Diffusion Model
For the fine-tuning phase of our model, we leveraged the DiffuSeq codebase (DiffuSeq's code). The DiffuSeq framework, originally designed for text data, required several significant adaptations to handle network traffic data and to meet the objectives of our diffusion model for adversarial packet generation. The primary adaptation to the DiffuSeq codebase involves the data preparation phase,
14

---

Table 5: Comparison of BERT Models.
Parameters / Metries
Tiny BERT
Small BERT
Medium BERT
Large BERT
Model Specifications
Embedding Size
128
512
768
1024
Feedforward Size
512
2048
3072
4096
Hidden Size
128
512
768
1024
Activation Function
GELU
GELU
GELU
GELU
Number of Heads
8
12
Number of Layers
4
12
24
Training
Training Time (hours)
13.5
18
23
30.5
Performance
MUM Loss
1.028
0.822
0.667
0.610
SSP Loss
0.295
0.204
0.099
0.061
MUM Accuracy
0.844
0.887
0.905
0.914
SSP Accuracy
0.846
0.902
0.972
0.982
specifically the concatenation strategy used to blend normal and malicious packet sequences. This adaptation is crucial for the model to learn how to generate adversarial packets that can mimic normal packets and evade detection by NIDS.
In our concatenation strategy, we identified pairs of normal and malicious packet sequences that exhibit strong contextual alignments unlike DiffuSeq, which, randomly merge the text sequences. This means that the patterns and behaviors within these sequences are similar, ensuring that the concatenated sequences are blended seamlessly. We concatenated these contextually aligned normal and malicious packet sequences to form a new sequence, Sben@mal, The strategy leverages the inherent patterns and behaviors within normal packet sequences, which is crucial for the mimicking process of malicious packet sequences.
We utilized the pre-trained BERT embeddings in our concatenation strategy. BERT is designed to capture deep semantic relationships within sequences. This design makes BERT making highly effective for analyzing complex patterns in network traffic. By employing the pre-trained embeddings, DiffuPac can accurately transform discrete packet data into continuous feature spaces, capturing the intricate dependencies between packets. This transformation is critical for the subsequent similarity analysis and pairing process. The pre-trained embeddings offer a rich representation of the packet sequences, so that the model can discern subtle contextual similarities between normal and malicious packet sequences. The detailed of the concatenation strategy is demonstrated Algorithm 1.
Here, we set the similarity threshold, to 0.9. This is a strategic decision aimed at balancing the need for high contextual alignment with the practicalities of ensuring sufficient number of matched pairs. Setting to 0.9 indicates that we are looking for pairs of packet sequences that are highly similar, capturing nearly all the essential contextual information shared between normal and malicious packet sequences. This high threshold ensures that the concatenated sequences are blended seamlessly, maintaining the realistic traffic patterns and behaviors required evade detection by NIDS. From practical standpoint, threshold of 0.9 is chosen because it strikes an optimal balance between precision and recall. higher threshold (closer to 1.0) would result in fewer matched pairs, as only the most similar sequences would be selected. This could limit the model's ability to generate a diverse set of adversarial packets. Conversely, a lower threshold would increase the number of matched pairs but might include sequences that are not contextually well-aligned, reducing the effectiveness of the adversarial packets in mimicking normal traffic.
During the reverse process, we introduced guidance mechanism using normal packet sequences. This mechanism treats the concatenated sequences of normal and noise-added malicious packets as a unified entities, allowing the model to denoise the malicious packets in the context of the normal packet sequences. We recalibrated the variational lower bound LVL.B to emphasize the integration of normal sequence guidance. The objective focuses on accurately reconstructing the initial state, oo from the noised states, so that the malicious components are seamlessly integrated into the normal packet sequences pattern.
15

---

Table 6 lists the parameters used to run our diffusion model. The other parameters are set according the default values of DiffuSeq. All training, experimentation, and sampling processes are executed on single NVIDIA ADI02 (GeForce RTX 4090) GPU. The training process for BERT required approximately 23 hours, while fine-tuning with the diffusion model took roughly 10 hours.
Table 6: Parameters used for the diffusion model.
Parameter
Value
Diffusion Steps
2000
Learning Rate
le-4
Learning Steps
50000
Seed
102
Noise Schedule
sqrt
Batch Size
64
Microbatch
64
Sequence Length
128
Hidden Time Dimension
128
Hidden Dimension
128
Schedule Sampler
uniform
Algorithm Finding contextually relevant packet sequences.
Require: Normal packet sequences Shen Malicious packet sequences Smaal Model parameters A
Similarity threshold e
Ensure: Matched pairs of normal and malicious packet sequences (Shen, sypal)
Initialize: Load pre-trained BERT model with parameters
2: Embed Normal Sequences:
3: for each sequence Shen in Shen do
4: Compute embedding Eben EMB(S)en 8)
5: end for
6: Embed Malicious Sequences:
7: for each sequence Syual in Sual do
8: Compute embedding E EMB(S 0)
9: end for
Find Contextually Aligned Pairs:
11: for each embedding E in Emall do Initialize best match by <- None
13: Initialize highest similarity Fmax 6-1
for each embedding Epen in Eben do
Compute similarity dij <- cosine -esssooossiillleeeeeee similarity (Emal E.onn 16: if Jij > mmx then
17:
Update best match by + Shen Update highest similarity daax +- Oij
19: end if
end for
21: if T
> then
22: Add pair (bj, Stial) to matched pairs
23:
end if
24:
end for
25:
Return Matched Pairs: Return all matched pairs
16

---

B Experimental Results
B.1 Wireshark Results
2130000
01:00:0:4
14.11
Freent: oi
190 UOC AD 14
1E.MA1
00.39
2BH (ND 2111
at De 5016, 01 letel (1h-- 1011
Oeteer mieaai waite 2011 Rd: ac 10.11.2
denri fonerrs lets 1861, Sey Ale
neettu cerii meead in 1911 Mnn,, 11f nitt du 1e0
Nolia Neli
Siren idee
renfateee Develite
laper LP1 1I
Seakcs baleer
HOUS14 oebert
Sapato kutte::
(1cathe Higeee catert)
emnotapaen satur 1ie Lonne
Neve 30 Catu 18
DEAAEER NOUI BES1
Decual Saick (uee
Nears 12 Niti, Oa baft, Ne-gee 3011
(a) Botnet attack before in Wireshark.
(b) Botnet attack after in Wireshark.
Figure 2: Comparison of Botnet attack before and after in Wireshark.
To demonstrate that our model intelligently alters specific fields to seamlessly blend malicious packets into normal traffic, we analyzed the packets using Wireshark. Our analysis covered 6 attack types: MITM, Botnet, Brute Force, DDoS, Port Scan, and Infiltration. The results showed that different specific fields were modified in the packets corresponding to each type of attack. This variation in field alteration confirms that model adapts its modifications according to the nature of the attack, enhancing the stealthiness of the malicious packets. In Figure 2, we can see that there is a change in the window size of Botnet packet. Botnet attack can be described (bots) controlled by an attacker to perform various malicious activities. Botnet traffic often shows abnormal patterns in window size due automated control and data bursts. Here, we can conclude that DiffuPac made the intelligent decision by itself in modifying the window size to fall within typical user traffic patterns. This helps to camouflage the botnet activity.
network of infected devices
MITM, on the other hand, demonstrated attacks whereby an attacker intercepts communication between two parties without their knowledge, often to cavesdrop alter the data being sent. Abnormal TTL values (time-to-live field) can reveal this kind of interception and redirection. Here, once again, model intelligently adjusts TTL value to match the expected values in aiding the masking of MITM presence. As shown in Figure 3, DiffuPac adjusts the TTL values to effectively conceal signs of interception. This ensures that the adversarial packets blend seamlessly with normal data exchanges, mimicking legitimate communication between parties.
DDoS attacks involve a large number of packets with similar characteristics, often reusing or predictably incrementing the IP Identification (ID) field. In Figure 4, it can be shown that DiffuPac addresses this by randomizing the IP IDs to prevent detection. When we analyzed the traffic using Wireshark, we observed varied and less predictable normal traffic patterns. This randomness helps avoid forming detectable patterns and evades detection systems that rely on identifying repetitive IP ID sequences. Thus, DiffuPac successfully masks the attack's presence while preserving the attack functionality.
IDs in the modified packets, aligning with
Port Scan often involve specific flags, such as SYN, to probe for open ports. This behavior is distinct from normal traffic, which uses a variety of flags. In Figure 5, we found that DiffuPac diversifies the use of flags to include SYN, ACK, and FIN, similar to normal connections. By normalizing the traffic in this way, DiffuPac helps disguise scanning activity. For example, instead of sending many SYN
17

---

4046
0360 0
21111
11984
144000 01
06/333
2 0220 0135
bv/rea Tort
Pru 2/27 No AL 03
MIL Iagu the
(a) MITM attack before in Wireshark.
(b) MITM attack after in Wireshark.
Figure 3: Comparison of MITM attack before and after in Wireshark.
M
11:89
18:333
11.102
11
12280
1:000
2.341
16.25 3
2 1250e
2,233
haar unde
enC
(a) DDoS attack before in Wireshark.
(b) DDoS attack after in Wireshark.
Figure 4: Comparison of DDoS attack before and after in Wireshark.
packets, it includes a mix of SYN, ACK, and FIN packets, which helps blend the scan into typical network behavior and evade detection.
Brute force attacks generate numerous login attempts, leading to irregular sequence numbers due to the repeated connections. By analyzing the traffic with Wireshark, we observed that PiffuPac intelligently normalizes sequence numbers to mimic regular user connections. Normal user login attempts typically show sequentially increasing sequence numbers. In Figure 6, it shows that DiffuPac adjusts these numbers to follow similar pattern, thereby camouflaging the brute force activity. This adjustment helps blend the attack packets seamlessly into benign traffic, making it harder for NIDS to spot the abnormality.
Infiltration attacks involve moving laterally within a network, often generating unusual acknowledg. ment numbers as the attacker accesses various systems. In Figure 7, it demonstrated that DiffuPac
18

---

A 4040
4046 8
(a) Port Scan attack before in Wireshark.
(b) Port Scan attack after Wireshark.
Figure 5: Comparison of Port Sean attack before and after in Wireshark.
4010
AAAM
19911
2/4
15:00
15.011
11:.0 24341
1.080
11
Fanh Stfee
LPC
(a) BruteForce attack before in Wireshark.
(b) BruteForce attack after in Wireshark.
Figure 6: Comparison of bruteForce attack before and after Wireshark.
intelligently modifies acknowledgment numbers to follow expected sequences. In normal communi- cation, acknowledgment numbers increment predictably based on the received data. DiffuPac ensures that infiltration attempts have acknowledgment numbers that follow these normal patterns, effectively masking the lateral movement within the network and blending the attack traffic with normal traffic, making it difficult for detection systems to identify the anomaly.
DiffuPac demonstrates its ability to modify specific fields intelligently in the packet headers of different attack types, such as the window size for botnet traffic and TTL for MITM, to blend malicious packets into normal packets. This capability consistently proven through detailed packet analysis with Wireshark. Indeed, the detailed packets analysis shows that critical fields defining packet identity and practicality (e.g., IP addresses, ports, payload) remain intact after modification, ensuring the functionality of the attack while evading detection by sophisticated NIDS. The comparisons
19

---

4046 10081
4
$1241
Scidda bu-daat
kade uS locurd tencheto
41100
PL:41
06.842
18
Sealle EC
(a) Infiltration attack before in Wireshark.
(b) Infiltration attack after in Wireshark.
Figure 7: Comparison of infiltration attack before and after in Wireshark.
for each attack type before and after the modificatinos further highlight DiffuPac's effectiveness in maintaining attack practicality while enhancing stealth.
20

---

B.2 Evasion Rate
We evaluated the evasion rates of the other 4 attacks DDoS. Port Scan, Brute Force, and Infil- DiffuPac and the baseline models, with the detailed results summarized in Table 7. According to the results, we can observed that the trends in the 4 attacks are consistent with the attacks shown in Table 2 in the main paper.
Table 7: Comparative analysis of attack detection and evasion rates of the other attacks.
(c) DDoS
(d) Port Scan
Feature Extractor
Classifier
Detection
Evasson (MER)
Feature Extractor
Classifier
Detection
Evasive (MER)
GAN
GAN
P
R
F1
& PSO
LSTM Ours
P
R
F1
& PSO
LSTM Ours
KRNEET
0.94 0.90
0.92 33.19% 41.69%
52.06%
KiNNTT
0.96 0.91 0.94 33.46% 40.0%% 55.03%
CIC
DT
0.73
0.81 0.72 39.92% 40.80%
46.81 %
CIC FLowMeier
DT
0.78
0.72
0.73
34.01%
61.23% 59.25%
FLowMeier
0.99 0.91
0.93 26.74%
19.49
51.45% %
IF
0.99
0.90
0.94
22.73%
29.49% 45.26%
MLP
0.72
0.71 0.74 52.03%
48%17%
69.87%
MLP
0.76
0,71 0.74
59.72%
57.56%
67.92%
SVM
58.47% 54.57%
SVM
68.91%
60.95%
LR
0.75
0.79
0.76
48.79%
51.08% 58.95%
LR
0.78
0.83
0.79
42.049
49.78% 54.21%
KiNNET
0.95
59.84%
74.06%
62.31%
KiNNTT
0.94
0.96
64.97% 55,87%
DTT
0.78 0.92 0.85
72.88 % 63.27%
59.47%
DYT
0.77 0.91 0.82 69.78%
62.01% 70.47%
AfterImage
AfterTmage
0.86 0.84 0,88
22.63%
36.33% 48.91% %
IF
0.84
0.91 0.89 18.38%
22.15% 48.30%
MLP
0.91 0.94 0.92 55, 55.45%
57.77% 63.25% %
MLP
0.96
0.94 0.97 49.73%
56.62% 64.02%
SVM
0.99 0.90 0.94 78.51% 79.97%
68.91%
SVM
0.97
0.96 0.98 76.40%
78.96% 75.81%
LR
0.97 0.91 71.68% % 69.95% 56.66%
LR
0.95 0.92
67.29%
70.07% 73.94%
(c) BruteForce
(f) Infiltration
Feature Extractor
Classiñerr
Detection
Evasive (MER)
Feature Extractor
Classifier
Detection
Evasive (MER)
GAN
GAN
P
F1
& PSO
LSTM Ours
R
F1
&
LSTM Ours
PSO
KiNET
0.97 0.94
0.97 36.94% 43.13% 50.32%
KiRNET
0.91 0.90 0.96 41.71% 40.68% 45.04%
CIC
DT
0.77
0.72
44.87% 42.79%
CIC
0.72
0.79
32.47%
62.67%
.eeee
IF
0.91
0.90
0.94 27.01
39.14% 41.64%
FLowMeter
IF
0.99
0.91
0.94
28.10%
42.19% 45.88%
MLP
0.88 0.85 0.83 52.32%
64.58% 67.18%
MLP
0.77
0.71
0.74
50.09%
53.68% 63.14%
SVM
0.80
0.81 0.72 47.31%
51.54% 56.74%
SVM
0.74
0.76
0.79
62.29%
64.51% 56.11%
LR
0.79
0.75 0.78 49.08%
58.77% 61.89%
LR
0.71 0.80
0,76
43.05%
50.67%
61.94%
KiNEET
0.96
0.93 0.95 81.84%
86.39% 69.31%
KiNETT
0.92 0.98
0.97
73.44%
70.58
51.26%
DT
0.82 0.79 0.78 41.07%
43.56% 55.07%
DT
0.77
0.91 0.82
69.23%
72.32%
62.54%
Afterlmage
Afterlnage
0.98 0.92
22.99% 39.71%
0.84
23.61%
27.51% 42.59%
ML.P
0.91 0.94 0.92 59.69%
63.00% 68.25%
MLP
0.96
0.94
0.97 50.65%
48.75 13.73% 72.25%
SVM
0.94 0.92 0.96 68.51 %
61.79% 58.93%
SVM
0.94
0.96
0.98
72.48
68.95%
64.73%
LR
0.97 0.93 0.94 61.68%
57.25% 64.06%
LR
0.93
0.92
0.95
66.04%
65.93% 68.37%
21

---

B.3 Malicious Functionality Evaluation
Experimental Setup. We conducted a robust evaluation of both Port Scan and Brute Force attacks to assess their malicious functionality in controlled, isolated network environments, The primary goal of this evaluation was determine whether adversarially modified packets retain their ability to perform malicious actions. Throughout the evaluation, we focused on analyzing the response of both the original and adversarial packets using Wireshark, carefully observing how the target system reacted to each type of attack.
B.3.1 Port Scan
Before proceeding to evaluate the generated adversarial packets of DiffuPac, we first demonstrated a successful Port Scan attack within our controlled environment. This initial step is fundamental step to ensure the validity and reliability of subsequent tests with both original and adversarial packets. The network configurations depicted in Figure 8(a) and Figure 8(b) confirm that both virtual machines were correctly placed within the same subnet. As demonstrated Scan successfully identified open ports the Ubuntu, highlighting port 22 as vulnerable.
Figure 8(c), the Port
We initiated a Port Scan attack from the Kali Linux machine targeting the Ubuntu server. During the execution of this attack, we captured all packets sent from the attacker to the victim, saving them as a peap file. This capture was timed precisely to begin as the Nmap tool launched the Port Scan, ensuring that only the relevant src-to-dst packets were recorded. This focus aligns with the operational framework of DiffuPac, which concentrates on modifying packets sent from the source.
Using the captured src-to-dst packets, we employed DiffuPac to generate adversarial Port Scan packets, which were then saved in separate peap file. These adversarial packets were subsequently replayed using the Tepreplay tool, directing them towards the Ubuntu server. Concurrently, we used Wireshark on the Ubuntu machine to capture the incoming responses, allowing us to analyze the effectiveness of the adversarial packets in real-time.
To ensure balanced evaluation, also replayed the original Port Scan peap file using Tepreplay, capturing the responses in Wireshark on the Ubuntu side. This dual replay enabled direct comparison between the packets' responses generated by the original and adversarial Port Scan packets.
As depicted Figure 9, the response to the adversarial Port Scan packets closely mirrors that the original Port Scan. A detailed inspection of the TCP flags reveals that packets with SYN and ACK flags targeted port 22, corroborating the initial vulnerability identified in Figure 8(c). This consistent response across both original and adversarial packets provides compelling evidence that DiffuPac successfully generates adversarial packets that retain their intended malicious functionality.
22

---

edat AARAAAAAA EHORIAA SOTN AARUL ate 33MM
Uet de reCnush 235
332, 248 215
iieth free TRoant offeefll 18266 prefiatre scapete fa28ell88
por (Laten sopurs DroRclirko
M. pREsets THE8 OUTeR ONESO (oh-3 #35)
pockets 1966 Intes VID.3 01155 183
errors, oropped CARPLLER Frane
11 aorkeno 1806 PyTer 82122 (nd. K18)
collisters
errors dropped cverrers cerrler coltisiers
1ust 12V U.S. nelinas 233 21.00.00
feat 127.8.8-1 rarnork 255.0.0.5
unets 11d praftstan acspetc Ex2l-tosts FX postets 2006 bysee 206306 (266.2 #33 errors spond PANPFAES Tram 1996
KL 9E2629 conoped 04452aR
frams
TX erreco crocped overeure
corrLer collisaara
TX errors cropped overrurs cernter colscators
a) Kali Linux network configurations
b) Ubuntu network configurations
Ls suda mnap -55 -T4 192. 58.
(a) [~]
[sudo] password for a1:
Starting Mnap 7.94SVN httos: mapporg at 2024-06-25 14:22 JST
Kmap scan report for 192-168. 56...66.888 182
088425 lateney).
Not shown: 999 closed tcp ports (ceset)
SERVICE
22/tcp open sah
Address: 77332147761122160 (Unknown)
done: address (1 hest seanned in 39 seconds
c) Successfully executed Port Scan using Nmap tool
Figure 8: Port scan attack demonstration on isolated environment
a) Original Port Scan Response
b) Adversarial Port Scan Response
Figure 9: Comparison of Victim Response demonstrated in wireshark
23

---

B.3.2 Brute Force
For the Brute Force attack evaluation, the experimental setup followed a similar approach to that of the Port Scan, focusing on capturing and analyzing both original and adversarial packets. However, in this case, we leveraged Metasploit's ssh_login module to perform the attack. After demonstrating successful Brute Force attack using Metasploit, we captured the relevant packets and utilized DiffuPac to generate adversarial versions of those packets. The evaluation focused on comparing the response of original and adversarial packets, observing how successful logins were achieved in both cases.
As shown in Figure 10, we utilized the Metasploit Framework to perform a Brute Force attack targeting the SSH service on the Ubuntu server. After setting the RHOSTS to the Ubuntu machine, we provided both a user file and password file containing potential login credentials. The login module systematically attempted each combination of usernames and passwords, ultimately leading to successful login.
The success of the Brute Force attack is demonstrated in Figure 10, where Metasploit successfully opened an SSH session on the target machine. This provided the necessary confirmation that the attack could be executed under normal conditions, laying the foundation for the subsequent evaluation of adversarial packets.
Following the successful attack, similar to the Port Scan setup, captured the src-to-dst packets exchanged during the Brute Force attack. These packets were saved as a peap file, which was subsequently used as input for DiffuPac to generate adversarial Brute Force packets.
Using Tepreplay, both the original and adversarial Brute Force packets were replayed towards the Ubuntu server. As shown in Figure 11, the response to the adversarial Brute Force packets closely mirrored that of the original Brute Force packets. The packet exchange between the attacker and the target server increased significantly as the correct login credentials were identified, culminating in successful SSH session for both the original and adversarial attacks.
To further analyze and validate the outcomes of both the original and adversarial Brute Force attacks, we utilized pam__mnt unix(sshd) on the Ubuntu server to log and compare successful login attempts. As depicted in Figure 12, the log entries confirmed that the SSH service recorded successful logins for both the original Brute Force attack and the adversarial Brute Force attack.
File Actions
Edit
View
Help
Actions
Edit View
CreateSession
truc
Epplotte:
20
coult: Create now session for every succossful, login
se CO_AAL__REEE
false
2a
Try wach want/ passward csup atored the current datab
ast
False
in current database the list
5R ALL USERS
false
add all users
08_ 28_SKIP_ EXISTING
none
10
5kip esisting credentials stared in the current database
Eiploit
(Accpptdd: none, user, userBreals)
PANSOWOR
80
sperificc passaurd to authertrate with File containing posswards, one line pp metasplost//eee httt/o/sss////sstttiii The target. port.
FALS_FILE
80
Lergel
www.wwttlt.t.......m.mm... /ww.//////////o//oo/// ittal
SPORT
22
Linus
STOP ONL SUCCESS
false
Limus
pms
Stop guessing when credential ial, works for host
THREADS
L1008
pes
The number of cancurrent threads (max one per host) 11000/10
USERNANE
40
A spocific ussenaaee to authenlicate
rantateing
separated by space,
'ine
USER AS PASS
false
80
Try the usernass the passaurs for all users
USER FILE
80
File containing usernanes, one line
VERBOSE
f31se
Stolico
yes
Rostl
whether to print output for alt atteapts
View full module Info with
Exft daxit bavttaarysscamer/// scanee//ss//stn login) set rhosts 192.168.56, 192
rhosts 192.158.56.182
rxft wuxil iwry( acameer/ hwtp///s/sshhaaaaaaaaaaaaaa Lngin) set user, file /hrew/ /a y /aek lherrs.tat
[itr_ /hano//3//exxtto//ccrr...
tadh
pasa_
f1.6 auxitiary( scamme///a///staaa Lagin) exploit
[+] 192, 168, 00.55...22 182:22 Starting bruteforce
[+] 188, 20.1220.222
Success: 'aj:illy" 'sid-1es((a)) a)) xid-188((3)) gecups- 33, 24 ,71(s
akpp), (i(guudee) 101(kee)
sjnatsn
-i---ergrrrr ci5-thurtu
May
e7:52:29 utc
särcné4 5ei seaxion opened 1992.16.56, 56..6. 1et (weses weses
sarches GMa/Linux
147, (2..100.9..18077)) 5874 07 13:33:50 +Ronn
[+] Scanned 1 of 1 hosts (1008. complete)
[+] Auxiliary module execution compleced
taft assiliery( (wc.iiryyyyccaayaaaaaaaa wwtpa///////sssaaaaaaaaaaaaaasaa laggis
Figure 10: Brute Force Attack Successfully conducted using metasploit
24

---

R C
1
71400
12605
10V, 1pmm P1
a) Original Brute Force
b) Adversarial Brute Force
Figure 11: Response Comparison of Original Brute Force and Adversarial Brute Force in Wireshark
sustirr aprtte and set, and autht suther momtun Welerle inplee vid wid the 4th
Pn suskiis abitte and masl and astii sathe contte relorie les viaio wire strean
woodiee sethi: ertientiaein feclure fagroso clove solees rovee
apoine
sut8 aprtoe and radi vaacca aotii sater Coutter fetorie wiei work tiyee
wwwouaal seth) antirntaation Facture, Lage Liee sotale
speine
sakee bouttee faborre wiki wish tayee
tobh) mntirnttat and facher ietiii 10 (yree
Chuisi ahitne (198-22041
atrtts oerection
cudaud dt glrtee csal (aay) and fie anc and
1u.0 100h snd
a) Original Brute Force
b) Adversarial Brute Force
Figure 12: Response Comparison of Original Brute Force and Adversarial Brute Force in ssh login
C
Ablution Study
This ablation study assesses the impact of substituting the pre-trained BERT model with a standard transformer architecture in the denoising process of DiffuPac. The focus is on assessing each model's ability to accurately reconstruct and generate adversarial packets that can effectively evade detection.
Non-BERT-based. This variant of DiffuPac employs a standard transformer architecture for the denoising process, outlined in the DiffuSeq methodology (Gong et al. [2023]). Notably, this model is trained from scratch without leveraging any pre-trained weights, thus relying solely on the information provided during the training phase.
BERT-based. The existing DiffuPac configuration, which incorporates a pre-trained BERT model for the denoising process, was utilized. This setup takes advantage of BERT's pre-trained weights, enabling enhanced semantic understanding and contextual relevance during packet reconstruction.
Both models were trained under identical conditions and evaluated based on their success in generating adversarial packets that evade detection. The results, depieted in Table 8, highlight the comparative effectiveness of each model.
As shown in Table 8, the BERT-based DiffuPac significantly outperforms the non-BERT variant, demonstrating superior ability to reconstruct packets that blend scamlessly into normal traffic. The BERT-based model excels in pairing malicious packets with normal packets that exhibit high contex- tual relevance. This strategic pairing allows the generated adversarial packets to blend seamlessly into normal traffic, thereby increasing their likelihood of evading detection. In contrast, the non- BERT-based model, which lacks this pre-trained knowledge, resorts to random matching of malicious and normal packets, leading to noticeable reduction in the effectiveness of the generated adversarial traffic.
An intriguing aspect of DiffuPac is its dual utilization of the pre-trained BERT model: first, as the denoising engine during the packet reconstruction process, and second, as critical tool for identifying contextually relevant packet pairs. To the best of our knowledge, this dual application of
25

---

Table 8: Comparative analysis of attack detection and evasion rates.
(a) Botnet
(b) MITM
Festure Extractor
Classifier
Detoction
Evasive (MER)
Festure Extractor
Classilier
Detection
Evasive (MER)
Noa-
Non-
BERT- Rased
BERT
Based
based
based
KinNET
0.84
0.94
0.92
38.93%
46.48 %
KHNET
0.92
0.94
0.91
45. 45.24%
55.87%
CIC
DT
0.79
0.91
0.82
50.24%
60.98 %
CIC
DT
0.74
0.79
0.76
47.83%
64.12%
0.99
0.90
0.95
39.18%
49.71 %
IF
0.99
0.92
0.94
36.91%
52.99%
54.98%
73.21%
SVM
0.92
52.31%
64.49%
SVM
0.74
56. 10%
60.46%
LR
0.84
0,91
0.89
35.02%
42.08 %
0.73
0.78
35.27%
47.48%
KRINET
0.96
0.90
0.94
69.15%
74.46 %
KRNET
0.94
0.96
0.93
52.55%
58.48%
Aftorlmage
DT
0.80
0,90
0.82
65.77%
72.13% %
Afterlmage
DT
0,75
0.89
0.84
61.47%
70.04%
IF
0.99
0.92
0.94
46.83%
$2.79 %
IF
0.81
0.83
0.86
31.33%
45.71% 7
MLP
0.96
0.96
0.95
52.04%
64.92%
MLP
0.92
0.90
0.93
64.17%
71.45%
SVM
0.94
%
0.99
56.44%
66.53%
LR
0.93
53.98%
58.98%
0.91
44.83%
52.38 %
tre-trained model within diffusion framework is unprecedented, further highlighting the innovation and effectivvness of our approach.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the key contributions of our paper, including the development of DiffuPac, the integration of a pre-trained BERT model with diffusion model, and the novel approach to generating adversarial packets. These claims are substantiated by both theoretical analysis and experimental results presented in the paper. The scope of our research, including the focus on real-world constraints and the effectiveness of the proposed model against advanced NIDS, is accurately reflected the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have included the limitations in the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The proofs for all theorems and formulas are clearly provided both in the main paper and supplemental material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides detailed explanation of the model architecture and the experimental setups. This includes thorough description of the pre- pre-trained BERT model integration, the diffusion model, and the specific configurations used in the experiments. Additionally, all hyperparameters, dataset details, and evaluation metrics are disclosed to enable replication of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
28

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Due to ethical concerns, we have decided not to publicly share the data and code. Additionally, much of the code is based on existing codebases from other sources. However, we have provided a full and detailed explanation of the model architecture and its implementation the paper and supplemental material. This includes comprehensive descriptions of the experimental setups, hyperparameters, and step-by-ste step instructions to facilitate reproducibility.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ public/guiden Sodeeoomisssonnn
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (httpa: / crrisscccppublll
geidee//eeeeeee Sate//uddeiiseeecceeee ssi omPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have provided the full detailed explanation regarding the training process and the experimental details both in the main paper and supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
29

---

Answer: [No]
Justification: The experiments conducted in this paper primarily focus evaluating the success of generating adversarial packets. Metrics such as evasion rate, packet field analysis, and probabilistic differences between original and adversarial packets are used to assess the effectiveness of the proposed method.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have detailed the types of computational resources used and the time of execution for BERT and diffusion model the supplemental materials.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://eeuuu.peeeeee cttpiippp.cppppeeee http:/uullc/t/ttt csGuidelulllliie ltw.a//eeeeectieieeeeeeeeeeeeeee ines?
Answer: [Yes]
Justification:
We have read the Code Of Ethics and strictly follow
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
30

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the positive societal impacts and negative societal impacts in the main paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
Due to ethical concerns of our paper, we have implemented stringent measures.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us, should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
31

---

Justification: All existing assets used in this paper. including code, data, and models, have been properly credited their original creators. We have cited the original papers that produced these assets and included URLs where applicable.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper thoode. www.cotmcaaa.c com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces a new model, DiffuPac, which integrates pre-trained BERT with diffusion model to generate adversarial packets. Detailed documentation for DiffuPac is provided, including descriptions of the model architecture, training process, implementation details, and usage guidelines. This documentation is included in the main paper and supplemental materials.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
32

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "HSRSRKcIdA", "reviewer": "Reviewer_CanA", "summary": "This work proposes DiffuPac, an adversarial packet generation model that aims to evade Network Intrusion Detection Systems (NIDS) without depending on detailed knowledge of NIDS components. DiffuPac combines a pre-trained BERT model with a diffusion model. Experimental results show that DiffuPac outperforms traditional methods by 8.83%.", "questions": "Please see the above-mentioned weaknesses. I will confirm the author's response during the rebuttal.", "limitations": "The authors discussed the limitations in this work.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This work studies an interesting and important issue about evading NIDS detection based on deep learning methods.\n- I appreciate that the author divides the traffic data into mutable fields and immutable fields for modification.\n- The experimental results support the effectiveness of the proposed method to evade detection.", "weaknesses": "- The paper achieves the following goals through expensive experiments (more than 30 hours of training on RTX 4090): select the normal packet sequence that is most similar to the malicious packet sequence, and modify the limited mutable fields (e.g. TCP flags, TTL, and window size) in the malicious packet sequence header according to the normal packet sequence. Given the very limited mutable fields modified within the packets and the lack of ablation studies in the experimental section, I doubt the necessity of using BERT and DiffuSeq. It seems that a simpler approach involving feature similarity matching, coupled with selective field selection and replacement, could potentially achieve the same goal without the complexity introduced by these advanced models.\n- In the diffusion model where all mutable fields of the entire malicious packet sequence are regenerated, there is a concern about whether this process affects its original functionality. Although the modification effects on specific fields of individual packets are shown in the appendix, whether the author replays these generated packets in a real network environment to verify its functionality, rather than simply viewing them through Wirshark.\n- Some minor issues: The paper needs improvement in writing quality. The connections between different components are not explicitly explained, and the specific format of the packet sequence input to the diffusion model is not detailed. In addition, the BERT-related content is very redundant in Sec. 3 and Figure 1. However, the related content is highly similar to ET-BERT and is only used to calculate the most relevant benign and malicious packet sequences in the \"fine-tuning\" stage (in fact, I think the operations in this stage do not belong to the category of \"fine-tuning\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewers’ constructive review and insightful suggestions regarding our paper.\nOur responses are given in a point-by-point manner for each comment.\n\n**W1.**\n\nSimpler methods (e.g., [1], [2], [3]) often require surrogate classifiers or access to NIDS components. \nThis is because the generated adversarial examples need to be evaluated and refined using these components to ensure they evade detection. \nThe dependency on internal NIDS mechanisms limits the practicality of these methods in real-world scenarios where attackers typically lack such insider access.\n\nIn contrast, DiffuPac operates independently of NIDS components by leveraging the capabilities of pre-trained BERT and diffusion models. \nThis approach allows DiffuPac to intelligently alter specific packet fields, seamlessly blending malicious packets into normal traffic without needing detailed knowledge of the NIDS. \nThe use of BERT is crucial in this process, as it provides the deep contextual understanding required to identify and modify fields in a way that maintains the packets' stealth and effectiveness.\n\nOur ablation study compared DiffuPac's performance with a standard transformer trained from scratch against the BERT-based model. \nThe results clearly demonstrate the BERT-based model's superiority in both evasion capabilities and accurate packet reconstruction. This highlights the essential role of BERT in DiffuPac, enabling the model to make sophisticated, contextually-aware modifications that simpler approaches, which lack such depth of understanding, cannot achieve.\nWe kindly ask the reviewer to refer to the \"Ablation Study\" section in the main author rebuttal for more detailed insights into the necessity of BERT in DiffuPac's architecture.\n\nWe agree with the reviewer that simpler approaches could be effective when contextual relevance with normal packets is identified. \nThis will be an important topic for future research, where we will explore the potential of such methods in conjunction with or as alternatives to more complex models like BERT and diffusion models.\n\n**W2.**\n\nWe really appreciate the reviewer's concern regarding the capability of DiffuPac in retaining the malicious functionality of generated adversarial packets. \nTo address this concern, we conducted a detailed malicious functionality evaluation within a controlled, isolated network environment. \nWe kindly ask the reviewer to refer to the \"Malicious Functionality Evaluation\" section in the main author rebuttal for a more detailed explanation.\nCurrently, we are also conducting malicious functionality evaluation on Brute Force Attacks, and other type attacks. These experimental results and their fully-detailed explanations will be added in the revised paper.\n\n**W3.**\n\nWe sincerely appreciate the reviewer’s insightful feedback and understand the importance of refining our explanations. \nOur methodology involves a structured process where each component plays a crucial role:\n\n- **Pre-Training Phase:**  \n  The BERT model is pre-trained using tasks like the Masked Unidirectional Flow Model and Same Sequence-origin Prediction. These tasks are essential for BERT to develop a deep understanding of network traffic patterns and dependencies, which are critical for the later stages.\n\n- **Fine-Tuning with Diffusion Models:**  \n  In this phase, the pre-trained BERT model enhances the diffusion process, allowing for the generation of adversarial packets that are contextually aligned with normal traffic. The packet sequences are formatted with token, positional, segment, and time step embeddings, providing a comprehensive input structure that the diffusion model uses to accurately reconstruct and generate packets.\n\nIn the revised paper, we will take the following steps to improve clarity and conciseness:\n\n- **Enhanced Component Integration:**  \n  We will provide a more explicit explanation of how each component of our methodology connects and contributes to the overall process. This will include clearer descriptions of the role of embeddings and how they prepare the data for effective processing by the diffusion model.\n\n- **Streamlining BERT-Related Content:**  \n  To address concerns about redundancy, we will condense the BERT-related sections, focusing on its unique application in DiffuPac. This will involve removing repetitive content and emphasizing only the most critical aspects that highlight DiffuPac’s innovations.\n\n- **Revisiting Terminology:**  \n  We will re-evaluate and revise terminology, particularly around the \"fine-tuning\" stage, to ensure it accurately reflects the specific operations and the unique way BERT is used in DiffuPac. This will help avoid any confusion and better align with the expectations of readers familiar with existing methodologies.\n\n- **Detailed Input Format Explanation:**  \n  We will include a more detailed explanation of the packet sequence input format to the diffusion model, ensuring readers understand the significance of each embedding type and how they contribute to the model’s performance.\n\n[1] Homoliak et al., Improving Network Intrusion Detection Classifiers by Non-Payload-Based Exploit-Independent Obfuscations\n\n[2] Hashemi et al., Towards Evaluation of NIDSs in Adversarial Setting \n\n[3] Kuppa et al., Black Box Attacks on Deep Anomaly Detectors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FmDRoqpOKx", "reviewer": "Reviewer_ogST", "summary": "This work presents DiffPac, a method that leverages BERT and a diffusion model to generate adversarial packets aimed at evading network \nintrusion detection systems. Compared to the approach by Han et al. (2021), DiffPac demonstrates superior effectiveness, with most machine learning-based classifiers trained on adversarial packets generated by DiffPac successfully evading detection across six different attack scenarios.", "questions": "* Data Pre-processing: Could you explain the rationale and advantage of transforming IP numerical values into hexadecimal representation and treating them as discrete values for workpiece segmentation? What benefits does this approach offer?\n\n* Formula (4): In Formula (4), does  S^{ben+mal}  denote z_t ? \n\n* Dataset: How many adversarial packets are included in the test set? Providing a statistical description of the datasets used would enhance clarity and help understand the dataset composition.", "limitations": "The limitation section highlights three main concerns:\n\n* The actual malicious functionality of the generated adversarial packets has not been thoroughly examined.\n\n* There is inconsistent performance across the six attack scenarios tested.\n\n* There is a potential risk associated with making the approach open-source, which could be exploited maliciously.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Approach: The paper presents an interesting intuition for generating adversarial packets using a diffusion model, which is a creative approach in the field.\n\n* Methodology: The methodology is clearly presented, making it easy to understand the steps and processes involved in the proposed DiffPac method.", "weaknesses": "* Motivation: The research motivation could be more clearly and concretely articulated. It would be beneficial to highlight the specific niche of this study in relation to existing work, such as Network Emulator (NetEM) and Metasploit (Homo-liak et al., 2018), Generative Adversarial Network (GAN) and Particle Swarm Optimization (PSO) (Han et al., 2021), and Reinforcement Learning (RL) (Hore et al., 2023). Rather than broadly stating that \"their efficacy in evading detection in real-world conditions remains suboptimal\" (line 64), the paper should detail the unique challenges and gaps that DiffPac addresses.\n\n* Method for Preserving Packets’ Integrity: The approach to preserving packets' integrity by only mutating TCP flags, TTL, and window size, rather than the payload, is unrealistic for real-world conditions where payloads often contain the malicious behaviors. This limitation should be addressed to enhance the practical applicability of the method.\n\n* Evaluation: The evaluation settings could be more closely aligned with those used by Han et al. (2021) to make it easier for the readers to understand the differences and contributions of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewers’ constructive review and insightful suggestions regarding our paper.\nOur responses are given in a point-by-point manner for each comment.\n\n**W1.**\n\nWe agree that a more detailed discussion on the limitations of existing works and how DiffuPac addresses specific gaps will strengthen the motivation of our study.\n\nTraditional approaches, such as those by Homoliak et al., Han et al., and Hore et al., often rely on access to NIDS components or surrogate classifiers. \nThese methods assume attackers have detailed knowledge of the NIDS, which is unrealistic in real-world scenarios. \nThis dependency limits their practical applicability, as attackers typically operate with minimal insider knowledge of the target's system.\nDiffuPac addresses these limitations by adopting a classifier-free approach, leveraging pre-trained BERT and diffusion models to generate adversarial packets without relying on NIDS components or surrogate classifiers. \nThis method allows DiffuPac to create adversarial traffic that mimics normal network behavior, making it more effective in evading detection in real-world conditions. \nAdditionally, DiffuPac’s adaptive field alteration ensures that the modifications are contextually appropriate, maintaining the malicious functionality of the packets while enhancing their stealthiness. \nIn the revised paper, we will expand on these points to highlight how DiffuPac innovates beyond existing methodologies, offering a practical solution for adversarial packet generation in real-world environments.\n\n**W2.**\n\nWe understand the concern that in real-world conditions, payloads often contain the core malicious behaviors, and thus, not modifying the payload may seem limiting. \nModifying the payload, which often contains core malicious behaviors, poses a significant risk of disrupting the packet's integrity, especially without detailed knowledge of its structure.\nWe acknowledge the potential benefits of payload modification and plan to explore methods like Deep Packet Inspection (DPI) and selective payload adjustments in future work to enhance the realism of adversarial packets while maintaining their functionality.\n\n**W3.**\n\nWe recognize the importance of such comparisons, particularly in demonstrating the unique contributions of DiffuPac in the context of existing methodologies.\nOur choice to diverge from the more complex evaluation settings used by Han et al. stems from the fundamentally different nature of our methodology. \nSince DiffuPac does not involve feature-level generation or direct interaction with NIDS components, the traditional metrics used in Traffic Manipulator's evaluation, which are deeply tied to feature extraction, are not directly applicable to our approach.\nInstead, by focusing on MER, we provide a clear and focused evaluation of DiffuPac's effectiveness in generating adversarial packets that evade detection in a more realistic context. \n\nWe recognize the value of aligning more closely with other methodologies.\nIn future work, we plan to expand our evaluation framework to include additional metrics and settings that facilitate direct comparisons with existing methods like Traffic Manipulator.\n\n**Q1.**\n\nWe really appreciate the reviewer's inquiry. \nWe convert IP numerical values to hexadecimal and treat them as discrete values for WordPiece tokenization to maintain the native structure and precision of network data. \nThis approach ensures uniform tokenization across diverse protocol fields, preserves semantic meaning crucial for understanding network traffic patterns, and allows for efficient vocabulary management, enhancing the model's ability to accurately process and learn from the data.\n\n**Q2.**\n\nWe sincerely apologize for any confusion caused by the insufficient explanation of Formula (4) in our original submission. We appreciate the reviewer's attention to detail and would like to clarify the notation and its context.\n\nTo clarify, $ \\mathbf{S}^\\mathrm{ben \\oplus mal} $ does not directly denote $ \\mathbf{z}_t $ at each time step $ t $, where the original concatenated sequence $ \\mathbf{S}^\\mathrm{ben \\oplus mal} $ is gradually perturbed.\n\nThe expression $ q_\\phi(\\mathbf{z}_0 \\mid \\mathbf{S}^{\\mathrm{ben \\oplus mal}}) = \\mathcal{N} (\\text{EMB}(\\mathbf{S}^{\\mathrm{ben \\oplus mal}}), \\beta_0 \\mathbf{I}) $ describes the initial state $ \\mathbf{z}_0 $, which is derived from the embedding of the concatenated packet sequences.\n\nSubsequent steps in the diffusion process perturb this state to produce the series of latent variables $ \\mathbf{z}_1, \\mathbf{z}_2, \\dots, \\mathbf{z}_t $.\n\nWe acknowledge that the initial explanation may not have fully conveyed these relationships, and we will ensure that the revision includes a more detailed and refined explanation.\n\n**Q3.**\n\n| Stage                             | Percentage of Total Data | Number of Normal Packets | Number of Malicious Packets |\n|-----------------------------------|--------------------------|--------------------------|-----------------------------|\n| Pre-training of the BERT model    | 60%                      | 5,079,021                | 2,734,858                    |\n| Fine-tuning phase                 | 20%                      | 1,693,007                | 911,619                      |\n| Training the classifier and NIDS  | 10%                      | 846,512                  | 455,819                      |\n| Testing phase                     | 10%                      | 846,504                  | 455,810                      |\n\n**Table 1**: Distribution of normal and malicious packets across different stages of data processing.\n\nIn the testing phase, we included 455,810 malicious packets. DiffuPac generated an equal number of adversarial packets, ensuring a direct comparison. A detailed statistical description of the dataset composition, including normal and malicious packets across all stages of the process, will be included in the revised paper for enhanced clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dgS1gOATiL", "reviewer": "Reviewer_vp4b", "summary": "The paper presents DiffuPac, a novel solution that integrates Bidirectional Encoder Representations from Transformers (BERT) and diffusion models to generate adversarial packets that can evade detection by sophisticated Network Intrusion Detection Systems (NIDS).\nDiffuPac leverages the extensive contextual understanding provided by BERT, which has been trained on diverse datasets representing a wide range of network behaviors, along with the generative capabilities of diffusion models. This fusion results in a sophisticated adversarial tactic where the elements of the attack are seamlessly integrated into the network traffic, making them indistinguishable from legitimate data.\nExtensive experimental evaluations on real-world datasets demonstrate that DiffuPac significantly outperforms existing methods in terms of evasion effectiveness, establishing new benchmarks for the generation of adversarial packets.", "questions": "1. For the high-level idea, I admit that there have been attempts to transform packet bytes as tokens and use language models for traffic analysis (e.g., [a,b,c]). However, most of them (also including DiffuPac) do not discuss if and how this type of method can handle encrypted traffic. As you may know, modern encryption algorithms like AES can significantly reduce entropy and there will be no one-to-one relations between plaintext and ciphertext; so from my perspective, learning from the encrypted payload is meaningless. Some methods may only use the header fields for tokenization. As there lack of such details, I'm not sure how DiffuPac deals with this concern.\n2. On page 5, line 199: \"... this task employs a dedicated binary classifier to determine the directional origin of network traffic, specifically whether packets in a sequence originate from src-to-dst or dst-to-src.\" Though the idea seems reasonable, I'm curious will there be any cases where packets from both sides are very similar so that the task can be too difficult to converge (e.g., in P2P apps where there are no explicit servers or clients)?\n3. On page 5, line 217: \"the initial packets in a flow contain the most significant information, we limit our analysis to the first three packets of each heavy flow\". Will the first three packets be too few? For TCP connections, the first three packets are typically just for handshakes so the model probably won't learn any useful information. Can the authors explain this, or give some experiments on this parameter?\n4. Could the authors list the mutable and immutable fields in the appendix?\n5. For experiment setup, one of my concerns is about the data preparation. Will the pre-trained model use malicious traffic for training? For one thing, the pattern of malicious traffic can be very abnormal so will that affect the BERT's understanding of normal traffic? For another, in practice, malicious traffic is much less than in the dataset so this setting can be somewhat impractical for a pre-trained model.\n6. Another major weakness of the experiment is that there is only one baseline. However, there are other works on generating (malicious or generic) traffic data, including those generating on feature spaces ([d,e]) or generating on raw bytes ([a,b]). The authors may compare them to improve the experiment, or discuss them in related work and present their fatal limitations that hinder the direct comparison.\n7. As there are two main components in DiffuPac (BERT and diffusion model), an ablation study on the pre-trained model is needed to show the necessity of using the pre-trained representations instead of some simpler representations.\n8. In the limitations, I suggest that the authors discuss the implications of your method facing signature-based NIDS (e.g., Snort). You may conclude with a consideration of the current situation that encrypted traffic is the majority, but that will go back to question 1 again.\n9. I like the authors providing the screenshots of Wireshark as parts of the results. One weakness is that the authors use their own domain knowledge to reveal the important packet fields that should be altered to avoid NIDS. However, the ML/DL-based NIDS may use some other features as the decision reasons which can be different from yours. I think the authors may use some global explanation models (e.g., [f,g]) to verify if the adversarial fields are really the features that NIDSes care.\n\n[a] NetGPT: Generative Pretrained Transformer for Network Traffic, arxiv 2023\n\n[b] TrafficGPT: Breaking the Token Barrier for Efficient Long Traffic Analysis and Generation, arxiv 2024\n\n[c] Yet Another Traffic Classifier: A Masked Autoencoder Based Traffc Transformer with Multi-Level Flow Representation, AAAI 2023\n\n[d] Flow-based network traffic generation using Generative Adversarial Networks, SIGCOMM 2022\n\n[e] Knowledge Enhanced GAN for IoT Traffic Generation, WWW 2023\n\n[f] AI/ML for Network Security: The Emperor has no Clothes, CCS 2022\n\n[g] Interpreting Unsupervised Anomaly Detection in Security via Rule Extraction, NeurIPS 2023", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Integrates BERT and diffusion models for adversarial packet generation\n\n* Addresses limitations of traditional methods that rely on unrealistic attacker knowledge\n\n* Introduces unique concatenation and targeted noising techniques for seamless blending of adversarial packets\n\n* I really appreciate that authors consider many realistic attacker constraints (e.g., packet integrity, replayability, only control on src-to-dst flows), showing that the authors have solid domain knowledge in network security", "weaknesses": "* There are many unclear points about the high-level idea, methodology, and experiments; see them below. If provided with compelling evidence, I may increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewers’ constructive review and insightful suggestions regarding our paper.\nOur responses are given in a point-by-point manner for each comment.\n\n**Q1.**\n\nWe appreciate the reviewer's insightful observation regarding the challenges of handling encrypted traffic.\nIn DiffuPac, we treat the payload as immutable, especially when encrypted. \nThis decision is based on the complexities of modern encryption algorithms like AES, which significantly reduce entropy and break the one-to-one relationship between plaintext and ciphertext. \nModifying encrypted payloads without understanding their structure risks rendering the packets non-functional or easily detectable.\nOur current focus is on modifying packet headers, which offer a safer and more navigable path for creating adversarial effects. \nHeader fields provide ample opportunity for evasion tactics while preserving the packet's integrity and functionality.\nFuture work may explore techniques for selective and safe payload modification, possibly leveraging advanced machine learning models to identify modifiable regions within encrypted payloads.\n\n**Q2.**\n\nWhile the task of distinguishing similar packets from both sides can indeed be challenging, our classifier has shown a strong ability to converge, even in these complex scenarios. \nThis is largely due to the deep learning capabilities of the transformer architecture, which uses self-attention mechanisms to capture subtle contextual cues and dependencies within packet sequences.\nWe trained the classifier using datasets like CICIDS2017 and Kitsune, which include scenarios with P2P traffic. \nThese datasets provide the necessary complexity to train the model to differentiate between src-to-dst and dst-to-src flows, even in environments where the distinction is not immediately apparent.\nWe plan to incorporate more diverse traffic scenarios in future work, including different types of encrypted and IoT traffic. \n\n**Q3.**\n\nWe apologize for any confusion caused by the initial explanation and appreciate the reviewer's insightful observation.\nWe will refine the explanation in the revised version to prevent any misunderstanding.\n\nTo clarify, in our preprocessing stage, each session is split into unidirectional flows—specifically src-to-dst and dst-to-src sequences. \nThis effectively means our analysis considers up to six packets per session: three from the src-to-dst flow and three from the dst-to-src flow.\n\nBy analyzing the first three packets from each unidirectional flow, we capture the initial exchanges in both directions. \nThis includes not just the TCP handshake but also the initial data packets, which often contain valuable information.\nConsidering packets from both directions ensures a more complete view of the communication, enhancing the model's ability to learn meaningful patterns even from early-stage data.\n\n**Q4.**\n\nWe cannot update the supplementary materials during the rebuttal period, in the revised paper we will provide fully-detailed explanation on mutable and immutable fields.\n\n**Q5.**\n\nThe pre-training phase utilizes an unlabeled dataset comprising a mix of normal and malicious traffic. \nThis allows BERT to learn general network traffic patterns and dependencies without bias toward specific traffic types.\nWhile malicious traffic can exhibit abnormal patterns, BERT's generalization capabilities ensure that these do not overshadow its understanding of normal traffic. \nThe focus during pre-training is on capturing the overall structure of network flows rather than distinguishing between normal and malicious traffic.\n\n**Q6.**\n\nWe acknowledge that including only one baseline may not provide a comprehensive comparison, especially in light of the existing body of work on generating packets. \nIn future work, we will conduct a thorough comparison with a broader range of methods within the Packet-Level Attacks category [1] to better position DiffuPac within the field of adversarial packets generation. \n\n**Q7.**\n\nWe kindly ask the reviewer to refer to the \"Ablation Study\" section in the main author rebuttal for a more detailed explanation.\n\n**Q8.**\n\nWe appreciate the reviewer's suggestion to discuss the implications of DiffuPac when facing signature-based NIDS.\nThe discussion regarding the relevance of signature-based NIDS on DiffuPac will be included in the revised paper.\nThis will involve evaluating how well DiffuPac's techniques, such as header modification and traffic pattern manipulation, can evade detection by these systems.\nFuture work will involve testing DiffuPac against a range of signature-based NIDS configurations and exploring enhanced header manipulation and potential payload modification techniques that respect encryption constraints.\n\n\n**Q9.**\n\nDiffuPac does not depend on predefined domain knowledge to reveal important packet fields. \nInstead, it intelligently alters specific fields within packets to seamlessly blend malicious packets into normal traffic, enhancing evasion capabilities. \nThis auto-altering capability is a key innovation of DiffuPac. In our revised paper, we will provide more refined explanations to prevent any misunderstandings regarding DiffuPac's intelligent field alteration capabilities.\n\nWe acknowledge that there may be differences between the packet fields modified by DiffuPac and the features ML/DL-based NIDS consider critical. \nThis potential disparity arises because DiffuPac independently identifies and modifies fields based on the inherent characteristics of network traffic and attack types.\n\nIn future work, we will use global explanation models, such as SHAP or LIME, in a controlled setting to gain insights into the influential features in NIDS detection processes. \nBased on these findings, we may enhance DiffuPac's targeting of significant features while maintaining its independent operational framework.\n\n[1] K. He et.al, Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WIM1cub1xs", "reviewer": "Reviewer_BP27", "summary": "This paper proposes a novel strategy named DiffuPac to generate adversarial packets to bypass NIDS detection. DiffuPac encompasses two critical components including the BERT, which captures the semantic meaning of packets and facilitates the embedding and contextual paring. The other one is DiffSeq which makes the malicious part seamlessly integrated into the normal packet pattern thereby improving the evasion against detection. Experiments showcase that DiffuPac outperforms the baseline and is capable of breaking practical detection through simulation.", "questions": "Please refer to weakness.", "limitations": "The authors have listed their limitations and have addressed most of them. One concern about functionality preservation is mentioned in the weakness part.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written, with a clear method description.\n\n2. The proposed DiffPac is effective against bypassing detection, the proposed strategy seems reasonable with good explanation.", "weaknesses": "1. As stated in the limitation, whether the functionality of these adversarial packets is preserved is not verified. From my understanding, this point is quite important given the practical aspect of DiffPac. If the adversarial package can only bypass detection without functioning as intended, it would undermine the practical utility of DiffPac.\n\n2. It would be beneficial to include additional analysis, such as evaluating the performance without applying contextual pairing and assessing the robustness against potential packet-level defenses. These analyses would provide further insights into the effectiveness and resilience of DiffPac.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewers’ constructive review and insightful suggestions regarding our paper.\nOur responses are given in a point-by-point manner for each comment.\n\n**W1.**\n\nWe conducted a detailed malicious functionality evaluation within a controlled, isolated network environment. \nWe kindly ask the reviewer to refer to the \"Malicious Functionality Evaluation\" section in the main author rebuttal for a more detailed explanation.\n\n**W2.**\n\nWe have conducted an ablation study that directly addresses the impact of contextual pairing, comparing DiffuPac’s performance with and without the use of the pre-trained BERT model for contextual pairing. \nThis study has been included in our main author rebuttal, and we encourage the reviewer to refer to \"Ablation Study\" section in the main author rebuttal for detailed insights into how contextual pairing enhances the model’s effectiveness.\n\nWe acknowledge that while Kitsune NIDS provides a solid foundation for evaluating packet-level defenses, it represents only one approach among many. \nTo further strengthen DiffuPac’s robustness and address the reviewer's concerns, our future work will explore additional NIDS that utilize advanced packet-based feature extractors. \nBy broadening our evaluation framework to include a variety of packet-level defenses, we aim to gain a deeper understanding of DiffuPac’s effectiveness across different NIDS architectures and feature extraction methodologies.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "epVf9NjfDY", "reviewer": "Reviewer_rUa8", "summary": "This paper proposes an adversarial packet generation method, named DiffuPac, to evade detection. DiffuPac integrates BERT and a diffusion model to make the generated packets indistinguishable. To better fit the task, a concatenation strategy and a classifier-free approach are proposed. The experimental results are promising.", "questions": "1.\tI understand the effectiveness of DiffuPac in an experimental environment but in the real world, usually, an ML model is adopted to detect evasion for fast response requirements but not a DNN. As DNN is less robust to adversarial examples, will this affect the practicability of DiffuPac?", "limitations": "Mentioned in Sec.5.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper proposes a novel adversarial packet generation scheme to test the evasion detection ability of NIDS. \n2.\tThe paper is well-written and easy to follow.\n3.\tThe performance is good.", "weaknesses": "1.\tThe primary concern is the efficiency of DiffuPac. As a diffusion model is adopted to generate adversarial packet, it may be time-consuming. Can you conduct some experiments to validate the efficiency of your method?\n2.\tCan you also consider some defenses against such adversarial packets? For example, adversarial training (AT), feature selection (FS), and adversarial feature reduction (AFR) as shown in work [1]\n\n\n[1] Han, Dongqi, et al. \"Evaluating and improving adversarial robustness of machine learning-based network intrusion detectors.\" IEEE Journal on Selected Areas in Communications 39.8 (2021): 2632-2647.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewers’ constructive review and insightful suggestions regarding our paper.\nOur responses are given in a point-by-point manner for each comment.\n\n**W1.**\n\nWe conducted a comprehensive evaluation of different BERT model configurations to identify the most efficient option that still maintains high performance.\nAs for the diffusion model,  we have implemented several optimizations to improve the training and inference efficiency of the model.\nWe tested four BERT model variants: Tiny, Small, Medium, and Large, comparing their specifications, training time, loss, and accuracy. \nThe results are summarized in the Table 1 below:\n\n| **Parameters**          | **Metrics**            | **Tiny BERT** | **Small BERT** | **Medium BERT** | **Large BERT** |\n|-------------------------|------------------------|---------------|----------------|-----------------|----------------|\n| **Model Specifications**| **Embedding Size**     | 128           | 512            | 768             | 1024           |\n|                         | **Feedforward Size**   | 512           | 2048           | 3072            | 4096           |\n|                         | **Hidden Size**        | 128           | 512            | 768             | 1024           |\n|                         | **Activation Function**| GELU          | GELU           | GELU            | GELU           |\n|                         | **Number of Heads**    | 2             | 8              | 12              | 16             |\n|                         | **Number of Layers**   | 2             | 4              | 12              | 24             |\n| **Training**            | **Training Time (hours)**| 13.5         | 18             | 23              | 30.5           |\n| **Performance**         | **MUM Loss**           | 1.028         | 0.822          | 0.667           | 0.610          |\n|                         | **SSP Loss**           | 0.295         | 0.204          | 0.099           | 0.061          |\n|                         | **MUM Accuracy**       | 0.844         | 0.887          | 0.905           | 0.914          |\n|                         | **SSP Accuracy**       | 0.846         | 0.902          | 0.972           | 0.982          |\n\n**Table 1**: Comparison of Tiny, Small, Medium, and Large BERT models in terms of their specifications, training time, loss, and accuracy.\n\nThe Medium BERT model was chosen as the optimal configuration for DiffuPac because it offers a balanced trade-off between accuracy and loss metrics.\n\nAs for the diffusion model, we utilized FP16 (half-precision floating-point) for GPU computations. This approach significantly reduced the training time from 10 hours to 4 hours, nearly doubling the speed without sacrificing model performance. FP16 allows us to leverage the GPU's computational power more effectively, enabling faster iterations and reduced memory usage.\nDuring the sampling phase, we initially encountered issues with memory over-utilization, which was a significant bottleneck given our hardware limitations. \nTo resolve this, we integrated the FAISS library, which is optimized for efficient similarity search and clustering. FAISS allowed us to handle the nearest neighbor search more efficiently, preventing memory overflow and ensuring smoother sampling operations. This optimization not only improved the stability of our sampling phase but also reduced the overall memory footprint of the model.\n\nAll of the results including the fully-detailed explanation of them will be added in the revised paper.\n\n**W2.**\n\nAdversarial Defences is a critical area of research that we are actively exploring, particularly in the context of DiffuPac's unique capabilities.\n\nIn addition to the more established defense strategies mentioned, we are currently developing a novel approach to defending against adversarial packets: adversarial purification using diffusion models. \nThis technique has shown great promise in the field of image processing, and we believe it holds significant potential for network security as well.\n\nOur approach leverages the inherent strengths of diffusion models, which are ideally suited for the task of adversarial purification. \nBy treating adversarial perturbations as noise, we utilize the diffusion model's reverse process to \"purify\" incoming network packets, effectively stripping away malicious modifications and restoring the packet to its original, clean state before it is analyzed by a NIDS.\n\nThis research is still in progress, and we are excited about its potential. We aim to fully develop and test this method, positioning it as a powerful alternative to traditional adversarial defenses. \nAs far as we are aware, this research work will be the first to apply adversarial purification to network packets, which we believe could set a new standard in the field.\n\n**Q1.**\n\nIn future work, we will propose a hybrid approach that combines the strengths of DiffuPac with more robust and faster-responding machine learning (ML) models. \nIn practice, DiffuPac could be used to generate adversarial examples during the training phase, which can then be used to harden simpler, more interpretable models like decision trees, random forests, or support vector machines through adversarial training. \nThis approach allows for the creation of a robust detection system that can quickly respond to threats while benefiting from the advanced adversarial generation capabilities of DiffuPac.\n\nAdditionally, while DiffuPac focuses on generating adversarial examples, the integration of adversarial purification, as discussed in our ongoing research, offers another layer of defense. \nThis process can help mitigate the effects of adversarial examples on DNNs, thereby enhancing their robustness in real-world applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewer’s thoughtful feedback and constructive insights. 
Here, we summarized the experiments that we conducted for the ablation study and maliciousness functionality evaluation.
These experiments will be included in the revised version of the paper.

### Ablation Study

**Objective:** This ablation study assesses the impact of substituting the pre-trained BERT model with a standard transformer architecture in the denoising process of DiffuPac. 
The focus is on assessing each model's ability to accurately reconstruct and generate adversarial packets that can effectively evade detection.

**Experimental Setup:**

- **Non-BERT-based:** This variant of DiffuPac employs a standard transformer architecture for the denoising process, as outlined in the DiffuSeq methodology [1]. 
    Notably, this model is trained from scratch without leveraging any pre-trained weights, thus relying solely on the information provided during the training phase.
- **BERT-based:** The existing DiffuPac configuration, which incorporates a pre-trained BERT model for the denoising process, was utilized. 
    This setup takes advantage of BERT's pre-trained weights, enabling enhanced semantic understanding and contextual relevance during packet reconstruction.
   
**Training and Evaluation:** Both models were trained under identical conditions and evaluated based on their success in generating adversarial packets that evade detection.
The results are shown in the accompanying PDF.

**Result And Analysis:** As shown in Table 1, the BERT-based DiffuPac significantly outperforms the non-BERT variant, demonstrating superior ability to reconstruct packets that blend seamlessly into normal traffic. 

The BERT-based model excels in pairing malicious packets with normal packets that exhibit high contextual relevance. This strategic pairing allows the generated adversarial packets to blend seamlessly into normal traffic, thereby increasing their likelihood of evading detection. 
In contrast, the non-BERT-based model, which lacks this pre-trained knowledge, resorts to random matching of malicious and normal packets, leading to a noticeable reduction in the effectiveness of the generated adversarial traffic.

An intriguing aspect of DiffuPac is its dual utilization of the pre-trained BERT model: first, as the denoising engine during the packet reconstruction process, and second, as a critical tool for identifying contextually relevant packet pairs. 
To the best of our knowledge, this dual application of a pre-trained model within a diffusion framework is unprecedented, further highlighting the innovation and effectiveness of our approach.


### Malicious Functionality Evaluation

**Objective:** To validate that DiffuPac-generated adversarial packets retain their original malicious functionality.

**Experimental Setup:** We evaluated DiffuPac’s efficacy in generating adversarial packets that maintain their malicious functionality in a controlled test environment using UTM hypervisor technology. 
This setup included two virtual machines: one running Kali Linux (attacker) equipped with Nmap, and another running Ubuntu 24.04 (victim). The isolation ensured a realistic yet controlled network environment. The results are shown in the accompanying PDF.


**Initial Attack Demonstration:**
Before proceeding to evaluate the generated adversarial packets of DiffuPac, we first demonstrated a successful Port Scan attack within our controlled environment. 
This initial step is a fundamental step to ensure the validity and reliability of subsequent tests with both original and adversarial packets.
The network configurations depicted in Figures 1(a) and 1(b) confirm that both virtual machines were correctly placed within the same subnet. 
As demonstrated in Figure 1(c), the Port Scan successfully identified open ports on the Ubuntu, highlighting port 22 as vulnerable. 

**Detailed Methodology:**

- **Initial Packet Capture:** We initiated a Port Scan attack from the Kali Linux machine targeting the Ubuntu server. 
    During the execution of this attack, we captured all packets sent from the attacker to the victim, saving them as a pcap file. 
    This capture was timed precisely to begin as the Nmap tool launched the Port Scan, ensuring that only the relevant src-to-dst packets were recorded. 
    This focus aligns with the operational framework of DiffuPac, which concentrates on modifying packets sent from the source.
- **Adversarial Packet Generation and Replay:** Using the captured src-to-dst packets, we employed DiffuPac to generate adversarial Port Scan packets, which were then saved in a separate pcap file. 
    These adversarial packets were subsequently replayed using the Tcpreplay tool, directing them towards the Ubuntu server. 
    Concurrently, we used Wireshark on the Ubuntu machine to capture the incoming responses, allowing us to analyze the effectiveness of the adversarial packets in real-time.
- **Fair Comparative Analysis:** To ensure a balanced evaluation, we also replayed the original Port Scan pcap file using Tcpreplay, capturing the responses in Wireshark on the Ubuntu side. This dual replay enabled a direct comparison between the packets' responses generated by the original and adversarial Port Scan packets.

**Results And Analysis:** 
As depicted in Figures 2(a) and 2(b), the response to the adversarial Port Scan packets closely mirrors that of the original Port Scan.
A detailed inspection of the TCP flags reveals that packets with SYN and ACK flags targeted port 22, corroborating the initial vulnerability identified in Figure 1(c). 
This consistent response across both original and adversarial packets provides compelling evidence that DiffuPac successfully generates adversarial packets that retain their intended malicious functionality.

Currently, we are also conducting evaluation on other type of attacks, that will be included in the revised paper.

[1] Gong S et. al., Diffuseq

Meta-review: The paper provides a novel method to evade Machine Learning-based Network Intrusion Detection Systems (ML-NIDS). From a security viewpoint, the major contribution of the paper is that the proposed method can work even in a very constrained setting wherein the attacker has no knowledge/access to the targeted ML-NIDS. Such a property is provided by leveraging models "pre-trained" on (well-known) datasets. The experiments validate the effectiveness of the method.

No reviewer pointed out critical flaws in the evaluation, and all reviewers praised the quality of writing of the paper. The major concerns on this paper were related to the guarantee that the method achieves evasion while preserving the original malicious functionality of the generated data. The authors have responded to this concern by carrying out some validatory experiments during the rebuttal. Nonetheless, such a verification is challenging to carry out, and few works have addressed this aspect (which is crucial from a security viewpoint).

Given the above, the paper does provide a meaningful contribution in the domain of "applied" ML for cybersecurity, and while the results are not groudbreaking and may present some uncertainties, there is reason to believe that this paper will inspire future work in this domain.

I also favor making the discussion between authors and reviewer publicly visible. There is much to learn in the comments/remarks generated during the discussion phase.
Decision: Accept (poster)