DiffuPac: Contextual Mimicry in Adversarial Packets Generation via Diffusion Model

Authors: abdullah bin jasni
akiko  manada
kohei  watabe
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:49:15

Abstract: In domains of cybersecurity, recent advancements in Machine Learning (ML) and Deep Learning (DL) have significantly enhanced Network Intrusion Detection Systems (NIDS), improving the effectiveness of cybersecurity operations. However, attackers have also leveraged ML/DL to develop sophisticated models that generate adversarial packets capable of evading NIDS detection. Consequently, defenders must study and analyze these models to prepare for the evasion attacks that exploit NIDS detection mechanisms. Unfortunately, conventional generation models often rely on unrealistic assumptions about attackers' knowledge of NIDS components, making them impractical for real-world scenarios. To address this issue, we present DiffuPac, a first-of-its-kind generation model designed to generate adversarial packets that evade detection without relying on specific NIDS components. DiffuPac integrates a pre-trained Bidirectional Encoder Representations from Transformers (BERT) with diffusion model, which, through its capability for conditional denoising and classifier-free guidance, effectively addresses the real-world constraint of limited attacker knowledge. By concatenating malicious packets with contextually relevant normal packets and applying targeted noising only to the malicious packets, DiffuPac seamlessly blends adversarial packets into genuine network traffic. Through evaluations on real-world datasets, we demonstrate that DiffuPac achieves strong evasion capabilities against sophisticated NIDS, outperforming conventional methods by an average of 6.69 percentage points, while preserving the functionality and practicality of the generated adversarial packets.

Main paper: 
 introduction: 
 Network Intrusion Detection Systems (NIDS) play a pivotal role in safeguarding the vast array of digital devices and infrastructures that permeate our lives. As of 2023, the global count of active IoT devices is expected to reach approximately 15.14 billion, with projections suggesting a rise to 30 billion by 2030 (Statista [2023] ). This explosive growth, fueled by applications spanning from consumer electronics to industrial automation and healthcare, presents formidable security challenges. To meet these challenges, advancements in Machine Learning (ML) and Deep Learning (DL) have significantly bolstered the efficacy of NIDS in monitoring IoT traffic and detecting malicious activities (Talaei Khoei and Kaabouch [2023] ; Talaei Khoei et al. [2023] ).However, the rapid evolution of generative AI technologies has ushered in a new era of cybersecurity threats, notably through the creation of adversarial packets designed to evade detection by even the most sophisticated NIDS. These AI-driven attacks can emulate and synthesize legitimate network behaviors, presenting an unprecedented challenge to existing security paradigms. Generative models, particularly those trained on extensive datasets of genuine network traffic, can generate adversarial packets that blend malicious functionalities within seemingly normal packet sequences, thus effectively camouflaging their malicious intents.In response to these evolving threats, it is critical for cybersecurity defenders to deepen their understanding of these generative AI models. By scrutinizing the mechanisms through which these models generate adversarial packets, defenders can better anticipate and counteract adversarial tactics that compromise the detection capabilities of NIDS (Ibitoye et al. [2019] ; Khazane et al. [2024] ). The urgency to develop innovative solutions that can adapt to and preempt these adversarial tactics is paramount, ensuring the reliability and robustness of NIDS in an increasingly complex threat landscape.Table 1 : Summary of recent literature on adversarial packets generation.Traditional methods for generating adversarial packets have primarily relied on direct engagement with NIDS or the use of surrogate classifiers, often assuming unrealistic levels of attacker access to NIDS configurations. Table 1 summarizes recent literature on adversarial packet generation, highlighting the limitations of these approaches. Studies using techniques such as Network Emulator (NetEM) and Metasploit (Homoliak et al. [2018] ), Generative Adversarial Network (GAN) and Particle Swarm Optimization (PSO) (Han et al. [2021] ), and Reinforcement Learning (RL) (Hore et al. [2023] ) demonstrate that while these methods successfully modified the packets behavior, their efficacy in evading detection in real-world conditions remains suboptimal. The reliance on detailed knowledge of NIDS models is flawed, as attackers typically operate with limited information about the underlying security infrastructure. This gap underscores the necessity for more practical and effective adversarial generation methods that align with the realistic constraints faced by attackers.To address these challenges, we introduce DiffuPac: Contextual Mimicry in Adversarial Packet Generation via Diffusion Model, a novel solution that leverages the combined strengths of Bidirectional Encoder Representations from Transformers (BERT) and diffusion model. This innovative fusion not only promises high accuracy in generating adversarial packets but also operates under the realistic assumption that attackers lack direct access to NIDS models. DiffuPac leverages the extensive contextual understanding provided by BERT, which has been trained on diverse datasets representing a wide range of network behaviors, along with the generative capabilities of diffusion models. This fusion results in a sophisticated adversarial tactics where the elements of the attack are seamlessly integrated into the network traffic, making them indistinguishable from legitimate data. This capability represents a significant leap forward, offering a stealthy approach that outmaneuvers current NIDS through advanced mimicry rather than direct confrontation.In summary, the principal contributions of this paper are as follows: (a) we have pioneered the integration of BERT and diffusion models to create DiffuPac, marking a first in the cybersecurity domain. This novel methodology sets a precedent in the field by blending the advanced contextual comprehension of network traffic with sophisticated generative capabilities to produce adversarial packets that are both stealthy and indistinguishable from genuine traffic; (b) we introduce a unique concatenation strategy coupled with targeted noising techniques. These innovations ensure that the adversarial packets not only blend seamlessly into the network environment but also dynamically adapt to evade modern detection systems; (c) DiffuPac advances a classifier-free approach to adversarial packet generation. This approach challenges traditional dependency on surrogate classifiers, offering a new paradigm that more accurately reflects the constraints and capabilities of real-world attackers; (d) Lastly, our extensive experimental evaluations, conducted on real-world datasets, demonstrate that DiffuPac significantly outperforms existing methods in terms of evasion effectiveness, establishing new benchmarks for the generation of adversarial packets. These contributions collectively push the boundaries of what is possible in the realm of network security, paving the way for more resilient cybersecurity defenses and a deeper understanding of adversarial tactics in network environments. 
related works: 
 Adversarial Attacks on ML/DL-based NIDS. NIDS are crucial for protecting digital infrastructures by monitoring network traffic and identifying potential threats through signature-based and anomalybased detection paradigms. Signature-based NIDS use pattern matching against predefined threat databases, while anomaly-based NIDS employ machine learning models to detect deviations from benign traffic patterns, providing an advantage in detecting sophisticated threats. The transition to DL in anomaly-based NIDS, as highlighted by Ahmad et al. [2021] , has enhanced threat detection due to the capability to learn abstract patterns, though this shift has introduced a vulnerability to adversarial attacks. These attacks modify network data subtly to evade NIDS, a phenomenon first noted in computer vision (Szegedy et al. [2014] ) and now challenging DL-based NIDS. Adversarial attack tactics vary based on attacker knowledge, ranging from white-box attacks with full system knowledge to black-box attacks with no classifier knowledge, as outlined by McCarthy et al. [2022] . Black-box scenarios are common in real-world threats since attackers generally lack direct NIDS access, necessitating sophisticated modifications of data to exploit DL model vulnerabilities and blur the distinction between normal and malicious traffic.Adversarial Attacks Evading NIDS. In the study of NIDS, researchers explore feature-level and packet-level attacks to enhance robustness. Feature-level attacks modify input network features using methods like GANs to mislead classifiers without direct knowledge of their mechanisms. For instance, Yang et al. [2018] used transfer-based and score-based attacks to deceive a Deep Neural Network (DNN) model on the NSL-KDD dataset, while Sheatsley et al. [2022] developed an Augmented JSMA (AJSMA) to ensure realistic feature modifications. However, these still fail to generate executable malicious packets since they do not provide a method for converting modified features into packet sequences. Conversely, packet-level attacks, which modify network packets directly to maintain their malicious intent while evading detection, are particularly effective. As detailed in Table 1 , studies such as Hashemi et al. [2019] show successful modifications using non-payload based and mimicking operations. In the survey He et al. [2023] , authors emphasized that the practicality and replayability of packet-level attacks make them more dominant, ensuring adversarial packets evade detection while remaining executable. Given this efficacy, our research model, DiffuPac, is designed to excel in this domain, outperforming previous models in robustness and detection evasion, and setting a new standard for NIDS efficacy in adversarial cybersecurity.Diffusion Models. Represent a breakthrough in generative modeling, offering a novel framework for understanding data as a dynamic and stochastic process. Based on the works of Sohl-Dickstein et al. [2015] and Ho et al. [2020] , these models conceptualize data points z 0 ∈ R d (where d is a positive integer) as the end result of a reverse Markov chain. In the forward diffusion process, data points z 0 are gradually transformed into a noise distribution . This transformation is modeled by a Markov chain starting from the data distribution q(z 0 ) and ending in a Gaussian distribution z T ∼ N (0, I), where T represents the total number of timesteps in the forward diffusion process. The transitions are defined q(zt |z t-1 ) = N (z t ; √ 1 -β t z t-1 , β t I) for 1 ≤ t ≤ T, where β t as the variance scale. In the reverse process, the model f θ (usually a U-Net or a Transformer) learns to reconstruct the original data z 0 from the noised data z T . It does so by iteratively estimating the parameters p θ (z t-1 |z t ) = N (z t-1 ; µ θ (z t , t), σ 2 t I), where µ θ and σ 2 t predict the mean and variance of the distribution.BERT and Diffusion Models Approaches in Cybersecurity and Other Fields. The application of pre-trained models, such as ET-BERT (Lin et al. [2022] ) and NetGPT (Meng et al. [2023] ), has revolutionized understanding of network traffic by capturing the intricacies of language and interpreting complex network patterns. Meanwhile, diffusion models have emerged as powerful generative tools capable of producing high-fidelity data for applications like image generation and notably, synthetic network traffic generation. These models leverage a unique denoising training objective to closely mimic the original data distribution, making them promising for generating realistic network IP 192.168.1.1.52255 > 40.103.10.38.443: Flags [.] 5056 C05A 7508 …. A814 06C0 A900 ….Position Embedding traffic for testing and analysis. Despite these advancements, combining the contextual understanding of pre-trained models with the generative prowess of diffusion models remains underexplored in adversarial packet generation. This integration offers a synergistic approach, merging the nuanced comprehension of network behaviors by pre-trained models with the high-quality data generation of diffusion models. By leveraging these technologies together, cybersecurity researchers can enhance the realism of simulated network environments and develop more advanced evasion tactics.Src-to-dst … … … … … … 𝐸 ! 𝐸 ! 𝐸 " 𝐸 " 𝐸 " Segment Embedding Dst-to-src 𝐸 # 𝐸 $ 𝐸 %&' 𝐸 % 𝐸 %(' 𝐸 ) 𝐸 *+*, 𝐸 -+*. 𝐸 [!01] 𝐸 [%.!3] 𝐸 *+*, 𝐸 1." ℎ["#$] 𝐸 [-4!] 𝐸 ' 𝐸 ! ℎ["#$] ℎ&'&( ℎ&'&( ℎ"'&) ℎ "'&) ℎ[$*+] ℎ[$*+] ℎ[,)$-] ℎ &'&( ℎ&'&( ℎ [,)$-] ℎ[+).] ℎ[+).] … … … … ℎ [-4!] ℎ *+*, ℎ -+*. ℎ [!01] ℎ *+*, ℎ [%.!3] ℎ [1."] … … BERT ℎ [-4!] ℎ *+*, ℎ -+*. ℎ [!01] ℎ *+*, ℎ [%.!3] ℎ [1."] … …𝑧 ! 𝑧 " 𝑧 #$% 𝑧 # … … 𝑝 & (𝑧 #$% |𝑧 # ) 𝑞(𝑧 # |𝑧 #$% ) Forward Process Reverse Process 𝑆 '() 
diffupac: 
 In this section, we introduce DiffuPac, a first-of-its kind adversarial packet generation model. Figure 1 illustrates the overall framework of the proposed model, consisting of three main phases: pcap pre-processing, pre-training and fine-tuning with diffusion model.In real network environments, traffic contains diverse flows from various applications, protocols, and services, complicating the learning of stable representations. Therefore, we first split pcap (packet capture) files into sessions (bidirectional flows) based on IP addresses, port addresses, and protocols. To refine the training of the BERT model, we further split sessions into unidirectional flows, categorizing them as either source-to-destination (src-to-dst) packet sequence or destination-tosource (dst-to-src) packet sequence. This categorization is crucial for the BERT model's pre-training. Network traffic varies due to diverse protocols and network services, resulting in different formats and patterns. To handle these variations and encoding requirements, we convert each byte to its corresponding hex number and tokenize using WordPiece (Wu et al. [2016] ). Each token ranges from 0 to 65535, with a dictionary size of 65536. We also incorporate special tokens [CLS], [SEP],[PAD], and [MASK] for training tasks.[CLS] is used at the beginning of each sequence and helps in classification tasks.[SEP] separates different sequences or segments within the same input. [PAD] ensures sequences are of uniform length and satisfy the minimum length requirement.[MASK] is used in pre-training task, where it temporarily substitutes tokens to be predicted.As shown in Figure 1 , the input tokens are processed using an embedding strategy that involves the sum of three types of embeddings: token embeddings, positional embeddings, and segment embeddings. Each embedding has a dimension of 768. Token Embeddings are high-dimensional vectors that uniquely represent each token, acting as their exclusive identifiers. Positional Embeddings are used to capture the temporal relationships of tokens, ensuring that the model learns to focus on the order of data transmission. Segment Embeddings differentiate packets within a single flow, as packets may not inherently share semantic associations, and preserving the order of packets to maintain the temporal sequence of events in a session flow.We design our pre-training tasks based on the approach described in Devlin et al. [2019] . Our two proposed pre-training tasks aim to capture the contextual relationships between traffic bytes. The first task involves predicting masked tokens to learn the underlying patterns and dependencies within the traffic data. The second task predicts the transmission order by determining whether packets belong to src-to-dst or dst-to-src sequences, thereby capturing the directional flow of the network traffic.Masked Unidirectional Flow Model. Inspired by BERT's Masked Language Model (MLM) in natural language processing, we adapt this approach for network traffic analysis through our Masked Unidirectional Flow Model. This model is designed to understand and predict the semantic patterns within bidirectional network flows; that are src-to-dst sequence and dst-to-src sequence. During pre-training, each token in the input sequence is masked with a probability of 15%. Amongst these masked tokens 80% are replaced with a [MASK] token, 10% are replaced with a random token from the vocabulary, and 10% are left unchanged. This introduces variability that mimics real-world data inconsistencies. The training objective is to minimize the negative log-likelihood of correctly predicting the original tokens at the masked positions. Formally, the loss function L MUM for this task is defined as:EQUATIONwhere n is the total number of tokens in the sequence, 1(m i ) is an indicator function that is 1 if the token v i was masked (0 otherwise). m i indicates the masking status of the i-th token and V masked is the masked sequence. v i is the actual token at position i and P (v i | V masked ; θ) is the probability of predicting the original token v i for given masked sequence and model parameters θ. The transformer encoder, characteristic of BERT, processes V masked to predict each masked token. This architecture leverages self-attention mechanisms to capture dependencies and context effectively, which is crucial for understanding complex patterns in network traffic.Same Sequence-origin Prediction. Inspired by Lin et al. [2022] , this task employs a dedicated binary classifier to determine the directional origin of network traffic, specifically whether packets in a sequence originate from src-to-dst or dst-to-src. This classification enhances the model's understanding of network flows and the contextual relationships between packets, which is crucial for recognizing communication patterns and dependencies in network traffic. For this task, each packet a r is paired with another packet a s . 50% of the time, a s is the next logical packet in the flow (src-to-dst or dst-to-src); the other 50% of the time, it is a randomly chosen packet from the opposite flow. The classifier then predicts whether a s follows a r in the correct directional sequence. This pairing strategy improves the model's capability to discern patterns in packet flows. Let A = {(a r , a s )} be the set of packet pairs, where each pair is labeled with b w ∈ {0, 1} (1 if a s follows a r in the correct flow, 0 otherwise). The loss function L SSP for this task can be formulated as:EQUATION)where K is the total number of packet pairs and θ represents the trainable parameters of the classifier. P (b w = 1 | a r , a s ; θ) is the probability predicted by the classifier that a s correctly follows a r in the given network flow direction.In summary, the final pre-training objective is the sum of the above two losses, which can defined as:L = L SSP + L MUM .(3) Fu et al. [2021] mentioned that the number of packets within each flow can vary significantly. Given the constraints imposed by packet size and the potential volume of network traffic, computational efficiency is a paramount concern. In addressing this problems, as demonstrated in Dai et al. [2023] that the initial packets in a flow contain the most significant information, we limit our analysis to the first three packets of each heavy flow. This means that for each session, we analyze a total of six packets-three from the src-to-dst flow and three from the dst-to-src flow. This strategy ensures a comprehensive view of the session while maintaining efficiency.Forward Process with Packet Sequences Concatenation Strategy. In the fine-tuning phase, the goal is to train the model so that malicious packets can mimic normal packets to bypass NIDS. This involves using packet sequences that are only from the src-to-dst, reflecting the adversary's control over the packets being sent (Hore et al. [2023] ). The forward process of our diffusion model begins by embedding both normal (benign) packet sequences S ben and malicious packet sequences S mal . This transformation of discrete packet data into a continuous feature spaces uses an embedding function adapted from Li et al. [2022] . Building upon the groundwork of DiffuSeq (Gong et al. [2023] ) approach, which typically involves random merging, our model innovates by leveraging the deep contextual insights provided by the pre-trained BERT model (Details in A.2). This allows us to strategically pair normal and malicious sequences with strong contextual alignments that show similarity in network behavior patterns. These contextually aligned pairs are crucial for mimicking normal traffic and increasing the chances of bypassing NIDS. By integrating these contextually aligned pairs into our diffusion process, the model extends the original forward chain to a new Markov transition q ϕ (z 0 | S ben⊕mal ) = N (EMB(S ben⊕mal ), β 0 I), where EMB(S ben⊕mal ) symbolizes the embedding transformation and concatenation of normal and malicious packet sequences. While S ben⊕mal denotes the initial concatenated sequence, the forward process gradually perturbs this initial state z 0 through a series of transitions, producing latent variables z 1 , z 2 , . . . , z t .Targeted Noising. To enhance our diffusion model's capability for adversarial packet generation, we simplify the model's state transitions by defining z t = x t ⊕ y t , where x t and y t correspond to the portions of z t that belong to S ben and S mal , respectively. This setup allows us to strategically inject noise into only the malicious packet sequences (represented by y t ), rather than the entire state z t , during each forward step q(z t | z t-1 ). This targeted noising, inspired by DiffuSeq, is pivotal for adapting conventional diffusion models for targeted modification.Reverse Process With Normal Packet Guidance. Building upon the foundational principles of DiffuSeq, our model introduces an innovative reverse process tailored for more nuanced handling of network traffic data. This process distinctively utilizes normal packet sequences as a guiding framework, enabling a sophisticated denoising technique that treats the concatenated sequences of normal and noise-added malicious packets as a unified unit. This approach effectively "teaches" the model to perceive these malicious elements as integral parts of the normal traffic pattern. A key aspect of this reverse process is the use of the pre-trained BERT model as the denoising engine during packet reconstruction. In this phase, the BERT model undergoes fine-tuning within the diffusion framework, ensuring that it is specifically adapted for reconstructing packets. As a result, the BERT model is not only responsible for contextual understanding but also plays a central role in recovering malicious packets that have been integrated into normal traffic patterns. The conditional denoising process effectively employs Bayesian inference to parameterize the transition probabilities between states, ensuring precise control over each step in the reverse process. These parameterizations are mathematically articulated through the equations:p θ (z 0:T ) := p(z T ) T t=1 p θ (z t-1 |z t ) and p θ (z t-1 |z t ) = N (z t-1 ; µ θ (z t , t), σ θ (z t , t), where µ θ (•) and σ θ (•) model the mean and variance necessary for the stochastic reverse transition from z t to z t-1 , respectively. To rigorously ensure that the malicious packets are not only recovered but also convincingly mimic normal packets, our model optimizes a specially formulated variational lower bound L V LB . This objective underscores the model's effectiveness in blending malicious packets within normal packets:min θ LV LB = min θ T t=2 ∥z0 -f θ (zt, t)∥ 2 + ∥EMB(S ben⊕mal ) -f θ (z1, 1)∥ 2 -log p θ (S ben⊕mal | z0) (4)This objective is particularly focused on accurately reconstructing the initial state z 0 from the noised states, with a distinct emphasis on ensuring that the malicious components are seamlessly integrated into the normal packets pattern.Preserving Packets' Integrity. We propose an innovative approach that utilizes a parallel data structure called "model dictionary" that clearly distinguishes between mutable and immutable fields within each packet. By categorizing fields into mutable (e.g. TCP flags, TTL, and window size) and immutable (including the critical 5-tuple information and payload), we ensure that modifications during training do not compromise the packet's integrity. The model dictionary serves a dual purpose: it retains the original values of immutable fields and establishes a link to their mutable counterparts. This approach allows for dynamic modification of mutable fields during training without affecting the core attributes of the packet. After training, any changes made to mutable fields are seamlessly integrated with the preserved immutable fields. This recombination ensures that the modified packets maintain their operational integrity and are indistinguishable from genuine traffic in real-world scenarios. By bypassing traditional domain constraints-which typically complicate training and pose convergence challenges-our method simplifies the training process and obviates the need for additional compensatory loss functions (Letcher [2021] ). The outcome is a highly effective generation of adversarial packets that are capable of evading detection without compromising the essential characteristics of the original packets. 
experiments: 
 We conduct experiments to validate the performance of DiffuPac on 6 types of attacks, against 6 classifiers.Dataset. The datasets used for this model are Kitsune Dataset (Mirsky et al. [2018] ) and CICIDS-2017 Dataset (Sharafaldin et al. [2018] ). Initially, pre-training of the BERT model utilizes a large subset of unlabeled network traffic to leverage the model's capability to capture diverse traffic patterns, accounting for 60% of the total data. Fine-tuning phase, the focus shifts to a smaller, labeled dataset, which constitutes 20% of the total data. This dataset is distinctly partitioned into malicious and normal packets, which is crucial for training the model to mimic malicious packets as normal. Training the classifier and NIDS then utilizes another 10% of the total data consisting of labeled portion. Testing phase is conducted with the remaining 10% of the data, reserved exclusively for evaluating the model's efficacy. This phase includes testing both original and mimicked malicious packets to rigorously assess the model's real-world applicability and its capability to generalize across unseen data.Baseline Models. We evaluate two baseline models-Traffic Manipulator (Han et al. [2021] ) and TANTRA (Sharon et al. [2021] )-both of which represent current strategies for evading NIDS but share a significant limitation in their reliance on specific operational assumptions. Traffic Manipulator formulates the evasions as a bi-level optimization problem, where the first level uses GAN to identify adversarial features closely mimicking the original network features, and the second level utilizes PSO to modify packet characteristics to exploit vulnerabilities in NIDS. TANTRA employs a three-step process involving an LSTM trained to model benign traffic by predicting inter-packet delays. This LSTM then adjusts malicious traffic to mimic these benign patterns before deployment. Both models, however, assume a degree of accessibility to NIDS configurations or feature extractors that is often unrealistic in practical scenarios. This reliance on specific knowledge about NIDS configurations creates a critical dependency that can limit the deployment and scalability of these evasion techniques in diverse operational environments. Conversely, DiffuPac operates under the assumption of zero prior knowledge about NIDS configurations or feature extractors.Implementation Details. DiffuPac leverages a BERT model with 12 transformer blocks, each featuring 12 attention heads and an embedding dimension of 768, to capture nuanced relationships in network traffic. This setup supports the maximum sequence length of 512. The diffusion model incorporates time step embedding to provide temporal context, enhancing the model's understanding of the reverse process. To manage the computational demands, particularly during the sampling phase, a projection layer reduces the embedding dimension to 128. Additionally, DiffuPac utilizes the FAISS library (Douze et al. [2024] ), renowned for its capability to handle large-scale similarity search and clustering of dense vectors.Experimental Details. Our experimental framework integrates advanced feature extraction tools and a diverse set of machine learning classifiers, building on methodologies established in the Traffic Manipulator research. We employ 2 feature extractors: AfterImage (Mirsky et al. [2018] ), which provides detailed packet statistics, and CICFlowMeter (Draper-Gil et al. [2016] ), which assesses connection-level metrics. Recognizing the documented shortcomings of the original CICFlowMeter, our research employs a revised version that integrates enhancements and corrections proposed in recent studies (Engelen et al. [2021] , Liu et al. [2022] ). These modifications are crucial in ensuring more accurate and reliable feature extraction for our analysis. AfterImage and CICFlowMeter offer a comprehensive view of network traffic, capturing both packet-level and flow-level data essential Evaluation. To assess the effectiveness of the mimicked packets in evading the 6 classifiers, we employed a singular, highly illustrative metric from the Traffic Manipulator: the Malicious traffic Evasion Rate (MER). This metric is calculated using the formula: MER = 1 -N adv /N mal , where N adv and N mal represent the number of detected adversarial and detected malicious packet sequences, respectively. This equation captures the percentage of adversarial packet sequences that goes undetected as compared to the original malicious packet sequences that was detected, effectively measuring the evasion capability of the adversarial packet sequences.We employed the two-sample Kolmogorov-Smirnov (K-S) test, a powerful non-parametric method used to determine the probabilistic differences between two data samples. Specifically, we compare the empirical cumulative distribution functions (eCDFs) of both the original malicious packets and the adversarial packets generated by our model. The K-S test calculates the maximum distance (K-S statistic, D) between these two eCDFs. This statistic measures the greatest deviation between the distribution of our adversarially modified packets and the original malicious packets (Hore et al. [2023] ; Gretton et al. [2012] ). A D value exceeding the critical threshold at a chosen significance level suggests a rejection of the null hypothesis-that is, the two samples are not derive from the same distribution. We demonstrated adversarial packets that were found to be out-of-distribution (OOD) at 95% significance level.Additionally, we used Wireshark, a widely recognized network protocol analyzer to ensure that during the modifications were made, the immutable fields crucial for the packet's attack functionality were preserved. This visual verification via Wireshark confirms that our adversarial packets maintain their structural integrity and functionality, successfully mimicking genuine network traffic while evading detection (Due to limited space, results and analysis are in Appendix B.1).We also evaluated DiffuPac's efficacy in generating adversarial packets that maintain their malicious functionality in a controlled test environment using UTM hypervisor technology. This setup included two virtual machines: one running Kali Linux (attacker) and another running Ubuntu 24.04 (victim).The isolation ensured a realistic yet controlled network environment. We conducted the evaluation on two type of attacks: Port Scan and Brute Force. (Due to space constraints, results and analysis of the two attacks are in Appendix B.3.1 and B.3.2).Evasion Rate. We evaluated the evasion rates of the 6 attacks using DiffuPac and two baseline models, and compared the results in TANTRA also demonstrated high evasion rates, particularly with AfterImage, thanks to its use of LSTM models trained on benign traffic from the targeted network. This training allows TANTRA to reshape the timing between packets. This specific adjustment of interpacket delays directly impacts the time-based features extracted by AfterImage, making it more difficult for the classifiers to distinguish between normal and malicious packets.Quantitatively, DiffuPac outperformed Traffic Manipulator by an average of 9.12 percentage points and TANTRA by an average of 4.26 percentage points across all attack types. The overall average improvement of DiffuPac over both baselines is 6.69 percentage points, demonstrating its superior capability to generate adversarial packets that effectively evade detection.Based on the results of all the attacks, we can conclude that ML classifier are more robust than DNNs to adversarial attacks. Traditional ML models, like DT and IF, have simpler and more interpretable decision boundaries, making them less susceptible to subtle adversarial modifications (Sauka et al. [2022] ). In contrast, DNNs with their complex and high-dimensional decision boundaries are more easily misled by the nuanced modifications introduced by DiffuPac.Statistical Difference (K-S test). The percentage of successful adversarial packets that were found to be OOD are demonstrated in Table 3 . From the result, we can obeserve that Brute Force and DDoS attacks show the highest percentage of OOD samples. The grounded reasons for these kind of attacks are due to the inherent nature of their traffic patterns. The nature of Brute Force attacks involves numerous failed login attempts, leading to highly irregular sequence numbers (Javed and Paxson [2013] ). Even after modifications by DiffuPac, these attempts will stand out due to their frequency and pattern, resulting in significant deviations in the eCDFs. The sheer volume and repetitive nature of DDoS traffic (Haseeb-ur rehman et al. [2023] ) make it difficult to disguise effectively. Despite efforts of intelligent mutation process, the high volume and distinctive traffic patterns will cause substantial deviations in the eCDFs as well. On the other hand, the MITM show the lowest percentage of OOD samples. Adjusting TTL values is straightforward as they can be set to typical ranges seen in normal packets without significantly altering packet behavior, resulting in minimal deviation in the eCDFs.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The paper provides a detailed explanation of the model architecture and the experimental setups. This includes a thorough description of the pre-trained BERT model integration, the diffusion model, and the specific configurations used in the experiments. Additionally, all hyperparameters, dataset details, and evaluation metrics are disclosed to enable replication of the results. 
limitations: 
 Our evaluations demonstrated DiffuPac's strong evasion capabilities, enabling the generation of adversarial packets that convincingly mimic legitimate network traffic, particularly in packet header fields. While these results highlight DiffuPac's ability to evade detection by NIDS, there remain areas for further study.In particular, while we assessed the malicious functionality of generated adversarial packets in a controlled environment using UTM hypervisor technology, this evaluation was limited to Port Scan and Brute Force attacks. Although DiffuPac successfully retained the malicious intent of these attacks, further evaluations are required to determine its effectiveness across a broader range of attacks, especially those with more complex behaviors such as DDoS or Botnet attacks.We also noted that DiffuPac's evasion performance is not uniform across all scenarios, notably DDoS and Brute Force. This variability is due to the distinct nature of each attack's traffic patterns, which may not be fully captured by the model, especially in highly repetitive or anomalous behaviors. To enhance the model's adaptability and generalizability, we plan to enrich the training dataset with a broader spectrum of real-world attack scenarios.Moreover, the dual-use nature of adversarial generation models like DiffuPac presents significant ethical and legal challenges. While designed to improve security defenses, these technologies could be misused for malicious activities, such as facilitating cyber-attacks or disrupting services. To mitigate these risks, we have opted not to publicly release saved model checkpoints, aiming to prevent exploitation by malicious entities and ensure that our advancements in adversarial packet generation are used responsibly and within ethical bounds.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Due to ethical concerns, we have decided not to publicly share the data and code. Additionally, much of the code is based on existing codebases from other sources. However, we have provided a full and detailed explanation of the model architecture and its implementation in the paper and supplemental material. This includes comprehensive descriptions of the experimental setups, hyperparameters, and step-by-step instructions to facilitate reproducibility.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion and future directions: 
 In this study, we introduced DiffuPac, an intelligent generative model that successfully generates adversarial packets capable of evading advanced NIDS while maintaining attack functionality. Unlike previous research, which often assumes attacker access to NIDS, DiffuPac operates under constraints of limited attacker knowledge, reflecting more realistic scenarios. Here, we present the in-depths insights obtained from this study: (a) DiffuPac uniquely combines normal and malicious packet sequences using contextual alignments, ensuring seamless integration into genuine traffic while employing these normal packet sequences to guide the denoising process. Through the performance evaluations with various NIDS, our model achieved an average improvement of approximately 6.69 percentage points in evasion rate compared to the two baselines across all attack types. (b) Evasion rates varied notably across attack types, with DDoS and Brute Force attacks showing higher probabilistic differences due to their complex, repetitive nature. This highlights potential areas for further refinement in DiffuPac's approach to handling voluminous attacks. (c) Simpler ML models like DT and IF displayed surprising resilience due to their less complex decision boundaries, limiting the effectiveness of DiffuPac's modifications. In contrast, DNNs, with their intricate decision boundaries, were more vulnerable, underscoring the complexities inherent in designing robust adversarial tactics.(d) DiffuPac's capability to balance sophisticated attacks with operational stealth makes it especially suitable for environments where attackers lack comprehensive NIDS configurations, enhancing its utility in realistic defense testing.As future studies, we will compare DiffuPac against a broader generative adversarial packets model and extend testing across more diverse NIDS to better understand its relative strengths and limitations.Next, we will further expand the range of attack types included in the malicious functionality evaluation. Moreover, adversarial defenses are a key focus of our ongoing research, particularly in relation to DiffuPac's capabilities. One promising direction is the development of adversarial purification using diffusion models, a technique successfully applied in image processing. This approach treats adversarial perturbations as noise, utilizing the diffusion model's reverse process to restore network packets to their original state before analysis by NIDS. We believe this method has significant potential for enhancing network security and aim to further develop and test it as a novel defense strategy.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have provided the full detailed explanation regarding the training process and the experimental details both in the main paper and supplemental material. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The experiments conducted in this paper primarily focus on evaluating the success of generating adversarial packets. Metrics such as evasion rate, packet field analysis, and probabilistic differences between original and adversarial packets are used to assess the effectiveness of the proposed method.Guidelines:• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We have discussed the positive societal impacts and negative societal impacts in the main paper. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Due to ethical concerns of our paper, we have implemented stringent measures. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All existing assets used in this paper, including code, data, and models, have been properly credited to their original creators. We have cited the original papers that produced these assets and included URLs where applicable.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The paper introduces a new model, DiffuPac, which integrates a pre-trained BERT with a diffusion model to generate adversarial packets. Detailed documentation for DiffuPac is provided, including descriptions of the model architecture, training process, implementation details, and usage guidelines. This documentation is included in the main paper and supplemental materials.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 BERTThe core of DiffuPac is the BERT model, specifically adapted to serve as the denoising component in the diffusion process. The architecture consists of 12 bidirectional Transformer blocks and each block incorporates 12 attention heads within its self-attention layers (Vaswani et al. [2017] ). These layers are crucial for capturing the intricate and implicit relationships between traffic bytes. Each input token has a dimension of 768, and the model can process sequences with up to 512 tokens. This setup allows the BERT model to effectively handle extensive sequences of traffic data, making it well-suited for the task of denoising during the diffusion process.To implement of our pre-trained BERT model, we utilized the codebase from the UER (Universal Encoder Representations)-py project (Zhao et al. [2019] ). UER-py is a versatile toolkit designed for pre-training and fine-tuning various NLP models, including BERT. In UER-py, the MLM is designed to predict masked words within a sentence. We adapted this approach to our masked unidirectional flow model, which focuses on predicting masked tokens within network traffic sequences. Instead of textual tokens, our model deals with traffic units from src-to-dst and dst-to-src sequences. We modified the MLM implementation to handle these traffic tokens, ensuring that each token is masked with a probability of 15%. Our model then predicts the original token from the masked sequence, capturing semantic patterns within bidirectional network flows.The Next Sentence Prediction (NSP) task in UER-py involves prediction step to determine whether one sentence follows another. We transformed this concept into the same sequence-origin prediction task to determine the origin of directional network traffic. We adapted the binary classifier used in NSP to classify whether packets are originated from src-to-dst or dst-to-src. By pairing packets and predicting the directional flow, our model learns the contextual relationships and patterns in network traffic, similar to how NSP learns sentence relationships.Table 4 lists the parameters are set to run our pre-trained BERT model. The remaining parameters were set to their default values.We also conducted a comprehensive evaluation of different BERT model configurations to identify the most efficient option that still maintains high performance. We tested four BERT model variants: Tiny, Small, Medium, and Large, comparing their specifications, training time, loss, and accuracy. The results are summarized in Table 5 . The Medium BERT model was chosen as the optimal configuration for DiffuPac because it offers a balanced trade-off between accuracy and loss metrics. For the fine-tuning phase of our model, we leveraged the DiffuSeq codebase (DiffuSeq's code). The DiffuSeq framework, originally designed for text data, required several significant adaptations to handle network traffic data and to meet the objectives of our diffusion model for adversarial packet generation. The primary adaptation to the DiffuSeq codebase involves the data preparation phase, In our concatenation strategy, we identified pairs of normal and malicious packet sequences that exhibit strong contextual alignments unlike DiffuSeq, which, randomly merge the text sequences. This means that the patterns and behaviors within these sequences are similar, ensuring that the concatenated sequences are blended seamlessly. We concatenated these contextually aligned normal and malicious packet sequences to form a new sequence, S ben⊕mal . The strategy leverages the inherent patterns and behaviors within normal packet sequences, which is crucial for the mimicking process of malicious packet sequences.We utilized the pre-trained BERT embeddings in our concatenation strategy. BERT is designed to capture deep semantic relationships within sequences. This design makes BERT making it highly effective for analyzing complex patterns in network traffic. By employing the pre-trained embeddings, DiffuPac can accurately transform discrete packet data into continuous feature spaces, capturing the intricate dependencies between packets. This transformation is critical for the subsequent similarity analysis and pairing process. The pre-trained embeddings offer a rich representation of the packet sequences, so that the model can discern subtle contextual similarities between normal and malicious packet sequences. The detailed of the concatenation strategy is demonstrated in Algorithm 1.Here, we set the similarity threshold, ϵ to 0.9. This is a strategic decision aimed at balancing the need for high contextual alignment with the practicalities of ensuring a sufficient number of matched pairs. Setting ϵ to 0.9 indicates that we are looking for pairs of packet sequences that are highly similar, capturing nearly all the essential contextual information shared between normal and malicious packet sequences. This high threshold ensures that the concatenated sequences are blended seamlessly, maintaining the realistic traffic patterns and behaviors required to evade detection by NIDS. From a practical standpoint, a threshold of 0.9 is chosen because it strikes an optimal balance between precision and recall. A higher threshold (closer to 1.0) would result in fewer matched pairs, as only the most similar sequences would be selected. This could limit the model's ability to generate a diverse set of adversarial packets. Conversely, a lower threshold would increase the number of matched pairs but might include sequences that are not contextually well-aligned, reducing the effectiveness of the adversarial packets in mimicking normal traffic.During the reverse process, we introduced a guidance mechanism using normal packet sequences. This mechanism treats the concatenated sequences of normal and noise-added malicious packets as a unified entities, allowing the model to denoise the malicious packets in the context of the normal packet sequences. We recalibrated the variational lower bound L V LB to emphasize the integration of normal sequence guidance. The objective focuses on accurately reconstructing the initial state, z 0 from the noised states, so that the malicious components are seamlessly integrated into the normal packet sequences pattern. if σ max > ϵ then To demonstrate that our model intelligently alters specific fields to seamlessly blend malicious packets into normal traffic, we analyzed the packets using Wireshark. Our analysis covered 6 attack types: MITM, Botnet, Brute Force, DDoS, Port Scan, and Infiltration. The results showed that different specific fields were modified in the packets corresponding to each type of attack. This variation in field alteration confirms that our model adapts its modifications according to the nature of the attack, enhancing the stealthiness of the malicious packets. In Figure 2 , we can see that there is a change in the window size of a Botnet packet. Botnet attack can be described as a network of infected devices (bots) controlled by an attacker to perform various malicious activities. Botnet traffic often shows abnormal patterns in window size due to automated control and data bursts. Here, we can conclude that DiffuPac made the intelligent decision by itself in modifying the window size to fall within typical user traffic patterns. This helps to camouflage the botnet activity.MITM, on the other hand, demonstrated attacks whereby an attacker intercepts communication between two parties without their knowledge, often to eavesdrop or alter the data being sent. Abnormal TTL values (time-to-live field) can reveal this kind of interception and redirection. Here, once again, our model intelligently adjusts TTL value to match the expected values in aiding the masking of MITM presence. As shown in Figure 3 , DiffuPac adjusts the TTL values to effectively conceal signs of interception. This ensures that the adversarial packets blend seamlessly with normal data exchanges, mimicking legitimate communication between parties.DDoS attacks involve a large number of packets with similar characteristics, often reusing or predictably incrementing the IP Identification (ID) field. In Figure 4 , it can be shown that DiffuPac addresses this by randomizing the IP IDs to prevent detection. When we analyzed the traffic using Wireshark, we observed varied and less predictable IP IDs in the modified packets, aligning with normal traffic patterns. This randomness helps avoid forming detectable patterns and evades detection systems that rely on identifying repetitive IP ID sequences. Thus, DiffuPac successfully masks the attack's presence while preserving the attack functionality.Port Scan often involve specific flags, such as SYN, to probe for open ports. This behavior is distinct from normal traffic, which uses a variety of flags. In Figure 5 , we found that DiffuPac diversifies the use of flags to include SYN, ACK, and FIN, similar to normal connections. By normalizing the traffic in this way, DiffuPac helps disguise scanning activity. For example, instead of sending many SYN packets, it includes a mix of SYN, ACK, and FIN packets, which helps blend the scan into typical network behavior and evade detection.Brute force attacks generate numerous login attempts, leading to irregular sequence numbers due to the repeated connections. By analyzing the traffic with Wireshark, we observed that DiffuPac intelligently normalizes sequence numbers to mimic regular user connections. Normal user login attempts typically show sequentially increasing sequence numbers. In Figure 6 , it shows that DiffuPac adjusts these numbers to follow a similar pattern, thereby camouflaging the brute force activity. This adjustment helps blend the attack packets seamlessly into benign traffic, making it harder for NIDS to spot the abnormality.Infiltration attacks involve moving laterally within a network, often generating unusual acknowledgment numbers as the attacker accesses various systems. In Figure 7 , it demonstrated that DiffuPac intelligently modifies acknowledgment numbers to follow expected sequences. In normal communication, acknowledgment numbers increment predictably based on the received data. DiffuPac ensures that infiltration attempts have acknowledgment numbers that follow these normal patterns, effectively masking the lateral movement within the network and blending the attack traffic with normal traffic, making it difficult for detection systems to identify the anomaly.DiffuPac demonstrates its ability to modify specific fields intelligently in the packet headers of different attack types, such as the window size for botnet traffic and TTL for MITM, to blend malicious packets into normal packets. This capability is consistently proven through detailed packet analysis with Wireshark. Indeed, the detailed packets analysis shows that critical fields defining packet identity and practicality (e.g., IP addresses, ports, payload) remain intact after modification, ensuring the functionality of the attack while evading detection by sophisticated NIDS. The comparisons for each attack type before and after the modificatinos further highlight DiffuPac's effectiveness in maintaining attack practicality while enhancing stealth.We evaluated the evasion rates of the other 4 attacks-DDoS, Port Scan, Brute Force, and Infiltration-using DiffuPac and the baseline models, with the detailed results summarized in Table 7 .According to the results, we can observed that the trends in the 4 attacks are consistent with the 2 attacks shown in Table 2 in the main paper. Experimental Setup. We conducted a robust evaluation of both Port Scan and Brute Force attacks to assess their malicious functionality in controlled, isolated network environments. The primary goal of this evaluation was to determine whether adversarially modified packets retain their ability to perform malicious actions. Throughout the evaluation, we focused on analyzing the response of both the original and adversarial packets using Wireshark, carefully observing how the target system reacted to each type of attack.Before proceeding to evaluate the generated adversarial packets of DiffuPac, we first demonstrated a a successful Port Scan attack within our controlled environment. This initial step is a fundamental step to ensure the validity and reliability of subsequent tests with both original and adversarial packets. The network configurations depicted in Figure 8 (a) and Figure 8 (b) confirm that both virtual machines were correctly placed within the same subnet. As demonstrated in Figure 8 (c), the Port Scan successfully identified open ports on the Ubuntu, highlighting port 22 as vulnerable.We initiated a Port Scan attack from the Kali Linux machine targeting the Ubuntu server. During the execution of this attack, we captured all packets sent from the attacker to the victim, saving them as a pcap file. This capture was timed precisely to begin as the Nmap tool launched the Port Scan, ensuring that only the relevant src-to-dst packets were recorded. This focus aligns with the operational framework of DiffuPac, which concentrates on modifying packets sent from the source.Using the captured src-to-dst packets, we employed DiffuPac to generate adversarial Port Scan packets, which were then saved in a separate pcap file. These adversarial packets were subsequently replayed using the Tcpreplay tool, directing them towards the Ubuntu server. Concurrently, we used Wireshark on the Ubuntu machine to capture the incoming responses, allowing us to analyze the effectiveness of the adversarial packets in real-time.To ensure a balanced evaluation, we also replayed the original Port Scan pcap file using Tcpreplay, capturing the responses in Wireshark on the Ubuntu side. This dual replay enabled a direct comparison between the packets' responses generated by the original and adversarial Port Scan packets.As depicted in Figure 9 , the response to the adversarial Port Scan packets closely mirrors that of the original Port Scan. A detailed inspection of the TCP flags reveals that packets with SYN and ACK flags targeted port 22, corroborating the initial vulnerability identified in Figure 8(c ). This consistent response across both original and adversarial packets provides compelling evidence that DiffuPac successfully generates adversarial packets that retain their intended malicious functionality. For the Brute Force attack evaluation, the experimental setup followed a similar approach to that of the Port Scan, focusing on capturing and analyzing both original and adversarial packets. However, in this case, we leveraged Metasploit's ssh_login module to perform the attack. After demonstrating a successful Brute Force attack using Metasploit, we captured the relevant packets and utilized DiffuPac to generate adversarial versions of those packets. The evaluation focused on comparing the response of original and adversarial packets, observing how successful logins were achieved in both cases.As shown in Figure 10 , we utilized the Metasploit Framework to perform a Brute Force attack targeting the SSH service on the Ubuntu server. After setting the RHOSTS to the Ubuntu machine, we provided both a user file and a password file containing potential login credentials. The ssh_login module systematically attempted each combination of usernames and passwords, ultimately leading to a successful login.The success of the Brute Force attack is demonstrated in Figure 10 , where Metasploit successfully opened an SSH session on the target machine. This provided the necessary confirmation that the attack could be executed under normal conditions, laying the foundation for the subsequent evaluation of adversarial packets.Following the successful attack, similar to the Port Scan setup, we captured the src-to-dst packets exchanged during the Brute Force attack. These packets were saved as a pcap file, which was subsequently used as input for DiffuPac to generate adversarial Brute Force packets.Using Tcpreplay, both the original and adversarial Brute Force packets were replayed towards the Ubuntu server. As shown in Figure 11 , the response to the adversarial Brute Force packets closely mirrored that of the original Brute Force packets. The packet exchange between the attacker and the target server increased significantly as the correct login credentials were identified, culminating in a successful SSH session for both the original and adversarial attacks.To further analyze and validate the outcomes of both the original and adversarial Brute Force attacks, we utilized pam_unix(sshd) on the Ubuntu server to log and compare successful login attempts. As depicted in Figure 12 , the log entries confirmed that the SSH service recorded successful logins for both the original Brute Force attack and the adversarial Brute Force attack. This ablation study assesses the impact of substituting the pre-trained BERT model with a standard transformer architecture in the denoising process of DiffuPac. The focus is on assessing each model's ability to accurately reconstruct and generate adversarial packets that can effectively evade detection.Non-BERT-based. This variant of DiffuPac employs a standard transformer architecture for the denoising process, as outlined in the DiffuSeq methodology (Gong et al. [2023] ). Notably, this model is trained from scratch without leveraging any pre-trained weights, thus relying solely on the information provided during the training phase.BERT-based. The existing DiffuPac configuration, which incorporates a pre-trained BERT model for the denoising process, was utilized. This setup takes advantage of BERT's pre-trained weights, enabling enhanced semantic understanding and contextual relevance during packet reconstruction.Both models were trained under identical conditions and evaluated based on their success in generating adversarial packets that evade detection. The results, as depicted in Table 8 , highlight the comparative effectiveness of each model.As shown in Table 8 , the BERT-based DiffuPac significantly outperforms the non-BERT variant, demonstrating superior ability to reconstruct packets that blend seamlessly into normal traffic. The BERT-based model excels in pairing malicious packets with normal packets that exhibit high contextual relevance. This strategic pairing allows the generated adversarial packets to blend seamlessly into normal traffic, thereby increasing their likelihood of evading detection. In contrast, the non-BERT-based model, which lacks this pre-trained knowledge, resorts to random matching of malicious and normal packets, leading to a noticeable reduction in the effectiveness of the generated adversarial traffic.An intriguing aspect of DiffuPac is its dual utilization of the pre-trained BERT model: first, as the denoising engine during the packet reconstruction process, and second, as a critical tool for identifying contextually relevant packet pairs. To the best of our knowledge, this dual application of a pre-trained model within a diffusion framework is unprecedented, further highlighting the innovation and effectiveness of our approach.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: The proofs for all theorems and formulas are clearly provided both in the main paper and supplemental material.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The paper provides a novel method to evade Machine Learning-based Network Intrusion Detection Systems (ML-NIDS). From a security viewpoint, the major contribution of the paper is that the proposed method can work even in a very constrained setting wherein the attacker has no knowledge/access to the targeted ML-NIDS. Such a property is provided by leveraging models "pre-trained" on (well-known) datasets. The experiments validate the effectiveness of the method.

No reviewer pointed out critical flaws in the evaluation, and all reviewers praised the quality of writing of the paper. The major concerns on this paper were related to the guarantee that the method achieves evasion while preserving the original malicious functionality of the generated data. The authors have responded to this concern by carrying out some validatory experiments during the rebuttal. Nonetheless, such a verification is challenging to carry out, and few works have addressed this aspect (which is crucial from a security viewpoint).

Given the above, the paper does provide a meaningful contribution in the domain of "applied" ML for cybersecurity, and while the results are not groudbreaking and may present some uncertainties, there is reason to believe that this paper will inspire future work in this domain.

I also favor making the discussion between authors and reviewer publicly visible. There is much to learn in the comments/remarks generated during the discussion phase.

Reviews: review: paper_summary: paper_summary: This work proposes DiffuPac, an adversarial packet generation model that aims to evade Network Intrusion Detection Systems (NIDS) without depending on detailed knowledge of NIDS components. DiffuPac combines a pre-trained BERT model with a diffusion model. Experimental results show that DiffuPac outperforms traditional methods by 8.83%.
strength_weakness: strength_weakness: - The paper achieves the following goals through expensive experiments (more than 30 hours of training on RTX 4090): select the normal packet sequence that is most similar to the malicious packet sequence, and modify the limited mutable fields (e.g. TCP flags, TTL, and window size) in the malicious packet sequence header according to the normal packet sequence. Given the very limited mutable fields modified within the packets and the lack of ablation studies in the experimental section, I doubt the necessity of using BERT and DiffuSeq. It seems that a simpler approach involving feature similarity matching, coupled with selective field selection and replacement, could potentially achieve the same goal without the complexity introduced by these advanced models.
- In the diffusion model where all mutable fields of the entire malicious packet sequence are regenerated, there is a concern about whether this process affects its original functionality. Although the modification effects on specific fields of individual packets are shown in the appendix, whether the author replays these generated packets in a real network environment to verify its functionality, rather than simply viewing them through Wirshark.
- Some minor issues: The paper needs improvement in writing quality. The connections between different components are not explicitly explained, and the specific format of the packet sequence input to the diffusion model is not detailed. In addition, the BERT-related content is very redundant in Sec. 3 and Figure 1. However, the related content is highly similar to ET-BERT and is only used to calculate the most relevant benign and malicious packet sequences in the "fine-tuning" stage (in fact, I think the operations in this stage do not belong to the category of "fine-tuning").
questions: questions: Please see the above-mentioned weaknesses. I will confirm the author's response during the rebuttal.
limitations: limitations: The authors discussed the limitations in this work.
paper_summary: paper_summary: This work presents DiffPac, a method that leverages BERT and a diffusion model to generate adversarial packets aimed at evading network 
intrusion detection systems. Compared to the approach by Han et al. (2021), DiffPac demonstrates superior effectiveness, with most machine learning-based classifiers trained on adversarial packets generated by DiffPac successfully evading detection across six different attack scenarios.
strength_weakness: strength_weakness: * Motivation: The research motivation could be more clearly and concretely articulated. It would be beneficial to highlight the specific niche of this study in relation to existing work, such as Network Emulator (NetEM) and Metasploit (Homo-liak et al., 2018), Generative Adversarial Network (GAN) and Particle Swarm Optimization (PSO) (Han et al., 2021), and Reinforcement Learning (RL) (Hore et al., 2023). Rather than broadly stating that "their efficacy in evading detection in real-world conditions remains suboptimal" (line 64), the paper should detail the unique challenges and gaps that DiffPac addresses.

* Method for Preserving Packets’ Integrity: The approach to preserving packets' integrity by only mutating TCP flags, TTL, and window size, rather than the payload, is unrealistic for real-world conditions where payloads often contain the malicious behaviors. This limitation should be addressed to enhance the practical applicability of the method.

* Evaluation: The evaluation settings could be more closely aligned with those used by Han et al. (2021) to make it easier for the readers to understand the differences and contributions of this work.
questions: questions: * Data Pre-processing: Could you explain the rationale and advantage of transforming IP numerical values into hexadecimal representation and treating them as discrete values for workpiece segmentation? What benefits does this approach offer?

* Formula (4): In Formula (4), does  S^{ben+mal}  denote z_t ? 

* Dataset: How many adversarial packets are included in the test set? Providing a statistical description of the datasets used would enhance clarity and help understand the dataset composition.
limitations: limitations: The limitation section highlights three main concerns:

* The actual malicious functionality of the generated adversarial packets has not been thoroughly examined.

* There is inconsistent performance across the six attack scenarios tested.

* There is a potential risk associated with making the approach open-source, which could be exploited maliciously.
paper_summary: paper_summary: The paper presents DiffuPac, a novel solution that integrates Bidirectional Encoder Representations from Transformers (BERT) and diffusion models to generate adversarial packets that can evade detection by sophisticated Network Intrusion Detection Systems (NIDS).
DiffuPac leverages the extensive contextual understanding provided by BERT, which has been trained on diverse datasets representing a wide range of network behaviors, along with the generative capabilities of diffusion models. This fusion results in a sophisticated adversarial tactic where the elements of the attack are seamlessly integrated into the network traffic, making them indistinguishable from legitimate data.
Extensive experimental evaluations on real-world datasets demonstrate that DiffuPac significantly outperforms existing methods in terms of evasion effectiveness, establishing new benchmarks for the generation of adversarial packets.
strength_weakness: strength_weakness: * There are many unclear points about the high-level idea, methodology, and experiments; see them below. If provided with compelling evidence, I may increase my score.
questions: questions: 1. For the high-level idea, I admit that there have been attempts to transform packet bytes as tokens and use language models for traffic analysis (e.g., [a,b,c]). However, most of them (also including DiffuPac) do not discuss if and how this type of method can handle encrypted traffic. As you may know, modern encryption algorithms like AES can significantly reduce entropy and there will be no one-to-one relations between plaintext and ciphertext; so from my perspective, learning from the encrypted payload is meaningless. Some methods may only use the header fields for tokenization. As there lack of such details, I'm not sure how DiffuPac deals with this concern.
2. On page 5, line 199: "... this task employs a dedicated binary classifier to determine the directional origin of network traffic, specifically whether packets in a sequence originate from src-to-dst or dst-to-src." Though the idea seems reasonable, I'm curious will there be any cases where packets from both sides are very similar so that the task can be too difficult to converge (e.g., in P2P apps where there are no explicit servers or clients)?
3. On page 5, line 217: "the initial packets in a flow contain the most significant information, we limit our analysis to the first three packets of each heavy flow". Will the first three packets be too few? For TCP connections, the first three packets are typically just for handshakes so the model probably won't learn any useful information. Can the authors explain this, or give some experiments on this parameter?
4. Could the authors list the mutable and immutable fields in the appendix?
5. For experiment setup, one of my concerns is about the data preparation. Will the pre-trained model use malicious traffic for training? For one thing, the pattern of malicious traffic can be very abnormal so will that affect the BERT's understanding of normal traffic? For another, in practice, malicious traffic is much less than in the dataset so this setting can be somewhat impractical for a pre-trained model.
6. Another major weakness of the experiment is that there is only one baseline. However, there are other works on generating (malicious or generic) traffic data, including those generating on feature spaces ([d,e]) or generating on raw bytes ([a,b]). The authors may compare them to improve the experiment, or discuss them in related work and present their fatal limitations that hinder the direct comparison.
7. As there are two main components in DiffuPac (BERT and diffusion model), an ablation study on the pre-trained model is needed to show the necessity of using the pre-trained representations instead of some simpler representations.
8. In the limitations, I suggest that the authors discuss the implications of your method facing signature-based NIDS (e.g., Snort). You may conclude with a consideration of the current situation that encrypted traffic is the majority, but that will go back to question 1 again.
9. I like the authors providing the screenshots of Wireshark as parts of the results. One weakness is that the authors use their own domain knowledge to reveal the important packet fields that should be altered to avoid NIDS. However, the ML/DL-based NIDS may use some other features as the decision reasons which can be different from yours. I think the authors may use some global explanation models (e.g., [f,g]) to verify if the adversarial fields are really the features that NIDSes care.

[a] NetGPT: Generative Pretrained Transformer for Network Traffic, arxiv 2023

[b] TrafficGPT: Breaking the Token Barrier for Efficient Long Traffic Analysis and Generation, arxiv 2024

[c] Yet Another Traffic Classifier: A Masked Autoencoder Based Traffc Transformer with Multi-Level Flow Representation, AAAI 2023

[d] Flow-based network traffic generation using Generative Adversarial Networks, SIGCOMM 2022

[e] Knowledge Enhanced GAN for IoT Traffic Generation, WWW 2023

[f] AI/ML for Network Security: The Emperor has no Clothes, CCS 2022

[g] Interpreting Unsupervised Anomaly Detection in Security via Rule Extraction, NeurIPS 2023
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a novel strategy named DiffuPac to generate adversarial packets to bypass NIDS detection. DiffuPac encompasses two critical components including the BERT, which captures the semantic meaning of packets and facilitates the embedding and contextual paring. The other one is DiffSeq which makes the malicious part seamlessly integrated into the normal packet pattern thereby improving the evasion against detection. Experiments showcase that DiffuPac outperforms the baseline and is capable of breaking practical detection through simulation.
strength_weakness: strength_weakness: 1. As stated in the limitation, whether the functionality of these adversarial packets is preserved is not verified. From my understanding, this point is quite important given the practical aspect of DiffPac. If the adversarial package can only bypass detection without functioning as intended, it would undermine the practical utility of DiffPac.

2. It would be beneficial to include additional analysis, such as evaluating the performance without applying contextual pairing and assessing the robustness against potential packet-level defenses. These analyses would provide further insights into the effectiveness and resilience of DiffPac.
questions: questions: Please refer to weakness.
limitations: limitations: The authors have listed their limitations and have addressed most of them. One concern about functionality preservation is mentioned in the weakness part.
paper_summary: paper_summary: This paper proposes an adversarial packet generation method, named DiffuPac, to evade detection. DiffuPac integrates BERT and a diffusion model to make the generated packets indistinguishable. To better fit the task, a concatenation strategy and a classifier-free approach are proposed. The experimental results are promising.
strength_weakness: strength_weakness: 1.	The primary concern is the efficiency of DiffuPac. As a diffusion model is adopted to generate adversarial packet, it may be time-consuming. Can you conduct some experiments to validate the efficiency of your method?
2.	Can you also consider some defenses against such adversarial packets? For example, adversarial training (AT), feature selection (FS), and adversarial feature reduction (AFR) as shown in work [1]


[1] Han, Dongqi, et al. "Evaluating and improving adversarial robustness of machine learning-based network intrusion detectors." IEEE Journal on Selected Areas in Communications 39.8 (2021): 2632-2647.
questions: questions: 1.	I understand the effectiveness of DiffuPac in an experimental environment but in the real world, usually, an ML model is adopted to detect evasion for fast response requirements but not a DNN. As DNN is less robust to adversarial examples, will this affect the practicability of DiffuPac?
limitations: limitations: Mentioned in Sec.5.

Comments: comment: We deeply appreciate the reviewer's feedback and apologize for not fully addressing the concerns during the rebuttal phase.

In the revised paper, we will thoroughly address all feedback by increasing the necessary experimental evidence. We are confident that the additional data and analysis will alleviate the reviewer's concerns.
Thank you for the detailed feedback and for highlighting key areas for improvement. We fully agree with the reviewer's suggestions and would like to provide an overview of the approach we will take in the revised paper to address the concerns.

Regarding the baseline, we are currently conducting evaluation on the TANTRA model (Y. Sharon et.al, 2022), and will provide a valuable comparison to DiffuPac. The results of this evaluation will be included in the revised paper, offering a more comprehensive understanding of DiffuPac's performance relative to existing methods.

We will also include a thoroughly detailed description of our threat model in the revised paper. This will clarify the settings in which DiffuPac operates, such as the positions of adversaries and NIDS, and their capabilities, especially in scenarios involving encrypted traffic and signature-based NIDS.

We completely agree with the reviewer that many related works often overlook the evaluation of retaining malicious functionality or do not provide a fully detailed description of this crucial aspect. In our paper, we aim to set a new benchmark in the community by offering a comprehensive, fully-detailed account of how we conduct malicious functionality evaluations. Furthermore, we provide clear evidence that our model is capable of retaining malicious functionality, demonstrating a significant advancement in the field.
We deeply appreciate the response from the reviewer.

First of all, we sincerely apologize for not fully addressing the concerns during the rebuttal phase. Due to time and resource constraints, we were only able to include the Port Scan attack evaluation. However, we have completed the evaluation of Brute Force attacks and are pleased to report that the results are promising. The responses from both the original Brute Force packets and the adversarial Brute Force packets indicate successful SSH access, demonstrating that DiffuPac preserves the original functionality of the traffic. We will include these results in the revised paper, along with evaluations of other attack types. We hope these additional evaluations will alleviate the reviewer's concerns.

Regarding the inconsistency between the BERT-based results in the rebuttal PDF (Table 1) and the submitted manuscript (Table 2), we deeply appreciate the feedback and acknowledge the discrepancy. We suspect that the variations might be due to potential minor differences in the random seeds used during training.

Despite these minor differences, the overall trends remain consistent, and the results continue to demonstrate the promising capabilities of our model. Moving forward, we will do our utmost to ensure that the revised paper reflects consistent and verified results. We will carefully re-run experiments, double-check our settings, and align all reported data to maintain accuracy and clarity.
Thank you for bringing this important issue to our attention. We deeply appreciate your feedback regarding the CICIDS2017 dataset and its known flaws, as highlighted in the works by Engelen et al. (2021) and Liu et al. (2022).

In the field of adversarial attack generation for cybersecurity, CICIDS2017 and its extended version, CICIDS2018, are indeed widely used and recognized datasets. Prominent studies, including recent research like Deep Pack Gen (supported by the U.S. Military Academy) and TANTRA (2022), have continued to utilize these datasets. Additionally, influential surveys, such as "Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey," acknowledge CICIDS2017/2018 as standard benchmarks in the field.

Given the pioneering nature of our work in generating adversarial packets, we aimed to align with the commonly accepted practices in the community by using these datasets. However, we were unaware of the specific flaws that have been mentioned, as they have not been widely addressed in the literature we have encountered. We genuinely appreciate the guidance on this matter.

In response to this valuable feedback, we have already conducted experimental evaluations using a refined version of the CICIDS2017 dataset. The preliminary results from these evaluations indicate an even higher evasion rate than initially reported, which is a promising outcome. We are currently extending these evaluations to the refined version of the CICIDS2018 dataset and will include the results in our revised paper.

Furthermore, in our revised paper, we will include a thorough discussion on the necessity of using the refined CICIDS2017/2018 datasets for evaluating adversarial attack generation. We believe this discussion will have a significant impact on the community, as it addresses a critical issue that has often been overlooked. By introducing this dialogue, we aim to set a new benchmark for the field.

Given that our work introduces a first-of-its-kind adversarial packet generation model, we are confident that this discussion will draw substantial attention from the community. We hope to inspire further research that prioritizes dataset integrity, ultimately contributing to the ongoing improvement of dataset quality in the ML-based/DL-based NIDS community.
Thanks for the author's response. However, only discussion is provided but lacking some experiment results to address my concern. Thus, I tend to keep my score.
Thanks for the response. Some of my concerns have been answered. However, I think the current version of the evaluation using only one baseline is not ready for publication. Besides, I agree with the other reviewers that the discussion about original malicious functionality and payload modification is crucial. In fact, these points are also aligned with my comments on encrypted content and signature-based NIDS which fundamentally care about payload. When revising the paper, I suggest the authors include a better-described threat model to define your settings, such as where adversaries and NIDSes monitor the traffic and what their abilities are. For example, suppose they are all on the ISP side, given the majority of encrypted traffic nowadays. In that case, adversaries cannot modify payloads feasibly, nor can NIDSes use adequate signatures to detect malicious activities. For another, if the NIDSes are on the boundary of the end servers (which have the keys to decrypt the ciphertext), then things will be different. I would like to maintain my score.

Again, I do like the authors considering the practicality of the generated traffic, as I've seen several works that do not have enough consideration while still getting published at high-level conferences and journals; maybe they "luckily" didn't meet expert-level reviewers as you did. This is why I encourage the authors to absorb the comments, in order to get ready for publication on top-tier venues like NeurIPS or Big 4 and make some real contributions to the community.
Thank you for your response. I'm pleased to see the inclusion of payload modification and the plan to expand the evaluation settings in future work. I will be keeping my score unchanged.
Hi Authors,

I have noticed that you used the CICIDS2017 dataset to carry out your experiments. Were you aware that the CICIDS17 dataset is flawed?

This fact has been known since 2021 in the ML-based NIDS community. See: [Engelen, Gints, Vera Rimmer, and Wouter Joosen. "Troubleshooting an intrusion detection dataset: the CICIDS2017 case study." 2021 IEEE Security and Privacy Workshops (SPW). IEEE, 2021.].

Do note that even its extended version (CICIDS2018) is also flawed; this was found in the 2022 best paper award at CNS (see [Liu, Lisa, et al. "Error prevalence in nids datasets: A case study on cic-ids-2017 and cse-cic-ids-2018." 2022 IEEE Conference on Communications and Network Security (CNS). IEEE, 2022.])
Thanks to the authors for their response. I have reviewed the rebuttals. However, showing only the case of Port Scan attack cannot solve my concern about whether DiffuPac affects the original functionality of the traffic. Once the original functionality is destroyed, the practical value of this work becomes questionable. Therefore, this should be a primary consideration before designing the method. In addition, I suggest that the authors carefully check their experimental results, as the BERT-based results in the rebuttal PDF (Table 1) seem inconsistent with those in the submitted manuscript (Table 2). Overall, I am inclined to keep the original scores.
Thanks for the response. The 'Malicious Functionality Evaluation' refers to Lines 382 - 384. While I acknowledge the discussion in the Limitation section, preserving the intended malicious effect is essential to ensure the practicality of your method.