Latency NMS Attacks: Is It Real Life or Is It Just Fantasy?
Authors: Jean-Philippe Monteuuis
Cong Chen
Jonathan Petit
Venue: NeurIPS 2025 poster ()
Keywords: evaluation
adversarial attacks
latency attacks
NMS attack
Object Detection
adversarial defenses
Primary area: evaluation

Abstract: ``Caught in a landslide, no escape from reality" summarizes the state of the research in AI offense: an attack might work on paper but does not necessarily in practice. In the last 5 years, we have seen the rise of latency attacks against computer vision systems. Most of them targeted 2D object detection, especially its Non-Max-Suppression (NMS) block, via adversarial images. However, we uncovered that, when tested in realistic deployment settings, the NMS latency attacks, accepted to top conferences, have very limited negative effects. In this paper, we define an evaluation framework (EVADE) to assess the practicality of attacks, and apply it to state-of-the-art NMS latency attacks. Attacks were tested on different hardware platforms, and different model formats and quantization. Results show that these attacks are not able to generate the claimed latency increase, nor transfer to other models (from the same family or not). 
Moreover, the latency increases remain within the latency requirements of downstream tasks in our evaluation, suggesting limited practical impact under these conditions. We also tested three defenses, which were successful in mitigating the NMS latency attacks. Therefore, in their current form, NMS latency attacks are just fantasy.

Latency NMS Attacks: Is It Real Life or Is It Just Fantasy?
Jean-Phillipe Monteuuis Qualcomm Technologies, Inc. Jonathan Petit
Cong Chen
(jmonteuu, congchen, petit) petitt@@ti..ualccc..... qua.. com
Abstract
"Caught in a landslide, no escape from reality" summarizes the state of the research in Al offense: an attack might work on paper but does not necessarily in practice. In the last 5 years, we have seen the rise of latency attacks against computer vision systems. Most of them targeted 2D object detection, especially its Non-Max- Suppression (NMS) block, via ad images. However, uncovered that, when tested in realistic deployment settings, the NMS latency attacks, accepted top conferences, have very limited negative effects. In this paper, define evaluation framework (EVADE) to assess the practicality of attacks, and apply to state-of-the-ar NMS latency attacks. Attacks were tested on different hardware platforms, and different model formats and quantization. Results show that these attacks not able to generate the claimed latency increase, models (from the same family or not). Moreover, the latency increases remain within the latency requirements downstream tasks in our evaluation, suggesting limited practical impact under these conditions. We also tested three defenses, which were successful in mitigating the NMS latency attacks. Therefore, in their current form, NMS latency attacks are just fantasy.
transfer to other
Introduction
Perception systems use camera inputs to perform tasks such as object detection, object classification, or segmentation. They rely on the performance of machine learning models to offer proper services. Especially, the latency of perception output plays an important role in delivering satisfactory user experience safety-critical functions.
The maximum latency for real-time object detection models depends on the target application. For example, for autonomous driving, the maximum latency is typically around 100 ms to ensure safety and responsiveness (Lin al.) 2018).For surveillance systems, latency of up to 200 ms is acceptable,
as these systems often prioritize accuracy over immediate response (Hussain 2024). In industrial roboties, latencies of up 100 ms are often acceptable, but for more precise tasks, lower latencies around 10-20 ms are preferred (Vijayakumar and Vairavasundaram 2024).
Most of the object detection models use Non-Max Suppression (NMS) process all the candidate bounding boxes and output the most probable ones. It has been demonstrated that an attacker could craft perturbations to create many high-probability bounding boxes that would not be filtered by the NMS, and hence, increase the latency Wang el al. (202T).
going demonstrate in this paper, the prior art omitted many important aspects of deployed ML models that seriously question the relevance of current NMS latency attacks. More precisely, tackle the following research questions.
However,
RQQ: How does the hardware affect the Attack Success Rate (ASR) of latency attacks against NMS?
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Al Hardware
EVADE
Hif
EVADE
Evaluate and Validate Adversarial Digital
HH
x 4
EVADE
Examples
HIT
C
HI
X x x X
Object Detection Models
Figure 1: Overview our evaluation framework (EVADE) to assess practicality of attacks. An input image is adversarially perturbed (to aim increasing latency here). EVADE tests different defenses, model formats, checks impact on performance (c.g., accuracy, number of bounding boxes) to assess the attack risk level.
RQ2: How does neural network architecture affect the ASR of NMS latency attacks?
RQ3: How does the size of the model affect the ASR of NMS latency attacks?
RQ4: When deployed, the models are compiled to a target format (e.g., QNN, ONNX.
TensorRT). Does the model format affect the ASR of NMS latency attacks?
RQS: Models deployed on device are quantized to optimize their performance (e.g., memory and latency). How does quantization affect the effectiveness of NMS latency attacks?
RQ6: How does "Domain Shift" affect the ASR of universal NMS latency attacks?
RQ7: How applicable are existing defenses (c.g., adversarial training, compression, purifi- cation) against NMS latency attacks?
To answer these research questions and assess the practicality of NMS latency attacks, we designed framework to Evaluate and Validate Adversarial Digital Examples (EVADE) see Figure Then, we devised experiments (see Section 4 that demonstrated the lack of effectiveness, transferability. and robustness of state-of sate-off-the-n NMS latency attacks. Especially, NMS latency attacks did not succeed when NMS runs on GPU (which is commonly the case), and NMS latency attacks were not transferable across different models (size, or version from the same family), nor able bypass existing defenses.
Our contributions are as follows:
The EVADE framework that supports evaluation of practicality of attacks. We applied EVADE to latency NMS attacks.
Generation 114 adversarial datasets to support reproducibility and comparison future research.
Comprehensive evaluation of hardware platforms, 4 latency attacks 15 models, and against defenses.
2
Related Work
2.1
Latency Attacks
There have been several papers on the topic of NMS latency attacks, which aim increase the processing time of the Non-MaxxSupppessinn maximize the number of bounding box proposals being processed by NMS, and second, minimize the number of proposals being filtered during NMS. As a result, the high number of box proposals increases the NMS processing time, and thus, results in a delay of the Object Detection pipeline.
(NMS) Lgoorithm by fulfilling two objectives. First,
More details NMS be found in Appendix

---

Table 1: Our evaluation of NMS Attacks compared SOTA
Evaluation Criteria
D
0
PS
BPS
EVADE
Iranslerability of the atlack across different Hardware
4
2
7
Transferability of universal patch attack across datasets (Domain shift)
NA
NA
X
Transferability across different export formats of model
X
X
V
Transferability across different models (sizes and architectures)
X
3
10
Transferability from a non-quantized model to quantized models
X
X
Robustness against defenses
X
X
V
Looking at the prior art O there are two categories NMS latency attack: unique patch and universal patch.
Unique patch attacks aim craft a perturbation patch be added "on the top of" genuine image by optimizing, through iteration, the perturbation for this single genuine image. Daedalus (D) Wang al. (202T) and Overload (O) Chen al. (2024) state-of-the-art unique patch NMS latency attacks.
Universal patch attacks aim to craft perturbation patch to be added "on the top of" genuine image by optimizing, iteratively, the perturbation across multiples images (dataset). From an attacker perspective, the advantage of universal patch attacks is for multiple images. Phantom Sponges (PS) Shapira et al. (2023) and Beyond Phantom Sponges (BPS) Schoof et al. (2024) are oftte-of-the-att art universal patch NMS latency attacks.
single perturbation that works
2.2 Evaluation of Adversarial Attacks
The evaluation of digital adversarial attacks is a well-explored topic. For instance, several works have been released such Adversarial Robustness Toolbox (ART) Nicolae et al. (2018) Ro- bustBench Croce et al. (020). However, these works do not evaluate adversarial examples from feasibility perspective. Indeed, current evaluation frameworks are limited. For instance, it is well-known that an attack optimized on a non-quantized model may not transfer to its quantized counterpart Li et al. (2024); (Shrestha and Großmann (2024). Such aspect mandatory assess the feasibility of the attack because models running on device are quantized. Therefore, it is important know if the attacker can use surrogate cunntiuadzzed model to perform its attack or if the attacker needs to optimize the attack on the quantized model. A follow-up feasibility aspect to consider is the transferability of the attack on the same model but with different sizes. Concretely, is common practice to train model with different sizes (e.g., small, medium, large) Jocher et al. (2023). Therefore, one may wonder if attacks transfer successfully to any model from the same family. In Table we summarize the evaluation criteria used in the four NMS latency attack papers. In comparison, our framework (EVADE) offers a comprehensive evaluation of the practicality of the attacks.
Evaluation Framework: EVADE
To assess the feasibility and practicality of NMS latency attacks, designed an evaluation method- ology as shown in F First, we describe the components that will be tested (hardware, model family, model size, model format, model quantization, domain shift, defenses). Then, we describe the datasets. attacks, and defenses specifically used for our NMS latency attack testing, Finally, we explain the relevant metries.
3.1 Components
Hardware: The Hardware component aims to solely evaluate the processing time of the NMS component. Since NMS the component targeted by the attack, its processing time is supposed to increase greatly under attack compared to the genuine scenario (no attack). Knowing that the processing time of NMS is hardware dependent, our evaluation considers wide range of hardware to cover all kind of use cases such as edge device, desktop, and cloud server. For instance, the attack may not affect a model running on a cloud server because of its powerful hardware. However, an
Forr the sake of reproducible and fair evaluation caused by re-implementation errors, we discard work without publicly available code. However, those work are cited in Appendix C
3

---

Inputs (section 4.2)
EVADE
(section
Metrics (section 4.3)
Datasets
Quanization
Format
4 Attacks
114 Adversarial
Hardware
Size
& Results Jsection 5)
3 Dafenaes
15 Models
Datasets
Defenses
Family
Haroware
Domain Shift
Figure 2: Proposed evaluation framework (EVADE) and corresponding sections where each phase discussed. Block shows the inputs used perform the component-level assessment, Numbers represent what was used to assess NMS latency attacks. Block B lists the components evaluated.
edge device may be impacted by the attack because it may not have powerful hardware to run the model.
Model Family: This component aims to evaluate if an attack optimized for a specific version of the model family (e.g., YOLOv8) can affect different version of the model (e.g., YOLOv11). If an attack, optimized for given model version, works for all versions of YOLO, then the attacker does need generate attack on each model version.
This component aims to evaluate an attack optimized for specific size of the model (e.g., nano) can affect model with different size (e.g., large). If an attack, optimized for given model size, works for any size of the targeted model, then the attacker does not need to generate attack each model size.
Model Size:
This component aims to understand if an attack optimized for model under a specific framework (e.g.. PyTTorch) can affect model exported to different framework (e.g.. CoreML). For instance, is common to have the model specification, training, and evaluation being done under PyTorch TensorFlow. However, to deploy models iPhone, they must be exported to framework different than PoTch,, namely CoreML. an attack, optimized for a model under one framework, works for any framework, then the attacker does not need to worry about model export.
Model Format:
Model Quantization: This component aims to understand an attack optimized for a non-quantized model works against quantized version of this model. The motivation behind this evaluation is that all latency attacks have been developed for non-quantized model. However, models are quantized to reduce their size and improve their efficiency, particularly for deployment on resource-constrainet devices like mobile phones or edge devices. Moreover, quantization has demonstrated some negative and positive effect attack success rate Bernhard al. (2019). Thus, one must test attacks against quantized models to assess its practicality.
This component aims to assess the universality of universal latency attacks. As reminder, universal attacks aim to create unique patch that fools any images. For instance, an attacker generates universal patch optimized on an autonomous driving dataset and on an object detection model. We test this patch by applying on "unseen" images and check if the attack still works.
Domain Shift:
Defenses: This component aims to understand if latency attacks are robust against existing defenses. We recommend to focus on diversifying the categories of defenses for the evaluation instead of having multiple defenses from the same category. Indeed, there little to interest to evaluate two defenses that are fundamentally similar because if the attacks fails against one defense, then the attack will fail against the second defense.
3.2 Inputs
To evaluate the feasibility of NMS attacks, EVADE requires diversity of inputs such as genuine and adversarial datasets, models, NMS attacks, different hardware, and defenses. The following sections describe each input.
Datasets:
For all our experiments, we use COCO 2017 validation as our baseline dataset. COCO the standard large-scale dataset for object detection with 5K images. One main reason for choosing COCO 2017 validation as our baseline dataset is the heterogeneity of the images. In other words, the images' semantics are diverse and provide more entropy.

---

Adversarial Datasets: As done in the prior art Chen et al. (2024); Wang et al. (2021) Shumailov al. (202T); Schoof et al. (2024). the common practice evaluate latency attack generate an adversarial dataset. The process of creating an adversarial dataset is to generate a unique perturbation (optimized for target Al model) for each image, repeated N times (where N the requested number of iterations for optimizing the perturbation). In nutshell, each adversarial dataset contains adversarial images optimized using a given latency NMS attack and a given targeted model. this paper, we generated 1 114 adversarial datasets to test attack effectiveness and transferability.
Models: To match the experiments from the ofate-of-the-art on NMS latency attacks, we use the same family of model (YOLO). We test different versions (v3, v5, v8, v11, v12), sizes (nano, small, medium, large, extra-large). export formats, and quantization schemes.
Attacks: Our focus is to evaluate latency NMS attacks. Therefore, in this paper, we evaluate the four state-of-the-art NMS lateney attacks, namely, Daedalus (D), Overload (OL., Phantom Sponge (PS), and Beyond Phantom Sponge (BPS). We reused the settings defined in their respective papei
Defenses: To assess existing defenses against NMS latency attacks, we selected three categories of defenses: an active defense, a passive defense, and a natural defense. An active defense is an extra component added to the inference pipeline (e.g., dedicated defensive AI model). A passive defense a defense that does not tamper with the image being processed during inference. An example of passive defense can be filtering technique such limit the number of bounding boxes being processed by NMS Chen et al. (2024). Lastly, a natural defense uses data processing schemes within the inference pipeline that may affect the attack. For instance, data transformations such as image compression or image resizing would fall under natural defenses.
3.3 Metrics
Number of bounding boxes: When it comes to evaluating NMS latency attacks, the evident metric is the inference time. However, the inference time is influenced not only by the attack, but also by the hardware used to run the model (CPU and GPU), the type of model being used for the experiment, and quantization. Therefore, to independently evaluate each component discussed in Section 3.] (except hardware). we set the number of bounding boxes after NMS as our core metric. Indeed, we found out that the number of bounding boxes was reproducible when testing the same model on the same dataset but with different hardware. Also, the number of bounding boxes is highly correlated to the outcome of the attack.
Visualization of attack effectiveness: We wanted to provide a visual interpretation of the effect of the attack. For each table, we follow a four color scale: green, yellow, orange, and red, Green and red highlight the extreme cases such as when the attack failed or is able to attain the maximum NMS detection. Yellow and orange reflect intermediate cases such as an abnormal number of bounding boxes but the attack is not at its full potential. For most of the evaluations, we divided the maximum number of bounding boxes allowed by the model (300 boxes) into 4 ranges such green (0-75), yellow (76-125), orange (126-225), red (226-300).
Evaluation
In this section, for each component, first detail the experiment, and then discuss the results. We remind the reader that the objective of the evaluation is to assess the attack practicality. Understanding the reason behind the results is out of scope. In the following Tables, abbreviated the attack names when needed and as follows: G (Genuine), D (Daedalus), O (Overload), PS (Phantom Sponge), BPS (Beyond Phantom Sponge). Each cell with numerical value represents the round up average number of bounding boxes for targeted dataset.
4.1 Hardware (RQ1)
Details about each attack and their parameters can be found in Appendix
5

---

Iw__ppe. Twi_name
3000
SPA, IVVIDAS ALCO SXMM4 S0DB
GRID A100E3-10
2500
gou,
time NMSS LmS
cpu (msii1 XcontFO Gold 6938N CPU @ 200HE
Appdie
2000
qpi
cpu, ARM Conoe ATBAE v8 12 Core 22222nz
1500
36
500
0
5k
10k
15k
20k
25k
30k
n_boxes
Figure 3: Inference time of NMS (ms) for different hardware platform in function of the number of bounding boxes that went through the NMS process. Each data point uses 30k bounding boxes proposals before NMS.
Table 2: Theoretical evaluation of attack success if NMS outputs 25k (yolov5) or 8k (yolov8)
bounding box proposals.
Hardware
Attack Success (ms)
type GPU
name
use case
100
500
1000
2000
NVIDIA A A100-SXM4-80 GB
Server
GRID ATOOD-IOC
Desktop
NVIDIA Jetson AGX ORIN 64GB
Edge-device
/
CPU
Intel Xeon Gold 6338N
Desktop
1 /
v
Apple MI Max
Laptop
/
AMD EPYC
Server
1
ARM C78AE
Edge-device
4
Experiment: Our focus is to understand how the number of bounding box affects the inference time of NMS. To this end, we use the implementation of NMS from Ultralytics Jocher et al. (2023), which is derived from the official PyTorch NMS implementation Paszke et al. (2019).
3 the NMS inference time function of the number of bounding box proposals. We plot the results for the seven hardware configurations. We observe that the NMS attack is the most successful when the NMS computation is performed on CPU with NMS inference time reaching more than 3 seconds in one instance. Now, if we want to translate Figure feasibility the attack, then Table would the closest answer.
Results:
3 into an analysis of the
Answer to RQI: NMS latency attacks cannot achieve the latency increase needed affect applica- tions when the NMS processing runs GPU. The attack must generate more than 10k bounding box proposals to start seeing a latency greater than 500 ms on CPU. However, models can cap the maximum number of proposals to lower number, making the attack unsuccessful
4.2 Model Family Transferability (RQ2)
Experiment: We use COCO 2017 validation dataset, attacks, and model versions of YOLO (v3, v5n, v8n, vl In, v12n). We generate 20 adversarial datasets (4 attacks x 5 models). In total, we run 100 experiments (20 adversarial datasets models).
Results: Table 3 shows the effect of each latency attack on different model version from the same family (YOLO). Results show that the attacks do not transfer across model versions of YOLO. Each attack only works when the source model and the target model are the same. From an attacker perspective, means she must craft adversarial examples for each model version.
MMore details of bounding box generation algorithm can be found in Appendix D

---

Table 3: Evaluation of each attack's transferability across the YOLO model family. The adversarial dataset is a perturbed COCO 2017 validation. Each cell with a numerical value represents the round up average number of bounding boxes for targeted dataset. Colors include green (0-75), yellow (76-125), orange (126-225), red (226-300). G stands for Genuine. TLDR: NMS attacks do not transfer across different version of the same model.
Tangttd
Spingo
212
v5n
Table 4: Evaluation of each attack's 's transferability across different model (YOLOv8) sizes. TLDR: NMS attacks do not transfer across different size of the same model.
2017
Sponge
Phantom Sporge
5
300
5
152
214
8
192
7
6.
12
205
12
9
300
16
13
5
222
8
8
192
7
11
49
240
70
9
75
100
44
8
219
8
8
12
172
10
72
112
276
9
103
126
300
9
9
212
14
10
116
Answer to RQ2: NMS latency attacks not transfer between YOLO model versions. The attacker must know the target model. The question remains if this result applies to another model families.
4.3 Model Size Transferability (RQ3)
Experiment: We follow the same experiment as in Section 4.2 but with five sizes of YOLOv8 (nano, small, medium, large, XL).
Table shows that latency attacks do not transfer across different size of the model. Each attack only works when the source model and the target model are the same. We observe mild transferability of Daedalus and Overload from YOLOv8x to YOLOv8I, but insufficient to reach the critical latency threshold (i.e., cells are orange yellow). This means the attacker must know the target model size.
Results:
Answer to RQ3: NMS latency attacks do not transfer to other sizes of the same YOLO model.
4.4 Model Export Transferability (RQ4)
We use 5 datasets (one genuine and four adversarial), 3 export formats of the model YOLOv8n (onnx, coreml, and openvino). CoreML is the model format used for running AI model on Apple's AI processor. VPenVINO
Experiment:
VINO is the model format optimized for Intel's 's processor. ONNX an
open-source format. We generate four adversarial datasets (4 attacks x 1 model) based on COCO 2017 validation. total, we run an inference on the 4 adversarial datasets per model format.
Table shows that latency attacks transfer across export model formats. Each attack generated a PyTorch model format works equally well the same model exported ONNX, CoreML, or VINO format. From an attacker perspective, the attacker does not need to worry about the model format.
Results:
Answer to RQ4: Model format does not affect the attack success. The model conversion (i.c., optimized operation, compilation) does not provide intrinsic robustness against NMS latency attacks.
4.5 Model Quantization Transferability (RQ5)
Experiment: We use 5 datasets (one genuine and four adversarial), 4 models based on YOLOv8n (non-quantized PyTorch model, non-quantized Open VINO model, quantized GPU model (TensorRT, PTQ, INT8 activations), quantized CPU model VINO, PTQ, INT8). We generate four adversar- ial datasets (4 attacks x model) based on COCO 2017 validation. total, we run an inference on the 4 adversarial datasets per model.
7

---

Table 5: Evaluation of each attack's transferability when tested on quantized models. TLDR: Overload is the only attack that perfectly transfers when targeting quantized version of the source model.
Data type FPI6
Device type
0
PS
BPS
CPU (Open Vino)
6
233
300
224
152
GPU (TensorRT)
233
300
224
152
INTS (PTQ)
CPU (OpenVino)
230
300
216
143
GPU (TensorRT)
195
67
Table 8: Domain Shift evaluation of universal patch attacks (Phantom Sponge and Beyond Phantom Sponge). TLDR: Universal Patch attacks do not work under domain shift.
Dalaset (Target)
Phantom Sponge
Beyond Plantoon Sponge
Split
Genuine
COCO
BDD 100K
nulmages
COCO
BDD
100k
nulmages
val
Train
val
Drain
train
val
train
val
train
val
Train
COCO
val
224
7
5
5
152
6
BDD-100k
val
16
300
26
14
5
11
288
20
9
36
300
6
9
296
nulmages
val
9
300
4
300
Imin
9
300
4
300
From Table 5 we can say that latency attacks transfer to quantized models Therefore, an attacker create perturbations non-quantized models and expect similar ASR quantized models.
Results:
Answer to ROS: NMS latency attacks transfer from non-quantized quantized models (INT8 Post-Training Quantization).
Table 6: Evaluation of each attack's transferabil-Table 7: Performances of each attack against three ity across model formats. TLDR: NMS attacks different types defense. TLDR: Each attack transfer across different model formats.
becomes harmless when facing defense.
Role
Fomat Name
G
D
0
PS
BPS
Detense
G
D
0
PS
BPS
Source
pytorch
6
233
300
132
None
233
300
224
Target
6
233
300
224
152
PDM
5
5
5
5
Target
coreml
6
233
300
224
151
JPEG
6
8
55
6
Target
openvino
6
233
300
224
152
Maximum Detection (10)
6
4.6 Domain Shift Transferability (RQ6)
Experiment: we use two universal patch attacks (Phantom Sponge and Beyond Phantom Sponge).
For dataset, we use two automotive datasets BDD-100k and nulmages in addition to our core dataset COCO 2017. For each dataset, both training and validation splits are used the experiment, As for adversarial datasets, we have 72 of them for this experiment. We generated 12 adversarial patches (2 attacks x 3 datasets X splits) that are applied to a total genuine datasets (3 datasets x 2 splits).
Results: A seen in Table 8 both attacks do not transfer across dataset, and this, even within the same domain (e.g., BDD-100k nulmages). Therefore, if attacker wanted to create adversarial patch on an automotive dataset and apply this patch on an image with an automotive context, then the attack will not work.
Answer to RQ6: Universal patch latency attacks do not transfer to other datasets.
4.7 Defenses (RQ7)
Experiment: we use datasets (one genuine and 4 adversarial), model (YOLOv8n), and 3 defenses (PDM-Pure, JPEG, and maximum detection threshold). We the maximum detection threshold to 10 based on the maximum number of objects present in individual images of COCO 2017 dataset. All 4 adversarial datasets (4 attacks x I model) based COCO 2017 validation. In total, run 20 experiments (5 datasets x defensive cases and one genuine case).
Note that this paper did not aim providing comprehensive study of quantization techniques.

---

Table 9: EVADE: Overall evaluation results for lateney NMS attacks. XIV means that the attack failed succeeded when tested against this component. "N/A means "Not Applicable".
Attack
Hardware
Model
Domain Shift
Defenses
Score
Result
Family
Size
Formal
Quantization
D
X
X
X
N/A
X
1/6
harmless
O
X
X
X
N/A
X
2/6
harmless
PS
X
X
X
V
X
X
1/7
harmless
BPS
X
X
X
V
X
X
1/7
harmless
Results:Table shoows that all attacks were filtered by the three defenses. Since all attacks failed, there is no need to expand this evaluation with more defenses. A special mention goes to the attack Overload which shows some resistance against the JPEG defense.
Answer to RQ7: NMS latency attacks couldn't defeat the defenses. New (realistic) attacks needed.
4.8 Summary
As seen Table / all attacks went through 6 evaluations (7 for universal attacks) and were found harmless in real-world scenario. The best attack is Overload, which passed only 2 out of 6 evaluations. However, this analysis demonstrates the lack of practicality of the current NMS latency attacks.
Discussion
5.1 Is NMS Latency Attack Dead End?
Thanks to our evaluation framework EVADE, we uncovered many limitations of state-of-the-art latency NMS attacks. Actually, our results suggest that these attacks did not pose practical threat when tested in realistic settings. That being said, because NMS attacks create unwanted bounding boxes, would interesting analyze their impact on the processes after object detection such as Object Tracking.
Another idea for latency attacks target NMS-free object detection model (e.g., DETR and YOLOv10), Obviously, existing frameworks for generating latency NMS attack do not apply here because they require access to the number of proposals before NMS (which not available in NMS-free models), One would have to devise a new attack and test it using EVADE.
5.2 Enhancing existing NMS
Clarification on low transferability While low transferability may limit the generalizability of latency-based attacks, does not imply that such attacks are ineffective. As demonstrated in Tables 3 and 4, the attack is successful when targeting the same model used to generate the adversarial input, consistent with white-box assumptions. Importantly, latency attacks remain credible threat in scenarios where the target model is known or can be reasonably guessed.
Boosting attack transferability Looking at the attacks' transferability in Sections 4.2 and 4.3 we can conclude existing NMS latency attacks do not transfer because they are white-box attacks. There- fore, the attacks work only against the targeted model. However, fill this gap, an interesting future work could be to add techniques for transferability boosting (e.g., Quantization Aware Attack Yang al. (2024) and Momentum- based iterative Dong et al. (2018)) the attack generation pipeline of existing NMS latency attack and see if the transterability improves.
Generating high-confidence fake boxes Most fake bounding boxes currently have lower confi- dence scores than genuine ones. As result, defenses that limit the number of displayed boxes tend to retain genuine boxes, since they have higher confidence. an attack could generate fake boxes with higher confidence than the genuine ones, the defense might mistakenly discard the genuine boxes.

---

Study on quantization A comprehensive evaluation of diverse quantization techniques-including varying bit-widths, post-training quantization, quantization-ane training, and mixed-precision strategies aeraaiiess-eemain latency attack robustness.
remains an important direction for future work to better understand their impact on NMS
5.3 Expand EVADE
We envision a couple of interesting additions to EVADE. First, we would like to add new metrics, such as image similarity score (e.g.. SSIM, LPIPS) to evaluate how close are the adversarial images from the genuine ones. This would support assessing the perceptibility of the attack.
Second, EVADE could be applied other vision models such audio speech recognition natural language processing. For instance, it has been shown that attacks targeting LLM are hardly transferable across models Lin et al. 2025 Liu et al. (2024) do not work quantized version of the model Dong et al. (2025) Yi cl al. (2025).
Conclusion
In recent academic papers, computer vision models have been shown be vulnerable to latency attacks. Notably, object detection models were dramatically slowed down by adversarially-crafee perturbations that exploit the Non-Max Suppression algorithm. However, in this paper, we put the state-of-the-art NMS latency attacks to the test. We focused on investigating their practicality. We designed an evaluation framework (EVADE) to test key deployment components. We uncovered that NMS latency attacks suffer from set of weaknesses, some specific to latency attacks, some valid for all digital adversarial examples. demonstrated this paper, without addressing the identified weaknesses, current NMS latency attacks targeting object detection have very limited practical impact, and hence, just fantasy.
10

---

References
Rémi Bernhard, Pierre-Alain Moellic, and Jean-Ma Max Dutertre. Impact of low-bitwidth quantization on the adversarial robustness for embedded neural networks. In 2019 International Conference on Cyberworlds (CW), pages 308-315. IEEE, 2019.
Erh-Chung Chen, Pin-Yu Yu Chen, Chung, Che-Rung Lee, al. Overload: Latency attacks on object detection for edge devices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24716 241211,2475, 2024.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang. Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint
2020.
Peiran Dong, Haowei Li, and Song Guo. Durable quantization conditioned misalignment attack on large language models. In The Thirleenth International Conference Leurning Representations, 2025.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9185-9193, 2018.
Muhammad Hussain. Yolov5, yolov8 and yolov10, 10: The go-to detectors for real-time vision, 2024.
Glenn Jocher, Jing Qiu, and Ayush Chaurasia. Ultralytics YOLO, 2023.
Qun Li, Yuan Meng, Chen Tang, Jiacheng Jiang, and Zhi Wang. Investigating the impact of quantization on adversarial robustness. arXiv preprint arXiv: Xix::2444400033) 2024.
Runqi Lin, Bo Han, Fengwang Li, and Tongling Liu. Understanding and enhancing the transferability of jailbreaking attacks. arXiv preprint arXiv:2502.03055. 2025.
Shih-Chieh Lin, Yunqi Zhang, Chang Hong Hsu, Matt Skach, Md E Haque, Lingjia Tang, and Jason Mars. The architectural implications of autonomous driving: Constraints and acceleration. In Proceedings of the twenty-third international conference on architectural support for programming languages and operating systems, pages 751-766, 2018.
Hanqing Liu, Lifeng Zhou, and Huanqian Yan. Boosting jailbreak transferability for large language models. arXiv preprint arXiv:2410. 15645, 2024.
Raymond Muller, Ruoyu Song, Chenyi Wang. Yuxia Zhan, Jean- -hin-ppil Monteuuis, Yanmao Man, Ming Li, Ryan Gerdes, and Z Berkay Celik. Investigating physical latency attacks against camera- based perception. In 2025 IEEE Symposium Security and Privacy (SP), pages 4202-4219, IEEE Computer Society, 2025.
Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, lan Molloy, and Ben Edwards. Adversarial robustness toolbox v1.2.0. CoRR, 1807.01069, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- perfrrrrrome deep learning library, 2019.
Coen Schoof. Stefanos Koffas, Mauro Conti, and Stjepan Picek. Beyond phantomsponges: Enhancing sponge attack on object detection models. In Proceedings of the 2024 ACM Workshop on Wireless Security and Machine Learning, pages 14-19, 2024.
Avishag Shapira, Alon Zolfi, Luca Demetrio, Battista Biggio, and Asaf Shabtai. Phantom sponges: Exploiting non-maximum suppression to attack deep object detectors. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 457 1-5800 2023.
11

---

Abhishek Shrestha and Jürgen Großmann. Properties that allow or prohibit transferability of ad- versarial attacks among quantized networks. Proceedings of the Sth ACM/IEEE International Conference on Automation of Software Test (AST 2024), pages 99-109, 2024.
Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. Sponge examples: Energy-latency attacks on neural networks. 2021 IEEE European symposium security and privacy (EuroS&P), pages 212-231. IEEE, 2021.
Manojna Sistla, Yu Wen, Aamir Bader Shah, Chenpei Huang, Lening Wang, Xuqing Wu, Jiefu Chen, Miao Pan, and Xin Fu. Bit-flip induced latency attacks in object detection. In 2025 IEEE/CVF Winler Conference on Applications of Computer Vision (WACV), pages 6709-6718 18. IEEE, 2025.
Ajantha Vijayakumar and Subramaniyaswamy Vairavasundaram. Yolo-based object detection models: A review and its applications. Multimedia Tools and Applications, pages 1-40, 2024.
Chenyi Wang, Yanmao Man, Raymond Muller, Ming Li, Z. Berkay Celik, Jonathan Petit, and Ryan Gerdes. Physical ID-Transfer Attacks against Multi-Object Tracking via Adversarial Trajectory. In Annual Computer Security Applications Conference (ACSAC), pages 1-15, 2024.
Derui Wang. Chaoran Li, Sheng Wen, Qing-Long Han, Surya Nepal, Xiangyu Zhang, and Yang Xiang. Daedalus: Breaking nonmaximum suppression in object detection via adversarial examples. IEEE Transactions Cybernetics, 52(8))7427-7440, 2021.
Yong Xiao, Jin Ma, Ping Yi, and Xiuzhen Chen. Sponge backdoor attack: Increasing the latency of object detection exploiting non-maximum suppression. In 2024 International Joint Conference on Neural Networks (IJCNN, pages 1-8. IEEE, 2024.
Yulong Yang, Chenhao Lin, Qian Li, Zhengyu Zhao, Haoran Fan, Dawei Zhou, Nannan Wang. Tongliang Liu, and Chao Shen. Quantization aware attack: Enhancing transferable adversarial attacks by model quantization, 2024.
Sibo Yi, Tianshuo Cong, Xinlei He, Qi Li, and Jiaxing Song. Behind the tip efficiency: Uncovering the submerged threats of jailbreak attacks in small language models, 2025.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction describe the contributions and main results of this paper. They also provide context and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section S provides suggestion on how to expand the evaluation framework such as new metries or looking at new application domains such as Natural Language Processing (NLP) and Audio Speech Recognition (ASR).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [NA]
Justification:
The paper does not include theoretical results. The work based on empirical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Section and the Appendix describe in detail the dataset, models, attacks, and metries used to reproduce the experiments.
Justification: Section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
The dataset, attacks, and models are already publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ priips.cc/pbbliccguu.eee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 3.1 Section 4 and the Appendix describe in detail the dataset, models, attacks, and metrics used reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments. The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
When relevant, Figures include error bars. We explain the variability in the
text.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Section 3.1 and Section describe the specification of the hardware used for
each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://eerrrrss.cc/bbbbbbbhhb
Answer: [Yes]
Justification: We read the Code of Ethics. We evaluated adversarial attacks that were publicly released and we found out they were harmless.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We study renowned adversarial attacks that are deemed harmless. Those attacks do not have any societal impacts a socictal impact. It would have been the case if the attacks had proven real life societal impacts (which not the case).
to begin with. Therefore, our study does have
Guidelines:
The answer NA means that there is no societal impact of the work performed.
16

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We are evaluating publicly available attacks. Therefore, this question is NA.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
Yes, the models, code, attacks, and datasets are all cited and used according to their respective license and terms of use.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
17

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper does not involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: ftttp:://neuris... /atps////coooeeeeeeee////////// for what should or should be described.
19

---

Appendix
A Non-Maximum Suppression (NMS)
Mon-Maximum Suppression (NMS) is a post-processing step commonly used in object detection pipelines prune redundant bounding boxes. During inference, detection model often outputs multiple overlapping bounding boxes for the same object-each with its own location, size, and con- fidence score. NMS serves to suppress (i.e., remove) boxes whose overlap with higher-confidence box is too large (above a predefined loU threshold), thereby retaining only the most representative candidates.
A.1 Background on NMS
Mon-Maximum Suppression (NMS) is a post-processing step commonly used in object detection pipelines to prune redundant bounding boxes. During inference, detection model often outputs multiple overlapping candidate bounding boxes for the same object- each with its own location, size, and confidence score. NMS takes all candidate bounding boxes and filters out the low-confidence ones, and boxes that have large IoU with higher-confidence boxes. NMS outputs the most representative candidate bounding boxes (also called proposals).
We would like bring attention to important parameters: the maximum number of bounding box proposals (set by the model), and the maximum detection (set by NMS). In the YOLO model family, the former parameter has been reduced with every version. fact, YOLOv5 caps the number of proposals to 25,000, while YOLOv8 caps it to 8, 8,000. Py PyToochh NMS implementation sets the maximum number of bounding box proposals to 30,000. As we will discuss in Section 4 this has huge impact on the attack success. The maximum detection set by NMS defines the maximum number of proposals to output. This means that even if the attack forces model to feed more bounding box proposals than what NMS accepts, it won't 't necessarily affect the latency.
In the standard implementation of NMS (Table TO). the maximum detection is to 300. This parameter directly limits the effect of the fake bounding boxes on the downstream tasks (e.g.. multi- object tracking Wang et al. (2024). However, the four state-of-the-ar NMS latency attacks (described in Section 2 consider a maximum detection of 30,000 which is not realistic in most use cases (c.g.. autonomous driving, surveillance, or traffic monitoring).
A.2 Parameters
Table T0 we describe the parameters of NMS used the hardware evaluation.
Parameter
Description
Value
conf thres
The confidence threshold helow which boxes will be filtered out The loU threshold helow which boxes will be filtered out during NMS
0.25
iou iou_thres
0.45
classes
A list of class indices to consider. If None, all classes will be considered True, the model is agnostic the number of classes.
None
agnostic
False
And all classes will be considered as one.
multi label
If True, each box may have multiple labels. The list contains the apriori labels for given image. The maximum number of boxes to keep after NMS The number of classes output by the model.
False
labels
O
maximum detection ne
300
0
Any indices after this will be considered masks. (seconds) for processing one image.
max time img max nms max wh in place rotated end2end
The maximum
0.05
max_
The maximum number of boxes into auchvisionnops.nmss. The maximum box width and height in pixels.
30000
7680
If True, the input prediction tensor will be modified in place.
True
Oriented Bounding Boxes (OBB) are being passed for NMS.
False
If the model doesn't 't require NMS.
False
Table 10: Parameters of Moximmx Suppression by Ultralytics

---

A.3 NMS processing time
Figure 10 depicts the line of code (328) of the function named non, san__ma___se wtt_maaa_pppppp swxxxssppression located in ultralytics. utils. orciis..pspp.... py from the library Ultrayticc containing the inference time for the NMS processing. For our hardware evaluation, we captured (time. time () -t) into a vari- able name time_ nms which was returned by non, non_man __x_________ _aa_max___pppp suppression in addition to the output of non moo__mx_______m _expxssupeessoon For those who wish to reproduce our approach, we use the version 8.3.34
of Ultralytics.
327
output [xi] [xi] x[i]
328 if (time. time() t) time_ limit:
329
worningwwrning (f"WARIING A NMS time limit {time_ limit: 3f)s exceeded")
330
break # time limit exceeded
331
332
return output
Figure Location of inference time for NMS within the Ultralytics library
B
Commands used for evaluation
All commands are based the documentation from the Ultralytics library.
B.1 Format
to export our model to a different format, we use the mode export available with the model of our choice. Then, specify the model format (e.g., ONNX) required for the export such as model.export (format="onnx")
B.2 Quantization
To quantize our model, we need to export model in a specific format (like in the previous section). Then, we need to enable the int8 quantization during the model export. For instance, if we want a quantized int8 version of OpenVINO. then we just need to run the following command such as model.e export "eoooaa=""ppeviiii
int8=True, nms=True). The mms argument includes the NMS post- proteprocessiig in addition to the quantized model during the export process.
C
NMS Latency Attacks
C.1 Prior Art
Previous work on latency attacks exploits the observation that generating numerous bounding boxes resistant NMS elimination. There an extensive literature on NMS Latency Attacks targeting Object Detection such as backdoor attack Xiao et al. (2024), bit-flip attack (Sistla et al. (2025). and adversarial inputs attack. However, the first two categories rely on strong assumptions. For backdoor attacks, those assumptions include having access to the training dataset (which is not publicly disclosed) or being able to inject sufficient malicious data into the training dataset (which implies the attacker knows which data will be picked to train the model). For bit-flip attack, the attacker somehow has access to the weights of the model used in production and can tamper with the weights. In general, model weights are not casily accessible because they are private and there are plenty security mechanisms to monitor any change made to the model such as cryptographic signature, encryption for data obfuscation, access control for people accessing the model weights. With such privileges, an attacker can have a better impact by using a malware attack or by leaking the model weights to the public on more realistic assumptions such as tampering inputs at inference by probing a model a product or by using surrogate model that may be the same as the one in production.
competitors. Adversarial inputs attack, the other hand, is based
For adversarial inputs attack, There are five attacks which focus on Camera Object Detection: Dacdalus, Overload, Phantom Sponge, Beyond Phantom Sponge, and DetStorm.

---

Daedalus (D) Wang et al. 2021) pioneered latency attacks, using adversarial examples against NMS achieve image-wide perturbation.
Overload (O) Chen et al. (2024) improved on DS by proposing a spatial attention mechanism, that prioritizes bounding box creation in less occupied areas to reduce overlap.
Phantom Sponge (PS) Shapira et al. (003)) advances latency attacks using universal adversarial
perturbations (UAP) to apply single pattern across multiple images, removing generation time.
Beyond Phantom Sponges (BPS) Schoof et al. (2024) is a spin-off of the original Phantom Sponge Attack. BPS replaces the bounding box area loss from PS with new loss (named loU loss). This new loss function decreases the IoU of given bounding box and every other bounding boxes instead of just decreasing the bounding box area.
DetStorm (DS) Muller et al. (2025) is another spin-off of the original Phantom Sponge Attack. DS modifies the original PS attack to make it work as a physical adversarial example (real-world). Since DetStorm is based on PS, it has the same flaws as Phantom Sponge.
C.2 Evaluation Settings
Each latency attack relies on a set of parameters and values that are common to all attacks but also specific to an attack (see Table TTp. Here is list of all the parameters with their description.
maximum iterations is the maximum number of iterations allowed to optimize the adversarial noise to be added to the image.
initial constant (ca) is the value of the constant (c) at initialization when generating the Daedalus attack. The constant (c) helps to balance the distortion and the adversarial loss function related to the attack.
binary search steps is the maximum number of search steps used during the binary search to find the best value for the constant c.
confidence (r) the minimal threshold value for the confidence of adversarial bounding boxes.
grid size the size the grids used to divide the image for the spatial attention mechanism Chen cet al. (2024).
epsilon is the maximum size of the perturbation.
lambda (A1) is a weighting factor affecting the maximum object loss and the maximum loU loss Shapira et al. (2023); Schoof al. (2024).
lambda (A2) a weighting factor affecting the bounding box area loss Shapira et al. 2023
Schoof
al. 2024).
Parameter
D
0
PS
BPS
max. iterations
1000
initial constant (op)
2
NA
NA
NA
binary search steps
NA
NA
NA
confidence
0.3
NA
NA
NA
grid size
NA
10x10
NA
NA
NA
15
70
A1
NA
NA
I
NA
NA
Table 11: Attacks' parameters and values used in our paper. "N/A" means "Not Applicable".
D Generating box proposals for NMS
For the hardware evaluation, we use Algorithm T to generate non-overlapping bounding boxes for given image size (W H). The motivation to create such a script because existing NMS attacks do not always For instance, if we use 640 640 images, then we will generate 409600 boxes which way above the maximum number of bounding box proposals that can be handled by Ultralytics NMS. Therefore, we make sure to cap total number of proposals with the parameter M. By

---

default, M is set to 30000 because it is the maximum number of proposals allowed to be processed by Ultralyties' NMS. Now, if we want to choose the number proposals being processed by the NMS function, then we are using the parameter N. For instance, if the NMS must process 5000 proposals, then N equals 5000. The remaining 25000 proposals will have their confidence value set. to and thus, they will be filtered out as part of the NMS pre-processing.
Algorithm 1 Generate non-overlapping box proposals
Require: W, H: List
I: B 0
I/ Set of bounding boxes proposals Max. proposals for Py PyTorhhss s NMS
2: M 30000
3:
// Width of the box If Height of the box Number of proposals // Identifier of the object class for this proposal
4: h =1
5: = 0
6: / = 1
7: while n M do
8:
for = 0 - H do
9:
for 0 + W do +
10:
x + 0.5 - 0.5
Box's 's center (x) coordinate
12:
// Box's 's center (y) coordinate
n < N then
14:
// Wanted number of proposals for NMS
c=
16:
else
17:
e=
18:
end if
19:
b = Ir, y. w. h, c,
Define proposal box I/ Append the 6 to B
20:
B +
21:
end for
22:
end for
23:
end while
24:
return B
E Additional Details regarding the evaluation
E.1
Hardware: measuring inference time
To measure the inference time (in milliseconds), we use the time computed in the Ultralytics NMS code (see Appendix A.3)). Because existing attacks couldn't generate the maximum bounding box proposals of 30,000 and need to evaluate the worst generate a tensor of non-overlapping 30,000 bounding box proposals, where 30,000 is the maximum number of proposals allowed by the NMS function in the Ultralytics library. Then, for each proposal, we set its confidence score and its class probability to 1. For the NMS function, we use the default value of each parameter (see Appendix A.2 for details).
scenario, created python script to
To measure the inference time of NMS for a specific number of bounding boxes (N), we set the confidence score to 1 for N proposals out of the 30,000 proposals. The remaining proposals (30,000 minus N) have their confidence score set to 0. Therefore, thanks to the confidence threshold, there will be N bounding boxes after NMS. In this study, the value of N ranges from to 30,000. Also, it is important to know that models may have a threshold in terms of number of proposals such as YOLOv5 (25,000) and YOLOv8 (8, (8,000). This factor may prevent latency attack to succeed due to potentially insufficient number of bounding boxes proposals passed to the NMS.
F.2 Hardware: determining the success of an attack
We define successful NMS attack as attack capable to increase the processing time of the NMS above defined threshold. In this paper, we set this threshold to 100 ms, 500 ms, 1000 ms, and 2000 ms. The value of each threshold serves an indicator of the performance of the attack for specific

---

use case. For instance. maximum latency above 100 milliseconds is unsafe in the autonomous driving domain Lin et 2018). In different context, a 100 ms delay will remain unnoticed by someone using an AI model for editing pictures on his phone or on his computer. However, a 2 seconds delay would be noticeable. Going back Table 2 we observe NMS attack will not work the NMS processing is performed by GPU instead of CPU. Therefore, the evaluation shows that NMS latency attacks are harmless when NMS runs on GPU, which is becoming best practice.
Another factor impacting the success of the attack is the maximum number of proposal output by the Al model. As discussed in Appendix A.I and shown Table 2 YOLOv8 caps the number of proposals to 8,000. Hence, the attack can only generate sub-500 ms latency, considering it successful only for applications with latency threshold of 100-400 ms. For YOLOv5, which has cap of 25,000 proposals, NMS attacks are capable to reach the 2 seconds threshold.
E.3 Metrics
E.3.1 Metric: definition of intervals for number of bounding boxes (colors)
The definition of each interval is unique to this work. Our approach does not rely on empirical distributions, previous work, or some application-speciffe-rriiii on arbitrary decision. To elaborate on the empirical distribution, this approach would have tied the definition of the intervals to the specific dataset used in our evaluation. Whereas, on the contrary, we wanted to have metric dataset- -agnostcc. In our opinion, it was important to highlight at least the following scenarios: the attack does not work (" I'grren'), ) the attack works ('red"), ), and the attack has some effect on the system while remaining unsuccessful ('yellow" and 'orange"), ). Using three intervals instead of four would a valid alternative. At the end, the intention of our metric is to provides pointers for future research who wish to understand why an attack partially failed or succeed. Using a binary classification ('red' and "green") would not provide such insights.
The definition purely based
E.3.2
Additional metric: number of bounding boxes before and after the attack
Another way to provide more insights the evaluation would be compute the boxes ratio under attack (brua) (Equation] [ to assess the strength of the attack. This metric computes the ratio between the number of bounding boxes for the genuine image and the number of bounding boxes for the adversarial image. For instance, a genuine image contains 300 objects (bounding boxes) such as candies in candy factory. Whereas, the adversarial version of this image contains 315 objects. Using our current evaluation, we may think the latency attack is working because there is a high number of bounding boxes. However, the high number of bounding boxes is due to the scene captured in the genuine image. Therefore, brua can be handy metric to assess the attack strength in particular for an image or dataset of images with large number of bounding boxes. Although, we believe brua will be better suited as a loss function to ensure the attack version of the image has more bounding boxes than the genuine version the image.
Niboz after_atttet attach
brua
N bboz be fore__ttact
E.4
Evaluation setup
Table 12 lists each setup used in our evaluation. As pointed out to us, in practice, latency can vary lot depending things like OS-level optimizations, driver versions, or background load. And, indeed, this study can be part of an extension for EVADE.
GPU / CPU
Operating System
CUDA Driver Version
NVIDIA A A100-SXM4-80 GB AMD EPYC
Ubuntu 20.04.6
CUDA 12.2
GRID A100D- 10C Intel Xeon
Ubuntu 20.04.6
CUDA 12.2
ORIGN / ARM Cortex
Ubuntu 20.04
CUDA 11.4
Apple MI Max (CPU only)
macOS Sequoia 15.2
N/A
Table 12: Hardware and software configurations used in experiments
5

---

Defenses
F.1
Prior Art on defenses
In this paper. PDM-Pur is our active defense. PDM-Pure is an off-the-shelf adversarial purifier based on a pixel diffusion model (PDM). PDM-Pure aims effectively eliminate adversarial patterns generated by latent diffusion models, thereby maintaining the integrity of images.
Our passive defense is Maximum detection threshold, defense advertised in the Overload paper/Chen Chen et al. (2024). However, the impact this defense on Overload has not been tested despite being already implemented parameter of the NMS component used by all YOLO models Jocher et al. (2023).
For our natural defense, we choose JPEG, popular image compression algorithm, used in many use cases. For instance, an image may need to compressed the image needs to transits through network with limited data bandwidth. This scenario can occur in embedded system such within an autonomous driving car or in collective perception where a system shares images to an external party. During those cases, compression is necessary.
F.2 Expanding the attack evaluation against defenses: model sizes and family
Table 13 extends the evaluation done Section 4.7 by evaluating the performance of the defenses against. attacks targeting models different than the original model (YOLOv8n) used in Table 7 These models different from the original model on two aspects which are the model size (YOLOv8x) and the model family (YOLOvI2n) Results-wise Table 13 is consistent with Table 7 because each defense decreases drastically the number bounding boxes. From an attack goal perspective, each defense prevents the attack reach its goal which is critical increase of inference time. However, preventing the attack's s goal does not mean the defense prevented all goals of the attack. For instance in Table 13 remains. While not impact the inference time, this high number bounding boxes still translate in poor detection performance because the model generates fake bounding boxes. This evaluation shows the importance of not choosing the defense from a single perspective (e.g., inference time).
with defense such JPEG, an abnormally high number of bounding boxes
Table 13: Model Comparison Under Various Defenses
Defense
Model
Genuine
DS
0
PS
BPS
None
v8n
6
233
300
224
152
v8x
7
276
300
212
116
v12n
7
284
300
195
218
PDM
v8n
5
5
5
5
5
v8x
5
5
v12n
4
4
4
5
4
JPEG
v8n
6
8
55
5
6
v8x
6
8
7
7
v12n
6
64
21
4
Max Det(10)
v8n
5
6
v8x
5
7
7
6
6
v12n
5
8
5


Reviews:
[{"review_id": "TGf523fuvV", "reviewer": "Reviewer_eEBD", "summary": "### Summary\nThe paper investigates the transferability and robustness of NMS latency attacks on object detection models. The authors evaluate the effectiveness of these attacks across different model formats, quantization levels , and versions of the same model. They also examine how defenses impact the success rate of these attacks.", "questions": "The main conclusion—that NMS latency attacks are generally ineffective—might be seen as discouraging for those looking for practical adversarial strategies. However, this is balanced by the identification of new research avenues. Somewhat, these methods provide insight on the robustness of NMS mechanism. Do you have more insightful suggestions?", "limitations": "1. While the paper suggests exploring NMS-free models, it does not delve into developing specific attacks for these models. Creating and evaluating new attack vectors tailored for NMS-free architectures would be a valuable extension.\n2. More suitable for the position paper track.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. They perform comprehensive experiments. The study uses an evaluation framework -  EVADE to systematically assess various aspects of NMS latency attacks, including transferability, quantization effects, and defense mechanisms.\n2. The experiments involve real-world datasets such as COCO 2017 and consider practical factors like model conversion and optimization.\n3. The paper provides valuable insights into the limitations of current NMS latency attacks and suggests directions for future research, particularly in targeting NMS-free models or analyzing post-object detection processes.\n\n### Weaknesses\n1. The paper does not provide a comprehensive study of quantization techniques, which could limit its applicability in some contexts.\n2. While the paper identifies the need for more effective attacks, it does not propose concrete solutions or improvements.", "was_revised": true, "final_justification": "The author's rebuttal solved my concerns; I raise my rating to acceptance.", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to provide constructive feedback on our submission. We appreciate your insights and would like to address the concerns and questions you raised in your review.\n\n## Question:\n\n> The main conclusion—that NMS latency attacks are generally ineffective—might be seen as discouraging for those looking for practical adversarial strategies. However, this is balanced by the identification of new research avenues. Somewhat, these methods provide insight on the robustness of NMS mechanism. Do you have more insightful suggestions?\n\n**TLDR**: \n- Our conclusion aims to provide **a different point of view and new research directions**\n- We have provided 2 insights to tackle existing limitations:\n  - **`The first insight` provides directions to boost latency attacks transferability**\n  - **`The second insight` provides a new attack goal for latency attacks**\n\n\n**Detailed answer**\n\nCertainly, we can provide some suggestions.\n\nThe major limitation of all the NMS attacks is that they are white-box attacks. As a result, these attacks are optimized to work for a single model, as demonstrated in our transferability results.\n\nOn the contrary, we see this paper as an opportunity to highlight the practical limitations encountered when deploying these attacks against object detection models in production. These limitations can serve as new research directions for developping more effective latency attacks. Such challenges have been largely overlooked in previous work. One possible reason is that prior research has focused on maximizing attack strength, often assuming no practical constraints on deployment. After several years and numerous papers on this topic,our goal with this paper is toraise awareness in the research community about these real-world limitations, which have not been previously addressed. \n\n\nThat said, there are several ways to tackle these challenges. We share 2 insights for future research.\n\n1. **Improving Attack Transferability**:\nOne solution could be to develop a black-box version of the NMS attack, such as an optimization-free attack [1]. his would avoid tailoring the attack to a specific model and instead focus on identifying adversarial patterns that are model-agnostic, thereby improving transferability. Another approach could involve combining existing NMS attacks with transferability-boosting techniques such as `Quantization Aware Attack (QAA)` [2] and `Momentum-based iterative (MI)` [3]\n\n2. **Generating High-Confidence Fake Boxes**:\nMost fake bounding boxes currently have lower confidence scores than genuine ones. As a result, defenses that limit the number of displayed boxes tend to retain genuine boxes, since they have higher confidence. If an attack could generate fake boxes with higher confidence than the genuine ones, the defense might mistakenly discard the genuine boxes.\n\nWe would be happy to include these insights in a discussion section.\n\n**References**\n[1] Liu, Hangcheng, et al. \"Optimization-Free Patch Attack on Stereo Depth Estimation.\" arXiv preprint arXiv:2506.17632 (2025).\n[2] Yang, Yulong, et al. \"Quantization aware attack: Enhancing transferable adversarial attacks by model quantization.\" IEEE Transactions on Information Forensics and Security, 2024.\n[3] Yinpeng Dong, et al. Boosting adversarial attacks with momentum. CVPR, 2018.\n\n## Weakness and limitations:\n\n> The paper does not provide a comprehensive study of quantization techniques, which could limit its applicability in some contexts.\n\nOur primary objective in including quantization was to highlight it as a realistic and practical factor that significantly impacts the effectiveness of current NMS latency attacks in real-world deployments. We showed that even basic quantization strategies can reduce attack effectiveness. We agree that a comprehensive analysis of various quantization methods (e.g., different bit-widths, post-training vs. quantization-aware training, mixed-precision) is a valuable direction for future research.\n\n> While the paper identifies the need for more effective attacks, it does not propose concrete solutions or improvements.\n\nThe main objective of this work is to introduce EVADE, a framework for evaluating the effectiveness of current latency attacks. The limitations we raise point to new challenges that future work can address. Additionally, we have provided insights into how these challenges might be tackled (see the reponse provided in the question section above)\n\n> While the paper suggests exploring NMS-free models, it does not delve into developing specific attacks for these models. Creating and evaluating new attack vectors tailored for NMS-free architectures would be a valuable extension.\n\nOur paper has two goals:\n- questions the effectiveness of existing latency attacks\n- proposes a new way to evaluate latency attacks\n\nDesigning a latency NMS-free attack would require a dedicated paper, as it would involve\n- introducing the design of the attack\n- presenting the evaluation methodology (EVADE)\n- Analyzing the attack's performance.\n\nDue to space constraints, we could not include such an attack in this paper. However, we do plan to pursue this direction in future work.\n\nAs highlighted in Section 5.2, valuable extensions to our work could be:\n - to expand EVADE to other types of attacks such as misclassification attacks and removal attacks\n - to expand EVADE to other types of modalities such as text and audio\n\n> More suitable for the position paper track\n\nWe were indeed debating between the evaluation track and the position paper track. Either option works for us!"}, {"review_id": "BJlP31EUan", "reviewer": "Reviewer_3QsT", "summary": "This paper challenges the practicability of the NMS attacks, which aims to increase the latency of object detector by increase the number of object proposals. To challenges the effectiveness of these existing attacks, this work proposed a new evaluation benchmark that studies the effectiveness of the existing attacks when (1) hardware changes, (2) architecture changes, (3) model size changes, (4) model format changes, (5) model is quantized, (6) input image has domain shift and (7) model has defense mechanism. The entire paper focus on the experimental discoveries. It concludes that the NMS attack remains effective when the model format and quantization changes and the NMS attack is no longer effective when the remaining factor changes.", "questions": "Please refer to the weakness section.", "limitations": "Yes, the author has addressed the limitations of this work and proposed serveral directions to expand the proposed dataset.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n•\tThe paper is well organized and easy to follow and read. \n•\tThe evaluation benchmark can be helpful for the larger community for testing whether the new developed NMS attack is generalizable. \n•\tThis paper seems to be the pioneering in studying the effectiveness of existing NMS attack.\n•\tThe experiments investigate the effectiveness of the attack in multiple factors, which are well designed.\n\nWeakness\n•\tThis paper is a discovery type of work, where only observation is presented but there is not further explanation about why the attack fails or how to make the attack more effective.  For example, why switching across different hardware will decrease the effectiveness of the attack. \n•\tThe only contribution is the evaluation benchmark itself (perhaps also the conclusion), but it remains debatable whether this is sufficient for Neurips acceptance\n•\tWill the author release the evaluation benchmark for reproducibility? For the “Hardware” component, how does other researcher in the community reproduce the result without having the hardware? This is important as the benchmark is the core and only contribution of this work.\n•\tIn section 3.3, the author categorizes the number of bbox into 4 different ranges. However, it is, by nature, that some of the images in the evaluation benchmark has more bounding box (i.e. objects) than other images, without being attacked. How does the metric account for this? Doesn’t it make more sense to produce a normalize score, where the normalized score is (# bbox after attack) / (# of bbox before attack) ?", "was_revised": true, "final_justification": "Thanks the author for providing the rebuttal. After reading the rebuttal, I remain the score as I still think the paper is interesting and this is one of the pioneering works to provide this study (from my best understanding). While there are many attacks proposed in the literature, whether these attacks are practical in the real-world scenario remains unknown, and this is where this paper contributes. \n\nYet, as mentioned by other reviewers and also the initial comment, this contribution of this work is the testbench and evaluation protocol itself. Again it remains questionable whether this is beyond Neurips acceptance. \n\nOverall, I still think this is a good work", "rebuttal": "Dear reviewer,\n\nThank you for taking the time to provide constructive feedback on our submission. We appreciate your insights and would like to address the concerns and questions you raised in your review.\n\n# Weakness\n\n> This paper is a discovery type of work, where only observation is presented but there is not further explanation about why the attack fails or how to make the attack more effective. For example, why switching across different hardware will decrease the effectiveness of the attack. \n\nOur primary objective with EVADE is to highlight the broader challenges and disconnects between theoretically strong attacks and their practical effectiveness in real-world systems. We are particularly interested in exploring why these challenges arise and how they can be addressed in future research.\n\n> Will the author release the evaluation benchmark for reproducibility? For the “Hardware” component, how does other researcher in the community reproduce the result without having the hardware? This is important as the benchmark is the core and only contribution of this work.\n\nYes, we plan to release the benchmark and dataset upon publication of the paper, and we will include a link in the final manuscript.\nRegarding hardware requirements, EVADE can be applied to any hardware researchers wish to evaluate. The hardware listed in our paper serves as a representative sample to demonstrate the challenges of current attacks.\n\n> In section 3.3, the author categorizes the number of bbox into 4 different ranges. However, it is, by nature, that some of the images in the evaluation benchmark has more bounding box (i.e. objects) than other images, without being attacked. How does the metric account for this? Doesn’t it make more sense to produce a normalize score, where the normalized score is (# bbox after attack) / (# of bbox before attack) ?\n\nThe attacks we evaluated aim to generate a high number of additional bounding boxes, far exceeding the natural count in any typical image. Therefore, our ranges were set to assess how many additional, fake bounding boxes the attack could force the system to process. \nHowever, we agree that a normalized score, such as $\\frac{N\\\\_bbox\\\\_after\\\\_attack}{N\\\\_bbox\\\\_before\\\\_attack}$, offers a valuable perspective, particularly for understanding the relative impact on individual images, especially those with a naturally higher baseline. Actually, we can also add a variable $\\epsilon$ to the denominator to cover the case where a genuine image may have no bounding box (to prevent a division by 0) such as $\\frac{N\\\\_bbox\\\\_after\\\\_attack}{N\\\\_bbox\\\\_before\\\\_attack\\ +\\ \\\\epsilon}$\nWe will clarify this observation and incorporate a discussion on normalized score to address this issue."}, {"review_id": "ikteEO7Hv2", "reviewer": "Reviewer_TrTJ", "summary": "The author proposes an evaluation framework named EVADE to assess the practicality of latency attacks, especially those targeting NMS in real-world scenarios, which is a novel direction. This framework comprises multiple components, providing a comprehensive combination of models, attacks, datasets, and more. It also employs different metrics to evaluate various aspects of the attacks. The extensive evaluation across different components reveals some interesting observations.", "questions": "1. While the authors refer to EVADE as a “framework,” I feel it is closer to a structured evaluation protocol or a testbed, as it lacks clear relationships between the components. In my view, a framework should be a more integrated and end-to-end system that not only provides a methodology for combining findings and reasoning about them but also offers an analytical structure to explain how different factors jointly influence the outcome.\n\n\n2. The bar set for declaring an attack practical seems overly strict. In real-world systems, even an increase from 10ms to 50ms in the NMS step could break timing guarantees, especially in time-critical applications. Since object detection is typically just one stage in a tightly constrained processing pipeline, even small delays can compromise the overall system functionality. Evaluating latency variance, jitter, and system-level behavior under load would provide a more realistic assessment of an attack’s effectiveness.\n\n\n3. On the platform side, the paper only lists the hardware type but doesn’t provide details about the actual deployment environment. In practice, latency can vary a lot depending on things like OS-level optimizations, driver versions, or background load. It would be helpful if the authors could clarify how sensitive their results are to these factors and whether the conclusions would still hold under different system configurations.\n\n\n4. While the authors seem to treat lack of transferability as a sign that the attack isn’t effective, that mixes up two different things. Just because an attack doesn’t transfer well doesn’t mean it’s not dangerous, especially when the model is known or guessable. \n\n\n5. What does “GRID A100D-10C” stand for in Table 2? Also, the Jetson AGX Orin has much stronger performance than typical edge devices, so a small latency increase on this platform may not reflect what would happen on more constrained hardware. It would be good to clarify whether the conclusions would still hold under truly resource-limited edge settings.\n\n\n6. In the caption of Table 3, the authors mention using color “green (0–75), yellow (76–125), orange (126–225), red (226–300)” to divide the average number of bounding boxes. What’s the rationale behind choosing these specific thresholds? Are they based on empirical distributions, prior work, or some application-specific criteria?\n\n\n7. While the results in Sections 4.2 and 4.3 show limited transferability across different model versions and sizes, it looks like the attacks are optimized separately for each model. It might be worth discussing whether known techniques for improving attack generalization could help with cross-model or cross-size robustness. Adding a short discussion on this could make the conclusions more broadly applicable.\n\n\n8. In Section 4.7, the authors evaluate the defense methods using only one model, which makes it unclear whether the results generalize to other settings. Also, the evaluation focuses only on reducing the number of NMS outputs, without considering possible side effects of the defenses, such as degradation in performance on benign inputs. If a defense severely hurts normal performance, wouldn't that also count as a kind of attack success, even if latency remains unchanged?\n\n\n9. In Section 4.8, the authors use a yes/no checklist over 6 or 7 criteria to conclude whether an attack is harmless. But this feels like an oversimplified evaluation. Just because an attack doesn't meet all the listed conditions doesn't mean it's completely ineffective or irrelevant. Some of the criteria don’t directly reflect actual impact, and the evaluation ignores cases where an attack could still be harmful in specific deployment settings. The overall reasoning seems too rigid to support such a broad conclusion.\n\n\n10. While the paper claims to evaluate the real-world feasibility of NMS latency attacks, it mainly focuses on adversarial inputs. Other attack types like backdoors or bit-flips are dismissed as unrealistic, but in practice, these aren’t that uncommon, especially with social engineering or insider access. Also, just because an attacker has access to the model doesn’t mean they wouldn’t prefer a stealthy latency-based attack. The threat model feels a bit too narrow.", "limitations": "The discussion section does not clearly address the work's limitations. It mostly repeats the conclusion that the attacks are harmless, without reflecting on the scope or assumptions of the study. The suggestions for future work are vague and lack concrete methods or supporting evidence. A clearer discussion of what the current analysis does not cover would improve the paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n1. This paper focuses on the real-world practicality of NMS-based latency attacks, which is a novel and valuable direction. It offers a new perspective for evaluating such attacks and lays a solid foundation for future research in this area.\n\n2. The proposed EVADE framework consists of multiple components and configurations, serving as a flexible and effective testbed for adversarial image-based latency attacks.\n\n3. The paper presents a well-structured set of research questions, which are carefully designed to cover diverse evaluation dimensions. This provides a clear and thoughtful basis for analyzing the effectiveness of latency attacks.\n\n4. The experimental evaluation is extensive and well-aligned with the research questions. Each component is evaluated through detailed experiments, which demonstrate substantial effort and contribute to a comprehensive analysis of the proposed framework.\n\n## Weaknesses:\n1. The author calls EVADE a framework, but it feels more like a collection of inputs and questions. There’s no clear internal structure connecting the parts, which makes it look more like a testbed than a true framework.\n\n2. The paper only focuses on adversarial image attacks at the input level, which limits the scope of the work.\n\n3. The paper sets a very high bar for what counts as a “successful latency attack,” and some of the main conclusions are stated quite definitively, even though the evidence doesn’t always seem strong enough to fully support them.\n\n4. The analysis mainly stays at the model level, without a broader system-level view, which is quite important when considering real-world practicality.  \n\n5. Most importantly, the authors didn't prove that NMS latency attacks are harmless. Instead, they show that some existing attack methods fail under specific experimental settings, and from that, draw a broad conclusion that the overall attack idea is ineffective. However, the failure of current attacks does not mean the general approach is invalid. Similarly, lack of transferability does not imply real systems are safe, and partial defense success does not prove the attack is harmless. Overall, the conclusion feels overstated compared to the evidence provided.", "was_revised": true, "final_justification": "I appreciate the authors’ detailed responses and their willingness to revise the language in the paper to avoid overly strong terms such as “harmless” or “fantasy.” It's encouraging to see the authors recognize the need to frame the discussion more carefully, especially given the complexity of evaluating attack effectiveness across different deployment scenarios. \n\nI have increased my score from 3 to 4 based on these clarifications and the authors’ commitment to revisions. Regarding Q2, I still feel that the paper would benefit from a clearer and more structured definition of what constitutes a “successful” attack under various system constraints. As suggested, including a table that explicitly outlines thresholds for latency increase under different deployment settings would make the paper more accessible and actionable for readers. I also support the idea of adding a brief discussion on system-level considerations, even if detailed evaluation is beyond the current scope. Overall, I appreciate the authors’ effort to engage constructively and improve the clarity and positioning of their work.", "rebuttal": "Dear reviewer,\n\nThank you for your time in providing constructive feedback on our submission. We appreciate your insights and would like to address your concerns and your questions.\n\n## Q1\n\nThe reason for considering EVADE as a \"framework\" was that **our methodology integrates the evaluation from multiple components**. As detailed in Section 3 and seen in Figure 2, EVADE comprises distinct, orthogonal components (Hardware, Model Family, Quantization, etc.). The results from evaluating each of these components are not presented in isolation. Instead, they are **systematically aggregated** into a final score and a conclusive, practical assessment, as demonstrated in our summary as shown in Table 9.\n\nAlso, the framework is **extensible** -- it can incorporate new metrics (e.g., image similarity scores), new evaluation modules (e.g., at the Operating System level), and new domains like Natural Language Processing (NLP) and Audio Speech Recognition (ASR). \n\nIn the revised manuscript, we will add a paragraph in Section 3 to explicitly define our use of the term \"framework,\" emphasizing both the integration of findings (as shown in Table 9) and the extensible nature of the components to better align with the reviewer's valuable perspective. **Alternatively, we are open to replacing \"framework\" with \"evaluation protocol\" if that better reflects the intended meaning**.\n\n## Q2\n\nGood point, it is true the attack is effective when latency increases from 10 ms to 50 ms. Our framework is designed to be flexible; a user with stricter latency requirements (e.g., 50ms) can use our findings to determine how many proposals are needed to exceed the targeted threshold for a successful attack and evaluate the attack's risk accordingly. For example, Table 2 reflects this observation. if the NMS inference time increases beyond 100 ms (or even 10 ms or 50 ms), the attack is effective. \n\nWe agree that evaluating the attack from a system-level perspective is important. However, most latency attack papers claim effectiveness without considering system-specific factors. To align with their evaluation scope, we have not included a system-level evaluation at this stage.\n\n## Q3\n\nWe appreciate the reviewer's insightful comment regarding the detailed characterization of the deployment environment and its potential impact on latency measurements. Below is the list of OS and driver versions used in our paper:\n\n\t- Setup #1 (NVIDIA A100-SXM4-80 GB (GPU) & AMD EPYC (CPU))\n\t\tUbuntu 20.04.6 (OS) \n\t\tCUDA version 12.2 (driver).\n\n\t- Setup #2 (GRID A100D-10C (GPU) & Intel Xeon (CPU)), \n\t\tUbuntu 20.04.6 (OS) \n\t\tCUDA version 12.2 (driver).\n\n\t- Setup #3 (ORIGN (GPU) and ARM Cortex (CPU))\n\t\tUbuntu 20.04 (OS)\n\t\tCUDA 11.4 (driver).\n\n\t- Setup #4 (Apple M1 Max CPU),\n\t\tSequoia 15.2 (OS).\n\nAlso, we acknowledge that factors such as OS-level optimizations, specific driver versions, and background system load can indeed influence the observed latency. However, delving into the exhaustive analysis of every possible permutation of operating systems and drivers is: \n- **beyond the scope of this paper**, \n- **not a mandatory requirement (according to prior art) for the success of an attack** \n- **beyond our available resources allocated to this initial study** \n\n\nTherefore, we plan to investigate those aspects in future iterations of the EVADE framework. \n\n## Q4\n\nYes, we agree with the reviewer's observation and we have the same observation as seen in Tables 3 and 4. The attack is ineffective (i.e., does not transfer) when targeting a model different from the one used to optimize the adversarial input. As expected from a white-box attack, the attack is only successful when targeting the same model.\n\nWe will clarify that while low transferability limits the efficacy and generalizability of latency attacks, it does not diminish the threat posed by latency attacks against known models.\n\n## Q5\n\n\"GRID A100D-10C\" refers to NVIDIA vGPU (Virtual GPU) A100D-10C. This configuration allocates a virtual A100 GPU instance with 10 GB of GPU memory and is commonly used in cloud-based deployments for efficient resource sharing. We will clarify this abbreviation in the updated version of Table 2 and the surrounding text.\n\nRegarding Jetson AGX Orin and resource-limited edge devices, we agree that the Jetson AGX Orin is a relatively high-performance edge device with growing adoption in more demanding edge AI applications. While we did not test on ultra-low-power devices, **our results on the Orin suggest that these specific attacks are generally impractical on such platforms.**\n\nFor more constrained devices like the Jetson NX, Table 3 of the Overload paper [1] shows that inference time can peak at 230 ms under attack. This implies that, based on our Table 2, the attack is effective if the application requires inference times below 230 ms, but harmless if the threshold is higher (e.g., 500, 1000, or 2000 ms). We will add a discussion to the manuscript to highlight this point.\n\n[1] Chen, Erh-Chung, et al. \"Overload: Latency attacks on object detection for edge devices.\" CVPR. 2024.\n\n## Q6\n\nFor now, the interval are purely informative. The key takeaway is:\n - the attack is effective if the attack is tagged as `red`\n - the attack is ineffective if the attack is tagged as `green`, `yellow`, or `orange` \n\n\nColors `yellow` and `orange` highlight images where the attack had some success in transferability. This information provides pointers for future research who wish to understand why. Such pointers would not exist if we had chosen a binary classification (`red` and `green`).\n\n\n## Q7\n\nYes all the attacks are white-box attacks and thus, they are optimized for a single model. \n\nWe agree with the reviewer and will add a discussion section highlighting the need to explore transferability boosting techniques such as QAA [2] and MI [3] to see if those attacks are more successful.\n\n[2] Yang, Yulong, et al. \"Quantization aware attack: Enhancing transferable adversarial attacks by model quantization.\" IEEE Transactions on Information Forensics and Security 19 (2024).\n[3] Yinpeng Dong, et al. Boosting adversarial attacks with momentum. CVPR, 2018\n\n## Q8\n\nWe tested various model sizes and versions and consistently observed the same outcome.\n\n## Table: Model Comparison Under Various Defenses\n\n|Defense|Model|Genuine|Daedalus|Overload|Phantom Sponge | Beyond Phantom Sponges |\n|-|-|-|--|-|-|-|\n|None|v8n|6|233|300|224|152|\n||v8x|7|276|300|212|116|\n||v12n|7|284|300|195|218|\n|PDM|v8n|5|5|5|5|5|\n||v8x|5|5|5|5|4|\n||v12n|4|4|4| 5|4|\n|JPEG|v8n|6|8|55|5|6|\n||v8x|6|8|9|7|7|\n||v12n|6|64|21|4|4|\n|Max_Det(10)|v8n|5|7|8|6|6|\n||v8x|5|7|7|6|6|\n||v12n|5|8|7|5|6|\n\n\nThese defenses consistently generalize across different model types and sizes, significantly reducing the number of bounding boxes.\n\nWe acknowledge that some defenses may degrade system performance. However, this is a limitation of the defense itself, not the attack. Our goal was to demonstrate that there are multiple ways to mitigate the attack’s objective—namely, increasing inference time.\n\nWe also want to point out that the strong reduction in mAP caused by poorly designed defenses has never been adequately addressed in previous latency attack papers. Notably, after our submission, a team published a defense specifically targeting latency attacks [4]. Their approach successfully removes adversarial bounding boxes while preserving mAP performance. This shows that effective defenses can mitigate latency attacks without compromising detection accuracy.\n\n[4] Wang, Tianyi, et al. \"Can't Slow Me Down: Learning Robust and Hardware-Adaptive Object Detectors against Latency Attacks for Edge Devices.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n\n## Q9\n\nWe agree that the attack could work under very specific conditions (e.g., non-quantized, no defense, the attacker knows the model). This makes the attack relatively niche, as it relies on strong assumptions about the system.\n\nOur goal with EVADE was to establish practical benchmarks based on realistic deployment scenarios. The \"harmless\" conclusion was drawn in the context of exposing the broader challenges and to disconnect between theoretically potent attacks and their practical impact in real-world systems.\n\nWe will clarify this motivation in final manuscript, emphasizing that the checklist serves as a pragmatic framework to assess the general practicality of attacks in the common scenarios we investigated, thereby encouraging future research to account for these real-world constraints from the outset. \n\n## Q10\n\nAbout the attacker model, we intended to include the latency backdoor attack and the latency bit-flip attack. However, the papers did not provide their code. As a matter of fact, we contacted the authors without success. We could reproduce their work but then people could challenge the fidelity of our reproduction of the attack. \n\nWe agree that this remains a promising direction for future exploration."}]

Author rebuttal:

Meta-review: All reviewers suggest accepting the paper, so will be accepted. Please address the remaining concerns in the camera ready version.
Decision: Accept (poster)