Overcoming Open-Set Approaches to Adversarial Defense
Authors: Edgar Wilfred Jatho
Armon Barton
Matthew Wright
Patrick McClure
Venue: Accepted by TMLR ()

Abstract: Machine learning (ML) models are increasingly proposed to replace or augment safety-critical information processing systems, yet their fragility to evasion attacks remains a well-documented, open problem. This work analyzes a class of deep neural network defenses that add a none-of-the-above (NOTA) class as an open-set-inspired, closed-set adversarial defense. We analyze seven prominent adversarial evasion attacks developed for computer vision classification and one attack developed for natural language processing classification, identifying how these attacks fail in the presence of a NOTA defense. We use this knowledge to adapt these attacks and provide empirical evidence that adding a NOTA class alone does not solve the core challenge of defending DNNs against evasion attacks. We release our adapted attack suite to enable more rigorous future evaluations of open-set-inspired defenses.

Published in Transactions on Machine Learning Research (12/2025)
Overcoming Open-Set Approaches to Adversarial Defense
Edgar W. Jatho III
@assma edu
Deportment of Computer Science United States Naval Academy
Armon C. Barton
armon. bww.oobbrrtot@vp....o edu
Department of Computer Science Naval Postgraduate School
Matthew Wright
matthew. matthew..righttäri edu
Deportment of Cybersecurity
Rochester Institute of Technology
Patrick McClure
patrick.
matrick.mchhure mechure @maps. edu
Deportment of Computer Science Noval Postgraduate School
Reviewed on OpenReview: https: // openreview. net/ forum? id= tuQ9r8VSIX
Abstract
Machine learning (ML) models are increasingly proposed to replace or augment safety- critical information processing systems, yet their fragility to evasion attacks remains a well-documented, open problem. This work analyzes a class of deep neural network defenses that add none-of-the-above (NOTA) class as an open-sel-inspirel, closed-sel adversarial defense. We analyze seven prominent adversarial evasion attacks developed for computer vision classification and one attack developed for natural language procossing classification, identifying how these attacks fail in the presence of a NOTA defense. We use this knowledge to adapt these attacks and provide empirical evidence that adding NOTA class alone does not solve the core challenge of defending DNNs against evasion attacks. We release our adapted attack suite to enable more rigorous future evaluations of open-sot-inspired defenses.
1 Introduction
Recent years have seen steep increase in the number of successful applications of Deep Neural Networks (DNNs) across the sciences, industry, and business. This technology has enabled strides forward in areas disparate as machine vision (Krizhevsky et al., 2012), neuroimaging analysis (McClure et al., 2019), astronomy (Valizadegan et al., 2022), cancer diagnosis (Savage, 2020), protein folding (Callaway, 2020), and natural language processing (NLP) (Brown et al., 2020). Despite these advances, efforts to leverage DNN technology in safety-critical systems have been hampered by the fact that current approaches create models that are highly susceptible deception, particularly dcception in the form of what are known as evasion attacks adversarial examples. This is well-documented,, open problem that persists to today (Carlini, 2024).
To address this, many defense methods have been proposed to increase the robustness of DNNs to adversarial examples (Costa et al., 2024). The most widely implemented lype of adversarial defense is adversarial training (Szegedy et al., 2013; Goodfellow et al., 2015; Madry et al., 2017). These defenses seek to create adversarial examples for particular DNN from clean training examples. These adversarial examples are then labeled with the same label as the clean training examples used generate them and are then used train that DNN. This is done with the goal of making the DNN's predictions robust to the addition of adversarial noise

---

Published in Transactions on Machine Learning Research (12/2025)
to the input. This robustness, however, may not defend against adversarial examples distant in input space from clean training examples (Costa et al., 2024).
One approach to addressing more distant adversarial attacks is to look to the open-set paradigm and create a cone-of-the-Abovy (NOTA) class, a new class in addition to the existing classes in a particular dataset. In contrast to adversarial training, generated adversarial examples act as boundaries, are put into the NOTA class, and are used as training examples. The premise is that with carefully crafted NOTA augmentation examples, one can continuously seed the data-point-sparee space between data-dense regions of classifier's input space throughout training and leverage the DNN to classify this vast space as NOTA. This approach is suggested as potential remedy to the "Inevitability" of adversarial examples under the standard DNN training paradigm (Shafahi et al., 2019). While covering all of the data-sparse regions of input space would be impractical, such approaches hold it might be sufficient to cover the regions that define the boundaries between and adjacent to classes and let the DNN generalize the remaining input space as NOTA.
Such defensive training strategies can be agnostic of the attack method the neural network architecture. Instead, they seek to change the structure of how the imput space is partitioned by the classifier through training with an additional NOTA class. The method is modular and can be added to any DNN classifier and may augment most present or future defense strategies. The entire approach has a fixed computational cost which only occurs during training, similar to adversarial training defenses. This makes it an attractive potential solution. Barton (2018) first demonstrated these open-set approaches to closed-set adversarial robustness and deep neural network defense with Boundary Padding (BP), a data augmentation approach which sought to insert a "padding class" between various classes in input space by linearly interpolating between two images and subsequently labeling the resulting example a padding class (i.e., NOTA). This defense was successful against several standard evasion attacks.
However, evaluating open-set or thone-of-the-Above adversarial defenses is held back by inadequate attack evaluations. A long line of work-from Athalye et al. (2018) through Tramer et al. (2020) to Suya et al. (2024) shows that adversarial defenses judged "robust" often collapse once the attacker adapts attacks to a given defense. Once a researcher has conceived and built a defense, it is then incumbent on them to "switch hats," and apply their full knowledge and effort to break the specific defense they are proposing, by altering attacks necessary (Carlini et al., 2019). Adapted attacks have been proposed for many defenses, but allacks adapted to defeat NOTA defenses have not been well-studied.
In this paper, we analyze seven prominent adversarial evasion attacks developed for computer vision classifi- cation and one attack developed for NLP classification, idontifying how these attacks fail in the presence of a NOTA defense. We use this knowledge to adapt these allacks and provide empirical evidence that adding a NOTA class alone does not solve the core challenge of defending DNNs against evasion attacks. We release our adapted attack suite to enable more rigorous future evaluations of open-set-inspired defenses.
2
Background
Before analyzing NOTA-adapted attack strategies, it is necessary to discuss deep neural network models, evasion attacks, adversarial training, and NOTA defenses.
2.1 Deep Neural Network Classifiers
Adversarial attacks are most commonly executed against DNN models. In general, a DNN classifier can be described as function f(): : . + R Where the input is Rd, and the output is in R and is often called the logits. In the image classification domain, input x is an h * * pixel image such that E [0, I and e is the number of classes. In the text classification domain, an input text is a sequence of n tokens, where each loken is an element of (0, embedding vector e; E R,, where is the token number and m is the dimensionality the embedding. The input, ær, to the classification function, f, is then sequence of n embedding vectors. Classes are denoted by integer codes ranging from 0 to c The prediction of the DNN for an input is given by the equation y argmax(((rr)
1}, where is the vocabulary size. Each loken is mapped to an
2

---

Published in Transactions on Machine Learning Research (12/2025)
2.2 Evasion Attacks
Evasion attacks imperceptibly modify the imput model to produce a change in classification from the clean (i.e. originally intended) class some other, untrue class (Chakraborty et al., 2018). This modified input is called an adversarial example. Adversarial examples crafted such that given classifier f, identifying distinct classes of objects yo, V1.--ee-1, and clean input x, belonging to the class Vtrue: input x' can be crafted such that (fgma((f(x") = Mpred, where Upred # Vtrue- This is accomplished by adding some perturbation 6g to x such that x + 6z = x', where 6g corresponds to only a small change in x (Goodfellow et al., 2015). This small change can be defined differently for different input modalities and methods (Madry et al., 2017; Guo al., 2021).
Our evasion atlack threat model assumes the adversary's attack occurs after training and system deployment, (i.e., the adversary cannot manipulate training data). The adversary is assumed to be able to do one of two things. One, they can change the actual artifact in the real world. Examples of this approach would be donning anti-facial-recognitiii glasses to defeat identification systems (Sharif et al., 2016), applying an A Al-camofflae pattern to a ship or tank to evade wide area motion imagery detection, or simply applying tape to precise positions on stop sign to fool a self-driving car's image classification system into missing the sign (Eykholt et al., 2018). Two, an adversary with insider access to the data stream can change the direct input to the model by; for instance, altering pixels by imperceptible amounts to fool classification system into misclassifying the pictured object (Goodfellow et al., 2015; Madry et al., 2017; Carlini & Wagner, 2017). In our threat model, the modifications to an input needed to change the predicted class of a model can be found using complete information of the classifier being attacked (i.e., a whitebox scenario).
2.3 Relevant Adversarial Defense Paradigms
Given the extensive literature that has accumulated regarding new methods to find effective, fast, and cheap evasion attacks, significant effort has likewise been expended pursuing effective measures to mitigate or climinate these threats. Here describe two DNN defense paradigms which, although near-complements of one another, have significantly different implications, strengths, and weaknesses. These include the well-known and ubiquitons adversarial training and the less well-exercised open-set approaches, like paradigms. The former's success at defending against existing state-of-the-art attacks is used as a baseline for evaluating the latter's robustness to original and adapted versions of the same attacks.
2.3.1 Adversarial Training Data Augmentation
The best known method for increasing DNN robustness remains adversarial training (Goodfellow al., 2015; Szegedy al., 2013). This is a data augmentation technique which adds adversarial examples to the true label class. The most common form, particularly for images, uses PGD, which, in many cases, leads to significantly improved adversarial robustness (Madry et al., 2017). Related gradient-based adversarial at tacks have likewise been successfully employed to defend against adversarial attacks on NLP tasks.
Attacking to Training (A2T) is an adversarial training regime designed to defend LLM-based tools from adversarial attacks (Yoo & Qi 2021). A2T introduces a gradient-based word-importance ranking derived from the gradient of the loss with respect to the embodding produced for cach token the input. It uses this basis to select the most impactful words swap out and replace in order to achieve successful attack (a change in classification). Once the attack is successful, the new example is added to the training dataset with label that matchos the original input's ground truth label. These examples are created on the fly during training as in other forms of adversarial training.
2.3.2 NOTA Defenses
In contrast to adversarial training, NOTA defenses generate adversarial examples or other data points in order to create training examples for the NOTA class. The premise is that adding carefully crafted NOTA augmentation examples training updates can continuously sow the data-poor regions of input space with NOTA examples. This is particularly useful for using NOTA examples to separate data-rich regions of input.
3

---

Published in Transactions on Machine Learning Research (12/2025)
space which correspond to different classes. The DNN then learns to classify this space along with the adversarial examples which are perturbed into it, as NOTA.
Shafahi et al. (2019) showed mathematically that if a class occupies less than or equal to 1/2 of the input space, then a random point from that class is, with high probability, either already misclassified or within small distance of an input region classified as another class. They suggest that having a classifier assign a large portion of imput space an "I don't know" class could circumvent the "inevitability" of adversarial examples. In particular, surrounding regions belonging to standard classes with regions corresponding to NOTA could make adversarial attacks return NOTA-classfieed examples. Shafahi et al. (2019) doesn't 't guarantee that every attack will result in a NOTA example, but it does offer theoretical rationale to support the idea that a NOTA buffer could absorb adversarial attacks.
Class
Class
(a) No Defense
(b) NOTA
Figure 1: A conceptual illustration of linear classifier for a binary classification task without (a) and with (b) NOTA defense.
2.3.2.1 NOTA, Open-Set, and Dutto--Distiibution Methods
Although NOTA defenses leverage the open-sct concept, their goal differs from most open-sct constructs in the literature. The "I don't i know" category is not generally used as defense, but rather as method to identify novel classes of data not already defined. Shao et al. (2020) characterize a research problem they call Open-Set Adversarial Defense (OSAD), where adversarial attacks are studied under open-set settings. their framing, the goal to both identify open-set samples (representing new classes) and defend against adversarial evasion attacks. They demonstrate that open-set. classifiers were readily fooled using existing closed-set attack methods.
A related method is aut-of-distributtan defenses using thresholds, but this has been shown to be ineffective against simple adapted attacks. Enevoldsen et al. (2025) demonstrate that open-set recognition models that use thresholds of maximum softmax probability or maximum logit score to identify novel classes are also easily deceived using simple adaptations to existing adversarial attacks. Also, Grosse et al. (2018) show the ease with which adversarial attacks can achieve high confidence and low uncertainty adversarial examples which are misclassified by MI models, but not detected by an out-of-distribution threshold approach. Additionally, they demonstrate that such examples successfully transfer wetw different Bayesian models and approaches. Thus, their research implies that confidence and uncertainty alone cannot be used as basis for defense against adversarial examples.
NOTA-type defenses are different in that they leverage the open-set concept to provide closed-set adversarial defense. NOTA defenses, therefore, do not facilitate or enable the identification of novel categories, nor do they use logit or uncortainty thresholds to identify adversarial examples. Rather, NOTA defenses leverage an additional non--of-the-above class to serve as the label for all adversarial examples, their derivatives, and open-set examples, relying the DNN to generalize and identify adversarial examples the NOTA class.
4

---

Published in Transactions on Machine Learning Research (12/2025)
2.3.2.2 Boundary Padding
BP was conceived after preceding rescarch showed that various methods of creating NOTA class examples- such as using linear interpolation in the input space, or mixing methods in latent lower dimensional space using auto-encoders showee promise at defending DNNs against adversarial examples produced by the CW attack suite using confidences of 20 or higher (Barton, 2018; Barton et al., 2021). However these methods struggled to perform against low-confidence Carlini Wagner Lp attacks.
Another influence for BP resulted from Zhang et al. (2020), they introduce Minup, algorithm for instantiating linear behavior between training examples and increasing regularization and resistance to adversarial attack. Although BP discards the label-mixing aspect, it uses the simple mixing expression, A- 21 + (1 A) 22 for the image data as well as randomization of À in a new way to create its NOTA class examples. BP then is an attempt to more closely surround correctly labeled regions of input space with NOTA class examples. However, instead of "mixing" two separate training examples, as well as their labels, and training as that label combination, as is done in mixup, BP "mixes" a single training set example 21 with an adversarial example, z1, derived using the PGD attack. The resulting BP image is labeled as the pure NOTA class and added to training on the fly. Note that no mixing of the labels occurs, all produced examples instead labeled as the NOTA class and trained as such. This can lead to NOTA becoming a "padding" class that separates data from other classes (Figure 1).
In BP two variations of NOTA are produced on the fly and added to the training batch before batch-trainin trainitraining commences, mean BP and uniform BP. In mean BP, set to 0.5 and limited gaussian noise added to the resulting image. In uniform BP, À is set a random number between 0.05 and 0.95 and weighted average of the clean and adversarial example is performed. The resulting NOTA to clean data ratio is two to one, making it by far the largest class in the dataset. This large representation of NOTA vs. any other class reflects the intuition that, regardless of the number of finite classes that are defined, the vast majority of possible inputs in the input space do not correspond to any of the specified classes and, thus, should instead be assigned to the NOTA class.
3 Methods
3.1
Novel NOTA Defenses
In this paper, we not only evaluate NOTA-adapted attacks against BP, but also against two novel NOTA defenses, one that builds on BP and one that extends NOTA defenses to LLM-based classifiers.
3.2 Adversarial NOTA Envelopment
Given BP's existing loss formulations, as NOTA regions are planted and reinforced over the course of training epochs, it is likely that new NOTA examples will be planted in nearly the same input space locations epoch after epoch. Each new NOTA example then results in steeper gradient to that portion of the input space, which in turn, will result in higher likelihood that future NOTA data augmentation will be created in close to the same place. This can have reinforcing effect creating essentially a funnel or entrapment zone. As a result, NOTA regions clump near but not surrounding a class's partition space. Instead, the desired behavior of a NOTA defense should be to surround or envelop homogenoussy-classed datapoint regions of input space with NOTA, while also populating data-sparse regions between them with NOTA.
Toward this end, the ANE defense retains all previous characteristics of BP, except it switches between two different losses in creating its NOTA data. One PGD loss maximizes the cross entropy (CE) with respect to the true label class as in BP. The second loss maximizes the cross entropy loss with respect to the NOTA class. This strategy is elegant in its simplicity, alternating between pushing away from the true label class to plant NOTA, and pushing away from existing NOTA partition to plant NOTA examples where they do not already exist. This leads to an attack loss of
Cxx, Vtruei f) = CE(ytrue softmax(f(z)))
when B 0.5 otherwise,
(1)
CE(yNOTA, [FlIyorrssottmx(((())) (oftmax(f(=)))
5

---

Published in Transactions on Machine Learning Research (12/2025)
where 3 ~ (0, 1). B is used to randomly choose between pushing adversarial example away from the true class and pushing the adversarial example away from the NOTA class.
3.3 NOTA A2T
To create NOTA examples for LLM-based models, we used modified version of A2T (Yoo & Qi, 2021). Specifically, we labeled successful adversarial examples created by A2T (sec Section 2.3.1) as the NOTA class, instead of the ground truth label. As in standard A2T, these examples are then used to augment. the original training dataset during training.
3.4 NOTA-Adapter Attacks
In this paper, we analyze and adapt seven computer vision evasion attacks, CW L2 and CW Los (Carlini & Wagner, 2017), AutoPGD with CE and AutoPGD with difference logits ratio (DLR) (Croce & Hein, 2020b), DeepFool (Moosavi-Dezfooli et al., 2016), Square Attack (Andriushchenko et al., 2020), and Auto Croce & Hein (2020b), and one evasion attack for LLs-eased classifiers, Gradient-based Distributional Attack (GBDA) (Guo et al., 2021).
3.4.1 Adversarial Attack Success for NOTA-Defended Models
When attacking NOTA-defended DNN classifier, f, with an input x', an attack is successful only the prediction neither the true class, Vrue: nor the NOTA class, ЧNOTA- In other words, argmax(f(z')) + Vtrue A argmax(f(x")) # μNOтa- For every attack, it is important to evaluate whether the stopping criteria needs to be modified to use this NOTA-aware definition of successful attack.
3.4.2 Projected Gradient Descent (PGD) and AutoPGD (APGD)
Projected gradient descent is a ttraight-forwart attack that leverages the same optimization that makes DNNs possible in the first place. In untargeted PGD, adversarial examples, x' are discovered through gradient ascent and backpropagation (Madry et al., 2017). We let f(x, yi f) represent any loss function whose minimun results in argmax(](a)) C maximally increases and the resulting input leads to argmax(f(x") # Vtrue.
Strue- Gradient ascent is then employed to iteratively update x' such that the loss,
After each gradient update, x' is projected to be within an Lp-bound, e, of æ and be in the set [0, 1, when dealing with images. The most common T---norms used in PGD and AutoPGD, and most other evasion attacks, are La and Lox. L2 x') is the magnitude of the adversarial noise and Loo (x, x²) is the maximum of the absolute adversarial noise.
For L2, x' is updated using
2/41 = 2f + a VoC(a' yi f))
(2)
V C('' yi f))ll2
For Loo, x' is updated using
2/41 = of - a sign Lz ys f)).
(3)
Here, the gradient vector VzL(a', . y: f) is the rate of change of the loss, L, with respoct to the input, æ, and a is the learning rate. This procedure produces a perturbation for x that pushes the DNN's prediction away from the true class, Strue- AutoPGD (Croce & Hein, 2020b) modifies the gradient-based update by using momentum and adaptive learning rate.
In AutoPGD, two losses are commonly used: (1) CE and (2) DLR.
This is equivalent to gradient decent the negative loss.

---

Published in Transactions on Machine Learning Research (12/2025)
3.4.2.1 Cross Entropy (CE)
When performing untargeted PGD and AutoPGD with CE loss, the probability of the true class, Vtrue: is minimized, as done in APGD-CE. The corresponding loss is
Lcn(e', Vtruei f) C softmax([(xz)))
(4)
When no NOTA defense present, making any non-true class have the highest, predicted probability by perturbing the input will lead to a successful adversarial example. However, this not the case when a NOTA class is present. Making NOTA the highest probability class does not result in succossful adversarial example. We investigated using a linear combination of the cross entropy loss for the true class and the NOTA class in order to create an example that is not classified as either the true or NOTA class. However, found that just maximizing the CE loss for the NOTA class led to a stronger attack, which we call Auti-NOTA (AN). This results in the AN loss being
LAN(E', UNOTAi f) softuaax
(5)
3.4.2.2 Difference Logits Ratio (DLR)
CE is shift-invariant loss, meaning the order of logits does not alter the output. Croce & Hein (2020b) introduces DLR, which is both shift and scale invariant. Scale invariance implies that rescaling the logits by non-zero constant will not change the loss value. This ensures that the learning process and by extension attacks seeking adversarial examples using gradient descent not be sensitive to or affected by the scale of the logits.
The DLR loss defined as:
Prru (x') - max fi(x') Vfrue
LDER Vtruei f)
(6)
fr, (x') (2')
where the elements of f(x') are the logits output by the model for x', fyrme (x') is the logit of the true class, and is the ordering of the components of f(x') in decreasing order. DLR has been reported to be sometimes better-performing than CE with respect to allack success and more stable than Mallni--agner loss (Croce & Hein, 2020b).
When attacking a NOTA-defonded model, DLR runs into a "target selection" " issue. That is, the loss can actively incentivize making the logit for the NOTA class the largest logit. This occurs when the NOTA logit is the socond-highest logit, after the true class logit. In the DLR loss, this issue arises when yeu fi INOTA (x'). Making NOTA the most likely predicted class can be problematic given the definition of attack success in the presence of NOTA. To address these issues, we modified the DLR. loss to
Spems (x') max fa(x' i i
ENOTA-DLRLZ' Vtruei f)
(7)
fm. (x²) fmy (x')
This adaptation seeks to prevent the loss from seeking to maximize the logit of the NOTA class.
3.4.3 Carlini-Wagner Attacks
Carlini- Marlini-Wagner (CW) attacks (Carlini & Wagner, 2017) are suite of attacks that use optimization methods find succossful adversarial examples that are created using the smallest possible perturbations, according some metric (1r,x'. The two most common CW attacks use Lg or Lee metries. CW attacks use the optimization loss of

---

Published in Transactions on Machine Learning Research (12/2025)
Low (x, x', Utrues f) = x') + de- low(n', Yerues f)
(8)
The CW constraint. term, lew, for untargeted attacks often takes the form of the margin loss
Low (a', Yrrue' f) max (fuerne (x') max f((e)) + K. K).
where i. is the "confidence" and controls how much the highest non-true logit, ma exceeds the logit for the true class, furru (x'). As in PGD, Carlini & Wagner (2017) add a box constraint that ensures x' E [0, 1] hot for images. The constant Xa is obtained through binary search and is used to increment or decrement the weight of the margin loss.
Like DLR, CW attacks suffer from a "target selection" issue when ma fila') fNOTA(x'). To address this, the margin loss was adapted to
ENOTA-CW(a', Btruei max (Syemee (x')
max
fi(x') + k. K.
(10)
Varue
This prevents the loss from incentivizing increasing the NOTA logit.
3.4.4 DeepFool
Algorithm 1: DocpFool: Multi-Class (Moosavi-Dezfooll et al., 2016)
input:
Image I, classificr f
output:
Perturbation 8;
3 (x)) = =rggax(f(x)))
Initialize 2/ + 2, 0;
while Uproa(ar()) yprrad (166) do for + Vonea(a')) do
7
wy + (fy(2() - V (oor (2/);
fg fy(x)) I Inrrea (x():
i argmin yoth Mpread (mr6) [Twr 11
9
15/1
10
6,
11
2/11
24 + 6z;
12
i
return: 8- = Σ1 8;
13
The DoepFool attack (Moosavi-Dezfooli et al., 2016) can be seen as a gradient descent approach that sets the learning rate using the estimated minimum La distance to a desired decision boundary. In the linear case, this method is optimal. However, that is not guaranteed for non-linear models. In practice, DeepFool often generates successful adversarial examples. The pscudocode is shown in Algorithm 1.
Two modifications are made to DeepFool to adapt it for NOTA, changing the stopping criteria to account. for NOTA and changing the target selection to not select NOTA. To adapt the stopping criteria, the logical statement Line of Algorithm
modified to Ypred (xς) = ('rea((xx) (x.) = INOTA- This means that the attack will continue until the model classifies the adversarial example a class that neither the initially predicted class nor the NOTA class, unless maximum number of iterations is used. To adapt the

---

Published in Transactions on Machine Learning Research (12/2025)
target selection, the logical statement in Line 6 of Algorithm 1 is changed to y & Uprea(z]) Ag # YNOTA- This prevents DeepFool from intentionally creating adversarial noise that would drive the input into NOTA region.
3.4.5 Square Attack
Square attack is a score-based black-box L2 and Tro adversarial attack that does not use local gradient information and thus is immune gradient masking (Andriushchenko et al., 2020). It uses randomized search scheme and perturbations are introduced such that they lie the boundary of the Leghhperspper or hys-hhye before x' is projected back inside the box constraint (i.e., x' E [0, 1)(wh) for images. First, a side-length for the square that will be perturbed is chosen, according to a decreasing schedule. Next, 5 is chosen. If, on applying the 6, the loss decreases, it is accepted. If not, it is rejected. If the new image classifies in a non-true class, the image is accepted. If not, the algorithm continues, repeating until it is either successful or the max number of iterations has been completed. Square attack seeks to minimize
CSA(2's Merue' f) Sperae (z') max f((x'
(11)
We adapted the attack by first preventing the NOTA class from being chosen as the initially correct label in the event the true labels were not provided. We altered the stopping criteria so that the attack was only successful if it resulted in a classification other than the true class and the NOTA class. We investigated modifying the loss to not incentivize increasing the NOTA logit, but, counterintuittvely, found that doing so slightly weakened the attack.
3.4.6 AutoAttack
AutoAttack (Croce & Hein, 2020b) is considered the state-of-the-ars adversarial evasion attack for computer vision. It is a highly effective ensemble of parameter-free attacks, combining the CE version of AutoPGD, the DLR version of AutoPGD, SquareAttack and the Fast Adaptive Boundary Attack (FAB) (Croce & Hein, 2020a). These separate allacks are used in sequence and until argıma((((x")) + ytrue- The adversarial robustness (ART) library (Nicolae et al., 2018), which we use in our computer vision evaluations, substitutes the DeepFool attack in for the FAB Attack. FAB is very similar to DeepFool, except that it uses additional iterative procedures to generate adversarial examples closer the decision boundary. However, weakness of untargeted FAB is its extensive computat ional cost as dataset complexity and the number of classes increase (Croce & Hein, 2020a). For ease of testing and standardization, the hyperparameters for each attack in AutoAttack are constant across models, datasets, and mcasurement norms. Given its extensive strength and effectiveness, untargeted AutoAttack is a standardized benchmark used by the adversarial robust.nsss community to compare model defense robustness to adversarial evasion attacks (Croce et al., 2020).
As AutoAttack leverages four separate subordinate attacks, it is necessary to ensure that the adapted NOTA version of AutoAttack calls the most effective adapted versions of each of these subordinate attacks. Additionally, the stopping criteria evaluated after cach subordinate attack must be adjusted to exclude the prediction of NOTA as a successful outcome for the attack. We test both a NOTA-aware version of AutoAttack with NOTA-Award APGD-CE and a NOTA-aware version of AutoAttack that replaces APGD-CE with APGD-AN.
3.4.7 Gradient- Cradient-Based Distributional Attack
GBDA is an adversarial evasion attack that targets transformers used in LLM-based text classification, seeking to produce fluent and semantically similar adversarial examples (Guo et al., 2021). A major challenge in this approach is overcoming the non-differentiable nature of the mapping from the input text token numbers to embeddings vectors. GBDA seeks to address this challenge by using the Gumbel softmax to create a differentiable method for linking token numbers with an embedding.
The GBDA loss function is a linear combination of three components: (1) a CW margin loss (Equation 9), (2) fluency constraint that preserves natural language flow, and (3) a semantic similarity constraint that.
9

---

Published in Transactions on Machine Learning Research (12/2025)
preserves meaning. This results in GBDA minimizing
LGBDA v. Pa- (ow(e(T), Veruei f)
Amm
NLLg (T) + Asimpa(
(12)
margin
fluency
semantic similarity
In GBDA's 's objective function, Pe is the learned adversarial Gumbel-softmax distribution for a particular input. is a sampled probability distribution from Pe over the vocabulary for each token in the input. e is function that computes the expected embodding vectors according to æ. (Z) is the perplexity (an analog of fluency) of the expected embeddings according to the language model 9, which in this case is GPT-2 (Radford et al., 2019). Py is the BERTScore (Zhang et al., 2019) (semantic similarity term) computed using g. Alm Asim > 0, are pyper-parameters that adjust fluency and semantic similarity with attack strength. At each gradient-
update step, the attack draws multiple soft samples, म, from Pe, calculates the loss LGBDA, and then calculates the gradients to update O. After @ has been learned, the attack samples text. inputs using Pe until successful adversarial text is generated or the maximum number of samples has been reached.
Two modifications were made to adapt GBDA for NOTA, modifying the margin loss target selection and changing the stopping criteria when sampling potential adversarial text examples. As in CW attacks, the NOTA class must. be eliminated from consideration ion when selecting a largel for the margin loss. This was accomplished by setting the margin loss to (NOTA-CW(E(T)) Mtrue: f), where (NOTA-CW defined in Equation 10. After learning the adversarial Gumbel softmax distribution for imput example, GBDA samples from this distribution until a successful adversarial example is found the maximum number of samples is reached. In the adapted attack, the stopping criterion for this procedure was modified so that successful adversarial example is defined to be one that is both not classified as the ground truth class and not classified as NOTA,
4 Experiments and Results
In this section, we discuss the evaluation of NOTA-adapted attacks against NOTA-defended models in both the compuler vision and natural language processing domains. In the computer vision domain, we compare the effectiveness of adapted versions of seven benchmark evasion attacks against NOTA- models to the effectiveness of the original attacks against NOTA-defended models, adversarial-trained models, and undefended models. In the NLP domain, we compare the effectiveness of NOTA-adapted version of a prominent NLPP classification attack, GBDA, to the effectiveness of the original GBDA attack against. both undefended and NOTA-defend LLM-based classifiers.
4.1
Experimental Setup
4.1.1
CIFAR-10 and CIFAR-100
We performed computer vision experiments using the CIFAR-10 and CIFAR-100 datasets. Our datasets are split before training such that 4% of the former training set are quarantined as validation set to enable early-stopping model selection, based on combination of best validation accuracy and best validation adversarial robustness. In very close models, we slightly favored best validation accuracy over validation adversarial robustness. The test set is strictly reserved for testing a specific model that has been previously selected using only ASR and accuracy performance from the validation set. In testing, accuracy was calculated from the full test set.
For CIFAR-10 and CIFAR-100, we used standard Wide Residual Networks (WRNs) with dropout and batch normalization, configured as suggested by Zagoruyko & Komodakis (2016). Dropout is set to 30% drop probability during training. For all models, common setup for these datascts. For cach model, we used an ADAM (adaptive moment estimation) optimizer with default settings, bn = 0.9 and b2 0.999 (Kingma & Ba, 2017) with sharpness-awar minimization (SAM) (Foret et al., 2021). ). ADAM assisted the model in efficiently converging on a solution, whereas SAM, by soeking out minima of the training loss landscape and minimizing loss curvature, smooths the boundaries between partitions expressed in input space. When performing mini-batch gradient descent training, two separate batches of 32 were drawn from the training set for each training cycle. The first used standard
used a WRN-12-6 (i.e. 12-unit deep and 6-unit wide WRN),
10

---

Published in Transactions on Machine Learning Research (12/2025)
dataset augmentation, random up to 10% shift up or down, and left or right, well as random horizontal flipping and a random, up to a 15 degree rotation clockwise counter clockwise. The second batch was a "clean" " batch drawn from a separate iterator without data augmentation. Each batch was used both as a benign training batch and also to create NOTA examples based on each training example.
Every 150 batches during training, the same 30 examples were used (previously separated from the validation set) to create 30 untargeted, zero-confidence CW L2 adversarial examples with a maximum of 10 iterations. These adversarial examples were then used to calculate validation attack success rate (ASR), which is used in model selection. ASR was calculated by determining the number of adversarial examples that successfully drove the model to classify the image as a class other than the NOTA class or the true label. The process detailed here was the same for all model training and model selection, whether NOTA, adversarial training, or undefended models.
Table 1: Clean Model Test Accuracies
CIFAR-10
Model
Accuracy
No Defense
92.31% %
Adversarial Training
90.71%
Boundary Padding
90.93%
Adversarial NOTA Envelopment
92.20%
CIFAR-100
No Defense
70.53%
Adversarial Training
66.47%
Boundary Padding
68.13%
Adversarial NOTA Envelopment
69.89%
models WRN-12-6, with dropout of 30% and batch nocmalizasion.
Test set ASR is calculated on the selected model using adversarial examples created using sufficient test set samples to ensure reasonably small confidence intervals. We state our findings along with their 95% binomial confidence intervals. We test against CW L2, CW Loe: AutoPGD-CE, AutoPGD-DLR, DeepFool, Square Attack, and AutoAttack. We adapt each of these attacks to counter the NOTA defense, and add an additional variant of AutoPGD, AutoPGD-AN, In parameterized atlacks, we set max iterations to 100, iee, CW Lp attacks, square attack and DeepFool (L2). All AutoPGD attacks, Square Attack, and AutoAttack are performed in both L2 with max epsilon of 0.5 (maximum distance between x and x' by specified 4, metric) and Lee with max epsilon of 8/255, the distances specified for each by Robust Bench (Croce et al., 2020). Finally, DeepFool is tested in default L2 with the standard max 0.5.
4.1.2 IMDB
We performed NLP experiments using the IMDB binary sentiment classification dataset (Maas et al., 2011 1). We used the same training and test sets as in Guo et al. (2021). We trained and evaluated a model that consists of BERT (Devlin et al., 2019) to create text embeddings, followed by a dropout layer with p = 0.1, a linear layer, and, finally, a softmax operation, as done Morris et al. (2020); Guo et al. (2021). We fine-tuned this model for 5 epochs uisng AdamW (Loshchilov & Hutter, 2017) with a batch size of 16, a learning rate of sequence 2e-05, a weight decay of 0.01, and maximum sequence length of 512.
The NOTA defense and the attack evaluated also required several hyperparameters to be set. When creating NOTA examples during training, we used similar hyperparancters to those used for adversarial training by Geisler et al. (2024). Namely, we started generating adversarial examples for the NOTA class after the model had been trained for one epoch and created a number of successful adversarial examples equal to 20% of the training datasct cach epoch afterwards. For original and NOTA-adapted GBDA, we closely followed the hyperparameters used by Guo al. (2021). Specifically, we used a learning rate of 0.3, a CW confidence of 5, perplexity coefficient. of 1, similarity coefficient of 20, and 100 Gumbel samples.
11

---

Published in Transactions on Machine Learning Research (12/2025)
4.2 Results
4.2.1 CIFAR-10 and CIFAR-100
In Tables and 3, attack success rates (ASR) are reported for original attacks against both undefended and defended models (adversarial training models, boundary padding models, and adversarial NOTA envelopment. models, respectively).
Table 2: Attack Success Rates (ASRs), with 95% binomial confidence intervals, for Original and NOTA Attacks for CTFAR-10.
Models
C&W Suite
APGD CE
APCD AN
APGD DLR
Square Aikk
DF
AA. Untegtd
AA-AN, Untrgid
Lg.
Lon,
Lg.
Lg-
Lgg.
Loo.
Zg.
Lgg
Lg.
Lee.
Cout:0
Cout.
c:0.6
c:0.031
c:0.081
c:0.031
No Defanse
100
99.3%,
100.0%,
05.0%,
12.055.
88.056
98.0%6
100.0%,
100.0%,
Orip. Attacks
12.0%
10.00
41.0%
10.0%
1.00%
4.33%
46.4%
19.7%
49.7%
10.0%
10.00 0%
Adv. Train
100%,
59.2%,
96.6%,
46. 46.4%,
875
49;
35.6%,
02.4%,
53. 63.4%,
96.8%
Orig. Attacks
12.0%
10. 0%
44.4%
11.0%
4.44%
9.
42.9%
14.2%
49.3%
14.4%
11 5%
BP Orig.
0%
1.0%,
6.0%
6.0%
0%.
14.0%;
64.0%,
9.0%
6.0%,
6.0%%
Attacia
+7.29
+2.05%
+4.77%
+4.75
+9.7%%
+ +.9.8%
+5085%
+9.45%
+6.0%
+4.7%
+447%%
ANE Orig. Attucias
12 0%.
0.0%,
5.0%,
5.0%,
15.0%.
50.0%,
5.0%,
40.05,
6.0%,
6.0%
5.0%
+9.4%
+0.05%
+44.956
+4.3%
+7.0%
+9900%
+44 35%
+9.0%%
+4.3%
+4.33%
+4.33%
HP ve. NOTA Atlucia
99.0%,
1.0%,
9.0%,
9.2%
18.8%,
91.8%,
91..0%,
100
006,
15.0%%
63.056
13.0%,
93.4%%
99.6%,
92.6%, 99.6%,
+3.0%
+2.00%
096
+2.0%
=3.4%
-.44%
+.5.0%
+:0. +0.00%
47.0%
4-0.0%
+6.0%
+22.2%
2.0.0%
4-2 3%
+0000%
ANE va. NOTA
9.0%.
5.0%,
8.0%
0.4%5
40.8%,.
74.4%.
65.0%,
01.00%
5.0%,
37.075,
9.0%.
62.0%,
46.4%.
64.0%,. 95.0%,,
Atlucia
+5.0%
44.395
+2.00%
+2.0%
44.4%
-3.5%
+.9.8%
#.5 0%6
#4.8%
4900%
+6.0%
+33.4%
#1.88%
+.4.4%
+1.0%
The blacke repeessen original attacks, for which ANTI-NOTA (AN) loss does nos exist as It is inbereutty adapted for NOTA,
Table 3: Attack Success Rates (ASRs), with 95% binomial confidence intervals, for Original and NOTA Attacks for CIFAR-100.
Models
C&W Suite
APGD CE
APCD AN
APCD DLR
Square Atkk
DF
AA, Untegld
AA-AN, Untegtd
Lg.
Loc
Lg.
Loc
Lg.
Loo.
Lg-
Loc
Lg.
Loc
Zg.
Lg.
Loo. e50.031
Lg.
Confi
+:0.5
+:0.5
No Defeuse
38%
100.0%.
00. 90.7%5
100.0%
41.375.
87.375.
83.5%
00.57
100%
10.05%
10.35%
10.05
1:2.75
20.33%
Train
100%
78.8%
Orig. Attacks
100.0%
1.3.8%
7.
1.5
1:4.35
Orig.
Allacla
+1.3%5
+4..1%
+4.1%
+33.0%
4:4. 0%
+4.4%%
+-4.2%
4.8.8%
144.1%
ANE Ozia. Attacla
2%
0%
31.2%
31.2%
54.0%%
51.2%%
68.0%5
10%
81.2%
81.2%
+4.9%
+00.0%
4.4.1%
4%
44.1%%
+.4.2%
4:8.
+:4.
#:4.1% 1%
BP va. NOTA
54. 4%,
34.885
33. 256
333.2%,,
58.2%.
98%
99..6%,
49.45.
48. 256.
4u.2%
322.8%
47.4%%
98.8%%
98.44%
99.4%, 04.6%,
Attacks
4:4.47
1:4. 2196
15%
24.3%
= 2%%
2:0 0%%
1:0 796
4:4. 4%
1:4. 4%
4:4.1%
3:1 .3%
1..0%
4:1 176
4:0 715
ANE va. NOTA
49.4%,
30. 33.
31.2%
81.2%,
88.4%,
00.0%%
80.81%,
80. 4%,
31, 315,
08.075
20.6%
20.
06.6%,
08.6%, 11.0%
Attacks
144.4%
7.3
1%%
44.1%
=2.07
=1 6%
2.2 7%
2. 0%
43.7%
1.4.
33.0%
$9.75%
#: 0%
4:2 0%
The blanke represeen original artacks, for which A NNTI-NOTA (AN) loss dowe not weist as it 1s inherently ada pted for NOTA.
4.2.1.1 Unadapted Attacks vs. Defenses
With unadapted attacks, we find that both BP and ANE appear highly robust, as can be seen in Tables 2 and 3. Even strong baselines like AutoAttack show low ASR at or below natural error, and several other attacks show substantial reductions as well. Overall, considering the substantially successful defense against these benchmark attacks, NOTA defenses would seem an impressive advance in favor of increased adversarial robustness in classification systems. However, these results stem from unadapted attacks treating NOTA as a regular class, meaning that they often stop once they cross the true class boundary even if they result in a NOTA classification. This also explains the apparent anomaly of how Square Attack appears more effective on its own than AutoAttack (which includes Square Attack as a component). Once an carlier component. attack reaches NOTA, the attack pipeline stops rather than continuing to non-NOTA misclassification. To sum up, these unadapted results overstate robustness.
12

---

Published in Transactions on Machine Learning Research (12/2025)
4.2.1.2 NOTA-Aware Adaptive Attacks vs. Defenses
Adapting attacks to the NOTA paradigm largely restore high ASR against both NOTA-defended models (Tables 2 and 3). The main exceptions are APGD-CE, CW Lx. and DeepFool, where robustness remains closer to the unadapted baseline. By contrast, APGD-DIR and our APGD-AN consistently drive ASR up, demostrating the PGD-based NOTA training may not protect against other PGD-based attacks that use a different loss function. Importantly, no evaluated atlack version becomes weaker after being adapted.
For BP, NOTA-aware adaptations produce large ASR jumps, especially for APGD-DLR and AutoAttack. CW L and APGD-CE show limited gains-consistent with BP's CE-driven training-wii while SquarcAttack and DeepFool change little. BP's apparent robustness under unadapted evaluation evaporates once largel selection and stopping criteria exclude the NOTA "honeypot".
ANE looks strong against unadapted attacks, but is similarly vulnerable under NOTA-aware evaluation. On both CIFAR-10 and CIFAR-100, APGD-DLR, APGD-AN, and AuloAllack regain high ASR. CW L2 is a partial exception (weaker on CIFAR-10, stronger on CIFAR-100). As with BP, SquareAttack and DeepFool remain close their original behavior. Notably, ANE retains some ClE-specific resilience, which makes sense due to CE being used to generate NOTA examples during training.
4.2.1.3 Comparing Adversarial Training with Adversarial NOTA Envelopment
With the advent effective adapted attacks against the best NOTA defenses, a far more balanced comparison can be made between the standard for adversarial defense, adversarial training, and this complement of its defense paradigm, NOTA. First and foremost, our testing confirms that neither is solution to the problem of adversarial allacks alone. Nevertheless, the comparison apparent in tables 2 and are still of interest. Overall, there are some mixed performances reported here, with most, though not all, resulting in adversarial training (AT) providing less robustness against standard attacks than ANE provides against NOTA-aware adapted versions of those attacks. All results considered, ANE would appear to confer modestly more general robustness to a model than adversarial training, however, considering the results from the adapted NOTA- attacks, both collapse and fail to defend models from evasion attacks overall, with several attacks achieving 90% or greater attack success rates on both.
4.2.2 IMDB
In Table 4, the clean accuracies and ASRs for the LLM-based classifier with and without the NOTA A2T defense are reported. While the NOTA defense does decrease clean test set accuracy from 93.66% to 87.75%, it even more icalal reduces the ASR of GBDA from 98.40% to 31.79%. 31.79% demonstrates a very effective defense, considering that the natural error rate the NOTA-defended model is 12.25%. When GBDA is adapted for NOTA, however, the ASR becomes 100.00%, demonstrating that NOTA-defended, LLM-based classifiers are still vulnerable to appropriately adapted adversarial evasion attacks.
Table 4: Test Clean Accuracies and Attack Success Rates (ASRs), with 95% binomial confidence intervals, for IMDB.
IMBD
Model
Clean Accuracy
GBDA ASR
NOTA GBDA ASR
No Defense
93. 33.66%
98.40% & 1.10%
NOTA A2T
87.75%
31.79% E 4.08%
100.00% 0.00%
5
Discussion
In this paper, we bogin by discussing and evaluating group of open-sot adversarial defense approaches which employ a
class defend deep neural networks against evasion attacks. In investigating why this genre of defense, on its surface, is effective against many attacks, we discover common aack failure
13

---

Published in Transactions on Machine Learning Research (12/2025)
modes. Particularly, we find that attacks often fail because: (1) they consider adversarial attack that changes the predicted label to NOTA as success and (2) they can incentivize making the NOTA class have the highest predicted probability. We solved (1) by defining a successful adversarial attack as one that leads to a predicted class that neither the true class nor NOTA. This failure often occurs when defining stopping criteria. To address (2), we adjusted atlacks that use "targel selection" to not select the NOTA class. If "target selection" not used, we investigated changing the loss. Using these, we modified eight prominent and highly effective benchmark attacks. In many cases, this largely recovered attack potency against NOTA defenses for both computer vision and NLP tasks.
We observe that our adapted attacks clearly show that present NOTA defenses are not sufficient to solve the core challenge of defending DNNs against evasion attacks. However, NOTA defenses can confer some small residual resilience to some adapted attacks that at least rivals adversarial training for the analyzed computer vision datasets. With several adapted attacks recovering ASRs back to 90% and above for both the evaluated computer vision and NLP tasks, we advise that future evaluations of NOTA-type or open-sct-inspinn must begin by lesting with NOTA-award altacks. To this end, we make the NOTA Aware adaptations to attacks created in this paper available for public use. Finally, if a NOTA-aware version of attack is not available, the common failures and adaptations discussed in this paper could help practitioners adapt new attacks.
defenses
While these findings are critical, the evaluations of NOTA defenses and attacks could be improved in several ways. In this paper, the NOTA class is populated using methods based on common adversarial training approaches (Madry et al., 2017: Yoo Qi, 2021). However, other promising methods exist for creating NOTA examples. One set of such methods are open-set training approaches (Ge et al., 2017; Chen et al., 2021). Also, other LLM adversarial attacks, could be adapted for NOTA defense, such as Geisler et al. (2024). Additionally, evaluations could be enhanced by using higher-resolution image classification datasets and more text tasks, such as Yelp Reviews (Zhang et al., 2015), AG News (Zhang et al., 2015), and LLM jailbreaking (Geisler et al., 2024).
Acknowledgments
The views expressed in this article are those of the authors and do not reflect the official policy or position of the U.S. Naval Academy, the Naval Postgraduate School, Department of the Navy, the Department of Defense, or the U.S. Government. This work is funded in part by National Science Foundation under Grants nos. 2040209, and 2429835.
References
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484 501. Springer, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated Gradients Give False Sense of Security: Circumventing Defenses to Adversarial Examples. arXiv preprint arx: 00420, 2018
A. Barton, E. Jatho, and V. Berzins. Defending Against Adversarial Examples in Deep Neural Network Classifiers. Technical Report NPS-CS-21-002, Naval Postgraduate School, Monterey, CA 93941, 2021. URL https://calhoun. mps. edh//hadd1e100444/6664
Armon Barton. Defending Neural Networks Against Adversarial Examples. PhD thesis, 2018. URL https://rec httt:://rc..ibbary... wwwpaar...a.a.eetttaaeaeeaa httpaa/ttt//aaaaaaaaaaa/////////
Tom B. Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesso, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πyya Sutskever, and Dario Amodei. Language models few-shot learners, 2020. URL https://arras htt////xiv..r//bbbbaaa wrg/abs///00...o./ 14055.
14

---

Published in Transactions on Machine Learning Research (12/2025)
Ewen Callaway. 'It will change everything:: DeepMind's AI makes gigantic leap in solving protein structures. Nature, 388:203-204 12 2020. doi: 10.1038 00.1088/4411586-02000033333
Nicholas Carlini. (Yet another) broken adversarial example defense at ieee sp 2024. Nicholas Carlini, May cht..deeeeeeeeell....... wwwpohilala..aaan.ia............ carlini.......... http://ttt.../eeaeeaaaeeeeeeeeee
2024. URL https://nicchllas.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pp. 39-57. IEEE, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Tan Goodfellow, Alcksander Madry, and Alexey Kurakin. On Evaluating Adversarial Robustness, February 2019. URL
19t.//090000000.5.. arXiv:1902.06705 [es, stat].
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. Adversarial attacks and defences: A survey, 2018.
Guangvao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pallern Analysis and Machine Intelligence, 14(11):8065-808 2021.
Joana C Costa, Tiago Roxo, Hugo Proença, and Pedro RM Inácio. How deep learning sees the world: A survey on adversarial attacks & defenses. IEEE Access, 2024.
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning, pp. 2196 2205. PMLR, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206 2216. PMLR, 2020b.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. Xiv preprint Xtv:2200 09670, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toulanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171 4186, 2019.
Philip Enevoldsen, Christian Gundersen, Nico Lang, Serge Belongic, and Christian Igel. Familiarity- based open-set recognition under adversarial allacks. In Teliana Lutchyn, Adín Ramírez Rivera, and Benjamin Ricaud (eds.), Proceedings of the 6th Northern Lights Deep Learning Conference (NLDL), volume 265 of Proceedings of Machine Learning Research, pp. 58 65. PMLR, 07 09 Jan 2025. URL attps://proceeding mlr. wwt.pp.pws.....///// prees//255/ennovoleeen252 html.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-word attacks on deep learning models, 2018. URL attps://arxiss org/ htgg/abe//// /100 .89455.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness Aware Minimization for Effi ciently Improving Generalization, April 2021. URL http:://axvvvv.oa.babee 20000000000000011112...... arXiv:2010.01412 [cs, stat].
ZongYuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. Xrx preprinl arXiw:1707.0744 18, 2017.
Simon Geisler, Tom Wollschläger, Johannes Gasteiger, and Stephan Günnemann. Attacking large language models with projected gradient descent. arXio preprint 24o:2402.09144 2024.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICL.R155, 2015.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Kathrin Grosse, David Pfaff, Michael T. Smith, and Michael Backes. The limitations of model uncertainty in adversarial settings. CoRR, 0bs//112.006000 2018. URL. http:///wvv..orra/a////aaa/
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5747 5757, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017. URL http://arren org/ hts/abb/14111 /etss//11...0 6980. arXiv:1412.6980 [cs].
Alex Krizhevsky, Nya Sutskever, and Geoffrey E, Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097 1105, 2012.
lyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint ar Xr1::1111..05101 2017.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Polts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arxi: 1706: 1766. 06083, 2017.
Patrick
McClure, Nao Rho, John A. Lee, Jakub R. Kaczmarzyk, Charles Y. Zheng, Satrajit S. Ghosh, Dylan M. Nielson, Adam G. Thomas, Peter Bandettini, and Francisco Pereira. Knowing what you know in brain segmentation using bayesian deep neural networks. Frontiers in Neuroinformatics, 13, 2019. ISSN 1662-5196. doi: 10.3389 10.3389/fninf.22 2an.::011900067. fninf.2019.0.m 22019000067
URL https://ww..rrattieeiii...ro h
Moosavi-Dezfooll, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574 2282, 2016.
John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. EMNLP 2020, pp. 119, 2020.
Maria-Irina Nicolae, Mathieu Sinn, Tran Ngoc Minh, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Ian M. Molloy, and Benjamin Edwards. Adversarial robustness toolbox v0.2.2. CoRR, abs/1807.0106%, 2018. URL httt:://axxxt wwrrxivv.org/aan http//bb/1880..0000
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Tlya Sutskever, el. al. Language models unsupervised multitask learners. OpenAT, 2019.
Neil Savage. How AI is improving cancer diagnostics. Nature, 579:S14-S16, 03 2020. doi: 10.1038/ (415886-000-00847-2
Ali Shafahi, W. Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? Tn International Conference on Learning Representations, 2019. URL https://openrevill
Rui Shao, Pramuditha Perera, Pong C. Yuen, and Vishal M. Patel. Open-set adversarial defense. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XVII, pp. 682 698, Berlin, Heidelberg, 2020. Springer-Verlag. ISBN 978-3-030-58519.3 doi: 10.1007 978-3-030-58520-4 40. URL / http//d....//11..1111010./.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to crime: Real and stcalthy attacks on state-of thate-tf-hh-art face recognition. In Proceedings of the 2016 aem sigsac conference on compuder and commenications securily, pp. 1528- 1540, 2016.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Fnu Suya, Anshuman Suri, Tingwei Zhang, Jingtao Hong, Yuan Tian, and David Evans. Sok: Pitfalls in evaluating black-box attacks. In 2024 IEEE Conference on Secure and Trusbworthy Machine Learning (SaTML), pp.
IEEE Computer Society, 2024.
Christian Szegedy, Wojeiech Zaremba, Nyy Sutskever, Joan Bruna, Dumitru Erhan, Tan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXi::132..60 139..9999. 2013.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. arXie preprint arXiv:2002. 08347, 7, 2020.
Hamed Valizadegan, Miguel J. S. Martinho, Laurent S. Wilkens, Jon M. Jenkins, Jeffrey C. Smith, Douglas A. Caldwell, Joseph D. Twicken, Pedro C. L. Gerum, Nikash Walia, Kaylie Hausknecht, Noa. Y. Lubin, Stephen T. Bryson, and Nikunj C. Oza. ExoMiner: highly accurate and explainable deep learning classifier that validates 301 new exoplancts. The Astrophysical Journal, 926(2):120, feb 2022. doi:
00.3847/1538-4457/aaalo
URL https://doi..org/00 wttp:/orrr11...8///11155555
Jin Yong Yoo and Yanjun Qi. Towards improving adversarial training of NLP models. In Pindings of the Association for Computational Linguistics: EMNLP 2021, pp. 945-956, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. ar preprinl arXie:1605.0711 071466, 2016.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. CoRR, abs/ 1904.999449955 2019. URL http://aree /ttp://aaxvvvvvva hwtp://bbbb//....09.00.
Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial attacks on deep-learning models in natural language processing: A survey, ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1 41, 2020. Publisher: ACM New York, NY, USA.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-leve convolutional networks for text classification. A Advances in newrol information processing systems, 28, 2015.
17


Reviews:
[{"review_id": "mUNdOetkZC", "reviewer": "Reviewer_HvvB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors provide several contributions: a clear list of failures of existing evaluations in existing attack evaluations (target selection, stopping criteria, objective function), they propose a complete benchmark with 7 distinct attacks, some of which are specifically tailored to open-set defense approaches, and experiments on CIFAR-10/100.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The article is very clear and well-structured. The limitations outlined by the authors in existing evaluations are very well explained and illustrated in the experiments.", "audience_yn": "Yes", "audience_explanation": "The topic is very relevant both from a theoretical perspective and for application. The presentation is very didactic and effectively reviews the domain.", "requested_changes": "It would be interesting to include other open-set methods (e.g., G-OpenMax, ARPL) or hybrid defenses (e.g., OpenSet-AT) to show the generalizability of the findings. On the same note, cifar-10 and 100 are low-resolution datasets. It would be interesting to see if this is also true for high-resolution datasets, such as Imagenet (or perhaps a suggestion for future work).", "broader_impact_concerns": "no ethical concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6icYvKWNyc", "reviewer": "Reviewer_7CjZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies existing adversarial attacks for ML model and defense techniques; and reports that NOTA-defenses results in too successful.  It then analyzes why it is so and proposes a way to attack NOTA-defenses more effectively and realistically.", "claims_and_evidence_yn": "No", "claims_explanation": "The current form is not really well written and have the following strength and weakness (concerns).\nHowever, I think the paper tries to do thorough study, so clarifying the texts should show its contribution better.\n\nStrength: This paper is trying to do thorough study of existing work and experiments; as long as those are better structured and clarified, it may be an interesting contribution.\n\nMajor concern: \n\nThe main concern is that this paper uses large part of the paper for describing existing methods in text with some figures that may or may not help understand the context.  If this is to compare different existing methods to highlight their strength and weakness etc. it is better to use tables or, the authors may want to add clear figures that show illustrations of each method concisely. \nAlso, the authors’ contributions, existing work, their weakness etc. are somewhat entangled in the text (even though they itemize contributions in the introduction).  It is a bit hard from the introduction etc. to see what is the main theme of the paper.  \nFurther, in page 8 for example, the authors introduce target selection, stopping criteria… but they also mention that those aspects have already been discussed in the past work; here again it is a bit hard to see what is the exact contribution of the authors.\nEspecially, section 3 and 4 are very hard to read...", "audience_yn": "Yes", "audience_explanation": "While it is a bit hard to gain clear insights in its current form, the study seems thorough and it should give an interesting finding to the community.", "requested_changes": "1. Restructure the paper to clarify the contributions, existing work comparison etc.  Simple flow, if possible, is preferable (e.g. general overview of attack/defense, about NOTA (that may not be analyzed thoroughly in the past), contribution (discovering its weakness or effective attack?) and proposed approach?)\n\n2. Related to the above; any problem raised in 3.1 and 3.2 should be mentioned earlier and 3. should focus on the authors’ contributions or discoveries. \n\n3. For experiment section, please summarize what you do concisely (while putting setups in tables etc.) and state observations clearly.  Currently, existing approaches, authors’ thoughts, results etc. are mixed up and it is hard to parse what is going on at first sight.\n\nMinor changes:\n\n1. page2 bottom; arrow (map) for f(x) is strange\n2. page3 explanation of evasion attacks is a bit ambiguous; “an input x’ can be crafted…” this statement itself does not imply x’ is close to x.  (I know you explain it just after, but this sentence itself is not well written)\n3. page 4, D is taking one or two inputs?\n4. page 5, what is xhat?  And what is f_{x_i}?", "broader_impact_concerns": "N/A", "additional_comments": "I am happy to see a revised version during this review process.", "extra_scores": {}, "extra_text": {}}, {"review_id": "RiZmbyp9zZ", "reviewer": "Reviewer_fbSB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Modern DNN models are vulnerable to adversarial examples and evasion attacks. A defense method creating None-of-the-Above (NOTA) class is proposed to defend against such attacks. Authors conducted experiments to show that existing NOTA solutions cannot completely resolve the challenges existing in evasion attacks. Authors give a clear introduction about the evasion attacks studied and carried certain number of experiments to support purposed argument.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In section 4, the authors presented results over CIFAR10 and CIFAR100 to show that NOTA based systems can effectively defend against certain attacks. On the other hand, they also showed that some core issues are not completely resolved and evasion attacks can still pose a threat to the scenario.\n\nHowever, it is suggested that since the paper heavily depends on empirical observation. Experiments with other datasets such as ImageNet and other models are needed. Apart from that, tasks apart from image classification should also be studied.", "audience_yn": "Yes", "audience_explanation": "Some individuals would be interested in knowing the findings of the paper as the paper studied on a popular question in terms of evasion attacks of DNN. The paper provides the idea that previous promising solutions for NOTA based defense still cannot resolve the core issues and makes the later researchers think on how they can effectively evaluate NOTA related solutions.", "requested_changes": "- Some sections are redundant. For example, the introduction of classification models in DNN.\n- Can authors elaborate in the section: threat model about in what case an attack can conduct PGD and AutoPGD, etc. and make the classification task fail?\n- \"Researchers Moosavi-Dezfooli, Fawzi and Frossard\": please use \\citep, \\citet, \\citeauthor or similar commands instead of directly writing authors names to do such reference.\n- What is $\\beta$ is section 3.2?\n- Equation should be numbered.\n- In section 3.3, why each attack needs to be proposed with a solution adapting to NOTA. Has anyone previously tried that?\n- Apart from CIFAR10 and CIFAR100 and WRN, other models and datasets should also be taken into consideration. Also, for classification tasks, NLP tasks are suggested apart from image classification.", "broader_impact_concerns": "The paper does not have particular further concerns regarding Boarder Impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)