Generating Less Certain Adversarial Examples Improves Robust Generalization
Authors: Minxing Zhang
Michael Backes
Xiao Zhang
Venue: Accepted by TMLR ()

Abstract: This paper revisits the robust overfitting phenomenon of adversarial training. Observing that models with better robust generalization performance are less certain in predicting adversarially generated training inputs, we argue that overconfidence in predicting adversarial examples is a potential cause. Therefore, we propose a formal definition of adversarial certainty that captures the variance of the model's predicted logits on adversarial examples and hypothesize that generating adversarial examples after the optimization of decreasing adversarial certainty improves robust generalization. Our theoretical analysis of synthetic distributions characterizes the connection between adversarial certainty and robust generalization. Accordingly, built upon the notion of adversarial certainty, we develop a general method to search for models that can generate training-time adversarial inputs with reduced certainty, while maintaining the model's capability in distinguishing adversarial examples. Extensive experiments on image benchmarks demonstrate that our method effectively learns models with consistently improved robustness and mitigates robust overfitting, confirming the importance of generating less certain adversarial examples for robust generalization. Our implementations are available as open-source code at: \url{https://github.com/TrustMLRG/AdvCertainty}.

Published in Transactions on Machine Learning Research (10/2024)
Generating Less Certain Adversarial Examples Improves Robust Generalization
Minxing Zhang
minzing.zhanggllcis de
CISPA Helmholtz Center for Information Security
Michael Backes
backes&ccispa.c de
CISPA Helmholtz Center for Information Security
Xiao Zhang
riao zhang @cispp..d. de
CISPA Helmholtz Center for Information Security
Reviewed on OpenReview: https: // openreview. net/ forum? MdtKKUUUUM
Abstract
This paper revisits the robust overfitting phenomenon of adversarial training. Observing that models with better robust generalization performance are less certain in predicting adversarially generated (raining inputs, we argue that overconfidence in predicting adversar- ial examples is a potential cause. Therefore, we propose a formal definition of advessarial certainty that captures the variance of the model's predicted logits on adversarial examples and hypothesize that generating adversarial examples after the optimization of decreasing adversarial certainty improves robust generalization. Our theoretical analysis of synthetic distributions characterizes the connection between adversarial certainty and robust generaliza- lion. Accordingly, built upon the notion of adversarial certainty, we develop general method to search for models that can generate training-tinm time adversarial inputs with reduced certainty, while maintaining the model's capability in distinguishing adversarial examples. Extensive ex- periments on image benchmarks demonstrate that our method effectively learns models with eotsystently improved robustness and mitigates robust. overfit ting, confirming the importance of generating less certain adversarial examples for robust generalization. Our implementations are available open-source code at: attps://gitie wtcchhu...ommmm/m wtt.////////iii http://usstMLRRcAdvCetaaalll
1 Introduction
Deep neural networks (DNNs) have achieved exceptional performance and have been widely adopted in various applications, including computer vision He et al. (2016), natural language processing Devlin et al. (2019) and recommendation systems Covington et al. (2016). However, DNNs have been shown highly vulnerable to classifying inputs, known as adversarial examples Szegedy et al. (2014); Goodfellow et al. (2015), crafted with imperceptible perturbations that are designed to trick the model into making wrong predictions. The prevalence of adversarial examples has raised serious concerns regarding the robustness of DNNs, especially when deployed in security-critical applications such as self-driving cars Chen et al. (2015), biometrie facial recognition Komkov & Petiushko (2021) and medical diagnosis Finlayson al. (2019); Ma et al. (2021). To improve the resilience of deep neural networks against adversarial perturbations, numerous defenses have been proposed, such distillation Papernot et al. (2016), adversarial detection Ma et al. (2018), feature denoising Xie et al. (2019), randomized smoothing Cohen et al. (2019), and semi-supervised methods Alayrac et al. (2019). Among them, adversarial training Madry et al. (2018); Zhang et al. (2019) is by far the most popular approach to learn robustness against adversarial perturbations. Nevertheless, even the sato-of-thhe-rt. adversarial training methods Croce et al. (2020); Rebuffi et al. (2021): Wang al. (2023) cannot achieve satisfactory robustness performance on simple classification tasks like classifying CIFAR-10 images.

---

Published in Transactions on Machine Learning Research (10/2024)
Witnessing the empirical challenges for improving model robustness, many recent works focus on understanding the behavior of adversarial training Tu et al. (2019); Gao et al. (2019); Wu et al. (2020); Zhang et al. (2020); Yu et al. (2022). In particular, Rice et al. observed that test robust accuracy of intermediate models produced during adversarial training immediately increases by a large margin after the first learning rate decay but keeps decreasing afterward, known as robust overfilling Rice et al. (2020). Robust overfitting has recently altracted a lot of attention, since is not an issuc for standard deep learning but appears to be dominant in adversarial training. Therefore, recognizing the fundamental cause of robust overfitting may provide important insights for designing better ways to produce more robust models. In this paper, revisit the robust overfitting phenomenon and provide a potential reason for why it happens. More concretely, we observe that models produced during adversarial training tend to be overconfident in predicting the class labels of adversarial inpuls, whereas models with better robust generalization exhibit much less significant overconfidence issues. By introducing the notion of adversarial certainty, which captures the variation of a model's output logits in prodicting adversarial examples generated by the model itsclf, we provide thcorctical evidence and empirical results showing that generaling adversarial examples after the optimization of decreasing adversarial certainty helps produce models with improved robust generalization.
Contributions. By visualizing the label predictions of adversarial examples generated at different epochs, we observe that adversarial training is prone to produce overconfident models, which further induces decreased test robust accuracy. Thus, we argue that generating training-time adversarial inputs after the optimization of decreasing adversarial certainty can improve robust generalization (Section 3). To study the hypothesis more rigorously, we first introduce a formal definition of adversarial certainty, and then provide thcoretical results on synthetic distributions that characterize the connection between adversarial certainty and robust. gencralization (Section 4).
Built upon the definition of adversarial certainty, we propose a general method to explicitly Decrease A Advrsarial Certainty (DAC) during adversarial training (Section 5). At a high level, DAC is designed to find training-time adversarial examples with lower certainty for improving model robustness. In particular, DAC first finds the steepest descent direction of model weights to decrease adversarial certainly, and then the newly generated adversarial examples with lower certainty are used to optimize model robustness. As the model learns from less certain adversarial examples, the aforementioned overconfidence issuc is expected to be largely mitigated. In addition, provide correlation analysis between adversarial certainty and robust generalization, which illustrates the importance of imposing proper constraints on model search space for DAC. By conducting extensive experiments on image benchmark datasets, we demonstrate that our method consistently produces more robust models when combined with various adversarial training algorithms, and robust overfitting is significantly mitigated with the involvement of DAC (Section 6.1). Moreover, we find that our proposed adversarial certainty has an implicit effect on existing robustness-ehhancing techniques that are even designed based on different insights (Section 6.2). Besides, we provide a more intuitive demonstration of DAC's efficacy (Section 6.3), and update the explicit optimization of adversarial certainty by using regu- larization term to improve the efficiency (Section 6.4). These empirical results again indicate the importance of adversarial certainty in understanding adversarial training and bring a further comprehension of our work.
Notation. We use lowercase boldfaced letters for vectors, and (-)) for the indicator function. For any x E Rd and i E {1, (1,2,...dd] dl, let x; be the i-th element of æ. For any sinite-sample set S, let |S] be the cardinality of S. Let (X, A) be a metric space, where A : X' X - R denotes a distance metric. For any E X and e 2 0, let. (x;; A) (a' E X A(x', 2) S }} be the ball centered at æ with radius and metric A. When A is free of context, we simply write (x) = (x; A). Let μ be probability distribution on X' x V where y/ denotes label space. The empirical distribution of i with respect to a sample set S is defined as: (Cs(C) = wess 1 ((x,8) y) E C)/[S for any measurable set C C X x V. Let N(y,0²) be the Gaussian distribution with mean 7 and standard deviation o > O.
2 Related Work
Adversarial training a promising defense framework for improving model robustness against adversarial examples Goodfellow et al. (2015); Madry et al. (2018); Zhang al. (2019); Wang et al. (2020); Tramèr et al. (2017); Shafahi et al. (2019); Andriushchenko & Flammarion (2020); Wong et al. (2020); Jin et al. (2022). Tn
2

---

Published in Transactions on Machine Learning Research (10/2024)
particular, Goodfellow et al. proposed to adversarially train models using perturbations generated by the fast gradient sign method (FGSM) Goodfellow et al. (2015). Later on, Madry et al. incorporated perturbations produced by iterative projected gradient descent (PGD) into adversarial training Madry et al. (2018), which learns models with more reliable and robust performance. Other variants of adversarial training have been proposed, which lypically modify the training objective but also use PGD altacks to approximately solve the inner maximization problem. For instance, Zhang et al. designed TRADES, which considers optimizing the standard classification loss while encouraging the decision boundary be smooth Zhang al. (2019). Wang et al. proposed MART to emphasize the importance of misclassified examples during adversarial training Wang et al. (2020). In this work, we demonstrate how to improve the robust generalization performance of these adversarial training algorithms by searching for models with lower adversarial certainty.
Apart from improving adversarial training, several recent works focus on understanding robust generalization and leveraging the gained insight build more robust models Rice et al. (2020); Stutz et al. (2021); Hwang et al. (2021); Chen et al. (2021); Yu et al. (2022); Xu et al. (2023). In particular, Rice et al. discovered that, unlike standard deep learning, robust overfitting is dominant phenomenon for adversarialll-Itaanee DNNs that hinders robust generalization, and advocated the use of early stopping Rice et al. (2020). Wu et al. discovered that the flatness of weight loss landscape is an important factor related to robust generalization, which inspires them to adversarially perturb the model weights during adversarial training Wu et al. (2020). Besides, Tack et al. proposed a consistency regularization term based on data augmentation to mitigate robust. overfitting Tack et al. (2022). Our work complements these methods, where we explain why overconfidence in generating adversarial examples is highly related to robust overfitting and illustrate how to improve robust generalization by promoting less certain perturbed inputs for adversarial training. Moreover, we are also awarc of two existing works that focus on improving the performance of adversarial training with the consideration of model overconfidence Stutz et al. (2020); Setlur et al. (2022). However, these works target different objectives from ours. More specifically, Stutz et al. developed a saffieene--alibratee adversarial training method that achieves better robustness against unseen attacks Stutz et al. (2020). Setlur et al. proposed a regularization technique to maximize the entropy of model predictions on out-of-distribution data with larger perturbations, thus improving model accuracy on unseen examplesSetlur et al. (2022).
3 Overconfidence Compromises Robustness
In this section, we first introduce the most relevant concepts, including adversarial robustness, adversarial training and robust overfitting, of which the complete introduction and discussion are detailed in Appendix A. Next, we visualize the label predictions of adversarially trained models by heatmaps, and propose our hypothesis that model overconfidence is potential cause of the decreased robust generalization in adversarial training, where robust overfitting occurs.
Preliminaries. In this work, we focus on the most widely-studied fn-norm bounded perturbations, and work with the following definition of adversarial robustness:
p)) = - Eaayi max (ff(x') # 4y), (x)
where fo is an arbitrary classifier, μ denotes the underlying data distribution, and 2 0 captures the adversarial strength. In practice, adversarial robustness estimated based on set of testing examples R.(fo; Ps..)) is typically used as the evaluation metric for measuring the robust generalization of fo. Adversarial training aims to improve model robustness by training on adversarialll-pertuubee inputs Goodfellow et al. (2015); Madry et al. (2018); Zhang et al. (2019), which can be formulated as a min-max optimization problem:
min 1 BEe
max Σ
((fo,z',))
(1)
S
x'EB.(
(o.))ES.r
where θ represents the model class, Ser is a set of training examples independently and identically sampled from μi and denotes some convex surrogate loss such as cross-cntropy. Note that PGD attacks Madry et al. (2018) are typically employed in adversarial training to provide approximated solutions to the inner maximization problem in Equation (1). Nevertheless, PGD-based adversarial training and its variants Madry
3

---

Published in Transactions on Machine Learning Research (10/2024)
Aingaon
0:91
0.00
0:02
0.00
0:00
0.000
5:00
0.00 0.04 0.00
Aurclank
0:52
0.05
0:08
0.04
0.04
0.00
0:01
0.00
(.27
0.05
Audo
0.00 0.50
5.05
0.00
0:00
0.00
9:00
0.00 0.00 0.00
Asto
0.02 0:7
0.01
0.03
0:41
0.01
0:01
0.00
0.05 0.19
Ground- Truth Label
Bir
0.32
0.00
0.00
Ground- Truth Label
Bed
0.10
0.04
0.83
2.00
0.00
0:01
0.62
0:05 0.07 D/04 0.01 0.00 0.00
CaR
0.03
0.01
0.08
0.17
0.13
0.33
0.13 0.06 0.63 0.03
5.00
0.00
9.51
0.02
0:0 0.02
0.08
0.04
0.00 1000
Deeer
0.03
0.00
9.02
0:14
0:28
0.1
0:15
0.14
0.04 6.01
6:40
6:33
Model
Model
0.00
From
0.00
0.00
0.02
0.03
0:00
0.01
0.54
0.00
0.00
0.00
0.02 Forks
0.01 0.07
0:20
0.17
0.10
0:38
0.01 0.01 0.0L
0.25
Hone
0.00 0:00 6:03 0.00
5:00
0.96
0.00
Horet
0.02 6.04 0:17
0:11
0:00
0.60
0.300
6.16
969
0.00 3.00 0:00 0.00
0.00
0.99
4//9
0.12
0.03 0.04
0.03 0.01
0.01
0:03
0.30
Trosk
0.00 0.01 0:00 0.00 6:00 0.00 0:00 0.00
0.00
0.57
Trolk
0:03 0:37 0.00 0.05 0.43 0.02 0:00 0:03
0.09
0.57
6.00
Not goo CI guel gut qurhyur giit qeadts
Aodion No00 gigh git oot guol qurth gii qual
6:00
Ovand
upp & ofe Caee o
Predicted Label
Predicted Label
(a) Last Model (TTrain)
(b) Last Model ('Test)
(a) Label-level Variance
Alogloo
0.29 DO1
0.0L
0:00
0.01 0.00 C.D 0.01 5.15 0.
Aingbon
0.53
0.04
0.05
0.02 0.02 0.01 0.02 0.00
0.00 0.06
Adversriial Cortainty
6.0
Best Model
Last Model
Auto
0.00
0.00
0.02
0.01
0.00 3:01
0.14
Ground. Label
0.05
0:01
0.52
0.06 0.13 0.05 0.14 0.05 582 0.0L
Ground-Truth Labol
0.00
0.08
0.31
0.11 0.13 0.06 5.15 0.06 0.23 0.03
2-4
0:02
0.03
0:13
5.15
0.05 0:01 0.0M
0.04
0:07
0.22 0.01 0.17 0.17 0.07 0:04 0.00
1.6
Dest
Cent
0.00
0.6
Det.
0.01
0:03 0.04
0.16
0.05
0.53
0.03
0.06
D:02 0.02
0.03 0.52
0.00
0.20
0.05
0.57
2.03 0.10
0.33 0.03
Eook
0.01 0.04 0.06
0.08
0.70
5:02
0.02
Evet
0.03 0.08 0.10 0.11 0:55
0.00 0.03
Overall
Airplane
0.01
0.01 0.58
0.02
0.02
0.62
2:00
Herwh
0.03
0.00
0.06
0.04
0.62
0.05
(b) Adversarial Certainty
559
0.04
0:03 0.00 0.00 0.00 0.00 0.00
0.00
2:81
0.00
SH9P
(0.11 0.07 0.02 0.03 0.01 0.01 0:61 0.00
0:38
0.06
Track
0.0L
0.00 0.01 0.00 0.0L
002
Trask
0.04 0.19 0.08 0.00 0:01
0.62
9 of ged 9otk yitk work
Noche gid of get yor git yort
Predicted Label
Predicted Label
Figure 2: Model confidence in pre- dicting training-time adversarial ex- amples conditioned on the ground- truth class label using different met- rics: (a) label-level variance, and (b) adversarial certainty.
(c) Best Model (Train)
(d) Best Model (Test)
Figure 1: Heatmaps of the label predictions of training- and testing-time generated adversarial examples with respect to models produced from the last and best epochs of adversarial training.
et al. (2018); Zhang et al. (2019) suffer from the robust overfitting phenomenon Rice et al. (2020): The test-time robustness of intermediate models produced during the training process sharply increases after the first learning rate decay but keeps decreasing aflerward. As result, the model produced from the last. training epoch cannot achieve a satisfactory robust generalization performance.
Heatmap Visualizations. To gain a doeper understanding of robust overfitting, we visualize the heatmaps of the label predictions for adversarially- perturbbd CIFAR-10 images. Given that robust overfitting captures the gap of robust generalization performance with respect to models produced the last and best epochs, we first plot Figures (bb) and (d). Since only the training process is accessible in adversarial training, we also depict the corresponding training-time heatmaps in Figures 1(a) and 1(c). Here, the ground- troth label represents the underlying class of clean images and the predicted label denotes the class of adversarial examples predicted by the corresponding model. More experimental details about Figure 1 are provided in Appendix B. Specifically, when comparing Figures (()) and 1(c), we find that the predictions of Last Model mainly concentrate on the ground-truth class, which means the model is overconfident in predicting adversarial examples generated by itself. In contrast, the heatmap of Best Model, which achieves better robust generalization performance, depicts less overconfidence. Moreover, by comparing the same model between the training and testing time, ie.. Figure 1(a) versus Figure 1(b) and Figure 1(c) versus Figure (dd), we discover that the train-test gap is significantly smaller with respect to the Best Model. We note that this result is aligned with the classical machine learning theory: If the testing distribution deviates more from the training distribution, standard learners will show a decreased generalization performance.
According to the above findings we hypothesize that the overconfidence property is detrimental to robust generalization. To be more specific, if the model cannot generate perturbed training inputs with sufficient uncertainty, the model will not be able to well predict the less certain adversarial examples during the
4

---

Published in Transactions on Machine Learning Research (10/2024)
inference time. Thus, mitigating model overconfidence could be potential solution to improve robust generalization for adversarial training. In Section 4, we will introduce novel notion of Adversarial Certainly, which is proposed to measure the degree of model overconfidence and is essential for designing our DAC method to help robust generalization as demonstrated in Section 5.
4 Introducing Adversarial Certainty
To numerically summarize our findings from the heatmaps, we measure the variance of the class probabilities of the predicted labels, denoted as label-level variance, with respect to the training-time adversarial examples for each ground-truth category in Figure 2(a). A lower label-level variance indicates the prediction confidences of different labels are closer, which corresponds to less certainty. More specifically, we observe that the Best Model with better robust generalization performance exhibits lower label-level variance than that of the Last Model, which is consistent with the results illustrated in Figure 1. Even though the label-level variance can characterize how certain the training-time generated adversarial examples are, such statistics are on a class level, which is not easy for optimization. Thus, we propose the following logit-level definition, termed adversarial cerlainly, to caplure the certainly of model in classifying the adversarial examples generated by itself, where lower score of adversarial certainty suggests the model has stronger ability to generate less certain adversarial inputs:
Definition 1 (Adversarial Certainty). Let X be the input space and y {1, 2, .,..., m} be the label space. Suppose μ is the underlying distribution and S is a set of sampled examples. Let 2 0, A be the perturbation metric. For any fo X + ), we define the adversarial certainly of fo as:
AC.(fo: A) = S
Σ (H.Y)ES
Var(Fo[A(a: so,
where A denotes an attack method such as PGD attacks for generating adversarial examples, Fe : X - R" represents the mapping from the input space X' to the logit. layer of fe, and Var(u) = Exe(m) (ux T)² hm///m, with ug and u denoting the k-th element and mean of u E Rm respcctively.
Different from label-level variance, adversarial certainty is an averaged sample-wise metric, which calculates the variance of the logits returned by the model fo for each to Figure 2()), we visualize the adversarial certainty of the Best Model and the Last Model in Figure 2(b).
example A(x: y, fo,c). Similar
Since predicted labels are decided by the class with the highest predicted probabilities, adversarial certainty depicts similar pattern to the label-level variance expected. Based on Definition 1, our hypothesis then be specifically formulated as:
Decreasing adversarial certainty during adversarial training can improve robust generalization.
We note that there also exist other alternative metrics, such as confidence and entropy, which can capture a model's certainty in predicting adversarial examples and summarizee the observations of the heatmaps depicted in Figure 1. As will be discussed in Section 6.1, we choose logit-level variance as the metric to define adversarial certainly, mainly because our DAC method illustrated in Section 5 always achieves the best robust generalization performance with such a choice.
Theoretical Analysis. To better understand the proposed definition adversarial certainty, we further study connection with robust. generalization using synthetic data distributions. Following existing works Tsipras et al. (2019); Wei et al. (2023), we assume the following data gencrating procodure for any example (æ,, y) ~ pr: The binary label y is first sampled uniformly from = {-1, +1}, then the robust feature x1 = y with sampling probability p and æ1 -y otherwise, while the remaining non-robust features T2, Fd+1 are sampled i.i.d. from the Gaussian distribution (ny, 1). Here, p E (1/2, 1) and 7 < 1/2 is small positive number. Following Wei et al. (2023), we consider linear SVM classifiers: (2) = sgn(z1 + eatuottau ) with w > O, where sgn(-) denotes the sign operator. Subsequently, we assume all the adversarial examples w' are sampled from the following adversarial distribution Paay with E 0: x = 21, and zy..... 2/441 iid. A( -)yy 1). Detailed discussions about the configurations of this synthetic robust classification task are provided in Appendix C.
5

---

Published in Transactions on Machine Learning Research (10/2024)
The following theorem, proven in Appendix C.1, characterizes connection between the certainty of adversarial examples and the robust generalization performance of an SVM classifier after single step of gradient update.
Theorem 1. Consider the aforementioned data distribution μ and robust classification task. Let Ete (1, 2n) and fw be an arbitrary SVM classifier with w > 0. For any E [n- a n), (f: M. Madv(e)), the adversarial certainty of fw: is monotonically decreasing with respect to E. Suppose we conduct one-step gradient update on w using adversarial examples sampled from (e)v(e): 1 = + v. RiJu [aadv())), where a >> 0 stands for the learning rate. Then, Rffo (atv(Ste)) the robust generalization performance fø, also increases as e increases.
Note that, since we consider the adversarial data distribution Padv instead of Lp perturbations, we now generalize the notion of adversarial certainly and robust generalization correspondingly Theorem suggests that if we decrease the certainty of the adversarial examples sampled from Madv(e), the robustness of the SVM classifier Ao will increase after one-step gradient update based on the sampled adversarial examples, confirming the importance of less certain adversarial examples for robust generalization. We remark that our theoretical analysis can also be extended to the typical setting of fo-morm bounded perturbations. In Appendix C.2, we show that considering too perturbations is equivalent to considering the adversarial data distribution of z = 21 - yE and "., " ii.d. sampled from N((n - (0y,n--y,,10 1) for any w > O, and derive similar results to Theorem
5 Decreasing Adversarial Certainty Helps Robust Generalization
Previous sections illustrate why decreasing the certainty of adversarial inputs used for adversarial training is beneficial for robust generalization. To further validate our hypothesis, this section proposes a novel method to explicitly Decrease Adversarial Certainty (DAC) based on adversarial training. In particular, DAC is designed to find less certain adversarial examples that are used to improve robust generalization, which aims to solve the following optimization problem:
min 6E0
Str
Σ
max L(fer, x', .W) where 0' argmin AC; Str; x')
(2)
('EC(0)
(m.
Ser the clean training dataset, denotes a specific attack method (e.g., PGD attacks Apga), and C(0) represents the fcasible scarch region for 0'_ We remark that imposing the constraint of C(6) is necessary, because the goal of DAC is to improve robust generalization of adversarial training, instead of merely obtaining adversarial certainty as low as possible. Without such a constraint, minimizing adversarial certainty will cause 0' to significantly deviate from the initial 0. This will render the adversarial examples generated with respect to less useful, thereby inducing a negative impact on robust generalization (sce Figure 3(a) and our correlation analysis for more discussions regarding the design choice of imposing such a constraint set).
Directly solving the min-max-min problem introduced in Equation (2) is challenging, due the non-convex nature of the optimization and the implicit definition of C(f). Thus, resort to gradient-based methods for approximate solver. To be more specific, take the t-th iteration of adversarial training example to illustrate our design of DAC. Given set of clean training examples Str, a specific attack method A, and a classification model fa, our DAC method can be formulated as a two-step optimization:
0.4.. = 02 - VoAC Str. (6r..A)) AA 0=0,
(3)
0.+1 O2+0.5 - 7 VoIrob(Jo: Str, A) 0=00
where À >> 0 and 7 > 0 represent the step sizes of the two optimization steps, ACe A) denotes the adversarial certainty of fo with respect to Sür and A, and Lrob(fo; Sir, can be roughly understood as the robust. loss except that the inner maximization is approximated using some attack method such as Angd- The (3) optimizes the adversarial certainty, which adjusts the model parameters 0 in a direction such that the generated training-time adversarial examples are less cortain, whereas the sccond stop in Equation (3) optimizes the model's ability in distinguishing adversarial examples generated by the model itself as in standard ad vereariia training.
first step in Equation
6

---

Published in Transactions on Machine Learning Research (10/2024)
Adversaril Certainty
Adveerari Certainty
Advercarial Certainty
Robolt Accuracy
Robust Accuracy
Robuss Accuracy
1:
cc
0000030 404008000
25
Step Size
Step Size
(a) Correlation Analysis (Last)
(b) Correlation Analysis (t steps)
(c) Correlation Analysis (Best)
Figure 3: Correlation between adversarial certainty and robust generalization under different configurations.
Correlation Analysis. Since our work aims to improve robust generalization by finding less certain
adversarial examples, it is natural to ask the following question:
Does decreasing adversarial certainty always induce better robust generalization?
Recall that in Equation (2), C(O) defines the feasible region for optimizing adversarial certainty. Therefore, the answer would be affirmative within this region, i.e. decreasing adversarial certainty will increase test robust accuracy. To support the answer to this question with evidence, we conduct correlation analysis between adversarial cortainty and robust generalization. The results are illustrated Figure 3(a). Specifically, derived. Then,
we use an AT-trained model as the starting point, from which the healmaps in Figure respectively update the model with one more epoch using DAC with different step sizes, ranging from 0.1 to 2.0, in the 0₂ + 0440.5 step of Equation (2) to decrease adversarial certainty. Finally, we measure the training-lime adversarial certainly (i.e., the blue bars) and robust Lest accuracy (i.e., the orange curve) of the result models. Figure 3(a) shows that adversarial certainty keeps decreasing as the step size increases. Meanwhile, the model's robustness first keeps improving but then decreases when the step size beyond the value of 1.3. This result suggests that if the model parameters lie in the feasible search region with a properly selected step size, lower adversarial certainty leads to higher test robust accuracy. However, when the model is out of the feasible search region, decreasing adversarial certainty will no longer improve robust generalization.
The above investigation suggests a negative answer to the question by performing one-epoch optimization with different step sizes on the last model of AT, i.e., the model of Figures (a) and 1(b). Here, we demonstrate that our finding is indeed dcrived from the changes of adversarial certainty but not from the onc-epoch setting. To be more specific, we separate one step of decreasing adversarial certainty with size 0.7 into steps, i.e., each step corresponds to the size of 0.7/t, where {1, 4, 5, 6, 7, 9, 10]. The results of training-time adversarial and testing-time time robust generalization are depicted in Figure 3(b), where
observe consistent pattern with Figure 3(a) that robust generalization first gains more improvements but then less with the decrease of adversarial certainty.
Moreover, we apply the one-epoch optimization on the best AT model that corresponds to Figures 1(c) and 1(d) to derive more observations. From Figure 3(c), we can find that the changes of robust generalization with the increase of adversarial certainty depict a different pattern from Figure 3(a). First, the robust. generalization improvements are slighter than those in the last model. Besides, there is no trend of robust generalization decreasing even if the step size arrives at 2.0. These results suggest that the feasible region of the best model is larger than that of the last model, which suffers from more severe robust overfitting. Consequently, when applying our DAC method, an overconfident model is supposed to select the optimization step size carefully; otherwise, the selection could be more ambitious.
6 Experiments
This section examines the performance of our DAC method under Eoo perturbations with = 8/255 on various model architectures, including PreActResNet-IIS denoted as PRN1S, and WideResNet-344 denoted as WRN34. And we train a model for 200 epochs using SGD with a momentum of 0.9. Besides, the initial learning rate is 0.1, and is divided by 10 at the 100-th epoch and the 150-th epoch. The adversarial attack used in training is PGD-10 with step size of 1 1/255 for SVHN, and 2/255 for CIFAR-10 and CIFAR-100, while we
7

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Testing-tinn time robustness (%) with/withhon DAC on CIFAR-10 under &o perturbations across different architectures and adversarial training methods. The best performance is highlighted in bold.
Architecture
Method
Clean
PGD-20
PGD-100
CWoc 0c
AutoAttack
AT
82.88 (82.68)
41.51 (49.23)
40.96 (48.92)
41.61 (48.07)
39.66 (45.71)
+ DAC
84.64 (83.55)
45.55 (52.20)
44.94 (51.87)
44.55 (50.05)
42.78 (48.20)
PRNIS
TRADES
82.10
(81.33)
47.44 (51.65)
46.95 (51.42)
46.64 (49.18)
44.99 (48.06)
+ DAC
83.18 (82.80)
49.32 (52.90)
48.81 (52.67)
48.30 (50.11)
46,40 (48.96)
MART
80.85
(78.27)
50.23 (52.28)
49.71 (52.13)
46.88 (47.83)
44.68 (46.01)
+ DAC
81.12
(79.37)
52.38 (53.25)
52.04 (53.14)
48.97 (49.25)
47.24 (47.69)
AT
86.47 (85.86)
47.25 (55.31)
46.73 (55.00)
47.85 (54.04)
45.84 (51.94)
+ DAC
86.48
(85.10)
52.02 (57.93)
51.69 (57.68)
51.51 (54.98)
49.75 (53.33)
WRN34
TRADES
83.37
(81.40)
51.51 (58.78)
51.28 (58.72)
49.26 (53.33)
47.74 (52.63)
+ DAC
85.04 (84.55)
58.97 (60.96)
58.97 (60.81)
52.79 (55.00)
51.80 (53.99)
MART
83.11 (83.30)
48.93 138.
48.31 (57.75)
46.32 (52.22)
41.99 (50.31)
+ DAC
84.69 (80.09)
52.00 (59.31)
51.32 (59.26)
49.50 (53.02)
47.65 (51.48)
utilize the commonly-used attack benchmarks of PGD-20 Madry et al. (2018), PGD-100 Madry et al. (2018), CW Carlini & Wagner (2017) and AutoAllack Croce & Hein (2020) for evaluation. In addition, we measure the Clean performance to investigate the influence on clean images. Regarding other hyperparameters, we follow the settings described in their original papers. In all cases, we evaluate the performance of the last (best) model in terms of testing- testing-time robust accuracy.
In Section 6,1, we evaluate the effectiveness of our DAC method in improving robust generalization on three widely-used benchmark datasets: CIFAR-10 Krizhevsky & Hinton (2009), CIFAR-100 Krizhevsky & Hinton (2009) and SVHN Netzer et al. (2011) based on three baseline adversarial training methods: AT Madry et al. (2018), TRADES Zhang et al. (2019) and MART Wang et al. (2020). To study the generalizability of our method, we further conduct experiments under &g perturbations, where we set <= 128/255 with a step size of 15/255 for all datasels. In Section 6.2, we associate with other robustness-enhancingg techniques to further investigate the effect of adversarial certainty in adversarial training. Finally, we demonstrate the efficacy of DAC under simplified one-step optimization setting in Section 6.3, and improve the DAC efficiency by regularizing decreasing adversarial certainty in loss term, i.e., DAC Reg that will be detailed in Section 6.4.
6.1 Main Results
We first evaluate the robust generalization performance of our proposed DAC method on the benchmark CIFAR-10 image dataset. The comparison results are depicted in Table 1, showing that DAC significantly enhances model robustness across different adversarial attacks, such as PGD attacks Madry et al. (2018), CW attacks Carlini & Wagner (2017) and AutoAttack Croce & Hein (2020). These results demonstrate the effectiveness of DAC, indicating the significance of generating less certain adversarial examples for robust generalization. Besides, we observe that although WRN34 suffers from more severe robust overfitting using baseline adversarial training methods, it achieves more robustness improvement by our method. This suggests that WRN34 superior to PRN18 in terms of robust generalization with the help of DAC. In addition to adversarial robustness, it is also worth noting the effect of DAC on clean test accuracy, which captures the standard generalization ability of the model. Table reveals that DAC consistently improves the clean test. accuracy under all experimental settings. This promotion shows that DAC could also help models gain better generalization performance on unseen clean images even by learning from adversarial examples. The results that include evaluations on more benchmark datasets (i.e., SVHN and CIFAR-100) are depicted in Tables 2 and 3, of which the full versions are shown in Tables 7 and 8 (Appendix E) due to space limit, showing a similar pattern of improvements.
Morcover, we empirically study the impact of DAC on the phenomenon of robust overfitting. More specifically, evaluate the gap of testing-time adversarial robustness between the best and the last models. The results
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: Testing-time adversarial robustness (%) of AT on SVHN with/without DAC/DAC Ree under loo perturabtions across different model architectures and benchmark datasets. The complete results are shown in Table 7, including all dataset settings of SVHN, CIFAR-10 and CIFAR-100.
Dataset
Architecture
Method
Clean
PGD-20
PGD-100
CW
AutoAttack
AT
89.63 (88.64)
42.25 (51.00)
41.37 (50.30)
42.84 (48.19)
39.52 (46.02)
PRNI8
+ DAC
90.58 (89.63)
45.86 (54.42)
43.92 (58.78)
48.75 (50.15)
40.68 (48.23)
+ DAC Reg
90.65 (90.21)
45.39 (53.06)
43.77 (52.28)
43,66 (19.64)
41.10 (17.39)
SVHN
AT
91.51 (89.72)
46.81 (53.43)
44.94 (52.77)
45.76 (50.43)
41.71 (49.50)
WRN34
DAC
91.26 (91.83)
60.42 (67.95)
50.71 (64.85)
56.98 (65.09)
42.33 (50.42)
+ DAC Reg
91.76 (92.13)
62.19 (65.96)
59.54 (63.68)
60.05 (63.87)
42.46 (49.95)
Table 3: Testing-timm adversarial robustness (%) of AT, TRADES and MART with/without DAC on SVHN under Loo perturbations. The complete results are shown in Table 8, including all dataset settings of SVHN,
CIFAR-10 and CIFAR100.
Dataset
Method
Clean
PGD-20
PGD-100
CW
AutoAttack
AT
89.63 (88.64) 64)
42.25 (51.00)
41.37 (50.30)
42.84 (48.19)
39.52 (46.02)
+ DAC
90.58 (89.63)
45.86 (54.42)
43.92 (53.78)
43.75 (50.15)
40.68 (48.23)
+ DAC _eg
90.65 (90.21)
45.39 (53.06)
43.77 (52.28)
43.66 (49.64)
41.10 (47.39)
TRADES
89.12 (87.75)
51.50 (55.19)
50.69 (54.50)
45.50 (50.32)
45.02 (48.69)
SVHN
+ DAC
90.24 (89.59)
52.24 (57.09)
51.14 (56.39)
46.34 (52.22)
46.20 (50.52)
+ DAC Reg
90.03 (89.75)
51.78 (56.10)
50.92 (54.83)
45.86 (51.35)
45.30 (49.06)
MART
89.68 (81.48)
19.07 (52.30)
48.30 (52.22)
45.48 (48.04)
44.54 (47.38)
+ DAC
88.90 (84.64)
51.04 (53.64)
50.91 (52.70)
46.94 (49.96)
46.18 (48.50)
+ DAC Reg
90.18 (88.47)
50.94 (52.94)
49.87 (52.46)
46.32 (49.18)
45.86 (47.73)
Table 4: Testing- time adversarial robust (%) of AT with/withoud DAC on PreAcdResNel-188 under &
perturabtions against PGD-20 across different benchmark datasets.
SVHN
CIFAR-10
CIFAR-100
Method
Best
Last
Best
Last
Best
Last
AT
66.45
63.20
66.02
65.18
39.23
35.68
+ DAC
69.11
67.44
69.10
67.37
40.75
36.32
are shown in Figure 4(a), where DAC consistently mitigates robust overfitting across different settings. These results indicate that decreasing adversarial certainty can successfully mitigate robust overfitting. Besides, we also mcasure the adversarial certainty gap between the best model and the last model produced by AT and AT- DAC in Figure 4(b). It can be observed that the adversarial certainty gap of AT-DAC is significantly smaller than that of AT, which is aligned with the closer adversarial robustness of the best model and the last model.
Comparison with Other Metrics. Recall our discussions in Section 3, we propose the notion of adversarial certainty based on logit-level variance (Definition 1), which further used our design of DAC. Noticing that confidence and entropy are also relevant metrics that can capture the model's overconfidence in prodicting adversarial examples, we conduct case study to illustrate why we choose to define adversarial certainty based variance. For ease of presentation, we only present results on CIFAR-10 and AT as illustration, where similar trends are observed among other settings. Table 5 reports the test-time adversarial robustness of models learned using AT-DAC with different metrics used in the definition of adversarial certainty. We can see that the last and best models produced using our method with the variance metric achieve the best. robustness performance, which empirically supports our design choice.
fg-Norm Bounded Perturbations. In the above evaluation, we focus on the loo norm-bounder pertur- bations. Meanwhile, the &g norm is also prevalent perturbation setting in advessaiial training. Thus, in
9

---

Published in Transactions on Machine Learning Research (10/2024)
Adversarial Certainty Gap
4.2
Robust Accuracy, Gap
With DWC
Without DAC
1.2
AT-DAC AT
Adversarial Certainty
DAC
1.0
0,0
DAC. Reg
98
86
22.4
1.8
AT-P
TRADES MART-P
AT-W
TRADES
MART-W
Overal
Aiplayg
Aurio
Bidd
Oct
Truck
40
100
120 Epoch
140
180
200
(a) Robust Overfitting
(b) Adversarial Certainty Gap
(c) Training Curves
Figure 4: (a) Robust overfitting across different methods, where "-P" and *-W" represent PRN18 and WRN34 respectively. (b) Adversarial certainty gap with respect to AT and AT-DAC conditioned on different
ground-truth classes. (c) Training curves of adversarial cortainty with respect different adversarial training algorithms.
Table 5: Comparison results (%)
4.5
3.4
Adversarial Cortainty
AT
AT-Conistencyy
Adversarial Certainty
3.2
AT-Conistency
of different metrics defining ad- versarial certainty on PRN18 and
4.0
AT-AWP
3.0
AT-AWP
3.5
28
CIFAR-10 at the last and best epochs.
3.0
2.2
2.0
20
Overal
Aisplane
huto
giob
ghes yoth
Horse Ship
Truck
1.8
Ownadl
Airplane
Aito
Bird
Dest Dott feod
Horse
Ship
Truck
Last
Best
Confidence
44.40
51.14
Last Model
(b) Best Model
Entropy
44.27
51.00
Figure Adversarial certainty across different CIFAR-10 classes with on the last and best models.
Variance
45.55
52.20
Table evaluate our method under the l2 perturbations. Similarly, DAC depicts consistent improvements in adversarial robustness best and last epochs across different benchmark datasets, which shows the efficacy of DAC against adversarial attacks with & perturbations.
6.2 Effect of Adversarial Certainty on Other Robustness- Ennanesss--Enaiinngg Techniques
We note that several recent works also focus on understanding robust generalization and developing methods to improve adversarial training, including adversarial weight perturbation Wu et al. (2020) (AWP), and consistency regularization Tack et al. (2022) (Consistency). More concretely, Wu et al. discovered that the flatness of the weight. loss landscape is an important factor related to robust generalization Wu al. (2020). And the method of Consistency regularizes the adversarial consistency based on various data augmentations Tack et al. (2022). However, since these methods focus different stratogics to improve robust generalization, it is unclear whether our proposed adversarial certainty has any connection with them. Therefore, we study the changes in adversarial certainty when involving AWP and Consistency in adversarial training, respectively, which are shown in Figure 5. Surprisingly, we find that AWP and Consistency, which improve the robust generalization of AT on both the last and best models, can gain lower adversarial certainty. These findings are consistent with the idea behind our DAC method decreasing adversarial certainty helps robust generalization. In other words, AWP and Consistency, which are designed toward their specified directions, will implicitly decrease adversarial certainty. Note that, even if AWP and Consistency have influences adversarial certainty, it does not mean that work proposes similar concept to them. Specifically, adversarial certainty is derived by observing an adversarial-trainnng-nnog overfitting, meanwhile, AWP is inspired by the theory of weight loss landscape from standard learning and Consistency considers the augmentation scope. Consequently, our proposed adversarial certainty is a crucial property in adversarial training, which can either explicitly or implicitly affect robust generalization.
phenomenon robust.
As AWP and Consistency can implicitly improve adversarial certainty, we then investigate the compatibility of DAC method with AWP and Consistency by a naive attempt. To incorporate DAC in AWP, add step
10

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: Testing-time adversarial robustness (%) of AWP and Consistency with/without DAC on CIFAR-10 and PRNI8 under So perturabtions.
Method
Clean
PGD-100
CW
AutoAUtack
AT-AWP
81.76.0.006 (82.37 L0.07)
$2.71
(3.28)
51.0710.24 (51.2210.24)
48.75.10.23 (49:33110.27)
+ DAC
84.07 13 182.67+0.10)
54.30-00.66
(55.00-0.)
51.76-00.25
(52.03+0.2) 0.022)
49.80-00226
(49.96+-0.20
TRADES-WWP
8144600 13 (81 $11.2800008)
52. 02.51
(03.66+0.266)
50.37=0.23
(50.61+0.21) 21)
(19. (19,92+0.gg 23) 49.5+++025
+ DAC
82.69-00.06 (82.85+0.00%
53.80-+0.29
(54.49:0..3
5..4440.28
(...51)
50.51 0.26
(50.63--0225)
MART-AWP
0300000 (77.27+0.00 09)
03.00:0..25 (52.58 (62.88+0.31)
49 0..0050.22 28 (48.39+40 22)
46.53+0.22 22 (47.01 17.01+0..0 26)
+ DAC
80.03+0.06 (78.65=0.11)
51.67:0029
(31.93=0.30)
05
(49.111:0211)
17.17=0.27 (17.73=:.21)
AT Cousistency
65.2820.06 (84.6610.D0))
05.1610.33
(56.46.10.27))
50.8110.23 (51.13.10.22)
48.08.10.21
(48, (48.4810.22 23)
DAC
85.36+0.09 (85.17=0.13)
56.31=0.26
(66.9000.26)
61.20=0.27 (51.72=0.22)
19.00=0.21
049.16:0225)
63.6820.12 (0..))
52.7810.20 26 (32.79:10.31)
48.85.10221 (48.891100.2) 28)
47.75.1::211
(47.77:0.23
+ DAC
84.7810.12 (84.73. 0.066)
53.4810.22
(53.72. 0.26)
49.37 10.27 (49.41 L0.28)
18.15.10.21
(48.19
SAART-Consistenuy
08.21=0.10 (78. 11..11::.. 01.9)
36.3110.21
(36.81 1:0 (8)
47, 0.23
(47.4710022))
05,53.10.27
(15.73:0.22)
+ DAC
$1.9110.06 (81.35 1.
58.2910.33 10.33
(58.56
.0.28)
50.08 10.27 (50.21 L0.27)
48.28 10.22
(48.59100.2
Adversaial Certaney
-345
Advers Certainty
Advestriall Cortainty
Rebust Acairacy
TRADES
MART
AT-DAC
TRADES-DAC
MART-DAC
30
120
10
20
30
10
6x
UK 30 18
Epoch
Epoch
Epoch
(a) AT
(b) TRADES
(c) MART
Figure 6: Visualization results for comparing the adversarial certainty and robust generalization of different adversarial training methods with and without the involvement of DAC.
before weight perturbation to optimize the certainty of adversarial examples. Then the updated intermediate model is used generate new adversarial examples for the following AWP optimization. Similarly, we first explicitly update the adversarial certainty on augmented samples, and then follow the Consistency optimization. The results are shown in Table 6. As expected, since AWP and Consistency have already implicitly decreased adversarial certainty, even if DAC conducts an explicit optimization, our method can only gain limited benofit. Neverthcloss, our repeated trials demonstrate that the improvements, even slight, are indeed derived from our method rather than randomness. Further, we conduct a significance test, which shows that the improvements of robust generalization on AWP and Consistency are statistically significant, as fully presented in Appendix D. The goal of our work is propose adversarial certainty and clarify its significance in adversarial training, thus better designs of involving adversarial certainty in existing robustuess-cnhancing strategies are left as future work.
6.3 Further Discussion on DAC
Based on previous results, we demonstrate the benefits of involving our DAC method in adversarial training. To more intuitively demonstrate the efficacy of our method, we empirically measure the performance improvements derived by conducting DAC for single epoch starting with different models. First, we train a sequence of models by AT and TRADES for 200 epochs, and by MART for 120 epochs, respectively. For every 20 epochs, we then update the same intermediate model by one further epoch using each of the three adversarial training methods with and without the help of DAC. Finally, we mcasure the adversarial certainty and robust generalization for all the updated models. Figure 6 summarizes the results, where the blue color represents the original method without DAC and orange corresponds to results with our DAC. The bars show adversarial certainty and the curves depict robust generalization. It can be seen from Figure that starting with different intermediate models, DAC can consistently gain less certain adversarial examples, from which the updated model attains better robust generalization performance, which is aligned with our theoretical
11

---

Published in Transactions on Machine Learning Research (10/2024)
results shown in Theorem 1. By optimizing the same model with only one epoch, these comparison results clearly show the efficacy of DAC for adversarially-trainee models.
6.4 Improvement on DAC Efficiency
To gain better understanding of our method, we explicitly examine our proposed adversarial certainty by involving two steps for each iteration ie., DAC, as formulated in Equation (3). In this section, we propose more efficient method, denoted as DAC_ Reg, by regularizing the optimization of adversarial certainty as a term in adversarial training loss. More concretely, the optimization problem with the additional regularizer can be cast as:
min liee |
[ffe, + Sf: (v.y)ES.r
Σ
where 3 > O denotes the trade-off parameter between the regularization of adversarial certainty and the robust loss. Similar to adversarial training, the model parameters are iteratively updated using stochastic gradient, descent (SGD) with respect to the regularized robust loss. Benefiting from the regularization design, DAC_Reg requires similar training time to the standard adversarial learning, which 0.56 of that of DAC. For instance, for a PRN18 model of AT and CIFAR-10 on a single NVIDIA A100 GPU, DAC averagely costs 143s for each training epoch while DAC Reg costs 80s. The comparison results of AT, TRADES and MART models SVHN, CIFAR-10 and CIFAR-100 datasets
shown in Table 2 and Table 3. We can see that DAC Reg achieves comparable performance, due to the additional penalty adversarial certainty, which is only a bit inferior to DAC. In few cases, DAC could bring better and more stable improvements. For instance, when a PRN18 model is trained on CIFAR-100 by AT, DAC Reg can only gain the improvement on the last epoch but not on the best opoch. In addition, we mcasure the adversarial certainty of a sequence of models trained by AT, DAC and DAC Reg, respectively, in Figure 4(c). We observe that DAC gains the lowest adversarial certainty with slight advantage over DAC Reg, again indicating that lower adversarial certainty corresponds to higher robust generalization.
7 Conclusion and Future Work
We revisited the robust overfitting phenomenon of adversarial training and argued that model overconfidence in predicting training-tine time adversarial examples is potential cause. Accordingly, we introduced the notion of adversarial certainty to capture the degree of overconfidence and designed a strategy to decrease adversarial certainty for models produced during adversarial training. Experiments on image benchmarks demonstrate the effectiveness of our method, which confirms the importance of generating less certain ad adversarial examples for robust generalization. Our work aims to gain a better understanding of robust generalization through obscrvations from robust overfitting. We believe our work provides a significant contribution to advancing the field of adversarial machine learning, which might inspire practitioners to look into the important role of less certain adversarial examples when building real-world robust systems against adversarial examples.
Investigating whether the notion of adversarial certainty connects with the robustness of language models could be an interesting future direction of our work. Nevertheless, natural language processing (NLP) is a different field from computer vision (CV), as it focuses on the discrete space while that CV is continnous. Thus, directly transferring our method to the NLP domain is non-trivial. According to the pipeline of vanilla adversarial training for text classification (Morris et al., 2020), the generation of adversarial examples is substitute some selected words. Such generation scheme is different from that of CV and does not support the mudient-bascd optimization of decreasing adversarial certainty. In that case, we will need to design an additional objective function to measure how certain NLP model is about its generated sentences, which could be the metric for selecting and substituting words. We regard the exploration of DAC in other domains like NL. as interesting future work but is beyond the scope of our work. We believe our proposed concopt of adversarial certainty will provide important insights for the development of robust machine learning systems.
12

---

Published in Transactions on Machine Learning Research (10/2024)
References
Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In Annual Conference on Newral Information Processing Systems (NeurIPS), pp. 22192-12202 NeurIPS, 2019.
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. Annual Conference on Newral Information Processing Systems (NeurIPS). NeurIPS, 2020.
Nicholas Carlini and David Wagner. Towards Evaluating the Robustness of Neural Networks. In IEEE Symposium on Security and Privacy (SBP), pp. 39-57. IEEE, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C. Duchi, and Percy Liang, Unlabeled data improves adversarial robustness. In Anmual Conference Neural Information Processing Systems (NeurIPS), pp- 11190 11201. NeurIPS, 2019.
Chenyi Chen, Ari Seff, Alain Kornhanser, and Jianxiong Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. In IEEE International Conference on Computer Vision (ICCV), pp. 2722-2730. IEEE, 2015.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may mitigated by properly learned smoothening. In International Conference on Learning Representations, 2021.
Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified Adversarial Robustness via Randomized Smoothing, In International Conference Machine Learning (ICML), pp. 1310 1320. PMLR, 2019.
Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In ACM Conference on Recommender Systems (RecSys), pp. 191-198. ACM, 2016.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning (ICML), pp. 2206 2216. PMLR, 2020.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. aixiv preprint ar Xix:220010. 09670, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pro-training of Deep Bidirec- tional Transformers for Language Understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HITT), pp. 4171-4186. ACL, 2019.
Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan Zittrain, Andrew L Boam, and Isaac S Kohanc. Adversarial attacks on medical machine learning. Science, pp. 1287-1289, 2019.
Ruiqi Gao, Tianle Cai, Haochuan Li, Cho-Jui Hsieh, Liwei Wang, and Jason D Lee. Convergence of adversarial training in overparametrized neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Examples. In International Conference on Learning Representations (ICI.R, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778. IEEE, 2016.
Joong-wer won won Hwang, Youngwan Lee, Sungchan Oh, and Yuseok Bac. Adversarial training with stochastic weight average. In IEFE International Conference Image Processing (TCIP), pp. 814-818. IEFE, 2021.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial Examples Are Not Bugs, They Are Features. In Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 125 136. NeurLPS, 2019.
Gaojic Jin, Xinping Yi, Wei Huang, Sven Schewe, and Xiaowei Huang, Enhancing adversarial training with second-order statistics of weights. In JEEE Conference on Compuler Vision and Pullern Recognition (CVPR), pp. 15252- 15262. IEEE, 2022.
Can Kanbak, Scyod-Mohsen Moosavi-Dezfool; and Pascal Frossard. Gcometric robustness deep networks: analysis and improvement. Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, pp. 4441 1449, 2018.
Stepan Komkov and Alcksandr Petiushko. Advhat: Real-world adversarial attack on areface face id system. In IEEE International Conference on Pottern Recognition (ICPR), pp. 819 826. IEEE, 2021.
Alex Krizhevsky and GeolTrey Hinton. Learning multiple layers of features from tiny images. 2009.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi N. R. Wijewickrema, Grant Schoenebeck, Dawn Song, Michael F. Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In International Conference Learning Representations (ICLR), 2018.
Xingjun Ma, Yuhao Niu, Lin Gu, Yisen Wang, Yitian Zhao, James Bailey, and Feng Lu. Understanding adversarial attacks on deep learning based medical image analysis systems. Paltern Recognition, 110: 107332, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Atackk.. In International Conference on Learning Representations (ICLR), 2018
John X. Morris, Eli Lifland, Jack Lanchantin, Yangfong Ji, and Yanjun Qi. Roevaluating adversarial examples natural language. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 3829 3839. ACL, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits natural images with unsupervised feature learning. 2011.
Nicolas Papernot, Patrick D. McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation Defense Adversarial Perturbations Against. Deep Neural Networks. In IEFF Symposium Securily and Privacy (S&P), pp. 582 597. IEEE, 2016.
Sylvestre-Alvise Rebufi,, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Timothy Mann. Fixing data augmentation to improve adversarial robustness. ar preprint ar arXi:220030000 03.01946, 2021.
Leslie Rice, Eric Wong, and J. Zico Kolter, Overfitting in adversarially robust deep learning. In International Conference on Machine Learning (ICML), pp. 8093-8104. PMLR, 2020.
Amrith Setlur, Benjamin Eysenbach, Virginia Smith, and Sergey Levine. Adversarial unlearning: Reducing confidence along adversarial directions. In Annual Conference on Neural Information Processing Systems (N)urIPS). NeurIPS, 2022.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zhong Xu, John P. Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 3353-3364 3364. NeurIPS, 2019.
Mahmood Sharif, Lujo Baner, and Michael K Reiter. On the suitability of Ip-norms for creating and preventing adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1605-1613, 2018.
14

---

Published in Transactions on Machine Learning Research (10/2024)
David Stutz, Matthias Hein, and Bernt Schicle. Conffdence-calibrated adversarial training: Generalizing to unseen attacks. In International Conference on Machine Learning (ICML), volume 119, pp. PMLR, 2020.
David Stutz, Matthias Hein, and Bernt Schiele. Relating adversarially robust generalization to flat minima. In JEEE International Conference on Computer Vision (ICCV), pp- 7787-7797. TEEE, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob
Fergus. Intriguing Properties of Neural Networks. In International Conference on Learning Representations (ICLR), 2014.
Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung Ju Hwang, and Jinwoo Shin. Consistency regularization for adversarial robustness. In AAAI Conference on Artificial Intelligence (AAAT), pp- 8414-8422. AAAI, 2022.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble Adversarial Training: Attacks and Defenses. In International Conference on Learning Represen- lations (ICLR), 2017.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations (ICLR), 2019.
Zhuozhuo Tu, Jingwei Zhang, and Dacheng Tao. Theorctical analysis of adversarial learning: A minimax approach. Advances in Neural Information Processing Systems, 32, 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving Adver- sarial Robustness Requires Revisiting Misclassified Examples. In International Conference on Learning Representations (ICLR), 2020.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. arXiv preprint 2XXx:2302.046333 2023.
Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang, CFA: class-wise calibrated fair adversarial training. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations (ICLR), 2020.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In Annual Conference on Newral Information Processing Systems (NeurIPS). NeurIPS, 2020.
Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. arXiv preprint (xXi::1801.02611 2018
Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L. Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 501-509. IEEE, 2019.
Yuancheng Xu, Yanchao Sun, Micah Goldblum, Tom Goldstein, and Furong Huang. Exploring and exploiting decision boundary dynamics for adversarial robustness. CoRR abs/2303 03015, 2023.
Chaojian Yu, Bo Han, Li Shen, Jun Yu, Chen Gong, Mingming Gong, and Tongliang Liu. Understanding robust overfitting of adversaria training and beyond. In International Conference on Machine Learning (ICML), pp. 25595 25610. PMLR, 2022.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Thcoretically Principled Trado-off between Robustness and Accuracy. In International Conference on Muchine Learning (ICML), pp. 7472 7482. PMLR, 2019.
Yi Zhang, Orestis Plevrakis, Simon S Du, Xingguo Li, Zhao Song, and Sanjeev Arora. oarrmparaeeterize adversarial training: An analysis overcoming the curse of dimensionality. Advances in Neural Information Processing Systems, 33:679 -68, 2020.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Appendix
A Complete Introduction of Preliminaries
For the sake of completeness, this section presents the detailed definitions and discussions of the preliminary concepts introduced in Section 3, including adversarial robustness, robust generalization and adversarial training. Let (X, A) be metric space. For any set C C X' and any æ E X, let Ilc(x) argmin A(x', be the projection of x onto C.
Adversarial Robustness.
Adversarial robustness captures the classifier's resilience to small adversarial perturbations. In particular, we work with the following definition of adversarial robustness:
Definition 2 (Adversarial Robustness). Let A' C I" be imput space, y be label space, and be the underlying distribution of inputs and labels. Let A be a distance metric on X' and 2 0. For any classifier So:X - the adversarial vobusiness of So with respect to ,4, 6 and A is defined as:
Re( =1
[ E Ba())) s.t. So(x') # v]- (m.
Pr
(4)
When 0, Ro(fe; (4) is equivalent to the clean accuracy of fe. In practice, the probability density function of the underlying distribution μ is typically unknown. Instead, we only have access to set of test examples Ste
ii.d. sampled from μ. Thus, classifier's adversarial robustness is estimated by replacing μ in Equation (4) with its empirical counterpart based on Ste. To be more specific, the testing- time adversarial robustness of fo with respect to Ste, e and A is given by:
R₂ R. = =1
Σ
max 1(Jo(z') + v), (
55)
Stel
(Z..))ESL
where sss denotes the empirical measure of y based on Ste- We remark that robust generalization, the main subject of this study, captures how well a model can classify adversarially-pertubeed inputs that are not used for training, which is essentially the lesting-time adversarial robustness (fo, As..). And we Re(fo: Ps..) in the following discussions when Ps... is free of context. In this work, we focus on the f--norm distances as the perturbation metric A, since they are most widely-used in existing literature on adversarial examples. Although lp distances may not best reflect the cumman-perceptual similarity Sharif et al. (2018) and perturbation metrics beyond f,-norm such as geometrically transformed adversarial examples Kanbak et al. (2018); Xiao et al. (2018) were also considered in literature, there is still a significant amount of interest in understanding and improving model robustness against lp perturbations. We hope that our insights gained from lp perturbations will shed light on how to learn better robust models for more realistic adversaries.
write R. (fo)
Adversarial Training. Among all the existing defenses against adversarial examples, adversarial train- ing Madry et al. (2018); Zhang et al. (2019); Carmon et al. (2019) is most promising in producing robust models. Given a set of training examples Sp, sampled from μ4, adversarial training aims to solve the following min-max optimization problem:
min S.r; where (fo: = S OEE
Σ
max afoo ,9)
(6)
a'EB. (x)
(E.))SS..
Here, O denotes the set of model parameters, and L is typically set as a convex surrogate loss such that (fee 2) is upper bound on the 0-1 loss 1 (fo(xx # y) for any (z,y). For instance, L is set as the cross- entropy loss in vanilla adversarial training Madry et al. (2018), whereas the combination of cross-entropy loss for clean data and a regularization term for robustness is used in TRADES Zhang et al. (2019). In theory, if Ser well captures the underlying distribution and the robust loss (Jo: is sufficiently small, then fe is guaranteed to achieve high adversarial robustness R. (f.:
However, directly solving the min-max optimization problem (6) for non-convex models such as deep neural networks is challenging. Tt is typical to resort to some good heuristic algorithm to approximately solve the
16

---

Published in Transactions on Machine Learning Research (10/2024)
problem, especially for the inner maximization problem. In particular, Madry et al. proposed to alternatively solve the inner maximization using an iterative projected gradient descent method (PGD) and solve the outer minimization using SGDMadry et al. (2018), which is regarded as the go-to approach in the research communtty We further explain its underlying mechanism below. For any intermediate model fo produced during advesaarial training, PGD updates the (perturbed) inputs according to the following update rule:
1s+1 = (a)) (I, + t...0.00 sgn Vgn(Va, L,o, v.. for any (æ,y) and E (0. S-1}.
....
(7)
where 20 2 > denotes the step size and S denotes the total number of iterations. For the ease of presentation, we Apgd to denote PGD attacks such that for any example (x..) and classifier fo, it generates x' = Øs = Apga(a;; y, f), based on the update rule (7). After generating the perturbed input for each example in a training batch, the model parameter 0 is then updated by single SGD step with respect to L(Jo, x/,y) y) for the outer minimization problem in Equation (6).
B More Details of Figures in Sections 3 and 4
This section provides all the experimental details for producing the heatmaps and the histograms illustratod in Sections 3 and 4. Given model So (e-g., Best Model and Last Model) and set of examples S sampled from the underlying distribution (e.g., CIFAR-10 training and testing datasets), adversarial examples are generated by PGD attacks within the perturbation ball (xx) centered at æ with radius c 8/255 under the for perturbations, which follows the settings of generating training samples considered in Sect ion 6, e.g., PGD is iteratively conducted by 10 steps with the step size of 2/255. We record and plot the label predictions of the generated adversarial examples with respect to each model as heatmaps in Figure 1.
Let HM be the m x m matrix representing the heatmap, where y {11,2.... m} denotes the label space. For any j, k E V, the (j, k)-th entry of HM with respect to fo and S is defined as:
{ar,y) E ES: y j and (a y, Jo, = k}
HM,
(8)
{(x, v) S y:y=:y10 =}}
where Apgd denotes PGD attacks defined by the update rule (7). More specifically, for any (2,y) E ES, the PGD atlack produces the corresponding adversarial example x' (ag y, fe, Then, we measure the predicted label fo(=)) In that case, for the given training data, we could construct (ground-truth, predicted) label pairs, simply denoted by {v,,)), Afterward, we first cluster ((,,)) separately by the ground-truth label, e.g., the subset of ground-truti label j includes all pairs such that y = j (denoted by {(v, ((v,));)) which corresponds to the rows of heatmaps. Further, for each subset, we group it into sub-subsets separately by the predicted labels, e.g., {(V, })},.k contains all pairs in {(y B)}, such that ø = k. Consequently, the number of adversarial examples of the ground truth label j is calculated as:
Kl 0)bb = [(x,)) E ::y 53
Meanwhile, the number of adversarial examples of ground truth label j but predicted as label k is measured as:
Ml 1)).a ((x,)) v) e S: s:y=j and (e (x; fe <)) =k]
where 9 - So (x; y. Sa, e)) Finally, we compute the ki, entry of the heatmap HMG as the ratio of Kl, 1l to Kly 0),1 (Equation (8)). Following the same settings, we plot the corresponding label-level Specifically, we first mcasure the label-level variance of
variance and adversarial cortainty in Figure the training-time time adversarial examples of the last model (Figure 1(a)) and the best model (Figure 1(c)) conditioned on the ground-truth label in Figure 2(a). Taking the ground-truth label j an example, the
17

---

Published in Transactions on Machine Learning Research (10/2024)
label-level variance can be formulated as:
Var (label)
Vaaelabel)
HHM j,h HM,)),
kEY
where HM, averages all HMj.k with different h and y = {1, 2.. m) is the label space. According to Definition 1, we measure the adversarial certainty of the last and the best models, as illustrated in Figure 2(b), with respect to the prodicted logits of all the adversarial examples with respect to each ground-truth label class.
...
C Proofs of Theoretical Results in Section 4
To gain a better understanding of the proposed definition of adversarial certainty, we further study its connection with robust generalization using theoretical data distributions. Following existing works Tsipras et al. (2019); Wei et al. (2023), we consider a simple binary classification task, but further step of gradient update is considered based on our work. First, we lay out the mathematical formulations of the important concepts under the assumed setting that will be used for the proofs.
Data Distribution. For this binary classification task, we assume the following procodure of data gencration for any example (æ, y) ~ μe The binary label y is uniformly sampled, i.e., y/ u.a.F. u.f.F.
[-1,+1}, then the robust. feature Ø1 y with sampling probability p and æ1 -y otherwise, while the remaining non-robust features
$2.-. Idd= are sampled i.i.d from the Gaussian distribution N(ny, 1). Here, E ( 1) and 1 3 is a small positive number. In general, the data distribution can be formulated as:
+y. w.p. P
and -2,-. Td+1 i.d. N(w, 1).
T1
(9)
-y w.p. 1 1-p
SVM Classifier, Without bias term, an SVM classifier is used, ie., f(x) = sgu 21 where sgn(-) denotes the sign operator. And for brevity, we assume Wi, w2 + 0 and w2 _... Wd+1 1
22.-.. Fd411 are equivalent. Let "11 the classifier is simplified (x) sgn(x1 + . ). And
wa iddd.
without loss of generality, since x2, 2d+1 ~ N(yy 1) tend to share the same sign symbol with y, we further assume w >
Adversarial Distribution. As discussed in Tsipras et al. (2019) and llyas et al. (2019), ii is robust to perturbation but not perfect (as p 1), while .z.... Td+1 useful for classification but sensitive to small perturbation. Following the setting of Tsipras et al. (2019), the non-robust features are shifted towards -y by an adversarial bias distribution for constructing adversarial examples. More specifically, the adversarial examples x' are sampled from the following ad adversarial distribution Madv(e) with e >> 0:
2 æ1, and 2/2 2'a4 i.d.d. N(((1 -)y, 1).
(10)
Note that, in this task, no perturbation bound c is involved, which is different from PGD-Attack. Instead, the distribution bias ε is used to find/sample adversarial examples, which is independent of the attacker's budget. Besides, the goal of this work is to find less certain adversarial examples in the training time. As e can directly decide the distribution of adversarial examples, there is no need to vary adversarial certainty by finding new model status.
Robust Generalization. Since we do not use PGD-based attacks to find adversarial examples as the empirical parts, instead of Definition 2, we utilize the corresponding version of robust generalization based on the adversarial distribution (Edv(E). Accordingly, given the model fw: the clean and robust generalizations are separately denoted by R(SwiN u1) and R(Ju: (e)), which are simply written as Ro(Jw) and R=(aa)) when p and (E) are free of context:
Ro(fw) = - 1 (fr = v),
(11)
R₂ (fw) = B(mr))) (z = v).
18

---

Published in Transactions on Machine Learning Research (10/2024)
For the sake of simplicity, let robust error (f() ) be the robust loss for the optimization, i.c.,
(c) = 1 - (mw)
(12)
The normal distribution ((0,,) 1) is defined by the distribution function Φ(x) and the probability density function (x)):
((x) |
2 - dt P(V(0,)) <.).
(13)
<(z) 1
I (x))
2
Recall that w > 0, according to Wei et al. (2023), we have
Rolfw) - pdl dn + (1 - p) dn
(14)
vd
Based on the distribution of non-robust features of adversarial examples, i.e, 2 ~ Non e)y, 1), we simply replace with (1 E) in Equation 14, Vw 0, have
Re(Fw) = =p d(1 5) - w
d(n E) w
+ (1 p) (1--))0(
(15)
dd
Vd
Consequently, we have
d(n c) w
d(n - -c) w) p) (1-p)
(f(fw) = 1 1
(16)
vd
vd
Adversarial Certainty. In Section 4, we provide our definition of adversarial certainty (Definition 1) by using the empirical counterpart of adversarial distribution. However, the thcoretical part, adversarial distribution Padv(e) is accessible. Thus, we use Madvís) to directly define the adversarial certainty for this binary classification task. In general, adversarial certainty measures how certain a model predicts the training-time adversarial examples, i.e, the variance of different cases of the ground-truth and the predicted labels. Based Equation (15), all probable cases of robust generalization
(a) +1 and fw = +1, which corresponds to robust, generalization Re (Sw);
(b) y = +1 and fw - 1, which corresponds to robust generalization 1 - Re
(c) = 1 and for 1,, which corresponds to robust generalization Rf
(d) y = and fn +1, which corresponds to robust generalization Rs (fw).
As y u.r.r (11,+1}, it yields Pr(y +1) Pr(u -1)=
For simplicity, we let AC(f n.e) = (Cw (E)v(E)) in the following discussions. According to the above discussions, the adversarial certainty can be formulated as
AC(fwi ()) = Var (R(fw)) al. x'4+1)
2 (e - + +G (fu))) 1
(e (() (+)()(((w)))
(17)
(Re(fu)- -1)2 (fw)) 2
2R. (fw)) (Rx(wv)) +
Now we ready to proof Theorem 1.
19

---

Published in Transactions on Machine Learning Research (10/2024)
C.1 Proof of Theorem 1
Proof of Theorem 1. We start by showing the monolonicity of adversarial certainly with respect to e.
Monotonicity of Ww(wwi According to Equation (17), we have AC v, s) = ( (s))
]]. Thus, the derivative to e is
(= (fr( VeRe(fw) (F
1 Re((w) - 5 (eRe(fu).
In that case, to study the monotonicity of AC(Soi n,),, there is a need to discuss the sign of (Ra()w) and (fw).
2
VeRe(fw) - -app o( d(n c) + w =))
d(n - - w vā(1 -d(1p)) p) 0(
(18)
vd
vd
As > 0, 0, p) < p, and o(x) > 0, in Equation (18), (eRe(fw) < 0, i.e, Re (fw) is monotonically decreasing with respect to 8.
According to Equation (15), have
(e( p- o d(1 8)
+ w
+ (1 (p) ( d(1 E) w
Vd
Vd
din-e)+
din-o)-w
1
= P
c $ dt + (1 - p)
e
T dt.
2TT
v 2T
When 8 1/, d(n-- -) = 0, thus
1
Re (fw) = p.
dt
+ (1 -p).
T dt
2
- 2TT
1
v1
2
p
dl + (1 1-p) - -11 p)
4 dt
2T
v 2m
= (2p
1)
C
dt (1 - p)
-00 2m
(2p 1) 2 + p1-p)
p 2 -1- 2'
Since (e( monotonically decreasing with respect to E, when e E (0,n), we have R=(sw) 1 >
In that case,
ACCCfin) - - (((() - 1 VeRe(u) O,
that is, AC(fwin, is monotonically decreasing with respect to when (0, v/. Monotonicity of Rew (fa). As aforementioned, robust error (x(ww) = 1-ph( din d(n-s)+ - (1-1)))))) 4) vd
d(n-s)-w
vd
is used as the robust loss optimize w. In that case, the derivative of (f(ww) to w is
Vd" o d(y 8) +
(1 - p)
d(1 E) - w
V VwEc(fo)
(19)
Vd
Vd
Vd
20

---

Published in Transactions on Machine Learning Research (10/2024)
Accordingly, the optimized parameters ve by step size of is derived as
1 VwEe(fu)
op Vd
Φ1 d(n - +
vd o( d(1 E) a(1 p).
(20)
vd
Following the evaluation of Tsipras el al. (2019), the non-robust. features are shifted Lowards -y to mislead (a()), i.e, Ete E [n, 2nj, where the sampled adversarial examples follow x NN(n Ete)U, 1) 22 a+1
Based on v and &te, the robust generalization
Re (fa) p $ d(n Ete) + 10 = (1 p) . d(n Ete) 20 )
(21)
Vd
vd
Accordingly, the derivative to à is
P o( d(n Ete)
(d
+ +
1-p p vd
d(1 Etc) 1
VæRen Vd
(22)
Vd
As Ete E [n, 2n), d(1 Ece) S O. Thus, of (1
2 0
d(n ((0=))e)==)
Consequently, V (fe) > 0 when
Ete in, 2ml, i.., Re, (fa) monotonically increasing with respect w.
Monotonicity of áh Based Equation (20),
à w
op
d(n 2)
a(1 -p)
di
d(n =)) w
Vd
Vd (ain ssw))
Vd
Vd
(ui a) w)²
w + op
a11 - p)
24
2Td
V2Td
In that case, the derivative of 1 to e
V T
ap
(d(n- =) + ww e
(din k)n
23
a(1 p)
(c(n =) -w)e
(2(n-e)-
2d
(23)
2nd
2
When E [n- n, d(1 -) + > 0 and d(n- 5) - S. thus Vel > 0, i.e., is monotonically increasing with respect to E.
Summary. From Monotonicity of AC(Swin,)) we have
AC(fwi n,E) is monotonically decreasing with respect to when (0, n).
From Monotonicity of Re, (fæ), we have
Rew (fe) is monotonically increasing with respect to wb
From Monotonicity of w we have
w
i is monotonically increasing with respect to e when 8 E In
J m).
Consequently, holds that given fw = sgn(æ1 + 2at (w > 0) and E E [7 3/4 n). lower AC n, E), which corresponds larger e, can yields a fa with better Re... (fa) under the testing-time distribution bias Ets E in. 2n). This theoretical insight theoretically characterizes the connection between adversarial certainty and robust generalization.
21

---

Published in Transactions on Machine Learning Research (10/2024)
C.2 Extension of Theorem 1 to &o Perturbations
Theorem suggests that if we decrease the cerlainly of the adversarial examples sampled from Padv Padv(e), (E), the robustness of the SVM classifier fee will increase after one-step gradient update based on the sampled adversarial examples. In this section, we generalize our theoretical analysis to the typical sctting of -oom bounded perturbations, First, we prove the following lemma to derive the adversarial data distribution with respect worst-case lo. perturbations under our problem setup.
Lemma 2. Consider the same data distribution and SVM classifiers as assumed in Theorem 1. For any w > and (x, y) sampled from μ, the distribution of worst-case adversarial example (x', y) under lo perturbations by using the distribution bias E is equivalent to the following adversarial data distribution:
z . ye and '22,'" F4441 Lid. N( ((,,-))y,1),
In other words, the adversarial data distribution is obtained by shifting all features of æ including the robust. feature in by ye. Accordingly, the robust generalization can be computed
Re(sw) = pp. d(n E) + w(1 =e) + (1- o d(n e) w(1 ++)
(24)
Vd
Vd
Proof of Lemma 2. According to the definition of adversarial robustness in Equation (4), for any (æ, v) ~ p and w > O, the worst-case adversarial example x' under eo-ppetuurattons by using the distribution bias e is defined as:
82 +
Tdt2
x' = argmax Pr (i) argmax Pr || Se
sgn
Ø1 +
* y
(25)
Maximizing the objective in Equation (25) is equivalent to perturbing in direction such that 5 #2 + +2--1 has opposite sign to the ground-truth y. In the following, going to prove the following claim: for any z such that و| 2llo0 S,
Pr y
81
32 + + Ta+1
<0 Pr y 21 sh +..... +.....o x'441 +....
< <0
(26)
w
provided that æ' is defined as 2²j Fÿ y for all j f1....,, d 1
First, we have || alloc = which means that of is feasible ad adversarial example. In addition, we know that for any feasible é\
82
+ Td+1
y
b1
+
w
12 + .... Td41
T2 + ... .
Ta+t
(
#1 + 12 + ... + Ed+1
2 y.
f1 +
T1 +
W
W
02 + Id+1
2 y
f1 +
1
e
w
32
y.
1 +
Based on the above incqualitios, we immediately know that our claim specified in Equation (26) holds for any æ. Based on the distribution of robust feature and non-robust features T ld+ and some simple algebra to compute the robust generalization (with respect to æ'), we complete the proof of Lemma 7
Now we lay out the extension of Theorem to Eoo perturbations and its proof.
Theorem 3. Consider the aforementioned data distribution j and robust classification task. Let Ete E (n, 1) and fw be arbitrary SVM classifier with +164 (d-dd)) dn 0. For any
22

---

Published in Transactions on Machine Learning Research (10/2024)
E (0. w+d' w+y - A where Ae E (0, w+d',, AC, (fwi H. padv(E)), the adversarial certainty fws is monotonically decreasing with respect to Suppose we conduct one-step gradient update on w using adversarial examples sampled from (E): à w + VuRo(fwi (adv(E)), where 0 stands for the learning rate. Then, Ro(fai Madv (adv( the robust generalization performance of fa, also increases as e increases.
Proof of Theorem 3. Similar to the proof of Theorem 1, we start by showing the monotonicity of adversarial certainty.
Monotonicity of AC( As defined in Equation (17), the adversarial certainty in this binary classification task is
AC(Suin,)) 8 2R (Rww) 2R (Sw) + 1].
And accordingly,
VcAC((mn))) = 2 Re(fm) 2 Re(M)).
Similarly, to study the AC( monotonicity, it is necessary to discuss the sign of (Sw) (fw). According to Equation 24, the derivative of (=( to is
1"
and
2
(eRe((m) p(d +w)
d(1 E) + w(1 (o(l=))
Vd
Vd
(27)
(1-p)(d pd + w)
- (d(n - E) w(1 + +E))
vd
Vd
As (1 (1- < 1, d 0, > and Vu, (u) 0, it yields (e < 0. That is, Re (fer is
monotonically decreasing with respect to
dn When == wtd' we have d(n - c) + w(1 E) w and d(n - =) w(1 + +c) w. Thus,
Re( =p. Φ
w
+ (1 p) 1 (-- w
Vd
Va
W
W
p +
+ (1 p) (1 p) ①
(2p 1) &
W
+ +(1 -) p)
Vdl
> (2p 1) 2 (1 p) ~
In that case, Ve E (0, dg it yields Re(fw) - 1/2 > 0. Consequently,
VcAC(fu:n,e) 1 2.1 [R(((f)) 21 VeRelfw) <0. 2
that is, AC(fw:n,e) is monotonically decreasing with respect to when E E (0, d
Monotonicity of Re.. (fa). Similarly, the robust error &c(fu) 1 Re(fw) involved as the robust loss to optimize W. In that case, the derivative of (Fwr) to is
p(1 - E)
d(n E) + w(1 - ) 0
Vd
Vd
(28)
(1 - p) +8)
d(1 - E) w(1 + t0)) $
Vd
dd
23

---

Published in Transactions on Machine Learning Research (10/2024)
Accordingly, the optimized parameters ve by a step size of a 0 is derived as
i- (wEc(fw Based on di and Ete, the robust gencralization is
ap(1 - 5) o d(n E) + w(1 ) a(1 p)(1 + +&)
of ( (d(n-))-w( 5 vd
ww(1 t
(29)
vd
vd
vd
Rec (fa) p- 1 d(n Esa) + w011 - Exx)
d(1) 8tc) - =w01 + Exn)
+ (1 - p) 1
(30)
vd
Vd
Accordingly, the derivative to á is
RæRen (fe) = p(1 -tte)
1 (dll - Ete) + 20(1 - -Ete)
Vd
Vd
(31)
(1- p)(1 + Ete) o (dlnn Ele) - :0(1 + Ete)
Vd
Vd
As 17 S Ete I (2p 1), it yields 0 < (1-N)((+Ete)
p((-swe) and 0 < of din (dlo-et))-
va
of ( ), thus V Re.. (Fx) > 0. That is, Res (fa) is monotonically increasing with respect to i.
Monotonicity of a. Based on Equation 29,
①- = w ap(1- 6)
d(n 5) + w(1 -)) a(1 - (1++)) +e) of (((nn Vd
8) w(1 + )
Vd
Vd
Vd
In that case, the derivative of s to is
Veu ap
d(n -) 5) + w(1 =)
d(n E) + w(1 - -E) )
+ (1 (1- Vedl
Vd
Vd
Vd
a(1
p)
[6( d(n s)- - w(1 =)-w(l + + +(1+ V. d(1 8) w(1 (EE)
Vd
Vd
vd
ap
(1 - e) w) (d(n E) + w011 - +0)]
d(n E) + w -66) E)
1+
.0
Vd
d
Vd
(32)
a(1 - p)
+ (1+ + (d(n e) - w(1 + 8))
d(n 8) w(l +9-
Vd
d
vd
dvd [(w + d) (w + d)) ((w d)s - (w + dn)) - d 66( d(n - -E) + w(1 -
ap
Vd
+ a(1 -p)
((w + d)E + (w + d)) ((w + d)c + (w - dn)) - d) o d(n - -w(1 + (w(++t) +8)
5)
dvd
Vd
As dn < v(d=dn))+1Gd-(ddd (d- < w, it yields ((w + d)e + (w + d)) (ww + d)e + (w - dn)) - d > 0. As
2
0 <
w+dd wid
<
w-d
w-d and ((w + d)s (w + d)) ((w + d)= (w + dr))
d, it yields A E (0, wtd
such that ((w + d)e (w + d)) (ww + d)c (w + dn))
d. In that case, it holds that Ve E
widy =
(0, w+ w+4
AE] Vew > 0, that is, à is monotonically increasing with respect to e when 8 E (0,
w+dy
Ae].
wtd
Summary, From Monotonicity of AC(fui 2), we have
dn
AC(swin,)) is monotonically decreasing with respect to e when e E (0,
w +
From Monotonicity of Rec (fre), we have
Rec (fo) is monotonically increasing with respect to in
24

---

Published in Transactions on Machine Learning Research (10/2024)
From Monotonicity of úü have
w + dn w + d
i is monotonically increasing with respect to e when e E (0,
-aj.
Consequently, it holds that given fw sgn(X1 + #2+ t2am- (0 dn (d+dy)) +16d-(d-d)) (d-da) < w) and dy w+-dy E (0, min( w+dd wtd - As)], where As E (0, w+d is
w lower AC(Swi (, which corresponds to a larger e, can yields a fa with better Re, (fØ) under the testing-ting time distribution bias Ete E [17, 2p 1].
2
w+dg)
Significance Test for the Improvements of DAC on AWP and Consistency
As discussed in Section 6.2, our DAC method can only bring slight improvements in robust generalization for AWP and Consistency. Although repeated trials have suggested that the improvements are the consequence of DAC (see Table 6), it is helpful to provide some statistical support. Therefore, in this section, we conduct a 1-test to measure the statistical significance of our DAC method. Specifically, we first make a null hypothesis, i.e,,
Ho Our DAC method does not improve the robust generalization of AWP and Consistency.
We then collect the robust generalization under AutoAtack without and with DAC from Table 6, which are separately denoted two samples X1 and X2- This decision is because AutoAttack is more powerful and comprehensive than other adversarial attacks used in our evaluation, and AutoAttack is now the default metric for the leaderboard of adversarial defenses." 2 In that case, the null hyperthesis Ho can be informally understood as X1 2 X2. Next, we calculate the mean of X1 and X2, which can be formulated as:
B1
1
B2
X1
X1s, and X₂
X26
n1
72
where n, and n2 are the size of X, and X2 this case. Subsequently, the standard deviation of X, and X2 can be calculated as:
(X x1)², and $2
1
X22 X₂)²
$1
112
i=1
Accordingly, the pooled standard deviation of the two samples is represented by $1 and $2, i.c.,
(n) 1)s + (n2 1)s3
8p
n1 n2
where ni n1 2 2 is the total number of degrees of freedom. Given X1, Xo and Sp; we have i-statistic
X
X₂
t
Sp
R3
In this case, we have the t-statistic 2.141 and total number of degrees of freedom nn + n22 - 2 22. By comparing to the t-Table, our t-statistic is larger than the element of t.975 2.074, i.e., we have 95% confidence to reject the null hypothesis Ho.3 3 In other words, our DAC method can bring statistically significant improvements to AWP and Consistency.
E
Complete Results
Due to the space limit, we present the full results of Table 2 and Table 3 in Table 7 and Table 8, respectively.
https: http:///wwwiien www.wwikiiiii..oom oocilaarrg/wiaiii http:/wkki/Stuuent//eee www.kpppia..og/iiiii///tteee/e
t-test
https://rohuuaae htane////nnttbbechggiiie hwt.hhhh.g.giio/////
https://wwwe htpp://wwwsjjs.eduu htt////www..dd//llllt////a/aaaa/ /tattty//eeetaaa///aaaaP/e//aaae
25

---

Published in Transactions on Machine Learning Research (10/2024)
Table 7: Testing-tine time adversarial robustuess (%) of AT with/withouu DAC/DAC Reg under loo perturabtions across different model architectures and benchmark datasets.
Dataset
Architecture
Method
Clean
PGD-20
PGD-100
CW
AutoAttack
AT
89.63 (88.64)
42.25 (51.00)
41.37 (50.30)
42.84 (48.19)
39.52 (66002)
PRNIS
+ DAC
90.58 (89.63)
45.86 (54.42)
43.92 (53.78)
43.75 (50.15)
40.66 (48.23)
+ DAC Reg
90.65 (90.21)
45.30 (53.06)
43.77 (52.28)
43.66 (49.64)
41.10 (47.39)
SVHN
AT
91.51 (89.72)
46.81 (53.43)
41.94 (52.77)
45.76 (50.43)
41.71 (49.50)
WRN34
+ DAC
91.26 (91.83)
60.42 (67.95)
56,71 (64.85)
56.98 (65.09)
42.33 (50.42)
+ DAC Reg
91.76 (92.13)
62.19 (65.96)
59.54 (63.68)
60.05 (63.87)
42.46 (49.95)
AT
82.88 (82.68)
41.51 (49:23)
40.96 (48.92)
41.61 (48.07)
39.66 (45.71)
PRN18
DAC
84.64 (83.55)
45.55 (52.20)
44.94 (51.87)
44.55 (50.05)
42.78 (48.20)
DAC_Res
83.78 (83.54)
45.39 (50.86)
44.87 (50.49)
44.18 (48.96)
42.41 (47.022
CIFAR-10
AT
86.47
(85.86)
47. 25 (55.31)
16.73 (55.00)
47.85 (54.04)
45.84 (51.94)
WRN34
+ DAC
86.48 (85. 10)
52.02 (57.93)
51.69 (57.68)
51.51 (54.98)
49.75 (53.33)
DAC_Reg Reg
85.69 (76.89)
48.81 (48.91)
47.54 (48.86)
47.55 (45.98)
44.24 (44.99)
AT
54.58
(53.64)
20.29 (27.80)
20.00 (27.66)
20.18 (25.40)
18.52 (23.45)
PRN18
+ DAC
54.85 (55.01)
22.46 (27.73)
22.19 (27.48)
21.11 (25.37)
19.09 (23.95)
+ DAC_ Reg
54.67 (58.11)
21.78 (28.86)
21.50 (28.70)
20.56 (26.00)
19.29 (23.40)
C00
AT
57.23 (54.45)
25.64 (30.30)
25.38 (29.97)
24.09 (27.57)
22.76 (25.46)
WRN34
DAC
58.15 (58.04) 04)
26.08 (31.55)
25.89 (31.43)
24.77 (29.19)
23.66 (27.08)
DAC_ Reg
57.57 (58.34)
24.46 (30.97)
24.13 (30.89)
24.04 (28.92)
22.68 (26.71)
Table 8: Testing-ting time adversarial robustness (%) of AT, TRADES and MART with/without DAC on SVHN, CIFAR-10 and CIFAR-100 under for perturbations.
Dataset
Method
Clean
PGD-20
PGD-100
CW 00
AutoAttack
AT
89.63 (88.64)
42.25 (51.00)
41.37 (50.30)
42.84 (48.19)
39.52 (46.02)
+ DAC
90.58 (89.63)
45.86 (54.42)
43.92 (53.78)
43.75 (50.15)
40.68 (48.23)
+ DAC_
90.65 (90.21)
45.39 (53.06)
43.77 (52.28)
43.66 (49.64)
41.10 (47.39)
TRADES
89.12 (87.75)
51.50 (55.19)
50.69 (54.50)
45.50 (50.32)
45.02 (48.69)
SVHN
DDC
90.24 (89.59)
52.24 (57.09)
51.14 (56.39)
46.34 (52.22)
46.20 (50.52)
+ DAC Reg
90.03 (89.75)
51.78 (56.10)
50.92 (54.83)
45.86 (51.35)
45.30 (49.06)
MART
89.68 (84.48)
49.07 (52.30)
48.30 (52.22)
45.48 (48.04)
44.54 (47.38)
+ DAC
88.90 (84.64)
51.04 (53.64)
50.91 (52.70)
46.94 (49.96)
46.18 (48.50)
+ DAC Reg
90.18 (88.47)
50.94 (52.94)
49.87 (52.46)
46.32 (49.18)
45.86 (17.73)
AT
82.88 (82.68) 68)
41.51 (49.23)
40.96 (48.92)
41.61 (48.07)
39.66 (45.71)
DAC
84.64 (83.55)
45.55 (52.20)
44.94 (51.87)
44.55 (50.05)
42.78 (48.20)
+ DAC Reg
83.78 (83.54)
45.39 (50.86)
44.87 (50.49)
44.18 (48.96)
42.41 (47.02)
TRADES
82.10 (81.33)
47.44 (51.65)
46.95 (51.42)
46.64 (49.18)
44.99 (48.06)
CIFAR-10
+ DAC
83.18 (82.80)
49.32 (52.90)
48.81 (52.67)
48.30 (50.11)
46.40 (48.96)
+ DAC Reg
82.97 (82.37)
47.87 (53.33)
47.33 (8188)
46.80 (49.68)
45.07 (18.70)
MART
80.85 (78.27)
50.23 (52.28)
49.71 (52.13)
46.88 (47.83)
44.68 (46.01)
+ DAC
81.12 (79.37)
52.38 (53.25)
52.04 (53.14)
48.97 (49.25)
47.24 (47.69)
+ DAC Reg
80.81 (79.07)
50.35 (52.71)
50.06 (52.54)
47.50 (49.32)
45.72 (17.19)
AT
54.58
(53.64)
20.29 (27.80)
20.00 (27.66)
20.18 (25.40)
18.52 (23.45)
+ DAC
54.85 (55.01)
22.46 (27.73)
22.19 (27.48)
21.11 (25.37)
19.09 (23,95)
+ DAC_ Reg
54.67 (53.11)
21.78 (28.86)
21.50 (28.70)
20.56 (26.00)
19.29 (23.40)
TRADES
55.40 (53.98)
22.40 (28.31)
22.32 (28.
21.42 (25.82)
20.55 (24.29)
CIFAR-100
DAC
56.66 (54.67)
25.54 (29.56)
25.43 (29.35)
23.32 (25.92)
22.35 (24.87)
+ DAC Reg
54.36 (53.86)
23.16 (27.96)
23.13 (27.88)
22.34 (24.35)
21.08 (23.43)
MART
55.73 (52.48)
24.18 (27.17)
24.00 (27.12)
22.41 (24.89)
21.56 (23.06)
+ DAC
55.94 (54.23)
24.81 (28.23)
24.65 (28.13)
23.53 (25.19)
22.06 (24.00)
+ DAC Reg
55.52 (52.77)
24.33 (27.40)
24.21 (23.41)
22.83 (25.04)
21.73 (23.28)
26

---

Published in Transactions on Machine Learning Research (10/2024)
F Additional Evaluations
In this section, provide additional experimental results better to comprehend work, including the influence of on DAC and training efficiency.
F.1 Influence of on DAC
Table 9: Testing- Desting-time robust generalization (RG%) on the last and best epochs, where the training time is set to 8/255 while the testing one varies in (4/255, 8/ 8/255, 12/255}.
Cr 8/255
Cte 4/255
<tc 8/255
Cte 12/255
RG /ast///rbbet RGGestt
66.03/70.09
45.55/52.20
29.91/34.88
Table 10: Testing-time robust generalization (RG%) the last and best epochs, where the testing-time is set to 8/255 while the training one varies in {4/255, 8/255, 12/255}.
Éte 8/255
Ctr 4/255
ftr 8/255
Ctr 12/255
|atast/ RRGbest
41.49/45.51
45.55/52.20
47.96/53.32
In the previous evaluation, our work focuses on the setting of c <= 8/255 in both the training and testing time for -nom perturbations, which widely used in this field the adversarial strength. However, it is interesting to investigate the influence of on our DAC method. Thus, separately fixed the training- and testing-time to 8/255, and vary the other one in {4/255, 8/255, 12/255), as shown in Tables 9 and 10. Specifically, Table 9 depicts that the model of fixed training-time etr is more vulnerable to larger testing-lime ête, on both the last and best epochs. On the other hand, Table 10, shows that a stronger adversarial ability in the training time can derive a better robustness when facing the same testing- time Ete- Consequently, these results suggest that model robustness will benefit from a larger Ety and smaller Ete-
usually associated with the assumption of
F.2 Training Efficiency
Table 11: Testing-time robust generalization (RG %) on the last and best epochs, where DAC, indicates that decrcasing adversarial certainty starts from the f-th epoch.
RG (%)
Last
Best
DAC1
45.55
52.20
DAC101
45.84
50.49
DAC1sI
45.68
49.85
In Section 6.4, proposed DAC Reg to reduce the training cost. However, it modifies the optimization flow. Therefore, to enhance DAC's efficiency, conduct additional attempt by decreasing adversarial certainty only when robust overfitting occurs. In evaluation, start the DAC method from the 101-th and 151-th epochs, respectively, corresponding to the first and second time of lcarning rate decay. From Table F.2, we can see that an earlier application of DAC brings better "Best" robustness, while the "Last" " one is comparable. Consequently, improving training efficiency by reducing the epochs that consider DAC can only derive limited performance promotion. other words, the potential direction of developing efficient. methods might be regularizing some specific insights in a loss term, such the design of DAC DAC_Reg. Still, the effort to guarantee stability is necessary.
27

---

Published in Transactions on Machine Learning Research (10/2024)
Potential Limitation. Table 11 depicts that applying DAC earlier achieve better robustness. Thus, 1AC101 and DAC151, decreasing adversarial certainty when robust overfitting happens, cannot achieve comparable robustness on the best epoch with DAC₁ that works from scratch. That is, our method requires a warm-up instead of an immediate functionality. Consequently, our method might not be suitable for the real-time scenario, where DAC is deployed only when robust overfitting happens, which limits the flexibility of our method.
28


Reviews:
[{"review_id": "aVjTyGmAnV", "reviewer": "Reviewer_Nu3j", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nInnovative Concept: The introduction of adversarial certainty is novel and provides a way of understanding adversarial training dynamics.\n\nTheoretical Results are Provided: The paper includes theoretical analysis to support the proposed approach, adding credibility.\n\nEmpirical Validation: Experiments across different datasets and adversarial training methods (AT, TRADES, MART) demonstrate the effectiveness of the method.\n\nWeaknesses:\n\nRobustness of $\\epsilon$: One concern is how to choose $\\epsilon$ prior to training. Do you have any suggestions on selecting $\\epsilon$? Additionally, how does a change in $\\epsilon$ affect performance?\n\nThe overall flow of the paper is not very strong. It consistently mentions concepts that have not yet been introduced, which leads to confusion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper revisits the problem of robust overfitting in adversarial training and introduces the novel concept of adversarial certainty. The authors argue that model overconfidence in predicting adversarial examples during training is a key factor causing reduced robust generalization. \n\nThe paper contributes in:\n\nDefining adversarial certainty to quantify the model’s certainty in adversarial predictions.\n\nProposing a method to generate less certain adversarial examples during training, which helps in improving robust generalization.\n\nProviding theoretical results and empirical results that show decreasing adversarial certainty improves robustness across various image benchmark datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Hyperparameter tuning on $\\epsilon$. Please provide extra results on how the DAC method performs under different values of $\\epsilon$. As $\\epsilon$ directly affects the strength of the adversarial perturbations, understanding the model's performance across a range of $\\epsilon$ values could give further insight into the robustness of your method under various adversarial conditions. After this, please also provide suggestions/discussions on $\\epsilon$ choosing if the authors have.\n\nStreaming issues:\n\nIn the contribution section, it is not recommended to cite equations or figures from later sections. Please try to summarize them instead. If you must cite them, you should provide details or define the variables in the equation before doing so.\n\nImportant notation, like adversarial certainty, is not properly defined until Chapter 4, yet it is frequently mentioned after just a one-line description. Please provide at least a high-level introduction before that.\n\nSimilarly, important methods like DAC_Reg are defined in Section 6.4, but are referenced earlier in Section 6.1.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q2glmWWFc0", "reviewer": "Reviewer_XCB7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Theoretical Contribution: The introduction of adversarial certainty provides a new perspective for understanding model behavior in adversarial training, supported by theoretical analysis that suggests reducing certainty can enhance model robustness.\n- Experimental Validation: Extensive experiments across various standard datasets validate that reducing adversarial certainty improves robust generalization capabilities, particularly highlighted through comparisons with existing technologies.\n- Practicality: The proposed method can be directly applied to existing adversarial training frameworks, showing good practicality and potential for widespread adoption.\n\nWeaknesses\n- Increased Complexity: Although theoretically effective, this method may increase the complexity and computational demands of the training process, particularly in large-scale datasets and models, potentially significantly extending training times.\n- Generalization Verification: The paper focuses on image recognition tasks, and the effectiveness of this approach for other types of tasks (such as natural language processing or audio processing) remains to be further researched and verified.\n- Parameter Sensitivity: The model and methods discussed may be sensitive to hyperparameter choices, such as step size and regularization coefficients, which could impact final performance and stability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses a key issue in adversarial training—robust overfitting—and introduces a new concept, \"adversarial certainty,\" to quantify the model's confidence in its predictions. By reducing the certainty in adversarial training, i.e., increasing the variability of model outputs, the paper aims to enhance the model's robust generalization against adversarial samples. Through extensive experiments and theoretical analysis, the paper demonstrates that the proposed method effectively mitigates robust overfitting in adversarial training and showcases performance improvements on multiple standard datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It is suggested to either extend the method to additional data types and tasks (e.g., natural language processing or audio tasks) to further validate its generalization capability, or if adding experiments is difficult, provide relevant discussion on how the method might generalize to other domains.\n\n2. While improving training efficiency through experiments would be beneficial, if conducting additional experiments is challenging, discussing potential optimizations or alternatives could provide valuable insights. This can include suggestions for future work on how the method could be made more computationally efficient.\n\n3. If possible, include detailed case studies or additional experiments where the method may not perform optimally. However, if resource constraints prevent additional experimentation, offering a discussion on potential limitations based on existing results would also help to clarify the boundaries of the method's effectiveness.", "broader_impact_concerns": "No ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HLE0mt2chU", "reviewer": "Reviewer_5svJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed idea is sensible. The method, being a 2 line gradient add-on to existing adversarial training method, is simple and easy to implement.\n2. The idea and method is decently covered by theoretical analysis. Theoretically, authors first prove in the special case of linear SVM classifier, as adversarial perturbation increases, adversarial certainty decreases and (after gradient update) robust generalization increases monotonically. Authors then show 1) in terms of numerical correlation, (in Fig. 2c) adversarial certainty is negatively correlated with robust generalization when step size is sufficiently small, 2) in terms of theory, *possibly* when in feasible search region with small step size, decreasing adversarial certainty induces better robust generalization; this was based on observation but not proven. \n3. Numerical results show this simple add-on approach usually achieves some improvement in robustness, when added onto sufficiently many existing adversarial training methods, or added on top of other robust add-ons. \n\nWeaknesses:\n1.  For the main hypothesis that less adversarial certainty means better robustness, the theoretical analysis is not complete proof, but only supporting evidence (see my strength 2). There are very significant gaps from theoretical analysis to completely proving the hypothesis. Even the limited claim in [strength 2.2)], as an attempt to bridge above gap, is only observed but not proven. \n2. Some short summaries in early pages are unclear and confusing. One example is requested changes 1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Robust overfitting is widely observed in existing adversarially trained models, where robust generalization keeps decreasing after an early time. This paper claims this is because those models are too confident classifying adversarial examples. This paper introduces ‘adversarial certainty’, hypothesizes generating less certain adversarial examples improves robust generalization, and backs this up with theory and numerics. Theoretically, connection between adversarial certainty and robust generalization are studied through synthetic distributions. Based on above hypothesis, the proposed method, used as additional updates (2 gradient updates in eqn. 3) after existing robust methods, first explicitly decreases adversarial certainty (DAC) by gradient descent (GD), then use new uncertain adversarial examples to optimize robustness by GD. This approach mitigates adversarial overconfidence. More analysis shows the proposed adversarial certainty is equivalent to a certain regularization term. Numerical results on image data show proposed method combined after existing methods produces decently more robust models than existing adversarial training methods alone.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. [unclear grammar] The meaning of ‘generating less certain adversarial examples’ in [1) abstract, line 5; 2) P2, line 4; 3) P2, line 15] is unclear and confusing. This should be replaced by more sensible phrases such as ‘decreasing adversarial certainty’, or more detailed words such as [P2, line 5]’s next sentence.\n2. [P7, correlation analysis] ‘Does decreasing adversarial certainty always induce better robust generalization?’ is a crucial question, but currently only addressed mostly by heuristics. The partially theoretical claim ‘this result suggests that…’ is not sufficiently rigorous and sensible. If possible, more rigorous details are desirable.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers acknowledge the novelty of the adversarial certainty. The paper gives comprehensive theoretical analysis and empirical analysis to show the connection of adversarial certainty and robust generalization. All the reviewers acknowledge the proposed method to generate adversarial training examples with less certainty, and the correlation analysis  between adversarial certainty and robust
generalization. The experimental comparison is made on various datasets and strong baselines.
Decision: Accept (poster)