One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
Authors: Jiacheng Zhang
Benjamin I. P. Rubinstein
Jingfeng Zhang
Feng Liu
Venue: ICML 2025 poster (2025)
Keywords: adversarial defense
adversarial robustness
accuracy-robustness trade-off
Primary area: deep_learning->robustness

Abstract: *Statistical adversarial data detection* (SADD) detects whether an upcoming batch contains *adversarial examples* (AEs) by measuring the distributional discrepancies between *clean examples* (CEs) and AEs. In this paper, we explore the strength of SADD-based methods by theoretically showing that minimizing distributional discrepancy can help reduce the expected loss on AEs. Despite these advantages, SADD-based methods have a potential limitation: they discard inputs that are detected as AEs, leading to the loss of clean information within those inputs. To address this limitation, we propose a two-pronged adversarial defense method, named ***D***istributional-discrepancy-based ***A***dversarial ***D***efense (DAD). In the training phase, DAD first optimizes the test power of the *maximum mean discrepancy* (MMD) to derive MMD-OPT, which is *a stone that kills two birds*. MMD-OPT first serves as a *guiding signal* to minimize the distributional discrepancy between CEs and AEs to train a denoiser. Then, it serves as a *discriminator* to differentiate CEs and AEs during inference. Overall, in the inference stage, DAD consists of a two-pronged process: (1) directly feeding the detected CEs into the classifier, and (2) removing noise from the detected AEs by the distributional-discrepancy-based denoiser. Extensive experiments show that DAD outperforms current *state-of-the-art* (SOTA) defense methods by *simultaneously* improving clean and robust accuracy on CIFAR-10 and ImageNet-1K against adaptive white-box attacks. Codes are publicly available at: https://github.com/tmlr-group/DAD.

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
Jiacheng Zhang' Benjamin I. P. Rubinstein Jingfeng Zhang? 23 Feng Liu 13
Abstract
1. Introduction
Statistical adversarial data detection (SADD) de- tects whether an upcoming batch contains adver- sarial examples (AEs) by measuring the distri- butional discrepancies between clean examples (CEs) and AEs. In this paper, we explore the strength of SADD-basee methods by theoretically showing that minimizing distributional discrep- ancy can help reduce the expected loss on AEs. Despite these advantages, SADD- methods have a potential limitation: they discard inputs that are detected AEs, leading the loss of use- ful information within those inputs. To address this limitation, we propose a two-pronged ad- versarial defense method, named Distributional- discrepancy-based Adversarial Defense (DAD). In the training phase, DAD first optimizes the test power of the maximum mean discrepancy (MMD) to derive MMD-OPT, which is stone that kills two birds. MMD-OPT first serves as a guiding signal to minimize the distributional discrepancy between CEs and AEs to train a denoiser. Then, it serves as a discriminator to differentiate CEs and AEs during inference. Overall, in the infer- ence stage, DAD consists of two-pronged pro- cess: (1) directly feeding the detected CEs into the classifier, and (2) removing noise from the detected AEs by the cnstribuutionalldiieeppaam based denoiser. Extensive experiments show that DAD outperforms current state-of-ther aht (SOTA) defense methods by simultaneously im- proving clean and robust accuracy on CIFAR-10 10 and ImageNet- against adaptive white-box at- tacks. Codes are publicly available at: https: / //github. com / wwwrtmllrrrroo r-group/DAD.
The discovery of adversarial examples (AEs) has raised security concern for deep learning techniques in recent decades (Szegedy et al., 2014; Goodfellow et al.,, 2015). AEs are often crafted by adding imperceptible noise to clean ex- amples (CEs), which can easily mislead a well-trained deep learning model to make wrong predictions. Considering the extensive use of deep learning- -basid systems, AEs pose a significant security threat for real-world applications (Dong et al., 2019; Cao et al., 2021; Han et al., 2025). Therefore, it is imperative to develop advanced defense methods to defend against AEs (Goodfellow et al., 2015; Madry et al., 2018; Zhang et al., 2019; Wang et al., 2020; Yoon et al., 2021; Nie et al., 2022).
Recently, statistical adversarial data detection (SADD) has gained increasing attention due its effectiveness in detect- ing AEs (Gao et al., 2021: Zhang et al., 2023). Unlike other detection- methods that train detector for specific classifiers (Stutz et al., 2020; Deng et al., 2021; Pang et al., 2022b), SADD leverages statistical methods, for example, maximum mean discrepancy (MMD) (Gretton et al., 2012), to measure the discrepancies between clean and adversarial distributions. Given the fact that clean and adversarial data are from different distributions, SADD-based methods have been shown empirically effective against adversarial attacks (Gao et al., 2021; Zhang et al., 2023).
To understand the intrinsic strength of SADD-based meth- ods from a theoretical standpoint, we establish a relationship between distributional discrepancy and the expected loss on adversarial data. Our theoretical analysis demonstrates that minimizing distributional discrepancy can help reduce the expected loss on adversarial data, revealing the poten- tial value of leveraging distributional discrepancy to design more effective defense methods (see Section 2 and 3).
However, despite their effectiveness from both empirical and theoretical perspectives, detection-based methods (e.g.. SADD-based methods) have a potential limitation: they discard inputs if they are detected AEs, leading to the loss of, e.g., semantic information within those inputs. This issue is more prominent in SADD-based methods, where inputs are often processed in batches, potentially resulting
'School of Computing and Information Systems, The Univer- sity of Melbourne "The University of Auckland 'RIKEN Center for Advanced Intelligence Project (AIP). Correspondence to: Feng Liu <fengliu.ml @ gomi..com>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
in the unintended loss of some CEs along with AEs if batch contains mixture of CEs and AEs (Gao al., 2021; Zhang et al., 2023). Furthermore, in many domains, ob- taining large quantities of high-quality data is challenging due to factors such as cost, privacy concerns, or the rarity of specific data, for example, obtaining medical images for rare diseases is challenging (Litjens et al., 2017). As a result, all possible samples with useful information are critical in these data-scarce domains (Gandhar et al., 2024). Therefore, given the effectiveness of SADD-based methods, the above- mentioned challenges naturally lead us to pose the following question: Can we design an adversarial defense method that leverages the effectiveness of SADD- basedd methods, while a the same time, preserves all the data before feeding them into a classifier?
pared to the most successful denoiser-based methods
(known as adversarial purification) that rely density estimation (e.g., Nie et al. (2022) and Lee & Kim (2023)), learning distributional discrepancies between AEs and CEs is a simpler and more feasible task, especially on large-scale datasets such as ImageNet- IK (Section 5.1).
Through extensive evaluations on benchmark image datasets such as CIFAR-I0 (Krizhevsky al., 2009) and Imagenet- IK (Deng et al., 2009), we demonstrate the effectiveness of DAD in Section 5. Compared to current state-of-the- art (SOTA) adversarial defense frameworks (i.c., adversar- ial training and adversarial purification), DAD can notably improve clean and robust accuracy simultaneously against well-designed adaptive white-box attacks (see Section 5. Furthermore, experiments show that DAD can generalize well against unseen transfer attacks (see Section 5.2).
The answer to this question is affirmative. Motivated by our theoretical analysis, we propose a two-pronged ad- aetibaattioon--diipppaae.. based Adversarial Defense (DAD). Specifically, we leverage an ad- vanced MMD statistic (named MMD-OPT) in our pipeline, which is obtained by maximizing the testing power of MMD (see Algorithm 1). MMD-OPT, as one stone, essentially kills two birds, i.e., serving two roles in DAD: in the training phase of the denoiser (see Algorithm 2), it acts as a guiding signal to minimize the distributional discrepancy between AES and CEs. Then, by simultaneously minimizing the cross-entropy loss, we aim to train a denoiser that can min- imize the distributional discrepancy towards the direction of making the classification correct. In the inference phase (see Section 4.3), MMD-OPT acts as a discriminator to dis- tinguish between CEs and AEs. Then, our method applies two-pronged process: (1) directly feeding the detected CEs into the classifier, and (2) removing noise from the detected AES by the well-trained denoiser. We provide a visual illustration in Figure 1.
versarial defense called
2. Problem Setting and Assumptions
In this section, we discuss the problem setting for the ad- versarial classification in detail. We formalize our problem for K-class classification as follows. We define a domain as a pair consisting of distribution 'D on inputs X' and a labelling function f X + {I, (L.... K}. Specifically, we consider a clean domain and an adversarial domain. The clean domain is denoted by (De.fe), and the adversarial domain is denoted by (DA, We define a hypothesis as a function h X' + {I,..,, K} from the hypothesis space H. The probability according to the distribution D that a hypothesis h disagrees with a labelling function f (which can also be hypothesis) is the risk:
R( f. = EX<D (((hxx).((x))
where [()()), (x)) is a loss function that measures the disagreement between l((x) and f(x). We consider the clean risk of hypothesis as R(h, fc, De). and the adversar- ial risk as R(h, fA: A)) In our problem, adversarial data are generated based the given clean data. Therefore, De is fixed and we use D to represent a set of valid adversarial distributions such that all possible DA e ID.
The success of DAD in adversarial classification takes root in the following aspects:
Statistical principle. Minimizing distributional discrep- ancies has been proven significant in controlling the ex- pected loss on AEs. Thus, our new defense is built upon a solid theoretical foundation.
Assumption 1. For any valid adversarial attack, adversarial data are generated by adding an -normmbounddd impercep- tible perturbation e' to the given clean data without changing its semantic meaning. Assume valid ground-truth labelling function fA exists, SA satisfies the following property:
One stone, two birds. DAD combines the strengths of SADD-based and denoiser-based methods while ad- dressing their limitations: SADD-based methods will discard the useful information within AEs. In contrast, denoiser-based methods cannot distinguish AES and CEs beforehand, which often results in a drop in clean accu- racy. Our method, on the other hand, separates CEs and AEs in the inference phase, thereby keeping the accuracy for CEs nearly unaffected. At the same time, AEs can be properly handled by the denoiser.
Vc' s.t. ||<'|p S, fa(x + c' = fa(x),
where 6 is the maximum allowed perturbation budget, and I|p is the threat model's tp norm.
A ssumption 2. Attacks in the adversarial domain will not change the labelling from the clean ground truth, i.c., math- ematically:
Discrimination is easier than data generation. Com-
Ve' s.t. |clll S.C, sa(x+ ()x++)== fe(x),
2

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
where is the maximum allowed perturbation budget.
Normally, because an attacker aims to attack the well-trained classifier on clean data (i.e., ideally the clean risk is mini- mized), according to Theorem 1, we have
3. Motivation from Theoretical Justification
In this section, we study the relationship between adversarial risk and distributional discrepancy, aiming to shed some light on designing effective adversarial defense methods.
fa. & fc, + dD (I)
Since h2. fc and De are fixed, R(h fc. Dc) is possibly a small constant (according to Definition 2). In our problem, the objective of an attacker can be considered finding an optimal DA E I that maximizes R(ha, fA. DA). Now, as- sume we have a detector that leverages the distributional dis- crepancies to identify AEs. Then, to break the defense, the attacker must generate AEs that could minimize the distribu- tional discrepancies between CEs and AEs (i.e., to mislead the detector to identify AEs as CEs). However, according to Eq. (1), reducing the distributional discrepancy d1 (De, DA) can help reduce adversarial risk R(hh. A. Da). which vi- olates the objective of adversarial attacks. This intriguing phenomenon helps explain why SADD based methods are effective against adaptive attacks in practice and inspires the design of our proposed method in this paper (see Section 4),
3.1. Relation between labelling functions
We first reveal a simple yet important relation between la- belling functions of CEs and AEs.
Corollary 1. If Assumptions I and 2 both hold, then we have:
X [c(x)=[A(x))
=
Remark 1. Assumptions 1 and 2 are more like inherent truths here, as attacks should only generate valid examples that abide by the original label (Bartoldson et al., 2024). Therefore, compared to the setting of common domain adaptation problems (Ben-David et al., 2006; 2010), the ground-trui labelling functions for the clean and adversar- ial domains are equal in our problem setting.
Comparison with previous studies. Previous studies have attempted to use distributional discrepancy in adversarial defense. For example, at the early stage of adversarial train- ing (AT), Song et al. (2019) propose to treat adversarial attacks as a domain adaptation problem. However, to the best of our knowledge, the relationship between adversar- ial risk and distributional discrepancy has not been well investigated yet from a theoretical perspective. In previ- ous domain adaptation literature, the upper bound of the risk on the target domain is always bounded by one extra constant (Mansour et al., 2009; Ben-David et al., 2010), e.g., R(hh- fA, DA) - R(hG, Dc) + d1 (De, DA) + C. This constant C may prevent decreasing the risk on the target domain from minimizing the distributional discrep- ancy between the source domain and the target domain. By contrast, we find that adversarial classification is special domain adaptation problem where the ground truth labelling functions are equivalent for both source and target domain. Based on this, we derive an upper bound without any extra constant, i.e., distributional discrepancy minimization can directly reduce the expected loss on adversarial domain.
3.2. Theoretical justifications
For simplicity, we analyze our problem for binary clas- sification, i.e., a labelling function f is simplified to f X + {0, 1) and hypothesis h H. is simplified to h X -> 10, 1}. The loss function is simplified to 0-1 loss (i.e., ((((),, (x)) = lh(x) fx)).. Otherwise, other settings (c.g., the definition of risks) are the same as described in Section
Definition 1. For simplicity, we use T.1 divergence, one of the most distinguishable metrics, natural measure of distributional discrepancies between two distributions:
di (D, D') = 2 sup Pr
Pr - Br]
BEB
D
where B is the set of measurable subsets under D and D'.
Theorem For a hypothesis E H and a distribution DA E D:
R(h, & R(h, Dc) d (De, (DDeDA)). DA)-
The proof of Theorem 1 can be found in Appendix A. Based on this theorem, we can easily find that distributional dis- crepancy is very important in adversarial defense.
4. A New Framework: Distributional Discrepancy-based Adversarial Defense
Motivated by our theoretical analysis above, we pro- pose two-pronged adversarial defense framework called Distributional discrepancy-
Significance of distributional discrepancy to adversarial defense. We first give the definition of the well-trained classifier h in the adversarial attack scenarios.
Adversarial Defense (DAD) in this section. We first introduce the concepts of maximum mean discrepancy (MMD). This will be followed by a detailed discussion of the training and inference process of DAD which is illustrated in Figure 1. Detailed description of mathematical notations are in Appendix B.
The optimal hypothesis that minimizes the clean risk is defined as:
Definition 2.
ho arg min R(h, fc, Dc).
REH
3

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
Training
Inference
Clean Data
Test Data
Parameters trained
Parameters are frozen
MMD-OPT
Maximize
Detected as clean
Adversarial Data
Detected adversarial
Discriminator Based on MMD-OPT
Train MMD-OPT
Gaussian Noise
Train Denoiser
Adversarial Pata
Denoiser
Denoised
MMD-OPT
MMD-OFT
Denoiser
Classifier
Minimize
Gaussian Noise
Classifier
Cross-enropy
Cat
Figure 1. The illustration of cuurrbattioal-ddseeeaaeeeyyeeeee maximum mean discreponcy (MMD) to derive MMD-OPT, which is stone that kills two birds. MMD-OPT first serves as a guiding signal minimize the distributional discrepancy between CEs and AEs discriminator to differentiate CEs and AEs during inference. Then, our method applies two two-pronged process: (1) directly feeding the detected CEs into classifier, (2) removing noise from detected AEs by the well-trained denoiser.
Adversarial Defense (DAD). DAD first optimizes the test power of the
denoiser. In the inference stage, also serves as
4.1. Preliminary
For characteristic kernels, μi = μiQ implies IP = Q, and thus, MMD(P, Q; H.)) 0 if and only if P = Q. In practice, we the estimator from recent work that can effectively measure the discrepancies between AEs and CEs (Gao et al., 2021), which is defined
Maximum mean discrepancy. The problem using L divergence in practice that it does not have unbiased estimators. This because the supremum can hardly be approximated by finite samples. Hence, in practice, it is challenging to estimate L' divergence accurately, especially in nigh-dimensional settings, where the bias and variance of the estimation can become significant. Therefore, in this paper, we use MMD to measure the distributional discrep- ancies between AEs and CEs. MMD has unbiased estima- tors and can effectively distinguish the difference between two distributions using small batches of data (Liu et al., 2020; 2021a; Zhang et al., 2023). Following Gretton al. (2012), let X' C Rdd denote a separable metric space, and let P and Q represent Borel probability measures defined on X. Given two sets of IID observations Sx {x( and Sz = (2) 1 sampled from distributions and Q respectively, kernel-based MMD (Borgwardt al., 2006) quantifies the discrepancy between these two distributions:
MMD k2ikw)= n(n - 1)
Σ Hij. iti
(2)
where Hij ki (x; Xi) + kn 2n) kew (Zi, x;), and (x, z) is defined as:
kiw (x;, 2ÿ)
(x. z) + 30 g(x,z), lx, z) = (1-Bu)s (1 Bu)
(3)
where So E (0, and ((x, z), i.c., the Gaussian kernel with bandwidth ag. are two important components ensuring that Ai (x, 2) serves as a characteristic kernel (Liu et al., 2020). Additionally, * (x, 2) represents a deep kernel function designed to measure the similarity between x and by utilizing semantic features extracted via the second last layer in hi (i.e., well-trained classifier CEs), In practice, $5 (x, is well-trained feature extractor (e.g.. classifier without the last layer).
(MD(P, Q H ||pe pq Ila
/ ( X + E (. (2) 2E[A(X, Z)),
4.2. Training Process of DAD
where k: X X + the kernel of reproducing kernel Hilbert space HA. μi = Elk(- x)] and μr := ( 2)] are the kernel mean embeddings of and Q respectively.
In this section, we discuss the training process of DAD, which includes optimizing MMD and training the denoiser.

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Diserepancy
Algorithm 1 Optimizing MMD (Liu et al., 2020).
Algorithm 2 Training the denoiser.
1: Input: clean data St adversarial data " learning rate
1: Input: clean data-label pairs (St y)")), optimized char- acteristic kernel k. by Algorithm 1, pre-trained classifier ha- denoiser 9 with parameters O. learning rate v,. epoch T:; Initialize /l 0; < 0.25; a 10 2:
17. epoch T'
2: Initialize w + wui 1
3: for epoch = I,... do
4: St f- minibatch from Stain.
3: for epoch = 1... T do
5: S'A +- minibatch from S
4: (Sc, Y2) + minibatch from (S ")"
6:
kw kernel function with parameters w using Eq. (3);
5: S'A < AEs generated from (5". Yč);
7:
M(w) + MMD S'a kw) using Eq. (2);
generate Gaussian noise:
(8),,
8:
Vi(W) + xx (Sc, S'As kw) using Eq. (5);
Swise f- S'x + n:;
9:
JA(W) < /w)/ / / VV (w) using Eq. (4);
8: Compute MMD-OPT(SC: (DDD-OPTSS''. (o (S(we))))) by Eq. (6);
9: Update 0 by Eq. (7);
10: w + w + 7f A (w);
10: end for
11:
end for
11: Output: denoiser g with well-trained parameters 0*
12:
Output: K
generated from clean data-label pairs (Sc, Yc), resulting in noise-injected AEs:
We provide detailed algorithmic descriptions for the training process of DAD in Algorithms and 2.
Snoise S + n.
optimized MMD. Following Liu et al. (2020), the test power of MMD can be maximized by maximizing the following objective (i.e., optimize kw)::
One stone:
The design of injecting Gaussian noise is inspired by previ- ous works showing that applying denoised smoothing to a denoiser-classifer pipeline can provide certified robustness (Salman al., 2020b; Carlini al., 2023). Following Lin et al. (2024), we set 0 and 0.25 by default. Then, we can obtain denoised samples Sdenoised by feeding Smoise to a denoiser g with parameters O:
((P:qsk)) = MMD3 Q: kw)/o
(P.) == 44E - and H12 13 refer to the HEj in Section 4.1. However, (IP Q: ) can- not be directly optimized because MMD2 (P, Q: lw) and (P,, Qi kw) depend on P and Q that are unknown. There- fore, instead, we can optimize an estimator of [IP, kw):
Sdemoised = go (Suisse).
Ideally, Saenoised should perform in the same way as its clean counterpart Sc. To achieve this, motivated by our theo- retical analysis in Section 3, the optimized parameters 0* are obtained by minimizing the distributional discrepancy towards the direction of making the classification correct, i.e., minimize MMD-OPT and the cross-entropy loss Lee simultancously:
(x kkikw) 2 == (Se, SA: k/ok))/0x(( SA;
(4)
Here Se are clean samples, SA are adversarial samples, o3 a regularized estimator of a2
go- = arg min MMD-OPT(Se,, go (Smoise)
n Σ
Hij
Hij
+ X
(5)
3
n
+ ax Lee ( (go Yc);
iml
where 0 is regularization term (10-2 2 by default) and he is the pre-trained classifier.
where is small constant avoid O division (here we assume to obtain the asymptotic distribution of the MMD estimator).
4.3. Inference Process of DAD
We can obtain optimized le (we denote it as ky) by maxi- mizing Eq. (4) on the training set. Then, we define MMD- OPT as the MMD estimator with the optimized kernel k
In this section, we demonstrate the two-pronged inference process of DAD in detail.
MMD-OPT(Sxx S'y) (S' kD),
Second bird: discriminator based MMD-OPT. In the inference phase, we define a batch of clean validation data as Sy and the test data as ST. In practice, Sy is extracted from the training data and is completely inaccessible during the training. Then Sy serves as a reference to measure the distributional discrepancy. According to Eq, (6), the distributional discrepancies between Sy and ST can
(6)
where S'x and S22 can be any two batches of samples from either the clean or the adversarial domain.
First bird: MMD-OPT-basee denoiser: In this paper, we use DUNET (Liao et al., 2018) as our denoising model. To train the denoiser, we first randomly generate noise n from a Gaussian distribution N(u, o²) and add n to SA that are
MMD-OPT(Ss, ST) MMD ST: KG). (8)
5

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
Then, incorporating a threshold and MMD-OPP(Sy,, ST), can discriminate batch containing sufficient AEs.
Algorithm Adaptive white-box PGD+EOT attack for DAD.
1: Input: clean data-label pairs (Se, Yc), optimized character- istic kernel k by Algorithm 1, pre-trained classifier hě. de- noiser g with parameters 6, maximum allowed perturhation 6,
The two-pronged inference process. Based on the denoiser and the discriminator based on MMD-OPT, the two-pronged inference process is demonstrated below.
stop size PGD iteration T. EOT iteration K;
2: Initialize adversarial data SA <- Sc: P t- 0; a <- 0.25;
If MMD-OPT(Ss, ST) in Eq. (8) is less than a threshold
a 4- 10 2 0.05;
t, i..., MMD-OPT(SV, ST) < t, then Sy will be treated
3: for PGD iteration 1...., T do
as CEs and directly fed into the classifier. Then the output
4: Initialize gradients over EOT GEor + 0:
will be (ST), where li is a well-trained classifier; Otherwise, ST will be treated AEs and denoised by
5:
Compute MMD-OPT(Sc, S.A) by Eq. (6);
6:
for EOT iteration 1,... K do
7:
if MMD-OPT(SC,
< then GEOT 4- 9Eor + Vsa 5s (MMD-OPT(SE;,5) SA) + a. L. (c))
the well-trained denoiser gø-. Then, the output will be (e(go-(S7))
8:
9:
elsc
5. Experiments
10:
Generate Gaussian noise: n~ N(u,
11:
Suise +- SA +
We briefly introduce the experiment settings here and pro- vide more detailed version in Appendix C.
12:
GEoT + GEDT + Va (MMD-OPT(Se, SA) + a Lee (o Yc)):
13:
end if
Dataset and target models. We evaluate DAD on two benchmark datasets with different scales, i.e. CIFAR-10 (Krizhevsky et al., 2009) and ImageNet-II (Deng et al., 2009). For the target models, we mainly use three archi- tectures with different capacities: ResNet (He et al., 2016), WideResNet (Zagoruyko & Komodakis, 2016) and Swin Transformer (Liu et al., 2021b).
14:
end for
15: GEOT + 1 GEOT:
16: Update SA <- (Se) (Sa + sign(Gror));
17:
end for
18:
Output: SA
box AutoAttack with the same configurations mentioned above. Notably, we find that our method achieves the worst- case robust accuracy adaptive white-box PGD+EOT at- tack. Therefore, we report the robust accuracy of our method on adaptive white-box PGD+EOT attack for Table and 2. On CIFAR-10, the maximum allowed perturbation budget for fo norm-based attacks and é é--norm-base based attacks is set to 8/255 and 0.5, respectively. While on ImageNet- 1K, we set E = 4/255 for -x--norm-baeed attacks.
Baseline settings. DAD is a two-pronged adversarial de- fense method, which is different from most existing defense methods. In terms of the pipeline structure, MagNet (Meng & Chen, 2017) is the only similar defense method to ours, which also contains a two-pronged process. However, Mag- Net is now considered outdated, making it unfair for DAD to compare with it. Therefore, to make the comparison as fair as possible, we follow a recent study on robust evalua- tion (Lee& & Kim, 2023) to compare our method with SOTA adversarial training (AT) methods in RobustBench (Croce et al., 2020) and adversarial purification (AP) methods se- lected by Lee & Kim (2023).
Implementation details of DAD. To avoid the evaluation bias caused by seeing similar attacks beforehand during training, we train both the MMD-OPT and the denoiser us- ing norm MMA attack (Gao et al., 2022), which differs significantly from PGD+EOT and AutoAttack. Then, we use unseen attacks to evaluate DAD. For optimizing the MMD, following Gao et al. (2021), we set the learning rate to be 2 x 10-4 and the epoch number to be 200. For train- ing the denoiser, we set the epoch number to be 60. The initial learning rate is set to 1 x 10 for both datasets and is divided by 10 at the 45th and 60th epoch to avoid robust overfitting (Rice et al., 2020). We provide more detailed implementation details in Appendix C.
Evaluation settings. We mainly use PGD+EOT (Atha- Iye et al., 2018b) and AutoAttack (Croce& & Hein, 2020a) to compare our method with different baseline methods. Specifically, following Lee & Kim (2023), we evaluate AP methods on the PGD+EOT attack with 200 PGD iterations for CIFAR-10 and 20 PGD iterations for ImageNet- 1K We set the EOT iteration to 20 for both datasets. We evaluate AT baseline methods using AutoAttack with 100 update iterations, as AT methods have seen PGD attacks during training, leading to overestimated results when evaluated on PGD+EOT (Lee& & Kim, 2023). For our method, we implic- itly design an adaptive white-box attack by considering the entire defense mechanism of DAD (see Algorithm 3). To make a fair comparison, we evaluate our method on both adaptive white-bos PGD+EOT attack and adaptive white-
5.1. Defending against Adaptive White-box Attacks
Result analysis on CIFAR-10. Table 1 shows the eval- uation performance of DAD against adaptive white-box PGD+EOT attack with fo = 8/255) and 6 = 0.5) on CIFAR-10. Compared to SOTA defense methods, DAD improves clean and robust accuracy by a notable margin.
6

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Diserepancy
Table 1. Clean and robust accuracy (%) against adaptive white-box attacks (left: foe (e (e=8/255) right: & (e 0.5)) CIFAR-1O 10.
means this method uses 1 means this method is trained with extra data. We report averaged results standard deviations of our method for 5 runs. We show the most successful defense in bold.
oo (E 8/255)
f (c = 0.5)
Type
Method
Clean
Robust
Type
Method
Clean
Robust
WRN-28-10
WRN-28-10
Gowal et al. (2021)
87.51
63.38
Rebuffi et al. (2021)*
91.79
78.80
AT
Gowal et al. (2020)*
88.54
62,76
AT
Augustin et al. (2020)!
93.96
78.79
Pang al. (2022a)
88.62
61.04
Sehwag al. (2022)1
90.93
77.24
Yoon et al. (2021)
85.66
33.48
Yoon et al. (2021)
85.66
73.32
AP
Nie al. (2022)
90.07
46.84
Nie et al. (2022)
91.41
79.45
Lee & Kim (2023)
90.16
55.82
Lee & Kim (2023)
90.16
83.59
Ours
DAD
94.16 + 0.08
67.53 + 1.07
Qurs
DAD
94.16
0.08
84.38 + 0.81
WRN-70-16
WRN-70-16
Rebuffi et al. (2021)*
92.22
66.56
Rebuffi al. (2021)*
95.74
82.32
AT
Gowal et al. (2021)
88.75
66.10
AT
Gowal et al. (2020)*
94.74
80.53
Gowal al. (2020)*
91.10
65.87
Rebuffi al. (2021)
92.41
80.42
Yoon et al. (2021)
86.76
37.11
Yoon et al. (2021)
86.76
75.66
AP
Nie et al. (2022)
90.43
51.13
Nie et (2022)
92.15
82.97
Lee & Kim (2023)
90.53
56.88
Lee & Kim (2023)
90.53
83.57
Ours
DAD
93.91 + 0.11
67.68 + 0.87
Ours
DAD
93.91 0.11
84.03 1: 0.75
feasible task than estimating data density, especially large-scale datasets such ImageNet-IK.
Table Clean and robust accuracy (%) against adaptive white-box attacks loo. (E = 4/255) on ImageNer-IN K. We report averaged results and standard deviations of methad for 3 runs. We show the most successful defense in bold.
5.2. Defending against Unseen Transfer Attacks
foo (c 4/255)
Since DAD requires AEs train the MMD-OPT and the denoiser, it is important for us to evaluate the transferability of our method. We report the transfcrability of our method (trained on WideResNet-255 10) under different threat mod- ResNet-18, ResNet-
Type
Method
Clean
Robust
RN-50
Salman ul. (2020a)
64.02
34.96
els, which include WideResNet- and Swin Transformer in Table We use PGD+EOT to and C&W &g (Carlini Wagner, 2017) with 200 iterations for evaluation. Experiment results show that method generalizes well to unseen transfer attacks.
AT
Engstrom (2019)
62.56
29.22
Wong et al. (2020)
55.62
26.24
Nie el al. (2022)
71.48
38.71
AP
Lee & Kim (2023)
70.74
42.15
Ours
DAD
78.61 + 0.04
53.85 + 0.23
5.3. Ablation Studies
Ablation study batch size. Identifying distributional
The evaluation results against BPDA+EOT on CIFAR-10 can be found in Appendix D.
discrepancies requires the data to be processed in batches. Therefore, we aim to determine how much data in a batch will not affect the stability of our method. Figure (top) shows the clean accuracy of our method on CIFAR-10 10 with different batch sizes, ranging from 10 to 110. We find that once the batch size exceeds 100, the performance of our method is stable. Therefore, in this paper, we the test batch size to 100 for all evaluations.
Result analysis on ImageNet-1K, Table 2 shows the eval- uation performance of DAD against adaptive white-box PGD+EOT attack with lx (e 4/255) on ImageNet-I1 The advantages our method over baselines become more significant large-scale datasets. Specifically, compared with AP methods that rely on density estimation (Nie et al., 2022; Lee & Kim, 2023), our method improves clean ac- curacy by at least 7.13% and robust accuracy by 11.70% on ResNet-50. This empirical evidence supports that iden- tifying distributional discrepancies is a simpler and more
Ablation study on mixed data batches. We explore more challenging scenario for our method, in which each data batch contains mixture of CEs and AEs. Figure 2 (bottom)

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Diserepancy
Table Robust accuracy (%) of DAD trained on WideResNet-28-10 against unseen transfer attacks CIFAR-JO. Attackers cannot
access parameters of MideResNNet22-U1 10, thereby it is in gray-bex setting. We report averaged results and standard deviations of 5 runs.
Trained WRN-28-10
Unseen Transfer Attack
WRN-70-16
RN-18
RN-50
Swin-T
8/255
80.84 + 0,46
80.78 + 0.60
81.47 = 0.30
81.46 - 0.29 80.40 + 0.41
PGD+FOT (fox)
12/ /55
80.26 + 0.60
80.54 + 0.45
80.98 + 0.36
0.5
82.45 + 0.19
91.30 + 0.20
89.26 = 0.11
93,45 + 0.17
C&W (62)
1.0
81.20 + 0.39
90.37 1 0.17
88.65 0.22
93.41 + 0.18
to CEs than AEs with e 8/255 (i.e., AEs for CIFAR-10). Intuitively, when e decreases AEs are the same as CEs (i.e., the distribution of AEs and CEs will be the same).
Clean Acc (%)
92.5
90.0
Ablation study on the regularization term a in Eq. (7). We explore the impact of a in Appendix D.4. Notably, when increases, the robust accuracy will decrease. This is because increasing a in Eq. (7) reduces the relative influence of the MMD-OPT, thereby diminishing its contribution to robustness. This observation aligns with the theoretical findings presented in Section 3.
87.5
85.0
20
40
60
80
100
120
Batch size
100
Mixed (%)
90
80
Ablation study on injecting Gaussian noise. We provide evaluation results of our method against adaptive white- box PGD+EOT attack with and without injecting Gaussian noise on CIFAR- 10 in Appendix D.5. We find that injecting Gaussian noise can make DAD generalize better.
70
60
Proportion of AEs in every batch (%)
Figure Top: clean accuracy (%) vs. batch size; Bottom: mixed accuracy (%) vs. proportion of AFs in batch (%). We plot averaged results and standard deviations of 5 runs.
Ablation study on the two-pronged process. We provide evaluation results of our method against adaptive white-box PGD+EOT attack with and without MMD-OPT on CIFAR- 10 in Appendix D.6. We find that using the two-pronged process can largely improve clean accuracy.
shows the mixed accuracy (i.e., the accuracy on mixed data) of our method CIFAR-10 with different proportions of AEs (generated by adaptive white-box PGD+EOT too with == 8/255) in each batch, ranging from 0% (i.e., pure CEs) to 100% (i.e., pure AEs). Initially, the mixed accuracy drops from over 90% to approximately 80%. This is because, with a high proportion CEs, the MMD-OPT has high chance regard the entire batch as clean data. After that (i.e., from 30% onward), the mixed accuracy degrades gradually to approximately 70%. This is because, the proportion of AEs increases, the MMD-OPT regards the entire batch as adversarial and feeds it into the denoiser. Notably, DAI) can still outperform baseline methods (see Appendix D.2).
5.4. Compute Resource of DAD
We report the compute resources used for training and eval- uating DAD in Appendix D.7. Compared to AT baselines, DAD offers better training efficiency (c.g., it can scale to large datasets like ImageNet- Additionally, although DAD requires training an extra denoiser and MMD-OPT, it significantly outperforms AP baselines in inference speed. Furthermore, relying a pre-trained generative model is not always feasible, as training such models at scale can be highly resource-intensive. Therefore, in general, DAD provides a more lightweight design.
Ablation study on threshold values of MMD-OPT. We explore the impact of threshold values MMD-OPT in Appendix D.3. We select the threshold based the perimental results on the validation data. Specifically, threshold value of 0.5 is selected for CIFAR-10 10 and 0.02 is selected for ImageNet-1K threshold for ImageNet-IK because the distribution AEs with = 4/255 (i.e., AEs for ImageNet- I will be closer
5.5. Combination with Adversarial Training
We also combine several well-known AT methods with DAD: Vanilla AT (Madry et al., 2018). TRADES (Zhang et al., 2019), MART (Wang et al., 2020), and MART (Wu et al., 2020) to see whether our method can be combined with AT-based methods. Following our pipeline, detected CEs are directly classified by the AT- classifier, while
is reasonable to use a smaller
8

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
detected AEs are first denoised by our method before being classified by AT-based methods. Experimental results show
which requires the data to be processed in batches for ad- versarial detection. When the batch size is too small, the stability of DAD will be affected (see Figure 2). To address this issue, each single sample provided by the user can be dynamically stored in a queue. Once the queue accumulates enough samples to form a batch, our method can then pro- cess the batch collectively using the proposed approach. A direct cost of this solution is the waiting time, as the system must accumulate enough samples (e.g., 50 samples) to form batch before processing. However, in scenarios where data arrives quickly, the waiting time is typically very short, mak- ing this approach feasible for many real-time applications. Overall, it a trade-off problem: using our method for user inference can obtain high robustness, but the cost is to wait for batch processing. Based on the performance improve- ments our method obtains over the baseline methods, we believe the cost is acceptable. Another possible solution is to find more robust statistics that can measure distributional discrepancies with fewer samples. We leave finding such statistics as future work.
that combining AT methods with DAD can consistently im- prove adversarial robustness under different proportions of AES in each batch (see Appendix D.8).
6. Related Work
We briefly review the related work here, and more detailed version can be found in Appendix E.
Statistical adversarial data detection. Recently, statistical adversarial data detection (SADD) has attracted increas- ing attention in defending against AEs. For example, Gao et al. (2021) demonstrate that maximum mean discrepancy (MMD) is aware of adversarial attacks and leverage the distributional discrepancy between AEs and CEs to filter out AEs, which has been shown effective against unseen attacks. Based on this, Zhang et al. (2023) further propose expected perturbation score to measure the expected score of a sample after multiple perturbations.
Practicability beyond user inference. Other than user in- ference, our method is suitable for cleaning the data before fine-tuning the underlying model. In many domains, obtain- ing large quantities of high-quality data is challenging due to factors such as cost, privacy concerns, or the rarity of specific data. As a result, all possible samples with clean in- formation are critical in these data-scarce domains. Then, a practical scenario is that there exists a pre-trained model on large-scale dataset (e.g.. a DNN trained on ImageNet- 1K) and clients want to fine-tune the model perform well on downstream tasks. If the data for downstream tasks contain AEs, our method can be applied to batch-wisely clean the data before fine-tuning the underlying model.
Denoiser-based adversarial defense. Denoiser- ad- versarial defense often leverages generative models to shift AEs back to their clean counterparts before feeding them into a classifier. In most literature, it is called adversarial purification (AP). At the early stage of AP, Meng & Chen (2017) propose a two-pronged defense called MagNet to remove adversarial noise by first using a detector to dis- card the detected AEs, and then using an autoencoder to purify the remaining samples. The following studies mainly focus on exploring the use of more powerful generative models for AP (Liao et al., 2018; Samangouei et al., 2018; Song et al., 2018; Yoon et al., 2021; Nie et al., 2022). Re- cently, the outstanding denoising capabilities of pre-trained diffusion models have been leveraged to purify AEs (Nie et al., 2022; Lee & Kim, 2023). The success of recent AP methods often relies on the assumption that there will be a pre-trained generative model that can precisely estimate the probability density of the CEs (Nie et al., 2022; Lee & Kim, 2023). However, even powerful generative models (e.g.. dif- fusion models) may have an inaccurate density estimation, leading to unsatisfactory performance (Chen et al., 2024). By contrast, instead of estimating probability densities, our method directly minimizes the distributional discrepancies between AES and CEs, leveraging the fact that identifying distributional discrepancies is simpler and more feasible.
8. Conclusion
SADD-based defense methods empirically show that lever- aging the distributional discrepancies can effectively defend against adversarial attacks. However, potential limita- tion of SADD-based methods is that they will discard data batches that contain AEs, leading to the loss of clean infor- mation. To solve this problem, inspired by our theoretical analysis that minimizing distributional discrepancy can help reduce the expected loss on AEs, we propose a two-pronged adversarial defense called petriibttioaaddiiceppeeee Adversarial Defense (DAD) that leverages the effective- ness of SADD-based methods without discarding input data, which kills two birds with one stone. Extensive experiments demonstrate that DAD effectively defends against various adversarial attacks, simultaneously improving both robust- ness and clean accuracy. In general, we hope this simple yet effective method could open up new perspective on adversarial defenses based on distributional discrepancies.
7. Discussions on Batch-wise Detections
We briefly discuss the limitation, solutions and practicability here, and see Appendix F for detailed discussions.
Limitation and its solutions for user inference. DAD leverages statistics based distributional discrepancies,
9

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Diserepancy
Acknowledgements
Borgwardt, K. M., Gretton, A., Rasch, M. J., Kriegel, H., Schölkopf, B.,, and Smola, A. J. Integrating structured biological data by kernel maximum mean discrepancy. In Proceedings 14th International Conference on Intelligent Systems for Molecular Biology, 2006.
JCZ is supported by the Melbourne Research Scholarship and would like thank Qiwei Tian and Ruijiang Dong for productive discussions. FL is supported by the Australian Research Council (ARC) with grant number DE240101089, LP240100101, DP230101540 and the NSF&CSIRO Re- sponsible AI program with grant number 2303037. BR is supported by ARC DP220102269. This research was supported by The University of Melbourne's Research Com- puting Services and the Petascale Campus Initiative.
Cao, Y., Wang, N., Xiao, C., Yang, D., Fang, J., Yang, R., Chen, Q. A., Liu, M., and Li, B. Invisible for both camera and lidar: Security of multi-sensor fusion based perception in autonomous driving under ahysical-world attacks. In IEEE Symposium Security and Privacy, pp. 176-194, 2021.
Impact Statement
Carlini. N. and Wagner, D. A. Towards evaluating the robustness of neural networks. In /EEE Symposium on Security and Privacy, pp. 39-57. IEEE, 2017.
This study on adversarial defense mechanisms raises im- portant ethical considerations that we have carefully ad- dressed. We have taken steps to ensure our adversarial defense method is fair. We use widely accepted public benchmark datasets to ensure comparability of our results. Our evaluation encompasses wide range of attack types and strengths to provide comprehensive assessment of our defense mechanism. The proposed defense algorithm con- tributes to the development of more robust machine learning models, potentially improving the reliability of Al systems in various applications. We will actively engage with the research community to promote responsible development and use of adversarial defenses.
Carlini, N., Tramèr, F., Dvijotham, K. D., Rice, L., Sun, M., and Kolter, J. Z. (certified!!) adversarial robustness for free! In ICLR, 2023.
Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, J. C.,, and Liang, P. Unlabeled data improves adversarial robustness. In NeurlPS, 2019.
Chen, H., Dong, Y., Wang, Z., Yang. X., Duan, C., Su, H., and Zhu, J. Robust classification via single diffusion model. In ICML, 2024.
References
Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with ensemble of diverse parameter-fiee attacks. In ICML, 2020a.
Andriushchenko, M.. Croce, Flammarion, N., and Hein, M. Square attack: A query-efficient black-box adversarial attack via random search. In ECCV, 2020.
Croce, F. and Hein, M. Minimally distorted adversarial examples with a fast adaptive boundary attack. In ICML, 2020b.
Athalye, A., Carlini, N., and Wagner, D. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018a.
Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
Athalye, A.,
Engstrom, I.,, Ilyas, A., and Kwok, K. Synthe- sizing robust adversarial examples. In ICML, 2018b.
Augustin, M., Meinke, A., and Hein, M. Adversarial robust- ness on in- and out-distribution improves explainability. In ECCV, 2020.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Bartoldson, B. R., Diffenderfer, J., Parasyris, K., and Kailkhura, B. Adversarial robustness limits via scaling- law and human-alignment studies. In ICML,, 2024.
Deng, Z., Yang, X., Xu, S., Su, H., and Zhu, J. LiBRe: practical Bayesian approach to adversarial detection. In CVPR, 2021.
Ben-David, S., Blitzer, J., Crammer, K., and Pereira, F. Analysis representations for domain adaptation. In NeurlPS, 2006.
Ding, G. W., Sharma, Y., Lui, K. Y. C., and Huang, R. MMA training: Direct input space margin maximization through adversarial training. In ICLR, 2020.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W. A theory of learning from different domains. Mach. Learn., 1911-2): 2010.
Dong, Y., Su, H., Wu, B., L., Liu, W., Zhang, T., and Zhu, J. Efficient decision- based black-box adversarial attacks on face recognition, In CVPR, 2019.
10

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Diserepancy
Engstrom, L.., Ilyas, A., Salman, H., Santurkar, S., and Tsipras, D. Robustness (python library), 2019. URL https://gitee hwt.getth.......aeeeeeeeeeeeeeee com/MadryLab/ robustness.
Laidlaw, C., Singla, S., and Feizi, S. Perceptual adversarial robustness: Defense against unseen threat models. In ICLR, 2021.
Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018.
Fang, Z., Li, Y., Lu, J., Dong, J., Han, B., and Liu, F. Is out- of-distribution detection learnable? NeurlPS, 2022.
Lee, M. and Kim, D. Robust evaluation of diffusion-based adversarial purification. In ICCV, 2023.
Gandhar, A., Gupta, K., Pandey, A. K., and Raj. D. Fraud detection using machine learning and deep learning, SN Comput. Sci., 5(5):453, 2024.
Liao, F., Liang, M., Dong, Y., Pang, T., Hu, X., and Zhu, J. Defense against adversarial attacks using high-level representation guided denoiser. In CVPR, 2018.
Gao, R., Liu, F., Zhang, J., Han, B., Liu, T., Niu, G., and Sugiyama, M. Maximum mean discrepancy test aware of adversarial attacks. In ICML, 2021.
Lin, G., Li, C., Zhang, J., Tanaka, T., and Zhao, Q. Ad- versarial training on purification (atop): Advancing both robustness and generalization. In ICLR, 2024.
Gao, R., Wang, J., Zhou, K., Liu, F., Xie, B., Niu, G., Han, B., and Cheng, J. Fast and reliable evaluation of adversarial robustness with minimum margin attack. In ICML, 2022.
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., and Sánchez, C. I. A survey on deep learning medical image analysis. Medical Image Anal., 42:60-88, 2017.
Goodfellow, J., Shlens, J., and Szegedy, Explaining and harnessing adversarial examples. In ICLR, 2015.
Gowal, S., Qin, C., Uesato, J., Mann, T. A., and Kohli, P. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.03593, 2020.
Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A., and Suther- land, D. J. Learning deep kernlls for non-parametric two-sample tests. In ICML, 2020.
Liu, F. Xu, W., Lu, J., and Sutherland, J. Meta two- sample testing: Learning kernels for testing with limited data. In NeurlPS, 2021a.
Gowal, S., Rebuffi, S., Wiles, O., Stimberg, F., Calian, D. A., and Mann, T. A. Improving robustness using generated data. In NeurIPS, 2021.
Liu, Z.. Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021b.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, J. A kernel two-sample test. Mach. Learn. Res., 13:723-773, 2012.
Ma, X., Li, B., Wang, Y., Erfani, S. M., Wijewickrema, S. N. R., Schoenebeck, G., Song, D., Houle, M. E., and Bailey, J. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018.
Han, B., Yao, J., Liu, T., Li, B., Koyejo, S., and Liu, F. Trustworthy Machine Learning: From Data to Models. Now Foundations and Trends, 2025.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Hill, M., Mitchell, J. C., and Zhu, S. Stochastic security: Adversarial defense using long-run dynamics of energy- based models. In ICLR, 2021.
Mansour, Y., Mohri, M., and Rostamizadeh, A. Domain adaptation: Learning bounds and algorithms. In COLT, 2009.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR, 2015.
Meng. D. and Chen, H. Magnet: A two-pronged defense against adversarial examples. In CCS, 2017.
Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-10 (cana- dian institute for advanced research). 2009. URL http: //www. cs wow.otoooooo.oooooo edu/- -wti//riiiiiei ci sirarrneeee .tmll.
Naseer, M., Khan, S. H., Hayat, M., Khan, F. S., and Porikli, F. A self-supervised approach for adversarial robustness. CVPR, 2020.
Kurakin, A., Goodfellow, I. J., and Bengio, S. Adversarial examples in the physical world. In ICLR, Workshop Track Proceedings, 2017.
Nayak, G. K., Rawal, R., and Chakraborty, A. DE-CROP: data-efficient certified robustness for pretrained classifiers. WACV, 2023.

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
Nie, W. Guo, B., Huang, Y., Xiao, C., Vahdat, A., and Anandkumar, A. Diffusion models for adversarial purifi- cation. In ICML, 2022.
Szegedy, C., Zaremba, W. Sutskever, Bruna, J., Erhan, D., Goodfellow, J. and Fergus, R. Intriguing properties of neural networks. In ICLR, 2014.
Pang, T., Yang, X., Dong, Y., Su, H., and Zhu, J. Bag of tricks for adversarial training. In ICLR, 2021.
Wang. J., Lyu, Z., Lin, D., Dai, B., and Fu, H. Guided dif- fusion model for adversarial purification. arXiv preprint arXiv: /4969, 2022.
Pang. T., Lin, M., Yang, X., Zhu, J., and Yan, S. Robustness and accuracy could be reconcilable by (proper) definition. In ICML, 2022a.
Wang. Q.. Liu, F., Han, B., Liu, T., Gong, C., Niu, G., Zhou, M., and Sugiyama, M. Probabilistic margins for instance reweighting adversarial training. In NeurIPS, 2021.
Pang, T., Zhang, H., He, D., Dong, Y., Su, H., Chen, W., Zhu, J., and Liu, T. Two coupled rejection metrics can tell adversarial examples apart. In CVPR, 2022b.
Wang. Y., Zou, D., Yi, J., Bailey, J., Ma, X., and Gu, Q. Improving adversarial robustness requires revisiting mis- classified examples. In ICLR, 2020.
Poursaeed, O., Jiang, T., Yang, H., Belongie, S. J., and Lim, S. Robustness and generalization via generative adversarial training. In ICCV, 2021.
Wong, E., Rice, L., and Kolter, J. Z. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
Raghuram, J., Chandrasekaran, V., Jha, S., and Banerjee, S.
Wu, D., Xia, S., and Wang, Y. Adversarial weight perturba- tion helps robust generalization. In NeurIPS, 2020.
A general framework for detecting anomalous inputs to
DNN classifiers. ICML, 2021.
Yang, Y., Zhang, G., Xu, Z., and Katabi, D. ME-Net: To- wards effective adversarial robustness with matrix estima- tion. In ICML, 2019.
Rebuffi, S., Gowal, S., Calian, D. A., Stimberg, F,, Wiles,
O., and Mann, T. A. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.01946 2021.
Yoon, J., Hwang. S. J., and Lee, Adversarial purification with score-based generative models. In ICML, 2021.
Rice, L., Wong, E., and Kolter, J. Z. Overfitting in adversar- ially robust deep learning. In ICML, 2020.
Zagoruyko, S. and Komodakis, N. Wide residual networks. In BMVC, 2016.
Salman, H., llyss A., Engstrom, L., Kapoor, A., and Madry, A. Do adversarially robust ImageNet models transfer better? In NeurlPS, 2020a.
Zhang, H., Yu. Y., Jiao, J., Xing, E. P., Ghaoui, L. E., and Jordan, M. I. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Salman, H., Sun, M., Yang, G., Kapoor, A., and Kolter, J. Z. Denoised smoothing: provable defense for pretrained classifiers. In NeurlPS, 2020b.
Zhang, J., Zhu, J., Niu, G., Han, B., Sugiyama, M., and Kankanhalli, M. S. Geometry-aware instance-reweighted adversarial training. In ICLR, 2021.
Sanangouei, P., Kabkab, M., and Chellappa, R. Defense- GAN: Protecting classifiers against adversarial attacks using generative models. In ICLR, 2018
Zhang, J., Liu, F., Zhou, D., Zhang, J., and Liu, T. Improv- ing accuracy-robustness trade-off via pixel reweighted adversarial training. In ICML, 2024.
Schwag, V., Mahloujifar, S., Handina, T., Dai, S., Xiang, C., Chiang, M., and Mittal, P. Robust learning meets genera- tive models: Can proxy distributions improve adversarial robustness? In ICLR, 2022.
Zhang, S., Liu, F., Yang, J., Yang, Y., Li, C., Han, B., and Tan, M. Detecting adversarial data by probing multiple perturbations using expected perturbation score. ICML, 2023.
Song, C., He, K. Wang, L., and Hopcroft, J. E. Improving the generalization of adversarial training with domain adaptation. In ICLR, 2019.
Zhou, D.,, Chen, Y., Wang, N., Liu, D., Gao, X., and Liu, T. Eliminating adversarial noise via information discard and robust representation restoration. In International Con- ference on Machine Learning, pp. 42517-42534 PMLR, 2023a.
Song, Y., Kim, T., Nowozin, S., Ermon, S., and Kushman, N. PixelDefend: Leveraging generative models to under- stand and defend against adversarial examples. In ICLR, 2018.
Zhou, D., Wang, N., Yang, H., Gao, X., and Liu, T. Phase- aware adversarial defense for improving adversarial ro- bustness. In International Conference on Machine Learn ing. pp. 42724-42741. PMLR, 2023b.
Stutz, D., Hein, M., and Schiele, B. Confidence-calibrated adversarial training: Generalizing to unseen attacks. In ICMI., 2020.
12

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
A. Proof of Theorem 1
Theorem 1. For a hypothesis H and distribution DA D::
fu. S R(h, fc. Dc) + (Dc. Da).
Proof. Let de and ΦA be the density functions of De and DA:
R(h, DA) = R(h, DA) R(,, Dc) - R(h, Sc. Dc) + LA, Dc) Pa,,, LA Dc)
- Re, De) + R(, Dc) R(h, fc. Dc) + RR(h, fA. Da) R(h, fa. Dc))
R(h, fe, Dc) [lfe(x)) fa(x)]l + [R(h, fA. DA) fa.
R(h, Sc, Dc) E [[x(()) fa(x)]I / [6c(x) (x(x))1((())) (x(x))dxx
(6)
S R(h, fc, Dc) (x) fx(x)) + (D, DA)
( R(h, fc. Dc) + B llf((()) (x) fc(x))l + (Dc, DA)
R(h, Sc, Dc) + di (De, DA):
where (a) is based on Definition and (b) based on Corollary 1.
B. Mathematical Notations in Section 4
X'
A separable metric space in gd
P, Q
Borel probability measures defined on X
Sx
IID observations sampled from IP, i.e., {x(0)
Sz
m IID observations sampled from Q i.e., {z mm
H2
reproducing kernel Hilbert space kernel of HIg with parameters w The kernel mean embedding The kernel mean embedding Q kw X;) k. (Z:, 2,) hew (x,, z;) ky (z,,
kw
Hi
deep kernel function that measures the similarity between x and z
he
A well-trained classifier
Bo
A constant E 1)
q
The Gaussian kernel with bandwidth og
J
The objective function of optimizing MMD
Mean and standard deviation
A
A small constant to avoid 0 division Gaussian noise, i.e., N(u, o²)
ge
A denoiser with parameters 0 Clean samples
Se
Ye
Ground truth labels of Se
SA
Adversarial examples
Spoise
Noise- injectede adversarial examples
Stenoised
Denoised samples
A regularization term
13

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
C. Detailed Experiment Settings
C.1. Dataset and target models
We evaluate the effectiveness of DAD on two benchmark datasets with different scales, i.., CIFAR-I0 (Krizhevsky et al., 2009) (small scale) and ImageNet- IK (Deng et al., 2009) (large scale). Specifically, CIFAR-I0 contains 50,000 training images and 10,000 test images, divided into 10 classes. ImageNet-1 a large-scale dataset that contains 1,000 classes and consists of 1, 1,281,167 training images, 50,000 validation images, and 100,000 test images. For the target models, three widely used architectures with different scales: ResNet (He et al., 2016), WideResNet (Zagoruyko & Komodakis, 2016) and Swin Transformer (Liu et al., 2021b). Specifically, following Lee & Kim (2023), we use WideResNet-22 28- 10 and WideResNett-70- 16 to evaluate the performance of defense methods on CIFAR-10 and we use ResNer-50 to evaluate the performance of defense methods on ImageNet-1KK Additionally, we examine the transferabilityy of our method under different threat models, which include ResNet-18, 18, ResNet-50, WideResNet-70-16 and Swin Transformer.
C.2. Baseline Settings
DAD is a two-pronged adversarial defense method, which is different from most existing defense methods. In terms of the pipeline structure, MagNet (Meng & Chen, 2017) is the only similar defense method to ours, which also contains a two-pronged process. However, MagNet is now considered outdated, making it unfair for DAD to compare with it. Therefore, to make the comparison as fair as possible, we follow a recent study robust evaluation (Lee & Kim, 2023) to compare our method with SOTA adversarial training (AT) methods in RobustBench (Croce et al., 2020) and adversarial purification (AP) methods selected by Lee & Kim (2023).
C.3. Evaluation Settings
We mainly use PGD+EOT (Athalye et al., 2018b) and AutoAttack (Croce& & Hein, 2020a) to compare our method with different baseline methods. Specifically, following Lee & Kim (2023), we evaluate AP methods on the PGD+EOT attack with 200 PGD iterations for CIFAR-10 10 and 20 PGD iterations for ImageNet-IK. We set the EOT iteration to 20 for both datasets. We evaluate AT baseline methods using AutoAttack with 100 update iterations, as AT methods have seen PGD attacks during training, leading to overestimated results when evaluated on PGD+EOT (Lee & Kim, 2023). For our method, we implicitly design an adaptive white-box attack by considering the entire defense mechanism of DAD. To make fair comparison, evaluate our method both adaptive white-box PGD+EOT attack and adaptive white-box AutoAttack with the same configurations mentioned above. Notably, we find that our method achieves the worst-case robust accuracy on adaptive white-box PGD+EOT attack. Therefore, we report the robust accuracy of our method on adaptive white-box PGD+EOT attack for Table 1 and 2. The algorithmic descriptions of the adaptive white-box attack is provided in Algorithm 3. On CIFAR-10, e for to -norm- based attacks and ba-norm-based attacks is set to 8/255 and 0.5, respectively. While on ImageNet-IK: IK, we set E = 4/255 for -oom--be attacks. We also evaluate our method against BPDA+EOT (Hill et al., 2021) on CIFAR-10. For BPDA+EOT, we use the implementation of Hill et al. (2021) with default hyperparameters for evaluation. For transferability experiments, we use PGD+EOT loo and C&W E (Carlini & Wagner, 2017) for evaluation. Specifically, the iteration number of C&W lg is set 200. For lo -nom transfer attacks, we examine the robustness of our method under 8/255 and 12 12/255. For C&W W f2, we examine our method under 0.5 and 1.0.
C.4. Implementation Details of DAD
To avoid the evaluation bias caused by learning similar attacks beforehand during training. we train both the MMD-OPT and the denoiser using the MMA attack with -nom (Gao et al., 2022), which differs significantly from PGD+EOT and AutoAttack. Then, we use unseen attacks to evaluate DAD. We set e = 8/255 with a step size of 2/255 for CIFAR-10 10, and <= <= 4/255 with a step size of 1 1/255 for ImageNet- K. For optimizing the MMD, following Gao et al. (2021), we set the learning rate to be 2 x 10-4 and the epoch number to be 200. For training the denoiser, we set the initial learning rate to 1 x 10-3 for both CIFAR- 10 and ImageNet-1KK We set the epoch number to be 60 and divide the learning rate by 10 at the 45th epoch and 60th epoch avoid robust overfitting (Rice et al., 2020). The training batch size set to 500 for CIFAR-10 10 and 128 for ImageNet- IK. The optimizer we use is Adam (Kingma & Ba, 2015). To improve the training efficiency on ImageNet-IK, 1K, we randomly select 100 samples from each class, resulting in 100,000 training samples in total. Notably, during the inference time, we evaluate our method using the entire testing set for both CIFAR-10 and ImageNet- K. The batch size for evaluation is set to 100 for all datasets.
14

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Diserepancy
D. A Additional Experiments
D.I. Defending against BPDA+EOT Attack
Table Clean (%) and robust accuracy (%) defense methods against BPDA+EOT attack under & (é = 8/255) threat models on
CIFAR-JO 10. We report averaged results and standard deviations of DAD for 5 runs. We show the most successful defense in bold.
Category
Model
Method
Clean
Robust
Average
Madry al. (2018)
87.30
45.80
66.55
RN-18
Zhang et al. (2019)
84.90
45.80
65.35
Adversarial Training
Carmon et al. (2019)
89.67
63.10
76.39
WRN-28-10
Gowal et al. (2020)
89.48
64.08
77.28
RN-18
Yang et al. (2019)
94.80
40.80
67.80
RN-62
Song et al. (2018)
95.00
9.00
52.00
Hill et al. (2021)
84.12
54.90
69.51
Adversarial Purification
Yoon et al. (2021)
86.14
70.01
78.08
WRN-28-10
Wang et al. (2022)
93.50
79.83
86.67
Nie et al. (2022)
89.02
81.40
85.21
Lee & Kim (2023)
90.16
88.40
89.28
Ours
WRN-28-10
DAD
94.16 + 0.08
87.13 + 1.19
90.65
D.2. Ablation Study on Mixed Data Batches
Table Mixed accuracy (%) of defense methods against adaptive white-box attacks to 8/255) on CIFAR-10 under different proportions of AEs. The target model is WRN-28-10 10. We report averaged results and standard deviations of 5 runs. We show the most successful defense bold.
Proportion of AEs in Each Batch (%)
Method
60
70
80
90
100
Rebuffi al. (2021)
85.10
82.68
80.27
77.86
75.45
73.03
70.62
68.21
65.79
63.38
Augustin
(2020)
85,96
83.38
80.81
78.23
75.65
73.07
70.49
67.92
65.34
62.76
Sehwag et al. (2022)
85.86
83.10
80.35
77.59
74.83
72.07
69.31
66.56
63.80
61.04
Yoon et al. (2021)
81.80
76.83
71.87
66.90
61.94
56.97
52.01
47.04
42.08
37.11
Nie et al. (2022)
85.75
81.42
77.10
72.78
68.46
64.13
59.81
55.49
55.16
46.84
Lee Kim (2023)
86.73
83.29
79.86
76.42
72.99
69.56
66.12
62.69
59,25
55.82
91.22
87.15
81.77
79.94
77.78
76.14
74.22
72.37
69.56
67.53
Ours
0.47
0.58
0.66
0.66
0.51
0.69
0.53
0.74
0.83
1.07
D.3. Ablation Study on Threshold Values of MMDDOPP
Table Sensitivity of DAD to the threshold values of MMD-OPT on CIFAR-10. We report clean and robust accuracy (%) against adaptive
white-hox attacks (e 8/255). The classifier used is WRN-28-10.
PGD+EOT
AutoAttack foo
Threshold Value
Clean
too
&g
0.05
94.16
66.98
73.40
72.21
85.96
0.07
94.16
66.98
73.40
72.21
85.96
0.10
94.16
66.98
73.40
72.21
85.96
0.50
94.16
66.98
84.38
72.21
85.96
0.70
94.16
66.98
84.38
72.21
85.96
1.00
94.16
64.75
84.38
72.21
85,96
15

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Diserepancy
D.4. Ablation Study on the Regularization Term a in Eq. (7)
Table Sensitivity of DAD to the regularization term a on CIFAR-IO. We report clean and robust accuracy (%) against adaptive
white-hox PGD+FOT (< 8/255). The classifier used is WRN-28-10.
Clean
PGD+EOT
0.01
94.16
67.53
0.05
94.16
50.70
0.10
94.16
45.35
D.5. Ablation Study on Injecting Gaussian Noise
Table 8, Robust accuracy (%) of our method with and without injecting Guassian noise against adaptive white-box PGD+EOT loc 8/255) and (e 0.5) on CIFAR-1O. We report averaged results and standard deviations of runs. show the most successful defense in bold.
Gaussian Noise
Model
PGD+EOT (foe)
PGD+EOT (f2
65.31 0.67
81.04 + 0.52
WRN-28-10
67.53 + 1.07
84.38 + 0.81
D.6. Ablation Study on the Two-pronged Process
Table 9. Clean and robust accuracy (%) of our method with and without the two-pronged process against adaptive white-box PGD+EOT loo 8/ 3/255) and Ez(e 0.5) CIFAR-10. We report averaged results and standard deviations of runs. We show the most successful defense in bold.
Module
Model
Clean
PGD+EOT (6so)
PGD+EOT (f2)
Denoiser only
85.07 0.16
71.76
0.65
85.01 0.50
WRN-28-10
Denoiser MMD-OPT
94.16 0.08
67.53 = 1.07
84.37 + 0.81
D.7. Compute Resources
Table 10. Training time (hours: minutes: seconds) and memory consumption (MB) for DAD on CIFAR-1O and ImageNet-IK: This table reports the compute resources for the entire training process of DAD (i.e.. optimizing MMD + training the denoiser).
Dataset
GPU
Batch Size
Classifier
Training Time
Memory
RN-18
00:28:17
5927
CIFAR-10
2 NVIDIA A100
500
WRN-28-10
00:55:34
6276
ImageNet-IK
NVIDIA A100
128
RN-50
09:52:50
97354
Table 11. Inference time (hours: minutes: seconds) for DAD on CIFAR-10 10 and ImageNer-IKK. This table reports the comput resources for evaluating the entire test set of CIFAR-10 (i.c., 10,000 images) and ImageNet-1K IK (i.c., 50,000 images).
Dataset
GPU
Batch Size
Classifier
Inference Time
CIFAR-10
1 x NVIDIA A100
100
WRN-28-10
00:00:32
ImageNet-1K
2 x NVIDIA A100
100
RN-50
00:03:08
Table 10 presents the compute resources for DAD, which include GPU configurations, batch size, classifier, training time, and memory usage for each dataset. For CIFAR-10, using NVIDIA A100 GPUs with batch size of 500, our method's training time is approximately 28 minutes with ResNet- 18 and 55 minutes with WideResNet-28-- 10. The memory consumption is 5927 MB and 6276 MB, respectively. For ImageNet-1K 1K, using 4 NVIDIA A100 GPUs with a batch size of 128, our method's 's training time is approximately 10 hours, with memory consumption of 97354 MB. Compared to AT baseline methods, DAD offers better training efficiency (e.g., it can scale to large datasets like ImageNet-IK), This is mainly because we directly use the pre-trained classifier. Furthermore, training MMD is extremely fast (usually less than 1 minute on CIFAR-10) and we use a lightweight denoiser.
16

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
Table 11 presents the compute resources for evaluating DAD, which include GPU configurations, batch size, classifier and inference time for each dataset. For CIFAR-10, using 1 NVIDIA A100 GPU with batch size of 100, method's inference time is approximately 32 seconds over the entire test set of CIFAR-10. For ImageNet-IK, using 2 NVIDIA A100 GPUs with a batch size of 100, our method's 's inference time is approximately 3 minutes over the entire test set of ImageNet- IK. Although DAD requires training an extra denoiser and MMD-OPT, it significantly outperforms AP baselines in inference speed. Furthermore, relying on a pre-trained generative model is not always feasible, as training such models at scale can be highly resource-intensive Therefore, considering considering the trade-off between computational cost and the performance of DAD, we believe that training an additional detector and denoiser is feasible and worthwhile.
D.8. Combination with Adversarial Training
Table 12. Mixed accuracy (%) our method with and without different adversarial training methods under different proportions of AEs in each batch and different batch sizes against PGD+EOT (e 8/255). We show the most successful defense in bold.
Proportion of AEs in Each Batch (%)
Method
10
20
30
40
50
60
70
80
90
100
Batch Size 25
Vanilla AT (Madry et al., 2018)
71.89
68.80
65.72
62.63
59.55
56,46
53.38
50.29
47.21
44.12
+ Ours
72.81
69.08
66.32
62.67
60.69
56.98
54.60
50.82
48.87
45.75
TRADES (Zhang et al., 2019)
70.29
67.70
65. 11
62.52
59.92
57.33
54.74
52.15
49.56
46.97
+ Ours
70.99
67.73
65.67
63.46
60.88
57.87
55.80
52.92
50.98
48.10
MART (Wang ct al., 2020)
68.63
66.25
63.86
61.47
59.08
56.70
54.31
51.92
49.54
47.15
+ Ours
69.34
66.67
64.34
61.78
60.05
57.27
55.16
52.48
51.10
48,46
TRADES-AWY AWP (Wu et al., 2020)
69.52
67.22
64.92
62.62
60.32
58.02
55.72
53.42
51.12
48.
+ Ours
70.22
66.98
65.56
62.46
61.12
58.18
57.03
54.65
53.40
49.99
Batch Size
50
Vanilla AT (Madry et al., 2018)
71.89
68.81
65.73
62.65
59.57
56.49
53.41
50.33
47.25
44.17
+ Ours
72.84
69.57
66.55
62.93
60.74
57.12
53.88
51.21
48.82
46.91
TRADES (Zhang et al., 2019)
70.28
67.69
65.09
62.50
59.90
57.30
54.71
52.11
49.52
46.92
+ Ours
70.17
67.80
65.38
62.64
59.99
57.47
55.45
52.90
51.10
49.27
MART (Wang et al., 2020)
68.63
66.24
63.86
61.47
59.08
56.69
54.30
51.92
49.53
47.14
+ Ours
68.75
66.34
64.54
62.60
59.41
58.07
56.78
55.14
53.51
51.35
TRADES-AWP (Wu et al., 2020)
69.52
67.23
64.93
62.64
60.34
58.04
55.75
53.45
51.16
48.86
Ours
69.44
67.10
65.18
62.80
60.49
57.98
56.18
54.25
52.34
51.10
Batch Size = 100
Vanilla AT (Madry et al., 2018)
71.88
68.79
65.71
62.62
59.53
56.44
53.35
50.27
47.18 18
44.09
+ Ours
72.23
69.14
65.82
63.74
59.87
58.01
56.08
53.36
51.50
49.20
TRADES (Zhang et al., 2019)
70.28
67.69
65.09
62.49
59.89
57.30
54.70
52.10
49.51
46.91
Ours
70.36
67.79
65.22
63,58
60.62
58.31
57.04
55.51
53.38
51.58
MART (Wang et al., 2020)
68.64
66.26
63.87
61.49
59.11
56.73
54.35
51.96
49.58
47.20
Ours
68.75
66.35
64.50
62.61
59.50
58.02
56.77
54.97
53.51
51.40
TRADES-AWP (Wu et al., 2020)
69.52
67.22
64.92
62.62
60.31
58.01
55.71
53.41
$1.11
48.81
Ours
69.64
67.49
64.97
63.61
61.10
59.30
57.87
56.40
54.24
52.06
Batch Size
150
Vanilla AT (Madry et al., 2018)
71.89
68.80
65.71
62.62
59.54
56.45
53.36
50.27
47.18
44.09
+ Ours
72.06
69.17
65.96
62.86
59.68
58.92
55.09
52.39
49.90
47.97
TRADES (Zhang et al., 2019)
70.27
67.67
65.08
62.49
59.89
57.30
54.71
52.12
49.52
46,93
+ Ours
70.59
67.87
65.43
63.35
61.58
60.05
58.23
56.13
53.68
51.68
MART (Wang et al., 2020)
68.64
66.26
63.87
61.48
59.09
56.71
54.32
51.93
49.55
47.
Ours
68.56
66.48
64.07
62.06
60.99
59.37
57.47
55.69
53.55
51.68
TRADES-AWP (Wu et al., 2020)
69.53
67.23
64.93
62.63
60.34
58.04
55.74
53.44
51.14
48.
Ours
69.70
67.43
64.98
63.53
62.26
60.05
57.99
56.14
54.25
52.18
17

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
E. Detailed Related Work
Adversarial attacks. The discovery of adversarial examples (ABs) has raised a security concern for AI development in recent decades (Szegedy et al., 2014; Goodfellow et al., 2015). AEs are often crafted by adding imperceptible noise to clean images, which can easily mislead a classifier to make wrong predictions. The algorithms that generate AEs are called adversarial attacks. For example, the Fast Gradient Sign Method (FGSM) perturbs clean data in the direction of the loss gradient (Goodfellow et al., 2015). Expanding on FGSM, the Basic Iterative Method (BIM) (Kurakin et al., 2017) iteratively applies small noises to the clean data in the direction of the gradient of the loss function, updating the input at each step to create more effective AEs than single-step methods such as FGSM. Madry et al. (2018) propose the Projected Gradient Descent (PGD), which further improves the iterative approach of BIM by adding random initialization to the input data before applying iterative gradient- based perturbations. Beyond con-targeted attacks, the Carlini Wagner attack (C& W) specifically directs data towards a chosen target label, which crafts AEs by optimizing a specially designed objective function (Carlini & Wagner, 2017). AutoAttack (AA) (Croce& & Hein, 2020a) is ensemble of multiple adversarial attacks, which combines three non-target white-box attacks (Croce & Hein, 2020b) and one targeted black-box attack (Andriushchenko et al., 2020), which makes AA a benchmark standard for evaluating adversarial robustness. However, the computational complexity of AA is relatively high. Gao et al. (2022) propose the Minimum-mary faster alternative to AA. Beyond computing exact gradients, Athalye et al. (2018b) propose Expectation over Transformation (EOT) to correctly compute the gradient for defenses that apply randomized transformations to the input. Athalye et al. (2018a) propose the Backward Pass Differentiable Approximation (BPDA), which approximates the gradient with an identity mapping to effectively break the defenses that leverage obfuscated gradients. According Lee & Kim (2023), PGD+EOT is currently the best attack for denoiser-based defense methods.
attack (MMA), which can be used as a
Adversarial detection. The most lightweight method to defend against adversarial attacks is to detect and discard AEs in the input data. Previous studies have largely utilized statistics on hidden-layer features of deep neural networks (DNNs) to filter out AEs from test data. For example, Ma et al. (2018) utilize the local intrinsic dimensionality (LID) of DNN features as detection characteristics. Lee et al. (2018) implement a Mahalanobis distance- based score for identifying AEs. Raghuram et al. (2021) develop a meta-algorithm that extracts intermediate layer representations of DNNs, offering configurable components for detection. Deng al. (2021) leverage a Bayesian neural network to detect AEs, which is trained by adding uniform noises to samples. Another prevalent strategy involves equipping classifiers with rejection option. For example, Stutz et al. (2020) introduce a manfidence-calibaate
adversarial training framework, which guides the model to make low-confidence predictions on AEs, thereby determining which samples to reject. Similarly, Pang et al. (2022b) integrate confidence measures with a newly proposed R- R-Con metric to effectively separate AEs out. However, these methods, train a detector for specific classifiers or attacks, tend to neglect the modeling of data distribution, which can limit their effectiveness against unknown attacks. Recently, statistical adversarial data detection (SADD) has delivered increasing insight. For example, Gao et al. (2021) demonstrate that maximum mean discrepancy (MMD) is aware of adversarial attacks and leverage the distributional discrepancy between AEs and CEs to filter out AEs, which has been shown effective against unseen attacks. Based on this, Zhang et al. (2023) further propose a new statistic called expected perturbation score (EPS) that measures the expected score of a sample after multiple perturbations. Then, an EPS-based MMD is proposed to measure the distributional discrepancy between CEs and AEs. Despite the effectiveness of SADD, an undeniable problem of SADD- methods is that they will discard data batches that contain AEs. To solve this problem, in this paper, we propose a new defense method that does not discard any data, while also inherits the capabilities of SADD-based detection methods.
Adversarial training. Another prominent defensive framework is adversarial training (AT). Vanilla AT (Madry et al., 2018) directly generates and incorporates AEs during the training process, forcing the model to learn the underlying distributions of AEs. Besides vanilla AT, several modifications have been developed to enhance the effectiveness of AT. For instance, at the early stage of AT, Song et al. (2019) propose to treat adversarial attacks as a domain adaptation problem and enhance the generalization of AT by minimizing the distributional discrepancy. Zhang et al. (2019) propose optimizing a surrogate loss function based on theoretical bounds. Similarly, Wang et al. (2020) explore how misclassified examples influence a model's robustness, leading to an improved adversarial risk through regularization. From the perspective of reweighting. Ding et al. (2020) propose reweight adversarial data with instance-dependdett perturbation bounds e and Zhang et al. (2021) introduce geometry- aware mnstance-reweighte data points to the class boundary. Wang et al. (2021) build upon GAIRAT by leveraging probabilistic margins to reweight AEs due to their continuous nature and independence from specific perturbation paths. Zhou et al. (2023b) propose a joint adversarial defense method that combines a phase-level adversarial training mechanism to enhance robustness against phase-based attacks with an amplitude-based preprocessing operation to mitigate perturbations in the amplitude domain.
AT (GAIRAT) framework, which differentiates weights based on the proximity of
18

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
More recently, Zhang et al. (2024) propose to pixel-wisely reweight AEs by explicitly guiding them to focus important pixel regions. Other modifications include improving AT using data augmentation methods (Gowal et al., 2021; Rebuffi et al., 2021) and hyper-parameter selection methods (Gowal et al., 2020; Pang et al., 2021). Although AT achieves high robustness against particular attacks, it suffers from significant degradation in clean accuracy and high computational complexity (Wong et al., 2020; Laidlaw et al., 2021; Poursaeed et al., 2021). Different from the AT framework, our method does not train a robust classifier. Instead, by directly feeding detected CEs to a pre-trained classifier, our method can effectively maintain clean accuracy. By using a lightweight detector and denoiser model, our method can alleviate the computational complexity.
Denoiser-based adversarial defense. Another well-known defense framework denoiser- based adversarial defense, which often leverages generative models to shift AEs back to their clean counterparts before feeding them into classifier. In most literature, it is called adversarial purification (AP). Previous methods mainly focus on exploring the use of more powerful generative models for AP. For example, at the early stage of AP, Meng & Chen (2017) propose a two-step process called MagNet, which first discards detected AEs using a detector, then uses an autoencoder to purify the rest by guiding them toward the manifold of clean data. After MagNet, Liao et al. (2018) design a denoising UNet that can denoise AEs to their clean counterparts by reducing the distance between adversarial and clean data under high-level representations. Samangouei et al. (2018) use a GAN trained on clean examples to project AEs onto the generator's manifold. Song et al. (2018) find that AEs lie in low- pww-pabbbbilty regions of the image distribution and propose to maximize the probability of given test example. Naseer et al. (2020) focus on training conditional GAN, which engages in a min-max game with a critic network, to differentiate between adversarial and clean data. Yoon et al. (2021) propose to use the denoising score- score-based model to purify adversarial examples. Nie et al. (2022) propose to use diffusion models to remove adversarial noise by gradually adding Gaussian noise to AEs, and then wash out the noise by solving the reverse-time stochastic differential equation. Zhou et al. (2023a) propose to utilize complementary masks to disrupt adversarial noise and employs guided denoising models to recover robust and predictive representations from the masked samples. The success of recent AP methods often relies on the assumption that there will be pre-trained generative model that can precisely estimate the probability density of the CEs (Yoon al., 2021; Nie et al., 2022). However, even powerful generative models (c.g., diffusion models) may have an inaccurate density estimation, leading to unsatisfactory performance (Chen et al., 2024). By contrast, instead of estimating probability densities, our method directly minimizes the distributional discrepancies between AEs and CEs, leveraging the fact that identifying distributional discrepancies is simpler and more feasible than estimating density. Nayak et al. (2023) propose to use MMD as a regularizer during the training of the denoiser. Different from their work, we use an optimized version of MMD (i.e., MMD-OPP OPT), which more sensitive to adversarial attacks. Furthermore, the MMD-OPT not only acts guiding signal to minimize the distributional discrepancy between AEs and CEs, but also acts discriminator to distinguish between CEs and AEs, which kills two birds with one stone.
F. Discussions on Batch-wise Detections
Benefits of using batch-wise statistical hypothesis test. A main benefit of using a batch-wise statistical hypothesis test is that it can effectively control the false alarm rate. For example, for DAD, we set the maximum false alarm rate to be 5%. Fang et al. (2022) theoretically prove that for instance-wise detection methods work perfectly, there must be a gap in the support set between IID and futtof-distribbutin (OOD) data. This theory also applies to adversarial problems, but such support set does not exist in adversarial settings, making perfect instance-wise detection generally difficult.
Limitation and its solutions for user inference. DAD leverages statistics based on distributional discrepancies (i.c., MMD-OPT), which requires the data to be processed in batches for adversarial detection. However, when the batch size is too small, the stability of DAD will be affected (see Figure 2). To address this issue, for user inference, single samples provided by the user can be dynamically stored in a queue. Once the queue accumulates enough samples to form batch, our method can then process the batch collectively using the proposed approach. A direct cost of this solution is the waiting time, as the system must accumulate enough samples (e.g., 50 samples) to form batch before processing. However, in scenarios where data arrives quickly, the waiting time is typically very short, making this approach feasible for many real-time applications. For applications with stricter latency requirements, the batch size can be dynamically adjusted based on the incoming data rate to minimize waiting time. For instance, if the system detects a lower data arrival rate, it can process smaller batches to ensure timely responses. Overall, it is a trade-off problem: using our method for user inference can obtain high robustness, but the cost is to wait for batch processing. Based on the performance improvements our method obtains over the baseline methods, we believe the cost is feasible and acceptable. Another possible solution is to find more robust statistics that can measure distributional discrepancies with fewer samples. Recently, measuring the expected score of sample after multiple perturbations has proven useful for this purpose (Zhang et al., 2023). However, computing the
19

---

One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy
expected score is time-consuming. We emphasize that this paper primarily focuses the relationship between distributional discrepancies and adversarial risk, aiming inspire the design of new defense method. We leave it future work.
Practicability beyond user inference. On the other hand, our method is not necessarily used for user inference. Instead, our method is suitable for cleaning the data before fine-tuning the underlying model. In many domains, obtaining large quantities of high-quality data is challenging due to factors such as cost, privacy concerns, or the rarity of specific data. As result, possible samples with clean information are critical in these data-scarce domains. Then, practical scenario is that there exists pre-trained model on a large-scale dataset g., DNN trained on ImageNet- and clients want to fine-tune the model to perform well on downstream tasks. the data for downstream tasks contain AEs, our method can be applied batch-wisely wisely clean the data before fine-ture fine-tuning the underlying model.
20


Reviews:
[{"review_id": "tAt5MUIp9x", "reviewer": "Reviewer_Ryke", "summary": "The paper introduces Distributional-Discrepancy-based Adversarial Defense (DDAD), a two-pronged approach that leverages Maximum Mean Discrepancy (MMD) to detect adversarial examples (AEs) and a denoiser to restore them. The paper provides a theoretical justification linking distributional discrepancy minimization to a reduction in expected adversarial loss. Then the MMD-OPT is used for training the detector and denoiser.  Extensive experiments on CIFAR-10 and ImageNet-1K demonstrate that DDAD achieves higher clean and robust accuracy than state-of-the-art (SOTA) adversarial defense methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Comparison to MagNet\nWe acknowledge that MagNet [1] is a very good work. The main reasons we did not include MagNet as a baseline are: (1) MagNet is outdated since it was published 8 years ago, and (2) MagNet cannot defend against adaptive attacks, placing it significantly behind current SOTA methods. Please kindly check Table 1.\n\nTable 1: Clean and robust accuracy (%) against adaptive white-box AutoAttack and PGD+EOT attacks on CIFAR-10. We show the most successful defense in **bold**. \n|Method|Clean|AutoAttack|PGD+EOT|\n|-|-|-|-|\n|WRN-28-10|\n|MagNet|84.53|0.00|0.00|\n|DDAD|**94.16**|**72.21**|**67.53**|\n\n[1] MagNet: A Two-pronged Defense against Adversarial Examples.\n\n## 2. Novelty of Theoretical Analysis\nThe theoretical bound we derive is **tighter** than previous work [1] [2].  Previously, the upper bound of the risk on the target domain is **always** bounded by one extra constant, e.g., $R(h, f_\\mathcal{A}, \\mathcal{D_A}) \\leq R(h, f_\\mathcal{C}, \\mathcal{D_C}) + d_1(\\mathcal{D_C}, \\mathcal{D_A}) + C$.  If $C$ is large,  minimizing $d_1(\\mathcal{D_C}, \\mathcal{D_A})$ can hardly reduce  $R(h, f_\\mathcal{A}, \\mathcal{D_A})$.  In contrast, we derive an upper bound **without any extra constant C**, which means minimizing $d_1(\\mathcal{D_C}, \\mathcal{D_A})$ can more effectively reduce $R(h, f_\\mathcal{A}, \\mathcal{D_A})$. This is a **major novelty** of our theoretical analysis.\n\n[1] Domain adaptation: Learning bounds and algorithms.\n\n[2] A theory of learning from different domains.\n\n## 3. Novelty of DDAD\nIn our humble opinion, our proposed method is significantly different from existing SOTA AP methods. We summarize the novelty of DDAD from several key perspectives:\n- **Philosophically**, DDAD focuses on minimizing distributional discrepancies, which fundamentally differs from existing AP methods relying primarily on density estimation.\n- **Theoretically**, we derive a tighter and more informative theoretical bound to support the design of DDAD, which is a major contribution of our work.\n- In terms of **training efficiency**, precise density estimation typically requires powerful models (e.g., diffusion models) that are computationally intensive and time-consuming to train. In contrast, learning distributional discrepancies is inherently simpler and more feasible, making DDAD both effective and efficient.\n- In terms of **inference efficiency**, diffusion-based AP methods suffer from slow inference speeds due to repeated calls to the forward process of diffusion models. Our method, however, achieves significant performance improvements without compromising inference speed.\n\n## 4. Adaptive Attack\nAccording to [1], PGD+EOT is considered the **strongest** attack against diffusion-based AP methods, and AutoAttack is **strongest** against AT-based methods. Our proposed adaptive attacks (both PGD+EOT and AutoAttack) additionally target DDAD's detection mechanism (i.e., MMD-OPT), and adaptive PGD+EOT proves **most effective** in breaking DDAD. For **gradient obfuscation checks**, DDAD achieves the **best** average performance against adaptive BPDA+EOT attack across various baselines (**see Table 3 in our paper**).  Also, please kindly check the results against adaptive AutoAttack in **Section 2 & 3 of Reviewer 3bTb's responses**.\n\n[1] Robust Evaluation of Diffusion-based Adversarial Purification.\n\n## 5. Transferability Across Different Defenses\n- Thank you for your concern! We would like to clarify that the purpose of conducting transferability experiments is not to claim SOTA performance, but rather to ensure our method maintains good robustness against unseen attacks, as it relies on AEs to train MMD-OPT and the denoiser. Table 4 demonstrates our method's strong transferability across different attacks, architectures, and perturbation budgets. \n- Due to the time-consuming nature of these baseline experiments, we will upload the corresponding results once completed. In the meantime, we provide some intuitions here: AT-based methods are expected to exhibit weaker transferability, as they encounter specific AEs during training [1], while AP-based methods are likely to demonstrate better transferability since they are independent of AE types [2].\n\n[1] Geometry-aware Instance-reweighted Adversarial Training.\n\n[2] Diffusion Models for Adversarial Purification.\n\n## 6. Clarification of MMD-OPT\nMMD-OPT builds upon MMD, which has a learnable kernel $k_w$. We can obtain optimized $k_w$ (we denote it as $k^*_w$) by maximizing Eq.(5) in our paper. Then, MMD-OPT is the MMD estimator with an optimized kernel $k^*_w$. During training, MMD-OPT serves as an objective to update the denoiser's parameters. During inferencing, MMD-OPT serves as a metric to measure the distributional discrepancies between two batches. We will clarify it in the updated version of our paper!\n\n---\nThank you for your time! If our response has resolved your major concerns, we would appreciate a higher score :)", "claims_and_evidence": "**CLAIM 1**: The paper proposed a novel DDAD method.\n* EVIDENCE 1: The paper introduces DDAD, which integrates Maximum Mean Discrepancy (MMD) for adversarial detection and a denoiser for adversarial purification. The method is novel in that it does not discard detected adversarial examples but instead attempts to restore them using a denoiser. The authors provide a clear methodology, including the training and inference process, as well as a theoretical justification for minimizing distributional discrepancy to reduce adversarial risk (Sections 3 and 4).\n* CONCERN 1: While the combination of MMD-based detection with a denoiser is a meaningful extension, the novelty might be incremental given existing adversarial purification techniques. The authors compare their method to adversarial training (AT) and adversarial purification (AP), but they do not extensively benchmark against prior hybrid methods. Although the authors claim that the previous two-pronged method, MagNet (Meng& Chen, 2017), is outdated, it is not fair to exclude them. Without the comparison with MagNet (Meng & Chen, 2017), the advance in the landscape of two-pronged methods is not clear.\n\n**CLAIM 2**: DDAD can improve clean and robust accuracy by a notable margin against well-designed adaptive white-box attacks.\n* EVIDENCE 2: The paper provides experimental results in Tables 1 and 2, showing that DDAD outperforms state-of-the-art adversarial training and purification methods in both clean and robust accuracy under adaptive white-box attacks (PGD+EOT and AutoAttack). The results are consistent across different architectures and datasets (CIFAR-10 and ImageNet-1K). Additionally, the ablation studies (Section 5.4) indicate that both MMD-OPT and the denoiser contribute to the improved performance.\n* CONCERN 2: The paper claims to evaluate against adaptive white-box attacks, but it primarily uses existing attack frameworks (PGD+EOT and AutoAttack). The authors implement an “adaptive” attack specific to DDAD (Algorithm 3), but it is unclear if this attack is truly optimized to break the defense. A stronger evaluation would involve gradient obfuscation checks and testing stronger adaptive attacks, such as AutoAttack variants specifically tuned against detection-based methods. Without this, the robustness improvements could be overestimated.\n\n**CLAIM 3**: DDAD can generalize well against unseen transfer attacks (see Section 5.3).\n* EVIDENCE 3: The paper presents results in Table 4, showing that DDAD-trained models maintain high robust accuracy against transfer attacks from different model architectures (e.g., WideResNet-28-10 -> ResNet-50, Swin Transformer). The experiments use PGD+EOT l_inf and C&W l2 attacks with various perturbation budgets, demonstrating DDAD’s transferability.\n* CONCERN 3: The paper does not compare DDAD’s transfer robustness against other methods—while the absolute numbers are reported, it is unclear if DDAD generalizes better than baseline adversarial purification or training methods in the same setting. A fairer evaluation would compare relative transfer robustness across different defenses.", "methods_and_evaluation_criteria": "**Method**: MMD-based detection aligns well with the problem of distinguishing adversarial vs. clean distributions. Unlike traditional SADD-based methods that discard adversarial examples, DDAD attempts to recover adversarial examples, which could preserve useful information. The theoretical justification (Section 3) provides a formal grounding for the method by linking distributional discrepancy minimization to reducing adversarial risk.\n\n**Evaluation**: The authors evaluate DDAD on CIFAR-10 and ImageNet-1K, two well-established image classification benchmarks in adversarial robustness research, using standard metrics (clean and adversarial accuracy). The comparison includes state-of-the-art adversarial training (AT) and adversarial purification (AP) methods.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "The benchmark used CIFAR-10 and ImageNet-1K which are widely adopted in the area. The method is compared to multiple methods but the two-pronged method is ignored, which is my major concern.", "relation_to_broader_scientific_literature": "The paper builds on prior work in adversarial detection, purification, and theoretical robustness analysis by proposing DDAD, a hybrid approach that detects and denoises adversarial examples instead of discarding them. It extends statistical adversarial detection methods (e.g., MMD-based detection) by introducing MMD-OPT, an optimized discrepancy measure for more effective adversarial differentiation. Unlike prior adversarial purification methods that rely on generative models (e.g., GANs, diffusion models), DDAD uses distributional discrepancy minimization for adversarial recovery, improving computational efficiency. The paper also provides theoretical insights, linking distributional discrepancy reduction to adversarial risk minimization, refining previous bounds in domain adaptation and adversarial training. These advancements position DDAD as a novel, theoretically grounded, and empirically validated defense mechanism.", "essential_references_not_discussed": "Not found.", "other_strengths_and_weaknesses": "Strength:\n* The authors gave an in-depth understanding of the success of the proposed DDAD method.\n\nWeakness\n* (minor) The contributions of the paper are not easy to capture from the introduction. What is the novelty of the theoretical analysis? Existing theoretical insights are not revisited leaving the novelty unclear. I tried to summarize my understanding but the points are not clear from the paper.", "other_comments_or_suggestions": "In the abstract, the high-level concept of MMD-OPT is not clearly introduced, causing some confusion. Is it a loss function?", "questions_for_authors": "* Are there other major methodological and theoretical contributions in the paper except the proposed DDAD method? What is the major novelty in the DDAD compared to previous methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4KGTbDGwJX", "reviewer": "Reviewer_3Zsd", "summary": "This paper proposes a two stages adversarial defence method based on distribution discrepancy between clean samples and adversarial samples. Firstly, the authors train a model called MMD-OPT by maximise the MMD between distribution of clean data and adversarial data. Then, the MMD-OPT can act as a guidance to train a denoiser to denoise the adversarial sample to the clean one. During the inference stage, the MMD-OPT can act as a detector to detect adversarial samples, and the trained denoiser can purify the perturbation inside the adversarial samples. With the purified data and clean data, the classifier will obtained higher accuracy on this mixed dataset. Experiments are conducted on CIFAR10 and ImageNet-1K demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Evidence of 'Minimizing distributional discrepancy can reduce the expected loss on AEs.'\nIn our paper, we derive a theoretical bound to support our claim, i.e., $R(h, f_\\mathcal{A}, \\mathcal{D_A}) \\leq R(h, f_\\mathcal{C}, \\mathcal{D_C}) + d_1(\\mathcal{D_C}, \\mathcal{D_A})$. In previous literature [1] [2], the upper bound of the risk on the target domain is **always** bounded by one extra constant, e.g., $R(h, f_\\mathcal{A}, \\mathcal{D_A}) \\leq R(h, f_\\mathcal{C}, \\mathcal{D_C}) + d_1(\\mathcal{D_C}, \\mathcal{D_A}) + C$.  If $C$ is large,  minimizing $d_1(\\mathcal{D_C}, \\mathcal{D_A})$ can hardly reduce  $R(h, f_\\mathcal{A}, \\mathcal{D_A})$.  In contrast, we derive an upper bound **without any extra constant C**, which means minimizing $d_1(\\mathcal{D_C}, \\mathcal{D_A})$ can more effectively reduce $R(h, f_\\mathcal{A}, \\mathcal{D_A})$. This is a **major contribution** of our work. We will clarify this more in the updated version of our paper!\n\n[1] Domain adaptation: Learning bounds and algorithms.\n\n[2] A theory of learning from different domains.\n\n## 2. Recent SOTA Baseline Comparison\nWe compare several **most recent SOTA defense methods** on RobustBench with DDAD. We will include the results in the updated version of our paper.\n\nTable 1: Clean and robust accuracy (%) of recent SOTA methods on CIFAR-10. We show the most successful defense in **bold**. [3]* uses an ensemble of networks (ResNet-152 + WRN-70-16) and 50M synthetic images.\n|Method|Clean|AutoAttack|Avg|\n|-|-|-|-|\n|WRN-28-10|\n|[1]|92.16|67.73|79.95|\n|[2]|92.44|67.31|79.88|\n|DDAD|**94.16**|**72.21**|83.19|\n|WRN-70-16|\n|[2]|93.25|70.69|81.97|\n|[3]*|**95.19**|69.71|82.45|\n|DDAD|93.91|**72.58**|**83.25**|\n\n[1] Decoupled Kullback-Leibler Divergence Loss, NeurIPS 2024\n\n[2] Better Diffusion Models Further Improve Adversarial Training, ICML 2023\n\n[3] MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers, TMLR 2024\n\n## 3. Plot Other Methods in Figure 2\nThank you for your suggestion! The main reason is that including all baseline methods will make Figure 2 look **very messy**. Therefore, we use a table (please see Table 5 in our paper) in Appendix E.3 to clearly compare our method with baseline methods under different proportions of AEs in a batch.\n\n## 4. Discuss Other Distributional Discrepancy Metrics\nWe discuss two representative statistical alternatives of MMD here: **Wasserstein distance** and **energy distance**.\n- Compared to Wasserstein distance, MMD has two major advantages: \n    - The estimator of MMD is **unbiased**, while the estimator of Wasserstein distance is **biased**. Therefore MMD estimator is **more accurate** than Wasserstein estimator, especially when the dimension of the data is large. \n    - Wasserstein distance requires solving a transportation problem, which is **computationally more expensive than MMD**, which means using Wasserstein distance will be slow for large datasets.\n- Compared to energy distance, **MMD offers greater flexibility** due to its use of kernel functions, which allow it to capture intricate differences between distributions and adapt to various tasks by selecting appropriate kernels. It is particularly well-suited for high-dimensional data, as its reliance on embeddings in a reproducing kernel Hilbert space **mitigates issues like the \"curse of dimensionality\" that can affect energy distance.** Moreover, MMD is sensitive to higher-order statistics of distributions, enabling it to **capture subtle discrepancies beyond mean and variance**.\n\nWe will add a short section to discuss these statistical measurements in the updated version of our paper!\n\n## 5. Assumptions for Training Setting\nTo train the MMD-OPT, we only require access to the **clean training data**. AEs used for MMD-OPT are generated based on the clean training data. This assumption is reasonable and commonly used in adversarial training-based methods and other defenses requiring AEs during training.\n\n## 6. Ablation study of $\\alpha$ in Eq.(8) \nIncreasing $\\alpha$ in Eq.(8) means focusing less on the effect of MMD-OPT, and thus the robust accuracy will decrease, as supported by our theoretical analysis. We will include the results in the updated version of our paper!\n\nTable 1: Our clean and robust accuracy (%) under different $\\alpha$ against adaptive white-box PGD+EOT $\\ell_\\infty (\\epsilon = 8/255)$ on CIFAR-10. \n|$\\alpha$|Clean|PGD+EOT|\n|---|---|---|\n|0.01|94.16|67.53|\n|0.05|94.16|50.70|\n|0.1|94.16|45.35|\n\n## 7. AEs in the Latent Space\nThank you for your insightful question! The philosophy is grounded in the findings of [1]: Regardless of the type of latent space (Hilbert space or otherwise), if adding perturbations into the latent representation of CEs does not increase the MMD value significantly, it indicates minimal distributional discrepancy between AEs and CEs. Consequently, these AEs can hardly deceive classifiers.\n\n[1] Maximum Mean Discrepancy Test is Aware of Adversarial Attacks, ICML 2021", "claims_and_evidence": "The authors made several claims in the paper. However, I do think several claims lack clear evidence. For example, the paper states that minimizing distributional discrepancy can reduce the expected loss on adversarial examples (AEs). This is quite intuitive, however, there is no evidence to support this claim, either from theoretical aspect or the empirical aspect.", "methods_and_evaluation_criteria": "The proposed method is a two-stage method, whose idea is to regard the test data includes both clean samples and adversarial samples. The method firstly identifies the adversarial samples within the dataset, then purifies these adversarial samples. For the detector of adversarial samples, the authors propose to this detector by maximising its output of MMD distance between clean samples and adversarial samples. The detector then acts as a guidance for the denoiser network of adversarial samples to make the denoised data has smaller MMD distance with the clean data. This whole pipeline is clear motivated and makes sense.", "theoretical_claims": "I do not check the theorem in the paper very carefully. For me, the contribution mainly lies in the proposed pipeline. The theoretical part is just an auxiliary to improve the soundness of motivation from theoretical aspects.", "experimental_designs_or_analyses": "The experiment part follows common practice of adversarial defence problems in the field, which evaluate the proposed method with small-scale CIFAR10 dataset and large-scale ImageNet-1K dataset. However, I think the compared methods are somehow too out-of-date since most of them are published at conferences at least two years ago. More recent methods are needed to be added to compare with. For the ablation study about the ratio of AEs within the mixed dataset, I am wondering why do not plot the curves of other compared methods in Figure 2.", "relation_to_broader_scientific_literature": "This paper may have potential to benefit the researchers in the field of adversarial defence and adversarial purification.", "essential_references_not_discussed": "I do not have more references to suggest.", "other_strengths_and_weaknesses": "Weaknesses:\n\n* The authors highlight the effectiveness of using distributional discrepancy to indicate clean samples and adversarial samples. However, there is no clear evidence to support this point. Additionally, the authors choose to use MMD to measure the discrepancies, there is a need to discuss or compare with other distributional discrepancies.\n\n* In this paper, the authors assume that we have access to a set of labeled clean data and adversarial data. Given these labeled data, the authors are able to train the MMD-OPT to act as a guidance to train the denoiser in the next step. However, this assumption is not satisfied in the plain adversarial defence setting.\n\n* The experiment part lacks an important ablation study, which is to the effectiveness of two terms in Eq. (8) respectively. As indicated in line 272, $\\alpha$ is set to be 0.01, which is a quite small value. How much does the cross-entropy constraint will affect the performance of  the trained denoiser?", "other_comments_or_suggestions": "I do not have further comments. Please refer to weaknesses and questions part.", "questions_for_authors": "A question just comes up to me after I read the paper. For an adversarial sample, it can be viewed as some invisible perturbation $\\epsilon$ into the clean image so that the perturbed image can cheat the classifier with no clear change from the visual perspective. While the perturbation in adversarial images is quite minor, the distance of distribution between AEs and CEs might not be too large. However, as suggested by the authors, it seems that after projecting AEs and CEs into the Hibert space, the distance between these two projected distributions becomes larger. Does this phenomenon can also be observed in other latent space? Moreover, what if we add perturbation into the Hibert space of CEs to form the adversarial samples? Does MMD can still be used indicate CEs and AEs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "947TZS1Uv7", "reviewer": "Reviewer_3bTb", "summary": "This work first validates the effectiveness of the SADD-based approach through theoretical analysis and mathematical proofs. To address the limitations of traditional SADD-based methods in utilizing AEs, the authors innovatively propose the DDAD method. By introducing an additional denoiser training module, the proposed method effectively eliminates adversarial noise interference, thereby significantly improving data utilization in data-constrained scenarios. Experimental results demonstrate that the DDAD method achieves SOTA performance on two benchmark datasets, providing a novel solution for enhancing model robustness in few-shot learning scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Rendering Errors of  LaTeX\nThank you for pointing out this unexpected rendering issue caused by LaTex! The entire sentence in line 112 is: *'...the ground-truth labelling functions for the clean and adversarial domains are equal in our problem setting.'* This statement is supported based on Assumptions 1&2. We will fix this issue in the updated version of our paper!\n\n## 2 & 3. Clarification of Experimental Settings\nSorry for the confusion! Indeed, we did not omit AutoAttack results intentionally. Instead, we follow the principle of evaluating each method under its **worst-case scenario** (i.e., the strongest attack against that method). Therefore, 'Robust Accuracy' in Tables 1 and 2 refers to:\n- For AT-based methods, we measure the robust accuracy of AutoAttack, since it is empirically the strongest attack for AT-based methods, as demonstrated by [1].\n- For AP-based methods, we measure the robust accuracy of PGD+EOT attack, since it is empirically the strongest attack for AP-based methods, as shown in [2].\n- For DDAD, we measure the robust accuracy of adaptive PGD+EOT attack, which additionally targets both the denoiser and the detector modules (see Algorithm 3). We empirically observed that adaptive white-box PGD+EOT provides the worst-case scenario for DDAD compared to adaptive white-box AutoAttack.\n\nThus, each defense method is evaluated **under its own strongest attack setting**, ensuring a **relatively fair** comparison since it mitigates evaluation bias (e.g., AT-based methods perform well on PGD+EOT because they have seen PGD examples during training). Similarly, if we only consider attacking the denoiser and the classifier (i.e., the white-box setting for AP-based methods), DDAD can achieve around 77% robustness on PGD+EOT and 81% on AutoAttack, which is **clearly not fair** for both AT and AP methods. We provide the full results in Table 1 below.\n\nTable 1: Clean and robust accuracy (%) against adaptive white-box PGD+EOT ($\\ell_\\infty, \\epsilon = 8/255$) and adaptive white-box AutoAttack ($\\ell_\\infty, \\epsilon= 8/255$) on CIFAR-10. * means this method is trained with extra data. We show the most successful defense in **bold**.\n\n|Type|Method|Clean|PGD+EOT|AutoAttack|\n|---|---|---|---|---|\n|AT|Gowal et al. (2021)|87.51|66.01|63.38|\n|AT|Gowal et al. (2020)*|88.54|65.10|62.76|\n|AT|Pang et al. (2022a)|88.62|64.95|61.04|\n|AP|Yoon et al. (2021)|85.66|33.48|59.53|\n|AP|Nie et al. (2022)|90.07|46.84|63.60|\n|AP|Lee & Kim (2023)|90.16|55.82|70.47|\n|Ours|DDAD|**94.16**|**67.53**|**72.21**|\n\n[1] RobustBench: a standardized adversarial robustness benchmark, NeurIPS D&B 2021.\n\n[2] Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV 2023.\n\n## 4. Training and Inference Efficiency\nWe provide comparisons of training time with 3 representative AT-based methods [1][2][3] in Table 1. Notably, the current SOTA AT method [4] requires generating 50M synthetic images, making it extremely time-consuming. The point we want to highlight is: even compared to simpler AT methods, our method demonstrates significantly better efficiency. We also provide comparisons of inference time with 2 SOTA diffusion-based AP methods [5][6] in Table 2.\n\nTable 1: Training time (hours: minutes: seconds) of different methods on CIFAR-10 with 2 x NVIDIA A100. The target model is RN-18.\n|Method|Training Time|\n|-|-|\n|[1]|00:55:54|\n|[2]|01:27:28|\n|[3]|01:04:19|\n|DDAD|00:28:17|\n\nTable 2: Inference time per image (seconds) of different methods on CIFAR-10 with 1 x NVIDIA A100. The target model is WRN-28-10.\n|Method|Inference Time per Image|\n|-|-|\n|[5]|3.934|\n|[6]|14.902|\n|DDAD|0.003|\n\n[1] Towards Deep Learning Models Resistant to Adversarial Attacks, ICLR 2018.\n\n[2] Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019.\n\n[3] Improving Adversarial Robustness Requires Revisiting Misclassified Examples, ICLR 2020.\n\n[4] Better Diffusion Models Further Improve Adversarial Training, ICML 2023.\n\n[5] Diffusion Models for Adversarial Purification, ICML 2022.\n\n[6] Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV 2023.\n\n## 5. Sensitivity of DDAD to the Threshold Values\nThe performance degradation occurs in ImageNet-1K, and this sensitivity can be attributed to 2 major reasons: (1) the evaluation of ImageNet-1K uses AEs with lower perturbation budgets compared to CIFAR-10, which makes it reasonable to use a smaller threshold value for ImageNet-1K. (2) ImageNet-1K is a large-scale dataset, so it may require more samples to optimize MMD-OPT. It is possible that the current MMD-OPT for ImageNet-1k has not been fully optimized, leading to such sensitivity (for now, we only use 1000 training samples to train MMD-OPT for ImageNet-1K). Luckily, we still have a range of threshold values that can produce stable and competitive results on ImageNet-1K. In the future, it would be interesting to see whether MMD-OPT will be more stable to threshold values on ImageNet-1K if we increase the training samples.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, I have thoroughly verified and validated all mathematical derivations and formula proofs presented in this work.", "experimental_designs_or_analyses": "Yes, I have checked the experimental designs and analyses.", "relation_to_broader_scientific_literature": "This work builds upon the existing SADD-based framework while addressing its limitations through the innovative integration of a denoiser module. Experimental results demonstrate significant performance gains, validating the effectiveness of this architectural improvement.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The strengths of this work have been previously highlighted. Below are some questions and concerns regarding this study:\n1.\tIn Remark 1 of the 'Problem Setting' section, there appears to be a potential typographical issue in the final sentence: '..., the ground-truth ial domains are equal in our problem setting'. Could the authors please verify and clarify this statement?\n2.\tIn the 'Evaluation Settings' subsection of the 'Experiments' section, the authors state: \"Notably, we find that our method achieves the worst case robust accuracy on adaptive white-box PGD+EOT attack, Therefore, we report the robust accuracy of our method on adaptive white-box PGD+EOT attack for Table 1 and 2.\" Could the authors please clarify:\na) The authors mention using both PGD+EOT and AutoAttack for evaluation at the beginning of this section, but only report PGD+EOT results in Tables 1 and 2. Could you explain why AutoAttack results were omitted?\n3.\tRegarding the experimental design in the 'Evaluation Settings' subsection, the authors employ different attack methods to evaluate AT, AP, and the proposed method. Could the authors please:\na) Justify the fairness of this experimental setup?\nb) Clarify the specific meaning of 'Robust Accuracy' in Tables 1 and 2, given that different attacks were used for different methods?\nA more consistent evaluation framework using uniform attack methods across all compared techniques would strengthen the comparative analysis.\n4.\tWhile the authors describe computational resources in Section E.8, could the authors please provide a more detailed comparison of the training efficiency with traditional methods?\n5.\tIn the ablation studies, Table 7 shows the \"Sensitivity of DDAD to the threshold\" where the robust accuracy drops to nearly zero when the threshold value increases from 0.01 to 0.03. Could the authors please:\na) Explain this dramatic performance degradation?\nb) Provide insights into the underlying mechanism causing this sensitivity?", "other_comments_or_suggestions": "See ‘Other Strengths And Weaknesses’", "questions_for_authors": "See ‘Other Strengths And Weaknesses’", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZZOo3R9G9q", "reviewer": "Reviewer_3JF8", "summary": "This paper proposes DDAD (Distributional-Discrepancy-based Adversarial Defense), a novel two-pronged adversarial defense method that leverages statistical adversarial data detection (SADD) to improve robustness against adversarial attacks. The paper's contributions include:\n\n1. Demonstrates that minimizing distributional discrepancy (via Maximum Mean Discrepancy, MMD) can reduce adversarial risk.\n\n2. Introducing Two-Pronged Defense Mechanism: Combines: 1)Detection: Uses an optimized MMD (MMD-OPT) to distinguish clean examples (CEs) from adversarial examples (AEs). 2) Denoising: Applies a denoiser to transform detected AEs before classification, instead of discarding them.\n\n3. Extensive experiments on CIFAR-10 and ImageNet-1K, showing improved clean and robust accuracy over state-of-the-art (SOTA) adversarial defenses against adaptive white-box attacks", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## 1. Batch-wise Processing\nIn our humble opinion, the practicality of a method should be evaluated in the context of specific scenarios and application requirements, which means there is no absolute 'practical' or 'impractical' method. The key message we want to deliver here is: **batch-wise evaluation is not impractical, but it will have some costs**:\n- **Proposed solution:** For user inference, single samples provided by the user can be dynamically stored in a queue. Once the queue accumulates enough samples to form a batch, our method can then process the batch collectively using the proposed approach.\n- **Costs for this solution:** The main cost is waiting time to accumulate enough samples (e.g., 50). However, in high-throughput scenarios (e.g., Google's terminals), this delay is minimal (often <2 seconds). For applications with stricter latency requirements, the batch size can be dynamically adjusted based on the incoming data rate to minimize waiting time. For instance, if the system detects a lower data arrival rate, it can process smaller batches to ensure timely responses.\n- **Comparison with SOTA AP methods:** Diffusion-based AP methods can handle single-sample inputs but suffer from slow inference speeds (e.g., DiffPure [1] takes ~4 seconds per CIFAR-10 image on an A100 GPU). In contrast, our method averages only ~0.003 seconds per image. Assuming there are 1000 images, DiffPure would take 4000 seconds to complete the inference, while our method only takes 3 seconds. Therefore, if the waiting time to form a batch is less than 3997 seconds, our method is more time-efficient than DiffPure. Thus, diffusion-based AP methods can hardly be applied to a system where data arrives quickly. Instead, our method can handle it, demonstrating that batch-wise evaluation is not impractical.\n\nOverall, it is a trade-off problem: using our method for user inference can obtain high robustness, but the cost is to wait for batch processing. Based on the performance improvements our method obtains over the baseline methods and the fact that current SOTA AP methods are generally slow at inference, we believe the cost is feasible and acceptable.\n\n[1] Diffusion Models for Adversarial Purification, ICML 2022.\n\n## 2. Training and Inference Efficiency \nWe provide comparisons of training time with 3 representative AT-based methods [1][2][3] in Table 1. Notably, the current SOTA AT method [4] requires generating 50M synthetic images, making it extremely time-consuming. The point we want to highlight is: even compared to simpler AT methods, our method demonstrates significantly better efficiency. We also provide comparisons of inference time with 2 SOTA diffusion-based AP methods [5][6] in Table 2.\n\nTable 1: Training time (hours: minutes: seconds) of different methods on CIFAR-10 with 2 x NVIDIA A100. The target model is RN-18.\n|Method|Training Time|\n|-|-|\n|[1]|00:55:54|\n|[2]|01:27:28|\n|[3]|01:04:19|\n|DDAD|00:28:17|\n\nTable 2: Inference time per image (seconds) of different methods on CIFAR-10 with 1 x NVIDIA A100. The target model is WRN-28-10.\n|Method|Inference Time per Image|\n|-|-|\n|[5]|3.934|\n|[6]|14.902|\n|DDAD|0.003|\n\n[1] Towards Deep Learning Models Resistant to Adversarial Attacks, ICLR 2018.\n\n[2] Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019.\n\n[3] Improving Adversarial Robustness Requires Revisiting Misclassified Examples, ICLR 2020.\n\n[4] Better Diffusion Models Further Improve Adversarial Training, ICML 2023.\n\n[5] Diffusion Models for Adversarial Purification, ICML 2022.\n\n[6] Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV 2023.\n\n## 3. Adaptive Attack\nAccording to [1], PGD+EOT is considered the **strongest** attack against diffusion-based AP methods, while AutoAttack is **strongest** against AT-based methods. Our proposed adaptive attacks (both PGD+EOT and AutoAttack) additionally target DDAD's detection mechanism (i.e., MMD-OPT), and adaptive PGD+EOT proves most effective in breaking DDAD. Moreover, Table 3 in our paper demonstrates DDAD achieves the **best** average performance against **adaptive BPDA+EOT attack** across various baselines. Finally, please kindly check the results against adaptive AutoAttack in **Section 2 & 3 of Reviewer 3bTb's responses**.\n\n[1] Robust Evaluation of Diffusion-based Adversarial Purification, ICCV 2023.\n\n## 4. Applicability of DDAD\nThank you for your concern! As specified in our problem setting, we primarily focus on robust classification, which is the standard setting adopted by most existing defense methods. In our humble opinion, even though classification is relatively straightforward, achieving fully robust model predictions remains challenging, indicating that robustness would likely be even harder to attain in more complex tasks. However, we agree that generalizing DDAD to other tasks/domains is an interesting direction and we leave it as future work.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "Pros:\n1. The combination of MMD-based detection and adversarial denoising is novel and addresses the limitations of traditional adversarial detection methods. Unlike previous SADD-based methods that discard detected adversarial samples, DDAD recovers useful information via denoising.\n\n2. Establishes a formal connection between adversarial risk and distributional discrepancy. Proves that minimizing MMD can reduce the expected loss on adversarial examples, strengthening the theoretical foundation of SADD-based defenses.\n\n3. Evaluates multiple model architectures (ResNet, WideResNet, Swin Transformer). Tests against strong adaptive white-box attacks (PGD+EOT, AutoAttack). Demonstrates generalization to transfer attacks and robustness improvements on ImageNet-1K.\n\n4. Unlike generative-based adversarial purification methods, DDAD does not rely on explicit probability density estimation, which is often unreliable for large datasets. Balances robustness and accuracy trade-offs better than existing denoiser-based approaches.\n\nCons:\n1. DDAD relies on batch-wise processing for adversarial detection, requiring a minimum batch size for stable performance. This limitation makes real-time, single-sample inference challenging, which reduces practicality for real-world applications.\n\n2. The training phase requires optimizing MMD-OPT and training a separate denoiser, which introduces additional computational overhead.\nThe paper does not compare training time or efficiency trade-offs with SOTA defenses.\n\n3. While DDAD is evaluated against PGD+EOT and AutoAttack, stronger adaptive adversaries (e.g., BPDA, multi-step AutoAttack, or gradient-free query-based methods) could be tested. Adaptive attacks that target the MMD-OPT feature space should be explored.\n\n4. The method is evaluated primarily on CIFAR-10 and ImageNet-1K, but its applicability to real-world security-sensitive domains (e.g., medical imaging, autonomous driving) is not discussed. How well DDAD generalizes to non-vision domains (e.g., NLP, speech models) remains an open question.", "theoretical_claims": "yes, it is ok", "experimental_designs_or_analyses": "see Methods And Evaluation Criteria", "relation_to_broader_scientific_literature": "see Methods And Evaluation Criteria", "essential_references_not_discussed": "see Methods And Evaluation Criteria", "other_strengths_and_weaknesses": "see Methods And Evaluation Criteria", "other_comments_or_suggestions": "see Methods And Evaluation Criteria", "questions_for_authors": "see Methods And Evaluation Criteria", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "no need"}}]

Author rebuttal:

Meta-review: This paper introduces DDAD, a novel adversarial defense method that combines detection and denoising to improve model robustness. Unlike traditional approaches that discard detected adversarial examples (AEs), DDAD uses Maximum Mean Discrepancy (MMD-OPT) to distinguish clean and adversarial distributions and then applies a denoiser to restore AEs, preserving useful information. Theoretically, the paper proves that minimizing distributional discrepancy reduces adversarial risk, deriving a tighter bound than prior work. Empirically, DDAD achieves state-of-the-art performance on CIFAR-10 and ImageNet-1K against strong adaptive attacks (PGD+EOT, AutoAttack), outperforming adversarial training (AT) and purification (AP) baselines while maintaining computational efficiency. 

Before the rebuttal, reviewers raised several key concerns. The adaptive attack evaluation is incomplete, and some stronger attacks (e.g., BPDA, AutoAttack variants) are required to be evaluated to ensure robustness claims hold. Furthermore, the comparison with older two-pronged methods like MagNet, is not included in baselines. In addition, the theoretical contribution, while sound, was not clearly highlighted in the paper, making it difficult to assess its significance. During the rebuttal, the authors addressed these concerns effectively. Additional results showed MagNet fails against adaptive attacks, and more comparisons with recent SOTA methods are provided. For adaptive attacks, the authors provide additional results against BPDA+EOT. For the theoretical contribution, the authors emphasized that their bound eliminates an extra constant (C) present in prior work, makingthe  defense more effective. The paper presents a theoretically grounded and empirically strong defense method with clear advantages over existing approaches. The authors’ responses adequately address the concerns. Thus, I recommend the acceptance.
Decision: Accept (poster)