Adversary Aware Optimization for Robust Defense
Authors: Daniel Wesego
Pedram Rooshenas
Venue: NeurIPS 2025 poster ()
Keywords: Adversarial robustness
Adversarial purification
optimization
Adversarial training
Primary area: general_machine_learning
TL;DR: We propose a novel adversarial purification technique based on optimization that regards the attack landscape.

Abstract: Deep neural networks remain highly susceptible to adversarial attacks, where small, subtle perturbations to input images may induce misclassification. We propose a novel optimization-based purification framework that directly removes these perturbations by maximizing a Bayesian-inspired objective combining a pretrained diffusion prior with a likelihood term tailored to the adversarial perturbation space. Our method iteratively refines a given input through gradient-based updates of a combined score-based loss to guide the purification process. Unlike existing optimization-based defenses that treat adversarial noise as generic corruption, our approach explicitly integrates the adversarial landscape into the objective. Experiments performed on CIFAR-10 and CIFAR-100 demonstrate strong robust accuracy against a range of common adversarial attacks. Our work offers a principled test-time defense grounded in probabilistic inference using score-based generative models. Our code can be found at \url{ https://github.com/rooshenasgroup/aaopt }.

Adversary Aware Optimization for Robust Defense
Daniel Wesego
Pedram Rooshenas
Department of Computer Science University of Illinois Chicago
Department Computer Science University of Illinois Chicago
Chicago, IL 60607 tweseg2Quic.edi
Chicago, II. 60607 pedram@uic.. edu
Abstract
Deep neural networks remain highly susceptible to adversarial attacks, where small, subtle perturbations input images may induce misclassification. We propose a novel optimization-basel purification framework that directly removes these perturbations by maximizing a Bayesian-inspired objective combining pretrained diffusion prior with likelihood term tailored to the adversarial perturbation space. Our method iteratively refines given input through gradient- updates of a combined woore-based loss to guide the purification process. Unlike existing optimization- based defenses that treat adversarial noise as generic corruption, our approach explicitly integrates the adversarial landscape into the objective. Experi- ments performed on CIFAR-10 10 and CIFAR- --0 demonstrate strong robust accuracy against range of common ad adversarial attacks. Our work offers principled test- time defense grounded probabilistic inference using score-based generative mod- els. Our code can be found at wtthttt::/// http//gttttt....t/// thub. hwtp://ooooeeeagrrooooooaooeo
Introduction
Neural networks have surpassed human accuracy on many vision benchmarks, yet they remain alarm- ingly vulnerable to adversarial examples worst-case perturbations that change model predictions with high confidence (Szegedy et al. 2014 Goodfellow et al. 2015 Carlini and Wagner, [017b]. Research increasingly suggests that this fragility arises from multiple interacting geometric factors as deep models partition the input space into mosaic of piecewise linear (or gently nonlinear) regions with large local Lipschitz constants, which means that even tiny steps can radically reorder the logits
(Hein and Andriushchenkol 2017 Fawzi et al. 2018 Yang et al. 2020].
The generation of an adversarial example can be expressed perturbed version of an input that changes the classification output. Given clean image x with true label y. and model an adversary seeks a perturbation y bounded by norm constraint || S that maximizes the classification loss:
problem of finding minimally
max Se
C(f(x + +).
(1)
Powerful gradient-basse attack methods are commonly used to find such perturbations. For in-
stance, projected gradient descent (PGD) [Madry et al. [018a] iteratively crafts adversarial example by taking steps in the direction of the sign of the gradient: x(++1) II rll, (x(+)
sign V (t) C( (x)))) v))) Another notable approach, the Carlini- (CW) attack (Carlini and Wagner 2017a], solves margin-based based optimization problem to find the minimum perturbation y required induce misclassification, i.e., f(x + 7) # y. These methods underscore how model
gradients can be exploited to identify subtle yet effective input alterations.
By ascending the input gradient of the loss, which often has significant component orthogonal the data manifold, gradient based attacks pierce through the fragile margins between classes, While
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

gradients are not strictly orthogonal. even small &o perturbations can cross decision boundaries in high- nigh-dimensional pixel spaces by targeting the most sensitive non-robust features. The structure of ReLU networks further exacerbates this vulnerability: within each activation pattern the model behaves affinely, allowing first-order optimization to be highly effective, while rapid switching between activation regions ensures that vulnerable directions are densely interleaved with benign ones [Qin et al. 2019].
Adversarial training (AT) defends a classifier by augmenting each mini-batch with worst-case perturbations and solving
max objective [Madry et al., 2018b]. Empirically, this paradigm remains the de facto gold standard for ls and lg threat models: models trained with AT are the hardest to break on standard robustness benchmarks. Subsequent variants such as TRADES [Zhang] et al.] 2019]. MART [Wang et al., 2020]. and AWP [Wu et al., [020] explicitly balance clean accuracy and robust loss or perturb the weights to enlarge margins, partially alleviating the well-known rccuracy-robustness trade-off.
Despite these gains, AT still faces significant practical hurdles. Generating multi-step adversar- ial examples for every mini-batch inflates training cost by 2-30x, even with accelcrations like Free (Shafahi et al., 2019] and Fast AT (Wong et al. 2020 , When access is limited to a fixed pretrained network, common in industrial pipelines or third-party deployments, full adversarial retraining may be infeasible. While carameter-efficient robust adapters and adversarial fine-tuning offer partial solutions, they still incur non non-trivial compute. Furthermore, robustness learned under one norm and radius does not necessarily transfer to other threat models or distribution shifts, though recent work on multi-norm training and data augmentation has made progress in this direction (Gowal et al. 202T].
A complementary line of work completely avoids retraining by purifying the input at test time, In this paradigm, the adversarial input & = x + 7 is treated corrupted observation, and the goal is to recover a reconstruction X that both lies near & and has high likelihood under a learned data prior. Early approaches such PixelDefend (Song et al. 2018] and Defense-GAN (Samangouei al. 2018) frame this as an optimization problem or projection onto the manifold of generative model trained on clean data. A more general perspective offered by the framework of Regularization by Denoising (RED) [Romano et al., [017]. which shows that any powerful denoiser implicitly defines a regularizer corresponding to an implicit data prior. Under this view, purification reduces to solving the following optimization problem:
min p(t x) + Ax- [x D(x)].
X
where D(x) is the denoising function.
Modern work replaces heuristic denoisers with score-based generative models. A diffusion model trained to predict the score Vx log px() under Gaussian noise at multiple scales; at test time, a stochastic sampler is run that iteratively nudges & toward high-density regions (Zhang et al. 2023 Nic et al., 2022b). Because the score network is trained solely on Gaussian Gaussian-corrupted data and never sees adversarial perturbations, it implicitly treats the perturbation generic Gaussian noise- an assumption that can fail when adversarial attacks exploit directions lying in low- wwwpprbbability yet classifier-sensitvv subspaces.
Adversarial perturbations, in general, differ from random corruptions in that they are intentional, geometry-awane and often exploit highly structured, non-robust directions in the input space. Cru- cially, these perturbations are not merely local, and they exhibit global regularities. Moosavi-Dezfool et al. [2017) shows that a single, fixed perturbation vector can fool most inputs, implying that decision boundaries are consistently oriented across different regions of the input space. Taken together, these local (gradient-based) and global (shared-structue properties suggest that adversarial perturbations are not arbitrary noise, but follow statistically learnable patterns.
This insight motivates the development of learned perturbation models, which aim to approximate the distribution of adversarial directions, improving purification strategies that go beyond the generic denoising distribution of adversarial directions. Our method draws inspiration from test-time opti- mization strategies [Zhang et al. 2023 Mardani et al. 2024 Cohen et al. [202T] that recover inputs by minimizing a tailored loss function. Rather than assuming simple Gaussian model to approximate p(& x), we explicitly capture the true adversarial perturbation distribution by training a score-based diffusion model. This model learns the gradient of the likelioel under realistic attack patterns,

---

which we then combine with the pretrained diffusion prior's score to direct the optimization. Because training a conditional score model for the likelihood term, similar to the prior, is going to be expen- sive, we propose lightweight score network to learn only the attack perturbations. This principled approach retains the strong generative power of the prior diffusion model and explicitly accounts for the adversarial structure using the perturbation model. Because it is based on optimization, it avoids the full reverse sampling process, offering a more efficient and robust defense mechanism.
Our contributions can be summarized as follows:
We introduce novel adversarial defense framework that employs score-based adversarial optimization with diffusion models, designed to maximize the posterior probability p(x |x) of clean images given inputs that may be clean adversarially perturbed.
Our proposed score-based diffusion optimization formulation integrates knowledge of the adversarial landscape into the purification process, making the defense more robust to attacks. The likelihood term p( x) is learned using an efficient, trainable perturbation model implemented as score- basee diffusion network without retraining the classifier.
We validate our approach CIFAR-10, CIFAR-100, and other datasets against strong adversarial attacks, achieving superior robust accuracy compared to state-of ofte-of-theear diffusion- based defenses. Our approach removes adversarial distortions while faithfully retaining the underlying image content, demonstrating its potential for practical robust systems.
2 Method
In this section, we describe our score-based optimization framework for adversarial purification. We begin with preliminary sections on diffusion models for adversarial purification. We then continue to present our approach, starting with the motivation and then formulating purification maximum a posteriori (MAP) estimation problem, deriving a tractable objective using pretrained diffusion and perturbation score networks, and then presenting an efficient iterative algorithm for test-time refinement.
2.1 Diffusion Models for Purification
Diffusion models (Sohl-Dickstein et al. 2015 Ho et al. 2020) (also known as score-based generative models|Song and Ermon 2019], Song al. 2021
class of deep generative models that learn to synthesize data by reversing a gradual noising process. During the forward diffusion process, data sample Xg ~ 4x) is progressively corrupted by adding Gaussian noise according
schedule {3:)-1- After I' steps, Xy is nearly an isotropic Gaussian regardless of Xg- A diffusion model learns the reverse denoising dynamics: starting from pure noise X ~ A/0 I) to produce samples that follow the data distribution. Training of diffusion models is typically done by denoising score matching Song and Ermon 2019) or an equivalent variational bound(Ho
al. (2020). In practice, neural network s t) is trained to predict either the added noise e, or the score (gradient of the log density) at each timestep using squared error loss. Sampling then proceeds by reversing this
process, either via discrete timesteps (Ho et al. 2020 or continuous SDE/ODE formulations [Song al. 202T].
Building on this generative capability, diffusion models have become a cornerstone of adversarial purification. In this setting, a potentially perturbed input x x + 7 is first injected with small amount of Gaussian noise by forward diffusing it to get x₃ and then denoised through reverse diffusion steps to produce purified output x ((x). The classifier f is applied to hopes of recovering the original prediction, f(x) = fx). While diffusion purification methods are effective mitigating adversarial perturbations, they are not without drawbacks. The stochastic and non-differentiable nature of the reverse process can obscure gradients, yet remains vulnerable to proper adaptive attacks such as PGD-FOT [Lee and Kim 2023
2.2 Motivation
Diffusion purification methods mainly rely on an unspervised approach where a pretrained diffusion prior can be applied to broad range of classifiers without retraining. While this classifier-agnostic

---

(a) Clean
Adversarial
(c) Perturbation
Figure 1: Samples from the clean, adversarial, and the difference perturbation between them. Image normalized and only showing values above threshold value of 0.03
approach broadens applicabilityy it leaves unexploited the rich structure of adversarial attacks them selves. In this work, we ask: can we augment diffusion purification with an attack-aware component that directly models the adversarial landscape, improving robustness, yet remains applicable across different classifiers? Our proposed Adversary Aware Optimization (AAOpt) answers this question by integrating perturbation model into the diffusion framework, delivering substantial gains in defense performance without sacrificing the plug-and-pla convenience of unsupervised purification.
2.3 Proposed Approach
Our goal is to recover the clean data given adversarial perturbation, which is equivalent to finding the maximum a posteriori (MAP) estimation for the following problem:
arg max log (xx(() x logp(*|x)) + log)
The prior p(x) supplied by a fixed, pretrained diffusion model whose score function directs samples toward the natural-image manifold. Rather than ignoring the likelihood term p(x x) or assuming a Gaussian distribution, we explicitly train it using a perturbation model that estimates the residual x & between clean and adversarial inputs. We model both the clean data prior p(x) and the perturbation likelihood p(& x) with score- diffusion models. Although we never evaluate these densities in closed form, can query their noise-prediction networks, (xe, [), which implicitly parameterize the corresponding score functions *o(x,,1 R Vx, (x..) 2 (-1/os obtained at diffusion time For single noise level 0, and corrupt X2 + O.E, (O.. we can show that (see Appendix for the derivation):
f)
log)(x) & -x [eo(xe,)) t) c].
(4)
Therefore, maximizing the right- hand side increases the likelihood of the data, even though p(x) itself is never evaluated.
The surrogate in eq, iis reminiscent of p(x) x x" D x)
x which has been used in RED Roo-
mano et al. (017). Mardani et al. [2024] and /Zang et al. [023 Mardani et al. [2024) derive similar
equation for their prior from the diffusion loss X Es Xt: t)
by using a linear loss based on
the gradient under some conditions.
We use a similar approach to obtain the surrogate for log x). Let y = x x the adversarial perturbation. We simply define p(x x) = p(y).
We train the diffusion model d for the adversarial perturbations following Song al. 2021 to eq. for e' (N, we have:
Similar
logp(y) & -(x - x) (dd(((()):
(5)
Finally, we can approximate the MAP solution in eq. as:
arg max max p(x x) Ra arg max x [cx, t) c) -xdc7t))))
(6)
x
where À balances the log data prior vs log probability of the perturbation.
Algorithm ssoos how we perform test-time adversarial purification by iteratively refining the given image & toward its MAP estimate x". To avoid propagating the gradients through the parameters of

---

Algorithm Adversarial Purification Optimization Algorithm
Require: x; pretrained diffusion net EØ; perturbation diffusion net dæ; total steps M; time range fmax]; schedule {at weight À
for to M do
2: Sample t ~ Uniform(Lwnn fmax), e, ~ NN011)
>> sample time and noise
3: X4 t vaix(i + v1 0g6
4: fprior + (x)(c((x,,))) c) x(
> prior loss & compute perturbation
5:
+ x(1-1) -X
6:
7t. - -avy + v
7:
Apent [sg(ds -
e(]² x(i-1)
> perturbation loss >> total loss
8:
et Epcior + + Eppr
9:
x(9) + -OptimieerStepx""
MAP update
end for
return x (M)
(a) Sample
(b) Sample 2
(c) Sample
Figure 2:
Different random samples generated from the perturbation model. Image is normalized and only showing values above a threshold value of 0.03
the wooe-bbsed diffusion models during the optimization, evaluating the score-based diffusion models, which will make the optimization efficient.
stop-gradient (sg) operations when
To train the perturbation model dæ. at each iteration, we sample minibatch of clean images x and pair each with randomly chosen sample x from the combined set of clean and adversarial examples length K. We combine clean samples the combined set because can be either adversarial clean in real test settings. Adversarial examples are generated only once using a combination of attacks on the classifier (2). We form the residual 7² = X-lean - x and then diffuse to timestep f by adding noise according the diffusion schedule at and train the network as normal diffusion model. The details the training can found in the Appendix section 66.4 Figure T shows clean and adversarial samples from the CIFAR-10 dataset with the adversarial perturbation plotted next to them. Figure showw samples from the perturbation model that capture the patterns of the perturbation of the attack landscape. For the prior score network, any diffusion model trained to generate clean data can be used.
We refer to our full test-time defense as Adversary Aware Optimization (AAOpt). In AAOpt, the perturbation score diffusion network de explicitly captures the adversarial likelihood p( x) by learning to predict the residual at multiple noise levels, while the pretrained score diffusion provides a strong prior p(x) over natural images. At inference, AAOpt iteratively refines the input via small number of madientt-basdd updates: each update minimizes combined loss that blends the diffusion prior and perturbation terms. This compact, MAP-driven loop delivers strong robustness against fully adaptive PGD-EOT and BPDA-ECC attacks, without modifying the underlying classifier or resorting to expensive reverse diffusion sampling.
3 Related Work
variety of defense strategies have been proposed to tackle adversarial attacks. Early purification approaches used simple image transformations, such as bit depth reduction compression, to remove high-frequency noise, but these filter-based methods offered limited robustness and were easily bypassed by adaptive attacks [Xu et al. 2018). Generative-model defenses then leveraged GANs or
5

---

VAEs to project inputs onto a learned manifold of clean images; DefenseGAN [Samangouei et al. 2018 optimizes a latent code reconstruct purified image, yet it introduces significant inference latency and can suffer from mode collapse.
Energy-based models (EBMs) were also applied adversarial purification by Hill et al. (222] using long-run Langevin dynamics to project adversarial inputs onto the natural data manifold and averaging over multiple stochastic samples via Expectation-Over -eee---eeee---Sfonnnee
(EOT). However, contrastive divergence training and MCMC sampling make EBMs slow and unstable. Yoon et al. [02TT] replaced EBMs with score- baree networks trained by denoising score matching [Song and Ermon 2019), and proposed a deterministic sampling technique with only few Langevin iterations.
They also improved robustness by injecting noise into inputs before purification.
The advent of diffussin priors ushered in more powerful purification techniques. DiffPure (Nie et al. 2022a] demonstrated that adding small amount of noise
adversarial example followed by a fixed number of reverse diffusion steps can effectively remove perturbations, achieving robust accuracy. However, the reverse chain often comprising of more than 50 timesteps entails high computational cost and may also drift the sample's 's semantics away from the original content depending on the selected timestep t. ScoreOpt (Zhang et al. 2023] improved efficiency by performing test-time optimization using the diffusion model's 's score function at random noise levels to guide a small number of gradient updates rather than executing the entire sampling chain. Chen et al. [2023) integrates classification directly into the diffusion process to jointly denoise and classify. They first optimize the input to remove adversarial perturbations and then use diffusion classifier to perform the classification. Lin et al. [2024) proposed a guidance method on top of diffusion purification based on distance metric between the features of adversarial examples and forward diffused x; to improve robustness.
Concurrently, works on diffusion-based inverse problems have introduced various formulations for solving both linear and non-linear measurement tasks. Song et al. [202T] showed that diffusion models can be applied inverse problems via a controllable generation process, sampling from p(X, y) using Bayes' rule and factoring the score as Vx, log (x,) ) + V log P: x1). Kawar ct al. [2022 proposed Denoising Diffusion Restoration Models (DDRM), which provides an efficient unsupervised method for solving inverse tasks by utilizing pretrained denoising diffusion models
and guidance from the observed input y. The guidanee is performed by considering the SVD of the degradation matrix and noise levels, which are then incorporated into the sampling process. |Mardani al. [2024 introduced RED-Diff, another method of optimization to solve inverse problems on images using diffusion prior models. They propose variational lower bound that consists of measurement matching loss and a score-based regularization term with pretrained diffusion models.
Experiments
We evaluate our score-based adversarial optimization method mainly on CIFAR-10 10 and CIFAR-100 datasets [Krizhevsky et al. 2009), benchmarking against suite strong and adaptive boite-box attacks and established defense baselines. All experimental results are reported on the test splits, and
we report both the clean (standard) accuracy and the robust accuracy when the model is under attack.
We compare our approach to different A Adversarial training (AT) methods [Pan et al. 2022 Gowal
cct al.
2021]
and differrntt diffusion purification methods [Nic et al., 2022a Zhang et al. 2023 Chen
ct al.
2023 We reuse some results from already implemented works [Lee and Kim 2023 Zhang
ct al. 2023 while we reimplement Score-Opt-O and Score-Opt-N Opt-N methods with the same pretrained
diffusion model and classifier checkpoints as our proposed method.
We employ a pretrained diffusion model using the EDM preconditioning scheme of Karras et al.) 2022). trained to minimize Ex-paata 2n= D(x n) - x|| where D(-, a)) predicts denoised estimate at each noise level. We extract the predicted noise from the model, where co (x, D(xx))/or. For all methods, we use a WideRes.Net 10 or WideResNet-70.. 16 classifier Zagoruyko and Komodakis] (016) trained on clean data, except where baselines specify adversarial training. To train our perturbation score model, we first generate adversarial examples for the classifiers using gradient attack with the cross-entropy loss from the training set once. Unless otherwise mentioned, we use Lx and l2 norms with combination of = (2, 4, 8) for L and
< <= (0.5, 1) for la norm attacks. Those will be combined with the clean data to train the perturbation model using the DDPM diffusion setup with 1000 discrete steps
(Ho et al. 2020].

---

We evaluate robustness under multiple adaptive attacks. We first show our results on the PGD-EOT (Projected Gradient Descent with Expectation Over Transformation) with Lo budget and l2 budget = 0.5 with 20 steps and 20 EOT samples per step [AAthalye et al., 2018 We then use the BPDA-FOT (Backward Pass Differentiable Approximation) attack setup with 50 BPDA iteration steps and 15 EOT samples per step. These attacks are crafted against the full purification (optimization) with the classification pipeline to measure end-to-end robustness of the models. We follow similar experimental setup with [ee and Kim [2023] Zhang et al. (2023). Remaining hyperparameter details and additional experiments can be found in the Appendix.
8/255
Table Standard and robust accuracy against PGD+EOT on CIFAR-10. Left: lac (c = 8/255); Right: 22 (c 0.5). Compared with adversarial training (AT) and adversarial purification (AP) methods.
Type
Method
Standard
Robust
Type Method
Standard
Robust
Net
WideResNett28-100
Default
94.78
0.0
Default
94.78
0.0
Pang
2022
88.62
64.95
Sehwag et al.
2021
90.93
83.75
AT
Gowal et a
2020
88.54
65.
AT
Rchutti ct al
2021
91.79
85.05
Gowal et al
2021
87.51
66.01
Augusun et
2020
93.96
86.14
Yoon ct al
2021
85.66
37.27
Yoon ct al.
2021
85.66
74.26
Nie ct al.
2022a
90.07
51.25
Nic ct al.
2022g
91.41
82.11
AP
AP
Score-Oper
2023
91.21
64.96
Score 0
91.21
79.09
Score-Opt-NN 2023
94.43
65.62
Score-Upt-N]20233
94.43
84.86
Lin cl al.
2024
90.42
64.06
Lin el al.
2024
90.42
85.55
AAOpt (ours)
91.91
86.75
AAOpt (ours)
91.91
87.61
WietReeNet-77-166
Default
95.19
0.0
Default
95.19
0.0
Gowal al.
2021
88.75
69.03
Rebufli
2021
92.41
86.24
AT
AT
Wang
2023
92.97
72.46
Wang
2023
96.09
86.72
Yoon al.
2021
86.76
41.02
Yoon al.
2021
86.76
75.90
Nie et al.
2022a
90.43
$7.03
Nie el al.
2022a
92.15
84.80
AP
Chen el al
2023
87.89
71.68
AP
Chen et al
2023
87.89
75.00
Score- Opt-e- 20223
92.61
68.86
Score 0[2033
92.61
79.55
Optro-00t-N
95.11
70.2
Score-Opt-N120231
95.11
85,21
A.AOpt (ours)
91.59
86.24
AAOOt (ours)
91.59
87.49
Table reports the results of clean and robust accuracy on the CIFAR10 dataset when performing the PGD-EOT attack. The PGD-EOT attack is performed under the budgets of loo budget = 8/255
and l2 budget 0.5. We use 20 steps of the optimization algorithm shown in Algorithm [ When generating adversarial attacks, we approximate the gradients of our model by performing single step of the exact optimization method, which will ensure the attack has full knowledge and access to all components of our defense. We compare against leading adversarial training (AT) and diffusion- purification (AP) baselines using two backbone classifiers (WRN-28- 10 and WRN-70-16)) 16). Across both architectures and attack norms, our adversary-aware optimization consistently delivers the highest robust accuracy, often outperforming prior defenses by many points, while maintaining clean accuracy on par with the best competing methods. These results demonstrate that our proposed optimization method offers state-of-the-art resilience to adaptive adversaries without sacrificing clean performance.
compontion
Table 2 higghlights the effectiveness of different gradient-approximatio strategies in the optimization loop, all evaluated on CIFAR-10 (foe, 8/255) with WRN-28- 10 classifier. Using single step of our exact nawward-ppimizzann strength of our adversary-and aware refinement. Even when omitting the stop-gradiemt (i.e., allowing gradients to backpropagate through the score networks) when generating adversarial examples, we still achieve 84.08% robust accuracy which substantially outperforms the other baselines shown in Table [ addition to this, we compare our approach with ADBM [Li al. 2025). which finetunes
update yields the highest robust accuracy of 86.75%, showing the
diffusion model with adversarial data, under different settings as detailed in Section 6.3 of the
7

---

Appendix. Finally, we use a single step of denoising through the prior diffusion to approximate the gradients similar Zhang al. [2023]. which also attains robust accuracy of 75.11% that exceeds the best Infffusion-purfficaltl and adversarial-trainig methods. These results underscore that our core optimization framework is both powerful and resilient, delivering leading adversarial robustness results.
Table 2: Robust accuracy on CIFAR-10 (Lxc, 8/255) for different gradient approximation methods using the WRN-28-10 classifier.
Gradient Approximation Method
Robust Accuracy (CIFAR- 8/255)
One step of exact forward optimization
86.75
One step of forward optimization with stop-grad One step of denoising
84.08
75.1 11.2
Table
3a compares clean and robust accuracy under strong BPDA-EOT attack on fo with
8/ 255 across adversarial training (AT), diffusion-bal based purification (AP), and our proposed udversary-award score optimization method. Adversarial Training methods plateau around 64% robust accuracy (Gowal et al. [2020) achieves 64.1%) while diffusion purifiers substantially improve robustness with Score-Opt-N pushing the result to 90. %. Our approach further advances the result by achieving robust accuracy of 91.4%, while maintaining clean accuracy of 92.1 1.. These results demonstrate that our proposed method withstands different kinds of adaptive attacks but also outperforms prior works on adversarial training and purification defenses. Table 3b shows the respective results for CIFAR-100 dataset. Our method is also robust on this data, achieving robust accuracy of 66.56% while maintaining standard accuracy of 69.98% which outperforms other diffusion purification methods under the same attack conditions on robustness with comparable standard accuracy.
Table 3: BPDA+EOT attack on CIFAR-10 and CIFAR-100 (Lac, 8/255) threat models (all architectures WRN-28-10).
(a) CIFAR-10 (WRN-28-10)
CIFAR-
(WRN-28-10
Type
Method
Std.
Rob.
Type
Method
Std.
Rob.
Default
94.78
0.0
Default
81.55
0.0
Carmon et al.
2019
89.67
63.10
Yoon et al.
2021
60.66
39.72
AT
Gowal et al.
2020
89.48
64.08
Hill et al
2021
51.66
26.10
Score Opt 2023
70.53
66.11
Wang et al.
2022
93.50
79,83
Score-Upt-N10023
74.18
60.21
Yoon et al
2021
86.14
70,01
AP
Nic ct al.
20224
89.02
81.40
AAOpt (ours)
69.98
66.56
Score-Opt-0J0025
90.23
81.36
Score-Upt-Nj2022
93.94
90.07
AAOpt (ours)
92.10
91.36
Table compares baseline purification pipeline, omitting our perturbation score correction and only using the prior pretrained diffusion score-regularization method against our full AAOpt method on CIFAR-10 under both PGD-EOT and BPDA-EOT attacks with a WRN-70-16 and WRN-28-11 classifiers, respectively. Incorporating our adversary-awarr perturbation score model boosts clean accuracy by at least 1% and yields substantial gains in robust accuracy under both PGD-EOT and BPDA-EOT attacks. Figure shows how the standard and robust accuracies change as À is increased from 0 to 1 on WRN-28-10 BPDA-EOT attack. The accuracies start to increase initially and plateau around À 0.6 and then decrease.
Generalization Table levaluates the transferability of our perturbation model across different classifier architectures. We compare performance on WRN-70-16 when using perturbation model trained on its own adversarial examples versus one trained on WRN-28-18 10. The minimal drop in accuracy demonstrates that our learned attack distribution generalizes effectively to classifier it was not originally trained on. Table 6 also shows the robustness of our model against different unseen corruptions [Hendrycks and Dietterich 2019 We evaluate against CIFAR10-C data

---

Table 4: Baseline (without perturbation-scon model correc- tion) similar to scoue-regularizztiion of Mardani et al. (2024)
Full Method (AAOpt) comparison on CIFAR-10 10 (2...
66-
é
8/255) under PGD-EOT using WRN-70-16 and BPDA-
EOT using WRN-28-10.
64-
65.
Method
PGD-EOT
BPDA-EOT
Standard
Robust
Standard
Robust
Baseline
82.88 89.7
91.05
81.95
Figure 3: Accuracy as À in- creased from O to 1 for AAOpt on BPDA-EOT on CIFAR-10 (foo. = 8 8/255)
AAÖpt (ours)
91.59+0.7
86.24
92.10+0.7
91.36=0.8
Table 5: Transferability of perturbation models: accuracy WRN-70-16 16 using perturbation models trained on WRN-70-16 and WRN-28-10 adversarial data under PGD-EOT attacks (loo, = 8/255 and fg, c 0.5).
Perturbation Model
PGD-EOT (Exc, c= 8/255)
PGD-EOT (l2, 0.5)
Standard
Robust
Standard
Robust
Perturbation Model of WRN-70-11 10-16
91.59+0.7
86.24 85.90
91.59+u.77
87.49=0.7
Perturbation Model of WRN-28 10
91.51+1.1
91.51+1.1
87.20
Table 6: Robustness against common corruptions on CIFARIO-C on WRN-70-16
clastic transform
jpeg compression
Method
gaussian
snow
brightness
Default
45.6
82.9
75,8
83.5
93.8
Score-Opt-O
79.8
78.5
86.1
82.67
89.6
AAOpt
85.3
84.81
89.5+0.1
85.340.4
90.5
combination of five different severity levels of transformations. These results confirm that AAOpt's perturbation-avee aware optimization yields strong, classifier- defenses across both adversarial and natural distribution shifts.
Beyond the transfer experiments in Table 5 which apply WRN-28-10 trained perturbation model to WRN-70-16 classifier, we further assess AAOpt with WRN-28- trained perturbation model on CLIP zero-shot model applied to CIFAR-10. We compare against the adversarially trained Robust- CLIP FARE2 and FARE models [Schlarmann et al. 2024], which were trained with foo budgets
of = 2/255 and e 4/255 respectively as shown in Table 7 Despite never seeing CLIP during the perturbation model training, AAOpt, using only the WRN-28- 10 perturbation model, achieves superior robust accuracy on this entirely different architecture, underscoring the wide applicability and weassiffer--gnostte nature of our adversary-aware optimization. In addition, it exhibits consistent performance across different attack budgets, unlike adversarial training models, which drastically lower their performance when attacked with a budget different from the one on which the model was trained.
Finally, we present experiments on norm- trantrrrssferaiiitt.. We train perturbation model using single norm and evaluate it on a different norm to assess generalization across attack types. As shown in Table the performance remains consistent, indicating that the proposed approach exhibits strong transferability between for and & perturbation spaces. Specifically, when trained on l2(c = 0.5) and evaluated with an &xe (c = 8/255) PGD-EOT attack, the model achieves robust accuracy of 86. 86.22% which is close to the results reported on Table [ where the perturbation model is trained on the combination of the two norm attacks. Similar performance also reported when trained on (x = 8/255) and evaluated under fe 0.5) perturbations. These results demonstrate that the perturbation model generalizes well across different norm constraints, confirming the robustness and adaptability of our method.

---

Table Transfer-PGD attack on CIFAR-10 10 (lxc, c 4/255 and 8/255) on the CLIP classifier
using perturbation model of WRN-28-1U classifier.
Type
Method
Standard
Robust (c = 4/255)
Robust (c = 8/255)
CLIP
95.2
0.0
0.0
FARE2
89.5
25.9
2.22
AT
FARE4
77.7
34.6
9.01
Score- Ocore-Opt-O(20233
89.21
88.74
88.70
AP
Score-Opt-N[2023
90.42
89.87
90.70
AAOpt (ours)
91.52+0.9
91.63=1.3
90.30=1.1
Table 8: Norm transferability: accuracy of WRN-28-10 using perturbation models trained on different norm under PGD-EOT attacks (E 8/255 and e,, 6 = 0.5).
Perturbation Model Trained On Evaluated On
Rob. Acc.
loc. 8/255
b, 0.5) loc. E 8/255
87.07
l2, c = 0.5)
86.22
We report additional results and implementation details in Appendix including theoretical deriva- tions, extended robustness evaluations (AutoAttack and ADBM comparisons), perturbation analyses, classifier transfer-attack results, and experiments on TinylmageNet. The appendix also provides full training and dataset specifications, including hyperparameters and compute resources, and concludes with discussion of the limitations of this work.
5 Conclusion and Discussion
We have introduced Adversary Aware Optimization (AAOpt), novel framework for test-time defense that explicitly incorporates both a diffusion prior and learned perturbation model. By training perturbation model to capture the adversarial landscape and combining it with a pretrained diffusion score model, AOOp performs small number gradient-based refinement steps to recover purified inputs. Extensive experiments on CIFAR-10 and CIFAR-100 under strong, adaptive PGD-EOT and BPDA-EOT attacks demonstrate that AAOpt consistently outperforms statc-of-the art adversarial training and purification basclines, achieving robust accuracy improvements while maintaining high clean accuracy. The limitations of our work are that AAOpt requires generating adversarial examples once for training the perturbation model and incurs additional test-time cost due to a few gradient-based refinement iterations. However, these overheads are modest compared to reverse-diffun diffusion sampling repeated classifier retraining, and the benefit of robust accuracy across multiple norms and attacks, coupled with minimal impact on clean performance, far outweighs the costs. Our framework is also highly modular: the perturbation model and diffusion prior can be improved independently. Consequently, AAOpt represents a practical path toward robust, test-time defenses in real-world settings.
References
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 274-283, PMLR, 10-15 Jul 2018. URL ltttps://proceeding: mlr. caa...pres//vve athalyee8a.laaata html
Maximilian Augustin, Alexander Meinke, and Matthias Hein. Adversarial robustness on in-and out-distribution improves explainability. arXiv preprint 200iv:2000300946/ 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 20/7 IEEE Symposium Security and Privacy, pages 39-57. IEEE, 2017a.
10

---

Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, pages 39-57. IEEE, 2017b.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. Advances neural information processing systems, 32. 2019.
Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang. Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via a single diffusion model. In Forty-first International Conference on Machine Learning, 2023.
Regev Cohen, Michael Elad, and Peyman Milanfar. Regularization by denoising via fixed fixed-point projection (red-pro). SIAM Journal on Imaging Sciences, 14(3):1374-1406 2021.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fre attacks. International conference on machine learning. pages 2206-2216. PMLR, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255. IEEE, 2009. doi: 10. 1119/C/PR.20000 200.556684..
Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. Adversarial vulnerability for any classifier. Advances in neural information processing systems, 31, 2018.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Sven Gowal, Chongli Qin. Jonathan Uesato, Timothy Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm-bounded adversarial examples. arXiv preprint arXiv:2010.0359.3 2020.
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in neural information processing systems, 34:4218 4233, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of classifier against adversarial manipulation. Advances in neural information processing systems, 30, 2017.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR). OpenReview.net, 2019. URL URL httpps:/ wwt.:/eeef.oommmiieeeeee.m
Mitch Hill, Jonathan Craig Mitchell, and Song-Chun Zhu. Stochastic security: Adversarial defense using long-run dynamics of energy-basee models. In International Conference on Learning Representations, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Ad- vances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 6840-6851, Cur- ran Associates, Inc., 2020. URL Abstract. html
htwpeeedddinns..uuriissse. nwwrins. cttppps..ccppppppee httpp//ss...pppppp/////////eee// papppp/p/////////////
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In S. Koyejo, S. Mohamed, A. Agar- wal, D. Belgrave, K. Cho, A. Oh, and T. White, editors, Advances in Neurul Infor- mation Processing Systems 35 (NeurIPS 2022), pages 12914-12928. Curran Associates, Inc., 2022. URL https: / www.wnnuuiicoom htwpedddngnnnerppp..ceppaaerrrre fiapper_flle//eee wwtesffflleesleeeeeeeeeeee 2ttapper..leeeppppe//2///////// 2atp//paar2/22/2ss/ 44932668e00000111177700006666660 oegglsaccccccaoooooooooooaaaaaar

---

Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In S. Koyejo. S. Mohamed, A. Agarwal. D. Belgrave, Cho, A. Oh, and T. White, editors, Advances in Neural Information Processing Systems 35 (NeurIPS 2022), pages 11803-11817. Cur- ran Associates, Inc., 2022. URL
neurips cwtpaper__facaeeeeeeee filesffile/eeereeeereeee httpe//aapeeeee paper/ html
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Minjong Lee and Dongwoo Kim. Robust evaluation of Iiffusion-basee adversarial purification. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 134-144. IEEE, 2023.
Xiao Li, Wenxuan Sun, Huanran Chen, Qiongxiu Li, Yingzhe He, Jie Shi, and Xiaolin Hu. ADBM: Adversarial diffusion bridge model for reliable adversarial purification. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL forum?id forumf/d-gornZeBguu
net/
Guang Lin, Zerui Tao, Jianhai Zhang, Toshihisa Tanaka, and Qibin Zhao. Robust diffusion models for adversarial purification. CoRR, abs/2403.16067, 2024. URL http:://////oo.rrrg////// 48550/ arrXiv 2403. 16067]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations (ICLR). OpenReview.net, 2018a. URL https://openrevieer net/
forum/id=r.z.IBEIN
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018b.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. variational perspective on solving inverse problems with diffusion models. In The Twelfth International Conference on Learning Representations (ICLR). OpenReview.net, 2024. URL URUhatpp://oggereeeeee...e wtwp:rrrwwwiieeeeeeooomommmm 1Y04E63SPB
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 1765-1773, 2017.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning. PMLR, 2022a.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. arXiv preprint 0274000205.074600, 2022b.
Tianyu Pang, Min Lin, Xiao Yang, Jun Zhu, and Shuicheng Yan. Robustness and accuracy could be reconcilable by (proper) definition. In International conference on machine learning, pages PMLR, 2022.
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. Advances in neural information processing systems, 32, 2019.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A Calian, Florian Stimberg, Olivia Wiles, and Tim- othy Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprint a1X1v:2103.01946. 2021.
Yaniv Romano, Michael Flad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 1804-1844444 2017.
12

---

Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-GAN Protecting classifiers against adversarial attacks using generative models. In International Conference on Learning Representations, 2018. URL htt.ttt::://pppeeeeeeeeeee/ net/ffrum 2d-.33/3b------3
Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust CLIP: unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 2/- 27, 2024. OpenReview.net, 2024. URL httpps://openreviee
Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? In International Conference on Learning Representations, 2021.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in neural information processing systems, 32, 2019.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamies. In International Conference on Machine Learning, PMLR, 2015.
pages
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In Interna- tional Conference on Learning Representations, 2018. URLLattps://pperriiiie wtw.eeerrreiieeneeffforeur net/forum?
]d=rJUYGxbCM
Yang Song, Jascha Dich--Dicstten, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Jinyi Wang, Zhaoyang Lyu, Dahua Lin, Bo Dai, and Hongfei Fu. Guided diffusion model for adversarial purification. arXiv preprint arXiv:2205. 14969, 2022.
Yisen Wang, Difan Zou, Jinfeng Yi, James R. Glass, and Quanshi Zhang. Improving adversarial robustness via metric learning. International Conference on Learning Representations, 2020.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International conference on machine learning. pages 36246-36263. PMLR, 2023.
Daniel Wesego and Pedram Rooshenas. Score-based multimodal autoencoder. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL httppp::/pppprrrreee.etee//o/aet
Id=JbuP6UV3FR
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020. URL https:://peerrvieey net/ Forum?id=B.Jx00EER
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. Advances in neural information processing systems, 33:2958-2969, 2020.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In NDSS Symposium. Internet Society, 2018.
YuanYuan oe Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaud- huri. A closer look at accuracy vs. robustness. Advances in nearal information processing systems, 33:8588-8607 2020.
13

---

Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based generative models. In International Conference on Machine Learning, pages 12062-12072. PMLR, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 1rXiv:660.007140 2016.
Boya Zhang, Weijian Luo, and Zhihua Zhang. Enhancing adversarial robustness via score- based optimization. In Alice H. Oh, Tristan Neumann, Amir Globerson, Sara Hooker, Ratthachat Sinthong, Shweta Israni, and Virginia Tseng, editors, Advances in Neural Information Processing Systems (NeurIPS 2023), pages 11547-11561. Curran Asso- ciates, Inc., 2023. URL https:// nips. wwcpppppr__eteeeeeeeeee files/illlee 6227773443399444488.8838
/ pptpp//ppp//aaaalll// hhml
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric P. Xing. Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7482. PMLR, 2019.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang. Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract are supported by experiments and they are reflected throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: Limitation has been discussed in the Conelusion and Discussion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
14

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our thcoretical results and assumptions are based on previously established formulations correctly cited whenever needed.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully disclose information needed to reproduce the experiments. Detailed hyperparameters will be in the Appendix section, along with the code attached.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (c.g., in the case
15

---

of a large language model), releasing of a model checkpoint, or other means that are
appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, we have attached our code along with how to reproduce the results. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce ec/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https Siaeeee//ooSSSeeeeeeeeee
//nips. //nips.cc/pul ctttps.cccpubblee
ssionnolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide experimental details in the Experiments section and in detail in the Appendix section.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We deviation of our results.
main experiments over 5 trials and report the and standard
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma eтror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We add this detail the Appendix section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We respect and follow the Neurips Code of Ethics.
17

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We do not believe our work has any societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
18

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited properly when used data and models from other works.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpppppppshtaeea thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use any crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not any crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The core research doesn't 't involve any LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy htttess//earriss.cc//ooffeeeeeee for what should or should not be described.
20

---

Appendix
This appendix provides supplementary material the main text. We first derive the surrogate equation in section 6.1 In Section 6.2.1 we report AutoAttack results; Section 6.2.2 compares our results with ADBM et al. 2025|: Section 6.2.3 analyzes the perturbation patterns of adversarial examples; Section 6.2.4 presents transfer-ates -ttackk experiments; and Section 6.2.5 shows results on TinylmageNet. Next, we will discuss datasets used in Section 66.3] We then detail our experimental setup, including hyperparameters, optimization, training, and compute resources in Section 6.4, and conclude with an extended discussion of limitations in Section
6.1 Derivation Surrogate Loss
Assuming x + OrE and e ~ (0 f), and the true marginal distribution at time é is (x.),,, the reverse conditional distribution modeled
p xx) (x: 1), of
with
(x t) X, occo(x(,t).
1. From Baye's identity for posterior log-density:
log p(x) = log Pe (xx) + px xx) - log g(x; x).
where
g(xe x) = o(
Note that
1
|xx x|| const } 2 _llll1 + const.
log q(x; xx
207
Therefore,
logp(x) = (x(:) + log (x x.) + 2 ||lel? + const.
2. From Taylor's expansion of (x) around Xx:
log Re (x.):((:) + (x- xx) Vx. logg)(x.).
Rearranged as:
(xg)r(x;) logp((x)) x1) Vx. (x-xx)) T. (x.))
Substitute log (x., in eq.
log ((x) log (x - Vx, logp:(x.) + log xx) + _1le112 const.
which can be written
1 7 |||| logp(x) G. log.:(xx) + log xx)) +
(8)
2
3. Expressing log px xx) using diffusion model cØ:
log (x xx) x 20? x px( 1) + const,
where
(x((),)) = =x -x
Note that
M4 -(x, -0,00) = (x xx) o:E9 -O + oFE9 0.(Eg E).
21

---

Therefore,
logpo(X xt) x llco -cl2 + const.
(9)
4. Substituting log (x xx) back in eq.
1 log p(x ofe Vx, logpi(x)) Expanding the quadratic term,
-22222le 211co ccl2
2 /llel + const.
Eg e|12 21eee11 > 2| ETE..
Thus,
logp(x) x "V log + E ETEg -
~ - const.
logp(x) x (o, Vx, (x) + +60)
2 |6al l
logp(x) x (x+-x)² (or Vx, Vx, log):(x;) + c0) - 2 =11co11
0t
x (or Vo, loggt(x)) + =e)
1
log ((x) x Tt
(o, logpe(x;) + ED)
||
at
Dropping the non x terms:
logp(x) x (x)T (or log))(x;) + co)
p(x) x -x" Vx, (x))) + c&)
Finally, replace score the conditional Gaussian score in expectation make it tractable Zhou et al. 2024
Vx, logp: so Vx. log g(xe x)
logp(x) x -x" (Eo(x,,)) () -
Which is the same as eq Similarly, the loss will be:
- log p(x x x (X,, - E)
which is the same loss used by Mardani et al. [2024] for their prior, but they formulated it in different way.
6.2
Additional Results
6.2.1
AutoAttack
In addition to PGD-EOT and BPDA-FOT attacks presented in the main experiment, we assess the robustness of our model using AutoAttack [Croce and Hein, 2020). an ensemble of complementary. parameter-free attacks that provides a strong, automated benchmark for robustness. We conduct these evaluations on a WRN-28-100 classifier, generating adversarial examples via an exact one-step gradient through our optimization loop. Table paalee]preeentt Auto/ttack robustness alongside PGD-EOT robust accuracy for direct comparison; baseline results are taken from Zhang al. [2023). Lee and Kim [2023), Lin et al. [024]. Although running the full multi-step optimization for Auto/tttak yields very similar performance, repeating those experiments across multiple seeds and attack configurations is prohibitively time-consuming, confirms that AtOt maintains high robust accuracy even under this rigorous, adaptive attack suite, demonstrating its superior performance against diverse basclines and attacks.
report the one-step exact-gradient results here. This evaluation
22

---

Table AutoAttack and PGD-EOT on CIFAR-10 (Lxo, 8/255) using the WRN-28-10 classifier.
Type
Method
Standard
PGD
AutoAttack
WideRessNe-28-10
Default
94.78
0.0
0.0
Pang et al.
2022
88.62
64.95
61.04
AT
Gowal et al.
2020
88.54
65.10
62.76
Gowal ct al.
2021
87.51
66.01
63.38
Yoon et al.
2021
85.66
37.27
59.53
Nie ct al.
2022a
90.07
51.25
63.6
AP
Lee and Kim
[2023
90.16
55.82
70.47
Lin et al.
2024
90.42
64.06
78.12
Ours (A.Optt)
91.91
86.75+0.7
88.28=005
6.2.2 Comparison with ADBM
In general, our optimization has stop gradient on the diffusion models during optimization to remove adversarial perturbations. PGD-based gradient-based attacks are also generated this way (remember we need second-order derivative for generating attacks), where gradients don't pass through the diffusion model, similar to our optimization mechanism. While this is ok, stronger attack can be generated by backpropagating through score-based diffusion models. the main text, we added Table 2 which shows the different performance of our model under different gradient approximations, even with no stop-gradient. In this section, we want to compare our work with ADBM [Li al.) 2025], which modifies the diffusion objective itself to guide adversarial examples toward the clean distribution. They also propose a full gradient attack where they show the performance of different diffusion purification methods perform very poorly. Unlike diffusion purification works discussed in [L al. 2025). AAOpt performs very well in this scenario also The results are summarized Table 10 and reveal that AAOpt significantly outperforms ADBM in robust accuracy. The experiments were conducted under a similar setup to ADBM for a fair comparison with PGD-EOT, with 20 attack iterations, setup similar to the one found in their appendix, and BPDA-EOT with 200 attack iterations.
The PGD results for AOpt are with gradients passing through both diffusion models to generate adversarial data, approximated with one optimization step. Even with full gradient passing through all the steps, similar to the ADBM full grad setup for the PGD, we get a robust accuracy x 82%, further confirming its efffcctiveness time and the resources takes to get the full gradients with no stop-grad through all the optimization steps, approximate it with one step, which provides close results but is more efficient to conduct.
under stronger attack variants. But due to the computation
Table 10: Comparison with ADBM CIFAR-10 (l 8/255) using the WRN-28-10 classifier.
Method
BPDA-EOT
PGD-EOT
Standard
Robust
Standard
Robust
ADBM
91.93
70.51
92.50
42.20
Ours (AAOpt)
92.10.0.7
90.8911.2
91.91
84.0
6.2.3 Attack Patterns
In this section, we analyze the structure of adversarial perturbations and their relationship to the original images. Figure displays clean image, its adversarially perturbed counterpart, and the difference between them. Although these difference patterns may appear noisy at first glance, they exhibit systematic correlations with the underlying content of the clean image. Capturing these consistent perturbation patterns is the key motivation for our perturbation-score network. As illustrated by the samples in Figure our model learns to represent and predict these adversarial residuals.
23

---

(a) Clean
(b) Adversarial
Perturbation
(d) Clean
(e) Adversarial
(1) Perturbation
Figure 4: Clean, Adversarial, and Perturbation of a sample image from CIFAR-10, perturbation is normalized and only showing values above threshold value of 0.03
6.2.4 Transfer PGD Attack
We evaluate transfer-based robustness by generating -oonnddd PGD adversarial examples on the classifier alone and then applying various purification methods frthhods-including our AAOpt framework- these fixed attacks. Table I reports clean and robust accuracy for each purifier. Our AAOpt method delivers robustness on par with the strongest diffusion purifiers, closely following Score-Opt-N, which achieves the highest robust accuracy in this setting. These results confirm that AAOpt's adversary-awar modeling substantially improves resilience to transfer attacks without sacrificing clean accuracy.
Table 11: TransferPPGD attack on CIFAR-10 (lo., 8/255) using the WRN-28-10 classifier.
Method
Standard
Robust
Default
94.78
0.0
Yoon et al. 2021
93.09
85.45
Score-Opt-0]220
89.79
87.45
Score-Opt-N[zuzz
92.65
92.76
Ours (A.Optt)
92.1440.6
91.01
6.2.5 TinyImageNet Results
To demonstrate that AAOpt scales TinylmageNet- - 200-class subset of ImageNet with images resized to 64x64 [Deng et al. 2009). We fine-tune a ResNett50 classifier [He et al. 2016 on this dataset and attack it using BPDA EOT with 50 iterations and 15 EOT samples. 12ble compares our method against Score-Opt- bascline. Even in this challenging setting, AAOpt achieves the highest robust accuracy, confirming its effectiveness on larger, more diverse benchmarks. The result can be improved further with more powerful classifiers and better prior models but we leave that for future work.
complex, higher-dimensional data, we evaluate
Table 12: BPDA-EOT attack on TinylmageNet (fac, 8/255 using ResNet50 classifier.
Method
Standard
Robust
ResNet50
76.90
0.0
Score-Opt-O0 Opore-0pt-0[2023]
40.89
21.2
Ours (AAOpt)
56.10.1.9
53.75+1.5

---

6.3
Datasets
Our evaluations leverage several widely used benchmark datasets. We use CIFAR-10 and CIFAR- 100 (Krizhevsky et al. 2009), each comprising 50 000 training and 10 000 test images at 32x32 resolution; CIFAR- I0 spans 10 classes, while CIFAR- 100 covers 100 classes. These datasets are publicly available from https:: https://www.com toronto. edu/"kriz ciirriiccffa... html though no explicit license specified by the authors.
To assess robustness under common corruptions, we employ CIFAR-10-C [Hendrycks and Dietterich 2019), which augments the original CIFAR-I0 10 test set with different types of corruptions five severity levels. CIFAR-IO-C is released under a CC BY 4.0 International license (DOI: https //doi.org/10 5281/zemodo. 25359677)
Finally, we include TinylmageNer200,, a 200-class subset of ImageNet containing 100 000 training and 10 000 validation images resized to 64x64. While the Stanford CS231N release (available at httt::////// cs231n stanf ord. edu/ imagenet 200. zip) does not list its own license, inherits the ImageNet terms of use, which permit non-commercial research and educational use only [Dngg et al. 2009].
6.4 Experiment details
In this subsection, cover the experimental details of training and inference for our proposed model. We first list the hyperparameters in the model, then we go over optimization, training, and compute details. We have also attached the code used generate the outputs, which can be found in the attached supplementary material for more details. We run the main experiments in the paper 5 times and report the and standard deviation. We use 512 samples for each trial because of the heavy computation requirements to generate adversarial examples.
6.4.1 Hyperparameters
This section details the hyperparameters for AAOpt framework. For each attack setting, specify the Adam learning rate, the number of optimization iterations, the weight À balancing the diffusion prior and perturbation model losses, and the number of forward diffusion timesteps. Table summarizes all selected values.
Table 13: Details of hyperparameters used for our proposed model AAOpt
Attack Type
Perturbation Budget
LR
Iterations
A
Timesteps
CIFAR-10
PGD+EOT
(E 8/255)
0.1
20
0.25
Uniform (0.15,0.35)
PGD+EOT
12(E 0.5)
0.1
20
0.25
Uniform (0.15,0.35)
AutoAttack
Lx (E 8/255)
0.1
20
0.25
Uniform (0.15,0.35)
BPDA+EOT
1o (E 8/255)
0.1
5
0.5
[0.5, 0]
Transfer-PGD
(E 8/255)
0.02
20
0.6
[0.2, .... 0]
Transfer-PGI Biaffer-PGDCLIII
To (E = 8/255)
0.01
20
0.2
[0.1, ..., 0]
CIFAR 10 10-Corrputions
Corruptions
0.1
5
0.1
[0.25, ..., 0]
CIFAR-100
BPDA+EOT
(c 8/255)
0.1
3
0.4
[0.15,
6.4.2 Optimization
The core purification loop is detailed in Algorithm [T where at each iteration we update the current estimate x by taking an Adam step on combined loss. make the entire procedure end-to-end differentiable, so that adaptive attacker can compute gradients through our optimization update, we implement a custom differentiable Adam optimizer PyTorch rather than relying on the built-in version, which uses in-place operations and is not differentiable. In addition, when generating
25

---

adversarial examples via a single exact optimization step. we set the create graph flag to True, which yields the correct gradients of the adversary's loss with respect to through the optimization. Without this, attacks like PGD-FOT would treat the optimization updates as constants, potentially generating a weaker form of attack.
6.4.3 Training
We used a pretrained EDM diffusion for the prior model [arras et al. and CIFAR-100 datasets. For training the perturbation model, we use UNET architecture with
2022 for both the CIFAR-10
three input and output channels. We train the model according to HHo et al. diffusion loss. The residual difference between the clean and adversarial images diffused a random timestep t by adding noise e¹ according to the diffusion schedule {ar). The network dø takes the noisy perturbation of the residual Y and predicts the injected noise ě". We then minimize normal diffusion DDPM model over the
2020 with unweighted
the denoising objective ||ž - e'2 and train the network perturbation to capture the score-based diffusion of the residual distribution at varying noise levels. Note that the residual 7 can also be defined in the reverse direction y' x Xelean with the same setup as the negative sign in the difference will cancel out with the perturbation loss term. The details of the training are shown in Algorithm
To train our perturbation score model, we generate adversarial examples once for the classifiers. We gradient attack with the cross-entropy loss from the training set. We use Loo and l2 attack norms with a combination e = (2, 4, 8) for Loe and = (0.5, 1) for le attack budgets. We combine these with the clean data, and one adversarial example is randomly sampled uniformly to form the pair for training the perturbation model.
For the classifiers, we used WideResNet architecture classifiers with WRN-28-18 10 and WRN-70-16 16 types [Zagoryyko and Komodakis] 2016). For the CIFAR-10 10 dataset, the WRN-28-1U 10 architecture yields an accuracy of 94.78, while the WRN-70-16 architecture yields an accuracy of 95.19 The robust accuracy of these default classifiers is approximately 0% for both Lo and (2 attack budgets of é 8/255 and 0.5 respectively.
Algorithm 2 Perturbation Model Training
Require:
Clean dataset {Xelean J. combined dataset of adversarial and clean dataset fx* }k=1 (where each x* is either clean or adversarial), score network d. diffusion timesteps I schedule {ar)
1: for each training iteration do
2: Sample (Xetean 2= from clean data and x from {xady]k
JK
3: Find perturbation 7 = Xleean X
4: Sample timestep t' Uniform(1......
5:
Sample noise e¹ ~ *O I)
6:
Compute noisy diff: 7i = voz 42 + VI a.e.
7: Predict: é' = ( (²)
8: Compute loss: L = ||ē* - c'l
9: Update Parameters: dz
10:
end for
6.4.4 Compute
We use NVIDIA A100 GPU to train the models and perform the inference. Training the perturbation UNET model takes approximately 6GB of GPU memory for batch size of 256. We trained the perturbation model for 3000 epochs, and one epoch approximately takes 18 seconds. We use the Adam optimizer with constant learning rate of 2c-5. More details and the exact training code can be found in the attached supplementary material.
Table 14 shows how much time the optimization algorithm takes compared to Score-Opt- baselines tested on our environment on an 100 GPU. The iteration steps for the optimizations on this table are 5 steps on the CIFAR- 10-C dataset, which is similar in characteristics to the CIFAR-10 dataset. As seen the table, our model is slightly faster compared to the baselines. Note that the time shown is for the optimization to purify x, and not for generating adversarial examples.
26

---

Table 14: Optimization speed of our model compared with baselines on CIFAR-IO-C
Model
Batch Size
Time Consumed
Score-Opt-O
128
1.16 seconds
Score-Opt-
128
1.26 seconds
Ours (AAOpt)
128
1.05 seconds
6.5 Limitation and Discussion
While AAOpt achieves state-of-the-art robustness, it relies on several assumptions and incurs non- trivial computational costs that may limit its performance. First, our perturbation model requires generation of adversarial examples once. Although we have shown this be enough based on our experiments as tested in different scenarios, the effectiveness of our perturbation model may degrade the attacker uses a substantially different threat model or if the perturbation distribution shifts too much in practice. Second, test-time optimization requires multiple gradient-basee refinement steps (3-20 iterations), which increases inference latency compared to a standard forward pass models. Third, AAOpt depends on well-traines diffusion prior; if the diffusion model is poorly matched to the target domain or exhibits artifacts under certain noise schedules, the denoising updates may drift the image away from the true manifold. Finally, while our experiments span CIFAR- 10, CIFAR- 100, TinylmageNe with different classifiers of WideResNet and a CLIP zero-shot classifier, further evaluation diverse modalities, and real-time scenarios is needed to fully characterize AAOpt's generality and scalability. Future work will focus on improving the model and extending this work to multimodal generative (Wesego and Rooshenas
2024 models.
27


Reviews:
[{"review_id": "ruKjW2piqV", "reviewer": "Reviewer_Ds3y", "summary": "The paper proposes a new adversarial defense method through adversarial purification that recovers the clean image from a perturbed observation. It uses two diffusion models to help construct the objective function for optimizing the clean image estimation. The proposed method is compared against a series of adversarial defense methods, including both adversarial training and adversarial purification, through benchmark testing. The results demonstrate superior robust accuracy with satisfactory standard accuracy for the proposed method, under different attacks.", "questions": "I invite the authors to clarify a couple of things as below. The answers will help me assess the quality, particularly on “Is the submission technically sound?”, based on which I will adjust my score.\n\nCan the authors expand on how to derive Eq. (4) by substituting the two mentioned equations?\n\nAccording to Section 2.3, the two diffusion models for modelling clean data and perturbation distributions are both score based, i.e., both s_{\\theta} and d_{\\phi} predict the score functions. However, according to Section 4, the used pretrained diffusion model for clean data predicts the denoised estimate. Can the authors explain how do they adapt such a predictor for score function prediction? Also, according to Algorithm 2 in Appendix, the adversarial predictor d_{\\phi} predicts the noise (line 7). Can the authors explain why it is ok to use this noise predictor as the score model required by Eq. (5)?\n\nIn line 174/175, it mentions that both clear and adversarial examples can be selected to pair with a clean image. But line 2 of Algorithm 2 in Appendix does not seem to reflect this. Can the authors check and explain this?\n\nWhen preparing image pairs for training the perturbation diffusion model, by randomly selecting images to pair, there is a chance to produce image pairs each containing two very different images. Isn’t this violating the perturbation strength constraint, e.g., roughly ||x - \\hat{x}|| <= \\epsilon? Can the authors comment on this?", "limitations": "Yes, there is limitation discussion in appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThe paper has introduced a novel adversarial purification method, advancing the research area of adversarial defence. The proposed method is elegant, achieving good performance. The paper is in general well-written, easy to follow.\n\nWeaknesses:\n\nThere are a few technical details that are not exactly clear and  require explanation and clarification. For instance, some more details on stop-gradient and how the objective function is optimised would be helpful. Also see my question later.\n\nAll evaluations and experiment design are aimed to demonstrate the success of the proposal. There is no experiment dedicated to evaluate/analyse in depth the weakness or failure cases of the proposed method.", "was_revised": true, "final_justification": "The authors clarified my questions, and responded on experiment side. Based on these I raise my rating.", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful review. We address your concerns below.\n\n*1. \"Can the authors expand on how to derive Eq. (4) by ...\"*\n\nWe will derive the equation below. There was a minor error when writing the equation, which we will fix below and in the camera-ready version if accepted.\n\nUsing Taylor expansion of $p_t$ distribution around $\\mathbf{x}_t$:\n\n$\\log p_t(\\mathbf x) \\approx \\log p_t(\\mathbf x_t) + (\\mathbf x-\\mathbf x_t)^T \\nabla_{\\mathbf x_t} \\log p_t(\\mathbf x_t) = \\log p_t(\\mathbf x_t) - \\sigma_t \\boldsymbol{\\epsilon}^T s_\\theta(\\mathbf x_t, t)$\n\nUsing Bayes rule, we have:\n\n$\\log p(\\mathbf x) \\approx \\log p_t(\\mathbf x_t) + \\log p(\\mathbf x| \\mathbf x_t) + \\frac{1}{2\\sigma_t^2}|\\mathbf x- \\mathbf x_t|^2$\n\nBy replacing $\\log p_t(\\mathbf x_t)$ from the Taylor approximation, we get:\n\n$\\log p(\\mathbf x) \\approx \\sigma_t \\boldsymbol{\\epsilon}^T s_\\theta(\\mathbf x_t, t) + \\frac{1}{2\\sigma_t^2}|\\mathbf x- \\mathbf x_t|^2 + \\log p_t(\\mathbf x) + \\log p(\\mathbf x|\\mathbf x_t)$\n\nThe MAP estimate of $\\log p(x)$ would have the same solution as:\n\n$\\log p(\\mathbf x) \\propto \\sigma_t \\boldsymbol{\\epsilon}^T s_\\theta(\\mathbf x_t, t) + \\frac{1}{2\\sigma_t^2}|\\mathbf x- \\mathbf x_t|^2$\n\nwhich can be rephrased as:\n\n$\\log p(\\mathbf x) \\propto \\sigma_t \\epsilon^T s_\\theta(\\mathbf x_t, t) + \\frac{1}{2} \\boldsymbol \\epsilon^T \\boldsymbol \\epsilon = \\epsilon^T(\\sigma_t s_\\theta(\\mathbf x_t, t) + \\frac{1}{2}\\epsilon) = - \\frac{1}{\\sigma_t}(x - x_t)^T(\\sigma_t s_\\theta(\\mathbf x_t, t) + \\frac{1}{2}\\epsilon) = \\mathbf x^T (-s_\\theta(\\mathbf{x}t,t) - \\frac{1}{\\sigma_t} \\boldsymbol \\epsilon) + \\mathbf x_t^T (s\\theta(\\mathbf{x}_t,t) + \\frac{1}{\\sigma_t} \\boldsymbol \\epsilon)$\n\nSince we only care about the MAP estimate on $p(\\mathbf x)$, we can ignore the last term as it does not directly depend on $\\mathbf x$.\n\nTherefore:\n\n$\\log p(\\mathbf x) \\propto \\mathbf x^T (-\\sigma_t s_\\theta(\\mathbf{x}_t,t) - \\boldsymbol \\epsilon)$\n\n\n*2. \"According to Section 2.3, the two diffusion model ... \"*\n\nThank you for the observation. Since we can extract the noise $\\epsilon_\\theta$ from the score function $s_\\theta$ regardless of how the diffusion/score/denoiser model was trained, we used the terms somewhat loosely in the main text. To clarify: we indeed extract $\\epsilon_\\theta$ for both the prior and perturbation terms in our framework, finally. \n\nScore-based models and diffusion models are part of the same family of generative models that transform noise into a data distribution [3,4]. The score $\\nabla_{x_t} \\log p_t(x_t)$ can be expressed in terms of the noise as $\\nabla_{x_t} \\log p_t(x_t) = -\\frac{1}{\\sigma(t)}\\epsilon $, which can be easily derived by differentiating the Gaussian distribution $\\log p_t(x_t)$ with respect to $x_t$. We extract the $\\epsilon_{\\theta}$ from the prior diffusion model which predicts the denoised estimate as $\\epsilon_{\\theta} = \\frac{x_t - D(x_t)}{\\sigma(t)}$ where $D(x_t)$ is the denoised estimate [5]. We trained a model that directly predicts the $\\epsilon_{\\theta}$ for the perturbation model, so we can directly take $\\epsilon_{\\theta}$ for that part.  \n\n*3. \"In line 174/175, it mentions that both ... \"*\n\nThank you for reviewing our paper in detail. In fact, you are the only person who read the appendix well, and we thank you for that. Algorithm 2 at the top says ${x^k}_{k=1}^K$ is composed by combining clean and adversarial sets. So, even if line 2 labels it as such, the set is composed of the combination. We do this because in the real-world application of our method, we don't know if a given $\\hat{x}$ is clean or adversarial. To ensure that the perturbation model also learns about data that doesn't have any adversarial perturbation, we included the clean examples in the set. \n\n\n*4. \"When preparing image pairs for training ...\"*\n\nThe image pairs are created by randomly selecting samples from the dataset, but they are originally constructed as pairs — each image has both a clean version and an adversarial (or possibly clean) counterpart. This ensures that each pair corresponds to the same base image, maintaining consistency during training.\n\nWe hope the above answers clarified your questions. We will be happy to answer if you have any further questions.\n\n\n1. Mardani, M., Song, J., Kautz, J., & Vahdat, A. (2023). A variational perspective on solving inverse problems with diffusion models. ICLR 2024. \\\\\n2. Vahdat, A., Kreis, K., & Kautz, J. (2021). Score-based generative modeling in latent space. Advances in neural information processing systems, 34, 11287-11302. \\\\\n3. Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2020). Score-based generative modeling through stochastic differential equations. ICLR 2021. \\\\\n4. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Advances in neural information processing systems, 33, 6840-6851. \\\\\n5. Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35, 26565-26577."}, {"review_id": "N2phei0PJa", "reviewer": "Reviewer_K1n9", "summary": "This paper introduces Adversary Aware Optimization (AAOpt), a test-time defense framework that combines a pretrained diffusion prior with a learned perturbation score model to perform input purification. The method formulates the purification process as a MAP estimation problem, integrating both the data prior and an adversarial likelihood term. The likelihood is modeled by a lightweight score network trained on adversarial residuals. Compared with prior diffusion-based defenses, the proposed method demonstrates improved robust accuracy under strong adaptive attacks on CIFAR-10 and CIFAR-100.", "questions": "See the weaknesses given above.", "limitations": "The paper briefly mentions test-time cost and one-time adversarial data generation as limitations, but it does not discuss the lack of evaluations on more complex datasets or the theoretical gap in the objective derivation. Including a clearer discussion of generalization and theoretical limitations would help strengthen the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper addresses a critical issue in adversarial purification, namely the failure of diffusion models to account for structured adversarial perturbations, by explicitly incorporating an adversarial score model.  The proposed optimization framework is efficient and avoids full reverse diffusion sampling.  Experimental results show consistent improvement over both adversarial training and prior purification methods across CIFAR-10 and CIFAR-100, under PGD-EOT and BPDA-EOT attacks.\n\nWeaknesses:\n1. **Limited Dataset Scope**  \nThe experiments are restricted to CIFAR-10 and CIFAR-100. Previous works on purification (e.g., DiffPure, ScoreOpt) often include TinyImageNet and ImageNet subsets to better evaluate scalability and generalization. The lack of larger-scale datasets raises doubts about the method’s applicability beyond small resolution benchmarks.\n\n2. **Unclear Derivation in Method Section**  \nThe derivation of Equation (4) appears problematic. Specifically, the term involving `−ϵ` in the surrogate for  \n\\[\n\\log p(x) \\propto x^\\top \\left[ s_\\theta(x_t, t) - \\epsilon \\right]\n\\]  \nis not rigorously justified and seems to be heuristically adapted from Tweedie’s formula. Given that this surrogate plays a central role in the optimization objective, a more precise derivation or theoretical justification is needed.\n\n3. **One-Time Attack Generation Overhead**  \nThe proposed perturbation score model, although efficient, requires a full round of adversarial data generation prior to test-time deployment. This pre-processing cost is non-trivial and could limit scalability.\n\n4. **Hyperparameter Sensitivity**  \nAlthough some ablation studies are presented, it remains unclear how sensitive the method is to the choice of \\( \\lambda \\) and other hyperparameters, especially on unseen architectures.\n\n5. **Lack of Comparison with Stronger Adaptive Attacks**  \nAlthough PGD-EOT and BPDA-EOT are included, the paper does not evaluate against more recent and stronger purification-aware adaptive attacks such as done in [1].\n\n6. **Unclear Scalability to High-Resolution Tasks**  \nThe method’s practicality on high-resolution datasets remains questionable. The reliance on pixel-level gradient updates and score models trained at small resolution may not transfer well to more realistic image sizes used in safety-critical applications.\n\n7. **Limited Theoretical Depth**  \nWhile the method is motivated from a MAP perspective, the paper lacks a rigorous theoretical analysis regarding convergence guarantees, stability of the optimization procedure, or robustness bounds. This makes it difficult to assess the theoretical soundness of the proposed surrogate objective.\n\n[1] DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification, NeurIPS 2024", "was_revised": true, "final_justification": "The authors have addressed some concerns, but not all. I will maintain my original rating, as it already accounted for some possibly positive responses from the authors.", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful review. We address your concerns below.\n\nWe submitted our appendix as a zip file with the supplementary materials, following one of the two allowed methods for NeurIPS submissions. Although we understand you are not obligated to review it, some content couldn’t fit in the main paper, necessitating its inclusion there. The appendix answers most of your questions, and we hope you look at it.\n\n*1. \"Limited Dataset Scope \"*\n\nWe included a comparison with Score-Opt under the same setup on the TinyImageNet dataset (Section 6.1.5). While the results could potentially be improved further by using stronger diffusion models and classifiers, our method already outperforms Score-Opt under the exact same conditions.\n\n*2. \"Unclear Derivation in Method Section \"*\n\nWe will derive the equation below. There was a minor error when writing the equation, which we will fix below and in the camera-ready version if accepted.\n\nUsing Taylor expansion of $p_t$ distribution around $\\mathbf{x}_t$:\n\n$\\log p_t(\\mathbf x) \\approx \\log p_t(\\mathbf x_t) + (\\mathbf x-\\mathbf x_t)^T \\nabla_{\\mathbf x_t} \\log p_t(\\mathbf x_t) = \\log p_t(\\mathbf x_t) - \\sigma_t \\boldsymbol{\\epsilon}^T s_\\theta(\\mathbf x_t, t)$\n\nUsing Bayes rule, we have:\n\n$\\log p(\\mathbf x) \\approx \\log p_t(\\mathbf x_t) + \\log p(\\mathbf x| \\mathbf x_t) + \\frac{1}{2\\sigma_t^2}|\\mathbf x- \\mathbf x_t|^2$\n\nBy replacing $\\log p_t(\\mathbf x_t)$ from the Taylor approximation, we get:\n\n$\\log p(\\mathbf x) \\approx \\sigma_t \\boldsymbol{\\epsilon}^T s_\\theta(\\mathbf x_t, t) + \\frac{1}{2\\sigma_t^2}|\\mathbf x- \\mathbf x_t|^2 + \\log p_t(\\mathbf x) + \\log p(\\mathbf x|\\mathbf x_t)$\n\nThe MAP estimate of $\\log p(x)$ would have the same solution as:\n\n$\\log p(\\mathbf x) \\propto \\sigma_t \\boldsymbol{\\epsilon}^T s_\\theta(\\mathbf x_t, t) + \\frac{1}{2\\sigma_t^2}|\\mathbf x- \\mathbf x_t|^2$\n\nwhich can be rephrased as:\n\n$\\log p(\\mathbf x) \\propto \\sigma_t \\epsilon^T s_\\theta(\\mathbf x_t, t) + \\frac{1}{2} \\boldsymbol \\epsilon^T \\boldsymbol \\epsilon = \\epsilon^T(\\sigma_t s_\\theta(\\mathbf x_t, t) + \\frac{1}{2}\\epsilon) = - \\frac{1}{\\sigma_t}(x - x_t)^T(\\sigma_t s_\\theta(\\mathbf x_t, t) + \\frac{1}{2}\\epsilon) = \\mathbf x^T (-s_\\theta(\\mathbf{x}t,t) - \\frac{1}{\\sigma_t} \\boldsymbol \\epsilon) + \\mathbf x_t^T (s\\theta(\\mathbf{x}_t,t) + \\frac{1}{\\sigma_t} \\boldsymbol \\epsilon)$\n\nSince we only care about the MAP estimate on $p(\\mathbf x)$, we can ignore the last term as it does not directly depend on $\\mathbf x$.\n\nTherefore:\n\n$\\log p(\\mathbf x) \\propto \\mathbf x^T (-\\sigma_t s_\\theta(\\mathbf{x}_t,t) - \\boldsymbol \\epsilon)$\n\n\n*3. \"One-Time Attack Generation Overhead \"*\n\nWe do not require adversarial generation for test-time, but only during training of our perturbation model. And\ncompared to adversarial training papers that continuously generate adversarial examples, our method only requires the initial one-time generation of adversarial examples. Even though this may have some cost, it's limited and can be saved once and reused again and again during training, which significantly reduces the cost. And this is not required for test-time, it's only required once during training of the perturbation model. Crucially, no adversarial generation is needed during inference—only optimization is performed at test time.\n\n*4. \"Hyperparameter Sensitivity \"*\n\nAs shown in our ablation study (Figure 2), performance improves consistently over the prior diffusion as the hyperparameter $\\lambda$ varies within the range 0 to 1. Similar to other hyperparameters in deep learning, $\\lambda$ can be selected using a validation set (or a subset of the training data if no separate validation set is available). We observe that the optimization is generally stable, and tuning $\\lambda$ is not particularly sensitive or difficult in practice.\n\n*5. Lack of Comparison with Stronger Adaptive Attacks\nAlthough PGD-EOT and BPDA-EOT are included, the paper does not evaluate ...*\n\nWe used PGD-EOT because it is currently one of the strongest attacks. In [1], it can be seen that the Robust Accuracy of DiffPure under PGD-EOT is lower than the Robust Accuracy of DiffPure under DiffAttack, showing that PGD-EOT is stronger than DiffAttack, which is proposed by [2]. We also observed that our model performs well under DiffAttack, which we have shown below. We used the same setup and hyperparameters as our model used for the PGD-EOT attack. \n\n--------------------------------------------------\n |       AAOpt Clean Acc |    91.6                  |\n\n |    AAOpt Robust Acc  |    87.5                   |\n\nTab1: DiffAttack AAOpt Performance on $l_\\infty(\\epsilon = \\frac{8}{255})$ using WRN-28-10\n\n*6. \"Unclear Scalability to High-Resolution Tasks \"*\n\nWe have demonstrated our results and comparisons on widely used adversarial benchmarks such as CIFAR-10 and CIFAR-100, and additionally included TinyImageNet to extend the evaluation. As long as a diffusion model is well-trained on higher-resolution datasets, our framework remains applicable. Naturally, scaling any generative model to high-resolution tasks presents challenges, and generating adversarial examples at those scales—especially with iterative gradient-based methods—can be computationally intensive. This is also why many prior works limit evaluations to smaller datasets. Nonetheless, the focus of our current study is to address adversarial robustness on standard, widely-used benchmarks. \n\n*7. \"Limited Theoretical Depth\"*\n\nWhile our method is motivated by a MAP formulation, we acknowledge that the paper does not present formal convergence guarantees or robustness bounds. However, our approach is rooted in established likelihood maximization principles commonly used in generative modeling and optimization literature. We have expanded the theoretical background in the revised version to clarify the underlying rationale and assumptions. In practice, our optimization procedure is stable, as supported by extensive empirical results.\n\nAlthough we do not provide formal robustness bounds, we believe this does not undermine the soundness of the approach. Moreover, the response to Question 2 provides further insight into the surrogate objective and its practical generalization.\n\nWe hope the above responses have clarified your concerns. Many of the points raised were also addressed in the appendix, and we appreciate your comments. Let us know if you have any further questions, and we kindly ask you to reconsider your score in light of these clarifications.\n\n1. Lee, M., & Kim, D. (2023). Robust evaluation of diffusion-based adversarial purification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 134-144). \n\n2. Kang, M., Song, D., & Li, B. (2023). Diffattack: Evasion attacks against diffusion-based adversarial purification. Advances in Neural Information Processing Systems, 36, 73919-73942."}, {"review_id": "e45Agnb2Wk", "reviewer": "Reviewer_PpXk", "summary": "This paper proposes a novel diffusion model–based purification method to defend against adversarial perturbations. The authors identify two limitations of prior work: adversarial training incurs high retraining costs, and existing purification approaches via diffusion model without retraining implicitly treat adversarial perturbations as generic Gaussian noise, resulting in insufficient robustness. To address this, the authors introduce a new score-based optimization framework. Specifically, they train a perturbation score network to estimate the distribution of adversarial directions and leverage it to integrate knowledge of the adversarial landscape into the purification process.", "questions": "• Compared to methods that integrate adversarial perturbation information into the purification process, what novel aspects does this approach introduce?\n• How does robustness change when the attack settings or datasets used at inference differ from those in training? Does the defense generalize across such variations?\n• How much larger is AAOpt’s inference cost compared to existing methods?\n• Could this framework be extended to provide certified defenses?", "limitations": "The authors claim that inference-time cost is high, but it is unclear how much overhead there is. Thus, the inference cost should be compared with existing methods.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strength:\nThis paper clearly identifies the weakness in existing purification methods that implicitly treat adversarial perturbations as Gaussian noise, and proposes a lightweight score-based approach to address it. Empirical results demonstrate its effectiveness across some settings, and the provided code ensures reproducibility.\n\nWeakness:\nThe technical contribution of the proposed method is incremental. Previous studies have already applied score‐based diffusion models to adversarial purification [1, 2], and other works have incorporated adversarial‐perturbation information directly into the purification process [3, 4]. Furthermore, the paper does not cite [3, 4] and fails to explain how its method differs from theirs or to provide any experimental performance comparison.\n\nMinor Comments\n• The proposed method still depends on training the perturbation‐score network with adversarial examples, making it vulnerable to unseen threat models and distribution shifts. Consequently, robustness may degrade when the attack budget or dataset changes. However, such robustness under varying conditions is not evaluated in the paper.\n• There is no quantitative assessment of the inference‐time cost. While the authors acknowledge this as a limitation, they do not provide experimental measurements.\n• The proposed method lacks theoretical guarantees. Some prior diffusion-based purification methods provide formal robustness proofs [5].\n\n[1] Nie, Weili, et al. \"Diffusion Models for Adversarial Purification.\" International Conference on Machine Learning. PMLR, 2022.\n[2] Zhang, Boya, Weijian Luo, and Zhihua Zhang. \"Enhancing adversarial robustness via score-based optimization.\" Advances in Neural Information Processing Systems 36 (2023): 51810-51829.\n[3] Li, Xiao, et al. \"ADBM: Adversarial diffusion bridge model for reliable adversarial purification.\" ICLR. 2025.\n[4] Bai, Mingyuan, et al. \"Diffusion models demand contrastive guidance for adversarial purification to advance.\" Proceedings of the 41st International Conference on Machine Learning. 2024.\n[5] Zhang, Jiawei, et al. \"{DiffSmooth}: Certifiably robust learning via diffusion models and local smoothing.\" 32nd USENIX Security Symposium (USENIX Security 23). 2023.", "was_revised": true, "final_justification": "Thank you very much for providing the additional experiments. My concerns have been fully addressed. I will raise my score.", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful review. We address your concerns below.\n\nWe submitted our appendix as a zip file with the supplementary materials, following one of the two allowed methods for NeurIPS submissions. Although we understand you are not obligated to review it, some content couldn’t fit in the main paper, necessitating its inclusion there. The appendix answers most of your questions, and we hope you look at it.\n\nYour questions and minor comments overlap for the most part. So, we will try to answer them together as much as possible.\n\n*1. \"Compared to methods that integrate adversarial perturbation information into the purification process, what novel aspects does this approach introduce? \"*\n\n*\"The technical contribution of the proposed method is incremental. Previous studies have already applied score‐based diffusion models to adversarial purification, and other works have incorporated adversarial‐perturbation information directly into the purification process ...\"*\n\nWe believe our method introduces a novel and distinctive contribution to adversarial purification. Specifically, we are the first to propose incorporating a perturbation model within a MAP (Maximum A Posteriori) optimization framework to explicitly maximize the likelihood of clean data during purification.\n\nOur model uniquely combines:\n\n  - A prior diffusion model, trained on clean data, which captures the data manifold accurately.\n\n  - A modular perturbation model, trained to capture the common structures of adversarial perturbations, which serves as a flexible guide toward clean data.\n\nThis perturbation model enhances purification performance by learning the attack distribution. It can be adjusted via a weighting parameter λ, or even removed entirely, offering flexibility depending on the scenario.\n\nIn contrast, related works such as [2] (which resembles [3] in our citations) employ guidance mechanisms during reverse diffusion, and [4] modifies the diffusion objective itself to guide adversarial examples toward the clean distribution. This method typically requires retraining the diffusion model and generating adversarial examples over many iterations for training the model, unlike ours.\n\nOur approach avoids these drawbacks by first keeping the diffusion prior intact, ensuring robustness across general settings. Then, our modular and lightweight perturbation model can generalize to different attacks. Our results also outperform existing methods, as demonstrated in our empirical results. We believe this formulation provides a more flexible, efficient, and principled way to integrate adversarial information into the optimization process.\n\n*2. \"How does robustness change when the attack settings or datasets used at inference differ from those in training? Does the defense generalize across such variations?\"*\n\nOur approach exhibits strong robustness and generalization across varying attack settings and classifier architectures.\n\n**Classifier Transferability**: Table 5 in the main paper shows that our perturbation model trained with one classifier (e.g., WRN-28-10) continues to perform well when applied to a different classifier (WRN-70-16), despite architectural differences. While these two models are relatively similar, we went further to assess robustness more thoroughly.\n\n**Cross-Architecture Generalization**: To test generalization across significantly different classifier families, Table 9 reports results when the perturbation model trained on WRN-28-10 is applied to a CLIP zero-shot classifier—a radically different architecture. The performance remains strong, demonstrating our method’s generalization beyond similar backbones.\n\n**Cross-Norm Generalization**: We also evaluated how a perturbation model trained under one threat model generalizes to another. Specifically, a model trained on $l_\\infty$-bounded attacks ($\\epsilon = \\frac{8}{255}$) performs well when tested on $l_2$-bounded attacks ($\\epsilon = 0.5$), confirming its cross-norm robustness.\n\n-----------------------------------------------\n |       AAOpt Clean Acc |    91.57             |\n\n |    AAOpt Robust Acc  |    86.22            |\n\nTab1: AAOpt Performance on $l_\\infty(\\epsilon = \\frac{8}{255})$ PGD-EOT attack using a perturbation model trained on $l_2 (\\epsilon =0.5)$\n\n -----------------------------------------------\n |       AAOpt Clean Acc |    91.71              |\n\n |    AAOpt Robust Acc  |    87.07              |\n\n Tab2: AAOpt Performance on $l_2 (\\epsilon =0.5)$ PGD-EOT attack using a perturbation model trained on $l_\\infty(\\epsilon = \\frac{8}{255})$\n\n**Robustness to Natural Corruptions**: Additionally, Table 6 in the main paper shows our defense is effective not just against adversarial examples, but also against common corruptions such as noise, blur, and weather-based distortions.\n\nOverall, these experiments confirm that our method generalizes effectively across classifier architectures, perturbation norms, and natural distribution shifts.\n\n*3. \"How much larger is AAOpt’s inference cost compared to existing methods?\"*\n\nThis question is addressed in the ablation study (Table 12 in the appendix).\n\nWe compare AAOpt to Score-Opt [1], which also purifies adversarial examples using iterative optimization.\n\nScore-Opt [1] has demonstrated that it is faster than reverse diffusion-based methods (e.g., DiffPure), since optimization requires fewer steps than full reverse sampling.\n\nDespite being iterative, our method is faster than Score-Opt for purifying adversarial examples, due to the efficiency of our optimization procedure, as shown in Table 12.\n\n*4. \"Could this framework be extended to provide certified defenses? Some prior diffusion-based purification methods provide formal robustness proofs.\"*\n\nOur current focus is on optimizing the posterior likelihood to remove adversarial perturbations and move samples closer to the clean distribution.\n\nAt this time, our method does not provide certified robustness guarantees, but we view this as a valuable direction for future work and plan to explore it.\n\nWe hope the responses above help clarify your questions. Many of these points are addressed in the appendix, but we’ve expanded them here for clarity. We would be happy to respond to any follow-up questions you may have, and we kindly ask you to reconsider the score if this has addressed your concerns.\n\n[1] Zhang, B., Luo, W., & Zhang, Z. (2023). Enhancing adversarial robustness via score-based optimization. Advances in Neural Information Processing Systems, 36, 51810-51829.\n\n[2] Bai, Mingyuan, et al. \"Diffusion models demand contrastive guidance for adversarial purification to advance.\" Proceedings of the 41st International Conference on Machine Learning. 2024.\n\n[3] Lin, G., Tao, Z., Zhang, J., Tanaka, T., & Zhao, Q. (2025). Adversarial guided diffusion models for adversarial purification. Neural Networks, 107705.\n\n[4] Li, X., Sun, W., Chen, H., Li, Q., Liu, Y., He, Y., ... & Hu, X. (2024). Adbm: Adversarial diffusion bridge model for reliable adversarial purification. ICLR 2025."}, {"review_id": "EfzWlXMQfX", "reviewer": "Reviewer_9umZ", "summary": "This work introduces AAOpt, an adversarial purification method for test-time adversarial defense. Compared with existing purification methods, the key difference/novelty lies in that AAOpt considers an extra diffusion model that explicitly characterizes the adversarial perturbations (rather than \"normal\" diffusion models that assume a gaussian perturbation). Experiments on CIFAR-10 and CIFAR-100 demonstrates improved results compared with adversarial training and other purification baselines.", "questions": "Please see Strengths and Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "While my expertise is in adversarial attacks and adversarial training, I don't closely follow the line of works on diffusion-based adversarial purification, so I may not be able to accurately assess the strengths/significance and weaknesses of this work.\n\n### Strengths\n1. It definitely makes sense to consider adversary-aware diffusion models for purification-based defense.\n\n### Weaknesses\n1. The setup in Table 5 regarding transferablity is too optimistic, i.e., models from the same architecture family (WRN) is being used. I think it's important to evaluate transferability between more distinct models, as I would imagine that the proposed method (which trains the diffusion model with certain attacks generated with a certain network) would be less transferable than previous purification-based methods that use pre-trained diffusion models.\n\n### Questions\n\n1. The perturbation model $d_\\phi$ is trained with $\\ell_\\infty$ and $\\ell_2$ attacks. What if the test-time attacks are with different norm? What would be the consideration here to make it generalize?\n2. Is it a standard to evaluate purification methods with EOT attacks? What about AutoAttack?", "was_revised": true, "final_justification": "My questions are addressed in the rebuttal, and I don't have any major concern left. I raise my score from 3 to 4.", "rebuttal": "Dear Reviewer,\n\nThank you for your thoughtful review. We address your concerns below.\n\nWe submitted our appendix as a zip file with the supplementary materials, following one of the two allowed methods for NeurIPS submissions. Although we understand you are not obligated to review it, some content couldn’t fit in the main paper, necessitating its inclusion there.\n\n*Comment: \"The setup in Table 5 regarding transferablity is too optimistic, i.e., models from the same architecture family (WRN) is being used. I think it's important to evaluate transferability between more distinct models, as I would imagine that the proposed method (which trains the diffusion model with certain attacks generated with a certain network) would be less transferable than previous purification-based methods that use pre-trained diffusion models.\" and \nQ1: \"The perturbation model $d_{\\phi}$ is trained with $l_2$ and $l_\\infty$\nattacks. What if the test-time attacks are with different norm? What would be the consideration here to make it generalize?*\n\nWe have thought about that and have already addressed some of your concerns in the Appendix. In Table 9, we study the transferability of our attack using a completely different classifier architecture of a CLIP one-shot model. Our model performs better than the baselines. This experiment, in addition to Table 5, shows that using a specific perturbation model is a better way to generalize to different classifiers and attack techniques. \n\nIn addition to that, to answer your Question 1, we have added experiments on norm transferability. We train a perturbation only on one type of norm and evaluate it on a different norm. The results are very close to the one trained using joint attack norms. The results are shown below:\n\n--------------------------------------------------\n |       AAOpt Clean Acc |    91.57                  |\n\n |    AAOpt Robust Acc  |    86.22                   |\n\nTab1: AAOpt Performance on $l_\\infty(\\epsilon = \\frac{8}{255})$ PGD-EOT attack using a perturbation model trained on $l_2 (\\epsilon =0.5)$ using WRN-28-10\n\n --------------------------------------------------\n |       AAOpt Clean Acc |    91.71                   |\n\n |    AAOpt Robust Acc  |    87.07                   |\n\n Tab2: AAOpt Performance on $l_2 (\\epsilon =0.5)$ PGD-EOT attack using a perturbation model trained on $l_\\infty(\\epsilon = \\frac{8}{255})$ using WRN-28-10\n\n\n*2. \"Is it a standard to evaluate purification methods with EOT attacks?\"*\n\nYes, it's a standard technique and very needed as diffusion-based methods vary, and averaging over multiple transformations ensures accurate results [1].\n\n*3. \" What about AutoAttack? \"*\n\nWe have already added AutoAttack results in the Appendix because we ran out of space in the main paper. Table 7 demonstrates the results during AutoAttack, but this attack is not as strong as the PGD-EOT attack [1]. Our model performs well in this evaluation too.\n\nWe hope we have answered your questions, and we kindly ask you to reconsider the score if this has addressed your concerns. We invite you to also look at the appendix, if possible.\n\n1. Lee, M., & Kim, D. (2023). Robust evaluation of diffusion-based adversarial purification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 134-144)."}]

Author rebuttal:

Meta-review: This paper proposes Adversary Aware Optimization (AAOpt), an adversarial purification framework for defending deep neural networks against adversarial attacks. The key innovation lies in explicitly modeling adversarial perturbations using a lightweight score-based diffusion model, which is combined with a pretrained diffusion prior to guide the purification process. The method formulates purification as a Maximum A Posteriori (MAP) estimation problem, integrating both the clean data distribution and adversarial perturbation structure. Experiments on CIFAR-10/100 demonstrate improved robust accuracy against adaptive attacks compared to existing adversarial training and purification baselines. The authors also validate generalization across classifier architectures and perturbation norms.

This paper has several strengths. It shows strong defense against adaptive attacks, with ablation studies showing consistent improvements over baselines, such as DiffPure and Score-Opt. The modular design allows the perturbation model to be adjusted, offering adaptability to different threat scenarios. The generalization is evidenced by robustness across classifier architectures and perturbation norms. However, the reviewers also raised several concerns. The scope of evaluation is limited, where the evaluation is conducted on small-resolution datasets, and the scalability to high-resolution tasks remains unverified. Additionally, the derivation of the surrogate objective (Eq. 4) lacks rigorous justification, and no formal robustness guarantees are provided. Also, the computational overhead has not been clearly stated. While efficient compared to full reverse diffusion, the inference-time cost is not thoroughly benchmarked.

During the rebuttal phase, the authors have addressed most concerns raised by the reviewers. Specifically, Reviewer Ds3y and K1n9 questioned the derivation of Eq. 4 and score/noise predictor consistency. The authors provided detailed derivations and clarified implementation alignment with prior work. Reviewer PpXk raised concern about the incremental contribution and missing comparisons to ADBM. The authors added experiments showing AAOpt’s superiority over ADBM. Reviewer 9umZ questioned about cross-norm and cross-architecture transferability. The authors provided new results $l_\infty$-trained model tested on $l_2$ attacks. Although some concerns remain after the rebuttal, such as evaluation on high-resolution tasks, most concerns have been addressed, and all the reviewers lean towards acceptance.

The paper’s empirical contributions and further clarifications outweigh its limitations, making it a solid addition to the adversarial robustness literature. The authors are encouraged to improve the paper by considering the reviewers’ comments carefully.
Decision: Accept (poster)