DiffBreak: Is Diffusion-Based Purification Robust?
Authors: Andre Kassis
Urs Hengartner
Yaoliang Yu
Venue: NeurIPS 2025 poster ()
Keywords: Adaptive Adversarial Attacks
Adversarial Purification
Diffusion Models
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: DiffBreak provides the first reliable framework for differentiating through diffusion-based purification, revealing key vulnerabilities under adaptive attacks.

Abstract: Diffusion-based purification (DBP) has become a cornerstone defense against adversarial examples (AEs), regarded as robust due to its use of diffusion models (DMs) that project AEs onto the natural data manifold. We refute this core claim, theoretically proving that gradient-based attacks effectively target the DM rather than the classifier, causing DBP's outputs to align with adversarial distributions. This prompts a reassessment of DBP's robustness, accrediting it two critical factors: inaccurate gradients and improper evaluation protocols that test only a single random purification of the AE. We show that when accounting for stochasticity and resubmission risk, DBP collapses. To support this, we introduce DiffBreak, the first reliable toolkit for differentiation through DBP, eliminating gradient mismatches that previously further inflated robustness estimates. We also analyze the current defense scheme used for DBP where classification relies on a single purification, pinpointing its inherent invalidity. We provide a statistically grounded majority-vote (MV) alternative that aggregates predictions across multiple purified copies, showing partial but meaningful robustness gain. We then propose a novel adaptation of an optimization method against deepfake watermarking, crafting systemic perturbations that defeat DBP even under MV, challenging DBP's viability.

DiffBreak: Is Diffusion-Based Purification Robust?
Andre Kassis, Urs Hengartner, Yaoliang Yu
Cheriton School of Computer Science, University of Waterloo Waterloo, Ontario, Canada
fakassis, urs hengartter, yaoliang. yu) @uwaterroooo ca
Abstract
Diffusion- based purification (DBP) has become a comnestone defense against adversarial examples (AEs), regarded as robust due to its use of diffusion models (DMs) that project AEs onto the natural data manifold. We refute this core claim, theoretically proving that gradient- based attacks effectively target the DM rather than the classifier, causing DBP's outputs to align with adversarial distributions. This prompts a reassessment of DBP's s robustness, accrediting it two critical fac- tors: inaccurate gradients and improper evaluation protocols that test only single random purification of the AE. We show that when accounting for stochasticity and resubmission risk, DBP collapses. To support this, introduce DiffBreak, the first reliable toolkit for differentiation through DBP, eliminating gradient mismatches that previously further inflated robustness estimates. We also analyze the current defense scheme used for DBP where classification relies on a single purification, pinpointing its inherent invalidity. We provide statistically grounded majority- vote (MV) alternative that aggregates predictions across multiple purified copies, showing partial but meaningful robustness gain. We then propose novel adapta- tion of an optimization method against deepfake watermarking, crafting systemic perturbations that defeat DBP even under MV, challenging DBP's viability.
Introduction
ML classifiers are vulnerable to Adversarial Examples imppereippeee perturbations that induce misclassification [37, 15]. Adversarial Training [28, 54] is attack-specific and costly [44], while other defenses [16, 13, 3, 46, 49, 32, 50, 9] are vulnerable to adaptive attacks [38]. Diffusion- Based Purification (DBP) [30, 40]. which leverages diffusion models (DMs) [36, 34], has emerged
promising defense [6, 45, 7, 53, 31, 42]. DBP purifies inputs by solving the reverse stochastic differential equation (SDE) associated with DMs, diffusing the input with random noise to dilute adversarial changes and iteratively denoising it. Given DMs' data modeling abilities [12, 39]. DBP operates under the assumption that each denoising (reverse) step's output belongs to a corresponding marginal natural distribution (a distribution obtained by introducing Gaussian noise into natural inputs, with decreasing variance proportional to that step).
This is the foundation for DBP's robustness as it ensures its outputs are from the natural distribution, while AEs lie outside this manifold and are unlikely be preserved. Recent works treat DBP as static pre- pre-processor that standard meddient-based attacks cannot directly manipulate but rather only attempt to evade. Thus, attackers are assumed to either fail due to randomness that masks the classifier's vulnerabilities [26]. or be forced to rely surrogate losses and heuristic approximations [20, 41]. We theoretically refute DBP's robustness rooted its ability to perform projection onto the natural manifold. This is paradoxical as it assumes correct behavior of the score model So used by the DM to mimic the gradients of the marginal distributions. Yet, 88 is an ML system with exploitable imperfections. In $3.1, we prove that standard adaptive attacks that simply backpropagate the classifier's s loss gradients through DBP effectively target 89 rather than the classifier, forcing it to generate samples from an adversarial distribution. Hence, DBP's theory longer holds.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

fa Forward diffasion
V
30 Pade
r Reverse diffusion step
M- Classifier
Aa- Adv. attuck $4.3
Gradient
&ade (0)
Budu (t')
Binde (t2)
Beede (1*)
Rudo (9)
PPado (1))
Pusto
Executed in forward propagation.
Executed in backpropagation only.
Figure I: DiffGrad. æ is given to the iterative attack algorithm As to generate wadv. At each iteration, Bade is propagated through DBP. yielding (0) that is given to M while storing each intermediate (.).(() (bottom replicas) from DBP's reverse pass (see $2) but without saving any graph dependencies. Backpropagation uses the stored samples: Starting from = (dl < 0-see $2), each (ta,(t) is used to recompute (f- retrieving the required dependencies. Then, we recursively obtain the gradient w.r.t. a (f) from the gradients w.r.t. (t+ using this recovered sub-graph (see $3.2).
Finally, gradients are backpropagated in standard manner from "t,.. (t*) to Aa to update Tade.
In $3.2 and $3.3, revisit DBP's previous robustness. attributing it to backpropagation issues and improper evaluation protocols. Most works [20, 30, 26, 40] judge attacks based on a single purification of the final adversarial input. While this seemingly mirrors DBP's intended "purify once and classify" deployment [30, 40], is statistically invalid: since DBP samples noise randomly, evaluation of the AE fails to capture true single misclassification probability across possible purifications. Moreover, due to DBP's memory-intensive gradients, prior works implement exact backpropagation [20, 26, 41] via checkpointing [8]. We discover and fix issues in all previous implementations, introducing DiffGrad the first reliable module for exact backpropagation through DBP (see Fig. 1). In $4, we show that even under the one- one-sample evaluation protocol, AutoArtack [10] significantly outperforms prior works, exposing their gradient fallacies. We further use sample evaluation that captures DBP's stochasticity and resubmission risk-aligned with DiffHammer [41], which also identified the evaluation gap, but retained problematic gradients. By fixing both, we prove DBP even more vulnerable than they report, with robustness dropping below 17. 19.. Finally, we show prior attack enhancements tailored to DBP (e.g., DiftHammer, DiffAttack [20]), being oblivious to the ability of standard gradient-bart based attacks to reshape behavior, in fact, disrupt optimization, lowering attack performance, and confirming our theoretical scrutiny.
To replace the current inherently vulnerable single-evaluation DBP defense scheme, we propose more robust majority-vote (MV) setup, where multiple purified copies are classified and the final decision is based on the majority label. Yet, even under MV, DBP retains only partial robustness against norm-bounden attacks- see $4.2, supporting our theoretical findings. As we established the inefficacy of intermediate projections, this resistance due to DBP's vast stochasticity: common AEs that introduce large changes to individual pixels are diluted and may not impact most paths. Instead, we require stealthy modifications that affect many pixels. $5, we propose a novel adaptation of recent strategy [21] from image watermarking that crafts low-frequency perturbations, accounting for links between neighboring pixels. This technique defeats DBP even under MV
Contributions. (i) Analytically scrutinizing adaptive attacks on DBP, proving they nullify its theoretical robustness claims. (ii) Addressing protocol and gradient issues in prior attacks, enabling reliable evaluations, and demonstrating degraded performance of DBP and the ineffectiveness of recent attack enhancement strategies [20, 41]. (iii) Introducing and evaluating a statistically grounded MV defense protocol for DBP. (iv) Proposing and adapting low-frequency (LF) attack optimizations to DBP, achieving unprecedented success even under MV. (v) Availability: aside from scalability and backpropagation issues, existing DBP implementations and attacks lack generalizability. We provide DiffBreak' the first toolkit for evaluating any classifier with DBP under various optimization methods, including our novel I.F, using our reliable DiffGrad module for backpropagation. (vi) Extensive evaluations on ImageNet and CIFAR- R-AR-10 prove our methods defeat DBP, bringing its robustness to ~ 0%, outperforming previous works by large margin.

---

2 Background & Related Work
Adversarial Attacks. Given æe TRd with true label y, classifier M, and preprocessing defense G (G=Id if no defense), attackers aim to generate a similar Fado s.t. M (odu)) ) *y. Formally:
T dvd arg min
C v) D( w)EC
for loss eg. Typically, (x, v) (())) y). where l is a loss over M's output that captures the desired outcome. For instance, ((M((())) y) can be chosen the probability that the classifier's output label is y, which we strive to minimize. 'D is distance metric that ensures similarity if kept below some c These untargeted attacks are the focus of many works [30, 40, 32. 29]. The expected value accounts for potential stochasticity in G (c.g.. DBP below).
Diffusion models (DMs) [34, 36] learn to model distribution p on RR by reversing the process that diffuses inputs into noise. DMs involve two stochastic processes. The forward pass converts samples into pure Gaussians, and is governed by the following SDE for infinitesimal step dt 0:
dx f(x, t)dt gf)
eq. (I) describes stochastic integral whose solution up to /* *= *10,, 1] gives x=(*). Here, f: Rd F is the drift. w: R2 is a Wiener process, and g: R focus on VP-SDE, which is the most common DM for DBP [30, 48, 40J. Yet, our insights generalize to all DMs (see [36] for review). In VP-SDE, f(x, t) - [(()))((() and g(t) = BBt)), where 3(t) is noise scheduler outputting small positive constants. These choices yield the solution:
R is the diffusion coefficient. We
x(t*) va(t* + +1 a(t)
(2)
for C~N(O, Ia) and - Io ((a) da With proper parameters, we have x(1) (1)>~N(O Ia). Thus, process that inverts eq. (1) from 1 to allows generating samples in p from random noise. Due to Anderson [1], the reverse pass is known to be a stochastic process with:
dē [ ((), () g²(V log ( + (())dw.
Defining (tt*) := x(t*), the process evolves from t* 0 with a negative time step dt and reverse-time Wiener process (t). Let p(x) be the probability of æ under p. and (ār()) the conditional density at & given a=0) æ. Then, the marginal density S ((( (ml=)) dar, where Po = p. Solving eq. (3) requires the score Vact) log Pe (o(()), which can be approximated via trained model 89 s.t. sg((((), t)
given by pe(
V Va(o) log pe pe((()))) [36], yielding:
dæ - (() + 28 (+()) + /B(l)dw.
(4)
As no closed-form solution exists, the process runs iteratively over discrete negative steps dt. Starting from 1*, dē is calculated at each i= 1 until -=0. This continuous-lime DM describes stochastic integral (despite the discretized implementations). An alternative, Denoising diffusion probabilistic modeling (DDPM) [19, 34), considers discrete-time DM. The two are equivalent (see Appendix B).
DBP [30, 48, 40, 51] performs purification by diffusing each input until optimal time t* that preserves class semantics while still diminishing malicious interruptions. xxt*) ) is then given to the reverse pass, reconstructing clean (0) s.t. a(0) ~ p for correct classification.
Certified vs. Empirical Robustness. While recent work has explored certified guarantees for DBP via randomized smoothing (RS) [48, 5, 55, 6, 45), these guarantees hold only under restrictive assumptions - e.g., small & perturbations and thousands of Monte Carlo samples per input. In contrast, empirical defenses [40, 30, 31, 53] aim for practical robustness under realistic threats [10] and efficient inference. Like prior work [41, 20, 30, 40, 24], we explicitly target these empirical defenses not orthogonal certified variants whose guarantees do not hold under stronger practical perturbations or operational constraints. Certification protocols are computationally prohibitive, and their bounds fail capture threats that remain imperceptible but exceed certified radii.
3 Revisiting Diffusion-Based Purification
We study gradient-based attacks. $3.1, we theoretically prove that adaptive attacks invalidate DBP's principles. Then, reconcile this with previous findings of DBP's robustness, attributing them to
3

---

backpropagation issues and improper evaluation protocols. In $3.2, we analyze backpropagation mismatches in previous works and propose fixes. Finally, we present an improved evaluation protocol for DBP in $3.3 to better measure its robustness.
3.1 Why DBP Fails: Theoretical Vulnerability to Adaptive Attacks
DBP's robustness is primarily attributed its ability project the purified e()) onto the natural man- ifold [30, 48, 20J. This foundational assumption- rooitodd-roteeddeeee in the inherent behavior of the purification process-has remained unchallenged. In fact, the most advanced attacks have been explicitly tailored to exploit or circumvent it: tiffAttack [20] introduces suboptimal per-step losses to perturb the purification trajectory (see $4), while DiffHammer [41] constrains its optimization to feasible paths that evade detection by the DBP process itself. DBP is often justified through its marginal consistency: since 80² log Pt. {a+(t))sa0 se and ((()))e().1) follow the same marginals [1]. yielding e()) ~ p.
Specifically, Xiao et al. [48] show that:
a((+))a(0)-a'
(r()((0)zz) x p()(0) E 2(1 )(+*))
where p the density of the natural data distribution and (tt*) is the variance schedule time t* Thus, DBP is expected to reject adversarial examples by construction: the probability of producing any (0) that is both adversarial and lies off-manifold is exponentially suppressed by the score model. However, this reasoning assumes the reverse process remains faithful to p. In practice, the score function $y is itself an ML model differentiable and susceptible adversarial manipulation. Our key insight is that traditional gradient-based -bsed attacks, when backpropagated through DBP, do not simply bypass the purifier- they implicitly target it, steering the score model to generate samples from an adversarial distribution. This challenges the assumptions underlying prior attack strategies, many of which distort gradients or constrain optimization ways that ignore the score model's vulnerability, undermining their own effectiveness. Below, we formally characterize this vulnerability.
Definitions. Diffusion Process Score Model. For t 2 t2, let tt sta represent the joint reverse Let 8g denote the score model used by DBP to approximate the gradients of the natural data distribution at different time steps. 8gt is the abstract score model invoked at time step with parameters H', corresponding to ( t). Given that the score model 8a is an ML model that interacts with adversarial input æ, we denote the parameters at each reverse step as 6' capturing their dependence on æ. That is, 0% = (z)) which makes explicit that the purification process is not immutable, as adversarial modifications to x can shape its behavior.
diffusion trajectory ((41), (t1 + di),..., .(2)). ...,,,,(((,)))))
Classifier. For a classifier M and label y, let (M) denote the probability that u belongs to class y.
Adaptive Attack. A gradient- algorithm that iteratively updates x with learning rate 1) >> 0:
1 N Σ (=()))
where V
Here, Va is the empirical gradient estimate over N purified samples z(0).. obtained via Monte Carlo approximation and backpropagated through DBP's stochastic process.
Theorem 3.1. The adaptive attack optimizes the entire reverse diffusion process, modifying the parameters (0's hesr such that the output distribution x(0)= DBP(()) (x), where (BP((2) is the DBP pipeline with the score model's 's weights (0t ) adversarially aligned. That is, the adversary implicitly controls the purification path and optimizes the weights 1 to maximize:
max E a(00) ( (0
[Pr(- [Pr(-y)(((0]]-
Since (0%) tec depend on the purification trajectory T.000- optimizing a under the adaptive attack directly shapes the purification process itself, forcing DBP to generate adversarially structured samples that maximize misclassification.
Impact. As DBP is widely perceived as immutable, its robustness is often attributed to gradient masking (vanishing or explosion) due to its presumed ability to project inputs onto the natural manifold. However, this reasoning treats DBP as a static pre-processing step rather than dynamic optimization target. Our result overturns this assumption: rather than resisting adversarial influence, DBP itself becomes active participant in adversarial generation. Crucially, standard

---

adaptive attacks that backpropagate accurate gradients through DBP do not merely evade it they implicitly exploit and reshape its behavior, forcing the purification process to align with the attacker's 's objective. The introduced perturbations directly shape the purification trajectory, causing DBP to generate, rather than suppress, adversarial samples. Thus, DBP does not inherently neutralize ad adversarial perturbations but instead shifts the optimization target from the classifier to the score model 89. leaving this paradigm highly vulnerable.
Proof. (Sketch) The adaptive attack maximizes the probability that the classifier mislabels the purified output, which is expressed as an expectation over the stochastic purification process. Applying the law of the unconscious statistician (LOTUS), we reformulate this expectation over the joint distribution of the reverse diffusion trajectory, shifting the dependency away from the classifier's 's decision at the final step. Leveraging smoothness, interchange differentiation and expectation, revealing that the adaptive attack's s gradient corresponds to the expected gradient of the purification trajectory, which is governed by the score model's parameters. Crucially, these parameters are optimized implicitly through perturbations to æ, rather than being explicitly modified. This demonstrates that the adaptive attack does not merely navigate DBP-it fundamentally shapes its behavior, exploiting the score model rather than directly targeting the classifier. The full proof is in Appendix C.
3.2 Precise DBP Gradients with DiffGrad
Building our theoretical analysis, we examine why prior adaptive attacks [30, 40, 20, 26, 24] underperform and show how DiffGrad, provided by our DiffBreak toolkit, resolves these limitations.
We reserve the terms forward/reverse pass for DBP and forward/hackpropagatiio for gradient computations. Let db(*)= := a (1*). The reverse process follows:
a( dt) (tt) + dæ(t), dt < O,
(5)
and by chain rule the gradient of any function F of @(t + dt) w.r.t. lc() is:
V2(1) F ( + di), V Va(++a)FF)
(6)
Applying this recursively yields gradients w.r.t. ((t)), and via eq. (2), the input æ. Yet, standard automatic differentiation is impractical for DBP due to excessive memory overhead from depen- dencies between all @()). Prior work resorts to approximations such as the adjoint method [25], or checkpointing [8] to compute the exact gradients. However, approximations yield suboptimal attack performance [24]. On the other hand, we identify critical issues in existing checkpointing-oee baased implementations that previously led to inflated estimates of DBP's robustness (see Appendix D for detailed analysis, DiffGrad's pseudo-code, and empirical evaluations of backpropagation issues):
1) High- Variance Gradients: Gradient-basee attacks require estimating expected gradients using N Monte Carlo (EOT) samples. When N is small, gradient variance is high, leading to unreliable updates. Prior works used small N due to limitations in how EOT samples were purified-one at time via serial loops in standard attack benchmarks (e.g., AutoAttack [10]. DiffGrad parallelizes EOT purifications. This integrates seamlessly with standard attacks. Coupled with our termination upon success protocols see $4.2, this enables us to use up to 128 EOT samples (for CIFAR-10) with specdup of up to 41 upon early termination, drastically reducing variance.
2) Time Consistency: torchsde2 is the de facto standard library for SDE solvers. Hence, several prior checkpointing approaches likely use torchsde as their backend. Yet, torchsde internally converts the integration interval into a PyTorch tensor, which causes a discrepancy in the time steps on which the score model is invoked during both propagation phases due to rounding issues if the checkpointing module is oblivious to this detail. DiffGrad ensures time steps match in both phases.
3) Reproducing Stochasticity: Forward and backward propagation should reuse identical randomness to preserve fidelity. In vanilla DBP, diffusion noise cancels in the gradient, so not preserving stochastic components has little effect. Yet, for guided schemes whose computations depend on randomness (some current ones and potential future variants), failing to reproduce the noise realizations can bias gradients and derail optimization. We thus introduce structured Noise Sampler NS that records all noise during the forward pass and reuses in backpropagation. We also enable deterministic CuDNN kernel selection to avoid backend nondeterminism see Appendix D. for details.
5

---

4) Guidance Gradients: Guided DBP [40] uses guidance metrics along the reverse pass. This guid- ance obtained by applying a guidance function 9 (potentially) both the original input z and the reverse-diffused (()). As such, it creates paths from I to the loss that basic checkpointing fails to consider. Furthermore, the guidance itself is often in the form of gradient, necessitating second-order differentiation during backpropagation, which is, by default, disabled by differentiation engines. We extend DifIGrad to include these "guidance" gradients. Critically, recent SOTA DBP defenses generate purified outputs from pure noise, relying æ only via guidance (see $6). As such, the lack of support for guidance gradients in all previous implementations leads to a false sense of security. To our knowledge, DiftGrad is the first to account for this component.
5) Proper Use of The Surrogate Method [24]: Upon inspecting DiffHammer's code, we find their checkpointing implementation effectively applies Lee and Kim [24]'s s surrogate approximation to calculate the gradients, which slightly degrades gradient quality with no practical performance gains-see Appendix F. Yet, the real issue lies in DiffHammer's s implementation of the surrogate, which diverges from [24], further corrupting the gradients. We defer details to Appendix 1.1.5.
3.3 On The Issues in DBP's Evaluation Protocols.
DBP is typically deployed in a single-purification (SP) setting: an input a is purified once via a stochastic reverse process and classified as c = MM)(()) [30, 40J. This assumes (0)) lies on the natural manifold, yielding correct label with high probability- see $3.1. 1. Yet, prior studies [20, 30, 40] fail to assess this very property: they evaluate robustness by purifying only the final adversarial iterate, testing just a single stochastic path, yielding noisy, unrepresentative estimates.
DammHr [41] addressed this with worst-case robustness Roor: the attack succeeds if any of N purifications (evaluated at each attack step) leads to misclassification. The metric is Wor.Rob == 1 - 1 s Σi= max A("), where A!") = 1 if the i-th purification of sample j fails, and NE
S is the dataset size. They justiffy this protocol by modeling resubmission attacks (e.g., in login attempts), thereby limiting N to (typically) N = 10 after which the attacker will be blocked. However, attackers can retry inputs arbitrarily in stateless settings like spam, CSAM, and phishing. Even in stateful systems, the defender has no control over the stochastic path. Thus, SP evaluations require many purifications per input for statistically meaningful conclusions.
Yet, any stochastic defense fails given enough queries [27]. Even if DBP nearly always projects to the manifold, SP is only reliable if misclassification probability is negligible over all paths-rare rare in practice ($4). Robust predictions must hence aggregate over multiple purifications. We thus propose:
Majority Vote (MV). Given input æ, generate K purifications and predict by majority: c MV (xr == Mode(M(@( (O)... M(a(K) (0))). Unlike certified smoothing [48], MV requires no excessive sampling, yet offers stable, variance-tolerant robustness estimate, albeit without formal
guarantees. We use the majority metric: MV.Rob - - 1 [MV K
(x.y) y/}.
where yj is the ground-truth label. DitTHammer's additional Avg. : 1 1
NS Σソ= Σ A
averages path-level failures across all samples, conflating individual copy failures with sample robustness. A few fragile samples can dominate this metric, making it unreliable for deployment.
4 Experiments
We reevaluate DBP, demonstrating its degraded performance when evaluation and backpropagation issucs are addressed, cementing our theoretical findings from $3.1.
Setup. We evaluate on CIFAR-10 [22] and ImageNet [11] similar to previous work [41, 20, 30, 40]. We consider two foundational DBP defenses: The VP-SDE DBP (DiffPuree [30] and the Guided- DDPM (see $3.2), GDMP [40]. We use the DMs [12, 19, 36) studied in the original works, adopting the same purification settings See Appendix E. Following Nie et al. [30], we use WideResNet-258 10 and WideReesNet-70-- 16 [52] for CIFAR-10, and WideResNet-50-- and DeiT-S [14] for ImageNet. As in previous work [40, 30, 20, 41, 26, 24), we focus on the white-box setting and use AutoAttack-ls: (AA-(a) [10] with exc 8/255 for CIFAR-10 and Ez = 4/255 for ImageNet. Existing works focus on norm-bounded (f.g and f2). For DBP, & has repeatedly proven superior [20, 24, 26), making it the
focus of our evaluations. We report clean accuracy (Cl-Acc) without attacks- and robust accuracy

---

(Rob-Ace) the fraction of correctly classified attack samples. We use 256 random test samples per dataset, consistent with prior DBP work [41, 71. Because DBP is a computee-ntenssvve defense. we prioritize breadth over scale-two two datasets (CIFAR-10, ImageNet), multiple classifiers and DBP schemes, diverse attacks and baselines, and MV ablations so readers see the same qualitative conclusions across settings (see $4- $6; extended results in Appendix F-Appendix H).
4.1 Reassessing One-Shot DBP Robustness with Accurate Gradients
As noted in $3, DBP's robustness stems from two factors: inaccurate gradients and improper evaluation. As our work offers enhancements on both fronts, we evaluate each factor separately, Here, we isolate the gradient issue by re- re-running prior experiments under the same (problematic) single-evaluation protocol, but with accurate gradients via our DiffGrad module and compare the results to those from the literature. Despite using only 10 optimization steps (vs. up to 100 in prior work), our method significantly outperforms all gradient approximations and even recent full-gradient methods. For DiffPure, we lower robust accuracy from 62. 12%1 (reported by Liu et al. [26]) to 48.05%, and for GDMP, from 24.53% (Surrogate (24]) to 19.53%. These results expose the issues in existing gradient implementations and invalidate claimed gains from enhancement strategies like DiffAttack. Full breakdowns and additional baselines are provided in Appendix F.
Evaluations Under Liu et al. [26]'s Protocol. Liu et al. [26], like DiffHammer [41] and our own analysis, note that evaluating a single purification attack termination inflates robustness scores. To address this, they propose a refined I-evaluation protocol, which tests 20 purifications of the final AE and declares success if any fail. This offers a stricter assessment than earlier one-shot methods, though still weaker than Ror.Rob (see $3.3). Accordingly, Liu et al. [26] group their method with one-shot evaluations. Repeating their setup using DiffGrad (20 attack iterations, N observe dramatic drop in robust accuracy: on WideResNet-28-100 10, we improve upon Liu et al. [26]'s results by 25.39% and 30.42% for DiffPure and GDMP, respectively- bringing the Rob-Accs down to 30.86% and 10.55%. These results confirm the superiority of DiffGrad's gradients and expose DBP's realistic vulnerability. Detailed results are Appendix G.
EOT), we
4.2 DBP Under Realistic Protocols
Here, we continue highlight the shortcomings of one-shot evaluation and the strength of DiffGrad, while exposing the invalidity of previous at- tack enhancements over the standard gradient-based methodologies. To do so, we adopt a Rob protocol, similar to DiflHammer--se see $3.3 and also include our majority-vote variant (MV.Rob). We reimplement DiffHammer and DiffAttack using DiffGrad, and compare them their original versions from [41] and our standard AA-Ea- All evaluations use
Table 1: AA-Ea, performance comparison on CIFAR-I0 (cm=8/2255) under realistic threat models. Metries include Wob and MV.Rob under a 10-evaluation protocol. indi- cates strategy is PGD.
Per:
Gradien Mothed
Cl-Acc Rob-Ac
Cl-Ace Rob-Au
MV
BPDA
32.81
72:27
AiifAtiack HSiiHammeer [F]]
33.79
NA
89,06
15.63
91,00
39.45
DijPare
DeMHammer (DifHommer (4T]
22.
Defflammer DiffGrad
10.16
38.25
Full (Duilllamezr [+]]]]
36.91
NA
Full-DilIGrad
17.19
39.45
ARlAAiackk (ISFHanmmer (PT))
$7.7
91.00
3.9]
92.19
14.45
GOMP (40)
Deffllamerr (Diffilammer (41))
27.54
NA
Diffliammer DiffGrod
7.81
25.39
CIFAR- with WideResNNet-0-11 We also report ImageNet results be- low. Full denotes the standard AA attack that uses the full exact gradients (i.e., via checkpointing).
31.05
Full-DifGrrad
16.8
Choice of batch size (N): ): We set N 10 for CIFAR-10 to match [41] (which uses N 10 for both EOT and evaluation copies) and to balance oobustnsss/gradien accuracy with runtime and memory; these values reflect practical deployment and evaluation constraints. For ImageNet, we choose N 8 for similar reasons. Ablations with larger N, latency measurements, and practical recommendations appear in Appendix H; complexity details are in Appendix 1.1.3 and Appendix D.3.
Results. Table reports Wor.Rob and RV.Rob scores. The attacks run for 100 iterations, terminating upon success. For all attacks Full, DiffAttack, and -aaaaamm-Dffrrraa yields significantly lower Wor. Rob compared to [41], confirming the gradient mismatches discussed in $3.2 (MV.Rob
is unique to our work). This establishes the need for our reliable DiffGrad an essential tool for
7

---

future progress in the field, given the repeated problems that continue to surface in implementations of the checkpointing method. With correct gradients (i.e., DiffGrrdd), all three attacks yield Wor.Rob < 20%; for GDMP, < 10%, exposing the failure of the single-purificttoon defense and reinforcing the claim that a statistically resistant alternative (e.g., our MV.Rob) must be used with DBP. Similarly, this highlights the issues in the attack evaluation protocols that consider single sample (i.e., 1-evaluation) as they drastically inflate robustness estimates.
Notably, minor Wor. differences among attacks in this range where predictions are highly noisy reflect random variation, not real gains. Hence, we must focus on the MV.Rob when comparing the three: For the same N, our MV is strictly harder as it takes the majority label meaning success requires misclassifying > [ of the N purified copies, whereas Wor. counts single error. Thus, MV.Rob is an upper bound on Wor.ob,, which aligns with the empirical findings attained with DiffGrad, showing significant robustness gains. Violations arise only when our MV.Rob results are compared to the Wor.Rob values reported by DiffHammer [41] and reflect implementation differences in gradient computation. As DiffGrad removes backpropagation mismatches that can reduce gradient fidelity and inflate robustness (both Rob and MV.Rob), MV-versus-worst-caae comparisons should be made using a common, DiffGrad implementation of each attack.
Throughout $4, we have thus far shown that DBP's reported robustness has been overstated due to gradient inaccuracies. Yet, some recent works introduce DBP-specific attack augmentations to improve success rates, but these overlook our theoretical finding that accurate gradients alone can reshape DBP's output cecrbbution-enndeere builds on the assumption that DBP's trajectories split into "vulnerable" and "non- nhon-vuulnrrable groups, with misleading gradients from the latter, but our analysis (see Theorem 3.1) shows that correct differentiation allows the attack to steer DBPPs stochastic process toward adversarial outcomes, contradicting that fixed partition. Likewise, DiffAttack's ' per-step losses artificially strive to alter the output distribution, ignoring the standard pradient-based attack's inherent ability to do so, which could lead to suboptimal optimization updates. Below, we show that both these methods offer no advantages over the standard (full-gradient) attack and can, in fact, be counter-productive.
such enhancements unnecessary. DiffHammer
Under MV.oob, MiIAAttck-DiffGrrat and Full-DiffGrad are identical on DiffPure, and DiffHammer- Diffrrr leads to MV. lower by a mere 1.17%. which amounts to only 3 samples out of 256 and is thus statistically insignificant. On GDMP, hifHHammer-DifGraal performs significantly worse than both others. Hence, DiffHammer worsens attack performance in accordance with our theory. DiffAttack-DffGrr a 2.35% edge, which despite the questionable statistical significance given the test set size, could indicate a potential advantage. Yet, our ImageNet comparisons below refute this hypothesis.
matches Full-DiffGradd on DiffPure: on GDMP, shows
ImageNet. We evaluate AA-Ea on ImageNet us- ing WedeResNet-50-22 and DeiT-S classifiers under 6x 4x:4/25 (100 iterations), following standard practice. For DeiT-S, we also reim-
Table AA-Laa comparison on ImageNet (co 4/255).
Wor % Rob-Ace
Rob
Models
Par.
Gradient Method
Cl-Ace
Cl-Axc
Rob-Acc
DifPPwe (30)
Full-DiffGrad
74.22
12.11
77.02
29,69
DeiT.S
DiffPwee (30)
Full-DiffGrad
DiffAttack- DiffGrad
73.63
21.99
25
77.34
32.81
42.21
Fill-DilGraad
69.14
20.70
7500
32.33
plement DiffAttack via DiffGrad for comparison. We use 16 EOT samples (two batches of N 8)
and 8 samples for prediction (Wor.Rob/MV.Rol
As with CIFAR-IO. 10, robustness drops sharply: Wor.Rob ranges from just 12.11 2..1% % to 21. 21.09% while MV.Rob peaks at 32.8%% This confirms the vul- nerability of single-purinn puriffiaaiioiiio and the strength of gradient-based attacks. DiffAttack underperforms our standard attack by 9.4% MV.Rob on DeiT-S, reinforcing its inferiority.
5 Defeating Increased Stochasticity
DBP's stochasticity boosts its robustness under MV (see $4.2). Typical adversarial strategies incur nigh-frequency changes as they directly operate on pixels, altering each significantly w.r.t. its neighbors. This leads to visual inconsistencies, limiting the distortion budget. Such modifications are also easily masked by DBP's noise. Instead, systemic, low wwwffrequency (LF) changes allow larger perturbations and resist randomness.
Our I. method is inspired by a recent attack -UnMaker [21]-on on image watermarking that employs novel units termed Optimizable Filters (OFs). In signal processing, filter is smoothing operation
8

---

defined by a kernel KCE BA (with values that sum to 1), with which the input is convolved. The output each pixel is a weighted average of all pixels in its MxN vicinity, depending on the weights assigned by K. Hence, filters incur systemic changes. Yet, they apply the same operation universally, unable to produce stealthy AEs, as the changes required to alter the label will be uniformly destructive. OF's allow each pixel (i to have its own kernel KC to customize the filtering at each point. K'* is the set of all per-pixel KC s. The weights 0x* are learned via feedback from a perceptual metric (lpips) [56], leading to an assignment that ensures similarity while maximizing the destruction at visually non-critical regions to optimize a specific objective. Note that the lpips constraint replaces the traditional norm constraint. To guarantee similarity, they also impose geometric constraints via color kernels de, similar to guided filters (details in Appendix I.I).
We subject a to a chain J. [[=
OF am of OFs similar to UnMarker, replacing the objective pertaining to watermark removal in the filters' weights' learning process with the loss over M. Each OF has a kernel set K,,"" (with wights 0xc and shape MyxNs), and oe We optimize:
f (a + 8), y)
Tade
argmin 10xq 1.6
+c me-mmalllpppsta A(x + 6)) Tp. 0}
OF
(M denotes any loss as defined in $2 8 is a modifier that directly optimizes æ, similar to traditional attacks. AES are generated by manipulating x via ō and propagating the result through the filters. While direct modifications alone do not cause systemic changes, with OFs, they are smoothed over neighbors of the receiving pixels. 8 allows disruptions beyond interpolations. Similar to UnMarker. we chain several OFs with different shapes to explore various interpolations. Optimization is iterative (code in Appendix 1.3). max (Ipips(x, [] (x (ll++))) 6)) Ty. 0) enforces similarity: If the distance exceeds
Tp. the lpips gradients lower it in the next iteration. Otherwise, it returns O, minimizing e.M unconditionally. This gives a solution within the Tp constraint (violating outputs are discarded), yielding optimal (6x4): 8 s.t. Tadu= u oF
(x+ where [J are the filters with (8xq))
of
5.1 DBP Against Low-Frequency AEs
Our final question is: Can DBP be de-
Table 3: Performance of LF attack under MV.
graded further under MV? Based on $5, this is possible with LF, which we test
Pur.
Dataset
Models
Cl-Ace 5
Rob-Ace
in this section. For LF (see $5), we use
ResNet--U
72.54
0.00
ImageNNtt
WideResNet--n-.
77:02
(1.00
VGG-LPIPS [56] as the distance metric with T, = 0.05, ensuring imperceptibil- itt [17,1]] Remaining parameters are sim- ilar to Un.arkkr'ss (see Appendix 1.2). We use 128 EOT samples for CIFAR-10 (same for label predictions since increasing the sample set size leads to enhanced robustness of DBP- see Appendix H). For ImageNet, the numbers are identical to $4.2 (note that the dimensionality of ImageNet makes larger sample sets prohibitive and our selected set sizes reflect realistic deployments the largest number of samples that can fit into a modern GPU simultaneously).
DofiPure [30]
DeiT-S
77.34
0.00
CIFAR-10
Nedt
Wedt
92.19
3.13
ResNet-S0
73.05
0.39
ImogenNet
WideResNei-S00
50-2
71.88
0.00
GDMP (40)
75
CIFAR-10
WiddResNett00..
Widekes Ne5-38-1
92.19
0.39
Choice of Perceptual Constraint: We use lpips with LF because it is widely adopted, and prior work [21, 17] provides known thresholds we can reuse for adversarial optimization. For AEs, calibrated thresholds are essential; otherwise, overly permissive constraint can label visibly altered adversarial images as successful, which defeats the point and inflates attack success (i.e., metrics (c.g..
underestimates robustness). Both independently calibrating other non -o----ooomonnene via user studies) and re-running robustness evaluations are time-consuming and require additional resources. Thus, we limit our scope to lpips but encourage future work reproduce our experiments with alternative perceptual metrics for which calibrated thresholds may exist or become available.
Results. The LF results (using Full-DiffGrad for backpropagation) a 55 classifier for ImageNet. Not only does it defeat all classifiers completely, leaving the strongest with Rob-Acc of 3. 13%, but it also does so in the MV setting, where previous attacks fail.
in Table 3. We also include
Concluding Remarks. Our findings highlight a limitation in current robustness evaluations, which focus heavily on norm-bounder attacks while overlooking powerful alternatives like low- ooeffrque

---

(LF) perturbations. Though not norm-bounded, LF is percepiually constraine--simee to StAdv [47], a longstanding benchmark against DBP Nie et al. [30]. LF's imperceptibility is guaranteed due to using a perceptual threshold Tp = 0.05 that has previously been proven to guarantee stealthiness and quality [17, 21J. We include qualitative results in Appendix J confirming this. One may question existing attacks like StAdy can also defeat DBP under MV, rendering LF incremental. We address this in Appendix J. showing StAdy fails to produce stealthy AEs in this setting. While other techniques may also be adaptable to attacking certain DBP variants in the future, LF enjoys a solid theoretical foundation such alternatives may lack, limiting their generalizability to all DBP defenses.
Future Extensions. While we focus on vision models similar to earlier DBP work, both Theorem 3.1 and DiffGrad are not structure. We thus expect the same vulnerabilities for, e.g., speech or video purification. The arguments in support of our L.F attack's efficacy due to its low-frequency nature also hold across various domains. Yet, the specific implementation that relies on spectral filter networks would require domain- based adaptations to apply it to different tasks (e.g. replacing 2D Fourier filters with their /D counterparts for audio). We therefore urge future work to extend our methods to other modalities.
they depend only on the diffusion dynamics and implementation.
6 Potential Countermeasures
Adversarial Training (AT). While AT remains leading defense, it performs poorly on unseen threats. As expected, it fails against our LF. On adversarially trained 22 for CIFAR-10 (DiffPure), LF reduces robust accuracy to just 0.78%, confirming this limitation.
SOTA DBP: MimicDiffusion, One might ask whether recent variants offer improved robustness. Most build incrementally on the foundational defenses in $4, meaning our results broadly generalize. One notable exception is MinicDiffusion [35], which generates outputs entirely from noise, using the input only as guidance (see $3.2). Its goal is to preserve semantic content for classification while eliminating adversarial perturbations, and it reports SOTA robustness to adaptive attacks.
theoretical analysis, we hypothesize this robustness is illusory-an an artifact of improper evaluation and broken gradient flow. Unlike GDMP, which involves both direct and guidance paths from æ to the loss, MimicDiffusion relies solely on guidance. Since guidance gradients require second-order derivatives (disabled by default), the original paper fails to compute meaningful gradients, severely overestimating robustness. We correct this using DiftGrad, the first method to properly differentiate through guidance paths. On CIFAR-IO. 10, using AA-Ex (we exclude LF for time limitations) with N 128 for both EOT and evaluation, we find that the original Rob- Ace scores of 92.67% (WRN-28-18 10) and 22. (WRN-70-16) collapse under proper evaluation: Wor.Rob drops to 2.73% and 4.3%, respectively. Even under stricter MV. MV.Rob, MimicDiflusion performs worse than the standard DBP defenses from $4, with accuracies of just 25. 75. and 27. 27%... This confirms that MimicDiifission's s robustness is not real, and collapses under proper gradients.
However, based
Outlook. Our findings highlight the need for more robust defenses. DBP remains promising as it improves clean accuracy compared prior purification methods. Yet, Theorem 3.1 shows it is vulnerable to adaptive attacks when accurate gradients are available. We believe future work should explore DBP variants that relax the assumptions of Theorem 3.1. One possible direction is to modify the purification process at inference so that the model follows private stochastic dynamics that are not directly exposed to adversaries. Such approaches, if carefully designed to avoid gradient masking [2], could reduce the transfcrability of uttacker-optimet gradients and increase robustness.
7 Conclusion
We scrutinized DBP's theoretical foundations, overturning its core assumptions. Our analysis of prior findings revealed their reliance on inaccurate gradients, which we corrected to enable reliable evaluations, exposing degraded performance under adaptive attacks. Finally, we evaluated DBP in stricter setup, wherein we found its increased stochasticity leaves it partially immune to norm- bounded AEs. Yet, our novel low-frequency approach defeats this defense in both settings. We find current DBP is not a viable response to AEs, highlighting the need for improvements.
Acknowledgements. This work was supported by NSERC Discovery Grant RGPIN-2020-0477
10

---

References
[1] Brian DO Anderson. Reverse-time diffuuson equation models. Stochastic Processes and their Applications. 1982.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning (ICML). 2018.
[3] Jacob Buckman, Aurko Roy, Colin Raffel, and J. Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International Conference on Learning Representations (ICLR), 2018.
[4] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEFE Symposium on Security and Privacy (SP), 2017.
[5] Nicholas Carlini, Florian Tramer, Krishnamurthy Dj Dvijotham, Leslic Rice, Mingjie Sun, and J Zico Kolter. (Certified!!) adversarial robustness for free! In International Conference on Learning Representations (ICLR), 2023.
[6] Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, and Jun Zhu. Your diffusion model is secretly certifiably robust classifier. In International Conference Machine Learning (ICML), 2024.
[7] Huanran Chen, Yinpeng Dong. Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu. Robust classification via single diffusion model. In International Conference on Machine Learning (ICML.), 2024.
[8] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint
04,74,1000 2016.
[9] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), 2019.
[10] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference on Machine Learning (ICML), 2020.
[11] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hicrarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2009.
[12] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[13] Guncet S. Dhillon, Kamyar Azizzadenesheli, Zachary C. Lipton, Jeremy Bernstcin, Jean Kossaifi, Aran Khanna, and Animashree Anandkumar. Stochastic activation pruning for robust adversarial defense. In International Conference on Learning Representations (ICLR), 2018.
[14] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations (ICLR), 2021.
[15] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (JCLR),2015
[16] Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations (ICLR), 2018.
[17] Qingying Hao, I Licheng Luo, Steve TK Jan, and Gang Wang. It's not what looks like: Manipulating perceptual hashing based applications. In ACM SIGSAC Conference on Computer and Communications Security (CCS), 2021.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[19] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
[20] Mintong Kang. Dawn Song, and Bo Li. DiffAttack: Evasion attacks against diffusion- adversarial purification. In Conference Neural Information Processing Systems (NeurIPS), 2024.
11

---

[21] Andre Kassis and Uns Hengartner. UnMarker: A Universal Attack on Defensive Image Watermarking, In IEEE Symposium on Security and Privacy (SP), 2025.
[22] Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009.
[23] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security. Chapman and Hall/CRC, 2018.
[24] Minjong Lee and Dongwoo Kim. Robust evaluation of ceffusion-based adversarial purification. In IEEE/CVF International Conference Computer Vision (ICCV), 2023.
[25] Xuechen Li, Ting- Bing-Kam Leonard Wong, Ricky TQ Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. In International Conference Artificial Intelligence and Statistics (AISTATS), 2020.
[26] Yiming Liu, Kezhao Liu, Yao Xiao, ZiYi Dong, Xiaogang Xu, Pengxu Wei, and Liang Lin. Towards understanding the robustness of diffusion ciffusioo-bbsed purification: A stochastic perspective. In The Thirteenth International Conference on Learning Representations, 2025. URL https: htttps://operreviex. net/forum? id=aA
[27] Keane Lucas, Matthew Jagielski, Florian Tramer, Lujo Bauer. and Nicholas Carlini, Randomness ML defenses helps persistent attackers and hinders evaluators. CoRR, 2023.
[28] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018
[29] Dongyu Meng and Hao Chen. MagNet: a two-pronged defense against adversarial examples. In ACM SIGSAC Conference on Computer and Communications Security (CCS), 2017.
[30] Weili Nic, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning (ICML), 2022.
[31] Yidong Ouyang, Liyan Xie, and Guang Cheng. Improving adversarial robustness through the contrastive- guided diffusion process. In International Conference Machine Learning (ICMI.), 2023.
[32] Tianyu Pang, Kun Xu, and Jun Zhu, Mixup inference: Better exploiting mixup defend adversarial attacks. In International Conference on Learning Representations (ICLR), 2019.
[33] Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop. 34th International Conference Machine Learning, 2017. URL http://arrke org/ hts/////s///// /1007, .00131.
[34] Jascha Sohl-Dickstein, Fric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning (ICML). 2015.
[35] Kaiyu Song, Hanjiang Lai, Yan Pan, and Jian Yin. MimicDiffusion: Purifying adversarial perturbation via mimicking clean diffusion model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[36] Yang Song. Jascha Sohl-Die Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
[37] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
[38] Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks adversarial example defenses. Conference on Neural Information Processing Systems (NeurIPS), 2020.
[39] Arash Vahdat, Karsten Kreis, and Jan Kautz, Score-based based generative modeling in latent space. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[40] Jinyi Wang, Zhaoyang Lyu, Dahua Lin, Bo Dai, and Hongfei Fu. Guided diffusion model for adversarial purification. arXiv preprint 2205 14969, 2022.
12

---

[41] Kaibo Wang, Xiaowen Fu. Yuxuan Han, and Yang Xiang. DiftHammer: Rethinking the robustness of diffusion-basse adversarial purification. In Conference Neural Information Processing Systems (NeurlPS), 2024.
[42] Zekai Wang. Tianyu Pang. Chao Du, Min Lin. Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning (ICML), 2023.
[43] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Bero P Simoncelli. Image quality assessment: From
error visibility to structural similarity. IEEE Transactions on Image Processing, 2004,
[44] Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations (ICLR), 2020.
[45] Quanlin Wu, Hang Ye, and Yuntian Gu. Guided diffusion model for adversarial purification from random noise. arXiv preprint XrXiv:2206.1086 10875, 2022.
[46] Chang Xiao, Peilin Zhong, and Changxi Zheng. Enhancing adversarial defense by k-winners-take-all. In International Conference on Learning Representations (ICLR), 2020.
[47] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In International Conference on Learning Representations (ICLR), 2018
[48] Chaowei Xiao, Zhongzhu Chen, Kun Jin, Jiongxiao Wang, Weili Nie, Mingyan Liu, Anima Anandkumar, Li, and Dawn Song. DensePure: Understanding diffusion models towards adversarial robustness. In Workshop on Trustworthy and Socially Responsible Machine Learning, NeurlPS, 2022.
[49] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations (ICLR), 2018.
[50] Yuzhe Yang, Guo Zhang, Dina Katabi, and 7hi Xu. ME-Net: Towards effective adversarial robustness with matrix estimation. In International Conference on Machine Learning (ICML.), 2019.
[51] Jongmin Yoon, Sung Hwang, and Juho Lee, Adversarial purification with score-based generative models. In International Conference on Machine Learning (ICML), 2021.
[52] Sergey Zagoruyko. Wide residual networks. arXiv preprint arXiv: 160tv:6005.00 1605.07146, 2016.
[53] Boya Zhang, Weijian Luo, and Zhihua Zhang. Purify++: Improving diffusion purificctioniiiii with advanced diffusion models and control of randomness. arXiv preprint arXiv:2310. 18762, 2023.
[54] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML), 2019.
[55] Jiawei Zhang, Zhongzhu Chen, Huan Zhang, Chaowei Xiao, and Bo Li. DiffSmooth: Certifiably robust learning via diffusion models and local smoothing. In USENIX Security Symposium (USENIX Security), 2023.
[56] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features a perceptual metric. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide a "Limitations" section in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

The assumptions underlying Theorem 3.1 are in Section 3.1 and the proof in Appendix C.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Reproducibility information is given in Section 4 and Appendices 1.2 and E.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Our code is available here: https://githin www. com/andrekassis//
DiffBreak. We use public datasets CIFAR-I0 10 and ImageNet.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c co/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips.. "nips.ccppppii cccppblic/guu wwwggguddeceeseii coiaiisssooopicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Experiment settings and details are given in Section 4 and Appendices I.2 and F.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Computing error bars would have been too computationally expensive given our limited computational resources and the extensive scope of our experiments, far exceeding typical settings in relevant prior work.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Information about compute resources given Section 4.2. Experiments are conducted 40GB NVIDIA A100 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:p/ccccpbbbcciiicccciiieeeee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use public datasets CIFAR-10 and ImageNet and give corresponding
citations.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

is If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code repository (tttg:://ggtaall.ccc/////aaaaaas well documented.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /ttpa/////eeu../eeeeee ceturipss.ccConfeeeeneeeee /c/Connferences//2255LL.. for what should or should be described.
20

---

A Broader Impact & Limitations
This work advances adversarial robustness by exposing a fundamental vulnerability in Diffusion- Based Purification (DBP). We show that adaptive attacks do not merely circumvent DBP but repurpose as an adversarial generator, invalidating its theoretical guarantees. While this insight highlights a critical security risk, it also provides foundation for designing more resilient purification strategies. To facilitate rigorous and reproducible evaluation, we introduce DiffBreak, an open-source toolkit providing the first reliable gradient module for DBP-based defenses, in addition wide array of attack implementations. including our low-frequency (LF) strategy that outperforms existing approaches, seamlessly applicable to a broad range of classifiers. Similar existing adversarial attack toolkits (e.g.. Foolbox [33], AutoArtack [10]), our framework is intended for research and defensive purposes. While this tool enhances robustness assessments, we acknowledge its potential misuse in adversarial applications. To mitigate risks, we advocate its responsible use for research and defensive purposes only. Finally, we urge developers of ML security systems integrate our findings to design more resilient defenses against adaptive attacks. We encourage future research to explore fundamentally secure purification methods that are inherently resistant to manipulations.
Limitations, Our work focuses on dissecting the robustness of DBP defenses, both theoretically and empirically, with an emphasis on attack strategy. While briefly consider existing mitigation techniques, future work should explore how diffusion models can be more effectively leveraged for defense. Classifiers could also be adversarially trained on LF perturbations improve robustness. Yet, the vast hyperparameter and architecture space of LF potentially creates an effectively infinite threat surface, limiting the practicality of adversarial training. We leave this to future investigation.
While we do not report formal error bars or confidence intervals, this decision aligns with standard practice in the literature (e.g., [20, 26]) due to the high computational cost of gradient-based based DBP attacks. Instead, we perform extensive experiments and also provide repeated empirical trials and variance analyses in Appendix D.2 that confirm our conclusions are robust, despite the absence of formal statistical intervals.
Although our main text focuses conceptual and empirical issues in DBP, Appendix D.1.3 and Appendix D.3 provide theoretical and asymptotic cost analysis, and Appendix H presents latency measurements and real-workd speedup comparisons. Our module is efficient, scales linearly with diffusion steps, and outperforms prior implementations in both latency and throughput. Thus, while cost is not emphasized in the main paper, it is extensively addressed and poses limitation in practice. Nonetheless, even with our efficient implementation, the dominant cost from DBP itself is inherited by attacks and robustness evaluations, rendering them costly regardless. That said, DBP's practicality may improve with advanced hardware or more efficient schemes in the future.
B Equivalence of DDPM and VP-SDE
An alternative to the continuous- time view described in $2 (i.e., VP-SDE) is Denoising diffusion probabilistic modeling (DDPM) [19, 34], which considers a discrete-time framework (DM) where the forward and reverse passes are characterized by maximum number of steps T. Here, the forward pass is Markov chain:
c 111--,,,111 Bi,z
where z; NOO,, Ia) and 3, small positive noise scheduling constant. Defining dt 4. we know due to Song et al. [36] that when T - op (i.e, dt-00, 0, which is the effective case of interest), this converges to the SDE in eq. (/) (with the drift and diffusion function f and described in $2). The reverse pass is also a Markov chain given as:
dē ã.-1 c.
B, ((1 11 B,)2, + (,,)( B.z.
(8)
V1
When T- dă converges to eq. (4) (see [36]). Thus, the two views are effectively equivalent. Accordingly, we focus on the continuous view, which encompasses both frameworks. For discrete time, ætt) will be used to denote 2 [
21

---

C Proof of Theorem 3.1
Theorem 3.1. The adaptive attack optimizes the entire reverse diffusion process, modifying the
parameters {0 1t such that the output distribution z(0) DBP(()) where DBP(%) (x) is the DBP pipeline with the score model's weights {6 Jtge* adversarially aligned. That is, the adversary implicitly controls the purification path and optimizes the weights {0 to maximize:
max E #(0))-Dplet (Pr(-y)((0)]} (0 ce1
Proof. The adversary aims to maximize:
((x)
Expanding this expectation yields the following alternative representations:
f pr((=l)500))((())44)))
(xx))
f
Pr Pr(-y)((0))
((#(())=
The first transition is due to the definition of expectation, the second follows from the definition of marginal probability, and the final transition replaces the joint density function with its compact form.
Since this formulation abstracts away a and (eage, we explicitly include them as p(o = p( p( x, ( Jege)). For notation simplicity, we define pea (ăt+al (x) p(0 (z, (0t (0])))) Thus, the final optimization objective is:
Q(x) = Pr( Pr(-vl((0))pe, x))
Since assume smoothness (C²), we interchange gradient and integral:
[<[[(=)] Pr(-yl=(0) V vPo (#re. [))
where the last step is because Pr(-yl((00) does not depend æ, but only z(0) which is independent of æ (though its probability of being the final output DBP is a function of æ)). Optimizing the objective via gradient ascent relies altering DBP's output distribution alone, evident in its gradient where Pr(-y) (0)) assigned by the classifier is a constant with its gradient ignored. While we lack direct access the gradients of the probabilistic paths above, may still attempt to solve the optimization problem.
Recall that by definition, Yu (u) Pr(vlu). Thus, the required gradient can also be expressed as:
[ Pr( (onlyy=(00))pe (#.*0o x (z)
V [(() V
E 1 (+ phg k))
-
E [MM (#.
Yet, this expectation is over probabilistic paths whose randomness due to the noise e of the forward pass and the Brownian motion dw, each reverse step that are independent. Hence, by the law of the unconscious statistician:
[[MP(E(0))] = - pho (2)
E [MM
p-(e, ................................ dd00))
where (r-e,, dures .dwo)) denotes the joint distribution of the noise and the Brownian motion vectors in the reverse pass. Note that z(0) the RHS denotes the output obtained by invoking the
22

---

DBP pipeline with some assignment of these random vectors on the sample æ. As earlier, we can interchange the derivation and integration, obtaining:
Va[MM(((0)]]
E
(e deno)
Let GO denote the random variable that is assigned values from V (²))), where z²(0) is as described above, and denote its covariance matrix by Mg.. Essentially, we are interested in [B we define V = as:
V (M(
where each a(),, is the output of DBP invoked with certain random path (e", d Lid p(e, diū,,,..duro) and N is sufficiently large number of samples. Then due to the central limit theorem, V, (E _g²) That is. propagating multiple copies through DBP and then averaging the loss gradients computes the required gradient for forcing DBP to alter its output distribution. The larger the number N of samples is, the lower the variance of the error, can be seen above. Note that the adaptive attack operates exactly in this manner, assuming it uses M as the loss it minimizes (but the soundness of the approach generalizes intuitively any other loss with the same objective over the classifier's 's logits), proving our claim.
D Details on Our DiffGrra Gradient Module
In Appendix D.I, we identify key implementation challenges in prior work, detailing how DiffGrad systematically resolves each one. Then, in Appendix D.2, we empirically demonstrate their impact on DBP's gradients when left unaddressed, reinforcing their practical significance and validating the need for our reliable module. Appendix D.3 presents the pseudo-code for our memory checkpointing algorithms, offering a transparent blueprint for reproducible and correct gradient- enabled purification. Finally, in Appendix D.4, we verify the correctness of our implementation, confirming that the gradients computed by DiffGrad match those produced by a true differentiable purification pipeline.
D.1
The Challenges of Applying Checkpointing to DBP and How DiffGrad Tackles Them
D.1.1 High-Variance Gradients
The challenge of high-variance gradients due to insufficient EOT samples is introduced $3.2. We defer its empirical analysis to Appendix D.2, where we quantify the gradient variability under different sample counts. Practical implications for performance and throughput- enabled by DiffGrad's design-art explored in the ablation study in Appendix H. Accordingly, we focus below the remaining implementation issues.
D.1.2 Time Consistency
Rounding issues may emerge when implementing checkpointing over torchsde solvers: Given the starting time t*, the solver internally converts it into PyTorch tensor, iteratively calculating the intermediate outputs by adding negative time increments to this tensor. On the other hand, checkpointing requires re-calculating the intermediate steps' samples during back-propagation. If this code is oblivious to torchsde's 's conversion of the initial time into a PyTorch tensor, it will continue to treat it as a floating point number, updating it with increments of the time step to obtain each intermediate used to re-compute the dependencies. The PyTorch implementation does not aim for 100% accuracy, leading to minute discrepancies the current value of / compared to pure floating-pot point operations. These inaccuracies accumulate over the time horizon, potentially severely affecting the gradients. Instead, we ensure both the solver and checkpointing code use the same objects (either floating points tensors).
23

---

D.1.3 Reproducing Stochasticity
CuDNN-Induced Nondeterminism. We found that even in deterministic purification pipelines, subtle nondeterminism from PyTorch's backend can cause discrepancies between forward and backward propagations. Specifically, CuDNN may select different convolution kernels during the recomputation of intermediate states checkpointing. leading to minute numerical drift.
To eliminate this, DiffBreak explicitly forces CuDNN to behave deterministically, guaranteeing consistent kernel selection and numerical fidelity. While this source of error subtle, it is measurable: for DiffPure with t* 0.1 and dt = 0.001, we observed raw and relative gradient discrepancies of le 4 and 7.28e 5 when all other issues have been addressed. With this fix, the discrepancy dropped to exactly zero. Although less severe than other sources of error (e.g., time inconsistency or missing gradient paths), this level of precision is critical in setups where gradients are small to prevent sign changes that can affect optimization trajectories. Our inspection of prior work's 's code reveals this drift was not previously addressed.
Noise Sampler for Guidance Robustness. While the contribution of the stochastic component of dē often cancels analytically in vanilla VP-SDE-style purification (see $2), meaning it does not affect the gradient and therefore is not required to be accurately reproduced during checkpointing, this is not guaranteed in all variants. In particular, many guidance-baser schemes-see $3.2-integrate stochastic terms within the guidance function itself. That is, the guidance function may incorporate stochasticity. When this occurs, gradients become sensitive to noise realizations, and failure to preserve them could break correctness. Yet, standard checkpointing only stores the intermediate (t)s. Hence, (tt+ + dt) will differ between the propagation phases as dá is computed via different random variables.
We restructure the logic computing dæ: We define function cale_ da that accepts the noise as an external parameter and utilize a Noise Sampler NS, initialized upon every forward propagation. For each f, NS returns all the necessary random noise vectors to pass to cale dx. Instead of 2(0) only, forward propagation also outputs N and a state S containing all @(fss. These objects are used to restore the path in backpropagation. This design also future -uture-proofs DiffBreak for emerging guided or hybrid purifiers: any randomness on which the guidance depends is captured by N V and replayed exactly during differentiation.
The memory cost of our design is O {* (aq) storing only NS, and all (()ss, each with a negligible footprint (0(1)) compared to graph dependencies [20].
D.1.4 Guidance Gradients
In schemes that involve guidance, it is typically obtained by applying function Ifn to (() each step (note that I may involve stochasticity- sse Appendix D.1.3 as is often the case in GDMP [40]). For instance, in Srideed-DDPM [40], the original sample æ is used influence the reverse procedure to retain key semantic information, allowing for an increased budget t* to better counteract adversarial perturbations. Effectively, it modifies eq. (8) describing the reverse pass of DDPM as:
dā VI -1-B. (11 (1-
(1- B,) + (,88((;;))) V a. GC x) + VB
where GC is guidance condition (typically distance metric), each step minimizes by moving in the opposite direction of its gradient, while the scale 8 controls the guidance's ' influence.
That is, in the specific case of Guided-DDPM a distance metric. Nonetheless, other choices for 9 may be employed in general. Typically, as the goal of the guidance is to ensure key information from the original sample æ is retained, fn will also directly involve this æ in addition @tt) (e.g., GC above). Yet, naive implementation would back propagate the gradients to æ by only considering the path through z(t). Yet, when g_ fn relies on a guide constructed from E to influence &(t) (e.g., guide = x in Cuided-DDPM. creates additional paths from æ to the loss through this guide at each step t. Accordingly, DiffGrad augments the process to include the gradients due to these paths. may not be identical to but can rather be computed based on or even completely independent (in which case no guidance gradients are collected). DiffGrad captures this nuance through an abstract function g aux that, given æ, outputs the guide. Similar to eg. (6), we have that for each t, the
(()) = GC((((), x). where GC is
the general case, this guide
24

---

gradient of any function F applied to ž0) w.r.t. guide due to the path from @(t + dt) to (( is given by:
V V
guide F V guide (t + dt), V dt( F>
as (t + dt) is a function of guide. Recall that we are interested in F, which is the loss function over the classifier's output on (0). The gradient of F w.r.t. guide is a superposition of all these paths' gradients, given as:
V guide F
VV guide F
(10)
By the chain rule, F's gradient w.r.t æ due to the guidance paths, which we denote as Vg
is:
P = V (guuiee V guidee
(II)
As a convention, for an unguided process or when guide and a are not not related, we define the gradient returned from eq. (//) as Vg F= O, preserving correctness in general. Since x in this guided scenario traverses both the guidance and standard purification paths, the final gradient is the sum of both components.
Finally, automatic differentiation engines, by default, generate gradients without retaining dependen- cies on the inputs that produced them. Thus, when the guidance itself is in the form of a gradient as in the case of Meddedd-DDMMMM or other potential alternatives, its effects will not be back procc-pppggtted to through any of the two paths described above, despite our proposed extensions. DiffGrad alters this behavior, retaining the dependencies of such gradient- guidance metrics as well.
D.1.5 Proper Use of The Surrogate Method [24]
As noted in $3.2, DiffHammer [41] adopts the surrogate method (originally proposed by Lee and Kim [24]) for gradient computation, as acknowledged in their Appendix C. This approach approximates DBP's gradients by replacing the standard fine- fine-grained reverse process (c.g.. dt = 0.001) with a coarser one (e.g., dt 0.01), therehy enabling memory-efficient differentiation using standard autodiff. For instance, when using diffusion time f* = 0.1 for DBP (see $2), the fine- grained process would require 100 steps, which is computationally infeasible for gradient computation without checkpointing. In contrast, the coarse-grained process would only require 10 steps, which is possible.
Specifically, the forward pass (diffusion) is computed using the closed-form solution from eq. (2), while the reverse pass uses dt, enabling efficient gradient computation. Notably, this replacement only occurs for the purpose of updating the attack sample (i.e.. computing the gradients) and happens during both forward and backpropagation so that the two phases match. In contrast, when evaluating the generated AE, it is purified via the standard fine-grained process that uses dt, as this is the true (full) procedure the model owner (defender) would execute. Although the surrogate process yields approximate gradients for a slightly different process, Lee and Kim [24] found it effective and more accurate than other approximations like the adjoint method [25] (despite remaining slightly suboptimal-set Appendix F).
However, manual inspection of HammHammer's code reveals that their implementation of this surrogate method contains critical issue. Rather than computing the reverse process with the coarse dt as intended for attack optimization during both forward and backpropagation, they first run the reverse pass during forward propagation with the standard dt. storing intermediate states. Then, during backpropagation, they attempt to backpropagate gradients using checkpoints spaced at dt intervals, but instead reuse the stored dt-based forward states. This leads to mismatch: gradients intended for steps of size dt are applied to states computed with dt, effectively disconnecting the computation graph and introducing significant gradient errors. In practice, this behaves like hybrid between checkpointing and BPDA (2), where gradients are approximated around fixed anchor points without correctly tracking the reverse dynamics.
As we show in $4.2, this implementation leads to significant attack performance degradation compared to the full gradients. It illustrates the fragility of gradient computation in diffusion attacks and further motivates the need for our reliable DiffGrad module.
http:::///giccuu..oo//aa//////// wwt.://.ooaaaabo/aa Hameeee (accessed in May 2025)
25

---

D.2 Demonstrating The Effects of Incorrect Backpropagation
We empirically demonstrate the impact of the identified issues on DBP's gradients Fig. 2 and briefly explain each result below. As numerical results for unhandled stochasticity) are already presented in Appendix D.1.3, we focus here on the remaining four issues detailed in Appendix D.1:
nondeterminism (i.e.,
(a) Effect of insufficient EOT samples. Fig.2a illustrates the impact of using too few EOT samples when computing gradients-sees33 and Appendix D.1.1. While prior works [41, 20, 24, 26, 30] typically use 10-20 samples, such low counts, while somewhat effective, may introduce significant variance, resulting in noisy gradients and suboptimal attacks.
To quantify this, run the following experiment 001, dt= 10 9). For each EOT count N (shown on the x-axis), we generate N purified copies, compute the average loss, and backpropagate obtain the EOT gradient. This is repeated 20 times to yield 20 gradients g .....I2) for each N. We then define:
CIFAR-10 sample a purified with Dure
1
mean (N) 9d
mar 1
9 ll2
20
This metric reflects the worst-case deviation between gradients under repeated EOT sampling. By the central limit theorem (see Appendix C), the variance of EOT gradients decays with N, and thus (N) - as N - oc. The curve in Fig. 2a captures this decay and lets us identify a threshold where variance becomes acceptably low.
The plot confirms that gradient variance decreases sharply with the number of EOT samples. While N 1hethe typical choice in prior work-is somewhat effective and viable under resource constraints, it remains suboptimal. We observe that variance continues to drop until around N 64, after which it plateaus. This suggests that although larger N values like 128 yield marginally better stability, the bulk of the benefit is realized by N = 32-64. Thus, N = 10 is a practical baseline, but N > 64 should be preferred when accuracy is critical and compute allows, coroborating our claims in $3.2.
94
10
0.9
10
30
8
30
10
01
EOT Somples
(a) Effect of insufficient EOT samples
(b) Effect of time inconsistency
104
20
110
100
10"2
10
10
&
9u (dt=1e-02)
10
(dt=1e-02)
10"
(dt=Se-03)
9: (dt=5e-03)
102
10a
9u (dt=1e
(di=1e-03)
0.505
0010
8015
0.020
0.525
0.030
0.035
100
15a
dRidt
(c) Effect of ignoring guidance gradients
(d) Effect of incorrect surrogate gradients
Figure 2: Effects of the identified issues in DBP's backpropagation. Each subfigure visualizes specific error source.
26

---

(b) Effect of time inconsistency. As detailed in $3.2 and Appendix D.1.2, rounding inconsistencies can emerge when using checkpointing with torchsde solvers. These stem from how the solver inter- nally handles time as a PyTorch tensor, which may diverge subtly from floating-poil point representations if not carefully synchronized. While such discrepancies are negligible for short purification paths (small E*), they accumulate significantly over longer trajectories-partiiee particularly at standard DBP settings like 1* =01 with dtt=10-3 (i.e., 100 reverse steps), as in DiffPure on CIFAR-10.
To quantify this effect, we purify fixed CIFAR-10 sample x using values of t* from 0.005 to 0.1 in steps of 0.005. For each t², we compute gradients using DiffGrad under two conditions: (1) without correcting the inconsistency (yielding Inf), and (2) with the issue fixed (yielding gy), both using the same stochastic path. We then compute two metries: (1) gd | gn (absolute error),
and (2) ge gg 921 (relative error). These quantify the divergence introduced by uncorrected 19
rounding- both in magnitude and in proportion to the true gradient-and illustrate its increasing severity with longer reverse trajectories.
Fig. 2b confirms that even minute rounding discrepancies in time handling, if uncorrected, can completely corrupt gradients over longer diffusion trajectories. Both the absolute error 9d (blue) and the relative error ge (orange) grow rapidly with t" By t* > 0.03, ge exceeds 90%, and for the typical setting of t* = 0.1, the relative error reaches 100%, indicating that the gradients are nearly orthogonal to the correct direction. This validates our claim that unpatched checkpointing introduces severe errors and demonstrates that our DiffGrad fix is essential for correct gradient computation in practical DBP setups.
(c) Effect of ignored guidance gradients. We evaluate the impact of neglecting guidance gradi- ents, a critical problem in current attacks against Guided DBP defenses (e.g., GDMP, MimicDiffi- sion)-see $3.2 and Appendix D.1.4. As explained earlier, guidance introduces additional gradient paths from the input to (t), typically requiring second-order derivatives to capture. Existing approaches ignore these paths, leading to incomplete gradients and suboptimal attacks. To our knowledge, DiffGrad is the first system to correctly handle them.
To quantify this effect, we purify a fixed CIFAR-I0 sample x using GDMP (setup in Appendix E) while varying the purification horizon 1 E {0.006, 0.012,.... 0.036). For each 1*, we compute gradients using DiffGrad with and without accounting for guidance-yieldingg g and gnft respec- tively-under the same stochastic path. We then compute the absolute and relative gradient error (ga. ge) as in previous experiments.
The findings in Fig. 2c clearly demonstrate the severe impact of neglecting guidance gradients in Guided DBP setups. Both the raw error (gu) and the relative error (ge) increase steeply with t*, confirming that as the purification path grows longer. the omitted gradient paths sttamming from guidance dependencies-ddominiie dominate the overall gradient signal. Most notably, Je exceeds 10t at 1 0.036, indicating that the gradient used in prior works is essentially meaningless for optimization, as it no longer aligns with the true direction of steepest descent.
These findings directly explain the poor attack success rates previously observed on guided defenses like GDMP and MimicDiffussin, and reinforce why our attacks, which correctly account for these guidance gradients via DifTGrad, achieve drastically superior performance-sed $4.2 and $6. In short, the guidance mechanism, when improperly handled, not only fails to help but actively hinders attack effectiveness by yielding incorrect gradient signals.
(d) Effect of incorrect surrogate gradients. As explained in $3.2 and Appendix D.1.5, prior work [41] uses the surrogate method [24] to enable efficient gradient computation for DBP. How- ever, identify critical divergence in DiffHammer's [41] implementation of this method that substantially compromises attack performance. This not only leads to inflated robustness estimates for DBP, but also to misleading conclusions about the relative effectiveness of the standard gradient- based attacks versus Dim'Hammer's proposed enhancements-see $4.2. Specifically, as detailed in Appendix 5.1.5, DiffHammer mismatches the time discretization used in the forward and backward passes: the forward pass computes intermediate denoised states ((t) using a fine step size dt, while the backward pass attempts to propagate gradients at a coarser granularity di > dl using those same (misaligned) forward states. This inconsistency leads to gradients being applied to states generated under a different dynamics, corrupting the computation and severely degrading the attack.
27

---

To validate this, we conduct a final experiment: For three different values of dt E {0.001, 0.005, 0.01} and fixed t 0.1, we purify fixed CIFAR-10 sample a. For each dt. we evaluate four surrogate variants using dt E {dt, 2dt, 5dt, 10dt). For every configuration, we compute f using DiflHammer's surrogate implementation and 9 using the correct implemen- tation. As before, we report both Ja and 9. quantify the absolute and relative gradient error due to the mismatch between dt and dt. Each experiment is repeated 10 times, and we report the mean values.
The results in Fig. 2d clearly validate our claim: the gradient errors induced by DiffHammer's surrogate implementation grow substantially with increasing di/dl, saturating quickly even for modest mismatches. Both absolute error Ja and relative error le consistently worsen across all di configurations, indicating that gradients being applied the wrong points in the computation graph.
Crucially, DiffHammer uses dt 0.01 and dt 2dt 0.02, a setup which already produces substantial degradation (ga 2 3. 9e ~ 1), confirming that their reported results rely on gradients that diverge significantly from the correct ones. These errors propagate into the attack, weakening its effectiveness and misleadingly suggesting that first-order attacks are inherently inferior. This experiment conclusively demonstrates that DiffHammer's surrogate misuse is not a minor detail, but a critical bug that undermines their central claims.
We note that DiffGrad also provides a correct implementation of the surrogate method (despite the full correct gradient being the main method considered in the paper).
D.3 Pseudo-Code for Our Memory-Efiicee Efficient Staddeett-Eabbles Purification with DiffGrad
Algorithm 1 Differentiable Purification with DiffGGad - Forward Propagation
Require:
Sample æ, Score model so, Optimal diffusion time 1*, step size dt,, Noise scheduler B Reverse diffusion function calc_dx, Noise sampler initializer wwl__noissssppppp Guidance condition gf. Guidance scale s, Auxiliary guidance extractor g_aux
1: steps guide (_)
1 Cale. #steps and init. guide */
2: disable dependencies()
/*
Dependencies enabled during forward
propagation. Disable them. #/
3: S
|* Saved state (will eventually hold all
intermediate reverse steps' outputs). */
4: seed - random raadom_seed( seed()
1* Seed to initialize noise path */
5: NS init noise sampler(seed)
Reproducible sampler #/
6: a calc_alpin (Bl__alph((3)
1* Calculate factors from eq. (2) */
7: Draw c C~ Ia)
8: -- --Va(()) + v - 0x (*(())
/* Diffuse according to eq. (2) */
9: for sleps, steps 1, ..., do
10: S.append(2)
/ Set S a */
step noise NS.sample((
Sample the random noise used to calculate di at step i. */ 1 Calc. dă according to
12: dā cxlc_dx(2___ 80, i, dt, 8,
13:
sste_noiise gi,, 8, guide)
eq. (4) */
14:
Φ 1-2 dē
/* Update +c( + di). #/
15: end for
16: enable dependencies()
8 Re-enable dependencies. #/
17: return &, S, NS, guide
Forward Propagation. DiffGrad's forward propagation logic is in Algorithm 1. blue is optional, pertaining to the use of guidance. We highlight in red the portions that differ from standard forward propagation. First, we generate the guidance guide from æ (line 1) and disable all graph dependency storage (line 2), enabling our code to run efficiently without attempting to store graphs that will lead to memory failures. Afterward (lines 3-5), we initialize S as empty list and
The code in
28

---

draw a random seed that is then used to invoke the abstract function init cett_ooiee____ee sampler, which returns noise sampler that provides a reproducible random path for the backpropagation phase (see $3.2). After the input is diffused (lines 6-8) via eq. (2), lines 9-15 correspond to the reverse pass: At each step (effectively i), & (that now represents a +({) is first appended to S, which will eventually contain all such intermediate outputs (line 10). The noise provided by NS for the current step i is then retrieved (line I1) and used to compute dē (line 12). dē is then added to ø so that its current value becomes @tt + dt). This repeats until & (()). Unlike the naive implementation, we only store the intermediate results. For efficiency, we also avoid saving the random noise for each step but utilize NS to reproduce those variables dependency storage (line 16) to ensure our code does not interfere with the execution of any other modules. Finally, øb (0) is returned together with the state S and the sampler NS, which are stored internally for reproducibility during backpropagation.
demand. Before termination,
re-enable
DBP Parallelism. Although Algorithm 1 is presented for single purification path, DiffGrad natively supports parallel purification of N stochastic copies to potentially obtain a significant speedup when computing higher-quality EOT gradients-sct see $3.2. 2. For simplicity, we abstracted away the batch logic in the pseudo-code. In practice, prior to line 1, the input æ is replicated N times (N is an additional argument that can be provided to Algorithm 1). Line 4 returns N random seeds, and the resulting sampler NS manages N reproducible random paths. Line 7 draws N distinct noise samples to generate N diffused versions of æ, and all subsequent steps operate copy-wise -wiie in parallel across these N instances.
Backpropagation. DiffGrad's backpropagation logic is in Algorithm 2. Similar to earlier, red text refers to operations that deviate from traditional backpropagation, while blue lines are optional (guidance-related). In addition to the usual gradient grad w.r.t. (()), the inputs include multiple parameters normally exclusive to forward propagation, as they are required to re-calculate the dependencies. Additionally, the algorithm accepts the saved state S, and the same noise sampler N S to retrieve the stochastic path of the forward propagation. Before providing details, we note that by definition VA, B e Rd, holds that:
where O denotes the element-wise product. Therefore, in order to calculate the gradients w.r.t. .(t) and guide as described in eq. (6) and eq. (9), we may define an objective at each step & as:
Obj E(E(t dt) + V Va((+a))F)) F)
(12)
and take its gradient w.r.t. the two elements of interest above, which explains the steps in our pseudo-code in Algorithm 2.
The procedure begins by creating a variable g grad and setting it to 0 (line 2). This will later be used to store the guidance gradients (see Appendix D.1.4). For each time step t (i.e., step i). starting from f' = dd (i 1), the process (lines 3-14) first retrieves æ(t) from the saved state S (line 4) and the corresponding random noise for that step used during forward propagation (line 5) and computes a(l + di), denoted as a (lines 7-9). Importantly, these computations are performed while storing graph dependencies (enabled on line 6 and re-disabled on line 11 to restore the normal execution state). Specifically, during the first step. we calculate a0() from d()d Afterward, we compute the objective Obj, (line 10) following eq. (12) that allows us to pack-prop the gradient from (0) to (--df) and guide using the stored dependencies, as per eq. (6) and eq. (9). grad is then updated to hold the gradient of the loss function w.r.t. -(-dt) as desired (line 12), and the gradient of guide due this guidance path (i.e., from guide to the loss due to guide participating directly in the calculation of ăt + dd)) dd) see Appendix D. 1.44 is added to g grad (line 13). This process repeats until grad finally holds the gradient w.r.t. (tt") and g g_grad holds the sum of gradients due to all guidance paths w.r.t. guide (eq. ( (10)). Note that after the required gradients w.r.t. Z(t) and guide are obtained at each step, the dependencies are no longer needed and can be discarded. This is where our approach differs from traditional backpropagation algorithms, enabling memory-efficies gradient calculations (at the cost of an additional forward propagation in total). At this point (line 14), we have the gradient V Va( F and all is required is to use it to calculate F, which is trivial
29

---

Algorithm 2 Differentiable Purification with DiffGGrad Backpropagation
Require:
Loss gradient grad w.r.t Zn, Sample æ, Score model Sp. Optimal diffusion time tM step
size dt, Noise scheduler (3, Reverse diffusion function calc, dx, State S { [[ di IT], Noise sampler NS, Auxiliary guidance input guide, Guidance function gfn Guidance scale 8
**
dt 8
1: steps
2: g grad
1 Init. gradient w.r.t guidance input */
3: for i 1.2, steps do
4: ④ S[i]
/a Set ã =(t)) */ 1" Retrieve noise for step */
5: step_noise - NS NS.saplle()
6: enable )
7:
dā - calcdd_ dx ét 88, di, B,
8:
step_noise gm, 8, guide)
9: d+ - + dæ
1 i d(l + dl) */
Obje - E( O grad)
1* Objective due to eq. (12) */
11: disable dependencies()
grad Obbb
1* Update gradient w.r.t @(t) (eq. (6)) */
13: g g_rrad - g grad + V guidde Obj
1 Update guide gradient (eg. (I0)) */
end for
15: a calc_all (glc_alpha(3)
16: grad - grad ((()*)
1 Loss gradient w.r.t x (eq. (2)) */
17: g ggradd - V > gugde G grad)
1 Guidance gradient w.r.t æ (eq. (I1)) */
18: grad - grad g grad
1 Merge loss and guidance gradients */
19: return grad
due to the chain rule since the closed-form solution for x((*) x<(*) from eq. (2) indicates that this is equivalent to VzF = (a(*) (Va(+*)E) as we compute on line 16. We then calculate the guidance paths' gradient w.r.t following eq. (11) on line 17. Finally, we sum both components, returning the precise full gradient w.r.t. x.
DBP Parallelism. Algorithm 2 is written for single purified copy but in practice operates over batch of N stochastic instances like Algorithm 1. As the forward pass stores trajectories, the backward pass propagates gradients independently for each. The operations in the pseudo-code are thus applied copy-wise in parallel across all instances. Finally, after line 18, the gradients are averaged, yielding the EOT gradient.
D.4 Verifying the Correctness of DiffGrad
To ensure the correctness of DiffGrad, all that is required to verify that each øtt + dt) computed during backpropagation (line in Algorithm 2) exactly matches the corresponding neward-computte æ(t + dt) (line 14 in Algorithm 1). This equality guarantees that the reconstructed computation graph faithfully mirrors the one produced by standard autodiff engines during their normal (non- checkpointed) operation, ensuring exact gradient recovery since we use them to perform the necessary backpropagation between each #t dt) and ((t).
Since our forward pass explicitly stores all intermediate states in S, we compare each recomputed @tt + dt) with its forward counterpart during backpropagation. An exact match confirms that the system is computing precise gradients. We manually validated this for all timesteps.
For guidance gradients, correctness follows from the derivations in Appendix D.1.4. Once incorpo- rated, the same matching procedure ensures their correctness as well.
E Additional Details on Systems & Models
WideRessNet-28-11 and WideResNet-700-16 [52] are used for CIFAR-1O, and ResNet-50 [18], WideResNNet-5-22 and DeiT-S [14] for ImageNet, similar to [30, 20]. For VP-SDE DBP (Diff- Pure) [30], the DMs [12, 36] are those from the original work. We also experiment with the Guided-
DMPM DDPM (see $3.2), GDMP [40], due to its SOTA robustness, using the author-evaluated
30

---

DMs [12. 191. The settings match the original optimal setup [40, 30]: For Diffpure, t" = 0.1 for CIFAR-10 and t*=0.15 for ImageNet. For GDMP. CIFAR-IO sample is purified m=4 times, with each iteration running for 36 steps (L*=0..36), using MSE guidance [40]. ImageNer uses 45 steps (m=1) under DDPM-acceleration [12] with SSIM guidance [43].
F One-Shot DBP Baseline Comparisons Against DiffGrad's Accurate Gradients
As noted in $3, DBP's robustness stems from sources: inaccurate gradients and improper evaluation. As our work offers enhancements on both fronts, we evaluate each factor separately. Here, we isolate the gradient issue by re-running prior experiments under the same one-shot evaluation protocol, but with accurate gradients via our DiffGrad module and compare the results to those from the literature. We restrict our attacks to 10 optimization steps (with AA-Ls)), while prior works often use up to 100, giving them clear advantage. We evaluate on CIFAR-1O with WideResNet-288 10, using N 128 EOT samples as justified in $3.2. For papers reporting several results, we chose their best.
Baselines. To demonstrate the efficacy of exact full gradients, compare our DiffGrad module against prior gradient approaches. For DiffPure, the adjoint method was originally used [30], while GDMP was initially evaluated using BPDA [2] and blind variant (ignoring the defense entirely) in Wang et al. [40]. These approximations were later criticized for poor attack performance [41, 26, 20, 24]. More recently, Lee and Kim [24] proposed the surrogate process approximate the gradients. performing the reverse pass with fewer steps during backpropagation to reduce memory usage, enabling approximate gradients via standard autodiff tools. DiffAttack [20], DiffHammer [41], and Liu et al. [26] proposed checkpointing for memory-efficien full-gradient backpropagation. However, DiffHammer avoids the standard one-shot evaluation (1-evaluation) protocol and reports no results under it, while DiffAttack does not evaluate GDMP and continues to use the adjoint method for DiffPure. We focus here on existing results under the 1-evaluation protocol and defer the discussion of these works' conceptual issues (see $1) to $4.2.
Metric. Following prior works, we report robust accuracy (Rob-Acc): the fraction of samples correctly classified after the attack completes and the final adversarial example is purified and evaluated (once).
Results. Table 4 shows comparison. All methods achieve similar clean accuracy (Cl-Acc) without attacks, with minor variation due to sample selection. Thus, robust accuracy (Rob-Acc) differences reflect the effect of gradient methods. Full denotes the standard AA attack that uses the full exact gradients (i.e., via checkpointing).
Our approach significantly out- performs gradient approxima- tions such as Adjoint, Blind, and BPDA, reaffirming their known weaknesses. More notably, de- spite using only 10 optimization steps, our method reduces Diff- Pure's Rob-Ace by 14.06% com- pared to Liu et al. [26], who also use exact gradients, confirm- ing their backpropagation mis- matches (see $3.2).
Table 4: One-shot AA-La comparison on CIFAR-IO (E,-8/255). f indicates strategy is PGD.
Models
Gradient
Cl-Acc 8
Rob-Acc
Adjoint (Nic et al. [30])
89.02
70.64
DiffAttack (Kang al. [20p
89.02
46.88
DijPPre [30]
Surmggte (Loc and Kim (24))*
90.07
48.22 28
Full (1.iu et al. [26))
89.26
11
Full-DiftGrad (Ours)
89.46
48,05
Blind al, [40p
93.50
9016
BPDA (1.ce and Kim (24])
89:96
75.59
GDMP (40)
Surrogate (l.ce Kim (24))
89.96
24.53
Full-DiffGrad (Ours)
93.36
19.53
While DiffAttack remains
slightly stronger, the difference is extremely small (1.4%) and can be safely attributed to differing evaluation samples. Importantly, the original gap reported in Kang et al. [20] between DiffAttack and the standard AA-La dropped by 23. 26%, undermining its claimed advantage due to the per-step deviated reconstruction losses and aligning with theoretical findings in $3.1. As DiffAttack involves broader architectural changes beyond gradient logic, we provide detailed comparisons in $4.2, where clearly showcase its inferiority.
Our method also slightly outperforms Surrogate for DiffPure (0.23%). but we caution against overinterpreting this small gap: under the improper she- evaluation, several purification paths can
31

---

still cause misclassification even if the majority yield correct labels. As such, good approximation methods like Surrogate may appear closer in performance than they truly are. To verify the superiority of our DiffGrad's s full gradients over the surrogate approximation against DiffPure, we thus further compare the two under the realistic MV protocol studied in $4.2, executing AA- with the surrogate process to obtain the gradients when attacking the same CIFAR-10 R-0 classifier considered in $4.2 (i.e., WideResNeet-7--1E 16) and using the same number of samples (N = 10) over which the majority vote is taken. We find that the surrogate process brings MV.Rob to 45% only, whereas our Full-DiffGrad lowers this number to 39.45%, achieving a considerable improvement of 4.3% and unequivocally proving the advantage of exact gradient computations. Finally, for GDMP, our method outperforms Surrogate by 5% even in the one-shot evaluation protocol (see Table 4). largely due to our incorporation of guidance gradients-entien unique strength of DiffGrad.
absent in all prior approaches highlighting the
Note that the surrogate method shortens the trajectory and backpropagates through this shorter hori- zon but retains the full graph over that path. Hence, one may argue that the attack success degradation is an acceptable tradeoff' when using the surrogate method given the potential speedup. Yet. in DBP, memory-not FLOPs-is the bottleneck; Checkpointing (e.g., our DiffGrad) recomputes individual path segments, yielding exact gradients with lower peak memory. The surrogate may improve speed but not memory pressure, and its truncation can miss dependencies, weakening gradients as shown above. In practice, attackers precompute adversarial examples offline since DBP's latency makes real-time adversarial generation infeasible either way, so runtime savings alone do not translate to a practical advantage.
All all, the results demonstrate the fragility of DBP in the face of accurate gradients and highlight the issues in previous works' backpropagation. Nonetheless, this one-shot evaluation protocol remains problematic, as previously stated, leading to an inflated robustness estimate. In fact, under more realistic settings (see $4.2), we demonstrate that this gradient-based attack almost entirely defeats DBP when only one sample is used to predict the label (i.e., Wor.Rob).
G Evaluations with Liu et al. [26]'s Fixed AutoAttack
Liu et al. [26]. like DiffHammer [41] and own analysis, note that evaluat- ing a single purification at attack termination inflates robustness scores. Liu et al. [26) address this by evaluating 20 replicas of the final AE, declaring success if any is misclassified. (see $3.3), this protocol is more limited: it evalu- ates only at the final step, whereas Wor.Rob evaluates N copies at each attack it- eration. Accordingly. Liu et al. [26] group their method with one-shot eval- uations, providing a slightly more realistic assessment that leads to results similar to those attained via PGD [23].
While similar in spirit to the Wor.Rob metric we consider
Table 5: AA-La comparison on CIFAR-10 under Liu et al. [26]'s protocol (i.e., Fixed AutoAttack) (6x 8/255).
Models
Pur.
Gradient Method
Cl-Ace $
Rob-Ace
Full (1.iu et [26])
89.26
56.25
WideReaNet-28-10
DiffPure [30]
Full-DiffGrad
89.46
30.86
Full (Lau et 126)
91.80
40.97
GDMP [40]
Full-DiffGrad
93.36
10.55
DiffPhne (30)
Full-DifTGrad
89.06
35.16
GDMP [40]
Full-DifGGrad
91.8
8.59
We replicate their setup using DiffGrad (Table 5), running 20 iterations with N = 10 BOT samples per step and evaluating 20 final replicas. Under this protocol, our improvements are stark. On WideResNNt-22-11 10, we reduce Rob-Acc by 25.39% and 30.42% for DiffPure and GDMP, bringing fi- nal accuracy to 30.863 86% and 10.55%, respectively. These results confirm the superiority of DifIGrad's gradients and expose DBP's realistic vulnerability. We observe similar results on WideResNNet77-16 (not evaluated in [26]).
H Ablation Study with Different Numbers of Samples for Label Prediction
To assess the impact of the number of evaluation samples N, we test both single-purification and majority-vote (MV) settings with N E (1, 10, 128). Our goal to highlight the brittleness of DBP's standard deployment, which classifies based on single purified copy. As explained in $3.3, in
32

---

stateless setups (e.g., phishing, spam, CSAM), adversaries can resubmit identical queries indefinitely, Since randomized defenses like DBP can fail along certain stochastic paths, any neglliiiee misclassification probability compounds with repeated attempts- see $3.3. Although DBP assumes this probability is negligible, our proof in $3.1, results in $4, and the ablation experiments here contradict this. Furthermore, we wish to showcase the advantages of our proposed majority-vote (MV) setting that strives to mitigate this vulnerability by, instead, classifying based on expectation of the randomized defense.
We evaluate DiffPure and GDMP on CIFAR-IO using WideResNet-700 70-16 and (our) AA-Ezo (with e = 8/255 and 100 iterations). For N = 1 and N = 10, we use 10 EOT samples; for N = 128, we reuse the same 128 samples for both EOT and evaluation. This improves gradient quality, thus strengthening attacks, yet still leads to higher MV robustness, proving the superiority of our proposed method despite the use of more accurate gradients.
For Rob, clean accuracy (CI-Acc) remains constant across all N values since we always compute it using single purified copy (N 1), independent of the number N of samples used in the attack. This reflects the actual standard single-purification deployment of DBP, where the defender classifies a single output, while the attacker may retry multiple times. Hence, for Rob-Acc we consider a batch of multiple samples (the corresponding N in that row) and then declare attack success if a single misclassification occurs, revealing the gap between measured and effective robustness. In contrast, MV.Rob clean accuracy varies with N, as the prediction always aggregates over N purified samples, consistent with our proposed deployment.
Table 6 confirms that Wor. con- sistently declines with N, revealing the illusion of robustness under sin- gle evaluation. For instance, DiffPure shows drop from 16% (at N 1) to 17.58% at N 128 (far lower than the inflated numbers reported in previous studies- see Appendix F),
Table 6: AA-Lc Performance under various evaluation sam- ple counts
Wor $
MV. & Cl-Axc Rob-Acc
Pur.
#Samples
Cl-Acc
Rob-Acc
N=1
35.16
89.05
35.16
DiffPure [30]
N=10
89.06
17.19
91.02
39.45
N=128
17.58
92.19
47.72
confirming that many stochastic paths yield incorrect predictions, as our
N=1
8.59
91.8
K.59
GDMP [40]
N=10
91.8
7.03
92.19
16.8
properly implemented gradient-based attack (see $3.2) lowers the expected classification confidence, making such failure modes more likely. For GDMP, we observe similar trend.
N=128
5.47
92.19
32.81
In contrast, MV.Rob improves with N. rising from 35.16% to 47.72% for DiffPure, and from 8.59% to 32.81% % for GDMP. This affirms MV as more stable and accurate evaluation method that must be adopted as the de facto standard for DBP evaluations in the future.
Yet, despite MV's benefits, our gradient- based AA-Ez attack still degrades robustness, which never exceeds 50%. validating our theoretical finding from $3.1 that such attacks repurpose DBP into an adversarial distribution generator.
Computational Cost. While larger N increases computational overhead, we identify N 128 as the max batch size fitting in a single A100 GPU run requiring ~ 26.53s for inference, with N 10 offering practical middle ground (6.54s vs. 5.29s for N 1). Hence, for practicality and if throughput is critical, one should opt for N = 10. However, in security-critical systems where latency is not crucial, larger N is favorable. Yet, additional tests (not shown) suggest MV.Rob plateaus near N = 128. Note that these latencies refer to purification inference alone (classification excluded). During attacks (not standard deployment), there is also the cost of backpropagation. When accounting for the cost of classification and backpropagation, the latency becomes ~ 17s for N 1 and N 10 compared to 53s for N 128. As explained in $3.2, our design allows purifying multiple stochastic copies of the same sample in one step, in contrast to previous work. This reduces the runtime per EOT gradient from up to N 17s where N stochastic purifications are run serially-to just T SS where TN is the latency incurred by purifying N samples in batch, yielding a speedup of up to N 17/TNN for single-sample attacks, which amounts to 41.06x for N 128 allowing for a considerable speedup when the objective is to utilize many EOT samples to obtain accurate gradient estimates. For batch attacks, prior methods purify one stochastic copy per sample per EOT step and require all samples to converge before proceeding to the next batch of
33

---

inputs. blocking early termination. In contrast, DiffGraa purifies N copies of single sample in parallel, allowing samples to terminate independently. This greatly improves overall throughput by freeing compute sooner, especially when convergence varies across inputs.
ImageNet timings: While we limit ablation studies to CIFAR- R--0 for feasibility, ImageNet timings for completeness. At ImageNet resolution on a 40 GB A100 GPU, DBP inference (forward propagation only) requires 7s for N 1 and ~ 25s for N 8. A single attack iteration (i.e., forward and backward propagation) lasts 16s (N = 1) and ~ 75s (N = 8). Thus, batching EOT copies yields a significant speedup over serial purification for both attacks and inference, but models and activations dominate cost at this scale, so batching returns diminish faster than on CIFAR-10.
provide
Recommendations for Choosing The Batch Size N. We found N = 128 for CIFAR-1O and N = for ImageNet to the largest feasible batch sizes to fit into the 40 GB A100 GPU memory, making larger batches impractical for large-scale evaluations. That said, control experiments with larger N values for both datasets (split into batches) showed negligible robustness or attack performance gains, indicating saturation.
Following the above discussion, we recommend N 10 for CIFAR-1O as a practical default and N = 64 - 128 for courity-critical inference (MV) and highly accurate gradients when evaluating new attack methods such as our LF strategy-see $5. For ImageNet, we recommend N 8 for similar reasons; larger N values remain preferable when added latency is acceptable.
I Details on Our Low-Frequency (LF) Adversarial Optimization Strategy
L11 Understanding Optimizable Filters
In practice, OFss extend an advanced class of filters, namely guided filters, that improve upon the basic filters discussed in $5. Guided filters employ additional per-pixel color kernels that modulate the distortion critical points: Since filters interpolate each pixel with its neighbors, they are destructive at edges (intersections between different objects in the image), while the values of non-edge pixels similar to their neighbors, making such operations of little effect on them. Depending on permissiveness guided filters construct, for each pixel (i. j) a color kernel c of the same dimensionality MxN as IC that assigns multiplier for each of (f, j)'s neighbors, that decays with the neighbor's 's difference in value from (i, j)'s. 'ss. The output at (i, j) involves calculating the effective kernel =. ciu x (normalized) which then multiplies (i, j)'s vicinity, taking the sum of this product. Thus, contributions from neighbors whose values differ significantly are diminished, better preserving information.
Guided filters still employ the same K. for all pixels, changing only the color kernel that is computed similarly for all pixels. Thus, to incur sufficient changes, they would also require destructive parameters despite them still potentially performing better compared their pristine counterparts. Their parameters are also predetermined, making it impossible to optimize them for specific purpose. The OF's by Kassis and Hengartner [21] build upon guided filters but differ in two ways: First, instead of using the same K, they allow each pixel to have its own kernel K to better control the filtering effects at each point, ensuring visual constraints are enforced based on each pixel's visual importance. In this setting, K** denotes the set including all the per-pixel kernels K Second, the parameters Ox* of each filter are learnable using feedback from perceptual metrie ((pips) [56] that models the human vision, leading to an optimal assignment that ensures similarity while maximizing the destruction at visually non- non-critical regions. To further guarantee visual similarity, they also include color kernels similar to guided filters (see original paper for details [21]).
I.2 Attack Hyperparameters
Through experimentation, we found that the loss balaneing constants c 100 for CIFAR-IO and = 10 for ImageNet lead to the fastest convergence rates and selected these values accordingly (although other choices are also possible). UnMarker's s filter network architecture for ImageNet is identical to that from the original paper [21]. For CIFAR-10, since the images are much smaller, we found the original architecture unnecessarily costly and often prevents convergence since larger filters group pixels from distant regions together in this case, easily violating visual constraints upon each
34

---

update, resulting in the ipips condition being violated. Thus, opt for more compact network that includes filters with smaller dimensions, which was chosen based on similar considerations [21], allowing us to explore several interpolation options. The chosen architecture for CIFAR-1O includes filters, whose dimensions are: (5, 5), (7, 7), 55,5), (3,3). We use fixed learning rates of 0.008 for the direct modifier ö and 0.05 for the filters' weights, optimized using Adam. The remaining hyperparameters were left unchanged compared to [21].
1.3 Pseudo-Code
The pseudo-code for our low-frequency (LF) strategy (see $5) is in Algorithm 3. Importantly, as each " values should be non-negative and sum to 1, the values for each such per-pixel kernel are effectively obtained by softmaxing the learned weights. Initially, the modifier å is initialized to 0 and the weights (0x²) } are selected the filters perform the identity function (line 1). As a result, the attack starts with Fade æ that iteratively optimized. Similar C& W [4], we directly optimize Lado (i.e., using the modifier 8) in the arctanh space, meaning transform the sample first to this space by applying arctanh (after scaling Fadv arctanh's 's valid range [-1, 1] where å is added and then restore the outcome to the original problem space (i.e., [min_val, max_vall, which typically [0, 1]) via the tanh operation. Further details on this method and benefits can be found in [4]. All other steps correspond to the description brought in $5. Unlike 13.1, we assume the classifier M outputs the logit vector rather than the probabilities (i.c., we omit the so ftmaa layer over its output, which me may not be re-introduced by the loss f), as is traditionally done for variety of adversarial optimization strategies (e.g.. C& W [4]) to avoid gradient vanishing. We use the known max-margin loss [4]- Ellogits, y) =logits|; max (logits(. j]}.
379
Algorithm 3 Low Low-Frrquency (LF) Adversarial Optimization
Require: Sample x, Model (classifier) M, DBP pipeline D, Loss function True label y of σ, Perceptual loss Ipips, lpips threshold Tp. Filter architecture fi Balancing constant c, Iterations max_iters, Success condition Cond, Filter weights learning rate Ir Iror. Modifier learning rate (rs. Number of purified copies n, Number of EoT eot_ sot_steps, Input range limits (min_val, mar_val)
1: (0x²) - identity { f))55 -
/* Initialize attack parameters. */
2: Optim Adam([(@rec] }, 8). [rror, Cooorrrg Iral)
3: Pinn - inc cattscale_ande aannndarcaan((.. min_val, marr_al) /* Scale x to [-1, 1] and take arctanh */
4: for i - 1 to max max_iters cot__teep do
5: Cado - -it daunhaandsccall
ô, min_val, max_val)
Generate adversarial input using
new (0x3) and via (eq. (7)) scaled to mmin_val, max__al]. */
6:
dist - tpips( xx xadv)
ie Calculate perceptual distance. */
7: 40 D(rppeat((dde n))
I* Get purified outputs. #/
8: logits - M M(#0 2
1e Compute model output. */
9: if Cond(logits, y) and dist < then
return Fadv
/* Success. Return Bade- #/
11: end if
12:
Objective (loggits, + max Tp: 0) (eneeiieebccwarr((( if mod eot steps then Optim.step()
* Compute loss (eq. (7J). */
13:
|* Get gradients for parameters. #/
15:
oe Update parameters. #/ 1 Reset gradients. #/
Optim.zero_grade
17:
end if
end for
19: return
1 Failure. Return original æ. */
35

---

To average the gradients over multiple (N) paths as per the adaptive attack's s requirements from $3.1, we generate several purified copies by repeating the sample addo under optimization times before feeding it into the DBP pipeline (line 9). Here, corresponds to the maximum number of copies we can fit into the GPU"s memory during a single run. However, as this n may be smaller than the desired N from $3.1 (i.e., number of Eo'l' samples) that allows us to sufficiently eliminate the error in the computed gradient, we use gradient accumulation by only making updates to the optimizable parameters (and then resetting their gradients) every sot_steps iterations (lines 16-19). By doing so, the effective number of used copies becomes n eot_steps, which can represent any N of choice that is divisible by n. Note that if is not divisor of N, we can always increase N until this condition is met, as a larger N can only enhance the accuracy). This also explains why the algorithm runs for mar mar_iters jiijtters cot steps (line 5).
Finally, the condition Cond captures the threat model (either SP or MV-see $2): When the logits for the batch of n copies are available together with the target label y. Cond outputs a success decision based on whether we seek misclassification for the majority of these purified copies or single copy only. Note that, as explained in $4.2, we take the majority vote over the maximum number of copies we can fit into the GPU (i.e., n) for MV. As this choice was only made for practical considerations, one may desire to experiment with different configurations wherein another number of copies is used. Yet. this is easily achievable by simply modifying Cond accordingly: For instance, we may augment it with history that saves the output logits over all cot eot_ steps (during which zadv is not updated). Then, the majority vote can be taken over all copies in this window. Note that by increasing the number of cot__teee steps, we can use as many copies for the majority vote decision as desired in this case. That said, the attack will become significantly slower.
In addition to the precise gradient module DiffGrad, our DiffBreak toolkit provides the implemen- tation of our LF strategy as well as various other common methods (e.g., AA and StAdv), to enable robust and reliable evaluations of DBP, All strategies are optimized for performance to speed up computations via various techniques such as just-in-time (JI7) compilation. Our code is available at https://8 thub. com http://areeaassssiifffef.
J Example Attack Images
In Appendix J.1-J.10, we provide variety of successful attack images that cause misclassification in the rigorous MV setting, generated using our cow-frequency (LF) strategy against all systems considered in $.. 1. For configurations that were also evaluated against AA-La under the same MV setting (i.e., using the same sample counts N as used for LF in $5.1), we include successful AEs generated with this method for direct comparison. Specifically, for ImageNet, both the LF attack in $5.1 and the AA attack in $4.2 were evaluated under MV with N = 8; hence, we include AA samples for ImageNet directly from $4.2. For CIFAR-1O, the AA experiments in $4.2 use the more permissive N = 10 setting. Therefore, we instead draw samples from the corresponding N = 128 AA experiments reported in Appendix H to ensure a fair comparison under equal majority-vote conditions. Note that all samples are crafted using the parameters listed in $4.2 and 15.1. That is, =p= 0.05 for LF and cxc = 8/255 for AA against CIFAR-10 and 6o = 4/ 4/255 against ImageNet.
For the configurations that were evaluated against both strategies, we provide two sets of samples: 1) Three triplets containing the original image, the AE generated using AA, and the AE crafted using LF. Importantly, all original samples in this set are inputs for which both methods can generate successful AES, and we provide these to allow for a direct comparison between the two strategies' output quality on a sample-by-sample basis. Yet, as AA is inferior to our approach (LFF,, resulting in the systems retaining robustness on many inputs for which it fails to generate successful AEs under MV. it is essential to inspect LF's outputs on such more challenging samples to demonstrate that it still preserves quality despite its ability to fool the target classifiers. Thus, we include a second set of 2) Three successful AEs generated with LF from inputs which AA fails under MV. For the remaining configurations that were not evaluated against AA under the same MV sample counts from 15.1, we provide six successful AES generated with LF.
In Appendix J.11, 11, we present attack images generated by the non-norm-bounde StAdy [47] strategy under MV (with the above sample counts N). This method has demonstrated superior performance to nase techniques the past against DRP [30] even in the absence of the correct exact gradients, indicating it could be viable attack strategy with gradient computation fixes, thereby making
36

---

our LF approach unnecessary. Yet, previous evaluations only considered StAdv against DBP for CIFAR-IO [30]. While we find StAdv capable of defeating all systems (for both CIFAR-IO and ImageNer), it leads to severe quality degradation when used to attack DBP-protected classifiers for high-resolution inputs (i.e., ImageNet), leaving the AĖS no utility. Thus, we deem it unsuitable, excluding it from the main body of the paper accordingly. Further details are in Appendix J.1
All samples below are originally (without adversarial perturbations) correctly classified.
J.1 Attack Samples Generated Against CIFAR-1O's WideResNee-70-11 with GDMP Purification
Figure 3: Successful attacks generated by LF and AA-Ex. Left -original image. Middle AA. Right LF.
Figure 4: Successful LF attacks on inputs for which AA-Ez fails. Left original image. Right LF.
J.2 Attack Samples Generated Against CIFAR-I0's WideResNet-28-110 with GDMP Purification
Note: While WideResNNe-22. 10 was not evaluated against AA under MV with N 128 in Appendix H, we include samples here to complement our CIFAR-10 unalyses-ou our primary benchmark for AA-based evaluation. Among the two main purification paradigms considered (GDMP and DiffPwre), we randomly selected GDMP for this illustrative example.
Figure 5: Successful attacks generated by LF and AA-Loo. Left -original image. Middle AA. Right LF.
Figure 6: Successful LF attacks on inputs for which AA-La, fails. Left original image. Right L.F.
37

---

J.3 Attack Samples Generated Against CIFAR-IO's WideResNet-70-10 with DiffPure Purification
Figure 7: Successful attacks generated by LF and AA-Lo. Left original image. Middle -A. Right
LF.
Figure 8: Successful LF attacks on inputs for which AA-lx fails. Left original image. Right LF.
J.4 Attack Samples Generated Against CIFAR-IO's WideResNet-28-I1 with DiffPure Purification
Figure 9: Successful attacks generated with LF. Left -original image. Right LF.
38

---

J.5
Attack Samples Generated Against ImageNet's DeiT-S with GDMP Purification
TURN Enginering
TURN
Enginering
TURN
rehitecture Techeology Bow
rohitectere Technology User
rchitectere Tochnology User Natwork
Figure 10: Successful attacks generated by LF and AA-los Left -orignnaa image. Middle AA. Right LF.
Figure 11: Successful LF attacks on inputs for which AA-L fails. Left original image. Right LF.
39

---

J.6 Attack Samples Generated Against ImageNet's WideResNee-50-22 with GDMP Purification
Speead
Speedd
C
P
Figure 12: Successful attacks generated with LF. Left -original image. Right LF.
J.7 Attack Samples Generated Against ImageNet's ResNet-50 with GDMP Purification
Figure 13: Successful attacks generated with LF. Left -original image. Right LF.
40

---

J.8
Attack Samples Generated Against ImageNet's DeiT-S with DiffPure Purification
Figure 14: Successful attacks generated by LF and AA-los Left -orginall image. Middle AA. Right LF.
Figure 15: Successful LF attacks on inputs for which AA-L fails. Left original image. Right LF.
41

---

J.9 Attack Samples Generated Against ImageNet's WideResNet-50-2 with DiffPure Purification
SPORTS ALT &HORITY 10
SPORTS AET AUTHORE &HORITY
SPORTS R At &HORITY 10
10
Figure 16: Successful attacks generated by LF and AA-lx- Left -original image. Middle AA. Right
LF.
Figure 17: Successful LF attacks on inputs for which AA-L fails. Left original image. Right LF.
42

---

J.10 Attack Samples Generated Against ImageNet's ResNet-50 with DiffPure Purification
uukatn
uukatos
Figure 18: Successful attacks generated with LF. Left -original image. Right LF.
J.11 Quality Comparison with StAdy
We found StAdv capable of generating outputs that defeat DBP even under MV. However, is not suitable for targeting DBP-defended classifiers that operate on high-resolution images. The reason that StAdy performs spatial transformations that relocate the different pixels. Thus, its changes quickly become visible when applied excessively. Due to the considerable stochasticity of DBP (see $4.2), the required displacements (especially in the MV setting) are significant, which in turn can severely impact the quality. For low-resolution inputs (e.g., CIFAR-10), StAdy still be effective, with the quality degradation remaining unnoticeable due to the size of the images that renders them blurry by default, masking StAdy's 's effects. For high-resolution inputs, the degradation substantial, leaving the outputs useless as stealthiness is key requirement from practical AES [21]. StAdy's 's successfully misclassified samples (under MV) below prove these claims. We use Full-DiffGrad for backpropagation and run StAdy with its default parameters [47]. When the parameters are changed to better retain quality, StAdy ceases to converge for ImageNet, making it of no use. All provided samples originally correctly classified.
43

---

Figure 19: StAdv attacks against CIFAR-10's WideResNet-70-16 with GDMP purification. Left
original image. Right StAdv.
TURN
TURA
Figure 20: StAdy attacks against ImageNet's DeiT-S with DiffPure purification. Left original image. Right StAdv.
44


Reviews:
[{"review_id": "TOZkqmyPDz", "reviewer": "Reviewer_6Kap", "summary": "This paper argues that Diffusion-Based Purification (DBP), a defense against adversarial attacks, is not robust. The authors claim that attacks can manipulate the core diffusion model instead of just tricking the classifier. They introduce a toolkit called DiffBreak to show this, which includes a more accurate way to calculate attack gradients (DiffGrad) and a better evaluation method (Majority-Vote). They also propose a \"low-frequency\" attack that they claim breaks DBP completely.", "questions": "- The LF attack uses LPIPS as a perceptual constraint. Have you experimented with other perceptual metrics, and do you believe the attack's success is specific to LPIPS or would generalize to other perceptual similarity measures?\n- Could the principle of implicitly optimizing the parameters of a pre-processing defense be generalized to other types of defenses that are often treated as fixed, such as JPEG compression or other input transformations? \n- The Majority-Vote protocol is a key contribution. The experiments use K=10 or K=8 samples. How was this number determined? Is there a performance/robustness trade-off curve as K increases, and what would be a practical recommendation for K in a real-world deployment?", "limitations": "The authors admit their methods are very computationally expensive.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper points out that testing a randomized defense like DBP with just one try isn't reliable. Their proposed Majority-Vote (MV) method is a more common-sense approach to measuring robustness.\n- The DiffGrad module seems like a solid piece of engineering. It fixes several technical issues in previous attack implementations, which could help the research community conduct more reliable experiments in the future.\n- The authors test their claims across several standard datasets and models (CIFAR-10, ImageNet), which makes their empirical findings convincing.\n\nWeaknesses:  \n- The paper exhibits weaknesses in its treatment of prior work and internal consistency. It repeatedly characterizes methods like DiffHammer and DiffAttack as having \"flawed\" gradients, overlooking that approaches such as surrogate approximations represent intentional computational trade-offs rather than fundamental errors. For example, DiffHammer’s checkpointing balances memory efficiency against gradient precision, a reasonable engineering compromise.\n- The paper first evaluates DBP against standard, norm-bounded attacks and shows it still has some robustness. Then, it introduces a completely different type of attack (the \"LF attack\") which is not norm-bounded and uses it to declare that DBP is broken. This seems like shifting the evaluation criteria.", "was_revised": true, "final_justification": "Thank you for the detailed and constructive rebuttal. Your responses have clarified several of my primary concerns and have convinced me to raise my score.", "rebuttal": "We’re thankful for the thoughtful feedback and glad the reviewer found our engineering, MV protocol, and empirical results convincing.\n# Q1. Clarifying “flawed gradients”\nThank you for the helpful comment. We agree surrogate approximations and checkpointing are legitimate trade-offs. By “flawed gradients,” we mean robustness overstatements due to ***(i) implementation variants that may affect gradient fidelity or (ii) auxiliary objectives that don’t improve performance once gradients are accurate.*** We appreciate the valuable observation, and we’ll adopt more neutral wording in the paper and make this distinction explicit. We provide details below:\n## DiffHammer [41] (implementation divergence)\nWe have no issue with checkpointing (also used in our DiffGrad). Yet, as noted in App. D.1.5, DiffHammer combines it with Lee & Kim’s [24] surrogate process (see DiffHammer’s App. C.1.1), but DiffHammer’s code diverges from [24], introducing gradient deviations. Re-running the standard AutoAttack on CIFAR‑10 with the surrogate implementation that matches [24] drops worst-case robustness (N=10) from DiffHammer’s reported 33.79% for the same attack (their Table 1) to 14.45%—even below the DiffHammer attack (reported 22.66%)—confirming the inflation came from gradient fidelity, not compute trade-offs.\n### On the surrogate method [24]\nIt shortens the trajectory and backprops through this shorter horizon but retains the full graph over that path. In DBP, memory—not FLOPs—is the bottleneck; Checkpointing (our DiffGrad, Liu et al. [26]) recomputes individual path segments, yielding exact gradients with lower peak memory. The surrogate may improve speed but not memory pressure, and its truncation can miss dependencies, weakening gradients. Empirically, checkpointed full-gradient attacks are stronger (App. G). Attackers precompute adversarial examples offline (DBP’s latency makes real-time adversarial generation infeasible either way), so runtime savings alone don’t translate to a practical advantage.\n## DiffAttack [20] (design choice)\nDiffAttack adds per-step losses to AutoAttack, increasing overhead. It also originally used the adjoint method [25] instead of standard backprop for checkpointing (see App.G), which is similar in cost but known to yield weaker attacks [24]. With our full gradients (DiffGrad), standard AutoAttack matches (CIFAR-10) or outperforms (ImageNet) DiffAttack without extra losses (see S4). We’ll clarify these points in S4.2.\n\n***Summary.*** Our concern is limited to implementations and additions that don’t improve attacks once gradients are accurate. We'll follow the reviewer’s suggestion and better characterize these previous works to avoid confusion and enhance clarity.\n# Q2. Does LF shift the evaluation criteria?\nThanks for the valuable question. ​​Our aim is to evaluate DBP practically—not just within an ℓ‑norm ball. As an empirical defense that must resist various practical threats, DBP isn’t tied to a specific norm, and prior work includes non-norm-bounded attacks (e.g., StAdv [47] in DiffPure [30], Chen et al. (2024)) alongside ℓ-bounded ones. We follow this precedent, proceeding in two steps:\n## 1.Norm-bounded (S4.2)\nWe first isolate the effect of accurate gradients. While evaluations often include non-norm attacks, recent attack papers focus on enhancing the norm-bounded AutoAttack. Our comparison with these works validates our theory: with accurate gradients, standard (i.e., gradient-based) attacks can alter DBP (S3.1) without extra surrogates (DiffHammer/DiffAttack). This allows us to focus on standard gradient-based approaches as strong DBP attacks when developing an effective strategy later as well.\n## 2.Imperceptible, non-norm-bounded (S5)\nWe then ask the broader question: How robust is DBP overall? Though robustness drops under ℓ-bounded attacks, some remains—motivating further evaluation for a thorough assessment. We enforce imperceptibility via an LPIPS threshold akin to an ℓ-budget, but that also permits low-frequency perturbations via our optimizable filters. Under MV, StAdv (the standard non-norm baseline) fails at high resolutions (S5/App.J); Our LF attack is the first to reliably break this sound DBP setup, showing the key factor is a stronger, DBP-tailored attack—not relaxed constraints.\n\n***Summary.*** We didn’t shift criteria: we validated our theory under standard ℓ-bounds, then assessed general robustness in an established imperceptible, non-norm setting—***both under MV***. LF shows DBP fails against stealthy attacks, the core practical concern—despite partial robustness to ℓ-bounded attacks, which we acknowledge.\n\n*Chen et al. Robust classification via a single diffusion model. ICML, 2024*\n# Q3. Perceptual metrics other than LPIPS for LF\nLPIPS is solely a perceptual constraint while LF’s success comes from its structured perturbations (see S5). We chose LPIPS since it’s the standard metric, with user validation and published thresholds enabling a clear, auditable similarity bound (see S5.1). LF isn’t tied to LPIPS; other metrics could work if they have validated thresholds to enforce similarity. Our aim isn’t to benchmark metrics, but to show LF breaks DBP under a proper perceptual constraint. App.J’s qualitative examples and the chosen LPIPS threshold confirm imperceptibility.\n# Q4. Does the principle of implicitly optimizing the pre-processing defense generalize beyond DBP?\nThanks for the great question. It depends on the pre-processor. Our findings apply when the defense has ***input-dependent, differentiable parameters***—e.g., ML-based purifiers like the score model in DBP or other learned purifiers. ***They don’t apply similarly to fixed, non-adaptive transforms.***\n\nFor instance, in DBP, if the exact score $\\nabla_{\\hat{x}(t)}\\log p_t(\\hat{x}(t))$ were available (as in the idealized analysis by Xiao et al. [48] brought in S3.1), the purification function would be fixed (predetermined). Here, the attacker cannot “optimize the defense,” and the robustness argument from S3.1 would hold even against adaptive attacks, leaving DBP highly robust. In practice, a perfect score oracle is unavailable and replaced by an ML model, whose parameters can be exploited—causing deviation from the ideal behavior (see S3.1)—precisely what we prove in S3.1 and verify empirically: with accurate gradients, DBP can be driven to generate adversarial—not natural—distributions. This vulnerability extends to all learned purifiers; the significance of our work lies in showing that it holds even for stochastic, iterative processes, forcing them to output adversarial ***distributions***. This is more significant than a similar finding for deterministic, parameterized defenses that yield a single output.\n\nA fixed pre-processor $h$ (e.g., JPEG or blur) has no input-controllable parameters. Attackers can’t change the defense’s fixed functionality $h$, but only optimize $x$ s.t. $h(x)$ ***(where h is fixed)*** lands in the classifier’s vulnerable region. Conceptually, this is ***evasion***, which yields weaker attacks (e.g., if a perfect score existed, attacks would fail against DBP—see above). Yet, common fixed transforms (unlike a perfect score oracle), such as JPEG, can’t even resist the weaker evasion attacks, as repeatedly shown in prior work.\n\n***Takeaway.*** Implicit optimization applies to input-adaptive (learned) pre-processors. For fixed ones, attackers optimize the transform’s output—not the transform. Our results show that even stochastic, iterative purifiers can be implicitly optimized.\n# Q5. Determining the MV sample count K\nWe chose K=10 for CIFAR-10 to match prior work’s worst-case evaluations (see S4.2). Ablations with K=1→10→128 in App.I (referenced in S4.2) show MV.Rob rises with K (DiffPure: 35.16%→39.45%→47.72%; GDMP: 8.59%→16.80%→32.81%), while Wor.Rob drops significantly under repeated resubmissions (larger K). This supports MV’s role (S3.3) in reducing variance. On a 40 GB A100 GPU, inference takes ~5.29s (K=1), ~6.54s (K=10), and ~26.53s (K=128). Batching the K samples, thus, makes K=10 nearly cost-free; K=128 adds ~21s per step. We stop at K=128 due to memory limits (largest feasible batch size)—control experiments with K=256 (split batches) showed negligible robustness gains, indicating saturation. App.I recommends K≈10 as a practical default and K=64–128 for security-critical use. For LF (S5.1), we use K=128 to evaluate our attack in the most robust MV setup.\n\nAs in [41], we restrict ablations to CIFAR-10 for feasibility. For ImageNet, we use K=8 due to similar memory limits and diminishing MV returns beyond that. Inference rises from 7s (K=1) to 25s (K=8), with batching offering ~2.2× speedup (56s→25s). Yet, larger models and data size make batching less effective than in CIFAR-10. We recommend K≈8 unless higher values yield clear MV gains. This will be added to App.I.\n# Q6. Computational Cost\n***We agree the experiments are compute-heavy—but the cost stems from DBP itself, not our attacks. Our DiffGrad matches prior work in attack-side cost and avoids memory bottlenecks.***\n\nDiffGrad uses checkpointing as in [20,26] but fixes backprop issues (S3.2, App. D) without extra cost. Its complexity matches the adjoint method [25] but is more efficient than DiffAttack [20], which adds per-step loss overhead. DiffHammer [41] also adds post-hoc computations. We further accelerate attacks by EOT batching (see S3.2). Backprop-free methods (BPDA) are faster but far less effective (see Table 1). The surrogate [24] improves speed only, but doesn't improve practicality (Q1).\n## LF overhead\nThe small LF filter network updates only once per backprop, adding negligible overhead.\n## MV protocol\nMV is costlier than single-purification, but isn’t an optional add‑on—it’s the correct deployment for this stochastic defense (see S3.3). It reduces variance-driven errors, making it essential. Still, its cost is offset by batching and choosing K based on the MV gain-latency trade-off (see Q5)."}, {"review_id": "qmdAK1cHT3", "reviewer": "Reviewer_4tho", "summary": "This paper challenges the robustness of diffusion-based purification (DBP) methods. This paper nullifies DBP’s core premise by proving that standard gradient-based attacks can manipulate the diffusion model’s score function to generate adversarial distributions rather than natural ones. To address these issues, this paper proposes DiffGrad, which is a reliable backpropagation toolkit for DBP that fixes gradient inaccuracies in prior works. Experiments show that DBP’s robustness drops significantly when using DiffGrad to calculate the gradients.", "questions": "Regarding the future of adversarial purification, do you view diffusion models as a promising direction if further refined, or would you recommend exploring fundamentally different architectures beyond diffusion models? Looking forward, what key innovations would be needed to make diffusion models robust against adversarial attacks? Any discussion is appreciated.", "limitations": "Yes, the authors provide a detailed discussion of limitations and broader impacts in Appendix A.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n- This paper tackles a very important & interesting problem in DBP methods. Evaluation is a long-standing problem for DBP methods. Due to the computational cost of accessing Diffusion model’s gradient, many methods rely on inappropriate approximations of the gradients, which in general may lead to the overestimated robustness of DBP methods.\n- This paper is well-written and theoretically sound. The theoretical analysis proves DBP’s vulnerability to adaptive attacks.\n- Experiments are solid and sufficient.  The robustness of DBP methods drops significantly after applying DiffGrad.\n- The observation that the low-frequency AEs can beat both DBP and AT methods is interesting and surprising, especially for DBP methods since they do not implicitly make  assumptions on the types of incoming AEs.\n\n**Weaknesses**:\n\n- Experiments use a small test set (256 samples per dataset), which is not sufficient, although the setting aligns with previous work.\n- The core insight that attacks can exploit DBP’s stochasticity aligns with previous work (i.e., DiffHammer), though this work goes further with low-frequency attack.", "was_revised": true, "final_justification": "All my concerns are well-addressed, so I increase my confidence score accordingly. I initially gave a very positive score (see strengths for why) and I will keep my positive score towards accepting this paper.", "rebuttal": "We thank the reviewer for their thoughtful and supportive evaluation, and for recognizing the paper’s theoretical soundness, empirical strength, and the significance of our findings.\n# Q1. Evaluation set size\nThank you for the observation and for noting the alignment with prior work. We agree the per-dataset evaluation set is small. However, as noted by the reviewer, this follows standard DBP practice (e.g., DiffHammer [41]; Chen et al., 2024) because DBP is a compute-intensive defense to evaluate. Additionally, we emphasized breadth over scale: two datasets (CIFAR-10, ImageNet), multiple classifiers and DBP schemes, diverse attacks/baselines, and MV ablations (see App.I). The same qualitative conclusions hold across these settings (see S4-S6, extended results in App.G–App.I), which we believe is more informative for assessing DBP than enlarging a single test set. We will make this trade-off explicit in S4 and App.F.\n\n*Chen et al. Robust classification via a single diffusion model. ICML, 2024*\n# Q2. Does the core insight that attacks can exploit DBP’s stochasticity align with DiffHammer?\nThank you for emphasizing the significance of our contribution and low-frequency attack. While both works analyze DBP’s stochasticity under attacks, the conclusions differ. DiffHammer posits that DBP’s purified trajectories partition into two groups—those sharing adversarial vulnerabilities and those that do not—and that gradients from the latter can impede convergence (“misleading gradients”). The attack proposed in DiffHammer [41] builds on this assumption. Our analysis (see S3.1, Thm 3.1) shows that, under correct differentiation, an adaptive attack can shape the purification distribution, driving the score‑driven dynamics so that the terminal sample $\\hat{x}(0)$ is drawn from an adversarial rather than a natural distribution. As such, our analysis indicates that DBP’s stochasticity is not an intrinsic obstacle but a process that can be steered when gradients are computed accurately. Accordingly, this analysis does not support a fixed partition of DBP’s output space as assumed by DiffHammer; when the attack manipulates the purifier, it can alter the output distribution.\n\nEmpirically, under identical settings and with DiffGrad’s accurate gradients, gradient-based attacks strengthen (both the standard AutoAttack and DiffHammer), with AutoAttack matching or outperforming DiffHammer (see S4.2) and confirming our theory.\n# Q3. Do you view DBP as a promising direction? What refinements are needed?\nThanks for the question. DBP remains a promising defense as diffusion purifiers can improve clean accuracy compared to prior purification methods. However, as a standalone defense, Theorem 3.1 shows DBP is vulnerable to adaptive attacks ***when accurate gradients are available***. We are therefore actively exploring a direction in which the same purifier is configured to follow a different, private SDE at deployment than the standard public SDE that can be used for attack optimization. Thus, the inference‑time purification dynamics differ from the public dynamics an attacker optimizes against. The private dynamics are not exposed under our threat model and are intended to be difficult to infer from model I/O alone. Under this design, attack‑optimized gradients need not transfer to the behavior used at inference—without resorting to gradient masking (a strategy repeatedly shown vulnerable in previous work). This circumvents the setting analyzed by Theorem 3.1 rather than contradicting it. Initial results are promising, and we will report a full treatment in a separate work."}, {"review_id": "IdgsIbFD2J", "reviewer": "Reviewer_7gyq", "summary": "The paper critically evaluates the popular diffusion-based purification (DBP) defense against adversarial examples and highlights the flaws in the existing approaches. The paper uses adaptive attacks to show that the robustness claimed by prior works is inflated due to the reliance of these defenses on flawed gradients. The paper then proposes a toolkit, DiffBreak, to eliminate the gradient flaws and curb the inflated robustness estimates. The paper also proposes a low-frequency attack inspired by [21] that further degrades the performance of the DBPs.", "questions": "See above", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper highlights a new vulnerability of existing DBP-based defenses, as well as previous methods such as DiffusionHammer, which further reduce the robustness performance of DBP-based defenses. In particular, the paper’s insight that the use of accurate gradients for generating adversarial examples ends up using the purification process to the attacker’s advantage is critical.\n2. The paper proposed a toolkit that could be used by model builders to evaluate their models’ robustness to adversarial examples.\n3. The low-frequency adversarial examples while not novel, have been shown to be effective at bringing down the robustness of various models.\n4. The paper thoroughly evaluates various methods, including some recently proposed defenses, and highlights that their robustness is also flawed.\n\n\nWeaknesses/Questions\n\n1. The proposed Majority Voting (MV) based metric is weaker than the worst-case robustness metric proposed by [41]. What is the purpose of this new metric?\n2. The evaluation consists of very low-resolution data. Do the findings transfer to much larger image sizes?\n3. What is the computational complexity for computing an attack against DBP for higher resolution images?\n4. Are the proposed findings transferable to different modalities of data or limited to just images?", "was_revised": true, "final_justification": "My concerns have been addressed. I maintain my rating.", "rebuttal": "We thank the reviewer for recognizing the paper’s core insights and contributions. We appreciate the thoughtful questions, which helped us clarify key aspects of the work.\n# Q1a. What is the purpose of the new Majority Voting (MV) metric?\nThanks for the question. Our goal is to measure DBP’s ***true*** robustness. To achieve this, we must define a realistic operational scenario reflecting how it should be deployed by a careful defender, rather than rely on a simplified protocol that is fragile to variance. Prior evaluations made two questionable choices:\n\n***(1) Single‑run defense.*** Each input was assumed to be purified and classified once to obtain the label, so random diffusion variance alone could cause misclassification. \n\n***(2) Single‑path attack.*** Adversarial examples were evaluated on a single purification path alone (upon attack termination), ignoring repeated‑query risk and inflating robustness scores.\n\nDiffHammer’s “worst‑case” metric [41] addresses (2) by letting the attacker resubmit the same sample N times (each purified along a new path), but still leaves issue (1): the defender also purifies each submission only once. An attack, therefore, “succeeds” if any of the N purifications errs, so the robustness estimate is dominated by stochastic noise. Yet, Lucas et al. [27] show that such one‑shot stochastic defense deployments are breakable with enough queries.\n\nMajority Vote (MV) resolves both problems. It models a realistic and more robust (responsible) defender that purifies each input N times in parallel and returns the ***majority label***, reducing variance-driven errors. MV suppresses variance while retaining manageable cost; it can be seen as a lightweight analogue of randomized smoothing—achieving practical stability with N~8−10 samples rather than thousands, albeit without a formal certificate, which is acceptable because many realistic attacks (***ℓ∞​, LF, StAdv***) are beyond current certification theory. MV is thus the simplest statistically sound deployment, essential for isolating genuine vulnerabilities from incidental variance-driven failures. Otherwise, any robustness numbers—ours or those of previous studies—are entangled with avoidable sampling noise, obscuring whether failures stem from DBP itself or an unrealistically fragile deployment. Since MV inherently aggregates predictions over multiple copies, the threat of resubmissions is readily accounted for as well.\n\nWe therefore adopt MV as the primary metric. In App.I, we conduct ablation studies to explore the effect of the set size N on robustness and efficiency, and highlight best practices and recommendations for future work.\n# Q1b. MV robustness appears weaker than the worst-case robustness proposed by DiffHammer [41]\nFor the same N, MV is strictly harder for the attacker as it takes the ***majority label***: success now requires misclassifying > ⌊N/2⌋ of the N purified copies, whereas DiffHammer’s [41] worst-case counts a single error. ***Hence, MV​ is an upper bound on worst-case robustness​. This aligns with our empirical findings.***\n\nTables 1–2 show that, when gradients are computed with DiffGrad, worst-case robustness (Wor.Rob) is always no greater than MV robustness (MV.Rob) for all attacks we evaluate. ***Apparent violations arise only when our MV.Rob results are compared to the Wor.Rob values originally reported by DiffHammer [41]***. These inversions reflect implementation differences in gradient computation. As discussed in S3.2, DiffGrad removes back-prop mismatches that can reduce gradient fidelity and inflate reported robustness (for both Wor.Rob and MV.Rob). Thus, MV-versus-worst-case comparisons should be made using a common, DiffGrad-based implementation of each attack (note that DiffHammer doesn’t report MV.Rob originally). *Note: Wor.Rob ≤ MV.Rob assumes the same inputs and the same set of N paths. Our MV vs. DiffHammer’s Wor.Rob comparisons use different samples/paths, so small sampling noise may appear; the main driver is still the gradient-computation differences.* We include the original Wor.Rob numbers from [41] for completeness and to illustrate the impact of gradient fidelity (see S3.2), not as cross-implementation evidence about MV versus worst-case. *We will also correct misleading boldface in Table 1 in an updated version.*\n\nTable 3 shows MV robustness near zero under a ***different attack*** (our ***LF attack***)—by design, this stronger attack defeats DBP even under MV, so the corresponding worst‑case numbers are, by definition, no higher and are omitted.\n# Q2. Evaluation is based on low-resolution data. Do the findings transfer to larger image sizes?\nThanks for the valuable question. Tables 2–3 already evaluate three ImageNet classifiers at 224×224 (upsampled to 256×256), the standard resolution in prior DBP work [20, 30, 40], so our results are not limited to CIFAR-10; they also apply to more practical, real-world resolutions. DBP is a computationally intensive defense, so the prior DBP work (e.g., DiffPure [30], GDMP [40], DiffAttack [20])—and our own—focuses on CIFAR‑10 and ImageNet, as for much larger image sizes, score model capacity and memory/latency grow accordingly, making large-scale studies impractical. Still, we consider multiple classifiers, DBP schemes, attack settings, and majority vote (MV) sample counts to ensure breadth and robustness of conclusions.\n\nBoth the mechanism and our evidence support that the findings transfer beyond small images. Theorem 3.1 is dimension-agnostic (accurate differentiation lets an attacker steer the purification distribution regardless of resolution), the gradient mismatches we address in S3.2 (DiffGrad) are implementation-level and apply at any size, and MV (S3.3) addresses variance in randomized purifiers, which is resolution-independent. For LF (see S5), we optimize small filter networks to induce structured, low‑frequency perturbations under a fixed LPIPS bound—neither the construction nor the constraint assumes small images. We already show stealthy, effective LF adversarials on ImageNet-resolution models (see S5.1, App.J); as resolution grows, these perturbations can be distributed over more pixels while remaining within the LPIPS threshold. Related evidence from UnMarker [21] (see S5)—though targeting watermark removal, not DBP, via different optimization objectives—demonstrates imperceptible low‑frequency edits at 512×512.\n\nIn short, our results already cover ImageNet‑resolution DBP and include extensive MV/EOT evaluations. Exploring ultra‑high‑resolution regimes (≫256²) is valuable future work, but is orthogonal to the core mechanisms we study and to prevailing evaluation practice for DBP.\n# Q3. What is the computational attack complexity for higher resolution images?\nAs mentioned in App.A (Broader Impact & Limitations), we provide an asymptotic analysis in App.D.1.3 with details in App.D.3. ***Bottom line:*** The bottleneck is DBP itself (long reverse diffusion and higher-capacity score models at larger resolutions). Our implementation keeps peak memory near the per-timestep cost and reuses DBP compute; ***if DBP is practical at a given resolution, the attack is practical too. That is, whenever the dimensionality allows for training score models and using DBP as a defense, running attacks is also feasible.*** Compared to a non-checkpointed attack implementation (which is memory-exhaustive), we roughly only require an additional forward propagation in total in terms of latency. Further details are below:\n\nAs in prior work (DiffAttack [20], Liu et al. [26]), the attack scales with the DBP pipeline. With our checkpointing-based DiffGrad, the computation graph is kept for one diffusion step (t) at a time (otherwise attacks become infeasible due to memory overhead [40]): during forward propagation, memory is essentially that of a single score-model invocation to compute $\\hat{x}(t)\\to\\hat{x}(t{+}\\mathrm{d}t)$; internal activations are discarded after the step, and we retain only the lightweight sequence $\\\\{\\hat{x}(t)\\\\}$ (which can be offloaded to CPU). During backprop, we re-invoke the score model on each saved $\\hat{x}(t)$, build the graph for that step only, take gradients, then drop the graph before moving to the next step. This avoids graph accumulation across the trajectory, so peak memory is dominated by the per-step score model activations (and backprop) cost (scaled by the number of batched EOT/MV samples N), not by the number of diffusion steps. Wall-clock time is proportional to the per-step model latency × number of steps; checkpointing does not introduce extra multiplicative overheads. For our LF attack (see S5), we add a small LF filter network that updates only once per backprop, adding negligible overhead.\n\nConcretely, at ImageNet scale and on a 40 GB A100 GPU, we observe DBP inference times of ~7s (N=1) and ~25s (N=8), compared to attack iteration times (forward+backprop) of ~16s (N=1) and ~75s (N=8). CIFAR-10 timings are reported in App.I; we will add these ImageNet timings there and cross-reference them in S4.2.\n# Q4. Are the proposed findings transferable to different data modalities?\nThanks for the interesting question. Our experiments target vision because DBP has so far been studied primarily on images, allowing direct comparison to prior work. Nevertheless, both Theorem 3.1 and the DiffGrad backprop module are modality‑agnostic: they depend only on the diffusion dynamics and implementation, not on pixel structure. We therefore expect the same vulnerabilities for, e.g., speech or video purification. Furthermore, all the arguments in support of the LF attack, including its efficacy stemming from its low-frequency nature, also hold across various domains (i.e., audio and video). However, the specific implementation that relies on spectral filter networks would require domain-based adaptations (e.g., replacing 2D Fourier filters with their 1D counterparts for audio) to apply it to these different tasks."}, {"review_id": "O1FdQLtUAU", "reviewer": "Reviewer_Q9d3", "summary": "This paper attempts to break diffusion based adversarial defenses empirically with (1) improved implementation details (2) additional techniques like LF. The authors present siginificant vulnerabilities in the DBP methods.", "questions": "Please see weakness.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength\n1. I believe this paper conveys a very strong and impactful message to the community by exposing the vulnerability of diffusion based defenses. DBP methods have been widely studied and are considered promising given their effciency compared to AT. However, if all the previous success should be attributed to improper and inadaquate evaluation protocal, then it is of great importance to convey this to the reserach community.\n2. The technical insights and implementation details are solid. \n3. The introduction of LF methods is novel and exhibits high effectiveness.\n\nWeakness\nGenerally, there is no major technical flaws in this paper. There are two minor weakness from my standpoint.\n1. I personally do not favor discussing much about Theorem 3.1, since both the theoretical result and the proof process are relatively intuitive. Clearly stating the reasons why the certifiable robustness of DBP might not realizable in practise is already a meaningful complement to related theoretical works.\n2. The \"Reproducing Stochasticity\" assumption is valid for worse case evaluation. However, this might be a assumption too strong to hold in practise.", "was_revised": true, "final_justification": "This is an impactful paper to the community that studies diffusion based purification defenses. The vulnerabilities uncovered by the authors are critical and the technical contributions are solid.", "rebuttal": "We thank the reviewer for the encouraging and insightful feedback, and we’re glad the contributions and message of the paper resonated.\n# Q1.Emphasize practical reasons why DBP’s certified robustness may fail in practice, rather than detailing Theorem 3.1.\nThank you for the valuable suggestion. We agree the proof does not rely on heavy machinery. Theorem 3.1 is included to settle a core premise in the DBP literature. The result—that, with accurate gradients, the adaptive attack can drive the purification pipeline so that $\\hat{x}(0)$ is drawn from an adversarial rather than the natural distribution—while intuitive in hindsight, runs counter to the working premise implicitly adopted in prior attack designs (S3.1): for example, DiffHammer [41] treats stochasticity chiefly as an obstacle rather than an exploitable mechanism (see our response to ***Reviewer 4tho’s Q2***), and DiffAttack [20] introduces per‑step surrogate losses under the implicit assumption that the purifier isn’t directly steered by standard gradient-based attacks. Our experiments (see S4) show that, once gradients are corrected, standard adaptive attacks realize the steering predicted by the theorem, meeting or surpassing these approaches. \n\nStating the theorem, thus, avoids overestimation or misattribution of robustness, and sets a clear methodological baseline so that future work can develop sound defenses and evaluate them with principled attacks—i.e., accurate, end‑to‑end gradients without unnecessary augmentations. In fact, in our response to ***Reviewer 4tho’s Q3***, we already explain how we’re currently designing an adversarial defense that aims for robustness by building on the conclusions of Theorem 3.1. To our knowledge, we provide the first formal treatment showing that this “purifier steering” extends beyond deterministic, one‑shot preprocessors to long, stochastic, iterative purifiers, namely, diffusion‑based pipelines—i.e., the stochastic distribution over purification outcomes can itself be altered under accurate differentiation.\n\nThat said, if the presentation feels dense, we are happy to streamline the main text—foreground practical implications (why certification may not be realizable for DBP under practical conditions), keep a short intuition in S3.1, and move derivational details to the appendix.\n# Q2. \"Reproducing Stochasticity\" assumption might be too strong in practice\nAgreed—thanks for pointing this out. As noted in App. D.1.3, failing to reproduce stochasticity is typically less severe than the other backprop concerns we address. In vanilla (non-guided) DBP, non‑deterministic cuDNN kernel choices induce only minute gradient discrepancies (raw ≈ 1e‑4, relative ≈ 7.3×10⁻⁵), so this factor is often secondary. However, stochasticity reproduction can be crucial in certain guided schemes: when the guidance function depends on randomness, gradients become tied to the specific noise realizations, and not preserving them can bias gradients and derail attack optimization. The impact is guidance‑dependent and can be substantial, especially for future DBP variants that may rely on stochasticity-dependent guidance.\n\nTo preclude such confounders, DiffBreak/DiffGrad includes a structured Noise Sampler that records and reuses noise during backprop—alongside our other fixes—to eliminate all gradient miscalculations as a source of over‑estimated robustness. Our goal is a robust, generalizable reference implementation for differentiating through DBP. We believe that providing such a reliable tool to the community is crucial for future research in the field, given the current struggle to implement backprop through DBP properly.\n\nWe agree that the main text is brief; we will add a short summary in S3.2 and point to App. D.1.3 for details.\n# Q3. Figure 1 is overly stretched horizontally\nThank you for the suggestion; we’ll correct Figure 1’s aspect ratio and switch to a vector graphic to avoid stretching in an updated version."}]

Author rebuttal:

Meta-review: The present work is a study on diffusion-based purification (DBP). This study challenges the popular view that DBP is robust thanks to the denoising mechanism of diffusion models. This work identifies several reasons that make existing adversarial attack methods ineffective. Moreover, it proposes a new attack method against DBP. The reviewers agree that this paper makes an interesting contribution and is well written. The majority of the weaknesses pointed out by the reviewers have been properly addressed in the rebuttal.
Decision: Accept (poster)