Towards Understanding the Robustness of Diffusion-Based Purification: A Stochastic Perspective
Authors: Yiming Liu
Kezhao Liu
Yao Xiao
ZiYi Dong
Xiaogang Xu
Pengxu Wei
Liang Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Adversarial Defense
Adversarial Purification
Diffusion Training
Randomized Defense
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We identify stochasticity as the primary factor of the adversarial robustness of diffusion-based purification. We further propose a novel method, ADDT, to improve the purification ability of diffusion models.

Abstract: Diffusion-Based Purification (DBP) has emerged as an effective defense mechanism against adversarial attacks. The success of DBP is often attributed to the forward diffusion process, which reduces the distribution gap between clean and adversarial images by adding Gaussian noise. Although this explanation is theoretically grounded, the precise contribution of this process to robustness remains unclear. In this paper, through a systematic investigation, we propose that the intrinsic stochasticity in the DBP procedure is the primary factor driving robustness. To explore this hypothesis, we introduce a novel Deterministic White-Box (DW-box) evaluation protocol to assess robustness in the absence of stochasticity, and analyze attack trajectories and loss landscapes. Our results suggest that DBP models primarily leverage stochasticity to evade effective attack directions, and that their ability to purify adversarial perturbations can be weak. To further enhance the robustness of DBP models, we propose Adversarial Denoising Diffusion Training (ADDT), which incorporates classifier-guided adversarial perturbations into diffusion training, thereby strengthening the models' ability to purify adversarial perturbations. Additionally, we propose Rank-Based Gaussian Mapping (RBGM) to improve the compatibility of perturbations with diffusion models. Experimental results validate the effectiveness of ADDT. In conclusion, our study suggests that future research on DBP can benefit from the perspective of decoupling stochasticity-based and purification-based robustness.

Published as a conference paper at ICLR 2025
TOWARDS UNDERSTANDING THE ROBUSTNESS OF DIFFUSION-BASED PURIFICATION: A STOCHASTIC PERSPECTIVE
Yiming Liu Kezhao Liu";, Yao Xiao', Ziyi Dong', Xiaogang Xu², Pengxu Wei Liang Lin Sun Yat-Sen University, "Chinese University of Hong Kong, *engg Cheng Laboratory f1iuym225, 1iukzh9, xiaoy39, dongzy6 amail2.... sas.ii.ssssu...... sysu. edu. cn
xi kiaogangxuu00@mall (emgaaie00000mmmmmmmmmmmmmma.a com, weipx3 sysu. edu. en, linliang@ieee.org
ABSTRACT
Diffusion-Bann Purification (DBP) has emerged as an effective defense mech- anism against adversarial attacks. The success of DBP is often attributed to the forward diffusion process, which reduces the distribution gap between clean and adversarial images by adding Gaussian noise. While this explanation is theoret- ically sound, the exact role of this mechanism in enhancing robustness remains unclear. In this paper, through empirical analysis, we propose that the intrinsic stochasticity in the DBP process is the primary factor driving robustness. To test this hypothesis, introduce novel Deterministic White-Ban Box (DW-box) setting assess robustness in the absence of stochasticity, and we analyze attack trajectories and Ioss landscapes. Our results suggest that DBP models primarily rely on stochas- ticity to avoid effective attack directions, while their ability to purify adversarial perturbations may be limited. To further enhance the robustness of DBP models, we propose Adversarial Denoising Diffusion Training (ADDT), which incorpo- rates classifier-gun guided adversarial perturbations into the diffusion training process, thereby strengthening the models' ability purify adversarial perturbations. Ad- ditionally, we propose Rank-Based Gaussian Mapping (RBGM) to improve the compatibility of perturbations with diffusion models. Experimental results validate the effectiveness of ADDT. conclusion, our study suggests that future research DBP can benefit from clearer distinction between stochasticity-drivs and purification-drit -rrive robustness.
INTRODUCTION
Deep learning has achieved remarkable success across var- ious domains, including computer vision (He et al. (2166), natural language processing (OpenAl 2023), and speech recognition (Radford et al. 2023). However, within this flourishing landscape, the persistent specter of adversarial attacks casts shadow over the reliability of these neural models. For vision models, adversarial attacks typically involve introducing imperceptible perturbations into input images, tricking the models into producing incorrect outputs with high confidence (Goodfellow et al. 2015 (Szgedy et al. 2014). This vulnerability has spurred substantial research into adversarial defenses (Zhang et al., 2019 Samangouei al. 2018 Shatahi et al. 2019 Wang al. 2023).
Loss
DW-box
White-hox-Eal
White-box
30
Figure 1: Comparison of attack trajec- tories under different evaluation set- tings. The attack trajectory in the stan- dard white-box setting significantly de- viates from the DW-box trajectory and demonstrates lower effectiveness.
Recently, diffusion-based purification (DBP) (Nie et al., 2022) has emerged
promising defense against adver- sarial attacks. Existing studies suggest that DBP robustness primarily stems from the forward diffusion process, which reduces the distribution gap between clean and adversarial images by applying Gaussian noise (Nie et al.) 2022 Wang et al.) 2022). While this reduction is theoretically supported, its contribution DBP robustness has not been sufficiently
Equal contribution, *Corresponding author.

---

Published as a conference paper at ICLR 2025
validated through empirical studies. Additionally, experimental results suggest that the stochastic nature of DBP might also play a significant role in enhancing its robustness (Nie et al. (022).
In this paper, we present new perspective that emphasizes the role of stochasticity throughout the DBP process as a key contributor to its robustness, challenging the traditional focus on the forward diffusion process. To assess the impact of stochasticity, (DW-box) attack setting, in which the attacker has complete knowledge of both the model parameters and the stochastic elements. Our findings reveal that DBP models experience significant loss of robustness when the process is made entirely deterministic from the attacker's perspective. Further analysis of attack trajectories and the loss landscape shows that DBP models do not defend against adversarial perturbations by relying on a flat loss landscape, as is common in adversarial training (AT) (Madry et al. 2018); instead, they leverage stochasticity to bypass the most effective attack directions, as illustrated in Figure]]
propose Deterministic White-box
Building on this new perspective of DBP robustness, we hypothesize that strengthening the diffusion model's ability to purify adversarial perturbations could further improve the robustness. To test this hypothesis, we propose Adversarial Denoising Diffusion Training (ADDT) for DBP models. This method follows an iterative two-step process: first, the Classifie--Guided Perturbation Optimization (CGPO) step generates adversarial perturbations; then, the diffusion model training step updates the parameters of the diffusion model using these perturbations. To better integrate these perturbations into the diffusion framework, we propose Rank-Based Gaussian Mapping (RBGM). which adjusts the adversarial perturbations to more closely resemble Gaussian noise, in line with the theoretical foundation of diffusion models. Experiments confirm that ADDT consistently enhances the robustness of DBP models. Through further empirical analysis and discussion, we argue that future research on DBP should separate the robustness derived from stochasticity and that achieved through purification. This distinction points to two complementary directions for improving DBP: (1) enhancing its ability to purify adversarial perturbations through efficient training methods, and (2) defending against Expectation over Transformation (FoT) attacks (Athalye et al. 2018b) by increasing the variance of attack gradients.
Our main contributions are as follows:
We offer a novel perspective on DBP robustness, highlighting the crucial role of stochasticity while challenging the conventional, purification- based view that robustness primarily arises from distribution gap reduction during the forward diffusion process.
We introduce a new Deterministic White-box attack setting and demonstrate that DBP models rely stochastic attack gradients evade the most effective attack directions, revealing key differences from adversarially trained models.
We show that DBP robustness can be further improved by enhancing the diffusion model's ability to purify adversarial perturbations through the proposed ADDT method.
RELATED WORK
Adversarial training (AT). First introduced by Madry al. (0118), AT seeks to develop robust. classifier by incorporating adversarial examples into the training process. It has nearly become the de facto standard for improving the adversarial robustness of neural networks (Athalye et al./ [0188]] Gowal et al.] 2020; Rebuffi et al.] 2021). Recent advances in AT harness the generative power of diffusion models augment training data and prevent overfitting (Gowal et al.) 202T Waang et al. 2023 However, the application of AT to DBP methods has not been thoroughly explored.
Adversarial purification. Adversarial purification utilizes generative models to remove adversarial perturbation from inputs before they are processed by downstream models. Traditionally, generative adversarial networks (GANs) (Samangouei al., 2018) or autoregressive models (Song al., 2018) have been used as purification models. More recently, diffusson models have been introduced for adversarial purification a technique known as diffusion-basse purification (DBP), which has shown promising results (Nie et al., 2022 Wang et al. 2022 Wu et al.) 2022 Xiao et al.) 2022). The robustness of DBP models is often attributed to the wash-out effect of Gaussian noise introduced during the forward diffusion process. Nie et al. (2022) propose that the forward diffusion process reduces the Kullback- Leibler (KL) divergence between the distributions of clean and adversarial images. Gao et al. (2022) suggest that while the forward diffusion process improves robustness
2

---

Published as a conference paper at ICLR 2025
by reducing model invariance, the backward process restores this invariance, thereby undermining robustness. However, these theoretical explanations lack strong experimental support.
PRELIMINARIES
Ad Adversarial training. Adversarial training aims build a robust model by including adversarial samples during training (Madry et al. 2018). This approach can be formulated as a min-max problem, where it first generates adversarial samples (the maximization step) and then adjusts the parameters to resist these adversarial samples (the minimization step). Formally, this can be represented as:
0' arg min) [f
6). v)],
where L is the loss function. f is the classifier, (æ, v) ~ D denotes the sampling of training data from the distribution D, and B defines the set of permissible perturbation d.
Diffusion models. Denoising Diffusion Probabilistic Models (DDPM) (Ho et al. 2020) and Denoising Diffusion Implicit Models (DDIM) (Song et al. 2020) simulate a gradual transformation in which noise is progressively added to an image and then removed to restore the original image. The forward process can be represented as:
V 0,30 + vI- e~ 10,,1,,
where mo is the original image and De is the noisy image. ā. is the cumulative noise level at step t (I < t < T, where 7' is the number of diffusion training steps). The parameters 0 are optimized by minimizing the distance between the actual and estimated noise:
arg ming [||e - < (v + v1 - 62 1)
where 0 represents the optimized parameters, and eg is the model's 's predicted noise. Using E can estimate tn in a single step:
2o (x. vI TzEa- (x,,t)) / /Va..
where Eg the recovered image. DDPM typically takes an iterative approach restore the image, removing small amount of Gaussian noise at time:
3r V1- Tr
c0- (ate /V1 B- + VBe,
where 8e is the noise level at step -b=-1 is the image recovered at step ( 1, and is sampled from N(O, 1). DDIM speeds up the denoising process by skipping certain intermediate steps. Recent work suggests that DDPM could also benefit from a similar approach (Nichol & Dhariwal 2021) Score SDEs (Song et al. 2021a) provide a score function perspective on DDPM and further lead to the derivations of DDPM+ ++ (VPSDE) and EDM (Karras et al. 2022). In this diffusion process, the noise terms e in Equation ) and Equation (5) represent the key stochastic elements that control the
randomness of the process. More discussion of stochastic elements provided in Appendix C.
Diffusion-based purification (DBP). DBP uses diffusion models to remove adversarial perturbation from images. Instead of using complete diffusion process between the clean image and pure Gaussian noise (between = 0 and = T), they first diffuse Do to a predefined timestep / = (1"
T) via Equation ( and recover the image do via the reverse diffusion process in Equation 5
4 STOCHASTICITY-DRIVER ROBUSTNESS OF DBP
4.1 STOCHASTICITY AS THE MAIN FACTOR OF DBP ROBUSTNESS
As discussed in Section previous studies primarily attribute the robustness of DBP models to the forward diffusion process, which perturbs inputs with Gaussian noise to reduce the distribution gap between adversarial and clean images (Nie et al. 2022; Wang et al. 2022). As a result, adversarial perturbations can be "washed out" by the Gaussian noise. However, it has also been observed that the robustness of DiffPure diminishes when switching from Stochastic Differential Equation (SDE) sampling Ordinary Differential Equation (ODE) (Nie et al.] (022), which introduces less stochasticity. This reduction robustness cannot be fully explained by the "wash out" effect of Gaussian noise, suggesting that stochasticity plays role in DBP robustness.
3

---

Published as a conference paper at ICLR 2025
To assess the impact of stochasticity on DBP robustness, we implemented both DDPM and DDIM within the DiffPure framework (Nie et al., 2022) and compared their performance (referred to as DDPM and DDIM). Note that the original implementation of DiffPure adopts a DDPM discretization form of DDPM++ (VPSDE), which differs only minimally from DDPM (Ho et al. 2020). Thus, the primary difference between DiffPure and our DDPM is that DiffPure employs larger UNet. DDPM utilizes a stochastic SDE-based reverse process, which introduces Gaussian noise in both the forward and reverse processes, making the entire process stochastic. In contrast, DDIM uses deterministic ODE-based reverse process and introduces Gaussian noise only in the forward process. The results are shown in Figure (labeled as Clean and White), where White refers to bx white-box PGD (Madry et al.) 2018) EoT (Athalye et al. (018b) attacks (as detailed in Section (6.T) Although DPDDPM achieves higher clean accuracy, it shows lower robust accuracy under adaptive white-box attacks, consistent with the findings Nie et al. (2022). This suggests that the
stochasticity in the reverse diffusion process may also play a crucial role in DBP robustness.
To better control the stochasticity in both the forward and reverse diffuusio processes and to isolate its ef- fect on DBP robustness, we introduce novel attack
85.04
88.88
DP
DPomn
Accuracy (%)
100
setting called the Deterministic White-Box (DW- box) setting. In this setting, the attacker not only has complete knowledge of the model parameters but also of the exact values of the stochastic elements sampled during evaluation, effectively making the diffusion process deterministic from the attacker's perspective. This setting could be realistic if the attacker knows the seed or the initial random state used for pseudo- random number generation in the model. For our evaluations, we define three levels of attacker knowl- edge: (1) Conventional White-box setting, where the attacker has access to the model parameters but not the stochastic elements; (2) DW -ow//DDeeWWrr-aaa setting, where the attacker knows the stochastic ele-
47.27
42.19
43.41
38 00
42.
50
80
4.98
4,08
Clean White DWPsa DW DWava
Figure 2: DPDDPM and DDIM robust accu- racy under different attack settings on CIFAR- 10. Both models lose most of their robustness only when the attacker knows all stochas- tic elements (highlighted in bold: DWBoh box for DPDD and DW -box for DPDDIM).
ments in the forward/reverse process in addition to the model parameters; (3) DW Both Both-hox box setting, where the attacker has complete knowledge of the model parameters and all stochastic elements in both the forward and reverse processes. Details of these settings are provided in Appendix +pp
With the Deterministic White-Box attack, we are able to compare traditional theories with our proposed hypothesis. These explanations diverge in behavior when stochasticity controlled. Traditional theories emphasize the forward diffusion process as the primary defense mechanism, suggesting that both DPDDPM and DDIM should behave similarly under the DWrou-box setting. In contrast, our hypothesis emphasizes the stochasticity throughout the diffusion process as the crucial factor. We hypothesize that as DPDDIM becomes deterministic under the Dbox setting, it should experience a significant reduction in robustness mimstla We evaluated adversarial robustness on CIFAR-10 using foo attacks (see Section 6.11.. The results shown in Figure 2 Under the DWFdd-bor Bud-box setting, DD maintains a significant portion of its robustness, whereas DDM loses almost all resistance to adversarial attacks. This phenomenon can not be explained by the "wash out" theory. Furthermore, under the DW Both -bx setting, DDPM shows a substantial drop similar to DP in the DWrwd setting. This suggests that stochasticity in both the forward and reverse diffusion processes plays a critical role in maintaining robustness.
similar to DPDDPM under the DW -box setting.
Our findings suggest that DBP models primarily rely on stochasticity to resist adversarial attacks, rather than depending solely on the forward diffusion process, and also reveals that DBP models lack the ability to effectively purify adversarial perturbations.
4.2 EXPLAINING STOCHASTICITY-DRRVV ROBUSTNESS
When attacking stochastic models, a commonly used technique is EoT (Athalye et al. 2018b). To assess the influence of adaptive attacks, we compare the performance of white-box attacks with and without using EoT. The results in [ahle[]show that HoT-baaed adaptive attacks have only modest impact on robustness, which contrasts sharply with DW-box attacks. A more detailed discussion of EoT steps is provided in Appendix A
4

---

Published as a conference paper at ICLR 2025
To compare different attacks, we visualize the at- tack trajectories using t-SNE. These trajectories projected onto the xy-plane, with loss values plotted on the z-axis. We compare three types of at- tacks: white-hox without EoT (White-box), white- box with EoT White--ox-EooT), and Deterministic White-box (DW-box). As shown in Figure || the trajectories vary significantly across all settings. re- flecting the stochasticity of DBP models. Notably, DW-box attacks lead to significant increase in loss values, whereas white-box attacks-even with EoT-result in only moderate increases. This observation suggests that stochasticity prevents attackers from finding the optimal attack direction. Even if the FoT estimate accurately captures the mean gradient direction, the high variance in attack gradients may prevent alignment with the optimal attack direction (the direction of the DW-box attack). This leads to reduced attack performance. Additional evidence is provided in Appendix B
Table I: Evaluation of DBP methods under vari- ous attack settings shows that EoT significantly affects the model's robustness accuracy (%).
Metric
DittPure
GDMP (MSE)
Clean
89.26
91.80 53.13
85,94
88.35 28.38
PGD20
69.04
60.25
54,59
PGD20+FoT10
55.96
40.97
42.19
To study how different attacks impact robust ac- curacy, compared the loss landscapes under White-box-EoT and Deterministic White-box attacks. We computed the average loss varia- tion across a batch of images and present the results in Figure 3 The trajectory of the White- box-EoT attack deviates from that of the Deter- ministic White-box attack, resulting in a flatter loss landscape along its path, whereas the De-
DW--box
White-bore-B EoI
terministic White-box attack produces a steep increase in loss. This observation suggests that the inherent stochasticity in DBP models pre- vents White-box-EoT attacks from identifying the optimal attack direction, and that removing this stochasticity makes the model vulnerable to adversarial perturbations. These findings con- trast with adversarially trained models, where the loss landscape remains flat even along adver- sarial directions (Shatahi al. 2019). Further details on the loss landscape visualization can be found in Appendix F
16.8
47.27
20
40
00
80
20
40
80
80
Figure 3: Visualisation of attack trajectories for White-box-Eoo attacks and DW-box attacks on the loss landscape. The loss landscape is steep in the direction of the DW-box attack. The plot is based on the first 128 images of CIFAR-10.
In conclusion, we propose that DBP models, instead of exhibiting a flat loss landscape, leverage
stochasticity to evade the most effective attack directions. Note that while certified defense methods such as random smoothing also incorporate stochasticity (Xiao et al. 2022 Carlini et al., 2022), their mechanisms and implications differ from those of DBP methods (see Appendix D
TOWARDS IMPROVING THE PURIFICATION CAPABILITY OF DBP MODELS
Based on the analysis in Section and considering the loss increase along the DW-box attack direction, we propose that as an alternative to stochasticity-driven robustness, the performance of DBP can be further improved by flattening the loss landscape. Achieving this requires incorporating adversarial samples into the training of DBP models. From the perspective of adversarial purification, this involves enhancing the diffusion model's 's ability to purify adversarial perturbations.
To address this, we propose Adversarial Denoising Diffusion Training (ADDT), a method that incorporates adversarial perturbations into the training of diffusion models in DBP. ADDT follows an iterative two-step process: (1) Massifier-Guidel Perturbation Optimization (CGPO), which generates adversarial perturbations by maximizing the classification error of a pre-trained classifier; (2) Diffusion Model Training, which trains the diffusion model on these perturbations to improve its ability to purify adversarial perturbations.
Integrating adversarial perturbations into diffusion training is challenging due to the Gaussian noise assumption inherent in diffusion models. To overcome this, we propose Rank-Based Gaussian Mapping (RBGM), technique that transforms adversarial perturbations into form more consistent
5

---

Published as a conference paper at ICLR 2025
Map 8 cx(e)
Map é (6
Generate i and astimate Zo
2 Ganerata at, and estimat R
landivve
Clossily 2g and
3 Compare
with
Update
+ Update
CGPO Step Update
Training Step
P(O)
(RBGM)
Update
(RDGM)
Figure 4: Overview of Adversarial Denoising Diffusion Training (ADDT). ADDT alternates between a CGPO step (left grey box) to refine the perturbations with a frozen diffusion model and classifier, and a training step (right grey box) to update the diffusion model with the refined perturbation. Throughout the process, RBGM is used to make the perturbation more "Gaussian-like".
with the Gaussian noise assumption, thereby facilitating their integration into the diffusion training process.
Figure provides an overview of ADDT, and pseudocode is available in Appendix G subsections detail the components of ADDT.
The following
5.1 ADVERSARIAL DENOISING DIFFUSION TRAINING
Classifier-Guided Perturbation Optimization (CGPO) step. In this step, we refine the adversarial perturbations to maximize the classification error pre-trained classifier C, as shown on the left side of Figure 4 The process begins by reconstructing clean image zo from the perturbed
input x using the diffusion model P. The perturbed input x is generated from the original image Tn. Gaussian noise and the RBGM-mapped perturbation ea (6),
detailed in Section 5.2 Here,
P(O, at, t) represents a one-step diffusion process based on the formulation in Equation 4
reconstructs the image žy from the noisy input æ/. The classifier C' is then used to predict label for this reconstructed image Lo. The optimization objective is to maximize the discrepancy between the classifier's 's prediction and the ground trrut label y. This can be formulated as:
8 arg maxg E,,,,,c t.c [L (C (O, x² (wo, e, cx())), t)) .3),
(6)
where L( y) denotes the loss function measuring the discrepancy between the classifier's prediction and the ground-truth label y. The pre-trained classifier C provides semantic guidance and need not be identical to the protected classifier, detailed in Section 6.2 To optimize ö, we employ the PGD attack (Madry et al. 2018). Since RBGM is non non-differentiable. we apply the Backward Pass Differentiable Approximation (BPDA) approach (Athalye et al.) 2018a). By applying BPDA, we employ RBGM during forward propagation but bypass it during backpropagation. Note that BPDA/REGM is only applied during training and does not affect the evaluation of DBP robustness.
Diffusion Model Training step. The objective of this step is to update the diffusion model's parameters that illustrated on the right side of Figure the model aims to remove both Gaussian noise and adversarial perturbations, effectively denoising the input. The optimization objective is defined as:
reconstruct the original image 2o from its perturbed counterpart a'/ As
ar a2
0" = arg mitlo Emg...e
Po - P(0, (an,, (6))),
where the scaling factor / vT// Tie ensures consistency with the standard DDPM/DDIM loss formulation.
5.2 LANK-BASHR GAUSSIAN MAPPING
Traditional diffusion models assume that input images corrupted by independent Gaussian noise e. To preserve Gaussian-like characteristics while encoding adversarial features, we introduce Rank- Based Gaussian Mapping (RBGM), shown in Figure RBGM, denoted as es c8))), retains the

---

Published as a conference paper at ICLR 2025
ordering of the elements in the input tensor ô while replacing their original values with samples drawn from a standard Gaussian distribution cş. Specifically, we first sample a Gaussian tensor e, with dimensions matching 6. Next, sort the elements of both ő and E, in ascending order. By mapping the sorted elements of g to the corresponding elements of 6s. we obtain 6 (6), which approximates Gaussian distribution while preserving the structural information in 8. To further refine the perturbation towards Gaussianity, we blend the RBGM-mapped perturbation with additional randomly sampled Gaussian noise. The adversarial input 2 then defined follows:
(2 6, (6) Vo
X2v1 ThE + d.v' (6),
(8)
where A modulates the level of adversarial perturbation. This ensures that the overall perturbation remains largely independent of zo. preventing the perturbations from overwhelming the denoising model's learning process. We determine X using the following formulation:
A clip("xAumtt Aman: Amaz), Vt vārl L/a//vi-B Ce.
where the clip p function constrains d between Amin and Amaz. Additional details and discussions about RBGM can be found in Appendix K
6 EXPERIMENTS AND DISCUSSIONS
6.1 EXPERIMENT SETUP
Classifier. We train WideResNet-28- 10 model for 200 epochs following the
Input (6)
Arbifarr Dishibution
Sampled
Cutput (ev(F))
Dusiiutioon
Naice
methods in Yoon al. (2021); Wang ct al. (2022). achieving an accuracy of
RBGM
92.12% on CIFAR-10 10 and 76.66% on CIFAR- 100 dataset.
Sample Gaussian Nolse
DBP timestep. For the diffusion forward process, we adopt the same DiffPure (Nie) et al.] 2022). In continuous-tine mod- els, such as the VPSDE (DDPM++) variant, with the forward time param- eter 0 < f < 1, we set t" 0.1, bal- ancing noise introduction and compu- tational efficiency. For discrete-time models, such DDPM and DDIM, where = 0, 1, ..., T. we similarly set the timestep to t* 0.1 T. Addi- tional settings and results for DPEDM are provided in Appendix !
Rank
Match
Input (d)
Perturbation
timestep settings
VI
Replace
Output (e)())
Gausslan Mapped Perturbation
Figure 5: Rank-Based Gaussian Mapping, RBGM trims the input to follow Gaussian distribution. samples a Gaussian noise and then replaces elements in the input with corre- sponding values from the noise, based on matching ranks.
Robustness evaluation. We as- sess model robustness using the PGD20+E6T10 attack (Athalye) et al. 8b). For foom at- tacks, we set the step size a
Table Clean and robust accuracy (%) on CIFAR-10, 10, obtained by different classifiers. ADDT (WRN-28-1P 10 guidance) improves robustness in protecting different subsequent classifiers. (*: the classifier used in ADDT fine-tuning).
8/255: for fz- norm attacks, we use = 0.1 and 6 0.5. Due the high computational cost of EoT at- tacks, we evaluate the models on the first 1024 images from the CIFAR-10 and CIFAR-100 datasets. For comprchensive evaluation settings, including an analysis of AutoAttack (Croce & Hein 2020 performance on
2/255 and the maximum per- turbation
DBP method
Clasifier
Glean
Vinilla
[=
Cleam
ADDT
Exc
(Smonyynn an Zisserman
2014
DP
WRN-ZN- 10
2016
85.94
47.27
65.64
51.46
70.12
WRN--70-1 10-17
65.43
48.91
70.31
67.34
52.54
70.70
VIT
83.45
45.61
69.53
85.25
48.63
69.92
VGG5-16
Sumooyan & 2i5s man
2014
K7.16
20.00
61 61.821
87.55
35.06
66.11
Reftbrriir 72
85.04
62.1
86.
38.77
63.82
DP
10"
Zapocivku fohnn
88.00
47,85
70.01
WRN--70-1
Dosorioldy
88.77
68
41.02
68.65
DP
WRN-38-100 10
WRN- 70-16
Zaguruykor & Kamodatts 2020
Zaseenck d Komedabe Mi
86.02
86.43
65.02
62.50
70.45
76
86.43
86
69.63
66.41
79.16
73.91
DBP models, refer Appendix F
7

---

Published as a conference paper at ICLR 2025
Table 3: Clean and robust accuracy (%) on CIFAR- 10 obtained by different DBP methods. All meth- ods show consistent improvement fine-tuned with ADDT.
Table Clean and robust accuracy (%) on DDDDPM- ADDT improves robustness across different NFEs, especially at lower NFES (*; default DDPM generation setting; classifier only).
Dillusion model
DBP mechod
Cleen
Exc
88.3
95.12
Dutaset
NFLs
Clean
Vamilla
Clean
ADDT
DisnntAADD
88.77
95.12
0.00
1,46
95.12
0.00
1.46
DDPM
GDMP (SSIM)
GDMP
INo
Wanngr
Wanr
(aan st dl M
85.9c
92.19
91.
67.27
69.34
68 68.95
09:63
CIFAR-10
20
50
10
73.34
49.51
36.72
21.78
65
55.47
36.13
83.89
59.95
78.91
50.20
43.07
30.27
48.44
62.97
69.04
69.82
41.99
ADDD
85.64
51,46
70.12
51.46
70.12
DDPM+
COUP Zhong al. 2024
DiD
89.26
90.33
89.94
55.96
50.77
62.11
79.19
76.66
75.73
CIFAR-LOD 10
20
10
48 05.05
34.08
17.29
76.66
10.55
17.68
50.66
19.24
9.28
53.52
40.62
14,55
18.65
6.25
0.00
27.25
13:77
36.13
EDM
Daan (Appendis]] DP Tuox+ADD AADDT [Appehmix[]]
86.43
86.35
62.50
66.41
76.86
79.16
100"
50
$7.52
55.57
20.02
20.41
37.70
$7.89
39.47
39.18
23.75
22.75
40.72
41.70
ADDT. ADDT fine-tuning is guided by the pre-trained WideResNeet22-110 classifier. For the CIFAR- 10 dataset, we utilize the pre-trained exponential moving average (EMA) diffusion model developed by Ho et al. (2020) (converted to Huggingface Diffusers format by Fang et al. (2023)). For the CIFAR- 100 dataset, we fine-tune the CIFAR- 10 diffusion model for 100 epochs. In CGPO, we set the
hyperparameters to Xunit = 0.03, Amin O, and Xmaz 0.3, and iteratively refine the perturbation for steps. Additional details regarding computational cost are provided Appendix O
6.2 DEFENSE PERFORMANCE UNDER DIFFERENT CONDITIONS
Performance across different DBP models. We evaluate the effectiveness of ADDT by applying it various diffusion models and comparing their performance with existing methods. The results shown in Table 3 indicate that ADDT consistently enhances model robustness while maintaining competitive accuracy on clean data.
Performance across different downstream classifiers. We evaluate ADDT's cross-classifier protection perfor- mance by applying a diffusion model specifically ADDT fine-tuned with WRN-28- 10 guidance, to protect variety of classifiers. The results in Table 2 show that these mod- els effectively protect classifiers with diverse architectures. Notably, employing DREDM model trained with WRN- 28- 10 guidance achieves a 69.63% loo robust accuracy on WRN-70-16 classifier. This ability to protect different classifiers without classifier-specific fine-tuning highlights the effectiveness and feasibility of ADDT.
Table 5: Clean and robust accuracy (%) CIFAR-10 fine-tuned with dif- ferent training samples. (None: no fine- tuning)
DBP method
Training samples
Cleen
Es
42
None
85.94
47.27
69 34
DP
MSE-quided
Cleun
26.91
$5.25
47.27
46.97
70,30
68.26
CGPO
85.64
51.46
70.12
Nane
43.16
Clein
41.41
DPPc04
MSB-guide guided
40.92
CGPO
$8.18
47.85
70.61
Performance under acceleration. Speeding up the diffusion process by omitting intermediate steps has become common practice in diffusion models (Song al., 2020 Nichol & Dhariwal, 2021). In this context, we evaluate the robustness of accelerated DBP models. To quantity computational cost, we utilize the number Neural Function Evaluations (NFEs), which reflects the number of evaluation steps executed during the DBP reverse process. For our experiments, we set t = 0.1 x T and accelerate the process by excluding intermediate time steps. For example, with 5 NFEs, the time steps for the DBP reverse process would be / [100, 80, 60, 40, 20, 0j. The results in Table 4 validate the effectiveness of ADDT in improving the robustness of accelerated DPDDPM models. Note that the performance of DPDDPM varies significantly with different NFE values. This may be because DDPM introduces stochasticity (Gaussian noise) at each reverse step, and with fewer reverse steps, the influence of this stochasticity diminishes. Additionally, the generative performance of DDPM is sensitive to the omission of intermediate steps. We also evaluated DP models, as detailed in Appendix H
6.3 ABLATION STUDY AND ANALYSIS
RBGM. We compare the generative ability diffusion models fine-tuned from the same pre-trained
8

---

Published as a conference paper at ICLR 2025
models, using two different perturbations: RBGM- mapped perturbations and & perturbations. To quan- tify generation quality, we use Fréchet Inception Dis- tance (FID) scores (Heusel et al.] 2017), as shown in Table 6 The results demonstrate that diffusion models fine-tuned with RBGM-mapped perturbations maintain generation quality comparable the vanilla diffusion model, while models directly fine-tuned with &or perturbations without RBGM show degraded performance. Additionally, we observe that training with RBGM-mapped perturbations generalizes better to different types of attacks. Experimental details and further tests are provided in Appendix L
Table 6: FID score of DDPM fine-tuned on CIFAR-10 10 with different perturbations (lower is better). Fine-tuning with RBGM-mappe perturbations results in lower FID scores pared to &a perturbations (without RBGM).
Vanilla
Ckam Fine-tuning ADDT ADDT wi RBGM
FID 3.196
3,500
5.190
13,608
CGPO. We analyze the impact of fine-tuning with different training samples in Table [ Specifically, we compare the performance of samples generated with classifier guidance in the CGPO step, referred as "CGPO", against those generated with Mean Squared Error (MSE) loss, denoted "MSS-guidedd.. The evalua- tion results are presented for DDPM with 100 NFEs and DDIM with 10 NFEs. The results demonstrate that samples generated by CGPO significantly outperform MSE-guided samples in terms of enhancing DBP robustness.
DPoOPM
DPDDM
Accuracy (%)
100
47.27
42:19
100
51.46
46.48
39. 16.16
50
16.80
17.09
4.96
White DW
White
DW
without ADDT
with ADDT
Figure 6: Revisiting robustness under Determinis- tic White-box setting. ADDT improves robustness under both White-box and Deterministic White- box settings, implying that ADDT strengthens the models' ability to purify adversarial inputs.
Revisiting DBP robustness. We re-examine robustness under the Deterministic White-box setting by comparing the performance of diffu- sion models with and without ADDT fine-tuning, as shown in Figure 6 The fine-tuned models show significantly higher robust accuracy under the DW-box setting, indicating that ADDT improves non-stochastiiitt--aass robustness. Further experiments across different models and NFEs, presented in Appendix M confirm these improvements in robustness. We also compare the loss landscapes of ADDT fine-tuned models and vanilla diffusion models, shown in Figure [3] This comparison reveals that our method effectively smooths the loss landscape of DBP models, enhancing their ability resist adversarial perturbations.
Evaluation with stronger PGD+EoT attacks. To balance computational cost and attack strength, primarily employ the PGD20+E6TT00 configuration in evaluations. To further validate the efficacy of ADDT under stronger attack settings, we assess its performance using the more challenging PGD200+EoT20 setup. The results presented in Table and Table D show that under these intensified attacks, ADDT's robust accuracy experiences moderate 5% drop compared to the PGD20+EoTI setting. Nevertheless, across various configurations and datasets, ADDT consistently outperforms the baseline in terms of robust accuracy.
6.4 SCALING MORE COMPLEX AND HIGH-DIMENSION DATA
To assess the scalability of DBP and ADDT
Table 7: Robust accuracy (%) on CIFAR-10 under stronger attacks.
on more complex datasets, we extend our
experiments to include Tiny-Image ImageNet (Le
Yang 2013) and ImageNet- 1k (Deng
DBP method
PGD200-EET20 Vanilla (5xc)
ADDT (f.e)
Vanilla (fxz)
PGD2I-E6T1O
ADDT (fxo)
et al. 2009). For Tiny-ImageNet, we
DPaors
41.02
46.19
47.27
51.46
trained the diffusion model from scratch
DPannt
36.23
41.11
43.16
47,85
DiffPure
48.93
55.76
55.96
62.11
for 200 epochs and then fine-tuned it with
ADDT for 50 epochs, using a pretrained WRN-28-IL 10 classifier for guidance. For ImageNet- Ik, the diffusion model was trained from scratch for 12 epochs, followed by 8 epochs of ADDT fine-tuning, with guidance from a pretrained ResNet- 101 classifier.
As shown in Table and Table 9 ADDT improves the robustness of DBP on these complex datasets. However, the additional robustness provided by ADDT appears relatively modest. This suggests that, to scale ADDT more effectively, adequate model capacity and sufficient volume of data are

---

Published as a conference paper at ICLR 2025
Table 8: Clean and robust accuracy (%) on Tiny- ImageNet with WRN-28-10 10 classifier. ADDT improves DBP robustness on Tiny- ImageNet (+: classifier only).
Table 9: Clean and robust accuracy (%) on ImageNet- 1k with ResNet- clas- sifier. Experiments are conducted under to perturbation bound of e 4/255.
DBP method
Clean
Vanilla foo
Eg
Cleun
ADDT
Eoo
Metric
Vanilla
ADDT
71.37
0.00
0.00
DP Does
57.13
11.82
46.68
56.15
13.57
Clean
80,31
80.20
60.35
4.79
39.75
60.45
5,86
40.82
PGD20+E0TI0
46.92
48.02
Daau
57.03
19.14
46.00
36.45
20.61
47.95
PGD200+BeT20
35.31
35.83
essential. This is further supported by the results in Table 3 where the larger DifIPur+AADDT model demonstrates greater robustness compared to the smaller ADDD model, aligning with findings from traditional AT' (Wang et al. 2023 Huang et al. 2023).
It is also important to note that scaling to high- resoluton images poses significant challenge for attackers, as performing strong EoT attacks on high- righ-resolution images is computationally expensive. For instance, our evaluation with PGD200+EoT22
224 224 images at resolution of 1024 takes approximately 7 days of computation on 8 NVIDIA RTX 4090 GPUs.
6.5 DISCUSSIONS ON IMPROVING STOCHASTICITY-BASSEE DBP ROBUSTNESS
As discussed in Section 4.2 DBP robustness can be primarily attributed to the high variance of the stochastic attack gradients. We argue that increasing the variance of attack gradients can improve the stochasticity--asse attacks. Specifically, on one hand, higher variance leads to larger errors in estimating the expected attack gradient direction. To reduce these errors, more EoT steps are required. On the other hand, higher variance also means that the DW-box attack gradient (which indicates the most effective attack direction) deviates more from the EoT attack gradient, even if the estimation of the mean attack gradient is accurate. As discussed in Section 422 this deviation leads to lower increase in classification loss after one attack step, suggesting that a successful attack may not be achieved or require more PGD steps.
robustness of DBP models by reducing the effectiveness of EoT
To increase the variance of attack gradients, a natural approach is to introduce more stochasticity. In initial experiment, we augment the DBP framework's stochasticity by integrating a Corrector sampler. Specifically, Song et al. (2021b) propose a Predictor-Corrector (PC) sampler framework. While standard VPSDE (DDPM++) implementations typically use only the predictor component, we add Corrector sampler to increase stochasticity in the reverse diffusion process, thereby enhancing the overall variance of attack gradients. As detailed in Appendix J our preliminary results indicate that this modification improves the robustness of DBP models against adaptive White-box attacks. However, there a trade-off: the model's clean accuracy decreases slightly. These observations align with the findings Nie et al. (2022), where randomizing the diffusion timesteps also improves robust- ness at the expense of clean accuracy, as well as with previous research on stochastic preprocessing defenses (Gao et al. 2022).
CONCLUSION
This study provides a new perspective on the robustness of Diffusion-Basee Purification (DBP), highlighting the critical role of stochasticity and challenging the traditional view that robustness primarily arises from minimizing the distribution gap through the forward diffusion process. We introduce a Deterministic white-box (DW-box) attack setting and demonstrate that DBP models rely on stochastic elements to evade effective attack directions but lack the ability to purify adversarial perturbations. This shows distinct differences compared to models trained with Adversarial Training. To further improve the robustness of DBP models, we propose Adversarial Denoising Diffusion Training (ADDT) and Rank-Based Gaussian Mapping (RBGM). ADDT integrates adversarial pertur- bations into the training process, while RBGM trims perturbations to better approximate Gaussian distributions. Experiments across various diffusion methods, attack settings, and datasets confirm the effectiveness of ADDT in enhancing robustness. In conclusion, this study emphasizes the decoupling of stochasticity- based and purification-bast robustness of DBP models for deeper analysis, and suggests that combining both approaches can lead to improved robustness in practice.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported in part by National Natural Science Foundation of China (NSFC) under Grant No.62376292, U21 A204700, Guangdong Provincial General Fund No. 11501000000
We would like to express our sincere gratitude to Hongjun Wang, Zhaoyu Chen Huanran Chen Ruoxin Chen and Conghan Yue for their invaluable discussions, which greatly contributed to the development of this work.
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning, pp. 274-283. PMLR, 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pp. 284-293. PMLR, 2018b.
Nicholas Carlini, Florian Tramer, Krishnamurthy Dj Dvijotham, Leslie Rice, Mingjie Sun, and J Zico Kolter. (certified!!) adversarial robustness for free! In The Eleventh International Conference Learning Representations, 2022.
Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu.
Robust classification via single diffusion model. arXiv preprint arXiv:2305. 2023.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 2016.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. international conference on machine learning, pp. 1310-1320. PMIR, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pp. PMLR, 2020.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint D.XXi:201000967700 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11.22990..1199,... 1929, 2020.
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. arXiv preprint arXiv:2305.10922 2023.
Yue Gao, Ilia Shumailov, Kassem Fawaz, and Nicolas Papernot. On the limitations of stochastic pre-processing defenses. Advances in Neural Information Processing Systems, 35:24280-2422 242944 2022.
Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL htL:ttp:://ee http::/tp:/ii....tt/// org. abs 1412.4111100 6572
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy A. Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norm- bounded adversarial examples. CoRR, abs/2010.03593, 2020. URL htt.ttttt//tecttm org wwwp/aa/aaaaaeooooerooooooaaaaea 033503030000000303333

---

Published as a conference paper at ICLR 2025
Sven Gowal, Sylvestre-Allise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in Neural Information Processing Systems, 34:4218 0144218-4335 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in newral information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffuusion probabilistic models. arXiv preprint arxiv:2006.1 1/339, 2020.
Shihua Huang, Zhichao Lu, Kalyanmoy Deh, and Vishnu Naresh Boddeti. Revisiting residual networks for adversarial robustness. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8202-8211, 2023.
Mintong Kang, Dawn Song, and Bo Li. Diffattack: Evasion attacks against diffusion- based ad adversarial purification. Advances in Neural Information Processing Systems, 36, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances Neural Information Processing Systems, 35:26565-26577, 2022.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference net forum?id
Track Proceedings. OpenReview.net, 2018. URL
IJ2IBE2AD
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning (ICML), 2022.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL https:/ Semanticschon wttp:aaccocccoooo0cccre0.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. International conference on machine learning, pp. 28492-285188 PMLR, 2023.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timo- thy A. Mann. Fixing data augmentation to improve adversarial robustness. ArXiv, abs/2103.01946, 2021. URL //api//ppppae semant icscholar axg/CooppsID::3223333
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint 1805:068006663 2018.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.1556 2014.
Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models.
arXiv:2010.005502, October 2020. URI https:tts::// httth:t/p::/aaatt //tttp:://axiv..... www.arawwwwwwgaaaaaaaaaaaaaeaeaa /2000. 02502
12

---

Published as a conference paper at ICLR 2025
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In Inter- national Conference on Learning Representations, 2018. URL
forum? (imum?ideee d
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differcential equations. International Conference on Learning Representations, 2021a. URL
[d==xTTG12RRHS3
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. score-based generative modeling through stochastic differrential equations. In International Conference on Learning Representations, 2021b. URL /ttpbbuuutttttts/////eeaaaoa htttpss://penneeeew..ee..
id=PxTTG12RB 2RRH5
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.). 2nd International Conference on Learning Representations, ICLR 2014. Banff, AB, Canada. April 14-16, 2014, Conference Track Proceedings, 2014. URL http:://axxttit org/abs. 1312. 6199
Jinyi Wang, Zhaoyang Lyu. Dahua Lin, Bo Dai, and Hongfei Fu. Guided diffusion model for adversarial purification. arXiv preprint 2rXiv:2005.14999 2022.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference on Machine Learning (ICML), 2023.
Quanlin Wu, Hang Ye, and Yuntian Gu. Guided diffusion model for adversarial purification from random noise. arXiv preprini arXiv:2206. 10875, 2022.
Chaowei Xiao, Zhongzhu Chen, Kun Jin, Jiongxiao Wang, Weili Nie, Mingyan Liu, Anima Anand- kumar, Bo Li, and Dawn Song. Densepure: Understanding diffusion models towards adversarial robustness. arXiv preprint arXiv:221 2022.
Jongmin Yoon, Sung Ju Hwang, and Juho Lee. Adversarial purification with score-based generative models. In International Conference on Machine Learning, pp. 12062-12072. PMLR, 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 1605.07146. 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference machine learning, pp. 7472-7482. PMLR, 2019.
Mingkun Zhang, Jianing Li, Wei Chen, Jiafeng Guo, and Xueqi Cheng. Classifier guidance enhances diffusion-based adversarial purification by preserving predictive information, 2024. URL https
//openreview net wm um uddd-ydy2yt222222
13

---

Published as a conference paper at ICLR 2025
A INFLUENCE OF EOT ITERATIONS ON DBP ROBUSTNESS EVALUATION
In this section, we examine how the number of EoT iterations influences the DBP robustness evaluation. As previously discussed in Section 4.1 the Deterministic White-box attack could find the most effective attack direction. To quantify the impact of EoT iterations, we compare the attack direction of the standard White-box-Eo? across various numbers of EoT iterations with that of the Deterministic White-box.
See Figure for visual explanation, where the red line shows the classification accuracy. and the blue line shows the similarity between the attack directions of the White-box-Eae and Deterministic White-box. The results show clear trend: increasing the BoT iterations raises the similarity between the attack directions and reduces model accuracy.
Note that both the increase in similarity and the decline in accuracy converge as the number of iterations increases. Balancing computational cost and evaluation accuracy, we adopted the PGD20- EoT10 configuration for our robustness evaluation.
65
C Robust Accuracy
600
Gradient Similarity
0.3
55
50
0.2
45
0.1
40
35
100
10a
102
EoT iterations
Figure 7: Robust accuracy (%) and gradient similarity on DP for 10, obtained by different EoT iterations. As the number of EoT iterations increases, the gradient similarity between the White-box-EoT attack direction and the Deterministic White-box attack direction increases and the robust accuracy decreases.
B DBP MODELS WITH DIFFERENT STOCHASTIC ELEMENTS CANNOT BE ATTACKED SIMULTANEOUSLY
Previous research has raised concerns about whether stochasticity enhances robustness, suggesting it may create obfuscated gradients that provide a false sense of security (Athalye et al.) 2018a). To investigate this, we implement DW Semiibbxx, semi-stochastic framework that restricts the available stochastic elements to limited set of options, in order to explore whether stochasticity can genuinely improve robustness.
DW semi-128 builds on the concept of Deterministic White-box. In contrast to the Deterministic White-box approach, where the attacker exploits the exact stochastic noise used during evaluation, DW semi-128 limits the stochastic elements to constrained set of possibilities. To mount an attack across 128 distinct sets of stochastic noise, the attacker can employ the average adversarial direction across these 128 noise settings (EoT-128) to perturb the DBP model. We evaluate the impact of stochasticity by comparing the loss changes under DW-box and DW DW semi 128 attacks. This is done by adjusting factor & to modify
image with perturbation and evaluating the loss at + ko for & ranging from -16 16 to 16. Perturbations are generated using the for Fast Gradient Sign Method (FGSM) (Goodfellow et al. 2015) with magnitude of 1/255. The experiment is conducted using the WideResNee-28. 10 with DP over the first 128 images from the CIFAR- 10 dataset.
As shown in Figure in the Deterministic White-box setting, perturbations lead to significant increase in loss, demonstrating their effectiveness. However, under DW weesiiin 128, the loss increase more moderate. This suggests that even when stochastic elements are limited small set of
14

---

Published as a conference paper at ICLR 2025
0.12
Loss
0.08
Non adversarial
DW
0.04
DW seni 128
16
16
Figure 8: Impact of stochasticity perturbation efficacy. Perturbations created under the DW box setting are less effective compared to those under the DW-box setting. For con-adversarial perturbations, each element is randomly assigned value of either 1/255 -1/255.
possibilities, and the attacker has fully considered all options, stochasticity still contributes to the robustness of the DBP model. These results contradict the hypothesis that there might be universally vulnerable direction when DBP applies different stochastic elements, thereby challenging concerns about potential insecurity.
C IMPACT OF ATTACKERS' KNOWLEDGE ON ROBUSTNESS: COMPARISON OF ATTACK SETTINGS
This section explores how the robustness of diffusion-based models is affected by the levels of knowledge that attackers possess concerning stochastic components in diffusion processes. We analyze how the knowledge of forward and reverse diffusion processes impacts model robustness in various attack settings.
C.I STOCHASTIC ELEMENTS IN THE DIFFUSION PROCESSES
Unlike deterministic models, where outputs are solely determined by the inputs, diffusion models introduce stochastic elements that also influence the outcomes. To explain the attacker's knowledge, we first explain the stochastic components involved in diffusion processes.
In the forward diffusion process, Gaussian noise incorporated into the input data to derive a noisy version zy:
x = Va, x + VI āi Es,
(10)
where ~ I) is sampled once per input.
In the reverse diffusion process, the model progressively denoises x; through iterative steps. For the Denoising Diffusion Probabilistic Model (DDPM), the reverse process is inherently stochastic:
at at
C
+ orEt,
(11)
where E, ~~,0 is sampled at each reverse step. In contrast, for the Denoising Diffusion Implicit Model (DDIM), the reverse process is deterministic, and no noise {et]T) is added.
C.2 ATTACK SETTINGS AND ATTACKER KNOWLEDGE
We define four attack settings based on information accessible to the attacker, particularly regarding the Gaussian noise variables involved in the diffusion process. Table 10 provides summary of the attacker's knowledge in each setting.
In the conventional white-box attack setting, the attacker possesses comprehensive knowledge of the model architecture and parameters but lacks insight into the stochastic elements used during inference (c, and {c, )T The DWrwd Fwd setting grants the attacker knowledge of the Gaussian noise in the forward diffusion process (Cy). Conversely, the DW Rev setting provides the attacker with knowledge of the Gaussian noise introduced during the reverse diffusion steps ([e,)T>)) The DW Both setting offers the attacker complete access all stochastic elements, E, and (E)) By manipulating the
15

---

Published as a conference paper at ICLR 2025
Table 10:
Information accessible the attacker in different attack settings. Ey denotes the Gaussian noise in the forward process, and {Ex]i=1 represents the Gaussian noise in the reverse process.
Attacker's Knowledge
White-box
DW Pwd
DW Rev
DW Both
Model Architecture and Parameters Input Images and Class Labels
v
Forward Process Noise Ef
X
v
X
Reverse Process Noise (Erst=1
X
V
attacker's knowledge in this manner, we isolate the individual effects of the forward and reverse diffusion processes model robustness.
C.3 IMPLICATIONS OF THE ATTACKER'S KNOWLEDGE OF STOCHASTIC ELEMENTS
The attacker's capability to craft potent adversarial examples is significantly influenced by their knowledge of the stochastic elements in diffusion processes. When these elements unknown to the attacker, they must independently sample noise variables, leading to discrepancies between their approximations and the actual behavior of the victim model. Conversely, if the attacker has access the exact noise variables used during inference, they can accurately replicate the model's 's behavior, significantly enhancing the effectiveness of their attack.
Attacker Without Knowledge of Stochastic Elements. In scenarios where the attacker lacks access the specific noise variables Ey and ) the model's output becomes unpredictable from the attacker's 's perspective. Consequently, the attacker must optimize the expected value of the loss function over the distribution of these stochastic elements. The optimization problem for generating an adversarial example a: is formulated as:
x adv arg mäX = (e, [L (E(
; ;e,, (E))))))
(12)
where 5 specifies the permissible perturbation magnitude, is the loss function, f represents the model's output given the input and stochastic elements, and y the actual class label.
Attacker With Knowledge of Stochastic Elements. If the attacker possesses precise knowledge of the noise variables e and {c,] used during the model's inference, they can accurately replicate the behavior of the victim classifier. In this case, the stochastic processes become deterministic from the attacker's perspective, enabling the optimization problem to be formulated
add arg |
|- = L (f E (==))))))) v).
(13)
This precise knowledge allows the attacker to adopt the exact noise that will be used during the target evaluation, allowing effective evaluation.
C.4 EFFECT OF ATTACKER'S KNOWLEDGE ON MODEL ROBUSTNESS
We test the robustness of DDPM under these four settings, and Table encapsulates the result.
Table 11: Robust accuracy (%) of DDPM under different attack settings.
Attack Setting
Robust Accuracy (E.o)
Conventional White-Bor Attack
47.27
DW Pwd
45,41
DW Rev
35.25
DW Both
16.80
Conventional White-Box Attack. In this setting, the attacker has full knowledge of the model's architecture and parameters but lacks access to the stochastic components (E, and {<( involved during inference. Hence, the model's output is unpredictable due to the stochasticity of both diffusion processes, making it challenging for the attacker to generate effective adversarial examples (reaching robust accuracy of 47.27%)
16

---

Published as a conference paper at ICLR 2025
Ewd- Here, the attacker is aware of the Gaussian noise ey used in the forward diffusion process but does not have knowledge of the noise {er (= used in the reverse process. This partial information enables the attacker to replicate the forward process, reducing uncertainty. However, the reverse process remains unpredictable. The exposure of the forward process leads to slight decrease in robust accuracy to 45.41% %.
DW Rev- In this case, the attacker knows the noise variables {cr ) used in the reverse diffusion process but lacks knowledge of the forward process noise e,. This enables the attacker to align their strategy more closely with the behavior of the model during the reverse diffusion phase, resulting in a more substantial drop in robust accuracy to 35.25%. These findings suggest that the stochasticity of the reverse process plays more critical role in maintaining model robustness than that of the forward process.
DW Both- In this case, the attacker possesses full knowledge of the noise variables for both the forward and reverse diffusion processes, allowing them to precisely replicate both processes and eliminate stochasticity from their perspective. This complete predictability enables the attacker to craft highly effective adversarial examples, leading to a significant reduction robust accuracy to 16.80% %. These results underscore the importance of the stochastic elements in preserving model robustness; when fully exposed, the model's defense mechanisms are substantially weakened.
D THE ROLE OF STOCHASTICITY IN DBP COMPARED TO CERTIFIED DEFENSE METHODS
In this appendix section, we delve deeper into the role of stochasticity in Diffusion -Bsedd Prediction (DBP) models and contrast it with its role in certified defense methods such as randomized smooth- ing (Cohen et al. 2019). While both approaches incorporate stochasticity, their mechanisms and implications for adversarial robustness differ significantly.
Conventionally, the classification models discussed in the studies of adversarial robustness can be viewed as mappings from input space X' to the label space y. However, DBP additionally involves random variable e E E that determines the random sampling in the forward and reverse processes (which can be the random seed in implementation). Hence, a DBP model protected classifier f can be viewed as the mapping f (X,E) + y.
Previous studies on randomized smoothing treat the randomized model f as a mapping f: T'X' + Py, where Py is the space of label distribution. Typically, the final prediction can be formulated as F(x)= = arg max, [ax)] Je. i.e., the class e with the highest probability in the output distribution U(x). Apparently, F deterministically maps X' to V consistent with the conventional models.
Recent studies on DBP also regard the model as f - Py, without explicitly studying the role of e. The key difference between DBP and randomized smoothing is that the final prediction for an input is directly sampled from the distribution f(x) for once, instead of sampling multiple times to approximate F(x) as in randomized smoothing.
In this paper, we revisit DBP by treating the randomized model f as the mapping X,E) + y' and studying the role of 6 E as an input of f. From this perspective, the conventional adversarial setting assuming full knowledge of the model parameters (but not é) not a complete white box, which motivates us to study the DW-box setting.
From our perspective, we can clearly point out the difference between DBP and randomized smoothing in terms of the loss landscape. Given an input Fo. the local loss landscape for a DBP model f is not deterministic as it also depends on c. Although the expected loss landscape over e E È may be smooth, does not suggest the robustness of DBP, as is fixed during single inference fixed E,, the local landscape of DBP is likely not smooth. In contrast, the loss landscape of a randomized smoothing model F may be smooth is the average landscape over multiple e. To conclude, we argue that the random noise itself may not smooth the loss landscape, but the average over random noises may.
of DBP. Indeed, our study suggests that given wo and
17

---

Published as a conference paper at ICLR 2025
E EVALUATION SETTINGS
Previous assessments of DBP robustness have often utilized potentially unreliable methods. In partic- ular, due to the iterative denoising process in diffusion models, some studies resort to mathematical approximations of gradients reduce memory constraints (Nie et al. 2022 or to circumvent the diffusion process during backpropagation (Wang et al. (022). Furthermore, the reliability of AutoAt- tack, widely used evaluation method, in assessing the robustness of DBP models is questionable. Although Attokt includes a Rand version designed for stochastic models, Nie et al. (2022) have found instances where the Rand version is less effective than the Standard version in evaluating DBP robustness.
To improve the robustness evaluation of diffusion-based purification (DBP) models, we implement several modifications. First, to ensure the accuracy of the gradient computations, we compute the exact gradient of the entire diffusion classification pipeline. To mitigate the high memory requirements in diffusion iterative denoising steps, we use gradient checkpointing (Chen et al.) (016) techniques to optimize memory usage. In addition, to deal with the stochastic nature of the DBP process, we incorporate the Expectation over Transformation (FoT) method to average gradients across different attacks. We adopt FoT with 10 iterations, and detailed discussion of the choice of EoT iterations can be found in Appendix [A] We also use the Projected Gradient Descent (PGD) attack instead of AutoAttack for our evaluations. We discover bug in the Rand version of AutoAttack that causes to overestimate the robustness of DBP. After fixing this, AutoAttackyuun gives similar results to PGD attacks, but at a much higher computational cost. Our revised robustmess evaluation revealed that DBP models, such DiffPure and GDMP, perform worse than originally claimed. DiffPure's accuracy dropped from a claimed 70.64% to an actual 55.96%, and GDMP's from 90.10% to 40.97%. These results emphasize the urgent need for more accurate and reliable evaluation methods to properly assess the robustness of DBP models. Similar evaluation protocols are also applied in Chen et al. (2023): Kang al. (2024).
E.1 EVALUATION WITH FIXED AUTOATTACK
AutoAttack (Croce & Hein 2020), an ensemble of White-box and Black-box attacks, is a popular benchmark for evaluating model robustness. It used in RobustBench (Croce al. 2020) to evaluate over 120 models. However, Nie et al. (202)) finds that the Rand version of Attack designed to evaluate stochastic defenses, sometimes yields higher accuracy than the Standard version that is intended for deterministic methods Our comparison of AutoAttack and PGD20-EoTIl in Table [2] also shows that the Rand version of AutoAttack gives higher accuracy than the PGD20-EOT10 attack.
We attribute this to the sample selection of AutoAttack. As an ensemble of attack methods, Auto/ttack selects the final adversarial sample from either the original input or the attack results. However, the original implementation neglects stochasticity and considers an adversarial sample to be sufficiently adversarial if it gives false result in one evaluation. To fix this, we propose a 20-iteration evaluation and select the adversarial example with the lowest accuracy. The flawed code is in the official GitHub (el.dl/a556444966655533
main branch, git version 43922004883e9
and specifically in lines We
#1 #125, #129, #133- #157, #221-225, #227-228, #231 of the file autooatacc/auttatttak... will open-source our updated code and encourage future stochastic defense methods to be evaluated against the fixed code. The code can now be found at: http://ggthuu..com/Buutteeedteee -ttack
Following the fix, robust accuracy under AutoAttackpıedd decreases by up to 10 points, producing results comparable our PGD20-E6T10 test outcomes. However, using AutoAttack on DP took nearly 25 hours, five times longer than PGD20-E6T10. Therefore, we will use PGD20-EoTT0 for most robustness evaluation.
F EXPERIMENTAL SETTING OF VISUALIZATION OF THE ATTACK TRAJECTORY
We visualize the attack by plotting the loss landscape and tracing the trajectories of EoT attack under White-box setting and the Deterministic White-box setting in Figure 3 We run a vanilla PGD20-EoTI0 attack under White-box setting and PGD20 attack under Deterministic White-box setting. We then expand a 2D space using the final perturbations from these two attacks, draw the loss landscape, and plot the attack trajectories it. Note that the two adversarial perturbation directions
18

---

Published as a conference paper at ICLR 2025
Table 12: AutoAttack (Rand version) and PGD20-EoTIO performance on DBP methods for CIFAR- 10 (the lower the better). The original Auto/ttack produces high accuracy (%), after fixing, it achieves similar results PGD20+EoT10 attack.
DBP method
AutoAttack
AutoAttackfised
PGD20-FBTIO
AutoAttack AutoAltackrixed
PGD20-FoTI0
DiffPure
62.11
56.25
55.96
81.84
76.37
75.78
DDPM
57.81
46.88
48.63
71.68
71.09
72.27
DDIM
50.20
40.62
44.73
77.15
70.70
71.68
Table 13: Clean and robust accuracy (%) on different DBP methods for CIFAR-10, evaluated with AutoAAtackfiedd (Rand version). All methods show consistent improvement when fine-tuned with
ADDT.
Vanilla foo
ADDT Eoo
DBP method
Clean
82
Clean
82
DurtPure
89,26
56.25
76.37
89.94
58.20
77.34
DDPM
85.94
46.88
71.09
85.64
48.63
72.27
DPDDIM
88.38
40.62
70.70
88.77
44.73
71.68
are not strictly orthogonal. To extend this 2D space, we use the Deterministic White-box attack direction and the orthogonal component of the EoT attack direction. Note that the endpoints of both trajectories lie exactly on the loss landscape, while intermediate points are projected onto it. The plot evaluated using Net 10 with DP over the first 128 images of CIFAR-10 dataset.
G PSEUDO-CODE OF ADDT
The pseudo-code for adopting ADDT within DDPM and DDIM framework is shown in Algorithm
Algorithm 1 Adversarial Denoising Diffusion Training (ADDT)
Require:
is image from training dataset, is the class label of the image, C' is the classifier, P is one-step diffusion reverse process and 0 is it's parameter, L is CrossEntropy Loss.
I: for æu. y in the training dataset do
t ~~ (((,,...,
d = =clipp -clipyy-Auii Amin; Amax), where 7/1
vi-a;
Init 8 to small random vector.
5:
for to ADDTinenaion do
6:
E No00 T)
7:
é' (8GM(65,) E)
8:
T: = v Va -
9:
8 8 + L(C(P(O,
10:
end for
TO.
12:
e' = RBGM(O,
13:
Ø; = vārzo + V1 11 V -
14:
Take gradient descent step on:
Voll vas γI (wo P(O, æ. 1))
end for
Diffusion model es predicts the Gaussian noise added to the image, adopting Equation in the paper, we have P(O, w., 1) = (æ. vI a.ea( 1)) /var
H ADDT RESULTS ON DP
As shown in Table 14 the performance of DP is less sensitive to the number of function evaluations (NFEs). Additionally, ADDT consistently improved the robustness of DP
19

---

Published as a conference paper at ICLR 2025
Table 14: Clean and robust accuracy (%) on DDDDIM. ADDT NFEs (#: default DDIM generation setting, +: classifier only ).
improve robustness across different
Vanilla &0o
ADDT
Dataset
NFBs
Clean
Eg
Clean
&oo
82
95.12
0.00
1.46
95.12
0.00
1.46
5
89.65
42.19
68.65
88.57
47.27
70.61
10
88.96
43.16
67.58
88.18
47.85
70.61
CIFAR-10 10
20
87.89
41.70
69.24
88.67
48.63
69.73
50
88.96
42.48
68.85
88.57
46.68
69.24
100
88.38
42.19
70.02
88.77
46.48
71.19
76.66
0.00
2.44
76.66
0.00
2.44
5
62.11
15.43
35.74
62.79
17.58
38.87
10
62.21
15.33
36.52
64.45
20.02
39.26
CIFAR-100
20
63.67
15.62
37.89
65.23
18.65
40.62
50
62,40
16.31
37.79
63.87
19.14
39,94
100
63.28
15.23
36.62
66.02
18.85
39.84
I ADOPTING VPSDE(DDPM++) AND EDM MODELS IN DBP
In the previous discussion of the robustness of DBP models, as detailed in Section 4.1 our focus was primarily on the DDPM and DDIM models. We now extend our analysis to include VPSDE (DDPM++) and EDM (Karras et al. 2022 models. VPSDE (DDPM++) is the diffusion model used in DifiPure.
From a unified perspective, the diffusion process can be modeled by stochastic differential equations (SDE) (Song al. (2021b). The forward SDE, as described in Equation (14), converts complex initial data distribution into a simpler, predetermined prior distribution by progressively infusing noise. This also be done in single step, shown in Equation (I5), mirroring the strategy of DDPM described in Equation (2). Reverse SDE, as explained in Equation (16), reverses this process, restoring the noise distribution to the original data distribution, thus completing the diffusion cycle.
dx = f(x, t)dt + g())du.
(14)
(ax())(t) a=)))= N (x(());e (Bman (mmi))mmmm) (amm)-36 318mm xr (0), Ie (. (0m8)x--mi)-BBa Bmmn)- 1 E [0.
(15)
dæ (f g V lo (z)) dt g(t)
(16)
The reverse process of SDEs also derives equivalent ODEs Equation (17) for fast sampling and exact likelihood computation, and these Score ODEs correspond to DDIM.
1 dæ [[or.)) - 91) -0000 (x)) dt.
(17)
By modulating the stochasticity, we can craft a spectrum of semi-stochastic models that bridge pure SDEs and deterministic ODEs, offering a range of stochastic behaviors.
EDM provides a unified framework to synthesize the design principles of different diffusion models (DDPM, DDIM (Nichol & Dhariwal (02I), VPSDE, VESDE (Song et al.) (021b)). Within this framework, EDM incorporates efficient sampling methods, such as the Heun sampler, and introduces optimized scheduling functions (I) and (l).. This allows EDM achieve state-of-the-art performance in generative tasks.
EDM forward process could be presented as:
Tt. 3o a(t)), E~~(0,, I),
(18)
where we choose att*) = 0.5 for clean and robust accuracy tradeoff. And for the reverse process, EDM incorporates parameter Schurn to modulate the stochastic noise infused during the reverse
20

---

Published as a conference paper at ICLR 2025
process. For experiments, choose 50 reverse steps (50 NFES, NFEs is Function of Neural Function Evaluations), configured the parameters with Smin 0.01, Smaz 0.46, Snoise 1.007,
and designate Schurn 0 to represent EDM-ODE. Schurn 6 to represent EDM-SDE.
As shown in Table 15 our ADDT could also increase the robustness DDMM-
Table 15: Clean and robust accuracy (%) on DPEDM for CIFAR-10. ADDT improves robustness in both DEEDM-SDE and DDED-ODE..
Vanilla
ADDT
Metric
DPEDM-DDE
DPEDM-DDR
DPEDM-SDE
DPEDM-ODE
Clean
86.43
87.99
86.33
87.99
80o
62.50
60.45
66.41
64.16
l2
76.86
75.49
79.16
77.15
J STRENGTHENING DBP VIA AUGMENTED STOCHASTICITY
(Song et al. b021b) present a Predictor-Corrector sampler for SDEs reverse process for VPSDE (DDPM++), detailed in Appendix | However, standard implementations of VPSDE (DDPM++) typically use only the Predictor. Given our hypothesis that stochasticity contributes to robustness, we expect that integrating the Corrector sampler into VPSDE (DDPM++) would further enhance the robustness of DBP models. Our empirical results, as shown in Table 16 confirm that the inclusion of Corrector to VPSDE (DDPM++) ++) indeed improves the model's s defense ability against adversarial attacks with fx norm constraints. This finding supports our claim that the increased stochasticity can further strengthen DBP robustness. Adding Corrector also consistent with ADDT. Note that the robustness against & norm attacks does not show significant improvement with the integration of the Extra Corrector. A plausible explanation for this could be that the robustness under le attacks is already quite strong, and the compromised performance clean data counteracts the increase in robustness.
Table 16: Clean and robust accuracy (%) on DP for CIFAR-10. Both extra Corrector and ADDT fine-tun fine-tuning improved robustness.
Metric
Vanilla
Extra Corrector
ADDT
ADDT+Extra Corrector
Clean
89.26
85.25
89.94
85.55
lx
55.96
59.77
62.11
65.23
8
75.78
74.22
76.66
76.66
K DISCUSSION ABOUT RBGM-MAPPED PERTURBATIONS
K. MOTIVATION AND ADVANTAGES oF RBGM
To elaborate, conventional diffusion forward process is based on the equation:
3;, = a 2ig + V at
(19)
where R; represents the noisy image at time f, 2o is the initial input, ā. is a time-dependent scaling factor, and is random Gaussian noise. Our proposed method, ADDT, modifies this equation to include an adversarial component:
c; = Von DD + V1 X- 1 (8)
(20)
In this revised formulation, (ō) represents the adversarial perturbation, and X is a parameter that controls the blend between traditional and adversarial noise. The core objective of ADDT training is
21

---

Published as a conference paper at ICLR 2025
Distribution Normalized Perturbation
Image
Image
Irage
Ubinttution of RG2Mkmmpppe Perlubation
Inage
returbaiinn
Perturtratoon Timesteps
1.50
1.00
3.25
Tinectes
Lo4
Timette
Tiranee 50
30
Peturtalion Moueh
Figure 9: Raw perturbation values exhibit diverse distributions across images and time steps. RBGM maps these perturbations to uniform Gaussian distribution.
to generate perturbations that emulate the charucteristies of Gaussian noise conventional diffasion training while incorporating adversarial disturbances. This leads to our Rank- Gaussian Mapping (RBGM) technique, which preserves the relative ordering of perturbation magnitudes while adjusting their values more closely resemble Gaussian distribution. The advantages of RBGM are twofold:
Enhancing statistical consistency. Raw adversarial perturbation values often exhibit non-standard distributions, and RBGM serves to recalibrate these perturbations, aligning them more closely with Gaussian distribution. To elaborate, rather than enforeing multivariate Gaussian distribution for the entire perturbation, RBGM ensures that the distribution of individual perturbation values adheres to Gaussian characteristics.
The benefit of this transformation can be illustrated in Figure and Figure I0 For fair comparison, the perturbation values have been normalized. In Figure 9 the original perturbation values display wide array of distributions across different images and time steps. After the mapping of RBGM, these values are transformed to exhibit uniform Gaussian distribution.
In Figure 10 the raw perturbations show irregular and inconsistent behavior when mixed with Gaus- sian noise at varying ratios. However, after RBGM mapping, the perturbations and the mixture exhibit consistent statistics with the pure Gaussian noise. The statistical consistency of the perturbation values may ease the training of the diffusion model and avoid significant deviation from the normal diffusion process.
Reducing image-specific dependence. In the training of diffusion models, the Gaussian noise independent of specific images time steps. This characteristic contrasts with adversarial perturbations, which are typically tailored to each input. RBGM addresses this issue by introducing stochasticity into the construction of perturbations while preserving the ranks of the values of the inage-dependent adversarial perturbations. This approach effectively reduces image-specific dependence. Consequently, RBGM enhances ADDT's ability to better mimic the diffusion training process and potentially mitigates the overfitting of training images.
K.2 RBGM-MAPPED PERTURBATIONS PRESERVE ADVERSARIAL. CHARACTERISTICS
While RBGM mappedd perturbations are "selected from a Gaussian distribution". their actual dis- tribution deviates from a pure Gaussian distribution and is adversarial for models. To substantiate
22

---

Published as a conference paper at ICLR 2025
Varue distibution changed by maing rato
Variia
PERGM
Wise distibttion remains constant
0 Asuaa)
1000 Avan)
More similar Generated Portubaiin
More smilarto Nolso
dfferent
Figure 10: RBGM ensures that mixing perturbations with Gaussian noise at any ratio yields a consistent value distribution.
this claim, compare the influence of RBGM-mapped perturbations and Gaussian noise model performance. In our experiments, we perturb clean images by adding RBGM-mapped perturbations and Gaussian noise, each scaled by factor of 0.03. The results present in Table 17 demonstrate that BBGM-mapped perturbations effectively act as adversarial inputs to the model. These perturbations drastically reduce the accuracy of pre-trained clean WRN-28-11 10 from 95.12% 4.47%.
Table 17:
Comparison model accuracy % under different conditions. RBGM--apped perturbations lead to significant reduction in accuracy compared to Gaussian noise.
Classifier
Clean
Gaussian noise
RBGM-mapped perturhation
WRN-28-10 95.12
81.54
4.47
K.3 BLENDING ADVERSARIAL PERTURBATIONS INTO DIFFUSION MODEL. TRAINING
In conventional adversarial attacks, perturbations are directly applied to the image, resulting in an adversarial image:
Cadv 0o + 8,
(21)
where is the original input image, and 5 is the adversarial perturbation. In ADDT, we incorporate this concept into the diffusion process, redefining the noisy image at time step as:
zt (a, + 8) + v1 a.e,
(22)
To enable the diffusion model to effectively purify adversarial perturbations during training, we reformulate the above equation by merging the perturbation 5 with the noise e. This results in:
var To + vi- Ci + 7/ 5),
(23)
where is scaling factor defined as:
2
(24)
VI a
Since a is time-dependent parameter that monotonically decreases from 1 to 0 during the diffusion process, 7t spans the range from 0 to cc. To ensure the adversarial perturbation remains within manageable intensity, we constrain its value to the range between Amin and Xmax-
23

---

Published as a conference paper at ICLR 2025
K.4 RBGM ENHANCES PERTURBATION COMPATIBILITY WITH DIFFUSION MODEL TRAINING
To illustrate how RBGM enhances the compatibility of perturbations with diffusion model training, we conduct comparative analyses in two scenarios. First, we assess the impact RBGM on statistical consistency by comparing Gaussian noise with adversarial perturbations reordered based on Gaussian noise ranks. Second. we evaluate the effectiveness of RBGM -mapped perturbations in improving model robustness while maintaining generative performance by comparing them with ez-normalized perturbations.
Gaussian noise vs. adversarial perturbations ordered by Gaussian noise We begin by examining RBGM's influence on statistical consistency through two training methodologies:
1. Vanilla: Trained with standard Gaussian noise.
2. GDDTGansiin reordered: Trained with adversarial perturbations reordered according to Gaus- sian noise ranks. To ensure fair comparison, the perturbations are normalized to have a mean of 0 and a variance of I, as their original magnitudes (derived from accumulated gradients) are significantly smaller than those of standard Gaussian noise. Note that this approach-reordees adversarial perturbations based on Gaussian noise ranks-is distinct from RBGM, where Gaussian noise is reordered based on adversarial perturbation ranks.
The results presented in Table [8 reveal that models trained with Gaussian noise reordering using adversarial perturbation values exhibit lower accuracy on both clean and adversarial samples compared to vanilla models. This decline in performance underscores RBGM's ability to enhance perturbation compatibility with diffusion model training by improving statistical consistency.
Table 18: Comparison of DPDDPM accuracy under different conditions and perturbation types. Train- ing with perturbations reordered by Gaussian noise and adversarial perturbation values degrades performance.
Vanilla foo
ADDT Caussinn reordered
ADDT
NFEs
Clean
f
Cleon E
&g
49.51
21.78
36.13
48.40
21.10
33.40
73.34
36.72
55.47
71.78
34.07
52.98
20
81.45
45.21
65.23
79.99
42.43
64.21
50
85.54
46.78
68.85
83.90
46.73
68.17
100
85.94
47.27
69.34
84.54
46.98
69.33
RBGMM-mapped perturbations vs. fg-normalized perturbations To further investigate RBGM's effectiveness in making adversarial perturbations compatible with diffusion model training, we compare:
1. ADDT: Trained with RBGM-mapped perturbations.
2. ADDT t--normalized: Trained with raw adversarial perturbations, scaled match the & norm of standard Gaussian noise. This scaling ensures that the perturbations share the same & norm as those mapped by RBGM, which we refer to eg-nomallieed perturbations.
As shown in Table models trained with normalzle perturbations tend to perform better under l2 attacks in some scenarios (possibly because these perturbations are more similar to those generated by f2 attack during testing). ADDT generally achieves better results. This advantage particularly pronounced under fx attacks and in scenarios with higher NFEs. Furthermore, shown in Table 20 ADDT yields a lower FID value, reflecting better preservation generative capabilities.
As discussed in Appendix K.I the goal of utilizing RGBM to generate perturbations is mimic the characteristic of Gaussian noise, hence aligning ADDT closer to traditional diffusion model training. In this context, both RBGM and lg normalization RBGM provides a more precise approximation, enhancing robustness and maintaining the generative performance more effectively than &g normalization.
as approximations of Gaussian noise. Yet.
24

---

Published as a conference paper at ICLR 2025
Table 19: Comparison of DDPM accuracy under different perturbation conditions. Training with ADDT generally achieves better results. This advantage is particularly pronounced under & attacks and in scenarios with higher NFEs.
NFEs
ADDT too
ADDT
E e-nommalize
Clean
82
Clean
8
8
59.96
30.27
41.99
60.40
28.47
44.58
10
78.91
43.07
62.97
79.29
41.90
63.72
20
83.89
48.44
69.82
83.59
47.85
67.68
85.45
50.20
69.04
84.83
49.12
69.24
100*
85.64
51.46
70.12
84.97
49.95
69.29
Table 20: FID score of DDPM for CIFAR-10 fine-tuned under different conditions. Fine-tuning with ADDT results in lower FID score compared to Eg normalization.
Clean fine-tuning
ADDT
ADDT noromallize
FID
3,50
5.190
5.678
L COMPARING REGM-MAPPEL PERTURBATIONS WITH &oo PERTURBATIONS
Section 6.3 we briefly explore the generation capabilities diffusion models trained with RBGM- mapped and co perturbations. Here, provide further experiments and delve deeper into their robustness comparison. To train with for perturbations, we adjust ADDT, replacing RBGM-mapped perturbations with for perturbations. Here, instead of converting accumulated gradients Gaussian- like perturbations, we use a 5-step projected gradient descent (PGD-55 approach. For fair comparison, Amaz 10 and refer this modified training protocol as ADDT,
we also Aunit 1, Aman
We evaluate the clean and robust accuracy of ADDT and ADDT; fine-tuned models. These models exhibit different behaviors. As shown in Table 22 while Gaussian-mapped perturbations can simultaneously improve clean accuracy and robustness against both & and foc attacks, training with lo perturbations primarily improves performance against foo attacks.
Table 21: Clean and robust accuracy (%) DBP models trained with different perturbations for CIFAR-10. 10. While ADDT simultaneously improves clean accuracy and robustness against both & and lo attacks. ADDT, to primarily improves performance against &oo attacks.
Vanilla too
ADDT foc
ADDT, 800
DBP method
Dataset
Clean
f
Clean
8
Clean
loc
82
CIFAR- 10 CIFAR-100
85.94
47.27
69.34
85.64
51.46
70.12
84.47
52.64
68.55
DPPM
57.52
20.41
37.89
59.18
23.73
41.70
57.81
23.24
40.04
CIFAR-10 10 CIFAR-100
88.38
42.19
70.02
88.77
46.48
71.19
88.48
50.49
70.31
DP
63.28
15.23
36.62
66.02
18.85
39.84
64.84
20.31
39.36
M ADDITIONAL EXPERIMENTS UNDER DETERMINISTIC WHITE-BOX SETTING
Evaluation across different NFEs We also investigate the robustness under the Deterministic White-box Setting across varying NFEs. The comparison of performance between vanilla models and ADDT fine-tuned models, shown in Figure [I] highlights that ADDT consistently enhances model performance at different NFEs. This improvement particularly pronounced at lower NFEs, further confirming that ADDT enables diffusion models more effectively counter adversarial perturbations.
25

---

Published as a conference paper at ICLR 2025
Table 22: DW-box accuracy (%) under to perturbations for various models. ADDT consistently improves robustness across all models.
DBP method
Vanilla
ADDT
DDDM
16.80
39.16
DDID
4.98
17.09
DiftPure
22.76
51.63
DPEDM
13.33
32,94
Accuracy
40
ADDT, White-box (EoT)
30
ADDT, DW-box
Vanilla, White-box (EoT)
20
Vanilla, DW-hox
10
10a
10a
NFES
Figure 11: Revisiting Deterministic White-box Robustness. ADDT consistently improves robustness under both White-box and Deterministic White-box settings, implying that ADDT strengthens the models' ability to handle adversarial inputs.
N SENSITIVITY ANALYSIS OF Aunit
we choose =0.03 as the magnitude of such noise similar to the adversarial perturbation of common adversarial attack setting. We also provide an ablation study here, which shows that the performance of ADDT is insensitive to Aunit and gets consistent improvement.
In Section 6
Table 23:
The performance of ADDT fine-tuned DDPM with varying Aunit values, ADDT shows insensitivity to Aunit and consistently improves robust accuracy (%).
too
NFEs
Vanilla
0.02
0.03
0.04
Vanilla
0.02
0.03
0.04
21.78
24.02
30.27
31.25
36.13
41.99
41.99
49.02
10
36.72
40.92
43.07
44.92
55.47
61.72
62.97
64.45
20
45.21
48.14
48.44
50.68
65.23
67.48
69.82
69.24
50
46.78
48.83
50.20
51.07
68.85
69.82
69,04
69.53
100*
47.27
48.93
51.46
50.88
69.34
70.31
70.12
69.92
O COMPUTATIONAL COST ANALYSIS FOR TRAINING AND INFERENCE
Fine-tuning DDPM and DDIM models using ADDT to achieve near-optimal performance requires 50 epochs and approximately 12 hours of training on 4 NVIDIA GeForce RTX 2080 Ti GPUs. This efficiency matches that of traditional adversarial training approaches and is notably faster than recent adversarial training techniques that utilize diffusion models for dataset augmentation (Wang et al.) 2023). However, testing DP and DDDDIM involves significant computational expense due to the use of Expectation over Transformation (EoT). For instance, validating 1,024 images on the (IFAR10/CIFAL 100 datasets takes approximately 5 hours the same GPU configuration.
One the key advantages of ADDT its "train-once" methodology. Once the initial training is complete, ADDT protect multiple classifiers without requiring additional fine-tuning as demonstrated Table 2 This is stark contrast to adversarial classifier training, where each classifier demands individual training.
26

---

Published as a conference paper at ICLR 2025
During inference, models trained with ADDT have similar complexity to standard DBP. However, their performance gains in accelerated scenarios offer the potential for a reduction in computational overhead. As shown in Table 4 DP ADDT achieves comparable performance to DP while requiring only 20 NFEs, resulting in up to an 80% reduction in computation time compared to the 100 NFEs required for DPDDPM-
P CREDIBILITY OF OUR PAPER
The code was developed independently by two individuals and mutually verified, with consistent results achieved through independent training and testing. The open-source code is available at https//gghbbbcoom///DDLUUA/A
Q BROADER IMPACT AND LIMITATIONS
Our work holds significant potential for positive societal impact across range of sectors, including autonomous driving, facial recognition payment systems, and medical assistance. We are dedicated enhancing the safety and trustworthiness of global Al applications. However, acknowledge the potential negative societal impacts, particularly concerning privacy protection, due to adversarial perturbations. Despite this, we believe that the overall positive impacts outweigh these risks.
In terms of limitations, our approach could benefit from incorporating insights from traditional adversarial training methods (Zhang et al.) 2019)] Shafahi al., 2019) Wang al.) (003), such as through more extensive data augmentation and refined ADDT loss design. Nevertheless, these limitations are relatively minor and do not diminish the overall contributions of this paper. We believe these findings and perspectives will have sustained impact future research in DBP, which a promising approach to adversarial defense and could prove even more valuable for real-world applications, despite the early stage of existing studies DBP.
27


Reviews:
[{"review_id": "MRHwrz5ECR", "reviewer": "Reviewer_TLr9", "summary": "The author argues that the current effectiveness of the DBP method, attributed to the forward process of diffusion models, lacks sufficient empirical validation. While they acknowledge the theory of mitigating distribution gap, they question whether the forward and backward diffusion stochasticity in DBP is indeed what enhances model robustness. To address this, they propose a new attack evaluation, the DW-Box attack, which captures both stochasticity information and parameters of defense system. Experiments reveal that models like DDPM and DDIM lack strong robustness against this new attack. Additionally, the authors introduce ADDT, a fine-tuning technique for diffusion models in the existing DBP methods. This approach first optimizes the generated adversarial perturbation using classifier guidance, and then performs further training on images with both Gaussian noise and optimized perturbations. They aim to enhance the purification capability of diffusion models. This method significantly enhances the robustness of existing DBP methods. Overall, the paper critically examines the underlying principles that make DBP methods effective.", "questions": "1. As the perturbation after Rand-Based Gaussian Mapping is still a Gaussian noise, can I view the idea of ADDT is just using further perturbed images for additional training, if so, is the improvement relying on the additional data?\n\n2. What is the reason of choosing such a range for the modulation term in line 362-364?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The work has included extensive experiment contents, such as exploration on accelerated generation of diffusion model, stronger white-box attack with more attack iterations and included various dataset results.\n\n2. The author proposed an appropriate attack evaluation, DW-box attack, which makes the performance comparison reliable.\n\n3. The ADDT technique has improved the robustness of all baseline DBP methods.", "weaknesses": "1. The author should provide a few more comparisons for the same content in figure 6, which shows the performance of the ADDT fine-tuning under the proposed DW-box attack. \n\n2. In figure 3, it is not convincing if the author only draws one complete attack trajectory. \n\n3. The whole idea can be viewed as an additional training by using samples with more Gaussian noises at the same timestep, hence the diffusion model achieves better purification ability, I would like to see whether the RBGM perturbation is different with using a pure additional Gaussian noise with trivial order. Hence, we cannot say the model has improved its ability of enhancing purification of adversarial perturbations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vc7rOwyKbK", "reviewer": "Reviewer_gEWD", "summary": "This paper investigates the robustness of Diffusion-Based Purification (DBP) methods in adversarial attacks, proposing that its defensive effectiveness relies primarily on stochasticity rather than the traditional noise ablation process. By introducing a deterministic white-box evaluation approach, the authors demonstrate the critical role of stochasticity in DBP's robustness. To enhance DBP's denoising capability, the paper presents Adversarial Denoising Diffusion Training (ADDT) and Rank-Based Gaussian Mapping (RBGM), with experimental results showing significant improvements in defense performance. The authors suggest that future research should focus on decoupling stochasticity from purification capability to optimize defensive performance.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper identifies that the robustness provided by diffusion models is likely due to stochastic elements within the model rather than solely the diffusion process.\n2. The authors propose a new training method for diffusion models that incorporates robustness information, enhancing the model's purification capability.", "weaknesses": "1. While the study contributes positively to understanding purification using diffusion models, there are concerns about the complexity of purification. It significantly increases the model's complexity, potentially multiple times compared to the original model, and this real-world cost needs to be experimentally clarified (complexity analysis).\n2. If the training data for the diffusion model originates from the same data as the original model, a comparison with methods using the diffusion model to generate new samples for robust model training would be valuable, as this approach does not add runtime complexity [1].\n3. Questions remain about why perturbations are not directly modified by mean and variance adjustments to achieve Gaussian distribution, instead requiring replacement.\n\n**Reference**:\n\n[1] Wang, Zekai, et al. \"Better diffusion models further improve adversarial training.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ww7XhT7QQ0", "reviewer": "Reviewer_nzsE", "summary": "The paper explores the reason behind the success of Diffusion-Based Purification (DBP) as a defense against adversarial attacks, highlighting that its effectiveness comes from randomness rather than just reducing the difference between clean and adversarial images. The authors introduced a new evaluation method, Deterministic White-Box (DW-box), and found that DBP’s ability to purify adversarial perturbations is weak without stochasticity. To improve DBP, they developed Adversarial Denoising Diffusion Training (ADDT), which incorporates adversarial examples into the training process, and Rank-Based Gaussian Mapping (RBGM), which makes adversarial perturbations more Gaussian-like. Their experiments showed that these methods enhance DBP’s robustness.", "questions": "1) Based on the conclusion of your paper that stochasticity is the main ingredient of the success of the DBP, do you think a diffusion model even necessary for the defense? What if we just add a certain amount of gaussian noise directly to the adversarial example and then just denoise it with an off-the-shelf image denoising network that was specifically trained to remove gaussian noise? Do you think it will work just as well as a defense method?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is well written and relatively easy to follow.\n2) Novel perspective on why Diffusion-Based Purification works and how it can be further improved.\n3) Claims are well supported by experiments.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vrw4YjsgTN", "reviewer": "Reviewer_mx31", "summary": "This paper examines how stochastic elements contribute to robustness in DiffPure by systematically ablating these components (DW-box). The authors propose incorporating adversarial training into DiffPure and addressing the \"robustness-generalization tradeoff\" through a novel technique, RGBM, which preserves both robustness and generative performance.", "questions": "I'm quite curious about the following (I would like to raise my score if the authors could provide more comprehensive studies on the mechanism of their proposed methods):\n\n- Are the points in Fig. 1 obtained by sampling at intervals of a certain number of attack steps? If so, does this imply that the attack process initially moves the farthest distance and moves the fastest, while the loss increases the slowest, and then the movement gradually slows as time progresses but the loss increases more rapidly?\n\n- How did you come up with RGBM? What inspired this approach?\n\n- RGBM uses values from a Gaussian distribution but follows the order of adversarial perturbations. What would happen if we used the order of Gaussian noise but the values of adversarial perturbations? I’m curious if the main mechanism of RGBM is to eliminate large (outlier) values.\n\n- I am very interested in whether \"ADDT w/o RGBM\" and \"ADDT\" in Table 6 were fine-tuned from pretrained models or trained from scratch. Was \"ADDT w/o RGBM\" fine-tuned from \"ADDT\"?\n\n- Can I understand RGBM as maintaining robustness while improving generalization and generation performance across different attacks?\n\n- I am very curious about why RGBM works. First, I notice that the result mapped by RGBM does not follow a Gaussian distribution, nor even a non-isotropic Gaussian (since each element is not Gaussian). Thus, the RGBM-mapped result is a biased estimator rather than an unbiased one. The only similarity I see to a Gaussian distribution is that, like a high-dimensional Gaussian, it also lies on a spherical shell. Is this the main reason it preserves the generalization of pretrained diffusion models? If so, simply \"normalizing\" the adversarial perturbation to an \\(\\ell_2\\) norm of \\(O(\\sqrt{D})\\) should also work (since the \\(\\ell_2\\) norm of a high-dimensional Gaussian converges to \\(\\sqrt{D}\\)). A simple way like yours (your illustration) to understand this is first to sample a Gaussian noise, compute its norm, and normalize the adversarial perturbation to this norm. Could you provide an ablation study on this?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I like the following contributions in this paper:\n\n- The authors systematically study how each of the stochastic elements in DiffPure contributes to robustness, including the \"forward process,\" \"backward process,\" and the combination of both, clearly demonstrating how much each component contributes to robustness.\n\n- The authors propose RGBM, which significantly alleviates the previous limitation of adversarial training compromising the generative ability of diffusion models.\n\n- Fig. 5 is excellent. Without Fig. 5, it would be challenging to understand RGBM clearly due to existing ambiguities (see question 3 on the two types of RGBM).", "weaknesses": "There are no severe weaknesses in this paper, but one notable issue:\n\n- \"Note that DDPM is mathematically equivalent to the DDPM++ (VPSDE) model used in DiffPure but employs a smaller UNet architecture. DDIM, on the other hand, is a mathematical improvement over DDPM that facilitates an ODE-based diffusion process.\" This statement lacks formality. DDPM is a mathematical framework independent of neural network architecture. Additionally, the DDPM framework represents only one discretization of the VP-SDE, while other discretizations also exist. The authors should address this non-formal claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)