Sample-specific Noise Injection for Diffusion-based Adversarial Purification
Authors: Yuhao Sun
Jiacheng Zhang
Zesheng Ye
Chaowei Xiao
Feng Liu
Venue: ICML 2025 poster (2025)
Keywords: adversarial purification
adversarial robustness
diffusion-based adversarial purification
accuracy-robustness trade-of
Primary area: deep_learning->robustness

Abstract: *Diffusion-based purification* (DBP) methods aim to remove adversarial noise from the input sample by first injecting Gaussian noise through a forward diffusion process, and then recovering the clean example through a reverse generative process. In the above process, how much Gaussian noise is injected to the input sample is key to the success of DBP methods, which is controlled by a constant noise level $t*$ for all samples in existing methods. In this paper, we discover that an optimal $t*$ for each sample indeed could be different. Intuitively, the cleaner a sample is, the less the noise it should be injected, and vice versa. Motivated by this finding, we propose a new framework, called Sample-specific Score-aware Noise Injection (SSNI). Specifically, SSNI uses a pre-trained score network to estimate how much a data point deviates from the clean data distribution (i.e., score norms). Then, based on the magnitude of score norms, SSNI applies a reweighting function to adaptively adjust $t*$ for each sample, achieving sample-specific noise injections. Empirically, incorporating our framework with existing DBP methods results in a notable improvement in both accuracy and robustness on CIFAR-10 and ImageNet-1K, highlighting the necessity to allocate *distinct noise levels to different samples* in DBP methods. Our code is available at: https://github.com/tmlr-group/SSNI.

Sample-Specific Noise Injection For Diffusion-Based Adversarial Purification
Yuhao Sun Jiacheng Zhang"
Zesheng Ye"
Chaowei Xiao Feng Liu
Abstract
predictions by adding imperceptible adversarial noise to clean examples, which pose significant security threat in critical applications (Dong et al., 2019; Cao et al., 2021; Jing et al., 2021; Han et al., 2025). To defend against ad- versarial examples, adversarial purification (AP) stands out as a representative defensive mechanism, by leveraging pre- trained generative models to purify adversarial examples back towards their natural counterparts before feeding into pre-trained classifier (Yoon et al., 2021; Nie et al., 2022). Notably, AP methods benefit from their modularity, as the purifier operates independently of the downstream classifier, which facilitates seamless integration into existing systems and positions AP as practical approach to improve the adversarial robustness of DNN-based classifiers.
Diffusion- based purification (DBP) methods aim to remove adversarial noise from the input sample by first injecting Gaussian noise through for- ward diffusion process, and then recovering the clean example through reverse generative pro- cess. In the above process, how much Gaussian noise injected to the input sample is key to the success of DBP methods, which is controlled by a constant noise level t* for all samples in existing methods. In this paper, we discover that opti- mal 1o for each sample indeed could be different. Intuitively, the cleaner sample is, the less the noise it should be injected, and vice versa. Mo- tivated by this finding, we propose a new frame- work, called Sample-specific Score-aware Noise Injection (SSNI). Specifically, SSNI uses a pre- trained score network estimate how much a data point deviates from the clean data distribu- tion (i.e., score norms). Then, based on the magni- tude of score norms, SSNI applies a reweighting function to adaptively adjust t* for each sample. achieving sample-specific noise injections. Empir- ically, incorporating our framework with existing DBP methods results in notable improvement in both accuracy and robustness on CIFAR-10 10 and ImageNet- K, highlighting the necessity to allo- cate distinct noise levels to different samples in DBP methods. Our code is available at: https: / //github. cttp:/..oommmmmmmmeem wwtp:/oooootoooooo//oooo SSNI.
Recently diffusion- purification (DBP) methods have gained much attention promising framework in AP, which leverage the denoising nature of diffusion models to mitigate adversarial noise (Nie et al., 2022; Xiao et al., 2023; Lee & Kim, 2023). Generally, diffusion models train a forward process that maps from data distributions to simple distributions, e.g., Gaussian, and reverse this mapping via reverse generative process (Ho et al., 2020; Song et al., 2021b). When applied in DBP methods, the forward process gradually injects Gaussian noise into the input sample, while the reverse process gradually purify noisy sample to recover the clean sample. The quality of the purified sample heavily depends on the amount of Gaussian noise added to the input during the forward process, which can be controlled by a noise level parameter t*. Existing DBP methods (Nie et al., 2022; Xiao et al., 2023; Lee & Kim, 2023) manually select constant t" for all samples.
1. Introduction
However, we find that using a sample-shared t* may over- look the fact that an optimal 1" indeed could be different at sample-level, as demonstrated in Figure 1. For exam- ple, in Figure la, t 100 is too small, resulting in the adversarial noise not being sufficiently removed by the dif- fusion models. This is because diffusion models are good at denoising samples that have been sufficiently corrupted by Gaussian noise through the forward process (Ho et al., 2020; Song et al., 2021b). With a small t", the sample re- mains insufficiently corrupted, which limits the denoising capability the reverse process and thereby compromising the robustness against adversarial examples. On the other hand, in Figure 1b and Ic, t" = 100 is too large, resulting
Deep neural networks (DNNs) are vulnerable to adversarial examples, which is a longstanding problem in deep learning (Szegedy et al., 2014; Goodfellow et al.. 2015). Adversar- ial examples aim to mislead DNNs into making erroneous
'Equal contribution SSchool of Computing and Infor- mation Systems. The University of Melbourne Univeersity of Wisconsin, Madison. Correspondence to: Feng Liu <fengliu.ml@ waw.:mmm@@mmammmmm.... com..
Proceedings of the 4rnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sample-Specific Noise Injection For Diffusion-Baree Adversarial Purification
100
100
1-0
100
t=
Ayd
Proficion: Deer
Prodetion Frog
Vlle
Prekilee
Dog
L-0
>1- 120
L--
1-0
1" 60
11-
30 30
[-0
Sad
Proficiio::
Prodetion
Nale
Prodictin:
Bird
Dog
Cat
(a)
(b)
(c)
Figure 1. For each sub siguri the Ist column contains the input (i.c., could either be AEs or CEs), the 2nd column contains noise-injected examples with different i' and the 3rd column contains purified examples. We use DiffPure (Nic al., 2022) with a sample- shared X" = 100 selected by Nie et al. (2022) to conduct this experiment on CIFAR-10 (Krizhevsky et al., 2009). The globally shared L* = 100 offers baseline, but results in suboptimal prediction performance compared what could be achieved by tuning the noise level for individual samples. Notably, while the recovered images obtained by different noise levels may be visually indistinguishable. they earry different semantics. For instance, the image classified as "frog" (incorrect) with t" 100 but "dog" (correct) with t 60 (Figure lb). These highlight the need for sample-wise noise level adjustment.
in excessive disruption of the sample's s semantic informa- tion during the forward process, which makes it difficult to recover the original semantics in the reverse process. In this case, both robustness and clean accuracy are compromised, as the purified samples struggle to preserve the semantic consistency of clean samples. These observations motivate us make the first attempt to adjust the noise level on samplc-specific basis.
such as CIFAR-10 (Krizhevsky et al., 2009) and ImageNet- IK (Deng et al., 2009), we demonstrate the effectiveness of SSNI in Section 5. Specifically, combined with different DBP methods (Nie et al., 2022; Xiao et al., 2023; Lee & Kim, 2023), SSNI can boost clean accuracy and robust accuracy simultaneously by notable margin against the well-designed adaptive white-box attack (see Section 5.2).
The success of SSNI takes root in the following aspects: (I) an optimal noise level t* for each sample indeed could be different, making SSNI an effective approach to unleash the intrinsic strength of DBP methods; (2) existing DBP meth- ods often inject excessive noise into clean samples, resulting in a degradation in elean accuracy. By contrast, SSNI injects less noise clean samples, and thereby notably improving the clean accuracy. Meanwhile, SSNI can effectively handle ad adversarial samples by injecting sufficient noise on each sample; (3) SSNI is designed as general framework instead of a specific method, allowing it to be seamlessly integrated with a variety of existing DBP methods.
In this paper, we propose Sample-specific Score-awar aware Noise Injection (SSNI), a new framework that leverages the distance of a sample from the clean data distribution to adaptively adjust on a sample-specific basis. SSNI aims to inject less noise to cleaner samples, and vice versa.
To implement SSNI, inspired by the fact that scores (i.e., Vxlog (x)) reflect the directional momentum of sam- ples toward the high-density areas of clean data distribu- tion (Song & Ermon, 2019), we use score norms (i.e., [x as a natural metric to measure the deviation of data point from the clean data distribution. In Section 3, we establish the relationship between the score norm and the noise level required for different samples. Specifically, sam- ples with different score norms tend to have accumulated different noise levels. Furthermore, we empirically show that the cleaner samples those closer to the clean data dis- tribution - exhibit lower score norm, justifying the rationale of using score norms for reweighting t². Concretely, we use a pre-trained score network estimate the score norm for each sample. Based this, we propose two reweighting functions that adaptively adjust t* according to its score norm, achieving sample-specifie noise injections (see Sec- tion 4.3). Notably, this reweighting process is lightweight, ensuring that SSNI is computationally feasible and can be applied in practice with minimal overhead (see Section 5.6).
2. Preliminary and Related Work
In this section, we first review diffusion models and scores in detail. We then review the related work of DBP methods. Detailed related work can be found in Appendix A.
Diffusion models are generative models designed to ap- proximate the underlying clean data distribution p(xp). by learning parametric distribution (x(Xx) with a forward and a reverse process. In Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020), the forward process of a diffusion model defined on X C Rd can be expressed by:
q(x+(xo) N Xxi Vāxo, (1 a.))).
(I)
Through extensive evaluations benchmark image datasets
where ār = I -3) and (B. (B:)e(o.rT are predefined
2

---

Sample-Specific Noise Injection For Diffusion-Barse Adversarial Purification
noise scales with 3. (0,1) for all I. As increases, X converges toward isotropic Gaussian noise. In the reverse process, DDPM seeks to recover the clean data from noise by simulating a Markov chain in the reverse direction over T steps. The reverse transition at each intermediate step is modeled by
Perturbaiin Budgees
hot
2/255
4/253
6/255
8/255
900
850
750
soore Norn
Po (x 1xx)) = (Xe- (x:,
700
650
where Us (xx, V1-3, X VI-a. (x(x)) t) predicted mean at i, and 0r is fixed variance (Ho al., 2020). More specifically, the model learns to predict the noise Eg- (x,,)) t) added to the data. The training objective minimizes the distance between the true and predicted noise, accounting for different noise levels controlled by to
is the
vi-a.
550
500
10 20
10
40
50
60
Image Index
Figure 2. Relationship between score norms and perturbation bud- gets. We use one batch of clean data from CIFAR-10 and employ PGD+FOT (e 8/255) as the attack.
Φ arg min Ex..,, [||e - cs (Varxo + V1 M [12] &
The generative process starts from pure noise XT ~ NO I) and progressively removes noise through T steps, with a random noise term E ~ NO(, I) incorporated at each step:
with pre-trained diffusion model by first corrupting an adver- sarial input through the forward process and then iteratively reconstructing it via the reverse process. This projects the in- put back onto the clean data manifold while stripping away adversarial artifacts. Wang et al. (2022) introduce input guidance during the reverse diffusion process to ensure the purified outputs stay close to the inputs. Lee & Kim (2023) propose fine-tuned gradual noise scheduling for multi-step purifications. Bai et al. (2024) improve the reverse diffusion process by incorporating contrastive objectives.
3e
Eg- (2..1) + v B.e. ae
81-1=
&
VI- D
Score and score norm. In diffusion models, score refers to the gradient of the log-probabilipg density Vxlog Pe (xx, which maps the steepest ascent direction of the log-density in the probability space (Song& & Ermon, 2019). The asso- ciated score norm || Volog p[x)]| measures the gradient's ss magnitude, reflecting how much a data point deviates from the clean data distribution: points located in low ooo-probayility regions typically exhibit larger score norms, while those sit- uated closer to high-density regions of clean data manifest smaller score norms. This property is valuable for adversar- ial example detection, as demonstrated by Yoon et al. (2021) who establish that score norms can effectively distinguish between adversarial and clean examples. However, direct computation of the score is often intractable for complex data distributions (e.g., images). In practice, it is estimated using neural networks so X' R2+ X' that take input both a data point x and a specified timestep 1 E IR, to approximate V Vxlog Pa (x) by minimizing the Fisher diver- gence between the true and estimated scores. Also, so (x, /5) enables efficient computation of V/ p(x)]l.
3. Motivation
We now elaborate on the motivation of our method by con- necting the impact of different perturbation budgets e to the required noise level t" of each sample through score norm.
Sample-shared noise-level 1. fails to address diverse ad- versarial perturbations. We empirically observe that an optimal noise level t* for each sample indeed could be differ- ent: While a constant noise level Nie et al. (2022), yields strong performances for certain sam- ples, it leads to suboptimal results for others (from Figure 1), Since DBP relies on adequate t* for forward noise injection to remove adversarial perturbation, shared t* cannot adapt to the distinct adversarial perturbations of individual exam- ples, leading to a suboptimal accuracy-robustnes trade-off. Specifically, Figure la shows that t* 100 is insufficient to remove the adversarial noise, leaving residual perturbations that compromise robustness. In contrast, t" = 100 overly suppresses the other sample's semantic information during the forward process, making it difficult to recover the origi- nal semantics via the reverse process (Figure 1b- These findings highlight the need for sample-spectes noise injec- tion levels tailored to individual adversarial perturbations.
100, as suggested by
Diffusion-based purification. Adversarial purification (AP) leverages generative models as an add-on module to purify adversarial examples before classification. Within this context, diffussin--bsee purification (DBP) methods have emerged as a promising framework, exploiting the inherent denoising nature of diffusion models to filter out adversarial noise (Nie et al., 2022: Wang et al., 2022; Xiao et al., 2023; Lee & Kim, 2023). Specifically, DBP works
3

---

Sample-Specific Noise Injection For Diffusion-Bas Based Adversarial Purification
Input
Diflused Images with Sample Specific t(x;
Purified Images
Predicted Labels
Off-the-shelf Diffusion Model
Forward
{(21)= tj
Reverse
Hummingbird
Forward
Reverse
222
1(zz) =
Off-the-shelf Classifler
Goose
Forward
t(#x)) t
Reverse
Wombat
Score Network 8y
Score
|=o(x)]
Reweighting Function f
Norm
Figure 3. An overview of the proposed SSNI framework. SSNI introduces novel sumple-specifie mechanism to adaptively adjust the noise injection level for each sample, enhancing purification effectiveness in DBP methods. The process begins by forwarding each sample image x; through the forward diffusion process using an off-the-shelf diffusion model. To determine how much noise to inject into each sample, SSNI employs pre-trained score network se compute the from the clean data distribution. Based on this score norm, rewcighting function adaptively determines the optimal noise level r for each sample. Finally, each sample purified through reverse diffusion process before being classified. Notahly, SSNT designed general framework rather than a specific method, which can be seamlessly integrated with a wide range of existing DBP methods.
[==)))))ll which reflects the distance of the sample
Score norms vary across perturbation budgets. Adversar- ial and clean examples are from distinct distributions (Gao et al., 2021). Motivated by this, we further investigate how different perturbation budgets affect score norms under adversarial attacks (Figure 2). Specifically, we compute the score norm of different samples undergoing PGD+EOT (c 8/255) with perturbation budgets varying between O and 8/255 CIFAR-10 10. We observe consistent pat- tern: score norms scale directly with perturbation strength: larger values lead to higher norms, whereas smaller per- turbations (i.e., cleaner samples) lead to lower norms The findings extend the role score norms from differentiating adversarial/clean samples (Yoon et al., 2021) to differentiate adversarial examples based on their perturbation strength.
the SSNI framework, followed by connection with existing DBP methods and the empirical realization of SSNI.
4.1. Framework of SSNI
Overview. SSNI builds upon existing DBP methods by reweighting the optimal noise level t" from global, sample- shared constant to sample-specific quantity. At its core, SSNI leverages score norms to modulate the noise injected into each sample during diffusion, ensuring more targeted denoising process tailored each sample. We visually illustrate SSNI in Figure 3, and describe the algorithmic workflow in Algorithm
DBP with sample-shared noise level t'. Existing DBP methods use an off-the-shelf diffusion model for data pu- rification, and a classifier responsible for label prediction. Let y be the label space for classification task. Denote the forward diffusion process by D process by R X X, and the classifier by C X -> y. For classification, each adversarial sample goes through
Different norms imply sample-specific t². Higher score norms signal greater deviation from the clean data, often caused by larger e. Intuitively, samples with elevated score norms demand higher t" (i.e., more aggressive noise injection) to remove adversarial patterns. This dependency creates a link: score norms can act as proxies for estimating the optimal sample-specific 1*. Doing so successfully leads to sufficient purification for adversarial examples while pre- serving fidelity for cleaner inputs.
the reverse
M(x) = C Ro D(x), with X0-
In this context, obtained after T steps of diffuusio, and X R(xT) rep- resents the corresponding recovered images through the reverse process. Specifically, these methods predetermine a constant noise level t for all samples, following shared noise schedule { | The outcome of the forward process defined in Eq. (1) can be expressed as:
D(xp) refers to the noisy image
4. Sample-specifie Score-aware Noise Injection
Motivated by Section 3, we propose Sample-specific Score- aware Noise Injection (SSNI), a generalized DBP frame- work that adaptively adjusts the noise level for each sample based on how much it deviates from the clean data distribu- tion, measured by the score norm. We begin by introducing
XT (1 8))x +
II (1 B.)e,

---

Sample-Specific Noise Injection For Diffusion-Baan Based Adversarial Purification
for x Xo: Vx X, where Xy represents the original data, and IO, denotes the Gaussian noise.
Algorithm 1 Diffusion-based Purification with SSNI.
Input: test samples x. score network so, reweighting function f()), noise level i for score evaluation, and a pre-determined noise level t*.
From sample-shared to sample-specific noise level. SSNI takes step further by transforming the sample-shared noise level (x) = t" into sample-specific noise level
1: Approximate the score by Sy: sa(x, t
(x)(x) = t(x), computed through
2: Obtain the sample-specifie noise level: t(x) =
(x1,0((x, {5)] ,t*)
f(x) = f[]|=o(x, t*),
(4)
3: Execute forward diffusion process (((x) + --q.. (5)
4: fort t(x)...... do
using a pre-trained score network 0 where (x td) ( is the score evaluated at an arbitrary reference noise level 15, and -(..) is a reweighting function (detailed in Section 4.3) that adjusts t" based on the score norm. The forward process of SSNI then takes the form:
5: Execute Reserve diffusion process x <- Eq. (2)
6: end for
7: return purified samples &
xx(x) II (1 B1) + V
((x) II 1=1 (1 3,),,
Realization of the score. A challenge to obtaining sample- specific noise levels lies in the score network's 's dependency on evoreevvaluation noise level as input. Estimating scores of different adversarial samples at a single fixed is intro- duces sensitivity: scores computed at different reference levels (e.g., 1 * +t5) yield inconsistent norms, biasing the selection of 1" Crucially, the "true" optimal i for each sample is unknown a priori, which leads to a circular de- pendency between reference noise level selection and score estimation. To alleviate this, we leverage expected pertur- bation score (EPS), which aggregates scores of perturbed samples across a spectrum of noise levels rather than rely- ing on a single i This integration reduces sensitivity to individual 1 choices (Zhang et al., 2023). EPS is defined as
(5)
for x = Xo, Vx E X. In this way, SSNI establishes link between the sample's 's deviation from the clean data and the intensity of noise injected during diffusion.
4.2. Unifying Sample-shared and Sample-specific DBP
We define generalized purification operator encompassing both sample-shane shared and sample-specific noise based DBP methods as (xx = R(xT(x)) where It denotes the reverse process, XT(x) is the noisy version of x after [xx) steps of diffusion, and T : X² > T is a function that determines the noise level for each input, with T = [0. T maxx] being the range of possible noise levels. Based (-), we derive the
EPS(x) = EE-U(O,Es) Vx (x),,(x),
(6)
following understandings (justifications are in Appendix B).
Sample-shared DBP is a special case of SSNI. The noise level of a sample-shares
where Pa (x) is the marginal probability density and t$ is the maximum noise level for EPS. EPS computes the ex- pectation of the scores of perturbed images across different noise levels t ~ U(0, 19), making it more robust to the changes in noise levels. Notably, this i is different from the sample-shared noise injection level t". We will omit i from the notation of EPS(x) for brevity hereafter. Fol- lowing Zhang et al. (2023), we set f 20. In practice, score Vx log,, (x) can be approximated by a score network. Specifically, we employ score network pre-trained using the score matching objective (Song & Ermon, 2019).
DBP, denoted by (x) & t", is a constant for Vx E X. while for SSNI, we have: (x) A t(x) = f[[l s (x, { (l) Clearly, any Tsn(x) can be expressed by (x), implying that any sample-shared noise level t" is equivalently represented by SSNI with a constant reweighting function.
SSNI has higher purification flexibility. To compare the purification capabilities of different DBP strategies, we intro- duce the purification range Ω. For an input x E X, we define !(x) {x) (x)
R(xx(x)), T X' + J). which characterizes all possible purified outputs that DBP strat- egy can generate for x when using different noise levels T(x). We find that AsH C Ωsi holds for any E X, and there exists at least one x E X' for which the inclusion is strict, i.e., ΩSH Ç ΩsI. These results show that SSNI ex- pands the space of possible purified outputs beyond what sample-shared DBP can achieve, thus enabling greater flexi- bility in the purification process.
Realization of the linear reweighting function. We first design a linear function to reweight t":
HPS [x] - Emin
finear EPS( {*)
xt + 6, (7)
Saax - Emin
where b is bias term and t* denotes the optimal sample- shared noise level selected by Nie et al. (2022). To implement this reweighting function, we extract 5,000 validation clean examples from the training data (de- noted as x,) and we use [[ES((x)]] as a reference to indicate the approximate EPS norm values of clean
4.3. Realization of SSNI
We now detail the empirical realizations of SSNI.
5

---

Sample-Specific Noise Injection For Diffusion-Basce Adversarial Purification
data, which can help us reweight 1* Then we de- fine Emin = min(]]EPS(())]] [I)))( and mmax = max (X))S( [[EPS(x))])) to normalize [[PS(()]] such that the coefficient of t" is within a range of [0, 1I, ensuring that the reweighted '* stays positive and avoids unbounded growth, thus preserving the semantic information.
et al., 2022) and GNS (Lee & Kim, 2023). The detailed configurations can be found in Appendix C. For the reverse process within diffusion models, we consider DDPM sam- pling method (Ho et al., 2020) in the DBP methods.
Evaluation settings for DBP baselines. Following Lee & Kim (2023), we use a fixed subset of 512 randomly sampled images for all evaluations due to high computational cost of applying adaptive white-box attacks to DBP methods. Lee & Kim (2023) provide robust evaluation framework for existing DBP methods and demonstrate that PGD+EOT (Madry et al., 2018; Athalye et al., 2018b) is the golden standard for DBP evaluations. Therefore, following Lee & Kim (2023), we mainly use adaptive white-box PGD+EOT attack with 200 PGD iterations for CIFAR-10 and 20 PGD iterations for ImageNet-IK, We use 20 EOT iterations for all experiments to mitigate the stochasticity introduced by the diffusion models. As PGD is a gradient-based attack, we compute the gradients of the entire process from a surrogate process. The details of the surrogate process is explained in Appendix D. We also evaluate DBP methods under adaptive BPDA+EOT attack (Athalye et al., 201 8a), which leverages an identity function to approximate the direct gradient rather than direct computing the gradient of the defense system.
Realization of the non-linear reweighting function. We then design a non-linear function based on the sigmoid function, which has two horizontal asymptotes:
t" +
([ t²) 1 + exp ( x) ||
/T) -
(8) where b is bias term and I* denotes the optimal sample- shared noise level selected by Nie et al. (2022) and is temperature coefficient that controls the sharpness of the function. We denote the mean value of |JES((x,) ]I as μ. This ensures that when the difference between [|EPS(x)] and j is large, the reweighted t' can approach to the maxi- mum 1" a more smooth way, and vice versa.
Adding a bias term to the reweighting function. One lim- itation of the above- mentioned reweighting functions is that the reweighted t" cannot exceed the original t", which may result in some adversarial noise not being removed for some adversarial examples. To address this issue, we introduce an extra bias term (i.c., b) to the reweighting function, which can increase the upper bound of the reweighted t* so that the maximum possible reweighted 1* can exceed original 1". Empirically, we find that this can further improve the robust accuracy without compromising the clean accuracy.
Evaluation settings for SSNI. Since SSNI introduces an extra reweighting process than DBP baselines, we implicitly design two adaptive white-box attacks by considering the entire defense mechanism of SSNI (i.c., adaptive white-box PGD+EOT attack and adaptive white-box BPDA+EOT at- tack). To make a fair comparison, we evaluate SSNI on adaptive white-box attacks with the same configurations mentioned above. The algorithmic descriptions for the adap- tive white-box PGD+EOT attack and adaptive white-box BPDA+FOT attack is provided in Appendix E and F. In addition, to evaluate the generalization and adaptability of SSNI to diverse adversarial attacks, we further include Au- toAttack (Croce & Hein, 2020). DiffAttack (Chen al., 2024b) and the Diff-PGI attack (Xue al., 2023) in Sec- tion 5.4. We set the iteration number to 5 for Diff-PGD.
5. Experiments
In this section, we use SSNI-L to denote our method with the linear reweighting function, and use SSNI-V to denote our method with the non-linear reweighting function.
5.1. Experimental Settings
Datasets and model architectures. We consider two datasets for our evaluations: CI (Krizhevsky al., 2009), and ImageNet-IK IK (Deng al., 2009). For classifiers, we use the pre-trained WideResNNe-22-10 and WedeResNet- 70-16 for CIF and the pre-trained ResNet-50 for ImageNet- 1 K. For diffusion models, we employ two off-the- shelf diffusion models trained on CIFAR-I0 and ImageNet- IK (Song et al., 2021b: Dhariwal & Nichol, 2021).
5.2. Defending Against Adaptive White-box PGD+EOT
We mainly present and analyze the evaluation results of SSNI-N in this section and the experimental results of SSNI- L can be found in Appendix G.
Result analysis on CIFAR-10. Table shows the standard and robust accuracy against PGD+EOT ( = 8/255) and Ez(e = 0.5) threat models on CI respec- tively. Notably, SSVI-N effectively improves the accuracy- robustness trade-off on PGD+EOT lzo 8/255) com- pared to DBP baselines. Specifically, SSNI-N improves stan- dard accuracy of DiffPure by 3.58% on WideResNee-28- and WideResNeet77-116 16 without compromising robust accu- racy. For GDMP, the standard accuracy grows by 1.63% on
Evaluation metrics. For all experiments, consider the standard accuracy (i.e., accuracy on clean examples) and robust accuracy (i.e., accuracy on adversarial examples) as the evaluation metrics.
Baseline settings. We use three well-known DBP methods as our baselines: DiffPure (Nie et al., 2022), GDMP (Wang
6

---

Sample-Specific Noise Injection For Puntusion-Basee Adversarial Purification
Table 1. -Stnndadd and robust accuracy of DBP methods against adaptive white-hox PGD+FOT (left: fo = 8/255), right: &a = 0.5)) on CIFAR-IO. 10. WideResNet-28- 10 and WedeRessNet700- 16 are used as classifiers. We compare the result of DBP methods with and without SSNI-N. We report mean and standard deviation over three runs. We show the most successful defense in bold. The performance improvements and degradation reported in green and red.
PGD+EOT 20 8/255)
PGD+EOT &g (c - 0.5)
DBP Method
Standard
Robust
DBP Method
Standard
Robust
Nie et al. (2022)
89.71
1..9810.64
Nie et al. (2022)
91,80
82.81 1.997
WRN-28-10
+ SSM--N
93.29=0.37 (43.58)
48.63+00.56 (+0.65)
WRN-28-10
SSNI-N
93.95=00.70 (+2.15)
82.75=1.01 (-0.06)
Wang et al. (2022)
92.45:10.64
36.72=1.05
Wang et al. (2022)
92.45=0..6
82.29:10.82
+ SSMI-N
0..0 (+1.63)
00.95:10.65 (+4.233
SSNI-N
94.08.10.33 (+1.63)
82.4910.75 (+0.20)
Lee & Kim (2023)
00.10:00.18
56.05 1111
Lee & Kim (2023)
00
83.66 0..46
SSNN-N
0.55 (+3.45)
06.45:+0.28
(+0.40)
-SSNI-N
93.55=+0.55 (+3.45)
84.05=0.33 (+0.39)
Nie et al. (2022)
1..8
52.1510.330
Nic al. (2022)
92.90-10.40
82.94
WRN-70-16
+ +SSNI-
94.47-0.51 (43.58)
52.47+00.66 (+0.32)
WRN-70-16
+ +SNNI-
15.12-00 (+2.22)
84.8 (+1.44)
Wang et (2022)
10
43.55+00588
Wang et al. (2022)
13.10+0.51
85.03++ 0.49
+ SSM-NN
95.57=0.24 (42.47)
46.03++1.33 (42.48)
+ SSNI-N
95.57=10.24 (42.47)
84.64=0.51 (-0.39)
Lee & Kim (2023)
89.39111.1
L6.97
Lee & Kim (2023)
89.3911.12
84.3710.37
+ -SSN-N
93.82-0.24 (+4.43)
57.03++0.28 (+0.06)
SSNI-N
93.82++0.24 (+4.43)
84.83+0.33 (+0.32)
accuracy-robustness trade-off.
Table 2. Standard and robust accuracy (%) against adaptive white- box PGD+EOT foo (c = 1/255) on ImageNet-IK.
Result analysis on ImageNet-1K. Table 2 presents the evaluation results against adaptive white-box PGD+EOT (x = 4/255) on ImageNet-1K. SSNI-N outperforms all baseline methods by notably improving both the standard and robust accuracy, which demonstrates the effectiveness of SSNI-N in defending against strong white-box adaptive attack and indicates the strong scalability of SSNI on large- scale datasets such as ImageNet-IK.
PGD+FOT f (e = 4/255)
DBP Method
Standard
Robust
Nie et al. (2022)
68.23.10.92
30.3410.72
SSNI-N
70.25:+0.56 (+2.02)
33.66=1.04 (+3.32)
RN-50
Wang et al. (2022)
0..22++0.12
0.39++0.03
SSNI-N
75.07:40.18 (+0.85)
5.21=:0.44 (+4.82)
Lee Kim (2023)
70.18++0.60
42.45+0092
SSNINN
72.6911..80 (+2.51)
43.48_0.255 (+1.03)
5.3. Defending Against Adaptive White-box BPDA+EOT
Table Standard and robust accuracy (%) against adaptive white- box BPDA+EOT (c = 8/255) attack on CIFAR-10.
We mainly present and analyze the evaluation results of SSNI-N in this section and the experimental results of SSNI- 1. can be found in Appendix G.
BPD s0 (c 8/255)
DBP Method
Standard
Robust
We further evaluate the performance of SSNI-N against adaptive white-box BPDA+EOT (x 8/255) which is an adaptive attack specifically designed for DBP methods (Tramèr et al., 2020; Hill et al., 2021), as demonstrated in Table 3. Specifically, incorporating SSNI-N with DiffPure can further improve the standard accuracy by 3.58% with- out compromising robust accuracy. Notably, incorporating SSNI-N with GDMP can improve the standard and robust accuracy simultaneously by a large margin. Despite some decreases in robust accuracy when incorporating SSNI-N with GNS (i.e., 1.10%), SSNI-N can improve standard ac- curacy significantly (i.e., 3.45%), and thus improving the roburaeyss trade-off by notable margin.
Nic al. (2022)
89.71=00.72
81.90+0.449
SSNI-N
93.29.10.37 (43.58)
82.10 1.15 (+0.20)
WRN-28-10
Wang et al. (2022)
92.45-1.64
79.88.10.89
SSNI-N
94.08 0.33 (+1.63)
80.99 1.09 (+1.11)
Lee Kim (2023)
90.10-10.18
88.40.10.88
SSNN-N
93.55+0..55 (+3.45)
87.30+0.42 (-1.10)
WideResNNet-28- and by 2.47% on WideResNeet70-11
respectively. Notably, SSNI-N improves the robust accuracy of GDMP by 4.23% on WidcResNet-28-U1 10 and by 2.48% on 7eteReesNet-70-166 For GNS. both the standard accu- racy and robust accuracy are improved by notable margin. We can observe a similar trend in PGD+EOT E2(E = 0.5). Despite some decreases in robust accuracy (e.g., 0.06% on DiffPure and 0.39% on GDMP), SSNI-N can improve stan- dard accuracy by a notable margin, and thus improving
5.4. Defending Against Additional Attacks
Table reports the standard and robust accuracy of various DBP methods on CIFAR-10 under three additional white-
7

---

Sample-Specific Noise Injection For Diffusion--aasee Adversarial Purification
Table Standard and robust accuracy (%) against Attoct (random version), DiffAttack and Diff-PGD attack with 6 (r = 8/255) on CIFAR-IO. We report mean and standard deviation over three runs. We show the most successful defense in bold.
for (r = 8/255)
DBP Method
Standard
AutoAttack
Diff'tttack
Diff-PGD
Nie et al. (2022)
89.71
66.73.10.21
47.16.10.48
54,95110.77
SSNI-N
93.29-0.37 (+3.58)
66.94+0.44 (+0.21)
48.15++0.22 (+0.99)
56.10=0.35 (+1.15)
WRN MRNN-28-10
Wang et al. (2022)
92.45=00.64
64.48::0.62
54.27::0.22
41.45:1:.60
SSNINN
94.08 0.33 (+1.63)
6..43 (+2.05)
55.81 (+1.54)
42,91 (+1.46)
Lee & Kim (2023)
90.10+0.18
69.92++0.30
56,04=0.58
59.02++0.28
SSNI-N
93.55_ (+3.45)
72.17.10.19 (+2.35)
56.80.10.41 (+0.76)
61.43100.8 (+2.41)
Table Ablation study on different sampling methods during the reverse diffusion process. We measure the standard and robust accuracy (%) against PGD+EOT fe = 8/255) on CIFAR-1O. We use DiffPure the baseline method and we set t" 100. WideResNet-233 28-100 is used as the classifier. We report mean and the standard deviations over three runs.
Aouuracy (kg
Standerd Acouraty
Tempersture Cooffilemt
8 88 6
Accansoy 184
Robust Accuraty
Sampling Method
Standard
Robust
sdeint solver
89.06=0.48 0..7
47.72::0.24
DDPM
47.98=:0.64
Temperature Couficcinn
DDIM
01.5441..2
37.50=10.80
Figure 4. Standard (top) and robust (bottom) accuracy (%) vs. We report mean and the standard deviations over three runs.
box attacks: AutoAttack, DiffAttack, and Diff-PGD, all under the ao = 8/255) threat model. For the DiffPure, SSNI-N improves standard accuracy by 3.58% and brings robustness gains under AutoAttack (i.e., 0.21% ), DiffAt- tack (i.e., 0.99%), and Diff-PGD (i.e., 1.15)). For GDMP, SSNI-N increases standard accuracy by 1.63%, and achieves significant robustness improvements under AutoAttack (i.e., 2.05%) and DiffAttack (i.e., 1.54%), with a slight gain un- der Diff-PGD (i.e., 1.46%). Regarding GNS, SSNI-N boosts the standard accuracy by 3.45%, and the robust accuracy improves under Auto/ttack (i.e., 2.35%), DiffAttack (i.e., 1.70%), and Diff-PGD (i.e.. 2.41%), respectively. Overall, SSNINN consistently enhances both standard and robust ac- curacy across all three attack types, demonstrating its strong generalization and adaptability to diverse adversarial threats.
T 20 for the non-linear reweighting function to optimize the accuracy-robustnes trade-off for DBP methods.
A blation study on sampling methods. DiffPure originally used an adjoint method to efficiently compute the gradients of the system, but Lee & Kim (2023) and Chen al. (2024a) suggest to replace adjoint solver with sdeint solver for the purpose of computing full gradients more accurately (Li et al., 2020; Kidger et al., 2021). Therefore, we investi- gate whether using different sampling methods affect the performance of DBP methods (here we use DiffPure as the baseline method). We further compare the results with DDIM sampling method (Song et al., 2021a), which is a faster sampling method than DDPM (Ho et al., 2020). From Table 5, DDPM achieves the best accuracy-robustne off among the three sampling methods, and thus we select DDPM as the sampling method for all baseline methods.
trade-
5.5. Ablation Study
Ablation study on 7 in SSNI-N. We investigate how the temperature coefficient т in Eq. (8) affects the perfor- mance of SSNI-N against adaptive white-box PGD+EOT ( = 8/555) attack on CIFAR-IO in Figure 4. The tem- perature coefficient 7 controls the sharpness of the curve of the non-linear reweighting function. A higher 7 leads more smooth transition between the low and high values of the reweighting function, resulting in less sensitivity to the changes of the input. From Figure 4, the standard accuracy remains stable across different rs, while the robust accuracy increases to the climax when 7 = 20. Therefore, we choose
A Ablation study on score norms. We investigate the effect of using single score norm (i.e., | Vx log Pt (x) []) for SSNI in Appendix H. We find that although single score norm can notably improve the standard accuracy, reduces robust accuracy. This might be attributed to the fact that single score norm is sensitive to the purification noise levels.
Ablation study on the bias term b. We investigate how the bias term b in reweighting functions affects the performance of SSNI in Appendix I. We find that the selection of bias term will not significantly impact the performance of our
8

---

Sample-Specific Noise Injection For Diffusion-Baan Based Adversarial Purification
Table Inference time of the DEP methods with and without SSNI for single image running on one A100 GPU on CIFAR-10 and ImageNet-IK. We use WideResNet-28 10 as the classifier for CIFAR- 10 and 50 for ImageNet-IK.
DBP Method
Noise Injection Method
Time (s)
DBP Method
Noise Injection Method
Time (s)
3,934
8,980
Nie et al. (2022)
SSNI-L SSNI-N
4.473
Nie et al. (2022)
SSNI-L SSNI-N
14.515
4.474
14.437
5.174
11.271
Wang et al. (2022)
SSNI-L
5.793
Wang et al. (2022)
SSNI-L SSNI-N
16.657
SSNI-N
5,829
16.747
14,902
35.091
Lee & Kim (2023)
SSNI-L
15.624
Lec & Kim (2023)
SSNI-L
40.526
SSNI-N
15,534
SSNI-N
40.633
framework under CIFAR-I0 and ImageNet-IK: IK. Note that when bias increases, there is general observation that the clean accuracy drops and the robust accuracy increases. This perfectly aligns with the understanding of optimal noise level selections in existing DBP methods, where a large noise level would lead to a drop in both clean and robust ac- curacy and small noise level cannot remove the adversarial perturbation effectively.
6. Limitation
Maximum level 1 for EPS. We use EPS to replace the single reference noise level with an integrated approach. Still, the maximum level is defining the upper bound of the expectation range is shared across samples, which may not be the optimal choice though. Refining sample-sensitive maximum levels could potentially improve the adaptability and robustness of EPS in different scenarios, and we leave it as future work.
Ablation study model architectures. We also investi- gate how the choice of model architecture affects the perfor- mance of our method in Appendix J. Specifically, we evalu- ate SSNINN on a Swin-Transformer (Liu et al., 2021) under the PGD+EOT (E = 8/255) on CIFAR-IO. We find that the improvements brought by SSNI-N are consistent with those observed on CNN-based models. In particular, SSNI- N enhances both standard and robust accuracy across all evaluated DBP baselines. Notably, the relative improvement on robust accuracy is more significant for seafformer-baseed classifiers. This observation suggests that SSNI-N can effec- tively complement the inherent robustness of transformer models and generalizes well across different architectures.
The design of reweighting functions. The proposed reweighting functions (i.e., the linear and non-linear reweighting functions) may not be the optimal ones for SSNI. Further efforts are needed to explore data- or learning-base based strategies to discover more effective func- tion forms. Meanwhile, designing an effective reweighting function is an open question, and we leave it as future work.
Extra computational cost. The integration of an extra reweighting process will inevitably bring some extra cost. Further efforts are needed to optimize the trade-off between accuracy gains and computational overhead. Luckily, we find that this reweighting process is lightweight, making SSNI computationally feasible compared to existing DBP methods (see Section 5.6).
5.6. Compute Resource
The inference time (in seconds) for incorporating SSNI modules into existing DBP methods on CIFAR-10 and ImageNet- 1K are reported in Tables 6. The inference time is measured the time it takes for single test image to complete the purification process. Specifically, SSNI is ap- proximately 0.5 seconds slower than baseline methods on CIFAR-10 and 5 seconds slower than baseline methods on ImageNet- IK. Thus, compared with DBP baseline meth- ods, this reweighting process lightweight, ensuring that SSNI is computationally feasible and can be applied in prac- tice with minimal overhead. We implemented our code Python version 3.8. CUDA version 12.2.0, and PyTorch version 2.0.1 with Slurm Workload Manager. We conduct each of the experiments on up to 4 x NVIDIA A100 GPUs (see https: /ttpt/tttttbb//e////////tt com/ tmlr (ttpo//ttllggrroo/SSS3//
7. Conclusion
In this paper, we find that an optimal t" indeed could be dif- ferent on a sample basis. Motivated by this finding, we pro- pose a new framework called Sample-specific Score-aware Noise Injection (SSNI). SSNI sample-wisely reweights t* for each sample based on its score norm, which generally injects less noise clean samples and sufficient noise to adversarial samples, leading notable improvement in the accuracy -oburacy-robusteess trade-off. We hope this simple yet effective framework could open up new perspective in DBP methods and lay the groundwork for future methods that account for sample-specific noise injections.
9

---

Sample-Specific Noise Injection For Diffusion-Bars Baaeod Adversarial Purification
Acknowledgements
adversarial attack. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2024b.
JCZ supported by the Melbourne Research Scholar- ship and would like to thank Shuhai Zhang and Huanran Chen for productive discussions. ZSY is supported by the Australian Research Council (ARC) with grant num- ber DE240101089. FL is supported by the ARC with grant number DE240101089, LP240100101, DP230101540 and the NSF&CSIRO Responsible AI program with grant num- ber 2303037. This research was supported by The Univer- sity of Melbourne's Research Computing Services and the Petascale Campus Initiative.
Croce, and Hein, M. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML,
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Dhariwal, and Nichol, A. Q. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
Dong, Y., Su, H., Wu, B., Li, Z., Liu, W., Zhang, T., and
Impact Statement
Zhu, J. Efficient pecision-based black-box adversarial attacks on face recognition. In CVPR, 2019.
This study on adversarial defense mechanisms raises im- portant ethical considerations that we have carefully ad- dressed. We have taken steps to ensure our adversarial defense method is fair. We use widely accepted public benchmark datasets to ensure comparability of our results. Our evaluation encompasses wide range of attack types and strengths to provide a comprehensive assessment of our defense mechanism. We have also carefully considered the broader impacts of our work. The proposed defense algorithm contributes to the development of more robust machine learning models, potentially improving the reliabil- ity of AI systems in various applications. We will actively engage with the research community promote responsible development and use of adversarial defenses.
Gao, R., Liu, F., Zhang, J., Han, B., Liu, T., Niu, G., and Sugiyama, M. Maximum mean discrepancy test is aware of adversarial attacks. In ICML, 2021.
Goodfellow, J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In ICLR, 2015.
Han, B., Yao, J., Liu, T., Li, B., Koyejo, S., and Liu, F. Trustworthy Machine Learning: From Data to Models. Now Foundations and Trends, 2025.
Hill, M., Mitchell, J. C., and Zhu, S. Stochastic security: Adversarial defense using long-run dynamics of energy- based models. In ICLR, 2021.
References
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurIPS, 2020.
Athalye, A., Carlini, N., and Wagner, D. A. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018a.
Jing, P., Tang, Q., Du, Y., Xue, L., Luo, X., Wang, T., Nie, S., and Wu, S. Too good to be safe: Tricking lane detection in autonomous driving with crafted perturbations. In USENIX Security Symposium, pp. 3237-3254, 2021.
Athalye, A., Engstrom. L., Ilyas, A., and Kwok, K. Synthe- sizing robust adversarial examples. In ICML, 2018b.
Kidger, P., Foster, J., Li, X., Oberhauser, H., and Lyons, T. GANs, In ICML,
Bai, M., Huang, W., Li, T., Wang, A., Gao, J., Caiafa, F. and Zhao, Q. Diffusion models demand contrastive guidance for adversarial purification to advance. In ICML, 2024.
Neural SDEs as 2021.
Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-10 (cana- dian institute for advanced research). 2009, URL http: / //wwwwcc cs toronto.......... edw/ /ecio-kkiii z/ ci htmml
Cao, Y., Wang, N., Xiao, C., Yang, D., Fang. J., Yang, R., Chen, Q. A., Liu, M., and Li, B. Invisible for both camera and lidar: Security of multi- fusion based perception in autonomous driving under physical-wond world attacks. In IEEE Symposium on Security and Privacy, pp. 176-194, 2021.
Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018
Lee, M. and Kim, D. Robust evaluation of diffusion-based adversarial purification. In ICCV, 2023.
Chen, H., Dong, Y., Wang, Z., Yang, X., Duan, C., Su, H., and Zhu, J. Robust classification via a single diffusion model. In ICML., 2024a.
Li, X., Wong, T. L., Chen, R. T. Q., and Duvenaud, D. Scal- able gradients for stochastic differential equations. In The 23rd International Conference on Artificial Intelligence and Statistics, pp. 3870-3882, 2020.
Chen, J.. Chen, H., Chen, K., Zhang, Y., Zou, Z., and Shi, Z. Diffusion models for imperceptible and transferable
10

---

Sample-Specific Noise Injection For Diffusion- Based Adversarial Purification
Liao, F., Liang, M., Dong, Y., Pang. T., Hu, X., and Zhu, J. Defense against adversarial attacks using high-level representation guided denoiser. In CVPR, 2018.
Wang. J., Lyu, Z., Lin, D., Dai, B., and Fu, H. Guided dif- fusion model for adversarial purification. arXiv preprint arXiv:2205. S2ti::2200.14966 14969, 2022.
Liu, Z.,, Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Wang. Q. Liu, F., Han, B., Liu, T., Gong, C.,, Niu, G., Zhou, M., and Sugiyama, M. Probabilistic margins for instance reweighting in adversarial training. In NeurIPS, 2021.
Ma, X., Li, B., Wang, Y., Erfani, M., Wijewickrema, S. N. R., Schoenebeck, G., Song, D., Houle, M. E., and
Wang. Y., Zou, D., Yi, J., Bailey, J., Ma, X., and Gu. Q. Improving adversarial robustness requires revisiting mis- classified examples. In ICLR, 2020.
Bailey, J. Characterizing adversarial subspaces using
local intrinsic dimensionality. In ICLR, 2018
Xiao, C., Chen, Z., Jin, K., Wang, J., Nie, W., Liu, M., Anandkumar, A., Li, B., and Song, D. Densepure: Under- standing diffusion models for adversarial robustness. In ICLR, 2023.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Naseer, M., Khan, S. H., Hayat, M., Khan, F. S., and Porikli, F. A self-supervised approach for adversarial robustness. In CVPR, 2020.
Xue, H., Araujo, A., Hu, B., and Chen, Y. Diffusion- adversarial sample generation for improved stealthiness and controllability. In NeurIPS, 2023.
Nie, W. Guo, B., Huang, Y., Xiao, C., Vahdat, A., and Anandkumar, A. Diffusion models for adversarial purifi- cation. In ICML,, 2022.
Yoon, J., Hwang, S. J., and Lee, J. Adversarial purification with score-based generative models. In ICML, 2021.
Zhang, H., Yu, Y., Jiao, J., Xing, E. P., Ghaoui, L. E., and Jordan, M. I. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Pang, T., Zhang, H., He, D., Dong, Y., Su, H., Chen, W., Zhu, J. and Liu, T. Two coupled rejection metrics can tell adversarial examples apart. In CVPR, 2022.
Zhang, J., Rubinstein, B. P., Zhang, J., and Liu, One stone, two birds: Enhancing adversarial defense through the lens of distributional discrepancy. URL https: a arxiv. org/ / abs 2503. 021 69.
Raghuram, J.. Chandrasekaran, V., Jha, S., and Banerjee, S.
A general framework for detecting anomalous inputs to DNN classifiers. In JCML, 2021.
Samangouei, P., Kabkab, M., and Chellappa, R. Defense- GAN: Protecting classifiers against adversarial attacks using generative models. In ICLR, 2018.
Zhang, J., Zhu, J., Niu, G., Han, B., Sugiyama, M., and Kankanhalli, M. S. Geometry-award instance-reweighted adversarial training. In ICLR, 2021.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In ICLR, 2021a.
Zhang, J., Liu, F., Zhou, D., Zhang, J., and Liu, T. Improv- ing accuracy-robustnes trade-off via pixel reweighted adversarial training. In ICML, 2024.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In NeurlPS, 2019.
Zhang, S., Liu, F,, Yang, J., Yang, Y., Li, C., Han, B., and Tan, M. Detecting adversarial data by probing multiple perturbations using expected perturbation score. ICML, 2023.
Song. Y., Kim, T., Nowozin, S., Ermon, S., and Kushman, N. PixelDefend: Leveraging generative models under- stand and defend against adversarial examples. In ICLR, 2018.
Zhuang, J., Dvornek, N. C., Li, X., Tatikonda, S., Pa- pademetris, X., and Dunean, J. S. Adaptive checkpoint adjoint method for gradient estimation in neural ODE, In ICML, 2020.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In ICLR, 2021b.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, J., and Fergus, Intriguing properties of neural networks. In ICLR, 2014.
Tramèr, F., Carlini, N., Brendel, W., and Madry, A. On adap- tive attacks to adversarial example defenses. In NeurIPS, 2020.

---

Sample-Specific Noise Injection For Diffusson-Baan Bassd Adversarial Purification
A. Detailed Related Work
Adversarial attack. Adversarial examples (AEs) have emerged as a critical security concern in the development of AI systems in recent years (Szegedy et al., 2014; Goodfellow et al., 2015). These examples are typically generated by introducing imperceptible perturbations to clean inputs, which can cause a classifier make incorrect predictions with high confidence. The methods used to generate such examples are known as adversarial attacks. One of the earliest attack methods, the fast gradient sign method (FGSM), which perturbs clean data in the direction of the gradient of the loss function (Goodfellow et al., 2015). Building on this idea, Madry et al. (2018) propose the projected gradient descent (PGD) attack, which applies iterative gradient-based baasee updates with random initialization. AutoAttack (AA) (Croce & Hein, 2020) integrate multiple attack strategies into a single ensemble, making it a widely adopted benchmark for evaluating adversarial robustness. To address defenses that apply randomized input transformations, Athalye et al. (201 8b) introduce the expectation over transformation (EOT) framework for computing more accurate gradients. Additionally, Athalye et al. (2018a) propose the backward pass differentiable approximation (BPDA), which approximates gradients using identity mappings to circumvent gradient obfuscation defenses. According to Lee & Kim (2023), the combination of PGD and EOT, i.e., PGI++OT, is currently considered the most effective attack strategy against DBP methods. More recently, adversarial attacks that specifically designed for DBP methods are proposed. For example. Xue et al. (2023) propose diffusion- based projected gradient descent enabling the generation of more stealthy AEs. Chen et al. (2024b) propose DiffAttack, which crafts perturbations in the latent space of diffusion models, rather than directly in pixel space, enabling the generation of human- inseeniiiiesiiiiee yet semantically meaningful AEs through content-preserving structures.
which leverages an off thel-heehfff diffusion model to guide perturbation optimization,
Adversarial defense. To counter the threat posed by adversarial attacks, numerous defense strategies have been developed, including adversarial detection (AD), adversarial training (AT), and adversarial purification (AP). (7) AD: AD is the most lightweight approach to defending against adversarial attacks is to detect and remove AFs from the input data. Earlier studies typically trained detectors tailored to specific classifiers or attack types, often overlooking the underlying data distribution, which limits their generalization to unseen attacks (Ma et al., 2018; Lee et al., 2018; Raghuram et al., 2021; Pang et al., 2022). Recently, statistical adversarial data detection has attracted growing attention for its ability to address this limitation. For instance, Gao et al. (2021) show that the maximum mean discrepancy (MMD) is sensitive to adversarial perturbations, and use distributional discrepancies between AEs and CEs to effectively identify and filter out AEs, even under previously unseen attacks. Building on this insight, Zhang et al. (2023) introduce novel statistic called the expected perturbation score (EPS), which quantifies the average score of a sample after applying multiple perturbations. They then propose an epseb variant of MMD to capture the distributional differences between clean and adversarial examples more effectively. (T)AT: AT: Vanilla AT (Madry et al., 2018) directly generates adversarial examples during training, encouraging the model to learn their underlying distribution. Beyond vanilla AT, various extensions have been proposed to improve its effectiveness. For instance, Zhang al. (2019) introduce a surrogate loss optimized based on theoretical robustness bounds. Similarly, Wang al. (2020) examine the role of misclassified examples in shaping model robustness and enhance adversarial risk via regularization techniques. From reweighting perspective, Zhang et al. (2021) propose geometry-aware instance-reweightee AT (GAIRAT), which adjusts sample weights based on their distance to the decision boundary. Building on this, Wang et al. (2021) utilize probabilistic margins to reweight AEs in a continuous and aath-independent manner. More recently, Zhang et al. (2024) suggest pixel-wise reweighting of AEs to explicitly direct attention toward critical image regions. (3) AP: AP typically employs generative models to transform AEs back into their clean counterparts before classification (Liao et al., 2018; Samangouei et al., 2018; Song et al., 2018: Naseer et al., 2020; Zhang et al.). Within this context, diffusion-based purification (DBP) methods have emerged as a promising framework, exploiting the inherent denoising nature of diffusion models to filter out adversarial noise (Nie et al., 2022; Wang et al., 2022; Xiao et al., 2023; Lee & Kim, 2023). DBP works with pre-trained diffuusion model by first corrupting an adversarial input through the forward process and then iteratively reconstructing it via the reverse process. This projects the input back onto the clean data manifold while stripping away adversarial artifacts. Wang et al. (2022) introduce input guidance during the reverse diffusion process to ensure the purified outputs stay close to the inputs. Lee & Kim (2023) propose fine-tuned gradual noise scheduling for Bai et al. (2024) improve the reverse diffusion process by incorporating contrastive objectives.
purifications.
12

---

Sample-Specific Noise Injection For Diffusion-Baan Based Adversarial Purification
B. Justification of Section 4.2
Sampled-shared DBP is special case of SSNL.
Proof. Let PSH be sample-shared purification operator with constant noise level t". We can express Pss as a sample-
specific purification operator (sst by defining the reweighting function as
f(z,t)) =t E R, E m
Then, for any E N, we have
p5)(x) = R(xT(x))
R(xx.)
(x).
SSNI has Higher Purification Capacity.
Statement I [Comparison of Purification Range]: For any E X, we have (SH(x) S (x)(
Proof: Let y E (sH(x). Then 3t" E Tmax] such that R(xx-). Define f(z,t" = t" for all 2 and t", then (xx) = (xx)==(xxx"")l.. = L" Therefore, y = R(x+-) = R(xx()) E (x)(x). This completes the proof of Qsn(x) C (x).
Statement 2 [Strict Inclusion]: There exists x E X, we have (x)( S Qsi(x).
Proof. Consider non-constant score function so(x, 1) and non-trivial reweighting function f. We choose such that t(x) * t* for any fixed t". Then R(x(()) @st(x) but R(x((x)) * 4sn- This completes the proof of (s)(x) ç (x).
C. Defense Methods Configurations
For chosen DBP methods, we utilize surrogate process to obtain gradients of the defense system during white-box adaptive attack, but we directly compute the full gradients during defense evaluation. Furthermore, we consistently apply DDPM sampling method to the selected DBP methods, which means replace the numeric SDE solver (sdeint) with DDPM sampling method in DiffPure (Nie et al., 2022) and GDMP (Wang et al., 2022). The reason is that the SDE solver does not support sample-specific timestep input. For DDPM sampling, we can easily manipulate sample-specific timestep input by using matrix operation.
C.1. DiffPure
Existing DBP methods generally follow the algorithm of DiffPure (Nie et al., 2022). DiffPure conducts evaluation on AutoAttack (Croce & Hein, 2020) and BPDA+EOT adaptive attack (Athalye et al.. 2018a) to measure model robustness. DiffPure chooses optimal t" = 100 and t" = 75 on CIFAR-10 against threat models t (c = 8/255) and f2(c = 0.5), respectively, also tests on high-resolution dataset like ImageNet- with 150 against threat models 6 (e = 4/255). Calculating exact full gradients of the defense system of DiffPure impossible since one attack iteration requires 100 function calls (with t" 100 and step size of 1). DiffPure originally uses numerical SDE solver for calculating gradients. However, the adjoint method is insufficient measure the model robustness since it relies the performance of the underlying SDE solver (Zhuang et al., 2020; Lee & Kim, 2023; Chen et al., 2024a). Therefore, we apply surrogate process to efficiently compute gradients of direct back-propagatie
in our evaluation. To overcome memory constraint issue, we align the step size settings of denoising process in adversarial attack with the evaluation settings in (Lee & Kim, 2023) and keep the timestep t" consistent with DiftPure. For ImageNet- 1K evaluation, we can only afford a maximum of 10 function calls for one attack iteration.
13

---

Sample-Specific Noise Injection For Diffusion-Basse Adversarial Purification
C.2. GDMP
GDMP basically follows the purification algorithm proposed in DiffPure (Nie et al., 2022: Wang et al., 2022), but their method further introduces novel guidance and use multiple purification steps sequentially. GDMP proposes to use gradients of distance between initial input and target being processed to preserve semantic information, shown in Eq. (9).
X N(po 2Eo xx D(x', x En);
Given DDPM (1,(x,)) (x,,)) gradient scale of guidance s. X; is the data being purified, and x is the adversarial example at t. Also, GDMP empirically finds that multiple purification steps can improve the robustness. In the original evaluation of GDMP, the defense against the PGD) attack consists of four purification steps, with each step including 36 forward steps and 36 denoising steps. For BPDA+EOT adaptive attack, GDMP uses two purification steps, each consisting of 50 forward steps and 50 denoising steps.
Lee & Kim (2023) evaluated GDMP with three types of guidance and concluded that No-Guidance provides the best robustness performance when using the surrogate process to compute the full gradient through direct backpropagation. In our evaluation, we incorporate the surrogate process with No-Guidance evaluate GDMP. Since it is impossible calculate the gradients of the full defense system, we use surrogate process consisting of same number of purification steps but with larger step size in the attack (with 6 denoising steps and 10 denoising steps for PGD+EOT and BPDA+EOT attack, respectively). Notably, GDMP only uses one purification run with 45 forward steps to evaluate on ImageNet- KK, which we keep consistent with this setting.
C.3. GNS
Lee & Kim (2023) emphasizes the importance of selecting optimal hyperparameters in DBP methods for achieving better robust performance. Hence, Lee & Kim (2023) proposed Gradual Sohse-Scheduling (GNS) for multi-step purification, which based on the idea of choosing the best hyperparameters for multiple purification steps. is basically the same architecture as GDMP (no guidance), but with different purification steps, forward steps and denoising steps. Specifically, GNS sets different forward and reverse diffusion steps and gradually increases the noise level at each subsequent purification step. We just keep the same hyperparameter settings and also use ensemble of 10 runs to evaluate the method.
D. Surrogate Process of Gradient Computation
The surrogate process is an efficient approach for computing approximate gradients through backpropagation, as proposed by (Lee & Kim, 2023). White-box adaptive attacks, such as PGD+EOT, involve an iterative optimization process that requires computing the exact full gradients of the entire system, result high memory usage and increased computational time. DBP methods often include diffusion model calls during reverse generative process. Hence, it is hard to compute the exact full gradient of DBP systems efficiently. The surrogate process takes advantage of the fact that, given fixed total amount of noise, we can denoise it using fewer denoising steps (Song et al., 2021a), but the gradients obtained from the surrogate process slightly differ from the exact gradients. Instead of using the full denoising steps, we approximate the original denoising process with fewer function calls, which allows us to compute gradients by directly propaproggg through the forward and reverse processes.
add-on purifier, which the model requires extensive function
There are other gradient computation methods such as adjoint method in DiffPure (Li et al., 2020; Nie et al., 2022). leverages underlying numerical SDE solver to solve the reverse-time SDE. The adjoint method can theoretically compute exact full gradient, but in practice, it relies on the performance of the numerical solver, which is insufficient to measure the model robustness (Zhuang et al., 2020; Lee & Kim, 2023; Chen et al., 2024a). Lee & Kim (2023) conducted a comprehensive evaluation of both gradient computation methods and concluded that utilizing the surrogate process for gradient computation poses a greater threat to model robustness. Hence, we use gradients obtained from a surrogate process in all our experiments.
14

---

Sample-Specific Noise Injection For Diffusion- Based Adversarial Purification
E. Adaptive White-box PGD+EOT Attack for SSNI
Madry et al. (2018) proposed Projected Gradient Descent (PGD), which a strong iterative adversarial attack. Combining PGD with Expectation Over Transformation (EOT) (Athalye et al., 2018b) has become powerful adaptive white-bo box attack against AP methods. In our defense system, we add a reweighting module existing DBP methods, so this information must be included during white-box attack. We decide to place the reweighting process outside the EOT loops because EOT provides more samples with different random transformations. It effectively reduces the randomness of gradient computation but this process does not greatly affect the EPS value of the data. In addition. this approach further reduces the computational cost.
Algorithm 2 Adaptive white-box PGD+EOT attack for SSNI.
Require: clean data-label pairs (x, y): purifier fp; classifier fe: noise level T; score network sx T); objective function L: perturbation budget e; step size a; PGD iterations K; EOT iterations N.
1: Initialize xdd <- x
2: for i == O....
do
3: Computing cample-specifie noise levels: t(xx1)) + -[[[so(xxtt) (T) :T)
4: Average the gradients over EOT: gk Ei
Vx & (fc(F((x()) ,t(xxx)))
5: Update adversarial examples: x <- (x) (xc + sign(
6: end for
7: return xddv = X
adv
F. Adaptive White-box BPDA+EOT Attack for SSNI
Athalye et al. (2018a) proposed Backward Pass Differentiable Approximation (BPDA), which is popular adaptive attack to DBP defense methods. When a defense system contains non-differentiable components where gradients cannot be directly obtained, the BPDA method substitutes these non-differentiable operations with differentiable approximation during backpropagation in order to compute the gradients. typically utilizes an identity function fx) = x as the differentiable approximation function. In specific, computing the exact full gradients via direct backpropagation through a diffusion model is time-consuming and memory-intensivee so BPDA attack assumes the purification process is an identity mapping. This implies that we ignore the impact of the purification on the gradients and directly treat the gradient with respect to the purified data Xg as the gradient with respect to the input Xg-
Algorithm 3 Adaptive white-box BPDA+EOT attack.
Require: clean data-label pairs (x, u); purifier fr: classifier foi approximation function Sape: noise level T; score network (x, I';; objective function L; perturbation budget e; step size at PGD iterations K; EOT iterations N.
1: Initialize x < x
2: for < to K do
3: Computing sample-specific 1(x)x*) <- S( (x (T),,T)
4: Average the gradient over BOT samples: Sh +- I Ei V Vxrar L ((SolSap ((o([))((
N
5: Update adversarial examples: xddv [x (x a. sign(gk)))
6: end for
7: return ady adv
G. Performance Evaluation of SSNI-L
We also incorporate SSNI-L reweighting framework with existing DBP methods for accuracy-robustnes evaluation. In Table 7, we report the results against PGD+EOT (c = 8/255) and f2(c = 0.5) threat models on CIFAR-10, respectively. We can see that SSN/- can still support DBP methods to better trade-off between standard accuracy and robust accuracy. Also, we report the results against BPDA+EOT l (c 8 8/255) threat model on CIFAR-10 Table 3. Overall, SSNI-L slightly decreases the robustness of DBP methods against PGD+EOT attack and maintain the robustness of DBP methods against BPDA+EOT attack, but there is notable improvement in standard accuracy.
15

---

Sample-Specific Noise Injection For Diffusion-Basse Adversarial Purification
Table Standard and robust accuracy of DBP methods against adaptive white-hox PGD+FOT (left: (e = 8/255), right: &a = 0.5)) CIFAR-IO. WideResNet-28 10 and WedeResNet-700 16 used as classifiers. We compare the result of DBP methods with and without SSNI-L. We report mean and standard deviation over three runs. We show the most successful defense in bold.
PGD+EOT &oo (c = 8/255)
PGD+EOT & (e 0.5)
DBP Method
Standard
Robust
DBP Method
Standard
Robust
Nie al. (2022)
89.71+-0.72
4..98+0066
Nie al. (2022)
91.80-+0.84
82.81
WRN-28 -0 10
+ SSN--L
92.97-+0.42
06.35++0772
WRN-28 10
+ SSNI-I
93.82-+0.37
81.12++0.80
Wang et al. (2022)
92.45-+0.664
36.72:t1.05
Wang al. (2022)
92.45-+0.64
02.2
+ SSNI-L
93.62:t:0.49
36.59
+ SSNI-L
93.62-+:0.9
80.66
Lee & Kim (2023)
90.1
56.05-+1.11
Lee Kim (2023)
90.10+0..18
83.66+00.46
+ SSNI-L
93.49.10.33
53.71
+ -SNNI-L
93.49110.33
85.29.10.24
Nic (2022)
1..89+1.113
52.15++2.30
Nic et al. (2022)
92.90+t0.40
82, 1.19
WRN-70-16
SSNN-L
93.82:1..49
09.9
WRN-70-16
+ +SSNI-L
94.99-10.24
84.44
Wang et al. (2022)
93.10:H:0.5
43.55++0.58
Wang al. (2022)
93.10-+0.51
85.03-1:0.49
SSNI-L
93.88:1-0.4
03.0
+ SSNI-I
93.88-+0.49
82.88
Lee & Kim (2023)
1.12
56.97+00.33
L.ce & Kim (2023)
89.39-+
84.51+00.77
SSN--L
92.64+-0.40
52.86+0.46
+ -SSN--
92.64++0.40
84.90++0.09
Table 8. Standard and robust accuracy (%) against adaptive white-hox BPDA+EOT (e = 8/255) attack on CIFAR- 10. We compare the result of DBP methods with and without SSNI-L. We report mean and standard deviation over three runs. We show the most successful defense in bold.
BPDA+EOT Eso (e = 8/255)
DBP Method
Standard
Robust
Nie al. (2022)
89.71
81.90- 0.49
+ SSNI-L
92.97-0.44
80.08-0.00
WRN- 28-
Wang al. (2022)
92.45-00.64
+ SSNI-L
93.62-10.49
79.95
Lee Kim (2023)
90.10110.18
88.40-10.88
+ SSNI-L
93.49
88.41 0.09
H. Ablation Study on Score Norm
We further provide experiments with the single score norm instead of the EPS norm. Yoon et al. (2021) shows score norm V lolog (x) is a valid measurement for adversarial detection. Incorporating single score norm with our SSNI-N framework, it still achieves notable improvement on standard accuracy, but the robustness drops. The single score norm is highly sensitive to noise levels, which makes it insufficient to completely distinguish between natural and adversarial examples.
Table Standard and robust accuracy (%) against adaptive white-box PGD+EOT (e = 8/255) attack on CIFAR-IO. We use single score norms (i.e., || log (xx) [). We report mean and standard deviation over three runs. We show the most successful defense in bold.
PGD+EOT L. (e 8/255)
DBP Method
Standard
Robust
Nic et al. (2022)
89.71 0.72
0.64
+ SSNI-L
91.31=0.24
46.92=0.52
WRN-28-10
+ +SNNI-
92.84=0.18
47.20
Wang et al. (2022)
92.45=0.64
3.072=1.05
SSNI-L
03.1
35.7
+ SSNI-N
93.42 0.60
1..24=1.45
Lee Kim (2023)
90.10+0.18
16.05-1.111
+ SSNI-L
93.40=0.49
54.52-0.46
+ SSNI-N
93.55-0042
15.47=1.115
16

---

Sample-Specific Noise Injection For Diffusion-Baan Bassed Adversarial Purification
I. Ablation Study on Bias Term
Table Ablation study on term b. We report standard and robust accuracy DBP methods against adaptive white-box PGD+EOT CIFAR-10. 10. WideResNet-28- 10 and WideResNNt--NN 70-16 are used classifiers. We report mean and standard deviation over three runs. We show the most successful defense in bold.
PGD+EOT t. (e 8/255)
Bias
10
15
20
25
30
WRN-28-11 10
Standard
9..34.1..43
93.95.1.117
93.17.11055
1..38.11.29
92.38.11.29
92.10
92.97
SSNA-
Robust 05.27=0075
57.23+1.62
57.03=0.54
57.6441..99
57.64+0.89
58.24+1.22
59.18=1.65
WRN-70-16
Standard
94.34:10.55
13.8
94.7311.34
92.7910.89
92.79:0.89
92.58=0.67
92.77=0.94
Robust
56.45+1.22
57.03
08.10
58.79:1.48
59.57=1.19
58.59=0.52
58.59=0.52
WRN-28-10
Standard
92.91+0.555
92.97-H0.42
93.01 0.78
92.64+0.28
0..5
91.69+1.02
91.45 - 0.88
SSNI-
Robust
46.29:10.46
46.35.10.72
46.28
46.75.10.3
47.11
47.05
47.23 0.76
WRN-70-16
Standard
93.79-10.53
93.82.10.49
93.7710.64
92.85.1..2
92.32.1..00
90.08
91.89-0.52
Robust
49.34:1..85
49.94.10.33
48.8311092
50.72.10.77
50.80.10.4
51.05-1.11 0.66 0.66
Table 11. Ablation study on the bias term b. We report standard and robust accuracy of DBP methods against adaptive white-box PGD+EOT on ImageNet-IK. ResNet-50 used the classifier. report mean and standard deviation over three runs. show the most successful defense in bold.
PGD+FOT Lo (e 4/255)
Bias
25
50
75
100
125
150
RN-50
SSAS
Standard 1..68
7..73.11.49 40.28.10.28
71.96.10.3 43.88
68.80.10.74
68.41
66.45 40.8710.92
67.63 1.08
Robust
39.33110.34
41.45.0.38
43.02.10.41
40.05-00.67
J. Ablation Study on Model Architecture
Table 12. Standard and robust accuracy (%) against adaptive white-box PGD+EOT foo (c = 8/255) attack on CIFAR-JO. The target classifier is Swin-Tre Tran-orranfommer. We report mean and standard deviation three runs. We show the most successful defense in bold.
PGD+EOT ta (e 8/255)
DBP Method
Standard
Robust
Nic et al. (2022)
88.48-0.49
37.81+0..3
+ SSNI-N
88.67-0.55
19.65-0.47
Swin-T
Wang et al. (2022)
88.09-0.72
27.34=0.18 18
+ SSNI-N
89.45-0.34
28.71=0.27
Lee & Kim (2023)
88.67=0.20
52.54=0.14
+ SSNI-N
91.21-10.11
54.10-0.222
17

---

Sample-Specific Noise Injection For Diffusion-Basee Adversarial Purification
K. Additional Justification of the Relationship Between Score Norms and Noise Levels
Extending the empirical findings from Section 3, we further justify how score norms of different samples correlate with the noise level s" in diffusion models. With Proposition K.5, we find that the score norm varies with t, and the variation exceeds threshold e > 0 when the time difference I22 &y] is sufficiently large. Thus, for two noisy samples æ1 and 22 with different score norms, the monotonic increase of { [0 with t implies that different score norms correspond to different noise levels, i.e., 1i + l2. It implies that, for samples with different score norms, it is likely that they have undergone different noise injection steps.
Definition K.1 (Marginal Probability Density). Let X' be the image sample space, p(xo) be the natural data distribution over X. and the diffusion process kernel defined in Eq. (I). The marginal probability density pt. X'' - R4 at time s t S T can be expressed as:
(x(x) = J 9(xx)x))((xx))xx
(10)
where (() describes how a natural sample X, evolves under the forward process to x at time t.
Before proceeding with the proof of Lemma K.4 and Proposition K.5, we start by presenting two lemmas to facilitate the proof.
Lemma K.2. Let pe(x) denote the marginal probability density of at time t. For any x E X and 0 S <ts T, the score
function Vx log (x( ) time be expressed as:
V looogg px() = Ep(xo(x) [x [xlggg(x(xo))
where p(x)(x) is the posterior given by Bayes' Rule:
((x(xx)p(xx))
p(x x) = J ((x((x))) ((())))))x)))))) (dxx
Proof.
Vx log / 1ece//(xx20 )dbo
Vx log
(by Def. K.1)
Vx / g(x (x (ooooxooppxx) )
(chain rule)
x(x
dx
J V q (x
dbco
(Leibniz integral rule)
I q(x xx x cdxo
(ag(x(xo) (xx ( xxxo
(1I)
o(x(xx)
(manipulate g(x(xo))
J (((x| Xo (X jdxo
J(Vx log q(x (((Xxx) g (x ((xx))dxx))
(chain rule)
dxd
(p(xo/x) A ((x(xo))(Xx))
( log ((x (p)))))))(((())))) x)dxo
J g(( Xo (xx)))(xxx))) )dxo
This ends up with the expectation over the posterior pxx) as E (x(x)))) [ log g xo)).
Lemma K.2 establishes a relationship between the marginal density and the forward process, where we can express the behavior of Vx log.o(x (x) ) in terms of the well-defined g(x(xo).
Lemma K.3. Consider the forward process q(x, (x0) is defined with noise schedule (Boheto,rT) where Br (0, 1) for all
S StST T. Then, for any x E X and 0 & it T, the score function Vx log Px(x) can be decomposed as:
Vx loggo = g(())(++++ + Ep(xolx) [g(t)
where g(t = 1/(1- and ā2 = II B.).
'However, we note that this analysis only focuses the property of diffusion models, disregarding the potential interactions between Gaussian and adversarial noise (which is often difficult analyze directly). In this sense, does not formally motivate the proposal of SSNI and not considered core contribution this study. We include this part of the content for completeness.
18

---

Sample-Specific Noise Injection For Diffusion-Baan Basood Adversarial Purification
Denote the score function sx(x) Vx log (x) for convenience. By applying the results from Lemma K.2, write:
Proof.
sx(x) = xxxx)(>) [[(x|xo)]
Provided that g(x[xo)= = N (x; vauxo: (1 (x)) II, we compute the gradient of the log-probability of the forward process
as:
Vx (x = xx (x; vaixo, (1 (
(x- - vārxo) Txx vāxxo) + +c
Vx
2(1 (t))
(x vaxXo);
where C' is a constant term independent of x. Substitute the result back into se (x(x), we have:
sx() = pxxxl(x) x)) [x(x(x))]
Ep(xo/x)
xx varxul
1
v V Ex(x)1x)(xo)
att
1 az
=[()x [x(x)xx (9(()
The Lemma is completed by substituting g(1) I- into the last equation.
Lemma K.3 expresses the score function by the noisy data x with noise level and the clean data We next examine how the score norm evolves to further understand its behavior as time increases.
Lemma K.4. Suppose there exists a constant K >> O such that for all t 2 0 and all X₁ E X, the expected norm of the clean
data given X₁ satisfies: Exx-p(xo | S K ||x, ||. Then, there exist constants 0 < C < and To > 0 such that for all t To:
V (x()] [x
> C.
Proof. Again, denote (x) = Vx (x). From Lemma K.3, we have:
sxx) -t())x Ep(x)/x) [g(t)varx))
Applying the triangle inequality to it, we have:
[x)]( (t) | |xpp())
gt) ||*|| gt) / (x [|[x,]]]+
where the second inequality comes from applying Jensen's inequality: S(E[X) < [E[(X)) for convex function f and random variable X, which leads to:
[x(x))x (o(t) || s E Ex(x) [|lo())
g(t) (xp [|]xo|]]
As we have assumed the existence of K > 0 that makes Exo ||| xoll| |I| S K ||x/|] holds for all x E X, we have:
(x)]] 2 g(t) ||||| - g(t) Va |||||
=(t) |x| (1 K V
We then look into the asymptotic behavior of gt). We have 11: (1 9.). Since 0 B, for all s,
and the sequence is decreasing (as 8g is increasing), it is easy check that
lim Tr = 0 2 00
19

---

Sample-Specific Noise Injection For Diffusion-Baan Basedd Adversarial Purification
and
lim s(t) lim 1 =1. 1. 1000
1-000
at
This can then be formalized as, for any e >> 0, there exists a T. such that for all t >>
Te:
Im(() - 1| < e, and vā <e.
Then, for L Te, have
[x)[
2 g(t |x|| (1 - K Vvā))
> (1 <) | (1 - Kc),
(1 c Ke+ + Kc²) ||X
(1 - (K 1)e + Kε²) ||x||
To establish the desired inequality || (x)ll 2 C ||| for constants > 0, we next investigate whether this quatratic
inequality Ke² + (K + 1)e + (1 - C) > can solved. Denote the discriminant (K + 1)2 4K(1 - CC), we have
D (K + 1)2 -K((1 C)
K² -2K + +1+
= (K -1)2 + +KC
> 0 (since (K -1)2 > 0 and K > 0).
Then, let 61 < be two real roots of K2² + (K + 1)c (1 C) 0, as the parabola opens upwards, i.e., K > 0, note
that O, we further need to ensure the smaller root 6
K+1-v/
> 0, such that
2K
K+1> V(K + 1)2 4K C)
(K + 1) (K- (K-1 4KC
4K > 4KC
1> C.
Putting them logether, we have e E (1) U (ez, oo) that makes [s,(x)]] 2 C [||x|| hold, given a constant 0 < < in relation to K and e. Setting To = T, completes the proof.
Proposition K.5. Consider the diffusion model satisfying all conditions as specified in Lemma K.4. Assume that there exist constants K O, such that Be S K for all Additionally, suppose |x|| S M for any x X. for some M > Then, for any there exists constant A 2t/(CK) such that for 1, L22 2 O, we have:
V log Pt, (x)ll - V log Ptg (x)ll 6, with 141
12
A
Proof. Denote sx(x) Vx (xlogp((x) Recall that we can express the score function as
(x(x) E xx)
x
[g(x(x))]
1
āt E Ep(x)/x) [x Vā(xo]
Vār
x
Epixo (x) [xo]
āt
1 1- Cr
Let g(t) = [x)] We compute
ag(t) ăt
using the chain rule and the product rule:
0g(t)
0s((x)
= 2 3t (x),
Øt
Øt
2
(x),
0
1
x
+
Vas
Epixo xx) (xx)
Øt
ai
1 ãi
20

---

Sample-Specific Noise Injection For Diffusion-Basct Adversarial Purification
Recall that di I (1 B.), we next derive the derivative of öt- We consider the continuous approximation of 4, where the product is approximated exponential of the integral
8t = exp
- -2)di)
As Be is typically assumed to small (which an implicit common practice (Ho et al., 2020)), we further simplify log(1 3.) with its first-order Taylor approximation, i.e., log(1 3,) s -3,, which thus leads to the approximated a exp (-Jo B.di). This way we compute the derivative of 0z as
0
- / 8,di)
a. Ot
exp
Ot
Øt (- L Brdi)
0
(-
exp
Badi
% Bydi
exp
(-3c)
-B,a
We can then compute the derivatives of the coefficients:
0 1
0
(1 a.)
Ot
1 - σ
=
(1 āz)² Ot
B,ag
(1 ā,)²
Va.
V 81
(1 at)
vār 7
a (1 (1)
0r
01
1 04
(1 -x,))
1/2 aa. 7 σ
(1 - āz) + Vāa
dā, 0t
(1=
( 1/2 (-3rat)
(1-ar) + var (-3:āt)
(1)
1 - k1)=
Blaa 3/2
(1 - -xa)
1/2 1 (1 az)) + at
=B
(1 ā.) 2
1 0t
Bi
2(1 - ,,)²
Using the computed derivatives:
(x) 0t
1 1 ōr
x :Bvār at
Bp(xo (x) [xo]
(1 az)²
2(1 āz)²
Substituting these back, we get:
0g
V
3r0
1 + Cee
2
+
E Fp(x)|x) (x0);
x Br
2(1 ā.)² (o(xolb)
0t
1 04
1 Ct
(1
az)
23,00
1 1 āz
[x(x))x
Ep Ep(2o (x) (xo)
(1-āz) )²
2āc
21

---

Sample-Specific Noise Injection For Diffusion-Basee Adversarial Purification
Next, under Cauchy-Schwarz inequality: (a, b) S ||a] |[b]]
we have
dg 23 (1 - ot)²
1+ ate 2var
|sxx())l
x
Bpixo xx [x0]
Then, by the triangle inequality for vectors a and b:
||a bll = a (-b)ll S ||a|| + 11-bb| |||||| + |bll
and the assumption |||||
M, we know that [p(x))x))
[xo||
M almost surely. Thus,
0g
- (1 x,)
23,00
||s((x)]
+ āe x
E Bx(xo)s))xx) (xu]
01
2
23,
+ ae
E3 1x(xolx) ))
||s((x)]|
||x|||
(1 a,)²
2
23,00
M + 1 Oz M
$ (1 0,)²
2/00
1+ As < ā < 1, is easy to check the maximum value of 1+& Let C1 = supr20 (1-0,)a and C2 = 22p
20.
2
2va,
i.e. C2 achieved when ãe gets close to This also aligns with the condition where sup (1-ā,)2 reached. Thus,
(I+ 1+ 2/a.
M S +2) C3M for some constants C3 2, since C2 + 1 when ã₂ 1. As result, have
constant C' > 0 leading to the upper bound as
Og 0t
< (x)]l, with C C1 (CaM).
Mean Value Theorem states that: for any I, 12, there exists a between ( and l2 such that: (9((2) g(h)) = 7t (5)11/2
00
t1 - (xx))l - t1l-
Applying this to g(t), for any t1, 12, there exists É between t and f2, such that:
0g (k) t1l
(g(t2) g(ti)l
S C3 || (x)] It2 til
s CK [x{(x))] Itz t1l
(by assumption 8, s K)
Then, express
[x)) || || (x)
|118, (x)]l ||,t, (x)ll
(11s,, (x) ||| + || (x)ll) CK [|s{{())]] 1t2 41l
|| (x)] + ||s,, (xx]l
From Lemma K.4, we have: there exists C' 0 and To > such that for all 2 To: [xs) Cx|[x| Applying this t, tg, and & with 8 > 0, we find that there exists C' such that:
||se, (x)] > C*
||s,, (x)] > C"
[|s] > C
||x]
Substituting these back, we have:
CK2 ||x]
122
[x)]] ||,9, (x)11
2C' |xX||
CK₂
122 -11.
2
For any > 0, let A CK. Then for It2 - 41 A, we have:
|| (x)] ||,,, (x) > c. This completes the proof.
22

---

Sample-Specific Noise Injection For Diffusion- Based Adversarial Purification
Remark K.6 (Relationship between score norms and (*). According Proposition K.5, for distinct noise levels 4 and 12, would obtain different score norms for the sample E A. Hence, consider that different t" would change the score norm of the same sample In addition, also imply that the score norms of the same sample differ if and only if the noise levels t" input to the score network Sg are different. This is because the score network only takes two input arguments s0(x, 1*), which sample x and noise level t", For natural example x, we treat as a sample with no adversarial perturbation, such that e = 0. From Figure 2, cleaner samples typically exhibit smaller score norms, which means the data points are close to the high data density region (Song Ermon, 2019). Then, we should not diffuse natural data further with a large t" retain the original semantic information. Samples subjected to stronger perturbations exhibit larger score norms, which means the adversarial examples are far from the high data density area. We then inject certain amounts of Gaussian noise to the sample to cover its adversarial perturbation during forward diffusion. We identify the score norm metric assess the sample's proximity to the original data distribution p(x).
L. Visualizations for Purification Results
(a) Adversarial Examples
(b) DiffPure
(c) DiffPure + SSNI-L
(d) DiffPure + SSNI-N
(a) Adversarial Examples
(b) DifiPure
(c) DifiPure + SSNI-L
(d) DiffPure + SSNI-N
(a) Natural Examples
(b) DiffPure
(c) DiffPure + SSNI-L
(d) DiffPure SSNI-N
23


Reviews:
[{"review_id": "N4neB4O0yh", "reviewer": "Reviewer_4Rp6", "summary": "This paper presents a method to enhance existing diffusion-based adversarial purification techniques. The authors build on the intuitive idea that adversarial samples with higher noise levels require larger diffusion timesteps for effective purification. To explore this, they analyze the output of the diffusion model when processing adversarial samples and observe that samples with greater noise tend to exhibit larger output norms. Leveraging this insight, the authors propose an adaptive approach to selecting the optimal diffusion timestep based on the noise level of each adversarial sample. They categorize noise into sample-shared and sample-specific components. Empirical results on CIFAR-10 demonstrate that the proposed method is compatible with existing diffusion-based adversarial purification techniques and further enhances their performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Evidence of Claim**\n\n**R1**:  Thanks for pointing it out. We now show that samples with larger deviation (caused by larger perturbation, leading to higher score norm) need stronger denoising (higher $t^*$).\n\nWith a DBP method [1], we assess the robust accuracy of WRN-28-10 against AEs from PGD+EOT $\\ell_{\\infty}$ attacks with varying perturbation budgets on CIFAR-10, finding a shared $t^*$ leads to poor robust accuracy for the other three groups (**[results](https://shorturl.at/q71D6)**).\n\n[1] Robust evaluation of diffusion-based adversarial purification. ICCV 2023.\n\n**Q2: Visualization of ImageNet**\n\n**R2**: Thanks for the advice! We will include visualizations of ImageNet. Due to character limits, we kindly refer you to **Response to Reviewer xxTh Q5** where we also discuss this in detail.\n\n**Q3: Clarify adversarial perturbation from diffusion noise**\n\n**R3**: This confusion is caused by inconsistent $\\epsilon$ between motivation and experiment sections.\n\nWe now clarify the difference between *adversarial perturbation budget* $\\epsilon$ and the adaptive diffusion noise $t(x)$ applied by SSNI.\n\nOur experiments indeed use a fixed $\\epsilon$, defining *attack* strength used to generate adversarial examples (AEs) for controlled evaluation.\nHowever, SSNI's *sample-specific noise* refers to the amount of Gaussian noise injected during DBP process, not $\\epsilon$ of adversarial attack.\n\nOur motivation is that different samples benefit from sample-specific diffusion noise levels for purification.\nFig.1 implies this need. Fig.2 then shows that score norms (our chosen metric) correlate with perturbation intensities (using different attack budgets $\\epsilon$ is a clear way to show this correlation). It establishes score norms as a proxy for deviation from the clean manifold.\n\nWe still need sample-specific diffusion noise level $t(x)$ even when evaluating against the same budget $\\epsilon$.\n- The **same attack** (shared $\\epsilon$) applied to **different** clean examples (CEs) will produce AEs perturbed to varying degrees relative to the clean manifold, depending on each CE's own property and its interactions with the attack.\n- So, even under a fixed $\\epsilon$, AEs have **different score norms**.\n- This leads to **different diffusion noise level** $t(x)$ tailored to sample-specific deviations (Eq.4).\n\n*So, the same $\\epsilon$ in evaluation does not omit sample-specific diffusion noise levels $t(x)$*.\n\n**Q4: NN-based reweighting function**\n\n**R4**: The presented SSNI-L/N are instantiations of the reweighting mechanism within our SSNI framework.\nOur primary goal was to introduce the core SSNI concept and show its effectiveness using simple and computationally lightweight functions working entirely at inference time without additional training.\n\nUsing a NN as the reweighting function is indeed a valid extension.\nBut this would require a training phase for optimization, shifting from the current training-free paradigm and introducing training overhead.\nWe agree that investigating such learnable functions is a promising direction.\n\n**Q5: Pre-defined noise level**\n\n**R5**: Yes, we used a pre-defined base noise level for score estimation via EPS (L252). \nBypassing these pre-defined levels is challenging within the training-free paradigm.\nSimilar to the reweighting function, one could train NN to predict a proper level per sample.\nYet this would also introduce a training phase and associated overhead.\nDeveloping a purely analytical training-free method to determine these levels without any reference is non-trivial.\n\nWe appreciate the reviewer's question and leave this interesting avenue for future work.\n\n**Q6: Score network choice**\n\n**R6**: Thank you for this valuable advice. We used a standard SDE-based score network [1] pre-trained on CIFAR-10 and a guided diffusion model [2] pretrained on ImageNet, **following** previous score estimation studies (Zhang et al. 2023).\n\nMoreover, SSNI should *not* be sensitive to a specific score network, as we use EPS, which averages scores over multiple noise levels for more robust score estimation (L264-274, Eq.6), supported by Appendix G. Thus, we believe SSNI can generalize to other score networks (e.g. LDM, EDM), as long as they provide *reliable score estimations*.\n\nWe are empirically evaluating with other score networks and will report the results once we obtain them.\n\n**Q7: Related work**\n\n**R7**: Thanks for providing the reference [3].\nBoth [3] and SSNI leverage diffusion models and analyze choice of noise level in different contexts.\n[3] denoises imitation learning demonstrations with an optimal $t^*$ before IL, whereas SSNI targets adversarial defense with sample-specific $t^*(x)$ selection based on score norms.\nWe will include the discussion in the revision.\n\n[1] Score-Based Generative Modeling through Stochastic Differential Equations ICLR 2021\n\n[2] Diffusion Models Beat GANs on Image Synthesis NeurIPS 2021\n\n[3] Imitation Learning from Purified Demonstrations ICML 2024", "claims_and_evidence": "Yes, most of the claims are well supported. In the motivation section, the authors highlight three aspects of the relationship between perturbation $\\epsilon$ and the noise level $t^\\ast$. The first two claims are substantiated with examples. However, I do not find any evidence supporting the intuitive assumption that samples with larger perturbations should require a higher timestep $t^\\ast$.", "methods_and_evaluation_criteria": "The authors propose to leverage pre-trained score networks to estimate the gradient of the log-data density is a valid technique from score-based generative modeling, and nusing score norms as a proxy for deviation from the clean data distribution is a reasonable heuristic. \n\nThe authors use standard datasets like CIFAR-10 and ImageNet-1K and evaluating both clean accuracy and robustness against adversarial attacks are standard and appropriate evaluation criteria for adversarial defense methods.", "theoretical_claims": "This criteria does not apply to this paper, as there are no theoretical claims presented in the main text. The mention of score norms and references to [1] hint at a theoretical underpinning in score-based generative modeling, but this is not formalized as theorems or propositions.\n\n[1] Song et al. Score-based generative modeling through stochastic differential equations. ICLR 2021", "experimental_designs_or_analyses": "The experimental design is soundness. The authors basically follow the experimental design of existing works, which is comprehensive. \n\n1. One issue is that the authors claim they will conduct experiments on ImageNet-1K, however, I do not see any visualised results in the paper. Since the visualised samples of CIFAR10 is quite blur, I can not see any useful information from the compared results in Figure 5-7. Can you further explain how to tell the purification effects from the aspect of visualization?\n\n2. One thing I am confused about is that in the methodology part, the authors claim that their proposed method are used to address sample-specific noise presented in the adversarial purification. However, it seems that the authors also use the same level of perturbation for the whole dataset. For example, in Table 1, the perturbation is set to be 8/255 and 0.5 respectively. Where does the sample-specific noise come from?", "relation_to_broader_scientific_literature": "The contributions are related to the literature on Adversarial Purification (AP) and Score-based Adversarial Detection. For the former area, the paper direclty builds upon the field, specifically DBP. The authors adequately cite a number of representative works in AP and example DBP methods. The paper positions itself as an improvement with a more general framework to these existing DBP techniques. As for the later area, the paper mentioned Yoon et al. (2021)'s use of score norms for adversarial example detection, connecting the proposed method to this line of work and extending the use of score norms beyond detection level adaptation in purification.", "essential_references_not_discussed": "I believe that some of works [1] have been investigating the level of noise problem in diffusion purification. In this work, the authors investigate how the level of noise could be connected to the timestep in diffusion purification. The authors are encouraged to discuss this paper and illustrate the difference between this paper and their work.\n\n[1] Wang et al., Imitation Learning from Purified Demonstrations. ICML 2024.", "other_strengths_and_weaknesses": "Strengths:\n\nI appreciate that the proposed method is simple enough to understand, and appears to be a general framework, allowing it to be compatible with diverse DBP methods.\n\nWeaknesses:\n\n1. The methodology section only covers two reweighting functions - it appears that there are also other possibilities which could be considered as well.\n\n2. Still, a pre-defined noise level needs to be specified before reweighting.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have two questions directly relevant to the weaknesses:\n\n1. Have the authors considered other forms of reweighting functions, for example, can we have optimizable ones using neural networks?\n\n2. Do you have alternatives to bypass pre-defined noise levels?\n\nIn addition, How sensitive is the performance of SSNI to the choice of score network? If the method is highly dependent on a specific, perfectly trained score network, its practical applicability and robustness could be limited.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MdxiHFlzzb", "reviewer": "Reviewer_ynLt", "summary": "This paper examines the problem of choosing a sample-dependent number of forward/reverse diffusion steps to use in diffusion-based purification (DBP) adversarial defense. Prior works typically use a fixed number (e.g. t=100) forward/reverse steps to secure an input before sending it to the classifier. The method is motivated by the intuition that different samples have difference number of forward/reverse steps needed for security (more secure samples need less diffusion, less secure samples need more diffusion). The key problem then becomes the method for estimating the ideal number of diffusion steps from a sample. The work proposes to use the score of the input sample in the diffusion network as a way to predict the optimal number of steps. Samples with lower score norms are believed to be closer to natural images and require less purification, while samples with higher score norms are believed to be further from natural images and require more purification. Simple linear and non-linear functions are used to reweight a baseline timestep into a sample-adjusted timestep using the score norm. Experimental results show the proposed method can reliably increase the natural accuracy and often increase the robust accuracy of existing diffusion defenses compared to the baselines using a fixed number of steps.\n\n\\## After rebuttal: My view of this work remains similar. The proposed method appears to be a reliable and cost-efficient way to provide a modest increase in security for diffusion defense.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Discussion on performance gain and robust accuracy**\n\n**R1**: We appreciate the reviewer for raising this concern. We'd like to first clarify our contribution and provide a clearer context.\n\n### Contribution\nThe central goal of this paper (and SSNI) is to achieve a more favorable accuracy-robustness balance, which is a crucial consideration in adversarial training/defense studies [1]. Our contributions are\n- identifying a critical limitation in existing DBPs: they rely on a fixed diffusion noise level $t^*$ injected during the forward pass, forcing a single, yet often suboptimal, denoising effort across all samples. For any clean or adversarial sample, an **inappropriate** noise level might \n    - fail to remove adversarial perturbations sufficiently (hurting robustness) or \n    - excessively corrupt sample semantics (hurting accuracy).\n- (*core contribution*) proposing a **new framework for sample-specific noise injection (SSNI)** to directly address this limitation. Based on the estimated deviation from the clean data distribution, SSNI adaptively sets the diffusion noise level $t(x)$ for each sample, assigning lower noise for cleaner samples to preserve accuracy, and potentially higher noise for more perturbed samples to enhance robustness. \n- confirming that SSNI has a **larger purification capacity** and is  **more flexible** than sample-shared-noise DBPs, supported by empirical results and theoretical justification (Appendix A)\n- (central implication) SSNI **improves the overall accuracy-robustness trade-off** in DBP, particularly achieved in a **training-free manner**.\n\n### Significance and risk\nOur results show SSNI's success in achieving this better balance. We observe that **in most cases**, both standard and robust accuracy are *improved simultaneously* (e.g., GDMP under BPDA+EOT, Table 3: +1.63% Std Acc, +1.11% Rob Acc).\nEven in **rare cases** where robust accuracy sees a minor decrease (e.g., -0.06% for DiffPure PGD-L2, Table 1), it is often coupled with a substantial gain in standard accuracy (+2.15% in that case). \nWe argue this often represents a preferable trade-off point, recovering clean performance sacrificed by using fixed $t^*$.\n\nThe **consistent improvements** on ImageNet (Table 2) further confirm that SSNI can effectively balance these targets on large-scale tasks. We thus argue that SSNI **generally enhances the robustness aspect of the accuracy-robustness balance** compared to baselines.\n\nIn summary, we believe that SSNI represents a principled framework to address the fundamental accuracy-robustness trade-off in DBP via adaptive denoising budgets, providing flexibility and improved overall balance that fixed $t^*$ methods cannot achieve.\n\n[1] Improving Accuracy-robustness Trade-off via Pixel Reweighted Adversarial Training, ICML 2024.\n\n**Q2: Results with additional DBP backbone (DiffAttack)**\n\n**R2**: Thank you for the suggestion. We have supplemented experiments and reported these results as requested. \n\n**Please see the [DiffAttack results](https://shorturl.at/YI8tU)**.\nThe table provides results of utilizing DiffAttack [2] with $\\ell_{\\infty}$ ($\\epsilon=8/255$) on target classifier WRN-28-10 on CIFAR-10 (**due to limited time frame within the rebuttal phase, we can only include partial results therein; we will report full results in the revision**).\n\nIt is easy to observe **consistent performance gains** over all DBP baselines with our SSNI integrated.\n\n\n[2] DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification, NeurIPS 2023\n\n---\nThank you very much again for your time! We hope our response addresses your concern. If you have any further questions, please feel free to ask, we’d be happy to provide more clarification.", "claims_and_evidence": "The claims and evidence in this paper are generally solid. The criterion of using the score norm is a reasonable way to judge how easy/difficult a sample will be to classify and builds upon similar observations in prior work. It is intuitively reasonable that adapting the number of diffusion steps based on this uncertainty measure could improve defense performance compared to the scenario of using a fixed number of steps. The increases in natural/robust accuracy from the proposed method compared to baselines is consistent and reasonable.", "methods_and_evaluation_criteria": "The methodology is suitable for the problem at hand. Use of score norm is a reasonable way to measure the classification difficulty of an input. The timestep selection function is relatively lightweight and doesn't add unreasonable computational burdens. Evaluation is performed according to the same attack protocol as the Lee & Kim, which is a representative state-of-the-art attack against diffusion models.", "theoretical_claims": "This paper does not make any theoretical claims.", "experimental_designs_or_analyses": "The experimental design and analyses are straightforward and appropriate. An ablation for timestep selection hyperparameters, method for calculating attack gradient (full checkpointing vs. surrogate vs. DDIM), use of single versus multiple score norms is presented.", "relation_to_broader_scientific_literature": "The key contribution of this work is a relatively fast method for selecting the number of diffusion forward/reverse steps for DBP. The method is quite general and can be incorporated with different DBP variations. The experimental results show that consistent gains can be achieved. While these gains are not especially large, on the other hand I feel fairly convinced that it is still worth using this method rather than using a fixed timestep.", "essential_references_not_discussed": "DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification\nhttps://arxiv.org/abs/2311.16124\n\nThis attack method produces similar results to Lee & Kim against DBP. It might be worth including the results from this attack with and without the proposed method in Table 4.", "other_strengths_and_weaknesses": "The main strengths are that the method is straightforward and produces a consistent benefit, more for natural accuracy but usually for robust accuracy as well. The main weakness is that the gains are not especially significant and that there is still a risk of reducing robust accuracy.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CPcTYAu1CP", "reviewer": "Reviewer_xxTh", "summary": "This paper proposes a new perspective on diffusion-based purification (DBP) methods. The authors first show the score norms $||\\Delta_{x}log\\ p_{t}(x)||$ of input samples $x$ are highly related to the noise level of Gaussian noise that should be injected when performing diffusion-based adversarial purification, then they develop a Sample-specific Score-aware Noise Injection(SSNI) method based on a pre-trained score network to control the level of injuected Gaussian noise, which improves the performance in terms of clean accuracy and robust accuracy when integrated with existing DBP methods.\n\n## update after rebuttal. We thank the detailed rebuttal and explanations, which addressed our concerns. Our views towards the paper remain unchanged. The paper is acceptable because of its method's novelty, generalization, and effectiveness in improving the performance in defending against adversarial attacks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Evaluation with transformer-based classifiers, Diff-PGD, and unrestricted attacks**\n\n**R1**: We have supplemented the transformer-based model and Diff-PGD experiments. **Due to character limits, please see [here](https://shorturl.at/2b8Qs) for results**\n\nRegarding unrestricted attacks: In this paper, we primarily focus on defending **human-imperceptible adversarial perturbations**, while unrestricted attack is a **different problem setup**, which breaks the assumption of human-imperceptible perturbations in most adversarial defense literature. However, we acknowledge this is an interesting open question to be explored.\n\n[1] Diffusion models for adversarial purification. ICML, 2022.\n\n**Q2: Discussion on the innovation**\n\n**R2**: We thank the reviewer for acknowledging SSNI's generality.\n\nWe'd like to respectively argue that customizing sample-specific noise level $t^*$ **is new** because $t^*$ is **fundamentally critical** to DBP performances, and the fixed $t^*$ used in existing studies is a **core limitation** causing suboptimal accuracy-robustness trade-offs.\n\nSSNI's main contribution is introducing a **conceptual advance** of sampled-adaptive $t(x)$, leveraging score norms that represent a sample's deviation from clean data, tailoring purification strengths based on *instance-specific* score norms.\nWith extensive experiments, we confirm this 'simple' modification leads to substantial gains of **accuracy-robustness balance**.\n\nWe believe the **simplicity and effectiveness** of SSNI, while being **easily integrated** into diverse DBP methods (as the reviewer mentioned), is a *key strength*.\n\nTo our knowledge, SSNI is the **first framework** to systematically implement and validate score-norm-driven adaptive $t^*$ for DBP, which offers a practical, impactful and thus novel purification principle.\n\n**Q3: Discussion on the performance-time trade-off**\n\n**R3**: Thank you for raising this discussion. We acknowledge that increased inference time **is a limitation** of SSNI, however, it is primarily attributed to the inherent limitation of diffusion models, which our method relies on in estimating score norms. We look forward to more efficient strategies to accelerate this step.\n\nOn the other hand, we argue the **benefits often justify the cost**. On SSNI delivers absolute gains of +2.0-2.5% standard and +1.0-4.8% robust accuracy on ImageNet with 1000 classes (PGD $\\ell_\\infty$), gains **often considered substantial** in robustness contexts.\nMoreover, SSNI improves the overall **accuracy-robustness balance**, which is a qualitative benefit beyond only numerical results.\n\nUltimately, the cost-benefit analysis is indeed **context-dependent**. For *security-critical offline* tasks, the improved accuracy-robustness profile provided by SSNI could well justify the additional inference time.\nAs a modular enhancement, SSNI provides practitioners **an option** when the computational budget allows for improved DBP effectiveness and accuracy-robustness trade-off.\nWe also note that ongoing advances in *efficient score estimator* will help to reduce this overhead.\n\n**Q4: Black-box settings**\n\n**R4**: We focused on adaptive white-box attacks (PGD+EOT, BPDA+EOT), aligning with the **standard evaluation protocol** for DBP methods (Lee&Kim ICCV 2023), which is **common practice** in recent DBP literature as it directly stress-tests the defense pipeline against the **worst-case threat model**. \nRobustness against these *strong* attacks typically implies robustness against *weaker* black-box threats. \nThus, we prioritized demonstrating effectiveness against the established challenging white-box benchmarks.\n\nStill, we provide results for a gray-box attack setting with PGD+EOT $\\ell_{\\infty}$ ($\\epsilon=8/255$) and $\\ell_{2}$ ($\\epsilon=1$) on CIFAR-10 dataset (partial results only, we'll report full results in the revision), where the attacker can access the target classifier, not the entire defense system.\n**Due to character limits, please see [here](https://shorturl.at/s8bnJ) for results**\n\n**Q5: Visualization of Imagenet-1K**\n**R5**: Thank you for the feedback. Regarding ImageNet-1K, we will include high-quality ImageNet-1K purification visualizations in the revision.\n\nFor the CIFAR-10 images (Fig.5-7), visual differences are indeed subtle due to low resolution. Our main goal here was **not necessarily** to showcase visually superior cleanness, but rather to show that SSNI maintains **semantic integrity**. Even when SSNI adaptively uses different (sometimes higher) noise levels $t^*$ per sample, the mechanism driving *improved robust accuracy* does **not corrupt the essential semantic information**, unlike the failure cases illustrated in Fig.1 where improper $t^*$ leads to misclassification.\nThis thus confirms that more flexible purification does not** come at the cost of distorting image content and compromising clean/robust accuracy.\n\n---\nThank you for your time again! Hope our responses address your concerns.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem.", "theoretical_claims": "This paper proposes the proof of the relationship between score norms and noise level. We check the correctness of the proofs of  Lemma J.1, J.2, J.3, J.4, J.5, and J.6 in Appendix J carefully and find no obvious issues.", "experimental_designs_or_analyses": "The paper does experiments on two datasets: CIFAR-10 and Imagenet-1K, and evaluates the robustness of the convolutional network-based classifiers against white-box PGD+EOT and BPDA+EOT attack method under the SSNI+BPD framework for each dataset, validates the superiority of the SSNI method in improving the accuracy of classifiers against both normal inputs and adversarial inputs in some circumstances. However, the experiments are not persuasive enough, other model architectures such as transformer-based classifiers should be included.\\\nThe paper only shows the qualitative results of purified images from the CIFAR-10 dataset in the main paper and appendix, visualization of purified images from Imagenet-1K should be included to show the performance on complex and high-resolution datasets.\\\nAblation sturdy on hyperparameters validates the framework's hyperparameters selection.", "relation_to_broader_scientific_literature": "Prior diffusion-based purification(DBP) methods inject a constant level of Gaussian noise into the input sample and leverage the sampling process of diffusion models to remove probably existing adversarial perturbations from the input samples, which achieves good performance in defending adversarial attacks. \\\nHowever, such DBP methods may also decrease the clean accuracy of classifiers, this paper attributes it to the fixed Gaussian noise level, since the Gaussian noise level for different kinds of input samples should be different (eg: a high noise level may destroy the semantics of clean samples, thus reducing the clean accuracy; a low noise level may not remove all adversarial perturbations, thus reducing the robust accuracy. ). \\\nThen this paper solves this problem by proposing the SSNI method, which applies a score network to control the forward diffusion step t* to inject an adaptive level of Gaussian noise into the input samples, improving both clean accuracy and robust accuracy of classifiers.\\\nThe SSNI method is general enough to be integrated with existing DBP methods and improve their performance further.", "essential_references_not_discussed": "To the best of our knowledge, no.", "other_strengths_and_weaknesses": "Strengths\n\n1. The writing of the paper is fluent, the structure is clear, and there are no obvious grammar errors.\n\nWeaknesses\n\n1. The authors only consider modifying the noise level of the entire DBP framework and developing their SSNI method. Though this improves the generalization of their method, which allows it to be integrated with existing DBP methods, the innovation of the entire paper appears insufficient.\n\n2. The SSNI method employs a pre-trained score network to estimate the score norm of input samples. As empirically validated in the study, the framework achieves a 2-3% accuracy improvement on ImageNet-1K while incurring a 5-second time increase per image. This presents a critical trade-off consideration: Given that DBP methods are inherently time-consuming, the justification for further escalating computational complexity to pursue marginal performance gains warrants rigorous cost-benefit analysis and domain-specific evaluation.", "other_comments_or_suggestions": "In the caption of Figure 1, there's some typo in the citation: Nie et al.(2022).", "questions_for_authors": "1. We are curious about the performance of SSNI against other attack methods, especially diffusion-based methods like Diff-PGD [1]. \n\n2. Do the authors evaluate their method against adversarial attacks under the black-box setting? \n\n3. We are also curious about how can the SSNI method adapt the noise level for unrestricted adversarial attacks which modify the semantics of images on a large scale. (eg. DiffAttack [2],  ACA [3])\n\n[1] Xue H, Araujo A, Hu B, et al. Diffusion-based adversarial sample generation for improved stealthiness and controllability[J]. Advances in Neural Information Processing Systems, 2023, 36: 2894-2921.\n\n[2] Chen J, Chen H, Chen K, et al. Diffusion models for imperceptible and transferable adversarial attack[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.\n\n[3] Chen Z, Li B, Wu S, et al. Content-based unrestricted adversarial attack[J]. Advances in Neural Information Processing Systems, 2023, 36: 51719-51733.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qAp4NIWNPs", "reviewer": "Reviewer_oarL", "summary": "This paper focus on the diffusion model-based purification methods. They proposed SSNI to find the optimal $t^{\\ast}$ based on Diffpure paradigm. The weakness of the Diffpure is that the robust of the purification depends on the setting of the optimal $t^{\\ast}$ i.e., how much Gaussian noise should be injected into the adversarial samples. Too much noises will break the semantic during the reverse process. Too less noises will be useless to filter the adversarial noise. SSNI proposed an adaptive way to find the optimal $t^{ast}$ for each sample since different samples will be injected different level of the adversarial noise. In this way, SSNI achieves the more robust purification method. The experiments reported on CIFAR-10 and ImageNet show that SSNI could increase the robust accuracy while keeping the standard accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Contribution of this study**\n\n**R1**: Due to character limits, please refer to **Response to Reviewer ynLT - Q1** where we clarify our contribution.\n\n### Discussion on [1]\nWe acknowledge that score-based metrics, including EPS [1], are established tools for distinguishing between clean and adversarial samples. Though score norms can estimate sample deviation [3], we follow the motivation in [1] and **adopt** their proposed EPS as it offers more robust estimates. \n\nFig.2 serves only as **motivation** of SSNI, **building on understandings from** [1]. It empirically visualizes correlations between score norms and perturbation levels, justifying why score norm is suitable for guiding SSNI's adaptive mechanism. We do **not** claim it as a contribution and will cite [3] and [1] near Fig.2 to credit foundational concepts, ensuring this is clear in our revision. \n\nSo, while we leverage EPS from [1], our core contribution is the **SSNI framework**, enabling sample-specific noise injection to address the accuracy-robustness trade-off problem in DBP.\n\n**Q2: Generality of SSNI**\n\n**R2**: We consider SSNI a **general framework** because its principle - adaptively adjusting the denoising level $t(x)$ based on a sample's deviation using a reweighting function $f$ - is applicable to **various DBP methods** and across **different datasets**.\n\nEq.7-8 are presented as **proof-of-concept instantiations** of $f$ within this framework. $b$ is a hyperparameter *within these instantiations*, allowing reweighted $t^*$ to exceed the baseline $t^*$ to improve robustness.\n\n*Tuning hyperparameter per dataset* is a *common practice* in machine learning and *does not invalidate* the framework's generality. Many 'general' methods (including **baseline DBPs that require tuning $t^*$**) do have hyperparameters that benefit from dataset-specific tuning for better performance.\n\nIn this sense, we argue that SSNI is a *general framework of sample-specific noise injection* for DBP, rather than a specific method implementation. **Reviewer xxTh also acknowledged SSNI's generality**.\n\n**Q3: More Evaluations on AutoAttack and [2]**\n\n**R3**: Thanks for bringing [2] to our discussion, which addresses a similar challenge (accuracy-robustness trade-off) in DBP to SSNI but takes a different approach.\n\n[2] learns adversarial guidance during the reverse diffusion step, requiring **training** an auxiliary network to modify the diffusion direction. Instead, SSNI is **training-free**, adaptively adjusting diffusion noise levels per sample before standard diffusion at inference time, based on pre-computed score norms.\n**We will include the discussion of [2] in our revision**.\n\nThese two methods are thus complementary. In principle, SSNI can be integrated with this method. However, as this paper has **yet open-sourced the code**, we are unable to get the results now but willing to include them later.\n**See [link](https://shorturl.at/1j18r) for required results.**\n\n**Q4: Usefulness of SSNI-L**\n\n**R4**: We included SSNI-L as a first step when exploring *training-free* reweighting functions for SSNI. Its purpose was to establish a simple baseline with linear mapping before investigating more complex non-linear ones.\n\nHowever, SSNI-N consistently provided a superior accuracy-robustness trade-off, possibly due to simple linear mapping can't fully model the complex reweighting operation, justifying our focus on SSNI-N in the main text.\n\nIn the revision, we will clearly state the role of SSNI-L as a simpler baseline and summarize its relative performance.\n\n**Q5: Triangle inequality**\n\n**R5**: Thank you for carefully reading our proof. We'd like to clarify it is correctly used.\n$$\n\\begin{aligned}\n||x + y|| & \\leq ||x|| + ||y|| \\\\\\\\\n||(x - y) + y|| & \\leq ||x - y|| + ||y|| \\\\\\\\\n||x|| - ||y|| & \\leq ||x - y||\n\\end{aligned}\n$$\n\n**Q6: Theoretical Claim**\n\n**R6**: We'd like to kindly recall our core contribution is the SSNI framework for sample-specific noise injection in DBP.\n\nTheoretically proving the optimality of $t^*$ is challenging, as there is no clear definition of 'true optimal' noise level. To be clear, we do **not claim to derive the 'optimal' noise level**; our focus is to emphasize **noise level should be sample-specific**. We will **ensure this is clarified in the revision to avoid any potential overclaiming.**\n\nThe effectiveness of SSNI is empirically validated and 'theoretically' justified (Recall our response to Q1). \n\nWe believe identifying the true optimal $t^*$ is an interesting open question. We will ensure our revision accurately reflects the scope of our claims and contributions.\n\n[1] Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score. ICML 2023\n\n[2] Robust Diffusion Models For Adversarial Purification. ArXiv 2024\n\n[3] Adversarial purification with Score-based generative models. ICML 2021\n\n---\nThanks for your review! If our response addresses your concern, we hope you might consider increasing score.", "claims_and_evidence": "This paper have two contributions: 1) finding the norm of the score function could be used as a metric to measure the level of an unknown adversarial noise hidden in given sample. 2) proposing the adaptive way to find $t^{\\ast}$.\n\nThe contribution 1 seems questionable. EPS [1] has already proposed that the norm of the score function could be the indicator to difference the natural sample and adversarial samples. Thus, the author claims: \" Motivated by this, we further investigate how different perturbation budgets ϵ affect score norms under adversarial attacks (Figure 2)\" is not the contribution for this paper while lacking the citation [1]. \n\nThe author claims the contribution includes proposing a general framework since SSNI is adaptive way, which seems questionable. Eq. 7 and Eq. 8, the key of the adaptive method, contain bias term $b$. Then, the ablation study reported in the Appendix show $b$ will make big influence for the overall method and has the different scale for CIFAR-10 and ImageNet. In such case, SSNI should be specific design based on the different datasets thus breaking the claim of general framework.\n\n[1] Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score. Shuhai Zhang, Feng Liu, Jiahao Yang, Yifan Yang, Changsheng Li, Bo Han, Mingkui Tan. ICML 2023", "methods_and_evaluation_criteria": "1. SSNI has been evaluated on PGD and BPDA attacks, which seems not enough. Although Lee et al. [1] suggest that PGD is more threaten to the diffusion-based adversarial attacks, they does not deny the threaten of the AutoAttack. Therefore, AutoAttack should be considered as the baseline at the same time.\n\n2. Lacking the ablation study to support the SSNI-linear (SSNI-L). The all experimental results about the robust performance seems not include the SSNI-L. I have checked all the results and there is only inference comparison between SSNI-L and SSNI-nonlinear (SSNI-N) shown in Table 10, Appendix. This leads that SSNI-L seems redundant.\n\n\n\n[1] Lee, M. and Kim, D. Robust evaluation of diffusion-based adversarial purification. ICCV 2023.", "theoretical_claims": "I have checked all the proof including these in Appendix.\n\nThere are two main weaknesses:\n\n1) Mistakes. For example, the triangle inequality in 946 lines is wrong. It should be $||E(\\ast)|| - ||g(t)x|| > ||E(\\ast) - g(t)x|| $, where $E(\\ast)$ is the abbreviated  for the expectation term.\n\n2) Lacking the main theoretical claims. The key for this paper is indicating that the propose method could calculate more precisely $t^{\\ast}$. However, no theoretical claims to support this.", "experimental_designs_or_analyses": "It lacks the latest baseline such as [1].\n\n[1] Robust Diffusion Models for Adversarial Purification. Guang Lin, Zerui Tao, Jianhai Zhang, Toshihisa Tanaka, Qibin Zhao. Arxiv:2403.16067.", "relation_to_broader_scientific_literature": "No, i think they have cited enough related works.", "essential_references_not_discussed": "Yes, they does not discuss what is different between their score norm finding and EPS [1] \n\n[1] Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score. Shuhai Zhang, Feng Liu, Jiahao Yang, Yifan Yang, Changsheng Li, Bo Han, Mingkui Tan. ICML 2023", "other_strengths_and_weaknesses": "1, The strength is that they use the surrogate process, which is more robust gradient approximation for the diffusion models.\n\n2. The additional weakness is that the overall paper seems the combination among EPS, Diffpure and Lee et al. The evidence is that the adaptive way is mainly based on the metric proposed in EPS. Then, lacking the theoretical proof makes weaken the contribution for this paper.", "other_comments_or_suggestions": "1. The author should make clear clarification for the contribution.\n2. More meaningful theoretical proof should be added.", "questions_for_authors": "No, please see above contents", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No"}}]

Author rebuttal:

Meta-review: All reviewers have provided positive scores for this submission, highlighting its strengths in novelty and experiments. Given the unanimous positive feedback and the recognition of its contribution to the area, the AC carefully reviewed the paper and concurred with the reviewers' assessments, therefore supporting the decision to accept this submission.
Decision: Accept (poster)