Consistency Purification: Effective and Efficient Diffusion Purification towards Certified Robustness

Authors: yiquan  li
zhongzhu  chen
kun  jin
jiongxiao  wang
jiachen  lei
bo  li
chaowei  xiao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:02:18

Abstract: Diffusion Purification, purifying noised images with diffusion models, has been widely used for enhancing certified robustness via randomized smoothing. However, existing frameworks often grapple with the balance between efficiency and effectiveness. While the Denoising Diffusion Probabilistic Model (DDPM) offers an efficient single-step purification, it falls short in ensuring purified images reside on the data manifold. Conversely, the Stochastic Diffusion Model effectively places purified images on the data manifold but demands solving cumbersome stochastic differential equations, while its derivative, the Probability Flow Ordinary Differential Equation (PF-ODE), though solving simpler ordinary differential equations, still requires multiple computational steps. In this work, we demonstrated that an ideal purification pipeline should generate the purified images on the data manifold that are as much semantically aligned to the original images for effectiveness in one step for efficiency. Therefore, we introduced Consistency Purification, an efficiency-effectiveness Pareto superior purifier compared to the previous work. Consistency Purification employs the consistency model, a one-step generative model distilled from PF-ODE, thus can generate on-manifold purified images with a single network evaluation. However, the consistency model is designed not for purification thus it does not inherently ensure semantic alignment between purified and original images. To resolve this issue, we further refine it through Consistency Fine-tuning with LPIPS loss, which enables more aligned semantic meaning while keeping the purified images on data manifold. Our comprehensive experiments demonstrate that our Consistency Purification framework achieves state-of-the-art certified robustness and efficiency compared to baseline methods.

Main paper: 
 introduction: 
 Diffusion models were first proposed for high-quality image generation [1, 2, 3, 4, 5] and have been extended to generative tasks across various modalities, including audio [6, 7, 8] , video [9, 10] , and 3D object [11, 12, 13] . A diffusion model for image generation typically involves two key processes:(1) a forward diffusion process, which transforms the source image into an isotropic Gaussian by gradually adding Gaussian noise, and (2) the reverse diffusion process, which uses a Deep Neural Network (DNN) to perform iterative denoising starting from random Gaussian noise.Due to the inherent denoising capability of diffusion models, there have been widely applied to improve the robustness of DNNs. This enhancement is achieved by Diffusion Purification [14, 15, 16, 17, 18] , which purifies the network inputs to reduce the effects of various types of unforeseen corruptions or adversarial attacks. Among these, one particularly suitable and effective scenario of purification is to improve certified robustness through randomized smoothing [19] for image classification tasks. This method guarantees a tight robustness in the ℓ 2 norm with a smoothed classifier. However, many previous works [19, 20, 21, 22, 23, 24] have shown that it still requires retraining with Gaussian augmented examples for each noise level to optimize the smoothed classifier. Diffusion models, capable of purifying Gaussian perturbed images before classification, can be seamlessly integrated with any base classifier to produce a smoothed classifier for arbitrary noise levels. This integration has been demonstrated to effectively enhance certified robustness, as supported by numerous studies [18, 25, 26, 27] . However, current diffusion purification for certified robustness via randomized smoothing still faces significant trade-offs between efficiency and effectiveness. Although Denoising Diffusion Probabilistic Model (DDPM) [28] only requires one single network evaluation in the purification process [25] , it generates the mean of the posterior data distribution conditioned the noisy sample, which does not necessarily locate on the data manifold and may exhibit ambiguity during classification. To further improve diffusion purification, various methods such as DensePure [26] , Local Smoothing [27] and Noised Diffusion Classifiers [29] are applied. However, these methods are considerably less efficient as they require multiple times of the computational costs compared to one-step DDPM. Another promising approach involves using the Probability Flow Ordinary Differential Equation (PF-ODE) [3] . It has offered a method to accelerate the sampling process [4] and achieved a closer distribution to the original data, well balancing efficiency and effectiveness. However, several computational steps are still needed to solve the ODE numerically.To find a Pareto superior solution in terms of efficiency and effectiveness, we introduce a new framework, Consistency Purification, which integrating consistency models into diffusion purification with Consistency Fine-tuning. The consistency model is a novel category of diffusion models that learns the trajectory of the PF-ODE that transits the data distribution to the noisy distribution. It is trained to map any point along this trajectory back to its starting point. This property is desirable for diffusion purification, as it allows images with any scale of Gaussian noise to be directly purified to the clean images. Distilled from a pre-trained diffusion model by simulating the PF-ODE trajectory, the consistency model can generate high-quality in-distribution images in a single step, thereby ensuring both efficiency and effectiveness. However, since consistency models are primarily trained for image generation, it may not suffice to guarantee that the purified image that maintains the same semantic meaning as the original image. To address this issue, we propose adding a Consistency Fine-tuning step into the purification framework, which further fine-tunes the consistency model using Learned Perceptual Image Patch Similarity (LPIPS) [30] loss, aiming to minimize the perceptual differences between the purified and original images, thereby ensuring better semantic alignment, while at the same time, ensuring the purified images still lie on the data manifold.We show that Consistency Purification is Pareto superior compared to baselines from two aspects. First of all, compared with effective methods like DensePure [26] , Local Smoothing [27] and Noised Diffusion Classifiers [29] , Consistency Purification is much more efficient since it enables single-step purification. Secondly, compared with efficient method like onestep-DDPM [25] , we provide both theoretical analysis and experiment results to support the effectiveness improvement of Consistency Purification. In Example 3.1, we show an one-dimensional example demonstrating that consistency model can generate on-manifold purified samples while onestep-DDPM does not have this property.In Theorem 3.3, we show an important theoretical result that given a purifier, the lower the transport from the original distribution to the purified distribution (a measure of distance between probability distributions, see [31] ), the higher the probability that the purified sample is sufficiently close to the original sample, and thus the better purification outcomes. Our experiment results verify that both the integration of consistency model in Consistency Purification and the further Consistency Fine-tuning decreases such transport and achieves better semantic alignment between purified samples and original samples.Beyond the validation of our theory, we conduct comprehensive experiments to demonstrate the empirical improvements of Consistency Purification. Compared to various baseline settings, our approach has shown significant improvements, achieving an average 5% gain in performance over the previous onestep-DDPM under the same cost with single-step purification. These observations underscore our success in finding a Pareto superior diffusion purification framework in both efficiency and effectiveness for certified robustness. 
backgrounds: 
 Randomized Smoothing [19] . Randomized smoothing is designed to certify the robustness of a given classifier under ℓ 2 norm perturbations. Given a base classifier f and an input x, randomized smoothing first defines the smoothed classifier by g(x) = arg max c P ϵ∼N (0,σ 2 I) (f (x + ϵ) = c), where σ is the noise level, which controls the trade-off between robustness and accuracy. [19] shows that g(x) induces the certifiable robustness for x under the ℓ 2 norm with radius R, whereR = σ 2 Φ -1 (p A ) -Φ -1 (p B ), where p A and p B are the probability of the most probable class and "runner-up" class respectively; Φ is the inverse of the standard Gaussian CDF. The p A and p B can be estimated with arbitrarily high confidence via the Monte Carlo method. [3] . The diffusion model has two components: the diffusion process followed by the reverse process. Given an input random variable x 0 ∼ p, the diffusion process adds isotropic Gaussian noises to the data so that the diffused random variable at time t is x t = √ α t (x 0 + ϵ t ), s.t., ϵ t ∼ N (0, σ 2 t I), and σ 2 t = (1 -α t )/α t , and we denote x t ∼ p t . The forward diffusion process can also be defined by the stochastic differential equation dx = D(x, t)dt + G(t)dw, (SDE) where x 0 ∼ p, D : R d × R → R d is the drift coefficient and typically has the form D(x, t) = D(t)x. G : R → R is the diffusion coefficient, dt is an infinitesimal time step, and w(t) ∈ R n is the standard Wiener process.The reverse process exists and removes the added noise by solving the reverse-time SDE [32] dx = [D(t)x -G(t) 2 ▽ x log p t (x)]dt + G(t)dw, (reverse-SDE) where p t (x) denotes the marginal distribution at time t, and w(t) is a reverse-time standard Wiener process. [3] defined the probability flow ODE (PF ODE) which has the same marginal distribution as reverse-SDE but can be solved much fasterdx = D(t)x -1 2 G(t) 2 ∇ x log p t (x) dt. (PF-ODE)As shown in [4] , the perturbation kernel of SDE has the general form Discrete-Time Diffusion Model (DDPM [28] ). DDPM constructs a discrete Markov chainp 0t (x(t) | x(0)) = N x(t); s(t)x(0), s(t) 2 σ(t){x 0 , x 1 , • • • , x i , • • • , x N }as the forward process for the training data x 0 ∼ p, such thatEQUATIONare predefined noise scales such that x N approximates the Gaussian white noise. Denoteα i = N i=1 (1 -β i ), we have P(x i |x 0 ) = N (x i ; √ α i x 0 , (1 -α i )I), i.e., x t (x 0 , ϵ) = √ α i x 0 + (1 -α i )ϵ, ϵ ∼ N (0, I).The reverse process of DDPM learns a reverse direction variational Markov chain p θ (x i-1 |x i ) = N (x i-1 ; µ θ (x i , i), Σ θ (x i , i)). [28] defines ϵ θ as a function approximator to predict ϵ fromx i such that µ θ (x i , i) = 1 √ 1-βi x i -βi √ 1-αi ϵ θ (x i , i) .Then the reverse time samples are generated byxi-1 = 1 √ 1-βi xi -βi √ 1-αi ϵ θ * ( xi , i) + √ β i ϵ, ϵ ∼ N (0, I), and the optimal parameters θ * are obtained by solvingθ * := arg min θ E x0,ϵ ∥ϵ -ϵ θ ( √ α i x 0 + (1 -α i ), i)∥ 2 2 .[28] also provided a one-step approximate reconstruction of x 0 from any x t ,x 0 ≈ x0 = x t - √ 1 -α t ϵ θ (x t ) / √ α t . (onestep-DDPM)Consistency Model [33] . Given a solution trajectory of PF-ODE, the consistency model is defined as D : (x t , t) → x ϵ . The model exhibits the property of self-consistency, ensuring that its outputs are consistent for arbitrary pairs of (x t , t) from the same PF-ODE trajectory; specifically,D(x t , t) = D(x t ′ , t ′ ) for all t, t ′ ∈ [ϵ, T ].As shown by the definition, consistency models are suitable for oneshot denoising, allowing for the recovery of x ϵ from any noisy input x t in one network evaluation. Two distinct training strategies can be employed for training the consistency models: distillation mode and isolation mode. The primary distinction lies in whether the models distill the knowledge from pre-trained diffusion models or train from initial parameters. According to the experiments reported in [33] , consistency models trained in the distillation mode have been shown to outperform those trained in isolation mode for generating high-quality images. Consequently, our paper only considers consistency models trained in the distillation mode. 
theoretical analysis: 
 In this section, we provide theoretical explanations on the advantages of Consistency Purification, with a focus on its purification performance improvement in terms of certified robustness over [25] .As demonstrated in [3] , PF-ODE maintains the marginal distribution of reverse-SDE, thereby establishing a deterministic mapping between the noisy distribution x t and the data distribution x 0 . In other words, PF-ODE guarantees that the purified sample lies on the data manifold, unlike onestep-DDPM, which lacks this assurance. We present here a simple one dimensional example for illustration. Example 3.1. Consider a one-dimensional space with a data set consisting of two samples {y 1 , y 2 }, where y 1 = 1 and y 2 = -1. The distribution can be represented as a mixture of Dirac delta distributions: p data (x) = 1 2 (δ(x -y 1 ) + δ(x -y 2 )). By setting s(t) = 1 and σ(t) = t in perturbationkernel, the distribution at time t becomes:p t (x) = 1 2t √ 2π e -1 2 ( x-1 t ) 2 + e -1 2 ( x+1 t ) 2 .Thend log pt(x) dx = -( x-1 t )e -1 2 ( x-1 t 2 ) 2 -( x+1 t )e -1 2 ( x+1 t 2 ) 2 2t √ 2πpt(x) = -x t 2 + e -1 2 ( x-1 t ) 2 -e -1 2 ( x+1 t ) 2 e -1 2 ( x-1 t ) 2 +e -1 2 ( x+1 t ) 2 .From the derivative formula d log pt(x) dx , it's evident that x = 0 is an equilibrium point, and the right-hand side expression is Lipschitz continuous around x = 0 by L'Hôpital's rule. Thus, according to the Picard-Lindelöf theorem, any trajectory starting on either side of x = 0 will not cross this point. As PF-ODE drives p t (x) closer to the Dirac delta distribution p data (x) as t approaches zero, any initial point on positive/negative side of x = 0 will eventually approach 1 or -1, i.e., the data manifold. Furthermore, in this example, PF-ODE generates not only a purified sample on the data manifold but also closest to the noisy sample. This property is desirable as it establishes a relatively large "robust" neighborhood around each true data point, which implies high certified robustness and a significant certified radius, which will be further discussed later. With the consistency model, we do not need to solve the ODE but rather directly map the noisy sample to either 1/ -1 depending on its location relative to x = 0.For comparison, given any x and t, the onestep-DDPM will output a posterior mean that ise -1 2 ( x-1 t ) 2 -e -1 2 ( x+1 t ) 2 e -1 2 ( x-1 t ) 2 +e -1 2 ( x+1 t ) 2 = e 2x t 2 -1 e 2x t 2 +1.The posterior mean will be near 1 or -1 only when t is sufficiently small compared to ∥x∥. Otherwise, it deviates from the data manifold. In the case when t is large, the posterior mean will be close to zero, locating in an ambiguous classification region. In adversarial purification [25, 26, 14] , we typically select t based on the variance of the noise added to the data sample rather than using an very small t. This practice helps avoid significant deviations in the posterior mean estimation due to the imperfect estimation of score/noise. With a very small t, even a slight bias in score/noise estimation can lead to a substantial deviation, resulting in a denoised sample even farther from the data manifold represented by p data (x).Additionally, PF-ODE is deterministic, eliminating the overhead of majority voting required when using reverse-SDE as a purifier [26] . The consistency model, which reduces ODE solving to a one-step mapping, further ensures purification has the same efficiency as onestep-DDPM while keeping the in-distribution property.Though the consistency model enjoys both in-distribution property and one-step efficiency, it does not guarantee that the purified sample has the same semantic meaning as the original sample. This is because the derivation of PF-ODE only guarantees a mapping between noisy distribution and data distribution, which is sufficient for generation, but not enough for denoising purposes.To address this concern, we first delineate the desired characteristics of the purifier. As evidenced in prior works [14, 25, 26, 34] , an ideal purifier should yield a purified output situated within a proximate vicinity of the original input. It is generally presumed that such purified outputs retain the semantic meaning of the original inputs with a high probability. The disparity in semantic consistency between the noisy input and the purified output generated by PF-ODE arises due to the proximity of the purified output to other samples. In this regard, we propose quantifying this disparity through the notion of transport between the data distribution and the purified distribution, derived by introducing Gaussian perturbations to the data distribution and subsequently applying denoising via PF-ODE. Given an original sample x, Gaussian noise ϵ, and purifier d, the mapping in the transport process is defined as T : x → d(x + ϵ), which is probabilistic. We aim to demonstrate that a diminished transport between the data distribution and the purified distribution is conducive to a higher likelihood of the purified output being situated in proximity to the original sample, thereby preserving its semantic meaning.We will leverage the following definition. Definition 3.2. Given the data distribution p, Gaussian noise ϵ, timestep t, and a purifier d, we define π t : x → d(x + tϵ) and the "transport" under g t between the data distribution and purified distribution asT πt (p) := ∥x -π t (x)∥ • p(x)dx.Intuitively, transport measures the distance between the original and purified samples, which should be small by an effective purifier. Below, we quantify this intuition and present our main theorem. See the detailed proof in Appendix B. Theorem 3.3. Given the transport T πt (p) between the data distribution p and the corresponding purified distribution under g t , then for any r > 0, the probability that the distance between the original sample x and purified sample x = π t (x) is larger than r is upper bounded by Tπ t (p) r . Remark 3.4. By Theorem 3.3, the efficacy of the purifier hinges on two crucial factors: the transport T πt (p) and the radius r. A theoretically perfect purifier would yield zero transport; however, this is unattainable due to the inherent randomness of g t . Typically, we can optimize the parameter t to minimize the transport, denoted as T * = min t Tπ t (p) r . In the context of classification tasks, the selection of r also depends on the robustness of the classifier; a more robust classifier allows a larger r to be chosen, thereby guarantee better purification efficacy. For ensuring consistency in semantic meaning between the original and purified samples, it is insufficient merely to minimize their distance; it is also necessary that the purified sample resides on the data manifold, which is the in-distribution property we previously mentioned. To concurrently achieve both objectives, rather than solely focusing on minimizing the Euclidean distance between the original and purified samples, we opt to minimize the Learned Perceptual Image Patch Similarity (LPIPS) loss between them. This strategy aids in mitigating the risk of the purified sample deviating from the data manifold, thereby preserving semantic meaning. In Table 1 , we show that using LPIPS is better than ℓ 1 and ℓ 2 loss for Consistency Fine-tuning when we want to guarantee the generated images are in-distribution, where lower FID scores indicate better in-distribution properties.Figure 2 validates the effectiveness of Consistency Purification based on our results in Theorem 3.3, it shows that both the integration of consistency model in Consistency Purification and the further Consistency Fine-tuning can decrease the transport from the original distribution to the purified distribution. Specifically, we can see that Consistency Purification achieves a lower average distance from the purified sample to the original sample compared with onestep-DDPM, and Consistency Fine-tuning further decreases this average distance, indicating both components result in a lower transport and thus a better semantic alignment between purified samples and original samples. 
method: 
 We propose our framework, Consistency Purification, with a further improved version using Consistency Fine-tuning.We introduce Consistency Purification, directly applying consistency model as a purifier to integrate with a base classifier into smoothed classifier for randomized smoothing.Following Diffusion Denoised Smoothing outlined in [25] , it is necessary to establish a mapping between Gaussian noise augmented images required by randomized smoothing and the noised image in the ODE trajectory of consistency model. For a given consistency model purifier D θ , any noisy input x t ∼ N (x, t 2 I) can be recovered to the trajectory's start x ϵ by directly passing it through the model with time t:x ϵ = D θ (x t , t).When comparing this to the image augmented with additive Gaussian noise x rs ∼ N (x, σ 2 I), which is required by randomized smoothing, we observe that x rs and x t share the same formula when t = σ. However, since the variances σ ∈ {σ i } m i=1 may not be used during the training of the consistency model, we empirically select the nearest time step t from the discrete time steps used in training for each σ.For the entire time horizon [ϵ, T ] with N -1 sub-interval boundaries t 1 = ϵ < t 2 < • • • < t N = T , the time steps used in training are computed by:t i = (ϵ 1/ρ + i-1 / N -1 (T 1/ρ -ϵ 1/ρ )) ρ , where ρ = 7.Given the variance σ of Gaussian noise used in randomized smoothing, we select the corresponding time step t * σ for Consistency Purified Smoothing byEQUATION]}.To optimize the consistency model for aligning semantic meanings during purification, we fine-tune the purifier D θ by minimizing the following loss function:L θ = E∥x -D θ (x σ , t * σ )∥ LPIPS , where the expectation is taken with x ∼ p data , σ ∼ U {σ i } m i=1 , x σ ∼ N (x, σ 2 I).Here LPIPS denotes the distance computed by the Learned Perceptual Image Patch Similarity [30] . p data represents the distribution of the training data, from which clean images x are sampled. U{σ i } m i=1 denotes the uniform distribution over m different noise scales σ i used for randomized smoothing. Typically, we select the scale set σ i ∈ {0.25, 0.5, 1.0}, which is commonly used to compute the certified radius via randomized smoothing.After obtaining the fine-tuned consistency model purifier D θ * ,it can replace the original model used in Consistency Purified Smoothing to purify any noised image x rs with Gaussian variance σ i , resulting in the final purified image x p by x p = D θ * (x rs , t * σi ). We present the detailed algorithm of our Consistency Purification in Appendix A. 
experiments: 
 In this section, we begin by detailing the experimental settings, followed by our main results. Additionally, we conduct ablation studies to further demonstrate the effectiveness of our framework. All experiments are conducted with 1×NVIDIA RTX A5000 24GB GPU.Dataset. We evaluate the Consistency Purification framework on both CIFAR-10 [35] and ImageNet-64 [36] . CIFAR-10 contains 32 × 32 pixel images across 10 different categories while ImageNet-64 includes 64 × 64 pixel images across 1000 categories. Due to limited computational resources, we select 500 test images for CIFAR-10 from the 10,000 CIFAR-10 test set, choosing every 20th example in sequence (e.g., the 1st, 21st, 41st, etc.). Similarly, for the ImageNet-64 dataset, we sample 500 test examples from its 50,000 test examples using a fixed interval of 100.Consistency Purification. For CIFAR-10, to demonstrate the effectiveness of Consistency Purification, we first perform purification with a public unconditional consistency model [37] . After that, to further improve the performance, we fine-tune the model with noise levels σ sampling from {0.25, 0.5, 1.0}, shown as the (+ Consistency Fine-tuning). However, currently there is no publicly available unconditional consistency model checkpoint for the ImageNet dataset that can be used directly for purification purposes. The only available model is the conditional consistency model on ImageNet-64. Thus, here we trained an unconditional consistency model on ImageNet-64, initializing it with the existing conditional consistency model checkpoint. Details of the training process are included in Appendix C. Additionally, we also conduct Consistency Fine-tuning on ImageNet-64 model with noise levels σ ∈ {0.05, 0.15, 0.25}.Baselines. For comparative analysis of CIFAR-10, we conduct baseline experiments under various settings. The first baseline involves onestep-DDPM, where we employ the 50-M unconditional improved diffusion models from [2] utilizing the one-shot denoising method [25] for purification. Given that our consistency model is distilled from an EDM model [4] , we include EDM as our baselines, applying both one-shot denoising (onestep-EDM) and ODE solver (PF-ODE EDM) for purification. Additionally, we include the recent advancement in diffusion purification methods, Diffusion Calibration, as a baseline following [38] , which fine-tunes the diffusion model with the guidance of classifier WideResNet28-10 to improve the purification accuracy under the specific classifier. While for ImageNet-64, due to the lack of public unconditional EDM model, we only include the comparison baseline with onestep-DDPM.Randomized Smoothing Settings. We set N = 10000 for both CIFAR-10 and ImageNet as the number of sampling times used in randomized smoothing. We compute the certified radius for each test example at three different noise levels σ ∈ {0.25, 0.5, 1.0} for CIFAR-10 and σ ∈ {0.05, 0.15, 0.25} for ImageNet-64. Then we calculate the proportion of test examples whose radius exceeds a specific threshold ϵ. The highest accuracy among these noise levels is reported as the certified accuracy at ϵ.For the classifier used after purification for CIFAR-10, we employ ViT-B/16 model [39] , which is pretrained on ImageNet-21k [36] and finetuned on CIFAR-10 dataset. In our ablation studies, we also use ResNet [40] and WideResNet [41] trained on CIFAR-10. For ImageNet-64, we make up-sampling on the 64×64 images and directly apply ViT-B/16 as the classifier.We present the certified accuracy of Consistency Purification for both CIFAR-10 and ImageNet-64 dataset, with the results presented in Table 2 . We also include the purification steps which decide whether the purifier needs multiple evaluation times through the networks (Multi Steps) other than a single network evaluation (One Step). As observed from Table 2 , Consistency Purification significantly outperforms onestep-DDPM for both CIFAR-10 and ImageNet-64 with even higher certified accuracy with Consistency Fine-tuning. Besides, for CIFAR-10, the results also suggest the effectiveness of Consistency Purification with Consistency Fine-tuning when compared with more baseline methods such as onestep-EDM, PF-ODE EDM and Diffusion Calibration. For the detailed certified accuracy evaluation of fine-grained ϵ at different noise levels σ, we present the results in Figure 3 compared with the onestep-DDPM setting. All results have demonstrated that Consistency Purification is able to certify robustness with both efficiency and effectiveness. To better illustrate the significant improvement in certified robustness brought by Consistency Purification, we present visualizations of images after diffusion purification in Figure 4 for CIFAR-10 at a noise level of σ = 0.5, compared with the onestep-DDPM approach. As shown, our method produces significantly higher-quality purified images than onestep-DDPM. Furthermore, these purified images achieve a notably higher classification accuracy when evaluated by the same classifier. Additional visualization examples for ImageNet-64 are included in Appendix D. We conduct various ablation studies to evaluate the effectiveness of our proposed method. Comparison with Non-Diffusion-based Baselines. To compare Consistency Purification with various non-diffusion-based approaches, we conducted additional experiments to compute the certified accuracy under three non-diffusion-based methods [19, 22, 42] . [19] first proposed training a classifier with noisy images to ensure certified robustness. Subsequent works [22, 42] build on [19] 's methodology, attempting to enhance the smoothed classifier by adding prediction consistency regularization, or incorporating per-sample bias. The experimental results presented in Tabel 3 show that our method surpasses all previous non-diffusion-based methods in achieving higher certified accuracy, particularly with a significantly high clean performance at ϵ = 0.0. Furthermore, in contrast to non-diffusion-based methods, which incur significant costs by requiring additional fine-tuning of robust classifiers for each specific noise level, our method can be applied directly to any off-the-shelf classifiers, significantly broadening its practical applications.Fine-tuning Loss Functions. To further demonstrate that LPIPS loss is the best choice considering both on-manifold purification and semantic meaning alignment, we assess the certified accuracy of Consistency Purification using different loss functions during Consistency Fine-tuning. Instead of LPIPS distance between the clean and purified images as the loss function, we experiment with ℓ 1 and ℓ 2 distances. Results in Table 4 indicate that Consistency Purification with LPIPS loss achieves the highest Certified Accuracy. In contrast, fine-tuning with ℓ 1 and ℓ 2 distances compromises the purification performance for certification. This demonstrates that fine-tuning with LPIPS loss function effectively aligns semantic meanings, whereas ℓ 1 or ℓ 2 distances may hurt them.Noise Levels Sampling Schedules during Consistency Fine-tuning. In our experiments of Consistency Fine-tuning, we simply select the same sampling schedules of noise levels σ ∼ U{0.25, 0.5, 1.0}, uniformly sampling σ used in randomized smoothing. To empirically demonstrate its effectiveness, we compare this approach with continuous sampling schedules where σ ∼ U[0, 1].Results presented in Table 5 show that our discrete sampling schedule achieves higher certified accuracy. This indicates that fine-tuning with a discrete scale, aligned with the noise levels used in randomized smoothing, enhances certified robustness. Generalizability with Different Classifiers. We compute certified accuracy with various classifiers to test if our framework maintains its effectiveness with arbitrary classifiers. The results, presented in Table 6 , compare Consistency Fine-tuning with Diffusion Calibration, an alternative method to fine-tune diffusion models for improving the certified robustness. When evaluated across different classifiers, including ViT-B/16, ResNet56, and WideNet28-10, our method outperforms Diffusion Calibration except certified accuracy at ϵ = 0.0 on WRN28-10 model. It is worth noting that the Diffusion Calibration, which requires a specific classifier for guidance during fine-tuning, exhibits limitations, only achieving comparable performance with the guidance classifier WRN28-10. This demonstrates the advantages of Consistency Fine-tuning in generalizing across different classifiers.Fine-tuning Classifier vs. Fine-tuning Diffusion Model. A potential concern with Consistency Fine-tuning is the higher certified accuracy and lower training cost associated with Fine-tuning the Classifier (CLS-FT) compared to our approach of Fine-tuning the Diffusion Model (DM-FT). However, our experiments, as shown in Table 7 , indicate that DM-FT does not conflict with CLS-FT; rather, combining these two methods achieves even higher certified accuracy. On another hand, although CLS-FT yield slightly higher certified accuracy than DM-FT, its requirement for fine-tuning a specific classifier compromises the natural property of diffusion purification frameworks with arbitrary off-the-shelf classifiers, thus limiting the practical applicability.  
conclusion: 
 In this paper, we introduced Consistency Purification, a novel framework proposed to enhance certified robustness via randomized smoothing. By incorporating consistency models into diffusion purification approach and further refining them through Consistency Fine-tuning, our empirical experiments have demonstrate the framework's ability to achieve high certified robustness efficiently with one single network evaluation for purification.Limitations. A notable limitation of our study is that our empirical results do not include computing certified robustness of high-resolution images such as ImageNet 256×256. This constraint is due to the absence of publicly available checkpoints for the consistency model at this resolution. Additionally, training a consistency model for ImageNet 256×256 would require huge computing resources, which are currently beyond our affordability. However, our framework is designed for adaptability and could be easily extended to ImageNet 256×256 once these checkpoints become available. As a result, our empirical evaluations in this paper are limited to the CIFAR-10 and ImageNet 64×64 datasets. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our paper does not have such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have included licenses of original owners in our code and made citations in our papers. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: Our paper does not release new assets. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We provide detailed descriptions of Consistency Purification in the following algorithms. Algorithm 1 presents the function of Consistency Fine-tuning and Consistency Purification respectively. Algorithm 2 shows the randomized smoothing algorithm from [19] with applying Consistency Purification to do prediction and compute the certified radius. x σ ← x + N (0, σ 2 I) 5: t * σ ← GETTIMESTEP(σ) 6: L ← LPIPS(x, D θ (x σ , t * )) 7: θ ← θ -η∇ θ L 8: until convergence 9:return D θ 10: end function11: function CONSISTENCYPURIFICATION(f clf , x, σ) 12: t * σ ← GETTIMESTEP(σ) 13:x rs ← x + N (0, σ 2 I) 14:x p ← D θ * (x rs , t * σ ) 15: y ← f clf (x p ) 16:return y 17: end function Given the transport T πt (p) between the data distribution p and the corresponding purified distribution under g t , then for any r > 0, the probability that the distance between the original sample x and purified sample x = π t (x) is larger than r is upper bounded byt i ← (ϵ 1/ρ + i-1 N -1 (T 1/ρ -ϵ 1/ρ )) ρ for i ∈ {1, . . . , N } 20: t * σ ← find {t i |σ ∈ ti-1+ti 2 , ti+ti+1Tπ t (p) r .Proof. We can leverage the Markov's inequality. BecauseE[∥x -x∥] = ∥x-x∥≤r ∥x -x∥ • p(x)dx + ∥x-x∥>r ∥x -x∥ • p(x)dx ≥ ∥x-x∥>r ∥x -x∥ • p(x)dx ≥ ∥x-x∥>r r • p(x)dx = r • P (∥x -x∥ > r),we haveP (∥x -x∥ > r) ≤ E[∥x -x∥] r = E[∥x -π t (x)∥] r = T πt (p) r .Algorithm 2 Randomized Smoothing [19] Input: Sampling times for prediction n. Sampling times for certification N . Significant confidence level α. Function LOWERCONFBOUND(k, n, 1 -α) returns a one-sided (1-α) lower confidence interval for the Binomial parameter p given that k ∼ Binomial(n, p).1: function PREDICT(f clf , x, σ, n, α) 2: counts ← 0 3: for i ∈ {1, 2, . . . , n} do 4: y ← CONSISTENCYPURIFICATION(f clf , x, σ) 5: counts[y] ← counts[y] + 1 6: end for 7:ŷA , ŷB ← top two labels in counts 8:EQUATION2 ) ≤ α then p A ← LOWERCONFBOUND(counts[ŷ A ], N, 1 -α) 29: if p A > 1 2 then 30:return prediction ŷA and radius σΦ -1 (p A ) We train an unconditional consistency model for ImageNet-64 from the public available conditional version by transiting the class embedding layers to a learnable token, initialization with average class embeddings. For each model forwarding, this token will be combined with the time embeddings for computation. After that, we train the conditional consistency model, initialized with the unconditional model's parameters, on ImageNet-64 training set for 120k steps.We present visualization images after diffusion purification by applying onestep-DDPM and Consistency Purification for ImageNet-64 under the noise level σ = 0.25 in Figure 5 . Justification: There is no societal impact of our work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper introduces Consistency Purification, a novel framework that integrates consistency models with diffusion purification to enhance the certified robustness of deep neural networks.  The paper provides both theoretical and experimental verification of the proposed method, and is well-written.  Some concerns raised by the Reviewers on the limited novelty and experimental evaluations were successfully addressed by the authors’ rebuttal.  The authors should incorporate all revision and improvement promised in the rebuttal into the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces Consistency Purification, a novel framework that integrates consistency models with diffusion purification to enhance the certified robustness of deep neural networks, achieving efficient and effective image purification. The framework is further refined through Consistency Fine-tuning with LPIPS loss to ensure semantic alignment between the original and purified images, demonstrating state-of-the-art performance in both certified robustness and efficiency compared to existing methods.
strength_weakness: strength_weakness: It's noteworthy that there have been non-diffusion-based prior studies for enhancing certified robustness. A comparative analysis with those works would add depth to this study.
The experimental setting, including baseline selection and the selection of evaluation data, is not consistent with the baseline method One-Step DDPM. In this paper, only 100 samples from ImageNet are selected for evaluation. It would be beneficial to evaluate the method across multiple benchmarks and with more test samples.
questions: questions: See Weakness.
limitations: limitations: None
paper_summary: paper_summary: The paper introduces the Consistency Model for one-step purification, ensuring the data manifold of the purified samples while maintaining the efficiency of one-step purification. At the same time, the paper proposes Consistency Finetuning to fine tune the Consistency Model to ensure semantic consistency of the purified samples.
strength_weakness: strength_weakness: 1. The paper's innovation is limited, using existing Consistency Models and LPIPS loss to form the diffpure framework.
2. There are some descriptions not easy to understand. Suggest the author to provide a brief explanation and clarification when transport (Page 3, line 77) first appears in the Introduction. Does Remark 3.4 (line202) need to be bolded?
3. The paper's validation set on ImageNet-64 is too small, and 100 images are difficult to accurately reflect its validity. Meanwhile, one of the purposes of this paper is to enhance the effectiveness, but the experimental dataset does not align with this original intention.
questions: questions: 1. In Example 3.1, the author states in lines 177-178 that if distribution consistency is already being ensured, it should already be a relatively strong constraint. Why can only ensure generation and not enough for denoising?
2. The paper's statements in 188-191 and the proof in Theorem 3.3 demonstrate the importance of small transport. However, the ideal denoising should be to transform d into x. For a defined transport, it is natural and reasonable to require it to be small. What the author needs to prove more is whether this Gaussian transport lower can be transferred to adversarial attacks, rather than just proving that lower is good?
limitations: limitations: The paper has addressed the limitations.
paper_summary: paper_summary: This paper presents an innovative approach to image purification using diffusion models, known as Consistency Purification. Traditional methods like the Denoising Diffusion Probabilistic Model (DDPM) and the Stochastic Diffusion Model face challenges in balancing efficiency and effectiveness. While DDPM provides efficient purification, it fails to ensure that purified images lie on the data manifold. On the other hand, the Stochastic Diffusion Model successfully places images on the data manifold but is computationally intensive. The proposed Consistency Purification leverages a consistency model distilled from the Probability Flow Ordinary Differential Equation (PF-ODE), achieving on-manifold purification in a single step. This approach is further refined through Consistency Fine-tuning with LPIPS loss, enhancing semantic alignment between purified and original images. Comprehensive experiments demonstrate that this framework achieves state-of-the-art certified robustness and efficiency compared to baseline methods.
strength_weakness: strength_weakness: 1. There is no comparison with multi-step DDPM methods. Providing these results and the corresponding speedup would make the work more solid.
2. The impact of noise intensity on experimental results is not mentioned. Could you please show this in the rebuttal phase? 
3. It is unclear how the model performs if the noise is not Gaussian. Providing results for different noise distributions would enhance the robustness and applicability of the method.
4. Including visualizations of the purified images before and after the purification process would make the results more intuitive and compelling.
questions: questions: refer to weakness
limitations: limitations: refer to weakness

Comments: title: Look forward to your reply
Look forward to your reply
Response to your valuable feedback
comment: I understand the authors’ response and efforts, i keep my rating.
Dear Reviewer aFWW,

The deadline for the discussion period is approaching. We have provided our rebuttal material and hopefully could address your concerns. Your feedback is highly valuable to us, and we would greatly appreciate it if you could take some time to review our response.

Best Regards,

Authors
Dear Reviewer 43Xy,

The deadline for the discussion period is approaching. We have provided our rebuttal material and hopefully could address your concerns. Your feedback is highly valuable to us, and we would greatly appreciate it if you could take some time to review our response.

Best Regards,

Authors
Thank you for your response! We have expanded the 500 test images to 512 for evaluation to align with common diffusion purification settings ('Diffusion Models for Adversarial Purification' and 'Robust Evaluation of Diffusion-Based Adversarial Purification'). The results in the following table show that consistency purification with consistency fine-tuning, consistently achieves the highest certified accuracy, demonstrating the effectiveness of our approach.

|                       | 0.0  | 0.05 | 0.15 | 0.25 | 0.35 |
|----|----|----|----|----|----|
| onestep-DDPM          | 55.3 | 44.7 | 33.4 | 15.2 |  8.8 |
| Consistency Purification | 62.3 | 54.3 | 35.2 | 19.7 | 13.1 |
| + Consistency Fine-tuning   | **68.8** | **58.0** | **37.5** | **23.4** | **17.4** |
Thank you for your detailed response. Regarding the model evaluation using a subset of 500 images, it is better to use 512 images for evaluation to align with many diffusion purification settings ( "Diffusion Models for Adversarial Purification" and "Robust Evaluation of Diffusion-Based Adversarial Purification"). I appreciate your detailed explanation of the contribution, and the rating has been adjusted accordingly.