Efficient randomized smoothing by denoising with learned score function

1 INTRODUCTION . The deep image classifiers are susceptible to deliberate noises as known as adversarial attacks ( Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ; Carlini & Wagner , 2017 ) . Even though many works proposed heuristics that can annul or mitigate adversarial attacks , most of them were broken by stronger attacks ( Athalye et al. , 2018 ; Athalye & Carlini , 2018 ) . The vulnerability of empirical defenses had led the researchers to scrutinize on certified defenses , which ensure the models to have constant output within the allowed set around given input . Unfortunately , many provable defenses are not feasible to large-scale neural networks because of their constraints on the architecture . On the other hand , randomized smoothing is a practical method that does not restrain the choice of neural networks . The randomized smoothing converts any base classifier to a smoothed classifier by making predictions over randomly perturbed samples . Then the smoothed classifiers are guaranteed to have a ` p certified radius , which is theoretically derived by the noise type used for smoothing . Since Cohen et al . ( 2019 ) derived tight ` 2 certified radius for Gaussian randomized smoothing , sequential works studied the certification bounds for various distributions ( Teng et al. , 2020 ; Yang et al. , 2020 ) . As base classifiers are required to predict randomly perturbed samples , natural classifiers are not sufficient for randomized smoothing . Therefore , many works proposed training ensemble of base classifiers accustomed for randomized smoothing . However , since each trained classifier only applies to specific noise distribution and level , it is expensive to protect against various ` p adversaries and robustness strength . In this work , we tackle the inefficiency of training random-ensemble of base classifiers by using one universal image denoiser to the pre-trained classifier . The idea of using denoiser for randomized smoothing was first introduced by Salman et al . ( 2020 ) and is refer to denoised smoothing . One step further , we study general image denoising problem for randomized smoothing with two different approaches : 1 ) direct training of image denoiser , and 2 ) solve the optimization problem by using a generative model to project to the learned data manifold . Then , we show that the score function , which is the gradient of log-density , is crucial for both approaches . We exploit multi-scale denoising score matching ( Song & Ermon , 2019 ) for score estimation , and propose an efficient algorithm simulated annealing for image denoising . Remark that we only require one score network to certify various noise distributions and levels . We provide experimentations on ImageNet and CIFAR-10 datasets to show the efficacy of our methods . Specifically , our denoisers perform better than original denoised smoothing , while can be applied to various noise types without any re-training . Further- more , we compare with the random-ensemble based method , which we refer to white-box smoothing , and show that our method works are comparable to them . In sum , we list our contributions : • We propose novel score-based image denoisers for randomized smoothing . • We improve denoised smoothing , which was originally proposed by Salman et al . ( 2020 ) and generalize to other distributions without training any neural networks . 2 RANDOMIZED SMOOTHING AND DENOISED SMOOTHING . 2.1 BACKGROUNDS ON RANDOMIZED SMOOTHING . Let f : Rd → Y be a classifier and q be a distribution on Rd . Then the randomized smoothing with q is a method that converts the base classifier f to the associated smoothed classifier g , where g ( x ) returns the class which is most likely to be predicted by the base classifier f when x is perturbed by a random noise sampled from q , i.e. , g ( x ) = arg max c∈Y Pr u∼q ( u ) [ f ( x + u ) = c ] . ( 1 ) The noise distribution is usually a symmetric log-concave distribution , i.e . q ( u ) = exp ( −φ ( u ) ) for some even and convex φ . Note that to control the robustness/accuracy tradeoff , we embed the noise level λ to q , then we have qλ ( u ) = exp ( −φ ( uλ ) ) . We mix the notations q and qλ throughout the paper . Robustness guarantee for smoothed classifiers Suppose an adversary can perturb the input x inside the allowed set B , which is usually an ` p ball centered at x . For the case when B is ` 2 ball and q is Gaussian distribution N ( 0 , σ2I ) , g ( x ) is robust within the radius R = σ 2 ( Φ−1 ( p1 ) − Φ−1 ( p2 ) ) ( 2 ) where Φ is inverse cumulative distribution function , and p1 = maxc Pr [ f ( x + u ) = c ] and p2 = maxc6=g ( x ) Pr [ f ( x + u ) = c ] . Cohen et al . ( 2019 ) first derived the certified radius by using Neyman-Pearson lemma , and later Salman et al . ( 2019a ) showed alternative derivation using the Lipschitz property of smoothed classifier . Furthermore when q is a centered Laplace distribution , the robustness certificate for ` 1 radius was derived by Teng et al . ( 2020 ) . Later , the proof methods are generalized to various distributions ( may not be log-concave ) that can certify various ` p radius ( Yang et al. , 2020 ) . Remark that the robustness guarantee depends on the noise distribution qλ and the performance of base classifier f under random perturbation with qλ . 2.2 RANDOMIZED SMOOTHING VIA IMAGE DENOISING . Even though the randomized smoothing can convert any classifier to a provably robust classifier , the smoothed classifier from natural classifiers are below the standard as they are not capable of predicting randomly perturbed samples . Many previous studies focused on training classifiers accustomed to randomized smoothing , which spans from noisy data augmentation ( Cohen et al. , 2019 ; Li et al. , 2019 ) to its variants such as adversarial training ( Salman et al. , 2019a ) or stability training ( Lee et al. , 2019 ; Zhai et al. , 2019 ) . However , such methods are computationally expensive and require a massive number of classifiers per noise types and levels . The idea of prepending denoiser to the classifier was first introduced by Salman et al . ( 2020 ) . By training denoiser Dθ : Rd → Rd , the smoothed classifier converted from f ◦ Dθ outperforms ’ no-denoiser ’ baseline . They proposed training denoisers with mean squared error ( MSE ) loss or classification ( CLF ) loss , or combining both methods . Formally , they are LMSE ( θ ) = Ex∼p , u∼q [ ‖Dθ ( x + u ) − x‖2 ] , ( 3 ) LCLF ( θ ) = Ex∼p , u∼q [ LCE ( F ( Dθ ( x + u ) ) , f ( x ) ) ] . ( 4 ) where LCE is the cross-entropy loss and F is soft version of hard classifier f . They showed that training with CLF loss makes perform better than denoiser with only MSE loss . Alternatively , Saremi & Srivastava ( 2020 ) trained neural empirical bayes estimator that can refine the white noise . Nonetheless , those methods still suffer from expensive training of numerous denoisers with respect to each noise types and levels . 3 SCORE-BASED IMAGE DENOISING . 3.1 FORMULATION OF IMAGE DENOISING PROBLEM . The image denoising is an example of linear inverse problem , which can be formulated as following : given an observation y = x + u with u ∼ q ( u ) finds x̂ ( y ) that is close to original x . Let x ∼ p ( x ) then the distribution of y is pq ( y ) = ∫ p ( y , x ) dx = ∫ p ( y|x ) p ( x ) dx = ∫ q ( y − x ) p ( x ) dx = ( p ∗ q ) ( y ) . One-step denoiser Like equation 3 , the most common approach to achieve denoiser is to train denoising autoencoder ( DAE ) Dθ with MSE loss ( Zhang et al. , 2017 ; ? ) . Suppose q is a Gaussian distributionN ( 0 , σ2I ) and let the distribution of y by pσ2 . Then the following proposition ( Robbins , 1956 ; Lu & Stephens , 2019 ; Saremi & Hyvarinen , 2020 ) reveals the relationship between the optimal denoiser Dθ∗ and pσ2 . Proposition 3.1 . Assume θ∗ ∈ arg minθ LMSE ( θ ) , then the following equation holds : Dθ∗ ( y ) = y + σ2∇y log pσ2 ( y ) ( 5 ) The proof of proposition 3.1 is in Appendix A . Let us define the score function of density p ( x ) by ∇x log p ( x ) , then the optimal DAE can be obtained by estimating the score of pσ2 . Let sθ ( · ; σ ) be score network that estimates score of smoothed density pσ2 . Then the denoiser from sθ is given by x̂ ( y ) = y + σ2sθ ( y ; σ ) . ( 6 ) Remark that it is only valid when q is Gaussian distribution . Multi-step denoiser Consider the maximum a posteriori ( MAP ) estimator that maximizes the conditional distribution p ( x|y ) . Formally the MAP loss is given by , arg min x LMAP ( x ; y ) = arg min x − log p ( x|y ) ( 7 ) = arg min x − log p ( x ) − log p ( y|x ) + log p ( y ) ( 8 ) = arg min x − log p ( x ) − log q ( y − x ) ( 9 ) = arg min x − log p ( x ) + φ ( y − x ) . ( 10 ) Note that we simply remove density term p ( y ) and rewrite with q. Lastly , we rewrite q with φ . Since the density p ( x ) is usually intractable for high-dimensional dataset , one may use approximation to make the MAP loss tractable . Many recent works focused on using cutting edge generative models such as generative adversarial network ( GAN ) or invertible neural networks to approximate p ( x ) in equation 9 ( Ulyanov et al. , 2018 ; Whang et al. , 2020 ; Asim et al. , 2020 ) . However , GAN suffer from mode collapse , and invertible neural networks require extremely long steps to reach local minima , which are not sufficient for randomized smoothing . Instead , we aim to approximate the gradient of LMAP by the score of Gaussian smooth densities . Let the approximate MAP loss with σ̃ by LMAP , σ̃ ( x ; y ) = − log pσ̃2 ( x ) + φ ( y − x ) . ( 11 ) Then we can approximate the gradient of LMAP , σ̃ ( x ; y ) by score network and perform gradient descent initialized with x0 = y as following : xt+1 = xt − α∇xtLMAP , σ̃ ( x ; y ) ≈ xt + α ( sθ ( xt ; σ̃ ) +∇xtφ ( y − xt ) ) . ( 12 ) Remark that the proposed method can be applied to any log-concave noise distributions . Following theorem shows the recovery guarantee of our methods when q is a Gaussian distribution . Theorem 3.2 . Let x∗ be local optimum of p ( x ) , and y = x∗ + u where u ∼ N ( 0 , σ2I ) . Assume − log p is µ-strongly convex within the neighborhood Br ( x ) = { z : ‖z − x‖ ≤ r } . Then , the gradient descent method on approximate loss LMAP , σ̃2 ( x ; y ) initialized by x0 = y converges to its local minima x̂ ( y ; σ̃ ) ∈ arg minLMAP , σ̃2 ( x ; y ) that satisfies : E‖x̂ ( y ; σ̃ ) − x∗‖2 ≤ σ √ d ( 1 + µσ̃2 ) 1 + µσ̃2 + µσ2 + σ̃ √ d ( 13 ) The proof of theorem 3.2 is in Appendix A . Remark that the upper bound in equation 13 increases as σ increases , which shows that the recovery becomes harder as σ becomes larger . Also the upper bound is strictly increasing function of σ̃ , and has the minimum when σ̃ = 0 .

Summary: This paper presents a denoising-based method for randomized smoothing that converts a base classifier into a smoothed one with p-robustness to adversarial examples. It considers a practical setting where the retraining/finetuning of the base classifier is largely inapplicable (e.g. the commercial classification service with only API provided to users).  To do this, it adopts a recently proposed methodology termed denoised smoothing [1] by prepending a custom-trained denoiser to the pretrained classifier. The major novelty of this work lies at the proposed denoising method using learned score function. The new denoising method only requires training one score network and is readily applicable to defend various $l_p$ adversaries, which is a key feature not available in [1].  The experiments show the proposed method outperforms the previous denoising-based approach, and is sometimes on par with the white-box approach [2] that manipulates the classifier.