GSmooth: Certified Robustness against Semantic Transformations via Generalized Randomized Smoothing

1 INTRODUCTION . Although deep learning models have achieved remarkable success on various applications ( LeCun et al. , 2015 ) , they are vulnerable to adversarial examples ( Biggio et al. , 2013 ; Szegedy et al. , 2013 ; Goodfellow et al. , 2014 ) and semantic transformations ( Hendrycks & Dietterich , 2019 ) . The vulnerability of deep learning models can limit their applications on many important tasks . For example , the autonomous driving system can be misled even by a small adversarial patch on the road mark ( Jing et al. , 2021 ) . Compared with the maliciously crafted adversarial examples , semantic transformations are more practical in real-world scenarios , such as rotation , translation , blur , bad weather , and so on . Such transformations do not damage the semantic features of images and can be easily recognized by humans , but they also degrade the performance of deep learning models . Therefore , it is imperative to improve model robustness against semantic transformations . To develop more reliable machine learning systems , many efforts have been made to design defense techniques against adversarial attacks or semantic transformations . The existing defense methods can be categorized into empirical defenses and certified defenses . Adversarial training ( AT ) ( Madry et al. , 2017 ; Zhang et al. , 2019a ) is one of the most effective empirical defenses against ` p-norm bounded adversarial examples . Moreover , methods based on data augmentation ( Hendrycks et al. , 2019 ; Wang et al. , 2019 ; Calian et al. , 2021 ) have been proposed to empirically improve the performance under semantic transformations . However , the performance of empirical defenses is difficult to be fully justified and these defenses can be further broken by new adaptive attacks ( Athalye et al. , 2018 ; Tramer et al. , 2020 ) . In contrast , the certified defenses aim to theoretically provide a certified region where the model is theoretically safe under any attack or perturbation ( Wong & Kolter , 2018 ; Cohen et al. , 2019 ; Gowal et al. , 2018 ; Zhang et al. , 2019b ) . Along this line , developing certified defense methods is a crucial step towards reliable machine learning systems . Although certified defenses have achieved great success , most of them are limited to defend against ` p-norm bounded attacks . However , the ` p distance between the original image and its corrupted counterpart by a semantic transformation ( e.g. , translation , rotation ) would be large even when the corruption is slight . Therefore , the current methods are incapable of certifying robustness against such semantic perturbations . To solve this problem , several recent works ( Fischer et al. , 2020 ; Mohapatra et al. , 2020 ; Li et al. , 2021 ) attempt to extend the certified defenses to several simple semantic corruptions , including translation , rotation , and Gaussian blur . However , these works are not scalable to certify robustness against complex and general semantic perturbations . First , deterministic certified defenses ( Mohapatra et al. , 2020 ) based on convex relaxation for the activation function require solving a complex optimization problem for computing bound which is computationally expensive . Second , probabilistic approaches based on randomized smoothing also demand a handcrafted Lipschitz bound ( Li et al. , 2021 ) , which is intractable for complicated semantic transformations . For example , many semantic transformations such as glass blur and pixelate do not have a closed form expression or they are black boxes and hard to be analyzed theoretically , but they are common in real-world scenarios . Therefore , it is still highly challenging to certify robustness against these complex and realistic semantic transformations . To address the aforementioned challenges , we propose a generalized randomized smoothing framework ( GSmooth ) . First , we provide a unified framework of GSmooth for certifying general semantic transformations . Then we categorize the transformations into resolvable transformations ( e.g. , translation ) and non-resolvable transformations ( e.g. , rotational blur ) similar with Li et al . ( 2021 ) . As mentioned above , most non-resolvable transformations are complex and the existing methods can not provide their certified radius . To handle the challenge , we propose to use an image-to-image translation neural network to approximate all these transformations . Due to the strong capacity of neural networks , our method is flexible and scalable to model these complex semantic transformations . By introducing an augmented noise in the layers of the surrogate model , we can theoretically provide the certified radius for the proxy neural networks which can be used for certifying the original transformations . Next , we provide theoretical analysis and error bounding for the approximation . Finally , we validate the effectiveness of our methods on several publicly available datasets . Extensive experimental results demonstrate that our methods are effective for certifying complex semantic transformations including different types of blur or image quality corruptions . 2 RELATED WORK . 2.1 ATTACKS AND DEFENSES FOR SEMANTIC TRANSFORMATIONS . Unlike ` p perturbation which adds small noise to every pixel of an image , semantic attacks or physical attacks are usually unrestricted . Brown et al . ( 2017 ) ; Song et al . ( 2018 ) use a small patch added to the image to mislead the classifier or the object detector . Engstrom et al . ( 2019 ; 2018 ) ; Xiao et al . ( 2018 ) construct adversarial examples using spatial transformations like rotation or translation . Hendrycks & Dietterich ( 2019 ) show that a wide variety of semantic perturbations degrade the performance for many deep learning models . Many works ( Cubuk et al. , 2019 ; Hendrycks et al. , 2019 ; 2020 ; Robey et al. , 2020 ) propose diverse data augmentation techniques to enhance robustness under semantic perturbations . Calian et al . ( 2021 ) propose adversarial data augmentation that can be viewed as adversarial training for defending semantic perturbations . Beyond empirical defenses , several works ( Mohapatra et al. , 2020 ; Madry et al. , 2017 ; Singh et al. , 2019 ; Balunović et al. , 2019 ) attempt to certify some simple geometric transformations . However , all of them belong to deterministic certification approaches and their performance on realistic datasets are unsatisfactory . 2.2 RANDOMIZED SMOOTHING . Randomized smoothing is a novel certification method originated from differential privacy ( Lecuyer et al. , 2019 ) . Cohen et al . ( 2019 ) then improve the certified bound and apply it to large scale deep neural networks and datasets . Yang et al . ( 2020 ) exhaustively analyze the robust radius by using different noise distribution and norms . Hayes ( 2020 ) ; Yang et al . ( 2020 ) point out that randomized smoothing suffers from curse of dimensionality for the l∞ norm . Salman et al . ( 2019 ) adopt adversarial training to train smoothed classifiers to obtain better robustness guarantees . Li et al . ( 2021 ) ; Fischer et al . ( 2020 ) extend randomized smoothing to certify some simple semantic transformations , e.g. , image translation and rotation . It shows that randomized smoothing could be generalized to certify more diverse attacks or corruptions . However , their methods are limited to simple semantic transformations , which are easy to analyze their mathematical properties . 3 PROPOSED METHOD . In this section , we present the framework and theoretical analyses of our Generalized Randomized Smoothing ( GSmooth ) . We first introduce the basic notations . Then we divide the semantic transformations into resolvable transformations and non-resolvable transformations similar with Li et al . ( 2021 ) . Next we introduce the details of our GSmooth for these two types of semantic transformations , respectively . Finally , we show the theoretical insight and proof sketch of our main results . 3.1 NOTATIONS . We first introduce the notations and formulation of the task . Given the input of x ∈ Rn and the labels of Y = { 1 , 2 , . . . p } , we denote the classifier as f ( x ) : Rn → [ 0 , 1 ] p , which outputs predicted probabilities over all p classes . The prediction of f is arg maxi∈Y f ( x ) i , where f ( · ) i denotes the i-th element of f ( · ) . Let τ ( θ , x ) : Rm×Rn → Rn be a semantic transformation of raw input x with parameter θ ∈ Rm . We define the smoothed classifier as G ( x ) = Eθ∼g [ f ( τ ( θ , x ) ) ] , ( 1 ) which is the average prediction for the samples under a smoothing distribution g ( θ ) where g ( θ ) = exp ( −ψ ( θ ) ) and ψ ( θ ) is a smooth function from Rm → R. Let ||u|| = 1 be any vector with unit norm and a random variable γu = 〈u , ∇ψ ( δ ) 〉 where δ ∼ g and ∇ is the gradient operator of a function . The complementary CDF is ϕu ( c ) = P [ γu > c ] and the inverse complementary CDF is ϕ−1u ( p ) = inf { c|P ( γu > c ) 6 p } . Following Yang et al . ( 2020 ) , we define a function Φ as Φ ( p ) = max||u||=1 E [ γuI { γu > ϕ−1u ( p ) } ] , ( 2 ) which will be used to represent the certified radius . Let yA= arg maxi∈YG ( x ) i be the predicted label by the smoothed classifier G ( x ) and yB = arg maxi∈Y\yA G ( x ) i is the runner-up class . Without causing confusion , we use G ( x ) A to denote the probability of the top class G ( x ) yA ; likewise for G ( x ) B . 3.2 CERTIFIED BOUND FOR RESOLVABLE SEMANTIC TRANSFORMATIONS . We first discuss a class of transformations that are resolvable — the composition of two transformations with parameters belonging to a perturbation set θ , ξ ∈ P ⊂ Rm is still a transformation with a new parameter γ ( θ , ξ ) ∈ P ⊂ Rm , where γ ( · , · ) : P × P → P is a function depending on these parameters . For resolvable semantic transformations , we have the following theorem . Theorem 1 . Let f ( x ) be any classifier and G ( x ) be the smoothed classifier defined in Eq . ( 1 ) . If there exists a function M ( · , · ) : P × P → R , the transformation τ ( · , · ) satisfies ∂γ ( θ , ξ ) ∂ξ = ∂γ ( θ , ξ ) ∂θ M ( θ , ξ ) , and there exist two constants pA , pB satisfying G ( x ) A > pA > pB > G ( x ) B , then yA = arg maxi∈Y G ( τ ( ξ , x ) ) i holds for any ‖ξ‖ 6 R where R = 1 2M∗ ∫ pA pB 1 Φ ( p ) dp , ( 3 ) and M∗ = maxξ , θ∈P ||M ( ξ , θ ) || . Remark . The settings of Theorem 1 are similar with Li et al . ( 2021 ) for resolvable semantic transformations . But here we adopt a different presentation and proof for the theorem which could be easier to extend to our GSmooth framework for general semantic transformations . Specifically , we show two examples of the theorem which are additive transformations and commutable transformations . A transformation is additive if τ ( θ , τ ( ξ , x ) ) = τ ( ξ + θ , x ) for any θ , ξ ∈ P ; or it is commutable if τ ( θ , τ ( ξ , x ) ) = τ ( ξ , τ ( θ , x ) ) for any θ , ξ ∈ P . For these two types of transformations , it is straightforward to verify that they satisfy the property proposed in Theorem 1 . As an example , we simply apply Theorem 1 for isotropic Gaussian distribution g ( θ ) = N ( 0 , σ2I ) and get the certified radius R = σ 2 ( Ψ ( pA ) −Ψ ( pB ) ) , ( 4 ) where Ψ is the inverse CDF of the standard Gaussian distribution . These two kinds of transformations include image translation and Gaussian blur , which are basic semantic transformations and widely discussed in previous works ( Li et al. , 2021 ; Fischer et al. , 2020 ) . The certification of these simple transformations only requires applying translation or Gaussian blur to the sample and gets the average classification score under the noise distribution .

Summary: The authors propose a randomized smoothing based certification algorithm  for general semantic transformations. The key idea of the work is to use a neural surrogate for the semantic transformations, and add noise in the latent space of the surrogate for randomized smoothing. The neural surrogate appears to convert the non-linear, multiplicative transformation into an additive operation in the latent space, thus allowing for randomized smoothing methods to be applied. Their approach proposes to decompose the resolvable and unresolvable parts of the semantic transformation by lifting the data+transformation parameters into a larger augmented latent space defined by an image-to-image network. The resolvable parts of the transform can then be certified similar to previous works (Yang'2021, Salman'2019) using the Lipschitzness of the smoothed transforms. The non-resolvable part of the transform however are assumed to be Lipschitz in the latent space. The authors provide theoretical and empirical evidence for GSmooth and show improvement on contemporary methods.