MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius

1 INTRODUCTION . Modern neural network classifiers are able to achieve very high accuracy on image classification tasks but are sensitive to small , adversarially chosen perturbations to the inputs ( Szegedy et al. , 2013 ; Biggio et al. , 2013 ) . Given an image x that is correctly classified by a neural network , a malicious attacker may find a small adversarial perturbation δ such that the perturbed image x + δ , though visually indistinguishable from the original image , is assigned to a wrong class with high confidence by the network . Such vulnerability creates security concerns in many real-world applications . Researchers have proposed a variety of defense methods to improve the robustness of neural networks . Most of the existing defenses are based on adversarial training ( Szegedy et al. , 2013 ; Madry et al. , 2017 ; Goodfellow et al. , 2015 ; Huang et al. , 2015 ; Athalye et al. , 2018 ; Ding et al. , 2020 ) . During training , these methods first learn on-the-fly adversarial examples of the inputs with multiple attack iterations and then update model parameters using these perturbed samples together with the original labels . However , such approaches depend on a particular ( class of ) attack method . It can not be formally guaranteed whether the resulting model is also robust against other attacks . Moreover , attack iterations are usually quite expensive . As a result , adversarial training runs very slowly . Another line of algorithms trains robust models by maximizing the certified radius provided by robust certification methods ( Weng et al. , 2018 ; Wong & Kolter , 2018 ; Zhang et al. , 2018 ; Mirman et al. , 2018 ; Wang et al. , 2018 ; Gowal et al. , 2018 ; Zhang et al. , 2019c ) . Using linear or convex relaxations of fully connected ReLU networks , a robust certification method computes a “ safe radius ” r for a classifier at a given input such that at any point within the neighboring radius-r ball of the input , the classifier is guaranteed to have unchanged predictions . However , the certification methods are usually computationally expensive and can only handle shallow neural networks with ReLU activations , so these training algorithms have troubles in scaling to modern networks . In this work , we propose an attack-free and scalable method to train robust deep neural networks . We mainly leverage the recent randomized smoothing technique ( Cohen et al. , 2019 ) . A randomized smoothed classifier g for an arbitrary classifier f is defined as g ( x ) = Eηf ( x + η ) , in which η ∼ N ( 0 , σ2I ) . While Cohen et al . ( 2019 ) derived how to analytically compute the certified radius of the randomly smoothed classifier g , they did not show how to maximize that radius to make the classifier g robust . Salman et al . ( 2019 ) proposed SmoothAdv to improve the robustness of g , but it still relies on the expensive attack iterations . Instead of adversarial training , we propose to learn robust models by directly taking the certified radius into the objective . We outline a few challenging desiderata any practical instantiation of this idea would however have to satisfy , and provide approaches to address each of these in turn . A discussion of these desiderata , as well as a detailed implementation of our approach is provided in Section 4 . And as we show both theoretically and empirically , our method is numerically stable and accounts for both classification accuracy and robustness . Our contributions are summarized as follows : • We propose an attack-free and scalable robust training algorithm by MAximizing the CErtified Radius ( MACER ) . MACER has the following advantages compared to previous works : – Different from adversarial training , we train robust models by directly maximizing the certified radius without specifying any attack strategies , and the learned model can achieve provable robustness against any possible attack in the certified region . Additionally , by avoiding time-consuming attack iterations , our proposed algorithm runs much faster than adversarial training . – Different from other methods ( Wong & Kolter , 2018 ) that maximize the certified radius but are not scalable to deep neural networks , our method can be applied to architectures of any size . This makes our algorithm more practical in real scenarios . • We empirically evaluate our proposed method through extensive experiments on Cifar-10 , ImageNet , MNIST , and SVHN . On all tasks , MACER achieves better performance than state-of-the-art algorithms . MACER is also exceptionally fast . For example , on ImageNet , MACER uses 39 % less training time than adversarial training but still performs better . 2 RELATED WORK . Neural networks trained by standard SGD are not robust – a small and human imperceptible perturbation can easily change the prediction of a network . In the white-box setting , methods have been proposed to construct adversarial examples with small ` ∞ or ` 2 perturbations ( Goodfellow et al. , 2015 ; Madry et al. , 2017 ; Carlini & Wagner , 2016 ; Moosavi-Dezfooli et al. , 2015 ) . Furthermore , even in the black-box setting where the adversary does not have access to the model structure and parameters , adversarial examples can be found by either transfer attack ( Papernot et al. , 2016 ) or optimization-based approaches ( Chen et al. , 2017 ; Rauber et al. , 2017 ; Cheng et al. , 2019 ) . It is thus important to study how to improve the robustness of neural networks against adversarial examples . Adversarial training So far , adversarial training has been the most successful robust training method according to many recent studies . Adversarial training was first proposed in Szegedy et al . ( 2013 ) and Goodfellow et al . ( 2015 ) , where they showed that adding adversarial examples to the training set can improve the robustness against such attacks . More recently , Madry et al . ( 2017 ) formulated adversarial training as a min-max optimization problem and demonstrated that adversarial training with PGD attack leads to empirical robust models . Zhang et al . ( 2019b ) further decomposed the robust error as the sum of natural error and boundary error for better performance . Finally , Gao et al . ( 2019 ) proved the convergence of adversarial training . Although models obtained by adversarial training empirically achieve good performance , they do not have certified error guarantees . Despite the popularity of PGD-based adversarial training , one major issue is that its speed is too slow . Some recent papers propose methods to accelerate adversarial training . For example , Freem ( Shafahi et al. , 2019 ) replays an adversarial example several times in one iteration , YOPO-m-n ( Zhang et al. , 2019a ) restricts back propagation in PGD within the first layer , and Qin et al . ( 2019 ) estimates the adversary with local linearization . Robustness certification and provable defense Many defense algorithms proposed in the past few years were claimed to be effective , but Athalye et al . ( 2018 ) showed that most of them are based on “ gradient masking ” and can be bypassed by more carefully designed attacks . It is thus important to study how to measure the provable robustness of a network . A robustness certification algorithm takes a classifier f and an input point x as inputs , and outputs a “ safe radius ” r such that for any δ subject to ‖δ‖ ≤ r , f ( x ) = f ( x + δ ) . Several algorithms have been proposed recently , including the convex polytope technique ( Wong & Kolter , 2018 ) , abstract interpretation methods ( Singh et al. , 2018 ; Gehr et al. , 2018 ) and the recursive propagation algrithms ( Weng et al. , 2018 ; Zhang et al. , 2018 ) . These methods can provide attack-agnostic robust error lower bounds . Moreover , to achieve networks with nontrivial certified robust error , one can train a network by minimizing the certified robust error computed by the above-mentioned methods , and several algorithms have been proposed in the past year ( Wong & Kolter , 2018 ; Wong et al. , 2018 ; Wang et al. , 2018 ; Gowal et al. , 2018 ; Zhang et al. , 2019c ; Mirman et al. , 2018 ) . Unfortunately , they can only be applied to shallow networks with limited activation and run very slowly . More recently , researchers found a new class of certification methods called randomized smoothing . The idea of randomization has been used for defense in several previous works ( Xie et al. , 2017 ; Liu et al. , 2018 ) but without any certification . Later on , Lecuyer et al . ( 2018 ) first showed that if a Gaussian random noise is added to the input or any intermediate layer . A certified guarantee on small ` 2 perturbation can be computed via differential privacy . Li et al . ( 2018 ) and Cohen et al . ( 2019 ) then provided improved ways to compute the ` 2 certified robust error for Gaussian smoothed models . In this paper , we propose a new algorithm to train on these ` 2 certified error bounds to significantly reduce the certified error and achieve better provable adversarial robustness . 3 PRELIMINARIES . Problem setup Consider a standard classification task with an underlying data distribution pdata over pairs of examples x ∈ X ⊂ Rd and corresponding labels y ∈ Y = { 1 , 2 , · · · , K } . Usually pdata is unknown and we can only access a training set S = { ( x1 , y1 ) , · · · , ( xn , yn ) } in which ( xi , yi ) is i.i.d . drawn from pdata , i = 1 , 2 , · · · , n. The empirical data distribution ( uniform distribution over S ) is denoted by p̂data . Let f ∈ F be the classifier of interest that maps any x ∈ X to Y . Usually f is parameterized by a set of parameters θ , so we also write it as fθ . We call x′ = x + δ an adversarial example of x to classifier fθ if fθ can correctly classify x but assigns a different label to x′ . Following many previous works ( Cohen et al. , 2019 ; Salman et al. , 2019 ) , we focus on the setting where δ satisfies ` 2 norm constraint ‖δ‖2 ≤ . We say that the model fθ is l 2-robust at ( x , y ) if it correctly classifies x as y and for any ‖δ‖2 ≤ , the model classifies x+δ as y . In the problem of robust classification , our ultimate goal is to find a model that is l 2-robust at ( x , y ) with high probability over ( x , y ) ∼ pdata for a given > 0 . Neural network In image classification we often use deep neural networks . Let uθ : X → RK be a neural network , whose output at input x is a vector ( u1θ ( x ) , ... , u K θ ( x ) ) . The classifier induced by uθ ( x ) is fθ ( x ) = arg maxc∈Y u c θ ( x ) . In order to train θ by minimizing a loss function such as cross entropy , we always use a softmax layer on uθ to normalize it into a probability distribution . The resulting network is zθ ( · ; β ) : X → P ( K ) 1 , which is given by zcθ ( x ; β ) = e βucθ ( x ) / ∑ c′∈Y e βuc ′ θ ( x ) , ∀c ∈ Y , β is the inverse temperature . For simplicity , we will use zθ ( x ) to refer to zθ ( x ; β ) when the meaning is clear from context . The vector zθ ( x ) = ( z 1 θ ( x ) , · · · , zKθ ( x ) ) is commonly regarded as the “ likelihood vector ” , and zcθ ( x ) measures how likely input x belongs to class c. Robust radius By definition , the l 2-robustness of fθ at a data point ( x , y ) depends on the radius of the largest l2 ball centered at x in which fθ does not change its prediction . This radius is called the robust radius , which is formally defined as R ( fθ ; x , y ) = { inf fθ ( x′ ) 6=fθ ( x ) ‖x′ − x‖2 , when fθ ( x ) = y 0 , when fθ ( x ) 6= y ( 1 ) Recall that our ultimate goal is to train a classifier which is l 2-robust at ( x , y ) with high probability over the sampling of ( x , y ) ∼ pdata . Mathematically the goal can be expressed as to minimize the expectation of the 0/1 robust classification error . The error is defined as l 0/1 −robust ( fθ ; x , y ) : = 1− 1 { R ( fθ ; x , y ) ≥ } , ( 2 ) and the goal is to minimize its expectation over the population L 0/1 −robust ( fθ ) : = E ( x , y ) ∼pdata l 0/1 −robust ( fθ ; x , y ) . ( 3 ) 1The probability simplex in RK . It is thus quite natural to improve model robustness via maximizing the robust radius . Unfortunately , computing the robust radius ( 1 ) of a classifier induced by a deep neural network is very difficult . Weng et al . ( 2018 ) showed that computing the l1 robust radius of a deep neural network is NP-hard . Although there is no result for the l2 radius yet , it is very likely that computing the l2 robust radius is also NP-hard . Certified radius Many previous works proposed certification methods that seek to derive a tight lower bound of R ( fθ ; x , y ) for neural networks ( see Section 2 for related work ) . We call this lower bound certified radius and denote it by CR ( fθ ; x , y ) . The certified radius satisfies 0 ≤ CR ( fθ ; x , y ) ≤ R ( fθ ; x , y ) for any fθ , x , y . The certified radius leads to a guaranteed upper bound of the 0/1 robust classification error , which is called 0/1 certified robust error . The 0/1 certified robust error of classifier fθ on sample ( x , y ) is defined as l 0/1 −certified ( fθ ; x , y ) : = 1− 1 { CR ( fθ ; x , y ) ≥ } ( 4 ) i.e . a sample is counted as correct only if the certified radius reaches . The expectation of certified robust error over ( x , y ) ∼ pdata serves as a performance metric of the provable robustness : L 0/1 −certified ( fθ ) : = E ( x , y ) ∼pdata l 0/1 −certified ( fθ ; x , y ) ( 5 ) Recall that CR ( fθ ; x , y ) is a lower bound of the true robust radius , which immediately implies that L 0/1 −certified ( fθ ) ≥ L 0/1 −robust ( fθ ) . Therefore , a small 0/1 certified robust error leads to a small 0/1 robust classification error . Randomized smoothing In this work , we use the recent randomized smoothing technique ( Cohen et al. , 2019 ) , which is scalable to any architectures , to obtain the certified radius of smoothed deep neural networks . The key part of randomized smoothing is to use the smoothed version of fθ , which is denoted by gθ , to make predictions . The formulation of gθ is defined as follows . Definition 1 . For an arbitrary classifier fθ ∈ F and σ > 0 , the smoothed classifier gθ of fθ is defined as gθ ( x ) = arg max c∈Y Pη∼N ( 0 , σ2I ) ( fθ ( x+ η ) = c ) ( 6 ) In short , the smoothed classifier gθ ( x ) returns the label most likely to be returned by fθ when its input is sampled from a Gaussian distribution N ( x , σ2I ) centered at x. Cohen et al . ( 2019 ) proves the following theorem , which provides an analytic form of certified radius : Theorem 1 . ( Cohen et al. , 2019 ) Let fθ ∈ F , and η ∼ N ( 0 , σ2I ) . Let the smoothed classifier gθ be defined as in ( 6 ) . Let the ground truth of an input x be y . If gθ classifies x correctly , i.e . Pη ( fθ ( x+ η ) = y ) ≥ max y′ 6=y Pη ( fθ ( x+ η ) = y ′ ) ( 7 ) Then gθ is provably robust at x , with the certified radius given by CR ( gθ ; x , y ) = σ 2 [ Φ−1 ( Pη ( fθ ( x+ η ) = y ) ) − Φ−1 ( max y′ 6=y Pη ( fθ ( x+ η ) = y ′ ) ) ] = σ 2 [ Φ−1 ( Eη1 { fθ ( x+η ) =y } ) − Φ −1 ( max y′ 6=y Eη1 { fθ ( x+η ) =y′ } ) ] ( 8 ) where Φ is the c.d.f . of the standard Gaussian distribution .

Summary: This paper improves the robustness of smoothed classifiers by maximizing the certified radius, which is more efficient than adversarially train the smoothed classifier and achieves higher average robust radius and better certified robustness when the radius is not much larger than the training sigma. It proposes a novel objective which is derived by decomposing the 0/1 certified loss into the sum of 0/1 classification error and 0/1 robustness error. Three conditions are identified to make the optimization doable. Two surrogate losses (CE and hinge loss on the certified radius) for the two 0/1 errors are proposed as upper bounds of the 0/1 loss. Certified radius is derived as a function of the logits of Soft-RS to make the hinge loss differentiable. Numerical stability of the proposed objective is also analyzed by showing its gradient is bounded.