Exploiting Safe Spots in Neural Networks for Preemptive Robustness and Out-of-Distribution Detection

1 INTRODUCTION . Deep neural networks have achieved significant performance on various artificial intelligence tasks such as image classification , speech recognition , and reinforcement learning . Despite the results , Szegedy et al . ( 2013 ) demonstrated that deep neural networks are vulnerable to adversarial examples , minute input perturbations designed to mislead networks to yield incorrect predictions . There have been a large number of studies to improve the robustness of networks against adversarial perturbations ( Song et al. , 2017 ; Guo et al. , 2018 ) , while many of the proposed methods have been shown to fail against stronger adversaries ( Athalye et al. , 2018 ; Tramer et al. , 2020 ) . Adversarial training ( Madry et al. , 2017 ) and randomized smoothing ( Cohen et al. , 2019 ) are some of the few methods that survived the harsh verifications , each focusing on empirical and certified robustness , respectively . To summarize , the study of adversarial examples has been an arms race between adversaries , who manipulate inputs to raise network malfunction , and defenders , who aim to preserve the network performance against the corrupted inputs . In this paper , we approach the adversarial robustness problem from a different perspective . Instead of defending networks from already perturbed examples , we assume the situation where the defenders can also influence inputs slightly for their interest before the adversaries ’ incursion . The defenders ’ goal for this manipulation will be to improve robustness by searching for spots in the input space that are resistant to adversarial attacks , given a pre-trained classifier . We explore methods for finding those safe spots from natural images under a given input modification budget and the degree of robustness achievable by utilizing these spots , which we denote as preemptive robustness . Ultimately , we tackle the following question : • Do safe spots always exist in the vicinity of natural images ? One practical example of the proposed framework is the case where a user uploads his or her photo from local storage ( e.g. , mobile device ) to social media ( e.g. , Instagram ) , as illustrated in Figure 1 . Suppose there is an uploader ( A ) who posts a photo on social media , a web user ( B ) who queries a search engine ( e.g. , Google ) for an image , and a search engine that crawls images from social media , indexes them with a neural network , and retrieves the relevant images to B . Our threat model considers an adversary ( M ) that can download A ’ s image from social media , perturb it maliciously , and re-upload the perturbed image on the web , where the search engine may crawl and index images from . The classifier on the search engine will wrongly index the perturbed image , causing the search engine to malfunction . Suppose an African-American uploader ( A ) posts a photo of him or herself on social media , and a racist adversary ( M ) perturbs it to be misclassified as “ gorilla ” by the search engine . When another person ( B ) searches “ gorilla ” on Google , the perturbed image would appear , though the image content shows a photo of A . This attack fools both A and B since the perturbed image is used contrary to A ’ s purpose and is not the image B wanted . To prevent this , the social media company , cooperating with the search engine company , could ask if A agrees that the images are slightly changed when uploaded to make them robust to such attacks . The purpose of the modification process , corresponding to the “ safe spot filter ” in Figure 1 , will be to ensure that the uploaded images are used under A ’ s intention and provide more accurate search results to B . We develop a novel optimization problem for searching safe spots in the vicinity of natural images and observe that over 90 % of the correctly classified images have safe spots nearby for adversarially trained models on both CIFAR-10 ( Krizhevsky & Hinton , 2009 ) and ImageNet ( Russakovsky et al. , 2015 ) . We also find that safe spots can enhance both empirical and certified robustness when applied on smoothed classifiers . Furthermore , we propose a novel safe spot inducing model training scheme to improve the preemptive robustness . By exploiting these safe spot-aware classifiers along with our safe spot search method , we also propose a new algorithm for out-of-distribution detection , which is often addressed together with robustness ( Hendrycks et al. , 2019a ; c ) . Our algorithm outperforms other baselines on near-distribution outlier datasets such as CIFAR-100 ( Krizhevsky & Hinton , 2009 ) . 2 RELATED WORKS . Adversarial training Goodfellow et al . ( 2015 ) first show that the robustness of a neural network can be enhanced by generating adversarial examples and including them in training set . PGD adversarial training improves the robustness against stronger adversarial attacks by augmenting training data with multi-step PGD adversarial examples ( Madry et al. , 2017 ) . Some recent works report performance gains over PGD adversarial training by modifying the adversarial example generation procedure ( Qin et al. , 2019 ; Zhang & Wang , 2019 ; Zhang et al. , 2020 ) . However , most of the recent algorithmic improvements can be matched by simply using early stopping with PGD adversarial training ( Rice et al. , 2020 ; Croce & Hein , 2020 ) . Other line of works achieve performance gains by utilizing additional datasets ( Carmon et al. , 2019 ; Wang et al. , 2020 ; Hendrycks et al. , 2019a ) . Randomized smoothing Injecting random noise during the forward pass can smooth the classifier ’ s decision boundary and improve empirical robustness ( Liu et al. , 2018 ) . Using differential privacy , Lecuyer et al . ( 2019 ) give theoretical guarantees for ` 1 and ` 2 robustness of classifiers smoothed with Gaussian and Laplacian noise . Cohen et al . ( 2019 ) provide a tight bound of ` 2 robustness of networks smoothed with Gaussian noise via the Neyman-Pearson lemma . Another proof of the robustness bound was given in Salman et al . ( 2019 ) using Lipschitz property of smoothed classifiers , where they also propose a new adversarial training scheme for building robust smoothed classifiers . Out-of-Distribution detection with deep networks Although deep networks achieve high performance on various classification tasks , they also tend to yield high confidence in out-of-distribution samples ( Nguyen et al. , 2015 ) . To filter out the anomalous examples , Hendrycks & Gimpel ( 2017 ) use the maximum value of a classifier ’ s softmax distribution as a score function , while Lee et al . ( 2018 ) propose Mahalanobis distance-based metric which spots out-of-distribution samples using hidden features . Hendrycks et al . ( 2019b ) show that leveraging auxiliary datasets disjoint from test-time data can improve the detection performance . Recently , Sastry & Oore ( 2020 ) characterize activity patterns of hidden features by Gram matrices and use the matrix values to identify anomalies . 3 METHODS . 3.1 GENERAL DEFINITION OF SAFE SPOT AND PREEMPTIVE ROBUSTNESS . We first establish a formal definition of safe spot and preemptive robustness . Let c : X → Y be a classifier which maps images to class labels . We define the safe region of the classifier c as the set of images that c can output robust predictions in the presence of slight adversarial perturbations . Definition 1 ( -safe region ) . Let c : X → Y be a classifier and ∈ R+ be the perturbation budget of an adversary . The -safe region of the classifier c is defined by S ( c ) : = { x ∈ X | c ( x′ ) = c ( x ) , ∀x′ ∈ B ( x ) } . Here , B ( x ) = { x′ ∈ X | d ( x , x′ ) ≤ } is the x-centered -ball . In this paper , we assume ` p threat model , i.e. , d ( x , x′ ) = ‖x − x′‖p , which is the most common setting on adversarial robustness literature , and consider p ∈ { 2 , ∞ } . Now , suppose a defender can preemptively manipulate a natural image xo under a small modification budget , knowing its ground-truth label yo . We denote the modified output image as xs . Then , the defender ’ s objective is to make xs be correctly classified as yo and locate in the safe region S ( c ) to improve the robustness against adversarial attacks . If xs satisfies these two conditions , then we say xo is preemptively robust and xs is a safe spot of xo . Definition 2 ( Preemptive robustness ) . Let c : X → Y be a classifier and δ , ∈ R+ be the modification budgets of the defender and the adversary , respectively . A natural image xo with its ground-truth label yo is called ( δ , ) -preemptively robust on the classifier c if there exists a safe spot xs ∈ Bδ ( xo ) such that ( i ) c ( xs ) = yo and ( ii ) xs ∈ S ( c ) . 3.2 SAFE SPOT SEARCH ALGORITHM . In this subsection , we develop an algorithm for searching a safe spot with a natural image . Given a classifier c , finding a safe spot xs from a natural image xo can be formulated as the following problem , which is directly from the definition of safe spot : minimize xs 1c ( xs ) 6=yo + 1xs /∈S ( c ) subject to ‖xs − xo‖p ≤ δ , where 1 is the 0-1 loss function . Note in this formulation the defender requires the ground-truth label yo for the safe spot search . However , images in the real-world ( e.g. , social media ) are usually unlabeled , unless uploaders annotate labels to their images by hand . So , it is natural to assume that the defender can not access the ground-truth label yo . In this case , we utilize the classifier ’ s prediction c ( xo ) instead of yo : minimize xs 1c ( xs ) 6=c ( xo ) + 1xs /∈S ( c ) subject to ‖xs − xo‖p ≤ δ . As xs /∈ S ( c ) implies there exists an adversarial example xa ∈ B ( xs ) such that c ( xa ) 6= c ( xs ) , we can reformulate the optimization problem as minimize xs 1c ( xs ) 6=c ( xo ) + sup xa 1c ( xa ) 6=c ( xs ) subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ . Since the 0-1 loss is not differentiable , we employ the cross-entropy loss ` : X × Y → R+ of the classifier c as the convex surrogate loss function : minimize xs ` ( xs , c ( xo ) ) + sup xa ` ( xa , c ( xs ) ) ( 1 ) subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ . Let h ( xs ) denote the objective in Equation ( 1 ) . Instead of minimizing h ( xs ) , we minimize h̃ ( xs ) = supxa ` ( xa , c ( xo ) ) , since it upper bounds h ( xs ) when sufficiently minimized by Lemma 1 . Algorithm 1 Finding a safe spot input An image and its prediction ( xo , c ( xo ) ) , the cross-entropy function ` xs = xo for i = 1 , . . . , MAXITER do Generate N adversarial examples for n = 1 , . . . , N do x ( 0 ) a , n = xs + ηn where ηn ∼ U ( B ( 0 ) ) for t = 1 , . . . , T do x ( t ) a , n = Πxs , ( f ( x ( t−1 ) a , n ; c ( xo ) , ` ) ) end for end for xs ← Πxo , δ ( xs − β · 1 N N∑ n=1 ∂ ` ( x ( T ) a , n , c ( xo ) ) ∂xs ) end for output xs Lemma 1 . If h̃ ( xs ) ≤ − log ( 0.5 ) ' 0.6931 , then h ( xs ) ≤ 2h̃ ( xs ) . Proof . See Supplementary A.1 . Finally , we have the following optimization problem : minimize xs sup xa ` ( xa , c ( xo ) ) ( 2 ) subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ . To solve Equation ( 2 ) , we first approximate the inner maximization problem by running T -step PGD ( Madry et al. , 2017 ) whose dynamics is given by x ( 0 ) a = xs + η ( random start ) x̃ ( t ) a = f ( x ( t−1 ) a ; c ( xo ) , ` ) ( adversarial update ) x ( t ) a = Πxs , ( x̃ ( t ) a ) , ( projection ) where η is a random noise uniformly sampled from the ` p zero-centered -ball , f is FGSM ( Goodfellow et al. , 2015 ) defined by f ( x ; y , ` ) = x+ α · sgn ( ∇x ` ( x , y ) ) if p =∞ x+ α · ∇x ` ( x , y ) ‖∇x ` ( x , y ) ‖2 if p = 2 , and Πxs , is a projection operation to B ( xs ) . Then , we iteratively solve the approximate problem given by replacing xa to x ( T ) a in Equation ( 2 ) . To update xs , we need to compute the gradient of ` ( x ( T ) a , c ( xo ) ) with respect to xs expressed as ∂ ` ( x ( T ) a , c ( xo ) ) ∂xs = ∂f̃ ( x ( 0 ) a ) ∂x ᵀ · · · ∂f̃ ( x ( T−1 ) a ) ∂x ᵀ · ∇x ` ( x ( T ) a , c ( xo ) ) , where ∂f̃/∂x is the Jacobian matrix of f̃ = Πxs , ◦ f which is easily computed via back-propagation . After computing the gradient , we update xs by projected gradient descent method : x ( i+1 ) s = Πxo , δ ( x ( i ) s − β · ∂ ` ( x ( T ) a , c ( xo ) ) ∂xs ) . Note that the loss ` ( x ( T ) a , c ( xo ) ) is now a random variable dependent on η . Therefore , we generate N adversarial examples { x ( T ) a , n } Nn=1 with different noises and optimize the sample mean of the losses instead . Algorithm 1 shows the overall safe spot search algorithm and Figure 2 illustrates our optimization process .

Summary: The authors argue that there are some safe "spots" in the data space that are less prone to adversarial attacks. The authors propose a technique to identify such "safe spots". They then leverage them for robust training and observe higher robust accuracy than baseline. Finally, they leverage this observation to identify out of distribution data.