Exploiting Safe Spots in Neural Networks for Preemptive Robustness and Out-of-Distribution Detection

1 INTRODUCTION . Deep neural networks have achieved significant performance on various artificial intelligence tasks such as image classification , speech recognition , and reinforcement learning . Despite the results , Szegedy et al . ( 2013 ) demonstrated that deep neural networks are vulnerable to adversarial examples , minute input perturbations designed to mislead networks to yield incorrect predictions . There have been a large number of studies to improve the robustness of networks against adversarial perturbations ( Song et al. , 2017 ; Guo et al. , 2018 ) , while many of the proposed methods have been shown to fail against stronger adversaries ( Athalye et al. , 2018 ; Tramer et al. , 2020 ) . Adversarial training ( Madry et al. , 2017 ) and randomized smoothing ( Cohen et al. , 2019 ) are some of the few methods that survived the harsh verifications , each focusing on empirical and certified robustness , respectively . To summarize , the study of adversarial examples has been an arms race between adversaries , who manipulate inputs to raise network malfunction , and defenders , who aim to preserve the network performance against the corrupted inputs . In this paper , we approach the adversarial robustness problem from a different perspective . Instead of defending networks from already perturbed examples , we assume the situation where the defenders can also influence inputs slightly for their interest before the adversaries ’ incursion . The defenders ’ goal for this manipulation will be to improve robustness by searching for spots in the input space that are resistant to adversarial attacks , given a pre-trained classifier . We explore methods for finding those safe spots from natural images under a given input modification budget and the degree of robustness achievable by utilizing these spots , which we denote as preemptive robustness . Ultimately , we tackle the following question : • Do safe spots always exist in the vicinity of natural images ? One practical example of the proposed framework is the case where a user uploads his or her photo from local storage ( e.g. , mobile device ) to social media ( e.g. , Instagram ) , as illustrated in Figure 1 . Suppose there is an uploader ( A ) who posts a photo on social media , a web user ( B ) who queries a search engine ( e.g. , Google ) for an image , and a search engine that crawls images from social media , indexes them with a neural network , and retrieves the relevant images to B . Our threat model considers an adversary ( M ) that can download A ’ s image from social media , perturb it maliciously , and re-upload the perturbed image on the web , where the search engine may crawl and index images from . The classifier on the search engine will wrongly index the perturbed image , causing the search engine to malfunction . Suppose an African-American uploader ( A ) posts a photo of him or herself on social media , and a racist adversary ( M ) perturbs it to be misclassified as “ gorilla ” by the search engine . When another person ( B ) searches “ gorilla ” on Google , the perturbed image would appear , though the image content shows a photo of A . This attack fools both A and B since the perturbed image is used contrary to A ’ s purpose and is not the image B wanted . To prevent this , the social media company , cooperating with the search engine company , could ask if A agrees that the images are slightly changed when uploaded to make them robust to such attacks . The purpose of the modification process , corresponding to the “ safe spot filter ” in Figure 1 , will be to ensure that the uploaded images are used under A ’ s intention and provide more accurate search results to B . We develop a novel optimization problem for searching safe spots in the vicinity of natural images and observe that over 90 % of the correctly classified images have safe spots nearby for adversarially trained models on both CIFAR-10 ( Krizhevsky & Hinton , 2009 ) and ImageNet ( Russakovsky et al. , 2015 ) . We also find that safe spots can enhance both empirical and certified robustness when applied on smoothed classifiers . Furthermore , we propose a novel safe spot inducing model training scheme to improve the preemptive robustness . By exploiting these safe spot-aware classifiers along with our safe spot search method , we also propose a new algorithm for out-of-distribution detection , which is often addressed together with robustness ( Hendrycks et al. , 2019a ; c ) . Our algorithm outperforms other baselines on near-distribution outlier datasets such as CIFAR-100 ( Krizhevsky & Hinton , 2009 ) . 2 RELATED WORKS . Adversarial training Goodfellow et al . ( 2015 ) first show that the robustness of a neural network can be enhanced by generating adversarial examples and including them in training set . PGD adversarial training improves the robustness against stronger adversarial attacks by augmenting training data with multi-step PGD adversarial examples ( Madry et al. , 2017 ) . Some recent works report performance gains over PGD adversarial training by modifying the adversarial example generation procedure ( Qin et al. , 2019 ; Zhang & Wang , 2019 ; Zhang et al. , 2020 ) . However , most of the recent algorithmic improvements can be matched by simply using early stopping with PGD adversarial training ( Rice et al. , 2020 ; Croce & Hein , 2020 ) . Other line of works achieve performance gains by utilizing additional datasets ( Carmon et al. , 2019 ; Wang et al. , 2020 ; Hendrycks et al. , 2019a ) . Randomized smoothing Injecting random noise during the forward pass can smooth the classifier ’ s decision boundary and improve empirical robustness ( Liu et al. , 2018 ) . Using differential privacy , Lecuyer et al . ( 2019 ) give theoretical guarantees for ` 1 and ` 2 robustness of classifiers smoothed with Gaussian and Laplacian noise . Cohen et al . ( 2019 ) provide a tight bound of ` 2 robustness of networks smoothed with Gaussian noise via the Neyman-Pearson lemma . Another proof of the robustness bound was given in Salman et al . ( 2019 ) using Lipschitz property of smoothed classifiers , where they also propose a new adversarial training scheme for building robust smoothed classifiers . Out-of-Distribution detection with deep networks Although deep networks achieve high performance on various classification tasks , they also tend to yield high confidence in out-of-distribution samples ( Nguyen et al. , 2015 ) . To filter out the anomalous examples , Hendrycks & Gimpel ( 2017 ) use the maximum value of a classifier ’ s softmax distribution as a score function , while Lee et al . ( 2018 ) propose Mahalanobis distance-based metric which spots out-of-distribution samples using hidden features . Hendrycks et al . ( 2019b ) show that leveraging auxiliary datasets disjoint from test-time data can improve the detection performance . Recently , Sastry & Oore ( 2020 ) characterize activity patterns of hidden features by Gram matrices and use the matrix values to identify anomalies . 3 METHODS . 3.1 GENERAL DEFINITION OF SAFE SPOT AND PREEMPTIVE ROBUSTNESS . We first establish a formal definition of safe spot and preemptive robustness . Let c : X → Y be a classifier which maps images to class labels . We define the safe region of the classifier c as the set of images that c can output robust predictions in the presence of slight adversarial perturbations . Definition 1 ( -safe region ) . Let c : X → Y be a classifier and ∈ R+ be the perturbation budget of an adversary . The -safe region of the classifier c is defined by S ( c ) : = { x ∈ X | c ( x′ ) = c ( x ) , ∀x′ ∈ B ( x ) } . Here , B ( x ) = { x′ ∈ X | d ( x , x′ ) ≤ } is the x-centered -ball . In this paper , we assume ` p threat model , i.e. , d ( x , x′ ) = ‖x − x′‖p , which is the most common setting on adversarial robustness literature , and consider p ∈ { 2 , ∞ } . Now , suppose a defender can preemptively manipulate a natural image xo under a small modification budget , knowing its ground-truth label yo . We denote the modified output image as xs . Then , the defender ’ s objective is to make xs be correctly classified as yo and locate in the safe region S ( c ) to improve the robustness against adversarial attacks . If xs satisfies these two conditions , then we say xo is preemptively robust and xs is a safe spot of xo . Definition 2 ( Preemptive robustness ) . Let c : X → Y be a classifier and δ , ∈ R+ be the modification budgets of the defender and the adversary , respectively . A natural image xo with its ground-truth label yo is called ( δ , ) -preemptively robust on the classifier c if there exists a safe spot xs ∈ Bδ ( xo ) such that ( i ) c ( xs ) = yo and ( ii ) xs ∈ S ( c ) . 3.2 SAFE SPOT SEARCH ALGORITHM . In this subsection , we develop an algorithm for searching a safe spot with a natural image . Given a classifier c , finding a safe spot xs from a natural image xo can be formulated as the following problem , which is directly from the definition of safe spot : minimize xs 1c ( xs ) 6=yo + 1xs /∈S ( c ) subject to ‖xs − xo‖p ≤ δ , where 1 is the 0-1 loss function . Note in this formulation the defender requires the ground-truth label yo for the safe spot search . However , images in the real-world ( e.g. , social media ) are usually unlabeled , unless uploaders annotate labels to their images by hand . So , it is natural to assume that the defender can not access the ground-truth label yo . In this case , we utilize the classifier ’ s prediction c ( xo ) instead of yo : minimize xs 1c ( xs ) 6=c ( xo ) + 1xs /∈S ( c ) subject to ‖xs − xo‖p ≤ δ . As xs /∈ S ( c ) implies there exists an adversarial example xa ∈ B ( xs ) such that c ( xa ) 6= c ( xs ) , we can reformulate the optimization problem as minimize xs 1c ( xs ) 6=c ( xo ) + sup xa 1c ( xa ) 6=c ( xs ) subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ . Since the 0-1 loss is not differentiable , we employ the cross-entropy loss ` : X × Y → R+ of the classifier c as the convex surrogate loss function : minimize xs ` ( xs , c ( xo ) ) + sup xa ` ( xa , c ( xs ) ) ( 1 ) subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ . Let h ( xs ) denote the objective in Equation ( 1 ) . Instead of minimizing h ( xs ) , we minimize h̃ ( xs ) = supxa ` ( xa , c ( xo ) ) , since it upper bounds h ( xs ) when sufficiently minimized by Lemma 1 . Algorithm 1 Finding a safe spot input An image and its prediction ( xo , c ( xo ) ) , the cross-entropy function ` xs = xo for i = 1 , . . . , MAXITER do Generate N adversarial examples for n = 1 , . . . , N do x ( 0 ) a , n = xs + ηn where ηn ∼ U ( B ( 0 ) ) for t = 1 , . . . , T do x ( t ) a , n = Πxs , ( f ( x ( t−1 ) a , n ; c ( xo ) , ` ) ) end for end for xs ← Πxo , δ ( xs − β · 1 N N∑ n=1 ∂ ` ( x ( T ) a , n , c ( xo ) ) ∂xs ) end for output xs Lemma 1 . If h̃ ( xs ) ≤ − log ( 0.5 ) ' 0.6931 , then h ( xs ) ≤ 2h̃ ( xs ) . Proof . See Supplementary A.1 . Finally , we have the following optimization problem : minimize xs sup xa ` ( xa , c ( xo ) ) ( 2 ) subject to ‖xs − xo‖p ≤ δ and ‖xa − xs‖p ≤ . To solve Equation ( 2 ) , we first approximate the inner maximization problem by running T -step PGD ( Madry et al. , 2017 ) whose dynamics is given by x ( 0 ) a = xs + η ( random start ) x̃ ( t ) a = f ( x ( t−1 ) a ; c ( xo ) , ` ) ( adversarial update ) x ( t ) a = Πxs , ( x̃ ( t ) a ) , ( projection ) where η is a random noise uniformly sampled from the ` p zero-centered -ball , f is FGSM ( Goodfellow et al. , 2015 ) defined by f ( x ; y , ` ) = x+ α · sgn ( ∇x ` ( x , y ) ) if p =∞ x+ α · ∇x ` ( x , y ) ‖∇x ` ( x , y ) ‖2 if p = 2 , and Πxs , is a projection operation to B ( xs ) . Then , we iteratively solve the approximate problem given by replacing xa to x ( T ) a in Equation ( 2 ) . To update xs , we need to compute the gradient of ` ( x ( T ) a , c ( xo ) ) with respect to xs expressed as ∂ ` ( x ( T ) a , c ( xo ) ) ∂xs = ∂f̃ ( x ( 0 ) a ) ∂x ᵀ · · · ∂f̃ ( x ( T−1 ) a ) ∂x ᵀ · ∇x ` ( x ( T ) a , c ( xo ) ) , where ∂f̃/∂x is the Jacobian matrix of f̃ = Πxs , ◦ f which is easily computed via back-propagation . After computing the gradient , we update xs by projected gradient descent method : x ( i+1 ) s = Πxo , δ ( x ( i ) s − β · ∂ ` ( x ( T ) a , c ( xo ) ) ∂xs ) . Note that the loss ` ( x ( T ) a , c ( xo ) ) is now a random variable dependent on η . Therefore , we generate N adversarial examples { x ( T ) a , n } Nn=1 with different noises and optimize the sample mean of the losses instead . Algorithm 1 shows the overall safe spot search algorithm and Figure 2 illustrates our optimization process .

Summary: This paper proposes a new adversarial framework where the defender could preemptively modify classifier inputs to find safe spots that are robust to adversarial attacks. They then introduce a novel bi-level optimization algorithm that can find safe spots on over 90% of the correctly classified images for adversarially trained classifiers on CIFAR-10 and ImageNet datasets and show that they can be used to improve both the empirical and certified robustness on smoothed classifiers. Besides, they propose a new training scheme based on their conjecture about safe spots for out-of-distribution detection which achieves state-of-the-art results on near-distribution outliers.