What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space

1 INTRODUCTION . Deep neural networks ( DNNs ) are a family of powerful models that have demonstrated superior learning capabilities in a wide range of applications such as image classification , object detection and natural language processing . However , DNNs are often applied as a black box with limited understanding of what the model has learned from the data . Existing understandings about DNNs have mostly been developed in the deep representation space or using the attention map . DNNs are known to be able to learn high quality representations ( Donahue et al. , 2014 ) , and the representations are well associated with the attention map of the model on the inputs ( Zhou et al. , 2016 ; Selvaraju et al. , 2016 ) . It has also been found that DNNs trained on high resolution images like ImageNet are biased towards texture ( Geirhos et al. , 2019 ) . While these works have significantly contributed to the understanding of DNNs , a method that can intuitively visualize what DNNs learn for each class in the input space ( rather than the deep representation space ) is still missing . Recently , the above understandings have been challenged by the vulnerabilities of DNNs to backdoor ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) and adversarial attacks ( Gu et al. , 2017 ; Chen et al. , 2017 ) . The backdoor vulnerability is believed to be caused by the preference of learning high frequency patterns ( Chen et al. , 2017 ; Liu et al. , 2020 ; Wang et al. , 2020 ) . Nevertheless , no existing method is able to reliably reveal the backdoor patterns , even though it has been well learned into the backdoored model . Adversarial attacks can easily fool state-of-the-art DNNs by either sample-wise ( Goodfellow et al. , 2016 ) or universal ( Moosavi-Dezfooli et al. , 2017 ) adversarial perturbations . One recent explanation for the adversarial vulnerability is that , besides robust features , DNNs also learn useful ( to the prediction ) yet non-robust features which are sensitive to small perturbations ( Ilyas et al. , 2019 ) . Adversarial training , one state-of-the-art adversarial defense method , has been shown can train DNNs to learn sample-wise robust features ( Madry et al. , 2018 ; Ilyas et al. , 2019 ) . However , it is still not clear if adversarially trained DNNs can learn a robust pattern for each class . In this paper , we focus on image classification tasks and propose a visualization method that can reveal the pattern learned by DNNs for each class in the input space . Different from sample-wise visualization methods like attention maps , we aim to reveal the knowledge ( or pattern ) learned by DNNs for each class . Moreover , we reveal these patterns in the input space rather than the deep representation space . This is because input space patterns are arguably much easier to interpret . Furthermore , we are interested in a visualization method that can provide new insights into the backdoor and adversarial vulnerabilities of DNNs , both of which are input space vulnerabilities ( Szegedy et al. , 2014 ; Ma et al. , 2018 ) . Given a target class , a canvas image , and a subset of images from the nontarget classes , our method searches for a single pattern ( a set of pixels ) from the canvas image that is highly predictive of the target class . In other words , when the pattern is attached to images from any other ( i.e . nontarget ) classes , the model will consistently predict them as the target class . Figure 1 illustrates a few examples of the class-wise patterns revealed by our method for DNNs trained on natural ( clean ) CIFAR-10 ( Krizhevsky , 2009 ) and ImageNet ( Deng et al. , 2009 ) datasets . In summary , our main contributions are : 1 ) We propose a visualization method to reveal the classwise patterns learned by DNNs in the input space , and show the difference to attention maps and universal adversarial perturbations . 2 ) With the proposed visualization method , we show that DNNs trained on natural datasets can learn a consistent and predictive pattern for each class , and the pattern contains abstract shapes along with some texture . This sheds new lights on the current texture bias understanding of DNNs . 3 ) When applied on backdoored DNNs , our method can reveal the trigger patterns learned by the model from the poisoned dataset . Our method can serve as an effective tool to assist the detection of backdoored models . 4 ) The existence of class-wise predictive patterns in the input space indicates that even DNNs trained on clean data can have backdoors , and the class-wise patterns identified by our method can be readily applied to “ backdoor ” attack the model . 5 ) By examining the patterns learned by DNNs trained in the adversarially setting , we find that adversarially trained models learn more simplified shape patterns . 2 RELATED WORK . General Understandings of DNNs . DNNs are known to learn more complex and higher quality representations than traditional models . Features learned at intermediate layers of AlexNet have been found to contain both simple patterns like lines and corners and high level shapes ( Donahue et al. , 2014 ) . These features have been found crucial for the superior performance of DNNs ( He et al. , 2015 ) . The exceptional representation learning capability of DNNs has also been found related to structures of the networks like depth and width ( Safran & Shamir , 2017 ; Telgarsky , 2016 ) . One recent work found that ImageNet-trained DNNs are biased towards texture features ( Geirhos et al. , 2019 ) . Attention maps have also been used to develop better understandings of the decisions made by DNNs on a given input ( Simonyan et al. , 2014 ; Springenberg et al. , 2015 ; Zeiler & Fergus , 2014 ; Gan et al. , 2015 ) . The Grad-CAM technique proposed by Selvaraju et al . ( 2016 ) utilizes input gradients to produce intuitive attention maps . Whilst these works mostly focus on deep representations or sample-wise attention , an understanding and visualization of what DNNs learn for each class in the input space is still missing from the current literature . Understanding Vulnerabilities of DNNs . Recent works have found that DNNs are vulnerable to backdoor and adversarial attacks . A backdoor attack implants a backdoor trigger into a victim model by injecting the trigger into a small proportion of training data ( Gu et al. , 2017 ; Liu et al. , 2018 ) . The model trained on poisoned dataset will learn a noticeable correlation between the trigger and a target label . A backdoored model behaves normally on clean test data , yet consistently predict a target ( incorrect ) label whenever the trigger appears in a test example ( Zhao et al. , 2020 ; Yao et al. , 2019 ; Liu et al. , 2020 ) . This is believed to be caused by the fact that DNNs tend to learn more high frequency ( e.g . backdoor ) patterns ( Chen et al. , 2017 ; Liu et al. , 2020 ; Wang et al. , 2020 ) . However , it is still unclear whether DNNs can learn such patterns from natural ( clean ) data . Moreover , despite a few attempts ( Wang et al. , 2019 ; Qiao et al. , 2019 ) , the trigger pattern still can not be reliably revealed , even though it has been well learned by the backdoored model . DNNs can also be easily fooled by small , imperceptible adversarial perturbations into making incorrect predictions ( Szegedy et al. , 2014 ; Goodfellow et al. , 2016 ) . Adversarial perturbations can be either sample-wise ( Madry et al. , 2018 ) or universal ( Moosavi-Dezfooli et al. , 2017 ) . This has been found to be caused by learning useful ( to prediction ) but nonrobust ( to adversarial perturbation ) features ( Ilyas et al. , 2019 ) . Meanwhile , adversarial training has been shown to learn more robust features and deliver effective defenses ( Madry et al. , 2018 ) . However , existing understandings of adversarial training are established based on sample-wise attention ( Ilyas et al. , 2019 ) . It still unclear , from the class-wise perspective , what robust or nonrobust input patterns look like . In this paper , we will propose a method to reveal the patterns ( e.g . backdoor or adversarially robust/nonrobust ) learned by DNNs for each class . 3 PROPOSED VISUALIZATION METHOD . In this section , we first define the input space class-wise pattern searching problem , then introduce our proposed searching method . Motivation and Intuition . We focus on image classification with deep neural networks . We denote the training and test dataset as Dtrain and Dtest , respectively . Given a DNN model f trained on a K-class Dtrain and a target class y ∈ { 1 , · · · , K } , our goal is to find an input space pattern , i.e , a small set of pixels , that are extremely predictive of the target class . A highly predictive pattern of a class can largely capture the knowledge the model learned for the class . In backdoor attack , a predictive ( i.e . backdoor trigger ) pattern learned by the model can even control the model ’ s prediction . Intuitively , a predictive pattern of a target class should be able to make the model consistently predict the target class whenever it is attached to images from any other ( e.g . nontarget ) classes . Class-wise Pattern Searching . For a target class y , our method searches for a predictive pattern py from a canvas image xc , based on a small test subset Dn of images from the nontarget classes ( i.e . Dn ⊂ Dtest ) . The canvas image xc is the image where the pattern ( a set of pixels ) is extracted . The search is done via an optimization process based on a mixed input between the canvas image xc and an image xn ∈ Dn . The mixed input x̃ is defined as follows : x̃ =m ∗ xc + ( 1−m ) ∗ xn , ( 1 ) where m is a mask that has the same size as either xc or xn , and mij ≥ 0 . The mixed input image is labeled as the target class y regardless of its original class . This mixing strategy is reminiscent of the mixup ( Zhang et al. , 2018 ) data augmentation algorithm . However , we do not mix the class labels and our purpose is for pattern optimization rather than data augmentation . During the searching process , the mask is iteratively updated to minimize the following loss : L = − log fy ( x̃ ) + α 1 n ‖m‖1 , ( 2 ) where , fy is network ’ s probability output with respect to target class y , ‖·‖1 is the L1 norm , α is a parameter that balances the two loss terms , and n is the size of the input image as well as the mask . The first loss term is the commonly used cross entropy loss . The second term increases the sparsity of mask as we are interested in simple patterns with a small number of highly predictive pixels . During the search process , we pair the canvas image xc randomly with images from Dn , and iteratively update the mask m using standard Stochastic Gradient Decent ( SGD ) while keeping the model parameters unchanged . At each iteration , the mask m will also be clipped into [ 0 , 1 ] . Once a mask is learned , we further clip the values in the mask that are smaller than γ to zero , larger than γ to one . We denote this clipped mask by mγ . We then extract the pattern from the canvas image by py =mγ ∗ xc . The γ parameter can be flexibly determined in different applications . A large γ may lead to less predictive pattern while a small γ will produce more of a sample-wise pattern that overfits to the canvas image . The above search method is repeatedly applied to N canvas images to generate N patterns for each class . We then select the pattern that has the lowest loss value as the final pattern of the class . This additional step is to find the most predictive pattern by exploring different canvases . The complete procedure of our method is described in Algorithm 1 in Appendix A. Canvas Sampling . We propose four different sampling strategies for the selection of the N canvas images : positive sampling , negative sampling , random sampling and white canvas . Positive sampling selects the top-N confident images from the target class according to the logits output of model f . Negative sampling selects the top-N most confidently misclassified images from any nontarget class into the target class . The random sampling randomly chooses N images from the target class y . The white canvas simply uses an image with all white pixels as the canvas . Both positive and the negative sampling aim to find the most well-learned examples by the model , but from different perspectives : well-learned correctly ( e.g . positive ) vs. well-learned incorrectly ( e.g . negative ) . The white canvas is interesting since the pattern found from the white canvas will have the texture “ removed ” , which is useful for scenarios where only the shape features are of interest . The patterns found based on different canvases are compared in Figure 4 . After applying our method on each class , we can obtain a set of class-wise patterns : P = { p1 , · · · , pK } . This set of predictive patterns can revel the knowledge learned by model f for each class from a unique perspective . Why is it Class-wise ? At first sight , one might wonder if the discovered pattern could be samplewise , rather than class-wise , given the use of the canvas sample . Note that , however , even though we are using a single sample as a canvas , the pattern found by the optimization algorithm is dependent on how the model has learnt the entire class , in terms of its loss . This is particularly evident in the case of the all white canvas , which bears no relation to any input sample . Hence our designation of the pattern as being “ class-wise ” . While our method can find consistent and predictive classwise patterns in the experiments , it might still be extendable . For example , using multiple positive canvas images at the same time , using noise rather than the non-target images , or using universal adversarial perturbation ( UAP ) ( Moosavi-Dezfooli et al. , 2017 ) but in a more controlled manner . We leave further explorations of these methods as our future work . Difference to Universal Adversarial Perturbation . UAP can also be applied to craft class-wise adversarial patterns that can make the model predict an adversarial target class . In this view , both UAP and our method find predictive patterns to the target class . However , the two methods work in different ways . By fooling the network , UAP explores the unlearned space ( low-probability “ pockets ” ) of the network ( Szegedy et al. , 2014 ; Ma et al. , 2018 ) . In contrast , our method is a searching ( rather than perturbing ) method that does not rely on adversarial perturbations . Thus , it has to find the optimal pixel locations in the input space that are well-learned by the model for the pattern to be predictive of the class . In Section 4.2 and Appendix E , we have experiments showing the difference of the patterns found by class-wise UAP and our method .

Summary: The papers proposes a simple method for visualizing the patterns learned by deep neural networks in the supervised classification setting. Informally, suppose you have an image x that is "representative" of the class y and let X be a set of images that belong to other classes. The authors propose an optimization problem that looks for a mask (i.e. set of pixels) along with values of those pixels such that when this pattern is added to any image in X, the model will predict the new image to have the label y. This optimization problem can be solved using iterative thresholding and one may control the level of sparsity as the authors studied. Despite its simplicity, it can reveal clear patterns, particularly on high resolution images, such as ImageNet. The authors, then, show how this method can be used to interpret neural networks, detect backdoor attacks during training, and verify robustness.