What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space
Authors: Shihao Zhao
Xingjun Ma
Yisen Wang
James Bailey
Bo Li
Yu-Gang Jiang
Venue:  (2021)
Keywords: deep neural networks
deep learning understanding
backdoor vulnerability
adversarial vulnerability

Abstract: Deep neural networks (DNNs) have been widely adopted in different applications to achieve state-of-the-art performance. However, they are often applied as a black box with limited understanding of what the model has learned from the data. In this paper, we focus on image classification and propose a method to visualize and understand the class-wise patterns learned by DNNs trained under three different settings including natural, backdoored and adversarial. Different from existing class-wise deep representation visualizations, our method searches for a single predictive pattern in the input (i.e. pixel) space for each class. Based on the proposed method, we show that DNNs trained on natural (clean) data learn abstract shapes along with some texture, and backdoored models learn a small but highly predictive pattern for the backdoor target class. Interestingly, the existence of class-wise predictive patterns in the input space indicates that even DNNs trained on clean data can have backdoors, and the class-wise patterns identified by our method can be readily applied to “backdoor” attack the model. In the adversarial setting, we show that adversarially trained models learn more simplified shape patterns. Our method can serve as a useful tool to better understand DNNs trained on different datasets under different settings.

Under review as a conference paper at ICLR 2021
WHAT Do DEEP NETS LEARN? CLASS-WISE PAT- TERNS REVEALED IN THE INPUT SPACE
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks (DNNs) have been widely adopted in different applications to achieve sfate-of-the-ar performance. However, they are often applied as black box with limited understanding of what the model has learned from the data. In this paper, we focus on image classification and propose method to visualize and understand the class-wise patterns learned by DNNs trained under three different settings including natural, backdoored and adversarial. Different from existing class-wise deep representation visualizations, our method searches for a single predictive pattern in the input (i.e. pixel) space for each class. Based on the pro- posed method, we show that DNNs trained on natural (clean) data learn abstract shapes along with some texture, and backdoored models learn a small but highly predictive pattern for the backdoor target class. Interestingly, the existence of class-wise predictive patterns in the input space indicates that even DNNs trained on clean data can have backdoors, and the class-wise patterns identified by our method can be readily applied to "backdoor" attack the model. In the adversar- ial setting, we show that adversarially trained models learn more simplified shape patterns. Our method can serve as a useful tool to better understand DNNs trained on different datasets under different settings.
INTRODUCTION
Deep neural networks (DNNs) are family of powerful models that have demonstrated superior learning capabilities in a wide range of applications such as image classification, object detection and natural language processing. However, DNNs are often applied as black box with limited understanding of what the model has learned from the data. Existing understandings about DNNs have mostly been developed in the deep representation space or using the attention map. DNNs are known to be able to learn high quality representations (Donahue et al. 2014), and the representations are well associated with the attention map of the model on the inputs (Zhou et al. 2016 Selvaraju let al. 2016). It has also been found that DNNs trained on high resolution images like ImageNet are biased towards texture (Geirhos et al. 2019). While these works have significantly contributed to the understanding of DNNs, a method that can intuitively visualize what DNNs learn for each class in the input space (rather than the deep representation space) is still missing.
Recently, the above understandings have been challenged by the vuinerabilities of DNNs to back- door (Szegedy et al.] 2014] Goodfellow et al. 2015) and adversarial attacks (Gu et al. 2017 Chen et al. 2017). The backdoor vulnerability is beviee to be caused by the preference of learning high trequency patterns (Chen et al. 2017 Liu et al. 2020 Wang et al. (020).. Nevertheless, no existing method is able to reliably reveal the backdoor patterns, even though has been well learned into the backdoored model. Adversarial attacks can easily fool statc-of-thh the-art art DNNs by either sample-wise (Goodfellow et al.) 2016) or universal Aloossvi--ezzoou et al. 2017) adversarial perturbations. One recent explanation for the adversarial vulnerability is that, besides robust features, DNNs also learn useful (to the prediction) yet non-robust features which are sensitive to small perturbations (Ilyas et al., 2019). Adversarial training, one state-of-the-art adversarial defense method, has been shown can train DNNs to learn sample-wise robust features (Madry et al., [0188] Ilyas et al. 2019). However, is still not clear if adversarially trained DNNs can learn a robust pattern for each class.
In this paper, we focus on image classification tasks and propose a visualization method that can reveal the pattern learned by DNNs for each class in the input space. Different from sample-wist visualization methods like attention maps, we aim to reveal the knowledge (or pattern) learned by DNNs for each class. Moreover, we reveal these patterns in the input space rather than the deep representation space. This is because input space patterns are arguably much easier to interpret. Furthermore, are interested in a visualization method that can provide new insights into the

---

Under review as a conference paper at ICLR 2021
"airplane"
"deer"
"dog"
"acoustic guitar"
"tabby"
"padlock"
CIFAR-10
ImageNet
Figure I: Example images (top row) and the class-wise patterns (bottom row) learned by a 8 on CIFAR-10 10 (left three columns) and ResNet-50 on ImageNet (right three columns) and revealed by our method. The ImageNet classes are "n02676566" ("guitar"), "n02123045"("cat"" and "n03874599" ("padlock"). The pattern size is set 5% of the image size.
backdoor and adversarial vulnerabilities of DNNs, both of which are input space vulnerabilities (Szegedy et al. 2014 Ma et al. 2018).
Given a target class, a canvas image, and subset of images from the nontarget classes, our method searches for single pattern (a set of pixels) from the canvas image that is highly predictive of the target class. In other words, when the pattern attached to images from any other (i.e. nontar- get) classes, the model will consistently predict them as the target class. Figure Iillustrates a few examples of the class-wise patterns revealed by our method for DNNs trained on natural (clean) CIFAR-10 (Krizhevsky) 2009) and ImageNet (Deng al. 2009) datasets.
In summary, our main contributions are: 1) We propose visualization method to reveal the class- wise patterns learned by DNNs in the input space, and show the difference to attention maps and universal adversarial perturbations. 2) With the proposed visualization method, we show that DNNs trained on natural datasets can learn consistent and predictive pattern for each class, and the pattern contains abstract shapes along with some texture. This sheds new lights on the current texture bias understanding of DNNs. 3) When applied on backdoored DNNs, our method can reveal the trigger patterns learned by the model from the poisoned dataset. Our method serve an effective tool to assist the detection of backdoored models. 4) The existence of class-wise predictive patterns in the input space indicates that even DNNs trained on clean data can have backdoors, and the class-wis patterns identified by our method can be readily applied "backdoor" attack the model. 5) By examining the patterns learned by DNNs trained in the adversarially setting, we find that adversarially trained models learn more simplified shape patterns.
RELATED WORK
General Understandings of DNNs. DNNs are known to learn more complex and higher quality representations than traditional models. Features learned at intermediate layers of AlexNet have been found to contain both simple patterns like lines and corners and high level shapes (Donahuel al. 2014). These features have been found crucial for the superior performance of DNNs (He et al. 2015). The exceptional representation learning capability of DNNs has also been found re- lated to structures of the networks like depth and width (Safran & Shamir 2017 Telgarsky| 2016) One recent work found that ImageNet-traiie trained DNNs are biased towards texture Teatures (Geirhos ctt al. 2019). Attention maps have also been used to develop better understandings of the decisions made by DNNs on a given input (Simonyan et al. 2014 (Springenberg et al.) 2015 [015]/eiier [Zeiler & Fergus, 2014 Can et al.) 2015). The Grad-CAM technique proposed by Selvaraju al. (2016) utilizes input gradients to produce intuitive attention maps. Whilst these works mostly focus on deep represen- tations or sample-wiss attention, an understanding and visualization of what DNNs learn for each class in the input space is still missing from the current literature.
Understanding Vulnerabilities of DNNs. Recent works have found that DNNs are vulnerable to backdoor and adversarial attacks. A backdoor attack implants a backdoor trigger into a victim model by injecting the trigger into small proportion of training data (Gu et al. 2017 Liu et al.) 2018). The model trained on poisoned dataset will leamn noticeable correlation between the trigger and
2

---

Under review as a conference paper at ICLR 2021
target label. A backdoored model behaves normally clean test data, yet consistently predict a target (incorrect) label whenever the trigger appears in a test example (Zhao et al.) 2020 Yao jet al. 2019; Liu et al.] 2020). This is believed to be caused by the fact that DNNs tend to learn more high frequency ge.g. backdoor) patterns (Chen et al.] 2017 Liu et al., 2020: Wang et al.) 2020). However, it is still unclear whether DNNS can learn such patterns from natural (clean) data. Moreover, despite a few attempls (Wang et al. 2019 Qiao et al. 2019), the trigger pattern still can not be reliably revealed, even though it has been well learned by the backdoored model. DNNs can also be easily fooled by small, imperceptible adversarial perturbations into making incorrect predictions (Szegedy et al. 2014; Goodffllow et al. 2016). Adversarial perturbations can be either sample-wise (Madry et al. 2018) or universal (Moosavi-Dezfooll et al. 2017). This has been found to be caused by learning useful (to prediction) but nonrobust (to adversarial perturbation) features (llyas et al. 2019). Meanwhile, adversarial training has been shown to learn more robust features and deliver ellective defenses (Madry et al. 2018). However, existing understandings of adversarial training are established based on sample-we wise attention (Ilyas et al., 2019). It still unclear, from the class-wise perspective, what robust or nonrobust input patterns look like. In this paper, we will propose a method to reveal the patterns (e.g. backdoor or adversarially robust/nonrobust) learned by DNNs for each class.
paper
PROPOSED VISUALIZATION METHOD
In this section, we first define the input space class-wise pattern searching problem, then introduce our proposed searching method.
Motivation and Intuition. We focus on image classification with deep neural networks. We denote the training and test dataset as Dtrain and Drest: respectively. Given DNN model f trained on K-class Dtrain and a target class y {1,... K). our goal is find an input space pattern, i.e, a small set of pixels, that are extremely predictive of the targer class. A highly predictive pattern of a class can largely capture the knowledge the model learned for the class. In backdoor attack, a predictive (i.e. backdoor trigger) pattern learned by the model can even control the model's 's predic- tion. Intuitively, a predictive pattern of a target class should be able make the model consistently predict the target class whenever it is attached to images from any other (e.g. nontarget) classes.
Class-wise Pattern Searching. For a target class y, our method searches for a predictive pattern Py from canvas image Bey based on small test subset D., of images from the nontarget classes (i.e. Dn C Drest). The canvas image V. is the image where the pattern (a set of pixels) is extracted. The search is done via an optimization process based on a mixed input between the canvas image a and an image xn E D,, The mixed input æ is defined as follows:
x = m x Te + (1 - (1-m)= * Tn.
where m is a mask that has the same size as either I. or nn, and m.j 2 0. The mixed input image labeled as the target class y regardless of its original class. This mixing strategy is reminiscent of the mixup (Zhang et al., 2018) data augmentation algorithm. However, we do not mix the class labels and our purpose is for pattern optimization rather than data augmentation.
During the searching process, the mask is iteratively updated to minimize the following loss:
C - log fu (x) + a- 72
||m]]]
where, fy is network's 's probability output with respect to target class y. ||-1|, is the L1 norm, a is a parameter that balances the two loss terms, and n is the size of the input image as well as the mask. The first loss term is the commonly used cross entropy loss. The second term increases the sparsity of mask as we are interested in simple patterns with small number of highly predictive pixels.
During the search process, we pair the canvas image x, randomly with images from Da, and it- eratively update the mask m using standard Stochastic Gradient Decent (SGD) while keeping the model parameters unchanged. At each iteration, the mask m will also be clipped into (0, 1). Once a mask learned, we further clip the values the mask that are smaller than Y to zero, larger than to one. We denote this clipped mask by m,. We then extract the pattern from the canvas image by Py m. Ze. The Y parameter can be flexibly determined in different applications. A large ? may lead less predictive pattern while a small y will produce more of sample-wiss pattern that overfits to the canvas image.
The above search method is repeatedly applied to N canvas images to generate N patterns for each class. We then select the pattern that has the lowest loss value as the final pattern of the class. This
3

---

Under review as a conference paper at ICLR 2021
additional step is to find the most predictive pattern by exploring different canvases. The complete Appendix A
procedure of our method is described in Algorithm I
Canvas Sampling. We propose four different sampling strategies for the selection of the N canvas images: positive sampling, negative sampling, random sampling and white canvas.
Positive sampling selects the top-N confident images from the target class according to the logits output of model f. Negative sampling selects the top-N most confidently misclassified images from any nontarget class into the target class. The random sampling randomly chooses N images from the target class y. The white canvas simply uses an image with all white pixels as the canvas.
Both positive and the negative sampling aim find the most well-learned examples by the model, but from different perspectives: well-learned correctly (e.g. positive) vs. well-learned incorrectly (e.g. negative). The white canvas is interesting since the pattern found from the white canvas will have the texture "removed", which is useful for scenarios where only the shape features of interest. The patterns found based on different canvases are compared in Figure
4 Afte applying
our method on each class, we can obtain a set of class-wise patterns: P = {pr.... p This set of predictive patterns can revel the knowledge learned by model / for each class from a unique perspective.
Why is it Class-wise? At first sight, one might wonder if the discovered pattern could be sample- wise, rather than class-wise, given the use of the canvas sample. Note that, however, even though we using single sample
canvas, the pattern found by the optimization algorithm is dependent on how the model has learnt the entire class, in terms of its loss. This is particularly evident in the case of the all white canvas, which bears no relation to any input sample. Hence our designation of the pattern as being "class-wise". While our method can find consistent and predictive class- wise patterns in the experiments, it might still be extendable. For example. using multiple positive canvas images at the same time, using noise rather than the non-target images, or using universal adversarial perturbation (UAP) (Moosavi-Deztooll el al. (017) but in more controlled manner. We leave further explorations of these methods as future work.
Difference to Universal Adversarial Perturbation. UAP can also be applied to craft class-wise adversarial patterns that can make the model predict an adversarial target class. In this view, both UAP and our method find predictive patterns to the target class. However, the two methods work in different ways. By fooling the network, UAP explores the unlearned space (low-probability "pockets") of the network (Szegedy et al.] 2014; Ma et al. 2018). In contrast, our method is a searching (rather than perturbing) method that does not rely on adversarial perturbations. Thus, it has to find the optimal pixel locations in the input space that are well-learned by the model for the pattern be predictive of the class. In Section 4.2 and Appendix [E] we have experiments showing the difference of the patterns found by class-wise UAP and our method.
4 EXPERIMENTS
In this section, we will apply our method to reveal the class-wise patterns learned by DNNs trained on natural data, backdoored data, adversarial settings with intentionallly robust or nonro- bust datasets, and adversarially trained DNNs. We also conduct comparisons with attention maps and universal adversarial perturbations.
Experimental Setting, We consider ResNet-50 (He et al. 2016) on two benchmark image datasets: CIFAR-10 (Krizhevsky) (209) and ImageNet (Deng et al. 2009). The models are trained on the training set of the datasets using standard training strategies, except the adversarial experiments in (ection[4.4] We then apply our method to search for the class- wise patterns based on canvas images and nontarget-class images selected from the test sets of the datasets. Here, we set the nontarget- class subset to be 20% of the test images (more analysis of the nontarget-class subset size can be found in Appendix C.2). The sparsity regularization parameter a is slightly adjusted around 0.2 for different classes and datasets. The clipping parameter y is set to discover a pattern of a predefined size (e.g. 5% or 10% of the image size), which will be explicitly stated in each experiment. The searching process is run for 5 epochs with respect to the nontarget-class subset. All experiments run on GeForce GTX 1080Ti GPUs
Predictive Power. We define the predictive power of the pattern Py for target class as fol-
lows: PW- ACC(F(a, tp,),) where ACC(((x, + Pp). v) is the model's accuracy on nontarget ACC(M(E)).))
images FP E Dreet when attached with the target-class pattern Py with respect to label y, and (C))) y) is the model's original accuracy the target class images and xy E D D Intu-
4

---

Under review as a conference paper at ICLR 2021
"horse"
"bird"
"ballpoint"
miciPipa
"analog clock"
Figure Class-wise patterns (second column) learned by a ResNet-50 on natural CIFAR-10 (top two rows) and ImageNet (bottom two rows). Class names are shown in the first column, while nontarget-class images with the pattern attached and also being misclassified as the tar- get class are shown in columns 3-8. The ImageNet classes are "10278316l""bballoonn" and (002700993"("anatli" clock"). Pattern size is 5% of the image size.
itively, the PW metric reflects the degree to which the pattern can determine the model's prediction towards the target class, relative to the true target-class images. Note that, in most of our experi- ments, the pattern Py is searched on Dn, C 'Deest which obtained from 20% of the test set. And the PW metric is computed on the entire (i.e. 100%) test set. We have an experiments in Appendix [B.2 showing consistent results of PW on the 80% test set that has not been used for pattern searching.
4.1 CLASS-WISD PATTERNS LEARNED BY DNNS FROM NATURAL DATA
We first show the class-wise patterns learned by ResNet-50 on natural CIFAR-10 and ImageNet datasets. Here,
positive canvases sampled using the positive sampling (see Section 3 A
more detailed analysis on the impact of different positive canvases can be found in Appendix D.3
The patterns revealed at 5% image size for two CIFAR-I0 and ImageNet classes are illustrated in
Figure 2 More patterns for all 10 CIFAR-IO classes and 20 ImageNet classes are provided in Figure
9 and 10 Appendix C.I The predictive power of different sizes (0% 10%) of patterns are shown
in Figure The detailed predictive power results for each of the 10 CIFAR-10 classes and 50 randomly selected ImageNet classes can be found in Appendix B.
Patterns Revealed by Positive Canvas. As shown in Figure 2 the class-wise patterns learned from natural data contain shapes that are closely associated with the object class, although some of the shapes are quite abstract, especially for the CIFAR-10 classes. Interestingly. DNN can learn more than one shape for a given class, for example the "tabby" class in Figure O and "ballpoint" class in Figure 2 And the shapes can be found at different locations of the input space (e.g. "padlock" in Figure T. This phenomenon is more common on ImageNet, as shown in Figure TO This is because even the same class of objects may have different appearances and appear at different locations of the image. As result, the model learns be sensitive to more than one patterns at different locations of the input space. According to the mean predictive power shown Figure (left subfigure), the patterns revealed by our method represent roughly 60% of the predictive information learned by the model. And the predictive power transfers well across separately trained models, shown in the right subfigure Figure 7 This indicates that DNNs can learn capture some common shapes for each class, even though the training images in the same class might be quite diverse.
The abstract shapes are not the whole story here as there are also some textures (c.g. color) in the pattern. Shapes
class-wise features our method iteratively searches for the optimal positions of the pixels that are predictive to the entire class. By contrast, the textures are more sample-wise features carried over from the canvas image by the P, m. * Ee operation. The use of a white-canvas
remove sample-wiss textures while keeping only class-wise shapes (see Figure 4). In relation existing texture bias understanding of DNNs trained on ImageNet (Geirhos) ctt al. 2019), our result implics that shape also plays an important role in DNNs. Note that the LT regularization of our method does not restrict the pixels in the pattern to be texture or shape. The lack of texture in the patterns indicates that DNNs do learn to be more sensitive certain positions of the
5

---

Under review as a conference paper at ICLR 2021
Mean Prediction Power
CIFAR-10
Imagehist
12
Prodecte
%
Po
- Fantan
11h
720
Randor
Wib
0.
Negattn
000
25
25
0% 7%
15%
25%
35
Fottem S2zo
Pattem Size
Potter Steo
Figure 3: Left: mean predictive power of the class-wise patterns of different sizes, over all 10 CIFAR-10 classes and 50 randomly selected ImageNet classes. Middle: mean predictive power of the patterns found with different types of canvases on CIFAR-10. Right: mean predictive power of the patterns found with different types of canvases on ImageNet. The patterns are searched on ResNet-50 models trained on clean CIFAR-10 and ImageNet.
Positive
Negative sampling
Random sampling
White
Positive sampling
Negative sampling
Random sampling
White canvas
sampling
carvva
"ship"
"hook"
Figure 4: Patterns (second row) found by our method on different types of canvases (the first row) for CIFAR-10 "ship" class (left) and ImageNet "hook" ""oo3"((n05322672""" class (right).
input space. Note that, besides shape and texture, DNNs also learn a certain amount of background noise, especially on ImageNet.
Patterns Revealed by Different Types of Canvases. An example the patterns found by the
types of canvases are illustrated in Figure 4 More visualizations are provided in Figure 13 (CIFAR- 10) and Figure 14 (ImageNet) in Appendix D.2 As can be observed, the abstract shapes found on different canvases are all closely associated with the object class, although there are some variations in the details and locations. As shown Figure 14| for some of the canvases, the revealed pattern has a noticeable correlation with the canvas, especially for the positive canvas. For example, the first two patterns for the "analog clock" class (the second row Figure This implies that the patterns found by our method do carry sample-wiss features from the canvas image, though they are very predictive of the entire class. Note that the patterns revealed by the white canvas also contain clear shapes, though there are sample-wise features. More analysis of the impact of different canvas on the predictive power can be found in Appendix [D. and Appendix [D.3] While the mean (over all target classes) predictive powers are similar for patterns found on different canvases (Figure 12), the individual predictive power varies across different canvas images (Table 4 and Table 5). This is also why we suggest to use N (i.e. 55 different canvases to help find the most predictive patterns.
4.2 COMPARISON TO ATTENTION MAP AND UNIVERSAL ADVERSARIAL PERTURBATION
Comparison to Attention Map. Here, we compare the predictive power of the class-wiss patterns found by our method to the key areas identified by using the attention map. This is somewhat not a fair comparison since attention maps were designed for sample-wise explanations. However, we believe it is interesting to see if high attention areas are also predictive of the class. We use Grad- al. 2016) to extract the high attention
of the positively sampled canvas image for each of the 50 ImageNet target classes (class names are in Table 2]. The size of the high attention area is set to be 10% of the image. We then attach the extracted attention area to all non-target class test images to compute its predictive power (the same testing scheme as for our method). Figure 5 illustrates iillustatts few example attention areas (left grid) and the mean predictive power over the 50 classes (right plot). We find that the predictive power of the attention patterns are considerably low. This indicates that sample-wis wise attention maps may not be good representation of the plasswiiee knowledge learned by the model. As the attention visualization shown in Appendix
CAM (Selvaraju

---

Under review as a conference paper at ICLR 2021
Images with high atlention pallern
Input
Attention
ImageNet
1.0
Grad-Cam
Our
Moan Prodictive Power
"tench'
0.6
"Tibetan
0.2
terrier"
0.0
05
196
24
2%
8% Pattem Size
4%%
5%
7%
8%
9%
10%
Figure 5: Left grid: input image (first column), attention map (second column) and images (third column) attached with high attention patterns extracted from the input image. Right plot: the mean predictive power of high attention patterns over 50 ImageNet target classes. The attention pattern size is 10% of the image size.
H the patterns identified by our method indeed have noticeable correlation to the class and can consistently attract the network's ' attention when attached to different images.
Comparison to Universal Adversarial Perturbations. Here we visualize the patterns revealed by UAP, our method and one variant of our method with the total-variation (TV) regularization (Fong Vedaldi 2017). We apply UAP targeted attack in a class- wisse fashion, accumulating the perturbations separately for each target class. The class-wise UAP is crafted on the entire test set for 11FAR-10 and 10K randomly selected test images for ImageNet. The loss function used by the TV variant of our method is defined as follows:
L = - log ( + a- ||mm]| + 3- /
(3)
where is a parameter that controls the strength of the TV regularization (the third term). The reason we consider the TV regularization is that it can reduce the variation of the mask, producing more smooth patterns (Fong & Vedaldi 2017).
As shown in Figure 6 the UAP patterns generated on CIFAR-10 also contain structures of the object. However, the patterns found by our method are much cleaner, although there are still some background noises. The UAP patterns generated on ImageNet are notably more abstract than our methods. This is because, by fooling the network, UAP explores the unlearned space of DNNs. By contrast, our method is searching (rather than perturbing) method that does not rely on adversarial perturbations. Therefore, it needs to find the well-learned locations in the input space for the pattern to be predictive of the class. We believe UAP can be applied in more controlled manner to explore the well-learned (rather than unlearned) space of the network. We find that the TV regularization can indeed help produce smooth patterns with less background noise, however, the main patterns are generally the same as those found without using TV. More visualizations and predictive power analysis of UAP and our methods can be founded in Appendix B
4.3 CLASS-WISE PATTERNS LEARNED BY BACKDOORED DNNS
Here, we apply our method reveal the patterns learned by a backdoored ResNet-50 model by BadNets (Gu et al. 2017) on CIFAR-10. The model was trained without data augmentation (see Appendix | for the results with data augmentations). The patterns revealed on different canvases are illustrated in the left figure of Figure 7 Here, set the pattern size to be small (c.g. 1%) as here we are interested in examining whether the model has learned a small, extremely predictive but meaningless pattern. As can be observed, our method can reliably reveal the trigger pattern, especially on the white canvas. More visualizations with three types of backdoor triggers can be found in Figure 16 in Appendix F
We then test the predictive power of the recovered trigger patterns on the backdoored model. For a comparison, we also test the predictive power of the same size of pattern identified on clean model. Here, we apply BadNets to attack all CIFAR- 10 classes individually. and compute the mean predictive power over the 10 classes. As shown in the middle figure of Figure 7 the patterns are highly predictive on the backdoored model, yet the same size of pattern identified on natural has almost zero predictive power on the natural model itself. This confirms that the model indeed learns to memorize the trigger pattern. On the other hand, the existence of small but extremely predictive
7

---

Under review as a conference paper at ICLR 2021
Ours + TV (B=0.2)
Ours TV (B=1.0)
UAP (e-8)
UAP (c=16)
UAP (x=32)
Ours
"airplane"
"deer"
"tabby"
"analog clock"
Figure 6: A comparison of the class-wise patterns revealed naturally trained ResNet-50 models by universal adversarial perturbation (UAP), our method and our method with the total-variation (TV) regularization (defined Equation 3). For our methods, the pattern size is set to 5% of the image size, while UAP perturbs the entire (i.e. 100%) image. The top and bottom two rows show the patterns for two CIFAR-10 and ImageNet classes, respectively.
Recovered triggers
1.0
images
on fuker
Mean Predictive Power
0.8
Pusnsee on fartser
Imsggehie
Backdoored images
Trannfer Rele
0.6
0.4
0.2
0.0
11 2% 5%
10%
Figure 7: Left: Backdoor trigger patterns recovered by our method from a backdoored (by BadNets with target class "airplane") ResNet-50 model on CIFAR-I0 10 using different canvas images. The trig- ger pattern is black-white checkerboard at the bottom right corner the image. Middle: The mean (over all 10 CIFAR-10 backdoor target classes) predictive power of 1) the recovered trigger pattern on the backdoored model (Pbackdoor on )backdoor), and 2) the same size of natural pattern on natural model (Pnaturai naturai)) Right: The transferability of the patterns revealed for clean ResNet-50 models. The transfer rate defined as PW "Woorrce/ PWP target. the ratio between the predictive power on the source and the target model. The patterns are searched on the source model.
patterns can be used assist the detection backdoored models. The effectiveness our method on more complex backdoor triggers is worth further exploration.
Revisiting the patterns learned on natural data (Figure 2). we find that DNNs trained on natural data also have backdoors, since those patterns be immediately applied to "backdoor" attack the model. The images shown columns 3-8 in Figure are the backdoored images. The predictive powers shown Figure Finurr33]nndicatt oatte-of-the-art (Chen et al. 2017: Liu al.) (020)) and stealthiness is not guaranteed. Interestingly, these results indicate that DNNs tend memorize certain patterns for each class, which explains why backdoor attacks can casily succeed by poisoning only a small proportion of the training data.
iiddcaeee that the attack success rate will be high, although not as high as the
4.4 CLASS-WISE PATTERNS LEARNED BY DNNS IN ADVERSARIAL. SETTINGS
Here, we apply our method to identify the class-wise patterns learned by ResNet-50 on three dif- ferent versions of CIFAR-10 dataset: natural (D), robust (Dp) and nonrobust v2) The D and D versions generated in previous work (Ilyas et al. 2019) by perturbing the training images
8

---

Under review as a conference paper at ICLR 2021
D
Dx
Dr
AdvTrain
NR
"airplane"
"deer"
"frog"
Figure 8: Examples of class-wise patterns learned by ResNet-50 and revealed by our method on natural (D), nonrobust RD and robust (DR) data CIFAR-10, as well as adversarially trained model (AdvTrain) on CIFAR-10. Pattern size is 10% of the image size.
of the original dataset to have only robust or nonrobust features in an adversarial setting. In other words, Dr contains only the robust features learned by robust (adversarially trained) model, while DNR contains only the nonrobust features learned by a nonrobust model. was shown that DNNs exhibit moderate adversarial robustness when trained on DR using even standard training (Ilyas itt al. 2019). We train the ResNet-50 model independently using standard training on D, D and Dn, and adversarial training on DD
The class- wiiss patterns learned under different settings are illustrated in Figure Compared to D. the patterns learned on DNn are mostly background noise. However, patterns learned by the adversarially trained model contains clear shapes and much less background noise. Compared to that learned on the shapes learned by adversarially trained DNNs are more simplified and have more connected regions. This provides an interesting visualization of the robust patterns learned by adversarially trained DNNs from a class-wise perspective. The robust dataset DR is interesting it is dataset that is intentionally perturbed to be robust. The patterns learned on D reveal that the "robust" perturbation can remove much of the background noise (e.g. the "airplane" class). However, it may also remove a certain part of the robust shape (e.g. the body of the "deer"). This might produce DNNs that have poor generalization performance in real-world scenarios.
DISCUSSION AND CONCLUSION
In this paper, propose method to identify the class-wise patterns learned by deep neural net- works (DNN) and reveal the patterns in the imput space. The patterns revealed by method indicate that DNNs can learn abstract but consistent shape patterns that are associated with the tar- get class. By further examining the patterns learned by backdoored DNNs, we find that our method can reveal the trigger pattern, and more interestingly. DNNs trained clean data may also have "backdoors". Patterns revealed in adversarial settings indicate that adversarially trained DNNs can indeed learn more robust shape patterns, however, internationally perturbed robust datasets may lose certain shape features. Our method can serve as useful tool for communities to better understand DNNs trained on different datasets under different settings.
pertuuree
One application of our method DNN understanding and interpretation. Based on our method, one can develop metrics to measure the strength, weakness or biases (e.g. gender, color and other attributes) of the knowledge learned by DNNs. The other potential application is using our method develop effective backdoor defense methods by monitoring and avoiding the learning of high predictive patterns during training. Our method can also motivate more effective adversarial defense methods, for example, regularized adversarial training methods that can help DNNs learn more robust shape features.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning, arXiv preprint arXiv:/712.05526, 2017.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national Conference on Machine Learning (ICML). 2014.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In ICCV, pp. 3429 3437, 2017.
Chuang Gan, Naiyan Wang, Yi Yang. Dit- Yan Yeung, and Alexander G. Hauptmann. Devnet: A deep event network for multimedia event detection and evidence recounting. In Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. CoRR, abs/ 700.00033,,, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In International Conference on Computer Vision (ICCV), 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Network and Distributed System Security Sym- posium (NDSS), 2018
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. European Conference on Computer Vision (ECCV), 2020.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M Erfani, Sudanthi Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations (ICLR), 2018
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Conference on Computer Vision and Pattern Recognition (CVPR),
2017.
Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
10

---

Under review as a conference paper at ICLR 2021
Itay Safran and Ohad Shamir. Depth-width tradeoffs in approximating natural functions with neural networks. In International Conference on Machine Learning (ICML), 2017.
Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. CoRR, abs/1610.023911000 2016.
Carl-Johann Simon-Gaan Gabriel, Yann Ollivier, Léon Bottou, Bernhard Schölkopf, and David Lopez- Paz. Adversarial vulnerability of neural networks increases with input dimension. ICLR, 2018.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. In International Conference Learning Representations (ICLR), 2014.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In International Conference on Learning Representations (ICLR), 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference Learning Representations (ICLR), 2014.
Matus Telgarsky. benefits of depth in neural networks. In Conference Learning Theory (COLT), 2016.
Bolun Wang. Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Sym- posium on Security and Privacy (SP), 2019.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Latent backdoor attacks on deep neural networks. In Conference on Computer and Communications Security (CCS), 2019.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), 2014.
Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018.
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. Clean- label backdoor attacks on video recognition models. In Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep fea- tures for discriminative localization. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016

---

Under review as a conference paper at ICLR 2021
A ALGORITHM AND SEARCH SETTINGS
The proposed Input Space Class-wise Pattern (ISCP) searching method described in Algorithm I Given a DNN model f (trained on Drrain), a target class y and a canvas image H, (in the case of positive and negative canvas, F. is sampled from Pest), the pattern search produce is executed as follows. The mask is initialize have the same size with the input image (without the channel dimension) and all 0.5 values (line 1). We randomly sample a subset of non-target class images Dn from the test set Diest. which in most of our experiments, is set to the non-target class images from 20% of the test set.
The searching procedure iterates for T epochs (T' times full pass on 2n)) with minibatch size B = 4. In each epoch, for each minibatch {am } in Da, we first mix each image in the minibatch with the canvas image se using the mask m (line 4). We then compute the loss on the mixed inputs (line 5). and backpropagate to update the mask according to step size 17 0.02 (lines 6-7). The updated mask is then clipped into the value range of [0, 1].
After T epochs of optimization, we obtain mask m. We then clip the values according threshold 7, so that only a certain proportion (e.g. 5%) of the values are clipped to 1 (the rest are clipped to 0). We then apply the clipped mask on the canvas image I. to obtain the final pattern Py for target class y. The loss term balancing parameter a [0.1, 0.3] is slightly adjusted around 0.2 to produce the most predictive pattern. The y parameter set the value that only a certain percentage of the pixels are remained, which in most of our experiments is 5% of the image size.
Algorithm 1 Input Space Class-wise Pattern (ISCP) Searching Method
Require:
A DNN model f. canvas image Z., nontarget-classes subset D, C Drest, target class regu- larization balancing parameter a, clipping threshold y, input image size W H, epochs T = 5, batch size B step size 7) 0.02.
Ensure: Pattern p., for target class
1: m= =InitializeMask]
2: for in range(T") do
3: for (a/ (5) 18
J==1 in D, do
4:
a(1)
= m * F. + (1 m) xn
5:
L (- f2 (x(()) + a |ml|]
6:
8 Dc
8m
7:
m = m
1) si
8: Clip all the values in m to [0. 1]
9: end for
10: end for
11: Clip the values in m that are smaller than r to 0, larger than y to 1, and get m.
12: Py my Ic
13: return P
B MORE PREDICTIVE POWER RESULTS
B. PREDICTIVE POWER FOR EACH CLASS
Here, we show the detailed predictive powers of the class-wise patterns revealed by method for all 10 CIFAR- 10 classes and 50 randomly selected ImageNet classes. The experimental settings follows Section 4. ResNet-50 models trained on natural CIFAR- and ImageNet. The patterns are searched based on 20% of the test images, and the predictive power (defined in Section 4 is computed all (e.g. 100%) ) test images. In Figure 3 we have shown the mean predictive power over all 10 classes CIFAR- 10 and 50 classes of ImageNet, here we show the detailed predictive power for each class in Table I and Table 2 Note that our method only searches one pattern for each class, and the pattern size is set to be 1 1% 10% of the image size. Positive sampling of the canvases are used in these experiments.
As shown in Table ! and 5% of the image size. This confirms our findings in the backdoor trigger recovery experiment in Section 4.3. on naturally trained models, small patterns that are of 1% the image size are not
2 the predictive power is extremely low when the pattern size is below
12

---

Under review as a conference paper at ICLR 2021
Table I: Detailed predictive power of different sizes of the patterns found by method for ResNet- 50 trained on natural CIFAR-10. The predictive power shown separately for each CIFAR- 10 class. mPW: mean predictive over all classes; STD: standard deviation over all classes. Positive canvases used.
Pattern Size
Class
Original Acc
1 %
2%
3%
5 %
10%
airplane automobile
0.031
0,126
0.357
0.745
0.892
0.953
0,032
0.081
0.246
0,663
0.921
0.979
bird
0,043
0.161
0.333
0.665
0.955
0,932
cat
0.047
0.077
0.162
0.431
0.814
0.851
dear
0.021
0.104
0.365
0.698
0.771
0.960
dog
0.011
0.099
0.270
0.571
0.867
0.910
frog
0,010
0.041
0.104
0.383
0.838
0.967
horse
0.040
0.151
0.278
0.596
0.865
0.948
ship
0,010
0.124
0.248
0.589
0.901
0.968
truck
0.032
0.167
0.212
0.570
0.928
0.957
mPW
0.028
0.113
0.258
0.591
0.875
0.942
STD 0.013
0.038
0.079
0.107
0.053
0.036
predictive unless the model has learned backdoor trigger. At pattern size 5%, the predictive power on CIFAR-10 is 0.59, which means our method has found pattern that can represent 59% of the original accuracy on the target class. While these patterns exhibit different predictive power on different classes, the variation is fairly low. This indicates that method can found representative and predictive patterns reliably for different classes. At pattern size 10%, the mean predictive power across all 10 CIFAR-10 10 classes is 0.875, that is, the pattern can represent 87.5% of the original accuracy average. Large patterns are generally more predictive, but contains more background noise. As shown in Table the results ImageNett
similar that CIFAR-10 except that for the same pattern size, the patterns are more predictive on ImageNet and the variation is higher. This is because ImageNet has more diverse images in the same class than CIFAR-10.
B.2 PREDICTIVE POWER ON DIFFERENT TEST SETS
In our setting, the patterns are searched with the assistance of 20% clean test data, after which the patterns applied to the entire (i.e. 100%) test test the predictive power on the rest the 80% test data that were not used for pattern search. This is to rule out the influence of the 20% searching subset on the predictive power. We compute the CIFAR- 10 and ImageNet, and take the mean predictive power over all 10 classes CIFAR-10 and 50 randomly selected ImageNet classes. We also vary the pattern size from 1 % to 10% of the image size to visualize the trend.
compute their predictive powers. Here,
predictive power for naturally trained ResNet-50 50 models
The results are visualized in Table 3 As can be inferred, the mean predictive powers tested on the entire test
very close to that tested the 80% test set that were used for pattern searching. This confirms that the patterns identified by our methods remain predictive on unseen images. formal transferability study of our pattern across separately trained models can be found in Figure (right panel).
C MORE VISUALIZATIONS OF THE CLASS-WISE PATTERNS
C.I CLASS-WISE PATTERNS FOR MORE CIFAR-10 AND IMAGENET CLASSES
Figure illustrates the patterns found by our method for each of the 10 CIFAR-IO classes, while Figure 10 shows the patterns for 20 out of the 50 selected ImageNet classes. As be observed in Figure 9 our method can find interpretable patterns for all CIFAR-10 classes, and the patterns generally contain shape features that well describe the object class. The color the pattern reflects the texture information learned by the model to some extent. Comparing the patterns found on CIFAR-10 that on ImageNet in Figure 10 DNNs learn more detailed shapes on high resolution images. Another observation is that DNNS can learn more than one shapes at different locations of the input space, especially on ImageNet. The patterns revealed by our method seem to have noticeable correlation with the high occurrence objects of the class.
13

---

Under review as a conference paper at ICLR 2021
"airplane"
"automobile"
"bird"
"cat"
"deer"
"dog"
"frog"
"horse"
"ship"
"truck"
+0
Figure 9: Class-wise patterns found by our method CIFAR-10 pattern size 5%. We show the pattern for each of the 10 CIFAR-10 10 classes. The predictive powers of these patterns are reported in Table Positive canvases are used.
C.2 PATTERNS FOUND WITH DIFFERENT SIZES OF THE NONTARGET- SUBSET
Here, we show the patterns revealed by our method with different sizes of the nontarget-clan subset selected from the original test set. We apply our method to search for the patterns with different nontarget-class subsets of sizes ranging from 20% to 100% of the test powers of the patterns found each setting are still computed on the entire (i.e. 100%) test set. This experiment is conducted on ResNet-50 model trained on natural CIFAR-10 dataset, and the pattern size is fixed to 5% of the image size. The class-wiss patterns and their predictive powers are illustrated Figure 1T As the results show, the patterns are fairly consistent under different subset sizes (left figure), and 20% of the test set is sufficient for our method to find highly predictive patterns (right figure).
Note that the predictive
D MORE ANALYSES OF THE CANVASES
D.1 EFFECT OF DIFFERENT CANVASES ON PREDICTIVE POWER
Here, present detailed results of the effect of the canvas type on the predictive power. The four types of canvases (e.g. random, positive, negative and white) are descrihed in Section 3 The experiments are conducted with naturally trained ResNet-50 models on CIFAR-10 and ImageNet. The patterns
searched 20% of the test images and the predictive power is computed (c.g. 100%) of test images. The search setting of our method follows that described in Section For CIFAR-10, 10, we compute the mean predictive power (mPW) over all 10 classes, while for ImageNet, we compute the mPW on 50 randomly selected classes.
all
Figure 12 illustrate the mPW CIFAR-10 and ImageNet. We find that patterns found random and negative canvases are similarly predictive of the class. interesting observation that, on CIFAR-10, patterns found on the white canvas have even higher predictive power than those found on other canvases. However, the white canvas is less effective because CIFAR-I0 10 images are all row-rusolution images, on which the model tends to learn patterns that can be well represented by one single type of pixels. On high-resolution ImageNet images, the model learns more complex patterns that are associated with the textures. In this case, white pixels become less effective to represent the high-resolution patterns learned by the model.
ImageNet. We suspect this is
14

---

Under review as a conference paper at ICLR 2021
"black widow"
"bagel"
"wood rabbit"
"tram"
"acorn"
"drake"
"ant"
"table lamp"
"ladle"
"vending machine"
"electric fan
"leahhopper"
"ccaemicc gown"
"platypus" "Newfoundland dog"
"badger"
"rose hip"
"tench"
"slot"
"hotdog"
Figure 10: Class-wise patterns found by our method on ImageNet at pattern size 5%. We only show the pattern for 20 out of the 50 tested ImageNet classes. The predictive powers of these patterns are reported in Table 2 Positive canvases are used.
D.2 PATTERNS REVEALED ON DIFFERENT CANVASES
Following the experiment Section D.I here we show the class-wise patterns revealed on the four types of canvases: random, positive, negative, and white. For random, negative and white canvas, we show the pattern revealed on canvas images, while for the positive canvas, show the pattern revealed on canvas images (i.e. Positivel and Positive2).
Figure 13 illustrates the patterns revealed for 3 CIFAR-10 10 classes. As can be observed, the patterns found on different canvases share common shapes, although there are some variations in the details and locations of the shape. This confirms that the patterns revealed by method represent some
15

---

Under review as a conference paper at ICLR 2021
20%
40%
60%
80%
100%
CIFAR-10
"airplane"
Predicive
0.4
"deer"
W
0.3
0,0
Nontargel-dass Subset Size (%)
Figure 11: Left grid: Patterns found by methods under different nontarget-class subset size (% to the test set), for ""irplann"" and "deer" classes in CIFAR-10. Right plot: The mean predictive power of the class-wise patterns on CIFAR-10. The pattern size is fixed to 5% of the image size.
Cifar-10
Imagenet
Moan prodicivv power
0.8
Mean prodictiv power
0.8
0.8
0.2
0.2
Postive Negative
Random
White
Negative
0
5 8
4 5
8 10
Pattern size (%)
Pattern size (%)
Figure 12: Mean predictive power (mPW) of the class-wise patterns found by our method over all 10 CIFAR-I0 classes or 50 randomly selected ImageNet classes for naturally trained ResNet-50 models. The mPW is shown for each of the sampling strategies: random, positive, negative and white. The pattern size varies from 0% to 10% of the image size.
consistent information learned by the model. This also implies that the model may learn to be sensitive to those patterns different locations of the input space. In other words, DNNs tend to learn certain class-wis wise patterns at multiple (rather than one) locations of the input space. other hand, this also means that the shape features exist in one pattern may not be the only pattern learned by the model. Note that the texture (e.g. color) of the pattern is highly correlated to the canvas image, which we have discussed Section 4.1
the
Figure 14 shows the patterns found on ImageNet. For some of the canvases, the revealed pattern has a noticeable correlation with the object in the canvas, especially for the positive canvases. For example, the first two patterns of the "analog clock" class. Note that the patterns revealed by the white canvas also have clear shape features, though there are no canvas dependent features. The shapes in white-canvas patterns tend to appear at more locations of the input space, which may be related to the spatial distributions of the objects in the class.
D.3 EFFECT OF DIFFERENT POSITIVE CANVASES ON PREDICTIVE POWER
Here, we further show how different positive canvases affect the predictive power. We test positive canvases selected from the top-N most confident and correctly classified images. We reveal the patterns learned by ResNet-50 models trained on natural (clean) CIFAR-10 and ImageNet datasets. Same as in Section 4.1 and Appendix [_II] here we search for the pattern on 20% of the test images and test its predictive power the full test set. The results reported in Table 4 and Table [5] where it shows that patterns found on the top-1 positive canvas are not necessary the most predictive ones. As we have shown in Figure [3 and different sampling strategies can reveal similar shapes, but the details and the locations of the shapes can be quite different. This is also why we suggest to use N (i.e. N=5) different canvases help find the most predictive patterns.
16

---

Here, we visualize more patterns found by the universal adversarial perturbation (UAP), our method and the TV variant of our method (defined in Equation 3 We naturally trained ResNet-50 models on clean CIFAR-10 10 and ImageNet. For each dataset and model, the patterns are generated based on 20% of the test images and tested on the full test set. For UAP, we use the targeted version and set the target class the class reveal. The UAP is accumulated in class-wise manner to generated the pattern separately for each target class. For our methods, the pattern size set to 5% of the image size. UAP will perturb the entire image, which corresponds to the 100% image size. [8, 32). For the TV variant of our method, we set Under review as a conference paper at ICLR 2021
Positivel
Positive2
Negative
Random
White
"airplane"
"deer"
"frog"
Figure 13: Class-wise patterns revealed by our method using different canvases (e.g. two positive, negative, random and one white canvases) on CIFAR-10 for naturally trained ResNet-50. The pattern size is fixed 5% of the image size. Here we only show classes as an example.
E MORE COMPARISONS TO UNIVERSAL ADVERSARIAL PERTURBATION
UAP is bounded by the maximum perturbation parameter 8 0.2 or 3 1 (see Equation 3).
Figure 15 illustrates the class-wise patterns revealed by different methods. The UAP patterns gen- erated on CIFAR-10 share some similarities with our method. However, our method can find much cleaner patterns. The UAP patterns generated ImageNet terns. This is because UAP explores the unlearned space of DNNs, and the unlearned space expands with the increase of the input dimensionality (Simon-Gate Gabriel et al. 2018). By contrast, our method searching (rather than perturbing) method that does not rely adversarial perturbations. There- fore, it needs find the optimal pixel locations the input space for the pattern be predictive of the class. This forces the searching algorithm to find the well-learned positions of the input space. We believe UAP can be applied in more controlled manner to explore the well-learned (rather than unlearned) space of DNNs. We find that the TV regularization terns with less background noise, however, the main patterns are generally the same as those found without using TVV.
more abstract than CIFAR-10 10 pat-
help produce smooth pat-
Table reports the the mean predictive power of the patterns revealed by the above three methods. The mean predictive power is computed over all 10 CIFAR- 10 classes 50 randomly selected ImageNet classes. For UAP, we use the attack success rate (ASR) as the predictive power. As can be inferred from Table 6 the predictive powers of our method are similar to UAP with maximum perturbation e=8. UAP has much higher predictive power e=16 or c=32, expected. For the TV
17

---

Under review as a conference paper at ICLR 2021
Positive1
Positive2
Negative
Random
White
"tabby"
"analog clock"
"bagel"
Figure 14: Class-wise patterns revealed by our method using different canvases (e.g. two positive, one negative, one random and one white canvases) ImageNet for a naturally trained ResNet-50. The pattern size is fixed to 5% of the image size. Here we only show classes as an example.
version of our method, increasing the regularization strength from 3=0.2 to 3=1 tends find less predictive patterns. This is because TVV removes the predictive background noise. This confirms that background noise also plays important role in DNNs (Ilyas et al. 2019) which also aligns with our findings in the adversarial setting in Section 4.4
MORE PATTERNS REVEALED ON BACKDOORED MODELS
In addition our analysis in Section 4.3 here we test two more types BadNets triggers and apply our method to reveal these triggers. We train a ResNet-50 on BadNets poisoned CIFAR-I -00 set to
obtain the backdoored ResNet-50, i.e., 10% of the data is poisoned following (Gu et al. 2017). We the backdoor target class to "airplane". and use three types of backdoor triggers (Shown Figure 16). The backdoored models are trained without using any data augmentations. This is typical setting for backdoor attack as data augmentation will make it more difficult for the model to remember the trigger pattern (Gu et al. 2017). As comparison, here also reveal the patterns for backdoored models trained with data augmentations: random shift, random crop and random horizontal flip.
Figure 16 illustrates the patterns revealed by our method at pattern size 1% (of the image size). As shown the figure, method reliably reveal the backdoor trigger pattern learned by the model, for backdoored models trained with or without data augmentations. In the no data augmentation case, the recovered trigger patterns are very close to the ground truth triggers. This indicates that DNNs can memorize both the trigger pattern and its location. Interestingly. in some of the recovered patterns, there are also trigger-sima -imglrs patterns different places of the input space. For example, the middle two columns in right subfigure sigger- patterns from natural data, as we discussed in 4.3 As illustrated in the left panel of Figure 16, data augmentations tend to shift the trigger location.
Figure [6] This indicates that the model may learn
18

---

Under review as a conference paper at ICLR 2021
Ours TV
Ours TV (B=1.0)
UAP (E=8)
UAP (c=16)
UAP (c=32)
Ours
(B=0.2)
"airplane"
"deer"
*frog"
"tabby"
"analog clock"
"bagel"
Figure 15: A comparison of the class- wise patterns revealed for naturally trained ResNet-50 models by universal adversarial perturbation (UAP), our method and our method with the total-variation (TV) (Fong & Vedaldi 2017) regularization (defined in Equation 3). For our methods, positive canvas used and the pattern size set 5% of the image size. UAP perturbs the entire (i.e. 100%) image. parameter is the maximum perturbation constraint for UAP, while 3 is the parameter for TV regularization. The top and bottom 3 rows show the patterns for CIFAR- 10 and ImageNet classes, respectively.
G ATTENTION SHIFT WHEN CLASS-WISE PATTERN IS ATTACHED
We use the attention map to check the network's 's attention shift when the class-wise pattern found by out method is attached to an image. As shown in Figure I7 we find that class-wise pattern has very strong attention around the areas that contain clear shapes, and the attention of the network significantly shifted towards those areas when the pattern attached different images. This confirms that the patterns identified by our method are indeed universal patterns that carry consistent information.
H EXTENSION TO LEARNED CANVAS
Here an extension of method simultaneously perturb the mask and the canvas x. as follows:
min log (y x. + (í m) xr.) + a- |m]] m.
4)
This is a direct extension our method defined in Equation 2 with the canvas x. also being per- turbed during the searching process. This extension can also be interpreted combination of our mask perturbation canvas image Ze. Note that, different from UAP, here the perturbation on the canvas is not bounded. By solving Equation 4 the canvas is also learned to be predictive, which might potentially lead to more precise and predictive patterns.
Equation 2 and the universal adversarial perturbations (UAP) applied on the
19

---

Under review as a conference paper at ICLR 2021
Backdoored
images
Canvas
Patterns
Data Augmentation
No Data Augmentation
Figure 16: Backdoor patterns revealed by our method for different backdoor triggers. The back- doored ResNet-50 model is trained on CIFAR-10 10 and poisoned by BadNets towards the target class "airplane". The pattern size is set 1% of the image size, and the patterns are searched based on 20% of the test set. Both positive and white canvases are used. Data augmentation means the back- doored model was trained with data augmentation techniques. For the data augmentation setting, we also use the negative canvas (the middle column in the left panel).
Pattern: "tench"
Orig
Orig+ Pattern
Orig
Orig + Pattern
Orig
Orig+Pattent
A
Figure 17: The attention (visualized by Grad-CAM) shift of ResNet-50 model (on ImageNet) when class-wise pattern is attached to different images. The pattern from (11144400""""eeeee""
In the above objective, there are two different types of variables that need to optimized the same time: the mask m and the canvas Te- To tackle this, we apply an alternating optimization strategy as follows. We update the mask and the canvas each for steps alternatively with step size 2/255 for the canvas and step size 0.02 for the mask. The same normalization and clipping techniques used here as in our original method. We use positive sampling to select the initial canvas image and set the pattern size to 5% of the image size. The patterns are searched for based on 20% of the test set images. We compare side-by-side the patterns revealed by the two methods in Figure [8]
On CIFAR-10 dataset, the patterns found on learned canvas also contain shape features, but for some classes, it tends to focus on only the most important part of the object. For example, the antlers found for the "deer" class. This is because, by perturbing the canvas, the method will explore the most vulnerable part of the canvas that can lead minimized loss towards the target class. On ImageNet, perturbing the canvas tends to find patterns at the four corners or edges of the canvas image, shown in the right panel of [8] This is because those regions are the most under-learned and vulnerable space that can most effectively make the network predict the target class. Overall, perturbing the canvas tends to force the method explore the most vulnerable and unlearned feature space of DNNs.
The predictive powers of the patterns found fixed versus learned canvases reported in Table The predictive power is tested on the entire test set of either CIFAR-10 or ImageNet. Overall, the predictive powers the patterns found on learned canvases are similar to those found on fixed can- vases, although there are some variations. Surprisingly. the learned (perturbed) canvas does not lead more predictive patterns, considering the high effectiveness of adversarial perturbations. Upon
20

---

Under review as a conference paper at ICLR 2021
"airplane"
"deer"
"frog"
"tabby"
"analog clock"
"bagel"
Our Canvases
Our Patterns
Learned Canvases
Learned
Patterns
CIFAR-10
ImageNet
Figure 18: A comparison of the class-wise patterns revealed by our method and an learned canvas version of our method (defined in Equation 4) for naturally trained ResNet-50s CIFAR-10 10 and ImageNet. The pattern size fixed to 5% of the image size. The first row shows the patterns found by the original method, while the learned canvas and the revealed patterns are shown the second and third rows respectively.
further investigation, find that this is due to three reasons: 1) the pattern revealed when using the learned canvas class-wise universal adversarial pattern, which less effective (predictive) than a sample-wise adversarial perturbation; 2) only 5% of the perturbed canvas is used to extract the pattern, which significantly reduces the adversarial (predictive) effect; 3) the pattern needs to be attached other clean images compute the predictive power. This is transfer scenario that also reduces the adversarial effect. This eventually results in similarly predictive patterns for both the fixed canvas and the learned canvas.
21

---

Under review as a conference paper at ICLR 2021
Table Detailed predictive power of different sizes the patterns found by our method for ResNet- 50 trained on natural ImageNet. The predictive power is shown separately for each of the 50 Im- ageNet classes. mPW: mean predictive over all 50 classes; STD: standard deviation over all 50 classes, Positive canvases are used.
Pattern Size
Class
Original Acc
1%
2 2%
3%
5%
10%
tench black widow
0.000
0.054
0.293
0.609
0.641
0.920
0,000
0.116
0.661
0.768
0.779
0,860
drake
0.000
0.083
0.523
0.821
0.952
0.840
platypus
0.000
0.000
0.013
0.275
0.750
0.800
dowitcher
0,000
0.062
0.300
0.656
0.856
0,900
oystercatcher
0,000
0.007
0.177
0.469
0.677
0,960
chesapeake bay retriever
0.000
0.057
0.193
0.636
0.852
0.880
schipperke
0,000
0.036
0.155
0.667
0.929
0.840
newfoundland dog toy poodle leopard ant leafhopper
0,000
0.183
0.695
0.878
0.902
0.820
0.000
0.146
0.667
1.270
1.292
0,480
0.000
0.091
0.625
0.705
0.659
0.880
0.000
0.105
0.776
0.921
0.987
0.760
0.000
0.044
0.378
0.656
0.778
0.900
wood rabbit
0.000
0.000
0.070
0.465
0,802
0.860
badger
0.000
0.000
0.081
0.453
0.674
0.860
gorilla academic gown
0.000
0.013
0.113
0.563
0.675
0.800
0.000
0.000
0.026
0.632
1.289
0.380
backpack
0.000
0.000
0.045
0.955
1.841
0.440
bicycle-
bosyele-built-fo-twwe
0.000
0.081
0.302
0.698
0.814
0.860
bookcase
0.000
0.000
0.015
0.409
0.788
0.660
castle
0.000
0.025
0.163
0.538
0.750
0.800
chain
0,000
0.154
1.731
2.385
2.654
0.260
church
0.000
0.029
0.353
0.912
1.088
0.680
cradle
0.000
0.000
0.100
0.800
1.925
0.400
electric fan
0.000
0.000
0.273
0.580
0.625
0.880
go-kart
0.000
0.000
0.011
0.319
0.702
0.940
holster
0.000
0.013
0.066
0.395
0.658
0.760
ladle
0.000
0.081
0.291
0.639
0.663
0.860
lifeboat
0.000
0.117
0.596
0.766
0.702
0.940
loupe
0.000
0.130
0.761
1.391
1.717
0.460
paper towel
0.000
0.000
0.068
0.405
0.703
0.740
ping-pong ball
0.000
0.133
0.389
0.667
0.711
0.900
punching bag
0.000
0.214
0.600
0.914
1.000
0.700
saltshaker
0.000
0.106
0.576
0.742
0.788
0.660
sax
0.000
0.103
0.515
0.632
0.588
0.680
slot spotlight
0.000
0.117
0.511
0.713
0.713
0.940
0.000
0.033
0.133
1.033
1.633
0.300
teapot
0.000
0.150
0.625
0.886
0.913
0.800
tram table lamp
0.000
0.020
0.200
1.000
1.260
0.500
0.000
0.012
0.105
0.360
0.674
0.860
vending machine
0.000
0.011
0.076
0.050
0.674
0,920
wool
0.000
0.000
0.075
0.525
1.350
0.400
traffic light
0,000
0.057
0.341
0.670
0.841
0.880
ice lolly
0,000
0.045
0.341
0.727
0.807
0,880
bagel
0,000
0.029
0.286
1.000
1.214
0.700
hotdog
0.000
0.000
0.012
0.267
0.616
0.860
spaghetti squash
0.000
0.000
0.012
0.159
0.671
0.820
acorn
0.000
0.104
0.531
0.739
0.792
0.960
rose hip
0.000
0.065
0.261
0.696
0.935
0.920
coral fungus
0,000
0.000
0.010
0.188
0.521
0.960
mPW
0.000
0.056
0.322
0.701
0.936
0.761
STD
0.000
0.057
0.308
0.347
0.405
0.188
22

---

Under review as a conference paper at ICLR 2021
Table 3: Predictive power tested on 80% test images that are not used for pattern searching. The results in the brackets are predictive powers tested on the entire (i.e. 100%) test set. The results are shown for ResNet-50 models trained natural CIFAR-10 10 and ImageNet, and averaged (i.e. mean predictive power) over all 10 classes of CIFAR-10 and 50 randomly selected ImageNet classes.
Pattern Size
Dataset
1 %
2%
3%
5 %
10%
CIFAR-10
0.028 (0.028)
0.112 (0. 113)
0.256 (0.258)
0.599 (0.591)
0.861 (0.875)
ImageNet
0.000 (0.000) 0.067 (0.055)
0.322 (0.316)
0.680 (0.685)
0.932 (0.930)
Table 4: Predictive power of class-wise patterns revealed by different positive canvases naturally trained ResNet-50 on CIFAR-10. The canvases are selected using positive sampling based on the top-N (N=5) most confident and correctly classified images. Pattern size fixed to 5% image size. STD: the standard deviation of the predictive power over all 5 canvases. For each class, the
predictive power of the best canvas highlight in bold.
Positive Canvas top-3
Class
STD
top-1
top-2
top-4
top-5
airplane
0.473
0.651
0.745
0.729
0.392
0.141
bird
0.665
0.623
0.470
0.220
0.591
0.161
dear
0.622
0.698
0.659
0.511
0.553
0.068
frog Ship 0.312
0.135
0.383
0.377
0.246
0.276
0.092
0.262
0.507
0.550
0.589
0.132
Table Predictive power of class-wise patterns revealed by different positive canvases on a naturally trained ResNet-50 ImageNet. The canvases selected using positive sampling based on the top-N (N=5) most confident and correctly classified images. Pattern size fixed 5% image size. STD: the standard deviation of the predictive power over all canvases. For each class, the predictive power of the best canvas is highlight in bold.
Positive Canvas top-3
Class
STD
top-1
top-2
top-4
top-5
tench
0.599
0.609
0.433
0.420
0.476
0.081
tibetan terrier
0.760
0.793
0.707
0.861
0.682
0.064
academic gown
0.411
0.572
0.301
0.632
0.596
0.126
hook
1.271
0.823
0.511
0.591
0.615
0.274
Slot
0.713
0.702
0.590
0.655
0.652
0.044
Table 6: Attack success rate and predictive power of class-wise patterns generated by universal adversarial perturbation (UAP) "Moosavi-Deztoon et al., (017) and our method. For our method, we also test the use of the votal-variation (TV) (Fong & Vedaldi 2017) regularization along with L1 regularization (this variant of method is defined Equation 3). The UAP pattern size is 100% of the image size while our patterns (L, or TV) are 5% of the image size.
Methods
UAP (c=8)
UAP (c=16)
UAP (c=32)
Ours
Ours+TV (8=0.2)
Ours+TV (8=1)
CIFAR-10
0.621
0.967
0.982
0.609
0.580
0.302
ImageNet
0.607
0.683
0.944
0.623
0.597
0.272
23

---

Under review as a conference paper at ICLR 2021
Table Predictive power class-wise patterns revealed by our method and by the extended version of our method (defined in Equation 4) for naturally trained ResNer-50s on CIFAR-10 and ImageNet. Pattern size is fixed to 5% image size.
CIFAR-10 Class
ImageNet Class analog clock
Method
airplane
deer
frog
tabby
bagel
Fixed Canvas
0.745 0.760
0.698
0.383
0.547 0.164
0.711 0.667
1.000
Learned Canvas
0.659
0.515
0.816
24


Reviews:
[{"review_id": "uL6bGJF7T3K", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nThis work proposes a technique to compute sparse patterns in the input space that are highly predictive of a class, even when added to samples from a different class (non-target samples).\nPatterns for class $c$ are produced by optimizing a point-wise weighted average of two samples (a fixed sample called the canvas and one that changes after each optimization step referred to as $x_s \\in X_s$) s.t. the result is highly predictive (yields a low error) for $c$. Learned weights for the average (the mask) are thresholded and used as a binary mask for the canvas (which is either a sample that has been labeled/predicted as class $c$ or a white image).\nAnalysis of the resulting patterns focuses on explainability on various fronts: first, the consistency of the pattern is evaluated (via a custom metric called 'predictive power') which shows that resulting patterns do cause a CNN to predict the class that has been optimized for.\nA few remarks on the patterns are made and also some claims regarding the link to the model's learned features (like position, structure, and number of shapes).\nAttention maps confirm that the saliency of the patterns prevails over the original content of the canvas image. Finally, models trained adversarially and with backdoors are analyzed by computing the sparse-mask patterns.  \n\n\n#### Strong and weak points of the paper:\n##### Strengths:\n - The paper is easy to read and the main ideas can be followed. A wide array of experiments are conducted with a small and (at least a subset of) a large dataset.\n - The proposed method is straightforward and described in enough detail to be easily reproducible.\n - Comprehensive and relevant related work has been cited and compared.\n - Section 4.2: an interesting baseline showing that the resulting patterns are more effective at causing the model to predict a target class than highly responsive areas of a \"natural\" sample.\n - Section 4.3 and 4.4: the inclusion of an analysis of training regimes that are known to affect the feature space learned by a neural network (adversarially trained, trained with robust features, and trained with backdoors) provides relevant use-cases for a method that is proposed as an interpretability tool.\n - Section 4.4: interesting results that seem to agree with Ilyas et al. 2019 in that more robust models require a perturbation that looks more like the adversarial target.\n\n##### Weaknesses:\n - The main concern is that the goal of the paper is not aligned with the findings: I consider that the patterns shown are a valuable tool to analyze the behavior of a neural network, but from a completely different perspective. These patterns (and the algorithm to obtain them) suggest that there are global-targeted adversarial perturbations (a close analogous to the work of Moosavi et al.: global perturbation because it is one perturbation applied to a set of samples; targeted because each perturbation has been designed to cause attacked samples to predict one specific class). In turn, the patterns' appearance is as difficult to interpret as those of classical adversarial perturbations or global perturbations. Claims regarding shape, texture, position, etc, are not well supported by measurable evidence (beyond a few observational remarks supported by a handful of examples), and are more likely the result of the method by which these perturbations are obtained (e.g, the regularization w.r.t. the mask, clipping, binarization, point-wise multiplication). Instead of trying to interpret the appearance of such perturbations, a focus on the implications of global-targeted adversarial perturbations can provide valuable insights into the manifold's structure learned by modern neural networks as well as their shortcomings. \n - Most experiments are missing key details regarding their setup. In particular, the canvas image is often not shown (especially relevant for Figures 1, 2, and 7). From figures where it is shown (like Figure 4), it seems like regardless of the canvas, a binary pattern with a global-targeted perturbation is learned.\n - An analysis of the impact of the canvas is missing w.r.t. predictive power which could support/invalidate the idea of the learned pattern to be a global-targeted adversarial perturbation.\n - Section 4.1, Patterns Revealed by Positive Canvas: several claims regarding the properties of the patterns (e.g., structural consistency, number, and position of shapes) in the mask are only supported by a qualitative evaluation of a few samples. No metrics are proposed.\n - Section 4.1, Patterns Revealed by Positive Canvas: although a metric (predictive power) is used to measure the consistency of the pattern w.r.t. a target class, results are based on a few classes. Why not report it for all 10 classes on CIFAR10? Also, results on 5 classes of Imagenet is not representative of the variety found in that dataset. A simple average of the predictive power over each class would provide stronger support for the results found in the subset of selected classes.\n - Section 4.1, Patterns Revealed by Different Canvases: the absence of texture in the patterns is more likely caused by the sparsity term on equation 2 (together with the threshold and the point-wise multiplication with the canvas) rather than by an intrinsic property learned by the model.\n\n\n#### Recommendation:\n - In its current form and with the proposed focus, I consider that this paper cannot be accepted. The focus of the analysis can yield stronger, more conclusive results from the standpoint of global adversarial perturbations.\n\n#### Supporting arguments for recommendation:\n - Without the right context (i.e., not that of explainability but adversarial attacks) the analysis of what the patterns represent is poorly supported by experiments. There is a lot of work done in the direction of adversarial attacks that findings in this work could re-purpose. Meanwhile, the impact of proposing a \"global-targeted adversarial attack\" based on one sample seems promising, but it is out of the scope of this work's analysis. Refer to the list of weaknesses for more details.\n\n#### Questions to clarify:\n - How are the class-patterns related to universal perturbations?\n - How is this method fundamentally different (i.e., more advantageous) than a universal adversarial perturbation (Moosavi-Dezfooli 2017) generated for each class?\n - If the learned pattern says something about the (clean) feature space of the classifier, what do these patterns reveal about classes with a large intra-class variation? For example, the class \"clock\" contains digital and analog clock faces (on the extreme, digital clocks show only numbers, and analog ones only the hands of the clock). Apart from their semantic relationship, there is no visual correspondence between the two sub-categories. Would this scenario be a limitation of the proposed method or what does it reveal about the model/the data?\n - Section 3, Equation 2: what happens to the mask (therefore to the conclusions from the patterns in the mask) with other regularization metrics like Total Variation (TV)? TV is used in [2] to find masks with connected areas that are representative for each sample in a counterfactual manner (i.e., blurring the area of the mask, causes the model to predict something else). In fact, not using TV makes the mask converge to an adversarial perturbation. Comparing those findings to those on this work (in particular Equation 2), makes me think that the resulting mask is more of a targeted adversarial attack rather than a representative pattern of the class. Moreover similar work published in ICLR 2016 [3] shows how almost any source image can be 'anchored' to the prediction of a guide image (equivalent to the canvas image in this work). This paper is essentially computing a sparse version of that attack via a masked input and the output layer (in [3] they don't apply masks and the deep representation was not constrained to the output but an arbitrary layer). It would be interesting to show the reconstruction of samples that have been \"attacked\" with the patterns in this work (following [3]).\n - Section 3, sampling canvas and results in Figure 4: how is the predicted power affected by the choice of canvas (positive, negative, or white)? Evaluating on a few classes can be misleading and although the metric of predictive power (PW) seems adequate, I suggest presenting a global metric by taking the mean PW (mPW) to better support the generality of the corresponding results.\n - Experiments with Imagenet: can you specify which classes the chosen IDs map to? Are they all related somehow? What were the criteria to select those 5? Why only 5?\n - Figure 2: it would be important to see what the canvas image looks like in order to establish how much of the pattern is just coming from that canvas.\n - Section 4.1, Patterns revealed by Positive Canvas: how are the number of shapes measured? Where does one shape start and one ends? It seems this observation is rather empirical and only a few samples are shown as evidence (one may ask how often does this happen and whether it happens because the dataset also has multiple shapes -which requires no mask, just simply looking at the samples). The same issues arise with claims about the position where the patterns occur (how is this measured?).\n - Section 4.1, Patterns Revealed by Different Canvases: how is the consistency of patterns evaluated? Is there further evidence beyond the 4 samples shown in Figure 4 for CIFAR10? Those seem to share structural similarity but, is this always the case? How do you establish structural similarity on the samples from Imagenet?\n - Section 4.2: taking one test image per class (source) whose highest activation area (with GradCAM) is pasted into ...how many target images? In other words, are the results in figure 5 (right) based on how many samples overall? This experiment needs to be described in more detail to be reproducible.\n - Claims for backdoor images about the trigger being recoverable is rather weak for two reasons: (1) there is only one class being evaluated and more importantly (2) there is only one (simple) pattern that has been analyzed. From this experiment, there are some indications that backdoor patterns may be reliably retrieved but nothing conclusive (the full trigger pattern has not been reconstructed and other, more complex patterns have not been evaluated).\n\n\n#### Additional feedback (not necessarily part of the decision assessment):\n - The notion of 'backdoor' is not explained in the abstract and it is not clear from the context what it could be. Including a brief description about it on the abstract could improve readability.\n - In the second paragraph of Section 1: references for \"backdoor\" attacks and \"adversarial\" attacks are swapped.\n - Section 1, 4th paragraph: the notion of \"canvas image\" is used without it being defined.\n - Section 2, \"General Understandings of DNNs\": there is work that has also focused on modeling properties of the input space over the dataset (and not only sample-wise) [1].\n - Section 3 paragraph \"Canvas Sampling\": typo for \"reveal\".\n\n\n#### References:\n[1] Palacio, Sebastian, et al. \"What do deep networks like to see?.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\n\n[2] Fong, Ruth C., and Andrea Vedaldi. \"Interpretable explanations of black boxes by meaningful perturbation.\" Proceedings of the IEEE International Conference on Computer Vision. 2017.\n\n[3] Sabour, Sara, et al. \"Adversarial manipulation of deep representations.\" ICLR (2016).", "extra_scores": {}, "extra_text": {}}, {"review_id": "goQsuc5B_4E", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The papers proposes a simple method for visualizing the patterns learned by deep neural networks in the supervised classification setting. Informally, suppose you have an image x that is \"representative\" of the class y and let X be a set of images that belong to other classes. The authors propose an optimization problem that looks for a mask (i.e. set of pixels) along with values of those pixels such that when this pattern is added to any image in X, the model will predict the new image to have the label y. This optimization problem can be solved using iterative thresholding and one may control the level of sparsity as the authors studied. Despite its simplicity, it can reveal clear patterns, particularly on high resolution images, such as ImageNet. The authors, then, show how this method can be used to interpret neural networks, detect backdoor attacks during training, and verify robustness.\n\n\nOverall, it is a great paper, insightful, and a pleasure to read. \n\nSome comments:\n\n1- The authors insist throughout the paper that this is a method for discovering class-wise patterns. To me, it appears to be a sample-wise pattern since a single representative sample x is used. When multiple samples are used, multiple patterns are detected at different locations as shown in some of the figures. I don't understand why the authors make such an emphasis. The algorithm seems great for providing interpretability per sample. Perhaps, this is because other methods for sample-wise interpretability already exist, such as attention maps, but I don't think that is a serious issue since it does not hurt to have multiple approaches for the same problem. I would suggest that the authors mention, at least, in the paper that this can be used sample-wise as well for interpretability. \n\n2- I am curious to know if upsampling CIFAR10 to have 224x224 images with a reasonable interpolation method would generate better patterns (closer to the patterns observed in ImageNet).\n\n3- In Section 4.3, the authors speculate that augmentation is responsible for the fact that the method finds the trigger patterns at different locations. This is easy to verify by simply training without augmentation. Has this been done? \n\n4- It would be better for readability if all images are placed at the top of the pages, rather than in arbitrary places. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8dGPa9j84_l", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a visualization method to reveal the class-specific discriminative patterns of DNNs in the input space. When added to images from another class, such patterns can lead the DNN to classify the images into the pattern's class. From the experimental results, the authors conjecture that images trained on natural data can have backdoors. It also claims that the method reveals the trigger patterns of backdoor attacks, that adversarially trained models learn simplified shape patterns, but an intentionally-perturbed robust dataset improves model robustness by sacrificing its ability to represent shapes. \n\nDespite feeling unsure about certain points, I think this paper is well-written. However, some of its main conclusions do not seem to be well-supported by the experimental results and analysis. \n\nStrengths:\n1. Proposing the interesting idea and method to find class-wise patterns that change the DNNs' prediction into a certain target image when added to images from another class. Such patterns are shown to reveal the abstract shapes and texture of the target class. \n\n2. Generalizing the analysis to various scenarios including backdoored models and adversarially-robust models.\n\nWeaknesses:\n1. The conjecture that \"DNNs trained on natural data can also have backdoors\" does not seem to be well-supported by the \"predictive power\" in Figure 3, due to the definition of predictive power. My understanding of a backdoor trigger for image classification is some pattern that generalizes well to images that are not used to craft such triggers. However, by definition of predictive power, it is evaluated on the examples that are used to generate such patterns, since $ACC(f(x_n+p_y), y)$ seems to be defined on the nontarget-class images. This is more like finding universal targeted adversarial attacks in the white-box setting, which may not be so surprising to succeed. I think it would be much more convincing if $ACC(f(x_n+p_y), y)$ is defined on a different set of nontarget-class images that are not used to craft the patterns.\n\n2. It is shown in Figure 6 (right) that the patterns from backdoored models do not transfer well to natural models, and it is suggested that such a method can be used to detect backdoored models. However, the results for transferability across natural models trained from different initializations seems to be missing. We do not know how the pattern's predictive power changes on another natural model that is not used to craft the pattern, and if it drops significantly, we cannot expect to use the method to detect backdoored models.\n\n3. Patterns shown in Figure 6 (left) do not seem to reveal the trigger at all. Only the pattern from the white canvas match the pattern, but its position is not correct. It says the position mismatch may be caused by data augmentation, but it is not convincing enough since no results on backdoored model trained without data augmentation is shown. It would be much better if such results can be included.\n\n4. The patterns from Figure 7 are also not conclusive enough. For example, patterns for the \"deer\" class seems to be in a worse shape when crafted from $\\hat{\\mathcal{D}}_R$ than ${\\mathcal{D}}$, but it is the opposite for the \"airplane\" class. Therefore, I do not think we can conclude that \"intentionally-perturbed dataset improves model robustness by sacrificing its ability to\nrepresent shapes\".\n\n\n### Updates after the rebuttal\nI appreciate the authors' timely response. The latest update explaining the regularization effect of using a fixed canvas $x_c$ and a learnable mask to avoid overfitting to spurious features, which seems to be inevitable if using a learnable $x_c$, does seem plausible and differentiates itself from the general idea of UAT. From the updated content in Appendix H, using a learnable $x_c$ does not seem to be able to find interpretable patterns. I feel contributing a new method with some improvements is worth an accept.\n\nHowever, I still feel the results are not conclusive enough. Regardless of the final decision, I hope to see more comparisons with simpler baselines in the future version. To highlight the novelty and effectiveness of the proposed method, the authors should try to compare with more baseline approaches that do not have presumptions on $x_c$ and initialize $x_c$ randomly. Currently, $x_c$ are selected as images having highest prediction scores from the model. Instead, we can learn $x_c$ from random initializations by using some approaches that enhance the transferability of adversarial examples. There are simple methods on improving the transferability of adversarial examples, e.g., adding Gaussian noise when crafting adversarial examples [1], or adding different random data augmentations [2]. These methods may also lead to interpretable patterns but the effectiveness of the resulting patterns could be stronger. I do not fully agree that a desirable canvas has to be a neutral or unbiased image. \n\n[1] Wu, Lei, Zhanxing Zhu, Cheng Tai, and Weinan E. \"Understanding and enhancing the transferability of adversarial examples.\" arXiv preprint arXiv:1802.09707 (2018).\n[2] Huang, Qian, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. \"Enhancing adversarial example transferability with an intermediate level attack.\"  CVPR (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "5g_ryOYkNP0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studied what the model has learned from the data. It proposed a class-wise pattern searching method to demonstrate the existence of class-wise predictive patterns in the input space. It studied three training settings, including natural, backdoored and adversarial, and revealed different characteristics of these training settings. \n\nI think that the motivation is very clear,  and the writing is readable, though some minor typos or inconsistencies should be corrected. \n\nThe proposed class-wise pattern searching method is very simple but reasonable. My opinion is that it may get somewhat inspiration from the backdoor triggers, since the format of the mixed input in Eq. (1) is same with the poisoned sample with triggers in the backdoor learning. If a discussion about this connection is added, then the derivation will be more smooth. \n\nMore details about the minimization of $\\mathcal{L}$ in Eq. (2) should be added. Although one algorithm is presented in Appendix A, there is still no sufficient explanations and analysis (e.g., the convergence). And, there are many hyper-parameters in the algorithm. Their settings and adjustments should be more clear, though there are some brief descriptions at the beginning of Section 4. \n\nThe studies about three trainings settings appreciated. And the presented results indeed provide some insights of these training settings. \n\nThe proposed method reveals many interesting insights of what DNN has learned from the data under different training settings. However, if the insights could be used to improve the current training method or DNNs, the value of this work could be larger. I would like to see some discussions about this point in this manuscript. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose an algorithm that learns sparse patterns of images that are highly predictive of a target class, even if added to a non-target class. The reviewers agree that the algorithm is novel, is tested on a wide array of experiments, and the paper well written.

Unfortunately, it seems that some of the main claims, such as DNNs trained on clean data "learn abstract shapes along with some texture", resort to qualitative evaluation of the few examples shown in the paper. Furthermore, two reviewers were concerned with how one particular design choice in the algorithm might bias the authors' claims. In particular, pointed out that the patterns learned are highly to the initial canvas used, which is not necessarily strongly motivated. 

As these two issues are integral parts of the paper, I hesitate to recommend Acceptance at this point.  That said, the approach looks very promising and I hope the authors continue to pursue this idea.
Decision: Reject