Visualizing the Emergence of Intermediate Visual Patterns in DNNs
Authors: Mingjie Li
Shaobo Wang
Quanshi Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Visualization
Deep Learning

Abstract: This paper proposes a method to visualize the discrimination power of intermediate-layer visual patterns encoded by a DNN. Specifically, we visualize (1) how the DNN gradually learns regional visual patterns in each intermediate layer during the training process, and (2) the effects of the DNN using non-discriminative patterns in low layers to construct disciminative patterns in middle/high layers through the forward propagation. Based on our visualization method, we can quantify knowledge points (i.e. the number of discriminative visual patterns) learned by the DNN to evaluate the representation capacity of the DNN. Furthermore, this method also provides new insights into signal-processing behaviors of existing deep-learning techniques, such as adversarial attacks and knowledge distillation.

Visualizing the Emergence of Intermediate Visual Patterns in DNNs
Mingjie Li Shanghai Jiao Tong University
Shaobo Wang Harbin Institute of Technology 18111c315estu, hit edu en
miinggie en
Quanshi Zhang
Shanghai Jiao Tong University
zqs10228 jtt. edu.cr
Abstract
This paper proposes method to visualize the discrimination power of intermediate- layer visual patterns encoded by DNN. Specifically, we visualize (1) how the DNN gradually learns regional visual patterns in each intermediate layer during the training process, and (2) the effects the DNN using non-diseriminative patterns in low layers to construct disciminative patterns in middle/high layers through the forward propagation. Based on our visualization method, we can quantify knowledge points (i.e. the number of discriminative visual patterns) learned by the DNN to evaluate the representation capacity of the DNN. Furthermore, this method also provides new insights into gegnal-rroesssngg behaviors of existing deep-learning techniques, such as adversarial attacks and knowledge distillation.
Introduction
Deep neural networks (DNNs) have achieved superior performance in various tasks, but the black-box nature of DNNs makes it difficult for people understand its internal behavior. Visualization methods are usually considered as the most direct way to understand the DNN. Recently, several attempts have been made to visualize the DNN from different aspects, e.g. illustrating the visual appearance that maximizes the prediction score of a given category [50] 666| (35. inverting intermediate- features to network inputs [12], extracting receptive fields of neural activations (73), estimating visualizing the sample distribution, such as
saliency/importtnce/ttbiitiiii maps 74 45 75
PCA 388 t-SNE [54]. etc.
In spite of above explanations of the DNN, there still a large gap between visual explanations of the patterns in the DNN and the theoretical analysis of the DNN's discrimination power. In other words, visualization results usually cannot reflect the discrimination power of features in the DNN.
Therefore, instead of simply visualizing the entire sample, we divide intermediate- features into feature components, each which represents a specific image region. We visualize the discrimination power of these feature components, and we consider discriminative feature components as knowledge wlege points learned by the DNN. Based on above methods, we can diagnose the feature representation of pre-trained DNN from the following perspectives.
We visualize the emergence of intermediate visual patterns in cemporal-spatio manner and evaluate their discrimination power. (1) We visualize how the discrimination power of each individual visual
*Corresponding author. This work done under the supervison of Dr. Quanshi Zhang. He is with the John Hopcroft Center and MoE Key Lab Artificial Intelligence, Institute, at the Shanghai Jiao Tong University, China.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

tamporal emergence regional patterss through training process
analysis of regional features' quantity and quality
leration
iteration 10
Ieerat 100
lle lleraton
spatial emergence regionall pattems through forward propagation
rrput image
reliable knowledge point
unreliable knowledge point
ayer
layer 2
layer
aaer
non-discriminative feature
Figure 1: Diagrammatic sketch for the emergence of visual patterns encoded by the DNN in temporal spartal manner. The visualization result enables people analyze the quantity and quality intermediate features.
pattern increases during the learning process. (2) We illustrate effects of using non-discriminative patterns low layers gradually construct discriminative patterns in high layers during the forward propagation. As Figure T shows, the regional feature of the cat head emerges as a discriminative pattern for the cat category, while wall features are non discriminative.
Based on the the visualization result, we can further measure the quantity and quality of intermediate patterns encoded by DNN. In Figure count knowledge points encoded in DNN regional patterns with strong discrimination power, and further evaluate whether each knowledge point is reliable for classification. This provides new perspective analyze the DNN.
A distinct contribution of this study is to bridge the empirical visualization and the quantitative analysis of a DNN's discrimination power. In comparison, Kim et al. (23] used concept activation vectors to model the relationship between visual features and manually annotated semantic concepts. Cheng et al. 77 quantified the number of visual concepts encoded by the DNN. However, these two methods cannot reflect the discrimination power of regional visual concepts. On the other hand, some researchers derived mathematical bounds the representational power of DNN [68] [3] under certain assumptions of the network architecture. To this end, we believe that bridging regional patterns and DNN's discrimination power is convincing and more intuitive way reveal the internal behavior a DNN than mathematical bounds under certain assumptions.
Besides, method provides insightful understanding towards existing deep-learning techniques, such as adversarial attacks and knowledge distillation. (1) For adversarial attacks, we discover that adversarial attacks mainly affect unreliable regional features in high layers of the DNN. The visualization result also enables us to categorize attacking behaviors of all image regions into four types. (2) For knowledge distillation, knowledge points, compared with the teacher DNN. Although knowledge distillation is able to force the student DNN to mimic features of a specific layer in the teacher DNN, there is still big difference of features in other layers between the student DNN and the teacher DNN.
discover that the student DNN usually encodes less reliable
Contributions of this paper can be summarized follows. (1) We propose method to visualize the discrimination power of intermediate-layer features the DNN, and illustrate the emergence of intermediate visual patterns in temporal-spatial manner. (2) Based on the visualization result, we quantify knowledge points encoded by DNN. (3) The proposed method also provides new insights into existing deep-learning techniques, such as the adversarial attack and the knowledge distillation.
2
Related work
Visual explanations for DNNs. Visualization of DNNs is the most direct way to explain visual patterns encoded in the DNN. Some studies reconstructed the network input based on given intermediate-layee feature activations in given feature map 67 Zhou et al. [73) extracted the actual image-resolution receptive
while others aimed generate an input image that cause high

---

field of neural activations in DNNs. Recently, Goh et al. [7 visualized the multi-modal neurons in the CLIP models 39 Kim et al. 23 proposed TCAV to represent manually annotated semantic concepts.
Another type of researches diagnosed and visualized the pixel-wise safiencyimporancc/aribiiiio
on real input images [4
5 What more, the distribution of intermediate- layer features could be visualized via dimensionality reduction methods, e.g. PCA [38], LLE [3]]
32 24
MDS [00, ISOMAP [52), t-SNE [54), etc. Recently, Law et al. [25] visualized the feature of each sample in low-dimensional space by exploiting probability scores of the DNN, while Li et al. [27]
achieved this by assigning each category in the task with a main direction. Sophisticated interfaces
have been built up to visualize the architecture and the knowledge of DNNs [19 60 [3].
Instead of visualizing visual appearance or merely visualizing the distribution of sample features, our method visualizes the emergence of intermediate visual patterns in a temporal-spain manner, which provides a new perspective to explain DNNs.
Theoretical analysis of the representation capacity of DNNs. Formulating and evaluating the discrimination power of DNNs is another direction explain DNNs. The rnformation-bottlenee theory 664 47) provides a generic metric to quantify information encoded in DNNs, which is further exploited to evaluate the representation capacity of DNNs 18 65]. Achille and Soatto M further used the iooommtiion-bttleeee
to improve the feature representation. Furthermore, several metrics were also proposed analyze the robustness or generalization capacity of DNNs. such as the CLEVER score [62], the stiffiness [15], the sensitivity metrics [37], etc. Zhang et al. [69] studied the role of different layers towards the generalization of deep models. Based on this, module criticality f was proposed to analyze DNN's generalization power. Cheng et al. ( quantified visual concepts in input images from the perspective of pixel-wise entropies. Liang et al. (29 diagnosed feature representation between different DNNs via knowledge consistency. Some studies also theoretically
proved the generalization bound for two-layer neural networks [68 13 36) and deep CNNs [31 28).
In fact, our research group led by Dr. Quanshi Zhang have proposed game-theoretic interactions as a new perspective to explain the representation capacity of trained DNNs. The interactions have been used to explain the hierarchical structure of information processing in DNNs 770 72), generalization power [71), the complexity [8] and the aesthetic level [9] of the encoded visual concepts, adversarial robustness 40 (5). and adversarial transferability [57] of DNNs. The interaction was also used to
learn baseline values of Shapley values
41
However, there is still a lack of connections between the explanations of visual concepts and the analysis of DNN's discrimination power. To this end, our method enables people use the discrimination power of local regions to explain the overall discrimination power of the entire DNN.
3 Algorithm
Given a pre-trained DNN, we propose an algorithm to project the feature of an entire sample and features corresponding to different image regions into low-dimensional space, in order to visualize (1) how the discrimination power of a sample feature increases during the learning process, (2) effects of the DNN constructing discriminative features in high layers using non-discriminative features in low layers, and (3) how discriminative regional features gradually emerge during the training process. The visualization result reflects whether the sample feature and each regional feature are biased to incorrect categories, which regional features play crucial role in classification, whether regional feature is shared by multiple categories, etc. Furthermore, the visualization enables people to quantify knowledge points encoded in the DNN, Such analysis can help people explain existing deep-learning techniques, e.g. adversarial attacks and knowledge distillation.
3.1 Radial distribution to analyze the discrimination power of features
Preliminaries: the vMF distribution. Given a pre-trained DNN and an input image IR., let us consider the output feature of specific layer, denoted by f IR.. To analyze the discimination power of f. previous literature usually considered the feature to follow radial distribution of massive pseudo-categories (much more than the real category number) 66 56 25 As Figure rright shows, each category/pseudo-categen
has mean direction He (c 1, 1.... C) in the feature space. The significance of classifying x towards category e is measured by the projection f "ee and cos(f. indicates the similarity between f and category c. For example, a typical case is the sofumax operation,
3

---

plyll) Softmax(W n), W E B The e-th row of W indicates the direction corresponding to the e-th category.
To this end, the von Mises-Fisher (vMF) mixture model 3
20 was proposed to model the radial
distribution, where the e-th mixture component assumes that the likelihood each feature f belonging to category C {1, C'} follows vMF distribution.
p(f) Σ p(y c)) (fly c). Poma (fly = = = Ca(ise) explre- cos(pe: f)l,
(1)
4/2
where Ca(Ke)= (20))/2ra/a is the normalization constant. Actually, the vMF distribution can be
considered as a spherical analogue to the Gaussian distribution on the unit sphere. Me E ER" measures the mean direction of category The increase of 2 0 decreases the variance of f's orientation w.E. the mean direction p.a Please see the supplementary material for more details.
Radial distribution with noise. The vMF distribution assumes that f is a clean feature without noise, which makes the inference of f purely based on its orientation and independent of the strength Ilfll.. However, in real applications. usually contains the meaningful and clean feature f and the meaningless noise (00, of i.e. f = f + classification confidence if the strength of f low. Therefore, we have
The existence of the noise decreases the
pfll c) c) p(e) PAME (5" = c/y = de.
(2)
For simplicity, we can assume that all features f of a specific strength / follow a vMF distribution, because they have similar vulnerabilitiess to noises. Specifically, let f o, 4). where / = fl and = f/l represent the strength and orientation of f. Then, the likelihood of f belonging to category e is given follows (proof in the supplementary material).
p( [o, flyy = c) p(1 = c) po y c, = p(ly = c) posr(olue,
where <l)) increases along with = 1lffl2, and p(ly = c) is the prior distribution of [l for category The variance parameter x(I) is determined based on statistics of all features of the same strength l. We can prove the classification result p(y= clf) is confident when the feature f has a large strength.
3.2 Visualization of the sample-wise discrimination power
In this section, we visualize the discrimination power of the feature f E 1ed of each entire sample x E X. The visualization is supposed illustrate the classification confidence of each feature towards different categories. Therefore, the goal is to learn a linear transformation to project f into a low-dimensional space, ie. 9 = Mf E Rd (d' < d), which ensures that the similarity between each sample feature f and different categories is preserved. The basic idea of learning the linear transformation M is to use the projected feature g for classification, and to force the classification based on g to mimic the classification based on the original feature f. Let E Y = {1, C) denote category. Thus, the objective is to minimize the KL divergence between the classification probability of the DNN P(ylaz and the classification probability Qu (y((y) based on the projected feature g.
min KL ( (Y|x)) + min E
Σ, P(yla) ()) log (y/()
Ply x))
(4)
where Ply|=) is usually computed using a softmax operation. (v)()) is computed by assuming
the distribution p(g) as a mixture model, where each mixture component p(gly) follows a revised vMF distribution in Eq. (3). Let 0g], where I, = l|gl|2 and og = g/l, denote the strength and
orientation of g. Then, we have
( = -, Thy (Talo) Pour (ogli,, x(,)),
(5)
where Ty denotes the prior of the y-th category. We assume the prior of g strength is independent with the category, i.e. p(lgly) = p(()). Then, Qu (wlxx can be measured by the posterior probability p(ylg) in the mixture model, i.e.
Qv((x p(y) p(olv)
Ty PoMa (og ( k(la))
(6)
p(9)
Ey My. Pomar Myy (((g))
The training of the sample-level visualization alternates between the following two steps. (i) Given the current linear transformation M, we update the mixture-model parameters {a. = {Ty, PyEe via the maximium likelihood estimation (MLE) max II, p(g). (ii) Given the current state of [4) we update M to minimize the KI. divergence supplementary material provides more discussions and derivations about the learning process.
in Eq. 4 The

---

3.3 Visualization of the regional discrimination power
In this section, we visualize the discrimination power of features extracted from different regions in each input sample. Let /e S KxXl be an intermediate- of HW regional features for HW positions. Each r-th regional feature is K K-dimensional vector, and is supposed to mainly describe the r-th region in the input, corresponding to the receptive field (region) of f(r). In fact, the actual receptive field of ((+) much smaller than the theoretical receptive field 73. The discrimination power of each regional feature f is analyzed in terms of both the importance and reliability, ie. (1) whether y(r) has a significant impact on the classification, and (2) whether f(r) pushes the classification towards the ground-truti category without significant bias.
feature of a sample which is composed
The visualization of regional discrimination power needs two overcome to challenges. First, we need to formulate and estimate specific importance of different regions for inference during the learning of visualization. Second, the visualization of regional discrimination power is supposed to be aligned with the coordinate system for sample-wiss wise discrimination power.
The goal of the visualization is to project regional features into low- -wwdiimensionll space via linear transformation A, i.e. A = Af E R= (d' < K). Each projected regional feature (()) is supposed to reflect the importance and reliability of the original feature f(r) The strength [|/4*)]22 reflects the importance, and the orientation of h(") represents the reliability of classification towards different categories. Just like t-SNE [54), we use the projected features h- = (hh) : ...., (HW)) to infer the similarity between samples. In this study, the distinet idea of learning A is use the regional similarities (based on h) as the hidden mechanism of mimicking the sample-wise similarity. Let 41, .y E X be two samples, and the probability of æ2 conditioned on ω represents the sample-wist similarity of æ2 to æ1. Then, the objective of learning A is to minimize the KL divergence between the conditional probability P(xu|æ;1 inferred by the DNN and the conditional probability Qx (22(21) inferred by the projected regional features h. In this way, each regional feature ( can well reflect feature representation f(r) used by the DNN. L --Epentsa) (r(ezle=n))
a Olog Q2 aA
Camiuny K XLIP((X
+
0A
P()a(=1) reflects the similarity of 22 to 21 encoded by the DNN, which is computed using DNN's categorical outputs 22, E C We assume 22 follows a vMF distribution with mean direction 21, Lee. P(x2|=1) = 1/2 exp(Np cos(22; =1)], where Z = = exp[*ip cos(22, a1)].
Qx reflects the similarity of 12 to æ1 inferred by the projected regional features ha and h.. Just like the bag-of-words model [11 each projected regional feature h: is assumed to independently contribute to the inference of Qx (halh) to simplify the computation. Furthermore, A((") is weighted
by its importance w/!") > 0, i.e. (x2l(az) x ПI. Qa (hf"") (hi) (r) Just like in [59], a large value of w6 means the r-th region in 22 is important for inference and peaks A 's contribution Q (x (h1).
while a weight w/?" near zero flattens out (x (hi). Details of the estimation of w will be introduced later. In this way, we have
Ölog (1ala))) = =, w 0A
a log Qa(h ( (h)).
(8)
0A
where Q (f. ( that is similar to the regional feature 1.5") in sample r2. Thus, we compute Qu Qx(1(²) (1)) as follows.
(h)) represents the likelihood of the sample æn containing a regional feature 1. E hi,
Qa (h. (b.) - (1 (n(")) =PvMF
(nL p= "==n("),((11"""""" x<(I)(""))) s.t. r'=arg max Qx ( (n"") (9)
Here, we assume A(r²) follows revised vMF distribution in Eq. 3 with mean direction k(-²), where the r'tth region in œ1 is selected as the most similar region to the r-th region in 22.
As is shown above, the loss Liimilarit enables (r) to mimic feature representation of f(r) in terms of encoding the sample-wise similarity. Furthermore, we also expect (r)) to reflect the discrimination power of each regional feature. Therefore, we align the regional features h to the coordinate system of g representing the sample-wise discrimination power, order to represent the regional discrimination power. To this end, we maximize the mutual information between the regional features and the sample features, as the second loss. In other words, this loss enables the discrimination power of regional feature to infer that of the corresponding sample feature.
aa 0A
a ao Qx
Lalign MM((((()))()))
EQa(h.s)
(10)
0A
5

---

bridge
dataset
Tany ImageNet
COCO 2014
CUB-200-2011
DNN
VGG-16
ResNet-34
MobilcNet-22
ResNet-50
ResNei-34
cat
desk
PCA
-0.65
-0.78
-0.81
-0.56
-0.78
dog
I-NNB
-0.50
-0.36
-0.66
-0.67
-0.50
frog
LLE
-0.38
-0.51
-0.27
-0.99
-0.58
ipod
Infebcat
orange
DRPR
-0.87
-0.89
-0.88
.0.7
-0.77
Visualization of sample features Table 1: The negative correlation (1) between the visualized sample learned by VGG- 16 the coordinate sys- features' strength and the samples' classification uncertainty. tem of the Tiny ImageNet categories'.
Figure 2:
The joint probability Qa(h, .) p(g) -1, (h (g) (g) reflects the fitness between the discrim- ination power of sample and that of its compositional regions. Qx (1()))))) reflects the fitmess between the sample feature & and each r-th regional feature h²*,, which is assumed to follow a vMF distribution with mean direction g. i.e. Qa(h"")g) Pome(h"") (g,"") k'). ). In this way. the second loss can be equivalently written as Lalign = -B, w()) cos(g, )), where k'' has been eliminated (proof in the supplementary material).
In sum, the loss functions in Eq. and 10 enable (r) to reflect both the feature representation of ((r) and align (rr) to the coordinate system of g's discrimination power. Thus, we learn A using both losses L = Lamiunit + a Laign (a > 0).
Estimation of each region's 's importance (") In Eq. 8 we need to estimate the importance of
each r-th region as (t) Just like Eq. 7 the objective of estimating w = [w ...., w w. (()))) in each
sample is also formulated as the minimization of the KL divergence between P(z2(=1) inferred by
the DNN and ( (21) inferred by f.
min XL(P((x|Xx))IQ4 (XIIIX))) > min Bay Σ P(x2(z1) log (()2(
(11)
Qu2
we estimate by formulating Qw (xelar) using raw features instead of the projected features h. for more accurate estimation. We assume each regional feature f(² con- tributes independently to (x2la1)) i.e. (x2(21) x II. (t Is)"""" Then, just like Eq. D ((r) [ ). In the quantification of Qw (fL (f"") we further consider the different importance of each channel in f To this end, we further estimate the importance of each channel of fz as = [o ..... 112
Unlike Eq. 8
(f lfi) max,, Q (f1
(K) E RR where (k) R denotes the importance of the k-th
channel. In this way, we quantify Qw " (
follows.
Qu Quo x exp R Σ
U2
(12)
112
11
where #12 and f1 are neural activations of the ketth channel in the r-th region in f2 and fi, respectively. In our experiments, between different samples, we force each element wa and es to be non-negative, and force their Li Li-norm to be 1. L.e. wa > 0, v2 ž 0, ||w2||1 = 1, and ||0z]] = 1. This ensures that magnitudes of region's s/channel's importance stabilize the optimization process of w2 and v2. The optimization process of u2 and alternates between the following two steps. (i) We first update wa and v22 via Eq. (TT) using the gradient descent method. (ii) We force each element in u2 and v2 to be non-negative and force their L -n-oomm to be 1,
and were jointly optimized via Eq. (II). For fair comparison
different samples are similar. Besides, this constraint can also
1w (r)
i.e. the importance of the r-th region uc(") is normalized to
(r 1,... HW), and the importance walll
(n (k
of the k-th channel 02 is normalized to Telli
L,..., K).
3.4 Quantifying knowledge points and the ratio of reliable knowledge points
Visualizing the discrimination power of regional features provides us a new perspective to analyze the representation capacity of a DNN, i.e. counting knowledge points encoded in different layers, and quantifying the ratio of reliable knowledge points. Up now, Cheng et al. [7] was the first attempt to quantify the knowledge points encoded in an intermediate layer using the information theory, but

---

1600
10000
iterationn
Figure 3: The emergence of regional patterns through the learning process in the coordinate system of visualizing the Tiny ImageNet categories
selected reg Ions
regional fratures of selecced regions
selected regions
regional festures Feat sellcctd regians
selected regi
regional festures of selected reglons
features
Pdesk
each layer
conv_22
feed aee
conv_33
conv .43
convy
-2
The emergence of regional patterns through the forward propagation in the coordinate system of visualizing the Tiny ImageNet categories Coordinates along the vertical axis reflect the discrimination power of the target category.
Figure 4:
the knowledge points were extracted based on the discard of the pixel- information, instead of representing the discrimination power of regional features. In comparison, we quantify the total knowledge points and reliable ones, in terms of their discrimination power: Experiments show that the quantity and quality of knowledge points can well explain knowledge distillation in practice.
Specifically, given a regional feature 1,) if (r discriminative enough for classification of any category, i.e. max. p(y ()) then we count this regional feature as a knowledge point. The classification probability p(y = e/h Ee s geeppp
jx ([[A((()) ( (k(() which similar to Eq. (6. Furthermore, among all knowledge points, those pushing the classification towards the correct classification, i.e. knowledge points satisfying cnumh arg maxe ply = ))h''),, are taken as reliable knowledge points. In this way, the ratio of reliable knowledge points is defined as the ratio of reliable knowledge points to total knowledge points, which reflects the quality of visual patterns.
explxxi((] ] cos(k [te]l
Experiments
In this section, we used our method to visualize sample features and regional features in VGG-16 49
ResNet-34/50 (21, MobileNet-V2 [44, which were learned for object classification, based on the Tiny ImageNet dataset (26), the MS COCO 2014 dataset (30], and the CUB-200-2011 dataset (55] For the MS COCO 2014 dataset and the CUB-200-2011 dataset, we used images cropped by the annotated bounding boxes for both training and testing. Note that the analysis of classification for massive categories requires a large number of category directions in the coordinate system, which will hurt the visualization clarity of the radial distribution. Therefore, to clarify the visualization result, we randomly selected 10 categories from each datase Please see the supplementary material for details the DNNs and datasets.
Visualization and verification of sample features' discrimination power. this experiment, we projected sample features f into 3-dimensional space (i.e. d' 3) for visualization. Specifically, we selected the feature before the last fully-connected layer as the sample feature f. Figure 2 shows the projected sample features and each category direction Me. The visualization result revealed the semantic similarity between categories. For example, cul features were similar to dog features, and bus features were similar to lifeboat features. Besides, the supplementary material shows how the discrimination power of sample features gradually increased through the training process.
Furthermore, in order to examine whether g reflected the discrimination power of sample features, we evaluated the Pearson correlation coefficient between the strength ||//|2 and the classification uncertainty of each sample To this end, the classification uncertainty of each sample was measured as the entropy of its output probability, i.e. H(Y = x). In Table I we compared our method with several visualization methods, such as PCA 388 t-SNE [54], LLE [43] ISOMAP (52), and the
For the Tiny ImageNet dataset, we selected steel arch bridge, school bus, sports car, tabby cat, desk, golden retriever, tailed frog. iPod, lifeboat, and orange. Please see the supplementary material for other datasets.
7

---

WP input imagee
dataset
DNN
correlation
VGG-16
0.77772016
Tiny ImageNet
ResNet-34
0.82482005
0.81690013 MobileNet-V2
COCO 2014
ResNer-50
0.7572
CUB-200-2011
ResNet-34
0.7765=0.1
Table The Pearson correlation coef- ficient between ||h**) and w()), The feature strength and feature importance were positively related to each other.
Figure 5: Visualization of the regional importance estimated by our method. Our regional importance similar the Shapley value of f(r), which verifies the trustworthiness of our method.
of knowlede points
comv_33
CORM
40-
37 05% unreliakce
67 ...02% unreliable
54, 12% unrellable
20-
Testiata 62.94%
70-
Taaliabe 32, 32.98%
Frobable 88%
reliable
10000 20000 20000
10000 20000 30000
10000
20000 30000 Iteration
knowledge poiTts briggn calrgory
knowledge points oringe colegory
Ieration
heration
Figure 6: (left) The increase of total knowledge points and reliable knowledge points during training. The ratio of reliable knowledge points, Tistable = # reliable points ## of all points, increases through the forward propagation. (right) Visualization of image regions of knowledge points towards different categorics.
recent DRPR [25]. Compared with baseline methods, the strength of our projected featuree 9 was more strongly correlated to the classification uncertainty. The supplementary material also verified the effectiveness of our method by illustrating contour maps of the classification probability.
Emergence of discriminative regional features. Next, we projected regional features f into 3-dimension space (i.e. d' = 3) to analyze the importance and reliability of each f(r) towards classification. We set = 0.1. Figure 3 shows the emergence of projected regional features h!m) through the training process, when we selected the output feature of the conv_5, _53 layer of VGG-16 as f(r) The ellipsoid represented the estimated Gaussian distribution of A for image regions cropped from each category. The visualization result showed that the discrimination power and reliability of regional features gradually increased during training. Besides, Figure visualizes regional features extracted from diffcrent layers of VGG-16. For clarity, we further selected image regions corresponding to reliable knowledge points in the conv_ 53 layer. Figure 4 visualizes the selected regions, as well as their regional features. It showed that these regions were not discriminative in low layers, but became discriminative in high layers.
Visualization and verification of the estimated regional importance. In this experiment, we estimated regional importance (*) with R set to 1000. The estimated (* was further verified from the following two perspectives. From the first perspective, we compared the estimated regional importance w(" and the Shapley value [46 32] (()) of each r-th region, when we selected the output feature of the conv_53 layer of VGG- 16 as regional features f(r), To this end, the Shaplcy value (10) was computed the numerical contribution of f()) to the DNN output. The Shapley value is the unique unbiased and widely-used 16 63 metric that fairly allocates the numerical contribution to input features, which satisfies the linearity axiom, the dummy axiom, the symmetry axiom, and the efficiency axiom (2]. Figure 5 shows the high similarity between wrr and of among different regions r, which demonstrated the trustworthiness of the estimated regional importance mr))
Besides, we calculated the Pearson correlation coefficient between the strength of projected features [| and their corresponding importance (r) Table 2 shows the mean value and the standard deviation of the Pearson correlation coefficient through all input samples in each dataset, when we used the output feature of the last convolutional layer regional features f. This proved that feature strength and feature importance were significantly and positively related to each other.
Quantifying knowledge points and the ratio of reliable knowledge points. Figure Figure/[left) leftt) shows the increase of knowledge points in different layers through the training of VGG-16. For fair comparison, we normalized the average strength of regional features [=) |l given =] in each layer to the average strength of regional features in the conv_5, layer, and therefore we could simply set = 0.4. Besides, we also computed the ratio of reliable knowledge points in each layer. Tigure[]leff shows that the ratio of reliable knowledge points in high layers (e.g. the conv_ 53 layer) was higher than that in low layers (e.g. the conv_33 333 layer), which demonstrated the increasing quality of visual
8

---

knowledge points the original category before attack knowledge points the target category
ingrona fosuue
1.0
after attack
((r) before
Figure 7: (left) Image regions corresponding knowledge points in original and adversarial samples. (right) We selected important regions for the target category from adversarial samples, and evaluated the selected regions' utilities/mportance of classifying original images to the category. Most important regions after the attack were not so important before the attack.
2
Type
20
Aprtentation Aetrength
etiwenstiall itertion
16
0.2
Pow
12
0.01
32
30
come 43
St
Figure 8: (left) The adversarial attack usually made significant effects regional features in high layers. (right)
Visualization of four types of regional features' trajectories during the attack.
Sllckntt Cniy de warninng wing PC laye
SudertDN collldd ding byer
dilleei onny Silyer
Figure 9: (left) Knowledge distillation caused the DNN to encode less reliable knowledge points. (right) The dissimilarity of features between each student DNN and the teacher DNN, in terms of orientation and strength.
e resional features
30+
enf paits
40
20
layer
patterns through the forward propagation. Besides, Figure F rightt highlights the image regions of knowledge points towards different categories. Regional features in high layers were usually more likely to localized on the foreground than regional features in low layers.
The adversarial attack mainly affected unreliable regional features in high layers. We used our method analyze the change of regional features when we applied the adversarial attack (33] to VGG-16. Given a normal sample x, the adversarial sample Tadv x + was generated via the untargeted PGD attack (33, subject to |o]|0 S 255 The attack was iterated for 20 steps with the step size of 0.1 In Figure leeft,, we found that important regions for the classification of 255
the original image (the first row) were usually different from important regions that attacked the classification towards the target category (the second row). More specifically, let (()) and (() denote two corresponding regional features in the same layer before and after the attack. Let us select important regions {r} for attacking from adversarial samples, satisfying p(v = ea (k(2)) > 0.4. Figure 7 right) illustrates the histogram for the selected regions' classification utilities p(y = (A in the original image. We found that most important regions after the attack were not so important before the attack. Besides, we compared the utility of the attack to regional features in different layers.
Let Aurienation = E., [E, (cos((c)) ((2)) )] and Aumuhh [E, (1)) 1|2 || [a]]] measure the utility of the attack to regional features' orientation and strength. Figure 8 shows that the adversarial attack mainly affected regional features in high layers, e.g. the conv_53 layer in VGG-16. We further categorized all image regions into four types, in terms of their attacking behaviors. To this end, we visualized the trajectories of regional features in the conv_5, during the attack. As Figure 3 right) shows, Type I illustrates important image regions for the dog category that were directly transferred to the cat category without much difficulties. Type 2 illustrates important dog regions, in which dog features were first damaged and then cat features were built up and became important cat regions. Type 3 indicates unimportant dog regions that were pushed to important cat regions. Type indicates important dog regions that were damaged by the attack and became unimportant regions.
The DNN learned via knowledge distillation encoded less reliable visual patterns. In this ex- periment, we learned two student DNNs (two VGG-16 nets) for knowledge distillation [22]. One student DNN was learned by distilling the output feature of the conv_53 layer after the ReLU operation in the teacher DNN (a pre-trained VGG-16) to the corresponding layer the student DNN. The other student DNN was learned by distilling the output feature of the penultimate fully-

---

connected layer after the ReLU operation in the teacher DNN to the corresponding layer in the student DNN. Figure (leftt compares the number all knowledge points and reliable knowledge points encoded by the teacher DNN and the two student DNNs, when we quantified knowledge points in conv_33/comt / conv_5o __3 layers. We found that student DNNs usually encoded less reliable knowledge points than the teacher DNN.
Furthermore, the student DNN usually learned even less reliable concepts in a layer, if this layer was farther from the target layer used distillation. To verify this conclusion, we compared the number of knowledge points between the above two student DNNs. As Figure [eeft) shows, the student DNN distilled using features of the colly-connected layer encoded much less reliable concepts than the student DNN distilled using features of the ccnv_ _5 layer, which verified our conclusion.
Although the knowledge distillation could force the student DNN to well mimic features of specific layer in teacher DNN, there was still big difference of other layers' regional features between the student DNN and the teacher DNN. To this end, we evaluated the quality of student DNNs
nimicking the teacher DNN. We selected (r) and Meneher as two corresponding regional features of the student DNN and the teacher DNN in the same layer. Then, we used cos( studem h tescher) and || || 2 - || llz to measure the difference of orientation and the difference of strength between the two regional features. Figure rrghtt shows the histogram of 1 cos(h M and ( ||2 | Meather ll,, when we used the conv_53 layer to evaluate the similarity between the student DNN and the teacher DNN. The similarity between student DNN features and teacher DNN features was lower when the student DNN was distilled based on features in the fully-connected layer (far from the conv_53 layer), which verified our conclusion.
Conclusion
In this paper, we propose method visualize intermediate visual patterns in a DNN. The vi- sualization illustrates the emergence of intermediate visual patterns in a temporal-spatial manner. The proposed method also enables people measure the quantity and quality of visual patterns encoded by the DNN, which provides a new perspective to analyze the discrimination power of DNNs. Furthermore, the proposed method provides insightful understanding towards the signal-processing behaviors of existing deep-learning techniques.
Acknowledgments and Disclosure of Funding
This work is partially supported by the National Nature Science Foundation of China (No. 61906 U19B2043), Shanghai Natural Science Fundation 2LJC1400800,21121144400 Shanghai Munici- pal Science and Technology Major Project (2021SHZZDZZ00022,
References
[1] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, (1112)):29772905 2018
[2] Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with polynomial time algorithm for shapley values approximation. arXiv: 10992, 2019.
[3] Arindam Banerjee, Inderjit Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Cluster- ing on the unit hypersphere using mises-fisher distributions, Journal of Machine Learning Research, 6(9), 2005.
[4] Niladri S Chatterji, Behnam Neyshabur, and Hanie Sedghi. The intriguing role of module criticality in the generalization of deep networks. arXiv preprint /rXiv:/9/2.000 2019.
[5] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad-cam++: Generalized gradient- visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 839-847. IEEE, 2018
10

---

[6] Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. L-shapley and c-shapley: Efficient model interpretation for structured data. In ICLR, 2019.
[7] Xu Cheng, Zhefan Rao, Yilan Chen, and Quanshi Zhang. Explaining knowledge distillation by quantifying the knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[8] Xu Cheng, Chuntung Chu, Yi Zheng, Jie Ren, and Quanshi Zhang. A game-theoretic taxonomy of visual concepts in dnns. arXiv preprint arXiv:2106. 10938, 2021.
[9] Xu Cheng, Xin Wang, Haotian Xue, Zhengyang Liang, and Quanshi Zhang. A hypothesis for the aesthetic appreciation in neural networks. arXiv preprint arXiv:2108.0280 026.6 2021.
[10] Michael AA Cox and Trevor F Cox. Multidimensional scaling. In Handbook of data visualiza- tion, pages 315-347. Springer, 2008
[11] Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. Visual categorization with bags of keypoints. In Workshop statistical learning in computer vision, ECCV. Prague, 2004.
[12] Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4829-4837, 2016.
[13] Simon S Du, Yining Wang, Xiyu Zhai, Sivaraman Balakrishnan, Ruslan Salakhutdinov, and Aarti Singh. How many samples are needed to estimate convolutional neural network? In NeurIPS, pages 371-381, 2018.
[14] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision, pages 3429-3437, 2017.
[15] Stanislav Fort, Pawel Krzysztof Nowak, and Srini Narayanan. Stiffness: A perspective generalization neural networks. arXiv preprint arXiv:1901.09491, 2019.
[16] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. ICML, 2019.
[17] Gabriel Goh, Nick Cammarata f. Chelsea Voss t. Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021. doi: 10.239 5/distill.
http://distill.pub/000 htwla//mmallllmmuurrrrrta
[18] Ziv Goldfeld, Ewout Van Den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow deep neural networks. In International Conference Machine Learning, pages 2299-23088 2308, 2019.
[19] Adam W Harley, An interactive node-link visualization of convolutional neural networks. In International Symposium Visual Computing, pages 867-877. Springer, 2015.
[20] Md Hasnat, Julien Bohné, Jonathan Milgram, Stéphane Gentric, Liming Chen, et al. von mises-fisher mixture model-based deep learning: Application to face verification. arXiv preprint arXiv: /TG6:/776..444 042644 2017.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[22] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: /503.02531, 2015.
[23] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pages 2668-26888 2676. PMLR, 2018.

---

[24] Pieter-Jan Kindermans. Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller. Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternattribution. arXiv preprint arXiv: arXiv: 1705.05598, 2017.
[25] Mare T Law, Jake Snell, Amir-massoud Farahmand, Raquel Urtasun, and Richard S Zemel. Di- mensionality reduction for representing the knowledge of probabilistic models. In International Conference Learning Representations, 2019.
[26] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7:7, 2015.
[27] Mingwei Li, Zhenge Zhao, and Carlos Scheidegger. Visualizing neural networks with the grand tour. Distill, 2020. doi: 00.223155disstll..omm
[28] Xingguo Li, Junwei Lu, Zhaoran Wang, Jarvis Haupt, and Tuo Zhao. On tighter generalization bound for deep neural networks: Cnns, resnets, and beyond. arXiv preprint 1oX.v:/806.05/59 2018.
[29] Ruofan Liang, Tianlin Li, Longfei Li, and Quanshi Zhang. Knowledge consistency between neural networks and beyond. In International Conference on Learning Representations, 2019.
[30] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[31] Philip M Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks In International Conference on Learning Representations, 2020.
[32] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https:://proceeddnggg neurips http:////oooaaoaooooo/////////// 33220286222233322227334453222223
pdf
[33] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1eXiv:770600600 2017.
[34] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by invert- ing them. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5188-5196, 2015.
[35] Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015. URL [RRLstttt////eeearddat hotp::cchoooooooe.oooooooo/////0 incepticni www.onniiiioooonnooooooo-ooo oog.geig-neeeee---ooooeeeeeeeeee hhtmma..mmmaaammmmmm
[36] Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over-parametrization in generalization of neural networks.
arXiv preprint (ttiv:I805.110111 12076, 2018
[37] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: empirical study. arXiv preprint vrXiv:1802.08760 2018
[38] Karl Pearson. Liii on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):55-577, 1901.
[39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint (tt.:2203300000 2021.
[40] Jie Ren, Die Zhang, Yisen Wang, Lu Chen, Zhanpeng Zhou, Xu Cheng, Xin Wang, Yiting Chen, Jie Shi, and Quanshi Zhang. Game-theoretic understanding of adversarially learned features. arXiv preprint
2021.
12

---

[41] Jie Ren, Zhanpeng Zhou, Qirui Chen, and Quanshi Zhang. Learning baseline values for shapley values. arXiv preprint 1iti:22005.10119 2021.
[42] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pages 133--114. ACM, 2016.
[43] Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290(5500):2323-23288 2000.
[44] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 15100-520, 2018
[45] Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[46] Lloyd S Shapley. A value for n-person games. Contributions the Theory of Games, 2(28): 307-317, 1953.
[47] Ravid Shwartz-ZZi and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 2017.
[48] K Simonyan, A Vedaldi, and A Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint 1312.6034 2017.
[49] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arxivv 1556, 2014.
[50] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6000 13/2.6034, 2013.
[51] Josef Sivic and Andrew Zisserman. Video google: A text retrieval approach to object matching in videos. In IEEE International Conference on Computer Vision, volume 3, pages 1470-1470. IEEE Computer Society, 2003.
[52] Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500) 2000.
[53] Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, et al. The language interpretability tool: Extensible, interactive visualizations and analysis for nlp models. arXiv preprint arXiv:2008.00 05 /22, 2020.
[54] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
[55] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011 1-001, California Institute of Technology, 2011.
[56] Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L.2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia, pages 1041-1049, 2017.
[57] Xin Wang. Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach interpreting and boosting adversarial transfcrability. In International Conference on Learning Representations, 2020.
[58] Xin Wang, Shuyun Lin, Hao Zhang, Yufei Zhu, and Quanshi Zhang. Interpreting attributions and interactions of adversarial attacks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1095-1104, 2021.
13

---

[59] Yixin Wang, Alp Kucukelbir, and David M Blei. Robust probabilistic modeling with bayesian data reweighting. In International Conference on Machine Learning, pages 3646-3655. PMLR, 2017.
[60] Zijie J Wang, Robert Turko, Omar Shaikh, Haekyu Park, Nilaksh Das, Fred Hohman, Minsuk Kahng, and Duen Horng Chau. Cnn explainer: Learning convolutional neural networks with interactive visualization. IEEE Transactions on Visualization and Computer Graphics, 2020.
[61] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pages 499-515. Springer, 2016.
[62] Tsui-Wei Weng, Huan Zhang, Pin-Yu Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsich, and Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. arXiv preprint 1801.
2018.
[63] Brian Williamson and Jean Feng. Efficient nonparametric statistical inference on population feature importance using shapley values. In ICML, 2020.
[64] Natalie Wolchover. New theory cracks open the black box of deep learning. In Quanta Magazine, 2017.
[65] Aolin Xu and Maxim Raginsky. Information-theeretiic analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems, pages 2524-2533, 2017.
[66] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In International Conference on Machine Learning. 2015.
[67] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833 Springer, 2014.
[68] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1X/1:/6/1.003550 2016.
[69] Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? arXiv preprint 19Xiv:990..01996 2019.
[70] Die Zhang, Huilin Zhou, Hao Zhang, Xiaoyi Bao, Da Huo, Ruizhao Chen, Xu Cheng, Mengyue Wu, and Quanshi Zhang. Building interpretable interaction trees for deep nlp models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14328-14337, 2021.
[71] Hao Zhang, Sen Li, YinChao Ma, Mingjie Li, Yichen Xie, and Quanshi Zhang. Interpreting and boosting dropout from game- theorethc view. In International Conference on Learning Representations, 2020.
[72] Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting mul- tivariate shapley interactions dnns. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10877-10886, 2021.
[73] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. In International Conference on Learning Representations, 2015.
[74] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929 2016.
[75] Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. In International Conference on Learning Representations, 2017. 2017.
14


Reviews:
[{"review_id": "QC4ME_rHMJw", "reviewer": "Reviewer_et3m", "summary": "Paper proposes a visualization method for probing into the intermediate visual patterns learned by the DNN layers. The idea is to learn a linear transformation to project the original DNN features into a lower dimensional space for mimicing the classification based on the original features. In other words, the projected low-dim features infer the similarity among samples. Experiments are conducted multiple datasets and DNN architectures.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Experiments reveal that the low dimensional projections preserve semantic similarity. Also, the magnitude of the projections is shown to be correlated (better than baselines and existing works) to the confidence of classification. Similarly, in case of regional features also, they show that both the discriminative ability and reliability improve temporally with training.\n- Regional importance estimated by the proposed approach is shown to be aligned with the Shapeley value (contribution of that region to the DNN o/p).\n- Paper presents multiple use cases of the proposed visualization along with understanding the DNN learnings and verification of the known notions.\n- Proposed approach infers that the adversaries majorly affect the easy to attack (or, unreliable) regional features from the semantic (higher) layers.\n- Similarly, interesting findings are inferred by the paper about the process of knowledge distillation (although the faithfulness of the metrics such as #knowledge points that led to these inferences has not been studied rigorously).\n- In summary, the paper presents a different (and seemingly complex) method for going into the learning aspects of the DNNs. However, it sort of gives a feeling that multiple inferences are not strong novel findings (yet they may be considered for adding reliability on the proposed method). For instance, evolution of discriminative features with training, increase in the discriminative ability of the features (and number of reliable knowledge points) towards the deeper layers, etc. In this regard, authors' attempt to explain phenomena such as adversarial attacks and knowledge distillation helps.\n\nMinor\n\n- line 164: 'needs two overcome to challenges' -> 'needs to overcome two challenges'", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I do not see potential -ve societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~6"}, "extra_text": {}}, {"review_id": "PfasK45vDNu", "reviewer": "Reviewer_uv31", "summary": "This paper quantifies the number of discriminative patterns in DNN to evaluate the network capacity through visualization method, and also provides some insights for adversarial attack and knowledge distillation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#Originality: The method connects the visual explanation and DNN's discriminative power, which has not been well studied yet.\n\n#Quality: This paper addresses an important problem (evaluate network's discriminative power via visual concepts), and provides a new perspective to analyze the discriminative powers via measure the quantity and quality of visual concepts learned by DNN.\n\n#Clarity: The paper is well-written and easy to follow.\n\n#Significance: The results show some insights for later work, it will be interesting to further explore.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In line 295, the paper claims that adv attack mainly affect unreliable regional features in high layers, but lack of full analysis. No experiment shows that the reliable regional features are more robust to attack (or does it mean the reliable region is hard to attack?). Besides, VGG is easy to attack, how is the results for different network (maybe adversarially trained, more robust)? Why these models are hard to attack, do they have less unreliable regional features? I'm also concern about how to apply this method into adversarial attack (sparsely attack the unreliable region) and defense (how to increase the number of reliable regional features).\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Gtolpt_5Uxc", "reviewer": "Reviewer_kQ6q", "summary": "The authors proposed an approach to visualize the regional visual patterns learned by the DNN models during the training. The method shows that (1) adversarial attacks mainly affect the regional features in depth layers. The authors also categorize the adversarial attack impact in four types; and (2) in the distillation scenario, the student learns to extract less reliable knowledge points than the teacher. It is an interesting approach with promising results, quantitative and analysis.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method divides intermediate layer features into feature components, each of which represents a specific image region. Then, the technique visualizes these feature components' discrimination power and considers discriminative feature components as knowledge points learned by the DNN. Moreover, the authors show how the discrimination power of each visual pattern increases during the learning process and illustrate the effect of using non discriminative patterns in low layers to gradually construct discriminative patterns in high layers during the forward propagation.\n\nI have the following recommendations and suggestions on how this paper can be improved:\n\n(Line 45) \"To this end, we believe that bridging regional patterns and a DNN's discrimination power is a more convincing and more intuitive way to reveal the internal behavior of a DNN than mathematical bounds under certain assumptions.\" The authors need to justify this claim because the paper also made some assumptions, such as lines 126-127 and 147-148.\n\n(Line 214) Can the authors show any quantitative evaluation of this claim? Since the work uses the projected features in other scenarios, it is essential to know how accurate it is.\n\n(Line 245) The input features of fully connected layers are 1D vectors, so what is the shape of this sample feature? Is it (N, 1, 1)? Are the authors using the features before some average pooling layer?\n\n(Lines 266)  How is the mapping between input image regions and the reliable knowledge points made? This is an important step to improve the paper's reproducibility.\n\n(Lines 275-279) The presented qualitative analysis of few examples is interesting, but why not present the quantitative analysis for all images of the dataset?\n\nThe quality of the text regions on the Figures should be improved (e.g., Figure 1).\n\nWhat is the statistical relevance of the results in Figure 6?\n\nWhat is the impact of the sample confidence in the ratio of reliable knowledge points (RRKP)? Maybe the samples when the model returns the correct answer with high confidence have a higher RRKP than images with low confidence. It is also valid to consider the distillation scenario because the student model may produce low confidence predictions than the teacher.\n\t\nWhat is the method performance on out-of-distribution (OOD) samples?\n\nSince distillation can be used as adversarial defense [1], what is the relation between the results of the student model and the adversarial attacks? \n\nThere has been little discussion on the limitations of the proposed approach. What are the possible technical limitations? It's not clear how the method deals with noise regions when formulating and estimating the specific importance of different regions. In other words, how accurate is it to assume that all features of a specific region have similar vulnerabilities to noise?\n\t\nRevise the paper to eliminate the typos in the text (e.g., section 3.1 discrimination as \"discimination\")\n\n[1] Papernot, Nicolas, et al. \"Distillation as a defense to adversarial perturbations against deep neural networks.\" 2016 IEEE symposium on security and privacy (SP). IEEE, 2016.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors made no comments about the Limitations And Societal Impact. The authors could address how your research could affect ML applications as possible uses, range of social impacts, and address the positive effects and reduce the downsides of the current approaches.", "review_text": "", "extra_scores": {"ethical_concerns": "I see no problem.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "FO-lYhLxrp", "reviewer": "Reviewer_naDB", "summary": "The authors present an interpretability method that shows the regional visual patterns (ie, knowledge points) that are important for the discriminative power of a network on a given image. They also explore this in the context of model distillation and adversarial attacks\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\nThis paper definitely seems original to me. It is an interesting method that goes beyond just looking at image pieces that are relevant for a given prediction, to try and match this with the discriminative power of the network.\n\n**Quality**\nI believe this is a high quality submission. The methods are sound (if a bit convoluted-- see clarity section below), and the analysis of model distillation and adversarial attacks is really compelling\n\n\n**Clarity**\nMost of the paper is clear, but I found section 3 a bit dense and difficult to understand. I know part of this is inherent since this section is math-heavy, but it could also really benefit from another diagram similar to figure 1 or figure 3 to just explain at a high level what the intuition is for the algorithm, before deeply diving into the derivations.\n\nAlso, I'm still a bit confused about algorithm and want to make sure I'm understanding it correctly. Is this correct?\n- Chunk the image into pieces (ie, geographic regions-- might be a cat ear, section of the wall, etc)\n- Run each of these image pieces through the CNN to get its embedding at each layer\n- Get the dot product of the image piece embeddings with feature space directions for each class\n- For these dot products, high value = high discriminative power = knowledge point\n- To make visualization, show the regions that had high dot products\n\nIf this is correct, section 3 probably doesn't need to be as long (parts could be moved to the appendix). That being said, I also definitely might be missing something more complicated about the method!\n\nAlso, one typo:\n>while others aimed to generate an input image that cause high\n(cause => causes)\n\n**Significance**\nThis is an interesting and novel take on visualizing image regions that are relevant to DNN predictions.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "45 min"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new method for examining the patterns learned by intermediate layers of a DNN, by projecting features into a lower dimensional space that preserves discriminative structure. The reviewers were consistently positive about the paper’s novelty and impact, though some concerns were raised regarding clarity. The authors have done a good job of satisfying reviewer concerns in their rebuttals, and I trust that the authors can incorporate these clarifications into the final version without radically reshaping the paper. All in all, interesting and exciting work.
Decision: Accept (Poster)