Visualizing the Emergence of Intermediate Visual Patterns in DNNs
Year: 2022

Abstract:
This paper proposes a method to visualize the discrimination power of intermediatelayer visual patterns encoded by a DNN. Specifically, we visualize (1) how the DNN gradually learns regional visual patterns in each intermediate layer during the training process, and (2) the effects of the DNN using non-discriminative patterns in low layers to construct disciminative patterns in middle/high layers through the forward propagation. Based on our visualization method, we can quantify knowledge points (i.e. the number of discriminative visual patterns) learned by the DNN to evaluate the representation capacity of the DNN. Furthermore, this method also provides new insights into signal-processing behaviors of existing deep-learning techniques, such as adversarial attacks and knowledge distillation.

heading: 1 Introduction
text: Deep neural networks (DNNs) have achieved superior performance in various tasks, but the black-box nature of DNNs makes it difficult for people to understand its internal behavior. Visualization methods are usually considered as the most direct way to understand the DNN. Recently, several attempts have been made to visualize the DNN from different aspects, e.g. illustrating the visual appearance that maximizes the prediction score of a given category [50, 66, 35], inverting intermediate-layer features to network inputs [12], extracting receptive fields of neural activations [73], estimating saliency/importance/attribution maps [74, 45, 75, 32], visualizing the sample distribution, such as PCA [38], t-SNE [54], etc.
In spite of above explanations of the DNN, there is still a large gap between visual explanations of the patterns in the DNN and the theoretical analysis of the DNN’s discrimination power. In other words, visualization results usually cannot reflect the discrimination power of features in the DNN.
Therefore, instead of simply visualizing the entire sample, we divide intermediate-layer features into feature components, each of which represents a specific image region. We visualize the discrimination power of these feature components, and we consider discriminative feature components as knowledge points learned by the DNN. Based on above methods, we can diagnose the feature representation of a pre-trained DNN from the following perspectives.
•We visualize the emergence of intermediate visual patterns in a temporal-spatial manner and evaluate their discrimination power. (1) We visualize how the discrimination power of each individual visual ∗Corresponding author. This work was done under the supervison of Dr. Quanshi Zhang. He is with the John Hopcroft Center and the MoE Key Lab of Artificial Intelligence, AI Institute, at the Shanghai Jiao Tong University, China.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
pattern increases during the learning process. (2) We illustrate effects of using non-discriminative patterns in low layers to gradually construct discriminative patterns in high layers during the forward propagation. As Figure 1 shows, the regional feature of the cat head emerges as a discriminative pattern for the cat category, while wall features are non-discriminative.
• Based on the the visualization result, we can further measure the quantity and quality of intermediate patterns encoded by a DNN. In Figure 1, we count knowledge points encoded in a DNN as regional patterns with strong discrimination power, and further evaluate whether each knowledge point is reliable for classification. This provides a new perspective to analyze the DNN.
A distinct contribution of this study is to bridge the empirical visualization and the quantitative analysis of a DNN’s discrimination power. In comparison, Kim et al. [23] used concept activation vectors to model the relationship between visual features and manually annotated semantic concepts. Cheng et al. [7] quantified the number of visual concepts encoded by the DNN. However, these two methods cannot reflect the discrimination power of regional visual concepts. On the other hand, some researchers derived mathematical bounds on the representational power of a DNN [68, 13] under certain assumptions of the network architecture. To this end, we believe that bridging regional patterns and a DNN’s discrimination power is a more convincing and more intuitive way to reveal the internal behavior of a DNN than mathematical bounds under certain assumptions.
Besides, our method provides insightful understanding towards existing deep-learning techniques, such as adversarial attacks and knowledge distillation. (1) For adversarial attacks, we discover that adversarial attacks mainly affect unreliable regional features in high layers of the DNN. The visualization result also enables us to categorize attacking behaviors of all image regions into four types. (2) For knowledge distillation, we discover that the student DNN usually encodes less reliable knowledge points, compared with the teacher DNN. Although knowledge distillation is able to force the student DNN to mimic features of a specific layer in the teacher DNN, there is still a big difference of features in other layers between the student DNN and the teacher DNN.
Contributions of this paper can be summarized as follows. (1) We propose a method to visualize the discrimination power of intermediate-layer features in the DNN, and illustrate the emergence of intermediate visual patterns in a temporal-spatial manner. (2) Based on the visualization result, we quantify knowledge points encoded by a DNN. (3) The proposed method also provides new insights into existing deep-learning techniques, such as the adversarial attack and the knowledge distillation.

heading: 2 Related work
text: Visual explanations for DNNs. Visualization of DNNs is the most direct way to explain visual patterns encoded in the DNN. Some studies reconstructed the network input based on a given intermediate-layer feature [34, 48, 12], while others aimed to generate an input image that cause high activations in a given feature map [67]. Zhou et al. [73] extracted the actual image-resolution receptive
field of neural activations in DNNs. Recently, Goh et al. [17] visualized the multi-modal neurons in the CLIP models [39]. Kim et al. [23] proposed TCAV to represent manually annotated semantic concepts. Another type of researches diagnosed and visualized the pixel-wise saliency/importance/attribution on real input images [42, 32, 24, 75, 14, 74, 5]. What is more, the distribution of intermediatelayer features could be visualized via dimensionality reduction methods, e.g. PCA [38], LLE [43], MDS [10], ISOMAP [52], t-SNE [54], etc. Recently, Law et al. [25] visualized the feature of each sample in a low-dimensional space by exploiting probability scores of the DNN, while Li et al. [27] achieved this by assigning each category in the task with a main direction. Sophisticated interfaces have been built up to visualize the architecture and the knowledge of DNNs [19, 60, 53].
Instead of visualizing visual appearance or merely visualizing the distribution of sample features, our method visualizes the emergence of intermediate visual patterns in a temporal-spatial manner, which provides a new perspective to explain DNNs.
Theoretical analysis of the representation capacity of DNNs. Formulating and evaluating the discrimination power of DNNs is another direction to explain DNNs. The information-bottleneck theory [64, 47] provides a generic metric to quantify information encoded in DNNs, which is further exploited to evaluate the representation capacity of DNNs [18, 65]. Achille and Soatto [1] further used the information-bottleneck to improve the feature representation. Furthermore, several metrics were also proposed to analyze the robustness or generalization capacity of DNNs, such as the CLEVER score [62], the stiffiness [15], the sensitivity metrics [37], etc. Zhang et al. [69] studied the role of different layers towards the generalization of deep models. Based on this, module criticality [4] was proposed to analyze a DNN’s generalization power. Cheng et al. [7] quantified visual concepts in input images from the perspective of pixel-wise entropies. Liang et al. [29] diagnosed feature representation between different DNNs via knowledge consistency. Some studies also theoretically proved the generalization bound for two-layer neural networks [68, 13, 36] and deep CNNs [31, 28].
In fact, our research group led by Dr. Quanshi Zhang have proposed game-theoretic interactions as a new perspective to explain the representation capacity of trained DNNs. The interactions have been used to explain the hierarchical structure of information processing in DNNs [70, 72], generalization power [71], the complexity [8] and the aesthetic level [9] of the encoded visual concepts, adversarial robustness [40, 58], and adversarial transferability [57] of DNNs. The interaction was also used to learn baseline values of Shapley values [41].
However, there is still a lack of connections between the explanations of visual concepts and the analysis of a DNN’s discrimination power. To this end, our method enables people to use the discrimination power of local regions to explain the overall discrimination power of the entire DNN.

heading: 3 Algorithm
text: Given a pre-trained DNN, we propose an algorithm to project the feature of an entire sample and features corresponding to different image regions into a low-dimensional space, in order to visualize (1) how the discrimination power of a sample feature increases during the learning process, (2) effects of the DNN constructing discriminative features in high layers using non-discriminative features in low layers, and (3) how discriminative regional features gradually emerge during the training process. The visualization result reflects whether the sample feature and each regional feature are biased to incorrect categories, which regional features play a crucial role in classification, whether a regional feature is shared by multiple categories, etc. Furthermore, the visualization enables people to quantify knowledge points encoded in the DNN. Such analysis can help people explain existing deep-learning techniques, e.g. adversarial attacks and knowledge distillation.

heading: 3.1 Radial distribution to analyze the discrimination power of features
text: Preliminaries: the vMF distribution. Given a pre-trained DNN and an input image x ∈ Rn, let us consider the output feature of a specific layer, denoted by f ∈ Rd. To analyze the discimination power of f , previous literature usually considered the feature to follow a radial distribution of massive pseudo-categories (much more than the real category number) [61, 56, 25]. As Figure 1(right) shows, each category/pseudo-category c has a mean direction µc (c = 1, ..., C) in the feature space. The significance of classifying x towards category c is measured by the projection f>µc, and cos(f, µc) indicates the similarity between f and category c. For example, a typical case is the softmax operation,
p(y|f) = Softmax(Wf), W ∈ RC×d. The c-th row of W indicates the direction corresponding to the c-th category.
To this end, the von Mises-Fisher (vMF) mixture model [3, 20] was proposed to model the radial distribution, where the c-th mixture component assumes that the likelihood of each feature f belonging to category c ∈ {1, ..., C} follows a vMF distribution.
p(f) = ∑
c p(y = c)pvMF(f |y = c), pvMF(f |y = c) = Cd(κc) · exp[κc · cos(µc, f)], (1)
where Cd(κc) = κ d/2−1 c
(2π)d/2Id/2−1(κc) is the normalization constant. Actually, the vMF distribution can be
considered as a spherical analogue to the Gaussian distribution on the unit sphere. µc ∈ Rd measures the mean direction of category c. The increase of κc ≥ 0 decreases the variance of f’s orientation w.r.t. the mean direction µc. Please see the supplementary material for more details.
Radial distribution with noise. The vMF distribution assumes that f is a clean feature without noise, which makes the inference of f purely based on its orientation and independent of the strength ‖f‖2. However, in real applications, f usually contains the meaningful and clean feature f? and the meaningless noise ∼ N (0, σ2Id), i.e. f = f? + . The existence of the noise decreases the classification confidence if the strength of f is low. Therefore, we have
p(f |y = c) = ∫ p( ) · pvMF ( f? = f − |y = c ) d . (2)
For simplicity, we can assume that all features f of a specific strength l follow a vMF distribution, because they have similar vulnerabilities to noises. Specifically, let f = [o, l], where l = ‖f‖2 and o = f/l represent the strength and orientation of f . Then, the likelihood of f belonging to category c is given as follows (proof in the supplementary material).
p(f = [o, l]|y = c) = p(l|y = c) · pvMF(o|y = c, l) = p(l|y = c) · pvMF(o|µc, κ(l)), (3)
where κ(l) increases along with l = ‖f‖2, and p(l|y = c) is the prior distribution of ‖f‖2 for category c. The variance parameter κ(l) is determined based on statistics of all features of the same strength l. We can prove the classification result p(y=c|f) is confident when the feature f has a large strength.

heading: 3.2 Visualization of the sample-wise discrimination power
text: In this section, we visualize the discrimination power of the feature f ∈ Rd of each entire sample x ∈ X. The visualization is supposed to illustrate the classification confidence of each feature towards different categories. Therefore, the goal is to learn a linear transformation to project f into a low-dimensional space, i.e. g = Mf ∈ Rd ′ (d′ d), which ensures that the similarity between each sample feature f and different categories is preserved. The basic idea of learning the linear transformation M is to use the projected feature g for classification, and to force the classification based on g to mimic the classification based on the original feature f . Let y ∈ Y = {1, ..., C} denote a category. Thus, the objective is to minimize the KL divergence between the classification probability of the DNN P (y|x) and the classification probability QM (y|x) based on the projected feature g.
min M KL [P (Y |X)‖QM (Y |X)]⇒ min M
Ex [∑
y P (y|x) log P (y|x)
QM (y|x)
] , (4)
where P (y|x) is usually computed using a softmax operation. QM (y|x) is computed by assuming the distribution p(g) as a mixture model, where each mixture component p(g|y) follows a revised vMF distribution in Eq. (3). Let g = [lg, og], where lg = ‖g‖2 and og = g/lg denote the strength and orientation of g. Then, we have
p (g) = ∑
y πy · p (lg|y) · pvMF (og|µy, κ(lg)) , (5)
where πy denotes the prior of the y-th category. We assume the prior of g’s strength is independent with the category, i.e. p(lg|y) = p(lg). Then, QM (y|x) can be measured by the posterior probability p(y|g) in the mixture model, i.e.
QM (y|x) = p(y) · p(g|y)
p(g) = πy · pvMF(og|µy, κ(lg))∑ y′ πy′ · pvMF(og|µy′ , κ(lg)) . (6)
The training of the sample-level visualization alternates between the following two steps. (i) Given the current linear transformation M , we update the mixture-model parameters {π, µ} = {πy, µy}y∈Y via the maximium likelihood estimation (MLE) max{π,µ} ∏ g p(g). (ii) Given the current state of {π, µ}, we update M to minimize the KL divergence KL(P (Y |X)‖QM (Y |X)) in Eq. (4). The supplementary material provides more discussions and derivations about the learning process.

heading: 3.3 Visualization of the regional discrimination power
text: In this section, we visualize the discrimination power of features extracted from different regions in each input sample. Let f∈RK×H×W be an intermediate-layer feature of a sample x, which is composed of HW regional features for HW positions. Each r-th regional feature is a K-dimensional vector, and is supposed to mainly describe the r-th region in the input, corresponding to the receptive field (region) of f (r). In fact, the actual receptive field of f (r) is much smaller than the theoretical receptive field [73]. The discrimination power of each regional feature f (r) is analyzed in terms of both the importance and reliability, i.e. (1) whether f (r) has a significant impact on the classification, and (2) whether f (r) pushes the classification towards the ground-truth category without significant bias.
The visualization of regional discrimination power needs two overcome to challenges. First, we need to formulate and estimate specific importance of different regions for inference during the learning of visualization. Second, the visualization of regional discrimination power is supposed to be aligned with the coordinate system for sample-wise discrimination power.
The goal of the visualization is to project regional features into a low-dimensional space via a linear transformation Λ, i.e. h(r) = Λf (r) ∈ Rd ′ (d′ K). Each projected regional feature h(r) is supposed to reflect the importance and reliability of the original feature f (r). The strength ‖h(r)‖2 reflects the importance, and the orientation of h(r) represents the reliability of classification towards different categories. Just like t-SNE [54], we use the projected features h = {h(1), ..., h(HW )} to infer the similarity between samples. In this study, the distinct idea of learning Λ is to use the regional similarities (based on h) as the hidden mechanism of mimicking the sample-wise similarity. Let x1, x2 ∈ X be two samples, and the probability of x2 conditioned on x1 represents the sample-wise similarity of x2 to x1. Then, the objective of learning Λ is to minimize the KL divergence between the conditional probability P (x2|x1) inferred by the DNN and the conditional probability QΛ(x2|x1) inferred by the projected regional features h. In this way, each regional feature h(r) can well reflect feature representation f (r) used by the DNN.
Lsimilarity =KL[P (X2|X1)‖QΛ(X2|X1)]⇒ ∂Lsimilarity ∂Λ =−Epdata(x1) [ EP (x2|x=x1) ∂ logQΛ(x2|x1) ∂Λ ] (7)
P (x2|x1) reflects the similarity of x2 to x1 encoded by the DNN, which is computed using DNN’s categorical outputs z2, z1 ∈ RC . We assume z2 follows a vMF distribution with mean direction z1, i.e. P (x2|x1) = 1Z exp[κp · cos(z2, z1)], where Z = ∑ x2 exp[κp · cos(z2, z1)].
QΛ(x2|x1) reflects the similarity of x2 to x1 inferred by the projected regional features h2 and h1. Just like the bag-of-words model [51, 11], each projected regional feature h(r)2 is assumed to independently contribute to the inference of QΛ(h2|h1) to simplify the computation. Furthermore, h(r)2 is weighted by its importance w(r)2 > 0, i.e. QΛ(x2|x1) ∝ ∏ r QΛ(h (r) 2 |h1)w (r) 2 . Just like in [59], a large value of w (r) 2 means the r-th region in x2 is important for inference and peaks h (r) 2 ’s contribution QΛ(h (r) 2 |h1), while a weight w(r)2 near zero flattens out QΛ(h (r) 2 |h1). Details of the estimation of w (r) 2 will be introduced later. In this way, we have ∂ logQΛ(x2|x1)
∂Λ = ∑ r w (r) 2 ∂ logQΛ(h (r) 2 |h1) ∂Λ , (8)
where QΛ(h(r)2 |h1) represents the likelihood of the sample x1 containing a regional feature h (r′) 1 ∈ h1,
that is similar to the regional feature h(r)2 in sample x2. Thus, we compute QΛ(h (r) 2 |h1) as follows.
QΛ(h (r) 2 |h1)=QΛ(h (r) 2 |h (r′) 1 )=pvMF ( h (r) 2 ∣∣∣µ=h(r′)1 , κ(‖h(r)2 ‖)), s.t. r′=arg max r′ QΛ(h (r) 2 |h (r′) 1 ) (9)
Here, we assume h(r)2 follows a revised vMF distribution in Eq. (3) with mean direction h (r′) 1 , where
the r′-th region in x1 is selected as the most similar region to the r-th region in x2.
As is shown above, the loss Lsimilarity enables h(r) to mimic feature representation of f (r) in terms of encoding the sample-wise similarity. Furthermore, we also expect h(r) to reflect the discrimination power of each regional feature. Therefore, we align the regional features h to the coordinate system of g representing the sample-wise discrimination power, in order to represent the regional discrimination power. To this end, we maximize the mutual information between the regional features and the sample features, as the second loss. In other words, this loss enables the discrimination power of regional feature to infer that of the corresponding sample feature.
Lalign = −MI(h(X); g(X))⇒ ∂Lalign ∂Λ = −EQΛ(h,g) [ ∂ logQΛ(h|g) ∂Λ ] (10)
The joint probability QΛ(h, g) = p(g) ·
∏ r QΛ(h (r)|g)w (r)
reflects the fitness between the discrimination power of a sample and that of its compositional regions. QΛ(h(r)|g) reflects the fitness between the sample feature g and each r-th regional feature h(r), which is assumed to follow a vMF distribution with mean direction g, i.e. QΛ(h(r)|g) = pvMF(h(r)|g, κ′). In this way, the second loss can be equivalently written as Lalign = −Ex[ ∑ r w
(r) · cos(g, h(r))], where κ′ has been eliminated (proof in the supplementary material).
In sum, the loss functions in Eq. (7) and (10) enable h(r) to reflect both the feature representation of f (r) and align h(r) to the coordinate system of g’s discrimination power. Thus, we learn Λ using both losses L = Lsimilarity + α · Lalign (α > 0).
Estimation of each region’s importance w(r). In Eq. (8), we need to estimate the importance of each r-th region as w(r). Just like Eq. (7), the objective of estimating w = [w(1), ..., w(HW )] in each sample is also formulated as the minimization of the KL divergence between P (x2|x1) inferred by the DNN and Qw(x2|x1) inferred by f .
min w KL(P (X2|X1)‖Qw(X2‖X1))⇒ min w
Ex1 [∑
x2 P (x2|x1) log P (x2|x1) Qw(x2|x1)
] (11)
Unlike Eq. (8), we estimate w by formulating Qw(x2|x1) using raw features f , instead of the projected features h, for more accurate estimation. We assume each regional feature f (r)2 contributes independently to Qw(x2|x1), i.e. Qw(x2|x1) ∝ ∏ r Qw(f (r) 2 |f1)w (r) 2 . Then, just like Eq. (9), Qw(f (r) 2 |f1) = maxr′ Qw(f (r) 2 |f (r′) 1 ). In the quantification of Qw(f (r) 2 |f (r′) 1 ), we further consider the different importance of each channel in f (r)2 . To this end, we further estimate the importance of each channel of f2 as v2 = [v(1)2 , ..., v (K) 2 ] ∈ RK , where v (k) 2 ∈ R denotes the importance of the k-th channel. In this way, we quantify Qw(f (r)2 |f (r′) 1 ) as follows.
Qw(f (r) 2 |f (r′) 1 ) ∝ exp [ κ′ · ∑ k ( v (k) 2 · f (r) 2,k ‖f (r)2 ‖2 · f (r) 1,k ‖f (r)1 ‖2 )] , (12)
where f (r)2,k and f (r) 1,k are neural activations of the k-th channel in the r-th region in f2 and f1, respectively. In our experiments, w2 and v2 were jointly optimized via Eq. (11). For fair comparison between different samples, we force each element in w2 and v2 to be non-negative, and force their L1-norm to be 1. I.e. w2 0, v2 0, ‖w2‖1 = 1, and ‖v2‖1 = 1. This ensures that magnitudes of region’s/channel’s importance in different samples are similar. Besides, this constraint can also stabilize the optimization process of w2 and v2. The optimization process of w2 and v2 alternates between the following two steps. (i) We first update w2 and v2 via Eq. (11) using the gradient descent method. (ii) We force each element in w2 and v2 to be non-negative and force their L1-norm to be 1, i.e. the importance of the r-th region w(r)2 is normalized to |w(r)2 | ‖w2‖1 (r = 1, ..., HW ), and the importance of the k-th channel v(k)2 is normalized to |v(k)2 | ‖v2‖1 (k = 1, ...,K).

heading: 3.4 Quantifying knowledge points and the ratio of reliable knowledge points
text: Visualizing the discrimination power of regional features provides us a new perspective to analyze the representation capacity of a DNN, i.e. counting knowledge points encoded in different layers, and quantifying the ratio of reliable knowledge points. Up to now, Cheng et al. [7] was the first attempt to quantify the knowledge points encoded in an intermediate layer using the information theory, but
the knowledge points were extracted based on the discard of the pixel-wise information, instead of representing the discrimination power of regional features. In comparison, we quantify the total knowledge points and reliable ones, in terms of their discrimination power. Experiments show that the quantity and quality of knowledge points can well explain knowledge distillation in practice.
Specifically, given a regional feature h(r), if h(r) is discriminative enough for classification of any category, i.e. maxc p(y = c|h(r)) > τ , then we count this regional feature as a knowledge point. The classification probability p(y = c|h(r)) = πc·exp[κ(‖h
(r)‖2)·cos(h(r),µc)]∑ c′ πc′ ·exp[κ(‖h(r)‖2)·cos(h(r),µc′ )]
, which is similar to Eq. (6). Furthermore, among all knowledge points, those pushing the classification towards the correct classification, i.e. knowledge points satisfying ctruth = arg maxc p(y = c|h(r)), are taken as reliable knowledge points. In this way, the ratio of reliable knowledge points is defined as the ratio of reliable knowledge points to total knowledge points, which reflects the quality of visual patterns.

heading: 4 Experiments
text: In this section, we used our method to visualize sample features and regional features in VGG-16 [49], ResNet-34/50 [21], MobileNet-V2 [44], which were learned for object classification, based on the Tiny ImageNet dataset [26], the MS COCO 2014 dataset [30], and the CUB-200-2011 dataset [55]. For the MS COCO 2014 dataset and the CUB-200-2011 dataset, we used images cropped by the annotated bounding boxes for both training and testing. Note that the analysis of classification for massive categories requires a large number of category directions in the coordinate system, which will hurt the visualization clarity of the radial distribution. Therefore, to clarify the visualization result, we randomly selected 10 categories from each dataset2. Please see the supplementary material for details on the DNNs and datasets.
Visualization and verification of sample features’ discrimination power. In this experiment, we projected sample features f into a 3-dimensional space (i.e. d′= 3) for visualization. Specifically, we selected the feature before the last fully-connected layer as the sample feature f . Figure 2 shows the projected sample features g and each category direction µc. The visualization result revealed the semantic similarity between categories. For example, cat features were similar to dog features, and bus features were similar to lifeboat features. Besides, the supplementary material shows how the discrimination power of sample features gradually increased through the training process.
Furthermore, in order to examine whether g reflected the discrimination power of sample features, we evaluated the Pearson correlation coefficient between the strength ‖g‖2 and the classification uncertainty of each sample x. To this end, the classification uncertainty of each sample xwas measured as the entropy of its output probability, i.e. H(Y |X = x). In Table 1, we compared our method with several visualization methods, such as PCA [38], t-SNE [54], LLE [43], ISOMAP [52], and the
2For the Tiny ImageNet dataset, we selected steel arch bridge, school bus, sports car, tabby cat, desk, golden retriever, tailed frog, iPod, lifeboat, and orange. Please see the supplementary material for other datasets.
recent DRPR [25]. Compared with baseline methods, the strength of our projected sample features g was more strongly correlated to the classification uncertainty. The supplementary material also verified the effectiveness of our method by illustrating contour maps of the classification probability.
Emergence of discriminative regional features. Next, we projected regional features f into a 3-dimension space (i.e. d′ = 3) to analyze the importance and reliability of each f (r) towards classification. We set α = 0.1. Figure 3 shows the emergence of projected regional features h(r) through the training process, when we selected the output feature of the conv_53 layer of VGG-16 as f (r). The ellipsoid represented the estimated Gaussian distribution of h(r) for image regions cropped from each category. The visualization result showed that the discrimination power and reliability of regional features gradually increased during training. Besides, Figure 4 visualizes regional features extracted from different layers of VGG-16. For clarity, we further selected image regions corresponding to reliable knowledge points in the conv_53 layer. Figure 4 visualizes the selected regions, as well as their regional features. It showed that these regions were not discriminative in low layers, but became discriminative in high layers.
Visualization and verification of the estimated regional importance. In this experiment, we estimated regional importance w(r) with κ̃ set to 1000. The estimated w(r) was further verified from the following two perspectives. From the first perspective, we compared the estimated regional importance w(r) and the Shapley value [46, 32] φ(r) of each r-th region, when we selected the output feature of the conv_53 layer of VGG-16 as regional features f (r). To this end, the Shapley value φ(r) was computed as the numerical contribution of f (r) to the DNN output. The Shapley value is the unique unbiased and widely-used [6, 16, 63] metric that fairly allocates the numerical contribution to input features, which satisfies the linearity axiom, the dummy axiom, the symmetry axiom, and the efficiency axiom [2]. Figure 5 shows the high similarity between w(r) and φ(r) among different regions r, which demonstrated the trustworthiness of the estimated regional importance w(r).
Besides, we calculated the Pearson correlation coefficient between the strength of projected features ‖h(r)‖2 and their corresponding importance w(r). Table 2 shows the mean value and the standard deviation of the Pearson correlation coefficient through all input samples in each dataset, when we used the output feature of the last convolutional layer as regional features f . This proved that feature strength and feature importance were significantly and positively related to each other.
Quantifying knowledge points and the ratio of reliable knowledge points. Figure 6(left) shows the increase of knowledge points in different layers through the training of VGG-16. For fair comparison, we normalized the average strength of regional features Ex,r[‖h(r)‖2 given x] in each layer to the average strength of regional features in the conv_53 layer, and therefore we could simply set τ = 0.4. Besides, we also computed the ratio of reliable knowledge points in each layer. Figure 6(left) shows that the ratio of reliable knowledge points in high layers (e.g. the conv_53 layer) was higher than that in low layers (e.g. the conv_33 layer), which demonstrated the increasing quality of visual
10
20
30
40 total
reliable
# o
f k n o w
le d g e p
o in
ts
conv_33 conv_43 conv_53 layer
# o
f re
g io
n a l fe
a tu
re s
100
102
104
1 − cos ℎdistill 𝑟 , ℎoriginal 𝑟 0.0 0.5 1.0 1.5 2.0 -3 -2 -1 0 1 2 ℎdistill 𝑟 2 − ℎoriginal 𝑟 2
teacher DNN student DNN distilled using FC layer student DNN distilled using conv_53 layer student DNN distilled using conv_53 layer student DNN distilled using FC layer Figure 9: (left) Knowledge distillation caused the DNN to encode less reliable knowledge points. (right) The dissimilarity of features between each student DNN and the teacher DNN, in terms of orientation and strength.
patterns through the forward propagation. Besides, Figure 6(right) highlights the image regions of knowledge points towards different categories. Regional features in high layers were usually more likely to be localized on the foreground than regional features in low layers.
The adversarial attack mainly affected unreliable regional features in high layers. We used our method to analyze the change of regional features when we applied the adversarial attack [33] to VGG-16. Given a normal sample x, the adversarial sample xadv = x + δ was generated via the untargeted PGD attack [33], subject to ‖δ‖∞ ≤ 1255 . The attack was iterated for 20 steps with the step size of 0.1
255 . In Figure 7(left), we found that important regions for the classification of
the original image (the first row) were usually different from important regions that attacked the classification towards the target category (the second row). More specifically, let h(r)ori and h (r) adv denote two corresponding regional features in the same layer before and after the attack. Let us select important regions {r} for attacking from adversarial samples, satisfying p(y = cadv|h(r)adv ) > 0.4. Figure 7(right) illustrates the histogram for the selected regions’ classification utilities p(y = cori|h(r)ori ) in the original image. We found that most important regions after the attack were not so important before the attack. Besides, we compared the utility of the attack to regional features in different layers. Let ∆orientation = Ex[Er(cos(h(r)ori , h (r) adv ))] and ∆strength = Ex[Er(|‖h (r) ori ‖2 − ‖h (r) adv‖2|)] measure the utility of the attack to regional features’ orientation and strength. Figure 8 shows that the adversarial attack mainly affected regional features in high layers, e.g. the conv_53 layer in VGG-16. We further categorized all image regions into four types, in terms of their attacking behaviors. To this end, we visualized the trajectories of regional features in the conv_53 during the attack. As Figure 8(right) shows, Type 1 illustrates important image regions for the dog category that were directly transferred to the cat category without much difficulties. Type 2 illustrates important dog regions, in which dog features were first damaged and then cat features were built up and became important cat regions. Type 3 indicates unimportant dog regions that were pushed to important cat regions. Type 4 indicates important dog regions that were damaged by the attack and became unimportant regions.
The DNN learned via knowledge distillation encoded less reliable visual patterns. In this experiment, we learned two student DNNs (two VGG-16 nets) for knowledge distillation [22]. One student DNN was learned by distilling the output feature of the conv_53 layer after the ReLU operation in the teacher DNN (a pre-trained VGG-16) to the corresponding layer in the student DNN. The other student DNN was learned by distilling the output feature of the penultimate fully-
connected layer after the ReLU operation in the teacher DNN to the corresponding layer in the student DNN. Figure 9(left) compares the number of all knowledge points and reliable knowledge points encoded by the teacher DNN and the two student DNNs, when we quantified knowledge points in conv_33/conv_43/conv_53 layers. We found that student DNNs usually encoded less reliable knowledge points than the teacher DNN.
• Furthermore, the student DNN usually learned even less reliable concepts in a layer, if this layer was farther from the target layer used for distillation. To verify this conclusion, we compared the number of knowledge points between the above two student DNNs. As Figure 9(left) shows, the student DNN distilled using features of the fully-connected layer encoded much less reliable concepts than the student DNN distilled using features of the conv_53 layer, which verified our conclusion.
• Although the knowledge distillation could force the student DNN to well mimic features of a specific layer in teacher DNN, there was still a big difference of other layers’ regional features between the student DNN and the teacher DNN. To this end, we evaluated the quality of student DNNs mimicking the teacher DNN. We selected h(r)student and h (r) teacher as two corresponding regional features of the student DNN and the teacher DNN in the same layer. Then, we used 1 − cos(h(r)student, h (r) teacher) and ‖h(r)student‖2 − ‖h (r) teacher‖2 to measure the difference of orientation and the difference of strength between the two regional features. Figure 9(right) shows the histogram of 1− cos(h(r)student, h (r) teacher) and ‖h(r)student‖2−‖h (r) teacher‖2, when we used the conv_53 layer to evaluate the similarity between the student DNN and the teacher DNN. The similarity between student DNN features and teacher DNN features was lower when the student DNN was distilled based on features in the fully-connected layer (far from the conv_53 layer), which verified our conclusion.

heading: 5 Conclusion
text: In this paper, we propose a method to visualize intermediate visual patterns in a DNN. The visualization illustrates the emergence of intermediate visual patterns in a temporal-spatial manner. The proposed method also enables people to measure the quantity and quality of visual patterns encoded by the DNN, which provides a new perspective to analyze the discrimination power of DNNs. Furthermore, the proposed method provides insightful understanding towards the signal-processing behaviors of existing deep-learning techniques.
Acknowledgments and Disclosure of Funding
This work is partially supported by the National Nature Science Foundation of China (No. 61906120, U19B2043), Shanghai Natural Science Fundation (21JC1403800,21ZR1434600), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).

Review summaries:
Paper proposes a visualization method for probing into the intermediate visual patterns learned by the DNN layers. The idea is to learn a linear transformation to project the original DNN features into a lower dimensional space for mimicing the classification based on the original features. In other words, the projected low-dim features infer the similarity among samples. Experiments are conducted multiple datasets and DNN architectures.

The authors proposed an approach to visualize the regional visual patterns learned by the DNN models during the training. The method shows that (1) adversarial attacks mainly affect the regional features in depth layers. The authors also categorize the adversarial attack impact in four types; and (2) in the distillation scenario, the student learns to extract less reliable knowledge points than the teacher. It is an interesting approach with promising results, quantitative and analysis.