Visualizing the Emergence of Intermediate Visual Patterns in DNNs

1 Introduction . Deep neural networks ( DNNs ) have achieved superior performance in various tasks , but the black-box nature of DNNs makes it difficult for people to understand its internal behavior . Visualization methods are usually considered as the most direct way to understand the DNN . Recently , several attempts have been made to visualize the DNN from different aspects , e.g . illustrating the visual appearance that maximizes the prediction score of a given category [ 50 , 66 , 35 ] , inverting intermediate-layer features to network inputs [ 12 ] , extracting receptive fields of neural activations [ 73 ] , estimating saliency/importance/attribution maps [ 74 , 45 , 75 , 32 ] , visualizing the sample distribution , such as PCA [ 38 ] , t-SNE [ 54 ] , etc . In spite of above explanations of the DNN , there is still a large gap between visual explanations of the patterns in the DNN and the theoretical analysis of the DNN ’ s discrimination power . In other words , visualization results usually can not reflect the discrimination power of features in the DNN . Therefore , instead of simply visualizing the entire sample , we divide intermediate-layer features into feature components , each of which represents a specific image region . We visualize the discrimination power of these feature components , and we consider discriminative feature components as knowledge points learned by the DNN . Based on above methods , we can diagnose the feature representation of a pre-trained DNN from the following perspectives . •We visualize the emergence of intermediate visual patterns in a temporal-spatial manner and evaluate their discrimination power . ( 1 ) We visualize how the discrimination power of each individual visual ∗Corresponding author . This work was done under the supervison of Dr. Quanshi Zhang . He is with the John Hopcroft Center and the MoE Key Lab of Artificial Intelligence , AI Institute , at the Shanghai Jiao Tong University , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . pattern increases during the learning process . ( 2 ) We illustrate effects of using non-discriminative patterns in low layers to gradually construct discriminative patterns in high layers during the forward propagation . As Figure 1 shows , the regional feature of the cat head emerges as a discriminative pattern for the cat category , while wall features are non-discriminative . • Based on the the visualization result , we can further measure the quantity and quality of intermediate patterns encoded by a DNN . In Figure 1 , we count knowledge points encoded in a DNN as regional patterns with strong discrimination power , and further evaluate whether each knowledge point is reliable for classification . This provides a new perspective to analyze the DNN . A distinct contribution of this study is to bridge the empirical visualization and the quantitative analysis of a DNN ’ s discrimination power . In comparison , Kim et al . [ 23 ] used concept activation vectors to model the relationship between visual features and manually annotated semantic concepts . Cheng et al . [ 7 ] quantified the number of visual concepts encoded by the DNN . However , these two methods can not reflect the discrimination power of regional visual concepts . On the other hand , some researchers derived mathematical bounds on the representational power of a DNN [ 68 , 13 ] under certain assumptions of the network architecture . To this end , we believe that bridging regional patterns and a DNN ’ s discrimination power is a more convincing and more intuitive way to reveal the internal behavior of a DNN than mathematical bounds under certain assumptions . Besides , our method provides insightful understanding towards existing deep-learning techniques , such as adversarial attacks and knowledge distillation . ( 1 ) For adversarial attacks , we discover that adversarial attacks mainly affect unreliable regional features in high layers of the DNN . The visualization result also enables us to categorize attacking behaviors of all image regions into four types . ( 2 ) For knowledge distillation , we discover that the student DNN usually encodes less reliable knowledge points , compared with the teacher DNN . Although knowledge distillation is able to force the student DNN to mimic features of a specific layer in the teacher DNN , there is still a big difference of features in other layers between the student DNN and the teacher DNN . Contributions of this paper can be summarized as follows . ( 1 ) We propose a method to visualize the discrimination power of intermediate-layer features in the DNN , and illustrate the emergence of intermediate visual patterns in a temporal-spatial manner . ( 2 ) Based on the visualization result , we quantify knowledge points encoded by a DNN . ( 3 ) The proposed method also provides new insights into existing deep-learning techniques , such as the adversarial attack and the knowledge distillation . 2 Related work . Visual explanations for DNNs . Visualization of DNNs is the most direct way to explain visual patterns encoded in the DNN . Some studies reconstructed the network input based on a given intermediate-layer feature [ 34 , 48 , 12 ] , while others aimed to generate an input image that cause high activations in a given feature map [ 67 ] . Zhou et al . [ 73 ] extracted the actual image-resolution receptive field of neural activations in DNNs . Recently , Goh et al . [ 17 ] visualized the multi-modal neurons in the CLIP models [ 39 ] . Kim et al . [ 23 ] proposed TCAV to represent manually annotated semantic concepts . Another type of researches diagnosed and visualized the pixel-wise saliency/importance/attribution on real input images [ 42 , 32 , 24 , 75 , 14 , 74 , 5 ] . What is more , the distribution of intermediatelayer features could be visualized via dimensionality reduction methods , e.g . PCA [ 38 ] , LLE [ 43 ] , MDS [ 10 ] , ISOMAP [ 52 ] , t-SNE [ 54 ] , etc . Recently , Law et al . [ 25 ] visualized the feature of each sample in a low-dimensional space by exploiting probability scores of the DNN , while Li et al . [ 27 ] achieved this by assigning each category in the task with a main direction . Sophisticated interfaces have been built up to visualize the architecture and the knowledge of DNNs [ 19 , 60 , 53 ] . Instead of visualizing visual appearance or merely visualizing the distribution of sample features , our method visualizes the emergence of intermediate visual patterns in a temporal-spatial manner , which provides a new perspective to explain DNNs . Theoretical analysis of the representation capacity of DNNs . Formulating and evaluating the discrimination power of DNNs is another direction to explain DNNs . The information-bottleneck theory [ 64 , 47 ] provides a generic metric to quantify information encoded in DNNs , which is further exploited to evaluate the representation capacity of DNNs [ 18 , 65 ] . Achille and Soatto [ 1 ] further used the information-bottleneck to improve the feature representation . Furthermore , several metrics were also proposed to analyze the robustness or generalization capacity of DNNs , such as the CLEVER score [ 62 ] , the stiffiness [ 15 ] , the sensitivity metrics [ 37 ] , etc . Zhang et al . [ 69 ] studied the role of different layers towards the generalization of deep models . Based on this , module criticality [ 4 ] was proposed to analyze a DNN ’ s generalization power . Cheng et al . [ 7 ] quantified visual concepts in input images from the perspective of pixel-wise entropies . Liang et al . [ 29 ] diagnosed feature representation between different DNNs via knowledge consistency . Some studies also theoretically proved the generalization bound for two-layer neural networks [ 68 , 13 , 36 ] and deep CNNs [ 31 , 28 ] . In fact , our research group led by Dr. Quanshi Zhang have proposed game-theoretic interactions as a new perspective to explain the representation capacity of trained DNNs . The interactions have been used to explain the hierarchical structure of information processing in DNNs [ 70 , 72 ] , generalization power [ 71 ] , the complexity [ 8 ] and the aesthetic level [ 9 ] of the encoded visual concepts , adversarial robustness [ 40 , 58 ] , and adversarial transferability [ 57 ] of DNNs . The interaction was also used to learn baseline values of Shapley values [ 41 ] . However , there is still a lack of connections between the explanations of visual concepts and the analysis of a DNN ’ s discrimination power . To this end , our method enables people to use the discrimination power of local regions to explain the overall discrimination power of the entire DNN . 3 Algorithm . Given a pre-trained DNN , we propose an algorithm to project the feature of an entire sample and features corresponding to different image regions into a low-dimensional space , in order to visualize ( 1 ) how the discrimination power of a sample feature increases during the learning process , ( 2 ) effects of the DNN constructing discriminative features in high layers using non-discriminative features in low layers , and ( 3 ) how discriminative regional features gradually emerge during the training process . The visualization result reflects whether the sample feature and each regional feature are biased to incorrect categories , which regional features play a crucial role in classification , whether a regional feature is shared by multiple categories , etc . Furthermore , the visualization enables people to quantify knowledge points encoded in the DNN . Such analysis can help people explain existing deep-learning techniques , e.g . adversarial attacks and knowledge distillation . 3.1 Radial distribution to analyze the discrimination power of features . Preliminaries : the vMF distribution . Given a pre-trained DNN and an input image x ∈ Rn , let us consider the output feature of a specific layer , denoted by f ∈ Rd . To analyze the discimination power of f , previous literature usually considered the feature to follow a radial distribution of massive pseudo-categories ( much more than the real category number ) [ 61 , 56 , 25 ] . As Figure 1 ( right ) shows , each category/pseudo-category c has a mean direction µc ( c = 1 , ... , C ) in the feature space . The significance of classifying x towards category c is measured by the projection f > µc , and cos ( f , µc ) indicates the similarity between f and category c. For example , a typical case is the softmax operation , p ( y|f ) = Softmax ( Wf ) , W ∈ RC×d . The c-th row of W indicates the direction corresponding to the c-th category . To this end , the von Mises-Fisher ( vMF ) mixture model [ 3 , 20 ] was proposed to model the radial distribution , where the c-th mixture component assumes that the likelihood of each feature f belonging to category c ∈ { 1 , ... , C } follows a vMF distribution . p ( f ) = ∑ c p ( y = c ) pvMF ( f |y = c ) , pvMF ( f |y = c ) = Cd ( κc ) · exp [ κc · cos ( µc , f ) ] , ( 1 ) where Cd ( κc ) = κ d/2−1 c ( 2π ) d/2Id/2−1 ( κc ) is the normalization constant . Actually , the vMF distribution can be considered as a spherical analogue to the Gaussian distribution on the unit sphere . µc ∈ Rd measures the mean direction of category c. The increase of κc ≥ 0 decreases the variance of f ’ s orientation w.r.t . the mean direction µc . Please see the supplementary material for more details . Radial distribution with noise . The vMF distribution assumes that f is a clean feature without noise , which makes the inference of f purely based on its orientation and independent of the strength ‖f‖2 . However , in real applications , f usually contains the meaningful and clean feature f ? and the meaningless noise ∼ N ( 0 , σ2Id ) , i.e . f = f ? + . The existence of the noise decreases the classification confidence if the strength of f is low . Therefore , we have p ( f |y = c ) = ∫ p ( ) · pvMF ( f ? = f − |y = c ) d . ( 2 ) For simplicity , we can assume that all features f of a specific strength l follow a vMF distribution , because they have similar vulnerabilities to noises . Specifically , let f = [ o , l ] , where l = ‖f‖2 and o = f/l represent the strength and orientation of f . Then , the likelihood of f belonging to category c is given as follows ( proof in the supplementary material ) . p ( f = [ o , l ] |y = c ) = p ( l|y = c ) · pvMF ( o|y = c , l ) = p ( l|y = c ) · pvMF ( o|µc , κ ( l ) ) , ( 3 ) where κ ( l ) increases along with l = ‖f‖2 , and p ( l|y = c ) is the prior distribution of ‖f‖2 for category c. The variance parameter κ ( l ) is determined based on statistics of all features of the same strength l. We can prove the classification result p ( y=c|f ) is confident when the feature f has a large strength .

Summary: Paper proposes a visualization method for probing into the intermediate visual patterns learned by the DNN layers. The idea is to learn a linear transformation to project the original DNN features into a lower dimensional space for mimicing the classification based on the original features. In other words, the projected low-dim features infer the similarity among samples. Experiments are conducted multiple datasets and DNN architectures.