Visualizing the Emergence of Intermediate Visual Patterns in DNNs

1 Introduction . Deep neural networks ( DNNs ) have achieved superior performance in various tasks , but the black-box nature of DNNs makes it difficult for people to understand its internal behavior . Visualization methods are usually considered as the most direct way to understand the DNN . Recently , several attempts have been made to visualize the DNN from different aspects , e.g . illustrating the visual appearance that maximizes the prediction score of a given category [ 50 , 66 , 35 ] , inverting intermediate-layer features to network inputs [ 12 ] , extracting receptive fields of neural activations [ 73 ] , estimating saliency/importance/attribution maps [ 74 , 45 , 75 , 32 ] , visualizing the sample distribution , such as PCA [ 38 ] , t-SNE [ 54 ] , etc . In spite of above explanations of the DNN , there is still a large gap between visual explanations of the patterns in the DNN and the theoretical analysis of the DNN ’ s discrimination power . In other words , visualization results usually can not reflect the discrimination power of features in the DNN . Therefore , instead of simply visualizing the entire sample , we divide intermediate-layer features into feature components , each of which represents a specific image region . We visualize the discrimination power of these feature components , and we consider discriminative feature components as knowledge points learned by the DNN . Based on above methods , we can diagnose the feature representation of a pre-trained DNN from the following perspectives . •We visualize the emergence of intermediate visual patterns in a temporal-spatial manner and evaluate their discrimination power . ( 1 ) We visualize how the discrimination power of each individual visual ∗Corresponding author . This work was done under the supervison of Dr. Quanshi Zhang . He is with the John Hopcroft Center and the MoE Key Lab of Artificial Intelligence , AI Institute , at the Shanghai Jiao Tong University , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . pattern increases during the learning process . ( 2 ) We illustrate effects of using non-discriminative patterns in low layers to gradually construct discriminative patterns in high layers during the forward propagation . As Figure 1 shows , the regional feature of the cat head emerges as a discriminative pattern for the cat category , while wall features are non-discriminative . • Based on the the visualization result , we can further measure the quantity and quality of intermediate patterns encoded by a DNN . In Figure 1 , we count knowledge points encoded in a DNN as regional patterns with strong discrimination power , and further evaluate whether each knowledge point is reliable for classification . This provides a new perspective to analyze the DNN . A distinct contribution of this study is to bridge the empirical visualization and the quantitative analysis of a DNN ’ s discrimination power . In comparison , Kim et al . [ 23 ] used concept activation vectors to model the relationship between visual features and manually annotated semantic concepts . Cheng et al . [ 7 ] quantified the number of visual concepts encoded by the DNN . However , these two methods can not reflect the discrimination power of regional visual concepts . On the other hand , some researchers derived mathematical bounds on the representational power of a DNN [ 68 , 13 ] under certain assumptions of the network architecture . To this end , we believe that bridging regional patterns and a DNN ’ s discrimination power is a more convincing and more intuitive way to reveal the internal behavior of a DNN than mathematical bounds under certain assumptions . Besides , our method provides insightful understanding towards existing deep-learning techniques , such as adversarial attacks and knowledge distillation . ( 1 ) For adversarial attacks , we discover that adversarial attacks mainly affect unreliable regional features in high layers of the DNN . The visualization result also enables us to categorize attacking behaviors of all image regions into four types . ( 2 ) For knowledge distillation , we discover that the student DNN usually encodes less reliable knowledge points , compared with the teacher DNN . Although knowledge distillation is able to force the student DNN to mimic features of a specific layer in the teacher DNN , there is still a big difference of features in other layers between the student DNN and the teacher DNN . Contributions of this paper can be summarized as follows . ( 1 ) We propose a method to visualize the discrimination power of intermediate-layer features in the DNN , and illustrate the emergence of intermediate visual patterns in a temporal-spatial manner . ( 2 ) Based on the visualization result , we quantify knowledge points encoded by a DNN . ( 3 ) The proposed method also provides new insights into existing deep-learning techniques , such as the adversarial attack and the knowledge distillation . 2 Related work . Visual explanations for DNNs . Visualization of DNNs is the most direct way to explain visual patterns encoded in the DNN . Some studies reconstructed the network input based on a given intermediate-layer feature [ 34 , 48 , 12 ] , while others aimed to generate an input image that cause high activations in a given feature map [ 67 ] . Zhou et al . [ 73 ] extracted the actual image-resolution receptive field of neural activations in DNNs . Recently , Goh et al . [ 17 ] visualized the multi-modal neurons in the CLIP models [ 39 ] . Kim et al . [ 23 ] proposed TCAV to represent manually annotated semantic concepts . Another type of researches diagnosed and visualized the pixel-wise saliency/importance/attribution on real input images [ 42 , 32 , 24 , 75 , 14 , 74 , 5 ] . What is more , the distribution of intermediatelayer features could be visualized via dimensionality reduction methods , e.g . PCA [ 38 ] , LLE [ 43 ] , MDS [ 10 ] , ISOMAP [ 52 ] , t-SNE [ 54 ] , etc . Recently , Law et al . [ 25 ] visualized the feature of each sample in a low-dimensional space by exploiting probability scores of the DNN , while Li et al . [ 27 ] achieved this by assigning each category in the task with a main direction . Sophisticated interfaces have been built up to visualize the architecture and the knowledge of DNNs [ 19 , 60 , 53 ] . Instead of visualizing visual appearance or merely visualizing the distribution of sample features , our method visualizes the emergence of intermediate visual patterns in a temporal-spatial manner , which provides a new perspective to explain DNNs . Theoretical analysis of the representation capacity of DNNs . Formulating and evaluating the discrimination power of DNNs is another direction to explain DNNs . The information-bottleneck theory [ 64 , 47 ] provides a generic metric to quantify information encoded in DNNs , which is further exploited to evaluate the representation capacity of DNNs [ 18 , 65 ] . Achille and Soatto [ 1 ] further used the information-bottleneck to improve the feature representation . Furthermore , several metrics were also proposed to analyze the robustness or generalization capacity of DNNs , such as the CLEVER score [ 62 ] , the stiffiness [ 15 ] , the sensitivity metrics [ 37 ] , etc . Zhang et al . [ 69 ] studied the role of different layers towards the generalization of deep models . Based on this , module criticality [ 4 ] was proposed to analyze a DNN ’ s generalization power . Cheng et al . [ 7 ] quantified visual concepts in input images from the perspective of pixel-wise entropies . Liang et al . [ 29 ] diagnosed feature representation between different DNNs via knowledge consistency . Some studies also theoretically proved the generalization bound for two-layer neural networks [ 68 , 13 , 36 ] and deep CNNs [ 31 , 28 ] . In fact , our research group led by Dr. Quanshi Zhang have proposed game-theoretic interactions as a new perspective to explain the representation capacity of trained DNNs . The interactions have been used to explain the hierarchical structure of information processing in DNNs [ 70 , 72 ] , generalization power [ 71 ] , the complexity [ 8 ] and the aesthetic level [ 9 ] of the encoded visual concepts , adversarial robustness [ 40 , 58 ] , and adversarial transferability [ 57 ] of DNNs . The interaction was also used to learn baseline values of Shapley values [ 41 ] . However , there is still a lack of connections between the explanations of visual concepts and the analysis of a DNN ’ s discrimination power . To this end , our method enables people to use the discrimination power of local regions to explain the overall discrimination power of the entire DNN . 3 Algorithm . Given a pre-trained DNN , we propose an algorithm to project the feature of an entire sample and features corresponding to different image regions into a low-dimensional space , in order to visualize ( 1 ) how the discrimination power of a sample feature increases during the learning process , ( 2 ) effects of the DNN constructing discriminative features in high layers using non-discriminative features in low layers , and ( 3 ) how discriminative regional features gradually emerge during the training process . The visualization result reflects whether the sample feature and each regional feature are biased to incorrect categories , which regional features play a crucial role in classification , whether a regional feature is shared by multiple categories , etc . Furthermore , the visualization enables people to quantify knowledge points encoded in the DNN . Such analysis can help people explain existing deep-learning techniques , e.g . adversarial attacks and knowledge distillation . 3.1 Radial distribution to analyze the discrimination power of features . Preliminaries : the vMF distribution . Given a pre-trained DNN and an input image x ∈ Rn , let us consider the output feature of a specific layer , denoted by f ∈ Rd . To analyze the discimination power of f , previous literature usually considered the feature to follow a radial distribution of massive pseudo-categories ( much more than the real category number ) [ 61 , 56 , 25 ] . As Figure 1 ( right ) shows , each category/pseudo-category c has a mean direction µc ( c = 1 , ... , C ) in the feature space . The significance of classifying x towards category c is measured by the projection f > µc , and cos ( f , µc ) indicates the similarity between f and category c. For example , a typical case is the softmax operation , p ( y|f ) = Softmax ( Wf ) , W ∈ RC×d . The c-th row of W indicates the direction corresponding to the c-th category . To this end , the von Mises-Fisher ( vMF ) mixture model [ 3 , 20 ] was proposed to model the radial distribution , where the c-th mixture component assumes that the likelihood of each feature f belonging to category c ∈ { 1 , ... , C } follows a vMF distribution . p ( f ) = ∑ c p ( y = c ) pvMF ( f |y = c ) , pvMF ( f |y = c ) = Cd ( κc ) · exp [ κc · cos ( µc , f ) ] , ( 1 ) where Cd ( κc ) = κ d/2−1 c ( 2π ) d/2Id/2−1 ( κc ) is the normalization constant . Actually , the vMF distribution can be considered as a spherical analogue to the Gaussian distribution on the unit sphere . µc ∈ Rd measures the mean direction of category c. The increase of κc ≥ 0 decreases the variance of f ’ s orientation w.r.t . the mean direction µc . Please see the supplementary material for more details . Radial distribution with noise . The vMF distribution assumes that f is a clean feature without noise , which makes the inference of f purely based on its orientation and independent of the strength ‖f‖2 . However , in real applications , f usually contains the meaningful and clean feature f ? and the meaningless noise ∼ N ( 0 , σ2Id ) , i.e . f = f ? + . The existence of the noise decreases the classification confidence if the strength of f is low . Therefore , we have p ( f |y = c ) = ∫ p ( ) · pvMF ( f ? = f − |y = c ) d . ( 2 ) For simplicity , we can assume that all features f of a specific strength l follow a vMF distribution , because they have similar vulnerabilities to noises . Specifically , let f = [ o , l ] , where l = ‖f‖2 and o = f/l represent the strength and orientation of f . Then , the likelihood of f belonging to category c is given as follows ( proof in the supplementary material ) . p ( f = [ o , l ] |y = c ) = p ( l|y = c ) · pvMF ( o|y = c , l ) = p ( l|y = c ) · pvMF ( o|µc , κ ( l ) ) , ( 3 ) where κ ( l ) increases along with l = ‖f‖2 , and p ( l|y = c ) is the prior distribution of ‖f‖2 for category c. The variance parameter κ ( l ) is determined based on statistics of all features of the same strength l. We can prove the classification result p ( y=c|f ) is confident when the feature f has a large strength .

Summary: The authors proposed an approach to visualize the regional visual patterns learned by the DNN models during the training. The method shows that (1) adversarial attacks mainly affect the regional features in depth layers. The authors also categorize the adversarial attack impact in four types; and (2) in the distillation scenario, the student learns to extract less reliable knowledge points than the teacher. It is an interesting approach with promising results, quantitative and analysis.