Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Paper proposes a visualization method for probing into the intermediate visual patterns learned by the DNN layers. The idea is to learn a linear transformation to project the original DNN features into a lower dimensional space for mimicing the classification based on the original features. In other words, the projected low-dim features infer the similarity among samples. Experiments are conducted multiple datasets and DNN architectures.
main_review: - Experiments reveal that the low dimensional projections preserve semantic similarity. Also, the magnitude of the projections is shown to be correlated (better than baselines and existing works) to the confidence of classification. Similarly, in case of regional features also, they show that both the discriminative ability and reliability improve temporally with training.
- Regional importance estimated by the proposed approach is shown to be aligned with the Shapeley value (contribution of that region to the DNN o/p).
- Paper presents multiple use cases of the proposed visualization along with understanding the DNN learnings and verification of the known notions.
- Proposed approach infers that the adversaries majorly affect the easy to attack (or, unreliable) regional features from the semantic (higher) layers.
- Similarly, interesting findings are inferred by the paper about the process of knowledge distillation (although the faithfulness of the metrics such as #knowledge points that led to these inferences has not been studied rigorously).
- In summary, the paper presents a different (and seemingly complex) method for going into the learning aspects of the DNNs. However, it sort of gives a feeling that multiple inferences are not strong novel findings (yet they may be considered for adding reliability on the proposed method). For instance, evolution of discriminative features with training, increase in the discriminative ability of the features (and number of reliable knowledge points) towards the deeper layers, etc. In this regard, authors' attempt to explain phenomena such as adversarial attacks and knowledge distillation helps.

Minor

- line 164: 'needs two overcome to challenges' -> 'needs to overcome two challenges'
limitations_and_societal_impact: I do not see potential -ve societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present an interpretability method that shows the regional visual patterns (ie, knowledge points) that are important for the discriminative power of a network on a given image. They also explore this in the context of model distillation and adversarial attacks

main_review: **Originality**
This paper definitely seems original to me. It is an interesting method that goes beyond just looking at image pieces that are relevant for a given prediction, to try and match this with the discriminative power of the network.

**Quality**
I believe this is a high quality submission. The methods are sound (if a bit convoluted-- see clarity section below), and the analysis of model distillation and adversarial attacks is really compelling


**Clarity**
Most of the paper is clear, but I found section 3 a bit dense and difficult to understand. I know part of this is inherent since this section is math-heavy, but it could also really benefit from another diagram similar to figure 1 or figure 3 to just explain at a high level what the intuition is for the algorithm, before deeply diving into the derivations.

Also, I'm still a bit confused about algorithm and want to make sure I'm understanding it correctly. Is this correct?
- Chunk the image into pieces (ie, geographic regions-- might be a cat ear, section of the wall, etc)
- Run each of these image pieces through the CNN to get its embedding at each layer
- Get the dot product of the image piece embeddings with feature space directions for each class
- For these dot products, high value = high discriminative power = knowledge point
- To make visualization, show the regions that had high dot products

If this is correct, section 3 probably doesn't need to be as long (parts could be moved to the appendix). That being said, I also definitely might be missing something more complicated about the method!

Also, one typo:
>while others aimed to generate an input image that cause high
(cause => causes)

**Significance**
This is an interesting and novel take on visualizing image regions that are relevant to DNN predictions.

limitations_and_societal_impact: n/a
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors proposed an approach to visualize the regional visual patterns learned by the DNN models during the training. The method shows that (1) adversarial attacks mainly affect the regional features in depth layers. The authors also categorize the adversarial attack impact in four types; and (2) in the distillation scenario, the student learns to extract less reliable knowledge points than the teacher. It is an interesting approach with promising results, quantitative and analysis.

main_review: The method divides intermediate layer features into feature components, each of which represents a specific image region. Then, the technique visualizes these feature components' discrimination power and considers discriminative feature components as knowledge points learned by the DNN. Moreover, the authors show how the discrimination power of each visual pattern increases during the learning process and illustrate the effect of using non discriminative patterns in low layers to gradually construct discriminative patterns in high layers during the forward propagation.

I have the following recommendations and suggestions on how this paper can be improved:

(Line 45) "To this end, we believe that bridging regional patterns and a DNN's discrimination power is a more convincing and more intuitive way to reveal the internal behavior of a DNN than mathematical bounds under certain assumptions." The authors need to justify this claim because the paper also made some assumptions, such as lines 126-127 and 147-148.

(Line 214) Can the authors show any quantitative evaluation of this claim? Since the work uses the projected features in other scenarios, it is essential to know how accurate it is.

(Line 245) The input features of fully connected layers are 1D vectors, so what is the shape of this sample feature? Is it (N, 1, 1)? Are the authors using the features before some average pooling layer?

(Lines 266)  How is the mapping between input image regions and the reliable knowledge points made? This is an important step to improve the paper's reproducibility.

(Lines 275-279) The presented qualitative analysis of few examples is interesting, but why not present the quantitative analysis for all images of the dataset?

The quality of the text regions on the Figures should be improved (e.g., Figure 1).

What is the statistical relevance of the results in Figure 6?

What is the impact of the sample confidence in the ratio of reliable knowledge points (RRKP)? Maybe the samples when the model returns the correct answer with high confidence have a higher RRKP than images with low confidence. It is also valid to consider the distillation scenario because the student model may produce low confidence predictions than the teacher.
	
What is the method performance on out-of-distribution (OOD) samples?

Since distillation can be used as adversarial defense [1], what is the relation between the results of the student model and the adversarial attacks? 

There has been little discussion on the limitations of the proposed approach. What are the possible technical limitations? It's not clear how the method deals with noise regions when formulating and estimating the specific importance of different regions. In other words, how accurate is it to assume that all features of a specific region have similar vulnerabilities to noise?
	
Revise the paper to eliminate the typos in the text (e.g., section 3.1 discrimination as "discimination")

[1] Papernot, Nicolas, et al. "Distillation as a defense to adversarial perturbations against deep neural networks." 2016 IEEE symposium on security and privacy (SP). IEEE, 2016.

limitations_and_societal_impact: The authors made no comments about the Limitations And Societal Impact. The authors could address how your research could affect ML applications as possible uses, range of social impacts, and address the positive effects and reduce the downsides of the current approaches.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper quantifies the number of discriminative patterns in DNN to evaluate the network capacity through visualization method, and also provides some insights for adversarial attack and knowledge distillation.
main_review: #Originality: The method connects the visual explanation and DNN's discriminative power, which has not been well studied yet.

#Quality: This paper addresses an important problem (evaluate network's discriminative power via visual concepts), and provides a new perspective to analyze the discriminative powers via measure the quantity and quality of visual concepts learned by DNN.

#Clarity: The paper is well-written and easy to follow.

#Significance: The results show some insights for later work, it will be interesting to further explore.
limitations_and_societal_impact: In line 295, the paper claims that adv attack mainly affect unreliable regional features in high layers, but lack of full analysis. No experiment shows that the reliable regional features are more robust to attack (or does it mean the reliable region is hard to attack?). Besides, VGG is easy to attack, how is the results for different network (maybe adversarially trained, more robust)? Why these models are hard to attack, do they have less unreliable regional features? I'm also concern about how to apply this method into adversarial attack (sparsely attack the unreliable region) and defense (how to increase the number of reliable regional features).
review_title: null

Meta-review: This work proposes a new method for examining the patterns learned by intermediate layers of a DNN, by projecting features into a lower dimensional space that preserves discriminative structure. The reviewers were consistently positive about the paper’s novelty and impact, though some concerns were raised regarding clarity. The authors have done a good job of satisfying reviewer concerns in their rebuttals, and I trust that the authors can incorporate these clarifications into the final version without radically reshaping the paper. All in all, interesting and exciting work.
Decision: Accept (Poster)