Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper approaches the topic of network explainability and visualization with a novel approach for isolating the factors contributing to a model's behaviour that focuses on network parameters rather than the previously more common approach of identifying image regions of highest impact. This is a subtle but non-trivial change, and leads to a number of useful new analyses of network behaviour, and in particular shows great promise in selectively training network parameters to correct errors.
strengths_and_weaknesses: Strengths:
This paper presents a novel idea that is intuitive (and almost obvious in retrospect, but many great ideas seem so!) and quite broad in application and effect. The experiments that are performed are easy to follow and clearly developed, offering strong evidence for the potential of this method, particularly in terms of targeted tuning and updating of network behaviour.

Weaknesses:
- Perhaps I missed it, but what is the effect on network behaviour for objects that are not neighbours to the misclassified sample? Are new errors introduced (and if so, at what rate? Is this still a strong net benefit, or are we trading one set of errors for another)?
- The visualization aspect of this method appears to essentially still end up looking similar to class activation maps in methods like Grad-CAM (i.e. it gets projected down to a pixel-wise map showing image regions that are most affecting the model behaviour on the given sample). While I am not sure that this is in and of itself a major weakness in the method (it is an intuitive visualization technique), the lack of comparison between the visualizations produced by the proposed method with prior image region based methods (e.g. Grad-CAM) makes it harder to evaluate their role and effectiveness.
questions: I do not have any questions beyond the minor points raised above. The paper was overall clearly written.
limitations: I do not foresee any negative societal impacts as a direct result of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors analyze salient network parameters, which are responsible for erroneous decisions. They conduct experiments to find salient parameters responsible for faulty predictions and show the relation between these parameters and regions in the input image.
strengths_and_weaknesses: Strengths: Instead of focusing on saliency maps like other work, this paper pays attention to salient network parameters, and the findings are interesting.

Weaknesses: I am not expert in this research field. However, I have a concern about the novelty of this paper. Parameter saliency is not new, and it’s already done in adversarial example research. The authors also do not clarify differences between this paper and adversarial example.

questions: How is this paper different from adversarial example?
limitations: I believe there would be no potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: "Where do Models go Wrong" presents a saliency method that highlights model filters important to the output. The paper validates these parameters are salient by turning off the salient parameters on misclassified predictions and showing an increase in corrected classification. They evaluate the semantic meaning of parameter space saliency by quantifying each input by its parameter saliency and analyzing the resulting space. The paper finds neighbors often share misclassification causes instead of visual similarities, showing that parameter saliency captures patterns in model behavior. Parameter saliency can also identify filters to fine-tune as a way to correct model mistakes. Finally, the paper shows how to use parameter saliency to analyze an image in an explainability task by masking various portions of the image related to salient parameters.
strengths_and_weaknesses: **Strengths**
* *Experimental baselines* --- The paper evaluates the saliency map by pruning salient filters and measuring the change to the model's output. They compare this procedure to pruning random filters and pruning the least salient filters. These comparisons show a significant difference in the model's output when salient filters are pruned.
* *Thorough evaluation* --- The paper validates their results on various models architectures trained on image and language modalities. They also test their saliency method using existing model randomization tests and find it model dependent.
* *Clarity* --- The paper is very well written and precise.

**Weaknesses**
* *Analysis of other saliency methods* --- The proposed parameter saliency method computes the gradient of the loss with respect to each model parameter and aggregates the gradients per filter. While other methods could compute parameter saliency, the paper does not include evaluations of any alternatives. It would be informative to see how consistent the parameter saliency is between methods. For instance, SmoothGrad could be used in this method by perturbing the input features but taking the gradient with respect to the model parameter.
* *Analysis of correctly classified samples* --- The method focuses on identifying filters contributing to misclassification. While this is undoubtedly important to identify and fix model errors, it is equally important to understand why the model made a correct classification. For example, it is critical to know why a model correctly predicted disease in a medical setting.

**Related Work**
* *Model error patterns* --- Figure 8 classifies different types of mistakes based on where the saliency is with respect to the main object. "Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior" by Boggust et al. has a similar metric for identifying and measuring these patterns. You might consider citing this method or using it to compute the Shared Interest of the different mistake categories you identified.
* *Saliency method evaluation* --- I appreciate that you evaluate your saliency method via the model parameterization test in the appendix. You might also consider discussing or validating your input-wise saliency on other proposed tests like the pointing game ("Top-down Neural Attention by Excitation Backprop" by Zhang et al.) 
* *Removing salient parameters experiments* --- The experiment removing saliency parameters to validate parameter-space saliency is analogous to removing salient features to validate input-space saliency. These validation approaches have been proposed in "Explaining recurrent neural network predictions in sentiment analysis" by Arras et al. and "Evaluating the visualization of what a deep neural network has learned" by Samek et al.

**Minor Issues**
* Figures 3, 6, and 7 are too small and hard to read.
* Please move figures closer to their corresponding text. Having figures pages away from the reference text is challenging to read.
questions: * The proposed parameter saliency method computes the gradient of the loss with respect to each model parameter. This seems equivalent to Vanilla Gradients applied to model parameter space. Is it possible to use variants of other input-space saliency methods to compute parameter-space saliency (e.g., SmoothGrad)? If so, have you tried any, and how did they compare?
* Could you use parameter saliency to prune models for model compression? Have you tried this or compared it to other pruning techniques?
* Can you provide a similar analysis for correctly classified inputs? It would be valuable to confirm that pruning salient filters decreases model output confidence for correctly classified images. Similarly, it would be interesting to see if the nearest neighbors of correct classifications are other correct classifications. Are neighbors from correctly classified images more likely to be in the same class, or would we find neighbors with similar confidence in the output class?
limitations: The appendix provides an interesting and thorough discussion about adversarial attacks, a comparison to GradCAM, and limitations. I would suggest moving at least some of this to the main text as it provides compelling context and considerations for the method.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: An interesting paper to establish a link between parameter space saliency and input saliency for the purpose of diagnosis of misclassification in deep NNs. The paper demonstrates that by turning salient filters on/off the prediction of NNs could be corrected. 
strengths_and_weaknesses: The paper addresses an interesting question that is to study the relation between saliency and network predictions. The approach is thought provoking. Nonetheless, the paper is  vague about the details when it comes to saliency profile calculation and could have a better representation. The experiments are limited but convincing with respect to the conclusion. There is, however, an overlooked area, in my opinion. That is the dataset bias and label noise, when the most visually salient input is not the same as the image label. It remains unanswered what really goes wrong, is it hidden in the data or in the model or both!?
questions: - How does the saliency in input space correlate to the true image label?
- Could it be the findings are more relevant to datasets than models?
limitations: It would be interesting to see if similar results could be obtained with other NN architectures.
review_title: null

Meta-review: metareview: This work was already appreciated in its original version by all reviewers: it presents a simple novel idea, intuitive, and with high impact for the community and application-wise. The paper was considered well written and clear, methodological and experimental/evaluation parts are well developed and further improved after rebuttal, also providing additional experimental evidences. Some issues and requests of clarifications were of course highlighted by the reviewers, and the authors feedback was more than adequate, fully satisfying the raised concerns.
In the end, after rebuttal, all reviewers confirm the validity of the work and its acceptance to NeurIPS 2022.



confidence: Certain
Decision: Accept