Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability
Authors: Roman Levin
Manli Shu
Eitan Borgnia
Furong Huang
Micah Goldblum
Tom Goldstein
Venue: NeurIPS 2022 Accept (2022)
Keywords: parameter saliency
explainability
saliency maps

Abstract: Conventional saliency maps highlight input features to which neural network predictions are highly sensitive. We take a different approach to saliency, in which we identify and analyze the network parameters, rather than inputs, which are responsible for erroneous decisions. We first verify that identified salient parameters are indeed responsible for misclassification by showing that turning these parameters off improves predictions on the associated samples more than turning off the same number of random or least salient parameters. We further validate the link between salient parameters and network misclassification errors by observing that fine-tuning a small number of the most salient parameters on a single sample results in error correction on other samples which were misclassified for similar reasons -- nearest neighbors in the saliency space. After validating our parameter-space saliency maps, we demonstrate that samples which cause similar parameters to malfunction are semantically similar. Further, we introduce an input-space saliency counterpart which reveals how image features cause specific network components to malfunction.


Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability
Roman Levin
Manli Shu"
Department of Applied Mathematics University of Washington
Department of Computer Science University Maryland umd. edu
rilevin@uw. edu
Eitan Borgnia'
Furong Huang
Department of Computer Science
Department Computer Science University of Maryland furonghêcs. umd edu
University of Maryland
eborgnia20gmail com
Micah Goldblum
Tom Goldstein
Center for Data Science New York University
Department of Computer Science
University of Maryland tomg@cs umd. edu
wolyblum@nyu.. edu
Abstract
Conventional saliency maps highlight input features to which neural network predictions highly sensitive. We take different approach to saliency, in which we identify and analyze the network parameters, rather than inputs, which responsible for erroneous decisions. We first verify that identified salient parameters are indeed responsible for misclassification by showing that turning these parameters off improves predictions the associated samples, more than turning off the same number of random or least salient parameters. We further validate the link between salient parameters and network misclassification errors by observing that fine-tuning small number of the most salient parameters on a single sample results in error correction other samples which were misclassified for similar reasons nearest neighbors in the saliency space. After validating our parameter-space saliency maps, we demonstrate that samples which cause similar parameters to malfunction are semantically similar. Further, we introduce an input-space saliency counterpart which reveals how image features cause specific network components to malfunction.
Introduction
With the widespread deployment of deep neural networks in high-stakes applications such as medical imaging [20), credit
assessment [49], and facial recognition [10), practitioners need under- stand why their models make the decisions they do. In fact, "right to explanation" legislation in the European Union and the United States dictates that relevant public and private organizations must be able to justify the decisions their algorithms make [46] [4]. Diagnosing the causes of system failures particularly crucial for understanding the flaws and limitations of models intend to employ.
Equal contribution
I This work was completed prior to the author joining Amazon
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Average gredient magnitudes
0.3
0.2
0.1-
30
50
Layer ID
Figure 1: Filter-wise parameter salieney profile. ResNet-50 filter-wise saliency profile (without standardization) averaged over samples in ImageNet validation set. The filter saliency values in each layer are sorted in descending order, and each layer's saliency values are concatenated. The layers are displayed left-to-right from shallow to deep and have equal width on x-axis.
Conventional saliency methods focus on highlighting sensitive pixels 37 image regions that maximize specific activations [3]. However, such maps may not be useful in diagnosing undesirable model behaviors they do not necessarily identify areas that specifically cause bad performance since the most sensitive pixels may not be the ones responsible for triggering misclassification.
We develop an alternative approach to saliency which highlights network parameters that influence decisions rather than input features. These parameter saliency maps yield number of useful analyses:
We verify that identified salient parameters indeed responsible for misclassification by showing that turning these parameters off improves predictions on the associated samples more than turning off the same number of random least salient parameters.
Nearest neighbors in parameter saliency space share common semantic information. That is, samples which are misclassified for similar reasons and cause similar parameters to malfunction semantically similar.
We further validate the link between salient parameters and network misclassification errors by observing that correcting a small number of the most salient parameters by fine-tuning them single sample results in error correction on other samples which were misclassified for similar reasons.
By first identifying the network parameters responsible for an erroneous classification, we can then visualize the image regions that interact with those parameters and trigger the identified misbehavior obtaining interpretable insights into model mistakes and neural network's 's reliance on spurious correlations.
Our code for computing parameter-saliency maps is available https://gi com/ LerinRonan/paraanttrsssss sarecce-salenny...
1.1 Related Work
Neural network interpretability and parameter importance. A major line work in neural network interpretability focuses convolutional neural networks. Works visualizing, interpreting, and analysing feature maps [52 51 27 25 provide insight into the role of individual convolutional
filters. These methods, together with other approaches for filter explainability 7 56 57 find that individual convolutional filters often are responsible for specific tasks such as edge, shape, and texture detection. Other ideas in neural network interpretability include identifying recurring patterns in model behavior [8].
The ideas of measuring neural network parameter importance has been studied in multiple contexts. Notions of neuron (34], feature [26 35|
41 30 and parameter [40 47 importance have been used
for Al explainability, manipulating model behavior (6]. model debugging 24 53 and pruning
23 42 32 (50 for improving model performance.
Input space saliency maps. A considerable amount of literature focuses on identifying input features that are important neural network decisions. These methods include using deconvolution approaches (52) and data gradient information [37]. Several works build on these ideas and propose

---

0.0
kverage saleeccy profile of correctly classifted samples
Saliency
- -.2
Average saliency profile of ctty classified samples
Saliency
20
30
40
50
Layer ID
Figure 2: Standardized filter-wise saliency profiles, correctly incorrectly classified samples. Top: Standardized saliency profiles averaged over correctly classified samples in the ImageNet validation set. Bottom: Standardized saliency profiles averaged over incorrectly classified samples in the ImageNet validation set. On both panels, the filter saliency values in each layer are sorted in descending order, and each layer's salieney values are concatenated. The layers displayed left-to- right from shallow to deep and have equal width on x-axis. Both profiles are generated on ResNet-50.
improvements such as Integrated Gradients [44], SmoothGrad (38, and Guided Backpropagation [39] which result in sharper and more localized saliency maps. Other approaches focus on the use of class activation maps (55) with improvements incorporating gradient information 3d and more
novel approaches to weighting the activation maps 48] In addition. various saliency methods are
based on manipulating the input image [15 52 30 3 Another line of work is aimed at evaluating the
effectiveness of saliency maps and developing validation techniques
33
Our work combines the ideas of saliency maps and parameter importance and evaluates saliency directly on model parameters by aggregating their absolute gradients filter level. We leverage the resulting parameter saliency profiles as an explainability tool and develop an input-space saliency counterpart which highlights image features that cause specific filters to malfunction study the interaction between the image features and the erroneous filters.
Method
It is known that different network filters responsible for identifying different image properties and objects [52 51 27 255. This motivates the idea that mistakes made on wrongly classified images can be understood by investigating the network parameters, rather than only the pixels, that played role in making a decision. We develop parameter-space saliency methods geared towards identifying and analyzing neural network parameters that are responsible for making erroneous decisions. Central to our method is the use of gradient information of the loss function as a measure of parameter sensitivity.
2.1 Parameter Saliency Profile
Let a sample in the validation set D with label y, and suppose trained classifier has parameters 0 that minimize a loss function L. We define the parumeter-wise saliency profile of as vector s(x, y) with entries s(x, y) := Co v)). the magnitudes of the gradient of the loss with respect to each model parameter. Because the gradients on training data for a model trained to convergence are near zero, it is important to specify that be a validation, or holdout, set. Intuitively, a larger gradient norm at the point (z, y) indicates greater inefficiency in the network's 's classification of sample x, and thus each entry of s(x, y) measures the suboptimality of individual parameters.
Aggregation of parameter saliency. Convolutional filters are known to specialize in tasks such as edge, shape, and texture detection [5] 7 27 We therefore choose to aggregate saliency on the filter-wise basis by averaging the gradient magnitudes of parameters corresponding to each convolutional filter. This allows us to isolate filters to which the loss most sensitive (i.e. those which, when corrected, lead to the greatest reduction in loss).
3

---

Formally, for each convolutional filter Fss in the network, consider its respective index set dg which gives the indices of parameters corresponding to the filter FR (the number of parameters corresponding to Fk is then given by |ou|). The filter-wise saliency profile of sample x with label defined to vector s(, y) with entries
((z v)
s v):,
(I)
resulting from averaging the parameter-wise saliency profile of the sample with label y the filter level.
Standardizing parameter saliency. Figure I exhibits the ResNet-50 [8] filter-wise saliency profile averaged over the ImageNet [9] validation set, where filters within each layer are sorted from highest to lowest saliency. One clear observation is the difference in the scale of gradient magnitudes shallower filters are more salient than deeper filters. This phenomenon might occur for number of reasons. First, early filters encode low-level features, such as edges and textures, which are active across a wide spectrum of images. Second, typical networks have fewer filters in shallow layers than in deep layers, making each individual filter more influential at shallower layers. Third, the effects of carly filters cascade and accumulate as they pass through a network.
To isolate filters that uniquely cause erroneous behavior on particular samples, we find filters that are abnormally salient for a sample, x, but not for others. That is, we further standardize the saliency profile of 2 with respect to all filter-wise saliency profiees of D.
Formally, let μ be the average filter-wise saliency profile across all D, and let be equal- length vector with the corresponding standard deviation for each entry. We use these statistics to produce the standardized filter-wise saliency profile as follows:
3 (2 w
8(z, v)
(2
The resulting tensor $(z, y) is of length equal to the number of convolutional filters in the network, and we henceforth call it the saliency profile for sample x. By standardizing saliency profiles, we create saliency map that activates when the importance of a filter is unusually strong relative to other samples in the dataset. This prevents the saliency map from highlighting filters that are uniformly important for all images, and instead focuses saliency on filters that uniquely important and serve an mengeddeeendent role. In the rest of the paper, unless explicitly noted otherwise, we use 8( y) and refer to it as parameter saliency.
Incorrectly classified samples are more salient. Empirically, we observe the saliency profiles of incorrectly classified samples exhibit, average, greater values than those of correctly classified examples. This bolsters the intuition that salient filters are precisely those malfunctioning - the classification is correct, there should be few malfunctioning filters none at all. Moreover, we see deeper parts of the network appear to be most salient for the incorrectly classified samples while earlier layers often the most salient for correctly classified samples. An example of these behaviors for ResNet-50 is shown in Figure 2 which presents standardized filter-wise saliency profiles averaged over the correctly and incorrectly classified examples from the ImageNet validation set. Additionally, we note the improved relative scale of the standardized saliency profile across different layers compared to the absolute gradient magnitudes in Figure [ Saliency profiles for other architectures could be found in Appendix A Henceforth, we will focus specifically on saliency profiles of misclassified samples in order to explore how neural networks make mistakes.
2.2 Input-Space Saliency for Visualizing How Filters Malfunction
The parameter saliency profile allows us to identify filters that are most responsible for mistakes and erroncous network behavior. In this section, we develop an input-space counterpart our parameter saliency method to understand which features of the image affect the saliency of particular filters. Geiping et al. [17 show that the gradient information of network is invertible, providing link between input space and parameter saliency space. This work, along with existing input-space saliency map tools (37 39 38 55 34). inspires our method.
Given a parameter saliency profile = 8(x,, y) for an image x with label y, our goal is to highlight the input features that drive large filter saliency values. That is, we would like to identify image

---

Incorrect class contidenos
Ronfer
12
True class cucfdenze
Lear subeet
corrected samples
30
cbange 176
-10
change (8)
0.-
Percentage
-15-
-0.
25-
-1.0-
rallent
Nost adiess
Rander
Racdon
Leaat abin:
Least sabeot
80
20
60
100
20
d0
6o
106
Number of pruned fibbers
Number of pruned Eilters
Number of pruned filters
(a)
(c)
Figure 3: Effect of turning salient filters off. (a) Change in incorrect class confidence score. (b) Change true class confidence score. (c) Percentage of samples that were corrected as the result of pruning filters. These trends are averaged across all images misclassified by ResNet-50 in the ImageNet validation set. The error bars represent 95% bootstrap confidence intervals.
pixels altering which make filters more salient. To this end, first select set F of the most salient filters that we would like to explore. Then, we create boosted saliency profile 8'p by increasing the entries of & corresponding to the chosen filters F:
[Bi i kis
(5))
(ks, F,
k > 1 (we used k = 100 in our experiments). Now, we can find pixels that are important for making the chosen filters F more salient and, equivalently, making the filter saliency profile s(x, y) close to the boosted saliency profile s'p by taking the following gradients:
MF = V.Dc(((2..)) v).
(4)
where De(s)) .) is cosine distance.
The resulting input saliency map Mr contains input features (pixels) that affect the saliency of the chosen filters the most.
Experiments
In this section, aim validate the meaningfulness of our parameter saliency method. First, we verify that salient parameters are indeed responsible for misclassification by showing on the dataset level that turning them off improves predictions the associated samples more than turning off the same number of random or least salient parameters. This experiment is similar in spirit to removing salient features validate input-space saliency methods (5] 333. We then find that samples which cause similar filters to malfunction are semantically similar. We also show on the dataset level that fine-tuning small number of the most salient parameters error correction on other samples which were misclassified for similar reasons and cause similar parameters to malfunction. We then use our input-space saliency technique in conjunction with its parameter-space counterpart as an explainability tool to explore how neural networks make mistakes and how salient filters interact with visual input features.
single sample results in
We evaluate our saliency method the context image classification on CIFAR-10 21 and ImageNet 19. Images
for visualization, unless otherwise specified, sampled from ImageNet validation set. Throughout the experiments, we use pre-teri ResNet-18 18 classifier [8 on CIFAR-10 and a pre-trained ResNet-50 ImageNet. Both models trained in a standard fashion on the corresponding datase
3.1 Turning Off Salient Filters
We begin validating parameter-space saliency maps by observing the effect of turning off the salient filters. In order to remove the influence of a particular salient filter, we prune - zero out
https://githubbcc "hanps//gituub..om//unagg http:///jtttl.cmm/uuunnaaahorbrr (under MIT license)
bttps://gthhib.ccopyyrrrrrrheeee (under BSD 3-Clause License)
5

---

(a) Great pyrenees < Kuvasz
(b) Basset hound < Beagle
Figure 4: Examples of nearest neighbors in parameter saliency space (from ImageNet).
the filter weights and also the biases of the associated batch normalization layers. This procedure guarantees that the corresponding input feature map to the next convolutional layer is always zero.
We gradually increase the number of pruned most salient filters and track three metrics: the change in the incorrect class confidence, the change in the true class confidence, and the percentage of the samples that flip their label to the correct class. In every case, we compare pruning the most salient filters against pruning the same number of random filters and the least salient filters. These experiments are performed on the dataset level: we average the trends across all misclassified images in the ImageNet validation set.
As shown in Figure 3 pruning the most salient filters significantly more effective for decreasing the incorrect class confidence than random least salient filters. Specifically, gradually pruning the top 100 salient filters achieves up to 30% drop in the incorrect class confidence score while pruning random filters yields only about 7% decrease. We also note that pruning the least salient filters does not produce any effect on the incorrect class confidence.
We repeat the same experiment with the true class confidence and observe that the highest true class confidence gain occurs when we prune around 20 most salient filters. Pruning enough salient filters eventually leads to a gradual decrease in the true class confidence. We note that this behavior is expected since we are destroying, not correcting, the inference power of all the most sensitive filters for an image, some of which may be essential for inference. Finally, pruning random filters provides a much slower increase in the true confidence class while the least salient filters again do not produce significant effect.
In addition, we count the number of images that were corrected as a result of pruning and find that pruning around 30 most salient filters results in the best correct classification rate of 12%. Similar to the true class confidence, the trend decreases beyond this point. Pruning random filters increases the percentage of corrected samples at much slower rate and does not perform better than the most salient filters when pruning up to 100 filters. Notably, pruning the least salient filters manages to correct nontrivial number of samples but still much smaller than pruning random filters.
These experiments suggest that the identified salient parameters are indeed responsible for misclassi- fication since turning these parameters off improves predictions on the associated samples more than turning off the same number of random or least salient parameters.
3.2 Nearest Neighbors in Parameter Saliency space
We validate the semantic meaning of our saliency profiles by clustering images based on the cosine similarity of their profiles. In this section, we present visual depictions of nearest neighbor search among all images in the ImageNet validation set. We also conduct this analysis CIFAR-10 images, and this can be found in Appendix
We find that the nearest neighbors of misclassified images in saliency space are mostly other mis- classified images from the same pair of predicted and true classes but possibly in reverse order. For example, in Figure 4 the reference image in (a) is great Pyrenees misclassified as kuvasz, and the 4 images with the most similar profiles exhibit either the same misclassification or the reverse (i.e., kuvasz misclassified as great Pyrenees). Intuitively, the common salient parameters across these neighbors are those which are important for discriminating between the two classes question not well-tuned for this purpose.

---

Figure 5: Neighbors in parameter saliency space found using only early or only deep layers. The reference image is in the first column. Images in the top row resemble the reference image in the saliency on early layers of VGG-19 19, and images in the bottom row are found using deeper layers.
Nwongg perceri ofcomnerit beneeeeeeeeeeeeoo (P4)
Procuuue of onnerrt samples (83)
Levi
Tale codf change
300
113
10
100
number of tunable filters
number of tunable filters
number of tunahle filters
(b)
Figure 6: Effect of updating a limited number of filterr. (a) Percentage of misclassified samples corrected after fine-tuning. (b) Average percentage of nearest neighbors that are also corrected after fine-tuning. (c) Average change in the confidence score of the true class among nearest neighbors. The horizontal dashed line each plot is the effect of updating the entire network.
Note that we find the concept of "being similar" in parameter saliency space to different from the one in image space. The nearest neighbors we find are often not similar in pixel-wise sense, but rather they are similar their reason for causing misclassification. For example, images Figure (b) are beagles mistaken by network for basset hounds and vice versa. We find that these pictures are either taken from high angle or do not include the dog's 's legs, making the leg length, a major distinction between the two breeds, indistinguishable from the picture. We include more example images along with their nearest neighbors in Appendix A
In addition, we compute nearest neighbors when only considering filters in specific range of layers in order to visualize the types of misbehavior triggered by network components (filters) at various network depths. We search for similar images using parameter saliency in the shallow and deep layers of VGG-19 19 network [3)], which we divide into the shallow and deep parts that respectively occur up to and after layer relu4_1. The top row of figure Figure 51sowss neighbors found using shallow parameters, which share basic image attributes such as color histogram, while images the bottom row share more abstract similarities.
3.3 Correcting Mistakes by Fine- Fine-Tuning Salient Filters
To validate that salient filters are more responsible for the erroneous behavior of neural networks, we show that updating a limited number of salient filters on single misclassified image can correct the mistakes made by neural network on other images that were misclassified because of similarly malfunctioning filters experiment we fine-ture tune-tunea pretrained ResNet-50 for one step on single image for which the network makes a wrong prediction and track the model's performance on the image's nearest neighbors found through the process introduced in Section 3.2 While filter saliency profiles are standardized with respect to the ImageNet validation set, we find the nearest neighbors from an independent test set ImageNet-v2, collected by Recht et [31 separately from the original ImageNet data. This way, the label information of the nearest neighbors is not used in fine tuning in any way.
nearest neighbors of that image in the parameter saliency space. In this
7

---

Most calwen
Fiber selerryy (54)
-20
-40
-00-
-50
Maaking stop
(8) Reference image
(b) Input-space salieney
(c) Saliency values change
4.07%
great white shark
the most erroneous filters
78.36
killer whale
masking experiments
(d) Mask seal
(c) Mask boat and seal
(f) Extended mask
11.83%
great white shark
43.97%
great white shark
58. 58.73%
great white shark
53.47%
killer whale
24.24%
whale
whale
Figure 7: Interaction between input features and salient filters. (a) Reference image of "great white shark" misclassified by ResNet-50 "killer whale" with confidence scores. (b) Input-space saliency visualization. Pixels that cause the top 10 salient filters to have high saliency. (c) Change in saliency values of the erroneous filters across masking experiments. The vertical bars represent the standard deviation of the change across 10 most salient filters. (d)-(f) Masking experiments.
restrict the number tunable filters to be no more than 1.0% of the total number of filters the network, and we update the chosen filters by taking step of gradient descent with normalized step. We compare tne-ttunng the most salient filters with two other choices of tunable filters: the least salient filters and random filters. We misclassified images from the ImageNet validation set, making up to over 10,000 samples. We evaluate the effect of fine-tuning on each of these images independently and find the corresponding nearest neighbors from holdout ImageNet-v2 test set where we limit the search scope exclusively to misclassified images too.
In Figure 6 compare the average performance of our three choices of tunable filters under three metrics. Panel (a) presents a the percentage of samples that are corrected after fine-tuning as sanity check and we see that updating 150 most salient filters (~0.6% 6% of total filters) is enough achieve the same percentage of corrections as updating the entire network.
The second and third metrics evaluate the effect on the nearest neighbors of the training sample (Figure 6 (c). Note that for given training sample, its nearest neighbors are not involved in our one-step single sample fine-tuning process. By tracking model predictions and true class confidence scores among the nearest neighbors of each sample, we find that fine-tuning salient filters significantly more effective than other choices. Results in Figure (b) and (c), also imply that the nearest neighbors found using our method are the images that are wrong for similar reasons and that
they can be corrected altogether by only updating the salient filters on single image. We note that we not propose new pruning or fine-tuning method. Rather, we use these experiments to verify that the salient filters are indeed responsible for misclassification.
3.4 Interpretable Network Failures: Visualizing Input Features Which Trigger Filter Malfunctions
Case study. To present example of how our approach could be used explainability tool, we begin this section with case study of an image misclassified by ResNet-50 as "killer whale" (Figure 7 aa).. The correct label of the image is "great white shark". Our goal is to study the interaction between

---

the most salient filters and input features. We first identify filters most responsible for misclassificatiin by computing the filter saliency profile and visualize parts of the image that drive the high saliency values for those filters using the input-space saliency counterpart (as introduced in Section 2.2
Panel (b) of Figure 7 presents our image-space visualization for the most salient filters the pixels which trigger misbchavior in these filters are highlighted. For example, we see that the seal and boat are both triggers. One natural hypothesis is that the seal looks like a killer whale to the network and is the source of the classification error. We check this by masking out the seal ( However, although the probability of "killer whale" goes down and the probability of the correct class increases, the network still misclassifies the image as "killer whale".
Now, if mask out exactly the most salient areas of the image according to visualization (see Figure [ (b), (e)); the network manages to flip the label of the image and classify it correctly. If we extend our mask to the less pronounced, but still salient, areas of the image in Figure 7 (f), we observe that the correct class confidence increases even more while the probability of the incorrect "killer whale" label further decreases. Additionally, we find that masking out the non-salient parts of the image results in even worse misclassification confidence than that of the original image (see Appendix [A]. In order to further investigate the effect of the salient region, we pasted it from this image onto other great white shark images (see Appendix A) and observed that this drives the probability of "killer whale" up for 39 out of 40 examples of great white sharks from the ImageNet validation set with an average increase of 3.75%.
Our experiments suggest that secondary objects in the image are associated with the misclassification. However, we see that the erroneous behavior of the model does not just stem from classifying a non-target object in the image. It is possible that the model correlates the combination of sea creatures (e.g. a seal) and man-made structures (e.g. a boat) with the "killer whale" label. We note that images of killer whales in ImageNet often have man-made structures which look similar the boat (see Appendix A for examples).
Finally, at each step of our masking experiments, we recompute the saliency values of the originally chosen 10 filters (i.e. the filters that caused erroneous behavior on the reference image). From Figure (c), we observe that as we mask out the input features according to our input-saliency. the saliency values of those filters decrease gradually and reach an 80% drop, confirming that highlighted regions indeed drive the high saliency of the chosen filters.
Dataset-level masking experiment. We performed the experiment of masking the foregroddd image areas on the dataset level and observed similar trends to the case study. Specifically, we trained a ResNet-50 on half of the ImageNet train set, and then used the incorrectly classified images from the other half of the train set with target object bounding boxes for the experiment (we used annotations from the ImageNet website We computed our input saliency for every image and masked out the most salient regions of those images while preserving the target object. We observed that as result of masking, the softmax output corresponding to the correct class increases (p < 0.05) on average and decreases for the incorrect class (p- 0.05), while masking the same number of randomly selected pixels (similarly outside of the target object bounding box) does not produce a statistically significant change in either correct or incorrect class confidence scores (p 0.854 and p = 0.695, respectively).
Figure g showcases input space visualizations with different illustrative examples of neural network mistakes. For a thorough discussion of mistake categories we find using our method, we refer to Appendix A.9
4 Discussion
Numerous applications demand that practitioners be able to understand the decisions their models make, especially when those decisions are incorrect. Existing methods for explainability focus on locating the input regions to which the network's 's output is sensitive or on associating network components with specific roles. In contrast, develop framework for finding the exact filters which are responsible for faulty predictions and studying the interactions between these filters and images. This direction yields an interpretable understanding of model behaviors.
htaps///magg-nnttorr/doooo bbx.eadddddbbxepppppppop
9

---

(a)
dalmation soccer hall
(h)
pizza
(c)
loopadd
(d)
junco
passenger car Incomotive
bell pepper
jaguar
house finch
Figure 8: Different types of network mistakes. All of the presented images are misclassified by ResNet-50. The correct class label is specified in the top row and the incorrect class label - in the bottom row of the subcaption on each panel. (a)-(b) The target object is confused with another object in the image. (c) A regular mistake. The salient pixels are focused the target object features which confuse the network. (d) Background features confuse the network. (e) An example of a noisy label where the network is "more correct" than the target label. These are examples where masking top 5% of the salient pixels corrects the misclassification.
Acknowledgments
This work was supported by the DARPA YFA program, DARPA GARD, the ONR MURI program, the National Science Foundations Division of Mathematical Sciences, the AFOSR MURI program, and Capital One Bank.
References
[1] R. Abbasi-Asl and Yu. Interpreting convolutional neural networks through compression. arXiv preprint
1.02329, 2017.
[2] J. Adebayo, J. Gilmer, M. Muelly. I. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. arXiv preprint arXiv:1810.03200 2018.
[3] C. Agarwal and A. Nguyen. Explaining image classifiers by removing input features using generative models. In Proceedings of the Asian Conference on Computer Vision, 2020.
[4] A. Alqaraawi, M. Schuessler, P. WeiB, E. Costanza, and N. Berthouze. Evaluating saliency map explanations for convolutional neural networks: user study. In Proceedings of the 25th International Conference on Intelligent User Interfaces, pages 275-20 775-285, 2020.
[5] L. Arras, G. Montavon, K.-R. Müller, and W. Samek. Explaining recurrent neural network predictions in sentiment analysis. arXiv preprint arXiv: 0720000660070.6. 2017.
[6] A. Bau, Belinkov, H. Sajjad, N. Durrani, F. Dalvi, and Glass. Identifying and controlling
important neurons in neural machine translation. arXiv preprint
2018.
[7] D. Bau. B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6541-6549, 2017.
[8] A. Boggust, B. Hoover, A. Satyanarayan, and H. Strobelt. Shared interest: Measuring human-ai alignment to identify recurring patterns in model behavior. In CHI Conference on Human Factors in Computing Systems, pages 1-17, 2022.
[9] J. Deng, W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[10] J. Deng, J. Guo, N. Xue, and S. Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 4690-4699, 2019.
10

---

[11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018
[12] Du, Yan, Feng, J. T. Zhou, L.. Zhen, R. M. Goh, and Y. Tan. Efficient sharpness- aware minimization for improved training of neural networks. arXiv preprint arXhv:2110.03141, 2021.
[13] D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
[14] European Commission. Recital 71 EU General Data Protection Regulation. 2018. URL
https://www http://ww....aaayyrraaatiaiiiaaa pzt1aaggrrggplttttoooe///a///// eu/en/r71. 20mmmm
[15] R. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In ICCV, 2017.
[16] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur. Sharpness- aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
[17] Geiping, H. Bauermeister, H. Dröge, and M. Moeller. Inverting gradients-hov how easy is it to break privacy in federated learning? arXiv preprint arXiv:2003, 240553 2020.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[19] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[20] E. Kang, J. Min, and J. C. Ye. A deep convolutional neural network using directional wavelets for low-dose x-ray et reconstruction. Medical physics, 44(10)=e30-<<35. 2017.
[21] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[22] Kwon, J. Kim, H. Park, and I. K. Choi. Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. arXiv preprint arXiv:2/02. /6000, 2021.
[23] C. Liu and H. Wu. Channel pruning based mean gradient for accelerating convolutional neural networks. Signal Processing, 156:84-91, 2019.
[24] S. Ma, Y. Liu, W.-C. Lee, X. Zhang, and A. Grama. Mode: automated neural network model debugging via state differential analysis and input selection. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 175-186, 2018
[25] A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015.
[26] A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, and M. Botvinick. On the importance of single directions for generalization. arXiv preprint 1rXiv:1803.06953 2018
[27] C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2017. doi: 10.23915/ distill.00007. https://distill.pubb pttp:2////ffauuueviiiiiiiiiiiii
[28] Petroni, Rocktäschel, A. H. Miller, P. Lewis, A. Bakhtin, Y. Wu, and S. Riedel. Language models as knowledge bases? In In: Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing (EMNLP), 2019, 2019.
[29] F. Petroni, P. Lewis, A. Piktus, T. Rocktäschel, Y. Wu, A. H. Miller, and S. Riedel. How context affects language models' factual predictions. In Automated Knowledge Base Construction, 2020. URL WtL/https:///c/t http://://////ee//eeeeeeeee net/ tott/fooummidd005502PP
[30] V. Petsiuk, A. Das, and K. Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprini arXiv:1806.07422/ 2018.
[31] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do ImageNet classifiers generalize to ImageNet? In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, pages 5389-5400. PMLR. 2019.
[32] M. Sabih, F. Hannig, and J. Teich. Utilizing explainable ai for quantization and pruning of deep neural networks. arXiv preprint arXiv:2008.09072 2020.

---

[33] W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and K.-R. Müller. Evaluating the visual- ization of what deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(1 1):2660-2675 2016.
[34] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
[35] A. Shrikumar, P. Greenside. and A. Kundaje. Learning important features through propagating activation differences. In ICML, 2017.
[36] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[37] K. Simonyan, Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR, 2014.
[38] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint ar 1706 1706.0770,,,5 2017.
[39] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. In ICLR, 2015.
[40] S. Srinivas and F. Fleuret. Full-gradient representation for neural network visualization. arXiv preprint arvv Teti::900.0000 1905.00780, 2019.
[41] E. Štrumbelj and I. Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems, 41(3):647-663 2014.
[42] X. Sun, X. Ren, S. Ma. and H. Wang, meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting. In International Conference on Machine Learning, pages 3299 -3008. PMLR, 2017.
[43] X. Sun, Z. Zhang, X. Ren, R. Luo, and L. Li. Exploring the vulnerability of deep neural networks: A study of parameter corruption. arXiv preprint arXiv:2006. 2xXiv:2006.05620, 2020.
[44] M. Sundararajan, A. Taly, and Q Yan. Axiomatic attribution for deep networks. In ICML, 2017.
[45] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 28 8-2266, 2016.
[46] United States Congress Senate Committee on Banking and Housing and Urban Affairs. Equal Credit Opportunity Act. [electronic resource]. S. Rpt. 94-685. Washington, 1976.
[47] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint 1wX.:::905.004110 2019.
[48] H. Wang, Z. Wang, M. Du, F. Yang, Z. Zhang, S. Ding, Mardziel, and X. Hu. Score-cam: Score-weighted visual explanations for convolutional neural networks. In CVPR, 2020.
[49] D. West. Neural network credit scoring models. Computers & Operations Research, 27(11-12): 1131-1152, 2000.
[50] S.-K. Yeom, P. Seegerer, S. Lapuschkin, A. Binder, S. Wiedemann, K.-R. Müller, and W. Samek. Pruning by explaining: novel criterion for deep neural network pruning. Pattern Recognition, 115:107899, 2021.
[51] Yosinski, Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv: 06579, 2015.
[52] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818-833. Springer. 2014.
[53] H. Zhang and W. Chan. Apricot: A weight-adaptation approach to fixing deep learning models. In 2019 34th IEFF/ACM International Conference on Automated Software Engineering (ASE), pages 376-387. IEEE, 2019.
[54] J. Zhang, S. A. Bargal, 7. Lin, J. Brandt, X. Shen, and S. Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 1 126( 1084-11000 2018.
[55] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 2921-2929, 2016.
12

---

[56] B. Zhou, D. Bau, A. Oliva, and A. Torralba. Interpreting deep visual representations via network dissection. IEEE transactions on pattern analysis and machine intelligence, 41(9):2131-2155, 2018.
[57] B. Zhou, D. Bau, A. Oliva, and A. Torralba. Comparing the interpretability of deep networks via network dissection. Explainable Interpreting, Explaining and Visualizing Deep Learning, pages 243-252. Springer, 2019.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes], [No] or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix C
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix C
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We include the code in the supplementary material well detailed instructions how it.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix B
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See footnotes
(c) Did you include any assets either in the supplemental material We include the code in the supplementary material as well as detailed instructions on how to use it.
URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] We do not work with human subjects or with user data
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We do not work with human subjects with user data
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "nrC-y2ZY8jx", "reviewer": "Reviewer_PwHG", "summary": "\"Where do Models go Wrong\" presents a saliency method that highlights model filters important to the output. The paper validates these parameters are salient by turning off the salient parameters on misclassified predictions and showing an increase in corrected classification. They evaluate the semantic meaning of parameter space saliency by quantifying each input by its parameter saliency and analyzing the resulting space. The paper finds neighbors often share misclassification causes instead of visual similarities, showing that parameter saliency captures patterns in model behavior. Parameter saliency can also identify filters to fine-tune as a way to correct model mistakes. Finally, the paper shows how to use parameter saliency to analyze an image in an explainability task by masking various portions of the image related to salient parameters.", "questions": "* The proposed parameter saliency method computes the gradient of the loss with respect to each model parameter. This seems equivalent to Vanilla Gradients applied to model parameter space. Is it possible to use variants of other input-space saliency methods to compute parameter-space saliency (e.g., SmoothGrad)? If so, have you tried any, and how did they compare?\n* Could you use parameter saliency to prune models for model compression? Have you tried this or compared it to other pruning techniques?\n* Can you provide a similar analysis for correctly classified inputs? It would be valuable to confirm that pruning salient filters decreases model output confidence for correctly classified images. Similarly, it would be interesting to see if the nearest neighbors of correct classifications are other correct classifications. Are neighbors from correctly classified images more likely to be in the same class, or would we find neighbors with similar confidence in the output class?", "limitations": "The appendix provides an interesting and thorough discussion about adversarial attacks, a comparison to GradCAM, and limitations. I would suggest moving at least some of this to the main text as it provides compelling context and considerations for the method. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* *Experimental baselines* --- The paper evaluates the saliency map by pruning salient filters and measuring the change to the model's output. They compare this procedure to pruning random filters and pruning the least salient filters. These comparisons show a significant difference in the model's output when salient filters are pruned.\n* *Thorough evaluation* --- The paper validates their results on various models architectures trained on image and language modalities. They also test their saliency method using existing model randomization tests and find it model dependent.\n* *Clarity* --- The paper is very well written and precise.\n\n**Weaknesses**\n* *Analysis of other saliency methods* --- The proposed parameter saliency method computes the gradient of the loss with respect to each model parameter and aggregates the gradients per filter. While other methods could compute parameter saliency, the paper does not include evaluations of any alternatives. It would be informative to see how consistent the parameter saliency is between methods. For instance, SmoothGrad could be used in this method by perturbing the input features but taking the gradient with respect to the model parameter.\n* *Analysis of correctly classified samples* --- The method focuses on identifying filters contributing to misclassification. While this is undoubtedly important to identify and fix model errors, it is equally important to understand why the model made a correct classification. For example, it is critical to know why a model correctly predicted disease in a medical setting.\n\n**Related Work**\n* *Model error patterns* --- Figure 8 classifies different types of mistakes based on where the saliency is with respect to the main object. \"Shared Interest: Measuring Human-AI Alignment to Identify Recurring Patterns in Model Behavior\" by Boggust et al. has a similar metric for identifying and measuring these patterns. You might consider citing this method or using it to compute the Shared Interest of the different mistake categories you identified.\n* *Saliency method evaluation* --- I appreciate that you evaluate your saliency method via the model parameterization test in the appendix. You might also consider discussing or validating your input-wise saliency on other proposed tests like the pointing game (\"Top-down Neural Attention by Excitation Backprop\" by Zhang et al.) \n* *Removing salient parameters experiments* --- The experiment removing saliency parameters to validate parameter-space saliency is analogous to removing salient features to validate input-space saliency. These validation approaches have been proposed in \"Explaining recurrent neural network predictions in sentiment analysis\" by Arras et al. and \"Evaluating the visualization of what a deep neural network has learned\" by Samek et al.\n\n**Minor Issues**\n* Figures 3, 6, and 7 are too small and hard to read.\n* Please move figures closer to their corresponding text. Having figures pages away from the reference text is challenging to read.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "U0eYGbiGAK8", "reviewer": "Reviewer_ywxe", "summary": "In this paper, the authors analyze salient network parameters, which are responsible for erroneous decisions. They conduct experiments to find salient parameters responsible for faulty predictions and show the relation between these parameters and regions in the input image.", "questions": "How is this paper different from adversarial example?", "limitations": "I believe there would be no potential negative societal impact.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Instead of focusing on saliency maps like other work, this paper pays attention to salient network parameters, and the findings are interesting.\n\nWeaknesses: I am not expert in this research field. However, I have a concern about the novelty of this paper. Parameter saliency is not new, and it’s already done in adversarial example research. The authors also do not clarify differences between this paper and adversarial example.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ov6m19KAeBO", "reviewer": "Reviewer_N8GQ", "summary": "This paper approaches the topic of network explainability and visualization with a novel approach for isolating the factors contributing to a model's behaviour that focuses on network parameters rather than the previously more common approach of identifying image regions of highest impact. This is a subtle but non-trivial change, and leads to a number of useful new analyses of network behaviour, and in particular shows great promise in selectively training network parameters to correct errors.", "questions": "I do not have any questions beyond the minor points raised above. The paper was overall clearly written.", "limitations": "I do not foresee any negative societal impacts as a direct result of this work.", "rating": 9, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper presents a novel idea that is intuitive (and almost obvious in retrospect, but many great ideas seem so!) and quite broad in application and effect. The experiments that are performed are easy to follow and clearly developed, offering strong evidence for the potential of this method, particularly in terms of targeted tuning and updating of network behaviour.\n\nWeaknesses:\n- Perhaps I missed it, but what is the effect on network behaviour for objects that are not neighbours to the misclassified sample? Are new errors introduced (and if so, at what rate? Is this still a strong net benefit, or are we trading one set of errors for another)?\n- The visualization aspect of this method appears to essentially still end up looking similar to class activation maps in methods like Grad-CAM (i.e. it gets projected down to a pixel-wise map showing image regions that are most affecting the model behaviour on the given sample). While I am not sure that this is in and of itself a major weakness in the method (it is an intuitive visualization technique), the lack of comparison between the visualizations produced by the proposed method with prior image region based methods (e.g. Grad-CAM) makes it harder to evaluate their role and effectiveness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "51s6dRf5H9A", "reviewer": "Reviewer_pViJ", "summary": "An interesting paper to establish a link between parameter space saliency and input saliency for the purpose of diagnosis of misclassification in deep NNs. The paper demonstrates that by turning salient filters on/off the prediction of NNs could be corrected. ", "questions": "- How does the saliency in input space correlate to the true image label?\n- Could it be the findings are more relevant to datasets than models?", "limitations": "It would be interesting to see if similar results could be obtained with other NN architectures.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses an interesting question that is to study the relation between saliency and network predictions. The approach is thought provoking. Nonetheless, the paper is  vague about the details when it comes to saliency profile calculation and could have a better representation. The experiments are limited but convincing with respect to the conclusion. There is, however, an overlooked area, in my opinion. That is the dataset bias and label noise, when the most visually salient input is not the same as the image label. It remains unanswered what really goes wrong, is it hidden in the data or in the model or both!?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept