Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability
Authors: Roman Levin
Manli Shu
Eitan Borgnia
Furong Huang
Micah Goldblum
Tom Goldstein
Venue: ICLR 2022 Submitted (2022)
Keywords: explainability
interpretability
saliency maps
parameter saliency

Abstract: Conventional saliency maps highlight input features to which neural network predictions are highly sensitive. We take a different approach to saliency, in which we identify and analyze the network parameters, rather than inputs, which are responsible for erroneous decisions.  We first verify that identified salient parameters are indeed responsible for misclassification by showing that turning these parameters off improves predictions on the associated samples, more than pruning the same number of random or least salient parameters.  We further validate the link between salient parameters and network misclassification errors by observing that fine-tuning a small number of the most salient parameters on a single sample results in error correction on other samples which were misclassified for similar reasons -- nearest neighbors in the saliency space. After validating our parameter-space saliency maps, we demonstrate that samples which cause similar parameters to malfunction are semantically similar.  Further, we introduce an input-space saliency counterpart which reveals how image features cause specific network components to malfunction.

Under review as a conference paper at ICLR 2022
WHERE DO MODELS GO WRONG? PARAMETER-SPACE SALIENCY MAPS FOR EXPLAINABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
Conventional saliency maps highlight input features to which neural network predictions are highly sensitive. We take a different approach to saliency, in which we identify and analyze the network parameters, rather than inputs, which are responsible for erroneous decisions. We first verify that identified salient parameters are indeed responsible for misclassification by showing that turning these parameters off improves predictions on the associated samples, more than pruning the same number of random or least salient parameters. We further validate the link between salient parameters and network misclassification errors by observing that fine-tuning small number of the most salient parameters on a single sample results in error correction other samples which were misclassified for similar reasons - nearest neighbors in the saliency space, After validating our parameter-space saliency maps, we demonstrate that samples which cause similar parameters to malfunction are semantically similar. Further, we introduce an input-space saliency counterpart which reveals how image features cause specific network components to malfunction.
INTRODUCTION
With the widespread deployment of deep neural networks in high-stakes applications such medical imaging (Kang et al. 2017). credit score assessment (West 2000), and facial recognition (Deng et al. (019), practitioners need understand why their models make the decisions they do. In fact,
"right to explanation" legislation in the European Union and the United States dictates that relevant
public and private organizations must be able to justify the decisions their algorithms make (United
States Congress Senate Committee on Banking and Housing and Urban Affairs 1976 European
Commission 2018). Diagnosing the causes of system failures is particularly crucial for understanding the flaws and limitations of models we intend to employ.
Conventional saliency methods focus on highlighting sensitive pixels (Simonyan et al. 2014) or image regions that maximize specific activations (Erhan et al.] 2009). However, such maps may not be useful in diagnosing undesirable model behaviors as they do not necessarily identify areas that specifically cause bad performance since the most sensitive pixels may not be the ones responsible for triggering misclassification.
We develop an alternative approach to saliency which highlights network parameters that influence decisions rather than input features. These parameter saliency maps yield a number of useful analyses:
Nearest neighbors in parameter saliency space share common semantic information. That is, samples which are misclassified for similar reasons and cause similar parameters to malfunction are semantically similar.
By first identifying the network parameters responsible for an erroneous classification, we can then visualize the image regions that interact with those parameters and trigger the identified mishehavior.
We verify that identified salient parameters are indeed responsible for misclassification by showing that turning these parameters off improves predictions on the associated samples, more than pruning the same number of random or least salient parameters.
We further validate the link between salient parameters and network misclassification errors by observing that fine-tuning small number of the most salient parameters single

---

Under review as a conference paper at ICLR 2022
sample results in error correction other samples which misclassified for similar reasons.
After carefully delineating our methodology and experimentally validating the meaningfulness of parameter saliency maps, we showcase the practical utility of this paradigm explainability tool with a case study in which we are able to uncover a neural network's reliance on a spurious correlation which causes interpretable failures.
1.1 RELATED WORK
Neural network interpretability and parameter importance. major line of work in neural
network interpretability focuses on convolutional neural networks, Works visualizing, interpreting. and analysing feature maps (Zeiler Fergus 2014 Yosinski et al. 2013 Olah et al. 2017 |017 Mahendran
& Vedaldi, 2015) provide insight into the role of individual convolutional filters. These methods, together with other approaches for filter explainability (Bau et al. 2017 Zhou et al.)
2018 2019) find that individual convolutional filters often are responsible for specific tasks such edge, shape, and texture detection.
The idea of measuring neural network parameter importance has been studied in multiple contexts.
Notions of neuron and parameter importance have been used for Al explainability (Srinivas & Fleuret 2019 Selvaraju al. 2017; Morcos al. 2018 Shrikumar 2017 Shrikumar et al.).
manipulating model behavior [Bau et al. 2018). and parameter pruning (Abbasi-Ae Asl & Yu 2017
Wu 2019).
Input space saliency maps. A considerable amount of literature focuses on identifying input features that are important for neural network decisions. These methods include using deconvolution approaches (Zeiler & Fergus 2014) and data gradient information (Simonyan et al. (014)). Several works build on these ideas and propose improvements such as Integrated Gradients (Sundararajan et al. 2017). SmoothGrad (Smilkov et al.) 2017). and Guided Backpropagation (Springenberg et al. 2015 which result sharper and more localized saliency maps. Other approaches focus on the use of class activation maps (Zhou et al. 2016) with improvements incorporating gradient information (Selvaraju et al., 2017) and more novel approaches to weighting the activation maps (Wang et al.) 2020). In addition, various saliency methods are based manipulating the input image (Fong & Vedaldi 2017] Zeiler & Fergus, 2014). Another line of work is aimed at evaluating the effectiveness of saliency maps (Adebayo al. 2018 Allaraawi al.) 2020).
Although extensive work studies how different regions of images affect a network's predictions, limited work (Srinivas & Fleuret (019)) aims distinguish important network parameters. Our work combines the ideas of saliency maps and parameter importance and evaluates saliency directly on model parameters by aggregating their absolute gradients on filter level. We leverage the resulting parameter saliency profiles as an explainability tool and develop an input-space saliency counterpart which highlights image features that cause specific filters to malfunction to study the interaction between the image features and the erroneous filters.
METHOD
is known that different network filters are responsible for identifying different image properties and objects (Zeiler & Fergus 2014 Yosinski et al.) 2015 Olah et al. 20177: Mahendran & Vedaldi 2015). This motivates the idea that mistakes made on wrongly classified images can be understood by investigating the network parameters, rather than only the pixels, that played role in making decision. We develop parameter-space saliency methods geared towards identifying and analyzing neural network parameters that are responsible for making erroneous decisions. Central to our method is the use of gradient information of the loss function as measure of parameter sensitivity and optimality of the network at a given point in image space.
2.1 PARAMETER SALIENCY PROFILE
Let x be a sample in the validation set D with label y. and suppose trained classifier has parameters i that minimize loss function L. We define the parameter-we saliency profile of vector
2

---

Under review as a conference paper at ICLR 2022
0.4
Average gredient magnitudes
Sallency
0.3
0.2
Layer ID
Figure 1: Filter-wise parameter saliency profile. ResNet-50 filter-wise saliency profile (without standardization) averaged over samples in ImageNet validation set. The filter saliency values in each layer are sorted in descending order, and each layer's 's saliency values are concatenated. The layers displayed left-to-right from shallow to deep and have equal width x-axis.
s(r, y) with entries s(z, y); == VV, (x; y)l, the magnitudes of the gradient of the loss with respect to each model parameter. Because the gradients on training data for model trained convergence are near zero, it is important to specify that D) be a validation, or holdout, set. Intuitively, a larger gradient norm at the point (z, y) indicates greater inefficiency in the network's s classification of sample x, and thus each entry of s(x v) measures the suboptimality of individual parameters.
Aggregation of parameter salieney. Convolutional filters are known specialize in tasks such as edge, shape, and texture detection Yosinski et al. 2015 Bau et al. 2017; Olah et al., 2017). We therefore choose to aggregate saliency on the filter-wise basis by averaging of parameters corresponding to each convolutional filter. This allows us to isolate filters to which the loss most sensitive (i.e. those which, when corrected, lead to the greatest reduction in loss).
gradient magniudess
Formally, for each convolutional filter F& in the network, consider its respective index set ag, which gives the indices of parameters corresponding to the filter Fu. The filter-wise saliency profile is defined to be vector s(x, y) with entries
(2 v)
s(x, v)ir
lagel
iEa
the parameter-we wise saliency profile aggregated by averaging the filter level.
Standardizing parameter saliency. Figure exhibits the ResNet-50 (He et al. 2016) filter-wise saliency profile averaged over the ImageNet (Deng et al. (209) validation set, where filters within each layer are sorted from highest to lowest saliency. One clear observation the difference in the scale of gradient magnitudes shallower filters might occur for a number of reasons. First, early filters encode low-level features, such as edges and textures, which are active across wide spectrum of images. Second, typical networks have fewer filters in shallow layers than in deep layers, making each individual filter more influential at shallower layers. Third, the effects of early filters cascade and accumulate as they pass through a network.
salient than deeper filters. This phenomenon
To isolate filters that uniquely cause erroneous behavior on particular samples, we find filters that are abnormally salient for a sample, x, but not for others. That is, we further standardize the saliency profile of & with respect to all filter-wise saliency profiles of D.
Formally, let μ be the average filter-wise saliency profile across all D, and let be an equal- length vector with the corresponding standard deviation for each entry. We use these statistics to produce the standardized filter-wise saliency profile as follows:
$(x, [s y) 14|
(2)
The resulting tensor $(z, y) is of length equal to the number of convolutional filters in the network, and we henceforth call it the saliency profile for sample x. By standardizing saliency profiles, we create saliency map that activates when the importance of a filter is unusually strong relative to other samples in the dataset. This prevents the saliency map from highlighting filters that are uniformly important for all images, and instead focuses saliency filters that uniquely important and serve
3

---

Under review as a conference paper at ICLR 2022
0.0
Average sal oney profile of correctly classified samples
Sallency
0.22
-0.4
10
20
30
40
50
Average saliency profile of incorrectly classified samples
Saliency
10
20
30
40
50
Layer
Figure 2: Standardized filter-wise saliency profiles, correctly vs incorrectly classified samples. Top: Standardized saliency profiles averaged over correctly classified samples in the ImageNet validation set. Bottom: Standardized saliency profiles averaged over incorrectly classified samples in the ImageNet validation set. On both panels, the filter saliency values in each layer are sorted in descending order, and each layer's 's saliency values are concatenated. The layers are displayed left-to-right from shallow to deep and have equal width on x-axis. Both profiles are generated on ResNet-50.
an image-dependent role. In the rest of the paper, unless explicitly noted otherwise, we use ä(z, v) and refer to it as parameter saliency.
Incorrectly classified samples are more salient. Empirically, we observe the saliency profiles of incorrectly classified samples exhibit, on average, greater values than those of correctly classified if
examples. This bolsters the intuition that salient filters are precisely those malfunctioning the classification is correct, there should be few malfunctioning filters or none at all. Moreover, see deeper parts of the network appear be most salient for the incorrectly classified samples while earlier layers are often the most salient for correctly classified samples. An example of these behaviors for ResNet-50 is shown in Figure 2 which presents standardized filter-wise saliency profiles averaged over the correctly and incorrectly classified examples from the ImageNet validation set. Additionally, we note the improved relative scale of the standardized saliency profile across different layers compared to the absolute gradient magnitudes in Figure | Saliency profiles for other architectures could be found in Appendix [A] Henceforth, we will focus specifically on saliency profiles of misclassified samples in order to explore how neural networks make mistakes.
2.2 INPUT-SPACE SALIENCY FOR VISUALIZING HOW FILTERS MALFUNCTION
The parameter saliency profile allows us to identify filters that are most responsible for mistakes and erroneous network behavior. In this section, we develop an input-space counterpart our parameter saliency method to understand which features of the image affect the saliency of particular filters.
Geiping et al.) (2020) show that the gradient information of a network invertible, providing a
link between input space and parameter saliency space. This work, along with existing input-space saliency map tools (Simonyan et al. 2014, (Springenberg et al., 2015) [Smilkov et al. 2017 Zhou
et al. 2016 Selvaraju al.) 2017) 1)), inspires our method.
Given a parameter saliency profile 8 = 8( y) for an image æ with label y, our goal is to highlight the input features that drive large filter saliency values. That is, we would like to identify image pixels altering which can make filters more salient. To this end, we first select some set F' of the most salient filters that we would like to explore. Then, we create boosted saliency profile s'p by increasing the entries of & corresponding to the chosen filters F (e.g.. multiplying by a large constant). Now, we can find pixels that are important for making the chosen filters F more salient and, equivalently, making the filter saliency profile s(x, y) close to the boosted saliency profile s'p by taking the following gradients:
Mr = V VV
(3)
where : ) is cosine distance.
4

---

Under review as a conference paper at ICLR 2022
Incorrent class nontidenon
17a0 contidencs
Lestt sileet
Perventage of
-30
35-
change
chazge
2,0
Mast milen
-30
Raylen
Randon
Lest cilen
Load rdies:
20
60
100
03
20
60
60
160
Number of prused fittrs
Number of pruned fibters
Number of pruned filters
(b)
(c)
Figure 3: Effect of turning salient filters off. (a) Change in incorrect class confidence score. (b) Change in true class confidence score. (c) Percentage of samples that corrected as the result of pruning filters. These trends are averaged across all images misclassified by ResNet-50 in the ImageNet validation set. The error bars represent 95% bootstrap confidence intervals.
The resulting input saliency map Mr contains input features (pixels) that affect the saliency of the chosen filters / the most.
EXPERIMENTS
In this section, we aim to validate the mcaningfulness of our parameter saliency method. First, we show the dataset level that turning salient parameters off improves predictions the associated samples thus verifying that the salient parameters are indeed responsible for misclassification. We then find that samples which cause similar filters malfunction are semantically similar. We also show on the dataset level that fine-tuning a small number of the most salient parameters on a single sample results eгrоr correction on other samples which were misclassified for similar reasons. We then use our input-space saliency technique in conjunction with its parameter-span counterpart as an explainability tool explore how neural networks make mistakes and how salient filters interact with visual input features.
We evaluate our saliency method the context of image classification on CIFAR-10 (Krizhevsky) 2009) and ImageNet (Deng et al. (009).. Images we use for visualization, unless otherwise specified, are sampled from ImageNet validation set. Throughout the experiments, we use pre-trained ResNet- 18 classifier (He et al. 2016) on CIFAR-10 and pre-trained ResNet-50 on ImageNet. Both models are trained in a standard fashion on the corresponding datase
3.1 PRUNING SALIENT FILTERS
We begin validating the meaningfulness of our parameter-space saliency maps by turning off the malfunctioning filters which cause misclassification of the associated image. We prune away the most salient convolutional filters (i.e., filters identified by our method as malfunctioning). In order to remove the influence of a particular salient filter, we zero out the filter weights and also the biases of the associated batch normalization layers. This procedure guarantees that the corresponding input feature map to the next convolutional layer always zero.
Remarkably, we find that this simple procedure improves the network's 's behavior on the associated samples. In particular, we gradually increase the number of pruned most salient filters and track three metrics: the change in the incorrect class confidence, the change in the true class confidence, and the percentage of the samples that flip their label the correct class. In every case, we compare pruning the most salient filters against pruning the same number of random filters and the least salient filters. These experiments are performed on the dataset level: we average the trends across misclassified images in the ImageNet validation set.
As shown in Figure 3 pruning the most salient filters significantly more effective for decreasing the incorrect class confidence than random least salient filters. Specifically, gradually pruning the
httgs//giittu..ommuuuggai/rooooo hhtps://gittuuh.o//yyorccoo/oiie (under BSD 3-Clause License)
(under MIT license)
5

---

Under review as a conference paper at ICLR 2022
top 100 salient filters achieves up 30% drop in the incorrect class confidence score while pruning random filters yields only about 7% decrease. We also note that pruning the least salient filters does not produce any effect on the incorrect class confidence.
We repeat the same experiment with the true class confidence and observe that the highest true confidence gain occurs when prune around 20 most salient filters. Pruning enough salient filters eventually leads to a gradual decrease in the true class confidence. We note that this behavior is expected since we destroying, not correcting, the inference power of all of the most sensitive filters for an image, some of which may be essential for inference. Finally, pruning random filters provides a much slower increase in the confidence class while the least salient filters again do not produce a significant effect.
In addition, we count the number of images that were corrected as a result of pruning and find that pruning around 30 most salient filters results in the best correct classification rate of 12%. Similar to the true class confidence, the trend decreases beyond this point. Pruning the random filters increases the percentage of corrected samples at much slower rate and does not perform better than the most salient filters when pruning up to 100 filters. Notably, pruning the least salient filters manages to correct a nontrivial number of samples but still much smaller than pruning random filters.
Given the trends in panels (b) and (c) of Figure 3 and given that pruning is coarse tool for fixing misbehavior, we explore the natural idea of correcting the most salient filters instead of removing them altogether our fine-tuning experiments Section 3.3
3.2 NEAREST NEIGHBORS IN PARAMETER SALIENCY SPACE
We validate the semantic meaning of our saliency profiles by clustering images based on the cosine similarity of their profiles. In this section, we present visual depictions of nearest neighbor search among all images the ImageNet validation set. We also conduct this analysis on CIFAR-10 images, and this can be found in Appendix A
(a) Great pyrenees ++ Kuvasz
(b) Basset hound <+ Beagle
Figure 4: Examples of nearest neighbors in parameter saliency space (from ImageNet).
We find that the nearest neighbors of misclassified images in saliency space are mostly other mis- classified images from the same pair of predicted and true classes but possibly in reverse order. For example, in Figure 4 the reference image in (a) is great Pyrenees misclassified as kuvasz, and the 4 images with the most similar profiles exhibit either the same misclassification or the reverse (i.e.. kuvasz misclassified as great Pyrenees). Intuitively, the common salient parameters across these neighbors are those which are important for discriminating between the two classes in question but are not well-tuned for this purpose.
Note that we find the concept of "being similar" in parameter saliency space to different from the one in image space. The nearest neighbors we find are often not similar in pixel-wise sense, but rather they are similar their reason for causing misclassification. For example, images Figure (b) are beagles mistaken by network for basset hounds and vice versa. We find that these pictures are either taken from a high angle or do not include the dog's 's legs, making the leg length, a major distinction between the two breeds, indistinguishable from the picture. We include more example images along with their nearest neighbors in Appendix
In addition, compute nearest neighbors when only considering filters in specific range of layers in order to visualize the types of misbehavior triggered by network components (filters) at various network depths. We search for similar images using parameter saliency in the shallow and deep

---

Under review as a conference paper at ICLR 2022
STRE
Figure 5: Neighbors parameter saliency space found using only early or only deep layers. The reference image is in the first column. Images in the top resemble the reference image in the saliency on early layers of VGG-19, and images in the bottom row are found using deeper layers.
layers of a VGG-19 19 network (Simonyan & Zisserman 2015 which we divide into the shallow and deep parts that respectively occur up to and after layer relu4_1. The top row of figure Figure shows neighbors found using shallow parameters, which share basic image attributes such as color histogram, while images in the bottom row share more abstract similarities.
3.3 CORRECTING MISTAKES BY FINEIGUNING SALIENT FILTERS
To validate whether salient filters are more responsible for the erroneous behavior of neural networks, we show that updating salient filters alone sufficient for correcting the mistakes made by a neural network. In this experiment, for pre-trained image classification network, fine-tune it for one step on a single image for which the network makes the wrong prediction. We restrict the number of tunable filters to be more than 1.0% of the total number of filters in network, and we update the chosen filters by taking one step of gradient descent with a fixed step size. To validate the effectiveness of optimizing salient filters, compare it with other choices of tunable filters: the least salient filters and random filters. For a more general evaluation, we use images from the ImageNet validation set that misclassified by ResNet-50, making up to over 10,000 samples. We evaluate the effect of fine-tuning each of these images independently.
Percentage of corrected samples (%)
True class confidence change
Random
corrected neighboors (%)
filinn
Ranalon
Random Salient
Moet
Lesc Salient
Moet Salient
Leaet Soliont
Solient
Average per pereentage
in neighbors (%)
15
118
number of tunable filters
number of tunable filters
number of tunable filters
(a)
(b)
(c)
Figure Effeet of updating a small number of filters. (a) Percentage of samples that are corrected
after fine-tuning. (b) Average percentage of nearest neighbors that are also corrected after fine-tuning.
(c) Average change in the confidence score of the true class among nearest neighbors. The horizontal line in each plot is the effect of updating the entire network.
In Figure 6] we compare the average performance of our three choices of tunable filters under three evaluation metrics. For given sample image and set of tunable filters, update step is considered to effective the updated network corrects its mistake on the sample image. addition, it is more useful if the updated network also corrects its mistake other images that resemble the sample image but are not seen during fine-tuning.
(a)), First, by inspecting the percentage of samples that are corrected after fine-tuning (Figure (a)),
find that updating 150 salient filters (~-0.6% of total filters) can achieve the same result as updating the entire network. The second and third metrics evaluate the effect on the nearest neighbors of the
7

---

Under review as a conference paper at ICLR 2022
+ Nait salee: dlars
-24-
-22
Vihe
Reference image
(b) Input-space space saliency
(c) Saliency values change of the most erroncous filters across musking experiments
4.07%
white shark killer whale
78.36
(d) Mask seal 83%
(e) Mask bost and seal
(1) Extended mask 73%
great white shark killer whale
great white shark
great white shark
53.47%
24.24%
killer whale
12 88%
killer whale
Figure 7: Interaction between input features and salient filters. (a) Reference image of "great white shark" misclassified by ResNet-50 as "killer whale" with confidence scores, (b) Input-space saliency visualization. Pixels that cause the top 10 salient filters have high saliency, (c) Change in saliency values of the erroneous filters across masking experiments. The vertical bars represent the standard deviation of the change across 10 most salient filters. (d)-(f) Masking experiments.
training sample (Figure 6 (c). We find nearest neighbors for each training sample through the process introduced Section 3.2 where we limit the search scope exclusively to other misclassified images. Note that for given training sample, its nearest neighbors are not involved in one-step single sample fine-tuning process. By tracking model predictions and true class confidence scores among the 10 nearest neighbors of each sample, more effective than other choices. Results in Figure 6 (b) and (c), also imply that the nearest neighbors found using method the images that wrong for similar reasons and that they be corrected altogether by only updating the salient filters on single image. We note that we do not propose new pruning fine-tuning method. Rather, use these experiments verify that the salient filters are indeed responsible for misclassification.
find that fine-tuning salient filters is significantly
3.4 INPUT FEATURES THAT CAUSE FILTERS TO MALFUNCTION: A CASE STUDY
We consider case study of an image misclassified by ResNet-50 as "killer whale" (Figure 7 a)). The correct label of the image is "great white shark". Our goal is study the interaction between the most salient filters and input features. We first identify filters most responsible for misclassification by computing the filter saliency profile and visualize parts of the image that drive the high saliency values for those filters using the input-space space saliency counterpart (Section 2.2)
Panel (b) of Figure presents our spage visualization, which depicts the causes of misbehavior for the ten most salient filters - the pixels that trigger mishehavior in these filters are highlighted. For example, we see that the seal and boat are both triggers. One natural hypothesis that the seal looks like killer whale the network and is the source of the classification error. We test this hypothesis by masking out the seal (Figure (d)) However, although the probability of "killer whale" goes down and the probability of the correct class increases, the network still misclassifies the image as "killer whale".
Now, if mask out exactly the most salient areas of the image according to visualization (see Figure (b)). (c)). the network manages to flip the label the image and classify it correctly. If extend mask to the less pronounced, but still salient, areas of the image in Figure
8

---

Under review as a conference paper at ICLR 2022
(a)
dalmation
(b)
pizza
loopard
(d)
junco finch
(e)
passenger car locomotive
soccer ball
bell pepper
jaguar
Figure 8: Different types of network mistakes. All of the presented images are misclassified by ResNet-50. The correct class label is specified in the top row and the incorrect class label in the bottom row of the subcaption on each panel. (a)-(b) The target object is confused with another object in the image. (c) A regular mistake. The salient pixels confuse the network. (d) Background features confuse the network. (e) An example of a noisy label where the network is "more correct" than the target label. These are examples where masking top 5% of the salient pixels corrects the misclassification.
focused the target object features which
(f), observe that the correct class confidence increases more while the probability of the incorrect "killer whale" label further decreases. Additionally, we find that masking out the nan-salient parts of the image results in even worse misclassification confidence than that of the original image (see Appendix[A] In order to further investigate the effect of the salient region, we pasted it from this image onto other great white shark images (see Appendix (A) and observed that this drives the probability of "killer whale" up for 39 out of 40 examples of great white sharks from the ImageNet validation set with an average increase of 3.75%.
Our experiments suggest that secondary objects in the image are associated with the misclassification. However, we see that the erroneous behavior of the model does not just stem from classifying non-target object in the image. It is possible that the model correlates the combination of sea creatures (e.g. seal) and man-made structures (e.g. a boat) with the "killer whale" label. We note that images of killer whales in ImageNet often have man-made structures which look similar to the boat (see Appendix A for examples of other "killer whale" images).
Finally, at each step of our masking experiments, we recompute the saliency values of the originally chosen 10 filters (i.e. the filters that caused erroneous behavior on the reference image). From Figure (c), observe that as mask out the input features according to our input-saliency. the saliency values those filters decrease gradually and reach an 80% drop, confirming that highlighted regions indeed drive the high saliency of the chosen filters.
More visualizations of input space saliency showcasing different illustrative examples of neural network mistakes can be found in Figure For thorough discussion of mistake categories identify using our saliency method, we refer to Appendix
DISCUSSION
Numerous applications demand that practitioners be able to understand the decisions their models make, especially when those decisions are incorrect. Existing methods for explainability focus on locating the input regions to which the network's 's output is sensitive or on associating network components with specific roles. In contrast, develop framework for finding the exact filters which are responsible for faulty predictions and studying the interactions between these filters and images. This direction yields both interpretable and intuitive understanding of model behaviors.

---

Under review as a conference paper at ICLR 2022
5 ETHICS STATEMENT
Although our formulation of parameter saliency is not restricted to image datasets and CNNs, we only conduct experiments in these settings. In contrast, real-world data and models come in many forms. Explainability methods which shed light in some settings may fail to do so in others, Moreover, we emphasize that some erroneous model behaviors are simply difficult to understand through existing methods, and the capabilitiiss of parameter saliency are limited. In many applications, it is imperative that practitioners understand why their models behave as they do and that they are able to diagnose problems when they arise. We hope that our work helps to enable solutions to real-world problems. However, we caution against a false sense of security. Our visual interpretations of model behavior should be viewed as approximations since neural networks are incredibly complex.
6 REPRODUCIBILITY STATEMENT
We include our implementation of our parameter saliency method as well as the input-space saliency counterpart as a supplementary material. All the datasets used in the experiments are publicly available. The implementation details available in Appendix B
REFERENCES
Reza Abbasi-Asl and Bin Yu. Interpreting convolutional neural networks through compression. arXiv preprint arXiv: 1..0229, 2017.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. arXiv preprint arXiv:/810.03292, 2018.
Ahmed Alqaraawi, Martin Schuessler, Philipp WeiB, Enrico Costanza, and Nadia Berthouze. Evaluat- ing saliency map explanations for convolutional neural networks: user study, In Proceedings of the 25th International Conference on Intelligent User Interfaces, pp. 275-285, 2020.
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. arXiv preprint 1r1iv:1811.01111 2018.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541 6551-6549, 2017.
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4690-4699, 2019.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18X0v:1810.444 040005 2018.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangl Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural networks. arXiv preprint 10.00421 2021.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
European Commission, Recital 71 EU General Data Protection Regulation. 2018. URL https www.privacy 1at at i oon eu/ wu/en//eeeeeeeeeeeeeeeeeeee 1ltmmmmmmmmmmmm
A. H. Miller P. Lewis A. Bakhtin Y. Wu F. Petroni, T. Rocktäschel and S. Riedel. Language models as knowledge bases? In In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019, 2019.
10

---

Under review as a conference paper at ICLR 2022
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In ICCV, 2017.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-how easy is it to break privacy in federated learning? arXiv preprint arXiv:2003.14053, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Gao Huang, Zhuang Liu,, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Eunhee Kang, Junhong Min, and Jong Chul Ye. A deep convolutional neural network using directional wavelets for low-dose x-ray ct reconstruction. Medical physics, 44(10)::300-<375, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. arXiv preprint arXiv:2102.11600, 2021.
Congcong Liu and Huaming Wu. Channel pruning based on mean gradient for accelerating convolu- tional neural networks. Signal Processing, 156:84-91, 2019.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015.
Ari S Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. arXiv preprint arXiv: /8Xx:://03306959, 2018.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.2391 109911/diitill.o.. 00000.. http:://diisiil.pp///0111 //taa//////////eeaaeeeraaaaaaiia
Fabio Petroni, Patrick Lewis. Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models' factual predictions. In Automated Knowledge Buse Construction, 2020. URL,
wwt.eeeww.eeee..eeeeeeeeeeeee.ee ner/foru??
id=025x0zPff.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In ICCV, 2017.
A Shrikumar, P Greenside, A Shcherbina, and A Kundaje. Not just a black box: Learning important features through propagating activation differences. 2016. arXiv preprint 1605.011
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In ICML, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg, Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: arXiv: /7061038260.5... 2017.

---

Under review as a conference paper at ICLR 2022
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. In ICLR, 2015.
Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. 2019.
arXiv preprini
Xu Sun, Zhiyuan Zhang, Xuancheng Ren, Ruixuan Luo, and Liangyou Li. Exploring the vulnerability of deep neural networks: A study of parameter corruption. arXiv preprint (rxiv:2000.056200 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826, 2016.
United States Congress Senate Committee on Banking and Housing and Urban Affairs. Equal Credit Opportunity Act. [electronic resource). S. Rpt. 94-685. Washington, 1976.
Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu. Score-cam: Score-wer Score-weighted
weighted visual explanations for convolutional neural networks. In
CVPR, 2020.
David West. Neural network credit scoring models. Computers & Operations Research, 27(11-12): 1131-1152, 2000.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv: 10X65:1506.06559, 2015.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 118-833. Springer, 2014.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. IEEE transactions on pattern analysis and machine intelligence, 41(9): 2131-2145, 2018.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Comparing the interpretability of deep networks via network dissection. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 243-254 Springer, 2019.
12

---

Under review as a conference paper at ICLR 2022
025
gradient nageRatte
0.0
(a) VGG-19
Ansruge ynalaai augastuios
Salancy
0.4
0.0
26
80
Laper ID
(h) Inception v3
0.3
gradient magributes
(c) DenseNet
Figure 9: Filter-wise saliency profiles for other architectures. (a) VGG-19 saliency profile (with- standardization). (b) Inception v3 saliency profile (without standardization). (c) DenseNet saliency profiles (without standardization). In each panel the filter-wise saliency profile is averaged over the ImageNet validation set. every panel, the filter saliency values in each layer are sorted in descending order, and each layer's 's saliency values are concatenated. The layers are displayed left-to-right from shallow deep and have equal width on x-axis.
A ADDITIONAL EXPERIMENTS
A.I PARAMETER SALIENCY PROFILES FOR OTHER NETWORK ARCHITECTURES
In this section, we present average saliency profiles for several popular network architectures other than ResNet-50 (He al. 2016). Analogously Figure Figure presents average gradient magnitudes for VGG-19 (Simonyan & Zisserman, 2015). Inception v3 (Szegeyy et al. 2016). and DenseNet (Huang et al. 2017). Similarly to Figure we also present Figure 100 standardized filter-wise saliency profiles for those architectures averaged across correctly and incorrectly classified
ImageNet (Deng et al. 2009) samples.
A.2 MORE EXAMPLES OF NEAREST NEIGHBORS
We present more examples of nearest neighbors in our parameter saliency space. Figure 12 are nearest neighbors in CIFAR- 10 (Krizhevsky) 2009) dataset, where reference images are chosen from samples misclassified by our classifier. Figure 13 are examples from ImageNet, where images are captioned with the true label of the reference images.
In addition, in comparison with the nearest neighbor in our parameter saliency space, we also conduct the nearest neighbor search in the feature representation space. We take the feature representation from the conv5 3 layer of a ResNet-50, and run the nearest neighbor search using the same reference images and candidate pool as in Figure Results are shown in Figure II We find that nearest neighbors in the feature space bear more resemblance in image structures, but it fails to identify samples that share the same mistakes. fact, most of the nearest neighbors in Figure are correctly classified samples.
13

---

Under review as a conference paper at ICLR 2022
Avedage profiee of correctly classified samples
Saleenyy
-0.4
8
10
12
14
16
Average protil of incorrectly clasatted samples
2
8
300
12
141
16
Layer ID
(a) VGG-19
0.0
Average salency profile of correctly cinssified samples.
Salwnnoy
-0.4-
Average saliency proflle of incorrectly classified samples
Lager ID
(b) Inception v3
0.0
Average sabency protile of correctly classthedd samples
Seliorry
-0.4-
120
1.5
Aveer agg sa Day profile of Incorrettly classiffed samples
Salienty
2.0-
0.5
0.0-
160
Lager
(c) DenseNet
Figure 10: Standardized saliency profiles averaged over correctly vs incorrectly classified sam- ples. (a) VGG-19 saliency profiles. (b) Inception v3 saliency profiles. (c) DenseNet saliency profiles. In each panel, the top row presents the standardized saliency profiles averaged over correctly classified samples and the bottom row shows standardized saliency profiles averaged over incorrectly classified samples. On every panel, the filter saliency values in each layer are sorted in descending order, and each layer's saliency values are concatenated. The layers are displayed left-to-right from shallow to deep and have equal width on x-axis.
(a)
(b)
Figure 11: Examples of nearest neighbors in the feature representation space (from ImageNet).
14

---

Under review as a conference paper at ICLR 2022
Furthermore, present preliminary results applying method to language models. We a BERT (Dellin et al. 2018) model, pre-trained on the task of predicting the word at masked position. We consider independent dataset for evaluation in experiments. which consists of short sentences with masked words, provided by LAMA (F. Petroni & Riedel 2019 Petroni et al. 2020), open-source language model analysis framework. Similar to the filter-wise aggregation in convolutional networks, we adopt column-wise aggregation for obtaining our saliency profiles in the transformer-basee architecture. We conduct the nearest neighbor search by comparing sentences from the dataset in saliency space and analyzing the top-5 nearest neighbors.
We present four examples below, where the numbering indicates n-th nearest neighbor sentence, and the italic word is the ground truth. Each example is accompanied with description of similarities between the neighbors:
Reference: "Cany Ash and Robert Sakula are both Architects." incorrectly predicted as: actors
"David Castlles-Quintane and Vicente Royuela are economists." incorrectly predicted as:
actors
2. "Raghuram Rajan is an economist." incorrectly predicted as: actor
3. "Richard G. Wilkinson and Kate Pickett are British." incorrectly predicted as: actors
4. "Nathan Alterman was a poer." correctly predicted: poet
5. "Zbigniew Badowski is an architect." incorrectly predicted as: author
Note that all 5 neighboring sentences here share the common structure of being declarations of profession for specifically named people. Interestingly, the first three closest sentences to the reference incorrectly predict the profession be an actor as well. Moreover, the ground truth of the reference and its closest neighbor is economist/s.
Reference: "D)' Olier Street is in Dublin." incorrectly predicted as: Paris
1. "A group who call themselves Huguenots lives in Australia." incorrectly predicted as: France
2. 'Huguenots and Walloons settled in Canterbury." incorrectly predicted as: France
3. "In the Treaty of Lisbon 2007 Ireland refused to consent changes." incorrectly predicted as: it
4. "Samuel Marsden Collegiate School is located in Wellington." incorrectly predicted as: Melbourne
5. "Konstantin Mereschkowski has Russian nationality." correctly predicted: Russian
Again, we see all five nearest neighbor sentences are semantically similar to the reference, this time relating to national affiliations and geography. In the first two closest sentences, the model incorrectly fills a location with France, which similar to reference which incorrectly predicts Paris.
Reference: "The Super Bowl sponsor was the Gap clothing company." incorrecly predicted as Nike
"During Super Bowl 50 the Nintendo gaming company debuted their ad for the first time." incorrectly predicted as: video
2. "Experimental measurements on a model steam engine was made by Wat.." incorrectly predicted as: Siemens
3. "ABC's programming strategy was criticized in May 1961 by Life magazine." incorrectly predicted as: Time
4. "In 2009, Doctor Who started be shown Canadian cable station Space." incorrectly predicted as: CBC
5. "To emphasize the 50th anniversary the Super Bowl the gold color was used." incorrectly predicted as: blue
All but one the nearest neighbors in this example relate to some kind of TV programming, and two also mention the Super Bowl. Much like the reference sentence, which incorrectly predicts the name corporation, the first four neighbors have a ground truth or incorrect prediction that also corporation.
15

---

Under review as a conference paper at ICLR 2022
Reference sample: "Tetzel's collections of money free souls from purgatory objected by Luther." incorrectly predicted as: some
1. "Newcastle was granted new charter in 1589 by Elizabeth." incorrectly predicted as: Parliament
2. "The suggestion that imperialism was the "highest" form of capitalism is from Lenin." incorrectly predicted as: Aristotle
3. "Fritschel said the man's sleep was disturbed by dreams." incorrectly predicted as: lightning
4. "The concept that falling objects fell the speed regardless of weight introduced by Galileo." incorrectly predicted as: NASA
5. "One of the earliest examples Civil Disohedience was brought forward by the Egyptians." incorrectly predicted government
This is an example where there is weak relation between the incorrect classifications (three of five involve some kind of government or government organization) the nearest neighbors, but the sentences still highly semantically similar. All of the neighbors except the third are declarations of historical actions performed by specific person or group of people.
(a) bird cat
(b) frog cat
(c) airplane <-> ship
(d) horse <> deer
Figure 12: CIFAR-10 examples of nearest neighbors in parameter saliency space. On CIFAR-10 images that cause similar filters to malfunction often misclassified similar way.
(a) coyote
(b) goldfish
(c) terrapin
(d) great white shark
Figure 13: ImageNet examples of nearest neighbors in parameter saliency space. In every panel, the reference image is in the left column and its nearest neighbors are in the right column. Panels are captioned by the true label of their reference image.
A.3 CORRECTING MISTAKES ON OTHER DATASETS
We have shown in Section 3.3 that updating a few salient parameters of an ImageNet pre-trained network is enough for effectively correcting mistakes the ImageNet validation set. Moreover, in this section, we show that this effect can be extended to other independent datasets. We use the ImageNet-v2 v2 test set, consisting of 10,000 images collected by Recht et al. (2019). independent from the original ImageNet data.
observe the similar pattern in Figure With test samples independent of the training data, salient parameters still demonstrate the most strength in correcting mistakes and their nearest neighbors. The advantage is more prominent when inspecting model's predictions of unseen nearest neighbors.
In Figure |4]
A.4 FINE-TUNING SALIENT FILTERS OF A VGG-19
In this section, we conduct the fine-tume tuning experiment introduced in Section3.33 on a VGG-19 network trained ImageNet, which has total of 5504 filters. The learning rate for training our VGG network
16

---

Under review as a conference paper at ICLR 2022
Paoentgge of onreded snntes (1g)
Lead Sidea
Aveig parcerng ooreere bnihher
number tunable filters
number of tunable filters
number tunable filters
(b)
(0)
Figure 14: Effect of updating a small number of filters on the ImageNet-v2 test data. (a) Percentage of samples that corrected after fine-tuning. that are also corrected after fine-tuning, (c) Average change in the confidence score of the true class among nearest neighbors. The horizontal line in each plot is the effect of updating the entire network.
(b) Average percentage of nearest neighbors
1/10 of that for the ResNet, so we decrease the fine-tuming step size by 10 in this experiment. Figure 15 shows the effect of updating salient filters of VGG-19. Note that the same range for the number tunable filters in this experiment; 300 filters correspond to 5.5% of total filters in VGG-19, while it is 1.2% for ResNet-50.
Poccingg of cxrecte smpiss (1)
socoree parcrttagc cunooaed ssigboons
Martille
Teme dhe confilnna wempg-innngghlae nnggllu
Fubic
Luckdiar
1s6
300
130
120
308
numher of tunabll filters
number of tumable filters
number of tumahle filters
(n)
(e)
Figure 15: Effect of updating a small number of filters on VGG-19, (a) Percentage of samples that are corrected after fine-tuning. (b) Average percentage of nearest neighbors that are also corrected after fine-tuning, (c) Average change in the confidence The horizontal line in each plot the effect of updating the entire network.
of the true class among nearest neighbors.
A.5 RANDOM PERTURBATION OF SALIENT FILTERS
As an alternative the pruning approach described in Section 3.J random perturbations could be used to show that the most salient filters are indeed responsible for misclassification. We perturbed the filters using small Gaussian noise (O, 0.001). Figure presents the effect of randomly perturbing salient filters, we observe similar trends to our pruning experiments in Section 3.1
A.6 CONNECTION TO ADVERSARIAL ATTACKS IN PARAMETER SPACE
Adversarial attacks in parameter space have been used for optimizers which find loss minima (Foret et al. 2020 Kwon et al., 2021 Du et al. (222] and for improving model robustness through paraamter--coruption-cct resssssant training (Sun et al. 2020).
One could instead apply adversarial attacks to construct parameter saliency profiles choose constraint space and perturb parameters in order to minimize loss, subject to the constraint, using the perturbation parameters several advantages and disadvantages of this alternative. On the one hand, the "adversarial" approach requires choice of constraint space and may require compute (our method results in the exact
saliency profile (perhaps standardizing afterwards). We notice
17

---

Under review as a conference paper at ICLR 2022
Most exilene
0.35
Bandars
A.5
-0.03
0.20
Losst sakent
2.0
0.25
-0.10-
0:15-
0:30
18.20
sxlezt
0:65
0.0
Namber perturttd
Namber preturted fiters
Number perturbed fiterss
(b)
(c)
Figure 16: Effect of randomly perturbing filters. (a) Change in incorrect class confidence score. (b) Change true class confidence score. (c) Percentage of samples that were corrected the result of pruning filters. These trends are averaged across all images misclassified by ResNet-50 in the ImageNet validation set. The error bars represent 95% bootstrap confidence intervals.
same saliency profile as a single-step adversary). On the other hand, the choice of constraint space and optimizer the adversarial approach yields more flexibility.
In this section, we compare our method with the Snivessriall-attck---e for given sample (x., y), we perturb parameters either to maximize or minimize the loss subject to an L2-norm constraint the parameter change:
method. More specifically,
min max Lo v)
(4)
s.t. 110 0o || <E
Then, given the adversarially perturbed parameters 0*, define the aiversaril-atttck-baee param- eter saliency profile of the sample (z, y) as the absolute difference from the initial parameters 06: s(x,))
10* Oul.
We compare the resulting adversarial saliency profiles with our original method on a random sample of 100 images from the ImageNet validation set. We observe that for reasonably small constraint (e= 10 10-4), the resulting adversarial saliency profiles are similar to our original parameter saliency profiles (as would expect for smooth loss) with the average cosine similarity between the saliency profiles generated by each method for the same images reaching 0.99 (the average is taken over the random sample of 100 images). We also see that both methods agree on the top-k (we tried k= 100) most salient filters: on average, 95% of filters identified by our original method as top-k salient filters were also identified top-k salient filters by the adversarial parameter saliency. The differences were gradually more distinct with larger constraints, however, we note that the smaller epsilons are of greater interest since they reflect the intuition of perturbing only the most important parameters.
We also tried I. l.Irregularized adversarial attacks. We can similarly enforce sparsity in our original method by simply adding the regularizer to our loss before computing the gradient:
s(z, v); (1 V (Co(z,)) + +al0 foll))
(5)
This modification can be seen as one step of the Li-regularized adversarial attack, and we experimen- tally checked that it produces very similar results with the cosine similarity between the resulting saliency profiles of 0.97 on average (for sufficiently large a = 0.99).
A.7 ADDITIONAL CASE STUDY FIGURES
Masking non-salient parts of the image. As noted in section 3.4 and presented in Figure 17 masking the non-salient parts of the image results in even worse misclassification confidence with the incorrect class confidence increasing compared to the reference image.
Pasting the salient region from the reference image onto other "great white shark" images. As mentioned in section 3.4 in order to further investigate the effect of the salient region, we pasted it (i.e., the seal and the boat) from the original image onto other images with "great white shark" ground truth label that were correctly classified by ResNet-50 (see Figure [8 examples). We observed that this increased the probability of "killer whale" for 39 out of 40 examples of great white sharks from the ImageNet validation set with average increase of 3. 3.75%.
18

---

Under review as a conference paper at ICLR 2022
(a) Reference image
(b) Input-space saliency for 10 salient filters (pixels most responsi- ble for misclasstication)
4.07%
great white shark
78.36
killer whale
(c) Masked buman
(d) Masked non-salient water
3.84%
great
great
86.04%
killer whale
79.38
killer whole
Figure 17: Masking non-salient parts of the image. Reference image of "great white shark" misclassified by the model as "killer whale" and the corresponding confidence scores. (b) Pixels that cause the top 10 most salient filters have high saliency. (c) Masked (non-salient) human. (d) Masked non-salient water region.
Examples of images with "killer whale" label. As we discussed section 3.4 the model might have learned to correlate combination of sea creatures (e.g. seal) and man-made structures (e.g. a boat) with the "killer whale" label. Images of killer whales ImageNet often have man-made structures which look similar to the boat, provide examples of that in Figure T9
by Maa
Figure 18: Sample "great white shark" images with boat and seal. The salient region from the case study image pasted onto other "great white shark" images.
Figure 19: ImageNet examples of "killer whale".
19

---

Figure 20: Pixels responsible for mistakes focused the target object. (a)-(b) Masking the salient pixels corrects the misclassification where masking confusing features (e.g. dog ears or spot patterns) helps distinguish animals. (c)-(d) Masking the salient pixels results in correct class confidence decrease, when the salient pixels are densely focused on the target object. The correct Under review as a conference paper at ICLR 2022
(a)
malamute
stingray
appenzeller senmenhund
recbone
husky
electric
beagle
class label is specified in the top row and the predicted incorrect class label in the bottom row of the subcaption on each panel.
A.8 EXPLORING NEURAL NETWORK MISTAKES
In Section 3.4, we apply our parameter-space saliency method as an explainability tool using it alongside our imput-space technique to study how image features cause specific filters to malfunction. In our case study, the salient pixels that confuse the network are focused less on the target object than on other image features, and masking the salient regions which are on the target object improves the network behavior. Such examples expose the network's 's reliance spurious correlations and constitute an interesting type of model mistake.
The masking approach can be adopted to explore network mistakes and find other interesting cases (e.g., cases where the salient pixels are not concentrated on the target object). Investigating examples where masking the most salient pixels improves performance may provide insights into the model's misbehavior as well as expose dataset noise and biases. We select misclassified samples where masking the top 5% of salient pixels leads to increase of least 25% in confidence corresponding to the correct class. We showcase representative examples of different types of mistakes that we observe in those samples in Figure 8 Many of the neural network misclassifications stem from classifying non-target object in images with multiple objects (see Figure S (a). (b)). However, other mistakes are triggered by background features (Figure & (d)), dataset biases (as our case study experiments in Figure 7 suggest), and label noise (Figure 8 (e)).
Interestingly, in some of the selected cases the salient regions still focus on the target object features (see Figure 8 (c)), and masking them improves the model's behavior. Masking salient target object features that confuse the network seems be particularly beneficial for images of animals; for example, masking dog ears helps the network identify the correct breed (see Figure 20(a)) or masking
spot patterns helps distinguish different types of rays (see Figure 20 b)).
While masking the top 5 % of salient pixels results in correct classification for all samples in Figure and in Figure a)) (b), this is, course, not always the case. Sometimes, pixels which cause large filter saliency values are focused densely on the target object, and masking them results in decreased confidence corresponding to the correct class. Selecting such samples can be used to find situations where the network is genuinely confused by the target object rather than background features (Figure (c)) or samples with multiple objects present and with salient pixels more focused on the target object (Figure (d))
20

---

Under review as a conference paper at ICLR 2022
A.9 COMPARISON TO GRADCAM
Existing imput saliency maps used with the predicted label can highlight features which are related to misclassification. However, they are not specifically geared towards that goal. Our input saliency technique highlights image features that cause specific filters malfunction and those features, while they might in some cases coincide with the features that explain high class confidence score corresponding to the predicted label, may not be the same.
In this section, we will compare our input-space saliency technique which highlights pixels that drive high parameter saliency values of specific filters (i.e., pixels that confuse the network) to the visual explanations produced by GradCAM with the predicted label (Selvaraju et al. 2017 one of the most popular and high quality input-saliency methods.
Figures 21 and 22 present panels of images comparing method to GradCAM explanations computed with the predicted label. From the perspective of highlighting pixels responsible for neural network mistakes and for driving high filter saliency values, we note the following:
GradCAM highlights the object that corresponds to the incorrect label, and the entire target object is highlighted in the images where only the target object is present (see Figure 21 (c)-(e), Figure 22 (a)-(c)). However, when our method focuses on the target object, it highlights specific features of that object. Those are the features that confuse the network, and masking them can correct the misclassification.
In cases where the network classifies the non-target object in the image (see Figure 21 (a)-(b), 2igure(22(c)) (d)), both methods highlight the non-target objects. However, GradCAM is more localized to the non-target object. This is expected since GradCAM produces visual explanations for the predicted label (and has been shown to produce highly localized saliency maps (Selvaraju al., 2017)) while our method highlights all pixels that drive the filter saliency, and these pixels may be located on the target object well.
In cases where the misclassification does not stem from confusion by the target object or classifying the non-target object (see Figure 21 (d)-(e), Figure (2 our method highlights background features and/or combination of non con-target object features, while GradCAM still highlights the target object. For example, in Figure (2 (e), our method highlights the boat and the sky much more than GradCAM, and our case study masking experiments in section 3.4 show that those regions indeed confuse the network.
In addition, we emphasize that our input saliency technique specific to the chosen filter set F' and is introduced to study the interaction between the image features and the malfunction- ing filters. contrast, GradCAM is not able to relate image-space mistakes an arbitrary set of model parameters or filters chosen by the user.
To summarize, GradCAM (as well as many other input-space saliency methods) was designed to be highly localized to the object correponding to the label of interest, while method highlights sparse fine-grained features of images which we believe is a desirable property for our specific application. Therefore, Gradient (Sioonyan et al. 2014) method. However, instead of class confidence scores, we use a different loss cosine distance the boosted parameter-saliency profile (as described in Section 2.2 which allows us to explore how image features cause specific filters to malfunction.
opt for using grpudigra information similar to the original Vanilla
A. INPUT-SALIENCY SANITY CHECK
To assure that our input-space saliency technique is model dependent, we performed the model randomization test from (Adebayo et al. 2018). We can see that the input saliency map is model dependent. We note that the data randomization test is not applicable in our case because our input- space saliency map is based on the parameter-saliene saliency profile and parameter-saliency is designed to investigate pretrained model with particular weights.
Implemmntation from https///gttuuh.co//kannnn/ saraadcca--ytoorcchh under MIT license
21

---

Under review as a conference paper at ICLR 2022
dalmation soccer ball
(b)
pizza
(c)
Ieopard
(d)
junco
passenger car locomotive
bell pepper
jaguar
house finch
Comparison to GradCAM. Top row: original image. Middle GradCAM input- space saliency map for the predicted label. Bottom row: our input-space saliency technique which highlights pixels that drive high parameter saliency values of specific filters (i.e., pixels that confuse the network). The correct class label is specified in the top row and the predicted incorrect class label the bottom row of the subcaption on each panel.
Figure 21:
B IMPLEMENTATION DETAILS
B. HYPER-PAR AMETER SETTING FOR FINE-TUNING SALIENT FILTERS
When tuning a small number of random or least salient filters, we re-normalize the gradient magnitude of these parameters to be the same as the salient filters for fair comparison: otherwise the gradients for these parameters are always smaller than the salient ones by the definition of our saliency profile, and updating them would make less change to model than updating the salient ones. In addition to re-normalizing the gradients, we also multiply them with a step size, similar to the concept of learning rate in stochastic gradient descent. For ResNet-50, equals to the learning rate of the last epoch when training the ResNet-50 on ImageNet from scratch. For 19, also set the fine-tuning step size be the learning rate from the last training epoch, which is 0.0001. We also note that the batch normalization layers were set to the test mode for our fine-tuning experiments
set the step size to be 0.001, which
B.2 INPUT SALIENCY VISUALIZATION
The number of top salient filters to boost was chosen to be 10 in all input-space saliency experiments. The filters were boosted by multiplying by 100. For visualization, absolute input gradients were thresholded at 90-th percentile and Gaussian Blur with (3, 3) kernel was applied.
B.3 HARDWARE
The experiments were run on Nvidia GeForce RTX 2080Ti GPUs with 1 G1 GPU memory on machine with 4 cpu cores and 64Gb RAM. The input-space and parameter-saliency profiles take seconds to compute for a single sample.
22

---

Under review as a conference paper at ICLR 2022
(a)
malamute husky
(h)
stingray electric ray
appenzeller
(d)
redhone
(e)
great white shark killer whale
sennenhund
beagle
Figure 22: Comparison to GradCAM. Top row: original image. Middle row: GradCAM input- space saliency map for the predicted label. Bottom row: our input-space saliency technique which highlights pixels that drive high parameter saliency values of specific filters (i.e., pixels that confuse the network). The correct class label specified the top row and the predicted incorrect class label in the bottom row of the subcaption on each panel.
(a) Original
(b) Stage 4
(c) Stages 3-4
(d) Stages 2-4
(c) Stages 1-4
Figure 23: Sanity checks. (a) No randomization ResNet-50. (b) Only stage of ResNet-50
randomized. (c) Stages 3-4 of 55 are randomized. (d) Stages 2-4 of ResNet-50 are randomized, (e) The entire ResNet-50 is randomized.
23


Reviews:
[{"review_id": "v-Hofo75rUe", "reviewer": "Reviewer_tkQp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper devises an approach so look at saliency of filters by aggregating the gradients w.r.t. the parameters belonging to that filter. In that sense it is an extension of sensitivity analysis. The paper then shows that only a limited number of filters need updating to modify the prediction. Given that some of these filters might have very many parameters the resulting change can be quite big. \n\nI believe there are 2 clear limitations in the paper.\n- it is not clear how this differs from an adversarial attack in parameter space/simply exploiting the sensitivity to certain parameters. I.e. changing parts with large gradients will change the behavior. The fact that this is done for a limited number of parameters is similar to using L1 normed adversarial attacks. \n- The evaluation in section 3.4 is quite limited in scope and looks to be more anecdotal. Extending this to an in depth evaluation would improve the paper.\n\n\nDetailed comments below\n\n\n# Intro/Methods\n- Figure 1/2: . I believe this figure can be improved by assigning each layer space corresponding of \"unit 1\" on the x-axis. Right now, the decay for the earlier layers (with fewer filters) is hard to see. By normalizing the space used by a single layer this might become easier to read.\n\n- Figure 1 and 2: I am very surprised that in Figure 1 and Figure 2-bottom the saliency averaged over filters is always positive. In contrast it is always negative in Figure 2 top.  Fig 1: states that it is not standardized. Fig 2. does not specify this but it seems to me that there was an inconsistency. \n\n- Section 2.2/ Eq 3  I did not find later on in the manuscript how the magnitude of the selected filters was increased. Instead of using the cosine distance, one could also try to only pass the gradients through the filters that were deemed most salient. But this might result in a very sparse heatmap.\n\n# Experiments\n- Section 3.1/Figure 3:  If I understand it correctly these plots only contain data from initially misclassified samples. While the evaluation is done over all misclassifications, it is the sum of experiments on single images. I.e. when a filter is removed this is done only for a single example and evaluated on this example. In that sense it is similar to an adversarial perturbation in the middle of the network. The fact that this changes the confidence should not be surprising per se.\n\n- Section 3.2/Figure 4/5: The nearest neighbor approach is done by computing the cosine similarity of the saliency maps. Since this is dominated by the higher layers it should not be surprising that this results in conceptual similarity vs input similarity.  This can also explain why the same classes are misclassified since higher level concepts are also higher in the layers. In Fig. 5 similarity of low vs higher level layers is shown. The texts states that this is done to show misbehavior, but it is not clear what is misbehaving in this plot. \n\n\n- Section 3.3 Correcting mistakes by updating salient filters is evaluated on all images independently. How was batch norm used in this case for the update? Setting it to training mode could have weird effects.  I am not convinced that it should be surprising that updating the ones with large gradients results in changes at the output. \n\n- Section 3.4\nThis section is the most problematic to me since it lacks large scale evaluation and is mostly anecdotal. \n\nFigure 8 is a set of \n# General\nMake the labels of the axes in plots larger. They are hard to read.", "summary_of_the_paper": "The paper tries to identify the most salient parameters in a neural network and shows that when these parameters are modified the predictions change. They also show that there is a relation between these parameters and regions in the input that impact the classification result.", "summary_of_the_review": "I tend to vote for rejection of this paper. The effects described are similar to adversarial attacks and in most but not all cases the influence of a parameter is only investigated on the image where the saliency was computed. \nThe evaluation of the proposed approach in terms of providing pixel wise interpretability section 3.4 is limited.\n\nThis review was done after the rebuttal period as an emergency review. I had no access to the discussion with the authors which might impact my understanding.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "k5VSqenWf3-", "reviewer": "Reviewer_3o7Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ Though the idea of parameter-wise visualization is not new, and widely studied in other works, this paper provides an interesting perspective on the parameters responsible for misclassification. \n+ Meanwhile, the authors provides the thorough analysis and potential for applying the salient parameters on improving classification accuracy. This makes the paper with pratical use.\n\n- The proposed idea is somehow similar to the parameter-space adversarial attacks. What if use attack methods to filter out the salient parameters that mislead the final classfications? This seems more intuitive.\n- For the experiments of parameters pruning, it may be not reasonable for directly turn off the parameters, what if give them a perturbation instead? It can still reveal the sensitivity of the parameters.", "summary_of_the_paper": "This paper introduces a parameter-space saliency map to explore the salient parameters that are responsible for miscalssification. A set of experiments and visulizations are conducted on the salient parameters, leading to several interesting findings, such as, the nearest parameter neighbors share similar semantic information. Besides, the authors are also trying to improve the prediction accuarcy by turning off or fine-tuning the salient parameters. ", "summary_of_the_review": "The proposed method is new as a parameter-wise visualization, but it becomes somehow similar to the  parameter-space adversarial attacks. The authors would better claim the similarity and difference of these 2 topics. Besides, adversarial attack needs to be reviewed as the related work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VA6NcLqM32r", "reviewer": "Reviewer_sCeW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[1] Eq. 1 is not clear enough, what's the meanings of K. \n\n[2] If the proposed analytic method only suits for the image classification task or the test models, the contribution would be limited. \n\n[3] As stated in Section 3.3, we can correct mistakes by fine-tuning salient filters. Does it mean a model of general performance could become a top-performing model benefiting from the proposed fine-tuning method?\n\n[4] Whether the performance-gain obtained by fine-tuning salient filters can transfer to the independent database, rather than the closed test set. ", "summary_of_the_paper": "This paper devises an analytic method for explainability based on the observation of filter-wise parameter saliency distribution, and tests on several models. And several experiments are conducted to deminstrate the conjecture. The motivation is straightforward and easy to understand. ", "summary_of_the_review": "The contribution might be not enough for iclr.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RChSRKbbSwY", "reviewer": "Reviewer_qR4u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. Different from the previous papers focusing on the saliency maps, this paper proposes to analyze the network parameters, which is novel and interesting.\n2. Quantitative and qualitative experiments are conducted to verify the proposed standpoints.\n3. The discoveries in the paper is interesting and inspired.\n", "summary_of_the_paper": "This paper conducts some basic experiments based on the network parameters which are responsible for erroneous decisions. To support the proposed standpoints, the authors conduct a lot of quantitative and qualitative experiments.", "summary_of_the_review": "\nI am not an expert in this field and I did read carefully about this paper. I could not find any weakness in this paper. The contributions proposed in this paper are really interesting and are verified by the proposed experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: For this paper initially the reviews were 6,8,5,5. All the reviewers have provided constructive and substantial feedback. The authors have incorporated changes to address some of these comments and some of the comments could not be addressed. The main criticism of the reviewers have been that the Reviewer tkQp finds two clear limitations in the paper, Reviewer 3o7Z finds that the proposed idea is similar to the parameter-space adversarial attacks and Reviewer sCeW questions the generalisability of the method to other tasks. After the rebuttal the reviewers have reached the consensus that the paper may not be above the acceptance threshold (final scores: 6,6,5,5). Following the reviewers' recommendations, the meta reviewer recommends rejection.
Decision: Reject