Improving Deep Learning Interpretability by Saliency Guided Training
Authors: Aya Abdelsalam Ismail
Hector Corrada Bravo
Soheil Feizi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Interpretability

Abstract: Saliency methods have been widely used to highlight important input features in model predictions. Most existing methods use backpropagation on a modified gradient function to generate saliency maps. Thus, noisy gradients can result in unfaithful feature attributions. In this paper, we tackle this issue and introduce a {\it saliency guided training} procedure for neural networks to reduce noisy gradients used in predictions while retaining the predictive performance of the model. Our saliency guided training procedure iteratively masks features with small and potentially noisy gradients while maximizing the similarity of model outputs for both masked and unmasked inputs. We apply the saliency guided training procedure to various synthetic and real data sets from computer vision, natural language processing, and time series across diverse neural architectures, including Recurrent Neural Networks, Convolutional Networks, and Transformers. Through qualitative and quantitative evaluations, we show that saliency guided training procedure significantly improves model interpretability across various domains while preserving its predictive performance.

Improving Deep Learning Interpretability by Saliency Guided Training
Aya Abdelsalam Ismail, Soheil Feizi Héctor Corrada Bravo a aasalam, sfeizi miiii]p @css, umd. edu, corradaht @gne. com Department of Computer Science, University of Maryland Data Science and Statistical Computing, Genentech, Inc.
Abstract
Saliency methods have been widely used highlight important input features in model predictions. Most existing methods use backpropagation on a modified gradient function generate saliency maps. Thus, noisy gradients can result in unfaithful feature attributions. In this paper, we tackle this issue and introduce a saliency guided training procedure for neural networks to reduce noisy gradients used in predictions while retaining the predictive performance of the model. Our saliency guided training procedure iteratively masks features with small and poten- tially noisy gradients while maximizing the similarity of model outputs for both masked and unmasked inputs. We apply the saliency guided training procedure to various synthetic and real data sets from computer vision, natural language processing, and time series across diverse neural architectures, including Recurrent Neural Networks, Convolutional Networks, and Transformers. Through qualitative and quantitative evaluations, we show that saliency guided training procedure sig- nificantly improves model interpretability across various domains while preserving predictive performance.
Introduction
Deep Neural Networks (DNNs) have been widely used in a variety of different tasks 31 26 37 yet interpreting complex networks remains challenge. Reliable explanations necessary for critical domains like medicine, neuroscience, finance, and autonomous driving [9,34]. Explanations
35 As result, various interpretability methods were 51 A common approach for understanding model decisions is identify features in the input that highly influenced the final classification
also useful for model debugging (63
developed to understand DNNs [5]
49
22
53
52
32
52 47 36 64). Such approaches, known as saliency maps, often use gradient calculations assign an importance score to individual features, reflecting their influences on the model prediction.
decision [6] 63 53
Saliency methods aim to highlight meaningful input features in model predictions to humans; however, the maps produced are often noisy (i.e., contain visual noise). To improve the faithfulness of saliency maps, explanations methods that depend on more than were developed. For example, SmoothGrad (52] reduces saliency noise by adding noise to the input multiple times and then taking the average of the resulting saliency maps for each input. Integrated gradients (53] DeepL.IFT [7 and Layer-wise Relevance Propagation [5] backpropagate through a modified gradient function 3 while Singla al. [5I] studies the use of higher-order gradients in saliency maps.
higher-order gradient calculations
Authhrrs contrihuted equally (Code:
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this paper, we take different approach to improve the interpretability of deep neural networks instead of developing yet another saliency method, we propose a new training procedure that naturally leads to improved model explanations using current saliency methods. Our proposed training procedure, called saliency guided training, trains models that produce sparse, meaningful, and less noisy gradients without degrading model performance. This is done by iteratively masking input features with low gradient values (i.e., less important features) and then minimizing a loss function that combines (a) the KL divergence [27] between model outputs from the original and masked inputs, and (b) the appropriate loss function for the model prediction. This procedure reduces noise in model gradients without sacrificing its predictive performance.
To demonstrate the effectiivnees of our proposed saliency guided training approach, we consider a variety of classification tasks for images, language, and multivariate time series across diverse neural architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Network (RNNs), and Transformers. In particular, we observe that using saliency guided training in image classification tasks leads to a reduction in visual saliency noise and sparser saliency maps, as shown Figure 2 Saliency guided training also improves the comprehensiveness of the produced explanations for sentiment analysis, and fact extraction tasks as shown in Table [
In multivariate time series classification tasks, we observe an increase in the precision and recall of saliency maps when applying the proposed saliency guided training. Interestingly, we also find that the saliency guided training reduces the vanishing saliency issue of RNNs [20) as shown in Figure Finally, we note that although we use the vanilla gradient for masking in the saliency guided training procedure, we observe significant improvements in the explanations produced after training by several other gradient based saliency methods.
o
2 Background and Related Work
Interpretability is rapidly growing area with several diverse lines of research. One strand of inter- pretability considers post-hoc explanation methods, aiming to explain why a trained model made a specific prediction for given input. Post-hoc explanation methods can be divided into gradient-based methods 53 52 (7 36 46) that can be reformulated as computing backpropagation for a modified that perturb areas of the input and measure how much this changes the model output. Perkins et al. 40) uses gradients for feature selec- tion through Grafting. Another line of works aims measure the reliability of interpretability methods. This can be done by creating standardize benchmarks with interpretability metrics 18, 21, 11 56 45]
gradient function and perturbation-basee approaches [63 15 42 57]
41 or debugging explanations
by identifying test cases where explanations fail. Others 44 61] 20) focus on modifying neural architectures for better interpretability. Similar to our line of work, Ghaeini et al. [14) and Ross et al. (44) incorporate explanations into the learning process. However, Ghaeini et al. 14 relies on the existence of the ground truth explanations while Ross et al. 44 relies on the availability of annotations about incorrect explanations for particular input. Our proposed learning approach does not rely on such annotations; since most datasets only have ground truth labels, may not be practical to assume the availability of positive or negative explanations.
[ 24 15
Input level perturbation during training has been previously explored. B3
use attention incorporates attention maps into training to improve classification accuracy. De and Taylor 100 masks out
60
50
maps improve segmentation for weakly supervised localization. Wang al. 59
square regions of input during training as regularization technique to improve the robustness and overall performance of convolutional neural networks. Our work focuses on a different task which is increasing model interpretability through training in a self-supervised manner.
In this paper, we evaluate our learning procedure with the following saliency methods: Gradient (GRAD) [6] is the gradient of the output w.r.t the input. Integrated Gradients (IG) (53) calculates a path integral of the model gradient to the input from a non-informative reference point. DeepLIFT
(DL) 47 compares the activation of each neuron to reference activation; the relevance is the difference between the two activations. SmoothGrad (SG) [52] samples similar input by adding noise to the input and then takes the average of the resulting sensitivity maps for each sample. Gradient SHAP (GS) [36] adds noise to the input, then selects a point along the path between a reference point and input, and computes the gradient of outputs w.r.t those points.
We demonstrate the effectiveness of our training procedure using several neural network architectures: Convolution neural networks (CNNs) including VGG-16 (48) ResNet [16] and Temporal Convolu-

---

tional Network (TCN)
38 28 a CNN that handles sequences; Recurrent neural networks (RNNs)
including LSTM and LSTM with Input-Cell Attention 20 as well Transformers [58].
3 Notation
First, consider a classification problem on the imput data {(X4, such that each X [e1.-....2)
.yy] .Nn] E TR has N features and is the label. Let fo denote a neural network parame- terized by 0. The standard training of the network involves minimizing the cross-entropy loss L over the training set as follows:
minimize
(Jo (
The gradient of the network output fo (X) with respect to the input X is given by Vx So (X). Let S(.) be sorting function such that (2) is the ethh smallest element in Z. Hence, (V x fe (X)) is the sorted gradient. We define the input mask function M.).) such that My (x I; where S(x;) E {S, (e)))= with a mask distribution, i.e., (s()) X) removes the lowest features from X based on the order provided by S(X).
replaces all
For a language input, we use X = [Zi...., NN where x; E R is the feature embedding representing the hhh word of the input. that case, S(X) would sort sort elements of X based on the sum of the gradient of the embeddings for each word x and M& (SX), would mask the bottom words according to that sorting. For multivariate time series input, we use = [1,, ...,..... IFTT E RRxT where T is the number of time steps and F is the number of features per time step. Datt is the input feature at time t; sorting and masking would be done at the 24, level.
For two discrete probability distributions P and Q defined the same probability space X, the Kullback- Leibler (KL) divergence [27 (or, relative entropy) from Q to given as DKL:
P(x)
Dx (P
Σ zEX
P(x) log
(2)
Q(x)
Saliency Guided Training
Existing gradient-basedd methods produce noisy saliency maps as shown in Figure The saliency map noise may be partially due to some uninformative local variations in partial derivatives. Using standard training procedure based ERM (expectation risk minimization), the gradient of the model w.r.t. the input (i.e., V ff (X)) may fluctuate sharply via small input perturbations [52).
Image
Regular model guided model
Figure 1: Saliency maps produced by typical training versus saliency guided training.
gradient-based explanation methods faithfully interpret the model's predictions, irrelevant features should have gradient values close to zero. Building on this intuition, we introduce saliency guided training, a procedure to train neural networks such that input gradients computed from trained models provide more faithful measures to downstream (gradient-based saliency methods. Saliency guided training aims reduce gradient values of irrelevant features without sacrificing the model performance. During saliency guided training, for every input X, we create a new input by masking the features with low gradient values follows:
x = (s f (X)).
(3)
3

---

X is then passed through the network which results in an output fx)). In addition to the classifica- tion loss, the saliency guided training minimizes the KL divergence between fo (X) and fo (x( to ensure that the trained model produces similar output probability distributions over labels for both masked and unmasked imputs. The optimization problem for the saliency guided training is:
n
minimize Σ (1o T
(Xi). ADRI (Jo (x()))
where À is a hyperparameter balance between the cross-entropy classification loss and the KL divergence term. Since this loss function is differentiable with respect to 6, it can be optimized using existing gradient- based optimization methods. The KL divergence term encourages the model to produce similar outputs for the original input X and masked input X. For this to happen, the model will need to learn to assign low gradient values to irrelevant features in model predictions. This potentially results sparse and more faithful gradients as shown in Figure I
Masking functions: In images and time series data, features with low gradients are replaced with ran- dom values within the feature range. In language tasks, the masking function replaces the low salient word with the previous high salient word. This allows to emphasize on high salient words and re- move non-salient ones while maintaining the sentence length. The selection of & dataset-dependent It depends on the amount of irrelevant information in training sample. For example, since most pixels in MNIST are uninformative, larger is desired. Detailed hyperparameters used is available in the appendix. Note that, only input features masked during the saliency guided training.
Limitations: (a) Compared to traditional training, our proposed training procedure is more computa- tionally expensive. Specifically, the memory needed is doubled since now in addition to storing the batch, we are storing the masked batch as well. Similar adversarial training, this training process is slow and takes larger number of epochs to converge. For example, the standard training of CIFAR-10 model usually takes on average 118 epochs to converge where each epoch is roughly 24 seconds. Using the saliency guided training, the convergence takes about 124 epochs where each epoch takes roughly 75 seconds (all experiments on the same GPU). (b) Our training procedure requires two hyprpaaraeeeess k and which might require hyperparameter search (we find that works well in all of our experiments).
Algorithm 1: Saliency Guided Training
Given: Training samples X, # of features to be masked k, learning rate t. hyperparameter À
Initialize fo
for 4-1 to epochs do
for minibatch do
Compute the masked input:
Get sorted index for the gradient of output with respect the input.
I= S V (x))
Mask bottom & features of the original input.
x MT,
Compute the loss function:
L. L fo. (X),y)
, 3) ADRL (fo. (X) ll fo, (x))
Use the gradient to update network parameters:
So = So. - Vo
end
end

---

Experiments
experiments have been repeated 5 times; the results reported below are the average of the runs. Hyperparameters used for each experiment, along with the standard error bars and details on computational resources are available in supplementary materials.
5.1 Saliency Guided Training for Images
In the following section, we compare gradient-baat explanations produced by regular training versus saliency guided training for MNIST 33 trained on a simple CNN [29], for CIFARIO 25 trained on
R [16] and for BIRD B trained on VGG-16 [48]. Further details about the datasets and
models are available in the supplementary material.
(A)
Tradoood Teleoo Dutter Diiittioon
(B)
4
Traditonal
Salkenyy Guided
(C)
CFFAR Ciorbuton Owe Semples
BIR
Vear Diptbution Dyer Al Sempler
0.10
0.05
Condteett
0.00
0..55
i
-0.1
-0.10
-02-
-0.15
Traditioal
Salienay Guiced
Traditonaal
Sallenyy Guided
Training
Figure 2: (A) Comparison between different training methods on MNIST along with distributions of gradient values each sample. (B) Saliency maps for CIFARIU 10 and BIRD datasets using regular and saliency guided training. (C) Distribution of gradient means across examples. Maps produced by saliency guided training are more precise: most features have gradient values around zero with large gaps between mean and outliers. Here gradients around zero indicate uninformative features, while very large and very small gradients indicate informative features. Saliency guided training helps reduce noisy fluctuating gradients in between as shown in the box plots.
Quality of Saliency Maps for Images
For an image classification problem, in many cases, most features are redundant and not needed by the model make the prediction. Consider the background of an object in an image; although covers most of the image, backgrounds are often not essential in the classification task. If the model focusing on the object rather than the background, we would want background gradient (i.e., most of the features) to be close to zero.
The examples shown Figure were correctly classified by both models. Gradients are scaled per sample to have values between 1 and 1. In Figure 2 (A) and F (B), saliency maps produced by model trained with saliency guided training were more precise than that trained traditionally, Most saliency maps produced by saliency guided training highlight the object itself rather than the background across different datasets. The distributions gradient values per sample Figure (A) show that most features have small gradient values (near zero) with large separation of high salient features away from zero for saliency guided training. Similarly, (C) we find that over the entire dataset, gradient values produced by the saliency guided training tend to be concentrated around zero with large separation between the mean and outliers (highly salient features), indicating the model's ability to differentiate between informative and non-in non-informative features.
5

---

Traditional Saliency Guided training model accuracy drop different saliency methods MNIST
Gradient
SmoothGrad
Integrated Gradient
DeepLift
Gradient SHAP
DeepSHAP
100
Trad.
Model Accuracy
Sal.
Random
60
20
50
50
50
50
50
features
festuress
festures
Figure 3: Model accuracy drop when removing features with high saliency using traditional and the saliency guided training for different gradient-based methods against random baseline. A steeper drop indicates a better performance. We find that regardless of the saliency method used, the performance improves by the saliency guided training.
Model Accuracy Drop
We compare the saliency guided training and traditional training for different saliency methods with modification-basee evaluation 45
First, features are ranked according the saliency values. Then, higher-ranked features are recursively eliminated (the original background in MNIST replaces the eliminated features). Finally, the degradation the trained model accuracy reported. This is done at different feature percentages. A steeper drop indicates that the removed features affected the model accuracy more. Figure comparee the model performance degradation different gradient- gradient-based methods; the saliency guided training shows steeper accuracy drop regardless of the saliency method used.
This experiment can only be performed on a dataset like MNIST since the uninformative feature distribution is known (black background), while this is not the case in other datasets that have considered. Although such modification-basee evaluation methods have been applied Hooker et al. [18] showed that removing features produces samples from different data distribution violating the underlying IID assumption (i.e., the training and evaluation data come from identical distributions). When the feature replacement comes from different distribution, it is unclear whether the degradation the model performance from the distribution shift or the removal of informative features. For that reason, we need to make sure that the model is trained on the mask used during testing to avoid this undesired effect.
Feature replacement methods
80
other datasets, 45 23
80-
20-
20
4C
50
80
36.0 Soatures removed
Figure 4: Accuracy drop in different modification- based evaluation masking approaches.
Hooker al. 18 proposes ROAR where the model is retrained after the feature elimination. However, due to the data redundancy, the retrained model can rely on different features to achieve the same accuracy. Figure shows the model accuracy drop on traditionally trained MNIST when removing the salient features. The IID line represents replacing features with the black MNIST background (known uninformative distribution), which acts as the ground truth in this particular dataset. The OOD line represents replacing the features with the mean image pixel value as done by [45] 41 23 and ROAR shows replacing features with the mean value and retraining the model as proposed by Hooker et al. [18]. Since neither OOD nor ROAR produce results similar to those produced evaluation methods may provide
by the IID feature replacement, we argue that modification-baat unreliable results unless the uninformative IID distribution is known. We leave further exploration of modification-bart based evaluation methods to future work.
5.2 Saliency Guided Training for Language
We compare the interpretability of recurrent models trained on language tasks using the ERASER benchmark. ERASER was designed to capture how well an explanation provided by models aligns

---

with human rationales and how faithful these explanations are (i.e., the degree to which explanation influences the predictions). For our purpose, we only focus on the faithfulness of the explanations.
ERASER provides two metrics to measure interpretability. Comprehensiveness evaluates if all features needed make a prediction are selected. To caleulate explanation comprehensiveness, new input X; is created such that X; X; R, where R₂ is predicted rationales (i.e. the words selected by saliency method as informative). Let fo (X:), be the prediction of model for class j. The model comprehensiveness is calculated as:
Comprehensiveness- = So (X:), - So (X:),
high score here implies that the explanation removed was influential the predictions. The second metric is Sufficiency that evaluates if the extracted explanations contain enough signal make prediction. The following equation gives the explanation sufficiency:
Sufficiency fo (Xx), - fo (Ri);
lower score implies that the explanations are adequate for a model prediction. The comprehensive- ness and sufficiency were calculated at different percentages of features (similar to [11] percentages are 1%, 5%, 10%, 20% and 50%), and Area Over the Perturbation Curve (AOPC) is reported.
We focus on datasets that can be formulated as a classification problem: Movie Reviews: [62 positive/negative sentiment classification for movie reviews. FEVER: (55) fact extraction and verification dataset where the goal is verifying claims from textual sources; each claim can either be supported or refuted. e-SNLI: [8] natural language inference task where sentence pairs are labeled as entailment, contradiction, neutral and, supporting.
Word embeddings generated from Glove [9]; then passed to a bidirectional LSTM [17] for classification. Table compares the scores produced by different saliency methods for traditional and saliency guided training against random assignment baseline. We found that saliency guided training results in a significant improvement in both comprehensiveness and sufficiently for sentiment analysis task Movie Reviews dataset. While for fact extraction task FEVER dataset, and natural language inference task e-SNLI dataset saliency guided training improves comprehensiveness and there is no obvious improvement in sufficiency (this might be due the adversarial effect of shrinking the sentence to much smaller size since the number of words identified as "rationales" is smaller than the remaining words).
Gradient
Integrated Gradient Sal. Guided
SmoothGrad Sal. Guided
Random
Trad. Sal. Guided
Trad.
Trad.
Movies
Comprehensiveness 1
0.200
0,240
0.265 0.054
0.306
0.198 0.034
0.256
0.056
Sufficiency +.
0.042
0,013
0.002
0.008
0.294
FEVER
Comprchensivenesst
0.007
0.008
0.008
0.009
0.007
0.008
0.001
Sufficiency.
0.012
0.011
0.005
0.004
0.006
0.006
0.003
e-SNLI
Comprchensiveness f
0.117
0.126
0.099
0.104
0.117
0.118 0.455
0.058
Sufficiency!
0.420
0.387
0.461
0.419
0.476
0,366
Table 1: Eraser benchmark scores: Comprehensiveness and sufficiency are in terms of AOPC. 'Random' is baseline when words are assigned random scores.
5.3 Saliency Guided Training for Time Series
We evaluated saliency guided training for multivariate time series, both quality on multivariate time series MNIST and quantitatively through synthetic data.
Saliency Maps Quality for Multivariate Time Series
We compare the saliency maps produced on MNIST treated as multivariate time series where one image axis is time. Figure 5 shows the saliency maps produced by different (neural architecture, saliency method) pairs when different training procedures were used. There a visible improvement in saliency quality across different networks when saliency guided training used.
7

---

LSTM call att.
Transformer
Figure Saliency maps produced for (neural architecture. saliency method) pairs. Traditional training was used for networks in the 1st row, while saliency guided training was used for the 2nd row. Grad, DL, GS and DLS stand for Gradient, DeepLift, Gradient SHAP and DeepSHAP, respectively. There is an improvement in the quality of saliency maps when saliency guided training is used.
Quantitative Analysis on Synthetic Data
We evaluated the saliency guided training on a multivariate time series benchmark proposed by Ismail al. [21] The benchmark consists of 10 synthetic datasets, each examining different design aspects in typical time series datasets. Informative features are highlighted by the addition of a constant j the positive class and subtraction of μ from the negative class. Following Ismail al. (21), we compare 4 neural architectures; LSTM [17]. LSTM with Input-Cell Attention (20), Temporal Convolutional Network (TCN) (8 and, Transformers [58]. Additional details about the dataset and architectures are provided in the supplementary material.
Quantitatively measuring the interpretability of (neural architecture, saliency method) pair involves applying the saliency method, ranking features according to the saliency values, replacing high salient features with uninformative features from the original distribution at different percentages. Finally, the area under the precision curve (AUP) and the area under the recall curve (AUR) is calculated by the precision/recall values different levels of degradation. Similar to Ismail et al. [21]. compare the AUP and AUR with a random baseline; since the baseline might be different for different models, we reported the difference between metrics values generated using the saliency method and the baseline. For example, the difference between gradient and random baseline model is trained traditionally calculated as:
when the
(ifd(AU))rrdd.Tree AU PUrod,,TTrad AU PUPPE Random, Grrnd
Similarly difference when the model is trained using saliency guided training is:
Diff(AUU)Grad Grad, AU AUPd,dd AU
PRandom, Random,,all
(8)
The mean metrics over all 10 datasets is shown in Table Higher values indicate better performance; negative values indicate performance similar to random feature assignment. Overall, the best performance was achieved by (TCN, Integrated gradients) when using saliency guided training. Detailed results for each dataset are available in the supplementary material.
Gradient
Integrated Gradient
DeepLIFF
Gradient
DeepStAP
SmoothGrad
Menre
Architecture
Trad.
Sal.
Trad.
Sal.
Trad.
Sal.
Trad.
Sal.
Trad.
Sal.
Trad.
Sal.
LSTM
.0.113
40.199
.0.083
-0.024
.0.097
.0.108
-0.088
-0.069
-0.098
-0.109
-0.110
-0.097
(If((AUP)
LSTM Inpul.
TCN Transformer
0.106
0.168
0.233
0.245
0.291
0.202
0.248
0.263
0,270
0.198
0.235
0.250
0,288
0,263
0.280
0.088
0.155
.0.054
-0.062
0.067
0.044
.0.040
.0.032
0.069
0.023
.0.014
-0.055
00.188
.0.046
LSTM
-0.017
0.047
0.089
0.060
DIF(AUU)
LSTM + Input.
0.075
0.136
0.185
0.798
0.187
0.204
0.182
0.196
0.183
0.201
0.843
0.111
TCN
0.125
0.171
0.191
0.220
0.202
0.204
0.185
0,209
0,196
0.192
0.016
0,138
Transformer 0.102
0.104
0,182
0.176
0.145
0.145
0.171
0.162
0.101
0.065
0.640
0.018
Table 2: The mean difference in weighted AUP and AUR for different (neural architecture, saliency
method) pairs. Overall, the best preference was achieved by TCN when using Integrated gradients as
a saliency method and saliency guided training procedure.
Saliency Guided Training reduces vanishing saliency of recurrent neural networks
Ismail et al. (2 showed that saliency maps in RNNs vanish over time, biasing detection of salient features only later time steps. This section investigates if using saliency guided training reduces
the vanishing saliency issue in RNNs. Repeating experiments done by Ismail et al. 220). three
8

---

synthetic datasets were generated as shown Figure (A). The specific features and the time intervals (boxes) on which they are considered important are varied between datasets to test the model's ability to capture importance at different time intervals. We trained an L.TTM with traditional and saliency guided training procedures.
The area under precision curve (AUP) and the area under the recall curve (AUR) are calculated by the precision/recall values at different levels of degradation. Higher AUP and AUR suggest better performance. Results are shown in Figure (B).
A traditionally trained LSTM shows clear bias in detecting features in the later time steps; AUP and AUR increase as informative features move to later time steps. When saliency guided training is used, LSTM was able to identify informative features regardless of their locations in time.
(A)
AUP
AUR
Karterr
Calence Carded
Interrediaas
0.4-
Laber
Earder itemmedtee Laber
Farter Intermedatee Later
Figure 6: (A) Samples from 3 different simulated datasets, informative features are located at the earlier, intermediate, and later time steps. (B) AUP and AUR were produced by LSTM by traditional and saliency guided training procedures. Traditionally trained LSTM shows clear bias in detecting features in the later time steps. When saliency guided training is used, there is no time bias.
6 Summary and Conclusion
We propose saliency guided training as a new training procedure that improves the quality of explanations produced by existing gradient-based saliency methods. saliency guided training is optimized to reduce gradient values for irrelevant features. This is done by masking input features with low gradients and then minimizing the KI. divergence between outputs from the original and masked inputs along with the main loss function. We demonstrated the effectiveness of the saliency guided training on images, language, and multivariate time series.
Our proposed training method encourages models to sharpen the tradient-based explanations they provide. It does this however without requiring explanations as input. It instead may be cast reg- ularization procedure where regularization is provided by feature sparsity driven by a gradient-based feature attribution. This is an alternative approach using ground truth explanations force the model to be right the right reasons [44]. We found that training model explanations in an un- supervised fashion also improves model faithfulness. This opens an interesting avenue for other unsu- pervised, perhaps regularization-bass methods to improve the interpretability of prediction models.
7 Acknowledgments
This project was supported in part by NSF CAREER AWARD 1942230, grant from NIST 60NANB20D134, NSF award CDS&E: 1854532, ONR grant 13370299 and AWS Machine Learning Research Award.
References
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, 2018.

---

[2] Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging tests for model explanations. arXiv preprint arXiv:2011.05429 2020.
[3] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of mradient-based attribution methods for deep neural networks. International Conference on Learning Representations, 2018.
[4] Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in Neural Information Processing Systems, 2014.
[5] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. In PLOS ONE, 2015.
[6] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert MÄäller. How to explain individual classification decisions. In Journal of Machine Learning Research, 2010.
[7] Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint 1rXiv:/803.0000 1803.0127,, 2018.
[8] Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natu- ral language inference with natural language explanations. arXiv preprint 192 2018.
[9] Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In International conference on knowledge discovery and data mining, 2015.
[10] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:/708. 04552,, 2017.
[11] Jay De Young, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace. Eraser: A benchmark to evaluate rationalized nlp models. arXiv preprint arXiv: 1911.0399 2019.
[12) Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into soft decision tree. arXiv preprint
2017.
[13] Gerry. 265 bird species, 2021. URL. https: httpss://www.kagg kaggle wwtpgggll.....///o
rdd-bird-species!
[14] Reza Ghaeini, Xiaoli Z Fern, Hamed Shahbazi, and Prasad Tadepalli. Saliency learning: Teaching the model where to pay attention. arXiv preprint arXiv: DrrXiv:/002.00m 19000000099999 2019.
[15] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In AAAI Conference on Artificial Intelligence, 2019.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
[17] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. In Neural computation, 1997.
[18] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for inter- pretability methods in deep neural networks. In Advances in Neural Information Processing Systems, 2019.
[19] Qibin Hou, Peng- Tao Jiang, Yunchao Wei, and Ming-Ming Cheng. Self-erasing network for integral object attention. arXiv preprint
2018.
[20] Aya Abdelsalam Ismail, Mohamed Gunady, Luiz Pessoa, Hector Corrada Bravo, and Soheil Feizi. Input-cell attention reduces vanishing saliency of recurrent neural networks. In Advances in Neural Information Processing Systems, 2019.
10

---

[21] Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi. Benchmark- ing deep learning interpretability in time series predictions. arXiv preprint arX. 2020.
[22] Pieter-Jan Kindermans, Kristof Schütt, Klaus-Robert Müller, and Sven Dähne. Investigating the influence noise and distractors on the interpretation of neural networks. arXiv preprint urXiv:161..0727770 2016.
[23] Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Roben Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: Patternnet and patternattribution. arXiv preprint arXiv: 1rXxv:1705.05595 2017.
[24] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo. Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning. 2019.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009. 2009.
[26] Alex Krizhevsky, llya Sutskever, and Geoffre E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012.
[27] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 1951.
[28] Colin Lea, Michael Flynn, Rene Vidal, Austin Reiter, and Gregory Hager. Temporal convolu- tional networks for action segmentation and detection. Conference on Computer Vision and Pattern Recognition, 2017.
[29] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
[30] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database, 2010.
[31] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 2015.
[32] Alexander Levine, Sahil Singla, and Soheil Feizi. Certifiably robust interpretation in deep 2019.
learning, arXiv preprint
[33] Kunpeng Li, Ziyan Wu, Kuan-Chuan Peng. Jan Ernst, and Yun Fu. Tell me where to look: Guided attention inference network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[34] Zachary C Lipton. The mythos of model interpretability. In Queue, 2018
[35] Yin Lou, Rich Caruana, and Johannes Gehrke Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, 2012.
[36] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, 2017.
[37] Ziad Obermeyer and Ezekiel J Emanuel. Predicting the future-big data, machine learning, and clinical medicine. In The New England journal of medicine. 2016.
[38] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint a6Xiv: 1e0.:03999999 2016.
[39] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014.
[40] Simon Perkins, Kevin I Lacker, and James Theiler. Grafting: Fast, incremental feature selection by gradient descent in function space. The Journal of Machine Learning Research, 2003.
11

---

[41] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.07421. 2018.
[42] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, 2016.
[43] Michael L Rich. Machine learning, automated suspicion algorithms, and the fourth amendment. In University of Pennsylvania Law Review, 2016.
[44] Andrew Slavin Ross, Michael C Hughes, and Finale Doshi Velez. Right for the right rea- sons: Training differentiable models by constraining their explanations. arXiv preprint urXiv:1703.0377 2017.
[45] Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus- Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 2016.
[46] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, 2017.
[47] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, 2017.
[48] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[49] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, 2013.
[50] Krishna Kumar Singh and Yong Jae Lee. Hide-and-seck: Forcing a network to be meticulous for weakly-supervised object and action localization. In 2017 JEEE international conference on computer vision (ICCV). IEEE, 2017.
[51] Sahil Singla, Eric Wallace, Shi Feng, and Soheil Feizi. Understanding impacts of high-order loss approximations and features in deep learning interpretation. International Conference on Machine Learning. PMLR, 2019.
[52] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint WvXiv:1706.0382. 2017.
[53] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 2017
[54] Harini Suresh, Nathan Hunt, Alistair Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Clinical intervention prediction and understanding using deep networks. arXiv preprint htt.:1705.04999 2017.
[55] James Thome, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: large-scale dataset for fact extraction and verification. arXiv preprint arXiv: 1803.05355 2018 8.
[56] Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece. Sanity checks for saliency metrics. In Proceedings of the AAAI conference on artificial intelligence, 2020.
[57] Sana Tonekaboni, Shalmali Joshi, Kieran Campbell, David Duvenaud, and Anna Goldenberg. What went wrong and when? instance- wise feature importance for seness models, arXiv preprint wwXiv:2003.0282/1 2020.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 2017.
12

---

[59] Lezi Wang, Ziyan Wu, Srikrishna Karanam, ChannChuann Peng, Rajat Vikram Singh, Bo Liu, and Dimitris N Metaxas. Sharpen focus: Learning with attention separability and consistency. In Proceedings of the IEEF/CVF International Conference on Computer Vision, 2019.
[60] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In Proceedings of the IEEE conference on computer vision and paltern recognition, 2017.
[61] Mike Wu, Michael C Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi- Velez. Beyond sparsity: Tree regularization of deep models for interpretability. In AAAI Conference Artificial Intelligence, 2018.
[62] Omar Zaidan and Jason Eisner. Modeling annotators: A generative approach to learning from annotator rationales. In Proceedings of the 2008 conference on Empirical methods in natural language processing, 2008.
[63] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, 2014.
[64] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
Checklist
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above, along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] section 4
(c) Did you discuss any potential negative social impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] In the supplemental material
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In the supplemental material
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In the supplemental material
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "hsG2oSfBDmA", "reviewer": "Reviewer_jiTu", "summary": "The authors propose an interpretable training procedure, in which the input features with the bottom k input gradients are masked. The loss term is comprised of the original task loss, plus a KL divergence between the model outputs on the original and masked inputs. The authors perform experiments on 3 image datasets, 3 language datasets, and a suite of synthetic time series datasets. They use model accuracy drop as a metric to evaluate saliency map quality.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*** Originality *** There are many prior works in \"interpretable training\", and it seems uneasy to name the method after this general field, especially given the prior works. Some works which incorporate interpretability into the training process in a self-supervised fashion include:\n\n- Tell Me Where to Look: Guided Attention Inference Network by Kunpeng Li et. al.\n- Sharpen Focus: Learning with Attention Separability and Consistency by Lezi Wang et. al.\n- Self-Erasing Network for Integral Object Attention by Qibin Hou et. al. - from NeurIPS 2018\n- Object Region Mining with Adversarial Erasing: A Simple Classification to\nSemantic Segmentation Approach by Yunchao Wei et. al.\n\nAlthough \"interpretable training\" has been around for a while, the authors do not include another interpretable training baseline. And although the papers I have mentioned evaluate on vision tasks, many of the ideas are extendable to the settings which the authors consider here, in the same manner (e.g., running the saliency methods on text instead of images).\n\n*** Quality *** \n\nI did not understand the ability of the box plots to show interpretability. As described in Figure 2: \"most features have gradient values around zero with large gaps between mean and outliers, indicating the model’s ability to differentiate between informative and non-informative features.\" - Why is this true?\n\nAlso, I am wary of these results, since they seem to go against Goodhart's Law: \"When a measure becomes a target, it ceases to become a good measure.\" The input gradients were used directly in the model optimization process. If a saliency method is used for evaluating the interpretability, it should be a different one than what was used in optimization - for example, something black-box like RISE (by Vitali Petsiuk et. al.). Furthermore, there is prior work which shows that input gradients are often not interpretable (e.g., Bridging Adversarial Robustness and Gradient Interpretability by Beomsu Kim et. al.).\n\nIt is admirable that the authors evaluate their algorithm on many benchmarks, in a variety of forms of data - MNIST, CIFAR, and BIRD for images, Movies, FEVER, and e-SNLI for language, and synthetic time series benchmarks. They also use a variety of model architectures.\n\n*** Clarity *** The paper contains many grammar errors, although they are generally not a hindrance to understanding. For example, Line 119: \"Although we use on vanilla gradients during training to mask unimportant features which is not computationally expensive, this training procedure (similar to other training procedures like adversarial training as an example) is more expensive than traditional training.\"\n\nOn Line 138, the statement: \"We would expect the background gradient (i.e., most of the features) to be close to zero.\" This is not always true (e.g., the background can easily be correlated with the object, and thus feature importance on the background is likely to be non-zero). But, the model focusing on object features rather than background features can instead be stated as a desirable quality to have, rather than the expectation.\n\nTypo Line 172 - MINST\n\n*** Significance *** It is great that the method the authors propose is self-supervised, and does not rely on the need for ground truth saliency maps, which are often not available (at least, from true GT annotations. It could be possible to get pseudo-GT saliency guidance from a pretrained model which is, say, good at recognizing \"objectness\"). The loss function is also nice in its simplicity. However, as stated above, there are many prior works in interpretable training, which this paper has not compared to.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address the limitations of their method around Line 119, about the additional computation needed for training and about the extra hyperparameters their method introduces. The authors mention briefly at the start of the introduction about high-risk settings where interpretability is needed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Z40kpbGTcXt", "reviewer": "Reviewer_zFG8", "summary": "The paper proposes a new interpretable training procedure for neural networks. The paper argues that the existing model’s prediction explanation through saliency maps uses backpropagation with noisy gradients. The approaches use gradient calculations to assign an importance score to individual features, reflecting their influence on the model prediction. As the importance scoring is dependent on the Gradients, noisy gradients can result in unfaithful feature attributions.  The problem was attempted to tackle by introducing an interpretable training procedure to reduce noisy gradients of the existing saliency methods: Gradient (GRAD), Integrated Gradients (IG), DeepLIFT (DL), SmoothGrad (SG), and Gradient SHAP (GS). The proposed interpretable training procedure (interpretable training) iteratively masks features with small and potentially noisy gradients while maximizing the similarity of the model outputs for both masked and unmasked inputs using a loss function that combines KL divergence and another loss function. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The contribution and clarity of the paper can be improved by addressing the following questions:\n\nRe: “most features have gradient values around zero with large gaps between mean and outliers, indicating the model’s ability to differentiate between informative and non-informative features.” --- I don’t see the connection between the large gap in gradient values (between mean and outliers) and models’ ability to differentiate between informative and non-informative features. Gradient values can be related to the long-term dependency in backpropagation algorithms, however, I am not sure how that can be related to informative and non-informative features.\n\nRelevant to the above, Re:” If gradient-based explanation methods faithfully interpret the model’s predictions, irrelevant features should have gradient values close to zero.”: The vanishing gradient problems in neural network (e.g. LSTM) training suggest that the small gradients are related to the longer sequence (long term dependency) due to the repeated application of chain rule in the backpropagation algorithms. Thus, gradients get smaller not because the features in the long sequences are irrelevant.\n\nIt would be useful to have elaborated examples presented to show how the small gradients and irrelevant features are related. For example, there are two examples in a relevant previous work [1] show how gradient was used for feature selection.\n\nRe: “S(X) would sort elements of X based on the sum of feature embeddings of each word x ”: What is the interpretation of the sum of the feature embeddings of each word? Feature embeddings are points in a vector space and summing up the coordinate values in that vector space for a word for sorting might not have any meaning. \n\n\nMinor comments:\n\nLn 25: “the maps produced are often noisy. ”--- Not sure what does this line means.\n\n\n[1] Perkins, S., Lacker, K., & Theiler, J. (2003). Grafting: Fast, incremental feature selection by gradient descent in function space. The Journal of Machine Learning Research, 3, 1333-1356.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "XOueSQJFPa", "reviewer": "Reviewer_KCTi", "summary": "This paper proposes a method (interpretable training) to improve the interpretability of saliency methods. The interpretable training method iteratively masks the features with small vanilla gradients and maximizes the similarity of the model outputs for both masked and unmasked inputs. They demonstrate this training procedure can improve interpretability in various data sets from CV, NLP, and time series across neural architectures, including RNNs, CNNs, and Transformers. They also find this method can reduce the vanishing saliency issue of RNNs. Lastly, they demonstrate improvements of the explanations produced after training by several gradient-based saliency methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Based on an intuitive idea that if gradient-based explanation methods faithfully interpret the model's predictions, irrelevant features should have gradient values close to zero, the authors provide a method (interpretable training) to improve interpretability. The proposed method combines the masking approach (line 103) and the KL divergence regularization (line 97) to solve the optimization problem in different tasks. The authors formulated this interpretable training with a concise equation of loss function (i.e., the prediction loss and the KL loss, line 107). The algorithm for interpretable training (Algorithm 1) is simple but novel for computing the masked input in each minibatch. Nevertheless, the reviewer thinks it is an effective approach to improve explainability.  \n\nPresentation Quality: The authors give extensive empirical analysis, especially for CV tasks (i.e., qualitative analysis in Figures 2A and 2B, quantitative analysis in Figure 2C and Figure 3), to demonstrate the better performance for interpretable training compared with other regular training methods. However, the following substantial issues should be addressed: (1) In Table 1, the performance for interpretable training for NLP tasks does not demonstrate consistent performance; the reason why a lower sufficiency for both FEVER and e-SNLI is not explained.  (2) In Table 1, for NLP tasks, only the vanilla gradient-based method is used; how about the more popular ones like DeepLift or IG? (3) In lines 114-116, the authors give how to implement masking functions in image, language, but how about for the time series?  Why can't these operations replace the low salient word with the previous high salient word (line 115) guarantee the argument of assigning low gradient values to irrelevant features in model predictions (in line 112), any theoretical explanation?  (4) Since the parameter k is very important for masking operation, in lines 116 -117, the authors point out how to select k for images (MNIST dataset); however, the selection of k for language and time series is not clear. (5) Other minor problems, e.g., the sequence number should be added for other formulas other than the KL-divergence (Eq (1)),  line number for Algorithm 1. \n\n-----New comments after reading authors' response\n\nI appreciate the response given by the authors, and most of my concerns have been successfully addressed by the responses. I think this work is  novel compared to other existing works on \"interpretable training\" in that this method can be used in more broad areas, NLP, CV, and time series prediction tasks, which is achieved via masking gradients without using attention maps. As such, I raise my rating to 6 to reflect my new assessment of the updated manuscript.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Ema6ht2uogm", "reviewer": "Reviewer_ooFj", "summary": "This paper addresses the noise in saliency map techniques, which seek to explain what part of the input is responsible for a model's output decision, by introducing a novel training paradigm in which noise is added to the least important input features and the model is encouraged to produce similar outputs for original and noise-augmented inputs. This paper shows extensive experimental results in a variety of ML domains, such as image, language, and time series data. Additionally, the paper adds interesting commentary on input feature replacement paradigms such as ROAR and OOD as well as analysis that shows how their interpretable training paradigm reduces the vanishing saliency problem in RNNs.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well motivated -- a number of saliency methods are affected by noisy gradients, often mitigating this effect with solutions that aren't well motivated (e.g., changing backprop rules). The paper is extensive, thoughtful, and thorough in its experimentation and analysis, covering a wide range of ML domains and showing convincingly that their paradigm works across different settings, for instance, by reducing the noise in saliency maps, widening the separation between uninformative and informative input features as highlighted by saliency methods, improving performance of saliency map methods when evaluated on the model accuracy drop benchmark. In particular, its analysis on feature replacement paradigms (section 5.1, under \"model accuracy drop\") a useful insight into how to consider methods like ROAR and OOD. Additionally, its experiment showing that its interpretable training paradigm mitigates the vanishing saliency of recurrent neural networks is an exciting result. Both the insights presented in this paper as well as the novel training paradigm introduced can be used and built upon in future works.\n\nThe rest of this review contains large and small suggestions for the authors to consider.\n\nLarge:\n* Could this paradigm be adapted and used as a fine-tuning procedure? Would we see the same benefits (for less training time)?\n* How does this training paradigm impact performance (e.g., accuracy)? Accuracy numbers provided for language but not other domains\n* Figure 2A -- distribution plots are hard to interpret -- use the box plot versions for MNIST (as shown in supp mat)\n* Table 1 -- include additional results when using other saliency methods as well as LSTM Interpretable + other saliency methods\n\nSmall: \n* Additional related works from weak localization literature that use input-level perturbations during training: Wei et al., CVPR 2017 (Object Region Mining with Adversarial Erasure), Singh & Lee, ICCV 2017 (Hide-and-Seek), DeVries 2017 (Cutout).\n* Line 119: Expand a bit more on what additional computational cost is needed (does it take the same # of epochs to converge; what's the added memory cost?)\n* Explain more clearly why we should expect feature replacement methods to match IID replacement\n* Line 114: Be more explicit that only input features are being masked (e.g., not intermediate features), be more clear in main paper about what masked features are being replaced by (e.g., copy parts from supp mat)\n* Line 132: Spell out which VGG network is used (e.g., VGG-16?)\n* Line 186: \\tilde{X} is already used on line 103 - use different notation for clarity\n* Line 186: Explain what R_i (predicted rationales) are in this context\n* Figure 5 -- in caption, explain what the abbreviations for saliency methods mean (Grad, DL, GS, DLS - last few are a bit unclear)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly mention some basic limitations of their method (computational cost, hyperparameter tuning, requiring training); however, they can improve their limitations section by fleshing out in more detail the additional computational cost. Furthermore, they did not discuss potential negative impacts of their work. Some directions for discussing potential negative impacts include misusing saliency methods, lack of consensus on how to evaluate saliency methods, etc. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new procedure for training neural networks that achieves improved interpretability by masking the bottom k input gradients, helping address the noise in saliency map techniques. The reviewers found the paper well motivated, liked the experiments (especially the fact that they’re conducted on many domains), and found the method novel. 

The reviewers appreciated the thoroughness of the rebuttal, both in terms of experiments and responses, clarifying the points around related works, writing and additional discussions.  I therefore recommend acceptance.   I encourage the authors to integrate all the suggestions from the reviewers into the camera ready.

Decision: Accept (Poster)