Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a new interpretable training procedure for neural networks. The paper argues that the existing model’s prediction explanation through saliency maps uses backpropagation with noisy gradients. The approaches use gradient calculations to assign an importance score to individual features, reflecting their influence on the model prediction. As the importance scoring is dependent on the Gradients, noisy gradients can result in unfaithful feature attributions.  The problem was attempted to tackle by introducing an interpretable training procedure to reduce noisy gradients of the existing saliency methods: Gradient (GRAD), Integrated Gradients (IG), DeepLIFT (DL), SmoothGrad (SG), and Gradient SHAP (GS). The proposed interpretable training procedure (interpretable training) iteratively masks features with small and potentially noisy gradients while maximizing the similarity of the model outputs for both masked and unmasked inputs using a loss function that combines KL divergence and another loss function. 
main_review: The contribution and clarity of the paper can be improved by addressing the following questions:

Re: “most features have gradient values around zero with large gaps between mean and outliers, indicating the model’s ability to differentiate between informative and non-informative features.” --- I don’t see the connection between the large gap in gradient values (between mean and outliers) and models’ ability to differentiate between informative and non-informative features. Gradient values can be related to the long-term dependency in backpropagation algorithms, however, I am not sure how that can be related to informative and non-informative features.

Relevant to the above, Re:” If gradient-based explanation methods faithfully interpret the model’s predictions, irrelevant features should have gradient values close to zero.”: The vanishing gradient problems in neural network (e.g. LSTM) training suggest that the small gradients are related to the longer sequence (long term dependency) due to the repeated application of chain rule in the backpropagation algorithms. Thus, gradients get smaller not because the features in the long sequences are irrelevant.

It would be useful to have elaborated examples presented to show how the small gradients and irrelevant features are related. For example, there are two examples in a relevant previous work [1] show how gradient was used for feature selection.

Re: “S(X) would sort elements of X based on the sum of feature embeddings of each word x ”: What is the interpretation of the sum of the feature embeddings of each word? Feature embeddings are points in a vector space and summing up the coordinate values in that vector space for a word for sorting might not have any meaning. 


Minor comments:

Ln 25: “the maps produced are often noisy. ”--- Not sure what does this line means.


[1] Perkins, S., Lacker, K., & Theiler, J. (2003). Grafting: Fast, incremental feature selection by gradient descent in function space. The Journal of Machine Learning Research, 3, 1333-1356.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose an interpretable training procedure, in which the input features with the bottom k input gradients are masked. The loss term is comprised of the original task loss, plus a KL divergence between the model outputs on the original and masked inputs. The authors perform experiments on 3 image datasets, 3 language datasets, and a suite of synthetic time series datasets. They use model accuracy drop as a metric to evaluate saliency map quality.
main_review: *** Originality *** There are many prior works in "interpretable training", and it seems uneasy to name the method after this general field, especially given the prior works. Some works which incorporate interpretability into the training process in a self-supervised fashion include:

- Tell Me Where to Look: Guided Attention Inference Network by Kunpeng Li et. al.
- Sharpen Focus: Learning with Attention Separability and Consistency by Lezi Wang et. al.
- Self-Erasing Network for Integral Object Attention by Qibin Hou et. al. - from NeurIPS 2018
- Object Region Mining with Adversarial Erasing: A Simple Classification to
Semantic Segmentation Approach by Yunchao Wei et. al.

Although "interpretable training" has been around for a while, the authors do not include another interpretable training baseline. And although the papers I have mentioned evaluate on vision tasks, many of the ideas are extendable to the settings which the authors consider here, in the same manner (e.g., running the saliency methods on text instead of images).

*** Quality *** 

I did not understand the ability of the box plots to show interpretability. As described in Figure 2: "most features have gradient values around zero with large gaps between mean and outliers, indicating the model’s ability to differentiate between informative and non-informative features." - Why is this true?

Also, I am wary of these results, since they seem to go against Goodhart's Law: "When a measure becomes a target, it ceases to become a good measure." The input gradients were used directly in the model optimization process. If a saliency method is used for evaluating the interpretability, it should be a different one than what was used in optimization - for example, something black-box like RISE (by Vitali Petsiuk et. al.). Furthermore, there is prior work which shows that input gradients are often not interpretable (e.g., Bridging Adversarial Robustness and Gradient Interpretability by Beomsu Kim et. al.).

It is admirable that the authors evaluate their algorithm on many benchmarks, in a variety of forms of data - MNIST, CIFAR, and BIRD for images, Movies, FEVER, and e-SNLI for language, and synthetic time series benchmarks. They also use a variety of model architectures.

*** Clarity *** The paper contains many grammar errors, although they are generally not a hindrance to understanding. For example, Line 119: "Although we use on vanilla gradients during training to mask unimportant features which is not computationally expensive, this training procedure (similar to other training procedures like adversarial training as an example) is more expensive than traditional training."

On Line 138, the statement: "We would expect the background gradient (i.e., most of the features) to be close to zero." This is not always true (e.g., the background can easily be correlated with the object, and thus feature importance on the background is likely to be non-zero). But, the model focusing on object features rather than background features can instead be stated as a desirable quality to have, rather than the expectation.

Typo Line 172 - MINST

*** Significance *** It is great that the method the authors propose is self-supervised, and does not rely on the need for ground truth saliency maps, which are often not available (at least, from true GT annotations. It could be possible to get pseudo-GT saliency guidance from a pretrained model which is, say, good at recognizing "objectness"). The loss function is also nice in its simplicity. However, as stated above, there are many prior works in interpretable training, which this paper has not compared to.
limitations_and_societal_impact: The authors address the limitations of their method around Line 119, about the additional computation needed for training and about the extra hyperparameters their method introduces. The authors mention briefly at the start of the introduction about high-risk settings where interpretability is needed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method (interpretable training) to improve the interpretability of saliency methods. The interpretable training method iteratively masks the features with small vanilla gradients and maximizes the similarity of the model outputs for both masked and unmasked inputs. They demonstrate this training procedure can improve interpretability in various data sets from CV, NLP, and time series across neural architectures, including RNNs, CNNs, and Transformers. They also find this method can reduce the vanishing saliency issue of RNNs. Lastly, they demonstrate improvements of the explanations produced after training by several gradient-based saliency methods.
main_review: Originality: Based on an intuitive idea that if gradient-based explanation methods faithfully interpret the model's predictions, irrelevant features should have gradient values close to zero, the authors provide a method (interpretable training) to improve interpretability. The proposed method combines the masking approach (line 103) and the KL divergence regularization (line 97) to solve the optimization problem in different tasks. The authors formulated this interpretable training with a concise equation of loss function (i.e., the prediction loss and the KL loss, line 107). The algorithm for interpretable training (Algorithm 1) is simple but novel for computing the masked input in each minibatch. Nevertheless, the reviewer thinks it is an effective approach to improve explainability.  

Presentation Quality: The authors give extensive empirical analysis, especially for CV tasks (i.e., qualitative analysis in Figures 2A and 2B, quantitative analysis in Figure 2C and Figure 3), to demonstrate the better performance for interpretable training compared with other regular training methods. However, the following substantial issues should be addressed: (1) In Table 1, the performance for interpretable training for NLP tasks does not demonstrate consistent performance; the reason why a lower sufficiency for both FEVER and e-SNLI is not explained.  (2) In Table 1, for NLP tasks, only the vanilla gradient-based method is used; how about the more popular ones like DeepLift or IG? (3) In lines 114-116, the authors give how to implement masking functions in image, language, but how about for the time series?  Why can't these operations replace the low salient word with the previous high salient word (line 115) guarantee the argument of assigning low gradient values to irrelevant features in model predictions (in line 112), any theoretical explanation?  (4) Since the parameter k is very important for masking operation, in lines 116 -117, the authors point out how to select k for images (MNIST dataset); however, the selection of k for language and time series is not clear. (5) Other minor problems, e.g., the sequence number should be added for other formulas other than the KL-divergence (Eq (1)),  line number for Algorithm 1. 

-----New comments after reading authors' response

I appreciate the response given by the authors, and most of my concerns have been successfully addressed by the responses. I think this work is  novel compared to other existing works on "interpretable training" in that this method can be used in more broad areas, NLP, CV, and time series prediction tasks, which is achieved via masking gradients without using attention maps. As such, I raise my rating to 6 to reflect my new assessment of the updated manuscript.



limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper addresses the noise in saliency map techniques, which seek to explain what part of the input is responsible for a model's output decision, by introducing a novel training paradigm in which noise is added to the least important input features and the model is encouraged to produce similar outputs for original and noise-augmented inputs. This paper shows extensive experimental results in a variety of ML domains, such as image, language, and time series data. Additionally, the paper adds interesting commentary on input feature replacement paradigms such as ROAR and OOD as well as analysis that shows how their interpretable training paradigm reduces the vanishing saliency problem in RNNs.
main_review: This paper is well motivated -- a number of saliency methods are affected by noisy gradients, often mitigating this effect with solutions that aren't well motivated (e.g., changing backprop rules). The paper is extensive, thoughtful, and thorough in its experimentation and analysis, covering a wide range of ML domains and showing convincingly that their paradigm works across different settings, for instance, by reducing the noise in saliency maps, widening the separation between uninformative and informative input features as highlighted by saliency methods, improving performance of saliency map methods when evaluated on the model accuracy drop benchmark. In particular, its analysis on feature replacement paradigms (section 5.1, under "model accuracy drop") a useful insight into how to consider methods like ROAR and OOD. Additionally, its experiment showing that its interpretable training paradigm mitigates the vanishing saliency of recurrent neural networks is an exciting result. Both the insights presented in this paper as well as the novel training paradigm introduced can be used and built upon in future works.

The rest of this review contains large and small suggestions for the authors to consider.

Large:
* Could this paradigm be adapted and used as a fine-tuning procedure? Would we see the same benefits (for less training time)?
* How does this training paradigm impact performance (e.g., accuracy)? Accuracy numbers provided for language but not other domains
* Figure 2A -- distribution plots are hard to interpret -- use the box plot versions for MNIST (as shown in supp mat)
* Table 1 -- include additional results when using other saliency methods as well as LSTM Interpretable + other saliency methods

Small: 
* Additional related works from weak localization literature that use input-level perturbations during training: Wei et al., CVPR 2017 (Object Region Mining with Adversarial Erasure), Singh & Lee, ICCV 2017 (Hide-and-Seek), DeVries 2017 (Cutout).
* Line 119: Expand a bit more on what additional computational cost is needed (does it take the same # of epochs to converge; what's the added memory cost?)
* Explain more clearly why we should expect feature replacement methods to match IID replacement
* Line 114: Be more explicit that only input features are being masked (e.g., not intermediate features), be more clear in main paper about what masked features are being replaced by (e.g., copy parts from supp mat)
* Line 132: Spell out which VGG network is used (e.g., VGG-16?)
* Line 186: \tilde{X} is already used on line 103 - use different notation for clarity
* Line 186: Explain what R_i (predicted rationales) are in this context
* Figure 5 -- in caption, explain what the abbreviations for saliency methods mean (Grad, DL, GS, DLS - last few are a bit unclear)

limitations_and_societal_impact: The authors briefly mention some basic limitations of their method (computational cost, hyperparameter tuning, requiring training); however, they can improve their limitations section by fleshing out in more detail the additional computational cost. Furthermore, they did not discuss potential negative impacts of their work. Some directions for discussing potential negative impacts include misusing saliency methods, lack of consensus on how to evaluate saliency methods, etc.
review_title: null

Meta-review: The paper proposes a new procedure for training neural networks that achieves improved interpretability by masking the bottom k input gradients, helping address the noise in saliency map techniques. The reviewers found the paper well motivated, liked the experiments (especially the fact that they’re conducted on many domains), and found the method novel. 

The reviewers appreciated the thoroughness of the rebuttal, both in terms of experiments and responses, clarifying the points around related works, writing and additional discussions.  I therefore recommend acceptance.   I encourage the authors to integrate all the suggestions from the reviewers into the camera ready.

Decision: Accept (Poster)