Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper investigates feature important methods in NLP. The authors argue that counterfactual inputs should be in-distribution, and propose training on counterfactual inputs to reduce this distribution shift. They also investigate the best approach to producing counterfactuals and the best search method for finding FI explanations, finding that one of their proposed methods does best. 
main_review: This paper addresses an important problem and has the potential to be a strong paper. It is somewhat original and includes many experiments. However, overall it feels unpolished and does not yet seem ready for publication.

The paper claims to make a new argument for why OOD counterfactuals are bad; namely, that they are socially misaligned. I strongly agree with the conclusion that OOD counterfactuals can be misleading for interpretability, as other authors have pointed out before. However, I don't feel like the argument that they're socially misaligned really adds anything. 

The authors then propose training on counterfactual inputs to reduce distribution shift. This basic idea seems natural and reasonable to me. However, there are a few cases where they make claims about what approach to producing counterfactual inputs is "best" by using accuracy as a proxy for how in-distribution examples are. I think this is kind of misleading; accuracy is affected both by how OOD the data is and by how much actual information is lost (e.g. you could train on counterfactuals where information is removed, and this would then be not OOD but still have lower accuracy). 

Moreover, I think it's unclear how I should think about explanations from counterfactually-trained models. It seems plausible but not obvious to me that I should now basically trust FI explanations from counterfactually-trained models (whereas FI explanations from non-CT models are misleading because of the OOD gap). To the extent that you're proposing a new approach to explaining models, you should explain to me in what sense it's an explanation, why I should trust it, etc. 

Relatedly, the authors use metrics of sufficiency and comprehensiveness, which seem to capture meaningful aspects of how good an explanation is. However, I'm not convinced they capture everything we care about. There are also some mysterious results, such as how some "strong" baseline methods do terribly according to these metrics. This makes me suspicious that either these baselines are not strong, or that these metrics are unreliable. 

In general, I think some of the results are potentially interesting, but in many cases I'm not sure how much to trust them or how I should best interpret them. I think a clearer exposition would go a long way toward these problems, which I think make this work not yet publishable.

---------------
Update: I've read the rebuttal. While it clarifies some minor points, overall I still don't think this work is ready for publication, so I'll choose to keep my score.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper considers different aspects of feature-importance-based explanations. In particular, the authors propose a ‘counterfactual training’ approach to deal with observations that are out of the original training distribution (OOD smaples). Moreover, the authors illustrate different approaches to replace input features with a pre-defined baseline, i.e., a common approach to assess the quality of an attribution/explanation method, which however may lead to OOD samples. Finally, the authors propose search algorithms to find binary-valued, feature-based explanations that have been shown to improve relevant test measures.
main_review: The paper discusses interesting aspects of feature-attribution-based explainations and introduces a novel approach to deal with the out-of-distribution problem of ablated observations. The use of search methods as an alternative to local (linear) approximations for the identification of important features seems interesting.

Unfortunately, there are several concerns regarding the overall quality and contribution of the paper, which I outline below:
The main contributions of this paper (line 72) do not form a consistent story line. While the first two contributions consider the evaluation of local attribution methods, the third contribution (search methods for local explainability) is somewhat isolated from the rest of the paper.
The term ‘counterfactual’, as used in this paper, does not adhere to common definitions in the literature. The authors speak of a counterfactual as an input, ‘which has its important features replaced with a user-specified (uninformative) feature’ (line 45). In general, however, a counterfactual explanation is considered the smallest change made to an input vector to influence its prediction. The authors might want to talk of ‘ablated inputs’ (or similar) instead.
In the related work section, the authors should also consider the recent paper of Haug et al. [1], when talking about the effect of different replace (baseline) methods.
The specified main objective (line 136) is to generate explanations that provide good sufficiency/comprehensiveness statistics in the evaluation. However, existing work on local attributions already specifies much stricter and more meaningful properties (e.g. Lundberg et al. [2]). The authors should argue, why their objective should be preferred.
The authors acknowledge the (negative) effect of out-of-distribution samples in the evaluation of local explanations. However, rather than improving the explanation or evaluation process, the authors manipulate the model training to obtain ‘expected behaviour’ (line 162). This is a highly questionable approach, as the very purpose of an explanation technique is to communicate what a given (!) black-box model does/has learned. In fact, the authors' approach to bring ablated inputs in line with the training distribution, is to alter the original training distribution (and therefore the trained black-box model). In this context, the proposed ‘counterfactual training’ is an approach to make predictive models more robust, but, in my opinion, it is not a meaningful technique to improve the quality of explanations.
The description of the ‘Replace’ functions (lines 221-230) is insufficient. For example, it remains unclear what a ‘MASK token’ is. Besides, the different ‘Replace’ functions are compared with respect to random explanations. The differences observed in Figure 2 might be attributed to this random sampling. The authors should therefore discuss, why random explanations are to be preferred here over real explanations (e.g. LIME).
In the experimental description, the authors describe that they ignore negative attributions when selecting the top $k$ features (line 262). However, (strongly) negative attributions can indicate important features as well (see SHAP and LIME, for example). The authors might want to reconsider this approach.
The advantage of the proposed search methods for local attribution finding remains unclear. Other than empirically improved sufficiency and comprehensiveness scores, the authors fail to provide any sensible properties or strong guarantees on the generated attributions. In fact, the search methods only yield 0/1-indications of important features and thus contain much less information than existing attribution methods (which usually generate scalar weights and thus also allow a ranking of features).
Finally, in the discussion of the experiments (line 357), the authors find that the sufficiency/comprehensiveness scores are often much worse when using the proposed counterfactual training. This contradicts the earlier recommendation to use counterfactual training ‘whenever one intends to create feature-importance-based explanations’ (line 253).

[1] Haug, Johannes, et al. "On Baselines for Local Feature Attributions." arXiv preprint arXiv:2101.00905 (2021).
[2] Lundberg, Scott M., and Su-In Lee. "A unified approach to interpreting model predictions." Proceedings of the 31st international conference on neural information processing systems. 2017.


limitations_and_societal_impact: The authors have addressed some general limitations of local explanation methods. However, there are several more specific limitations and problems that the authors have not addressed yet (see detailed comments).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Summary

Explanations methods based on feature importance (FI) assign an importance score to each input feature. This is often evaluated by (1) removing the k most important features and replacing it with a feature that is in some sense defined as uninformative (also referred to by “Replace” function in the paper), (2) measuring how much the confidence in the original prediction drops (= sufficiency metric), (3) a good explanation is determined by still having a high confidence in the original prediction.

The paper argues that this approach is problematic because the explanations, were some features are replaced with uninformative features, represent a counterfactual out-of-distribution (OOD) example that the model is not equipped to handle correctly. As a consequence, the paper argues, the “best” explanation, as determined by the above approach, does not necessarily provide the information the user expects, making this explanation “socially misaligned”.

The paper studies the problem and explores alternative options, which are tested on six text classification tasks.

Contributions

-	Arguments on why the current sufficiency-based evaluation of FI explanation methods is problematic
-	Investigation of which “uninformative” features are best for avoid the outlined problem (see second paragraph above)
-	Novel search-based methods for finding FI explanations

main_review: Strengths

-	Interesting discussion on how OOD explanations do not reflect what a user expects and how this is an issue. Considerations of how explanations actually affect users is important in order for the research field of explainable AI to head into a direction where explanations truly reflect what is needed in applications by users.
-	Systematic comparison of various FI explanation methods and different choices for selecting uninformative features. This gives interesting insights and is particularly value for anyone who wants to apply FI explanation methods in practice.
-	A new search methods that finds the best explanation iteratively (with regards to sufficiency metric), rather than taking the top k highest as features
-	Experimental results are thorough and follow good practice (e.g. 10 random seeds, error bars)

Weaknesses

-	Some questions remain open but can easily be addressed.
-	It is not clear that the insights drawn will really generate explanations that are truly more socially aligned. Of course, answering this question is in itself tricky and the paper acknowledges this in the broader impact statement 

After Authors' Response
Thank you for answering my questions! I like the idea of this paper, although I do agree with some of the concerns raised by other reviewers, particularly that the story line is in parts not clear to follow. Thus, I will keep my score unchanged.

Questions

-	Line 152: Why do you say a MASK token is OOD, i.e. not seen in training? At least in pre-training of a masked LM, the MASK token is part of training.
-	Paragraph starting at line 163: Yes, model prior and random seed affect an explanation, but they should, correct? A different seed might really lead to a different model and therefore a different explanation? [see also footnote 1 below] My guess is that the main difference is: models with different random seeds also handle the same OOD counterfactual differently and how different models react to OOD counterfactuals remains intransparent, leading to the social misalignment? Do you agree? Could the argument be made clearer?

[footnote 1] This also relates to what Jacovi&Goldbert term faithfulness, in that context for an explanation to be faithful, a model with a different random seed and a different prediction also might have a different explanation and it needs to have a different explanation because it is indeed a different model, this is needed for the explanations to be faithful.

-	Line 224, “Marginalize”: I’m not sure I understood this paragraph correctly, is the following correct: For the input feature to be removed, you use BERT to replace the token at the to-be-removed feature position? E.g. in “A cat is sat on a mat” and “cat” is to be replaced, you ask BERT what other word to put there and it e.g. give you “dog”?
-	Line 267, Vanilla Gradients: Which replace function do you use there?
-	LIME vs Integrated Gradients: Why does one use the replace function attention mask and the other mask token? 
-	Table 1: The difference between e.g. LIME and random search is that LIME uses the top-k features whereas random sample k at random? Sorry if I missed it, but what is the value for k? How did you choose it?

Typos, representation, minor questions etc.

-	115: explanation -> explanations
-	It is strange that the sufficiency metric is introduced twice
-	It took me a few reads of the beginning of section 5 to understand what exactly the setup is and why, maybe this could be rephrased some? For example, “where the set of input features to be removed is fixed” -> explicitly state here why this is important and needed
-	Line 290: delete “produce”
-	Line 294: delete “with”
-	Line 295: what is the regularizer here?
-	Line 334: It would be great if you could list the budget (forward/backward passes) required by the different methods that you test.
-	Table 1: Bold in last section, second last column is missing; same for last column of “Evidence Inference”


limitations_and_societal_impact: -	The paper provides a good overview of the potential risks of FI explanation methods.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper addresses a core limitation of existing feature importance methods, where importance scores are generated by comparing original examples to unrealistic counterfactuals. To solve this issue, the authors propose a solution where counterfactuals used to compute feature importance are generated by manipulating the attention mask and not by editing the actual text of the original examples. They also propose training models on the generated counterfactuals and show that it improves model robustness. Finally, they introduce a novel method producing model explanations and show that it outperforms strong baselines.

main_review: The paper contains some interesting findings and proposes a novel search-based method which improves on existing feature-importance solutions. It is also clearly written and easy to follow through.

However, in its current state, the contributions presented in the paper are not substantial enough in my opinion. First, the realization that unrealistic counterfactuals harm our ability to estimate the effect of variables is a pillar of causal research, and there are existing methods in ML and in NLP specifically which deal with this problem in the context of causal explanations (for example, see CausaLM, INLP, CaCE and their follow-up work).

Second, the use of counterfactuals for training can be misleading. While it can improve model performance (as demonstrated in the counterfactually augmented data literature), measuring model performance on in-domain data and calling it more robust is problematic. Alternatively, counterfactually trained models should be tested on types of counterfactuals which were not seen in training. Also, counterfactual training can harm our ability to identify model weaknesses, which is a main motivation for generating model explanations in the first place. 

The two main arguments raised in this paper, that the OOD-ness of counterfactuals leads to poor explanations and that training on OOD data improves robustness, are inherently causal arguments which I think already exist in the literature. While the results in the paper are surely interesting, I would like to see the authors address the existing literature on the subject and try to explain the similarities and differences in the theoretical approach. While I am pointing to one prism of thought that might be useful for analyzing the paper’s findings, there might be others. To improve the paper, I would like to see the authors address similar arguments made in the literature.

limitations_and_societal_impact: This paper can potentially have a substantial positive societal impact, as understanding what features drive model decisions can uncover hidden biases and fairness issues which govern the model’s behavior.
review_title: null

Meta-review: The reviews for this paper were mostly borderline, with one review strong negative. While the reviewers thought that the paper addresses an important problem in a novel way, and provides interesting insights, the author response did not lead them to change their scores. 

Despite this lukewarm assessment, I believe the paper makes contributions that likely merit acceptance: (1) Tackling an important problem in generating and evaluating explanations in NLP and ML more broadly (the OODness of feature importance attribution methods that are based on masking, ablation, etc., since they produce counterfactual/OOD examples); (2) providing a clear approach to mitigate this problem (adding counterfactuals at training time), which can work well (little drop compared to standard models); (3) evaluating various methods to create explanations and finding that some work well in the new counterfactually trained models (achieve good sufficiency/comprehensiveness scores, which are standard). 

Given these contributions, and considering that no paper is perfect, I recommend acceptance. Since my recommendation goes somewhat against the general sense of the reviews, I provide here a detailed discussion of the main issues raised by the reviewers, how the authors responses, and my opinion of the issue. In any case, I urge the authors to take these issues into account in their next revision. 

**Comments by Reviewer NMtf**
- argument on social alignment doesn't add much. Authors: it's another perspective on why FI OODness is problematic, and may help. AC: natural application of the social alignment argument from Jacovi and Goldberg to another class of explanations.
- Using accuracy to assess distribution shift is misleading, can conflate OODness and information removal. Authors: remove the same information in all conditions, difference is due to OODness. AC: agree that accuracy change between iid and ood data is a reasonable metric of OODness. 
- Not convinced about sufficiency and comprehensiveness metrics. Authors: These are standard in the literature. AC: agree, these are standard faithfulness metrics. What's missing here is an evaluation of plausibility, for example by comparing with human-annotated explanations (highlights, rationales). 
- Baseliens do terrible. Authors: LIME does well, IG is likely a poor method. AC: It's tricky to compare when the benchmark is not common. It could have been better to use standard datasets from explanations in NLP, where prior results are available. But, LIME results are indeed quite good. 

**Reviewer Lhz2**
- The contributions do not form a consistent story line. Authors disagree and explain that the part on local attributions depends on fixing the OOD problem of explanations. AC: agree, the first part is a nice contribution in itself and it makes the rest possible. 
- Counterfactual is an incorrect term here, since a counterfactual explanation means something different. Authors:  did not use counterfactual explanation. AC: best to clarify that counterfactual input is not to be confused with the technical term of counterfactual explanation. 
- questions the sufficiency/comprehensiveness metrics and prefers more meaningful metrics like SHAP. Here I agree with the authors that it is useful to apply these empirical metrics, which are standard in the literature, even if they do not have some nice theoretical properties like SHAP.
- Changing the training distribution is questionable. Authors: to use FI explanations, have to align training and test distributions, so if we want to keep test time metrics, must change training distribution. AC: indeed, I was also concerned by changing the training distribution and hence the model, but was pleased with the minor drop in accuracy, at least for some of the methods. However, it's possible that a model trained on the original distribution arrives at decisions in a different way than a model trained on the counterfactually-modified distribution, but they have similar performance. This is a hard problem to get around, but one that should be discussed. 
- Search methods provide binary importance while existing methods provide scalar scores. Authors argue that their method still brings useful rankings (15 out of 24 vs LINE) and that for selecting features it works best. AC: not convinced by the author response, as 15/24 seems weak, and also as scalar weights can indeed be useful, both for interpretation (users might prefer them) and for downstream tasks (training with explanations, etc.). 
- Sufficiency and comprehensiveness are often worse when using the proposed counterfactual training. Authors explain they recommend using CT regardless of these metrics, because using CT guards again OODness. AC: while I agree with the idea in principle, I'm unconvinced by the argument. If using a standard model leads to better explanations according to the metrics in question, then the argument for using CT grows much weaker. 

**Reviewer 2JSj**
- Had some questions that seemed to have been answered.
- Not clear that the explanations will truly be more socially aligned. AC: did not see a clear answer to this. The answer to the issue of social alignment and how different models react to OOD counterfactuals is part of the story, but probably not all. 

**Reviewer abDP**
- there are existing method to deal with the problem of estimating effects in the case of counterfactuals (mentioning specific studies: CausaLM, INLP, CaCE). Authors: don't see how these particular studies lessen the contribution, since they don't discuss how OODness affects social alignment. Also say that the mentioned methods are not meant to use with individual data points. AC: agree. Those should probably be discussed, but are not at the core of the current work.
- Counterfactual training can be misleading in terms of evaluation. Authors argue that in their case it makes sense, because the CT models work better on OOD data, and they're not interested in different kinds of OODness. AC: agree, this actually seems reasonable for the present use case of evaluating explanations, if not for robustness more broadly. The authors should clarify their goals in this evaluation and distinguish from general model robustness. 


Decision: Accept (Poster)