The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
Authors: Peter Hase
Harry Xie
Mohit Bansal
Venue: NeurIPS 2021 Poster (2021)
Keywords: feature importance
interpretability
explainability
social alignment
search methods
feature attribution
salience methods

Abstract: Feature importance (FI) estimates are a popular form of explanation, and they are commonly created and evaluated by computing the change in model confidence caused by removing certain input features at test time. For example, in the standard Sufficiency metric, only the top-k most important tokens are kept. In this paper, we study several under-explored dimensions of FI explanations, providing conceptual and empirical improvements for this form of explanation. First, we advance a new argument for why it can be problematic to remove features from an input when creating or evaluating explanations: the fact that these counterfactual inputs are out-of-distribution (OOD) to models implies that the resulting explanations are socially misaligned. The crux of the problem is that the model prior and random weight initialization influence the explanations (and explanation metrics) in unintended ways. To resolve this issue, we propose a simple alteration to the model training process, which results in more socially aligned explanations and metrics. Second, we compare among five approaches for removing features from model inputs. We find that some methods produce more OOD counterfactuals than others, and we make recommendations for selecting a feature-replacement function. Finally, we introduce four search-based methods for identifying FI explanations and compare them to strong baselines, including LIME, Anchors, and Integrated Gradients. Through experiments with six diverse text classification datasets, we find that the only method that consistently outperforms random search is a Parallel Local Search (PLS) that we introduce. Improvements over the second best method are as large as 5.4 points for Sufficiency and 17 points for Comprehensiveness.

The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
Peter Hase, Harry Xie, and Mohit Bansal
Department of Computer Science University of North Carolina at Chapel Hill {peter, fengyu. mbansal)@cs. unc. edu
Abstract
Feature importance (FI) estimates are popular form of explanation, and they are commonly created and evaluated by computing the change in model confidence caused by removing certain input features at test time. For example, in the standard Sufficiency metric, only the top-k most important tokens are kept. In this paper, we study several under-explored dimensions of Fl explanations, providing conceptual and empirical improvements for this form of explanation. First, we advance a new argument for why it can be problematic to remove features from input when creating or evaluating explanations: the fact that these counterfactual inputs are out-of-distribution (OOD) to models implies that the resulting explanations are socially misaligned. The crux of the problem that the model prior and random weight initialization influence the explanations (and explanation metrics) in unintended ways. To resolve this issue, we propose a simple alteration to the model training process, which results in more socially aligned explanations and metrics. Second, we compare among five approaches for removing features from model inputs. We find that some methods produce others, and we make recommendations for selecting a feature-replacemmnt function. Finally, introduce four search-based methods for identifying FI explanations and compare them to strong baselines, including LIME, Anchors, and Integrated Gradients. Through experiments with six diverse text classification datasets, we find that the only method that consistently outperforms random search is Parallel Local Search (PLS) that we introduce. Improvements over the second best method are as large as 5.4 points for Sufficiency and 17 points for Comprehensiveness.
OOD counterfactuals than
Introduction
Estimating feature importance (FI) is a common approach to explaining how learned models make predictions for individual data points [51, 46, 34, 57, 36, 11]. Fl methods assign scalar to each feature of an input representing its "importance" to the model's output, where feature may be an individual component of an input (such as a pixel or a word) or some combination of components. Alongside these methods, many approaches have been proposed for evaluating FI estimates (also known as attributions) [41, 1, 13, 22, 20, 67]. Many of these approaches use test-time input ablations, where features marked as important are removed from the input, with the expectation that the model's confidence its original prediction will decline if the selected features were truly important.
All supporting code for experiments in this paper is publicly available https:/ httg/gttthbbccoo peterbbaase/Exxlaaaattoneeet
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

For instance, according to the Suffiiincyy metric [13], the best Fl explanation is the set of features which, if kept, would result in the highest model confidence in its original prediction. Typically the top-k features would be selected according to their Fl estimates, for some specified sparsity level k. Hence, the final explanation is a k-sparse binary vector {0, 134, where d the dimensionality of the chosen feature space. For an explanation e and model f that outputs a distribution over classes p(ylz) = f(x), Sufficiency can be given as:
Suff(f, - f(x), - (Replace(z, e)ls
where أ = arg max, f((x)y is the predicted class for x and the Replace function replaces features in x with some uninformative feature at locations corresponding to Os in the explanation e.
The Replace function plays key role in the definition of such metrics because it defines the counterfactual input that we are comparing the original imput with. Though FI explanations are often presented without mention of counterfactuals, all explanations make use of counterfactual situations [38], and Fl explanations are no exception. The only way we can understand what makes some features "important" to particular model prediction is by reference to counterfactual input which has its important features replaced with a user- specisee (uninformative) feature.
In this paper, we study several under-explored dimensions of the problem of finding good explanations according to test-time ablation metries including Sufficiency and related metric, Comprehensiveness, with focus on natural language processing tasks. We describe three primary contributions below.
First, we argue that standard FI explanations are heavily influenced by the out-of-distribution (OOD) nature of counterfactual model inputs, which results in socially misaligned explanations. We use this term, first introduced by Jacovi and Goldberg [24], to describe situation where an explanation communicates a different kind of information than the kind that people expect it to communicate. Here, we do not expect the model prior random weight initialization influence FI estimates. This is a problem insofar as Fl explanations are not telling us what we think they are telling us. We propose training algorithm to resolve the social misalignment, which is to expose models to counterfactual inputs during training. so that counterfactuals are not out-of-distribution at test time.
Second, we systematically compare Replace functions, since this function plays an important role in evaluating explanations. To do so, we remove tokens from inputs using several Replace functions, then measure how OOD these ablated inputs are to the model. We compare methods that remove tokens entirely from sequences of text [41, 13), replace token embeddings with the zero embedding or special token [34, 57, 1, 65, 57], marginalize predictions over possible counterfactuals [68, 30, 63], and edit the input attention mask rather than the input text. Following our argument regarding the OOD problem (Sec. 4), we recommend the use of some Replace functions over others.
Third, we provide several novel search-based methods for identifying Fl explanations. While finding the optimal solution to arg max, Suff[ e) is natural example of binary optimization, problem for which search algorithms are a common solution [43, 49, 31, we are aware of only a few prior works that search for good explanations [19, 47. 15]. We introduce our novel search algorithms for finding good explanations by making use of general search principles [43]. Based on experiments with two Transformer models and six text classification datasets (including FEVER, SNLI, and others), we summarize our core findings as follows:
1. We propose to train models on explanation counterfactuals and find that this leads to greater model robustness against counterfactuals and yields drastic differences in explanation metrics.
We find that some Replace functions are better than others at reducing counterfactual OOD-ness, although ultimately our solution to the OOD problem is much more effective.
3. We introduce four novel search-based methods for identifying explanations. Out of all the methods we consider (including popular existing methods), the only one that consistently outperforms random search is the Parallel Local Search (PI.) that we introduce, often by large margins of up to 20.8 points. Importantly, we control for the compute budget used by each method.
2 Related Work
Feature Importance Methods. A great number of methods have been introduced for Fl estimation,
drawing upon local approximation models [46, 47, 31], attention weights [25, 61, 66), model gradients [5L, 50, 57, 52], and model-based feature selection [4, 2, 62, 42, 8. 11]. While search approaches are

---

regularly used to solve combinatorial optimization problems in machine learning [49, 5, 16, 40],
we know of only few FI methods based on search [19. 47. 15]. Fong and Vedaldi [19] perform gradient descent in explanation space, while Ribeiro et al. [47] search for probably-sufficiee
subsets concurrent work, Du and Xu [15] propose a genetic search algorithm for identifying Fl explanations. We introduce several novel search algorithms for finding good explanations, including (1) a gradient search similar to Fong and Vedaldi [19], a (2) local heuristic search inspired by adversarial attack method [16], (3) global heuristic search, and (4) a parallel local search (PLS) making use of general search principles [43].
of the input (under a perturbation distribution).
Choice of Replace Function. Past evaluations of explanation methods typically remove tokens or words from the text entirely [41, 13] or replace them with fixed feature values [22. 64]. Methods for creating explanations also use several distinct approaches, including (1) replacing token embeddings
with the zero vector [34, 57, 1J. (2) using a special token [65, 57], (3) marginalizing predictions
over a random variable representing an unobserved token [68, 30, 63, 27], and (4) adversarially selecting counterfactual features [23]. Sturmfels et al. [55] carry out a case study involving a few Replace functions for vision model, which they compare via test-time ablations with image blurring techniques, though the case study is not intended to be a full comparison of methods. Haug et al. [21] assess number of Replace functions for explanation methods used with tabular data, but they compare between functions to use when generating explanations, rather than when evaluating explanations, for which they offer no recommendation. In addition to evaluating Replace functions from the above works, we also consider setting attention mask values for individual tokens to O.
The Out-of-distribution Problem of Explanations. Many papers have expressed concerns over how removing features from an input may result in counterfactuals that are out-o--distribution to a trained model [65, 57, 19, 7, 22, 30, 23, 63, 56, 27, 45, 21, 48, 28, 59]. In response to the problem, some have proposed to marginalize model predictions over possible counterfactual inputs 168, 30, 63, 27], use counterfactuals close to real inputs [7, 48), weight their importance by their closeness to real inputs [45], or to adversarially select counterfactual features rather than use any user-specified features [23]. Others reject the whole notion of test-time ablations, preferring metrics based on train-time ablations [22]. Jethani et al. [28] propose a specialized model for evaluating explanations that is trained counterfactual inputs in order to make them in-distribution, but since the evaluation model is distinct from the model used to solve the task, explanation metrics calculated using this evaluation model may not reflect the faithfulness of explanations to the task model. In concurrent work, Vafa et al. [59] independently adopt a solution equivalent to our Counterfactual Training with Attention Mask Replace function, approach which we empirically justify in Sec. 5. In general, prior works make arguments for their approach based on intuition or basic machine learning principles, such as avoiding distribution shift. In Sec. 4, we give a more detailed argument for preferring in-distribution counterfactuals on the basis of social alignment, a concept introduced by Jacovi and Goldberg [24], and we propose a solution to the OOD problem. Our solution allows for rest-time evaluation of explanations of particular model's 's decisions for individual data points, unlike similar proposals which either evaluate large sets of explanations all at once [22] or use a separate model trained specifically for evaluation rather than the blackbox model [28].
3 Problem Statement
Feature Importance Metrics. The problem we are investigating is to find good feature importance explanations for single data points, where explanation are evaluated under metrics using test-time ablations of the input. In this context, an explanation for an input in a d dimensional feature space is a binary vector E {0, 1]d, which may be derived from discretizing an FI estimate E Rd We consider two primary metrics, Sufficiency and Comprehensiveness [13]. Sufficiency measures whether explanations identify a subset of features which, when kept, lead the model to remain confident in its original prediction for a data point. Comprehensiveness. meanwhile, measures whether an explanation identifies all of the features that contribute model's 's confidence in its prediction, such that removing these features from the input lowers the model's confidence.
The Sufficiency metric for an explanation E {0, 1]d and model p(y|>) = fe():) is given as Zu, e, s) = fo(x) fe (Beplace, (x,)))
(1)
where ý arg max, y [x) is the predicted class for x, and the Replace, function retains a pro- portion of the input features (indicated by e) while replacing the remaining features with some
3

---

user seee---eeiied feature. In order to control for the explanation sparsity, i.e. the proportion s of tokens in the input that may be retained, we average Sufficiency scores across sparsity levels in (.05.. 20, .50), meaning between 5% and 50% of tokens in the input are retained [13]. A lower Sufficiency value is better, as it indicates that more of the model's confidence is explained by just the retained features (increasing fe (Replacc, e))g)-
Similarly, Comprehensiveness is given as 2,, = with sparsity values in [.95, 900 .80, .50). as we are looking to remove features that are important to the prediction (while keeping most features). A higher Comprehensiveness value is better, as indicates that the explanation selects more of the evidence that contributes to the model's 's confidence in its prediction (resulting in a lower fo (Reelacee(; e))
e,,,) ( fo (Replace, (z)) but
Overall Objective. Finally, our overall Sufficiency and Comprehensiveness objectives are given by averaging Suff (or Comp) scores across several sparsity levels. With a model p(y) (z) [xx), single data point x with d features, and a set of sparsity levels S, the Sufficiency objective is optimized by
obtaining set E {ei)/11 with one explanation per sparsity level
|8| S
1
2.ffff.z.ei,.. s.t. e, (0, 1]d and Σ c S ceiling(s; d)
arg max F
where the ceiling function rounds up the number s; d of tokens to keep. When optimizing for Comprehensiveness, we use the Comp and arg min functions, and the inequality is flipped. In general, we will optimize this objective using limited compute budget, further described in Sec. 6.2.
4 The Out-of-Distribution Problem in Explanations
In this section, we first give a full argument for why it is problematic for explanations to be created or evaluated using OOD counterfactuals. Then, we propose a solution to the OOD problem. We rely on this argument in our comparison of Replace functions in Sec. 5. We also assess our proposed solution to the OOD problem in Sec. and later make use of this solution in Sec. 6.
The OOD problem for explanations occurs when a counterfactual input used to create or evaluate an explanation is out-of-distribution (OOD) to model. Here, we take OOD to mean the input is not drawn from the data distribution used in training (or for finetuning, when a model is finetuned) [39]. In general, counterfactual data points used to produce FI explanations will be OOD because they contain feature values not seen in training, like a MASK token for a language model. A long line of past work raises concerns with this fact [65, 57, 19, 7, 22, 30, 23, 63, 56, 27, 45, 21, 48, 28, 59]. The most concrete argument on the topic originates from Hooker et al. [22], who observe that using OOD counterfactuals makes it difficult to determine whether model performance degradation is caused by "distribution shift" or by the removal of information. It is true that, for given counterfactual, a model might have produced different prediction if that counterfactual was in-distribution rather than cut-of-distributie there is no ambiguity about what causes a drop model confidence when replacing features: the features in the input were replaced, and this changes that model's, 's prediction. If the counterfactual was in-distribution, we would be talking about different model, with a different training distribution. Hence, we believe we need a stronger argument for why we should not use OOD counterfactuals when explaining a trained model's 's behavior.
But this is a question we cannot ask about single, trained model, where
Our principal claim is that feature importance explanations for a standardly trained neural model socially misaligned, which is undesirable. Jacovi and Goldberg [24] originally introduce this term as they describe shortcomings of explanations from a pipeline (select-predict) model, which is kind of classifier designed to be interpretable. Explanations are socially misaligned when people expect them to communicate one kind of information, and instead they communicate a different kind of information. For example, if we expected an explanation to be the information that a model relied on in order to reach a decision, but the explanation was actually information selected after a decision was already made, then we would say that the explanations are socially misaligned. Our argument now proceeds in two steps: first, we outline what the social expectations are for feature importance explanations, and then we argue that the social expectations are violated due to the fact that counterfactuals are OOD.

---

We suggest that, for a particular trained model and particular data point, people expect a feature importance explanation to reflect how the model has learned interpret features as evidence for or against a particular decision." This social expectation is upheld if FI explana- tions are influenced only by the interaction of an untrained model with a training algorithm and data. But expectations are violated to the ex- tent that FI explanations are influenced by factors such as the choice of model prior and random seed (which we do not intend to influence Fl explanations). We depict these possible upstream causes of individual Fl explanations in Fig. 1. In fact, the model prior and random seed are influential to Fl explanations when the counterfactuals employed in these explanations are OOD the model. A simple example clearly illustrates the potential influence of model priors: Suppose one trained a BERT model to classify the sentiment of individual words using training data from sentiment dictionary, then obtained feature importance explanations with the MASK token Replace function. In this situation, model predictions on counterfactual data are always equal to the prediction for a single MASK token, fe (MASK). So, by construction, the MASK token never appears in the training data, but FI explanations for this model make use of the quantity fo (MASK). Since a model could not have learned its prediction fe (MASK) from the data, this quantity will be largely determined by the model prior and other training hyperparameters. and therefore explanations based on this prediction are socially misaligned. Now, in general, we know that neural models are sensitive to random parameter initialization, data ordering (determined by the random seed) [14], and hyperparameters (including regularization coefficients) [9, 44, 60), even as evaluated on in-distribution data. For OOD data, then, a neural model will still be influenced by these factors, but the model has no data to learn from in this domain. As result, Fl explanations are socially misaligned to the extent that these unexpected factors influence the explanations (while the expected factors like data are not as influential). In other words, we do not expect explanations to influenced by random factors, the priors of the model designer, or uninterpretable hyperparameters, but we do expect them to be influenced by what the model learns from data.
Cless
Seed
Algerithim
Input
Explanation
Figure 1: Causal diagram of a feature importance explanation for a trained model and an input.
The argument applies equally explanation metrics. When metrics are computed using OOD counterfactuals, the scores are influenced by unexpected factors like the model prior and random seed, rather than the removal of features that a model has learned to rely on. As a result, the metrics are socially misaligned. They do not represent explanation quality in the way we expect them to.
The solution the OOD problem is align the train and test distributions, which we do by exposing the model to counterfactual inputs during training, a method we term Counterfactual Training. Since common explanation methods can require hundreds or thousands of model forward passes when explaining predictions [46, 57], explanations from these methods would be prohibitively expensive obtain during training. We therefore propose to train with random explanations that remove most of the input tokens, which provides a good objective in theory for models to learn the counterfactual distribution that will be seen at test time [28]. Specifically, we double the inputs in each training batch, adding Replace(1; e) version of each input (with the same label) according to a random explanation with sparsity randomly selected from (.05, .1, .2, .5). The (CT) models make in-distribution predictions for both observed
resulting Counterfactual--rainee and counterfactual data points. While we cannot guarantee that this approach fully eliminates the influence of the model prior and random seed on Fl explanations, the fact that explanations are influenced by what the model learns from data will resolve social misalignment to a great extent. We find that these models suffer only slight drops in test set accuracy, by 0.7 points on average across six datasets (see Table ?? in Supplement). But we observe that this approach greatly improves model robustness to counterfactual inputs, meaning the counterfactuals are now much more in-distribution to models (described further in Sec. 5). Similar to the goals of ROAR [22] and EVAL-X [28], our proposed solution also aims align the train and test-time distributions. However, our approach allows for test-time evaluation of individual explanations for a particular trained model, while ROAR only processes large of explanations all at once and EVAL-X introduces specialized model for evaluation, which may not reflect the faithfulness of explanations to the task model.
We mean "people" to refer to the typical person who has heard the standard description of these explanations, iec that they identify features that are "important" to model decision. Of course, there will be some diversity in how different populations interpret feature importance explanations [18].
5

---

Model Sensitivity Replace Function
Counteffccuull-rrannc Model Robustness
BEXT Booe
RoBERDe Banc
Countbifactel-TTui Trained
Standeed
0.0
-0.1
-0.3
-0.2
FINS
-0.1
Replace Function
-0.4
Attention Mask
Change
00.5
-02
Marginalize
in Acc.
0.0
-0.3
Slice Out
Misk Token
-0.1
Zero Embodding
-0.2
0.4
0.3
-0.4
-0.5
0.2
0.8
Removed
Tokens Removed
Figure 2: Model sensitivity to input ablations for several choices of Replace function and training algorithm. On the left we show the sensitivity of standardly trained models. On the right we show the effect of using Counterfactual- Trainne models.
Analysis of Counterfactual Input OOD-ness
Here, assess how aut-of-distributiio the counterfactual inputs given by Replace functions models, and we measure the effectiveness of Counterfactual Training. We do this before designing or evaluating explanation methods because, given our argument in Sec. 4, it is important to first identify which Replace ace function and training methods are most appropriate to use for these purposes.
Experiment Design. We compare between Replace functions according to how robust models are to test-time input ablations using each function, where the set of input features to be removed is fixed across the functions. We measure robustness by model accuracy, which serves proxy for how in in-distribution or outtof-distributtin the ablated inputs are. we observe differences in attribute
model accuracies between Replace functions for given level of feature sparsity, the change in the input OOD-ness to the Replace function itself. In the same manner, we compare between Counterfactual- Trained (CT) models and standardly trained models (termed Standard).
Specifically, we train 10 BERT- Base [12] and RoBERTa- [35] on two benchmark text classifica- tion datasets, SNLI [6] and SST-2 [54]. These are all Standard models, without the counterfactual training we propose. We use ten random seeds for training these models. Then, we evaluate how robust the models are to input ablations, where we remove proportion of random tokens using one of five Replace functions (i.e. we Replace according to a random explanation). We evaluate across proportions in {0.2, 0.5, 0.8). The five Replace functions we test are:
Attention Mask. We introduce this Replace function, which sets a Transformer's attention mask values for removed tokens to 0. meaning their hidden representations are never attended to. Marginalize. This method marginalizes model predictions over counterfactuals drawn from generative model (I) of the data, i.c. as arg max, In Lā=po(ālr..) Pe (y P. (z) (#|2, e), where Pa(ā]z, e) is the distribution over tokens to be replaced given by c.g. BERT [68, 30, 7, 63].
MASK Token. this method, we simply replace tokens with the MASK token.
4. Slice Out. This approach removes selected tokens from the input sequence itself, such that the resulting sequence has a lower number of tokens.
5. Zero Vector. Here, we set the token embeddings for removed tokens to the zero vector.
We train additional CT models for BERT-Base on SNLI. with ten random seeds per model, for all Replace functions except Marginalize, since this function is exceedingly expensive to use during Counterfactual Training. For further details, see the Supplement.
Results for Replace ace functions. We show the results of this experiment in Fig. 2, via boxplots of the drops in accuracy for each of the 10 models per condition. First, we describe differences in Replace functions for Standard models, then we diseuss the effect of Counterfactual Training. On the left in Fig. 2, we see that Standard models are much more sensitive to some Replace functions than others. The Attention Mask and Mask Token functions are the two best methods. The best these two methods outperforms the third best method by up to 1.61 points with BERT and SNLI (p .0005),'
---vul for two-sided difference in means tests are calculated by block bootstrap with all data points and model seeds being resampled 100k times [17].

---

5.48 points with RoBERTa and SNLI (p < 1e- 2.42 points with BERT and SST-2 (p - 0.0008), and 4.72 points with RoBERTa and SST-2 (p- < 1e-4). The other methods often far underperform the best method. For instance, with BERT on SST-2, Zero Embedding is up to 10.45 points worse than Mask Token (p < 1e-4), and with RoBERTa on SST-2, Slice Out underperforms Attention Mask by up to 4.72 points (p < le-4). Marginalize is regularly more than 10 points worse than the best method. Overall, we recommend that, when not using Counterfactual Training, researchers use either the Attention Mask or Mask Token Replace functions.
Counterfactual Training vs. Standard Training. On the right side of Fig. 2, we see the effect of Counterfactual Training on model robustness for several Replace functions. We find that coun- terfactual inputs are much less OOD for Counterfactual Trained models than for Standard models, regardless of the Replace function used. The improvement in robustness is up to 22.9 points. More- over, the difference between Replace functions is almost entirely erased, though we do observe a statistically significant difference between Attention Mask and Zero Embedding with 80% of tokens removed (by 2.23 points, p < 1e-4). Given these results, and following Sec. 4, we ultimately rec- ommend that researchers use Counterfactual Training with the Attention Mask, Mask Token, or Slice Out Replace function whenever they intend to create FI explanations.
6 Explanation Methods and Experiments
6.1 Explanation Methods
We describe explanation methods we consider below, with implementation details in the Supplement. Salience Methods. One family of approaches we consider assigns a scalar salience to each feature of an input. The key property of these scores is that they allow one rank-order features. We obtain binarized explanations through selecting the top-k features, or up to the top-k features when some scores are negative (suggesting they should not be kept). We list the methods below:*
1. LIME LIME estimates Fl by learning a linear model of a model's predicted probabilities with samples drawn from local region around input [46]. Though it is common to use the Slice Out Replace function with LIME, we use the Attention Mask deplace function (following Sec. 5), meaning we sample local attention masks rather than local input sequences.
Vanilla Gradients. We obtain model gradients w.r.t. the model input as salience scores, one early method for explanation [33]. We compute the gradient of the predicted probability w.r.t. input token embeddings, and we obtain a single value per token by summing along the embedding dimension.
3. Integrated Gradients. We evaluate the Integrated Gradients (IG) method of Sundararajan et al. [57]. This method numerically integrates the gradients of model output w.r.t. its input along a path between the observed input and a user-selected baseline. Given our results in Sec. 5, we use a repeated MASK token embedding for our baseline ă rather than the all-zero input suggested by Sundararajan et al. [57] for text models. We use the model's predicted probability as the output, and to obtain token-level salience scores, we sum the output of IG along the embedding dimension.
Search Methods. An alternative class of methods searches through the space of possible explana- tions. Search methods are regularly used solve combinatorial optimization problems in machine learning [49, 5, 3, 16, 40J. All search methods use the Attention Mask Replace function, and the search space is restricted to explanations of the maximum allowable sparsity (or minimum, with Comprehensiveness), except for Anchors which takes a maximum number of features as a parameter.
1. Random Search. For each maximum explanation sparsity is (or minimum, for Comprehensiveness) we randomly sample a set of k-sparse explanations, compute the current objective for each of them, and choose the best explanation under the objective.
Anchors. Ribeiro et al. [47] introduce a method for finding a feature subset that almost always yields the same model prediction as its original prediction for some data point, as measured among data points sampled from a distribution centered on the original data point. Explanations are also preferred to have high coverage, meaning the feature subset is likely to be contained in a local sample. The solution is identified via a Armed Bandit method combined with a beam search.
'In early experiments, we found that parametric model (similar to [4. 2, 42)) performed far worse than other salience methods, and hence we leave out parametric models from further consideration.
7

---

3. Exhaustive Search. Exhaustive search returns the optimal solution after checking the entire solution space. This is prohibitively expensive with large inputs, as there is combinatorial explosion in the number of possible explanations. However, we are able to exactly identify optimal explanations for short sequences, typically with 10 or fewer tokens.
4. Gradient Search. Fong and Vedaldi [19] propose to search through a continuous explanation space by gradient descent. We introduce a Gradient Search that uses discrete explanations, because continuous explanations do not reflect test-time conditions where discrete explanations must be used. For an input of length L. this method sequentially updates a continuous state vector s E R via gradient descent to minimize a regularized cross- eross-entropy loss between the model prediction on the input x and the model prediction for Replace(z, et), where e is discrete explanation sampled from a distribution p(e, (t) using the Gumbel-Soffmnn estimator for differentiability [37, 26]. The regularizer maintains sparsity of the solution. The final explanation is obtained from the last state 8t-
5. Taylor Search. Inspired by HotFlip [16], this method explores the solution space by forecasting the change in the objective using a first-order Taylor approximation. Specifically, this is local search method with a heuristic function computed as follows: We first calculate the gradient E R2 of the cross-entropy loss (same loss as Gradient Search) with respect the explanation es. Then we find the two indices and j as the solution to arg m g( glú)) The next state is obtained by
setting (1) and (i) O. This is a first-order approximation to the change in loss between the new and old state, based on the Taylor expansion of the loss [16]. Note when optimizing for Comprchensiveness,
the arg min. Following Ebrahimi et al. [16]. we ultimately search heuristic within a beam search, starting from a random explanation, with width to = 3.
this
6. Ordered Search. Next, we introduce global search algorithm, Ordered Search. This method searches through all explanations in an order given by a scoring function f : - RR We only require that f is linear in e, this allows for efficient ordering of the search space using priority queue [3]. The algorithm proceeds by first estimating parameters for fe, then searching through explanations in order of their score, 0 e. For the first stage, we obtain parameters for So from the per-token salience scores given by LIME, which is the best salience method evaluated in Sec. 6. In the second stage, we enumerate the search space in order of the score given by fe. We allocate 25% of the compute budget to the first stage and 75% to the second (measured in terms of forward passes).
7. Parallel Local Search (PLS). Lastly, we again consider the class of local search algorithms, which have a long history of success with constrained optimization problems [43, 3]. We propose a parallelized local search algorithm (PLS) tailored to the problem at hand. Given number r of parallel runs to perform, a proposal function Propose, and compute budget per run, an individual search proceeds as follows: (1) Sample random initial explanation ei and compute the objective function for that explanation. (2) For the remaining budget of b-1 steps: sample not-already-seen explanation e, according Propose and adopt e, the state only if the objective is lower at e, than at the current state. The Propose function samples not-already-seen explanations from neighboring states to the current state. We use 10 parallel runs following tuning results.
6.2 Experimental Setup
Data. We compare the above explanation methods six benchmark text classification datasets: SNLI [6]. BoolQ [10], Evidence Inference [32], FEVER [58], MultiRC [29], and SST-2 [53]. One important distinction among these datasets is that BoolQ, FEVER, MultiRC, and Evidence Inference data points include both a query and an accompanying document. The query is typically critical information indicating how the model should process the document, and therefore we never replace query tokens. We use 500 test points from each dataset to compare methods. See Table ?? in the Supplement for dataset statistics, including average sequence length.
Models. We train ten seeds of BERT-Base models on each dataset [12], which we term Standard models. For each dataset we train another ten Counterfactual-Tie Trained (CT) models using the Attention Mask Replace function, following the approach outlined in Sec. 4 (further details in Supplement).
Controlling for Compute Budget. We wish to control for the available compute budget in order to fairly compare explanation methods. Some explanations require single forward and backward pass [51, 33], while others can require hundreds of forward and backward passes [57] thousands of
8

---

Table 1: Explanation metrics across methods and datasets
Sufficiency 4
Comprehensiveness f
Dataset
Method
Standard Model
CT Model
Standard Model
Model
LIME
20.00 (2.02)
27.08 (1.68)
82.18 (2.82)
75.34 (1.93)
Int-Grad
43.76 (3.27)
32.91 (2.36)
34.01 (2.55)
43.22 (2.28)
Anchors
11.93 (1.53)
30.96 (1.87)
55.72 (2.60)
48.86 (2.38)
Gradient Search
17.55 (1.47)
33.98 (1.43)
53.15 (2.53)
49.36 (1.95)
SNLI
Taylor Scarch
6.91 (1.10)
28.00 (1.46)
73.20 (2.57)
66.76 (2.12)
Ordered Search
-1.45 (0.93)
15.06 (1.37)
87.78 (2.41)
84.67 (1.61)
Random Search
-1.54 (0.96)
15.38 (1.39)
87.36 (2.47)
84.63 (1.68)
PLS
-1.55 (1.07)
14.16 (1.38)
87.95 (2.55)
86.18 (1.45)
LIME
2.15 (1.75)
-1.56 (0,63)
52.02 (3.69)
36.25 (3,45)
nut-Grad
20.78 (3.57)
9.05 (1.53)
16.80 (1.57)
12.20 (1.68)
Anchors
11.98 (2.62)
6.07 (1.06)
29.87 (4.17)
15.46 (1.97)
Gradient Search
5.12 (1.41)
1.65 (0.81)
30.04 (2.58)
17.65 (1.85)
BoolQ
Taylor Search
6.01 (1.33)
2.28 (0.87)
46.32 (3.89)
26.65 (2.68)
Ordered Search
0.09 (0.84)
-2.58 (0.70)
51.59 (3.52)
34.36 (3.34)
Random Search
-0.58 (0.63)
-2.11 (0.70)
55.78 (3.71)
31.62 (3.06)
PLS
-1177 (0.47)
-3.52 (0.88)
72.78 (4.06)
47.80 (3.57)
LIME
-16.07 (2.84)
-14992 (1.38)
47.60 (5.66)
33.97 (4.22)
Int -rad
1.22 (4.42)
-2988 (1.68)
26.51 (2.68)
20.87 (2.57)
Anchors
7.08 (4.70)
3.04 (0.99)
25.01 (6.52)
13.89 (1.55)
Evidence Inference
Gradient Search
-0.57 (2.58)
-7.56 (1.46)
31.73 (4.43)
18.07 (2.13)
Taylor Search
4.55 (2.66)
-3.33 (1.27)
41.95 (5.63)
26.70 (3.00)
Ordered Search
16.80 (2.75)
-14.26 (1.36)
45.37 (5.53)
31.14 (3.73)
Random Search
17.05 (2.83)
-12.69 (1.30)
42.81 (6.00)
26.48 (3.15)
PLS
-00.76 (3.77)
-20.33
(2.65)
56.31 (9.81)
38.71 (3,91)
LIME
-0.24 (0.50)
0.39 (0.96)
33.86 (3.43)
22.06 (2.36)
Int-Grad
9.72 (1.80)
4.99 (1.40)
17.81 (2.47)
13.69 (1.71)
Anchors
6.19 (1.22)
6.36 (1.10)
20.82 (2.58)
11.94 (1.84)
Gradient Search
0.66 (0.68)
2.63 (1.12)
19.26 (2.68)
11.44 (1.65)
FEVER
Taylor Search
4.17 (0.96)
4.20 (1.20)
24.51 (2.78)
15.62 (1.85)
Ordered Scarch
-1.26 (0.41)
-0.01 (0.90)
31.79 (3.28)
18.90 (2.46)
Random Scarch
-1.51 (0.51)
-1.24 (2.33)
32.47 (3.33)
18.84 (2.11)
PLS
-2.4 (0.62)
-3.66 (0.82)
37.72 (3.28)
24.07 (2.46)
LIME
-5.20 (1.18)
-5.90
(1.19)
39.75 (4.84)
28.57 (2.18)
Int-Grad
13.19 (3.14)
4.66
(1.71)
15.53 (3.39)
11.84 (1.31)
Anchors
5.40 (3.34)
3.33 (1.27)
24.53 (8.77)
14.55 (1.66)
Gradient Search
-0.09 (1.33)
-0.73 (1.18)
20.16 (2.92)
11.41 (1.13)
MultiRC
Taylor Search
7.54 (2.53)
1.43 (1.47)
30.76 (4.04)
20.15 (1.83)
Ordered Search
-6.43 (0.98)
-5.49
(1.13)
35.70 (4.40)
24.38 (2.03)
Random Search
-7.42 (1.08)
-5.97 (1.22)
35.29 (4.59)
22.19 (1.81)
PLS
10.17 (1.43)
-9.77
(1.49)
39.95 (5.44)
26.96 (2.19)
forward passes [46]. Since this expensive to perform, we limit each method to a fixed number of forward and backward passes (counted together), typically 1000 in total, to obtain single explanation.
6.3 Main Results
In Table 1, we show Suff and Comp scores across methods and datasets, for both the Standard and Counterfactual- Trained (CT) models. 95% confidence intervals are shown in parentheses, obtained by bootstrap with data and model seeds resampled 100k times. Bolded numbers are the best in their group at a statistical significance threshold of < .05. We give results for SST-2 in the Supplement, short sequences there, well for Vanilla Gradient
including for Exhaustive Search since as it performs much worse than other methods. We summarize our key observations as follows:
1. PLS performs the best out of all of the explanation methods, and it is the only method to consistently outperform Random Search. Improvements in Sufficiency are statistically significant for every dataset using both Standard and CT models, with differences of up to 12.9 points over

---

LIME and 7.6 points over Random Search. For Comprehensiveness, PLS is the best method in 9 of 10 comparisons, 7 of which are statistically significant, and improvements are large as 20.8 points over LIME and 17 points over Random Search.
2. LIME is the best salience method on both Suff and Comp metrics, but it is still outperformed by Random Search on Sufficiency in 9 of 10 comparisons, by up to 21.5 points. LIME does appear to perform better than Random Search on Comprehensivennss with three of five datasets for Standard models and four of five with CT models, where the largest improvement over Random Search is 7.49 points.
3. Suff and Comp scores are often much worse for CT models than for Standard models. With Random Search, for instance, Comp scores are worse for all datasets (by up to 24.16 points), and Suff scores are worse by 16.92 points for SNLI, though there are not large changes in Suff for other datasets. The differences here show that the OOD nature of counterfactual inputs can heavily influence metric scores, and they lend support to our argument about the OOD problem in Sec. 4. In particular, these metrics are more easily optimized when counterfactuals are OOD because is easier to identify feature subsets that send the model confidence to I or 0.
Given the results above, we recommend that explanations be obtained using PLS for models trained with Counterfactual Training. Though explanation metrics are often worse for CT models, the only reason for choosing between Standard and CT models is that CT models' explanations aligned, while Standard models' explanations are socially misaligned. It would be a mistake to prefer standardly trained models on the grounds that they are "more easily explained" when this difference is due to the way we unintentionally influence model behavior for OOD data points. When using CT models, however, should be comfortable optimizing for Sufficiency and Comprehensiveness scores, and PLS produces the best explanations under these metrics.
socially
We give additional results for RoBERTa and reduced search budgets in the Supplement.
7 Conclusion
In this paper, we provide a new argument for why it is problematic to use out-of-distriutioo counterfactual inputs when creating and evaluating feature importance explanations. We present Counterfactual Training solution to this problem, and we recommend certain Replace functions over others. Lastly, we introduce Parallel Local Search method (PLS) for finding explanations, which is the only method we evaluate that consistently outperforms random search.
8 Broader Impacts and Limitations
There are several positive broader impacts of improved feature importance estimation methods and solutions to the OOD problem. When model developers and end users wish to understand the role of certain features in model decisions, FI estimation methods help provide an answer. Access FI explanations can allow for (1) developers to check that models are relying on the intended features when making decisions, and not unintended features, (2) developers to discover which features are useful to model accomplishing a task, and (3) users to confirm that decision was based on acceptable features or dispute the decision if it was not. Our solution to the OOD problem helps align Fl explanations with the kind of information that developers and users expect them to convey, e.g. by limiting the influence of the model prior on the explanations.
Nevertheless, there are still some risks associated with the development of Fl explanations, mainly involving potential misuse and over-reliance. FI explanations are not summaries of data points or justifications that given decision was the "right" one. When explanations are good, they reflect what the model has learned, but it need not be the case that what the model has learned good and worth basing decisions on. It is also important to emphasize that FI explanations are not perfect, as there is always possibly some loss of information by making explanations sparse. Trust in and acceptance of these explanations should be appropriately calibrated to the evidence we have for their faithfulness. Lastly, we note that we cannot guarantee that our Counterfactual Training will eliminate the influence of the random seed and model prior on explanations, meaning that Fl explanations for the models we consider will still not be perfectly socially aligned. will be valuable for future work to further explore how these factors influence the explanations seek for models.
10

---

Acknowledgements
We thank Serge Assaad for helpful discussion of the topics here, as well as Xiang Zhou, Prateek Yadav, and Jaemin Cho for feedback on this paper. We also want to thank our reviewers and area chair for their thorough consideration of this work. This work was supported by NSF-CAREER Award 1846185, DARPA Machine-Commonsense (MCS) Grant N66001-19-2-4031, Royster Society PhD Fellowship. Microsoft Investigator Fellowship, and Google and AWS cloud compute awards. The views contained in this article are those of the authors and not of the funding agency.
References
[1] Leila Arras, Ahmed Osman, Klaus-Robert Müller, and Wojciech Samek. Evaluating recur- rent neural network explanations. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 113-126, Florence, Italy, Au- gust 2019, Association for Computational Linguistics. doi: 10. 10.18653/1//11-44833. URL https:/ wwwww///w/wwwwwwbb..............
aclweb. wrw./ogg///aaaaaaa Beathhooggy/W19-440
2,
[2] Seo-Jin Bang, Pengtao Xie, Wei Wu, and Eric P. Xing. Explaining black-box using deep variational information bottleneck approach. ArXiv, abs/1902.06918, 2019. URL https: //arxiv. org/ aaaaab//999// 06918. 2,
[3] Ricardo Baptista and Matthias Poloczek. Bayesian optimization of combinatorial structures.
In International Conference on Machine Learning, pages 462-471. PMLR, 2018. URL http: /pproceedings. mlr. wwtp.//.....//////////a///// ppeess/rt//aappiissatt///aaaaap
pdf. 2, 3, 7,
[4] Jasmijn Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with dif-
ferentiable binary variables. In ACL 2019, pages 2963-2977, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v//P19 1284. URL https:
/ wwww.aclved. org http//anhoooogP11---11 1284. 2,7
[5] David Belanger, Bishan Yang. and Andrew McCallum. End-to-end learning for structured prediction energy networks. In Doina Precup and Yee Whye Teh, editors, Proceedings the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 429-439. PMLR, 2017. URL http: / http:///ppceeeeeeeeee/ee mlr. press/v700 v0/ hts/bbllanng anger 17a. html. 3.7
[6] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP 2015, 2015. URL https://arrn org/ abs/1508..... 05326. 6, 8
[7] Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In ICLR, 2019. URL https://arxiv, wwtpxiv.orr//ppe org/pdf/ 1807 08024. pdf. 3, 4, 6
[8] Hanjie Chen and Yangfeng Ji. Learning variational word masks to improve the interpretability of neural text classifiers. EMNLP, pages 4236 4251, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2000 wttp://2220.mplppp/. main.347. URL https: //www. aclweb. org/ anthology/22000 emnle- main 347. 2
[9] Mare Claesen and Bart De Moor. Hyperparameter search machine learning. arXiv preprint 120. 2015. URL https: //axxiv org/ htfp//ff11100 /1502. 02127 pdf. 5
[10] Christopher Clark, Kenton Lee, Ming- Beig-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-29366 Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: (0.18863/v1//19911333 1300. URL. https: /www. aclweb. org/ anthology/ #19-1300. 8

---

[11] Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. How do deci- sions emerge across layers neural models? interpretation with differentiable masking. In EMNLP,, pages 3243-3255, Online, November 2020. Association for Computational Linguis- tics. doi: 10. http://////22220mmmmmmmmm 2020. emnlp-main.... 262. 1,2
URLL https::///// h http://wwwwwwwwbb/////../oot aclweb. org/ anthology/
[12] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lec, and Kristina Toutanova. of deep bidirectional transformers for language understanding. In ACL 2019, 2019. URL 04805. pdf. 6, 8
Bert: Pre-training
https://aat /atrsi////rivv.//a/ wttpi/.orrgp///////eeeee
[13] Jay Young,, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. Eraser: A benchmark to evaluate rationalized nip models. In ACL 2020, volume abs/191 1.03429, 2020. URL httph::////a/ //axxiv. org/pdf 19111 0ee.1..349..pp. pdf. 1, 2, 3,4
[14] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. arXiv:2002.0643 [cs], February 2020. URL http: /at/aaavvv...abaaaaa//aeeeeaeaee
06305. arxiv: 2002.06305. 5
[15] Qingfeng Du and Jincheng Xu. Model-agnostic local explanations with genetic algorithms for text classification. In The 33rd International Conference Software Engineering & Knowledge Engineering, 2021. URL https: /tsiss///sieeeerchh... org/ htt//seke/eeeeee seskee/eeee2ppppr/
paper040. påf. 2,3
[16] Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-box adversarial examples for text classification. In ACL, pages 31-36, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.1865//1//18-2000 URL htttps://wet 7.8
aclweb. org/ Saathology/PP8--oo
[17] Bradley Efron and Robert J Tibshirani. An Introduction to the Bootstrap. CRC press, 1994. 6
[18] Upol Ehsan, Samir Passi, Q Vera Liao, Larry Chan, Lee, Michael Muller, Mark O Riedl, et al. The who in explainable ai: How ai background shapes perceptions of ai explanations. arXiv preprint arXiv:2107.13500 2021. URL https: / //axxvx.vvvv./////////////aaa pwrriviorgg// påf /2107 13509. pdf. 5
[19] Ruth C Fong and Andrea Vedaldi. Interpretable explanations black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference Computer Vision, pages 3429-3437, 2017. URL httts::tt/aariiiiee http://aaxvvvoo.g..//./ / / 03296. pdf.. 2, 2,
[20] Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? In ACI. 2020, 2020. URL. 01831.pda pdf.
[21] Johannes Haug, Stefan Zürn, Peter El-Jiz, and Gjergji Kasneci. On baselines for local feature attributions. In AAAI, 2021. URL
htt..oorr//pp//11..000001 00905, pdf. 3,
[22] Sara Hooker, Dumitru Erhan, Picter-Jan Kindermans, and Been Kim. A benchmark for in- terpretability methods in deep neural networks. In Proceedings of the 33rd International Conference Neural Information Processing Systems, 2019. URL https: //arxiv org/ abs/1806. 10758. 1, 1,3,4,5 4,5
[23] Cheng-Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu, Pradeep Ravikumar, Seungyeon Kim, Sanjiv Kumar, and Cho-Jui Hsieh. Evaluations and methods for explamation through robustness analysis. arXiv preprini arXiv:2006.00444. 2020. URL https: /ttpps://arzae //taxi/aaiivi////a/ org/ pttp: 3, 4
/200 00442. pdf.
[24] Alon Jacovi and Yoav Goldberg. Aligning faithful interpretations with their social attribution. Transactions of the Association for Computational Linguistics, 9:294-310, 2021. URL https:
/ //direct. mit. maiteedddatcee www/tac//tttetteeeeeeeee http://acc/articlee/o////// 11w.0/1.111111.111 httptaaacaaaa006//aa11
000367/98620/A1ggn1n.. Aligning.
12

---

[25] Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Pupers), pages 3543-3556, 2019. 2
[26] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriizatoo with gumbel-softmax. In ICLR, 2016. URL https: httpp:://axxiva /tta//xxvvorrg.. wwtp:////gpp////t///////o http//dff/61111 .1144.114444.. 0.f.4..df..
[27] Dominik Janzing, Lenon Minorics, and Patrick Blöbaum. Feature relevance quantification in explainable ai: A causal problem. In International Conference on Artificial Intelligence and Statistics, pages 2907-2916. PMLR, 2020. URL https: pdf. 3.4
org/ wwtpp//g////ooooooo/oo/t///te/// 19t.:/911..1.. 13413.
[28] Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath. Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations. In International Conference on Artificial Intelligence and Statistics, pages 1467- PMLR, 2021. URL https: arxiv..ivv....... org/
wwt/oo//g/////p/ooooooo/ /2103 01890. pdf. 3, 4,5
[29] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v 1/N18- 1023. URL https: / /www. aclweb. org/ wwwwanntto/ayaaaeaa /N18 1023. 8
[30] Siwon Kim, Jihun Yi, Eunji Kim, and Sungroh Yoon. Interpretation of NLP models through input marginalization. In EMNLP, pages 3154-3167, Online, November 2020. Association for Computational Linguistics. doi: 10. 1 11653/11/22200 w111200..mllmmmmm 1alipp-mii..5555 255. URL https::/www/ aclweb. org/ anthology/2220. emn mamnlimmminm 255. 2, 3, 4, 6
[31] Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Faithful and customizable explanations of black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 131-138, 2019. URL //ttpps://pprrs .nwss.iiii...................... cc/ cc/paper/2017/ Papp-rappppppp
[32] Eric Lehman, Jay De Young, Regina Barzilay, and Byron C. Wallace. Inferring which medical treatments work from reports of clinical trials. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 3705-3717, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19- 1371. URL bttps://wwae aclweb. org/ anthologyh/aaaaaeaaaaaaaww Mrtcchooogy/N9---o
8
[33] Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. arXiv preprint arXiv: 2015. URL 1506. 1506.01066 pdf. 7,8
[34] Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220, 2016. URL /ttp///xiiiv.or///////.... 08220. odf. 2,
[35] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11199. 2019. URL https: /tpps:///rrxiie org/ httg//df///// /9077 11692. pdf. 6
[36] Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Process- ing Systems 30, pages 4775-4774, 2017. URL /ttp://papers.. nips. ctppssccc/p//////ppppp/ wwwuuuiidddpppppaae to- interpreting-mddel/ model-predictions pdd.tiiiiissn...........
[37] Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In ICLR 2017, 2017. URL. aattps://arnts org/ 0bs/1611.00712. 00712. 8
13

---

[38] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell., 267:1-38, 2019. doi: 10. wtt.....ttin.2018..777 URL https://doi. org/10. 1016// artint.2011 2018.00010.1111000 0011.07.007.2
[39] Jose G Moreno- Torres, Troy Raeder, Rocío Alaiz-Rodriguez, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern recognition, 45(1))521- 530, 2012. URL https://// cis upenn. edu/cis700 101770002199//1 wwwppsppptaaeeeeeeeeeeppppaappse tastasst-shitt/ dataset- - shiftssshiitttte termitttterinoloyy pdt.loo...p.......... 4
cis.upenn.
[40] Ramaravind Kommiya Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Mireille Hildebrandt, Carlos Castillo, Elisa Celis, Salvatore Ruggieri, Linnet Taylor, and Gabricla Zanfir-Fortuna, editors, FAT* '20: Conference on Fairness, Accountability, and Transparency, pages 607-617. ACM, 2020. doi: 10.1145/3350095.337777 33372850. URL https: https://dooo http://oooooog/...//o 1145/3351095 00555/33119966337885000. 3372850. 3,7
[41] Dong Nguyen. Comparing automatic and human evaluation of local explanations for text classification. In NAACL-HLI 2018, 2018. URL htttps:/wwwt aclweb. org/ anthology/ N18-1097 pdf. 1, 3,
[42] Bhargavi Paranjape. Mandar Joshi, John Thickstun, Hannanch Hajishirzi, and Luke Zettlemoyer. An information bottleneck approach for controlling conciseness in rationale extraction. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1938- 1938-1952, Online, November 2020. Association for Computational Linguis- tics. doi: 10.18653/v1/2022 http://ee.000.002mmiaeeee memne-mminnnl.5... URL https: / /www. aclweb. org/ anthology/ 2020 emn 153. 2, 7
[43] Mare Pirlot. General local search methods. European journal of operational research, 92(3):
493-511, 1996. 2, 3, 8
[44] Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. Tunability: Importance of hyper- parameters of machine learning algorithms. J. Mach. Learn. Res., 20(53):1-32, 2019, URL https://arxiv owg/oof.///./ooooooooo/.gg/../ 1100/ 09596. ww059...f......... 5
[45] Luyu Qiu, Yi Yang. Caleb Chen Cao, Jing Liu, Yueyuan Zheng, Hilary Hei Ting Ngai, Janet Hsiao, and Lei Chen. Resisting preprint arXiv:2 14000, 2021. 3.4 4
data problem in perturbation of xai. arXiv
[46] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD In- ternational Conference on Knowledge Discovery and Data Mining, 2016. URL https:
//arxiv. llaaxii.org/ppeeeeeeeeeeooeeeeee wtg/ 1602. 04938. pdf. 2. 7,
[47] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model- agnostic explanations. In AAAI Conference on Artificial Intelligence, 2018. URL https:
/ //homes... cs. washington. edu/ wwwrc/ttraaaaaaaiiaaaaaaaaaaaaaa aaf.fa.................m.mmm...a 3.
[48] Soumya Sanyal and Xiang Ren. Discretized integrated gradients for explaining language models. arXiv preprini arXiv:2 1354,, 2021. URL https: :tttkps:///riiiz org/pdt /2108. 13654. 1df...pdf.
[49] Christian Schäfer. Particle algorithms for optimization on binary spaces. ACM Transactions on Modeling and Computer Simulation (TOMACS), 23(1):1-25, 2013. URL https://arxiv. org/ wrg/pdf// /t////111111..11111111 odf. 2, 3,
[50] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3 1455-3 3153, 2017. URL https: //te:////xxxiii org/ /atf///////// /7044 02685. . pdf. 2
[51] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. Workshop at International Confer- ence on Learning Representations. 2013. URL https:
btaaps://axivv //t////xvvv///////////e/////aav/ //ta//avvrg//pdaaaeeee org/pdf/ /1312. 6034. pdf.
1, 2, 8
14

---

[52] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint arXiv:7706.33825, 2017. URL
https://arxiv.orr www. 11t. 03825. pdf. 2
[53] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural
Language Processing, pages 1-442, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: bestps:/wwii /www.
http:/wwwwwww.././//w/t///twtw.p aclweb. org/ wwt.l/otthooooaoaeeaa DD13 1170. 8
[54] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013. URL sttps://wwwx aclweb. org/ anthology/ D13-1170. pdf. 6
[55] Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 5(1):e22, 2020. URL https://disttil.pbb/22 pub/2020/ atriibutionn-aeeelneee.
[56) Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation.
In International Conference on Machine Learning, pages 9269-9278. PMLR, 2020. URL
https://arxix: wttg//////////////////// 190// 08474, pdf. 3,4 4
[57] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks.
In International Conference on Machine Learning, pages 3319,3328, 2017. URL https: 01365. pdf. 1, 2, 3, 4, 4, 7,8
/ //arxiv. orgoorgg/...o
[58] James Thorne, Andreas Vlachos, Christos Christodoulopoulos and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pupers), pages 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 10.18653/v1/N18- 1074. URL https: wwwww..wwwwbbbb../////.//////... aclweb. org/ httpa/anoooogg/NNo-oooo 1074. 8
[59] Keyon Vafa, Yuntian Deng, David M Blei, and Alexander M Rush. Rationales for sequential predictions. In EMNLP, 2021. URL https: /ww.vv.ivvvvvv///vv/ wwggppff/te /2109. 06387 pdf.. 3, 4
[60] Jan N Van Rijn and Frank Hutter. Hyperparameter importance across datasets. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2367-2376, 2018. URL https: /twpvv..ivvvv/////////a///a.aa org/pd/t htfp//ff/111/ /1710. 111111771100775511 pdf.
[61] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (FMNI.P-JCCNLP), pages 11 11-20, 2019. URL /90/.. 04626. .1122..p. pdf. 2
https:/ http///xxxvvvrrr/
[62] Maksymilian Wojtas and Ke Chen. Feature importance ranking for deep learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5105-5114. Curran As- sociates, Inc., 2020. URL https: / http:///pppooo..neeeeeee www.oeedddinnnnnuissseeeoee wwwripp..ccppppeepee cctpapper//22//110 Paper. ..teer.dffff. pdf. 2
[63] Jihun Yi, Eunji Kim, Siwon Kim, and Sungroh Yoon. Information-theoreti visual explanation for black-box classifiers. arXiv preprint arXiv:2009. 2020. URL https:://aat //trxxaaaivvv///////// org/ pdf /2009.00.. 11150. .df. 2, 3, 4,
[64] Fan Yin, Zhouxing Shi, Cho-Jui Hsich, and Kai-Wei Chang. On the faithfulness measurements for model interpretations, 2021. URL https: http:::///xxvvvv////ee
087822.0......... .df. 3
[65] Omar Zaidan, Jason Eisner, and Christine Piatko. Using "Annotator Rationales" to Improve Machine Learning for Text Categorization. In Human Language Technologies 2007: The Confer- ence of the North American Chapter of the Association for Computational Linguistics; Proceed- ings of the Main Conference, pages 260-267, Rochester, New York, April 2007. Association for
15

---

Computational Linguisties. URL htttps://www.acllee hww..//..rrggaaaaooog///0//----- 1033 2,
4
[66] Ruiqi Zhong, Steven Shao, and Kathleen McKeown. Fine-grained sentiment analysis with faithful attention. arXiv preprint Stti::/900.0687700 2019. URL https://ant http::/aaaxv.o/ppppeee 1908.06870. pdf.
[67] Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? arXiv preprint arXiv:210.144033 2021. URL https: //arxiv. wwtpooggg/////p//////o////////// /2104. 14403.pdl pdf.
[68] Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. ICLR, 2017. URL https://arxi/ org/ pdf/1702. 04595. adf. 2, 3,6
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]. See the Broader Impacts and Limitations
(c) Did you discuss any potential negative societal impacts of your work? [Yes]] See the Broader Impacts and Limitations section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Code and data are provided the Supplement.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes]. See Supplement and ??.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Footnote 2 and Table captions (e.g. in Table 1). (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Supplement ??.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]. We cite the original authors contributing each dataset.
(b) Did you mention the license of the assets? [Yes] See Supplement ??.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A] We use publicly available datasets in accordance with their licenses.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] This issue is not applicable in the case of these public datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
16

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
17


Reviews:
[{"review_id": "jHzGdLrs7u2", "reviewer": "Reviewer_NMtf", "summary": "This paper investigates feature important methods in NLP. The authors argue that counterfactual inputs should be in-distribution, and propose training on counterfactual inputs to reduce this distribution shift. They also investigate the best approach to producing counterfactuals and the best search method for finding FI explanations, finding that one of their proposed methods does best. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an important problem and has the potential to be a strong paper. It is somewhat original and includes many experiments. However, overall it feels unpolished and does not yet seem ready for publication.\n\nThe paper claims to make a new argument for why OOD counterfactuals are bad; namely, that they are socially misaligned. I strongly agree with the conclusion that OOD counterfactuals can be misleading for interpretability, as other authors have pointed out before. However, I don't feel like the argument that they're socially misaligned really adds anything. \n\nThe authors then propose training on counterfactual inputs to reduce distribution shift. This basic idea seems natural and reasonable to me. However, there are a few cases where they make claims about what approach to producing counterfactual inputs is \"best\" by using accuracy as a proxy for how in-distribution examples are. I think this is kind of misleading; accuracy is affected both by how OOD the data is and by how much actual information is lost (e.g. you could train on counterfactuals where information is removed, and this would then be not OOD but still have lower accuracy). \n\nMoreover, I think it's unclear how I should think about explanations from counterfactually-trained models. It seems plausible but not obvious to me that I should now basically trust FI explanations from counterfactually-trained models (whereas FI explanations from non-CT models are misleading because of the OOD gap). To the extent that you're proposing a new approach to explaining models, you should explain to me in what sense it's an explanation, why I should trust it, etc. \n\nRelatedly, the authors use metrics of sufficiency and comprehensiveness, which seem to capture meaningful aspects of how good an explanation is. However, I'm not convinced they capture everything we care about. There are also some mysterious results, such as how some \"strong\" baseline methods do terribly according to these metrics. This makes me suspicious that either these baselines are not strong, or that these metrics are unreliable. \n\nIn general, I think some of the results are potentially interesting, but in many cases I'm not sure how much to trust them or how I should best interpret them. I think a clearer exposition would go a long way toward these problems, which I think make this work not yet publishable.\n\n---------------\nUpdate: I've read the rebuttal. While it clarifies some minor points, overall I still don't think this work is ready for publication, so I'll choose to keep my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "YM1yIKcMxph", "reviewer": "Reviewer_abDP", "summary": "This paper addresses a core limitation of existing feature importance methods, where importance scores are generated by comparing original examples to unrealistic counterfactuals. To solve this issue, the authors propose a solution where counterfactuals used to compute feature importance are generated by manipulating the attention mask and not by editing the actual text of the original examples. They also propose training models on the generated counterfactuals and show that it improves model robustness. Finally, they introduce a novel method producing model explanations and show that it outperforms strong baselines.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper contains some interesting findings and proposes a novel search-based method which improves on existing feature-importance solutions. It is also clearly written and easy to follow through.\n\nHowever, in its current state, the contributions presented in the paper are not substantial enough in my opinion. First, the realization that unrealistic counterfactuals harm our ability to estimate the effect of variables is a pillar of causal research, and there are existing methods in ML and in NLP specifically which deal with this problem in the context of causal explanations (for example, see CausaLM, INLP, CaCE and their follow-up work).\n\nSecond, the use of counterfactuals for training can be misleading. While it can improve model performance (as demonstrated in the counterfactually augmented data literature), measuring model performance on in-domain data and calling it more robust is problematic. Alternatively, counterfactually trained models should be tested on types of counterfactuals which were not seen in training. Also, counterfactual training can harm our ability to identify model weaknesses, which is a main motivation for generating model explanations in the first place. \n\nThe two main arguments raised in this paper, that the OOD-ness of counterfactuals leads to poor explanations and that training on OOD data improves robustness, are inherently causal arguments which I think already exist in the literature. While the results in the paper are surely interesting, I would like to see the authors address the existing literature on the subject and try to explain the similarities and differences in the theoretical approach. While I am pointing to one prism of thought that might be useful for analyzing the paper’s findings, there might be others. To improve the paper, I would like to see the authors address similar arguments made in the literature.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper can potentially have a substantial positive societal impact, as understanding what features drive model decisions can uncover hidden biases and fairness issues which govern the model’s behavior. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "XJUldUx5Nf1", "reviewer": "Reviewer_Lhz2", "summary": "The paper considers different aspects of feature-importance-based explanations. In particular, the authors propose a ‘counterfactual training’ approach to deal with observations that are out of the original training distribution (OOD smaples). Moreover, the authors illustrate different approaches to replace input features with a pre-defined baseline, i.e., a common approach to assess the quality of an attribution/explanation method, which however may lead to OOD samples. Finally, the authors propose search algorithms to find binary-valued, feature-based explanations that have been shown to improve relevant test measures.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper discusses interesting aspects of feature-attribution-based explainations and introduces a novel approach to deal with the out-of-distribution problem of ablated observations. The use of search methods as an alternative to local (linear) approximations for the identification of important features seems interesting.\n\nUnfortunately, there are several concerns regarding the overall quality and contribution of the paper, which I outline below:\nThe main contributions of this paper (line 72) do not form a consistent story line. While the first two contributions consider the evaluation of local attribution methods, the third contribution (search methods for local explainability) is somewhat isolated from the rest of the paper.\nThe term ‘counterfactual’, as used in this paper, does not adhere to common definitions in the literature. The authors speak of a counterfactual as an input, ‘which has its important features replaced with a user-specified (uninformative) feature’ (line 45). In general, however, a counterfactual explanation is considered the smallest change made to an input vector to influence its prediction. The authors might want to talk of ‘ablated inputs’ (or similar) instead.\nIn the related work section, the authors should also consider the recent paper of Haug et al. [1], when talking about the effect of different replace (baseline) methods.\nThe specified main objective (line 136) is to generate explanations that provide good sufficiency/comprehensiveness statistics in the evaluation. However, existing work on local attributions already specifies much stricter and more meaningful properties (e.g. Lundberg et al. [2]). The authors should argue, why their objective should be preferred.\nThe authors acknowledge the (negative) effect of out-of-distribution samples in the evaluation of local explanations. However, rather than improving the explanation or evaluation process, the authors manipulate the model training to obtain ‘expected behaviour’ (line 162). This is a highly questionable approach, as the very purpose of an explanation technique is to communicate what a given (!) black-box model does/has learned. In fact, the authors' approach to bring ablated inputs in line with the training distribution, is to alter the original training distribution (and therefore the trained black-box model). In this context, the proposed ‘counterfactual training’ is an approach to make predictive models more robust, but, in my opinion, it is not a meaningful technique to improve the quality of explanations.\nThe description of the ‘Replace’ functions (lines 221-230) is insufficient. For example, it remains unclear what a ‘MASK token’ is. Besides, the different ‘Replace’ functions are compared with respect to random explanations. The differences observed in Figure 2 might be attributed to this random sampling. The authors should therefore discuss, why random explanations are to be preferred here over real explanations (e.g. LIME).\nIn the experimental description, the authors describe that they ignore negative attributions when selecting the top $k$ features (line 262). However, (strongly) negative attributions can indicate important features as well (see SHAP and LIME, for example). The authors might want to reconsider this approach.\nThe advantage of the proposed search methods for local attribution finding remains unclear. Other than empirically improved sufficiency and comprehensiveness scores, the authors fail to provide any sensible properties or strong guarantees on the generated attributions. In fact, the search methods only yield 0/1-indications of important features and thus contain much less information than existing attribution methods (which usually generate scalar weights and thus also allow a ranking of features).\nFinally, in the discussion of the experiments (line 357), the authors find that the sufficiency/comprehensiveness scores are often much worse when using the proposed counterfactual training. This contradicts the earlier recommendation to use counterfactual training ‘whenever one intends to create feature-importance-based explanations’ (line 253).\n\n[1] Haug, Johannes, et al. \"On Baselines for Local Feature Attributions.\" arXiv preprint arXiv:2101.00905 (2021).\n[2] Lundberg, Scott M., and Su-In Lee. \"A unified approach to interpreting model predictions.\" Proceedings of the 31st international conference on neural information processing systems. 2017.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed some general limitations of local explanation methods. However, there are several more specific limitations and problems that the authors have not addressed yet (see detailed comments). ", "review_text": "", "extra_scores": {"ethical_concerns": "There are no apparent ethical concerns regarding this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "0qXOgSCuLPU", "reviewer": "Reviewer_2JSj", "summary": "Summary\n\nExplanations methods based on feature importance (FI) assign an importance score to each input feature. This is often evaluated by (1) removing the k most important features and replacing it with a feature that is in some sense defined as uninformative (also referred to by “Replace” function in the paper), (2) measuring how much the confidence in the original prediction drops (= sufficiency metric), (3) a good explanation is determined by still having a high confidence in the original prediction.\n\nThe paper argues that this approach is problematic because the explanations, were some features are replaced with uninformative features, represent a counterfactual out-of-distribution (OOD) example that the model is not equipped to handle correctly. As a consequence, the paper argues, the “best” explanation, as determined by the above approach, does not necessarily provide the information the user expects, making this explanation “socially misaligned”.\n\nThe paper studies the problem and explores alternative options, which are tested on six text classification tasks.\n\nContributions\n\n-\tArguments on why the current sufficiency-based evaluation of FI explanation methods is problematic\n-\tInvestigation of which “uninformative” features are best for avoid the outlined problem (see second paragraph above)\n-\tNovel search-based methods for finding FI explanations\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n-\tInteresting discussion on how OOD explanations do not reflect what a user expects and how this is an issue. Considerations of how explanations actually affect users is important in order for the research field of explainable AI to head into a direction where explanations truly reflect what is needed in applications by users.\n-\tSystematic comparison of various FI explanation methods and different choices for selecting uninformative features. This gives interesting insights and is particularly value for anyone who wants to apply FI explanation methods in practice.\n-\tA new search methods that finds the best explanation iteratively (with regards to sufficiency metric), rather than taking the top k highest as features\n-\tExperimental results are thorough and follow good practice (e.g. 10 random seeds, error bars)\n\nWeaknesses\n\n-\tSome questions remain open but can easily be addressed.\n-\tIt is not clear that the insights drawn will really generate explanations that are truly more socially aligned. Of course, answering this question is in itself tricky and the paper acknowledges this in the broader impact statement \n\nAfter Authors' Response\nThank you for answering my questions! I like the idea of this paper, although I do agree with some of the concerns raised by other reviewers, particularly that the story line is in parts not clear to follow. Thus, I will keep my score unchanged.\n\nQuestions\n\n-\tLine 152: Why do you say a MASK token is OOD, i.e. not seen in training? At least in pre-training of a masked LM, the MASK token is part of training.\n-\tParagraph starting at line 163: Yes, model prior and random seed affect an explanation, but they should, correct? A different seed might really lead to a different model and therefore a different explanation? [see also footnote 1 below] My guess is that the main difference is: models with different random seeds also handle the same OOD counterfactual differently and how different models react to OOD counterfactuals remains intransparent, leading to the social misalignment? Do you agree? Could the argument be made clearer?\n\n[footnote 1] This also relates to what Jacovi&Goldbert term faithfulness, in that context for an explanation to be faithful, a model with a different random seed and a different prediction also might have a different explanation and it needs to have a different explanation because it is indeed a different model, this is needed for the explanations to be faithful.\n\n-\tLine 224, “Marginalize”: I’m not sure I understood this paragraph correctly, is the following correct: For the input feature to be removed, you use BERT to replace the token at the to-be-removed feature position? E.g. in “A cat is sat on a mat” and “cat” is to be replaced, you ask BERT what other word to put there and it e.g. give you “dog”?\n-\tLine 267, Vanilla Gradients: Which replace function do you use there?\n-\tLIME vs Integrated Gradients: Why does one use the replace function attention mask and the other mask token? \n-\tTable 1: The difference between e.g. LIME and random search is that LIME uses the top-k features whereas random sample k at random? Sorry if I missed it, but what is the value for k? How did you choose it?\n\nTypos, representation, minor questions etc.\n\n-\t115: explanation -> explanations\n-\tIt is strange that the sufficiency metric is introduced twice\n-\tIt took me a few reads of the beginning of section 5 to understand what exactly the setup is and why, maybe this could be rephrased some? For example, “where the set of input features to be removed is fixed” -> explicitly state here why this is important and needed\n-\tLine 290: delete “produce”\n-\tLine 294: delete “with”\n-\tLine 295: what is the regularizer here?\n-\tLine 334: It would be great if you could list the budget (forward/backward passes) required by the different methods that you test.\n-\tTable 1: Bold in last section, second last column is missing; same for last column of “Evidence Inference”\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-\tThe paper provides a good overview of the potential risks of FI explanation methods.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews for this paper were mostly borderline, with one review strong negative. While the reviewers thought that the paper addresses an important problem in a novel way, and provides interesting insights, the author response did not lead them to change their scores. 

Despite this lukewarm assessment, I believe the paper makes contributions that likely merit acceptance: (1) Tackling an important problem in generating and evaluating explanations in NLP and ML more broadly (the OODness of feature importance attribution methods that are based on masking, ablation, etc., since they produce counterfactual/OOD examples); (2) providing a clear approach to mitigate this problem (adding counterfactuals at training time), which can work well (little drop compared to standard models); (3) evaluating various methods to create explanations and finding that some work well in the new counterfactually trained models (achieve good sufficiency/comprehensiveness scores, which are standard). 

Given these contributions, and considering that no paper is perfect, I recommend acceptance. Since my recommendation goes somewhat against the general sense of the reviews, I provide here a detailed discussion of the main issues raised by the reviewers, how the authors responses, and my opinion of the issue. In any case, I urge the authors to take these issues into account in their next revision. 

**Comments by Reviewer NMtf**
- argument on social alignment doesn't add much. Authors: it's another perspective on why FI OODness is problematic, and may help. AC: natural application of the social alignment argument from Jacovi and Goldberg to another class of explanations.
- Using accuracy to assess distribution shift is misleading, can conflate OODness and information removal. Authors: remove the same information in all conditions, difference is due to OODness. AC: agree that accuracy change between iid and ood data is a reasonable metric of OODness. 
- Not convinced about sufficiency and comprehensiveness metrics. Authors: These are standard in the literature. AC: agree, these are standard faithfulness metrics. What's missing here is an evaluation of plausibility, for example by comparing with human-annotated explanations (highlights, rationales). 
- Baseliens do terrible. Authors: LIME does well, IG is likely a poor method. AC: It's tricky to compare when the benchmark is not common. It could have been better to use standard datasets from explanations in NLP, where prior results are available. But, LIME results are indeed quite good. 

**Reviewer Lhz2**
- The contributions do not form a consistent story line. Authors disagree and explain that the part on local attributions depends on fixing the OOD problem of explanations. AC: agree, the first part is a nice contribution in itself and it makes the rest possible. 
- Counterfactual is an incorrect term here, since a counterfactual explanation means something different. Authors:  did not use counterfactual explanation. AC: best to clarify that counterfactual input is not to be confused with the technical term of counterfactual explanation. 
- questions the sufficiency/comprehensiveness metrics and prefers more meaningful metrics like SHAP. Here I agree with the authors that it is useful to apply these empirical metrics, which are standard in the literature, even if they do not have some nice theoretical properties like SHAP.
- Changing the training distribution is questionable. Authors: to use FI explanations, have to align training and test distributions, so if we want to keep test time metrics, must change training distribution. AC: indeed, I was also concerned by changing the training distribution and hence the model, but was pleased with the minor drop in accuracy, at least for some of the methods. However, it's possible that a model trained on the original distribution arrives at decisions in a different way than a model trained on the counterfactually-modified distribution, but they have similar performance. This is a hard problem to get around, but one that should be discussed. 
- Search methods provide binary importance while existing methods provide scalar scores. Authors argue that their method still brings useful rankings (15 out of 24 vs LINE) and that for selecting features it works best. AC: not convinced by the author response, as 15/24 seems weak, and also as scalar weights can indeed be useful, both for interpretation (users might prefer them) and for downstream tasks (training with explanations, etc.). 
- Sufficiency and comprehensiveness are often worse when using the proposed counterfactual training. Authors explain they recommend using CT regardless of these metrics, because using CT guards again OODness. AC: while I agree with the idea in principle, I'm unconvinced by the argument. If using a standard model leads to better explanations according to the metrics in question, then the argument for using CT grows much weaker. 

**Reviewer 2JSj**
- Had some questions that seemed to have been answered.
- Not clear that the explanations will truly be more socially aligned. AC: did not see a clear answer to this. The answer to the issue of social alignment and how different models react to OOD counterfactuals is part of the story, but probably not all. 

**Reviewer abDP**
- there are existing method to deal with the problem of estimating effects in the case of counterfactuals (mentioning specific studies: CausaLM, INLP, CaCE). Authors: don't see how these particular studies lessen the contribution, since they don't discuss how OODness affects social alignment. Also say that the mentioned methods are not meant to use with individual data points. AC: agree. Those should probably be discussed, but are not at the core of the current work.
- Counterfactual training can be misleading in terms of evaluation. Authors argue that in their case it makes sense, because the CT models work better on OOD data, and they're not interested in different kinds of OODness. AC: agree, this actually seems reasonable for the present use case of evaluating explanations, if not for robustness more broadly. The authors should clarify their goals in this evaluation and distinguish from general model robustness. 


Decision: Accept (Poster)