Counterfactuals As a Means for Evaluating Faithfulness of Attribution Methods in Autoregressive Language Models

Authors: sepehr  kamahi
yadollah  yaghoobzadeh
Venue: EMNLP_2024_Workshop_BlackBoxNLP
Published: 2024-08-20 11:07:14

Abstract: Despite the widespread adoption of autoregressive language models, explainability evaluation research has predominantly focused on span infilling and masked language models. Evaluating the faithfulness of an explanation method—how accurately it explains the inner workings and decision-making of the model—is challenging because it is difficult to separate the model from its explanation. Most faithfulness evaluation techniques corrupt or remove input tokens deemed important by a particular attribution (feature importance) method and observe the resulting change in the model's output. However, for autoregressive language models, this approach creates out-of-distribution inputs due to their next-token prediction training objective. In this study, we propose a technique that leverages counterfactual generation to evaluate the faithfulness of attribution methods for autoregressive language models. Our technique generates fluent, in-distribution counterfactuals, making the evaluation protocol more reliable.

Main paper: 
 introduction: 
 Most modern NLP systems rely on autoregressive, transformer-based language models (Brown et al., 2020; Touvron et al., 2023; Groeneveld et al., 2024) . These models are inherently opaque, creating a strong need to understand their decision-making processes. As a result, explanation methods have become increasingly important in the field.A widely-used approach for model explainability is attribution, also known as feature importance (FI) (Zhao et al., 2023) . Attribution methods aim to identify which input features contribute most to a model's predictions, assigning a scalar value to each feature that reflects its relevance in the decision-making process. In typical NLP tasks, input features are often subwords or their combinations.A key challenge in evaluating the faithfulness of attribution methods is that many existing techniques are designed for denoising or masked language models (MLMs) (Kobayashi et al., 2020 (Kobayashi et al., , 2021;; Ferrando et al., 2022b; Modarressi et al., 2022 Modarressi et al., , 2023;; Mohebbi et al., 2023) . Recent work on autoregressive models has primarily focused on the plausibility of attributions (Yin and Neubig, 2022; Ferrando et al., 2023) . While plausible (or persuasive) explanations might be the objective of the explainer, the core objective for the user is to truly understand the model's decision-making process, rather than simply being convinced that the model's decisions are correct (Jacovi and Goldberg, 2021) .Nearly all previous methods for faithfulness evaluation modify the input in some way, such as masking or removing important tokens based on the attribution results, and then measuring the impact on the model's predictions. These methods tend to work well for MLMs, which are specifically trained for tasks like span or mask infilling. However, in the case of autoregressive models like GPT-2, which predict the next token, such modifications produce out-of-distribution (OOD) inputs. This raises a crucial question: are these evaluation methods truly assessing the informativeness of the selected tokens, or merely testing the model's robustness to unnatural text and the artifacts introduced by testing modifications (Hooker et al., 2019) ? Moreover, the OOD nature of these inputs results in explanations that become socially misaligned (Hase et al., 2021) . In other words, the expectations of users-who seek to understand which features are most relevant to the model's decision-no longer align with the actual output of the attribution method. Instead, feature importance becomes influenced by the model's priors rather than the learned features that truly drive predictions.In this work, drawing inspiration from coun-terfactual generation-where the input is altered to flip the model's output-we propose a new technique to evaluate the faithfulness of attribution methods in autoregressive language models. Specifically, we use counterfactual generators to modify the input by focusing on tokens highlighted by attribution methods, while ensuring that the altered input remains natural, fluent, and within the model's original distribution. This ensures that any observed change in the model's predictions is due to the modification of the important tokens, rather than an effect of OOD inputs.We argue that if an attribution method enables a counterfactual generator to modify fewer tokens to change the model's prediction, then it demonstrates a stronger understanding of the model's inner workings, indicating higher faithfulness. To validate our approach, we apply this faithfulness evaluation technique to several attribution methods-including gradient norm, gradient × input, erasure, KernelSHAP, and integrated gradients-within the context of next-word prediction for two language models: the fine-tuned Gemma-2b and the off-the-shelf Gemma-2b-instruct (Team et al., 2024) .Our contributions are as follows: (i) We introduce a novel faithfulness evaluation protocol that preserves the model's input distribution, designed for attribution methods in autoregressive language models. (ii) We apply this protocol to evaluate and rank widely-used attribution methods, showcasing differences in sensitivity between fine-tuned and off-the-shelf models when handling OOD data and proposing a solution.1 2 Related work Evaluating Explanations. Most current metrics for evaluating faithfulness involve either removing important tokens or retraining the model using only those identified as important by attribution methods (Chan et al., 2022) . For instance, Abnar and Zuidema (2020) assess explanations by comparing them with gradient and ablation techniques. Although Wiegreffe and Pinter (2019) caution that gradients should not be considered ideal or the "ground truth," they still utilize gradients as a proxy for the model's intrinsic semantics. Importantly, the trustworthiness of explanations is both taskand model-dependent (Bastings et al., 2022) , and different attribution methods frequently produce inconsistent results (Neely et al., 2022) . As a result, it is not justifiable to treat any single explanation method as a universal standard across all contexts.In their work, DeYoung et al. ( 2020) introduce two key concepts: comprehensiveness (whether the important tokens identified are the only ones necessary for making a prediction) and sufficiency (whether these important tokens alone are enough to make the prediction). Carton et al. (2020) build on this by proposing normalized versions of these concepts, comparing comprehensiveness and sufficiency to the null difference-the performance of an empty input (for sufficiency) or a full input (for comprehensiveness). However, it remains unclear whether these corruption techniques evaluate the informativeness of the corrupted tokens or merely the robustness of the model to unnatural inputs and artifacts introduced during evaluation.Further, Han et al. (2020) and Jain et al. ( 2020) frame attribution methods as either faithful or unfaithful, with no consideration for degrees of faithfulness. They describe attribution methods that are "faithful by construction." In contrast, other researchers propose that faithfulness exists on a spectrum and suggest evaluating the "degree of faithfulness" of explanation methods (Jacovi and Goldberg, 2020) . Our approach aligns with this view, as we aim to find explanation methods that are sufficiently faithful for autoregressive models. Atanasova et al. (2023) evaluate the faithfulness of natural language explanations using counterfactuals, applying techniques from Ross et al. (2021) to assess how well explanations align with the model's decision-making. This line of work offers valuable insights into the use of counterfactuals, which we build upon for evaluating attribution methods in language models. Another relevant direction is the evaluation of explanations using uncertainty estimation. For example, Slack et al. (2021) develop a Bayesian framework that generates feature importance estimates along with their associated uncertainty, expressed through credible intervals, highlighting the importance of uncertainty in faithfulness evaluationsThe OOD Problem in Explainability.The issue of OOD inputs in explainability has been raised by several works. Hooker et al. (2019) and Vafa et al. (2021) suggest retraining or finetuning the model using partially erased inputs to align training and evaluation distributions. However, this process can be computationally expen-sive and is not always practical. An alternative approach by Kim et al. (2020) aims to ensure that the explanation remains in-distribution to mitigate OOD problems. Our work addresses this concern by preserving the input distribution during faithfulness evaluation, particularly for autoregressive models.Feature Importance (Attribution). Attributions, or feature importance scores, are local explanations that assign a score to each input feature-typically token embeddings in NLP tasks-indicating how crucial that feature is to the model's prediction. Attribution methods can be categorized into four types: i) Perturbation-based methods, which alter or mask input features to assess their importance by observing changes in the model's output (Li et al., 2016 (Li et al., , 2017;; Feng et al., 2018; Wu et al., 2020) . ii) Gradient-based methods, which calculate the derivative of the model's output with respect to each input to measure the influence of each feature (Mohebbi et al., 2021; Kindermans et al., 2019; Sundararajan et al., 2017; Lundstrom et al., 2022; Enguehard, 2023; Sanyal and Ren, 2021; Sikdar et al., 2021) . iii) Surrogatebased methods, which explain a complex black-box model using a simpler, interpretable model (Ribeiro et al., 2016; Lundberg and Lee, 2017; Kokalj et al., 2021) . iv) Decomposition-based methods, which break down the overall importance score into linear contributions from the input features (Montavon et al., 2019; Voita et al., 2021; Chefer et al., 2021; Modarressi et al., 2022; Ferrando et al., 2022a) . 
our method: 
 Our faithfulness evaluation protocol involves two models: a counterfactual generator model and a predictor model. Our goal is to evaluate the faithfulness of attribution methods for the predictor model. Due to the large output space of autoregressive language models (LMs), which often includes thousands of vocabulary items, examining the entire output space does not provide much insight. Therefore, we use the contrastive explanations proposed by Yin and Neubig (2022) , which measure the attribution of input tokens for a contrastive model decision. Contrastive attributions aim to identify the most important tokens that led the model to predict the target y t instead of a foil y f . We then use a separate editor model to modify these important tokens to generate counterfactuals-examples that make the original predictor model more likely to predict the foil.Our protocol for evaluating attributions consists of two phases. The first phase involves creating the editor that can generate counterfactuals. In the second phase, we use the editor and predictor together to determine what percentage of tokens the editor needs to change to flip the predictor model's prediction. Figure 2 illustrates the second phase.To create the editor, we fine-tune an autoregressive language model specifically for counterfactual generation. During fine-tuning, we add two tokens to the embedding space and the tokenizer: '<mask>' and '<counterfactual>'. Inspired by Wu et al. (2021) and Donahue et al. (2020) , we create training examples for our counterfactual generator by randomly masking between 5% and 50% of the tokens. We then append each example's label (e.g., positive or negative for the SST-2 dataset), the '<counterfactual>' token, and finally the original unmasked example. The process of creating training examples is shown in Figure 3 .In the second phase of evaluating attributions, we first input a sentence into the predictor and apply an attribution method to identify the most important tokens influencing the predictor's decisionmaking process. We begin by replacing 10% of these most important tokens with '<mask>' and present the masked sentence along with the foil label (the label with the second-highest logit) to the editor to generate a counterfactual sentence-one that flips the prediction of the predictor model. If unsuccessful in flipping the prediction, we incrementally increase the masking by 10% until we either flip the prediction or reach a masking threshold of 50%. This evaluation protocol is depicted in Figure 2 . The prompting technique used for counterfactual generation during this phase is shown in Figure 1 . The attribution technique that identifies the most critical tokens for creating counterfactuals and enables counterfactuals with the least amount of change to the original text is considered to provide the most faithful representation of the predictor's decision-making process. We use three datasets for evaluating faithfulness: SST-2 (Socher et al., 2013) and IMDB (Maas et al., 2011) , which are both binary classification datasets, and AG-News (Zhang et al., 2015) , a four-class classification dataset.Faithfulness evaluation datasets should not have gold attribution labels because we do not want human intuition to influence the evaluation. Instead, we aim to understand how the model makes predictions (Jacovi and Goldberg, 2020) . 
editor models: 
 For the editor model, our method is similar to Wu et al. (2021) , which uses GPT-2, a decoder-only causal model, for generating counterfactuals. We extend this by using three more modern decoderonly models: GPT-J-6B (Wang and Komatsuzaki, 2021) , which we refer to as "gptj," and two sizes of Pythia: Pythia-1.4B (pythia1) and Pythia-2.8B (pythia2) (Biderman et al., 2023) . We fine-tune these models following the process described in Section 3. The pythia1 model is fully fine-tuned, while the other two (gptj and pythia2) are finetuned using Low-Rank Adaptation (LoRA) (Hu et al., 2022) . All models are trained for 8 epochs using dynamic masking (Liu et al., 2019) , meaning each example is masked differently in each epoch.We use Gemma-2b (Team et al., 2024) as the predictor model. We fine-tune the raw language model for the three datasets (referred to as gemma-ft) using Low-Rank Adaptation (LoRA). Additionally, we employ an off-the-shelf instruct-tuned version (gemma-it) for zero-shot evaluation. We then conduct a detailed comparison between these two versions-fine-tuned (gemma-ft) and non-finetuned (gemma-it)-to assess their differences in attribution evaluation.Here we detail the six widely used attribution methods employed in our study. We use all attribution methods in a contrastive way (Yin and Neubig, 2022) . Contrastive attributions measure which features from the input make the foil token y f more likely and the target token y t less likely. We denote contrastive, target, and foil attributions by S C , S t , and S f respectively:S C = S t -S f (1)We use the implementation of these attribution methods provided by Yin and Neubig (2022) (for Gradient × input, gradient norm and erasure) and by Captum (Miglani et al., 2023) (for KernelSHAP and Integrated Gradient).We can calculate attributions based on the norm of the gradient of the model's prediction with respect to the input x (Simonyan et al., 2013; Li et al., 2016) .The gradient with respect to feature x i is given by:g(x i ) = ∇ x i q(y t |x)Where q(y t |x) is the model output for token y t given the input x. The contrastive gradient:g C (x i ) = ∇ x i (q(y t |x) -q(y f |x))We will use both norm one (gradnorm1) and norm two (gradnorm2):S C GN 1 (x i ) = ||g C (x i )|| L1 S C GN 2 (x i ) = ||g C (x i )|| L2 4.3.2 Gradient × InputIn gradient × input (gradinp) method (Shrikumar et al., 2016; Denil et al., 2014) , we compute the dot product of the gradient and the input token embedding x i :S GI (x i ) = g(x i ) • x iBy multiplying the gradient by the input embedding, we also account for how much each token is expressed in the attribution score. The Contrastive Gradient × Input is:S C GI (x i ) = g C (x i ) • x iErasure-based methods measure the importance of each token by erasing it and observing the effect on the model output (Li et al., 2017) . This is achieved by taking the difference between the model output with the full input x and the model output with the input where token x i is zeroed out, denoted as x ¬i :S t E (x i ) = q(y t |x) -q(y t |x ¬i )For the contrastive case, S C E (x i ) becomes:(q(y t |x) -q(y t |x ¬i )) -(q(y f |x) -q(y f |x ¬i ))KernelSHAP (Lundberg and Lee, 2017) explains the prediction of a classifier q by learning a linear model ϕ locally around each prediction. The objective function of KernelSHAP constructs an explanation that approximates the behavior of q accurately in the neighborhood of x. More important features have higher weights in this linear model ϕ. Let Z be a set of N randomly sampled perturbations around x:S t ϕ = arg min ϕ z∈Z [q(y t |z) -ϕ T z] 2 π x (z) (2)KernelSHAP uses a kernel π x that satisfies certain principles when input features are considered agents of a cooperative game in game theory. We use equation 2 in a contrastive way. First we normalize S t ϕ and S f ϕ by dividing by their L2 norm and then subtracting:S C ϕ = S t ϕ ||S t ϕ || - S f ϕ ||S f ϕ ||(3)Integrated Gradients (IG) (Sundararajan et al., 2017 ) is a gradient-based method which addresses the problem of saturation: gradients may get close to zero for a well-fitted function. IG requires a baseline b as a way of contrasting the given input with the absence of information. For input i, we compute:S t IG (x i ) = 1 m m k=1 ∇ x i q y t b+ k m (x-b) •(x i -b i )(4) That is, we average over m gradients, with the inputs to q being linearly interpolated between the baseline b and the original input x in m steps. We then take the dot product of that averaged gradient with the input embedding x i minus the baseline.We use a zero vector baseline (Mudrakarta et al., 2018) and five steps. The contrastive case becomes:S C IG = S t IG ||S t IG || - S f IG ||S f IG || (5)5 Results and Discussion 
the out-of-distribution problem: 
 Why should we use counterfactuals instead of erasing important tokens or replacing them with unimportant ones? First, we demonstrate that our counterfactual generators produce indistribution text for the predictor. Second, we show that the rankings of attribution methods' faithfulness are consistent when using a counterfactual generator for token replacement, but these rankings differ when other replacement methods are used.To achieve our first goal-demonstrating that the generated counterfactuals are in-distribution-we employ an out-of-distribution (OOD) detection technique to measure the percentage of our generated inputs that are OOD. Prominent OOD detection methods use a threshold, considering any input with a value higher than this threshold as OOD (Chen et al., 2023) . For each dataset, we calculate the threshold by measuring the negative loglikelihood (NLL) of 200 original examples using different predictors (fine-tuned and instruct-tuned) and consider the 99th percentile of these NLLs as the OOD threshold. We use NLL to detect OOD because the type of shift we aim to detect is background shift. OOD data can be classified as either semantic or background shift (Arora et al., 2021) . Semantic features have a strong correlation with the label, and semantic shift occurs when we encounter unseen classes at test time. In contrast, background features consist of population-level statistics that do not depend on the label and focus on the style of the text.In evaluating faithfulness by corrupting the input, we do not introduce new labels or classes; instead, we change the style of the text. Therefore, we aim to detect background shift. There are two common types of OOD detection methods: calibration and density estimation. Density estimation methods, such as perplexity (PPL), outperform calibration methods under background shifts, while the opposite is true under semantic shift. We use NLL, which is closely related to PPL.An attribution method shows us which tokens are important, and we replace those tokens in four ways: (i) using an editor to replace the tokens (our method), (ii) using tokens that are considered semantically unimportant (the <unk> token and the <mask> token), (iii) erasing the tokens, and (iv) zeroing out the attention mask for important tokens without altering the text itself (att-zero).The baselines (ii) through (iv) are similar to previous work (Hase et al., 2021) . Table 1 shows that for both fine-tuned and instruct-tuned predictors, the generated counterfactuals are mostly indistribution. Specifically, we present results for Chen et al. ( 2023) demonstrate that fine-tuning renders a model insensitive to non-semantic shifts. Their research indicates that fine-tuning eliminates pre-trained, task-agnostic knowledge about general linguistic properties, which is crucial for detecting non-semantic shifts. Our findings align with these observations. When a predictor is fine-tuned for a specific classification task, such as sentiment analysis on the SST-2 dataset, it is optimized to assign high probabilities to the correct labels for the training data. Consequently, this fine-tuned model becomes less sensitive to input corruptions. In our experiments, regardless of the replacement method employed, the resulting inputs tend to remain indistribution for the fine-tuned predictors. As evidenced in Table 1 , under the Gemma-ft columns, the percentage of out-of-distribution (OOD) examples approaches zero.In contrast, Gemma-it, an off-the-shelf model that is not optimized for a specific dataset, exhibits different behavior. When subjected to various input modifications-such as replacing important tokens with semantically neutral ones (e.g., <unk> or <mask> tokens), completely removing tokens, or zeroing out the attention mask for important tokens without altering the text itself-the Gemma-it predictor frequently categorizes these modified inputs as OOD. This disparity in behavior between fine-tuned and off-the-shelf models underscores the impact of task-specific optimization on a model's sensitivity to input perturbations. However, when the counterfactual generator is used to modify the inputs, the examples remain in-distribution even for the instruct-tuned predictor. This observation demonstrates that when we do not want to change the predictor model and prefer to use an off-theshelf model as our predictor, using a counterfactual generator is helpful in evaluating the faithfulness of attribution methods.To achieve our second goal-demonstrating the consistency of the faithfulness rankings of attribution methods when using a counterfactual generator, and the lack of consistency when another replacement method is applied-we use Spearman's rank correlation, as in previous works (Rong et al., 2022) . For each example, we rank the attribution methods based on the percentage of the mask needed to flip the label. We then compute the correlations among these rankings across all seven replacement methods (our three editors, Erase, <unk>, <mask>, and att-zero) and average the results over 200 examples.We present this analysis for the SST-2 dataset in Figure 4 . Other datasets yield similar results and are shown in Appendix B. In the top correlation matrix of Figure 4 , these average correlations are shown for the fine-tuned predictor. For the fine-tuned predictor, all replacement methods have high average correlations with each other. The middle matrix in Figure 4 shows these correlations when the predictor model is an off-the-shelf instruct-tuned model. For the off-the-shelf predictor, only when a counterfactual generator is used do the rankings have high correlations with each other; other replacement methods have low correlations with the counterfactual generators. This is likely because, when using an instruct-tuned predictor, replacement methods other than counterfactual generators create OOD inputs.The bottom matrix of Figure 4 displays the difference between the first and second matrices. It shows that the correlation difference between fine-tuned and instruct-tuned predictors is near zero when using editors as the replacement method. However, the difference is significant when using other replacement methods (<unk>/Erase/<mask>/att-zero). This suggests that when evaluating explanations on an off-the-shelf instruct-tuned model, it is crucial to avoid using corrupted OOD text.In Tables 2 and 3 , we show the average masking percentage required (the average percentage of tokens the counterfactual generator should change) to flip the label for fine-tuned and instruct-tuned predictor models, respectively. The masking percentage is highly correlated with the flip rate-the percentage of labels each counterfactual generator is able to flip by altering the corrupted tokens. In Appendix C, we show the flip rate for both finetuned and instruct-tuned predictor models. Attribution methods that can flip the labels with less masking (i.e., fewer changes) are also able to flip more labels.For the fine-tuned predictor (Table 2 ), gradient norm methods consistently outperform others on the SST-2 and IMDB datasets. In contrast, for AG-News, the Erasure method consistently performs the best or near the best. Our results suggest that straightforward methods, such as gradnorm1, grad-norm2, and Erasure, consistently deliver superior performance regardless of the editor used.For the instruct-tuned predictor (Table 3 ), the Erasure method yields the best results for the SST-2 dataset, while gradinp demonstrates the best performance on the IMDB dataset. However, no attribution method consistently outperforms random selection for the AG-News dataset. Overall, these findings suggest that attribution methods are less effective when the model is not fine-tuned for the specific task, indicating the need for cautious application of these methods to pretrained and instructtuned language models. 
conclusion: 
 In this work, we designed a faithfulness evaluation protocol based on counterfactual generation. We demonstrated that the efficacy of attribution methods varies between models fine-tuned on our specific dataset and off-the-shelf, instruct-tuned models. We showed that counterfactual generators are effective for evaluating feature attribution because they can produce mostly in-distribution text for the predictor model. This approach allows us to separate the evaluation of the model from the evaluation of the attribution method, as the examples used are mostly in-distribution. We also found high consistency between different counterfactual generators and a lack of consistency with other replacement methods, highlighting the importance of being in-distribution, particularly when evaluating attributions on off-the-shelf models. Finally, we used our protocol to compare different attribution methods. 
limitations: 
 Our work is limited in several aspects: First, we rely on generating counterfactuals, which requires a strong generative model. Generating counterfactuals-especially for long sequences-is computationally expensive. Second, the counterfactual generator might unintentionally incorporate the artifacts and shortcuts used by the predictor to flip the label, potentially limiting the intended application of our approach.Third, we applied our protocol only to classification tasks; evaluating it on other tasks, like translation, is left for future work. The bottom matrix illustrates the difference between the fine-tuned and instruct-tuned models, indicating that when editors are used as the replacement method, the difference in correlation is near zero. In contrast, using other replacement methods (i.e., <unk>, erase, <mask>, att-zero) results in significant inconsistencies between the two predictor types, likely due to the creation of out-of-distribution (OOD) text for the instruct-tuned model. 
 
 Appendix: Table 4 is the OOD percentages for other attribution methods in SST-2 dataset that were not in Table 1 . Tables 5 and 6 show OOD percentages in AG-News dataset.Figures 5 and 6 show the difference of correlations in IMDB and AG-News datasets respectively.Tables 7 and 8 show flip-rates for fine-tuned and instruct-tuned predictor models respectively. The code is available at https://github.com/ Sepehr-Kamahi/faith 


Decision: The paper introduces a novel method for evaluating the faithfulness of feature attribution methods for language models, based on a incremental masking procedure that measures the amount of tokens that need to be corrupted in order to flip a model prediction. The reviewers commend the novelty of this method, as well as its evaluation with respect to other well known attribution methods. The authors are encouraged to incorporate a reflection of the weaknesses pointed out by reviewers 5Cwj and Kqp4 in the revised version of the paper.

Reviews: review: paper_summary: paper_summary: This paper expands the traditional faithfulness evaluation method for attribution with a counterfactual generator. The authors argue that the current faithfulness evaluation often involves text replacement methods that create OOD inputs. The counterfactual generator ensures in-distribution input, and could help to boost the consistency across different attribution methods.
strength_weakness: strength_weakness: 1. The literature regarding counterfactual generation and evaluation in LLMs is missing: 
e.g. Qin, L., Bosselut, A., Holtzman, A., Bhagavatula, C., Clark, E., & Choi, Y. (2019). Counterfactual story reasoning and generation. arXiv preprint arXiv:1909.04076.
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35, 17359-17372.
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021). Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9, 160-175.
Li, J., Yu, L., & Ettinger, A. (2023). Counterfactual reasoning: Testing language models' understanding of hypothetical scenarios. arXiv preprint arXiv:2305.16572.

2. I am concerned about the quality of the text generated by counterfactual generator. For instance, the most intuitive counterfactual by masking “clear” in "a clear sense of purpose" would be "an unclear sense of purpose", but the current generator might create text like "a purposeful sense of purpose". If I am understanding it correctly, the current counterfactual generator shown in Fig 3 seems to be more a like a conventional unmasker, the training example does not have a contrastive original-counterfactual pair, and the only hint comes from the token "<counterfactual>". 

3. I am not sure if I have fully understand the editor model in the attribution evaluation process. Does it only perform masked token prediction, or does it jointly predict the label for generated counterfactual texts? I assume that the label for the generated counterfactuals would also be predicted, but it is not clear to me where this step is being done from Fig 2.
questions: questions: Line 179 attributions -> Attributions
Line 132 hence -> Hence
Line 71 wrong citation format for Hase et al. 2021.
paper_summary: paper_summary: In this manuscript, the authors propose an algorithm to evaluate explanations of LLMs by leveraging datasets of counterfactuals. Specifically, the algorithm fits a counterfactual generator model which, when given an instance and a mask of possible tokens to perturb, attempts to generate counterfactual by perturbing said mask. The higher the success rate, the higher the importance of said mask.
strength_weakness: strength_weakness: 1. Some claims ought to be better justified, see observations below
questions: questions: - "We use counterfactual generators to change the input, focusing on the important tokens specified by the attribution methods, and make sure that the input to the model is natural, fluent, and in-distribution." How do you make sure?
- "We argue that if an attribution method helps the counterfactual generator to change the model’s prediction with fewer changes, that method knows more about the model’s inner workings, which means it is more faithful." I think this needs a justification.
- Row 242, a fragment?
paper_summary: paper_summary: This paper proposes a new method to evaluate the faithfulness of feature importance (FI) methods.
Given a **predictor model** to be analysed and an input $x$, first run a feature importance method on it; this process outputs a list of which tokens in $x$ are most important for a prediction (and in which order).
Second, mask the top 10% tokens in $x$, and use a second **editor** model to create a counterfactual input $x’$ which should have an alternative label.
If the predictor model’s output changes on $x’$, stop; else, iteratively also mask the following top 10% tokens (according to the FI method), until labels change or 50% of tokens have been masked.
A better FI method should require changing less tokens to  flip the predictor model’s predictions.


Editor model training. Given a dataset of inputs + labels $(x, y)$, mask from 5 to 50% of the tokens in $x$, than create a sequence `mask(x) y <counterfactual> x`  and train an autoregressive editor model to reconstruct $x$ this way. This autoregressive model is initialised from a pretrained model, such as Pythia.

The paper evaluates their faithfulness evaluation protocol against prior methods using two experiments:
* The number of out-of-distribution inputs these faithfulness evaluation protocols generate. Their method generates less out-of-distribution inputs than other methods, at least when analysing instruction tuned models (results for fine-tuned models are mixed).
* The correlation between the rankings output by different faithfulness evaluation protocols. They show that their method, when using an editor model based on different pretrained models, leads to rankings which correlate with one another. Prior methods do not correlate among each other.

The paper then evaluates a number of FI methods using their metric, showing that simple methods—such as gradient norm—often over perform more complex methods.
strength_weakness: strength_weakness: One way in which the paper evaluates their method is by showing the correlation between rankings using different faithfulness evaluation protocols. However, I do not think this is sufficiently motivated. Why should a high correlation in this experiment represent a better method? To my understanding, a high correlation between rankings when starting the editor model with different pretrained models only shows that the method is robust to this choice, but not necessarily that this is a better method.

The paper evaluates the presence of out-of-distribution inputs according to their negative likelihood. However, to the best of my knowledge, likelihood only correlates weakly with text quality and how out-of-distribution it is (Holtzman et al. 2020). Running human evaluations could increase confidence in these results.


* Holtzman et al. (2020). The Curious Case of Neural Text Degeneration
questions: questions: * Parenthetical and inline citations are used incorrectly throughout the text. E.g., in line 293 you should have a parenthetical citation: “We use Gemma-2b Team et al. (2024) as the predictor.”
* In sections 4.3.1 and 4.3.2, the FI methods are given as a function of a specific feature $x_i$ (which I believe represents each token in the input). Sections 4.3.3, 4.3.4 and 4.3.5 do not have an $x_i$ input, however. Further, how per-token scores are computed for 4.3.4 is not clear in my opinion.