The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models (2024)

Abstract: In order to oversee advanced AI systems, it is important to understand their underlying decision-making process. When prompted, large language models (LLMs) can provide natural language explanations or reasoning traces that sound plausible and receive high ratings from human annotators. However, it is unclear to what extent these explanations are faithful, i.e., truly capture the factors responsible for the model’s predictions. In this work, we introduce Correlational Explanatory Faithfulness (CEF), a metric that can be used in faithfulness tests based on input interventions. Previous metrics used in such tests take into account only binary changes in the predictions. Our metric accounts for the total shift in the model’s predicted label distribution, more accurately reflecting the explanations’ faithfulness. We then introduce the Correlational Counterfactual Test (CCT) by instantiating CEF on the Counterfactual Test (CT) from Atanasova et al. (2023). We evaluate the faithfulness of free-text explanations generated by few-shot-prompted LLMs from the Llama2 family on three NLP tasks. We find that our metric measures aspects of faithfulness which the CT misses.

heading: 1 Introduction
text: In many applications of ML systems it is important to understand why the system came to a particular answer (Rudin, 2018), and the field of explainable AI attempts to provide this understanding. However, relying on subjective human assessment of explanations can be misleading: humans sometimes prefer interpretability techniques that provide little information about model predictions (Adebayo et al., 2018). It is therefore important to clearly assess the extent to which explanations inform us about ML systems, both for current high-stakes applications such as medicine and criminal justice (Rudin, 2018), as well as potential scenarios involving highly general systems (Shah et al., 2022; Ngo
et al., 2023; Ward et al., 2023). If we can ensure that explanations are faithful to the inner-workings of the models, we could use the explanations as a channel for oversight, scanning them for elements we do not approve of, e.g. racial or gender bias, deception, or power-seeking (Lanham, 2022).
We make the following contributions:
1. We argue that in order to be informatively faithful, it is not enough to test whether explanations mention significant factors: we also need to test whether they mention significant factors more often than insignificant ones.
2. We introduce Correlational Explanatory Faithfulness (CEF), a novel faithfulness metric that improves upon prior work by capturing both the degree of impact of input features, as well as the difference in explanation mention frequency between impactful and non-impactful factors.
3. We introduce the Correlational Counterfactual Test (CCT), where we instantiate CEF on the Counterfactual Test (CT) from Atanasova et al. (2023) and use statistical distance between predictions to measure impact.
4. We run experiments with the Llama2 family of LLMs on three datasets and demonstrate that CCT captures faithfulness trends that the existing faithfulness metric used in CT misses.

heading: 2 Related Work
text: There has been much discussion on what it means for an explanation to be “faithful”. Jacovi and Goldberg (2020) survey literature on the term and define an explanation as faithful insofar as it “accurately represents the reasoning process behind the model’s prediction”. Wiegreffe and Marasović (2021) review datasets for explainable NLP and identify three predominant classes of textual
530
explanations: highlights (also called extractive explanations ), free-text (also called natural language explanations or NLEs), and structured. Prior work on faithfulness has mostly focused on highlights and NLEs. We chose to focus on NLEs in this work because highlight-based explanations are highly restrictive in what they can communicate (Camburu et al., 2021; Wiegreffe et al., 2020), while NLEs allow models to produce justifications that are as expressive as necessary (e.g. they can mention to background knowledge that is not present in the input but that the model made use of for its prediction). Moreover, there is increasing work on NLEs in high-stakes areas, such as healthcare (Kayser et al., 2022), where having faithful explanations is crucial.
Parcalabescu and Frank (2023) review a range of recent NLE faithfulness tests and claim that many are instead measuring “self-consistency”. See Appendix C for further discussion.

heading: 2.1 “Explanatory” vs. “Causal” Faithfulness
text: We identify two types of faithfulness being researched in the literature, which we refer to as “explanatory” and “causal”. Explanatory faithfulness asks the question: does the explanation reflect the decision-making process of the model? This is often measured by intervening on the input, such as with the metrics of sufficiency and comprehensiveness for highlight-based explanations (DeYoung et al., 2019; Camburu et al., 2021), or the counterfactual test (CT) for NLEs (Atanasova et al., 2023). Causal faithfulness adds the criterion: does the model’s prediction causally depend on the generated reasoning trace? (Creswell and Shanahan, 2022; Lanham et al., 2023; Radhakrishnan et al., 2023; Turpin et al., 2023) Causal faithfulness requires structural restrictions on the prediction system (at a minimum, that the explanation is generated before the prediction), such as in chain-of-thought (Wei et al., 2023) or selectioninference (Creswell et al., 2022). Explanatory faithfulness, however, can be measured for a more general class of rationales, including post-hoc explanations (DeYoung et al., 2019; Atanasova et al., 2023). We focus on explanatory faithfulness in this work; see Appendix A for further discussion of causal faithfulness.
Some authors also distinguish between “explainability” and “interpretability/transparency” as approaches for understanding models (e.g. Rudin (2018)). While the concept of faithfulness is appli-
cable to both approaches, we primarily focus on “explainability” in this work. See Appendix B for further discussion.

heading: 2.2 The Counterfactual Test
text: In order to measure whether an explanation captures the true factors responsible for a model’s prediction, we need to know which factors are relevant. However, deep neural networks like LLMs are often difficult to interpret (Fan et al., 2020).
To address this problem, Atanasova et al. (2023) introduce the Counterfactual Test (CT). The CT inserts some text into an input query, which we refer to as an interventional addition (IA). If the model’s prediction changes, then the IA was relevant to the model’s new prediction, and we check if it is mentioned in the new explanation. Counterfactual edits have the advantage of easily generating features that we know are relevant to the model’s prediction. We choose to focus our analysis on this method, and identify ways to improve it.

heading: 3 Methods
text: We identify two significant drawbacks with the CT:
1. It does not test whether impactful features are more likely to be mentioned than less impactful ones. There is a trivial strategy that leads to 0% unfaithfulness as measured by the CT: repeat all input t ext verbatim as the explanation, which means explanations will never fail to mention the IA. This demonstrates an important property of useful explanations: they are useful only if they both mention impactful features and leave out non-impactful features.
2. It measures impactfulness as binary, i.e. whether the intervention results in a change in the model’s top predicted label. But this ignores changes in the model’s predicted class likelihoods: it would label an intervention that changes the predicted probability of a class from 49% to 51% as relevant, while an intervention that changes the probability from 1% to 49% would be labelled as irrelevant, even though the latter caused a larger shift.
To address these drawbacks, we propose the metric Correlational Explanatory Faithfulness (CEF), which can be applied to any tests with three given properties:
1. An intervention: a function mapping an input example to its modified version.
2. A prediction impact measure: a function mapping an input example, intervention, and model to a scalar representing how impactful the intervention was on the model’s prediction. We call the output of this function the prediction impact or I.
3. An explanation mention measure: a function mapping an input example, intervention, and explanation to a scalar representing the extent to which the explanation attributes importance to the intervened factors. We call the output of this function the mention importance or M.
If an intervention has higher prediction impact, a faithful explanation should assign it higher mention importance. We quantify this relationship by measuring the Pearson correlation coefficient between prediction impact and mention importance:
CEF = ∑n i=0 ( Ii − I ) ( Mi −M ) √∑n
i=1
( Ii − I )2√∑n i=1 ( Mi −M )2 (1)
where x = 1n ∑n
i=1 xi (the sample mean). Being a correlation, it lies in the interval [−1, 1], with 0 indicating no relationship and positive values indicating higher mention importance for more impactful interventions.
We can then apply this metric to the CT, which gives us the Correlational Counterfactual Test (CCT). In our work, the intervention inserts an IA. To quantify the degree of prediction impact in a continuous manner, we measure the total shift in the model’s predictions due to the IA. There are a number of ways to measure shifts in probability distributions over discrete classes; we use the total variation distance (TVD), i.e:
TVD(P,Q) = 1
2
∑
x
|P (x)−Q(x)| (2)
where P and Q are probability distributions over discrete classes. We take P and Q to be the model’s predicted distributions before and after the intervention, so that TVD measures the absolute change in probabilities assigned to each class. Compared to other common statistical distances such as the relative entropy (KL divergence), TVD gives less weight to shifts between very small probabilities (which are unlikely to impact classification) and has the advantage of symmetry.
To measure mention importance, we use the original CT’s binary metric: does the explanation mention the word? Note that in this case our metric represents the point-biserial correlation, a special case of the Pearson correlation coefficient where one variable is continuous and the other is dichotomous. We can then write CCT as:
CCT = EM (TVD)− E¬M (TVD) STD(TVD)
√ |M ||¬M |
|M ∪ ¬M |2 , (3)
where M indicates that the explanation mentions the IA, and |M | indicates the number of examples with explanation mentions. For the binary mentions we study, CCT is maximized when explanations mention IAs exactly when their TVD is above a certain threshold (where the threshold depends on the distribution of TVDs). Table 1 shows an example application of our method. Future work could explore the case where explanations can assign weights to different features. We test alternatives to TVD and CCT in Appendix F.
CCT addresses the mentioned drawbacks of the CT. Unlike the CT, it cannot be trivially gamed:
achieving maximum correlation requires explanations to mention impactful IAs while not mentioning non-impactful IAs, which requires a signal about which words are impactful.

heading: 4 Experiments
text: In this section, we describe our experimental setup. We first generate predictions and NLEs using LLMs on a set of three natural language classification tasks. We then study the faithfulness of these NLEs, comparing the CT and CCT.

heading: 4.1 Datasets
text: Following Atanasova et al. (2023), we evaluate on three popular classification datasets including human-written NLEs:
e-SNLI (Camburu et al., 2018): Sentence pairs labeled with entailment, contradiction, or neutral.
ComVE (Wang et al., 2020): Sentence pairs where one violates common sense.
ECQA (Aggarwal et al., 2021): Multiple choice common sense questions with 5 options each.
We use ECQA in place of CoS-E (Rajani et al., 2019) as a more recent dataset also based on CQA with more detailed explanations that both justify the correct answer and refute the incorrect answers. Note that the ground-truth NLEs are not necessarily faithful explanations for an LLM: there may be multiple equally valid justifications for a groundtruth label on an instance (e.g., multiple reasons why two sentences are contradictory), or the LLM could rely on other reasoning, such as spurious correlations. We use the original train/test splits and evaluate on test sets, containing 9,842 (e-SNLI), 2,194 (ECQA), and 999 (ComVE) examples.

heading: 4.2 Models and Prompts
text: We use the Llama-2 series of LLMs (Touvron et al., 2023). We focus on the few-shot imitation setting: we use the pretrained foundation models (Llama2-7B, Llama-2-13B, and Llama-2-70B) prompted with a brief description of the dataset followed by 20 randomly selected examples from the training set including label and explanation. When prompting the model, we can have it generate NLEs either after its prediction, as an explanation conditioned on the prediction (predict-then-explain, PE), or before the prediction, which is conditioned on the explanation (explain-then-predict, EP)1 (Camburu
1Using this terminology, chain-of-thought (Wei et al., 2023) is EP.
et al., 2018). We provide full example prompts in Appendix G. When generating text with these models, we use greedy sampling to reduce variation during evaluation. However, we still record the probabilities assigned to tokens corresponding to predicted classes, which we use for computing the TVD.

heading: 4.3 Counterfactual Interventions
text: We use the random intervention proposed in Atanasova et al. (2023): we insert a random adjective before a noun or a random adverb before a verb, randomly selecting 4 positions where we insert the said words, and for each position selecting 20 random candidate words. The candidates are chosen from the complete list of adjectives or adverbs available in WordNet (Fellbaum, 2010), and nouns and verbs are identified with spaCy (Orosz et al., 2022) using the model "en_core_web_lg". In order to avoid highly unnatural sentences, we use an instruction-tuned LLM, Llama-2-70b-chat, to identify interventions that the model judges as not making sense, and keep only the top 20% of interventions for each example (prompt shown in subsection G.4). See Appendix E for examples of interventions and their effect on model predictions and explanations. We determine whether an explanation includes an IA by case-insensitive substring matches, either on the original strings or stemmed versions (Porter, 2001).
For each model, prompting strategy (PE vs. EP), and dataset, we first run the model on each example in the test set and measure its predicted class probabilities. Next, we perform counterfactual interventions on each example and re-run the model on each intervention. Using TVD to measure impactfulness, we can study whether explanations are more likely to mention IAs that are more impactful, and compare the CT and CCT.

heading: 5 Results
text: Figure 1 plots intervention importance as measured by TVD vs. the fraction of the time that IAs are mentioned in explanations. A model with faithful explanations should show an upward trend in mentions, being more likely to mention highly impactful IAs than less impactful IAs. We note that while explanation mentions for e-SNLI show a clear upward trend, ECQA has a relatively flat trend: most ECQA explanations mention IAs, but they are not much more likely to mention highly impactful IAs
than non-impactful ones. This may be because they tend to be verbose and repeat large portions of their inputs, as can be seen frm the examples on Table 4.
Table 2 shows the quantitative results of our experiments. Classification accuracy before intervention is above random for all models and datasets (except possibly Llama2-7B on ComVE), indicating that the models are capable of performing some aspects of the tasks. Note that ECQA explanations have the lowest CT unfaithfulness of any dataset, i.e. they frequently mention IAs which cause predictions to change. But Figure 1 shows that this is misleading: ECQA explanations succeed in frequently mentioning impactful IAs because they frequently mentions any IAs; the fact that a word appears in an ECQA explanation gives little signal about whether that word was impactful or not for the model’s prediction.
The CCT is more informative of the qualitative results from Figure 1 than CT: model explanations provide more information about the relevance of
IAs for e-SNLI than for ECQA, and are thus more faithful. Additionally, we see that the largest model, Llama2 70B, produces the most faithful explanations on e-SNLI and ComVE.

heading: 6 Summary and outlook
text: We introduced Counterfactual Explanatory Faithfulness and the Correlational Counterfactual Test, allowing us to measure how informative explanations are about the importance of the factors they mention. Model explanations are more likely to mention inserted words when they’re more impactful to the model’s predictions, suggesting a degree of faithfulness on these tasks which increases with model size. However, there is significant variation between datasets, which could be due to either the nature of the task or the annotator-provided explanations. Future work could apply the CCT to instruction-tuned models, as well as explanations generated using strategies such as question decomposition (Radhakrishnan et al., 2023).
Limitations
While our analysis identifies and corrects some shortcomings of prior work on measuring the faithfulness of NLEs, it does inherit some of the limitations of the original CT (Atanasova et al., 2023). The counterfactual interventions only insert adjectives and adverbs, and only single words at a time, so our experiments do not measure sensitivity to other parts of speech. Our random intervention can generate text which lacks semantic coherence, despite our LLM filtering step. We do not test for synonyms, which could inaccurately label some explanations. Additionally, we do not consider the semantic usage of word mentions: for example, our metrics would not penalize the faithfulness of illogical explanations as long as they had the correct pattern of word inclusion. Some of these drawbacks could potentially be addressed by further filtering or analysis by more advanced LLMs, taking advantage of their semantic understanding.
We study LLMs generating predictions and explanations using few-shot prompting, with example explanations taken from human-generated NLEs. These explanations can be highly dependent on annotation instructions. For example, CoS-E (Rajani et al., 2019) and ECQA (Aggarwal et al., 2021) both use CQA (Talmor et al., 2019) as a base dataset, but ECQA explanations are significantly longer than those for CoS-E. As such, care should be taken when extrapolating our results to other tasks: in the few-shot setting, the example explanations provided can have just as much impact on faithfulness as the model being used.

heading: Acknowledgements
text: We would like to thank Zac Kenton for feedback on a draft of this paper. Oana-Maria Camburu was supported by a Leverhulme Early Career Fellowship. The work of Perez-Ortiz was partially supported by the European Commission-funded project “Humane AI: Toward AI Systems That Augment and Empower Humans by Understanding Us, our Society and the World Around Us” (grant 952026).

heading: A "Causal" vs. "Explanatory" Faithfulness
text: Rather than generating post-hoc explanations, there have been calls to instead build interpretability into the prediction process, where the prediction causally follows from the explanation (Rudin, 2018; Chattopadhyay et al., 2023). In the context of LLMs, this can be done by having models generate chains-of-thought (CoT) (Wei et al., 2023), a series of intermediate reasoning steps before producing their prediction. In addition to improving final task accuracy, this has been hypothesized to be a way to improve faithfulness: rationales may be more likely to accurately represent a model’s true reasoning process if they are generated first, so that they can inform the prediction (Lanham, 2022; Camburu et al., 2018). However, it has been shown that even if reasoning is generated before the prediction, it may still be unfaithful (Turpin et al., 2023; Atanasova et al., 2023). Work on CoT has often focused on measuring (Lanham et al., 2023) and increasing (Radhakrishnan et al., 2023) the degree to which the model’s final answer depends on its reasoning (i.e. the extent to which editing or removing reasoning steps change the model’s answer). Studying faithfulness and causal dependence in reasoning tackle complementary ideas, and we believe there are reasons to measure them separately:
1. It may be difficult to ensure reliance on CoT reasoning for some tasks: Lanham et al. (2023) found relatively minor accuracy gains from CoT outside of math-focused domains. In particular, as models become more powerful, they may be capable of solving increasing sets of tasks without verbalised CoT.
2. Causal dependence alone doesn’t ensure the usefulness of an explanation: models could use language in ways different from humans, either unintentionally (e.g. semantic drift) or as a result of some optimization pressure (e.g. steganography Roger and Greenblatt (2023)). Separate from causal dependence, it will still be necessary to measure whether the textual content of reasoning provides useful information on the factors leading to the model’s prediction.
B “Explainability” vs. “Transparency/Interpretability”
There isn’t currently a clear consensus on the usage of the terms “explainability” and “interpretability”: they are sometimes used interchangeably (e.g. Jacovi and Goldberg (2020)), while other times a distinction is made between “interpretability” or “transparency” involving the creation of systems constrained in model form so its inner mechanics can be observed and understood, and “explainability” involving the creation of auxiliary models to explain an existing black-box model (e.g. Rudin (2018)). Marcinkevics and Vogt (2020) also survey some existing usages of these terms.
Because “interpretability” is used in these different ways, when discussing this distinction, we’ve found it least ambiguous to refer to the two sides as “explainability” and “transparency”.
The definition of faithfulness we adopt is that an explanation is faithful insofar as it “accurately represents the reasoning process behind the model’s prediction” (Jacovi and Goldberg, 2020). Under Rudin (2018)’s distinction, both transparent systems and explainable systems can in principle be faithful if their explanations accurately represent the model’s reasoning process. However, explainable systems in particular are at risk of post-hoc rationalization: producing explanations that sound plausible to humans but that don’t capture the true features that led to the prediction. This is our motivation for introducing improved metrics for faithfulness in explanations.
C “Faithfulness” or “Self-Consistency”?
Recent work (Parcalabescu and Frank, 2023) has argued that many metrics claiming to measure “faithfulness” (including the Counterfactual Test (Atanasova et al., 2023)) are in fact only measuring a weaker property, which they refer to as “selfconsistency”, because these tests fail to take into account mechanistic inner workings.
However, we still believe it is useful to refer to these tests as faithfulness metrics rather than self-consistency tests. Using Jacovi and Goldberg (2020)’s definition of faithfulness, if we intervene on an input and the model’s output distribution changes, we have learned a property of the model’s true reasoning process, i.e. that it depends on the intervened input in the current context. We can then measure the extent to which the explanation reflects this dependency, as in our proposed test.
Additionally, a test being mechanistic is not a guarantee of its robustness. Parcalabescu and Frank (2023) argue that “a test that is able to interrogate a model’s inner workings would be akin to a lie detector that uses more internal cues that cannot be easily suppressed”. Indeed, this has been the motivation for some prior approaches: Burns et al. (2022) proposed Contrast Consistent Search, a test using internal model activations to detect when a model gives an answer it “knows” is untrue. However, later work found that this method often identifies spurious non-knowledge-related features (Farquhar et al., 2023). Robustly measuring faithfulness may require a combination of tests, both mechanistic and black-box.
D Intervention Impactfulness with Explain-then-Predict
Figure 2 shows intervention impactfulness vs. explanation mention measure, equivalent to Figure 1 but in the Explain-then-Predict (EP) setting.

heading: E Example Interventions
text: In this section we show randomly selected examples of interventions on the three datasets, as well as model responses. For each example, we show the original problem and resulting prediction first, followed by the modified problem and predictions with the IA highlighted in red. We also highlight any mentions of the IA in the model’s explanation for the modified problem. For conciseness we show only the case of Llama2 70B using predict-thenexplain prompting. See Table 3 for interventions on e-SNLI, Table 4 for interventions on ECQA, and Table 5 for interventions on ComVE.

heading: F CCT Variants
text: We chose to use TVD as our distance metric because it gives less weight to shifts between very small probabilities (which are unlikely to impact the classification decision), and we chose to use Pearson as our correlation coefficient because it takes cardinality into account, unlike rank correlation coefficients which only use ordinality. However, our approach can also be computed using other choices of distance and correlation.
We can compute our metric in the predict-thenexplain setting under two other plausible configurations: CCT (Jensen-Shannon) using JensenShannon divergence, a symmetric divergence based
on KL) in place of TVD, and CCT (Spearman) using Spearman’s rank correlation in place of Pearson. Table 6 shows our results.
These variants show similar qualitative trends, with the highest values assigned to e-SNLI explanations, lower values for ECQA and comVE, and slightly more faithful explanations for the largest model (except for CCT (Spearman) EP, where the 13B model has the highest value).

heading: G LM Prompts
text: In this section we describe the prompts we use. Each few-shot prompt consists of three parts: the prefix describes the format verbally; 20 few-shot examples sampled uniformly without replacement from the training set, providing demonstrations of predictions and explanations; and the query, consisting of the input for a new problem instance to be evaluated. To avoid dependence on a single prompt sample, we independently sample new few-shot examples for each evaluation example. However, to ensure our word insertion interventions are the only thing changing model predictions, we use the same few-shot examples for the model’s prediction before and after interventions.
The following are randomly selected examples of prompts for each dataset. We show predict-thenexplain prompts; explain-then-predict prompts have the same format, with the only difference being that the order of the label and explanation lines is reversed and the query ends with "EXPLANATION:" rather than the label title.
G.1 e-SNLI Example Prompt
The following are examples from a dataset. Each example consists of a pair of statements, "TEXT" and "HYPOTHESIS". Each pair is labeled with a "JUDGEMENT": given the text, is the hypothesis definitely true ("entailment"), maybe true ("neutral"), or definitely false ("contradiction")? "EXPLANATION" explains why the selected judgement is chosen.
TEXT: a dog chases another dog. HYPOTHESIS: The dog is wanting to get the ball first. JUDGEMENT: neutral EXPLANATION: The dog may not be wanting anything. There may not be a
ball present to get first.
TEXT: A woman carried a cake ito the room with three candles as another woman holding a flute glass of wine, holds up her hand. HYPOTHESIS: Two women were celebrating. JUDGEMENT: neutral EXPLANATION: Eating a cake and drinking one doesn’t imply celebrating.
TEXT: A man in a wetsuit is surfing up and over a wave. HYPOTHESIS: A man is surfing over a wave. JUDGEMENT: entailment EXPLANATION: A man surfing would do so over a wave.
TEXT: Rugby players tackling each other. HYPOTHESIS: The rugby players are getting physical. JUDGEMENT: entailment EXPLANATION: Tackling is a very physical action.
TEXT: Some students saying prayer outside. HYPOTHESIS: A dog barks inside. JUDGEMENT: contradiction EXPLANATION: the dog is not students outside and the dog is inside.
TEXT: Three women are posing together and smiling while one holds up a hand signal. HYPOTHESIS: Two women are yelling at each other and pointing fingers. JUDGEMENT: contradiction EXPLANATION: There is either three women or two women.
TEXT: Three people are checking out a piece of art at the local museum. HYPOTHESIS: Three women are at a museum. JUDGEMENT: entailment EXPLANATION: Three people could be women and they are at a museum
TEXT: Four people are in a group hug near a soda machine. HYPOTHESIS: A group of friends in a huddle. JUDGEMENT: neutral EXPLANATION: a hug is not a huddle
TEXT: A young boy wearing black pants and a pinstriped shirt looks at something on a computer screen. HYPOTHESIS: A young boy is doing his homework on the computer. JUDGEMENT: neutral EXPLANATION: Looking at screen doesn’t imply doing homework.
TEXT: A man is rollerblading down a rail. HYPOTHESIS: There is a man rollerblading quickly. JUDGEMENT: neutral EXPLANATION: Not all people rollerblading are doing so quickly.
TEXT: Pedestrians strolling along a brick walkway tween high buildings. HYPOTHESIS: People walk through town. JUDGEMENT: entailment EXPLANATION: Strolling means casually walking while a simple "walk"
doesn’t have any connotation.
TEXT: a group of people sitting on the ground on the sidewalk HYPOTHESIS: A group of people sit around in a circle. JUDGEMENT: neutral EXPLANATION: Sitting on the ground does not have to be in a circle.
TEXT: A man with an arm cast films something on video while another man is looking at the camera.
HYPOTHESIS: The man does not have a cast. JUDGEMENT: contradiction EXPLANATION: The man can’t have a cast while not having a cast.
TEXT: Young woman in blue shirt checking out merchandise. HYPOTHESIS: The woman is shopping. JUDGEMENT: entailment EXPLANATION: One is shopping by checking out merchandise.
TEXT: A woman carries a young girl on her shoulders HYPOTHESIS: A woman carries her purse with her to the concert. JUDGEMENT: contradiction EXPLANATION: A woman can either carry a young girl or her purse at a
time.
TEXT: A man cooking in a restaurants. HYPOTHESIS: A lady is cooking in a restaurant. JUDGEMENT: contradiction EXPLANATION: A man and a lady are two different people.
TEXT: A white dog travels along a narrow path in a park setting. HYPOTHESIS: The animal is going along the path. JUDGEMENT: entailment EXPLANATION: The dog traveling is the animal going on the path.
TEXT: One guy wearing black shirt sitting at table working on computer project. HYPOTHESIS: There is a man indoors with a computer. JUDGEMENT: entailment EXPLANATION: Guy is a synonym for man. Working on a computer project
would likely require a computer.
TEXT: A man in blue shorts lays down outside in a parking lot. HYPOTHESIS: Nobody is laying. JUDGEMENT: contradiction EXPLANATION: A man is laying down so there is somebody laying.
TEXT: Girl running in a marathon, wearing a black shirt with a white tank top, with the numbers 44 on it. HYPOTHESIS: There is boy sitting at his house. JUDGEMENT: contradiction
EXPLANATION: a girl is not a boy and running is not sitting
TEXT: Two women are embracing while holding to go packages. HYPOTHESIS: The sisters are hugging goodbye while holding to go packages
after just eating lunch. JUDGEMENT:
G.2 ComVE Example Prompt
The following are examples from a dataset. Each example consists of a pair of sentences, "SENTENCE 0" and "SENTENCE 1". One of these sentences violates common sense. Each pair of these is labeled with "FALSE SENTENCE", followed by the label of the false sentence, 0 or 1. "EXPLANATION" explains why sentence is chosen.
SENTENCE 0: You can use a holding bay to store an item SENTENCE 1: You can use a holding bay to delete an item FALSE SENTENCE: 1 EXPLANATION: Deleting items is not a holding bay function
SENTENCE 0: Rainbow has five colors SENTENCE 1: Rainbow has seven colors FALSE SENTENCE: 0 EXPLANATION: The seven colors of the rainbow are red, orange, yellow,
green, blue, blue, and purple
SENTENCE 0: You are likely to find a cat in ocean SENTENCE 1: You are likely to find a shark in ocean FALSE SENTENCE: 0 EXPLANATION: Cats do not feed on ocean lives
SENTENCE 0: The caterpillar eats the rose bud SENTENCE 1: Roses buds eat caterpillars FALSE SENTENCE: 1 EXPLANATION: Caterpillars have mouths while rose buds don’t
SENTENCE 0: playing frisbee is for people who like to play frisbee SENTENCE 1: playing frisbee is for people who like to play football FALSE SENTENCE: 1 EXPLANATION: People avoid doing things they dislike so if they like play
frisbee they do that sport
SENTENCE 0: A recipe is great way to cook a gourmet meal and avoid minor mistakes in the kitchen. SENTENCE 1: Cooking gourmet meals is the number one way to make mistakes such as kitchen fires. FALSE SENTENCE: 1 EXPLANATION: Kitchen fires, and or mistakes are not a direct result of
cooking gourmet meals.
SENTENCE 0: Nail is a small piece of metal which is inserted into a lock and turned to open or close it SENTENCE 1: Key is a small piece of metal which is inserted into a lock and turned to open or close it FALSE SENTENCE: 0 EXPLANATION: Usually people use key to unlock a lock
SENTENCE 0: She put a Turkey in the oven. SENTENCE 1: She put a desk in the oven. FALSE SENTENCE: 1 EXPLANATION: A desk can not fit in a oven.
SENTENCE 0: A lemon has stripes. SENTENCE 1: A tiger has stripes. FALSE SENTENCE: 0 EXPLANATION: Lemons are yellow fruits.
SENTENCE 0: Burning trash purifies air quality. SENTENCE 1: Burning trash aggravates air quality. FALSE SENTENCE: 0 EXPLANATION: Burning trash will produce a lot of harmful gases and can’t
purify the air.
SENTENCE 0: my favorite thing is skiing in the lake SENTENCE 1: my favorite thing is boating in the lake FALSE SENTENCE: 0 EXPLANATION: a lake is not the right place for skiing
SENTENCE 0: He talked to her using a book shelf SENTENCE 1: He talked to her using a mobile phone FALSE SENTENCE: 0 EXPLANATION: Book shelves are for keeping books
SENTENCE 0: People are so glad to see the heavy smog in the winter morning SENTENCE 1: People are so glad to see the blue sky in the winter morning FALSE SENTENCE: 0 EXPLANATION: Smog is a kind of pollution, it makes people sad and angry
SENTENCE 0: A towel can not dry the water on your body SENTENCE 1: A towel can dry the water on your body FALSE SENTENCE: 0 EXPLANATION: Towels have a certain degree of water absorption.
SENTENCE 0: There are four mountains around the table SENTENCE 1: There are four stools around the table
FALSE SENTENCE: 0 EXPLANATION: Mountains need a great space and cannot be so close to a
table
SENTENCE 0: If I have no money, I would lent it to you SENTENCE 1: If I have any money, I would lent it to you FALSE SENTENCE: 0 EXPLANATION: He cannot lent money he doesn’t have
SENTENCE 0: people go to see a doctor because they fall ill SENTENCE 1: people go to see a doctor so they fall ill FALSE SENTENCE: 1 EXPLANATION: a doctor is meant to cure diseases
SENTENCE 0: Metro door is closing, please be quick SENTENCE 1: Metro door is closing, please step back FALSE SENTENCE: 0 EXPLANATION: People should step back and wait for the next train if the
door is closing
SENTENCE 0: There are many aliens in China. SENTENCE 1: There are many people in China. FALSE SENTENCE: 0 EXPLANATION: There aren’t aliens in the world.
SENTENCE 0: People usually go to bars for drinks SENTENCE 1: People usually go to bars for milk FALSE SENTENCE: 1 EXPLANATION: Bars mainly sell drinks
SENTENCE 0: A red lion will match that suit. SENTENCE 1: A red tie will match that suit. FALSE SENTENCE: 0 EXPLANATION: no one puts a lion on their clothes.
SENTENCE 0: I have two eyes SENTENCE 1: I have five eyes FALSE SENTENCE: 1 EXPLANATION: Usually, humans have two eyes
SENTENCE 0: drinking milk can help teenagers grow shorter SENTENCE 1: drinking milk can help teenagers grow taller FALSE SENTENCE: 0 EXPLANATION: it’s impossible for people to grow shorter
SENTENCE 0: She ate her ballet shoes. SENTENCE 1: She wore her ballet shoes. FALSE SENTENCE: 0 EXPLANATION: she cannot eat ballet shoes
SENTENCE 0: HE PUT HIS FOOT INTO THE SHOE IN ORDER TO TRY IT ON. SENTENCE 1: HE ALSO PUT HIS HAND IN THE SHOE TO SEE IF IT FITS. FALSE SENTENCE: 1 EXPLANATION: HANDS DON’T FIT WELL INSIDE OF SHOES.
SENTENCE 0: He poured orange juice on his cereal. SENTENCE 1: He poured milk on his cereal. FALSE SENTENCE:
G.3 ECQA Example Prompt
The following are examples from a dataset. Each example consists of a question followed by five multiple choice options. The option that makes the most sense as answer to the question is labelled as "CORRECT OPTION". "EXPLANATION" explains why the selected option is chosen.
QUESTION: The chief saw his entire tribe wiped out, he was a leader with a single what? OPTION 1: peon OPTION 2: indian OPTION 3: minister OPTION 4: follower OPTION 5: employee CORRECT OPTION: 4 EXPLANATION: Leaders have followers who are supporters unlike peon,
Indian or minister. Followers do not work for money while employees do.
QUESTION: The drive was full of obstacles, he really had to what? OPTION 1: listen to radio OPTION 2: get into vehicle OPTION 3: hole in one OPTION 4: sleep OPTION 5: pay attention CORRECT OPTION: 5 EXPLANATION: Drive full of obstacles really needs to pay attention from
driver.You cannot listen radio when the drive is full of obstacles as it may distract you. you cannot get into vehicle as you are already into the vehicle when driving.Hole in one is not things to do. You cannot sleep when the drive is full of obstacles as it may result in accident.
QUESTION: What can’t viruses do without infecting a host cell? OPTION 1: reproduce OPTION 2: make computer malfunction
OPTION 3: infect OPTION 4: hack computer OPTION 5: mutate CORRECT OPTION: 1 EXPLANATION: Viruses can’t reproduce instead of infecting a host cell.
Viruses can make a computer malfunction. Virus can infect. A virus can hack the computer system. Virus do mutate the system.
QUESTION: How might a automobile get off a freeway? OPTION 1: exit ramp OPTION 2: garage OPTION 3: driveway OPTION 4: repair shop OPTION 5: stop light CORRECT OPTION: 1 EXPLANATION: Exit ramp is the end of a freeway from where people get off
the freeway in their automobiles. All the other options are not from where automobiles get off freeways.
QUESTION: It was impossible to find a parking garage, so James took a bus whenever he wanted to go where? OPTION 1: neighbor’s house OPTION 2: car OPTION 3: building OPTION 4: restaurant OPTION 5: downtown CORRECT OPTION: 5 EXPLANATION: Downtown is or is relating to the central and main part of
a city. James takes a bus to go downtown since he wouldn’t find a parking garage. One won’t take a bus to go to his neighbor’s house and restaurants usually have a parking area. Building can be any building and a car is not a place to go to.
QUESTION: He made another call, he did this all day hoping people would what well to his offer? OPTION 1: hang up OPTION 2: respond OPTION 3: contact OPTION 4: answer OPTION 5: attracting ducks CORRECT OPTION: 2 EXPLANATION: A response could get an offer while contacting and
answering do not. Responding means answering unlike hanging up or attracting ducks.
QUESTION: Where are people likely to sing? OPTION 1: apartment OPTION 2: supermarket OPTION 3: train station OPTION 4: opera OPTION 5: conference CORRECT OPTION: 4 EXPLANATION: Opera is an ancient musical art form including theatrical
work. Opera includes singing. People usually sing at Opera. Apartment is not a common place where people sing. People do not sing at train stations. People do not sing at conferences of supemarkets.
QUESTION: What might people do to protect their legs from getting dirty on the farm? OPTION 1: wear jeans OPTION 2: milk cow OPTION 3: believe in god OPTION 4: avoid mud OPTION 5: plant flowers CORRECT OPTION: 1 EXPLANATION: People wear full clothing in order to avoid getting dirty.
Jeans is a full clothing for legs. People on farms wear jeans to protect their legs from getting dirty. Milking cow does not help in avoiding dirty legs. Believe in god is an irrelevant option. Avoiding mud does not always help in protecting legs from getting dirt on them. Plant flowers is an irrelevant option.
QUESTION: Where would you get a toothpick if you do not have any? OPTION 1: box OPTION 2: grocery store OPTION 3: eyes OPTION 4: chewing OPTION 5: mouth CORRECT OPTION: 2 EXPLANATION: You would get a toothpick from a grocery store because it
is available there. Box isnt a place from where youn can get a toothpick. Eyes or Chewing is not a place. You cant get a toothpick from mouth if you dont have any.
QUESTION: What is smaller than a country but larger than a city? OPTION 1: town OPTION 2: france OPTION 3: continent OPTION 4: state OPTION 5: metal CORRECT OPTION: 4 EXPLANATION: Country is a collection of states and state is a collection
of cities. So State is smaller than a country and larger than a city. Metal is not a place and all the other options are not smaller than a country and larger than a city.
QUESTION: With all the leaves falling each year, a natural compost keeps the soil healthy for all the trees where? OPTION 1: garden
OPTION 2: useful for recycling OPTION 3: surface of earth OPTION 4: forest OPTION 5: orchard CORRECT OPTION: 4 EXPLANATION: A natural compost keeps the soil healthy for all the trees
in a forest which is a large area covered chiefly with trees. Compost is decayed or decaying organic matter like leaves. A garden may or may not have trees. Useful for recycling is not a geographical place where trees exist. Trees do not exist across all surface of earth. Leaves of fruit trees in an orchard may or may not fall every year.
QUESTION: What must one be careful about when learning about science? OPTION 1: become educated OPTION 2: frustration OPTION 3: accidents OPTION 4: smiles OPTION 5: basketball CORRECT OPTION: 3 EXPLANATION: Accident is an unfortunate incident that happens
unexpectedly and unintentionally. One must be careful about accidents when learning about science. Become educated is not being careful of. Frustration is the feeling of being upset as one doesn’t get frustrated when learning about science. Smile is amused expression whereas being careful about smile is not necessary when learning about science. Basketball is not true as learning about science is not related with basketball.
QUESTION: Where can you learn about the anatomy of a blowfish in print? OPTION 1: cuba OPTION 2: fish market OPTION 3: books OPTION 4: france OPTION 5: canada CORRECT OPTION: 3 EXPLANATION: Anatomy exists in living beings including fishes and can be
accessed in books. Cuba, France and Canada are countries and are not material to be printed on. Fish market cannot be printed on.
QUESTION: If you ate some spicy food, what could happen to you? OPTION 1: medium OPTION 2: illness OPTION 3: throwing up OPTION 4: heartburn OPTION 5: sleepiness CORRECT OPTION: 4 EXPLANATION: spicy food causes you heartburn.Medium is not that can
happen to you.spicy food doesn’t cause illness or throwing up or sleepiness.
QUESTION: She let him know he was being over the top, and that his antics where a little what? OPTION 1: much OPTION 2: plenty OPTION 3: larger OPTION 4: lot of OPTION 5: big CORRECT OPTION: 1 EXPLANATION: The behaviour of the person was getting unbearble and a
little much signifies something excess beyond capacity. All the other options are either grammatically or contextually incorrect.
QUESTION: Where can a child learn about the adventures of a talking monkey? OPTION 1: rain forest OPTION 2: tropical areas OPTION 3: pet store OPTION 4: library OPTION 5: story book CORRECT OPTION: 5 EXPLANATION: Story books are books which are used for teaching children
about various things like talking monkeys. Both tropical area sand rain forest are wild areas which are not a thing to teach child. Pet store and library are a diffrent type of place but cannot be used to teach children.
QUESTION: You’ll likely have a kitchenette in what place where you sleep away from home? OPTION 1: house OPTION 2: hotel room OPTION 3: apartment OPTION 4: allen key OPTION 5: dormroom CORRECT OPTION: 2 EXPLANATION: Hotel room is a bedroom usually with bath in a hotel.
You’ll likely have a kitchenette in a hotel room where you sleep away from home. House is a home where you live permanently and not away from home. Apartments are house and is not where you sleep away from home. Allen key is not a room where you can sleep. Dorm room usually comes without a kitchen.
QUESTION: It was the only way out of town, the police parked their vehicles and drew their guns to create a what? OPTION 1: war OPTION 2: sporting goods store OPTION 3: military base OPTION 4: roadblock OPTION 5: fun CORRECT OPTION: 4
EXPLANATION: A roadblock is a barrier or barricade on a road which is set up to stop people passing through a road. Roads are ways of out towns. The police parked their vehicles to create a roadblock. Parking vehicles and drawing guns does not create fun all the other options.
QUESTION: Sahmbi was lying about the fugitive’s location. He was lying because he wanted to avoid legal what? OPTION 1: confusion OPTION 2: being found out OPTION 3: hurt feelings OPTION 4: being fired OPTION 5: trouble CORRECT OPTION: 5 EXPLANATION: People lie to avoid legal troubles as they involve lot of
hassle. All the other options have no legal implication and meaning.
QUESTION: What does getting in line for a long time require in a person? OPTION 1: intention OPTION 2: getting in the front of the line OPTION 3: basic organization OPTION 4: early childhood socialization OPTION 5: patience CORRECT OPTION: 5 EXPLANATION: Patience is the capacity to accept or tolerate delay,
problems, or suffering without becoming annoyed or anxious which is what required in a person to get in line for a long time. Getting in front of the line is not something in a person and getting in line for a long time does not require the things given in the other options.
QUESTION: What might a person see at the scene of a brutal killing? OPTION 1: bloody mess OPTION 2: pleasure OPTION 3: being imprisoned OPTION 4: feeling of guilt OPTION 5: cake CORRECT OPTION:
G.4 Naturalness Test Example Prompt The following is the prompt to filter examples for the naturalness of our interventions. Because this prompt is designed for instruction-tuned Llama2 models, it surrounds the instruction with 
 tags, matching the format these models were finetuned on.

 I’m going to show a sentence, and followed by the same sentence with a word added. It’s fine if the added word changes the meaning of the sentence. However, I want you to tell me if the second sentence still makes sense with the added word.
Sentence 1: "The children throw rocks at the militant threatening their safety."
Sentence 2: "The stuck children throw rocks at the militant threatening their safety."
Does the second sentence make sense with the added word? Please begin your answer with "Yes" or "No". 


heading: Future Work
text: Future work could apply the CCT to instruction-tuned models, as well as explanations generated using strategies such as question decomposition (Radhakrishnan et al. , 2023). Limitations
While our analysis identifies and corrects some shortcomings of prior work on measuring the faithfulness of NLEs, it does inherit some of the limitations of the original CT (Atanasova et al. , 2023). The counterfactual interventions only insert adjectives and adverbs, and only single words at a time, so our experiments do not measure sensitivity to other parts of speech. Our random intervention can generate text which lacks semantic coherence, despite our LLM filtering step. We do not test for synonyms, which could inaccurately label some explanations. Additionally, we do not consider the semantic usage of word mentions: for example, our metrics would not penalize the faithfulness of illogical explanations as long as they had the correct pattern of word inclusion. Some of these drawbacks could potentially be addressed by further filtering or analysis by more advanced LLMs, taking advantage of their semantic understanding. We study LLMs generating predictions and explanations using few-shot prompting, with example explanations taken from human-generated NLEs. These explanations can be highly dependent on annotation instructions. For example, CoS-E (Rajani et al. , 2019) and ECQA (Aggarwal et al. , 2021) both use CQA (Talmor et al. , 2019) as a base dataset, but ECQA explanations are significantly longer than those for CoS-E. As such, care should be taken when extrapolating our results to other tasks: in the few-shot setting, the example explanations provided can have just as much impact on faithfulness as the model being used