Are self-explanations from Large Language Models faithful? (2024)

Abstract: Instruction-tuned Large Language Models (LLMs) excel at many tasks and will even explain their reasoning, so-called selfexplanations. However, convincing and wrong self-explanations can lead to unsupported confidence in LLMs, thus increasing risk. Therefore, it’s important to measure if self-explanations truly reflect the model’s behavior. Such a measure is called interpretability-faithfulness and is challenging to perform since the ground truth is inaccessible, and many LLMs only have an inference API. To address this, we propose employing self-consistency checks to measure faithfulness. For example, if an LLM says a set of words is important for making a prediction, then it should not be able to make its prediction without these words. While selfconsistency checks are a common approach to faithfulness, they have not previously been successfully applied to LLM self-explanations for counterfactual, feature attribution, and redaction explanations. Our results demonstrate that faithfulness is explanation, model, and taskdependent, showing self-explanations should not be trusted in general. For example, with sentiment classification, counterfactuals are more faithful for Llama2, feature attribution for Mistral, and redaction for Falcon 40B.

heading: 1 Introduction
text: Instruction-tuned large language models (LLMs), such as Llama2 (Touvron et al., 2023), Falcon (Penedo et al., 2023), Mistral (Jiang et al., 2023), or GPT4 (OpenAI, 2023), are increasingly becoming mainstream among the general population, due to their capabilities and availability.
Instruction-tuned LLMs can even provide very convincing explanations for their utterances and will often do so unprompted. Because LLMs produce these explanations themselves and they provide justification for their own behavior, we term them self-explanations. Importantly, one may also judge the ethicality of models not just based on
their predictions but also on their self-explanations. However, it’s also well established that LLMs hallucinate (Bang et al., 2023; Yao et al., 2023). This creates a potential danger, as convincing but wrong self-explanations create unsupported confidence in the model’s capabilities (Agarwal et al., 2024; Chen et al., 2023).
295
Previous work on hallucination analysis has focused on a different kind of correctness, namely if the explanations reflect true world-knowledge (Bang et al., 2023; Yao et al., 2023; Rawte et al., 2023; Lin et al., 2022; Lee et al., 2021). Although important, such analysis can only be applied to selfexplanations about facts, not the model’s opinions. It also doesn’t tell us if the self-explanations truly reflect the model’s reasoning process, known as interpretability-faithfulness (Jacovi and Goldberg, 2020; Doshi-Velez and Kim, 2017).
Interpretability-faithfulness is hard to measure because the models are too complex for humans to manually analyze, making the ground-truth inaccessible (Jacovi and Goldberg, 2020). With selfexplanations, there is an additional challenge because of their free-formed nature, which makes them hard to analyze (Parcalabescu and Frank, 2023). Additionally, many contemporary LLMs only provide an inference API and often refuse to make discrete predictions when critical information is missing; these properties make previous faithfulness metrics that depend on confidence scores impractical to apply (Huang et al., 2023).
To solve these challenges, we propose a faithfulness metric that only uses an inference API and takes advantage of the model’s reluctance to answer when critical information is missing. We achieve this by limiting the scope of self-explanations to those verifiable using self-consistency checks and by carefully prompting the model regarding both the explanation and classification generation.
A self-consistency check uses re-evaluation to check if the explanation is consistent with the model’s behavior. For example, consider a hiring recommendation system. In 2018, Amazon found that “Women’s chess club membership” was a negative signal while “chess club membership” was a positive signal, indicating a harmful gender bias (Kodiyan, 2019). Such a contrastive example is known as a counterfactual. With LLMs, one can ask the model itself to edit the resume such that it would make the opposite prediction. We can then learn about the model behavior by comparing the two resumes, assuming the counterfactual is faithful. To check faithfulness, we ask the model for its hiring recommendation using the counterfactual resume. If the recommendation changed, as requested, it’s a faithful explanation. Figure 1 shows the workflow of prediction, explanation, and self-consistency check.
In this paper, we evaluate the faithfulness of the
following types of self-explanations: • Counterfactual explanations replace content, to
get the opposite prediction (Madsen et al., 2022b; Miller, 2019). For example, replace “Women’s chess club” with “Chess club”.
• Feature attribution explanations list the necessary words for making a prediction, such that without these words a prediction can not be made. For example, 1) “Women” 2) “BSc degree”.
• Redaction explanations remove all relevant words for making a prediction, such that a prediction can no longer be made. For example, “[REDACTED] chess club”. We demonstrate our approach on four datasets
with varying tasks: sentiment classification (IMDB Maas et al. 2011), multi-choice classification (bAbI and MCTest Weston et al. 2016; Richardson et al. 2013), and two-paragraph classification (RTE Dagan et al. 2006). Additionally, we apply the approach to Llama2 (70B, 7B), Falcon (40B, 7B), and Mistral (7B). The variability of these tasks and models shows the generality of our approach.
We find that the faithfulness of instruction-tuned LLMs depends on the model, explanation, and task. For example, regarding Llama2 (70B), counterfactuals only work with IMDB, and feature attribution only works with RTE and bAbI, clearly showing a task dependence. Additionally, we show our findings are robust to prompt variations.
Because faithfulness is explanation and taskdependent, we suggest self-explanations cannot generally be trusted and propose how future work might address this challenge.

heading: 2 Self-explanations
text: LLMs’ ability to produce highly convincing selfexplanations is a new development in the field of interpretability. Previously, a separate model or algorithm generated the explanation (Madsen et al., 2022b), not the predictive model itself. This development creates new challenges and opportunities (Singh et al., 2024).
For example, counterfactuals are often generated using an explanation model which is almost completely independent of the classification model they explain (Ross et al., 2021; Kaushik et al., 2020). The explanations may be generated by fine-tuning on a dataset with human-annotated explanations, where the classification model is only used to filter the generated explanations (Wu et al., 2021; Li et al., 2022). This is a problem because humans
have no insight into the classification model’s behavior (Wiegreffe and Marasović, 2021; Jacovi and Goldberg, 2020). Therefore, there is little reason to think that the counterfactuals represent the classification model (Hase et al., 2020a).
Instead, we let the language model generate both the classification and the counterfactual explanation. In principle, because the weights are the same, the explanation generation can access or simulate the classification behavior; a property known as self-model capabilities. Kadavath et al. (2022) explores the self-model capabilities of LLMs using a self-consistency check and find that LLMs have poor self-modeling capabilities.
That being said, it is possible that self-model capabilities are helpful but not necessary for generating self-explanations. If the LLM reproduces human behavior sufficiently well in both the classification and counterfactual case, then the explanations would be self-consistent and thus faithful.
Because Kadavath et al. (2022) have already explored self-model capabilities, and it might not be necessary for faithful self-explanations, we limit the scope to faithfulness evaluation.

heading: 3 Interpretability-faithfulness of self-explanations
text: This section describes the general methodology we propose and its principles, using counterfactuals as an example. Section 4 then goes into detail on the different kinds of explanations and datasets where this approach is possible and how we precisely formulate the prompts.

heading: 3.1 Self-consistency check
text: Counterfactuals are explanations where the input is modified such that the model predicts the opposite label. The intention is to provide a contrastive example to explain the model’s behavior (Wu et al., 2021; Ross et al., 2021; Madsen et al., 2022b). For example, in the hiring-recommendation example (Figure 1), instead of asking for a general explanation (e.g. “Why are you recommending not to hire”), we directly asked the language model to modify the resume such that it would yield a positive hiring recommendation.
Because the goal is to produce the opposite label, we can re-evaluate the prediction with the counterfactual and check that this goal is satisfied. If satisfied, the counterfactual is faithful. We refer to such re-evaluation as a “self-consistency check”.
Note that when using an instruction-tuned LLM in a conversational setting, it’s important to perform the re-evaluation using a new chat session. Otherwise, the chat model may predict the opposite label only because it was previously prompted to do so within the chat history.

heading: 3.2 Negative results and robustness
text: If the self-consistency check does not pass, we don’t know if the model cannot generate faithful counterfactuals in general or if a different prompt or generation sample would generate interpretabilityfaithful explanations. Therefore, negative results are less informative than positive results.
However, it’s important to consider the practical application where an end user asks a chat model for an explanation. Regular users may not be critical regarding the explanation. As such, we desire models that are robust to natural prompt variations and minor sampling differences, by providing faithful explanations in all cases.
Therefore, when measuring interpretabilityfaithfulness, we use six different prompt patterns in order to assess the LLMs’ robustness. Furthermore, we consider finding just one non-faithful prompt pattern significantly problematic.

heading: 4 Prompt-patterns for self-explanation and self-consistency checks
text: This section provides the specific details and considerations behind the prompts used in the experiment section. Specifically, the prompts that generate the counterfactual, feature attribution, and redaction explanations. As well as the prompt for generating the classification.
To facilitate the “specific details,” we use sentiment classification as an example. However, the methodology generalizes to multi-choice Q&A datasets (e.g., bAbI) and multi-paragraph classification (e.g., NLI). Prompts for those tasks are in Appendix E.2 and Appendix E.3, respectively.
All prompts are used with chat-tuned models, as these models are typically deployed and provide a well-defined framing mechanism between input and output1. Such framing helps to avoid some out-of-distribution issues compared to purely instruction-tuned models. However, the prompts only convey a one-time input-output relation and
1For example, Llama2 frames the user message with special 
 and 
 tokens (Touvron et al., 2023).
thus could be used with purely instruction-tuned models given the proper framing.
It should be noted that it’s impossible to prove that the model understands the user’s intent. However, it’s worth prioritizing the user’s intent rather than the model’s understanding when evaluating the potential impact of deployed models.

heading: 4.1 Counterfactual explanation
text: We generate counterfactuals by asking for an edit that generates the opposite sentiment. In Figure 2. we explicitly express the target sentiment in the prompt. To evaluate robustness to prompt variations, we also consider an implicit version by replacing “is {opposite sentiment}” with “becomes the opposite of what it currently is”.
The “Do not explain the answer.” part of the prompt template in Figure 2 (and the other prompts) is there to prevent the model from providing additional details about why it produced the counterfactual. These details prolong inference time and cannot be validated using our self-consistency framework. Despite this instruction, some models will occasionally provide them regardless; if this happens, the extra explanation is removed.
The model output is fairly systematic, often prefixing the counterfactual paragraph with “Paragraph:” or providing a clear separation, making it easy to extract the counterfactual paragraph.

heading: 4.2 Feature attribution explanation
text: A common alternative to counterfactual explanations is feature attribution. These explana-
tions highlight which input words are important for making a prediction. The faithfulness of these explanations can be evaluated using a selfconsistency check, where the important words are redacted/masked (Ribeiro et al., 2016; Karpathy et al., 2015; Sundararajan et al., 2017). Given a faithful explanation, it will be impossible for the model to perform the classification task (Samek et al., 2017; Hooker et al., 2019; Madsen et al., 2022a), assuming that the model understands the meaning of redaction/masking during classification. We discuss this and the classification setup in Section 4.4.
The model is only provided with the input prompt shown in Figure 3. The model response is a list of important words, and the matching words in the paragraph are replaced with “[REDACTED]”.

heading: 4.3 Redaction explanation
text: Redaction explanations are a less common variation of feature attribution explanations. Instead of algorithmically replacing important words with “[REDACTED]”, we ask the model to perform the replacement directly. The prompt-template is shown in Figure 4. As a prompt variation, we also use “[REMOVED]” instead of “[REDACTED]”.
Readaction may be easier for the model because the LLMs likely have built-in mechanisms for copying content (McDougall et al., 2023). In principle, the model could redact the entire paragraph, as we don’t constrain the redaction amount. This would be a faithful explanation but not a very useful explanation to humans (Doshi-Velez and Kim, 2017), we also don’t observe such behavior in practice (see for example Appendix E.1.3).
Besides the different replacement mechanisms, the interpretability-faithfulness metric works the
same. The explanation is faithful if the model can not classify the redacted paragraph.

heading: 4.4 Classification
text: So far, we have discussed how to generate explanations. However, the self-consistency evaluation depends on a classification of the original paragraph and the explanation paragraph (or, in the case of feature attribution, it’s the paragraph modified using the explanation).
In Section 4.2 and Section 4.3, we use a “[REDACTED]” string to indicate that content is missing. We do this rather than removing content, as removing it could create ungrammatical content that can create out-of-distribution issues (Madsen et al., 2023, 2022a). As such, the classification should be robust to [REDACTED] tokens.
Finally, in the case of feature attribution (Section 4.2) and redaction explanations (Section 4.3), the paragraph of faithful explanation can not be classified. Therefore, the classification should al-
low for an “unknown” class prediction. Importantly, the same prompt template is used
in all cases (Figure 5); as in, for all explanations and both before and after the explanation step. The “unknown” and “[REDACTED]” support is not required for the counterfactual case but is kept for consistency and comparability.

heading: 4.5 Persona robustness
text: In the past sections, we presented some prompt modifications specific to each explanation. Inspired by Deshpande et al. (2023), we propose the idea of using “persona” as a prompt modification that can be applied to any prompt. In our setup, a “persona” means that the subject of the explanation or classification request is either “you” or “a human”.
For example, the previously presented prompts (e.g., Figure 5) asked the question (e.g., “What is the sentiment ..”) in an objective manner. Instead, it’s possible to ask “What would you classify the sentiment as” or “What would a human classify the sentiment as” – the exact prompts are provided in Appendix E. We hypothesize that personas could be relevant for the model’s classification and explanation (Deshpande et al., 2023). For example, the “you” persona may be significant if the model has self-modeling capabilities (Kadavath et al., 2022).

heading: 5 Results
text: To evaluate prompt sensitivity, all the prompt variations from Section 4 (details in Appendix E) are evaluated using the Llama2-70B model fine-tuned for chatting (Touvron et al., 2023).
In addition, we use a default case to show differences across model types and sizes by evaluating on: Mistral v0.1 (7B) (Jiang et al., 2023), Falcon (7B and 40B) (Penedo et al., 2023), and Llama2 (7B and 70B) (Touvron et al., 2023). Model details are in Appendix B, with additional details (e.g., generation configuration) in Appendix C.
We perform all experiments with sentiment classification (IMDB), multi-choice Q&A tasks (bAbI1 and MCTest), and an entailment/NLI task (RTE), details of which are in Appendix A. The prompts for these datasets are slightly different; see Appendix E. The variability amoung the datasets, demonstrates that our methodology generalizes to both paragraph-based multi-choice questions and two-paragraph tasks. In all cases, the main paragraph is modified through the explanation. The questions, choices, or hypotheses are not modified.
Default case: The “[REDACTED]” token is used, counterfactuals use explicit targets, and all prompts use an objective persona.

heading: 5.1 Classification
text: Before evaluating the faithfulness, it’s worth first investigating the classification accuracy for each task. In addition, this experiment also investigates the effect of persona (you, human, and objective) and the redaction-instruction ([REDACTED] or [REMOVED]). To validate that the redaction instruction does not cause issues, an ablation study with no redaction instruction (None) is also performed.
Figure 6 shows that neither the redactioninstruction nor the persona affects the results much. This is the desired result, as the redaction instruction should not affect the classification and supports just analyzing the default case will be sufficient.
Using the default case, Figure 7 shows the effect
of different model types and sizes. There is quite a lot of variation between the different models. The most surprising result is that Falcon 40B performs slightly worse than Falcon 7B. Unfortunately, due to the complexity and lack of documentation regarding these models, it’s hard to make an educated guess on why this is.
Some task and model combinations do not perform well. This could be problematic if, for example, an LLM classifies a positive-sentiment input as negative, and the counterfactual explanation asks for it to become negative, then it would appear that the explanation is faithful despite the explanation having made potentially no changes to the input. As the scope of this paper is faithfulness evaluation, not classification performance, we do not attempt to improve the classification performance. Instead, only the correctly predicted observations are used to evaluate faithfulness; the rest are discarded.

heading: 5.2 Faithfulness
text: Using only the observations that are correctly classified and produce meaningful results (e.g., discarding “As an AI model I cannot do that.”), we evaluate the faithfulness of each observation. Because our self-consistency method determines whether or not an observation is faithful, faithfulness in this paper refers to the ratio of faithful observations.
Figure 8 show the faithfulness, for each promptvariation for Llama2-70B. Figure 9 show faithfulness with the default prompt settings for each model type and size.

heading: 5.2.1 Counterfactual
text: From the prompt-variation results in Figure 8, we find that the persona has little effect. Making the counterfactual target implicit or explicit also does not affect faithfulness much. The exception here is for MCTest and bAbI-1, which is expected as these are multi-choice datasets thus for an implicit-target there are multiple correct answers, while there is only one correct answer using an explicit-target. As such, this is all positive, as the goal is to have models that are robust to prompt variations.
From the model-variation results in Figure 9, we find that Llama2 and Mistral are only consistently faithful for IMDB sentiment classification, while Falcon never performs well.

heading: 5.2.2 Feature attribution
text: The feature attribution experiments show again that Llama2-70B is robust to prompt variations (Fig-
ure 8), which is the desired outcome. However, we find that only for RTE and bAbI-1 is Llama270B faithful. The size particularly affects faithfulness, where Llama2-7B and Falcon-7B perform very poorly. Despite this, Mistral (7B) is surprisingly good at identifying important words for sentiment classification, drastically outperforming every other model (Figure 9).
While it might be surprising that the models generally perform poorly, it makes sense that bAbI-1 is an easy task, as there are, from a human perspective, only two relevant words for answering the question (a location and individual).

heading: 5.2.3 Redaction
text: The redaction experiments show results similar to the feature attribution experiment. This makes sense as the two explanations are similar in nature. This is particularly true for Llama2-70B, which have nearly identical results for the different prompt variations (Figure 8).
This similarity also addresses a potential concern with feature attribution explanations, which is that masking (i.e. the [REDACTED] token) inputs might create out-of-distribution results. However, for the
redaction explanations, the masking is generated by the model, and sound therefore is in-distribution. The observation that redaction and feature attribution have similar results, validates that masking does not cause any issues for the feature attribution explanations.
Finally, we do observe that Falcon-40B generally performs much better for this task compared to other models, given the IMDB and MCTest datasets. Also, surprisingly Falcon-40B performs worse than Falcon-7B for bAbI-1. This is particularly surprising as bAbI-1 is a synthetic dataset with few important words, hence it should be an easy task.

heading: 6 Related work
text: Using self-consistency checks to measure faithfulness is not a new idea. One of the earliest uses was with feature attribution, where allegedly important information was removed from the input, and it was then checked that this removal altered the prediction (Samek et al., 2017; Hooker et al., 2019; Madsen et al., 2022a). These earlier works used an external explanation algorithm rather than self-explanations. Another important difference is that the classification was not discrete (e.g., posi-
tive or negative sentiment) but provided logits. The logit values were then measured before and after information removal, and the difference provides a more nuanced faithfulness metric.
The self-consistency idea has also been adopted for self-explanations. Huang et al. (2023), applied the same idea to self-explanation feature attribution from ChatGPT on a sentiment classification task (a 100 observation subset of SST (Socher et al., 2013)). Because ChatGPT does not provide logits, they instruct ChatGPT to provide a confidence score between 1 and 0 for either positive or negative sentiment. Similarly, they instruct ChatGPT to provide scores for the importance of each word. Using this approach, previous methods can in principle be easily applied. Unfortunately, they find their approach ineffective as the confidence and importance scores are not calibrated, and ChatGPT resists classifying as positive or negative sentiment when information is missing. For this reason, they call for better faithfulness metrics targeted LLM self-explanations.
In contrast, our approach does not depend on any scores (confidence or importance). Instead, the models perform discrete classification and can predict “unknown”. This solves the aforementioned issues identified by Huang et al. (2023).
Self-consistency checks have also been used to measure the faithfulness of mathematical counterfactuals. In particular, Lanham et al. (2023) applies self-consistency checks to Chain-of-Thought (CoT) self-explanations from LLMs, where the CoTs provide mathematical context (e.g. “2 + 3 = 5”) to a mathematical question. Lanham et al. (2023) then edits the CoT to contain false information (e.g. “2 + 3 = 6”) and checks that the prediction follows. The issue here is that injecting false facts may create out-of-distribution results or be interpreted as typos by the LLM, thus it’s unclear if this method is completely valid. Regardless, they find similar to our paper, that faithfulness is model and taskdependent.

heading: 6.1 Non-faithfulness works
text: Self-consistency checks also have been used to analyze other LLMs’ capabilities. For example, Kadavath et al. (2022) analyzed LLM self-modeling capabilities by comparing asking if the LLM knows the answer (Yes/No) and asking the LLM to answer directly. Li et al. (2023) performs a similar task with mathematical questions (e.g., comparing “What is 7 + 8?” with “7+8=15, True or False”). Fi-
nally, Hu and Levy (2023) evaluates meta-linguistic capabilities using self-consistency checks.
Outside of self-consistency checks, Chen et al. (2023) measures the simulatability of counterfactual self-explanations. Simulatability means that humans can predict the model’s behavior on input y, given an explanation for a similar input x. Chen et al. (2023) find that self-explanations are very convincing to humans (plausibility) but there is no correlation with simulatability. This shows that self-explanations from LLMs can be misleading (Agarwal et al., 2024). These findings highlight the importance of providing faithful explanations. A challenge with simulatability is it’s difficult to ensure that humans use the explanations and not their world-knowledge to predict the model. Faithfulness does not have this challenge and it’s therefore important to also measure faithfulness.
Finally, it’s worth briefly mentioning works on rationalizations. These are explanations by humans (e.g., e-SNLI (Camburu et al., 2018)) or by a model that has learned from human explanations. Rationalizations should not be used to explain the model (Jacovi and Goldberg, 2020) but may help convince humans of a prediction. As such, most works are on plausibility, but there are also simulatability studies (Hase et al., 2020b) and faithfulness evaluations (Atanasova et al., 2023). Although Parcalabescu and Frank (2023) argues that the works on faithfulness only measure consistency, not faithfulness.

heading: 7 Conclusion
text: Our investigation reveals that self-explanations’ faithfulness is highly model and dataset-dependent. This conclusion is similar to previous works (Lanham et al., 2023; Madsen et al., 2022a; Bastings et al., 2022). Our contribution is the ability to measure faithfulness on LLMs’ self-explanations, specifically counterfactuals, feature attribution, and redaction explanations.
The task dependence is concerning as it means LLM self-explanations cannot generally be trusted. There is also no reason to trust more free-form explanations where faithfulness can not be evaluated using self-consistency checks. This increases the risk with LLMs, as individuals may have the misconception that LLMs can explain themselves (Chen et al., 2023).

heading: 7.1 Future work
text: We propose that future work on developing instruction-tuned LLMs also evaluate the selfexplanation faithfulness using the methodology presented in this paper. We believe that this direction is necessary to decrease the risk of LLMs.
It may also be possible to fine-tune LLMs towards greater faithfulness. In particular, Kadavath et al. (2022) showed that it’s possible to fine-tune a model to improve its self-modeling capabilities. As such, it stands to reason that improving selfexplanation capabilities is also possible. Furthermore, as we suggest in Section 2, self-model capabilities may also improve self-explanation capabilities. Unfortunately, the model trained by Kadavath et al. (2022) is not available to the public, thus we were not able to evaluate this.
Finally, due to the difficulty in evaluating selfexplanations, we suggest that self-explanation faithfulness should be treated as an out-of-domain evaluation problem. For example, one could fine-tune for only counterfactual faithfulness and show that feature attribution and redaction explanations improve too. This would give credit to the model’s explanation capabilities in general, including more free-formed explanations, which cannot be as easily evaluated using self-consistency checks.

heading: Limitations

heading: Absolute faithfulness
text: Jacovi and Goldberg (2020) argue that the interpretability-faithfulness field should move towards a more nuanced view on interpretabilityfaithfulness, where the metric is not if an explanation is faithful or not, but how much faithfulness it has. We agree with this notion, but because the LLMs are designed to provide discrete outputs (a sequence of tokens), we have not been able to do that.
That being said, our absolute faithfulness is only for a single observation. We still provide an aggregation average, which provide the nuanced view on the model’s faithfulness as a whole.
Assuming the prompt is interpreted correctly Our work is based on a fundamental assumption that the prompts are interpreted correctly by the model. This is not possible to completely verify. However, we attempt to ensure this by providing natural sounding prompts. Hence, if the model doesn’t understand these prompts and therefore
generates unfaithful explanations or inaccurate classification, it’s a limitation of the model’s comprehension capabilities.
Additionally, we find that for at-least one dataset we get high classification accuracy or explanationfaithfulness. This indicates that the model does interpret these prompts sufficiently, it’s just not able to perform the requested task.

heading: No in-context learning
text: Because in-context learning have shown good results on many downstream tasks (Dong et al., 2022), it’s tempting to think the same is possible for generating explanations. Unfortunately, this is problematic as it would require known true explanations, which is not possible for humans to provide. Therefore, this paper only uses zero-shot explanations.
Poor classification performance creates challenging comparison
For most tasks we use, the LLMs do not perform well at the classification task and archives low accuracy. As the goal of this paper is to measure interpretability-faithfulness, not accuracy, we simply discard the incorrect observations. However, this does mean that comparisons will be more challenging for future work. There may also be a class imbalance, in what observations were selected.
Impossible to show lack of capability
Our results demonstrate that the LLMs will, in some cases, consistently not provide faithful explanations. However, this does not show that the model is fundamentally unable to provide the explanations. It only shows that for the specific prompttemplates, the explanations are not faithful. Because our work shows that it’s very challenging to generate faithful explanations, user’s should not trust the explanations. Unfortunately, users often find these explanations to be very convincing (Chen et al., 2023), we thus think this is problematic enough to warrant concern about AI Safety. However, it is not definitive evidence that the model’s can never explain themselves.
Limited number of explanation tasks
We provide interpretability-faithfulness results on 3 kinds of explanations: counterfactuals, importance measures, and redaction. Importance measures and redaction are also quite similar in the explanation they provide. As such, there is not a great variety of explanations in this paper. However, these
are currently the only explanations for which selfconsistently can be used to measure interpretabilityfaithfulness. We hope that future work can identify new ways of using self-consistently checks.

heading: Not measuring on ChatGPT, GPT-4
text: We do not evaluate on ChatGPT, GPT-4 and other popular closed models. This because such models are not static models. They are frequently updated, by for example OpenAI, and we don’t have the means to choose a specific model version. As such, it would not be possible to reproduce our results.
Additionally, These models use a license agreement with an unrestricted indemnity-clause. As such, if we were to show that the explanations are not faithful, this could damage the companies behind these products, and we may become financially liable for damages. Due to this personal risk, we do not believe a truly unbiased analysis can be provided.

heading: Ethics Statement
text: This paper discusses how to measure the interpretability-faithfulness of LLMs selfexplanations. As such, any mistakes in our methodology could lead to unsupported confidence or skepticism about LLMs. Additional skepticism likely won’t cause ethical issues. However, unsupported confidence can be very dangerous. The message of this paper, is that LLMs should not be assumed to provide faithful explanations. As such, we do not think this paper in itself could provide unsupported confidence.
As for model choices, we have specifically only used models where the license agreements do not restrict what we can say about the model. This should give the reader confidence that our views are unbiased. This is unlike ChatGPT or GPT4, which include an unrestricted indemnity-clause in their license agreement, which could make us financially liable for damages.

heading: Acknowledgements
text: Sarath Chandar is supported by the Canada CIFAR AI Chairs program, the Canada Research Chair in Lifelong Machine Learning, and the NSERC Discovery Grant.
Siva Reddy is supported by the Facebook CIFAR AI Chairs program and NSERC Discovery Grant.
Computing resources were provided by the Digital Research Alliance of Canada.

heading: C Experimental setup

heading: B Models

heading: A Datasets
text: Finally, we have made the code used for generating all the results in this paper publicly available at https://github.com/AndreasMadsen/ llm-introspection.
Parameter Value
temperature 0.1 top_p 0.95 repetition_penalty 1.2 top_k 50 seed 0

heading: D Compute
text: classification prompt is reused in each explanation variation.

heading: E Prompt examples
text: This section provides output examples for all the prompts used in this paper. The outputs are from the Llama2-70B chat-tuned model. Simple heuristics extract the content, such as the predicted label or the explanation. Correct extraction has been manually verified for every observation.

heading: E.1.1 Counterfactual

heading: E.1 Sentiment classification with IMDB

heading: E.1.2 Feature attribution

heading: E.1.3 Redacted
text: E.2 Multi Choice classification with bAbI-1

heading: E.2.1 Counterfactual

heading: E.2.2 Feature attribution

heading: E.2.3 Redacted

heading: E.3.1 Counterfactual

heading: E.3 NLI classification with RTE

heading: E.3.2 Feature attribution

heading: E.3.3 Redacted

heading: Future Work
text: However, this does mean that comparisons will be more challenging for future work. There may also be a class imbalance, in what observations were selected. Impossible to show lack of capability
Our results demonstrate that the LLMs will, in some cases, consistently not provide faithful explanations. However, this does not show that the model is fundamentally unable to provide the explanations. It only shows that for the specific prompttemplates, the explanations are not faithful. Because our work shows that it’s very challenging to generate faithful explanations, user’s should not trust the explanations. Unfortunately, users often find these explanations to be very convincing (Chen et al. , 2023), we thus think this is problematic enough to warrant concern about AI Safety. However, it is not definitive evidence that the model’s can never explain themselves. Limited number of explanation tasks
We provide interpretability-faithfulness results on 3 kinds of explanations: counterfactuals, importance measures, and redaction. Importance measures and redaction are also quite similar in the explanation they provide. As such, there is not a great variety of explanations in this paper. However, these
are currently the only explanations for which selfconsistently can be used to measure interpretabilityfaithfulness. We hope that future work can identify new ways of using self-consistently checks