Chain-of-Thought Unfaithfulness as Disguised Accuracy
Authors: Oliver Bentham
Nathan Stringham
Ana Marasovic
Venue: Accepted by TMLR ()

Abstract: Understanding the extent to which Chain-of-Thought (CoT) generations align with a large language model's (LLM) internal computations is critical for deciding whether to trust an LLM's output. As a proxy for CoT faithfulness, Lanham et al. (2023) propose a metric that measures a model's dependence on its CoT for producing an answer. Within a single family of proprietary models, they find that LLMs exhibit a scaling-then-inverse-scaling relationship between model size and their measure of faithfulness, and that a 13 billion parameter model exhibits increased faithfulness compared to models ranging from 810 million to 175 billion parameters in size. We evaluate whether these results generalize as a property of all LLMs. We replicate the experimental setup in their section focused on scaling experiments with three different families of models and, under specific conditions, successfully reproduce the scaling trends for CoT faithfulness they report.
However, after normalizing the metric to account for a model's bias toward certain answer choices, unfaithfulness drops significantly for smaller less-capable models. This normalized faithfulness metric is also strongly correlated ($R^2$=0.74) with accuracy, raising doubts about its validity for evaluating faithfulness.

Published in Transactions on Machine Learning Research (06/2024)
Chain-of-Though Unfaithfulness as Disguised Accuracy
Oliver Bentham"
oliver. bikeer.eenhhom@@taah.cc
Kahlert School of Computing University of Utah
Nathan Stringham"
nates utah. edu
Kahlert School of Computing University of Utah
Ana Marasović
www.marnasoric@uulaa..cc edu
Kahlert School of Computing University of Utah
Reviewed on OpenReview: https: openreview. net/ net/forum forum? id- ydcrP552ee
Abstract
Understanding the extent to which Chain-of Thouuht (CoT) generations align with a large language model's (LLM) internal computations is critical for deciding whether to trust an LLM's output. As a proxy for CoT faithfulness, Lanham et al. (2023) propose a metric that measures a model's dependence on its CoT for producing an answer. Within a single family of proprietary models, they find that LLMs exhibit ship between model size and their measure of faithfulness, and that 13 billion paramcter model exhibits increased faithfulness compared to models ranging from 810 million to 175 billion parameters in size. We evaluate whether these results generalize as property of all LLMs. We replicate the experimental sctup in their section focused on scaling experiments with three different families of models and, under specific conditions, successfully reproduce the scaling trends for CoT faithfulness they report. However, after normalizing the metric to account for a model's bias toward certain answer choices, unfaithfulness drops significantly for smaller less-capable models. This normalized faithfulness metric is also strongly corre- lated (R2=0.74) with accuracy, raising doubts about its validity for evaluating faithfulness.
relation-
1 Introduction
In Choughof-Thought (CoT) prompting, a large language model (LLM) is instructed to generate a step- by-step reasoning chain, typically in plain natural language, before providing its answer. Although it has been shown that this mothod improves zcro-shot performance for certain tasks (Kojima et al., 2022; Wei et al., 2022), the usefulness of the generated reasoning for explaining the model's behavior is less clear. This is partly because it is difficult determine if the generated CoT is coupled with the underlying model computations, and therefore,
faithfully represents the model's true reasoning process.
Most attempls to measure faithfulness of free-text explanations use tests which rule out specific cases where the model could behave unfaithfully (Atanasova et al., 2023; Turpin et al., 2023; Lanham et al., 2023). Lanham et al. (2023) derive a simpler metric for faithfulness which measures how often the answer produced by a model changes between a normal prompting setting and one where CoT used. If the answer does not change, they argue that the CoT reasoning is post-hoc, so "there no strong reason to believe that such reasoning would be faithful." " They show that this mcasure of faithfulness corrclates well with their own tests
Equal contributions.

---

Published in Transactions on Machine Learning Research (06/2024)
designed to rule out post-hoc reasoning and use it to measure changes in faithfulness across single family of unnamed proprietary models (Ganguli et al., 2023) ranging from 810 million to 175 billion parameters.
For eight multiple-choice NLP benchmarks they observe a trade-off between faithfulness and model size. Specifi- cally, they show that faithfulness, according to their met- ric, increases for models up 13 billion parameters fol- lowed by decrease for even larger models. This scaling followed by inverse scaling trend is illustrated by the V-shape in Figure 1. They discuss these results in re- lation to task accuracy, and interpret this scaling trend to mean that "...only models of a certain capability level (but no higher) on a task seem to produce faithful CoT." (Lanham et al., 2023, p. 8). This claim suggests that in settings where laithfulness requirements
Lanham et al. (reported)
100
Unfaithfulness (Lanham al.)
60-
40
AQUA
ARC (Chellenge)
TruthfuIQA
LagiQA
MMLU
ARC (Easy)
OpenBookQA
HellsSwag
high, choos-
10
100
ing a large, highly accurate model might come at the cost of less faithful explanations.
Model Size (Parameters)
Figure 1: Model size vs. unfaithfulness results reported in Lanham et al. (2023).
Since this conclusion is drawn from an experiment us- ing only one family of models, raises the question whether this finding is unique to the models evaluated or is a property of LLMs in general. To shed more light on this, we address the following research questions:
RQ1: Across various model familics, is inverse scaling observed in model size for CoT faithfulness once models reach certain capability level?
RQ2: If so, do other non-proprietary families of LLMs start to produce unfaithful CoTs at similar model size (13B parameters)?
RQ3: Does the optimally faithful model size depend on the difficulty of the task?
We measure the faithfulness of three families of openly accessible LLMs Llama 2 (Touvron et al., 2023). FLAN-T5 (Chung et al., 2022) FLAN-UL2 (Tay et al., 2023), and Pythia DPO (O'Mahony et al., 2024) using Lanham et al. (2023)'s 's metric the same set of multiple choice benchmarks and addition tasks.
Initially, our results show similar scaling trends, with unfaithfulness increasing for models above a certain capability level. However, after normalizing the metric to account for model's preference for certain answer choices, the high unfaithfulness scores for less capable models become low. That is, the rate at which these models produce the same answer with the addition of CoT largely accounted for by a bias select a certain letter option.
We discuss the implications of this finding with respect to measuring the faithfulness of CoTs and hypothesize another possible mechanism by which it could be confounded-the the ability of larger models to capture the knowledge of the CoTs in their weights. Although we are able to replicate these scaling trends for other LLMs under certain circumstances, we also find strong linear correlation between the faithfulness metric and accuracy, bringing into question how informative the metric is altogether.
2 Related Work
Faithfulness Tests The faithfulness of explanation measures the extent and likelihood that it accu- rately represents the reasoning process behind the model's prediction (Jacovi & Goldberg, 2020). Numerous proposals exist for mcasuring this, but they often lack direct comparability and produce inconsistent results (Lyu el al., 2022). For example, multiple methods have been used to verify if input lokens determined to be important by input attribution methods truly influence the model's decision decision-making process. These include (normalized) sufficiency and comprchensiveness (Yu et al., 2019; DeYoung et al., 2020; Carton et al., 2020), (recursive) and-ee-nndrettrin (Hooker et al., 2019; Madsen al., 2022), and recovering known artifacts (Bastings et al., 2022).
2

---

Published in Transactions on Machine Learning Research (06/2024)
Although it has been more challenging to measure the faithfulness of free-text explanations, that include chain-of-thoughts. several tests have still been proposed (Wiegreffe et al., 2021; Atanasova et al., 2023; Turpin et al., 2023; Lanham et al., 2023). A common approach is to intervene on either the input, free-text explanation itself, or some combination of the two and to observe its effect on the model's predictions. For example Atanasova et al. (2023) examine how the explanation changes afler counterfactually editing the imput sequence. They also attempt to reverse-engineer the input sequence from the generated explanations and then test whether it produces the original prediction when used as input to the model. Turpin et al. (2023) introduce biases into the imput and show that models can generate plausible explanations which fail to reference the specific bias. Lanham et al. (2023) intervene in various ways model CoTs during the decoding process including inserting a mistake into the reasoning chain, truncating it, paraphrasing it, and replacing it with filler lokens. They find the effects of truncaling the CoT and inserting mistakes correlate well with a simpler metric which compares the answers produced by a model on each instance with and without CoT. We investigate the generalizability of this faithfulness metric to three other familics of language models.
While these tests provide necessary condition for determining whether models behave faithfully following intervention, they are not sufficient to determine faithfulness of free-text explanations. Instead, they rule out specific ways by which the model explanation can be unfaithful to its prediction (Wiegreffe et al., 2021). Parcalabescu & Frank (2023) argue these should be seen as measuring ceff-coniistency of the output rather than faithfulness. They compare a group of tests and find large disparities in performance across datasets and models. They also introduce a new self-consistency measure which compares how much each input token contributes to both the CoT and the prediction. Our study complements this line of work by evaluating a different measure proposed by Lanham et al. (2023) and focusing explicitly on how it behaves across wide range of model sizes.
Scaling Laws With the training of increasingly large models (measured by mumber of parameters) has come an interest in deriving scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022). While many tasks exhibit trend of increased performance with larger model sizes, other trends such as inverse scaling (Lin et al., 2022; McKenzie et al., 2022) and U-shaped scaling (Belkin et al., 2019; Black et al., 2022; Ruis et al., 2023; Wei et al., 2023) also occur. Lanham et al. (2023)'s results can be viewed as an instance of inverse- scaling with respect to their measure of faithfulness for models larger than 13b. In our study, we ask whether the inverse scaling trend of their metric occurs for L.LMs generally.
3 Experimental Setup
In this section, we describe the metrics, models, tasks, and implementation details we followed to reproduce the scaling experiments.' We indicate where our methods align with Lanham al., and we motivate our alternative approaches where appropriate.
Lanham et al. Unfaithfulness To study the relationship between model size and faithfulness, Lanham et al. (2023) use metric of unfaithfulness that measures how often model M produces the same prediction with- and without-CoT on given task's dataset D.2 Equation (1) below shows this calculation, where NoCoT denotes the answer provided by the model without CoT prompting, and CoT denotes the answer provided by the model with CoT prompting.
UNFAITHFULNESS (MM (M,D)
1
Σ 1 (N) COT(M,,)) RED
(1)
D
Lanham et. al. (2023) argue that if a model's answers change with the CoT, it indicates the model's depen- dence CoT to produce given answer, which is necessary for faithfulness. They also support the inverse
We release our code at: "Note that Lanham al. (2023) introduce two additional measures of faithfulness based on early answering and adding mistakes before studying how faithfulncss scales with model sizes. Howeeer, for the scaling study, they say that the measurement in (1) is "highly predictive of overall early answering and adding mistakes results. :..] We [Lanham al.] thus this metric in lien of running the full early answering and adding mistakes experiments for computational reasons."
httpg//ttb...o//aaaaai////////// ciccccc_dddssedd_____________ _cccuracyccccycc
3

---

Published in Transactions on Machine Learning Research (06/2024)
interpretation, that when model produces the same answer with- and Without-CoT, it demonstrates that the model never needed CoT to produce the answer in the first place. This, they argue, is evidence that the CoT reasoning is post-hoc, ie., that the model was going to produce the same answer irrespective of CoT. Their use the metric erucially hinges on interpreting post-hoc CoT reasoning to imply lower faithfulness, establishing CoT reliance as a proxy for unfaithfulness.
Normalized Lanham et al. Unfaithfulness Prior work has shown that models are biased towards producing certain outputs, even when the imput has no content, e.g., "Input: N/A Sentiment:" leads to a higher rate of producing "positive" than "negative" (Zhao et al., 2021). Given this context, as well as the fact. that. TruthfulQA's 's validation dataset always has "A" the correct answer, we introduce normalized version of the unfaithfulness metric which accounts for model's tendency to select certain letter choices regardless of the content.
We first compute the normalization term as:
N(M D
Σ FED
(2)
where E is version of x where the answer choices have been randomly shuffled. In other words, Equation (2) measures how often the model selects the same letter choice when prompted twice in the No-CoT setting with different answer choice orderings.
Then the normalized metric becomes:
UNFATTHFULNEESS..Oo (M,D)
UNAATTUFULLESSSNormmneel
(3)
N(M,D)
This measures the frequency of answer changes with CoT, compared to changes expected from merely shuffling the answer order.
Models Our goal is to determine whether the scaling laws of chain-of-thought faithfulness that Lanham et al. (2023) observe in a specific proprictary model family, hold for LLMs in general. The model families we evaluate are described below.
Llama 2 (Touvron al., 2023) is a decoder-only family of LLMs available with 7B, 13B and 70B parameters, making this relevant comparison for evaluating whether inverse scaling begins at 13B parameters. In line with the original paper, we use the chat variant of the model that has been finetuned to provide helpful dialogue responses using reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020).
FLAN-T5 + UL2 (Chung et al., 2022; Tay et al., 2023) is an encoder-decoder family of LLMs available with 77M, 248M, 783M, 2.85B, 11.3B, and 20B parameters. Unlike Anthropic's models or the Llama 2 family of models, it has not undergone human preference alignment, however it has been instruction finetuned for over 1800 natural language tasks.
Pythia DPO (O'Mahony et al., 2024) extends Pythia pretrained decoder-only LLMs (Biderman et al., 2023), by finetuning on Anthropic's human preference dataset (Bai et al., 2022) using direct preference optimization (DPO; Rafailov et al., 2023). These models cover 70M, 160M, 410M, 1B, 1.4B and 2.8B parameters.
FLAN-T5 and Pythia DPO do not cover both sides of the 13B parameter point-of-interest where we would expect find the V-shape. However, they do provide context about the relationship between model scaling and CoT reliance for models smaller than 13B parameters. We do not include other common benchmark models like Mistral (Jiang et al., 2023) or MPT (Team, 2023b;a) because they do not have the range of model sizes needed to extrapolate patterns related to model scaling.
We also experimented with shuffling the ordering of answer choices between the No-CoT and CoT settings. These results found in Appendix A.2. 4 Links Pythia DPO Huggingface models: 70M, 160M, 410M, IB, 1.4B, 2.8B.
4

---

Published in Transactions on Machine Learning Research (06/2024)
AQUA-RAT MRC-Challenge
ARC-Easy
LogigA
OpenBookQA
Pythia DPQ
Liema
PLAN-TS + UL2
HellaSwag
TruthfulQA
Unfaithfulness (Lanham
Normalized Unfaithfulness
Accuracy with CoT
Unfelthfullness (Lanham eeneet. al)
100
Nortaa Itred Unfaltyuulnes fullaas
80
Accuracy with CoT
80
40
20
20
10a
10
10
10a
10a
10a
10"
10th
100
Normalized Unfaltyuuuess fuyeessss
Unfaitifulness [Lanhain et
80
Accuracy with CoT
80
40
40
20
20
Model Size (Parameters)
Model Size (Parameters)
Model Size (Parameters)
Figure 2: Lanham et al. (2023)'s unfaithfulness, our normalized unfaithfulness, and CoT prompting accuracy across different model sizes for Pythia DPO and Llama model families; see $3 for details. Each point a model, corresponding model family (symbol) and size, evaluated on given benchmark (color).
Ideally, we would include the Authropic models used in Lanham et al. (2023), taken from Ganguli et al. (2023). We cannot use these models, even through the Anthropic API, because we are restricted to a single model size, specifically, the size of Anthropic's Claude model available at the moment. We need access to all model sizes in Ganguli al. (2023) to be able to fully recreate the measurements necessary for drawing conclusions about model scaling. We also cannot guarantee that Anthropic's current Claude model is any of the models in Lanham et al. (2023) since Anthropic may have updated the publicly available model since the paper was released. Finally, even with access to all model sizes through an API, the decoding strategy described the original paper requires manual intervention in the decoding process to append special prompts for extracting answers. As a result, we assume the original findings by Lanham et al. (2023) are valid, and instead look evaluate other model families under the same conditions, where possible, validate whether their findings generalize to other LLMs..
Multiple Choice Benchmarks The multiple choice task is a question answering task where each example is presented as multiple choice question (MCQ), consisting of question and set of candidate answers, of which only one is correct. Following the original paper, we evaluate our models on the following MCQ datasets: AQuA-RAT (Ling et al., 2017), ARC-Challenge and ARC-Easy (Clark el al., 2018), HellaSwag (Zellers et al., 2019), LogiQA (Liu et al., 2023), MMLU (Hendrycks et al., 2021), OpenBookQAA (Mihaylov et al., 2018), and TruthfulQA (Lin et al., 2022).
In experiments, we the same prompting methods described in Lanham et al. (2023). Namely, we decode using mucleus sampling with p 0.95 and temperature 0.8. For Llama 2 and Pythia DPO, both decodcr-only models, we prompt in the same way as the original paper, taking turns and extracting the answer with special appended prompt, "So the right answer is (", that allows us to constrain our prediction to just the logits corresponding to the MCQ choices' letter tokens. For FLAN-T5, an encoder-decoder model not intended for dialogue, we follow the same procedure but without the turn based approach, directly prompting the model.
We "madel family" mean LLMss that range in parameter count, share similar pretraining procedures.
5

---

Published in Transactions on Machine Learning Research (06/2024)
Unfaithfulnes (Lanham et al.)
2-Digit Addition
3-Digit Addition
100
operands
operands
operands
40
operands
20
7B
108
138
70B
70
10B
138
70B
100
100
2 operands w/CoT
80
80
2 operands
Accuracy
4 operands w/CoT
60
60
4 operands
8 operands w/CoT
40
40
8 operands
20
16 operands w/CaT
16 operands
70B
Number of Model Parameters
Number of Model Parameters
Figure 3: Lanham et al. (2023)'s unfaithfulness and task accuracy for 2- and 3-digit addition problems containing 2, 4, 8, or 16 operands using Llama The bottom plots show the accuracy for each condition, where CoT prompting accuracy is represented dashed line, and without-Col' accuracy represented with a solid line. The x-axis for all plots is a log-scale with the model size (number of model parameters). For both tasks the optimally faithful model according to the metric occurs 13B; however, this might be due to the sparse nature of the x-axis
Addition Task In order to more carefully measure the effect of task complexity on their faithfulness metric, Lanham et al. (2023) propose an addition task where the difficully is controlled by varying the number of operands involved in the computation. We implement this task as 2-digit and 3-digit addition problems with 2, 4, 8, or 16 operands and prompt the model provide its answer in "<answor></answe XML tags. For each digit operands pair we generate a set of 2000 addition problems by sampling uniformly the operands from the appropriate range of integers (i.e. 10-99 for 2-digit and 100-999 for 3-digit). Using the same models previously, run inference with and without-Coll for each addition task and calculate the same answer percentage.
4 Results
In this section, we address whether other LLMs exhibit inverse scaling in model size for CoT faithfulness. If so, we aim to determine the parameter count at which the inflection point occurs and whether this depends on task difficulty. Figure shows how unfaithfulness (left), normalized unfaithfulness (center), and accuracy (right) change with different model sizes. We refer the reader to Tables 6-8 in the Appendix for a more detailed breakdown of these results.
Do we observe inverse scaling in faithfulness once models become sufficiently capable across different model families? Yes. In Figure 2, the Pythia DPO models exhibit V-shaped pattern of CoT unfaithfulness similar to that identified by Lanham et al. (2023) in Figure which flattens under the normalized metrie. The flattening happens because Pythia DPO models seem to favor specific answers over random selection when they are not yet capable of solving a task. Moreover, if we focus on the accuracy lines that are one standard deviation below the mean in Figure 2, we observe that only Llama 2 models and FLAN-UL2 over bocome better than a random bascline for benchmarks." We see inverse scaling in the
6 Appendix Figure shows this tendency. A baseline that selects an answer choice random ranges 25% depending on the benchmark. Accuracy results for individual benchmarks found in Appendix Figure 10.

---

Published in Transactions on Machine Learning Research (06/2024)
Accuracy vs. Unfaithfulness
Accuracy vs. Normalized Unfaithfulness
100
Unfaithfulness (Lanham et
X
tormalized Infarthfullees
Pythia DPO
60
?
Pythia DPO (CoT)
R²=0.025
Liama 2
40
Liama 2 (CoT)
FLAN-TS UL2
x
FLAN-TS + UL2 (CoT)
R²=0.740
20
40
60
80
100
20
40
60
80 100
Accuracy
Accuracy
Figure 4: Task accuracy vs. Lanham et (2023)'s unfaithfulness metric on the left and task accuracy vs. normalized unfaithfulness on the right. The dashed black line corresponds to a linear regression line fit to the data, along with its respective R correlation value. The correlation between accuracy and unfaithfulness is minimal (R2 = 0.025), but strong (R² 0.740) between accuracy and normalized unfaithfulness.
Llama 2 unfaithfulness scores for both the unnormalized and normalized metries; CoT' become less faithful as the models get bigger. These results support the takeaway of Lanham et al. (2023): "only models of a certain capability level (but no higher) on a task seem to produce faithful CoT."
The FLAN-T5 models do not show any particular trends in either condition when considered alone, but when considered with the capable FLAN-UL2, unfaithfulness might continue to increase with larger model sizes were they available.
increase in unfaithfulness. It is conceivable that
The observed connection between unfaithfulness and task accuracy leads us to further explore the correla- tion between these two factors. In our correlation the normalized condition. We contend that this strikingly clear correlation between normal- ized unfaithfulness and accuracy with Co suggests simplification of the intricate concept of measuring reasoning frithfulness that is too reductive.
plots, shown in Figure 4, we discover a high
Do all LLMs exhibiting inverse scaling in model size for faithfulness start to produce unfaithful CoTs at similar model size (13B)? No. Lanham al. (2023) find that inverse scaling only begins 13B parameters. The Llama 2 unfaithfulness scores suggest that inverse scaling begins at model size smaller than 7B parameters. The Pythia DPO unfaithfulness scores exhibit V-shape similar to the one found by Lanham et al. (2023), however, it indicates inverse scaling begins at 1B parameters. After normalization, the Pythia DPO model exhibits much shallower V-shape.
Does the optimally faithful model size depend on the difficulty of the task? No. To answer this, we consider Figure 3, which shows the Llama 2 models' performance on the addition task. Given that 2-digit addition is an easier task than 3-digit addition, we would expect the inverse scaling trend to begin at smaller models for 2- than for 3-digit addition. We do not see this pattern the Llama 2 unfaithfulness scores, where both scttings point to the 13B model having the most faithful CoTs. The discrepancy in our findings compared to Lanham el al. (2023) may be an issue about granularity, where our three Llama 2 model sizes difference between the addition conditions. Similar to findings in the
do not provide enough detail to original paper, we omit results for Pythia DPO and FLAN-T5 due to unreliability in extracting answers from the model's response for smaller models.
5 Discussion and Conclusions
In $4, we demonstrate that high unfaithfulness scores by models performing no better than random do not necessarily stem from genuinely unfaithful CoTs but due to the models' tendency prefer specific answers. Thus, a more accurate understanding of CoTs faithfulness can be obtained by computing the normalized

---

Published in Transactions on Machine Learning Research (06/2024)
version of Lanham et al. (2023)'s unfaithfulness measurement that use in this work. While it is possible that the smaller models used by Lanham et al. (2023) do not exhibit such bias, which we cannot test directly since they are proprietary, its existence in the models we test highlights a potential flaw in the unnormalized metric.
However, if the faithfulness of CoTs produced by less capable models is confounded by the order of answer choices, could be that the faithfulness of CoTss produced by more capable models also confounded by something? We hypothesize it might be. It is conceivable that more capable models can capture information present in CoT steps in their parameters and use it to predict the answer even if they not prompted to spell out this reasoning. Future work could explore this hypothesis by using emerging methods for causal tracing and editing of knowledge (Meng et al., 2023; Gupta et al., 2023). Specifically, use these methods to causally trace each step in a CoT, perturb localized parameters (or edit the knowledge), and observe the effect on answer prediction relative to perturbing random parameters. If CoTs are faithful, the ability answer should be affected more by intervening on localized parameters.
Moreover, after normalization, find strong linear correlation between task accuracy and unfaithfulness of generated Col's. We contest the notion that higher accuracy in one LLM compared to another means its CoTss are automatically less faithful. If these CoTs are also regularly plausible- which is often the case with highly capable models like GPT-4- the correlation suggests that such models reason differently from people in all cases where they solve the task well. Moreover, the notable change the perceived unfaithfulness of the Pythia DPO 70M model, simply by accounting for sensitivity to answer choice order, raises concerns about the sensitivity of this measurement. Together, these observations caution against using the rate which answers change with the provision of CoTs a measurement for assessing their faithfulness.
6 Acknowledgments
We would like to thank the anonymous reviewers for their helpful feedback which led to eliminating a potential confounding variable in our experiments and a clearer presentation of our findings. We also thank the members of the UtahNLP group for valuable insights and discussion. The support and resources from the Center for High Performance Computing the University of Utah are gratefully acknowledged.
References
Pepa Atanasova, Mana-Maria Camburu, Christina Lioma, Thomas Lukasiewicz, Jakob Grue Simonsen, and Isabelle Augenstein. Faithfulness tests for natural language explanations. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61sl Annual Meeling of the Association for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 283 294. Association for Computational Linguistics, 2023. doi: https://doi. org 10.g//1..110 18653/v1/2023 acl- 25.
URL
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Doddss Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforeement learning from human feedback. CoRR, hts/22044005822 2022. doi: 10.48550/AM 448550/ARXII/ ARXIV. 2200..5862.. URL https: //tp://do....o.. wt..../////...5/////....... 2286.05862.....
Jasmijn Bastings, Scbastian Ebert, Polina Zablotskaia, Anders Sandholm, and Katja Filippova. "will you find these shorteuts?" a protocol for evaluating the faithfulness of input salience methods for text classification. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 976-991, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10.18653/v 1122022..22aaee h/2022.emmlppmmatt... emnlp-main.64. URL https:/ htaps://acllanthology org.//22g2222222 aanlp-maing 64.
8

---

Published in Transactions on Machine Learning Research (06/2024)
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias varianer trade-off. Proceedings of the National Academy of Sciences, 116(32):15849- 15854, 2019. URL https://www pnas. wwwp//o//gg///////o/ooo//oo//e/e htt///////ab//1111////////// 10w.1/077p.....m 0000000000000/0/00000000000001 1903070116.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (cds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2028, Honolulu, Hawaïi, USA, volume 202 of Proceedings of Machine Learning Research, pp. 2397 2430. PMLR, 2023. URL https: : /ttp//eeeocce....//////// wetigs.mmr..prrrree.
bwdermann3a..aae...a html.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvan Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-2001 An open-source autoregressive language model. In A Angela Fan, Suzana Nlic, Thomas Wolf, and Matthias Gallé (eds.), Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Lan- guage Models, pp. 95 136, virtual+ +Dublin, May 2022. Association for Computational Linguistics. doi: 10.18653/vl http///22.igssiieieo 1.9. URL https : aalp//aaaantooggy...omo. or2/2022. bigscience-1.9.
Samuel Carton, Anirudh Rathore, and Chenhao Tan. Evaluating and characterizing human rationales. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pp. 9294-9307, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 00065///1/0200.emmppmmmaa https://aclanthol1. org/2020 mmnlp-main.76e
URL
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. 4299 4307, 2017. URL
waaedd9t-tbtttttthaeeaea .html.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob De- vlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetue models. CoRR, abs/2210 22111.11,,,,,, 2022. doi: 10.48550/ARXIV 2210.11441. URL https: //doi. org/10. 48550/arXiv. 2210.2202222222 11416.
language
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the AI2 reasoning challenge. CoRR, abs/1803 1803.0557, 2018. URL
wtg////////a/////// 05457.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Effi cient finetuning of quantized lims. Advances in Neural Information Processing Sys- tems, 36, 2024. URL https://prroceeding.. wwwpeeedddinnnnnnnn.oooeeeeeee.e www.ips.ccupappeee.... files/ wwwes//paeee////ee paals///pppr/2233//eee pdf.
Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: benchmark to evaluate rationalized NLP models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Anrual Meeting of the Association for Computational Linguistics, pp. 4443 4458, Online, July 2020. Association for Computational Linguis- tics. doi: 10.18853/vvl htt5:/33/11122.accl//// acl.02200ma..mmin.... 408. URL https: attps://aclana httpa:///ccaaaatooo/o// org/2020 acl-main. 408.
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamile Lukosiute, Anna Chen, Anna Goldic, Azalia Mirhoscini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemí Mercado, Nova DasSarma, Oliver Rausch,
9

---

Published in Transactions on Machine Learning Research (06/2024)
Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Tawton, Tom Henighan, Tristan Hume, Yun- tao Bai, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, and Jared Kaplan. The capacity for moral self- correction in large language models, 2023. URL https: org/10.
/000 48550/arKiv. 0t..00//02x1..220007755555 2302.07459.
Anshita Gupta, Debanjan Mondal, Akshay Sheshadri, Wenlong Zhao, Xiang Li, Sarah Wiegreffe, and Niket Tandon. Editing common sense in transformers. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 8214- 8232, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023, emnlp-main.51 m11. URL https: hatpss////caaatoooo/o// org/2023. emnlp oonlp-main...
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. 2021. URL https: htapp:///pppnnee http///ppprrvvveeeeeeeee wtw.rrrrrooo.eetoouoiiiddde
d7XBjmI3Gmm)
Jordan Hoffmann, Sebast Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katic Millican, George van den Driessche, Bogdan Damoc, Aurclia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-oplimal large language models. CoRR, 48550/arXiv. 2203. 15556.
2022. doi: 10.48550/AM ARXIV
URL https://doi.or org/10..
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Bue, Emily B. Fox, and Roman Garnett (ods.), Advances in Neural Information Processing Sys- tems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancoweer, BC, Canada, pp. 9734-9745, 2019. URL http:///pooceedns.neerrreee cc/ html.
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should define and evaluate faithfulness? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4198 4205, Online, July 2020. Association for Computational Linguistics. doi: 1118603//112022.aclmm/ www.g//...g.0000/
URL
386.
Albert Q, Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnicr, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06822. 2023. doi: 10.48550/ ARXIV .2310.06825. URL https://doi. org/10. 48544/855//// aetii0//rxvv.... 2310 06825.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08361, 2020. URL https://taaati http//axivvv.gr..r org//rss///...o httg/01..2000.... 08361.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large lan- guage models zero-shot reasoners. In Advances in Neural Information Processing Systems, vol- ume 35, pp. 22199-22213 2022. URL https://proocedddng/ neurues cc/paper files/ fwless/pper//00/ paper/202// 100.0e11110009911166222222000 Confere-noeeeeeeeeeeoe .df..
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamile Lukosiute, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac HatfieldDodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. Measuring faithfulness in chain-of of-thhtght reasoning, 2023. URL https: //doi. oooo..org/1... / 48550/arXii /ariiv. 2307.13700 .17002.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human false- hoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214 3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/202. acl-long.220 229. URL https://aclanthoooggy/ wrg/2022 ael-long.223
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. Regina Barzilay and Min-Yen Kan (eds.); Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158-167 Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10. 10.18653/v1 111517-1015. URL https://aclamthhooogg 1001111111...
Hanmeng Liu, Jian Liu, Leyang Cui, Zhiyang Teng, Nan Duan, Ming Zhou, and Yue Zhang. Logiqa 2.0 an improved dataset for logical reasoning in natural language understanding. IEEE ACM Truns. Audio Speech Lang. Process., 31:2947-2962, 2023. doi: 10.1109 10.1109/TASL...... 20A.S.....22.329900400 URL Attps://doi.oo org/ 10. 10.1199/TAALL ww2.:222...3022 3293046.
Qing Lyu, Marianna Apidianaki, and Chris Callison-Burchh Towards faithful model explanation in NLP: A survey. CoRR, hts//200.111380 2022. doi: 10. 0..48550/AAXTT 2222.22222.. URL https://dol.org// 48550/arXiv. 2209. 11326.
Andreas Madsen, Nicholas Meade, Vaibhav Adlakha, and Siva Reddy. Evaluating the faithfulness of importance measures in NLP by recursively masking allegedly important tokens and retraining. Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pp. 1731 1751, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 111883/v1120000 findings-cmnlp // aacclanthology.orr//0200 wrg//022. findings-enienn emnlp.
URL https:
Ian McKenzie, Alexander Lyzhov, Alicia Parrish, Ameya Prabhu, Aaron Mueller, Najoung Kim, Sam Bow- man, and Ethan Perez. The inverse scaling prize, 2022. URL https://gi htt//gtthuub..aa wttp:tttt.....eeeeeeeeeeeeeeeeee com//inverse-scallnng prize.
Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.ncc, 2023. URL https://openreviae http://pwrrriie..eeeeddd/d// net/pdf?id= MkbcAHIYgyS..
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, October-November 2018. Association for Computational URL https: htttpp://acaanthhooog http://cchaattoooggorrrr//////// org/D18-1260.
Linguistics. doi: 10. (653/v1/DD1-///1
Laura O'Mahony, Leo Grinsztajn, Hailey Schoelkopf, and Stella Biderman. Attributing mode collapse in the fine-tuning of large language models. In ICLR 2024 Workshop Matheraatical and Empirical Un- derstanding of Foundation Models, 2024. URL https: http:///poeeeeeeeet/ooorr///////
Letitia Parcalabescu and Anette Frank. On measuring faithfulness of natural language explanations. CoRR, hts//231..77466 2023. doi: 10.48550/ARXIV.22111 231..0...11..74666 URL https://doi.oon 10. wwtp///.....//////////.////////. 48550/arXiv. 07466.
Rafael Rafailov, Archit. Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Tn Thirty-seventh Con- ference on Neural Information Processing Systems, 2023. URL https://arxixe wrwp:/s//////////////
htsg/bb//2305 18290.
Laura Eline Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette.
The goldilocks of pragmatic understanding: Fino-tuning strategy matters for implicature resolution by LLMs. In Thirty-sevent Conference on Neural Information Processing Systems, 2023. URL https: / //openreview
11

---

Published in Transactions on Machine Learning Research (06/2024)
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Tearning to summarize with human leedback. 2020. URL. https:// proceedings.neuree www.ipe....pppppeeeeeeeee http:///........ooooooooeeeeeee0
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denuy Zhou, Neil Houlsby, and Donald Metzler. UL2: unifying language learning paradigıns. In The Eleventh International Conference on Learning Representations, TCLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL tttps://openreett net/ pdf7idd6ruVLB727MCC
MosaicML NLP Team. Introducing mpt-30b: Raising the bar for open-source foundation models, 2023a.
URL www.://iimc....ooooo/ooooaooa/aa
Accessed: 2024-02-07.
MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable llms,
2023b. URL www. www.mmsscc...... wttpmmm..omo/moo/m///////// Accessed: 2024-02-07.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton- Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lec, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subra- manian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Fduuov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: /tte:://doii http://oooooo/////////// 10. 100...8550aaa /er.vv0/arxi.... www....00.............. 09288.
Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don't always say what they think: Unfaithful explanations in chain-offthought prompting. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 74952 74965. Curran Associates, Inc., 2023. URL https: httpps://prooeeediing. nwwrsss.e.rppp.op wccppapprfiillleeeeeeeeeeeeeeeae files/ wttp://aaeeeeeeaaaaa----ee- Conperr-ooferennce pdf.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2022. URL http://papers. mips. wttpecepppppeaeeeeeeee files/ ftlees/paper/2222/hass html.
Jason Wei, Najoung Kim, Yi Tay, and Quoc Lo. Inverse scaling can become U-shaped. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15580- 15501,, Singapore, December 2023. Association for Computational Linguis- tics. doi: 10.18653/v1/2022 1t053///1/22..mmlpppp aannl-mminn.. main.963. URL https://acaanthology org/2023. emnlp- mamipp-mmin... 963.
Sarah Wiegreffe, Ana Marasović, and Noah A. Smith. Measuring association between labels and free-text rationales. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Pro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10266-10284 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. http://////222..mmmmmaamaaa
doi:
URL https://aclantholle org/2021 emnlp-main .044.
Mo Yu, Shiyu Chang, Yang Zhang, and Tommi Jaakkola. Rethinking cooperative rationalization: Introspec- live raction and complement control. In Kentaro Inui, Jing Jiang, Vincent. Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNIP)) pp. 4094-4103, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: (118653vv1/D11 14200. URL attpp:://cclaaaaooogy.o///////
12

---

Published in Transactions on Machine Learning Research (06/2024)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Llúís Mârquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791 4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: //aclanthology. beeelaarbooooyyorr//11---111 org/P19-1472.
URL https:
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few- shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12697 12706. PMLR, 18 24 Jul 2021. URL httts://proceediie
mlr. wal.r.peessvvvve zhwo/iz......... html.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Chris- tiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/ 1ess/9990.08993 2019. URL http:://aree www./bo/////.a.......oooo wttpaaa.////19../ 08593.
A Appendix
A.1 Implementation Details
Sampling from datasets As shown in Table 2, we do not always evaluate the entire test sets due to rosource constraints. Instead, we either use the full datasct or sample 500 examples, whichever less.
Decoding with quantized models Due to resource constraints, we are not able to fit Llama 2 70b or FLAN-UL2 single NVIDIA A100 GPU. In order to still get results, use quantized 4-bit and bit alternatives, respectively. For evaluation settings, these quantized models have shown similar performance as their unquantized counterparts (Dettmers et al., 2024).
A.2 Experiments with Different Answer Ordering
In Equation (3) we introduce normalized unfaithfulness which accounts for the known inductive bias in some models to consistently select the same answer choice. Another approach to mitigate this bias is by applying different shuffling strategies and comparing their effect on the unfaithfulness metric. The two strategies we use are described below, and are demonstrated in Figure 5.
Same Ordering. The MCQ choice ordering is shuflled as to different from the original test dataset. However, both the CoT and No-CoT conditions are presented in the same order of choices. Different Ordering. The MCQ choice ordering is shuffled such that the CoT and No-CoT conditions get different orderings from each other, and they are both different from the original dataset.
One problem with the different condition is that it introduces two treatments at once: the ordering of the answer and the provision of COT. This means that when we compare results of the unfaithfulness metric from this condition with those from the same condition, cannot precisely assign which treat- ment caused the effects. For example, it could be that changes in unfaithfulness in the different con- dition are due to sensitivity to answer ordering and not COT. As seen in Fig. 6 the effect of different ordering empirically similar to that of normalized unfaithfulness. This also the case Fig. where the observed correlation with accuracy even stronger (R² 0.862) than for normalized unfaithfulness. Duc to its cleaner interpretation, we opt to present our normalized unfaithfulness, but include the results for the different condition here.
largest organ in human body?
Same
Different
CoT
CoT
A) brain
brain
A brain
A] skin
B) lverr
B) iver
B) skin w
8: brain
C) stomach
stomasc
C) liver
C) stomach
D) skknn
Dj skin of
D) ststmmcc
D2 Pver
Figure 5: Illustration of the same vs. different order- ing conditions for MCQs.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Pythia DPO
Liama 2
+ FLAN-T5 UL2
Unfalthfulness (Lanham al.)
Normalized Unfaithfulness
Different-Ordering Unfaithfulness
Unfeltifulness (Larhain et al)
Unfoithffnnne hntss (Lanham all
-100
Normaltzed Unraithytllees
60
40
20
Model Size (Parameters)
Model Size (Parameters)
Model Size (Parameters)
Figure Lanham el al. (2023)'s unfaithfulness (left), normalized unfaithfulness (center), and Lanham et al. (2023)'s unfaithfulness when choice ordering is different with- and without- CoT (right). Normalized unfaithfulness shows V-shape, but normalized unfaithfulness and different-ordering unfaithfulness show scaling relationship.
Accuracy vs. Unfaithfulness (Different Ordering)
Unfalthfunness [Lanham all)
80
Pythia DPO
Pythia DPO (CoT)
60
Liama 2
Liama 2 (CoT)
40
FLAN-TS + UL2
FLAN-TS UL2 (CoT)
20
#7m0.862
Accuracy
Figure 7: Correlation plot showing accuracy vs. unfaithfulness when the prompt choices are shufled bet wetw the CoT and no-CoT conditions. The two metrics are highly correlated (R² 0.862).
No-CoT Same Letter
No-CoT Same Answer
100
100
Same Letter
80
Same Answer
80
Pythia DPO
60
60
Llama 2
FLAN-T5
40
40
20
20
10a
109
1020
10a
10a
1020
Model Size (Parameters)
Model Size (Parameters)
Figure 8: Given a different ordering of MCQ options, how often each model responds with the same letter (left) and same answer (right). As model size increases, models favor the same answer more and the same letter less.
Model
Number Parameters
Chat Tuning
Type
Source
Llama-2
RLHF None DPO
Decoder
htsge://hugginggacc co/neta: 1lans/Llams
FLAN-To
80M, 222M, 737M. 288, 11.3B
Fneoder-eecode
htspe://hugginng
Pythis-DPO
70M, 160M, 110M. 1B, 1.B., 2.8B
Decoder
htspe://buggingfaee wwt.aaaee......aaoeeeeeeeeeeaeae co/lonshonyy/////oooooooooocooce ththiis 2.88 h-8.heelppul dgo
Table 1: The models used in our evaluation expand the scope Lanham et al. (2023) to inchude different model architectures (FLAN-T5) as well as different methods for chat tuning.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Task
Split
#
Examples
Source
AQuA-RAT (Ling et ol., 2017)
[est.
254
htcps://huggingffacc.
ARC-Challenge (Clark et al., 2018)
ARC-Easy (Clark al,, 2018)
HellaSwag (Zellers al., 2019)
validation
LogiQA (Tiu et al., 2023)
test
500
MMLU (Hendrycks et al., 2021)
test
262
OpenBookQA (Mihaylov et al., 2018)
test validation
287
TruthfulQA (Liu et ul., 2022)
500
Table 2: Evaluation setups used for the multiple choice benchmarks. We include the number of examples
used and include link to cach data source.
Original Ordering
Same Ordering
Different Ordering
100
100
100
80
80
80
Unfatthtunesss (Lanham
Liama
40
40
40
20
20
20
10
10
10
Unfaithfulness (Lanham et ai)
80
80
80
FLAN-T5 UL2
60
60
60
40
40
40
109
10a
10.0
109
10P
10
108
102
1040
100
100
100
Unfaithfuiness (Lanham et al.)
80
80
80
Pythia DPO
40
40
40
20
20
20
10a
10"
10a
10a
108
Model Size (Parameters)
Model Size (Parameters)
Model Size (Parameters)
AQUA-RAT
ARC-Easy HellaSwag
Logi MMLU
OpenBookQA TruthfulQA
ARC-Challenge
Figure 9: Scaling of unfaithfulness scores for model families on 8 multiple choice benchmarks. Each contains particular model family and each column represents different ordering condition for the multiple choice question answers.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Original Ordering
Same Ordering
Different Ordering
100
100
100
80
80
80
Liama 2 Accuracy
60
40
40
40
20
20-
20
1040
10:0
1040
100
100
80
80
80
FLAN-T5 UL2
Accuracy
60
60
60
40
20
20
20
0
10a
10a
10
10a
10a
10a0
10a
10a
10th
100
100
100
Pythia DPO
Accuracy
60
60
60
40
40
40
20
20
20
10a
10a
10a
10"
Model Size (Parameters)
Model Size (Parameters)
Model Size (Parameters)
AQUA-RAT CoT
ARC-Easy CoT
OA w/ CoT
OpenBookOA w/ CoT
AQUA-RAT
ARC-Easy
Log QA
OpenBookQA
ARC-Challenge w/ COT
HellaSwag w/ CoT
MMLU w/ CoT
TruthfulQA w/ CoT
ARC-Challenge
HellaSwag
MMLU
TruthfulQA
Figure 10: Scaling of accuracy for 3 model families on 8 multiple choice benchmarks. Each row contains particular model family and each column represents a different ordering condition for the multiple choice question answers.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Model Family
# Parameters
Examples
Accuracy (No CoT)
Accuracy (CoT)
Unfaithfulness
FLAN-T5
77m
254
23.23
22.83
46.06
FLAN-T5
248m
254
25.20
19.29
31.89
FLAN-T5
783m
254
24.02
23.62
28.35
FLAN-T5
2.85b
254
25.59
21.65
FLAN-TS
11.3b
254
27.56
22.44
30.31
FLAN-T5
20b
254
27.56
29.13
29.13
AQUA-RAT
Llama
254
22.44
22.44
45.67
IJama 2
13h
254
20.87
24.41
40.16
Llama 2
70b
254
25.98
35.43
39.37
Pythia DPO
70m
254
24.80
24.80
99.61
Pythia DPO
160m
254
24.80
21.65
64.57
Pythia DPO
410m
254
25.59
79.13
Pythia DPO
254
20.47
18.50
6.69
Pythia DPO
1.4b
254
14.17
22.83
19.69
Pythia DPO
2.8b
254
23.23
22.05
25.59
FLAN-TS
77m
1172
25.85
26.28
64.42
FLAN-T5
248m
1172
38.74
38.57
62.20
FLAN-T5
783m
46.76
51.19
FLAN-T5
2.85b
1172
64.85
60.15
60.58
FLAN-T5
11.3h
1172
25.85
26.28
64.42
ARC- Ce-Challenge
FLAN-T5
20h
500
79.80
66.80
71.60
Llama 2
7b
1172
57.59
58.36
66.64
Llama 2
13b
697
63.41
67.00
70.88
Llama
70b
988
76.72
76.62
76.42
Pythia DPO
70m
500
23.60
24.00
99.60
Pythia DPO
160m
1172
70
23.98
82.34
Pythia DPO
410m
1172
23.2 23
22.87
89.33
Pythis DPO
1b
500
21.80
28.00
35.60
Pythia DPO
1.4b
1172
26.28
27.13
16.47
Pythia DPO
2.8b
500
23.80
25.80
33.60
FLAN-T5
77m
2376
28.41
25.84
62.04
FLAN-T5
248m
2376
53.20
51.18
65.57
FLAN-T5
783m
2376
53.49
57.53
54.17
FLAN-T5
2.85h
2376
28.41
25.84
62.04
FLAN-T5
11.3b
2376
25.84
62.04
FLAN-TS
20b
500
93.60
84.20
85.40
ARC-Easy
Llama
2376
75.42
72.77
75.93
Llama
13b
1250
83.52
83.12
82.88
Llama 2
70b
1098
92.35
91.80
91.89
Pythia DPO
70m
500
22.80
100.00
Pythia DPO
160m
2376
25.17
25.34
82.53
Pythia DPO
410m
2122
25.12
25.45
90.86
Pythia DPO
500
23.20
26.60
39.60
Pythia DPO
1.4b
500
27.60
28.60
31.80
Pythia DPO
2.8b
500
24.40
24.40
34.20
Table 3: Full results for models evaluated on the multiple-choice benchmarks using the original answer choice ordering (continued in Tables 4-5).
17

---

Published in Transactions on Machine Learning Research (06/2024)
Model Family
# Parameters
Examples
Accuracy (No CoT)
Accuracy (CoT)
Unfaithfulness
FLAN-T5
77m
500
42.00
31.40
42.60
FLAN-T5
248m
10042
42.70
38.41
59.04
FLAN-T5
783m
10042
78.26
64.66
70.58
FLAN-T5
2.85b
10042
86.21
89.73
FLAN-TS
11.3b
10042
90.54
85.18
87.79
FLAN-T5
20b
500
83.60
75.20
81.80
HellaSwag
Llama
4434
51.71
44.72
55.28
IJama 2
13h
627
59.49
53.43
60.29
Llama 2
70b
500
72.01
62.69
69.03
Pythia DPO
70m
6891
25.13
25.03
98.77
Pythia DPO
160m
3940
24.97
25.00
99.52
Pythia DPO
410m
2117
25.22
96.03
Pythia DPO
2731
25.38
24.79
34.60
Pythia DPO
1.4b
2236
24.82
23.61
32.07
Pythia DPO
2.8b
1536
25.52
27.08
24.54
FLAN-TS
77m
651
28.26
26.42
83,26
FLAN-T5
248m
651
30.88
31.64
58.22
FLAN-T5
783m
651
28.88
27.19
54.84
FLAN-T5
2.85b
651
31.80
33.49
50.84
FLAN-T5
11.3h
651
37.17
31.95
41.01
FLAN-T5
20h
500
33.80
31.20
48.40
LogiQA
Llama 2
7b
651
36.41
31.49
51.77
Llama 2
13b
651
39.78
34.41
51.77
Llama
70b
651
41.63
57.91
Pythia DPO
70m
500
20.40
20.20
99.40
Pythia DPO
160m
651
20.12
22.89
77.73
Pythia DPO
410m
651
20,74
21.81
90.48
Pythia DPO
1b
651
26, (6.42
23.81
8.91
Pythia DPO
1.4b
500
24.80
22.60
25.00
Pythia DPO
2.8b
500
29.80
27.80
33.20
FLAN-T5
77m
13985
27.93
26.05
38.30
FLAN-T5
248m
13985
33.54
32.53
56.40
FLAN-T5
783m
6379
42.19
35.57
44.19
FLAN-T5
2.85h
500
42.20
43.40
46.60
FLAN-T5
11.3b
500
37.20
40.40
FLAN-TS
20b
500
50.40
46.40
50.20
Llama
4766
50.67
52.45
59.04
Llama
13b
3520
53.81
55.62
63.07
Llama 2
70b
500
54.80
64.40
56.60
Pythia DPO
70m
500
20.80
21.00
99.00
Pythia DPO
160m
2213
22.77
22.86
99.50
Pythia DPO
410m
1991
22.70
23.00
89.00
Pythia DPO
500
21.43
35.32
Pythia DPO
1.4b
2172
27.85
27.03
33.38
Pythia DPO
2.8b
1533
27.27
25.83
33.66
Table 4: Full results for models evaluated on the multiple-choice benchmarks using the original answer choice ordering (continued in Table 5).
18

---

Published in Transactions on Machine Learning Research (06/2024)
Model Family
# Parameters
# Examples
Accuracy (No CoT)
Accuracy (CoT)
Unfaithfulness
FLAN-T5
77m
500
27.80
21.60
59.60
FLAN-T5
248m
500
41.40
67.20
FLAN-T5
783m
500
44.80
47.60
47.20
FLAN-T5
2.85h
500
70.40
65.20
64.00
FLAN-T5
11.3h
500
74.40
33.40
29.20
Open
FLAN-TS
20b
500
79.60
69.60
71.20
Llama 2
7b
500
59.40
57.2
62.00
Llama
13h
500
63.80
65.00
67.20
Llama 2
70h
500
75.20
74.20
73.80
Pythia DPO
500
27.60
28.20
89.60
Pythia DPO
160m
500
27.60
25.40
80.80
Pythis DPO
410m
500
27.60
25.80
91.80
Pythia DPO
1b
500
25.20
27.80
15.00
Pythia DPO
1.4b
500
26, 80
23.80
34.80
Pythia DPO
2.8b
500
22.87
22.59
23.69
FLAN-T5
77m
817
5.88
15.42
70.13
FLAN-T5
248m
817
27.66
12.12
59.00
FLAN-T5
783m
817
8.08
12.36
49.82
FLAN-T5
2.85b
817
8.32
20.93
44.55
FLAN-T5
11.3b
817
23.62
29.62
43.33
TruthfulQA
FLAN-T5
20b
500
47. 77.60
35.40
51.60
Llama
817
28. 89
35.37
58.02
Llama 2
13h
817
24.11
46.39
45.41
Llama 2
70b
817
41.98
45.78
57.04
Pythia DPO
70m
500
100.00
98.20
98.20
Pythia DPO
160m
817
99.88
66.95
66.83
Pythia DPO
410m
817
97.55
82.50
80.66
Pythia DPO
1b
817
10.16
60.59
13.71
Pythia DPO
1.4b
500
0.80
15.00
39.00
Pythia DPO
2.8b
500
10.80
22.60
31.20
Table 5: Full results for models evaluated on the multiple-choice benchmarks using the original answer choice ordering (continuation of Table 4).
19

---

Published in Transactions on Machine Learning Research (06/2024)
Model Family
# Parameters
Examples
Accuracy (No CoT)
Accuracy (CoT)
Unfaithfulness
FLAN-T5
77m
254
20.08
21.65
46.46
FLAN-T5
248m
254
21.65
17.72
34.65
FLAN-T5
783m
254
22.83
23.23
27.56
FLAN-T5
2.85b
254
23.23
22.44
FLAN-TS
11.3b
254
22.05
22.44
28.74
AQuA-RAT
FLAN-T5
20b
254
26.38
24.02
26.38
Llama
254
29.92
27.56
40.16
IJama 2
13h
254
26.77
28.35
45.28
Llama 2
70b
254
27.56
37.80
36.61
Pythia DPO
70m
254
19.69
20.08
99.61
Pythia DPO
160m
254
24.41
23.62
99.21
Pythia DPO
410m
254
18.91
18.57
85.43
Pythia DPO
254
25.59
19.69
42.13
Pythia DPO
1.4b
254
20.87
18.54
20.47
Pythia DPO
2.8b
254
20.47
22.83
20.08
FLAN-TS
77m
1172
24.23
22.27
65.10
FLAN-T5
248m
1172
37.12
37.46
62.20
FLAN-T5
783m
46.42
49.91
FLAN-T5
2.85h
1172
63.91
61.69
62.29
FLAN-T5
11.3h
1172
25.43
25.60
62.20
ARC- C--Challenge
FLAN-T5
20h
500
78.80
65.80
69.40
Llama 2
7b
1172
56.57
56.14
65.10
Llama 2
13b
1172
62.37
63.14
68.60
Llama
70b
1027
74.68
76.83
Pythia DPO
70m
500
24.60
24.80
99.80
Pythia DPO
160m
500
24.85
25.44
99.41
Pythia DPO
410m
500
25.00
25.40
$4.00
Pythia DPO
1b
500
29.20
25.40
38.20
Pythia DPO
1.4b
389
24.42
22.11
32.39
Pythia DPO
2.8b
500
27.60
25.40
32.20
FLAN-T5
77m
2376
26.89
25.97
62.75
FLAN-T5
248m
2376
53.83
49.16
65.24
FLAN-T5
783m
2376
53.20
57.70
55.68
FLAN-T5
2.85h
2376
27.48
26.85
61.24
FLAN-T5
11.3b
2376
28.62 62
27.90
62.16
FLAN-TS
20b
500
93,20
84.80
85.00
ARC-Easy
Llama
2376
75.97
72.35
75.51
Llama
13b
1010
84.55
83.86
83.66
Llama 2
70b
1111
93.07
90.91
91.09
Pythia DPO
70m
500
25.60
99.43
Pythia DPO
160m
500
28.60
28.60
98.87
Pythia DPO
410m
500
24.20
24.80
83.81
Pythia DPO
500
27.60
23.40
33.60
Pythia DPO
1.4b
500
26.00
24.20
30.64
Pythia DPO
2.8b
500
28.60
26.40
31.
Table 6: Full results for models evaluated on the multiple-choice benchmarks using the same answer choice ordering (continued in Tables 7-8).
20

---

Published in Transactions on Machine Learning Research (06/2024)
Model Family
# Parameters
Examples
Accuracy (No Co
Accuracy (CoT)
Unfaithfulness
FLAN-T5
77m
500
44.40
33.80
40.80
FLAN-T5
248m
10042
42.52
38.24
59.91
FLAN-T5
783m
10042
78.36
65.02
70.92
FLAN-T5
2.85b
10042
86.32
89.65
FLAN-TS
11.3b
10042
90.75
85.11
87.93
FLAN-T5
20b
500
83.20
72.60
79.20
HellaSwag
Llama
4557
50.34
43.16
55.48
IJama 2
13h
3847
57.68
47.78
54.02
Llama 2
70b
500
73.80
57.40
66.00
Pythia DPO
70m
500
27.60
27.40
98.20
Pythia DPO
160m
500
24.60
24.40
99.60
Pythia DPO
410m
500
24.63
24.63
91.71
Pythia DPO
500
23.
25.80
31.00
Pythia DPO
1.4b
500
24.20
26.20
37.00
Pythia DPO
2.8b
500
25.40
21.80
22.00
FLAN-TS
77m
651
24.58
24.58
84.95
FLAN-T5
248m
651
29.49
27.50
56.53
FLAN-T5
783m
651
24.27
27.65
56.22
FLAN-T5
2.85h
651
32.72
32.72
48.39
FLAN-T5
11.3h
651
36.56
31.03
39.78
FLAN-T5
20h
500
37.00
33.80
48.40
LogiQA
Llama 2
7b
651
34.41
33.64
51.31
Llama 2
13b
651
36.87
33.33
51.31
Llama
70b
651
42.09
39.63
58.53
Pythia DPO
70m
500
26.20
26.20
99.80
Pythia DPO
160m
500
25.40
99.80
Pythia DPO
410m
500
28.60 60.
29.20
78.80
Pythia DPO
1b
500
19.77 70
25.12
33.00
Pythia DPO
1.4b
500
24.00
23.40
26.40
Pythia DPO
2.8b
500
25.00
23.40
33.00
FLAN-T5
77m
13985
27.87
26.24
38.85
FLAN-T5
248m
13985
33.96
33.26
54.67
FLAN-T5
783m
6497
39.97
35.31
45.37
FLAN-T5
2.85h
500
39.80
40.00
44.20
FLAN-T5
11.3b
500
34.00
41.40
FLAN-TS
20b
500
47.60
44.80
56.80
MMMI
Llama
4901
50.17
49.89
59.36
Llama
13b
3760
54.81
55.59
63.01
Llama 2
70b
500
52.80
64.40
56.20
Pythia DPO
70m
500
29.00
29.00
98.80
Pythia DPO
160m
3904
26.46
26.43
99.46
Pythia DPO
410m
500
23.40
25.00
84.40
Pythia DPO
2627
25.31
25.43
31.56
Pythia DPO
1.4b
2305
25.03
24.21
33.84
Pythia DPO
2.8b
1504
26.66
26.33
34.11
Table 7: Full results for models evaluated on the multiple-choice benchmarks using the same answer choice ordering (continued in Table 8).
21

---

Published in Transactions on Machine Learning Research (06/2024)
Model Family
# Parameters
# Examples
Accuracy (No CoT)
Accuracy (CoT)
Unfaithfulness
FLAN-T5
77m
500
24.40
25.40
61.40
FLAN-T5
248m
500
28.60
24440
61.60
FLAN-T5
783m
500
27.20
28.20
61.80
FLAN-T5
2.85b
500
29.60
27.00
58.80
FLAN-T5
11.3h
500
26.20
24.00
59.80
OpenBook
FLAN-TS
20b
500
79.20
70.40
72.80
Llama 2
7b
500
58.40
55.80
61.20
Llama
13h
500
65.00
63.80
65.60
Llama 2
70h
500
80.00
73.40
74.20
Pythia DPO
70m
500
25.20
99.40
Pythia DPO
160m
500
24.20
24.20
99.80
Pythis DPO
410m
500
25.60
26.60
90.80
Pythia DPO
1b
500
24.00
22.60
34.80
Pythia DPO
1.4b
500
25. 80
26.40
32.80
Pythia DPO
2.8b
500
24.74
26.48
26.83
FLAN-T5
77m
817
19.34
20.81
71.73
FLAN-T5
248m
817
18.97
21.79
71.24
FLAN-T5
783m
817
23.26
23.87
70.62
FLAN-T5
2.85b
817
18.85
19.83
70.75
FLAN-T5
11.3b
817
20.93
20.93
71.60
TruthfulQA
FLAN-T5
20h
500
46.20
36.60
55.80
Llama
817
28.
43.21
53.98
Llama 2
13b
817
31.09
47 47.86
50.92
Llama 2
70b
809
46.35
56.00
60.44
Pythia DPO
70m
500
26.00
26.60
99.20
Pythia DPO
160m
500
21.40
21.60
99.20
Pythia DPO
410m
500
22.80
82.20
Pythia DPO
1b
500
23.00 60
21.20
40.80
Pythia DPO
1.4b
500
24.40
23.40
45.80
Pythia DPO
2.8b
500
23.40
24.80
34.40
Table 8: Full results for models evaluated on the multiple-choice benchmarks using the same answer choice ordering (contimation of Table 7).
22

---

Published in Transactions on Machine Learning Research (06/2024)
Model Family
# Parameters
Examples
Accuracy (No CoT)
Accuracy (CoT)
Unfaithfulness
FLAN-T5
77m
254
24.02
20.08
22.44
FLAN-T5
248m
254
15.75
23.62
23.23
FLAN-T5
783m
254
22.44
23.23
25.20
FLAN-T5
2.85b
254
22.44
28.74
25.59
FLAN-TS
11.3b
254
24.02
27.56
21.26
AQUA-RAT
FLAN-T5
20b
254
27.56
29.53
31.10
Llama
254
24.41
24.41
22.83
IJama 2
13h
254
22.83
25.59
26.77
Llama 2
70b
254
22.83
38.19
24.41
Pythia DPO
70m
254
18.90
19.29
24.02
Pythia DPO
160m
254
16.54
19.29
16.14
Pythia DPO
410m
254
21.26
25.59
21.26
Pythia DPO
254
21.65
21.65
18.90
Pythia DPO
1.4b
254
20.08
20.08
20.87
Pythia DPO
2.8b
254
22.83
20.87
24.41
FLAN-TS
77m
1172
24.15
25.60
22.95
FLAN-T5
248m
1172
38.82
36.60
52.30
FLAN-T5
783m
42.75
40.70
FLAN-T5
2.85b
1172
64.33
59.73
59.73
FLAN-T5
11.3h
1172
26.11
26.19
26.19
ARC- P--Challenge
FLAN-T5
20h
500
78.60
66.00
67.60
Llana 2
7b
1172
55.20
54.44
52.13
Llama 2
13b
1172
64.16
65.44
62.54
Llama
70b
988
77.13
77.02
75.20
Pythia DPO
70m
500
25.00
26.20
23.00
Pythia DPO
160m
500
25.42
25.06
22.81
Pythia DPO
410m
500
26.00
23.40
24.60
Pythis DPO
1b
500
25.20
25.40
22.80
Pythia DPO
1.4b
500
22.81
26.32
25.00
Pythia DPO
2.8b
500
24.60
28.00
22.00
FLAN-T5
77m
2376
27.06
25.59
25.25
FLAN-T5
248m
2376
53.07
49.16
58.96
FLAN-T5
783m
2376
53.41
57.45
46.59
FLAN-T5
2.85h
2376
24.49
25.34
25.84
FLAN-T5
11.3b
2376
26.77
27.61
24.49
FLAN-TS
20b
500
93.60
83.40
$3.60
ARC-Easy
Llama
2376
74.71
71.04
67.26
Llama
13b
2376
81.57
79.04
75.38
Llama 2
70b
1109
91.61
91.07
89.18
Pythia DPO
70m
500
26.40
25.00
Pythia DPO
160m
500
22.87
26.52
22.63
Pythia DPO
410m
500
24.70
29.17
26.49
Pythia DPO
500
26.00
23.20
23.00
Pythia DPO
1.4b
500
25.40
25.20
27.00
Pythia DPO
2.8b
500
26.80
24.40
24.20
Table 9: Full results for models evaluated on the multiple-choice benchmarks using the different answer choice ordering (continued in Tables 10-11).
23

---

Published in Transactions on Machine Learning Research (06/2024)
Model Family
# Parameters
Examples
Accuracy (No CoT)
Accuracy (CoT)
Unfaithfulness
FLAN-T5
77m
500
45.99
28.47
35.77
FLAN-T5
248m
10042
42.57
38.56
55.28
FLAN-T5
783m
10042
77.90
64.51
68.00
FLAN-T5
2.85b
10042
85.61
88.22
FLAN-TS
11.3b
10042
90.38
85.39
$6.00
FLAN-T5
20b
500
82.80
71.80
75.60
HellaSwag
Llama
4358
51.68
44.63
48.81
IJama 2
13h
1975
58.68
51.44
54.84
Llama 2
70b
588
73.13
58.16
66.67
Pythia DPO
70m
500
22,20
26.00
25.60
Pythia DPO
160m
500
23.00
23.40
23.80
Pythia DPO
410m
500
24.80
23.80
Pythia DPO
500
20.53
26.24
25.10
Pythia DPO
1.4b
500
24.20
24.00
27.20
Pythia DPO
2.8b
500
25.68
27.05
28.42
FLAN-TS
77m
651
24.12
24.73
27.96
FLAN-T5
248m
651
28.26
29.03
48.69
FLAN-T5
783m
651
27.65
33.64
FLAN-T5
2.85b
651
27.65
32.10
42.55
FLAN-T5
11.3h
651
37.33
29.65
38.40
FLAN-T5
20h
500
38.60
34.20
45.40
LogiQA
Llama 2
7b
651
34.25
32.26
44.55
Llama 2
13b
651
35.48
36.71
43.47
Llama
70b
651
42.86
38.86
52.69
Pythia DPO
70m
500
23.80
24.80
24.40
Pythia DPO
160m
500
23.20
25.80
Pythia DPO
410m
500
25, 25,80
22.40
29.20
Pythia DPO
1b
500
23.60
23.80
26.80
Pythia DPO
1.4b
500
24.20
25.40
27.00
Pythia DPO
2.8b
500
23.80
21.40
28.20
FLAN-T5
77m
13985
28. 24
26.04
30.22
FLAN-T5
248m
13985
33.66
33.11
50.45
FLAN-T5
783m
6158
41.51
35.37
43.54
FLAN-T5
2.85h
500
44.20
37.80
46.00
FLAN-T5
11.3b
500
48.80
35.40
37.
FLAN-TS
20b
500
45,20
48.00
52.20
MINI
Llama
4698
49.98
49.17
51.34
Llama
13b
3628
55.40
56.89
56.01
Llama 2
70b
500
56.00
63.00
53.80
Pythia DPO
70m
500
24.00
24.60
23.80
Pythia DPO
160m
3718
25,26
24.93
25.34
Pythia DPO
410m
500
24.00
24.00
24.20
Pythia DPO
2565
25.5 54
23.90
25.07
Pythia DPO
1.4b
2224
25.40
26.17
25.58
Pythia DPO
2.8b
500
25.60
22.80
21.00
Table 10: Full results for models evaluated on the multiple-choice benchmarks using the different answer choice ordering (continued in Table 11).
24

---

Published in Transactions on Machine Learning Research (06/2024)
Model Family
# Parameters
# Examples
Accuracy (No CoT)
Accuracy (CoT)
Unfaithfulness
FLAN-T5
77m
500
26.40
24.20
28.60
FLAN-T5
248m
500
25.60
28.40
FLAN-T5
783m
500
24.60
28.40
27.60
FLAN-T5
2.85h
500
27.80
25.00
28.20
FLAN-T5
11.3h
500
31.40
25.60
27.00
Open
FLAN-TS
20b
500
78.40
69.20
68.80
Llama 2
7b
500
57.60
57.60
57.80
Llama
13h
500
63.40
63.80
57.00
Llama 2
70h
500
77.60
76.40
73.00
Pythia DPO
500
30.00
40.00
10.00
Pythia DPO
160m
500
23.35
21.83
21.83
Pythis DPO
410m
500
22.00
24.00
26.80
Pythia DPO
1b
500
24.40
25.80
21.60
Pythia DPO
1.4b
500
25.40
23.00
25.60
Pythia DPO
2.8b
500
24.00
25.80
25.00
FLAN-T5
77m
500
22.77
20.44
22.03
FLAN-T5
248m
500
21.54
21.79
22.28
FLAN-T5
783m
500
20.20
20.20
22.40
FLAN-T5
2.85b
500
23.62
21.91
FLAN-T5
11.3b
500
21.30
23.99
22.64
TruthfulQA
FLAN-T5
20b
500
48.00
36.40
52.40
Llama
500
30.97
40.51
39.90
Llama 2
13h
500
32.93
48.96
47.98
Llama 2
70b
500
46.88
56.79
57.77
Pythia DPO
70m
500
23.80
25.00
25.60
Pythia DPO
160m
500
26.60
26.40
25.40
Pythia DPO
410m
500
26.80
23.00
23.00
Pythia DPO
1b
500
26, 80
25.80
30.20
Pythia DPO
1.4b
500
22.40
23.20
27.20
Pythia DPO
2.8b
500
26.00
23.00
26.00
Table 11: Full results for models evaluated on the multiple-choice benchmarks using the different answer choice ordering (contimuation of Table 10).
25


Reviews:
[{"review_id": "BN29pndXDp", "reviewer": "Reviewer_H9RD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The paper tackles an under-explored aspect of AI interpretability by focusing on the faithfulness of CoT reasoning, which is crucial for trusting AI systems in critical applications.\n- It expands on previous research by testing across multiple model families and sizes, which enhances the generalizability of the findings.\nQuantitative and Qualitative Insights:\n- The combination of quantitative metrics and qualitative examples provides a robust analysis of how and why model outputs may vary with changes in CoT presentation.\n- Discovering that the faithfulness metric is sensitive to the order of answer choices highlights an important aspect of model behavior that could impact applications requiring reliable reasoning.\n\n## Weaknesses\n- The generalizability of results across languages other than English and non-multiple-choice formats is not addressed, which could be important in global or varied application contexts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the reliability of chain-of-thought (CoT) reasoning in large language models (LLMs). It explores how faithfully these models represent their internal reasoning processes when generating CoT outputs. The study aims to assess the extent to which CoT generations align with the internal computations of LLMs and to question the trustworthiness of LLM outputs based on CoT reasoning. It builds on prior work by Lanham et al. (2023) that proposed a metric for evaluating the faithfulness of CoT by examining if the model's answers change when CoT is applied compared to standard prompting. The authors replicate the methodology of Lanham et al. across multiple LLM families, including proprietary models of varying sizes (from 810 million to 175 billion parameters), to see if the previous findings generalize. They introduce a metric that quantifies unfaithfulness based on whether the model's output changes with the introduction or removal of CoT. The paper evaluates this across various NLP benchmarks and controlled task settings.\n\nThe paper reports a scaling-then-inverse-scaling relationship between model size and faithfulness, where mid-sized models (around 13 billion parameters) show higher faithfulness compared to both smaller and larger models. However, a critical finding is that simply changing the order of answer choices can drastically alter the faithfulness metric, reducing it by up to 73 percentage points. This suggests the metric may not robustly measure faithfulness. There is also a high correlation ($R^2 = 0.91$) between the faithfulness metric and task accuracy, which raises questions about whether the metric truly captures faithfulness or merely accuracy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don't have any proposed changes. The paper is pretty straightforward.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mFzGVHarfc", "reviewer": "Reviewer_GdPT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The main Figure 2 is terrific!  The top row showing wild differences in faithfulness metric is quite the contrast with the bottom row showing no difference is accuracy (as would be expected, so good control).  This is a convincing warning sign about the metric.\n2. Figure 3 showing the accuracy/faithfulness correlation with different orderings is very nice as well, and I agree it's suspicious.\n2. The papers conclusions and strength of conclusions are well calibrated to the experiments run: it's hard to be certain what it is going on, and the paper does not overclaim.  In particular, I found it tempting through out to go into more philosophical objections to the faithfulness metrics, but it is reasonable to keep this paper scoped to concrete empirical concerns.\n\n### Weaknesses\n\n1. We don't know what ordering method Lanham et al. used!  You should ask them!  I realize it's hard to do this with deanonymizing, but the paper would be significantly more useful if one could easily compare with the plots in Lanham et al.  If the authors don't want to not deanonymize, perhaps the TMLR editor could ask?\n2. It's sad that the x-axis in Figure 4 is so sparse, and the conclusion (that the optimally faithful model is the same size across problems) seems weak as a result.  I would suggest either caveating this plot, or finding a different model family with a smoother range of sizes for the experiment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a clean experiment which indicates that a chain-of-thought faithfulness metric from [Lanham et al. 2023](https://arxiv.org/abs/2307.13702) may be confounded.  Specifically, the average difference in answer between CoT vs. NoCoT depends wildly on whether it is measured with the same random answer ordering for both CoT and NoCoT or different answer ordering.  In particular, weak models may seem more faithful because they derive more of their answer distribution from letter preference (e.g., for A).\n\nThe paper is clean and to-the-point, and I believe it should be accepted.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The most important one is to determine what ordering Lanham et al. used, to make the comparisons easier.\n\nAn improvement to Figure 4 to make the x-axis smoother would be nice as well, though I realize that may be hard.  If no improvement is feasible, it may be worth caveating the results a bit.", "broader_impact_concerns": "No ethical concerns: the paper shows a potential confound in a method for investigating safety concerns (whether humans would infer the correct things from seeing chain-of-thought transcripts), which is a good safety contribution.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ZiUtGNOWi", "reviewer": "Reviewer_u5Tm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- It's very important for science to reproduce results of experiments on proprietary models with open-access models. \n\n- The authors reproduce results of the original paper on three different open-access model families. \n\n**Weaknesses**\n\n- Only part of the Lanham results are reproduced, which make the conclusions of the paper not sound. The original authors (Lanham et al.) claim this metric measures unfaithfulness **by virtue of it's correlation with another method that measures faithfulness**. In and of itself, the metric that is used in the submission need not say anything about faithfulness. If an answer changes from zero-shot to CoT, that does not need to indicate faithfulness. If the answer does not change, it might mean the model just also knows the answer zero-shot, and with CoT it still follows the reasoning from the CoT faithfully. The model might even employ a different reasoning for zero-shot than CoT, this metric says nothing about that. Further, if the answer does change from zero-shot to CoT, that also does not mean the model is faithful to it's reasoning. The authors are up front about these limitations here and there in the paper, but the problem is that the experiment that the original Lanham paper does (intervening on the inputs to the model), is not done. Without this experiment, and showing that indeed that correlates with the unfaithfulness metric for the open-access models, you cannot say something about how well this unfaithfulness metric works for other LLM families than presented in Lanham et al.\n\n- The additional experiments the authors propose (ordering MC answers differently between zero-shot and CoT) are not sound due to a confounder that is unaddressed. The authors adjust the original unfaithfulness metric, which tests whether an answer changes if using CoT prompting instead of zero-shot prompting, by also changing the ordering of answers between zero-shot and CoT prompt. This is explicitly introducing a confounder, and experiments need to be done to test if this is the reason for the different unfaithfulness results due to this metric. The authors seemingly have the data necessary to test for this confounder already; does a models answer change within prompt if you change the ordering? I.e., if the answer changes from zero-shot to zero-shot different ordering, perhaps an answer change from zero-shot to CoT doesn't mean that the model is more faithful, but just that it's sensitive to answer orders.\n\n- The above makes me wonder if one of the main conclusions of this paper holds: because you observe a high correlation in the different condition between accuracy and unfaithfulness you claim that unfaithfulness as proposed by Lanham et al is too reductive. I think the metric is indeed too reductive, but not because of the presented correlation. As mentioned above, I think the \"different\" condition introduces a confounder which makes it essentially a different and less sound metric than the one Lanham et al propose. You first need to investigate the confounder that the different condition introduces before you can say anything about the original unfaithfulness metric. Furthermore, even if the confounder is addressed, a high correlation between accuracy and unfaithfulness doesn't necessarily mean there is something wrong with the metric. It might really just be that models that have more capabilities are less faithful. \n\n- The presentation of the results is very difficult to interpret because the authors present an average over about 8 different benchmarks (of which 1 is MMLU). An average like that is really difficult to interpret, and a variance over benchmarks seems even less useful. For example, it could now be that for certain benchmarks the unfaithfulness becomes less with scale.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper reproduces part of the results of Lanham et al. on unfaithfulness in CoT generations for 3 open LLM families. \"Faithfulness\" measures the extent to which an explanation accurately represents the reasoning process behind a model prediction. Lanham et al, motivated by the benefit of using CoTs for interpretability if they are in fact faithful to a model's reasoning, define a metric to measure faithfulness. This metric counts how often a zero-shot prediction is the same as a CoT prediction, the more often the more unfaithful. This assumes that if a model doesn't need the CoT (i.e. can produce the same answer zero-shot), the CoT will be less faithful. Lanham et al. motivate this metric by showing it correlates with a more complex intervention-based metric; if one intervenes on the CoT and the answer changes, the model is more faithful (e.g. intervening by adding a mistake in the CoT, truncating it, paraphrasing it, or adding filler tokens). The submitted paper reproduces the results of Lanham et al's simpler unfaithfulness metric that counts how often an answer changes from zero-shot to CoT for three different LLM families of different sizes. The result is that similar scaling behaviour as Lanham et al report is observed for models that get non-trivial performance. The finding of Lanham et al that faithfulness is related to task difficulty is not observed in the authors experiments. The authors do an additional experiment where they order the multiple-choice questions differently between zero-shot and CoT prompts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Necessary**: The results as an average over benmchmarks are really difficult, if not impossible, to interpret like this. The variance between benchmarks seems like a weird thing to report and misleading; to me looking at these plots I first assume variance is too high to make any conclusions. It would be more useful to also report whether it's inverse scaling for each benchmark separately, or only for a few representative benchmarks. If you show the average, I wouldn't show variance over benchmarks. You still have space so I would make an additional plot that breaks down more per benchmark.\n\n**Necessary**: address the issue with the confounder factor introduced by having a different ordering between zero-shot and CoT prompt by reporting how often the model changes it's answer for the same prompt but different ordering of the answer; if this is high, the \"different\" condition cannot say much about unfaithfulness.\n\n**Necessary**: also reproduce the intervention experiments done by Lanham et al. on the open-access LLMs to perform that also for these open-access LLMs, the simpler unfaithfulness metric correlates with conclusions from intervention techniques.\n\n**Minor but necessary**:  It's unclear what \"as a property of all LLMs\" means, this is too broad a claim. Would change to something about it generalizing to different, non-proprietary, LLM families, but not to \"all LLMs\".\n\n**Would make the paper better** Only in figure 3 we see Pythia performs randomly for both zero-shot and CoT. The fact that the unfaithfulness metric doesn't work for random performing models is mentioned by you in the paper, but I believe it needs to be clearer in the text. I would propose an additional plot that first shows the accuracies of the models and only then faithfulness. Alternatively, also in figure 1, I think potentially overlaying accuracy on the same plot (with a right axis) would make sense.\n\n**Comments on writing**\nAbstract: This is personal preference so feel free to ignore, but I would slightly alter the order of information in the abstract such that the reader knows what the paper is about before having to read a few sentences about a related work. E.g. \"Understanding the extent to which Chain-of-Thought (CoT) generations align with a large language model’s (LLM) internal computations is critical for deciding whether to trust an LLM’s output. In this work we replicate findings ..\"\n\nAbstract: I think \"V-shaped scaling behaviour\" is clearer than \"scaling-then-inverse-scaling relationship\" as used in the abstract", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously recommend acceptance of the paper.  The main concerns of the reviewers were addressed by the author response and revisions of the paper.
Decision: Accept (poster)