Measuring Faithfulness in Chain-of-Thought Reasoning
Authors: name: Tamera Lanham; name: Anna Chen; name: Ansh Radhakrishnan; name: Benoit Steiner; name: Carson Denison; name: Danny Hernandez; name: Dustin Li; name: Esin Durmus; name: Evan Hubinger; name: Jackson Kernion; name: Kamilė Lukošiūtė; name: Karina Nguyen; name: Newton Cheng; name: Nicholas Joseph; name: Nicholas Schiefer; name: Oliver Rausch; name: Robin Larson; name: Sam McCandlish; name: Sandipan Kundu; name: Saurav Kadavath; name: Shannon Yang; name: Thomas Henighan; name: Timothy Maxwell; name: Timothy Telleen-Lawton; name: Tristan Hume; name: Zac Hatfield-Dodds; name: Jared Kaplan; name: Jan Brauner; name: Samuel R. Bowman; name: Ethan Perez
Published: 2023-07-17 01:08:39

Abstract: Large language models (LLMs) perform better when they produce step-by-step,
"Chain-of-Thought" (CoT) reasoning before answering a question, but it is
unclear if the stated reasoning is a faithful explanation of the model's actual
reasoning (i.e., its process for answering the question). We investigate
hypotheses for how CoT reasoning may be unfaithful, by examining how the model
predictions change when we intervene on the CoT (e.g., by adding mistakes or
paraphrasing it). Models show large variation across tasks in how strongly they
condition on the CoT when predicting their answer, sometimes relying heavily on
the CoT and other times primarily ignoring it. CoT's performance boost does not
seem to come from CoT's added test-time compute alone or from information
encoded via the particular phrasing of the CoT. As models become larger and
more capable, they produce less faithful reasoning on most tasks we study.
Overall, our results suggest that CoT can be faithful if the circumstances such
as the model size and task are carefully chosen.