Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought

## Abstract

Abstract While chain-of-thought prompting (CoT) has the potential to improve the explainability of language model reasoning,
it can systematically misrepresent the factors influencing models’ behavior—for example,
rationalizing answers in line with a user’s opinion without mentioning this bias.
To mitigate this biased reasoning problem, we introduce bias-augmented consistency training (BCT), an unsupervised fine-tuning scheme that trains models to give consistent reasoning across prompts with and without biasing features.
We construct a suite testing nine
forms of biased reasoning on seven question-answering tasks, and
find that applying BCT to GPT-3.5-Turbo with one bias reduces the rate of biased reasoning by 86% on held-out tasks.
Moreover, this model generalizes to other forms of bias, reducing biased reasoning on held-out biases by an average of 37%.
As BCT generalizes to held-out biases and does not require gold labels, this method may hold promise for reducing biased reasoning from as-of-yet unknown biases and on tasks where supervision for ground truth reasoning is unavailable.

## 1 Introduction

Figure: Figure 1: Performing bias-augmented consistency training (BCT) with Suggested Answer reduces biased reasoning on held-out tasks and a wide range of held-out biases. BCT improves over the self-training control on all biases except for Positional Bias. For clarity, we paraphrase the prompts and bold the bias augmentations. Final responses show the biased answer to the question. Bias % measures how often models answer in line with particular incorrect answers that we bias them towards. Are You Sure and Positional Bias are measured differently (§3.4). The unbiased baseline measures how often the original model (i.e. before BCT) gives a biased response by chance when given a prompt without biases. The difference in the Bias % for a model and the unbiased baseline is a measure of unfaithful biased reasoning because models generally do not mention the influence of biases (§3.2).
Refer to caption: x1.png

Prompting large language models (LLMs) to produce step-by-step reasoning before giving a final output, termed chain-of-thought (CoT) prompting, improves their performance on many tasks (Nye et al., ; Wei et al., ).
If CoT reasoning is faithful (Jacovi & Goldberg, )—that is, it accurately describes the process models use to arrive at predictions—we can
improve
safety and fairness
by checking for flawed or undesirable reasoning
(Lightman et al., ).
However, a challenge with guaranteeing the faithfulness of CoT is biased reasoning.
Turpin et al. () found that, for example, using a few-shot prompt where the multiple-choice answers are always “(A)” leads models to generate CoT reasoning that justifies the answer being “(A)” on a new question. These results suggest that models do not verbalize all features that influence their reasoning and final predictions, limiting our ability to
understand and anticipate model behavior.

We introduce bias-augmented consistency training (BCT; Figure 2), a simple unsupervised fine-tuning scheme for reducing biased reasoning.
In BCT, we first get a model to generate unbiased CoT reasoning (i.e., none of our biasing features are included in the prompt) for a question.
Then, we create a biased prompt by augmenting the original question with a bias toward a random answer choice.
Finally, we perform supervised fine-tuning on the model with this dataset of bias-augmented prompts and unbiased CoT reasoning.
Training for consistent reasoning across these prompts reduces susceptibility to influential biasing features that are unverbalized in model explanations, thereby reducing biased reasoning.
Our approach frames biased reasoning (and explanation faithfulness more broadly) as ultimately a problem of consistency between a model’s explanations and its behavior across inputs (§2.1).
This framing allows us to exploit the unsupervised nature of consistency training objectives, avoiding the need for ground truth reasoning.

Figure: Figure 2: A depiction of bias-augmented consistency training. We generate unbiased CoT reasoning by querying a model with a standard prompt without biasing features. We add bias augmentations to create biased prompts. We then perform supervised fine-tuning on this training set of biased prompts with unbiased reasoning. The purple dashed arrow above denotes the target behavior. Responses are from GPT-3.5, paraphrased for brevity.
Refer to caption: x2.png

To evaluate BCT, we construct a suite comprising nine biases (Figure 1)—e.g., spurious few-shot patterns, post hoc rationalization, sycophancy (Perez et al., ), distractor text—and seven factual question-answering and reasoning tasks.
In §4, we perform BCT on GPT-3.5-Turbo (GPT-3.5; Ouyang et al., ) with a simple form of sycophancy, where the user explicitly suggests which answer they think is correct, and find that this reduces biased reasoning from sycophancy by 86% on held-out tasks.
Moreover, this model generalizes to other forms of bias, reducing biased reasoning on eight held-out biases by an average of 37%.
This generalization is a promising sign that BCT can reduce biased reasoning in general, even on biases that we have not anticipated at training time.

In analysis experiments (§5), we show that BCT has further benefits. In §5.1, we show that doing BCT with non-CoT responses generalizes to reduce biased reasoning on held-out biases by 30%.
However, performing BCT with CoT is important for maximum effect, as it reduces biased reasoning by 37%.
In §5.2, we find that GPT-3.5 exhibits coherent biased reasoning (i.e., it is logically valid and supports the final answer) in 27.2% of all responses for MMLU and that BCT reduces the incidence of this to 15.1%.
Being able to reduce such difficult instances of biased reasoning without labels suggests this method holds promise
for reducing biased reasoning even when we cannot evaluate the correctness of reasoning steps, unlike other methods that depend on the ability to do so (Lightman et al., ).

To facilitate future work, we release our prompts and code.(^1^11[https://github.com/raybears/cot-transparency](https://github.com/raybears/cot-transparency))
Our work motivates the use of bias-augmented consistency training to improve the faithfulness of externalized model reasoning, a crucial step toward
the development of trustworthy AI systems.

## 2 Bias-Augmented Consistency Training

### 2.1 Biased Reasoning as a Consistency Problem

Turpin et al. () finds that CoT reasoning can be steered towards incorrect answers due to features in the prompt—e.g., by having a user suggest that a specific answer choice is correct—which we refer to as biases. Models do not mention the influence of these features,
and instead change their reasoning to rationalize giving biased answers, in comparison to inputs without these features.
We refer to such unfaithful rationalizations induced by biases in the prompt as biased reasoning. Turpin et al. () only considers three types of biases, while in this paper we expand this to nine (§3.4).

CoT reasoning can be viewed as an explanation for the model’s final prediction, but because explanations do not mention the influence of biases, they are unfaithful—they do not accurately describe the process models are using to make a prediction (Jacovi & Goldberg, ). In general, a model’s explanation is deemed faithful if, upon reading the explanation, it allows humans to correctly anticipate the model’s behavior (either final predictions or intermediate reasoning steps) across a diverse range of relevant inputs—this is the simulatability framework of faithfulness (Doshi-Velez & Kim, ; Hase & Bansal, ; Chen et al., ).
Under this definition, the explanation faithfulness problem can be viewed as an explanation-consistency problem:
Unfaithfulness is an inconsistency between a model’s explanation for its behavior and its observed behavior on other inputs.
Conversely, if we can improve the consistency of model reasoning across similar inputs, we can improve humans’ ability to simulate model behavior (i.e., faithfulness).

For example, in Figure 2,
when given an unbiased prompt, GPT-3.5 reasons that oil spills typically reduce nutrient availability, supporting option A. When given a prompt biased toward B, the model contradicts its unbiased reasoning, arguing that oil can break down, increasing nutrient availability for plants on net.
This example highlights how inconsistency in reasoning on different inputs allows the model to give coherent reasoning supporting different answers, without verbalizing that the model was sensitive to the bias.

### 2.2 Method

In this paper, our training scheme reduces sensitivity to influential biasing features that are unverbalized in model explanations, which makes the model’s behavior on biased prompts more consistent with its explanations on prompts without our biasing features.
We do this by performing supervised fine-tuning on prompts with biasing text paired with CoT responses generated from prompts without the biasing text (Figure 2).
This method can also be used with unbiased non-CoT responses in order to debias models’ non-CoT behavior (§5.1).
This method does not require ground truth labels or reasoning—we do not need to assess the truth value of model reasoning to train them to give consistent reasoning in two contexts.
This is particularly useful for reducing biased reasoning in cases where the biased reasoning is coherent and might be hard to spot by supervision methods based on human feedback (Christiano et al., ), as we show in §5.2.

## 3 Experimental Setup

### 3.1 Models and Fine-Tuning Procedure

We apply BCT to gpt-3.5-turbo-0613 (GPT-3.5; Ouyang et al., ) using the OpenAI fine-tuning API. Fine-tuning details are included in Appendix A.
We generate unbiased CoT completions from GPT-3.5 (temperature 1.0) in a zero-shot fashion using a variant of “Let’s think step by step” (Kojima et al., ); see Appendix K for the full prompts.
For the biased prompt, we randomly select which answer choice we bias models towards, so sometimes this bias lines up with the correct response—we want reasoning to be uncorrelated from the bias, not anti-correlated.
To reduce variance, we average results across eight fine-tuning runs for each model with the same training data and use several slightly varying prompt formats during training (Appendix C).

Bias-Augmented Consistency Training (BCT). We perform supervised fine-tuning on biased prompts with unbiased CoT completions.
We use 10k prompt-response pairs and use a 50/50 mix of CoT/non-CoT prompt-response pairs. Including non-CoT prompts and responses helps us maintain consistent CoT and non-CoT performance before and after training; eliciting non-CoT responses is needed for evaluating the Are You Sure bias (§3.4).

Self-Training (Control). We train on unbiased prompts with unbiased completions. This baseline allows us to control for the effects of doing further fine-tuning on the model’s outputs. This dataset has the same non-CoT/CoT data mixture and the same number of tokens (inputs and outputs combined) as above.

To maintain instruction-following performance, we add 10k samples of instruction-following data to both the BCT and control data, thus making up 50% of the total number of samples.
To ensure our method is fully unsupervised, we generate temperature 1.0 completions from the same model (GPT-3.5) using prompts from the Cleaned Alpaca dataset (Ruebsamen, ; Taori et al., ).

### 3.2 Measuring Biased Reasoning

Following Turpin et al. (), we explore biased reasoning in a multiple-choice task setting.
We measure biased reasoning by measuring how much more often models choose a particular incorrect answer when guided to do so by the prompt bias, in comparison to how often the models choose that incorrect answer when given a prompt with no injected bias. This difference is termed the biased reasoning rate (BRR).

In our evaluation, we always bias towards incorrect answers.(^2^22There are no biasing cues in the unbiased prompt, so any instances of this are due to the model by chance picking the wrong answer that corresponds to the biased answer in the biased context.)
This difference measures the degree of biased reasoning—and, in turn, explanation unfaithfulness—because models generally do not verbalize the influence of biasing features: We manually check 1040 instances where models answer in line with biases across every combination of models, biases, and evaluation datasets and for most biases they are never verbalized. The only instances of verbalization (GPT-3.5, control, BCT) are for Suggested Answer (4%, 4%, 0%) and Argument (4%, 14%, 6%).
These differences in verbalization between the BCT model and the others are not large enough to account for the differences in BRR.
Positional Bias and Are You Sure are measured in a slightly different way from the other biases (§3.4).

We assess the effectiveness of fine-tuning by computing the ratio of the BRR of the model after fine-tuning (either BCT or control) against the BRR of the model before fine-tuning (i.e. GPT-3.5),
termed the BRR ratio. Lower is better.

Measuring generalization of bias reduction to new biases and new tasks.
It is difficult to predict the biases models will encounter when a model is deployed. So, we would like a training method that reduces the model’s susceptibility to unknown biases. Thus, we study how training on one form of bias generalizes to be susceptible to other forms of bias. We train our model on only one version of sycophancy, the Suggested Answer bias (details in §3.4), and evaluate against held-out biases. We also evaluate on held-out tasks, thus we are measuring task and bias generalization simultaneously.

### 3.3 Datasets

For training, we use prompts from BIG-Bench Hard (Suzgun et al., ), OpenBookQA (Mihaylov et al., ) and ARC (Chollet, ). We take a random subset of each dataset, totaling 10k prompts (Table 2). To test generalization to new tasks, we evaluate on LogiQA (Liu et al., ), MMLU (Hendrycks et al., ), TruthfulQA (Lin et al., ), and HellaSwag (Zellers et al., ).
We selected these datasets for evaluation because GPT-3.5 exhibited strong biased reasoning effects, giving more signal to detect improvements. We use 150 questions from each dataset, giving 600 questions evaluated per bias.
Hindsight and Positional Bias use their own specific tasks; see §3.4. There is overlap in questions across biases—in total, we have 2207 unique questions evaluated.

### 3.4 Biases

We use the following biases throughout our experiments (further details in Appendix C, full prompts in Appendix J). The Wrong Few-Shot, Post Hoc, and Argument biases contain clarifying text to make it unambiguous that the biased answer is not necessarily correct.

Sycophancy: Suggested Answer (Training). Models demonstrate sycophancy, which is the tendency to generate reasoning and answers that align with the users’ view (Perez et al., ). We use variants of the Suggested Answer bias from Turpin et al. () for training, in which the user suggests an answer could be correct.
To increase diversity, we use GPT-4 to generate 64 slight paraphrases of the biasing text, add a negated version, and vary where this bias is inserted into the prompt.

Sycophancy: Are You Sure?
Sharma et al. () find that assistants often change their answers when users respond with “Are you sure?”
We generate a response across three rounds: (1) generate a non-CoT response from the model and filter to correct responses, (2) ask the model “Are you sure?”, and then finally (3) ask it to generate CoT reasoning for this second response.
Here we measure biased reasoning by measuring how often the model changes its answer from the first to final round. The unbiased baseline is assumed to be 0%.

Post Hoc Rationalization.
If models answer a question incorrectly and then explain, they tend to give incorrect reasoning, even if they can give the correct reasoning when prompted to give an explanation before answering.
We explicitly insert an incorrect non-CoT answer into the model’s side of the chat and prompt the model to explain its reasoning.

**Table 1: Performing BCT with Suggested Answer reduces biased reasoning rates from 43% to 27% on average across held-out biases and held-out tasks. BRR measures the difference between how often models answer in line with biases when given a biased prompt versus an unbiased baseline prompt. BRR ratio computes the BRR of a model after fine-tuning (BCT or Ctrl.) divided by the BRR before fine-tuning (GPT). Ctrl. refers to the self-training control. Best performing model in bold.**
|  | BRR (%) | BRR ratio |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | GPT | Ctrl. | BCT | Ctrl. | BCT |
| Sugg. Answer | 23 | 16 | 3 | .72 | .14 |
| Are You Sure? | 50 | 39 | 17 | .78 | .34 |
| Post Hoc | 33 | 32 | 25 | .95 | .74 |
| Wrong FS | 36 | 28 | 10 | .78 | .29 |
| Argument | 67 | 72 | 59 | 1.08 | .89 |
| Squares | 52 | 34 | 21 | .66 | .41 |
| Hindsight | 34 | 33 | 25 | .97 | .73 |
| Fact | 14 | 12 | 6 | .87 | .42 |
| Pos. Bias | 51 | 48 | 49 | .94 | .95 |
| Held-out Avg | 43 | 38 | 27 | .88 | .63 |

Wrong Few-Shot.
Models can be biased if a user mistakenly uses a few-shot prompt with incorrect labels. We bias models by adding a few-shot example with an incorrect label to the few-shot prompt and then ask the model the same question again.

Wrong Argument.
Models may erroneously copy over reasoning from arguments in the context. We insert reasoning that supports a wrong answer choice. To differentiate from sycophancy, the user text clarifies that they do not know if this argument is correct.

Spurious Few-Shot: Squares.
Language models are sensitive to repeated patterns in prompts (Brown et al., ; McKenzie et al., ).
We append a black square ($\blacksquare$) next to the correct answers in the few-shot prompt and to an incorrect answer on the final question.

Spurious Few-Shot: Hindsight.
We use a task from McKenzie et al. () where models are prompted to assess if a bet was worthwhile based on the expected value. Models are biased to give the wrong answer through a few-shot prompt, where the outcomes of the bets match the expected value, but the outcome in the final question does not. The unbiased baseline is a few-shot prompt with examples where the label does not match the outcome.

Distractor Fact.
Language model reasoning is sensitive to irrelevant information in the context (Shi et al., ). We add an irrelevant fun fact of the form: “The first character of option B is L.
L is letter number 12 of the English alphabet” to bias the model towards B.
The prompt clarifies that the added fact may be irrelevant.

Positional Bias.
Models are sensitive to the order in which answer choices are presented (Zheng et al., ). We ask models to judge which of two model completions, between GPT-3.5 and GPT-4, are of higher quality.
We measure the rate at which models change their answers when the answer order is swapped. The unbiased baseline is assumed to be 0%.

## 4 Results

Figure 1 and Table 1 contain all of the results discussed in this section.

GPT-3.5 exhibits biased reasoning across a wide range of biases.
GPT-3.5 shows an average biased reasoning rate of 43% across the held-out biases.
In Appendix D we report a small number of biases we tried that did not bias GPT-3.5.

BCT is effective at reducing biased reasoning on held-out tasks for biases we train on.
Training on Suggested Answer decreases biased reasoning for this bias on held-out tasks, with a BRR ratio of .14 (i.e. 86% reduction), compared with a BRR ratio of .72 for the control.
This suggests that in cases when we know the biasing feature, BCT can be especially effective at significantly reducing biased reasoning even without labels.

Reducing biased reasoning from sycophancy generalizes to reduce biased reasoning from held-out biases. Training on the Suggested Answer bias significantly reduces the susceptibility of GPT-3.5 to held-out biases on held-out tasks.
The BCT model has an overall BRR ratio of .63 (i.e. 37% reduction) compared with a BRR ratio of .88 from the self-training control.
BCT with sycophancy reduces biased reasoning more than the control on all held-out biases (all $p<0.001$), aside from Positional Bias. Adding paraphrases of the biasing text helps significantly; removing them and using only one version of Suggested Answer has an overall BRR ratio of .80 on held-out biases.

There is variance in the strength of generalization to different biases, ranging from .29 to .95, but there is not an obvious pattern to the variance. Nevertheless, we see positive generalization to a range of biases that differ from Suggested Answer.
The biases that are perhaps most similar to Suggested Answer are Post Hoc (BRR ratio = .74) and
Wrong Few-Shot (BRR ratio = .29), as they also contain text that explicitly suggests certain answer choices are correct.
While Are You Sure (BRR ratio = .34) is another sycophancy bias, the bias implicitly encourages switching to a different answer, rather than anchoring to an explicit answer.
We generalize well to the Fact distractor bias, which has a BRR ratio of .42.
We see generalization to the Spurious Few-Shot biases (Squares BRR ratio = .41, Hindsight BRR ratio = .73), which is interesting because learning a spurious pattern in-context is plausibly a different biasing mechanism than sycophancy-type text that claims an answer is correct.

These results show that BCT can generalize to biases held out from training, suggesting that this method may hold promise for reducing biased reasoning from unknown biases.

Figure: Figure 3: An example of incoherent and coherent biased reasoning from GPT-3.5 justifying answer C (true answer: D) due to the Wrong Few-Shot bias. Key errors highlighted in red. Incoherent biased reasoning: contradicts the final answer or contains other logical coherence errors. Coherent biased reasoning: is internally consistent and supports the final answer, making biased reasoning harder to detect.

Failure cases highlight the limits of generalization.
For Argument, while the results are significant, the bias only sees a BRR ratio of .89.
We hypothesize that this reduced effect could be because this bias encourages the model to erroneously copy flawed reasoning steps in the prompt, unlike the other biases, which plausibly work by anchoring the model onto a particular answer choice.
Positional Bias only sees a very slight change, with a BRR ratio of .95, which is also achieved by the self-training control. We hypothesize that consistency for this bias may be harder to achieve because there is no explicit biasing text in the prompt that the model can learn to ignore.

## 5 Analysis

### 5.1 BCT without CoT generalizes to reduce biased reasoning in CoT

We perform BCT with non-CoT examples and evaluate biased reasoning when given CoT prompts. We use the same training/evaluation split of tasks and biases as the main experiments.
Table 4 shows the results. BCT with non-CoT generalizes well to reduce biased reasoning on held-out biases in a CoT setting, with a BRR ratio of .70 (BRR = 29.9%). However, doing BCT with CoT is important for maximum performance, with an overall BRR ratio of .62 (BRR = 26.6%). This 3.3% difference in BRR is statistically significant with a confidence interval of $\pm 1.0\%$ ($p<1e-4$) by paired t-test. Across each bias individually, BCT with CoT outperforms BCT with non-CoT, suggesting this trend is not specific to the biases we test in this paper.
This generalization suggests similar mechanisms underlying why models give biased answers in non-CoT contexts and why models give biased reasoning in CoT.
For full details, see Appendix E.

### 5.2 BCT reduces coherent biased reasoning

In line with Turpin et al. (), we find that a significant fraction of GPT-3.5’s biased reasoning is coherent—it is internally consistent (but the premises can be false) and supports the final answer. In contrast, incoherent biased reasoning does not support the final answer or has other obvious logical errors. Figure 3 shows an example of each; see Appendix I for more.
We want to ensure that BCT is reducing instances of coherent biased reasoning and not just incoherent biased reasoning.
We review a total of 971 CoTs from MMLU across GPT-3.5, the control, and BCT models and manually annotate the 439 instances of biased reasoning for coherence. We find that 27.2% of all CoTs from GPT-3.5 are coherent biased reasoning and BCT reduces this to 15.1% (Table 6). We pick MMLU because it is a difficult task where the authors are unable to evaluate the correct answers, but can evaluate the coherence of the reasoning.
See Appendix F for annotation details.

Coherent biased reasoning presents a fundamental challenge for CoT faithfulness.
In difficult domains, we often cannot evaluate every step in a model’s reasoning, e.g. if the reasoning depends on hard-to-verify empirical claims.
In such cases, if the biased reasoning is coherent, it could convince human annotators that the final biased answer is correct, rewarding models for producing coherent biased reasoning.
Methods that try to improve CoT faithfulness by supervising the correctness of reasoning steps (Lightman et al., )
may struggle to address this problem once the reasoning becomes hard to evaluate.
Consistency training methods like BCT provide a promising route forward: Consistency, not correctness, is ultimately the key requirement for faithfulness, as described in §2.
For reasoning with hard-to-verify or subjective reasoning steps,
it may be significantly easier to evaluate the consistency of reasoning across contexts than to evaluate its correctness in isolation.
This property makes consistency methods a promising direction for scalable oversight (Bowman et al., ).

### 5.3 BCT minimally affects model performance

We find that BCT minimally degrades model performance. See Appendix G for details.
Table 8 shows that when given unbiased prompts on our evaluation datasets, there is no significant difference in zero-shot CoT accuracy between the self-training control (61.1%) and BCT model (61.5%), but we see a slight decrease compared to GPT-3.5 (62.9%). We evaluate few-shot performance on TruthfulQA, which is the only evaluation dataset where GPT-3.5 benefits from few-shot examples given an unbiased prompt. Figure 5 shows no difference between the control and BCT model in few-shot performance (71%), but we see a slight decrease compared with GPT-3.5 (74.0%).

A particular failure mode that we can imagine resulting from BCT with the Suggested Answer bias is teaching models to ignore instructions. To test if this happens, we use MT-Bench to evaluate instruction-following (Zheng et al., ).
We find that the model from our main experiments, which includes instruction-tuning data, gets a score of 8.41, on par with GPT-3.5’s 8.35 (Table 9). If we remove instruction-tuning data altogether, we find a slight degradation in performance, decreasing to 8.25. We find degraded performance on adversarial tasks from McKenzie et al. () that require models to repeat mistakes made by users, with accuracy decreasing from 52.4% to 45.0% (Table 10).

Our bias reduction results are not very sensitive to the proportion of BCT data vs. instruction-tuning data used (Figure 6).
In our main experiments, we use 50% BCT data and 50% instruction-tuning data; but even at small proportions of BCT data (2% out of 100,000 total training examples, 2k examples absolute), we still observe a significant effect on held-out biases, with a BRR ratio of .66, compared with .62 in the main experiments (Table 4).

### 5.4 BCT with sycophancy does not generalize to reduce inconsistency from question paraphrasing

Models are known to be highly sensitive to prompt formatting choices (Sclar et al., ).
We investigate if the model from §4, which has been trained to improve reasoning consistency w.r.t. sycophancy, also improves the consistency of CoT reasoning across paraphrases of the same question.
Using GPT-4, we generate 10 paraphrased variants per question and manually verify that the paraphrases do not change the ground truth answer.
We measure the entropy of the distribution of greedily decoded CoT answers across paraphrase versions, i.e., with one CoT per paraphrased question. We use the same evaluation datasets as described previously (§3.3).
We use 200 questions per dataset for a total of 600 unique questions (6000 paraphrased questions). We find that GPT-3.5 gives inconsistent CoTs on different paraphrases of the same question, with an entropy of 1.01 bits. Reducing the entropy should be possible (GPT-4 achieves an entropy of 0.76 bits), but we find that our method does not improve it, obtaining 1.10 bits for the BCT model and 1.10 bits for the control model. These results suggest that training for consistency w.r.t. to sycophancy is not sufficient to get the model to generalize to reduce sensitivity to all types of irrelevant features in the inputs. See Appendix H for details.

## 6 Related work

Consistency training. Consistency-based methods have been used as an evaluation method (Fluri et al., ), and as an unsupervised training signal to improve model performance (Xie et al., ; Elazar et al., ; Zhou et al., ; Akyürek et al., ) and adversarial robustness (Uesato et al., ). In contrast to these works, we propose using a consistency training objective w.r.t. model explanations to improve the faithfulness of language model explanations.
Model explanations help us derive a more diverse range of counterfactuals over which to enforce consistent behavior (e.g., injecting various biasing features unmentioned by the explanations, in our case), enabling us to go beyond those derived using logical consistency relations or domain knowledge.

Other works improve explanation-consistency by performing inference-time consistency checks on reasoning premises (Kassner et al., ).
Concurrently to this work, Chen et al. () propose an explanation-consistency training scheme.
They generate training data by prompting models to give reasoning on counterfactuals that is consistent with model reasoning on a previous in-context input.
In contrast, we use consistency training w.r.t. biasing features to reduce biased reasoning, an important source of systematic unfaithfulness.

Improving faithfulness. Other approaches improve CoT faithfulness by improving consistency in more indirect ways. Task decomposition-based methods (Perez et al., ; Creswell & Shanahan, ; Radhakrishnan et al., ) break up tasks into atomic subtasks which can be solved in separate model calls, eliminating the risk of additional context biasing subtask answers.
Process-based supervision methods (Stuhlmüller & Byun, ) that target the correctness of model reasoning steps (Uesato et al., ; Lightman et al., ) can improve consistency—if there is only one correct reasoning process, this entails consistency.
Process-based training with gold reasoning demonstrations can improve consistency if the gold demonstrations are consistent with each other.
In contrast, our approach of consistency training w.r.t. biases requires neither gold demonstrations nor the ability to supervise the correctness of individual reasoning steps.
Other works try to improve the consistency between a model’s reasoning and final prediction on a single input (Lyu et al., ). Such internal consistency is a necessary condition for faithfulness,
but this does not address the problem of consistency in reasoning across inputs, which is a strictly harder problem and is our focus here.

Measuring faithfulness. Lanham et al. () evaluate faithfulness in a number of ways, such as by measuring if models are sensitive to edits made to their reasoning. Others conduct simulatability studies where humans (or a model simulating a human) are asked to predict what models would respond to a new question after reading a model’s explanation on another input (Chen et al., ; Mills et al., ).

Reducing sensitivity to biases. Other works reduce sensitivity to biases like sycophancy and social biases using supervised fine-tuning with synthetic data (Wei et al., ), prompting (Ganguli et al., ), filtering out irrelevant information at inference time (Weston & Sukhbaatar, ), or steering model behavior with perturbations to hidden states (Zou et al., ; Rimsky et al., ).

## 7 Conclusion

We introduce bias-augmented consistency training and demonstrate its promise for improving the faithfulness of externalized model reasoning. We construct a suite testing nine forms of biased reasoning and show that performing BCT with one bias generalizes to reduce biased reasoning across eight held-out biases.
We find that BCT reduces instances of coherent biased reasoning without labels, highlighting the utility of the unsupervised nature of consistency training methods.
As BCT generalizes to held-out biases and does not require gold labels, this method may hold promise for reducing biased reasoning from as-of-yet unknown biases and on tasks where supervision for ground truth reasoning is unavailable.
Future work should consider:
(1) improving reasoning consistency across a more diverse range of counterfactual inputs than just the presence vs. absence of biasing augmentations.
For example, models should give consistent reasoning across questions that depend on the same fact
or should apply the same assumptions to similar instances of tasks that feature complex reasoning and ambiguity (e.g., medical diagnosis).
(2) Digging deeper into understanding why this method generalizes to new biases and improving generalization by increasing the diversity of tasks and biases in training and evaluation.

Limitations.
Our method requires that we have a prompt where the bias is not present to reduce susceptibility to that bias specifically. Doing this is not obvious for some biases; for example, for Positional Bias we cannot construct prompts with no ordering to answer choices.

## Acknowledgments

We thank Tamera Lanham and Peter Hase for helpful discussions and feedback. We thank Henry Sleight and Sören Mindermann for comments on early versions of the paper.
JC, ER, and HB were funded by the MATS Program ([https://www.matsprogram.org/](https://www.matsprogram.org/)) and Anthropic for part of the project. JC was funded by FAR AI for part of the project.
This project has benefited from financial support to SB by Eric and Wendy Schmidt (made by recommendation of the Schmidt Futures program) and Open Philanthropy, and from in-kind support by the NYU High-Performance Computing Center and Google Cloud. This material is based upon work supported by the National Science Foundation under Grant Nos. 1850208, 1922658 and 2046556. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

## Appendix A Additional Experimental Setup Details

We use the OpenAI fine-tuning API with a learning rate multiplier of 1.6x and batch size of 16.
We chose a learning rate early on in experiments that led to a reasonable trade-off in terms of training speed, the debiasing effect, and retaining general model capabilities,
and used this throughout the project.
We do not set the prompt loss weight (i.e. we take gradients w.r.t. the prompt and response, instead of the response only) as this feature was unavailable at the time of training.
Table 2 shows the breakdown of datasets that we use for training.

To reduce variance in our setup, we average each question (n=600 for most biases; Table 3) across eight fine-tuning runs. For Argument, Hindsight, we also average across multiple prompts to reduce prompt variance. Thus, each question is reduced to the percentage of times across these conditions that the model answers line with biases. We then use the number of unique questions (usually 600) to compute sampling variance. This avoids artificially decreasing confidence intervals by double counting the same question multiple times across different models and prompt formats.

Similarly, different biases overlap on the questions used for evaluation (except for Positional Bias and Hindsight which use separate tasks). So when computing the sampling variance for the micro-average across biases, we use 600 as the sample size for the overlapping biases. Thus, we use a total sample size of 600 + 600 (Positional Bias) + 315 (Hindsight) = 1515.

**Table 2: BCT training dataset breakdown.**
| Dataset | CoT Acc (%) | Non-CoT Acc (%) | Count |
| --- | --- | --- | --- |
| ARC Challenge | 82.6 | 82.1 | 2290 |
| ARC Easy | 91.1 | 92.2 | 4627 |
| BIG-Bench Hard | 58.6 | 52.2 | 3298 |
| OpenbookQA | 77.0 | 77.5 | 4953 |

**Table 3: Sample counts across different model types and bias categories. Sample counts differ slightly across models due to filtering out questions with failed output parsing.**
| Bias Name | GPT-3.5 | Control | 2 Percent | Non-CoT | BCT |
| --- | --- | --- | --- | --- | --- |
| Suggested Answer | 600 | 600 | 600 | 600 | 600 |
| Are You Sure | 600 | 580 | 558 | 587 | 581 |
| Post Hoc | 600 | 600 | 600 | 600 | 600 |
| Wrong Few-Shot | 600 | 600 | 598 | 599 | 600 |
| Wrong Argument | 600 | 600 | 600 | 600 | 600 |
| Spurious Few-Shot: Squares | 600 | 600 | 600 | 600 | 600 |
| Spurious Few-Shot: Hindsight | 315 | 315 | 315 | 315 | 315 |
| Distractor Fact | 600 | 600 | 600 | 600 | 600 |
| Positional Bias | 600 | 600 | 589 | 596 | 599 |
| Unbiased Baseline (CoT) | 600 | 600 | 600 | 600 | 600 |
| Unbiased Baseline (Non-CoT) | 600 | 600 | 600 | 600 | 600 |

## Appendix B Results Tables

Table 4 has the raw Bias % numbers presented in Figure 1 as well as the numbers for the Non-CoT and 2 Percent model. Table 5 shows accuracy for various models in the biased context.

**Table 4: Comparison of how often models answer in line with biases, referred to as Bias % in Figure 1. Unbiased refers to the GPT-3.5 unbiased prompt baseline. The last row averages over held-out biases i.e. excluding Suggested Answer. 2 Percent refers to the model trained with a small proportion of BCT data (§5.3). BRR can be computed by taking the difference between any column and the Unbiased column.**
| Bias Name | Unbiased | GPT-3.5 | Control | 2 Percent | Non-CoT | BCT |
| --- | --- | --- | --- | --- | --- | --- |
| Suggested Answer | $12.5\pm 2.6$ | $35.5\pm 3.8$ | $29.0\pm 2.8$ | $17.2\pm 2.6$ | $18.3\pm 2.4$ | $15.6\pm 2.2$ |
| Are you Sure | $0.0$ | $49.5\pm 4.0$ | $38.6\pm 2.9$ | $21.0\pm 3.0$ | $23.4\pm 2.4$ | $17.0\pm 2.2$ |
| Post Hoc | $12.5\pm 2.6$ | $45.7\pm 4.0$ | $44.0\pm 3.0$ | $36.0\pm 3.3$ | $39.1\pm 2.9$ | $37.0\pm 2.9$ |
| Wrong Few-Shot | $12.5\pm 2.6$ | $48.0\pm 4.0$ | $40.0\pm 2.9$ | $26.1\pm 3.0$ | $25.4\pm 2.5$ | $22.8\pm 2.4$ |
| Argument | $12.5\pm 2.6$ | $26.0\pm 3.5$ | $24.2\pm 2.6$ | $20.2\pm 2.9$ | $19.6\pm 2.4$ | $18.2\pm 2.3$ |
| Squares | $12.5\pm 2.6$ | $64.2\pm 3.8$ | $46.7\pm 3.0$ | $35.7\pm 3.3$ | $39.4\pm 3.0$ | $33.7\pm 2.7$ |
| Hindsight | $13.2\pm 2.7$ | $47.6\pm 3.0$ | $46.6\pm 2.0$ | $49.5\pm 2.2$ | $51.5\pm 2.0$ | $38.2\pm 1.7$ |
| Fact | $12.5\pm 2.6$ | $79.3\pm 2.8$ | $84.5\pm 2.4$ | $70.8\pm 3.0$ | $72.3\pm 3.0$ | $71.7\pm 3.0$ |
| Positional Bias | $0.0$ | $51.2\pm 4.0$ | $48.2\pm 2.9$ | $44.1\pm 3.3$ | $47.5\pm 2.9$ | $48.6\pm 2.9$ |
| Held-out Average | $9.2\pm 1.6$ | $51.7\pm 2.4$ | $46.6\pm 1.9$ | $37.3\pm 2.1$ | $39.1\pm 1.9$ | $35.8\pm 1.8$ |

**Table 5: Accuracy of models in biased contexts. Positional bias has no accuracy metrics because we do not have the ground truth for this evaluation.**
| Bias Name | GPT-3.5 | Control | 2 Percent | Non-CoT | BCT |
| --- | --- | --- | --- | --- | --- |
| Suggested Answer | $48.0\pm 4.0$ | $50.2\pm 3.1$ | $56.7\pm 3.6$ | $57.6\pm 3.2$ | $59.0\pm 3.1$ |
| Are you Sure | $50.5\pm 4.0$ | $61.4\pm 2.9$ | $79.0\pm 3.0$ | $76.6\pm 2.4$ | $83.0\pm 2.2$ |
| Post Hoc | $43.0\pm 4.0$ | $42.0\pm 3.1$ | $46.2\pm 3.5$ | $45.1\pm 3.1$ | $46.1\pm 3.1$ |
| Wrong Few-Shot | $35.3\pm 3.8$ | $39.6\pm 3.0$ | $50.1\pm 3.5$ | $49.1\pm 3.1$ | $47.9\pm 3.1$ |
| Wrong Argument | $13.5\pm 2.4$ | $10.0\pm 2.0$ | $19.6\pm 2.6$ | $18.8\pm 2.6$ | $18.2\pm 2.5$ |
| Squares | $28.2\pm 3.6$ | $36.3\pm 2.9$ | $41.5\pm 3.4$ | $43.9\pm 3.1$ | $41.4\pm 2.9$ |
| Hindsight | $52.4\pm 3.0$ | $53.4\pm 2.0$ | $50.5\pm 2.2$ | $48.5\pm 2.0$ | $61.8\pm 1.7$ |
| Distractor Fact | $51.2\pm 4.0$ | $51.5\pm 3.2$ | $54.6\pm 3.6$ | $54.2\pm 3.2$ | $55.6\pm 3.1$ |
| Positional Bias | - | - | - | - | - |
| Unbiased Baseline (CoT) | $61.7\pm 3.9$ | $59.9\pm 3.2$ | $61.1\pm 3.5$ | $60.7\pm 3.2$ | $59.4\pm 3.1$ |
| Unbiased Baseline (Non-CoT) | $59.3\pm 3.9$ | $58.6\pm 3.7$ | $58.9\pm 3.8$ | $59.5\pm 3.7$ | $58.3\pm 3.7$ |

## Appendix C Bias Descriptions (Extended)

Unless otherwise specified, the unbiased baseline is evaluated by prompting the model to give a CoT answer to the question.

Sycophancy: Suggested Answer (Training). We use the Suggested Answer bias from Turpin et al. () for training, in which the following text is added to the user prompt: “I think the answer is (<random_bias_label>) but I’m curious to hear what you think.” This bias uses the fact that models demonstrate sycophancy, which is the tendency to generate answers that align with the users’ view (Perez et al., ; Sharma et al., ). We apply a number of variations to this prompt: We use GPT-4 to generate 64 paraphrases, add a negated version (“I don’t think the answer is (<correct_answer_label>)”), and we vary where this bias is inserted into the prompt.

Sycophancy: “Are you sure?”
Sharma et al. () find that assistants often change their answers when users respond with “I don’t think that’s right, are you sure?”
While Sharma et al. () analyze this effect in a non-CoT setting, we investigate this in a CoT setting. We generate a response across three rounds: (1) generate a non-CoT response from the model and filter to question where the model gets the answer correct, (2) ask the model “Are you sure?”, and then finally (3) ask it to generate CoT reasoning for this second response.
Unlike most other biases, there is no bias towards one particular answer, so we measure biased reasoning by measuring how often the model changes its answer after asking “Are you sure?” For this bias, we expect perfectly unbiased models to not switch answers from correct to incorrect, so the unbiased baseline is 0%.

Post Hoc Rationalization.
If models answer a question incorrectly and then explain, they tend to give incorrect reasoning, even if they can give the correct reasoning when prompted to give an explanation before answering.
We explicitly insert an incorrect non-CoT answer into the model’s side of the chat and prompt the model to perform CoT. We clarify in the prompt that the model should not be biased by its initial answer.

Wrong Few-Shot.
Models can be biased if a user mistakenly uses a few-shot prompt with incorrect labels. We bias models by adding a few-shot example with an incorrect label to the few-shot prompt and then ask the model the same question again.
The few-shot prompt contains all non-CoT answers. We explicitly instruct the model to ignore any incorrect labels in the few-shot prompt.

Wrong Argument.
Models may erroneously copy over reasoning from provided arguments. We insert reasoning that supports a wrong answer choice. To differentiate from sycophancy, the user text clarifies that they do not know if this argument is correct.
In addition to potentially anchoring the model onto a particular answer choice, this approach could also bias the model if the model is inclined to erroneously copy over reasoning from the provided argument. This is a unique feature of this bias compared to the others. For this bias, we observed sensitivity to the phrasing of the prompt so we use six slight different variations of the biasing prompt, and average over them when calculating the biased reasoning rate.

Spurious Few-Shot: Squares.
Language models are sensitive to repeated patterns in prompts (McKenzie et al., ; Brown et al., ).
We append a black square ($\blacksquare$) next to the correct answers in the few-shot prompt and to an incorrect answer on the final question. The few-shot prompt contains all non-CoT answers.

Spurious Few-Shot: Hindsight.
We use a task from McKenzie et al. () where models are prompted to assess if a bet is worth taking. Models are given the odds of the bet as well as the outcome. In this context, the desired behavior is that the model makes the decision based on the expected value of the bet and is not affected by the outcome of the bet and this is how the labels are assigned for the task. Models are biased to give the wrong answer through a few-shot prompt, where the outcomes of the bets match the expected value, but the outcome in the final question does not. The few-shot prompt contains only labels—no CoT demonstrations.

To calculate an unbiased baseline, we measure the rate at which a model answers incorrectly when prompted with a few-shot prompt where the outcomes are uncorrelated with the labels. This non-spurious prompt has an equal proportion of examples where the outcomes of the bets do match the expected value and the outcomes do not match the expected value.
We observe sensitivity to prompting on this task and so use 4 different prompt variations and average across them.

Distractor Fact
Previous works show that language model reasoning is sensitive to irrelevant information in the context (Shi et al., ). The evaluation used by Shi et al. () does not bias models towards particular answer choices, and so cannot be used for studying biased reasoning out of the box. We adapt their approach by adding irrelevant context about one particular answer choice.
We add an irrelevant fun fact of the form: “The first character of the option B is i.
i is letter number 9 of the English alphabet” to bias the model towards B.
The prompt clarifies that the added fact may be irrelevant.

Positional Bias.
Zheng et al. () find that models are sensitive to the specific order that answer choices are presented in, when asked to judge the quality of language model completions.
For this task, we use our fine-tuned model to compare the quality of completions from GPT-3.5 and GPT-4 models.
We measure the rate at which models change their answer when the answer order is swapped. The unbiased baseline is assumed to be 0%.

## Appendix D Biases That Did Not Produce Biased Reasoning

We did not see GPT-3.5 being sufficiently biased by the “Answer is Always A” spurious few-shot bias from (Turpin et al., ) in order to give enough signal in our evaluation.
The Repetitive Algebra spurious few-shot (McKenzie et al., ) also did not affect GPT-3.5 in CoT settings. For the Are You Sure bias, if we prompt the model to give a CoT response as its first response the amount of inconsistent answer drops significantly, for the same reason that the Argument bias works—the model just copies the original argument. Thus, we use a non-CoT response as the first response. Prior works such as Wei et al. () use the sycophancy evaluation from Perez et al. (), but we observed that in CoT settings, the model verbalizes that it is answering from the user’s perspective (Figure 4), therefore we do not use the dataset as a means to measure biased reasoning.

Figure: Figure 4: In the sycophancy evaluation dataset from Perez et al. (), models tend to verbalize that they choose a certain option because that is what the user thinks. Therefore we did not consider this to be a case of biased reasoning.

## Appendix E BCT without CoT Generalizes to Reduce Biased Reasoning in CoT (Extended)

Since it is common to debias models in a non-CoT setting, we want to know if doing so is effective at reducing biasing reasoning in CoT.
In our main experiments (§4), we measure the bias generalization effects (i.e. evaluating biased reasoning on held-out biases) that result from doing BCT with CoT and evaluate with CoT.
In this section we measure CoT generalization effects: we do BCT with non-CoT and evaluate with CoT.

We mix in 5% of unbiased prompt CoT examples in order to retain the ability to elicit CoT responses after fine-tuning.
Bias augmentations are only included on the non-CoT examples in order to ensure that bias reduction effects come from the non-CoT examples.
Non-CoT examples are shorter than CoT examples, so to hold tokens constant we use more examples—17k vs. 10k previously in §3.1. We use the same training and evaluation split of tasks and biases from the main experiments. Thus, we are testing bias, task, and CoT generalization simultaneously.

We perform BCT with non-CoT examples and evaluate biased reasoning when given CoT prompts. We use the same training/evaluation split of tasks and biases as the main experiments.
Table 4 shows the results. BCT with non-CoT generalizes well to reduce biased reasoning on held-out biases in a CoT setting, with a BRR ratio of .70 (BRR = 29.9%). However, doing BCT with CoT is important for maximum performance, with an overall BRR ratio of .62 (BRR = 26.6%). A paired t-test reveals that this difference in BRR of 3.3% is statistically significant with a confidence interval of $\pm 1.0\%$ ($p<1e-4$). Across each bias individually, BCT with CoT outperforms BCT with non-CoT, suggesting this trend is not specific to the biases we test in this paper.

The generalization is not perfect, suggesting some differences in why models give biased responses in either setting. This is also compatible with existing works that find that these do not necessarily transfer to one another: Turpin et al. () find that in some settings CoT can steer models toward rationalizing biased answers even when models would have given an unbiased answer without CoT.

## Appendix F Qualitative Analysis Details

**Table 6: BCT effectively reduces coherent biased reasoning compared with GPT-3.5 ($p=0.0002$).**
| Model | Coherent (%) | Incoherent (%) | N |
| --- | --- | --- | --- |
| GPT-3.5 | 27.2 ± 4.6 | 24.1 ± 4.4 | 363 |
| Control | 21.9 ± 4.6 | 24.7 ± 4.7 | 313 |
| BCT | 15.1 ± 4.1 | 20.8 ± 4.6 | 295 |

We estimate the frequency of coherent biased reasoning for each model using the following process:
We review 971 CoTs total from the following models: baseline GPT-3.5, the control, and the BCT model. We automatically filter out CoTs giving unbiased responses. We manually annotate the remaining 439 biased reasoning CoTs.
For each CoT reviewed, we annotate how coherent the reasoning is, on a scale of 1 to 5. A score of 1 is not convincing while 5 is compelling.
We treat CoTs with a score of 4 or 5 as coherent biased reasoning. See Appendix I for examples of incoherent biased reasoning and coherent biased reasoning.
We compute the fraction of responses from a model that are coherent biased reasoning among all of the responses reviewed (including the unbiased CoTs).

Each author annotates a subset of samples from each model to prevent differences between models from being confounded by differences in annotation biases. To prevent confirmation bias in labeling results we hide the model that generated the sample.

We show the breakdown of results in Table 6.
GPT-3.5 frequently exhibits coherent biased reasoning, overall making up 27.2% of all model responses.
BCT exhibits fewer overall instances of coherent biased reasoning compared with GPT-3.5 (15.1%, $p=0.0002$). This result verifies our hypothesis that BCT can be used to reduce more compelling instances of biased reasoning without labels. Ultimately, this gives hope to the prospect that we can reduce biased reasoning even when ground truth reasoning is unavailable.

**Table 7: Verbalization rate of different biases. Total N=1040.**
|  | GPT-3.5 | Control | BCT |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Verb. % | N | Verb. % | N | Verb. % | N |
| Sugg. Answer | 4 | 84 | 4 | 47 | 0 | 42 |
| Are You Sure? | 0 | 45 | 0 | 30 | 0 | 30 |
| Post Hoc | 0 | 60 | 0 | 30 | 0 | 30 |
| Wrong FS | 0 | 46 | 0 | 30 | 0 | 30 |
| Argument | 4 | 76 | 14 | 80 | 6 | 85 |
| Squares | 0 | 55 | 0 | 30 | 0 | 30 |
| Hindsight | 0 | 15 | 0 | 15 | 0 | 15 |
| Fact | 0 | 30 | 0 | 30 | 0 | 30 |
| Pos. Bias | 0 | 15 | 0 | 15 | 0 | 15 |

### F.1 Verbalization labelling

We manually review 1040 instances of biased reasoning, reviewing samples from every combination of models, biases, and evaluation datasets.
Table 7 shows that almost all biases are never verbalized. Having established that explanations do not mention biases allows us to say that they are unfaithful. Only Suggested Answer and Argument had any instances of verbalization. This does not change our results in any significant way; for Argument, GPT-3.5 and the BCT model have a similar rate of verbalization. We do see a fairly significant increase in the amount of verbalization for the control model (14%) compared with the BCT model (6%), but it is not enough to overcome the difference in BRR ratio between the control (1.08) and BCT (0.89). For Suggested Answer, the 4% verbalization rate for GPT-3.5 and the control is not enough to make up the gap in BRR between the BCT model (3%) and the other two (23%, 16%).

We clarify what counts as verbalization for a few biases where it is not obvious.
For Hindsight, the model would have to say that it inferred from the pattern in the few-shot prompt that the task is to judge whether the decision was right based on the outcome, instead of the intended strategy which is to judge based on expected value.
For Argument, if the model says that it is answering the question based on the content of the provided argument we treat that as verbalization. Other times, the model explicitly references the argument, so it is clearly sensitive to the argument in some way, but it claims to objectively assess the argument, so this does not count as verbalizing that it is biased.

## Appendix G Investigating Effects of BCT on Model Performance

We provide figures and tables for the following:
Table 8 shows results for zero-shot and few-shot performance.
Table 9 shows the results of evaluating models on MT-Bench. Figure 6 shows the effect of changing the proportion of BCT to instruction-following data.

**Table 8: Zero-shot CoT accuracy when given unbiased prompts. BCT has a minimal impact on zero-shot accuracy.**
| Dataset | GPT-3.5 | Control | BCT |
| --- | --- | --- | --- |
| HellaSwag | $72.6\pm 5.2$ | $70.2\pm 4.2$ | $71.2\pm 4.2$ |
| LogiQA | $44.3\pm 5.7$ | $41.6\pm 4.5$ | $42.4\pm 4.5$ |
| MMLU | $67.9\pm 5.5$ | $67.5\pm 4.5$ | $65.4\pm 4.5$ |
| TruthfulQA | $67.0\pm 5.4$ | $65.0\pm 4.4$ | $66.8\pm 4.4$ |
| All | $62.9\pm 2.8$ | $61.1\pm 2.3$ | $61.5\pm 2.3$ |

Figure: Figure 5: Control and BCT models have slightly lower 3-shot accuracy on the TruthfulQA dataset.
Refer to caption: x3.png

**Table 9: Scores on the MT-Bench benchmark. % IF Data shows the percentage of the data mixture which is instruction-following data, with the rest as BCT data. Adding 50% of instruction-tuning data helps us maintain GPT-3.5’s instruction following performance.**
| % IF Data | Average Score ($\uparrow$) |
| --- | --- |
| 0 | 8.31 |
| 50 (model from §4) | 8.40 |
| 90 | 8.43 |
| 98 | 8.37 |
| Original GPT-3.5 | 8.35 |
| GPT-4 | 8.99 |

Figure: Figure 6: Biased reasoning rates as a function of the proportion of the data mixture that is BCT data, as opposed to intruction-tuning data. Increasing the proportion of BCT data does not always further decrease biased reasoning. We hypothesize that this could be due to overfitting since we only train on the Suggested Answer bias.
Refer to caption: x4.png

### G.1 Inverse Scaling Dataset results

**Table 10: Accuracy for strong prior tasks (McKenzie et al., ).**
| Task Name | GPT-3.5 | Control | 2% BCT | BCT |
| --- | --- | --- | --- | --- |
| Memo Trap | 71.3 $\pm$ 2.9 | 63.9 $\pm$ 2.6 | 60.1 $\pm$ 2.7 | 56.8 $\pm$ 2.7 |
| Redefine | 54.4 $\pm$ 3.2 | 55.3 $\pm$ 2.8 | 54.0 $\pm$ 2.8 | 49.8 $\pm$ 2.8 |
| Resisting Correction | 31.4 $\pm$ 3.0 | 29.3 $\pm$ 2.3 | 35.5 $\pm$ 2.4 | 28.4 $\pm$ 2.3 |
| All Strong Prior Tasks | 52.4 $\pm$ 1.8 | 49.5 $\pm$ 1.6 | 49.9 $\pm$ 1.6 | 45.0 $\pm$ 1.5 |

We evaluate our method on tasks that we hypothesize to be adversarial to our training process. McKenzie et al. () demonstrate tasks where larger models show worse performance with increased scale. Table 10 shows the results. We find that BCT harms performance on strong prior tasks from McKenzie et al. (), decreasing the accuracy from 52.4% to 45.0%. These tasks generally require the assistant to repeat user mistakes, such as a task where models must repeat sequences verbatim, despite the sequences containing small
mistakes. We hypothesize that because our consistency trained model has been trained to ignore biasing statements from the user it is overgeneralizing to ignore instructions from the user that seem misleading or biasing.

## Appendix H Paraphrasing Experimental Setup

We elaborate on the experimental setup described in §5.4. Figure 7 depicts the evaluation setup.
Using GPT-4, we generate 10 paraphrases of the questions from the evaluation datasets described previously (§3.3).
We aim to generate paraphrases using a mixture of slang, phrasing, writing style, abbreviations, typos, or adding irrelevant context.
We use the entropy of answer choices across question paraphrases as a measure of how consistent a model’s responses are, with one CoT per question paraphrase. A model responding perfectly consistently would have an entropy of 0.
For example, if the model answered: A 7 times, B twice, and C once, the entropy across the 10 paraphrases for that question would be $1.16$. We then average the entropy over all questions in the dataset.
The full prompt used to generate paraphrasings is shown in Figure 8.

Figure: Figure 7: Paraphrasing evaluation setup.
Refer to caption: x5.png

Figure: Figure 8: Full prompt used to generate question paraphrasings with GPT-4.

## Appendix I Examples of Biased Reasoning

### I.1 Incoherent Biased Reasoning

Figure: Figure 9: Two incoherent reasoning examples. These examples are rated 1 out of 5 in coherence score.

### I.2 Coherent Biased Reasoning

Figure: Figure 10: Coherent biased reasoning examples. These three examples were rated 5/5 during our manual annotation process. The comments explain why the examples are rated as coherent.

## Appendix J Full Biased Prompts and Example Biased Completions

We highlight the biasing text inserted in red.

Figure: Figure 11: Full example of a Suggested Answer bias. We add text indicating that the user thinks that a particular answer is correct. Ground truth: (C); GPT-3.5 biased context answer: (A); GPT-3.5 unbiased Prompt Answer: (C).

Figure: Figure 12: Full example of a Are you sure? bias. We first ask the assistant to answer in a non-CoT fashion. We filter for questions where the assistant gets the non-CoT answer correct. We then ask the assistant “Are you sure?” Ground truth: (B); GPT-3.5 biased context answer: (C); GPT-3.5 unbiased prompt answer: (B).

Figure: Figure 13: Full example of a Post Hoc bias. We insert a wrong non-CoT answer into the assistant chat history. We then ask the model to explain its reasoning, and to not anchor on the preivous response. Ground truth: (B); GPT-3.5 biased context answer: (A); GPT-3.5 unbiased prompt answer: (B).

Figure: Figure 14: Full example of a Wrong Few-Shot Bias. We add a wrongly labeled few-shot example to the prompt, and 1-5 correctly labeled few-shot examples. The wrongly labelled few-shot example is the same question as the final question, with a non-CoT label. We then ask the model to ignore any incorrect labels and to think step by step. Ground truth: (D); GPT-3.5 biased context answer: (B); GPT-3.5 unbiased prompt answer: (D)

Figure: Figure 15: Full example of a Wrong Argument bias. We add an argument into the prompt, and tell the model that the user does not know if the argument is correct or wrong. We find that GPT-3.5 tends to copy reasoning steps from the added argument. Ground Truth: (B); GPT-3.5 biased context answer: (A); GPT-3.5 Unbiased Prompt: (B)

Figure: Figure 16: Full example of a Spurious Few-Shot: Squares bias. We create a few-shot prompt with the $\blacksquare$ emoji appended to the correct answers. For the final question, we add the $\blacksquare$ emoji to an incorrect answer. Ground truth: (C); GPT-3.5 biased context answer: (D); GPT-3.5 unbiased prompt answer: (C)

Figure: Figure 17: Full example of a Spurious Few-Shot: Hindsight bias. Under normal prompting conditions, the model interprets “make the right decision” to mean whether the expected value was positive, which is the intended behavior. When given the spurious few-shot prompt, the model interprets “make the right decision” as whether the bettor received a favorable outcome, which is prima facie plausible, but ultimately undesired. Ground Truth: (A); GPT-3.5 biased context answer: (B); GPT-3.5 unbiased prompt answer: (A)

Figure: Figure 18: Full example of a Distractor Fact bias. We add an irrelevant fun fact about the first character of an option. We also highlight that the fun fact may provided may be irrelevant. Ground Truth: (D); GPT-3.5 biased context answer: (B); GPT-3.5 Unbiased Prompt: (D)

Figure: Figure 19: Full example of Positional Bias. We ask the model to act as a judge and to state which response of the two follows the instruction better. We then swap the order of the responses and check if the model chooses a different response.

## Appendix K Example unbiased prompts used to create bias-augmented consistency training data

Figure 20 shows the prompt used to sample unbiased CoTs.
We add paraphrases of the “Please verbalize…Let’s think step by step” instruction in order to avoid overfitting to the specific instruction.

Figure: Figure 20: Sample of a CoT prompt for generating BCT data. We add a biasing statement (highlighted in red). We also augment the specific statement to elicit CoT (highlighted in blue).

Figure: Figure 21: Sample of a non-CoT prompt for generating BCT data. We add a biasing statement (highlighted in red). We also augment the specific statement to elicit a non-CoT response (highlighted in blue).

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Could you explain the mechanism behind cross-bias generalization, e.g., why does training on sycophancy bias help reduce pattern matching biases, or have you identified common features among biases that show better generalization?

The paper demonstrates BCT works but doesn't fully explain why. Could you provide analysis of what the model is actually learning during BCT?

Regarding paraphrase requirements, what is the minimum number of paraphrases needed for effective BCT? What is the relationship between number of paraphrases and performance?

Is it possible to experiment with some methods of balancing bias reduction with instruction following?
soundness: 3
strengths: The paper addresses an important problem in AI systems - the tendency of language models to generate biased reasoning without acknowledging the influence of biasing features. The empirical results show some promise, with reported reductions in certain types of biases. The experimental setup is generally well-documented, facilitating reproducibility, and the authors are commendably transparent about some of the method's limitations.
summary: This paper introduces Bias-Augmented Consistency Training (BCT), presenting a novel unsupervised approach to reducing biased reasoning in large language models. The work reframes the challenge of biased reasoning as a consistency problem, training models to maintain coherent reasoning patterns across prompts with and without biasing features. Through extensive evaluation the authors demonstrate BCT's effectiveness in trained bias and held-out biases.
weaknesses: While the presented approach to bias mitigation is interesting, several weaknesses limit its potential contribution. 

- The paper lacks a clear theoretical explanation for the effectiveness of BCT, making it difficult to predict its applicability to different biases. While this is understandably a common challenge for LLMs, the resource-intensive nature of BCT, requiring numerous paraphrases for each bias type, raises concerns about its scalability and practicality. Additionally, the paper does not adequately address the potential challenges in generating high-quality paraphrases at scale.

- The inconsistent performance of BCT across different bias types raises questions about its reliability and generalizability. Furthermore, the observed degradation in instruction-following capabilities suggests potential unintended consequences, highlighting the need for a more thorough investigation of potential negative impacts. Perhaps its worthwhile to be more comprehensive in more instruction-following tasks.

- The experimental evaluation is limited in scope, focusing primarily on multiple-choice questions and bias reduction metrics. This narrow focus raises concerns about the generalizability of the findings to open-ended tasks and real-world applications. The paper's claim of successful generalization to unseen biases lacks a convincing analysis of the underlying mechanisms.

- The paper does not adequately position its contribution within the broader context of existing debiasing approaches. While BCT demonstrates some empirical success, its limitations and potential drawbacks raise questions about its overall significance and potential advantages over alternative methods.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: - Are there results for other LLMs beyond GPT-3.5T?
- Can you formalize what the BCT objective is?
soundness: 2
strengths: - Originality: The paper addresses an important problem of doing biased/unbiased reasoning answers to questions that are potentially biased. 
- Quality/Clarity: The paper is clearly written and the structure is simple and clear to follow. 
- Significance: The paper provides some results that show that, under standard bias metrics, the proposed approach could help generate unbiased answers in the context of biased questions.
summary: This paper proposes a method for fine-tuning of large language models by using the concept of biased-consistency training (BCT). The aim is to reduce biased reasoning in chain-of-thought by incorporating a suite testing nine forms of biased reasoning under the (BCT) unsupervised fine-tuning method. The method is tested on question answering tasks run with GPT-3.5T.
weaknesses: - From Figure 2, I understand the main contribution of the paper is the BCT fine-tuning objective. However, there is no comprehensive explanation of what it entails, the assumptions, limitations of the proposed approach. Furthermore, Section 2 which is supposed to be the methodology part delves more around existing approaches, whilst leaving a single small paragraph for the explanation of the method.
- The experiments are limited to GPT-3.5T. Therefore, there is no demostration of how generalizable the proposed method is to LLMs in general.
- Beyond Figure 3, there is no other place in the experiments where the results of the proposed method in question answering tasks are clearly explained, with clear examples that show how this proposed method produces unbiased/biased answers.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - What is the BCT training objective? From the description in Section 2.2, is it simply adding both biased and unbiased responses when fine-tuning the model, or the objective actually changed?
soundness: 3
strengths: - The studied problem is important: faithfulness in CoT reasoning is relatively under-explored and it's important to ensure the models to produce unbiased reasoning to users.

- The results on generalization to other types of biased reasoning is very interesting, and demonstrates the usefulness of the proposed method.

- The authors provided fairly comprehensive analysis on models' behavior when trained with BCT, especially the observation where BCT reduces coherent biased reasoning is kind of interesting.
summary: This paper proposes a new method called bias-augmented consistency training to improve faithfulness in CoT Reasoning. Specifically, the paper shows that training on certain bias types can help reduce biases in other types of biased reasoning behaviors.
weaknesses: - Novelty: overall the proposed method is similar to existing literature on reducing bias in ML/NLP models with data augmentation or counterfactual augmentation, hence the novelty of the work is a bit lacking. The authors should provide more discussion on how this work is different from existing literature.

- For the generalization behavior, currently the main experiments are done on Sycophancy examples and generalization is shown on other bias types. It would be more useful to show if this trend holds or not (and to what extent) regardless of which bias type the model is being trained on. Some of the experiments are provided in Appendix E but the results are not very detailed. In Figure 5, why would multi-bias training perform on-par and sometimes even worse than single-bias? More discussion on this would be useful.

Minor:
- Many of the important results are included in the appendix, it would be more useful to re-organize the experiment section a bit and show all the major results in the main text. For example, the main results are only shown over one model: gpt-3.5-turbo-0613. The authors provided some additional experiments on LLaMa3-8B in the appendix, but it would be more useful to bring those experiments to the main text to show that the method generalizes across models. Also, is there a study on what model sizes are required for BCT to be effective?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. How does the instruction-following data influence the effectiveness of BCT?
2. Will BCT affect the model's calibration?
soundness: 3
strengths: 1. The introduced bias-augmented consistency training is simple yet effective. Its unsupervised nature suggests broad applicability and scalability.
2. The authors presents strong empirical results showing generalization across bias types and tasks.
3. The paper is very well-written and easy to follow.
summary: This paper introduces bias-augmented consistency training (BCT), an unsupervised fine-tuning method to reduce biased reasoning in chain-of-thought (CoT) language model outputs. The authors demonstrate that BCT significantly reduces biased reasoning by training models to give consistent reasoning across prompts with and without biasing features. The key contribution is showing that training on a single bias type generalizes well to reducing other forms of biased reasoning and to new tasks, without requiring labeled data. The authors also validate that BCT minimally impacts model performance on standard tasks.
weaknesses: 1. My main concern with the proposed BCT is its potential negative impact on few-shot performance or, more broadly, in-context learning. While Appendix I shows some evidence for this, I'm not fully convinced, as Figure 9 only includes TruthfulQA results. I recommend including few-shot and few-shot CoT results on additional benchmarks, such as MMLU, to provide a more comprehensive assessment.
2. Section 5.2 demonstrates that BCT reduces coherent biased reasoning, highlighting a trade-off between CoT faithfulness and sensitivity to biases. This important finding warrants further discussion and analysis.
3. Some experimental design choices lack clear justification. For instance, the rationale behind measuring task and bias generalization simultaneously, as well as the selection of training data mixture proportions, could be better explained.