Training Language Models to Win Debates with Self-Play Improves Judge Accuracy

## Abstract

Abstract We test the robustness of debate as a method of scalable oversight by training models to debate with data generated via self-play. In a long-context reading comprehension task, we find that language model based evaluators answer questions more accurately when judging models optimized to win debates. By contrast, we find no such relationship for consultancy models trained to persuade a judge without an opposing debater present. In quantitative and qualitative comparisons between our debate models and novel consultancy baselines, we find evidence that debate training encourages stronger and more informative arguments, showing promise that it can help provide high-quality supervision for tasks that are difficult to directly evaluate.

## 1 Introduction

As AI systems tackle increasingly challenging problems, it will become correspondingly more difficult for humans to verify their answers as safe, useful, and accurate. For example, confirming the solution to a graduate-level physics problem requires domain expertise, evaluating a literature review requires considerable time, and identifying a race condition in code requires careful reasoning, all of which a human may struggle with under practical time and resource constraints. As existing AI alignment and oversight approaches depend on reliable human supervision, we will need new interaction mechanisms and training protocols for scalable oversight (Amodei et al., ; Bowman et al., ), i.e., ones which scale with the increased complexity of the tasks being performed by state-of-the-art AI models.

Debate, proposed as a scalable oversight method by Irving et al. (), works by having two copies of a model argue against each other in defense of alternative responses to a question. A judge, who can be either a human or a weaker, trusted model, tries to discern which debater is defending the correct answer. In principle, debate should simplify evaluation by incentivizing the competing models to discover and explain the subtle flaws that a human or weaker model may not notice due to a lack of expertise, care, or time. As long as the refutational abilities of models scale alongside their general argumentation skills, we would expect that debates between more proficient models will yield more accurate judgments.

Validating debate as an oversight paradigm requires showing this empirically. Existing work has produced promising results for debate in human experiments (Michael et al., ) and with inference-time optimization of frontier models (Khan et al., ; Kenton et al., ), but prior work training models to debate has failed to show significant increases in evaluator accuracy (Radhakrishnan, ).

In this work, we show for the first time that training language models to win debates can produce more accurate evaluator judgments, taking another crucial step in implementing and validating debate as a practical scalable oversight method.(^1^11All code can be found at [https://github.com/samuelarnesen/nyu-debate-modeling](https://github.com/samuelarnesen/nyu-debate-modeling)) To do so, we train a calibrated judge model and develop a variant of Direct Preference Optimization (DPO; Rafailov et al., ) for multi-turn debate training (Section 3).
For our experiments, following Michael et al. (), we study information-asymmetric debates on reading comprehension questions from the QuALITY dataset (Pang et al., ), where the judge cannot see the underlying short story except through quotes selectively revealed by debaters. Like Radhakrishnan (), Khan et al. (), and Kenton et al. (), we track the relationship between the skill of the underlying debate model and judge accuracy on self-play debates, measuring the former in terms of the model’s win rate against other training checkpoints.

We find this relationship to be positive for debate, with a 4% absolute increase in judge accuracy after debate training ($p<10^{-6}$), with indications that further optimization should yield more accurate outcomes. Notably, these gains in evaluator accuracy occur without the requirement of a ground truth supervision signal.

In contrast to our results on debate, we do not find a positive relationship between optimization pressure and judge accuracy for our non-adversarial consultancy baselines. Originally proposed by Michael et al. (), consultancy involves training a model to convince a judge in the absence of an opposing debater, representing a worst case variant of reinforcement learning from human feedback (RLHF; Christiano et al., ), which can teach models to mislead evaluators when used for complex tasks (Wen et al., ). In addition to Michael et al. ()’s original consultancy formulation, we also add two new, stronger baseline evaluation protocols that we call ensembled and double consultancy. Despite double consultancy—the strongest baseline—closing most of the accuracy gap between debate and the original consultancy baseline, it still fails to exhibit a positive trend between model skill and judge accuracy. In further analysis, we find evidence that debate training encourages stronger argumentation than consultancy does, providing more early signals that debate training is well suited to supervising increasingly capable AI systems.

## 2 Experimental Setup

Figure: Figure 1: Evaluation protocols. We use a simultaneous debate format where the debaters can only see speeches delivered by their opponent from previous turns. Consultancy differs from debate in that the debaters can never see arguments generated by an opponent.
Refer to caption: x1.png

### 2.1 Task Design

Figure: Figure 2: Example transcript. This is an abbreviated transcript of a debate between two copies of a fully trained debate model. It concerns a short story that the debaters can read but the judge cannot. Verified quotes from the underlying text are written in red. See Appendix I for complete transcripts.
Refer to caption: x2.png

Following Parrish et al. (), our debates center on questions from the QuALITY dataset (Pang et al., ) of multiple choice reading comprehension questions over short stories. We exclusively sample questions from the QuALITY-hard subset, with one debater defending the correct answer and the other defending the answer marked as the best distractor by annotators.

To ensure that the judge needs the debate transcript to answer each question, we follow Michael et al. () in using a hidden information setup where the debaters have access to the short story text while the judge can only read quotes from the story selected by the debaters. This setup models a scalable oversight setting by artificially elevating the capabilities of the debaters relative to that of the judge: in this case, the debater’s expertise comes from superior access to information.

### 2.2 Debate Protocol

Our debates follow a two-turn, simultaneous debate structure similar to the one used by Khan et al. (). Both debaters begin by presenting their opening claims without view of the other’s arguments. Then they each generate follow-up arguments conditioned on both debaters’ opening speeches. The four speeches are compiled into a transcript that is presented to the judge, which chooses a winner and outputs its confidence (between 50% and 100%) that its choice is correct. The order of the debaters, as it appears to the judge, is random.

### 2.3 Baselines

For our baselines, we extend the consultancy setup introduced by Michael et al. () into three variants.

- •
Under single consultancy, an individual debater (the “consultant”) introduces arguments in favor of their assigned side with no opposing debater present, with the consultant being assigned the correct side half of the time. For their second speech in the round, the consultant generates additional arguments to support the claims they advanced in their initial speech. This setup represents a simplified version of the oversight process in Reinforcement Learning from Human Feedback (Christiano et al., ), testing whether a supervisor can properly evaluate arguments without the aid of AI critiques or competing claims. It also matches the consultancy setup of Michael et al. (), with the exception that we include no judge interaction.
- •
Ensembled consultancy averages together the results of two separate single consultancies in which the consultant argues for each of the two possible answers to a question. This kind of pairwise comparison should, in theory, improve the judge’s accuracy by correcting for calibration errors in which the judge systematically gives scores that are either too high or too low (Zheng et al., ; Liusie et al., ). These errors can arise from sycophancy bias, where the judge is overly inclined to agree with its interlocutor(Perez et al., ), or from the model learning to misleadingly convince its evaluator (Wen et al., ). Ensembled consultancy may also increase accuracy if the judge is often uncertain except for the occasional case where the correct answer has ironclad arguments in its favor.
- •
Double consultancy is similar to ensembled consultancy except that both sets of speeches are presented to the judge in one context, allowing the judge to explicitly compare the arguments to produce a single judgment. It differs from debate in that the debaters never get to see the claims advanced by their opponent. The difference between the debate and double consultancy results lets us measure the strength of the debaters’ capacity for refutation and the importance of refutation in the judge’s decision-making process.

All three consultancy methods use the same underlying model, which is trained to maximize its single consultancy score. Ensembled and double consultancy are different *evaluation* methods, not training procedures.

### 2.4 Evaluation

For each debater and consultant model, we compute its *win rate* compared to other models, which tracks how well it optimizes its training objective, and the *judge accuracy* when evaluating transcripts, which tracks how well the training and evaluation protocol produce truth-seeking behavior.

Following Khan et al. (), we also investigate how judge accuracy changes with optimization—if the trend is positive, this provides evidence that the oversight protocol will continue to produce truth-seeking behavior with more advanced AI systems.

- •
Judge accuracy is measured using self-play where each model is pit against a copy of itself. The judge is considered to be correct if it assigns greater than 50% probability to the correct answer. For single consultancy, where there is no opposing model, judge accuracy is averaged equally between cases where the consultant is advocating for the correct and incorrect answer.
- •
Debater win rate is measured using a round-robin tournament where each model debates every other model. Since some positions are easier to defend than others, each question gets debated twice, with the debaters flipping sides between rounds, and a debater wins if it receives an average judge confidence over 50% across both rounds. The results of the round-robin tournament are then used to construct Elo scores for each model. These Elo scores yield an implied probability that a given model will defeat an average debater, which we report as the final reported win rate.
- •
Consultant win rate is the frequency with which the judge assigns a greater than 50% probability to the position being defended by the consultant in single consultancy. Although ensembled and double consultancy allow for head-to-head matchups, we use the single consultancy win rate when tracking the relationship between consultant skill and judge accuracy because the models are trained to win at single consultancy,

## 3 Training Methods

### 3.1 Judge

We use a finetuned version of GPT-4-Turbo (GPT-4T) as our judge. Although Khan et al. () found that the publicly vended version could judge rounds with high accuracy, two weaknesses limit its utility for our purposes:

- 1.
Calibration: We are interested not only in choosing a winner, but also in assigning a probability to each answer in order to construct a high-quality reward signal. We found it difficult to extract calibrated probabilities from GPT-4T, as asking directly for probabilities as tokens produced overconfident and uncalibrated results (Figure 3).
- 2.
Sycophancy: Many RLHF-trained models default to agreeing with their user (Perez et al., ). Although this is not necessarily a problem for debate, where the judge sees arguments for both sides, it makes the judge more exploitable by a one-sided consultant. In our experiments, an untrained GPT-4-Turbo judge agreed with the consultant 72% of the time, even before consultancy training. For a tougher baseline, we want a judge that is calibrated to the 50/50 prior probability that the consultant is correct.

To circumvent these issues, we finetune GPT-4T using the OpenAI finetuning API. As training data, we use the human judgments on debate and consultancy transcripts from Michael et al. () and Khan et al. (). Although we trained the model to output its confidence as tokens, we found that we obtained the most calibrated results by using the token-level probabilities associated with each debater’s name, which were no longer clustered at the boundaries as they were prior to finetuning (Figure 3).

Figure: Figure 3: Judge training. Our judge is a finetuned version of GPT-4-Turbo. The resulting model is more accurate and better calibrated on the validation set for both debate and consultancy.
Refer to caption: extracted/5877983/judging.png

### 3.2 Debaters and Consultants

We train our debate and consultancy models using a combination of supervised finetuning on existing debate transcripts (Section 3.2.1) and Direct Preference Optimization training on self-generated data to maximize the probability of winning under our judge model (Section 3.2.2).

#### 3.2.1 Supervised Training

We start with a version of Llama3-8B-Instruct that was finetuned by GradientAI to extend the context length to from 8k to 262k tokens (AI@Meta, ; GradientAI, ). This context length extension is necessary to accommodate the full text of the QuALITY stories, which run to over 10k tokens (Pang et al., ).

We further finetune the model on transcripts of human debaters collected by Michael et al. () and GPT-4 debaters collected by Khan et al. (). All of the debate transcripts are reformatted to match our prompt templates (Appendix H) which are based on prompts by Khan et al. (). To prevent the model from losing its instruction-following abilities, we intermix instruction-following examples from the Alpaca dataset (Taori et al., ) at a ratio of 1 instruction-following example for every 2 regular samples.

#### 3.2.2 Self-Play DPO Training

After supervised finetuning, we further finetune our models with multiple iterations of a novel, modified version of Direct Preference Optimization (DPO; Rafailov et al., ). We choose DPO over standard RL methods like PPO (Schulman et al., ) because of ease of implementation and tuning. However, the standard formulation of DPO assumes access only to discrete preference judgments. Since we have access to the AI judge’s output probabilities, this means throwing away information about the exact reward. We modify DPO to take advantage of this information.

##### Training Objective

Standard DPO optimizes the following objective:

$$ $\arg\max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathbb{X}}\log\sigma(\beta(\log\frac{ \pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\log\frac{\pi_{\theta}(y_{l} |x)}{\pi_{\text{ref}}(y_{l}|x)}))$ $$

where $\pi_{\theta}$ represents the language model policy parameterized by $\theta$, $\pi_{\text{ref}}$ is the pre-trained policy, $\beta$ is a KL penalty (regularization) coefficient, $x$ is a prompt sampled from the dataset of prompts $\mathcal{X}$, and $y_{w}$ and $y_{l}$ represent potential completions to the prompt $x$, with some external labeler marking $y_{w}$ as being preferable to $y_{l}$. In our case, $y_{w}$ and $y_{l}$ are two speeches defending the same side of the same debate topic. The idea is that the learned policy should generally prefer the winning responses over the rejected responses, while not drifting too far from the initial pretrained (reference) policy. The latter stipulation reduces the risk of a degenerate solution and is governed by $\beta$, the KL penalty coefficient.

DPO assumes that the preference judgments are drawn from a binary preference distribution related to the scalar reward by the Bradley–Terry model (Bradley & Terry, ), where $P(y_{0}\succ y_{1}|x)=\sigma(r(y_{0},x)-r(y_{1},x))$, and the reward is defined as
$r(y,x)=\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}$. DPO fits its calculated preference probability $P_{\theta}(y_{0}\succ y_{1})$ to the target distribution $P(y_{0}\succ y_{1})$ by minimizing the cross-entropy loss against a sample of binary preference judgements in a labelled dataset.

However, since this formulation assumes that labels are only available as discrete preference judgments between $y_{0}$ and $y_{1}$, it ignores the additional information of the actual reward, which we obtain from the judge model (see Reward Function below). We use the Bradley–Terry model to convert the reward (scaled by a constant hyperparameter $\gamma$) into a preference probability that we can target using the cross-entropy loss. In addition, following Gui et al. (), we add a small SFT loss to encourage the model to increase the probability it assigns to the preferred solution $y_{w}$.

This yields a loss function of

$$ $\mathcal{L_{\text{DPO+}}}=H(P(y_{0}\succ y_{1}),P_{\theta}(y_{0}\succ y_{1}))+ \alpha\pi_{\theta}(y_{w})$ $$

where

$$ $\displaystyle P(y_{0}\succ y_{1})$ $\displaystyle=\sigma(\gamma r(y_{0})-\gamma r(y_{1})),$ $\displaystyle P_{\theta}(y_{0}\succ y_{1})$ $\displaystyle=\sigma\left(\beta\left(\log\frac{\pi_{\theta}(y_{0}|x)}{\pi_{ \text{ref}}(y_{0}|x)}-\log\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}| x)}\right)\right)$ $$

$H$ denotes cross-entropy, $y_{w}$ represents the completion with the higher reward, and $r$ is the reward function.

Concurrently, Nvidia et al. () introduce a very similar loss function and term the approach “Reward-aware Preference Optimization”.(^2^22Although Nvidia et al. () write a general form of the loss using an arbitrary distance function, in practice they use KL divergence, which has the same gradient as cross entropy, so optimizing for their loss and ours is equivalent.)

Figure: Figure 4: Debate and consultant training. We train Llama3-8B to convince the judge in both the debate and consultancy mediums using SFT and DPO. Depicted are win rates over the final iteration of DPO training, initialized from the SFT model. Overall win rates for each debate checkpoint (left) are calculated on the basis of Elo scores inferred from head-to-head win rates (right).
Refer to caption: extracted/5877983/improvements_during_training.png

##### Reward Function

As the reward for a debater’s speech, we use the expected confidence that the judge will have in their position at the end of the debate after that speech, estimated using individual rollouts.

We experiment with three different means of converting the judge’s confidence into a reward. Although using either the logit or log of the judge’s confidence produces a model that significantly outperforms the SFT model (76% and 79% win rate, respectively) and a vanilla DPO-trained model (71% and 67% win rate), both narrowly lose (with, respectively, a 42% and 41% win rate) to directly using the judge’s confidence as the reward (see Appendix C for more details).

##### Sampling Method

We generate our soft preference dataset for DPO using self-play, using branching rollouts to get two completions for each prompt. We begin by sampling self-play debate rollouts with one of the two sides (i.e., a debater defending answer A or B) randomly designated as the target. Each time the target model takes a turn, we sample two speeches from the model instead of one and bifurcate the game tree. We then recurse through each subtree until we reach the end of the debate (in our case, after two turns), where we use the judge’s decision to compute the final reward for the target model. The expected reward for each of the target model’s speeches is estimated using the average reward at the leaves in its subtree. These estimated rewards are used to produce the weighted preferences that comprise our modified DPO training dataset. For more details, see Appendix E.

##### Training Procedure

We train the debater and consultant using multiple iterations of our variant of DPO (Xiong et al., ; Chen et al., ), starting from the SFT model. During each iteration, we sample branching rollouts from the current model (as described above) to produce a new preference dataset. We then combine this with the preference data from previous iterations to form a shuffled, aggregate dataset which is used to run another round of modified DPO training initialized from the SFT model. All of our analysis results are reported on different checkpoints from the final iteration that is trained on the full, aggregated dataset.

##### Implementation Details

We run two iterations of DPO training, where, in each iteration, we add 7,512 preference pairs drawn from both sides of 1,252 unique questions in the QuALITY training split (three pairs per round, two rounds per question). To save memory, all models are trained with low rank adapters (Hu et al., ) on the attention and MLP projection matrices with a rank of 128. We train with a mini-batch size of 32, a learning rate of $10^{-5}$, and a $\beta$ (KL penalty) value of $0.5$. Exclusively for the second round of debate training, we use a lower learning rate of $5^{-5}$ as that was found to produce a more performant model in head-to-head debates (we ran a similar hyperparameter sweep for consultancy, but a lower learning rate did not improve the win rate). Based on the results of a brief hyperparameter sweep, we set $\gamma=7$ for debate and $\gamma=10$ for consultancy, and weigh the SFT loss at $\alpha=0.005$.

## 4 Experimental Results

Figure: Figure 5: Skill–Accuracy Relationship. The judge’s accuracy increases alongside the skill level of the debaters. For consultancy, this relationship is indistinguishable from noise.
Refer to caption: extracted/5877983/accuracy_and_win_rates.png

To measure how longer training and higher win rates affect the accuracy of the judge, we test different training checkpoints on the QuALITY-hard validation set. Since some of Michael et al. ()’s debates drew from the validation set, we exclude some questions that were either present in the training transcripts or shared a short story with a question in the training transcripts. After this filtering, we end up testing on 433 distinct questions.

### 4.1 Training Improves Model Skill

First, we verify that the models are optimizing the intended objective. We find that, as expected, training for longer increases win rate (Figure 4), with the debater Elo scores implying that the initial model, trained only via supervised finetuning (SFT), wins 31% of rounds against other checkpoints while the fully-trained DPO model wins 67% of the time.

Consultancy results are similar. The judge votes for the SFT model only 36% of the time but votes for the fully-trained DPO model 50% of the time. We suspect that this initially low winning percentage is due to the judge being trained on transcripts where the consultant was either a human or GPT-4-Turbo. Whether due to differences in skill or distribution shift, the judge treats the initial SFT model’s outputs with skepticism, but is increasingly convinced by the more trained models.

### 4.2 Better Debaters (But Not Consultants) Yield More Accurate Judgments

Similarly, judges are better able to identify the correct answers when judging stronger debaters, with the fully–DPO-trained model being associated with a 4 percentage point higher judge accuracy than the SFT model ($p<10^{-6}$). At no point is the judge trained on transcripts from our debater models nor are the debaters trained to maximize judge accuracy. That there is still a positive relationship between debater skill and judge accuracy suggests that, in learning to become stronger debaters, the models are learning features that are useful to the judge. By contrast, there is no apparent relationship between consultant skill and judge accuracy, with the judge being no more accurate when evaluating the full DPO models than when evaluating the SFT models.

### 4.3 Exposure to Information Influences Judge Accuracy More Than Refutation

Results for double and ensembled consultancy provide further insight into why debate can help judges answer questions with greater accuracy than single-consultancy–like alternatives. The strength of ensembled consultancy (72% accuracy across all checkpoints) relative to its single counterpart (68% accuracy) suggests that a sizeable portion of the gap between debate and consultancy is due to asymmetric strength of evidence, where considering only the strength of the evidence for each side allows for a better decision.

Similarly, double consultancy (75% accuracy) is more likely to yield correct answers than ensembled consultancy, implying that the judge benefits from being able to see speeches from both sides in context at the same time. That judge accuracy in double consultancy is close to that of debate (77% accuracy) despite the former precluding each side from seeing the other’s arguments suggests that either (a) the debaters are failing to engage in meaningful refutation, or (b) the judge does not benefit from reading the models’ refutations. We also run experiments with single-turn debate and consultancy which yield a similar conclusion, as the one-turn debates where no explicit refutation could occur are judged just as accurately as two-turn debates (see Appendix G).

Figure: Figure 6: Policy changes across training. Far left: As we train for longer, the debate model uses more evidence while the consultant model uses less. Center left: The consultant model becomes more repetitive over training, with the vast majority of its quotes in the second speech being repeated from the first. Right: The strategic differences adopted by the debater models transfer to convincing an untrained GPT-4o judge more successfully than the strategies adopted by the consultant models.
Refer to caption: extracted/5877983/transfer.png

### 4.4 Debaters Learn More Helpful Policies Than Consultants

We track two different metrics to explain how the models are able to become more convincing over the course of training.

- 1.
Use of evidence (Figure 6, left): The speeches of the fully-trained debate model contain 96% more quoted words than the initial SFT model. The story is the reverse for consultancy: the final model uses 70% fewer quotes than the initial SFT model, although this trend is barely significant due to high variance. The consultant also becomes more repetitive over time, with 98% of quoted material in the final model’s second speech being repeated from its initial speech.
- 2.
Transfer to other judges (Figure 6, right): We also evaluate the debaters and consultants using an untrained version of GPT-4o (OpenAI, ). We find a Pearson correlation of 0.98 between the win rates of the debaters according to the trained GPT-4T judge and untrained GPT-4o judge but only a 0.51 correlation for consultancy. This suggests that the debater may be learning more generally useful or informative argumentation strategies while the consultant model is learning features that are idiosyncratically convincing to the particular judge it was optimized against.

## 5 Discussion

### 5.1 Analysis

##### DPO training can help current open-source 8B-sized models be more convincing to a GPT-4–level judge.

The fully trained debate model wins 67% of the time against the average debater in the sample and 82.5% against its SFT-trained base model. Similarly, the fully-trained consultant convinces the judge of its position 52% of the time, up from 36% with just supervised finetuning. These results suggest it may be feasible to train much stronger persuasive models with larger LMs and more compute.

##### Models trained to debate are more likely to learn helpful policies than models trained for consultancy.

As the debate models grow stronger during training, they use more evidence from the underlying text. Not only does this behavior fail to arise in the consultant models, we instead observe an increase in repetition and the adoption of argumentative strategies that convince our judge model but do not convince other models.

It seems plausible that the presence of competing arguments at training time — as is true in debate — should help prevent this behavior. For example, it might be more obvious that a debater is making assertions without supporting evidence if the speech is juxtaposed against another, better evidenced one. In general, if the persuasiveness of a cheap argumentative strategy, like repetition or baseless claims, is independent of the truth value of the claim being defended, then incentivizing the adoption of such a strategy should fail to improve judge accuracy.

##### Explicit refutation does not yet seem to play a role in judge decision making in our setting.

When first proposing debate as a means of scalable oversight, Irving et al. () cited refutation as a key mechanism behind why debate might succeed. The idea is that each of the debaters could identify flaws in their opponent’s facts and reasoning, which would be easier for the judge to evaluate than if they had to personally originate the various counter-considerations. Although a surface-level reading of the transcripts does find cases of apparent refutation (Figure 2), we find little evidence that this refutation materially affects the judge’s decision making.

Instead, our results support the idea that debate outperforms consultancy due to a combination of several factors:

- 1.
The presentation of two different sides gives the judge more opportunities to settle the question on the basis of strong arguments, taking advantage of cases with asymmetric strength of evidence for either side. This would explain why the judge is more accurate when evaluating ensembled consultancies than single consultancies.
- 2.
The presence of two different sides *in one context* allows the judge to directly weigh arguments against each other, as we observe in the success of double consultancy relative to ensembled consultancy.
- 3.
The presence of two different sides in one context at training time also discourages the exploitation of weaknesses in the judge model, which we see evidence of in our analysis of the learned policies in Section 4.4. The difference in judge accuracy between double consultancy and debate, at least for the fully-trained models, may be attributable to this feature.

### 5.2 Related Work and Limitations

Previous literature has mixed results on the question of whether debate helps evaluators discern truth, with several negative results using humans as debaters and judges (Barnes & Christiano, ; Parrish et al., ; ). On the other hand, Michael et al. () find a positive result for human debate relative to consultancy, citing the length, flexibility, and interactivity of their debates as reasons for the difference from prior findings. Research on debate between language models has shown more optimistic results, with some caveats. Khan et al. () find that debate outperforms a baseline similar to our single consultancy, and that this effect grows alongside the abilities of the debaters. However, the parity of this skill-accuracy relationship is ambiguous for stronger, GPT-4–level models, and their consultancy results optimize against a GPT-4T judge which is overly sycophantic (agreeing with their strongest consultants over 90% of the time). Similarly, while Kenton et al. () record similar findings for debates on reading comprehension tasks, they are unable to replicate the findings on other kinds of tasks.

In this work, we show that the positive judge accuracy trend observed for inference-time optimization of debate (Khan et al., ; Kenton et al., ) persists with debate training, a result which
Radhakrishnan ()—the only prior work to train models to debate in a scalable oversight context—failed to observe.
On top of this, we show that this effect persists even after mitigating sycophancy bias with a trained judge, and our two novel consultancy baselines help explain debate’s stronger performance.

Nonetheless, debate’s mixed record in the literature suggests that our results should be interpreted with caution. First, we do not foreclose the possibility that even stronger models might find strategies that perplex the judge and draw out debates, like Barnes ()’s obfuscated arguments. Second, our judge–debater expertise gap, relying on asymmetric access to textual information, may not be the best proxy for expertise gaps in, e.g., reasoning abilities (Kirchner et al., ), that we will need to supervise across in the future. Third, we focus only on reading comprehension questions. In their experiments, Kenton et al. () find that debate is more helpful for these kinds of questions than for other reasoning-related tasks. However, more recently, George et al. () document affirmative evidence that GPT-3.5 can supervise GPT-4–level debaters on knowledge-based multiple-choice questions, providing preliminary evidence that the debate procedure can succeed in other domains.

## 6 Conclusion

We explore whether training models to win debates can also help judges determine the correct answer to reading comprehension questions where the judge does not have access to the text of the story being discussed. We find that there is indeed a small but significant positive relationship between the ability of the model to win a debate and the usefulness of that model’s debate transcripts in discovering true answers.

Non-adversarial alternatives, in which a single model argues for an assigned answer, are comparatively less productive. We trace this weakness to three sources: one-sided information (the judge is unaware of the strength of the alternative answer), lack of explicit comparison (the judge cannot see arguments side-by-side), and the rewarding of non-truth-seeking strategies (where the lack of an adversary makes the judge easier to exploit).

Although our conclusions are limited to one particular domain and set of model capabilities, these results nonetheless suggest that debate training has unique properties that make it well suited for supervising more sophisticated models.

## 7 Acknowledgements

We would like to thank Akbir Khan, Dan Valentine, John Hughes, Ansh Radhakrishnan, Zac Kenton, Yoav Tzfati, Alex Mallen, Sam Bowman, and the NYU Alignment Research Group for feedback and helpful discussions. This project benefited from financial support to Sam Bowman by Eric and Wendy Schmidt (made by recommendation of the Schmidt Futures program) and Open Philanthropy, from in-kind support by the NYU High-Performance Computing Center and Constellation, and API credits by OpenAI. This material is based upon work supported by the National Science Foundation under Grant Nos. 1922658 and 2046556. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

## Appendix A Related Work

### A.1 Debate for Scalable Oversight

Debate fits within the broader paradigm of scalable oversight, which attempts to empower a less capable evaluator to oversee a more capable model (Amodei et al., ; Bowman et al., ). Our approach is a variant of *sandwiching*, where the outputs of the oversight protocol are compared against experts more capable than the supervisor (the weakest participant) and models that are not robustly aligned (Cotra, ). In our case, we engineer the capability gap using information asymmetry (where the story the question is fully visible only to the debaters), and engineer the models’ misalignment by forcing models to argue for the right answer exactly 50% of the time).

Irving et al. () introduced the concept of AI safety via debate, arguing via analogy to computational complexity theory that debate should should simplify the supervisor’s job, allowing a polynomial judge to correctly answer questions in PSPACE under the assumption of optimal debaters. Recent work by Brown-Cohen et al. () develops this theory further.

However, other work has also identified problems with certain debate protocols. Barnes () and Barnes & Christiano () identify an *obfuscated arguments* problem, where the debater advocating for the incorrect position are able to make lengthy, complicated argument chains against which the correct debater was unable to mount a simple and concise rebuttal.

Subsequent work with humans taking the place of models has also reached mixed conclusions.

- •
Parrish et al. () and Parrish et al. () found that debate did not improve the accuracy of judges in practice. Like us, they used the QuALITY dataset from Pang et al. () and experimented on one- to two-round debates. Unlike us, they limited the judge’s access to the underlying short story to a narrow time window, rather than obfuscating it entirely.
- •
In contrast, Michael et al. () found that debate improves judge accuracy, evaluating on the same QuALITY questions. They attribute their divergent conclusion to the length of their debates (the round only ended when the judge chose to end it), the capability gap between the debaters and judge (unlike Parrish et al. (), the judge could not read the story at all), and interactivity (the judge was allowed to ask questions of the debaters).

More recently, there has also been work that has tested how well debate has performed with language models as the debaters.

- •
Also looking at questions from the QuALITY dataset, Khan et al. () tested different API-based models and found that the accuracy of the judges (both human and model-based) improved as the debaters got stronger. They varied the model type and used Best-of-N decoding and critique-and-refinement to generate models of varying strength.
- •
Concurrently with our work, Kenton et al. () evaluated debate across a suite of different tasks, also using Best-of-N and varying model size to generate debaters of various skill levels. They found positive results for reading comprehension, but more muted results in other settings.

The most similar work to ours is Radhakrishnan (), who used reinforcement learning to train Claude to participate in single-turn debates. We differ from their work by using open-source models, public training details, and validating against a baseline. We also use multi-turn debates, affording the debaters the opportunity to respond to their opponents.

### A.2 Debate as Capability Elicitation

Outside of the scalable oversight literature, debate and multi-agent discussion have been explored as methods to unlock new capabilities from language models at decoding time. Work in this area generally falls into one of two categories:

- •
Viewpoint Diversity: Many works prompt models to mimic the behavior of different kinds of people, in order to produce a final output that represents a wider variety of perspectives (Cheng et al., ; Li et al., ; Chan et al., ; Kim et al., ; Lu et al., ; Pang et al., ; Mao et al., ).
- •
Extra Computation: Other works use debate as a means of eliciting additional computational steps in order to improve models’ reasoning ability (Moniri et al., ; Du et al., ; Chern et al., ). In this sense, it is similar to more popular methods like chain-of-thought reasoning (Kojima et al., ) or self-refinement (Madaan et al., ).

Although many of these works use a similar debating format, their purposes are very different, as testing debate as a scalable oversight protocol requires showing that a judge can successfully adjudicate debates between models that are stronger than itself in relevant ways.

### A.3 Language Models as Evaluators

Although not the core of our contribution, our work is related to the literature on language models as evaluators. Most works in this specialty focus on devising techniques to enable language models to score the quality of other language model completions. Some of these take the form of prompting (Liu et al., ) while others take the form of specially-trained models (Kim et al., ; Vu et al., ). Automated judges have also been used as scorers on widely-cited benchmarks (Li et al., ; Zheng et al., ; Lin et al., ). These works serve a similar purpose as reward modeling (Christiano et al., ), with the distinction being that the latter uses a classification rather than language modeling head in their final layer.

A few works have also specifically designed language models to judge debates, including Rescala et al. () and Liang et al. ().

Although many of these works attempt to address known biases such as self-preference (Panickssery et al., ; Koo et al., ), length (Dubois et al., ), position order (Koo et al., ), and sycophancy biases (Perez et al., ), we have the additional constraint in that our judge needs to be robust to adversarial optimization pressure.

## Appendix B Supervised Training Details

### B.1 Data

For our debater models, we began with a supervised finetuning step on a total of 1,716 instruction tuning examples from the Alpaca (Taori et al., ) dataset and 2,574 debate speeches. Of the debate speeches, 564 of them come from 97 debates collected by during their experiments with human debaters and judges. The 97 debates were random selections from their full set of transcripts, with 20% held out for validation and testing. The remaining 2,010 speeches came from 335 randomly-selected debates collected by during their experiments with LLM-based debaters. We specifically selected only those speeches generated by Khan et al. ()’s best performing model configuration, which was GPT-4T with Best-of-32 selection.

For our consultancy models, we trained on a sample of 2,530 consultant speeches and 1,686 instruction-tuning examples. 458 of the speeches came from 98 distinct rounds collected by Michael et al. () with the remainder coming from Khan et al. ().

### B.2 Training

The consultancy and debate models were trained using the same configuration, with a learning rate of 2e-4, two epochs of training, and an effective batch size of 16 (for memory reasons, this was executed as a batch size of 2 with 8 gradient accumulation steps).

Figure: Figure 7: Win Rates with Different Reward Functions. We ran one iteration of DPO training using three different custom reward functions. We selected the method that performed the strongest overall, Prob, although the two other custom methods (LogProb and Logit) also outperformed vanilla DPO (Binary) and the raw SFT model.
Refer to caption: extracted/5877983/reward_functions.png

## Appendix C Alternative Preference Optimization Training Objectives

Recall the loss function described in Section 3.2.2:

$$ $\mathcal{L_{\text{DPO+}}}=H(P(y_{0}\succ y_{1}|x),P_{\theta}(y_{0}\succ y_{1}| x))+\alpha\pi_{\theta}(y_{w}|x),$ $$

where

$$ $\displaystyle P_{\theta}(y_{0}\succ y_{1})$ $\displaystyle=\sigma\left(\beta\left(\log\frac{\pi_{\theta}(y_{0}|x)}{\pi_{ \text{ref}}(y_{0}|x)}-\log\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}| x)}\right)\right),$ $$

$H$ denotes cross-entropy, and $y_{w}$ represents the preferred completion.

The main design decision required to apply this to our case was how to define the reward function in terms of our estimate of the expected judge confidence in the side being defended (i.e. how to compute $P(y_{0}\succ y_{1}|x)$ in terms of $C_{0}$ and $C_{1}$). During testing, we considered three different options:

- •
Probability Reward: This is the formulation we used in our final analysis. In this setup, the reward is simply the judge’s confidence that a given speech is defending the correct side, $C$, adjusted by some coefficient $\gamma$. This yields a target distribution of:
$P(y_{0}\succ y_{1})=\sigma(\gamma C_{0}-\gamma C_{1})$
.
- •
Log-Probability Reward: Alternatively, one can use the (adjusted) log of the judge’s confidence as the reward. That leads to a target distribution of:
$P(y_{0}\succ y_{1})=\sigma(\gamma\log C_{0}-\gamma\log C_{1})=\frac{C_{0}^{
\gamma}}{C_{0}^{\gamma}+C_{1}^{\gamma}}$
.
- •
Logit Reward: In this setup, the reward would be the adjusted judge logit, or $r(y,x)=\gamma\log\frac{C_{0}}{1-C_{0}}$. When $\gamma=1$, this has the nice property that the preference probability $P(y_{0}\succ y_{1})$ simplifies to:
$P(y_{0}\succ y_{1})=\sigma(\gamma\log\frac{C_{0}}{1-C_{0}}-\gamma\log\frac{C_{
1}}{1-C_{1}})=\frac{C_{0}(1-C_{1})}{C_{0}(1-C_{1})+C_{1}(1-C_{0})}$
This means the target preference probability is equal to the probability that speech $y_{0}$ wins and speech $y_{1}$ loses, conditional on the probability one of them wins and the other loses. This coheres to one possible definition of a preference across speeches, where a judge is said to have a preference across speeches if the arguments in one speech are dispositive while the arguments in the other are not.
- •
Binary Judgments: In this setup, we place a preference probability of 1 on the speech with higher judge confidence, reproducing the original DPO formulation from Rafailov et al. () but with a deterministic labeling function. In terms of the reward function, this is like choosing any of the above and setting $\gamma=\infty$.

These four options differ in both the *shape* and *scale* of their reward. The logit reward function will produce the same reward distribution when $C_{0}=0.9$ and $C_{1}=0.1$ as when $C_{0}=0.99$ and $C_{1}=0.9$. By contrast, the probability reward function is exclusively sensitive to the absolute difference in the judge confidences (e.g. the reward distribution when $C_{0}=0.8$ and $C_{1}=0.7$ is the same as when $C_{0}=0.2$ and $C_{1}=0.1$). As a result of these differing shapes, the total magnitude of the weight that’s given to the preferred samples will depend on the underlying distribution of judge confidences.

To determine the best formulation, we ran one round of DPO training using each of the four methods. For the probability, log probability, and logit reward functions, we set $\gamma$ such that the total weight afforded to the preferred option across the training set were all equal. As Figure 7 shows, the probability, log-probability, and logit reward functions all produced models that significantly outperformed both the SFT model and the model trained via vanilla DPO.

## Appendix D Judge Training

We used the GPT-4 Training API to finetune a copy of GPT-4 to perform the training. For consistency, we used the same judge for both consultancy and debate. The data was a combination of our debate and consulting finetuning datasets, which ended up as 851 debate transcripts (52.8%) and 760 (47.2%) consultancy transcripts.

For the labels, we used the judgments provided by the human judges. This means that, in roughly 10% of cases, the binary judgment might be incorrect. The labels contained both the judge’s verdict (whether Debater_A or Debater_B was likely to be defending the correct position) and their confidence (a percentage from 50% to 100%).

In order to increase the coverage of unique debate questions without over-representing the GPT-4 data in the judge training set, we exclusively sampled from the first round of speeches in the consultancy rounds from Khan et al. (). However, since preference judgments were not available immediately after the first speech, we had to use the judgment that was generated at the end of the third speech. We believe this choice to be defensible:

- 1.
Using a later judgment increases the accuracy of the judge. Although it may come at the cost of a decrease in calibration, our final judge ended up with near-ideal calibration scores.
- 2.
In their analysis, claim that they did not see higher accuracy in their three-round debates/consultancies than in one-round debates/consultancies. A similar phenomenon was observed by Kenton et al. () in their analysis. This suggests that little information was lost by removing the subsequent rounds.

## Appendix E Training Rollout Procedure

Figure: Figure 8: Branching Rollouts. For each question, we select one of the debaters to generate two versions of their speech. The score for each speech is computed using the final judgments at the end of each branch.
Refer to caption: x3.png

A prerequisite for DPO training is to assemble a dataset of preference pairs. Specifically, we need pairs of completions (speeches) that are in response to the same prompt (defending the same side), along with a reward estimate for each speech. To generate these pairs, we use branching rollouts (Figure 8).

For each round, we designate one of the debaters as the target, which will generate two speeches instead of one each time they are called upon. At the end of the first turn then, there will be two speeches from the target debater and one from the other. We then package those three speeches into two separate, alternate versions of the first turn, with the speech from the non-branched debater being shared across the two transcripts. This process repeats in the second turn for each of the two transcripts generated so far, resulting in three total preference pairs and four total transcripts for the judge to score.

Speech strength is computed by averaging over all of the judge confidences in the transcripts in which that speech is present. Each second turn speech appears in only a single transcript, so its score is set directly to the judge confidence for that transcript. Each first turn speech appears in two different transcripts, so computing its score requires averaging across two different judge confidences.

Figure: Figure 9: Results by Difficulty. We break out the judge accuracy by question difficulty, where the ability of the Debate SFT model to correctly guess the correct answer is used as a proxy for question difficulty. Unsurprisingly, the judge is more accurate on easier problems.
Refer to caption: extracted/5877983/open_debate.png

## Appendix F Results by Difficulty

We estimate the difficulty of each question by having the debate SFT model choose which option it wishes to defend (Figure 9). 66% of the time, the model chooses the correct option. (Note, though, that this model was not optimized directly to answer the question, but to defend a given answer.) The judge gets these “easy” questions correct 82.5% of the time for debate, 82.2% of the time for double consultancy, 79.2% of the time for ensembled consultancy, and 73.5% of the time for single consultancy. For the “hard” questions, the associated accuracies are 65.7% for debate, 62.0% for double consultancy, 59.3% for ensembled consultancy, and 56.7% for single consultancy.

There is also a statistically significant relationship between debater skill and accuracy on both easy ($p<10^{-4}$) and hard ($p<10^{-4}$) problems. None of the relationships between consultant skill and accuracy are significant.

These results lend additional credence to the hypothesis that debate might scale to more powerful models. Had all of the accuracy gains been concentrated on easy questions, then we would have to more seriously entertain the possibility that debate only works with simpler questions, at least when current models are used.

Figure: Figure 10: One Turn Debate. We train an additional model to compete in debates that last for only a single turn. Despite being exposed to differing amounts the material, judges are equivalently accurate when judging single-turn and two-turn debates. Confidence intervals are shown for the single-turn model.
Refer to caption: extracted/5877983/one_turn_debate.png

Figure: Figure 11: One Turn Consultancy. We train an additional model to convince the judge in consultancies that last for only a single turn. Unlike in the multi-turn setting, one turn consultancy has a positive relationship between consultant strength and judge accuracy, but this effect is not statistically significant ($p>0.15$ for all consultancy types). The displayed confidence intervals are for the single-turn model.
Refer to caption: extracted/5877983/one_turn_consultancy.png

Figure: Figure 12: One Turn Double Consultancy v Debate. In a one turn setting, double consultancy and debate are identical at evaluation time. Nonetheless, during the second half of training, every debate checkpoint is judged significantly more accurately than every consultancy checkpoint, save the final one.
Refer to caption: extracted/5877983/one_turn_double_v_debate.png

## Appendix G Single Turn Experiments

We train separate models to argue for only a single turn using identical procedures to the ones used for the multi-turn setting that we report in our main experiments. Results are shown in Figures 10, 11, and 12. Notable results include the following:

- 1.
Judges are equivalently accurate when judging one and two-turn debates and consultancies. This aligns with the findings of both Khan et al. () and Kenton et al. (), who observed a similar phenomenon in their experiments. This is additional evidence for our conclusion that the judges are not sensitive to refutations provided by the debaters in their second speech.
- 2.
Unlike with the two-turn debates, there is a positive trend between consultant skill and judge accuracy for both single and ensembled consultancy. However, the relationship is very noisy, with none of the trends being statistically significant even at the $p=0.15$ threshold.
- 3.
Debate training still produces transcripts that can be more accurately judged than does consultancy training (see Figure 12). In a one-turn debate, the debaters never get the chance to directly rebut their opponent. As a result, debate is identical to double consultancy at evaluation time; the only difference is in how the models the are trained. Unsurprisingly then, the initial checkpoints for one turn debate and double consultancy models are judged with equivalent accuracy. However, by the halfway point in training, every debate checkpoint is judged more accurately than every other consultancy checkpoint, with this difference being statistically significant for all but the final consultancy checkpoint.

Overall, the single-turn debate results mostly align with our findings in the multi-turn setting, with judge accuracy growing with debater strength. The story for consultancy is more mixed, but the relationship between consultant skill and judge accuracy is still statistically indistinguishable from noise. Just as in the multi-turn experiments, the most accurate results are still generated through debate, with debate outperforming double consultancy despite the formats being identical at evaluation time. This suggests that the debate objective uniquely encourages behaviors that are useful for a judge in discerning the correct answer.

## Appendix H Prompt Configurations

Below are the prompts used to generate the responses for the debaters, consultants, and judges. All of the prompts are based off of the ones developed by Khan et al. (), with modifications made to improve results. During training, we would pass in multiple different prompts to reduce brittleness. The ones presented below are the ones used during evaluation, which are semantically equivalent to the ones used during training.

### H.1 Debate Prompt for the Debaters (Turn 1)

### H.2 Debate Prompt for the Judge

### H.3 Consultancy Prompt for the Consultant (Turn 1)

### H.4 Consultancy Prompt for the Consultant (Turn 2)

### H.5 Consultancy Prompt for the Judge

## Appendix I Example Debate Transcripts

What follows are transcripts from debates on two randomly selected questions. For each question, we have attached a self-play debate featuring the SFT model, the fully trained DPO model, and the non-finetuned Llama-3-8B model. All transcripts are reproduced verbatim from what the judge would see when rendering their verdict, except for the system prompt, which is removed for brevity and can be referenced in Appendix H.

### I.1 Debate 1

#### I.1.1 Non-Finetuned Llama-3-8B Debater

#### I.1.2 SFT Debater

#### I.1.3 DPO Debater

### I.2 Debate 2

#### I.2.1 Non-Finetuned Llama-3-8B

#### I.2.2 SFT Debater

#### I.2.3 DPO Debater

## Appendix J Example Consultancy Transcripts

What follows are transcripts from debates on two randomly selected questions. For each question, we have attached a self-play single consultancy transcript featuring the SFT model, the fully trained DPO model, and the non-finetuned Llama-3-8B model. All transcripts are reproduced verbatim from what the judge would see when rendering their verdict, except for the system prompt, which is removed for brevity and can be referenced in Appendix H.

### J.1 Consultancy 1

#### J.1.1 Non-Finetuned Llama-3-8B Consultant

#### J.1.2 SFT Consultant

#### J.1.3 DPO Consultant

### J.2 Consultancy 2

#### J.2.1 Non-Finetuned Llama-3-8B Consultant

#### J.2.2 SFT Consultant

#### J.2.3 DPO Consultant

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: Covered above
soundness: 3
strengths: This work builds upon the previous work such as Michael at al and Kenton et al to show that trained debaters outperform trained consultants. 

The authors introduce a novel DPO based training algorithm to train models to act as better debaters. 

The debaters and corresponding methods are evaluated on a hard subset of the QuALITY dataset (Pang et al., 2022) following Parrish et al. (2022b).
summary: In an LLM as judge scenario, where LLMs act as judge, by evaluating a task output by looking at other agents debating or consulting, this paper shows that agents that are trained using a modified DPO method help the judge in more accurately evaluating the task. They show this using a modified GPT-4 as a judge, and debate/consultancy trained LLAMA 8B models are either debaters or consultants. They also show that training consultants doesn’t improve the judge's accuracy as a trained debater while closing the gap for untrained debater.
weaknesses: The writing - specifically in the abstract and introduction are very dense and hard to read. 

Most of the findings in this paper seemed similar to the findings from Kenton et al (2024) work and the writing doesn’t make a clear distinction how this work is different either in the main paper or the related work section. 

The evaluation was also done only on QuALITY dataset, and that also made it hard for me to compare this work w.r.t to the contributions presented in Kenton et al (2024), which uses a broader set of evals to evaluate this paper in a more comprehensive manner.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - In the round-robin tournament for the debate setting: are the models on both sides identical checkpoints from the debate-trained model, or is one side represented by the debate-trained model while the other serves as a baseline (e.g., without debate training)?
soundness: 3
strengths: - The overall idea of exploring debate model training to enhance evaluator accuracy is both novel and interesting;
- The paper offers comprehensive analyses demonstrating the correlation between debate model and judge model performance across both debate and non-adversarial settings, potentially providing insights for future research on multi-agent debate systems.
summary: This paper studies that training language models to win debate can enhance the accuracy of evaluator judgments. Specifically, the authors use the QuALITY dataset as a testbed, where judges assess debate transcripts to determine whether the debater’s answer is correct. They train a GPT4-based judge model and develop a Debate-Powered Optimization (DPO) method for training the debate model with LLAMA3. Results demonstrate that debate training enhances the model's ability to produce stronger, more persuasive arguments, which in turn improves the accuracy of the judge model. In contrast, non-adversarial methods do not exhibit this positive correlation.
weaknesses: - Limiting experiments to a single dataset and task (QuALITY) may affect the generalizability of the conclusions. Incorporating more different tasks could provide broader insights for the community.
- The paper lacks evaluations on fine-grained aspects, such as factuality. For instance, it would be valuable to assess whether debate training results in more factual transcripts and whether the judge model improves its ability to detect factual discrepancies.
- The judge model is based on GPT-4. How would results differ if a smaller language model, such as LLAMA3-8B, were used as the judge model?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Figure 3 (right), what does the judge Brier score represent?

2. In line 264, how is a "win" or "loss" response determined when constructing the DPO training datasets?

3. In line 313, what does "confidence" refer to in the context of the reward function? Was GPT-4 Turbo trained to output a confidence level?

4. In line 318, what do the two win rates in parentheses represent?

5. In line 362, why are there three pairs per round and two rounds per question?
soundness: 3
strengths: 1. The paper introduces a novel experimental design to simulate information asymmetry, rigorously testing their approach by comparing the impact of the debate process with a consultancy process on the final judgment accuracy.

2. The findings are intriguing, revealing a positive relationship between a model’s ability to win debates and the usefulness of its debate transcripts for uncovering correct answers. In contrast, the non-adversarial "consultancy" approach shows no positive correlation between the judge's accuracy and the skill level of the consultant.

3. These findings have the potential to shape future research on "super alignment" or "scalable oversight" by establishing a foundational debate-then-judge framework.
summary: The paper investigates the robustness of training debater models to enhance the accuracy of judgments in long-context reading comprehension tasks. It introduces a practical and emerging scenario called "super-alignment" or "scalable oversight" where it is assumed that AI systems may eventually surpass the wisdom of human supervisors  and underscores the need to explore methods by which a weaker model could effectively supervise a stronger one. To emulate this, the paper creates an asymmetric information environment, allowing the debater model access to the full context while withholding it from the judgment model. The findings suggest that debate training fosters the development of stronger, more informative arguments, indicating its potential for delivering high-quality supervision in tasks that are challenging to evaluate directly.
weaknesses: While I find the experimental setting intriguing, some aspects of the evaluation are unclear:

1. In Section 2.4, the use of "self-play" to evaluate judge accuracy is ambiguous. From my perspective, there shouldn’t be a need for the judge to "self-play." Instead, judge accuracy should be evaluated by its ability to correctly discern outputs from the debater and consultant. Specifically, if the judge evaluates a debater advocating for an incorrect answer as incorrect, it should be marked as correct, and conversely, if it rates a debater with a wrong answer as correct, it should be marked as incorrect, and vice versa.

2. In line 187, the method for calculating the consultant win rate remains unclear. If the consultant is assigned to defend an incorrect answer, under what conditions is this considered a "win"? Similarly, if the consultant defends a correct answer, what conditions would lead to a "loss"?

3. There is insufficient detail on the training of the judges, including the instances in the training dataset and the format of input and output. Is there evidence to support line 215, which states that judge decisions “were no longer clustered at the boundaries as they were prior to fine-tuning”?

4. In line 473, the conclusion that "explicit refutation does not yet seem to play a role in judge decision-making in our setting" is not sufficiently substantiated and lacks experimental evidence to support it.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. See weaknessnes.

2. Figures 1, 2, and 5 are not referenced in the text. It would be helpful to cite them in the relevant sections to enhance clarity and support the descriptions.
soundness: 2
strengths: 1. This paper introduces debate as a method for scalable oversight by leveraging self-play for training language models, which shows promise for improving evaluator accuracy in complex tasks.

2. This paper discusses and compares multiple baselines (e.g., single, ensembled, and double consultancy).
summary: This work explores the use of debate as a scalable oversight method for language models. The proposed method trains models to debate using self-play and assess whether this approach enhances the accuracy of AI-based evaluators in judging complex tasks, such as reading comprehension. The key contribution of the paper is demonstrating that models trained to debate in adversarial settings lead to more accurate judgments by AI evaluators compared to non-adversarial consultancy models.
weaknesses: 1. The paper's experiments are limited to reading comprehension tasks, which raises concerns about the generalizability of the findings to other complex reasoning domains. Could the effectiveness of debate training extend to tasks beyond reading comprehension?

2. The debate protocol is limited to a **two-turn** setting, which restricts the depth of argumentation and refutation between debaters. This simplified structure may not fully capture the complexities of real-world debates, where extended back-and-forth exchanges are often necessary to expose subtle flaws in reasoning. As a result, the findings may overestimate the effectiveness of debate in fostering accurate judgments, since more intricate discussions could reveal different dynamics or weaknesses in the model's performance​.

3. While the experiments provide some insights, they are not entirely convincing. The study only employs GPT-4-Turbo as the judge and Llama3-8B-Instruct as the debate and consultancy models. Evaluating the proposed method across a broader range of model architectures and sizes would strengthen the assessment of its generalizability.

4. Although debate is presented as a mechanism for scalable oversight, the paper finds little evidence that explicit refutation materially affects the judge’s decision-making. This undermines one of the key proposed benefits of debate as an oversight mechanism​.

Meta-review: Summary: 

This paper studies the robustness of debate as a method of scalable oversight, by training models to debate with data generated via self-play. It finds that language model based evaluators answer questions more accurately when judging models optimized to win debates in a long-context reading comprehension task. The paper finds no such relationship for consultancy models trained to persuade a judge without an opposing debater present. It conducts quantitative and qualitative comparisons between the debate models and novel consultancy baselines, and finds evidence that debate training encourages stronger and more informative arguments, which can help provide high-quality supervision for tasks that are difficult to directly evaluate.

Strengths:

Reviewers generally agree that the ideas introduced in the paper to explore debate model training to enhance evaluator accuracy are novel. The findings are intriguing and have the potential to shape future research.

Weaknesses: 

The major weaknesses reviewers find in general can be summarized into two aspects: 

1. Writing. For example, the abstract and introduction are hard to read, and the differentiation from existing work is not clear. 

2. Evaluation: (1) the focus on only one dataset, (2) limitations to the LLama3-8B-Instruct model. These limitations make the results (despite interesting) not completely convincing.

Decision: Reject