Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks
Authors: Linbo Cao
Jinman Zhao
Venue: COLM 2025 (2025)
Keywords: debate-driven evaluation
QA benchmarks
multi-agent debate
language model evaluation
benchmark contamination
model memorization
adversarial evaluation
dynamic assessment
TL;DR: We introduce a debate-driven evaluation paradigm that transforms existing QA benchmarks into adversarial debates between models, providing a more robust assessment of reasoning abilities while penalizing shallow memorization.

Abstract: As frontier language models increasingly saturate standard QA benchmarks, concerns about data contamination, memorization, and escalating dataset creation costs persist. We propose a debate-driven evaluation paradigm that transforms any existing QA dataset into structured adversarial debates—where one model is given the official answer to defend, and another constructs and defends an alternative answer—adjudicated by a judge model blind to the correct solution. By forcing multi-round argumentation, this approach substantially increases difficulty while penalizing shallow memorization, yet reuses QA items to reduce curation overhead. We make two main contributions: (1) an evaluation pipeline to systematically convert QA tasks into debate-based assessments, and (2) a public benchmark that demonstrates our paradigm's effectiveness on a subset of MMLU-Pro questions, complete with standardized protocols and reference models. Empirical results validate the robustness of the method and its effectiveness against data contamination—a Llama 3.1 model fine-tuned on test questions showed dramatic accuracy improvements (50% → 82%) but performed worse in debates. Results also show that even weaker judges can reliably differentiate stronger debaters, highlighting how debate-based evaluation can scale to future, more capable systems while maintaining a fraction of the cost of creating new benchmarks. Overall, our framework underscores that "pretraining on the test set is no longer all you need," offering a sustainable path for measuring the genuine reasoning ability of advanced language models.

Published as a conference paper at COLM 2025
Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks
Linbo Cao *f Faculty of Mathematics University of Waterloo Waterloo, ON, Canada 16cao@uwaterloo..
Jinman Zhao
Department of Computer Science University of Toronto Toronto, ON, Canada jzhao@cs, toronto. edu
Abstract
As frontier language models increasingly saturate standard QA bench- marks, concerns about data contamination, memorization, and escalat- ing dataset creation costs persist. We propose a debate-driven evaluation paradigm that transforms any existing QA dataset into structured adversar- ial debates- one model is given the official answer to defend, and another constructs and defends an alternative answer adjuur--jjudieteedd by a judge model blind to the correct solution. By forcing multi-round argu- mentation, this approach substantially increases difficulty while penalizing shallow memorization, yet reuses QA items to reduce curation overhead. We make two main contributions: (1) an evaluation pipeline to systemat- ically convert QA tasks into debate-based assessments, and (2) public benchmark that demonstrates our paradigm's effectiveness on subset of MMLU-Pro questions, complete with standardized protocols and reference models. Empirical results validate the robustness of the method and its effectiveness against data contamination Llama 3.1 model fine-tuned on test questions showed dramatic accuracy improvements (50% + 82%) but performed worse in debates. Results also show that even weaker judges can reliably differentiate stronger debaters, highlighting how debate-based evaluation scale to future, more capable systems while maintaining a fraction of the cost of creating new benchmarks. Overall, our framework underscores that "pretraining on the test set is longer all you need," offering a sustainable path for measuring the genuine reasoning ability of advanced language models.
1 Introduction
Benchmark saturation is critical challenge in NLP evaluation (Ott al., 2022), driven by rapid advances in large language models (L.LMs) such GPT-4 (OpenAI, 2023; Bubeck et al., 2023), OpenAl's ol (OpenAT, 2024c), and DeepSeek-RU (DeepSeek-AT, 2025). Continuous model improvements quickly exhaust existing benchmarks, evidenced by the progression from GLUE (Wang et al., 2019b) to SuperGLUE (Wang et al., 2019a), and from MMLU (Hendrycks et al., 2021) to MMLU-Pro (Wang et al., 2024), as well the emergence of tougher tests like ARC-AGI (Chollet, 2019; Chollet et al., 2024) and HLE (Phan et al., 2025). Such developments highlight fundamental evaluation challenges Al capabilities approach artificial general intelligence (AGI) or artificial superintelligence (ASI). Another major issue, data contamination, occurs when models achieve artificially inflated scores due to training on benchmark test data, as illustrated by Schaeffer (2023). Comprehensive reviews by Sainz al. (2023) emphasize this issue, prompting solutions like LiveBench (White al., 2025), which regularly introduces new questions to minimize contamination, and interactive evaluations like KIEval (Yu et al., 2024) that distinguish memorized responses from true understanding. Resource limitations further complicate benchmark creation, exemplified by
"Equal contribution.
I 'Corresponding author.

---

Published as a conference paper at COLM 2025
Question: 1+1=?
Standard Answer (A): 2
Pro is given answer and must defend it.
Strong Model
Weak Model
Con must propose alternative answer and defend it.
Scenarlo (Strong) Con (Weak)
Scenarlo 2: Pro (Weak) vs Con (Strong)
1+1=2. This fundamental. arithmetic operation where we combine one unit with another unit,
1+1-2 because that's always been that way. Count it on your fingers one fin ger plus another finger equais two fingers.
school and
resulting in two units total.
1-1-1, in abstract ways. For example, mixing water drop with another still gives you water drop. Similarty, you combine cloud. These examples show that in certain contexts, combining two units can result in a single unit.
Not necessarly. Alternative answers can 1+1=10 binary cases) or 1+1-0 modular Both atternative answers hold credibliity, so not be appropriate to conclude 1-1-2 without further information
clouds, you still
number system being
Figure 1: Illustration of how debates might encourage deeper reasoning, distinguishing strong and weak models beyond superficial correctness.
HLE's reliance on nearly 1000 experts from over 500 institutions (Phan et al., 2025), making such efforts increasingly unsustainable.
Current evaluation alternatives include holistic frameworks such as HELM (Liang et al., 2023), offering multidimensional metric analyses, and head-to-head evaluations like Chatbot Arena (Chiang et al., 2024), though it might suffer from inconsistent evaluation criteria. Automated assessments with L.M-baasd judges, such as Auto-Arena (Zhao et al., 2024), face reliability challenges. Existing debate-based evaluations, including FlagEval (Beijing Academy of Artificial Intelligence, 2025) and methods from Bandi Harrasse (2024), often lack standardization, typically relying open-ended questions (e-g., MT-Bench (Zheng et al., 2023)). Our method addresses this gap by converting structured QA tasks into adversarial debates, utilizing official answers to facilitate objective assessment, reducing biases common in open-ended evaluations.
Our core approach, debate-driven QA evaluation, modifies existing QA tasks by retaining only questions and correct answers, removing incorrect alternatives. A Pro model supports the official answer, while Con model proposes and defends alternative. Keeping all original options would compel defending obviously incorrect choices, creating alignment issues; conversely, fully open-ended formats risk amplifying evaluator bias both debaters produce incorrect solutions. This structured debate system integrates smoothly with existing datasets, featuring double round-robin formats with role reversals to eliminate positional bias. Assigning explicit adversarial roles incentivizes deeper reasoning, penalizing super- ficial memorization and requiring coherent logical arguments, as illustrated in Figure 1. Finally, answer-bling judge assesses arguments purely based quality, prioritizing genuine reasoning over memorized answers.
We introduce systematic, reproducible evaluation pipeline transforming standard QA datasets into structured adversarial debates, effectively reducing subjective biases, clearly signaling model reasoning capabilities, and significantly decreasing benchmark curation costs. Empirical experiments with various commercial and open-source models on MMLUU Pro (Wang et al., 2024) validate method's robustness and resistance to data contami- nation. Additionally, weaker judges reliably distinguished stronger debaters, reinforcing findings by Khan et al. (2024), thereby ensuring scalability and future-proof applicability for advanced AI evaluations. We plan to publicly release standardized benchmark derived from 5,500 structured debates debates-totalingg totaling over 11,000 rounds of argumentation inventationiinoolvinng eleven reference models, providing the research community with dependable evaluation resource.
"Cdde for this project available at https://ee thub. wwtp:///ww./ebaaeeeeeaooeeeeeeee htt.aa////aaaat----iivvvellaaaaa
2

---

Published as a conference paper at COLM 2025
2 Related Work
Traditional QA Benchmarks and Limitations. The evolution of QA benchmarks has wit- nessed increasing saturation as language models rapidly advance. Early benchmarks like GLUE (Wang et al., 2019b) quickly led to SuperGLÜE (Wang al., 2019a), while MMLU (Hendrycks et al., 2021) spawned more challenging variants including MMLU-Pro (Wang et al., 2024), MMLU-Pro+ (Taghanaki et al., 2024), and C-MMLU (Li et al., 2024b). Other specialized benchmarks emerged, such as GSM8K (Cobbe et al., 2021) for mathematical reasoning, GPQA (Rein et al., 2023) for expert-level questions, and BIG-Bench (Srivastava et al., 2023) with over 200 diverse tasks. Advanced models including OpenAT's GPT-4 (Ope- nAI, 2023), Claude 3.5 Sonnet (Anthropic, 2024), Google's Gemini (Pichai et al., 2024), and open-source alternatives like DeepSeek-V3 V3 (DeepSeek-Al) 2024), and recent advancements in reasoning models such as OpenAl's ol (OpenAI, 2024c) and DeepSeek-RI (DeepSeek-A) AI, 2025), employ chain of thought (Wei et al., 2022), demonstrating performance gain with sound reasoning (Bi et al., 2025). Concurrently, the community's ' enthusiasm for enhancing LLM reasoning has remained high, yielding significant new methods continuously, such as Shain-of-Thoughh
prompting (Wei et al., 2022), Self-Consistency (Wang et al., 2023), Multi-Agent Debate (Liang et al., 2024), and deductive-inductive frameworks (Cai et al., 2025). Such models and methods post threat to saturation of these benchmarks, necessitating increasingly difficult challenges such as ARC-AGI (Chollet, 2019), ARC-AGI-2 (ARC Prize Foundation & Kamradt, 2025), Humanity's 's Last Exam (Phan et al., 2025), and FrontierMath (Glazer et al., 2024). Economic constraints further compound these involved 450 authors across 132 institutions, HLE mobilized nearly 1000 subject experts from over 500 institutions, and FrontierMath assembled 60+ mathematicians including IMO gold medalists- heedalstt--highlighttng the unsustainability of frequent benchmark creation.
BIG-Bench
Data contamination represents another critical challenge to benchmark integrity. Sainz et al. (2023) highlight how test set leakage threatens evaluation validity, while Schaeffer (2023) satirically demonstrate the possibility of achieving perfect scores by directly training test data. Studies by Balloccu et al. (2024), Golchin & Surdeanu (2024), and Xu et al. (2024) document widespread contamination issues, especially in closed-source models, undermining the reliability of performance com comparisons. To this end, researchers have developed sophisticated detection techniques, such as showing that models can guess masked test answers (Deng et al., 2024) or using statistical tests to prove contamination from model outputs (Oren et al., 2024; Shi et al., 2024). Researchers have proposed various solutions to address this problem, including continuous benchmark refreshes via LiveBench (White et al., 2025), filtering contaminated examples (Gupta et al., 2024), and dynamic evaluation protocols like KIEval (Yu et al., 2024) and DyVal (Zhu et al., 2024). However, these approaches either rely on expensive private datasets or cannot fully resolve contamination in established benchmarks, leaving significant challenges that our debate-driven methodology directly addresses.
Multi-Agent Debate (MAD). Multi-agent debate originated from AI Safety via Debate (Irving et al., 2018), introducing adversarial dialogues where agents advocate positions evaluated by human judges. Recent studies like Liang et al. (2024) extended this to foster divergent reasoning in L.Mss. Debates help achieve truth emergence, with Khan et al. (2024) showing debates among advanced models enable weaker judges to better discern truthful answers, while Lang et al. (2025) demonstrated debate's 's capacity to enhance weak-to-strong model alignment and reduce hallucinations. Du et al. (2024) and Li et al. (2024c) confirmed iterative model interactions significantly mitigate LLM hallucinations. The success of debate frame- works depends on judge models-Zheen et al. (2023) found LLM-based judges effectively approximate human judgment, Liu et al. (2024) confirmed LLM judges can surpass human accuracy but warned of lexical biases, and Khan et al. (2024) showed even weaker judges can effectively evaluate stronger debaters. Implementation variants include Bandi & Harrasse (2024)'s 's courtroom-style multi-agent debate, Moniri et al. (2024)ss automated model ranking, Beijing Academy of Artificial Intelligence (2025)'s 's broader framework, and Rahnamoun & Shamsfard (2025''s multi-layered metrics. Unlike existing approaches, our work specifically adapts debates to structured QA tasks, addressing data contamination while ensuring clear reasoning assessments.
3

---

Published as a conference paper at COLM 2025
Dynamic Evaluation Methods. Dynamic evaluation methods offer flexible assessments of language models beyond static benchmarks. Chatbot Arena (Chiang et al., 2024) exemplifies head-to-head comparison through user-driven evaluation, though with inconsistent judging criteria. LLM--bsee automated evaluation has emerged as an alternative-Auto Arena (Zhao et al., 2024) utilizes LLM judges for model comparison, FlagEval (Beijing Academy of Artificial Intelligence, 2025) implements debate frameworks, KlEval (Yu et al., 2024) focuses on distinguishing understanding from memorization, and others use structured or Socratic questioning to evaluate discourse comprehension and faithfulness (Wu et al., 2023; -aalaag---odee---------------
(Bai et al., 2023) also creates adaptive questions. Comprehensive surveys by Li et al. (2024a) and Gu et al. (2024) examine such LLM-as-judge systems. These approaches require robust ranking mechanisms like Elo (Elo, 1978) for skill progression, Bradley-Tere models (Bradley & Terry, 1952) for probabilistic comparison, and TrueSkill (Herbrich et al., 2006) for Bayesian inference. Despite their advantages, these methods struggle with itttucibblity-vvriattion in prompts, match ordering, and evaluator biases yield inconsistent results. Current research aims to balance flexibility with consistency for reliable model evaluation.
Miao et al., 2024).
3 Debate-Based Evaluation Framework
This section describes our methodology for adapting QA benchmarks into structured debates, systematic evaluation protocols, and standardized benchmark creation from debate outcomes for efficient model comparisons.
if round
if round
round
continue
Winner: (Pro
Postiiv
Négativa
Jucge
Input: (Pro Instruction.
Inout: (Con Instructinn. (Quastion).
(Question)
Input: [Audge Insiruction) (Debate History)
Winner: (Winss))
(Correct Answer),
(Correc) Answer Rejected) (Debato History)
if Foonnd
(Debate History)
Output: (Winner Continus)
Output: (Pasitiva Argumenty
Oulput: (Negative Argumenty
Single debate pipeline showing question transformation, role assignment, multi- round exchanges, and blind judging process.
Figure 2:
Our framework converts QA datasets with clearly defined answers, such as MMLU-Pro (Wang et al., 2024), into structured debates by removing multiple-choice distractors. This grants the Con side freedom to propose alternatives while preserving the original correct answer for the Pro side. Keeping all original distractors would force debaters into defending obviously incorrect answers, creating alignment concerns, while entirely open-ended formats could amplify judge bias when both sides are incorrect.
Debate Overview:
Our pipeline assigns distinct roles: the Pro model defends the provided correct answer, and the Con model, instructed that the official answer is incorrect, proposes and defends an alternative. This adversarial approach incentivizes deeper reasoning rather than memoriza- tion. Debates consist of multiple rounds (2-5), range chosen to balance argumenta tive depth with computational efficiency. This choice is informed by prior work (Liang et al., 2024; Du et al., 2024), which found diminishing returns in debate quality beyond this scope. A minimum of two rounds mitigates judge biases, particularly in data contamination scenarios. no clear winner emerges after five rounds, the Pro side is awarded victory for demonstrating sustained defense. Figure illustrates the debate pipeline, including question transformation, role assignment, multi-round exchanges, and blind judging.
Judges evaluate debates blindly, seeing only the question and debate transcript, and return verdicts as "Positive" (Pro wins), "Negative" (Con wins), or "Continue" (another round required), ensuring unbiased evaluation based on argument quality alone.
Evaluation Protocols: Our evaluation employs a double round-robin format, where each model debates all others in both Pro and Con roles to mitigate positional biases. Models
4

---

Published as a conference paper at COLM 2025
For all M, M, where i+t
Models
Dataset
Question
Pro
Con Winner
M1
<Dataset>
2 5 Rounds
Q2
Mi
M|
Mi / Mj
Judge
<Judge
Mn
Model>
2 5 Rounds
Qn
M
M
M. / M,
Figure 3: Double round-robin evaluation process showing model pairings, role alternation, and aggregate scoring system.
effective at defending correct answers may struggle in adversarial roles, and contamination may artificially enhance defensive performance. Figure 3 depicts the double round-robin process, including model pairings, role switching, and scoring methods.
The primary evaluation metric is total win count, enabling reproducible, order-independent rankings. Debate results are logged for analysis and future use.
Benchmark Creation: We create standardized benchmarks through reference debates be- tween established models, reusable in subsequent evaluations. New models only debate against selected reference models in both roles, supplemented by stored reference debates. Introducing one new model at time preserves integrity, enabling consistent scoring across evaluations.
For ranking, we examine various methods: simple win counts (complete tournaments required), Elo ratings (Elo, 1978) (progressively updated scores), Bradley-Terry models (Bradley& & Terry, 1952) (probabilistic pairwise comparisons), and TrueSkill (Herbrich et al., 2006) (Bayesian rankings suited for incomplete comparisons). Our approach first prioritizes stability and usability of reference model scores, ensuring minimal fluctuation as new models are introduced, and subsequently emphasizes producing benchmark outcomes that are globally comparable.
This framework supports reproducible, consistent evaluation across implementations, al- lowing reliable comparison of model reasoning without creating additional QA datasets.
Experimental Setup
We randomly sampled 50 questions from the MMLU-Pro benchmark (Wang et al., 2024) as our evaluation dataset. Our main experiment involved eleven diverse models: DeepSeek V3 (DeepSeek-AI, 2024), Claude 3.5 Sonnet (Anthropic, 2024), GPT-40 (OpenAI, 2024a), GPT-40 mini (O,enAL, 2024b), CPT-3.5-turbo, (MistralAI, 2025), Mixtral 8B (Jiang et al., 2024), Mixtral 8x 22B (MistralAI, 2024), Mistral 7B (Jiang et al., 2023), and Llama 3.1 8B (Grattafiori et al., 2024). For our main experiment, we used GPT-4o as the judge model. To confirm the generalizability conducted an evaluation the GPQA main test set (Rein et al., 2023) (448 questions) using several open-source models, with the full setup detailed in Appendix C.
Claude 3.5 Haiku (Anthropic, 2025), Mistral Large
our method, we also
To assess data contamination effects, we performed fine-tuning analysis of Llama 3.1 8B using LoRA (Hu et al., 2022) trained explicitly on the test set. We also conducted separate experiment to test judge model variation, using seven different judges: Mistral Large, GPT-4o, GPT-4o mini, Mixtral 8 x 7BB, Mistral 7B, along with the original Llama 3.1 8B and its newly fine-tuned variant. Each debate followed our structured protocol (Section 3)
Phttps: htt.a//awa....ppeaaccc////////ee com/doccs//d/ellll

---

Published as a conference paper at COLM 2025
Table QA Accuracy (50 MML.--rr 0-shot CoT) vs. Debate Win Counts.
Model
QA Acc (%)
QA Rank
Debate Wins
Debate Rank
A
Claude 3.5 Sonnet
80
1
718
2
-1
DeepSeek
74
2
759
1
+1
GPT-4o
72
3
581
5
-2
Mistral Large
70
4
636
3
+1
GPT-4o mini
62
5
564
6
-1
Claude 3.5 Haiku
60
6
621
4
+2
Mixtral 8x 22B
56
7
385
8
-1
Liama 3.1 8B
50
8
348
9
-1
GPT-3.5-turbo
48
9
263
10
Mixtral 8x 7B
38
10
395
7
+3
Mistral 7B
34
11
230
11
with double round-robin format, aggregating model win counts as the primary metric. For benchmark standardization, we included our dataset of 5,500 debate transcripts-which collectively contain over 11,000 argumentative rounds-spanning the full double round- robin interactions among the eleven reference models, tested various scoring methods including Elo, Bradley-Terry, and TrueSkill, and ultimately selected TrueSkill due its robustness on partial data.
Results and Analysis
5.1 Overall Performance Comparison
Traditional QA Accuracy. We first measure standard QA accuracy for all eleven models on 50 MMLU-Pro (0-shot CoT) questions. Each model's final answer checked for exact match with the official solution in single-pass format. Despite notable gaps-with Claude 3.5 Sonnet scoring 80% and Mistral 7B trailing 34% severl models still achieve relatively scores
high scores, suggesting partial saturation even under conventional single-turn QA.
Debate Win Counts. We next conduct multi-round, double round-robin debates, with models alternating between defending (Pro) and questioning (Con) the official answers. Debate outcomes, summarized in Table 1, show generally strong correlation with single- turn QA accuracy-most models shift by only one or two positions. However, debates add valuable nuance: Claude 3.5 Haiku climbs two ranks due to superior argumentation, while GPT-4o drops slightly, highlighting that debate evaluation effectively captures deeper reasoning skills beyond standard QA performance.
We further analyze debate performance by isolating results from models exclusively in the defender (Pro) role, reflecting their ability justify and maintain correct solutions. Figure 4 compares total debate wins (combined Pro and Con) to Defending-Oni Wins and Questioning-On Wins, illustrating the consistency of model rankings across these roles.
5.2 Role-Specific Performance Analysis
The rankings based solely Defending-Only Wins perfectly align with overall debate win rankings for the top nine models, underscoring the robustness of our evaluation framework. The minor discrepancy observed between GPT-3.5-turbo and Mistral 7B at the bottom is negligible, indicating that a model's overall debate success consistently reflects its genuine understanding and problem-solving capabilities.

---

Published as a conference paper at COLM 2025
Comparison of Total, Defending-Only, and Questioning-Only Wins
739
718
Total Wins
700
633
Defending Oifending--Only Wins
600
564
Questioning- Only Wins
500
wins
400
401
858
350555
365
300
937
2.99
91.7
202
276
277
212
2.99
230
200
L76
751
100
91
DeepSeek V3
Claude 3.5 Sonnet
Mistral Large
Claude 3.5 Haiku
GPT-40
GPT-4o Mini
Mixtral 8
Mixtral 8x22B
Llama 3.1 8B
GPT-3.5 GPT-3.5-turba -.ttubbb
Mistral 7B
Figure 4: Comparison of Total Debate Wins vs. Defending-Onie and Questioning-Onl Wins.
SeepSeek V3 aax bika bod
084 0.4 6/
DeepSeck V3 33s bise
2-1 0.
358
0:01
Claude 3.5 Sonnet -448
0.02
0:5
0.01
1.0
Claude 3.5 Sonnet-cxe
3:64 0.00
0.02
0.30
1.0
Mistral Large 625 933
0:77
Mistral Large 4:2 a32
353
0.00
0.02
0.0
Claude Haiku
Claude 3.5 Haiku n:
Model
GPT44 mini-
GPT-4o
0.6
Moda
GPT-4o mini
0.6
0.4
0.4
432
Llama 3.1
<37 334
0.2
Llama 3.1
021
0.2
GPT-3. 5-turbo
Fe 412 2: can
0.0
GPT-3. 5-turbo
1 814
BM
0.0
Mistral 78
mm at1 12
Mistral 78
te 112
Claude 3.5 Sonnat
Mistral Large
Claude Halku
GPT-4
mini
Mixtral 8
Mixtral Bx223-
Liama 3.1 8B
CPT-3 5-turbo
DecepSekk V3
Claude 3.5 Sonnet
Mistral Large
Halku
GPT-40
GPT 1:40 mini-
Mixtral 6x78
Mixtral 8x228-
Liama 3.1 88
5-turbo
DeepSeck
Mistral
Mistraal
Opponent
Opponent
(a) Combined Win Rates
(b) Defending-Only Win Rates
Figure 5: Pairwise head-to-head win rate heatmaps across all 11 models. Each cell (i, shows the win rate of model against model j. (a) combines Pro and Con roles, while (b) focuses only on defending correct answers. These reveal clear ranking structure and robustness against positional and contamination effects.
5.3 Head-to-Head Model Comparisons
Pairwise Heatmaps and Transitivity Analysis. We constructed pairwise win rate heatmaps across three crenarios-questionee
defending, and overall-to refine the rankings in Table 1. Stronger models consistently dominated weaker ones, typically securing 70-90% win rates. While few minor loops were observed in the defending-onle and questioning- only settings, notably, the combined heatmap resolved almost all of these inconsistencies. Out of 55 unique model pairings, the combined results produced a remarkably consistent hierarchy, with only single, marginal transitivity violation. This near-perfect (98%+) transitivity strongly suggests our role-switching approach effectively mitigates positional bias and data contamination, where contaminated models may artificially excel in defense roles. This high degree of fidelity implies our evaluation genuinely captures meaningful differences in models' reasoning capabilities. For instance, even when provided with
7

---

Published as a conference paper at COLM 2025
the correct answer, the weaker model (Mistral 7B) achieved only 10% win rate against DeepSeek V3, clearly underscoring our framework's effectiveness in distinguishing true model competence. Furthermore, this high level of consistency was confirmed in our supplementary evaluation on the GPQA benchmark (Appendix C), where the rankings exhibited perfect transitivity, underscoring the robustness of the method across different datasets.
Partial Tournament Validity, A key finding from our large-scale experiments is the near- perfect transitivity of debate outcomes. As evidenced by both the combined win rate heatmap (Figure 5a) and our confirmatory evaluation on the full GPQA dataset (Ap- pendix w--hhich seldom showed intransitive loops-a clear and stable performance hierarchy consistently emerges. This high degree of consistency has a critical practical implication: if a new model is evaluated and found to win against reference model A but lose model B, we can conclude with high confidence that its capability lies within the range of A to B. This property enables a highly efficient, binary search-like evaluation proto- col. In practice, this shifts the computational requirement for benchmarking a new model from a costly linear scan against all reference models (O(n) to much more manageable logarithmic one (O(log n)), where n represents the number of reference models in the debate benchmark set.
5.4 Fine-Tuning Impact Assessment
Table Debate performance (Win Rate) of Llama 3.1 8B before and after fine-tuning. "Theoretical win rate for self-match; role-specific rates (-) are not applicable.
Model
Llama 3.1 8B Defending Questioning
DeepSeek V3
Overall
Overall
Defending
Questioning
Llama 3.1 8B
0.50*
0.17
0.22
0.12
Llama 3.1 8B Finetuned
0.46
0.48
0.44
0.16 L
0.26
0.06 |
We conducted a fine-tuning experiment using LoRA on the Llama 3.1 8B model with the evaluation set, resulting in significant accuracy improvement from 50% to 82%. However, this improvement in standard QA did not translate to enhanced debate performance (Ta- ble 2). Compared its non-fine-tuned baseline, the fine-tuned model performed worse overall against both the original Llama 3.1 8B (0.50-0.46) and the state-of-the-art DeepSeek V3 (0.17-+0.16), particularly suffering a substantial drop in questioning ability (0.12-0..06.
The disparity between improved static accuracy and decreased debate efficacy suggests that fine-tuning primarily enhanced memorization without developing deeper comprehension, aligning with similar findings by KIEval (Yu et al., 2024). Crucially, our debate-based approach exposes and mitigates such contamination effects, effectively distinguishing genuine understanding from superficial recall. This demonstrates that our method robustly evaluates model reasoning capabilities without requiring dataset filtering or the creation of new benchmarks.
5.5 Judge Model Variations
Multi-Judge Experiment. To assess judge model variation, we conducted seven indepen- dent debate toumaments. Each tournament featured the same four debater models (Mistral Large, GPT-4o Mini, Mixtral 8x7B, and Mistral 7B) but was adjudicated by one of seven distinct judges, including one contaminated model (Llama 3.7 8B FT). Table 3 summarizes the cumulative win counts from each of these judge-led tournaments (pairwise results provided in Appendix Figure 7 and Table 4). Despite varied judging capabilities, six of the seven judges- incluudin the contaminated model produuced identical debater rankings: Mistral Large > GPT-4o Mini > Mixtral 8x7B > Mistral 7B, confirming the robustness of the debate outcomes across different adjudicators.
Rank Consistency and Judge Limitations. Six judges, including the contaminated one, as evidenced by the pairwise heatmaps (Figure 7), yield entirely consistent rankings with
8

---

Published as a conference paper at COLM 2025
Table 3: Total win counts for each debating model (columns) as evaluated by different judge
models (rows). Judges are ordered by capability (strongest to weakest). The rankings of debaters remain identical, with the exception of the weakest judge (Mistral 7B), which failed to differentiate between models.
Judge Model
\
Debater
Mistral Large
GPT-4o Mini
Mixtral 8x
Mistral 7B
Mistral Large
209
176
122
93
GPT-4o
215
191
126
68
GPT-4o Mini
233
192
114
61
Mixtral 8
202
190
127
81
Llama 3.1 8B
167
162
144
127
Llama 3.1 8B Finetuned
169
167
147
117
Mistral 7B
150
151
150
149
minimal intransitive results (e.g, no significant cycles such as A> A>B>C>> A). However, Mistral 7B exhibited poor discrimination because it appears lack sufficient long-context evaluation and instruction-folloinng capability. It consistently failed to output the required judgment formats, causing the system to keep falling back the default positive option. Thus, we conclude that limited by its of judging. Yet, given the closely aligned results from the remaining judges, we may conclude the ranking is fairly robust and consistent across judge models, as long as they possess sufficient instruction-folloiing capabilities.
capabilities, it is not yet capable
Superintelligence Readiness. Importantly, our experiments support conclusions from Khan et al. (2024), demonstrating weaker models can still reliably evaluate stronger ones within structured debate settings. Unlike conventional QA tasks with fixed performance ceilings, our framework's ' adversarial nature provides theoretically unlimited measurement range. Consequently, even future superintelligent models would be meaningfully assessed, suc- cess in debates necessitates robust and explicit reasoning that weaker judges can effectively evaluate.
5.6 Ranking Algorithm Comparison
Why Not Simple Win Counts, Elo, or Bradley-Teny? A naive approach is to rank by total wins, but this requires complete double round robin and is unsuited for incremental participation. Elo simplifies sequential updates yet suffers from order dependence and partial-match instability. Bradley-Terry (BT) fits pairwise outcomes via logistic regression, which reduces sequence effects; however, as shown in Tables 6 and 7, even a single new model can shift reference-model scores drastically (e.g., DeepSeek V3 from 1963.0 to 1981.0), undermining benchmark stability.
TrueSkill for Stability. TrueSkill uses Bayesian factor-graph framework to handle skill inference in tournaments with incomplete or disordered match data. Our chosen parameters 25, 8.333, 8 == 4.5,
0.01 consistently align scores with observed win counts, while limiting rating shifts when new participants appear (e.g., DeepSeek V3's score moving only 492.0 to 489.4). Such minimal disturbance preserves previously established orders and ensures reliable integration of newly added models.
Practical Considerations. Real-world leagues rarely conduct full round robins; new models enter on rolling bases, some matchups remain unplayed, and computational resources vary. TrueSkill's -art capability ensures that incomplete data still yields coherent rankings, where established references only marginally affected by new arrivals. Conse- quently, our benchmark employs TrueSkill as the primary ranking mechanism, guaranteeing consistent, reproducible standings across both partial and complete tournaments.

---

Published as a conference paper at COLM 2025
6 Discussion and Conclusion
Our experiments have demonstrated that debate-driven evaluation provides a robust al- ternative to traditional QA benchmarks, addressing key challenges in Al assessment. The framework's effectiveness is validated by several key findings: highly transitive rankings (98%+ consistency) across a diverse set of models, aligned results across different judge models, and reliable detection of shallow memorization in contaminated models.
Our approach directly address data contamination, showing that while fine-tuning can ar- tificially inflate traditional QA performance (50% - 82%), it fails to improve-and can even harm Sarm-perfoomance in debate scenarios. This aligns with findings from KTEval (Yu et al., 2024), confirming that debate formats successfully distinguish genuine understanding from memorization. Furthermore, our method offers remedy for benchmark saturation without creating entirely new datasets-a crucial advantage as creating high-quality benchmarks becomes increasingly expensive (e.g,, HLE's 1000 experts from 500+ institutions (Phan et al., 2025)). By transforming existing QA tasks into more challenging debate scenarios, we extend the useful lifespan of established datasets while raising the evaluation ceiling.
Importantly, our approach demonstrates ASI readiness by providing a theoretically un- bounded measurement space. Unlike conventional benchmarks with fixed performance ceilings, debate assessments scale with model capabilities. This scalability operates on multiple fronts: the methodology can be seamlessly applied to any QA dataset, including the most difficult existing benchmarks and those yet to be created for future systems. The performance ceiling is also practically limitless; achieving a perfect score requires not merely correctness, but comprehensively out-arguing all opponents on all questions-an an excep- tionally high bar. The foundation of this auture-proof design lies in its ability to facilitate weak-to-strong evaluation through structured debate. Our empirical results strongly ground this theoretical soundness: we demonstrated that even significantly weaker or contaminated models can reliably adjudicate debates and produce consistent rankings of stronger debaters (Table 3), corroborating findings from Khan et al. (2024). This unique combination of adapt- ability, high performance ceiling, and empirically validated weak-to-strong assessment makes our framework robust and valuable tool for evaluating advanced systems as they approach superintelligence.
The cost efficiency of our method represents strategic trade-off. While evaluating new model is more computationally intensive than standard few-shot methods, this is mitigated by one-time benchmark creation cost and efficient partial tournaments for subsequent models. Critically, this approach becomes increasingly favorable over time: inference costs are predictably declining, while the expense of creating new, sufficiently hard benchmarks is escalating sharply (e-g., FrontierMath's mobilization of 60+ mathematicians (Glazer et al., 2024)). By recycling existing high-quality datasets, our framework offers sustainable and future-proof ev evaluation path, sidestepping the prohibitive costs of perpetual benchmark creation while providing a high ceiling for even the most advanced models.
Our study primarily employed only 50 MMLU-Pro questions, limiting domain- specific analysis reliability. While our complete evaluation of 5,500 debates generated over 11,000 rounds of argumentation across 11 models, the computational demands remain mubstantial-thouge justified when compared to creating entirely new benchmarks. Judge model biases, particularly toward persuasiveness rather than correctness, warrant further investigation (Liu al., 2024). The applicability to QA-formatted visual benchmarks for Vision-Language Models (VLMs) also requires further study, especially given VLM-specifii challenges like data contamination (Chen et al., 2024) and adversarial robustness (Dong et al., 2025a;b). Finally, the debate format's fixed structure (2-5 rounds) may not fully capture the nuances of complex reasoning, suggesting room for format optimization in future work.
Limitations:
debate-driven approach offers reproducible, contamination-rrsistaa method for evaluating advanced language models, demonstrating that "pretraining on the test set is no longer all you need" for meaningful AT assessment.
In summary,
10

---

Published as a conference paper at COLM 2025
Ethics Statement
Our debate-driven evaluation framework introduces several design decisions that warrant ethical consideration. We identify and address these considerations below:
Adversarial Role Assignment. Our framework intentionally assigns "Con" model to argue against objectively correct answers. While this might initially appear to reward factual incorrectness, this adversarial approach serves critical evaluation purpose: it forces deeper reasoning from both models, exposing superficial memorization. By removing multiple- choice distractors and allowing the Con model construct its own alternatives, we mitigate the ethical concern of forcing models to defend demonstrably false positions. This approach aligns with common academic practices such as debate competitions and devil's advocate exercises, where arguing opposing viewpoints develops critical reasoning skills.
Persuasiveness vs. Correctness. A potential concern with debate-based evaluation is that might reward persuasiveness over factual correctness. We acknowledge this risk but find it diminishes significantly as question difficulty increases. For complex questions (e.g,, in specialized domains like mathematics, physics, or medicine), persuasive rhetoric without substantive correctness becomes srrfectiveesiillee to how academic evaluation prioritizes sound reasoning over presentation style. Our experiments demonstrate that judge models consistently favor debaters with stronger reasoning capabilities, and nop-performing models in debates align well with traditional QA accuracy.
Judge Model Bias. We recognize that L.LM-based judges may inherit biases present in their training data. Our research mitigates this through multiple safeguards: (1) using structured QA with known correct answers rather than open-ended questions, (2) employing multiple judge models to verify consistency, (3) implementing double round-robin formats with role reversals to eliminate positional advantages, and (4) blind judging protocols that focus only argument quality. These measures collectively reduce the impact of potential judge
biases.
Benchmark Accessibility. By publicly releasing our benchmark, detailed debate logs, and evaluation methodology, we promote transparency and reproducibility in Al evaluation. This open approach allows for community scrutiny and improvement of the framework while democratizing access to evaluation tools that would otherwise require substantial resources to develop.
We believe this framework represents an ethical advancement in Al evaluation that helps distinguish genuine reasoning from shallow pattern matching a crucial capability as AI systems become increasingly integrated into high-stakes domains where real understanding, not mere memorization, is essential.
References
Anthropic. Claude 3.5 sonnet. Online at httpp://wwww.aathhorcccc//o///ss sonnet, 2024. URL https: /wwww www..anthropcc..om. cotp///////////////eeeeeeeee 3- 3-5-
Anthropic. Claude 3.5 haiku. httpss//ww..attrrpot...c/////aaa 2025. 2025.
ARC Prize Foundation and Greg Kamradt. ARC-AGI-2 ARC Prize 2025 is Live! Blog 2025, March
Post, https: /ttpp//i///prrieeee/// http:sss/arrrrr....o//////////// 24 2025.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. Benchmarking foundation models with -aaauagge--mdde--------aaaaaaa
In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL wwtpt::tt///eetteee //apxiv////xi...//// wwt.rrwww......///..ee..... 04181.
11

---

Published as a conference paper at COLM 2025
Simone Balloccu, Patricia Schmidtová, Mateusz Lango, and Ondřej Dušek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms. In Proceedings of the 18th Conference of the European Chapter of the Association for Compu- tational Linguistics, pp. 67-93. Association for Computational Linguistics, 2024. URL https://aclanttologg/ www.y...rg/eeee. org/2024. eacl-long.5.
Chaithanya Bandi and Abir Harrasse. Adversarial multi-agent evaluation of large language models through iterative debates. arXiv preprint arXiv:2410.04663, 2024. URL https: //arxiv.org/abs httpvvvv.org/bb/244000000 24110.4666......
Beijing Academy of Artificial Intelligence. Flageval. Website, 2025. URL URtsttt:://fLlaeeall baai.ac.com
Jinhe Bi, Danqi Yan, Yifan Wang, Wenke Huang, Haokun Chen, Guancheng Wan, Mang Ye, Xun Xiao, Hinrich Schuetze, Volker Tresp, and Yunpu Ma. Cot-kinetics: A theoretical modeling assessing Irm reasoning process, 2025. URL https://arxiv.org/bbb///// 13408.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/ 44)3334444345 1952. doi: 10.2307 /330/2334029. URL http:s//www...oooooogg//aaaaaa/ org/stabble/233402.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. URL https:
httpps://ariiv http://aaii..oo/ab//////// 12712.
Chengkun Cai, Xu Zhao, Haoliang Liu, Zhongyu Jiang, Tianfang Zhang, Zongkai Wu, Jenq-Neng Hwang, and Lei Li. The role of deductive and inductive reasoning in large language models, 2025. URL / http::///xxxi.or///bbb.s. org/abs/2411 241000292...
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 27056-27087. Curran Asso- ciates, Inc., 2024. URL https://proceeedingg neurips.r.... cc/paper files/paper/2004 file/228eee.22766644262222 8e555b5aeb339 oaaaraaco666.ba.222833a33aaaarrr Parpsaaaaebbbppaaprrooooeeeeeee .df..
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael I. Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference. In Proceedings of the Forty-first International Conference on Machine Learning (ICML), 2024. URL https://openrevient net /tt//ffrmmm/--3wwekyyy....
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Are prize 2024: Technical report, 2024. URL htttttsss///xiiv http://aaxxxvv.g/aaaaaa/ wrgp:/s//////////////// 2412/2242.22 04604.
François Chollet. Abstraction and reasoning corpus for artificial general intelligence (arc-agi), 2019. URL https: http:///gtttt....c httpp:://githubb.cooffftt// www.//chollee//RC----C.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:2110.1416: 2021. URL
14168.
DeepSeek-AL. Deepseek-v3 technical report. CoRR, abs/2412.19937, 2024. URL https: //arxiv.org/abs//442..22
DeepSeek-AN Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL httpp::://axxi.orr//bbbbba .221//2211..1.8.... 12948.
12

---

Published as a conference paper at COLM 2025
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investi- gating data contamination in modern benchmarks for large language models. Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Associalion for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8706-8719, Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi: 10.18653/v http://122222/2/al--/aa https://aclanthollen
naw.a-oooo.ag-2222 482. URL
/tgg//22.4422 naacl-long. 482/.
Junhao Dong, Piotr Koniusz, Xinghua Qu, and Yew-Soon Ong. Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 236-247 247, 2025a.
Junhao Dong, Piotr Koniusz, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew- Soon Ong. Improving zero-shot adversarial robustness in vision- lisoun- models by closed-form alignment of adversarial path simplices. In Forty-second International Confer- ence Machine Learning, 2025b.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. In Forty-first International Conference Machine Learning (ICML), 2024. URL https: //tp/v/awvvv///////////eeeeaiiva org abs/ 2305. 14325.
Arpad E. Elo. The rating of chessplayers : past and present. Ishi Press International, Bronx, NY, 2nd printing, 2008 edition, 1978. ISBN 9780923891275. URL https://lccn.1.. 1oc. gov/ 201 2010549499.
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, Lionel Levine, Grant Barkley, Natalie Stewart, Bogdan Grechuk, Tetiana Grechuk, Shreepranav Varma Enugandla, and Mark Wildon. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2024. URL https: //arxiv.org/abs 22/p//11///1111111 04872.
Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamination in large language models. In The Twelfth International Conference Learning Representations (ICLR), 2024. URL https://openrevieee net fo
Aaron Grattafiori, Abhimanyu Dubey, et al. The llama herd of models. arXiv preprint arXiv:2407.21789, 2024. URL https:://arxiie http://axx....r//bbaa// absp 24172/1133
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. A survey on llm-as-a-judge. CoRR, 2bb1/241.155594 2024. URL httpp:://axiv..ort www. 15594.
Vipul Gupta, Candace Ross, David Pantoja, Rebecca J. Passonneau, Megan Ung, and Adina Williams. Improving model evaluation using smart filtering of benchmark datasets, 2024.
URL https://arxiv. ab/////s/////// 24100.//211... 2440..2245.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021. URL https: httt::://ppereviieee www.eew.eeefoorroe n
Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill A bayesian skill rating system. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https: http:///pooocccn...//////// neuriise..s..... cc/ cc/paper filess/paprr/2086/F wwtpeee/ffllle wwteee/file//feee 2tt.:222666.111e4442222255566660 01t1222666651110bb81111111 229d1 wtt f- .pdf.
13

---

Published as a conference paper at COLM 2025
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL https: /wwps//ee//eeeeeeeeett.e. forum?id=nneWWeeFFY.
Geoffrey Irving, Paul Christiano, and Dario Amodei. abs/1805.00899, 2018. URL http::://aaxxiv.org//bb/////
Ai safety via debate. CoRR, 00899.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. URL http:://arxivvorr/aas//////////
06825.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, et al. Mixtral of experts. arXiv 04088.
preprint arXiv:2401.04000 2024. URL http::///aaxiv.og//bbssss
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with more persuasive lims leads to more truthful answers. In Forty-first International Conference Machine Learning (ICML), 2024. URL https::/openreview.neettt net/oormm??dd iLCZt17FTa.
Hao Lang, Fei Huang, and Yongbin Li. Debate helps weak-to-strong generalization. In Proceedings of the AAAI 2025 Special Track on AI Alignment (Oral presentation), 2025. URL 13124.
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims-as-judges: A comprehensive survey on llm-based evaluation methods. CoRR, abs/24122 /00/2241,05579, 2024a. URL https: orgo 2241 055799
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understand- ing in chinese. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 11260-11285. Association for Computational Linguistics, 2024b. doi: 10.18653 11/22/.FINNIINNSAACCAC https://arxiv.orr wttp://..wr//b///////e/e 09212.
URL
Ray Li, Tanishka Bagade, Kevin Martinez, Flora Yasmin, Grant Ayala, Michael Lam, and Kevin Zhu. A debate-driven experiment on Ilm hallucinations and accuracy. arXiv preprint arXiv:2410.19485, 2024c. URL httph::t/////iiei /tttps://arxiv.rrg.... www.:/r//////////////
19485.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Ni- ladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Ko- reeda. Holistic evaluation of language models. Transactions on Machine Learning Research (TMLR), 2023, 2023. URL http::/://///eeeeeeii..e/. http:://pgerrrevvvv...///oooooor www.reee/oruum1.cci044zEEE
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shum- ing Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Process- ing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 17889-17904 Association for Computational Linguistics, 2024. URL https:::: //as::///llaaa http:/aallanttt http:///laaaannl.....// wrw.///...22..... wwn.e-maiiiieeeeeee 992.
Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen
Xinyi Liu, Pinxin Liu, and Hangfeng He. An empirical analysis on large language models in debate evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (ACL) - Short Papers, pp. 470-487, Bangkok, Thailand, 2024. Association for Computational Linguistics. URL https: /actat///allannhoooogy. orgp//24.....222..../// acl short, short.44, .4.
14

---

Published as a conference paper at COLM 2025
Meta AI The Llama 4 herd: The beginning of new of natively multimodal AI
innovation. https://ant /taps://ai..o http://a.mmmaa...e/ee meta. wwt.e..........oeeeeeeeeeeeeetee 4atl:://aaaaa-mmmlmmmoeeeo intellleeceelleeee// 2025.
Yisong Miao, Hongfu Liu, Wenqiang Lei, Nancy Chen, and Min-Yen Kan. Discursive socratic questioning: Evaluating the faithfulness of language models' understanding of discourse relations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meefing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 62776667766 6295, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 11116633/1120244.chaa 341. URL https://ache https://aallantholoogyy wwwpegg...rg////o 2024. acl-long.341 341/.
Microsoft.
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRRs, 2025. URL httpss//arxxi..or//aab///// 01743.
MistralAI. Cheaper, better, faster, stronger. http:://misttralla//new/m/ssiss 8x221 2024.
MIstralAI. Mistral large.
2025.
Behrad Moniri, Hamed Hassani, and Edgar Dobriban. Evaluating the performance of large language models via debates. arXiv preprint arXiv:2006.10140 2024. URL https: 11044.
OpenAL. Gpt-4 technical report, 2023. URL utt.tssss///rrrvvorrr/bbb////// 08774
OpenAl. GPT-4o System Card. arXiv preprint ariiv:22122:222 2024a. URL https://aarxi.. org/abs/2410.211
OpenAL. GPT-4o mini: advancing cost-efficient intelligence. https://openaii.com/inee
oostteffiiieent-nnelllggeeeee 2024b.
OpenAL. Openai ol system card, 2024c. URL.ttp:::/axxv..or//bbbb
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: /tte:://oopoee http///ppppeeeeeeeeeeeeeee wwwprrewwwwweeeffruun//.e.e nwt.:/rmffffridd KS8mIvetg2.
Simon Ott, Adriano Barbosa-Silva. Kathrin Blagec, Jan Brauner, and Matthias Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13(1);6793, 2022. doi:
URL https:
//www. nawure.tur...m wwt./o/oaaaataico /4tpee/eecmmmmtrteee///44666-- 34591---
Long Phan, Alice Gatti, Ziwen Han, et al. Humanity's 's last exam. CoRR, abs/2501.14249, 2025. URL https: http:::/dooo.rrro// 48550/arXiv..... 2501. 14249.
Sundar Pichai, Demis Hassabis, Koray Kavukcuoglu, and Google DeepMind. In- troducing gemini 2.0: our new model for the agentic era. Blog post wttpa/////gooollledeemnee
The Keyword, 2024. URL https://blog saaplle--emmii-----pa---------ee
Rashin Rahnamoun and Mehrnoush Shamsfard. Multi-layered evaluation using a fusion of metrics and LLMs judges in open-domain question answering, In Proceedings of the 31st International Conference on Computational Linguistics, pp. 6088-6104, Abu Dhabi, UAE, 2025. Association for Computational Linguistics. URL https://aclaadie Besing-main.408..
202/2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google- proof q&a benchmark. arXiv preprint arXiv:2311.12002 2023. URL https://arxiv... abs/2311. 12022.
15

---

Published as a conference paper at COLM 2025
Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. Nip evaluation in trouble: On the need to measure Ilm data contami- nation for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10776-10787, Singapore, 2023. Association for Computational Linguis- tics. doi: 10. 10.18653/10 vv1 htv1//22..fmmmm htt///22...iiingeeemlllpp.p Sindings-emnlp. 722/.
URL https::/aclaatthollll .ogpl/ooooooooooooooooooe org/2023
Rylan Schaeffer. Pretraining on the test set is all you need, 2023. URL https://doi.nnn 48550/arXiv. 2309. 230..08632........ arXiv preprint arXiv:2309.08633 08662..
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Tweelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forummid=z2ar3MM
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023, URL https://openreee www.eeeeeeeeeeooornniiiooooooo net.fforum1dd-uyy15Bv53333
Saeid Asgari Taghanaki, Aliasgahr Khani, and Amir Khasahmadi. MMLU-Pro+: Evaluating Higher-Order Reasoning and Shortcut Learning LLMs. In Proceedings of the 38th Conference on Neural Information Processing Systems, NeurIPS 2024, Safe Generative Al Track, 2024. URL https: / /ttp/vaaivvvvv.////// www.ass/////ooooooooo.o.oo....a /wwpaab///.//.///// 2409 02257.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pp. 3261-3275 2019a. URL https::/ppooceediiii newriis.ee.ppp...... www.iee..ucppppeeeeeeeeeeeeeeeee pacc/appr/22/ htt///221//hhhhs html.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, net/forum -itrummmi-dr4425557.......
USA, May 6-9, 2019, 2019b. URL
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL https://openrrviiee http://ppprrviie.....o.. wwwpenrwewwwweweeeeeeeeeeeeeeeee ior.oornnudd1pp 111MMMW..
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, Decem- ber 10 15, 2024, 2024. URL http://papers. nips cctpaper.files/ppeeeeeeeeeeeeee /illes//aper/222//haaee ad236edc564f3e31 3ee lb2feafb99a24 Datasttss and Trenchaarks.Trakk html.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824-24837. Cur- ran Associates, Inc., 2022. URL https::/proceedd// ngs. n iisuipp.... pss cc/ /appr Files/paper/ 2022/file/9d5600 0tt.///6606111224444444551161110 orp244/ffffft00r711aaaaaaaaprrpr wwwer-aapeeeeeeeeeeee .ddff
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination--rre LLM benchmark. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
16

---

Published as a conference paper at COLM 2025
Yating Wu, Ritika Mangla, Greg Durrett, and Junyi Jessy Li. QUDeval:
The evaluation of questions under discussion discourse parsing. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, Pp. 5344-5363, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v/ /v1 22023.emnlpmmaiii 325. URL https://acllntthollgg org/ 2023. emnlp-main. 3225/
Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi, Benchmark data contamina- tion large language models: A survey, 2024. URL hwtps: org/ aas//b///// /206. 04244.
Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. Kieval: A knowledge-groundeed interactive evaluation framework for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5967-5985. Association for Computational Linguistics, 2024. doi: 111865//v12222.acl-ooogg wwrrxiv..rg/bbbs. .ow.org///aa. 2402.. 15043.
URL https:
//arxiv.
Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Weiwen Xu, Deli Zhao, and Lidong Bing. Auto-arena: Automating 1lm evaluations with agent peer battles and committee discussions, 2024. URL https: https://arxiv... wtt.aaiwww...///////......5 20267. arXiv:2405.20267.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yong- hao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decem- ber 10 16, 2023, 2023. URL http://papert /ttp://aapppepeeaaaaeeeeee/essee nips cc/paper.p.pp..eceeeceeeeeee..ee files/paperr / 2223//aahh (FFF18122773388888822266666 -1s---Aatattttattt-ttasssee and Benchmarks..hhh.ee html.
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL httph::t////oe h/tp:://///eeeiiww net/fforum2idcgjfooL555K
17

---

Published as a conference paper at COLM 2025
A
Additional Debate Evaluation: Questioning-Only Heatmap
DeepSeek V3
0.54
1.0
Claude 3.5 Sonnet - 0.44
0.64
0.70
0.64
0.66
0.76
0.78
0.76
0.84
0.88
Mistral Large 0.32
0.40
0.50
0.54
0.44
0.62
0.72
0.80
0.78
0.86
0.8
Claude 3.5 Haiku
0.68
GPT-4o
0.26
0.36
0.42
0.50
0.52
0.54
0.64
0.70
0.80
0.78
0.6
Model
GPT-4o mini
0.22
0.32
0.32
0.52
0.38
0.64
0.72
0.76
0.78
0.74
Mixtral 8x
0.46
0.66
0.4
Mixtral 8x22B
0.12
0.18
0.18
0.22
0.32
0.34
0.44
0.48
0.66
0.58
0.2
Llama 3.1 8B
0.12
0.24
0.14
0.14
0.26 0.24
0.54 0.52
0.68
0.58
GPT-3.5-turbo
0.04
0.08
0.20
0.34
0.32
0.0
Mistral 7B
0.04
0.10
0.10
0.08
0.12
0.16
0.30
0.20
0.3.
0.42
DeepSeek V3
Claude 3.5 Sonnet
Mistral Large
Claude 3.5 Haiku
GPT-40
GPT-4o mini
Mixtral 8x
Mixtral 8x22B
Llama 3.1 8B
Mistral 7B
GPT-3.5-turia
Opponent
Figure Head-to-head win rate heatmap (Questioning-Only Each entry reflects win rates when models challenge the official answer. While noisier than defending-only results, the structure is broadly consistent.
B Judge Model Variation Results
Below are detailed win-rate tables showing pairwise comparisons between models across different judge models.
Table 4: Judge Failure: Mistral 7B. The model failed adhere output format instructions, leading to default judgments and the indiscriminate win rates shown.
Opponent- +
GPT-4o Mini
Mistral Large
Mistral 8x7B
Mistral 7B
GPT-4o Mini
0.50
0.50
0.51
Mistral Large
0.50
0.50
0.50
Mistral 8x7B
0.50
0.50
0.50
Mistral 7B
0.49
0.50
0.50
18

---

Published as a conference paper at COLM 2025
0.88
0.86
Mistral Large-
0.56
0.71
0.88
Mistral Large
0.65
0.82
0.86
0.73
0.72
GPT-4o Mini
0.44
0.68
0.79
GPT-4o Mini
0.35
0.73
0.84
Model
0.58
Model
0.57
Mistral 8B
0:29
0.32
0.65
0.42
Mistral 8x7B
0.18
0.27
0.69
0.43
0.27
0.28
Mistral
Mistral
0.12
0.14
Mistral Large
GPT-4o Mini
Mistral 8x78
Mistral 7B
Mistral Large
GPT-4o Mini
Mistral 8x7B
Mistra 7B
Opponent
Opponent
(a) Judge Model: GPT-4o
(b) Judge Model: GPT-4o Mini
0.82
0.78
Mistral Large
0.55
0.72
0.82
Mistral
0.59
0.66
0.77
0.69
0.67
GPT-4o Mini
0.45
0.64
0.67
GPT-4o
0
0.71
0.78
Model
0.56
Model
0.56
Mistral 8x7B
0.28
0.36
0.58
0.44
Mistral 8x7B
0.34
0.29
0.64
0.44
0.31
0.33
Mistral
0.18
0.33
0.42
Mistral 7B
0.23
0.36
0.18
0.22
Mistral Large
GPT-4o Mini
Mistral 8x7B
Mistral 7B
Mistral Large
GPT-4o Mini
Mistral 78
Mistral
Opponent
Opponent
(c) Judge Model: Mistral Large
(d) Judge Model: Mistral 8x7B
0.58
0.64
Mistral Large-
0.52
0.58
0.57
Mistral Large
0.51
0.54
0.64
0.55
0.58
GPT-4o Mini
GPT-4o
Model
0.52
Model
0.53
Mistral 8x7B
0.42
0.44
0.58
0.48
Mistral 8x7B
0.46
0.44
0.57
0.47
0.45
0.42
Mistral 7B
0.43
0.42
0.42
Mistral 7B
0.36
0.38
0.43
0.42
0.36
Mistral Large
GPT-4o Mini
Mistral 8x7B
Mistral 78
Mistral Large
GPT-4o Mini
Mistral 8x7B
Mistral
Opponent
Opponent
(e) Judge Model: Llama 3.1 8B
(f) Judge Model: Llama 3.1 8B Finetuned
Figure 7: Pairwise win rate heatmaps for each judge model. Each subfigure (a)-(f) shows the win rates between models, as judged by the corresponding model.
19

---

Published as a conference paper at COLM 2025
C Confirmatory Evaluation on the GPQA Dataset
To confirm the generalizability of our framework beyond the MMLU-Pro dataset, we conducted full evaluation on the GPQA main set (Rein al., 2023), which consists of 448 graduate-level questions. This experiment involved round-robin tournament with five open-source models: Llama Scout (Meta AI, 2025), Liama 3.1 8B, Mixtral 8x Mistral 7B, and Phi-4 Multimodal (Microsoft, 2025).
The debates followed the same double round-robin protocol as the main experiment, with Llama-4 Scout serving as the judge model. The aggregated pairwise win rates are presented in Table 5. The results demonstrate clear and consistent performance hierarchy with no intransitive loops (e.g., A > B C > A), reinforcing the robustness and reliability of our debate-driven evaluation method on a different, challenging benchmark.
Table 5: Pairwise win rates the GPQA main test set. Each cell (i,) shows the win rate of model against model j. The judge model was Llama Scout.
Opponent
Llama 4 Scout
Llama 3.1 8B
Mixtral 8x7B
Mistral 7B
Phi-4 MM
Llama 4 Scout
0.78
0.83
0.80
0.87
Llama 3,1 8B
0.22
0.51
0.61
0.83
Mixtral 8x7B
0.17
0.49
0.56
0.82
Mistral 7B
0.20
0.39
0.44
0.71
Phi-4 Multimodal
0.13
0.17
0.18
0.29
D Detailed TrueSkill and Bradley-Terry Scores
This appendix provides the full TrueSkill and Bradley-Terry (BT) rating tables illustrating how model scores change before and after introducing fine-tuned Llama 3.1 8B model into the reference set of eleven models:
DeepSeek V3
Claude 3.5 Sonnet
Mistral Large
Claude 3.5 Haiku
GPT-4o
GPT-4o mini
Mixtral 8x7B
Mixtral 8x22B
Llama 3.1
GPT-3.5-turbo
Mistral 7B
All references below use GPT-40 as the judge model during debates.
From Tables 6 and 7, TrueSkill scores exhibit minimal shifts among reference models, whereas Bradley-Terry adjustments can be substantially larger. This highlights the greater stability of TrueSkill for incremental or partial-toumament settings.
20

---

Published as a conference paper at COLM 2025
Table 6: TrueSkill scores comparing reference-only results to updated ratings after adding the fine-tuned Llama 3.1 8B.
Model
Score (ref models)
Score (with new model)
DeepSeek V3
492.00
489,40
Claude 3.5 Sonnet
468.10
468.20
Mistral Large
447.80
447.90
Claude 3.5 Haiku
436.20
436.30
GPT-4o
414.50
414.90
GPT-4o mini
411.60
412.00
Mixtral 22B
341.30
341.40
Mixtral 8x
346.50
346.60
Llama 3.1 8B
326.30
326.50
GPT-3.5-turbo
289.70
290.10
Mistral 7B
281.70
281.80
Llama 3.1 8B Finetuned
0.00
<315.20>
Table 7: Bradley-Terry scores comparing reference-only results to updated ratings after adding the fine-tuned Llama 3.1 8B.
Model
Score (ref models)
Score (with new model)
DeepSeek V3
1963.00
1981.00
Claude 3.5 Sonnet
1881.00
1904.00
Mistral Large
1731.00
1755.00
Claude 3.5 Haiku
1705.00
1728.00
GPT-4o
1637.00
1660.00
GPT-4o mini
1608.00
1632.00
Mixtral 8x 22B
1308.00
1331.00
Mixtral 8x
1325.00
1349.00
Llama 3.1 8B
1243.00
1271.00
GPT--35-ttrbo
1083.00
1107.00
Mistral 7B
1015.00
1039.00
Llama 3.1 8B Finetuned
0.00
<1244.00>
E
Debate Prompts
This appendix provides the exact prompts used structure debates between models in our evaluation framework.
E.1
Positive (Pro) Side Prompt
The model assigned to defend the correct answer receives the following prompt:
You are the POSITIVE side. Argue in favor the correct answer.
Consider and address any previous counter-arguments: Question: (question_text)
Your answer: (correct_answer
Debate history: (lebate_transcript)
E.2 Negative (Con) Side Prompt
The model assigned to challenge the correct answer receives this prompt:
21

---

Published as a conference paper at COLM 2025
You the NEGATIVE side. The system has rejected the answer incorrect. Provide an alternative answer and argue against the original answer, considering previous arguments: Question: (question_text)
Claimed correct answer: (correct_answer) Debate history: (debate_transcriit)
E.3 Judge Prompt
The judge model, which evaluates debates without knowing the official answer, receives this prompt:
You are an impartial judge evaluating debate about the answer to question. Based solely on the quality of arguments presented by both sides, determine which side has made the stronger case.
The debate transcript is provided below. You do not need to determine the correct answer yourself only which debater presented more convincing arguments.
[debate_transcrippt)
After careful consideration of all arguments, which side made the stronger case overall? Respond with only "positive", "negative", or "continue" (if more debate rounds are needed for clear determination).
22


Reviews:
[{"review_id": "dfEanbWOmM", "reviewer": "Reviewer_wZxD", "summary": "This paper proposes a debate-like evaluation method that addresses benchmark saturation and data contamination. The paper is clearly motivated, presents a well-structured and reproducible pipeline for converting existing benchmarks into a debate-style format, and concludes with sound reasoning and empirical evidence on the cost-effectiveness and contamination robustness of the proposed approach. Although debate-based evaluation has been explored before, the structured and standardized format proposed here is novel and shows future potential.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**S1: The presentation of the paper is good.**  \nThe paper begins by clearly presenting benchmark saturation and data contamination as its motivation. In the conclusion, it clearly summarizes how the proposed method addresses these two problems. The writing style is coherent and easy to follow.\n\n**S2: The evaluation appears to be OK.**\n\n- I especially enjoyed the visualization in Figure 7 (although Figures 5 and 6 should be renamed as Figure 7a and 7b).\n- The results in Section 5.4 provide strong evidence for the paper's claim: improvement on the standard benchmark by fine-tuned models does not translate to improvement on the proposed debate benchmark.", "reasons_to_reject": "**W1: Correlation with existing benchmarks/human judgments.**  \nMy primary criticism is that, although the paper shows that improvement on standard benchmarks does not translate to the proposed evaluation, it does not clearly demonstrate whether the evaluation correlates with existing benchmarks or human judgments. In other words, while the paper shows that the proposed evaluation can disentangle data contamination, its reliability remains unclear due to the lack of grounding in established methods. Can the authors explore other works on automatic model evaluation and report correlation? (Please correct me if I missed this in the paper).\\\nI think a good or moderate correlation with existing metrics would be sufficient, it would show that the proposed evaluation shares common ground with established metrics while also highlighting where it diverges.\n\n**W2: The soundness of evaluation can still be improved in the following aspects:**\n\n- Many MMLU-Pro questions have a specific correct answer (e.g., math problems). Forcing a model to argue for an incorrect answer may place it at a disadvantage, as it could be more difficult to find supporting evidence for an incorrect position. Although role-switching ensures both sides are disadvantaged equally, alternative answers are not fixed, and defending different incorrect answers may vary in difficulty. This undermines the fairness assumed by role-switching.\n- Figure 3 is somewhat vague regarding model pairing. Are models first paired in a round-robin format with a random question assigned to each match, or is a complete round-robin tournament played for each question?\n- In some evaluations, the same LLM is both contestant and judge. For example, Table 1 shows GPT-4o acting as both judge and contestant. Could this be unfair, as the model may favor its own chain of reasoning?\n- The paper does not provide judging rubrics to the judge LLM. The concept of \"making the better case\" is entirely subject to the judge model's interpretation. This could affect the fairness of the benchmark if different judges are used. It may be helpful to standardize the judge (or use an ensemble) to mitigate such bias.\n- While the paper thoroughly studies the impact of contamination on contestant models, it does not investigate contamination effects on judge models. Although the paper tries to make the judge models answer-blind by not revealing answers, they may still have encountered the questions and answers during training. It would be worth testing whether such prior exposure affects the fairness of judgments.", "questions_to_authors": "**Q1:** Can your method work with tasks other than QA?\n\n**Q2:** The paper does not mention how the 50 MMLU-Pro questions were sampled. Since MMLU-Pro spans a wide range of domains, domain selection could significantly affect debate outcomes--especially if some domains favor factual recall while others favor reasoning. Could the authors clarify the sampling methodology and the domain distribution of the 50 selected questions?\n\n**Recommended Citations**\n\nThere are recent methods from the computational linguistics (CL) community that are similar in spirit. I recommend the authors review and cite the following:\n\n- [QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing](https://aclanthology.org/2023.emnlp-main.325) from  EMNLP 2023\n- [Discursive Socratic Questioning: Evaluating the Faithfulness of Language Models' Understanding of Discourse Relations](https://aclanthology.org/2024.acl-long.341/) from ACL 2024\n\nBoth works use multiple question answering (similar to your debate approach) to evaluate how well models understand context.\n\nUpdate on June 16th: Increased my score slightly after author response.", "extra_scores": {}}, {"review_id": "TcdmGmnfRC", "reviewer": "Reviewer_SpHc", "summary": "This paper introduces a debate-driven approach to evaluate the reasoning capabilities of large language models (LLMs). Current evaluation benchmarks face two major challenges: the increasing cost of developing more complex benchmarks as models advance, and the issue of data contamination, where benchmark questions may overlap with training data. These limitations highlight the need for a new evaluation framework. In response, the authors propose a novel method that assesses the validity of answers to existing multiple-choice question-answering (MCQA) samples through structured debate. The effectiveness of this framework is demonstrated through experimental validation.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The motivation is well understood and important for the LLM community.\n- The experimental results (Figure 7) are consistent and appear to demonstrate the effectiveness of the evaluation method.\n- Fine-tuning impact assessment is an interesting finding.", "reasons_to_reject": "- Not only the cost of constructing the benchmark set, but also the cost of evaluation is an important issue, and we would like to see an explicit mention of how much computing cost is required.\n\n- (L.232) Since robustness is a very important issue for evaluation frameworks, it would be helpful to have a more detailed explanation to determine whether Partial Tournament Validity has been sufficiently evaluated.\nThe computing cost does not seem to be significantly different from other experiments, so it would be useful to be able to evaluate under different conditions if necessary.\n\n- Although it is understood that the purpose is not to create a model leaderboard, still, an explanation is desired why these 11 models were selected.", "questions_to_authors": "(L197) Since exact match requires instruction following ability, differences between models are expected to emerge. Is this a valid measurement method for traditional QA accuracy?", "extra_scores": {}}, {"review_id": "KZbIZaNotZ", "reviewer": "Reviewer_fs1s", "summary": "The paper proposed a debate-driven evaluation paradigm that transforms an existing QA dataset into structured adversarial debates, where one model is given the official answer to defend and another constructs and defends an alternative answer. Empirical results validated the robustness of the method and its effectiveness against data contamination.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed a debate-driven evaluation paradigm that transforms an existing QA dataset into structured adversarial debates. Empirical results validated the robustness of the method and its effectiveness against data contamination.", "reasons_to_reject": "1. While the authors claim their approach deals with any existing QA dataset, the experiments were done only for a dataset, MMLU-Pro. Further, the size of the dataset for evaluation, 50 randomly sampled questions, is rather small. I wonder it can yield a reliable result.\nMost importantly, the authors should investigate and discuss how large dataset should be involved for the debate to yield reliable judgments. \n2. While the authors claim retaining only questions and correct answers, and removing incorrect alternatives are better, it was not verified with any evidence. So it is better to show whether using the original incorrect answers in the debate is really harmful in the experiments. \n3. It was not clearly discussed why the rounds of the debate should be in the range of 2-5.\n4. While it costs less than constructing a new dataset, the proposed approach seems rather costly as an evaluation method. Thus, the authors should discuss its computational cost and compare it with other related approaches.", "questions_to_authors": "1. Figures 1, 5-7  were not referred to in the body of the paper. Tables 9 and 10 in Sec. 5.6 do not exist in the body but are in the Appendix. \n2. For the paragraph, Partial Tournament Validity, no results (evidence) seem shown.", "extra_scores": {}}, {"review_id": "4oAoLUdwHY", "reviewer": "Reviewer_5Sib", "summary": "This paper presents an innovative evaluation method to address potential benchmark contamination issues. Instead of relying on standard benchmarks, it introduces multi-round arguments between models to assess whether a model truly understands the content. I find this approach intuitive, as allowing a model to elaborate on facts makes it easier to evaluate its mastery. My detailed comments are below.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I believe this evaluation setup is straightforward and effective. The empirical results show that this method successfully differentiates genuine reasoning from memorization, as evidenced by the fine-tuned model's reduced debate performance despite improved QA accuracy. It also proves robust across various judge models.", "reasons_to_reject": "The primary concern with this method is that *evaluating data points is too expensive*. Assessing a single data point requires agent-style back-and-forth and using an LLM as a judge. Could you provide the time required to evaluate 1,000 data points with a specified length? I’m open to inference frameworks like vLLM or SGLang, but since this involves multi-turn debates, I suspect the speed gains may be limited. \n\nAdditionally, based on Table 1, the difference between debatable and simple evaluations isn’t significant. I think most would prefer evaluating across diverse benchmarks to demonstrate their model’s general capabilities rather than focusing on debatable evaluations for a single dataset. I’d appreciate a comparison of time and cost, especially since you’re using an LLM-as-a-judge approach.\n\nThere are also some reference I thought may be beneficial to include:\n- [1] Proving test set contamination in black-box language models. \n- [2] Investigating Data Contamination in Modern Benchmarks for Large Language Models. \n- [3] Detecting Pretraining Data from Large Language Models.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a debate-driven evaluation paradigm that can be applied to standard QA benchmarks. The results show that while fine-tuned models can show better QA accuracy, it is likely due to data contamination, because the same models perform worse in the structured debate setting. The proposed evaluation produces a robust ranking of models and is a promising way to address benchmark saturation.

**Strengths**

Reviewers point out that:

* The paper is well-written (wZxD) and clearly and compellingly motivated (wZxD, SpHc)
* The proposed evaluation method is effective and robust (5Sib)
* The claim is substantially supported by the experimental results (all reviewers)
* The finding about fine-tuning $\Rightarrow$ better standard benchmark performance but worse debate performance is especially interesting and compelling (wZxD, SpHc, 5Sib) 

**Weaknesses**

Reviewers' biggest concern is the viability of the proposed method: debate requires much more LLM use than standard evaluation, and thus is substantially more costly and time-consuming (SpHc, 5Sib, fs1s). The rebuttal argues that in the long run creating hard-enough benchmarks will grow more expensive, while compute gets cheaper over time, alleviating this concern. I am mostly convinced by the rebuttal (and the reviewers raising their scores suggests that they are too), although I agree that it is too early to judge the practical impact of the approach.

Other criticisms had to do with the validity and reliability of the method:

* It was not demonstrated that success in debates aligns with human judgments about the performance/soundness of the reasoning (wZxD). I understand the rebuttal's argument about human expert evaluation being costly to conduct, but it would have further strengthened the paper.
* It was not originally clear if the findings generalize to other datasets (fs1s), but the additional experiments in the rebuttal addressed this.

Overall, echoing Reviewer SpHc: this work is a promising first step in an important research direction. Even if we can't guarantee its viability and future impact, I consider it a sufficient contribution and recommend acceptance -- given that the new results/discussion from the rebuttal are added to the paper.
Decision: Accept