Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learning
Authors: Qihao Liu
Luoxin Ye
Wufei Ma
Yu-Cheng Chou
Alan Yuille
Venue: ICLR 2026 Poster (2026)
Keywords: Large language models
Reinforcement learning
Adversarial training
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) with explicit reasoning capabilities excel at mathematical reasoning yet still commit process errors, such as incorrect calculations, brittle logic, and superficially plausible but invalid steps. In this paper, we introduce Generative Adversarial Reasoner, an on-policy joint training framework designed to enhance reasoning by co-evolving an LLM reasoner and an LLM-based discriminator through adversarial reinforcement learning. A compute-efficient review schedule partitions each reasoning chain into logically complete slices of comparable length, and the discriminator evaluates each slice’s soundness with concise, structured justifications. Learning couples complementary signals: the LLM reasoner is rewarded for logically consistent steps that yield correct answers, while the discriminator earns rewards for correctly detecting errors or distinguishing traces in the reasoning process. This produces dense, well-calibrated, on-policy step-level rewards that supplement sparse exact-match signals, improving credit assignment, increasing sample efficiency, and enhancing overall reasoning quality of LLMs. Across various mathematical benchmarks, the method delivers consistent gains over strong baselines with standard RL post-training. Specifically, on AIME24, we improve DeepSeek-R1-Distill-Qwen-7B from 54.0 to 61.3 (+7.3) and DeepSeek-R1-Distill-Llama-8B from 43.7 to 53.7 (+10.0). The modular discriminator also enables flexible reward shaping for objectives such as teacher distillation, preference alignment, and mathematical proof-based reasoning.

Published as a conference paper at ICLR 2026
GENERATIVE ADVERSARIAL REASONER: ENHANCING LLM REASONING WITH ADVERSARIAL REINFORCE- MENT LEARNING
Qihao Liu, Luoxin Ye, Wufei Ma, Yu-Cheng Chou, Alan Yuille Johns Hopkins University qliu45e juu. edu
ABSTRACT
Large language models (LLMs) with explicit reasoning capabilities excel math- ematical reasoning yet still commit process errors, such as incorrect calculations, brittle logic, and superficially plausible but invalid steps. In this paper, we intro- duce Generative Adversarial Reasoner, an on-policy joint training framework de- signed enhance reasoning by co-evolving an LLM reasoner and an LLM-based discriminator through adversarial reinforcement learning. A compute-efficient re- view schedule partitions each reasoning chain into logically complete slices of comparable length, and the discriminator evaluates each slice's ' soundness with concise, structured justifications. Learning couples complementary signals: the LLM reasoner is rewarded for logically consistent steps that yield correct answers, while the discriminator earns rewards for correctly detecting errors or distinguish- ing traces in the reasoning process. This produces dense, well-calibrated, on- policy step-level rewards that supplement sparse exact-match signals, improving credit assignment, increasing sample efficiency, and enhancing overall reasoning quality of LLMs. Across various mathematical benchmarks, the method delivers consistent gains over strong baselines with standard RL poptstt post-postttt post-training. Specifically, on AIME24, we improve and DeepSeek-RI DeepSeek-R1-Distill- -i-----i-iitil--aaaaa--- criminator also enables flexible reward shaping for objectives such as teacher dis- tillation, preference alignment, and mathematical proof-based reasoning.
from 54.0 to 61.3 (+7.3) from 43.7 to 53.7 (+10.0). The modular dis-
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable mathematical reasoning abilities, often achieving expert-level performance across diverse benchmarks (Achiam et al., 2023 Dubey eet al.] [2024] Shao al. 2024: [DepSSek--A]] 2025). However, despite extensive training large- scale datasets with sophisticated paradigms, these models still suffer from errors in reasoning, such incorrect calculations, flawed logic, superficially plausible but invalid arguments, and repetitive or incoherent reasoning steps. To tackle these challenges, researchers have explored approaches such as model debate collaboration, in which models debate against each other (Du
2023
Liang et al.) 2023) or with themselves (Kuba et al., 2025) Liu et al., (025a). and Process Reward Models (Lightman al., 2023) Wang et al. (203), which aim to identify and mitigate process errors throughout the reasoning process. These methods provide finer- mer-ggaaied supervision and contribute to more robust and reliable LLM performance.
Among existing approaches, Process Reward Models (PRMs) have shown strong results on complex reasoning tasks, largely because they leverage detailed step-level annotations. However, PRMs face challenges related annotation costs and data quality (Lightman al.) (023). fine-grained labels are expensive and prone to subjective error, and are sometimes susceptible to over- or under-reward issues (Wen al., 2024 Lv al. 2025). Alternatively, prompt- based methods employ LLMs as critics for stepwise judgments at lower cost (Zhang et al. 2024 Gao et al.) [2024] Xia et al. 2025 However, their judgments sometimes be noisy, inconsistent, and less discriminative.
To bridge this gap, we retain a stepwise critic (referred to as the discriminator) but enable it to co- evolve with the LLMM reasoner through joint training, generating effective step-level signals with

---

Published as a conference paper at ICLR 2026
DS R1 LI--Dssiill-- ---ssill--wee-77 Qwen-7B
DS R1 RI-Distiil- Qwen-BB + GAR
+0.5%
DS NU-Distill-Lamaal
+1.8%
+2.4%
942
+6.5%
DS-RR- -isstill-Llam--83 + GAR
+2.88
DAL
+13.5%
+4.4%
+3.00%
54.0
+22.9%
6..66%
19.5%
03.7
+33..3%
121.1%
LiveMathBenca-Hard AIME25
OlympiadBench
AIME24
GSM8K
AMC23
MATH500
Pass@1 on Math Benchmarks
Figure 1: Pass@1 accuracy on seven mathematical reasoning benchmarks. Our Generative Adversarial Reasoner (GAR) consistently improves over strong baselines across both Seesssee-RI--iiill---
and Deepseck-RI Hi--Ditiill-Llama 8B. GAR achieves gains of +22.9% on AIME24 and +19.5% on AIME25 for the Llama backbone, well as +35.3% LiveMathBench- Hard for Qwen. These results demonstrate the robust- ness and generality GAR enhancing reasoning performance across diverse mathematical tasks (Tab. I
lower annotation costs and increased robustness to label noise and reward mis-specification. Con- cretely, we optimize the LLM reasoner and an LLM-based discriminator together: the discriminator judges the logical soundness of each intermediate reasoning step and explains its judgment, while the reasoner learns to produce steps the discriminator consistently endorses for valid logic. This co-adaptation dynamically aligns the reward signal with the model's evolving capabilities, reduces reliance on costly fine-grained annotations, and mitigates miscalibration by continually recalibrat- ing the discriminator to the reasoner's step distribution. As result, we obtain better-calibrated, on-policy stepwise judgments, thereby enhancing the reasoning capabilities (Fig. I
However, jointly training the LLM reasoner and its stepwise discriminator introduces several chal- lenges. First, stepwise analysis over long, intricate reasoning chains during training increases com- putational cost and system complexity. Second, ensuring the discriminator's 's judgments are rigorous and interpretable requires explicit scoring rubrics and structured rationales. Third, co-evolution can invite reward hacking: the discriminator may drift toward overly positive judgments, while the LLM may learn produce reasoning steps that look plausible but semantically shallow.
To address the challenges, we propose Generative Adversarial Reasoner (GAR), which incorpo- rates a compute-efficient review schedule and adversarial co-training framework. Specifically, partition each reasoning chain into logically complete slices of comparable length. The discrimina- tor evaluates each slice for logical soundness and generates a concise, structured rationale, providing localized and verifiable feedback on specific reasoning errors. For learning, we jointly update the an adversarial reinforcement learning scheme inspired by
LLM reasoner and the discriminator GANs (Goodfellow et al. 2014)). The reasoner is rewarded for logically consistent steps that lead to correct final answers. The discriminator receives two complementary rewards: an alignment reward for correctly detecting errors in the reasoning, and a discriminative reward for distinguishing the reasoner's is trajectories from reference rationales. Together, these signals improve sample efficiency, and deliver calibrated stepwise supervision under controlled compute budget.
Compared with previous methods, our model offers three key advantages: (i) slice-level evaluation that simplifies the discriminator's 's task and yields localized, interpretable justifications; (ii) on on-policy joint updates that keep rewards aligned with the model's current behavior and support continued improvement, with the diseriminator evolving detect subtler errors as the L.MM grows stronger; and (iii) dense step-level rewards that augment sparse exact match grading with continuous signals based on the fraction of correct steps, improving credit assignment and sample efficiency.
Our experiments show that, even compared to strong baselines (DeepSeek-Al) 2025), we achieve further improvements across various mathematical reasoning benchmarks, delivering significant gains over standard RL post-training approaches. For instance, on AIME24, we boost DeepSeek- -iiit-RR--DitiIILL-
RI-Distill-Qwen- from 54.0 to 61.3 (+7.3) and (+10.0). with comparable training time. These results highlight the consistent and substantial en- hancements in LLM reasoning with our GAR, achieved within comparable compute budget.
from 43.7 to 53.7
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Reinforcement Learning with Process Feedback. Process-level supervision improves reasoning
by evaluating intermediate steps rather than outcomes (Lightman al. 2023 Uesato et al.) 2022]] Ouyang et al.) 2022). Beyond costly human PRMs, automatic judges lower labeling cost but in- troduce noise; methods include Monte-Cario from final answers,
and formulating reward modeling as next-token prediction to improve stability (Zhang et al. 2025 2024 Gao et al, 2024). Math verifiers complement PRMs and favor concise, struc- tured rationales (Cobbe et al. 2021). RL methods such as DeepSeek-RI (DeepSeek-A]] (025) and
2024 Xia et al. [025 Xi et al.
FastCuRL. (Song al. 2025) encourage behaviors like self-reflection and verification but still rely
on static rewards or fixed crities that drift with the policy, Our approach jointly trains the reasoner and a discriminator under an on-policy scheme, yielding dense slice-level rewards with a compute- efficient review schedule and alignment regularization to curb reward hacking, addressing PRM mis-specification and noisy signals without sacrificing efficiency (Wen et al. 2024 [Lv al. 2025).
Self-Play, Multi-Agent, and Hame-Theerrtic Training, Self-play is a unifying mechanism: ad- versarial dynamics induce curricula, enabling iterative improvement from self-generated data with evolving critics (Chen et al., 2024 Yuan et al. 2024 Chen et al. 2025), J while robustness-oriented variants such as SPAG instantiate these dynamics via adversarial word games (Cheng et al.. 2024).
Extending to the zero-data regime, recent work replaces external traces with task-space exploration and language self-play (Zhao et al.] 2025; Kuba et al.)
2025) A complementary line recasts the adversary as agents whose discourse provides training signals, with debate-style setups improving factuality and robustness (Du et al.) 2023 Liang et al.) (023) To scale these dynamics, fully online
multi-agent RL systems implement distributed actor-learner frameworks with role-conditioned ad- vantages to sustain open-ended curricula transferable to math and science, while ParalleI-RI specif-
ically targets parallel thinking (Liu et al. 2025b Wan et al. 2025 Liu et al., 2025a Zheng al.] 2025). Distinct from these, we embed ad versarial dynamics inside training by co-evolving the critic with the policy to deliver fine- aine-graiddd, on-policy credit assignment, consistent with recent insights on exploration and entropy for reasoning RL. (Cui et. 2025
et al. 2025 Wang et al., 2025)
Thinking Scaffolds and Curriculum Learning. Reasoning can be scaffolded via distilled tem- plates, uncertainty-ave awwaee planning, and staged curricula to stabilize early steps and diversify so- lution paths (Yang et al.) 20044; [Hu et al., 22024; Zheng et al., 2025). Building on this, game- theoretic formulations treat reasoning as an interactive, multi-round protocol; Game-of-Thought shows that such interaction improves robustness and information secking (Kempinski et al., (025)). Complementing these macro-level curricula, verifier-guide scaffolds such as Math-Shephee Shepherd pro- vide lightweight stepwise signals without human labels, reinforcing intermediate decisions at low supervision costs (Wang et al., 2023). Our method internalizes game theory into discriminator and couples it with compute-efficient slice-level evaluation, providing dense, calibrated, on-policy rewards that improve mathematical reasoning and code generation within unified framework.
3 GAR: GENERATIVE ADVERSARIAN REASONER
We propose a modular formulation for GAR consisting of two components: Reasoner, denoted My. which general-purpose LLM that generates reasoning processes and final answers based on user input; and Discriminator, denoted Ma, which evaluates the outputs of M, slice by slice. The two models are jointly trained via reinforcement learning. We provide detailed descriptions of each model and the training procedure below.
The reasoner is implemented as an LLM that generates intermediate reasoning and final answers. In principle, any model capable of step-by-step reasoning can serve as the reasoner. In this work, to demonstrate the effectiveness of our approach, we instantiate the reasoner with several state-of-the-art, open-source reasoning models (namely, variants of the official DeepSeck-R. -Distill models (DeepSeck-Al Al 2025),, and show that our framework further improves their performance.
Reasoner.
Discriminator. The discriminator evaluates the quality and correctness of the reasoning process, assigning a reward signal to each generated response. In our implementation, it is instantiated as a smaller, pre-trained variant of the reasoner.
However, holistic evaluation of the entire reasoning trace with the discriminator often fails to yield reliable results. We hypothesize that lengthy and complex reasoning chains, which may span thou-
3

---

Published as a conference paper at ICLR 2026
Question: What is the ratio of the area inside the in circle to the of the hexagon?
Reasoner
Discriminator
Reasoning process:
Correctness of Slice
Reference / generated:
Slice 1, Slice 2, Slice 3
The reasoning in Slice i correctly notes that... YES.
Ref. i: Given Slice The reasoning... ...
Final answer is 12233)
Match Reward
Slice Reward
Alignment Reward
Disc. Reward
GAR architecture. GAR is a reinforcement learning framework that jointly trains an LLM reasoner and a slice-level discriminator to improve reasoning accuracy and explainability. Unlike standard RL for LLMs. which computes reward signals mainly based on final answers, GAR leverages discriminator to provide dense, slice-level rewards that evaluate the reasoning process at each intermediate step. More importantly, the discriminator is continuously co-evolved with the LLM reasoner, ensuring the reward signal remains aligned with the model's s current behavior and enabling sustained improvement and refinement of reasoning.
Figure 2:
sands of tokens, are difficult for language models to process and evaluate faithfully, hindering precise localization of reasoning errors. To mitigate this, we partition the generated reasoning into shorter, logically coherent slices, balancing slice length and semantic completeness. Specifically, we seg- ment the reasoning trajectory based delimiters, then merge adjacent segments until clear new semantic beginning is identified or a predefined token length L 320 is reached. For each slice i, the discriminator assigns binary slice reward r E {0, 1} to evaluate its reasoning quality, where rf = 1 indicates that the slice is logically sound. The overall reward is then computed as the mean of all slice-level scores: R²" === - r
This slice-level reward mechanism offers two main advantages. First, it improves reliability: assess- ing the correctness and internal consistency of short slices is substantially easier and more accurate than evaluating a lengthy reasoning chain in its entirety. Second, it provides denser and more in- formative training signal than simple answer matching: rather than a single binary label on the final answer, it scores the reasoning trajectory at the slice level and aggregates these scores into a fine- grained reward. Consequently, even when all final answers are wrong, the model can differentiate and reinforce better reasoning paths during RL training, improving sample efficiency and mitigating the problem of reward sparsity.
Reward Functions. As illustrated Fig. 2 we jointly train the reasoner and the discriminator, For the reasoner, that linearly combines (i) an exact-match term Rm E {0, 1}. which compares the final answer to
Group Relative Policy Optimization (GRPO) (Shao et al. 2024) with a reward
the ground truth, and (īi) the continuous reward R E [0, 1] from the discriminator (the mean of the slice-level scores). The overall reasoner reward is defined as Rrea = RT + R2 where d1 A2 are hyperparameters that weight the two components.
For the discriminator, we maximize two terms: a discriminator reward Rd and an alignment reward Ro. The discriminator reward Rd follows the standard GAN objective (Goodfellow et al. 20144:
Rd Eg-pa [log Ma(x)] + Serrrenloo(( - Ma(2))]
where Ma(x) represents the discriminator's estimated probability that slice is real, and Prefs Pgen denote the distributions of reference reasoning slices and model-ggneratte reasoning slices, respec- tively. The aligmment reward R quantifies the mean agreement between the discriminator's slice- level scores r and the correctness of the final answer produced by the entire reasoning sequence. Under the hypothesis that correct answers are more likely to be supported by logically sound reason- ing, this term encourages consistency between slice-level evaluation and answer-level correctness. The total discriminator reward is given by Rdis = Ag + X4 where λ3, X 2 control the relative contributions. This joint training encourages the discriminator provide calibrated, task- aligned feedback while the reasoner improves both reasoning quality and answer accuracy.
Training Procedure. For each batch of questions, the reasoner generates both answers and detailed reasoning steps, which we segment into multiple slices. We then mix these generated slices with an equal number of reference slices form balanced set and train the discriminator to distinguish
4

---

Published as a conference paper at ICLR 2026
between them. The discriminator scores each slice; these scores provide the slice reward R for the reasoner and contribute to the discriminator's 's own objectives (R' and R*). We jointly update both models with their respective objectives and iterate, yielding improvements in reasoning quality, answer accuracy, and the discriminator's evaluation accuracy.
In addition, it is known that generating the reasoning process enhances LLMs' capabilities in han- dling complex tasks (Wei et al.) 2022). However, this process can be computationally expensive for our task, requires analyzing each slice of the entire reasoning chain, potentially resulting in tens of slices per question. To improve efficiency, we modify the discriminator's workflow to (i) briefly analyze the reasoning chain, (ii) provide the evaluative judgment (slice reward rf), and (iii) provide a concise rationale for its assessment, rather than generating full reasoning chain before scoring. The rationale provided after the judgment is used mainly for explainability. During training, the discriminator is prompted to generate the analysis, rating (slice reward r:), and rationale in a single response, but with the maximum generation length limited to K 128 tokens to curtail the ratio- nale and accelerate training. Notably, results in Sec. 4.3 indicate that restricting the discriminator's s response at 128 tokens does not degrade performance: the final results remain comparable to those with unrestrieted response lengths, while substantially accelerating training.
To further improve the evaluation accuracy, especially after switching the discriminator to an analy- sis ses--scorr-rrtonee
format, we introduce supervised fine-tuning (SFT) stage for the discriminator. In this stage, we use a pre-trained L.MM to generate reasoning steps on a small subset of the training data. These reasoning steps then evaluated by GPT-o44mini, which provides brief analysis, an evaluative judgment, and a concise rationale for each example. To build a balanced SFT dataset and mitigate bias, we randomly sample equal numbers of examples labeled 'yes' and 'no', ensuring both classes are equally represented. We fine-tune the discriminator on this data with early stopping, enabling it to adapt to the new format while preserving the capabilities of the original model.
In summary, training proceeds in two stages: (1) SFT of the discriminator to adapt it to the evaluation format, and (2) joint optimization of the reasoner and discriminator with GRPO. At inference time, only the LLM reasoner is used to produce answers, following the standard inference procedure.
4 EXPERIMENTS
This section presents comprehensive evaluation of the mathematical reasoning capabilities of our model. Sec. 4.1 outlines the experimental setup. Sec. 3.2. proves over state-of-the-art models on mathematical benchmarks. Sec. 4.3 provides detailed analy- ses of GAR, and Sec. 4.4 reports ablation studies. Finally, Sec. 4.3 discusses potential applications and usage of our GAR. In the Appendix, we include additional experimental details (Sec. A and B results on code generation (Sec. C). and ablation studies on slice-segmentation design (Sec.
that GAR significantly im-
4.1 EXPERIMENTAL SETUP
Model Details. Our implementation builds on OpenRI (Hugging Face 2025) and vLLM K won
al. 2023). and we evaluate two backbones. For the Qwen-based setup (Qwen Team we use -isiecii-----iiii-----eeeee
2024),
as the reasoner and DeepSeck-RI-De Distill-Qwenn discriminator. For the Llama-based setup (Dubey et al. 2024). we use DeepSeek- RR1 -istill-LLanmae 8B for both the reasoner and the discriminator, as no smaller Llama reasoning variant is available.
as the
Tasks and Benchmarks. We evaluate GAR on mathematical reasoning tasks across seven public benchmarks: AIME 2024/2025 (MAA a). MATH500 (Hendrycks et al. 2021), GSM8K (Cobbe) al. 202T), AMC23 (MAA b). and LiveMathBench (Liu al., (024). For LiveMathBench, we evaluate its hard splits (v202505 all en and v202412 hard en) and report the average performance. For all evaluations, we adopt Pass accuracy (averaged over 30 samples) as the metric, and fix the decoding parameters to temperature 0.6, top p 0.95, and max tokens = 32K.
Dataset. All experiments are conducted using the OpenR I -Math- dataset from the OpenRI project. To construct instruction-tuning data for the discriminator, we randomly sample 10% of the training set, partition the dataset-prowe -rrosiet-rovided DeepSeek-RI chains of thought into slices, and annotate each slice with binary (yes/no) judgments evaluating its soundness, along with brief rationales, using the GPT-o44mini API. To mitigate class imbalance in these judgments, we downsample the majority class to achieve a 1:1 label ratio.
5

---

Published as a conference paper at ICLR 2026
Table 1: Pass accuracy on mathematical reasoning benchmarks. Reported scores are aver- aged over 30 runs per benchmark to reduce evaluation noise.
Olympiad Bench
LiveMath Bench-Hard
Model
AIME24
AIME25
MATH1500
GSM8K
AMC23
DS-RI-Distil-Queen Bissnll--wen-7BB
54.0
38.0
94.3
90.6
90.3
52.5
18.4
GARR (Ours)
61.3 (+7.3)
44.3 (16.3)
94.8 (+0.5)
92.2 (+1.6)
92.5 (+2.2)
54.8 (+2.3)
(46.)
Dh-RI-Distil--Llma---
43.7
30.3
88.1
82.9
84.5
48.2
18.5
GAR (Ours)
53.7 (+10.0)
36.2 (+5.9)
91.3 (+3.2)
85.2 (+2.3)
90.0 (+5.5)
50.9 (+2.)
22.4 (+3.9)
Training Details. The discriminator is first instruction-tuned with Adam (Loshchilov & Hutter, 2017) for 500 steps (learning rate x 10 100 warm-up steps, weight decay 0.0001) using a global batch size of 128 on H100 GPUs. Then for adversarial reinforcement learning, we jointly optimize the LLM reasoner and discriminator for 400 steps with AdamW (initial learning rate x 10 with a 10% warm-up and cosine learning rate decay to x 10 10-"), using a global batch size of 192 on
H100 GPUs Reward weights set to A1 X2 X and A 0.5.
4.2 ADVANCING STATE-OF-THE-ART MODELS ON MATHEMATICCLL REASONING
Table summarizes the Pass@1 accuracy of our method compared to strong baselines across diverse mathematical reasoning benchmarks. All results are averaged over 30 trials per bench- mark, ensuring reliability by conducting three independent training runs and evaluating with 10 inference seeds per To ensure fair comparison, we re-evaluate all baselines under unified evaluation protocol (Habib et al.] 2023) to climinate scripting variance. Despite starting from strong baselines, our method demonstrates consistent improvements across all benchmarks, espe- cially on challenging datasets such as AIME24, AIME25, and LiveMath
For example, on AIME24 by 7.3 and on LiveMathBench-Hian by 6.5, and achieves an even larger improvement of 10.0 on AIME24 when applied to DeepSeek-ER SeepSeek-R--iisil---- Ri--Dstilll-Llmm--B These results highlight our method's effectiveness in addressing difficult reasoning tasks. In particular, the discriminator model plays crucial role by supervising reasoning traces, thereby enhancing the system's 's ability to solve complex questions with greater accuracy. Beyond challenging datasets, our approach achieves notable improvements across all benchmarks, highlighting its versatility and robustness in enhancing reasoning capabilities.
our approach improves the accuracy of DeepSeek- MerSSeek-R--iisse RI-Distill-
4.3 ANALYSES AND DISCUSSIONS
This section presents a detailed analysis and discussion of the proposed method.
Slice-Level Feedback from Our Diseriminator. Table 2 presents wraiiingtimm examples of the LLM reasoning slices and our discriminator's judgments. GAR provides concise, structured assess- ments of each slice's soundness, yielding localized, checkable feedback. The discriminator is able to affirm correct algebraic and logical transformations, flag subtle arithmetic slips and flawed rea- soning steps, and identify the exact symbols or steps responsible, while keeping rationales brief to control cost. This dense, slice-level supervision localizes errors early, improves credit assignment, and yields better learning than sparse outcome-only rewards.
Discriminator with Improved Efficiency. To obtain accurate and comprehensive analysis, our discriminator evaluates each slice of the full reasoning chain, which can yield tens of slices per question and increase review costs. To im- prove efficiency, we modify the discriminator workflow as shown in Table 2 the discriminator first gives a brief analysis, then a binary 'yes/no' verdict on the slice's soundness, and finally a concise rationale. During joint training of the LLM reasoner and the discriminator, this design yields reliable judgments without requiring a full ra- tionale. Specifically, we cap the discriminator's output at 128 tokens, preserving the verdict and truncating any justification beyond that limit. Table B compares three training settings: standard RL without discriminator, diseriminator with truncation, and discriminator without truncation. Apply- ing the cap preserves accuracy while significantly improving training efficiency. It demonstrates that our analysis score- design delivers dense supervision with minimal overhead.
Table 3: Rollout speed.
AIME24
Training Time (hours)
Standard RL
56.3
16
GAR w/ truncation
61.3
19
GAR w/o truncation
60.8
43

---

Published as a conference paper at ICLR 2026
Table 2: Training-time examples of LLM reasoning slices and our discriminator judgments. We underline the discriminator's key analysis leading to its judgment. Our model provides localized, interpretable justifications for each reasoning slice. Additional examples in Appendix
LLM Reasoning Slices
Our Full Discriminator Judgments
Given n = (a (a + b), and both a h and a + b are either both even or both odd.
The reasoning correctly notes that for odd n, a rep- resentation always exists. For even it correctly deduces n must he divisible by 4 and sets up the parameterization a m + n. b
If is odd, then both a bad a- b-ban are odd, and their product n. So, any odd number can be expressed as such.
n m with mr n. = k = n/44. It requires n > m > 0 for 6 O. However, it then claims that as long as x/42 L. such a factor pair exists, which is not always true. * ***
If n is even, then both - and + must be even, so must be divisible by 4. Let's write where is a positive integer. Then, a b 2m and a + b = 2n, where m x n = k. Thus, a = m + n and = n m. For b to be positive, n > m. and both m and n must be positive integers.
4k:
The reasoning is flawed in the even case. While is true that n must be divisible by 4, the conclu- sion that n/41 2 guarantees such a factorization ignores the requirement that b itive, meaning m < n. However, if n /4 a prime square (e.g.. n = 4. so n/4 = 1). the only factor pair is m = 1, = 1. which gives 6 = O, invalid since b must he positive integer. Thus, expressible in the required form, contradicting the claim that all even n divisible by 4 with n/4 2 work. The reasoning fails to consider that n//4 must have a factor pair with distinct factors.
Therefore, for be wise number, if odd, it's automatically a wise number. n is even, it must be divisible by 4, and n//4 must be expressible as product of two positive integers m and where m. Wait, but 4 just needs to have at least one pair of factors where < n, which is true long as n/ 2 1, which is always true since n is positive
must be pos-
= 4 is not
2 1. (2,3, -4): Prefix sums are 2, 5, 1. All positive. Good.
The reasoning lists all permutations of (2, 3, -4), computes their prefix sums, and counts how many have all prefix sums positive. It finds permutations are "good" and then asks if this the minimum. ** *EES**
2. (2, 3,: Prefix sums 2, -2, 1. The second
prefix sum is negative. Not good.
3. (3. 2, -4): Prefix sums are 3, 5, 1. All positive. Good.
The thinking process enumerates all permutations of the specific numbers (2, 3, -4) and correctly com- putes their prefix sums to determine which per- mutations have all prefix sums positive (only two: [2 3, -4] and [3.2. -4). Fach listed prefix-sum sequence is arithmetically accurate, the classification (good/ / not good) follows the stated criterion, and the cautious conclusion ("at least 2"') logically sound (they do not overclaim a proven minimum). The only minor point is they stop short of proving minimality for 3, but that does not detract from the correct- ness of the presented analysis.
4. (3, -4, 2): Prefix sums are 3, -1, 1. The second prefix sum is negative. Not good.
5. (-4, 2,3): Prefix sums are -4, -2, 1. The first prefix sum is negative. Not good.
6. (-4,3,2) 2): Prefix sums are -4, -1, 1. The first prefix sum is negative. Not good.
So. out permutations, only 2 are good. Hmm, that's interesting. So, for n = 3, seems that we can have at Icast 2 good permutations exist. But is this the minimum? Let's see another example to confirm.
Selective-Entrop without Collapse. Recent studies on RL for LLMs warn that performance gains often come with policy entropy collapse and reduced diversity, hurting exploration and calibra- tion (Cui et al, 2025) Cheng et al. [025] Wang et al. 2025). However, as shown in Fig. (a), after RL training, our per- mean-entropy distribution remains comparable to the baseline model SeeepSeek-RI-Distil---eee
with only a mild left shift and preserved spread. Moreover, con- trary the common trade-off between accuracy and entropy, we raise AIME24 accuracy from 54.0 to 61.3 (+7.3) without global drop in entropy. In Fig. (b). both models exhibit the expected "wrong > correct" pattern. However, our wrong-case violin plot is markedly tighter with a shorter low-entropy tail, and our correct-case entropy is lower than R1's, demonstrating better calibration and fewer extreme failures. More importantly, Fig. (c) (computed after removing zero-entropy tokens) reverses the ordering for correct cases: our mean entropy over non-zero-yntropy tokens ex- ceeds Rl's, and is comparable between our correct and wrong groups, indicating that we suppress entropy only where the model is confident while retaining stochasticity on informative tokens.
The contrast between Fig. (b) and (c) reveals a selective-entropy mechanism: our on-policy slicing with adversarial discriminator encourages low entropy on deterministic slices (producing many zero-entropy tokens and lower global mean) while sustaining exploration on decision-critee critical slices (higher non-zero entropy), thereby reducing high-entropy outliers when the model is wrong. This
7

---

Published as a conference paper at ICLR 2026
Entropy Distribution
Entrapy by
(c.)
Non-zerr Entrupy Correctness
m=|
m=20
m/
m
m=17
n=13
z-10
DS-RI-7D
Ours
0.08
Dansity
Entropy
Entropy
0.06
0.54
0.04
0.52
0.02
Correct
Wrong
Correct
Wrong
Mean Entropy per Case
Correctness Groups
cinesss Groups
Figure 3: Improving model without entropy collapse. Compared to Direcc-----iitilll--ee
7B, our method raises AIME24 accuracy (+7.3), but maintains comparable overall mean-entropy distribution (5.20% vs. 5.27%) (a) and tightens the "wrong" distribution (b), indicating better cali- bration with fewer extreme-uncee santeme-unceraintty uncertainty failures. Removing zero-entropy tokens (c) flips the ordering (entropy is higher on correct cases), revealing a selective-entropy behavior (decisive on determinis- tic spans, exploratory decision-critical tokens) that aligns with the AIME24 accuracy gains.
Table 4: Ablation study. Beginning with the baseline (DeepSeek-B R1 Dei--isttll-we--7777 the effectiveness of each component.
verify
Exact Match Judger
Alignment
Discriminator
Model
AIME24 AIME25
Grading Rm Score R
Reward R Reward R
Baseline (BL.)
54.0
38.0
BI,+ Standard RI.
56.3
40.7
BI, + Fixed Standard Critic
56.7
40.4
BL- Fixed GAR Discriminator
58.6
42.0
BL Trainable GAR Discriminator
59.4
42.8
BL+ Trainable GAR Discriminator
60.2
43.3
BL++ Trainable GAR Discriminator
61.3
44.3
pattern explains the observed accuracy gains on AIME24 and suggests practical control signal: token- slice-level entropy can trigger self-checks adaptive sampling precisely where uncertainty concentrated, improving both efficiency and reliability.
4.4
ABLATION STUDY
We conduct various ablations in Table 4 progressively building the baseline DeepSeek-RI- to validate each component and culminate in our final model (row 7).
Diseriminator Design Analysis. We ablate the discriminator in Table (rows 2 4). Row baseline LLM fine-tuned with standard GRPO using an outcome- based exact-match reward, without any discriminator. Row 3 adds a fixed standard critic SreeSSek-R1-Diisee (iistil--wwen-155B1 to provide feedback. Row 4 keeps the LLM critic's capacity fixed, instruction-tur judgment format with brief rationales, and deploys under the compute-efficient review schedule.
to our slice-level
The table shows that row improves over row 2 on AIME24, confirming the benefit of adding a discriminator. Moreover, Row 4 consistently outperforms both, indicating that the discriminator drives the gains. By reframing the discriminator's role from holistic solution grading to slice-level soundness judgments with concise rationales, we obtain more accurate and interpretable feedback. The resulting dense, slice-level rewards provide continuous learning signals compared with sparse exact-match grading, improving credit assignment and sample efficiency, and thereby significantly boosting performance across benchmarks.
Reward for Discriminator Training, We ablate the discriminator's reward design in Table 4 (rows 5 7). Both the alignment and discriminator rewards individually improve performance over the baseline. Combining them yields the best results, indicating the two signals are complementary. The alignment term sharpens the discriminator's ability distinguish correct from incorrect reasoning, but its supervision can be noisy because it depends on the correctness of that step's generated final answer. The diseriminator term stabilizes learning by steering the discriminator toward reference judgments. Together, these complementary signals yield a stronger and more reliable training signal.
Effectiveness of Joint Training. Finally, we evaluate joint training of the LLM reasoner and the discriminator Table and 7). Compared with using a fixed discriminator (row 4), joint on- policy updates (row 7) yield consistent gains by keeping rewards aligned with the reasoner's 's currett
8

---

Published as a conference paper at ICLR 2026
behavior. As the reasoner improves, the co co-trained discriminator adapts to detect subtler errors and provides more informative slice-level feedback, which raises the performance ceiling and mitigates drift or overfitting to static reward signal.
4.5 GAR UNLOCKS NEW APPLICATIONS AND FUTURE DIRECTIONS
Finally, we briefly discuss novel use cases and capabilities enabled by our method in this section.
RL without Full Chain-of- Thought or Verifiable Final Answers. Another advantage of our approach that it decouples RL post-training from judgeable final answers. Standard RL-based post-training re- quires generating a complete chain of thought and an automatically verifiable final answer. This makes training much slower than supervised fine-tuning (which only predicts the next token) and restricts applicability to tasks with clear evaluators (e.g., requiring an execution engine for code generation and struggling with open-ended math proofs). In contrast, our model provides additional reward signals, enabling us remove the final-answer reward and update the model solely using our discriminator's scores on intermediate reasoning. We demonstrate this advantage in Table 5 Rather than generating a complete reasoning trajectory and a final answer, we stop after three reasoning slices and have the discriminator evaluate these partial traces, providing dense early feedback without a full chain of thought. This yields substantial effi ciency gains while improving accuracy: our method surpasses standard RI. with significantly less training time. Moreover, because it does not rely on final-answer rewards or external executors, naturally extends to tasks with tord-lo-evaluate outputs, such as mathematical proofs.
Table 5: Partial-trace evaluation without final-answer reward yields faster training and higher accuracy than standard RL.
AIME24 Training Time (hours)
Standard RL
56.3
16
Ours (3 slices)
57.7
6
Distilling Reasoning Patterns with GAR. GANs have been widely used for distilling patterns in tasks like im- age generation (Sauer et al. 2024j. Similarly, we show that GAR enables reasoning distillation, aligning stu- dent model's reasoning pattern with that of a teacher.
Table Distinguishability of reason- ing patterns. We report the success rate of human experts distinguishing gener- ated reasoning from Gemini reasoning before and after GAR training.
In demo experiment, we use the S1K-1.1 (Muennighoff eet al. 2025) dataset from the OpenR1 project, containing two reasoning trajectories, "gemini thinking trajectory" and waapoeeetttiinnnntrjaijeee
w/o GAR w/ GAR
Success rate
82.3%
55.9%
,,ary"""" to train the discriminator to distinguish between the two styles. We then use this discriminator to jointly train the reasoner and discriminator within our GAR frame- work. Notably, the reasoner is only trained on the Math220K dataset, without exposure to the Gemini trajectory during training. In the evaluation, human experts are asked to differentiate be- tween the generated and Gemini trajectories in randomized side-by-side comparison. As shown in Table 6 our model significantly increases similarity to the Gemini reasoning style, reducing the distinguishability success rate from 82.3% to 55.9% (close to 50% random guess baseline), making substantially harder for experts to tell the two apart. Experimental details are included the Ap- pendix A A natural extension of GAR 's distillation ability is human preference alignment, where teacher reasoning comes from human explanations, enabling alignment with human llie- reasoning. We leave empirical validation to future work.
5 CONCLUSION
In this paper, we presented an adversarial co-training framework that couples an LLM Reasoner with an -LM-based Discriminator to deliver dense, calibrated, slice-level rewards that supplement sparse exact-match grading. Our model partitions reasoning into logically complete slices, and the Discriminator provides concise and structured feedback. During training, the reasoner is rewarded for logically consistent steps that lead to correct answers, and the Discriminator is rewarded for correctly detecting reasoning errors. This design reduces annotation burden, mitigates reward mis- specification and reward hacking, and improves credit assignment and sample efficiency under a controlled compute budget. Empirically, the approach yields consistent gains over strong RL base- lines on mathematical tasks, including +7.3 on AIME24 for DeepSeek- -aiec-RR--Diseee -Dsstilll RRi--itilll-ween--- Qwen-7B and + +1..0 for DeepSeek-RI. R1- -8m along with better calibration and fewer extreme failures.

---

Published as a conference paper at ICLR 2026
Acknowledgements. Thanks to all who supported this project and to the anonymous reviewers for their constructive comments.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint (rXiv:2303.0877) 2023.
Jiaqi Chen, Bang Zhang. Ruotian Ma, Peisong Wang, Xiaodan Liang, Zhaopeng Tu, Xiaolong Li, and Yee K Wong. Spe: Evolving self-play critic via adversarial games for IIm reasoning. arXiv preprint arXiv:2504.1916 2025.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint urXiv:240.013335, 2024.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Bo Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective. arXiv preprint arXiv:2506. 14758, 2025.
Pengyu Cheng, Yong Dai, Tianhao Hu, Han Xu, Zhisong Zhang, Lei Han, Nan Du, and Xiaolong Li. Self-playing adversarial language game enhances Ilm reasoning. Advances in Neural Information Processing Systems, 37:126515-12654. 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10. 14168, 2021.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan. Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22611, 2025.
DeepSeek-AL. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 12948
2025. httpptttss///axxiv... wrt.:wo.g//aa/o.oe
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving fac- tuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning. 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Bofei Gao, Zefan Cai, Runxin Xu, Peiyi Wang, Ce Zheng, Runji Lin, Keming Lu, Dayiheng Liu, Chang Zhou, Wen Xiao, al. Llm critics help catch bugs in mathematics: Towards a better mathematical verifier with natural language feedback. arXiv preprint arXiv:2406. 14024, 2024.
lan J Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Nathan Habib, Clémentine Fourrier, Hynck Kydlíčck, Thomas Wolf, and Lewis Tunstall. Lighte- val: A lightweight framework for 1lm evaluation, 2023. URL https: com/ hucginggaac//lgghtee.e
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint S03::2203008844 2021.
Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty- planning enhances information seeking in large language models. arXiv preprint arXiv:2402.03221, 2024.
10

---

Published as a conference paper at ICLR 2026
Hugging Face. Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https: aape/peeeeee/eeeeeeeeoe
Benjamin Kempinski, Ian Gemp, Kate Larson, Marc Lanctot, Yoram Bachrach, and Tal Kachman. Game of thoughts: Iterative reasoning in name-theoretic domains with large language models. 2025.
Jakub Grudzien Kuba, Mengting Gu, Qi Ma. Yuandong Tian, and Vijai Mohan. Language self-play for data-free training. arXiv preprint arXiv:2509.0744 0444444499, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujuu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint arXiv:2305.191) 2023.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv preprint arXiv:2506.22 22125006..1111 19, 2025a.
Junnan Liu, Hongwei Liu, Linchen Xiao, Ziyi Wang, Kuikun Liu, Songyang Gao, Wenwei Zhang, Songyang Zhang, and Kai Chen. Are your iims capable of stable reasoning? arXiv preprint arXiv:2412.13147, 2024.
Mickel Liu, Liwei Jiang, Yancheng Liang, Simon Shaolei Du, Yejin Choi, Tim Althoff, and Natasha Jaques. Chasing moving targets with online self-play reinforcement learning for safer language models. arXiv preprint arXiv:2506.07468 2025b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rX1v:1711.05101, 2017.
Ang Lv, Ruobing Xie, Xingwu Sun, Zhanhui Kang, and Rui Yan. The climb carves wisdom deeper than the summit: On the noisy rewards in learning to reason. arXiv preprint arXiv:2505.22653, 2025.
MAA. American invitational mathematics m examinationn (aime). Mathematics Competition Series, aime Accessed: October 5, 2023.
URL
MAA. American mathematisccs competiiionn (ame 10/12). Mathematics Competition Series, b. URL /amc Accessed: October 5, 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zeutlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19399 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Qwen Team. Qwen2.5: A party of foundation models, September 2024, URL github. tithub.i./blal wtt.l///lllooooooo q/w.b//.5////2////////
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis-
tillation. In European Conference Computer Vision, pp. 87-103. Springer, 2024.
11

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with stage-wise context scaling for efficient train- ing ri-kike reasoning models. arXiv preprint arXiv:2503.17287 2025.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song. Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022.
Ziyu Wan, Yunxiang Li, Xiaoyu Wen, Yan Song, Hanjing Wang. Linyi Yang, Mark Schmidt, Jun Wang, Weinan Zhang, Shuyue Hu, et Rema: Learning meta-think for ilms with multi-agent reinforcement learning. arXiv preprint arXiv:2503, 09501, 2025.
Peiyi Wang. Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math :ath-shepherd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:2312.08923 2023.
Shenzhi Wang, Le Yu. Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, et al. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning. arXiv preprint S0.iv:250660993333 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. -hain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? arXiv preprint arXiv:2410.05584 2024.
Zhiheng Xi, Wenxiang Chen, Boyang Hong, Senjie Jin, Rui Zheng, Wei He, Yiwen Ding, Shichun Liu, Xin Guo, Junzhe Wang, al. Training large language models for reasoning through reverse curriculum reinforcement learning. arXiv preprint arXiv:2402.008808 2024.
Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 27723-27730, 2025.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui. Buffer of thoughts: Thought-augmentted reasoming with large language models. Advances in Neural Information Processing Systems, 37:113519- 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprini arXiv:2401.10020 3, 2024.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. 15240, 2024.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025.
Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang. Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03:: 03335, 2025.
Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, Xinyu Yang, Runpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, et al. Parallel-rl: Towards parallel thinking via reinforcement learning. arXiv preprint arXiv:2509.07988 2025.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
In the appendix, we provide additional information as listed below:
Sec.
provides experimental details for reasoning distillation.
Sec.
lists the system prompts used for the reasoner and the discriminator.
Sec.
provides additional experimental results on coding ability.
Sec.
presents ablation results the slice segmentation design.
Sec.
provides additional training-ting examples of the LLM reasoning slices and our
discriminator judgments.
Sec.
discusses the limitations of method.
Sec. G
provides the ethics statement.
Sec. provides the reproducibility statement.
A EXPERIMENTAL DETAILS FOR REASONING DISTILLATION.
In the demo experiment, OpenR1 project, which contains two reasoning trajectories, "gemini_thinking didepseek_thinkingg.fe thhinking_trajectrry
the S1K-1.1 MMuennighoff al. 2025) dataset from the
and train discriminator distinguish between the two styles. We use the same settings as our main experiments and train the discriminator for 1,000 iterations. We then integrate this discriminator into our GAR framework and jointly train the reasoner and the dis- criminator under the Partial trace setting: instead of generating the full reasoning trajectory and final answer, we stop after three reasoning slices and have the discriminator evaluate these partial traces. We train this model for 3,000 steps, and all other hyperparameters are the same in the main experiment. The reasoner is trained only on the Math220K dataset, without exposure to Gemini trajectory during training.
For the human preference study, we generate 200 reasoning slices, pair them side by side with the corresponding Gemini slices, and randomly shuffle the order. We then ask 10 experts from three academic institutions, including 3 undergraduate students and PhD students, to first familiarize themselves with the differences between Gemini and DeepSeek-R1 reasoning styles by reviewing the S1K-1.1 dataset, and then conduct the evaluation. We compare the success rate of experts dis- tinguishing generated reasoning from Gemini reasoning before and after GAR training, and report the results in Table
B SYSTEM PROMPTS
We provide the system prompts for LLM reasoner and the discriminator as follows:
System prompt for the LLM reasoner:
You are a helpful AI Assistant that provides well-reasoned and detaile led responses. You first think about the reasoning process as an internal monelogue and then provide the user with the answer. Respond in the following An... \n</answer>
System prompt for the LLM discriminator:
You are an evaluator responsible for assessing whether reasoning thinking process is reasonable, rigorous, and accurate. Based on these criteria, determine if the analysis analyze the reasoning very briefly, then respond with ' for high quality or ' if it is not. Finally, provide a brief but specific explanation for your judgment. Hint: You can first summarize the given thinking process to identify the main reasoning chain, then ana maae the reasoning chain sentence by sentence.
of high quality. First,
13

---

Published as a conference paper at ICLR 2026
C PERFORMANCE ON CODE GENERATION
We further evaluate our model on coding- based reasoning tasks. As shown in Table 7 our method yields substantial improvements across multiple coding benchmarks. To ensure statistical reliabil- ity, all results are averaged over 30 trials per benchmark (three independent training runs, each evaluated with 10 inference seeds).
Experiments are conducted on the CodeForces-Col dataset, where the (approximated) ground- truth reasoning traces are produced by DeepSeek-R1. The maximum rollout length during training is fixed at 800 steps. The reward function is an equal-weight combination of three components:
CF-Code Reward: Computes a weighted sum over public test cases, assigning for pass and for fail.
Code-Format Reward: Assigns reward of 1 when the model output includes valid code-block wrapper (e.g., python *))
Incorporates alignment and discriminative terms, using the same structure as in the math-reasoning setting.
Critic Reward:
Table 7: Pass@1 accuracy on coding benchmarks. Similar to the results reported in the main paper, scores are averaged over 30 runs per benchmark to reduce evaluation noise. Our method significantly improves coding performance.
Model
LiveCodeBench
HumanEval
HumanEval+
DS-R -i-----issill-weeen 7B DS-RI-Dissiil-Qween-TT GAR (Ours)
37.4
40.4
37.8
43.6
42.7
39.3
D ABLATIONS OF SLICE SEGMENTATION DESIGN
D.1 SEGMENTATION STRATEGY
In this section, we clarify the rationale behind adopting the proposed segmentation method based on explicit delimiters combined with token-length range. Chain-of-thought (CoT) generated by contemporary reasoning models (e.g., DeepSeck-RI, Gemini) is typically organized into coherent fragments separated by line breaks (""nn") and discourse cues such as "Wait," "Since," and "There- fore,". These natural markers provide reliable boundaries for forming semantically self-contained slices. Applying additional token-length constraint prevents segments from being excessively short or long, while keeping computation and implementation overhead minimal.
Table compares this rule-based method with two alternatives: (1) pure fixed-length token windows and (2) L.LM-based semantic segmentation, where a model is prompted to automatically partition the COT. The results show that the alternatives either disrupt coherent reasoning steps require substantially higher training cost, without yielding performance improvements over our method.
Table 8: Ablation on different slice segmentation strategies.
Method
AIME24 Training Time (hours)
Pure fixed-length token windows
58.7
19
LLM-based semantic segmentation
61.6
35
Ours
61.3
19
D.2 SENSITIVITY TO SLICE LENGTH VARIATION
We further add a sensitivity analysis over slice length. As shown in Table 9 model performance is highest and most stable when slices contain approximately 320-560 tokens. Shorter slices of- ten contain no explicit reasoning error, making it difficult for the discriminator to learn informative supervision signals. In contrast, very long slices typically include at least flaw, causing most
14

---

Published as a conference paper at ICLR 2026
segments be labeled "incorrect" and reducing label diversity, which in turn weakens the dis- criminator's 's effectiveness.
Table 9: Ablation over slice length (tokens) on AIME24.
Slice Length
160
320
480
560
800
960
1120
1440
@ accuracy 57.4
61.3
61.5
61.4
61.0
59.3
56.5
56.8
E SLICE-LEVEL. FEEDBACK FROM OUR DISCRIMINATOR.
We provide additional examples of the LLM reasoning slices and our discriminator's judgments during joint training in Table 10 and Table
F LIMITATIONS AND FUTURE WORK
The proposed GAR has a few remaining limitations. First, remains challenging balance the dis- criminator's ' reasoning depth with compute efficiency. In this work, we reformulate the format into an maalysi--score-rtiiie serving its reasoning ability. Nevertheless, there is room to further improve compute-effin compute-efficient rea- soning. Promising directions include adaptive early-exit mechanisms, dynamic truncation of anal- ysis conditioned confidence, etc. Second, although we compute slice-level dense rewards, the final objective aggregates them into single trajectory-level signal by averaging, which can dilute local credit assignment and increase variance. Better ways to leverage slice-wist information could improve training efficiency and stability.
format to make the discriminator more efficient while pre-
G ETHICS STATEMENT
This work seeks to improve model reasoning and training efficiency, but we acknowledge risks including bias amplification, spurious correlations, reward hacking, and potential misuse for gener- ating harmful or deceptive content. We rely on publicly available datasets (Math-220K) and adhere their licenses, avoid collecting or exposing personal data, and assess safety and fairness where applicable, documenting observed failure modes. We will not release artifacts that materially in- crease misuse risk and provide usage guidelines support responsible deployment. Downstream use should include human oversight, continuous monitoring, and domain-specine risk assessments.
H REPRODUCIBILITY STATEMENT
We will release code, pretrained checkpoints, and detailed scripts to reproduce all results, including data preparation, training, and evaluation. The repository will provide exact configurations (model architectures, the analysis-score-ratinn aggregation, loss functions, optimizers, schedules), dataset splits and preprocessing. and the evalu- ation harness with metrics. We document software and hardware environments (package versions, CUDA/cuDNN, GPU types), supply containerized setups (Conda/Docker), and fix random seeds: results reported mean over 30 runs. The implementation details are also provided in Sec. 4.1
discriminator format, slice-level reward computation and
15

---

Published as a conference paper at ICLR 2026
Table 10: Training-time examples of LLM reasoning slices and our discriminator judgments. We underline the discriminator's key analysis leading to its judgment. Our model provides localized, interpretable justifications for each reasoning slice.
LLM Reasoning Slices
Our Full Discriminator Judgments
Case 1: p divides 2", Since 2" is a power of 2. p must
The reasoning examines the case where the prime p divides 2". correctly concludes p = 2, substitutes into the equation 2"q = 2 1, and argues that the left side 2 is even while the right side 2'g + 1 is odd, leading to contradiction. **YES**
Case 2: p divides Since is a prime, = 9-
Let's explore each case.
**Case 1: p 2**
Substituting p = into the equation, we get:
The reasoning is rigorous and accurate. It system- atically considers the case p = 2, substitutes it into the equation, and rearranges to 2 = 2"q + 1. It correctly observes that 2 is even, while 2'g T is odd (since 2'y is even for s 2 1 ). leading a contradiction. The logic is clear, step-by-step. and based on fundamental parity properties, leaving room for ambiguity. This is a high-quality analysis for this specific case.
q²q== = 2 1
Rearranging:
2'
Since 2 is even, the right-hand side must also be even. However, 2'q is even if s 2 1, so 2"g g + is odd. This is a contradiction because an even number cannot equal an odd number. Therefore. p 2 is not valid solution.
2
Let start with the first part. know that a wise number can be written as a² 12 where and are positive integers. I can factor this expression as (a-b) So, any wise number can be expressed as the product of two integers (a b) and (a + b). Since a and are positive integers, + b must greater than a b, and both a + b and be positive integers. Moreover, both a + b and a b must have the same parity because + b + (a is even.
The reasoning shows that a² - 62 = (a (a-bb(ee b) + 6) requires both factors to have the same parity, so the product is either odd or divisible by 4. Thus, numbers of form 4k + 2 (like 2) cannot be wise numbers. YES**
must
correctly identifies that a wise number is of the form a² -8² ====0 (a b)(a + b) and notes that a + b and b must be positive integers with the same parity (both even or both odd) because their sum is 2o (even). This leads to the key insight: the product (a b)(a + b) must be either odd or divisible by 4. Consequently. numbers of the form 4k + 2 (which even but not divisible by 4) cannot he expressed as such a product. The example of (which is x 0 + 2) solidifies the argument. The logic is clear, step-by-step, and based on sound number the- ory principles.
b) 2a must be even, meaning their Therefore, both a + and a must be either both even or both odd. This implies that their product (a 1--bj () + b) must he either odd or divisible by 4. So, if a number is of the form 4k 22, it cannot be ex- pressed the product of two integers with the same parity. For example, number like 2 cannot be writ- ten as the difference of two squares because 2 is of the form 4k + 2 (where k- = 0), and there are no two integers and that would satisfy a² - b² = 2.
3 But since this is an approximation, I need to check if 2686 2687 is wise number. Let me check if 2686 is wise number.
The reasoning checks if 2686 and 2687 are wise numbers, correctly using the parityldivssiiility rule. It then concludes 2687 is the 2015th wise number without verifying the count. N*
2686 divided by 2 is 1343, which is odd. So, 2686 is 2 times an odd number, which is not multiple of 4. Therefore, 2686 is not a wise number hecause it cannot he expressed as the difference of squares. Next, check 2687. Since 2687 odd, it can be ex- pressed as a difference of squares. So, 2687 is a wise number.
While it correctly identifies that 2686 (even but not divisible by 4) cannot he wise number, and that 2687 (odd) can be a wise number, it fails to verify whether 2687 is actually the 2015th wise number. The reasoning jumps to a conclusion without cal- culating how many wise numbers exist up to 2686 and 2687. Specifically, fact that not all odd numbers are necessarily wise (c.g.. 1 is odd and wise, but other constraints might exist) and, more importantly it does not compute the cumulative count of wise numbers to confirm the index. This lack of systematic counting or verifica- tion makes the reasoning incomplete and potentially incorrect.
Therefore, the 2015th wise number is 2687. Wait, but let me make sure. Let's verify how many wise numbers are less than or equal to 2686 and 2687.
does not account for the
16

---

Published as a conference paper at ICLR 2026
Table 11: Training-time examples of LLM reasoning slices and our discriminator judgments. We underline the discriminator' key analysis leading to its judgment. Our model provides localized, interpretable justifications for each reasoning slice.
LLM Reasoning Slices
Our Full Discriminator Judgments
But wait, should double-check if this is indeed the minimum. Maybe I made mistake in applying the Lagrange multipliers. Alternatively, perhaps there's s geometric interpretation that can confirm this re- sult.
The reasoning cross-checks a calculus result (La- grange multipliers) with geometric interpretation: the sum of distances from points the axes (12,9) is minimized by the straight-line distance from the origin to (12,9), which is 15, The results match
Thinking geometrically, as considered carlier, the expression represents the sum of distances from points the coordinate axes the point (12,9) The minimal total distance should be the straight line distance from the origin to (12, 9). which is V12² + 92 vI44 + 8I /225 15. That matches the result got using calculus, so it seems correct.
*
The reasoning is rigorous and accurate. It cross- validates the result obtained from calculus (La- grange multipliers) with a geometric interpretation:
expression va² 144 + +15 11:5-10 (1]1-=)3-1 + 81 represents the sum of distances from point on the x-axis to fixed points, and the minimal path is indeed straight line from the origin to (12,9), yielding a distance of 15. This geometrie insight is sound and confirms the calculus result, demonstrat- ing thoroughness and multiple perspectives. The logic is clear and well-supported.
So, the minimal value is indeed 15.
2 From this table, I notice a pattern. It seems that (1) alternates between I and 0, but it's not straightfor- ward. Let try to see if there's 's a base-3 pattern something related to the recursive formula involving division by 3.
The reasoning observes a pattern in f(x) and re- calls the recursive definition [Tr) = 1 f(n 1) f [[n/3]). It tests simple hypothesis (based on divisibility by 3) but immediately notes counterexamples (like f(6)=1. f(9)=1), showing the pattern is more complex.
Looking at the recursive formula:
**NO**
f(n - 1). ((as))
T2
The reasoning is not rigorous accurate. It tempts to identify a pattern (e.g.. alternation or mod- ulo 3 behavior) but quickly contradicts itself with counterexamples (e.g.. S(3)
This suggests that the value of ((n) depends on the previous value f'n - 1) and the value at [n/3]. Let me consider the soquence more carefully. It seems that (() is except when = 0 mod 3. but that's 's not entirely accurate because f(3) O.
1, ((9) 1). The analysis is superficial and does not dceply engage with the recursive formula. It mentions considering the recursion but does not systematically compute values or derive propertics.
O. f(6)
f(6) = 1, (()) = and (12) would depend on f(11) and f(1).
3 In a geometric sequence, each term is obtained by multiplying the previous term by a constant ratio r. Given d2 2, we have:
The reasoning uses the geometric sequence defini- tion to set up equations for 02 and as. solves for the ratio by dividing the equations, finds = 1/2, then solves for a1 4, and derives the general for- mula a., 23-0 **
42 = 0 X xr=2=aa X (1)
Given d5
7. have:
The reasoning is rigorous and accurate. It correctly uses the definition of geometric sequence, sets up equations based on the given terms (a2 2 and as = 1/4), and solves for the common ratio r by di- viding the equations. The calculation 1 = , is correct, leading to r = 1 Then, it substi- tutes back to find 01 4. Finally, it expresses the general term an in both exponential and simpli- fied form (23-)). The steps are logical, clear, and mathematically sound, with no errors or gaps. This is a high qual-iyultty analysis.
as 01
a x
(2)
Dividing equation (2) by equation (1):
01 x p 1 X = =r V
1
Substituting
1/2 into equation (1):
2 a - a: 44
Thus, the geometric sequence is:
1-1
On = 4x
2 x x ++1 = 23-
17


Reviews:
[{"review_id": "EI1XsWGYES", "reviewer": "Reviewer_mfoq", "summary": "This paper introduces Generative Adversarial Reasoner (GAR), a novel framework that jointly trains an LLM reasoner and an LLM-based discriminator through adversarial reinforcement learning to enhance mathematical reasoning capabilities. The key innovation is partitioning reasoning chains into logically complete \"slices\" that the discriminator evaluates for soundness, providing dense step-level rewards rather than relying solely on sparse final-answer feedback. The reasoner is rewarded for producing logically consistent steps leading to correct answers, while the discriminator receives rewards for correctly detecting errors and distinguishing between generated and reference reasoning. Experiments on mathematical benchmarks show consistent improvements over strong baselines, with notable gains on challenging datasets like AIME24 (+7.3 for DeepSeek-R1-Distill-Qwen-7B and +10.0 for DeepSeek-R1-Distill-Llama-8B).", "questions": "1. Why use a smaller model (1.5B vs 7B) for the discriminator in the Qwen setup but the same size for Llama? How does discriminator capacity affect performance?\n\nOthers see Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel approach to LLM reasoning improvement by adapting GAN-style adversarial training to the reasoning domain. The slice-level evaluation mechanism is creative and addresses computational efficiency concerns while maintaining granular feedback. The joint training paradigm where both reasoner and discriminator co-evolve is innovative compared to static reward models.\n\n2. The experimental methodology is solid with comprehensive evaluation across seven mathematical benchmarks. The authors provide proper statistical analysis (averaged over 30 runs) and fair comparisons using unified evaluation protocols. The ablation studies systematically validate each component's contribution. The selective-entropy analysis provides valuable insights into how the method avoids common pitfalls of RL training.\n\n3. The paper is generally well-written with clear motivation and methodology. The slice-level feedback examples in Table 2 effectively demonstrate the discriminator's capabilities. The figures and experimental results are presented clearly.", "weaknesses": "1. Limited Theoretical Foundation: While the empirical results are strong, the paper lacks theoretical analysis of the joint training dynamics. There's no convergence analysis or guarantees about the co-evolution process, which is concerning given the potential for reward hacking mentioned by the authors.\n\n2. Slice Segmentation Methodology: The slice partitioning approach appears somewhat ad-hoc. The paper doesn't provide sufficient justification for these choices or analysis of how sensitive the results are to different segmentation strategies.\n\n3. Computational Overhead Analysis: While the authors claim efficiency improvements, the analysis is incomplete. Table 3 shows training time comparisons, but there's no detailed breakdown of computational costs during inference or comprehensive analysis of the discriminator's computational burden across different problem complexities.\n\n4. Limited Baseline Comparisons: The paper primarily compares against variants of DeepSeek-R1-Distill models. Comparisons with other recent reasoning enhancement methods (beyond standard RL) would strengthen the evaluation. The related work section mentions several relevant approaches that aren't empirically compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tn0VqtQpMq", "reviewer": "Reviewer_iNKw", "summary": "This paper introduces Generative Adversarial Reasoner (GAR), an on-policy framework that jointly trains an LLM reasoner and an LLM discriminator via adversarial reinforcement learning. The discriminator evaluates slices of reasoning, providing dense, on-policy, step-level rewards. This method significantly improves mathematical reasoning, showing strong gains on benchmarks like AIME24.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's core idea and method hold research value. Judging from the experimental results, the proposed method indeed brings about a relatively clear improvement compared to standard reinforcement learning methods.\n\n2. The paper conducts thorough ablation studies, sufficiently demonstrating the effectiveness of each component within the designed framework. Furthermore, the discriminator's truncation experiment design gives consideration to its impact on training efficiency.\n\n3. The paper includes several interesting discussions. First, the discussion on the experimental conclusions regarding policy entropy collapse is quite thorough. Additionally, the discussion on \"RL without Full Chain-of-Thought or Verifiable Final Answers\" is, in my opinion, quite intriguing and holds potential reference value for the community's understanding of RL.", "weaknesses": "1. First, I believe the paper aims to propose a denser reward strategy. However, the final method is still based on GRPO, which assigns a reward to the entire sequence. Therefore, although the proposed method can provide step-level rewards, it essentially still provides a denser Outcome Reward during training. I think the authors' description of their contribution requires careful consideration, as the PRM is not used in a step-level way. Admittedly, in Section 4.5, the authors mention the possibility of \"RL without Full Chain-of-Thought or Verifiable Final Answers.\" Detailing these experiments would clarify the paper's contribution.\n\n2. The comparison with other RL methods is insufficient. The authors only include standard RL as the sole baseline. However, there is a significant amount of other work on LLM Reasoning and RL. I believe the authors need to compare their method against some of the more influential works in this area to further demonstrate its effectiveness.\n\n3. The \"slice\" plays a crucial role in the proposed method. The paper defines it as \"logically coherent slices\" segmented by \"delimiters.\" This definition is somewhat heuristic. Is the model sensitive to the choice of $L$? Furthermore, what exactly are these \"delimiters\" (e.g., newlines, specific punctuation, or semantic boundaries)? A clearer description of the slicing algorithm is needed for reproducibility. Additionally, the text mentions mixing generated slices with an \"equal number of reference slices.\" Do these \"reference slices\" originate from the baseline model’s (DeepSeek-R1) COT? If so, this should be clarified in methodology (Section 3). This element seems to function more like a regularizer (to prevent the discriminator from deviating excessively from reasonable reasoning patterns) than a traditional GAN objective (imitating $p_{ref}$).\n\n4. The proposed GAR framework is quite complex, involving numerous training hyperparameter settings ($\\lambda_{1}=\\lambda_{2}=\\lambda_{3}=1, \\lambda_{4}=0.5$). The paper lacks an analysis of these weight choices. If performance is highly sensitive to minor variations in these weights, the framework will be relatively difficult to reproduce and apply.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fgUGGjg0CJ", "reviewer": "Reviewer_gLeA", "summary": "This paper introduces the Generative Adversarial Reasoner, a framework that co-trains an LLM reasoner and an LLM discriminator through adversarial reinforcement learning. \nThe discriminator evaluates \"slices\" of the reasoning chain, providing dense, step-level rewards that improve credit assignment and sample efficiency over sparse final-answer rewards. \nThis method enhances performance on mathematical benchmarks and offers a flexible, modular approach for enhancing reasoning quality.", "questions": "See the Weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a co-training approach where the reasoner and discriminator evolve together, providing dense, calibrated, step-level rewards that significantly improve credit assignment over sparse outcome-based methods.\n\nIt shows improvements across multiple mathematical reasoning benchmarks (e.g., +7.3 on AIME24), outperforming strong RL baselines while maintaining comparable training efficiency.\n\nIt implements compute-efficient innovations like slice-level evaluation and response length truncation (128 tokens) that maintain performance while substantially reducing training time and resource requirements.", "weaknesses": "The adversarial training setup may lead to reward hacking, where the discriminator and reasoner adapt to each other's weaknesses rather than improving reasoning quality. How does this paper mitigate the risk of reward hacking?\n\nIt is recommended that the method proposed in this paper be validated on a broader range of tasks, such as coding and commonsense reasoning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MCkqEpbnJG", "reviewer": "Reviewer_JCij", "summary": "The paper proposes Generative Adversarial Reasoner (GAR): a joint RL framework that co-trains a reasoner LLM with a slice-level LLM discriminator to provide dense, on-policy process rewards. Reasoning chains are partitioned into “logically complete” slices; the discriminator outputs an analysis → YES/NO → brief rationale per slice, and training combines exact-match and aggregated slice rewards. On seven math benchmarks,  GAR improves over strong DeepSeek-R1-Distill baselines; authors report efficiency tricks and partial-trace RL without final answers.", "questions": "How do you detect and mitigate reward hacking (e.g., producing slice-looking “clean” text without real progress)? Any concrete failure cases? Why binary slice labels (YES/NO) instead of graded scores or token-level attributions? Did you try continuous rewards?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1. Stepwise tables show improvements from: (i) adding any critic, (ii) reframing to slice-level judgments, and (iii) joint on-policy updates with alignment + discriminator rewards—culminating in the strongest results.\n\nS2. The discriminator operates on logically complete slices with an analysis → YES/NO → brief rationale format, producing localized, checkable feedback that improves credit assignment without expensive human PRM labels.", "weaknesses": "W1.  Novelty is overstated vs. existing PRM/LLM-judge and adversarial/self-play lines What is addressed as “co-evolving slice-level discriminator” is very close to (i) PRMs trained on stepwise feedback (human or LLM-as-judge) and (ii) self-play / adversarial critics already explored for reasoning. The paper doesn’t clearly isolate what’s genuinely new beyond (a) cutting long traces into “slices” and (b) mixing a GAN-style real/fake discriminator with an answer-aligned term. Without tighter theoretical or empirical differentiation from PRM-as-judge (e.g., Generative Verifiers, Math-Shepherd, debate/multi-agent critics), the claimed contribution reads incremental. Also, GAN objectives assume a meaningful data distribution for “real” vs. “generated.” Here, “real slices” are reference CoT fragments from existing models, not ground-truth human-verified reasoning. The discriminator can then learn stylistic artifacts of reference traces rather than logical soundness. This undermines the premise that Rd improves correctness (vs. style mimicry). No analysis disentangles style detection from error detection.\n\nW2. Slicing heuristic (L=320 tokens, delimiter rules) is arbitrary and unvalidated\nThe core claim—“slice-level evaluation improves reliability”—depends on how you segment traces. But the paper gives no sensitivity study over L, no human audit of “logical completeness,” and no robustness to alternate chunking strategies. If performance hinges on a specific, hand-tuned slicer, the method risks brittleness and poor reproducibility.\n\nW3. All evidence comes from seven mathematical benchmarks. Math affords crisp local checks (algebraic identity, parity, etc.), which makes slice-level “soundness” judgments unusually reliable. It’s unclear that the same discriminator will remain calibrated on domains without local verifiers (open-ended QA, multi-hop fact retrieval, safety reasoning, tool use, code with flaky unit tests, long-form writing).\n\nW4. The “analysis–score–rationale” prompt and binary slice labels fit math well (short, checkable steps). In natural language reasoning, law, or medical QA, slices can be ambiguous and correctness is non-binary. The paper doesn’t show how slice segmentation and binary scoring handle ambiguity, partial credit, or evidence attribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)