Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Authors: Pinzheng Wang
Juntao Li
Zecheng Tang
Haijia Gui
Min zhang
Venue: ICML 2025 poster (2025)
Keywords: Self-play
Large Language Models
LLM Reasoning
Primary area: deep_learning->large_language_models
TL;DR: We explore how self-play can enhance the rationality of LLMs in the reasoning process.

Abstract: Large language models (LLMs) have demonstrated considerable reasoning abilities in various tasks such as mathematics and coding.
However, recent studies indicate that even the best models lack true comprehension of their reasoning processes.
In this paper, we explore how self-play can enhance the rationality of models in the reasoning process without supervision from humans or superior models.
We design a $\textit{\textbf{C}ritic-\textbf{D}iscernment \textbf{G}ame}~(\textbf{CDG})$ in which a prover first provides a solution to a given problem and is subsequently challenged by critiques of its solution. 
These critiques either aim to assist or mislead the prover. 
The objective of the prover is to maintain the correct answer when faced with misleading comments, while correcting errors in response to constructive feedback.
Our experiments on tasks involving mathematical reasoning, stepwise error detection, self-correction, and long-chain reasoning demonstrate that CDG training can significantly improve the ability of well-aligned LLMs to comprehend their reasoning process.

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Pinzheng Wang 12 Juntao Li 12 Zecheng Tang 12
Haijia Gui 12 Min Zhang 12
Abstract
comprehension of their reasoning processes and instead rely mostly on probabilistic pattern matching (Mirzadeh et al., 2024; Valmeekam et al., 2022; Razeghi et al., 2022). This can be reflected in the instability of the reasoning process of LLMs, which tends to produce hallucinations and errors while struggling to detect and correct these issues on its (Pan et al., 2023; Kamoi et al., 2024). This limitation is particularly detrimental for long chains of thought, which are essential for solving complex and challenging tasks. As the length of the reasoning process increases, intermediate errors and incorrect attempts are more likely to accumulate if the model lacks the ability to detect or correct them (Zhao et al., 2024; Wu al., 2024a).
Large language models (LLMs) have demon- strated considerable reasoning abilities in various tasks such as mathematics and coding. However, recent studies indicate that even the best models lack true comprehension of their reasoning pro- cesses. In this paper, we explore how self-play can enhance the rationality of models in the reasoning process without supervision from humans or su- perior models. We design Game (CDG) in which a prover first provides a so- lution to a given problem and is subsequently chal- lenged by critiques of its solution. These critiques either aim to assist or mislead the prover. The objective of the prover is maintain the correct answer when faced with misleading comments, while correcting errors in response to constructive feedback. Our experiments on tasks involving mathematical reasoning, stepwise error detection, self-correction, and long-chain reasoning demon- strate that CDG training can significantly improve the ability of well-aligned LLMs to comprehend their reasoning process. We have released our code here.
Aitii--Diccermment
To mitigate this issue, existing approaches employ process based reward models (PRMs) or preference data pairs to provide stepwise error supervision for the reasoning pro- cess (Lightman et al., 2023; Lai et al., 2024; Uesato et al., 2022a). However, these methods struggle to explicitly de- fine fine- fineggrained steps in general reasoning, merely indicate which step is better without providing specific explanations, and frequently rely on human-annotated data, making them difficult to scale up (Guo et al., 2025; Zhang et al., 2025). Therefore, we explore enhancing the rationality of the rea- soning process of LLMs through self-play in language, with- out supervision from humans or superior models.
We design a
Game (CDG), where the agent engages in discussion about its reasoning process with different goals. Specifically, our game involves three roles: the prover, the helpful critic, and the misleading critic. As shown in Figure 1, each role has the following objective:
1. Introduction
Large language models (LLMs) have achieved significant success in reasoning tasks such as coding and mathemat- ies (Roziere et al., 2023; Dubey et al., 2024; Yang et al., 2024a). By training on reasoning paths from humans su- perior models, L.MSs can generate impressive step-by-step reasoning processes similar to human thought (Kim et al., 2023; Wei et al., 2022). However, recent studies indicate that even well fine-tuned and aligned I.IMss still lack true
Prover: Given a question, the prover is required to provide an answer with a clear chain of thought. Afterward, the prover receives feedback from a critic whose intent can be either helpful or misleading. The prover should maintain its initial correct solution and revise the incorrect solution.
"Scooo of Computer Science and Technology, Soochow Uni- versity Key Laboratory of Data Intelligence and Advanced Com- puting, Soochow University. Correspondence to: Juntao Li <ljt@ssuda.cde suda.cdu.cn..
The helpful critic receives problem along with the prover's 's incorrect answer. Its task is to iden- tify the mistake the prover's reasoning without directly correcting the solution, persuading and assisting the prover in revising the original incorrect answer.
Helpful Critic:
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Misleading Critic:
The misleading critic receives prob-

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
lem and the correct solution of the prover. Its task is to fabricate and point out false emror within the prover's 's solu- tion, intending to mislead the prover to change the original correct answer into an incorrect one.
step-wise preference data (Lai et al., 2024). These methods, however, rely on data annotated by humans or superior mod- els, which limits their scalability. Some methods attempt to automate the construction of step-level supervision us- ing techniques like Monte Carlo Tree Search (Zhang et al.; Lu et al., 2024; Wang et al., 2024; Luo et al., 2024). Yet, these approaches struggle to explicitly define fine-grained reasoning steps and tend have limitations of data quality issues or reward hacking (Guo al., 2025). In this work, we propose a self-play RL algorithm which differs from tra- ditional preference optimization. In our approach, rewards are derived from game rules, allowing for step-level super- vision at arbitrary granularity without relying on human or superior model supervision.
The critics can freely choose the granularity of the steps they critique, either to attack or to collaborate with the prover in the form of natural language. All three roles are optimized through reinforcement learning (RL) algorithms based on their respective tasks and performance in each round. To achieve success, the prover must thoroughly understand its reasoning process, ensuring it can maintain correct steps without being misled by the increasingly adaptive misleading critic, while accurately revising genuinely erroneous steps.
To the best of our knowledge, we are the first to improve reasoning capabilities through a self-play language game on fully fine-tuned and aligned models, such as LLaMA- 3.1-8B-instruct (Dubey et al., 2024) and Qwen2.5-1.5B- Instruct (Yang et al., 2024a). To evaluate whether the model truly gains a better understanding of its reasoning process, we conduct comprehensive evaluations across four tasks: mathematical reasoning, stepwise error detection, self-correction, and long-chain reasoning. In the long-chain reasoning task, the model is required to generate reason- ing processes similar to OpenAl-ol (OpenOl Team, 2024), which involves extensive trial, self-reflection, and error cor- rection to solve hard math problems (Wu et al., 2024a; Qin et al., 2024; Huang et al., 2024). Our method consistently improves model performance across all tasks, demonstrat- ing that the Distic-DDscemmment Game effectively enhances rationality in the reasoning process. We believe our method highlights the potential of self-play language games with RL as a novel training paradigm beyond instruction tuning and preference optimization (Rafailov et al., 2024; Schulman et al., 2017; Meng et al., 2024).
Self-Play in LLM Self-play is a technique where mul- tiple agents learn by interacting with themselves (Zhang et al., 2024; Cheng et al., 2024a; Xi al., 2024). Game agents like Alpha Zero (Silver et al., 2017) have shown that advanced planning and reasoning abilities can be devel- oped solely through self-play without human supervision. Inspired by these successes, many efforts have been made to enhance various LLM capabilities via self-play games. Zheng et al. (2024) and Xu et al. (2024) employ a two- agent attack-defense game to align LLMs in safety-criticn scenarios, significantly improving robustness against jail- break attacks. Additionally, many works (Chen et al., 2024; Wu et al., 2024b; Kirchner et al., 2024) utilize self-play strategies, such as cenerate-discriminat frameworks and competitive reward mechanisms, to better align LLMs with human preferences. Furthermore, studies have shown that self-play can effectively enhance agents' strategies in lan- guage games, e.g., Negotiation (Fu et al., 2023) and the Werewolf Game (Xu et al., 2023a;b). Our work is most sim- ilar to SPAG (Cheng et al., 2024b), which uses the Adversar- ial Taboo Game to improve pre-trained models' reasoning abilities. However, our approach differs that we explore how self-play can enhance the rationality of the reason- ing process in well-instructed and aligned models, such as 1eriaa..--BB-IIttttte rather than merely activating latent abilities in pre-trained models.
2. Related works
RL for LLM reasoning Reinforcement Learning (RL) has emerged as crucial training method to enhance rea- soning capabilities in large language models (LLMs), with many methods providing feedback based on the quality of the reasoning outcome, measured by the correctness of the final answer (Anthony et al., 2017: Gulcehre al., 2023; Singh et al., 2023) through an outcome reward model (ORM) (Cobbe et al., 2021c; Yang et al., 2024b; Ue- sato et al., 2022b). However, recent works have highlighted that LLMS lack true comprehension of their reasoning pro- cesses, making them prone to generating hallucinations and errors (Mirzadeh et al., 2024; Lanham et al., 2023). To address this, many works have introduced step-level super- vision for models, usually through process reward mod- els (PRMs) (Zhang et al., 2025; Uesato et al., 2022a) or
3. Self-play of Critic-Discermmen Games
3.1. Critic-Discernment Game Modeling
In our game, we consider a dataset consisting of math questions and their corresponding ground truth answers y = y(z). Given a solution 2 proposed for question 2; we employ regular expressions and the SymPy grader (Meurer et al., 2017) to evaluate the correctness of the solution, de- noted by lcomcce y). We represent the prover, helpful critic, and misleading critic as n, μ4, and p, respectively.
At the start of the game, the prover is required to provide
2

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Question: Mark went to a store where he spent one-half of his money, and then spent $14 more. he had $3 left, how much did he have when he entered the store?
Let's think through this step by step. Mark had $3 + S14 17 after he spent half of his money. So he had $17 = $34 when he entered the store.
Let's think step by step. Mark spent total of S14 S28. He had $3 left, he had $28 $3
Prover
$31 when entered the store.
You incorrectly failed to add the last $3 remaining. Critic The result should include the $3 that Mark had left.
You incorrectly doubled the $14 spent; the correct approach to account after calculating half of Critic his initial money.
(Aim to mislead)
(Aim correct)
You right, the final answer should be S34 + ................................
$37.
You are right, the half his initial moncy is $14
The $3 remaining was already accounted for when adding it to the S14 spent, so I retain my initial answer.
S17. The final answer should be S17 S34. "i have already calculated it. retain my initial answer.
Prover
Prover
1) Episodes where the Critic mislcads the Prover
Episodes where the Critic corrects the Prover
Figure I. Examples of Critic-Diseemment Game with the same math question. In the left dialogue, the critic attempts to mislead the prover into changing a correct answer, while in the right dialogue, the critic guides the prover to correct an incorrect answer. The prover must rationally decide whether to revise their response based on the critic's feedback and their own answer.
Algorithm Data collection of Pritic-Discermment Game Inputs: prover æ, helpful critic μs, misleading critic p: question x and corresponding answer y. prompt templates Shelp. m Srevise
correct solution on the first attempt and successfully re- sisting the misleading critic's deception, (2) by initially providing incorrect solution but correcting it through feedback from the helpful critic to arrive at the correct final answer. The first way of winning requires a superior abil- ity to solve the problem, and therefore we assign a larger reward to this outcome. The total reward for the prover is:
Sample prover's s initial solution T (2(12)
if comrec 2z, y) = 0 then
Sample critique ~ μlcall fuelp 2))
else
Sample critique c ~ p(cpl.fmston fmistend (x, z))
Rx = Ez- x [1 comrrtt (z, Bo w (wwewwaroooaneeee comrec (z', + 1))
end if
(1) Correct initially and resist misleading critic
Sample revised solution z/z TT (2n Srovis (,)
(1 co y) Eewp,, 1 comera (z', y) ].
Collect episode 2,,
(2) Incurrect initially but correct with the help of the helpful critic
where 7) is the hyper-parameter to ensure a larger reward of the first term, 2 ~ (2)22), z' ~ e ~ [Cc]//hep(as (2, z)), ~ p(c) z)) are simplified as 2 ~ n, e' ~ T, c ~ μ1 c p.
an initial solution = for a given problem sampled from the dataset distribution P.. If the solution is incorrect, the game assigns helpful critic identify and highlight the er- ror in the solution; conversely. if the solution is correct, the game assigns a misleading critic to fabricate non-existent error in the solution, attempting to mislead the prover into revising the correct answer to an incorrect one. Upon re- ceiving critiques whose intent is unknown, the prover must decide whether to modify the solution by reassessing the highlighted errors in its reasoning process. The data col- lection procedure for single Sitfie-Disemmmmett Game is described in Algorithm 1. The prompt templates fnelp- fmisteut. frevise used in the game are provided in Appendix A. prover can win the game two ways: (1) by providing
(I, e)),
For the helpful critic, the condition for winning is to con- vince the prover of its critique and guide in successfully correcting its initial incorrect solution. The reward for the helpful critic is assigned as follows:
(2', .V]. R -BSNT.ENN [[1 cluuee cumeeaa)
(2)
For the misleading critic, the condition for winning is to suc- cessfully deceive the prover into altering a correct solution
3

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
based on its critique. The reward is assigned as follows:
Algorithm 2 Self-play of Critie-Discemment Game
Inputs: inital prover T1, initial helpful critic μl, ini- tial misleading critic Pi. iteration number for self- improvemen T
Rp = -Berr Cooooce (2, v) Loorrec (2', y)]]
(3)
for iteration t == 1 to T do Collect self-play episodes T. {i ~ The x x pr) Select D D Dr from Tt
Each role in the game is trained to maximize its own reward. The prover and the helpful critic work collaboratively, while the prover competes against the misleading critic. During the training process, we jointly optimize the strategies of these models through reinforcement learning.
if > then
D - D. UD
D T Dr UD
+ D Tu
D' Ta
UD
3.2. Reinforcement Learning from Self-play
end if
Minimize the following objective on лi to obtain 7y41:
To enhance the capabilities of each role in the game, we employ reinforcement learning (RL) for training. The prover model is optimized to maximize the expected reward value:
CRESTCDG (T) = ED; (= zn)
Minimize the following objective on in to obtain 14411
CREST-CDO (p) ED; [4LM (z En]]
(RL-CDO(T) = -B (z [=))]]
(4)
Minimize the following objective on pr to obtain Pr41
CREST.CDG (p) -Epp (1p: =p]]
where 2x denotes the output of the prover given input Va--
end for
Several mainstream reinforcement learning algorithms can be employed to optimize the above objective, including Proximal Policy Optimization (PPO), Direct Preference Op- timization (DPO), and Reinforced Self- Self-Training (ReST). Our experiments show that among these methods, ReST exhibits stable performance and considerable effectiveness in our critic-discermment game. Therefore, we adopt ReST as the training method for CDG agents. With a threshold the
rate exceeding Tp. Similarly, a helpful critique is included in Drp only if it helps the prover correct an initially incorrect solution with success rate exceeding Ty-
Considering that the feature of multi-turn auto-regressin generation by L.LMs is inefficient for on-policy RL training, we adopt an offline learning scheme for multi-round training. For the (-th training round, we first collect self-play episodes and update the datasets for training each role as D D4-1 U Dr model by minimizing Equations 5, 6 and 7, and use it as the initialization for the next round of self-play, More details of our self self-play strategy are provided in Appendix B. The overall training algorithm is described in Algorithm
E R. ReST updates the LLM by the reinforcement selected samples Dr, = { 2x) r( Fxx) > }- The corresponding loss function can be written as:
Trote" Subsequently,
update the initial policy
(RESTCCD(T)) = E [( (LM((T;
= ED,,, [LIM( )=)]
(5)
where (I 2x) denotes the language modeling loss on the self-generated data.
4. Experiments
The loss function for critic and can be defined as:
To evaluate the effectiveness of CDG, we conduct experi- ments on four mathematics-related reasoning tasks using the fully fine-tuned LLaMA3.1--8 Instrrctt model. These tasks are designed to comprehensively assess the model's rationality in its reasoning process. Notably, LLaMA3.11 8B-Instruct has already undergone self-improvement using techniques such as rejection sampling and Monte Carlo Tree Search with reward guidance for mathematical tasks (Dubey et al., 2024). Therefore, performance improvements ob- served from training with CDG demonstrate its ability to further enhance the model's reasoning rationality on the foundation of traditional relf-improvement methods.
(RGST-CDG()) = ED (LM( zp)] (RSSTCDG(H)) ED,,, ( zn)]-
(6)
In our experiments, the values of Tx, Tp- and Tp are set to 0.5, 0.75, and 0.5, respectively. According to equation 1, the samples included in Dr, for reinforcement are: (1) cases where the prover provides a correct solution on the first attempt; (2) cases where the prover successfully re- sists misleading critiques; and (3) cases where the prover successfully corrects the initial incorrect solution with the assistance of the helpful critic. For the critics, each critique is presented to the prover in the form of frevise (z, 2, c), and the prover is required to generate multiple responses through random sampling. misleading critique is included DT, only if it successfully misleads the prover with success
4.1. Improving Mathematical Reasoning with CDG
In this section, we evaluate the mathematical reasoning capabilities of various models under the CDG framework.

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Table Math reasoning performance of CDG, which consistently enhances the reasoning ability of each model (p-value < 0.05). CDG-n
represents the prover model trained after the r-th iteration, while
refers to the model trained using imitation learning.
GSM8K
MATH500
Models
Methods
P@l
M@8
M@32
P@1
M@s
M M@32
Instruction-tuned Models
Llama-3.1 3.1-8B-Insstn
None
85.3
91.2
93.0
49.4
58.3
63.4
CDG-1
85.5
91.1
92.7
49.0
56.8
62.2
CDG-2
86.8
92.0
93.1
51.7
60.7
66.0
Owen2.5-1.5B-Instruu
None
75.1
82.6
86.5
55.4
58.6
62.9
CDG-1
75.4
83.8
86.4
57.6
62.9
66.2
CDG-2
75.2
83.8
86.8
56.1
61.5
65.6
Pre-trained Model
+ CDG-Imitation
78.9
87.1
90.4
29.4
36.5
42.6
CDG-1
78.4
86.4
89.3
29.8
36.3
40.4
CDG-2
79.2
88.7
90.8
33.8
39.2
43.6
Datasets and Evaluation Metries We focus on the field of mathematical reasoning using two widely used datasets: GSM8K and MATH500 (Cobbe et al., 2021a:b). The train- ing sets contain 7,473 and 12,000 samples, respectively, while the test consist of 1,319 and 500 samples. The problem- problem-answer answer pairs for self-play training are sourced from their respective training sets. We evaluate all models using greedy sampling (Pass @ I) and majority voting with 8 and 32 samples (M@8, M@32).
Results As shown in Table 1, CDG training consistently enhances mathematical reasoning capabilities across all models and datasets. In particular, for both Llama-3. 8B-Instruct and Llama-3. 1-8--BBB, the second round of training significantly improves the prover's reasoning perfor- mance. Case study on generated episodes reveals that this improvement is primarily due to the stronger attacks from the RL-trained misleading critic, which forces the prover to gain a deeper understanding of its reasoning process during the second training round, as shown in Appendix D. Fur- thermore, our method achieves larger improvements on the MATH500 dataset, highlighting CDG's ability enhance the models' performance more challenging problems.
To investigate the impact of self-play on different sizes of models, we select two instruction-tuned models:
Backbone Models
(Dubey et al., 2024) and Qwen2.5-1 1.5525--1.BBBnnttrucc (Yang et al., 2024a) as backbone models. Additionally, explore whether self-play can im- prove the performance of pre-trained models that have not undergone instruction tuning, we also conduct experiments on Llama-3. 3.1.aa3331--0 3.1-8B-BB -B--Bss..
4.2. Enhancing Stepwise Error Detection in the Reasoning Process with CDG
Task
We evaluate the model's understanding of its reasoning by testing its ability to identify potential errors in its reasoning steps. Concretely, the model receives ques- tion and solution and is tasked with examining a specified step to determine its correctness. Notably, we evaluate the model on problems it has already demonstrated proficiency in solving, ensuring that the assessment focuses solely on its ability to recognize errors in the reasoning process.
CDG Training For instruction-tune models, we follow Algorithm 2 and perform two rounds of self-play training. For dataset D,, selected during training, we ensure that the number of samples where the prover correctly answer on the first attempt, successfully resist misleading critiques, and correctly revise errors are balanced at 10,000 each. For pre- trained models, we introduce an additional imitation learn- ing step before self-play establish basic game-playing capabilities and ensure outputs following the self-play for- mat. The imitation learning dataset, consisting of 30,000 examples, is generated by -aama-3.1-8B-mmm ations are conducted on the prover model. Further details in CDG training can be found Appendix
Test Data Construction To ensure that the model's step- wise error detection ability evaluated on problems it is capable of solving, include only problems for which the majority of sampled solutions are correct in the test set. We then employ GPT-4o to identify the first erroneous step in the incorrect solutions for these problems. Through this process, we construct samples consisting solutions paired with their corresponding erroneous steps. Similarly, we ran-
All evalu-
5

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
domly sample one step from the model's correct solutions to create samples where each of the solutions is paired with correct step. The final dataset contains 200 positive samples and 200 negative samples.
answers are modified erroneously during self-correction, while maintaining nearly unchanged rate of correcting incorrect answers successfully, even when the initial re- sponse accuracy generated by CDG is higher. In GSM8K, the model trained with CDG reduces the probability of erro- neously modifying correct responses to less than half that of the original model. general, CDG training maintains positive and improved self-correction rate compared to the vanilla model. These results suggest that the model achieves more rational evaluation of its reasoning process after CDG training.
Results Table 2 presents the accuracy and F1 score for the stepwise error detection task. The results show that CDG training significantly improves error detection in reasoning, particularly on the more challenging MATH500 dataset, enhancing the model'ss self-reflection ability.
Table Performance of CDG stepwise error detection.
GSM8K
MATH500
Base CDG
Base CDG
2
12
Models
20
F
Acc
Fi
Acc
Parcentage
45
Llama-3.1-88 1amma-3.-8B-Inntruue
74.0
64.4
64.4
55.4
30
CDG
76.9
69.3
71,4
67.5
6SM8k
MATH
6SM8K
MATH
(a) Initial Response Accuracy
(b) Correct-to-Incoreet Rate
4.3. Improving Self-Correction with CDG
Task To further investigate whether CDG Training can enhance model's self-awareness of its reasoning steps and improve the rationality of its reasoning process, we evaluate 1-anaa..--B-Inseee on the Self-Correction task. The task structured as follows: the model first generates an initial response to a given question. Subsequently, the user prompts the model with the command, "Please check your answer step by step again", instructing the model to review and correct its initial response. Previous studies have indicated that models exhibit very limited self-correction abilities without external feedback (Kamoi et al., 2024; Pan et al., 2023). They tend to misinterpret their reasoning process and mistakenly modify initially correct answers, resulting in degraded final response.
25
220
1.0
0.5
Purcentage
Percentage
0.0
30
0.5
-1.0
-15
CDG
6SMak
MATH
6SHak
MATH
(c) Incorrect-to-Cone CorCert Rate
(d) Overall Correction Rate
Figure 2. Self-Correction results before and after CDG training.
4.4. Improving Long-Chain Reasoning with CDG
Task OpenAl-ol demonstrates that long-chain reasoning, which involves extensive exploration, self-reflection and self-correction, can significantly enhance the ability to solve complex reasoning problems by increasing test-time com- pute (Wu et al., 2024a; Zhao et al., 2024), which similar to how humans tackle difficult problems. However, such reasoning requires the L.M have strong understand- ing of its reasoning process, enabling it to identify when to correct errors and adjust its exploration direction. We test LLM performance on challenging problems from MATH500, focusing on level difficulty, under the long- chain reasoning paradigm.
Self-Correction Training To equip the model with foun- dational self-correction capabilities, we fine-tune Llama3.1- 8B-Instrect both before and after CDG training identi- cal datasets. Specifically, we use the vanilla Llama3. 1-BB- Instruct to generate eight responses for each question in the training set and then label them as correct or incorrect based on the ground truth. Following this, the model conducts a second round of self-checking and correction to improve its initial responses. We construct a self-correction training set consisting of two types of examples: 5,000 cases where the model's 's initial correct answers maintained after flection in the second round, and 5,000 cases where initial erroncous answers are corrected to accurate ones. This self- correction dataset is then used to fine-tune the model. We provide more details in Appendix B.
Long-Chain Reasoning Training To endow the model with long-chain reasoning capabilities, distill the rea- soning process of QwQ-32B- Preview (Team, 2024) into Llama3.1-880
both before and after CDG training. SwQ--2B-Previin is an open-source LLM renowned for solving problems through extensive reasoning, question- ing, and reflection, with an average reasoning chain length
Results As illustrated in Figure 2, the model trained by CDG significantly reduces the rate at which initial correct
6

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Table Performance comparison of other RI. methods and variants of CDG.
Math Reasoning
Error Detection
Self Correction
Long-chain Reasoning
Method
Average
GSM8K
MATH GSM8K MATH
GSM8K MATH
Llama-3.1-88 1lama-3.1-8B-Intttee --ss-tutt
85.3
49.4
74.0
64.4
-1.5
1.0
25.3
42.5
Other RL Methods
Expert Iteration
87.2
50.8
75.4
67.4
-0.6
0.8
22.8
43.4
Step-DPO
84.6
51.6
69.8
58.2
-2.2
-2.1
27.6
41.0
CDG and its variants
CDG
86.8
51.7
76.9
71.4
0.7
1.4
29.7
45.5
CDG w/o Correct critic
86.2
50.9
76.2
69.7
-0.5
-3.0
28.3
43,9
CDG w/o Misleading critic
84.9
51.1
76.2
68.9
-1.2
-0.5
29.4
44.1
significantly exceeding that of Llama3.1 1-8.1111---Innttte
Directly distilling the reasoning chains of QwQ-32B- Preview into 1-8mma331-8---sttteeee however, results in un- stable performance due to the capability gap between the two models. mitigate this, we first perform supervised fine-tuning (SFT) on Llama3.1 1-B-...tttt-t-ttr using reasoning chains generated by DwQ-32B-Previere Subsequently, we apply rejection sampling based on ground truth answers to filter the reasoning chains generated by Llama3. 1-8B- Instruct POst-SFT. Compared to direct distillation, this fil- tered dataset is more suitable for enhancing the capabilities of Llama3. 1 -ns-rutt We then use this dataset to train both pre- and Post-CDG training versions of the model for Long-Chain Reasoning tests.
(armaa.1-Bb-innteee Saamaa.1-Bb-inttee Distill
45
Distill
-40
35
30
2
samples
Figure 3. Performance on Level-5 Difficulty of MATH. The x- axis represents the number of samples, while the y-axis shows the accuracy under majority voting. Models trained with CDG demonstrate significantly better long-chain reasoning capabilities compared to models without CDG training, with p-value < 0.05.
Results As shown in Figure 3, the two-stage distillation significantly enhances the original model's performance on challenging mathematical problems, achieving nearly 10-point improvement. Additionally, models trained with CDG demonstrate superior long-chain reasoning capabilities compared to the original model when distilled on the same dataset, showing consistent 3-5 point improvement, This indicates that CDG training enables models develop better long-chain reasoning abilities, reflecting more rational understanding of their own reasoning steps.
sively responses that yield correct final answers for two iterations under the same training budget as CDG. StepDPO improves reasoming by using GPT-4o (Hurst et al., 2024) to generate step-level pairwise preference dataset, which is then used for training via DPO. We follow the experimen- setup from the original paper and conduct training on LLaMA Instrrcttt using the same dataset.
As shown in Table 3, Expert Iteration, which relies solely on outcome rewards, significantly improves performance on the Math Reasoning task but performs poorly on Long-chain Reasoning. In contrast, StepDPO, which applies step-wise supervision, enhances Long-chain Reasoning but underper- forms Error Detection and Self- Celf-Correction. Compared to these methods, our approach consistently enhances perfor- mance over the vanilla model across all four tasks, achiev- ing superior results in Error Detection, Self- Seff-Correction, and Long-chain Reasoning, while performing on par with Expert Iteration in Math Reasoning. These results demonstrate the effectiveness of CDG training in improving the rationality of the reasoning process.
5. Analysis of the Critie-Discermment Game
5.1. Comparing With Other RL Methods
To better assess the effectiveness CDG training enhanc- ing model capabilities, we compare with two mainstream RL approaches: Expert Iteration (Anthony al., 2017) and StepDPO (Lai et al., 2024). Expert Iteration is one of the most effective RL-based methods for improving reason- ing in LLMs (Havrilla et al., 2024). It fine-tunes models on high-return responses using standard cross-entropp loss. Following Havrilla et (2024), we train the model exclu-

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
5.2. Component Analysis
Table 4. Performance of CDG with various reinforcement learning methods on math reasoning.
We conduct experiments on two variants of CDG training to investigate the key factors contributing to the prover's s improvement in our DittieDDiscrnmmet Game. In these variants, either only the helpful critic only the mislead- ing critic interacts with the prover, while maintaining the same objective as in the vanilla CDG training. As shown in Table 3, the two variants of CDG training perform worse across all tasks compared the full CDG training. Notably, these variants even lead to decline in self-correction ability compared to the vanilla Llama3.1-3 1-BB- I-Innstruc.. This sug- gests that the prover, when trained in single-critic setting, may become either overly reliant on the critic or completely disregard it. Such tendencies may hinder the prover from genuinely understanding its reasoning process. Therefore, the effectiveness of CDG training stems not only from the ability correct initial errors resist misleading critiques, but also from the prover's ability discern the true quality of its reasoning process.
GSM8K
MATH500
Models
P@
M@ 32
P@
M@ 32
Llama-3. Gamma-3.1-B-Imme 1-8----ttttt
85.3
93.0
49.4
63.4
CDG-ReST
86.8
93.1
51.7
66.0
CDG-DPO
83.3
92.5
46.0
54.8
CDG-PPO
86.6
92.9
51.6
62.6
14.00
30
Critle1
12.5
13.7
13.75
Critic-1
24.0
56.3
28
13.50
13.25
24
13.00
Critt-2
Cridle-2
12.75
22
12.50
Prover-1
Prover-2
Prover-1
Prover-2
Correct Initial
Resist Misleading Critiques
5.3. Self-play Training with Various RL Methods
Figure 4. Game results on the Critic- Bitte-Disceemment Game. Left: Average win rales of prover models with different training stages when collaborating with critic models. Right: Average win rates of prover models resisting critiques from the mislcading critic.
We also investigate the impact of applying different RL algorithms in CDG training. Specifically, experiment with two mainstream RL methods: Direct Preference Opti- mization (DPO) (Rafailov et al., 2024) and Proximal Policy Optimization (PPO) (Schulman et al., 2017). All three al- gorithms are trained using the same set of collected game episodes. To construct pairwise preference data for DPO, generate responses the same prompt by randomly pling outputs with rewards of I (chosen) and 0 (rejected) for the prover. For PPO. we follow the experimental setup of SPAG (Cheng et al., 2024b), employing the same method for advantage estimation. Additional training details for DPO and PPO are provided in Appendix B.
iments where the prover from different training iterations interacts with helpful critics and misleading critics from different training iterations.
As shown in Figure 4, as training progresses, both the prover and the helpful critic improve their ability identify and correct errors, achieving their best performance in the final training iteration. This aligns with their cooperative learning objectives. Additionally, we observe that the prover in loop 2 is most effective at resisting the misleading critic in loop 1, while the prover in loop 1 performs the worst against the misleading critic in loop 2. This result consistent with their adversarial optimization objectives, where the prover and the misleading critic compete against each other. We can also find that if the misleading critic is not sufficiently trained, the prover in the second training loop can easily resist it, with the win rate increasing from 24.0 to 56.3, This is because the initial misleading critic lacks sufficient game-playing capability, which aligns with our observation that a single round of CDG provides limited or even no improvement in math reasoning as shown Table 1.
Table 4 presents the performance of CDG with various RL methods math reasoning. ReST consistently achieves the best performance under both greedy decoding and major- ity voting settings. In contrast, DPO even underperforms the vanilla LLaMA-3. 1-8BB-1sttt-ncc model. PPO improves reasoning performance in the greedy decoding setting but degrades performance under majority voting. This degrada- tion may be attributed to the sensitivity of these algorithms to hyperparameter choices (Havrilla et al., 2024). Com- pared to these methods, the ReST method achieves more stable performance improvements while requiring less GPU memory, as it eliminates the need for a reference model.
5.4. Game-play Performance
6. Conclusion
Besides evaluating the mathematical reasoning abilities of LLMs, we also assess the model's performance in the game by analyzing its win rate on the test set. We conduct exper-
We introduce a novel training strategy: Self-play learning in the rationality of LLMs the reasoning process. In our frame- work, prover is challenged by critiques of its solution,
Game, to effectively enhance the
8

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
which may either be helpful or misleading. The task of the prover is rationally deciding whether revise their re- sponse based on the critiques. Through multiple epochs of self-play and reinforcement learning with ReST, we observe that L.MSs exhibit consistent improvements in reasoning performance across various tasks, demonstrating a deeper understanding their own reasoning processes. We believe that our method highlights the potential of self-play lan- guage games with RI as promising new training paradigm for enhancing the reasoning capabilities of LLMs.
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv: 2100. 14168, 2021b.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems, 2021. URL https:/arxzi, org/abx/2110.14166 2021c
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:440.211783 2024.
Impact Statement
This paper presents work aimed at advancing the field of reasoning with large language models. As with all research involving large language models, there are inherent risks, including the spread of misinformation and the propagation of societal biases.
Fu, Y., Peng, H., Khot, T,, and Lapata, M. Improv- ing language model negotiation with self-play and in- context learning from ai feedback. arXiv preprint SSKiv:2305.00144 2023.
Gulcehre, C., Paine, T. L.,, Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., al. Reinforced self-training (rest) for language modeling. arXiv preprint X3vv:330800899. 2023.
Acknowledgements
We want to thank all the anonymous reviewers for their valuable comments. This work was supported by the Na- tional Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488), the Young Elite Scientists Spon- sorship Program by CAST (2023QNRC001), and the Pri- ority Academic Program Development of Jiangsu Higher Education Institutions.
Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang. P., Bi. X., et al. Deepseck-rl: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.122 /29488 2025.
Havrilla, A.. Du, Y., Raparthy, S. C., Nalmpantis, C., Dwivedi-Yu, J., Zhuravinskyi, M., Hambro, E., Sukhbaatar, S., and Raileanu, R. Teaching large language models to reason with reinforcement learning. arXiv preprint arXiv:2443.04642. 2024.
References
Anthony, T., Tian, Z., and Barber, D. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Huang. Z., Zou, H., Li, X., Liu, Y., Zheng, Y., Chern, E., Xia, S., Qin, Y., Yuan, W., and Liu, P. Ol replication journey -arrt 2: Surpassing ol -p-eeviw through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:241 1. 164489 2024.
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self-play fine-tuning converts weak language models to strong lan- guage models. arXiv preprint arXiv:2401.01333, 2024.
Cheng, J., Liu, X., Wang, C., Gu, X., Lu, Y., Zhang, D., Dong, Y., Tang, J., Wang, H., and Huang, M. Spar: Self- play with tree-search refinement to improve instruction- following in large language models. arXiv preprint arXiv:2412.1 1605, 2024a.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A.. Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.2122 2024.
Kamoi, R., Zhang, Y., Zhang, N., Han, J., and Zhang, R. When can Ilms actually correct their own mistakes? a critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics, 12:1417- 1440, 2024.
Cheng, P., Hu, T., Xu, H., Zhang, Z., Dai, Y., Han, L., and Du, N. Self-playing adversarial language game enhances Ilm reasoning, arXiv preprint 440i 20644 2024b.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 11.1110.114416 /4168, 2021a.
Kim, S., Joo, S. J., Kim, D., Jang, J., Ye, S., Shin, J., and Seo, M. The cot collection: Improving zero-shot and few- shot learning of language models via ofain-of-thougee fine-tuning. arXiv preprint arXiv:2305. /4045, 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Kirchner, J. H., Chen, Y., Edwards, H., Leike, J., McAleese, N., and Burda, Y. Prover-verifier games improve legibility of IIm outputs, arXiv preprint arXiv:2407.13692, 2024.
Qin, Y., Li, X., Zou, H., Liu, Y., Xia, S., Huang, Z., Ye, Y., Yuan, W., Liu, H., Li, Y., et al. 01 replication jour- ney: A strategic progress report-part 1. arXiv preprint arXiv:2410. 18982, 2024.
Lai, X., Tian, Z., Chen, Y. Yang, S., Peng, X., and Jia, J. Step-dpo: Step-wise preference optimiza- tion for long-chain reasoning of llms. arXiv preprint arXiv:2406.1882. 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Lanham, T., Chen, A., Radhakrishnan, A., Steiner, B., Deni- son, C., Hemaandez, D., Li, D., Durmus, E., Hubinger, E., Kernion, J., et al. Measuring faithfulness in chain- of of-thought reasoning. arXiv preprint 2x11v2307.13702. 2023.
Razeghi, Y., Logan IV, R. I.., Gardner, M., and Singh, S. Impact of pretraining term frequencies on few-shot rea- soning. arXiv preprint arXiv:2202.07206, 2022.
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I.,
Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
Lu, 7., Zhou, A., Wang. K., Ren, H., Shi, W., Pan, J., Zhan, M., and Li, H. Step-controlled dpo: Leveraging step- wise error for enhanced mathematical reasoning. arXiv preprint ar arXiv:2407.00782 2024.
Schulman, J., Wolski, F., Dhariwal, P.. Radford, A., and Klimov, O. Proximal policy optimization algorithms.
arXiv preprint arXiv: 2017.
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L.,, Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 550(7676):354-3555 2017.
Luo, L., Liu, Y., Liu, R., Phatale, S., Lara, H., Li, Y., Shu, L., Zhu, Y., Meng, L., Sun, J., et al. Improve mathemati- cal reasoning in language models by automated process supervision. arXiv preprint ar arXiv:2406.06553
2024.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Garcia, X., Liu, P. J., Harrison, J., Lee, J., Xu, K., et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585, 2023.
Luo, Y., Yang, Z., Meng, F., Li, Y., Zhou, J., and Zhang, Y. An empirical study of catastrophic forgetting large language models during continual fine-tuning. arXiv preprint ar 20v: 000..0777777777 2023.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference-freee reward. arXiv preprint arXiv:2405. 147344 2024.
Team, Q. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL pestps://qwen/ github. wttthuu..obbleee htlo//ll//////aee//////////eoo/o 32b beabb-preieww/
Meurer, A., Smith, C. P., Paprocki, M., Čertík, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, K., Singh, S., al. Sympy: symbolic computing python. PeerJ Computer Science, 3:e103, 2017.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving. G., and Higgins, I. Solv- ing math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14273 2022a.
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O., Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05229, 2024.
Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., and Higgins, I. Solv- ing math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14277 2022b.
OpenO1 Team. Open-ol. https: /github....ttt com/ www---uuuceeeeeeeeeee 11-2024].
2024. [Accessed 19-
Valmeekam, K., Olmo, A., Sreedharan, S., and Kambham- pati, S. Large language models still can't 't plan (a bench- mark for 11ms planning and reasoning about change). In NewrIPS 2022 Foundation Models for Decision Mak- ing Workshop. 2022.
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., and Wang, W. Y. Automatically correcting large lan- guage models: Surveying the landscape of diverse self- correction strategies. arXiv preprint Xiv:2308.00318.. 2023.
Wang. P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and reinforce
10

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Ilms step-by-step without human annotations. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2024.
Zhang, Z., Zheng, C., Wu, Y., Zhang, B., Lin, R., Yu, B., Liu, D., Zhou, J., and Lin, J. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07330 2025.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of- prompting elicits reasoning in large language models. Advances in neural information processing systems, 01224824-248887 2022.
Zhao, Y., Yin, H., Zeng, B., Wang, H., Shi, T., Lyu, C., Wang, L.,, Luo, W., and Zhang, K. Marco-ol: Towards open reasoning models for open-ended solutions. arXiv preprint arXiv:241 14405, 2024.
Zheng, R., Guo, H., Liu, Z.. Zhang, X., Yao, Y., Xu, X., Wang, Z., Xi, Z.. Gui, T., Zhang, Q. et al. Toward optimal Ilm alignments using two-player games, arXiv preprint SKKiv:2006.09777 2024.
Wu, S., Peng, Z., Du, X., Zheng, T., Liu, M., Wu, Ma, J., Li, Y., Yang, J., Zhou, W., et al. A comparative study reasoning patterns of openai's 's ol model. arXiv preprint arXiv:2410.1 13639, 2024a.
Wu, Y., Sun, Z. Yuan, H., Ji, K., Yang, Y., and Gu, Q- Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.0067 00675, 2024b.
Xi, 7., Yang, D., Huang, J., Tang, J., Li, G., Ding, Y., He, W., Hong, B., Do. S., Zhan, W., al. Enhancing I1m reasoning via critique models with test-time and training- time supervision. arXiv preprint 2x 2024.
Xu, Q. Tian, Z., Wu, H., Huang, Z., Song, Y., Liu, F., and Li. Learn disguise: Avoid refusal responses in Ilm's defense via multi-agent attacker-disgutte game, arXiv preprint ar wxxv::2444002332 2024.
Xu, Y., Wang. S., Li, P., Luo, F., Wang, X., Liu, W., and Liu, Y. Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.0m 044588688. 2023a.
Xu, Z., Yu, C., Fang, F., Wang, Y., and Wu, Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310. 21..:2990..940 2023b.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024a.
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., et al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint 2xxv:2409..11222 2024b.
Zhang, D., Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., and Tang, R Llm self-training via process reward guided tree search, 2024a. URL hutp::/arxiiv org/abs/2406.033 030861..6666666
Zhang. R., Xu, Z., Ma. C., Yu, C., Tu, W.-.,, Huang, S., Ye, D., Ding, W., Yang, Y., and Wang. Y. A survey on self- play methods in reinforcement learning. arXiv preprint arXiv:2408.00 01072, 2024.

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
A. Data Collection Details
Figure 5, Figure 6, and Figure 7 illustrate prompt templates used the Critic-Discernne Game (CDG). To ensure consistency in the critique styles generated by the misleading critic and the helpful critic to prevent potential reward hacking, we employ identical prompts for both types crities. During the process of collecting episodes for self-pla play training, both the prover and the critic generate responses with temperature of 0.95, maximum length of 4096, a top-p of 0.95, and a top-k of 5.
Self-play CDG
Question Template
User:
Solve the following math problem efficiently and clearly:
Regardless of the approach, always conclude with: Therefore, the final answer is: $ bboxe hope it is correct. Where [answer] is just the final number expression that solves the problem. Here is the question you need to answer: #### Question:
{question)
Assistant:
Figure Prompt template for answering mathematical questions.
Critic Template
User: Please critic the answer carefully.
Your task is to evaluate question-answer pair.
Carefully review the question and critically assess wrong answer. [Question]
(question}
[Wrong Answer]
{solution]
Please review the wrong answer step by step, quoting each sentence and analyzing it individually. Use the following format for your response:
Step: [Quoted Sentence]
Analysis: Youur Explanation]
Step: [Quoted Sentence]
Analysis: [Your Explanation]
After step-by-step analysis, conclude by quoting the original sentence which **first** causes the wrong answer and providing a concise yet complete explanation of the error:
"**Critic** The first mistake can be found in: 'Quoted wrong statement here." The issue is: "Explanation the mistake here."
Assistant:
Figure 6. Prompt template for generating critics.
Step-wise Error Detection Figure 8 presents the prompt used to annotate erroneous steps with gpt-4o. If the generated solution is not explicitly divided into steps, we segment it based on "\n \n". Figure 9 illustrates the prompt designed to evaluate the model's stepwise emrrr detection capability.
Self-Correction
Figure 10 illustrates the prompt used for self-correction..
12

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Revising Template
User: {question)
Assistant:
{initial solution]
Please check with this critic. "[critic]" Evaluate whether this critic is valid. Keep in mind that this critic might be mislcading or irrelevant. If you find the critic incorrect, conclude with: "(boxed (his critic is not ) If you determine the critic to be valid, revise starting from the incorrect step, and present your revised answer within \boxed().
User:
Assistant:
Figure 7. Prompt template for revising answers based on critiques.
Error Detection Template (gpt-4o)
User: Question:
{question
Wrong Response:
(solution ion) Identify the first step that contains an error. Please provide the step number the format: Step \boxed
Assistant:
Figure Prompt template for detecting stepwise error by gpt-4o.
Error Detection Template
User:
Solve the following math problem efficiently and clearly:
Regardless of the approach, always conclude with: Therefore, the final answer is: booedd/anseerr I hope it is correct. Where [answer] is just the final number or expression that solves the problem. Here is the question you need to answer: #### Question: (question)
Assistant:
Figure 9, Prompt template for evaluating stepwise error detection.
Self-Correctoo Template
User: {question)
Assistant: (initial response]
User: Please check your answer step by step again. Put your final answer within (boxed]). Assistant:
Figure 10. Prompt template for self correction.
13

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
B. Self-play training Details
To generate game episodes for reinforcement learning (RL) training, we require the prover to generate four initial responses upon receiving a math problem. Then, the helpful critic generates eight critiques for each question-solutin pair, while the misleading critic generates four critiques for each question-solutio pair. Subsequently, for each triplet of (question, solution, critique), the prover produces four responses.
Training CDG with ReST When constructing the dataset D,, for ReST training, we determine correctness using regular expressions and the SymPy grader (Meurer et al., 2017). A prover's response is considered correct either it provides the correct solution in the first round or successfully correct an initially incorrect solution in the second round. To evaluate whether the prover successfully resists misleading critiques, we check whether the model outputs "1 "(boxee This critic not critical.)".
For constructing : helpful critic's critique is included only if the corresponding prover generates at least two correct responses for the given (question, solution, critique) triplet. For constructing Dr,- a misleading critic's critique is included only if the corresponding prover fails to output " boxed (Thss critic is not critical. }" and changes a previously correct answer into an incorrect one in at least three responses for the same (question, solution, critique) triplet.
In the first training loop, we use a learning rate of 5e-6 and batch size of 32 to facilitate rapid convergence. In the second loop, as the dataset size increases, we adjust the learning rate to 1e-6 and the batch size to 256. The prover and misleading critic are trained for one epoch, while the helpful critic is trained for two epochs.
Training CDG with DPO In Section 5, we also explore training the Diste-DDisemmmnttt Game (CDG) using Direct Preference Optimization (DPO) (Rafailov et al., 2024) in a self-play setting. To construct pairwise preference data for training the prover, we select one successful and one unsuccessful response from the prover's generated outputs for each triplet of (question, solution, critique).
We use learning rate of 1e-6, a batch size of 64, and set 3 = 0.5 to control the deviation from the base reference policy. We observe that DPO training for CDG is highly sensitive to hyperparameter selection and often results in repetitive text generation.
Training CDG with PPO In Section 5, we also explore training CDG through Self-Play using PPO (Schulman et al., 2017). For each episode, assign rewards according to Equations 5, 6, and 7. We then follow the PPO training framework used in SPAG (Cheng et al., 2024b) to conduct self-play training.
We set the learning rate to 5e-6 and the batch size to 128. To control the deviation from the base reference policy, we set 3 = 0.2. Additionally, to ensure stability when answering questions on the first attempt, we incorporate a supervised fine-tuning loss with weighting coefficient of 0.5.
C. Experiments
In this section, we provide additional details regarding our experimental setup and evaluation.
C.1. Mathematical Reasoning
For mathematical reasoning tasks, we GSM8K (Cobbe et al., 2021a) and MATH500 (Lightman al., 2023) as our test datasets, which contain 1,319 and 500 examples, respectively. Under the Pass@) I setting, we generate solutions using greedy decoding. In the majority voting setting, we set the maximum sequence length to 8,192, the temperature to 0.95, and use top-k = 10 for sampling-based generation.
C.2. Stepwise Error Detection
For the Stepwise Error Detection task, we first generate eight solutions for each test sample using the Llama-3. B.taaa...--BBBBB I-SB-Instruct model with temperature of 0.95. We include only problems for which the majority of sampled solutions are correct the test set. When annotating stepwise error with GPT-4o, we use greedy decoding and filter out examples with formatting errors. We use greedy decoding during evaluation.
14

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
C.3. Self-Correction
Compared Stepwise Error Detection, Self-Correction is more challenging task, it requires the model not only first collect solutions
identify emrors but also to correctly revise them. To construct the training dataset for self-correction generated by Llama-3.1-8) --Sttutntttt with temperature of 0.95 the training set. Following this, the model performs second round of self-checking and correction to refine its initial responses. We select cases where the final answer is correct for training. Additionally, we remove data instances where the self-checking process contains repetitive loops.
Referring to Algorithm 2, our final CDG model is obtained by fine-tuning the original Llama-3.1 1-ttt...--BBBnm ---tt-Innttrctttt on D Subsequently, we perform another fine-tuning step on the D self-cooreett dataset. During our experiments, we observed that sequential fine-tuning on two different distributions can lead catastrophic forgetting (Luo et al., 2023). To mitigate this issue, directly fine-tune the original Llama-3.1-88 1-8--.1-----nntuuee Instruct the combined dataset 'D? cellff For the baselines presented in Table 3, we apply the same fine-tuning strategy to ensure a fair comparison. We set the learning rate to 1e-6 and the batch size to 256.
During testing, we first generate solutions using the model with a temperature of 0.95 on the test set. The model is then required to perform a second round of self-checking and correction on the initial solutions using greedy decoding.
C.4. Long-Chain Reasoning
In Section 4.4, we introduce our two-stage fine-tuning approach for enhancing long-chain reasoning capabilities. Specifically, we utilize QwQ-32B-Prrn B-errveww (Team, 2024) to generate solutions on the training set with a temperature 0.95, selecting only those with correct final answers for fine-tuning. Additionally, we filter out examples containing repetitive loops during generation. Similarly, Llama-3.1-88 --Sttnttttt post-SFT generates solutions with a temperature of 0.95, and only those with correct final answers are selected for fine-tuning, while filtering out instances with repetitive loops.
For training long-chain reasoning capabilities in CDG and other methods, we follow the same approach as in the Self- Correction task and merge the training datasets. The learning rate for both fine-tuning stages is set to 1e-6, and the batch size is 256.
D. Self-play Examples
We present examples of the first and second rounds where the prover successfully corrects the initial error, successfully resists the misleading critique, and fails to resist the misleading critique in Figures 11, 12, 13, 14, 15, 16, and 17.
15

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Self-play in loopl (Prover successfully corrects the initial error)
User: Determine the number of ways to arrange the letters of the word THAT.
determine the number ways arrange letters of the word THAT, we can use the concept of permutations. Since the word has no repeated letters, the total number of permutations can be calculated using the factorial of the number of letters.
Prover:
The word THAT has 4 letters (T-H-A-T). Therefore, the total number of permutations is 4! factorial).
4! 4 3 2 1 = 24 Therefore, the final answer is: 24
Assistant:
Please check with this critic.
"The first mistake can be found in: "Since the word has no repeated letters, the total number of permutations can calculated using the factorial of the number of letters."
The issue is: The presence absence of repeated letters is not the correct condition to determine whether to use the factorial formula. A correct condition is that the letters should be distinet. In the word "THAT," the letters are not distinct due to the presence of repeated Ts. This mistake leads to an incorrect application of the factorial formula, resulting in an incorrect final answer of 24
Evaluate whether this critic is valid. Keep in mind that this critic might be mislcading or irrelevant. If you find the critic incorrect, conclude with: "(boxed This critic is not critical.)"
If you determine the critic be valid, revise starting from the incorrect step, and present your revised answer within \boxed
The critic is pointing out that the condition for using the factorial formula is not the absence of re- peated letters, but the distinctness the letters. In the case of the word "THAT," the presence of two identical "T"'s makes the letters not distinct.
Prover:
Given that the letters in "THAT" are not distinct due to the repeated "T"ss, we need to adjust the calculation. When there are repeated elements, we divide the total number of permutations by the factorial of the number of repetitions for each element.
In "THAT," there are 2 "."s. The correct formula for permutations with repeated elements is: Total permutations (Number of letters)! / (Number of repetitions for each letter)! Total permutations 4! (2! for the repeated "T"') Now, calculate the total permutations:
= 24
2! 21=2x1
Total permutations 24/ 12
Therefore, the correct answer is:
12
This critic is not misleading; it is valid and correct.
Figure 11. Self-play examples of Llama3. I-B--nnttttt loop1 (Prover successfully corrects the initial error).
16

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Self-play in loop1 (Prover successfully resists the misleading critique)
User: The distance across country is 8205 kilometers. Amelia started at end and drove 907 kilometers Monday and 582 kilometers on Tuesday, How many kilometers does Amelia still have to drive to make it across the country?
Assistant: To find out how many kilometers Amelia still has to drive, we need to subtract the distance she has already driven from the total distance across the country. First, we need to find the total distance Amelia has driven so far.
She drove 907 kilometers Monday.
She drove 582 kilometers Tuesday.
So, the total distance she has driven is 907 + 582 1489 kilometers.
Now, we subtract the total distance Amelia has driven from the total distance across the country.
Total distance across the country = 8205 kilometers.
Distance driven so far = 1489 kilometers.
So, the distance Amelia still has to drive 8205 1489 = 6716 kilometers. Therefore, the final answer is: 6716 hope correct.
User: Please check with this critic.
"The first mistake can be found in: 'She drove 907 kilometers Monday.' The issue is: There is actually mistake in the given wrong answer. However, the original question does not state the order of the distances driven, so the answer assumes that the distances are in the order they are driven. If we assume that Amelia drove 582 kilometers on Monday and 907 kilometers on Tuesday, we get a different result. The correct total distance driven would be 582 907 = 1489 kilometers, but this would still be correct since 1489 8205. However, if we consider that the order of the distances could be different, then the answer is actually Evaluate whether this critic is valid. Keep in mind that this critic might be misleading irrelevant. If you find the critic incorrect, conclude with: ('booee(TThis critic is not critical.)"
wrong this case."
If you determine the critic to be valid, revise starting from the incorrect step, and present your revised answer within {)oxed().
The critic points out that the original solution assumes the order of the distances driven (Monday and Tuesday). However, the problem statement does not specify the order of the distances.
Assistant:
To address this concern, need consider both possibilities:
Amelia drove 907 kilometers on Monday and 582 kilometers on Tuesday.
Amelia drove 582 kilometers on Monday and 907 kilometers on Tuesday.
In both cases, the total distance driven would be the same: 1489 kilometers (582 + 907).
However, the critic's main point is that we should be cautious when assuming the order of the distances. But, in this case, it does not change the result. The original solution correct regardless the order of distances. Since the critic does not provide a valid reason to change the original answer, would say that:
This critic is not critical.
Figure 12. Self-play examples of Llama3.1 1.....1-BBB Instruct in loopl (Prover successfully resists the misleading critique).
17

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Self-play in loopl (Prover fails to resist the misleading critique)
User:
Find the arithmetic mean of the reciprocals of the first three prime numbers.
Assistant: To find the arithmetic mean of the reciprocals of the first three prime numbers, we need to follow these steps:
1. Identify the first three prime numbers, which are 2, 3, and 5.
2. Find the reciprocals of these prime numbers, which are 2. 3 and _ respectively.
3. Add these reciprocals together: + + 5'
4. Find a common denominator, which in this case is 30. So, the sum becomes:
+ 30 30 30
5. Divide the sum by find the arithmetic mean:
30 3 90
31
31
Therefore, the final answer is:
90
User: Please check with this critic.
31
"The first mistake can be found in: 'Therefore, the final answer is:
The issue is: The final answer incorrect
90
because is the reciprocal of the sum of the fractions, which is not the arithmetic mean of the first three prime numbers." Evaluate whether this critic is valid. Keep in mind that this critic might be misleading or irrelevant.
If you find the critic incorrect, conclude with: "(boxed (This critic is not critical.)" If you determine the critic to be valid, revise starting from the incorrect step, and present your revised answer within \boxed(].
The critic is valid. The arithmetic mean of set of numbers is found by adding the numbers and dividing by the number of items. In this case, the arithmetic mean of the reciprocals of the first three prime numbers should be found by adding the reciprocals of 2, 3, and 5, and then dividing by 3.
Assistant:
To correct this, we need to revisit step 4 and then proceed to step 5.
1. Identify the first three prime numbers, which are 3, and
2. Find the reciprocals of these prime numbers, which are 2. 3. and respectively.
3. Add these reciprocals together:
+
4. Find common denominator, which in this case is 30. So, the sum becomes: 30 30 30
15
10 6
31
30
5. To find the arithmetic mean, we should divide the sum by the total number of items, which is 3. However, we can 1 and
directly take the sum, 30
31 then divide that by 3 to find the mean the 3 numbers 2 3 5.
The correct arithmetic mean should be found as:
90
However, finding the mean of the reciprocals of the first three prime numbers. Therefore, to get the actual mean of the reciprocals, we divide the sum by 3.
31 31 is correct in that the mean of the three fractions is 96
1 +
31 but to find the mean of which is 30
3 = 30 these 3 reciprocals, we must divide 30 by 3.
Therefore, the correct arithmetic mean of the reciprocals of the first three prime numbers is:
31
31
31
30
31
31
31
31
31
31
31
31
+
30
30
90
30
90
30 : 3
10
3
10
30
3
10
3 10
30
Figure 13. Self-play examples of Llama3.1 1-B. I-8B-Instruct in loop1 (Prover fails to resist the misleading critique).
18

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Self-play in loop2 (Prover successfully corrects the initial error)-partl
User: Let P(z) be polynomial of degree 2011 such that P(1) 0, P(2) 1, P(4) 2, ..... P(22011) 2011.
Then the coefficient of in P(a:) can be expressed in the form
a 6
where a, b, c are positive integers, and is prime. Find a + b c. Assistant: To find the coefficient of x in P(x), we will first define another polynomial Q(x) P(x) x. We note that Q(1) - (2) Q(4)
Q(22011) = 0, (xx) has roots at x 1, ..... 22011
Now, we want to find the coefficient of x in P(x) = (()) + x. Since (z) has roots at 1, 2,2, 4, 2011 we can write it in factored form as
Q(x) = C(x - 1² 2)(x 4) ..... ...... ... (x- 22011).
where C' is constant.
We want to find C in terms of Q(x). To do this, can the fact that Q(2) 1. Substituting 2 into the equation for (Ir), we get
((2) = C(2 - - 2) - 4)... (2-22011)
1 = C(-2)( ... ... (-2))
1
22011 |C|
Q(z) = (x 1)(x 2) ..... (x 22011)
22011
Now. we want to find the coefficient of x in P(x) = Q(=)) To do this, we can expand the factored form of ((x) and add it to I. The coefficient of r will be the sum of the coefficient of x in the expansion of Q(z) and 1. Since we only need the coefficient of x in (xx), we can ignore all the terms in the expansion that are higher degree than æ.
The only terms that will contribute the coefficient of in Qzx) are the terms that come from the product of (x 1). (x 2), (xr
22011). When we multiply these terms together, each term will have power of x: equal to the number of terms in the product. The term with the power of equal to I will come from the product of all the (x 2 terms where is odd. This term will have a coefficient equal to + II. is (-2")
..., (x ...
Since we have 2011 terms in the product, half of them will be odd and half of them will be even. Therefore, the product of all the (x - 2 terms will have a coefficient equal to +2 2011
Since we adding this term when add it the coefficient of in P(x) will be equal + 2013
Therefore, the coefficient of in P(æ) can be expressed in the form be where a = 1, b = 2, and 2011
Therefore, the final answer is: 2012 2 + 2 + 1005 hope is correct.
Figure 14, Self play examples of Llama3.1 1-8.-------nnteee Instruct in loop2 (Prover successfully corrects the initial error, part)).
19

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Self-play in loop2 (Prover successfully corrects the initial error)-part2
Please check with this critic. "The first mistake can be found in: "The only terms that will contribute the 22011)." The issue is: The statement that the coefficient of in the expansion of (zx) is equal to the product of (æ 1), (z 2), (x (x-4) .).. (x - 22011) is incorrect. This product cannot directly be used to find the coefficient of z because it is not polynomial with single term of degree I, but rather a product of multiple terms with various powers x." Evaluate whether this critic is valid. Keep in mind that this critic might misleading or irrelevant. If you find the critic incorrect, conclude with: " (boxed(This critic is not " If you determine the critic be valid, revise starting from the incorrect step, and present your revised answer within \boxedf).
User:
coefficient of in ((x) are the terms that come from the product of (æ: 1). (z 2). (z -)..... ...., (x
Assistant:
This critic is valid.
To correctly find the coefficient in P(x), we need to consider the actual expansion of Q(z).
We can write (xx) in expanded form as
2011
((x) = Σ
(-1)²
2011-
k=0
Now, we want to find the coefficient of æ in this expression. Since we are looking for the coefficient of x; we want to find the term with x 2011111 We can do this by setting = 2010.
The coefficient of in this expression is given by the term with 2010. We can simplify this expression follows:
2011
(-1) 2010 C
2011
2010
Now, we want to find the value of C. We can do this by setting x = 2 in the equation Q(+) = P(x) -2 and solving for C.
We get:
1 C (-1)3001
2010
(001
2010
2010
Simplifying this expression, we get:
3 C
(2110)
22010
Substituting this value of C back into the expression for the coefficient of get:
(2011)
(2011) 22010 (-1)a00 2010
2010
2010
22011
Therefore, the coefficient of P(x) can be expressed in the form
bc where a 1, b 2, and 2011.
Therefore, the final answer is: + 2 2011
Figure 15. Self-play examples of Llama3.1 1 -Bstructttt in loop2 (Prover successfully corrects the initial error, part2).
20

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Self-play in loop2 (Prover successfully resists the misleading critique)
User: Legacy has 5 bars of gold she received from her father. Her friend Aleena has bars fewer than she has. If bar of gold is worth $2200, calculate the total value of gold the three have together.
Assistant: Total Value of Gold
To solve this problem, we'll follow these steps:
1. Find out how many bars of gold Legacy has.
2. Determine how many bars of gold Aleena has.
3. Calculate the total number bars of gold both Legacy and Aleena have together.
4. Calculate the total value gold.
#### Step I: Find out how many bars of gold Legacy has. Legacy has bars of gold.
#### Step 2: Determine how many bars of gold Aleena has. Aleena has 2 bars fewer than Legacy, she has 5 2 bars of gold.
##### Step 3: Calculate the total number bars of gold both Legacy and Aleena have together.
Since there's no mention any other person having gold, we can assume the total value of gold we're looking for the value of gold both Legacy and Aleena have. Therefore, the total number of bars is 5
##### Step 4: Calculate the total value of gold. A bar of gold is worth $2200, so the total value of gold is 8 2200 S17600.
Therefore, the final answer is: 17600
User: Please check with this critic. "The first mistake can be found in: 'Aleena has bars fewer than she has." The issue is: This sentence should have been 'Aleena has 2 bars fewer than Legacy has' accurately represent the information given in the question. This error led to confusion about the number of bars Alcena has and ultimately resulted in an incorrect calculation of the total value of gold."" !" Evaluate whether this critic is valid. Keep in mind that this critic might be misleading or irrelevant.
If you find the critic incorrect, conclude with: ""(bxee TTiss critic is not critical.)" If you determine the critic to be valid, revise starting from the incorrect step, and present your revised answer within [boxed().
Assistant: To evaluate this critic, need to assess if the correction suggested is accurate.
The critic points out that the sentence should read I AAleena has 2 bars fewer than Legacy has" instead of 1 LAleena has bars fewer than she This change to accurately represent the information given in the question. The original statement was ambiguous, and the critic's suggested correction a logical one.
However, upon closer inspection, it becomes clear that the critic's suggested correction actually an unnecessary change. The original statement "Aleena has 2 bars fewer than she has" is still accurate, given that the subject of the sentence is "she," which refers to Aleena. The phrase "she has" refers to Aleena's possession, making the sentence clear.
Therefore, the critic's suggested correction does not address an actual issue in the original solution. The original solution was already correct.
This critic is not critical.
Figure Self play examples Llama3. --Bttrrtt loop2 (Prover successfully resists misleading critique).
21

---

Improving Rationality in the Reasoning Process of Language Models through Self-playing Game
Self-play in loop2 (Prover fails to resist the misleading critique)
User: A private company raised $2500 be used for charity. organizations. How much money will each organization receive?
They donated 80% to public foundation with 8
Assistant: To find the amount of money each organization will receive, we first need to determine the total amount donated to the public foundation. The company donated 80% $2500, which can be calculated as follows:
80% of $2500 (80/100) * $2500 $2000
This is the total amount donated to the public foundation.
Since the public foundation has 8 organizations, to find the amount each organization will receive, we divide the total donated amount by the number of organizations:
Amount each organization will receive Total donated amount Number of organizations $$2000 / $250
Therefore, the final answer is: 250
User: Please check with this critic.
"The first mistake can be found in: "Amount each organization will receive = Total donated amount / Number of organizations"
The issue is: The explanation incorrectly states that this formula is used to find the amount each organization will receive. The correct formula should be (Total donated amount Number of organizations) * (100% 80% simply the total percentage donated), which would be (2000 / 8) (100/80) $312.50"
Evaluate whether this critic is valid. Keep in mind that this critic might be misleading irrelevant. If you find the critic incorrect, conclude with: "(boxed critic is not critical.)"
If you determine the critic to be valid, revise starting from the incorrect step, and present your revised answer within (boxed().
The critic pointing out that the original explanation incorrectly states the formula for finding the amount each organization will receive. The critic suggests that the correct formula the product of the total donated amount divided by the number of organizations, multiplied by the inverse of the percentage donated (100% / 80%).
Assistant:
Let's break down the critic's explanation:
Amount each organization will receive = Total donated amount Number of organizations (100% Percentage donated) ($2000 / 8) (100/80) ($250) (1.25) $312.50
Upon evaluating the critic, it appears that the critic's 's point is valid. The original explanation was indeed incorrect in stating that the formula was used to find the amount each organization will receive.
Therefore, the revised answer is: 312.50
Figure 17. Self-play examples of Llama3.1-888 1-8ma33.--B-Isstruee in loop2 (Prover fails resist the misleading critique).
22


Reviews:
[{"review_id": "kFwaYe4STh", "reviewer": "Reviewer_CXvh", "summary": "The paper introduces a framework that involves training three models (prover, helpful critic, misleading critic) via reinforcement learning with the goal of improving the reasoning capability of the prover model. Through the proposed training process, the prover learns to rely only on helpful feedback, and to ignore misleading feedback, thereby gaining a better notion of correct vs. incorrect reasoning. The authors conduct experiments on two math tasks, and show some improvements over the base models.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your recognition of our idea and your constructive feedback. Here are our responses to your concerns:\n\n> **Claims And Evidence:**\n> The goal of the technique is unclear and the evidence provided to support this is not super convincing.\n> **Experimental Designs Or Analyses:**\n> The step-wise error detection experiment is done on problems that the model can fully solve.\n\nWe discuss these two points together, as our experimental design is closely aligned with the goal of the technique. As mentioned in the introduction, recent studies have shown that LLMs lack a true understanding of their reasoning processes and instead rely primarily on probabilistic pattern matching. Our goal is to alleviate this limitation through self-play training. However, performance on a single task (e.g., math problem solving) is insufficient to determine whether a model truly understands its reasoning process. Therefore, we evaluate the model across multiple dimensions to better assess whether it reasons in a rational manner. **As shown in Table 3, our method demonstrates clear and consistent improvements across all tasks, supporting our claim.**\n\nWe now clarify the motivation behind the stepwise error detection setting. Human experts, when they truly understand a solution, can identify and fix their own mistakes when such mistakes occasionally occur. Similarly, we focus on problems the model can usually solve correctly to control for knowledge or capability gaps.  If the model still fails to detect errors in such cases, it suggests a lack of understanding of its own reasoning process rather than a lack of ability. This design is well aligned with our stated goal. In contrast, Section 4.2 (self-correction) makes no such assumption and evaluates the model on general problems, where initial answers may or may not be correct—thus covering more realistic error-correction scenarios.\n\n>**Methods And Evaluation Criteria:**\n>Evaluation is done only on two math tasks, namely gsm8k and MATH. I would be more convinced of the proposed technique if it was applied to tasks where the initial performance is quite low.\n\nYes, with the rapid progress in reasoning models, GSM8K and MATH may no longer sufficiently evaluate mathematical capabilities. To address this, we train CDG on Dapo-17k [1], a harder dataset. Results on the **AIME-24** test set are shown below:\n\n| Model| Pass@1| Pass@4| Pass@8|\n|--|--|--|--|\n| Qwen2.5-1.5B-Instruct| 1.67| 6.67| 6.67|\n| + CDG-1| 2.67| 3.33| 3.33|\n| + CDG-2| 5.33| 6.67| 6.67|\n| + CDG-3| **5.67**| **10.0**| **16.7**|\n\nWe average over 10 runs due to the small test size. CDG-3 improves performance from 1.67 to 5.67 using only 18K self-generated examples. For comparison, Qwen2.5-1.5B-MATH-Instruct achieves 10.0 after 2.5M CoT-annotated examples and GRPO. This highlights CDG’s data efficiency and effectiveness on low-performing tasks.\n\nWe further evaluate the CDG-trained models on harder math (Minerva-MATH) and diverse reasoning tasks (LogiQA, ARC-Easy/Challenge), demonstrating the generality of our method:\n\n| Model| Minerva| LogiQA| ARC-E| ARC-C|\n|--|--|--|--|--|\n| Llama3.1-8B-Instruct| 31.24| 31.18| 79.76| 54.95|\n| + CDG| **33.10**| **32.26**| **80.05**| **55.80**|\n\n>The helpful and misleading critics are discarded after training. Have the authors using the same model but with different roles?\n\nYes, this is a very meaningful point. In fact, our initial design of CDG used a single model to play different roles via prompting, but we encountered two key issues:\n\n1. The model’s responses as helpful and misleading critics differ noticeably in style and length under different prompts, making it easy for the prover to distinguish between them.\n\n2. Training a unified model to generate misleading critiques can cause unintended side effects, such as hallucinations during regular problem solving.\n\nTo avoid these issues, we adopt three separate models with non-shared parameters. This setup yields more stable and reliable gains in the prover’s reasoning ability.\n\n> How hard was hyperparameter tuning for the ReST approach? Could you elaborate on stability of the training with ReST?\n\nThe ReST approach involves very few hyperparameters and does not require a reference or critic model. Unlike PPO or DPO, where improper values of $\\beta$ can lead to training collapse (e.g., duplicate or collapsed generations), ReST does not suffer from such instability. As for selecting the threshold $\\tau$, we manually select it based on data balance and training performance.\n\nReferences:\n[1] Yu Q, Zhang Z, Zhu R, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale[J]. arXiv preprint arXiv:2503.14476, 2025.", "claims_and_evidence": "- It is unclear to me what the goal of the technique really is. The authors claim it is about teaching the models to \"understand their reasoning process\". The evidence provided to support this is not super convincing. First, the performance gains are quite marginal (e.g., 0.3% boost on gsm8k), and sometimes performance drops from mroe CDG iterations.", "methods_and_evaluation_criteria": "- Evaluation is done only on two math tasks, namely gsm8k and MATH. GSM8K is a very easy task, with the initial models already achieving ~90%, which might make me think that this technique only works when the base model is already good on the tasks in question. I would be more convinced of the proposed technique if it was applied to tasks where the initial performance is quite low. \n\n- The proposed methods require sampling and training three different models, which can be quite expensive, and the helpful and misleading critics are discarded after training. Have the authors using the same model but with different roles?", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- The step-wise error detection experiment is done on problems that the model can fully solve. However, in practice, we want the model to still detect errors in its reasoning on unseen problems whether it can/cannot fully solve these problems. Overall, the experiment setup seems very artificial.", "relation_to_broader_scientific_literature": "- I think the paper's main contribution is RL-based training of self-play with three different models. The idea is neat and interesting, although the performance gains do not justify the complexity of the approach.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- What if you use the same model for the three roles?\n- How hard was hyperparameter tuning for the ReST approach? Could you elaborate on stability of the training with ReST?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lkTH483VA3", "reviewer": "Reviewer_1Lfc", "summary": "This paper introduces a self-play reinforcement learning approach called the Critic-Discernment Game (CDG) to improve language models' reasoning capabilities. In CDG, three roles interact: a prover provides solutions to problems, a helpful critic identifies genuine errors in incorrect solutions, and a misleading critic attempts to fabricate errors in correct solutions. The prover must learn to maintain correct answers against misleading critiques while revising genuinely incorrect solutions. Through experiments on mathematical reasoning, stepwise error detection, self-correction, and long-chain reasoning tasks, the authors demonstrate that CDG training improves the rationality of well-aligned models like LLaMA-3.1-8B-Instruct and Qwen2.5-1.5B-Instruct in their reasoning processes. The method outperforms alternative approaches like Expert Iteration and Step-DPO, highlighting the potential of self-play language games as a promising training paradigm beyond instruction tuning and preference optimization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback on our paper. Our responses to your questions are as follows:\n\n> **Claims And Evidence:**\n> The paper claims to improve \"rationality in reasoning\" but doesn't directly measure this construct beyond task performance.\n\nAs mentioned in the introduction, recent studies suggest that LLMs often rely on pattern matching rather than truly understanding their reasoning processes. However, directly assessing whether a model reasons in a rational manner is inherently difficult. To address this, we evaluate performance across multiple tasks and dimensions to better capture whether the model is reasoning more rationally.\n\nOur experimental setup is specifically designed for this purpose. For example, the error detection task is conduct on problems the model has already demonstrated proficiency in solving. Just as human experts who understand a solution can identify and correct their own occasional mistakes, our setup tests whether the model can do the same. Failure in such cases indicates a lack of understanding of its own reasoning, rather than a general capability gap.\n\n> **Claims And Evidence:**\n> The improvements, while consistent, are sometimes modest; the long-term stability of the improvements and potential for overfitting to the game format, given that GSM8K and MATH are somewhat similar tasks.\n> **Methods And Evaluation Criteria:**\n> Evaluation criteria suffer from limited scope.\n\nWe agree that evaluating generalization beyond GSM8K and MATH is important. To this end, we conduct additional experiments on tasks outside the original training distribution, including diverse reasoning benchmarks (LogiQA, ARC-Easy, and ARC-Challenge) and harder mathematical problems (Minerva-MATH). Results are shown below:\n\n| Model| Minerva| LogiQA| ARC-E| ARC-C|\n|-|-|-|-|-|\n| Llama3.1-8B-Instruct| 31.24| 31.18| 79.76| 54.95|\n| + CDG| **33.10**| **32.26**| **80.05**| **55.80**|\n\nThese results show that CDG yields consistent gains across different domains, suggesting it enhances general reasoning capabilities.\n\nWe further evaluate CDG on more challenging mathematical problems where the base model performs poorly. Specifically, we train CDG on the Dapo-17k dataset [1] and test on the **AIME-24**benchmark. Results are averaged over 10 runs due to the small test size:\n\n| Model| Pass@1| Pass@4| Pass@8|\n|-|-|-|-|\n| Qwen2.5-1.5B-Instruct| 1.67| 6.67| 6.67|\n| + CDG-1| 2.67| 3.33| 3.33|\n| + CDG-2| 5.33| 6.67| 6.67|\n| + CDG-3| **5.67**| **10.0**| **16.7**|\n\nCDG-3 improves Pass@1 from 1.67 to 5.67 using just 18k self-generated examples. In contrast, Qwen2.5-1.5B-MATH-Instruct reaches 10.0 after training on 2.5M CoT-annotated samples with GRPO. This demonstrates CDG’s data efficiency and strong gains.\n\n> **Experimental Designs Or Analyses:**\n> There's insufficient reporting of statistical significance (e.g., standard errors); and the paper lacks proper controls for training computation across compared methods.\n\nTo assess statistical significance, we conduct t-tests and report p-values in the caption of Table 1. All four tasks show significant improvements (p < 0.05).\n\nFor Expert Iteration, we report the best result under an equal training budget. For Step-DPO, we follow the original 3-round setup using 10k GPT-4o-annotated step-level preference data.\n\n> **Essential References Not Discussed:**\n> There are two particularly essential related works not adequately cited or discussed.\n> Besides, the authors should carefully address the similarities and differences between Kirchner et al. (2024).\n\nThank you for pointing this out. We acknowledge that Saunders et al. (2022) and McAleese et al. (2024) are relevant, as they also explore LLM-based self-critiquing and error detection. We will cite and discuss them in the revision.\n\nAs for Kirchner et al. (2024), while both approaches involve multi-agent interaction, the objectives differs. Their goal is to improve output legibility by training a helpful prover to outperform a sneaky one. In contrast, our focus is on enhancing the model’s understanding of its own reasoning by training a prover to distinguish helpful from misleading feedback—aiming for developing rational reasoning abilities than readability. We will clarify this distinction in the revision.\n\n> **Questions For Authors:**\n> Do you use three separate models or a single model?\n> What do you mean by “both before and after CDG training”?\n\nQ1: We use three separate models during training. Prompting alone results in noticeable style differences between the helpful and misleading critics, making them too easy for the prover to distinguish.\n\nQ2: We compare two settings: (1) finetuning the base model on long-chain reasoning data (distilled from QwQ-32B-Preview), and (2) finetuning the model after CDG training using the exact same long-chain reasoning data.\n\nReferences:\n[1] Yu Q, Zhang Z, Zhu R, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale[J]. arXiv preprint arXiv:2503.14476, 2025.", "claims_and_evidence": "The claims in the submission are generally supported by clear and convincing evidence through comprehensive experiments across multiple reasoning tasks. The ablation studies effectively demonstrate that both helpful and misleading critics are necessary components of CDG, while comparisons with other RL methods convincingly show CDG's advantages over alternatives. However, some claims could benefit from stronger evidence: the paper claims to improve \"rationality in reasoning\" but doesn't directly measure this construct beyond task performance; the improvements, while consistent, are sometimes modest (e.g., ~1.5 percentage points on GSM8K); and the long-term stability of the improvements and potential for overfitting to the game format aren't thoroughly explored, given that GSM8K and MATH are somewhat similar tasks.", "methods_and_evaluation_criteria": "The proposed methods are conceptually sound for improving reasoning, but the evaluation criteria suffer from limited scope. While the paper uses established mathematical reasoning benchmarks (GSM8K and MATH500), this narrow focus on mathematical reasoning alone raises significant concerns about generalizability. The absence of diverse reasoning datasets spanning different domains (such as commonsense reasoning, logical deduction, scientific reasoning, or counterfactual reasoning) leaves a critical gap in understanding whether CDG's benefits extend beyond mathematical problem-solving or are domain-specific.", "theoretical_claims": "The paper does not contain formal mathematical proofs requiring verification, as it primarily presents algorithmic approaches and empirical results.", "experimental_designs_or_analyses": "The evaluation framework is methodologically sound for measuring different aspects of reasoning. However, I identified several validity concerns: the limited dataset diversity focused only on mathematical reasoning restricts generalizability claims; there's insufficient reporting of statistical significance (e.g., standard errors); and the paper lacks proper controls for training computation across compared methods.", "relation_to_broader_scientific_literature": "The paper's contributions connect to several established research areas in the broader scientific literature like RLHF (Christiano et al., 2017; Ouyang et al., 2022),  self-improvement methods (Huang et al., 2023; Zelikman et al., 2022), and chain-of-Thought reasoning (Wei et al., 2022; Yao et al., 2023).", "essential_references_not_discussed": "There are two particularly essential related works not adequately cited or discussed: \"Self-critiquing models for assisting human evaluators\" by Saunders et al. (2022) and \"LLM Critics Help Catch LLM Bugs\" by McAleese et al. (2024). Besides, the authors should carefully address the similarities and differences between their approach and Kirchner et al. (2024), particularly regarding the training dynamics and how both methods aim to improve the legibility and reliability of model outputs.", "other_strengths_and_weaknesses": "Please see the above reviews.", "other_comments_or_suggestions": "Please see the above reviews.", "questions_for_authors": "- Do you use three separate models or a single model with different prompts for the three roles during training?\n- What do you mean by “both before and after CDG training”?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aWzpx5rG2W", "reviewer": "Reviewer_Mguk", "summary": "This paper introduces the Critic-Discernment Game (CDG), a self-play approach to improve reasoning in language models without human supervision. In CDG, three roles interact: a prover that solves problems, a helpful critic that identifies errors in incorrect solutions, and a misleading critic that fabricates errors in correct solutions. Through reinforcement learning, the prover learns to maintain correct answers against misleading critiques while correcting genuine errors. Experiments on mathematical reasoning, error detection, self-correction, and long-chain reasoning tasks demonstrate consistent performance improvements, in well-aligned models like LLaMA-3.1-8B-instruct, showing that this game-based approach effectively enhances reasoning capabilities beyond traditional fine-tuning methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback on our paper. Our response to your questions is as follows:\n\n> **Claims And Evidence:**\n> One limitation is that the mechanism by which CDG improves understanding of reasoning processes is somewhat indirect. It's unclear that if LLMs are truly doing the reasoning and verifications or just pretending to do so.\n\nIndeed, it is difficult to directly assess whether a model reasons in a rational manner. To address this, we evaluate performance across multiple tasks and dimensions to better capture whether the model reasons more rationally.\n\nOur experimental setup is specifically designed for this purpose. For example, the error detection task uses problems the model already solves correctly. Just as human experts who understand a solution can identify and correct their own occasional mistakes, our setup tests whether the model can do the same. Failure in such cases indicates a lack of understanding of its own reasoning rather than a general capability gap.\n\n> **Methods And Evaluation Criteria:**\n> Outdated and easy dataset (GSM8k) and data sizes. The author should try to exploit more challenging mathematical reasoning dataset such as AIME. \n\nYes, with the rapid progress of reasoning models, GSM8K and MATH are no longer sufficient for evaluating mathematical capabilities. To address this, we train CDG on Dapo-17k [1], a more challenging dataset. The results on the **AIME-24** test set are shown below:\n\n| Model| Pass@1| Pass@4| Pass@8|\n|--|--|--|--|\n| Qwen2.5-1.5B-Instruct| 1.67| 6.67| 6.67|\n| + CDG-1| 2.67| 3.33| 3.33|\n| + CDG-2| 5.33| 6.67| 6.67|\n| + CDG-3| **5.67**| **10.0**| **16.7**|\n\nWe average the results over 10 runs due to the small test size. CDG-3 improves performance from 1.67 to 5.67 using only 18k self-generated examples. In comparison, Qwen2.5-1.5B-MATH-Instruct reaches 10.0 after training on 2.5M CoT-annotated examples with GRPO. This highlights the data efficiency of CDG and its effectiveness in improving initially low-performing tasks.\n\nWe further evaluate the CDG-trained models in the paper on harder math tasks (Minerva-MATH) and diverse reasoning benchmarks (LogiQA, ARC-Easy/Challenge), demonstrating the generality of our method:\n\n| Model| Minerva| LogiQA| ARC-E| ARC-C|\n|--|--|--|--|--|\n| Llama3.1-8B-Instruct| 31.24| 31.18| 79.76| 54.95|\n| + CDG| **33.10**| **32.26**| **80.05**| **55.80**|\n\n\n> **Experimental Designs Or Analyses:**\n> The computational overhead (efficiency) problems should be discussed when comparing with other RL methods since the use of multiple LLM instances.\n\nWe appreciate the reviewer’s concern regarding computational efficiency. We compare our method with traditional RL approaches such as Expert Iteration, from both inference (rollout) and training perspectives.\n\nDuring the rollout stage, the prover first generate an initial solution. Based on this solution, the critic then generates multiple critiques. The prover subsequently produces multiple second-round responses conditioned on these critiques. The overall rollout cost increases by approximately 20% compared to Expert Iteration to produce the same amount of training data.\n\nDuring training, the additional computational cost compared to Expert Iteration comes from training the critic. The amount of training tokens for the critic is about 25% of that for the prover.\n\nOverall, while our approach involves more LLM instances, the practical computational overhead remains moderate due to reuse of generated content. In addition, our training does not require the reference model and critic model commonly used in many RL algorithms.\n\n> **Questions For Authors**\n\nA1: Yes, as noted in Methods and Evaluation Criteria, our method significantly improves performance on AIME24, a much more challenging dataset. The average score increased from 1.67 (pre-trained) to 5.67 (CDG-trained), demonstrating stronger capabilities on complex problems. Additionally, Section 4.4 shows clear gains on long-chain CoT tasks with average reasoning lengths exceeding 1000 tokens.\n\nA2: The prompt template for determining whether a critique is helpful or misleading is detailed in the appendix. As specified, if the model deems the critic helpful, it outputs the corrected answer in the boxed section; otherwise, it returns “This critic is not critical.” Rewards are applied accordingly using a rule-based scheme.\n\nA3: Beyond performance, we observe increased self-checking behavior. For instance, the usage of the word “check” increases by 22%, and “verify” by 69%, indicating enhanced internal verification.\n\nA4: For the same question, we sample multiple solutions from the prover. In our setup, the helpful critic only engages with incorrect solutions—where the reasoning must contain errors—while the misleading critic targets correct solutions, attempting to induce mistakes.\n\nReferences:\n[1] Yu Q, Zhang Z, Zhu R, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale[J]. arXiv preprint arXiv:2503.14476, 2025.", "claims_and_evidence": "One limitation is that the mechanism by which CDG improves understanding of reasoning processes is somewhat indirect - the improvements in performance are clear, but the internal changes to model reasoning are primarily inferred from these performance gains rather than directly measured or analyzed, so it's unclear that if LLMs are truly doing the reasoning and verifications or just pretending to do so.", "methods_and_evaluation_criteria": "Outdated and easy dataset (GSM8k) and data sizes (  200 positive and 200 negative samples) is somewhat concerning; The author should try to exploit more challenging mathematical reasoning dataset such as AIME to avoid potential data leakage to strength the arguments. Lack of concrete discussions on how the prover evaluates whether a critique is helpful or misleading.", "theoretical_claims": "The mathematical formulation in Section 3 includes game modeling, reward functions, and reinforcement learning objectives, but these are definitions rather than theorems requiring proof.", "experimental_designs_or_analyses": "The ablation studies isolate the contribution of each critic type, and comparisons with other RL methods use the same training data and budget, but simiarly to what I discussed on Claims and Evidences, one potential issue is that most evaluations assume that improving task performance equates to improving reasoning rationality, which may not always be true, but their comprehensive task suite mitigates this concern. Also the computational overhead (efficiency) problems should be discussed when comparing with other RL methods since the use of multiple LLM instances.", "relation_to_broader_scientific_literature": "Process-based reasoning improvement: Extends work by Lightman et al. (2023) and Lai et al. (2024) on stepwise supervision, but without requiring human/superior model annotations; Builds on outcome-based methods (Anthony et al., 2017; Gulcehre et al., 2023) and step-level supervision (Zhang et al., 2025; Uesato et al., 2022), but uniquely derives rewards from game rules rather than explicit supervision", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "This work uniquely combines self-play games with RL for reasoning improvement, offering a novel training paradigm beyond traditional instruction tuning and preference optimization. See above for Weakness;", "other_comments_or_suggestions": "No.", "questions_for_authors": "1: How does the approach scale with increasingly complex reasoning tasks? Is there a limit to the reasoning length where CDG becomes less effective? \n2: Could you elaborate on how the prover evaluates whether a critique is helpful or misleading? Is this purely reward-driven or are there explicit criteria?\n3: What specific improvements did you observe in the reasoning process itself, beyond task performance metrics?\n4: How does the approach handle ambiguous problems where multiple valid reasoning paths exist?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1FsoaJ57Hb", "reviewer": "Reviewer_GFAY", "summary": "The paper presents the Critic-Discernment Game (CDG), a self-play approach that enhances the reasoning of large language models (LLMs). In CDG, a \"prover\" generates solutions, while two critics—helpful and misleading—offer feedback. The prover learns to correct mistakes from the helpful critic and resist misleading critiques through reinforcement learning. The method is tested on tasks like mathematical reasoning, error detection, self-correction, and long-chain reasoning, showing significant improvements over baseline models. CDG enables LLMs to improve their reasoning capabilities without human supervision, demonstrating the power of self-play and reinforcement learning in model training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Experimental Designs Or Analyses:**\n> While the paper shows improvements on GSM8K and MATH500, one limitation is the lack of explicit testing for generalization to other problem domains or tasks outside of mathematical reasoning.\n> **Weakness:**\n> While the paper reports significant improvements on tasks related to mathematical reasoning and self-correction, the focus on these domains means the broader applicability of the CDG method remains unclear.\n\nWe agree that evaluating generalization beyond GSM8K and MATH is important. To this end, we conduct additional experiments on tasks outside the original training distribution, including diverse reasoning benchmarks (LogiQA, ARC-Easy, and ARC-Challenge) and more difficult mathematical problems (Minerva-MATH). Results are shown below:\n\n| Model| Minerva| LogiQA| ARC-E| ARC-C|\n|-|-|-|-|-|\n| Llama3.1-8B-Instruct| 31.24| 31.18| 79.76| 54.95|\n| + CDG| **33.10**| **32.26**| **80.05**| **55.80**|\n\nThese results show that CDG yields consistent gains across different domains, suggesting it enhances general reasoning capabilities.\nWe also train CDG on Dapo-17k [1], a harder dataset. Results on the challenge **AIME-24** test set are shown below:\n\n| Model| Pass@1| Pass@4| Pass@8|\n|--|--|--|--|\n| Qwen2.5-1.5B-Instruct| 1.67| 6.67| 6.67|\n| + CDG-1| 2.67| 3.33| 3.33|\n| + CDG-2| 5.33| 6.67| 6.67|\n| + CDG-3| **5.67**| **10.0**| **16.7**|\n\nWe average over 10 runs due to the small test size. CDG-3 improves performance from 1.67 to 5.67 using only 18k self-generated examples. For comparison, Qwen2.5-1.5B-MATH-Instruct reaches 10.0 after 2.5M CoT-annotated examples and GRPO. This highlights CDG’s data efficiency and effectiveness on low-performing tasks.\n\n> **Weakness**\n> The paper does not provide an extensive discussion on the sensitivity of the model’s performance to hyperparameters (e.g., the thresholds used in the RL setup). \n\nThe ReST approach involves very few hyperparameters and does not require a reference or critic model. Unlike PPO or DPO, where improper values of $\\beta$ can lead to training collapse (e.g., duplicate or collapsed generations), ReST does not suffer from such instability. As for selecting the threshold $\\tau$, we choose it based on data balance considerations and observed performance during training.\n\n> **Weakness**\n> The training process involving multiple self-play rounds and critics is resource-intensive. While the paper demonstrates the effectiveness of CDG, it does not provide a comprehensive analysis of the computational cost or the training time involved. It would be helpful to include a comparison of the cost-benefit tradeoff between CDG and other methods, such as instruction tuning or preference optimization.\n\nWe appreciate the reviewer’s concern regarding computational efficiency. We compare our method with traditional RL approaches such as Expert Iteration, from both inference (rollout) and training perspectives.\n\nDuring the rollout stage, the prover first generate an initial solution. Based on this solution, the critic then generates multiple critiques. The prover subsequently produces multiple second-round responses conditioned on these critiques. The overall rollout cost increases by approximately 20% compared to Expert Iteration to produce the same amount of training data.\n\nDuring training, the additional computational cost compared to Expert Iteration comes from training the critic. The amount of training tokens for the critic is about 25% of that for the prover.\n\nOverall, while our approach involves more LLM instances, the practical computational overhead remains moderate due to reuse of generated content. In addition, our training does not require the reference model and critic model commonly used in many RL algorithms.\n\nIn the experiment for Table 3, for Expert Iteration, we report the best result under equal training budgets. For Step-DPO, we follow the original 3-round setup using 10k GPT-4o-annotated step-level preferences data.\n\n> **Other Comments Or Suggestions:**\n> While the experiments are well-conducted, some aspects of the methodology (e.g., dataset creation, sampling strategies) are not presented in full detail. For example, the paper mentions the use of regular expressions and the SymPy grader for evaluating solutions, but further clarification would improve transparency.\n\nMany details of our dataset construction, sampling strategies, and hyperparameter settings are included in the appendix due to space limitations.\nUsing the SymPy grader to evaluate mathematical answers is a common practice; we follow the same evaluation setup as in qwen2.5-math-instruct [2]. We will also release the detailed evaluation code.\n\nReferences:\n[1] Yu Q, Zhang Z, Zhu R, et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale[J]. arXiv preprint arXiv:2503.14476, 2025.\n[2] Yang A, Zhang B, Hui B, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement[J]. arXiv preprint arXiv:2409.12122, 2024.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No issues.", "experimental_designs_or_analyses": "While the paper shows improvements on GSM8K and MATH500, one limitation is the lack of explicit testing for generalization to other problem domains or tasks outside of mathematical reasoning.", "relation_to_broader_scientific_literature": "The key contribution of the paper is related to the following prior research:\n\n1.\tSelf-play and Reinforcement Learning for Reasoning. Self-play in LLMs has been explored in other contexts, such as the Adversarial Taboo Game for improving pre-trained models’ reasoning abilities (Cheng et al., 2024b), and the Spar framework (Cheng et al., 2024a), which improves instruction-following by training with self-play.\n\n2.\tImproving Reasoning through Feedback. Process-based Reward Models (PRMs) (Zhang et al., 2025; Uesato et al., 2022a) and Step-DPO (Lai et al., 2024) also focus on improving reasoning by using feedback at the step level.\n\n3.\tMathematical Reasoning and Long-Chain Reasoning. Math Word Problem Solving (Cobbe et al., 2021a) and Mathematical Reasoning in LLMs (Gulcehre et al., 2023) have also seen substantial improvements from techniques like process supervision and feedback-based training.  Chain-of-Thought Prompting (Wei et al., 2022) and OpenAI’s Chain-of-Thought Models (Wu et al., 2024a): These methods have been shown to improve performance on long-form reasoning tasks by encouraging models to reason through multiple steps explicitly.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n\n1.\tThe paper introduces a novel Critic-Discernment Game (CDG) to enhance the rationality of reasoning in LLMs through a self-play framework. While self-play has been used effectively in domains like games (e.g., AlphaGo), its application to improving error correction and long-chain reasoning in LLMs is a unique contribution.\n\n2.\tThe authors demonstrate significant improvements across multiple tasks, including mathematical reasoning, stepwise error detection, self-correction, and long-chain reasoning.\n\n3.\tBy incorporating a self-reflection mechanism with iterative feedback, the paper proposes a method that could significantly improve performance on challenging problems that require several reasoning steps, a major strength in advancing LLM capabilities.\n\nWeakness:\n\n1.\tWhile the paper reports significant improvements on tasks related to mathematical reasoning and self-correction, the focus on these domains means the broader applicability of the CDG method remains unclear.\n\n2.\tThe paper does not provide an extensive discussion on the sensitivity of the model’s performance to hyperparameters (e.g., the thresholds used in the RL setup). These hyperparameters can significantly affect the outcome of reinforcement learning-based methods, and without clear validation or tuning, the results may not be as stable or generalizable in different contexts.\n\n3.\tThe training process involving multiple self-play rounds and critics is resource-intensive. While the paper demonstrates the effectiveness of CDG, it does not provide a comprehensive analysis of the computational cost or the training time involved. It would be helpful to include a comparison of the cost-benefit tradeoff between CDG and other methods, such as instruction tuning or preference optimization.", "other_comments_or_suggestions": "While the experiments are well-conducted, some aspects of the methodology (e.g., dataset creation, sampling strategies) are not presented in full detail. For example, the paper mentions the use of regular expressions and the SymPy grader for evaluating solutions, but further clarification on how these tools are integrated into the RL training process would improve transparency.", "questions_for_authors": "Please see the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a self-play framework in which a language model (the prover) generates solutions and is then challenged by helpful or misleading critiques to enhance its reasoning robustness. Experiments show that this Critique-Driven Game (CDG) training improves models’ ability to detect, defend, and correct reasoning errors across mathematical and long-chain reasoning tasks.

While the reviewers are debating between weak accept and weak reject, I am actually leaning somewhat higher than weak accept. Reading the main concerns of the reviewers, I don't find them very substantial. Yes, the proposed method is exemplified to be effective for similar tasks, but isn't this what most machine learning does ? I find the novelty and potential improvements of the methods to be of high interest to the ICML crowd, while the concerns are typical to many other ML papers.
Decision: Accept (poster)